Skip to content

feat: add is_finished method to RequestQueueClient interface#1982

Open
Mantisus wants to merge 1 commit into
apify:masterfrom
Mantisus:queue-client-is-finished
Open

feat: add is_finished method to RequestQueueClient interface#1982
Mantisus wants to merge 1 commit into
apify:masterfrom
Mantisus:queue-client-is-finished

Conversation

@Mantisus

Copy link
Copy Markdown
Collaborator

Description

  • Adds an is_finished method to the RequestQueueClient interface. It's not a breaking change. The base class defaults is_finished to is_empty when it isn't overridden. Splits the queue predicates. is_empty means there are no requests available to fetch. is_finished means all requests are fully processed (nothing to fetch and nothing in progress).
  • Implements is_finished in all built-in RequestQueueClient.
  • Updates is_empty in all built-in RequestQueueClient to match the new meaning.
  • Affects the AutoscaledPool. It now skips scheduling new tasks when no requests are available to fetch. For example, while the only request is being processed.

Issues

Testing

  • Added a test for RequestQueue that checks the is_empty and is_finished values through the queue lifecycle. It covers all built-in clients.
  • Added a regression test that checks the AutoscaledPool doesn't schedule new tasks while the only request is being processed.

@Mantisus Mantisus requested review from janbuchar and vdusek June 21, 2026 21:23
@Mantisus Mantisus self-assigned this Jun 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add an is_finished method to the StorageClient interface Crawler can get softlocked when consuming the RequestQueue

2 participants