Skip to content

fix: add S3 operation timeouts to prevent scheduler hangs#44

Open
NikhilSharmaWe wants to merge 1 commit into
subsquid:masterfrom
NikhilSharmaWe:fix/s3-operations-timeouts
Open

fix: add S3 operation timeouts to prevent scheduler hangs#44
NikhilSharmaWe wants to merge 1 commit into
subsquid:masterfrom
NikhilSharmaWe:fix/s3-operations-timeouts

Conversation

@NikhilSharmaWe

Copy link
Copy Markdown

Summary

  • Configure AWS SDK TimeoutConfig on the shared S3 client (60s per attempt, 180s per operation, 60s read timeout)
  • Hung S3 calls now return errors within a bounded time; failed datasets are skipped per Skip datasets that fail to load instead of aborting the entire scheduler run #41 instead of freezing the cron
  • Check dataset_load_timeout_sec before each listing/download batch, not only after batch completion
  • Add unit and runtime tests for timeout config wiring and bounded list_objects failure

Closes #14

Problem

The scheduler could hang indefinitely at "Downloading chunks from …" when an S3/R2 list_objects_v2 call stalled. AWS SDK for Rust has no default operation or read timeout (only a 3.1s connect timeout), so .send().await could block forever. A single hung dataset blocked the entire cron run and prevented assignment publication for all datasets.

Solution

Apply TimeoutConfig in S3Args::config(), which is shared by both S3Storage (listing + summary downloads) and Uploader (assignment uploads). On timeout, the dataset load returns an error and is skipped; other datasets continue and the assignment is still published.

Also tighten dataset_load_timeout_sec (PR #35) to evaluate the deadline before each batch, avoiding an extra list+download cycle after the budget is exhausted.

Test plan

  • cargo test --all-features (25 tests pass)
  • cargo clippy --all-targets --all-features
  • s3_timeout_config_sets_expected_values — verifies timeout values
  • s3_list_fails_within_bounded_time_on_unreachable_endpoint — proves list_objects_v2 fails in ~3s, not forever

Signed-off-by: Nikhil Sharma <nikhilsharma230303@gmail.com>
@NikhilSharmaWe

Copy link
Copy Markdown
Author

@kalabukdima, please take a look. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Set timeout for listing chunks

1 participant