fix: add S3 operation timeouts to prevent scheduler hangs#44
Open
NikhilSharmaWe wants to merge 1 commit into
Open
fix: add S3 operation timeouts to prevent scheduler hangs#44NikhilSharmaWe wants to merge 1 commit into
NikhilSharmaWe wants to merge 1 commit into
Conversation
Signed-off-by: Nikhil Sharma <nikhilsharma230303@gmail.com>
Author
|
@kalabukdima, please take a look. Thanks! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
TimeoutConfigon the shared S3 client (60s per attempt, 180s per operation, 60s read timeout)dataset_load_timeout_secbefore each listing/download batch, not only after batch completionlist_objectsfailureCloses #14
Problem
The scheduler could hang indefinitely at
"Downloading chunks from …"when an S3/R2list_objects_v2call stalled. AWS SDK for Rust has no default operation or read timeout (only a 3.1s connect timeout), so.send().awaitcould block forever. A single hung dataset blocked the entire cron run and prevented assignment publication for all datasets.Solution
Apply
TimeoutConfiginS3Args::config(), which is shared by bothS3Storage(listing + summary downloads) andUploader(assignment uploads). On timeout, the dataset load returns an error and is skipped; other datasets continue and the assignment is still published.Also tighten
dataset_load_timeout_sec(PR #35) to evaluate the deadline before each batch, avoiding an extra list+download cycle after the budget is exhausted.Test plan
cargo test --all-features(25 tests pass)cargo clippy --all-targets --all-featuress3_timeout_config_sets_expected_values— verifies timeout valuess3_list_fails_within_bounded_time_on_unreachable_endpoint— proveslist_objects_v2fails in ~3s, not forever