fix: avoid crashes in job bundle submission edge cases#1290
Open
crowecawcaw wants to merge 5 commits into
Open
fix: avoid crashes in job bundle submission edge cases#1290crowecawcaw wants to merge 5 commits into
crowecawcaw wants to merge 5 commits into
Conversation
When create_job_from_job_bundle is called with debug_snapshot_dir for a bundle that has no job attachments, the attachments block never runs, so the asset_manager local is never assigned. It was then passed unconditionally to _save_debug_snapshot, raising UnboundLocalError and making snapshotting crash. Initialize asset_manager to None before the conditional and accept Optional[S3AssetManager] in _save_debug_snapshot, which only dereferences it when the create_job args carry attachments. Signed-off-by: Stephen Crowe <6042774+crowecawcaw@users.noreply.github.com>
wait_for_create_job_to_complete read job["lifecycleStatusMessage"] with subscript access on both the success and failure branches. A get_job response that omits that field raised KeyError, turning an otherwise successful submission into a crash. Read it with .get(..., "") so a missing message degrades to an empty string instead of raising. Signed-off-by: Stephen Crowe <6042774+crowecawcaw@users.noreply.github.com>
_upload_attachments and _snapshot_attachments both accept a config parameter, and _upload_attachments uses it to select the telemetry client for the upload summary. create_job_from_job_bundle never passed its config through, so upload telemetry used the default on-disk config instead of the caller-supplied one. Forward config to both call sites. Signed-off-by: Stephen Crowe <6042774+crowecawcaw@users.noreply.github.com>
Adds a unit test for the debug-snapshot-WITH-attachments combination: verifies _snapshot_attachments is used (not _upload_attachments), receives the caller's config, the snapshot's create_job_args.json contains the attachments, and the generated submit_job.sh includes the 'aws s3 cp' upload commands. This locks in the non-None side of the Optional asset_manager change and the config forwarding on the snapshot path. Also tightens the no-attachments snapshot test to assert the generated script omits 'aws s3 cp' while still containing the create-job command. Red-green: the new test fails on the pre-fix parent commit because _snapshot_attachments was called without config. Signed-off-by: Stephen Crowe <6042774+crowecawcaw@users.noreply.github.com>
crowecawcaw
marked this pull request as ready for review
July 24, 2026 16:33
leon-li-inspire
approved these changes
Jul 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes:
What was the problem/requirement? (What/Why)
Three crash/robustness bugs in job-bundle submission (
api/_submit_job_bundle.py):asset_managerwas only assigned inside the attachments block but passed unconditionally to_save_debug_snapshot→UnboundLocalErrorwhen submitting a bundle with no attachments and a debug snapshot.job["lifecycleStatusMessage"]was read without a fallback →KeyErrormade a successful submit look like a crash.configwasn't forwarded to_upload_attachments/_snapshot_attachments, so telemetry used the wrong config.What was the solution? (How)
asset_manager = Nonebefore the conditional and acceptOptional[S3AssetManager]in_save_debug_snapshot(it's only dereferenced inside the attachments branch).job.get("lifecycleStatusMessage", "")in both waiter branches.config=configto both attachment helpers.What is the impact of this change?
Debug-snapshot submissions without attachments no longer crash, successful submits report success, and attachment telemetry uses the correct config.
How was this change tested?
Added red-green unit tests for each: no-attachments debug snapshot, missing
lifecycleStatusMessage, and config forwarded to the attachment helpers.test_job_bundle_submission.pyand adjacent bundle-submit tests (58 passed).Was this change documented?
Does this PR introduce new dependencies?
Is this a breaking change?
No.
Does this change impact security?
No. (Known-path containment (C-known-path) and the stack-trace sanitizer are handled in separate PRs; the session-context race and debug-snapshot command-injection findings are out of scope here.)
Testing
All verification is automated unit tests (no live AWS calls; deadline/boto3 clients mocked), each proven red-green against the pre-fix parent commit:
Fix (a) —
asset_managerUnboundLocalError:test_create_job_from_job_bundle_debug_snapshot_no_attachments: debug snapshot of a bundle with no attachments completes cleanly; snapshot files are written, no job is submitted, and the generatedsubmit_job.shcontainsaws deadline create-jobbut correctly omitsaws s3 cplines. Pre-fix failure:UnboundLocalError: cannot access local variable 'asset_manager'.test_create_job_from_job_bundle_debug_snapshot_with_attachments(new): the other side of theOptional[S3AssetManager]change — a snapshot with attachments uses_snapshot_attachments(not_upload_attachments) with a bound asset manager, writesattachmentsintocreate_job_args.json, and emits theaws s3 cpupload commands in the generated script.Fix (b) — missing
lifecycleStatusMessage:test_wait_for_create_job_to_complete_missing_status_message, parametrized over both waiter branches: aget_jobresponse omittinglifecycleStatusMessagereturns(True, "")forREADYand(False, "")forCREATE_FAILED. Pre-fix failure:KeyError: 'lifecycleStatusMessage'in each branch. (Note: the sharedMockDeadlineBackendused by UI/e2e tests always includeslifecycleStatusMessage, so this unit test is the permanent coverage for the omission case.)Fix (c) —
configforwarding:test_create_job_from_job_bundle_forwards_config_to_upload_attachments: normal submit forwards the caller'sconfigobject to_upload_attachments(verified by identity,is). Pre-fix failure:configabsent from the call kwargs.test_create_job_from_job_bundle_debug_snapshot_with_attachments(new) verifies the same for_snapshot_attachmentson the snapshot path; it fails pre-fix withconfigabsent.Suite runs:
test_job_bundle_submission.py(41 tests) plus the fulltest/unit/deadline_client/api/+test_cli_bundle_submit*.pysuites (578 passed).hatch run fmtandhatch run lint(ruff check, ruff format, mypy) pass; full CI matrix is green.Not covered (manual-only, not automatable): executing a generated
submit_job.sh/submit_job.batagainst a live farm, and observing a realCreateJobwhoseGetJobresponse omitslifecycleStatusMessage. These are optional hardening checks; the mocked flows exercise the exact code paths changed.