fix: make job trace-schedule robust to in-flight/partial jobs#1293
Open
crowecawcaw wants to merge 4 commits into
Open
fix: make job trace-schedule robust to in-flight/partial jobs#1293crowecawcaw wants to merge 4 commits into
crowecawcaw wants to merge 4 commits into
Conversation
The `job trace-schedule` summary computed several percentages and a per-action overhead by dividing by the total session duration and the session-action count. For an in-flight job with no completed durations (session_total_duration == 0) or with zero session actions, this raised a ZeroDivisionError and crashed the command. Add a _percent_of helper that returns "N/A" for a zero denominator, and guard the overhead-per-action division so it reports 0 instead of crashing. Signed-off-by: Stephen Crowe <6042774+crowecawcaw@users.noreply.github.com>
Partial job data crashed `job trace-schedule` with KeyErrors: - _get_all_sessions sorted sessions by session["startedAt"], but a session that hasn't started yet omits that key. Sort such sessions last via .get with a datetime.max fallback. - _build_trace_events read session["step"]["name"], but the step may be absent when BatchGetStep returns a partial result (a step could not be fetched). Fall back to a "<Unknown Step>" placeholder. Signed-off-by: Stephen Crowe <6042774+crowecawcaw@users.noreply.github.com>
_build_trace_events dereferenced session['startedAt'] unconditionally, so a session without startedAt (now tolerated by the sort) would still KeyError. Skip such sessions entirely, matching the per-action guard. Signed-off-by: Stephen Crowe <6042774+crowecawcaw@users.noreply.github.com>
crowecawcaw
marked this pull request as ready for review
July 22, 2026 17:05
kavmur
approved these changes
Jul 24, 2026
leon-li-inspire
approved these changes
Jul 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes:
What was the problem/requirement? (What/Why)
deadline job trace-schedulecrashed on in-flight jobs: the summary's percentage/overhead math divided by a zero session-total duration (ZeroDivisionError), and it hard-accessedsession["startedAt"]andsession["step"], raisingKeyErroron sessions that hadn't started or whose step hadn't been fetched.What was the solution? (How)
_percent_of(numerator, denominator)helper that returns"N/A"when the denominator is zero (an in-flight job with no completed durations yet), used for all percentage lines in the summary..get("startedAt", datetime.max)so unstarted sessions sort last, and usesession.get("step", {}).get("name", "<Unknown Step>")for the step name.What is the impact of this change?
trace-scheduleruns cleanly on jobs that are still running or have partial data, showingN/Afor percentages that can't yet be computed instead of crashing.How was this change tested?
Added unit tests (new file — none existed for this module) covering zero-denominator summary math, zero session-action count, missing
startedAt, and a session without a fetchedstep.test/unit/deadline_client/cli/test_cli_trace_schedule.py(4 passed).Was this change documented?
Does this PR introduce new dependencies?
Is this a breaking change?
No.
Does this change impact security?
No.
Testing
In addition to the unit tests, the full command was manually verified end-to-end with mocked AWS APIs (no live service), driving
deadline job trace-schedule --trace-format chrome --trace-file ...through Click'sCliRunnerwith fixture data for an in-flight job:startedAtbut noendedAt(still running).startedAt == endedAt).endedAt, a running action with noendedAt, and a queued action with nostartedAt.startedAtat all.ResourceNotFoundException) for one step.Results: exit code 0, no exceptions; the summary printed real percentages, the partial-step warning was surfaced, and the written Chrome trace file was validated — every event has a non-negative integer
ts(anddurforXevents), sessionB/Epairs are balanced, the not-started session and not-started action emit no events, the failed step falls back to<Unknown Step>, in-progress markers appear, and the zero-duration span is emitted withdur: 0. A second degenerate run (job started, zero sessions) also exited 0 and printedN/Afor all percentage lines instead of raisingZeroDivisionError.Not yet verified against a real running job in a live farm; a hands-on check tracing a job mid-render (while some sessions are still
STARTING) would fully close the loop.