Skip to content

fix: make job trace-schedule robust to in-flight/partial jobs#1293

Open
crowecawcaw wants to merge 4 commits into
aws-deadline:mainlinefrom
crowecawcaw:review-fix/trace-schedule
Open

fix: make job trace-schedule robust to in-flight/partial jobs#1293
crowecawcaw wants to merge 4 commits into
aws-deadline:mainlinefrom
crowecawcaw:review-fix/trace-schedule

Conversation

@crowecawcaw

@crowecawcaw crowecawcaw commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Fixes:

What was the problem/requirement? (What/Why)

deadline job trace-schedule crashed on in-flight jobs: the summary's percentage/overhead math divided by a zero session-total duration (ZeroDivisionError), and it hard-accessed session["startedAt"] and session["step"], raising KeyError on sessions that hadn't started or whose step hadn't been fetched.

What was the solution? (How)

  • Added a _percent_of(numerator, denominator) helper that returns "N/A" when the denominator is zero (an in-flight job with no completed durations yet), used for all percentage lines in the summary.
  • Sort sessions with .get("startedAt", datetime.max) so unstarted sessions sort last, and use session.get("step", {}).get("name", "<Unknown Step>") for the step name.

What is the impact of this change?

trace-schedule runs cleanly on jobs that are still running or have partial data, showing N/A for percentages that can't yet be computed instead of crashing.

How was this change tested?

Added unit tests (new file — none existed for this module) covering zero-denominator summary math, zero session-action count, missing startedAt, and a session without a fetched step.

  • Have you run the unit tests? Yestest/unit/deadline_client/cli/test_cli_trace_schedule.py (4 passed).
  • Have you run the integration tests? No (the e2e test needs a resolvable endpoint; not runnable in the dev sandbox).

Was this change documented?

  • No public-contract change; docstrings on the new helper added.
  • README.md not affected.

Does this PR introduce new dependencies?

  • This PR adds one or more new dependency Python packages.
  • This PR does not add any new dependencies.

Is this a breaking change?

No.

Does this change impact security?

No.

Testing

In addition to the unit tests, the full command was manually verified end-to-end with mocked AWS APIs (no live service), driving deadline job trace-schedule --trace-format chrome --trace-file ... through Click's CliRunner with fixture data for an in-flight job:

  • Job with startedAt but no endedAt (still running).
  • A completed session (step fetched via BatchGetStep) including a zero-duration action span (startedAt == endedAt).
  • An in-flight session with no endedAt, a running action with no endedAt, and a queued action with no startedAt.
  • A not-yet-started session with no startedAt at all.
  • A BatchGetStep partial failure (ResourceNotFoundException) for one step.

Results: exit code 0, no exceptions; the summary printed real percentages, the partial-step warning was surfaced, and the written Chrome trace file was validated — every event has a non-negative integer ts (and dur for X events), session B/E pairs are balanced, the not-started session and not-started action emit no events, the failed step falls back to <Unknown Step>, in-progress markers appear, and the zero-duration span is emitted with dur: 0. A second degenerate run (job started, zero sessions) also exited 0 and printed N/A for all percentage lines instead of raising ZeroDivisionError.

Not yet verified against a real running job in a live farm; a hands-on check tracing a job mid-render (while some sessions are still STARTING) would fully close the loop.

The `job trace-schedule` summary computed several percentages and a
per-action overhead by dividing by the total session duration and the
session-action count. For an in-flight job with no completed durations
(session_total_duration == 0) or with zero session actions, this raised
a ZeroDivisionError and crashed the command.

Add a _percent_of helper that returns "N/A" for a zero denominator, and
guard the overhead-per-action division so it reports 0 instead of
crashing.

Signed-off-by: Stephen Crowe <6042774+crowecawcaw@users.noreply.github.com>
Partial job data crashed `job trace-schedule` with KeyErrors:

- _get_all_sessions sorted sessions by session["startedAt"], but a
  session that hasn't started yet omits that key. Sort such sessions
  last via .get with a datetime.max fallback.
- _build_trace_events read session["step"]["name"], but the step may
  be absent when BatchGetStep returns a partial result (a step could
  not be fetched). Fall back to a "<Unknown Step>" placeholder.

Signed-off-by: Stephen Crowe <6042774+crowecawcaw@users.noreply.github.com>
@github-actions github-actions Bot added the waiting-on-maintainers Waiting on the maintainers to review. label Jul 21, 2026
Comment thread src/deadline/client/cli/_groups/_trace_schedule.py
_build_trace_events dereferenced session['startedAt'] unconditionally, so a
session without startedAt (now tolerated by the sort) would still KeyError.
Skip such sessions entirely, matching the per-action guard.

Signed-off-by: Stephen Crowe <6042774+crowecawcaw@users.noreply.github.com>
@crowecawcaw
crowecawcaw marked this pull request as ready for review July 22, 2026 17:05
@crowecawcaw
crowecawcaw requested a review from a team as a code owner July 22, 2026 17:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

waiting-on-maintainers Waiting on the maintainers to review.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants