fix: correct stale state and JSON output in queue sync-output

[crowecawcaw]

Fixes:

What was the problem/requirement? (What/Why)

Several state bugs in the queue sync-output incremental-download loop:

• A stale max_session_ended_timestamp from the last job of the first loop was reused in a second loop and written onto every job's checkpoint timestamp → missed output downloads / redundant re-downloads on the next sync-output.
• Raw print() DEBUG lines were emitted even under --json, corrupting the JSON output.
• Requeued jobs skipped the eventual-consistency window; running-only jobs overwrote a saved timestamp with None; session_action["manifests"] was accessed as a hard key (KeyError when absent).

What was the solution? (How)

• Removed the stale per-job timestamp reassignment in the second loop; seed each job's max_session_ended_timestamp from the prior checkpoint value.
• Route the DEBUG prints through the print_function_callback (suppressed under --json).
• Apply the eventual-consistency window to genuinely-new jobs (not requeued ones), and use .get("manifests", []).

What is the impact of this change?

sync-output downloads the correct set of outputs across runs, --json output is valid JSON again, and the command no longer crashes on partial/optional session data.

How was this change tested?

Added red-green unit tests for each case (stale-timestamp corruption, running-only overwrite, requeued consistency window, missing manifests, --json cleanliness). Also removed dead skipif(py<3.9) gates (3.9 is the minimum supported version) and hoisted in-body imports to the top of the test file.

• Have you run the unit tests? Yes — test/unit/deadline_client/cli/test_cli_queue_incremental_download.py (all pass, 0 skipped).
• Have you run the integration tests? No.

Was this change documented?

• Docstrings updated where behavior needed clarifying; no public-contract change.
• README.md not affected (no CLI argument changes).

Does this PR introduce new dependencies?

• This PR adds one or more new dependency Python packages.
• This PR does not add any new dependencies.

Is this a breaking change?

No.

Does this change impact security?

No.

Note

The Ctrl-C-mid-download checkpoint-advance issue (C9) is intentionally out of scope here — it needs a signal-handling design decision and is tracked separately.

Testing

Beyond the automated unit tests in this PR, the changed behavior was manually exercised locally (no live AWS — mocked deadline client + fixture data, temp checkpoint dirs, cleaned up afterward):

1. --json output cleanliness across two checkpoint runs — Drove the real deadline queue sync-output CLI (CliRunner + mocked deadline APIs) twice against the same checkpoint dir: a bootstrap run and a resume-from-checkpoint run, both with --json. stdout was completely empty (0 chars) on both runs — every line was validated as parseable JSON, and no DEBUG/progress text leaked. The checkpoint file was still written correctly. A contrast run without --json on the same checkpoint emitted the expected 60 lines of verbose output including the DEBUG: lines, confirming the suppression is mode-scoped rather than output being lost.
2. Brand-new vs requeued job eventual-consistency threshold — Built a checkpoint containing a previously-tracked (inactive) job with a saved session_ended_timestamp, then ran _get_job_sessions with both that job and a never-seen job categorized as added. Captured the per-job session_ended_threshold passed to the session-retrieval call: the brand-new job got the raw bootstrap timestamp (no window), while the requeued job got its saved timestamp minus the eventual-consistency window (observed 2025-01-04T23:58:00Z for a saved 2025-01-05T00:00:00Z with a 120s window).
3. Checkpoint session_ended_timestamp integrity across runs — Ran _update_checkpoint_jobs_list twice on one checkpoint: run 1 with two jobs having different session endedAt values verified each job records its own max timestamp (no cross-job stale-variable bleed); run 2 with only running sessions (no endedAt) verified the saved timestamps are preserved rather than overwritten with None.
4. Repo checks — hatch run lint (ruff + mypy) clean; the full test_cli_queue_incremental_download.py suite passes (19 passed, 0 skipped).

The one gap originally flagged for manual testing — --json cleanliness during an actual multi-file download with progress callbacks — is now closed by an automated test instead: test_incremental_output_download_json_mode_with_real_s3_download uploads real asset manifests and CAS data objects to a moto S3 bucket and drives deadline queue sync-output --json through an actual 3-file S3 download, with the real ProgressTracker firing progress callbacks. It asserts stdout is byte-clean (and that the downloaded files match the uploaded bytes), then re-runs without --json and asserts the DEBUG lines and the 100% progress message do appear — proving the callbacks fired and were suppressed, not that the download was skipped. The test was verified red-green: reintroducing one raw print() in _get_download_candidate_jobs makes it fail.

No hands-on testing against a real queue is required for the changes in this PR.