Preempt api response

[dslear]

What type of PR is this?

Enhancement

What this PR does / why we need it

When a job is preempted in Armada, it should have an error message that includes the user who requested preemption, as well as the reason if one exists. Currently, when a job running as part of a gang job is preempted, the rest of the jobs in the gang are also preempted but do not have the appropriate error message attached. This PR ensures that all jobs in the gang have the requesting user and reason as the error message.

Which issue(s) this PR fixes

Fixes #4832

Special notes for your reviewer

[greptile-apps]

Greptile Summary

This PR fixes two related issues: it propagates the preemption reason (including requesting user) to all jobs in a gang when one gang member is preempted, and it cleans up the preemption reason string format (removing the old \" | Preempted by user\" format with a leading space/pipe in favour of \"Preempted by user - reason\").

The core change in reconciliation.go introduces a gangPreemptionRequest struct that tracks the preemption reason per gang, so each gang's members can receive the correct formatted reason rather than a nil. The buildPreemptionReason helper in instructions.go centralises formatting with proper handling of all four user/reason combinations.

Confidence Score: 5/5

Safe to merge — logic is correct, all four reason/user combinations are tested, and gang propagation is properly isolated per gang.

No P0 or P1 issues found. The gang preemption reason extraction correctly uses jst.Job.LatestRun() (the post-reconciliation run with PreemptReason set) rather than the original job. The per-gang grouping via gangPreemptionRequests correctly handles multiple concurrently-preempted gangs with different reasons. The format change from the old " | Preempted by user" pattern (with leading space/pipe on empty reason) to "Preempted by user - reason" is an intentional improvement and is thoroughly tested.

No files require special attention.

Important Files Changed

Filename	Overview
internal/scheduler/jobdb/reconciliation.go	Refactors gang preemption logic to carry per-gang preempt reasons; markJobsAsPreemptionRequested now accepts and applies *string reason to all gang members' runs.
internal/scheduleringester/instructions.go	Extracts buildPreemptionReason helper; fixes old bug where empty reason produced a leading "
internal/scheduler/scheduler_test.go	Adds expectedPreemptReasons test field and a new table-driven case verifying that preempt reason propagates to all gang members after a single member's run update.
internal/scheduleringester/instructions_test.go	Updates existing preemption test expectation to match new format, adds "with reason" case, and adds a dedicated unit test for buildPreemptionReason covering all four input combinations.

Sequence Diagram

sequenceDiagram
    participant API as API Layer
    participant Ingester as scheduleringester
    participant DB as Database
    participant Reconciler as reconciliation.go
    participant GangMember as Gang Member Jobs

    API->>Ingester: JobPreemptionRequested(jobId, reason, user)
    Ingester->>Ingester: buildPreemptionReason(reason, user)
    Ingester->>DB: MarkRunsForJobPreemptRequested{jobId: formattedReason}
    DB-->>DB: Store PreemptReason on run

    Note over Reconciler: Next scheduler cycle
    Reconciler->>DB: Fetch run updates
    Reconciler->>Reconciler: reconcileJobDifferences(gangJob1) → jst.PreemptionRequested=true
    Reconciler->>Reconciler: jst.Job.LatestRun().PreemptReason()
    Reconciler->>Reconciler: Append gangPreemptionRequest{jobIds, reason}
    Reconciler->>GangMember: markJobsAsPreemptionRequested(jobIds, reason)
    GangMember-->>Reconciler: PreemptionRequested=true, reason propagated

Loading

_{Reviews (3): Last reviewed commit: "Merge branch 'master' into preempt-api-r..." | Re-trigger Greptile}