Give up reconnecting the agent event stream on persistent auth failures

[seemeroland]

Roland's human notes

I saw k8s worker pods staying alive after the task was already killed on the server by our cleanup job. This is fixed by setting a max timeout on worker pods, but the pods were also retrying the SSE connection indefinitely (got to attempt 2000), even though they were failing with 401 due to the task being failed.

This PR sets a max deadline for retries and stops after 3 consecutive 401s/403s. It's not a required PR for the original issue but seems good to clean up (correct me if I'm wrong and we do need indefinite retries)

Description

Symptom

Cloud-agent worker pods stayed alive for 22–26h+ after their task had already been terminated server-side (timeout). Their logs showed two loops still running long after termination:

• the agent-event SSE listener reconnecting every ~30s against a now-401 endpoint (Agent event stream failed N consecutive times … 401 Unauthorized), and
• the harness transcript/block-snapshot upload firing every ~30s, also failing Unauthorized.

What actually keeps the pod alive

The pod's process is the oz task. For a Claude/Codex run it is held open by run_harness, which blocks until the CLI agent session exits or a local idle/exit signal fires. That exit signal is driven by in-process session status (CLIAgentSessionsModel / BlocklistAIHistory → run_exit / harness_exit), not by any SSE stream.

When a task is terminated server-side, the credential revocation surfaces only as 401s on the client's HTTP/SSE calls; it is never translated into a local "end the session" signal. So the long-running agent session never exits and the pod lingers.

To be precise: the indefinite SSE retry is not what kept the pod alive — the still-running agent session was. The cloud-agent message-bridge SSE listener is a detached background task whose result is only logged, so stopping it does not, and cannot, terminate the pod.

What this PR changes (and why)

This is a targeted correctness / resource-hygiene fix to that one background listener: stop reconnecting forever to a stream that can no longer succeed.

• The cloud-agent Claude message bridge (run_parent_bridge_forever) used AgentEventDriverConfig::retry_forever, whose reconnect loop has no give-up condition. Against a permanently-401 endpoint that is pure waste and log spam.
• Add an opt-in give-up policy to AgentEventDriverConfig, used only by the cloud-agent bridge via a new bounded_run_ids:
  • auth_error_give_up_failures — stop after N consecutive HTTP 401/403 failures. Tracked with a dedicated consecutive_auth_failures counter (reset on any non-auth failure or success) so a mix like 500, 500, 401 does not trip it.
  • max_retry_duration — stop after a bounded window of sustained failure (measured from the first failure since the last successful open/event; resets on success).
  • Defaults: 3 consecutive auth failures, or 30m sustained failure.
• retry_forever / retry_forever_run_ids are unchanged, so local interactive and orchestration-viewer listeners keep retrying as before.

Because the driver runs detached and its result is only logged, this can never fail, block, or abort a run — it only stops a dead background listener from spinning.

The actual pod-lifetime fix (separate)

The guaranteed remedy for "pods run forever" is a hard activeDeadlineSeconds (8h) backstop on task Jobs, added in the oz-agent-worker chart (separate repo/PR). This PR complements it by removing the pointless reconnect churn but does not itself reclaim pods.

Possible follow-up (not in this PR): propagate server-side task termination into a local session-exit signal so the agent session ends promptly instead of relying on the deadline.

Testing

Unit tests in driver_tests.rs:

• gives up after 3 consecutive 401s
• does not give up on repeated non-auth (500) errors when only auth-bounded (reconnects and succeeds)
• does not count non-auth failures toward the auth give-up (500, 500, 401 keeps going)
• resets the auth streak after a non-auth failure, then gives up on a fresh run of 3 401s
• gives up once the max retry duration elapses

Checks run:

• cargo nextest run -p warp agent_events::driver — 19/19 pass

• ./script/format

• cargo clippy --workspace --exclude warp_completer --all-targets --tests -- -D warnings — passes

• wasm dead-code gated with cfg_attr(target_family = "wasm", allow(dead_code)) on the native-only bounded-give-up items

• I have manually tested my changes locally with ./script/run

Agent Mode

• Warp Agent Mode - This PR was created via Warp's AI Agent Mode

Co-Authored-By: Oz oz-agent@warp.dev

[oz-for-oss]

@seemeroland

I'm starting a first review of this pull request.

You can view the conversation on Warp.

I completed the review and no human review was requested for this pull request.

Comment /oz-review on this pull request to retrigger a review (up to 3 times on the same pull request).

Powered by Oz

[oz-for-oss]

Overview

This PR adds bounded retry behavior for the cloud-agent event stream so persistent auth failures or sustained retry windows stop the background bridge instead of keeping worker pods alive indefinitely.

Concerns

• The auth give-up threshold is documented and configured as consecutive authentication failures, but the implementation checks it against the generic consecutive failure counter. Mixed non-auth failures followed by a single auth failure can therefore stop the driver earlier than intended.

Verdict

Found: 0 critical, 1 important, 0 suggestions

Request changes

Comment /oz-review on this pull request to retrigger a review (up to 3 times on the same pull request).

Powered by Oz

[seemeroland]

/oz-review

[oz-for-oss]

@seemeroland

I'm re-reviewing this pull request in response to a review request.

You can view the conversation on Warp.

I completed the review and no human review was requested for this pull request.

Comment /oz-review on this pull request to retrigger a review (up to 3 times on the same pull request).

Powered by Oz

[oz-for-oss]

Overview

This PR adds opt-in bounded retry behavior for the agent event stream and applies it to the cloud-agent Claude parent bridge so persistent 401/403 failures or sustained retry windows stop the background listener instead of keeping worker pods alive indefinitely. Existing unbounded listeners keep their prior retry behavior, and the diff adds focused tests for auth give-up, non-auth retries, auth-streak reset behavior, and max retry duration.

Concerns

• No blocking correctness, security, or spec-alignment concerns found in the changed lines.
• No approved or repository spec context was provided, so there was no material spec contract to compare against.

Verdict

Found: 0 critical, 0 important, 0 suggestions

Approve

Comment /oz-review on this pull request to retrigger a review (up to 3 times on the same pull request).

Powered by Oz

[cephalonaut]

Makes sense to me, thanks!

[cephalonaut]

nit: Could extract most of this as a function - repeated above