fix(actions): self-heal duplicate tend-outage issues from concurrent leg failures#744
Conversation
…leg failures The jittered backoff added in #586 narrows but does not close the create-create race when many matrix legs fail within seconds (upstream 5xx hitting every leg). Jitter collisions plus issue-list index lag let two legs both read no existing issue and each file a tracker — observed 2026-05-22 (#575-#578) and again 2026-06-29 (#742/#743) despite #586. Add a convergent post-create reconciliation: keep the lowest-numbered open tend-outage issue and close the rest as duplicates. Idempotent across racing legs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
tend-agent
left a comment
There was a problem hiding this comment.
The convergent keep-lowest reconciliation is sound, and it's loop-safe — tend-triage/tend-mention both skip tend-outage-labeled issues, so the close-with-comment won't cascade. For the documented trigger (matrix legs of one run) there's no data loss, since all legs share GITHUB_RUN_ID and thus the same run link.
One edge to consider: the dedup is label-scoped and workflow-agnostic, so it also fires when duplicates come from different workflow runs — e.g. a tend-review and a tend-triage both passing the empty-EXISTING check within the same overload window. There the closed higher-numbered issue carries a distinct run link, and the keeper never accumulates it: enrich-tend-outage-issues.sh only iterates --state open, so the closed dup's run is never enriched onto the keeper. The run is still one click away via the "Duplicate of #KEEP" comment, so it's recoverable rather than lost — just not consolidated.
If that's worth closing, the cheap fix is to append the dup's run reference as a comment on KEEP before closing it (read the dup's body for its run URL), rather than only commenting on the dup. Not blocking — the matrix-leg case this targets is fully handled.
Submitting as a comment since GitHub rejects self-approval.
|
or more jitter? |
|
More jitter can't close this race — it only lowers the probability. The PR keeps the jitter for exactly that reason (fewer dupes created), but a deterministic reconcile is what actually fixes it. Two reasons jitter alone falls short:
So jitter is a probability knob; the reconcile is convergent — every racing leg computes the same keeper, so the window is closed rather than narrowed, at the cost of one |
Problem
The outage-tracker dedup in
shared/steps/report-failure.shis racy, and the jitter added in #586 to fix it has now been proven insufficient in production.When several matrix legs fail within a few seconds of each other (the common case for an upstream model-API 5xx/overload incident, which hits every leg at once), each leg runs the check-then-create dedup. The
sleep $((RANDOM % 30))jitter only narrows the create-create window — it can't close it. Two legs can still both readEXISTINGas empty: jitter collisions are likely with several legs drawing from a 30-slot range (birthday problem), and GitHub's issue-list index takes a few seconds to reflect a freshly-created issue, so even a staggered second leg can miss the first's issue.Evidence
tend-outageissues (Bot temporarily unavailable #575, Bot temporarily unavailable #576, Bot temporarily unavailable #577, Bot temporarily unavailable #578) created within 4 seconds. This prompted fix: jitter outage-issue dedup to handle concurrent matrix-leg failures #586 (jitter).review-reviewersfailure (run 28352782724, transient529 Overloaded/StopFailure) produced two duplicate trackers Bot temporarily unavailable #742 and Bot temporarily unavailable #743, created within 1 second (06:23:30 and 06:23:31), both referencing the same run.Every prior incident with only 1–2 failing legs produced a single tracker, so the failure mode scales with the number of simultaneously-failing legs. Impact is low (both dupes were auto-closed by nightly ~2h later when the outage resolved), but it's a recurring structural failure of a deployed fix, and it generates duplicate "Bot temporarily unavailable" notifications during every wide outage.
Fix
Keep the jitter (it still reduces contention and the number of dupes created) and add a self-healing reconciliation to the create path: after
gh issue create, settle briefly for the list index, then list every opentend-outageissue, keep the lowest-numbered, and close the rest as duplicates. It is idempotent and convergent — every racing leg computes the same keeper, so a second leg closing an already-closed dup is a no-op.This is shared verbatim by all three harness actions, so it reaches every consumer at the next release.
Verification
shellcheck shared/steps/report-failure.shpasses.[{742},{743}]→ keep Bot temporarily unavailable #742, close Bot temporarily unavailable #743; single-issue input closes nothing.Found by
review-runsanalyzing run 28361647677.