Skip to content

fix(actions): self-heal duplicate tend-outage issues from concurrent leg failures#744

Merged
max-sixty merged 1 commit into
mainfrom
daily/review-runs-28361647677
Jul 10, 2026
Merged

fix(actions): self-heal duplicate tend-outage issues from concurrent leg failures#744
max-sixty merged 1 commit into
mainfrom
daily/review-runs-28361647677

Conversation

@tend-agent

Copy link
Copy Markdown
Collaborator

Problem

The outage-tracker dedup in shared/steps/report-failure.sh is racy, and the jitter added in #586 to fix it has now been proven insufficient in production.

When several matrix legs fail within a few seconds of each other (the common case for an upstream model-API 5xx/overload incident, which hits every leg at once), each leg runs the check-then-create dedup. The sleep $((RANDOM % 30)) jitter only narrows the create-create window — it can't close it. Two legs can still both read EXISTING as empty: jitter collisions are likely with several legs drawing from a 30-slot range (birthday problem), and GitHub's issue-list index takes a few seconds to reflect a freshly-created issue, so even a staggered second leg can miss the first's issue.

Evidence

Every prior incident with only 1–2 failing legs produced a single tracker, so the failure mode scales with the number of simultaneously-failing legs. Impact is low (both dupes were auto-closed by nightly ~2h later when the outage resolved), but it's a recurring structural failure of a deployed fix, and it generates duplicate "Bot temporarily unavailable" notifications during every wide outage.

Fix

Keep the jitter (it still reduces contention and the number of dupes created) and add a self-healing reconciliation to the create path: after gh issue create, settle briefly for the list index, then list every open tend-outage issue, keep the lowest-numbered, and close the rest as duplicates. It is idempotent and convergent — every racing leg computes the same keeper, so a second leg closing an already-closed dup is a no-op.

This is shared verbatim by all three harness actions, so it reaches every consumer at the next release.

Verification
  • shellcheck shared/steps/report-failure.sh passes.
  • Logic unit-tested against today's numbers: [{742},{743}] → keep Bot temporarily unavailable #742, close Bot temporarily unavailable #743; single-issue input closes nothing.
  • Index-lag safety: the 5s settle plus the convergent keep-lowest rule means even if leg B's first re-list misses leg A's issue, both legs eventually compute the same keeper and the loser is closed.

Found by review-runs analyzing run 28361647677.

…leg failures

The jittered backoff added in #586 narrows but does not close the
create-create race when many matrix legs fail within seconds (upstream
5xx hitting every leg). Jitter collisions plus issue-list index lag let
two legs both read no existing issue and each file a tracker — observed
2026-05-22 (#575-#578) and again 2026-06-29 (#742/#743) despite #586.

Add a convergent post-create reconciliation: keep the lowest-numbered
open tend-outage issue and close the rest as duplicates. Idempotent
across racing legs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@tend-agent tend-agent added the review-runs Findings from review-runs skill label Jun 29, 2026

@tend-agent tend-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The convergent keep-lowest reconciliation is sound, and it's loop-safe — tend-triage/tend-mention both skip tend-outage-labeled issues, so the close-with-comment won't cascade. For the documented trigger (matrix legs of one run) there's no data loss, since all legs share GITHUB_RUN_ID and thus the same run link.

One edge to consider: the dedup is label-scoped and workflow-agnostic, so it also fires when duplicates come from different workflow runs — e.g. a tend-review and a tend-triage both passing the empty-EXISTING check within the same overload window. There the closed higher-numbered issue carries a distinct run link, and the keeper never accumulates it: enrich-tend-outage-issues.sh only iterates --state open, so the closed dup's run is never enriched onto the keeper. The run is still one click away via the "Duplicate of #KEEP" comment, so it's recoverable rather than lost — just not consolidated.

If that's worth closing, the cheap fix is to append the dup's run reference as a comment on KEEP before closing it (read the dup's body for its run URL), rather than only commenting on the dup. Not blocking — the matrix-leg case this targets is fully handled.

Submitting as a comment since GitHub rejects self-approval.

@max-sixty

Copy link
Copy Markdown
Owner

or more jitter?

@tend-agent

Copy link
Copy Markdown
Collaborator Author

More jitter can't close this race — it only lowers the probability. The PR keeps the jitter for exactly that reason (fewer dupes created), but a deterministic reconcile is what actually fixes it. Two reasons jitter alone falls short:

  1. Birthday collisions scale with legs. With the current RANDOM % 30 and 5 simultaneously-failing legs, P(some pair within the same slot) ≈ 30%. To push that low you'd need a range of minutes, which delays outage reporting on every failure to fix a rare one.

  2. Index lag is the dominant factor, and jitter can't bound it. Even legs that draw distinct slots can both read EXISTING empty, because GitHub's issue-list index takes a few seconds to reflect a fresh create. The 06-29 dupes (Bot temporarily unavailable #742/Bot temporarily unavailable #743) landed 1s apart — within that lag window. To beat lag with jitter alone, every leg would need spacing greater than the (variable, uncontrolled) index latency, which random jitter can't guarantee.

So jitter is a probability knob; the reconcile is convergent — every racing leg computes the same keeper, so the window is closed rather than narrowed, at the cost of one gh issue list + closes on the create path only. Happy to drop the jitter entirely if you'd rather not keep both, but I'd lean on keeping it since it cheaply reduces how many dupes the reconcile has to clean up.

@max-sixty max-sixty merged commit e127227 into main Jul 10, 2026
7 checks passed
@max-sixty max-sixty deleted the daily/review-runs-28361647677 branch July 10, 2026 23:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review-runs Findings from review-runs skill

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bot temporarily unavailable

2 participants