Skip to content

fix(supervisor): stop signature-failure log storm (dedup + backoff + quarantine)#32

Merged
TeoSlayer merged 1 commit into
mainfrom
fix/supervisor-signature-backoff
Jul 13, 2026
Merged

fix(supervisor): stop signature-failure log storm (dedup + backoff + quarantine)#32
TeoSlayer merged 1 commit into
mainfrom
fix/supervisor-signature-backoff

Conversation

@TeoSlayer

Copy link
Copy Markdown
Contributor

Problem

When a catalogue app's manifest failed ed25519 signature (or trust-anchor) verification, scanInstalled logged signature verification failed and skipped the app. A skipped app is never registered in the installed map, so the rescan ticker re-examined and re-logged it on every cycle — a ~2s hot loop that produced hundreds of thousands of identical log lines (683k observed in a local daemon.log). No backoff, no dedup, no quarantine.

Fix

Track signature-failure state keyed by (install dir, manifest content hash):

  • Dedup: log the failure at most once per manifest hash. An identical failure on the next rescan tick is silent; a NEW/changed manifest (different hash) re-attempts and re-logs.
  • Backoff: capped exponential backoff (2s → doubling → 5m) between re-verifications instead of re-checking every tick.
  • Quarantine: after 5 consecutive same-hash failures the app is retried only on the slow backoff tier. The supervisor is never crashed — the bad app is merely isolated.
  • Recovery: a later success clears the state, so a re-install / fixed manifest is picked up cleanly and a future transient failure re-logs.

Both catalogue failure modes (VerifySignature and VerifyTrustAnchor) route through the same throttling path.

Test

plugin/appstore/zz5_sigstorm_test.go reproduces the storm and asserts:

  • a repeatedly-failing manifest is logged exactly once across 200 rescans (was 200),
  • backoff grows and the app is quarantined after N failures,
  • a changed manifest hash re-logs,
  • recovery clears the quarantine and admits the app,
  • the trust-anchor failure mode is throttled too.

go build ./..., go test ./..., and go test -race ./plugin/appstore/ all pass.

🤖 Generated with Claude Code

…ailure

When a catalogue app's manifest failed ed25519 signature (or trust-anchor)
verification, scanInstalled logged "signature verification failed" and
skipped the app. Because a skipped app is never registered in the
installed map, the rescan ticker re-examined and re-logged it on every
cycle — a ~2s hot loop that produced hundreds of thousands of identical
log lines (683k observed in a local daemon.log).

Add signature-failure state keyed by (install dir, manifest content hash):

- Log the failure at most ONCE per manifest hash. An identical failure on
  the next rescan tick is silent; a NEW/changed manifest (different hash)
  re-attempts and re-logs.
- Apply capped exponential backoff (2s → doubling → 5m) between
  re-verifications instead of re-checking every tick.
- Quarantine after 5 consecutive same-hash failures: retried only on the
  slow backoff tier. The supervisor is never crashed — the bad app is
  merely isolated.
- A later success clears the state (recovery), so a re-install or a fixed
  manifest is picked up cleanly and a future transient failure re-logs.

Both catalogue failure modes (VerifySignature and VerifyTrustAnchor) are
routed through the same throttling path.

Adds zz5_sigstorm_test.go reproducing the storm and asserting: a
repeatedly-failing manifest is logged exactly once across 200 rescans,
backoff grows, the app is quarantined, a changed hash re-logs, recovery
clears state, and the trust-anchor failure mode is throttled too.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@TeoSlayer
TeoSlayer enabled auto-merge (squash) July 13, 2026 12:51
@TeoSlayer
TeoSlayer merged commit 7187377 into main Jul 13, 2026
4 checks passed
@codecov

codecov Bot commented Jul 13, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants