Skip to content

fix(failover): require a sustained stream before failing back to a recovered input#33

Merged
ntt0601zcoder merged 1 commit into
mainfrom
fix/failover-probe-sustained
Jun 18, 2026
Merged

fix(failover): require a sustained stream before failing back to a recovered input#33
ntt0601zcoder merged 1 commit into
mainfrom
fix/failover-probe-sustained

Conversation

@ntt0601zcoder

Copy link
Copy Markdown
Owner

Summary

The failback probe declared a degraded input "recovered" on a single payload packet from one read. An intermittent source that merely sputters (an occasional packet while effectively down) passed that check, so the manager failed back to it, then starved to the packet-timeout and degraded again — flapping between the source and its backup roughly every timeout window. The repeated input switches also churned ingest sessions into visible playback flicker on recovery.

Fix

ingestor.Probe now requires payload to keep arriving across a minimum span (1.5s), with a minimum packet count and byte total, before reporting recovery:

  • A sputtering feed never reaches the span → stays degraded → no premature failback → no flapping.
  • A genuinely recovered source streams continuously → adopted once, cleanly.

The manager's probe deadline is raised from 3s to 5s to fit the sustained window.

Changes

  • internal/ingestor/service.go — sustained-probe logic (probeStats accumulator + sustainedProbeOK); Probe reads until sustained or ctx expires.
  • internal/manager/service.goprobeTimeout 3s → 5s.
  • internal/ingestor/probe_test.go — table tests (silent / single blip / sparse blips / burst-then-quiet / sustained / boundaries).

Testing

  • go build, go vet, gofumpt, golangci-lint run ./... → clean
  • go test ./internal/ingestor/ ./internal/manager/ → pass (only the pre-existing MediaMTX docker integration test in ingestor/pull fails, unrelated and untouched)

…covered input

The failback probe declared a degraded input "recovered" on a single
payload packet from one read. An intermittent source that merely sputters
(an occasional packet while effectively down) passed that check, so the
manager failed back to it, then starved to the packet-timeout and degraded
again — flapping between the source and its backup roughly every timeout
window, and the repeated input switches churned ingest sessions into
visible playback flicker on recovery.

Probe now requires payload to keep arriving across a minimum span (1.5s),
with a minimum packet count and byte total, before reporting recovery. A
sputtering feed never reaches the span and stays degraded; a genuinely
recovered source streams continuously and is adopted once. The manager's
probe deadline is raised to 5s to fit the sustained window.
@ntt0601zcoder ntt0601zcoder merged commit 346bf2a into main Jun 18, 2026
4 checks passed
@ntt0601zcoder ntt0601zcoder deleted the fix/failover-probe-sustained branch June 18, 2026 06:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant