Failover sticks on broken provider when error is not in analyzeError pattern table

[anhthii]

Summary

The failover system in internal/rpc/failover.go can get stuck routing every request to a single broken provider when that provider returns an error message not listed in analyzeError's pattern table. Observed in production: Ethereum Sepolia (and reportedly Base) indexing stalled for an extended period while the failover never switched away from a provider returning RPC error -32701.

Incident logs (Ethereum Sepolia, 2026-05-23)

WRN Switching provider due to error provider=ethereum_sepolia-2 error_type=connection_error blacklist_duration=2m0s
... emergency recovery picks sepolia-2 back up ...
DBG Provider failed but not switching provider=ethereum_sepolia-2 state=healthy
    error="rpc error: RPC error -32701: Please specify an address in your request
    or, to remove restrictions, order a dedicated full node here..."
    consecutive_errors=0
DBG Provider failed but not switching ... state=healthy consecutive_errors=1
DBG Provider failed but not switching ... state=degraded consecutive_errors=2
WRN failed to query ERC20 transfers error="failed after 3 attempts: rpc error: RPC error -32701..."
... loop repeats, no switch ever happens ...

Root cause

analyzeError (internal/rpc/failover.go:589) only marks a provider unhealthy when the error string matches a hand-maintained pattern table (rate limit, 403, timeout, connection reset, -32007, etc.). The publicnode -32701 error matches none of these, so it falls through to:

issue := ProviderIssue{
    Reason:        "generic_error",
    Cooldown:      0,
    MarkUnhealthy: false,
}

With MarkUnhealthy=false, executeCore (failover.go:387-391) takes the handleProviderFailure branch — provider is not blacklisted, only provider.Fail() is called.

The follow-on stuck behavior is structural:

1. provider.Fail() only escalates state to Unhealthy after ErrorThreshold (default 5) consecutive errors.
2. Provider.IsAvailable() (internal/rpc/provider.go:27-31) returns true for every non-blacklisted state — including Unhealthy.
3. GetBestProvider returns the current provider as long as IsAvailable() is true, so currentIndex never moves.
4. The retry wrapper retries 3x via retry.Constant, all 3 attempts hit the same provider, then bubbles failed after 3 attempts to the worker.
5. Sibling providers (e.g. ethereum_sepolia-1) are never tried even when healthy.

Result: any error class not in the pattern table can cause an indefinite indexing stall on the affected chain.

Why this will keep recurring

The current design relies on a hand-maintained allowlist of error strings to decide whether to switch providers. New providers, new error codes, or changes in upstream error message text will silently re-create this incident. The pattern table is currently load-bearing for routing safety, but it has no feedback loop to catch unknown errors before they cause an outage.

Proposed fixes

1. Forced rotation on consecutive errors (recommended primary fix)

In GetBestProvider (or in the retry path), skip the current provider when consecutiveErrors >= N (e.g., 3) even if it's not blacklisted. This guarantees that N retries against the same broken endpoint can never happen, regardless of error classification.

Small, contained change in failover.go — no changes to analyzeError's classification logic.

2. Loud logging of unknown error types

When analyzeError falls through to the generic_error branch, log at WARN with the full error message and increment a dedicated unknown_error_type metric. Gives an early signal to extend the pattern table before the next outage.

[anhthii]

Detailed proposal for Fix #1 (Forced rotation on consecutive errors)

The idea

Add a second reason to skip a provider in GetBestProvider, independent of blacklist state and independent of error classification:

If a provider has failed N consecutive times, skip it on the next selection — regardless of whether the error was classified.

ConsecutiveErrors is already tracked by provider.Fail() (internal/rpc/provider.go:45) and already reset by provider.Success() (internal/rpc/provider.go:78). Routing logic just doesn't read it today.

Proposed change shape

In GetBestProvider (internal/rpc/failover.go:228):

if curIdx >= 0 && curIdx < len(providers) {
    cur := providers[curIdx]
    if cur.IsAvailable() && cur.ConsecutiveErrors < forceRotateThreshold {
        return cur, nil
    }
    // either blacklisted OR too many consecutive failures → fall through
}
return f.findNextAvailableProvider()

Add a config field (e.g., ForceRotateThreshold int) to FailoverConfig, default to 3.

Why this fixes the -32701 incident regardless of error type

1. sepolia-2 returns unknown error → generic_error → no blacklist, but ConsecutiveErrors increments.
2. After 3 consecutive failures, next GetBestProvider call skips sepolia-2 and findNextAvailableProvider picks sepolia-1.
3. We never had to classify -32701 — the system self-corrects on any unknown error.

Once sepolia-1 returns a success, its counter stays at 0, and if sepolia-2 ever recovers and returns a success, its counter resets — natural re-entry into rotation.

Blast radius

• ~2 lines added to GetBestProvider.
• 1 new config field + default value.
• Zero changes to analyzeError pattern table (kept as an optimization for cooldown selection).
• Zero changes to Provider state machine.
• Zero changes to executeCore or retry semantics.

⚠️ Please test thoroughly before merging

This changes routing decisions across every chain (EVM, Tron, Bitcoin, Solana, Sui, Cosmos, Aptos — all 7 NewFailover call sites in internal/worker/factory.go). Suggested test coverage:

Unit tests in internal/rpc/:

• Provider with ConsecutiveErrors >= threshold is skipped by GetBestProvider, even when state is Healthy/Degraded/Unhealthy.
• Provider re-enters rotation after a Success() call resets the counter.
• All providers having ConsecutiveErrors >= threshold triggers findNextAvailableProvider → emergency recovery path (verify no deadlock / infinite recursion).
• Single-provider failover with ConsecutiveErrors >= threshold: confirm behavior (likely should still return it via emergency recovery, since there's no alternative).
• Concurrent reads/writes against ConsecutiveErrors under -race — Fail() and Success() already lock, but verify the new read in GetBestProvider is also under the provider's lock.

Integration tests:

• Simulate a provider returning a non-classified error (e.g., -32701) repeatedly and assert the failover switches to a healthy sibling within N attempts.
• Simulate transient blips (1-2 failures then success) and confirm no spurious rotation — the threshold should absorb these.
• Verify metrics: ProviderSwitches increments on forced rotation, ErrorsByType["generic_error"] still records the unknown error so it's visible.

Manual / staging verification:

• Run against a real publicnode Sepolia endpoint that returns -32701 and confirm indexing proceeds via the sibling provider.
• Watch logs for any unexpected oscillation between providers (rotation thrash).

Tuning notes:

• forceRotateThreshold = 3 is a starting point. Too low → spurious rotation on flaky-but-recoverable providers. Too high → slow reaction to broken providers. May need tuning per chain (Bitcoin RPCs vs EVM RPCs have different failure characteristics).
• Consider whether the threshold should be exposed in YAML config (currently FailoverConfig from YAML isn't wired up at all — separate issue).

I'd recommend implementing Fix #2 (loud logging for unknown errors) in the same PR — it's a 2-line change and gives an immediate feedback loop for catching the next unknown error before it causes a stall.