Auto-restart to recover from wedged ASIC (Flatline of Death, #1053)

[ThymeKeeper]

What

Firmware-level detection and automatic recovery for the "Flatline of Death" (#1053): if the pool connection is demonstrably alive but the ASIC has not returned a single nonce for 10 minutes, the device logs the condition and restarts itself — the same recovery every affected user performs by hand or with external reboot scripts today.

Diagnosis on an affected unit

I'm the user who posted the external reboot-script workaround in #1053 (Nov 2025). This week I caught my Gamma 601 (BM1370, stock 525 MHz / 1150 mV) mid-flatline against a local public-pool instance and autopsied it before rebooting:

• The stratum TCP session was ESTABLISHED (verified from the pool host with ss), and the pool had logged zero disconnects during the 14+ hour flatline — the miner never reconnected because, at the network layer, nothing was wrong.
• Live device logs (via /api/ws) showed stratum_api: rx: {...mining.notify...} lines still arriving — the confirmation @skot asked reporters for — while zero asic_result lines appeared.
• Power draw stayed at full mining load (20.8 W) with normal temperatures; the web UI, WiFi, and API remained fully responsive.
• Share counter frozen; a restart instantly restored normal hashing, as it always does.

So at least on this unit, FoD is a wedged ASIC/serial-result path, not a stratum or network failure — which is why it survives every network-level fix and why only a reset clears it.

How the detection works

(Reworked after 48 h of field testing on the affected unit — see the comment below for the second wedge mode that motivated v2.)

• SYSTEM_notify_accepted_share — already the common accept path for both SV1 and SV2 — stamps last_accepted_share_time. Pool-accepted shares are the ground truth: a wedged chip may go silent or babble garbage nonces; garbage can fool local counters (it inflated the share-based hashrate estimator ~17× on my unit) but it can never pass pool validation.
• Both stratum receive loops call SYSTEM_check_flatline_watchdog() right after successfully receiving pool traffic: a restart can only trigger when the pool link is provably alive, so a pool outage can never cause a restart loop.
• The timeout is 25× the expected share interval at the current pool difficulty (from device frequency × small-core count), floored at 10 minutes — so vardiff can never outrun the watchdog.
• The clock resets on every fresh connection, and the check is skipped during overheat_mode or while the ASIC is uninitialized, so pauses can't cause a stale-timestamp restart on resume.

False-positive analysis

Share arrivals are Poisson. The timeout is ≥25 expected arrivals at the current difficulty, so the false-trigger probability per window is ≤ e^−25 ≈ 10⁻¹¹ regardless of what difficulty vardiff assigns — at typical difficulties the 10-minute floor dominates and the margin is far larger (e.g. ~73 expected shares per window at diff 2048 on a Gamma). It cannot fire on a healthy miner. The floor and multiplier live in one place if maintainers prefer different values.

Testing

• The identical detection logic (backported to a v2.10.1 base, which this unit is pinned to for unrelated reasons) is running on the affected Gamma 601 now; it previously flatlined as often as daily.
• This branch builds clean against master with ESP-IDF v5.5 (esp32s3).
• Recovery events are easy to audit pool-side: each one appears as a disconnect/reconnect pair with a young uptime after it.

The flatline on this unit, followed by restart and normal operation (note the reported hashrate frozen at a healthy-looking value the whole time — and ASIC temperature staying at full mining load):

Addresses #1053 (automatic recovery; the root cause of the wedge itself remains open).

[ThymeKeeper]

Field report after 48 h on the affected unit — and a v2 of the detection (bf81fed).

The nonce-based aliveness check from the original commit caught ~35 real wedges in two days (each recovered within 10 minutes). But this morning the unit revealed a second wedge mode that slipped past it: instead of going silent, the chip started babbling garbage results. Those refreshed the "chip is alive" stamp and inflated the share-based hashrate estimator to ~17.5 TH/s on a ~1 TH/s Gamma, while producing zero pool-valid shares for 2+ hours — pool link healthy the whole time, so the v1 check never fired.

The lesson: any locally-observable signal can be faked by a misbehaving chip. The only unfakeable signal is a pool-accepted share. v2 therefore tracks last_accepted_share_time via the existing SYSTEM_notify_accepted_share path (shared by SV1/SV2) and restarts when it goes stale for 25× the expected share interval at the current difficulty (floored at 10 min), checked only after receiving pool traffic. False-positive probability is ≤ e^−25 per window at any vardiff level.

Both wedge modes (silent and babbling) are now caught. Running on the affected unit.

[ThymeKeeper]

One more member of the FoD failure family, found and fixed (latest commit).

Today the test unit's flaky serial link failed chip detection on a cold boot. app_main returns in that case, which leaves a web-UI zombie: dashboard up, Chip count 0 status, 8 W idle draw, and no mining until a human intervenes — the watchdog can't act because the stratum task never starts. The very next soft reset initialized fine, so on marginal hardware this is a transient that deserves a retry, not a permanent strand. The fix keeps the UI reachable for 5 minutes (so debugging/flashing stays possible), then restarts to retry init.

With this, all three field-observed modes on this unit are handled: silent wedge, babbling wedge, and failed-init strand.