Retry blocks with empty debug traces/stateDiffs instead of serving them

[elina-chertova]

Problem

debug_traceBlockByHash (callTracer / prestateTracer) returns exactly one entry per transaction. But fetch_debug_frames / fetch_debug_state_diffs collapse any RPC error or a null result into an empty vec![]:

match result {
    Err(_) | Ok(Value::Null) => {
        out.push(vec![]);
    }
    ...

Critically, validate_error normalizes the transient "not found" / "cannot query unfinalized data" responses to Ok(Value::Null) — i.e. exactly the block-not-ready-yet case that should be retried. Instead, the empty vector is assigned onto the block and the whole-block retry loop (enrich_block_with_retry) treats it as ready, because its readiness check (!b.is_invalid) is only ever tripped by the logs/receipts hash checks (add_logs/add_receipts) — never by empty traces. So a block whose trace call momentarily errored or returned null is silently served (and cached) with empty traces/stateDiffs.

This reproduced in production on the Rust hotblocks providers (evm-data-service-rs:0.1.4): ~40% of a 20-block eth-sepolia window had empty trace data while the upstream returned it, which forced an operational revert of all trace/statediff hotblocks datasets back to the TypeScript image.

Note: the earlier v0.1.5 fix (64a709d) addressed the trace-API (trace_replayBlockTransactions) path only; the debug-API path above was untouched and still carried the bug.

Fix

Add a completeness check in add_traces, mirroring the existing add_logs/add_receipts not-ready handling: a block that has transactions but came back with no debug trace / stateDiff entries is marked invalid, so enrich_block_with_retry re-fetches the whole block instead of accepting the gap. A block with zero transactions correctly has no entries and is left untouched (the same way an empty eth_getLogs result is correct when logsBloom == 0).

This makes a transient upstream trace error survivable by retry rather than turning it into permanently cached corrupt data — it does not "tolerate" the bad data as valid.

Test

crates/evm-source/tests/debug_traces.rs drives the real fetch pipeline against a mock node that returns a block with one transaction but null for debug_traceBlockByHash, and asserts the block is flagged not-ready.

• Red on pre-fix code: FAILED ... block with transactions but empty debug traces must be marked invalid
• Green after fix: test result: ok. 1 passed

(Verified locally with cargo test -p evm-source --test debug_traces. The repo's pipeline.rs/parity.rs suites depend on git-LFS fixtures that were not available in my sandbox, so they could not be run here; this new test builds its block JSON inline and is unaffected.)

Falsification

If, after this lands, the trace/statediff datasets re-migrated to Rust still show empty traces for blocks that have transactions (compared against the TS image or a reference node), or if enrich_block_with_retry starts bailing/crash-looping on a healthy chain head because the retry window (10×50ms) is too short for normal head-following trace lag, then this fix is wrong/insufficient.