apollo_mempool: drop emptied gap accounts on tx expiry to fix accounts_with_gap leak

[matanl-starkware]

Problem

On mainnet Grafana (v0.14.3), mempool_accounts_with_gap never drops below 1, while mempool_stuck_txs sometimes hits 0 — an impossible pair, since a nonce-gap account must hold at least one stuck tx. One counter is leaking.

Root cause (Mempool::remove_expired_txs): when a gap account's stuck txs expire (TTL), the per-tx loop correctly decrements n_stuck_txs, but the AddressToNonce map it returns — the only input to update_accounts_with_gap — is built from queued_txs only. Stuck txs are never in tx_queue (fee mode enqueues only txs whose nonce == account nonce; a gap tx has a higher nonce), so a gap account whose stuck txs all expire is never re-evaluated and lingers in accounts_with_gap. There's no GC (only a TODO(clean_accounts)), so it self-heals only if a new tx for that account later arrives or it's committed/evicted → a persistent floor of ~1 ghost account (accounts_with_gap > 0 while stuck_txs == 0).

Evidence (GCP logs, apollo-mainnet-0, sequencer-mempool, 11:02–11:07 UTC)

• Bot accounts repeatedly enter the gap state (Account 0x04678eb4…61b5 has a nonce gap; 1 transaction(s) are now stuck., also 0x048ddc…441f, 0x05aed2…2cba).
• The same stuck tx hashes (0x3e17…ff87c, 0x4c30…1e84) are Removed expired transactions three times, exactly 5 min apart (10:51/10:56/11:01) — stuck txs expiring, P2P-rebroadcast, re-expiring.

Fix

After the existing decrement loop, drop any account expiry left with no pool txs (mirrors the emptiness cleanup already in try_make_space):

for tx_ref in &removed_txs {
    if !self.tx_pool.contains_account(tx_ref.address) {
        self.remove_from_accounts_with_gap(tx_ref.address);
    }
}

Must be a separate loop after the decrements, otherwise removing an account from the set early would make its remaining per-tx decrement_stuck_txs_if_gap_account calls no-op and undercount. remove_from_accounts_with_gap subtracts n_txs_for_address = 0 at cleanup time (no double-decrement) and is idempotent on duplicate addresses.

Note: deliberately not fixed by mapping all removed_txs into the returned nonce map — resolve_nonce(addr, hint) falls back to hint only when state has no committed/staged nonce; for a stuck tx hint = tx.nonce > account_nonce, which would mis-evaluate accounts that still hold txs. The correct signal is nonce-independent: the account has no pool txs left.

Tests

Added stuck_txs_counter_expiry_removes_emptied_gap_account (the existing _expiry_decrements_count only expires 1 of 2 txs, so the account legitimately stays and the leak was uncovered). A single stuck tx fully expires → asserts both counters return to 0. Fails pre-fix with accounts_with_gap mismatch left=1 right=0 (the exact prod symptom); passes post-fix.

Verified: full apollo_mempool suite 113/0, rust_fmt.sh clean, clippy -D warnings clean.

🤖 Generated with Claude Code

[cursor]

PR Summary

Low Risk
Targeted mempool metric/GC fix with focused tests; behavior change is limited to expiry cleanup of empty gap accounts.

Overview
Fixes a leak where mempool_accounts_with_gap could stay positive while mempool_stuck_txs hit zero after stuck (non-queued) transactions expired. Expiry already decremented stuck counts per tx, but gap accounts were only re-evaluated via queued-tx nonce updates—stuck gap txs never enter the queue in fee mode.

remove_expired_txs now runs a second pass after per-tx stuck decrements: for each removed tx, if the address has no txs left in the pool, call remove_from_accounts_with_gap. Order matters—removing the account before decrements would make later decrements no-ops and undercount stuck txs when multiple txs expire together.

Adds regression tests for one stuck tx and for two stuck txs expiring in one sweep (the latter documents the two-loop requirement).

^{Reviewed by Cursor Bugbot for commit 9c2cd20. Bugbot is set up for automated code reviews on this repo. Configure here.}

[reviewable-StarkWare]

This change is