Conversation
Promote four debug! reject-path log lines to info! so the ABI-decoded revert string + gas/profit context surface in docker logs / Loki without RUST_LOG=debug. - victim_halted: halt opcode + gas - negative_after_gas: gross, gas_cost, deficit_wei, arb_gas - arb_reverted: decoded reason, 4-byte selector, full output hex - arb_halted: halt opcode + gas Volume at current mempool rate is ~10 lines/min sustained (50% negative_after_gas + 25% arb_reverted of ~1.2k lifetime rejects). Each line still routes through the same RejectReason counter, so dashboards are unchanged. victim_reverted was already info!; arb_revert / negative-after-gas were the two biggest blind spots (75% of rejects combined).
Adds three Prom HistogramVec series, all labelled with the same
`reason` set as `aether_mempool_backrun_rejected_total` (plus
`"accepted"` for survivors):
aether_mempool_backrun_gross_profit_eth{reason}
aether_mempool_backrun_gas_cost_eth{reason}
aether_mempool_backrun_deficit_eth{reason}
The reject counter already tells us how many candidates landed in
each of the eight reject buckets. It does not tell us by how much
they missed. With 50% of rejects today being `negative_after_gas`
and 25% `arb_reverted`, the actionable next question is whether
those misses cluster near zero (gas-model tuning win) or span
orders of magnitude (paths fundamentally too thin) — a question
the counter cannot answer.
Buckets are log-scale 1e-7 → 10 ETH, shared across all three
histograms so PromQL can subtract / divide buckets cleanly. Gas
cost is pinned to `params.base_fee` so the histogram matches the
simulator's own `NegativeAfterGas` accounting; the executor's
real-time gas pricing remains a separate Go-side metric.
Deficit observation is gated on `arb_gas_used > 0` so
`rpc_transport` / `sim_timeout` rejects (which never reach the
arb leg) do not skew the deficit histogram toward 0.
Three Grafana panels appended to `mempool.json`:
- p50 / p95 deficit by reason (timeseries)
- negative_after_gas deficit heatmap
- p50 gross profit by reason (timeseries)
`wei_to_eth_f64` helper does a two-stage divide to stay under
the 2^53 f64 mantissa boundary for typical (sub-1 ETH) backrun
samples; covered by `wei_to_eth_f64_handles_typical_range`.
A second test (`backrun_reject_histograms_emit_per_reason_label`)
exercises every reject label + accepted, so a future rename of
`RejectReason::as_str()` cannot silently break the dashboard.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Releases two observability PRs already merged to
develop:obs(sim): surface backrun reject reasons in stdout by default— promotes 4debug!reject paths toinfo!so revert strings + deficit context appear in Loki withoutRUST_LOG=debugobs(metrics): histograms for backrun gross / gas / deficit per reason— adds 3HistogramVecseries labelled byRejectReason, plus 3 Grafana panels onmempool.jsonNo schema / proto / breaking changes. Purely additive observability.
Why now
Production funnel shows
attempted: 384 → validated: 0with 75% of rejects in two buckets (negative_after_gas50%,arb_reverted25%). Counters tell us which bucket; these PRs tell us why (Loki strings) and by how much (deficit p50/p95).Test plan
cargo buildclean across both crates touched (aether-grpc-server,aether-simulator)cargo clippy --lib --no-depsclean on both cratescargo test metrics::— 4 passed (2 new)cargo test mempool_backrun::— 9 passed, 1 ignored (live-fork only)mempool.jsonJSON valid (25 panels total, 3 new){service="aether-rust"} |= "mempool-backrun: arb leg reverted"returns lines withreason=field