feat: mempool shadow-run suite — discovery, WS keepalive, retention, config hardening, dashboard#184
Open
Pablosinyores wants to merge 2 commits into
Open
feat: mempool shadow-run suite — discovery, WS keepalive, retention, config hardening, dashboard#184Pablosinyores wants to merge 2 commits into
Pablosinyores wants to merge 2 commits into
Conversation
Land two net-new mempool features on develop's foundation. Hot-token discovery (AETHER_DISCOVERY=1, off by default): a HotTokenTracker records every decoded mempool swap; the discovery loop reads frequently-traded candidates, screens them with a fee-on-transfer round-trip (revm, own timeout), qualifies venue liquidity, and appends admitted pools to pools.toml + reloads the registry (capped at AETHER_MAX_ADMITTED_POOLS). The tracker is wired additively into SimContext (new field + a record() hook in the decode loop); develop's backrun design is otherwise untouched. WS idle keepalive: subscribe_once gains an idle-watchdog select arm (MEMPOOL_WS_IDLE_TIMEOUT_SECS, default 60, 0=off) that drops + reconnects when no pending tx arrives in the window, closing the silent-half-close stall. MempoolIngestMetrics is registered via with_metrics so idle_reconnect_total + the re-encode counter finally populate. Adds 4 discovery metrics (pools_admitted/rejected, fot_screen, hot_candidates). cargo clippy --workspace + grpc-server/ingestion/simulator tests green.
…+ funnel dashboard Operational hardening for the unattended public-mempool shadow run. - Reconciler retention sweep (PruneMempoolLedger + runRetentionLoop): deletes old mempool_predictions (children dropped ON DELETE CASCADE) on MEMPOOL_RETENTION_HOURS (def 48, <=0 disables) / _INTERVAL_MINS (def 60); metrics aether_mempool_retention_pruned_total / _errors_total. - demo.sh: AETHER_ARM gating (A = analytical-only, executor + backrun validator off), profit-scorer supervised under spawn_with_restart with a crash-loop bound, start_log_janitor (copytruncate $LOG_DIR/*.log past DEMO_LOG_CAP_BYTES, independent of respawn), and AETHER_DISCOVERY opt-in carrying the discovery->canary / freeze-pools.toml->measurement guardrail. - Prometheus aether.shadow.rules group (producer-down, predictions-stalled, retention-errors, reconciler-drops). - Postgres autovacuum + max_connections tuning; x-logging json-file 50MBx5 caps on all compose services. - Backrun funnel & diagnostics Grafana dashboard (auto-provisioned). go build/vet/test, promtool (12 rules), docker compose config, bash -n green.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Net-new features for the unattended public-mempool shadow run, landed on develop's foundation. No submission path is touched — all of this is observability / discovery / ops hardening.
What's included
AETHER_DISCOVERY=1, off by default) —HotTokenTrackerrecords every decoded mempool swap; the discovery loop screens frequently-traded candidates with a fee-on-transfer round-trip (revm, own timeout), qualifies venue liquidity, and appends admitted pools topools.toml+ reloads the registry (capped atAETHER_MAX_ADMITTED_POOLS). Wired additively intoSimContext(new field + arecord()hook in the decode loop) — develop's backrun design is untouched.subscribe_oncegains an idle-watchdog arm (MEMPOOL_WS_IDLE_TIMEOUT_SECS, default 60, 0=off) that drops+reconnects on a silent TCP half-close. RegistersMempoolIngestMetricssoidle_reconnect_total+ the re-encode counter populate.PruneMempoolLedger+runRetentionLoopin the reconciler (MEMPOOL_RETENTION_HOURSdef 48 /_INTERVAL_MINSdef 60; children droppedON DELETE CASCADE).AETHER_ARMgating (A = analytical-only), profit-scorer underspawn_with_restartwith a crash-loop bound,start_log_janitor(copytruncate, independent of respawn), andAETHER_DISCOVERYopt-in carrying the discovery→canary / freeze-pools.toml→measurement guardrail.aether.shadow.rulesgroup (producer-down, predictions-stalled, retention-errors, reconciler-drops).max_connections=200;x-loggingjson-file 50MB×5 caps on all compose services.Foundation-adaptation notes
router_decoder,mempool_backrun,fork,mempool_pipeline,metrics,mempool) plusengine.rs/cycle_gating.rs/price_graph.rs/pools.tomluse develop's version; only net-new feature deltas are layered on.input_amount_weiand prices gas in the revm sim — there is no static pre-revm coarse gate for it to make dynamic, so the feature is obsolete here.Validation
cargo clippy --workspaceclean · grpc-server (75) / ingestion (82) / simulator (96) /*-binstests green ·go build/vet/test·bash -n demo.sh·docker compose --profile ledger --profile node config·promtool(12 rules).