Skip to content

Recurring action plan rules (win slot 31 every epoch, withhold the payload) - #147

Merged
qu0b merged 4 commits into
mainfrom
qu0b/action-plan-recurring-rules
Jul 28, 2026
Merged

Recurring action plan rules (win slot 31 every epoch, withhold the payload)#147
qu0b merged 4 commits into
mainfrom
qu0b/action-plan-recurring-rules

Conversation

@qu0b

@qu0b qu0b commented Jul 24, 2026

Copy link
Copy Markdown
Member

Why

The per-slot action plan can already express "win this slot and don't reveal it" — but only slot by slot, for a bounded horizon. Devnet testing wants the scenario to hold for the whole run: the idea came out of a client-team discussion about testing an entire devnet with slot 31 missing, to exercise how clients attest when a slot at the epoch boundary produces no execution payload.

This adds recurring rules: a plan template matched against the slot index within the epoch.

What

curl -X POST .../api/buildoor/action-plan/rules -d '{"rules":[{
  "id":"slot31-withhold","enabled":true,"slots_in_epoch":[31],
  "bid":{"mode":"custom","bid_subsidy":1000000000,"ignore_missing_prefs":true},
  "reveal":{"mode":"disabled"}}]}'

Slot 31 of every epoch is now bid for (build forced past the schedule, exempt from the next_n budget) and, once won, never revealed — the beacon block exists, its execution payload does not. Set-and-forget, persisted in the state-db.

A SlotRule carries the same categories as a SlotPlan (bid, builder_api, reveal, build, transforms) plus a matcher: slots_in_epoch and optional from_epoch/to_epoch bounds.

Semantics

  • Precedence is wholesale, never merged: an explicit per-slot plan replaces the rule entirely. SlotPlan.IgnoreRules opts a single slot out completely — without it there is no way to make one rule-covered slot behave normally, since an otherwise empty plan is dropped and hands the slot straight back to the rule. Such a plan therefore counts as an instruction and is stored.
  • Rules resolve at freeze time, exactly like the global config baseline, so editing a rule never rewrites an already frozen slot. The materialized plan carries rule_id, is synthesized on every read and never persisted per slot — so it flows into FrozenPlan.Plan and into each slot result's applied_plan.
  • One matcher, no divergence: GetRange returns effective plans (stored + rule-derived), so the UI needs no matching logic. Rule plans are only synthesized for still-open slots (> currentSlot, not yet frozen) — past slots keep their recorded history instead of being described by today's rules.
  • Lowest id wins when several rules match. The set is replaced atomically (all-or-nothing validation, max 32 rules) and persists in the kv_store slot_rules namespace.

API

GET /api/buildoor/action-plan/rules rules, id-ascending (= match order)
POST /api/buildoor/action-plan/rules atomic full-set replace (auth + audit)
action_plan_rules_updated SSE with the authoritative set; the grid refetches its range

POST /api/buildoor/action-plan gains a top-level ignore_rules member. GET /api/buildoor/action-plan now also returns rule-derived plans (marked rule_id) for the open slots of the range.

WebUI

Action Plan tab gets a Recurring Rules panel (list / toggle / edit / delete) with two scenario presets, dashed chips on rule-derived cells, and an "ignore recurring rules" checkbox in the slot modal. Opening a rule-covered slot pre-fills the rule's values and explains that saving detaches the slot into an explicit plan. The category form widgets moved to planForms.tsx so the slot and rule editors share one implementation.

Notes

  • Scope is the ePBS flavour (win the bid, withhold the envelope). The pre-Gloas "serve the header, then refuse the unblind so the proposer misses the beacon block entirely" variant is not part of this PR.
  • The preset bids with a subsidy (added on top of the block value) rather than an absolute bid_value_gwei (which replaces it). Measured on devnet-7: block values run 0.25–0.43 ETH and bids land at 0.4–0.5 ETH, so the original absolute 1,000,000 gwei would have underbid every competitor — and a rule that never wins the slot can never withhold it, silently, because the bid submits fine and just loses.

Testing

go build ./... and go test ./... pass; gofmt and golangci-lint clean on the touched packages. New coverage in pkg/action_plan/rules_test.go (matching, epoch bounds, validation, atomic replace, clone isolation, wholesale precedence, the opt-out, freeze snapshotting, range synthesis, deterministic order, codec round-trip) and pkg/webui/handlers/api/action_plan_test.go (endpoint round-trip incl. rule_id in the range query, validation → 400 with nothing committed). Swagger regenerated, CLAUDE.md updated.

qu0b added 2 commits July 24, 2026 23:46
Sparse per-slot plans can only express a scenario for a bounded horizon,
but devnet testing needs scenarios that hold for the whole run - e.g.
"win slot 31 of every epoch and never reveal its payload", the ePBS
analogue of running a devnet with slot 31 missing.

A SlotRule matches slot indices within the epoch (slots_in_epoch, with
optional from_epoch/to_epoch bounds) and carries the same categories as
a SlotPlan:

  POST /api/buildoor/action-plan/rules
  {"rules":[{"id":"slot31-withhold","enabled":true,"slots_in_epoch":[31],
    "bid":{"mode":"custom","bid_value_gwei":1000000,"ignore_missing_prefs":true},
    "reveal":{"mode":"disabled"}}]}

Semantics:
- precedence is wholesale, never merged: an explicit plan for a slot
  replaces the rule, and SlotPlan.IgnoreRules opts a single slot out
  (such a plan is no longer "empty" and is stored)
- rules resolve at freeze time like the global config baseline, so a
  rule change never rewrites an already frozen slot; the materialized
  plan carries rule_id and reaches the slot result's applied_plan
- lowest id wins when several rules match; the set is replaced
  atomically (max 32) and persists in the kv_store "slot_rules"
  namespace
- GetRange returns effective plans (stored + rule-derived for the
  still-open slots), so the backend stays the only matcher

WebUI: a Recurring Rules panel with scenario presets, dashed chips on
rule-derived cells, and an "ignore recurring rules" opt-out in the slot
modal. The plan category form widgets move to planForms.tsx so the slot
and rule editors share one implementation.
- useActionPlan: deleting a stored plan hands the slot back to whatever
  recurring rule matches it, which only the server can resolve. Merging
  the null entry dropped the cell's chips until the next refetch (and a
  bulk delete blanked a whole range). Any in-range deletion now refetches
  instead of merging; setting a plan still merges, since a stored plan
  always wins over a rule.
- RuleEditModal: saving upserts by id, so creating a rule that reuses an
  existing id replaced it silently - both presets carry fixed ids, so
  applying one twice was enough. Create mode now rejects a colliding id;
  the server cannot catch this because it only sees the collapsed set.
- PlanService.Start: warn about rehydrated rules whose slot indices can
  never match the active chain spec (a state-db carried to a network with
  a different SLOTS_PER_EPOCH), instead of leaving them silently dead.
- Document that s.mu is what makes a rule swap atomic against Freeze.
- Test that rule-forced builds do not consume the next_n budget while
  scheduled builds do.
@qu0b

qu0b commented Jul 25, 2026

Copy link
Copy Markdown
Member Author

Review findings addressed in 911481f:

  • Stale grid on delete (useActionPlan.ts) — any in-range deletion in a change set now refetches instead of merging the null, so the slot correctly falls back to its recurring rule. Setting a plan still merges in place: a stored plan always wins over a rule, so no server round-trip is needed there. One extra GET per delete action; the no-op short-circuit for out-of-range changes is preserved.
  • Silent overwrite on "New Rule" (RuleEditModal.tsx) — create mode rejects an id that is already in use, with the existing rule named in the error. The server cannot catch this since the client collapses the upsert before the request; edit mode is unaffected.
  • Silently dead rules (PlanService.Start) — rehydrated rules whose slot indices exceed the active SLOTS_PER_EPOCH are logged at warn on startup. Extracted as unmatchableRules() so it is testable without log capture; covered by TestUnmatchableRulesAfterSpecChange, which also asserts such a rule does not resolve.
  • Lock invariant — comment on PlanService.mu recording that it, not the memstore locks, is what makes a rule swap atomic against Freeze.
  • Missing testTestRuleForcedBuildDoesNotConsumeNextN covers the accounting through OnSlotBuilt: forced builds leave the budget untouched, scheduled ones spend it, and the rule keeps forcing its slots after the budget is exhausted.

Not changed, deliberately: bulk ignore_rules stays write-only (the checkbox label says so, and delete covers the clearing case), rule ids stay immutable, and matchRule keeps the linear scan — the default 13-epoch window makes the 320-epoch worst case theoretical.

go build/go test ./... pass, golangci-lint clean on pkg/action_plan, tsc + webpack clean.

An absolute bid_value_gwei REPLACES the block value, so the preset's
1000000 gwei silently underbid every competitor on devnet-7, where block
values run 0.25-0.43 ETH and bids land at 0.4-0.5 ETH. A rule that never
wins the slot can never withhold it - the failure is invisible because
the bid is submitted successfully, it just loses.

A subsidy adds to the block value instead, so it outbids competitors
whatever the block is worth and needs no per-network tuning.
@qu0b

qu0b commented Jul 27, 2026

Copy link
Copy Markdown
Member Author

Deployed to devnet-7 as a single-host canary: buildoor-lodestar-ethrex-1 runs ethpandaops/buildoor:qu0b-action-plan-recurring-rules-latest (sha256:d700f47d), the other three buildoors stay on main-latest. Inventory change: ethpandaops/glamsterdam-devnets#49 + #50.

Host choice matters here: the scenario needs a builder that actually wins. lodestar-ethrex and prysm-ethrex win on devnet-7; lighthouse-geth currently does not (its bids are rejected with bid failed gossip verification: InvalidGasLimit), so a canary there could never win slot 31 and could never withhold it. prysm-ethrex stays on main-latest as a winning control.

Verified on the host: container running, GET /api/buildoor/action-plan/rules{"rules":[]} (the same path on a main-latest host returns the SPA index.html, i.e. the endpoint genuinely is new — same contrast holds publicly against the prysm-ethrex control), no errors in the log, and the builder is bidding and winning — Bid submitted ×2 plus Our payload was included in a beacon block! within 4 minutes of the restart.

Deploying the image arms nothing. The rule set starts empty and persists in the state-db (/data/buildoor.sqlite), so it survives container recreates and watchtower rolls. Turning the scenario on is one POST, and since it makes an epoch-boundary slot payload-less for the whole network it is the devnet owners' call, not something this PR does on its own:

curl -X POST https://api-buildoor-lodestar-ethrex-1.srv.glamsterdam-devnet-7.ethpandaops.io/api/buildoor/action-plan/rules \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" -d '{"rules":[{
    "id":"slot31-withhold","enabled":true,"slots_in_epoch":[31],
    "bid":{"mode":"custom","bid_subsidy":1000000000,"ignore_missing_prefs":true},
    "reveal":{"mode":"disabled"}}]}'

enabled:false (or the panel's pause button) stops it again without deleting the rule.

Also pushed in 8f8c1e9: the win-and-withhold preset now uses bid_subsidy instead of an absolute bid_value_gwei. Live devnet-7 numbers made the old preset a silent no-op — block values run 0.25–0.43 ETH and bids land at 0.4–0.5 ETH, against a preset bidding 0.001 ETH. A subsidy adds to the block value, so it outbids competitors whatever the block is worth and needs no per-network tuning.

Measured on devnet-7: competing builders bid 0.17-0.29 ETH, so +0.2 ETH on
top of the block value wins the auction. The previous 1 ETH was a round
number with no basis, and an oversized bid bought nothing.

Also drop ignore_missing_prefs from the preset - proposer preferences
arrive reliably over gossip, and bidding without them risks a
fee_recipient/gas_limit mismatch that CLs ignore silently.

Note in the docs that a rule losing the auction withholds nothing and
shows up only as the grid's "bid, not included" dot.
@qu0b

qu0b commented Jul 27, 2026

Copy link
Copy Markdown
Member Author

devnet-7 validation: 17 slot-31s observed

The scenario works end-to-end on a live multi-client devnet. Every slot-31 got a bid with the frozen rule’s economics, and every slot we won had its reveal suppressed — 17/17 rule correctness, 6/6 withholds. On-chain signature each time: beacon block present with our builder’s bid committed, execution payload absent, PTC voting payload_present=False, EL block number skipping the slot.

outcome n
won + payload withheld 6
no block at the slot (ambient) 4
proposer self-built 5
lost to a competing builder 1
my own misconfiguration (see below) 2

Win rate, excluding teku proposers: 6/7 = 86% of slot-31s that produced a block. All four self-build losses after the config fix had a teku proposer — four different teku nodes on four different ELs (reth, nimbusel, ethrex, geth), so it is the client, not a node or EL interaction.

No collateral damage

Measured against an 8-epoch pre-arming baseline: miss rate 18.4% → 16.3%, participation 85–89% → 86–90%, finality tracking normally throughout, beacon_builder_circuit_breaker_active 0 on lodestar (1 withheld payload per epoch stays well under its faults × window > allowedFaults × blocksPresent threshold — worth knowing before anyone widens slots_in_epoch). Our own win rate on non-slot-31 slots was 78.0% before arming and 73.5% after, inside the normal 50–93% per-epoch range: withholding does not cost us other auctions.

Two corrections to earlier claims in this PR

  1. The preset originally used an absolute bid_value_gwei of 1,000,000 — that silently underbid every competitor (devnet block values run 0.25–0.43 ETH). Fixed to a subsidy in 8f8c1e9.
  2. I then set that subsidy to 1 ETH, a round number with no basis; both slot-31s under it were lost. 98b2fb4 sizes it at +0.2 ETH against the competing builders’ observed 0.17–0.29 ETH bids, and every subsequent non-teku slot-31 was won. I could not reproduce a mechanism in any client that rejects a large bid, so I am not claiming the 1 ETH value caused those two losses — the proposers were both lodestar and the sample was 2.

Feature behaviour confirmed in production

  • Freeze semantics: a rule edit at 13:49 did not affect slot 94143, already frozen — it took effect from the next slot-31, exactly as designed.
  • rule_id provenance flows into applied_plan, so every slot result says which rule drove it.
  • Rules persisted in the state-db across the whole run; updated_by correctly recorded the JWT subject.
  • Per-slot bid_interval override worked: the default window fires exactly one bid, and 150ms produced two.

@qu0b
qu0b merged commit ccd9f30 into main Jul 28, 2026
15 checks passed
@qu0b
qu0b deleted the qu0b/action-plan-recurring-rules branch July 28, 2026 16:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants