Skip to content

ERA file consumer and producer via LCLI#69

Open
dapplion wants to merge 26 commits into
unstablefrom
era-lcli
Open

ERA file consumer and producer via LCLI#69
dapplion wants to merge 26 commits into
unstablefrom
era-lcli

Conversation

@dapplion
Copy link
Copy Markdown
Owner

@dapplion dapplion commented Mar 9, 2026 •
edited
Loading

Plan

  • Add ERA file support via LCLI commands to not touch production paths
  • Once that's properly battle tested we can add this functionality as part of the beacon CLI

PR summary

  • Import ERA files into Lighthouse's cold DB via lcli era-import, verify via
    historical_roots / historical_summaries
    • Export ERA files from the cold DB via lcli era-export (round-trip produces
      byte-identical output)
    • After import, advance store metadata (split, anchor, fork choice) so the chain
      boots via the regular resume_from_db path — no special startup logic needed
    • --era-trusted-state flag for faster import when a trusted state root is available
      (skips per-ERA verification)
    • Test vectors from an external minimal-preset testnet (downloaded at test time)

Design notes

  • ERA consumer verifies block roots against the boundary state's block_roots and era
    roots against historical_roots/historical_summaries. Block signatures are NOT
    verified.
  • init_genesis_store (in era/store_init.rs, not in the prod path) sets up the store
    from genesis before import
  • advance_store_to_era constructs fork choice via from_persisted rather than
    get_forkchoice_store, because the ERA boundary state's latest_block_header may
    reference a block in the next ERA that isn't stored
  • Writes both BeaconStateRoots (slot→root) and BeaconColdStateSummary (root→slot)
    indices for every slot so get_state works for intermediate cold slots

Existing tests

  • era_test_vectors — imports 13 ERA files, verifies block/state indices, round-trips
    through producer
  • chain_boots_from_imported_db — boots chain via resume_from_db, verifies canonical
    head, queries every slot's state and block through the HTTP API code path, validates
    parent chain linkage
  • Corruption tests: rejects corrupted blocks, states, wrong era roots, wrong block
    roots, corrupt block summaries
  • Trusted state root: accepts correct root, rejects wrong root, rejects mutated state
    with correct root

TODO tests

dapplion added 9 commits March 9, 2026 00:48
@dapplion
Implement ERA consumer and producer in lcli
0444731
@dapplion
Move ERA test vectors to external repo, download at test time
12efdae 
Test vectors are now hosted at dapplion/era-test-vectors and downloaded
via Makefile (same pattern as slashing_protection interchange tests).
@dapplion
Add module-level documentation to ERA consumer and producer
82d7e68
@dapplion
Improve ERA consumer: docs, cleanup, and tests
4ef1854 
- Add docs to EraFileDir, import_all, and module-level usage example
- Rename let _span to let _ for debug spans
- Remove unused _start_slot variable
- Extract parse_era_filename with unit tests
- Add rejects_wrong_trusted_slot test
@dapplion
Add EraImportTrust enum and --era-trusted-state CLI flag
b908151 
EraFileDir::new now takes genesis_validators_root and EraImportTrust:
- TrustedStateRoot(era_number, root): uses that ERA as reference,
  verifies its state root, imports only ERAs 0..=era_number
- Untrusted: uses highest ERA in directory as reference

Trust checks (genesis_validators_root, state root) moved from
import_all/import_era_file into EraFileDir::new. Removed all
expects/unwraps from production code.
@dapplion
Add happy-path test for TrustedStateRoot import
2aca773
@dapplion
Simplify ERA test helpers and reduce duplication
11c8d7c
@dapplion
Boot BeaconChain from ERA-imported store via resume_from_db
ddbda6f 
Add `init_genesis_store` + `advance_store_to_era` so that after ERA import
the store metadata (split, anchor, fork choice) is fully set up for the
regular `resume_from_db` → `build()` startup path.

Key changes in consumer.rs:
- `init_genesis_store`: standalone genesis init (block, state, anchor, fork choice)
- `advance_store_to_era`: advances split/anchor/fork choice to ERA boundary
- `write_state_root_index_for_era`: writes both BeaconStateRoots (slot→root)
  and BeaconColdStateSummary (root→slot) for every slot
- Uses `from_persisted` instead of `get_forkchoice_store` to avoid deriving
  a wrong anchor block root from the ERA boundary state's latest_block_header

Test `chain_boots_from_imported_db` verifies:
- canonical_head matches expected head root
- Every slot's state is accessible via state_root_at_slot → get_state
- Every slot's block is accessible via block_root_at_slot → get_blinded_block
- Blocks form a valid parent chain (parent_root linkage)

Also fixes producer to use get_blinded_block + make_full_block for cold blocks
where get_full_block fails when prune_payloads is enabled.
@dapplion
Move init_genesis_store to era/store_init.rs
d2b4a2b 
Separate store initialization from the ERA consumer since it's not part
of the production beacon node path.
@dapplion dapplion requested a review from michaelsproul as a code owner March 9, 2026 05:54
dapplion and others added 17 commits March 9, 2026 01:18
@dapplion
Fix cargo fmt
7dbb7c3
@dapplion
Fix clippy unnecessary_map_or lint
00941e5
@dapplion
fix boundary block bug, crash-resume, parallel reconstruction
059be09
@dapplion
fix cli init, max_era off-by-one, use store_cold_state_summary
8d0ed18
@dapplion
crash-resume, atomic metadata, remove dead code
6e01e80
@michaelsproul
Spec v1.7.0 alpha.3 (sigp#8988)
95b99ee 
Update spec code for compliance with spec v1.7.0-alpha.3: https://github.com/ethereum/consensus-specs/releases/tag/v1.7.0-alpha.3

The actual consensus changes are minimal. There are few more changes that are only relevant to fork choice or P2P validation that we will pick up in future PRs.

The change "Ignore beacon block if parent payload unknown" is currently covered in a hacky way by `load_parent` and can be improved once we have fork choice.

The change "Add parent_block_root to bid filtering key" is relevant to bid gossip validation, which we don't have at all in unstable yet.


Co-Authored-By: Michael Sproul <michael@sigmaprime.io>
@eserilev
Unknown block for envelope (sigp#8992)
17d183e 
Add a queue that allows us to reprocess an envelope when it arrives over gossip references a unknown block root. When the block is finally imported, we immediately reprocess the queued envelope.

Note that we don't trigger a block lookup sync. Incoming attestations for this block root will already trigger a lookup for us. I think thats good enough


  


Co-Authored-By: Eitan Seri- Levi <eserilev@gmail.com>
@macladson
Remove lighthouse/analysis endpoints (sigp#8968)
a965bfd 
Some of our custom `lighthouse/analysis` endpoints will require maintenance for the Gloas hard fork. We have decided instead to remove those endpoints. We don't utilize them internally and they have pretty limited utility and so we feel they are not worth maintaining.


  Remove `lighthouse/analysis/attestation_performance` and `lighthouse/analysis/block_packing_efficiency` endpoints.


Co-Authored-By: Mac L <mjladson@pm.me>
@michaelsproul
Gloas cold DB (sigp#8991)
0602522 
Closes:

- sigp#8958


  - Update the `HotColdStore` to handle storage of cold states.
- Update `BeaconSnapshot` to hold the execution envelope. This is required to make `chain_dump`-related checks sane, and will be generally useful (see: sigp#8956).
- Bug fix in the `BlockReplayer` for the case where the starting state is already `Full` (we should not try to apply another payload). This happens on the cold DB path because we try to replay from the closest cached state (which is often full).
- Update `test_gloas_hot_state_hierarchy` to cover the cold DB migration.


Co-Authored-By: Michael Sproul <michael@sigmaprime.io>

Co-Authored-By: Michael Sproul <michaelsproul@users.noreply.github.com>
@barnabasbusa
fix: update kurtosis apt source to sdk.kurtosis.com (sigp#9000)
54d62d0 
Co-Authored-By: Barnabas Busa <busa.barnabas@gmail.com>
@michaelsproul
Bump rustls and ignore unpatched version due to Warp (sigp#9010)
8f9c1ca 
Fix the cargo-audit failure caused by:

- https://rustsec.org/advisories/RUSTSEC-2026-0049

We can't fix it completely yet because `warp 0.3` is keeping us on an old version of `rustls`.

Mac's PR here will fix it:

- sigp#9001


Co-Authored-By: Michael Sproul <michael@sigmaprime.io>
@michaelsproul
Merge remote-tracking branch 'origin/release-v8.1' into backmerge-audit
de77da6
@eserilev
Carry forward withdrawals from the current BeaconState when a paren…
b3d5185 
…t envelope is missed (sigp#9014)

Co-Authored-By: Eitan Seri- Levi <eserilev@gmail.com>
@Alleysira @pawanjay176
fix(network): set ENR nfd to zero bytes when next fork is unknown (s…
7ffc637 
…igp#9009)

Fixes sigp#8996


  When no next fork is scheduled, the `nfd` field in the ENR was set to the current fork digest via
`.unwrap_or_else(|| ctx.fork_context.current_fork_digest())`.

According to the [spec](https://github.com/ethereum/consensus-specs/blob/1baa05e71148b0975e28918ac6022d2256b56f4a/specs/fulu/p2p-interface.md?plain=1#L636-L637), `nfd` should be zero-valued bytes when the next fork is unknown.


Co-Authored-By: Alleysira <1367108378@qq.com>

Co-Authored-By: Alleysira <56925051+Alleysira@users.noreply.github.com>

Co-Authored-By: Pawan Dhananjay <pawandhananjay@gmail.com>
@dapplion
merge unstable
06c9b34
@dapplion
fix BeaconSnapshot missing execution_envelope field
582b4a7
@dapplion
remove task docs and log file
2d42f63
This was referenced May 10, 2026
Open
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@michaelsproul michaelsproul Awaiting requested review from michaelsproul michaelsproul is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@dapplion @michaelsproul @eserilev @macladson @barnabasbusa @Alleysira