Skip to content

backport generic improvements from glamsterdam-devnets and bal-devnets - #180

Open
qu0b wants to merge 8 commits into
masterfrom
qu0b/backport-devnet-improvements
Open

backport generic improvements from glamsterdam-devnets and bal-devnets#180
qu0b wants to merge 8 commits into
masterfrom
qu0b/backport-devnet-improvements

Conversation

@qu0b

@qu0b qu0b commented Jul 27, 2026

Copy link
Copy Markdown
Member

Backports the generic improvements that ran ahead in glamsterdam-devnets and bal-devnets but never made it back here. Most of the earlier drift has already landed on master (coredevs SSH rosters, buildoor, the IPv6 rework, sync_mapping, the template-name guard, inventory-path derivation) — this is what was left.

The buildoor gap below is a bug found while auditing rather than a straight copy.

buildoor only covered 3 of 5 consensus clients

buildoor wins on group precedence via ansible_group_priority=100, so any client whose args it does not re-merge silently loses that client's own args — bootnodes included. Only prysm, lighthouse and lodestar were re-merged, which left teku and grandine buildoor nodes unusable.

Split teku/grandine into *_container_command_extra_simple_args (the pattern the other three already use), added their merge entries, and gave every client its subscribe-all-subnets flag — a builder has to see every attestation subnet to value a payload correctly.

Every flag was verified against the real binaries rather than trusted from the source repo:

Client Flags Verified against
teku --Xfork-choice-updated-always-send-payload-attributes=true, --p2p-subscribe-all-subnets-enabled consensys/teku:latest (the first is a hidden X option, probed by running it)
grandine --features=AlwaysPrepareExecutionPayload, --subscribe-all-subnets ethpandaops/grandine:develop (also confirmed repeated --features both parse)
lighthouse --always-prepare-payload, --subscribe-all-subnets, --import-all-attestations beacon_node/src/cli.rs
prysm --prepare-all-payloads, --subscribe-all-subnets, --p2p-colocation-whitelist cmd/flags.go, config/features/flags.go
lodestar --emitPayloadAttributes, --subscribeAllSubnets already live on glamsterdam-devnet-7

The rest

  • mitogen 0.3.43 → 0.3.49 (ansible.cfg + install_dependencies.sh in lockstep)
  • CI: install python build headers — asdf compiles python from source and the runner image lacks libbz2/readline/lzma, so the build silently drops those modules and pip fails later
  • noqa syntax#noqa rule is not recognised by ansible-lint; it needs # noqa: rule
  • prysm colocation whitelist — devnet peers are few and often share a /24, which prysm's colocation limit treats as a sybil cluster and refuses to peer with
  • Skip DigitalOcean VPCs on hetzner-only devnets (from bal-devnets) — an empty VPC per region is still billable surface and blocks project teardown
  • run.zsh: check_deps and send_fundscheck_deps reports every missing tool with an install hint instead of dying on the first one mid-run

Deliberately not included

Verification

  • ansible-lint playbook.yaml --profile production (the exact CI invocation) — passes
  • terraform fmt -check and terraform validate — clean
  • zsh -n scripts/run.zsh, plus run.zsh check_deps and run.zsh help executed
  • Jinja render check on all five buildoor merge expressions, confirming each client's base args survive the merge:
prysm      -> ['--a', '--bootstrap-node=enr:x', '--prepare-all-payloads', '--subscribe-all-subnets']
lighthouse -> ['--b', '--always-prepare-payload', '--subscribe-all-subnets', '--import-all-attestations']
lodestar   -> ['--c', '--emitPayloadAttributes', '--subscribeAllSubnets']
teku       -> ['--d', '--Xfork-choice-updated-always-send-payload-attributes=true', '--p2p-subscribe-all-subnets-enabled']
grandine   -> ['--e', '--features=AlwaysPrepareExecutionPayload', '--subscribe-all-subnets']

qu0b added 7 commits July 27, 2026 15:56
asdf compiles python from source and the runner image lacks libbz2/readline/lzma
headers, so the build silently drops those modules and pip fails later.
`#noqa rule` is not recognised; ansible-lint needs `# noqa: rule`.
buildoor wins on group precedence, so a client whose args it does not re-merge
loses its own base args (bootnodes included). Only prysm/lighthouse/lodestar
were re-merged, which left teku and grandine buildoor nodes unusable.

Split teku/grandine into _simple_args the same way the other three already are,
add their merge entries, and give every client its subscribe-all-subnets flag —
a builder has to see every attestation subnet to value a payload correctly.

Flags verified against consensys/teku:latest and ethpandaops/grandine:develop.

Also point buildoor at the config service's validator-ranges endpoint so the
overview can label builders by the range they serve.
Devnet peers are few and often share a /24, which prysm's colocation limit
treats as a sybil cluster and refuses to peer with.
An empty VPC per region is still billable surface and blocks project teardown.
check_deps reports every missing external tool with an install hint instead of
failing on the first one mid-run. send_funds sends plain ETH to an address.
@qu0b
qu0b force-pushed the qu0b/backport-devnet-improvements branch from d3a88ae to 9679bec Compare July 27, 2026 14:48
@qu0b-reviewer

qu0b-reviewer Bot commented Jul 27, 2026

Copy link
Copy Markdown

🤖 qu0b-reviewer

Summary

This PR backports 10 files from glamsterdam-devnets and bal-devnets, adding a check_deps/send_funds helper to run.zsh, splitting Ansible container args to allow higher-precedence groups (buildoor) to merge rather than replace them, adding --subscribe-all-subnets and related builder flags, skipping DO VPCs on Hetzner-only devnets, installing missing Python build deps, and bumping mitogen. The changes are generally well-structured and the patterns are consistent.

Issues

  • 🟡 scripts/run.zsh:585 — The new send_funds function (and pre-existing get_balance at line 206) call python -c "import sys; print(int(sys.stdin.read(), 16) / 1e18)", but the new check_deps function (line 21) verifies python3, not python. On macOS and some Linux systems (e.g. Ubuntu ≥20.04 without the python-is-python3 package), python is absent while python3 is present. A user who runs check_deps sees a clean bill of health, then send_funds (or get_balance) fails with python: command not found. This should use python3 to match what check_deps actually verifies.

Suggestions

  • scripts/run.zsh:585 — Replace python with python3 on this line (and optionally line 206 for consistency).

Reviewed @ 7e83ea3c
"Real artists ship." — Steve Jobs

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant