discv4/discv5 wire fixes + hardening: ENRRESPONSE, NEIGHBORS/NODES, ENR size, bond & flood

[pk910]

Overview

Follow-up to #43. Fixes a set of discv4/discv5 wire-compatibility and hardening gaps found by an audit against go-ethereum (devp2p) and sigp/discv5, and validated with a full multi-client glamsterdam-devnet-7 e2e (bootnodoor as the sole bootnode).

Each fix is its own commit so they can be cherry-picked independently. All CI checks pass (go mod verify, go vet, gofmt, staticcheck, go test -race, make build).

Commits (one fix each)

1. discv4: fix ENRRESPONSE serializing an empty ENR record. enr.Record.EncodeRLP() has signature ([]byte, error), which does not satisfy go-ethereum's rlp.Encoder; with no exported fields the embedded record serialized as an empty list (c0), so go-ethereum rejected the reply ("record contains less than two list elements") and discv4 ENR requests against the bootnode timed out. Implement rlp.Encoder on ENRResponse. Verified end-to-end against geth: discv4 RequestENR ok (was RPC timeout).
2. discv5: cap NODES response to ≤15 nodes / ≤5 packets. go-ethereum honours only the first packet's total and reads at most 5 NODES packets; beyond that nodes were silently dropped.
3. enr: enforce the 300-byte record size limit on decode/ingest. Matches go-ethereum and sigp/discv5 — an oversized but validly-signed ENR could otherwise be stored and re-served, and a single oversized record makes clients discard the whole NODES message.
4. discv4: always send NEIGHBORS and advertise the real TCP port. An empty routing table produced a silent (zero-packet) response, making go-ethereum's querier wait out its full timeout. Also advertise the node's real TCP port from its ENR instead of the UDP port.
5. discv4: bond only after the peer answers our ping (endpoint proof). Bonding on a received PING alone let a spoofed-source PING coax a large NEIGHBORS reply to a victim (amplification) and granted FINDNODE/ENRRequest access without endpoint proof. Establish the bond only when the peer answers our ping-back with a PONG (matching go-ethereum). The ping-back is unchanged, so real peers still bond — verified against geth (discv4 RequestENR ok).
6. discv4: evict a stale node instead of dropping new peers when the map is full. The node-map cap previously returned a non-retained node once full, so under a flood of distinct signed IDs a genuinely new peer could never bond (memory-growth DoS → bonding-lockout DoS). When full, evict one unbonded entry to admit the new node; bonded, endpoint-proven peers are never evicted this way.

E2E validation (glamsterdam-devnet-7)

All EL (geth, reth, nethermind, besu, erigon) + CL (lighthouse, prysm, teku, lodestar, nimbus, grandine) clients, bootnodoor as the only bootnode, discv5 enabled where supported.

• bootnodoor discovered all 12 nodes, 11 discv5 sessions, 0 EL fork-id rejections, no bans, no invalid handshakes.
• Every client bootstrapped through bootnodoor: CL 5/5 mesh, EL 4–5/5.
• erigon ran discv5-only (v4=false v5=true) and bootstrapped cleanly — the gloas full-discv5 scenario.
• Re-run with these fixes reproduced identical metrics with zero new errors (the 300-byte limit rejected no legitimate ENR; no decode failures/panics), and discv4 bonding/ENR resolution verified against go-ethereum's stack.

Out of scope

besu discv5 needs an ENR bootnode ("ENR bootnodes must start with 'enr:'"). Not a bootnodoor bug — orchestrators must hand besu bootnodoor's /enr (not its enode). Tracked as a separate ethereum-package change.