Skip to content

feat(merge): add SSZ-REST transport for Engine-API#11301

Merged
LukaszRozmej merged 91 commits into
NethermindEth:masterfrom
Dyslex7c:feat/ssz-engine-api
May 11, 2026
Merged

feat(merge): add SSZ-REST transport for Engine-API#11301
LukaszRozmej merged 91 commits into
NethermindEth:masterfrom
Dyslex7c:feat/ssz-engine-api

Conversation

@Dyslex7c
Copy link
Copy Markdown
Contributor

@Dyslex7c Dyslex7c commented Apr 22, 2026

Replaces #10728

Summary

Implements the binary SSZ-REST Engine API transport per ethereum/execution-apis#764. The binary transport coexists with the existing JSON-RPC endpoint on the same authenticated port (8551) and is negotiated via engine_exchangeCapabilities — no breaking changes to existing behaviour.

Architecture

  • Wire types (SszWireTypes.cs) — [SszContainer] definitions for every Engine-API structure across Paris → Amsterdam: payload V1–V4, forkchoice, blob bundles V1/V2, BlobAndProof V1/V2, payload bodies V1/V2, capabilities, client version, transition configuration. Driven by the existing Nethermind.Serialization.SszGenerator source generator.

  • Codec (SszCodec.cs) — Stateless bridge between domain objects (ExecutionPayload, ForkchoiceStateV1, …) and wire types. Encoders write straight into an IBufferWriter<byte>, returning a byte count instead of a pooled buffer. Decoders accept ReadOnlySequence<byte>. Status-enum mapping, nullable-list (List[T, 1]) handling, little-endian UInt256 layout for BaseFeePerGas / TerminalTotalDifficulty.

  • Middleware (SszMiddleware.cs) — ASP.NET Core middleware on the authenticated Engine-API port. Triggers on Content-Type: application/octet-stream (POST) or Accept: application/octet-stream (GET). Performs JWT auth, route resolution (/engine/v{N}/{resource}[/{extra}]), 16 MiB body-size enforcement, and dispatches to the appropriate versioned handler. Non-SSZ requests pass through to JSON-RPC unchanged. Error-code → HTTP-status mapping (404, 409, 413, 422, 500) is per-spec.

  • Handler dispatch — Generic versioned handlers parameterized by per-version descriptor structs (INewPayloadVersion<TWire>, IForkchoiceUpdatedVersion<TWire>, IGetPayloadVersion<TResult>, IPayloadBodiesByHash/RangeVersion<TResult>, IGetBlobsV2Version). Adding a new endpoint version is one descriptor struct + one DI registration — no version switches anywhere. Every descriptor calls into IEngineRpcModule.engine_*V{N}, so JSON-RPC and SSZ-REST share the same engine-module entry points.

  • Capabilities provider (EngineRpcCapabilitiesProvider.cs) — Single Build method emits both engine_*V{N} JSON-RPC method names and SSZ-REST path strings, each gated by the same fork checks via a shared helper. Capability options live in [Flags] enum RpcCapabilityOptions : byte { None, Enabled, WarnIfMissing } with IsEnabled() / ShouldWarnIfMissing() extensions.

  • DI bridge (SszMiddlewareConfigurer.cs) — IJsonRpcServiceConfigurer that bridges Autofac (where Engine-API handlers live) to ASP.NET's MS DI container so Kestrel can resolve SszMiddleware and its handlers without duplicating registration.

Performance

  • Encode side — Encoders write directly into ctx.Response.BodyWriter (a PipeWriter) via GetSpan / Encode / Advance / set ContentLength from the returned byte count / FlushAsync. No intermediate ArrayPoolSpan<byte> rent + copy, no MemoryStream rebuild. Connection is aborted if encode throws mid-stream so the CL never sees a 500 with garbled binary.

  • Decode side — Middleware reads via PipeReader.ReadAtLeastAsync(ContentLength) and hands the resulting ReadOnlySequence<byte> straight to the generator-emitted Decode(ReadOnlySequence<byte>, out T) overload. Single-segment input is zero-copy; multi-segment consolidates once via ArrayPool then dispatches to the existing span decoder. Eliminates the MemoryStream + ToArray intermediate copy that was dominant on blob-bearing payloads. SszLib gains primitive sequence overloads (bool/byte/ushort/uint/int/ulong/long/UInt128/UInt256/BitArray) as infrastructure for a future fully-recursive sequence-aware decoder.

  • RoutingFrozenDictionary with AlternateLookup<ReadOnlySpan<char>> for zero-alloc resource lookup. Path extra (e.g. payloadId) threaded as ReadOnlyMemory<char> end-to-end. Span<char>-based hex parse for payload IDs avoids substring allocation.

  • Wire-encoder dedupMerkleizeRefType{Vector,List,ProgressiveList} collapsed into one core helper; small chunks stack-allocate (≤4 chunks), larger payloads use ArrayPool. Single-allocation Blob V1/V2 encoders (count nulls first, allocate exact size).

Spec conformance

Aligned with ethereum/execution-apis#764:

  • application/octet-stream content/accept negotiation
  • 400 Bad Request for malformed SSZ bodies
  • 404 Not Found for unknown payload IDs / endpoints
  • 409 Conflict for InvalidForkchoiceState
  • 413 Payload Too Large past the 16 MiB limit
  • 422 Unprocessable Entity for InvalidPayloadAttributes
  • 500 Internal Server Error for InternalError

Follow-up

Issue #11525 tracks SSZ generator support for InlineArray<T, N> to remove the byte[] allocation in Bytes32 / Hash256 value paths.

What types of changes does your code introduce?

  • Bugfix (a non-breaking change that fixes an issue)
  • New feature (a non-breaking change that adds functionality)
  • Breaking change (a change that causes existing functionality not to work as expected)
  • Optimization
  • Refactoring
  • Documentation update
  • Build-related changes
  • Other: Description

Testing

Requires testing

  • Yes
  • No

If yes, did you write tests?

  • Yes
  • No

Notes on testing

  • SszCodecTests (29 tests) — Encode/decode roundtrips for payload status, forkchoice response, transition configuration, capabilities, blob requests, payload bodies V1/V2, and getPayload responses V1–V6. Spec-defined byte-offset assertions for fixed sections of ExecutionPayload, PayloadStatus, ForkchoiceUpdatedResponse, GetPayloadResponseV3, GetPayloadResponseV4, PayloadAttributesV3 (regression for the parent_beacon_block_root placement bug).

  • SszMiddlewareTests (26 tests) — Drive SszMiddleware through DefaultHttpContext (no Kestrel required). Verify handler dispatch per version, route fallback (versioned → version-agnostic), Cache-Control: no-store on getPayload responses, status-code mapping, capability advertising, and 401/404/413/422 error paths.

  • SszMultiSegmentDecodeTests (26 tests) — Cover the multi-segment ReadOnlySequence<byte> branch that production traffic from Kestrel actually hits (4 KB pooled blocks). Builds explicit multi-segment sequences via ReadOnlySequenceSegment<byte> chains at boundary sizes 1/3/7/4096 across five wire types (TransitionConfig, Capabilities, GetBlobsRequest, GetPayloadBodiesByRange, NewPayloadV3RequestWire) and the SszLib primitive boundary path (uint/ulong/UInt256).

@LukaszRozmej
Copy link
Copy Markdown
Member

@claude review

@claude
Copy link
Copy Markdown
Contributor

claude Bot commented Apr 22, 2026

Claude finished @LukaszRozmej's task in 4m 10s —— View job


PR Review: feat(merge): add SSZ-REST transport for Engine-API

Tasks

  • Gather context and read rule files
  • Read all changed files
  • Analyze for correctness, security, performance
  • Post inline comments for specific issues
  • Write final review summary

Summary

Good foundation — the middleware architecture, route dispatch, and SSZ codec coverage are well-structured. Several issues need to be addressed before merge:


High

1. Memory exhaustion / DoS in ReadBodyAsync (SszMiddleware.cs:458–476)

MemoryStream grows without bound before the second size check. When Content-Length is absent (e.g. chunked transfer encoding), an authenticated client can stream multiple gigabytes of data, causing the node to OOM before the 16 MiB gate fires. This is the authenticated port, but a stolen or leaked JWT, or a misconfigured CL client, would be sufficient.

// Current – dangerous with chunked bodies
using MemoryStream ms = new();
await ctx.Request.Body.CopyToAsync(ms);   // unbounded growth here
if (ms.Length > MaxBodySize) { throw}

Fix: pre-allocate the stream at the limit and abort early if the data overflows, or use Stream.CopyToAsync with a wrapper that stops at the byte limit:

using MemoryStream ms = new(MaxBodySize + 1);
byte[] buffer = ArrayPool<byte>.Shared.Rent(81920);
try {
    int read;
    while ((read = await ctx.Request.Body.ReadAsync(buffer, ctx.RequestAborted)) > 0) {
        if (ms.Length + read > MaxBodySize)
            throw new InvalidOperationException($"Request body exceeds {MaxBodySize} bytes");
        ms.Write(buffer, 0, read);
    }
} finally { ArrayPool<byte>.Shared.Return(buffer); }

2. int.Parse overflow on version number (SszMiddleware.cs:107)

The regex captures \d+ with no length cap. A URL like /engine/v99999999999/payloads causes int.Parse to throw OverflowException, which is caught by the outer catch and returns a 500 with the exception message. Should return 404 instead.

// Current
int version = int.Parse(match.Groups["version"].Value);  // OverflowException → 500

// Fix
if (!int.TryParse(match.Groups["version"].Value, out int version))
{
    await WriteErrorAsync(ctx, StatusCodes.Status404NotFound, $"Invalid version in path: {path}");
    return;
}

Medium

3. Internal exception messages leaked to callers (SszMiddleware.cs:130–132)

catch (Exception ex)
{
    if (_logger.IsError) _logger.Error($"SSZ-REST handler error for {path}", ex);
    await WriteErrorAsync(ctx, StatusCodes.Status500InternalServerError, ex.Message);
}

ex.Message for SSZ decode failures or null-ref exceptions can contain file paths, type names, or internal state. The spec only requires an HTTP 500 with an opaque error; return a generic message and keep the detail in the log:

await WriteErrorAsync(ctx, StatusCodes.Status500InternalServerError, "Internal server error");

4. Missing CancellationToken in async I/O (SszMiddleware.cs:468, 413)

Per robustness.md: "Async methods that perform I/O or network calls must accept a CancellationToken". Neither CopyToAsync (body read) nor WriteAsync (response write) is passed ctx.RequestAborted:

await ctx.Request.Body.CopyToAsync(ms);          // should use ctx.RequestAborted
await ctx.Response.Body.WriteAsync(data);         // should use ctx.RequestAborted

A client disconnect leaves the body-read in flight indefinitely.

5. No graceful shutdown check

The JSON-RPC fast lane in Startup.cs (line 298) checks _jsonRpcProcessor.ProcessExit.IsCancellationRequested before accepting work. SszMiddleware has no equivalent check, so during node shutdown the SSZ endpoint keeps processing requests while the rest of the pipeline is draining.

6. newPayloadV5 vs getPayloadV5 format asymmetry — needs spec reference

DecodeNewPayloadRequest for version >= 5 decodes an Amsterdam-format ExecutionPayloadV4Wire (with BlockAccessList/SlotNumber), but HandleGetPayloadAsync for version == 5 encodes a Pectra-format ExecutionPayloadV3Wire (no Amsterdam fields). A CL client that calls POST /engine/v5/payloads with an Amsterdam payload would then expect GET /engine/v5/payloads/{id} to return the same Amsterdam format — and would receive a Pectra-format response instead, which would fail deserialization.

Please add a comment citing the relevant section of ethereum/execution-apis#764 that defines this split, or fix the mapping so the same version number is consistent in both directions. Without the spec reference it's impossible for reviewers to verify this is intentional.


Low

7. Indentation inconsistency in SszCodec.cs (lines 495–656)

Private methods PayloadV3FromV1Wire, PayloadV3FromV2Wire, PayloadV3FromV3Wire, PayloadToV1WirePayloadV4FromV4Wire are indented at column 0 instead of the class-level 4 spaces. This is inconsistent with the rest of the file and breaks editorconfig formatting.

8. Duplicated pattern match for ExecutionPayloadV3 (SszCodec.cs:607–609)

BlobGasUsed   = ep is ExecutionPayloadV3 v3  ? v3.BlobGasUsed   ?? 0 : 0,
ExcessBlobGas = ep is ExecutionPayloadV3 v3b ? v3b.ExcessBlobGas ?? 0 : 0   // separate match, different var name

Extract once: ExecutionPayloadV3? v3 = ep as ExecutionPayloadV3;

9. PUT treated as a potential SSZ request (SszMiddleware.cs:444)

IsSszRequest checks is "POST" or "PUT". PUT is not defined by the Engine API SSZ spec; this is dead code that silently accepts any PUT with application/octet-stream content-type.

10. User-supplied path echoed in 404 error bodies (SszMiddleware.cs:103)

await WriteErrorAsync(ctx, StatusCodes.Status404NotFound, $"Unknown SSZ endpoint: {path}");

Harmless in plain text (text/plain), but good practice not to reflect raw request data in error bodies.

11. SszCapabilityName list byte limit (SszWireTypes.cs:312)

[SszList(64)] means the Name field supports up to 64 bytes. Capability strings like "POST /engine/v1/payloads/bodies/by-range" are 42 bytes; this is fine today, but a future long name could silently truncate. Consider bumping to 128 to match room for growth, and add a comment referencing the spec limit.


Code coverage gaps

  • No test for authentication failure path (401)
  • No test for oversized body (413)
  • No test for unknown path (404)
  • No test for malformed SSZ body (should return 500 without panicking)
  • No test for engine_getBlobsV1 with a null entry in the result (the 204 path)
  • No round-trip test for newPayloadV4/newPayloadV5 decode → encode

Comment thread src/Nethermind/Nethermind.Merge.Plugin/SszRest/SszMiddleware.cs Outdated
Comment thread src/Nethermind/Nethermind.Merge.Plugin/SszRest/SszMiddleware.cs Outdated
Comment thread src/Nethermind/Nethermind.Merge.Plugin/SszRest/SszMiddleware.cs Outdated
Comment thread src/Nethermind/Nethermind.Merge.Plugin/SszRest/SszMiddleware.cs Outdated
@LukaszRozmej LukaszRozmej marked this pull request as draft April 22, 2026 21:57
@LukaszRozmej
Copy link
Copy Markdown
Member

@claude re-review

@claude
Copy link
Copy Markdown
Contributor

claude Bot commented Apr 24, 2026

Claude finished @LukaszRozmej's task in 8m 0s —— View job


PR Re-Review: feat(merge): add SSZ-REST transport for Engine-API

Tasks

  • Load rule files
  • Read changed files
  • Analyze correctness, security, performance
  • Post inline comments
  • Write final review summary

What Was Fixed Since the Last Review

All six High issues from the previous review are resolved:

# Issue Status
1 Memory exhaustion in ReadBodyAsync (unbounded MemoryStream) ✅ Fixed — chunked read with ArrayPool enforcing limit during read
2 int.Parse overflow → 500 on long version numbers ✅ Fixed — int.TryParse
3 Internal exception messages leaked to callers ✅ Fixed — returns "Internal server error"
4 Missing CancellationToken in async I/O ✅ Fixed — ctx.RequestAborted passed to ReadAsync and WriteAsync
5 No graceful shutdown guard ✅ Fixed — _processExitToken.IsCancellationRequested check added
6 newPayloadV5/getPayloadV5 asymmetry undocumented ✅ Addressed — spec comment at lines 77–81

All five test-coverage gaps from the previous review are now covered: 401 auth failure, 413 oversized body, 404 unknown path, malformed SSZ → 500, 204 null-blob path.

PUT as fake SSZ method removed, indentation fixed, duplicate V3 pattern match consolidated.


Remaining Issues

High

1. PayloadAttributesWire is version-unaware — V1/V2/V3 forkchoice will fail SSZ decode (inline comment)

PayloadAttributesWire always encodes all six fields including SlotNumber (8 bytes). The SSZ-REST spec defines version-specific PayloadAttributes types:

  • V1 = timestamp + prevRandao + feeRecipient — 60-byte fixed, no variable section
  • V2 = V1 + withdrawals
  • V3 = V2 + parentBeaconBlockRoot
  • V4 = V3 + slotNumber

When a CL sends engine_forkchoiceUpdatedV1 with payload attributes, it sends the V1-format bytes (60 bytes fixed + variable offset header). The decoder, however, parses a struct that includes SlotNumber as a fixed field, producing a layout 8 bytes larger. The decode silently corrupts or throws.

The fix is to dispatch to version-specific payload-attributes wire types in SszCodec.DecodeForkchoiceUpdatedRequest.

2. ExecutionPayload SSZ includes SszWithdrawals, breaking V1 newPayload format (inline comment)

NewPayloadV1RequestWire embeds ExecutionPayload whose generated SSZ codec includes SszWithdrawals (a variable-length list with a 4-byte offset in the fixed header). Per the spec ExecutionPayloadV1 is fully fixed-size — no withdrawals field. A CL sending a spec-compliant V1 payload produces fewer bytes than this decoder expects, causing parse corruption or failure. Nulling Withdrawals after decode only affects the domain object; it does not fix the wire layout.

A separate V1 wire type (or suppressing SszWithdrawals from the V1 codec path) is required.


Medium

3. EncodeGetPayloadV1Response temporarily mutates a shared domain object (inline comment)

The method sets ep.Withdrawals = null, encodes, then restores. If the same ExecutionPayload is ever read concurrently (e.g. payload cache, background metric flush), a race will produce a corrupted SSZ body. Currently safe because handlers produce per-request objects, but this needs a code comment explaining the safety invariant so it isn't broken by future caching.

4. SszCapabilityName 64-byte limit (inline comment)

Still 64 bytes; unaddressed from the prior review. 128 bytes recommended with a spec citation.


Low

5. LINQ .ToArray() on IReadOnlyList<byte[]> (inline comment)

b.Transactions.ToArray() and b.Withdrawals?.ToArray() in EncodePayloadBodiesV1/V2Response use the LINQ extension, not a collection method. Per coding-style.md: "No LINQ when a simple for/foreach works."

6. Leftover commented-out using directive (inline comment)

// using Nethermind.Core.ExecutionRequest; at ExecutionPayload.cs:14.

7. User-supplied path echoed in 404 error body (SszMiddleware.cs:217)

$"Unknown SSZ endpoint: {path}" reflects the raw request path. Harmless in text/plain over the local authenticated port, but good hygiene to strip it.


Summary

Good progress — all previous High findings are resolved, and the new tests are a meaningful improvement. The two new High issues relate to SSZ wire-format mismatches for older versioned endpoints (PayloadAttributesWire has extra bytes for V1–V3, and ExecutionPayload has a withdrawals field that V1 doesn't). These will cause protocol-level decode failures when CL clients use the V1 SSZ endpoints. The Medium object-mutation issue should at minimum get a safety comment. Blocking on High items.

Comment thread src/Nethermind/Nethermind.Merge.Plugin/SszRest/SszWireTypes.cs
Comment thread src/Nethermind/Nethermind.Merge.Plugin/SszRest/SszWireTypes.cs Outdated
Comment thread src/Nethermind/Nethermind.Merge.Plugin/SszRest/SszCodec.cs Outdated
Comment thread src/Nethermind/Nethermind.Merge.Plugin/SszRest/SszCodec.cs Outdated
Comment thread src/Nethermind/Nethermind.Merge.Plugin/Data/ExecutionPayload.cs Outdated
Comment thread src/Nethermind/Nethermind.Merge.Plugin/SszRest/SszWireTypes.cs
Comment thread src/Nethermind/Nethermind.Merge.Plugin/SszRest/SszExecutionPayload.cs Outdated
Comment thread src/Nethermind/Nethermind.Merge.Plugin/SszRest/SszExecutionPayload.cs Outdated
Comment thread src/Nethermind/Nethermind.Merge.Plugin/SszRest/SszWireTypes.cs Outdated
@LukaszRozmej
Copy link
Copy Markdown
Member

@claude re-review

@claude
Copy link
Copy Markdown
Contributor

claude Bot commented Apr 27, 2026

Claude finished @LukaszRozmej's task in 9m 17s —— View job


PR Re-Review: feat(merge): add SSZ-REST transport for Engine-API

Tasks

  • Load rule files
  • Read changed files
  • Check previous High/Medium findings
  • Analyze new code for correctness, security, performance
  • Post inline comments
  • Write final review summary

What Was Fixed Since the Last Review

All findings from the previous re-review are addressed:

# Issue Status
High 1 PayloadAttributesWire version-unaware (V1/V2/V3 forkchoice) ✅ Fixed — separate PayloadAttributesV1Wire, V2Wire, V3Wire types with correct dispatch in SszCodec.DecodeForkchoiceUpdatedRequest
High 2 ExecutionPayloadV1 SSZ included withdrawals field ✅ Fixed — new ExecutionPayloadV1Ssz/ExecutionPayloadSsz/V3Ssz/V4Ssz wrapper classes in SszExecutionPayload.cs
Medium EncodeGetPayloadV1Response mutating a shared domain object ✅ Fixed — uses ExecutionPayloadV1Ssz.Wrap(ep)
Medium SszCapabilityName 64-byte limit ✅ Fixed — bumped to 128 bytes
Low LINQ .ToArray() on IReadOnlyList ✅ Fixed — collection expressions [.. b.Transactions]
Low Commented-out using in ExecutionPayload.cs ✅ Fixed
Low Raw request path echoed in 404 error body ✅ Fixed

Remaining Issues

Critical

1. Field ordering mismatch in SSZ payload wrappers breaks wire compatibility (inline comment)

ExecutionPayloadV1Ssz and ExecutionPayloadSsz (and by inheritance, V3Ssz, V4Ssz) declare BaseFeePerGas and BlockHash before ExtraData. The SSZ source generator uses property declaration order, so it emits the wrong fixed-section layout.

The Ethereum consensus spec (Bellatrix+) specifies:

...timestamp(8) | offset:extra_data(4) | base_fee_per_gas(32) | block_hash(32) | offset:transactions(4)...

The code produces:

...timestamp(8) | base_fee_per_gas(32) | block_hash(32) | offset:extra_data(4) | offset:transactions(4)...

baseFeePerGas lands at byte 436 in the spec but byte 500 in the code. Any spec-compliant CL client sending a newPayload request will have its baseFeePerGas decoded from the wrong bytes, and every field from extraData onwards will be misread. This affects all versioned payload types V1–V4.

Fix: in both ExecutionPayloadV1Ssz and ExecutionPayloadSsz, move the ExtraData property declaration to appear before BaseFeePerGas and BlockHash.

Medium

2. IJsonRpcProcessor not registered as interface in MS DI (inline comment)

JsonRpcRunner calls s.AddSingleton(_jsonRpcProcessor) where the field is typed JsonRpcProcessor (concrete class). This registers the service under typeof(JsonRpcProcessor). SszMiddleware's constructor requires IJsonRpcProcessor. SszMiddlewareConfigurer does not bridge this interface. When ASP.NET Core builds the middleware pipeline, it will fail to resolve IJsonRpcProcessor and throw at startup. Fix: add s.AddSingleton<IJsonRpcProcessor>(_jsonRpcProcessor) in JsonRpcRunner, or bridge the interface in SszMiddlewareConfigurer.

3. Round-trip tests cannot catch field ordering bugs (inline comment)

Every codec test encodes then decodes the same struct, so they pass even when the wire layout is wrong (because both sides use the same wrong layout). The Critical field ordering bug above was introduced by c4765b0 and is not caught by any existing test. Add at least one test that verifies known fields land at spec-defined byte offsets (e.g. baseFeePerGas at offset 440 for V1).


Summary

The SszExecutionPayload.cs refactor introduced in commit c4765b0 resolved two previous High findings cleanly, but introduced a new Critical field-ordering bug in all four payload wrapper types. No CL client that sends spec-compliant SSZ execution payloads would interoperate correctly until this is fixed. The DI wiring for IJsonRpcProcessor also needs to be verified before the middleware can load in production. Blocking on Critical item 1 and Medium item 2.

Comment thread src/Nethermind/Nethermind.Merge.Plugin/SszRest/SszExecutionPayload.cs Outdated
Comment thread src/Nethermind/Nethermind.Merge.Plugin.Test/SszRest/SszCodecTests.cs Outdated
@github-actions github-actions Bot added the dependencies Pull requests that update a dependency file label May 9, 2026
LukaszRozmej and others added 6 commits May 10, 2026 09:31
…engine, split codec/round-trip

The previous benchmark compared SSZ-through-real-middleware against a hand-rolled
JSON stub that bypassed JsonRpcProcessor and did its own ad-hoc parse-twice
(JsonDocument + per-payload Deserialize) with a static byte[] response. The two
paths weren't measuring equivalent work, which made the SSZ-vs-JSON comparison
misleading.

JSON path now goes through the production RpcModuleProvider + JsonRpcService +
JsonRpcProcessor with the engine module registered as a SingletonModulePool, so
both paths share the same engine call surface and only the transport + codec
differ.

Other changes:
- Replace 130-line StubEngineModule (mostly NotImplementedException placeholders)
  with NSubstitute on IEngineRpcModule.
- Reuse TestItem.KeccakA-E and TestItem.AddressA-B; drop local consts.
- Cache MediaTypeHeaderValue and the parsed Authorization as static fields;
  return int (status code) from round-trip benchmarks instead of HttpResponseMessage.
- Tighten EncodeSszBody (raw byte[], drops ArrayBufferWriter detour) and
  EncodeJsonBody (Utf8JsonWriter + JsonSerializer.Serialize directly into the
  writer; no JsonDocument injector pattern).
- Single BuildEngineHost helper; SSZ and JSON servers diverge only in
  service registration and the terminal middleware.
- Drop StubProcessExitSource from the JSON server (only SszMiddleware needs it).
- Split into 6 benchmarks: deserialize Ssz/Json, serialize PayloadStatus
  Ssz/Json, full round-trip Ssz/Json. [Params(0,1,3,6)] for blob count.
- Add NSubstitute and Nethermind.Core.Test references to the benchmark csproj.

Validated: 24/24 benchmarks pass. Headline (Blobs=6): SSZ decode 2.4us vs JSON
7.2us (3x); SSZ encode 27ns vs JSON 110ns (4x); round-trip SSZ 78us vs JSON
121us (1.5x).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…nchmarks; simplify builders; expand blob params

The class covers more than just an end-to-end round-trip — it also runs
codec-only deserialize and PayloadStatus serialize benchmarks. Rename the file
and class to reflect that. Update the runner's Program.cs reference accordingly.

BuildTransactions: drop the per-tx Random fill — content doesn't affect codec
work or output size, so the random bytes were noise. Just allocate and set the
EIP-1559 type byte.

BuildBlobVersionedHashes: lift the byte buffer out of the loop and use stackalloc
+ Hash256(ReadOnlySpan<byte>) so we don't allocate one byte[32] per blob. With
[Params] now extending up to 72 blobs, the per-iteration allocation is worth
removing.

[Params] expanded from {0, 1, 3, 6} to {0, 1, 3, 6, 12, 24, 36, 72} to surface
how cost scales beyond the spec maximum.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
BDN's default 120s build timeout for the auto-generated boilerplate trips
when the benchmark project's transitive dep graph is large (Merge.Plugin
pulls in JsonRpc, Synchronization, Init, etc., and a cold restore + build
of all of those takes longer than 2 minutes on a regular machine).

Set ManualConfig.BuildTimeout to 10 minutes — covers a clean build, no
effect once the dep tree is warm.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…rawals

The previous defaults (50 × 300 B = 15 KB tx data, 2 withdrawals) were nowhere
near a realistic mainnet block. The codec cost the benchmark measured was
mostly fixed-field overhead, not the variable payload weight that dominates in
production.

Bump the tx-data baseline to 250 × 600 B ≈ 150 KB — matches a typical mainnet
block. The constant is a single edit-point if someone wants to stress-test
worst case (e.g. 2500 × 600 B ≈ 1.5 MB observed on mainnet).

Withdrawals go from 2 to 16 (Capella max). BuildWithdrawals fills them from
TestItem.Addresses to keep the payload realistic.

Note: blob count is the right scaling axis for V3 specifically — the envelope
carries 32-byte versioned hashes only, not blob bytes (which travel in a
sidecar). So 72 blobs adds 2.3 KB to the envelope, not 9 MB.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Host.CreateDefaultBuilder() wires the Microsoft.AspNetCore.Hosting.Diagnostics
logger at Information level by default, which floods stdout with two lines
("Request starting" / "Request finished") per benchmark iteration. Drop all
log providers — BenchmarkDotNet has its own output pipeline and any other
log noise is benchmark-irrelevant.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…; reuse Eip4844Constants

Adds a sibling benchmark targeting engine_getPayloadV3, where the SSZ-vs-JSON
delta is most pronounced — every proposing slot returns ExecutionPayloadV3 +
BlobsBundle (commitments, proofs, ~131 KB blobs). JSON hex-encodes blob bytes,
doubling the wire size; SSZ keeps them as-is.

Layout matches NewPayloadSerializationBenchmarks: SSZ decode wire, JSON parse
JsonDocument (no STJ-constructible target — production CL has its own DOM),
SSZ/JSON encode of the result, full Kestrel round-trip via SszMiddleware /
real JsonRpcProcessor with engine_getPayloadV3 stubbed via NSubstitute.

Block construction uses Build.A.Block with real signed EIP-1559 transactions
(Build.A.Transaction...Signed). Going through ExecutionPayloadV3.TryGetBlock()
on a payload with synthetic byte[] transactions failed RLP decode, so the
benchmark constructs the Block directly.

Reuse existing constants where they exist:
- Eip4844Constants.GasPerBlob (= 131072) for blob byte size and BlobGasUsed.
- Eip4844Constants.BytesPerBlobVersionedHash (= 32) for hash buffer size.
- KzgPolynomialCommitments.KzgBlobHashVersionV1 (= 0x01) for the version byte.

48 (commitment / proof bytes) stays a local const — the canonical source is
Ckzg.BytesPerCommitment / BytesPerProof from CkzgLib, which would require a
new namespace reference for two named constants.

Validated: 48/48 benchmarks pass. At Blobs=72 (worst case): SSZ round-trip
4.1ms / 9.6 MB vs JSON 43.3ms / 115 MB — 10x faster, 12x fewer allocations.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@LukaszRozmej
Copy link
Copy Markdown
Member

Benchmark:


| Method                                        | Blobs | Mean          | Error        | StdDev        | Median        | Ratio  | RatioSD | Gen0   | Gen1   | Gen2   | Allocated | Alloc Ratio |
|---------------------------------------------- |------ |--------------:|-------------:|--------------:|--------------:|-------:|--------:|-------:|-------:|-------:|----------:|------------:|
| 'SSZ  decode NewPayloadV3 wire'               | 0     |  20,209.96 ns |   527.721 ns |  1,555.996 ns |  20,140.10 ns |  0.426 |    0.03 | 4.4250 | 1.2207 |      - |  165696 B |       0.997 |
| 'JSON decode ExecutionPayloadV3'              | 0     |  47,444.58 ns |   940.759 ns |  1,255.887 ns |  47,250.27 ns |  1.001 |    0.04 | 5.1270 | 0.9155 |      - |  166152 B |       1.000 |
| 'SSZ  encode PayloadStatus'                   | 0     |      27.93 ns |     0.584 ns |      0.759 ns |      27.79 ns |  0.001 |    0.00 | 0.0032 |      - |      - |     120 B |       0.001 |
| 'JSON encode PayloadStatus'                   | 0     |     124.69 ns |     2.505 ns |      2.343 ns |     124.61 ns |  0.003 |    0.00 | 0.0021 |      - |      - |      88 B |       0.001 |
| 'SSZ  NewPayloadV3 (full Kestrel round-trip)' | 0     | 146,742.29 ns | 2,813.201 ns |  2,631.470 ns | 146,979.64 ns |  3.095 |    0.10 |      - |      - |      - |  182791 B |       1.100 |
| 'JSON NewPayloadV3 (full Kestrel round-trip)' | 0     | 701,215.89 ns | 9,005.652 ns |  8,423.892 ns | 699,341.21 ns | 14.790 |    0.42 |      - |      - |      - |  199722 B |       1.202 |
|                                               |       |               |              |               |               |        |         |        |        |        |           |             |
| 'SSZ  decode NewPayloadV3 wire'               | 1     |  20,001.20 ns |   397.143 ns |    811.258 ns |  19,655.09 ns |  0.435 |    0.02 | 4.4556 | 1.1597 |      - |  165752 B |          NA |
| 'JSON decode ExecutionPayloadV3'              | 1     |  46,026.04 ns |   897.622 ns |  1,474.819 ns |  45,707.96 ns |  1.001 |    0.04 | 4.7607 | 0.7324 | 0.0610 |         - |          NA |
| 'SSZ  encode PayloadStatus'                   | 1     |      28.24 ns |     0.496 ns |      0.439 ns |      28.17 ns |  0.001 |    0.00 | 0.0040 |      - |      - |     120 B |          NA |
| 'JSON encode PayloadStatus'                   | 1     |     131.58 ns |     2.663 ns |      4.593 ns |     131.04 ns |  0.003 |    0.00 | 0.0021 |      - |      - |      88 B |          NA |
| 'SSZ  NewPayloadV3 (full Kestrel round-trip)' | 1     | 150,573.03 ns | 7,284.416 ns | 18,672.745 ns | 154,274.51 ns |  3.275 |    0.42 | 0.4883 |      - |      - |  182970 B |          NA |
| 'JSON NewPayloadV3 (full Kestrel round-trip)' | 1     | 686,438.68 ns | 8,432.350 ns |  7,887.625 ns | 686,295.31 ns | 14.929 |    0.49 |      - |      - |      - |  199866 B |          NA |
|                                               |       |               |              |               |               |        |         |        |        |        |           |             |
| 'SSZ  decode NewPayloadV3 wire'               | 3     |  19,449.61 ns |   517.738 ns |  1,518.436 ns |  18,860.06 ns |  0.429 |    0.04 | 4.4250 | 1.1597 |      - |  165864 B |       0.998 |
| 'JSON decode ExecutionPayloadV3'              | 3     |  45,439.89 ns |   904.142 ns |  1,583.532 ns |  45,775.19 ns |  1.001 |    0.05 | 5.3101 | 0.9155 |      - |  166152 B |       1.000 |
| 'SSZ  encode PayloadStatus'                   | 3     |      27.50 ns |     0.570 ns |      0.920 ns |      27.20 ns |  0.001 |    0.00 | 0.0032 |      - |      - |     120 B |       0.001 |
| 'JSON encode PayloadStatus'                   | 3     |     122.44 ns |     2.282 ns |      2.023 ns |     122.73 ns |  0.003 |    0.00 | 0.0021 |      - |      - |      88 B |       0.001 |
| 'SSZ  NewPayloadV3 (full Kestrel round-trip)' | 3     | 139,255.44 ns | 1,065.680 ns |    832.013 ns | 139,286.18 ns |  3.068 |    0.11 |      - |      - |      - |  182448 B |       1.098 |
| 'JSON NewPayloadV3 (full Kestrel round-trip)' | 3     | 679,700.57 ns | 7,468.966 ns |  6,621.043 ns | 678,033.98 ns | 14.976 |    0.54 |      - |      - |      - |  199992 B |       1.204 |
|                                               |       |               |              |               |               |        |         |        |        |        |           |             |
| 'SSZ  decode NewPayloadV3 wire'               | 6     |  17,420.09 ns |   346.002 ns |    437.582 ns |  17,430.84 ns |  0.396 |    0.01 | 5.3406 | 1.4648 |      - |  166032 B |       0.999 |
| 'JSON decode ExecutionPayloadV3'              | 6     |  44,062.61 ns |   873.180 ns |    896.691 ns |  44,130.65 ns |  1.000 |    0.03 | 4.4556 | 0.5493 |      - |  166152 B |       1.000 |
| 'SSZ  encode PayloadStatus'                   | 6     |      27.20 ns |     0.559 ns |      0.837 ns |      27.19 ns |  0.001 |    0.00 | 0.0052 |      - |      - |     120 B |       0.001 |
| 'JSON encode PayloadStatus'                   | 6     |     120.22 ns |     2.406 ns |      2.363 ns |     120.18 ns |  0.003 |    0.00 | 0.0021 |      - |      - |      88 B |       0.001 |
| 'SSZ  NewPayloadV3 (full Kestrel round-trip)' | 6     | 146,452.60 ns | 2,239.987 ns |  1,985.690 ns | 146,081.64 ns |  3.325 |    0.08 | 0.2441 |      - |      - |  182811 B |       1.100 |
| 'JSON NewPayloadV3 (full Kestrel round-trip)' | 6     | 678,194.02 ns | 7,101.521 ns |  6,642.767 ns | 676,712.70 ns | 15.398 |    0.34 |      - |      - |      - |  200328 B |       1.206 |
|                                               |       |               |              |               |               |        |         |        |        |        |           |             |
| 'SSZ  decode NewPayloadV3 wire'               | 12    |  18,921.77 ns |   394.710 ns |  1,157.616 ns |  18,593.89 ns |  0.423 |    0.03 | 4.4556 | 1.3428 |      - |  166368 B |       1.001 |
| 'JSON decode ExecutionPayloadV3'              | 12    |  44,777.56 ns |   881.709 ns |  1,292.397 ns |  44,622.67 ns |  1.001 |    0.04 | 5.1270 | 0.8545 |      - |  166152 B |       1.000 |
| 'SSZ  encode PayloadStatus'                   | 12    |      28.17 ns |     0.585 ns |      1.112 ns |      27.93 ns |  0.001 |    0.00 | 0.0057 | 0.0001 | 0.0001 |     115 B |       0.001 |
| 'JSON encode PayloadStatus'                   | 12    |     118.26 ns |     1.675 ns |      1.399 ns |     118.32 ns |  0.003 |    0.00 | 0.0021 |      - |      - |      88 B |       0.001 |
| 'SSZ  NewPayloadV3 (full Kestrel round-trip)' | 12    | 140,015.48 ns | 1,640.725 ns |  1,454.460 ns | 140,346.02 ns |  3.129 |    0.09 |      - |      - |      - |  183473 B |       1.104 |
| 'JSON NewPayloadV3 (full Kestrel round-trip)' | 12    | 691,192.64 ns | 6,514.374 ns |  6,093.549 ns | 691,866.21 ns | 15.448 |    0.45 |      - |      - |      - |  200811 B |       1.209 |
|                                               |       |               |              |               |               |        |         |        |        |        |           |             |
| 'SSZ  decode NewPayloadV3 wire'               | 24    |  18,839.80 ns |   354.937 ns |    716.991 ns |  18,698.85 ns |  0.418 |    0.02 | 4.4861 | 1.1902 |      - |  167040 B |       1.005 |
| 'JSON decode ExecutionPayloadV3'              | 24    |  45,110.37 ns |   890.998 ns |    914.989 ns |  45,062.18 ns |  1.000 |    0.03 | 4.7607 | 0.7324 |      - |  166152 B |       1.000 |
| 'SSZ  encode PayloadStatus'                   | 24    |      27.97 ns |     0.566 ns |      0.556 ns |      27.89 ns |  0.001 |    0.00 | 0.0040 |      - |      - |     120 B |       0.001 |
| 'JSON encode PayloadStatus'                   | 24    |     147.37 ns |     2.328 ns |      2.178 ns |     147.92 ns |  0.003 |    0.00 | 0.0029 | 0.0002 | 0.0002 |         - |       0.000 |
| 'SSZ  NewPayloadV3 (full Kestrel round-trip)' | 24    | 143,294.11 ns | 2,068.221 ns |  1,727.058 ns | 142,230.27 ns |  3.178 |    0.07 |      - |      - |      - |  184914 B |       1.113 |
| 'JSON NewPayloadV3 (full Kestrel round-trip)' | 24    | 691,187.55 ns | 4,419.173 ns |  4,133.697 ns | 691,629.49 ns | 15.328 |    0.31 |      - |      - |      - |  201856 B |       1.215 |
|                                               |       |               |              |               |               |        |         |        |        |        |           |             |
| 'SSZ  decode NewPayloadV3 wire'               | 36    |  19,946.22 ns |   387.241 ns |  1,111.068 ns |  19,619.72 ns |  0.428 |    0.03 | 4.4861 | 1.3428 |      - |  167712 B |       1.009 |
| 'JSON decode ExecutionPayloadV3'              | 36    |  46,587.79 ns |   888.374 ns |  1,057.546 ns |  46,802.19 ns |  1.000 |    0.03 | 5.1270 | 0.9766 |      - |  166152 B |       1.000 |
| 'SSZ  encode PayloadStatus'                   | 36    |      32.40 ns |     0.678 ns |      0.781 ns |      32.39 ns |  0.001 |    0.00 | 0.0033 | 0.0001 | 0.0001 |     536 B |       0.003 |
| 'JSON encode PayloadStatus'                   | 36    |     136.66 ns |     2.761 ns |      4.688 ns |     134.75 ns |  0.003 |    0.00 | 0.0021 |      - |      - |      88 B |       0.001 |
| 'SSZ  NewPayloadV3 (full Kestrel round-trip)' | 36    | 140,996.84 ns | 2,727.191 ns |  2,551.016 ns | 140,316.36 ns |  3.028 |    0.09 |      - |      - |      - |  186432 B |       1.122 |
| 'JSON NewPayloadV3 (full Kestrel round-trip)' | 36    | 679,566.41 ns | 4,013.623 ns |  3,351.557 ns | 679,414.45 ns | 14.594 |    0.34 |      - |      - |      - |  203216 B |       1.223 |
|                                               |       |               |              |               |               |        |         |        |        |        |           |             |
| 'SSZ  decode NewPayloadV3 wire'               | 72    |  19,009.60 ns |   368.665 ns |    868.986 ns |  18,782.61 ns |  0.427 |    0.02 | 4.5471 | 1.3428 |      - |  169728 B |       1.022 |
| 'JSON decode ExecutionPayloadV3'              | 72    |  44,585.64 ns |   883.928 ns |  1,117.886 ns |  44,345.06 ns |  1.001 |    0.03 | 4.4556 | 0.6714 |      - |  166152 B |       1.000 |
| 'SSZ  encode PayloadStatus'                   | 72    |      27.57 ns |     0.566 ns |      0.652 ns |      27.51 ns |  0.001 |    0.00 | 0.0043 |      - |      - |     120 B |       0.001 |
| 'JSON encode PayloadStatus'                   | 72    |     118.53 ns |     2.295 ns |      2.356 ns |     118.62 ns |  0.003 |    0.00 | 0.0023 |      - |      - |      88 B |       0.001 |
| 'SSZ  NewPayloadV3 (full Kestrel round-trip)' | 72    | 143,476.55 ns | 2,576.366 ns |  2,409.935 ns | 142,704.69 ns |  3.220 |    0.09 |      - |      - |      - |  190675 B |       1.148 |
| 'JSON NewPayloadV3 (full Kestrel round-trip)' | 72    | 710,506.85 ns | 7,159.184 ns |  6,346.430 ns | 709,196.97 ns | 15.945 |    0.41 |      - |      - |      - |  206704 B |       1.244 |

@LukaszRozmej
Copy link
Copy Markdown
Member

| Method                                        | Blobs | Mean         | Error        | StdDev       | Median       | Ratio  | RatioSD | Gen0     | Gen1     | Gen2     | Allocated   | Alloc Ratio  |
|---------------------------------------------- |------ |-------------:|-------------:|-------------:|-------------:|-------:|--------:|---------:|---------:|---------:|------------:|-------------:|
| 'SSZ  decode GetPayloadV3 wire'               | 0     |     19.57 us |     0.391 us |     1.129 us |     19.43 us |   1.81 |    0.11 |   4.0588 |   0.9155 |        - |    159648 B |     2,217.33 |
| 'JSON parse GetPayloadV3 (JsonDocument)'      | 0     |     10.82 us |     0.147 us |     0.130 us |     10.82 us |   1.00 |    0.02 |        - |        - |        - |        72 B |         1.00 |
| 'SSZ  encode GetPayloadV3 result'             | 0     |     89.49 us |     2.189 us |     6.173 us |     88.61 us |   8.27 |    0.58 |   4.6387 |   4.6387 |   4.6387 |    148406 B |     2,061.19 |
| 'JSON encode GetPayloadV3Result'              | 0     |    280.22 us |     6.120 us |    18.045 us |    280.32 us |  25.90 |    1.69 |   9.7656 |   9.7656 |   9.7656 |    291494 B |     4,048.53 |
| 'SSZ  GetPayloadV3 (full Kestrel round-trip)' | 0     |    170.00 us |     3.284 us |     8.765 us |    170.58 us |  15.71 |    0.83 |   2.9297 |   2.9297 |   2.9297 |    159906 B |     2,220.92 |
| 'JSON GetPayloadV3 (full Kestrel round-trip)' | 0     |  1,135.51 us |    21.233 us |    52.084 us |  1,132.16 us | 104.95 |    4.93 |  35.1563 |  35.1563 |  35.1563 |   1488379 B |    20,671.93 |
|                                               |       |              |              |              |              |        |         |          |          |          |             |              |
| 'SSZ  decode GetPayloadV3 wire'               | 1     |    130.38 us |     1.793 us |     1.677 us |    130.79 us |   8.32 |    0.16 |   8.0566 |   4.6387 |   4.1504 |    290923 B |     4,040.60 |
| 'JSON parse GetPayloadV3 (JsonDocument)'      | 1     |     15.67 us |     0.261 us |     0.231 us |     15.63 us |   1.00 |    0.02 |        - |        - |        - |        72 B |         1.00 |
| 'SSZ  encode GetPayloadV3 result'             | 1     |    183.04 us |     8.166 us |    24.079 us |    176.13 us |  11.68 |    1.54 |   8.0566 |   8.0566 |   8.0566 |    279703 B |     3,884.76 |
| 'JSON encode GetPayloadV3Result'              | 1     |    473.86 us |     2.785 us |     2.605 us |    473.21 us |  30.24 |    0.46 |  15.1367 |  15.1367 |  15.1367 |    553922 B |     7,693.36 |
| 'SSZ  GetPayloadV3 (full Kestrel round-trip)' | 1     |    251.26 us |     7.789 us |    22.966 us |    247.70 us |  16.03 |    1.48 |   4.3945 |   4.3945 |   4.3945 |    291360 B |     4,046.67 |
| 'JSON GetPayloadV3 (full Kestrel round-trip)' | 1     |  2,007.58 us |    38.488 us |    50.045 us |  1,998.18 us | 128.12 |    3.62 |  62.5000 |  62.5000 |  62.5000 |   2791533 B |    38,771.29 |
|                                               |       |              |              |              |              |        |         |          |          |          |             |              |
| 'SSZ  decode GetPayloadV3 wire'               | 3     |    242.98 us |     4.837 us |    13.879 us |    242.28 us |   9.99 |    0.58 |  16.1133 |  13.1836 |  12.9395 |    553376 B |     7,685.78 |
| 'JSON parse GetPayloadV3 (JsonDocument)'      | 3     |     24.32 us |     0.355 us |     0.332 us |     24.40 us |   1.00 |    0.02 |        - |        - |        - |        72 B |         1.00 |
| 'SSZ  encode GetPayloadV3 result'             | 3     |    389.59 us |     8.828 us |    26.031 us |    390.67 us |  16.02 |    1.09 |  14.1602 |  14.1602 |  14.1602 |    542283 B |     7,531.71 |
| 'JSON encode GetPayloadV3Result'              | 3     |    949.84 us |     7.626 us |     7.134 us |    951.65 us |  39.06 |    0.59 |  29.2969 |  29.2969 |  29.2969 |   1078735 B |    14,982.43 |
| 'SSZ  GetPayloadV3 (full Kestrel round-trip)' | 3     |    371.32 us |     8.040 us |    23.454 us |    370.03 us |  15.27 |    0.98 |   9.7656 |   9.7656 |   9.7656 |    554241 B |     7,697.79 |
| 'JSON GetPayloadV3 (full Kestrel round-trip)' | 3     |  3,680.51 us |    73.148 us |   212.216 us |  3,686.32 us | 151.36 |    8.92 | 101.5625 | 101.5625 | 101.5625 |   5447633 B |    75,661.57 |
|                                               |       |              |              |              |              |        |         |          |          |          |             |              |
| 'SSZ  decode GetPayloadV3 wire'               | 6     |    385.00 us |     7.651 us |     9.949 us |    387.24 us |  10.61 |    0.29 |  28.8086 |  25.3906 |  24.9023 |    947082 B |    13,153.92 |
| 'JSON parse GetPayloadV3 (JsonDocument)'      | 6     |     36.29 us |     0.429 us |     0.401 us |     36.21 us |   1.00 |    0.02 |        - |        - |        - |        72 B |         1.00 |
| 'SSZ  encode GetPayloadV3 result'             | 6     |    611.03 us |    11.567 us |    11.360 us |    611.49 us |  16.84 |    0.35 |  19.5313 |  19.5313 |  19.5313 |    936138 B |    13,001.92 |
| 'JSON encode GetPayloadV3Result'              | 6     |  1,359.62 us |    27.125 us |    45.319 us |  1,364.30 us |  37.47 |    1.30 |  31.2500 |  31.2500 |  31.2500 |   1865718 B |    25,912.75 |
| 'SSZ  GetPayloadV3 (full Kestrel round-trip)' | 6     |    574.81 us |    11.042 us |    15.115 us |    574.85 us |  15.84 |    0.44 |  16.6016 |  16.6016 |  16.6016 |    948216 B |    13,169.67 |
| 'JSON GetPayloadV3 (full Kestrel round-trip)' | 6     |  5,977.10 us |   118.963 us |   243.010 us |  6,013.71 us | 164.72 |    6.86 | 140.6250 | 140.6250 | 140.6250 |   9351819 B |   129,886.38 |
|                                               |       |              |              |              |              |        |         |          |          |          |             |              |
| 'SSZ  decode GetPayloadV3 wire'               | 12    |    661.64 us |    15.015 us |    44.272 us |    664.77 us |  10.58 |    0.75 |  53.7109 |  49.8047 |  49.8047 |   1734425 B |    24,089.24 |
| 'JSON parse GetPayloadV3 (JsonDocument)'      | 12    |     62.56 us |     1.201 us |     1.644 us |     62.14 us |   1.00 |    0.04 |        - |        - |        - |        72 B |         1.00 |
| 'SSZ  encode GetPayloadV3 result'             | 12    |    688.64 us |    13.362 us |    12.499 us |    682.18 us |  11.01 |    0.34 |  21.4844 |  21.4844 |  21.4844 |   1723755 B |    23,941.04 |
| 'JSON encode GetPayloadV3Result'              | 12    |  2,060.81 us |    40.994 us |    75.986 us |  2,040.28 us |  32.96 |    1.46 |  31.2500 |  31.2500 |  31.2500 |   3439740 B |    47,774.17 |
| 'SSZ  GetPayloadV3 (full Kestrel round-trip)' | 12    |    956.32 us |    13.949 us |    12.365 us |    957.51 us |  15.30 |    0.43 |  19.5313 |  19.5313 |  19.5313 |   1736205 B |    24,113.96 |
| 'JSON GetPayloadV3 (full Kestrel round-trip)' | 12    |  8,910.52 us |   143.131 us |   140.573 us |  8,887.19 us | 142.52 |    4.21 | 140.6250 | 140.6250 | 140.6250 |  17223641 B |   239,217.24 |
|                                               |       |              |              |              |              |        |         |          |          |          |             |              |
| 'SSZ  decode GetPayloadV3 wire'               | 24    |  1,120.70 us |    21.947 us |    30.041 us |  1,112.02 us |   9.58 |    0.43 |  74.2188 |  70.3125 |  70.3125 |   3308878 B |    45,956.64 |
| 'JSON parse GetPayloadV3 (JsonDocument)'      | 24    |    117.14 us |     2.292 us |     4.415 us |    116.15 us |   1.00 |    0.05 |        - |        - |        - |        72 B |         1.00 |
| 'SSZ  encode GetPayloadV3 result'             | 24    |  1,472.54 us |    10.429 us |     9.245 us |  1,475.15 us |  12.59 |    0.46 |  31.2500 |  31.2500 |  31.2500 |   3299069 B |    45,820.40 |
| 'JSON encode GetPayloadV3Result'              | 24    |  4,133.95 us |    81.832 us |   234.793 us |  4,081.89 us |  35.34 |    2.38 |  54.6875 |  54.6875 |  54.6875 |   6588201 B |    91,502.79 |
| 'SSZ  GetPayloadV3 (full Kestrel round-trip)' | 24    |  1,637.66 us |    33.765 us |    99.556 us |  1,631.15 us |  14.00 |    0.99 |  29.2969 |  29.2969 |  29.2969 |   3312314 B |    46,004.36 |
| 'JSON GetPayloadV3 (full Kestrel round-trip)' | 24    | 16,626.01 us |   330.034 us |   308.714 us | 16,640.73 us | 142.12 |    5.77 | 187.5000 | 187.5000 | 187.5000 |  32967970 B |   457,888.47 |
|                                               |       |              |              |              |              |        |         |          |          |          |             |              |
| 'SSZ  decode GetPayloadV3 wire'               | 36    |  1,926.67 us |    62.666 us |   184.771 us |  1,980.54 us |  11.37 |    1.10 | 136.7188 | 132.8125 | 132.8125 |   4883464 B |    67,825.89 |
| 'JSON parse GetPayloadV3 (JsonDocument)'      | 36    |    169.44 us |     3.238 us |     3.029 us |    169.02 us |   1.00 |    0.02 |        - |        - |        - |        72 B |         1.00 |
| 'SSZ  encode GetPayloadV3 result'             | 36    |  2,081.11 us |    46.186 us |   132.515 us |  2,049.82 us |  12.29 |    0.81 |  62.5000 |  62.5000 |  62.5000 |   4874409 B |    67,700.12 |
| 'JSON encode GetPayloadV3Result'              | 36    |  6,492.88 us |   129.813 us |   177.689 us |  6,528.20 us |  38.33 |    1.22 |  62.5000 |  62.5000 |  62.5000 |   9736194 B |   135,224.92 |
| 'SSZ  GetPayloadV3 (full Kestrel round-trip)' | 36    |  2,600.83 us |    75.013 us |   221.178 us |  2,682.61 us |  15.35 |    1.33 |  58.5938 |  58.5938 |  58.5938 |   4888602 B |    67,897.25 |
| 'JSON GetPayloadV3 (full Kestrel round-trip)' | 36    | 21,554.12 us |   427.512 us |   626.641 us | 21,631.02 us | 127.25 |    4.24 | 156.2500 | 156.2500 | 156.2500 |  48711867 B |   676,553.71 |
|                                               |       |              |              |              |              |        |         |          |          |          |             |              |
| 'SSZ  decode GetPayloadV3 wire'               | 72    |  3,682.43 us |    81.978 us |   236.526 us |  3,676.16 us |   6.67 |    0.51 | 207.0313 | 203.1250 | 203.1250 |   9607158 B |   133,432.75 |
| 'JSON parse GetPayloadV3 (JsonDocument)'      | 72    |    553.07 us |    11.019 us |    24.416 us |    548.02 us |   1.00 |    0.06 |        - |        - |        - |        72 B |         1.00 |
| 'SSZ  encode GetPayloadV3 result'             | 72    |  3,843.33 us |    66.855 us |    62.537 us |  3,809.48 us |   6.96 |    0.32 |  58.5938 |  58.5938 |  58.5938 |   9600432 B |   133,339.33 |
| 'JSON encode GetPayloadV3Result'              | 72    | 12,624.50 us |   251.244 us |   376.051 us | 12,620.12 us |  22.87 |    1.18 |  62.5000 |  62.5000 |  62.5000 |  19180949 B |   266,402.07 |
| 'SSZ  GetPayloadV3 (full Kestrel round-trip)' | 72    |  4,869.15 us |    91.389 us |    85.486 us |  4,874.63 us |   8.82 |    0.40 |  62.5000 |  62.5000 |  62.5000 |   9616550 B |   133,563.19 |
| 'JSON GetPayloadV3 (full Kestrel round-trip)' | 72    | 50,112.28 us | 1,696.819 us | 4,949.701 us | 49,933.84 us |  90.78 |    9.73 | 181.8182 | 181.8182 | 181.8182 | 108188398 B | 1,502,616.64 |

Copy link
Copy Markdown
Contributor

@rubo rubo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@flcl42 Could you please review the SSZ changes?

@benaadams
Copy link
Copy Markdown
Member

@claude confirm or refute these issues; comment on the code with fixes if found to be true

  1. Consensus correctness: empty withdrawals become “absent” src/Nethermind/Nethermind.Merge.Plugin/SszRest/SszExecutionPayload.cs:278 SszExecutionPayload.Withdrawals sets _inner.Withdrawals = null when the decoded SSZ list is empty. For post-Shanghai payloads, an empty withdrawals list is still present and must not be collapsed to pre-Shanghai shape. This can make valid SSZ newPayload requests fail version validation or build a block with no withdrawals root. Preserve [] as an empty array; only V1 should omit the field.

  2. Resource leak / DoS: SSZ result wrappers are never disposed src/Nethermind/Nethermind.Merge.Plugin/SszRest/Handlers/SszEndpointHandlerBase.cs:71 WriteSszResultAsync writes ResultWrapper data but never disposes the wrapper. That matters because ResultWrapper.Dispose() disposes disposable data, and engine_getBlobsV1 can return BlobsV1DirectResponse, which owns an ArrayPoolList. Repeated authenticated SSZ blob requests can keep pooled arrays alive and create memory pressure. Wrap the result in using or try/finally after encoding completes.

  3. Input validation vulnerability: malformed fixed-size SSZ lists can be accepted
    src/Nethermind/Nethermind.Serialization.SszGenerator/SszGenerator.cs:531 The generated decode path for List-style ref basic types computes count as slice.Length / itemSize and does not reject remainder bytes. A body with 32n + 1 bytes is silently truncated instead of rejected as non-canonical SSZ. Add a multiple-of-item-size validation before decoding fixed-size ref-type lists. This affects Engine request lists such as blob versioned hashes and block hashes.

  4. Unexpected Engine HTTP exposure on WS-only authenticated ports src/Nethermind/Nethermind.Merge.Plugin/SszRest/SszMiddleware.cs:121

The SSZ middleware checks url.IsAuthenticated but not url.RpcEndpoint.HasFlag(RpcEndpoint.Http). JSON-RPC startup has this HTTP endpoint guard, so SSZ REST can be served on authenticated ports configured as WebSocket-only. Require the HTTP flag, and ideally the Engine module flag, before handling SSZ REST requests.

  1. Startup race in capabilities cache
    src/Nethermind/Nethermind.Merge.Plugin/Handlers/EngineRpcCapabilitiesProvider.cs:54 EnsureBuilt() uses _jsonRpc as the initialized sentinel, but _ssz is assigned after it. A concurrent first call can observe _jsonRpc != null, skip build, then dereference _ssz! while it is still null. Publish a single immutable holder atomically or initialize under a lock.

@claude
Copy link
Copy Markdown
Contributor

claude Bot commented May 10, 2026

Claude finished @benaadams's task in 3m 29s —— View job


Analyzing @benaadams's findings — all five verified against the current HEAD.

Tasks

  • Read relevant files and analyze all five issues
  • Issue 1: Empty withdrawals → null (SszExecutionPayload.cs:278)
  • Issue 2: ResultWrapper never disposed (SszEndpointHandlerBase.cs:71)
  • Issue 3: Fixed-size SSZ list truncation (SszGenerator.cs:531)
  • Issue 4: WS-only port SSZ exposure (SszMiddleware.cs:121)
  • Issue 5: Startup race in capabilities cache (EngineRpcCapabilitiesProvider.cs:54)

Issue 1 — ✅ Confirmed: Empty withdrawals silently collapsed to null

SszExecutionPayload.cs:278

set
{
    field = value;
    if (value is null || value.Length == 0)   // ← collapses [] and null to the same thing
    {
        _inner.Withdrawals = null;
        return;
    }
    ...
}

A V2+ newPayload body with an empty withdrawals list (valid for blocks with no withdrawals post-Shanghai) will decode into a wire struct where SszWithdrawals is an empty array. The setter then sets _inner.Withdrawals = null. When the domain object reaches IExecutionPayloadParams.ValidateParams, the engine checks if (ExecutionRequests is null) → Fail and, more directly, a null Withdrawals on a V2 execution payload causes engine_newPayloadV2 to reject the block (the spec requires withdrawals to be present, even empty). Valid blocks with zero withdrawals will be incorrectly rejected on the SSZ path.

Fix: distinguish null (V1 — field absent) from [] (V2+ — field present, zero items):

set
{
    field = value;
    if (value is null)
    {
        _inner.Withdrawals = null;
        return;
    }
    Withdrawal[] result = new Withdrawal[value.Length];
    for (int i = 0; i < result.Length; i++)
        result[i] = new Withdrawal { ... };
    _inner.Withdrawals = result;
}

Fix this →


Issue 2 — ✅ Confirmed: ResultWrapper carrying BlobsV1DirectResponse is never disposed

SszEndpointHandlerBase.cs:71

protected static Task WriteSszResultAsync<T>(HttpContext ctx, ResultWrapper<T> result, ...) =>
    result switch
    {
        { Result.ResultType: not ResultType.Success } => WriteErrorAsync(...),
        { Data: null } => SetNoContent(ctx),
        { Data: var data } => WriteSszAsync(ctx, data, encode)
        // ← result is never disposed
    };

ResultWrapper<T>.Dispose() (confirmed in ResultWrapper.cs:60) calls Dispose() on Data if it implements IDisposable. engine_getBlobsV1 returns ResultWrapper<BlobsV1DirectResponse>, and BlobsV1DirectResponse wraps an ArrayPoolList<BlobAndProofV1?> that owns rented array segments. Since WriteSszResultAsync never calls result.Dispose(), every authenticated SSZ POST /engine/v1/blobs call leaks the ArrayPoolList allocation — the pool arrays are eventually GC-collected but defeat the pool's purpose and create steady GC pressure at blob-heavy load.

Fix: add a using wrapper:

protected static async Task WriteSszResultAsync<T>(HttpContext ctx, ResultWrapper<T> result, Func<T, IBufferWriter<byte>, int> encode)
{
    using (result)
    {
        await (result switch
        {
            { Result.ResultType: not ResultType.Success } => WriteErrorAsync(ctx, ErrorCodeToHttpStatus(result.ErrorCode), result.Result.Error ?? "Unknown error"),
            { Data: null } => SetNoContent(ctx),
            { Data: var data } => WriteSszAsync(ctx, data, encode)
        });
    }
}

Fix this →


Issue 3 — ✅ Confirmed: Fixed-size ref-type list decode silently truncates non-aligned input

SszGenerator.cs:539

string countExpr = property.Kind == Kind.Vector
    ? property.Length!.Value.ToString()
    : $"{sliceExpression}.Length / {itemSize}";     // ← integer division, no remainder check

For any [SszList(N)] Hash256[]? or [SszList(N)] Address[]? property, the generated decoder computes count = slice.Length / 32. A body with 32n + 1 bytes decodes n items and silently discards the trailing byte — non-canonical SSZ is accepted. The subsequent ValidateSszListLimit check (line 540) validates the upper bound but not alignment. This affects GetBlobsRequestWire.VersionedHashes, GetPayloadBodiesByHashRequestWire.BlockHashes, and any other ref-basic-type list in Engine API wire types.

Fix: emit a remainder check before computing count:

// Generated code should be:
if ({sliceExpression}.Length % {itemSize} != 0)
    throw new InvalidDataException($"Expected multiple of {itemSize} bytes for {typeName}.{fieldName}");
int __count = {sliceExpression}.Length / {itemSize};

In SszGenerator.cs, change line 541 to prepend the guard before the loop:

string guard = property.Kind == Kind.List
    ? $"if ({sliceExpression}.Length % {itemSize} != 0) throw new System.IO.InvalidDataException($\"{decl.Name}.{property.Name}: expected multiple of {itemSize} bytes, got {{{sliceExpression}.Length}}\");"
    : string.Empty;
string loop = $"{{ {guard} int __count = {countExpr}; ...";

Fix this →


Issue 4 — ✅ Confirmed: SSZ middleware ignores the HTTP endpoint flag

SszMiddleware.cs:121

else if (!_urlCollection.TryGetValue(ctx.Connection.LocalPort, out JsonRpcUrl? url) || !url.IsAuthenticated)
{
    await _next(ctx);
}

The JSON-RPC pipeline at Startup.cs:248 explicitly guards:

if (!jsonRpcUrlCollection.TryGetValue(ctx.Connection.LocalPort, out JsonRpcUrl jsonRpcUrl) ||
    !jsonRpcUrl.RpcEndpoint.HasFlag(RpcEndpoint.Http))          // ← present in JSON-RPC

The SSZ middleware omits the RpcEndpoint.Http flag check. An authenticated port configured as WebSocket-only (RpcEndpoint.Ws) would pass the IsAuthenticated gate and the middleware would serve SSZ-REST requests that should only reach the WS pipeline. Real Engine API configurations typically use separate HTTP and WS ports, so practical impact is low — but the inconsistency is a bug.

Fix:

else if (!_urlCollection.TryGetValue(ctx.Connection.LocalPort, out JsonRpcUrl? url)
         || !url.IsAuthenticated
         || !url.RpcEndpoint.HasFlag(RpcEndpoint.Http))     // ← add this
{
    await _next(ctx);
}

Fix this →


Issue 5 — ✅ Confirmed: Publication race — _ssz can be null when _jsonRpc is visible

EngineRpcCapabilitiesProvider.cs:52-57

private void EnsureBuilt()
{
    if (_jsonRpc is not null) return;
    Build(..., out Dictionary<...> json, out Dictionary<...> ssz);
    _jsonRpc = json.ToFrozenDictionary();   // ← sentinel written first
    _ssz   = ssz.ToFrozenDictionary();      // ← _ssz may not be visible before next reader
}

Without memory barriers, a CPU or JIT is free to reorder these writes. More concretely: Thread A writes _jsonRpc, then begins writing _ssz. Thread B enters EnsureBuilt(), observes _jsonRpc != null, and immediately returns. Thread B then calls GetEngineCapabilities() and reaches _ssz!.Count — which is still null. This is a null-reference exception on the authenticated engine-API path.

The comment in the class says "Build is pure and the last assignment wins" — that reasoning is correct for the case where both threads run Build, but it ignores the partial-publish case where only _jsonRpc has been stored.

Fix — publish _ssz first, then use Volatile.Write on _jsonRpc to enforce ordering:

private void EnsureBuilt()
{
    if (_jsonRpc is not null) return;
    Build(specProvider.GetFinalSpec(), out Dictionary<string, RpcCapabilityOptions> json, out Dictionary<string, RpcCapabilityOptions> ssz);
    _ssz   = ssz.ToFrozenDictionary();                    // assigned before sentinel
    Volatile.Write(ref _jsonRpc, json.ToFrozenDictionary()); // sentinel: guarantees _ssz visible
}

The corresponding read of _jsonRpc in the guard should use Volatile.Read (or the field can be volatile):

if (Volatile.Read(ref _jsonRpc) is not null) return;

Fix this →


Summary

# Issue Severity Verdict
1 Empty withdrawals → null: valid V2+ blocks rejected on SSZ path High ✅ Confirmed
2 ResultWrapper<BlobsV1DirectResponse> never disposed: ArrayPoolList leaked Medium ✅ Confirmed
3 Ref-basic-type SSZ list: non-aligned bytes silently truncated Medium ✅ Confirmed
4 SSZ middleware ignores RpcEndpoint.Http flag: serves WS-only ports Low ✅ Confirmed
5 EnsureBuilt() race: _ssz readable as null after _jsonRpc published Medium ✅ Confirmed

All five are real bugs. Issue 1 is the most impactful — it is a consensus correctness regression that rejects valid post-Shanghai blocks with zero withdrawals on the SSZ path. Issues 2, 3, and 5 are resource-management / thread-safety defects. Issue 4 is a configuration-boundary inconsistency.

@benaadams
Copy link
Copy Markdown
Member

@claude also consider and confirm or refute in same way

  1. [Input validation / SSZ spec] engine_getPayloadBodiesByRange does not enforce the SSZ limit of 32 src/Nethermind/Nethermind.Merge.Plugin/SszRest/SszCodec.cs:167 src/Nethermind/Nethermind.Merge.Plugin/SszRest/Handlers/GetPayloadBodiesByRangeSszHandler.cs:29 The SSZ spec’s MAX_PAYLOAD_BODIES_REQUEST is 32, and the response wire types enforce [SszList(32)], but the range handler forwards count to the existing JSON-RPC handler, which allows 1024. Counts 33..1024 can either incorrectly succeed or blow up during SSZ response encoding. Fix by validating count <= 32 in the SSZ range path and returning 413 before calling the engine handler.

  2. [Input validation] SSZ uint64 payload fields wrap into signed long src/Nethermind/Nethermind.Merge.Plugin/SszRest/SszExecutionPayload.cs:65 :71, :77, :188, :194, :200 block_number, gas_limit, and gas_used are decoded as SSZ uint64 and then cast unchecked to Nethermind’s signed long domain fields. Values above long.MaxValue wrap negative, affecting fork lookup and block header construction. Fix with checked/range validation and throw InvalidDataException so malformed SSZ returns 400.

  3. [DoS / generator validation] variable SSZ list limits are checked after allocation src/Nethermind/Nethermind.Serialization.SszGenerator/SszGenerator.cs:831 :833, :1044, :1046, :1244, :1245 For variable-size element lists, the generated decoder computes element count from the first offset and allocates the whole array before the property’s [SszList(limit)] validation runs. Through the new SSZ Engine endpoints, a body under the 16 MiB cap can still force large allocations before rejection. Fix by emitting a pre-allocation limit check for Kind.List properties, or pass the property limit into generated array decode.

@claude
Copy link
Copy Markdown
Contributor

claude Bot commented May 10, 2026

Claude finished @benaadams's task in 5m 5s —— View job


Analyzing @benaadams's findings — all three verified against current HEAD.

Tasks

  • Read relevant files and analyze all three issues
  • Finding 1: getPayloadBodiesByRange SSZ limit not enforced
  • Finding 2: uint64 payload fields wrap to signed long
  • Finding 3: Variable SSZ list limits checked after allocation

Finding 1 — ✅ Confirmed: getPayloadBodiesByRange does not enforce the SSZ limit of 32

GetPayloadBodiesByRangeSszHandler.cs:29 / SszCodec.cs:167-171

The SSZ handler decodes (start, count) from the wire and forwards both straight to TVersion.Call(engineModule, start, count) without capping count. The JSON-RPC handler (GetPayloadBodiesByRangeHandler) allows MaxCount = 1024. The response wire types enforce [SszList(32)], so ValidateSszListLimit fires during Encode when count > 32 — but only after the engine has already fetched up to 1024 blocks.

Concretely with count = 1024:

  1. Engine executes: up to 1024 block fetches from the block tree.
  2. EncodePayloadBodiesV1Response calls EncodeToWriter(new PayloadBodiesV1ResponseWire { PayloadBodies = arr }, writer).
  3. Inside PayloadBodiesV1ResponseWire.Encode(…), ValidationStatements calls ValidateSszListLimit(arr, 32, …)InvalidDataException.
  4. Middleware catches it as malformed SSZ → 422 (or 400 after the round-6 M1 fix).

So the endpoint does eventually reject counts > 32, but only after paying the full engine cost. The spec-correct behaviour is to validate count ≤ MAX_PAYLOAD_BODIES_REQUEST before touching the engine.

The correct status for this is 400 Bad Request (client sent an out-of-range parameter), not 413.

Fix this →


Finding 2 — ✅ Confirmed: BlockNumber, GasLimit, GasUsed cast ulonglong unchecked

SszExecutionPayload.cs:65, 71, 77 and :188, 194, 200

Both SszExecutionPayloadV1 and SszExecutionPayload setters do inner.BlockNumber = (long)value etc., where the domain fields (ExecutionPayload.BlockNumber, GasLimit, GasUsed) are long. The cast is unchecked in C#: any value > long.MaxValue wraps to a negative long.

A crafted SSZ newPayload body with block_number = 0xFFFFFFFFFFFFFFFF writes -1 into inner.BlockNumber. Downstream effects:

  • specProvider.GetSpec(BlockNumber, Timestamp) receives a negative block number — fork lookup returns wrong spec, potentially causing a valid block to be flagged with wrong EIP gating.
  • Gas accounting (capacity checks, gas-used vs. gas-limit comparison) operates on negative values.

These are consensus-correctness bugs triggered by unauthenticated SSZ input on an authenticated port (JWT required). The practical risk is a compromised or misconfigured CL client submitting a crafted payload that Nethermind accepts/rejects incorrectly.

Fix: add a range guard in each setter:

set
{
    if (value > (ulong)long.MaxValue)
        throw new InvalidDataException(
            $"SSZ uint64 value {value} exceeds maximum representable block field value");
    inner.BlockNumber = (long)value;
}

Throwing InvalidDataException means the middleware's catch will map this to 400 Bad Request.

Same fix needed in SszExecutionPayloadV1 (lines 65, 71, 77) and SszExecutionPayload (lines 188, 194, 200).

Fix this →


Finding 3 — ✅ Confirmed: Variable SSZ list limits validated after array allocation

SszGenerator.cs:541, 831-833, 1042-1046, 1244-1245

In the generated decode path for variable-size list properties, the pattern is:

// Inside a container Decode():
{property.Type.Name}.Decode(slice, out T[] variableName);  // allocates new T[length] inside
container.{property.Name} = [..variableName];               // assigns
ValidateSszListLimit(container.{property.Name}, limit,);  // checks limit AFTER

And for the basic ref-type list path (line 541):

int __count = slice.Length / itemSize;
T[] result = new T[__count];                 // allocates first
// ... fill loop
container.Property = result;
ValidateSszListLimit(result, limit,);      // limit check after allocation

Through the SSZ Engine API path, a crafted authenticated request body can trigger over-allocation before rejection. The 16 MiB body cap limits the theoretical maximum, but the cost is asymmetric:

  • GetBlobsRequestWire.VersionedHashes [SszList(4096)]: the item type is Hash256 (32 bytes). A 16 MiB body = 524,288 hashes → allocates new Hash256[524288] (each a managed 32-byte object) before ValidateSszListLimit rejects it at 4096. The 4096-element limit is checked, but only after 128× the limit worth of managed objects have been allocated.
  • GetPayloadBodiesByHashRequestWire.BlockHashes [SszList(32)]: same pattern, limit 32, body can cause ~500K allocations first.

Fix in SszGenerator.cs: for Kind.List properties with variable-size items, emit the limit check before the allocation in the generated Decode method body. For the basic-ref-type path (line 541), change the emitted code to check __count <= property.Limit before new T[__count].

Fix this →


Summary

# Finding Severity Verdict
1 getPayloadBodiesByRange count > 32 triggers full engine work before 422 rejection Medium ✅ Confirmed
2 BlockNumber/GasLimit/GasUsed unchecked ulong→long cast wraps to negative High ✅ Confirmed
3 Variable-length SSZ list elements allocated before ValidateSszListLimit fires Medium ✅ Confirmed

Finding 2 is the most impactful — an authenticated CL can set consensus-critical fields to negative values via crafted SSZ, potentially corrupting fork-spec lookups and gas accounting on the Nethermind side. Findings 1 and 3 are bounded-cost denial-of-service vectors that waste engine work or trigger over-allocation before rejection.

@benaadams
Copy link
Copy Markdown
Member

@claude re-review

@claude
Copy link
Copy Markdown
Contributor

claude Bot commented May 10, 2026

Claude finished @benaadams's task in 5m 32s —— View job


Re-review — all 8 findings confirmed fixed ✅

Verified each finding from both of @benaadams' comment rounds against the current branch.


Round 1 — all 5 fixed

# Finding Status
1 getPayloadBodiesByRange SSZ limit (32) not enforced before calling engine ✅ Fixed — GetPayloadBodiesByRangeSszHandler.cs:32-37 checks count > MaxPayloadBodiesRequest before forwarding to the engine
2 ResultWrapper<T> not disposed in WriteSszResultAsync ✅ Fixed — SszEndpointHandlerBase.cs:73 wraps the result in using (result)
3 Fixed-size list decode: no remainder check (could silently truncate) ✅ Fixed — SszGenerator.cs now emits a remainderGuard (if (slice.Length % itemSize != 0) throw InvalidDataException) before counting elements
4 WS-only port not rejected (SSZ middleware intercepted WebSocket-only listeners) ✅ Fixed — SszMiddleware.cs:122 now checks !url.RpcEndpoint.HasFlag(RpcEndpoint.Http) and falls through to _next
5 Race condition in EngineRpcCapabilitiesProvider lazy init ✅ Fixed — EnsureBuilt() assigns _ssz before Volatile.Write(ref _jsonRpc, ...), so any thread observing _jsonRpc != null is guaranteed to see a fully initialized _ssz

Round 2 — all 3 fixed

# Finding Status
1 ulong → long silent overflow in BlockNumber/GasLimit/GasUsed setters ✅ Fixed — both SszExecutionPayloadV1 and SszExecutionPayload now throw InvalidDataException when value > (ulong)long.MaxValue
2 Withdrawals setter didn't distinguish null (field absent) from empty array ✅ Fixed — setter in SszExecutionPayload now has separate if (value is null)_inner.Withdrawals = null and if (value.Length == 0)_inner.Withdrawals = [] paths
3 SSZ list decoder allocates array before checking count against the spec limit ✅ Fixed — SszGenerator.cs now emits a preAllocationListGuard that reads the element count (first offset ÷ pointer size for variable-size, slice ÷ item size for fixed-size) and throws InvalidDataException if it exceeds the declared limit, before new T[count]

One minor observation (non-blocking)

In SszCodec.DecodeGetPayloadBodiesByRangeRequest (line 170), wire.Count (ulong) is cast directly to long before being returned. Values in (long.MaxValue, ulong.MaxValue] wrap to negative, so a hostile Count = ulong.MaxValue would arrive at the handler as −1, pass the count > 32 SSZ-limit guard (since −1 is not > 32), and then be caught by the engine handler's existing count < 1 guard with a 400 InvalidParams. No correctness or security impact — the caller still gets a 400 — but the error message is the engine's "must be positive" rather than the SSZ spec limit message. See inline comment for a suggested one-liner fix.


All findings addressed. Ready to merge from a correctness standpoint.

Comment thread src/Nethermind/Nethermind.Merge.Plugin/SszRest/SszCodec.cs Outdated
Comment thread src/Nethermind/Nethermind.Merge.Plugin/SszRest/SszExecutionPayload.cs Outdated
LukaszRozmej and others added 5 commits May 11, 2026 00:22
NewPayloadSerializationBenchmarks and GetPayloadV3SerializationBenchmarks were
duplicating ~150 lines of identical plumbing: host builder, JSON-RPC processor
wiring, stub URL collection / auth / process-exit, header constants, and the
withdrawal factory.

Extract to EngineBenchmarkHost (internal static): Build(...) for the engine
TestServer scaffolding, BuildJsonServer(engine) for the JSON-RPC pipeline,
BuildWithdrawals(count), shared header values, and the three Stub* classes.
Both benchmark files now keep only their format-specific encoding / decoding
helpers and the SSZ handler registration.

No behavior change. Build clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…eGetPayloadBodiesByRangeRequest

Address benaadams's review on SszExecutionPayload.cs:70 — the 5 inline
"if (value > (ulong)long.MaxValue) throw …" property setters share an
identical shape but no shared code. Extract to a single
SszNumericChecks.CheckedLong helper with the suggested cold-path local
function + [DoesNotReturn, StackTraceHidden] for clean JIT codegen.

While there, fix the unchecked ulong→long cast in
SszCodec.DecodeGetPayloadBodiesByRangeRequest: wire.Start and wire.Count
arrived as long via direct (long)wire.Count, so values in
(long.MaxValue, ulong.MaxValue] wrapped to negative. The handler-side
"count > 32" guard then passed (negative isn't > 32), and the engine's
"count < 1" guard caught it with a generic 400 "must be positive". No
security impact, but the SSZ-spec limit message was being skipped. With
CheckedLong, the same out-of-range condition now produces a proper SSZ
400 "uint64 value … exceeds valid range for long" at decode time.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…nerator

The attribute existed so consumers could declare new SSZ basic types
out-of-tree, but the only two users (SszBytes8, SszKzgCommitment) lived
inside Nethermind itself. Per-compilation discovery (DiscoverKnownTypes
walking the namespace tree, attribute symbol lookup, WeakReference cache)
was ~80 lines of generator code in service of two types.

Move SszBytes8 and SszKzgCommitment to Nethermind.Serialization.Ssz so
every project that depends on the SSZ generator can see them, then hardcode
them in SszType.BasicTypes alongside SszBytes32. Generator no longer needs
the discovery pass; the [SszBasicType] attribute and its attribute class
are deleted.

Adding a new fixed-length basic SSZ type now means one entry in
SszType.BasicTypes — same friction as the existing Hash256 / Address / Bloom
entries.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
After SszBytes8 and SszKzgCommitment moved to Nethermind.Serialization.Ssz,
the remaining wire types in this file don't use System, InteropServices,
or CompilerServices types — only the SSZ generator attributes plus core/crypto.
Drops three dead usings; resolves IDE0005 warning surfaced by CI.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@LukaszRozmej LukaszRozmej merged commit 4144d02 into NethermindEth:master May 11, 2026
546 of 549 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dependencies Pull requests that update a dependency file new feature performance is good refactoring rpc taiko related to the taiko alethia rollup

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants