From 49dea04b197cce64fd18aec622b1339b18376429 Mon Sep 17 00:00:00 2001 From: developeruche Date: Fri, 20 Mar 2026 05:51:07 +0100 Subject: [PATCH 1/5] added REST spec for new-payload-with-witness --- src/engine/README.md | 2 +- src/engine/common.md | 4 + src/engine/rest.md | 171 +++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 176 insertions(+), 1 deletion(-) create mode 100644 src/engine/rest.md diff --git a/src/engine/README.md b/src/engine/README.md index c02c92e66..7d64028d3 100644 --- a/src/engine/README.md +++ b/src/engine/README.md @@ -3,7 +3,7 @@ The Engine JSON-RPC API is a collection of methods that all execution clients implement. This interface allows the communication between consensus and execution layers of the two-component post-Merge Ethereum Client. -This API is in *active development* and currently specified in markdown documents specified by fork scopes ([Paris](./paris.md), [Shanghai](./shanghai.md), [Cancun](./cancun.md), [Prague](./prague.md), [Osaka](./osaka.md), [Amsterdam](./amsterdam.md)). +This API is in *active development* and currently specified in markdown documents specified by fork scopes ([Paris](./paris.md), [Shanghai](./shanghai.md), [Cancun](./cancun.md), [Prague](./prague.md), [Osaka](./osaka.md), [Amsterdam](./amsterdam.md)), with shared definitions in [Common](./common.md), [Authentication](./authentication.md), and optional [HTTP REST](./rest.md) bindings. ## References * [Engine API: A Visual Guide](https://hackmd.io/@danielrachi/engine_api) diff --git a/src/engine/common.md b/src/engine/common.md index b76d03bf9..4fe04a59f 100644 --- a/src/engine/common.md +++ b/src/engine/common.md @@ -51,6 +51,10 @@ These methods are described in [Ethereum JSON-RPC Specification][json-rpc-spec]. Engine API uses JWT authentication enabled by default. JWT authentication is specified in [Authentication](./authentication.md) document. +## HTTP REST bindings (optional) + +Execution layer client software **MAY** expose additional `POST` routes on the same Engine HTTP server (same port and authentication rules) as specified in [Engine API -- REST](./rest.md). Those routes are not JSON-RPC; they use JSON request bodies and SSZ response bodies as defined in that document. + ## Versioning The versioning of the Engine API is defined as follows: diff --git a/src/engine/rest.md b/src/engine/rest.md new file mode 100644 index 000000000..237df6ecd --- /dev/null +++ b/src/engine/rest.md @@ -0,0 +1,171 @@ +# Engine API -- HTTP REST bindings + +This document specifies optional **HTTP REST** endpoints for a subset of the Engine API. They are exposed on the same authenticated Engine HTTP server and port as JSON-RPC (see [Common Definitions](./common.md)). + +JSON-RPC methods remain the canonical specification for request parameters and processing rules. REST endpoints **MUST** behave identically to their JSON-RPC counterparts except for the HTTP request shape and response encoding described here. + +## Table of contents + + + + +- [Structures](#structures) + - [SSZ PayloadStatusV1](#ssz-payloadstatusv1) + - [SSZ ExecutionWitnessV1](#ssz-executionwitnessv1) + - [SSZ NewPayloadWithWitnessResponseV1](#ssz-newpayloadwithwitnessresponsev1) +- [Endpoints](#endpoints) + - [new-payload-with-witness](#new-payload-with-witness) + - [Request](#request) + - [Successful response](#successful-response) + - [Error responses](#error-responses) + - [Specification](#specification) +- [Capabilities](#capabilities) + + + +## Structures + +### SSZ PayloadStatusV1 + +This type is an SSZ `Container` encoding the same logical object as JSON-RPC [`PayloadStatusV1`](./paris.md#payloadstatusv1). + +| Index | Field name | SSZ type | +| ----- | ---------- | -------- | +| 0 | `status` | `uint8` | +| 1 | `latest_valid_hash` | `Union[None, ByteVector[32]]` | +| 2 | `validation_error` | `Union[None, List[uint8, VALIDATION_ERROR_MAX]]` | + +Constants: + +* `VALIDATION_ERROR_MAX`: `8192` (maximum number of `uint8` elements in the list when the `validation_error` union variant is present). + +`uint8` values for `status` **MUST** be: + +| Value | JSON-RPC `status` string | +| ----- | ------------------------ | +| `0` | `VALID` | +| `1` | `INVALID` | +| `2` | `SYNCING` | +| `3` | `ACCEPTED` | +| `4` | `INVALID_BLOCK_HASH` | + +Mapping from JSON-RPC field values to SSZ: + +* `latestValidHash: null` **MUST** be encoded as the `None` variant of `latest_valid_hash`. +* `latestValidHash: <32-byte DATA>` **MUST** be encoded as the `ByteVector[32]` variant (raw bytes, **not** hex). +* `validationError: null` **MUST** be encoded as the `None` variant of `validation_error`. +* `validationError: ` **MUST** be encoded as UTF-8 bytes in the `List[uint8, VALIDATION_ERROR_MAX]` variant. If the UTF-8 encoding exceeds `VALIDATION_ERROR_MAX` bytes, client software **MUST** truncate to `VALIDATION_ERROR_MAX` bytes without splitting a multi-byte UTF-8 code point (i.e. truncate to the longest prefix that is valid UTF-8 and fits the limit). + +### SSZ ExecutionWitnessV1 + +This type is an SSZ `Container` encoding the execution witness produced during block processing. It contains the raw Merkle trie nodes, preimage keys, contract bytecodes, and block headers required to statelessly verify the block's execution. + +| Index | Field name | SSZ type | +| ----- | ---------- | -------- | +| 0 | `state` | `List[List[uint8, MAX_WITNESS_ITEM_BYTES], MAX_WITNESS_ITEMS]` | +| 1 | `keys` | `List[List[uint8, MAX_WITNESS_ITEM_BYTES], MAX_WITNESS_ITEMS]` | +| 2 | `codes` | `List[List[uint8, MAX_WITNESS_ITEM_BYTES], MAX_WITNESS_ITEMS]` | +| 3 | `headers` | `List[List[uint8, MAX_WITNESS_ITEM_BYTES], MAX_WITNESS_ITEMS]` | + +Constants: + +* `MAX_WITNESS_ITEMS`: `1048576` — maximum number of items per witness field. +* `MAX_WITNESS_ITEM_BYTES`: `1048576` — maximum byte length of a single witness item. + +Field semantics: + +* `state` — serialized Merkle Patricia Trie nodes (account trie and storage tries) touched during execution. +* `keys` — preimage keys (hashed keys → original keys) used to look up trie nodes. +* `codes` — contract bytecodes accessed during execution. +* `headers` — RLP-encoded block headers needed by the `BLOCKHASH` opcode. + +An empty list (`[]`) for any field indicates no data of that category was accessed during execution. + +### SSZ NewPayloadWithWitnessResponseV1 + +This type is an SSZ `Container` combining the payload validation result and the execution witness. + +| Index | Field name | SSZ type | +| ----- | ---------- | -------- | +| 0 | `status` | `uint8` | +| 1 | `latest_valid_hash` | `Union[None, ByteVector[32]]` | +| 2 | `validation_error` | `Union[None, List[uint8, VALIDATION_ERROR_MAX]]` | +| 3 | `witness` | `List[uint8, MAX_WITNESS_BYTES]` | + +Constants: + +* `VALIDATION_ERROR_MAX`: as defined in [SSZ PayloadStatusV1](#ssz-payloadstatusv1). +* `MAX_WITNESS_BYTES`: `1073741824` (1 GiB) — maximum byte length of the SSZ-encoded `ExecutionWitnessV1`. + +Fields `status`, `latest_valid_hash`, and `validation_error` carry the same semantics as [SSZ PayloadStatusV1](#ssz-payloadstatusv1). + +The `witness` field contains the SSZ serialization of an [`ExecutionWitnessV1`](#ssz-executionwitnessv1). When the `status` is not `VALID` or no witness was produced, the `witness` field **MUST** be an empty list (`[]`). + +Serialization of `Container`, `Union`, `List`, and `ByteVector` types **MUST** follow the Ethereum consensus [Simple Serialize (SSZ) specification][ssz]. + +[ssz]: https://github.com/ethereum/consensus-specs/blob/dev/ssz/simple-serialize.md + +## Endpoints + +### new-payload-with-witness + +This endpoint **MUST** implement the same payload validation and execution logic as JSON-RPC [`engine_newPayloadV5`](./amsterdam.md#engine_newpayloadv5). In addition, when execution succeeds with `VALID` status, the response **MUST** include the execution witness. + +#### Request + +* HTTP method: `POST` +* Path: `/new-payload-with-witness` +* Header `Content-Type` **MUST** be `application/json`. +* Body: a JSON array of exactly **four** elements, in order: + 1. `executionPayload`: [`ExecutionPayloadV4`](./amsterdam.md#executionpayloadv4) + 2. `expectedBlobVersionedHashes`: `Array of DATA`, 32 Bytes each + 3. `parentBeaconBlockRoot`: `DATA`, 32 Bytes + 4. `executionRequests`: `Array of DATA` — as defined for [`engine_newPayloadV5`](./amsterdam.md#engine_newpayloadv5) +* timeout: 8s + +#### Successful response + +* HTTP status: `200 OK` +* Header `Content-Type` **MUST** be `application/octet-stream` +* Body: the SSZ serialization of [`NewPayloadWithWitnessResponseV1`](#ssz-newpayloadwithwitnessresponsev1) as defined above. + +#### Error responses + +Unless otherwise specified, error bodies **MUST** use `Content-Type: application/json` and **MUST** be a JSON object with at least: + +* `code`: `integer` — same semantics as JSON-RPC `error.code` for Engine API errors (see [Errors](./common.md#errors)). +* `message`: `string` — same semantics as JSON-RPC `error.message`. + +| Condition | HTTP status | Notes | +| --------- | ----------- | ----- | +| Missing or invalid JWT | `401` | Per [Authentication](./authentication.md). | +| Malformed JSON body, or body is not a JSON array | `400` | e.g. `-32700` Parse error where applicable. | +| Wrong number of parameters or invalid parameter shapes | `400` | `-32602` Invalid params | +| Unsupported fork or other Engine errors that JSON-RPC surfaces as `error` | `400` or `500` as appropriate | Same `code` as JSON-RPC (e.g. `-38005` Unsupported fork). | +| HTTP method other than `POST` | `405` | | + +When JSON-RPC would return a **result** `PayloadStatusV1` (including `INVALID`, `SYNCING`, etc.), this endpoint **MUST** respond with `200 OK` and an SSZ body — **not** an HTTP error — because those outcomes are part of normal payload processing. + +#### Specification + +This endpoint follows the same specification as [`engine_newPayloadV5`](./amsterdam.md#engine_newpayloadv5) with the following additions: + +1. Client software **MUST** return `-38005: Unsupported fork` error if the `timestamp` of the payload does not fall within the time frame of the Amsterdam activation. + +2. When the payload status is `VALID`, the EL **MUST** include the execution witness in the `witness` field of the response. The witness **MUST** be the SSZ serialization of an `ExecutionWitnessV1` containing all state data accessed during block execution. + +3. When the payload status is not `VALID` (e.g. `INVALID`, `SYNCING`, `ACCEPTED`), the `witness` field **MUST** be empty (`[]`). + +4. The execution witness **MUST** contain sufficient data for a stateless verifier to re-execute the block and arrive at the same post-state root. Specifically: + - `state` **MUST** contain every Merkle Patricia Trie node (from both the account trie and any storage tries) that was read or written during execution. + - `keys` **MUST** contain the preimage mappings for every hashed key used to traverse the tries. + - `codes` **MUST** contain every contract bytecode that was loaded during execution. + - `headers` **MUST** contain the RLP-encoded headers of any ancestor blocks accessed via the `BLOCKHASH` opcode. + +## Capabilities + +Execution layer client software that implements the endpoint in this document **SHOULD** include the following string in the response array of [`engine_exchangeCapabilities`](./common.md#engine_exchangecapabilities) when the corresponding route is supported: + +* `rest_engine_newPayloadWithWitness` — `POST /new-payload-with-witness` is available + +Consensus layer client software **MAY** use this to discover REST support. This capability name is **not** a JSON-RPC method name; it identifies an optional REST feature. From 051a6f35190a5b79fd9b25bc8fd553a0a9956644 Mon Sep 17 00:00:00 2001 From: developeruche Date: Fri, 20 Mar 2026 12:50:52 +0100 Subject: [PATCH 2/5] updated REST intro to make it more clear --- src/engine/rest.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/engine/rest.md b/src/engine/rest.md index 237df6ecd..fbf695229 100644 --- a/src/engine/rest.md +++ b/src/engine/rest.md @@ -2,7 +2,7 @@ This document specifies optional **HTTP REST** endpoints for a subset of the Engine API. They are exposed on the same authenticated Engine HTTP server and port as JSON-RPC (see [Common Definitions](./common.md)). -JSON-RPC methods remain the canonical specification for request parameters and processing rules. REST endpoints **MUST** behave identically to their JSON-RPC counterparts except for the HTTP request shape and response encoding described here. +The Engine API's primary interface is JSON-RPC. This document introduces an **optional** REST endpoint, `POST /new-payload-with-witness`, that performs the same payload validation and execution as [`engine_newPayloadV5`](./amsterdam.md#engine_newpayloadv5) but additionally returns an execution witness. The endpoint accepts the same JSON parameters as `engine_newPayloadV5` and returns the response as SSZ-encoded bytes. All validation rules, processing logic, and error semantics are inherited from `engine_newPayloadV5` unless explicitly stated otherwise below. ## Table of contents @@ -103,7 +103,7 @@ The `witness` field contains the SSZ serialization of an [`ExecutionWitnessV1`]( Serialization of `Container`, `Union`, `List`, and `ByteVector` types **MUST** follow the Ethereum consensus [Simple Serialize (SSZ) specification][ssz]. -[ssz]: https://github.com/ethereum/consensus-specs/blob/dev/ssz/simple-serialize.md +[ssz]: https://github.com/ethereum/consensus-specs/blob/master/ssz/simple-serialize.md ## Endpoints From cc107ff51cf29c5867abafb20931dc36a70dcb13 Mon Sep 17 00:00:00 2001 From: developeruche Date: Tue, 14 Apr 2026 12:15:37 +0100 Subject: [PATCH 3/5] review impl --- src/engine/rest.md | 21 ++++++--------------- 1 file changed, 6 insertions(+), 15 deletions(-) diff --git a/src/engine/rest.md b/src/engine/rest.md index fbf695229..2318bb8da 100644 --- a/src/engine/rest.md +++ b/src/engine/rest.md @@ -63,21 +63,15 @@ This type is an SSZ `Container` encoding the execution witness produced during b | Index | Field name | SSZ type | | ----- | ---------- | -------- | | 0 | `state` | `List[List[uint8, MAX_WITNESS_ITEM_BYTES], MAX_WITNESS_ITEMS]` | -| 1 | `keys` | `List[List[uint8, MAX_WITNESS_ITEM_BYTES], MAX_WITNESS_ITEMS]` | -| 2 | `codes` | `List[List[uint8, MAX_WITNESS_ITEM_BYTES], MAX_WITNESS_ITEMS]` | -| 3 | `headers` | `List[List[uint8, MAX_WITNESS_ITEM_BYTES], MAX_WITNESS_ITEMS]` | +| 1 | `codes` | `List[List[uint8, MAX_WITNESS_ITEM_BYTES], MAX_WITNESS_ITEMS]` | +| 2 | `headers` | `List[List[uint8, MAX_WITNESS_ITEM_BYTES], MAX_WITNESS_ITEMS]` | Constants: * `MAX_WITNESS_ITEMS`: `1048576` — maximum number of items per witness field. * `MAX_WITNESS_ITEM_BYTES`: `1048576` — maximum byte length of a single witness item. -Field semantics: - -* `state` — serialized Merkle Patricia Trie nodes (account trie and storage tries) touched during execution. -* `keys` — preimage keys (hashed keys → original keys) used to look up trie nodes. -* `codes` — contract bytecodes accessed during execution. -* `headers` — RLP-encoded block headers needed by the `BLOCKHASH` opcode. +For detailed field semantics and specific encoding requirements, refer to the [Execution Witness specification](https://github.com/ethereum/execution-specs/blob/e2ff4cc00ca94cb5872090e0f813894e231f6be3/src/ethereum/forks/amsterdam/stateless.py#L27-L47). An empty list (`[]`) for any field indicates no data of that category was accessed during execution. @@ -120,7 +114,9 @@ This endpoint **MUST** implement the same payload validation and execution logic 1. `executionPayload`: [`ExecutionPayloadV4`](./amsterdam.md#executionpayloadv4) 2. `expectedBlobVersionedHashes`: `Array of DATA`, 32 Bytes each 3. `parentBeaconBlockRoot`: `DATA`, 32 Bytes - 4. `executionRequests`: `Array of DATA` — as defined for [`engine_newPayloadV5`](./amsterdam.md#engine_newpayloadv5) + 4. `executionRequests`: `Array of DATA` + + These parameters are as defined for [`engine_newPayloadV5`](./amsterdam.md#engine_newpayloadv5). * timeout: 8s #### Successful response @@ -156,11 +152,6 @@ This endpoint follows the same specification as [`engine_newPayloadV5`](./amster 3. When the payload status is not `VALID` (e.g. `INVALID`, `SYNCING`, `ACCEPTED`), the `witness` field **MUST** be empty (`[]`). -4. The execution witness **MUST** contain sufficient data for a stateless verifier to re-execute the block and arrive at the same post-state root. Specifically: - - `state` **MUST** contain every Merkle Patricia Trie node (from both the account trie and any storage tries) that was read or written during execution. - - `keys` **MUST** contain the preimage mappings for every hashed key used to traverse the tries. - - `codes` **MUST** contain every contract bytecode that was loaded during execution. - - `headers` **MUST** contain the RLP-encoded headers of any ancestor blocks accessed via the `BLOCKHASH` opcode. ## Capabilities From 6c22ae57b6a26ba6a801fb88eef1599ab5651fdb Mon Sep 17 00:00:00 2001 From: developeruche Date: Wed, 15 Apr 2026 06:52:08 +0100 Subject: [PATCH 4/5] review impl --- src/engine/rest.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/engine/rest.md b/src/engine/rest.md index 2318bb8da..cf523b539 100644 --- a/src/engine/rest.md +++ b/src/engine/rest.md @@ -58,7 +58,7 @@ Mapping from JSON-RPC field values to SSZ: ### SSZ ExecutionWitnessV1 -This type is an SSZ `Container` encoding the execution witness produced during block processing. It contains the raw Merkle trie nodes, preimage keys, contract bytecodes, and block headers required to statelessly verify the block's execution. +This type is an SSZ `Container` encoding the execution witness produced during block processing. It contains the raw Merkle trie nodes, contract bytecodes, and block headers required to statelessly verify the block's execution. | Index | Field name | SSZ type | | ----- | ---------- | -------- | From 324d0171142ac2c0953dc60a337d2b5d410eac43 Mon Sep 17 00:00:00 2001 From: developeruche Date: Fri, 17 Apr 2026 20:04:55 +0100 Subject: [PATCH 5/5] removed serialization optimization with little impact --- src/engine/rest.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/src/engine/rest.md b/src/engine/rest.md index cf523b539..729af580d 100644 --- a/src/engine/rest.md +++ b/src/engine/rest.md @@ -84,16 +84,15 @@ This type is an SSZ `Container` combining the payload validation result and the | 0 | `status` | `uint8` | | 1 | `latest_valid_hash` | `Union[None, ByteVector[32]]` | | 2 | `validation_error` | `Union[None, List[uint8, VALIDATION_ERROR_MAX]]` | -| 3 | `witness` | `List[uint8, MAX_WITNESS_BYTES]` | +| 3 | `witness` | `Union[None, ExecutionWitnessV1]` | Constants: * `VALIDATION_ERROR_MAX`: as defined in [SSZ PayloadStatusV1](#ssz-payloadstatusv1). -* `MAX_WITNESS_BYTES`: `1073741824` (1 GiB) — maximum byte length of the SSZ-encoded `ExecutionWitnessV1`. Fields `status`, `latest_valid_hash`, and `validation_error` carry the same semantics as [SSZ PayloadStatusV1](#ssz-payloadstatusv1). -The `witness` field contains the SSZ serialization of an [`ExecutionWitnessV1`](#ssz-executionwitnessv1). When the `status` is not `VALID` or no witness was produced, the `witness` field **MUST** be an empty list (`[]`). +The `witness` field is a `Union` type: the `None` variant **MUST** be used when the `status` is not `VALID` or no witness was produced; the `ExecutionWitnessV1` variant **MUST** be used when the `status` is `VALID` and a witness was generated. Serialization of `Container`, `Union`, `List`, and `ByteVector` types **MUST** follow the Ethereum consensus [Simple Serialize (SSZ) specification][ssz].