Skip to content

[one-shot implementation for discussion, not for merging]: feat(midnight): add first-class Midnight proof support (native + EVM) via protocol/model extensions and SDK adapter#2

Open
jtcoolen wants to merge 38 commits intoEYBlockchain:mainfrom
jtcoolen:julian@midnight-proof-support
Open

Conversation

@jtcoolen
Copy link

@jtcoolen jtcoolen commented Feb 25, 2026 •
edited
Loading

[Only the last 15 commits apply, the first commits come from #1]

Fixes [Midnight] issues

Context / Motivation

Midnight proofs are not wire-compatible with vanilla Halo2 in several places:

  1. Committed instance prefix: some leading instance columns are commitments (EC points), not scalar vectors.
  2. Instance-length hashing: transcript absorption can hash instance-column lengths before instance values.
  3. Trailing challenges: there can be extra challenges not tied to any witness phase.
  4. Extra commitments: there can be additional commitments after challenges are squeezed.
  5. Quotient recombination base: quotient chunk recombination uses (z^n / z) (vs Halo2’s z^n), with a fixed chunk count derived from Midnight domain configuration.

You can understand this branch as:

Make the generic verifier model express Midnight semantics, then translate Midnight VK/CS into that model.

Once the protocol language can encode the 5 differences above, the bulk of verification logic (native + EVM codegen) stays generic.

High-Level Design

  • Extend core PLONK protocol/proof structs to represent transcript/layout variants (the “protocol language”).

  • Keep legacy behavior as defaults so existing Halo2 flows remain unchanged.

  • Add a Midnight-specific KZG multiopen implementation (KzgAs<_, Midnight>).

  • Add a feature-gated SDK bridge that:

    • converts Midnight artifacts into halo2-axiom types,
    • builds a Midnight-correct PlonkProtocol,
    • reuses the same generic verifier path for both native verification and EVM codegen/runtime.

Detailed Changes

1) Core protocol model extensions

File: snark-verifier/src/verifier/plonk/protocol.rs

Protocol metadata added

  • committed_instance_count
  • hash_instance_lengths
  • trailing_challenges
  • extra_commitments

These fields are the main abstraction pivot: Midnight is represented as protocol metadata rather than a forked verifier.

Quotient model extended

  • QuotientChunkBase::{Zn, ZnMinusOne} where ZnMinusOne corresponds to (z^n / z)
  • num_chunk_override: Option<usize>

Behavior:

  • num_chunk() honors the explicit override when present.
  • Metadata is preserved across conversions (loaded, loaded_preprocessed_as_witness).

Legacy behavior preserved

  • Vanilla Halo2 compile path initializes neutral defaults so existing flows stay unchanged.

2) Proof parsing / verification updates

File: snark-verifier/src/verifier/plonk/proof.rs

Committed instance handling (absorption order)

  • A committed-instance-aware flow now:

    1. absorbs committed-instance EC points first (if configured),
    2. then absorbs scalar instances for non-committed columns,
    3. optionally absorbs each non-committed column length first when hash_instance_lengths=true.

New parser entrypoint

  • read_with_committed_instances(...) supports caller-provided committed-instance commitments.

Trailing challenges + extra commitments

  • After phase-driven witness/challenge parsing, the parser appends:

    • trailing_challenges
    • extra_commitments

Commitment indexing alignment

  • Committed-instance placeholders are appended to align commitment indexing with num_instance column layout.

Quotient recombination supports both bases

  • Verifier recombines quotient chunks using either:

    • Halo2 base: z^n (Zn)
    • Midnight base: (z^n / z) (ZnMinusOne)

Tests added

  • committed prefix ordering + scalar absorption order
  • deterministic errors for layout mismatches
  • committed-instance placeholder extension behavior

3) Midnight KZG PCS flavor (multiopen)

Files:

  • snark-verifier/src/pcs/kzg.rs
  • snark-verifier/src/pcs/kzg/multiopen.rs
  • snark-verifier/src/pcs/kzg/multiopen/midnight.rs

Added:

  • Midnight multiopen scheme and MidnightProof struct.

Transcript layout

  • Parsed as: x1, x2, f_com, x3, q_evals_on_x3, x4, pi

Verification routine

  • groups queries by point-set structure
  • reconstructs interpolation terms
  • performs batched inversion
  • maps Midnight pairing equation into snark-verifier accumulator convention

Optional: truncated challenges

  • truncated-challenges feature applies 128-bit truncation policy for challenge powers (native + EVM loaders).

4) EVM transcript/runtime correctness alignment

Files:

  • snark-verifier/src/system/halo2/transcript/evm.rs
  • snark-verifier/src/loader/evm/loader.rs
  • snark-verifier/src/loader/evm/code.rs

Fixes/hardening:

  • transcript absorption handles disjoint memory by copying into a contiguous transcript buffer
  • identity point encoding/decoding aligned to EVM sentinel (0,0)
  • constant EC point loader emits (0,0) for identity
  • scalar truncation helper for truncated-challenge mode
  • batched MSM precompile path for multi-scalar multiplications
  • zero-safe deterministic batch inversion path
  • generated Solidity pragma updated to 0.8.30

5) SDK Midnight adapter (translation layer)

Files:

  • snark-verifier-sdk/src/midnight_adapter/*

Added:

  • MidnightProofBundle for constructing verification/codegen inputs from Midnight params/VK/proof/instances
  • strict normalization/validation of committed-instance shape
  • Midnight→halo conversions for fields and curve points
  • MidnightProtocolBuilder translating Midnight VK/CS into PlonkProtocol

MidnightProtocolBuilder guarantees:

  • query/evaluation ordering matches Midnight transcript expectations

  • committed-instance semantics reflected in protocol metadata

  • inclusion of trash constraints + trash witness columns

  • quotient config:

    • chunk_base = ZnMinusOne
    • num_chunk_override = vk quotient degree (fixed chunk count from Midnight domain config)

EVM parity design

  • build_evm_verifier_loader() replays the same generic proof read + verify flow over EvmLoader,
    so emitted runtime code is derived from verifier execution (not a hand-written special-case path).

6) Feature plumbing

Files:

  • snark-verifier-sdk/Cargo.toml
  • snark-verifier-sdk/src/lib.rs
  • snark-verifier/Cargo.toml

Added:

  • SDK midnight feature gating Midnight deps + adapter module
  • truncated-challenges passthrough from SDK to core
  • example registration for Midnight EVM transcript flow

7) End-to-end example

Files:

  • snark-verifier-sdk/examples/midnight_poseidon_evm.rs
  • snark-verifier-sdk/examples/support/midnight_evm_transcript.rs

Example demonstrates:

  • build Poseidon proof with Midnight EVM transcript hash
  • verify natively
  • generate Solidity verifier + bytecode + calldata
  • optionally run local revm verification for gas/result sanity

Backward Compatibility

  • Existing non-Midnight flows remain unchanged.
  • New protocol fields are serde-defaulted and initialized to legacy values in standard Halo2 compile path.
  • Midnight behavior is opt-in via feature flags and adapter usage.

Risks / Review Focus

  • Query ordering and polynomial index offsets in MidnightProtocolBuilder.
  • Correct absorption order for committed vs non-committed instances (including optional length hashing).
  • Quotient chunk recombination base (ZnMinusOne) and fixed chunk count override.
  • KZG Midnight multiopen transcript layout compatibility with prover output.
  • EVM identity encoding consistency ((0,0) sentinel) across read/write/codegen.

Testing

  • cargo check -p snark-verifier-sdk --features midnight,loader_evm
  • cargo +1.88.0 check -p snark-verifier-sdk --features midnight,loader_evm,revm
  • cargo +1.88.0 check -p snark-verifier-sdk --features midnight,loader_evm,revm,truncated-challenges
  • cargo run --example midnight_poseidon --features midnight -p snark-verifier-sdk (if present in local tree)
  • RUN_REVM=1 cargo +1.88.0 run --example midnight_poseidon_evm --features midnight,loader_evm,revm -p snark-verifier-sdk
proof bytes: 3968
deployment code bytes: 22817
calldata bytes: 4000
wrote .../target/midnight/MidnightPoseidonVerifier.sol
wrote .../target/midnight/midnight_poseidon.bytecode
wrote .../target/midnight/midnight_poseidon.calldata
revm gas: 597142

@jtcoolen
build(evm): update generated verifier pragma to Solidity 0.8.30
d8f1bbf 
Align generated verifier source with Solidity 0.8.30 in the EVM codegen template.

- Change the emitted contract pragma in SolidityAssemblyCode from 0.8.19 to 0.8.30.
@jtcoolen
test(evm): raise local limits for large-verifier gas estimation
bcd1f54 
Adjust local EVM execution helpers so oversized verifier contracts can be deployed and profiled in simulation.

- Add serde_json dependency used by EVM utility flows.

- Silence non-critical solc stderr output during bytecode compilation.

- Relax revm code-size and gas-limit constraints in deploy_and_call for high-bytecode verifier benchmarks.
@jtcoolen
Copy link
Author

jtcoolen commented Feb 25, 2026

Only the last 10 commits apply, the first 21 commits come from #1

@jtcoolen jtcoolen changed the title feat(midnight): add first-class Midnight proof support (native + EVM) via protocol/model extensions and SDK adapter [one-shot implementation for discussion, not for merging]: feat(midnight): add first-class Midnight proof support (native + EVM) via protocol/model extensions and SDK adapter Feb 25, 2026
This was referenced Feb 27, 2026
Open
Open
jtcoolen
jtcoolen commented Mar 2, 2026
View reviewed changes
snark-verifier/src/loader/evm/loader.rs
.reduce(|acc, ec_point| acc.loader.ec_point_add(&acc, &ec_point))
.expect("pairs should not be empty")
// Route all MSM calls through the batched G1MSM precompile helper.
pairs.first().expect("pairs should not be empty").1.loader.ec_point_multi_scalar_mul(pairs)
Copy link
Author

@jtcoolen jtcoolen Mar 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of iteratively calling the Bls12_381G1Add/Bls12_381G1Msm precompile we call Bls12_381G1Msm directly with all points and scalars

Copy link
Author

@jtcoolen jtcoolen Mar 2, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixes https://github.com/EYBlockchain/Moonlight/issues/42, could be moved to #1

jtcoolen added 21 commits March 2, 2026 11:39
@jtcoolen
chore(vendor): vendor halo2-lib snapshot and route halo2-base/ecc loc…
6ceb111 
…ally

Vendor the halo2-lib workspace under vendor/halo2-lib and route both axiom-crypto and jtcoolen halo2-lib sources to local halo2-base/halo2-ecc paths.

This commit keeps the raw upstream snapshot state before applying the halo2-base BLS patches.
@jtcoolen
chore(vendor): apply halo2-base blsFr scalar-field patch (360a9d3)
5fc6269 
Apply upstream halo2-base commit 360a9d3 to the vendored tree so blsFr implements ScalarField in utils::scalar_field_impls.
@jtcoolen
chore(vendor): apply halo2-base blsFq 6-limb patch set (f56347d)
52272a3 
Apply upstream halo2-base commit f56347d after 360a9d3, including 6-limb blsFq ScalarField support and halo2curves revision pinning in the vendored halo2-lib metadata.
@jtcoolen
fix(vendor): make from_u64_digits width-agnostic
3521425 
Preserve the local vendored fix by reconstructing from little-endian bytes instead of assuming a [u64;4] representation.
@jtcoolen
chore(lockfile): resolve dependency graph for pinned halo2 stack
7c2e1b9 
Refresh Cargo.lock so halo2-base/halo2-ecc and halo2curves-axiom resolve to the pinned fork revisions deterministically.
@jtcoolen
chore(vendor): import halo2-base snapshot from f56347d
c5a590e 
Vendor halo2-base sources from jtcoolen/halo2-lib@f56347d without local modifications for reproducible builds.
@jtcoolen
fix(vendor): make from_u64_digits width-agnostic
37ddb9b 
Replace the hardcoded [u64;4] conversion with byte-based reconstruction so wider scalar fields are handled safely.
@jtcoolen
refactor(sdk): switch bn256 generics to bls12_381
37c0669 
Migrate SDK proving/verifying type aliases and generic bounds from Bn256 to Bls12 across halo2 and EVM entry points.
@jtcoolen
feat(evm-loader): support BLS byte widths for proof/ec-point IO
86c7d76 
Generalize EVM loader point encoding for wider base fields and update transcript IO sizing to distinguish proof-byte and EVM-memory representations.
@jtcoolen
feat(evm-decider): move G2 serialization to 8-word pairing inputs
63d9de6 
Serialize G2 points as eight 32-byte words and update the EVM pairing call interface to consume the expanded layout.
@jtcoolen
feat(evm-codegen): adopt BLS precompile call shapes in generated runtime
0acb7c4 
Switch generated runtime precompile identifiers and calldata/returndata sizing to BLS12-381 G1/G2 operation layouts.
@jtcoolen
fix(evm): correct EIP-2537 precompile addresses
de04db8 
Use Prague EIP-2537 precompile addresses for BLS12-381 add, MSM, and pairing operations.
@jtcoolen
fix(decider): serialize G2 in EIP-2537 c0,c1 order
594f95c 
Remove the temporary legacy c1,c0 env-toggle path and keep a single canonical ordering for EVM pairing inputs.
@jtcoolen
fix(field-arith): set LIMBS=3 and BITS=128 for BLS constraints
6e50139 
Adjust limb decomposition parameters for BLS12-381 base-field arithmetic compatibility in halo2-ecc.
@jtcoolen
chore(toolchain): bump rust-toolchain to 1.88.0
a089d14 
Align pinned toolchain with revm/alloy dependency MSRV requirements used by the BLS runtime path.
@jtcoolen
fix(revm): harden Prague config and revert diagnostics
7c3dd7f 
Update local EVM execution config/error reporting for Prague precompiles and tighten transcript runtime handling for allocated scalar/ec-point memory layout.
@jtcoolen
chore(lockfile): refresh lockfile after revm/runtime wiring
79bfb0a 
Update Cargo.lock to match runtime dependency/resolution changes introduced by the Prague revm hardening and toolchain alignment.
@jtcoolen
chore(evm): improve solc error surfacing
42c1649 
Capture solc stderr and include actionable diagnostics when bytecode generation fails or emits no output.
@jtcoolen
fix(transcript/evm): handle identity encoding and disjoint transcript…
728397d 
… memory

Harden EVM transcript absorption and decoding for identity points and non-contiguous memory sources.

- Re-copy disjoint memory-backed scalars/points into contiguous transcript buffer when required.

- Encode/decode point-at-infinity via EVM (0,0) sentinel consistently in native transcript flow.

- Add regression tests for identity encoding/decoding behavior.
@jtcoolen
perf(loader_evm): batch MSM precompile path and harden inversion sema…
f4e4942 
…ntics

Optimize and harden EVM loader arithmetic paths used by generated verifiers.

- Route multi-scalar multiplication through batched G1MSM precompile calls.

- Add reusable staticcall helper, identity-safe constant point encoding, and zero-preserving scalar negation.

- Rewrite batch inversion flow to be zero-safe for aliased inputs and deterministic in memory-backed execution.
@jtcoolen
chore(example): port standard_plonk to BLS12-381
0004143 
Keep the original arithmetization gate shape while moving the example flow to BLS12-381 params and EVM artifact generation/verification.
@jtcoolen jtcoolen force-pushed the julian@midnight-proof-support branch from f7e7216 to 17e162b Compare March 2, 2026 11:46
@jtcoolen
refactor(core): extract shared halo2 layout/expression helpers
31a4d58 
Factor reusable Halo2 layout and expression helpers so protocol builders can share remapping and selector logic.

- Add shared system::halo2::layout and system::halo2::expression modules.

- Switch Halo2 compiler internals to those helpers where metadata layout is unchanged.

- Replace iter::repeat_n with explicit repeat().take() and simplify native assertion control flow.
jtcoolen added 4 commits March 2, 2026 11:53
@jtcoolen
feat(plonk): add protocol metadata for transcript/layout variants
dec3323 
Extend PLONK protocol metadata so verifier flows can represent transcript/layout variants without overloading legacy fields.

- Add committed-instance and transcript extension fields on PlonkProtocol.

- Extend quotient metadata with chunk base and optional fixed chunk count.

- Wire Halo2 compiler defaults for the new metadata and re-export protocol types cleanly.
@jtcoolen
feat(plonk): support committed-instance aware proof parsing
8762fde 
Extend PLONK proof parsing so protocols can hash committed-instance prefixes, bind optional instance lengths, and parse transcript extensions.

- Add read_with_committed_instances entrypoint and shared committed-instance absorption flow.

- Parse trailing challenges and extra commitments declared by protocol metadata.

- Support ZnMinusOne quotient recombination and account for extra commitments in query indexing.
@jtcoolen
test(plonk): cover committed-instance transcript ordering
dd3b0cc 
Add deterministic unit tests for committed-instance transcript absorption and placeholder behavior.

- Verify committed-prefix plus hashed-length ordering in transcript writes.

- Verify deterministic errors for committed layout mismatches.

- Verify placeholder extension preserves declared instance-column layout.
@jtcoolen
feat(kzg): add Midnight multiopen PCS and truncated challenge mode
e342a31 
Add Midnight-compatible KZG multiopen verification and expose an optional truncated challenge policy used by Midnight transcript variants.

- Implement Midnight KZG PCS proof reader/verifier for native and EVM loaders.

- Re-export Midnight PCS flavor from kzg modules and add truncated-challenges feature flag.

- Add EVM scalar truncation helper used by truncated challenge verification flows.
@jtcoolen jtcoolen force-pushed the julian@midnight-proof-support branch from 17e162b to cbfb181 Compare March 2, 2026 11:54
jtcoolen added 10 commits March 2, 2026 12:00
@jtcoolen
build(sdk-midnight): wire Midnight feature gate and dependency graph
076394c 
Introduce the SDK plumbing required to compile Midnight adapter code behind an explicit feature flag.

- Add Midnight-related optional dependencies and feature wiring in snark-verifier-sdk/Cargo.toml.

- Re-export the Midnight adapter module from snark-verifier-sdk/src/lib.rs behind the Midnight feature gate.

- Refresh Cargo.lock to capture the resolved Midnight dependency tree used by subsequent adapter commits.
@jtcoolen
chore(sdk-midnight): seed protocol builder from halo2 baseline
6a4cd75 
Start the Midnight protocol adapter by copying the existing halo2 protocol compiler into a dedicated SDK module.

This commit is intentionally mechanical: it preserves the source behavior so follow-up commits can introduce Midnight-specific changes as isolated, reviewable deltas.

- Add snark-verifier-sdk/src/midnight_adapter/protocol_builder.rs as a direct baseline copy of snark-verifier/src/system/halo2.rs.
@jtcoolen
feat(sdk-midnight/protocol): retarget builder scaffold to Midnight types
a5cadaf 
Replace the copied halo2 compile/config shell with a Midnight-specific protocol builder skeleton.

- Switch imports and type parameters from generic halo2 primitives to Midnight VK/domain and halo conversion helpers.

- Introduce MidnightProtocolBuilder state for committed instances, phase remapping metadata, lookup/permutation dimensions, and trash column counts.

- Build PlonkProtocol with Midnight transcript metadata (committed_instance_count, hash_instance_lengths, transcript initial state).

- Remove multi-proof/config paths that are not part of the Midnight adapter flow.
@jtcoolen
feat(sdk-midnight/protocol): port query scheduling and witness indexing
eb2b0bb 
Port the query/index plumbing from the baseline compiler into Midnight layout semantics.

- Define Midnight witness offsets for advice, lookup permuted columns, permutation products, trash columns, random polynomial, and quotient polynomial.

- Rebuild committed-instance/advice/fixed/permutation/lookup query generation around Midnight column and phase metadata.

- Include trash witness columns in protocol query and evaluation order.

- Keep query ordering compatible with Midnight proof parsing and EVM codegen paths.
@jtcoolen
feat(sdk-midnight/protocol): finalize lookup/permutation math and quo…
1461a3d 
…tient settings

Complete the Midnight protocol builder transition for non-trash constraints and quotient recombination behavior.

- Normalize permutation and lookup constraint generation around Midnight field/domain types and strict zk-only checks.

- Switch helper APIs to explicit Result-based expression conversion paths and remove remaining generic halo2 compile scaffolding.

- Set quotient recombination to ZnMinusOne with Midnight domain chunk-count override.

- Keep quotient numerator as alpha-compressed gate+permutation+lookup constraints (trash integrated in a follow-up commit).
@jtcoolen
feat(sdk-midnight/protocol): add trash argument constraints to quotient
ae0c7da 
Model Midnight trashcans explicitly in the protocol builder and include them in quotient construction.

- Add trash_constraints() to compress each trashcan constraint vector with trash_challenge.

- Gate trash equalities by selector and bind them to dedicated trash witness evaluations.

- Fold trash constraints into the quotient numerator after gate/permutation/lookup constraints so verifier query/evaluation ordering remains aligned.
@jtcoolen
feat(sdk-midnight): add conversion layer between Midnight and halo2 c…
4415c6d 
…urve types

Introduce reusable conversion helpers used by the adapter bundle/protocol builder boundary.

- Add field conversion utilities for Midnight Fq/Fp/Fp2 into halo2 Fr/Fq/Fq2 with encoding-length validation.

- Add G1/G2 affine mapping helpers that preserve identity handling and coordinate validity checks.

- Add committed-instance normalization to enforce VK instance-column shape before proof parsing.

- Introduce midnight_adapter module exports and shared MidnightCommitment/HaloAs type aliases.
@jtcoolen
feat(sdk-midnight): add MidnightProofBundle artifact loader and verif…
9606ed0 
…ier flow

Provide a single typed container for Midnight params, VK, committed instances, scalar instances, and proof bytes.

- Add constructors from materialized VK and VK bytes (RawBytesUnchecked) with shared validation path.

- Build snark-verifier deciding key/protocol from Midnight artifacts, including committed-instance column expansion semantics.

- Add shared run_snark_verifier_flow helper for transcript parsing + verification across native/EVM paths.

- Add verifier-params G2 decoding helper and strict trailing-bytes checks for processed param encoding.
@jtcoolen
feat(sdk-midnight/evm): generate Solidity verifier, bytecode, and cal…
862dc8e 
…ldata

Expose EVM-facing helper methods on MidnightProofBundle to materialize deployable verifier artifacts.

- Build an EvmLoader by replaying Midnight proof parsing/verification with EVM transcript semantics.

- Expose Solidity source generation, deployment bytecode compilation, and calldata encoding helpers.

- Gate optional local revm deployment/call verification behind the revm feature.

- Ensure committed-instance points are loaded only when present to preserve runtime code shape.
@jtcoolen
feat(examples): add Midnight Poseidon EVM transcript end-to-end demo
3d38168 
Add an SDK example that exercises the Midnight adapter from proof artifacts to generated EVM verifier outputs.

- Add midnight_poseidon_evm example and transcript support helpers.

- Wire example output handling for Solidity, bytecode, and calldata artifacts.

- Update standard_plonk example wiring and gitignore entries required by generated assets.
@jtcoolen jtcoolen force-pushed the julian@midnight-proof-support branch from cbfb181 to 3d38168 Compare March 2, 2026 12:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@jtcoolen