Skip to content

Commit 7781028

Browse files
committed
ci(neon): aarch64 NEON SIMD parity gate under qemu (twin of wasm_simd)
The x86 `cargo test` suite never compiles `simd_neon.rs` (cfg'd to aarch64), so the native NEON SIMD lanes — including the new `U32x16` ARX lane the ChaCha20 backend rides — had no automated runtime verification. This closes that blind spot, mirroring the `wasm_simd`/node gate. - crates/neon-simd-parity — excluded bin that calls the REAL `ndarray::simd` aarch64 types (no copy → no drift) and self-checks each lane against scalar: U32x16 ARX (Add / BitXor / rotate_left over 0,1,7,8,12,16,24,31), F32x16 (splat / roundtrip / add / reduce_sum), I8x16 (roundtrip / add). Exit code points at the exact lane+op on mismatch. - scripts/neon-parity.sh — cross-build for aarch64-unknown-linux-gnu (via aarch64-linux-gnu-gcc) then run under qemu-aarch64-static. Exercises both integration (ndarray compiles for aarch64 on stable) and parity. - .github/workflows/ci.yaml — new `neon_simd` (neon-simd/parity-qemu) job installing gcc-aarch64-linux-gnu + qemu-user-static; added to `conclusion`. Verified GREEN locally: cross-built + run under qemu-aarch64-static → "U32x16 / F32x16 / I8x16 lanes bit-identical to scalar". Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01K6pT32kk6pnuAAqR3JiYqu
1 parent 83d2ca7 commit 7781028

8 files changed

Lines changed: 253 additions & 16 deletions

File tree

.claude/CHACHA20_MATRYOSHKA_PLAN.md

Lines changed: 17 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -81,21 +81,23 @@ below **superseded it** — it is now **RETIRED** (2026-07-12, see below).
8181
`[patch.crates-io] chacha20 = { path = "vendor/chacha20" }` pointing at a vendored
8282
copy of the fork, to inherit the acceleration. Documented, low-risk.
8383

84-
## DEFERRED — TO-DO (next session)
85-
86-
1. **Native neon `U32x16 = [U32x4; 4]`.** `simd_neon.rs` has native
87-
`U32x4(uint32x4_t)` with add/sub/min/max; add `bitxor` (`veorq_u32`) +
88-
`rotate_left` (shift-or via `vshlq_u32` with variable count vectors, `n%32`
89-
guard), then compose `U32x16([U32x4; 4])` with `impl Add`/`impl BitXor`/
90-
`rotate_left` (fan over 4 lanes, mirror the wasm shape in `simd_wasm.rs`), fix
91-
`F32x16::to_bits/from_bits` to `from_array`/`to_array`, and swap the `simd.rs`
92-
aarch64 arm's scalar `U32x16` re-export for the native one.
93-
2. **aarch64 cross parity CI.** Generalize `wasm-simd-parity` into a shared
94-
`simd-parity` harness (or add a sibling) run under **qemu** via `cross` in a
95-
new `neon_simd` CI job — same selfcheck, closing NEON's identical x86-suite
96-
blind spot (its `F32x16`/`I8x16` are unverified in CI today too).
97-
3. **avx2 native `U32x16`** (2×`__m256i`) — the TD-SIMD-3 lowering; optional, the
98-
scalar polyfill is correct meanwhile.
84+
## DONE / DEFERRED
85+
86+
1. ~~**Native neon `U32x16 = [U32x4; 4]`.**~~ **DONE 2026-07-12**`U32x4` gained
87+
`bitxor`(veorq_u32) + `rotate_left`(vshlq_u32 shift-or, n%32 guard); composed
88+
native `U32x16([U32x4;4])` with Add/BitXor/rotate_left; `F32x16::to_bits/
89+
from_bits` → from_array/to_array; `simd.rs` aarch64 arm re-exports the native
90+
lane. (+ fixed the pre-existing aarch64 stable-compile breakage.)
91+
2. ~~**aarch64 cross parity CI.**~~ **DONE 2026-07-12**`crates/neon-simd-parity`
92+
(excluded bin, real `ndarray::simd` aarch64 types) + `scripts/neon-parity.sh`
93+
(cross-build aarch64 + run under `qemu-aarch64-static`) + CI `neon_simd`
94+
(`neon-simd/parity-qemu`) job, added to the `conclusion` needs. Runtime-verifies
95+
U32x16 ARX (rotate 16/12/8/7 + edges) / F32x16 / I8x16 == scalar. Twin of
96+
`wasm_simd`. **Green locally under qemu.**
97+
3. **avx2 native `U32x16`** (2×`__m256i`) — the TD-SIMD-3 lowering; still optional,
98+
the scalar polyfill is correct meanwhile.
99+
4. **wasm matryoshka backend + cross-repo `[patch]`** — see the SHIPPED section's
100+
follow-ups above.
99101

100102
## The MATRYOSHKA — how the lane gets USED (the finalization)
101103

.claude/blackboard.md

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -654,3 +654,28 @@ Green jobs of note: `tests/{stable,beta,1.95.0}`, `blas-msrv`, `nostd/thumbv6m-n
654654

655655
Deferred (task #30, token-limit call): native neon `U32x16=[U32x4;4]` + aarch64 cross
656656
parity CI + the matryoshka chacha20 fork. Plan in `.claude/CHACHA20_MATRYOSHKA_PLAN.md`.
657+
658+
---
659+
660+
## 2026-07-12 — Matryoshka finalized + NEON cross-CI (deferred #30 CLOSED)
661+
662+
- **Native NEON `U32x16 = [U32x4;4]` ARX lane** (commit 06a61bf9): bitxor/
663+
rotate_left on U32x4, composed U32x16 Add/BitXor/rotate_left; also fixed the
664+
pre-existing aarch64 stable-compile breakage (`u16x8` alias; nightly-only
665+
`vdotq_s32` → stable widening NEON). aarch64 now `cargo check`-clean on stable.
666+
- **ChaCha20 matryoshka + `simd_crypto.rs` RETIRED** (commit 20dc6c3f):
667+
`vendor/chacha20/` fork (name/version kept, own [workspace]); the ONE delta is
668+
`backends/ndarray_simd.rs` — the transpose block16 over `ndarray::simd::U32x16`
669+
(pure +/^/rotate_left, no intrinsics, no unsafe), compile-time-selected under
670+
cfg(x86_64+avx512f), `[patch.crates-io]`-folded under encryption. Triple parity
671+
gate GREEN (fork RFC 8439 vectors through ndarray_simd @ v4; encryption 23 AEAD
672+
tests @ v3+v4). Deleted src/simd_crypto.rs + the parity test + the chacha20 dev-dep
673+
+ the ndarray::simd::chacha20_* surface.
674+
- **NEON cross parity CI**: `crates/neon-simd-parity` (excluded bin) +
675+
`scripts/neon-parity.sh` (cross-build aarch64 + run under qemu-aarch64-static) +
676+
CI `neon_simd` job (added to conclusion needs). Runtime-verifies U32x16 ARX /
677+
F32x16 / I8x16 == scalar on real aarch64. Green locally under qemu.
678+
679+
Follow-ups (documented in `.claude/CHACHA20_MATRYOSHKA_PLAN.md`): wasm matryoshka
680+
backend (simd128 branch); cross-repo `[patch]` for MedCare-rs; the workspace
681+
default is x86-64-v3 (avx2) so ndarray_simd activates on avx512 builds only.

.github/workflows/ci.yaml

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -135,6 +135,28 @@ jobs:
135135
- name: wasm SIMD parity (build + node run)
136136
run: ./scripts/wasm-parity.sh
137137

138+
neon_simd:
139+
# The x86 `cargo test` suite never touches `simd_neon.rs` (it is cfg'd to
140+
# aarch64), so NEON SIMD correctness is otherwise unverified. Cross-build the
141+
# real `ndarray::simd` aarch64 types via the excluded `neon-simd-parity` bin
142+
# and run its numeric selfcheck under qemu — both integration (the lib
143+
# compiles for aarch64 on stable) and parity (each lane == scalar). The
144+
# aarch64 twin of the wasm_simd job above.
145+
runs-on: ubuntu-latest
146+
name: neon-simd/parity-qemu
147+
steps:
148+
- uses: actions/checkout@v4
149+
- uses: dtolnay/rust-toolchain@stable
150+
with:
151+
targets: aarch64-unknown-linux-gnu
152+
# rust-toolchain.toml pins 1.95.0 — install the aarch64 target for the
153+
# pinned toolchain too (dtolnay installs for `stable`, which may differ).
154+
- run: rustup target add aarch64-unknown-linux-gnu
155+
- name: install aarch64 cross toolchain + qemu-user
156+
run: sudo apt-get update && sudo apt-get install -y gcc-aarch64-linux-gnu qemu-user-static
157+
- name: NEON SIMD parity (cross-build + qemu run)
158+
run: ./scripts/neon-parity.sh
159+
138160
tests:
139161
runs-on: ubuntu-latest
140162
needs: pass-msrv
@@ -359,6 +381,7 @@ jobs:
359381
- format
360382
- nostd
361383
- wasm_simd
384+
- neon_simd
362385
- tests
363386
- native-backend
364387
- hpc-stream-parallel

Cargo.toml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -402,7 +402,7 @@ members = [
402402
"ndarray-rand",
403403
"crates/*",
404404
]
405-
exclude = ["crates/burn", "crates/wasm-simd-parity", "vendor/chacha20"]
405+
exclude = ["crates/burn", "crates/wasm-simd-parity", "crates/neon-simd-parity", "vendor/chacha20"]
406406
default-members = [
407407
".",
408408
"ndarray-rand",

crates/neon-simd-parity/.gitignore

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
/target
2+
Cargo.lock

crates/neon-simd-parity/Cargo.toml

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
# neon-simd-parity — the run-under-qemu gate for the aarch64 NEON SIMD tier.
2+
#
3+
# The x86 `cargo test` suite never touches `simd_neon.rs` (it is `cfg`'d to
4+
# aarch64), so NEON SIMD correctness is otherwise unverified by CI — exactly the
5+
# blind spot `wasm-simd-parity` closes for wasm. This bin calls the REAL
6+
# `ndarray::simd` aarch64 types (no copy → no drift) and self-checks each lane's
7+
# arithmetic against scalar in-process; it is cross-built for
8+
# `aarch64-unknown-linux-gnu` and executed under `qemu-aarch64-static`, exiting
9+
# non-zero on any lane mismatch.
10+
#
11+
# EXCLUDED from the workspace (see root Cargo.toml `exclude`) so it has zero
12+
# effect on the x86 jobs; the CI `neon_simd` job builds it via --manifest-path.
13+
[package]
14+
name = "neon-simd-parity"
15+
version = "0.0.0"
16+
edition = "2021"
17+
publish = false
18+
19+
[[bin]]
20+
name = "neon-simd-parity"
21+
path = "src/main.rs"
22+
23+
[dependencies]
24+
# `simd` needs only ndarray's `std` feature, not the full HPC default set.
25+
ndarray = { path = "../..", default-features = false, features = ["std"] }
26+
27+
[profile.release]
28+
panic = "abort"
29+
opt-level = "z"
Lines changed: 123 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,123 @@
1+
//! Run-under-qemu parity gate for the aarch64 NEON SIMD tier.
2+
//!
3+
//! `selfcheck()` runs each NEON SIMD lane's arithmetic against the scalar
4+
//! `u32`/`f32`/`i8` reference **in the same process**, returning `0` iff every
5+
//! lane is bit-identical. `scripts/neon-parity.sh` cross-builds this bin for
6+
//! `aarch64-unknown-linux-gnu` and runs it under `qemu-aarch64-static`, asserting
7+
//! exit code `0`; the CI `neon_simd` job runs the script. This is the NEON twin
8+
//! of `wasm-simd-parity` (the wasm tier's node gate) — it tests the REAL
9+
//! `ndarray::simd` aarch64 types, so a regression in `simd_neon.rs` fails here
10+
//! even though the x86 `cargo test` suite never compiles that file. Extend the
11+
//! per-lane blocks below whenever a new `ndarray::simd` NEON lane lands.
12+
13+
fn main() {
14+
#[cfg(target_arch = "aarch64")]
15+
{
16+
let rc = checks::selfcheck();
17+
if rc != 0 {
18+
eprintln!("neon-simd-parity FAILED: lane/op code = {rc}");
19+
std::process::exit(rc as i32);
20+
}
21+
println!("neon-simd-parity OK: U32x16 / F32x16 / I8x16 lanes bit-identical to scalar");
22+
}
23+
#[cfg(not(target_arch = "aarch64"))]
24+
{
25+
eprintln!("neon-simd-parity is aarch64-only; nothing to check on this target");
26+
}
27+
}
28+
29+
#[cfg(target_arch = "aarch64")]
30+
mod checks {
31+
use ndarray::simd::{F32x16, I8x16, U32x16};
32+
33+
/// Distinct nonzero return codes make a CI failure point at the exact lane+op.
34+
pub fn selfcheck() -> u32 {
35+
if let Err(code) = check_u32x16() {
36+
return code;
37+
}
38+
if let Err(code) = check_f32x16() {
39+
return code;
40+
}
41+
if let Err(code) = check_i8x16() {
42+
return code;
43+
}
44+
0
45+
}
46+
47+
/// `U32x16` ARX triple (Add / BitXor / rotate_left) — the ChaCha20/BLAKE lane
48+
/// (native `[U32x4; 4]` on aarch64).
49+
fn check_u32x16() -> Result<(), u32> {
50+
let a_arr: [u32; 16] = [
51+
0x0000_0000, 0xFFFF_FFFF, 0x0000_0001, 0x8000_0000, 0x1234_5678, 0x9ABC_DEF0, 0xDEAD_BEEF, 0xCAFE_BABE,
52+
0x0F0F_0F0F, 0xF0F0_F0F0, 0x5555_5555, 0xAAAA_AAAA, 0x0000_00FF, 0xFF00_0000, 0x0101_0101, 0x8080_8080,
53+
];
54+
let b_arr: [u32; 16] = [
55+
0x9E37_79B9, 0x1111_1111, 0xDEAD_C0DE, 0x0BAD_F00D, 0x7FFF_FFFF, 0x0000_0000, 0xFFFF_FFFF, 0x1357_9BDF,
56+
0x2468_ACE0, 0xFEDC_BA98, 0x0000_0010, 0x0000_001F, 0xABCD_EF01, 0x1020_4080, 0x0F0F_F0F0, 0xC0DE_CAFE,
57+
];
58+
if U32x16::from_array(a_arr).to_array() != a_arr {
59+
return Err(10);
60+
}
61+
let a = U32x16::from_array(a_arr);
62+
let b = U32x16::from_array(b_arr);
63+
let add = (a + b).to_array();
64+
let xor = (a ^ b).to_array();
65+
for i in 0..16 {
66+
if add[i] != a_arr[i].wrapping_add(b_arr[i]) {
67+
return Err(11);
68+
}
69+
if xor[i] != a_arr[i] ^ b_arr[i] {
70+
return Err(12);
71+
}
72+
}
73+
// ARX rotate — ChaCha20 uses 16/12/8/7; edges (0, 1, 24, 31) included.
74+
for &n in &[0u32, 1, 7, 8, 12, 16, 24, 31] {
75+
let r = a.rotate_left(n).to_array();
76+
for i in 0..16 {
77+
if r[i] != a_arr[i].rotate_left(n) {
78+
return Err(13);
79+
}
80+
}
81+
}
82+
Ok(())
83+
}
84+
85+
/// `F32x16` — the float hot-path lane (splat / roundtrip / add / reduce_sum).
86+
fn check_f32x16() -> Result<(), u32> {
87+
let data: [f32; 16] = [
88+
0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0, 10.0, 11.0, 12.0, 13.0, 14.0, 15.0,
89+
];
90+
if F32x16::from_array(data).to_array() != data {
91+
return Err(20);
92+
}
93+
let a = F32x16::from_array(data);
94+
let b = F32x16::splat(2.0);
95+
let sum = (a + b).to_array();
96+
for i in 0..16 {
97+
if sum[i] != data[i] + 2.0 {
98+
return Err(21);
99+
}
100+
}
101+
// 0+1+…+15 = 120.
102+
if F32x16::from_array(data).reduce_sum() != 120.0 {
103+
return Err(22);
104+
}
105+
Ok(())
106+
}
107+
108+
/// `I8x16` — the byte lane (roundtrip / add).
109+
fn check_i8x16() -> Result<(), u32> {
110+
let a_arr: [i8; 16] = [-128, -1, 0, 1, 127, 2, -2, 3, -3, 42, -42, 100, -100, 7, -7, 64];
111+
let b_arr: [i8; 16] = [1, 1, 1, 1, 1, -1, -1, -1, 5, -5, 10, -10, 25, -25, 63, -64];
112+
if I8x16::from_array(a_arr).to_array() != a_arr {
113+
return Err(30);
114+
}
115+
let sum = I8x16::from_array(a_arr).add(I8x16::from_array(b_arr)).to_array();
116+
for i in 0..16 {
117+
if sum[i] != a_arr[i].wrapping_add(b_arr[i]) {
118+
return Err(31);
119+
}
120+
}
121+
Ok(())
122+
}
123+
}

scripts/neon-parity.sh

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
#!/usr/bin/env bash
2+
# NEON SIMD parity gate — cross-build the real ndarray::simd aarch64 types (via
3+
# the `neon-simd-parity` bin) for aarch64-unknown-linux-gnu, then run the numeric
4+
# selfcheck under qemu-aarch64. Exercises BOTH:
5+
# - correctness/integration: the full ndarray lib compiles for aarch64 on stable
6+
# (the harness path-deps ndarray, so building it builds ndarray for aarch64);
7+
# - parity: each NEON SIMD lane (U32x16 ARX / F32x16 / I8x16) is bit-identical
8+
# to scalar, run under qemu.
9+
# The x86 `cargo test` suite never touches simd_neon.rs, so this is the only
10+
# automated guard for the NEON SIMD tier (the twin of scripts/wasm-parity.sh).
11+
#
12+
# Requirements (Ubuntu CI): gcc-aarch64-linux-gnu (cross linker + sysroot),
13+
# qemu-user-static (qemu-aarch64-static), and `rustup target add
14+
# aarch64-unknown-linux-gnu`.
15+
set -euo pipefail
16+
17+
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
18+
MANIFEST="$ROOT/crates/neon-simd-parity/Cargo.toml"
19+
TARGET="aarch64-unknown-linux-gnu"
20+
QEMU="${QEMU_AARCH64:-qemu-aarch64-static}"
21+
SYSROOT="${AARCH64_SYSROOT:-/usr/aarch64-linux-gnu}"
22+
23+
echo "==> building neon-simd-parity (ndarray::simd aarch64 types) for $TARGET"
24+
CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER="aarch64-linux-gnu-gcc" \
25+
cargo build --release --manifest-path "$MANIFEST" --target "$TARGET"
26+
27+
BIN="$ROOT/crates/neon-simd-parity/target/$TARGET/release/neon-simd-parity"
28+
# Excluded crate → its own target dir under the crate; fall back to workspace target.
29+
[ -f "$BIN" ] || BIN="$ROOT/target/$TARGET/release/neon-simd-parity"
30+
31+
echo "==> running selfcheck under $QEMU (sysroot $SYSROOT): $BIN"
32+
"$QEMU" -L "$SYSROOT" "$BIN"
33+
echo "==> NEON SIMD parity: PASS"

0 commit comments

Comments
 (0)