You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The scalar scanner (src/scan/scalar.rs) already uses the fused scan_and_validate path that carries a depth stack inline and validates bracket balance in a single pass, eliminating the separate validate_brackets call.
The AVX2 and NEON scanners still run the two-pass design: emit structural offsets, then call validate_brackets over the emitted indices.
A working implementation was prototyped in PR #18 (closed without merge): emit_bits_validate carries a depth stack and dispatches on buf[pos] per emitted bit.
Why it was deferred
Measured ±2% (within noise) on the multimodal bench because:
The per-emit buf[pos] lookup adds back roughly what the eliminated pass saved.
Structural-char density is too low on string-heavy payloads for savings to dominate.
When to revisit
This becomes worth implementing once a structurally-dense bench fixture (config JSON, JSONL, object-shape JSON with hundreds of keys per chunk) is added to the harness and profiling shows validate_brackets as a measurable cost. See issue #24 for the fixture work.
Background
The scalar scanner (
src/scan/scalar.rs) already uses the fusedscan_and_validatepath that carries a depth stack inline and validates bracket balance in a single pass, eliminating the separatevalidate_bracketscall.The AVX2 and NEON scanners still run the two-pass design: emit structural offsets, then call
validate_bracketsover the emittedindices.A working implementation was prototyped in PR #18 (closed without merge):
emit_bits_validatecarries a depth stack and dispatches onbuf[pos]per emitted bit.Why it was deferred
Measured ±2% (within noise) on the multimodal bench because:
buf[pos]lookup adds back roughly what the eliminated pass saved.When to revisit
This becomes worth implementing once a structurally-dense bench fixture (config JSON, JSONL, object-shape JSON with hundreds of keys per chunk) is added to the harness and profiling shows
validate_bracketsas a measurable cost. See issue #24 for the fixture work.Implementation notes
emit_bits_validateprototype in PR perf(scan): fuse validate_brackets into SIMD emit loops #18 for the depth-stack approach.cargo test --no-default-featuresrun (scalar path must still pass).tests/scanner_crosscheck.rsis the main correctness gate.