Skip to content

perf(scan): validate_brackets fusion in AVX2 and NEON scanners #25

@membphis

Description

@membphis

Background

The scalar scanner (src/scan/scalar.rs) already uses the fused scan_and_validate path that carries a depth stack inline and validates bracket balance in a single pass, eliminating the separate validate_brackets call.

The AVX2 and NEON scanners still run the two-pass design: emit structural offsets, then call validate_brackets over the emitted indices.

A working implementation was prototyped in PR #18 (closed without merge): emit_bits_validate carries a depth stack and dispatches on buf[pos] per emitted bit.

Why it was deferred

Measured ±2% (within noise) on the multimodal bench because:

  • The per-emit buf[pos] lookup adds back roughly what the eliminated pass saved.
  • Structural-char density is too low on string-heavy payloads for savings to dominate.

When to revisit

This becomes worth implementing once a structurally-dense bench fixture (config JSON, JSONL, object-shape JSON with hundreds of keys per chunk) is added to the harness and profiling shows validate_brackets as a measurable cost. See issue #24 for the fixture work.

Implementation notes

  • Reference the emit_bits_validate prototype in PR perf(scan): fuse validate_brackets into SIMD emit loops #18 for the depth-stack approach.
  • Gate the change on a green cargo test --no-default-features run (scalar path must still pass).
  • The proptest cross-check in tests/scanner_crosscheck.rs is the main correctness gate.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions