[codex] Fix v0.5 release audit blockers

[Fieldnote-Echo]

Summary

• harden .ovfs FastScan load/write validation: invalid nibbles, b=2 composition violations, and nonzero block-tail padding now fail before safe search can observe the bytes
• make SDE-backed AVX-512 CI/coverage fail closed for push/release-gated runs while allowing PR-only outage warnings, and update release invariants so the SDE probe/tests cannot green-skip on main
• bound calibration-profile hashing in ordvec-manifest, align verification/cache reads to canonical paths, and mark persisted-format metadata enums non-exhaustive before v0.5 ships
• replace manifest metadata wildcard panics with fallible conversion so future unsupported core metadata reports artifact_kind_unsupported / artifact_params_unsupported instead of panicking
• consolidate the v0.5 changelog and document the b=8 boundary, .ovfs manifest/probe gap, FastScan dispatch reality, and Python/persistence limitations

Notes

• The trusted-publisher and environment findings were already handled outside this PR: crates-io and pypi environments audit clean for Project-Navi/ordvec.
• ordvec-manifest package/dry-run still must wait until ordvec 0.5.0 is present in the crates.io index; cargo package -p ordvec-manifest --locked currently fails for that expected dependency-resolution reason.
• cargo test --workspace --all-features --locked still cannot be used directly for the PyO3 extension crates in this local environment; it fails while linking extension-module tests as binaries with missing PyObject_* symbols. Python validation was run through maturin develop instead.

Validation

• cargo fmt --all -- --check
• cargo clippy --workspace --all-targets --all-features --locked -- -D warnings
• cargo test fastscan --locked
• cargo test --test index fastscan --locked
• cargo test -p ordvec-manifest --locked
• cargo test -p ordvec-manifest --no-default-features --locked
• cargo test -p ordvec-manifest --all-features --locked
• cargo build -p ordvec-ffi --locked
• cargo build -p ordvec-ffi --all-features --locked
• cargo test -p ordvec-ffi --test c_link_smoke --locked
• cargo test --workspace --all-features --locked --exclude ordvec-python --exclude ordvec-manifest-python
• python3 tests/release_publish_invariants.py
• bash tests/release_publish_invariants.sh
• bash tests/release_signed_release_invariants.sh
• REPO=Project-Navi/ordvec bash tests/release_environment_settings.sh
• cargo deny check
• cargo package -p ordvec --locked
• cargo publish -p ordvec --dry-run --locked
• pinned local workflow lint: go install github.com/rhysd/actionlint/cmd/actionlint@v1.7.12 then actionlint .github/workflows/ci.yml .github/workflows/coverage.yml
• temp venv: maturin develop --manifest-path ordvec-python/Cargo.toml
• temp venv: maturin develop --manifest-path ordvec-manifest-python/Cargo.toml
• temp venv: Python probe confirming ordvec.RankQuant(64, 8) raises ValueError

Focused Qodo follow-up validation

• cargo fmt --all -- --check
• cargo test -p ordvec-manifest --locked
• cargo test -p ordvec-manifest --no-default-features --locked
• cargo test -p ordvec-manifest --all-features --locked
• cargo clippy -p ordvec-manifest --all-targets --all-features --locked -- -D warnings
• git diff --check

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[qodo-code-review]

PR Summary by Qodo

Fail-closed SDE gating and harden FastScan/manifest validation for v0.5
🐞 Bug fix 🧪 Tests 📝 Documentation ⚙️ Configuration changes 🕐 40+ Minutes

Description

• Harden .ovfs FastScan load/write validation to reject malformed payloads before search.
• Make Intel SDE-backed AVX-512 CI and coverage jobs fail closed for release gating.
• Bound and canonicalize ordvec-manifest verification hashing and document v0.5 limits.

Diagram

graph TD
  CI["CI / Coverage workflows"] --> SDE[["setup-intel-sde"]] --> AVX["AVX-512 tests + coverage"]
  FUZZ["Fuzz + index tests"] --> CORE["ordvec FastScan (.ovfs)"]
  CORE --> IO["rank_io (.ovfs validate)"]
  MAN["ordvec-manifest verify"] --> IO --> DB[("SQLite cache key")]
  IO --> FFI["ordvec-ffi ABI v1"]

  subgraph Legend
    direction LR
    _wf["Workflow"] ~~~ _act[["Action"]] ~~~ _db[("Database")]
  end

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Self-hosted AVX-512 runner instead of Intel SDE

• ➕ Removes dependence on Intel download availability/WAF behavior
• ➕ Runs on real hardware for higher fidelity performance/correctness signals
• ➖ Higher operational/security burden (runner hardening, patching, isolation)
• ➖ May reduce contributor accessibility compared to GitHub-hosted runners

2. Mirror/pin Intel SDE artifact in a project-controlled bucket

• ➕ Keeps SDE-based approach while reducing third-party outage risk
• ➕ Preserves current job design and CPUID-probe semantics
• ➖ Introduces artifact distribution responsibilities and legal/licensing considerations
• ➖ Still relies on emulation rather than real AVX-512 hardware

3. Make `.ovfs` payload validation optional (strict mode)

• ➕ Avoids O(payload) validation cost for trusted, high-throughput local loads
• ➖ Weakens the security posture for a persisted format at first stable release
• ➖ Adds API complexity and increases the chance callers accidentally run non-strict loads

Recommendation: Keep the PR’s fail-closed SDE gating and always-on .ovfs validation for v0.5: they align with a security-first persisted-format contract and prevent silent coverage/test gaps. If Intel SDE availability remains a recurring release risk, prefer adding an internal mirror or a dedicated self-hosted AVX-512 lane rather than reintroducing green-skips.

Files changed (21) +465 / -135

Bug fix (4) +149 / -39

lib.rsFail gracefully on unknown IndexKind/IndexParams in ABI v1 +12/-0

Fail gracefully on unknown IndexKind/IndexParams in ABI v1

• Adds wildcard match arms so metadata probing returns an explicit unsupported-format error for future index kinds/parameters, avoiding accidental panics or misclassification as the core enums evolve.

ordvec-ffi/src/lib.rs

lib.rsCanonical-path hashing and bounded calibration-profile hashing +23/-15

Canonical-path hashing and bounded calibration-profile hashing

• Hashes and probes canonical paths (artifact, row identity, auxiliary artifacts) to eliminate path-reopen drift. Introduces 'max_calibration_profile_bytes' with bounded hashing and propagates the new limit into resource limits and limit-code classification; marks manifest metadata enums '#[non_exhaustive]'.

ordvec-manifest/src/lib.rs

sqlite.rsUse canonical paths and bounded calibration hashing in cache keys +9/-4

Use canonical paths and bounded calibration hashing in cache keys

• Builds cache keys by hashing the canonical artifact and row-identity paths, matching verifier behavior. Computes calibration profile SHA-256 with bounded hashing using the new calibration size limit.

ordvec-manifest/src/sqlite.rs

rank_io.rsValidate '.ovfs' payload bytes on load and before write +105/-20

Validate '.ovfs' payload bytes on load and before write

• Introduces 'validate_fastscan_payload()' to reject invalid FastScan nibbles, b=2 constant-composition violations, and non-zero tail padding. Runs this validation in both '.ovfs' load and write paths, with new tests ensuring rejected writes do not create/truncate files.

src/rank_io.rs

Tests (5) +206 / -20

load_fastscan.rsRequire safe search after successful '.ovfs' load in fuzz target +8/-5

Require safe search after successful '.ovfs' load in fuzz target

• Strengthens the fuzzing contract: if the loader returns 'Ok', the target performs a safe 'search()' to ensure accepted bytes remain safe through the public scan path.

fuzz/fuzz_targets/load_fastscan.rs

manifest.rsTest calibration size limits and sqlite cache-key behavior +33/-0

Test calibration size limits and sqlite cache-key behavior

• Adds regression tests ensuring calibration profile hashing respects 'max_calibration_profile_bytes' and produces 'calibration_profile_too_large'. Extends sqlite cache tests to validate the same behavior through cached verification paths.

ordvec-manifest/tests/manifest.rs

fastscan.rsDocument '.ovfs' invariants and test dispatch vs scalar reference +82/-5

Document '.ovfs' invariants and test dispatch vs scalar reference

• Updates FastScan module and persistence docs to reflect AVX-512→scalar dispatch and '.ovfs' payload requirements. Adds a unit test that round-trips persisted bytes and asserts the dispatched search matches a scalar reference implementation exactly.

src/fastscan.rs

fastscan.rsAdd '.ovfs' negative tests across all public loaders +72/-2

Add '.ovfs' negative tests across all public loaders

• Adds helpers to forge '.ovfs' bytes and asserts that path/reader/byte-slice load APIs reject invalid nibble bytes, non-zero tail padding, and composition violations. Makes temp filenames more collision-resistant by adding a timestamp nonce.

tests/index/fastscan.rs

release_publish_invariants.pyRequire SDE jobs to fail closed and disallow green-skips +11/-8

Require SDE jobs to fail closed and disallow green-skips

• Updates release invariant checks to require 'allow-unavailable: false' for Intel SDE setup. Ensures SDE-dependent steps are not conditionally skipped and forbids emitting outage notices that would mask missing AVX-512 coverage.

tests/release_publish_invariants.py

Documentation (9) +99 / -58

CHANGELOG.mdConsolidate v0.5.0 changelog and document release hardening +46/-36

Consolidate v0.5.0 changelog and document release hardening

• Rolls forward the v0.5.0 entry date and consolidates prior notes into a single release section. Adds explicit release notes for '.ovfs' validation hardening, SDE fail-closed gating, canonical-path hashing fixes, bounded calibration hashing, and non-exhaustive metadata enums.

CHANGELOG.md

README.mdClarify b=8 boundary and FastScan dispatch behavior +9/-4

Clarify b=8 boundary and FastScan dispatch behavior

• Documents that 'b=8' is Rust-only/in-memory in v0.5.0 and not persisted or exposed via Python. Corrects FastScan dispatch to AVX-512→scalar (not AVX2) and aligns high-level API descriptions with the actual behavior.

README.md

THREAT_MODEL.mdExtend persisted-input invariants and fuzzing expectations +9/-4

Extend persisted-input invariants and fuzzing expectations

• Adds '.ovfs' structural invariants (valid nibbles, b=2 constant composition, zero tail padding) to the threat model. Updates FastScan positioning and documents that fuzzing now executes a safe 'search()' after successful loads.

THREAT_MODEL.md

INDEX_PROVENANCE.mdDocument calibration byte ceiling and '.ovfs' manifest/probe gap +9/-4

Document calibration byte ceiling and '.ovfs' manifest/probe gap

• Updates verifier documentation to include calibration-profile byte ceilings. Clarifies that manifest v1 still does not bind/probe '.ovfs', while the direct loader enforces payload validity.

docs/INDEX_PROVENANCE.md

PERSISTED_FORMAT.mdState '.ovfs' loader validation guarantees +6/-2

State '.ovfs' loader validation guarantees

• Documents that direct '.ovfs' loads validate nibble domain, b=2 constant composition, and tail padding before any search path observes bytes. Reinforces that metadata probing rejects OVFS until a later contract promotion.

docs/PERSISTED_FORMAT.md

README.mdAdd calibration profile size limit and clarify hashing model +8/-4

Add calibration profile size limit and clarify hashing model

• Documents a stable 64 MiB calibration-profile artifact limit and the corresponding CLI override flag. Clarifies that calibration/encoder profile over-limit artifacts are treated as non-cacheable and force re-verification.

ordvec-manifest/README.md

README.mdDocument that b=8 RankQuant is not exposed in Python v0.5 +4/-0

Document that b=8 RankQuant is not exposed in Python v0.5

• Adds an explicit note that Python 'RankQuant' supports only bits 1/2/4 in v0.5.0 and cannot persist or construct 'b=8'.

ordvec-python/README.md

lib.rsCorrect top-level FastScan dispatch documentation +3/-3

Correct top-level FastScan dispatch documentation

• Updates crate-level docs and comments to state that FastScan dispatch is AVX-512→scalar (not AVX2). Keeps guidance that FastScan is specialized relative to the main RankQuant/Bitmap surfaces.

src/lib.rs

quant.rsClarify b=8 memory cost and cross-language/persistence constraints +5/-1

Clarify b=8 memory cost and cross-language/persistence constraints

• Expands 'RankQuant' documentation with the prepared-query LUT memory footprint for 'b=8' and reiterates that v0.5.0 Python and '.ovrq' persistence support only bits 1/2/4.

src/quant.rs

Other (3) +11 / -18

ci.ymlMake AVX-512 SDE CI job fail closed +5/-10

Make AVX-512 SDE CI job fail closed

• Removes the temporary SDE-unavailable escape hatch so the avx512 job must successfully install SDE, run the CPUID probe, and execute AVX-512 tests. Updates job commentary to reflect release gating expectations.

.github/workflows/ci.yml

coverage.ymlFail closed for SDE-backed coverage generation +1/-8

Fail closed for SDE-backed coverage generation

• Requires Intel SDE install to succeed and removes conditional skipping of llvm-cov and upload steps. Ensures coverage floor enforcement runs under SDE rather than being green-skipped.

.github/workflows/coverage.yml

main.rsExpose CLI override for calibration profile byte ceiling +5/-0

Expose CLI override for calibration profile byte ceiling

• Adds '--max-calibration-profile-bytes' and wires it into 'VerifyOptions' limit overrides to match library-level controls.

ordvec-manifest/src/main.rs

[qodo-code-review]

Code Review by Qodo

🐞 Bugs (1) 📘 Rule violations (0) 📜 Skill insights (0)

1. Unreachable panics on new variants 🐞 Bug ☼ Reliability

Description

ordvec-manifest uses unreachable!() in ManifestIndexKind::from_core /
ManifestIndexParams::from_core, so if ordvec::probe_index_metadata() ever returns a newly-added
IndexKind/IndexParams variant (now #[non_exhaustive]), manifest verification will panic
instead of reporting an unsupported-format error.

Code

ordvec-manifest/src/lib.rs[R2742-2755]

            CoreIndexKind::RankQuant => Self::RankQuant,
            CoreIndexKind::Bitmap => Self::Bitmap,
            CoreIndexKind::SignBitmap => Self::SignBitmap,
+            _ => unreachable!("unsupported ordvec index kind from current metadata probe"),
        }
    }
}

#[derive(Copy, Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
#[serde(tag = "kind", rename_all = "snake_case", deny_unknown_fields)]
+#[non_exhaustive]
pub enum ManifestIndexParams {
    Rank,
    RankQuant { bits: u8 },

Evidence

The ordvec metadata enums are now explicitly open to future expansion (#[non_exhaustive]).
ordvec-manifest imports these enums from ordvec and currently panics on any value outside the known
set via unreachable!(), whereas other consumers (FFI) already fail gracefully on unknown variants.

src/rank_io.rs[93-111]
ordvec-manifest/src/lib.rs[17-21]
ordvec-manifest/src/lib.rs[2728-2769]
ordvec-ffi/src/lib.rs[365-402]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`ManifestIndexKind::from_core` and `ManifestIndexParams::from_core` currently handle the wildcard (`_`) arm with `unreachable!()`. Since `ordvec::IndexKind` / `ordvec::IndexParams` are now `#[non_exhaustive]`, a future ordvec version may legitimately return new variants from `probe_index_metadata()`, at which point `ordvec-manifest` would panic during verification.

## Issue Context
This is a forward-compatibility/robustness issue: the `#[non_exhaustive]` change explicitly signals that callers must not assume the variant set is closed.

## Fix Focus Areas
- ordvec-manifest/src/lib.rs[2738-2769]

## Expected fix
- Replace `unreachable!()` with a non-panicking path:
 - Option A (preferred): make these conversions fallible (`try_from_core(...) -> Result<..., ...>`) and plumb an explicit verifier error code like `artifact_kind_unsupported` / `artifact_params_unsupported`.
 - Option B: introduce an internal/serialized `Unsupported`/`Unknown` variant for `ManifestIndexKind`/`ManifestIndexParams` and have verification mark the report as failed with a clear error.
- Ensure the verifier continues returning a `VerificationReport` rather than panicking when it encounters an unsupported metadata kind/params.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

[Fieldnote-Echo]

qodo please re-review this PR. I addressed your finding by replacing the non-exhaustive manifest metadata wildcard unreachable!() arms with fallible conversion: verification now reports artifact_kind_unsupported / artifact_params_unsupported instead of panicking, and manifest creation returns ManifestError for unsupported future metadata. Focused validation passed: cargo fmt --all -- --check, cargo test -p ordvec-manifest --locked, cargo test -p ordvec-manifest --no-default-features --locked, cargo test -p ordvec-manifest --all-features --locked, and cargo clippy -p ordvec-manifest --all-targets --all-features --locked -- -D warnings.

[qodo-code-review]

please re-review this PR. I addressed your finding by replacing the non-exhaustive manifest metadata wildcard unreachable!() arms with fallible conversion: verification now reports artifact_kind_unsupported / artifact_params_unsupported instead of panicking, and manifest creation returns ManifestError for unsupported future metadata. Focused validation passed: cargo fmt --all -- --check, cargo test -p ordvec-manifest --locked, cargo test -p ordvec-manifest --no-default-features --locked, cargo test -p ordvec-manifest --all-features --locked, and cargo clippy -p ordvec-manifest --all-targets --all-features --locked -- -D warnings.

Yes — finding 1 looks addressed.

What changed

• The manifest metadata conversions are now fallible instead of using wildcard unreachable!() arms.
• Verification now reports artifact_kind_unsupported / artifact_params_unsupported for future metadata variants.
• Manifest creation now returns a ManifestError when it encounters unsupported metadata.

Re-review result

• I don’t see the original panic-on-new-variant issue remaining.
• The added validation path is the right fix for forward compatibility.

Given the focused test runs you listed, this looks good to me.