feat(corruption-corpus): structural-corruption generator, decoupled from the codec-corpus fetcher#8
Open
lilith wants to merge 10 commits into
Open
feat(corruption-corpus): structural-corruption generator, decoupled from the codec-corpus fetcher#8lilith wants to merge 10 commits into
lilith wants to merge 10 commits into
Conversation
Three VarDCT bitstreams chosen to cover the distinct error variants the bug manifests as. All from the same root cause (LZ77 distance cluster shifted away from context_map.last() by AC-context padding) but each fails at a different validation point in the AC reader. libjxl djxl 0.12.0 and jxl-oxide 0.12.5 decode all three cleanly. libjxl/jxl-rs v0.4.3 rejects them; fix in libjxl/jxl-rs#766. | file | size | manifests as | | akfcrc022_e9_d3.0.jxl | 22 KB | Invalid AC: 8 nonzeros after decoding block | | gb82-sc_windows9_d0.25.jxl | 51 KB | Invalid AC: nonzeros 144 is too large for 1 8x8 blocks | | gb82-sc_terminal_d0.5.jxl | 58 KB | ANS stream checksum mismatch | Encoder: imazen/jxl-encoder at e9 + low distance on screen content (uses Optimal/Greedy LZ77 — libjxl never emits these for VarDCT, so its existing test corpora can't catch the bug). References regenerate via jxl/generate_references.sh. Refs: - libjxl/jxl-rs#765 (bug) - libjxl/jxl-rs#766 (fix) - imazen/zenjxl-decoder#15 (downstream tracking + regression test) - imazen/jxl-encoder#29 (encoder default rationale)
Add the `corruptions` module: a held-out falsification set for a perceptual metric's negative tail. Localized structural defects no honest encoder produces (channel swap, dropped block, off-by-one edge) that a faithful metric must rank below an honest lossy encode of the same reference — the gate score(corruption) < score(q20-anchor). Ten distortion families (channel/block/edge/noise/tone/overlay/ chroma-boundary/aliasing/geometric/composite), each parameterized by region size (whole -> 1/4 -> 1/16 -> 64x64 -> 16x16 -> 8x8) and severity (opaque -> 50% -> 20% opacity). Generators are deterministic via a dependency-free SplitMix64 (same bytes on every platform) and pure RGB-buffer math, so they stay in the default build and compile to WASM. Includes the ManifestEntry schema (reserving a `real_bug` source slot for historical-bug reproductions tracked elsewhere) and the catalog() / manifest_for_reference() sweep generators. 51 default-build unit tests cover determinism, dimension preservation, slug uniqueness, and tiny-image no-panic across every family.
Add the `driver` feature (optional `image` dep) and `corruptions::driver`: builds the score-ready quad (reference, corruption, q20-anchor, q10-anchor) for the metric gate. JPEG anchors are encoded then decoded back to pixels so the gate scores pixels, not bitstreams. Add the `corruption_corpus` example: given references, expands the full catalog, builds each quad, and writes the corrupted PNG + both JPEG anchors + a _MANIFEST.json to an output dir. No corrupted bytes are committed — the corpus is reproduced on demand from (ref_id, seed, params) plus the curated reference set already in this repo. Integration tests (driver feature) exercise the full quad builder across the catalog, manifest reproducibility, and lossless PNG roundtrip. README + CHANGELOG document the module, families, and usage.
Adds the scripts/ pipeline for the imazen-26 synthetic content corpus.
Existing pipeline (01-04): downscale/crop -> bpp matrix -> K-means(K=500)
clustered subset -> rename to NNNN-provenance-category-subcategory.png.
See scripts/README.md and ZENSIM_CORPUS_PLAN.md.
New (05): regenerates lilith/generated-graphics/ line-art with
anti-aliasing. Reproduces each curated PNG byte-exact from the seed in
its filename (replaying the driver's 512->1024 RNG reuse), then renders
1024sq via ScaledDraw supersample (SS=4) and Lanczos-down. --validate
asserts byte-exact + sort-consistent vs the originals; passes for all 45
line-art files. Sort rule: folder = f(type, kind). Output:
lilith/generated-graphics-aa/{polygons,lines,line-patterns,grids}/.
New (06): quickchart.io gallery covering every type the public endpoint
renders (20 types via Chart.js v2/v4) across 8 themes, with real-world
data triple-checked for validity (17 in-code asserts; --check-only runs
them). Heatmap via matplotlib (quickchart's public endpoint does not
register matrix/treemap/graph controllers). 1024sq at devicePixelRatio=2
Lanczos-down. Output: lilith/generated-graphics-aa/charts/ and grids/.
…x1024 Two AI-gen sporting-goods files were PNG (magic) misnamed .jpg with stale WxH in the name. Renamed to *_1536x1024.png; manifest path/width/height/format + original_filename corrected. (Images are untracked; this commits the manifest rows.)
Probe vs basename xref found 9231 (AI-gen accessory PNG) named 2160x3840 but actually 1024x1536. Renamed + manifest path/width/height/original_filename fixed.
Live-verified against R2 2026-07-22: the "no tower or R2 mirror" claim for stage 3 (the live corpus) was wrong — imazen-26-unprocessed/ has been in the codec-corpus bucket since 2026-06-09, predating this doc's own 2026-06-25 trace. Adds the R2 mirror location, the previously-undocumented ~178GB of codec-encoded derivatives under picker-sweep-2026-06-22/, and flags (but does not rewrite) PROVENANCE.md's mismatched folder references as apparent leftover from an earlier corpus iteration. Scope: this file only. Pre-existing uncommitted changes elsewhere in this repo (scripts/ reorg, README.md, .gitignore, untracked image folders) are unrelated prior work, left untouched.
…eparate corruption-corpus workspace member Decouple the corpus FETCHER (codec-corpus: download/cache/access test-image datasets) from the structural-corruption GENERATOR. The corruptions module was self-contained -- it imported only `image`, `serde`, and its own `prng` (SplitMix64), used none of codec-corpus's download/registry code, and the fetcher used none of it -- so this is a clean move; no logic changed. New workspace member crate/corruption-corpus/ (package `corruption-corpus`, lib `corruption_corpus`, v0.1.0). git-mv preserves history: corruptions/mod.rs -> src/lib.rs (crate root), families.rs/driver.rs/prng.rs -> src/, the example -> examples/, the integration test -> tests/. Imports updated codec_corpus::corruptions::* -> corruption_corpus::* (the module-internal super::/crate:: refs were already relative and are unchanged). Added #![forbid(unsafe_code)]; README + CHANGELOG. codec-corpus: removed `pub mod corruptions;`, the corruption-only optional `image` dep, the `driver` feature, and the `[[example]]`. README/CHANGELOG now point at the new crate. The default codec-corpus build no longer pulls `image`. The image-backed driver + `corruption_corpus` example still live in the shared workspace, so the example binary path (crate/target/release/examples/corruption_corpus), consumed by zensim's build_corruption_corpus.py, is unchanged; only the build invocation now needs `-p corruption-corpus --features driver`.
…ntegration test Pre-existing clippy::unnecessary_map_or lint at tests/integration.rs:37, unrelated to the corruption-corpus extraction; fixing it makes `cargo clippy --workspace --all-targets` clean with -D warnings.
The extraction moved the corruption generator into a separate workspace member. CI ran `cargo check/clippy/test` from crate/ with no `--workspace`, which -- with a root package present -- covers only codec-corpus, so the new crate would have shipped untested. Add `--workspace` to the native + wasm jobs, plus `-p corruption-corpus --features driver` steps so the image-backed driver + example are exercised. Verified locally: native + wasm (scalar & SIMD128 under wasmtime) all green; `cargo clippy --workspace --all-targets -- -D warnings` clean including the driver feature.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Introduces
corruption-corpus— a standalone workspace-member crate thatgenerates a held-out falsification set for a perceptual metric's negative
tail: localized structural defects no honest encoder would produce (a channel
swap, a dropped block, an off-by-one edge) that a faithful metric must rank
below an honestly-lossy encode of the same reference.
The gate every corpus entry asserts:
Built per the spec at
zensim/docs/structural_corruption_corpus_spec_2026-05-27.md.Structure — generator decoupled from fetcher
The structural-corruption generator now lives in its own crate
(
crate/corruption-corpus/, packagecorruption-corpus, libcorruption_corpus), decoupled from thecodec-corpusfetcher (whichdownloads / caches / serves test-image datasets). The module was self-contained
— it imported only
image,serde, and its ownprng(SplitMix64), used noneof codec-corpus's download/registry code, and the fetcher used none of it — so
the split is a clean move; no generator logic changed. Consequences:
codec-corpusno longer depends on theimagecrate in its default build;the
driverfeature + thecorruption_corpusexample moved to the new crate.target/(
crate/target/release/examples/corruption_corpus), so downstream consumersthat invoke it by path are unaffected; the build now just needs
-p corruption-corpus --features driver.Ten distortion families
channel · block · edge · noise · tone · overlay · chroma-boundary · aliasing ·
geometric · composite — each parameterized by region size (whole-image →
1/4 → 1/16 → 64×64 → 16×16 → 8×8, + 1px for edge/geometric) and severity
(opaque → 50% → 20% opacity). The subtle end (small region, low opacity) is the
hard case where a metric saturates and lets a real bug pass.
Design
(ref_id, seed, params)via a dependency-freeSplitMix64; same bytes on every platform. No
rand, no OS entropy.generators stay in the default build and compile to WASM.
#![forbid(unsafe_code)].curated reference set already in this repo (CLIC2025, CID22, KADID-10k, GB82,
GB82-SC, …).
driverfeature (optionalimagedep) builds the score-ready quad(reference, corruption, q20-anchor, q10-anchor); without it, only the seededin-memory generators + manifest types are compiled.
ManifestEntry { ref_id, content_class, family, params, seed, expected_below_lq, expected_negative, source }—sourcereserves areal_bugslot for historical-bug reproductions tracked separately.Files
crate/corruption-corpus/src/{lib,families,prng,driver}.rs— crate root, 10families, seeded PRNG, quad driver (moved via
git mv, history preserved)crate/corruption-corpus/examples/corruption_corpus.rs— emits quads +_MANIFEST.jsonon demandcrate/corruption-corpus/tests/corruption_corpus.rs— driver integration testscrate/corruption-corpus/{Cargo.toml,README.md,CHANGELOG.md,.gitignore}crate/Cargo.toml— new[workspace] members; dropped theimagedep,driverfeature, and[[example]]crate/src/lib.rs— droppedpub mod corruptions;crate/{README,CHANGELOG}.md— point at the new crateTests (measured on this branch)
cargo test -p corruption-corpus(default): 25 unit + 1 doc greencargo test -p corruption-corpus --features driver: 28 unit + 5 integration + 1 doc greencargo test -p codec-corpus: 26 unit + 4 integration (10 network-ignored) + 8 doc greencargo clippy --workspace --all-targets -- -D warnings(default +--features driver): cleancargo fmt --check: cleanNotes
bugs) are tracked in a parallel GitHub issue; this PR builds the synthetic
families and leaves the
real_bugsource slot for them.clippy::unnecessary_map_orlint intests/integration.rsis now fixed (
is_some_and) in its own commit, so the workspace isclippy-clean under
-D warnings..github/workflows/ci.yml) now runs the native + wasm jobs with--workspace(plus a-p corruption-corpus --features driverstep), so thenew member is exercised on every platform instead of being silently untested
after the split.
imazen-26corpus-data commits(synth pipeline + dimension/manifest/doc corrections) that were committed
locally on the branch before the extraction; they are independent of the
corruption-corpus change.