R2Corpus: anonymous-pull, auth-push corpus sync with tar bundling

[lilith]

Summary

Add an R2Corpus API to codec-corpus for syncing file corpora (fuzz seeds, test fixtures, conformance suites) to/from Cloudflare R2. Anonymous pull via public HTTP, authenticated push via S3 API.

The existing Corpus API (GitHub release tarballs) is widely used (~115 callsites across zen crates) and works well for curated, versioned datasets. ManifestCorpus (JSONL + per-blob fetch from R2) was never adopted outside this crate — its schema was overdesigned and the blob-by-hash mental model didn't match how people think about test data.

R2Corpus targets the gap: continuously-growing corpora (fuzz seeds, CI-generated test data) that are too large or too frequently updated for GitHub releases, but still need anonymous cross-platform fetch.

Design principles

1. Path-keyed, not hash-keyed. Files have meaningful paths. The API returns a directory you walk with std::fs.
2. Manifest is auto-generated, never hand-edited. The push command regenerates the .list file. No manual schema maintenance.
3. Minimal schema. .list contains {path: {size, sha256, in_bundle}}. No expected_behavior, no source_label, no classification metadata.
4. Smart bundling. For prefixes with many small files, a .tar.zst bundle + .list sidecar replaces thousands of individual HTTP requests.
5. Hierarchical prefixes. pull("fuzz/zentiff/") fetches a subtree. pull("fuzz/") fetches everything.
6. Same mental model as Corpus. One-line setup, returns a PathBuf, users walk it normally.

Storage layout

s3://codec-corpus/fuzz/zentiff/fuzz_decode.list           # auto-generated index
s3://codec-corpus/fuzz/zentiff/fuzz_decode.tar.zst        # frozen snapshot bundle
s3://codec-corpus/fuzz/zentiff/fuzz_decode/abc123...      # individual delta (added since last bundle)
s3://codec-corpus/fuzz/zentiff/fuzz_decode/def456...      # individual delta

.list format

{
  "version": 1,
  "prefix": "fuzz/zentiff/fuzz_decode/",
  "generated_at": "2026-04-08T03:15:00Z",
  "bundle": {
    "key": "fuzz/zentiff/fuzz_decode.tar.zst",
    "size": 18234000,
    "sha256": "0123abc...",
    "file_count": 1556,
    "uncompressed_size": 41499000
  },
  "files": {
    "abc123...": { "size": 1234, "sha256": "abc123...", "in_bundle": true },
    "fed987...": { "size":  512, "sha256": "fed987...", "in_bundle": false }
  }
}

Pull algorithm

1. Fetch .list (small, one request)
2. Diff against local cache (by sha256)
3. If >40% of bundled files are missing → download .tar.zst, extract (1-2 requests for cold start)
4. Fetch remaining individual deltas in parallel (for incremental updates)

Cold start for 1556 files / 41 MB: ~3 seconds (1 GET for list + 1 GET for bundle + extract) vs ~30-60s for 1556 individual GETs.

Push algorithm

1. Walk local dir, hash new files
2. Upload new files as individual objects + update .list
3. Optionally --rebundle to regenerate .tar.zst from all files (when too many deltas accumulate)
4. Auto-rebundle threshold (e.g., rebundle when >100 deltas exist since last bundle)

HTTP client approach

Note: The existing Corpus API deliberately avoids compiled-in HTTP dependencies. It shells out to OS tools (curl, wget, powershell Invoke-WebRequest, git sparse-checkout) detected at runtime, keeping the crate tiny (~1 KB on crates.io) with minimal deps (dirs, fd-lock).

R2Corpus should follow the same pattern for anonymous pull:

• Bundle download: curl -fSL / wget -q / Invoke-WebRequest (same as existing Corpus)
• .list download: same tool chain
• Tar extraction: tar (with zstd support, i.e. tar --zstd -xf or zstd -d | tar -xf)
• Individual file fetch: curl/wget with parallel invocations (e.g. xargs -P)

For authenticated push, the S3 API is more complex (SigV4 signing, multipart upload). Options:

• Shell out to aws s3 CLI (simple, user installs it themselves, push is admin-only anyway)
• Optional r2-push feature flag that pulls in object_store or rust-s3 (heavier, but self-contained)
• Separate r2-corpus-admin binary with its own Cargo.toml and heavier deps (keeps the library crate minimal)

The library crate (codec-corpus) stays minimal with OS-tool downloads. The admin binary (push/rebundle) can afford heavier deps since it's only used by maintainers.

Rust API

// Anonymous pull (no auth needed, R2 bucket has public-read)
// Uses curl/wget/powershell under the hood — no compiled-in HTTP client
let corpus = R2Corpus::pull("https://corpus.imazen.io", "fuzz/zentiff/fuzz_decode/")?;
for entry in std::fs::read_dir(corpus.path())? { /* ... */ }

// With options
let corpus = R2Corpus::pull_with_options(
    "https://corpus.imazen.io",
    "fuzz/zentiff/fuzz_decode/",
    PullOptions { mode: PullMode::ForceBundle, ..default() },
)?;

CLI (r2-corpus binary)

r2-corpus pull  fuzz/zentiff/                        # anonymous, downloads subtree
r2-corpus push  fuzz/zentiff/fuzz_decode/ --local ./fuzz/corpus/fuzz_decode  # auth required
r2-corpus push  fuzz/zentiff/fuzz_decode/ --rebundle  # regenerate tar.zst
r2-corpus list  fuzz/zentiff/                        # show .list contents
r2-corpus diff  fuzz/zentiff/fuzz_decode/            # local vs remote
r2-corpus login                                      # store R2 credentials

Per-project config

Projects configure their corpus sync in corpus-sync.toml:

[corpus]
base_url = "https://corpus.imazen.io"
local_dir = "fuzz/corpus"

[[sync]]
prefix = "fuzz/zentiff/fuzz_decode/"
local = "fuzz/corpus/fuzz_decode"

Dependencies

Library crate (anonymous pull): No new compiled deps — shells out to curl/wget/tar/zstd like the existing Corpus API. Keeps the crate tiny.

Admin binary (push/rebundle): Either shells out to aws s3 CLI or uses optional compiled deps:

• object_store or rust-s3 for S3-compatible push
• serde + serde_json for .list generation (already a dep via ManifestCorpus)
• clap for CLI
• dirs for cross-platform cache directory (already a dep)

Implementation plan

Phase 1 (~2 days): Basic R2Corpus with per-file pull/push and auto-generated .list. No bundle support. Proves the API works.

Phase 2 (~3 days): Add .tar.zst bundle support. Smart pull heuristic. Rebundle command. Transparent upgrade — existing prefixes get faster as soon as someone runs --rebundle.

Prerequisites

• Enable public-read on codec-corpus R2 bucket (or the fuzz/ prefix)
• Optionally set up custom domain (corpus.imazen.io → R2 bucket)

Motivation

This replaces the ad-hoc fuzz-corpus.sh shell script currently in imazen/zenextras (which shells out to aws s3 sync and requires AWS CLI + R2 credentials even for reads). Any zen crate could adopt this with codec-corpus = { version = "1.1", features = ["r2"] } and a one-line pull call.