Add dvx audit — blob classification, lineage, and cache analysis

README.md

@@ -276,6 +276,7 @@ with Repo() as repo:
 | `add` | Track file(s) with optional provenance |
 | `status` | Show freshness of tracked files (data & deps) |
 | `diff` | Content diff with preprocessing support |
+| `audit` | Blob classification, lineage, and cache analysis |
 | `cache` | Inspect cache (path, md5, dir) |
 | `cat` | View cached file contents |
 | `push` | Upload data to remote storage |
@@ -299,6 +300,7 @@ with Repo() as repo:
 ### Added in DVX
 - `dvx run` - Parallel pipeline execution with per-file provenance
 - `dvx diff` preprocessing - Pipe through commands before diffing (with `{}` placeholder)
+- `dvx audit` - Blob classification (input/generated/foreign), lineage, orphan detection
 - `dvx cache path/md5` - Cache introspection
 - `dvx cat` - View cached files directly
 - `dvx status --yaml` - Detailed status with hashes

specs/blob-audit-lineage.md

@@ -0,0 +1,158 @@
+# Blob Audit and Lineage Tracking
+
+## Problem
+
+DVX stores provenance per-artifact (`meta.computation` in `.dvc` files), but there's no way to query across artifacts: which blobs are used where, which are still necessary, which are orphaned, and what the full dependency graph looks like. DVC's `gc` operates at the hash level (keep referenced hashes, delete unreferenced), but doesn't reason about the *relationships* between blobs across commits.
+
+## Use Cases
+
+### 1. "What blobs does this commit need?"
+Given a commit SHA, enumerate all `.dvc` files, their output hashes, and transitively all input hashes. Answer: "to fully reproduce this commit's state, you need these N blobs totaling X MB."
+
+### 2. "Where is this blob used?"
+Given a blob hash (or path), find all commits/branches/tags that reference it — either as a direct output or as a transitive dependency. Answer: "this blob is referenced by 3 commits on `main` and 1 tag."
+
+### 3. "Which blobs are generated vs. input?"
+Classify all blobs in the cache (or remote) by their provenance:
+- **Input**: no computation, added directly via `dvx add` or `dvx import-url`
+- **Generated**: has computation, output of `dvx run`
+- **Foreign**: imported via `--no-download`, tracked by ETag but not cached locally
+- **Orphaned**: in cache but not referenced by any `.dvc` file in any branch/tag/commit
+
+### 4. "What's the minimal cache for this branch?"
+Given a branch, compute the minimal set of blobs needed:
+- All input blobs (not reproducible)
+- Generated blobs only if their inputs are unavailable
+- Total size of irreducible inputs
+
+### 5. "Can I safely delete this remote blob?"
+Before deleting a blob from S3, check:
+- Is it an input blob? (If so, it's irreplaceable — don't delete unless another copy exists)
+- Is it generated? (Safe to delete if inputs are available)
+- Is it referenced by any commit? (If not, it's orphaned — safe to delete)
+
+## Implementation
+
+### Module structure
+
+```
+src/dvx/audit/
+  __init__.py       # Exports: scan_workspace, audit_artifact, find_orphans
+  model.py          # BlobKind, Reproducibility, BlobInfo, AuditSummary
+  scan.py           # Scanning and classification logic
+src/dvx/cli/audit.py  # Click command
+```
+
+### Data model (`model.py`)
+
+- **`BlobKind`** enum: `INPUT`, `GENERATED`, `FOREIGN`, `ORPHANED`
+- **`Reproducibility`** enum: `REPRODUCIBLE`, `NOT_REPRODUCIBLE`, `UNKNOWN`
+- **`BlobInfo`** dataclass: path, md5, size, kind, reproducible, cmd, deps, git_deps, in_local_cache, in_remote_cache, is_dir, nfiles
+- **`AuditSummary`** dataclass: blobs list + computed aggregates (counts/sizes by kind, cache stats)
+  - `to_dict()` for JSON serialization — this is the data contract for the web UI
+
+### Classification logic (`scan.py`)
+
+`classify_blob(DVCFileInfo) → (BlobKind, Reproducibility)`:
+- No md5 on outs → `FOREIGN`
+- Has `cmd` → `GENERATED`
+  - `meta.reproducible: false` → `NOT_REPRODUCIBLE`
+  - `meta.reproducible: true` or absent → `REPRODUCIBLE` (positive default)
+- No cmd → `INPUT`
+
+Reuses from existing code:
+- `find_dvc_files()` from `cache.py` — discovers `.dvc` files
+- `read_dvc_file()` from `dvc_files.py` — parses `.dvc` YAML
+- `check_local_cache()` from `cache.py` — checks local cache existence
+- `read_dir_manifest()` from `dvc_files.py` — reads directory manifest entries for orphan detection
+
+### Orphan detection (`find_orphans`)
+
+1. Collect all md5 hashes referenced by `.dvc` files (output hashes + dep hashes + dir manifest entries)
+2. Walk `.dvc/cache/files/md5/` to enumerate all cached blobs
+3. Return blobs not in the referenced set, with their sizes
+
+### CLI (`cli/audit.py`)
+
+```
+dvx audit                         # workspace summary
+dvx audit <path>                  # per-artifact lineage
+dvx audit -o/--orphans            # list unreferenced cache blobs
+dvx audit -g/--graph              # DOT dependency graph (colored by kind)
+dvx audit --json                  # machine-readable (any mode)
+dvx audit -r/--remote <name>      # also check remote cache
+dvx audit -j/--jobs N             # parallel workers for remote checks
+```
+
+### DVCFileInfo extension
+
+Added `reproducible: bool | None = None` field to `DVCFileInfo` in `dvc_files.py`, read from `meta.reproducible` in `read_dvc_file()`.
+
+### Output formats
+
+**Summary** (no args):
+```
+Blobs in workspace:       34
+  Input:                  18 (810 MB)
+  Generated:              16 (80 MB, 14 reproducible)
+  Foreign:                 0
+
+Local cache:              30 of 34 (870 MB)
+  Missing:                 4 (20 MB)
+```
+
+**Per-artifact** (`dvx audit <path>`):
+```
+Path:    www/public/taxes-2025-lots.geojson
+MD5:     abc123...
+Size:    22.8 MB
+Type:    Generated (reproducible)
+Command: python -m jc_taxes.geojson_yearly --year 2025 --agg lot
+
+Dependencies (2 data + 2 code):
+  [data] data/taxrecords_enriched.parquet  (def456...)
+  [code] src/jc_taxes/geojson_yearly.py    (git: aabbcc)
+
+Cache:   local=yes  remote=not checked
+```
+
+**Orphans** (`dvx audit --orphans`):
+```
+4 orphaned blob(s) (12 MB):
+  a433cf78...  (8.2 MB)
+  b782ee41...  (3.8 MB)
+```
+
+**JSON** (`dvx audit --json`): full `AuditSummary.to_dict()` serialized.
+
+**Graph** (`dvx audit --graph`): Graphviz DOT with kind-based node coloring:
+- Input = palegreen, Generated = lightblue (lighter if reproducible), Foreign = gray dashed
+
+### Integration with `dvx dag`
+
+The graph output reuses the same conceptual DAG structure as `dvx dag` but colors nodes by `BlobKind` rather than by position (root/leaf/middle). For full graph features (clustering, Mermaid, HTML), use `dvx dag`.
+
+## ML Pipeline Considerations
+
+Large-scale ML training pipelines amplify the audit problem. A single training run may produce:
+- **Checkpoints**: dozens of multi-GB model snapshots at different training steps
+- **Evaluation artifacts**: metrics, predictions, confusion matrices at each checkpoint
+- **Intermediate data**: preprocessed datasets, tokenized corpora, embedding caches
+
+The `meta.reproducible: false` opt-out is particularly important here — expensive training outputs should be explicitly marked non-reproducible to prevent accidental eviction.
+
+## Future (not this PR)
+
+- Cross-commit scanning (which commits reference which blobs)
+- `dvx gc --evict-reproducible` (uses audit classification) — see `evictable-generated-blobs.md`
+- Remote cache size analysis
+- SQLite index for large repos
+- UI extension: audit view tab in `ui/` (Vite + React + @xyflow/react)
+
+## Open Questions
+
+- How expensive is scanning `.dvc` files across all commits? For repos with thousands of commits and hundreds of `.dvc` files, this could be slow. The SQLite index amortizes this but adds maintenance burden.
+- Should `dvx audit` also check dep *availability* (can this blob actually be regenerated right now)? This requires checking that all transitive inputs exist in cache or remote.
+- Should lineage be queryable in the other direction? ("What outputs does this input produce?" — useful for impact analysis when an input changes.)
+- For ML pipelines: should DVX integrate with experiment trackers (W&B, MLflow) to correlate blob lineage with training metrics? Or should it stay purely at the data layer and let users join the two?
+- Should `dvx audit` support a `--cost` flag that estimates regeneration cost from historical run times logged in `meta.computation`?

specs/evictable-generated-blobs.md

@@ -0,0 +1,141 @@
+# Reproducible Generated Blobs
+
+## Problem
+
+DVX tracks both **generated** artifacts (have `meta.computation` with `cmd` + `deps`) and **input** artifacts (no computation, or foreign imports). Currently GC treats them identically — keep or evict based on git scope. But generated artifacts are fundamentally different: they can be **regenerated** from their inputs + code at any commit, making them safe to evict from cache even when referenced.
+
+This matters for projects like jc-taxes where ~450MB of GeoJSON is regenerated from deterministic code + parquet inputs. Every regen creates 16 new cache blobs; old versions accumulate even though they're reproducible from the corresponding code.
+
+## Concepts
+
+### Blob taxonomy
+
+|  | **Has computation** | **No computation** |
+|--|--------------------|--------------------|
+| **DVX-tracked** (`.dvc` file) | Generated: `dvx run` output | Input: `dvx add`'d raw data |
+| **Foreign** (`import-url --no-download`) | n/a | External: tracked by ETag, not cached |
+
+Generated blobs are the only ones that can be safely evicted without data loss, because they satisfy:
+1. Their `.dvc` file records the exact `cmd` + `deps` + `git_deps`
+2. At any git commit, the code (`git_deps`) and data inputs (`deps`) are pinned
+3. Re-running the command with those inputs reproduces the output
+
+### Reproducibility spectrum
+
+Not all generated blobs are equally reproducible. The `reproducible` flag encodes a **confidence level about reproducibility**:
+
+| Level | Example | Reproducible? | Notes |
+|-------|---------|--------------|-------|
+| **Bit-reproducible** | jc-taxes GeoJSON from deterministic Python | Yes (default) | Same inputs + code → byte-identical output |
+| **Semantically reproducible** | ML inference, deterministic but float-sensitive | Mostly | May differ at last bits due to hardware/library versions |
+| **Reproducible with seed** | Single-GPU training with fixed seed | Cautiously | Reproduce given same hardware + library versions |
+| **Non-reproducible** | Distributed training with weight-update races | No | Even with same inputs, output differs per run |
+
+Note: even "non-reproducible" training is becoming tractable — Google's Marin 8B (JAX + TPU) achieved bit-reproducibility for large-scale training, specifically to enable debugging loss spikes in expensive runs. But this is the exception; most distributed training has inherent nondeterminism from collective ops ordering.
+
+**Key insight**: reproducibility is the positive default. Blobs with `computation` are assumed reproducible unless explicitly marked `meta.reproducible: false`. A `dvx run` output that took 10,000 GPU-hours to produce should be explicitly marked non-reproducible by the user.
+
+### Reproducibility
+
+A generated blob is **reproducible** (and thus safe to evict) when:
+- It has a `computation` block in its `.dvc` file
+- It is NOT marked `meta.reproducible: false` (opt-out)
+- All its `deps` are available (either cached or themselves reproducible)
+- All its `git_deps` are available (always true — they're in git)
+
+An input blob is **never evictable** unless backed up to a remote (the existing `--safe` GC behavior).
+
+## Proposed Changes
+
+### 1. `.dvc` file: `reproducible` flag
+
+The `meta.reproducible` field is opt-out — present only when a generated blob is NOT reproducible:
+
+```yaml
+outs:
+  - md5: abc123
+    size: 22851069
+    path: model-checkpoint.pt
+
+meta:
+  reproducible: false
+  computation:
+    cmd: python train.py --steps=50000 --seed=42
+    deps:
+      data/training_set.parquet: def456
+    git_deps:
+      train.py: aabbcc
+```
+
+- Default (absent): blobs with `computation` are assumed reproducible
+- `reproducible: false` — blob cannot be reliably regenerated, keep in cache
+- Classification is surfaced by `dvx audit` (see blob-audit-lineage spec)
+
+### 2. `dvx gc --evict-reproducible`
+
+New GC mode: evict blobs that are reproducible (have `computation` and not marked `reproducible: false`), keeping only input blobs and non-reproducible outputs.
+
+```bash
+# Evict reproducible blobs not used in current workspace
+dvx gc -w --evict-reproducible
+
+# Evict reproducible blobs from all commits (keep only inputs in cache)
+dvx gc -A --evict-reproducible
+
+# Dry run
+dvx gc -w --evict-reproducible --dry
+```
+
+Logic:
+1. Identify all `.dvc` files in scope
+2. For each, check if it has `computation` and is NOT `meta.reproducible: false`
+3. If reproducible: remove from cache (local and/or remote per `--cloud`)
+4. If not reproducible: keep (standard GC behavior)
+
+### 3. Non-reproducible marking
+
+For expensive or non-deterministic outputs, mark them explicitly:
+
+```yaml
+# In the .dvc file, set meta.reproducible: false
+meta:
+  reproducible: false
+  computation:
+    cmd: ...
+```
+
+This can be done manually in the YAML, or a future `dvx mark --not-reproducible <path>` command.
+
+### 4. `dvx regen` (optional, future)
+
+Regenerate evicted blobs on demand:
+
+```bash
+# Regenerate a specific evicted blob
+dvx regen www/public/taxes-2025-lots.geojson
+
+# Regenerate all evicted blobs needed for current workspace
+dvx regen --workspace
+```
+
+Uses `meta.computation.cmd` with the pinned deps. This is essentially `dvx run` but specifically targeting blobs that were evicted.
+
+## Interaction with `dvx push`
+
+When pushing, reproducible blobs could optionally be skipped:
+
+```bash
+# Push only non-reproducible (input) blobs
+dvx push --skip-reproducible
+
+# Push everything (default, current behavior)
+dvx push
+```
+
+This saves remote storage for blobs that are reproducible. The tradeoff: re-generating is slower than pulling from cache, but for deterministic pipelines the remote copy is pure redundancy.
+
+## Open Questions
+
+- Should there be a project-level default in `.dvc/config`? E.g., `core.assume_reproducible = true` (already the default behavior) or `false` to require explicit opt-in.
+- Should there be a "confidence" level? E.g., `reproducible: true` (fully reproducible) vs `reproducible: expensive` (reproducible but costly)?
+- How does this interact with remote storage billing? If reproducible blobs are skipped on push, the remote is smaller but regen requires local compute.

src/dvx/audit/__init__.py

@@ -0,0 +1,17 @@
+"""DVX blob audit — classification, lineage, and cache analysis."""
+
+from dvx.audit.model import AuditSummary, BlobInfo, BlobKind, Reproducibility
+from dvx.audit.repo_view import FilesystemRepoView, SnapshotRepoView
+from dvx.audit.scan import audit_artifact, find_orphans, scan_workspace
+
+__all__ = [
+    "AuditSummary",
+    "BlobInfo",
+    "BlobKind",
+    "FilesystemRepoView",
+    "Reproducibility",
+    "SnapshotRepoView",
+    "audit_artifact",
+    "find_orphans",
+    "scan_workspace",
+]