feat(sweep): spec provenance stamp, intent compilation, declared ladder trims#16
Conversation
…er trims sweep plan now covers the machine-specific intent that forced hand-authoring (vllm runtime path, GPU memory utilization, KV cache bytes) and accepts declared ladder trims with reasons. Generated specs carry a generator stamp with a canonical content hash; bench run prints the verified provenance and reports label runs Generated default sweep or Custom grid from the stored spec bytes, never on trust. Trimmed points render like adaptive skips with their reason, never as holes.
…ow real rows Codex review round 1: the content hash now excludes only generator.content_hash itself, so tampering with declared trims or intent demotes the spec; bench run verifies provenance from the raw file bytes; trims must reference generated contexts; synthesized trimmed rows never overwrite real measurements.
|
Final report: PR 3 (final) of docs/2026-07-05-report-integrity-plan.md. Codex round 1 found 1 P1 + 3 P2 (hash excluded stamp metadata; runner verified re-marshaled bytes; synthetic trim rows could shadow real measurements; trims accepted for absent contexts) — all fixed in be6af8f with regression tests. Round 2 found a single P2 (bench loader not following --vllm-command) — fixed in cf9539a; per the review-loop rule a P2-only round ends the loop. Validation: go test ./..., go vet, gofmt, simpledoc, slophammer check/dry (incl. dependency boundaries), check-crap.sh, golangci-lint, AGENTS.md dry-run + artifact check, plus an end-to-end smoke: sweep plan --trim '64k=8:…' --vllm-command … produces a stamped spec with the capped ladder that bench plan accepts. CI green. Merging per explicit authorization. |
2 participants
Summary
The four GB10 sweeps ran from hand-written spec files with silently trimmed 32k/64k ladders, and nothing could tell a generated spec from a hand-edited one.
This change makes
sweep planthe intent surface and the spec a verifiable build artifact.Machine-specific runtime choices become generator flags, deliberate concurrency caps become declared trims with reasons, and every generated spec carries a content-hashed provenance stamp that reports verify from the stored bytes.
This is PR 3 (final) of docs/2026-07-05-report-integrity-plan.md.
What Changed
The spec stops being an editing surface; intent goes through the generator and everything else is labeled.
sweep plangains--vllm-command,--gpu-memory-utilization,--kv-cache-memory-bytes, and repeatable--trim <context>=<max>:<reason>. Trims cap the generated ladders (and profilemax_num_seqs) and require a reason.generator: {tool, version, intent, ladder_trims, content_hash}(internal/artifact/provenance.go); the hash covers the canonical spec document minus the stamp, computed via generic-map canonicalization so any verifier gets identical bytes.bench runverifies the stamp at load time (against the raw file, before defaults mutate anything) and printsspec provenance: generated|edited|custom.Specmetadata item: "Generated default sweep (localperf sweep plan)" or "Custom grid (hand-authored / edited after generation)".Testing
Unit tests cover stamping, hash verification (generated/edited/custom), trim capping, runtime intent, trim validation branches, and report rendering of provenance + trimmed rows.
go test ./...,go vet,gofmt— cleansimpledoc check,slophammer check/dry,scripts/check-crap.sh,golangci-lint— cleanartifact check— ok (printsspec provenance: customfor the example spec, as expected)sweep plan --trim '64k=8:12 GiB KV budget' --vllm-command …produces a stamped spec with the 64k ladder capped at 8;bench planaccepts it.Risks
Hand-written specs keep working — they are just labeled "Custom grid".
The hash intentionally breaks on any edit to a generated spec; regenerate through
sweep planinstead of editing.Older artifacts without stored original specs render as "Custom grid (hand-authored)", which is accurate.