convert(fp8->int4): --indir copies the full metadata set and resumes interrupted passes (#383)

[anrasi]

Summary

Closes the two --indir gaps from #383, as requested there.

Metadata. The main --indir pass copied only config.json; the download path copies four files. A locally-converted container was left without tokenizer.json, so coli chat/serve refuse to start on it. The local pass now copies the same four (config.json, tokenizer.json, tokenizer_config.json, generation_config.json), prints what it copied, and warns when the source directory is missing any — tokenizer.json called out by name, since that's the one that breaks chat.

Resume. Local passes restarted from shard 0 on every rerun. The download path's if os.path.exists(outp): continue can't be mirrored directly: local out-NNNNN names count emitted shards, not input indexes (shards with no relevant tensors emit no file, so the numbering shifts under resume). Instead, a per-prefix sidecar manifest (.out-progress.json / .out-mtp-progress.json / .out-idx-progress.json) records input shard → output name (or "empty"), plus the conversion parameters, and is rewritten atomically after each shard. A rerun skips everything that matches and still exists on disk. A rerun with different parameters on the same outdir is refused with an explanatory error instead of silently mixing containers — the #355 failure mode, blocked one layer deeper.

Validation

• make -C c check — clean build, C unit tests, Python suite (73 tests) green (aarch64, gcc 13.3)

• Functional matrix on the tiny model (glm_tiny source, GB10):

  case	result
  fresh main pass	converts, copies all four metadata files
  rerun, nothing to do	[RESUME] 1 shard(s) already done, output byte-identical (md5)
  rerun with --ebits 8 on an --ebits 4 outdir	refused with ERROR, files untouched
  --mtp pass into the same outdir	own prefix + own manifest, main shards untouched (md5)
  source without tokenizer files	[META] WARNING: not found … — chat/serve need tokenizer.json
  output shard deleted, manifest kept	reconverted, byte-identical (md5)
  3-shard source, middle shard empty-for-main, output deleted, rerun	numbering stable under resume: skips 2 (one emitted, one empty), reconverts into the same out-00001, byte-identical (md5)

• CUDA — n/a (tools-only change)

• No performance claims

Compatibility

• The default CPU build remains dependency-free (converter already required numpy+safetensors)
• No model files, generated binaries, or benchmark artifacts (the manifest is a small dot-file sidecar inside the output container)

Two notes for review. The manifest lives in the outdir next to the shards it describes, so deleting the outdir resets everything consistently, and deleting shards while keeping the manifest is also safe (the exists-check reconverts them); if you'd rather have resume opt-in behind a flag, happy to gate it. And this is written to compose with #385 (local index resolution for --mtp/--indexer): it only touches emit/skip bookkeeping and metadata copy, not shard selection — if #385 lands first and the loop moves, happy to rebase promptly.

🤖 Generated with Claude Code