forensic-compare

Compare two LLM checkpoints for provenance signals:

• Tokenizer file hash equivalence
• CKA heatmap (representation similarity)
• Activation distribution fingerprints (skew/kurt/quantiles + KS)
• Optional null-model baselines for z-scores and distributions

Setup

uv sync
# optional: GPU extras (bitsandbytes)
uv sync --extra gpu
# optional: SentencePiece tokenizers
uv sync --extra spm

uv run forensic-compare \
  --a <MODEL_A> --b <MODEL_B> \
  --out_dir ./report \
  --max_seq_len 128 --num_prompts 32 --batch_size 1 \
  --layers_step 2 \
  --cka_samples 1024 --proj_dim 128 \
  --activation_samples 100000 \
  --null_models gpt2,distilgpt2 \
  --null_reference A

CUDA / torch wheel notes

100B-class models typically require CUDA. torch wheels and index URLs can vary by OS, CUDA toolkit version, and accelerator type, and mismatches are a common source of install failures. If uv sync fails or torch is CPU-only, install torch from the official PyTorch index for your platform first, then rerun uv sync.

Null model tests

Use 2-3 unrelated models as baselines and compare them to either A or B.

uv run forensic-compare \
  --a <MODEL_A> --b <MODEL_B> \
  --out_dir ./report \
  --null_models gpt2,distilgpt2,EleutherAI/gpt-neo-125M \
  --null_reference A

Notes:

• --null_models can be repeated or comma-separated.
• --null_reference selects which model (A or B) to compare nulls against.
• Outputs: report/null_model_report.json, report/null_*_hist.png, plus null_z_scores inside report/summary.json.