feat: extend vocabulary from 36,495 to 38,360 tokens (+5.1%)

.gitignore

@@ -1,3 +1,7 @@
 bazel-*
 *.jsonl
 *.pt
+__pycache__/
+.venv/
+results/
+.handoffs/

pyproject.toml

@@ -0,0 +1,13 @@
+[project]
+name = "ctoc-tools"
+version = "0.1.0"
+description = "Vocabulary extraction and analysis tools for ctoc"
+requires-python = ">=3.11"
+dependencies = [
+    "anthropic>=0.52",
+    "httpx[http2]>=0.28",
+    "tiktoken>=0.8",
+]
+
+[dependency-groups]
+dev = []

tools/README.md

@@ -0,0 +1,86 @@
+# ctoc tools
+
+Vocabulary analysis and extraction tools for ctoc. These complement the main
+C++ CLI by providing Python-based tooling for probing the Anthropic
+`count_tokens` API and analyzing tokenizer accuracy.
+
+## Setup
+
+```bash
+uv sync
+```
+
+Requires `ANTHROPIC_API_KEY` in your environment.
+
+## Tools
+
+### analyze_errors.py
+
+Shows where ctoc's greedy tokenizer diverges from the Claude API on real files.
+Reports per-file efficiency and identifies runs of short tokens that indicate
+over-segmentation.
+
+```bash
+uv run tools/analyze_errors.py FILE [FILE...]
+uv run tools/analyze_errors.py --summary FILE [FILE...]
+```
+
+### find_missing_tokens.py
+
+Discovers missing vocabulary entries by extracting frequent n-grams of short
+tokens from real files and probing them against the API. Useful for finding
+tokens that would have the most impact on accuracy for a given corpus.
+
+```bash
+uv run tools/find_missing_tokens.py FILE [FILE...]
+uv run tools/find_missing_tokens.py --top 200 --json-out results.json FILE [FILE...]
+```
+
+### recheck_vocab.py
+
+Re-probes the `vocab.json` "checked" (rejected) list using the sandwich method.
+The original extraction may have incorrectly rejected tokens due to baseline
+bugs documented in REPORT.md Section 4. See REPORT-ADDENDUM.md for context.
+
+Supports stop/resume via JSONL checkpoint files. Rate-limits automatically.
+
+```bash
+# Test run: 2-byte candidates (~53 min at Tier 1)
+uv run tools/recheck_vocab.py --max-bytes 2
+
+# Resume a previous run
+uv run tools/recheck_vocab.py --max-bytes 2 --resume results/recheck_TIMESTAMP.jsonl
+
+# Override RPM if you know your tier
+uv run tools/recheck_vocab.py --max-bytes 3 --rpm 2000
+```
+
+Results go to `results/` as JSONL. Each line:
+
+```json
+{"candidate": " = ", "sandwich_count": 1, "is_single": true, "timestamp": "2026-02-11T21:37:56Z"}
+```
+
+## Rate limits
+
+The `count_tokens` endpoint is free with its own rate limits separate from the
+Messages API:
+
+| Tier | RPM   | Full checked list (276K) | <= 3 bytes (31K) |
+|------|-------|--------------------------|-------------------|
+| 1    | 100   | 46 hrs                   | 5.3 hrs           |
+| 2    | 2,000 | 2.3 hrs                  | 16 min            |
+| 3    | 4,000 | 1.2 hrs                  | 8 min             |
+| 4    | 8,000 | 35 min                   | 4 min             |
+
+## The sandwich method
+
+All probing uses the "section sign sandwich" to normalize API framing overhead:
+
+```
+count("§" + candidate + "§") - count("§§") == 1  →  candidate is a single token
+```
+
+This avoids three bugs in the raw `count_tokens` baseline documented in
+REPORT.md Section 4: variable framing by character type, trailing newline
+stripping, and control character stripping.