Port: port/pr-17-extend-scanner-lang-support

[briansumma]

Ported from upstream cytostack/openwolf. Needs review before merge.

[briansumma]

Here's my review of PR #3:

---

PR #3 — Port: Extend Scanner Language Support

Status: DRAFT | Lines: +16 / -2
What it does: Ports upstream PR cytostack#17 from cytostack/openwolf. Expands both the CODE_EXTENSIONS set in anatomy-scanner.ts and the CODE_EXTS set in token-estimator.ts with ~18 new file extensions (Dart, Kotlin, Swift, C++ variants, C#, Ruby, PHP, Lua, Vue, Svelte, HTML, Protobuf, GraphQL, Terraform, and shell variants). Also adds "Keep in sync" comments linking the two parallel sets.

---

⚠️ Warnings

WR-01: Fragile duplicate Sets — "Keep in sync" comment doesn't prevent drift

Both CODE_EXTENSIONS (anatomy-scanner.ts:42) and CODE_EXTS (token-estimator.ts:4) are byte-for-byte identical sets maintained in two separate files. The sync comment is an improvement over the pre-existing state, but it's unenforceable — any future contributor who adds an extension to one file and misses the other will silently produce divergent behavior:

• anatomy-scanner.ts would use ratio 3.75 (default) instead of 3.5 for the missed extension
• token-estimator.ts would return "mixed" instead of "code" for detectContentType()

For a tool where token estimation accuracy is the core value proposition, this silent drift is a real risk. The correct fix is to extract the shared set into a single source of truth:

// src/utils/extensions.ts (new file)
export const CODE_EXTENSIONS = new Set([
  ".ts", ".js", ".tsx", ".jsx", ".py", ".rs", ".go", ".java",
  // ... full list ...
]);

export const PROSE_EXTENSIONS = new Set([".md", ".txt", ".rst", ".adoc"]);

Then import it in both files. This eliminates the sync problem entirely.

---

WR-02: .html/.htm classified as code applies too-low token ratio

anatomy-scanner.ts:49 / token-estimator.ts:11

HTML is classified as "code" type, applying a 3.5 chars/token ratio. HTML files are markup with prose content, attribute text, and only optionally embedded JS/CSS. The mixed ratio (3.75) or even prose ratio (4.0) would be more accurate.

With the code ratio, OpenWolf will consistently under-count tokens for HTML files, which means users could exceed their intended token budget. Given this is a token-budgeting tool, accuracy matters.

Suggested fix: Remove .html and .htm from CODE_EXTS/CODE_EXTENSIONS and either add a dedicated MARKUP_EXTENSIONS set with its own ratio, or leave HTML to fall through to the mixed default (3.75):

// In anatomy-scanner.ts estimateTokens():
if (CODE_EXTENSIONS.has(ext)) ratio = 3.5;
if (MARKUP_EXTENSIONS.has(ext)) ratio = 3.75; // Already the default, so just don't add to CODE_EXTENSIONS
if (PROSE_EXTENSIONS.has(ext)) ratio = 4.0;

---

ℹ️ Info

IN-01: No project-level tests

The project has no test suite (no *.test.ts or *.spec.ts files outside node_modules). The extension lists and the estimateTokens/detectContentType functions are untested. Not introduced by this PR, but worth noting before this pattern grows — a simple test asserting the two sets are identical would have caught any future WR-01 drift automatically.

---

✅ What's Good

• The "Keep in sync" comments are a meaningful improvement over the prior state
• Extension additions themselves are correct and well-chosen (Dart, Kotlin, Swift, Terraform, GraphQL, shell variants all reasonable as code type)
• Both files updated atomically in the same PR — the sets are in sync right now
• .hpp, .hh, .cc, .cxx filling out C++ variants was the right call

---

Verdict

Not ready to merge as-is. WR-01 is the primary concern — this PR is the perfect moment to fix the structural duplication rather than cement the comment-based sync pattern. WR-02 is lower priority but worth a conversation before merge. Neither is a showstopper if you want to accept the upstream port as-is, but they'd both be easier to fix now than after more files depend on the pattern.

[briansumma]

✅ Autofixer applied. Build and tests passed. Promoting to Ready for Review.