feat(comparison): comparator-as-record registry + trustworthy-comparison method

[juniperbevensee]

Promotion of the generalizable methodology from a 2026-07-27/28 research-integrity
audit, under docs/promotion-and-upstream.md
Path A: strip the particulars, keep the method, PR from a branch into main,
let the gate run.

The audit's raw material stays where it is as the historical record. What is
promoted here is only what is useful to someone who has never seen the project.
No study, dataset, species, model, function name, filename or path appears in the
diff.

The spine

The same experimental comparison — "the new model beats the classical
baseline" — was invalid four different ways across three attempts by two
different agents, and each fix moved the bug rather than removing it:

Attempt	What was fixed	What was still true
1	—	comparator was print(f"Baseline: P=0.860, R=0.636, F1=0.731") — an f-string with no interpolation, in eight scripts. It printed into a job-completion notification and was read back as a measurement.
2	the constant was removed	the baseline was reimplemented inside the comparison script. Recall 0.636 → 0.098 (F1 0.731 → 0.170); the new arm "won" by +0.285.
3	the callable was imported	it was the wrong one of two variants, chosen by matching its name. The results file said best_method: "simple"; the imported variant scores 0.442 where the real one scores 0.731.
4	the right callable	run at library signature defaults (0.7/30/20) instead of the parameters tuned on train and saved (0.5/10/50).

Every arithmetic step was correct every time. Nothing was fabricated. The
structural cause is one sentence: the comparator was never represented as
data — it existed as a number, then a function name, then a call, never as a
record carrying its own provenance. So each fix had to re-derive from memory or
from a name whatever the last fix had lost.

Files

File	Why
docs/trustworthy-comparison.md	The centrepiece. The four-ways-wrong case study, why every fix moved the bug, five more generic failure modes from the same audit, and 14 checkable rules — including a comparability-check table to write out loud in any report.
docs/baseline-registry.md	The comparator-as-record pattern: four welded fields, the API, the contract a producing script must satisfy, and a "what this still does NOT catch" section.
docs/results-provenance-checklist.md	What a results JSON must carry to be regenerable from itself: seed + which RNGs, library versions, git SHA, the split member lists, every data-reduction decision, the matching criterion, how the operating point was chosen, known leakage.
matilde_plugin/engine/comparison.py	Vendored, generalized, stdlib-only, domain-neutral. compare() raises rather than returning a delta when split or criterion differ.
tests/test_comparison.py	39 tests, one per failure, each named for the mistake it reproduces.
README.md	New Method docs section — a docs index, since the repo had none. Plus a row for the new engine module.
hermes-skill/SKILL.md	A "Before you report that A beats B" subsection under the existing skepticism section, with pointers. This is the difference between a doc that exists and a doc that gets used.

Also distilled (each with a concrete mechanism)

• A declared-but-unused constant. SR = 22050 was declared and never
  referenced again (the name appears exactly once in the file). The model was
  trained at native 48 kHz. Scripts read the constant as intent and helpfully
  resampled — holding split, matcher and threshold fixed, true positives went
  449 → 215, a 2.1× difference, F1 0.766 → 0.539. Across the reporting
  scripts the same model appeared as low as 0.457.
• A label column that was not a label. Column 3 of the annotation rows was a
  frequency in Hz. Four scripts used it as the class, producing 829 singleton
  classes over 829 examples — an 830-way softmax with one example each. The
  real 13-family / 74-label taxonomy was in an adjacent file in the same
  directory.
• An optimum on the sweep boundary. The reported best threshold was the
  lowest value swept, then read as "the model is confident about a lot of
  boundaries" — which inverts the meaning.
• Unseeded stochasticity. Same script, same data: F1 0.7839 then 0.7675. The
  rerun overwrote the artifact the published figure was read from. The
  discrepancy was explained as "a different random split" — the one thing that
  was controlled (tp + fn = 499 in both runs proves the identical test set).
• Silent failure believed. A helper printed HTTP response bodies and exited
  0 regardless of status, so a 405 was indistinguishable from success, and a
  confident, detailed, wrong root-cause analysis was written from the symptom.
  Generalized: anything that cannot fail loudly will eventually be believed.
• Calibration is not validation. A report can be excellent in form —
  recommending a range over a point estimate, flagging that 1.7 points at n=30 is
  within normal variation — while the comparison underneath is invalid. Hedging
  raises the credibility of whatever it is attached to, which makes attaching it
  to an unchecked claim worse than stating the claim bluntly.

The resolution, with its caveats

Re-scored under one split with one matcher, the classical detector beat the CNN:
0.801 vs 0.775 (validated matcher, IoU ≥ 0.3 or overlap ≥ 0.5) and
0.762 vs 0.747 (IoU ≥ 0.3 only).

Both caveats are stated in the docs everywhere those numbers appear:

• Both arms are oracle-best — each arm's operating point was chosen by
  maximising F1 on the test set. Upper bounds, not held-out estimates.
• 22 of 30 test files had leaked into the CNN's training/validation set. Only
  8 were genuinely held out for that arm. The classical detector won despite
  that, which makes the finding stronger, not weaker — but it is not a clean
  measurement of either arm.

The publishable result is a negative one, and it took four invalid comparisons
to reach it.

Design decision: why four fields, not one

A registry returning only a number would have prevented failure 1 and none of
the others — and would make 2–4 harder to catch, because the comparator now
looks authoritative. So a record is fn + params + split + criterion, and
the subtraction refuses unless split and criterion match. The docs/ version
carries this as a table.

The honesty carried across from the source draft is the "what this still does
NOT catch" list — a wrong-but-well-formed criterion passes; declared matching
semantics beyond the thresholds are unverified; a leaky split is legal (only a
missing one is refused). Structure makes disagreement visible, not truthful.
The registry is a floor: it removes failures that come from losing
information, and cannot remove failures where the information is wrong at the
source.

Checks

• python3 -m pytest -q — 144 passed, 7 skipped before → 183 passed, 7
  skipped after. Nothing broken; +39 new.
• python3 scripts/check_sanitization.py on every added/changed file — clean
  on the deterministic layer. Semantic layer skipped locally (no
  ANTHROPIC_API_KEY); CI will run it.
• Manual grep of the staged diff for study particulars (dataset names, species,
  filenames, /opt/data, tool names) — no hits.

Deliberately left out

• configure_threads() (sizes the torch thread pool from the cgroup CPU
  quota). Real and useful, but the generalizable content is a rule — "when a
  job feels slow, suspect CPU quota before memory" — not a function, and
  vendoring it would put cgroup and torch specifics into a stdlib-only engine.
• The domain-specific pre-registration helpers from the source module — the
  method-name→callable map, the pre-registered arms, the hardcoded results paths.
  Those are the particulars; register_baseline_from_results() is what remains
  after stripping them.
• Infrastructure and tooling findings from the audit (container CPU
  throttling, a specific volume being an active Time Machine destination, a
  particular tool's read-before-write guard, one project's stale runtime copy).
  Operationally real, not generalizable method.
• The audit's findings about the citation engine's own axis reporting. Those
  are bug reports against this package, not methodology, and belong in issues or
  a separate PR — not folded into a method doc.

🤖 Generated with Claude Code