Add unit tests for lint scripts in hack/

[fullsend-ai-retro]

What happened

On PR #36, the review agent caught a bug in the newly added extract_h1() function in hack/lint-experiment-frontmatter: the function did not skip YAML frontmatter when searching for H1 headings, meaning a YAML comment starting with # inside the frontmatter block could be misidentified as the document's H1 heading. The author fixed this in commit 05bd450e by adding frontmatter-skipping logic. The three lint scripts (lint-experiment-frontmatter, lint-experiment-numbers, lint-experiment-index) have zero test coverage — there are no test files in hack/ or elsewhere for these scripts.

What could go better

The frontmatter-skipping bug was only caught because the review agent reasoned about the code's behavior. A unit test with a frontmatter block containing a comment line (e.g., # this is a YAML comment) would have caught this deterministically during development, before the PR was even opened. As these lint scripts enforce core repo conventions and are modified when conventions evolve (as happened in this PR), they are high-value targets for test coverage. Without tests, future modifications to these scripts rely entirely on code review — human or agent — to catch regressions.

Proposed change

Add a test file (e.g., hack/test_lint_scripts.py) with pytest-based unit tests covering at minimum: (1) extract_h1 correctly skips YAML frontmatter and does not match comments inside it, (2) extract_h1 handles files without frontmatter, (3) H1/title mismatch detection in validate_experiment, (4) missing required frontmatter fields, (5) invalid status values, (6) topics field type validation. Wire the tests into CI via a GitHub Actions step or pre-commit hook. The lint-experiment-frontmatter script already uses standard Python with pyyaml as the only dependency, so tests should be straightforward.

Validation criteria

Tests exist for all three lint scripts covering at least the cases listed above. Tests run in CI on every PR. The specific frontmatter-skipping regression (YAML comment matching as H1) has a dedicated test case that would fail if the extract_h1 fix from commit 05bd450e were reverted.

---

Generated by retro agent from #36

[fullsend-ai-triage]

🤖 Finished Triage · ✅ Success · Started 10:27 PM UTC · Completed 10:29 PM UTC
Commit: 0a95cac · View workflow run →

[fullsend-ai-triage]

Triage Summary

This is a well-defined feature request to add unit tests for the three lint scripts in hack/. The issue was generated from a retro analysis of PR #36, where a bug in extract_h1() was caught during review rather than by automated tests.

What needs to happen:

• Create hack/test_lint_scripts.py (or similar) with pytest-based tests
• Cover at minimum the six test scenarios listed in the issue
• Include a regression test for the frontmatter-skipping bug (commit 05bd450e)
• Wire tests into CI so they run on every PR

Key test case — frontmatter comment regression:

def test_extract_h1_skips_frontmatter_comments():
    content = textwrap.dedent("""\
    ---
    title: "0001. Test"
    status: Active
    # this is a YAML comment
    ---
    # 0001. Test
    """)
    result = extract_h1(content)
    assert result == "0001. Test"

The lint scripts use standard Python with pyyaml as the only dependency, so test setup should be straightforward. The scripts enforce core repo conventions and are modified as conventions evolve, making them high-value targets for coverage.

Severity: Medium — no production impact, but undetected lint regressions could allow convention violations to slip through.

---

Labels: This is a new feature request to add test coverage for existing lint tooling.