docs: add governance shell and close documentation drift gaps#142
Merged
Conversation
Every path:line pointer in the rescue log had drifted or was wrong: the regex character-class fix moved to support/TextContent.java, and the teardown quit() guard no longer sat at the cited line. Anchor each entry to a stable symbol or test name instead so the references survive edits.
All three stack READMEs mixed repo-root Compose commands with stack-local build commands without saying which directory each one needs. The Java README was outright broken: bare 'mvn test' cannot resolve a POM from the repository root. State the convention once per README and show the cd before the first stack-local command.
Nothing stated what a documentation-compliant change means here, so the bar lived only in review habit. Encode setup, the fast validation loop, the catalog workflow, the commitlint rules, and the five compliance conditions. Record the two deliberate governance gaps as waivers rather than leaving them to look like oversights: CODEOWNERS and reviewer-assignment machinery would both deadlock a single-maintainer repository. Both revisit on the first sustained external contributor.
This repo has no deployed service, no users, and no dependents, so the realistic risk is not an exploit: it is committing something that should never have been public, into a repository that is public on purpose. Scope the guidance to that. Contributors need to know not to commit credentials, not to point a stack at a real system whose traces would inherit its secrets, and to redact captures before attaching them. Notes explicitly that the demo app's tomsmith login is a published fixture rather than a secret, so the rule does not read as if the suite already breaks it.
The drift surfaces this repo cares about are the ones CI cannot see: a README embed copied by hand, a workflow guide mirrored by habit, and docs whose commands are edited but never run. Put them in front of the author as a checklist at the moment they can still act on it. Paths are inline code rather than links: GitHub pre-fills this file into the PR body, where relative links would not resolve.
Every relative link resolves today, so this is preventive: it keeps the cross-document navigation the guides depend on from rotting silently as files move. Relative links are checked strictly. Only two things are excluded, both unverifiable rather than merely inconvenient: the upstream demo app, whose outages must not fail unrelated pull requests, and the badge endpoints, which rate-limit per request. Loopback URLs are skipped because no CI job serves them. lychee is pinned by digest, matching the actionlint step.
Fifteen documents with no index, and no stated rule for which one wins when two disagree. Readers had to infer authority from tone. Name the order explicitly: workflows, then the catalog, then the generated matrix, then stack READMEs, then guides, with the root README deferring to all of them. Give every document an update trigger, since a trigger is what actually keeps a doc honest; the review cadence is only a backstop. Versioning stays in Git history rather than in per-document version lines that rot faster than the prose.
The model was only inferable by reading the checker and three stacks: one catalog leads, three stacks answer to it, they share no code on purpose, and a pinned demo app keeps failures meaningful. Record the reconciler's real limit rather than implying it proves more than it does: TypeScript and Python IDs are matched anywhere in a file, so an ID in a comment counts as coverage, while Java is anchored to the TestNG testName attribute. Python IDs live in docstrings, so the caveat is load-bearing, not hypothetical. Links to the CI guide rather than restating its job tables, per the authority order in the index.
CI regenerated and diffed the matrix file, but the README section 5 copy of it was maintained by hand and checked by nobody. It happened to match; nothing kept it matching. The checker now rebuilds the expected table from the catalog and compares it against the README embed, printing a unified diff on divergence so the fix is mechanical. It runs on a plain invocation too, so the documented fast loop catches this before CI does. Also replaces the loose dict(r["coverage"]) reads with a typed helper: a cast, so runtime behavior is identical, and the file is now clean under a type checker.
The CI guide mirrors the workflows down to step order, with nothing enforcing it. That is the repo's highest-risk doc: it stays plausible long after it stops being true. Commit 0f45a76 changed all five workflows and would have been blocked by this check. Compares the PR diff against the base and fails when .github/workflows/ moves without docs/ci-workflows.md. A [ci-guide-exempt] token in the PR title skips it, for edits with no semantic effect. The title is read in the if: expression only, never interpolated into the run block, so a crafted PR title cannot inject shell.
Five decisions shaped this repo and none were written down: the catalog as canonical model, the UI/HTTP split, three tracks with Cypress and WebdriverIO argued but not implemented, tagging unstable tests out of the gates instead of deleting them, and refusing to chain the workflows. Each was reconstructed from the commits that made it and dated to when it actually landed, not to today. The Consequences sections record the costs accepted, since a record that only lists benefits is advertising. Records are immutable: supersede, never edit.
Cross-links ran root-inward only: the README pointed at these four guides, and each was a dead end. A reader who arrived from search had no route onward and no way to learn the index existed. Each guide now ends with the index plus the two or three documents its readers actually need next.
Nothing in the matrix file said a machine had written it, so the only thing stopping someone from editing it by hand was noticing the prose line. Emit a marker directly after the H1 instead. The ✅ and — cells were also the sole encoding of state, with no key anywhere. Add a legend to the generated file and to the README copy, which carries the same emoji-only encoding.
The CI guide says what runs; nothing said what to do when it goes red. Each failure class now names the first diagnostic step, where the artifacts are, and a reproduction command that matches the READMEs. Two entries carry most of the value: which matrix legs failed is itself the diagnosis, and the nightly grid/local pair localizes a fault for free - grid red with local green is never test logic. Leads with the rule that the rest of the repo assumes: never fix a red gate by weakening it.
Nothing existed to copy, so each new document reinvented its own shape. The stack template encodes the fourteen-section contract the three existing stack READMEs only partly meet, which makes those gaps visible rather than debatable. Placeholders are backticked: bare <angle brackets> parse as inline HTML and fail markdownlint, which is also why the ADR template's placeholder link had to go.
The fast start is the first thing a new contributor runs, so a stale command there is the most expensive rot in the repo — and renaming an npm script or moving a directory broke it silently. Walks the block and checks everything it names still exists: the compose file parses, the cd targets are real, the npm scripts are defined. It starts no browser and installs nothing, so it stays a lint, not a test. Renaming test:chromium:smoke now fails the gate.
The repo had consistent conventions and no record of them, so they were only enforceable by whoever remembered. Each rule here exists because its absence caused something: bare line numbers had already rotted, the missing working directory made mvn test fail from the root, and the matrix emoji were the sole encoding of state. Six terms are pinned because they are near-synonyms in English but distinct here — notably stack versus track, and gate, which now means merge-blocking rather than merely red.
TypeScript and Python IDs were matched anywhere in a test file, so '// UI-JSERROR-001 is not implemented yet' in a comment was enough to satisfy the checker that the scenario was covered. Java was already anchored to the TestNG testName attribute. Anchor TypeScript to the test title and Python to the docstring, marker, or parametrize ID — the contexts every existing test already uses. Counts are unchanged before and after (48 rows, 27 java, 47 ts, 25 py), so this tightens the gate without reclassifying any coverage. The regex is still not a parser: a correctly titled test that asserts nothing still counts. The architecture guide says so.
The two waivers were already recorded; nothing said how a third would be added, which makes the register a one-off rather than a process. Two rules stop it becoming a way to opt out of anything inconvenient: the reason must say why the requirement is wrong here rather than merely expensive, and the revisit trigger must be an observable event rather than a date nobody checks.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull request
What changed
A documentation audit scored this repository 66/100. The strong parts were already here — the catalog reconciler, copyable execution paths, real tradeoff writing. What was missing was a governance shell sized for a solo maintainer, and enforcement for the two documents that restate a source of truth and could drift silently.
Nineteen commits, one per finding:
Corrective fixes. Every
path:linereference in the anti-patterns log had rotted — one pointed at the wrong file entirely, andUITest.java:186cited a file 126 lines long. They now anchor to stable symbols. All three stack READMEs mixed repo-root Compose commands with stack-local build commands without saying which directory each needs; the Java README was outright broken, since baremvn testcannot resolve a POM from the repository root.Governance. A contributing guide with the fast validation loop, the catalog workflow, the commitlint rules, and the five documentation-compliance conditions. A pull-request template. A documentation index that names the authority order — workflows, then catalog, then generated matrix, then stack READMEs, then guides — and gives every document an update trigger. An architecture overview. Five backfilled ADRs, dated from the commits that made each decision.
Enforcement. CI now fails on: a broken relative link, a README scenario-matrix embed that disagrees with the catalog, and a workflow change that does not update the CI guide. The README fast start is validated against the repository, so renaming an npm script breaks the build rather than the newcomer.
Two deliberate gaps recorded as waivers.
CODEOWNERSand reviewer-assignment machinery would both deadlock a single-maintainer repository. Both revisit on the first sustained external contributor.Security guidance is scoped to what this repository actually is: a teaching repo with no deployed service and no users. The realistic risk is committing a credential or an unredacted trace into a public repo, so that guidance lives in the contributing guide as contributor hygiene. There is no vulnerability disclosure policy, because there is nothing to disclose against.
How it was verified
Every gate was run locally, exactly as CI runs it:
Each new gate was proven to fail before being trusted, then reverted: a broken relative link fails lychee, a single flipped README matrix cell fails the embed check, and renaming
test:chromium:smokefails the fast-start check. Commit0f45a76— a real workflows-only change from this repository's history — would have been blocked by the new CI-guide guard.The scenario-ID matching was tightened in a separate commit: TypeScript and Python IDs were matched anywhere in a file, so
// UI-JSERROR-001 is not implemented yetin a comment counted as coverage. Matching is now anchored to test titles, docstrings, markers, and parametrize IDs. Counts are identical before and after, so no coverage was reclassified.No stack source was touched — only stack READMEs — so the smoke and per-stack static gates are unaffected by this change.
Checklist
scenarios/catalog.yml) updated, or this change adds or removes no scenarios.python3 tools/check-scenarios.py --write-matrix, and the README section 5 embed matches it..github/workflows/are mirrored indocs/ci-workflows.mdin this pull request.docs/adr/, or this decision is not durable enough to need one.