feat(coverage): per-file erosion alert in the sticky PR comment#63
Open
armstrongsamr wants to merge 1 commit into
Open
feat(coverage): per-file erosion alert in the sticky PR comment#63armstrongsamr wants to merge 1 commit into
armstrongsamr wants to merge 1 commit into
Conversation
Add an advisory erosion alert section to the existing coverage sticky comment. The alert fires when any touched file's line coverage drops by ≥2 percentage points versus the latest passing main run. The alert is *advisory* — no CI gate is added — and the section is omitted entirely when no files qualify, so 90%+ of PRs see no change in the comment. This is a refinement of the existing "report, don't gate" doctrine in `docs/TESTING_ARCHITECTURE.md` §"Why Reporting-Only Coverage", not a new gate. The doctrine survives intact; the alert addresses the one class of regression the existing "review-time conversation" rule does not catch — a covered file silently rotting when a PR adds an untested branch or yanks the only assertion for an existing line. ## What changed in `coverage.yml` The existing github-script block (lines ~149-163 before this change) collected changed files by name only via `git diff --name-only`. It now collects richer metadata via `git diff --name-status --find-renames=90` + `git diff --numstat` so the alert filters can distinguish: - **Added (A)** — new file, no baseline to compare against. Skipped. - **Renamed (R)** — detected at 90% similarity (stricter than git's default 50%). Baseline lookup would miss anyway and the delta is spurious. Skipped. - **Deleted (D)** — head has no coverage row. Skipped. - **Modified (M) with deletions == 0** — pure-addition residue from file-split refactors. Skipped (per the "refactor moves numbers" anti-threshold concern preserved from the existing doctrine). The per-file table downstream (the existing collapsed `<details>` block) is unchanged in behaviour — it reuses `changedFiles` as a back-compat name derived from the new `changedFileMeta` array. The baseline-fetch block at lines ~98-142 now stores the full `baseSummary` as `baselineSummary` in addition to extracting `.total` into `baseline`. The aggregate consumer is unchanged; the new per-file consumer reads `baselineSummary[key].lines.pct`. The alert section itself sits between the project-wide table and the collapsed per-file details so the signal is visible without expanding the details block. Max 5 rows, sorted by absolute drop descending, with an "_…and N more file(s)_" footer if more qualify. ## What changed in `TESTING_ARCHITECTURE.md` Added a subsection "Erosion alerts (advisory, not blocking)" inside the existing "Why Reporting-Only Coverage" section, documenting: - the 2pp threshold + conditional emission; - the three filters (rename / delete / pure-addition) with their rationale grounded in the existing anti-threshold doctrine; - the explicit non-goal "new modules with zero coverage are still a review-time conversation, not an alert"; - a sub-section "What the alert does not catch" pointing at mutation testing for the assertion-deletion case that coverage Δ cannot see. No new ADR; the amendment lives in-doc because it is a refinement of an existing policy, not a contested new direction. ## What is not changed - No thresholds, no CI gate. The merge experience is unchanged on a PR that has no erosion. - The coverage workflow's existing `continue-on-error` on the github-script step is preserved — a bug in the alert math (e.g. a malformed baseline artifact) cannot break the comment-upsert path. - The unit slice config (`vitest.unit.config.ts`) is unchanged — the matching `coverage.all: true` honesty fix that the alert math assumes is a separate change. ## Verification - `actionlint .github/workflows/coverage.yml` clean (no new shellcheck findings introduced by this diff). - The github-script body parses as valid JS when wrapped in the same async-function shape github-script uses at runtime (`node --check`). - A behavioural test on a draft PR is the next step before merge — the github-script body cannot be exercised locally without the full GitHub Actions context (`context`, `github`, `core`).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Add an advisory "Coverage erosion alert" section to the existing coverage
sticky comment posted by
.github/workflows/coverage.yml. The alert fireswhen any touched file's line coverage drops by ≥2 percentage points versus
the latest passing main run. The section is advisory (no CI gate is
added) and omitted entirely from the comment when no files qualify.
Three filters drop refactor-artifact noise:
git diff --find-renames=90are skipped — baseline lookup would missthem anyway and the delta would be spurious.
deletions) are skipped — a file that gained code without losing any
cannot have lost coverage.
The 2pp threshold + conditional emission keep the signal high and the
comment-noise floor at zero on clean PRs.
Also amends
docs/TESTING_ARCHITECTURE.md§"Why Reporting-Only Coverage"with a new subsection documenting the alert and its rationale.
Worked examples
The alert fires on:
65% lines (−5pp), because new branches were added without
corresponding tests.
if/elsein (e.g.)
electron/agent/router.ts, dropping line coverage from 88%to 79% (−9pp) on that source file.
The alert never fires on:
.ts/.tsxchanges.foo.ts→bar.ts(rename detector catches it).What the alert does not catch
Four blind spots a reader should know about:
expect(...)while keepingthe surrounding
it()block leaves the lines/branches stillexecuted; line coverage does not move. For this class of regression,
mutation testing is the right tool.
fire. Across many PRs, sub-threshold drops can aggregate into a
double-digit slide that never produced an alert.
test on a file; PR2 adds a no-op test. Each PR alone passes the
alert; sequenced, the regression lands.
src/**. The unit slice(
vitest.unit.config.ts) excludes renderer files; they have nobaseline node in
coverage-summary.json, so the alert path skipsthem silently. Renderer-only PRs produce no alerts.
Why
The existing doctrine ("report, don't gate") is preserved unchanged. The
alert addresses the one class of regression the existing "review-time
conversation" rule does not catch: a covered file silently rotting
because a PR removed the only test exercising a previously-covered
branch. The PR title might say "fix bug" with no hint that test
coverage shifted.
Touched-file scoping (the codecov / coveralls default) keeps the signal
local to the diff under review. The 2pp threshold is the floor where a
single missing line is unlikely to flip the verdict on a moderately-
covered file; the filters keep the alert quiet on refactors.
Policy lives in
docs/TESTING_ARCHITECTURE.md§"Erosion alerts(advisory, not blocking)"; this is a refinement of the existing
"Reporting-Only Coverage" doctrine, not a new direction. ADRs in this
repo are reserved for contested multi-week decisions; in-doc
amendment is the right shape for a refinement.
Depends on
#60 (
coverage.all: truehonesty fix). The alert readsbaselineSummary[file].lines.pctfor each touched file. Files thathave no test on
maintoday are absent frombaselineSummary, andthe alert path's
if (!baseNode) continueguard atcoverage.yml:291silently skips them. Without #60, ~102 of the 156 main-process files
are absent from the baseline and the alert quietly does nothing on
that two-thirds of the tree — presenting as "no erosion" when the
truth is "never measured." The alert is technically sound either way;
reversing the merge order degrades the signal rather than failing
loudly.
Checklist
pnpm lintpassespnpm type-checkpassespnpm testpasses (or note "no tests yet for this area" with rationale)pnpm buildsucceedselectron/preload.ts) is updated and the type contract still holdsdesktopConfigPayload()allowlist is updatedChecklist notes
docs/TESTING_ARCHITECTURE.mdamended in-line with thenew subsection.
actionlint .github/workflows/coverage.ymlclean (no new shellcheckfindings introduced).
async-function shape github-script uses at runtime (
node --check).Behavioural shape cannot be exercised locally without GitHub Actions
context; the PR itself is the first real test.