Skip to content

feat(coverage): per-file erosion alert in the sticky PR comment#63

Open
armstrongsamr wants to merge 1 commit into
LegionIO:mainfrom
armstrongsamr:feat/coverage-erosion-alert
Open

feat(coverage): per-file erosion alert in the sticky PR comment#63
armstrongsamr wants to merge 1 commit into
LegionIO:mainfrom
armstrongsamr:feat/coverage-erosion-alert

Conversation

@armstrongsamr

Copy link
Copy Markdown
Contributor

What

Add an advisory "Coverage erosion alert" section to the existing coverage
sticky comment posted by .github/workflows/coverage.yml. The alert fires
when any touched file's line coverage drops by ≥2 percentage points versus
the latest passing main run. The section is advisory (no CI gate is
added) and omitted entirely from the comment when no files qualify.

Three filters drop refactor-artifact noise:

  • Renamed/moved files detected at ≥90% similarity via
    git diff --find-renames=90 are skipped — baseline lookup would miss
    them anyway and the delta would be spurious.
  • Deleted files are skipped — head has no coverage row.
  • Pure-addition files (status A, or modifications with zero
    deletions) are skipped — a file that gained code without losing any
    cannot have lost coverage.

The 2pp threshold + conditional emission keep the signal high and the
comment-noise floor at zero on clean PRs.

Also amends docs/TESTING_ARCHITECTURE.md §"Why Reporting-Only Coverage"
with a new subsection documenting the alert and its rationale.

Worked examples

The alert fires on:

  • A PR that modifies a 70%-covered file and the new version comes in at
    65% lines (−5pp), because new branches were added without
    corresponding tests.
  • A PR that deletes the only test exercising one branch of an if/else
    in (e.g.) electron/agent/router.ts, dropping line coverage from 88%
    to 79% (−9pp) on that source file.

The alert never fires on:

  • A PR with no .ts / .tsx changes.
  • A PR that adds a new module (no baseline to compare).
  • A PR that renames foo.tsbar.ts (rename detector catches it).
  • A PR that adds only new lines to a file with zero deletions.
  • A PR where every touched file held its coverage within ±2pp.

What the alert does not catch

Four blind spots a reader should know about:

  1. Assertion-only deletion. Removing an expect(...) while keeping
    the surrounding it() block leaves the lines/branches still
    executed; line coverage does not move. For this class of regression,
    mutation testing is the right tool.
  2. Sub-threshold drift. A 1.99pp drop on a single file does not
    fire. Across many PRs, sub-threshold drops can aggregate into a
    double-digit slide that never produced an alert.
  3. Cross-PR scope-splitting. PR1 deletes the only assertion-bearing
    test on a file; PR2 adds a no-op test. Each PR alone passes the
    alert; sequenced, the regression lands.
  4. Renderer code in src/**. The unit slice
    (vitest.unit.config.ts) excludes renderer files; they have no
    baseline node in coverage-summary.json, so the alert path skips
    them silently. Renderer-only PRs produce no alerts.

Why

The existing doctrine ("report, don't gate") is preserved unchanged. The
alert addresses the one class of regression the existing "review-time
conversation" rule does not catch: a covered file silently rotting
because a PR removed the only test exercising a previously-covered
branch. The PR title might say "fix bug" with no hint that test
coverage shifted.

Touched-file scoping (the codecov / coveralls default) keeps the signal
local to the diff under review. The 2pp threshold is the floor where a
single missing line is unlikely to flip the verdict on a moderately-
covered file; the filters keep the alert quiet on refactors.

Policy lives in docs/TESTING_ARCHITECTURE.md §"Erosion alerts
(advisory, not blocking)"; this is a refinement of the existing
"Reporting-Only Coverage" doctrine, not a new direction. ADRs in this
repo are reserved for contested multi-week decisions; in-doc
amendment is the right shape for a refinement.

Depends on

#60 (coverage.all: true honesty fix). The alert reads
baselineSummary[file].lines.pct for each touched file. Files that
have no test on main today are absent from baselineSummary, and
the alert path's if (!baseNode) continue guard at coverage.yml:291
silently skips them. Without #60, ~102 of the 156 main-process files
are absent from the baseline and the alert quietly does nothing on
that two-thirds of the tree — presenting as "no erosion" when the
truth is "never measured." The alert is technically sound either way;
reversing the merge order degrades the signal rather than failing
loudly.

Checklist

  • pnpm lint passes
  • pnpm type-check passes
  • pnpm test passes (or note "no tests yet for this area" with rationale)
  • pnpm build succeeds
  • If this changes IPC, the preload bridge (electron/preload.ts) is updated and the type contract still holds
  • If this adds a new config section, desktopConfigPayload() allowlist is updated
  • Doc impact considered (README, CONTRIBUTING, AGENTS, CLAUDE)

Checklist notes

  • N/A on IPC: workflow + docs only, no preload bridge change.
  • N/A on config: no schema additions.
  • Doc impact: docs/TESTING_ARCHITECTURE.md amended in-line with the
    new subsection.
  • actionlint .github/workflows/coverage.yml clean (no new shellcheck
    findings introduced).
  • The github-script body parses as valid JS when wrapped in the same
    async-function shape github-script uses at runtime (node --check).
    Behavioural shape cannot be exercised locally without GitHub Actions
    context; the PR itself is the first real test.

Add an advisory erosion alert section to the existing coverage sticky
comment. The alert fires when any touched file's line coverage drops by
≥2 percentage points versus the latest passing main run. The alert is
*advisory* — no CI gate is added — and the section is omitted entirely
when no files qualify, so 90%+ of PRs see no change in the comment.

This is a refinement of the existing "report, don't gate" doctrine in
`docs/TESTING_ARCHITECTURE.md` §"Why Reporting-Only Coverage", not a
new gate. The doctrine survives intact; the alert addresses the one
class of regression the existing "review-time conversation" rule does
not catch — a covered file silently rotting when a PR adds an untested
branch or yanks the only assertion for an existing line.

## What changed in `coverage.yml`

The existing github-script block (lines ~149-163 before this change)
collected changed files by name only via `git diff --name-only`. It
now collects richer metadata via `git diff --name-status
--find-renames=90` + `git diff --numstat` so the alert filters can
distinguish:

- **Added (A)** — new file, no baseline to compare against. Skipped.
- **Renamed (R)** — detected at 90% similarity (stricter than git's
  default 50%). Baseline lookup would miss anyway and the delta is
  spurious. Skipped.
- **Deleted (D)** — head has no coverage row. Skipped.
- **Modified (M) with deletions == 0** — pure-addition residue from
  file-split refactors. Skipped (per the "refactor moves numbers"
  anti-threshold concern preserved from the existing doctrine).

The per-file table downstream (the existing collapsed `<details>`
block) is unchanged in behaviour — it reuses `changedFiles` as a
back-compat name derived from the new `changedFileMeta` array.

The baseline-fetch block at lines ~98-142 now stores the full
`baseSummary` as `baselineSummary` in addition to extracting `.total`
into `baseline`. The aggregate consumer is unchanged; the new
per-file consumer reads `baselineSummary[key].lines.pct`.

The alert section itself sits between the project-wide table and the
collapsed per-file details so the signal is visible without
expanding the details block. Max 5 rows, sorted by absolute drop
descending, with an "_…and N more file(s)_" footer if more qualify.

## What changed in `TESTING_ARCHITECTURE.md`

Added a subsection "Erosion alerts (advisory, not blocking)" inside
the existing "Why Reporting-Only Coverage" section, documenting:

- the 2pp threshold + conditional emission;
- the three filters (rename / delete / pure-addition) with their
  rationale grounded in the existing anti-threshold doctrine;
- the explicit non-goal "new modules with zero coverage are still
  a review-time conversation, not an alert";
- a sub-section "What the alert does not catch" pointing at
  mutation testing for the assertion-deletion case that coverage Δ
  cannot see.

No new ADR; the amendment lives in-doc because it is a refinement of
an existing policy, not a contested new direction.

## What is not changed

- No thresholds, no CI gate. The merge experience is unchanged on a
  PR that has no erosion.
- The coverage workflow's existing `continue-on-error` on the
  github-script step is preserved — a bug in the alert math (e.g. a
  malformed baseline artifact) cannot break the comment-upsert path.
- The unit slice config (`vitest.unit.config.ts`) is unchanged — the
  matching `coverage.all: true` honesty fix that the alert math
  assumes is a separate change.

## Verification

- `actionlint .github/workflows/coverage.yml` clean (no new
  shellcheck findings introduced by this diff).
- The github-script body parses as valid JS when wrapped in the same
  async-function shape github-script uses at runtime
  (`node --check`).
- A behavioural test on a draft PR is the next step before merge —
  the github-script body cannot be exercised locally without the
  full GitHub Actions context (`context`, `github`, `core`).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant