Skip to content

fix(ci): stop green-washing coverage-floor failures in suite_runner#10

Merged
docxology merged 1 commit into
mainfrom
fix/coverage-green-wash
Jun 2, 2026
Merged

fix(ci): stop green-washing coverage-floor failures in suite_runner#10
docxology merged 1 commit into
mainfrom
fix/coverage-green-wash

Conversation

@docxology
Copy link
Copy Markdown
Owner

The defect

infrastructure/reporting/suite_runner.py forced exit_code = 0 whenever check_test_failures returned should_halt=False. But that helper returns should_halt=False whenever failed_count == 0 — and a --cov-fail-under failure is exactly every test passing while pytest exits 1. So a project dropping below its coverage floor reported PASS. (Found by Forge during the template_newspaper verification; empirically reproduced by revert-and-rerun.)

The fix

Only suppress the non-zero exit for tolerated test failures (failed_count > 0 within the configured max). A non-zero exit with zero test failures — coverage-floor failure or an internal pytest error — keeps its exit code.

Two regression tests pin both paths:

  • test_coverage_floor_failure_is_not_green_washed (exit 1 + 0 failed → stays 1)
  • test_tolerated_test_failures_are_still_suppressed (exit 1 + 2 failed, max 5 → suppressed to 0)

Coupled fix: a previously-vacuous test

test_run_infrastructure_tests_pipeline_smoke_real_subprocess only passed because of the green-wash: its synthetic repo couldn't import infrastructure, so the smoke subprocess exited 2 (collection error) and was masked to 0 — it never actually ran anything. It now seeds trivial passing tests at the smoke paths, genuinely exercising the runner's path-resolution + subprocess + exit-code propagation. (Symlinking the real smoke tests cascades into a conftest/helper dependency chain a synthetic repo can't satisfy.)

Verification (run, not asserted)

  • Targeted: 32 infra tests pass (incl. both regressions + the repaired smoke test); ruff/format/mypy clean.
  • Full infra suite: 7178 passed, 0 failed — confirms no other test was relying on the masking.

run_test_suite forced exit_code=0 whenever check_test_failures returned
should_halt=False — but that helper returns False whenever failed_count==0, so a
--cov-fail-under failure (every test passes, pytest still exits 1) was silently
passed. A project dropping below its coverage floor would report PASS.

Fix: only suppress the non-zero exit for *tolerated test failures*
(failed_count>0 within the configured max). A non-zero exit with zero test
failures (coverage-floor failure or internal pytest error) keeps its exit code.

Two regression tests pin both paths. Also repairs the coupled
test_run_infrastructure_tests_pipeline_smoke_real_subprocess, which previously
relied on the green-wash: its synthetic repo could not import infrastructure, so
the smoke subprocess exited 2 (collection error) and was masked to 0 — the test
never actually ran anything. It now seeds trivial passing tests at the smoke
paths, genuinely exercising the runner's path-resolution + subprocess + exit-code
propagation.
@docxology docxology merged commit 15ea5a7 into main Jun 2, 2026
23 checks passed
@docxology docxology deleted the fix/coverage-green-wash branch June 2, 2026 14:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant