fix(ci): stop green-washing coverage-floor failures in suite_runner#10
Merged
Conversation
run_test_suite forced exit_code=0 whenever check_test_failures returned should_halt=False — but that helper returns False whenever failed_count==0, so a --cov-fail-under failure (every test passes, pytest still exits 1) was silently passed. A project dropping below its coverage floor would report PASS. Fix: only suppress the non-zero exit for *tolerated test failures* (failed_count>0 within the configured max). A non-zero exit with zero test failures (coverage-floor failure or internal pytest error) keeps its exit code. Two regression tests pin both paths. Also repairs the coupled test_run_infrastructure_tests_pipeline_smoke_real_subprocess, which previously relied on the green-wash: its synthetic repo could not import infrastructure, so the smoke subprocess exited 2 (collection error) and was masked to 0 — the test never actually ran anything. It now seeds trivial passing tests at the smoke paths, genuinely exercising the runner's path-resolution + subprocess + exit-code propagation.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The defect
infrastructure/reporting/suite_runner.pyforcedexit_code = 0whenevercheck_test_failuresreturnedshould_halt=False. But that helper returnsshould_halt=Falsewheneverfailed_count == 0— and a--cov-fail-underfailure is exactly every test passing while pytest exits 1. So a project dropping below its coverage floor reported PASS. (Found by Forge during the template_newspaper verification; empirically reproduced by revert-and-rerun.)The fix
Only suppress the non-zero exit for tolerated test failures (
failed_count > 0within the configured max). A non-zero exit with zero test failures — coverage-floor failure or an internal pytest error — keeps its exit code.Two regression tests pin both paths:
test_coverage_floor_failure_is_not_green_washed(exit 1 + 0 failed → stays 1)test_tolerated_test_failures_are_still_suppressed(exit 1 + 2 failed, max 5 → suppressed to 0)Coupled fix: a previously-vacuous test
test_run_infrastructure_tests_pipeline_smoke_real_subprocessonly passed because of the green-wash: its synthetic repo couldn'timport infrastructure, so the smoke subprocess exited 2 (collection error) and was masked to 0 — it never actually ran anything. It now seeds trivial passing tests at the smoke paths, genuinely exercising the runner's path-resolution + subprocess + exit-code propagation. (Symlinking the real smoke tests cascades into a conftest/helper dependency chain a synthetic repo can't satisfy.)Verification (run, not asserted)