Skip to content

Add .codecov.yml with threshold-based coverage status checks#529

Open
khatchad wants to merge 2 commits into
wala:masterfrom
ponder-lab:add-codecov-config-upstream
Open

Add .codecov.yml with threshold-based coverage status checks#529
khatchad wants to merge 2 commits into
wala:masterfrom
ponder-lab:add-codecov-config-upstream

Conversation

@khatchad

Copy link
Copy Markdown
Collaborator

Summary

Adds a minimal .codecov.yml so Codecov's project/patch status checks tolerate small base-vs-head coverage drift instead of failing on every fractional drop. Mirrors the configuration in ponder-lab/ML.

Why

Without an explicit .codecov.yml, Codecov uses default target: auto with zero tolerance: any project-coverage decrease at all triggers codecov/project: FAILURE. Base-vs-head test-environment drift or flakiness can shave 1-2 lines off the project total and fail the gate even on PRs that don't touch Java code.

Concrete case: #516 (the POM cleanup mirror of ponder-lab/ML#272) has codecov/patch: SUCCESS (no Java changed, so the patch is trivially 100% covered) but codecov/project: FAILURE from a -0.03% project delta (-2 hits / +2 misses) that can't be attributed to the diff. The default Codecov policy fails on any non-zero negative delta; this configuration absorbs the noise.

What This Changes

coverage:
  status:
    project:
      default:
        target: auto
        threshold: 1%
        informational: false
    patch:
      default:
        target: auto
        threshold: 2%
        informational: false
github_checks:
  annotations: true
  • project threshold 1%—tolerates project-coverage drops of up to 1% before failing. Real regressions still fail.
  • patch threshold 2%—same idea for changed-line coverage.
  • github_checks.annotations: true—surfaces uncovered changed lines as inline annotations in the PR's "Files changed" view.

informational: false keeps the checks as real build signal rather than unconditionally green—the thresholds are the absorptive layer. The per-PR Codecov comment remains enabled by default (Codecov's default behavior with comment unspecified).

History Of The Equivalent ponder-lab/ML Config

For reference, the four commits that produced this state on the fork:

  • f7f610ff (#243): added .codecov.yml with informational: true (unconditionally green).
  • fd19bb91 (#263): disabled per-PR comment, enabled inline annotations.
  • ee247f0b (#264): flipped informational: false with the 1%/2% thresholds—"Now that there's no comment, we need to see the build failure."
  • 3eac854a (#266): re-enabled the per-PR comment.

This PR ports the current combined state in one commit.

Test Plan

🤖 Generated with Claude Code

Without an explicit `.codecov.yml`, Codecov uses default `target: auto` with zero tolerance: any project-coverage decrease at all triggers `codecov/project: FAILURE`. Base-vs-head test-environment drift or flakiness can shave 1-2 lines off the project total and fail the gate even on PRs that don't touch Java code (e.g., POM-only fixes—see PR wala#516 where a -0.03% project delta fails the gate despite the patch having no Java changes to cover).

Add a minimal `.codecov.yml` mirroring the configuration in `ponder-lab/ML`'s current state (commits `f7f610ff` / `fd19bb91` / `ee247f0b` / `3eac854a`):

* `coverage.status.project.default.threshold: 1%`—tolerate up to 1% project-coverage decrease before failing. Real regressions still fail; noise doesn't.
* `coverage.status.patch.default.threshold: 2%`—tolerate up to 2% patch-coverage decrease.
* `github_checks.annotations: true`—surface uncovered changed lines as inline diff annotations in the PR's "Files changed" view (alongside the per-PR Codecov comment, which remains enabled by default).

`informational: false` keeps the checks as real build signal rather than unconditionally-green—the thresholds are the absorptive layer.
Copilot AI review requested due to automatic review settings May 12, 2026 16:56

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a repository-level Codecov configuration to make codecov/project and codecov/patch status checks tolerant to small coverage fluctuations, reducing CI noise while keeping coverage regressions as real failures.

Changes:

  • Add .codecov.yml configuring project and patch coverage statuses to use target: auto with threshold tolerances (1% / 2%).
  • Enable github_checks.annotations so uncovered changed lines appear as inline PR annotations.

@khatchad khatchad enabled auto-merge (squash) May 12, 2026 16:58
@codecov

codecov Bot commented May 12, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 57.84%. Comparing base (33bbcf4) to head (813f4bc).

Additional details and impacted files
@@            Coverage Diff            @@
##             master     #529   +/-   ##
=========================================
  Coverage     57.84%   57.84%           
  Complexity      625      625           
=========================================
  Files           111      111           
  Lines          7690     7690           
  Branches        860      860           
=========================================
  Hits           4448     4448           
  Misses         3062     3062           
  Partials        180      180           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants