Skip to content

ci(codecov): wait for all coverage uploads before reporting status#497

Open
sandersaares wants to merge 3 commits into
mainfrom
sandersaares/codecov-after-n-builds
Open

ci(codecov): wait for all coverage uploads before reporting status#497
sandersaares wants to merge 3 commits into
mainfrom
sandersaares/codecov-after-n-builds

Conversation

@sandersaares

@sandersaares sandersaares commented Jun 12, 2026

Copy link
Copy Markdown
Member

The coverage job in main.yml runs a 3-OS matrix (ubuntu-latest, windows-latest, ubuntu-24.04-arm), each uploading coverage to Codecov independently. Codecov posted the codecov/project status as soon as the first upload arrived, before data from the other platforms had been received, which flagged us with a spurious "coverage check failed".

This sets codecov.notify.after_n_builds: 3 so Codecov holds the status until all coverage matrix jobs have uploaded and it has the full multi-platform dataset. Each matrix OS does a single upload (its comma-separated lcov files are merged into one report), so the value equals the number of OS entries in the matrix.

I added cross-referencing comments in both codecov.yml and the matrix in main.yml so maintainers keep the count in sync when adding/removing an OS.

Note: with after_n_builds: 3, if one coverage leg fails it uploads fewer than 3 reports and the status stays pending, blocking merge. This is acceptable since coverage is already a required check, so a failing leg must be fixed/re-run regardless.

The config was validated against https://codecov.io/validate (Valid!).

sandersaares and others added 3 commits June 12, 2026 14:23
The coverage job runs a 3-OS matrix, each uploading independently. Codecov posted the codecov/project status after the first upload, before data from the other platforms arrived, causing spurious coverage failures. Set codecov.notify.after_n_builds to 3 (the number of OS entries in the coverage matrix) so Codecov waits for all uploads before reporting, and add cross-referencing comments in codecov.yml and main.yml to keep the count in sync.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@sandersaares sandersaares marked this pull request as ready for review June 12, 2026 11:31
Copilot AI review requested due to automatic review settings June 12, 2026 11:32

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adjusts Codecov configuration so the codecov/project status isn’t reported until all OS legs of the coverage matrix have uploaded coverage, preventing premature/spurious failures on PRs.

Changes:

  • Add codecov.notify.after_n_builds: 3 to delay status reporting until all three matrix uploads arrive.
  • Add cross-referencing comments in codecov.yml and the coverage job matrix to keep the build count and OS list in sync.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File Description
codecov.yml Adds Codecov notify settings to wait for all expected coverage uploads before posting status.
.github/workflows/main.yml Documents the coupling between the coverage OS matrix size and codecov.notify.after_n_builds.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread codecov.yml
@codecov

codecov Bot commented Jun 12, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.9%. Comparing base (963e015) to head (01b8b47).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@          Coverage Diff          @@
##            main    #497   +/-   ##
=====================================
  Coverage   99.9%   99.9%           
=====================================
  Files        336     336           
  Lines      24829   24829           
=====================================
  Hits       24818   24818           
  Misses        11      11           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants