Skip to content

ci: aggregate cross-OS coverage via cargo-llvm-cov#168

Merged
bellini666 merged 2 commits into
masterfrom
ci/cross-os-coverage
Jun 4, 2026
Merged

ci: aggregate cross-OS coverage via cargo-llvm-cov#168
bellini666 merged 2 commits into
masterfrom
ci/cross-os-coverage

Conversation

@bellini666
Copy link
Copy Markdown
Owner

@bellini666 bellini666 commented Jun 4, 2026

Why

The coverage job ran cargo-tarpaulin on ubuntu-latest only, because tarpaulin is Linux-only (ptrace-based) and can't join the test matrix. Two problems followed:

  1. Coverage reflected Linux execution only. OS-specific branches were never counted, e.g. the cfg!(target_os = "macos") / "windows" arms in src/providers/mod.rs:90-102 and #[cfg(unix)] in scanner.rs. Codecov reported a Linux number, not the project's.
  2. The test suite ran twice per commit: once in test across all three OSes, then again from scratch under tarpaulin.

Changes

  • Swap tarpaulin for cargo-llvm-cov (source-based, cross-platform) and fold coverage into the test matrix, so tests run once per OS and emit coverage in the same pass.
  • Each OS uploads its own report tagged flags: os-${{ runner.os }}. Codecov merges the per-commit reports into one aggregated figure that includes the OS-specific branches. carryforward: true on each flag (in the new codecov.yml) keeps the merged number stable if one OS upload is missing, instead of letting it drop.
  • Drop the standalone coverage job and the now-redundant cargo build step (llvm-cov builds its own instrumented binary in a separate target dir).

The e2e tests that shell out to the binary now count toward coverage too: assert_cmd::Command::cargo_bin resolves CARGO_BIN_EXE_* to llvm-cov's instrumented build. Tarpaulin missed those.

Verification

Real check is this PR's own run: confirm the test job collects coverage on all three OSes, the old job is gone, Codecov receives three uploads, and the macOS/Windows arms in src/providers/mod.rs now show as covered.

Replace the Linux-only cargo-tarpaulin job with cargo-llvm-cov folded
into the test matrix. Coverage is now collected on Linux, macOS, and
Windows and merged by Codecov (after_n_builds: 3), so OS-specific
branches are counted. The test suite no longer runs twice per commit.
Copilot AI review requested due to automatic review settings June 4, 2026 21:46
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Jun 4, 2026

Dependency Review

The following issues were found:
  • ✅ 0 vulnerable package(s)
  • ✅ 0 package(s) with incompatible licenses
  • ✅ 0 package(s) with invalid SPDX license definitions
  • ⚠️ 1 package(s) with unknown licenses.
See the Details below.

License Issues

.github/workflows/ci.yml

PackageVersionLicenseIssue Type
taiki-e/install-action2.*.*NullUnknown License
Allowed Licenses: MIT, Apache-2.0, BSD-2-Clause, BSD-3-Clause, ISC, MPL-2.0, Unlicense, 0BSD, CC0-1.0, Zlib, CC-BY-4.0, Python-2.0, Unicode-3.0, Unicode-DFS-2016, BSL-1.0
Excluded from license check: pkg:cargo/rustpython-ast

OpenSSF Scorecard

PackageVersionScoreDetails
actions/taiki-e/install-action 2.*.* 🟢 7.4
Details
CheckScoreReason
Code-Review⚠️ 1Found 2/13 approved changesets -- score normalized to 1
Maintained🟢 1030 commit(s) and 25 issue activity found in the last 90 days -- score normalized to 10
Dangerous-Workflow🟢 10no dangerous workflow patterns detected
Packaging⚠️ -1packaging workflow not detected
Token-Permissions🟢 10GitHub workflow tokens follow principle of least privilege
Binary-Artifacts🟢 10no binaries found in the repo
CII-Best-Practices⚠️ 0no effort to earn an OpenSSF best practices badge detected
Pinned-Dependencies🟢 10all dependencies are pinned
Fuzzing⚠️ 0project is not fuzzed
License🟢 10license file detected
Signed-Releases⚠️ -1no releases found
Security-Policy🟢 10security policy file detected
Branch-Protection🟢 4branch protection is not maximal on development and all release branches
SAST🟢 10SAST tool is run on all commits

Scanned Files

  • .github/workflows/ci.yml

Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the CI pipeline to generate and upload cross-OS code coverage by replacing the prior Linux-only cargo-tarpaulin approach with cargo-llvm-cov integrated into the existing test matrix, enabling Codecov to aggregate coverage across Linux/macOS/Windows.

Changes:

  • Adds a codecov.yml configuration to define OS-specific flags and wait for 3 uploads before notifying.
  • Updates the test job matrix to install cargo-llvm-cov, run tests under coverage, and upload an LCOV report to Codecov with OS flags.
  • Removes the standalone Linux-only tarpaulin coverage job.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File Description
codecov.yml Adds Codecov config for OS-flagged uploads and aggregation timing (after_n_builds: 3).
.github/workflows/ci.yml Switches matrix tests to run under cargo-llvm-cov and upload per-OS LCOV reports; removes tarpaulin coverage job.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

- Use explicit taiki-e/install-action@v2 + tool: input instead of the
  tool-name ref shorthand (clearer, pins the action major).
- Add carryforward to per-OS flags so a flaky upload on one OS doesn't
  tank the merged number.
- Drop after_n_builds (left the coverage status pending if a matrix leg
  failed before uploading) and the redundant per-flag src/ paths.
- Add concurrency to cancel superseded CI runs.
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

Comment thread codecov.yml
@codecov
Copy link
Copy Markdown

codecov Bot commented Jun 4, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.84%. Comparing base (c848ff9) to head (0268406).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #168      +/-   ##
==========================================
+ Coverage   81.87%   90.84%   +8.96%     
==========================================
  Files          29       29              
  Lines        4684    10691    +6007     
==========================================
+ Hits         3835     9712    +5877     
- Misses        849      979     +130     
Flag Coverage Δ
os-Linux 90.77% <ø> (?)
os-Windows 90.21% <ø> (?)
os-macOS 90.78% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@bellini666 bellini666 merged commit 7f44c73 into master Jun 4, 2026
20 checks passed
@bellini666 bellini666 deleted the ci/cross-os-coverage branch June 4, 2026 21:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants