c8 + vitest: Some suites get covered other suites ...dont?

[osher]

c8 + vitest: Some suites get covered other suites ...dont?

I'm working on modernizing and open-sourcing some platform...
It's test strategy and test suite is built of two layers:

• test/e2e - end-to-end suites that uses child_process.exec to run the SUT as cli in multiple scenarios.
• test/unit - unit-level suites that add unit-tests to cover specific code-paths which are not covered by any of the e2e scenarios.

In the old setup I used nyc, but after modernizing it to recent version of node and refactoring from require to import I could not get nyc to produce coverage for the e2e suites. Then I found about c8.

Astonishingly, c8 produced coverage for the e2e right out of the box with zero configuration.
However, evidently, I was too fast to rejoice: It does not produce coverage for the unit tests... 🫨

The opened codebase is still young and does not take long to install or run - so I hope it can be used as a reproduction scenario...

Reproduction

0. Clone https://gitlab.com/prunjs/prun-core, use branch c8-bug-report.

1. install (npm i)

2. "vanilla" run - npm -w ecosystem/prun run cover
   

   • Note that 3 suites have run: 1 e2e and 2 unit tests.

3. run only the e2e suites: npm -w ecosystem/prun run cover -- test/e2e

   • expect: coverage numbers to be lesser than the vanilla run
   • found: coverage are exactly like the "vanilla" run (i.e the "vanilla" run gathers coverage only from the e2e part)

4. run only unit tests - npm -w ecosystem/prun run cover -- test/unit

   • expect: some coverage to be produced
   • found: The only covered file is vitest.config.js... no coverage is reported for any of the project files
     

I believe I could do a mish-mash where I ask both tools to produce coverage report in json for the parts it covers successfully, and ask either of the tools to merge them, but that would be a patch. I thought to ask here before I embark on something weird like that...

I hope it's just something silly I'm missing... donno...

🙏 thanks in advance 🙏

• Version:

10.1.2

• Platform:
  Reproduced in both
• fedora in WSL2 , with node v20.17.0, npm `v10.8.2'
• Win11 pro, with node v20.12.2, npm v10.5.0

[osher]

@bcoe - I'd really appreciate it if you could take a look, or direct me to who can help

Thanks for all the good work! :)

[cenfun]

You should use vitest coverage directly instead of c8 + vitest

vitest run --coverage

or c8 + mocha (or node test runner)

[bcoe]

You should use vitest coverage directly instead of c8 + vitest

@osher did running vitest's built in coverage support do the trick for you? I recommend this, because it's one less layer of indirection.

[tms0]

Hi @bcoe,

I have exactly the same issue. With vitest's built in coverage, only my unit tests are covered, and with c8 only my integration tests (that spawn an HTTP server in a child process) are covered.

Do you have an idea on how to get the best of both worlds ?