CI code coverage wrong

[tznind]

It looks like CI is not running all the tests

[BDisp]

Check if is some TestFixture that is using a class on a parameter. With xunit that happens and doesn't run the test at all.

[dodexahedron]

I'll flag this for follow-up.

Right now, I'm working in Terminal.Gui stuff that is going to have enough impact over here that I'd rather not fix something that ends up duplicating or obviating any work.

Now that the big blocker of the reformat push tig wanted to do is mostly done, I should be able to start making much quicker progress.

However, this is probably relevant to v1, yeah? Just want to be sure I can triage appropriately on my end.

[dodexahedron]

Check if is some TestFixture that is using a class on a parameter. With xunit that happens and doesn't run the test at all.

Good thinking. But, nunit doesn't have that issue. It's quite good with parameterized tests, and falls back to object if it can't figure it out, so they at least run, even if things get boxed or upcast.

Usually, it's related to either an attribute that causes it to be excluded without explicit mention in a filter for the test runner or something that makes the source generator skip the method, such as incorrect access modifiers or other syntactic stuff, or else a misbehaving or improperly referenced ValueSource or TestCaseSource, when those are in use.

[dodexahedron]

@tznind does a git blame on any particular commit result in better behavior?

Also, it's possible that the runner may just need to be tweaked, if this is only happening in CI runs. I seem to remember wanting to re-work that, in some work I was doing before I switched focus to TG v2 stuff. 🤔

[tznind]

this is probably relevant to v1, yeah?

This is only on v2 branch. Code coverage stat comes from a gist and theres only one so its 'last in first out' (not ideal but I can live with it).

@tznind does a git blame on any particular commit result in better behavior? [...]

Unfortunately the CI is only running on the merge commits so cannot narrow it down further.

It works locally though so its like the CI test runner is exiting early - but not returning an errror exit code. It shows the right number of tests found:

NUnit3TestExecutor discovered 1100 of 1100 NUnit test cases using Current Discovery mode, Non-Explicit run

But then it only reports running 396

I need to run the test on my linux machine to see if it is somehow OS related.

Run cd tests && dotnet test --configuration Debug --verbosity normal --collect:'XPlat Code Coverage' --settings coverlet.runsettings
Build started 02/10/2024 19:43:11.
     1>Project "/home/runner/work/TerminalGuiDesigner/TerminalGuiDesigner/tests/UnitTests.csproj" on node 1 (Restore target(s)).
     1>_GetAllRestoreProjectPathItems:
         Determining projects to restore...
 
[...]

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
NUnit Adapter 4.5.0.0: Test execution started
Running all tests in /home/runner/work/TerminalGuiDesigner/TerminalGuiDesigner/tests/bin/Debug/net8.0/UnitTests.dll
   NUnit3TestExecutor discovered 1100 of 1100 NUnit test cases using Current Discovery mode, Non-Explicit run
   
[...]  
   
Create_And_CreateT_ReturnEquivalentInstancesForSameInputs<ComboBox>(ComboBox()(0,0,0,0)): Multiple failures or warnings in test:
  1) View comparison only done by bounds check.
  2) View comparison only done by bounds check.
  3) View comparison only done by bounds check.
  4) Subviews not yet checked on ComboBox

[...]

  Passed MakeValidFieldName_ShouldPrependUnderscoreToReservedKeywords("void") [< 1 ms]
  Passed MakeValidFieldName_ShouldPrependUnderscoreToReservedKeywords("volatile") [< 1 ms]
  Passed MakeValidFieldName_ShouldPrependUnderscoreToReservedKeywords("while") [< 1 ms]
  Passed Test_GetUniqueFieldName_AfterAdding("bob","bob","bob2") [< 1 ms]
  Passed Test_GetUniqueFieldName_AfterAdding("blank","","blank2") [< 1 ms]

Test Run Successful.
Total tests: 402
     Passed: 396
    Skipped: 6
 Total time: 4.5190 Seconds
     1>Done Building Project "/home/runner/work/TerminalGuiDesigner/TerminalGuiDesigner/tests/UnitTests.csproj" (VSTest target(s)).

Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:06.66

[BDisp]

Here is a unit test which wont run even you comment any of the [Test]. They only passed using ReSharper. But why? Isn't only
dotnet dependent? I'm confused.