Microsoft.DotNet.Watch.UnitTests.FileWatcherTests.DeleteSubfolder has flaky test assertion

[ericstj]

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=1056990
Build error leg or test failing: dotnet-watch.Tests.dll.3.WorkItemExecution
Pull request: #49092

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "Failed Microsoft.DotNet.Watch.UnitTests.FileWatcherTests.DeleteSubfolder",
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Sample failing log: https://helixr1107v0xdeko0k025g8.blob.core.windows.net/dotnet-sdk-refs-pull-49092-merge-89284afdacb94b978c/dotnet-watch.Tests.dll.3/3/console.3de535af.log?helixlogtype=result

  Failed Microsoft.DotNet.Watch.UnitTests.FileWatcherTests.DeleteSubfolder(usePolling: True) [1 s]
  Error Message:
   Expected:
ChangedPath { Path = /tmp/dotnetSdkTests.igDIQiVU/True/subdir, Kind = Delete },
ChangedPath { Path = /tmp/dotnetSdkTests.igDIQiVU/True/subdir/foo1, Kind = Delete },
ChangedPath { Path = /tmp/dotnetSdkTests.igDIQiVU/True/subdir/foo2, Kind = Delete },
ChangedPath { Path = /tmp/dotnetSdkTests.igDIQiVU/True/subdir/foo3, Kind = Delete }
Actual:
ChangedPath { Path = /tmp/dotnetSdkTests.igDIQiVU/True/subdir, Kind = Update },
ChangedPath { Path = /tmp/dotnetSdkTests.igDIQiVU/True/subdir, Kind = Delete },
ChangedPath { Path = /tmp/dotnetSdkTests.igDIQiVU/True/subdir/foo1, Kind = Delete },
ChangedPath { Path = /tmp/dotnetSdkTests.igDIQiVU/True/subdir/foo2, Kind = Delete }

  Stack Trace:
     at Microsoft.DotNet.Watch.UnitTests.AssertEx.Fail(String message) in /_/test/dotnet-watch.Tests/Utilities/AssertEx.cs:line 214
   at Microsoft.DotNet.Watch.UnitTests.AssertEx.SequenceEqual[T](IEnumerable`1 expected, IEnumerable`1 actual, IEqualityComparer`1 comparer, String message, String itemSeparator, Func`2 itemInspector) in /_/test/dotnet-watch.Tests/Utilities/AssertEx.cs:line 169
   at Microsoft.DotNet.Watch.UnitTests.FileWatcherTests.TestOperation(String dir, ChangedPath[] expectedChanges, Boolean usePolling, Action operation) in /_/test/dotnet-watch.Tests/FileWatcherTests.cs:line 63
   at Microsoft.DotNet.Watch.UnitTests.FileWatcherTests.DeleteSubfolder(Boolean usePolling) in /_/test/dotnet-watch.Tests/FileWatcherTests.cs:line 353
--- End of stack trace from previous location ---
  Standard Output Messages:

Have a look at the assertion the test is making --

sdk/test/dotnet-watch.Tests/FileWatcherTests.cs

Lines 353 to 390 in 0934942

	await TestOperation(
	dir,
	expectedChanges: usePolling ?
	[
	new(subdir, ChangeKind.Delete),
	new(f1, ChangeKind.Delete),
	new(f2, ChangeKind.Delete),
	new(f3, ChangeKind.Delete),
	]
	: RuntimeInformation.IsOSPlatform(OSPlatform.OSX) ?
	[
	new(subdir, ChangeKind.Add),
	new(subdir, ChangeKind.Delete),
	new(f1, ChangeKind.Update),
	new(f1, ChangeKind.Add),
	new(f1, ChangeKind.Delete),
	new(f2, ChangeKind.Update),
	new(f2, ChangeKind.Add),
	new(f2, ChangeKind.Delete),
	new(f3, ChangeKind.Update),
	new(f3, ChangeKind.Add),
	new(f3, ChangeKind.Delete),
	]
	: RuntimeInformation.IsOSPlatform(OSPlatform.Windows) ?
	[
	new(subdir, ChangeKind.Update),
	new(subdir, ChangeKind.Delete),
	new(f1, ChangeKind.Delete),
	new(f2, ChangeKind.Delete),
	new(f3, ChangeKind.Delete),
	]
	:
	[
	new(subdir, ChangeKind.Delete),
	new(f1, ChangeKind.Delete),
	new(f2, ChangeKind.Delete),
	new(f3, ChangeKind.Delete),
	],

It's asserting it will see a series of events exactly on different operating systems. FileSystemWatcher (and file notification APIs in the platform for that matter) don't garuntee that. You can't count on seeing all changed/updated events. Especially in the case of polling you cannot garuntee you'll see an update event if it's followed by a delete since a polling implementation can only see a file or directory has been removed, and not see any changes that might have happened between the last poll and removal.

This test should be changed to just assert on seeing a minimum set of events, not exact set. In other words - a contains check rather than equals.

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=1056990
Error message validated: [Failed Microsoft.DotNet.Watch.UnitTests.FileWatcherTests.DeleteSubfolder]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/2/2025 8:08:17 PM UTC

Report

Build	Repository	Test	Pull Request
1461884	dotnet/sdk	Microsoft.DotNet.Watch.UnitTests.FileWatcherTests.DeleteSubfolder	#54726
1461879	dotnet/sdk	Microsoft.DotNet.Watch.UnitTests.FileWatcherTests.DeleteSubfolder	#54732
1461765	dotnet/sdk	Microsoft.DotNet.Watch.UnitTests.FileWatcherTests.DeleteSubfolder	#54748
1461532	dotnet/sdk	Microsoft.DotNet.Watch.UnitTests.FileWatcherTests.DeleteSubfolder	#54743
1460550	dotnet/sdk	Microsoft.DotNet.Watch.UnitTests.FileWatcherTests.DeleteSubfolder	#54661
1460375	dotnet/sdk	Microsoft.DotNet.Watch.UnitTests.FileWatcherTests.DeleteSubfolder	#54726
1460086	dotnet/sdk	Microsoft.DotNet.Watch.UnitTests.FileWatcherTests.DeleteSubfolder
1459664	dotnet/sdk	Microsoft.DotNet.Watch.UnitTests.FileWatcherTests.NewFile	#54708
1458122	dotnet/sdk	Microsoft.DotNet.Watch.UnitTests.FileWatcherTests.DeleteSubfolder	#54682
1456723	dotnet/sdk	Microsoft.DotNet.Watch.UnitTests.FileWatcherTests.DeleteSubfolder
1456698	dotnet/sdk	Microsoft.DotNet.Watch.UnitTests.FileWatcherTests.DeleteSubfolder	#54653
1455449	dotnet/sdk	Microsoft.DotNet.Watch.UnitTests.FileWatcherTests.DeleteSubfolder	#54127
1455000	dotnet/sdk	Microsoft.DotNet.Watch.UnitTests.FileWatcherTests.DeleteSubfolder	#54606
1454831	dotnet/sdk	Microsoft.DotNet.Watch.UnitTests.FileWatcherTests.DeleteSubfolder	#54518
1454694	dotnet/sdk	Microsoft.DotNet.Watch.UnitTests.FileWatcherTests.DeleteSubfolder	#54643
1445609	dotnet/sdk	dotnet-watch.Tests.dll.2.WorkItemExecution	#54555
1441133	dotnet/sdk	dotnet-watch.Tests.dll.6.WorkItemExecution	#54501
1440660	dotnet/sdk	dotnet-watch.Tests.dll.6.WorkItemExecution	#53945
1440543	dotnet/sdk	dotnet-watch.Tests.dll.6.WorkItemExecution	#54501
1440395	dotnet/sdk	dotnet-watch.Tests.dll.6.WorkItemExecution	#54445
1433748	dotnet/sdk	dotnet-watch.Tests.dll.6.WorkItemExecution
1432239	dotnet/sdk	dotnet-watch.Tests.dll.6.WorkItemExecution	#54386
1430148	dotnet/sdk	dotnet-watch.Tests.dll.6.WorkItemExecution
1429468	dotnet/sdk	dotnet-watch.Tests.dll.6.WorkItemExecution	#54355
1427239	dotnet/sdk	dotnet-watch.Tests.dll.6.WorkItemExecution	#54386

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
4	15	25