Add find-untested-sources skill (C# parse-only test pairing)#733
Open
Evangelink wants to merge 1 commit into
Open
Add find-untested-sources skill (C# parse-only test pairing)#733Evangelink wants to merge 1 commit into
Evangelink wants to merge 1 commit into
Conversation
Parse-only Roslyn analysis that maps C# source files to the test files referencing their declared types and lists production sources with no referring test. No Compilation, no MetadataReferences, no binding. - File discovery prunes bin/obj/node_modules/.git and generated *.g.cs. - Test classification by .csproj suffix or test-SDK reference. - Source index records (ShortName, Namespace, FilePath) per parsed file. - Test scan walks IdentifierTokens, disambiguates strictly against the test file's using directives + enclosing namespace. - Suggests a test-file path by mirroring the source under the test project that already <ProjectReference>s the source's project. Runs in ~9s on a ~3,900 .cs-file repo (AITestAgent + msbench fixtures). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
Skill Coverage Report
|
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new dotnet-test skill, find-untested-sources, which performs parse-only Roslyn analysis to map C# production source files to test files that reference their declared types, and emits a JSON report highlighting unpaired (“untested”) sources.
Changes:
- Added
SKILL.mddocumentation describing intended usage, output schema, and limitations. - Added
scripts/Find-UntestedSources.cs, implementing the parse-only index + test-scan + JSON report pipeline.
Show a summary per file
| File | Description |
|---|---|
| plugins/dotnet-test/skills/find-untested-sources/SKILL.md | Documents the new skill’s purpose, usage, output schema, and limitations. |
| plugins/dotnet-test/skills/find-untested-sources/scripts/Find-UntestedSources.cs | Implements the analyzer and JSON report generation logic. |
Copilot's findings
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Files reviewed: 2/2 changed files
- Comments generated: 5
Comment on lines
+87
to
+90
| while (cur is not null && cur.Length >= root.Length) | ||
| { | ||
| var csproj = Directory.EnumerateFiles(cur, "*.csproj").FirstOrDefault(); | ||
| if (csproj is not null) |
Comment on lines
+424
to
+428
| static string GetNamespaceFor(SyntaxNode node) | ||
| { | ||
| var ns = node.Ancestors().OfType<BaseNamespaceDeclarationSyntax>().FirstOrDefault(); | ||
| return ns?.Name.ToString() ?? ""; | ||
| } |
Comment on lines
+211
to
+214
| if (d.Namespace.Length == 0) | ||
| { | ||
| continue; | ||
| } |
Comment on lines
+448
to
+450
| var map = new Dictionary<string, string>(StringComparer.OrdinalIgnoreCase); | ||
| var allCsproj = Directory.EnumerateFiles(root, "*.csproj", SearchOption.AllDirectories).ToList(); | ||
| foreach (var testProj in allCsproj) |
Comment on lines
+371
to
+394
| static bool IsSkippedFile(string path) | ||
| { | ||
| if (path.EndsWith(".g.cs", StringComparison.Ordinal)) | ||
| { | ||
| return true; | ||
| } | ||
| if (path.EndsWith(".Designer.cs", StringComparison.Ordinal)) | ||
| { | ||
| return true; | ||
| } | ||
| if (path.EndsWith(".AssemblyInfo.cs", StringComparison.Ordinal)) | ||
| { | ||
| return true; | ||
| } | ||
| if (path.EndsWith(".AssemblyAttributes.cs", StringComparison.Ordinal)) | ||
| { | ||
| return true; | ||
| } | ||
| if (path.EndsWith(".GlobalUsings.g.cs", StringComparison.Ordinal)) | ||
| { | ||
| return true; | ||
| } | ||
| return false; | ||
| } |
Contributor
|
👋 @Evangelink — this PR has 5 unresolved review thread(s). When you're ready, please address the feedback and push an update; the triage bot will pick up the next state automatically. (Add the |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
find-untested-sources, a parse-only Roslyn static analyzer that maps C# source files to the test files referencing their declared types, and lists production sources with no referring test.No
Compilation, noMetadataReferences, no binding — runs in ~9 s on a 3,900-file repo (measured locally onAITestAgent+ msbench fixtures).What it produces
A deterministic JSON report:
{ "counts": { "source_files": 3036, "test_files": 867, "untested_files": 1852, "paired_files": 1184 }, "untested": [ { "source": "src/Foo/Bar.cs", "decl_count": 8, "suggested_test_path": "tests/Foo.Tests/Bar/BarTests.cs" } ], "source_to_tests": { "src/Foo/Baz.cs": ["tests/Foo.Tests/BazTests.cs"] } }untestedordered by API surface (decl_count) descending → drop-in worklist for any test-generation agent.suggested_test_pathderived from real<ProjectReference>edges → lands in a project that already compiles against the source.How it works
.cs, prunebin/obj/.git/node_modules/.vs/packages/and generated files..csprojas test/source by name suffix (.Tests,.UnitTests, …) or SDK reference (Microsoft.NET.Test.Sdk,MSTest.Sdk,xunit,NUnit,TUnit,Microsoft.Testing.Platform).(ShortName, EnclosingNamespace, FilePath)per type declaration.usingdirectives + enclosing namespace (avoids the false-positive explosion that naive identifier matching causes on common names likeSettingsorContext).<ProjectReference>s the source's project.Honest limitations
Static parse-only → known gaps documented in
SKILL.md: reflection-driven tests, DI-resolved interface types not named in source, extension-method calls as instance methods,var/target-typednew()/pattern matching. For these, the skill points users atcoverage-analysis.Measurement context — honest restatement
This skill was prototyped in response to a benchmark observation: test-generation agents spend significant tokens manually pairing source ↔ test files via repeated
find/grep/globcalls before writing any tests. A 5×136-instance internal experiment compared a baseline (skill installed, no doc pointer) against an arm with a pointer to the helper added tocode-testing-agent's SKILL.md (the doc edit is proposed separately in #734).After re-doing the analysis correctly (per-task mean rather than volume-weighted aggregate, with a non-.NET control bucket):
Differential ≈ 8.6 pp, Welch's t ≈ −2.02 (right at p ≈ 0.05). Pass rate neutral within noise.
Caveats you should know about
code-testing-agentSKILL.md, not from the helper executing. The runtime value of this skill (i.e. what happens when it actually runs) is still unmeasured.This PR ships the skill alone. The doc integration is in PR #734.
Test plan
dotnet run scripts/Find-UntestedSources.cs -- <repo-root>produces well-formed JSON on multiple .NET repos (AITestAgent, ocelot, eShop).SKILL.mddocumentation.Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com