Add JSON output for skill-validator check#601
Conversation
Fixes dotnet#600 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
Note This PR is from a fork and modifies infrastructure files ( Changes to infrastructure typically need to be submitted from a branch in Please consider recreating this PR from an upstream branch. If you don't have push access to |
There was a problem hiding this comment.
Pull request overview
This PR adds a --json output mode to skill-validator check by refactoring the command to build a structured in-memory report first, then render either human-readable console output or a slim JSON payload suitable for downstream tooling.
Changes:
- Refactor
checkto generate a sharedCheckReportand render it to console or JSON (--json). - Introduce new check result/report models (
CheckReport,CheckJsonOutput, per-domain result types) and JSON source-gen support. - Update README usage guidance and extend check-command tests to cover JSON output and early failure scenarios.
Show a summary per file
| File | Description |
|---|---|
| eng/skill-validator/tests/Check/CheckCommandTests.cs | Adds JSON-output tests and console capture utilities; introduces an xUnit collection to prevent parallel console mutation. |
| eng/skill-validator/src/SkillValidatorJsonContext.cs | Registers CheckJsonOutput for source-generated JSON serialization and enables string-enum conversion. |
| eng/skill-validator/src/README.md | Documents the new --json flag and clarifies stdout behavior. |
| eng/skill-validator/src/Check/PluginProfiler.cs | Updates plugin validation to return the new PluginCheckResult model. |
| eng/skill-validator/src/Check/Models.cs | Adds new report/result models and the CheckOutputMode enum used by check. |
| eng/skill-validator/src/Check/CheckCommand.cs | Implements report building + rendering pipeline and adds --json flag handling. |
Copilot's findings
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comments suppressed due to low confidence (1)
eng/skill-validator/src/Check/CheckCommand.cs:329
- When no agents are discovered, this method records an error but returns
Result = 0. In the combined skills+agents flow, this can allow a successful exit code even though the report contains an error, which is especially problematic for--jsonconsumers. Returning a failure result (when agent paths were explicitly provided) or makingExitCodereflectGeneralErrorswould avoid this inconsistency.
if (allAgents.Count == 0)
{
if (agentPaths.Count > 0)
{
var searched = string.Join(", ", agentPaths.Select(p => $"\"{Path.GetFullPath(p)}\""));
builder.AddPlainError($"No agents found in the specified paths: {searched}");
}
return ([], [], 0);
}
- Files reviewed: 6/6 changed files
- Comments generated: 2
|
@aaronpowell this looks great. Happy to take it after you resolved the copilot comments. |
Address PR feedback by failing when explicit skill or agent paths discover nothing, including the combined check path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
@ViktorHofer done |
|
Tagging @ViktorHofer for a re-review. |
|
@copilot resolve the merge conflicts in this pull request |
|
@ViktorHofer - done |
| var message = $"Plugin '{pluginName}' aggregate description size is {totalChars:N0} characters — maximum is {SkillProfiler.MaxAggregateDescriptionLength:N0}."; | ||
| if (builder.Plugins.FirstOrDefault(p => string.Equals(p.Name, pluginName, StringComparison.Ordinal)) is { } pluginResult) | ||
| pluginResult.Errors.Add(message); | ||
| return 1; |
| if (string.Equals(fullCandidatePath, fullContainerPath, StringComparison.OrdinalIgnoreCase)) | ||
| return true; | ||
|
|
||
| if (File.Exists(fullContainerPath)) | ||
| fullContainerPath = Path.GetDirectoryName(fullContainerPath)!; | ||
|
|
||
| var normalizedContainer = fullContainerPath.TrimEnd(Path.DirectorySeparatorChar, Path.AltDirectorySeparatorChar) | ||
| + Path.DirectorySeparatorChar; | ||
|
|
||
| return fullCandidatePath.StartsWith(normalizedContainer, StringComparison.OrdinalIgnoreCase); |
| WriteIndented = true, | ||
| PropertyNamingPolicy = JsonKnownNamingPolicy.CamelCase, | ||
| PropertyNameCaseInsensitive = true, | ||
| DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull)] | ||
| DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull, | ||
| UseStringEnumConverter = true)] |
| foreach (var dependency in report.ExternalDependencies) | ||
| { | ||
| switch (dependency.Kind) | ||
| { | ||
| case "plugin": | ||
| if (plugins.FirstOrDefault(plugin => string.Equals(plugin.Name, dependency.Name, StringComparison.Ordinal)) is { } plugin) | ||
| plugin.Warnings.Add(new CheckJsonWarning("externalDependency", dependency.Message)); | ||
| break; | ||
| case "skill": | ||
| if (skills.FirstOrDefault(skill => string.Equals(skill.Name, dependency.Name, StringComparison.Ordinal)) is { } skill) | ||
| skill.Warnings.Add(new CheckJsonWarning("externalDependency", dependency.Message)); | ||
| break; | ||
| case "agent": | ||
| if (agents.FirstOrDefault(agent => string.Equals(agent.Name, dependency.Name, StringComparison.Ordinal)) is { } agent) | ||
| agent.Warnings.Add(new CheckJsonWarning("externalDependency", dependency.Message)); | ||
| break; | ||
| } | ||
| } |
| public void AddPlainError(string text) => GeneralErrors.Add(text); | ||
|
|
||
| public void AddGeneralError(string text) => GeneralErrors.Add(text); |
|
✅ Automated diff scan completed for This is an automated static analysis of the PR diff. Note 🔒 Integrity filter blocked 51 itemsThe following items were blocked because they don't meet the GitHub integrity level.
To allow these resources, lower tools:
github:
min-integrity: approved # merged | approved | unapproved | none
|
|
✅ Automated diff scan completed for This is an automated static analysis of the PR diff. Note 🔒 Integrity filter blocked 7 itemsThe following items were blocked because they don't meet the GitHub integrity level.
To allow these resources, lower tools:
github:
min-integrity: approved # merged | approved | unapproved | none
|
|
✅ Automated diff scan completed for This is an automated static analysis of the PR diff. Note 🔒 Integrity filter blocked 68 itemsThe following items were blocked because they don't meet the GitHub integrity level.
To allow these resources, lower tools:
github:
min-integrity: approved # merged | approved | unapproved | none
|
|
/evaluate |
|
@aaronpowell do you want to resolve the remaining Copilot feedback as well? |
Skill Validation Results
[1] (Plugin) Quality unchanged but weighted score is -7.8% due to: tokens (13217 → 30098), tool calls (0 → 1) Model: claude-opus-4.6 | Judge: claude-opus-4.6 🔍 Full Results - additional metrics and failure investigation steps
▶ Sessions Visualisation -- interactive replay of all evaluation sessions |
Summary
Fixes #600
Here's an example of the JSON output from running against awesome-copilot's skills list (truncated).
{ "counts": { "pluginCount": 0, "skillCount": 325, "agentCount": 0 }, "plugins": [], "skills": [ { "name": "acquire-codebase-knowledge", "path": "..\\awesome-copilot\\skills\\acquire-codebase-knowledge", "skillMdPath": "..\\awesome-copilot\\skills\\acquire-codebase-knowledge\\SKILL.md", "errors": [ "File reference \u0027assets/templates/STACK.md\u0027 is 2 directories deep \u2014 maximum is 1 level from SKILL.md.", "File reference \u0027assets/templates/STRUCTURE.md\u0027 is 2 directories deep \u2014 maximum is 1 level from SKILL.md.", "File reference \u0027assets/templates/ARCHITECTURE.md\u0027 is 2 directories deep \u2014 maximum is 1 level from SKILL.md.", "File reference \u0027assets/templates/CONVENTIONS.md\u0027 is 2 directories deep \u2014 maximum is 1 level from SKILL.md.", "File reference \u0027assets/templates/INTEGRATIONS.md\u0027 is 2 directories deep \u2014 maximum is 1 level from SKILL.md.", "File reference \u0027assets/templates/TESTING.md\u0027 is 2 directories deep \u2014 maximum is 1 level from SKILL.md.", "File reference \u0027assets/templates/CONCERNS.md\u0027 is 2 directories deep \u2014 maximum is 1 level from SKILL.md.", "File reference \u0027assets/templates/STACK.md\u0027 is 2 directories deep \u2014 maximum is 1 level from SKILL.md.", "File reference \u0027assets/templates/STRUCTURE.md\u0027 is 2 directories deep \u2014 maximum is 1 level from SKILL.md.", "File reference \u0027assets/templates/ARCHITECTURE.md\u0027 is 2 directories deep \u2014 maximum is 1 level from SKILL.md.", "File reference \u0027assets/templates/CONVENTIONS.md\u0027 is 2 directories deep \u2014 maximum is 1 level from SKILL.md.", "File reference \u0027assets/templates/INTEGRATIONS.md\u0027 is 2 directories deep \u2014 maximum is 1 level from SKILL.md.", "File reference \u0027assets/templates/TESTING.md\u0027 is 2 directories deep \u2014 maximum is 1 level from SKILL.md.", "File reference \u0027assets/templates/CONCERNS.md\u0027 is 2 directories deep \u2014 maximum is 1 level from SKILL.md." ], "warnings": [], "profile": { "name": "acquire-codebase-knowledge", "chars4TokenCount": 2270, "bpeTokenCount": 2169, "complexityTier": "detailed", "sectionCount": 12, "codeBlockCount": 3, "numberedStepCount": 23, "bulletCount": 15, "hasFrontmatter": true, "hasWhenToUse": false, "hasWhenNotToUse": false }, "profileLine": "acquire-codebase-knowledge: 2,169 BPE tokens [chars/4: 2,270] (detailed \u2713), 12 sections, 3 code blocks" }, { "name": "add-educational-comments", "path": "..\\awesome-copilot\\skills\\add-educational-comments", "skillMdPath": "..\\awesome-copilot\\skills\\add-educational-comments\\SKILL.md", "errors": [], "warnings": [], "profile": { "name": "add-educational-comments", "chars4TokenCount": 1556, "bpeTokenCount": 1346, "complexityTier": "detailed", "sectionCount": 17, "codeBlockCount": 2, "numberedStepCount": 9, "bulletCount": 49, "hasFrontmatter": true, "hasWhenToUse": false, "hasWhenNotToUse": false }, "profileLine": "add-educational-comments: 1,346 BPE tokens [chars/4: 1,556] (detailed \u2713), 17 sections, 2 code blocks" }, { "name": "adobe-illustrator-scripting", "path": "..\\awesome-copilot\\skills\\adobe-illustrator-scripting", "skillMdPath": "..\\awesome-copilot\\skills\\adobe-illustrator-scripting\\SKILL.md", "errors": [], "warnings": [ { "kind": "profile", "message": "Skill is 5,832 BPE tokens (chars/4 estimate: 5,794) \u2014 \u0022comprehensive\u0022 skills hurt performance by 2.9pp on average. Consider splitting into 2\u20133 focused skills." }, { "kind": "profile", "message": "No numbered workflow steps \u2014 agents follow sequenced procedures more reliably." } ], "profile": { "name": "adobe-illustrator-scripting", "chars4TokenCount": 5794, "bpeTokenCount": 5832, "complexityTier": "comprehensive", "sectionCount": 64, "codeBlockCount": 26, "numberedStepCount": 0, "bulletCount": 43, "hasFrontmatter": true, "hasWhenToUse": true, "hasWhenNotToUse": false }, "profileLine": "adobe-illustrator-scripting: 5,832 BPE tokens [chars/4: 5,794] (comprehensive \u2717), 64 sections, 26 code blocks" }, { "name": "agent-governance", "path": "..\\awesome-copilot\\skills\\agent-governance", "skillMdPath": "..\\awesome-copilot\\skills\\agent-governance\\SKILL.md", "errors": [], "warnings": [ { "kind": "profile", "message": "Skill is 4,203 BPE tokens (chars/4 estimate: 4,626) \u2014 approaching \u0022comprehensive\u0022 range where gains diminish." }, { "kind": "profile", "message": "No numbered workflow steps \u2014 agents follow sequenced procedures more reliably." } ], "profile": { "name": "agent-governance", "chars4TokenCount": 4626, "bpeTokenCount": 4203, "complexityTier": "standard", "sectionCount": 33, "codeBlockCount": 14, "numberedStepCount": 0, "bulletCount": 19, "hasFrontmatter": true, "hasWhenToUse": true, "hasWhenNotToUse": false }, "profileLine": "agent-governance: 4,203 BPE tokens [chars/4: 4,626] (standard ~), 33 sections, 14 code blocks" }, { "name": "agent-owasp-compliance", "path": "..\\awesome-copilot\\skills\\agent-owasp-compliance", "skillMdPath": "..\\awesome-copilot\\skills\\agent-owasp-compliance\\SKILL.md", "errors": [], "warnings": [ { "kind": "profile", "message": "Skill is 2,754 BPE tokens (chars/4 estimate: 2,970) \u2014 approaching \u0022comprehensive\u0022 range where gains diminish." } ], "profile": { "name": "agent-owasp-compliance", "chars4TokenCount": 2970, "bpeTokenCount": 2754, "complexityTier": "standard", "sectionCount": 22, "codeBlockCount": 7, "numberedStepCount": 10, "bulletCount": 43, "hasFrontmatter": true, "hasWhenToUse": false, "hasWhenNotToUse": false }, "profileLine": "agent-owasp-compliance: 2,754 BPE tokens [chars/4: 2,970] (standard ~), 22 sections, 7 code blocks" }, { "name": "agent-supply-chain", "path": "..\\awesome-copilot\\skills\\agent-supply-chain", "skillMdPath": "..\\awesome-copilot\\skills\\agent-supply-chain\\SKILL.md", "errors": [], "warnings": [ { "kind": "profile", "message": "Skill is 2,515 BPE tokens (chars/4 estimate: 2,677) \u2014 approaching \u0022comprehensive\u0022 range where gains diminish." }, { "kind": "profile", "message": "No numbered workflow steps \u2014 agents follow sequenced procedures more reliably." } ], "profile": { "name": "agent-supply-chain", "chars4TokenCount": 2677, "bpeTokenCount": 2515, "complexityTier": "standard", "sectionCount": 13, "codeBlockCount": 8, "numberedStepCount": 0, "bulletCount": 10, "hasFrontmatter": true, "hasWhenToUse": true, "hasWhenNotToUse": false }, "profileLine": "agent-supply-chain: 2,515 BPE tokens [chars/4: 2,677] (standard ~), 13 sections, 8 code blocks" }, { "name": "agentic-eval", "path": "..\\awesome-copilot\\skills\\agentic-eval", "skillMdPath": "..\\awesome-copilot\\skills\\agentic-eval\\SKILL.md", "errors": [], "warnings": [ { "kind": "profile", "message": "No numbered workflow steps \u2014 agents follow sequenced procedures more reliably." } ], "profile": { "name": "agentic-eval", "chars4TokenCount": 1466, "bpeTokenCount": 1358, "complexityTier": "detailed", "sectionCount": 16, "codeBlockCount": 8, "numberedStepCount": 0, "bulletCount": 13, "hasFrontmatter": true, "hasWhenToUse": true, "hasWhenNotToUse": false }, "profileLine": "agentic-eval: 1,358 BPE tokens [chars/4: 1,466] (detailed \u2713), 16 sections, 8 code blocks" }, { "name": "ai-prompt-engineering-safety-review", "path": "..\\awesome-copilot\\skills\\ai-prompt-engineering-safety-review", "skillMdPath": "..\\awesome-copilot\\skills\\ai-prompt-engineering-safety-review\\SKILL.md", "errors": [], "warnings": [ { "kind": "profile", "message": "No code blocks \u2014 agents perform better with concrete snippets and commands." } ], "profile": { "name": "ai-prompt-engineering-safety-review", "chars4TokenCount": 2540, "bpeTokenCount": 2185, "complexityTier": "detailed", "sectionCount": 19, "codeBlockCount": 0, "numberedStepCount": 20, "bulletCount": 104, "hasFrontmatter": true, "hasWhenToUse": false, "hasWhenNotToUse": false }, "profileLine": "ai-prompt-engineering-safety-review: 2,185 BPE tokens [chars/4: 2,540] (detailed \u2713), 19 sections, 0 code blocks" }, { "name": "ai-ready", "path": "..\\awesome-copilot\\skills\\ai-ready", "skillMdPath": "..\\awesome-copilot\\skills\\ai-ready\\SKILL.md", "errors": [], "warnings": [], "profile": { "name": "ai-ready", "chars4TokenCount": 516, "bpeTokenCount": 521, "complexityTier": "detailed", "sectionCount": 2, "codeBlockCount": 3, "numberedStepCount": 4, "bulletCount": 0, "hasFrontmatter": true, "hasWhenToUse": false, "hasWhenNotToUse": false }, "profileLine": "ai-ready: 521 BPE tokens [chars/4: 516] (detailed \u2713), 2 sections, 3 code blocks" }, { "name": "ai-team-orchestration", "path": "..\\awesome-copilot\\skills\\ai-team-orchestration", "skillMdPath": "..\\awesome-copilot\\skills\\ai-team-orchestration\\SKILL.md", "errors": [], "warnings": [], "profile": { "name": "ai-team-orchestration", "chars4TokenCount": 1400, "bpeTokenCount": 1436, "complexityTier": "detailed", "sectionCount": 13, "codeBlockCount": 5, "numberedStepCount": 17, "bulletCount": 12, "hasFrontmatter": true, "hasWhenToUse": true, "hasWhenNotToUse": false }, "profileLine": "ai-team-orchestration: 1,436 BPE tokens [chars/4: 1,400] (detailed \u2713), 13 sections, 5 code blocks" }, { "name": "appinsights-instrumentation", "path": "..\\awesome-copilot\\skills\\appinsights-instrumentation", "skillMdPath": "..\\awesome-copilot\\skills\\appinsights-instrumentation\\SKILL.md", "errors": [], "warnings": [ { "kind": "profile", "message": "No code blocks \u2014 agents perform better with concrete snippets and commands." }, { "kind": "profile", "message": "No numbered workflow steps \u2014 agents follow sequenced procedures more reliably." } ], "profile": { "name": "appinsights-instrumentation", "chars4TokenCount": 616, "bpeTokenCount": 547, "complexityTier": "detailed", "sectionCount": 9, "codeBlockCount": 0, "numberedStepCount": 0, "bulletCount": 7, "hasFrontmatter": true, "hasWhenToUse": true, "hasWhenNotToUse": false }, "profileLine": "appinsights-instrumentation: 547 BPE tokens [chars/4: 616] (detailed \u2713), 9 sections, 0 code blocks" }, { "name": "apple-appstore-reviewer", "path": "..\\awesome-copilot\\skills\\apple-appstore-reviewer", "skillMdPath": "..\\awesome-copilot\\skills\\apple-appstore-reviewer\\SKILL.md", "errors": [], "warnings": [ { "kind": "profile", "message": "No code blocks \u2014 agents perform better with concrete snippets and commands." } ], "profile": { "name": "apple-appstore-reviewer", "chars4TokenCount": 2336, "bpeTokenCount": 2047, "complexityTier": "detailed", "sectionCount": 36, "codeBlockCount": 0, "numberedStepCount": 8, "bulletCount": 127, "hasFrontmatter": true, "hasWhenToUse": false, "hasWhenNotToUse": false }, "profileLine": "apple-appstore-reviewer: 2,047 BPE tokens [chars/4: 2,336] (detailed \u2713), 36 sections, 0 code blocks" }, { "name": "arch-linux-triage", "path": "..\\awesome-copilot\\skills\\arch-linux-triage", "skillMdPath": "..\\awesome-copilot\\skills\\arch-linux-triage\\SKILL.md", "errors": [], "warnings": [ { "kind": "profile", "message": "No code blocks \u2014 agents perform better with concrete snippets and commands." } ], "profile": { "name": "arch-linux-triage", "chars4TokenCount": 232, "bpeTokenCount": 212, "complexityTier": "compact", "sectionCount": 4, "codeBlockCount": 0, "numberedStepCount": 6, "bulletCount": 8, "hasFrontmatter": true, "hasWhenToUse": false, "hasWhenNotToUse": false }, "profileLine": "arch-linux-triage: 212 BPE tokens [chars/4: 232] (compact \u2713), 4 sections, 0 code blocks" }, { "name": "architecture-blueprint-generator", "path": "..\\awesome-copilot\\skills\\architecture-blueprint-generator", "skillMdPath": "..\\awesome-copilot\\skills\\architecture-blueprint-generator\\SKILL.md", "errors": [], "warnings": [ { "kind": "profile", "message": "Skill is 2,519 BPE tokens (chars/4 estimate: 3,340) \u2014 approaching \u0022comprehensive\u0022 range where gains diminish." }, { "kind": "profile", "message": "No code blocks \u2014 agents perform better with concrete snippets and commands." }, { "kind": "profile", "message": "No numbered workflow steps \u2014 agents follow sequenced procedures more reliably." } ], "profile": { "name": "architecture-blueprint-generator", "chars4TokenCount": 3340, "bpeTokenCount": 2519, "complexityTier": "standard", "sectionCount": 18, "codeBlockCount": 0, "numberedStepCount": 0, "bulletCount": 104, "hasFrontmatter": true, "hasWhenToUse": false, "hasWhenNotToUse": false }, "profileLine": "architecture-blueprint-generator: 2,519 BPE tokens [chars/4: 3,340] (standard ~), 18 sections, 0 code blocks" }, { "name": "arduino-azure-iot-edge-integration", "path": "..\\awesome-copilot\\skills\\arduino-azure-iot-edge-integration", "skillMdPath": "..\\awesome-copilot\\skills\\arduino-azure-iot-edge-integration\\SKILL.md", "errors": [], "warnings": [ { "kind": "profile", "message": "No code blocks \u2014 agents perform better with concrete snippets and commands." } ], "profile": { "name": "arduino-azure-iot-edge-integration", "chars4TokenCount": 1171, "bpeTokenCount": 929, "complexityTier": "detailed", "sectionCount": 18, "codeBlockCount": 0, "numberedStepCount": 10, "bulletCount": 44, "hasFrontmatter": true, "hasWhenToUse": true, "hasWhenNotToUse": false }, "profileLine": "arduino-azure-iot-edge-integration: 929 BPE tokens [chars/4: 1,171] (detailed \u2713), 18 sections, 0 code blocks" } ], "agents": [] }