feat(agents): align RAI planner with guide, remove scoring, improve UX#1287
feat(agents): align RAI planner with guide, remove scoring, improve UX#1287WilliamBerryiii wants to merge 25 commits intomainfrom
Conversation
- restructure Phase 2 into binary trigger assessment with T1/T2/T3 tiers
- replace likelihood-impact scoring with restricted-use gate framework
- adopt AI STRIDE extensions with eight AI element types in Phase 4
- unify threat IDs to T-RAI-{NNN} format across all phases
- add rai-sensitive-uses-triggers.instructions.md for Phase 2 depth
- update collection manifests, plugins, and documentation accuracy
- add Sign-RaiArtifacts.ps1 signing script with Pester tests
Closes #1281
🚀 - Generated by Copilot
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.OpenSSF Scorecard
Scanned Files
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #1287 +/- ##
==========================================
- Coverage 87.66% 87.64% -0.02%
==========================================
Files 61 63 +2
Lines 9328 9576 +248
==========================================
+ Hits 8177 8393 +216
- Misses 1151 1183 +32
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
raymond-nassar
left a comment
raymond-nassar
left a comment
There was a problem hiding this comment.
First impressions on this draft PR:
The three-part disclaimer formulation exceeds the baseline requirement: (1) covers legal, compliance, and ethics; (2) frames all outputs as suggestions; (3) establishes organizational policy precedence.
Replacing numerical scores with qualitative concern levels and maturity indicators strengthens the disclaimer posture. Qualitative assessments naturally invite human judgment.
Look forward to seeing this in action.
raymond-nassar
left a comment
raymond-nassar
left a comment
There was a problem hiding this comment.
Files affected: All files containing the current disclaimer blockquote:
.github/agents/rai-planning/rai-planner.agent.md
.github/instructions/rai-planning/rai-identity.instructions.md (Session Start Display, Exit Point Reminder, State Creation)
.github/prompts/rai-planning/rai-capture.prompt.md
.github/prompts/rai-planning/rai-plan-from-prd.prompt.md
.github/prompts/rai-planning/rai-plan-from-security-plan.prompt.md
Current text:
This tool provides structured prompts and frameworks to support responsible AI planning. It is not a substitute for professional legal, compliance, or ethics review. All outputs are suggestions for human evaluation. Organizational RAI policies and applicable regulations take precedence.
Required text:
This agent is an assistive tool only. It does not provide legal, regulatory, or compliance advice and does not replace Responsible AI review boards, ethics committees, legal counsel, compliance teams, or other qualified human reviewers. The output consists of suggested actions and considerations to support a user's own internal review and decision‑making. All RAI assessments, sensitive use screenings, security models, and mitigation recommendations generated by this tool must be independently reviewed and validated by appropriate legal and compliance reviewers before use. Outputs from this tool do not constitute legal approval, compliance certification, or regulatory sign‑off.
Rationale:
The current disclaimer is too general and does not adequately protect against misinterpretation. The replacement version:
- Enumerates specific professional roles the tool does not replace (RAI review boards, ethics committees, legal counsel, compliance teams)
- Names the specific output types requiring validation (RAI assessments, sensitive use screenings, security models, mitigation recommendations)
- Explicitly excludes certification and sign-off — preventing misinterpretation as approval
- States a mandatory validation requirement ("must be independently reviewed and validated") rather than a softer suggestion
- Addresses regulatory advice directly ("does not provide legal, regulatory, or compliance advice")
- This change should be applied everywhere the current disclaimer blockquote appears, including the Session Start Display and Exit Point Reminder sections in rai-identity.instructions.md which govern when the disclaimer is shown to users.
- update disclaimer in rai-planner agent, three RAI prompt files, and rai-identity instructions - regenerate plugin outputs and reformat doc tables ⚖️ - Generated by Copilot
This has been resolved. |
|
The disclaimer text was strengthened across all existing locations in Given that the new verbatim text explicitly states "Outputs from this tool do not constitute legal approval, compliance certification, or regulatory sign‑off", the handoff document is arguably the most critical place for this disclaimer to appear, since it's the artifact most likely to be shared outside the immediate user session with review boards, legal, or leadership. Suggested change: add between the handoff header metadata and the Work Item Summary. |
Agree and I am addressing this now; but I am still trying to figure out how to do this tastefully. My main area of consideration here is that the handoff artifacts, in their "intended use" are to be persisted locally (not saying they WILL be, but as HVE Core is currently designed, the "intended use" is user specific local storage). Local artifacts, just like GHCP generating code or documentation generation are intended for human review. As currently designed the agent receivers of these handoff artifacts (the backlog agents) must be operationalized by the user explicitly, the artifacts passed to them, and they are backed by 3 operational modalities:
All three operational modalities have multi-step human interaction requirements, and there is no masked workflows happening. The human is identifying and selecting a specific agent for execution, and any delegation (only available to subagents), only has the bounded context of the original agent's human request. I think at the end of the day, the thing we really care about here from a system/agent operations perspective is to ensure that a human has reviewed the artifacts that are intended for other humans to consume. If we can agree on that, then I think the most important thing about AI generation attribution is to provide, as part of the disclaim footer, a markdown checkbox that offers, "has" or "has not" been reviewed by a human, with a default selection of "has not". This sends a much stronger signal, of attestation of review ... which is really what we are trying to set the behavioral outcome for. Consuming agents can also gate on this field ... as an added benefit. |
…system - remove human review checkbox from agentic artifacts (Control Surface Catalog, Evidence Register) in impact assessment instructions - remove superseded qualifier prose from RAI Review Summary template in backlog handoff instructions - add Artifact Attribution and Review section to handoff pipeline docs with footer classification table - add footer classification notes to Phase 5 and Phase 6 outputs in phase reference docs - add conversational vs persisted disclaimer note to agent overview docs 📝 - Generated by Copilot
Artifact Attribution and Review — Summary of ChangesThank you for the thorough review, @raymond-nassar. We implemented a two-tier attribution system across the RAI planning artifact templates and added documentation to support the new conventions. Tier 1: AI-Content NoteAll persisted artifacts now include an AI-content note adapted from the Microsoft Learn per-article pattern:
Tier 2: Full DisclaimerThe handoff summary — the primary stakeholder-facing deliverable — includes the complete verbatim disclaimer after the AI-content note and human review checkbox. Human Review CheckboxHuman-facing artifacts include a review checkbox to track validation status:
Artifact Classification
Control Surface Catalog and Evidence Register are classified as agentic artifacts (consumed by later pipeline phases) and receive only the AI-content note. All other artifacts are human-facing and include the review checkbox. Files ChangedInstruction files:
Documentation:
|
- add centralized config files for disclaimers and footers with human review - add JSON schema for config file validation - add Validate-AIArtifacts.ps1 with scope-filtered artifact classification - add Pester tests (20 passing) for all validation functions - add ai-artifact-validation.yml reusable workflow gated in pr-validation - add lint:ai-artifacts and validate:ai-artifacts npm scripts 🔧 - Generated by Copilot
AI Artifact Validation Infrastructure — Update SummaryThis latest commit ( What was added
Expansion plansThe current scope filter targets
The goal is for each focus area to opt in by adding its scope patterns to the config files — keeping validation centralized while allowing incremental adoption across the codebase. 🔧 - Generated by Copilot |
Correction to previous commentThe scope in Updated scope: Updated expansion plan: When other focus areas (e.g., Validation results after fix: 2 files / 7 issues (down from 3 files / 9 issues), all within 🔧 - Generated by Copilot |
…iling newline Security instruction files were not modified in this PR. Remove .github/instructions/security/** scope patterns from footer-with-review.yml to avoid false positive validation warnings. Fix missing trailing newline in package.json. 🔧 - Generated by Copilot
- switch Find-ArtifactReferences from content-based to filename-based matching - update Pester tests for filename-based artifact matching logic - fix BOM encoding and empty catch blocks for PSScriptAnalyzer - update Docusaurus collection card counts for rai-planning - auto-fix markdown table formatting in handoff-pipeline 🔧 - Generated by Copilot
… consistency - prettify tables in 5 rai-planning instruction files using LF-normalized input - resolves CRLF/LF formatting divergence between Windows and Linux CI 🎨 - Generated by Copilot
…ess characteristics - replace 6 Microsoft RAI principles with 7 NIST trustworthiness characteristics - rewrite standards, identity, impact assessment, security model, and backlog handoff - update agent definition, prompts, and documentation for NIST alignment - migrate state schema to riskClassification - regenerate plugin outputs 🏛️ - Generated by Copilot
Re: Disclaimer text update (5 files)Addressed. All 5 files updated with the expanded disclaimer text. The updated disclaimer replaces the generic 1-line AI notice with a multi-sentence version specifying professional review requirements and output types (risk assessments, security models, planning artifacts). One modification from the suggested text: "sensitive use screenings" updated to "risk classification screenings" to align with the terminology redesign across this PR.
|
|
Thank you for the review! All feedback incorporated — see inline replies on each comment thread for specific before/after details and file references. |
NIST AI RMF 1.0 Sole Embedded Standard — Complete AlignmentThis PR migrates the RAI Planner from Microsoft's 6 proprietary RAI principles to NIST AI RMF 1.0's 7 trustworthiness characteristics as the sole embedded standard. All proprietary framework content has been removed. 29 files changed | 953 insertions | 744 deletions | 2 commits Architecture Changes
NIST 7 Trustworthiness Characteristics
Terminology Changes
Files Changed
Breaking Changes
Validation
Comment Resolution Checklist
|
- add author: Microsoft to 6 RAI planning doc pages for CI frontmatter validation - format tables in 4 RAI planning instruction files - format tables in 4 RAI planning doc files - fix symlink index for 2 plugin instruction files 🔧 - Generated by Copilot
🎨 - Generated by Copilot
…structions file - create disclaimer-language.instructions.md as single source of truth - replace inline CAUTION blockquotes with #file: references in agent files - add Startup sections with disclaimerShownAt logic in prompt files - correct step counts in session resume and post-summarization recovery - remove duplicate inline blockquotes from all prompt and agent files 🔧 - Generated by Copilot
Summary of Changes (
|
raymond-nassar
left a comment
raymond-nassar
left a comment
There was a problem hiding this comment.
Sharing three clarification questions from my review of the latest RAI planner changes.
There was a problem hiding this comment.
Can you clarify whether the shared disclaimer language is intentionally still using "sensitive use screenings"? I noticed the broader RAI planner updates appear to rename that phase and related terminology to "risk classification," so I wanted to check whether the shared startup language should also be updated for consistency.
There was a problem hiding this comment.
I'll remove that section and revert to the risk screenings. I had gone back to the original drafted disclaim language but will update that.
There was a problem hiding this comment.
Fixed — the shared disclaimer language in disclaimer-language.instructions.md now consistently uses "risk classification screenings" instead of "sensitive use screenings." This aligns with the broader rename from rai-sensitive-uses-triggers → rai-risk-classification and the NIST AI RMF 1.0 terminology used across the rest of the planner instructions.
There was a problem hiding this comment.
Can you clarify whether the startup experience is still expected to include both the disclaimer and the source context? In the earlier version, the RAI planner startup text referenced the Microsoft RAI Standard and NIST AI RMF directly, but after centralizing the disclaimer into the shared file, I only see the caution/disclaimer language reflected in the startup flow. I wanted to check whether that source context was intentionally removed, or whether it still needs to be preserved somewhere in the startup experience for consistency with the identity guidance.
There was a problem hiding this comment.
I'll review if we should add the source language back and do it based on if the user has overridden the NIST default config.
There was a problem hiding this comment.
Fixed — the Startup Announcement section now explicitly references both disclaimer-language.instructions.md (for the caution/disclaimer display) and rai-identity.instructions.md (for framework attribution), following the Session Start Display protocol. This restores the source context that was lost when the disclaimer was centralized into the shared file. The startup flow now surfaces both the disclaimer and the NIST AI RMF / configured framework attribution before presenting entry modes.
There was a problem hiding this comment.
I may be reading this incorrectly, but can you clarify whether the riskClassification state shape is intended to be the same across the RAI planner sources? I noticed some files describe riskClassification.framework as a string and use indicatorResults, while the risk-classification instruction appears to model framework as an object and uses indicators, activatedCount, and riskScore. Since the planner persists and resumes from state.json, should these definitions be aligned to a single canonical schema before merge?
There was a problem hiding this comment.
More than likely ... I think the state file is gonna get a bit of a revision when i back propagate this arch to the SSSC and Security planners ... but I'll at least align it for now.
There was a problem hiding this comment.
Fixed — the riskClassification state shape is now aligned to a single canonical schema across all RAI planner sources. The canonical shape uses riskClassification.indicators with dynamic keys (safety_reliability, rights_fairness_privacy, security_explainability), each containing an .activated boolean. Top-level fields include activatedCount and suggestedDepthTier. The old sensitiveUsesTriggers / indicatorResults / triggeredCount references have been migrated.
Additionally, a formal JSON Schema was created at scripts/linting/schemas/rai-state.schema.json (462 lines) that provides authoritative validation for the entire state.json structure — including the riskClassification section. This schema is integrated via VS Code json.schemas settings in .vscode/settings.json, so state files get real-time validation in the editor.
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Advisory review — this PR is from a maintainer. Findings are informational only.
Review Overview
This PR delivers a well-structured, comprehensive upgrade to the RAI Planner system: 6-phase restructuring, scoring removal, guide alignment, and a new config-driven AI artifact validation pipeline. The scope expansion beyond the 11-file issue estimate is clearly intentional and well-explained. Overall the change is solid; the findings below are advisory and intended to improve correctness and convention compliance before merge.
Issue Alignment
✅ Closes #1281 — all five acceptance criteria are addressed:
- References updated to the Impact Assessment Guide ✅
- Disclaimer present (moved to shared
disclaimer-language.instructions.md— satisfies the spirit of the requirement) ✅ - UX improvements implemented ✅
- Output preferences working ✅
- Validation passes per the Testing table ✅
Validate-AIArtifacts.ps1 pipeline. Consider whether the issue should be updated or a follow-up issue created to capture the new validation infrastructure.
PR Template Compliance
npm run plugin:generate and npm run docs:test. Given that plugin generation was explicitly run and passed (per the Testing table), these should be present and checked.
/prompt-analyze, prompt-builder feedback, verified common standards). The corresponding AI Artifact type checkboxes are checked. If the prompt-builder review was not performed, these items should remain unchecked (they are manual-only), but reviewers should be aware.
Coding Standards
See inline comments for:
ai-artifact-validation.ymlline 52 — invertedif:condition causes soft-fail mode to silently ignore all validation failures (functional bug)ai-artifact-validation.ymlline 23 — checkout SHAde0fac2e...differs from the repository-wide pinned SHA11bd7190...for the samev4.2.2tagscripts/linting/Validate-AIArtifacts.ps1line 1 — missing#!/usr/bin/env pwshshebang and script-level comment-based help blockscripts/linting/schemas/ai-artifact-config.schema.jsonline 3 —$idvalue isfooter-config.schema.json(old name) but file isai-artifact-config.schema.jsonscripts/security/Sign-RaiArtifacts.ps1line 83 — main execution not wrapped in invocation guard (if ($MyInvocation.InvocationName -ne '.')).github/workflows/copilot-setup-steps.ymlline 153 —sha256sumformat uses two spaces here but one space inon-create.sh; the single-space form inon-create.shis non-standard
Code Quality
✅ The Validate-AIArtifacts.ps1 script is well-structured: proper module imports, function decomposition, invocation guard, CI annotation integration, and JSON output.
✅ Sign-RaiArtifacts.ps1 gracefully handles missing cosign with a warning rather than a hard failure — appropriate for an optional signing feature.
✅ The JSON Schema for config validation (ai-artifact-config.schema.json) is thorough with additionalProperties: false throughout and a oneOf discriminator. The $id mismatch (see inline comment) is the only issue.
💡 lint:ai-artifacts vs validate:ai-artifacts in package.json — Both targets call the same script but with different strictness (-FailOnMissing vs no flag). The convention in this repo uses lint:* for enforcement and validate:* for optional checks, but having both targets for the same script with inverted roles may confuse contributors. Consider adding a comment in package.json or aligning naming with the existing pattern (validate:* is the non-failing form).
Action Items (Advisory)
| Priority | Finding | Location |
|---|---|---|
| 🔴 Medium | Inverted if: on "Check results" step |
.github/workflows/ai-artifact-validation.yml:52 |
| 🟡 Low | Checkout SHA mismatch | .github/workflows/ai-artifact-validation.yml:23 |
| 🟡 Low | Missing shebang + help block | scripts/linting/Validate-AIArtifacts.ps1:1 |
| 🟡 Low | Schema $id mismatch |
scripts/linting/schemas/ai-artifact-config.schema.json:3 |
| 🟡 Low | Missing invocation guard | scripts/security/Sign-RaiArtifacts.ps1:83 |
| i️ Info | sha256sum single-space in on-create.sh |
.devcontainer/scripts/on-create.sh |
| i️ Info | Missing PR template sections | PR description |
- add activeFrameworkProfile.name attribution to agent and prompt startup announcements - enhance identity instructions with custom framework announcement guidance - correct disclaimer wording from sensitive use to risk classification screenings 🔧 - Generated by Copilot
…cross all files - replace activeFrameworkProfile with framework object in identity and agent JSON blocks - update prose references from activeFrameworkProfile to framework in 6 files - align indicators shape, activatedCount, riskScore, and suggestedDepthTier defaults - remove legacy screeningCompleted, customFrameworkSource, and prohibitedUsesGateResult fields 🔧 - Generated by Copilot
- rename raiRiskSurfaceStarted to securityModelAnalysisStarted in phase-reference - fix suggestedDepthTier nesting in phase-reference - refresh State Fields table in agent-overview - remove duplicate referencesProcessed entry from agent file - regenerate plugins 📝 - Generated by Copilot
…/microsoft/hve-core into feat/rai-planner-guide-alignment
There was a problem hiding this comment.
Advisory Review — PR #1287 feat/rai-planner-guide-alignment
Advisory review: this PR is from a maintainer. Findings are informational only.
Overview
This PR adds substantial RAI planner infrastructure: 11 updated RAI agent/instruction/prompt files (the stated goal), plus a full AI artifact validation pipeline (Validate-AIArtifacts.ps1, ai-artifact-validation.yml, JSON schema, configuration YAML, Pester tests) and artifact-signing capability (Sign-RaiArtifacts.ps1). The core work is well-structured; a handful of convention and logic issues are noted below.
📋 Issue Alignment
Issue #1281 scoped this work to 11 RAI-related files. The PR changes ~50 files, adding an entire validation and signing pipeline not mentioned in the issue. This is likely intentional follow-up work, but the issue description was never updated to reflect the expanded scope. Consider either closing #1281 with a note about the additional scope, or filing a follow-up issue to track the validation pipeline separately for backlog clarity.
No missing requirements were identified — the linked issue's stated work appears fully addressed.
📝 PR Template Compliance
The following sections of the PR description have items that appear to require attention:
- GHCP Maturity Acknowledgment: Both checklist items are unchecked. Since the PR modifies
.agent.md,.prompt.md, and.instructions.mdfiles, the acknowledgment checkboxes should be checked if that work is intentional. - AI Artifact Contributions checklist (under "Checklist → AI Artifact Contributions"): All three items are unchecked (
/prompt-analyzereview, addressing prompt-builder feedback, verifying contribution follows standards). These are process requirements for AI artifact changes; if the steps were performed, please check the boxes. - Required Automated Checks: All seven items are unchecked. These are self-reported results of running validation commands — please check them off once the validation passes locally.
🔍 Coding Standards
Three inline comments have been posted on specific files:
| Finding | File | Severity |
|---|---|---|
Missing #!/usr/bin/env pwsh shebang |
scripts/linting/Validate-AIArtifacts.ps1:1 |
Low |
Missing invocation guard + $PWD path assumption |
scripts/security/Sign-RaiArtifacts.ps1:84 |
Medium |
Dead code Check results step |
.github/workflows/ai-artifact-validation.yml:51 |
Medium |
The invocation guard finding is the most impactful: the test suite currently works around its absence using [System.Management.Automation.Language.Parser] AST extraction — a fragile approach that will require maintenance if function signatures change. Adding the guard allows standard dot-source testing and aligns with every other script in the repo.
✅ Code Quality Positives
- SHA pinning: All third-party actions in
ai-artifact-validation.ymlandcopilot-setup-steps.ymlare pinned to full commit SHAs with semantic version comments. ✅ - Cosign SHA consistency: Identical SHAs used in both
copilot-setup-steps.ymlandon-create.sh. ✅ Validate-AIArtifacts.ps1: 612-line script is well-structured — proper copyright,[CmdletBinding()],$ErrorActionPreference = 'Stop', invocation guard, and#regionorganization. The invocation guard pattern here is the modelSign-RaiArtifacts.ps1should follow.package.json:lint:ai-artifactscorrectly wired intolint:all. Having both an enforcing (-FailOnMissing) and advisory (no flag) variant makes the distinction between CI-blocking and local-advisory validation explicit — a reasonable design..github/config/disclaimers.yml: Single-source-of-truth for disclaimer text consumed by the validator is a clean design. Prevents drift between the YAML config and hard-coded strings.- Pester test coverage for both
Validate-AIArtifacts.ps1andSign-RaiArtifacts.ps1is appreciated and thorough.
💡 Minor Observation — package.json script naming
validate:ai-artifacts and lint:ai-artifacts do the same thing with different failure modes. The validate: prefix in this repo is used for tools that are also run in CI checks (validate:copyright, validate:skills). The non-enforcing variant here is purely advisory, so callers need to remember which script has which behavior. A brief comment in the PR description or inline in package.json (as a README note) explaining the distinction would help future contributors.
- create rai-state.schema.json with full Schema B field definitions - add json.schemas VS Code setting for editor-time validation - reference formal schema from RAI identity instructions 🔧 - Generated by Copilot
There was a problem hiding this comment.
Advisory review — this PR is from a maintainer. Findings are informational only.
Review Overview
This PR delivers a substantial, well-documented set of changes: a 5→6 phase restructuring of the RAI Planner with scoring removal, standard alignment, and a new AI artifact validation pipeline. The implementation is thorough and the testing table is detailed. Four advisory findings below — one is a correctness issue worth addressing before merge.
Note: The PR reports 46 changed files in the description; the GitHub API counts 52. Minor inaccuracy, no action needed.
Issue Alignment
✅ Closes #1281 — The PR directly addresses the linked issue (Impact Assessment Guide alignment, disclaimer updates, UX improvements, user output preferences). The additional scope (AI artifact validation pipeline, Sign-RaiArtifacts.ps1, new CI workflow, schema files) goes beyond the original issue but is well-motivated by reviewer feedback documented in the discussion thread, and is fully described in the PR body.
PR Template Compliance
Several checklist sections in the PR body are incomplete or missing:
Missing section: The Security Considerations section from the template is absent from the PR body. This PR adds security-adjacent scripts (Sign-RaiArtifacts.ps1, Validate-AIArtifacts.ps1) and a new CI workflow; the three attestation checkboxes there are relevant.
Unchecked required items:
- 🔲 Required Automated Checks — all seven items are unchecked, though the Testing table confirms they were run and passed. Please tick the boxes that passed.
- 🔲 GHCP Maturity Acknowledgment — both acknowledgment checkboxes are unchecked despite the PR including 11 experimental artifacts listed in the maturity table.
- 🔲 AI Artifact Contributions — all three checklist items (
/prompt-analyze,prompt-builderfeedback, standards verification) are unchecked.
Checklist divergence from template: The standard template includes Plugin freshness: npm run plugin:generate and Docusaurus tests: npm run docs:test under Required Automated Checks. These are absent from the PR body (replaced with lint:ai-artifacts). The Testing table confirms plugin regeneration was run — please either restore the standard template items or note the substitution explicitly.
Coding Standards
See inline comments for specifics. Summary:
| File | Issue |
|---|---|
scripts/linting/Validate-AIArtifacts.ps1 |
Missing #!/usr/bin/env pwsh shebang (convention requires it on all .ps1 files) |
scripts/security/Sign-RaiArtifacts.ps1 |
Main execution not wrapped in invocation guard (if ($MyInvocation.InvocationName -ne '.')) |
Code Quality
🔴 Correctness issue (inline comment on ai-artifact-validation.yml line 52): The Check results step uses if: "!inputs.soft-fail", which is inverted. When soft-fail=true the step is skipped — the only scenario where a deferred env-var check is useful. When soft-fail=false the step cannot run because the validate step already terminated the job on failure. As written, $env:AI_ARTIFACT_VALIDATION_FAILED (set via Set-CIEnv in Validate-AIArtifacts.ps1) is never checked, making the soft-fail mode a silent pass. Suggested fix: if: always() && inputs.soft-fail.
Minor (inline comment on .devcontainer/scripts/on-create.sh line 94): Single-space sha256sum format differs from the double-space format in copilot-setup-steps.yml. Both likely work on Linux, but alignment keeps the two install paths consistent.
Collection completeness note: .github/instructions/shared/disclaimer-language.instructions.md is included in collections/hve-core-all.collection.yml but not in collections/rai-planning.collection.yml. Since RAI planning artifacts reference this file, consider whether it belongs in the RAI planning collection as well (or document that the shared collection is the intended distribution point).
Action Items
- Address the
soft-faillogic bug inai-artifact-validation.ymlbefore merge — the feature is otherwise a no-op. - Complete the PR checklist (Required Automated Checks, GHCP Maturity Acknowledgment, AI Artifact Contributions, Security Considerations).
- Add the shebang to
Validate-AIArtifacts.ps1and the invocation guard toSign-RaiArtifacts.ps1(convention alignment). - Align the sha256sum format in
on-create.shwithcopilot-setup-steps.yml.
- fix unreachable soft-fail check step in ai-artifact-validation workflow - add shebang and comment-based help to Validate-AIArtifacts.ps1 - correct schema $id to match filename in ai-artifact-config.schema.json - add invocation guard to Sign-RaiArtifacts.ps1 - fix sha256sum two-space format in devcontainer on-create script 🔧 - Generated by Copilot
There was a problem hiding this comment.
Advisory Review
Advisory review — this PR is from a maintainer. Findings are informational only.
Overview
This PR delivers a thorough, well-documented evolution of the RAI Planner agent and associated infrastructure. The new 6-phase workflow, qualitative concern-level model, AI artifact validation pipeline, and cosign signing infrastructure are all solid additions. CI/workflow conventions are properly followed across all new and modified files.
Two process compliance gaps and one structural observation are noted below.
Issue Alignment
✅ Linked issue: Closes #1281 — the core requirements (alignment with the Impact Assessment Guide, disclaimer upgrades, UX improvements, output preferences, validation passing) are addressed.
.github/agents/rai-planning/, .github/instructions/rai-planning/, and .github/prompts/rai-planning/. This PR changes 52 files, adding a new CI validation pipeline (Validate-AIArtifacts.ps1, ai-artifact-validation.yml, YAML config files, JSON schema, Pester suite), cosign signing infrastructure (Sign-RaiArtifacts.ps1, devcontainer, copilot-setup-steps.yml), and collection/plugin metadata regeneration. The additions are coherent and valuable, but they extend well beyond the original issue's stated scope. Consider opening a follow-up issue to track the AI artifact validation pipeline as a standalone work item, or annotate the PR description to acknowledge the expanded scope explicitly.
PR Template Compliance
✅ Description: Comprehensive and well-structured.
✅ Related Issue(s): Closes #1281 present.
✅ Type of Change: Multiple appropriate boxes checked.
✅ Sample Prompts: Entry modes, execution flow, output artifacts, and success indicators all documented.
✅ Testing: Detailed results table provided.
❌ AI Artifact Contributions checklist — all items unchecked: The PR includes agents, instructions, and prompts. The following checklist items under "AI Artifact Contributions" must be completed by the author before merge:
[ ]Used/prompt-analyzeto review contribution[ ]Addressed all feedback fromprompt-builderreview[ ]Verified contribution follows common standards and type-specific requirements
❌ Required Automated Checks — all checkboxes unchecked: The Testing section table shows all validations passed, but the formal checklist items in "Required Automated Checks" remain unchecked. These checkboxes should be marked once the corresponding commands have been run and confirmed passing:
[ ]Markdown linting:npm run lint:md[ ]Spell checking:npm run spell-check[ ]Frontmatter validation:npm run lint:frontmatter[ ]Skill structure validation:npm run validate:skills[ ]Link validation:npm run lint:md-links[ ]PowerShell analysis:npm run lint:ps[ ]AI artifact validation:npm run lint:ai-artifacts
Coding Standards Review
✅ GitHub Actions workflow (ai-artifact-validation.yml): SHA-pinned actions (actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd, actions/upload-artifact@bbbca2ddaa5d8feaa63e36b76fdaad77386f024f), correct top-level and job-level permissions: contents: read, persist-credentials: false, and ubuntu-latest runner. All conventions satisfied.
✅ Workflow integration (pr-validation.yml): Local reusable workflow reference correctly uses relative path — SHA-pinning exemption applies. Permissions correctly scoped.
✅ Devcontainer/setup (on-create.sh, copilot-setup-steps.yml): Cosign installation correctly downloads, verifies (SHA-256 with two-space format — note the sha256sum -c double-space fix applied consistently across all tools is correct), and installs via sudo install. Architecture detection and fallback error handling are sound.
✅ PowerShell conventions (Validate-AIArtifacts.ps1, Sign-RaiArtifacts.ps1): Copyright headers, #Requires -Version 7.0, [CmdletBinding()], $ErrorActionPreference = 'Stop', and comment-based help are all present.
✅ npm scripts (package.json): lint:ai-artifacts (CI enforcement with -FailOnMissing) and validate:ai-artifacts (informational, no failure flag) serve distinct purposes. The lint:all chain correctly uses the enforcement variant. The naming distinction is intentional and reasonable.
✅ SHA-staleness tests: Test-SHAStaleness.Tests.ps1 correctly updated to expect 3 tool release checks (actionlint + gitleaks + cosign) across all relevant test cases.
Code Quality
✅ No security vulnerabilities, injection risks, or secrets exposure detected in the diff.
✅ Error handling at system boundaries follows established patterns.
✅ Cosign SHA values are valid 64-character hex strings.
✅ The tool-checksums.json entry for cosign is consistent with the SHA values used in both on-create.sh and copilot-setup-steps.yml.
Required Actions Before Merge
- Check the three AI Artifact Contributions checklist items after performing the required
/prompt-analyzereview. - Check all seven Required Automated Checks items.
- Check both GHCP Maturity Acknowledgment items.
…Artifacts - add 29 tests for Validate-AIArtifacts covering config loading, compliance, and validation - add 3 tests for Sign-RaiArtifacts covering manifest generation, dry run, and cosign invocation - fix missing trailing newline in rai-state.schema.json 🧪 - Generated by Copilot
katriendg
left a comment
katriendg
left a comment
There was a problem hiding this comment.
What a huge amount of work and love the addition of the schema/footer validation stuff as well.
I left a few comments, and then when you got those done you will have to regenerate plugins. Other than that it should be ready for merging!
| <div> | ||
| <h3>RAI Control: {control_name}</h3> | ||
| <p><strong>RAI Principle:</strong> {principle}</p> | ||
| <p><strong>NIST Characteristic:</strong> {characteristicistic}</p> |
There was a problem hiding this comment.
Comment 1 (Lines 171, 217, 232)
- Category: Code Quality
- Severity: Medium
Typo: {characteristicistic} should be {characteristic} — three occurrences in template placeholders.
Suggested Change
Line 171:
<p><strong>NIST Characteristic:</strong> {characteristic}</p>Line 217:
rai_characteristic: {characteristic}Line 232:
**NIST Characteristic:** {characteristic}| - handoff-summary | ||
| - compact-handoff-summary | ||
| text: >- | ||
| > **Disclaimer** — This agent is an assistive tool only. It does not |
There was a problem hiding this comment.
Comment 2 (Lines 17-28)
- Category: Consistency
- Severity: Medium
The disclaimer text in disclaimers.yml says "sensitive use screenings" but the same disclaimer in disclaimer-language.instructions.md and rai-backlog-handoff.instructions.md says "risk classification screenings". Since the PR renames Phase 2 from "Sensitive Uses Assessment" to "Risk Classification", the YAML config text appears stale.
Additionally, the formatting prefix differs: the YAML uses > **Disclaimer** — while disclaimer-language.instructions.md uses > [!CAUTION] + **Disclaimer:**. These serve different purposes (artifact footer vs. startup announcement) but the body text should be consistent.
Suggested Change
Update the text field in disclaimers.yml to replace "sensitive use screenings" with "risk classification screenings" to match the other two sources.
|
|
||
| Activate the RAI Planner in **capture mode** for project slug `${input:project-slug}`. | ||
|
|
||
| ## Startup |
There was a problem hiding this comment.
Comment 3
- Category: Conventions
- Severity: Low
The three prompt files each add a ## Startup section that reproduces the disclaimer display protocol already defined in the agent's ## Startup Announcement section and the identity instructions file's ## Disclaimer and Attribution Protocol. Per the prompt-builder instructions, prompts delegating to an agent via agent: should avoid adding sections that duplicate the parent agent's protocol. The prompt-builder guidance says:
Avoid adding Required Phases, Required Steps, or Required Protocol sections that duplicate or conflict with the parent agent's protocol.
The agent already handles disclaimer display via disclaimerShownAt in state.json. Consider removing the ## Startup sections from the three prompt files or converting them to a brief reference: "Follow the Startup Announcement and Disclaimer and Attribution Protocol from the agent definition."
| # Agentic artifacts: Tier 1 note only (no human review required) | ||
| agentic: | ||
| scope: | ||
| - .github/instructions/rai-planning/** |
There was a problem hiding this comment.
Comment 4
- Category: Maintainability
- Severity: Low
All three artifact classification tiers scope exclusively to .github/instructions/rai-planning/**. The validation workflow is named "AI Artifact Validation" and the script Validate-AIArtifacts.ps1, suggesting broader applicability. If Security Planner or SSSC Planner adopt the same footer/disclaimer pattern later, the config will need expansion.
Consider either:
- Adding a comment in the YAML noting the current RAI-only scope and the extension path, or
- Pre-defining empty tier entries for security and SSSC planning scopes so the extension path is obvious
No current breakage — other planners won't see false positives since artifact name matching is basename-based and no matching filenames exist outside RAI planning instructions.
Optional if you want to change anything here, or close the comment.
| - path: .github/instructions/rai-planning/rai-risk-classification.instructions.md | ||
| kind: instruction | ||
| maturity: experimental | ||
| - path: .github/instructions/shared/hve-core-location.instructions.md |
There was a problem hiding this comment.
Comment 5
- Category: Conventions
- Severity: Medium
disclaimer-language.instructions.md is referenced via #file: by the RAI Planner agent, Security Planner agent, and SSSC Planner agent. It is included in hve-core-all.collection.yml but missing from three collections that include these agents:
rai-planning.collection.yml❌security.collection.yml❌project-planning.collection.yml❌
The copilot-instructions state: "Collections must include all subagent dependencies used by their referenced custom agents." The same principle applies to #file: dependencies — if the agent references a file via #file:, the collection should include it so plugin packaging delivers the file alongside the agent.
Suggested Change
Add disclaimer-language.instructions.md to each collection's items:
- path: .github/instructions/shared/disclaimer-language.instructions.md
kind: instruction5 participants
Description
Implements Issue #1281: RAI Planner Updates — Guide Alignment, Scoring Removal, and UX Improvements.
The RAI Planner agent expands from a 5-phase to a 6-phase workflow, replacing numeric scoring with qualitative assessment, aligning terminology and structure with the Microsoft Responsible AI Impact Assessment Guide, and introducing artifact signing infrastructure. Additionally, this PR establishes a config-driven AI artifact validation pipeline for enforcing footer and disclaimer standards across instruction files. 46 files changed across agent definitions, instructions, prompts, documentation, plugins, collection metadata, config, CI workflows, and scripts.
Key Changes
Basic/Standard/Comprehensive). Subsequent phases renumbered.Low/Moderate/High), a review quality checklist, maturity indicators, and audience adaptation profiles. Renamedrai-scorecard.md→rai-review-summary.md.T-RAI-{NNN}.Sign-RaiArtifacts.ps1for SHA-256 manifest generation with optional Sigstore cosign keyless signing, backed by 262-line Pester test suite. Installed cosign v3.0.5 in devcontainer..github/config/disclaimers.yml,.github/config/footer-with-review.yml) defining footer text, human review checkboxes, and tiered artifact classification with scope-aware glob patterns. CreatedValidate-AIArtifacts.ps1(612 lines) for CI enforcement, a JSON Schema for config validation, a reusable GitHub Actions workflow (ai-artifact-validation.yml), and a 20-test Pester suite. Wired intopr-validation.ymland thelint:allnpm chain.Related Issue(s)
Closes #1281
Type of Change
Select all that apply:
Code & Documentation:
Infrastructure & Configuration:
AI Artifacts:
prompt-builderagent and addressed all feedback.github/instructions/*.instructions.md).github/prompts/*.prompt.md).github/agents/*.agent.md).github/skills/*/SKILL.md)> Note for AI Artifact Contributors:
>
> * Agents: Research, indexing/referencing other project (using standard VS Code GitHub Copilot/MCP tools), planning, and general implementation agents likely already exist. Review
.github/agents/before creating new ones.> * Skills: Must include both bash and PowerShell scripts. See Skills.
> * Model Versions: Only contributions targeting the latest Anthropic and OpenAI models will be accepted. Older model versions (e.g., GPT-3.5, Claude 3) will be rejected.
> * See Agents Not Accepted and Model Version Requirements.
Other:
.ps1,.sh,.py)Sample Prompts (for AI Artifact Contributions)
User Request:
Invoke
RAI Plannerin the VS Code chat pane and use one of three entry prompts:/rai-capture— Start a new conversational RAI assessment from scratch/rai-plan-from-prd— Generate an RAI plan from an existing PRD/rai-plan-from-security-plan— Generate an RAI plan from an existing security planExecution Flow:
T-RAI-{NNN}threats.rai-review-summary.md, dual-format backlog (ADO + GitHub), and optional signed artifact manifest.Output Artifacts:
.copilot-tracking/rai-plans/{session}/state.json— Session state with phase progression.copilot-tracking/rai-plans/{session}/rai-review-summary.md— Qualitative review summary (replaces scored scorecard).copilot-tracking/rai-plans/{session}/rai-backlog-*.md— Dual-format work item backlog.copilot-tracking/rai-plans/{session}/rai-manifest.json— SHA-256 artifact manifest (optional signing)Success Indicators:
rai-review-summary.mdinstead ofrai-scorecard.mdT-RAI-{NNN}format consistentlyTesting
npm run plugin:generate) — 14 pluginsnpm run lint:ai-artifacts> Note: Manual testing was performed along side automated validation and sandbox evaluation as the primary verification methods.
GHCP Artifact Maturity
> [!WARNING]
> This PR includes experimental GHCP artifacts that may have breaking changes.
.github/agents/rai-planning/rai-planner.agent.md.github/prompts/rai-planning/rai-capture.prompt.md.github/prompts/rai-planning/rai-plan-from-prd.prompt.md.github/prompts/rai-planning/rai-plan-from-security-plan.prompt.md.github/instructions/rai-planning/rai-identity.instructions.md.github/instructions/rai-planning/rai-standards.instructions.md.github/instructions/rai-planning/rai-security-model.instructions.md.github/instructions/rai-planning/rai-impact-assessment.instructions.md.github/instructions/rai-planning/rai-backlog-handoff.instructions.md.github/instructions/rai-planning/rai-capture-coaching.instructions.md.github/instructions/rai-planning/rai-sensitive-uses-triggers.instructions.mdGHCP Maturity Acknowledgment
Checklist
Required Checks
AI Artifact Contributions
/prompt-analyzeto review contributionprompt-builderreviewRequired Automated Checks
The following validation commands must pass before merging:
npm run lint:mdnpm run spell-checknpm run lint:frontmatternpm run validate:skillsnpm run lint:md-linksnpm run lint:psnpm run lint:ai-artifacts> Repository template used:
.github/PULL_REQUEST_TEMPLATE.md