feat: add /qualify AI qualification workflow#474
Open
myakove wants to merge 8 commits into
Open
Conversation
Contributor
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughThis PR adds an AI-driven /qualify workflow: orchestration prompt, agent and skill specs (test-planner, cluster-verifier, proof-generator), templates, documentation, and a ChangesAI Qualification Workflow for MTV API Tests
Sequence Diagram(s)sequenceDiagram
participant User
participant Orchestrator
participant TestPlannerAgent
participant PythonWriter
participant RealCluster
participant ClusterVerifier
participant ProofGenerator
participant Git
User->>Orchestrator: invoke /qualify --type --source [--cluster]
Orchestrator->>TestPlannerAgent: generate test-plan.md from source
Orchestrator->>User: request human approval of test-plan.md
Orchestrator->>PythonWriter: write tests & config (branch for permanent tests)
PythonWriter->>RealCluster: run pytest (capture output -> test-output.log)
RealCluster-->>Orchestrator: pytest stdout/stderr + exit code
Orchestrator->>ClusterVerifier: run independent cluster verification
ClusterVerifier-->>Orchestrator: cluster verification report
Orchestrator->>ProofGenerator: assemble proof.md (tests + verification + versions)
Orchestrator->>Git: create PR (permanent-test path) with proof reference
Estimated code review effort🎯 2 (Simple) | ⏱️ ~12 minutes 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Tip 💬 Introducing Slack Agent: The best way for teams to turn conversations into code.Slack Agent is built on CodeRabbit's deep understanding of your code, so your team can collaborate across the entire SDLC without losing context.
Built for teams:
One agent for your entire SDLC. Right inside Slack. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Report bugs in Issues Welcome! 🎉This pull request will be automatically processed with the following features: 🔄 Automatic Actions
📋 Available CommandsPR Status Management
Review & Approval
Testing & Validation
Container Operations
Cherry-pick Operations
Label Management
✅ Merge RequirementsThis PR will be automatically approved when the following conditions are met:
📊 Review ProcessApprovers and ReviewersApprovers:
Reviewers:
Available Labels
AI Features
💡 Tips
For more information, please refer to the project documentation or contact the maintainers. |
myakove
force-pushed
the
feature/qualify-workflow
branch
from
33becb1 to
b8ea397
Compare
myakove
force-pushed
the
feature/qualify-workflow
branch
from
e42aac5 to
077b62d
Compare
|
Clean rebase detected — no code changes compared to previous head ( |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@llm/qualify/agents/cluster-verifier.md`:
- Around line 146-150: Add mandatory redaction rules to the evidence collection
section that currently instructs to store "The full output" and raw evidence
into proof.md: update the cluster-verifier.md checklist to require automatic
redaction before persisting raw logs (e.g., mask secrets, API keys, tokens,
emails, IPs, and YAML anchors), provide a short canonical redaction policy and
example regex patterns, and add a one-liner command or reference to the
redaction utility to run prior to saving evidence so any step that calls out
"The full output" or writes to proof.md enforces redaction.
- Line 35: The document currently has contradictory failure semantics ("stop
immediately" vs "continue checking other items"); update the verification
guidance to clearly distinguish connectivity/authentication failures from
individual check failures: state that failure of any cluster-level connectivity
or authentication check (the sentence currently saying "stop immediately") must
abort the entire verification and report no-trust, whereas non-connectivity
per-check failures should be logged and verification should continue to collect
all failures (the area currently saying "continue checking other items"); change
the two conflicting sentences so the first explicitly names
"connectivity/authentication checks" as abort conditions and the later paragraph
(around the per-check rules) explicitly documents that per-check failures do not
abort but are aggregated as partial failures, and ensure the same clarified rule
text replaces the existing lines referenced in the doc.
In `@llm/qualify/prompts/qualify.md`:
- Around line 136-139: The doc has a conflict between the temporary test
location and the verify-only execution command: update the verify-only example
to run pytest directly against the temporary test file path
(/tmp/qualify-<name>/<test_file>.py) instead of using the repo test selector
(tests/<path>::<TestClass>); change the command shown (the "uv run pytest ..."
example) to point to /tmp/qualify-<name>/<test_file>.py, keep the same pytest
flags (--tc-file, --tc-format, -p no:xdist), and continue piping output to
.qualify/<type>/<name>/test-output.log so verify-only remains isolated from repo
state.
In `@llm/qualify/skills/proof-generator/SKILL.md`:
- Around line 130-132: Update SKILL.md's evidence policy for proof.md so raw
YAML/logs are required to be sanitized before inclusion: change the bullet that
mandates raw evidence in collapsible <details> to require "sanitized, redacted
evidence" and add a short checklist in SKILL.md/proof.md that instructs
redacting secrets/PII (API keys, tokens, passwords, private IPs, certs),
replacing values with placeholders like <REDACTED> and documenting what was
removed; ensure the collapsible details still include enough context (file
names, non-sensitive fields, and diffs) and add a statement on how to mark
omitted sections (e.g., "[…redacted…]") so reviewers know evidence was
intentionally redacted.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: b340adfa-9cf1-4110-860e-e91ae4f91673
📒 Files selected for processing (15)
.gitignoreCLAUDE.mdREADME.mdllm/qualify/README.mdllm/qualify/agents/.gitkeepllm/qualify/agents/cluster-verifier.mdllm/qualify/agents/test-planner.mdllm/qualify/prompts/.gitkeepllm/qualify/prompts/qualify.mdllm/qualify/skills/proof-generator/.gitkeepllm/qualify/skills/proof-generator/SKILL.mdllm/qualify/templates/.gitkeepllm/qualify/templates/proof-template.mdllm/qualify/templates/test-plan-template.mdllm/qualify/workflow-diagrams.md
Full qualification workflow: test plan → write tests → verify on cluster → PR with proof. Components: - /qualify prompt template (orchestrator) - test-planner agent (reads feature/bug docs → test plans) - cluster-verifier agent (independent cluster state verification) - proof-generator skill (assembles proof.md reports) - Templates for test plans and proof reports
- Clarify failure semantics in cluster-verifier agent - Add redaction rules to proof-generator skill - Fix verify-only path in qualify prompt
Collaborator
Author
|
@coderabbitai review |
Contributor
✅ Actions performedReview triggered.
|
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@llm/qualify/prompts/qualify.md`:
- Around line 42-53: After the "Collect environment versions" block in
qualify.md, add explicit failure-handling instructions: if any of the three oc
commands (OCP, MTV, CNV) fail (missing CSV/namespace or RBAC denied), record
that specific version as `UNKNOWN` and capture the command's error message;
state that the proof report will include these `UNKNOWN` entries and that
missing versions will cause proof-generator/SKILL.md's qualification logic to
mark the report NOT QUALIFIED, mirroring the behavior described in
cluster-verifier.md.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: b19bc28d-78cd-4c42-8328-1589dd033318
📒 Files selected for processing (3)
llm/qualify/agents/cluster-verifier.mdllm/qualify/prompts/qualify.mdllm/qualify/skills/proof-generator/SKILL.md
Collaborator
Author
|
@coderabbitai review |
Contributor
✅ Actions performedReview triggered.
|
Contributor
There was a problem hiding this comment.
Actionable comments posted: 3
♻️ Duplicate comments (3)
llm/qualify/prompts/qualify.md (2)
65-67:⚠️ Potential issue | 🟠 Major | ⚡ Quick winHIGH: Bug artifact directory placeholder is inconsistent (
<name>vs canonical bug identifier).This can desynchronize artifact paths between prompt execution and README/consumers expecting
.qualify/bugs/<id>/....Suggested doc patch
- - Bug: `.qualify/bugs/<name>/` + - Bug: `.qualify/bugs/<id>/`🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@llm/qualify/prompts/qualify.md` around lines 65 - 67, Update the bug artifact directory placeholder so it uses the canonical bug identifier instead of a generic <name>; change occurrences of ".qualify/bugs/<name>/" to ".qualify/bugs/<id>/" in the qualify.md prompt documentation (keep ".qualify/features/<name>/" as-is if features use names) so artifact paths match README/consumers expecting .qualify/bugs/<id>/...
56-63:⚠️ Potential issue | 🟠 Major | ⚡ Quick winHIGH: Version collection commands can terminate the flow before
UNKNOWNfallback is applied.Plain
... | grep ...can exit non-zero on no match, which contradicts the “record as UNKNOWN with error message” behavior.Suggested doc patch
-oc get csv -n openshift-mtv -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.version}{"\n"}{end}' | grep mtv +oc get csv -n openshift-mtv -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.version}{"\n"}{end}' | grep mtv || true -oc get csv -n openshift-cnv -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.version}{"\n"}{end}' | grep kubevirt +oc get csv -n openshift-cnv -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.version}{"\n"}{end}' | grep kubevirt || true#!/bin/bash # Verify current prompt uses fatal grep pipelines for MTV/CNV collection. rg -n 'openshift-mtv|openshift-cnv|grep mtv|grep kubevirt|UNKNOWN' llm/qualify/prompts/qualify.md🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@llm/qualify/prompts/qualify.md` around lines 56 - 63, The grep pipelines in the version-collection snippets (the lines containing "oc get csv -n openshift-mtv ... | grep mtv" and "oc get csv -n openshift-cnv ... | grep kubevirt") can return a non-zero exit and abort scripts; change these snippets so the pipeline never causes a fatal exit and instead captures grep failure output and sets the version to "UNKNOWN" with the error message; implement this by making the grep step non-fatal (e.g., allow the pipeline to continue on no-match), detect grep's exit status or stderr, and explicitly emit the "UNKNOWN: <error message>" fallback in the snippet so the documented behavior ("record it as UNKNOWN with the error message") always occurs.llm/qualify/agents/cluster-verifier.md (1)
46-53:⚠️ Potential issue | 🟠 Major | ⚡ Quick winHIGH:
UNKNOWN with error messageis not operationally satisfiable with current version commands.
grep ... || trueprevents hard failure, but it also drops the signal needed to reliably capture why the version was missing. This can produce silentUNKNOWNwithout actionable diagnostics.Suggested doc patch
-oc get csv -n openshift-mtv -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.version}{"\n"}{end}' | grep mtv || true +MTV_VERSION_RAW="$(oc get csv -n openshift-mtv -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.version}{"\n"}{end}' 2>&1)" +MTV_VERSION="$(printf '%s\n' "$MTV_VERSION_RAW" | grep mtv || true)" +# if empty -> record: UNKNOWN: <MTV_VERSION_RAW or "no CSV match"> -oc get csv -n openshift-cnv -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.version}{"\n"}{end}' | grep kubevirt || true +CNV_VERSION_RAW="$(oc get csv -n openshift-cnv -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.version}{"\n"}{end}' 2>&1)" +CNV_VERSION="$(printf '%s\n' "$CNV_VERSION_RAW" | grep kubevirt || true)" +# if empty -> record: UNKNOWN: <CNV_VERSION_RAW or "no CSV match">#!/bin/bash # Verify that current doc suppresses grep failures but does not define error-capture variables. rg -n 'grep (mtv|kubevirt) \|\| true|UNKNOWN.*error message|_RAW=|2>&1' llm/qualify/agents/cluster-verifier.md🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@llm/qualify/agents/cluster-verifier.md` around lines 46 - 53, The current shell snippets use "grep ... || true" which hides failures and prevents capturing why a CSV/version lookup failed; modify the commands that run "oc get csv -n openshift-mtv ..." and "oc get csv -n openshift-cnv ..." to capture both stdout and stderr into variables (e.g., MTV_RAW and CNV_RAW), check the command exit status, and if non-zero set the version variable to "UNKNOWN" while storing the stderr/error message (e.g., MTV_ERROR or CNV_ERROR) for diagnostic output; remove the "|| true" pattern, use command substitution with redirection (cmd_output=$(oc ... 2>&1) && parse || { VERSION=UNKNOWN; ERROR="$cmd_output"; }) and ensure subsequent docs/logs print VERSION and ERROR when VERSION is UNKNOWN.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@llm/qualify/prompts/qualify.md`:
- Around line 3-25: The prompt currently treats `--cluster` as optional (see
`argument-hint` and "Phase 0: Parse Arguments & Setup") which conflicts with the
safety contract; update the prompt to require `--cluster` explicitly and to fail
fast when it is not provided: change the `argument-hint` text to mark
`--cluster` as REQUIRED and modify the Phase 0 parsing step to validate presence
of `--cluster` (or error/exit immediately with a clear message) so the
orchestrator will not proceed without an explicit kubeconfig path.
In `@llm/qualify/README.md`:
- Around line 45-47: Update the example CLI invocation in the README to include
the required --cluster argument so it matches the required-arguments table;
specifically modify the example line that currently reads "/qualify --type bug
--source https://issues.redhat.com/browse/MTV-5678 --name MTV-5678" to include a
placeholder cluster flag (e.g., --cluster <cluster-name>) so callers know to
supply the cluster when running the qualify command.
In `@llm/qualify/skills/proof-generator/SKILL.md`:
- Around line 166-169: Update the inconsistent bug proof path in SKILL.md:
replace the `.qualify/bugs/<bug-name-or-id>/proof.md` placeholder with the
canonical placeholder used elsewhere (e.g., `.qualify/bugs/<bug-name>/proof.md`)
so tooling and docs are consistent; edit the string in the SKILL.md content
where the proof path is documented to match the
`.qualify/features/<feature-name>/proof.md` style and ensure the directory
guidance line references `.qualify/bugs/<bug-name>/proof.md`.
---
Duplicate comments:
In `@llm/qualify/agents/cluster-verifier.md`:
- Around line 46-53: The current shell snippets use "grep ... || true" which
hides failures and prevents capturing why a CSV/version lookup failed; modify
the commands that run "oc get csv -n openshift-mtv ..." and "oc get csv -n
openshift-cnv ..." to capture both stdout and stderr into variables (e.g.,
MTV_RAW and CNV_RAW), check the command exit status, and if non-zero set the
version variable to "UNKNOWN" while storing the stderr/error message (e.g.,
MTV_ERROR or CNV_ERROR) for diagnostic output; remove the "|| true" pattern, use
command substitution with redirection (cmd_output=$(oc ... 2>&1) && parse || {
VERSION=UNKNOWN; ERROR="$cmd_output"; }) and ensure subsequent docs/logs print
VERSION and ERROR when VERSION is UNKNOWN.
In `@llm/qualify/prompts/qualify.md`:
- Around line 65-67: Update the bug artifact directory placeholder so it uses
the canonical bug identifier instead of a generic <name>; change occurrences of
".qualify/bugs/<name>/" to ".qualify/bugs/<id>/" in the qualify.md prompt
documentation (keep ".qualify/features/<name>/" as-is if features use names) so
artifact paths match README/consumers expecting .qualify/bugs/<id>/...
- Around line 56-63: The grep pipelines in the version-collection snippets (the
lines containing "oc get csv -n openshift-mtv ... | grep mtv" and "oc get csv -n
openshift-cnv ... | grep kubevirt") can return a non-zero exit and abort
scripts; change these snippets so the pipeline never causes a fatal exit and
instead captures grep failure output and sets the version to "UNKNOWN" with the
error message; implement this by making the grep step non-fatal (e.g., allow the
pipeline to continue on no-match), detect grep's exit status or stderr, and
explicitly emit the "UNKNOWN: <error message>" fallback in the snippet so the
documented behavior ("record it as UNKNOWN with the error message") always
occurs.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: b906dcdc-ef1b-4880-8451-498ce03b5b78
📒 Files selected for processing (7)
llm/qualify/README.mdllm/qualify/agents/cluster-verifier.mdllm/qualify/prompts/qualify.mdllm/qualify/skills/proof-generator/SKILL.mdllm/qualify/templates/proof-template.mdllm/qualify/templates/test-plan-template.mdllm/qualify/workflow-diagrams.md
- require --cluster, normalize placeholders, capture version errors
Collaborator
Author
|
@coderabbitai[bot] The following review comments were reviewed and a decision was made:
|
Contributor
|
Tip For best results, initiate chat on the files or code changes.
If you found this review helpful, would you consider giving us a shout-out on X? Thank you for using CodeRabbit! |
Collaborator
Author
|
@coderabbitai review |
Contributor
✅ Actions performedReview triggered.
|
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@llm/qualify/agents/cluster-verifier.md`:
- Line 4: Update the agents docs to clarify tool-name mappings: in
llm/qualify/agents/cluster-verifier.md locate the tools declaration "tools:
read, bash" and add a short mapping section that shows equivalent tool names for
other AI CLIs (e.g., map "bash" to "execute" or "run_command" for pi and note
"bash" for Cursor/Claude), and include a generic note to adapt shell execution
tool names for other frameworks so readers can easily translate the example to
their CLI.
In `@llm/qualify/prompts/qualify.md`:
- Around line 49-62: Update the CSV-version capture to record stderr into a raw
variable and derive the filtered version from it: run the oc get csv commands
with stderr redirected (capture into MTV_VERSION_RAW and CNV_VERSION_RAW), then
set MTV_VERSION and CNV_VERSION by grepping those raw vars (e.g., printf '%s\n'
"$MTV_VERSION_RAW" | grep mtv || true), and when the grep yields empty record
"UNKNOWN: <raw output or 'no CSV match'>"; apply this pattern for both MTV and
CNV (replace the current direct pipe + "|| true" uses) so diagnostic stderr is
preserved for the proof report.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: 652fb002-6616-4851-9268-559f63e42bf8
📒 Files selected for processing (4)
llm/qualify/README.mdllm/qualify/agents/cluster-verifier.mdllm/qualify/prompts/qualify.mdllm/qualify/skills/proof-generator/SKILL.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the
/qualifyAI qualification workflow — a fully automated end-to-end pipeline that takes a feature design doc or bug report and produces verified tests with cluster proof.What's Included
Prompt Template
llm/qualify/prompts/qualify.md— Main/qualifycommand that orchestrates all 4 phasesAgents
llm/qualify/agents/test-planner.md— Reads feature/bug docs → produces structured test plansllm/qualify/agents/cluster-verifier.md— Independently verifies OpenShift cluster state after test executionSkill
llm/qualify/skills/proof-generator/SKILL.md— Assembles proof.md reports with test results + cluster evidenceTemplates
llm/qualify/templates/test-plan-template.md— Test plan skeletonllm/qualify/templates/proof-template.md— Proof report skeletonDocumentation
llm/qualify/README.md— Full usage guide with setup instructions for pi, Claude Code, Cursor, and other AI CLIsllm/qualify/workflow-diagrams.md— Mermaid flowcharts (workflow, components, sequence diagram)Workflow Overview
Human Checkpoints
Do NOT auto-merge. This needs human review.
Summary by CodeRabbit
New Features
Documentation
Chores