Skip to content

plain language and report streamline changes for cli-review skill #13

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
hartmutobendorf merged 1 commit into from
Jul 8, 2026
+46 −60
Merged

plain language and report streamline changes for cli-review skill #13

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 17 additions & 16 deletions cli-skill/commands/cli-review.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ CLI standard compliance review only.
- `.cli-skill-infra/scripts/calculate_cli_score.py`
- `scripts/calculate_cli_score.py`
Execute: `python3 <resolved_script_path> <json_file>`.
10. Use the script JSON output values (`score`, `rating`, `rating_badge`, counts) in the summary. **Do not compute score manually.**
10. Use the script JSON output values (`score`, and counts) in the summary. **Do not compute score manually.**

### Hard Constraints

Expand Down Expand Up @@ -61,40 +61,41 @@ The output **must** use exactly these top-level headings in this order:

```
# Canonical CLI automated review report
## Summary (violation table + rating badge returned by the scoring script)
## Findings (table with columns: Severity | Rule Summary | Evidence | How to fix)
## Detailed analysis (with citations) (this section is encased in <detail> tag per cli-review-output-format.md)
## CLI changes in this PR (include only if the workflow was triggered from a PR creation/update)
## Compliance matrix (table with columns: Finding | Rule Summary | Evidence | Notes)
## Non-compliance Findings (with citations)
## Compliant Findings Summary (concise, without citations)
## Summary (violation table + score returned by the scoring script)
```

The final review file MUST begin with first character of the required report title: `# Canonical CLI automated review report`

Write:
- `cli-review/cli-review.md`

### Summary Requirements
The summary should list the number of violations, and their severity. It shall include these in a table. It shall give an overall score. This score is calculated by a script, DO NOT REASON ABOUT IT, AND DO NOT CHANGE THE ALGORITHM.
The summary should list the number of violations, and their severity. It shall include these in a table. It shall give an overall score. This score is calculated by a script, DO NOT REASON ABOUT IT, AND DO NOT CHANGE THE ALGORITHM.

To calculate the score:
1. Create a JSON table with structure: `{"commands": <int>, "issues": [{"severity": "High|Medium|Low|Unrated", "category": <str>, "message": <str>}, ...]}`
2. Execute: `python3 <resolved_script_path> <json_file>` using the path resolution defined in `Execution Order`
3. This script returns JSON with `score` (0-100), `passed` (boolean), `rating badge`, and severity counts
4. Use this `score` and `rating badge` in the summary section of the markdown output to report the compliance score
3. This script returns JSON with `score` (0-100), `passed` (boolean), and severity counts
4. Use this `score` in the summary section of the markdown output to report the compliance score

The script implements the standard algorithm: Start with 100%, weight W=100/#commands. For each High violation, reduce by W; Medium violation by 0.5*W; Low violation by 0.2*W. Clamp to 0-100%.

### CLI Change Requirements
Analyze the files that have been changes as part of this PR. Create a list of changes, each with a detailed summary of how each change affects the compliance of the CLI.

### Compliance Matrix Requirements
### Findings Table Requirements

The `Compliance Matrix` section must include a table with these columns:
The `Findings` section must include a table with these columns:

- `Finding` - In the form of `[SEVERITY-N]`, link to non-compliance finding in the `Non-compliance findings` section
- `Severity` - In the form of `[SEVERITY-N]`, link to non-compliance finding in the `Detailed analysis` section
- `Rule Summary`
- `Evidence`, include the name of the relevant section of the cli standard by using `reference to cli standard`
- `Notes`
- `Evidence`, include the name of the relevant section of the CLI standard by using `reference to cli standard`
- `How to fix`

### Non-compliance findings requirements
### Detailed analysis Requirements
Every non-compliance finding must use this exact heading format:

```
Expand All @@ -111,6 +112,6 @@ Each finding:
4. Should include whenever possible a remediation action that restores compliance
5. Cite the code block in markdown code format where the violation is detected

**Non-compliance findings checkpoint**: Verify that each non-compliance finding is linked in the compliance matrix, and that each row in the compliance matrix links to a finding. Do not proceed until confirmed.
**Detailed analysis checkpoint**: Verify that each non-compliance finding from `Detailed analysis` section is linked in the `Findings` table, and that each row in the `Findings` table has one corresponding non-compliance finding in `Detailed analysis` section. Do not proceed until confirmed.

**Command completion checkpoint**: Verify that the review file exists in `cli-review/cli-review.md` and is non-empty. Do not proceed until confirmed.
**Command completion checkpoint**: Verify that the review file exists in `cli-review/cli-review.md` and is non-empty, and its first non-empty line is exactly `# Canonical CLI automated review report`. Do not proceed until confirmed.
50 changes: 28 additions & 22 deletions cli-skill/references/cli-review-output-format.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,38 +3,26 @@

**Scope:** CLI standard compliance review of <paths for examined files>

## Summary

| **Severity** | **Count** | **Guideline Categories** |
| --- | --- | --- |
| High | <# high> | <main categories for high issues> |
| Medium | <# medium> | <main categories for medium issues> |
| Low | <# low> | <main categories for low issues> |
| Unrated | <# unrated> | <main categories for unrated issues> |
| **Total** | **<# total>** | |

**Overall rating:** <score> <rating badge>
> The scoring algorithm starts with 100%, number of commands N, weight W=100/N. For each High violation, reduce by 2*W; Medium violation by 1*W; Low violation by 0.5*W. Clamp to 0-100.

---
**Workflow:** [CLI skill version v.X](url)

## CLI changes in this PR
**Reference:** [Canonical CLI standards](url)

<describe findings: what code changes improve or decrease the CLI>
**Limitations:** Detection of clear CLI standards violations. Does not guarantee full compliance with CLI standards

---

## Compliance matrix
## Findings

| Finding | Rule Summary | Evidence | Notes |
| Severity | Rule Summary | Evidence | How to fix |
|---------|--------------|----------|-------|
<findings>
Example:
${| HIGH-1 | Every command acting on a primary object must be a verb. | `registration-csv` is a noun phrase, not a verb. | Should use a verb such as `generate-registration-csv`. |}
${| HIGH-1 | Commands are verbs - every command acting on a primary object must be a verb. | `registration-csv` is a noun phrase, not a verb. | Should use a verb such as `generate-registration-csv`. |}

---

## Non-compliance Findings (with citations)
<details>
<summary><H2>Detailed analysis</H2></summary>

Each finding heading must follow this exact format: `### [SEVERITY-N] <short description>`, where `SEVERITY` is `HIGH`, `MEDIUM`, `LOW`, or `UNRATED` (uppercase) and `N` is a per-severity counter starting at 1 (HIGH-1, HIGH-2, MEDIUM-1, LOW-1, …).

Expand All @@ -48,10 +36,28 @@ app.command(
)(_registration_csv_cmd)
```
The command name is a noun phrase. It should be a verb-noun compound (e.g. `generate-registration-csv`).

**Remediation:** Rename to `generate-registration-csv` or `export-registration-csv`.}

---

## Compliant Findings Summary
## CLI changes in this PR

<describe findings: what code changes improve or decrease the CLI>

---

## Summary

| **Severity** | **Count** | **Guideline Categories** |
| --- | --- | --- |
| High | <# high> | <main categories for high issues> |
| Medium | <# medium> | <main categories for medium issues> |
| Low | <# low> | <main categories for low issues> |
| Unrated | <# unrated> | <main categories for unrated issues> |
| **Total** | **<# total>** | |

**Overall score:** <score>
> The scoring algorithm starts with 100%, number of commands N, weight W=100/N. For each High violation, reduce by 1*W; Medium violation by 0.5*W; Low violation by 0.2*W. Clamp to 0-100.

<list of short bullet points of compliant findings>
</details>
23 changes: 1 addition & 22 deletions scripts/calculate_cli_score.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,39 +11,21 @@
]
}

Output: JSON with score, rating, and GitHub markdown badge.
Output: JSON with score, pass/fail status, and severity counts.
"""

import json
import sys
from pathlib import Path


def acceptance_rating(score: float) -> tuple[str, str]:
"""Return (label, github_markdown_badge) for a given score."""
if score > 95:
return "Excellent", "💚 **Excellent**"
elif score > 90:
return "Good", "🟢 **Good**"
elif score > 80:
return "Fair", "🟡 **Fair**"
elif score > 60:
return "Room for Improvement", "🟠 **Room for Improvement**"
else:
return "Need for Action", "🔴 **Need for Action**"


def calculate_score(issues_data: dict) -> dict:
"""Calculate compliance score from issues table."""
issues = issues_data.get("issues", [])
num_commands = issues_data.get("commands", 0)

if not issues:
label, badge = acceptance_rating(100)
return {
"score": 100,
"rating": label,
"rating_badge": badge,
"passed": True,
"summary": "No issues found",
"command_count": num_commands,
Expand All @@ -70,13 +52,10 @@ def calculate_score(issues_data: dict) -> dict:

score = max(0.0, min(100.0, score)) # Clamp to 0-100

label, badge = acceptance_rating(score)
passed = score > 80

return {
"score": score,
"rating": label,
"rating_badge": badge,
"passed": passed,
"summary": f"{high_count} High, {medium_count} Medium, {low_count} Low",
"command_count": num_commands,
Expand Down