feat(triage): add effort-based gating before auto-promoting to coder#36
Open
rh-hemartin wants to merge 2 commits into
Open
feat(triage): add effort-based gating before auto-promoting to coder#36rh-hemartin wants to merge 2 commits into
rh-hemartin wants to merge 2 commits into
Conversation
Add effort estimation to triage agent (0.25-3 scale matching RICE). Bug/docs/performance issues with effort >= 2.0 get triaged instead of ready-to-code, routing high-complexity work through human review before dispatch. Migrated from fullsend-ai/fullsend feat/triage-effort-analysis. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Hector Martinez <hemartin@redhat.com>
|
🤖 Review · |
PR Summary by QodoAdd effort scoring and auto-promotion gating to triage pipeline
AI Description
Diagram
High-Level Assessment
Files changed (4)
|
- Add regex guard for non-numeric effort values (fail closed to triaged) - Add test for missing-effort and non-numeric-effort fallback paths - Fix schema description to match triage.md effort scale labels Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Hector Martinez <hemartin@redhat.com>
Code Review by Qodo
Context used✅ Compliance rules (platform):
55 rules 1. Protected scripts/ files modified
|
![qodo-code-review[bot]](https://avatars.githubusercontent.com/in/484649?s=60&v=4){kind=small}
Comment on lines
263
to
+288
| # ready-to-code, which triggers the code agent. Feature work and anything | ||
| # else receives the triaged label and waits for human prioritization | ||
| # (per #561, only feature issues should require human review before coding). | ||
| # | ||
| # Effort-based gating (#2207): high-effort bug/docs/performance issues | ||
| # (effort >= 2.0) get triaged instead of ready-to-code to route them through | ||
| # human review before dispatching to the code agent. | ||
| CATEGORY=$(jq -r '.triage_summary.category // "unknown"' "${RESULT_FILE}") | ||
| echo "Category: ${CATEGORY}" | ||
| EFFORT=$(jq -r '.triage_summary.effort // "null"' "${RESULT_FILE}") | ||
| echo "Category: ${CATEGORY}, Effort: ${EFFORT}" | ||
| case "${CATEGORY}" in | ||
| bug|documentation|performance) | ||
| echo "Deferring ready-to-code label (${CATEGORY}) until after label_actions..." | ||
| DEFERRED_LABEL="ready-to-code" | ||
| # Effort threshold: >= 2.0 is substantial work, route to human review. | ||
| # Missing effort → conservatively route to human review (triaged). | ||
| if [[ "${EFFORT}" == "null" ]]; then | ||
| echo "Missing effort estimate — applying triaged label for human review..." | ||
| remove_label "ready-to-code" | ||
| add_label "triaged" | ||
| elif awk -v effort="${EFFORT}" 'BEGIN { exit (effort >= 2.0 ? 0 : 1) }'; then | ||
| echo "High effort (${EFFORT}) — applying triaged label for human review..." | ||
| remove_label "ready-to-code" | ||
| add_label "triaged" | ||
| else | ||
| echo "Low effort (${EFFORT}) — deferring ready-to-code label until after label_actions..." | ||
| DEFERRED_LABEL="ready-to-code" | ||
| fi |
There was a problem hiding this comment.
1. Protected scripts/ files modified 📜 Skill insight § Compliance
This PR modifies files under protected governance/infrastructure paths (e.g., scripts/ and agents/), which require explicit human review and must not be auto-approved. Protected-path changes increase governance and supply-chain risk if merged without manual oversight.
Comment on lines
263
to
+288
| # ready-to-code, which triggers the code agent. Feature work and anything | ||
| # else receives the triaged label and waits for human prioritization | ||
| # (per #561, only feature issues should require human review before coding). | ||
| # | ||
| # Effort-based gating (#2207): high-effort bug/docs/performance issues | ||
| # (effort >= 2.0) get triaged instead of ready-to-code to route them through | ||
| # human review before dispatching to the code agent. | ||
| CATEGORY=$(jq -r '.triage_summary.category // "unknown"' "${RESULT_FILE}") | ||
| echo "Category: ${CATEGORY}" | ||
| EFFORT=$(jq -r '.triage_summary.effort // "null"' "${RESULT_FILE}") | ||
| echo "Category: ${CATEGORY}, Effort: ${EFFORT}" | ||
| case "${CATEGORY}" in | ||
| bug|documentation|performance) | ||
| echo "Deferring ready-to-code label (${CATEGORY}) until after label_actions..." | ||
| DEFERRED_LABEL="ready-to-code" | ||
| # Effort threshold: >= 2.0 is substantial work, route to human review. | ||
| # Missing effort → conservatively route to human review (triaged). | ||
| if [[ "${EFFORT}" == "null" ]]; then | ||
| echo "Missing effort estimate — applying triaged label for human review..." | ||
| remove_label "ready-to-code" | ||
| add_label "triaged" | ||
| elif awk -v effort="${EFFORT}" 'BEGIN { exit (effort >= 2.0 ? 0 : 1) }'; then | ||
| echo "High effort (${EFFORT}) — applying triaged label for human review..." | ||
| remove_label "ready-to-code" | ||
| add_label "triaged" | ||
| else | ||
| echo "Low effort (${EFFORT}) — deferring ready-to-code label until after label_actions..." | ||
| DEFERRED_LABEL="ready-to-code" | ||
| fi |
There was a problem hiding this comment.
2. Docs contradict effort gating 📜 Skill insight ⚙ Maintainability
The triage documentation still states that ready-to-code applies to bug/documentation/performance issues as a category-level rule, but the post-triage logic now routes high-effort (and missing-effort) items to triaged. This mismatch can mislead maintainers and auditors about when auto-promotion to coder actually occurs.
Agent Prompt
## Issue description
`docs/triage.md` describes `ready-to-code`/`triaged` semantics that no longer match the updated post-triage behavior (effort-based gating for bug/docs/performance).
## Issue Context
The post-triage script now applies `triaged` (human review) for bug/documentation/performance items when `effort >= 2.0` or when `effort` is missing, instead of always applying `ready-to-code` for these categories.
## Fix Focus Areas
- docs/triage.md[32-44]
- scripts/post-triage.sh[262-288]
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
Comment on lines
+271
to
+288
| EFFORT=$(jq -r '.triage_summary.effort // "null"' "${RESULT_FILE}") | ||
| echo "Category: ${CATEGORY}, Effort: ${EFFORT}" | ||
| case "${CATEGORY}" in | ||
| bug|documentation|performance) | ||
| echo "Deferring ready-to-code label (${CATEGORY}) until after label_actions..." | ||
| DEFERRED_LABEL="ready-to-code" | ||
| # Effort threshold: >= 2.0 is substantial work, route to human review. | ||
| # Missing effort → conservatively route to human review (triaged). | ||
| if [[ "${EFFORT}" == "null" ]]; then | ||
| echo "Missing effort estimate — applying triaged label for human review..." | ||
| remove_label "ready-to-code" | ||
| add_label "triaged" | ||
| elif awk -v effort="${EFFORT}" 'BEGIN { exit (effort >= 2.0 ? 0 : 1) }'; then | ||
| echo "High effort (${EFFORT}) — applying triaged label for human review..." | ||
| remove_label "ready-to-code" | ||
| add_label "triaged" | ||
| else | ||
| echo "Low effort (${EFFORT}) — deferring ready-to-code label until after label_actions..." | ||
| DEFERRED_LABEL="ready-to-code" | ||
| fi |
There was a problem hiding this comment.
3. Missing effort blocks promotion 🐞 Bug ≡ Correctness
scripts/post-triage.sh routes bug/documentation/performance issues to triaged when triage_summary.effort is missing, which can suppress auto-promotion whenever the agent omits the field. Because effort is optional in schemas/triage-result.schema.json, these results still validate, making this behavior easy to trigger unintentionally.
Agent Prompt
## Issue description
The post-triage gating logic treats missing `triage_summary.effort` as a reason to apply the `triaged` label for bug/documentation/performance categories, but the triage result schema does not require `effort`. This means valid (schema-passing) agent outputs can unexpectedly disable auto-promotion.
## Issue Context
- Triage prompt guidance says to estimate effort for bug/docs/performance and to bias toward auto-promotion (round down), but the implementation routes *missing* effort to human review.
- The harness validates against `schemas/triage-result.schema.json` before running `post-triage.sh`, so schema constraints are the right place to prevent missing effort from reaching the gating logic (or the script should default missing effort to a low-effort value).
## Fix Focus Areas
- scripts/post-triage.sh[270-288]
- schemas/triage-result.schema.json[121-144]
- agents/triage.md[247-263]
## What to change
Choose one consistent policy and implement it end-to-end:
1) **Require effort for bug/docs/performance when action=sufficient**
- Update `schemas/triage-result.schema.json` with an `if/then` that, when `action == "sufficient"` and `triage_summary.category` is in `{bug, documentation, performance}`, requires `triage_summary.effort`.
- In `post-triage.sh`, replace the “missing effort => triaged” branch with a hard error (since it should be schema-impossible) or keep a defensive error in case the script is run without schema validation.
OR
2) **Default missing effort to low effort** (to preserve “bias toward auto-promotion”)
- In `post-triage.sh`, treat missing effort as low effort (e.g., default to `1.0` and defer `ready-to-code`) and log a warning.
- Optionally keep schema optional, but consider adding an agent-side requirement if you want stronger guarantees.
Also ensure the written guidance in `agents/triage.md` matches the chosen behavior (especially the handling of missing effort).
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
|
🤖 Finished Review · ✅ Success · Started 6:35 AM UTC · Completed 6:44 AM UTC |
Closed
5 tasks
ReviewVerdict: comment · 3 medium, 2 low findings The effort-based gating logic is well-designed: the bash branching correctly handles missing, non-numeric, and numeric effort values; the schema addition is backward-compatible (optional field); and tests cover the primary scenarios including boundary cases at 2.0 and edge cases for missing/non-numeric effort. The overall approach — gating auto-promotion for high-complexity issues — fits the existing triage pipeline architecture. Three areas worth addressing: Findings1. GHA workflow command injection via unsanitized EFFORT interpolation · medium · securityFile: The new The same risk applies to the earlier
Note: This is defense-in-depth — schema validation (effort type: number) runs before the post-script, so exploitation requires schema validation bypass. The existing Remediation: Sanitize EFFORT before all safe_effort="${EFFORT//$'\n'/}"
safe_effort="${safe_effort//$'\r'/}"
safe_effort="${safe_effort//::/}"
safe_effort="${safe_effort//%0A/}"
safe_effort="${safe_effort//%0a/}"
safe_effort="${safe_effort//%0D/}"
safe_effort="${safe_effort//%0d/}"2. Documentation now contradicts actual behavior —
|
fullsend-ai-review
Bot
added
the
requires-manual-review
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
effortfieldMigrated from fullsend-ai/fullsend
feat/triage-effort-analysis.Test plan
bash scripts/post-triage-test.sh— all cases pass including new effort-gating scenarioseffortfield with correct bounds🤖 Generated with Claude Code