Review agent verdict contradicted its own analysis on PR #32: chose comment despite saying findings should be addressed before merge

[fullsend-ai-retro]

What happened

On PR #32 (#32), the fullsend review agent identified 6 medium-severity findings and 6 low-severity findings. Its summary stated: "None are blocking, but several should be addressed before merge." Despite this statement, the agent chose comment (COMMENTED review state) rather than request-changes, which meant the findings had no blocking effect on the PR.

Per the review agent definition (agents/review.md), the verdict criteria are:

• request-changes: "one or more critical or high findings; multiple medium-severity findings which could affect the intended outcome of the PR"
• comment-only: "medium-severity findings worth noting but none that should block"

The agent had 6 medium findings, several of which were functional bugs affecting the PR's intended outcome (vouch-check fail-closed behavior, invalid schema terminology, unnecessary security permissions). These meet the request-changes criteria. The agent's own text ("should be addressed before merge") aligns with request-changes semantics but it chose comment-only.

What could go better

The review agent's verdict decision logic needs tighter calibration between its analysis text and its action selection. When the agent's own natural-language summary says findings "should be addressed before merge," the corresponding action should be request-changes, not comment. The disconnect between text and action removed the only automated blocking signal on the PR, contributing to all findings being merged unaddressed.

This is related to but distinct from #3438 (governance docs getting approve instead of comment-only). Here the issue is the opposite direction: the agent should have escalated from comment to request-changes based on the nature and quantity of medium findings.

Confidence: Medium-high. The criteria in the agent definition seem clear ("multiple medium-severity findings which could affect the intended outcome"), but the agent may have judged that docs/CI PRs have lower blast radius and calibrated accordingly. If that's intentional, the agent definition should make this explicit. If not, the verdict selection logic needs a consistency check.

Proposed change

In agents/review.md (Output format → Outcome section), clarify the request-changes criteria to reduce ambiguity:

1. Add explicit guidance: "If your summary states that findings should be addressed, fixed, or resolved before merge, the verdict must be request-changes, not comment. The summary text and the verdict action must be consistent."

2. Refine the medium-severity escalation criteria. The current language ("multiple medium-severity findings which could affect the intended outcome") is subjective. Consider: "Two or more medium-severity findings, at least one of which identifies a functional bug (incorrect behavior, permission error, schema violation, or silent failure), require request-changes."

3. In the pr-review skill's synthesis step, add a self-consistency check: before emitting the final verdict, verify that the verdict action is consistent with the language used in the summary paragraph.

Validation criteria

1. Re-run the review agent on PR docs: add CLAUDE.md, COMMITS.md, CONTRIBUTING.md, LICENSE, and repo workflows #32's diff (commit 7e7509f). The verdict should be request-changes (not comment) in at least 4 out of 5 trials.
2. On a docs-only PR with only style/process medium findings (no functional bugs), the agent should still use comment — the change should not over-escalate.
3. In any review where the summary text says findings "should be addressed before merge" or equivalent, the verdict is request-changes.
4. No increase in false-positive request-changes verdicts on PRs where the review agent currently correctly uses comment.

---

Generated by retro agent from #32

[fullsend-ai-triage]

🤖 Finished Triage · ✅ Success · Started 10:38 PM UTC · Completed 10:42 PM UTC
Commit: e8381e3 · View workflow run →

[fullsend-ai-triage]

Triage Summary

Severity: Medium · Category: Bug

Problem

The review agent's verdict selection logic is inconsistent with its own analysis text. On PR #32, the agent identified 6 medium-severity findings — several of which were functional bugs (vouch-check fail-closed, invalid schema terminology, unnecessary security permissions) — and wrote that findings "should be addressed before merge," yet selected comment instead of request-changes. This removed the only automated blocking signal, and all findings were merged unaddressed.

This is a systemic issue also tracked in fullsend-ai/fullsend#2940 with additional evidence from multiple other PRs.

Root Cause Hypothesis

Two compounding factors:

1. Subjective escalation criteria — the current language ("multiple medium-severity findings which could affect the intended outcome") gives the agent too much discretion to downgrade despite meeting the criteria
2. No self-consistency check — the agent can write blocking-sounding language in its summary while selecting a non-blocking verdict without detecting the contradiction

Recommended Fix

Three changes to agents/review.md (Outcome section) and skills/pr-review/SKILL.md (step 6f):

1. Add a self-consistency rule: if the summary states findings should be addressed before merge, the verdict must be request-changes
2. Replace the subjective "could affect the intended outcome" with an objective criterion: "Two or more medium findings, at least one identifying a functional bug, require request-changes"
3. Add a pre-emit consistency check step in the synthesis phase

Proposed Test Case

Test 1 — Escalation (PR #32 replay):
  Input: PR #32 diff (commit 7e7509f)
  Expected: verdict = request-changes (4/5 trials minimum)
  Rationale: 6 medium findings including functional bugs

Test 2 — Non-over-escalation (docs-only PR):
  Input: PR with only style/process medium findings, no functional bugs
  Expected: verdict = comment (not request-changes)
  Rationale: Style findings should not block

Test 3 — Language-verdict consistency:
  Input: Any PR where agent summary says "should be addressed before merge"
  Expected: verdict = request-changes
  Rationale: Summary language and verdict action must agree

---

Labels: This issue targets the review agent's verdict logic in agents/review.md and skills/pr-review/SKILL.md. It describes a bug in verdict calibration, not a new feature.

[fullsend-ai-coder]

🤖 Finished Code · ✅ Success · Started 10:43 PM UTC · Completed 10:52 PM UTC
Commit: e8381e3 · View workflow run →