Skip to content

fix(review-pr): stop posting a false LGTM when the agent exits 0 without a review#41

Closed
Sayt-0 wants to merge 1 commit into
mainfrom
fix/review-pr-false-lgtm-fallback
Closed

fix(review-pr): stop posting a false LGTM when the agent exits 0 without a review#41
Sayt-0 wants to merge 1 commit into
mainfrom
fix/review-pr-false-lgtm-fallback

Conversation

@Sayt-0

@Sayt-0 Sayt-0 commented Jul 6, 2026

Copy link
Copy Markdown
Member

Summary

When the review agent exits 0 without posting a review (a degenerate run, not a clean one), the action posted a fallback review starting with "No issues found" plus "LGTM", misrepresenting a failed run as a clean review. That body also matched COMPLETED_BODY_MARKERS in src/incremental-review, so the unreviewed head SHA was recorded as reviewed and subsequent incremental reviews skipped those commits permanently (silent coverage hole).

Evidence: https://github.com/docker/cli/actions/runs/28783584164 (PR docker/cli#7007). The drafter sub-agent returned a degenerate response, the re-delegated drafter got stuck in a read_file/list_directory loop, the agent exited 0 without posting anything, and the action then posted review 4634830529 as a clean LGTM.

Failure chain (before)

agent run degenerates (exit 0, no review posted)
  -> exit-0 branch posts the "No issues found" LGTM fallback
  -> fallback body matches COMPLETED_BODY_MARKERS
  -> head SHA recorded as reviewed
  -> incremental reviews skip those commits forever

Changes

File Change
review-pr/action.yml (exit-0 branch only) Fallback replaced by a neutral, non-approving notice: "Review did not complete, the agent finished without posting a review, re-request a review from docker-agent to retry", with a logs link. Contains neither ### Assessment: nor the old LGTM prefix, so it can never count as a completed review.
review-pr/action.yml (dedup guard) Paginated via gh api --paginate with per-page counts summed by jq -s 'add // 0', covering PRs with more than 30 reviews. Matches the new notice plus the legacy LGTM prefix on the reviews endpoint and keeps the legacy issue-comment check, so PRs that already received an old fallback never get a second fallback of either kind. The echo "0" error fallback became true: composite shell: bash runs with pipefail, and on a gh failure the old pattern would have emitted two lines ("0" from jq plus "0" from echo) and broken the numeric test; the existing ${VAR:-0} covers the empty case.
src/incremental-review/incremental-review.ts COMPLETED_BODY_MARKERS reduced to ['### Assessment:'], so past and future fallbacks never mark commits as reviewed. The legitimate zero-findings path is unaffected: the agent itself posts the ### Assessment: APPROVE body, which stays a completion marker. Header comments updated.
src/rate-limit/index.ts Comment wording only ("LGTM" to "incomplete-run"). Behavior unchanged: any non-empty-body bot review is still counted, including legacy LGTM fallbacks already in the wild.
src/incremental-review/__tests__/incremental-review.test.ts The "accepts the LGTM fallback" test now asserts the opposite (returns null), plus a new test asserting the new fallback text is not treated as completed. Net +1 test.

Behavior matrix

Scenario Before After
exit 0, review posted no fallback no fallback (unchanged)
exit 0, no review posted LGTM review posted, commits marked reviewed neutral notice posted, commits stay unreviewed
exit 0, no review, fallback already on PR (any page) duplicate possible beyond 30 reviews skipped (paginated, both generations matched)
exit 124 (timeout) timeout review unchanged
exit != 0 (failure) failure review unchanged

Validation

Check Result
pnpm build pass
pnpm test 746 passed (29 files), net +1 test
pnpm lint (biome + tsc + actionlint) pass, no new issues
Extracted post-summary step run under bash -eo pipefail with a stubbed gh posts the notice when no fallback exists; skips when a legacy LGTM is found on page 2; skips when the new notice exists; fails open on gh API error
jq expressions against realistic review payloads correct counts, multi-page summing works, empty input yields 0
Timeout and failure branches re-run in the simulation, bodies unchanged

Possible follow-ups (not included, out of the requested scope)

  • AGENTS.md line 183 still says completed reviews are "assessment or LGTM bodies only", now inaccurate.
  • SECURITY.md line 114 still mentions "timeout/error/LGTM fallbacks".
  • The rate-limit test "counts timeout / error / LGTM fallback reviews" stays valid for legacy bodies; the new notice body could be added to it.

…out a review

A degenerate agent run (exit 0, no review posted) triggered the fallback
"No issues found" LGTM review, misrepresenting a failed run as a clean
review, and incremental-review treated that body as a completed run,
permanently skipping those commits on subsequent incremental reviews.

- replace the fallback with a neutral incomplete-run notice
- dedup guard: paginate and match both the new notice and the legacy prefix
- drop the LGTM marker from COMPLETED_BODY_MARKERS (assessment bodies only)
- adjust now-stale comments in incremental-review and rate-limit
@Sayt-0

Sayt-0 commented Jul 6, 2026

Copy link
Copy Markdown
Member Author

Maybe should try something else

@Sayt-0 Sayt-0 closed this Jul 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant