Skip to content

feat: add post-PR code review using the reviewer agent#5

Open
Maxtanh-Meta wants to merge 3 commits into
logosc:mainfrom
Maxtanh-Meta:post-pr-code-review
Open

feat: add post-PR code review using the reviewer agent#5
Maxtanh-Meta wants to merge 3 commits into
logosc:mainfrom
Maxtanh-Meta:post-pr-code-review

Conversation

@Maxtanh-Meta

Copy link
Copy Markdown

Summary

After a PR is created, run the configured reviewer agent on the actual code diff and post its findings as a PR comment. This gives operators visibility into code quality without manual review.

How it works

  1. After PR creation (step 17 in the orchestration), the orchestrator gets the diff (git diff <base>..HEAD)
  2. Runs the configured reviewer agent (e.g. Claude) with a fixed code-review prompt
  3. Posts the review as a comment on the PR with a ✅ LGTM or ⚠️ Changes Requested verdict

Design decisions

  • Best-effort: If the reviewer fails or is not configured, the PR is still created normally
  • Non-blocking: Does not prevent the PR from being labeled pr-ready
  • Reuses existing infrastructure: Same reviewer provider/timeout config, just a different prompt
  • Fixed prompt: Not user-controllable via WORKFLOW.md (consistent with plan reviewer security model)
  • Diff truncation: Large diffs are capped at 30KB to stay within model context limits

Example output on PR

[symphony-go] 🔍 Code Review: ✅ LGTM

The implementation is clean, well-tested, and matches the planned scope.

Changes

  • internal/orchestrator/job.go: Add runPostPRCodeReview method + call it after PR creation

Testing

Tested end-to-end — Claude successfully reviews the diff and posts findings on the PR.

After a PR is created, run the configured reviewer agent on the actual
code diff (not just the plan). The reviewer evaluates the implementation
for bugs, style issues, missing error handling, test coverage gaps, and
security concerns, then posts its findings as a comment on the PR.

This is a best-effort step — if the reviewer fails or is not configured,
the PR is still created and labeled normally. The code review is
informational and does not block the PR.

The reviewer reuses the existing reviewer infrastructure (same provider,
same timeout config) but with a code-review-specific prompt that
includes the full diff against the base branch.
@Maxtanh-Meta

Copy link
Copy Markdown
Author
Screenshot 2026-05-14 at 12 16 14 PM

@Maxtanh-Meta

Copy link
Copy Markdown
Author

Code Review (via Codex)

Overall: Request Changes

Findings

1. State machine conflict with triggerRevision

A "changes requested" review calls triggerRevision, which closes the PR, deletes the remote branch, removes the worktree, and resets the job. But runImplementation continues after and still posts the "opened PR" comment, applies the PR-ready label, and saves job.Status = pr_ready. This can leave the system advertising a closed/deleted PR as ready — contradicting the "best-effort, non-blocking" design. If revision cycling is desired, runPostPRCodeReview needs to return a signal so the caller stops the PR-ready transition.

2. Synchronous review blocks PR transition

The reviewer runs synchronously before the issue gets the PR link comment, before labels transition, and before the job is saved as pr_ready. A slow or hanging reviewer leaves a created PR in limbo. Since reviewer timeout can be disabled, this is a real operational risk. For true "non-blocking" behavior: save/transition first, then run review afterward (or from a bounded async path).

3. Diff base mismatch

The diff is computed as git diff <baseBranch>..HEAD, but the worktree is created from origin/<baseBranch> and existing code uses origin/<baseBranch>...HEAD (three dots). A stale local base branch can cause the reviewer to see unrelated upstream changes or miss the actual merge-base diff. Use the same strategy as gitDiffNameOnly — ideally origin/<base>...HEAD.

4. Prompt/parser mismatch with approval.Reviewer

The code-review prompt asks for a ## Verdict with "LGTM" / "Changes Requested", but approval.Reviewer.Review wraps it in the existing plan-review prompt and parses only a ## Decision JSON block with approve / reject. This mismatch can produce "malformed reviewer output" errors or lose the actual review findings. Should use a dedicated code-review prompt/parser or align with the existing decision schema.

Design Feedback

Reusing the approval reviewer for post-PR review is reasonable, but it needs a separate contract from plan approval. Plan approval produces a gate decision; PR code review should produce review text and a severity/verdict without mutating orchestration state — unless that behavior is explicitly designed into the state machine.

@Maxtanh-Meta

Copy link
Copy Markdown
Author

Code Review — Round 2 (via Codex)

The previous 4 issues are all resolved ✅. One new finding:


[P2] Missing seedSubscriptionAuth for review HOMEjob.go:1289-1291

When the reviewer provider runs in subscription-auth mode (the documented Claude/Codex flow that relies on symlinks from seedSubscriptionAuth), the new reviewerHome created for post-PR review is separate from the already-seeded per-issue home and never gets those auth links.

In that configuration the reviewer subprocess starts with an isolated HOME lacking ~/.claude, ~/.codex, etc. — fails as unauthenticated, and silently skips posting the code review.

Fix: Either reuse the seeded layout home, or call seedSubscriptionAuth(reviewerHome) after creating it.


All other findings from round 1 confirmed fixed. The only pre-existing test failure (TestPerAxisWorkflowAndValidation) is unrelated to this PR.

Without this fetch, when multiple jobs run sequentially and earlier PRs
get merged between runs, origin/<base> becomes stale. The three-dot diff
(origin/main...HEAD) then includes commits from previously-merged PRs,
causing the reviewer to flag unrelated changes.

Observed on XF-APAC/one-minute-games PR #50: the code review saw README
changes from PR #49 (issue #38) because origin/main hadn't been updated
since that PR was merged.
@Maxtanh-Meta

Copy link
Copy Markdown
Author

Additional fix: fetch before diff

Added git fetch origin <base> before computing the code review diff.

Problem: When multiple issues are processed sequentially and earlier PRs get auto-merged, origin/main becomes stale. The three-dot diff (origin/main...HEAD) then includes commits from previously-merged PRs, causing the reviewer to flag unrelated changes.

Observed on: XF-APAC/one-minute-games PR #50 — code review incorrectly flagged README.md changes from the already-merged PR #49.

Fix: Fetch origin/<base> before computing the diff, matching what workspace.Create() already does.

Maxtanh-Meta added a commit to Maxtanh-Meta/symphony-go that referenced this pull request May 19, 2026
fix(orchestrator): detect '# Plan' heading to prevent duplicate in PR body
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant