Skip to content

feat: add prompt-type Stop hook for LLM completion verification#3192

Merged
mabry1985 merged 1 commit intodevfrom
feat/stop-hook-completion-verifier
Mar 30, 2026
Merged

feat: add prompt-type Stop hook for LLM completion verification#3192
mabry1985 merged 1 commit intodevfrom
feat/stop-hook-completion-verifier

Conversation

@mabry1985
Copy link
Copy Markdown
Contributor

@mabry1985 mabry1985 commented Mar 30, 2026

Summary

  • Adds a type: prompt Stop hook that runs an LLM evaluation before every session end to catch incomplete work
  • Detects four failure patterns: silent exit, partial work, error abandonment, and untested changes
  • Chains before the existing evaluate-session.js pattern miner — verifier runs first, miner only fires if work is complete

Context

Inspired by patterns from claude-howto. The prompt-type hook uses an LLM to evaluate last_assistant_message rather than regex pattern matching, catching nuanced failure modes like agents that hit an error and suggest the user handle it instead of investigating.

Test plan

  • CI passes
  • Verify hook fires on session end (check for 30s prompt evaluation)
  • Verify normal Q&A sessions pass through without blocking
  • Verify incomplete work (e.g., planned edits not made) triggers continuation

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Chores
    • Enhanced internal verification mechanisms to improve system reliability and error detection.

Adds a completion verifier that runs before evaluate-session.js on every
session end. An LLM evaluates the last assistant message against four
failure patterns (silent exit, partial work, error abandonment, untested
changes) and prevents premature stop if work is genuinely incomplete.
Normal conversation endings pass through cleanly.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@mabry1985 mabry1985 enabled auto-merge (squash) March 30, 2026 04:18
@coderabbitai
Copy link
Copy Markdown

coderabbitai bot commented Mar 30, 2026

Caution

Review failed

Pull request was closed or merged during review

📝 Walkthrough

Walkthrough

A new Stop hook was added to .claude/settings.json that deploys a prompt-type completion verifier with a 30-second timeout. The hook evaluates the assistant's final message for four incomplete-work patterns and returns either a description of remaining work or confirmation that work is complete.

Changes

Cohort / File(s) Summary
Completion Verification Hook
.claude/settings.json
Added a Stop hook that runs a prompt-type completion verifier (30-second timeout) to check for incomplete work patterns: silent exit, partial work, error abandonment, and untested changes. Returns remaining work or "Work is complete."

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~5 minutes

Poem

🐰 A hook now watches work with care,
Checking if the task is fair,
Silent exits, partial mends—
Our verifier sees where work ends!
Complete or not, the truth it tells,
In .claude where the checker dwells. 🌟

🚥 Pre-merge checks | ✅ 3
✅ Passed checks (3 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately describes the main change: adding a prompt-type Stop hook for LLM completion verification in .claude/settings.json. It is concise, specific, and clearly conveys the primary purpose of the changeset.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/stop-hook-completion-verifier

Comment @coderabbitai help to get the list of available commands and usage tips.

@mabry1985 mabry1985 merged commit d10fbd2 into dev Mar 30, 2026
3 of 4 checks passed
@mabry1985 mabry1985 deleted the feat/stop-hook-completion-verifier branch March 30, 2026 04:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant