fix(gate): fire failure nudge only on machine-confirmed status#15
Open
AnswerTheo wants to merge 1 commit into
Open
fix(gate): fire failure nudge only on machine-confirmed status#15AnswerTheo wants to merge 1 commit into
AnswerTheo wants to merge 1 commit into
Conversation
PostToolUse gate flagged 'observed a tool failure' on successful commands whose output merely contained a failure keyword (No such file or directory, failed, error:) — greps, log reads, and SSM runs that exit 0. The host Bash payload exposes no exit code, so exit_success() always fell through to a text regex. Split explicit_status() (machine-readable exit code / success bool) from the text guess, and gate detect_failure() on ground truth only. exit_success() keeps its text fallback for verification bookkeeping, so should_block_stop() is unaffected. Verified: benign exit-0 + 'No such file' and grep 'failed' no longer fire; real exit_code:1 (pytest) and Edit success:false still fire; test_gate.py (6/6) and test_gate_robustness.py (12/12) pass. Closes fivetaku#14 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #14.
Problem
The
PostToolUseobservation gate firesfablize gate observed a tool failureon successful commands whose output merely contains a failure-looking word (No such file or directory,failed,error:,N errors). In practice this misfires constantly on exit-0 commands:grepresults, log/file reads, and remote (SSM/ssh) runs.Root cause
The host's Bash
PostToolUsepayload carries noexit_code/exitCode/returncode/status/successfield, soexit_success()always falls through toFAILURE_RE.search(text)— a pure text guess.detect_failure()then flags any exit-0 command whose stdout contains a failure keyword.Edit/Writeare affected too when the edited text itself contains such words.Fix
Split machine-readable status (
explicit_status()) from the text guess, and gate the failure nudge on ground truth only.exit_success()keeps its text fallback for verification-command bookkeeping, soshould_block_stop()is unaffected. If a future host exposes a real exit code,explicit_status()picks it up automatically.Verification
hooks/gate_post_tool.py:No such file or directory→ was firing, now{}grepoutput containingfailed/error:→ was firing, now{}exit_code: 1(pytest) → still fires ✅Editsuccess: false→ still fires ✅tests/test_gate.py(6/6, incl. S3 pytest-exit-1 → BLOCK) andtests/test_gate_robustness.py(12/12).