Skip to content

fix(gate): fire failure nudge only on machine-confirmed status#15

Open
AnswerTheo wants to merge 1 commit into
fivetaku:mainfrom
AnswerTheo:fix/gate-text-heuristic-false-positive
Open

fix(gate): fire failure nudge only on machine-confirmed status#15
AnswerTheo wants to merge 1 commit into
fivetaku:mainfrom
AnswerTheo:fix/gate-text-heuristic-false-positive

Conversation

@AnswerTheo

Copy link
Copy Markdown

Closes #14.

Problem

The PostToolUse observation gate fires fablize gate observed a tool failure on successful commands whose output merely contains a failure-looking word (No such file or directory, failed, error:, N errors). In practice this misfires constantly on exit-0 commands: grep results, log/file reads, and remote (SSM/ssh) runs.

Root cause

The host's Bash PostToolUse payload carries no exit_code/exitCode/returncode/status/success field, so exit_success() always falls through to FAILURE_RE.search(text) — a pure text guess. detect_failure() then flags any exit-0 command whose stdout contains a failure keyword. Edit/Write are affected too when the edited text itself contains such words.

Fix

Split machine-readable status (explicit_status()) from the text guess, and gate the failure nudge on ground truth only. exit_success() keeps its text fallback for verification-command bookkeeping, so should_block_stop() is unaffected. If a future host exposes a real exit code, explicit_status() picks it up automatically.

Verification

  • Before/after on hooks/gate_post_tool.py:
    • benign exit-0 + No such file or directory → was firing, now {}
    • grep output containing failed/error: → was firing, now {}
    • real failure exit_code: 1 (pytest) → still fires ✅
    • real Edit success: false → still fires ✅
  • Regression suites pass: tests/test_gate.py (6/6, incl. S3 pytest-exit-1 → BLOCK) and tests/test_gate_robustness.py (12/12).

PostToolUse gate flagged 'observed a tool failure' on successful
commands whose output merely contained a failure keyword (No such
file or directory, failed, error:) — greps, log reads, and SSM runs
that exit 0. The host Bash payload exposes no exit code, so
exit_success() always fell through to a text regex.

Split explicit_status() (machine-readable exit code / success bool)
from the text guess, and gate detect_failure() on ground truth only.
exit_success() keeps its text fallback for verification bookkeeping,
so should_block_stop() is unaffected.

Verified: benign exit-0 + 'No such file' and grep 'failed' no longer
fire; real exit_code:1 (pytest) and Edit success:false still fire;
test_gate.py (6/6) and test_gate_robustness.py (12/12) pass.

Closes fivetaku#14

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

PostToolUse gate: false 'observed a tool failure' on successful commands (text-heuristic misfire)

1 participant