feat(guard): injection-phrase FP instrumentation (Refs #12)#81
Merged
Conversation
Ship the unblocked, non-default-changing slice of issue #12 so operators can collect field false-positive data for WRD-RES-INJECT-PHRASE without flipping any default. No tier, default action, or error-replacement membership changes. - Add a discrete matched_phrases[] to the INJECT-PHRASE finding, threaded through the JSONL record and a matchedPhrases SARIF property, so per-phrase aggregation reads a structured field instead of parsing the free-text message. - Count inspected tools/call result frames (frames_inspected) in guard and inspect and emit it as a run-summary JSONL record + SARIF run property, giving a per-phrase FP rate its base-rate denominator. - Add --block-inject-phrase-only <file>: a default-off, per-phrase opt-in that blocks ONLY the named exact phrases while all other curated phrases stay monitor. Narrower than --block-inject-phrase; the future deterministic-subset promotion mechanism. Surfaced honestly in the startup posture banner. - Document the operator record -> inspect -> label FP-collection loop in RESULT_INSPECTION.md section 10. Security: the telemetry surface emits only the curated denylist phrase, rule id, tool/server metadata, action, and counts -- never raw result content (which can carry secrets/PII). Local-only, no phone-home. Covered by a dedicated no-leak test asserting raw content never reaches the emitted record. Tests: matched_phrases correctness, frame denominator, per-phrase selective block (named blocks, unnamed stays monitor), and the no-raw-content assertion. Whole-project coverage 86.6% (floor 80).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Ships the unblocked slice of #12 — the instrumentation operators need to collect field false-positive (FP) evidence for
WRD-RES-INJECT-PHRASE— without changing any default posture. Per the CSO scoping, the default-block flip stays gated (no field FP data yet, and INJECT-PHRASE error-replaces the whole tool result), so this PR references #12, it does not close it.WRD-RES-INJECT-PHRASEkeeps its tier (MONITOR), its default action (monitor-only), and its position inguard_result.error_rules. No matching was broadened toward regex/fuzzy (RESULT_INSPECTION.md §9 respected).What shipped
matched_phrases. A discretematched_phrases[]on the INJECT-PHRASE finding, threaded throughemit_res.result_finding_to_dict(JSONL) and amatchedPhrasesSARIF property — per-phrase aggregation without parsing prose.frames_inspectedcounted in bothguardandinspect, emitted as arun-summaryJSONL record (kind: "run-summary") + SARIF run property — the base rate for a per-phrase FP rate.--block-inject-phrase-only <file>blocks ONLY the named exact phrases while every other curated phrase stays monitor. Default-off, narrower than--block-inject-phrase, and the future deterministic-subset promotion mechanism. Reflected honestly in the startup posture banner.docs/RESULT_INSPECTION.md§10 documents the record → inspect → label loop and the graduated promotion path.Security (non-negotiable)
The telemetry surface emits only the curated denylist phrase, rule id, tool/server metadata, action, and counts — never raw result content (which can carry secrets/PII, per
WRD-RES-SECRET-ECHO). Aggregation is local-only; no phone-home. A dedicated test (test_no_raw_result_content_leaks_into_inject_phrase_telemetry) asserts distinctive raw markers never reach the emitted JSONL/SARIF record.Verification
pytest— 604 passed, 2 skipped; whole-project coverage 86.6% (floor 80, replicated withCOVERAGE_PROCESS_START=pyproject.tomlas CI does).ruff check .— clean.pip-audit— no known vulnerabilities.Issue #12 status
Remains open. This is the prerequisite evidence-collection work; the default-block flip stays gated on the FP data this instrumentation collects.
🤖 Generated with Claude Code
https://claude.ai/code/session_01QofWZ4MivqEKPevjSYuwdk