Add --no-heuristics CLI flag and heuristics_filter audit (v0.21 E5)#121
Open
pengfei-threemoonslab wants to merge 2 commits into
Open
Add --no-heuristics CLI flag and heuristics_filter audit (v0.21 E5)#121pengfei-threemoonslab wants to merge 2 commits into
pengfei-threemoonslab wants to merge 2 commits into
Conversation
Closes round-3 / round-4 review's E5 carryover. `Finding.provenance_kind` has shipped on every report since v0.15 as required+non-nullable wire metadata but had no consumer for four review cycles. v0.21 lands the security/GRC-friendly consumer the field was always designed for: a filter that excludes findings whose provenance is `keyword_heuristic` or `regex_heuristic` from the active release-gating set. Wire surface (additive, schema bumped 0.20 -> 0.21): - New `--no-heuristics` flag on `agents-shipgate scan` (stable in 0.x). When set, findings whose `provenance_kind` is in the EXCLUDE set (today: keyword_heuristic, regex_heuristic) are marked `suppressed=True` with `suppression_reason="filtered by --no-heuristics"` BEFORE the release decision is built. Filtered findings remain in `findings[]` for transparency; they no longer gate release. - New top-level `report.heuristics_filter` audit envelope. Required + always present on emitted scans regardless of whether the flag was set (parallel to `privacy_audit`). Fields: `enabled`, `excluded_provenance_kinds`, `filtered_finding_count`, `filtered_by_kind`. Earns `provenance_kind`'s contract weight by giving it a first-class consumer. Decision recorded: round-4 review offered ship-or-retire. We ship. Retiring would force a deprecation cycle on a stable-contract field used by every report since v0.15; shipping the flag earns the weight and serves a real audience (security/GRC reviewers triaging declared-only findings). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…iew) Addresses PR #121 review: [P1] v0.21 schema now enforces heuristics_filter as a real, non-nullable, fully-shaped envelope. Without these changes the schema accepted payloads with the field missing, null, or {}, contradicting STABILITY/AGENTS saying it's required + always present. - scripts/generate_schemas.py: add `heuristics_filter` to the top-level `schema["required"]` list; tighten `properties["heuristics_filter"]` to `{"$ref": "#/$defs/HeuristicsFilter"}` (parallel to privacy_audit, reviewer_summary); set HeuristicsFilter's $defs `required` to all four fields (enabled, excluded_provenance_kinds, filtered_finding_count, filtered_by_kind). - docs/report-schema.v0.21.json regenerated. - New negative schema test in tests/test_no_heuristics.py (`test_v21_schema_requires_heuristics_filter_and_rejects_null`) parallel to the v0.12 agent_summary regression test: real payload validates, but strip-field / set-null / set-{} / drop-sub-field all raise jsonschema.ValidationError. [P2] docs/architecture.md pipeline diagram now shows `apply_no_heuristics_filter` BEFORE `core/findings.build_report` (which calls `build_release_decision`), matching the actual runtime ordering in `_run_checks_and_decide` (scan.py:641). Stamped doc no longer teaches the reverse ordering — preserves the main safety property of the flag (filtered findings cannot gate release). [P3] AGENTS.md stable-flag table for `agents-shipgate scan` now lists `--no-heuristics`, matching STABILITY.md. This is the file downstream agents ingest. Test gap: `test_keep_list_is_explicit_and_non_overlapping` was tautological (`kept = valid - excluded` then asserting the partition is exact). Replaced with explicit pin of EXPECTED_KEEP and EXPECTED_EXCLUDE sets — a new `ProvenanceKind` literal now fails the test until it's deliberately classified. llms-full.txt regenerated to pick up the AGENTS.md flag-table change. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes round-3 / round-4 review's E5: Either
--no-heuristicsflag OR formalprovenance_kindretirement carryover. We ship the flag.Finding.provenance_kindhas shipped on every report since v0.15 as required + non-nullable wire metadata, but had no consumer for four review cycles. v0.21 lands the consumer the field was always designed for: a security/GRC-friendly filter that excludes findings whose provenance iskeyword_heuristicorregex_heuristicfrom the active release-gating set.--no-heuristicsflag onagents-shipgate scan(stable in 0.x). Findings whoseprovenance_kind∈NO_HEURISTICS_EXCLUDED_PROVENANCE_KINDSare markedsuppressed=Truewithsuppression_reason="filtered by --no-heuristics"before the release decision is built. Filtered findings remain infindings[]for transparency; they no longer gate release.report.heuristics_filteraudit envelope (required, always present — parallel toprivacy_audit). Fields:enabled,excluded_provenance_kinds: list[str],filtered_finding_count,filtered_by_kind: dict[str, int]. Earnsprovenance_kind's contract weight by giving it a first-class consumer.checks.ignorekeeps the user's reason text. The audit envelope still counts the overlap.report_schema_version: "0.20"→"0.21". v0.20 moves to frozen-reference; existing v0.20 consumers ignore the new field (purely additive).docs/architecture.mdbumped to date2026-05-23, reportv0.21.Decision recorded — why ship, not retire
Round-4 review's E5 offered ship-or-retire on
provenance_kind. Retiring would have forced a deprecation cycle on a stable-contract field used by every report since v0.15. Shipping the flag earns the field's weight and serves a real audience (security/GRC reviewers triaging declared-only findings before promotion). The flag is opt-in; nothing changes for current callers.Test plan
pytest -q— 1820 passed, 4 skipped, 0 failedruff check .— cleantests/test_no_heuristics.pycovering:provenance_kind)enabled=True/FalseNO_HEURISTICS_EXCLUDED_PROVENANCE_KINDSis a realProvenanceKind; KEEP+EXCLUDE partition is exact)run_scan(no_heuristics=True)(envelope populated, blockers preserved, review_items shrink)test_architecture_doc_contract_stamp_matches_runtime) updated in lockstep🤖 Generated with Claude Code