Skip to content

feat(baseline): pin-and-diff suppression baseline (bash + PowerShell) - #1

Merged
eric-sabe merged 2 commits into
mainfrom
feat/suppression-baseline
Jul 8, 2026
Merged

feat(baseline): pin-and-diff suppression baseline (bash + PowerShell)#1
eric-sabe merged 2 commits into
mainfrom
feat/suppression-baseline

Conversation

@eric-sabe

Copy link
Copy Markdown
Owner

What & why

honey's verdict is worst-wins across bumblebee + every lens, and a lens flips to exposed on the first finding at any severity. A daily run over trusted, first-party agent skills emits 100+ static-analysis false positives (reference docs, LICENSE files, MCP examples) that drag OVERALL to EXPOSED every day and bury the real findings.

This adds a suppression baseline to acknowledge a reviewed-benign finding once — as a pin, not a mute. Each entry records a sha256 of the referenced file, so suppression and change-detection are the same mechanism:

State Result
pinned, file unchanged suppressed — dropped from the verdict, still counted
pinned, file changed 🔁 MUTATED — resurfaces loudly (the rug-pull tripwire)
pin past expiry (default 90d) resurfaces as a normal finding

Suppression is a report-layer re-rank, never a deletion (raw run records untouched), and a suppressed run never reads as a bare all-clear — the verdict always carries the tallies, e.g. OVERALL: CLEAN (12 suppressed) or OVERALL: EXPOSED — … (12 suppressed, 2 mutated).

On last night's real run this collapses the 123 skillspector false positives to "clean — all suppressed" and lets the 3 genuine npm CVEs (osv-scanner) drive the verdict alone — honestly annotated.

Design (see docs/BASELINE.plan.md)

Draws on adjacent ecosystems: content hash (vs gitleaks' fragile location-only fingerprint), ~-relative paths (portable across machines), Semgrep-style occurrence index + deterministic ordering (no fingerprint cycling), expiry (trivy/detect-secrets), and a bumblebee opt-in guard (--all excludes known-compromised matches; pinning one requires explicit --scanner bumblebee). The baseline file is committed and reviewable — that auditability is itself an anti-gaming control. The doc is explicit about what the design does not defend (sleepers, runtime-fetched includes, image payloads — things never flagged in the first place).

Multi-OS parity preserved

Mirrored in the win/ PowerShell port, verified behavior-identical (incl. byte-identical file hashes) on macOS bash 3.2 + pwsh 7.5.4.

  • bash: lib/baseline.sh (classifier), honey-baseline.sh (CLI); report.sh/daily-cycle.sh/doctor.sh integrate it; honey.baseline.json starter.
  • win: win/lib/Baseline.psm1, win/honey-baseline.ps1; report.ps1/daily-cycle.ps1/doctor.ps1 updated.

Manage it

./honey-baseline.sh status                    # dry-run tally
./honey-baseline.sh add --scanner skillspector --location references/ \
    --reason "first-party docs" --expires 90
./honey-baseline.sh list [--expired|--active]
./honey-baseline.sh remove --scanner skillspector --location references/
./honey-baseline.sh prune

Quality gates

  • report.sh output is byte-identical to before when the baseline is empty (no regression).
  • shellcheck clean (-x, 5 bash files); PSScriptAnalyzer clean under the repo's Warning-level settings (5 PS files).
  • Lifecycle verified on both platforms: suppressed / mutated / expired classification, occurrence-index disambiguation, idempotent add, verdict math, and the mutation tripwire.

Docs

New docs/BASELINE.plan.md; README (section + config var + layout); routine-prompt.md and triage-guide.md teach the CLEAN (N suppressed) / MUTATED semantics so the Claude routine won't re-alarm on pins or fold a MUTATED into "all clear"; win/TESTING.md adds a baseline test step.

🤖 Generated with Claude Code

honey's verdict is worst-wins across bumblebee + every lens, and a lens
flips to `exposed` on the first finding at any severity. A daily run over
trusted, first-party agent skills emits 100+ static-analysis false positives
(reference docs, LICENSE files, MCP examples) that drag OVERALL to EXPOSED
every day and bury real findings.

This adds a suppression baseline that lets you acknowledge a reviewed-benign
finding ONCE — but as a PIN, not a mute. Each entry records a sha256 of the
referenced file, so:
  • pinned + file unchanged  -> suppressed (dropped from the verdict, still counted)
  • pinned + file changed    -> MUTATED, resurfaces loudly (the rug-pull tripwire)
  • pin past its expiry       -> resurfaces as a normal finding
Suppression is a report-layer re-rank, never a deletion; a suppressed run
never reads as a bare all-clear (the verdict always carries the tallies,
e.g. `OVERALL: CLEAN (12 suppressed)`).

Design (docs/BASELINE.plan.md) follows adjacent ecosystems: content hash over
gitleaks' location-only fingerprint, ~-relative paths (portable), Semgrep-style
occurrence index + deterministic ordering, expiry, and bumblebee opt-in guard.
The baseline file is committed and reviewable — that auditability is itself an
anti-gaming control.

Multi-OS parity preserved: mirrored in the win/ PowerShell port.
  bash:  lib/baseline.sh, honey-baseline.sh; report.sh, daily-cycle.sh,
         doctor.sh integrate it; honey.baseline.json starter.
  win:   win/lib/Baseline.psm1, win/honey-baseline.ps1; report/daily-cycle/
         doctor updated.
Verified behavior-identical on macOS bash 3.2 + pwsh 7.5.4 (incl. byte-identical
file hashes); shellcheck clean; PSScriptAnalyzer clean; report.sh output is
byte-identical to before when the baseline is empty (no regression).

Docs updated: README (new section + config + layout), routine-prompt.md and
triage-guide.md (teach the CLEAN(N suppressed)/MUTATED semantics), win/TESTING.md.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Ubuntu's shellcheck flags SC2015 on the [ -f ] && [ -r ] || {...} hash guard
and the [ $# -gt 0 ] && shift || true dispatch line (info-level, but CI treats
it as a failure); homebrew shellcheck 0.11 does not. Rewrite both as explicit
if/then/else — behavior identical, CI green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@eric-sabe
eric-sabe merged commit ddbe95b into main Jul 8, 2026
2 checks passed
@eric-sabe
eric-sabe deleted the feat/suppression-baseline branch July 8, 2026 03:11
eric-sabe added a commit that referenced this pull request Jul 8, 2026
The merges added four lenses (smuggle, mcp, ocr, mcp-scan, garak), the
suppression baseline, and the verdict policy, leaving several sections stale or
self-contradictory. Fixes:
- intro lens table listed 4 of 8 lenses → now shows all native lenses + a
  signpost to the opt-in pair and the baseline/verdict features;
- Quickstart told readers to read manifest.json 'for the verdict' → corrected to
  report.sh's OVERALL line (manifest is bumblebee alone — the #1 mistake);
- 'How a scan cycle works' read as bumblebee-only → notes lenses run + worst-wins;
- 'offline vs external' miscounted ('first six'), dropped skillspector, and
  claimed 'nothing leaves your machine' while osv/govulncheck query public DBs →
  rewritten accurately (only mcp-scan/garak phone out, and they're opt-in);
- 'what each answers' + Layout tree + HONEY_SKILL_ROOTS text updated for the new
  lenses and lib/verdict.sh.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant