feat(baseline): pin-and-diff suppression baseline (bash + PowerShell) - #1
Merged
Conversation
honey's verdict is worst-wins across bumblebee + every lens, and a lens
flips to `exposed` on the first finding at any severity. A daily run over
trusted, first-party agent skills emits 100+ static-analysis false positives
(reference docs, LICENSE files, MCP examples) that drag OVERALL to EXPOSED
every day and bury real findings.
This adds a suppression baseline that lets you acknowledge a reviewed-benign
finding ONCE — but as a PIN, not a mute. Each entry records a sha256 of the
referenced file, so:
• pinned + file unchanged -> suppressed (dropped from the verdict, still counted)
• pinned + file changed -> MUTATED, resurfaces loudly (the rug-pull tripwire)
• pin past its expiry -> resurfaces as a normal finding
Suppression is a report-layer re-rank, never a deletion; a suppressed run
never reads as a bare all-clear (the verdict always carries the tallies,
e.g. `OVERALL: CLEAN (12 suppressed)`).
Design (docs/BASELINE.plan.md) follows adjacent ecosystems: content hash over
gitleaks' location-only fingerprint, ~-relative paths (portable), Semgrep-style
occurrence index + deterministic ordering, expiry, and bumblebee opt-in guard.
The baseline file is committed and reviewable — that auditability is itself an
anti-gaming control.
Multi-OS parity preserved: mirrored in the win/ PowerShell port.
bash: lib/baseline.sh, honey-baseline.sh; report.sh, daily-cycle.sh,
doctor.sh integrate it; honey.baseline.json starter.
win: win/lib/Baseline.psm1, win/honey-baseline.ps1; report/daily-cycle/
doctor updated.
Verified behavior-identical on macOS bash 3.2 + pwsh 7.5.4 (incl. byte-identical
file hashes); shellcheck clean; PSScriptAnalyzer clean; report.sh output is
byte-identical to before when the baseline is empty (no regression).
Docs updated: README (new section + config + layout), routine-prompt.md and
triage-guide.md (teach the CLEAN(N suppressed)/MUTATED semantics), win/TESTING.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This was referenced Jul 7, 2026
Ubuntu's shellcheck flags SC2015 on the [ -f ] && [ -r ] || {...} hash guard
and the [ $# -gt 0 ] && shift || true dispatch line (info-level, but CI treats
it as a failure); homebrew shellcheck 0.11 does not. Rewrite both as explicit
if/then/else — behavior identical, CI green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
eric-sabe
added a commit
that referenced
this pull request
Jul 8, 2026
The merges added four lenses (smuggle, mcp, ocr, mcp-scan, garak), the suppression baseline, and the verdict policy, leaving several sections stale or self-contradictory. Fixes: - intro lens table listed 4 of 8 lenses → now shows all native lenses + a signpost to the opt-in pair and the baseline/verdict features; - Quickstart told readers to read manifest.json 'for the verdict' → corrected to report.sh's OVERALL line (manifest is bumblebee alone — the #1 mistake); - 'How a scan cycle works' read as bumblebee-only → notes lenses run + worst-wins; - 'offline vs external' miscounted ('first six'), dropped skillspector, and claimed 'nothing leaves your machine' while osv/govulncheck query public DBs → rewritten accurately (only mcp-scan/garak phone out, and they're opt-in); - 'what each answers' + Layout tree + HONEY_SKILL_ROOTS text updated for the new lenses and lib/verdict.sh. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
honey's verdict is worst-wins across bumblebee + every lens, and a lens flips to
exposedon the first finding at any severity. A daily run over trusted, first-party agent skills emits 100+ static-analysis false positives (reference docs,LICENSEfiles, MCP examples) that dragOVERALLtoEXPOSEDevery day and bury the real findings.This adds a suppression baseline to acknowledge a reviewed-benign finding once — as a pin, not a mute. Each entry records a sha256 of the referenced file, so suppression and change-detection are the same mechanism:
Suppression is a report-layer re-rank, never a deletion (raw run records untouched), and a suppressed run never reads as a bare all-clear — the verdict always carries the tallies, e.g.
OVERALL: CLEAN (12 suppressed)orOVERALL: EXPOSED — … (12 suppressed, 2 mutated).On last night's real run this collapses the 123 skillspector false positives to "clean — all suppressed" and lets the 3 genuine npm CVEs (osv-scanner) drive the verdict alone — honestly annotated.
Design (see
docs/BASELINE.plan.md)Draws on adjacent ecosystems: content hash (vs gitleaks' fragile location-only fingerprint),
~-relative paths (portable across machines), Semgrep-style occurrence index + deterministic ordering (no fingerprint cycling), expiry (trivy/detect-secrets), and a bumblebee opt-in guard (--allexcludes known-compromised matches; pinning one requires explicit--scanner bumblebee). The baseline file is committed and reviewable — that auditability is itself an anti-gaming control. The doc is explicit about what the design does not defend (sleepers, runtime-fetched includes, image payloads — things never flagged in the first place).Multi-OS parity preserved
Mirrored in the
win/PowerShell port, verified behavior-identical (incl. byte-identical file hashes) on macOS bash 3.2 + pwsh 7.5.4.lib/baseline.sh(classifier),honey-baseline.sh(CLI);report.sh/daily-cycle.sh/doctor.shintegrate it;honey.baseline.jsonstarter.win/lib/Baseline.psm1,win/honey-baseline.ps1;report.ps1/daily-cycle.ps1/doctor.ps1updated.Manage it
Quality gates
report.shoutput is byte-identical to before when the baseline is empty (no regression).-x, 5 bash files); PSScriptAnalyzer clean under the repo's Warning-level settings (5 PS files).add, verdict math, and the mutation tripwire.Docs
New
docs/BASELINE.plan.md; README (section + config var + layout);routine-prompt.mdandtriage-guide.mdteach theCLEAN (N suppressed)/MUTATEDsemantics so the Claude routine won't re-alarm on pins or fold a MUTATED into "all clear";win/TESTING.mdadds a baseline test step.🤖 Generated with Claude Code