feat(watch): per-file acceptance granularity for high-churn vendor skills#69
Merged
Conversation
…ills Whole-skill-hash acceptance lapses on ANY upstream byte — agentforce-adlc shipped twice on 07-16/17 and turned the badge red both times over docs-only churn (#65, #67). Accept entries can now opt into granularity: finding-files: the acceptance keys to the reviewed finding-bearing files, and everything else in the skill must still scan clean. A reviewed fixture drifting re-flags; a new finding anywhere flags on its own; unrelated docs/version churn no longer lapses the review. No usable files map fails closed. - skill.mjs keeps per-file scan pieces so a subset can be re-scanned on exactly the text the full scan saw; scan targets and hashes unchanged - support/watch-accept.mjs authors entries: attributes findings per file and verifies with the same remainder-must-scan-clean predicate the watch uses - the three agentforce-adlc entries convert to per-file, computed from the catalog-pinned corpus at byte-identical fixture content to the reviewed accepts (whole-skill hashes verified equal before conversion) Closes #68
sprayberry-reviewer
approved these changes
Jul 17, 2026
sprayberry-reviewer
left a comment
Collaborator
There was a problem hiding this comment.
Automated review from the Sprayberry Labs fleet code reviewer.
Verdict: No blocking issues found — approving. The core covers() predicate is correct and well-tested; a couple of minor observations below, non-blocking.
What I checked
gh pr diff(5 files, +201/-10) plus a local shallow clone ofmaster/PR head to readsrc/skill.mjsaround thescanPieceschange (the diff rendered as "Binary files … differ" for that file ingh pr diff, so I diffed the two blobs directly withgit show— confirmed it's the same small, readable hunk shown in the PR description, not an actual binary change).- CI rollup:
gh pr checks 69— CodeQL, analyze, all 6 test matrix jobs (ubuntu/macos/windows × node 20/22), and "verify pinned skills" allpass. - Traced
covers()insupport/marketplace-watch.mjs:32-40againstloadSkillDir()insrc/skill.mjs:78-115:hashOfis built fromskill.files(every file's content-hash, via the existingentriesarray), andrestfiltersskill.scanPieces(text-only pieces) down to those whose current hash differs from the reviewed entry's recorded hash — i.e. exactly the not-yet-reviewed-or-drifted files.- When
restis empty (every scanned piece matches a reviewed byte-identical file), it returnstrue(accepted) without a re-scan — correct, since the original full-skill scan already ran findings against the whole joined blob and every constituent piece is reviewed. - When
restis non-empty, it re-scans just those pieces withkind: 'skill'soscanSkill()still filtersadvisory-severity findings — the same filtering the original full scan applied — and requires a clean verdict. A file that isn't in thefilesmap at all (reviewed[p.path]isundefined) always mismatches itshashOfvalue and is correctly folded intorest. - An entry with no usable
filesmap (a.filesmissing/non-object) setsreviewed = {}, so every current scanPiece counts as unreviewed →rest= the whole skill → fails closed, matching the PR's stated guarantee and covered by the'an entry with no files map fails closed'test.
test/marketplace-watch.test.mjsnew cases exercise: baseline per-file accept +(per-file)marker in WATCH.md, unrelated churn holding, a new finding in a new file lapsing, the reviewed file itself drifting lapsing, the no-files-map fail-closed case, and thewatch-accept.mjs --filesauthoring round-trip (including its own attribution-refusal path when a finding can't be pinned to one file). That maps directly onto the semantics claimed in the PR body.support/watch-accepted.json: all new/changedhash/filesvalues are well-formed 64-char hex (sha256), consistent with the existing whole-skill entries.
Minor (non-blocking)
support/watch-accept.mjs's--filesattribution check (bearing/remainder scan) re-scans each finding-bearing candidate individually plus once more for the remainder — O(n) re-scans over a skill's file count. Fine at today's skill sizes (this is an offline authoring helper run by a human, not the hot watch path), just flagging in case very large skills make it noticeably slow later.
What's good
- The rationale comments in
covers()andloadSkillDir()clearly state the invariant being preserved (scan targets/verdicts/hashes byte-for-byte unchanged for existing whole-skill entries) and why a clean remainder proves per-file coverage — easy to audit against the code. watch-accept.mjsrefusing to emit a per-file entry when a finding isn't attributable to a single file (crosses a file boundary) closes the obvious escape hatch of this feature before it ships.- Test coverage is thorough and specifically targets the drift/churn/new-finding/no-files-map edge cases the feature is supposed to get right.
This was referenced Jul 17, 2026
askalf
added a commit
that referenced
this pull request
Jul 17, 2026
) The weekly marketplace watch now publishes directory-manifest.json on the watch branch: name -> skill hash for everything it scanned, plus the currently-flagged names. New CLI command `check-manifest <file>` compares every INSTALLED marketplace plugin skill against it: - bytes differ from what the watch scanned -> drifted (exit 1) - watch-flagged -> fails even byte-identical (match is not endorsement) - unknown to the manifest -> unlisted (reported, never fatal) - manifest skills not installed here are ignored Offline like everything else: you fetch the manifest, truecopy reads it. Prototype-safe manifest lookups (Object.hasOwn; a skill named after a prototype member cannot read an inherited "expected hash"). Verified live on this machine: 29 installed official-directory plugin skills vs a manifest generated from the directory at tip -> all match, exit 0. Install-vs-corpus hash parity proven before building (installed marketplace trees hash byte-identical to the directory clone). 0.9.0: this plus the per-file acceptance internals (#69) ship to npm on merge via the usual auto-release path.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #68 — the badge stops flapping red on upstream docs releases, without weakening what the acceptance guarantees.
What changed
Accept entries in
watch-accepted.jsoncan now opt into per-file granularity:Semantics: the files listed (the reviewed finding-bearing files) are excluded at their exact reviewed bytes, and the remainder of the skill must scan clean on the same detection pipeline. Since acceptance is only consulted when the full skill flags, a clean remainder proves every current finding lives in a reviewed, byte-identical file:
filesmap → excludes nothing → fails closedDefault whole-skill-hash entries are untouched.
How
src/skill.mjs— skill dirs keep per-file scan pieces (path + the exact text the joined scan target saw). Scan targets, verdicts, and hashes are byte-for-byte unchanged; this only makes subset re-scans possible.support/marketplace-watch.mjs—covers()implements both granularities; per-file rows are marked inresults.json(granularity) and WATCH.md ((per-file)).support/watch-accept.mjs(new) — authors entries after a hand-review: attributes findings per file, then verifies the attribution with the exact predicate the watch uses; refuses per-file mode if findings aren't attributable to individual files (e.g. a match spanning a file boundary).Conversion of the agentforce entries
Computed from the catalog-pinned corpus (
1df4ca2f, the v0.9 merge reviewed in #67), not by hand:bddf5715/2f2bf93f/8a23ff1c), so the per-file hashes cover exactly the bytes the 07-09/07-17 reviews readagentforce-generate→ 1 fixture file,agentforce-secure→ 5,agentforce-test→ 1; remainder scans clean for eachVerification
poisoned 0 · accepted 3, exit 0; simulated docs release (new reference doc + README edit) holds (exit 0); tampered fixture file flags (poisoned 1, exit 1)Note
Reproducing the corpus hashes from a fresh clone on Windows needs
core.autocrlf=false— the default checkout rewrites every text file to CRLF and all three skills hash as "drifted" when nothing moved. The conversion and drills above were run LF-clean and match the accepted hashes exactly.