Skip to content

feat(watch): per-file acceptance granularity for high-churn vendor skills#69

Merged
askalf merged 1 commit into
masterfrom
feat/watch-per-file-acceptance
Jul 17, 2026
Merged

feat(watch): per-file acceptance granularity for high-churn vendor skills#69
askalf merged 1 commit into
masterfrom
feat/watch-per-file-acceptance

Conversation

@askalf

@askalf askalf commented Jul 17, 2026

Copy link
Copy Markdown
Owner

Closes #68 — the badge stops flapping red on upstream docs releases, without weakening what the acceptance guarantees.

What changed

Accept entries in watch-accepted.json can now opt into per-file granularity:

"agentforce-adlc:agentforce-generate": {
  "granularity": "finding-files",
  "files": { "references/safety-review-reference.md": "af86e02a…" },
  ...
}

Semantics: the files listed (the reviewed finding-bearing files) are excluded at their exact reviewed bytes, and the remainder of the skill must scan clean on the same detection pipeline. Since acceptance is only consulted when the full skill flags, a clean remainder proves every current finding lives in a reviewed, byte-identical file:

Default whole-skill-hash entries are untouched.

How

  • src/skill.mjs — skill dirs keep per-file scan pieces (path + the exact text the joined scan target saw). Scan targets, verdicts, and hashes are byte-for-byte unchanged; this only makes subset re-scans possible.
  • support/marketplace-watch.mjscovers() implements both granularities; per-file rows are marked in results.json (granularity) and WATCH.md ((per-file)).
  • support/watch-accept.mjs (new) — authors entries after a hand-review: attributes findings per file, then verifies the attribution with the exact predicate the watch uses; refuses per-file mode if findings aren't attributable to individual files (e.g. a match spanning a file boundary).

Conversion of the agentforce entries

Computed from the catalog-pinned corpus (1df4ca2f, the v0.9 merge reviewed in #67), not by hand:

  • whole-skill hashes at the pin verified byte-identical to all three accepted hashes before converting (bddf5715/2f2bf93f/8a23ff1c), so the per-file hashes cover exactly the bytes the 07-09/07-17 reviews read
  • agentforce-generate → 1 fixture file, agentforce-secure → 5, agentforce-test → 1; remainder scans clean for each

Verification

  • 146/146 tests pass; 3 new: churn-holds/lapse-on-drift/lapse-on-new-finding, no-files-map fails closed, and the authoring helper round-trips through the watch
  • real-corpus drills at the pin: baseline poisoned 0 · accepted 3, exit 0; simulated docs release (new reference doc + README edit) holds (exit 0); tampered fixture file flags (poisoned 1, exit 1)

Note

Reproducing the corpus hashes from a fresh clone on Windows needs core.autocrlf=false — the default checkout rewrites every text file to CRLF and all three skills hash as "drifted" when nothing moved. The conversion and drills above were run LF-clean and match the accepted hashes exactly.

…ills

Whole-skill-hash acceptance lapses on ANY upstream byte — agentforce-adlc
shipped twice on 07-16/17 and turned the badge red both times over docs-only
churn (#65, #67). Accept entries can now opt into
granularity: finding-files: the acceptance keys to the reviewed
finding-bearing files, and everything else in the skill must still scan clean.
A reviewed fixture drifting re-flags; a new finding anywhere flags on its own;
unrelated docs/version churn no longer lapses the review. No usable files map
fails closed.

- skill.mjs keeps per-file scan pieces so a subset can be re-scanned on
  exactly the text the full scan saw; scan targets and hashes unchanged
- support/watch-accept.mjs authors entries: attributes findings per file and
  verifies with the same remainder-must-scan-clean predicate the watch uses
- the three agentforce-adlc entries convert to per-file, computed from the
  catalog-pinned corpus at byte-identical fixture content to the reviewed
  accepts (whole-skill hashes verified equal before conversion)

Closes #68

@sprayberry-reviewer sprayberry-reviewer left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review from the Sprayberry Labs fleet code reviewer.

Verdict: No blocking issues found — approving. The core covers() predicate is correct and well-tested; a couple of minor observations below, non-blocking.

What I checked

  • gh pr diff (5 files, +201/-10) plus a local shallow clone of master/PR head to read src/skill.mjs around the scanPieces change (the diff rendered as "Binary files … differ" for that file in gh pr diff, so I diffed the two blobs directly with git show — confirmed it's the same small, readable hunk shown in the PR description, not an actual binary change).
  • CI rollup: gh pr checks 69 — CodeQL, analyze, all 6 test matrix jobs (ubuntu/macos/windows × node 20/22), and "verify pinned skills" all pass.
  • Traced covers() in support/marketplace-watch.mjs:32-40 against loadSkillDir() in src/skill.mjs:78-115:
    • hashOf is built from skill.files (every file's content-hash, via the existing entries array), and rest filters skill.scanPieces (text-only pieces) down to those whose current hash differs from the reviewed entry's recorded hash — i.e. exactly the not-yet-reviewed-or-drifted files.
    • When rest is empty (every scanned piece matches a reviewed byte-identical file), it returns true (accepted) without a re-scan — correct, since the original full-skill scan already ran findings against the whole joined blob and every constituent piece is reviewed.
    • When rest is non-empty, it re-scans just those pieces with kind: 'skill' so scanSkill() still filters advisory-severity findings — the same filtering the original full scan applied — and requires a clean verdict. A file that isn't in the files map at all (reviewed[p.path] is undefined) always mismatches its hashOf value and is correctly folded into rest.
    • An entry with no usable files map (a.files missing/non-object) sets reviewed = {}, so every current scanPiece counts as unreviewed → rest = the whole skill → fails closed, matching the PR's stated guarantee and covered by the 'an entry with no files map fails closed' test.
  • test/marketplace-watch.test.mjs new cases exercise: baseline per-file accept + (per-file) marker in WATCH.md, unrelated churn holding, a new finding in a new file lapsing, the reviewed file itself drifting lapsing, the no-files-map fail-closed case, and the watch-accept.mjs --files authoring round-trip (including its own attribution-refusal path when a finding can't be pinned to one file). That maps directly onto the semantics claimed in the PR body.
  • support/watch-accepted.json: all new/changed hash/files values are well-formed 64-char hex (sha256), consistent with the existing whole-skill entries.

Minor (non-blocking)

  • support/watch-accept.mjs's --files attribution check (bearing/remainder scan) re-scans each finding-bearing candidate individually plus once more for the remainder — O(n) re-scans over a skill's file count. Fine at today's skill sizes (this is an offline authoring helper run by a human, not the hot watch path), just flagging in case very large skills make it noticeably slow later.

What's good

  • The rationale comments in covers() and loadSkillDir() clearly state the invariant being preserved (scan targets/verdicts/hashes byte-for-byte unchanged for existing whole-skill entries) and why a clean remainder proves per-file coverage — easy to audit against the code.
  • watch-accept.mjs refusing to emit a per-file entry when a finding isn't attributable to a single file (crosses a file boundary) closes the obvious escape hatch of this feature before it ships.
  • Test coverage is thorough and specifically targets the drift/churn/new-finding/no-files-map edge cases the feature is supposed to get right.

@askalf
askalf merged commit 24312dd into master Jul 17, 2026
9 checks passed
@askalf
askalf deleted the feat/watch-per-file-acceptance branch July 17, 2026 22:06
askalf added a commit that referenced this pull request Jul 17, 2026
)

The weekly marketplace watch now publishes directory-manifest.json on the
watch branch: name -> skill hash for everything it scanned, plus the
currently-flagged names. New CLI command `check-manifest <file>` compares
every INSTALLED marketplace plugin skill against it:

- bytes differ from what the watch scanned -> drifted (exit 1)
- watch-flagged -> fails even byte-identical (match is not endorsement)
- unknown to the manifest -> unlisted (reported, never fatal)
- manifest skills not installed here are ignored

Offline like everything else: you fetch the manifest, truecopy reads it.
Prototype-safe manifest lookups (Object.hasOwn; a skill named after a
prototype member cannot read an inherited "expected hash").

Verified live on this machine: 29 installed official-directory plugin
skills vs a manifest generated from the directory at tip -> all match,
exit 0. Install-vs-corpus hash parity proven before building (installed
marketplace trees hash byte-identical to the directory clone).

0.9.0: this plus the per-file acceptance internals (#69) ship to npm on
merge via the usual auto-release path.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

watch: per-file acceptance granularity for high-churn vendor skills

2 participants