Turn the verification template into the Agent Skill Verifier CLI product (v0.1.0)

.github/workflows/ci.yml

@@ -0,0 +1,54 @@
+# Continuous integration: lint, typecheck, tests, build, and a packaged-CLI
+# smoke test. Runs fully offline (no provider credentials, no model calls).
+name: ci
+
+on:
+  pull_request:
+  push:
+    branches: [main]
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+concurrency:
+  group: ci-${{ github.ref }}
+  cancel-in-progress: ${{ github.ref != 'refs/heads/main' }}
+
+env:
+  # Pinned Node version used by all product builds (see docs/release-process.md).
+  NODE_VERSION: "22.13.1"
+
+jobs:
+  quality:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Setup Node.js (pinned)
+        uses: actions/setup-node@v4
+        with:
+          node-version: ${{ env.NODE_VERSION }}
+          cache: npm
+
+      - name: Install dependencies (locked)
+        run: npm ci
+
+      - name: Lint
+        run: npm run lint
+
+      - name: Typecheck
+        run: npm run typecheck
+
+      - name: Test
+        run: npm test
+
+      - name: Build (tsc)
+        run: npm run build
+
+      - name: Build portable CLI bundle
+        run: npm run build:cli
+
+      - name: Smoke-test the portable CLI (outside the repository)
+        run: node scripts/smoke-test.mjs --target dist/cli/agent-skill-verifier.cjs

.github/workflows/release.yml

@@ -0,0 +1,230 @@
+# Tag-driven release: builds standalone executables on native runners for
+# every supported platform, validates the archives and checksums, then
+# publishes a GitHub Release draft-first.
+#
+# Dry run: trigger manually (workflow_dispatch). A dispatch run builds and
+# validates everything and uploads workflow artifacts, but NEVER creates a
+# tag or a Release (the release job only runs for tag pushes).
+name: release
+
+on:
+  push:
+    tags:
+      - "v*.*.*"
+  workflow_dispatch:
+
+# Least privilege by default; only the `release` job below gets write access.
+permissions:
+  contents: read
+
+concurrency:
+  group: release-${{ github.ref }}
+  cancel-in-progress: false
+
+env:
+  # Pinned Node version for reproducible release builds. The Node SEA procedure
+  # used by scripts/build-standalone.mjs follows this version's documentation.
+  NODE_VERSION: "22.13.1"
+
+jobs:
+  verify:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Setup Node.js (pinned)
+        uses: actions/setup-node@v4
+        with:
+          node-version: ${{ env.NODE_VERSION }}
+          cache: npm
+
+      - name: Validate tag format and package-version match
+        if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags/')
+        run: node scripts/release-check.mjs --tag-only --tag "$GITHUB_REF_NAME"
+
+      - name: Install dependencies (locked)
+        run: npm ci
+
+      - name: Lint
+        run: npm run lint
+
+      - name: Typecheck
+        run: npm run typecheck
+
+      - name: Test
+        run: npm test
+
+      - name: Build portable CLI bundle
+        run: npm run build:cli
+
+      - name: Smoke-test the portable CLI
+        run: node scripts/smoke-test.mjs --target dist/cli/agent-skill-verifier.cjs
+
+  build-platform:
+    needs: verify
+    strategy:
+      matrix:
+        include:
+          - os: windows-latest
+            target: windows-x64
+          - os: ubuntu-latest
+            target: linux-x64
+          - os: macos-15-intel
+            target: macos-x64
+          - os: macos-14
+            target: macos-arm64
+    runs-on: ${{ matrix.os }}
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Setup Node.js (pinned)
+        uses: actions/setup-node@v4
+        with:
+          node-version: ${{ env.NODE_VERSION }}
+          cache: npm
+
+      - name: Install dependencies (locked)
+        run: npm ci
+
+      - name: Build and package the standalone executable (Node SEA)
+        run: node scripts/package-release.mjs --standalone-only
+
+      - name: Validate the archive and manifest
+        run: node scripts/release-check.mjs --require-commit-match
+
+      - name: Smoke-test the standalone executable (outside the repository)
+        run: node scripts/smoke-test.mjs
+
+      - name: Upload platform archive
+        uses: actions/upload-artifact@v4
+        with:
+          name: asv-${{ matrix.target }}
+          path: |
+            dist/release/*.zip
+            dist/release/*.tar.gz
+          if-no-files-found: error
+          retention-days: 7
+
+  build-portable:
+    needs: verify
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Setup Node.js (pinned)
+        uses: actions/setup-node@v4
+        with:
+          node-version: ${{ env.NODE_VERSION }}
+          cache: npm
+
+      - name: Install dependencies (locked)
+        run: npm ci
+
+      - name: Package the portable Node bundle
+        run: node scripts/package-release.mjs --portable-only
+
+      - name: Validate the archive and manifest
+        run: node scripts/release-check.mjs --require-commit-match
+
+      - name: Smoke-test the portable bundle (outside the repository)
+        run: node scripts/smoke-test.mjs --target dist/cli/agent-skill-verifier.cjs
+
+      - name: Upload portable archive
+        uses: actions/upload-artifact@v4
+        with:
+          name: asv-node
+          path: dist/release/*.zip
+          if-no-files-found: error
+          retention-days: 7
+
+  release:
+    # Publication happens ONLY for pushed version tags — never for
+    # workflow_dispatch dry runs and never from pull requests.
+    if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags/v')
+    needs: [verify, build-platform, build-portable]
+    runs-on: ubuntu-latest
+    permissions:
+      contents: write
+    env:
+      GH_TOKEN: ${{ github.token }}
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Setup Node.js (pinned)
+        uses: actions/setup-node@v4
+        with:
+          node-version: ${{ env.NODE_VERSION }}
+
+      - name: Download all build artifacts
+        uses: actions/download-artifact@v4
+        with:
+          pattern: asv-*
+          path: dist/release
+          merge-multiple: true
+
+      - name: Inspect downloaded assets
+        run: ls -l dist/release
+
+      - name: Validate tag, archives, manifests, and expected asset set
+        run: >
+          node scripts/release-check.mjs
+          --tag "$GITHUB_REF_NAME"
+          --require-commit-match
+          --require-assets windows-x64,linux-x64,macos-x64,macos-arm64,node
+
+      - name: Generate checksums
+        run: node scripts/generate-checksums.mjs dist/release
+
+      - name: Verify checksums
+        run: node scripts/generate-checksums.mjs dist/release --verify
+
+      - name: Generate release notes
+        run: node scripts/release-notes.mjs > dist/release-notes.md
+
+      - name: Ensure the release does not already exist
+        run: |
+          if gh release view "$GITHUB_REF_NAME" --repo "$GITHUB_REPOSITORY" >/dev/null 2>&1; then
+            echo "::error::Release $GITHUB_REF_NAME already exists — refusing to overwrite."
+            exit 1
+          fi
+
+      - name: Create draft release
+        run: |
+          VERSION="${GITHUB_REF_NAME#v}"
+          gh release create "$GITHUB_REF_NAME" \
+            --repo "$GITHUB_REPOSITORY" \
+            --draft \
+            --title "Agent Skill Verifier v${VERSION}" \
+            --notes-file dist/release-notes.md \
+            --verify-tag
+
+      - name: Upload assets to the draft (no clobbering)
+        run: |
+          gh release upload "$GITHUB_REF_NAME" \
+            --repo "$GITHUB_REPOSITORY" \
+            dist/release/*.zip \
+            dist/release/*.tar.gz \
+            dist/release/SHA256SUMS.txt \
+            dist/release/SHA256SUMS.json
+
+      - name: Confirm every expected asset was uploaded
+        run: |
+          gh release view "$GITHUB_REF_NAME" --repo "$GITHUB_REPOSITORY" \
+            --json assets --jq '.assets[].name' | sort > uploaded.txt
+          ls dist/release | grep -E '\.(zip|tar\.gz)$|^SHA256SUMS' | sort > expected.txt
+          echo "Expected:"; cat expected.txt
+          echo "Uploaded:"; cat uploaded.txt
+          diff expected.txt uploaded.txt
+
+      - name: Publish the release
+        run: |
+          gh release edit "$GITHUB_REF_NAME" \
+            --repo "$GITHUB_REPOSITORY" \
+            --draft=false --latest
+
+      - name: Show the published release
+        run: gh release view "$GITHUB_REF_NAME" --repo "$GITHUB_REPOSITORY"

.github/workflows/skill-eval.yml

@@ -11,6 +11,9 @@ on:
   pull_request:
   workflow_dispatch:
 
+permissions:
+  contents: read
+
 jobs:
   verify:
     runs-on: ubuntu-latest

.gitignore

@@ -8,6 +8,10 @@ dist/
 reports/*
 !reports/.gitkeep
 
+# CLI verification output and scratch space
+.agent-skill-verification/
+tmp/
+
 # Logs
 *.log
 npm-debug.log*

CHANGELOG.md

@@ -0,0 +1,72 @@
+# Changelog
+
+All notable changes to Agent Skill Verifier are documented here. The format is
+based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and the
+project follows semantic versioning.
+
+## 0.1.0 — 2026-07-17
+
+First downloadable release: the repository's verification harness is now a
+real cross-platform CLI product, **agent-skill-verifier**.
+
+### Added
+
+- **`verify` command** — runs every evaluation case N times against a model
+  adapter, applies the four structural validators (schema, citations,
+  unsupported claims, tool calls), enforces the quality gate
+  (threshold + per-case floors + optional flaky-rate ceiling), and writes the
+  full report bundle. Non-interactive by design; safe for CI.
+- **`validate` command** — static checks for skill contracts, evaluation
+  cases (duplicate ids, expected statuses, citation files), adapter names,
+  numeric ranges, and output-path safety. Executes no runs.
+- **`replay` command** — schema-validated inspection of stored replay
+  artifacts (input, output, tool trace, validation verdict). No model call;
+  artifacts are never modified.
+- **`report` command** — converts the canonical `summary.json` into
+  terminal, JSON, JUnit XML, or self-contained HTML without rerunning.
+- **Canonical result schema 1.0.0** (`summary.json`) — versioned,
+  machine-readable, schema-validated before writing; unsupported metrics are
+  `null`, never fabricated.
+- **Report bundle** — `summary.json`, `junit.xml`, `report.html` (fully
+  self-contained), `events.jsonl`, `metrics.json`, and one replay artifact
+  per run under `replays/`.
+- **Stable exit codes** — 0 passed · 1 gate failed · 2 invalid input ·
+  3 adapter unavailable · 4 runtime failure · 5 timeout/cancelled ·
+  6 report/artifact failure.
+- **Project configuration** — `skill-verification.yaml|yml|json` with
+  documented precedence (CLI flags → config file → defaults). Evaluation
+  cases may be YAML or JSON.
+- **Standalone executables** for Windows x64, Linux x64, macOS x64, and
+  macOS arm64 built with the official Node.js Single Executable Application
+  mechanism (pinned Node 22.13.1), plus a portable Node bundle. No Node
+  installation or `node_modules` required for the standalone binaries.
+- **Release engineering** — per-archive `release-manifest.json` with
+  per-file SHA-256, `SHA256SUMS.txt`/`SHA256SUMS.json`, archive content
+  validation, packaged-binary smoke tests, and a draft-first tag-driven
+  release workflow that verifies every expected asset before publishing.
+- **CI** — lint, typecheck, 116 offline tests, build, and packaged-CLI smoke
+  test on every push/PR with `contents: read` permissions.
+
+### Changed
+
+- Path resolution is workspace-rooted (the CLI pins it to the current working
+  directory), making the harness usable outside this repository.
+- The repository README now documents the CLI product; the original
+  template/tutorial documentation moved to `docs/reference-implementation.md`
+  unchanged in substance.
+
+### Security
+
+- Output confined to the working directory; sanitized replay file names;
+  HTML/XML escaping for hostile skill content; secrets only via environment
+  variables; release workflow uses least-privilege tokens and never
+  overwrites releases or assets. See `docs/threat-model.md`.
+
+### Known limitations
+
+- Binaries are **not code-signed**; OS trust prompts are expected on first
+  run. Checksums verify integrity, not publisher identity.
+- `replay` inspects recorded artifacts; deterministic model re-execution is
+  not claimed.
+- Live-adapter (`llm`) verification is non-deterministic by nature; mock
+  latency/token/cost metrics are labeled estimates.