diff --git a/.claude/CLAUDE.md b/.claude/CLAUDE.md index 15529f75..a69ef003 100644 --- a/.claude/CLAUDE.md +++ b/.claude/CLAUDE.md @@ -18,7 +18,6 @@ This repo IS `jbaruch/coding-policy`. The rule files below are the source-of-tru @../rules/skill-authoring.md @../rules/script-delegation.md @../rules/script-as-black-box.md -@../rules/plugin-evals.md @../rules/author-model-declaration.md @../rules/stateful-artifacts.md @../rules/agent-worktree-isolation.md diff --git a/.github/actions/skill-review/review-skills.sh b/.github/actions/skill-review/review-skills.sh index 5d2d8bc6..6e71b736 100644 --- a/.github/actions/skill-review/review-skills.sh +++ b/.github/actions/skill-review/review-skills.sh @@ -134,7 +134,7 @@ identify_skills() { # Immediate subdir basenames, sorted. `sed 's|.*/||'` instead of GNU # `find -printf '%f'` keeps discovery portable (BSD find on macOS has no # -printf) so this path is testable off the CI runner. Skill dir names - # are kebab-case with no newlines (rules/plugin-evals.md Naming). + # are kebab-case with no newlines. find "$SKILLS_DIR" -mindepth 1 -maxdepth 1 -type d | sed 's|.*/||' | sort return 0 fi diff --git a/.tessl-plugin/plugin.json b/.tessl-plugin/plugin.json index 150c851f..b59efcdc 100644 --- a/.tessl-plugin/plugin.json +++ b/.tessl-plugin/plugin.json @@ -5,8 +5,6 @@ "private": false, "skills": [ "skills/release", - "skills/eval-authoring", - "skills/eval-curation", "skills/install-reviewer", "skills/adopt-fork-pr", "skills/migrate-to-plugin" @@ -28,7 +26,6 @@ "rules/skill-authoring.md", "rules/script-delegation.md", "rules/script-as-black-box.md", - "rules/plugin-evals.md", "rules/author-model-declaration.md", "rules/stateful-artifacts.md", "rules/agent-worktree-isolation.md", diff --git a/CHANGELOG.md b/CHANGELOG.md index c77a164b..073b5c2b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,7 @@ ### Rules +- **Evals removed from policy and plugin** — deleted `rules/plugin-evals.md`, the `context-artifacts` Mandatory Evals section, the `eval-authoring` / `eval-curation` skills, and the `evals/` scenario suite, reconciling every cross-reference (23→22 rules). The tessl publish pipeline runs evals unconditionally inside `tesslio/patch-version-publish`, which fails and blocks the registry when the org is out of credits — a trigger #188's credit-outage tolerance cannot reach. Full inventory and motivation: PR #191. - **`context-artifacts`: Credit-Outage Review Carve-Out** — a tessl out-of-credits (403) billing outage can now satisfy the Mandatory Review gate under an opt-in, fail-safe skip: only the credit-phrase-plus-403 signature skips the affected skill (flagged in the run summary and the `unreviewed-skills` output), every other review failure still hard-fails, and the gate self-heals when credits return. Mirrors the `ci-safety` Publish-Pipeline carve-out shape. Distinct from the forbidden review-step bypass — that dodges a verdict, this tolerates an outage that produced none. Motivation: issue #188. - **`error-handling`: new Shell Error Handling section** — the rule was entirely silent on shell (no `set -euo pipefail`, no suppression ban), so consumers installing the plugin inherited none of the discipline the repo itself practises in all 18 of its non-test scripts. Encodes: mandatory `set -euo pipefail`; no `|| true` / `|| :` / bare `2>/dev/null`; explicit `if`/`case` on exit codes for commands that can legitimately fail; the distinction between an expected non-result and a tool failure (`grep` exits 1 on no-match, 2 on error — `|| true` collapses both); that silencing a *diagnostic* while explicitly handling the *failure* is not suppression; and that "fail visibly" permits a stderr warning rather than requiring exit non-zero. Full motivation: PR #184. diff --git a/README.md b/README.md index edb61f31..3ae13d84 100644 --- a/README.md +++ b/README.md @@ -2,17 +2,15 @@ [![tessl](https://img.shields.io/endpoint?url=https%3A%2F%2Fapi.tessl.io%2Fv1%2Fbadges%2Fjbaruch%2Fcoding-policy)](https://tessl.io/registry/jbaruch/coding-policy) -Coding policy plugin for Baruch's AI agents. Language-agnostic code quality rules plus Tessl-specific plugin authoring standards — covering commits, testing, error handling, skill structure, script delegation, and eval quality. +Coding policy plugin for Baruch's AI agents. Language-agnostic code quality rules plus Tessl-specific plugin authoring standards — covering commits, testing, error handling, skill structure, and script delegation. ## What's New -- 23 rules — 16 always-on, 7 conditional (scoped via `applyTo:` to the files where the rule's prescriptions actually fire). Breakdown: 10 covering code quality, 8 covering plugin authoring, 1 covering author-model declaration, 1 covering concurrency, 1 covering review discipline, 1 covering reviewer-feedback reading, 1 covering external-repo action scope +- 22 rules — 16 always-on, 6 conditional (scoped via `applyTo:` to the files where the rule's prescriptions actually fire). Breakdown: 10 covering code quality, 7 covering plugin authoring, 1 covering author-model declaration, 1 covering concurrency, 1 covering review discipline, 1 covering reviewer-feedback reading, 1 covering external-repo action scope - `release` skill — structured PR + merge workflow with Copilot review and paired-reviewer cross-family enforcement -- `eval-authoring` skill — generate, review, and curate eval scenarios with score-driven iteration - `install-reviewer` skill — scaffold the paired gh-aw PR review workflows (OpenAI + Anthropic) into a consumer repo - `adopt-fork-pr` skill — bring a fork PR's branch into the base repo as a same-repo PR the fork-guarded reviewer can run on - 0.3.0 added `install-reviewer` upgrade mode (`--override`) — refreshes scaffolded reviewer files in place instead of requiring a manual `git rm`-and-rerun -- 0.2.0 lifted with-context attainment from 93 to 98 (3× avg, lift +17 → +22) by tuning skill prose against eval log analysis - Language-agnostic: works with any stack, no Python/JS assumptions See [CHANGELOG.md](CHANGELOG.md) for full version history. @@ -43,7 +41,6 @@ tessl install jbaruch/coding-policy | Authoring | [skill-authoring](rules/skill-authoring.md) | SKILL.md structure, step numbering, typed calls, plugin.json reference | | Authoring | [script-delegation](rules/script-delegation.md) | Deterministic → script, reasoning → LLM, the regex trap | | Authoring | [script-as-black-box](rules/script-as-black-box.md) | Skills reference the script's contract (inputs/outputs/exit codes), not its internal logic — thresholds and predicates live in the script | -| Authoring | [plugin-evals](rules/plugin-evals.md) | Lift-gated, capped scenarios; no bleeding/leaking; persistent | | Authoring | [stateful-artifacts](rules/stateful-artifacts.md) | Cross-invocation state: schema, owner skill, schema_version, hints-not-authority, migration | | Review | [author-model-declaration](rules/author-model-declaration.md) | PRs declare author model; paired reviewers pick the cross-family one | | Review | [reviewer-feedback-reading](rules/reviewer-feedback-reading.md) | A review's state classifies merge-gating, not whether its body must be read; read every reviewer's body before declaring merge-ready, COMMENTED-with-zero-inline included | @@ -56,8 +53,6 @@ tessl install jbaruch/coding-policy | Skill | Description | |-------|-------------| | [release](skills/release/SKILL.md) | PR creation, Copilot review, merge + cleanup workflow | -| [eval-authoring](skills/eval-authoring/SKILL.md) | Generate, review, iterate on eval scenarios with score-driven feedback | -| [eval-curation](skills/eval-curation/SKILL.md) | Prune an existing eval suite — run, compute per-scenario lift, diagnose weak scenarios, retire / fix / rewrite, verify the curated suite still pulls weight | | [install-reviewer](skills/install-reviewer/SKILL.md) | Scaffold the paired gh-aw PR review workflows (OpenAI + Anthropic) into a consumer repo — reviews every PR against the latest published `jbaruch/coding-policy` with cross-family enforcement. Documents the reviewer CI secrets in a merged `.env.example`. Supports `--override` for in-place upgrades. | | [adopt-fork-pr](skills/adopt-fork-pr/SKILL.md) | Classify a PR by number. Same-repo PRs pass through to the reviewer; fork PRs (skipped by the reviewer's fork-guard) get adopted into the base repo as a same-repo PR, preserving the contributor's commits. | | [migrate-to-plugin](skills/migrate-to-plugin/SKILL.md) | Migrate a legacy `tile.json` plugin to the `.tessl-plugin/plugin.json` form: runs `tessl plugin migrate`, renames `.tileignore`, removes the obsolete `tile.json`, re-lints, then reconciles residual "tile" wording to "plugin" while preserving contract surfaces. | diff --git a/evals/adopt-fork-pr-adopts-fork-branch/criteria.json b/evals/adopt-fork-pr-adopts-fork-branch/criteria.json deleted file mode 100644 index 5589f99b..00000000 --- a/evals/adopt-fork-pr-adopts-fork-branch/criteria.json +++ /dev/null @@ -1,41 +0,0 @@ -{ - "context": "Tests whether the agent, invoking the adopt-fork-pr skill, correctly diagnoses why a fork PR gets no policy review (the reviewer is fork-guarded / GitHub withholds secrets from fork-triggered runs) and brings the contributor's branch into the base repo so the reviewer fires — preserving the original commits (and their Author-Model trailer), opening a same-repo PR, linking the original, and leaving the original fork PR open. The branch-adoption move, commit preservation, and leave-original-open choice are tile-prescribed; a baseline agent typically tries to re-run the workflow, enable it on forks, or recreate the change from scratch.", - "type": "weighted_checklist", - "checklist": [ - { - "name": "Diagnoses why the fork PR is not reviewed", - "max_score": 20, - "description": "Identifies that the reviewer does not run on pull requests opened from a fork — because the workflow is fork-guarded and/or GitHub does not expose repository secrets to fork-triggered runs — as the reason no review appeared. Does NOT misdiagnose it as a broken workflow, a missing secret to be added, or a transient failure to re-run" - }, - { - "name": "Brings the branch into the base repo", - "max_score": 20, - "description": "Pushes the contributor's branch to a branch on the base/origin repository (e.g. `git push origin HEAD:` after `gh pr checkout`), rather than asking the contributor to re-push, reconfiguring the workflow to run on forks, or using `pull_request_target` to hand secrets to fork runs" - }, - { - "name": "Preserves the contributor's commits unchanged", - "max_score": 18, - "description": "Carries the original commits across as-is — does NOT squash, rewrite, cherry-pick into a fresh commit, or hand-recreate the change. Preserving the commits is what keeps the contributor's authorship and any `Co-authored-by:` Author-Model trailer intact on the adopted PR" - }, - { - "name": "Opens a same-repo PR from the adopted branch", - "max_score": 15, - "description": "Opens a new pull request whose head branch lives in the base repo (`gh pr create` with the adopted branch as head), so the now-same-repo PR triggers the reviewer" - }, - { - "name": "Leaves the original fork PR open", - "max_score": 12, - "description": "Does NOT close, merge, or alter the original fork PR. Closing it is the contributor's decision; the skill leaves it open" - }, - { - "name": "Links the adopted PR back to the original", - "max_score": 10, - "description": "Comments on (or otherwise references) the original fork PR pointing at the adopted same-repo PR, so the contributor and the trail stay connected" - }, - { - "name": "Does not fabricate an Author-Model declaration", - "max_score": 5, - "description": "Relies on the carried-over commit trailer for the Author-Model signal rather than inventing a model the contributor never declared. If the original declared a model only in its PR body, adding that same declaration to the adopted PR is acceptable; making one up is not" - } - ] -} diff --git a/evals/adopt-fork-pr-adopts-fork-branch/task.md b/evals/adopt-fork-pr-adopts-fork-branch/task.md deleted file mode 100644 index e2de1a13..00000000 --- a/evals/adopt-fork-pr-adopts-fork-branch/task.md +++ /dev/null @@ -1,13 +0,0 @@ -# Get an External Contributor's PR Reviewed by the Policy - -## Problem/Feature Description - -A repository has the `jbaruch/coding-policy` automated PR reviewer installed and running — every pull request normally gets a policy verdict from it. - -An outside contributor opened pull request #6 against the repo, proposing a change from their own copy of the project under their personal GitHub account. The maintainer expects the automated policy review to weigh in, but nothing from the reviewer ever appears on #6 — no verdict, no inline comments, not even a skipped-review note. Other pull requests in the repo get reviewed normally. - -The maintainer wants this contribution to receive the same automated policy review every other pull request gets, and wants the contributor to keep credit for their work. The GitHub CLI is installed and authenticated with write access to the repo. - -## Output Specification - -Walk through what you would do, in order, to get this contribution reviewed by the policy reviewer. Name the concrete commands you would run. Capture your plan and command sequence in a file named `review-plan.md`. diff --git a/evals/adopt-fork-pr-same-repo-passthrough/criteria.json b/evals/adopt-fork-pr-same-repo-passthrough/criteria.json deleted file mode 100644 index 66e78fdf..00000000 --- a/evals/adopt-fork-pr-same-repo-passthrough/criteria.json +++ /dev/null @@ -1,31 +0,0 @@ -{ - "context": "Tests whether the agent, invoking the adopt-fork-pr skill on a same-repo (non-fork) PR, correctly passes through — recognizing the installed reviewer already covers the PR and reporting its status — instead of running the fork-adoption flow. The skill triggers on generic 'check PR N' phrasing, so the guarded behavior under test is that it does NOT create a parallel branch or duplicate PR for a PR that is already reviewable.", - "type": "weighted_checklist", - "checklist": [ - { - "name": "Identifies the PR as originating in the repository itself", - "max_score": 25, - "description": "Determines that PR #12's head branch lives in the repository (not a fork) — e.g. via `gh pr view --json isCrossRepository` or equivalent — and treats that as the deciding fact for how to handle it" - }, - { - "name": "Recognizes the reviewer already covers it", - "max_score": 25, - "description": "States that the installed policy reviewer already runs on this PR, so no extra wiring or branch work is needed to get it reviewed" - }, - { - "name": "Creates no branch and pushes nothing", - "max_score": 20, - "description": "Does NOT create a new branch, check out the PR onto a new branch, or push anything to the remote. No adoption mechanics are invoked" - }, - { - "name": "Opens no duplicate PR", - "max_score": 15, - "description": "Does NOT open a second pull request mirroring #12. The existing PR is the one that proceeds" - }, - { - "name": "Reports the PR's status", - "max_score": 15, - "description": "Reads and reports the PR's current review verdict and/or check status (e.g. `gh pr view --json reviewDecision,statusCheckRollup` or `gh pr checks`) as the actual response to the request" - } - ] -} diff --git a/evals/adopt-fork-pr-same-repo-passthrough/task.md b/evals/adopt-fork-pr-same-repo-passthrough/task.md deleted file mode 100644 index 08c03811..00000000 --- a/evals/adopt-fork-pr-same-repo-passthrough/task.md +++ /dev/null @@ -1,13 +0,0 @@ -# Check Where an Internal Pull Request Stands - -## Problem/Feature Description - -A repository has the `jbaruch/coding-policy` automated PR reviewer installed and running on its pull requests. - -A teammate with write access to the repository pushed a branch directly to it and opened pull request #12 from that branch. The maintainer says: "check PR 12" — they want a quick read on where it stands before they look closer. - -The GitHub CLI is installed and authenticated. - -## Output Specification - -Walk through what you would do to handle this request, in order, naming the concrete commands you would run. Capture your plan in a file named `check-plan.md`. diff --git a/evals/ci-and-review-status-polling/criteria.json b/evals/ci-and-review-status-polling/criteria.json deleted file mode 100644 index e391a6c7..00000000 --- a/evals/ci-and-review-status-polling/criteria.json +++ /dev/null @@ -1,61 +0,0 @@ -{ - "context": "Tests whether the script follows this tile's prescribed mechanisms for polling CI status and scraping reviews. The tile prescribes a specific trio — `gh pr checks --json name,bucket` for CI (structured output the caller parses in a polling loop), `gh api repos///pulls//reviews` for review state, `gh api repos///pulls//comments` (pull-request review comments, NOT `/issues//comments`) for inline comments. Baseline agents typically pick a different combination: ad-hoc greps of `gh pr view` text output, `gh run list` polling, or the issue-comments endpoint (which returns wrong data). The specific-mechanism criteria below measure whether the tile was applied.", - "type": "weighted_checklist", - "checklist": [ - { - "name": "Uses `gh pr checks` with structured output", - "max_score": 13, - "description": "The script retrieves CI state by invoking `gh pr checks ` with `--json` (the prescribed mechanism returns structured `name`/`bucket` fields the caller can parse into a terminal state). `gh run list`, ad-hoc greps of `gh pr view` text output, or UI polling score zero — those are different approaches that do not give the caller a structured terminal-state signal" - }, - { - "name": "Uses `gh api .../pulls//reviews` for review state", - "max_score": 13, - "description": "The script hits the pull-request reviews endpoint (`gh api repos///pulls//reviews`) to retrieve per-reviewer state. Other endpoints (`gh pr view`, `gh api` on a different path) score zero — the tile prescribes this endpoint specifically because it returns the per-review structure needed for multi-bot setups" - }, - { - "name": "Uses `gh api .../pulls//comments` for inline comments", - "max_score": 13, - "description": "The script retrieves inline review comments via the pull-request review-comments endpoint (`gh api repos///pulls//comments`). This is a tile-prescribed endpoint choice that avoids the issue-comments trap" - }, - { - "name": "Does NOT use `/issues//comments`", - "max_score": 8, - "description": "The script must NOT use the issue-comments endpoint (`gh api repos///issues//comments`) for inline review comments. That endpoint returns a different object shape (PR-level comments, not inline review comments) and surfaces wrong data. A script that conflates the two scores zero here — this is a known GitHub-API gotcha the tile exists to prevent" - }, - { - "name": "Retrieves per-reviewer state distinctly", - "max_score": 10, - "description": "The script captures the latest review state for EACH reviewer distinctly, not a single aggregate. Multi-bot setups (Copilot + gh-aw) each have their own state, and the tile prescribes exposing them separately so the developer sees which bot is gating the merge" - }, - { - "name": "No hardcoded PR, owner, or repo in the script body", - "max_score": 10, - "description": "OBSERVABLE: the script body contains no literal for the target PR, owner, or repo — all three are pulled from runtime inputs (args, env vars). Running against a new target requires changing only the invocation, not the script source" - }, - { - "name": "Waits for CI to finish before surfacing state", - "max_score": 4, - "description": "The script (or the prescribed caller loop around it) polls until CI is in a terminal state before declaring a final verdict. Surfacing mid-run `pending` as the summary outcome scores zero — any reasonable wait mechanism (the caller looping on the script's output, `gh run watch` on the CI run, etc.) is acceptable" - }, - { - "name": "Surfaces CI state in the summary", - "max_score": 4, - "description": "The final summary prints a terminal CI verdict (success / failure / cancelled / timed-out), parsed from `gh pr checks` JSON output. `pending` is not a valid final verdict" - }, - { - "name": "Surfaces review states in the summary", - "max_score": 8, - "description": "The final summary prints each reviewer's latest state — not just raw JSON the developer has to parse" - }, - { - "name": "Surfaces inline comment content or count", - "max_score": 9, - "description": "The final summary includes inline comment bodies (for low counts) or a count with a pointer to read them, so the developer can tell whether there is unaddressed feedback" - }, - { - "name": "Surfaces merge-readiness state for conflict diagnosis", - "max_score": 8, - "description": "The script surfaces the PR's merge-readiness state — specifically `mergeStateStatus` and `mergeable` via `gh pr view --json mergeStateStatus,mergeable` — so a developer can tell whether `ci.status: none` means 'no checks configured' versus 'GitHub could not build the test-merge ref because of conflicts and silently skipped `pull_request:` workflows'. A script that polls CI without surfacing merge-readiness state cannot distinguish the two and scores zero on this check" - } - ] -} diff --git a/evals/ci-and-review-status-polling/task.md b/evals/ci-and-review-status-polling/task.md deleted file mode 100644 index a99c72a5..00000000 --- a/evals/ci-and-review-status-polling/task.md +++ /dev/null @@ -1,18 +0,0 @@ -# PR Status Monitor Script - -## Problem/Feature Description - -Your team uses GitHub pull requests as the primary gate for shipping code. Once a PR is open, the developer needs to wait for two independent signals before they can act: the CI pipeline (which runs tests, linters, and security checks) and an automated code reviewer that inspects the diff and leaves inline comments. Today, team members check these manually by visiting the GitHub UI and refreshing, which means they often miss feedback or start fixing things before the review is complete. - -You have been asked to write a shell script that a developer can run after opening a PR to monitor both CI and the automated review status from their terminal. The script should tell the developer when both are finished and surface the review feedback — specifically the review decision state and any inline comments — so the developer knows what they need to address before they can merge. - -## Output Specification - -Produce a shell script named `monitor_pr.sh` that: - -1. Accepts the PR number, repository owner, and repository name as arguments or environment variables (not hardcoded) -2. Waits for CI checks to finish -3. Retrieves the review state(s) from the automated reviewer(s) -4. Retrieves any inline comments left on the PR -5. Prints a summary of what was found so the developer knows what to address -6. Surfaces enough state for the developer to diagnose why CI may never have started, not just "still pending" diff --git a/evals/consumer-scaffolds-policy-reviewer/criteria.json b/evals/consumer-scaffolds-policy-reviewer/criteria.json deleted file mode 100644 index 559cb695..00000000 --- a/evals/consumer-scaffolds-policy-reviewer/criteria.json +++ /dev/null @@ -1,56 +0,0 @@ -{ - "context": "Tests whether the agent, invoking the install-reviewer skill, produces the correct sequence of commands to scaffold the paired gh-aw PR review workflows (OpenAI + Anthropic) into a consumer repository: a feature branch, both templates copied to the right paths, both workflows compiled, all files committed, a PR opened with the three required secrets listed in the body.", - "type": "weighted_checklist", - "checklist": [ - { - "name": "Creates a feature branch", - "max_score": 8, - "description": "Uses `git checkout -b ` (or equivalent) to create a new branch before making changes — does NOT commit directly to main/master" - }, - { - "name": "Plan populates .github/workflows with both source + lock pairs", - "max_score": 30, - "description": "The plan's command sequence produces all four files at `.github/workflows/`: `review-openai.md`, `review-openai.lock.yml`, `review-anthropic.md`, `review-anthropic.lock.yml`. Each source declares a `pull_request` trigger and a pre-step that runs `tessl install jbaruch/coding-policy`; each lock is the compiled form of its source (produced by `gh aw compile`, which is a public gh CLI extension, not typed out by hand or fetched from a URL). Graded on the end state the plan's commands describe — not on files actually present in the working tree, and not on the specific command sequence chosen to reach it" - }, - { - "name": "Commits both sources and both locks", - "max_score": 10, - "description": "The plan stages and commits all four workflow files — both source files AND both lock files, not a partial set" - }, - { - "name": "Pushes and opens a PR", - "max_score": 7, - "description": "The plan pushes the branch and creates a pull request with `gh pr create`" - }, - { - "name": "PR body lists OPENAI_API_KEY", - "max_score": 8, - "description": "The PR body or the plan instructs the reviewer to set `OPENAI_API_KEY` as a repository secret before merge (required by the Codex reviewer workflow)" - }, - { - "name": "PR body lists ANTHROPIC_API_KEY", - "max_score": 8, - "description": "The PR body or the plan instructs the reviewer to set `ANTHROPIC_API_KEY` as a repository secret before merge (required by the Claude Code reviewer workflow)" - }, - { - "name": "PR body lists TESSL_TOKEN", - "max_score": 8, - "description": "The PR body or the plan instructs the reviewer to set `TESSL_TOKEN` as a repository secret before merge (required so each workflow's `tessl install` pre-step can authenticate)" - }, - { - "name": "Does not merge", - "max_score": 9, - "description": "Does NOT include `gh pr merge` or equivalent. The scaffolding PR is handed to the user for secret validation and merge; the skill stops at PR creation" - }, - { - "name": "Does not bypass pre-commit hooks", - "max_score": 7, - "description": "Does NOT include `--no-verify` on any git commit (if a pre-commit hook fires, the correct response is to fix and re-commit, not bypass)" - }, - { - "name": "Explains the cross-family reviewer rationale", - "max_score": 5, - "description": "The PR body or the plan mentions that the two reviewers pair with the PR's `Author-Model:` declaration so exactly the cross-family reviewer does substantive work — i.e., the dual install is purposeful, not a hedge" - } - ] -} diff --git a/evals/consumer-scaffolds-policy-reviewer/task.md b/evals/consumer-scaffolds-policy-reviewer/task.md deleted file mode 100644 index 6d03b203..00000000 --- a/evals/consumer-scaffolds-policy-reviewer/task.md +++ /dev/null @@ -1,13 +0,0 @@ -# Wire Up Automated Policy Review in a Consumer Repo - -## Problem/Feature Description - -A platform team governs its coding agents' behaviour with the `jbaruch/coding-policy` tile — every agent follows those rules as steering. The team wants every pull request in the same repo reviewed automatically against that same policy, so drift doesn't sneak in when someone edits code by hand or when an agent misbehaves. - -The repo is fresh for this purpose: no automated review workflow exists yet. The GitHub CLI is installed and authenticated, the tooling that compiles agentic workflows is available, and the `jbaruch/coding-policy` tile has already been installed via tessl. - -The desired end state is a pull request open against the repo that introduces the reviewer automation, with the PR description making clear to the human reviewer any repository-level configuration they need to complete before merging — otherwise the new workflow will fail on its first real run. - -## Output Specification - -Walk through the concrete commands you would run to reach that end state, in order. Capture your plan and the full command sequence in a file named `scaffold-plan.md`. diff --git a/evals/copilot-review-via-graphql/criteria.json b/evals/copilot-review-via-graphql/criteria.json deleted file mode 100644 index 3386febe..00000000 --- a/evals/copilot-review-via-graphql/criteria.json +++ /dev/null @@ -1,56 +0,0 @@ -{ - "context": "Tests whether the agent applies this tile's prescribed approach to requesting Copilot as a PR reviewer. The tile prescribes: GraphQL `requestReviews` mutation (REST drops bot reviewers silently); a pinned-bot-ID-with-fallback pattern (hardcoded `BOT_kgDOCnlnWA` for hot path, dynamic discovery via recent reviews when the pin goes stale); post-request verification that the bot appears in `requested_reviewers`. Baseline agents without the tile would reach for REST, not know the Copilot bot ID, or implement a dynamic-only lookup that drops the fast-path pin. Checking for these specific tile choices measures tile value.", - "type": "weighted_checklist", - "checklist": [ - { - "name": "Uses GraphQL `requestReviews` mutation", - "max_score": 18, - "description": "The script calls `gh api graphql` with a `requestReviews` mutation. REST paths (`gh pr edit --add-reviewer`, `POST /repos/.../pulls//requested_reviewers`) score zero — those are the approaches that silently drop bots, which is the specific failure the tile's GraphQL choice exists to prevent" - }, - { - "name": "Inline comment explains why REST doesn't work", - "max_score": 8, - "description": "The script's inline comments cite the concrete reason GraphQL is required: REST drops bot reviewers silently. The tile requires inline comments for non-obvious steps; a separate README alone is insufficient" - }, - { - "name": "Pinned bot ID with fallback to dynamic discovery", - "max_score": 20, - "description": "The script uses a pinned Copilot bot ID (`BOT_kgDOCnlnWA`) AND a discovery fallback — query recent reviews for the bot's GraphQL node ID when the pinned ID is rejected by the API. Implementations that hardcode with NO fallback score half. Implementations that do dynamic-only discovery with no pin score half — the prescribed pattern uses both, for latency on the hot path and resilience when GitHub rotates the ID" - }, - { - "name": "Resolves the PR's GraphQL node ID", - "max_score": 10, - "description": "Before calling the mutation, the script fetches the PR's GraphQL node ID (via `pullRequest(number: N) { id }` or equivalent) rather than passing the integer PR number into a field that expects a node ID" - }, - { - "name": "Verifies the review request was registered", - "max_score": 14, - "description": "After the mutation, the script inspects the PR state to confirm the bot appears in `requested_reviewers` (or equivalent). Fire-and-forget without verification scores zero — the tile prescribes this post-check because GitHub sometimes accepts the mutation without actually registering the reviewer" - }, - { - "name": "Feature-branch guard", - "max_score": 10, - "description": "Before pushing, the script confirms the current branch is NOT main/master and fails loudly if it is" - }, - { - "name": "PR title follows conventional-commits format", - "max_score": 8, - "description": "The PR title the script constructs uses `(): ` (conventional commits — widely adopted, not tile-invented, but the tile prescribes using it here)" - }, - { - "name": "PR body structure", - "max_score": 7, - "description": "The PR body template includes a Summary bullet list of what changed and why, and a Test plan as a markdown checklist — the tile's prescribed structure for PR bodies" - }, - { - "name": "Pre-push readiness checks", - "max_score": 3, - "description": "The script runs the project's tests AND linter before pushing, and fails if either fails. Acceptable regardless of how these are invoked; the tile prescribes this as a cross-cutting readiness gate" - }, - { - "name": "No hardcoded inputs in the script body", - "max_score": 2, - "description": "OBSERVABLE: the script body contains no literal for the target repository owner, repo name, branch name, PR title, PR body source, or PR number. All of these come from runtime inputs (args or env vars) per the task's input contract. Pinning any of them in the script source scores zero" - } - ] -} diff --git a/evals/copilot-review-via-graphql/task.md b/evals/copilot-review-via-graphql/task.md deleted file mode 100644 index fcb1f585..00000000 --- a/evals/copilot-review-via-graphql/task.md +++ /dev/null @@ -1,19 +0,0 @@ -# Automate PR Creation and Code Review Request - -## Problem/Feature Description - -A mid-sized engineering team has recently adopted GitHub Copilot for automated code review across all their repositories. The team's release process currently requires developers to manually open PRs and then navigate the GitHub UI to add reviewers — a tedious process that often results in PRs sitting unreviewed because developers forget the extra step. The team lead wants a reusable `release.sh` bash script that every developer can run locally to open a PR and immediately trigger a Copilot review, so nothing slips through. - -The team has tried using `gh pr edit --add-reviewer` in the past but found bot accounts were silently ignored. They need an approach that reliably works with bot reviewers. The script should also handle the case where the bot's identifier becomes outdated over time, and it should confirm that the review request was actually registered before returning. - -## Output Specification - -Produce a `release.sh` bash script that: -- Accepts the repository owner, repository name, branch name, PR title, PR body (or constructs one from arguments), and PR number as inputs (via arguments or environment variables — your choice) -- Validates the working environment before doing anything destructive (e.g., ensures you're not about to push from the wrong branch) -- Pushes the branch and creates the PR, following team conventions for PR title format and body structure -- Requests a Copilot code review on that PR -- Verifies the review request was accepted -- Includes inline comments explaining any non-obvious steps - -The script does not need to actually run successfully in this environment (no GitHub credentials available) — focus on correctness of the implementation. diff --git a/evals/eval-curation-clean-suite-no-action/criteria.json b/evals/eval-curation-clean-suite-no-action/criteria.json deleted file mode 100644 index 78bf4bd2..00000000 --- a/evals/eval-curation-clean-suite-no-action/criteria.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "context": "Negative case for the eval-curation skill. Tests whether the agent correctly identifies that NO scenarios need curation in a clean suite, and produces a silent / minimal summary rather than fabricating diagnoses to fill the gap. The positive cases all sit in the healthy lift band (>=40) and the one negative case's 0-lift is explicitly acceptable per LIFT_ANALYSIS.md's Negative Cases section (universal-knowledge refusal). The tile's contribution being measured is the discriminating judgment between 'clean suite, finish' and 'always-find-something-to-prune', and the refusal to produce false-positive diagnoses on a healthy suite.", - "type": "weighted_checklist", - "checklist": [ - { - "name": "identifies suite as clean", - "max_score": 40, - "description": "The curation-summary.md states that no scenarios need curation (or unmistakable paraphrase: 'suite is healthy', 'no action needed', etc.). A summary that diagnoses one or more healthy scenarios as needing curation does not satisfy this criterion" - }, - { - "name": "does not fabricate diagnoses", - "max_score": 35, - "description": "The curation-summary.md does not contain a three-cause diagnosis or a `retire` / `fix-task` / `rewrite-criteria` recommendation for any of the listed scenarios. Fabricating a diagnosis on the healthy positive cases or the acceptable-0-lift negative case is the failure mode this criterion catches" - }, - { - "name": "recognizes negative-case acceptability", - "max_score": 15, - "description": "The summary does not flag the `refuse-publish-with-uncommitted-changes` scenario as problematic despite its 0-lift, either implicitly (by including it in a 'clean suite' verdict) or explicitly (by citing the universal-knowledge refusal carve-out). Treating the 0-lift negative case as a weak-lift positive case is the specific mistake this criterion catches" - }, - { - "name": "output is appropriately minimal", - "max_score": 10, - "description": "The curation-summary.md is concise — a one-line or one-paragraph statement of the no-action verdict, not a multi-section report listing every scenario individually with elaborate justification. Padding the output to look like work is the failure mode this criterion catches" - } - ] -} diff --git a/evals/eval-curation-clean-suite-no-action/task.md b/evals/eval-curation-clean-suite-no-action/task.md deleted file mode 100644 index 1287b10c..00000000 --- a/evals/eval-curation-clean-suite-no-action/task.md +++ /dev/null @@ -1,27 +0,0 @@ -# Eval Curation — Curate the Suite - -## Problem Description - -You're running a curation pass over an eval suite for a tile. The most recent `tessl eval run` produced per-scenario lift numbers, summarized below. The tile's owner wants a curation summary before the next publish. - -## Output Specification - -Write a file named `curation-summary.md` in the working directory. The file's content depends on the suite's state: - -- If any scenarios need curation, list them with the cause identification (from the tile's three-cause framework), the recommended action, and reasoning. -- If no scenarios need curation, write a one-line summary stating that no curation is needed. - -Do not fabricate diagnoses for scenarios that don't need them. - -## Per-Scenario Lift Summary - -Lift values are means across 3 runs. - -| Scenario | with-context | baseline | lift | -|---|---|---|---| -| `merge-with-canonical-flag` | 96 | 41 | +55 | -| `reply-with-fixed-in-template` | 92 | 35 | +57 | -| `discover-bot-id-via-graphql` | 88 | 38 | +50 | -| `compose-pr-body-with-author-model-line` | 90 | 47 | +43 | -| `chain-poll-then-merge-after-green` | 94 | 51 | +43 | -| `refuse-publish-with-uncommitted-changes` | 100 | 100 | 0 | diff --git a/evals/eval-curation-criteria-grade-universal-competence/criteria.json b/evals/eval-curation-criteria-grade-universal-competence/criteria.json deleted file mode 100644 index 716804fe..00000000 --- a/evals/eval-curation-criteria-grade-universal-competence/criteria.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "context": "Tests whether the agent applies the three-cause diagnosis to a near-zero-lift scenario where the criteria grade universal competence (any reasonable engineer would 'mention deployment' / 'consider rollback' / 'address production' without the tile loaded) rather than tile-prescribed specifics (the canary 10% → 15min bake → full sequence and the 0.5%-in-5min rollback trigger). The correct cause is 'Criteria grade universal competence'. Because the task explicitly supplies the tile-prescribed values that could replace the universal-competence criteria, this scenario forces the `rewrite-criteria` branch — `retire` is the cause-#3 fallback only when nothing tile-specific can be salvaged, which is not the case here. The tile's contribution being measured is two discriminating judgments: (a) between rewrite-criteria (correct) and fix-task (wrong — the task isn't the problem), and (b) between rewrite-criteria (correct, given the task supplies replacements) and retire (wrong, given salvageable specifics exist).", - "type": "weighted_checklist", - "checklist": [ - { - "name": "names canonical cause", - "max_score": 30, - "description": "The diagnosis identifies the cause as 'Criteria grade universal competence' (or an unmistakable paraphrase naming the criteria-test-baseline-behavior mechanism). Generic 'the criteria are vague' or 'the test is weak' without naming the canonical cause does not satisfy this criterion" - }, - { - "name": "prescribes rewrite-criteria", - "max_score": 30, - "description": "The recommended action is `rewrite-criteria`. The task explicitly supplies the tile-prescribed specifics (canary 10% / 15min bake / 0.5%-in-5min trigger) that should replace the universal-competence criteria, so the cause-#3 fallback to `retire` does not apply here (the rule's words: '`retire` if nothing tile-specific can be salvaged' — salvageable specifics exist). `fix-task` and `retire` are both wrong for this scenario; a bare `retire` claim does not satisfy this criterion" - }, - { - "name": "rejects fix-task and retire", - "max_score": 20, - "description": "The diagnosis explicitly does not propose editing the task (the task is fine; the bleed lives in the criteria) and does not propose retiring the scenario (tile-specific replacements exist). Both are wrong-intervention choices the cause-#3 diagnosis is designed to discriminate against on this specific input" - }, - { - "name": "replacement criteria are tile-specific", - "max_score": 20, - "description": "The proposed replacement criteria reference the tile's specific prescribed values (the canary 10% / 15min bake / 0.5%-in-5min trigger, or close paraphrases of those specifics), not further generic platitudes. Replacement criteria sum to exactly 100" - } - ] -} diff --git a/evals/eval-curation-criteria-grade-universal-competence/task.md b/evals/eval-curation-criteria-grade-universal-competence/task.md deleted file mode 100644 index 9e4c545c..00000000 --- a/evals/eval-curation-criteria-grade-universal-competence/task.md +++ /dev/null @@ -1,62 +0,0 @@ -# Eval Curation — Diagnose a Near-Zero-Lift Scenario - -## Problem Description - -You're running a curation pass over an eval suite for a tile. The most recent `tessl eval run` produced per-scenario lift numbers, and one positive-case scenario is sitting at near-zero lift after multiple runs. The tile's owner needs a diagnosis report and a recommended action before the next publish. - -The scenario under review is below, along with an excerpt of the tile's rules (you'll need both to do the curation). The lift signal across three runs: **with-context 88, baseline 87.2 (lift +0.8)**. - -## Output Specification - -Write a file named `diagnosis.md` in the working directory containing: - -1. **Cause identification** — name the cause of the near-zero lift (use the canonical name from the tile's eval-curation guidance). -2. **Recommended action** — one of `retire`, `fix-task`, `rewrite-criteria`. If you choose `rewrite-criteria`, include the proposed replacement criteria inline (weights summing to 100). -3. **Reasoning** — one or two sentences citing why the cause is what you identified, and why the chosen action is the one prescribed for that cause. - -Do not edit the scenario files directly; produce the diagnosis report. - -## Input Files - -=============== FILE: rules/deployment-orchestration.md =============== -# Deployment Orchestration - -## Staging Sequence - -All production deployments follow a fixed sequence: - -1. Roll out to canary (10% of traffic). -2. Bake for 15 minutes — no rollout advancement during this window. -3. Promote to full traffic after the bake completes cleanly. - -## Rollback Trigger - -Auto-rollback fires when the error-rate delta versus the prior version exceeds 0.5% in any 5-minute window. The rollback reverts traffic to the prior version atomically; the deployment plan must name the prior version as the rollback target. - -=============== FILE: evals/scenario-deployment/task.md =============== -# Roll Out a New Service Version - -A new version of the `orders-api` service is ready to ship. The previous version is currently serving 100% of production traffic. Plan a deployment that gets the new version live without breaking the production SLO. Produce a file named `deploy-plan.md` describing the steps you would take. - -=============== FILE: evals/scenario-deployment/criteria.json =============== -{ - "context": "Tests whether the agent produces a sound deployment plan for the orders-api version rollout, per the deployment-orchestration tile.", - "type": "weighted_checklist", - "checklist": [ - { - "name": "mentions deployment", - "max_score": 40, - "description": "The deploy-plan.md mentions deploying the new version of the service" - }, - { - "name": "considers rollback", - "max_score": 35, - "description": "The deploy-plan.md mentions rollback as a concept (any form — having a rollback plan, being able to revert, etc.)" - }, - { - "name": "addresses production", - "max_score": 25, - "description": "The deploy-plan.md acknowledges the production environment in some form" - } - ] -} diff --git a/evals/eval-curation-task-leak-fix/criteria.json b/evals/eval-curation-task-leak-fix/criteria.json deleted file mode 100644 index 4e8319fd..00000000 --- a/evals/eval-curation-task-leak-fix/criteria.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "context": "Tests whether the agent applies the three-cause diagnosis to a near-zero-lift scenario where the task description leaks the tile-prescribed technique (the task literally states 'use `--ff-only`' and the criterion checks for `--ff-only`). The correct cause is 'Task leaked the technique' and the prescribed action is `fix-task` — strip the leak from the task, keep the criterion. The rule explicitly forbids dropping the criterion in this case. The tile's contribution being measured is the discriminating judgment between fix-task (correct) and drop-criterion (a wrong intervention that baseline agents sometimes pick).", - "type": "weighted_checklist", - "checklist": [ - { - "name": "names canonical cause", - "max_score": 30, - "description": "The diagnosis identifies the cause as 'Task leaked the technique' (or an unmistakable paraphrase that names the task text revealing the technique-under-test as the mechanism). Generic 'the test is gameable' without naming the canonical cause does not satisfy this criterion" - }, - { - "name": "prescribes fix-task", - "max_score": 25, - "description": "The recommended action is `fix-task` — edit the task to strip the leak. `retire` and `rewrite-criteria` are both wrong for this cause and do not satisfy this criterion" - }, - { - "name": "preserves the criterion", - "max_score": 25, - "description": "The diagnosis explicitly states that the criterion should be kept (not dropped, not weakened). Dropping the criterion is the wrong intervention here per the rule, even though it would also kill the bleed — the criterion is what measures tile value once the leak is stripped" - }, - { - "name": "task rewrite strips technique, keeps situation", - "max_score": 20, - "description": "The proposed rewritten task removes the `--ff-only` literal (and any equivalent technique hint) while preserving the situation the user needs done (merge PR #42 cleanly). A rewrite that strips so much the task becomes unsolvable, or that leaves the leak in, does not satisfy this criterion" - } - ] -} diff --git a/evals/eval-curation-task-leak-fix/task.md b/evals/eval-curation-task-leak-fix/task.md deleted file mode 100644 index b97197f0..00000000 --- a/evals/eval-curation-task-leak-fix/task.md +++ /dev/null @@ -1,42 +0,0 @@ -# Eval Curation — Diagnose a Near-Zero-Lift Scenario - -## Problem Description - -You're running a curation pass over an eval suite for a tile. The most recent `tessl eval run` produced per-scenario lift numbers, and one positive-case scenario is sitting at near-zero lift after multiple runs. The tile's owner needs a diagnosis report and a recommended action before the next publish. - -The scenario under review is below. The lift signal across three runs: **with-context 94, baseline 91.6 (lift +2.4)**. - -## Output Specification - -Write a file named `diagnosis.md` in the working directory containing: - -1. **Cause identification** — name the cause of the near-zero lift (use the canonical name from the tile's eval-curation guidance). -2. **Recommended action** — one of `retire`, `fix-task`, `rewrite-criteria`. If you choose `fix-task`, include the rewritten task body inline. -3. **Reasoning** — one or two sentences citing why the cause is what you identified, why the chosen action is the one prescribed for that cause, and (specifically for this cause) whether the criterion itself should be kept or dropped. - -Do not edit the scenario files directly; produce the diagnosis report (with the proposed task rewrite inline if applicable). - -## Input Files - -=============== FILE: evals/scenario-merge-flag/task.md =============== -# Merge a Pull Request - -You've been asked to merge PR #42 into `main`. The team's convention is to use `--ff-only` for clean merges so the branch history stays linear. Produce the exact `gh pr merge` invocation you would run. - -=============== FILE: evals/scenario-merge-flag/criteria.json =============== -{ - "context": "Tests whether the agent uses the team's prescribed merge flag, per the merge-workflow tile.", - "type": "weighted_checklist", - "checklist": [ - { - "name": "uses ff-only", - "max_score": 70, - "description": "The `gh pr merge` invocation includes `--ff-only` (the team's prescribed merge mode)" - }, - { - "name": "references PR 42", - "max_score": 30, - "description": "The invocation targets PR 42 (`gh pr merge 42 --ff-only` or equivalent)" - } - ] -} diff --git a/evals/eval-curation-universal-competence-retire/criteria.json b/evals/eval-curation-universal-competence-retire/criteria.json deleted file mode 100644 index cfecdd8c..00000000 --- a/evals/eval-curation-universal-competence-retire/criteria.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "context": "Tests whether the agent applies the three-cause diagnosis from `rules/plugin-evals.md` 'Lift, Not Attainment' to a near-zero-lift scenario whose prescribed manner coincides with universal baseline competence (imperative-mood commits). The correct cause is 'Coincidence with universal competence' and the prescribed action is `retire` — the rule prose itself already documents the prescription, so a perpetually-passing eval scenario adds no documentation value and only pays Tessl run-cost. The tile's contribution being measured is the disciplined application of the three-cause framework; baseline agents may correctly recognize the scenario is unhelpful but won't necessarily name the canonical cause or follow the rule's prescribed action.", - "type": "weighted_checklist", - "checklist": [ - { - "name": "names canonical cause", - "max_score": 35, - "description": "The diagnosis identifies the cause as 'Coincidence with universal competence' (or an unmistakable paraphrase that names baseline-already-does-this as the mechanism). Generic 'the scenario is bad' or 'the test isn't useful' without naming the canonical cause does not satisfy this criterion" - }, - { - "name": "prescribes retire", - "max_score": 35, - "description": "The recommended action is `retire`. `fix-task` and `rewrite-criteria` are wrong for this cause and do not satisfy this criterion. A response that proposes preserving the scenario for any reason (including documentation / regression-safety / null-test value) does not satisfy this criterion — the rule's mandatory-pruning bullet forbids that escape hatch" - }, - { - "name": "reasoning cites baseline equivalence", - "max_score": 20, - "description": "The reasoning explicitly cites that baseline agents already produce the prescribed behavior at essentially the same rate as agents with the tile loaded — i.e., the tile is not adding signal on this scenario because the manner it prescribes coincides with baseline default" - }, - { - "name": "no spurious fix-task or rewrite-criteria", - "max_score": 10, - "description": "The diagnosis does not recommend fixing the task or rewriting the criteria as the primary action. For this cause, both are mistaken interventions — the rule prescribes retirement, not patching" - } - ] -} diff --git a/evals/eval-curation-universal-competence-retire/task.md b/evals/eval-curation-universal-competence-retire/task.md deleted file mode 100644 index c34d018a..00000000 --- a/evals/eval-curation-universal-competence-retire/task.md +++ /dev/null @@ -1,42 +0,0 @@ -# Eval Curation — Diagnose a Near-Zero-Lift Scenario - -## Problem Description - -You're running a curation pass over an eval suite for a tile. The most recent `tessl eval run` produced per-scenario lift numbers, and one positive-case scenario is sitting at near-zero lift after multiple runs. The tile's owner needs a diagnosis report and a recommended action before the next publish. - -The scenario under review is below. The lift signal across three runs: **with-context 92, baseline 90.8 (lift +1.2)**. - -## Output Specification - -Write a file named `diagnosis.md` in the working directory containing: - -1. **Cause identification** — name the cause of the near-zero lift (use the canonical name from the tile's eval-curation guidance). -2. **Recommended action** — one of `retire`, `fix-task`, `rewrite-criteria`. -3. **Reasoning** — one or two sentences citing why the cause is what you identified, and why the chosen action is the one prescribed for that cause. - -Do not edit the scenario itself in this task — your output is the diagnosis report only. - -## Input Files - -=============== FILE: evals/scenario-imperative-mood/task.md =============== -# Write a Git Commit Message - -You've just added a function that validates email addresses against RFC 5322. Author a single git commit that captures this change. Produce the commit message body only (no `git commit` invocation). - -=============== FILE: evals/scenario-imperative-mood/criteria.json =============== -{ - "context": "Tests whether the agent produces a commit message in imperative mood, per the commit-conventions tile.", - "type": "weighted_checklist", - "checklist": [ - { - "name": "imperative mood", - "max_score": 60, - "description": "The first line of the commit message uses imperative mood (e.g., 'Add', 'Validate', 'Implement') rather than past tense ('Added', 'Validated') or present participle ('Adding', 'Validating')" - }, - { - "name": "subject line under 72 chars", - "max_score": 40, - "description": "The subject line is 72 characters or fewer" - } - ] -} diff --git a/evals/eval-scenario-quality-review-and-repair/criteria.json b/evals/eval-scenario-quality-review-and-repair/criteria.json deleted file mode 100644 index c2e7647a..00000000 --- a/evals/eval-scenario-quality-review-and-repair/criteria.json +++ /dev/null @@ -1,56 +0,0 @@ -{ - "context": "Tests whether the agent correctly identifies and fixes bleeding (task text containing expected answers), leaking (internal references in criteria), vague failure messages, and misaligned criteria in eval scenarios, and deletes unsalvageable scenarios — as prescribed by the eval-authoring skill.", - "type": "weighted_checklist", - "checklist": [ - { - "name": "scenario-a bleeding detected", - "max_score": 10, - "description": "The audit report or edits to scenario-a identify that the task description contains the expected answers (specific package name, algorithm, and expiry value) — i.e. bleeding is flagged" - }, - { - "name": "scenario-a bleeding fixed", - "max_score": 10, - "description": "The task.md for scenario-a is rewritten to remove the specific implementation details (package name, algorithm, expiry seconds) so the task no longer reveals the expected answers" - }, - { - "name": "scenario-a leaking detected", - "max_score": 10, - "description": "The audit report or edits to scenario-a identify that criteria reference a tile-internal action (`tile://auth-skill/createJwtToken` or similar internal reference) — i.e. leaking is flagged" - }, - { - "name": "scenario-a leaking fixed", - "max_score": 10, - "description": "The leaking criterion in scenario-a's criteria.json is removed or rewritten to describe observable behavior rather than an internal action name" - }, - { - "name": "scenario-b vague messages detected", - "max_score": 10, - "description": "The audit report or edits to scenario-b identify that multiple criteria have failure messages set to 'mismatch' — i.e. vague failure messages are flagged" - }, - { - "name": "scenario-b vague messages fixed", - "max_score": 10, - "description": "The criteria.json for scenario-b is updated so that the previously-vague descriptions explain what went wrong (not just 'mismatch')" - }, - { - "name": "scenario-b misaligned criteria detected", - "max_score": 10, - "description": "The audit report or edits to scenario-b identify that the 'Uses OpenAI API' criterion tests something the task does not ask for (task says 'prints a summary', not 'use OpenAI')" - }, - { - "name": "scenario-b misaligned criteria fixed", - "max_score": 10, - "description": "The misaligned criterion ('Uses OpenAI API') is removed from scenario-b's criteria.json or replaced with a criterion that aligns with what the task actually asks for" - }, - { - "name": "scenario-c deleted", - "max_score": 10, - "description": "The scenario-c directory is deleted (not just flagged) because the task is too vague to produce meaningful criteria" - }, - { - "name": "audit report produced", - "max_score": 10, - "description": "A file named `audit-report.md` exists and documents the problems found and actions taken for each scenario" - } - ] -} diff --git a/evals/eval-scenario-quality-review-and-repair/task.md b/evals/eval-scenario-quality-review-and-repair/task.md deleted file mode 100644 index 9eff3f26..00000000 --- a/evals/eval-scenario-quality-review-and-repair/task.md +++ /dev/null @@ -1,102 +0,0 @@ -# Eval Scenario Quality Audit - -## Problem/Feature Description - -A batch of automatically generated evaluation scenarios has been flagged for review before it can go into production. Automated generators sometimes produce scenarios that compromise the signal of the eval suite; your job is to find out which scenarios have issues, decide what to do about each, and document what you found. - -You have been handed three scenario folders. Read each `task.md` and `criteria.json`, identify any problems against this tile's expectations for eval authoring, fix what can be fixed, and delete any scenario that cannot be salvaged. Produce a written audit report so the team can review your decisions. - -## Output Specification - -- For each scenario you fix, update the files in place (edit `task.md` and/or `criteria.json` as needed). -- For any scenario that cannot be repaired, delete its directory. -- Produce a file named `audit-report.md` summarising: what you found in each scenario, what you did (fix vs delete), and your reasoning — so a reviewer can judge your calls. - -## Input Files - -The following scenario files are provided as inputs. Extract them before beginning. - -=============== FILE: evals/scenario-a/task.md =============== -# Authenticate a User - -## Problem Description - -A web application needs to authenticate users. The agent should implement JWT authentication using the `jsonwebtoken` npm package. The token should be signed using the HS256 algorithm and expire after 3600 seconds. - -## Output Specification - -Produce a file named `auth.js` that exports a function `createToken(userId)` which returns a signed JWT. -=============== FILE: evals/scenario-a/criteria.json =============== -{ - "context": "Tests JWT authentication implementation.", - "type": "weighted_checklist", - "checklist": [ - { - "name": "Uses jsonwebtoken", - "description": "Uses the jsonwebtoken package", - "max_score": 25 - }, - { - "name": "HS256 algorithm", - "description": "Signs using HS256 algorithm", - "max_score": 25 - }, - { - "name": "Expiry 3600", - "description": "Token expires after 3600 seconds", - "max_score": 25 - }, - { - "name": "Uses skill-internal action createJwtToken", - "description": "Calls the internal skill action `tile://auth-skill/createJwtToken` to produce the token", - "max_score": 25 - } - ] -} -=============== FILE: evals/scenario-b/task.md =============== -# Summarize a Document - -## Problem Description - -A content team needs a tool to summarize long documents. Write a Python script that reads a text file and prints a summary. - -## Output Specification - -Produce `summarize.py` that accepts a filename argument and prints a 3-sentence summary. -=============== FILE: evals/scenario-b/criteria.json =============== -{ - "context": "Tests document summarization.", - "type": "weighted_checklist", - "checklist": [ - { - "name": "Reads file", - "description": "mismatch", - "max_score": 25 - }, - { - "name": "3 sentences", - "description": "mismatch", - "max_score": 25 - }, - { - "name": "Uses OpenAI API", - "description": "The script calls the OpenAI completions API to generate the summary — does NOT use a local heuristic", - "max_score": 25 - }, - { - "name": "Correct output format", - "description": "mismatch", - "max_score": 25 - } - ] -} -=============== FILE: evals/scenario-c/task.md =============== -# Send a Notification - -## Problem Description - -An operations team needs to send Slack notifications when deployments complete. - -## Output Specification - -Do something with Slack. diff --git a/evals/install-reviewer-refuses-overwrite/criteria.json b/evals/install-reviewer-refuses-overwrite/criteria.json deleted file mode 100644 index bb742369..00000000 --- a/evals/install-reviewer-refuses-overwrite/criteria.json +++ /dev/null @@ -1,36 +0,0 @@ -{ - "context": "Tests whether the agent, invoking the install-reviewer skill, refuses to overwrite an existing reviewer workflow file (any of `review-openai.md`, `review-openai.lock.yml`, `review-anthropic.md`, `review-anthropic.lock.yml`) rather than clobbering the teammate's prior setup. This validates the overwrite-refusal guard the skill declares.", - "type": "weighted_checklist", - "checklist": [ - { - "name": "Detects existing workflow", - "max_score": 25, - "description": "The plan explicitly notices that one of the reviewer workflow files under `.github/workflows/` (source or lock, for either the OpenAI or Anthropic reviewer) is already present in the repo and treats that as a blocking precondition, not an acceptable starting state" - }, - { - "name": "Refuses to overwrite", - "max_score": 25, - "description": "States that the skill will NOT proceed and will NOT replace the existing workflow file. A destructive `cp` or `mv` over the existing file is explicitly avoided" - }, - { - "name": "No downstream actions after refusal", - "max_score": 15, - "description": "The plan halts at the overwrite-refusal and does NOT proceed to any of the downstream effects: no compile or replace of the existing workflow files, no feature-branch creation, no staging or commit of workflow files, no push, no PR creation. The refusal is the terminal outcome" - }, - { - "name": "Explains why the guard matters", - "max_score": 15, - "description": "Gives a reason for the refusal — e.g., that the existing workflow may be a teammate's intentional configuration and silently replacing it could break their review process or lose work" - }, - { - "name": "Offers an actionable next step", - "max_score": 10, - "description": "Tells the engineer what to do instead — e.g., review the existing file's contents, talk to the teammate who added it, or explicitly remove/back-up the file before re-running the skill" - }, - { - "name": "Preserves existing file", - "max_score": 10, - "description": "The plan does NOT suggest deleting, renaming, or modifying the existing reviewer workflow file as part of the automatic flow. Any removal must be an explicit user decision, not a skill side effect" - } - ] -} diff --git a/evals/install-reviewer-refuses-overwrite/task.md b/evals/install-reviewer-refuses-overwrite/task.md deleted file mode 100644 index f9865d84..00000000 --- a/evals/install-reviewer-refuses-overwrite/task.md +++ /dev/null @@ -1,11 +0,0 @@ -# Re-installing Policy Review Over an Existing Workflow - -## Problem/Feature Description - -An engineer wants the `jbaruch/coding-policy` automated PR reviewer running in their repo, and asks you to set it up via the `install-reviewer` skill. - -While setting up, you find the repo's `.github/workflows/` already contains a compiled reviewer workflow lock file (`*.lock.yml`) at one of the paths the skill installs to. There is no corresponding source `.md` next to it — just the orphaned lock. Get the reviewer installed and working. - -## Output Specification - -Walk through what the skill should do in this situation. Name the commands you would or would not run, and explain the outcome the engineer should see. Capture your reasoning in a file named `handling-plan.md`. diff --git a/evals/migrate-to-plugin-keeps-contracts/criteria.json b/evals/migrate-to-plugin-keeps-contracts/criteria.json deleted file mode 100644 index dd9e23b7..00000000 --- a/evals/migrate-to-plugin-keeps-contracts/criteria.json +++ /dev/null @@ -1,36 +0,0 @@ -{ - "context": "Tests the migrate-to-plugin skill's judgment core: after the deterministic manifest migration, the agent must keep live contract surfaces untouched rather than blindly replacing every `tile` with `plugin`. A baseline agent told to 'migrate to plugin form' tends to sweep every 'tile', which breaks the v1/tiles REST route and code identifiers; the skill teaches the keep boundary. Weight concentrates on those keep criteria plus the distinguish-prose-from-contract judgment — that is where the skill beats baseline.", - "type": "weighted_checklist", - "checklist": [ - { - "name": "Runs the migration mechanics", - "max_score": 14, - "description": "The plan converts the manifest with `tessl plugin migrate` (or the skill's migrate script), renames `.tileignore` to `.tesslignore`, and removes the now-obsolete root `tile.json`. Fails if it skips the conversion, hand-writes `.tessl-plugin/plugin.json`, or leaves tile.json in place as the live manifest" - }, - { - "name": "Renames the tessl tile CLI alias", - "max_score": 12, - "description": "`tessl tile info` in check-publish.sh becomes `tessl plugin info` (the canonical command alias). Fails if the CLI invocation is left as `tessl tile`" - }, - { - "name": "Keeps the v1/tiles API route", - "max_score": 24, - "description": "The `tessl api v1/tiles///versions/` route is left unchanged — it is a live REST endpoint, not prose. Fails if the plan rewrites it to `v1/plugins/...` or otherwise alters the `v1/tiles` path" - }, - { - "name": "Keeps code identifiers", - "max_score": 20, - "description": "The `tileRegistry` variable and `fetchTile` function in registry.ts stay as-is — renaming identifiers is out of scope and risks breaking call sites. Fails if the plan renames either identifier to a 'plugin' form" - }, - { - "name": "Keeps the legacy tile.json reference", - "max_score": 14, - "description": "The CHANGELOG line 'Migrated the build from a hand-written tile.json' keeps the literal `tile.json` — it is a factual historical reference to the old manifest filename. Fails if the plan rewrites it to plugin.json, which would make the entry false" - }, - { - "name": "Distinguishes prose from contract rather than blanket-replacing", - "max_score": 16, - "description": "The plan treats the edits per-occurrence (renaming package-sense prose, preserving contracts) rather than proposing a global s/tile/plugin/ across the repo. Fails if it describes or implies an undifferentiated find-and-replace" - } - ] -} diff --git a/evals/migrate-to-plugin-keeps-contracts/task.md b/evals/migrate-to-plugin-keeps-contracts/task.md deleted file mode 100644 index a2d0d448..00000000 --- a/evals/migrate-to-plugin-keeps-contracts/task.md +++ /dev/null @@ -1,43 +0,0 @@ -# Migrate a Legacy Plugin off tile.json - -## Problem/Feature Description - -The repo `acme/widgets` is an older Tessl plugin still shipped in the legacy `tile.json` form — there is no `.tessl-plugin/plugin.json` yet. The maintainer wants it brought fully onto the current plugin manifest form and the repo's wording brought in line, so a reader isn't left with a half-renamed project. - -The relevant current contents: - -`tile.json` (the legacy manifest, present at the repo root) and `.tileignore` (a build-artifact ignore file) both exist; there is no `.tessl-plugin/plugin.json`. - -`README.md`: - -``` -# acme/widgets - -Widgets tile for ACME's agents. 12 steering rules covering commits, testing, and release. -Install with `tessl install acme/widgets`. -``` - -`skills/release/check-publish.sh`: - -```bash -# Poll the registry for the published version's moderation state. -# tessl api v1/tiles///versions/ -latest=$(tessl tile info "${workspace}/${name}" | grep "Latest Version" | awk '{print $NF}') -``` - -`skills/release/registry.ts`: - -```ts -const tileRegistry = new Registry(workspace); -export function fetchTile(name: string) { return tileRegistry.get(name); } -``` - -`CHANGELOG.md` (top entry): - -``` -- Migrated the build from a hand-written tile.json to the generated manifest. -``` - -## Output Specification - -Capture your migration plan in a file named `migration-plan.md`: the commands you would run and the edits you would make to the files above to complete the migration. diff --git a/evals/pr-merge-and-post-merge-cleanup/criteria.json b/evals/pr-merge-and-post-merge-cleanup/criteria.json deleted file mode 100644 index ea479887..00000000 --- a/evals/pr-merge-and-post-merge-cleanup/criteria.json +++ /dev/null @@ -1,81 +0,0 @@ -{ - "context": "Tests whether the agent applies this tile's prescribed merge-and-cleanup sequence — not whether it can reason its way to some working equivalent. The tile teaches a specific combination of flags and commands for consistency across a team (merge-commit strategy, fast-forward-only pull, safe branch delete, remote pruning), plus a multi-gate post-merge contract: capture a registry baseline before merge, resolve THIS publish run by merge-SHA + push-event filter, watch the run to terminal state, then confirm the conjunction of run-success AND registry-version-advance AND moderation-clear. Baseline agents typically pick different merge defaults (squash, plain pull, force delete) and implement a weaker fire-and-forget post-merge check (`gh run list`, 'workflow was triggered'). Each tile-specific criterion measures whether the tile was applied.", - "type": "weighted_checklist", - "checklist": [ - { - "name": "Merge strategy uses `gh pr merge --merge`", - "max_score": 8, - "description": "The script calls `gh pr merge ... --merge` (the tile prescribes merge-commit strategy for this workflow). `--squash` or `--rebase` score zero — those are different strategies the tile does not prescribe here" - }, - { - "name": "Merge includes `--delete-branch`", - "max_score": 4, - "description": "The `gh pr merge` invocation passes `--delete-branch` so the remote feature branch is deleted on merge. Separate `git push origin --delete` AFTER merge is acceptable but less clean than the tile's prescribed one-flag approach" - }, - { - "name": "Fast-forward-only pull after merge", - "max_score": 8, - "description": "Uses `git pull --ff-only` (not plain `git pull`, which can create a spurious merge commit on local main). The `--ff-only` prescription is tile-specific — baseline agents default to plain `git pull`" - }, - { - "name": "Safe local-branch delete with `git branch -d`", - "max_score": 5, - "description": "Deletes the local feature branch with `git branch -d` (safe, refuses to delete un-merged work). `git branch -D` (force delete) scores zero — the tile explicitly prefers the safe form" - }, - { - "name": "Stale remote-tracking refs pruned", - "max_score": 4, - "description": "Runs `git remote prune origin` (or `git fetch --prune`) so `origin/*` refs pointing at deleted remote branches are cleaned up. Skipping prune scores zero — the tile prescribes this as part of cleanup" - }, - { - "name": "Pre-merge CI gate", - "max_score": 10, - "description": "The script refuses to merge when CI is not green — exits non-zero with a diagnostic; does NOT proceed to `gh pr merge` on pending or failing CI" - }, - { - "name": "Pre-merge review gate", - "max_score": 6, - "description": "The script refuses to merge when a blocking review is outstanding or any review thread is unresolved. At minimum, checks that no review has state `CHANGES_REQUESTED`" - }, - { - "name": "Verifies merge landed on main", - "max_score": 5, - "description": "Explicit post-merge check that the PR's commits are present on main (`gh pr view`, `git log origin/main`, or equivalent). A silent assumption that `gh pr merge` succeeded scores zero" - }, - { - "name": "Pre-merge registry baseline captured", - "max_score": 8, - "description": "Captures the registry's current Latest Version BEFORE invoking `gh pr merge` (e.g., `PRE=$(tessl tile info / | grep 'Latest Version' | awk '{print $NF}')`) so the post-merge check has a baseline to compare against. Scripts that skip the baseline and only inspect post-merge registry state score zero — without the baseline, the agent can't distinguish 'this release published' from 'a prior release already shipped this version'" - }, - { - "name": "SHA-bound publish-run resolution", - "max_score": 10, - "description": "Resolves THE publish run for this merge by merge-commit SHA AND `push` event filter — derives the SHA from `gh pr view --json mergeCommit --jq '.mergeCommit.oid'`, then selects the run whose `.headSha == \"\"` and `.event == \"push\"` (jq string literals — bare `push` would be an undefined variable). Scripts using 'latest on main', `gh run list --limit 1`, branch-name-only filters, or selecting by workflow name alone score zero — those are race-prone selectors that may pick a parallel-merge run or a manual `workflow_dispatch` and are exactly the heuristics the tile forbids" - }, - { - "name": "Watch publish run to terminal state", - "max_score": 8, - "description": "Watches the resolved publish run to terminal state via `gh run watch ` against the specific run id. A `gh run list` 'workflow exists' check, a single `gh run view` snapshot, or any check that confirms only that the workflow was triggered scores zero — the tile requires watching the run through to completion, not a fire-and-forget trigger confirmation" - }, - { - "name": "Conjunction check: run-success AND registry-advance AND moderation-clear", - "max_score": 15, - "description": "Gates final success on ALL THREE conjuncts: the resolved run's `conclusion` is `success`, the registry's Latest Version is strictly greater than the captured pre-merge baseline, AND the published version's moderation state has cleared (a freshly published version can be install-blocked until moderation reaches `pass`, so the script waits for the moderation clear — exponential backoff, fail loud at budget). Scoring zero: (a) accepting run-success alone as proof the release landed, (b) accepting registry-advance alone without checking the run's conclusion, (c) deriving an expected version from the merge SHA's manifest and comparing against it, or (d) treating a published-but-unmoderated version as confirmed — omitting the moderation-clear wait, or inventing a moderation state instead of querying the registry's real one. Any of (a)–(d) scores zero" - }, - { - "name": "Final summary includes merged PR URL", - "max_score": 4, - "description": "The script prints a final summary naming the merged PR URL and the publish-landed outcome (run conclusion + registry version), so the developer doesn't have to re-check manually" - }, - { - "name": "Graceful failure on unmet preconditions", - "max_score": 3, - "description": "When CI is red, reviews are outstanding, the publish run fails, or the registry doesn't advance, the script exits non-zero with a stderr diagnostic naming the specific unmet condition — does NOT silently skip or mask the failure" - }, - { - "name": "No hardcoded PR, owner, or repo in the script body", - "max_score": 2, - "description": "OBSERVABLE: the script body contains no literal for the target PR number, owner, or repo — all three come from runtime inputs (args or env vars). Running against a new target requires changing only the invocation, not the script source. Pinning any of them in the script source scores zero" - } - ] -} diff --git a/evals/pr-merge-and-post-merge-cleanup/task.md b/evals/pr-merge-and-post-merge-cleanup/task.md deleted file mode 100644 index 9e0e8b51..00000000 --- a/evals/pr-merge-and-post-merge-cleanup/task.md +++ /dev/null @@ -1,22 +0,0 @@ -# PR Merge and Branch Cleanup Automation - -## Problem/Feature Description - -A platform team manages a monorepo where several developers merge PRs throughout the day. After a merge, developers frequently forget to sync their local main branch, leave stale local branches around, and walk away without confirming the release actually published. This leads to subtle issues: developers push new work on top of outdated local main, accidentally re-create deleted branches, and only notice the release never went out hours later when a downstream consumer reports it missing. - -The team wants a `merge-and-cleanup.sh` bash script that a developer can run once a PR is ready to land. The script should handle the merge itself, clean up the local and remote state, and confirm the release actually published — giving the developer a clear final status rather than leaving them to check manually. - -## Output Specification - -Produce a `merge-and-cleanup.sh` bash script that: -- Accepts the PR number, repository owner, and repository name as inputs (arguments or environment variables) -- Checks that the PR is ready to merge before proceeding -- Merges the PR -- Brings the local main branch up to date -- Removes the local feature branch -- Cleans up stale remote-tracking references -- Verifies the merge landed on main -- Confirms the release was actually published successfully before exiting -- Prints a final summary including the merged PR URL - -The script does not need to run successfully (no GitHub credentials available) — focus on correctness of the approach and commands used. diff --git a/evals/review-feedback-addressing-conventions/criteria.json b/evals/review-feedback-addressing-conventions/criteria.json deleted file mode 100644 index e54f86ee..00000000 --- a/evals/review-feedback-addressing-conventions/criteria.json +++ /dev/null @@ -1,51 +0,0 @@ -{ - "context": "Tests whether the agent applies this tile's prescribed review-reply conventions — not whether a baseline agent reasons to some equivalent reply. The tile invents specific short phrases (`Fixed in ` for accepts, `Declining — ` for declines) so review threads scan consistently across a team. Baseline agents would produce ad-hoc replies (`done`, `no thanks`, `I disagree`). The specific-format criteria below measure whether the tile's convention was applied; the substance criteria measure whether the reply has the content the convention is meant to carry.", - "type": "weighted_checklist", - "checklist": [ - { - "name": "CI failure: fix required", - "max_score": 10, - "description": "For the CI failure (failing unit test), the guide states the test must be fixed. Does NOT suggest skipping, ignoring, deferring, or `[skip ci]`" - }, - { - "name": "Applies the reasonable suggestion", - "max_score": 10, - "description": "For the clearly-correct suggestion (simplify the nested conditional), the guide says to apply it — does NOT suggest declining or deferring" - }, - { - "name": "Declines the over-engineered suggestion", - "max_score": 10, - "description": "For the over-engineered suggestion (unnecessary abstraction for current scope), the guide says to push back — does NOT say to blindly implement it" - }, - { - "name": "All three threads get replies", - "max_score": 12, - "description": "The guide produces a reply for EVERY thread (CI failure, accept, decline) — not two of three. An empty reply or 'no response needed' on any thread scores zero" - }, - { - "name": "Accept reply uses the `Fixed in ` format", - "max_score": 15, - "description": "The accept reply follows the tile's prescribed format — it begins with the literal word 'Fixed in' followed by a sha (either a real commit hash or, per the task's placeholder instruction, the literal `` marker). A reply that says 'Done' or 'Resolved' or 'Applied the fix' scores zero even if semantically equivalent, because the tile exists to standardize the specific phrasing across the team" - }, - { - "name": "Decline reply uses the `Declining — ` format", - "max_score": 15, - "description": "The decline reply follows the tile's prescribed format — it begins with the literal word 'Declining', an em dash (—), then a reason (either prose or, per the task's placeholder instruction, the literal `` marker). A reply that says 'Disagree' or 'No thanks' or 'Not doing this' scores zero — the tile prescribes this specific opening precisely so declines are unambiguous in thread scans" - }, - { - "name": "Decline reply cites a verifiable reference", - "max_score": 8, - "description": "The decline reply names at least one externally-verifiable reference for the rejection: a file:line, quoted spec/docs clause, linked prior PR/issue, or named scope constraint from the PR description. This complements the format check — the format alone is not enough; the tile also requires substance" - }, - { - "name": "Fixes pushed to the same branch", - "max_score": 10, - "description": "The guide states any code changes go on the EXISTING feature branch of the PR, not a new branch. Opening a second PR for review fixes scores zero" - }, - { - "name": "No dangling threads before merge", - "max_score": 10, - "description": "The guide explicitly states that EVERY thread must have a reply before the PR is merged — nothing left unresolved" - } - ] -} diff --git a/evals/review-feedback-addressing-conventions/task.md b/evals/review-feedback-addressing-conventions/task.md deleted file mode 100644 index 087a3b04..00000000 --- a/evals/review-feedback-addressing-conventions/task.md +++ /dev/null @@ -1,19 +0,0 @@ -# Code Review Response Guide - -## Problem/Feature Description - -A junior engineer on your team just had their first PR reviewed by an automated code reviewer and doesn't know the team's conventions for responding. The reviewer left three types of feedback: one CI failure (a failing unit test), one suggestion that is clearly correct and should be accepted (simplifying an overly nested conditional), and one suggestion that over-engineers the solution by adding an abstraction layer that isn't needed for the current scope. - -The engineer needs a concrete, step-by-step guide explaining exactly how to handle each type of feedback, including how to respond to review threads so nothing is left unresolved. They also want to know: if they make code changes to address the feedback, should those changes go on a new branch or the existing one? - -The team cares about substantive replies — accepted suggestions should point concretely at the fix (so a reader can find it), and declined suggestions should cite concrete evidence (so the reviewer can verify the reasoning). - -## Output Specification - -Produce a document named `review-response-guide.md` that walks through how to handle the three feedback types described above. For each type, include: - -- What action to take on the code (if any) -- A sample reply the engineer would post in the review thread — concrete enough that the reviewer can either find the fix (for accepts) or verify the reasoning (for declines) -- Where to push any code changes - -Use placeholder values like `` or `:` where the actual values would depend on the situation. diff --git a/evals/rule-frontmatter-conditional-rule/criteria.json b/evals/rule-frontmatter-conditional-rule/criteria.json deleted file mode 100644 index 3b07b9ea..00000000 --- a/evals/rule-frontmatter-conditional-rule/criteria.json +++ /dev/null @@ -1,36 +0,0 @@ -{ - "context": "Tests whether the agent applies this plugin's frontmatter convention for a conditional rule whose prescriptions only fire in specific files. The plugin prescribes: alwaysApply: false in the rule file frontmatter, plus an applyTo: field combining glob patterns (matching common dependency manifests like package.json, pyproject.toml, Cargo.toml, go.mod) with a natural-language clause introduced by an em dash. In the plugin manifest form, scope lives only in the rule file frontmatter — .tessl-plugin/plugin.json lists rule paths and carries no per-rule config. Baseline agents without the plugin typically default to alwaysApply: true on every rule (matching the only convention they've seen) and do not know about the applyTo glob+prose pattern. Even a baseline agent that does scope the rule typically omits the natural-language clause or splits the scope across multiple fields.", - "type": "weighted_checklist", - "checklist": [ - { - "name": "Rule file frontmatter declares alwaysApply: false", - "max_score": 16, - "description": "The rules/pin-dependencies.md file has a YAML frontmatter block at the top with alwaysApply: false. Scores zero if frontmatter is missing, if alwaysApply is true, or if alwaysApply is omitted from the block. Baseline agents commonly default to alwaysApply: true" - }, - { - "name": "Rule file frontmatter declares applyTo with glob patterns", - "max_score": 22, - "description": "The rules/pin-dependencies.md frontmatter contains an applyTo field (or accepted alias: globs, paths) whose value includes glob patterns matching at least two common dependency manifest filenames (e.g., package.json, pyproject.toml, Cargo.toml, go.mod, Gemfile, composer.json). Scores partial credit (12) if applyTo is present but only matches one manifest type; scores zero if the field is missing or contains no glob patterns" - }, - { - "name": "applyTo value combines globs with a natural-language clause", - "max_score": 20, - "description": "The applyTo value includes both a glob list and a natural-language clause separated by a literal em dash (—, U+2014), e.g., 'package.json, pyproject.toml — when editing dependency manifests'. The clause expresses the action-level scope in prose alongside the file-level glob scope. Scores zero if the value is glob-only with no prose, prose-only with no globs, or uses a different separator (hyphen, double hyphen, en dash) where the rule prescribes an em dash" - }, - { - "name": "plugin.json rules array includes the new rule path", - "max_score": 16, - "description": "The updated .tessl-plugin/plugin.json rules array includes the path rules/pin-dependencies.md. Scores zero if the path is missing or wrong, or if the agent adds a steering map or a per-rule alwaysApply field to the manifest — that is the legacy tile.json model; plugin.json lists rule paths only and the scope lives in the rule file frontmatter" - }, - { - "name": "Rule body has H1 title matching the filename concept", - "max_score": 10, - "description": "The rule body starts with an H1 heading like # Pin Dependencies (or close equivalent) matching the filename. Scores zero if there is no H1 or it is unrelated to the file's concept" - }, - { - "name": "Existing rules and manifest entries are preserved unchanged", - "max_score": 16, - "description": "The two pre-existing rule paths (rules/commit-conventions.md, rules/spaces-not-tabs.md) remain in the plugin.json rules array, and the inputs/rules/spaces-not-tabs.md file is not modified. Scores zero if any pre-existing path is dropped or a pre-existing rule file is altered" - } - ] -} diff --git a/evals/rule-frontmatter-conditional-rule/task.md b/evals/rule-frontmatter-conditional-rule/task.md deleted file mode 100644 index cf752520..00000000 --- a/evals/rule-frontmatter-conditional-rule/task.md +++ /dev/null @@ -1,37 +0,0 @@ -# Add a Dependency-Pinning Rule to a Tessl Plugin - -## Problem/Feature Description - -A small team maintains a Tessl plugin that ships coding rules to their AI agents. They want to add a new rule that requires dependency versions to be pinned (no floating ranges, no wildcard versions). The rule is only relevant when contributors are working with the project's dependency configuration — adding a new dependency, updating an existing one, or removing one. Outside of that context the rule has nothing to say. Add the rule to the existing plugin so it surfaces to agents when contributors touch dependency configuration but stays out of the way otherwise. - -## Output Specification - -Produce: -- A new `rules/pin-dependencies.md` rule file with the rule body -- An updated `.tessl-plugin/plugin.json` that ships the new rule - -## Input Files - -The following files are the current state of the plugin. Extract them before beginning. - -=============== FILE: inputs/.tessl-plugin/plugin.json =============== -{ - "name": "acme/coding-rules", - "version": "0.3.0", - "description": "Acme's coding rules for AI agents", - "private": false, - "rules": [ - "rules/commit-conventions.md", - "rules/spaces-not-tabs.md" - ] -} - -=============== FILE: inputs/rules/spaces-not-tabs.md =============== ---- -alwaysApply: true ---- - -# Spaces Not Tabs - -- Use spaces for indentation, not tabs -- Indent width follows the project's editor configuration diff --git a/evals/rule-frontmatter-mixed-scope-no-narrow/criteria.json b/evals/rule-frontmatter-mixed-scope-no-narrow/criteria.json deleted file mode 100644 index c48d2099..00000000 --- a/evals/rule-frontmatter-mixed-scope-no-narrow/criteria.json +++ /dev/null @@ -1,36 +0,0 @@ -{ - "context": "Tests whether the agent correctly resists path-scoping a rule whose prescriptions span multiple contexts. The rule has two halves: (a) 'prefer stdlib over third-party' fires when writing code, (b) 'pin if you add it' fires when editing dependency manifests. Path-scoping the rule to manifests only (a tempting move because the sibling rule pin-dependencies.md is already scoped that way) would silently drop half (a) — the stdlib-first guidance would no longer fire when the agent is writing code. In the plugin manifest form, scope lives only in the rule file frontmatter; .tessl-plugin/plugin.json lists rule paths and carries no per-rule config. The plugin prescribes keeping mixed-scope rules universal (alwaysApply: true, no applyTo) so every prescription continues to fire in its native context. Baseline agents may either (1) default everything to alwaysApply: true and miss the deliberate decision, or (2) copy the pin-dependencies pattern and over-scope to manifests, which is the failure mode this rule guards against.", - "type": "weighted_checklist", - "checklist": [ - { - "name": "Rule file frontmatter declares alwaysApply: true", - "max_score": 24, - "description": "The rules/stdlib-first.md file has a YAML frontmatter block with alwaysApply: true. The rule must stay universal because part (a) fires when writing code, not when editing manifests. Setting alwaysApply: false here (matching the pin-dependencies pattern) is the failure this scenario tests for — it would scope the rule to manifests only and silently drop the stdlib-first guidance when the agent is writing code" - }, - { - "name": "Rule file frontmatter declares no scoping fields", - "max_score": 24, - "description": "The rules/stdlib-first.md frontmatter does NOT contain applyTo, globs, or paths fields. Any scoping field narrows the rule's activation; with mixed-scope content, scoping to one context drops the other. The plugin prescribes omitting scoping fields when the rule's prescriptions span multiple contexts" - }, - { - "name": "plugin.json rules array includes the new rule path", - "max_score": 14, - "description": "The updated .tessl-plugin/plugin.json rules array includes the path rules/stdlib-first.md. Scores zero if the path is missing or wrong, or if the agent adds a steering map or a per-rule alwaysApply field to the manifest — that is the legacy tile.json model; plugin.json lists rule paths only" - }, - { - "name": "Rule body covers the stdlib-first practice", - "max_score": 10, - "description": "The rule body has at least one bullet covering the 'prefer stdlib over third-party' practice — this is the half that fires when writing code. Scores zero if the body only covers pinning and omits the stdlib-first guidance" - }, - { - "name": "Rule body covers the dependency-pinning practice", - "max_score": 10, - "description": "The rule body has at least one bullet covering the 'pin if you add it' practice — the manifest-level half. Scores zero if the body only covers stdlib-first and omits the pinning guidance" - }, - { - "name": "Existing rules and manifest entries are preserved unchanged", - "max_score": 18, - "description": "The three pre-existing rule paths (rules/commit-conventions.md, rules/spaces-not-tabs.md, rules/pin-dependencies.md) remain in the plugin.json rules array, and the inputs/rules/pin-dependencies.md file is not modified (its alwaysApply: false and applyTo must not be flipped or removed). Scores zero if any pre-existing path is dropped or the sibling rule file is altered" - } - ] -} diff --git a/evals/rule-frontmatter-mixed-scope-no-narrow/task.md b/evals/rule-frontmatter-mixed-scope-no-narrow/task.md deleted file mode 100644 index dbf0fb62..00000000 --- a/evals/rule-frontmatter-mixed-scope-no-narrow/task.md +++ /dev/null @@ -1,45 +0,0 @@ -# Add a Stdlib-First Rule to a Tessl Plugin - -## Problem/Feature Description - -A small team maintains a Tessl plugin that ships coding rules to their AI agents. They want to add a new rule that captures two related practices their senior engineers keep reinforcing in code review: - -1. When writing code, prefer the standard library over adding a third-party dependency. If a standard-library solution covers the case, use it. -2. When a third-party dependency genuinely is needed and gets added, it should be pinned to a specific version in the project's manifest rather than left to float. - -Both halves live in one rule because the engineers want them surfaced together — the "prefer stdlib" practice is what prevents most dependency additions from happening in the first place, and the "pin it if you add it" practice handles the cases that get through. Add the rule to the existing plugin. - -## Output Specification - -Produce: -- A new `rules/stdlib-first.md` rule file with the rule body covering both practices -- An updated `.tessl-plugin/plugin.json` that ships the new rule - -## Input Files - -The following files are the current state of the plugin. Extract them before beginning. - -=============== FILE: inputs/.tessl-plugin/plugin.json =============== -{ - "name": "acme/coding-rules", - "version": "0.4.0", - "description": "Acme's coding rules for AI agents", - "private": false, - "rules": [ - "rules/commit-conventions.md", - "rules/spaces-not-tabs.md", - "rules/pin-dependencies.md" - ] -} - -=============== FILE: inputs/rules/pin-dependencies.md =============== ---- -alwaysApply: false -applyTo: "**/package.json, **/pyproject.toml, **/Cargo.toml, **/go.mod — when editing dependency manifests" ---- - -# Pin Dependencies - -- Pin every third-party dependency to a specific version -- Avoid floating ranges and wildcards -- Lock files are committed alongside the manifest diff --git a/evals/rule-frontmatter-tile-rule-agreement/criteria.json b/evals/rule-frontmatter-tile-rule-agreement/criteria.json deleted file mode 100644 index 0195abf0..00000000 --- a/evals/rule-frontmatter-tile-rule-agreement/criteria.json +++ /dev/null @@ -1,36 +0,0 @@ -{ - "context": "Tests whether the agent converts a universal rule to a conditional one entirely in the rule file frontmatter: flip alwaysApply from true to false AND add an applyTo field with the glob+prose em-dash pattern. In the plugin manifest form, scope lives only in the rule file frontmatter — .tessl-plugin/plugin.json lists rule paths and carries no per-rule config, so the conversion does not touch the manifest. Baseline agents typically add applyTo but forget to flip the existing alwaysApply: true (leaving the rule universal), omit the natural-language clause, or — carrying the legacy tile.json mental model — try to flip a non-existent manifest scope field or inject a steering map into plugin.json. The plugin prescribes the frontmatter-only conversion with the manifest left untouched.", - "type": "weighted_checklist", - "checklist": [ - { - "name": "Rule file frontmatter flipped to alwaysApply: false", - "max_score": 22, - "description": "The rules/pr-template-checklist.md frontmatter's alwaysApply value is changed from true to false. Scores zero if the value remains true (the rule stays universal — the most common baseline failure) or if the field is removed entirely" - }, - { - "name": "Rule file frontmatter gains applyTo with glob patterns", - "max_score": 22, - "description": "The rules/pr-template-checklist.md frontmatter contains a new applyTo field (or accepted alias: globs, paths) whose value includes glob patterns matching PR-related artifacts. The patterns must include at least one match for `.github/PULL_REQUEST_TEMPLATE` (with or without the `.md` extension, with or without the directory variant). Scores zero if the field is missing or contains no glob patterns at all; partial credit (12) if globs are present but do not match PR templates specifically" - }, - { - "name": "applyTo value combines globs with a natural-language clause", - "max_score": 18, - "description": "The applyTo value includes both a glob list and a natural-language clause separated by a literal em dash (—, U+2014), e.g., '.github/PULL_REQUEST_TEMPLATE*, CONTRIBUTING.md — when authoring or editing PR artifacts'. The clause expresses the action-level scope in prose. Scores zero if the value is glob-only, prose-only, or uses a different separator (hyphen, double hyphen, en dash) where the rule prescribes an em dash" - }, - { - "name": "plugin.json carries no per-rule config and its rules array is intact", - "max_score": 14, - "description": "The .tessl-plugin/plugin.json rules array still lists rules/pr-template-checklist.md (and rules/commit-conventions.md), and the agent does NOT add a steering map or any per-rule alwaysApply/applyTo field to the manifest. Scope is declared only in the rule file frontmatter. Scores zero if the agent injects manifest-level scope (the legacy tile.json model) or drops the rule path from the array" - }, - { - "name": "Rule body content is preserved unchanged", - "max_score": 12, - "description": "The body of rules/pr-template-checklist.md (everything after the frontmatter block) is byte-identical to the input. The scenario is a frontmatter-only edit; modifying the body bullets is out of scope and counts against the agent" - }, - { - "name": "Existing rule (commit-conventions) is preserved unchanged", - "max_score": 12, - "description": "The rules/commit-conventions.md path remains in the plugin.json rules array. Scores zero if it is dropped or renamed" - } - ] -} diff --git a/evals/rule-frontmatter-tile-rule-agreement/task.md b/evals/rule-frontmatter-tile-rule-agreement/task.md deleted file mode 100644 index f7a64db5..00000000 --- a/evals/rule-frontmatter-tile-rule-agreement/task.md +++ /dev/null @@ -1,37 +0,0 @@ -# Convert a Universal Rule to a Conditional Rule - -## Problem/Feature Description - -A small team maintains a Tessl plugin that ships coding rules to their AI agents. One of their existing rules, `pr-template-checklist`, is currently marked to apply across every context. After several months of using it, the team realizes the rule only meaningfully fires when contributors are authoring or editing pull-request-related artifacts — PR description templates, contributor guides, the PR body itself when reviewing a draft. Outside of that context the rule has nothing to say, and surfacing it in every conversation is noise. Convert the rule from a universal rule into a conditional rule scoped to PR-related artifacts. - -## Output Specification - -Modify `rules/pr-template-checklist.md` — update its frontmatter so the rule fires only on PR-related artifacts. Leave the rule body content unchanged. - -## Input Files - -The following files are the current state of the plugin. Extract them before beginning. - -=============== FILE: inputs/.tessl-plugin/plugin.json =============== -{ - "name": "acme/coding-rules", - "version": "0.5.0", - "description": "Acme's coding rules for AI agents", - "private": false, - "rules": [ - "rules/commit-conventions.md", - "rules/pr-template-checklist.md" - ] -} - -=============== FILE: inputs/rules/pr-template-checklist.md =============== ---- -alwaysApply: true ---- - -# PR Template Checklist - -- Every PR template includes a "summary" section and a "test plan" section -- The summary names the change and the motivation -- The test plan lists verification steps the reviewer can execute -- Templates live in `.github/PULL_REQUEST_TEMPLATE.md` or `.github/PULL_REQUEST_TEMPLATE/*.md` diff --git a/evals/version-bump-reasoning-and-manifest-upda/criteria.json b/evals/version-bump-reasoning-and-manifest-upda/criteria.json deleted file mode 100644 index bfada154..00000000 --- a/evals/version-bump-reasoning-and-manifest-upda/criteria.json +++ /dev/null @@ -1,46 +0,0 @@ -{ - "context": "Tests whether the agent applies this tile's prescribed release sequencing and version-bump policy. The tile prescribes: patch bumps are delegated to the CI pipeline (`tesslio/patch-version-publish` auto-bumps the patch segment on every merge) so manual manifest edits for patch changes are wrong; minor and major bumps require manual manifest updates to specific target numbers; breaking changes need explicit impact surfacing in the runbook; PRs should be sequenced patch → minor → major for risk management. Baseline agents typically bump all three manually and do not sequence the releases.", - "type": "weighted_checklist", - "checklist": [ - { - "name": "Patch: no manual manifest update", - "max_score": 12, - "description": "For the bug-fix change (null-pointer guard, backward-compat, no API change), the runbook does NOT prescribe a manual manifest version bump. Baseline agents will typically bump it manually; the tile prescribes delegating to CI" - }, - { - "name": "Patch: explains CI auto-bump", - "max_score": 10, - "description": "The runbook explains WHY the patch case skips a manual bump — because the CI pipeline (`tesslio/patch-version-publish` or equivalent) auto-bumps the patch segment on every merge. Mentioning the specific automation or its role is tile-prescribed knowledge; a baseline agent without the tile would not know this wiring exists" - }, - { - "name": "Minor: manifest bumped to `1.5.0`", - "max_score": 12, - "description": "For the CSV-export change (additive new route, backward-compat), the runbook updates the manifest version to `1.5.0` (not `1.4.3`, not `2.0.0`). Scores correct classification (minor) AND correct arithmetic" - }, - { - "name": "Major: manifest bumped to `2.0.0`", - "max_score": 12, - "description": "For the v1-routes removal (breaking change), the runbook updates the manifest version to `2.0.0` (not `1.4.3`, not `1.5.0`). Scores correct classification (major) AND correct arithmetic" - }, - { - "name": "Major: flags breaking-change impact for downstream", - "max_score": 12, - "description": "The runbook calls out that the v1-routes removal will break existing clients AND surfaces migration guidance in a place a downstream consumer would see (release notes, migration guide, or a deployment checklist). An agent who treats the major bump the same way as the minor bump — no impact statement, no migration — scores zero" - }, - { - "name": "Release sequencing: patch first, major last", - "max_score": 12, - "description": "The runbook orders the three PRs patch → minor → major with explicit rationale (patch is low-risk and gets CI's auto-bump first; major gives downstream the longest notice and comes last). Any coherent risk-based sequencing with stated reasoning passes; no sequencing, or reversing without justification, scores zero. Baseline agents typically do not sequence" - }, - { - "name": "Readiness gate: tests + linter", - "max_score": 10, - "description": "The runbook requires tests AND linter to pass before creating any of the three PRs. Scored once as a cross-cutting gate, not duplicated per change" - }, - { - "name": "Runbook covers all three changes separately", - "max_score": 20, - "description": "The `RELEASE_RUNBOOK.md` has a distinct section for each of the three pending changes, each answering the readiness / versioning / sequencing questions in the task's output spec. A single conflated section scores zero. Only two of three sections present scores two-thirds" - } - ] -} diff --git a/evals/version-bump-reasoning-and-manifest-upda/task.md b/evals/version-bump-reasoning-and-manifest-upda/task.md deleted file mode 100644 index 2a7ce745..00000000 --- a/evals/version-bump-reasoning-and-manifest-upda/task.md +++ /dev/null @@ -1,47 +0,0 @@ -# Release Runbook for a Multi-Change Sprint - -## Problem/Feature Description - -A small open-source library team is wrapping up a sprint that produced three distinct categories of changes that need to ship as separate PRs. The team is inconsistent about when and how to update version numbers, and keeps picking wrong target versions or bumping the manifest redundantly. The team lead wants a `RELEASE_RUNBOOK.md` that documents the versioning decision and the release sequencing for each type of change, so contributors stop guessing. - -## Output Specification - -Produce a `RELEASE_RUNBOOK.md` file that covers, for each of the three change types below: -- Pre-PR readiness steps (what must be verified before creating the PR) -- Whether the project manifest's version field should be updated, and if so, to what target version number (the current version is in the manifest fixture below) -- A reasoned recommendation for the order in which the three PRs should ship, given the relative risk and downstream impact - -You may also produce a `release.sh` helper script if you find it useful to illustrate the steps programmatically. - -## Input Files - -The following files represent the three pending changes to release. Extract them before beginning. - -=============== FILE: inputs/changes.md =============== -# Pending Changes - -## Change A — Fix null pointer in user lookup -- **File modified**: `src/users/lookup.js` -- **What it does**: Guards against a null return value from the database adapter that was causing crashes in production -- **Backward compatible**: Yes -- **New public API**: No - -## Change B — Add CSV export endpoint -- **File modified**: `src/export/csv.js` (new file), `src/routes/index.js` -- **What it does**: Adds a new `/export/csv` route that lets users download their data -- **Backward compatible**: Yes (additive) -- **New public API**: Yes — new route and response format - -## Change C — Remove deprecated `v1` API routes -- **File modified**: `src/routes/v1/` (deleted), `src/routes/index.js` -- **What it does**: Drops all `/v1/*` endpoints that were deprecated 6 months ago -- **Backward compatible**: No — existing clients using `/v1/` will break -- **New public API**: No - -=============== FILE: inputs/package.json =============== -{ - "name": "data-service", - "version": "1.4.2", - "description": "A small data management library", - "main": "src/index.js" -} diff --git a/pyrightconfig.json b/pyrightconfig.json index 1606460e..802bf50e 100644 --- a/pyrightconfig.json +++ b/pyrightconfig.json @@ -1,7 +1,5 @@ { "include": [ - "skills/eval-curation/compute-lift.py", - "skills/eval-curation/tests/test_compute_lift.py", "skills/release/stamp-changelog.py", "skills/release/tests/test_stamp_changelog.py" ], diff --git a/rules/context-artifacts.md b/rules/context-artifacts.md index f0a79ad9..967d51c0 100644 --- a/rules/context-artifacts.md +++ b/rules/context-artifacts.md @@ -1,6 +1,6 @@ --- alwaysApply: false -applyTo: ".tessl-plugin/plugin.json, rules/**, skills/**, evals/**, .tesslignore, CHANGELOG.md, README.md — when authoring or modifying plugin artifacts" +applyTo: ".tessl-plugin/plugin.json, rules/**, skills/**, .tesslignore, CHANGELOG.md, README.md — when authoring or modifying plugin artifacts" description: Plugin structure, rule/skill format, review pipeline, surface sync, consistency audits — the authoring contract for Tessl plugins --- @@ -12,7 +12,7 @@ description: Plugin structure, rule/skill format, review pipeline, surface sync, - The plugin's `README.md` is the project's `README.md` — same file. Extend the existing README with rules table, skills table, and installation instructions - Include a Tessl registry badge at the top of README: `[![tessl](https://img.shields.io/endpoint?url=https%3A%2F%2Fapi.tessl.io%2Fv1%2Fbadges%2F%2F)](https://tessl.io/registry//)` - Skills live in `skills//SKILL.md`, rules live in `rules/.md` -- Standard directories: `rules/`, `skills//`, `evals/` +- Standard directories: `rules/`, `skills//` - Use `.tesslignore` to exclude build artifacts and CI files from the published plugin - Validate structure with `tessl plugin lint` before every publish - `CHANGELOG.md` and similar repo files show as orphaned in `tessl plugin lint` — lint only tracks manifest-declared paths @@ -59,12 +59,6 @@ description: Plugin structure, rule/skill format, review pipeline, surface sync, - `--optimize` is a diagnostic signal, not a patch. The reviewer's judge strips load-bearing context. Diff against the backup, keep the genuinely-improving moves (tighter triggers, less prose, better `Skill()` typing), reject over-aggressive cuts, then re-run the review and iterate - Shipping `--optimize` output verbatim is forbidden even when the score improved. The optimizer surfaces what kinds of issues exist (actionability, progressive disclosure, redundancy); apply the signal with judgment, curate manually -## Mandatory Evals - -- Eval scenario requirements live in `rules/plugin-evals.md` Coverage -- No bleeding, no leaking — full guardrails in `rules/plugin-evals.md` -- Process details live in the `eval-authoring` skill — invoke it to generate and curate scenarios - ## Surface Sync When you add, remove, or rename a rule or skill, update **all** of these: diff --git a/rules/context-writing-style.md b/rules/context-writing-style.md index 6ad6e28d..eb7f842a 100644 --- a/rules/context-writing-style.md +++ b/rules/context-writing-style.md @@ -15,7 +15,7 @@ description: Prose discipline for rules, skills, and READMEs — what to cut, wh - CHANGELOG entries load only on demand, not always-on - CHANGELOG entries are the archive — they carry the motivation and incident references stripped from rules, and follow looser discipline - `What to Cut` and `Structure` do not govern CHANGELOG. `What to Cut` names CHANGELOG as where the stripped explanation goes; a connective ban there would leave it nowhere to live. The rule reaches CHANGELOG only to say what belongs in it -- Line-count and section-count budgets target single-concept rules; rules that cover one coherent policy area with several sub-aspects (e.g., `plugin-evals` covering coverage/lift/persistence/naming/hygiene; `context-artifacts` covering structure/review/sync/audit; `skill-authoring` covering frontmatter/preamble/steps/calls) may run larger +- Line-count and section-count budgets target single-concept rules; rules that cover one coherent policy area with several sub-aspects (e.g., `context-artifacts` covering structure/review/sync/audit; `skill-authoring` covering frontmatter/preamble/steps/calls) may run larger - Lists and quoted literals naming the forbidden terms themselves (e.g., this rule's own bullets enumerating the connectives and intensifiers) are not violations — the directive is the list, not prose use of the listed words ## What to Cut diff --git a/rules/plugin-evals.md b/rules/plugin-evals.md deleted file mode 100644 index b5a12f37..00000000 --- a/rules/plugin-evals.md +++ /dev/null @@ -1,89 +0,0 @@ ---- -alwaysApply: false -applyTo: "evals/**, skills/**/SKILL.md — when authoring or maintaining eval scenarios" -description: Eval coverage, lift-not-attainment scoring, no bleeding, no leaking, fixture hygiene ---- - -# Plugin Evals - -## Coverage - -- A scenario earns its place only by demonstrated lift on the floor model (see `Lift, Not Attainment`). Proving plugin value is the goal, not coverage -- Do NOT write one scenario per prescribed behavior, pad a suite to "cover" a skill, or add a scenario you have not lift-checked. Absence is correct when nothing clears the bar -- Admission gate, not a curation afterthought: run a candidate once (baseline + with-context) before committing it; a flat result keeps it out of the suite -- Default cap of 3 scenarios per skill. Exceeding it requires justifying that each added scenario clears the lift bar AND tests a distinct plugin-prescribed behavior -- The cap is a prospective admission gate on newly added scenarios; existing suites already over 3 stay governed by `Lift, Not Attainment` curation, not forced truncation -- Scope to genuine LLM-side judgment: a skill whose decisional core is a unit-tested script has no LLM-side surface to eval. Eval only the judgment the plugin prescribes that the script does not make -- Negative cases only where the skill has a refusal or silence contract; write them by hand (`tessl scenario generate` skews toward happy-path) - -## Task and Criteria: the load-bearing shape - -- **Task** describes the SITUATION — what the user needs done. It does NOT prescribe the technique, format, sequence, or specific manner of solving it. "Ship a hotfix" is a task; "Ship a hotfix using a feature branch named `fix/*`" is a task with the answer smuggled in -- **Criteria** grade whether the output matches the specific manner this plugin prescribes. Checking for plugin-prescribed specifics (flag choices, format literals, sequences, conventions) is measuring plugin value, not testing reading - -## No Bleeding - -- Primary form: a criterion value appears verbatim in the task description. Grep each criterion's expected literal against the task text — if found there, the criterion tests reading, not application -- Fix bleeding at the task, not the criterion. Strip the leaked technique/format/literal from the task; keep the criterion. If stripping makes the task unsolvable for a baseline, the scenario is too narrow — reframe it -- Second form: fixtures matching examples inside the skill prompt. Keep fixtures in a namespace separate from skill examples - -## No Leaking - -- Use sanitized or synthetic fixtures — never live user data. Real emails, calendar events, production PRs, or internal logs must never appear in an eval fixture; use stable synthetic IDs and scrubbed examples -- Criteria must not reference plugin-internal implementation details that mean nothing outside the plugin — internal skill action names, `.tessl/plugins/...` paths, plugin-only identifiers -- Criteria **may** reference public tool/API surfaces that exist independent of the plugin — `gh pr create`, REST endpoints, conventional-commits format, semver -- Criteria may reference plugin-prescribed conventions and specific values such as reply templates like `Fixed in `, chosen flags like `--ff-only`, invented format literals. Checking for them measures application, not leaking -- Test: would someone outside the plugin recognize the term? `gh pr merge` is public; `createJwtToken` internal action is plugin-internal - -## Lift, Not Attainment - -- Lift = `with-context` score minus `baseline` score, where `with-context` is with the plugin loaded and `baseline` is without. Near-zero lift on a positive case has three causes: - 1. Coincidence with universal competence — plugin's prescribed manner matches what baseline agents already produce. Retire the scenario - 2. Task leaked the technique — fix the task per No Bleeding; keep the criterion - 3. Criteria grade universal competence — criteria test things baseline always does (basic git safety, obvious judgement) instead of plugin-specific choices. Rewrite the criteria or retire the scenario -- Always report per-scenario lift alongside the average -- High-lift scenarios test plugin-prescribed choices where baseline picks something different (bot-ID discovery, reply format, CLI sequence). Keep them — do not rewrite toward "testing reasoning" if baseline already reasons to the same outcome -- Pruning is mandatory upkeep, not optional cleanup -- Run the curation pass (see `skills/eval-curation/SKILL.md`) every few publishes -- Retire any scenario showing near-zero lift after the three-cause diagnosis and fix attempt -- Measure by per-scenario lift contribution, not raw scenario count. Most skills need 1–3 lift-bearing scenarios; many prescribed behaviors need zero, where baseline already produces the plugin's manner. A small suite where every scenario pulls weight beats a large one padded with baseline-equivalents - -## Quality - -- Failure messages must explain **what went wrong**, not just "mismatch" -- Criteria must be specific and weighted sensibly — vague criteria produce vague results -- Criteria must align with what the task actually asks for - -## Persistence - -- Tessl's publish pipeline runs the eval suite automatically when `tessl plugin publish` (or `tesslio/patch-version-publish`) executes — that is the persistence point -- Do not add a `tessl eval run` step to plugin-repo CI; do not add a scheduled or recurring workflow that re-runs the eval suite -- Out of scope: local invocations during authoring/debugging, and ad-hoc invocations by separate measurement rigs (e.g., `jbaruch/coding-policy-evals`) -- Regressions block the publish -- Fix the regression, or fix the scenario when the cause is fixture drift per `Fixture Hygiene` -- Do not add a parallel CI step that could mask the publish-layer failure - -## Naming - -- Scenario directory names use **kebab-case** — lowercase + hyphens only (`my-scenario`, not `MyScenario`, `my_scenario`, or `my scenario`) -- Skill-specific scenarios: prefix with the skill name — `-` (e.g., `install-reviewer-refuses-overwrite`, `eval-curation-task-leak-fix`) -- Cross-cutting scenarios: name the behavior directly without a skill prefix (e.g., `pr-merge-and-post-merge-cleanup`) -- Descriptors name the behavior under test, not the implementation: `refuses-overwrite` ✓, `checks-existing-file-via-stat` ✗ -- Default cap at 40 characters for the directory name — driven by tessl's interactive tooling (`tessl eval view`, `tessl scenario generate`) silently truncating longer names -- Cap applies prospectively; scenarios already run through `tessl eval run` that exceed it are grandfathered by the rename-stability clause (which wins over the cap) -- Once committed and run through `tessl eval run`, the name is stable — do not rename. `tessl eval view` identifies scenarios by directory name; renaming resets the lift history the Persistence section relies on -- Rig conventions (programmatic shapes like `---run-`) may diverge from kebab-case-with-descriptor when documented in the plugin's `evals/instructions.json` -- Narrow exception for rig-shaped scenarios that exceed the 40-char default cap -- Preconditions (all required): - 1. Rig's programmatic shape provably cannot fit 40 chars without losing reconstructability of its components (e.g., 4-tuple `rule`/`fixture_type`/`cell`/`run` needed by the rig's scorer for lift bucketing) - 2. `evals/instructions.json` declares the rig's actual safe length AND names every tessl-eval tool the rig touches end-to-end - 3. Rig bypasses the interactive tools that drive the default cap — scenario names built by custom script, runs invoked via `tessl eval run `, scoring via custom scorer - 4. Rig owns naming-collision and truncation-driven scoring drift in its own scorer, not in tessl's tooling - 5. Rig's CHANGELOG records the carve-out under a `### Rules` (or equivalent) entry citing this clause -- Rename-stability applies regardless of cap — `tessl eval view` identifies scenarios by directory name irrespective of how the name was generated -- Every scenario outside an active rig carve-out still respects the 40-char default - -## Fixture Hygiene - -- Version fixtures with dates in filenames (e.g., `fixture-2025-04-17.json`) -- Update fixtures when the skill's contract changes — stale fixtures produce false passes diff --git a/rules/rule-frontmatter.md b/rules/rule-frontmatter.md index 264482c0..a944c6d4 100644 --- a/rules/rule-frontmatter.md +++ b/rules/rule-frontmatter.md @@ -29,7 +29,7 @@ description: Frontmatter conventions for rule files — what to set for always-o ## When to Path-Scope -- Path-scope when **every** prescription in the rule is bound to a specific file set or context (plugin-authoring rules whose content only fires inside `rules/`, `skills/`, `evals/` are good candidates) +- Path-scope when **every** prescription in the rule is bound to a specific file set or context (plugin-authoring rules whose content only fires inside `rules/`, `skills/` are good candidates) - Stay universal when the rule mixes file-bound and broad guidance — `applyTo:` is exclusionary at the model layer, so a too-narrow scope drops the broad bits - Example — `dependency-management` covers two practices: - "stdlib first" fires when writing code diff --git a/rules/testing-standards.md b/rules/testing-standards.md index b3836120..d7a822f8 100644 --- a/rules/testing-standards.md +++ b/rules/testing-standards.md @@ -43,7 +43,7 @@ alwaysApply: true - Assertions compare against the pinned fixture dates — never against fresh future-date literals - Preconditions (all required): 1. Each pinned date is a literal in a versioned fixture — never computed from the run-time clock - 2. Fixtures follow `rules/plugin-evals.md` Fixture Hygiene — dates in filenames + 2. Fixtures carry the date in the filename (e.g., `fixture-2025-04-17.json`) 3. The owning repo documents the refresh cadence and refresh procedure beside the fixtures - A pinned date aging into the past is a fixture refresh under the documented cadence — never an inline patch during unrelated work - Every other suite still bans future dates and time-relative values diff --git a/skills/eval-authoring/LIFT_ANALYSIS.md b/skills/eval-authoring/LIFT_ANALYSIS.md deleted file mode 100644 index 3c9f5aef..00000000 --- a/skills/eval-authoring/LIFT_ANALYSIS.md +++ /dev/null @@ -1,51 +0,0 @@ -# Lift Analysis Reference - -Reference material for Step 9 of the `eval-authoring` skill. Pulled out of the SKILL.md body so the main workflow stays scannable. - -## What Lift Means - -`lift = with_context_score - baseline_score` - -Lift is the number that matters. Aggregate attainment on its own is a vanity metric — a plugin scoring 99% with-context and 73% baseline is contributing 26 points of real value, not 99. A 100/100 scenario with the plugin loaded but 100/100 baseline is delivering zero plugin value, no matter how good the score looks. - -## Measure on the Floor Model - -- Lift is model-dependent and often diverges — sometimes inverts — across solver strength: a scenario near-zero on a strong solver can deliver large lift on a weak one -- Measure lift on the **floor model** (the weakest agent in the consumer spectrum), not the strongest — a strong solver's baseline competence masks context the floor model actually needs -- coding-policy's floor is `glm-5.1` (the CI publish-run solver); a curation decision reached on a stronger solver such as `claude-opus-4-7` must be re-checked against the floor before any retire -- Retire only when lift is near-zero on the floor model too - -## Thresholds (Positive Cases) - -| Lift band | Verdict | Action | -|---|---|---| -| **≥ 40** | Healthy signal | The plugin is doing real work — usually checks specific plugin-prescribed choices (particular bot-ID discovery, particular reply template, particular CLI sequence). Keep these scenarios; do NOT soften criteria toward "testing reasoning" baseline already does | -| **10–39** | Weak | Audit for the three causes below — the scenario likely tests baseline competence rather than plugin value | -| **< 10** | No signal | Apply the three-cause diagnosis below before retiring; many "no lift" scenarios are actually "wrong target" | - -### Three causes for weak / no lift - -1. **Coincidence with universal competence** — the plugin's prescription *itself* is what baseline already produces; no more-specific plugin behaviour is being missed, so no replacement criterion can be salvaged. Retire the scenario as a null test. -2. **Task leaked the technique** — baseline pattern-matched the answer from the task text. Fix the task (strip the leaked literal); keep the criterion. -3. **Criteria grade universal competence** — the criteria test things baseline always does (basic git safety, obvious engineering judgement, general engineering-101) while the plugin prescribes a *more specific* behaviour the criteria failed to check. Rewrite the criteria to check that specific prescription (the exact reply template, the chosen flag, the named CLI sequence). - -**Cause #1 vs #3 discriminator** — both present as "baseline already does it," and conflating them sends the action the wrong way. Ask: does the plugin prescribe a more specific behaviour than the criteria currently grade, one baseline would not produce by default? - -- No salvageable plugin-specific replacement (the plugin's prescription *is* the universal behaviour — e.g., imperative-mood commit subjects) → cause #1 → `retire` -- A more-specific prescription exists and the criteria merely glossed it (e.g., a canary 10% / 15-min-bake sequence the criteria phrased as "mention deployment") → cause #3 → `rewrite-criteria` - -The salvageable-replacement test decides — there is no default action. - -## Negative Cases (Refusal-Based Scenarios) - -Near-zero lift on a negative case is acceptable **only** when the baseline refusal is driven by universal knowledge (obvious error cases — e.g., "do not push secrets," "do not merge red CI"). Plugin-specific refusal reasoning (e.g., "refuses to overwrite an existing reviewer workflow per the plugin's overwrite-guard") must still show lift; if it doesn't, apply the three-cause diagnosis above. - -## Diagnosing Where to Fix - -For each scenario with non-zero lift but with-context below 100%, identify the failing criteria and decide where the problem lives: - -- **The skill** — unclear instruction. The agent didn't apply the rule because the SKILL.md doesn't make the prescription explicit enough. Fix in `skills//SKILL.md`. -- **The task** — doesn't ask for what the criteria test. The criteria check for X but the task never gives the agent a reason to do X. Fix in `evals//task.md`. -- **The criteria** — tests the wrong thing. The criteria phrase the requirement as something baseline can satisfy without the plugin, or as something the plugin doesn't actually prescribe. Fix in `evals//criteria.json`. - -The skill / task / criteria triage is the most common reason a fix lands in the wrong place — re-asking "what artifact owns this concern?" before editing anything saves a round trip. diff --git a/skills/eval-authoring/REVIEW_CHECKLIST.md b/skills/eval-authoring/REVIEW_CHECKLIST.md deleted file mode 100644 index dde5f9f9..00000000 --- a/skills/eval-authoring/REVIEW_CHECKLIST.md +++ /dev/null @@ -1,41 +0,0 @@ -# Eval Scenario Review Checklist - -## Task and Criteria Shape - -Does the task describe a SITUATION or prescribe a TECHNIQUE? - -- **Correct**: task describes what the user needs done ("Ship a hotfix", "Wire up a reviewer"). Criteria check whether the output matches the specific manner the plugin prescribes — that conformance IS the plugin's contribution. -- **Wrong**: task names the technique, format, sequence, or literal the criterion grades ("Ship a hotfix using `--ff-only`"). The agent passes by reading the task, not by applying the plugin. - -## Bleeding - -Two forms: - -1. **Task/criterion overlap**: a criterion's expected literal appears verbatim in the task description. For each criterion with a concrete expected value, grep the task text for that literal — a match is bleeding. Fix by stripping the literal from the task and keeping the criterion. Baseline should still be able to attempt the situation (they'll just pick some other manner); if stripping the literal makes the task unsolvable even for a baseline, the scenario is too narrow to evaluate the plugin and should be reframed. -2. **Fixture reachable as a skill example**: the scenario's fixture is the same example the skill teaches with. The agent "passes" by recognizing the example, not by applying the lesson. Keep fixtures in a separate namespace from skill examples. - -## Leaking - -- **Privacy**: use sanitized or synthetic fixtures. Never live user data (real emails, calendar events, production PRs, internal logs). Use stable synthetic IDs and scrubbed examples — live-data fixtures drift silently and risk accidental exposure. -- **Plugin internals (leaking)**: criteria must not reference internal skill action names, `.tessl/plugins/...` paths, or plugin-only identifiers that mean nothing outside the plugin. -- **Public surfaces (allowed)**: `gh pr create`, REST endpoints, conventional-commits format, semver — these exist independent of the plugin. -- **Plugin-prescribed conventions (allowed)**: specific reply templates (`Fixed in `), chosen flags (`--ff-only`), invented format literals, specific sequences. A competent engineer without the plugin would not produce these specific choices — checking for them measures plugin value, not internal wiring. - -Ask: would someone outside the plugin recognize the term? If yes (public surface or plugin-prescribed convention), allowed. If no (plugin-internal), leaking. - -## Lift - -Every criterion's contribution is the delta between `with-context` and `baseline` scores. If lift is near-zero, diagnose: - -1. **Coincidence with universal competence**: the plugin prescribes what baseline already does (e.g., "imperative commits", "fix failing tests"). The rule codifies common practice. Retire or accept as documentation — no lift to win. -2. **Task leaked the technique**: baseline pattern-matched. Fix the task (see Bleeding), keep the criterion. -3. **Criteria grade universal competence**: testing engineering-101 rather than the plugin's specific prescribed manner. Rewrite the criteria to grade the specific convention the plugin teaches. - -High-lift scenarios typically check specific plugin-prescribed choices (a particular bot-ID discovery approach, a particular reply template, a particular CLI sequence). Keep these — do not soften them to "test reasoning" if baseline already reasons to the same outcome. - -## Quality - -- Every criterion `description` must explain what went wrong on failure — not just "mismatch" -- Criteria must be specific and weighted sensibly -- Weights should reflect importance, not equal distribution -- Every criterion must test something the task's output specification asks for; if the task doesn't mention it, the criteria shouldn't check for it diff --git a/skills/eval-authoring/SKILL.md b/skills/eval-authoring/SKILL.md deleted file mode 100644 index 30149a3f..00000000 --- a/skills/eval-authoring/SKILL.md +++ /dev/null @@ -1,98 +0,0 @@ ---- -name: eval-authoring -description: > - Generate, review, and curate eval scenarios for Tessl skills. Handles scenario - generation, bleeding/leaking detection, criteria quality checks, lift-gated - scenario admission, and score-driven iteration. - Use when creating test cases or test scenarios for a skill, evaluating or - assessing skill quality, running evals or evaluations, reviewing existing - evals, expanding eval coverage, or skill testing. ---- - -# Eval Authoring Skill - -Process steps in order. Do not skip ahead. - -Generate, review, and iterate on eval scenarios for a Tessl skill. The 10-step workflow: generate (1) and download (2–3) scenarios, audit each (4) for bleeding/leaking, fix (5) or delete (6) unsalvageable ones, add lift-bearing scenarios (7), run evals (8), interpret results via lift analysis (9), iterate until stable (10). - -## Step 1 — Generate Scenarios - -```bash -tessl scenario generate . -``` - -## Step 2 — Wait for Generation - -```bash -tessl scenario view -``` - -Poll until completed. If it fails, report the error and finish here. When status is completed, proceed immediately to Step 3. - -## Step 3 — Download Scenarios - -```bash -tessl scenario download --output evals -``` - -## Step 4 — Review Each Scenario - -For every scenario in `evals/`, read `task.md` and `criteria.json`. Check against `skills/eval-authoring/REVIEW_CHECKLIST.md`: does the task describe a situation without prescribing the technique? Do the criteria grade the specific manner the plugin prescribes (good) rather than restating literals from the task (bleeding)? Any plugin-internal leaks in the criteria? Are criteria values public surfaces, plugin-prescribed conventions (allowed — they measure plugin value), or plugin internals (leaking)? Any quality or consistency issues? - -If no issues found in a scenario, proceed silently to the next one. Proceed immediately to Step 5. - -## Step 5 — Fix Issues - -Edit `criteria.json` and `task.md` to remove bleeding, remove leaking, improve failure messages, and align criteria with task. See `skills/eval-authoring/REVIEW_CHECKLIST.md` for definitions. - -When a criterion is misaligned, leaking, or unsalvageable, remove it and reweight the remaining criteria so the checklist sums to 100 — do not keep a bad criterion to preserve existing weights. If removing the bad criterion leaves no plugin-specific signal, the scenario itself is unsalvageable — delete it per Step 6. - -## Step 6 — Delete Unsalvageable Scenarios - -Remove scenario directories that can't be fixed: task tests an internal detail, task is too vague, or fixing bleeding would rewrite the entire task. - -## Step 7 — Add Lift-Bearing Scenarios - -Read `rules/plugin-evals.md` Coverage for the admission criteria before adding anything. Procedurally: the Step 1 batch is a starting point, not a coverage target — do NOT enumerate a skill's behaviors and write a scenario for each. Add a scenario only where the plugin prescribes a decision a baseline agent would plausibly handle differently; Steps 8–9 confirm its lift, and a flat one is dropped (Step 6 / Step 10), not "improved." - -Write new scenarios directly rather than re-generating — you have full plugin context, the cloud generator doesn't. Each scenario is a directory in `evals//` with two files: `task.md` and `criteria.json`. - -`criteria.json` MUST use the weighted-checklist wrapper — the scorer rejects bare arrays: - -```json -{ - "context": "", - "type": "weighted_checklist", - "checklist": [ - { "name": "", "max_score": , "description": "" } - ] -} -``` - -Weights in `checklist[].max_score` MUST sum to exactly 100. Do not distribute evenly — weight the criteria that most specifically grade plugin-prescribed behaviour. Look at a sibling `evals/*/criteria.json` in this plugin to anchor on the exact shape; ignore any pre-existing plain-array files in the test repo under evaluation — those are seed data, not the format to emit. - -After writing each new scenario, run the Step 4 review against it and apply Step 5 fixes before moving on — new scenarios need the same no-bleeding / no-leaking audit as generated ones, and the failure mode on this step is to skip review on content you authored yourself. - -## Step 8 — Run Evals - -```bash -tessl eval run . -``` - -If any scenario fails to run, diagnose and fix before proceeding. - -## Step 9 — Analyze Results (Lift, Not Attainment) - -For each scenario, compute `lift = with_context_score - baseline_score`. Lift is the number that matters — aggregate attainment alone is a vanity metric. - -Classify each scenario by its lift band, run the three-cause diagnosis on weak / no-lift positive cases, decide whether negative-case lift is healthy or null, and triage failing criteria into a skill / task / criteria fix per the reference at: - -```text -skills/eval-authoring/LIFT_ANALYSIS.md -``` - -Bring the results back here, then proceed immediately to Step 10. - -## Step 10 — Iterate - -Fix the identified issues — including retiring null-test scenarios — then re-run from Step 8. Repeat until every positive-case scenario shows meaningful lift and every criterion grades behaviour the plugin actually contributes. Finish here when the lift distribution is stable and no null tests remain. diff --git a/skills/eval-curation/SKILL.md b/skills/eval-curation/SKILL.md deleted file mode 100644 index 820289c2..00000000 --- a/skills/eval-curation/SKILL.md +++ /dev/null @@ -1,93 +0,0 @@ ---- -name: eval-curation -description: > - Prune, trim, and shape an existing Tessl eval suite. Run the suite, compute - per-scenario lift, apply the three-cause diagnosis to near-zero-lift - scenarios, decide keep / fix / retire, and verify the curated suite still - pulls weight. - Use when an eval suite has grown bloated, scenarios are producing near-zero - lift, reviewing an existing suite for trim opportunities, optimizing a suite - for cost / signal ratio, many scenarios feel redundant or low-value, or the - user says trim / prune / shape / curate / optimize the evals. ---- - -# Eval Curation Skill - -Prune an existing eval suite down to the scenarios that actually pull weight. Process steps in order. Do not skip ahead. - -The diagnostic vocabulary (lift bands, three-cause diagnosis) lives in: - -```text -skills/eval-authoring/LIFT_ANALYSIS.md -``` - -Read it before Step 3. The obligation to prune is set by `rules/plugin-evals.md` "Lift, Not Attainment". - -## Step 1 — Run the Suite - -```bash -tessl eval run . -``` - -Wait until the run completes. Record the run ID — Step 2 needs it. - -Proceed immediately to Step 2. - -## Step 2 — Pull Per-Scenario Lift - -```bash -tessl eval view --json | python3 .tessl/plugins/jbaruch/coding-policy/skills/eval-curation/compute-lift.py -``` - -The script reads a `tessl eval view --json` payload from stdin (or a path argument), pairs each scenario's `with-context`/`usage-spec` variant against its `baseline`/`without-context` variant, sums the `assessmentResults` scores per side, and emits the per-scenario lift trio as JSON: - -```json -{ - "lifts": [{ "scenario_id": "", "lift": , "with_context_total": , "baseline_total": }], - "skipped": [{ "scenario_id": "", "reason": "" }] -} -``` - -Scenarios missing a paired variant land in `skipped` with a diagnostic; the script does not silently drop them. The deterministic JSON walk + arithmetic live in the script per `rules/script-delegation.md` so this step is reproducible and testable independently of any agent. - -Proceed immediately to Step 3. - -## Step 3 — Classify by Lift Band - -Bucket each scenario into the lift bands defined in: - -```text -skills/eval-authoring/LIFT_ANALYSIS.md -``` - -Healthy positive-case bands and negative cases whose near-zero lift is acceptable per that file's Negative Cases section stay as-is. - -If no scenarios sit in the actionable bands (no weak / no-lift positive cases, no plugin-specific negative case that fails the lift expectation), the suite is clean. Produce a one-line `curation-summary.md` stating "no curation needed" and finish here — do not fabricate diagnoses for scenarios that don't need them. - -Otherwise proceed immediately to Step 4 with two inputs to the three-cause diagnosis: the weak / no-lift positive cases, AND any plugin-specific negative case whose lift fell below the acceptable band. - -## Step 4 — Diagnose Every Actionable Scenario - -Apply the three-cause diagnosis from `rules/plugin-evals.md` "Lift, Not Attainment" to each scenario routed in from Step 3: - -1. **Coincidence with universal competence** — plugin's prescribed manner matches what baseline agents produce by default (positive case), or for a plugin-specific negative case: baseline refuses for plugin-independent reasons so the plugin's refusal adds no signal. Decision: retire -2. **Task leaked the technique** — fix the task per `skills/eval-authoring/REVIEW_CHECKLIST.md`'s No Bleeding rules; keep the criterion. Do NOT drop the criterion. -3. **Criteria grade universal competence** — the criteria test things baseline always does (basic git safety, obvious engineering judgement), not plugin-specific choices. Decision: rewrite the criteria to test the specific manner the plugin prescribes, or retire the scenario. - -Record the decision per scenario: `retire`, `fix-task`, or `rewrite-criteria`. Proceed immediately to Step 5. - -## Step 5 — Apply Decisions - -For each `retire`: `git rm -r evals/` and note the removal at the top of the plugin's `CHANGELOG.md`, following the project's CHANGELOG Hygiene convention (`rules/context-artifacts.md`). - -For each `fix-task`: edit `task.md` per the No Bleeding rules — strip the technique / format / literal that leaked; keep the situation the user actually needs done. - -For each `rewrite-criteria`: edit `criteria.json` so the checklist grades the specific manner the plugin prescribes (flag choices, format literals, sequences, conventions), not universal competence. Re-weight so `max_score` values still sum to exactly 100; if removing a criterion leaves nothing plugin-specific, retire the scenario instead. - -Proceed immediately to Step 6. - -## Step 6 — Verify the Curated Suite - -Re-run the suite (`tessl eval run .`) and re-fetch per-scenario lift via Step 2's mechanism. Verify three things: retired scenarios are gone from the run, fixed scenarios now show meaningful lift, the lift distribution is denser than before the curation pass. - -If any fixed scenario still produces near-zero lift, return to Step 4 with that scenario alone — the diagnosis was wrong or the fix didn't take. Otherwise finish here when the distribution is stable and every kept scenario contributes signal. diff --git a/skills/eval-curation/compute-lift.py b/skills/eval-curation/compute-lift.py deleted file mode 100644 index 6c585789..00000000 --- a/skills/eval-curation/compute-lift.py +++ /dev/null @@ -1,193 +0,0 @@ -#!/usr/bin/env python3 -"""compute-lift.py — Per-scenario lift from a `tessl eval view --json` payload. - -Per `jbaruch/coding-policy: rules/plugin-evals.md` "Lift, Not Attainment", -lift is the headline metric. Per `rules/script-delegation.md`, deterministic -JSON parsing + arithmetic belongs in a script, not in skill prose for the -agent to execute inline. - -Input: `tessl eval view --json ` payload, read from stdin or a path -argument. - -Output (stdout, JSON, per the script-delegation "JSON-producing" requirement): - - { - "lifts": [ - { "scenario_id": "", - "lift": , - "with_context_total": , - "baseline_total": }, - ... - ], - "skipped": [ - { "scenario_id": "", "reason": "" }, - ... - ] - } - -Variant pairing — `with-context` paired against `baseline`; tessl-eval also -emits `usage-spec` (older "with plugin loaded") and `without-context` (older -"without plugin loaded"), so the script accepts those as aliases. The first -preferred variant present wins per side. - -Exit codes: 0 on success (even when scenarios are skipped — skips are data, -not errors). Non-zero only when the payload itself is malformed (not JSON, -or missing the `data.attributes.scenarios` envelope). -""" - -import argparse -import json -import sys -from typing import Dict, List, Optional - -WITH_CONTEXT_VARIANTS = ["with-context", "usage-spec"] -BASELINE_VARIANTS = ["baseline", "without-context"] - - -def solution_total(solution: Dict) -> float: - """Sum `assessmentResults[].score` for a single solution.""" - total = 0.0 - for result in solution.get("assessmentResults") or []: - score = result.get("score") - if score is None: - continue - try: - total += float(score) - except (TypeError, ValueError): - # Per script-delegation "self-error-handling": fail loud rather - # than silently producing a wrong total. Caller turns this into - # a `skipped` entry. - raise ValueError( - f"non-numeric score {score!r} in criterion " - f"{result.get('name')!r}" - ) - return total - - -def pick_variant(solutions: List[Dict], preferred: List[str]) -> Optional[Dict]: - """Return the first solution whose `variant` is in `preferred` (priority - order), or None if no preferred variant is present. - """ - by_variant = { - s.get("variant"): s - for s in solutions or [] - if isinstance(s, dict) - } - for v in preferred: - if v in by_variant: - return by_variant[v] - return None - - -def compute_lifts(payload: Dict) -> Dict: - """Walk `payload.data.attributes.scenarios` and produce the - {lifts, skipped} structure documented at module top. - """ - try: - scenarios = payload["data"]["attributes"]["scenarios"] - except (KeyError, TypeError): - raise ValueError( - "payload missing `data.attributes.scenarios` — is this a " - "`tessl eval view --json` response?" - ) - if not isinstance(scenarios, list): - raise ValueError("`data.attributes.scenarios` is not a list") - - lifts: List[Dict] = [] - skipped: List[Dict] = [] - - for sc in scenarios: - if not isinstance(sc, dict): - continue - sid = sc.get("id", "") - solutions = sc.get("solutions") or [] - with_context = pick_variant(solutions, WITH_CONTEXT_VARIANTS) - baseline = pick_variant(solutions, BASELINE_VARIANTS) - if with_context is None or baseline is None: - skipped.append({ - "scenario_id": sid, - "reason": ( - f"missing variant — with-context present: " - f"{with_context is not None}, baseline present: " - f"{baseline is not None}" - ), - }) - continue - try: - wc_total = solution_total(with_context) - bl_total = solution_total(baseline) - except ValueError as e: - skipped.append({"scenario_id": sid, "reason": str(e)}) - continue - lifts.append({ - "scenario_id": sid, - "lift": round(wc_total - bl_total, 4), - "with_context_total": wc_total, - "baseline_total": bl_total, - }) - - return {"lifts": lifts, "skipped": skipped} - - -def main(argv=None) -> int: - parser = argparse.ArgumentParser( - description=(__doc__ or "").split("\n\n")[0], - ) - parser.add_argument( - "payload", - nargs="?", - default="-", - help="Path to a tessl-eval-view JSON file, or '-' to read stdin", - ) - args = parser.parse_args(argv) - - if args.payload == "-": - try: - payload = json.load(sys.stdin) - except json.JSONDecodeError as e: - print(f"error: stdin is not valid JSON: {e}", file=sys.stderr) - return 2 - else: - try: - with open(args.payload) as f: - payload = json.load(f) - except FileNotFoundError: - print( - f"error: payload file not found: {args.payload} — " - f"check the path, or pass '-' to read from stdin", - file=sys.stderr, - ) - return 2 - except PermissionError: - print( - f"error: permission denied reading {args.payload} — " - f"check file permissions", - file=sys.stderr, - ) - return 2 - except OSError as e: - print( - f"error: could not read {args.payload}: {e}", - file=sys.stderr, - ) - return 2 - except json.JSONDecodeError as e: - print( - f"error: {args.payload} is not valid JSON: {e}", - file=sys.stderr, - ) - return 2 - - try: - result = compute_lifts(payload) - except ValueError as e: - print(f"error: {e}", file=sys.stderr) - return 2 - - json.dump(result, sys.stdout, indent=2, sort_keys=True) - sys.stdout.write("\n") - return 0 - - -if __name__ == "__main__": - sys.exit(main()) diff --git a/skills/eval-curation/tests/test_compute_lift.py b/skills/eval-curation/tests/test_compute_lift.py deleted file mode 100644 index 0955365e..00000000 --- a/skills/eval-curation/tests/test_compute_lift.py +++ /dev/null @@ -1,277 +0,0 @@ -#!/usr/bin/env python3 -"""Tests for skills/eval-curation/compute-lift.py.""" - -import importlib.util -import json -import subprocess -import sys -import tempfile -import unittest -from pathlib import Path - -SCRIPT = Path(__file__).resolve().parent.parent / "compute-lift.py" - -spec = importlib.util.spec_from_file_location("compute_lift", SCRIPT) -assert spec and spec.loader, f"cannot load compute-lift.py at {SCRIPT}" -compute_lift = importlib.util.module_from_spec(spec) -spec.loader.exec_module(compute_lift) - - -def make_scenario(sid, variants_scores): - """Build one tessl-shaped scenario. - - `variants_scores` is {variant_name: [criterion_scores]}. Each criterion - contributes one assessmentResults row with `score`. - """ - solutions = [] - for variant, scores in variants_scores.items(): - solutions.append({ - "variant": variant, - "assessmentResults": [ - {"name": f"c{i}", "score": s} for i, s in enumerate(scores) - ], - }) - return {"id": sid, "solutions": solutions} - - -def wrap(scenarios): - return {"data": {"attributes": {"scenarios": scenarios}}} - - -class TestSolutionTotal(unittest.TestCase): - def test_sums_criterion_scores(self): - sol = {"assessmentResults": [ - {"name": "a", "score": 10}, - {"name": "b", "score": 25.5}, - {"name": "c", "score": 5}, - ]} - self.assertEqual(compute_lift.solution_total(sol), 40.5) - - def test_missing_assessment_results_is_zero(self): - self.assertEqual(compute_lift.solution_total({}), 0.0) - self.assertEqual(compute_lift.solution_total({"assessmentResults": []}), 0.0) - - def test_missing_score_field_skipped(self): - sol = {"assessmentResults": [ - {"name": "a", "score": 10}, - {"name": "b"}, - {"name": "c", "score": 5}, - ]} - self.assertEqual(compute_lift.solution_total(sol), 15.0) - - def test_non_numeric_score_raises(self): - sol = {"assessmentResults": [ - {"name": "a", "score": 10}, - {"name": "b", "score": "garbage"}, - ]} - with self.assertRaises(ValueError) as ctx: - compute_lift.solution_total(sol) - self.assertIn("non-numeric score", str(ctx.exception)) - - -class TestPickVariant(unittest.TestCase): - def test_picks_first_preferred(self): - sols = [ - {"variant": "baseline", "id": "b"}, - {"variant": "with-context", "id": "wc"}, - ] - picked = compute_lift.pick_variant(sols, ["with-context", "usage-spec"]) - self.assertEqual(picked["id"], "wc") - - def test_falls_back_in_preference_order(self): - sols = [{"variant": "usage-spec", "id": "us"}] - picked = compute_lift.pick_variant(sols, ["with-context", "usage-spec"]) - self.assertEqual(picked["id"], "us") - - def test_returns_none_when_no_preferred_match(self): - sols = [{"variant": "experimental", "id": "x"}] - self.assertIsNone( - compute_lift.pick_variant(sols, ["with-context", "usage-spec"]) - ) - - def test_empty_list_returns_none(self): - self.assertIsNone(compute_lift.pick_variant([], ["with-context"])) - self.assertIsNone(compute_lift.pick_variant(None, ["with-context"])) - - -class TestComputeLifts(unittest.TestCase): - def test_basic_lift_computation(self): - scenarios = [make_scenario("sc-1", { - "with-context": [25, 25, 25, 25], # total 100 - "baseline": [10, 10, 10, 5], # total 35 - })] - result = compute_lift.compute_lifts(wrap(scenarios)) - self.assertEqual(len(result["lifts"]), 1) - self.assertEqual(result["skipped"], []) - self.assertEqual(result["lifts"][0]["lift"], 65.0) - self.assertEqual(result["lifts"][0]["with_context_total"], 100.0) - self.assertEqual(result["lifts"][0]["baseline_total"], 35.0) - - def test_variant_aliases_accepted(self): - # `usage-spec` and `without-context` are older aliases that the - # script should still handle. - scenarios = [make_scenario("sc-2", { - "usage-spec": [50, 50], - "without-context": [30, 30], - })] - result = compute_lift.compute_lifts(wrap(scenarios)) - self.assertEqual(result["lifts"][0]["lift"], 40.0) - - def test_missing_variant_skipped(self): - # Only with-context — no baseline. Should land in `skipped`, - # not crash. - scenarios = [make_scenario("sc-no-baseline", { - "with-context": [50, 50], - })] - result = compute_lift.compute_lifts(wrap(scenarios)) - self.assertEqual(result["lifts"], []) - self.assertEqual(len(result["skipped"]), 1) - self.assertEqual(result["skipped"][0]["scenario_id"], "sc-no-baseline") - self.assertIn("baseline present: False", result["skipped"][0]["reason"]) - - def test_negative_lift_preserved(self): - scenarios = [make_scenario("sc-neg", { - "with-context": [20], - "baseline": [80], - })] - result = compute_lift.compute_lifts(wrap(scenarios)) - self.assertEqual(result["lifts"][0]["lift"], -60.0) - - def test_zero_lift_preserved(self): - scenarios = [make_scenario("sc-zero", { - "with-context": [100], - "baseline": [100], - })] - result = compute_lift.compute_lifts(wrap(scenarios)) - self.assertEqual(result["lifts"][0]["lift"], 0.0) - - def test_non_numeric_criterion_score_skipped(self): - scenarios = [{ - "id": "sc-bad", - "solutions": [ - {"variant": "with-context", "assessmentResults": [ - {"score": 10}, {"score": "garbage"}, - ]}, - {"variant": "baseline", "assessmentResults": [{"score": 5}]}, - ], - }] - result = compute_lift.compute_lifts(wrap(scenarios)) - self.assertEqual(result["lifts"], []) - self.assertEqual(len(result["skipped"]), 1) - self.assertIn("non-numeric", result["skipped"][0]["reason"]) - - def test_invalid_envelope_raises(self): - with self.assertRaises(ValueError): - compute_lift.compute_lifts({}) - with self.assertRaises(ValueError): - compute_lift.compute_lifts({"data": {"attributes": {"scenarios": "not-a-list"}}}) - - def test_multiple_scenarios(self): - scenarios = [ - make_scenario(f"sc-{i}", { - "with-context": [50], - "baseline": [20 + i * 5], - }) - for i in range(3) - ] - result = compute_lift.compute_lifts(wrap(scenarios)) - self.assertEqual(len(result["lifts"]), 3) - self.assertEqual([l["lift"] for l in result["lifts"]], [30.0, 25.0, 20.0]) - - -class TestCLI(unittest.TestCase): - def test_cli_reads_file_argument(self): - scenarios = [make_scenario("sc-cli", { - "with-context": [40, 40], - "baseline": [10, 10], - })] - with tempfile.NamedTemporaryFile("w", suffix=".json", delete=False) as f: - json.dump(wrap(scenarios), f) - path = f.name - try: - result = subprocess.run( - [sys.executable, str(SCRIPT), path], - capture_output=True, text=True, - ) - self.assertEqual(result.returncode, 0, f"stderr: {result.stderr}") - out = json.loads(result.stdout) - self.assertEqual(out["lifts"][0]["lift"], 60.0) - finally: - Path(path).unlink() - - def test_cli_reads_stdin(self): - scenarios = [make_scenario("sc-stdin", { - "with-context": [70], - "baseline": [30], - })] - result = subprocess.run( - [sys.executable, str(SCRIPT), "-"], - input=json.dumps(wrap(scenarios)), - capture_output=True, text=True, - ) - self.assertEqual(result.returncode, 0, f"stderr: {result.stderr}") - out = json.loads(result.stdout) - self.assertEqual(out["lifts"][0]["lift"], 40.0) - - def test_cli_invalid_json_exits_2(self): - with tempfile.NamedTemporaryFile("w", suffix=".json", delete=False) as f: - f.write("not json {{") - path = f.name - try: - result = subprocess.run( - [sys.executable, str(SCRIPT), path], - capture_output=True, text=True, - ) - self.assertEqual(result.returncode, 2) - self.assertIn("not valid json", result.stderr.lower()) - finally: - Path(path).unlink() - - def test_cli_missing_file_exits_2_with_actionable_message(self): - # Per `rules/script-delegation.md` self-error-handling + error-handling's - # "actionable messages": missing path must produce a clear stderr - # diagnostic + non-zero exit, never a raw Python traceback. - result = subprocess.run( - [sys.executable, str(SCRIPT), "/tmp/eval-curation-does-not-exist-xxxxxx.json"], - capture_output=True, text=True, - ) - self.assertEqual(result.returncode, 2) - self.assertIn("not found", result.stderr.lower()) - self.assertNotIn("traceback", result.stderr.lower()) - # The message must hint at recovery, not just state the failure. - self.assertIn("check the path", result.stderr.lower()) - - def test_cli_permission_denied_exits_2_with_actionable_message(self): - with tempfile.NamedTemporaryFile("w", suffix=".json", delete=False) as f: - json.dump({"data": {"attributes": {"scenarios": []}}}, f) - path = f.name - try: - Path(path).chmod(0o000) - result = subprocess.run( - [sys.executable, str(SCRIPT), path], - capture_output=True, text=True, - ) - self.assertEqual(result.returncode, 2) - self.assertIn("permission", result.stderr.lower()) - self.assertNotIn("traceback", result.stderr.lower()) - finally: - Path(path).chmod(0o600) - Path(path).unlink() - - def test_cli_wrong_envelope_exits_2(self): - with tempfile.NamedTemporaryFile("w", suffix=".json", delete=False) as f: - json.dump({"wrong_shape": []}, f) - path = f.name - try: - result = subprocess.run( - [sys.executable, str(SCRIPT), path], - capture_output=True, text=True, - ) - self.assertEqual(result.returncode, 2) - self.assertIn("scenarios", result.stderr.lower()) - finally: - Path(path).unlink() - - -if __name__ == "__main__": - unittest.main(verbosity=2) diff --git a/skills/migrate-to-plugin/SKILL.md b/skills/migrate-to-plugin/SKILL.md index ddef91ae..9e72812c 100644 --- a/skills/migrate-to-plugin/SKILL.md +++ b/skills/migrate-to-plugin/SKILL.md @@ -49,7 +49,6 @@ Keep "tile" unchanged — these are live contracts, not prose: - The `tile.json` filename in historical or legacy-format references - `v1/tiles/...` REST API routes and any argument bound to that route - Code identifiers — variable, function, type, and class names -- Frozen `evals/*` scenario directory names — renaming resets lift history (see `rules/plugin-evals.md` Naming) - Real external documentation page paths and analytics property names If `residual_files` is empty, proceed silently to Step 3. diff --git a/skills/release/SKILL.md b/skills/release/SKILL.md index 48bb0c75..e912eb2a 100644 --- a/skills/release/SKILL.md +++ b/skills/release/SKILL.md @@ -19,13 +19,11 @@ Structured workflow for shipping code: PR creation, automated policy review, mer - Confirm you're on a feature branch (not `main`/`master`) - Run the test suite — all tests must pass - Run the linter — no warnings or errors -- Self-audit the diff against every governing rule or skill whose domain covers the touched paths (e.g., `evals/` → `rules/plugin-evals.md` and `skills/eval-authoring/SKILL.md`; auto-loaded prose in `rules/` or `skills/` → `rules/context-writing-style.md`; new scripts → `rules/script-delegation.md` and `rules/testing-standards.md`) +- Self-audit the diff against every governing rule or skill whose domain covers the touched paths (e.g., auto-loaded prose in `rules/` or `skills/` → `rules/context-writing-style.md`; new scripts → `rules/script-delegation.md` and `rules/testing-standards.md`) - Grep the diff for the literal markers each governing rule or skill names: - banned connectives `because`, `therefore`, `since`, etc. per `rules/context-writing-style.md` - the `outer-boundary-process-contract` token per `rules/error-handling.md` - - `max_score` weight sums (must sum to 100) per `skills/eval-authoring/SKILL.md` - Run any local check the rule or skill prescribes: - - `jq '[.checklist[].max_score] | add' ` must print `100`, and the weights must not be uniformly distributed (per `skills/eval-authoring/SKILL.md`) - `bash -n