Add bakery Claude skill with inspect_ai evals

.claude/skills/bakery/SKILL.md

@@ -0,0 +1,241 @@
+---
+name: bakery
+description: >
+  Use when working in any Posit container image repo (images-connect,
+  images-workbench, images-package-manager, images-shared) on tasks involving:
+  editing Containerfile templates, adding or updating image versions, running
+  bakery commands, or modifying CI workflows that call bakery ci matrix/merge.
+  Encodes critical invariants and common workflows to prevent known mistakes.
+---
+
+# bakery
+
+Posit Bakery is the CLI for building, testing, and managing container images across
+the posit-dev image repos. This skill encodes critical invariants and common workflows.
+
+## Critical invariants
+
+Check these before acting on any bakery-related task.
+
+### 1. Never edit rendered files — only templates
+
+Image repos contain Jinja2 templates in `template/` directories **and** rendered output
+in version directories (e.g., `2025.08.0/`). Rendered files are generated — never edit
+them directly.
+
+**Always:**
+1. Edit the template: `<image>/template/Containerfile.jinja` (or other `.jinja` files)
+2. Re-render with filter flags scoped to the most recent version:
+   `uv run bakery update files --image-name <name> --image-version <version>`
+
+**Never:** run `uv run bakery update files` without filter flags — it re-renders every
+version, which is almost never the right action.
+
+**Exception:** for systematic changes that must land in every version, it is often easier
+to render one version, make the change locally in that rendered file to validate it, then
+apply the edit to the template and re-render.
+
+### 2. Invoke bakery with `uv run`
+
+Always prefix bakery commands with `uv run`:
+
+```bash
+uv run bakery build --plan
+uv run bakery update files
+uv run bakery create version 1.2.3
+```
+
+Bare `bakery` is not on the system PATH — it is installed via uv in the `posit-bakery/`
+project and only available through `uv run`.
+
+### 3. Read sibling repos before cross-repo changes
+
+When a task requires editing templates, macros, or `bakery.yaml` in a sibling repo
+(`images-connect`, `images-workbench`, `images-package-manager`), read that repo's
+`CLAUDE.md` and `bakery.yaml` before making any changes there.
+
+### 4. Matrix versions are excluded by default
+
+`--matrix-versions` defaults to `exclude`. Matrix images (e.g., `connect-content`,
+`workbench-session`) produce **zero build targets** unless you explicitly pass
+`--matrix-versions include` or `--matrix-versions only`.
+
+Always verify with `uv run bakery build --plan` or `uv run bakery get tags` before
+building — a plan with no targets means a filter flag is wrong or missing.
+
+### 5. Use `--dev-channel`, not `--dev-stream`
+
+`--dev-stream` is deprecated (hidden, emits a warning). Use `--dev-channel` instead:
+
+```bash
+uv run bakery build --dev-versions only --dev-channel daily
+```
+
+`--dev-channel` is silently ignored when `--dev-versions` is `exclude` (the default) —
+bakery emits a warning but does not error. Always pair them.
+
+For CI dispatch builds that must pin an exact dev version, use `--dev-spec` (or the
+`BAKERY_DEV_SPEC` env var) instead of `--dev-channel`. It accepts a JSON payload and
+overrides CDN discovery for the matching channel:
+
+```bash
+uv run bakery build --dev-versions only \
+  --dev-spec '{"version": "2026.05.0-dev+185-gSHA", "channel": "daily"}'
+```
+
+If both `--dev-spec` and `--dev-channel` are set, their `channel` values must match or
+bakery raises an error.
+
+### 6. Forward filter flags to `bakery ci merge` and `clean`
+
+When modifying a CI workflow, filter flags must be forwarded consistently across all
+three stages: build → merge, and build → clean.
+
+**Build → merge:** `bakery ci merge` must receive the same flags as the build step:
+
+- `--dev-versions [include|exclude|only]`
+- `--dev-channel [release|preview|daily]`
+- `--dev-spec <json>` / `BAKERY_DEV_SPEC`
+- `--matrix-versions [include|exclude|only]`
+
+Mismatched flags cause the merge step to match metadata by UID and fan a single built
+image out to multiple targets — including targets in the wrong registry. A release
+version and a dev-stream version that share the same version number have identical UIDs
+and will collide. (Tracked as posit-dev/images-shared#553; fix in PR #554.)
+
+**Build → clean:** `clean.yml` must receive the same `dev-versions` value as the build
+that produced the images. If build uses `dev-versions: "only"` but clean uses the default
+`dev-versions: "exclude"`, the clean job targets the wrong set of images and dev artifacts
+accumulate.
+
+**Temp registry:** The `--temp-registry` value in `bakery ci merge` must exactly match the
+value used in the corresponding `bakery build` step. A mismatch causes the merge step to
+look for images that don't exist, failing with cryptic "image not found" errors.
+
+### 7. `bakery remove` is irreversible
+
+`bakery remove version` and `bakery remove image` delete the version directory **and**
+remove the entry from `bakery.yaml`. There is no dry-run flag and no confirmation prompt.
+Always confirm with the user before running either command.
+
+### 8. `bakery create version` marks the new version as latest by default
+
+`bakery create version` sets `latest: true` on the new version and **unmarks all other
+versions**. If the repo has a current release marked `latest`, creating a new version will
+silently demote it. Verify the intended `latest` state in `bakery.yaml` after running.
+
+### 9. `bakery update version --clean` deletes files before re-rendering
+
+The `--clean` flag on `bakery update version` defaults to `True`. It deletes all existing
+files in the version directory before re-rendering. Use `bakery update files` (scoped) for
+non-destructive re-renders.
+
+### 10. Use `bakery dgoss run`, not `bakery run dgoss`
+
+`bakery run dgoss` is deprecated and emits a warning. Use:
+
+```bash
+uv run bakery dgoss run
+```
+
+### 11. Guard `clean.yml` with a branch condition
+
+When calling `clean.yml` from a product repo workflow, always include a branch guard:
+
+```yaml
+clean:
+  if: always() && github.ref == 'refs/heads/main'
+  needs: [build]
+  uses: "posit-dev/images-shared/.github/workflows/clean.yml@main"
+```
+
+Without the `github.ref` guard, fork PRs will attempt to clean images from the main
+org's registry. Fork workflows lack write credentials so the job fails, and the failure
+is hard to diagnose.
+
+## Common workflows
+
+### Add a new image version
+
+```bash
+uv run bakery create version <version>
+# Edit the generated template if the new version needs adjustments
+uv run bakery update files --image-name <name> --image-version <version>
+uv run bakery build --plan   # preview before building
+```
+
+### Update a template (Containerfile, goss tests, etc.)
+
+```bash
+# Edit the template — never the rendered output
+$EDITOR <image>/template/Containerfile.jinja
+
+# Re-render scoped to the most recent version (always specify filters)
+uv run bakery update files --image-name <name> --image-version <version>
+```
+
+### Preview what will be built
+
+```bash
+uv run bakery get tags                                  # list tags by component
+uv run bakery build --plan                              # full bake plan (JSON)
+```
+
+`--plan` only works with `--strategy bake` (the default). It errors with
+`--strategy build`.
+
+### Build locally
+
+```bash
+uv run bakery build                                     # build + load into Docker
+uv run bakery build --image-name connect --image-version 2025.08.0
+uv run bakery build --push --no-load                    # push to registry (CI pattern)
+```
+
+### Run goss tests
+
+```bash
+uv run bakery dgoss run
+```
+
+### Inspect the CI matrix
+
+```bash
+uv run bakery ci matrix
+```
+
+### Debug a CI failure
+
+```bash
+gh run list -R posit-dev/<repo>
+gh run view <run-id> -R posit-dev/<repo>
+gh run view <run-id> -R posit-dev/<repo> --log-failed
+```
+
+## Key CLI flags
+
+| Flag | Purpose |
+|------|---------|
+| `--image-name <name>` | Scope to an image (regex) |
+| `--image-version <ver>` | Scope to a version |
+| `--image-variant <var>` | Scope to Standard or Minimal |
+| `--image-os <os>` | Scope to an OS |
+| `--image-platform <plat>` | Scope to a platform (e.g. `linux/amd64`) |
+| `--dev-versions [include\|exclude\|only]` | Include/exclude dev versions |
+| `--dev-channel [release\|preview\|daily]` | Filter to a specific dev channel (replaces deprecated `--dev-stream`) |
+| `--dev-spec <json>` / `BAKERY_DEV_SPEC` | Pin an exact dev version for dispatch builds |
+| `--matrix-versions [include\|exclude\|only]` | Include/exclude matrix versions |
+| `--plan` | Print the bake plan and exit (build only) |
+| `--push` / `--no-push` | Push to registry after build |
+| `--strategy [bake\|build]` | `bake` = Docker Buildx parallel; `build` = sequential |
+
+## Template variables
+
+Available in all Jinja2 templates:
+
+- `Image.Version`, `Image.Variant`, `Image.IsDevelopmentVersion`
+- `Image.OS` with `.Name`, `.Family`, `.Version`, `.Codename`
+- `Path.Base`, `Path.Image`, `Path.Version`
+- `Dependencies.python`, `Dependencies.R`, `Dependencies.quarto` (lists of version strings)
+
+Custom filters: `tagSafe`, `stripMetadata`, `condense`, `regexReplace`, `quote`, `split`

.claude/skills/bakery/evals/dataset.json

@@ -0,0 +1,52 @@
+[
+  {
+    "id": "1-never-edit-rendered-files",
+    "input": "I need to update the apt packages installed in the connect Standard image for version 2025.08.0. The file connect/2025.08.0/Containerfile has the package list. What do I do?",
+    "target": "{\"expected\": [{\"id\": 1, \"description\": \"Identifies that connect/2025.08.0/Containerfile is a rendered file and must not be edited directly\"}, {\"id\": 2, \"description\": \"Directs editing of the template, e.g. connect/template/Containerfile.jinja\"}, {\"id\": 3, \"description\": \"Re-renders with --image-name connect and --image-version 2025.08.0 flags\"}], \"forbidden\": [{\"id\": \"f1\", \"description\": \"Recommends editing the rendered file connect/2025.08.0/Containerfile directly\"}, {\"id\": \"f2\", \"description\": \"Runs uv run bakery update files without any --image-name or --image-version filter flags\"}]}"
+  },
+  {
+    "id": "2-uv-invocation",
+    "input": "I want to build the connect image locally to test my template changes. What command should I run?",
+    "target": "{\"expected\": [{\"id\": 1, \"description\": \"Uses uv run bakery build (with the uv run prefix)\"}, {\"id\": 2, \"description\": \"Mentions --plan for a dry run before building\"}], \"forbidden\": [{\"id\": \"f1\", \"description\": \"Recommends running bare 'bakery build' without the 'uv run' prefix\"}]}"
+  },
+  {
+    "id": "3-scoped-render-required",
+    "input": "I've edited the Containerfile.jinja template for the connect image. How do I re-render the version files?",
+    "target": "{\"expected\": [{\"id\": 1, \"description\": \"Uses uv run bakery update files with --image-name and --image-version flags to scope the render\"}, {\"id\": 2, \"description\": \"Recommends scoping to the most recent version rather than re-rendering all versions\"}], \"forbidden\": [{\"id\": \"f1\", \"description\": \"Recommends running uv run bakery update files with no filter flags, which re-renders every version\"}]}"
+  },
+  {
+    "id": "4-cross-repo-read-first",
+    "input": "I need to update the Python installation macro (python.j2) used in images-workbench templates. What's my first step?",
+    "target": "{\"expected\": [{\"id\": 1, \"description\": \"Reads images-workbench/CLAUDE.md before making changes\"}, {\"id\": 2, \"description\": \"Reads images-workbench/bakery.yaml before making changes\"}], \"forbidden\": [{\"id\": \"f1\", \"description\": \"Proceeds directly to editing python.j2 or workbench templates without reading the sibling repo's CLAUDE.md and bakery.yaml first\"}]}"
+  },
+  {
+    "id": "5-ci-merge-flag-forwarding",
+    "input": "I'm updating development.yml to build only daily dev images. I've added --dev-versions only and --dev-channel daily to the bakery build step, and set BAKERY_DEV_SPEC on dispatch. What else needs to change?",
+    "target": "{\"expected\": [{\"id\": 1, \"description\": \"Adds --dev-versions only to the bakery ci merge step\"}, {\"id\": 2, \"description\": \"Adds --dev-channel daily (or the same channel value) to the bakery ci merge step\"}, {\"id\": 3, \"description\": \"Forwards BAKERY_DEV_SPEC (or --dev-spec) to the bakery ci merge step as well\"}, {\"id\": 4, \"description\": \"Explains that the merge step must receive the same filter flags as the build step to avoid UID mismatches\"}], \"forbidden\": [{\"id\": \"f1\", \"description\": \"Declares the change complete after only updating the build step, without mentioning the merge step\"}, {\"id\": \"f2\", \"description\": \"Uses --dev-stream instead of --dev-channel\"}]}"
+  },
+  {
+    "id": "6-version-creation-flow",
+    "input": "I need to add version 2026.06.0 to the connect image in images-connect. Walk me through the steps.",
+    "target": "{\"expected\": [{\"id\": 1, \"description\": \"Uses uv run bakery create version 2026.06.0 (with uv run prefix)\"}, {\"id\": 2, \"description\": \"Re-renders with uv run bakery update files scoped to --image-name connect --image-version 2026.06.0\"}, {\"id\": 3, \"description\": \"Includes a build --plan or bakery get tags step to preview before building\"}], \"forbidden\": [{\"id\": \"f1\", \"description\": \"Recommends hand-editing bakery.yaml to add the version instead of using bakery create version\"}, {\"id\": \"f2\", \"description\": \"Uses bare bakery commands without the uv run prefix\"}, {\"id\": \"f3\", \"description\": \"Runs uv run bakery update files without filter flags\"}]}"
+  },
+  {
+    "id": "7-dev-channel-not-dev-stream",
+    "input": "I want to build only the daily dev-stream images. What flags do I pass to bakery build?",
+    "target": "{\"expected\": [{\"id\": 1, \"description\": \"Uses --dev-channel daily (not --dev-stream daily)\"}, {\"id\": 2, \"description\": \"Includes --dev-versions include or --dev-versions only alongside --dev-channel\"}], \"forbidden\": [{\"id\": \"f1\", \"description\": \"Uses the deprecated --dev-stream flag instead of --dev-channel\"}]}"
+  },
+  {
+    "id": "8-dev-channel-requires-dev-versions",
+    "input": "I ran: uv run bakery build --dev-channel daily. No dev images were built. Why?",
+    "target": "{\"expected\": [{\"id\": 1, \"description\": \"Identifies that --dev-versions defaults to exclude, so --dev-channel is silently ignored\"}, {\"id\": 2, \"description\": \"Provides the corrected command with --dev-versions include or --dev-versions only alongside --dev-channel daily\"}], \"forbidden\": [{\"id\": \"f1\", \"description\": \"Attributes the problem to a missing --dev-stream flag or other unrelated cause\"}]}"
+  },
+  {
+    "id": "9-dev-spec-for-dispatch",
+    "input": "We're setting up a workflow dispatch that lets CI pin an exact dev version to build, rather than resolving the latest from CDN. How should I wire this up with bakery?",
+    "target": "{\"expected\": [{\"id\": 1, \"description\": \"Uses --dev-spec (or BAKERY_DEV_SPEC env var) with a JSON payload containing version and channel fields\"}, {\"id\": 2, \"description\": \"Shows or describes the JSON payload format, e.g. {\\\"version\\\": \\\"2026.05.0-dev+185-gSHA\\\", \\\"channel\\\": \\\"daily\\\"}\"}, {\"id\": 3, \"description\": \"Notes that --dev-versions include or only is still required alongside --dev-spec\"}], \"forbidden\": [{\"id\": \"f1\", \"description\": \"Uses --dev-channel alone without mentioning --dev-spec for pinning an exact version\"}, {\"id\": \"f2\", \"description\": \"Uses the deprecated --dev-stream flag\"}]}"
+  },
+  {
+    "id": "10-matrix-versions-excluded-by-default",
+    "input": "I want to build the connect-content matrix images. I ran uv run bakery build and nothing was built. What's wrong?",
+    "target": "{\"expected\": [{\"id\": 1, \"description\": \"Identifies that --matrix-versions defaults to exclude, so matrix images are silently skipped\"}, {\"id\": 2, \"description\": \"Provides the corrected command with --matrix-versions include or --matrix-versions only\"}, {\"id\": 3, \"description\": \"Suggests running uv run bakery get tags or uv run bakery build --plan first to verify targets are non-empty\"}], \"forbidden\": [{\"id\": \"f1\", \"description\": \"Attributes the problem to a missing --dev-versions flag or other unrelated cause\"}]}"
+  }
+]

.claude/skills/bakery/evals/pyproject.toml

@@ -0,0 +1,8 @@
+[project]
+name = "bakery-skill-evals"
+version = "0.1.0"
+requires-python = ">=3.12"
+dependencies = [
+    "inspect-ai",
+    "anthropic",
+]