diff --git a/README.md b/README.md index 3b12955..e692e62 100644 --- a/README.md +++ b/README.md @@ -301,6 +301,12 @@ Once installed, OpenCode will automatically detect the skill when you ask it to OpenCode will load the opencode-skill-creator instructions and use the plugin tools to walk through the full workflow. +## Examples + +The fastest way to see the workflow end to end: start with the [Docker Compose helper example](examples/docker-compose-helper/). It includes a small complete skill, a trigger eval set in the exact `skill_eval` JSON format, and copy-pasteable prompts for `skill_eval`, `skill_optimize_loop`, and benchmark review. + +Use it after install to see how description evals work before creating your own skill. + ## Architecture This project has two components: diff --git a/examples/docker-compose-helper/README.md b/examples/docker-compose-helper/README.md new file mode 100644 index 0000000..7ebd311 --- /dev/null +++ b/examples/docker-compose-helper/README.md @@ -0,0 +1,72 @@ +# Docker Compose Helper Example + +This is a complete, lightweight example of the eval-driven skill workflow: +write a small skill, test whether its description triggers on the right +queries, optimize the description, then benchmark real task output if needed. + +## Files + +- `SKILL.md` - the example skill +- `evals/eval-set.json` - trigger evals consumed by `skill_eval` + +The eval file is a JSON array of `{ "query": string, "should_trigger": boolean }` +items. That matches the `EvalItem` interface used by the plugin. + +## Run A Trigger Eval + +In OpenCode, ask: + +```text +Use opencode-skill-creator to run skill_eval on examples/docker-compose-helper with examples/docker-compose-helper/evals/eval-set.json. +Use runsPerQuery 3 and triggerThreshold 0.5. +``` + +Equivalent tool arguments: + +```text +skill_eval: + skillPath: examples/docker-compose-helper + evalSetPath: examples/docker-compose-helper/evals/eval-set.json + runsPerQuery: 3 + triggerThreshold: 0.5 +``` + +Expected output: JSON with one result per query. Positive queries pass when the +skill triggers in at least 50% of successful runs; negative queries pass when it +triggers in less than 50%. With the defaults above, each query runs 3 times. + +## Optimize The Description + +Ask: + +```text +Use opencode-skill-creator to run skill_optimize_loop for examples/docker-compose-helper using examples/docker-compose-helper/evals/eval-set.json. +Use maxIterations 5, runsPerQuery 3, triggerThreshold 0.5. +``` + +Equivalent tool arguments: + +```text +skill_optimize_loop: + skillPath: examples/docker-compose-helper + evalSetPath: examples/docker-compose-helper/evals/eval-set.json + maxIterations: 5 + runsPerQuery: 3 + triggerThreshold: 0.5 +``` + +Expected output: JSON containing the best description found, train/test scores, +and per-query trigger rates. Copy `best_description` into the frontmatter only +if it improves held-out test behavior and still reads clearly. + +## Benchmark Real Output + +Trigger evals only test whether OpenCode chooses the skill. To compare task +quality, run paired task outputs with and without the skill, then ask: + +```text +Use opencode-skill-creator to run skill_aggregate_benchmark for my eval workspace, then open the review viewer. +``` + +Expected output: `benchmark.json`, `benchmark.md`, and a review viewer showing +pass rates, timing, tokens, and side-by-side run outputs. diff --git a/examples/docker-compose-helper/SKILL.md b/examples/docker-compose-helper/SKILL.md new file mode 100644 index 0000000..3405107 --- /dev/null +++ b/examples/docker-compose-helper/SKILL.md @@ -0,0 +1,59 @@ +--- +name: docker-compose-helper +description: Use when the user asks to create, debug, or modify Docker Compose files, compose.yaml, docker-compose.yml, service dependencies, healthchecks, ports, volumes, networks, environment variables, or local multi-container development stacks. Use this skill even if the user says "compose file" or "local stack" instead of "Docker Compose". +--- + +# Docker Compose Helper + +Help users create, review, and repair Docker Compose setups with the smallest +working change that fits their local development workflow. + +## Workflow + +1. Identify the user's goal: + - creating a new compose file + - adding or changing a service + - debugging startup, networking, volume, or environment issues + - translating a `docker run` command into compose +2. Inspect existing files before editing: + - `compose.yaml` + - `compose.yml` + - `docker-compose.yaml` + - `docker-compose.yml` + - `.env` + - Dockerfiles referenced by `build` +3. Prefer Docker Compose v2 syntax: + - omit the obsolete top-level `version` + - use `compose.yaml` if creating a new file + - keep service names lowercase and stable +4. Make the smallest useful change. Do not add orchestration, custom networks, + or named volumes unless the service needs them. + +## Compose Defaults + +- Use explicit image tags instead of `latest` when a stable tag is obvious. +- Use named volumes for database state. +- Use bind mounts for application source during local development. +- Put secrets and machine-specific values in `.env`, not directly in YAML. +- Add `depends_on` only for startup ordering; add `healthcheck` when readiness + actually matters. +- Expose only the host ports the user needs. + +## Debugging Checklist + +When debugging, check these in order: + +1. YAML validity and indentation. +2. Whether the service name, image, build context, and Dockerfile path exist. +3. Port conflicts between host ports and already-running services. +4. Volume paths and whether a bind mount hides files built into the image. +5. Environment variable names, `.env` loading, and missing required values. +6. Container-to-container networking: services should use service names as DNS + hosts, not `localhost`. +7. Readiness problems: add a healthcheck and condition only when the dependent + service truly must wait. + +## Output + +When editing files, show the changed compose snippet or file path and explain +the reason for each non-obvious setting in one short sentence. diff --git a/examples/docker-compose-helper/evals/eval-set.json b/examples/docker-compose-helper/evals/eval-set.json new file mode 100644 index 0000000..a05841d --- /dev/null +++ b/examples/docker-compose-helper/evals/eval-set.json @@ -0,0 +1,34 @@ +[ + { + "query": "Create a compose.yaml for a Node API, Postgres database, and Redis cache for local development.", + "should_trigger": true + }, + { + "query": "My docker-compose.yml starts Postgres but the app says ECONNREFUSED to localhost:5432. Help me fix the compose networking.", + "should_trigger": true + }, + { + "query": "Convert this docker run command into a Docker Compose service with a named volume and port mapping.", + "should_trigger": true + }, + { + "query": "Add a healthcheck and depends_on condition so my web service waits for the database to be ready in compose.yaml.", + "should_trigger": true + }, + { + "query": "Review my compose file for unnecessary ports, missing env vars, and volume mistakes before I share it with the team.", + "should_trigger": true + }, + { + "query": "Write a Kubernetes Deployment and Service for this container image.", + "should_trigger": false + }, + { + "query": "Explain how Docker image layers and build cache work at a high level.", + "should_trigger": false + }, + { + "query": "Fix this GitHub Actions workflow that builds and pushes a container image.", + "should_trigger": false + } +]