Make SkillOpt the core optimization workflow

.env.example

@@ -7,28 +7,22 @@
 BASE_URL=https://openrouter.ai/api/v1
 API_KEY=
 
-# Or point at ANY other OpenAI-compatible provider instead:
-#   Fireworks direct (zero data retention by default for open models on serverless,
-#   https://docs.fireworks.ai/guides/security_compliance/data_handling):
-#     BASE_URL=https://api.fireworks.ai/inference/v1   API_KEY=fw_...
-#     AGENT_MODEL=accounts/fireworks/models/qwen3p7-plus
-#     SKILLOPT_MODEL=accounts/fireworks/models/glm-5p2
-#     JUDGE_MODEL=accounts/fireworks/models/deepseek-v4-pro
-#   Together direct:   BASE_URL=https://api.together.xyz/v1   API_KEY=...
-#   Local vLLM/Ollama: BASE_URL=http://172.17.0.1:8000/v1     (no key needed)
-# Serving-role-only overrides (agent runs, A/B agents, candidate rollouts) for hybrid setups:
+# OpenRouter is the only hosted provider configuration. Fully local vLLM/Ollama remains supported:
+#   BASE_URL=http://172.17.0.1:8000/v1     # no key needed
+# Serving-role-only overrides let agent runs, A/B agents, and candidate rollouts run locally while
+# SkillOpt and the judge stay on OpenRouter:
 # MODEL_BASE_URL=
 # MODEL_API_KEY=
 
-# Optional overrides:
-# AGENT_MODEL=qwen/qwen3.6-27b      # the agent model: everything that *executes* skills (also
+# Optional OpenRouter model overrides. Use OpenRouter model slugs from https://openrouter.ai/models:
+# AGENT_MODEL=qwen/qwen3-32b        # the agent model: everything that *executes* skills (also
 #                                   # candidate rollouts); MODEL is the legacy alias
 # SKILLOPT_MODEL=z-ai/glm-5.2       # authors evals and skill revisions
 # STRONG_MODEL=z-ai/glm-5.2         # serves novel requests (no skill matched)
 # JUDGE_MODEL=google/gemini-2.5-flash   # the LLM judge, MUST differ from SKILLOPT_MODEL
 # JUDGE_MODELS=google/gemini-2.5-flash,deepseek/deepseek-chat-v3.1  # comma-separated = ensemble judge
-# OPENROUTER_PROVIDERS=fireworks,groq  # OpenRouter only: provider priority, tried in order; still
-#                                   # ZDR-only, uncovered models fall back to the open ZDR pool
+# OPENROUTER_PROVIDERS=groq          # provider priority, not an allowlist; still ZDR-only, and
+#                                   # uncovered models fall back to the open ZDR pool
 # Local inference (vLLM / Ollama, OpenAI-compatible), strongest privacy, no key needed when
 # nothing points at a hosted endpoint. From inside compose containers use your host's IP, not localhost:
 # MODEL_BASE_URL=http://172.17.0.1:8000/v1   # serving model only (agent, A/B, rollouts)
@@ -49,8 +43,9 @@ API_KEY=
 # SANDBOX_IMAGE=ingot-optimize     # image the sandbox containers run (repo code at /app)
 # SANDBOX_RUNTIME=runsc            # optional: gVisor runtime for kernel-level isolation
 
-# Candidate generation, SkillOpt reflective training loop (see README: Candidate generation).
-# OPTIMIZE_STRATEGY / OPTIMIZE_CANDIDATES were removed with the GEPA and best-of-N body loops.
+# SkillOpt optimization and reflective training (see docs/configuration.md).
+# OPTIMIZE_STRATEGY / OPTIMIZE_CANDIDATES were removed with the older body-search loops.
+# SKILLOPT_ROLLOUTS=direct         # direct (default) or agent (full scaffold, much higher cost)
 # SKILLOPT_EPOCHS=2                # passes over the train set
 # SKILLOPT_MINIBATCH=3             # train tasks reflected on per step
 # SKILLOPT_MAX_EDITS=3             # ceiling on edits applied per step (the learning-rate cap)

ARCHITECTURE.md

@@ -18,17 +18,19 @@ exposes only loopback ports.
    the first behavioral divergence, token cost, the split's leakage status, and the gate verdict.
    The description pass records router metrics (top-1, recall@3, no-route precision,
    cross-harness parity) with the same revisions and gate verdict; it has no rollouts to diff.
-4. The UI shows that evidence, the component diff, and one explicit approval action. Promotion
+4. The UI shows that evidence, a body-change risk summary, unified and side-by-side component
+   diffs, and explicit approve or reasoned-reject actions. Promotion
    re-checks the recorded gate verdict, that the champion revision still matches what is
    on disk, and that the challenger revision still matches the recorded evidence. It does not
    re-run collision checks or the held-out A/B.
    It then snapshots the revision it displaces into `runs/revisions/` and swaps directories
    atomically. The review card re-checks rewrite freshness when it renders, so a change whose
    champion has moved is refused before the approval click rather than after it.
 5. Rollback restores any snapshot the same way, and also snapshots what it displaces.
-6. Approval and rollback append metadata-only records to `runs/approval-audit.jsonl`. The actor is
+6. Approval, rejection, and rollback append metadata-only records to
+   `runs/approval-audit.jsonl`. A rejection may include an operator-supplied reason. The actor is
    the approver's authenticated identity (HTTP Basic username or OIDC email), or `local-operator` in
-   the zero-config open mode (see [SECURITY.md](SECURITY.md)).
+   explicit open mode (see [SECURITY.md](SECURITY.md)).
 
 ## Serving path
 
@@ -43,24 +45,38 @@ exposes only loopback ports.
    `LANGFUSE_*` points at); mining reads it back and has no local fallback. Hosted model calls use
    the configured OpenAI-compatible endpoint. OpenRouter calls always request ZDR providers.
 
-## Candidate generation (optional)
-
-Optimization proposes changes; it never makes them. It is off the review path and is expected to run
-in the background (`optimize.loop`) or on demand from the UI.
-
-Mining selects difficult, semantically diverse failures from real traces. The body pass runs one
-candidate search, `optimize/skillopt_loop.py` (SkillOpt's reflective training loop, driven
-per-component by `_greedy_search` in `optimize/ab.py`): bounded patch edits reflected from the seed's
-judged failures and prior rejected edits, accepted only against a held-out strictly-improving gate.
-The description pass (`optimize/routing.py`) scores
-candidate descriptions with the real embedding router and uses GEPA only for its reflection step.
+## SkillOpt optimization
+
+SkillOpt optimization is a core product capability. It proposes changes but never activates them.
+Runs start in the background (`optimize.loop`) or on demand from the UI, and every result enters the
+same human review path.
+
+Mining reads every usable Langfuse trace by default, with `--limit N` available only as an explicit
+operational cap. It attributes traffic by skill tag or embedding top-five relevance, clusters exact
+and semantic task variants locally, and judges one representative per cluster. Verdicts are cached
+by task, rubric, answer, judge configuration, and prompt. Cluster frequency weights health and
+failure counts, so every use contributes without one LLM call per use. A cold run judges at most
+`MINE_MAX_JUDGE_CALLS` new representatives, highest-frequency clusters first. The background loop
+defers health decisions while any cluster remains unjudged, then continues from the cache on its
+next run.
+
+Mined eval candidates are the lowest-scoring representatives selected by a deterministic
+`difficulty * novelty` greedy pass after train-set near-duplicates are removed. Frequency is
+reported as `occurrences`, but it does not multiply candidate rank. Mining never edits an eval set:
+an operator verifies each task and rubric, adds accepted failures to `train`, and authors new
+`holdout` cases separately so the promotion gate remains uncontaminated.
+
+The body pass runs one candidate search, `optimize/skillopt_loop.py` (SkillOpt's reflective training
+loop, driven per-component by `_greedy_search` in `optimize/ab.py`): bounded patch edits reflected
+from the seed's judged failures and prior rejected edits, accepted only against a held-out
+strictly-improving gate.
+The description pass (`optimize/routing.py`) uses SkillOpt bounded edits and scores candidate
+descriptions with the real embedding router.
 Both passes end at a quarantined pending record and an evidence bundle under `runs/evidence/`.
 
-There is one candidate search, by design. A second, sequential GEPA body loop was removed: it
-optimized the same objective at roughly twenty times the cost, was reachable only through an opt-in
-flag, and had no test coverage of its own. `OPTIMIZE_STRATEGY` is no longer read, and the removed
-pass flags (`--strategy`, `--gepa`, `--skip-gepa`, `--candidates`) fail the argument parse rather than
-being silently ignored. The scripts pass (`--scripts`) optimizes bundled
+There is one SkillOpt candidate search per component, by design. `OPTIMIZE_STRATEGY` and
+`OPTIMIZE_CANDIDATES` are no longer read, and their removed CLI flags fail argument parsing rather
+than being silently ignored. The scripts pass (`--scripts`) optimizes bundled
 `scripts/` files, but only when the skill's holdout carries execution-grounded `check:` assertions,
 since the judge alone cannot tell a broken script from a working one; when it runs, both the
 candidate rollouts and the A/B serve the assembled skill (body plus files), so a rewritten file is
@@ -83,7 +99,6 @@ text components (`OPTIMIZE_COMPONENTS=body,file:<path>`), diffed for review and
   constrains which skills the application can act on (promotion, rollback, candidate
   generation, and the review surface all refuse a non-slug name), not which ones a library root
   can serve.
-- Hosted credentials are read from environment variables and are never stored in traces.
 
 ## Stores and ownership
 
@@ -104,8 +119,7 @@ Appends to the approval trail open the file with `O_APPEND` and loop until the w
 written, so a short write cannot leave a half record; readers skip unparseable and non-UTF-8 lines
 either way.
 
-Pending records store candidate-search scores under `inner_loop`. Records written before the GEPA
-body loop was removed used `gepa`; the review API reads either, so an existing queue still renders.
+Pending records store SkillOpt candidate-search scores under `inner_loop`.
 
 ### Recovering a promotion killed between its two renames
 
@@ -131,8 +145,8 @@ revision the approval trail last recorded.
 
 `docker compose up` runs MCP, the UI, and the self-hosted Langfuse evals backend, with one-shot
 agent and candidate-generation containers on demand. Point `LANGFUSE_*` at an external Langfuse
-(Cloud or self-hosted) to send traces there instead; the bundled observability services still start
-unless you stop them. Fully local inference points model
+(Cloud or self-hosted) and include `docker-compose.external-langfuse.yml` to keep the bundled
+observability services stopped. Fully local inference points model
 URLs to vLLM or Ollama. Hosted inference sends prompts and outputs to the selected provider. Endpoint
 auth is independent of these modes: the UI has its own password/OIDC gate (`AUTH_MODE`), but MCP has
 none, so publishing MCP off-loopback requires an authenticating proxy and authorization appropriate to
@@ -165,8 +179,9 @@ candidate-generation and review services for execution-grounded judging. Do not
 for untrusted users. Prefer a dedicated Docker context or isolated host, and omit the socket when
 using static checks.
 
-MCP has no built-in authentication; the UI carries a password gate (or optional OIDC), unauthenticated
-only in the zero-config open mode, where anyone who can reach it can approve a change or roll one back.
+MCP has no built-in authentication. Compose password-gates the UI by default, and OIDC is available
+for shared deployments. In explicit open mode, anyone who can reach the UI can approve a change or
+roll one back.
 Skills can contain hostile instructions or executable assets. Hosted providers receive model traffic.
-Local traces can contain task and answer text. These boundaries and concrete deployment safeguards
+Langfuse traces can contain task and answer text. These boundaries and concrete deployment safeguards
 are expanded in [SECURITY.md](SECURITY.md).

CONTRIBUTING.md

@@ -20,6 +20,6 @@ Before opening a pull request:
 1. Run the full Docker test suite.
 2. Confirm `docker compose config --quiet` succeeds when Compose files change.
 3. Describe user-visible behavior, risks, and verification evidence.
-4. Run the repository's Code Health safeguard when available and fix regressions.
+4. Run CodeScene only when the pull request or reviewer explicitly asks for it.
 
 Report security issues through the private process in [SECURITY.md](SECURITY.md), not a public issue.

Dockerfile

@@ -18,15 +18,19 @@ RUN arch=$([ "${TARGETARCH:-$(dpkg --print-architecture)}" = "arm64" ] && echo a
     && rm -rf /tmp/docker.tgz /tmp/docker
 
 # Pre-download the CPU embedding models so the demo runs offline (no runtime download):
-# the Qwen3 q4 ONNX router default (~460MB), plus the bge-small fastembed fallback (~34MB) so an
-# EMBED_MODEL override to the previous default also works offline. Keep these in sync with the
+# the Qwen3 q4 ONNX router default (~925MB including its tokenizer), plus the bge-small fastembed
+# fallback (~34MB), so an EMBED_MODEL override to the previous default also works offline. Keep
+# these in sync with the
 # EMBED_MODEL / EMBED_ONNX_FILE envs the server reads, or the model downloads on first boot.
 ARG EMBED_MODEL=onnx-community/Qwen3-Embedding-0.6B-ONNX
 ARG EMBED_ONNX_FILE=onnx/model_q4.onnx
 RUN python -c "from huggingface_hub import hf_hub_download as fetch; \
 fetch('${EMBED_MODEL}', 'tokenizer.json'); fetch('${EMBED_MODEL}', '${EMBED_ONNX_FILE}')"
+ENV BAKED_EMBED_MODEL=${EMBED_MODEL} \
+    BAKED_EMBED_ONNX_FILE=${EMBED_ONNX_FILE}
 ARG FALLBACK_EMBED_MODEL=BAAI/bge-small-en-v1.5
 RUN python -c "from fastembed import TextEmbedding; TextEmbedding('${FALLBACK_EMBED_MODEL}')"
+ENV BAKED_FALLBACK_EMBED_MODEL=${FALLBACK_EMBED_MODEL}
 
 COPY mcp_server ./mcp_server
 COPY agent ./agent

PRODUCTION_SETUP.md

@@ -50,6 +50,13 @@ REDIS_AUTH=replace-with-a-random-password
 
 AUTH_USER=admin
 AUTH_PASSWORD=replace-with-a-strong-password
+
+BASE_URL=https://openrouter.ai/api/v1
+API_KEY=sk-or-replace-me
+AGENT_MODEL=qwen/qwen3-32b
+SKILLOPT_MODEL=z-ai/glm-5.2
+JUDGE_MODEL=google/gemini-2.5-flash
+OPENROUTER_PROVIDERS=groq
 ```
 
 Generate suitable random values with OpenSSL:
@@ -59,7 +66,10 @@ openssl rand -hex 32  # LANGFUSE_ENCRYPTION_KEY and LANGFUSE_NEXTAUTH_SECRET
 openssl rand -hex 16  # salts, API key suffixes, and service passwords
 ```
 
-Also configure the model provider variables described in [Configuration](docs/configuration.md).
+OpenRouter is the supported hosted model path. Provider priorities are preferences, not an
+allowlist: if Groq does not serve a configured role, Ingot falls back to another ZDR-qualified
+OpenRouter endpoint. Fully local vLLM or Ollama is also supported as described in
+[Configuration](docs/configuration.md).
 For a shared team deployment, consider OIDC for the change-control UI as described in
 [SSO](docs/sso.md).