Make Langfuse the default evals backend; remove local-JSONL lite mode

.env.example

@@ -12,7 +12,7 @@ API_KEY=
 #   https://docs.fireworks.ai/guides/security_compliance/data_handling):
 #     BASE_URL=https://api.fireworks.ai/inference/v1   API_KEY=fw_...
 #     AGENT_MODEL=accounts/fireworks/models/qwen3p7-plus
-#     GEPA_MODEL=accounts/fireworks/models/glm-5p2
+#     SKILLOPT_MODEL=accounts/fireworks/models/glm-5p2
 #     JUDGE_MODEL=accounts/fireworks/models/deepseek-v4-pro
 #   Together direct:   BASE_URL=https://api.together.xyz/v1   API_KEY=...
 #   Local vLLM/Ollama: BASE_URL=http://172.17.0.1:8000/v1     (no key needed)
@@ -23,10 +23,9 @@ API_KEY=
 # Optional overrides:
 # AGENT_MODEL=qwen/qwen3.6-27b      # the agent model: everything that *executes* skills (also
 #                                   # candidate rollouts); MODEL is the legacy alias
-# GEPA_MODEL=z-ai/glm-5.2           # the teacher LM: writes candidate skills, and reflects for the
-#                                   # description pass's GEPA loop (legacy name, kept for .env files)
+# SKILLOPT_MODEL=z-ai/glm-5.2       # authors evals and skill revisions
 # STRONG_MODEL=z-ai/glm-5.2         # serves novel requests (no skill matched)
-# JUDGE_MODEL=google/gemini-2.5-flash   # the LLM judge, MUST differ from GEPA_MODEL (anti reward-hacking)
+# JUDGE_MODEL=google/gemini-2.5-flash   # the LLM judge, MUST differ from SKILLOPT_MODEL
 # JUDGE_MODELS=google/gemini-2.5-flash,deepseek/deepseek-chat-v3.1  # comma-separated = ensemble judge
 # OPENROUTER_PROVIDERS=fireworks,groq  # OpenRouter only: provider priority, tried in order; still
 #                                   # ZDR-only, uncovered models fall back to the open ZDR pool
@@ -37,21 +36,13 @@ API_KEY=
 
 # Routing model + thresholds. The defaults are calibrated TOGETHER on a drafted routing eval;
 # if you override EMBED_MODEL, recalibrate the three scores with it (cosine distributions differ
-# per model — for the previous default BAAI/bge-small-en-v1.5 use 0.65 / 0.45 / 0.93):
+# per model. For the previous default BAAI/bge-small-en-v1.5 use 0.65 / 0.45 / 0.93):
 # EMBED_MODEL=onnx-community/Qwen3-Embedding-0.6B-ONNX   # or any fastembed model name
 # MIN_SCORE=0.53                   # at/above -> routable match; below -> related band or novel
 # RELATED_SCORE=0.37               # floor of the "related" (compose/extend) band; below it a task
 #                                  # is novel (weak/strong escalation)
 # COLLISION_SCORE=0.94             # a rewritten description this close to another skill is a routing hack
 
-# Local trace privacy and retention:
-# LOCAL_TRACE_ENABLED=true         # false disables the plaintext local JSONL store
-# LOCAL_TRACE_REDACT=true          # redact common credentials before writing
-# LOCAL_TRACE_MAX_AGE_DAYS=30      # prune older records on append; 0 disables age retention
-# LOCAL_TRACE_MAX_BYTES=10485760   # rotate after 10 MiB
-# LOCAL_TRACE_BACKUPS=3            # retained rotated files
-# TRACES_FILE=/app/runs/traces.jsonl
-
 # Execution-grounded judging (sandboxed by default; fails closed when docker is unreachable):
 # EXEC_SANDBOX=docker              # docker = locked-down sandbox container (default),
 #                                  # 1 = bare subprocess (legacy), off = static checks only
@@ -66,6 +57,11 @@ API_KEY=
 # SKILLOPT_GATE_METRIC=mixed       # inner accept/reject metric: hard | soft | mixed
 # SKILLOPT_GATE_MIXED_WEIGHT=0.5   # weight on soft (mean-judge) when metric=mixed
 
+# Production trace mining. The default reads every Langfuse trace page, clusters paraphrased uses
+# locally, and judges only one representative per cluster. Verdicts persist in runs/mine-cache/.
+# MINE_MAX_JUDGE_CALLS=24          # new representative judge calls per run; <=0 removes the cap
+# MINE_CLUSTER_THRESHOLD=0.90      # task cosine at/above this shares a representative verdict
+
 # Change-control UI login. Three modes; AUTH_MODE picks one (compose default: password).
 # See docs/sso.md. To share the UI beyond this machine, use the TLS front door:
 # `docker compose --profile lan up -d proxy` (docs/security.md "Network exposure").
@@ -94,14 +90,41 @@ API_KEY=
 # PROMOTE_MIN_SAMPLES=3            # min held-out tasks before a promote is allowed
 # PROMOTE_PASS_SCORE=0.5           # a task "passes" at/above this (used for the no-regression check)
 
-# Langfuse: by default the compose stack runs its own (UI http://localhost:3100, login
-# demo@local.dev / localdemo123, local demo literals, not secrets) and these stay unset.
-# To use your OWN Langfuse project instead, set all three to that project's values
-# (Project Settings -> API Keys) and restart the stack (docker compose up -d):
-# LANGFUSE_PUBLIC_KEY=pk-lf-...
+# Evals backend (Langfuse). Langfuse is the default and comes up with `docker compose up` (UI
+# http://localhost:3100, login demo@local.dev / localdemo123). Trace mining has NO local fallback:
+# it fails loudly unless a Langfuse-compatible endpoint is reachable, so either run the bundled
+# stack or point the three vars below at your own provider. These stay unset for the bundled stack.
+#
+# Use your OWN Langfuse (Cloud or self-hosted elsewhere) instead of the bundled one:
+# LANGFUSE_PUBLIC_KEY=pk-lf-...      # Project Settings -> API Keys
 # LANGFUSE_SECRET_KEY=sk-lf-...
-# LANGFUSE_BASE_URL=https://cloud.langfuse.com   # or your self-hosted URL, reachable FROM INSIDE
-#                                  # the containers: another server on this machine is
-#                                  # http://host.docker.internal:<port>, NOT http://localhost:<port>
-# LANGFUSE_PUBLIC_URL=https://cloud.langfuse.com # where YOUR BROWSER reaches that Langfuse
-#                                  # (the UI links traces here; defaults to localhost:3100)
+# LANGFUSE_BASE_URL=https://cloud.langfuse.com   # reachable FROM INSIDE the containers: another
+#                                  # server on this machine is http://host.docker.internal:<port>,
+#                                  # NOT http://localhost:<port>
+# LANGFUSE_PUBLIC_URL=https://cloud.langfuse.com # where YOUR BROWSER reaches it (UI trace links;
+#                                  # defaults to localhost:3100)
+# Start Ingot with docker-compose.yml plus docker-compose.external-langfuse.yml (Docker Compose
+# 2.24.4 or newer); the override removes bundled dependencies and leaves those containers stopped.
+# A different evals platform (Arize etc.) is not wired yet; see docs/mcp-integration.md
+# (Using your own evals platform) for the trace contract and the fetch_traces seam.
+#
+# LANGFUSE SECURITY. The bundled stack ships demo credentials, safe ONLY because it is bound to
+# 127.0.0.1 and its datastores publish no host port. Before the FIRST boot of any deployment that
+# will be exposed, set every value below. For an initialized deployment, changing .env alone does
+# not rotate stored database, user, or project credentials; follow docs/security.md first.
+# Then set LANGFUSE_HOST and LANGFUSE_BIND_ADDRESS to this box's trusted DNS name or private IP,
+# set LANGFUSE_PUBLIC_URL=https://<same-name-or-IP>:3443, and start the separate opt-in TLS endpoint
+# with `docker compose --profile langfuse-lan up -d langfuse-proxy`.
+# LANGFUSE_HOST=192.168.1.40         # exact DNS name or IP included in Caddy's certificate
+# LANGFUSE_BIND_ADDRESS=192.168.1.40 # trusted host interface that publishes port 3443
+# LANGFUSE_ENCRYPTION_KEY=          # 64 hex chars: `openssl rand -hex 32` (encrypts stored secrets)
+# LANGFUSE_SALT=                    # `openssl rand -hex 16`
+# LANGFUSE_NEXTAUTH_SECRET=         # `openssl rand -hex 32` (signs Langfuse web sessions)
+# LANGFUSE_INIT_USER_EMAIL=you@corp.example    # the Langfuse web login (host :3100)
+# LANGFUSE_INIT_USER_PASSWORD=      # a strong password; default is localdemo123
+# POSTGRES_PASSWORD=                # datastore creds; the same var drives Langfuse's connection
+# CLICKHOUSE_PASSWORD=
+# MINIO_ROOT_USER=
+# MINIO_ROOT_PASSWORD=
+# REDIS_AUTH=
+# (rotate LANGFUSE_PUBLIC_KEY / _SECRET_KEY above too; they double as the project API keys)

.github/workflows/ci.yml

@@ -14,3 +14,5 @@ jobs:
         run: docker build -t ingot-mcp .
       - name: Run tests
         run: docker run --rm -v "$PWD:/app" -w /app ingot-mcp python -m pytest tests -q
+      - name: Smoke test Compose and Langfuse TLS
+        run: ./scripts/compose_smoke.sh

ARCHITECTURE.md

@@ -39,7 +39,8 @@ exposes only loopback ports.
 3. One response is authoritative for the direct `match` or explicit `related_match`, loaded body,
    revision, root, body-free alternatives, and `novel` escalation signal. A related match is loaded
    for compose-or-extend use. The agent uses the weak model unless `novel` is true.
-4. The run can be recorded in local JSONL and, when configured, Langfuse. Hosted model calls use
+4. The run is recorded to Langfuse (the default evals backend, or a Langfuse-compatible endpoint
+   `LANGFUSE_*` points at); mining reads it back and has no local fallback. Hosted model calls use
    the configured OpenAI-compatible endpoint. OpenRouter calls always request ZDR providers.
 
 ## Candidate generation (optional)
@@ -92,8 +93,7 @@ rollback snapshots, plus a `.snapshots.json` index per skill that records when e
 last snapshotted; it sits beside the snapshot directories, never inside one, so a rollback restores
 the skill and nothing else. `runs/evidence/` contains evidence bundles, and pending records name
 them relative to the repository root. `runs/approval-audit.jsonl` contains the approval trail.
-`runs/traces.jsonl` contains local traces. Langfuse uses its own Compose Postgres, ClickHouse, and
-object stores.
+Traces live in Langfuse, which uses its own Compose Postgres, ClickHouse, and object stores.
 
 Local JSON and JSONL stores assume one logical writer per skill or file. Atomic replacement protects
 pending records and skill promotion, but concurrent candidate, approval, or rollback processes are
@@ -102,12 +102,7 @@ generation, and refuses a second approval or rollback with HTTP 409 while one is
 clicks cannot interleave the snapshot, stage, and swap steps of a promotion.
 Appends to the approval trail open the file with `O_APPEND` and loop until the whole record is
 written, so a short write cannot leave a half record; readers skip unparseable and non-UTF-8 lines
-either way. Trace appends use one operating-system append write per record. Operators should still
-mount a single local writer when using network filesystems.
-
-Local trace schema version 1 contains `schema_version`, Unix `ts`, `task`, `answer`, and `tags`.
-Readers also accept the original unversioned shape as version 1. See README privacy controls for
-opt-out, redaction, permissions, and rotation.
+either way.
 
 Pending records store candidate-search scores under `inner_loop`. Records written before the GEPA
 body loop was removed used `gepa`; the review API reads either, so an existing queue still renders.
@@ -134,8 +129,10 @@ revision the approval trail last recorded.
 
 ## Deployment modes
 
-Lite mode runs MCP and the UI, with one-shot agent and candidate-generation containers on demand.
-The optional `langfuse` profile adds local observability services. Fully local inference points model
+`docker compose up` runs MCP, the UI, and the self-hosted Langfuse evals backend, with one-shot
+agent and candidate-generation containers on demand. Point `LANGFUSE_*` at an external Langfuse
+(Cloud or self-hosted) to send traces there instead; the bundled observability services still start
+unless you stop them. Fully local inference points model
 URLs to vLLM or Ollama. Hosted inference sends prompts and outputs to the selected provider. Endpoint
 auth is independent of these modes: the UI has its own password/OIDC gate (`AUTH_MODE`), but MCP has
 none, so publishing MCP off-loopback requires an authenticating proxy and authorization appropriate to
@@ -144,8 +141,9 @@ the available tools.
 ## Failure and recovery
 
 MCP refresh compares file signatures and rebuilds only after change; unchanged description vectors
-are reused. A failed refresh keeps the previous state until a later request retries. Trace and
-Langfuse failures do not fail serving. Malformed local trace lines are skipped by mining.
+are reused. A failed refresh keeps the previous state until a later request retries. Langfuse
+callback failures do not fail serving, but mining fails loudly when the evals backend is unreachable
+rather than returning an empty result that would read as "nothing failing".
 
 Promotion stages changes and restores the prior directory if the swap fails. Every rewrite stores
 the displaced revision. Restore it from the UI's History section, or with: