Make Langfuse the default evals backend; remove local-JSONL lite mode#37
Merged
Conversation
…e mode
Langfuse is now the default trace/evals backend, up with `docker compose up`.
Trace mining has no local fallback: it fails loudly when no Langfuse-compatible
endpoint is reachable, instead of silently returning "nothing failing".
Code:
- Remove the local JSONL trace store (agent/traces.py) and its writer call in
agent/run.py; remove optimize.traces_file() and TRACES_FILE / LOCAL_TRACE_*.
- optimize/mine.py: drop the local-trace fallback (6 helpers); fetch_traces()
raises SystemExit with guidance when the backend is unreachable. Keep the
dual-shape _task_answer parsing (still used by BYOA connectors).
- Reword remaining "lite mode" comments in ab.py / compat.py (the gate's
Langfuse-free execution path is kept; only the trace store is gone).
- tests: delete test_local_traces.py; add a fail-loud fetch_traces test.
Compose:
- Bring the Langfuse stack up by default (remove `profiles: ["langfuse"]` from
the 6 services); wire depends_on: langfuse-web (healthy) on agent + the mining
services; fix the stale "falls back to local" comments.
- Security: keep everything bound to 127.0.0.1 with datastores unpublished, and
templatize every Langfuse secret as ${VAR:-demo} so it's overridable via .env
without editing compose (SALT, ENCRYPTION_KEY, NEXTAUTH_SECRET, DB/CH/MinIO/
Redis creds, project keys, web login).
Docs:
- .env.example: drop LOCAL_TRACE_*/TRACES_FILE; add a "LANGFUSE SECURITY" block
(rotate-before-exposing) and the your-own-Langfuse guidance.
- security.md: new "Securing the Langfuse deployment" section.
- mcp-integration.md: new "Using your own evals platform" section — bundled /
own Langfuse / other platform (Arize), documenting the fetch_traces read seam
as the extension point (write side is OTel today; read-side adapters planned).
- README, ARCHITECTURE, configuration, tutorial: replace lite-mode / local-JSONL
framing with Langfuse-as-default.
Verified in Docker: image builds, 476 passed / 3 skipped, `docker compose config`
valid with Langfuse in the default profile.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01URfKaRsuJqkcVQPkHzxKfP
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughThe change makes Langfuse the required trace backend, removes local JSONL tracing and fallback mining, adds clustered trace mining and Compose orchestration, and updates model configuration, documentation, connector setup, security guidance, and integration smoke tests. ChangesLangfuse trace backend
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant Agent as agent._serve
participant MCP as Ingot MCP
participant Langfuse
participant Miner as optimize.mine
participant Judge as Mining judge
Agent->>MCP: Route and load skill
MCP->>Langfuse: Record root trace
Miner->>Langfuse: Fetch paginated traces
Langfuse-->>Miner: Return normalized trace roots
Miner->>Judge: Judge clustered representatives
Judge-->>Miner: Return cached or new verdicts
Miner-->>Miner: Select weighted mined candidates
Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
optimize/mine.py (1)
24-45: 🔒 Security & Privacy | 🟡 Minor | ⚡ Quick winRestrict
LANGFUSE_BASE_URLto HTTP(S)
urllib.request.urlopenacceptsfile://and other schemes here, so a bad env value can read local files instead of reaching Langfuse. Allowlisthttpandhttps, or add a narrowly justified suppression if non-HTTP schemes are intentional.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@optimize/mine.py` around lines 24 - 45, Validate LANGFUSE_BASE_URL in fetch_traces before constructing or opening the request, allowing only HTTP and HTTPS schemes and rejecting all others with a clear configuration error. Preserve the existing Langfuse request behavior for valid URLs.Source: Linters/SAST tools
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@ARCHITECTURE.md`:
- Line 96: Update the tracing storage section in ARCHITECTURE.md around the
Langfuse statement and the local trace append/schema text to remove obsolete
local JSONL trace-store references. Ensure the section consistently documents
Langfuse-only storage and does not instruct operators to expect or configure a
local trace store.
In `@docker-compose.yml`:
- Line 330: Update the Redis healthcheck configuration associated with the Redis
service so its redis-cli ping authenticates using the configured REDIS_AUTH
password before checking availability. Preserve the existing default password
behavior from --requirepass and ensure the authenticated check allows
langfuse-web and the worker to start.
- Around line 60-64: Make the external-Langfuse configuration actually bypass
the bundled langfuse-web dependency: update docker-compose.yml around the mcp
depends_on entry with a profile or override keyed by LANGFUSE_* that prevents
starting or waiting for langfuse-web. Update the corresponding claims in
ARCHITECTURE.md, README.md (both affected sections), docs/configuration.md, and
docs/mcp-integration.md to match the implemented compose behavior; each listed
site requires documentation changes unless the compose implementation makes its
existing statement accurate.
In `@docs/mcp-integration.md`:
- Around line 21-25: Replace every em dash in the specified documentation prose
with permitted punctuation, preserving the existing meaning: update
docs/mcp-integration.md ranges 21-25, 32-37, 63-67, and 75-95, and
docs/security.md ranges 21-25 and 133-140. No direct changes beyond replacing
the em dashes are needed.
---
Outside diff comments:
In `@optimize/mine.py`:
- Around line 24-45: Validate LANGFUSE_BASE_URL in fetch_traces before
constructing or opening the request, allowing only HTTP and HTTPS schemes and
rejecting all others with a clear configuration error. Preserve the existing
Langfuse request behavior for valid URLs.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 4fc0422f-7b7a-401e-8c4a-2ff46216b1f5
📒 Files selected for processing (16)
.env.exampleARCHITECTURE.mdREADME.mdagent/run.pyagent/traces.pydocker-compose.ymldocs/configuration.mddocs/mcp-integration.mddocs/security.mddocs/tutorial.mdoptimize/__init__.pyoptimize/ab.pyoptimize/compat.pyoptimize/mine.pytests/test_local_traces.pytests/test_mine.py
💤 Files with no reviewable changes (4)
- agent/traces.py
- optimize/init.py
- tests/test_local_traces.py
- agent/run.py
- ARCHITECTURE.md: remove the leftover present-tense description of the deleted local JSONL trace store (append semantics + schema paragraph). - Redis healthcheck: authenticate it (`redis-cli -a "$REDIS_AUTH" ... ping`). The old unauthenticated ping exits 0 on NOAUTH so it never actually blocked langfuse-web, but now that REDIS_AUTH is overridable an authenticated check catches a mismatched password. Verified: passes with the right password, fails with a wrong one. - Correct the external-Langfuse claims: pointing LANGFUSE_* at your own backend does not skip the bundled containers (they are default-on and the agent depends_on langfuse-web). README / ARCHITECTURE / configuration / mcp-integration now say the bundled stack keeps running unless you stop it. - Replace em dashes with commas/colons/semicolons in the changed prose and comments, per the repo guideline (mcp-integration, security, compose, .env.example). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01URfKaRsuJqkcVQPkHzxKfP
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docker-compose.yml`:
- Around line 8-12: Update the docker-compose security comment to avoid claiming
every listed value is overridable through .env: either template
LANGFUSE_INIT_ORG_ID, LANGFUSE_INIT_ORG_NAME, LANGFUSE_INIT_PROJECT_ID,
LANGFUSE_INIT_PROJECT_NAME, and LANGFUSE_INIT_USER_NAME, or narrow the claim to
credentials and secrets only.
- Around line 331-333: Update the Redis service command configuration to list
form so --requirepass and the REDIS_AUTH value are separate argv items,
preserving the full password—including spaces or quotes—as one argument while
retaining the existing --maxmemory-policy option.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 68f11556-c6fd-4ac5-99c8-63578f805069
📒 Files selected for processing (7)
.env.exampleARCHITECTURE.mdREADME.mddocker-compose.ymldocs/configuration.mddocs/mcp-integration.mddocs/security.md
🚧 Files skipped from review as they are similar to previous changes (5)
- README.md
- docs/configuration.md
- docs/security.md
- .env.example
- ARCHITECTURE.md
- Redis command: switch the folded-scalar string to list form so REDIS_AUTH stays
a single argv item even with spaces/quotes (the healthcheck uses the full value;
the folded form would split it). Verified: `REDIS_AUTH='p a"b' docker compose
config` yields ["--requirepass","p a\"b",...]; redis comes up healthy.
- Narrow the compose security note: only secrets are ${VAR:-demo} overridable; the
non-secret LANGFUSE_INIT_ org/project/user names stay literal.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01URfKaRsuJqkcVQPkHzxKfP
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/mcp-integration.md`:
- Around line 38-40: Update the coding-agent connector contract in the
surrounding documentation to state “one of three shapes” instead of “one of two
shapes,” preserving the listed accepted input/output formats and setup guidance.
In `@ops/caddy/Langfuse.Caddyfile`:
- Around line 3-6: Update the :3443 TLS configuration in the Caddyfile to remove
on_demand certificate issuance, or replace it with an ask policy that authorizes
requests; ensure the listener is bound to the intended trusted LAN interface
rather than every interface when required.
In `@scripts/claude_setup.sh`:
- Around line 20-24: The Claude setup logic in scripts/claude_setup.sh must
inspect the user-scoped ingot entry explicitly before deciding setup is
complete, so entries from other scopes do not suppress the required user-scoped
add. In scripts/codex_setup.sh, validate the existing ingot URL against MCP_URL
and remove and re-add the entry when it differs; retain existing entries only
when both name and URL match.
In `@scripts/codex_setup.sh`:
- Around line 34-35: Update scripts/codex_setup.sh after the Langfuse plugin
installation to configure ~/.codex/config.toml with plugin_hooks = true and
enable tracing@codex-observability-plugin, and ensure TRACE_TO_LANGFUSE=true is
configured before setup completes.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: d4b81be9-0660-42b4-9d39-ee4b3396c90b
📒 Files selected for processing (10)
.env.exampleREADME.mddocker-compose.ymldocs/mcp-integration.mddocs/security.mdops/caddy/Langfuse.Caddyfileoptimize/mine.pyscripts/claude_setup.shscripts/codex_setup.shtests/test_mine.py
🚧 Files skipped from review as they are similar to previous changes (5)
- .env.example
- docs/security.md
- optimize/mine.py
- README.md
- docker-compose.yml
Contributor
Author
|
@coderabbitai review |
✅ Action performedReview finished.
|
There was a problem hiding this comment.
Actionable comments posted: 7
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@optimize/__init__.py`:
- Around line 31-33: Update skillopt_model() to treat an empty SKILLOPT_MODEL
environment value as unset by applying the same fallback behavior as
agent_model() and strong_model(), returning "z-ai/glm-5.2" for blank values. Add
a test verifying the empty environment case.
In `@optimize/mine.py`:
- Around line 61-97: Keep the API request page size constant in the trace-fetch
loop by using TRACE_PAGE_SIZE for every query, rather than deriving page_limit
from the remaining output limit. Continue enforcing the caller’s limit only when
appending to out and returning once it is reached; preserve the existing
pagination and termination checks.
In `@README.md`:
- Around line 70-74: Update the README tutorial sentence beginning “Then walk
the full loop” to remove the awkward comma construction and improve its flow,
while preserving the meaning and links. In the following sentence, change “open
source” to the hyphenated compound “open-source.”
In `@scripts/claude_langfuse_smoke.sh`:
- Around line 62-66: Add the Docker host-gateway mapping option to both
smoke-test containers: scripts/claude_langfuse_smoke.sh lines 62-66 and
scripts/codex_langfuse_smoke.sh lines 53-57. Update each docker run invocation
so host.docker.internal resolves to the host on plain Linux Docker Engine while
preserving the existing Langfuse environment variables and command.
In `@scripts/claude_setup.sh`:
- Around line 13-15: Update scripts/claude_setup.sh lines 13-15 to reject an
explicitly configured remote LANGFUSE_BASE_URL unless LANGFUSE_PUBLIC_KEY and
LANGFUSE_SECRET_KEY are both provided, while preserving local demo defaults.
Update docs/mcp-integration.md lines 89-90 and PRODUCTION_SETUP.md lines 141-143
to include LANGFUSE_BASE_URL, LANGFUSE_PUBLIC_KEY, and LANGFUSE_SECRET_KEY in
the documented remote Claude commands.
In `@scripts/codex_setup.sh`:
- Around line 80-91: Configure Codex hooks and the tracing plugin in
scripts/codex_setup.sh lines 80-91 by safely creating or updating
~/.codex/config.toml with [features] plugin_hooks = true and the enabled
tracing-plugin entry; update tests/test_setup_scripts.py lines 67-77 to assert
both settings are generated or preserved, and update docs/mcp-integration.md
lines 61-62 to claim tracing is enabled only after this configuration is
applied.
In `@tests/test_env_check.py`:
- Around line 198-204: Update test_judge_warns_when_skillopt_model_is_the_grader
to remove or override the inherited JUDGE_MODELS environment variable before
launching the subprocess, while preserving the SKILLOPT_MODEL and JUDGE_MODEL
values and the existing warning assertion.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 7d9c6c42-c370-43c4-814b-ce9985b8cd83
📒 Files selected for processing (33)
.env.example.github/workflows/ci.ymlPRODUCTION_SETUP.mdREADME.mdagent/run.pydocker-compose.external-langfuse.ymldocker-compose.ymldocs/configuration.mddocs/evidence-gate.mddocs/mcp-integration.mddocs/security.mddocs/sso.mddocs/tutorial.mdops/caddy/Langfuse.Caddyfileoptimize/__init__.pyoptimize/draft.pyoptimize/judge.pyoptimize/loop.pyoptimize/mine.pyoptimize/rollout.pyoptimize/usage.pyscripts/claude_langfuse_smoke.shscripts/claude_setup.shscripts/codex_langfuse_smoke.shscripts/codex_setup.shscripts/compose_smoke.shtests/test_env_check.pytests/test_lite_mode.pytests/test_loop.pytests/test_mine.pytests/test_run_task.pytests/test_security.pytests/test_setup_scripts.py
🚧 Files skipped from review as they are similar to previous changes (3)
- agent/run.py
- docs/security.md
- docker-compose.yml
JamesMaki
added a commit
that referenced
this pull request
Jul 22, 2026
- ARCHITECTURE.md: remove the leftover present-tense description of the deleted local JSONL trace store (append semantics + schema paragraph). - Redis healthcheck: authenticate it (`redis-cli -a "$REDIS_AUTH" ... ping`). The old unauthenticated ping exits 0 on NOAUTH so it never actually blocked langfuse-web, but now that REDIS_AUTH is overridable an authenticated check catches a mismatched password. Verified: passes with the right password, fails with a wrong one. - Correct the external-Langfuse claims: pointing LANGFUSE_* at your own backend does not skip the bundled containers (they are default-on and the agent depends_on langfuse-web). README / ARCHITECTURE / configuration / mcp-integration now say the bundled stack keeps running unless you stop it. - Replace em dashes with commas/colons/semicolons in the changed prose and comments, per the repo guideline (mcp-integration, security, compose, .env.example). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01URfKaRsuJqkcVQPkHzxKfP
JamesMaki
added a commit
that referenced
this pull request
Jul 22, 2026
- Redis command: switch the folded-scalar string to list form so REDIS_AUTH stays
a single argv item even with spaces/quotes (the healthcheck uses the full value;
the folded form would split it). Verified: `REDIS_AUTH='p a"b' docker compose
config` yields ["--requirepass","p a\"b",...]; redis comes up healthy.
- Narrow the compose security note: only secrets are ${VAR:-demo} overridable; the
non-secret LANGFUSE_INIT_ org/project/user names stay literal.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01URfKaRsuJqkcVQPkHzxKfP
JamesMaki
added a commit
that referenced
this pull request
Jul 22, 2026
Make Langfuse the default evals backend; remove local-JSONL lite mode
JamesMaki
added a commit
that referenced
this pull request
Jul 22, 2026
- ARCHITECTURE.md: remove the leftover present-tense description of the deleted local JSONL trace store (append semantics + schema paragraph). - Redis healthcheck: authenticate it (`redis-cli -a "$REDIS_AUTH" ... ping`). The old unauthenticated ping exits 0 on NOAUTH so it never actually blocked langfuse-web, but now that REDIS_AUTH is overridable an authenticated check catches a mismatched password. Verified: passes with the right password, fails with a wrong one. - Correct the external-Langfuse claims: pointing LANGFUSE_* at your own backend does not skip the bundled containers (they are default-on and the agent depends_on langfuse-web). README / ARCHITECTURE / configuration / mcp-integration now say the bundled stack keeps running unless you stop it. - Replace em dashes with commas/colons/semicolons in the changed prose and comments, per the repo guideline (mcp-integration, security, compose, .env.example).
JamesMaki
added a commit
that referenced
this pull request
Jul 22, 2026
- Redis command: switch the folded-scalar string to list form so REDIS_AUTH stays
a single argv item even with spaces/quotes (the healthcheck uses the full value;
the folded form would split it). Verified: `REDIS_AUTH='p a"b' docker compose
config` yields ["--requirepass","p a\"b",...]; redis comes up healthy.
- Narrow the compose security note: only secrets are ${VAR:-demo} overridable; the
non-secret LANGFUSE_INIT_ org/project/user names stay literal.
JamesMaki
added a commit
that referenced
this pull request
Jul 22, 2026
Make Langfuse the default evals backend; remove local-JSONL lite mode
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Langfuse becomes the default and required trace/evals backend — it comes up with
docker compose up, and trace mining has no local fallback: it fails loudly when no Langfuse-compatible endpoint is reachable, instead of silently returning "nothing failing".Behavior
docker compose upnow brings up the self-hosted Langfuse stack alongside MCP/UI/agent.optimize-mine/ the background loop require a reachable Langfuse (bundled, or your own viaLANGFUSE_*). Disable Langfuse without connecting a replacement → it fails loudly, by design._run_variant_local) and cross-model compat are kept — they generate their own eval runs and never needed the trace store.Code
agent/traces.py) + its writer call; removeoptimize.traces_file(),TRACES_FILE,LOCAL_TRACE_*.optimize/mine.py: drop the local fallback (6 helpers);fetch_traces()raises with guidance when the backend is unreachable. Keep the dual-shape_task_answerparsing (BYOA connectors rely on it).test_local_traces.py; add a fail-loudfetch_tracestest.Compose + security
profiles: ["langfuse"]from the 6 Langfuse services (default-on);depends_on: langfuse-web (healthy)on the agent + mining services; fix stale "falls back to local" comments.127.0.0.1with datastores publishing no host port, and every demo secret (SALT,ENCRYPTION_KEY,NEXTAUTH_SECRET, DB/ClickHouse/MinIO/Redis creds, project keys, web login) is templatized${VAR:-demo}so it's overridable via.envwith no file edit.Docs
.env.example"LANGFUSE SECURITY" block (rotate-before-exposing, withopenssl randrecipes).security.md→ "Securing the Langfuse deployment".mcp-integration.md→ "Using your own evals platform": bundled / your-own-Langfuse / a different platform (Arize etc.). The write side is OTel today; the read side is thefetch_traces()seam — first-class adapters for other platforms are planned.Verification (Docker)
docker compose configvalid with Langfuse in the default profile.Consequence
Every
docker compose upnow runs Postgres + ClickHouse + MinIO + Redis + Langfuse web/worker — heavier, and a memory consideration on constrained hardware. That's the tradeoff of dropping lite mode; pointLANGFUSE_*at an external/Cloud Langfuse to avoid the bundled containers.🤖 Generated with Claude Code
https://claude.ai/code/session_01URfKaRsuJqkcVQPkHzxKfP
Summary by CodeRabbit
LANGFUSE_*.--doctor/--repair.SKILLOPT_MODELand new mining tuning variables.