fix: tier 1 correctness and cleanup fixes from architecture audit

[nojibe]

Summary

Lands the "Tier 1" batch from the July architecture review: six correctness fixes across blueprint ingestion, the LLM-judge config surface, PR staging, and the analysis UI, plus a dead-code sweep (net −4,432 lines) and relocation of shared utils out of the sandbox route.

Motivation / context

A code audit following the architecture discussion with the founding engineer surfaced silent divergences between the platform's five blueprint-ingestion paths, dead configuration knobs that mislead blueprint authors, a broken UI alert, and ~4.8k lines of verified-dead code. Findings are catalogued in #56 and #57.

Changes

• Webhooks / ingestion:
  • Push-to-main webhook now uses the same canonical parser, path-derived IDs, and CORE-collection resolution as the PR webhook and scheduled evals — multi-document blueprints no longer silently skip production evaluation after merge, and dedupe hashes finally match across paths (src/app/api/webhooks/github-push/route.ts)
  • New shared parseSubmittedBlueprint() in src/lib/blueprint-ingestion.ts so ingestion behavior is defined once
  • Canonical JSON-Schema validation now fails fast at both webhooks, with Ajv errors surfaced in the PR validation comment instead of burning a staging run; per-file parse errors no longer 500 the whole webhook
• Eval pipeline:
  • useExperimentalScale is honored (default behavior unchanged: 9-point scale); the FORCE_EXPERIMENTAL override is gone; the ignored judgeMode parameter is removed and the config field documented as deprecated (src/cli/evaluators/llm-coverage-evaluator.ts, src/types/shared.ts)
  • PR staging samples prompts evenly (deterministically) across the blueprint instead of truncating to the first 10, so large multilingual/blocked blueprints get whole-file coverage pre-merge (src/lib/pr-eval-limiter.ts)
• Analysis UI:
  • excludedModels is computed server-side at save time into core.json, restoring the "Models Automatically Excluded" alert for artifact-format runs (src/lib/storageService.ts); applies to newly saved runs
  • Fixed 404 detail links on the redlines and ndeltas experiment index pages
• Cleanup:
  • Deleted /v2 (duplicate homepage prototype), NavBarV2, SearchAutocomplete, pilot/india-multilingual/_archive_v1/, and four dangling package.json scripts
  • Moved yaml-generator, json-response-parser (+ tests) to src/lib/ and useMobile to src/hooks/ — the sandbox route no longer owns code used by live surfaces
• Tests: 12 new tests for parseSubmittedBlueprint, exclusion-computation cases added to the artefact round-trip suite, sampling tests for sampleEvenlySpaced, and the test asserting first-N truncation updated to the new intended behavior

Test plan

• pnpm typecheck ✅ 0 errors · pnpm test:web ✅ 564 passed · pnpm test:cli ✅ 570 passed (includes all new tests)
• pnpm lint ⚠️ not runnable — the repo has no ESLint config and next lint prompts for interactive migration (pre-existing on main; worth a follow-up ticket)
• Canonical schema checked against every example blueprint before making validation fatal: all parseable examples pass (the four failures are env-var placeholder URLs that already hard-fail at parse time on every path)
• Manual: booted the app with seeded local fixtures; verified the fixed experiment links click through to 200s, old top-level paths and /v2 return 404, homepage renders cleanly after the deletions

Risks / rollback

Plain revert undoes everything — no migrations. Reviewer attention:

• One-time re-evaluation: a blueprint touched by a push after deploy will re-run once, because dedupe hashes now use resolved model lists (matching the cron path); dedupe then converges across paths.
• Fail-fast validation: a canonical-schema false positive would now block a submission instead of warn. Mitigated by the example-corpus check above; errors are surfaced verbatim in the PR comment.
• Staging prompt selection changes: PR staging runs exercise different (evenly-spaced) prompts than before; staging-run hashes for identical content remain deterministic.
• excludedModels only appears in newly saved runs; old core.json artifacts predate the field (alert stays inert for them unless backfilled).

Screenshots not committed: the changed routes are data-driven (/experiments/*), so captures were taken against a secret-free local dev server with mock fixtures and shared with the maintainer directly rather than embedding permanent raw URLs in a public branch.

Related Issues

Closes #56
Closes #57

🤖 Generated with Claude Code

https://claude.ai/code/session_017bEpzahprA9rLt1wxxTKgy

---

Generated by Claude Code

[claude]

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.

_{Tip: disable this comment in your organization's Code Review settings.}

[railway-app]

🚅 Deployed to the app-pr-58 environment in weval

Service	Status	Web	Updated (UTC)
weval-app	✅ Success (View Logs)	Web	Jul 16, 2026 at 5:32 am

[nojibe]

📸 Site tour: every surface this audit touched, as it looks in production today

Captured 2026-07-16 from the live site. Three groups: what this PR removes, the links it fixes, and the orphaned pages it deliberately leaves alone pending a product decision (issues #56 / #57 have the code-level detail).

---

🗑 Removed by this PR (live until merge)

/v2 — duplicate homepage prototype

A full redesigned homepage: "The Open Platform for AI Evaluation" with three capability leaderboards (Safety & Integrity, Global Fluency, Helpfulness & Reasoning), its own nav and search. Nothing links to it and it duplicates the real homepage's data. Note before merge: the layout is genuinely polished — if anyone wants to salvage design ideas, the shared leaderboard components it uses (components/home/*) survive the deletion; only the route + NavBarV2 + orphaned SearchAutocomplete go.

---

🔧 Broken links this PR fixes

The redlines/ndeltas index pages linked to top-level /redlines/{configId} and /ndeltas/{modelId}, which never existed:

/experiments/redlines — span-level critique feed

LLM-annotated "redline" highlights on model responses, showing exactly which rubric points a response failed (example in frame: suicide-response scenarios flagging a missing crisis pathway / 988 referral). The config links in each card now resolve.

/experiments/ndeltas — model weak-points index

Per-model table of the prompts where each model most underperforms the peer average (worst Δ, median Δ). The "View details" links now resolve. Data last generated 2025-08-12.

---

🧭 Orphaned surfaces — live, unlinked, decision pending (NOT touched by this PR)

/vibes — model similarity map

Force-directed graph of models (node size = coverage, edges = strongest similarities) with a target-model selector and sim/capability/coverage blend sliders. Renders fully with production data.

/compass — AI Personality Compass

Eight behavioral axes (Abstract Thinking, Proactivity, Epistemic Humility, Risk Appetite, Free-Thinking, Conscientiousness, Agreeableness, Extroversion) with the leading/trailing models per trait.

/regressions — version-regression dashboard

Tracks score changes across model versions (in frame: Claude 3.5 Sonnet → 3.7 Sonnet, 45 regressions / 39 improvements). ⚠️ Data was last generated 2025-10-10 — the generate-regressions CLI hasn't run in ~9 months, so keeping this page implies scheduling that job.

/what-is-an-eval — narrative explainer

A polished "What is an Eval?" walkthrough (the Dr. Sharma / rural-Montana framing) that links onward to /story. Orphaned by omission — recommendation is to link it from the nav, not delete it.

/pairs — pairwise human-preference arena

The "Help Us Evaluate AI" voting surface. In production it errors with "No comparison tasks are available in the index" — the collection machinery (task queue, voting UI, preference storage) works, but the queue was never populated and no aggregation layer consumes the votes. This is the revive-or-remove decision from the audit.

/story — conversational eval builder

"How has AI affected you?" → a guided chat that turns a person's experience into a draft blueprint. This is the genuinely simple creation flow the audit recommends promoting as the site's "Create" front door (currently linked from nowhere).

/workshop — collaborative sibling of /story

Same conversational engine, wrapped for facilitated group sessions: generate a shareable workshop ID (crimson-elephant-742) or join an ongoing one; includes publish/gallery flows. Reachable only by shared ID.

/model/[...modelId] — superseded model page

404s even for valid model ids — no URL builder anywhere in the app produces links to it; /cards/{modelId} is the living equivalent. Recommendation: redirect to /cards and delete.

/eval-card-api/evaluations — orphaned JSON API

Returns the full blueprint-metadata catalogue as raw JSON (with a bare "Pretty-print" checkbox). Zero references in the repo, and it emits self-URLs (/api/evaluations/…) that don't match its own mount path — evidence it was moved and left stale. Check production access logs for external consumers before removing.

---

Screenshots are viewport captures (1280×800) of public production pages, committed under .github/pr-media/site-tour/ on this branch.

🤖 Generated with Claude Code

---

Generated by Claude Code