feat(evals): add budget-matched Elite evaluation substrate

DOCUMENTATION_INDEX.md

@@ -30,6 +30,8 @@ the canonical authority spec on top of those.
 | Doc | Path | Purpose |
 |-----|------|---------|
 | **Decision Quality Roadmap** | [`docs/plans/2026-07-17-decision-quality-roadmap.md`](docs/plans/2026-07-17-decision-quality-roadmap.md) | H0-H4 prove-it roadmap: Elite correctness, randomized ablations, source-grounded Decision Briefs, buyer pull, outcome learning, old-backlog disposition, demand gates, and kill criteria. |
+| **H1 Evaluation Design** | [`docs/plans/2026-07-17-h1-budget-matched-evaluation-design.md`](docs/plans/2026-07-17-h1-budget-matched-evaluation-design.md) | DSE-708 experimental boundary, six frozen conditions, replay, blinding, and failure-inclusive analysis. |
+| **H1 Evaluation Plan** | [`docs/plans/2026-07-17-h1-budget-matched-evaluation.md`](docs/plans/2026-07-17-h1-budget-matched-evaluation.md) | TDD delivery plan for the offline, budget-matched evaluation substrate. |
 
 ---
 
@@ -60,6 +62,8 @@ Package root: `src/conclave/` (installed as the `conclave` package; console scri
 | Config | [`src/conclave/config.py`](src/conclave/config.py) | Loads/merges `~/.conclave/config.yml` over defaults; resolves model ids and named/CSV councils; parses the `endpoints:` section (custom OpenAI-compatible providers). |
 | Models | [`src/conclave/models.py`](src/conclave/models.py) | Stable Pydantic contract, including `EliteResult` phase artifacts and backward-compatible `CouncilResult.elite`. |
 | CLI | [`src/conclave/cli.py`](src/conclave/cli.py) | Five released `conclave ask` modes plus source-only Elite and `providers`; Elite exits 1 unless decision readiness is `ready`, emits full JSON before failure, and rejects streaming. |
+| Experimental evals | [`src/conclave/evals/`](src/conclave/evals/) | Versioned DSE-708 planning, strict replay, failure-inclusive running, blinding, scoring, and exploratory reports; no product-quality claim. |
+| Eval CLI | [`src/conclave/eval_cli.py`](src/conclave/eval_cli.py) | Offline-only `conclave eval plan/run/blind/report`; `run` validates replay artifacts and cannot call providers. |
 | Logging | [`src/conclave/logging.py`](src/conclave/logging.py) | Logger factory; stderr; verbosity via `CONCLAVE_LOG_LEVEL` (default `WARNING`). |
 
 ## Tests
@@ -74,6 +78,7 @@ Package root: `src/conclave/` (installed as the `conclave` package; console scri
 | Provider highway tests | [`tests/test_providers.py`](tests/test_providers.py) | `resolve_adapter` (built-in prefixes, per-provider URLs, custom endpoints, unknown-prefix raise), end-to-end `call_model`, and `redact()` (bearer/`sk-`/env-var-value/`x-api-key` scrubbing; pre-redacted provider errors). |
 | Registry/config tests | [`tests/test_registry_config.py`](tests/test_registry_config.py) | Name resolution, key-presence logic, config merge. |
 | CLI tests | [`tests/test_cli.py`](tests/test_cli.py) | Typer `CliRunner`: exit-code contract (0 usable result and ready Elite / 1 zero-usable or non-ready Elite / 2 usage error), `--json` payload + exit code, human renderers per mode, `providers` table never prints secrets, aclose lifecycle. |
+| Eval tests | [`tests/evals/`](tests/evals/) | Frozen matrix, replay fail-closed behavior, runner denominators, blinding, scoring, reporting, and offline CLI. |
 | Transport tests | [`tests/test_transport.py`](tests/test_transport.py) | `post_json` via httpx `MockTransport`: success/error-status/non-JSON fallback, timeout & connect/HTTP errors → `TransportError` (key never leaks), client reuse/pooling, aclose idempotency. |
 | Streaming tests | [`tests/test_streaming.py`](tests/test_streaming.py) | Per-adapter SSE via `MockTransport` (openai-compat/anthropic/gemini): incremental chunks + assembled answer == concatenation == buffered result; mid-stream malformed-frame/connection-drop/non-2xx → error set with partial text preserved (never raises); key redaction in stream errors; buffered `ask()` never opens a stream; `Council.ask_stream` interleaving + terminal `done` shape; CLI `--stream` smoke + exit-code contract + debate rejection; `--stream` + cache one-shot replay. |
 | Logging tests | [`tests/test_logging.py`](tests/test_logging.py) | `CONCLAVE_LOG_LEVEL` resolution (default `WARNING`, case-insensitive, unknown → `WARNING`), factory contract, one-shot configuration. |
@@ -107,6 +112,7 @@ Run: `pytest` (config in `pyproject.toml`, `asyncio_mode = "auto"`).
 
 | Date | Change |
 |------|--------|
+| 2026-07-17 | Added the experimental DSE-708 offline evaluation substrate and `conclave eval` artifact workflow; no live study or quality claim. |
 | 2026-07-17 | Reframed the product roadmap around empirically proven decision quality: narrowed current claims to execution traceability, identified answer IDs as internal provenance rather than external evidence, gated Elite merge on H0 correctness, and added H1-H4 evidence, buyer, and outcome gates. |
 | 2026-07-17 | Documented the implemented-but-unreleased Elite Decision Protocol: fixed three-success gates, initial/claim-audit/revision artifacts, existing final verdict, phased receipts, failure semantics, cost/latency tradeoff, and no streaming. |
 | 2026-07-13 | **Manifest-on-every-result invariant fix + doc reconciliation.** `ModelHarnessManifest` now attaches on all five modes (was `None` for `debate`/`adversarial`/`vote`) via a single-site fix at `Council._cached_run` → new `_ensure_manifest`; regression tests in `tests/test_manifest_all_modes.py`. Docs corrected so the manifest-on-every-result claim is accurate and the `vote` mode is documented as **shipped** (CAC-09 / #3), not "absorbed" — across README, `SYSTEM_CONTEXT_DIAGRAM.md`, PDD (§1/§3/§4a/§8/§9/§12), and `CHANGELOG.md` (`[Unreleased]`). Verdict scope unchanged (still `synthesize`/`ask`-only; not layered on `adversarial`). |

README.md

@@ -150,6 +150,19 @@ a future release explicitly includes it.
 of a council defined in your config (see below). The built-in `default` council
 is all known providers.
 
+### Experimental offline evaluation (DSE-708)
+
+The source branch also exposes an H1 evidence harness. It makes no decision-quality claim and
+never calls providers: `run` only validates an existing replay artifact against its frozen
+manifest. Grader output and the restricted blind map must remain separate.
+
+```bash
+conclave eval plan tasks.json manifest.json --study-id pilot --replicates 2 --seed 19 --max-output-tokens 1200
+conclave eval run manifest.json validated-run.json --replay-artifact recorded-run.json
+conclave eval blind validated-run.json grader.json restricted-map.json --seed 23
+conclave eval report manifest.json validated-run.json judgments.json report.json report.md --bootstrap-seed 29
+```
+
 Add `--stream` to render member (and synthesizer) tokens live as they arrive
 (`synthesize`/`raw` modes only):
 

SYSTEM_CONTEXT_DIAGRAM.md

@@ -32,6 +32,8 @@ flowchart TB
         council["Council orchestrator<br/>fan_out · synthesize_blocks · skip-no-key (council.py)"]
         modes["Deliberation modes<br/>debate · adversarial · vote (modes.py + prompts.py)"]
         elite["Elite (UNRELEASED)<br/>initial → claim audit → revision<br/>fixed 3-success gate each phase"]
+        evalcli["Experimental eval CLI (DSE-708)<br/>plan · replay validation · blind · report<br/>OFFLINE ONLY"]
+        evalartifacts["Versioned eval artifacts<br/>manifest · run · grader set + separate map<br/>exploratory report"]
         registry["Registry · name to model-id<br/>key PRESENCE only, never values (registry.py)"]
         config["Config loader · custom endpoints (config.py)"]
         models["Result contract · CouncilResult v2<br/>answers · verdict · consensus · manifest (models.py)"]
@@ -65,6 +67,8 @@ flowchart TB
     end
 
     user -->|"prompt + council + mode"| cli
+    user -->|"public tasks + offline artifacts"| evalcli
+    evalcli --> evalartifacts
     user -->|"import"| lib
     warden -.->|"imports at DEV time only · NOT a runtime dep"| lib
 
@@ -123,6 +127,10 @@ flowchart TB
 - **Two entry points, one core.** The CLI (`cli.py`) and the library API
   (`from conclave import Council`) are both thin drivers over the same `Council`
   orchestrator. There is no behavior in the CLI that the library can't reach.
+- **The DSE-708 eval lane is experimental and offline.** `conclave eval` freezes manifests,
+  validates pre-recorded run artifacts, blinds outputs into a grader set plus separate identity
+  map, and writes exploratory reports. It has no edge to the provider highway: live execution is
+  fail-closed, and these artifacts support measurement rather than a decision-quality claim.
 - **mcp-warden is dashed and dev-time.** The dotted edge from `mcp-warden` to the library
   is deliberate: warden imports conclave **only at design/eval time**. conclave is
   stochastic and must never sit in warden's deterministic runtime decision path. See PDD

docs/plans/2026-07-17-h1-budget-matched-evaluation-design.md

@@ -0,0 +1,49 @@
+# H1 Budget-Matched Evaluation Design
+
+**Linear:** DSE-708
+**Status:** Approved direction; implementation begins offline
+**Base:** Elite H0 commit `23e8fce437172c040084d31dcea4ed5737ebae2a`
+
+## Decision
+
+Build a narrow, experimental `conclave.evals` package that can plan, replay, blind, score, and report a fixed six-condition study. It is an evidence instrument for Elite, not a general evaluation platform and not a new public council mode.
+
+## Study contract
+
+The six frozen conditions are:
+
+1. `single_frontier`
+2. `self_refine`
+3. `independent_synthesis`
+4. `critique_only`
+5. `revision_only`
+6. `elite_full`
+
+Every task-condition-replicate cell is declared before execution. Missing, timed-out, malformed, abstained, and incomplete cells remain in the denominator. Conditions receive identical public task material and reference packets. Grader-only keys live in a separate file that the runner never loads.
+
+## Architecture
+
+- `models.py`: versioned Pydantic task, condition, run, score, and study-manifest contracts.
+- `dataset.py`: load and hash public tasks; separately load grader keys only for scoring.
+- `protocols.py`: immutable six-condition registry and token-budget allocation.
+- `replay.py`: record/replay at `conclave.transport.post_json`; match sanitized request identity plus occurrence index; reject missing, extra, incompatible, or secret-bearing artifacts.
+- `runner.py`: create a seeded task x condition x replicate matrix, enforce budgets, preserve failures, and emit atomic artifacts.
+- `blinding.py`: deterministic seeded opaque output IDs and a separate restricted blind map.
+- `scoring.py`: atomic judgments, adjudication without overwriting raw scores, Wilson intervals, paired bootstrap differences, and Cohen's kappa.
+- `eval_cli.py`: `plan`, `run`, `blind`, and `report` commands. The main CLI exposes these under `conclave eval` only after the offline substrate is stable.
+
+## Budget matching
+
+The study manifest declares one output-token ceiling per task-condition cell. Provider adapters receive `max_output_tokens` end to end. Planned ceilings must be within 5% across conditions; actual tokens, latency, failures, and unused budget are reported. Token ceilings are a reproducible provider-spend proxy; a later live-study manifest may add frozen price metadata without placing mutable pricing in library code.
+
+## Replay and security
+
+Replay performs zero network calls. Request identities include provider-safe URL components, model, normalized non-secret body, and occurrence index; authorization headers, API keys, raw endpoint credentials, and exception chains are never serialized. Replay fails closed on schema/version/hash mismatch or unmatched calls.
+
+## Analysis
+
+The primary endpoint is failure-inclusive critical-error-free decision rate. Reports retain task-level paired outcomes and show distributions and uncertainty: Wilson intervals for rates, seeded paired bootstrap intervals for condition differences, and kappa/adjudication rates for graders. Pilot results are exploratory; held-out results require a frozen preregistration.
+
+## Delivery boundaries
+
+This increment excludes provider spend, a hosted dashboard, retrieval, embeddings, routing, mutable pricing tables, LLM-as-primary-grader, and public product claims. H0 must be merged and pinned before a confirmatory live run. Any paid pilot requires an explicit spend ceiling.

docs/plans/2026-07-17-h1-budget-matched-evaluation.md

@@ -0,0 +1,55 @@
+# H1 Budget-Matched Evaluation Implementation Plan
+
+> **Linear:** DSE-708. Execute with test-driven development on stacked branch `feat/h1-budget-matched-eval`; make no live provider calls in this plan.
+
+**Goal:** Deliver an offline, reproducible substrate that can fairly compare Elite with five fixed budget-matched alternatives.
+
+**Architecture:** Add an experimental `conclave.evals` package around versioned artifacts. Keep provider I/O at the existing transport seam, thread output-token caps through adapters, and keep study execution separate from the public council API.
+
+## Phase 1 — Contracts and deterministic planning
+
+1. Add failing tests in `tests/evals/test_models.py` and `tests/evals/test_protocols.py` for schema versions, the exact six-condition registry, ±5% planned-budget enforcement, stable hashes, deterministic seeded order, and complete task x condition x replicate matrices.
+2. Run `pytest tests/evals/test_models.py tests/evals/test_protocols.py -q` and confirm RED.
+3. Implement `src/conclave/evals/__init__.py`, `models.py`, `dataset.py`, and `protocols.py` minimally.
+4. Re-run focused tests, Ruff, and commit.
+
+## Phase 2 — Output caps and strict replay
+
+1. Add failing adapter/provider tests proving `max_output_tokens` reaches OpenAI-compatible, Anthropic, and Gemini request bodies without changing default requests.
+2. Add failing `tests/evals/test_replay.py` cases for record/replay identity, repeated-call occurrence indexes, zero-network replay, secret exclusion, and fail-closed missing/extra/version-mismatched artifacts.
+3. Thread optional `max_output_tokens` through `ProviderAdapter`, concrete adapters, `call_model`, and streaming paths; implement `src/conclave/evals/replay.py` at `transport.post_json`.
+4. Run focused tests, existing adapter/provider/transport tests, Ruff, and commit.
+
+## Phase 3 — Runner and blinding
+
+1. Add failing `tests/evals/test_runner.py` and `test_blinding.py` for all six conditions, per-cell caps, immutable failures in denominators, receipts/totals, seeded opaque IDs, and a separate blind map.
+2. Implement `src/conclave/evals/runner.py`, `protocols.py`, and `blinding.py`. Protocol executors may use recorded fixtures only in this increment.
+3. Re-run focused tests and commit.
+
+## Phase 4 — Atomic scoring and reports
+
+1. Add failing `tests/evals/test_scoring.py` for raw grader preservation, adjudication, critical-error-free rates, Wilson intervals, seeded paired bootstrap intervals, and Cohen's kappa.
+2. Implement `src/conclave/evals/scoring.py` and machine-readable plus Markdown report writers.
+3. Add a small synthetic fixture set under `tests/fixtures/evals/`; label every result exploratory/synthetic.
+4. Re-run focused tests and commit.
+
+## Phase 5 — CLI and documentation
+
+1. Add failing `tests/evals/test_cli.py` for `conclave eval plan`, `run --replay`, `blind`, and `report`; reject live execution unless explicitly enabled and configured.
+2. Implement `src/conclave/eval_cli.py` and mount its Typer app in `src/conclave/cli.py`.
+3. Update `README.md`, `DOCUMENTATION_INDEX.md`, and `SYSTEM_CONTEXT_DIAGRAM.md` with the experimental boundary and reproducible commands.
+4. Run CLI tests and commit.
+
+## Final verification and handoff
+
+Run:
+
+```bash
+python -m pytest -q
+python -m ruff check .
+python -m ruff format --check .
+git diff --check
+gitleaks git --redact
+```
+
+Then push the stacked branch, open a PR linked to DSE-708 and PR #51, request independent review, and attach test/scan evidence to Linear. Do not merge, publish, or run paid models without the required approval.

src/conclave/adapters/anthropic.py

@@ -120,6 +120,7 @@ def build_request(
         timeout: float,
         api_key: str,
         output_contract: OutputContract | None = None,
+        max_output_tokens: int | None = None,
     ) -> tuple[str, dict[str, str], dict]:
         """Build the Messages POST, hoisting system out of the message array.
 
@@ -157,7 +158,7 @@ def build_request(
 
         body: dict = {
             "model": self._bare_model(model_id),
-            "max_tokens": self.max_tokens,
+            "max_tokens": max_output_tokens if max_output_tokens is not None else self.max_tokens,
             "messages": turns,
         }
         if temperature is not None:
@@ -261,6 +262,7 @@ def stream_request(
         timeout: float,
         api_key: str,
         output_contract: OutputContract | None = None,
+        max_output_tokens: int | None = None,
     ) -> tuple[str, dict[str, str], dict]:
         """Build the streaming POST: ``build_request`` + ``stream: true``.
 
@@ -271,7 +273,13 @@ def stream_request(
         fragments. See :meth:`ProviderAdapter.stream_request`.
         """
         url, headers, body = self.build_request(
-            model_id, messages, temperature, timeout, api_key, output_contract
+            model_id,
+            messages,
+            temperature,
+            timeout,
+            api_key,
+            output_contract=output_contract,
+            max_output_tokens=max_output_tokens,
         )
         body["stream"] = True
         return url, headers, body

src/conclave/adapters/base.py

@@ -274,6 +274,7 @@ def build_request(
         timeout: float,
         api_key: str,
         output_contract: OutputContract | None = None,
+        max_output_tokens: int | None = None,
     ) -> tuple[str, dict[str, str], dict]:
         """Build ``(url, headers, json_body)`` for this provider.
 
@@ -292,6 +293,8 @@ def build_request(
                 ``responseSchema`` / Anthropic tool ``input_schema``) is deferred
                 to the CAC-02-OAI/ANT/GEM tickets; today every adapter accepts and
                 ignores it.
+            max_output_tokens: Optional per-call output ceiling. ``None`` keeps
+                the adapter's existing provider default unchanged.
 
         Returns:
             A ``(url, headers, json_body)`` tuple ready for ``post_json``.
@@ -322,6 +325,7 @@ def stream_request(
         timeout: float,
         api_key: str,
         output_contract: OutputContract | None = None,
+        max_output_tokens: int | None = None,
     ) -> tuple[str, dict[str, str], dict]:
         """Build ``(url, headers, json_body)`` for a STREAMING request (issue #7).
 

src/conclave/adapters/gemini.py

@@ -290,6 +290,7 @@ def build_request(
         timeout: float,
         api_key: str,
         output_contract: OutputContract | None = None,
+        max_output_tokens: int | None = None,
     ) -> tuple[str, dict[str, str], dict]:
         """Build the generateContent POST.
 
@@ -320,7 +321,11 @@ def build_request(
             gemini_role = _ROLE_MAP.get(role, "user")
             contents.append({"role": gemini_role, "parts": [{"text": content}]})
 
-        generation_config: dict = {"maxOutputTokens": self.max_output_tokens}
+        generation_config: dict = {
+            "maxOutputTokens": (
+                max_output_tokens if max_output_tokens is not None else self.max_output_tokens
+            )
+        }
         if temperature is not None:
             generation_config["temperature"] = temperature
         # Conditional structured-output injection (no-op when contract is None,
@@ -367,6 +372,7 @@ def stream_request(
         timeout: float,
         api_key: str,
         output_contract: OutputContract | None = None,
+        max_output_tokens: int | None = None,
     ) -> tuple[str, dict[str, str], dict]:
         """Build the streaming POST against ``streamGenerateContent?alt=sse``.
 
@@ -380,7 +386,13 @@ def stream_request(
         # output_contract flows into build_request, which performs the
         # capability-gated responseMimeType/responseSchema injection.
         _url, headers, body = self.build_request(
-            model_id, messages, temperature, timeout, api_key, output_contract
+            model_id,
+            messages,
+            temperature,
+            timeout,
+            api_key,
+            output_contract=output_contract,
+            max_output_tokens=max_output_tokens,
         )
         model = self._bare_model(model_id)
         url = f"{GEMINI_BASE}/{model}:streamGenerateContent?alt=sse"