feat(evals): add capped live evaluation runner

CHANGELOG.md

@@ -30,6 +30,16 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - **Version-aware cache identity.** Cache keys cover the resolved roster, generation and mode
   settings, extraction behavior, sanitized endpoint routing, optional source-bundle digest, and
   cache/protocol/prompt/schema versions. Old incompatible envelopes are safe misses.
+- **Capped live evaluation runner.** The H1 lane is paid exploratory only and dry-run is the
+  default. Paid execution requires `--execute` plus exact USD 10.00 approval, persists each
+  reservation before allowing one in-flight provider call, and never repeats an interrupted
+  cell on resume. Its outputs are not decision eligible: the smoke verifies correctness only,
+  not efficiency or decision quality; the offline/open-book 24-task fixture remains separate.
+- **Authenticated live checkpoints and bounded UTF-8 estimates.** Paid execution additionally
+  requires an owner-only `--checkpoint-seal-key-file`; HMAC-SHA256 v2 checkpoints reject old,
+  forged, or wrong-key state without storing the key. Frozen price entries now bind
+  `max_output_bytes_per_token`, preventing one-byte dry-run placeholders from understating
+  multibyte upstream output expansion.
 
 ### Fixed
 

DOCUMENTATION_INDEX.md

@@ -7,23 +7,23 @@ the canonical authority spec on top of those.
 
 - **Repo:** `/Users/ernestprovo/dev/conclave/`
 - **Version:** 1.1.0 · **License:** MIT
-- **Last updated:** 2026-07-17
+- **Last updated:** 2026-07-18
 
 ---
 
 ## Core documentation
 
 | # | Doc | Path | Purpose |
 |---|-----|------|---------|
-| 1 | **README** (project overview) | [`README.md`](README.md) | Install, BYO-keys, five released-mode CLI/library quickstart, and the source-only unreleased Elite protocol. |
-| 2 | **System Context Diagram** | [`SYSTEM_CONTEXT_DIAGRAM.md`](SYSTEM_CONTEXT_DIAGRAM.md) | Mermaid context for the provider highway, five released modes, source-only Elite gated phases, verdict pipeline, phased manifest receipts, and mcp-warden boundary. |
+| 1 | **README** (project overview) | [`README.md`](README.md) | Install, BYO-keys, five released-mode CLI/library quickstart, source-only Elite, and the gated paid exploratory eval lane. |
+| 2 | **System Context Diagram** | [`SYSTEM_CONTEXT_DIAGRAM.md`](SYSTEM_CONTEXT_DIAGRAM.md) | Mermaid context for the provider highway, released/source modes, verdict pipeline, eval live guard, manifest receipts, and mcp-warden boundary. |
 | 3 | **Documentation Index** | [`DOCUMENTATION_INDEX.md`](DOCUMENTATION_INDEX.md) | This file. Master map of all docs + source layout. |
 
 ## Authority spec
 
 | Doc | Path | Purpose |
 |-----|------|---------|
-| **Product Design Document** | [`docs/PRODUCT_DESIGN_DOCUMENT.md`](docs/PRODUCT_DESIGN_DOCUMENT.md) | **Canonical** product spec: five released modes plus implemented-but-unreleased Elite, its fixed three-success phase gate and cost/latency tradeoff, the v1.1 execution-traceable verdict, manifest, architecture, boundaries, and roadmap. **When docs disagree, the PDD wins.** |
+| **Product Design Document** | [`docs/PRODUCT_DESIGN_DOCUMENT.md`](docs/PRODUCT_DESIGN_DOCUMENT.md) | **Canonical** product spec: released/source modes, the v1.1 execution-traceable verdict, manifest, paid exploratory evaluation boundary, architecture, and roadmap. **When docs disagree, the PDD wins.** |
 
 ## Product plans
 
@@ -34,6 +34,8 @@ the canonical authority spec on top of those.
 | **H1 Evaluation Plan** | [`docs/plans/2026-07-17-h1-budget-matched-evaluation.md`](docs/plans/2026-07-17-h1-budget-matched-evaluation.md) | TDD delivery plan for the offline, budget-matched evaluation substrate. |
 | **H1 Method Hardening** | [`docs/plans/2026-07-17-h1-method-hardening-design.md`](docs/plans/2026-07-17-h1-method-hardening-design.md) | Paid-study provenance, task-clustered inference, grader controls, confirmatory refusal boundaries, and QA design. |
 | **H1 Synthetic QA Pack** | [`studies/elite_qa_v1/README.md`](studies/elite_qa_v1/README.md) | Balanced 24-task open-book harness fixture, QA protocol, and confirmatory preregistration template. |
+| **H1 Live Runner Design** | [`docs/plans/2026-07-18-h1-live-evaluation-runner-design.md`](docs/plans/2026-07-18-h1-live-evaluation-runner-design.md) | Sequential paid-exploratory execution, hash-bound UTF-8 estimates, USD 10 cap, authenticated checkpoints, and no-repeat resume. |
+| **H1 Live Runner Plan** | [`docs/plans/2026-07-18-h1-live-evaluation-runner.md`](docs/plans/2026-07-18-h1-live-evaluation-runner.md) | Exact TDD tasks for the six live conditions, dry-run estimator, replay fixtures, CLI gate, and correctness-only paid smoke. |
 
 ---
 
@@ -65,7 +67,7 @@ Package root: `src/conclave/` (installed as the `conclave` package; console scri
 | Models | [`src/conclave/models.py`](src/conclave/models.py) | Stable Pydantic contract, including `EliteResult` phase artifacts and backward-compatible `CouncilResult.elite`. |
 | CLI | [`src/conclave/cli.py`](src/conclave/cli.py) | Five released `conclave ask` modes plus source-only Elite and `providers`; Elite exits 1 unless decision readiness is `ready`, emits full JSON before failure, and rejects streaming. |
 | Experimental evals | [`src/conclave/evals/`](src/conclave/evals/) | Versioned DSE-708 planning, strict replay, failure-inclusive running, blinding, scoring, and exploratory reports; no product-quality claim. |
-| Eval CLI | [`src/conclave/eval_cli.py`](src/conclave/eval_cli.py) | Offline-only `conclave eval plan/run/blind/report`; `run` validates replay artifacts and cannot call providers. |
+| Eval CLI | [`src/conclave/eval_cli.py`](src/conclave/eval_cli.py) | Offline `plan/run/blind/report` plus `eval live`; paid execution requires `--execute`, exact USD 10.00 approval, and an owner-only `--checkpoint-seal-key-file`. |
 | Logging | [`src/conclave/logging.py`](src/conclave/logging.py) | Logger factory; stderr; verbosity via `CONCLAVE_LOG_LEVEL` (default `WARNING`). |
 
 ## Tests
@@ -80,7 +82,7 @@ Package root: `src/conclave/` (installed as the `conclave` package; console scri
 | Provider highway tests | [`tests/test_providers.py`](tests/test_providers.py) | `resolve_adapter` (built-in prefixes, per-provider URLs, custom endpoints, unknown-prefix raise), end-to-end `call_model`, and `redact()` (bearer/`sk-`/env-var-value/`x-api-key` scrubbing; pre-redacted provider errors). |
 | Registry/config tests | [`tests/test_registry_config.py`](tests/test_registry_config.py) | Name resolution, key-presence logic, config merge. |
 | CLI tests | [`tests/test_cli.py`](tests/test_cli.py) | Typer `CliRunner`: exit-code contract (0 usable result and ready Elite / 1 zero-usable or non-ready Elite / 2 usage error), `--json` payload + exit code, human renderers per mode, `providers` table never prints secrets, aclose lifecycle. |
-| Eval tests | [`tests/evals/`](tests/evals/) | Frozen matrix, replay fail-closed behavior, runner denominators, blinding, scoring, reporting, and offline CLI. |
+| Eval tests | [`tests/evals/`](tests/evals/) | Frozen matrix, offline replay, live reservation/checkpoint/resume gates, blinding, scoring, reporting, and CLI assertions. |
 | Transport tests | [`tests/test_transport.py`](tests/test_transport.py) | `post_json` via httpx `MockTransport`: success/error-status/non-JSON fallback, timeout & connect/HTTP errors → `TransportError` (key never leaks), client reuse/pooling, aclose idempotency. |
 | Streaming tests | [`tests/test_streaming.py`](tests/test_streaming.py) | Per-adapter SSE via `MockTransport` (openai-compat/anthropic/gemini): incremental chunks + assembled answer == concatenation == buffered result; mid-stream malformed-frame/connection-drop/non-2xx → error set with partial text preserved (never raises); key redaction in stream errors; buffered `ask()` never opens a stream; `Council.ask_stream` interleaving + terminal `done` shape; CLI `--stream` smoke + exit-code contract + debate rejection; `--stream` + cache one-shot replay. |
 | Logging tests | [`tests/test_logging.py`](tests/test_logging.py) | `CONCLAVE_LOG_LEVEL` resolution (default `WARNING`, case-insensitive, unknown → `WARNING`), factory contract, one-shot configuration. |
@@ -114,6 +116,7 @@ Run: `pytest` (config in `pyproject.toml`, `asyncio_mode = "auto"`).
 
 | Date | Change |
 |------|--------|
+| 2026-07-18 | Implemented the sequential paid exploratory runner: default dry-run, exact USD 10.00 execute approval, reservation-before-call, one in-flight call, and no-repeat resume. The 24-task fixture remains offline/open-book and is not the paid smoke corpus; the smoke is correctness-only and not decision eligible. |
 | 2026-07-17 | Added the experimental DSE-708 offline evaluation substrate and `conclave eval` artifact workflow; no live study or quality claim. |
 | 2026-07-17 | Reframed the product roadmap around empirically proven decision quality: narrowed current claims to execution traceability, identified answer IDs as internal provenance rather than external evidence, gated Elite merge on H0 correctness, and added H1-H4 evidence, buyer, and outcome gates. |
 | 2026-07-17 | Documented the implemented-but-unreleased Elite Decision Protocol: fixed three-success gates, initial/claim-audit/revision artifacts, existing final verdict, phased receipts, failure semantics, cost/latency tradeoff, and no streaming. |

README.md

@@ -150,21 +150,22 @@ a future release explicitly includes it.
 of a council defined in your config (see below). The built-in `default` council
 is all known providers.
 
-### Experimental offline evaluation (DSE-708)
+### Experimental evaluation (DSE-708)
 
-The source branch also exposes an H1 evidence harness. It makes no decision-quality claim and
-never calls providers: `run` only validates an existing replay artifact against its frozen
-manifest. Grader output and the restricted blind map must remain separate.
-The open-book 24-task synthetic QA pack and its no-promotion boundary are documented in
-[`studies/elite_qa_v1/README.md`](studies/elite_qa_v1/README.md). Its committed answer keys
-are test fixtures, not access-controlled grader material, so this pack cannot be used for a
-paid or confirmatory study.
+Offline `conclave eval run` validates a frozen replay and never calls a provider. The live lane
+is **paid exploratory only**. Dry-run is the default; execution also requires `--execute`, exact
+`--approve-spend-usd 10.00`, and an owner-only `--checkpoint-seal-key-file` containing at least
+32 random bytes. Checkpoints use a versioned HMAC-SHA256 seal; the key is never serialized.
+Each frozen price entry attests `max_output_bytes_per_token`, so estimates bound inserted UTF-8 bytes. One provider call is in flight, a reservation is persisted before each call, and resume never repeats an
+interrupted cell. Smoke checks correctness only, not efficiency or decision quality; outputs
+remain paid exploratory, not decision eligible, and separate from the offline 24-task fixture.
 
 ```bash
-conclave eval plan tasks.json manifest.json --study-id qa --replicates 2 --seed 19 --max-output-tokens 1200
-conclave eval run manifest.json validated-run.json --replay-artifact recorded-run.json
-conclave eval blind validated-run.json grader.json restricted-map.json --seed 23
-conclave eval report manifest.json validated-run.json judgments.json report.json report.md --bootstrap-seed 29
+conclave eval plan path/tasks.json path/manifest.json --study-id qa --replicates 2 --seed 19 --max-output-tokens 1200
+conclave eval run path/manifest.json path/run.json --replay-artifact path/replay.json
+conclave eval live path/manifest.json path/tasks.json path/prices.json path/run.json path/checkpoint.json path/receipts.json
+conclave eval live path/manifest.json path/tasks.json path/prices.json path/run.json path/checkpoint.json path/receipts.json \
+  --execute --approve-spend-usd 10.00 --checkpoint-seal-key-file path/checkpoint-seal.key
 ```
 
 Add `--stream` to render member (and synthesizer) tokens live as they arrive

SYSTEM_CONTEXT_DIAGRAM.md

@@ -8,6 +8,7 @@ LLM-SDK dependency), how the v1.1 **verdict pipeline** turns the member answers
 structured, agreement-scored, **execution-traceable** verdict plus a redacted execution manifest, and
 how the implemented-but-unreleased **Elite Decision Protocol** adds gated claim audit and
 revision before that verdict. It also shows where **mcp-warden** sits as a dev-time consumer.
+The DSE-708 edge includes offline replay plus an opt-in, capped paid exploratory path.
 
 > Authority note: behavioral details here are descriptive. The canonical spec is
 > [`docs/PRODUCT_DESIGN_DOCUMENT.md`](docs/PRODUCT_DESIGN_DOCUMENT.md).
@@ -32,8 +33,9 @@ flowchart TB
         council["Council orchestrator<br/>fan_out · synthesize_blocks · skip-no-key (council.py)"]
         modes["Deliberation modes<br/>debate · adversarial · vote (modes.py + prompts.py)"]
         elite["Elite (UNRELEASED)<br/>initial → claim audit → revision<br/>fixed 3-success gate each phase"]
-        evalcli["Experimental eval CLI (DSE-708)<br/>plan · replay validation · blind · report<br/>OFFLINE ONLY"]
+        evalcli["Experimental eval CLI (DSE-708)<br/>offline replay · live dry-run<br/>gated paid exploratory execute"]
         evalartifacts["Versioned eval artifacts<br/>manifest · run · grader set + separate map<br/>exploratory report"]
+        liveguard["Live guard<br/>exact USD 10.00 approval · owner-only seal key<br/>HMAC checkpoint · one in flight · no-repeat resume"]
         registry["Registry · name to model-id<br/>key PRESENCE only, never values (registry.py)"]
         config["Config loader · custom endpoints (config.py)"]
         models["Result contract · CouncilResult v2<br/>answers · verdict · consensus · manifest (models.py)"]
@@ -67,8 +69,9 @@ flowchart TB
     end
 
     user -->|"prompt + council + mode"| cli
-    user -->|"public tasks + offline artifacts"| evalcli
+    user -->|"public tasks + frozen artifacts"| evalcli
     evalcli --> evalartifacts
+    evalcli -.->|"--execute + exact approval"| liveguard
     user -->|"import"| lib
     warden -.->|"imports at DEV time only · NOT a runtime dep"| lib
 
@@ -85,6 +88,7 @@ flowchart TB
     council --> elite
     elite -->|"3+ revisions: existing synthesis + verdict<br/>under 3: stop incomplete"| council
     council --> provider
+    liveguard --> provider
     provider --> adreg
     adreg --> oai
     adreg --> anth
@@ -127,10 +131,13 @@ flowchart TB
 - **Two entry points, one core.** The CLI (`cli.py`) and the library API
   (`from conclave import Council`) are both thin drivers over the same `Council`
   orchestrator. There is no behavior in the CLI that the library can't reach.
-- **The DSE-708 eval lane is experimental and offline.** `conclave eval` freezes manifests,
-  validates pre-recorded run artifacts, blinds outputs into a grader set plus separate identity
-  map, and writes exploratory reports. It has no edge to the provider highway: live execution is
-  fail-closed, and these artifacts support measurement rather than a decision-quality claim.
+- **The DSE-708 eval lane is paid exploratory only.** Offline `eval run` still validates
+  pre-recorded artifacts without provider calls. `eval live` defaults to dry-run; execution
+  requires `--execute`, exact USD 10.00 approval, and an owner-only checkpoint-seal key file.
+  HMAC-SHA256 authenticates checkpoint state, and the price hash includes each model's
+  `max_output_bytes_per_token` estimate bound. One call is in flight, its reservation persists
+  first, and resume skips an interrupted cell. The smoke checks correctness only—not efficiency
+  or decision quality—and remains not decision eligible.
 - **mcp-warden is dashed and dev-time.** The dotted edge from `mcp-warden` to the library
   is deliberate: warden imports conclave **only at design/eval time**. conclave is
   stochastic and must never sit in warden's deterministic runtime decision path. See PDD

docs/PRODUCT_DESIGN_DOCUMENT.md

@@ -410,7 +410,13 @@ council shipped in v1.1** (§4a — the wedge).
 The revised thesis is a **source-grounded, execution-traceable decision record with
 empirically proven quality**. Current answer IDs identify model outputs, not external evidence;
 source-auditable language is therefore too broad until source grounding ships. Elite remains
-implemented but unreleased on open PR #51, pending final review and merge.
+implemented but unreleased in source.
+
+H1 also includes an opt-in live runner that is **paid exploratory only**. Dry-run is the default;
+paid execution requires `--execute`, exact `--approve-spend-usd 10.00`, and an owner-only
+`--checkpoint-seal-key-file` with at least 32 random bytes. Versioned HMAC-SHA256 checkpoints
+never serialize that key; frozen `max_output_bytes_per_token` attestations bound inserted UTF-8
+bytes. One call is in flight, reservations persist first, and resume never repeats interrupted cells. The smoke proves correctness only—not efficiency or decision quality—and remains not decision eligible.
 
 The canonical roadmap is
 [`docs/plans/2026-07-17-decision-quality-roadmap.md`](plans/2026-07-17-decision-quality-roadmap.md):