fix(evals): harden H1 methods and add synthetic QA pack

DOCUMENTATION_INDEX.md

@@ -32,6 +32,8 @@ the canonical authority spec on top of those.
 | **Decision Quality Roadmap** | [`docs/plans/2026-07-17-decision-quality-roadmap.md`](docs/plans/2026-07-17-decision-quality-roadmap.md) | H0-H4 prove-it roadmap: Elite correctness, randomized ablations, source-grounded Decision Briefs, buyer pull, outcome learning, old-backlog disposition, demand gates, and kill criteria. |
 | **H1 Evaluation Design** | [`docs/plans/2026-07-17-h1-budget-matched-evaluation-design.md`](docs/plans/2026-07-17-h1-budget-matched-evaluation-design.md) | DSE-708 experimental boundary, six frozen conditions, replay, blinding, and failure-inclusive analysis. |
 | **H1 Evaluation Plan** | [`docs/plans/2026-07-17-h1-budget-matched-evaluation.md`](docs/plans/2026-07-17-h1-budget-matched-evaluation.md) | TDD delivery plan for the offline, budget-matched evaluation substrate. |
+| **H1 Method Hardening** | [`docs/plans/2026-07-17-h1-method-hardening-design.md`](docs/plans/2026-07-17-h1-method-hardening-design.md) | Paid-study provenance, task-clustered inference, grader controls, confirmatory refusal boundaries, and QA design. |
+| **H1 Synthetic QA Pack** | [`studies/elite_qa_v1/README.md`](studies/elite_qa_v1/README.md) | Balanced 24-task open-book harness fixture, QA protocol, and confirmatory preregistration template. |
 
 ---
 

README.md

@@ -155,9 +155,13 @@ is all known providers.
 The source branch also exposes an H1 evidence harness. It makes no decision-quality claim and
 never calls providers: `run` only validates an existing replay artifact against its frozen
 manifest. Grader output and the restricted blind map must remain separate.
+The open-book 24-task synthetic QA pack and its no-promotion boundary are documented in
+[`studies/elite_qa_v1/README.md`](studies/elite_qa_v1/README.md). Its committed answer keys
+are test fixtures, not access-controlled grader material, so this pack cannot be used for a
+paid or confirmatory study.
 
 ```bash
-conclave eval plan tasks.json manifest.json --study-id pilot --replicates 2 --seed 19 --max-output-tokens 1200
+conclave eval plan tasks.json manifest.json --study-id qa --replicates 2 --seed 19 --max-output-tokens 1200
 conclave eval run manifest.json validated-run.json --replay-artifact recorded-run.json
 conclave eval blind validated-run.json grader.json restricted-map.json --seed 23
 conclave eval report manifest.json validated-run.json judgments.json report.json report.md --bootstrap-seed 29

docs/plans/2026-07-17-h1-method-hardening-design.md

@@ -0,0 +1,53 @@
+# H1 Method Hardening Design
+
+**Linear:** DSE-708
+**Dependency:** PR #52 at `4365c28`
+**Gate:** Required before any paid study call
+
+## Decision
+
+Keep the existing H1 substrate as the execution primitive, then add a frozen study-method
+contract and a balanced 24-task open-book synthetic QA pack. The committed pack validates
+mechanics only; it cannot estimate unbiased effects or produce a product GO or KILL. A paid
+pilot requires separately access-controlled grader keys and a hash frozen before execution.
+
+## Frozen study contract
+
+Every manifest must bind the exact base commit, study phase, task-family map, two roster specifications, provider/model revisions, condition prompt and protocol versions, generation settings, public-task and private-key hashes, rubric and grader-instruction hashes, evaluator and analysis-code versions, randomization and bootstrap settings, timeout/retry policy, exclusion/deviation policy, price snapshot, approved spend ceiling, and preregistration identifier/hash.
+
+Confirmatory reports fail closed if any frozen field is missing or its observed receipt differs. Exploratory manifests are explicit and cannot be promoted after results exist.
+
+## Experimental unit and randomization
+
+The unit of analysis is the task. Roster and replicate are repeated measurements averaged within task. Create every task x roster x condition x replicate cell before execution. Derive an independent condition permutation from the master seed plus task and roster IDs so condition order varies across blocks. All failed, timed-out, malformed, over-budget, abstained, incomplete, ungraded, and unresolved cells remain failures.
+
+## Grading contract
+
+Two independent graders score every successful output. Atomic records include rubric item, error category and severity, severe-error flag, holistic dimensions, reviewer seconds, confidence or abstention, rubric version/hash, grader batch/order, and condition/provider guess. Adjudication sees only genuine disagreements, cites every source record, and never overwrites raw judgments.
+
+Non-success records bypass the human queue and are automatic failures. Successful grader views expose only normalized presentation text, scan it for predeclared provider/model/condition labels, and retain a separate hashed/access-controlled blind map. Paid and confirmatory scoring rejects direct planned-run targets, unhashed maps, and maps that do not cover exactly the successful output set.
+
+## Analysis and gates
+
+Study reporting uses failure-inclusive Wilson rates, task-clustered paired bootstrap differences with roster/replicate averaging, raw agreement, prevalence, kappa, adjudication rate, severe errors, reviewer effort, total and analysis-set cost, p95 latency, deviations, and retained leakage guesses. A paid workflow must preregister and apply its material-leakage rule before claiming method readiness.
+
+Reliability is undefined—not perfect—when expected agreement is one. A future paid pilot is
+method-ready only with at least 95% double grading, raw agreement at least 80%, kappa at least
+0.60 overall and no family below 0.50, adjudication at most 20%, and no material leakage.
+Otherwise redesign. The committed open-book QA pack cannot satisfy or measure this gate.
+
+Confirmation freezes one strongest baseline. GO requires a point gain of at least 10 percentage points with the paired 95% interval above zero, severe-error noninferiority within +2 points, readiness noninferiority within +5 points, reviewer-effort and latency gates, positive direction in every family and roster, and failure-inclusive validity. Simpler-baseline equivalence, harm, weak reliability, or post-hoc changes trigger redesign or kill per DSE-708.
+
+## Synthetic QA pack
+
+Use 24 synthetic, current-fact-free tasks: 12 operational-execution and 12
+organizational-stewardship, balanced across six subfamilies, three difficulty tiers, and
+`ready`/`not_ready`/`indeterminate`. Public packets and committed fixture keys are separate
+files, but not separate security domains. Paid-study keys must be provisioned outside the
+repository under access control and frozen by hash. Confirmatory tasks are independently
+authored, access-restricted, and screened for sentence, numeric, structural, and semantic
+overlap.
+
+## Non-goals
+
+No provider spend, held-out task publication, hosted dashboard, mutable pricing service, LLM primary grader, routing product, release, or quality claim in this increment.

docs/plans/2026-07-17-h1-method-hardening.md

@@ -0,0 +1,49 @@
+# H1 Method Hardening Implementation Plan
+
+> **Linear:** DSE-708. Use strict TDD on `feat/h1-pilot-pack`; make zero live provider calls.
+
+## Phase 1 — Freeze contract and blocked plan
+
+1. Add failing tests in `tests/evals/test_study_design.py` for complete provenance, two rosters, task-family mapping, cost/spend and policy freezes, tamper detection, exploratory/confirmatory separation, and deterministic task x roster blocked permutations.
+2. Extend `src/conclave/evals/models.py` and `protocols.py`; keep old manifests readable only as synthetic exploratory artifacts.
+3. Re-run focused tests and commit.
+
+## Phase 2 — Atomic grading and blinding
+
+1. Add failing tests for typed atomic errors, severe-error accounting, reviewer seconds, confidence/abstain, grader order/guess, complete double grading, raw agreement/prevalence, undefined constant kappa, and family/roster reliability.
+2. Extend `scoring.py`, `blinding.py`, and models. Queue only successful outputs, normalize views, scan leakage, and hash the separate blind map.
+3. Re-run focused tests and commit.
+
+## Phase 3 — Confirmatory estimands and refusal gates
+
+1. Add tests for task-level roster averaging, task-clustered paired bootstrap, one-sided severe-error and effort/latency noninferiority boundaries, symmetric task exclusions, deviations, complete cost/latency receipts and summaries, and report refusal on freeze drift or exploratory evidence.
+2. Extend `scoring.py`, `reporting.py`, and CLI artifact validation. Archive every gate input and seed.
+3. Re-run focused tests and commit.
+
+## Phase 4 — Synthetic QA pack
+
+1. Add `studies/elite_qa_v1/public_tasks.json`, committed fixture `grader_keys.json`,
+   `qa_protocol.md`, `confirmatory_preregistration.md`, and `README.md`. Classify the pack as
+   open-book harness QA, not a paid pilot.
+2. Add `tests/evals/test_qa_pack.py`: exactly 24 unique tasks, 12/12 families, 4 per
+   subfamily, balanced tier/readiness, matching nonempty fixture keys, packet citation IDs, no
+   grader material in public records, no duplicates, and documented leakage/holdout rules.
+3. Generate and validate an offline manifest only; do not create model outputs.
+4. Re-run focused tests and commit.
+
+## Phase 5 — Verify and hand off
+
+Run:
+
+```bash
+PYTHONPATH=src PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 python -m pytest -p pytest_asyncio.plugin -p no:cacheprovider -q
+python -m ruff check .
+python -m ruff format --check .
+git diff --check
+gitleaks git --redact
+```
+
+Push, open a draft PR stacked on #52, obtain independent methods/code review, and attach
+evidence to DSE-708. Paid pilot execution remains blocked until PR #51 is merged/pinned,
+Ernest approves a hard spend ceiling, and separately access-controlled grader keys are frozen
+by hash before execution.

src/conclave/eval_cli.py

@@ -62,11 +62,15 @@ def _validate_run_summary(study_run: StudyRun) -> None:
     expected_counts = dict(sorted(Counter(record.outcome for record in records).items()))
     expected_tokens = sum(record.completion_tokens or 0 for record in records)
     expected_latency = sum(record.latency_ms or 0.0 for record in records)
+    expected_cost = sum(record.cost_usd for record in records)
+    expected_deviations = sum(len(record.deviation_codes) for record in records)
     if (
         study_run.total_planned_runs != len(records)
         or study_run.outcome_counts != expected_counts
         or study_run.total_completion_tokens != expected_tokens
         or abs(study_run.total_latency_ms - expected_latency) > 1e-9
+        or abs(study_run.total_cost_usd - expected_cost) > 1e-9
+        or study_run.total_deviation_count != expected_deviations
     ):
         _abort("run artifact summary does not match its immutable records")
 

src/conclave/evals/blinding.py

@@ -3,12 +3,13 @@
 from __future__ import annotations
 
 import hashlib
+import json
 import random
 from collections.abc import Sequence
 
-from pydantic import BaseModel, ConfigDict, Field
+from pydantic import BaseModel, ConfigDict, Field, model_validator
 
-from .models import EVAL_SCHEMA_VERSION, RunOutcome, RunRecord, SchemaVersion
+from .models import EVAL_SCHEMA_VERSION, RunOutcome, RunRecord, SchemaVersion, Sha256Digest
 
 
 class BlindModel(BaseModel):
@@ -43,6 +44,37 @@ class BlindMap(BlindModel):
     """Separate restricted identity artifact, never included in grader output."""
 
     entries: tuple[BlindMapEntry, ...]
+    blind_map_hash: Sha256Digest | None = None
+
+    @model_validator(mode="after")
+    def validate_hash(self) -> BlindMap:
+        if self.blind_map_hash is not None and self.blind_map_hash != hash_blind_map(self.entries):
+            raise ValueError("blind map hash does not match its entries")
+        return self
+
+
+class GraderQueueOutput(BlindModel):
+    """One normalized successful output safe for paid human grading."""
+
+    opaque_output_id: str = Field(pattern=r"^output_[0-9a-f]{24}$")
+    presentation: str = Field(min_length=1)
+
+
+class GraderQueue(BlindModel):
+    """Successful-only grader queue with no execution labels or outcomes."""
+
+    outputs: tuple[GraderQueueOutput, ...]
+
+
+def hash_blind_map(entries: Sequence[BlindMapEntry]) -> str:
+    """Hash a restricted identity map in its frozen queue order."""
+
+    canonical = json.dumps(
+        [entry.model_dump(mode="json") for entry in entries],
+        sort_keys=True,
+        separators=(",", ":"),
+    ).encode("utf-8")
+    return f"sha256:{hashlib.sha256(canonical).hexdigest()}"
 
 
 def _opaque_id(*, planned_run_id: str, seed: int) -> str:
@@ -76,3 +108,38 @@ def blind_run_records(
         BlindedOutputSet(outputs=tuple(output for output, _ in blinded)),
         BlindMap(entries=tuple(entry for _, entry in blinded)),
     )
+
+
+def build_grader_queue(
+    records: Sequence[RunRecord], *, seed: int, forbidden_labels: Sequence[str]
+) -> tuple[GraderQueue, BlindMap]:
+    """Normalize, leak-check, and blind successful outputs for paid grading."""
+
+    if len({record.planned_run_id for record in records}) != len(records):
+        raise ValueError("run records must have unique planned_run_id values")
+    blinded = []
+    labels = tuple(label.casefold() for label in forbidden_labels if label)
+    for record in records:
+        if record.outcome != "success":
+            continue
+        if record.output is None:
+            raise ValueError("successful grader-queue records require output")
+        presentation = " ".join(record.output.split())
+        if not presentation:
+            raise ValueError("successful grader-queue records require nonempty output")
+        folded = presentation.casefold()
+        if any(label in folded for label in labels):
+            raise ValueError("grader presentation contains a forbidden label")
+        opaque_id = _opaque_id(planned_run_id=record.planned_run_id, seed=seed)
+        blinded.append(
+            (
+                GraderQueueOutput(opaque_output_id=opaque_id, presentation=presentation),
+                BlindMapEntry(opaque_output_id=opaque_id, planned_run_id=record.planned_run_id),
+            )
+        )
+    random.Random(seed).shuffle(blinded)
+    entries = tuple(entry for _, entry in blinded)
+    return (
+        GraderQueue(outputs=tuple(output for output, _ in blinded)),
+        BlindMap(entries=entries, blind_map_hash=hash_blind_map(entries)),
+    )