diff --git a/CHANGELOG.md b/CHANGELOG.md index 3261b50..3f49353 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,33 @@ All notable changes to conclave are documented here. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [Unreleased] + +### Fixed + +- **`ModelHarnessManifest` now rides on *every* mode's result — a true invariant.** + The auditable manifest was documented as first-class on every `CouncilResult`, but + `debate`, `adversarial`, and `vote` built their result directly in `modes.py` and + returned with `manifest = None` (only `ask`/synthesize attached one). The fix moves + manifest attachment to the single chokepoint every mode funnels through + (`Council._cached_run` → new `Council._ensure_manifest`): it fills the manifest from + the resolved membership + collected answers for `debate`/`adversarial`/`vote` + (including the zero-members early return and cache hits), is a no-op for the + synthesize/raw path that already builds its own richer manifest, and stamps + `secret_safety = verified_no_secrets` when the manifest is provably clean. Regression + tests (`tests/test_manifest_all_modes.py`) pin the invariant per mode so it cannot + drift again. The clustering **verdict** scope is unchanged: it still runs on + `synthesize`/`ask` only and is intentionally not layered onto `adversarial` (which + already emits a judge verdict) — see PDD §4a. + +### Documentation + +- Reconciled `README.md`, `SYSTEM_CONTEXT_DIAGRAM.md`, and the PDD so the + manifest-on-every-result claim is now accurate, and documented the constrained-choice + **`vote` mode** as **shipped** (CAC-09 / #3) rather than "absorbed by `provider_votes`" + — the two are complementary (a fixed ballot vs. clustered free-form stances), not the + same feature. Added the `--mode vote --choices` CLI example and a §4a verdict-scope note. + ## [1.1.0] - 2026-06-21 The **auditable council**. Every run now produces a structured, agreement-scored, @@ -58,10 +85,10 @@ reproducible arithmetic over the model's clustering — never an LLM-emitted fig ### Note -- **`vote` mode (council issue #3) is absorbed/superseded** by the verdict work: - `provider_votes` records which provider took which position (with evidence) and - `consensus_label`/`consensus_score` report the split deterministically — no separate - `vote` mode is shipped or planned. +- **`vote` (council issue #3):** at 1.1.0 the verdict's `provider_votes` + + `consensus_label`/`consensus_score` were considered to subsume it. **Superseded post-1.1.0** + — a real constrained-choice `vote` mode later shipped (CAC-09 / #3; see `[Unreleased]`), + complementary to `provider_votes`: a fixed ballot vs. clustered free-form stances. ## [1.0.0] - 2026-06-14 diff --git a/DOCUMENTATION_INDEX.md b/DOCUMENTATION_INDEX.md index b30b291..ab476f9 100644 --- a/DOCUMENTATION_INDEX.md +++ b/DOCUMENTATION_INDEX.md @@ -23,7 +23,7 @@ the canonical authority spec on top of those. | Doc | Path | Purpose | |-----|------|---------| -| **Product Design Document** | [`docs/PRODUCT_DESIGN_DOCUMENT.md`](docs/PRODUCT_DESIGN_DOCUMENT.md) | **Canonical** product spec and roadmap, reframed in v1.1 to **the auditable multi-model council**. Problem & vision, personas, BYO-keys/key-handling security, council modes (synthesize/raw/debate/adversarial; `vote` absorbed by the verdict), the v1.1 auditable verdict (§4a — CouncilResult v2, deterministic consensus, native structured output, the manifest), provider matrix, architecture, scope + non-goals, demand-gated v1.2 roadmap, the mcp-warden dev-time boundary, licensing & positioning, open questions. **When docs disagree, the PDD wins.** | +| **Product Design Document** | [`docs/PRODUCT_DESIGN_DOCUMENT.md`](docs/PRODUCT_DESIGN_DOCUMENT.md) | **Canonical** product spec and roadmap, reframed in v1.1 to **the auditable multi-model council**. Problem & vision, personas, BYO-keys/key-handling security, council modes (synthesize/raw/debate/adversarial/vote — `vote` shipped as a constrained-choice ballot, CAC-09 / #3, complementary to the verdict's `provider_votes`), the v1.1 auditable verdict (§4a — CouncilResult v2, deterministic consensus, native structured output, the manifest), provider matrix, architecture, scope + non-goals, demand-gated v1.2 roadmap, the mcp-warden dev-time boundary, licensing & positioning, open questions. **When docs disagree, the PDD wins.** | --- @@ -81,7 +81,7 @@ Run: `pytest` (config in `pyproject.toml`, `asyncio_mode = "auto"`). |------|------|---------| | Packaging | [`pyproject.toml`](pyproject.toml) | hatchling build, deps (httpx, pydantic, rich, typer, pyyaml — no LLM SDK), dev extras, console script, pytest config. License: MIT. **PyPI distribution name `conclave-cli`** (the name `conclave` is an unrelated project); command + import stay `conclave`. | | Release runbook | [`RELEASING.md`](RELEASING.md) | Operator runbook: one-time PyPI OIDC Trusted-Publisher setup for `conclave-cli`, cut-a-release checklist (bump→tag→publish Release), post-release verification (Sigstore bundle, PEP 740 attestations), rollback/yank. | -| Changelog | [`CHANGELOG.md`](CHANGELOG.md) | Keep-a-Changelog history per release (SemVer). The `[1.1.0]` (unreleased) entry covers the auditable council — CouncilResult v2, deterministic consensus, native structured output, the manifest, and the absorbed `vote` mode; the 1.0.0 entry covers the distribution rename, key-leak hardening, synthesizer versioning, and release engineering. | +| Changelog | [`CHANGELOG.md`](CHANGELOG.md) | Keep-a-Changelog history per release (SemVer). The `[Unreleased]` entry covers the manifest-on-every-result invariant fix and the `vote`-mode-shipped doc reconciliation; `[1.1.0]` covers the auditable council — CouncilResult v2, deterministic consensus, native structured output, the manifest; the 1.0.0 entry covers the distribution rename, key-leak hardening, synthesizer versioning, and release engineering. | | Dev lockfile | [`requirements-dev.lock`](requirements-dev.lock) | Hash-pinned dev + runtime tree for reproducible installs/CI. Regenerate via `uv pip compile --universal --generate-hashes --python-version 3.11 --extra dev pyproject.toml -o requirements-dev.lock`. | | License | [`LICENSE`](LICENSE) | MIT License. Copyright (c) 2026 Ernest Provo. Matches the `pyproject.toml` license field. | | Security policy | [`SECURITY.md`](SECURITY.md) | BYO-keys vulnerability reporting policy: how to report, scope, and the key-handling guarantees consumers can rely on. | @@ -100,6 +100,7 @@ Run: `pytest` (config in `pyproject.toml`, `asyncio_mode = "auto"`). | Date | Change | |------|--------| +| 2026-07-13 | **Manifest-on-every-result invariant fix + doc reconciliation.** `ModelHarnessManifest` now attaches on all five modes (was `None` for `debate`/`adversarial`/`vote`) via a single-site fix at `Council._cached_run` → new `_ensure_manifest`; regression tests in `tests/test_manifest_all_modes.py`. Docs corrected so the manifest-on-every-result claim is accurate and the `vote` mode is documented as **shipped** (CAC-09 / #3), not "absorbed" — across README, `SYSTEM_CONTEXT_DIAGRAM.md`, PDD (§1/§3/§4a/§8/§9/§12), and `CHANGELOG.md` (`[Unreleased]`). Verdict scope unchanged (still `synthesize`/`ask`-only; not layered on `adversarial`). | | 2026-06-21 | **v1.1 docs pivot — the auditable multi-model council.** PDD reframed to the auditable-council wedge (new §4a: CouncilResult v2, deterministic `position_cluster_ratio_v1` consensus, native + fallback structured output, the verdict-optional rule, the secret-free `ModelHarnessManifest`); `vote` mode marked **absorbed** by `provider_votes` across PDD §1/§4/§8/§9/§12; demand-gated v1.2 "Operable Council" roadmap. System Context diagram gains the verdict pipeline + manifest (mermaid re-validated). README gains an "Auditable verdict" section (CLI panel + library example). `CHANGELOG.md` `[1.1.0]` added. v0.x mode-detail prose archived to [`docs/archive/pdd-v0.x-modes-detail.md`](docs/archive/pdd-v0.x-modes-detail.md) to keep the PDD under 500 lines. New modules documented: `verdict.py`, `agreement.py`, `verdict_synthesis.py`, `manifest.py`. | | 2026-06-14 | v1.0.0 release. Version bump 0.3.0 → 1.0.0; new `CHANGELOG.md` (Keep-a-Changelog). Integrates the three v1.0 PRs below. | | 2026-06-14 | v1.0 distribution + release-engineering (PR-A): PyPI distribution name → `conclave-cli` (command + import stay `conclave`); OIDC Trusted-Publishing release workflow (`.github/workflows/release.yml`, SHA-pinned, PEP 740 attestations, Sigstore keyless signing, inert until a Release fires + publisher configured); `pip-audit` fail-closed CI job in `test.yml`; hash-pinned `requirements-dev.lock`; `RELEASING.md` operator runbook. | diff --git a/README.md b/README.md index 7732f7c..ab51f46 100644 --- a/README.md +++ b/README.md @@ -22,8 +22,11 @@ verdict you can act on — structured, scored for agreement, fully auditable**: `CouncilVerdict` exposing agreement, `conflicts`, `minority_reports`, and `provider_votes`; a deterministic `consensus_score` (arithmetic over the model's clustering, *never* an LLM-emitted number); and a redacted `ModelHarnessManifest` recording how the run executed and -which model produced the disagreement analysis. The verdict is **default-on**. (A `vote` mode -was once on the roadmap; it is **absorbed** by `provider_votes` — see below.) +which model produced the disagreement analysis. The verdict is **default-on**, and the manifest +rides on **every** result — for all five modes (`synthesize`, `raw`, `debate`, `adversarial`, +`vote`), not just `ask`. A constrained-choice **`vote` mode** (`--mode vote --choices ...`) also +shipped in v1.1 (CAC-09 / #3) — distinct from the verdict's `provider_votes`, which score +free-form agreement rather than tally a fixed ballot. See the canonical spec and design docs: @@ -118,16 +121,21 @@ conclave ask "Is a service mesh worth it for 8 services?" \ conclave ask "Defend event sourcing for this ledger." \ -c grok,gemini,perplexity --mode adversarial --proposer grok -# Machine-readable output (works for every mode; carries rounds/adversarial too) +# Vote: each member picks one labelled choice; plurality winner (or split) is tallied +conclave ask "Which datastore for this workload?" \ + -c grok,gemini,claude --mode vote --choices "Postgres,DynamoDB,MongoDB" + +# Machine-readable output (works for every mode; carries rounds/adversarial/vote too) conclave ask "..." -c grok,perplexity --mode debate --json ``` -Mode flags at a glance: `--mode synthesize|raw|debate|adversarial`. `--rounds N` +Mode flags at a glance: `--mode synthesize|raw|debate|adversarial|vote`. `--rounds N` (default 2) is the *maximum* round count for `debate`; `--converge-threshold FLOAT` (or `--converge`/`--no-converge`) optionally stops a debate early once answers stabilize round-over-round (off by default — `--rounds` runs in full). `--proposer -NAME` (default: first member) applies to `adversarial`. `--synthesizer/-s` overrides -the synthesizer *and* the adversarial judge. +NAME` (default: first member) applies to `adversarial`. `--choices "A,B,C"` (two or +more) is required for `vote`. `--synthesizer/-s` overrides the synthesizer *and* the +adversarial judge. Every mode's result carries the auditable `ModelHarnessManifest`. `--council` accepts either a comma-separated list of friendly names or the name of a council defined in your config (see below). The built-in `default` council diff --git a/SYSTEM_CONTEXT_DIAGRAM.md b/SYSTEM_CONTEXT_DIAGRAM.md index 4a9f41f..2485389 100644 --- a/SYSTEM_CONTEXT_DIAGRAM.md +++ b/SYSTEM_CONTEXT_DIAGRAM.md @@ -29,7 +29,7 @@ flowchart TB cli["CLI · conclave ask / providers (cli.py)"] lib["Library API · from conclave import Council (__init__.py)"] council["Council orchestrator
fan_out · synthesize_blocks · skip-no-key (council.py)"] - modes["Deliberation modes
debate · adversarial (modes.py + prompts.py)"] + modes["Deliberation modes
debate · adversarial · vote (modes.py + prompts.py)"] registry["Registry · name to model-id
key PRESENCE only, never values (registry.py)"] config["Config loader · custom endpoints (config.py)"] models["Result contract · CouncilResult v2
answers · verdict · consensus · manifest (models.py)"] @@ -147,7 +147,11 @@ flowchart TB with a prompt-level fallback for providers without strict support. The **`ModelHarnessManifest`** (`manifest.py`) rides on **every** result — first-class, not a debug flag — recording which model + prompt version produced the clustering (provenance) and stamping `secret_safety` - only after the serialized manifest is scanned clean. A verdict is *optional*: open-ended + only after the serialized manifest is scanned clean. This is enforced as a true invariant at + the single chokepoint `Council._cached_run` → `_ensure_manifest`: **all five modes** + (`synthesize`, `raw`, `debate`, `adversarial`, `vote`) funnel through it, so a result can + never escape without a manifest — including the zero-members early return and cache hits. + A verdict is *optional*: open-ended prompts, fewer than two responding members, or extraction failure leave `verdict = None` with the synthesis and member answers intact and the reason recorded on the manifest. - **Streaming shares the same boundary (PDD §9 #5).** A `--stream` run (and the library diff --git a/docs/PRODUCT_DESIGN_DOCUMENT.md b/docs/PRODUCT_DESIGN_DOCUMENT.md index 45c05c6..fa1109b 100644 --- a/docs/PRODUCT_DESIGN_DOCUMENT.md +++ b/docs/PRODUCT_DESIGN_DOCUMENT.md @@ -46,10 +46,10 @@ v1.1 makes the product identity precise: **a multi-model council verdict you can structured, scored for agreement, fully auditable.** Every run yields a `CouncilVerdict` exposing agreement, disagreement (`conflicts`), minority views (`minority_reports`), and per-provider votes (`provider_votes`); a deterministic `consensus_score` (arithmetic over the -model's clustering, *never* an LLM-emitted number); and a redacted `ModelHarnessManifest` -recording how the run executed and which model produced the disagreement analysis (§4a). The -roadmapped `vote` mode is therefore **absorbed and superseded** — "show me who voted for what" -is now `provider_votes`, with evidence, not a separate mode. +model's clustering, *never* an LLM-emitted number); and a redacted `ModelHarnessManifest` (how +the run executed + which model produced the analysis) riding on **every** result across all five +modes (one chokepoint, §4a). A constrained-choice **`vote` mode** also **shipped** (CAC-09 / #3, +`--mode vote --choices "A,B,C"` → plurality winner/split) — distinct from `provider_votes`. conclave's first real use was an **adversarial design review**: a council of Grok, Gemini, Perplexity, and Claude critiquing a security-tool strategy and catching flaws a single @@ -125,19 +125,18 @@ output. The v1.1 verdict layer (§4a) sits on top of whichever mode produced the | **raw** | **BUILT (v0.1)** | Fan out and return every member's raw answer with no synthesis. Not a deliberation mode — it is "synthesize off." Exposed as `--mode raw` / `ask(..., synthesize=False)`. | | **debate** | **BUILT (v0.2)** | N rounds (`--rounds`, default 2). Round 1 is an independent fan-out; rounds 2..N show each member its peers' **anonymized** prior-round answers (`Model A/B/C`) and ask it to revise or defend. A member that errors in a round drops out of later rounds; the debate continues with survivors. The synthesizer consolidates the final round. Exposed as `--mode debate` / `Council.debate()` / `debate_sync()`. | | **adversarial** | **BUILT (v0.2)** | Structured propose → refute → verdict. A `--proposer` (default: first member) answers; the remaining members are CRITICS explicitly prompted to refute it; the synthesizer acts as JUDGE, weighing proposal vs. critiques and issuing a verdict + strengthened answer. This is the mode conclave's origin story (the security design review) exercised by hand. Exposed as `--mode adversarial` / `Council.adversarial()` / `adversarial_sync()`. | -| ~~**vote**~~ | **ABSORBED (v1.1)** | ~~Structured majority with reported split.~~ Superseded by the v1.1 verdict: `provider_votes` records which provider took which position (with evidence) and `consensus_label`/`consensus_score` report the split deterministically — no separate mode needed. See §4a. | +| **vote** | **BUILT (v1.1, CAC-09 / #3)** | Constrained-choice ballot: each member sees a fixed labelled option set (`A, B, C, …`) and answers with one letter; responses are tallied to a plurality `winner` (or `split` on a tie) on `result.vote` (`VoteResult`). Exposed as `--mode vote --choices "A,B,C"` / `Council.vote()` / `vote_sync()`. Distinct from the verdict's `provider_votes`, which cluster free-form stances with evidence (§4a); `vote` tallies a fixed ballot. | **Mode algorithms (as built).** The step-by-step "as built" prose for synthesize / raw / -debate / adversarial (fan-out + partial-results, peer anonymization + drop-out, proposer → -critic → judge) is landed history and lives in -[`docs/archive/pdd-v0.x-modes-detail.md`](archive/pdd-v0.x-modes-detail.md). In brief: every -mode fans out concurrently, captures each call as a `ModelAnswer` (answer **or** redacted -error — `call_model` never raises), and survives partial failure; the deliberation modes -extend `CouncilResult` (`mode`, `rounds`, `adversarial`) backward-compatibly so v0.1 -`answers`/`synthesis` consumers keep working. The mode *text* output is a generative -reconciliation, inherently stochastic (load-bearing for the mcp-warden boundary, §10); the -v1.1 verdict (§4a) adds a *deterministic* agreement number on top, by arithmetic over a -clustering, never lifted from that free text. +debate / adversarial / vote (fan-out + partial-results, peer anonymization + drop-out, proposer → +critic → judge, ballot tally) is landed history and lives in +[`docs/archive/pdd-v0.x-modes-detail.md`](archive/pdd-v0.x-modes-detail.md). In brief: every mode +fans out concurrently, captures each call as a `ModelAnswer` (answer **or** redacted error — +`call_model` never raises), and survives partial failure; the deliberation modes extend +`CouncilResult` (`mode`, `rounds`, `adversarial`, `vote`) backward-compatibly so v0.1 +`answers`/`synthesis` consumers keep working. The mode *text* output is a generative, inherently +stochastic reconciliation (load-bearing for the mcp-warden boundary, §10); the v1.1 verdict (§4a) +adds a *deterministic* agreement number on top, by arithmetic over a clustering. --- @@ -242,25 +241,30 @@ verdict appears identically in the buffered result and the streaming `done` even `secret_safety` stamp is re-run over the final content. **Cost:** default-on adds exactly **one** extra synthesizer call per run; the opt-out exists for cost-sensitive callers. +**Verdict scope across modes (deliberate).** Posture: *manifest everywhere, clustering verdict only where unambiguously additive.* The manifest rides on all five modes; the clustering verdict still runs on `synthesize`/`ask` only — deferred (not forced) on `debate`/`vote` (a future change may opt them in behind `extract_verdict`), and **intentionally NOT on `adversarial`**, which already emits a judge verdict the clustering verdict would double-count. + ### ModelHarnessManifest — first-class, secret-free -The `ModelHarnessManifest` (`manifest.py`) rides on every `CouncilResult` — *not* behind a -debug flag. It records `request_id`, `conclave_version`, `mode`, `providers_considered/ -called/skipped` (each skip a `ProviderSkip{name, reason}`), `model_ids`, -`generation_settings`, `receipts` (each a `ProviderExecutionReceipt{name, provider, -model_id, generation_settings, latency_ms, usage, error(redacted), schema_valid}`), -`total_latency_ms`, `total_usage`, `schema_valid`, `redacted_errors`. Verdict-provenance -slots: `verdict_extraction: VerdictExtraction{model_id, prompt_version}` (which model + prompt -version produced the disagreement analysis — *the* auditability hook), `verdict_type`, -`consensus_method`, `verdict_absent_reason`. Two deliberate honesty choices: - -- **No invented pricing.** `estimated_cost` is `None` (a wrong number in an audit receipt is - worse than none); `pricing_snapshot_date` is the dated-estimate slot, `None` until a real - pricing table exists. Usage (tokens) is recorded; cost is not guessed. +The `ModelHarnessManifest` (`manifest.py`) rides on every `CouncilResult` — *not* behind a debug +flag, and now a **true invariant** enforced at one chokepoint: all five modes funnel through +`Council._cached_run` → `_ensure_manifest`, which attaches the manifest on every returned result +— including `debate`/`adversarial`/`vote` (built in `modes.py`), the zero-members early return, +and cache hits (synthesize/raw builds its own richer one earlier). Pinned by +`tests/test_manifest_all_modes.py`. It records `request_id`, `conclave_version`, `mode`, +`providers_considered/called/skipped` +(each skip a `ProviderSkip{name, reason}`), `model_ids`, `generation_settings`, `receipts` (each +a `ProviderExecutionReceipt{name, provider, model_id, generation_settings, latency_ms, usage, +error(redacted), schema_valid}`), `total_latency_ms`, `total_usage`, `schema_valid`, +`redacted_errors`, and verdict-provenance slots (`verdict_extraction: VerdictExtraction{model_id, +prompt_version}` — the auditability hook — plus `verdict_type`, `consensus_method`, +`verdict_absent_reason`). Two deliberate honesty choices: + +- **No invented pricing.** `estimated_cost` stays `None` (a wrong number in an audit receipt is + worse than none); `pricing_snapshot_date` is the dated-estimate slot until a real pricing table + exists. Usage (tokens) is recorded; cost is not guessed. - **Proven secret-safety.** `secret_safety` defaults to `unverified`, promoted to - `verified_no_secrets` **only** after `scan_for_secret_material()` proves the serialized - manifest free of forbidden substrings (`sk-`, `bearer`, `authorization`, `api_key`, - `x-api-key`). Key *values* never appear; errors are redacted upstream and re-redacted on - construction. + `verified_no_secrets` **only** after `scan_for_secret_material()` proves the serialized manifest + free of forbidden substrings (`sk-`, `bearer`, `authorization`, `api_key`, `x-api-key`). Key + *values* never appear; errors are redacted upstream and re-redacted on construction. --- @@ -308,7 +312,7 @@ consumers. The end-to-end flow — `CLI/Library → Council → call_model → a | Module | Responsibility | |--------|----------------| -| `council.py` | `Council` — primary importable entry point. Resolves names, partitions members, and exposes two reusable primitives: `fan_out` (the single concurrent + partial-failure call loop) and `synthesize_blocks` (the single synthesizer/judge call path). Hosts the public mode API: `ask`/`ask_sync` (synthesize/raw), `debate`/`debate_sync`, `adversarial`/`adversarial_sync`. Sync wrappers guard against a running event loop. Runs `_apply_verdict` (default-on; `extract_verdict=False` opts out) on both buffered + streaming paths after the manifest exists. | +| `council.py` | `Council` — primary importable entry point. Resolves names, partitions members, and exposes two reusable primitives: `fan_out` (the single concurrent + partial-failure call loop) and `synthesize_blocks` (the single synthesizer/judge call path). Hosts the public mode API: `ask`/`ask_sync` (synthesize/raw), `debate`/`debate_sync`, `adversarial`/`adversarial_sync`, `vote`/`vote_sync`. Sync wrappers guard against a running event loop. Every mode funnels through `_cached_run`, which enforces the manifest-on-every-result invariant via `_ensure_manifest`. Runs `_apply_verdict` (default-on; `extract_verdict=False` opts out) on the synthesize buffered + streaming paths after the manifest exists. | | `verdict.py` | Public verdict/member Pydantic types (`CouncilVerdict`, `CouncilPosition`, `CouncilConflict`, `ProviderVote`, `MinorityReport`) + the LCD JSON Schemas (`verdict_json_schema`/`member_answer_json_schema`/`verdict_extraction_json_schema`) usable across all three native structured-output surfaces; `VERDICT_SCHEMA_VERSION`. | | `agreement.py` | Deterministic consensus: `consensus_score` (`position_cluster_ratio_v1` — largest cluster / positioned members; `None` for N<2) + `consensus_label` buckets. Pure arithmetic, no `difflib`, never LLM-emitted. | | `verdict_synthesis.py` | `extract_verdict` engine: one extraction call (clusters stances, never emits a number), native `output_contract` enforcement + prompt-level fallback, validate → repair-once → graceful `verdict=None`; the three verdict-absent reasons; provenance on every return path. | @@ -349,28 +353,24 @@ Typer + Rich, PyYAML. **No LLM-SDK dependency.** hatchling build; console script Condensed history (v0.x mode-detail archived per §4, per-release changelog in `CHANGELOG.md`, verdict layer in §4a): - **v0.1:** `synthesize` + `raw` modes; first-class providers (5, now 9) + adapter - pass-through; BYO-keys by env-var name with graceful skip; concurrent fan-out; - structured `CouncilResult` (latency, usage, per-model error); CLI (`ask`/`providers`, - `--json`); config (named models/councils, default synthesizer); sync + async library - API; transport-mocked test suite (no network, no keys). -- **v0.2:** `debate` (multi-round, anonymized peers, drop-out, `CouncilResult.rounds`) and - `adversarial` (proposer → critics → judge, `CouncilResult.adversarial`); backward- - compatible `CouncilResult` extension; both on the library API + CLI with rich rendering. -- **v0.3:** **LiteLLM removed** → owned `httpx` provider highway + adapter registry (§6), - the only network dependency; three adapters cover all nine providers; custom - OpenAI-compatible `endpoints:` (config-only); key-leak hardening via `redact()` (was §9 - item 7); `call_model` signature + never-raises contract unchanged. -- **v1.0 (stable):** distribution name `conclave-cli`; OIDC Trusted-Publishing release - workflow + Sigstore + PEP 740; key-leak threat model (`SECURITY.md`, cause-chain fix, - transport-logging guard default-on); versioned synthesis prompt - (`SYNTHESIS_PROMPT_VERSION` → `result.prompt_version`); streaming (synthesize/raw) + - optional result cache + debate convergence early-stop. + pass-through; BYO-keys by env-var name with graceful skip; concurrent fan-out; structured + `CouncilResult`; CLI (`ask`/`providers`, `--json`); config; sync + async API; mocked suite. +- **v0.2:** `debate` (multi-round, anonymized peers, drop-out, `.rounds`) and `adversarial` + (proposer → critics → judge, `.adversarial`); backward-compatible `CouncilResult` extension. +- **v0.3:** **LiteLLM removed** → owned `httpx` provider highway + adapter registry (§6, the + only network dependency; 3 adapters cover 9 providers); custom OpenAI-compatible `endpoints:`; + key-leak hardening via `redact()`; `call_model` signature + never-raises contract unchanged. +- **v1.0 (stable):** dist name `conclave-cli`; OIDC Trusted-Publishing + Sigstore + PEP 740; + key-leak threat model (`SECURITY.md`, transport-logging guard default-on); versioned synthesis + prompt (`SYNTHESIS_PROMPT_VERSION`); streaming (synthesize/raw) + result cache + debate early-stop. - **v1.1 (the auditable council):** `CouncilResult` v2 — `verdict` + `consensus_*` + `conflicts`/`provider_votes`/`minority_reports` + first-class `manifest`; deterministic `position_cluster_ratio_v1` consensus; native + fallback structured output across OpenAI/Anthropic/Gemini; the verdict-optional rule; verdict default-on with - `Council(extract_verdict=False)` opt-out. The `vote` mode is **absorbed** by - `provider_votes`. Full detail in §4a. + `Council(extract_verdict=False)` opt-out; the auditable `manifest` made a true + every-mode invariant (one chokepoint). A constrained-choice **`vote` mode** shipped + (CAC-09 / #3) — `--mode vote --choices` — distinct from the verdict's `provider_votes`. + Full detail in §4a. --- @@ -383,9 +383,9 @@ Condensed history (v0.x mode-detail archived per §4, per-release changelog in ` - **Not an agent framework.** No tool-calling graphs, stateful agents, or orchestration DSL — we compete by being *small*. (Permanent.) - **Not a key manager / secrets vault.** conclave reads env vars; it does not provision, rotate, store, or proxy keys. (Permanent.) - **No hosted/proxied token path.** No conclave-operated endpoint that sees prompts or takes a margin — BYO-keys, direct-to-provider, always. (Permanent.) -- **No streaming for debate/adversarial** (synthesize/raw streaming landed in v0.3, #7). -- **No server mode** (possible Roadmap, §9). -- ~~**No `vote` mode** yet.~~ **Absorbed in v1.1** — `provider_votes` + `consensus_label`/`consensus_score` deliver "who voted for what, and the split" with evidence (§4a). +- **No streaming for debate/adversarial/vote** (synthesize/raw streaming landed in v0.3, #7). +- **No server mode** (possible Roadmap, §9; local HTTP spike #8 closed no-go — §9 item 6). +- ~~**No `vote` mode** yet.~~ **Shipped in v1.1** (CAC-09 / #3) — `--mode vote --choices "A,B,C"` tallies a fixed ballot (plurality winner or split). Complementary to the verdict's `provider_votes`, which cluster free-form stances with evidence (§4a); the two are not the same thing. --- @@ -408,7 +408,7 @@ score is validated against real runs. ### Landed history (kept struck-through for traceability) -1. ~~**`vote` mode**~~ **ABSORBED in v1.1** — `provider_votes` + `consensus_label`/`consensus_score` deliver "who voted for what + the split" with evidence; the structured-answer-schema prerequisite is satisfied by the LCD schemas + native structured output (§4a). +1. ~~**`vote` mode**~~ **SHIPPED in v1.1 (CAC-09 / #3)** — a real constrained-choice mode (`--mode vote --choices "A,B,C"` / `Council.vote()`): each member picks one labelled option, tallied to a plurality `winner`/`split` on `result.vote`. Complementary to the verdict's `provider_votes` (§4a), not the same thing. 2. ~~**Debate convergence/stop criteria**~~ **LANDED (#4)** — opt-in `converge_threshold` early-stop on round-over-round text stability (`difflib`); off by default; recorded on `CouncilResult.converged`/`convergence_score`. (NB: text-stability ≠ the verdict `consensus_score`, §4a.) 3. ~~**More first-class providers**~~ **LANDED (#5)** — `groq`/`deepseek`/`mistral`/`together` promoted to typed defaults (`registry.OPENAI_COMPAT_PROVIDERS`); no native adapter; aggregators excluded (§11). 4. ~~**Caching**~~ **LANDED (#6)** — opt-in result cache (`config.cache` / `--cache`), off by default, on-disk, secret-free SHA-256 key, corrupt = silent miss; hit on `CouncilResult.cached`. @@ -483,11 +483,12 @@ the auditable verdict, the owned provider layer, the no-aggregator posture, and **Open:** none currently. -2. ~~**`vote` answer schema.**~~ **RESOLVED in v1.1** — the question (constrained answer - format vs. post-hoc tally, and its structured-output prerequisite) is moot: the verdict - ships the LCD verdict/member JSON Schemas enforced via native structured output across - all three adapters, and `provider_votes` records the per-provider positions. The - structured-output prerequisite this question flagged is now landed (§4a). +2. ~~**`vote` answer schema.**~~ **RESOLVED in v1.1** by *two* complementary deliveries: (a) a + real constrained-choice **`vote` mode** (CAC-09 / #3) — fixed labelled ballot, plurality + winner/split on `result.vote`; and (b) the verdict's structured `provider_votes`, which + cluster free-form stances with evidence (LCD JSON Schemas + native structured output, §4a). + The constrained-answer-format question is answered by the ballot; the tally question by + `provider_votes`. **Resolved (2026-06-08):** questions 1 (synthesizer-in-council), 3 (per-member overrides), 4 (server-mode scope, plus the 2026-06-09 #8 spike outcome), and 5 (first-class provider diff --git a/src/conclave/council.py b/src/conclave/council.py index c8dbbb5..86d7b34 100644 --- a/src/conclave/council.py +++ b/src/conclave/council.py @@ -244,9 +244,18 @@ async def _cached_run( and the providers are not called. On a miss (or when caching is off) the live ``run`` executes; a successful live run is stored best-effort. Cache read/write failures never propagate -- they degrade to a normal live run. + + This is the single chokepoint every mode funnels through, so it is also + where the manifest-on-every-result invariant is enforced: each returned + result passes through :meth:`_ensure_manifest` (a no-op for the + synthesize/raw path, which builds its own richer manifest in + :meth:`_ask_uncached`; a fill for ``debate``/``adversarial``/``vote`` and + for a cache hit stored before the manifest existed). """ if not self.cache_enabled: - return await run() + result = await run() + self._ensure_manifest(result, mode) + return result key = self._cache_key( prompt, @@ -259,12 +268,52 @@ async def _cached_run( hit = cache_mod.load(key) if hit is not None: logger.info("cache hit for %s run (%s)", mode, key[:12]) + self._ensure_manifest(hit, mode) return hit result = await run() + self._ensure_manifest(result, mode) cache_mod.store(key, result) return result + def _ensure_manifest(self, result: CouncilResult, mode: str) -> None: + """Guarantee the manifest-on-every-result invariant at the single chokepoint. + + Every mode funnels through :meth:`_cached_run`, so attaching the auditable + :class:`ModelHarnessManifest` here makes it a true invariant rather than a + per-mode responsibility that can silently drift (the drift this method + fixes: ``debate``/``adversarial``/``vote`` used to return with + ``manifest is None``). + + A no-op when ``result.manifest`` already exists: the synthesize/raw path + builds its own manifest inside :meth:`_ask_uncached` so it can populate the + verdict-provenance slots and re-stamp secret-safety over them, and this + method must not overwrite that richer manifest. The + ``debate``/``adversarial``/``vote`` wrappers return a result with + ``manifest is None`` -- this fills it from the resolved membership and the + collected answers, including the zero-members early-return path and a stale + cache hit stored before this invariant existed, so no result ever escapes + without an audit manifest. + + Membership is re-resolved via :meth:`_available_members` (the same + resolution :meth:`_cache_key` already performs) rather than threaded back + through every mode's return value; this keeps ``providers_called`` / + ``model_ids`` reflecting the full resolved membership even for debate rounds + where a member later dropped out, while the per-answer receipts are built + from ``result.answers``. :meth:`_build_manifest` stamps ``secret_safety`` + VERIFIED when the assembled manifest is provably clean. + + Args: + result: The result to attach a manifest to. Mutated in place. + mode: The deliberation mode string recorded on the manifest. + """ + if result.manifest is not None: + return + members, skipped = self._available_members() + result.manifest = self._build_manifest( + mode=mode, members=members, skipped=skipped, answers=result.answers + ) + async def fan_out( self, members: list[tuple[str, str]], @@ -349,7 +398,10 @@ def _build_manifest( :func:`conclave.providers.receipt_from_answer`). It works for both the normal path (``answers`` populated) and the empty-members path (``members``/``answers`` empty, ``skipped`` listing every requested name) - so every live ``ask`` returns a manifest. + so every live run returns a manifest. It is called for the synthesize/raw + path directly in :meth:`_ask_uncached` and for ``debate``/``adversarial``/ + ``vote`` via :meth:`_ensure_manifest`, so the ``mode`` argument spans every + deliberation mode. The ``conclave_version`` is read via a deferred import: ``conclave`` imports this module at package init *before* it assigns ``__version__``, @@ -362,7 +414,8 @@ def _build_manifest( (:func:`conclave.manifest.verified_secret_safety`). Args: - mode: Deliberation mode (``"synthesize"``/``"raw"``). + mode: Deliberation mode (``"synthesize"``/``"raw"``/``"debate"``/ + ``"adversarial"``/``"vote"``). members: ``(friendly_name, model_id)`` pairs that were called. skipped: Friendly names skipped for a missing key. answers: The collected per-member answers (empty on the no-members path). diff --git a/tests/test_manifest_all_modes.py b/tests/test_manifest_all_modes.py new file mode 100644 index 0000000..643956d --- /dev/null +++ b/tests/test_manifest_all_modes.py @@ -0,0 +1,270 @@ +"""Regression guard: the ModelHarnessManifest rides on EVERY mode's result. + +The auditable manifest (CAC-04) is documented as first-class on every +:class:`conclave.models.CouncilResult` -- "it rides on every result, not behind +a debug flag." Historically only the synthesize/raw path (built in +``Council._ask_uncached``) satisfied this; ``debate``/``adversarial``/``vote`` +constructed their result directly in :mod:`conclave.modes` and returned with +``manifest is None``, so the invariant silently drifted for 3 of 5 modes and no +test guarded it. + +The single-site fix attaches the manifest in ``Council._cached_run`` (the one +chokepoint every mode funnels through) via ``Council._ensure_manifest``. These +tests pin the invariant for debate, adversarial, and vote -- including the +zero-members early-return and cache-hit paths -- so it cannot drift again. All +run offline through the shared ``patch_call_model`` seam; no network, no keys. +""" + +from __future__ import annotations + +import pytest + +from conclave import Council +from conclave.config import ConclaveConfig +from conclave.manifest import SECRET_SAFETY_VERIFIED, ModelHarnessManifest +from tests.conftest import make_response + + +def _all_keys(monkeypatch) -> None: + """Set every provider key to a dummy non-empty value.""" + for var in ( + "XAI_API_KEY", + "GEMINI_API_KEY", + "ANTHROPIC_API_KEY", + "PERPLEXITY_API_KEY", + "OPENAI_API_KEY", + ): + monkeypatch.setenv(var, "dummy-key") + + +def _config() -> ConclaveConfig: + """A deterministic config independent of any on-disk ~/.conclave file.""" + return ConclaveConfig( + models={ + "grok": "xai/grok-4.3", + "gemini": "gemini/gemini-2.5-pro", + "claude": "anthropic/claude-sonnet-4-6", + "perplexity": "perplexity/sonar-pro", + }, + councils={"default": ["grok", "gemini", "claude", "perplexity"]}, + synthesizer="claude", + ) + + +def _system_text(messages) -> str: + """Return the system-role content of a message list, or '' if none.""" + for m in messages: + if m.get("role") == "system": + return m.get("content", "") + return "" + + +def _assert_verified_manifest(result, expected_mode: str) -> None: + """The result carries a secret-safety-VERIFIED manifest stamped with the mode.""" + manifest = result.manifest + assert manifest is not None, f"{expected_mode} result must carry a manifest" + assert isinstance(manifest, ModelHarnessManifest) + assert manifest.secret_safety == SECRET_SAFETY_VERIFIED + assert manifest.mode == expected_mode + # request_id is a uuid4 hex (32 lowercase hex chars) -- a real, assembled manifest. + assert len(manifest.request_id) == 32 + assert all(c in "0123456789abcdef" for c in manifest.request_id) + + +# --------------------------------------------------------------------------- # +# Debate +# --------------------------------------------------------------------------- # + + +async def test_debate_result_carries_verified_manifest(monkeypatch, patch_call_model): + """A debate run attaches a secret-safe manifest (was manifest=None before the fix).""" + _all_keys(monkeypatch) + + def handler(model, messages, **kwargs): + if "synthesizer concluding a multi-round" in _system_text(messages): + return make_response("DEBATE SYNTHESIS") + return make_response(f"answer from {model}") + + patch_call_model(handler) + council = Council( + models=["grok", "gemini", "perplexity"], synthesizer="claude", config=_config() + ) + result = await council.debate("Is P=NP?", rounds=2) + + _assert_verified_manifest(result, "debate") + # Full resolved membership is recorded even though only survivors answer. + assert result.manifest.providers_considered == ["grok", "gemini", "perplexity"] + assert result.manifest.providers_called == ["grok", "gemini", "perplexity"] + + +async def test_debate_dropped_member_still_in_manifest_membership(monkeypatch, patch_call_model): + """A member that drops mid-debate stays in providers_called (full membership).""" + _all_keys(monkeypatch) + + def handler(model, messages, **kwargs): + system = _system_text(messages) + if "synthesizer concluding a multi-round" in system: + return make_response("SYNTH") + if model == "gemini/gemini-2.5-pro": # gemini fails every round -> drops out + raise RuntimeError("gemini down") + return make_response(f"answer from {model}") + + patch_call_model(handler) + council = Council( + models=["grok", "gemini", "perplexity"], synthesizer="claude", config=_config() + ) + result = await council.debate("q", rounds=2) + + _assert_verified_manifest(result, "debate") + # Membership reflects everyone that was called, not just final-round survivors. + assert set(result.manifest.providers_called) == {"grok", "gemini", "perplexity"} + + +async def test_debate_no_members_still_carries_manifest(monkeypatch, patch_call_model, clear_keys): + """The zero-members early return in run_debate still yields a manifest.""" + + def handler(model, messages, **kwargs): # pragma: no cover - never called + return make_response("unused") + + patch_call_model(handler) + council = Council(models=["grok", "claude"], config=_config()) + result = await council.debate("q", rounds=2) + + _assert_verified_manifest(result, "debate") + assert result.manifest.receipts == [] + assert result.manifest.providers_called == [] + assert {s.name for s in result.manifest.providers_skipped} == {"grok", "claude"} + + +# --------------------------------------------------------------------------- # +# Adversarial +# --------------------------------------------------------------------------- # + + +async def test_adversarial_result_carries_verified_manifest(monkeypatch, patch_call_model): + """An adversarial run attaches a secret-safe manifest (was manifest=None before).""" + _all_keys(monkeypatch) + + def handler(model, messages, **kwargs): + system = _system_text(messages) + if "judge of an adversarial review" in system: + return make_response("VERDICT TEXT") + if "critic on an adversarial review" in system: + return make_response(f"critique from {model}") + return make_response(f"proposal from {model}") + + patch_call_model(handler) + council = Council( + models=["grok", "gemini", "perplexity"], synthesizer="claude", config=_config() + ) + result = await council.adversarial("Defend microservices.") + + _assert_verified_manifest(result, "adversarial") + assert result.manifest.providers_considered == ["grok", "gemini", "perplexity"] + + +async def test_adversarial_no_members_still_carries_manifest( + monkeypatch, patch_call_model, clear_keys +): + """The zero-members early return in run_adversarial still yields a manifest.""" + + def handler(model, messages, **kwargs): # pragma: no cover - never called + return make_response("unused") + + patch_call_model(handler) + council = Council(models=["grok", "claude"], config=_config()) + result = await council.adversarial("q") + + _assert_verified_manifest(result, "adversarial") + assert result.manifest.providers_called == [] + assert {s.name for s in result.manifest.providers_skipped} == {"grok", "claude"} + + +# --------------------------------------------------------------------------- # +# Vote +# --------------------------------------------------------------------------- # + + +async def test_vote_result_carries_verified_manifest(monkeypatch, patch_call_model): + """A vote run attaches a secret-safe manifest (was manifest=None before the fix).""" + _all_keys(monkeypatch) + + def handler(model, messages, **kwargs): + return make_response("A") + + patch_call_model(handler) + council = Council( + models=["grok", "gemini", "perplexity"], synthesizer="claude", config=_config() + ) + result = await council.vote("Best option?", ["Alpha", "Beta"]) + + _assert_verified_manifest(result, "vote") + assert result.manifest.providers_called == ["grok", "gemini", "perplexity"] + # One receipt per member that voted. + assert len(result.manifest.receipts) == len(result.answers) + + +async def test_vote_no_members_still_carries_manifest(monkeypatch, patch_call_model, clear_keys): + """The zero-members early return in run_vote still yields a manifest.""" + + def handler(model, messages, **kwargs): # pragma: no cover - never called + return make_response("A") + + patch_call_model(handler) + council = Council(models=["grok", "claude"], config=_config()) + result = await council.vote("Pick", ["Yes", "No"]) + + _assert_verified_manifest(result, "vote") + assert result.manifest.receipts == [] + assert {s.name for s in result.manifest.providers_skipped} == {"grok", "claude"} + + +# --------------------------------------------------------------------------- # +# Cache-hit path also carries the manifest +# --------------------------------------------------------------------------- # + + +@pytest.mark.parametrize("mode", ["debate", "adversarial", "vote"]) +async def test_cache_hit_carries_manifest(monkeypatch, patch_call_model, tmp_path, mode): + """A cached result for each mode still carries a manifest on the hit path. + + Two identical runs with caching enabled: the second is served from cache + (``cached is True``) and must still expose a VERIFIED manifest, proving the + invariant holds on the cache-hit branch of ``_cached_run`` too. + """ + _all_keys(monkeypatch) + # Point the on-disk cache (XDG_CACHE_HOME/conclave) at an isolated tmp dir. + monkeypatch.setenv("XDG_CACHE_HOME", str(tmp_path)) + + def handler(model, messages, **kwargs): + system = _system_text(messages) + if "judge of an adversarial review" in system: + return make_response("VERDICT") + if "critic on an adversarial review" in system: + return make_response("critique") + if "synthesizer concluding a multi-round" in system: + return make_response("SYNTH") + return make_response("A") + + patch_call_model(handler) + council = Council( + models=["grok", "gemini", "perplexity"], + synthesizer="claude", + config=_config(), + cache=True, + ) + + async def run(): + if mode == "debate": + return await council.debate("q", rounds=2) + if mode == "adversarial": + return await council.adversarial("q") + return await council.vote("q", ["Alpha", "Beta"]) + + first = await run() + assert first.cached is False + _assert_verified_manifest(first, mode) + + second = await run() + assert second.cached is True + _assert_verified_manifest(second, mode)