Master index for conclave's documentation. conclave follows the 3-Core Documentation Rule: every project maintains exactly three core docs — an overview (README), a system context diagram, and this index linking everything together. The Product Design Document is the canonical authority spec on top of those.
- Repo:
/Users/ernestprovo/dev/conclave/ - Version: 1.2.0 · License: MIT
- Last updated: 2026-07-19
| # | Doc | Path | Purpose |
|---|---|---|---|
| 1 | README (project overview) | README.md |
Install, BYO-keys, six-mode CLI/library quickstart including Elite, and the gated paid exploratory eval lane. |
| 2 | System Context Diagram | SYSTEM_CONTEXT_DIAGRAM.md |
Mermaid context for the provider highway, released/source modes, verdict pipeline, eval live guard, manifest receipts, and mcp-warden boundary. |
| 3 | Documentation Index | DOCUMENTATION_INDEX.md |
This file. Master map of all docs + source layout. |
| Doc | Path | Purpose |
|---|---|---|
| Product Design Document | docs/PRODUCT_DESIGN_DOCUMENT.md |
Canonical product spec: six released modes, the v1.1 execution-traceable verdict, v1.2 Elite and evaluation guard, manifest, architecture, and roadmap. When docs disagree, the PDD wins. |
| Doc | Path | Purpose |
|---|---|---|
| Decision Quality Roadmap | docs/plans/2026-07-17-decision-quality-roadmap.md |
H0-H4 prove-it roadmap: released Elite baseline, active H1 exploratory pilot, randomized ablations, source-grounded Decision Briefs, buyer pull, outcome learning, demand gates, and kill criteria. |
| H1 Evaluation Design | docs/plans/2026-07-17-h1-budget-matched-evaluation-design.md |
DSE-708 experimental boundary, six frozen conditions, replay, blinding, and failure-inclusive analysis. |
| H1 Evaluation Plan | docs/plans/2026-07-17-h1-budget-matched-evaluation.md |
TDD delivery plan for the offline, budget-matched evaluation substrate. |
| H1 Method Hardening | docs/plans/2026-07-17-h1-method-hardening-design.md |
Paid-study provenance, task-clustered inference, grader controls, confirmatory refusal boundaries, and QA design. |
| H1 Synthetic QA Pack | studies/elite_qa_v1/README.md |
Balanced 24-task open-book harness fixture, QA protocol, and confirmatory preregistration template. |
| H1 Live Runner Design | docs/plans/2026-07-18-h1-live-evaluation-runner-design.md |
Sequential paid-exploratory execution, hash-bound UTF-8 estimates, USD 10 cap, authenticated checkpoints, and no-repeat resume. |
| H1 Live Runner Plan | docs/plans/2026-07-18-h1-live-evaluation-runner.md |
Exact TDD tasks for the six live conditions, dry-run estimator, replay fixtures, CLI gate, and correctness-only paid smoke. |
| Durable JSON Output Design | docs/plans/2026-07-21-durable-json-output-design.md |
Opt-in atomic user-private result persistence for long buffered council runs and detached supervisors. |
Package root: src/conclave/ (installed as the conclave package; console script
conclave = conclave.cli:app).
| Module | Path | Responsibility |
|---|---|---|
| Package API | src/conclave/__init__.py |
Public exports include Council, result types, verdict/manifest surface, EliteResult, and Elite protocol constants. |
| Council | src/conclave/council.py |
Six released async/sync modes; Elite keeps protocol completion separate from ready/not_ready/indeterminate decision readiness. |
| Modes | src/conclave/modes.py |
Debate, adversarial, vote, and Elite orchestration; Elite gates initial/claim-audit/revision at three successes. |
| Verdict types | src/conclave/verdict.py |
Public verdict/member Pydantic types (CouncilVerdict, CouncilPosition, CouncilConflict, ProviderVote, MinorityReport) + the LCD JSON Schemas (verdict_json_schema/member_answer_json_schema/verdict_extraction_json_schema) usable across all three native structured-output surfaces; VERDICT_SCHEMA_VERSION/VERDICT_EXTRACTION_PROMPT_VERSION. |
| Agreement | src/conclave/agreement.py |
Deterministic consensus: consensus_score (position_cluster_ratio_v1 — largest cluster / positioned members; None for N<2) + consensus_label buckets. Pure arithmetic, no difflib, never LLM-emitted. |
| Verdict synthesis | src/conclave/verdict_synthesis.py |
extract_verdict engine: one initial extraction call and at most one repair, native output_contract enforcement + prompt-level fallback, validate → repair-once → graceful verdict=None; the three verdict-absent reasons; provenance on every return path. |
| Manifest | src/conclave/manifest.py |
Secret-scanned manifest on every result; buffered Elite receipts cover all attempted phases through synthesis, extraction, and repair. Unknown cost remains None. |
| Streaming | src/conclave/streaming.py |
stream_ask — council-level streaming engine behind Council.ask_stream: concurrent member interleaving via an asyncio.Queue, optional synthesizer streaming, terminal done event with the full CouncilResult (synthesize/raw only). |
| Prompts | src/conclave/prompts.py |
Debate/adversarial/vote plus anonymized Elite claim-audit and revision prompts; answer IDs provide within-run provenance, not external grounding. |
| Providers | src/conclave/providers.py |
Async call_model (buffered) + call_model_stream (SSE) paths: resolve the adapter, read the key by name at call time, call transport, parse; latency/usage/redacted-error capture; never raises (partial text preserved on mid-stream failure). |
| Transport | src/conclave/transport.py |
post_json + stream_sse — the single async httpx network boundary (buffered POST and client.stream(...) SSE) for the whole provider highway. |
| Adapter registry | src/conclave/adapters/__init__.py |
resolve_adapter(model_id, config) — provider registry + extension seam (one registration per family; config-only for OpenAI-compatible endpoints). |
| Adapter base | src/conclave/adapters/base.py |
ProviderAdapter protocol (build_request/parse_response + stream_request/parse_sse_event), SSEDelta, ProviderError, and redact() (error-string secret scrubber). |
| OpenAI-compat adapter | src/conclave/adapters/openai_compat.py |
OpenAICompatAdapter — openai/xai/perplexity + custom OpenAI-compatible endpoints. |
| Anthropic adapter | src/conclave/adapters/anthropic.py |
AnthropicAdapter — native /v1/messages, system-prompt hoist, required max_tokens. |
| Gemini adapter | src/conclave/adapters/gemini.py |
GeminiAdapter — native generateContent, OpenAI-role mapping, usageMetadata. |
| Registry | src/conclave/registry.py |
Friendly-name → model-id defaults; provider → env-var mapping; key presence logic (never values). |
| Config | src/conclave/config.py |
Loads/merges ~/.conclave/config.yml over defaults; resolves model ids and named/CSV councils; parses the endpoints: section (custom OpenAI-compatible providers). |
| Models | src/conclave/models.py |
Stable Pydantic contract, including EliteResult phase artifacts and backward-compatible CouncilResult.elite. |
| CLI | src/conclave/cli.py |
Six released conclave ask modes plus providers; Elite exits 1 unless decision readiness is ready, emits full JSON before failure, and rejects streaming. |
| Experimental evals | src/conclave/evals/ |
Versioned DSE-708 planning, strict replay, failure-inclusive running, blinding, scoring, and exploratory reports; no product-quality claim. |
| Eval CLI | src/conclave/eval_cli.py |
Offline plan/run/blind/report plus eval live; paid execution requires --execute, exact USD 10.00 approval, and an owner-only --checkpoint-seal-key-file. |
| Logging | src/conclave/logging.py |
Logger factory; stderr; verbosity via CONCLAVE_LOG_LEVEL (default WARNING). |
| File | Path | Covers |
|---|---|---|
| Council tests | tests/test_council.py |
Fan-out, partial failure, synthesis behavior. |
| Synthesizer tests | tests/test_synthesizer.py |
Pins the synthesizer/judge contract: default + configurable (arg/config/CLI --synthesizer) selection; observable degradation (unkeyed/failed → synthesis_error/verdict_error, never silent) for synthesize, debate, and the adversarial judge; versioned synthesis prompt (SYNTHESIS_PROMPT_VERSION + result.prompt_version) with prompt-text + version pins. |
| Modes tests | tests/test_modes.py |
Debate multi-round flow, mid-round drop-out, peer anonymization; adversarial proposer/critic/verdict, proposal/critic failure paths, no-key judge, sync wrappers. |
| Elite tests | tests/test_elite_mode.py |
Three-phase gates, anonymized prompts, partial failures, synthesis/verdict behavior, cache and public API. |
| Adapter tests | tests/test_adapters.py |
Per-adapter build_request + parse_response for openai-compat/anthropic/gemini: system-hoist, max_tokens, role mapping, usage parsing, empty/malformed/error-status raises. |
| Provider highway tests | tests/test_providers.py |
resolve_adapter (built-in prefixes, per-provider URLs, custom endpoints, unknown-prefix raise), end-to-end call_model, and redact() (bearer/sk-/env-var-value/x-api-key scrubbing; pre-redacted provider errors). |
| Registry/config tests | tests/test_registry_config.py |
Name resolution, key-presence logic, config merge. |
| CLI tests | tests/test_cli.py |
Typer CliRunner: exit-code contract (0 usable result and ready Elite / 1 zero-usable or non-ready Elite / 2 usage error), --json payload + exit code, human renderers per mode, providers table never prints secrets, aclose lifecycle. |
| Eval tests | tests/evals/ |
Frozen matrix, offline replay, live reservation/checkpoint/resume gates, blinding, scoring, reporting, and CLI assertions. |
| Transport tests | tests/test_transport.py |
post_json via httpx MockTransport: success/error-status/non-JSON fallback, timeout & connect/HTTP errors → TransportError (key never leaks), client reuse/pooling, aclose idempotency. |
| Streaming tests | tests/test_streaming.py |
Per-adapter SSE via MockTransport (openai-compat/anthropic/gemini): incremental chunks + assembled answer == concatenation == buffered result; mid-stream malformed-frame/connection-drop/non-2xx → error set with partial text preserved (never raises); key redaction in stream errors; buffered ask() never opens a stream; Council.ask_stream interleaving + terminal done shape; CLI --stream smoke + exit-code contract + debate rejection; --stream + cache one-shot replay. |
| Logging tests | tests/test_logging.py |
CONCLAVE_LOG_LEVEL resolution (default WARNING, case-insensitive, unknown → WARNING), factory contract, one-shot configuration. |
| Key-leak audit tests | tests/test_keyleak_audit.py |
Threat-model regression guards: no API key (bearer/sk-/env-var value/x-api-key) leaks via exception messages, __cause__ chains, repr, or transport debug logging; transport-logging guard is default-on and opt-out only via allow_transport_debug_logging. |
| Fixtures | tests/conftest.py |
Shared fixtures; mocks the httpx transport so the suite needs no network and no API keys. |
Run: pytest (config in pyproject.toml, asyncio_mode = "auto").
| File | Path | Purpose |
|---|---|---|
| Packaging | pyproject.toml |
hatchling build, deps (httpx, pydantic, rich, typer, pyyaml — no LLM SDK), dev extras, console script, pytest config. License: MIT. PyPI distribution name conclave-cli (the name conclave is an unrelated project); command + import stay conclave. |
| Release runbook | RELEASING.md |
Operator runbook: one-time PyPI OIDC Trusted-Publisher setup for conclave-cli, cut-a-release checklist (bump→tag→publish Release), post-release verification (Sigstore bundle, PEP 740 attestations), rollback/yank. |
| Changelog | CHANGELOG.md |
Keep-a-Changelog history through v1.2.0. |
| Dev lockfile | requirements-dev.lock |
Hash-pinned dev + runtime tree for reproducible installs/CI. Regenerate via uv pip compile --universal --generate-hashes --python-version 3.11 --extra dev pyproject.toml -o requirements-dev.lock. |
| License | LICENSE |
MIT License. Copyright (c) 2026 Ernest Provo. Matches the pyproject.toml license field. |
| Security policy | SECURITY.md |
BYO-keys vulnerability reporting policy: how to report, scope, and the key-handling guarantees consumers can rely on. |
| Contributing guide | CONTRIBUTING.md |
Dev setup, the BYO-keys contract for contributors, and the PR checklist (tests, ruff lint/format, coverage). |
| Code of conduct | CODE_OF_CONDUCT.md |
Contributor Covenant v2.1. |
| Project | Relationship |
|---|---|
| mcp-warden (sibling) | Imports conclave as a DEV-TIME dependency (design review, taxonomy labeling). NOT a runtime dependency — security findings need determinism; a stochastic council is the wrong tool for runtime adjudication. See PDD §10. |
| Date | Change |
|---|---|
| 2026-07-19 | Reconciled the decision-quality roadmap to released v1.2.0 and active H1 pilot preparation. Recorded the 12-cell paid correctness smoke, isolated three-provider connectivity check, synthetic-pack evidence boundary, and two-human-grader calibration plan. Decision quality and efficiency remain unvalidated. |
| 2026-07-18 | Released v1.2.0: Elite, constrained-choice vote, every-mode manifest hardening, the capped paid-exploratory evaluation runner, strict OpenAI-compatible verdict extraction, and org-repository release identities. The live lane remains correctness-only; no quality or efficiency claim. |
| 2026-07-18 | Implemented the sequential paid exploratory runner: default dry-run, exact USD 10.00 execute approval, reservation-before-call, one in-flight call, and no-repeat resume. The 24-task fixture remains offline/open-book and is not the paid smoke corpus; the smoke is correctness-only and not decision eligible. |
| 2026-07-17 | Added the experimental DSE-708 offline evaluation substrate and conclave eval artifact workflow; no live study or quality claim. |
| 2026-07-17 | Reframed the product roadmap around empirically proven decision quality: narrowed current claims to execution traceability, identified answer IDs as internal provenance rather than external evidence, gated Elite merge on H0 correctness, and added H1-H4 evidence, buyer, and outcome gates. |
| 2026-07-17 | Documented the implemented-but-unreleased Elite Decision Protocol: fixed three-success gates, initial/claim-audit/revision artifacts, existing final verdict, phased receipts, failure semantics, cost/latency tradeoff, and no streaming. |
| 2026-07-13 | Manifest-on-every-result invariant fix + doc reconciliation. ModelHarnessManifest now attaches on all five modes (was None for debate/adversarial/vote) via a single-site fix at Council._cached_run → new _ensure_manifest; regression tests in tests/test_manifest_all_modes.py. Docs corrected so the manifest-on-every-result claim is accurate and the vote mode is documented as shipped (CAC-09 / #3), not "absorbed" — across README, SYSTEM_CONTEXT_DIAGRAM.md, PDD (§1/§3/§4a/§8/§9/§12), and CHANGELOG.md ([Unreleased]). Verdict scope unchanged (still synthesize/ask-only; not layered on adversarial). |
| 2026-06-21 | v1.1 docs pivot — the auditable multi-model council. PDD reframed to the auditable-council wedge (new §4a: CouncilResult v2, deterministic position_cluster_ratio_v1 consensus, native + fallback structured output, the verdict-optional rule, the secret-free ModelHarnessManifest); vote mode marked absorbed by provider_votes across PDD §1/§4/§8/§9/§12; demand-gated v1.2 "Operable Council" roadmap. System Context diagram gains the verdict pipeline + manifest (mermaid re-validated). README gains an "Auditable verdict" section (CLI panel + library example). CHANGELOG.md [1.1.0] added. v0.x mode-detail prose archived to docs/archive/pdd-v0.x-modes-detail.md to keep the PDD under 500 lines. New modules documented: verdict.py, agreement.py, verdict_synthesis.py, manifest.py. |
| 2026-06-14 | v1.0.0 release. Version bump 0.3.0 → 1.0.0; new CHANGELOG.md (Keep-a-Changelog). Integrates the three v1.0 PRs below. |
| 2026-06-14 | v1.0 distribution + release-engineering (PR-A): PyPI distribution name → conclave-cli (command + import stay conclave); OIDC Trusted-Publishing release workflow (.github/workflows/release.yml, SHA-pinned, PEP 740 attestations, Sigstore keyless signing, inert until a Release fires + publisher configured); pip-audit fail-closed CI job in test.yml; hash-pinned requirements-dev.lock; RELEASING.md operator runbook. |
| 2026-06-14 | Key-leak audit + threat model (v1.0 readiness): cause-chain leak fix (httpx exception dropped from TransportError.__cause__); transport-logging guard default-on in Council.__init__ (opt-out via allow_transport_debug_logging); SECURITY.md threat model; .gitleaks.toml + tests/test_keyleak_audit.py. |
| 2026-06-14 | Documented + tested synthesizer behavior (v1.0 readiness must-do #5): README "Synthesizer behavior" section (selection precedence, observable degradation, versioned prompt); synthesis prompt set now versioned via conclave.prompts.SYNTHESIS_PROMPT_VERSION, stamped onto every CouncilResult.prompt_version; confirmed (not silent) degradation across synthesize/debate/adversarial-judge paths; new tests/test_synthesizer.py (21 tests). No non-synthesis behavior changed. |
| 2026-06-09 | Roadmap features shipped: adversarial proposer resilience (#9), optional result cache (#6), debate convergence early-stop (#4), 4 first-class providers groq/deepseek/mistral/together (#5), streaming for synthesize/raw (#7); tests 121→191. #8 local-server-mode spike evaluated (no-go on HTTP). Doc sync: System Context diagram now shows all 9 providers; PDD §12 resolved questions archived to docs/archive/pdd-resolved-questions-2026-06-09.md (PDD back under 500 lines); config.example.yml stale "LiteLLM" comment fixed. |
| 2026-06-08 | v0.3.0 version bump; CI foundation (Actions matrix, ruff, coverage floor, gitleaks, branch protection); redact() custom-endpoint key-leak fix (#14); status_error consolidation + conditional temperature (#16/#22); provider-metadata single-source + import-time drift guard + config memoization (#19/#15); CLI exit-code contract + httpx client lifecycle (#17/#20); transport/cli/logging test backfill (#18); public release + community files. |
| 2026-06-08 | PDD §11 repositioned vs. new direct peers (llm-council-core, the-llm-council); §12 Q1/Q3/Q4/Q5 resolved. Index Tests table updated for the PR #2 split (test_adapters.py, test_providers.py). |
| 2026-06-07 | v0.3 provider-highway refactor (LiteLLM removed → owned httpx transport + adapter registry); 3-core docs + PDD authored. |
- Exactly 3 core docs (README, System Context Diagram, this index); the PDD is the authority spec layered on top.
- Each doc stays under 500 lines; overflow is archived to dated/topic files and linked from here.
- Prefer editing existing docs over creating new ones. Check this index before adding any new document.
- The PDD is canonical; on any conflict, defer to
docs/PRODUCT_DESIGN_DOCUMENT.md.