fix(manifest): attach ModelHarnessManifest on every mode's result#50
Merged
Conversation
The auditable ModelHarnessManifest was documented as first-class on every
CouncilResult ("rides on every result, not behind a debug flag"), but only the
ask/synthesize path built one. debate, adversarial, and vote constructed their
result directly in modes.py and returned with manifest=None, so the invariant
silently drifted for 3 of 5 modes and no test guarded it.
Make manifest attachment a true invariant with a single-site fix: all five
modes funnel through Council._cached_run, which now calls a new
Council._ensure_manifest on every returned result. It is a no-op for the
synthesize/raw path (which builds its own richer manifest in _ask_uncached so
it can populate verdict-provenance slots and re-stamp secret-safety), and fills
the manifest for debate/adversarial/vote from the resolved membership +
collected answers. This also covers the zero-members early return and the
cache-hit path, so no result can escape without a secret-safety-VERIFIED
manifest.
Verdict scope is deliberately unchanged: the clustering verdict still runs on
synthesize/ask only. It is not forced onto adversarial (which already emits a
judge verdict, so it would double-count the same adjudication) and is deferred
on debate/vote; rationale recorded in PDD 4a.
Add regression tests (tests/test_manifest_all_modes.py) asserting
manifest is not None and secret_safety == verified_no_secrets for debate,
adversarial, and vote, including the zero-members and cache-hit paths.
Reconcile docs so (a) the manifest-on-every-result claim is now true and (b)
the constrained-choice vote mode is documented as shipped (CAC-09 / #3), not
"absorbed by provider_votes": README, SYSTEM_CONTEXT_DIAGRAM, PDD,
DOCUMENTATION_INDEX, and CHANGELOG.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The auditable
ModelHarnessManifestis documented as first-class on everyCouncilResult("rides on every result, not behind a debug flag"), but in practice only theask/synthesize path built one.debate,adversarial, andvoteconstructed their result directly inmodes.pyand returned withmanifest = None, so the invariant silently drifted for 3 of 5 modes and no test guarded it.The fix — a single-site invariant
All five modes funnel through the one chokepoint
Council._cached_run. This PR addsCouncil._ensure_manifest, called there on every returned result:_ask_uncached, so it can populate verdict-provenance slots and re-stamp secret-safety).debate/adversarial/votefrom the resolved membership + collected answers.secret_safety = verified_no_secretsmanifest.Membership is re-resolved via the existing
_available_members()(the same resolution_cache_keyalready performs) rather than threaded through every mode's return — soproviders_called/model_idsreflect the full membership even for debate rounds where a member later dropped out, while per-answer receipts come fromresult.answers.Verdict scope (deliberate)
Manifest everywhere; verdict scope unchanged. The clustering verdict still runs on
synthesize/askonly. It is not forced ontoadversarial(which already emits a judge verdict, so it would double-count the same adjudication) and is deferred ondebate/vote. Rationale recorded in PDD §4a.Tests
New
tests/test_manifest_all_modes.pypins the invariant per mode:manifest is not Noneandsecret_safety == verified_no_secretsfordebate,adversarial, andvote, including the zero-members and cache-hit paths. Full suite: 499 passed, coverage 92.3% (gate 75%),ruff check+ruff format --checkclean.Doc reconciliation
SYSTEM_CONTEXT_DIAGRAM.md, PDD §4a).votemode is documented as shipped (CAC-09 / vote mode — structured majority with reported split #3), not "absorbed byprovider_votes" — the two are complementary (a fixed ballot vs. clustered free-form stances). Updated across README,SYSTEM_CONTEXT_DIAGRAM.md, PDD (§1/§3/§4a/§8/§9/§12),DOCUMENTATION_INDEX.md, andCHANGELOG.md. All core docs remain under 500 lines.🤖 Generated with Claude Code
https://claude.ai/code/session_01QofWZ4MivqEKPevjSYuwdk