Require auditable plan coverage for signed human requirements

[wwind123]

Problem

A valid signed human requirement on llm-dialectic#336 requested that a Grafana dashboard display the new verification attribution. The subsequent plan and reviews emitted <!-- HUMAN_REQUIREMENTS_RESOLVED -->, but the approved plan covered only an admin.html operations view and did not mention Grafana, dashboard provisioning, or any integration artifact.

The current flow accepts a reviewer-provided HUMAN_REQUIREMENTS_RESOLVED marker as sufficient. It does not require an explicit per-requirement disposition or verify that the approved plan covers the signed requirement.

Expected behavior

For every surfaced signed human requirement during plan-first review:

• The coder's structured plan/revision must provide a per-requirement disposition keyed by the generated Requirement N ID: addressed, blocked, or not-applicable, with a concise evidence note.
• A reviewer may emit HUMAN_REQUIREMENTS_RESOLVED only when each requirement has an explicit addressed or explicitly accepted not-applicable/blocked disposition and the current canonical plan contains concrete coverage.
• The orchestrator must reject or repair an approval marker that lacks the required structured dispositions, rather than treating it as proof of satisfaction.
• Prompt wording should distinguish a named external deliverable (for example, Grafana) from a superficially similar local UI (for example, admin.html) and require the plan to name the integration artifact when it is requested.

Regression coverage

Add a plan-first fixture with a signed requirement for a named external integration and a plan that implements only a similarly purposed local UI. Assert that approval is blocked until the revised plan explicitly covers the requested integration or records an explicit, reviewer-visible reason it cannot be satisfied.

[wwind123]

Plan

Make signed human requirements auditable in plan-first flows by requiring per-requirement dispositions, preserving them in the canonical plan, and refusing reviewer approval unless those dispositions and concrete plan coverage are validated.

Plan steps

1. Extend the structured plan protocol in src/coding_review_agent_loop/protocol.py with a validated per-requirement disposition model keyed by each surfaced Requirement N ID. Require every surfaced requirement in both plan_state and plan_revision to have exactly one addressed, blocked, or not-applicable disposition plus a non-empty evidence note; reject unknown, duplicate, missing, or malformed entries while retaining the existing omitted-context/direct-discussion behavior.
2. Update canonical/public plan rendering and plan-first orchestration in src/coding_review_agent_loop/comment_rendering.py and src/coding_review_agent_loop/orchestrator.py so the current canonical plan carries the requirement dispositions and evidence. Gate HUMAN_REQUIREMENTS_RESOLVED on reviewer-visible, valid dispositions and concrete canonical-plan coverage; convert invalid or unsupported approval markers into a blocking/revision path (including repair validation) instead of accepting the marker alone.
3. Revise plan creation, revision, and reviewer prompts in src/coding_review_agent_loop/prompts.py (and the skill-runner prompt/validation adapters under helpers/ that invoke the same plan-first protocol) to require the structured dispositions, tell reviewers how to evaluate them, and explicitly distinguish named external integrations from similarly purposed local UI. Require the plan to name the requested integration artifact (for example Grafana dashboard/provisioning) or record a reviewer-visible blocked/not-applicable reason.
4. Expand protocol, prompt, rendering, orchestrator, and skill-helper tests. Add the requested plan-first regression fixture: a signed requirement naming Grafana, an initial plan that only names an admin.html-style local view, and an approval marker. Assert it is blocked/repaired until a revised canonical plan explicitly covers Grafana provisioning/integration or contains a valid, accepted blocked/not-applicable disposition; also cover complete valid dispositions, missing/duplicate/unknown IDs, empty evidence, rejected marker repairs, and no-human-requirement compatibility. Run the focused test modules and the full python -m pytest suite.

-- OpenAI Codex: gpt-5.6-terra

[wwind123]

Review verdict: Blocking

The plan correctly targets the real gap (plan_state/plan_revision have no structured human_requirements field, unlike coder_followup, and the orchestrator today accepts HUMAN_REQUIREMENTS_RESOLVED as a bare marker check via human_requirements_resolved()/_plan_revision_missing_human_acknowledgement without verifying any per-requirement coverage), and the file/architecture targets (protocol.py, comment_rendering.py, orchestrator.py, prompts.py, helpers/skill_runner.py) are accurate. But it leaves unresolved exactly how 'concrete canonical-plan coverage' is validated, which is the crux of the issue and of the requested regression test.

Blocking plan issues

• Define the concrete validation mechanism for 'concrete canonical-plan coverage' before implementation starts, and reflect it in step 1/2's wording. As drafted, step 1 only requires a disposition value (addressed/blocked/not-applicable) plus a non-empty evidence note, and step 2 only requires that dispositions exist and are reviewer-visible. Neither is sufficient to prevent the original incident: a coder could set disposition=addressed with evidence text like 'Implemented via admin.html' for the Grafana requirement, which satisfies 'non-empty evidence note' and 'disposition present' while still not covering the named artifact -- exactly reproducing issue External coder plan_state (round 1) posted as raw JSON instead of rendered markdown #336. Since the issue's own text places 'concrete coverage' verification on the reviewer ('A reviewer may emit HUMAN_REQUIREMENTS_RESOLVED only when ... the current canonical plan contains concrete coverage'), confirm explicitly whether coverage validation is (a) purely reviewer-judgment, enforced only through updated prompt wording (in which case the plan's summary/step-2 phrasing 'refusing reviewer approval unless ... concrete plan coverage are validated' should be reworded to avoid implying orchestrator-side semantic validation that won't actually exist), or (b) partially mechanical (e.g. requiring 'addressed' evidence notes to reference an artifact keyword extracted from the signed requirement text). This decision changes the shape of the regression fixture: with (a), the fixture must script the reviewer to correctly block round 1 and only approve after the coder's revision names Grafana (testing that the structural/prompt machinery threads through correctly), not that the system catches a reviewer who is fooled by an admin.html-only plan. With (b), the fixture can assert the orchestrator itself rejects/repairs an admin.html-only 'addressed' disposition even when the reviewer wrongly emits the marker. Please state which behavior the regression fixture in step 4 is expected to exercise, since 'Assert that approval is blocked until the revised plan explicitly covers the requested integration' (from the issue) is ambiguous between these two readings and materially changes both the coder's protocol.py validation function and the test.

-- Anthropic Claude: claude-sonnet-5

[wwind123]

Review verdict: Blocking

Plan is conceptually sound but misses schema updates for the reviewer's approval marker and implies impossible semantic checks in the orchestrator.

Blocking plan issues

• The issue requires the orchestrator to reject or repair an 'approval marker that lacks the required structured dispositions'. Since the coder's plan is strictly validated for dispositions before reaching the reviewer, this implies the reviewer's approval response (plan_review) must also contain structured dispositions. Update step 1 and 2 to extend the plan_review schema to require these dispositions, and have the orchestrator validate their presence in the reviewer's payload.
• In step 2, clarify that orchestrator.py validates the structural presence and validity of dispositions. The orchestrator cannot natively perform semantic checks for 'concrete canonical-plan coverage'; this is the responsibility of the LLM reviewer as instructed by the prompts.

-- Google Antigravity: Gemini 3.1 Pro (High)

[wwind123]

Revised plan

Revise the plan-first protocol so coder plans and reviewer approvals both carry complete, auditable per-requirement dispositions. The orchestrator will mechanically validate those attestations and marker coupling; whether an addressed plan concretely covers a named deliverable remains an explicit reviewer judgment, enforced by the review prompt and regression flow rather than unreliable keyword matching.

Prior plan item dispositions

• [item-1] Blocking issue from Anthropic Claude, round 1: Define the concrete validation mechanism for 'concrete canonical-plan coverage' before implementa... -> resolved: The plan now chooses reviewer judgment for semantic canonical-plan coverage, limits orchestrator enforcement to structural validation, and scripts the regression reviewer to block the admin.html-only plan before approving a Grafana-specific revision.
• [item-2] Blocking issue from Google Antigravity, round 1: Extend the plan_review schema to require reviewer dispositions and have the orchestrator validate... -> resolved: The plan now requires a complete per-requirement disposition collection in structured plan_review responses and validates it before an approval marker can be accepted or repaired.
• [item-3] Blocking issue from Google Antigravity, round 1: Clarify that the orchestrator validates structural disposition validity while the reviewer judges... -> resolved: The plan explicitly separates structural orchestrator validation from the LLM reviewer's semantic determination that the canonical plan concretely covers the requirement.

Plan steps

1. Extend src/coding_review_agent_loop/protocol.py with one shared, typed human_requirement_dispositions payload for plan_state, plan_revision, and plan_review. Each entry will contain the generated Requirement N ID, one of addressed, blocked, or not-applicable, and a non-empty evidence note. Given the surfaced signed-requirement IDs for a planning round, validate exact one-to-one coverage: reject missing, duplicate, unknown, malformed, or empty-evidence entries; preserve the no-signed-requirement compatibility path by requiring an empty collection there. Carry the parsed collection through the structured plan and review dataclasses, while retaining existing prior-plan-item dispositions as a separate ledger.
2. Update src/coding_review_agent_loop/orchestrator.py and src/coding_review_agent_loop/comment_rendering.py to retain and render the coder's dispositions in the canonical plan and the reviewer's dispositions in the public review record, including resume metadata and canonical-plan subject inputs. Gate HUMAN_REQUIREMENTS_RESOLVED mechanically: an approved review may retain or recover the marker only when both the current canonical coder plan and that reviewer's structured payload have complete valid dispositions for every surfaced ID. Missing or invalid reviewer dispositions, including a marker-only approval, enter the existing structured-repair/blocking path. Do not implement artifact-keyword extraction or claim semantic verification by the orchestrator: the reviewer determines whether addressed evidence and the canonical plan actually cover the requested deliverable, and explicitly accepts any blocked or not-applicable reason before marking resolved.
3. Revise plan-creation, revision, repair, and plan-review instructions in src/coding_review_agent_loop/prompts.py, plus the matching plan-first protocol adapters in helpers/ if present in this checkout. Provide the new JSON examples and require coder and reviewer entries for every Requirement N. Direct reviewers to compare each disposition and evidence note against the current canonical plan, return blocking when an addressed claim lacks concrete plan coverage, and only emit the marker after the comparison succeeds. Explicitly state that a named external integration is distinct from a similarly purposed local UI: a Grafana request requires plan coverage for the Grafana dashboard/provisioning or other named integration artifact, not merely an admin.html view; an unsatisfiable request must have a visible blocked/not-applicable reason that the reviewer accepts.
4. Extend the relevant protocol, rendering, prompt, and plan-first orchestrator tests (including the existing structured fixtures in tests/) for complete valid coder/reviewer dispositions, no-requirement compatibility, missing/duplicate/unknown IDs, invalid status, empty evidence, marker-only approval repair/rejection, rendering/canonical-state persistence, and resume behavior. Add the issue regression fixture with a signed Grafana requirement: the initial canonical plan and its addressed disposition cite only an admin.html-style UI; the scripted reviewer returns blocking because it does not cover Grafana; a revised plan explicitly names the Grafana dashboard/provisioning artifact (or has a reviewer-accepted blocked/not-applicable reason) and receives a complete approving reviewer disposition plus the marker. This tests the intended prompt-and-attestation workflow, not a nonexistent orchestrator semantic keyword check. Run the focused modules and python -m pytest after confirming the assigned checkout with pwd and git status --branch --short.

-- OpenAI Codex: gpt-5.6-terra (high)

[wwind123]

Review verdict: Approved

Verified against the current codebase (protocol.py dataclasses/marker regexes, orchestrator.py plan-review and PR-review gating call sites, comment_rendering.py's freeform Human requirements block, and helpers/). The plan correctly identifies that plan-first flow today carries human-requirement acknowledgement via a freeform '### Human requirements' section plus a marker, parsed loosely by regex, with no per-requirement status/evidence and no orchestrator-side structural check. Moving to a shared typed human_requirement_dispositions payload on plan_state/plan_revision/plan_review, mechanically validated for one-to-one ID coverage and non-empty evidence, correctly resolves that gap. Keeping semantic 'does this plan concretely cover the named deliverable' judgment with the LLM reviewer (rather than orchestrator keyword matching) is the right call — the issue's own Grafana-vs-admin.html example requires semantic judgment that no reliable mechanical check could substitute for, and the plan enforces it via prompt wording plus a scripted regression reviewer. All four issue requirements (coder per-requirement disposition, reviewer-gated marker emission, orchestrator reject/repair of unsupported markers, and prompt wording distinguishing named external deliverables from similar local UI) are concretely covered, and the regression fixture matches the issue's exact ask. No implementation is deferred to future phases.

Future follow-ups

• The PR-review (post-plan-approval implementation) flow still gates HUMAN_REQUIREMENTS_RESOLVED only on marker presence via StructuredCoderFollowup's addressed_ids, without per-requirement addressed/blocked/not-applicable disposition and evidence. Consider extending the same auditable disposition mechanism to that flow in a later issue, since a requirement could still slip through during implementation follow-up even after this plan-first fix lands.

Prior unresolved plan item dispositions

• [item-1] Blocking issue from Anthropic Claude: claude-sonnet-5, round 1: Define the concrete validation mechanism for 'concrete canonical-plan coverage' before implementa... -> resolved: Plan defines the validation mechanism concretely: orchestrator performs structural validation only (one-to-one ID coverage, valid status enum, non-empty evidence note), while 'concrete canonical-plan coverage' remains an explicit reviewer judgment enforced through revised prompt wording and a scripted regression reviewer that blocks an admin.html-only plan against a Grafana requirement. This is the right boundary — no reliable mechanical/keyword check could make that judgment.
• [item-2] Blocking issue from Google Antigravity: Gemini 3.1 Pro (High), round 1: Extend the plan_review schema to require reviewer dispositions and have the orchestrator validate... -> resolved: Step 1 adds a shared, typed human_requirement_dispositions payload to plan_state/plan_revision/plan_review structured schemas with exact one-to-one ID coverage validation; step 2 has the orchestrator gate HUMAN_REQUIREMENTS_RESOLVED on completeness of both the coder's and reviewer's structured dispositions, routing incomplete/marker-only approvals into the existing repair/blocking path. Verified against the current StructuredHumanRequirementsPayload/StructuredPlanReview dataclasses and the existing _run_structured_repair gating code at both plan-review call sites in orchestrator.py, which today only check marker presence via human_requirements_resolved(), not per-requirement structure.
• [item-3] Blocking issue from Google Antigravity: Gemini 3.1 Pro (High), round 1: Clarify that the orchestrator validates structural disposition validity while the reviewer judges... -> resolved: Step 2 explicitly separates orchestrator-side structural validation (disposition completeness/shape) from reviewer-side semantic judgment (does the disposition's evidence and the canonical plan actually cover the requirement), and step 3 directs prompt wording accordingly. This matches the issue's own framing, which asks for auditable structure, not automated semantic verification.

-- Anthropic Claude: claude-sonnet-5

[wwind123]

Review verdict: Approved

The plan looks solid and covers all aspects of the issue. It properly extends the protocol, updates the orchestrator to perform structural validation of dispositions, and refines prompts so reviewers handle the semantic coverage judgment.

Prior unresolved plan item dispositions

• [item-1] Blocking issue from Anthropic Claude: claude-sonnet-5, round 1: Define the concrete validation mechanism for 'concrete canonical-plan coverage' before implementa... -> resolved: The plan clarifies that semantic reviewer judgment replaces keyword matching for canonical-plan coverage.
• [item-2] Blocking issue from Google Antigravity: Gemini 3.1 Pro (High), round 1: Extend the plan_review schema to require reviewer dispositions and have the orchestrator validate... -> resolved: The protocol schema is updated to include human_requirement_dispositions and the orchestrator validates their presence.
• [item-3] Blocking issue from Google Antigravity: Gemini 3.1 Pro (High), round 1: Clarify that the orchestrator validates structural disposition validity while the reviewer judges... -> resolved: The plan explicitly separates structural orchestrator validation from the LLM reviewer's semantic determination.

-- Google Antigravity: Gemini 3.1 Pro (High)

[wwind123]

Planning complete for issue #554.

Outcome: implement

Approved plan:

Revise the plan-first protocol so coder plans and reviewer approvals both carry complete, auditable per-requirement dispositions. The orchestrator will mechanically validate those attestations and marker coupling; whether an addressed plan concretely covers a named deliverable remains an explicit reviewer judgment, enforced by the review prompt and regression flow rather than unreliable keyword matching.

Prior plan item dispositions

• [item-1] Blocking issue from Anthropic Claude, round 1: Define the concrete validation mechanism for 'concrete canonical-plan coverage' before implementa... -> resolved: The plan now chooses reviewer judgment for semantic canonical-plan coverage, limits orchestrator enforcement to structural validation, and scripts the regression reviewer to block the admin.html-only plan before approving a Grafana-specific revision.
• [item-2] Blocking issue from Google Antigravity, round 1: Extend the plan_review schema to require reviewer dispositions and have the orchestrator validate... -> resolved: The plan now requires a complete per-requirement disposition collection in structured plan_review responses and validates it before an approval marker can be accepted or repaired.
• [item-3] Blocking issue from Google Antigravity, round 1: Clarify that the orchestrator validates structural disposition validity while the reviewer judges... -> resolved: The plan explicitly separates structural orchestrator validation from the LLM reviewer's semantic determination that the canonical plan concretely covers the requirement.

Plan steps

1. Extend src/coding_review_agent_loop/protocol.py with one shared, typed human_requirement_dispositions payload for plan_state, plan_revision, and plan_review. Each entry will contain the generated Requirement N ID, one of addressed, blocked, or not-applicable, and a non-empty evidence note. Given the surfaced signed-requirement IDs for a planning round, validate exact one-to-one coverage: reject missing, duplicate, unknown, malformed, or empty-evidence entries; preserve the no-signed-requirement compatibility path by requiring an empty collection there. Carry the parsed collection through the structured plan and review dataclasses, while retaining existing prior-plan-item dispositions as a separate ledger.
2. Update src/coding_review_agent_loop/orchestrator.py and src/coding_review_agent_loop/comment_rendering.py to retain and render the coder's dispositions in the canonical plan and the reviewer's dispositions in the public review record, including resume metadata and canonical-plan subject inputs. Gate HUMAN_REQUIREMENTS_RESOLVED mechanically: an approved review may retain or recover the marker only when both the current canonical coder plan and that reviewer's structured payload have complete valid dispositions for every surfaced ID. Missing or invalid reviewer dispositions, including a marker-only approval, enter the existing structured-repair/blocking path. Do not implement artifact-keyword extraction or claim semantic verification by the orchestrator: the reviewer determines whether addressed evidence and the canonical plan actually cover the requested deliverable, and explicitly accepts any blocked or not-applicable reason before marking resolved.
3. Revise plan-creation, revision, repair, and plan-review instructions in src/coding_review_agent_loop/prompts.py, plus the matching plan-first protocol adapters in helpers/ if present in this checkout. Provide the new JSON examples and require coder and reviewer entries for every Requirement N. Direct reviewers to compare each disposition and evidence note against the current canonical plan, return blocking when an addressed claim lacks concrete plan coverage, and only emit the marker after the comparison succeeds. Explicitly state that a named external integration is distinct from a similarly purposed local UI: a Grafana request requires plan coverage for the Grafana dashboard/provisioning or other named integration artifact, not merely an admin.html view; an unsatisfiable request must have a visible blocked/not-applicable reason that the reviewer accepts.
4. Extend the relevant protocol, rendering, prompt, and plan-first orchestrator tests (including the existing structured fixtures in tests/) for complete valid coder/reviewer dispositions, no-requirement compatibility, missing/duplicate/unknown IDs, invalid status, empty evidence, marker-only approval repair/rejection, rendering/canonical-state persistence, and resume behavior. Add the issue regression fixture with a signed Grafana requirement: the initial canonical plan and its addressed disposition cite only an admin.html-style UI; the scripted reviewer returns blocking because it does not cover Grafana; a revised plan explicitly names the Grafana dashboard/provisioning artifact (or has a reviewer-accepted blocked/not-applicable reason) and receives a complete approving reviewer disposition plus the marker. This tests the intended prompt-and-attestation workflow, not a nonexistent orchestrator semantic keyword check. Run the focused modules and python -m pytest after confirming the assigned checkout with pwd and git status --branch --short.

Filed future follow-up issues:

• Follow up future plan-review note: The PR-review (post-plan-approval implementation) flow still gates HUMAN_REQUIREMENTS #559

Reconciliation: 1 filed, 0 deduplicated, 0 skipped by cap.

These planning-stage future follow-ups are future work outside the current implementation scope. They are not carried into PR review and their plan item IDs are not PR prior review items.

-- coding-review-agent-loop

[wwind123]

Approved plan for issue #554 handed off to PR #560 for one-shot implementation.

Mode: implement-one-shot
Plan hash: c2556c46c9186469
Plan subject: c2556c46c918646923dbf47fc1f8733bb4fe77ac8291d60739733f1c026fcf56

Parent reruns will resume the PR review loop for this PR instead of re-implementing.

-- coding-review-agent-loop