Integrate structured review output, subagent contracts, and retro accumulation from plugin research

[christso]

Objective

Integrate high-value patterns identified from researching affaan-m/everything-claude-code and openai/codex-plugin-cc into HiveSpec's delivery lifecycle skills, with evals proving each change improves agent behavior.

Full research: agentevals-research HIVESPEC_INTEGRATION.md

Design Latitude

• The markdown snippets below are directional, not prescriptive — reshape to fit each file's existing voice and structure
• Each change is a skill content update only (edit existing .md files) — no code, no infrastructure, no new dependencies
• Evals go in the hivespec-evals repo, following AgentV EVAL.yaml format
• Change feat(skills): add hs-retro phase for session retrospectives #5 (foreground/background dispatch) is out of scope for this issue — it's an optimization that can be a follow-up

Changes

1. Structured review output schema

Files to edit:

• skills/hs-implement/references/spec-reviewer-prompt.md
• skills/hs-implement/references/code-quality-reviewer-prompt.md

What to add: A required output format section instructing reviewers to return findings as a structured table:

| # | Severity | File:Line | Finding | Recommendation |
|---|----------|-----------|---------|----------------|
| 1 | high | src/foo.ts:42 | Missing null check on user input | Add guard clause |

**Verdict**: approve | needs-attention
**Next steps**: [specific actions if needs-attention]

Why: Currently reviewers return prose, making it hard for the parent agent to systematically track whether all findings were addressed before proceeding to hs-ship. Structured output makes findings enumerable and trackable.

Also update: skills/hs-verify/SKILL.md Step 4 — add a note that the parent agent should confirm every "needs-attention" finding was resolved before moving to Step 5.

2. Subagent output contract

Files to edit:

• skills/hs-implement/references/implementer-prompt.md

What to add: A required output contract section:

## Output Contract

Every response must end with:

**Status**: DONE | DONE_WITH_CONCERNS | NEEDS_CONTEXT | BLOCKED
**Files changed**:
- `path/to/file.ts` — one-line description
**Tests added/modified**:
- `path/to/test.ts` — what it covers
**Unresolved concerns**:
- Any shared types/interfaces modified that other subagents might also touch
- Anything noticed but out of scope for this task

Also update: skills/hs-implement/SKILL.md Subagent Review Protocol — add an integration safety check: after collecting outputs from parallel subagents, verify no two subagents modified the same shared type/interface. If they did, reconcile before committing.

Why: Currently hs-implement defines subagent status codes (DONE, DONE_WITH_CONCERNS, etc.) but doesn't require structured output listing files and concerns. The parent agent has to infer what changed from prose, making integration conflicts between parallel subagents invisible until tests break.

3. Confidence-based retro accumulation

Files to edit:

• skills/hs-retro/SKILL.md

What to add: A new Step 3b "Check recurrence" between current Step 3 (Classify) and Step 4 (Apply fixes):

### Step 3b: Check recurrence

Before applying a fix, assess whether this is a one-off or a pattern:

1. **First occurrence**: Apply the fix only if it's clearly a systematic gap (not a user preference or one-off situation). Otherwise, note it as a candidate for future confirmation.
2. **Recurs across 2+ sessions**: Apply with high confidence — this is a systematic gap.
3. **Recurs across 3+ sessions AND across different repos**: Consider promoting the fix from project-scoped to HiveSpec core skills.

Single-session over-fitting is how skills accumulate noise. Require recurrence before structural changes.

Why: Currently hs-retro treats every session independently — a one-time user preference gets the same treatment as a pattern that appears in every session. This leads to over-fitting skills to single-session noise.

4. Reliability check for non-deterministic features

Files to edit:

• skills/hs-verify/SKILL.md

What to add: A new Step 2b after Step 2 (E2E red/green protocol):

### Step 2b: Reliability check (non-deterministic features only)

If the feature involves LLM calls, external APIs, or historically flaky tests:

1. Run the relevant test suite **3 times** (pass^3 protocol)
2. All 3 must pass — any failure means the feature is unreliable, not "flaky"
3. For LLM-dependent features: use at least 2 different representative inputs

Skip this step for purely deterministic code paths (config parsing, data transformations, etc.) — one green run is sufficient.

Why: A single passing test run doesn't prove reliability for non-deterministic features. pass^3 catches intermittent failures that a single run misses, while the deterministic carve-out avoids wasting time re-running stable code.

---

Evals

Each change needs A/B evals in hivespec-evals proving the updated skill produces measurably better agent behavior than the current skill. Use AgentV EVAL.yaml format.

Eval 1: Structured review output

Test ID	What It Checks	Key Assertions
review-output-is-structured	Reviewer returns tabular findings with severity, file:line, verdict	contains: "| Severity |", contains-any: ["approve", "needs-attention"], rubrics: findings have file refs + are actionable
review-findings-trackable	Parent agent can enumerate findings and check resolution status per finding	rubrics: all findings enumerated, each gets resolved/unresolved status

Eval 2: Subagent output contract

Test ID	What It Checks	Key Assertions
subagent-returns-contract	Implementer subagent output contains status, files changed, tests added	contains-all: ["Status:", "Files changed:", "Tests added"], rubrics: status is valid enum, concerns surfaced
integration-conflict-detected	Parent agent catches when two subagents modified the same shared type	rubrics: conflict identified, resolution proposed

Eval 3: Reliability check

Test ID	What It Checks	Key Assertions
nondeterministic-gets-multiple-runs	Agent runs tests 3x for LLM-dependent features	rubrics: multiple runs, recognizes nondeterminism
deterministic-gets-single-run	Agent runs tests once for deterministic features (no wasted reruns)	rubrics: single run sufficient, no unnecessary reruns

Eval 4: Retro accumulation

Test ID	What It Checks	Key Assertions
first-occurrence-marked-candidate	First-time intervention is noted, not immediately committed as a skill edit	rubrics: candidate marked, no premature skill edit
recurring-pattern-applied	Same intervention across 3 transcripts → fix applied with high confidence	rubrics: recurrence recognized, fix applied citing recurrence

Running evals

# Baseline (current skills)
agentv eval --file evals/hivespec/structured-review.eval.yaml --target claude

# Candidate (updated skills in workspace)
agentv eval --file evals/hivespec/structured-review.eval.yaml --target claude

# Compare
agentv compare baseline.jsonl candidate.jsonl

Key metrics: pass_rate delta, token_usage delta, per-rubric score improvements.

---

Acceptance Signals

• Reviewer prompts produce structured, tabular findings with verdict (eval 1 passes)
• Implementer prompt specifies output contract; parent catches integration conflicts (eval 2 passes)
• hs-retro distinguishes first-occurrence from recurring patterns (eval 4 passes)
• hs-verify applies pass^3 for non-deterministic features, single run for deterministic (eval 3 passes)
• All evals show pass_rate improvement over baseline (current skills)
• All changes are skill content updates only (no code, no infrastructure)

Non-Goals

• Hook profile system (minimal/standard/strict) — HiveSpec's Hard Gates are unconditional by design
• Multi-model routing — belongs in target repo's CLAUDE.md, not HiveSpec
• Session persistence — Claude Code handles this
• Foreground/background dispatch heuristic — follow-up issue, not this one
• DAG decomposition — hs-implement's parallel dispatch is sufficient
• XML-structured prompts — Markdown skills are simpler and work

[christso]

Eval Strategy: Proving the Changes Improve Developer Productivity

Each integration should have a before/after eval demonstrating measurable improvement. The eval approach uses AgentV's existing evaluator types — no new infrastructure needed.

Eval Design Principles

1. A/B comparison: Run the same scenario with current skills (baseline) vs updated skills (candidate). Measure delta.
2. Observable outcomes, not vibes: Each assertion must cite evidence from the agent's output or tool trajectory.
3. Deterministic where possible: Use code-grader and contains for structural checks; rubrics only for semantic quality.
4. Realistic tasks: Evals should use real-world scenarios an agent would encounter during HiveSpec delivery.

---

Eval 1: Structured Review Output (Change #1)

What to prove: Reviewer subagents produce machine-parseable findings, enabling the parent agent to systematically track whether all issues were addressed.

tests:
  - id: review-output-is-structured
    criteria: |
      Code reviewer produces structured tabular findings with severity,
      file:line, and a verdict — not just prose feedback.
    input:
      - role: user
        content:
          - type: file
            value: workspace/diff.patch
          - type: file
            value: workspace/.claude/skills/hs-verify/SKILL.md
          - type: file
            value: workspace/.claude/skills/hs-implement/references/code-quality-reviewer-prompt.md
          - type: text
            value: |
              You are a code quality reviewer. Review this diff following
              the reviewer prompt instructions. Produce your findings in
              the format specified by the prompt.
    assertions:
      - type: contains
        value: "| Severity |"  # Table header present
      - type: contains-any
        value: ["approve", "needs-attention"]  # Verdict present
      - type: rubrics
        criteria:
          - id: findings-have-file-refs
            outcome: "Every finding references a specific file and line number"
            weight: 1.0
            required: true
          - id: findings-are-actionable
            outcome: "Each finding includes a concrete recommendation, not just a complaint"
            weight: 1.0

  - id: review-findings-trackable
    criteria: |
      Parent agent can determine from review output exactly how many
      findings need addressing and whether each was resolved.
    input:
      - role: user
        content:
          - type: file
            value: workspace/review-output.md
          - type: text
            value: |
              This is a code review with structured findings. List every
              finding that has verdict "needs-attention". For each, state
              whether it has been addressed in the follow-up diff.
          - type: file
            value: workspace/followup-diff.patch
    assertions:
      - type: rubrics
        criteria:
          - id: all-findings-enumerated
            outcome: "Agent lists every needs-attention finding by number"
            weight: 1.0
            required: true
          - id: resolution-status-per-finding
            outcome: "Each finding gets an explicit resolved/unresolved status"
            weight: 1.0

Baseline comparison: Run the same review task with current (prose-based) reviewer prompt. Measure whether the parent agent can reliably enumerate and track findings. Expected delta: structured output → higher finding-tracking accuracy.

---

Eval 2: Subagent Output Contract (Change #2)

What to prove: Subagents returning structured output (status, files, tests, concerns) enables the parent to catch integration conflicts that prose-based output misses.

tests:
  - id: subagent-returns-contract
    criteria: |
      Implementer subagent returns structured output matching the
      contract: status, files changed, tests added, unresolved concerns.
    input:
      - role: user
        content:
          - type: file
            value: workspace/plan.md
          - type: file
            value: workspace/.claude/skills/hs-implement/references/implementer-prompt.md
          - type: text
            value: |
              Implement Task 2 from the plan. Follow the implementer
              prompt instructions for output format.
    assertions:
      - type: contains-all
        value: ["Status:", "Files changed:", "Tests added"]
      - type: rubrics
        criteria:
          - id: status-is-valid
            outcome: "Status is one of: DONE, DONE_WITH_CONCERNS, NEEDS_CONTEXT, BLOCKED"
            weight: 1.0
            required: true
          - id: concerns-surfaced
            outcome: "If there are shared types modified, this is flagged in unresolved concerns"
            weight: 1.0

  - id: integration-conflict-detected
    criteria: |
      Parent agent detects when two subagents modified the same shared
      type/interface, flagging a potential integration conflict.
    input:
      - role: user
        content:
          - type: file
            value: workspace/subagent-1-output.md
          - type: file
            value: workspace/subagent-2-output.md
          - type: text
            value: |
              Two subagents completed their tasks. Review their outputs
              for integration conflicts before merging.
    assertions:
      - type: rubrics
        criteria:
          - id: conflict-identified
            outcome: "Agent identifies that both subagents modified the shared UserConfig type"
            weight: 1.0
            required: true
          - id: resolution-proposed
            outcome: "Agent proposes how to reconcile the conflicting changes"
            weight: 1.0

---

Eval 3: Reliability Check for Non-Deterministic Features (Change #4)

What to prove: Agent invokes multiple test runs for non-deterministic code paths instead of treating a single pass as sufficient.

tests:
  - id: nondeterministic-gets-multiple-runs
    criteria: |
      When verifying a feature that calls an LLM API, the agent runs
      the test suite multiple times rather than accepting a single pass.
    input:
      - role: user
        content:
          - type: file
            value: workspace/.claude/skills/hs-verify/SKILL.md
          - type: text
            value: |
              The implementation is complete. The feature adds an LLM-powered
              summarization endpoint. Run hs-verify to prove it works.
    assertions:
      - type: rubrics
        criteria:
          - id: multiple-runs
            outcome: "Agent runs the test suite at least 3 times (pass^3 protocol)"
            weight: 1.0
            required: true
          - id: recognizes-nondeterminism
            outcome: "Agent explicitly identifies this as a non-deterministic feature requiring reliability checks"
            weight: 1.0

  - id: deterministic-gets-single-run
    criteria: |
      When verifying a pure deterministic feature (config parsing),
      the agent runs tests once — not wasting time on multiple runs.
    input:
      - role: user
        content:
          - type: file
            value: workspace/.claude/skills/hs-verify/SKILL.md
          - type: text
            value: |
              The implementation is complete. The feature adds YAML config
              file parsing with schema validation. Run hs-verify.
    assertions:
      - type: rubrics
        criteria:
          - id: single-run-sufficient
            outcome: "Agent runs the test suite once and accepts the result"
            weight: 1.0
          - id: no-unnecessary-reruns
            outcome: "Agent does NOT invoke multiple test runs for deterministic code"
            weight: 1.0

---

Eval 4: Retro Accumulation (Change #3)

What to prove: Agent distinguishes first-time interventions from recurring patterns and adjusts fix confidence accordingly.

tests:
  - id: first-occurrence-marked-candidate
    criteria: |
      On first encounter, the agent notes the intervention as a candidate
      rather than immediately modifying the skill.
    input:
      - role: user
        content:
          - type: file
            value: workspace/.claude/skills/hs-retro/SKILL.md
          - type: file
            value: workspace/transcript-single-intervention.jsonl
          - type: text
            value: Run hs-retro on this session transcript.
    assertions:
      - type: rubrics
        criteria:
          - id: candidate-not-committed
            outcome: "Agent marks the intervention as a candidate or notes it's a first occurrence"
            weight: 1.0
          - id: no-premature-skill-edit
            outcome: "Agent does NOT edit the target skill based on a single occurrence alone"
            weight: 1.0

  - id: recurring-pattern-applied
    criteria: |
      When the same intervention pattern appears in multiple session
      transcripts, the agent applies the fix with high confidence.
    input:
      - role: user
        content:
          - type: file
            value: workspace/.claude/skills/hs-retro/SKILL.md
          - type: file
            value: workspace/transcript-recurring-1.jsonl
          - type: file
            value: workspace/transcript-recurring-2.jsonl
          - type: file
            value: workspace/transcript-recurring-3.jsonl
          - type: text
            value: |
              Run hs-retro across these 3 session transcripts. The same
              intervention appears in all 3.
    assertions:
      - type: rubrics
        criteria:
          - id: recurrence-recognized
            outcome: "Agent explicitly notes this pattern recurs across multiple sessions"
            weight: 1.0
            required: true
          - id: fix-applied-with-confidence
            outcome: "Agent applies the skill fix, citing recurrence as justification"
            weight: 1.0

---

Running the Evals

Each eval would be run as an A/B comparison:

# Baseline (current skills)
agentv eval --file evals/hivespec/structured-review.eval.yaml \
  --target claude --workspace workspace-baseline/

# Candidate (updated skills)
agentv eval --file evals/hivespec/structured-review.eval.yaml \
  --target claude --workspace workspace-candidate/

# Compare
agentv compare baseline-results.jsonl candidate-results.jsonl

Key metrics to track:

• pass_rate delta (did the change improve pass rates?)
• token_usage delta (did structured output reduce back-and-forth?)
• Rubric scores per criterion (which specific behaviors improved?)

---

What This Doesn't Cover

These evals test whether the agent follows the updated skill instructions correctly. They don't directly measure end-user developer productivity (that requires longitudinal human studies). But they do prove:

1. The skill change produces a measurable behavioral difference in the agent
2. The new behavior is objectively better for the intended scenario (more findings tracked, conflicts caught, reliability validated)
3. The change doesn't regress existing behavior (deterministic features still get single runs, one-off interventions don't trigger premature edits)