Skip to content

iikit-clarify needs eval scenario coverage #85

Description

@jbaruch

The iikit-clarify skill has zero eval scenarios despite carrying substantial decisional logic:

  • Auto-detection of target artifact across 6 artifact types (spec, plan, checklist, testify, tasks, constitution)
  • Ambiguity scanning per artifact-specific taxonomy
  • Sequential questioning + answer integration
  • Per-artifact reference format (FR-xxx, US-x, SC-xxx, plan headers, check items, T-xxx, principle names)
  • Testify exception path (Gherkin can't hold markdown)
  • Checklist re-validation after spec clarification (added in feat(iikit-clarify): re-validate checklists after spec clarification #79)

Per jbaruch/coding-policy: plugin-evals: "Every skill with decisional logic ships eval cases."

Surfaced on PR #79 when the policy reviewer flagged the new branching behavior — but the underlying issue is that the whole skill has been untested in evals. Filing this for a focused eval-coverage effort rather than tacking one scenario onto every behavioral change PR.

Scope

Add scenarios covering at least:

  • Auto-detection picks correct artifact (positive case per artifact type)
  • Question generation respects per-artifact reference scheme
  • Integration writes Q&A back into the right section format
  • Testify exception writes to tests/clarifications.md not the .feature file
  • Checklist re-validation (from feat(iikit-clarify): re-validate checklists after spec clarification #79): stale-regression, resolved-gap, non-spec no-op

Use tessl scenario generate for happy-path then hand-author negative cases per plugin-evals guidance.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions