Skip to content

[Libretto] implement skip/xfail annotation fields at file and case levels of the test model #5

@typotter

Description

@typotter

Implement skip/xfail annotation fields at file-level and case-level in the test data schema. Skip prevents test execution; xfail marks tests as expected to fail. Both fields support SDK-scoped values to handle SDK-specific behavioral divergences and unimplemented features.

Acceptance Criteria

  • All evaluation-cases/test-*.json files migrated from bare array to wrapper object format with cases key
  • File-level skip and xfail fields supported on wrapper object
  • Case-level skip and xfail fields supported on each case object
  • SkipValue type: true | ["sdk", ...] | null
  • XfailValue type: true | "reason string" | {"sdk": "reason", ...} | {"sdk": true, ...} | null
  • Canonical SDK ids: "go", "java", "python", "dotnet"
  • Case-level annotations override file-level; null at case-level opts back in to normal run
  • Skip takes precedence over xfail when both apply
  • README.md updated to document new schema
  • CLAUDE.md updated to document new schema
  • At least one example file demonstrates annotations at both file and case levels

Metadata

Metadata

Assignees

No one assigned

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions