Skip to content

feat(cli): add --tag / --exclude-tag filtering for eval runs #900

@christso

Description

@christso

Problem

Eval files support tags: [...] in the schema, but the CLI has no way to filter by them. CI currently uses hardcoded glob patterns to exclude multi-provider evals, which is fragile.

Proposal

Add --tag and --exclude-tag flags to agentv eval run:

# Run only agent-tagged evals
agentv eval run 'examples/**/*.eval.yaml' --tag agent

# Run everything except multi-provider evals
agentv eval run 'examples/**/*.eval.yaml' --exclude-tag multi-provider

# Combine (AND logic, matching promptfoo convention)
agentv eval run 'examples/**/*.eval.yaml' --tag agent --exclude-tag slow

Prior art

  • promptfoo: --filter-metadata tags=X on test cases
  • pytest: @pytest.mark.slow + -m "not slow" marker filtering
  • Jest: --testPathPattern + --testNamePattern for filtering

Both promptfoo and pytest support inclusive and exclusive filtering with AND logic for multiple filters.

Implementation notes

  • Tags are already parsed into EvalFileMetadata.tags (packages/core/src/evaluation/metadata.ts)
  • Filter should apply at the eval-file level (skip entire files), not test-case level
  • --exclude-tag would replace hardcoded glob exclusions in CI workflows

Metadata

Metadata

Assignees

Labels

in-progressClaimed by an agent — do not duplicate work

Type

No type

Projects

Status

No status

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions