-
Notifications
You must be signed in to change notification settings - Fork 0
feat(cli): add --tag / --exclude-tag filtering for eval runs #900
Copy link
Copy link
Closed
Labels
in-progressClaimed by an agent — do not duplicate workClaimed by an agent — do not duplicate work
Description
Problem
Eval files support tags: [...] in the schema, but the CLI has no way to filter by them. CI currently uses hardcoded glob patterns to exclude multi-provider evals, which is fragile.
Proposal
Add --tag and --exclude-tag flags to agentv eval run:
# Run only agent-tagged evals
agentv eval run 'examples/**/*.eval.yaml' --tag agent
# Run everything except multi-provider evals
agentv eval run 'examples/**/*.eval.yaml' --exclude-tag multi-provider
# Combine (AND logic, matching promptfoo convention)
agentv eval run 'examples/**/*.eval.yaml' --tag agent --exclude-tag slowPrior art
- promptfoo:
--filter-metadata tags=Xon test cases - pytest:
@pytest.mark.slow+-m "not slow"marker filtering - Jest:
--testPathPattern+--testNamePatternfor filtering
Both promptfoo and pytest support inclusive and exclusive filtering with AND logic for multiple filters.
Implementation notes
- Tags are already parsed into
EvalFileMetadata.tags(packages/core/src/evaluation/metadata.ts) - Filter should apply at the eval-file level (skip entire files), not test-case level
--exclude-tagwould replace hardcoded glob exclusions in CI workflows
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
in-progressClaimed by an agent — do not duplicate workClaimed by an agent — do not duplicate work
Type
Projects
Status
No status