feat(ci): add marketplace, eval, and markdown validation workflow#861
Merged
feat(ci): add marketplace, eval, and markdown validation workflow#861
Conversation
… Actions - Add composite setup-bun action with caching (mirrors opencode pattern) - Add marketplace job: schema validation, sort check, sync check, frontmatter validation - Add evals job: installs agentv globally, validates eval dirs have eval files, runs agentv validate with glob patterns - Add glob pattern support to `agentv validate` command (uses fast-glob, already a dependency) - Sort marketplace.json plugins alphabetically and sync .github/plugin/ copy Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Instead of hardcoding known exceptions in the eval-dirs validation script, add proper eval YAML files to the compare and trace-analysis example directories so they pass validation like all other features. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Deploying agentv with
|
| Latest commit: |
4b42302
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://1b52946c.agentv.pages.dev |
| Branch Preview URL: | https://feat-ci-validate.agentv.pages.dev |
- Run agentv validate from built dist (bun apps/cli/dist/cli.js) instead of installing from npm, which lacks the new glob support - Replace import.meta.url with __dirname in pi-coding-agent.ts to eliminate the esbuild CJS warning about empty import.meta Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Enable `shims: true` in tsup config instead of manual runtime guards. tsup injects proper CJS shims (pathToFileURL(__filename)) so import.meta.url works in CJS output without esbuild warnings, while ESM output uses native import.meta.url directly. Reverts pi-coding-agent.ts to its original idiomatic ESM code since the build tool now handles cross-format compatibility. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The criteria field was set to ${{ EVAL_CRITERIA }} which resolves to
empty string when the env var isn't set, causing validation to fail in
CI. Move interpolation demo to expected_output and input fields instead,
keeping criteria as a static string.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
name and description are optional metadata — no reason to couple them. name can be derived from the filename if not provided. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Compare eval: match all 5 fixture test IDs (code-review-001/002/003, code-gen-001/002) so eval output is directly comparable - Remove $schema from eval YAML files — not needed - Verified: full eval run passes (5/5, 3/3), output is agentv-compare-compatible with matching test IDs Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
setup-bunGitHub Action with dependency caching (mirrors opencode pattern).mdfilesagentvglobally, verifies everyevals/directory has at least one eval file, runsagentv validatewith glob patterns against all example evalsagentv validatecommand (reusesfast-glob, already a CLI dependency)marketplace.jsonplugins alphabetically and sync.github/plugin/copycompareandtrace-analysisexamples (previously had only result files, no eval definitions)Changes
.github/actions/setup-bun/action.yml.github/workflows/validate.ymlmarketplaceandevalsjobsapps/cli/src/commands/validate/validate-files.tsexpandPaths()scripts/marketplace/*.tsscripts/validate-eval-dirs.tsexamples/features/compare/evals/dataset.eval.yamlexamples/features/trace-analysis/evals/dataset.eval.yaml.claude-plugin/marketplace.json.github/plugin/marketplace.jsonTest plan
agentv validatewith glob patterns finds and validates 52 eval files🤖 Generated with Claude Code