feat: plugin-validator skill config for claude and codex by s4kh · Pull Request #2468 · grafana/plugin-tools

s4kh · 2026-02-18T20:44:59Z

What this PR does / why we need it:

PoC for https://github.com/grafana/grafana-catalog-team/issues/677

/build-plugin skill

Figures out package manager using package.json
Checks if it has a backend based on backend prop of package.json
Build using instructions.md and run build and mage as fallbacks

Add skill to run /validate-plugin from claude code and codex:

Builds plugin
Creates zip with dist/ folder for validation
Calls plugin-validator with archive and source code DIR. Uses NPX for preferred way to call it and docker for fallback.
Shows report and suggests fixes.

Ex(claude code): /validate-plugin

Ex(codex): $validate-plugin

If we have an MCP server registered for validating the plugin, the skills will use the MCP server tool to call the plugin-validator.

github-actions · 2026-02-18T20:45:20Z

Hello! 👋 This repository uses Auto for releasing packages using PR labels.

✨ This PR can be merged but will not trigger a new release. To trigger a new release add the release label before merging.
NOTE: When merging a PR with the release label please avoid merging another PR. For further information see here.

s4kh · 2026-02-18T20:46:39Z

packages/create-plugin/templates/common/.codex/skills/validate-plugin.md

@@ -0,0 +1 @@
+@../../skills/validate-plugin.md


We have to create the skills inside codex, otherwise it does not directly recognize from root. Claude does it if the root has SKILLS.md.

Copilot

Pull request overview

This pull request adds AI agent skill configurations for validating Grafana plugins using the official @grafana/plugin-validator tool. The skill enables Claude Code and Codex to run plugin validation directly from their interfaces, providing developers with automated feedback on plugin compliance.

Changes:

Added validate-plugin skill that builds plugins (frontend and optionally backend), packages them as zip archives, and runs validation using npx or docker
Integrated the skill into both Claude Code (via .claude/skills/) and Codex (via .codex/skills/) by referencing the shared skill definition
Provided step-by-step instructions for AI agents to execute validation with error handling and user feedback

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 10 comments.

File	Description
packages/create-plugin/templates/common/.config/AGENTS/skills/validate-plugin.md	Main skill definition with bash commands for building, packaging, and validating Grafana plugins
packages/create-plugin/templates/common/.codex/skills/validate-plugin/SKILL.md	Codex skill registration with frontmatter referencing the shared skill definition
packages/create-plugin/templates/common/.claude/skills/validate-plugin/SKILL.md	Claude skill registration referencing the shared skill definition

packages/create-plugin/templates/common/.config/AGENTS/skills/validate-plugin.md

packages/create-plugin/templates/common/.claude/skills/validate-plugin/SKILL.md

packages/create-plugin/templates/common/.config/AGENTS/skills/validate-plugin.md

academo · 2026-02-19T09:20:17Z

packages/create-plugin/templates/common/.config/AGENTS/skills/validate-plugin.md

+
+Always use the bash commands below directly. Do NOT use MCP tools for validation.
+
+## Usage


these are instructions for the LLM. not the user. are you telling the LLM to use /validate-plugin or the user?

For the LLM, I am trying to compare like which is the better way to validate the plugin - through MCP or SKILLS. If we have an MCP registered with validated plugin in the description, it uses the MCP. So the above instruction is to directly use the CLI command.

academo · 2026-02-19T09:20:44Z

packages/create-plugin/templates/common/.config/AGENTS/skills/validate-plugin.md

+
+## Steps
+
+1. Check if `npx` or `docker` is available. npx is preferred, docker is the fallback:


docker is prefered for the validator, not npx

I see, when I ran from the Codex CLI or Claude code CLI npx runs way faster.

that's because it is skipping the security checks (which are inside the docker image) that's why it runs faster. But that's actually a good point. the security checks are very important at submission not necessarily on a fast-dev environment

Can we use the same runner as the plugin is scaffolded with?

For running the validator? Wouldn't it be better to use npx or docker? Since it is standalone CLI tool not tied to the plugin's package manager? and if nodejs is installed which is a requirement for plugin development npx is present.

packages/create-plugin/templates/common/.config/AGENTS/skills/validate-plugin.md

mckn

Great work on this and I think this is a great start! I'm a bit concerned that we are duplicating information about e.g. how to build the plugin. If we make updates to how the plugin is built we now also need to update this skill.

One idea is that we introduce another skill that provides information about how to, by reading the plugin, understand how to build the plugin. Then we can reference that skill in this skill and only say that the plugin needs to be built before running the validation.

The benefit of doing it that way would be that we can simplify this skill and it will only focus on the steps for how to validate the plugin.

mckn · 2026-02-24T13:20:56Z

packages/create-plugin/templates/common/.config/AGENTS/skills/validate-plugin.md

+
+## Steps
+
+1. Check if `npx` or `docker` is available. npx is preferred, docker is the fallback:


Can we use the same runner as the plugin is scaffolded with?

mckn · 2026-02-24T13:22:44Z

packages/create-plugin/templates/common/.config/AGENTS/skills/validate-plugin.md

+
+3. Build the frontend following the build instructions in `.config/AGENTS/instructions.md`:
+   ```bash
+   npm run build


We should use the package manager that the plugin is scaffolded with.

mckn · 2026-02-24T13:23:58Z

packages/create-plugin/templates/common/.config/AGENTS/skills/validate-plugin.md

+   ```
+   If the build fails, stop and report the error to the user.
+
+4. If `HAS_BACKEND` is non-zero (backend plugin detected), build the backend following the build instructions in `.config/AGENTS/instructions.md`:


Is it possible to introduce a skill that we can rely on regarding information about the plugin?

So provide a skill for how to detect what kind of plugin it and how it is built.

s4kh · 2026-02-24T17:05:57Z

Can we use the same runner as the plugin is scaffolded with?

For running the validator, we don't need to use the plugin's package manager, right?

One idea is that we introduce another skill that provides information about how to, by reading the plugin, understand how to build the plugin. Then we can reference that skill in this skill and only say that the plugin needs to be built before running the validation.

@mckn I totally agree with the package manager finding logic - but for build and other stuff, we are instructing the AI to read the AGENTS/instructions.md. Creating a skill to figure out what build is basically reading the instruction.md or reading the package.json - since skill can be invoked using /validate-plugin would we ever call the skill /how-to-build, we would directly ask the LLM 'How to build the plugin' should we create a separate skill for it?, I am thinking are we abstracting too much. Maybe /build-plugin is the way to go?

From claude's doc -> skills are way to orchestrate a sequence of actions to achieve something.
Validation workflow -> install, build, zip, validate.

s4kh · 2026-02-24T17:18:13Z

packages/create-plugin/templates/common/.config/AGENTS/skills/validate-plugin.md

+   ```
+   If `RUN_ENGINE` is `none`, stop immediately and tell the user: "Neither npx nor docker is installed. Please install Node.js (for npx) or Docker to run the plugin validator."
+
+2. Find `src/plugin.json` (or `plugin.json`) in the plugin directory and extract the plugin ID and whether it has a backend. Sanitize `PLUGIN_ID` to only allow characters valid in a Grafana plugin ID:


Figure out, if the plugin has the backend for building purposes.

s4kh · 2026-02-24T17:19:04Z

packages/create-plugin/templates/common/.config/AGENTS/skills/validate-plugin.md

+   HAS_BACKEND=$(grep -c '"backend" *: *true' src/plugin.json || true)
+   ```
+
+3. Detect the package manager. Check the `packageManager` field in `package.json` first, then fall back to lock file detection:


For front-end building, running the build with same package manager that plugin is scaffolded.

s4kh · 2026-02-24T18:08:48Z

packages/create-plugin/templates/common/.config/AGENTS/skills/build-plugin.md

+## Usage
+
+```
+/build-plugin


@mckn added separate skill for building the plugin.

Figures out package manager using package.json

Checks if it has a backend based on backend prop of package.json

Build using instructions.md and run build and mage as fallbacks

Nice! Looks great 🙌🏻

mckn

I think this looks great! Great work on this 🙌🏻

xnyo

Just one small comment but the rest LGTM! Great job! 🙌

xnyo · 2026-02-25T11:42:34Z

packages/create-plugin/templates/common/.config/AGENTS/skills/validate-plugin.md

+
+## Important
+
+Always use the bash commands below directly. Do NOT use MCP tools for validation.


Does the agent already know it should use the MCP server if it's available?

Yeah, it does, I was comparing the MCP vs Skills. But they are more of a complimentary to each other. Skills workflows, MCP specific single tool.

jackw

Approving to unblock (codeowners). Left a couple of comments.

jackw · 2026-02-26T09:29:23Z

packages/create-plugin/templates/common/.config/AGENTS/skills/build-plugin.md

+       grep '"packageManager"' package.json | sed -E 's/.*"packageManager" *: *"([^@]+).*/\1/'
+     elif [ -f "pnpm-lock.yaml" ]; then
+       echo "pnpm"
+     elif [ -f "bun.lock" ] || [ -f "bun.lockb" ]; then


I would suggest removing this as we don't officially support bun.

jackw · 2026-02-26T09:33:54Z

packages/create-plugin/templates/common/.config/AGENTS/skills/validate-plugin.md

+5. Run the validator with JSON output using `$RUN_ENGINE` from step 1 and `$ZIP_NAME` from step 4:
+   If `$RUN_ENGINE` is `npx`:
+   ```bash
+   npx --cache .cache/npm -y @grafana/plugin-validator@latest -jsonOutput $ZIP_NAME


What's the need to set the --cache path?

when running from codex it is using the ~/.cache for npx cache and permission denied error is returned, so we are setting it to use the plugins .cache folder.

github-project-automation bot added this to Grafana Catalog Team Feb 18, 2026

github-project-automation bot moved this to 📬 Triage in Grafana Catalog Team Feb 18, 2026

s4kh self-assigned this Feb 18, 2026

s4kh moved this from 📬 Triage to 🧑‍💻 In development in Grafana Catalog Team Feb 18, 2026

s4kh commented Feb 18, 2026

View reviewed changes

s4kh added chore Other changes that don't modify src or test files minor Increment the minor version when merged labels Feb 18, 2026

s4kh force-pushed the feat-skills branch from 044bfbb to d21877c Compare February 18, 2026 22:02

s4kh changed the title ~~feat: validate plugin skills config for claude and codex~~ feat: validate plugin skills config for claude Feb 18, 2026

s4kh requested review from academo, mckn, tolzhabayev and xnyo February 18, 2026 23:20

s4kh force-pushed the feat-skills branch from d21877c to 0d65ea0 Compare February 18, 2026 23:21

s4kh changed the title ~~feat: validate plugin skills config for claude~~ feat: plugin-validator skill config for claude and codex Feb 18, 2026

s4kh marked this pull request as ready for review February 18, 2026 23:23

s4kh requested a review from a team as a code owner February 18, 2026 23:23

s4kh requested review from Copilot, hugohaggmark and jackw February 18, 2026 23:23

Copilot started reviewing on behalf of s4kh February 18, 2026 23:23 View session

Copilot AI reviewed Feb 18, 2026

View reviewed changes

academo reviewed Feb 19, 2026

View reviewed changes

packages/create-plugin/templates/common/.config/AGENTS/skills/validate-plugin.md Outdated Show resolved Hide resolved

s4kh force-pushed the feat-skills branch from 1f79804 to 1355bc0 Compare February 19, 2026 13:47

s4kh moved this from 🧑‍💻 In development to 🔬 In review in Grafana Catalog Team Feb 20, 2026

s4kh requested a review from academo February 20, 2026 15:10

mckn reviewed Feb 24, 2026

View reviewed changes

s4kh force-pushed the feat-skills branch from 40b5ec6 to 3afb73c Compare February 24, 2026 17:16

s4kh commented Feb 24, 2026

View reviewed changes

s4kh force-pushed the feat-skills branch 2 times, most recently from 156100e to d177222 Compare February 24, 2026 17:30

s4kh commented Feb 24, 2026

View reviewed changes

s4kh requested a review from mckn February 24, 2026 18:09

s4kh force-pushed the feat-skills branch 3 times, most recently from 9360b45 to bf08111 Compare February 24, 2026 18:47

mckn approved these changes Feb 25, 2026

View reviewed changes

xnyo approved these changes Feb 25, 2026

View reviewed changes

s4kh force-pushed the feat-skills branch from bf08111 to 68ada7d Compare February 25, 2026 14:16

jackw approved these changes Feb 26, 2026

View reviewed changes

s4kh added 7 commits February 26, 2026 08:37

feat: validate plugin skills config for claude and codex

a25a8c7

fix: reviews addressed

3c7f0b6

ref: rename RUN ENGINE

5e51f39

ref: add package manager finding logic and extract build-plugin

9125ac0

ref: point to package section of instructions.md

7e03045

chore: network access config for codex

5dcb008

fix: use plugins cache folder for running the npx validator

26531d4

s4kh force-pushed the feat-skills branch from 6631bb0 to 26531d4 Compare February 26, 2026 13:38

s4kh merged commit f6b55c0 into main Feb 26, 2026
28 checks passed

s4kh deleted the feat-skills branch February 26, 2026 14:06

github-project-automation bot moved this from 🔬 In review to 🚀 Shipped in Grafana Catalog Team Feb 26, 2026


		Always use the bash commands below directly. Do NOT use MCP tools for validation.

		## Usage


		## Steps

		1. Check if `npx` or `docker` is available. npx is preferred, docker is the fallback:


		## Important

		Always use the bash commands below directly. Do NOT use MCP tools for validation.

		@@ -0,0 +1 @@
		@../../skills/validate-plugin.md

Conversation

s4kh commented Feb 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Feb 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mckn left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

s4kh commented Feb 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mckn left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xnyo left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jackw left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

s4kh commented Feb 18, 2026 •

edited

Loading

github-actions bot commented Feb 18, 2026 •

edited

Loading

s4kh commented Feb 24, 2026 •

edited

Loading

mckn left a comment •

edited

Loading