Create Plugin / Plugin E2e: Improve agentic e2e testing experience

[sunker]

Problem

AI coding agents are increasingly used to write and maintain e2e tests for Grafana plugins. Today, an agent dropped into a plugin project has no idea how to use @grafana/plugin-e2e. The existing scaffolded instructions.md says "use @grafana/plugin-e2e" but provides no guidance on the API surface - fixtures, page models, custom matchers, provisioning patterns or version-aware selectors.

This means agents can fall back to writing raw Playwright code (page.goto, page.click) instead of idiomatic @grafana/plugin-e2e code (createDataSourceConfigPage, panelEditPage.datasource.set, toBeOK). The result is brittle tests that break across Grafana versions - exactly the problem plugin-e2e was built to solve.

Solution

Two parts, in order of complexity:

Part 1: Plugin-type-specific e2e testing docs for agents (low-hanging fruit)

Ship a testing-focused context file (e.g. tests/AGENTS.md or .config/AGENTS/e2e-testing.md) in newly scaffolded plugins via create-plugin. This file would be plugin-type-specific (datasource, panel, app) and include:

• The @grafana/plugin-e2e fixture API relevant to that plugin type (e.g. datasource plugins get createDataSourceConfigPage, panelEditPage, readProvisionedDataSource; app plugins get gotoAppPage, appConfigPage, the fixtures.ts extension pattern)
• Common test patterns as copyable code examples (config editor, query editor, provisioned resources, error scenarios, mock responses)
• Custom matcher reference (toBeOK, toHaveAlert, toDisplayPreviews)
• Tricky parts: the selectors.apis.DataSource.proxy() URL construction, force: true for checkboxes/switches, provisioning file structure
• Links to the live docs via the llms.txt convention for deeper exploration

This works with every AI tool (no MCP dependency), is version-controlled alongside the plugin and costs nothing at runtime. Templates already exist per plugin type (templates/datasource/, templates/app/, templates/panel/) so this fits the existing scaffolding model.

Part 2: Grafana plugin test agents (planner, generator, healer)

Inspired by Playwright Test Agents, scaffold Claude Code subagent definitions in .claude/agents/ that are specialized for Grafana plugin e2e testing:

• Planner - Reads plugin.json to determine plugin type, knows the standard test surface (config editor, query editor, variables, annotations, alerts for datasources; pages and config for app plugins; options and rendering for panels), produces a structured test plan mapped to @grafana/plugin-e2e fixtures
• Generator - Writes idiomatic @grafana/plugin-e2e tests using the correct fixtures, page models and matchers. Uses Playwright MCP browser tools to verify locators against the live app before writing assertions
• Healer - Understands Grafana-specific failure modes (version-aware selector changes, checkbox actionability quirks, provisioning file mismatches), debugs and fixes broken tests

These are just markdown files (~1,000-1,500 tokens each) that follow the Claude Code subagent format. They reference the existing Playwright MCP tools for browser interaction - no custom MCP server needed. The agents would be templated per plugin type, matching the existing create-plugin template structure.

Additionally, we can create an add cmd script that wires this up for existing plugins.

[mckn]

I think this sounds great! It would be great to try to do this in a standardised way so we don't commit too much to a single platform.

https://agentskills.io/what-are-skills
https://agents.md/

Or use sub-agents but try to reference other docs as much as possible so you don't need to repeat the information multiple times.