diff --git a/src/content/docs/cli.mdx b/src/content/docs/cli.mdx index 5b249e1305..69f11553dc 100644 --- a/src/content/docs/cli.mdx +++ b/src/content/docs/cli.mdx @@ -67,3 +67,4 @@ mergify --token your_token_here | `mergify queue` | Monitor and control merge queue | [Monitoring](/merge-queue/monitoring) | | `mergify config` | Validate and simulate config | [Config](/configuration/file-format#validating-with-the-cli) | | `mergify ci` | Upload and analyze CI results | [CI Insights](/ci-insights) | +| `mergify tests show` | Search test health by name | [Test Insights CLI](/test-insights/cli) | diff --git a/src/content/docs/test-insights.mdx b/src/content/docs/test-insights.mdx index 31b3f771c2..2028dd65f2 100644 --- a/src/content/docs/test-insights.mdx +++ b/src/content/docs/test-insights.mdx @@ -29,6 +29,11 @@ of a test reliability problem: to unblock CI without removing them. Tests keep running, but their failures no longer block merges. +You can also search test health directly from the terminal or from AI coding +agents with the [Test Insights CLI](/test-insights/cli). This is useful when +an agent needs to decide whether a failing test is a real regression or +known flakiness. + ## Key concepts - **Flaky test**: A test that produces different results on the same commit. diff --git a/src/content/docs/test-insights/cli.mdx b/src/content/docs/test-insights/cli.mdx new file mode 100644 index 0000000000..a5f54fc02c --- /dev/null +++ b/src/content/docs/test-insights/cli.mdx @@ -0,0 +1,83 @@ +--- +title: CLI +description: Interact with Test Insights data from the terminal — search for tests, branch on health via exit codes, and chain into AI coding agents like Claude Code and Cursor. +--- + +The [Mergify CLI](/cli) lets you interact with Test Insights data from +the terminal. It's handy when you want to triage a failing test +quickly, or when an AI coding agent needs to decide whether to retry, +fix, or ignore a failure. + +## `mergify tests show` + +Search the Test Insights catalog by test name and print health, ratios, +and the most recent failure context for each match. + +```bash +mergify tests show -r owner/repo "tests/auth/test_login.py::test_login_timeout" +``` + +Names accept glob patterns (`*`, `?`), and several can be passed in one +call. Add `--json` for a machine-readable payload. Run +`mergify tests show --help` for the full list of flags, and see the +[API reference](/api/test-insights) for the underlying endpoints and +response schemas. + +### Exit codes + +The exit code reflects the worst health observed across the matched +tests, so callers can branch without parsing output: + +| Code | Meaning | +|---|---| +| `0` | All matched tests are healthy or unknown, or nothing matched. | +| `1` | At least one matched test is flaky. | +| `6` | At least one matched test is broken. | + +Exit code `2` is reserved for CLI argument errors. + +## AI coding agent recipes + +The exit-code contract is designed to be chained from agents that ran +a test, saw it fail, and need to decide what to do next. + +### Should the agent retry, fix, or ignore? + +```bash +if mergify tests show -r owner/repo "$FAILED_TEST" --json > /tmp/health.json; then + status=0 +else + status=$? +fi + +case $status in + 0) echo "Healthy or unknown: investigate as a real regression." ;; + 1) echo "Known flaky: retry before touching code." ;; + 6) echo "Broken: fix is required; skip the retry loop." ;; +esac +``` + +### Claude Code + +Add a [skill](https://docs.claude.com/en/docs/claude-code/skills) or a +prompt snippet that points the agent at the CLI: + +```md +When a test fails locally or in CI, before attempting a fix, run: + + mergify tests show -r OWNER/REPO --json "" + +Then branch on the exit code: +- 0, non-empty payload → healthy in Mergify; investigate as a real + regression. +- 0, empty payload → unknown to Mergify; treat as a new test or one on + a non-default branch. +- 1 → known flaky. Rerun once; only investigate if it fails again. +- 6 → known broken (pre-existing). Surface to the user instead of + attempting a fix. +``` + +### Cursor + +In Cursor, add the same guidance to a project rule +(`.cursor/rules/test-triage.mdc`) so it auto-loads in agent sessions. diff --git a/src/content/navItems.tsx b/src/content/navItems.tsx index d7515455fb..df28cb8115 100644 --- a/src/content/navItems.tsx +++ b/src/content/navItems.tsx @@ -123,6 +123,7 @@ const navItems: NavItem[] = [ { title: 'Detection', path: '/test-insights/detection', icon: 'lucide:search' }, { title: 'Mitigation', path: '/test-insights/mitigation', icon: 'lucide:radiation' }, { title: 'Quarantine', path: '/test-insights/quarantine', icon: 'lucide:radiation' }, + { title: 'CLI', path: '/test-insights/cli', icon: 'lucide:terminal' }, { title: 'Test Frameworks Setup', path: '/test-insights#test-framework-configuration',