feat(evals): add local report command and developer documentation

docs/behavioral-evals.md

@@ -0,0 +1,186 @@
+# Behavioral Evaluations & EDK Guide
+
+This guide introduces the **Eval Development Kit (EDK)** and details how to
+write, validate, run, and report on **behavioral evaluations** in the Gemini CLI
+codebase.
+
+---
+
+## Overview
+
+Behavioral evaluations are automated tests designed to assert on the
+**behavior** of the Gemini CLI agent (e.g., verifying which tools are called,
+checking call ordering, or avoiding destructive commands) rather than checking
+the final prose output.
+
+Evaluating agent behavior is critical because:
+
+1. Model responses are non-deterministic, making exact prose matching highly
+   fragile.
+2. We must ensure the model utilizes the most efficient tools (e.g., batching
+   files via `read_many_files` instead of sequential `read_file` calls).
+3. We must enforce safety boundaries (e.g., preventing execution of raw shell
+   commands when safe alternatives exist).
+
+All behavioral evaluations are stored under the
+[evals/](file:///c:/coding/gemini-cli/evals/) directory.
+
+---
+
+## EDK Developer Commands
+
+The EDK provides CLI tools under `scripts/` to help contributors audit, check,
+and monitor evals.
+
+### 1. `npm run eval:inventory`
+
+Scans all eval files under `evals/`, statically parses them, and provides a
+structured overview of what exists in the repository.
+
+- **Usage:**
+  ```bash
+  npm run eval:inventory
+  ```
+- **JSON Output:** For CI integration or inventory indexing, generate a
+  machine-readable JSON report:
+  ```bash
+  npm run eval:inventory -- --json
+  ```
+- **Custom Root:** Run against another directory or repository:
+  ```bash
+  npm run eval:inventory -- --root /path/to/other/repo
+  ```
+
+---
+
+### 2. `npm run eval:validate`
+
+A lint-like checker that validates eval source files against standard structural
+guidelines and best practices.
+
+- **Usage:**
+  ```bash
+  npm run eval:validate
+  ```
+- **Custom Scopes:** Validate a specific file:
+  ```bash
+  npm run eval:validate -- evals/my-test.eval.ts
+  ```
+
+#### Validation Rules & Severities
+
+| Rule ID              | Severity    | Description                                                                                                            |
+| :------------------- | :---------- | :--------------------------------------------------------------------------------------------------------------------- |
+| `file-naming`        | **Error**   | File must match `*.eval.ts` or `*.eval.tsx` naming conventions.                                                        |
+| `valid-policy`       | **Error**   | Policy must be one of `ALWAYS_PASSES`, `USUALLY_PASSES`, or `USUALLY_FAILS`.                                           |
+| `suite-metadata`     | **Error**   | Both `suiteName` and `suiteType` must be present as static string literals.                                            |
+| `prompt-presence`    | **Error**   | Every eval case must have a non-empty `prompt` string.                                                                 |
+| `case-name-static`   | **Error**   | The case name must be a static string literal, not computed dynamically.                                               |
+| `invalid-tool-refs`  | **Error**   | All tools referenced in assertions must match known built-in or legacy tools.                                          |
+| `positive-assertion` | **Error**   | Evaluation cases must assert on at least one tool call (e.g., check `waitForToolCall` has been invoked).               |
+| `workspace-setup`    | **Error**   | Workspace behaviors (like file-system edits/reads) must set up a `files` object.                                       |
+| `new-evals-policy`   | **Warning** | New evals must not use `ALWAYS_PASSES` policy initially (they should be promoted after nightly data proves stability). |
+
+Warnings (`new-evals-policy`) will be logged with `⚠` and will **not** cause
+the CLI process to exit with status `1`. Errors (`✗`) will block CI builds and
+return exit status `1`.
+
+---
+
+### 3. `npm run eval:report`
+
+Aggregates local vitest `report.json` artifacts, maps them against inventory
+policies, and summarizes the pass rates per model.
+
+- **Usage:**
+  ```bash
+  npm run eval:report
+  ```
+  By default, it scans `evals/logs/` recursively for `report.json` files.
+- **Specifying Directory:**
+  ```bash
+  npm run eval:report -- /path/to/logs
+  ```
+- **JSON Output:**
+  ```bash
+  npm run eval:report -- --json
+  ```
+
+---
+
+## Contributor Workflow
+
+When writing a new behavioral evaluation, adhere to this workflow to ensure
+high-quality, non-flaky test runs.
+
+### Step-by-Step Guide
+
+1. **Identify the Target Behavior**: Determine which tool calls need
+   verification (e.g., `web_fetch` must be called).
+2. **Author the Eval File**: Create your file under `evals/<name>.eval.ts`
+   naming it properly.
+3. **Configure Workspace Files**: If the eval reads or edits files, define them
+   inside the `files` metadata field.
+4. **Assert Behavior, Not Prose**: Ensure the `assert` block checks tool
+   interactions using `rig.waitForToolCall` or similar. Do not check final
+   prose.
+5. **Run Locally**:
+   ```bash
+   RUN_EVALS=true npx vitest run evals/my-test.eval.ts
+   ```
+6. **Deflake**: Run your eval at least 3 times locally to verify it does not
+   fail due to model variance.
+7. **Run Validation**: Run `npm run eval:validate` to ensure no linting errors
+   are present.
+
+### Acceptance Criteria Checklist
+
+- [ ] **Naming**: File ends with `.eval.ts` or `.eval.tsx`.
+- [ ] **Policy**: New evals start as `USUALLY_PASSES`.
+- [ ] **Metadata**: Static `suiteName` and `suiteType` (e.g. `'behavioral'`) are
+      specified.
+- [ ] **Assertions**: Uses `rig.waitForToolCall` or asserts tool arguments
+      explicitly.
+- [ ] **Clean workspace**: Does not write to files outside `rig.testDir`.
+
+### Common Anti-Patterns to Avoid
+
+- **Restricting core tools**: Never override `settings.tools.core` to limit
+  tools. Evals must run against the default toolset.
+- **Checking model prose**: Avoid `expect(result).toContain('something')` since
+  model wording is non-deterministic.
+- **Integration-only testing**: Evals that only write files without checking
+  realistic model prompts are integration tests and belong under
+  `integration-tests/`.
+
+---
+
+## CI & Dashboard Integration
+
+You can easily automate behavioral evaluations or compile dashboard data using
+EDK's JSON reporters.
+
+### CI Validation Block
+
+Add a step in your PR checks or GitHub workflows to automatically lint new evals
+and block pull requests containing validation errors:
+
+```yaml
+- name: Run Eval Validator
+  run: npm run eval:validate
+```
+
+### Publishing to a Dashboard
+
+To record nightly performance metrics across multiple models:
+
+1. Configure your workflow to run evaluations with the JSON reporter:
+   ```bash
+   cross-env GEMINI_MODEL=gemini-2.5-pro npx vitest run --config evals/vitest.config.ts --reporter=json --outputFile="evals/logs/eval-logs-gemini-2.5-pro/report.json"
+   ```
+2. Aggregate all test runs using the reporting tool:
+   ```bash
+   npm run eval:report -- evals/logs --json > aggregated_report.json
+   ```
+3. Upload `aggregated_report.json` to your dashboard storage backend to
+   visualize pass rates over time.

docs/sidebar.json

@@ -217,6 +217,10 @@
       {
         "label": "Development",
         "items": [
+          {
+            "label": "Behavioral evaluations",
+            "slug": "docs/behavioral-evals"
+          },
           { "label": "Contribution guide", "slug": "docs/contributing" },
           { "label": "Integration testing", "slug": "docs/integration-tests" },
           {

package.json

@@ -34,6 +34,7 @@
     "docs:keybindings": "tsx ./scripts/generate-keybindings-doc.ts",
     "eval:inventory": "tsx ./scripts/eval-inventory-cli.ts",
     "eval:inventory:json": "tsx ./scripts/eval-inventory-cli.ts --json",
+    "eval:report": "tsx ./scripts/eval-report-cli.ts",
     "build": "node scripts/build.js",
     "build-and-start": "npm run build && npm run start --",
     "build:vscode": "node scripts/build_vscode_companion.js",

scripts/eval-report-cli.ts

@@ -0,0 +1,81 @@
+#!/usr/bin/env tsx
+
+/**
+ * @license
+ * Copyright 2026 Google LLC
+ * SPDX-License-Identifier: Apache-2.0
+ */
+
+/**
+ * @fileoverview CLI entry point to summarize eval report.json files.
+ *
+ * Scans a directory for report.json files, groups them by model name,
+ * and prints pass rate summaries. Integrates with static inventory data
+ * to display static policies.
+ *
+ * Usage:
+ *   npm run eval:report
+ *   npm run eval:report -- <reports-directory> [--json] [--root <repo-root>]
+ */
+
+import path from 'node:path';
+import fs from 'node:fs';
+import { collectInventory } from './utils/eval-inventory.js';
+import {
+  summarizeReports,
+  formatReportSummary,
+  formatReportSummaryJson,
+} from './utils/eval-report.js';
+
+async function main() {
+  const args = process.argv.slice(2);
+
+  const jsonFlagIndex = args.indexOf('--json');
+  const jsonMode = jsonFlagIndex !== -1;
+  if (jsonMode) args.splice(jsonFlagIndex, 1);
+
+  const rootFlagIndex = args.indexOf('--root');
+  let repoRoot: string | undefined;
+  if (rootFlagIndex !== -1) {
+    repoRoot = args[rootFlagIndex + 1];
+    if (repoRoot === undefined || repoRoot.startsWith('--')) {
+      console.error('Error: --root requires a valid directory path.');
+      process.exit(1);
+    }
+    args.splice(rootFlagIndex, 2);
+  }
+
+  const resolvedRoot = repoRoot ? path.resolve(repoRoot) : process.cwd();
+
+  // The first positional argument is the directory of reports
+  const reportsDirArg = args.find((a) => !a.startsWith('--'));
+  const reportsDir = reportsDirArg
+    ? path.resolve(reportsDirArg)
+    : path.join(resolvedRoot, 'evals', 'logs');
+
+  if (!fs.existsSync(reportsDir)) {
+    console.error(`Error: Reports directory does not exist: ${reportsDir}`);
+    process.exit(1);
+  }
+
+  // Try to load inventory if available to match policies
+  let inventory;
+  try {
+    inventory = await collectInventory(resolvedRoot);
+  } catch {
+    // If inventory fails to load (e.g. running outside repo), proceed without it
+  }
+
+  const summary = await summarizeReports(reportsDir, inventory);
+
+  if (jsonMode) {
+    console.log(formatReportSummaryJson(summary, resolvedRoot));
+  } else {
+    console.log(formatReportSummary(summary, resolvedRoot));
+  }
+}
+
+main().catch((error) => {
+  console.error('Fatal error:', error);
+  process.exit(1);
+});