feat: add agent evaluation CLI

.claude/settings.json

@@ -1,22 +1,15 @@
 {
   "$schema": "https://json.schemastore.org/claude-code-settings.json",
-  "description": "Settings for Claude code agents",
   "permissions": {
-    "defaultMode": "default",
     "allow": [
       "Bash(pnpm lint:*)",
       "Bash(pnpm lint:fix:*)",
       "Bash(pnpm typecheck:*)",
       "Bash(pnpm build:*)",
       "Bash(pnpm format:*)",
       "Bash(pnpm format:check:*)",
-      "Bash(pnpm test:*)"
-    ],
-    "ask": [
-      "Bash(pnpm install:*)",
-      "Bash(pnpm add:*)",
-      "Bash(pnpm remove:*)",
-      "Bash(git commit:*)"
+      "Bash(pnpm test:*)",
+      "Bash(tsx scripts/scaffold-cli.ts:*)"
     ],
     "deny": [
       "Bash(curl:*)",
@@ -29,6 +22,14 @@
       "Read(**/secrets/**)",
       "Bash(git push:*)",
       "Bash(gh pr create:*)"
-    ]
-  }
+    ],
+    "ask": [
+      "Bash(pnpm install:*)",
+      "Bash(pnpm add:*)",
+      "Bash(pnpm remove:*)",
+      "Bash(git commit:*)"
+    ],
+    "defaultMode": "default"
+  },
+  "description": "Settings for Claude code agents"
 }

AGENTS.md

@@ -7,7 +7,7 @@
 **Repo:** `cli-agent-sandbox` — minimal TypeScript CLI sandbox built with `@openai/agents` and tool sandboxing under `tmp/`.
 
 1. Start at `src/cli/<cli>/main.ts` and the matching `src/cli/<cli>/README.md`.
-2. Follow the pipeline classes under `src/cli/<cli>/clients/*` and schemas under `src/cli/<cli>/types/*`.
+2. Follow the pipeline classes under `src/cli/<cli>/clients/*` and schemas under `src/cli/<cli>/types/schemas.ts`.
 3. Reuse shared helpers: `src/utils/parse-args.ts`, `src/utils/question-handler.ts`, `src/clients/logger.ts`.
 4. Keep `main.ts` focused on the basic agent flow; move non-trivial logic into `clients/` or `utils/`.
 5. Keep changes minimal; add/update **Vitest** tests (`*.test.ts`) when behavior changes.
@@ -94,6 +94,9 @@ All file tools are sandboxed to `tmp/` using path validation (`src/tools/utils/f
 - **`listFiles`** (`src/tools/list-files/list-files-tool.ts`)
   - Lists files/dirs under `tmp/`.
   - Params: `{ path?: string }` (defaults to `tmp/` root)
+- **`deleteFile`** (`src/tools/delete-file/delete-file-tool.ts`)
+  - Deletes a file under `tmp/`.
+  - Params: `{ path: string }` (path is **relative to `tmp/`**)
 - **`runPython`** (`src/tools/run-python/run-python-tool.ts`)
   - Runs a Python script from a configured scripts directory.
   - Params: `{ scriptName: string, input: string }` (input is JSON string; pass `""` for no input)
@@ -117,7 +120,7 @@ All file tools are sandboxed to `tmp/` using path validation (`src/tools/utils/f
 - Prefer TypeScript path aliases over deep relative imports: `~tools/*`, `~clients/*`, `~utils/*`.
 - Use Zod schemas for CLI args and tool IO.
 - Keep object field names in `camelCase` (e.g., `trainSamples`), not `snake_case`.
-- Keep Zod schemas in a dedicated `schemas.ts` file for each CLI (avoid inline schemas in `main.ts`).
+- Keep Zod schemas in a dedicated `types/schemas.ts` file for each CLI (avoid inline schemas in `main.ts`).
 - Keep constants in a dedicated `constants.ts` file for each CLI.
 - Move hardcoded numeric values into `constants.ts` (treat numbers as configuration).
 - For HTTP fetching in code, prefer `Fetch` (sanitized) or `PlaywrightScraper` for JS-heavy pages.

README.md

@@ -1,6 +1,6 @@
 # cli-agent-sandbox
 
-A minimal TypeScript CLI sandbox for testing agent workflows and safe web scraping. This is a single-package repo built with [`@openai/agents`](https://github.com/openai/openai-agents-js), and it includes a guestbook demo, a Finnish name explorer CLI, a publication scraping pipeline with a Playwright-based scraper for JS-rendered pages, an ETF backtest CLI, and agent tools scoped to `tmp` with strong safety checks.
+A minimal TypeScript CLI sandbox for testing agent workflows and safe web scraping. This is a single-package repo built with [`@openai/agents`](https://github.com/openai/openai-agents-js), and it includes a guestbook demo, a Finnish name explorer CLI, a publication scraping pipeline with a Playwright-based scraper for JS-rendered pages, an ETF backtest CLI, an agent evals CLI, and agent tools scoped to `tmp` with strong safety checks.
 
 ## Quick Start
 
@@ -9,9 +9,10 @@ A minimal TypeScript CLI sandbox for testing agent workflows and safe web scrapi
 3. Install Playwright system deps (Chromium): `pnpm exec playwright install-deps chromium`
 4. Set `OPENAI_API_KEY` (export it or add to `.env`)
 5. Run the demo: `pnpm run:guestbook`
-6. (Optional) Explore Finnish name stats: `pnpm run:name-explorer -- --mode ai|stats`
-7. (Optional) Run publication scraping: `pnpm run:scrape-publications -- --url="https://example.com"`
-8. (Optional) Run ETF backtest: `pnpm run:etf-backtest -- --isin=IE00B5BMR087` (requires Python setup below)
+6. (Optional) Run agent evals: `pnpm run:agent-evals -- --suite=example`
+7. (Optional) Explore Finnish name stats: `pnpm run:name-explorer -- --mode ai|stats`
+8. (Optional) Run publication scraping: `pnpm run:scrape-publications -- --url="https://example.com"`
+9. (Optional) Run ETF backtest: `pnpm run:etf-backtest -- --isin=IE00B5BMR087` (requires Python setup below)
 
 ### Python Setup (for ETF backtest)
 
@@ -29,6 +30,7 @@ pip install numpy pandas torch
 | Command                        | Description                                            |
 | ------------------------------ | ------------------------------------------------------ |
 | `pnpm run:guestbook`           | Run the interactive guestbook CLI demo                 |
+| `pnpm run:agent-evals`         | Run agent evaluation suites and generate reports       |
 | `pnpm run:name-explorer`       | Explore Finnish name statistics (AI Q&A or stats)      |
 | `pnpm run:scrape-publications` | Scrape publication links and build a review page       |
 | `pnpm run:etf-backtest`        | Run ETF backtest + feature optimizer (requires Python) |
@@ -87,17 +89,29 @@ Notes:
 - `--refresh` forces a refetch; otherwise cached data is reused.
 - Python scripts live in `src/cli/etf-backtest/scripts/`.
 
+## Agent evals
+
+The `run:agent-evals` CLI executes evaluation suites for agents and writes reports under `tmp/agent-evals/` by default.
+
+Usage:
+
+```
+pnpm run:agent-evals -- --suite=example
+pnpm run:agent-evals -- --all
+```
+
 ## Tools
 
 File tools are sandboxed to the `tmp/` directory with path validation to prevent traversal and symlink attacks. The `fetchUrl` tool adds SSRF protections and HTML sanitization, and `runPython` executes whitelisted Python scripts from a configured directory.
 
-| Tool        | Location                                  | Description                                                                    |
-| ----------- | ----------------------------------------- | ------------------------------------------------------------------------------ |
-| `fetchUrl`  | `src/tools/fetch-url/fetch-url-tool.ts`   | Fetches URLs safely and returns sanitized Markdown/text                        |
-| `readFile`  | `src/tools/read-file/read-file-tool.ts`   | Reads file content from `tmp` directory                                        |
-| `writeFile` | `src/tools/write-file/write-file-tool.ts` | Writes content to files in `tmp` directory                                     |
-| `listFiles` | `src/tools/list-files/list-files-tool.ts` | Lists files and directories under `tmp`                                        |
-| `runPython` | `src/tools/run-python/run-python-tool.ts` | Runs Python scripts from a configured scripts directory (JSON stdin supported) |
+| Tool         | Location                                    | Description                                                                    |
+| ------------ | ------------------------------------------- | ------------------------------------------------------------------------------ |
+| `fetchUrl`   | `src/tools/fetch-url/fetch-url-tool.ts`     | Fetches URLs safely and returns sanitized Markdown/text                        |
+| `readFile`   | `src/tools/read-file/read-file-tool.ts`     | Reads file content from `tmp` directory                                        |
+| `writeFile`  | `src/tools/write-file/write-file-tool.ts`   | Writes content to files in `tmp` directory                                     |
+| `listFiles`  | `src/tools/list-files/list-files-tool.ts`   | Lists files and directories under `tmp`                                        |
+| `deleteFile` | `src/tools/delete-file/delete-file-tool.ts` | Deletes files under the `tmp` directory                                        |
+| `runPython`  | `src/tools/run-python/run-python-tool.ts`   | Runs Python scripts from a configured scripts directory (JSON stdin supported) |
 
 `runPython` details:
 
@@ -109,21 +123,37 @@ File tools are sandboxed to the `tmp/` directory with path validation to prevent
 ```
 src/
 ├── cli/
+│   ├── agent-evals/
+│   │   ├── main.ts            # Agent evals CLI entry point
+│   │   ├── README.md          # Agent evals CLI docs
+│   │   ├── constants.ts       # CLI constants
+│   │   ├── types/             # CLI schemas
+│   │   │   └── schemas.ts     # CLI args + suite schemas
+│   │   ├── clients/           # Suite runner + report generator
+│   │   ├── utils/             # Assertion + formatting helpers
+│   │   └── suites/            # Example evaluation suites
 │   ├── etf-backtest/
 │   │   ├── main.ts            # ETF backtest CLI entry point
 │   │   ├── README.md          # ETF backtest docs
 │   │   ├── constants.ts       # CLI constants
-│   │   ├── schemas.ts         # CLI args + agent output schemas
+│   │   ├── types/             # CLI schemas
+│   │   │   └── schemas.ts     # CLI args + agent output schemas
 │   │   ├── clients/           # Data fetcher + Playwright capture
 │   │   ├── utils/             # Scoring + formatting helpers
-│   │   ├── types/             # ETF data types
 │   │   └── scripts/           # Python backtest + prediction scripts
 │   ├── guestbook/
 │   │   ├── main.ts            # Guestbook CLI entry point
-│   │   └── README.md          # Guestbook CLI docs
+│   │   ├── README.md          # Guestbook CLI docs
+│   │   └── types/             # CLI schemas
+│   │       └── schemas.ts     # Guestbook output schema
 │   ├── name-explorer/
 │   │   ├── main.ts            # Name Explorer CLI entry point
-│   │   └── README.md          # Name Explorer CLI docs
+│   │   ├── README.md          # Name Explorer CLI docs
+│   │   └── types/             # CLI schemas + data types
+│   │       ├── ai-output.ts   # Agent output schema
+│   │       ├── index.ts       # Type exports
+│   │       ├── schemas.ts     # CLI args schema
+│   │       └── stats.ts       # Statistics types
 │   └── scrape-publications/
 │       ├── main.ts            # Publication scraping CLI entry point
 │       ├── README.md          # Publication scraping docs
@@ -132,7 +162,8 @@ src/
 │       │   ├── publication-scraper.ts  # Link discovery + selector inference
 │       │   └── review-page-generator.ts # Review HTML generator
 │       └── types/
-│           └── index.ts       # Publication Zod schemas
+│           ├── index.ts       # Publication Zod schemas
+│           └── schemas.ts     # CLI args schema
 ├── clients/
 │   ├── fetch.ts               # Shared HTTP fetch + sanitization
 │   ├── logger.ts              # Shared console logger
@@ -142,6 +173,7 @@ src/
 │   ├── parse-args.ts          # Shared CLI arg parsing helper
 │   └── question-handler.ts    # Shared CLI prompt + validation helper
 ├── tools/
+│   ├── delete-file/           # Delete file tool
 │   ├── fetch-url/             # Safe fetch tool
 │   ├── list-files/            # List files tool
 │   ├── read-file/             # Read file tool

package.json

@@ -8,6 +8,7 @@
     "run:name-explorer": "pnpm -s node:tsx -- src/cli/name-explorer/main.ts",
     "run:scrape-publications": "tsx src/cli/scrape-publications/main.ts",
     "run:etf-backtest": "tsx src/cli/etf-backtest/main.ts",
+    "run:agent-evals": "tsx src/cli/agent-evals/main.ts",
     "scaffold:cli": "tsx scripts/scaffold-cli.ts",
     "node:tsx": "node --disable-warning=ExperimentalWarning --import tsx",
     "typecheck": "tsc --noEmit",
@@ -34,6 +35,7 @@
     "@ianvs/prettier-plugin-sort-imports": "4.7.0",
     "@openai/agents": "0.3.7",
     "@types/jsdom": "27.0.0",
+    "@types/lodash-es": "4.17.12",
     "@types/node": "25.0.6",
     "@types/sanitize-html": "2.16.0",
     "@types/slug": "5.0.9",
@@ -42,6 +44,7 @@
     "eslint-plugin-import": "2.32.0",
     "jiti": "2.6.1",
     "jsdom": "27.4.0",
+    "lodash-es": "4.17.23",
     "marked": "17.0.1",
     "node-html-markdown": "2.0.0",
     "playwright": "1.57.0",