diff --git a/packages/evals-core/README.md b/packages/evals-core/README.md new file mode 100644 index 00000000..5f28f906 --- /dev/null +++ b/packages/evals-core/README.md @@ -0,0 +1,28 @@ +# @a0/evals-core + +Core types, configuration, and utilities for the Auth0 eval framework. This package holds the framework's foundational building blocks — shared type definitions, config loading, eval discovery, and scoring primitives — that `@a0/evals` and `@a0/evals-reporter` build on. + +Part of the [`auth0-evals`](https://github.com/auth0/auth0-evals) monorepo. + +## What it provides + +- **Types** — agent/trace types (`AgentType`, `TraceStep`, `TurnMetricEntry`), job results (`JobResult`, `AgentJobResult`, `BaselineJobResult`), scoring types (`RunRecord`, `DimensionScore`), and eval definitions (`EvalDefinition`, `EvalConfig`). +- **Configuration** — `defineConfig` / `loadConfig` for typed `eval.config.js` files, `DEFAULT_FRAMEWORK_CONFIG`, and `deepMerge`. +- **Eval discovery** — `discoverEvals` and `loadEval` for auto-discovering evals from a directory. +- **Utilities** — `logger` / `setLogger`, `withRetry` / `isTransientLlmError`, `estimateCost`, and MCP auth helpers (`mintMcpToken`, `mcpBearerTokenEnvVar`). + +## Usage + +```ts +import { defineConfig, discoverEvals } from '@a0/evals-core'; +import type { EvalDefinition, JobResult } from '@a0/evals-core'; + +const config = defineConfig({ evalsDir: 'src/evals' }); +const evals = await discoverEvals(config.evalsDir); +``` + +Most consumers use this package indirectly through [`@a0/evals`](https://www.npmjs.com/package/@a0/evals). See the [monorepo README](https://github.com/auth0/auth0-evals) for the full framework guide. + +## License + +Apache-2.0 © Okta, Inc. See [LICENSE](https://github.com/auth0/auth0-evals/blob/main/LICENSE). diff --git a/packages/evals-graders/README.md b/packages/evals-graders/README.md new file mode 100644 index 00000000..811fce07 --- /dev/null +++ b/packages/evals-graders/README.md @@ -0,0 +1,52 @@ +# @a0/evals-graders + +Grader primitives and type definitions for the Auth0 eval framework. A grader is a single pass/fail check run against an eval's output; this package provides the factory functions used to define them in an eval's `graders.ts`. + +Part of the [`auth0-evals`](https://github.com/auth0/auth0-evals) monorepo. + +## Grader primitives + +| Primitive | What it does | +|---|---| +| `contains(needle)` | Substring present in any non-excluded workspace file | +| `notContains(needle)` | Substring must NOT appear in any non-excluded workspace file | +| `notContainsInSource(needle)` | Substring must NOT appear in source files (allowed in config) | +| `matches(pattern)` | Regex match in any non-excluded workspace file | +| `judge(question, framework?)` | LLM-as-judge yes/no question | +| `ranCommand(command, args, description, level)` | Agent ran a shell command containing `command` and all `args` | +| `ranCommandOneOf(commands, description, level)` | Agent ran at least one command from the list | +| `wroteFile(path, description, level, expected?)` | Agent wrote a file whose path contains the substring (optionally asserting content) | +| `compiles(description, level)` | Framework runs the eval's `compile_command` and passes/fails on exit code | +| `calledTool(toolName, description, level)` | Agent invoked an MCP tool whose name contains `toolName` | +| `calledToolOneOf(toolNames, description, level)` | Agent invoked at least one of the named MCP tools | + +## Grader levels + +`GraderLevel` classifies each grader by what it tests and which configurations it runs in: + +| Level | Enum value | Tests | +|---|---|---| +| L1 | `positive_presence` | Required SDK symbols, imports, config keys are present | +| L2 | `hallucination` | Hallucinated packages / wrong SDK variants are absent | +| L3 | `security` | No hardcoded credentials or tokens in insecure storage | +| L4 | `structural` | Code is correctly wired | +| L5 | `version_correctness` | Uses current API, not deprecated patterns | + +## Usage + +```ts +import { contains, notContainsInSource, judge, GraderLevel } from '@a0/evals-graders'; + +export const graders = [ + contains('@auth0/auth0-react', 'imports the Auth0 React SDK', GraderLevel.positive_presence), + notContainsInSource('client_secret', 'no client secret in source', GraderLevel.security), + // Holistic judge — no level, always runs + judge('Does the app correctly wrap the root with Auth0Provider?'), +]; +``` + +See the [monorepo README](https://github.com/auth0/auth0-evals) for the full framework guide. + +## License + +Apache-2.0 © Okta, Inc. See [LICENSE](https://github.com/auth0/auth0-evals/blob/main/LICENSE). diff --git a/packages/evals-reporter/README.md b/packages/evals-reporter/README.md new file mode 100644 index 00000000..8a32ca26 --- /dev/null +++ b/packages/evals-reporter/README.md @@ -0,0 +1,34 @@ +# @a0/evals-reporter + +Report generation and analytics for the Auth0 eval framework. Turns the JSON score files produced by `a0-eval` into a self-contained HTML report, and provides the processing helpers used to group and diff results. + +Part of the [`auth0-evals`](https://github.com/auth0/auth0-evals) monorepo. + +## What it provides + +- **`renderHtml(results, generatedAt)`** — render a full HTML report from an array of job results. +- **Processors** — `loadScores`, `groupResults`, `groupByVariant`, `computeDeltas`, `resultVariant`, and the `MODES` constant for building custom views over results. +- **Nunjucks filters** — `registerFilters` / `ALL_FILTERS` for the report templates. + +## Usage + +```ts +import { loadScores, renderHtml } from '@a0/evals-reporter'; +import { writeFileSync } from 'node:fs'; + +const results = loadScores(['scores-latest.json']); +const html = renderHtml(results, new Date().toISOString()); +writeFileSync('report.html', html); +``` + +Most consumers generate reports via the CLI instead: + +```bash +a0-eval report --input scores-latest.json --output report.html +``` + +See the [monorepo README](https://github.com/auth0/auth0-evals) for the full framework guide. + +## License + +Apache-2.0 © Okta, Inc. See [LICENSE](https://github.com/auth0/auth0-evals/blob/main/LICENSE).