Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 28 additions & 0 deletions packages/evals-core/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
# @a0/evals-core

Core types, configuration, and utilities for the Auth0 eval framework. This package holds the framework's foundational building blocks — shared type definitions, config loading, eval discovery, and scoring primitives — that `@a0/evals` and `@a0/evals-reporter` build on.

Part of the [`auth0-evals`](https://github.com/auth0/auth0-evals) monorepo.

## What it provides

- **Types** — agent/trace types (`AgentType`, `TraceStep`, `TurnMetricEntry`), job results (`JobResult`, `AgentJobResult`, `BaselineJobResult`), scoring types (`RunRecord`, `DimensionScore`), and eval definitions (`EvalDefinition`, `EvalConfig`).
- **Configuration** — `defineConfig` / `loadConfig` for typed `eval.config.js` files, `DEFAULT_FRAMEWORK_CONFIG`, and `deepMerge`.
- **Eval discovery** — `discoverEvals` and `loadEval` for auto-discovering evals from a directory.
- **Utilities** — `logger` / `setLogger`, `withRetry` / `isTransientLlmError`, `estimateCost`, and MCP auth helpers (`mintMcpToken`, `mcpBearerTokenEnvVar`).

## Usage

```ts
import { defineConfig, discoverEvals } from '@a0/evals-core';
import type { EvalDefinition, JobResult } from '@a0/evals-core';

const config = defineConfig({ evalsDir: 'src/evals' });
const evals = await discoverEvals(config.evalsDir);
```

Most consumers use this package indirectly through [`@a0/evals`](https://www.npmjs.com/package/@a0/evals). See the [monorepo README](https://github.com/auth0/auth0-evals) for the full framework guide.

## License

Apache-2.0 © Okta, Inc. See [LICENSE](https://github.com/auth0/auth0-evals/blob/main/LICENSE).
52 changes: 52 additions & 0 deletions packages/evals-graders/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
# @a0/evals-graders

Grader primitives and type definitions for the Auth0 eval framework. A grader is a single pass/fail check run against an eval's output; this package provides the factory functions used to define them in an eval's `graders.ts`.

Part of the [`auth0-evals`](https://github.com/auth0/auth0-evals) monorepo.

## Grader primitives

| Primitive | What it does |
|---|---|
| `contains(needle)` | Substring present in any non-excluded workspace file |
| `notContains(needle)` | Substring must NOT appear in any non-excluded workspace file |
| `notContainsInSource(needle)` | Substring must NOT appear in source files (allowed in config) |
| `matches(pattern)` | Regex match in any non-excluded workspace file |
| `judge(question, framework?)` | LLM-as-judge yes/no question |
| `ranCommand(command, args, description, level)` | Agent ran a shell command containing `command` and all `args` |
| `ranCommandOneOf(commands, description, level)` | Agent ran at least one command from the list |
| `wroteFile(path, description, level, expected?)` | Agent wrote a file whose path contains the substring (optionally asserting content) |
| `compiles(description, level)` | Framework runs the eval's `compile_command` and passes/fails on exit code |
| `calledTool(toolName, description, level)` | Agent invoked an MCP tool whose name contains `toolName` |
| `calledToolOneOf(toolNames, description, level)` | Agent invoked at least one of the named MCP tools |

## Grader levels

`GraderLevel` classifies each grader by what it tests and which configurations it runs in:

| Level | Enum value | Tests |
|---|---|---|
| L1 | `positive_presence` | Required SDK symbols, imports, config keys are present |
| L2 | `hallucination` | Hallucinated packages / wrong SDK variants are absent |
| L3 | `security` | No hardcoded credentials or tokens in insecure storage |
| L4 | `structural` | Code is correctly wired |
| L5 | `version_correctness` | Uses current API, not deprecated patterns |

## Usage

```ts
import { contains, notContainsInSource, judge, GraderLevel } from '@a0/evals-graders';

export const graders = [
contains('@auth0/auth0-react', 'imports the Auth0 React SDK', GraderLevel.positive_presence),
notContainsInSource('client_secret', 'no client secret in source', GraderLevel.security),
// Holistic judge — no level, always runs
judge('Does the app correctly wrap the root with Auth0Provider?'),
];
```

See the [monorepo README](https://github.com/auth0/auth0-evals) for the full framework guide.

## License

Apache-2.0 © Okta, Inc. See [LICENSE](https://github.com/auth0/auth0-evals/blob/main/LICENSE).
34 changes: 34 additions & 0 deletions packages/evals-reporter/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
# @a0/evals-reporter

Report generation and analytics for the Auth0 eval framework. Turns the JSON score files produced by `a0-eval` into a self-contained HTML report, and provides the processing helpers used to group and diff results.

Part of the [`auth0-evals`](https://github.com/auth0/auth0-evals) monorepo.

## What it provides

- **`renderHtml(results, generatedAt)`** — render a full HTML report from an array of job results.
- **Processors** — `loadScores`, `groupResults`, `groupByVariant`, `computeDeltas`, `resultVariant`, and the `MODES` constant for building custom views over results.
- **Nunjucks filters** — `registerFilters` / `ALL_FILTERS` for the report templates.

## Usage

```ts
import { loadScores, renderHtml } from '@a0/evals-reporter';
import { writeFileSync } from 'node:fs';

const results = loadScores(['scores-latest.json']);
const html = renderHtml(results, new Date().toISOString());
writeFileSync('report.html', html);
```

Most consumers generate reports via the CLI instead:

```bash
a0-eval report --input scores-latest.json --output report.html
```

See the [monorepo README](https://github.com/auth0/auth0-evals) for the full framework guide.

## License

Apache-2.0 © Okta, Inc. See [LICENSE](https://github.com/auth0/auth0-evals/blob/main/LICENSE).
Loading