Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 3 additions & 4 deletions docs/features/remote-evals/params/contracts.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ HTTP 200 OK
Content-Type: application/json
```

Body: a JSON object keyed by evaluator name. For each evaluator, the `parameters` field contains a `parameters` object serialized from the evaluator's `parameters` definition, or `null` if the evaluator defines no parameters.
Body: a JSON object keyed by evaluator name. For each evaluator, the `parameters` field contains a `parameters` object serialized from the evaluator's `parameters` definition. When the evaluator defines no parameters, the field must be **omitted entirely** — do not emit `"parameters": null`.

```json
{
Expand All @@ -87,8 +87,7 @@ Body: a JSON object keyed by evaluator name. For each evaluator, the `parameters
}
},
"text-summarizer": {
"scores": [],
"parameters": null
"scores": []
}
}
```
Expand All @@ -101,7 +100,7 @@ Body: a JSON object keyed by evaluator name. For each evaluator, the `parameters
| `schema` | `Record<string, parameter>` | Map of parameter name to definition |
| `source` | `null` | Always `null` for static parameters. Non-null values reference remotely-stored parameter definitions — out of scope for baseline. |

When the evaluator defines no parameters, set `"parameters": null` or omit the field.
When the evaluator defines no parameters, **omit the `parameters` field entirely**. Do not emit `"parameters": null`: the Playground validates each evaluator's `parameters` against an optional (not nullable) zod schema (`serializedParametersContainerSchema.optional()`), so a literal `null` fails parsing and causes the Playground to reject the entire `/list` response with "Failed to parse evaluator list".

> **Note for existing SDK implementors**: Prior to the introduction of the container format, some SDKs returned the `schema` map directly (i.e. `Record<string, parameter>`) rather than wrapping it in a `parameters` object with `type` and `source` fields. The container was introduced to distinguish static (inline) parameters from dynamic (remotely-stored) ones. If updating an existing SDK, check whether it predates this format and update accordingly.

Expand Down
2 changes: 1 addition & 1 deletion docs/features/remote-evals/params/design.md
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,7 @@ For a baseline implementation, supporting `"data"` type (generic values) is suff

### Evaluator parameters are optional

Evaluators that don't define parameters continue to work as before. The `parameters` field in both `GET /list` and `POST /eval` is optional. An evaluator without parameters responds to `GET /list` with `"parameters": null` (or omits the field).
Evaluators that don't define parameters continue to work as before. The `parameters` field is optional in both `GET /list` and `POST /eval`. In the `GET /list` response, an evaluator without parameters must **omit** the `parameters` field entirely (the Playground validates it against an optional, not nullable, schema, so `"parameters": null` fails parsing). In the `POST /eval` request, the client may send `"parameters": null`, `{}`, or omit it — all mean "no overrides".

## Process Flow

Expand Down
7 changes: 4 additions & 3 deletions docs/features/remote-evals/params/validation.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ This document describes the scenarios and behaviors that an implementation must
- Each declared parameter appears in the `"schema"` subfield with its `"default"` and `"description"` preserved
- Parameter names match what was declared in the evaluator definition

### 1.2 Evaluator Without Parameters Returns Null
### 1.2 Evaluator Without Parameters Omits the Field

**Purpose**: Confirm evaluators that define no parameters don't break the `/list` response.

Expand All @@ -31,7 +31,8 @@ This document describes the scenarios and behaviors that an implementation must

**Expected**:
- Response is `200 OK`
- The evaluator's entry has `"parameters": null` or omits the field entirely
- The evaluator's entry **omits** the `"parameters"` field entirely
- The entry must **not** contain `"parameters": null` — the Playground validates this field against an optional (not nullable) schema, so a literal `null` fails parsing and rejects the whole response

### 1.3 Mixed Evaluators in Same `/list` Response

Expand All @@ -44,7 +45,7 @@ This document describes the scenarios and behaviors that an implementation must
**Expected**:
- All evaluators appear in the response
- Parameter-enabled evaluators have valid `ParametersContainer` structures
- Parameter-free evaluators have `null` or omitted `parameters`
- Parameter-free evaluators omit the `parameters` field (not `null`)

---

Expand Down