From 6c2c4ed19b520dd47abb604aff17a4a63794cbd4 Mon Sep 17 00:00:00 2001 From: Andrew Kent Date: Fri, 5 Jun 2026 15:06:36 -0600 Subject: [PATCH] update remote eval params spec --- docs/features/remote-evals/params/contracts.md | 7 +++---- docs/features/remote-evals/params/design.md | 2 +- docs/features/remote-evals/params/validation.md | 7 ++++--- 3 files changed, 8 insertions(+), 8 deletions(-) diff --git a/docs/features/remote-evals/params/contracts.md b/docs/features/remote-evals/params/contracts.md index 0769d1c..84d80e8 100644 --- a/docs/features/remote-evals/params/contracts.md +++ b/docs/features/remote-evals/params/contracts.md @@ -61,7 +61,7 @@ HTTP 200 OK Content-Type: application/json ``` -Body: a JSON object keyed by evaluator name. For each evaluator, the `parameters` field contains a `parameters` object serialized from the evaluator's `parameters` definition, or `null` if the evaluator defines no parameters. +Body: a JSON object keyed by evaluator name. For each evaluator, the `parameters` field contains a `parameters` object serialized from the evaluator's `parameters` definition. When the evaluator defines no parameters, the field must be **omitted entirely** — do not emit `"parameters": null`. ```json { @@ -87,8 +87,7 @@ Body: a JSON object keyed by evaluator name. For each evaluator, the `parameters } }, "text-summarizer": { - "scores": [], - "parameters": null + "scores": [] } } ``` @@ -101,7 +100,7 @@ Body: a JSON object keyed by evaluator name. For each evaluator, the `parameters | `schema` | `Record` | Map of parameter name to definition | | `source` | `null` | Always `null` for static parameters. Non-null values reference remotely-stored parameter definitions — out of scope for baseline. | -When the evaluator defines no parameters, set `"parameters": null` or omit the field. +When the evaluator defines no parameters, **omit the `parameters` field entirely**. Do not emit `"parameters": null`: the Playground validates each evaluator's `parameters` against an optional (not nullable) zod schema (`serializedParametersContainerSchema.optional()`), so a literal `null` fails parsing and causes the Playground to reject the entire `/list` response with "Failed to parse evaluator list". > **Note for existing SDK implementors**: Prior to the introduction of the container format, some SDKs returned the `schema` map directly (i.e. `Record`) rather than wrapping it in a `parameters` object with `type` and `source` fields. The container was introduced to distinguish static (inline) parameters from dynamic (remotely-stored) ones. If updating an existing SDK, check whether it predates this format and update accordingly. diff --git a/docs/features/remote-evals/params/design.md b/docs/features/remote-evals/params/design.md index 2222276..2968978 100644 --- a/docs/features/remote-evals/params/design.md +++ b/docs/features/remote-evals/params/design.md @@ -112,7 +112,7 @@ For a baseline implementation, supporting `"data"` type (generic values) is suff ### Evaluator parameters are optional -Evaluators that don't define parameters continue to work as before. The `parameters` field in both `GET /list` and `POST /eval` is optional. An evaluator without parameters responds to `GET /list` with `"parameters": null` (or omits the field). +Evaluators that don't define parameters continue to work as before. The `parameters` field is optional in both `GET /list` and `POST /eval`. In the `GET /list` response, an evaluator without parameters must **omit** the `parameters` field entirely (the Playground validates it against an optional, not nullable, schema, so `"parameters": null` fails parsing). In the `POST /eval` request, the client may send `"parameters": null`, `{}`, or omit it — all mean "no overrides". ## Process Flow diff --git a/docs/features/remote-evals/params/validation.md b/docs/features/remote-evals/params/validation.md index 790ddb3..7288cb8 100644 --- a/docs/features/remote-evals/params/validation.md +++ b/docs/features/remote-evals/params/validation.md @@ -21,7 +21,7 @@ This document describes the scenarios and behaviors that an implementation must - Each declared parameter appears in the `"schema"` subfield with its `"default"` and `"description"` preserved - Parameter names match what was declared in the evaluator definition -### 1.2 Evaluator Without Parameters Returns Null +### 1.2 Evaluator Without Parameters Omits the Field **Purpose**: Confirm evaluators that define no parameters don't break the `/list` response. @@ -31,7 +31,8 @@ This document describes the scenarios and behaviors that an implementation must **Expected**: - Response is `200 OK` -- The evaluator's entry has `"parameters": null` or omits the field entirely +- The evaluator's entry **omits** the `"parameters"` field entirely +- The entry must **not** contain `"parameters": null` — the Playground validates this field against an optional (not nullable) schema, so a literal `null` fails parsing and rejects the whole response ### 1.3 Mixed Evaluators in Same `/list` Response @@ -44,7 +45,7 @@ This document describes the scenarios and behaviors that an implementation must **Expected**: - All evaluators appear in the response - Parameter-enabled evaluators have valid `ParametersContainer` structures -- Parameter-free evaluators have `null` or omitted `parameters` +- Parameter-free evaluators omit the `parameters` field (not `null`) ---