Skip to content

Document classifier function push #28

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Stephen Belanger (Qard) wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Document classifier function push #28

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 41 additions & 8 deletions docs/features/classifiers.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -125,6 +125,7 @@ When returning structured classifier results, each classification must be a non-
```

Additional field rules:

- `name` defaults to the classifier function's resolved span name when missing/empty (see Name Resolution above).
- `metadata` is an unconstrained `Record<string, unknown>`. SDKs **MUST NOT** impose size limits.
- Duplicate `{name, id}` pairs are allowed. Multiple items with the same `id` under the same name key **MUST** all be stored. Deduplication, if needed, is handled at the display layer.
Expand Down Expand Up @@ -152,6 +153,7 @@ rootSpan.log({ classifications });
Classifier failures **MUST NOT** abort the evaluation or affect other classifiers/scorers.

On failure:

1. Record the error under `classifier_errors` in eval metadata (maps classifier name to error message/stack)
2. Log the error to the root span's metadata
3. **SHOULD** emit a debug warning
Expand All @@ -166,11 +168,11 @@ This mirrors the `scorer_errors` pattern.

The storage format for a single classification. Derived from `Classification` by dropping `name`. `label` remains optional in the wire format.

| Field | Type | Required | Description |
|---|---|---|---|
| `id` | String | **REQUIRED** | Stable identifier for filtering and grouping. |
| `label` | String | **OPTIONAL** | Display label. Consumers **MAY** fall back to `id` when omitted. |
| `metadata` | Record\<string, unknown\> | **OPTIONAL** | Arbitrary metadata. |
| Field | Type | Required | Description |
| ---------- | ------------------------- | ------------ | ---------------------------------------------------------------- |
| `id` | String | **REQUIRED** | Stable identifier for filtering and grouping. |
| `label` | String | **OPTIONAL** | Display label. Consumers **MAY** fall back to `id` when omitted. |
| `metadata` | Record\<string, unknown\> | **OPTIONAL** | Arbitrary metadata. |

### Classifications on Events

Expand All @@ -179,9 +181,7 @@ Stored as a top-level `classifications` field on experiment and log events. **MU
```json
{
"classifications": {
"category": [
{ "id": "greeting", "label": "Greeting" }
],
"category": [{ "id": "greeting", "label": "Greeting" }],
"sentiment": [
{ "id": "positive", "label": "Positive" },
{ "id": "enthusiastic", "label": "Enthusiastic" }
Expand All @@ -192,6 +192,39 @@ Stored as a top-level `classifications` field on experiment and log events. **MU

---

## Function Push

Classifiers that are saved and pushed as project-level functions **MUST** use the same function registration path as scorers, tools, prompts, and other saved functions. SDKs **MUST NOT** emit pushed classifiers through a separate top-level `classifiers` registry.

For code-backed classifiers:

1. The function manifest entry **MUST** be a normal code function entry.
2. The persisted function row **MUST** set `function_type` to `"classifier"`.
3. The persisted function row **MUST** set `function_data.type` to `"code"`.
4. The function body, bundle metadata, runtime context, schema, tags, metadata, and `if_exists` behavior **MUST** follow the same rules as other pushed code functions.

```json
{
"name": "Category classifier",
"slug": "category-classifier",
"function_type": "classifier",
"function_data": {
"type": "code",
"data": {
"type": "bundle",
"runtime_context": {
"runtime": "node"
},
"bundle_id": "bundle-id"
}
}
}
```

Eval-local classifier arrays remain part of the evaluator API described above. They are used for running classifiers during an eval and producing the `classifications` event field. They are distinct from saved function push registries. When an SDK provides a project-level classifier API for function push, it **MUST** register the classifier in the SDK's normal saved function registry with classifier function type metadata.

---

## Examples

### Basic
Expand Down