Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,12 @@ All notable changes to `@tangle-network/agent-eval` and its sibling `agent-eval-

---

## Unreleased

### Breaking

- Removed `@tangle-network/agent-eval/primeintellect`, which generated a legacy single-turn Verifiers package with unsafe substring scoring. Use `@tangle-network/agent-runtime/primeintellect` for Verifiers v1 task packaging, real runtime execution, and full trace import.

## [0.119.1] — 2026-07-14 — portable improvement evidence

### Added
Expand Down
26 changes: 0 additions & 26 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -95,31 +95,6 @@ const traceRuns = fromOtelSpans({ spans: yourOtelSpans })
await analyzeRuns({ runs: [...runs, ...traceRuns], raterScores })
```

### 4. Export to PrimeIntellect

Use this when you want PrimeIntellect Verifiers to run or train against tasks produced by Tangle evals, runtime runs, or knowledge runs.

```ts
import {
primeIntellectRowsFromRunRecords,
writePrimeIntellectEnvironmentPackage,
} from '@tangle-network/agent-eval/primeintellect'

const rows = primeIntellectRowsFromRunRecords({
records, // RunRecord[] with scenarioId
scenarios, // prompts plus answer or requiredSubstrings
})

await writePrimeIntellectEnvironmentPackage('prime-envs/refund-policy', {
name: 'refund-policy',
rows,
runRecords: records,
})
```

The generated package contains `load_environment()`, `data/dataset.jsonl`, `data/run_records.jsonl`, and a manifest.
Run it with `uv run vf-eval refund-policy` or upload it with `prime env push`.

---

## Core concepts
Expand Down Expand Up @@ -149,7 +124,6 @@ Each example: `README.md` + a single `index.ts` runnable via `pnpm tsx`. Prints
|---|---|
| `…/contract` | **The headline, frozen surface — new code starts here.** `defineAgentEval`, `selfImprove`, `analyzeRuns`, `runEval`, `runCampaign`, `runImprovementLoop`, `diffRuns`; intake adapters (`fromFeedbackTable`, `fromOtelSpans`); proposers (`gepaProposer`, `evolutionaryProposer`); gates (`defaultProductionGate`, `heldOutGate`, `paretoSignificanceGate`, `neutralizationGate`, `composeGate`); the deployment-outcome store; storage; and the five core types `Scenario` / `Dispatch` / `JudgeConfig` / `SurfaceProposer` / `Gate`. |
| `…/hosted` | `createHostedClient` / `hostedClientFromEnv` + the wire types to ship eval-run events + trace spans to a hosted orchestrator (ours or your own implementation of the spec) |
| `…/primeintellect` | Export validated `RunRecord`s plus scenario prompts into a PrimeIntellect Verifiers package: Python `load_environment()`, `data/dataset.jsonl`, `data/run_records.jsonl`, and export manifest |
| `…/adapters/otel` | `createOtelBridge` — forwards OpenTelemetry-shape spans into the hosted-tier ingest, no `@opentelemetry/*` dependency |
| `…/adapters/langchain` | Wrap any LangChain `Runnable` as a `Dispatch` (or `JudgeConfig`), no `@langchain/core` peer dep |
| `…/adapters/http` | `httpDispatch` + `runDispatchServer` — run a campaign's worker on another machine (multi-region, remote worker execution) |
Expand Down
5 changes: 0 additions & 5 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -119,11 +119,6 @@
"import": "./dist/hosted/index.js",
"default": "./dist/hosted/index.js"
},
"./primeintellect": {
"types": "./dist/primeintellect/index.d.ts",
"import": "./dist/primeintellect/index.js",
"default": "./dist/primeintellect/index.js"
},
"./openapi.json": {
"default": "./dist/openapi.json"
}
Expand Down
1 change: 0 additions & 1 deletion scripts/build-entries.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,5 @@ export const buildEntries = {
'belief-state/index': 'src/belief-state/index.ts',
'contract/index': 'src/contract/index.ts',
'hosted/index': 'src/hosted/index.ts',
'primeintellect/index': 'src/primeintellect/index.ts',
cli: 'src/cli.ts',
}
Loading
Loading