feat(lifecycle): close the self-improve loop end-to-end over the SWE Docker judge#544
Draft
drewstone wants to merge 45 commits into
Draft
feat(lifecycle): close the self-improve loop end-to-end over the SWE Docker judge#544drewstone wants to merge 45 commits into
drewstone wants to merge 45 commits into
Conversation
Root-fix a scoring bug proven live: on django-12419 @ glm-4.6 the agent made the EXACT gold fix but scored 0 because the harness extracted the patch by parsing the model reply for a fenced diff block, and the model wrote prose instead. Two changes, standard SWE-bench practice (SWE-agent/OpenHands read the diff from repo state and pre-stage the checkout): - boxSetup(task): harness clones the instance repo at base_commit into a fixed /work BEFORE the agent shot (same session) so the agent only edits. A stochastic model cannot be trusted to clone to an exact path (observed cannot-change-to-/work failures). Adapter-agnostic seam on BenchmarkAdapter. - boxExtract(): git diff of the agent edits in /work (test files excluded; the judge applies the gold test_patch itself), preferred over the event-stream parse which stays the fallback. Prompt updated: repo pre-staged, agent only edits. Typecheck clean; calibration still green. Live glm-5.2 verification blocked on transient sandbox box-provisioning failures (box unavailable before start on a green health) - re-run when the platform host-agent recovers.
…proposer The code-surface proposer fed the coding agent the ~400-char DISTILLED findings (generationFailureDistiller), not the raw traces. The meta-harness edge (yoonholee.com/meta-harness) is raw-trace filesystem context: the coding agent greps/cats the FULL run traces of failed candidates to diagnose, rather than reading a pre-summary. Add rawTraceDistiller() — an additive analyzeGeneration producer that, instead of summarizing, points the proposer at the prior generation's real run traces already durable on disk under runDir (per-cell spans.jsonl event logs + cached-result.json scores + artifacts) with a grep/cat-to-diagnose instruction. It emits AnalystFinding[] with ABSOLUTE paths (the harness runs from a worktree cwd) so it drops into the same analyzeGeneration slot and renders through the same agenticGenerator prompt path. The existing distiller stays the default. improve() gains a one-line enable: opts.rawTraceContext = true wires rawTraceDistiller() when the caller has not supplied their own analyzeGeneration (explicit analyzeGeneration still wins; null still disables). Self-test builds a real tmp runDir with fake candidate + trace files and asserts the findings reference the actual absolute trace-file paths + the grep/cat instruction, plus worst-candidate-first ordering and the clean-generation fallback. tsc clean, biome clean, 3/3 new tests + 7/7 existing improve tests pass.
# Conflicts: # bench/src/benchmarks/swe-bench.ts # bench/src/run-benchmarks.ts # docs/api/primitive-catalog.md # src/improvement/raw-trace-distiller.test.ts # src/improvement/raw-trace-distiller.ts
…point Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…f-k + self-repair on docstring examples) Selection + repair grounded only on visible docstring >>> examples (doctest, nonce-sentinel), hidden check() grading physically after all harness decisions. Paired bootstrap + exact sign test. Calibrated: 164/164 hidden self-check, 75/164 coverage, 2/75 false-fail (dataset bugs). First result: Llama-3-8B 44.3%->57.3% pass@1 (+13.0pp, CI [+7.8,+18.4], p=1.5e-5, n=164). humaneval.ts: HUMANEVAL_GZ local-cache support (GitHub raw 429s under reps).
…ructural-lever rig Model writes single-line asserts from the prompt alone, BEFORE any candidate exists; asserts run individually in the jail and add to the doctest signal (split counts kept for audit). Extends honest-oracle coverage from 46% of HumanEval toward 100%. Opt-in via TESTGEN=<count>; default off keeps prior runs reproducible.
…all-k-samples-fail bucket
…EN), hidden=remaining asserts Same architecture as hev-structural (nonce judges, phase separation, incremental persistence, bootstrap+sign-test). Calibration: 427/427 hidden self-check, 0/427 visible false-fail, 0 dropped. Official-first scoring: MBPP one-sentence specs make model-guessed asserts ~70% wrong on passing code — shown assert ranks first, guesses break ties (pilot: selection -6.7pp -> +16.7pp).
… only net-new seam; extend strategy family, verifier-environment, field routing
…select + guarded repair (measured +8..+21pp)
…ull result (hev-eval, hev-improve)
…lRollout dials under the held-out gate
…tool-loop conversation runBrainLoop returned on a no-tool-call turn WITHOUT pushing the final assistant message, so ToolLoopResult.messages violated its own contract (seed + every assistant/tool turn): a shot that answers without tools came back with a transcript missing its own answer. Depth continuation then criticized a solution absent from the model's history, and structuralRollout's fenced-code candidate extraction (defaultExtractCandidate's designed fallback) read an empty conversation on non-tool-calling models (Llama-3-8B-Instruct-Lite never tool-calls under tool_choice:auto — measured on Together). Proven by the runtime smoke (bench/src/smoke-structural-rollout.mts): candidates now flow through the strategy on a real non-tool-calling model.
…ory-less consults consultAnalyst rendered EVERY raw consult in analyst framing (instruction as system, 'TASK: ...TRAJECTORY:' as the user turn). With no trajectory — the pre-task case, i.e. structuralRollout's authored-check generation — that user turn is just the coding task, which a weak model answers directly, ignoring the system instruction: measured on Llama-3-8B-Instruct-Lite, authored-assert yield 6/18 reps (0 asserts on the rest) vs 18/18 with the instruction and task fused into one user message (the proven hev-structural rig shape). Trajectory-bearing consults keep the analyst framing unchanged. Runtime smoke after the fix: authored checks on 20/20 HumanEval tasks (was 14/20).
…Eval, hidden grading script-side The ship gate for the ported strategy: runAgentic + structuralRollout(default policy: k=5, repairs<=2, testgen=6) over createVerifierEnvironment with an INERT check, so no hidden-test signal reaches selection/repair; visible checks are the strategy's own model-authored asserts run by the shipped sandboxCheckRunner over a docker --network=none exec channel (the script's thin adapter). The task's own check() suite grades every locked candidate script-side (runChecker) after each rollout finishes. Receipts are hard-verified against the recorded outcomes per candidate. Llama-3-8B-Instruct-Lite (Together), first 20 tasks: blind mean-of-k 63.0% -> selected 80.0% -> final 85.0% (+22pp), authored checks 20/20, 4 tasks rescued (selection or guarded repair passes hidden where sample 0 fails), 0 crashes.
…g (arms A-D, hard/control sets) Implements supervisor-lab PREREG-supervisor-showdown.md: fixed 62-hard/20-control HumanEval sets, equal-compute worker loop (k=5, testgen=6, <=2 repairs), visible-only evidence rendering, no-code plan contract with leak stripping + planLeaked audit, Phase A/B separation with the rig-local nonce hidden judge, per-model token+cost rows.
…glm plan calls exceed 240s under sustained load
…ator Per PREREG-swe-frontier Stage 0: glm-5.2 authors a repro script from the issue text (+ up to 3 requested file reads); validity = nonzero exit on the unpatched tree (1 feedback retry), soundness = exit 0 after the gold patch applied host-side to a COPY of the tree, run in the same :ro-mount jail as the env's run tool. Gold is script-side only, enforced by a leak assert on every outbound message. PYTHONPATH=/testbed pins imports to the mounted tree (script-path sys.path would silently test the image's baked-in install). swe-bench-env: image resolution extracted to exported resolveImageForMetadata (no behavior change).
…e are not a script Observed live on astropy__astropy-13033: the model fenced its READ: lines, extractScript shipped them to the jail, and the NameError crash registered as a false 'bug detected'. A fenced block consisting solely of READ: lines now routes to the read round.
…t the image's built /testbed Stage 0 measured the mount substrate killing 6/23 instances (astropy x2, matplotlib, sklearn x2, pytest) with import errors a fresh un-built clone cannot avoid (compiled extensions / generated version files live only in the image's own /testbed). The same 6 scripts were all valid+sound when executed in-image with the gold applied in-container (writable layer, discarded by --rm; __GOLD_APPLIED__ sentinel separates apply-failure from still-failing). REPRO_EXEC=image makes that substrate a first-class mode.
…ttempts) The 56s total backoff lost 5/23 instances to a sustained zai 429 burst at the tail of a conc-3 run (0 model calls, whole instances recorded as infra-error). Rate-limit retries now climb 30s/60s/120s/240s while transient errors keep the short ladder.
…script extraction Observed live on pylint-dev__pylint-7080: a reply opening with two consecutive ```python lines made the non-greedy fence match capture the empty span between them, discarding a complete script as authoring-failed.
… the grading substrate - canary: with the gold patch applied to the tree under test, import <pkg> must succeed AND resolve INTO that tree (exit 3 = site-packages shadow); a canary failure marks the instance env-unresolvable in that mode before any model call is spent - CANARY_ONLY=1: substrate-trust sweep with zero model calls (image mode also skips the host clone) - mount substrate: gold applied host-side up front, so the canary observes the tree exactly as the soundness run mounts it - 429 ladder raised to 60s/120s/240s over 7 attempts (the overnight code-1305 storm zeroed 6/23 instances on the 30s ladder) Measured on the 23-instance Stage 0 backbone: image canary 23/23 pass, mount canary 17/23 (the 6 compiled-package instances fail, matching the prereg amendment's diagnosis).
…ction + guarded repair vs solo swe-structural.mts runs both PREREG-swe-frontier Stage-1 arms on the cached backbone: ARM=system = canary-asserted image substrate, Stage-0 repro reuse (re-verified per instance: validity without gold, soundness with gold in-container), k independent emit-patch attempts, in-image candidate scoring (git apply to /testbed + repro), argmax (repro-pass first, crash-lowest, first-index ties), <=2 strictly-improving repair rounds, then the official swebench judge serialized in Phase B after every arm decision locks. ARM=solo = one emit-patch attempt, the July-protocol reference. - swe-jail.ts: the calibrator's canary-verified jail/zai-ladder/canary primitives extracted verbatim (swe-repro-calibrate now imports them — one jail, no fork) + assertNoHiddenLeak, the transport-chokepoint judge-separation guard (system/user messages only: assistant/tool text is the model's own or reads of the visible tree). - swe-bench-env.ts: opt-in cloneCache — one GitHub clone per instance, local copies per open, so k-attempt sampling cannot die on a clone flake mid-run. Default off, baseline unchanged. - Rows are incremental (OUT.phaseA then OUT post-judge) and both phases resume by instance id.
…ralRollout dials on real HumanEval with the library's held-out gate The first live campaign on the merged machinery: the 'rollout-policy' improve() surface (deterministic neighbor proposer) drives a REAL evaluator — runAgentic + structuralRollout over an inert verifier surface, visible checks via sandboxCheckRunner on a semaphored docker --network=none channel, script-side hidden grading by the nonce-sentinel judge — with fixed disjoint slices (DEV = HumanEval [0,60), HELD-OUT = [60,120)) passed as explicit budget.holdoutScenarios so the library enforces disjointness and its own defaultProductionGate (paired bootstrap, ship iff CI.low > 0.05) makes the ship/hold call. analyzeGeneration: null keeps the improver's context to DEV composites only. Worker default is Qwen2.5-7B-Instruct-Turbo: the original Meta-Llama-3-8B-Instruct-Lite and every 8B Llama variant were retired from Together serverless (model_not_available, verified 2026-07).
…ow-up on a headroom config Identical improve() rollout-policy loop as live-improve-campaign.mts (HumanEval, DEV saturated at 90%, gate HELD), moved to sanitized MBPP where Qwen2.5-7B measures 76.7% at k5/r2/t6 vs an 85.9% pass@5 bound. Shown assert rides task.meta.visibleChecks into officialChecksFromMeta so the official check outranks authored guesses; test_imports prepended for both judges; DEV = usable index [0,150), HELD-OUT = [150,300); MAX_CALLS hard cap + fail-loud model preflight + recompute-from-raw holdout cross-check added. mbpp-structural.mts gains an entrypoint guard (corpus-replay precedent) and exports basePrompt so the campaign imports the loader without launching the benchmark; direct execution is unchanged.
…mpty-holdout crash
When the worker stops answering mid-run (Together HTTP 402 credit exceeded /
429 rate limit / outage), the runtime swallows exhausted-retry shots as null,
so cells silently degrade (calls/cell 15 -> 3) and the run dies hundreds of
cells later at the holdout with a cryptic 'no scorable cells' gate error.
Now: when a cell produces zero candidates (repairStop=no-candidates, every shot
null), probe the worker directly and abort loud with the real HTTP status
(e.g. 402 credit exceeded) so the operator sees the actual cause immediately.
A healthy probe means the null was a one-off and only that cell fails.
Motivated by a live run: the loop completed all 1350 DEV/gen cells and picked
{k9,r2,t6} (+5.3pp DEV over the 82.0% baseline) but exhausted the Together
credit balance at the holdout phase before the gate could render a verdict.
…river over SWE-bench Verified Contract: supervisor-lab/docs/design/stream-loop.md. Base refactor of the Stage-1 structural pilot (swe-structural.mts @ 393ee50) into a running stream: F (frozen-v0: canary, manifest-repro reuse else fresh authoring, k=4 repro-argmax, <=2 guarded repairs) and L (byte-identical + memory recall via promptAppendix over an initially-empty agent-knowledge store, templated outcome-anchored notes, failure-class tally), interleaved per instance, official judge serialized per instance after arm decisions lock. Hard driver requirements (each traces to a measured Stage-1 failure): - per-instance DEADLINE_MS race (29h hang) -> error rows w/ partial receipts, stream continues; deadline reaches into the zai retry ladder - per-candidate TURN_CAP at the transport chokepoint (160-call blowup) -> synthetic no-tool-call completion, candidate scored as-is, receipted - zai conc <=2 total, 429 ladder 60/120/240s, 480s client timeout - explicit image pull -> run -> delete rotation with a hard presence assert; SWEBENCH_CACHE_LEVEL env override (default env) because the judge's cache_level=env silently pruned the 23-image fingerprint cache Ledger: one JSONL row per (instance x arm) with all structural receipts + recall/write receipts + cumulative resolved/$ per arm; events.jsonl incl. the stubbed every-25 batch-look; append-only persisted plan (stable streamIndex across continuations); resume by instance id. Shared protocol lift: repro-authoring prompt/parsers move from the Stage-0 calibrator into swe-jail (byte-identical), zaiChatRaw gains an optional deadlineAt, IMPORT_NAME covers seaborn/xarray.
…125 (5 attempts, backoff) Two defects the stream's all-error run exposed: (1) the docker-failure path read err.stdout but docker writes exit-125/mount/OCI errors to STDERR, so every infra failure surfaced as a bare 'docker: ' with no cause; (2) a transient daemon window (125) fatal-errored the instance — and since erroring instances burn through instantly with no model calls, one bad window nuked all 23 in ~3 min. Now: stderr captured, and 125/daemon-unreachable/overlay/no-space retried with backoff before being surfaced as infra error.
…r() honors TEMP as the temp DIR TEMP=0.8 (intended as sampling temperature) made os.tmpdir() return '0.8', so mkdtemp built relative paths like '0.8/swe-repro-XXXX' and docker rejected every -v mount as an invalid volume name — the true cause behind the swallowed 'docker: ' error that failed all 23 instances. TEMP still accepted iff it parses < 2 so a stray temp-DIR value can't leak in as a temperature.
Swap the worker to glm-4.5-air (both arms; SWE headroom) via WORKER_MODEL, and
inject at the shared makeTransport chokepoint + the repro-author call the
reasoning-budget knob the zai coding endpoint actually honors — thinking:{type:
enabled} — so arms F and L send byte-identical bodies (REASONING_EFFORT env,
default enabled; symmetry guaranteed by the shared transport). Probe on the live
endpoint: thinking lifts glm-4.5-air reasoning 831->1820 tok on a plain
completion, while reasoning_effort returns 200 with zero lift (silently ignored).
MAX_TOKENS stays 12000 (probe showed no content starvation). Record
maxTokens/reasoningEffort on each row so the worker-budget-identical audit reads
the ledger directly.
…failure gate)
Arm L now fetches a STRONGER model's (SUPERVISOR_MODEL=glm-5.2) no-code diagnosis
before repair, ONCE per instance, gated on a VERIFIED failure: the selected
candidate diff applied AND the gold-verified repro still reports the bug
(applyOk && severity===1). The strength gap over the glm-4.5-air worker is the
arm-C effect; a same-model pair would be the arm-B null.
Primitives (renderRepairEvidence, planContract, supervisorPrompt, stripPlanCode)
are REPLICATED from supervisor-arena.mts (that file runs main() on import, so it
cannot be imported). Evidence is execution-verified / model-visible ONLY — the
issue, the worker's own candidate diff, and the repro-output tail — never
FAIL_TO_PASS, never gold, never worker self-report. The supervisor call is
guarded by assertNoHiddenLeak (a trip HOLDS, never crashes) + the instance
deadline, runs on its OWN token budget (SUPERVISOR_MAX_TOKENS), and its plan is
injected into L's repair appendix; F's appendix is byte-identical (raw failure =
control). Receipts: Row.supervisorPlan {fired, reason, plan, leaked,
groundedOnReproTail, planTokensIn/Out} (populated L, null F) + supervisor-fired/
held events. Supervisor cost is added to L's $ but kept out of the worker token
counters so the worker-budget-identical audit stays clean.
…), not the worker The repro is a shared MEASUREMENT INSTRUMENT — it grades both arms' candidates and defines the supervisor-fire predicate (severity===1) — so authoring it with the weaker glm-4.5-air worker leaks worker weakness into the instrument (observed: a degraded-unsound matplotlib repro that fails on buggy code but the gold patch does not clear). Fresh authoring now runs on REPRO_MODEL (default = SUPERVISOR_MODEL, glm-5.2), the same author as the Stage-0 manifest, so the reused (18) and fresh (5) repros across the fingerprint-23 share one strong instrument. No thinking knob on the author call (Stage-0 authored without one; glm-5.2 reasons at baseline). The Stage-0 soundness gate is unchanged: a degraded-unsound/invalid/timeout fresh repro keeps script=null so it never gates. Worker solve/repair stay glm-4.5-air (the strength gap for the solve-vs-supervise contrast is preserved). Log the sound-repro rate per run; refine held-reason granularity (empty-diff vs apply-failed).
…— the weak worker's dominant failure Smoke (6 inst, glm-4.5-air worker): supervisor fired 0/6 — reasons apply-failed=4, no-repro=2. The gate required severity===1 (patch applied but repro still fails), but glm-4.5-air's dominant failure is severity 3 (git apply rejects the diff — stale context). apply-failed is execution-verified (objective, non-credulous) and stale-diff re-anchoring is exactly where a stronger reviewer helps. Gate now fires on severity 1 OR 3; evidence rendering + supervisor prompt branch on failureKind (wrong-fix vs apply-failed) so the plan targets the real problem.
…uced nothing, the case a plan helps most; hold only already-passing + repro-timeout
…ashboard from a stream dir's ledger/events (reproducible, re-runnable for live state)
…with labeled nodes (role/model/harness/transport), and correct the mental model (best-of-k + supervised-repair, NOT supervisor-spawns-workers)
…SWE Docker judge
Fills the EvalRunner hole (marginal-lift.ts): drive each pinned Verified
instance with the multi-turn atom (runAgentic + refine) under the profile's
prompt, capture the git diff, grade it with the swebench Docker judge held
outside the agent, and return { composite: mean(resolved), costUsd }.
The environment/tasks/judge are injected (bench owns them; importing bench
from src would invert the package dependency). The driven system prompt folds
profile.prompt.instructions in after systemPrompt — the surface applyArtifact
mutates for prompt artifacts — so a prompt candidate measurably changes the
worker instead of scoring a fake zero delta. All-judge-failures throws rather
than reporting a fabricated composite.
bench/src/self-improve-swe.mts wires it end-to-end: promptGenerator (one
deterministic candidate via the injected refine seam) -> runLifecycle ->
thresholdPromotionGate(0) -> composeProfile, with an SMOKE=1 import-only path.
…-loop tool builder Replace the thin builder/author prompts with one shared senior doctrine (src/improvement/optimizer-prompt.ts): diagnose the dominant failure mode, state a hypothesis with a predicted lift, decompose into checkable sub-goals, design to isolate the mechanism, generalize past the shown findings, preserve what works, verify for real, then reflect. Seeded from GEPA's REFLECTION_SYSTEM and the /evolve, /pursue, self-improving-loop skill docs; embedded by toolBuildPrompt / mcpBuildPrompt / defaultBuildPrompt and by authorStrategy's authoring instruction. Add driverLoopGenerator: the driver->worker CandidateGenerator on the shipped atom (runBrainLoop + ToolLoopChat, the same seam driverAgent runs on). A driver LLM authors each worker instruction, observes the session's diff / files / verifier output, rates it, and decides refine / re-scope / decompose - replacing agenticGenerator's canned EMPTY_TREE_NOTE/failureNote respawn. Workers stay runLocalHarness in the candidate worktree; agenticGenerator is kept intact as the offline path. worktreeBuildCandidate wires the driver loop as the tool/mcp build default whenever a driver brain is configured. Completion oracle stays code-owned: after the driver stops, ground truth (dirty tree + raw-trace evidence + verifier exit) decides applied, never the driver's prose. Scripted-brain unit tests prove instruction threading, refine on red verifier, fail-closed gating, and the session cap.
…P into the scorer Phase 3 of the self-improve loop: a worktree-BUILT MCP server (stdio, cwd on the host) was unreachable by every shipped backend — none spawned profile.mcp stdio servers same-host, so a built tool could never be scored live. - connectStdioMcp (runtime/stdio-mcp-client): the ONE persistent newline-JSON-RPC spawn+handshake, extracted from mcpServeVerifier's probe; the verifier is rebased on it so 'verified it serves' and 'served while scored' can never drift. Spawn faults stay McpSpawnFault (setup bug, thrown); crash/timeout stays a failed candidate with the stderr tail. - materializeLocalMcp: spawn every enabled stdio server in profile.mcp, namespace tools <server>__<tool> (provider-safe schemas via the shared sanitizeMcpToolSchema), fail-closed when a declared server cannot boot. - localSandboxClient + resolveSandboxClient backend:'local': the same-host pseudo-box — router-brain tool loop (runBrainLoop) with the profile's MCP tools live; profile arrives per-create on backend.profile; delete() kills the children. Event protocol matches inlineSandboxClient. - sweEvalRunner opts.materializeMcp: overlay the live MCP tools onto the driven surface (tools()/call()) for the eval's duration, close in finally. Default stays prompt-only. - bench self-improve-swe.mts SURFACE=mcp: buildableGenerator over worktreeBuildCandidate (driver-loop brain via routerBrain), ranked/gated by the SAME eval runner with materializeMcp on. Verified: tsc green (root+examples); vitest 1318 passed 0 failed; chain smoke mcpServeVerifier -> applyArtifact -> materializeLocalMcp round-trips a live cwd-bound tools/call.
…sers consume real runs
Phase 4 of the self-improve loop: kill the trace-ingestion rot between what
the loop RECORDS and what its analysts READ.
- campaign-otlp: the missing converter from the campaign trace writer's
per-cell spans.jsonl ({name, cellId, startMs, durationMs, ...attrs}) to the
OTLP-flat JSONL OtlpFileTraceStore/the trace-analyst registry parse, plus
campaignTraceResolver(runDir) — the resolveTraces implementation for
traceAnalystProposer/haloProposer: proposing generation g reads gen-(g-1)
(baseline for g=0) under the same runDir the loop records into. One trace
per cell keyed on the cell's on-disk path (the same cellId recurs across
baseline and every candidate); deterministic FNV folds to 32/16-hex ids.
- findings: isAnalystFinding + toAnalystFindings replace the blind
'as AnalystFinding[]' cast in improvement-driver — real findings pass by
reference, untyped seeds are lifted into makeFinding envelopes (claim = the
curator extraction order, original under metadata.raw), garbage dropped
without throwing. Build prompts can no longer render undefined claims.
- improve(): rawTraceDistiller is now the DEFAULT analyzeGeneration for
durable runs (real runDir — where the traces live); the in-memory digest
distiller stays for mem:// runs and now emits TYPED AnalystFindings too,
so exactly one findings shape crosses the wire. rawTraceContext becomes the
explicit override in either direction.
Proof: vitest drives the real published runCampaign recording -> resolver ->
OtlpFileTraceStore (2 traces indexed, tool names visible) ->
traceAnalystProposer returns a candidate, offline via its analyze/fetch seams.
Converter lives in this repo because agent-runtime-swe consumes the published
agent-eval (0.108.0); lifting it into agent-eval next release is the follow-up.
… via stdio MCP
Adds the 'memory' ArtifactKind + AgentMemorySpec payload. applyArtifact
mounts it twice: the typed spec on profile.metadata.memory (local
extension — the published AgentProfile has no memory field yet) and a
live stdio server entry on profile.mcp, so the same-host client
(materializeLocalMcp) boots the memory during a scored run and
sweEvalRunner({ materializeMcp: true }) measures memory-as-treatment
lift with zero scorer changes.
The memory MCP server (memory_search / memory_get, deterministic
lexical retrieval) serves on the extracted createStdioToolServer core
(committed here; the memory server is its first consumer), with a bin
entry (agent-runtime-memory-mcp) resolved per install: dist/mcp/
memory-bin.js under node when built, the TS source through tsx in
dev/test. Fail-closed: an empty memory never serves; a broken store
fails the materialization instead of faking the ablation.
Lesson ingestion: memoryArtifactFromLessons /
memoryArtifactFromCuratedSurface adapt agent-eval's
memoryCurationProposer block (markers duplicated by convention —
flagged) into memory artifacts; memoryGenerator is the native
CandidateGenerator (observed findings only — judge-derived rows are
firewalled; carried memory accumulates; no-change generations emit
nothing). AgentMemorySpec.logPath writes a JSONL row per memory_search
— the retrieval-log seam for agent-knowledge's RetrievalHoldout, which
stays cross-package (not a dependency of this repo).
Also fixes a stale improvement-driver test: its partial report-finding
fixture now conforms to the P4 toAnalystFindings pass-through contract
instead of asserting a pre-lift id.
…or the research optimizer
- 'connection' ArtifactKind: an ExternalMcpGrant (http url / stdio launch,
literal headers/env, secrets by provider KEY NAME, optional hub grant)
promotes onto profile.mcp[key] via connectionMcpServer and onto
profile.connections (artifact wins per connectionId)
- buildableGenerator: BuiltCandidate.remote branches the mcp emit off the
hardcoded stdio transport — the adopt-not-build path emits a
{ transport:'http', url, headers, env } server with adopt provenance
- KeyProvider (runtime/key-provider.ts): declarative secretEnv
(mcp[key].metadata.secretEnv, names only) resolved at materialize time;
envKeyProvider reads the dotenvx-loaded env; materializeLocalMcp injects
values into the spawned server child env only — fail-closed on a missing
provider/key, values never on the profile or in logs; sweEvalRunner
forwards keys so an adopted server scores with its credential live
- research seam: driverLoopGenerator gains an optional research{query} tool
+ researchDriverNote (adopt-before-build doctrine), offered only when
wired; mcpBuildPrompt instructs adopt-over-build; no live web backend
ships yet (the seam is worktreeBuildCandidate driver.research)
The e2e driver imports sweEvalRunner, which lives in ../src and resolves only through bench/tsconfig path aliases (tsx applies them at cwd=bench). Running tsx bench/src/... from the repo root resolved the published agent-runtime in bench/node_modules and crashed on the missing export. Add a bench package script (self-improve) so the invocation always runs at cwd=bench, and document the requirement in the driver header.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Closes the agent self-improvement loop end-to-end over the real SWE-bench Verified Docker judge, and fills every surface it can optimize: prompt, tools, MCP, memory, external connections.
The one previously-missing piece was the scorer —
EvalRunner(src/lifecycle/marginal-lift.ts) was a type with zero real implementations, sorunLifecycle(the clean generate → measure → promote → compose loop) had no way to produce a number and no real callers. This PR writes that function and everything downstream of it.Base branch
meta/swe-code-surface(38 commits, its own in-flight work). This PR's diff includes those base commits and is not cleanly mergeable tomainyet — it is a draft to surface the self-improve work for review. The 7 self-improve commits are16f98ae2..fb636d71.The 6 phases (+ 1 fix)
16f98ae2sweEvalRunner— drives each Verified instance with the multi-turn driver→worker atom (runAgentic+ refine) under the profile's prompt, captures thegit diff, grades it with the swebench Docker judge held OUTSIDE the agent. Closes the loop.7132f9ae24cfced2aee7d691spans → OTLPso the trace-native proposers consume real runs (stops theunknown[]down-cast).13cb8024memory_search/memory_get); lesson-curation seam + retrieval log for memory-as-treatment.32a3116dfb636d71pnpm --filter ./bench self-improve) so the e2e resolves localsrcinstead of the published dist.Verified at HEAD
pnpm -s typecheck→ exit 0pnpm -s test→ 1372 passed / 0 failed (2 skipped), 139 filesWORKER_MODEL=google/gemini-2.5-flash-lite,django__django-11532, n=1, budget=1): generate → drive (real 1036-byte patch) → grade (official Docker judge, container exec'd, both arms returned a real verdict) → gate → compose all fired.baseline composite=0.0000, candidatescoreDelta=0.0000,promoted=false— the gate correctly held a zero-lift candidate.Honest limits (follow-ups, not blockers)
gemini-flash-litesolves ~nothing ondjango-11532at n=1, so both arms fail and Δ=0. A lift proof needs a stronger worker (glm, currently provider-down) on a non-saturated multi-task holdout.AgentProfile.memory/connectionsextensions ride a local type because the publishedagent-interfacehas no such field yet (source repo absent); thespans→OTLPconverter lives here but belongs in agent-eval; no HTTP MCP client for same-host adoptedhttpgrants yet.Architecture map
The grounded inventory that drove this (what existed, why it couldn't self-improve an agent, where it was dumb, the phased plan) is at
research/…/EXP-037-selfimprove-architecture-search/INVENTORY-one-harness.md.