SWE-bench eval_limit=500 stalls: all runtime pods stuck in pending, no progress for 45+ minutes

[simonrosenberg]

Summary

SWE-bench ACP-claude eval with eval_limit=500 stalls completely — runtime pods never leave pending status, the orchestrator hangs with no log output for 45+ minutes, and 0/500 instances complete.

This affects ALL benchmarks using SDK >= v1.15.0 (commit b99f15f), not just SWE-bench. Confirmed broken: swebench, swebenchmultimodal, commit0. Only GAIA (binary build) is unaffected.

Details

Triggered run:

• K8s job: eval-23622798964-claude-4-6
• SDK run: 23622781040
• Eval run: 23622798964
• CID: 2A6EC49A
• Params: benchmark=swebench, model=claude-4.6-opus, agent_type=acp-claude, eval_limit=500, benchmarks_branch=fix/copy-python-runtime-for-eval-images

Other affected runs (same root cause):

Job	Benchmark	Branch	Status
eval-23622889791-claude-son	swebenchmultimodal	fix/swebenchmultimodal-api-timeout	All 5 instances CreateContainerError
eval-23624504367-claude-son	swebenchmultimodal	fix/swebenchmultimodal-api-timeout	All 5 instances CreateContainerError
commit0 (deleted)	commit0	unknown	All instances failed, job deleted

Concurrent GAIA run (same SDK) is progressing fine — eval-23622799654-claude-4-6 uses b99f15f-gaia-binary images (different build path, not affected).

Observed Behavior

1. Job starts, image build succeeds, 8 instances are launched at 00:01:28 UTC
2. All 8 instances immediately fail with Runtime not yet ready (status: pending) — runtime_id=None, session_id=None
3. Retries also fail with same error (32 total failures across 8 instances, all on retry 2-3)
4. Last log line at 00:21:52 UTC — no output for 45+ minutes after that
5. Progress bar stuck at 0/500
6. runtime-pods namespace shows no pods at all: No resources found in runtime-pods namespace
7. Orchestrator pod itself is healthy (Running, 1/1 READY)

Root Cause

SDK v1.15.0 broke all source-minimal eval images. PR OpenHands/software-agent-sdk#2567 changed the agent-server Dockerfile builder from --managed-python (self-contained venv) to --python-preference only-system (venv symlinks to system Python). The eval-base images don't have Python 3.13 at /usr/local/bin/, so the symlink is broken.

Every source-minimal runtime pod fails with:

CreateContainerError: exec: "/agent-server/.venv/bin/python":
stat /agent-server/.venv/bin/python: no such file or directory

Pod status logger confirms: 10 out of 13 runtime pods are in CreateContainerError (all source-minimal), while 3 GAIA pods (-binary build) run fine.

PR #578 fix is not sufficient. The Dockerfile.agent-layer changes are present on both branches (fix/copy-python-runtime-for-eval-images and fix/swebenchmultimodal-api-timeout), builds succeed, but pods still crash. The validation only tested 1 instance (sympy__sympy-23824). All other instances remain broken, likely because:

1. Stale images cached in GHCR from pre-fix builds (tag b99f15f-* already existed), or
2. Python 3.13 is not at /usr/local/bin/python3.13 in the builder (the assumed COPY path may be wrong)

Contributing factors

• Cluster resource exhaustion: 4 concurrent evals saturate the 23-node runtime cluster (17/23 insufficient CPU, 19/23 insufficient memory)
• Warm runtime pool broken: warm_runtimes.py crashes with relation "runtimes" does not exist — all sessions cold-start
• Orchestrator hangs silently after retries exhaust instead of failing fast

Suggested Fix

Step 1: Verify the Python path in the builder image

docker run --rm ghcr.io/openhands/eval-builder:b99f15f ls -la /usr/local/bin/python*
docker run --rm ghcr.io/openhands/eval-builder:b99f15f readlink -f /agent-server/.venv/bin/python

If Python is not at /usr/local/bin/python3.13, update the COPY paths in Dockerfile.agent-layer to match the actual location.

Step 2: Force-rebuild images (purge GHCR cache)

The b99f15f-* image tags in GHCR are stale (built before the Dockerfile fix). Either:

• Delete the stale tags from ghcr.io/openhands/eval-agent-server and re-trigger builds, or
• Use a new SDK commit so the image tags are fresh (e.g. abc1234-sweb.eval.* has never been built, guaranteeing no cache hit)

Step 3: Merge the corrected fix to main

Once verified, merge PR #578 (with correct COPY paths) to main so ALL benchmark branches inherit the fix. Currently every feature branch needs the fix independently.

Step 4: Address cluster saturation (separate issue)

• Add autoscaling or resource limits for concurrent eval jobs
• Run the missing database migration for the runtimes table to restore the warm runtime pool
• Add a circuit-breaker in the orchestrator to fail fast instead of hanging silently after retries exhaust

Timeline (UTC, 2026-03-27)

Time	Event
00:00:29	SWE-bench eval starts, 10 workers
00:01:15	prune_db.py crashes: relation "runtimes" does not exist
00:01:28	8 instances dispatched to runtime-api
00:01:32	warm_runtimes.py crashes: same DB error
00:11:32	All 8 instances timeout: Runtime not yet ready (status: pending), runtime_id=None
00:21:52	Retry round 2 also fails; orchestrator goes silent after this

[simonrosenberg]

Root Cause Analysis

Investigation traced the failure through K8s pod status logs, runtime-api logs, Datadog, and the evaluation job logs. There are two compounding root causes, plus an amplifying factor.

Root Cause 1: Broken SWE-bench eval images (CreateContainerError)

SDK v1.15.0 (PR OpenHands/software-agent-sdk#2567) changed the agent-server Dockerfile builder from --managed-python to --python-preference only-system, making the venv's bin/python a symlink to /usr/local/bin/python3.13 instead of a self-contained managed Python. The eval-base images don't have Python 3.13 at that path, so the symlink is broken.

Every SWE-bench runtime pod fails immediately with:

CreateContainerError: exec: "/agent-server/.venv/bin/python":
stat /agent-server/.venv/bin/python: no such file or directory

Pod status logger snapshot confirms 6 out of 10 active runtime pods stuck in CreateContainerError, all using b99f15f-sweb.eval.x86_64.*-source-minimal images. GAIA pods (b99f15f-gaia-binary) are unaffected and running fine.

Fix: PR #578 correctly addresses this by copying the Python 3.13 runtime from the builder stage.

Root Cause 2: Runtime cluster resource exhaustion (Unschedulable)

The 23-node eval-runtime cluster is fully saturated by concurrent eval workloads. Pod status logger shows:

0/23 nodes available: 17 Insufficient cpu, 19 Insufficient memory,
3 node(s) didn't match Pod's node affinity/selector,
8 Insufficient ephemeral-storage

At the time of this run, 4 eval jobs were running simultaneously:

• eval-23622798964-claude-4-6 — swebench, eval_limit=500 (this job)
• eval-23622799654-claude-4-6 — GAIA, 165 instances
• eval-23622889791-claude-son
• eval-23624504367-claude-son

This is why the validation run in #578 (eval_limit=1, idle cluster) succeeded but this 500-instance run failed — the fix is correct but the validation didn't test at scale.

Additionally, transient PVC provisioning races cause pods to initially fail with persistentvolumeclaim "runtime-*" not found before eventually being scheduled.

Amplifying Factor: Warm runtime pool is broken

The runtime-api's warm_runtimes.py and prune_db.py are both crashing with:

relation "runtimes" does not exist

All 3 runtime-api pods log Found 0 warm runtimes in the database and No warm runtimes found for API key mamoodi-eval for every session. This forces all sessions to cold-start, adding scheduling pressure to an already saturated cluster.

Why GAIA works but SWE-bench doesn't

	GAIA	SWE-bench
Image	b99f15f-gaia-binary	b99f15f-sweb.eval.x86_64.*-source-minimal
Python venv	Present (different build)	Missing (broken symlink)
Pod status	Running	CreateContainerError
eval_limit	165	500 (higher cluster pressure)

Timeline (UTC, 2026-03-27)

Time	Event
00:00:29	SWE-bench eval starts, 10 workers
00:01:15	prune_db.py crashes: relation "runtimes" does not exist
00:01:28	8 instances dispatched to runtime-api
00:01:32	warm_runtimes.py crashes: same DB error
00:11:32	All 8 instances timeout: Runtime not yet ready (status: pending), runtime_id=None
00:21:52	Retry round 2 also fails; orchestrator goes silent after this

Recommendations

1. Merge fix: copy Python 3.13 runtime into eval images for venv compatibility #578 — fixes the CreateContainerError for eval images
2. Run the missing database migration for the runtimes table to restore the warm runtime pool
3. Limit concurrent eval jobs or add cluster autoscaling — 4 concurrent evals with 500+ total instances overwhelm 23 nodes
4. Add a timeout/fail-fast in the orchestrator — currently hangs silently after retries exhaust instead of reporting failure

[simonrosenberg]

The core issue was fixed by OpenHands/software-agent-sdk#2656