HeadlessRim Docker: real automated benchmarks + leaderboard infrastructure

[jkbennitt]

Problem

Current --dry-run benchmarks are fake. They run against hardcoded static mock state (3 colonists, 8000 wealth, never changes). Mock LLM returns no_action every tick. Metrics are flat. This tests JSON parse rate and pipeline plumbing, not colony management.

Real benchmarks require a live RimWorld instance — and the leaderboard (#8) requires running 6 scenarios × N models × 4+ runs = hundreds of game sessions with statistical rigor. That's not possible manually.

Solution

HeadlessRim + HeadlessRimPatch + RIMAPI in Docker. Ilya confirmed RIMAPI endpoints work headlessly (HeadlessRim#1) and published HeadlessRimPatch the same day. Save files load via RIMAPI endpoints + Docker mount.

Our existing scoring, evaluation, and paired comparison code works unchanged — just swap mock transport for real HTTP to the containerized game.

Why this is the leaderboard blocker

The leaderboard vision from #8 ("Chatbot Arena for RimWorld") requires:

Requirement	Without Docker	With Docker
Run 96+ game sessions (6 scenarios × 4 models × 4 runs)	Human babysitting a PC for hours	Scripted, unattended
Statistical significance (N=4+ per model)	Impractical manually	Automated with parallel containers
Reproducibility	Different random events per manual session	Same Docker image + save = same start
New model drops, update leaderboard	Redo everything manually	CI triggers full matrix automatically
Community submissions	"Trust us, we ran it"	Publish Docker image, anyone can replicate

Without HeadlessRim, the leaderboard is a one-off manual effort. With it, it's a living automated pipeline.

Architecture

Docker container (HeadlessRim + HeadlessRimPatch + RIMAPI on :8765)
    ↕ REST API + SSE
RLE Python orchestrator (run_benchmark.py)
    ↕ hub-spoke
7 agents (Felix Agent SDK) + any LLM provider
    ↕
Score → paired stats → leaderboard update

Leaderboard pipeline:

for model in [claude, gpt-4o, nemotron, llama, ...]:
  for scenario in [crashlanded, first_winter, ...]:
    for run in range(4):
      docker start → load save via RIMAPI
      run_benchmark.py --model $model
      docker stop
    compute paired stats (agent vs baseline, Cohen's d, p-value)
publish leaderboard

Tasks

Docker image

• Clone HeadlessRimPatch, review Harmony patches
• Build Docker image: HeadlessRim + HeadlessRimPatch + RIMAPI + scenario save files
• Test single scenario run against containerized game
• Benchmark tick speed — can we push speed 4 headlessly?

Benchmark tooling

• Rename --dry-run to --smoke-test (honest about what it measures)
• Add --docker flag to run_benchmark.py (connect to containerized game)
• Multi-container parallel runs for N=4+ statistical significance

CI / leaderboard

• GitHub Actions workflow: spin up container → run full benchmark suite → collect results
• Leaderboard generation from benchmark_history.jsonl
• Publish to HuggingFace Hub (existing infra in run_benchmark.py)

References

• Continues from IlyaChichkov ecosystem integration — HeadlessRim, dashboard, SSE #4 (closed — ecosystem integration complete)
• Unblocks Agents must beat unmanaged baseline #6 (agents must beat baseline — needs real game data)
• Unblocks RLE v1.0: Multi-model colony management leaderboard #8 (multi-model leaderboard — needs automated runs)
• HeadlessRim#1 — Ilya's confirmation + our follow-up

[jkbennitt]

PR #15 landed the modules but deferred the wiring. Working through the remaining 6 items on feature/issue-13-wiring (754e953):

Done so far:

1. EventLog wired — 7 event types emitted throughout run_tick() + _deliberate_agent(), thread-safe via Lock. Smoke test produces 344 events across 6 scenarios.
2. CostTracker wired — token usage pulled from agent._last_usage after each deliberation. cost_snapshot + event_summary in benchmark_summary.json.
3. Process metrics live — ActionResolver.resolve() returns ResolverStats (conflict counts). MetricContext populated with conflicts + CentralPost message tracking. DEFAULT_WEIGHTS redistributed (proportional 20% cut): coordination=0.12, comm_efficiency=0.08. All 6 scenario YAMLs updated.
4. --ablation implemented — 8 passes (full + 7 agent-removed), respects --runs, prints delta matrix, saves ablation_results.json.
5. Weave traces — optional @weave.op() on _call_provider() via enable_weave(). Graceful if not installed.
6. Docker validated — image builds, fixed CRLF bug in entrypoint.sh. Full e2e needs Linux game files at runtime.

All green: mypy strict, ruff, 331 tests, smoke test. Testing more before PR.

[jkbennitt]

Docker e2e validated. 6 scenarios, 10 ticks, Nemotron 120B: avg 0.804 (vs 0.744 non-headless). Full results in results/docker-benchmark/. HeadlessRimPatch PR #6 + socat IPv6 bridge made it work.

[jkbennitt]

Merged in #16. Full infrastructure wired and validated:

• EventLog (7 event types, thread-safe), CostTracker, MetricContext with conflict + message tracking
• coordination (0.12) and communication_efficiency (0.08) weights activated across all 6 scenarios
• Ablation mode (full + 7 agent-removed passes)
• Docker e2e validated with entrypoint fixes (socat supervision, signal traps, privilege dropping)

Final run: 0.844 composite on Crashlanded Survival (10 ticks, Nemotron 120B). Four metrics hit 1.0 — survival, threat_response, wealth, coordination.