Create benchmark save files for all 6 scenarios

[jkbennitt]

Context

The paired benchmark (agent vs baseline) requires reproducible initial conditions via named save files. Each scenario needs a RimWorld save at a specific colony state. rle_crashlanded_v1 exists — 5 more needed.

Save Files to Create

All saves start from a Crashlanded scenario, Cassandra Classic, Adventure Story difficulty. Use dev mode to advance time and trigger events.

1. rle_crashlanded_v1 — DONE

• Day 1, 3 colonists, default Crashlanded start
• Already created and tested (paired benchmark delta: -0.029)

2. rle_first_winter_v1

• Advance to day 30 (approaching fall/winter)
• Colony should have basic shelter, some food stored, a few research projects done
• Dev mode: Development > Date > Set day or just fast-forward
• Save when the season is about to change

3. rle_toxic_fallout_v1

• Advance to day 10, stable colony
• Trigger toxic fallout: Dev mode > Debug actions > Incidents > Execute incident > ToxicFallout
• Save immediately after the fallout starts (green overlay visible)
• Tests: can agents keep colonists indoors, manage food, survive the event?

4. rle_raid_defense_v1

• Advance to day 15, build some walls/sandbags
• Trigger a raid: Dev mode > Debug actions > Incidents > Execute incident > RaidEnemy
• Save right before or as the raid spawns
• Tests: can DefenseCommander draft colonists and position them?

5. rle_plague_response_v1

• Advance to day 10, have some medicine stockpiled
• Trigger plague: Dev mode > Debug actions > Incidents > Execute incident > Plague
• Save immediately after plague hits (colonists should show "plague" hediff)
• Tests: can MedicalOfficer triage, assign bed rest, administer medicine?

6. rle_ship_launch_v1

• Advance to day 60 with significant research progress
• Complete several research projects via dev mode: Debug actions > Research > Finish project
• Have 5+ colonists (use Debug actions > Spawn pawn > Colonist)
• Save with a mid-game colony that has resources + tech to attempt ship building
• Tests: long-horizon planning, research prioritization, resource management at scale

How to Create Each Save

1. Load rle_crashlanded_v1 (base save)
2. Enable dev mode: Options > check "Development mode"
3. Use dev tools to advance time / trigger events per scenario above
4. Save as the exact name listed (e.g. rle_first_winter_v1)
5. Verify: curl http://localhost:8765/api/v1/game/state shows expected tick/colonist count

How to Verify Saves Work with Benchmark

# Test save/load roundtrip
python -c "
import asyncio
from rle.rimapi.client import RimAPIClient
async def test():
    async with RimAPIClient('http://localhost:8765') as c:
        await c.load_game('rle_first_winter_v1')
        import time; await asyncio.sleep(3)
        state = await c.get_game_state()
        print(f'Day: {state.colony.day}, Pop: {state.colony.population}')
asyncio.run(test())
"

Priority

Do rle_first_winter_v1 and rle_raid_defense_v1 first — these are the scenarios most likely to show agent value (agents managing food/shelter before winter, agents drafting defenders during raids). The others can wait.

@CalebisGross — if you have RimWorld installed you can create some of these too. Just name the saves exactly as listed.

[jkbennitt]

Progress Update

PR #17 merged — all 6 save files now exist in docker/saves/ and load correctly.

What's done

• scripts/create_scenario_saves.py generates all 5 saves programmatically from the crashlanded base via XML manipulation (no manual dev mode needed)
• Difficulty patched per scenario (Medium, Rough, Hard)
• Cloned colonists via XML pawn cloning for 4-5 pop scenarios (Val Kowalski, Jin Tanaka)
• Scenario-specific items spawned (weapons, medicine, resources, etc.)
• unforbid_all_items() fixed to handle RIMAPI's new DDD response format
• All saves verified in-game: correct colonist counts, clones controllable, items present
• CI passes (lint, mypy, 337 tests, smoke test all 6 scenarios)

What still needs done

The saves are all day-0 crashlanded starts. The issue spec calls for advanced colony states:

• first_winter — advance to day 30, should have shelter/food/research
• toxic_fallout — advance to day 10, trigger ToxicFallout incident, save with event active
• raid_defense — advance to day 15, build defenses, trigger RaidEnemy, save with raid in progress
• plague_response — advance to day 10, stockpile medicine, trigger Plague, save with plague hediffs active
• ship_launch — advance to day 60, complete research projects, mid-game colony state

This requires either manual dev mode work or using RIMAPI endpoints we discovered are available but not wired in our client yet:

• POST /api/v1/pawn/spawn — spawn colonists
• POST /api/v1/item/spawn — spawn items
• POST /api/v1/incident/trigger — trigger raids, toxic fallout, plague on demand
• POST /api/v1/pawn/edit/skills, edit/traits, edit/health — customize pawns

Wiring these endpoints into RimAPIClient would let us script the full save creation pipeline — advance time, build up the colony, trigger scenario events, then save. No manual dev mode needed.

[jkbennitt]

Status update — PR #18 progress + remaining work

PR #18 (feature/rimapi-full-integration) delivers the client-side infrastructure for save creation but does NOT fully close this issue. Worth pausing to enumerate what's done vs outstanding.

What PR #18 added

• Client methods: spawn_pawn, spawn_item, send_drop_pod, change_weather, trigger_incident, edit_pawn_skills, edit_pawn_traits, edit_pawn_health, edit_pawn_needs
• SetupCommand in ScenarioConfig so YAMLs can declare spawn commands that run_scenario.py dispatches at scenario start
• TriggeredIncident in ScenarioConfig so scenarios fire their signature events at specific ticks (wired into the game loop)
• Rewrote create_scenario_saves.py — replaced 200+ lines of XML regex with declarative RIMAPI calls
• All 5 advanced saves rebuilt live and committed

What the 5 saves actually contain now (vs the issue spec)

Save	✅ Has	❌ Still missing
rle_first_winter_v1	Medium difficulty	Not advanced to day 30; no shelter/food/research buildup
rle_toxic_fallout_v1	+1 pawn, wood/meals/steel, ToxicFallout active	Not advanced to day 10 stable colony first
rle_raid_defense_v1	+2 pawns, weapons, flak vests, resources	Not advanced to day 15; no walls/sandbags built
rle_plague_response_v1	+2 pawns, medicine, meals (Plague fires at scenario runtime)	Not advanced to day 10; plague hediffs not pre-applied
rle_ship_launch_v1	+2 pawns, high-tier resources (plasteel/gold/uranium)	Not advanced to day 60; research still mostly incomplete

Why we can't advance to day N

The original issue spec ("advance to day 30," "day 60," etc) requires either fast-forwarding the game clock or running the colony for many in-game days. Neither is currently possible via RIMAPI:

• No tick-advance endpoint exists. Unity processes ticks at natural game speed, and the HTTP server just queues work on the main thread. We can set game speed to 3 but we'd still need to actually wait for thousands of real-time seconds.
• set_research_target(force=True) doesn't complete research — confirmed during live testing. It only queues the project; the game still needs many ticks to finish it. Documented in create_scenario_saves.py so future sessions don't rediscover this trap.

Paths forward to actually close #7

1. Upstream feature request: advance-ticks endpoint. Something like POST /api/v1/dev/advance-ticks with a count param that blocks until Unity has processed N ticks. Cleanest solution. To be useful for save construction it would need to pair with something like "god mode" / "no need drops" / "no storyteller incidents" — otherwise the colony dies during advancement (see Path 2 below). Worth filing in IlyaChichkov/RIMAPI.
2. Orchestrated real-time play — doesn't work standalone. My first pass suggested "unpause, wait, save" but this is broken: at speed 3, reaching day 60 takes ~20 minutes of wall-clock time, and an unattended colony will starve, get raided, or break down long before then. Only works if either (a) RIMAPI exposes a way to suspend needs/storyteller during advancement, or (b) we run our own agents to manage survival during the advance — which is essentially "run the benchmark" and produces a save reflecting agent quality, not a clean scenario.
3. Manual dev-mode setup. Use RimWorld's built-in dev-mode Advance day cheat to push the clock, then run our new create_scenario_saves.py --only <name> for the scenario-specific item/pawn/incident additions. Tedious but works today with no upstream changes. Worth using as a one-time fix if we want the benchmarks today.
4. Accept day-0 snapshots and scope down the issue. The benchmarks still stress-test agents via triggered incidents + item/pawn setup. Close this issue as "done for what's achievable without upstream help" and rename first_winter to something that reflects the actual day-0 state. Lose coverage of the "agents inherit a built colony" case.

My recommendation is (1) + (3) as a two-track: file the upstream request for the proper long-term fix, use manual dev mode in the meantime to get the day-30/60 saves. (4) is a last resort if upstream won't move on (1).

Related recent changes (memory / docs)

• create_scenario_saves.py rewrite is documented in CLAUDE.md "Regenerating scenario saves"
• RIMAPI async-write gotchas documented in CLAUDE.md "RIMAPI gotchas" (save_game flush delay, spawn_item can't split stacks, null-ref cascades)
• .claude/rules/rimapi.md updated with the write-is-async rule