High 'crashed' session rate on instawp (43%) — not OOM; launch-failure + tmux-kill mislabeling

[vikasprogrammer]

Surfaced by the fleet-insights skill. instawp: 49/113 sessions (43%) marked crashed, all within the last ~1.5 days of the window.

Investigation (2026-07-10, read-only on the box)

• OOM ruled out. instawp has 15 GiB RAM, 9.8 GiB free, 0 swap used — not memory-constrained. (The 2 GB box is expresstech, a different machine.)
• Bimodal session lifespan (created → crash-swept): p50 = 46 s, p90 ≈ 8 h, max ≈ 1.8 days.
• 45/48 crashes are non-resident (member/automation), 3 resident. The resident idle-reaper correctly marks stopped, so residents aren't the issue.
• Three sampled crashed sessions had no transcript file at all — the headless lane writes the .log before running claude, so a missing log means the pane died before/at launch (or they were interactive/resident, which write no transcript).
• Crash-marks are spread across 07-04 → 07-10, not tightly clustered at restart events → not purely deploy churn.

Mechanism

listSessions sweeps a running row whose tmux pane is gone (no report/markEnded/notify_ended) → crashed. Two populations fit:

1. Short-lived (<1 min): spawns that fail at/near launch (claude auth expired, MCP server startup error, script error) and exit before notify_ended — with no durable breadcrumb, so they're undiagnosable and just read as "crashed".
2. Long tail (hours/days): interactive sessions killed when the deploy recipe runs tmux -S <sock> kill-server (still nukes in-flight sessions even after the KillMode=process fix).

Proposed work

• Launch-failure breadcrumb for interactive/resident lanes (capture the pane's first N lines / an early session.launch.failed audit) so early-death spawns are diagnosable. Prereq for everything else.
• Don't mark a session crashed from an operator tmux kill-server; drain to stopped in the deploy recipe, or detect the kill vs. an abnormal exit.
• Stop polluting the learning loop: a swept crashed with a <Ns lifespan and no work done shouldn't write an outcome:crashed episode (inflates importance via outcomeBoost, skews Dreaming/consolidation).
• Re-measure crash rate after the breadcrumb lands to split launch-failure vs. operator-kill.

Files: src/terminal.ts (listSessions sweep ~L350, writeEpisode), terminal/claude-launch.sh (notify_ended paths), deploy recipe in the deploy memory.

[vikasprogrammer]

Correction — OOM was likely the primary cause (owner context)

The "OOM ruled out / 15 GiB free" reading above was taken after the box was upgraded. Owner confirms instawp was just upgraded 8 GB → 16 GB precisely because it was crashing so much.

So OOM is back as the leading cause: an 8 GB box running 20+ concurrent claude+node+MCP processes (each easily 1–2 GB) would exhaust memory under load. This also explains the short-lived (<1 min, no-transcript) cluster — on a memory-starved box a heavy claude spawn OOM-fails at launch, before notify_ended, leaving only a swept crashed row.

Mitigation already applied: RAM 8→16 GB.

Remaining product work (still worth doing regardless of RAM):

• Verify: re-run fleet-insights in ~3–5 days; expect the crash rate to fall sharply now that OOM pressure is gone. If it doesn't, the launch-failure breadcrumb below is the next step.
• A concurrency cap on simultaneous headless spawns so a fleet burst can't OOM the box again (defense-in-depth beyond just more RAM).
• Launch-failure breadcrumb + stop writing false crashed episodes into the learning loop (as above) — these stand independent of the OOM fix.