fix(session): reap unattended runs that end via report (v0.141.1)#241
Merged
Conversation
An automation/cron/task agent that finishes by calling `report` flips its session row to `done` mid-turn. By the time the turn-end Stop beacon reached `markTurnIdle`, the status was already terminal, so teardown bailed on `status !== 'running'` and the interactive TUI pane kept running forever — dozens of orphaned tmux panes + claude processes piled up on the live fleet (some 20h+ old, all belonging to `done` sessions; `session.reaped` had fired only once). - markTurnIdle: accept `done` as well as `running` for unattended, non-resident runs, and reap a still-alive pane (honoring claimed / attached / blocked). A liveness poll skips an already-dead pane so a stray second beacon can't re-reap. - reapIdleSessions backstop: also sweep `done` unattended rows whose pane is still alive (self-heals orphans predating the fix / lost Stop beacons); only ever acts on a genuinely-alive pane, so a cleanly-reaped row is never re-killed/re-audited, and nothing outside term_sessions (non-agent-os tmux sessions) is ever touched. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…npollable The backstop's 'done'-orphan sweep needs a tmux liveness poll, but aliveNames() returns null on the Linux LauncherSessionBackend (and transient local failures). Gating the whole sweep on a non-null poll would regress running-straggler reaping on launcher backends. Fall back to the classic time-based running-only rule when liveness can't be polled; sweep 'done' orphans only when it can (per docs/concurrency-cap-plan.md, which flagged the aliveNames()===null fail-open). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Background sessions spawned by automations/cron/tasks don't stop after they finish. On the live instawp tenant: 31 orphaned tmux sessions + 61 lingering claude processes, some 20h+ old, every one belonging to a session the DB already marks
done.session.reapedhad fired once ever.Root cause
The fleet's automation prompts all end with "Finish with
report".report(terminal.ts) setsstatus='done'but does not kill the tmux pane. The unattended run is an interactive, attachable TUI (notclaude -p), so its only teardown is the turn-end Stop hook →/api/turn-idle→markTurnIdle. ButmarkTurnIdlebailed on:By the time the Stop beacon lands,
reporthas already flipped the row todone, so it returns early and never kills the pane (proof: these rows havelast_activity = null— it bailed before the stamp). The idle backstop only sweptstatus='running'rows with non-nulllast_activity, so nothing reaped them. The TUI sits idle forever. It also made the session list show adonerun as "live" (green), sinceisLive()keys off the still-alive pane.Fix
markTurnIdlenow acceptsdoneas well asrunningfor unattended, non-resident runs and reaps a still-alive pane (still honoring claimed / attached / blocked-on-human). A liveness poll skips an already-dead pane so a stray second beacon can't re-reap.reapIdleSessionsbackstop also sweepsdoneunattended rows whose pane is still alive — self-heals orphans that predate the fix or whose Stop beacon never landed. It only ever acts on a genuinely-alive pane (so a cleanly-reaped row is never re-killed/re-audited), and is keyed entirely offterm_sessionsrows, so tmux sessions not spawned through agent-os are never touched, and member interactive (headless=0) / chat resident (resident=1) sessions are excluded.Net: a finished background run stops for real, and the session list correctly reads "done".
Verification
npm run typecheck✓ ·npm run build✓ ·npm run test:governance→ 68/68 ✓done-orphan panes on the next 60s tick.🤖 Generated with Claude Code