Skip to content

fix(orchestrator): Preserve post-interrupt recovery state#4229

Draft
mwolson wants to merge 2 commits into
pingdotgg:t3code/codex-turn-mappingfrom
mwolson:fix/claude-codex-interrupt-recovery
Draft

fix(orchestrator): Preserve post-interrupt recovery state#4229
mwolson wants to merge 2 commits into
pingdotgg:t3code/codex-turn-mappingfrom
mwolson:fix/claude-codex-interrupt-recovery

Conversation

@mwolson

@mwolson mwolson commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Preserve the first Claude user turn after Stop when a stale background task
    notification arrives during session resume.
  • Terminalize Codex command projections when interrupted or failed turns omit
    the command completion event.
  • Terminalize run-owned subagent projections and linked child nodes when a
    stopped root turn leaves them open.
  • Add deterministic regressions for the observed Claude and Codex frame orders.

Stack

This draft targets t3code/codex-turn-mapping and is stacked on #4193, the same
post-merge fixture-fix base used by #4218. This branch contains one unique
commit above that stack base.

Problem and Fix

Problem and Why it Happened Fix
Claude can emit a background task_notification result between the first post-Stop prompt and its real response. The adapter treated that result as the active root result, completing the recovery run with no assistant output. Detect task-notification-origin results and ignore them for normal user turns before fallback text or terminalization. Provider continuation turns still consume the same result shape while draining buffered wakes.
Codex app-server can complete a turn as interrupted or failed without sending item/completed for an in-flight command. The projected command card remained running after the run settled. Track running command snapshots per native turn and emit terminal command node and turn-item updates before turn.terminal. Late completions are dropped without retaining or waking the stopped run.
Claude and Codex can stop the root run while its subagent entity, node, and turn item remain running; Claude can also leave the linked child root node running. Cascade interrupted, failed, or cancelled status to open run-owned subagent records and linked child nodes before root finalization, while preserving child linkage and partial result text.

Validation

  • vp check: pass with pre-existing warnings only
  • vp run typecheck: pass
  • Claude and Codex adapter tests: 62 passed
  • turn_interrupt_mid_tool replay fixture: 3 passed
  • claude-interrupt-direct-stop: pass with one interrupted run and two distinct
    completed recovery runs
  • Isolated paired browser verification: both Claude and Codex recovery messages
    rendered as distinct assistant responses with no stuck Working state
  • Packaged AppImage Claude race: an empty task-notification result arrived
    during recovery 1; both recovery runs completed with their expected markers
  • Packaged AppImage foreground-subagent probes: reproduced the stale parent row
    in Claude and Codex and the stale Claude child root node
  • Shared run-execution tests: 17 passed, including interrupt cascade,
    failed/cancelled mapping, supersede protection, linked-child cleanup,
    unreferenced-child exclusion, partial result preservation, and same-attempt
    and cross-attempt late-event rejection

Remaining Provider Behavior

The Codex app-server can report a turn interrupted while the provider-owned
foreground shell and child remain alive beyond the live suite's five-second
process-containment bound. This change settles T3's command projection and drops
late completion traffic, but it does not claim to terminate that external
process. The strict codex-interrupt-direct-stop pack therefore fails closed on
process containment even though the UI and recovery-state assertions pass.

@coderabbitai

coderabbitai Bot commented Jul 21, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 4564d317-fcbb-4a5d-8cdd-cdabfc0cbb7b

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added vouch:unvouched PR author is not yet trusted in the VOUCHED list. size:M 30-99 changed lines (additions + deletions). labels Jul 21, 2026
@mwolson
mwolson force-pushed the fix/claude-codex-interrupt-recovery branch from 12dfe4e to 2b5cda2 Compare July 21, 2026 12:55
@github-actions github-actions Bot added size:L 100-499 changed lines (additions + deletions). and removed size:M 30-99 changed lines (additions + deletions). labels Jul 21, 2026
@mwolson mwolson changed the title fix(claude): Preserve recovery turns across task notifications fix(orchestration): Preserve post-interrupt recovery state Jul 21, 2026
@mwolson mwolson changed the title fix(orchestration): Preserve post-interrupt recovery state fix(orchestrator): Preserve post-interrupt recovery state Jul 21, 2026
@mwolson
mwolson force-pushed the fix/claude-codex-interrupt-recovery branch from 2b5cda2 to 2557fda Compare July 21, 2026 14:28
if (belongsToRootRun) {
yield* Ref.update(openRunOwnedSubagents, (current) => {
const subagents = new Map(current.subagents);
if (isTerminalSubagentStatus(event.subagent.status)) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium orchestration-v2/RunExecutionService.ts:810

When a subagent reaches a terminal status before the root run is interrupted, trackChildLifecycle deletes it from openRunOwnedSubagents.subagents (and its turn item from .turnItems), which also discards the only recorded childThreadId. If that subagent's linked child-thread node is still pending, running, or waiting, cascadeTerminalizeRunOwnedSubagents still finds the node in open.nodes but can no longer match it to any childThreadId, so the node is never terminalized and remains permanently open in the projection. Consider preserving the childThreadId linkage until all linked nodes settle, or storing the linkage separately from the subagent/turn-item snapshots.

🤖 Copy this AI Prompt to have your agent fix this:
In file @apps/server/src/orchestration-v2/RunExecutionService.ts around line 810:

When a subagent reaches a terminal status before the root run is interrupted, `trackChildLifecycle` deletes it from `openRunOwnedSubagents.subagents` (and its turn item from `.turnItems`), which also discards the only recorded `childThreadId`. If that subagent's linked child-thread node is still `pending`, `running`, or `waiting`, `cascadeTerminalizeRunOwnedSubagents` still finds the node in `open.nodes` but can no longer match it to any `childThreadId`, so the node is never terminalized and remains permanently open in the projection. Consider preserving the `childThreadId` linkage until all linked nodes settle, or storing the linkage separately from the subagent/turn-item snapshots.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:L 100-499 changed lines (additions + deletions). vouch:unvouched PR author is not yet trusted in the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant