diff --git a/.gitignore b/.gitignore index 0662ae319..e74e1b852 100644 --- a/.gitignore +++ b/.gitignore @@ -27,3 +27,6 @@ docs/project_management/packs/**/logs/ .mcp.json .DS_Store **/.DS_Store +.gitnexus +.orchestrator-* +.substrate diff --git a/AGENTS.md b/AGENTS.md index dc878fb5c..9e54f5fb9 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -37,7 +37,7 @@ cargo clippy --workspace --all-targets -- -D warnings cargo test --workspace -- --nocapture # run full suite with output cargo bench # exercise hotspots when touching performance-sensitive crates ``` -- Avoid `--all-features` at the workspace level; enable graph features explicitly (e.g., `cargo build -p substrate-graph --features kuzu-dylib`). +- Avoid `--all-features` at the workspace level; enable graph features explicitly via the graph crate manifest (e.g., `cargo build --manifest-path crates/substrate-graph/Cargo.toml --features kuzu-dylib`). - Regenerate shims for manual testing with `substrate --shim-deploy`; inspect with `substrate --shim-status`. ## Coding Standards & Patterns @@ -91,3 +91,47 @@ cargo bench # exercise hotspots when touching pe - `scripts/dev-entrypoint.sh`: reproducible dev container bootstrap. - `docs/BACKLOG.md`: living backlog for DX improvements/bugs. - Update these guides when behavior changes; stale docs are treated as regressions. + + +# GitNexus — Code Intelligence + +This project is indexed by GitNexus as **substrate** (46052 symbols, 69611 relationships, 300 execution flows). Use the GitNexus MCP tools to understand code, assess impact, and navigate safely. + +> If any GitNexus tool warns the index is stale, run `npx gitnexus analyze` in terminal first. + +## Always Do + +- **MUST run impact analysis before editing any symbol.** Before modifying a function, class, or method, run `gitnexus_impact({target: "symbolName", direction: "upstream"})` and report the blast radius (direct callers, affected processes, risk level) to the user. +- **MUST run `gitnexus_detect_changes()` before committing** to verify your changes only affect expected symbols and execution flows. +- **MUST warn the user** if impact analysis returns HIGH or CRITICAL risk before proceeding with edits. +- When exploring unfamiliar code, use `gitnexus_query({query: "concept"})` to find execution flows instead of grepping. It returns process-grouped results ranked by relevance. +- When you need full context on a specific symbol — callers, callees, which execution flows it participates in — use `gitnexus_context({name: "symbolName"})`. + +## Never Do + +- NEVER edit a function, class, or method without first running `gitnexus_impact` on it. +- NEVER ignore HIGH or CRITICAL risk warnings from impact analysis. +- NEVER rename symbols with find-and-replace — use `gitnexus_rename` which understands the call graph. +- NEVER commit changes without running `gitnexus_detect_changes()` to check affected scope. + +## Resources + +| Resource | Use for | +|----------|---------| +| `gitnexus://repo/substrate/context` | Codebase overview, check index freshness | +| `gitnexus://repo/substrate/clusters` | All functional areas | +| `gitnexus://repo/substrate/processes` | All execution flows | +| `gitnexus://repo/substrate/process/{name}` | Step-by-step execution trace | + +## CLI + +| Task | Read this skill file | +|------|---------------------| +| Understand architecture / "How does X work?" | `.claude/skills/gitnexus/gitnexus-exploring/SKILL.md` | +| Blast radius / "What breaks if I change X?" | `.claude/skills/gitnexus/gitnexus-impact-analysis/SKILL.md` | +| Trace bugs / "Why is X failing?" | `.claude/skills/gitnexus/gitnexus-debugging/SKILL.md` | +| Rename / extract / split / refactor | `.claude/skills/gitnexus/gitnexus-refactoring/SKILL.md` | +| Tools, resources, schema reference | `.claude/skills/gitnexus/gitnexus-guide/SKILL.md` | +| Index, status, clean, wiki CLI commands | `.claude/skills/gitnexus/gitnexus-cli/SKILL.md` | + + diff --git a/AGENT_ORCHESTRATION_GAP_MATRIX.md b/AGENT_ORCHESTRATION_GAP_MATRIX.md index 915b3e52c..a05fb1d7c 100644 --- a/AGENT_ORCHESTRATION_GAP_MATRIX.md +++ b/AGENT_ORCHESTRATION_GAP_MATRIX.md @@ -25,7 +25,7 @@ In v1: - host orchestrator, - in-world member execution, - policy-gated orchestration, - - and shared-world reuse where needed for member sessions. + - and shared-world reuse for retained member sessions keyed by exact `backend_id`. ### later intent @@ -73,36 +73,63 @@ Terminology rule for this repository: ## Current State Summary -- Modeling is ahead of runtime: - - agent inventory, - - derived `backend_id`, - - role/scope validation, - - tuple-aware structured events, - - and observational CLI status surfaces are mostly in place. -- Gateway runtime is real for nested LLM/gateway lifecycle work, but that is not yet the same thing as live agent-hub orchestration. -- The main missing layer is the actual runtime control plane that opens, tracks, resumes, forks, stops, and correlates orchestrator/member sessions. +- This is no longer modeling-only. + - `crates/shell` now depends on `unified-agent-api` (`agent_api`) and the REPL boots a shell-owned UAA runtime in `crates/shell/src/repl/async_repl.rs`. + - Live orchestration and participant state is persisted by `AgentRuntimeStateStore` under `~/.substrate/run/agent-hub/sessions//...`, with compatibility snapshots under `~/.substrate/run/agent-hub/{sessions,participants}`. + - Host orchestration sessions now persist explicit durable posture and attachment truth: `posture`, `attached_participant_id`, `pending_inbox_count`, and per-session durable inbox artifacts under `sessions//inbox/.json`. + - That durable inbox state is intentionally narrow: persistence, posture normalization into `awaiting_attention`, internal ack/dismiss support, and dev-support/test ingress exist, but no public inbox command surface or automatic resume workflow is shipped. + - `parked_resumable` and `awaiting_attention` are retained durable host postures, not gone-session aliases; `terminal` remains the only non-routable posture family. + - Persistent world sessions now support explicit shared-world owner binding via `SharedWorldOwnerSpec` / `SharedWorldBindingSnapshot`, and the shell invalidates stale world members when `world_generation` rolls forward. + - Canonical `AgentEvent` production is live for the REPL-owned orchestrator/member runtime, and `substrate agent status` now prefers live runtime state with trace fallback while surfacing parked and attention-needed durable host sessions on the read path. + - The narrow public control plane is now live under `substrate agent start|turn|reattach|fork|stop`, with exact `--backend ` / `--session ` selectors, one-of prompt-source validation for `start` and `turn`, helper-owned streaming NDJSON for prompt-taking calls, authoritative completion-time `session_posture`, hidden-owner launch handoff, exact-session `reattach` recovery to durable attached truth, owner-transport stop routing as the canonical closeout path for attached and parked durable sessions, and explicit `Completed`/`Failed` terminal delivery after `Accepted`. +- Recent validation tightened several earlier suspected gaps: + - Linux shared-world replacement ordering and `session.json` durability are already landed in the current backend. + - Linux world-scoped member runtime placement over the existing host↔world execute-stream transport is now landed, including authoritative world binding validation, replacement-member relaunch, fail-closed startup semantics, and retained live-member coexistence for distinct backend ids such as `cli:codex` and `cli:claude_code` within one orchestration session / world generation. + - The selected-follow-up contract is now explicit and regression-proven for the Linux-first / REPL-first surface: exact `:: ` targeting stays fail-closed, host follow-up turns resume only the active orchestrator backend, same-generation world follow-up turns reuse the exact retained member, and stale world state relaunches the exact backend slot before typed `/v1/member_turn/stream` submit. + - The broad PID-based orchestration-stamping concern is mostly retired from production event emission; the remaining authority cleanup is narrower. +- The gateway runtime is also real for nested LLM/gateway lifecycle work, but it is still a separate runtime path from pure-agent orchestration. +- The main remaining gaps are: + - `substrate -c` is still shell wrap mode rather than an agent-prompt caller surface, + - the shipped public control plane is intentionally narrow: root `start` is host-only, existing-session prompt-taking accepts only exact `orchestration_session_id` plus exact `backend_id`, there is still no default-agent routing, and there is still no public world-root start, + - exact targeted turns now exist on both narrow caller surfaces, but they stay intentionally strict: the REPL keeps exact `:: `, public follow-up keeps exact `(orchestration_session_id, backend_id)`, detached world follow-up still requires `reattach`, and there is still no broader member-selector surface, + - broader operator-facing routing policy outside those exact caller paths is still unsettled; there is still no default-agent surface and no broader fuzzy non-REPL targeting contract, + - `substrate agent status` now degrades torn parent/session linkage into warnings on the read surface, but strict control-plane surfaces such as `toolbox status` / `toolbox env` still fail closed and the trace-only fallback remains coarse when the rows omit `participant_id`, + - macOS/Lima now uses the same shared-owner/member-runtime backend seam as Linux for the supported forwarded path, with regression coverage for shared-owner proof, member dispatch, targeted follow-up reuse, and guest-owned cancel. + - Linux remains the source-of-truth ownership implementation and still has the broadest platform maturity; Windows/WSL remains fail-closed outside the supported contract. ## Gap Matrix | Area | Status | What exists now | What is missing for the intended v1 path | |---|---|---|---| | Product intent | `Now explicit` | This file now records the intended v1 and later product shape | Keep downstream docs aligned to this statement | -| Unified Agent API adoption | `Decision made; runtime not landed` | ADR and contract work already point toward Unified Agent API; local authority now explicitly confirmed | Add a normal crates.io dependency on `unified-agent-api` and wire real usage into Substrate runtime paths | -| UAA vs local `agent-api-*` boundary | `Clarified conceptually` | Local crates exist for host/world-agent transport; UAA exists separately for CLI agents | Make the naming/ownership boundary explicit in code/docs so future runtime work does not mix them together | -| Agent config and inventory | `Landed` | Inventory schema, capabilities, scope, and derived `backend_id` in [agent_inventory.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_inventory.rs:13) | Integrate inventory selection with live UAA backend registration and launch paths | -| Agent CLI inspection surface | `Partially landed` | `substrate agent list|status|doctor|toolbox status|env` in [cli.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/cli.rs:399) and handlers in [agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:31) | The surfaces are still mostly inspection/projection, not live control-plane actions | -| Orchestrator eligibility rules | `Landed` | Host-only orchestrator validation and capability checks in [agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:1201) | Open a real host orchestrator session through UAA instead of stopping at validation | -| Agent event schema and trace flattening | `Landed` | Tuple-aware `AgentEvent` envelope in [agent_events.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/common/src/agent_events.rs:62) and persistence in [telemetry.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/telemetry.rs:187) | Real producers from orchestrator/member runtime rather than only shell/demo/world-lifecycle projections | -| Nested gateway identity split | `Landed observationally` | `agent status` separates pure-agent rows and nested gateway-backed rows from trace in [agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:386) | Drive this from live orchestrator/member runtime state, not only trace replay | -| Gateway lifecycle | `Landed for nested LLM runtime` | Real `status|sync|restart` endpoints in [world-agent lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/lib.rs:265) and shell lifecycle client in [world_gateway.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/builtins/world_gateway.rs:640) | Separate and add agent-session orchestration runtime; current gateway lifecycle is not yet agent-hub session lifecycle | -| Backend selection / allowlisting | `Landed for selection logic` | Inventory-backed selection for gateway backends in [agent_inventory.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_inventory.rs:191) | Apply the same selection and allowlist logic to live UAA backend registration and dispatch for orchestrator/member sessions | -| Live agent session registry | `Missing runtime` | Planning/history exists; observational status surfaces exist | Add a live source of truth for orchestration sessions and member sessions; this does not need to start as a full `/v1/agents` service if a thinner v1 slice is chosen | -| Agent session control plane | `Missing runtime` | ADR/pack semantics exist; UAA defines `run`, `run_control`, and session extensions | Add Substrate runtime paths for `start/resume/fork/stop` that delegate session semantics to UAA and add Substrate-owned orchestration metadata around them | -| Session handles | `Partial` | UAA has canonical session-handle capability/extension semantics; Substrate planning has `AgentSessionHandleV1` semantics | Add a Substrate-owned registry/store that records orchestration session id, role, world binding, and UAA session-handle metadata together | -| Host orchestrator process management | `Missing runtime` | Orchestrator can be selected and validated on paper | Launch and track a real host-scoped orchestrator backend via UAA | -| In-world member dispatch | `Missing runtime` | World scope and fail-closed checks exist | Launch and track one or more member backends via UAA inside the selected shared world boundary | -| World reuse across member sessions | `Partial` | World lifecycle alerting and generation metadata exist in trace/tests | Add actual shared-world ownership and reuse rules for live member sessions | -| Toolbox surface | `Partial` | Config keys exist in [config_model.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/config_model.rs:485); CLI `toolbox status|env` surface now exists; ADR contract exists in [ADR-0045](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/project_management/adrs/draft/ADR-0045-orchestration-toolbox-internal-mcp-identity-trace-contract.md:99) | Add the real internal MCP server, auth token flow, and tool-call audit emission | +| Unified Agent API adoption | `Landed for host orchestrator and supported Linux/macOS world member runtime` | `crates/shell/Cargo.toml` now pulls `unified-agent-api`, `agent_runtime/registry.rs` registers Codex / Claude Code backends, the REPL keeps the host orchestrator under attached control, and supported world-scoped members now launch through `world-agent` over the existing host↔world transport on Linux and the macOS/Lima forwarded path | Remaining v1 work is no longer basic UAA runtime adoption; it is on operator-facing controls, status resilience, Windows/WSL posture, and broader caller-surface productization | +| UAA vs local `agent-api-*` boundary | `Partially clarified` | Code now distinguishes shell-owned UAA runtime from local host/world transport; `PURE_AGENT_PROTOCOL` comments explicitly say `uaa.agent.session` is a Substrate-local normalized label, not an upstream protocol claim | Local crates are still named `agent-api-*`, and the `uaa.agent.session` label still exists, so naming confusion has been reduced but not removed | +| Agent config and inventory | `Landed` | Inventory schema, capability gates, derived `backend_id`, scope resolution, and backend selection are live in `agent_inventory.rs` and `agent_runtime/validator.rs` | No major v1 inventory gap remains; future work is mostly on runtime breadth rather than selection modeling | +| Agent CLI inspection surface | `Landed for inspection plus narrow public control` | `substrate agent list|status|doctor|toolbox status|env` is live, `agent status` now merges live runtime-state snapshots with trace fallback and surfaces durable `parked_resumable` / `awaiting_attention` host sessions, and the namespace now exposes first-class `start|turn|reattach|fork|stop` actions with exact selectors plus one-of prompt-source validation for `start` and `turn` | Remaining gaps are broader productization only: there is still no default-agent routing, no member-level public selector surface beyond exact `(orchestration_session_id, backend_id)`, and no public world-root start | +| REPL agent invocation grammar | `Landed, Linux-first and REPL-first` | The interactive shell now accepts exact single-line targeted follow-up turns as `:: `, routes them before shell fallback, preserves `:host` / `:pty`, and continues to treat plain REPL input as shell execution | Remaining work is broader product-surface breadth only: no default-agent routing exists, and non-REPL surfaces still need their own explicit caller contract | +| Non-interactive agent invocation surface | `Landed narrowly` | `substrate agent start --backend (--prompt | --prompt-file | --prompt-file -) [--json]` and `substrate agent turn --session --backend ...` now provide a real public prompt-taking caller surface with helper-owned streaming, completion-time `session_posture`, explicit fail-closed exact selector coverage, Linux world-member follow-up proof through typed `/v1/member_turn/stream`, and an explicit non-regression guard that keeps `substrate -c` / `--command` in shell-wrap mode | Remaining gaps are broader caller-surface breadth and parity only: there is still no default-agent routing, no public world-root start, and Windows/WSL world-sensitive follow-up still fails closed | +| Terminology guardrails from the proposed v1 slice | `Partially landed` | Backend-kind mapping for supported UAA agents exists in `agent_runtime/mapping.rs`; runtime selection fails closed for unsupported kinds; `agent_events.rs` documents the local meaning of `uaa.agent.session` | Local `agent-api-*` crate names are still in place, and the local trace/session label has not been renamed | +| Shell-owned UAA runtime | `Landed` | The REPL prepares orchestrator startup, validates allowlisting, builds a shell-owned UAA gateway, starts the orchestrator, retains cancel/event/completion handles, and persists lifecycle transitions in `repl/async_repl.rs` | This is still REPL-owned, not a general hub service or reusable daemonized control plane | +| Live agent session registry | `Landed` | `AgentRuntimeStateStore` persists authoritative parent sessions and participant records under `~/.substrate/run/agent-hub/sessions//session.json` plus per-session `participants/`, `leases/`, and durable `inbox/` artifacts; host sessions now persist explicit posture/attachment truth instead of inferring validity from attached-control diagnostics alone | Only a future product-surface rename to something like `agent-sessions` remains undecided; there is no current runtime gap here | +| Agent session control plane | `Landed for narrow public control, Linux-source with supported macOS/Lima world-sensitive parity` | Internal runtime paths now support attached-control `start`, public `reattach` / `fork`, prompt-taking `start` / `turn`, authoritative persistence, clean host detach parking into `parked_resumable`, pending-inbox escalation into `awaiting_attention`, exact-session `reattach` recovery back to durable attached truth, `stop`/cancel as the canonical closeout path for attached and parked durable sessions, replacement-member creation across world-generation rollover, the hidden owner-helper launch seam, the same private stop-owner plane for helper-owned and REPL-owned sessions, and explicit post-`Accepted` terminal envelopes | Remaining gaps are broader surface breadth and parity: no public world-root start, no member-level public selectors beyond exact session/backend targeting, and Windows/WSL world-sensitive control posture still fails closed | +| User-turn submission into active UAA sessions | `Landed for REPL and narrow public caller surfaces on Linux plus supported macOS/Lima` | The REPL can now submit arbitrary targeted follow-up turns into retained active sessions, and the public `substrate agent start|turn` surface can now submit exact prompts with helper-owned streaming: parked host follow-up turns resume the exact authoritative orchestrator session, supported world follow-up turns go through the typed `/v1/member_turn/stream` route using the persisted retained-member tuple (`participant_id`, `orchestrator_participant_id`, `backend_id`, `world_id`, `world_generation`), reusing the exact retained backend slot when current and relaunching that exact slot when stale, and detached-world follow-up remains fail-closed with `reattach` guidance | Remaining work is broader operator-surface breadth and platform parity. Public root `start` remains host-only, default-agent routing still does not exist, and Windows/WSL world follow-up still fails closed | +| Session handles | `Partial` | The runtime extracts and persists surfaced UAA session ids (`internal.uaa_session_id`) and correlates them with orchestration/participant metadata; parent sessions also track `active_session_handle_id` | The Substrate registry still mixes local participant ids, orchestration ids, and optional surfaced UAA ids rather than exposing a clean public session-handle contract | +| Host orchestrator process management | `Landed` | A real host-scoped orchestrator backend is launched via UAA, tracked through persisted lifecycle states, and stopped authoritatively on teardown | No major thin-slice blocker remains here | +| Explicit backend/member targeting | `Landed for the exact REPL and narrow public caller surfaces` | Effective inventory entries derive canonical `backend_id` values, targeted REPL follow-up turns route by exact `backend_id`, public `turn` requires exact `(orchestration_session_id, backend_id)` routing, host follow-up targeting fails closed unless it names the active orchestrator backend, Linux world reuse/relaunch is keyed by exact `backend_id` rather than `agent_id`, distinct retained members such as `cli:codex` and `cli:claude_code` can coexist in one orchestration session / world generation, and duplicate retained members for the same `backend_id` still fail closed | Remaining work is broader productization rather than runtime correctness: there is still no default-agent routing, and policy/default-selection decisions outside these exact caller paths are still open | +| Explicit shared-world ownership | `Landed on Linux and supported macOS/Lima; Linux remains source-of-truth` | Persistent world startup now sends `SharedWorldOwnerSpec`, world-agent echoes `SharedWorldBindingSnapshot`, session metadata persists `orchestration_session_id` / `world_generation`, shared-world replacement rolls back correctly on failure, and the shell invalidates stale world members after rollover on Linux plus the supported macOS/Lima forwarded path | Remaining work is broader platform scope and operator maturity, not missing shared-owner proof on the supported macOS/Lima path. Windows/WSL still does not implement this contract | +| Shared-world replacement ordering and world metadata durability | `Landed` | Linux replacement already uses a two-phase `Active -> Replacing -> Replaced` flow with rollback on creation failure, and world `session.json` persistence is atomic-by-rename with failure-preserving behavior | No current correctness gap was confirmed here; only future hardening beyond the current contract would remain | +| In-world member dispatch over existing host↔world transport | `Landed on Linux and supported macOS/Lima` | The shell now emits typed `member_dispatch` requests over `/v1/execute/stream`, `world-agent` validates the authoritative shared-world binding and launches the member UAA runtime inside the active world, `/v1/execute/cancel` reaches retained in-world control ownership, and regression tests cover lazy launch, reuse, replacement, and fail-closed startup behavior on Linux plus the supported macOS/Lima path | Remaining work is broader platform scope and follow-on hardening, not missing member-dispatch transport on the supported macOS/Lima path | +| macOS/Lima parity for agent orchestration | `Landed in the supported Lima-backed posture` | `world-mac-lima` now forwards the same shared-owner/member-runtime contract as Linux for the supported backend-detected path: `SharedWorldOwnerSpec` reaches the guest, `SharedWorldBindingSnapshot` returns through `WorldHandle.shared_binding`, `member_dispatch` and typed `/v1/member_turn/stream` follow-up are preserved, and cancel remains guest-owned through `/v1/execute/cancel` | Remaining work is outside this slice: Windows/WSL parity, broader public caller-surface breadth, and any future live-guest coverage beyond the documented macOS/Lima smoke plus regression suite | +| Replacement-member semantics across world-generation rollover | `Landed` | Restart/drift handling invalidates stale members, advances `world_generation`, creates a distinct replacement participant with preserved lineage, and relaunches the replacement member through the in-world transport path on Linux | Remaining work is mainly parity and operator-surface breadth rather than replacement correctness | +| Agent event schema and trace flattening | `Landed` | Tuple-aware `AgentEvent` schema is live, runtime events are emitted from orchestrator/member lifecycle code, and status surfaces validate world identity + nested parent correlation | Remaining work is narrower read-side cleanup and producer breadth, not missing support for the now-landed in-world member launch path | +| Event-emission authority plumbing | `Mostly landed` | Production REPL/host/world emitters now require explicit runtime-owned orchestration context before publishing orchestration-scoped `agent_event` rows | Remaining cleanup is narrower: command rows still use synthetic `run_id = cmd_id`, and read-side utilities still keep heuristic recovery helpers such as PID lookup / synthetic parent reconstruction | +| Bootstrap ordering | `Mostly landed` | Parent orchestration session persistence now happens before persisted child/runtime state, and the parent is not marked live until the runtime surfaces a session handle | The remaining nuance is in-memory construction order: the child manifest is still built with the new `orchestration_session_id` before the authoritative parent record is persisted | +| Nested gateway identity split | `Landed` | `substrate agent status` now separates pure-agent sessions from nested gateway-backed rows using live runtime state plus trace correlation | Remaining work here is secondary and mostly depends on broader runtime rollout, not a schema gap | +| Gateway lifecycle | `Landed for nested LLM runtime` | `status|sync|restart` are live end-to-end for the nested gateway lifecycle | This should still not be conflated with pure-agent session orchestration | +| Status ambiguity handling | `Partially landed` | `substrate agent status` now keeps valid rows visible while surfacing degraded warnings for torn parent/session state such as missing `active_session_handle_id`, missing parent-session metadata, and incomplete selected participants; durable parked and attention-needed host sessions remain visible on the read surface instead of collapsing into terminal absence | Strict selectors still fail closed by design on the control-plane surfaces (`toolbox status`, `toolbox env`, doctor/member selection), and status still needs a clearer operator-facing ambiguity policy for participant-less trace-only rows | +| Trace-only participant-aware fallback | `Partially landed` | Trace rows that carry `participant_id` now stay sibling-distinct, sibling-specific suppression is keyed by `participant_id`, and nested correlation honors `parent_participant_id` when same-agent siblings coexist in one `orchestration_session_id` | Participant-less trace rows still warn and fall back to coarse matching, so the fallback is not yet fully participant-aware unless the trace carries `participant_id` / `parent_participant_id` | +| Secret handoff into the world gateway | `Landed` | Host-side policy/auth selection still owns auth-source precedence, `world-agent` now delivers a read-once `GatewayAuthBundleV1` over inherited FD via `SUBSTRATE_LLM_AUTH_BUNDLE_FD`, and `gateway` integrated startup overlays that bundle in memory before provider construction | Remaining work is follow-on hardening and broader parity, not the default carrier: integrated delivery no longer depends on secret-bearing child env vars | +| Toolbox surface | `Partial` | Config, `toolbox status`, `toolbox env`, live-session endpoint derivation, and world-binding projection are present | The internal MCP server, mutation tools, and auth/audit plumbing are still unimplemented | | Toolbox role in orchestration | `Constrained by design` | ADR-0045 is introspection-only in [ADR-0045](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/project_management/adrs/draft/ADR-0045-orchestration-toolbox-internal-mcp-identity-trace-contract.md:77) | Nothing here should be treated as a substitute for member launch or control-plane execution | | Custom Substrate harness | `Deferred by intent` | Product intent allows for a future Substrate-native harness layer | Do not block v1 on this; revisit only after CLI-agent orchestration through UAA is working cleanly | @@ -110,44 +137,52 @@ Terminology rule for this repository: These are the decisions that still need to be made to keep the path forward clean. -1. Should Substrate’s first shipped runtime slice be: - - a thin vertical slice: - - one host orchestrator, - - one in-world member, - - one shared-world binding model, - - and a minimal session registry, - - or a broader hub service with a general `/v1/agents` registry and richer multi-agent session management from the start? - -2. What exact Substrate-owned session record should wrap the UAA session semantics? - - UAA already defines backend-facing session extensions and session-handle surfacing. - - Substrate still needs its own orchestration record for: - - `orchestration_session_id`, - - role, - - world binding, - - world generation, - - trace correlation, - - and restart invalidation semantics. - -3. Should `uaa.agent.session` remain as a Substrate-local trace/identity token, or should it be replaced? - - It should not be treated as if it were the canonical upstream UAA protocol token. - - If it remains, it should be explicitly documented as a Substrate-local identity label rather than an upstream UAA contract surface. - -4. How soon should the local `agent-api-*` crates be renamed or otherwise deconflicted? - - They are already easy to confuse with external `agent_api`. - - The longer runtime work continues without an explicit naming boundary, the more expensive the cleanup becomes. +1. How far to widen the now-shipped `substrate agent start|turn|reattach|fork|stop` namespace before broader hub work. +- Current repo evidence points toward “keep the shipped namespace narrow until non-interactive caller semantics are frozen.” +- Session start is now explicit for host orchestrators, but plain REPL input is still treated as shell execution rather than “message the default backend agent.” +- The open question is how much additional caller breadth belongs beside this first-class control surface before a broader hub/service story exists. + +2. What the explicit agent caller syntax should be in interactive and non-interactive surfaces. +- The REPL now has an explicit targeted-turn contract: `:: ` on a single line, with no default-agent routing and with plain REPL input still reserved for shell execution. +- The remaining decision is how that explicit caller contract should extend beyond the REPL, because `substrate -c` still means shell wrap mode today and any non-interactive agent caller must be introduced deliberately rather than implied. + +3. Whether `uaa.agent.session` should remain the Substrate-local normalized protocol-family label, or be renamed now that real UAA runtime integration is live. +- The recent runtime landing increases the cost of leaving this ambiguous, because the repo now contains both real external `agent_api` usage and local `uaa.*` labeling. +- The discussion above did not resolve this; it only reinforces that the label is still local and still easy to misread as an upstream protocol claim. + +4. How aggressively to deconflict the local `agent-api-*` crate names from external `agent_api` before more runtime code accumulates on both sides of that boundary. +- This is closely related to the `uaa.agent.session` decision above, but it is a crate/module/package naming question rather than a protocol/identity-label question. +- The recent runtime landing makes this more pressing from a governance/clarity standpoint, even though it is not a direct runtime-correctness blocker. + +5. Whether to stop at the newly landed hidden shell↔world-agent helper over the existing execute/stream transport, or continue on to a broader reusable world-owned UAA service surface. +- The current repo state has now landed the thinner option first: host orchestrator on the shell, world member launched through the existing world transport seam, without building a broad new hub service up front. +- The remaining question is product breadth and reuse, not whether member placement can work through the existing transport seam. + +6. Whether explicit backend targeting should be mandatory once multiple eligible world members exist. +- The current explicit REPL caller path already requires named `backend_id` selection and supports multiple retained live world members when the backend ids are distinct. +- Productizing targeted invocation beyond the current REPL path still means deciding whether named `backend_id` selection becomes mandatory anywhere multiple eligible members may exist outside this exact caller path, or whether a default-selection layer is introduced elsewhere. + +7. Whether and when to retire the remaining flat compatibility outputs after the `agent-hub` cutover is considered complete. +- Code and docs are already aligned that compatibility dual-writes are still intentional today. +- This is now a retirement-timing decision, not a current ambiguity about what the runtime is supposed to do. + +8. How much cross-platform parity is required before calling the v1 slice complete, given that Linux currently has the clearest shared-world ownership semantics. +- The current implementation evidence still says Linux is the strongest and clearest slice for shared-world ownership and replacement behavior. +- macOS/Lima is now on the supported shared-owner/member-runtime path through the forwarded backend seam; the next parity question is Windows/WSL or any broader non-Linux backend model, not whether Lima needs a different orchestration design. +- The open question is the parity bar for non-Linux backends after that, not whether the host-orchestrator / world-member model itself is still the intended placement rule. ## Recommended v1 Runtime Slice -If the goal is to move quickly without overbuilding the control plane, the recommended first executable slice is: +The thin-slice recommendation is still correct, but the remaining scope is now much narrower than when this file was first written: -1. Register supported UAA backends from inventory. -2. Open a host-scoped orchestrator session through UAA. -3. Open one world-scoped member session through UAA inside a Substrate-managed shared world. -4. Record a Substrate-owned orchestration session object that wraps: - - orchestrator/member role, - - world binding, - - world generation, - - and any surfaced UAA session handle. -5. Emit canonical trace/agent events from that runtime. +1. Keep the existing shell-owned orchestrator runtime and live state registry. +2. Reuse the existing shared-world owner binding model and world-generation invalidation rules. +3. Keep the newly landed Linux world-owned member launch over the existing host↔world execute-stream seam as the v1 baseline. +4. Keep the exact REPL targeted-turn grammar `:: ` as the explicit Linux-first / REPL-first caller surface rather than redesigning the grammar or adding implicit default-agent routing. +5. Extend the already-landed user-turn submission seam beyond the current REPL-first surface only through deliberate new caller contracts, not by changing `substrate -c` semantics implicitly. +6. Keep `start`, `turn`, `reattach`, `fork`, and `stop` under `substrate agent` as the clean public session-handle contract, with `reattach` reserved for exact-session attached-owner recovery and `stop` as the canonical closeout path for attached or parked durable sessions. +7. Keep `substrate agent status` on the current split contract: surface durable `parked_resumable` and `awaiting_attention` sessions on the read surface, degrade torn parent/session linkage into warnings, keep control-plane selectors fail-closed, and finish the remaining participant-less trace fallback hardening. +8. Keep the landed auth-bundle handoff as the default integrated carrier so nested/in-world gateway work does not regress back to secret-bearing child env vars. +9. Keep macOS/Lima on the same shared-owner/member-runtime path as Linux while deciding whether any future non-Linux backend should meet that same bar or stay fail-closed. -That slice is enough to prove the product direction without requiring a full general-purpose agent hub service surface up front. +At this point, the missing work is broader caller-surface productization, control-plane/status hardening, and parity decisions, not basic UAA adoption, REPL targeted-turn grammar, REPL user-turn dispatch, session persistence, event modeling, gateway secret-carrier honesty, or Linux member-runtime placement. diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 000000000..ab715ef4a --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,43 @@ + +# GitNexus — Code Intelligence + +This project is indexed by GitNexus as **substrate** (46052 symbols, 69611 relationships, 300 execution flows). Use the GitNexus MCP tools to understand code, assess impact, and navigate safely. + +> If any GitNexus tool warns the index is stale, run `npx gitnexus analyze` in terminal first. + +## Always Do + +- **MUST run impact analysis before editing any symbol.** Before modifying a function, class, or method, run `gitnexus_impact({target: "symbolName", direction: "upstream"})` and report the blast radius (direct callers, affected processes, risk level) to the user. +- **MUST run `gitnexus_detect_changes()` before committing** to verify your changes only affect expected symbols and execution flows. +- **MUST warn the user** if impact analysis returns HIGH or CRITICAL risk before proceeding with edits. +- When exploring unfamiliar code, use `gitnexus_query({query: "concept"})` to find execution flows instead of grepping. It returns process-grouped results ranked by relevance. +- When you need full context on a specific symbol — callers, callees, which execution flows it participates in — use `gitnexus_context({name: "symbolName"})`. + +## Never Do + +- NEVER edit a function, class, or method without first running `gitnexus_impact` on it. +- NEVER ignore HIGH or CRITICAL risk warnings from impact analysis. +- NEVER rename symbols with find-and-replace — use `gitnexus_rename` which understands the call graph. +- NEVER commit changes without running `gitnexus_detect_changes()` to check affected scope. + +## Resources + +| Resource | Use for | +|----------|---------| +| `gitnexus://repo/substrate/context` | Codebase overview, check index freshness | +| `gitnexus://repo/substrate/clusters` | All functional areas | +| `gitnexus://repo/substrate/processes` | All execution flows | +| `gitnexus://repo/substrate/process/{name}` | Step-by-step execution trace | + +## CLI + +| Task | Read this skill file | +|------|---------------------| +| Understand architecture / "How does X work?" | `.claude/skills/gitnexus/gitnexus-exploring/SKILL.md` | +| Blast radius / "What breaks if I change X?" | `.claude/skills/gitnexus/gitnexus-impact-analysis/SKILL.md` | +| Trace bugs / "Why is X failing?" | `.claude/skills/gitnexus/gitnexus-debugging/SKILL.md` | +| Rename / extract / split / refactor | `.claude/skills/gitnexus/gitnexus-refactoring/SKILL.md` | +| Tools, resources, schema reference | `.claude/skills/gitnexus/gitnexus-guide/SKILL.md` | +| Index, status, clean, wiki CLI commands | `.claude/skills/gitnexus/gitnexus-cli/SKILL.md` | + + diff --git a/Cargo.lock b/Cargo.lock index 89ca9b239..1b17ad09e 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -51,6 +51,7 @@ dependencies = [ "serde_json", "substrate-common", "thiserror 1.0.69", + "world-api", ] [[package]] @@ -157,12 +158,6 @@ version = "0.5.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7d902e3d592a523def97af8f317b08ce16b7ab854c1985a0c671e6f15cebc236" -[[package]] -name = "arrayvec" -version = "0.7.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7c02d123df017efcdfbd739ef81735b36c5ba83ec3c59c80a9d7ecc718f92e50" - [[package]] name = "assert-json-diff" version = "2.0.2" @@ -586,36 +581,6 @@ version = "1.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c8d4a3bb8b1e0c1050499d1815f5ab16d04f0959b233085fb31653fbfc9d98f9" -[[package]] -name = "cmake" -version = "0.1.58" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c0f78a02292a74a88ac736019ab962ece0bc380e3f977bf72e376c5d78ff0678" -dependencies = [ - "cc", -] - -[[package]] -name = "codespan-reporting" -version = "0.11.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3538270d33cc669650c4b093848450d380def10c331d38c768e34cac80576e6e" -dependencies = [ - "termcolor", - "unicode-width 0.1.14", -] - -[[package]] -name = "codespan-reporting" -version = "0.13.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "af491d569909a7e4dee0ad7db7f5341fef5c614d5b8ec8cf765732aba3cff681" -dependencies = [ - "serde", - "termcolor", - "unicode-width 0.2.2", -] - [[package]] name = "colorchoice" version = "1.0.5" @@ -669,7 +634,7 @@ dependencies = [ "encode_unicode", "libc", "once_cell", - "unicode-width 0.2.2", + "unicode-width", "windows-sys 0.59.0", ] @@ -919,66 +884,6 @@ dependencies = [ "windows-sys 0.61.2", ] -[[package]] -name = "cxx" -version = "1.0.138" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3956d60afa98653c5a57f60d7056edd513bfe0307ef6fb06f6167400c3884459" -dependencies = [ - "cc", - "cxxbridge-cmd", - "cxxbridge-flags", - "cxxbridge-macro", - "foldhash", - "link-cplusplus", -] - -[[package]] -name = "cxx-build" -version = "1.0.194" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b0f4697d190a142477b16aef7da8a99bfdc41e7e8b1687583c0d23a79c7afc1e" -dependencies = [ - "cc", - "codespan-reporting 0.13.1", - "indexmap", - "proc-macro2", - "quote", - "scratch", - "syn", -] - -[[package]] -name = "cxxbridge-cmd" -version = "1.0.138" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0f01e92ab4ce9fd4d16e3bb11b158d98cbdcca803c1417aa43130a6526fbf208" -dependencies = [ - "clap", - "codespan-reporting 0.11.1", - "proc-macro2", - "quote", - "syn", -] - -[[package]] -name = "cxxbridge-flags" -version = "1.0.138" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8c41cbfab344869e70998b388923f7d1266588f56c8ca284abf259b1c1ffc695" - -[[package]] -name = "cxxbridge-macro" -version = "1.0.138" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "88d82a2f759f0ad3eae43b96604efd42b1d4729a35a6f2dc7bdb797ae25d9284" -dependencies = [ - "proc-macro2", - "quote", - "rustversion", - "syn", -] - [[package]] name = "dashmap" version = "5.5.3" @@ -2191,21 +2096,6 @@ dependencies = [ "libc", ] -[[package]] -name = "kuzu" -version = "0.11.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "06952b00b1a2eabf4caf1426672395f9b1def04207d984af204693dec3da287a" -dependencies = [ - "cmake", - "cxx", - "cxx-build", - "rust_decimal", - "rustversion", - "time", - "uuid", -] - [[package]] name = "lazy_static" version = "1.5.0" @@ -2236,33 +2126,6 @@ dependencies = [ "redox_syscall 0.7.4", ] -[[package]] -name = "libseccomp" -version = "0.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "21c57fd8981a80019807b7b68118618d29a87177c63d704fc96e6ecd003ae5b3" -dependencies = [ - "bitflags 1.3.2", - "libc", - "libseccomp-sys", - "pkg-config", -] - -[[package]] -name = "libseccomp-sys" -version = "0.2.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9a7cbbd4ad467251987c6e5b47d53b11a5a05add08f2447a9e2d70aef1e0d138" - -[[package]] -name = "link-cplusplus" -version = "1.0.12" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7f78c730aaa7d0b9336a299029ea49f9ee53b0ed06e9202e8cb7db9bae7b8c82" -dependencies = [ - "cc", -] - [[package]] name = "linux-raw-sys" version = "0.4.15" @@ -3214,7 +3077,7 @@ dependencies = [ "strum_macros", "thiserror 2.0.18", "unicode-segmentation", - "unicode-width 0.2.2", + "unicode-width", ] [[package]] @@ -3371,16 +3234,6 @@ dependencies = [ "ordered-multimap", ] -[[package]] -name = "rust_decimal" -version = "1.41.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2ce901f9a19d251159075a4c37af514c3b8ef99c22e02dd8c19161cf397ee94a" -dependencies = [ - "arrayvec", - "num-traits", -] - [[package]] name = "rustc-hash" version = "1.1.0" @@ -3570,12 +3423,6 @@ version = "1.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "94143f37725109f92c262ed2cf5e59bce7498c01bcc1502d7b9afe439a4e9f49" -[[package]] -name = "scratch" -version = "1.0.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d68f2ec51b097e4c1a75b681a8bec621909b5e91f15bb7b840c4f2f7b01148b2" - [[package]] name = "sct" version = "0.7.1" @@ -4144,6 +3991,7 @@ dependencies = [ "serde", "serde_json", "sha2", + "substrate-common", "tempfile", "thiserror 1.0.69", "tiktoken-rs", @@ -4166,15 +4014,12 @@ dependencies = [ "anyhow", "chrono", "dirs 6.0.0", - "kuzu", "serde", "serde_json", "substrate-common", "substrate-trace", - "tempfile", "thiserror 1.0.69", "tokio", - "tokio-test", "tracing", ] @@ -4354,15 +4199,6 @@ dependencies = [ "windows-sys 0.61.2", ] -[[package]] -name = "termcolor" -version = "1.4.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "06794f8f6c5c898b3275aebefa6b8a1cb24cd2c6c79397ab15774837a0bc5755" -dependencies = [ - "winapi-util", -] - [[package]] name = "termios" version = "0.2.2" @@ -4933,12 +4769,6 @@ version = "1.13.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9629274872b2bfaf8d66f5f15725007f635594914870f65218920345aa11aa8c" -[[package]] -name = "unicode-width" -version = "0.1.14" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7dd6e30e90baa6f72411720665d41d89b9a3d039dc45b8faea1ddd07f617f6af" - [[package]] name = "unicode-width" version = "0.2.2" @@ -4953,9 +4783,9 @@ checksum = "ebc1c04c71510c7f702b52b7c350734c9ff1295c464a03335b00bb84fc54f853" [[package]] name = "unified-agent-api" -version = "0.2.3" +version = "0.3.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d5684c156f98a2cae7b5089aad0e449eb10e7420b6f733c00951bc4f5b59dc56" +checksum = "ed6756fe97439580f64d317a8b9e8b471e4c21d08e6e1681ff86850665239cdb" dependencies = [ "futures-core", "futures-util", @@ -4969,9 +4799,9 @@ dependencies = [ [[package]] name = "unified-agent-api-claude-code" -version = "0.2.3" +version = "0.3.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d955aa5a5d5d2c38263cfe2b2a8ab170fbbbfcf6dfd00dd8117a7c9e1a0938db" +checksum = "93a2b0d39a51ac9e642bebe947d377921b5a5ee366af733ac95d692be31fc915" dependencies = [ "futures-core", "portable-pty 0.9.0", @@ -4984,9 +4814,9 @@ dependencies = [ [[package]] name = "unified-agent-api-codex" -version = "0.2.3" +version = "0.3.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d37fe286c84074ea6cd25248de0f949cfaec05ad8b4867a69b42b0d0e52aaa2f" +checksum = "bbb1725e74fa859d0893a61380f3fe5b866a49471706e0280fdb88c9f9078a86" dependencies = [ "futures-core", "semver", @@ -5835,7 +5665,6 @@ dependencies = [ "chrono", "dns-lookup", "libc", - "libseccomp", "linux-raw-sys 0.4.15", "nix 0.27.1", "prctl", @@ -5866,7 +5695,6 @@ dependencies = [ "futures-util", "hyper 0.14.32", "libc", - "libseccomp", "once_cell", "portable-pty 0.8.1", "rand 0.8.5", @@ -5884,6 +5712,7 @@ dependencies = [ "tower 0.4.13", "tracing", "tracing-subscriber", + "unified-agent-api", "uuid", "world", "world-api", diff --git a/Cargo.toml b/Cargo.toml index 83124cc0e..0c68c43a3 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -58,9 +58,10 @@ members = [ "crates/broker", "crates/trace", # "crates/trace", # TODO: Implement after broker "crates/telemetry-lib", - "crates/replay", "crates/substrate-graph", "crates/forwarder", "crates/world-windows-wsl", + "crates/replay", "crates/forwarder", "crates/world-windows-wsl", "tools/version-bump", ] +exclude = ["crates/substrate-graph"] [workspace.dependencies] anyhow = "1.0" diff --git a/HOST_ORCHESTRATOR_INTENDED_BEHAVIOR_TRUTH.md b/HOST_ORCHESTRATOR_INTENDED_BEHAVIOR_TRUTH.md new file mode 100644 index 000000000..da4e49d38 --- /dev/null +++ b/HOST_ORCHESTRATOR_INTENDED_BEHAVIOR_TRUTH.md @@ -0,0 +1,245 @@ +# Host Orchestrator Intended Behavior Truth + +## Purpose + +This document records the intended behavior for the Substrate host orchestrator session model exactly as clarified during review of: + +- [docs/project_management/adrs/draft/ADR-0047-host-orchestrator-durable-session-and-parked-resumable-ownership.md](/home/spenser/__Active_code/substrate/docs/project_management/adrs/draft/ADR-0047-host-orchestrator-durable-session-and-parked-resumable-ownership.md) +- [llm-last-mile/22-broaden-caller-surfaces-from-repl-first-to-public-session-member-turns.md](/home/spenser/__Active_code/substrate/llm-last-mile/22-broaden-caller-surfaces-from-repl-first-to-public-session-member-turns.md) +- [llm-last-mile/23-host-orchestrator-durable-session-and-parked-resumable-ownership.md](/home/spenser/__Active_code/substrate/llm-last-mile/23-host-orchestrator-durable-session-and-parked-resumable-ownership.md) +- [llm-last-mile/24-fix-host-bootstrap-readiness-and-clean-detach-parking.md](/home/spenser/__Active_code/substrate/llm-last-mile/24-fix-host-bootstrap-readiness-and-clean-detach-parking.md) + +This document is not a design brainstorm. It is a truth record of intended behavior and the frozen public contract for the currently shipped durable host-session model. + +## Core Model + +The host orchestrator session is a Substrate-owned durable session. + +The durable authority is: + +- the Substrate orchestration session record +- the authoritative participant linkage +- the durable inbox / task state +- the Substrate-owned routing and lifecycle state + +The durable authority is not: + +- one currently attached backend process +- one currently running `codex exec` process +- one currently running helper PID + +A Codex-backed host process is an attachable execution client. It may attach, run a prompt, exit cleanly, and later resume against the same durable orchestration session. + +## Meaning Of `agent start` + +`substrate agent start --backend --prompt ...` starts the host orchestrator session. + +The command is intended to: + +- create or bind the durable Substrate-owned orchestration session +- run the user's initial prompt as the true initial backend prompt +- establish the orchestration session as active and durable +- leave that orchestration session available for future orchestration work until explicit stop + +The command is not intended to: + +- send a hidden bootstrap prompt as the real backend user prompt +- wrap the user's prompt inside hidden control instructions and treat that wrapped text as the real initial backend prompt +- treat the backend process itself as the durable orchestration authority +- end the orchestration session merely because the initial prompt-bounded backend process exits + +## Prompt Semantics + +For a Codex-style backend: + +- the initial user prompt for `agent start --prompt ...` must map to the real initial `codex exec` prompt semantics +- follow-up prompt-taking must map to resume semantics + +That means: + +- `agent start --prompt ...` must use the user prompt as the true initial backend prompt +- `agent turn --session ... --backend ... --prompt ...` is the follow-up prompt-taking resume path +- `agent reattach --session ...` is not a prompt-taking action + +The helper launched by `agent start` exists for Substrate-owned session setup, state bookkeeping, routing, and lifecycle management. It is not supposed to replace or rewrite the user's initial backend prompt semantics. + +## Session Lifetime + +When `agent start` succeeds, the orchestration session should remain open until: + +- `agent stop` is run +- or a future explicit stale/timeout lifecycle is added and deliberately transitions it + +The session should not stop being an open orchestration session simply because: + +- the initial `codex exec` process finished +- the foreground helper is no longer actively babysitting world agents +- the orchestrator is currently idle + +## Meaning Of `parked_resumable` + +`parked_resumable` does not mean the orchestration session ended. + +`parked_resumable` means: + +- the orchestration session is still active and durable +- no foreground attached host execution client is currently babysitting the session +- the session remains routable and resumable +- Substrate still owns the authoritative session state +- the session remains available to receive requests, messages, updates, approvals, completions, and other orchestration responsibilities + +More simply: + +- parked means idle or detached +- parked does not mean gone +- parked does not mean terminal + +## Meaning Of `active_attached` + +`active_attached` means: + +- the orchestration session is active +- an attached host execution client is currently present +- that attached client can immediately receive prompt traffic and foreground orchestration work + +## Meaning Of `awaiting_attention` + +`awaiting_attention` means: + +- the orchestration session is still active and durable +- no host client is currently attached +- pending durable inbox work exists +- host-side review or resumption is needed + +## Meaning Of `terminal` + +`terminal` means: + +- the orchestration session is no longer routable +- the orchestration session is closed, failed, invalidated, or otherwise no longer available for orchestration work + +## Parked Session Responsibilities + +While parked, the orchestrator session is still supposed to represent a live durable authority that can continue owning world-agent responsibilities. + +That means parked is intended to remain capable of: + +- receiving world-originated messages +- receiving world-originated updates +- receiving approvals +- receiving completion notices +- receiving follow-up work +- retaining those items durably while no host client is attached +- allowing later `turn` or `reattach` against that same session + +The orchestrator should not need to stay foreground-attached just to keep the session valid. + +## Meaning Of `agent turn` + +`substrate agent turn --session --backend --prompt ...` is the exact follow-up prompt-taking resume path. + +It is intended to: + +- target one exact durable orchestration session +- resume prompt-taking work against that same session +- preserve the durable session identity +- allow the session to return to parked afterward if no attached client remains + +`turn` is not intended to: + +- create a fuzzy new session +- bypass the durable session record +- require the original initial backend process to still be running + +## Meaning Of `agent reattach` + +`substrate agent reattach --session ` is attached-owner recovery only. + +It is intended to: + +- restore attached host ownership for the already-existing durable orchestration session +- leave the same durable session active +- make the session truly attached again + +It is not intended to: + +- submit a prompt +- implicitly consume inbox work +- merely return success while the session immediately falls back to parked + +If `reattach` reports success, the intended truth is that attached ownership was actually restored. + +## Meaning Of `agent stop` + +`substrate agent stop --session ` is the explicit shutdown action for the durable orchestration session. + +It is intended to: + +- stop the durable orchestration session cleanly +- be the canonical closeout path for an active orchestration session +- work for the durable session model, not only for a currently attached live owner process + +The session is intended to remain open until `stop` is run, subject only to future explicit stale/timeout lifecycle rules if those are later designed and implemented. + +## Meaning Of `agent status` + +`substrate agent status --json` is intended to surface the durable host orchestration session truth. + +That includes parked sessions. + +A parked session is still supposed to be visible as a real durable orchestration session, not disappear merely because no attached live owner process is currently present. + +## Durable Inbox Expectations + +The durable inbox is a narrow retained state surface, not a broad public product workflow. + +The shipped contract is: + +- inbox items can persist durably under the session root while no host client is attached +- pending inbox work can normalize posture from `parked_resumable` to `awaiting_attention` +- internal ack and dismiss support exists for retained items +- dev-support and test ingress exist for exercising that retained state +- no public inbox command surface is shipped +- no public inbox-driven automatic resume workflow is shipped + +## Frozen Public Contract + +The current public durable-session contract is: + +- `substrate agent turn --session ... --backend ... --prompt ...` is prompt-taking follow-up on the same durable orchestration session +- `substrate agent reattach --session ...` is attached-owner recovery only for that same durable orchestration session +- `substrate agent stop --session ...` is the canonical closeout path for attached and parked durable host sessions +- `substrate agent status --json` is the authoritative parked-session read surface for live-runtime durable-session posture truth +- detached-world follow-up remains fail-closed until `reattach` restores an active host owner + +## Intended Acceptance Shape + +The intended final shape is: + +1. `agent start --prompt ...` starts a durable active orchestration session and runs the user prompt as the true initial backend prompt. +2. That session remains open until explicit `agent stop` or a future explicitly designed stale/timeout lifecycle. +3. The session may be `active_attached`, `parked_resumable`, or `awaiting_attention` while still remaining an active durable orchestration session. +4. `parked_resumable` means the session is idle/detached but still alive, routable, resumable, and still Substrate-owned for orchestration responsibilities. +5. `turn` resumes prompt-taking against that same durable session. +6. `reattach` restores actual attached host ownership for that same durable session without submitting a prompt. +7. `stop` cleanly stops that same durable session. +8. `status` shows that same durable session even while parked. +9. Durable inbox state remains narrow: persistence exists, `awaiting_attention` posture normalization exists, internal ack/dismiss support exists, and no public inbox workflow is implied. +10. Detached-world follow-up remains fail-closed until `reattach` restores an active host owner. + +## Non-Negotiable Truths + +The following are the key truths this repository still needs to honor: + +- the durable authority is the Substrate orchestration session, not one backend process +- `agent start` is not supposed to end the orchestration session when the initial prompt-bounded backend run exits +- parked does not mean gone +- parked does not mean terminal +- parked means still alive as a durable orchestration session +- world agents must be able to keep working without requiring the host orchestrator to foreground-babysit them continuously +- the parked durable session must still be available to receive requests, messages, updates, approvals, completions, and follow-up orchestration work +- `reattach` must actually restore attachment if it reports success +- `stop` must be the real closeout path for the durable orchestration session +- `status` must represent the durable parked session truth +- detached-world follow-up must stay fail-closed until `reattach` restores an active host owner +- durable inbox wording must stay narrow and must not imply a public inbox product surface or automatic resume path diff --git a/Makefile b/Makefile index 34aa3052e..f34002794 100644 --- a/Makefile +++ b/Makefile @@ -112,6 +112,9 @@ CODEX_PROFILE ?= CODEX_MODEL ?= CODEX_JSONL ?= 0 EMIT_JSON ?= 0 +PROVING_RUN_FACTS ?= +PROVING_RUN_HUMAN_INPUTS ?= +PROVING_RUN_CLOSEOUT_OUTPUT ?= proving-run-closeout.json .PHONY: planning-validate planning-validate: @@ -245,6 +248,14 @@ pm-pre-planning-from-adr: $(MAKE) pm-pre-planning-research FEATURE_DIR="$$feature_dir" START_AT="$(START_AT)" POLL_S="$(POLL_S)" CODEX_PROFILE="$(CODEX_PROFILE)" CODEX_MODEL="$(CODEX_MODEL)" CODEX_JSONL="$(CODEX_JSONL)"; \ fi +.PHONY: pm-prepare-proving-run-closeout +pm-prepare-proving-run-closeout: + @if [ -z "$(PROVING_RUN_FACTS)" ]; then echo "ERROR: set PROVING_RUN_FACTS="; exit 2; fi + @set -euo pipefail; \ + cmd="python3 docs/project_management/system/scripts/execution/prepare_proving_run_closeout.py --facts \"$(PROVING_RUN_FACTS)\" --output \"$(PROVING_RUN_CLOSEOUT_OUTPUT)\""; \ + if [ -n "$(PROVING_RUN_HUMAN_INPUTS)" ]; then cmd="$$cmd --human-inputs \"$(PROVING_RUN_HUMAN_INPUTS)\""; fi; \ + eval "$$cmd" + .PHONY: pm-fse-pre-planning-from-adr pm-fse-pre-planning-from-adr: @if [ -z "$(ADR)" ]; then echo "ERROR: set ADR=docs/project_management/adrs//ADR-XXXX-....md"; exit 2; fi diff --git a/ORCH_PLAN.md b/ORCH_PLAN.md new file mode 100644 index 000000000..9729508d6 --- /dev/null +++ b/ORCH_PLAN.md @@ -0,0 +1,650 @@ +# ORCH_PLAN: Async Persistent-Session Bootstrap Readiness Split + +Authoritative plan source: [PLAN.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/PLAN.md) +Primary protocol doc: [persistent_session.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/internals/repl/persistent_session.md) +World backend doc: [WORLD.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/WORLD.md) +Stale note: the existing durable-session orchestration content is out of scope for this run and is replaced by this controller. +Live repo root: `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate` +Fresh worktree root: `/Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-slice-26-readiness-split` +Baseline branch: `feat/host-orchestrator-durable-session` +Run id: `slice-26-async-persistent-session-bootstrap-readiness` +Max concurrent workers: `2` +Parent role: only integrator, only gate authority, only writer of `.runs` artifacts + +## Summary + +This controller operationalizes [PLAN.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/PLAN.md) for the async persistent-session bootstrap readiness split. + +Execution shape is fixed: + +1. Parent source-locks the plan and creates a clean integration branch from the current slice branch. +2. Worker A lands `A1`, which freezes the shell-facing async readiness contract without widening sync surfaces. +3. Workers B and C run in the only parallel window, on disjoint backend crates. +4. Worker A returns for `A2` to wire the shell caller migration and sync-regression proof against the merged backend seams. +5. Worker D updates docs late. +6. Parent runs the full validation wall and decides final acceptance. + +This preserves the slice contract: + +1. no async `WorldBackend` rewrite +2. sync `ensure_ready` preserved for sync callers +3. macOS async persistent-session startup stops using the sync bridge +4. socket override bypass remains exact +5. backend-owned readiness stays backend-owned +6. WSL work is internal parity only +7. tests, docs, and validation are mandatory +8. concurrency stays honest + +## Hard Guards + +1. `WorldBackend` remains synchronous for this slice. +2. `PlatformWorldContext.ensure_ready` remains available and unchanged for sync callers. +3. The macOS persistent-session async startup path must stop calling the sync bridge. +4. `SUBSTRATE_WORLD_SOCKET` override behavior must remain exact, including bypass semantics. +5. Shell code may dispatch to backend-owned readiness. It may not duplicate VM, forwarding, client, or capability logic. +6. Linux behavior stays structurally unchanged unless a tiny compile-only adapter is needed. +7. WSL hardening is internal-only and must not create a shipped Windows persistent-session shell caller. +8. Readiness failures stay fail closed and return normal errors, never a Tokio panic and never best-effort degraded startup. +9. The `A1` shell contract must preserve a concrete backend-owned async-ready access path. It is not enough to add a free shell helper if `PlatformWorldContext` still erases the backend behind `Arc` with no async-ready access surface. +10. The no-panic regression must exercise a no-override path that previously crossed `ctx.ensure_ready()`. `SUBSTRATE_WORLD_SOCKET` override tests remain required, but they are not valid evidence for the original panic fix. +11. Docs ship only after merged behavior is stable. +12. Workers do not edit [PLAN.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/PLAN.md), [ORCH_PLAN.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/ORCH_PLAN.md), or `.runs/**`. +13. Every symbol edit requires prior GitNexus impact analysis. +14. Any `HIGH` or `CRITICAL` GitNexus impact result is a parent-only escalation point. +15. Every worker handoff must include `gitnexus_detect_changes()` status before the parent considers merge. +16. Parent runs a final `gitnexus_detect_changes()` on the merged branch before closeout. + +## Worktree Creation Order And Commands + +Parent creates worktrees in this exact order. + +### 1. Create parent integration worktree + +```bash +mkdir -p /Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-slice-26-readiness-split + +git -C /Users/spensermcconnell/__Active_Code/atomize-hq/substrate fetch origin + +git -C /Users/spensermcconnell/__Active_Code/atomize-hq/substrate worktree add \ + /Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-slice-26-readiness-split/parent \ + -b codex/slice-26-async-persistent-session-readiness \ + feat/host-orchestrator-durable-session +``` + +### 2. Create Worker A worktree for `A1` + +```bash +git -C /Users/spensermcconnell/__Active_Code/atomize-hq/substrate worktree add \ + /Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-slice-26-readiness-split/shell-lane \ + -b codex/slice-26-shell-readiness-seam \ + codex/slice-26-async-persistent-session-readiness +``` + +### 3. After `G1`, create Worker B and Worker C worktrees + +```bash +git -C /Users/spensermcconnell/__Active_Code/atomize-hq/substrate worktree add \ + /Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-slice-26-readiness-split/mac-lima-lane \ + -b codex/slice-26-mac-lima-readiness-split \ + codex/slice-26-async-persistent-session-readiness + +git -C /Users/spensermcconnell/__Active_Code/atomize-hq/substrate worktree add \ + /Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-slice-26-readiness-split/wsl-lane \ + -b codex/slice-26-wsl-readiness-parity \ + codex/slice-26-async-persistent-session-readiness +``` + +### 4. After `G2`, refresh Worker A for `A2` + +If the original Worker A worktree is still clean and easy to reuse, rebase it on the parent branch. Otherwise, remove and recreate it. + +```bash +git -C /Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-slice-26-readiness-split/shell-lane fetch origin + +git -C /Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-slice-26-readiness-split/shell-lane rebase \ + codex/slice-26-async-persistent-session-readiness +``` + +If recreate is cleaner: + +```bash +git -C /Users/spensermcconnell/__Active_Code/atomize-hq/substrate worktree remove \ + /Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-slice-26-readiness-split/shell-lane + +git -C /Users/spensermcconnell/__Active_Code/atomize-hq/substrate worktree add \ + /Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-slice-26-readiness-split/shell-lane \ + -B codex/slice-26-shell-readiness-seam \ + codex/slice-26-async-persistent-session-readiness +``` + +### 5. After `G3`, create Worker D worktree + +```bash +git -C /Users/spensermcconnell/__Active_Code/atomize-hq/substrate worktree add \ + /Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-slice-26-readiness-split/docs-lane \ + -b codex/slice-26-docs-readiness-split \ + codex/slice-26-async-persistent-session-readiness +``` + +## Parent-Owned Run-State Surface / Artifact Ledger + +Canonical run root: + +- `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/slice-26-async-persistent-session-bootstrap-readiness/` + +Required directory tree: + +```text +.runs/slice-26-async-persistent-session-bootstrap-readiness/ + run-state.json + source-lock.json + branch-map.json + ownership-boundaries.json + task-ledger.json + merge-log.md + final-summary.md + blocked.json # only if run blocks + sentinels/ + RUN_OPEN + RUN_BLOCKED # only if blocked + RUN_COMPLETE # only if complete + gates/ + G0-source-lock/ + gate.json + evidence.md + OPEN + G1-shell-contract-freeze/ + gate.json + evidence.md + OPEN + REOPENED # touched only if contract drift forces reset + G2-backend-parallel-window/ + gate.json + evidence.md + OPEN + G3-shell-migration/ + gate.json + evidence.md + OPEN + G4-docs-closeout/ + gate.json + evidence.md + OPEN + G5-final-acceptance/ + gate.json + evidence.md + OPEN + tasks/ + P0-source-lock/ + P1-parent-branch-init/ + A1-shell-contract-freeze/ + G1-shell-contract-accept/ + B1-mac-lima-readiness-split/ + C1-wsl-parity-hardening/ + G2-backend-accept/ + A2-shell-migration-and-sync-regressions/ + G3-shell-accept/ + D1-docs-closeout/ + G4-docs-accept/ + P2-final-validation-wall/ + P3-final-gitnexus-closeout/ + P4-final-summary/ +``` + +Required per-task artifact files under each `tasks//` directory: + +1. `task.json` +2. `owner.txt` +3. `status.txt` +4. `dependency-status.json` +5. `scope.txt` +6. `deliverable.txt` +7. `acceptance-notes.md` +8. `changed-files.txt` +9. `commands.txt` +10. `exit-codes.json` +11. `impact-analysis-summary.md` +12. `gitnexus-detect-changes.txt` +13. `handoff-notes.md` +14. `blocker-notes.md` if blocked +15. `HEAD_SHA.txt` +16. sentinel file exactly one of: + - `READY_FOR_REVIEW` + - `ACCEPTED` + - `REJECTED` + - `BLOCKED` + +Rules: + +1. Workers do not write these artifacts directly. +2. Workers return the content in their handoff. +3. Parent transcribes the handoff into `.runs/**`. +4. No task is considered complete until the parent has written the task artifacts and touched the correct sentinel. + +## GitNexus Workflow + +GitNexus is an active part of the run, not a passive guard. + +### Source-lock stage + +1. Parent checks GitNexus availability and index freshness. +2. If stale, parent runs `npx gitnexus analyze` from the parent worktree before any worker starts. +3. Parent records GitNexus freshness status in `tasks/P0-source-lock/impact-analysis-summary.md`. + +### Before each worker edits symbols + +The worker must run impact analysis on the symbols it expects to touch and include a concise summary in handoff. Minimum required targets by lane: + +- `A1` + - `PlatformWorldContext` + - `build_ws_and_start_session_frame` +- `B1` + - `MacLimaBackend::ensure_session` + - any backend-local readiness helper being split +- `C1` + - `WindowsWslBackend::ensure_session` + - any backend-local readiness helper being split +- `A2` + - `ReplPersistentSessionClient::start_with` + - `open_world_session` + - any sync shell caller or regression surface being touched +- `D1` + - no GitNexus impact required if docs-only, but worker still reports docs-only status + +Escalation rule: + +1. If any impact analysis returns `HIGH` or `CRITICAL`, the worker stops before editing and returns a blocker handoff. +2. Parent records the blocker, decides whether the slice still fits [PLAN.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/PLAN.md), and either relaunches with narrower scope or blocks the run. + +### Before each worker handoff + +Each coding lane runs `gitnexus_detect_changes()` and reports: + +1. pass or fail +2. changed symbols +3. changed execution flows +4. whether the result matches the intended lane scope + +### Before final closeout + +Parent runs `gitnexus_detect_changes()` on the merged parent branch and records the result under `tasks/P3-final-gitnexus-closeout/`. + +The run cannot close green if final GitNexus detection shows unexpected symbol drift that the parent cannot explain from the task ledger. + +## Frozen Ownership Boundaries + +### Worker A, phase `A1`: shell contract freeze only + +Owned files: + +1. [crates/shell/src/execution/platform_world/mod.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/platform_world/mod.rs) +2. [crates/shell/src/execution/routing/dispatch/world_persistent_session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_persistent_session.rs) +3. [crates/shell/src/execution/platform_world/windows.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/platform_world/windows.rs) if the context shape changes require the Windows detector to populate the same async-ready access surface + +Purpose: + +1. add the shell-facing async readiness seam +2. preserve sync `ensure_ready` +3. preserve a backend-owned async-ready access path despite backend type erasure in `PlatformWorldContext` +4. freeze helper names, dispatch shape, and error-shape expectations for backend lanes + +Not allowed in `A1`: + +1. no edits to `async_repl.rs` +2. no backend crate edits +3. no docs +4. no attempt to prove the macOS no-panic regression through `SUBSTRATE_WORLD_SOCKET` override-only tests +5. no sync regression additions outside minimal compile coverage + +### Worker B, phase `B1`: macOS backend split + +Owned files: + +1. [crates/world-mac-lima/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/lib.rs) + +Purpose: + +1. split shared setup and async verification from the sync wrapper +2. keep sync `ensure_session(...)` working +3. harden `block_on_compat(...)` as defense-in-depth + +### Worker C, phase `C1`: WSL parity hardening + +Owned files: + +1. [crates/world-windows-wsl/src/backend.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-windows-wsl/src/backend.rs) +2. [crates/world-windows-wsl/src/tests.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-windows-wsl/src/tests.rs) + +Purpose: + +1. mirror the readiness split internally +2. preserve existing sync behavior +3. explicitly avoid shipping a Windows persistent-session caller + +### Worker A, phase `A2`: shell caller migration and sync regressions + +Owned files: + +1. [crates/shell/src/execution/routing/dispatch/world_persistent_session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_persistent_session.rs) +2. [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) +3. [crates/shell/src/execution/routing/world.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/world.rs) +4. [crates/shell/src/execution/routing/dispatch/world_ops.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs) +5. [crates/shell/src/execution/workspace_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/workspace_cmd.rs) +6. [crates/shell/tests/repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs) +7. any directly related shell-side tests required by the slice + +Purpose: + +1. replace the macOS sync bridge with the async seam +2. preserve exact socket override behavior +3. land the honest no-override no-panic regression separately from the override-path regression +4. prove sync callers still use sync `ensure_ready` + +Reason `A1` and `A2` are serialized: + +1. `A1` defines the shell-to-backend seam consumed by `B1` and `C1` +2. `A2` cannot honestly finish until the backend adapters are merged and stable +3. merging them into one long shell lane would either block the backend window or force backend workers to guess the final shell contract + +### Worker D, phase `D1`: docs late + +Owned files: + +1. [docs/internals/repl/persistent_session.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/internals/repl/persistent_session.md) +2. [docs/WORLD.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/WORLD.md) +3. optional [llm-last-mile/README.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/README.md) only if needed for index hygiene + +Purpose: + +1. document the shipped caller-shape split +2. document the Windows parity decision as internal only + +## Detailed Task Ledger + +| Task ID | Owner | Worktree / Branch | Depends on | Deliverable | Acceptance notes | +| --- | --- | --- | --- | --- | --- | +| `P0` | Parent | `parent` / `codex/slice-26-async-persistent-session-readiness` | none | source lock, GitNexus freshness check, run-state init | `source-lock.json` exists, `RUN_OPEN` touched | +| `P1` | Parent | `parent` / `codex/slice-26-async-persistent-session-readiness` | `P0` | parent branch ready, Worker A launched | clean parent branch exists, `branch-map.json` written | +| `A1` | Worker A | `shell-lane` / `codex/slice-26-shell-readiness-seam` | `P1` | shell-facing async readiness seam only | sync `ensure_ready` preserved, no backend or `async_repl.rs` edits | +| `G1` | Parent | `parent` / same | `A1` | shell contract accepted | contract artifact written, gate `G1` opened | +| `B1` | Worker B | `mac-lima-lane` / `codex/slice-26-mac-lima-readiness-split` | `G1` | macOS backend shared sync/async readiness split | crate tests green, no shell edits | +| `C1` | Worker C | `wsl-lane` / `codex/slice-26-wsl-readiness-parity` | `G1` | WSL parity hardening and tests | crate tests green, no product-surface expansion | +| `G2` | Parent | `parent` / same | `B1`, `C1` | backend window accepted | both backend merges recorded, gate `G2` opened | +| `A2` | Worker A | `shell-lane` / `codex/slice-26-shell-readiness-seam` rebased | `G2` | shell caller migration plus sync regressions | macOS path off sync bridge, no-override panic regression present, socket override exact, shell tests green | +| `G3` | Parent | `parent` / same | `A2` | shell migration accepted | merged tree satisfies code-path split, gate `G3` opened | +| `D1` | Worker D | `docs-lane` / `codex/slice-26-docs-readiness-split` | `G3` | docs updates only | docs match merged code, no code edits | +| `G4` | Parent | `parent` / same | `D1` | docs accepted | gate `G4` opened | +| `P2` | Parent | `parent` / same | `G4` | final validation wall | all commands executed and logged | +| `P3` | Parent | `parent` / same | `P2` | final GitNexus detect-changes closeout | output matches expected scope | +| `P4` | Parent | `parent` / same | `P3` | final summary and run completion | `RUN_COMPLETE` touched | + +## Lane Handoff Requirements + +Every worker handoff must include, at minimum: + +1. `HEAD` commit SHA +2. exact changed file list +3. exact commands run +4. per-command exit codes +5. concise GitNexus impact summary +6. whether any impact call returned `HIGH` or `CRITICAL` +7. whether `gitnexus_detect_changes()` passed +8. concise explanation of any unexpected file drift +9. blocker notes if not green +10. statement of whether the lane stayed within ownership boundaries + +Parent records this under the matching `tasks//` artifact directory. + +### Minimum handoff packet format for coding lanes + +1. `changed-files.txt` +2. `commands.txt` +3. `exit-codes.json` +4. `impact-analysis-summary.md` +5. `gitnexus-detect-changes.txt` +6. `handoff-notes.md` +7. `HEAD_SHA.txt` + +### Required lane-specific notes + +`A1` must additionally state: + +1. the exact new shell-facing async helper name +2. which backend-facing contract points were frozen +3. how the platform-world layer preserved concrete backend-owned async-ready access +4. whether any shell test additions were deferred to `A2` + +`B1` must additionally state: + +1. which sync and async backend-local helpers now exist +2. whether `block_on_compat(...)` behavior changed only as hardening +3. whether any shell contract mismatch was discovered + +`C1` must additionally state: + +1. which backend-local parity helpers now exist +2. whether any Windows product-surface change was avoided explicitly +3. whether any shell contract mismatch was discovered + +`A2` must additionally state: + +1. which call site stopped using the sync bridge +2. how socket override bypass was preserved +3. where the no-override current-thread no-panic regression lives +4. which sync caller regressions were tested + +`D1` must additionally state: + +1. docs touched +2. exact statements added for macOS async split and Windows internal-only scope + +## Task Graph And Control Points + +```mermaid +graph TD + P0["P0 Source lock + GitNexus freshness"] --> P1["P1 Parent branch init"] + P1 --> A1["A1 Shell contract freeze"] + A1 --> G1["G1 Parent accepts shell contract"] + G1 --> B1["B1 macOS backend split"] + G1 --> C1["C1 WSL parity hardening"] + B1 --> G2["G2 Parent accepts backend window"] + C1 --> G2 + G2 --> A2["A2 Shell migration + sync regressions"] + A2 --> G3["G3 Parent accepts shell migration"] + G3 --> D1["D1 Docs closeout"] + D1 --> G4["G4 Parent accepts docs"] + G4 --> P2["P2 Final validation wall"] + P2 --> P3["P3 Final GitNexus closeout"] + P3 --> P4["P4 Final summary"] +``` + +## Validation Commands By Task + +These are the required commands, not examples. + +### `A1` validation commands + +Run from `/Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-slice-26-readiness-split/shell-lane`: + +```bash +cargo fmt --all +cargo test -p shell --lib -- --nocapture +``` + +### `B1` validation commands + +Run from `/Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-slice-26-readiness-split/mac-lima-lane`: + +```bash +cargo fmt --all +cargo test -p world-mac-lima -- --nocapture +``` + +### `C1` validation commands + +Run from `/Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-slice-26-readiness-split/wsl-lane`: + +```bash +cargo fmt --all +cargo test -p world-windows-wsl -- --nocapture +``` + +### `A2` validation commands + +Run from `/Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-slice-26-readiness-split/shell-lane` after rebase to the parent branch: + +```bash +cargo fmt --all +cargo test -p shell --lib -- --nocapture +cargo test -p shell --test repl_world_first_routing_v1 -- --nocapture +cargo test -p shell -- --nocapture +``` + +### `D1` validation commands + +Run from `/Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-slice-26-readiness-split/docs-lane`: + +```bash +git diff -- docs/internals/repl/persistent_session.md docs/WORLD.md llm-last-mile/README.md +``` + +### Parent final validation wall in `P2` + +Run from `/Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-slice-26-readiness-split/parent`: + +```bash +cargo fmt --all -- --check +cargo clippy --workspace --all-targets -- -D warnings +cargo test -p shell -- --nocapture +cargo test -p world-mac-lima -- --nocapture +cargo test -p world-windows-wsl -- --nocapture +script -q /dev/null zsh -lc 'RUST_BACKTRACE=1 ~/.substrate/bin/substrate' +``` + +## Conflict-Resolution Rule For Cross-Lane Contract Drift + +Contract drift after `G1` is handled explicitly. + +If `B1` or `C1` discovers that the `A1` shell contract is insufficient or wrong: + +1. The worker stops immediately. +2. The worker does not patch shell files locally. +3. The worker returns a blocker handoff describing the exact contract mismatch. +4. Parent writes `BLOCKED` for that task and touches `REOPENED` under `gates/G1-shell-contract-freeze/`. +5. Parent closes downstream acceptance for unmerged work based on the stale contract. +6. Parent either: + - launches a narrow `A1R1` correction on the shell lane, or + - blocks the run if the required change violates [PLAN.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/PLAN.md) +7. If `A1R1` lands, parent updates: + - `source-lock.json` only if the plan authority itself changed + - `gates/G1-shell-contract-freeze/gate.json` + - `ownership-boundaries.json` if file boundaries changed + - `task-ledger.json` +8. Parent then relaunches affected backend lanes from the new parent base. Old backend commits against the stale contract are not merged opportunistically. + +## Gate Rules And Stop Conditions + +### Gate `G1` acceptance rule + +Open only when: + +1. `A1` handoff is complete +2. parent confirms sync `ensure_ready` still exists for sync callers +3. parent confirms the frozen shell contract preserves a backend-owned async-ready access path for downstream backend lanes +4. parent confirms `A1` did not spill into backend crates or `async_repl.rs` +5. parent has a written shell contract artifact under `tasks/G1-shell-contract-accept/` + +### Gate `G2` acceptance rule + +Open only when: + +1. `B1` and `C1` are both merged into the parent branch +2. their required crate tests passed +3. neither lane expanded public Windows persistent-session surface +4. neither lane forced shell changes after `G1` without reopening the gate + +### Gate `G3` acceptance rule + +Open only when: + +1. `A2` is merged +2. macOS persistent-session startup no longer uses the sync bridge +3. socket override bypass is still exact +4. a no-override current-thread regression covers the former panic path without relying on `SUBSTRATE_WORLD_SOCKET` +5. sync caller regressions are covered and green + +### Gate `G4` acceptance rule + +Open only when: + +1. docs are based on the merged parent branch +2. docs explicitly describe: + - async persistent-session startup using a dedicated async readiness seam + - sync request-builders continuing to use sync `ensure_ready` + - WSL work in this slice being internal-only parity hardening + +### Run stop conditions + +Write `blocked.json`, touch `RUN_BLOCKED`, and stop if any are true: + +1. the slice requires an async `WorldBackend` +2. sync `ensure_ready` cannot be preserved +3. exact socket override bypass cannot be preserved +4. shell code must duplicate backend readiness logic to succeed +5. a backend lane requires a shipped Windows persistent-session caller to validate the design +6. final manual smoke still produces a Tokio panic +7. final GitNexus detect-changes shows unexpected scope the parent cannot justify +8. required macOS smoke cannot be run honestly and the user has not explicitly approved an evidence waiver + +## Final Acceptance Wall + +The run passes only if all of the following are true on the merged parent branch: + +1. Persistent-session startup no longer calls the sync readiness bridge on the macOS async path. +2. macOS current-thread REPL startup no longer panics on `block_in_place`. +3. Readiness failures surface as normal `Result` errors. +4. Existing sync bootstrap and request-builder callers still use sync `ensure_ready`. +5. Shared readiness rules remain backend-owned across sync and async entrypoints. +6. `SUBSTRATE_WORLD_SOCKET` override bypass remains exact. +7. WSL parity hardening stayed internal-only. +8. Required tests are present and pass, including a no-override current-thread regression for the former panic path and a separate override-path regression. +9. Docs clearly encode the caller-shape split and Windows scope decision. +10. Final GitNexus detect-changes matches expected scope. + +### Manual smoke expectation + +Primary honest smoke: + +1. Run the parent `script -q /dev/null zsh -lc 'RUST_BACKTRACE=1 ~/.substrate/bin/substrate'` check in a macOS environment where the current-thread REPL path is meaningful for this slice. +2. Pass if startup succeeds without panic. +3. Pass if startup fails as a normal readiness error without Tokio runtime panic. +4. Fail if a Tokio panic or `block_in_place` runtime invariant panic is observed. + +### If the environment cannot run the smoke honestly + +This does not default to pass. + +Rules: + +1. Parent records `SMOKE_UNAVAILABLE` in `tasks/P2-final-validation-wall/acceptance-notes.md`. +2. Parent explains exactly why the environment is not honest for this smoke. +3. Parent marks the run blocked unless the user explicitly accepts a smoke-evidence waiver. +4. If a waiver is granted, parent still records the gap in `final-summary.md`. The run is “accepted with waived manual smoke,” not “fully validated.” + +## Tests And Acceptance Mapping To PLAN.md + +The parent should use this direct mapping when deciding final acceptance: + +1. PLAN completion point `1`: covered by `A2`, `G3`, and final code inspection +2. PLAN completion point `2`: covered by the no-override current-thread regression in `A2` plus manual smoke +3. PLAN completion point `3`: covered by shell and backend error-path tests and smoke failure mode +4. PLAN completion point `4`: covered by `A1`, `A2`, and sync shell regressions +5. PLAN completion point `5`: covered by `B1`, `C1`, `A2`, and docs +6. PLAN completion point `6`: covered by `C1` scope notes and docs +7. PLAN completion point `7`: covered by `D1` and parent doc review + +## Assumptions + +1. The parent will create a clean integration worktree from the current slice branch, `feat/host-orchestrator-durable-session`, and treat the current [PLAN.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/PLAN.md) text as the locked slice authority in run-state artifacts. +2. If the manual smoke environment is unavailable, that is a real acceptance gap, not a silent skip. +3. The parent can access GitNexus through the required MCP workflow or equivalent repo-supported invocation before worker edits begin. diff --git a/PLAN.md b/PLAN.md new file mode 100644 index 000000000..874f75f78 --- /dev/null +++ b/PLAN.md @@ -0,0 +1,552 @@ +# PLAN: Async Persistent-Session Bootstrap Readiness Split + +Source SOW: [26-async-persistent-session-bootstrap-readiness.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/26-async-persistent-session-bootstrap-readiness.md) +Related slices: [21-macos-lima-shared-owner-and-member-runtime-parity.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/21-macos-lima-shared-owner-and-member-runtime-parity.md), [24-fix-host-bootstrap-readiness-and-clean-detach-parking.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/24-fix-host-bootstrap-readiness-and-clean-detach-parking.md), [25-host-durable-session-closeout-and-qa-hardening.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/25-host-durable-session-closeout-and-qa-hardening.md) +Primary protocol doc: [persistent_session.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/internals/repl/persistent_session.md) +World backend docs: [WORLD.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/WORLD.md) +Supersedes: previous root `PLAN.md`, which tracked durable host-orchestrator closeout instead of the slice-26 readiness split +Current workspace branch: `feat/host-orchestrator-durable-session` +Base branch: `main` +Plan type: backend/runtime correction, no UI scope, developer-facing runtime scope +Status: unified implementation plan, 2026-05-15 + +## Objective + +Fix the async persistent-session startup seam so REPL world-session bootstrap no longer crosses the synchronous readiness bridge on macOS, while preserving the existing synchronous bootstrap and request-builder surfaces. + +This is a design correction, not a trait rewrite. The slice lands one explicit async readiness path for persistent-session startup, keeps `WorldBackend` synchronous, preserves fail-closed behavior, and makes the Windows parity decision explicit. + +## Acceptance Criteria + +This plan is complete only when all of the following are true: + +1. Persistent-session startup no longer calls `PlatformWorldContext.ensure_ready` from [world_persistent_session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_persistent_session.rs). +2. macOS/Lima REPL startup under the current-thread Tokio runtime no longer panics with the `block_in_place` runtime invariant. +3. Readiness failures surface as normal `Result` errors, not panic unwinds. +4. Existing synchronous bootstrap and request-builder callers keep using `ensure_ready` with no async signature expansion. +5. Shared readiness rules remain consistent across sync and async entrypoints for VM start, forwarding, client construction, and backend-owned capabilities verification semantics. +6. The Windows/WSL backend is either hardened with the same split-ready internals or explicitly documented as a future-only consumer with no current persistent-session caller. +7. Tests and docs make the caller-shape split obvious enough that a future async caller does not regress back through the sync bridge. + +## Step 0: Scope Challenge + +### What already exists + +| Sub-problem | Existing code | Plan decision | +| --- | --- | --- | +| REPL async caller | [`open_world_session(...)` in async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs), [`ReplPersistentSessionClient::start_with(...)` in world_persistent_session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_persistent_session.rs) | Reuse. The caller is already async and stays async. | +| Persistent-session transport/bootstrap | [`build_ws_and_start_session_frame(...)` in world_persistent_session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_persistent_session.rs) | Reuse transport and handshake logic. Replace only the readiness bridge. | +| Sync platform-world context | [`PlatformWorldContext` in platform_world/mod.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/platform_world/mod.rs) | Keep `ensure_ready` for sync callers. Add a separate async-ready path instead of widening the existing one. | +| macOS readiness ownership | [`MacLimaBackend` in world-mac-lima/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/lib.rs) | Reuse backend-owned VM/forwarding/client logic. Split adapters, not ownership. | +| Windows readiness ownership | [`WindowsWslBackend` in world-windows-wsl/src/backend.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-windows-wsl/src/backend.rs) | Reuse. Harden parity logic now, but do not invent a Windows persistent-session feature in this slice. | +| Sync bootstrap callers | [`routing/world.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/world.rs), [`world_ops.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs), [`workspace_cmd.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/workspace_cmd.rs) | Preserve behavior. Regression-test them instead of rewriting them. | + +### Minimum honest diff + +The minimum honest implementation touches these module groups: + +1. [`crates/shell/src/execution/platform_world/`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/platform_world/) +2. [`crates/shell/src/execution/routing/dispatch/world_persistent_session.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_persistent_session.rs) +3. [`crates/world-mac-lima/src/lib.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/lib.rs) +4. [`crates/world-windows-wsl/src/backend.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-windows-wsl/src/backend.rs) +5. targeted tests in [`crates/shell/tests/`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/), macOS backend tests in [`crates/world-mac-lima/src/lib.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/lib.rs), and WSL tests in [`crates/world-windows-wsl/src/tests.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-windows-wsl/src/tests.rs) +6. docs in [persistent_session.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/internals/repl/persistent_session.md) and [WORLD.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/WORLD.md) + +Anything smaller is a shortcut. It would either leave the async caller on the sync bridge, duplicate readiness rules in shell code, or fail to prove the sync surfaces still behave correctly. + +### Repo-truth findings + +1. [`world_persistent_session.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_persistent_session.rs) and the REPL world-session entrypoints in [`async_repl.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) are compiled only on Linux and macOS today. The shell-side persistent-session seam is not a current Windows caller. +2. `PlatformWorldContext` still exposes one synchronous `ensure_ready` closure, and it is actively consumed by sync bootstrap and request-builder paths. Deleting or widening that surface in this slice is cross-cutting churn with no user payoff. +3. Both non-Linux backends already contain the same risk pattern: + - Lima uses `block_on_compat(...)` in [`world-mac-lima/src/lib.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/lib.rs). + - WSL uses `block_on(...)` in [`world-windows-wsl/src/backend.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-windows-wsl/src/backend.rs). +4. The real defect is not the websocket protocol and not `WorldBackend` itself. The defect is that an already-async caller is still forced through a sync bootstrap seam that uses sync-over-async capability verification. +5. [`PlatformWorldContext`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/platform_world/mod.rs) currently erases the concrete backend behind `Arc` and only stores the sync `ensure_ready` closure. Phase 1 therefore must preserve some concrete async-ready access path in the platform-world layer; a pure shell helper that "dispatches to the concrete backend" is not implementable unless that access path is added deliberately. + +### Complexity, completeness, and distribution verdict + +1. This is a medium slice. It touches more than 8 files once tests and docs are counted, but the spread is justified because the bug spans one async caller, one shared shell context, and two backend adapter implementations. +2. The complete fix is still the right fix. A hotfix inside `block_on_compat(...)` alone is cheaper to type, but it leaves the wrong architecture in place and guarantees the same confusion later. +3. No new binary, package, service, or distribution pipeline is introduced. Distribution is unchanged in this slice. +4. There is no repo-root `TODOS.md` today. Deferred work stays in `NOT in scope` and `Deferred follow-ups` instead of inventing a backlog surface mid-slice. + +### Scope ruling + +The scope is accepted as-is. It is bigger than a one-line hotfix, but smaller than a trait migration, and that is the right middle. + +## Architecture Review + +### Problem statement + +The repo already has a real async persistent-session startup path: + +```text +CURRENT BROKEN STARTUP +====================== +async REPL current-thread runtime + -> open_world_session(...) + -> ReplPersistentSessionClient::start_with(...) + -> build_ws_and_start_session_frame(...) + -> ctx.ensure_ready() [sync bridge] + -> backend.ensure_session(...) + -> ensure_agent_ready(...) + -> block_on_compat(client.capabilities()) + -> Tokio panic on macOS current-thread runtime +``` + +The mismatch is narrow but real: + +1. the caller is async, +2. the readiness bridge is sync, +3. the backend uses sync-over-async capability verification that is only safe under some runtime shapes. + +The fix is not "teach `block_in_place` a new trick." The fix is to stop sending an async caller through the sync bridge. + +### Architecture thesis + +Caller shape decides the bootstrap adapter. Readiness ownership stays with the backend, but the shell must stop pretending sync and async callers are the same thing. + +### Target architecture + +```text +TARGET STARTUP MODEL +==================== +async REPL current-thread runtime + -> open_world_session(...) + -> build_ws_and_start_session_frame(...) + -> ensure_persistent_session_ready_async(...) + -> backend-specific async readiness adapter + -> transport connect + -> websocket handshake + -> ready frame + +sync bootstrap / sync request-builders + -> ctx.ensure_ready() + -> backend.ensure_session(...) + -> backend-specific sync adapter + -> unchanged caller contract +``` + +### Readiness ownership split + +```text +BACKEND READINESS OWNERSHIP +=========================== +shared sync-safe steps + - ensure_vm_running() / warm backend + - ensure_forwarding() + - build_agent_client() + +shared async verification step + - capabilities probe + - preserve current per-backend timeout / retry behavior + - fail-closed error mapping + +sync adapter + -> shared setup + -> block_on(private_runtime_or_safe_bridge, async verification) + +async adapter + -> shared setup + -> await async verification directly +``` + +This is the whole design. No async `WorldBackend`, no shell-side Lima logic clone, no new protocol. + +### Caller contract matrix + +| Caller shape | Current entrypoint | Required readiness path after this slice | Notes | +| --- | --- | --- | --- | +| Async REPL persistent-session startup | [`open_world_session(...)` in async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) | `ensure_persistent_session_ready_async(...)` | Must never call `ctx.ensure_ready()` on macOS again. | +| Linux persistent-session startup | [`build_ws_and_start_session_frame(...)` in world_persistent_session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_persistent_session.rs) | existing Linux readiness path | Linux already uses the world-agent readiness flow and stays boring. | +| Sync platform bootstrap | [`routing/world.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/world.rs) | existing `ctx.ensure_ready()` | No signature or behavior rewrite. | +| Sync request builders | [`world_ops.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs) | existing `ctx.ensure_ready()` | Regression coverage required because these are easy to break silently. | +| Workspace sync capabilities checks | [`workspace_cmd.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/workspace_cmd.rs) | dedicated runtime `block_on(...)` already local to workspace sync | Not the main bug, but still part of the sync-surface preservation check. | + +### Frozen contract + +If implementation wants to violate any rule below, stop and revise the plan first. + +1. `WorldBackend` stays synchronous in this slice. +2. `PlatformWorldContext.ensure_ready` remains available and continues to serve sync callers. +3. Persistent-session startup must not call `ensure_ready` on macOS. Linux may keep its existing direct world-agent readiness path. +4. Explicit `SUBSTRATE_WORLD_SOCKET` overrides continue to bypass Lima detection and startup exactly as they do today. +5. VM/forwarder/client/capabilities rules remain backend-owned. Shell code may orchestrate, not duplicate. +6. All readiness failures remain fail closed. A missing VM, failed forwarder, failed capabilities probe, or bad websocket connect returns an error, never a best-effort degraded session. +7. Windows/WSL does not gain a new persistent-session product surface in this slice. Only internal readiness parity and hardening land there. +8. The platform-world layer must preserve a concrete backend-owned async-ready path for persistent-session startup, whether through an additional callback, a typed carrier, or another testable adapter surface. Shell code must not recover that access by duplicating backend logic. + +## Code Quality Review + +### Explicit design choices + +1. Add one shell-facing async helper for persistent-session startup. Do not generalize this into a broad "all world readiness is async now" abstraction. +2. Keep backend-owned readiness logic central. The shell helper dispatches to backend adapters; it does not reimplement VM boot, forwarding, or capability checks. +3. Split shared backend internals by behavior, not by platform folklore: + - shared setup, + - shared async verification, + - sync wrapper, + - async wrapper. +4. Because `PlatformWorldContext` currently erases concrete backend type information, Phase 1 must also define how the shell reaches the backend-owned async-ready adapter. The acceptable shapes are narrow: + - an additional async-ready callback stored alongside `ensure_ready`, + - a typed platform carrier that still preserves the sync surface, + - or an equivalent platform-world adapter that remains shell-thin and testable. +5. Harden sync bridge helpers as defense-in-depth only. If `block_on_compat(...)` or WSL `block_on(...)` are touched, the change is there to produce a controlled error instead of a panic, not to justify leaving the async caller on the sync bridge. +6. Preserve Linux simplicity. Do not backdoor a second non-Linux abstraction into Linux just for symmetry. + +### Naming and ownership expectations + +The plan is intentionally opinionated about the shape of the new seam: + +1. The shell helper should be named for the real caller, for example `ensure_persistent_session_ready_async(...)`, not a generic future-looking async world API. +2. macOS and WSL should each expose backend-local async readiness wrappers that reuse the same internal setup/verification steps as the sync wrapper. +3. `PlatformWorldContext` remains the sync compatibility carrier, but Phase 1 must deliberately preserve a backend-owned async-ready access path. A free helper is fine only if the platform-world layer exposes enough information to reach the real backend adapter without rebuilding readiness logic in shell code. + +### DRY guardrails + +The following are non-negotiable: + +1. No shell-side duplication of Lima forwarding or agent capability logic. +2. No one-off macOS-only bypass that leaves WSL with the same trap hidden in a different crate. +3. No separate definitions of "ready" for sync and async startup. + +## Implementation Plan + +### Phase summary + +| Phase | Purpose | Modules touched | Hard dependency | +| --- | --- | --- | --- | +| 1 | Define async shell bootstrap contract | `crates/shell/src/execution/platform_world/`, `crates/shell/src/execution/routing/dispatch/world_persistent_session.rs` | — | +| 2 | Split macOS readiness internals into shared setup plus sync/async adapters | `crates/world-mac-lima/` | 1 | +| 3 | Migrate persistent-session caller to async seam | `crates/shell/src/execution/routing/dispatch/`, `crates/shell/src/repl/` | 1, 2 | +| 4 | Harden WSL parity without adding new product surface | `crates/world-windows-wsl/` | 1 | +| 5 | Prove sync surfaces still behave the same | `crates/shell/tests/`, `crates/shell/src/execution/routing/` | 3 | +| 6 | Document the split and run smoke validation | `docs/`, `llm-last-mile/` | 3, 5 | + +### Phase 1: Add the async shell bootstrap seam + +Files: + +1. [platform_world/mod.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/platform_world/mod.rs) +2. [world_persistent_session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_persistent_session.rs) + +Actions: + +1. Introduce one shell-facing async helper for persistent-session startup. Its job is to: + - preserve the existing Linux readiness behavior, + - detect or reuse the platform world context for macOS, + - reach a backend-owned async-ready adapter through an explicit platform-world seam, + - and return only when readiness is sufficient to open the transport. +2. Because `PlatformWorldContext` currently stores only `Arc` plus sync `ensure_ready`, Phase 1 must add the missing async-ready access path explicitly. Acceptable implementations include an additional async callback or a typed platform carrier, but not shell-side duplication of backend logic. +3. Keep `PlatformWorldContext` usable by existing sync callers. Do not change its current `ensure_ready` contract. +4. Freeze the shell-to-backend async contract before parallel backend work starts. This is the merge point every other lane depends on. + +Exit criteria: + +1. The shell has one explicit async readiness seam for persistent-session startup. +2. No sync caller changes signature. +3. The shell contract is specific enough that macOS and WSL work can proceed in parallel without guessing. + +### Phase 2: Refactor macOS/Lima readiness into shared internals plus sync/async adapters + +Files: + +1. [world-mac-lima/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/lib.rs) + +Actions: + +1. Keep `ensure_vm_running()` and `ensure_forwarding()` as shared sync-safe setup steps. +2. Extract the capabilities verification portion of `ensure_agent_ready()` into an async helper that can be awaited directly after client construction, while preserving current Lima semantics. This slice does not add a new timeout or retry policy to macOS. +3. Keep `ensure_agent_ready()` as the sync wrapper used by `ensure_session(...)`. +4. Add a dedicated async readiness wrapper for persistent-session startup that reuses the same setup and verification rules. +5. Harden `block_on_compat(...)` so accidental future misuse returns a controlled error with context instead of a runtime panic. + +Exit criteria: + +1. macOS sync `ensure_session(...)` still works. +2. macOS async persistent-session bootstrap no longer touches `block_on_compat(...)` on the happy path. +3. Sync and async Lima readiness share the same setup and verification logic. + +### Phase 3: Migrate persistent-session startup to the async seam + +Files: + +1. [world_persistent_session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_persistent_session.rs) +2. [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) + +Actions: + +1. Leave `ReplSessionStartParams`, policy/env normalization, shared-world request wiring, and websocket framing alone. +2. Replace the macOS `ctx.ensure_ready` call with the new async readiness helper. +3. Preserve `SUBSTRATE_WORLD_SOCKET` short-circuiting exactly. The override path is already the clean escape hatch for tests and explicit custom sockets. +4. Keep Linux on its existing direct world-agent readiness path. + +Exit criteria: + +1. REPL startup remains one straight async flow. +2. The only semantic change is the readiness boundary before transport connect. +3. The macOS path no longer crosses the sync bridge. + +### Phase 4: WSL parity hardening, not product-surface expansion + +Files: + +1. [world-windows-wsl/src/backend.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-windows-wsl/src/backend.rs) +2. [world-windows-wsl/src/tests.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-windows-wsl/src/tests.rs) + +Actions: + +1. Split WSL capabilities verification and warm-retry logic into shared async internals plus a sync wrapper, preserving the existing WSL warm-and-retry behavior rather than introducing a new cross-platform policy. +2. Harden `block_on(...)` the same way macOS is hardened. +3. Do not add a Windows persistent-session shell caller in this slice. +4. Document in code comments and docs that this is backend-local parity work for future async callers, not shipped Windows persistent-session support. + +Exit criteria: + +1. WSL remains compile-safe and behaviorally unchanged for existing sync callers. +2. Future async callers have a safe backend-local seam waiting for them. +3. The Windows parity decision is explicit and test-backed. + +### Phase 5: Prove sync callers still work + +Files: + +1. [routing/world.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/world.rs) +2. [world_ops.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs) +3. [workspace_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/workspace_cmd.rs) + +Actions: + +1. Preserve call sites unless a tiny adapter shim is required for compilation. +2. Add regression tests that prove sync bootstrap and sync request-building still succeed and still fail closed. +3. Confirm that workspace sync capability checks remain unaffected by the new persistent-session path. + +Exit criteria: + +1. Sync callers remain boring. +2. No hidden async creep enters these surfaces. +3. Existing fail-closed semantics are preserved. + +### Phase 6: Docs and manual validation + +Files: + +1. [persistent_session.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/internals/repl/persistent_session.md) +2. [WORLD.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/WORLD.md) +3. optionally [llm-last-mile/README.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/README.md) if the packet index needs a note + +Actions: + +1. Document the caller-shape split explicitly. +2. Call out that persistent-session startup on macOS uses an async readiness adapter while sync request-builders still use the legacy sync seam. +3. Record the Windows parity decision so nobody mistakes "hardened internals" for "Windows persistent REPL shipped." +4. Run the manual repro smoke and confirm the failure mode is now either success or a normal readiness error. + +Exit criteria: + +1. The docs reflect the real architecture. +2. The next person reading the code does not have to reverse-engineer why the split exists. +3. The manual repro no longer panics. + +## Test Review + +Rust is the project runtime and `cargo test` remains the authoritative framework for this slice. + +### Code path coverage + +```text +CODE PATH COVERAGE +================== +[+] Async REPL startup + async_repl::start_world_session(...) + -> open_world_session(...) + -> ReplPersistentSessionClient::start_with(...) + -> build_ws_and_start_session_frame(...) + -> ensure_persistent_session_ready_async(...) + -> websocket connect / ready frame + Required tests: + - macOS current-thread startup succeeds without panic + - readiness failure returns Result error + - explicit SUBSTRATE_WORLD_SOCKET override still bypasses bootstrap + +[+] Sync bootstrap callers + routing/world.rs + -> ctx.ensure_ready() + -> backend.ensure_session(...) + world_ops.rs request builders + -> ctx.ensure_ready() + -> AgentClient transport selection + workspace_cmd.rs capabilities checks + -> dedicated runtime.block_on(...) + Required tests: + - sync bootstrap unchanged + - sync request builders unchanged + - fail-closed behavior preserved + +[+] Backend adapters + MacLimaBackend + -> ensure_vm_running() + -> ensure_forwarding() + -> async capabilities probe + WindowsWslBackend + -> warm / retry path + -> async capabilities probe + Required tests: + - async adapter success + - probe failure returns normal error + - sync adapter still works +``` + +### User-flow and error-state coverage + +```text +USER FLOW COVERAGE +================== +[+] macOS REPL startup, default transport + - world session starts under current-thread runtime + - user sees normal REPL startup, not Tokio panic + +[+] macOS REPL startup, readiness failure + - VM unavailable / forwarder unavailable / capabilities probe fails + - user sees explicit readiness failure + +[+] macOS REPL startup, explicit socket override + - no Lima bootstrap side effects + - custom socket still connects + +[+] Sync command surfaces + - world bootstrap still works + - world_ops request builders still work + - workspace capabilities checks still work + +[+] WSL backend hardening + - existing sync flows unaffected + - future async callers no longer depend on unsafe sync-over-async behavior +``` + +### Concrete test additions + +| Target | Test requirement | Type | +| --- | --- | --- | +| shell-side current-thread test near [`world_persistent_session.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_persistent_session.rs) or an adjacent focused shell test | Add a no-override current-thread regression that exercises the path which previously called `ctx.ensure_ready()` and proves startup now returns success or a normal error without panic. | focused unit/integration | +| [`crates/shell/tests/repl_world_first_routing_v1.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs) | Preserve explicit `SUBSTRATE_WORLD_SOCKET` override bypass regression coverage only. This harness is valid for override-path behavior, not for reproducing the original panic. | integration | +| [`world_persistent_session.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_persistent_session.rs) tests | Cover async readiness success, readiness error propagation, and any shell-side helper behavior needed to prove the caller-shape split. | focused unit/integration | +| [`world-mac-lima/src/lib.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/lib.rs) tests | Use the existing `session_setup_override` seam to prove sync and async adapters share readiness rules. | crate unit | +| [`world-windows-wsl/src/tests.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-windows-wsl/src/tests.rs) | Cover async-parity helper behavior and sync-wrapper preservation. | crate unit | +| [`routing/world.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/world.rs) and [`world_ops.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs) | Add sync regression coverage that proves unchanged success and fail-closed behavior. | integration | + +### Regression rule for this slice + +The macOS REPL panic is a regression against an existing command surface. That makes the no-panic REPL startup test mandatory. No defer, no follow-up issue, no "manual smoke is enough." + +The override-backed REPL harness is not sufficient by itself for this regression because `SUBSTRATE_WORLD_SOCKET` bypasses platform detection and the sync readiness bridge. The no-panic regression must execute a no-override path that would have crossed `ctx.ensure_ready()` before this slice. + +## Failure Modes Registry + +| Failure mode | Where it happens | Required handling | Test requirement | Critical gap if missing | +| --- | --- | --- | --- | --- | +| Tokio current-thread panic | macOS REPL startup | Eliminate by moving async caller off sync bridge | REPL startup regression | Yes | +| VM starts but forwarding is absent | macOS async adapter | Return fail-closed readiness error | macOS adapter test | Yes | +| Capabilities probe hangs or fails | macOS + WSL adapters | Return normal error with context, no panic, while preserving existing per-backend retry/timeout semantics | adapter timeout/failure tests | Yes | +| Explicit socket override accidentally triggers platform bootstrap | macOS startup override path | Preserve short-circuit bypass | override regression test | Yes | +| Sync bootstrap silently starts using async path | `routing/world.rs` or `world_ops.rs` | Keep sync callers on `ensure_ready` | sync regression tests | Yes | +| Future Windows async caller repeats the same trap | WSL backend | Land async-parity internals and harden sync bridge | WSL parity tests | No, but document if partially deferred | + +Any failure mode that has no test, no error handling, and only a silent failure is a release blocker for this slice. + +## Performance Review + +This slice is about startup correctness, but there are still performance guardrails: + +1. No duplicate readiness passes. The async path should perform one readiness preparation before transport connect, not re-probe capabilities again immediately after connect unless that is already part of the existing handshake contract. +2. No extra VM warm cycles. Shared setup must remain idempotent enough that moving to an async adapter does not accidentally start the Lima VM twice. +3. No hidden runtime creation on the async happy path. The point of the async seam is to avoid bridging through a blocking helper. +4. Timeout and retry policy must remain backend-owned and unchanged in spirit. Do not silently lengthen startup latency just to mask the design bug. +5. Steady-state REPL throughput is unchanged. There is no new hot-path work after session start. + +## NOT in scope + +1. Converting [`WorldBackend`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-api/src/lib.rs) into an async trait. +2. Rewriting LinuxLocalBackend. +3. Redesigning the persistent-session protocol or `ready` frame contract. +4. Broad transport refactors unrelated to readiness. +5. Shipping Windows persistent-session REPL support as a new product surface. +6. Changing public CLI behavior outside of replacing the panic with normal readiness errors. + +## Deferred Follow-Ups + +1. Full async `WorldBackend` migration, if future async callers justify the cost. +2. Broader unification of sync request-builders if those surfaces ever move fully async. +3. Any Windows persistent-session feature work beyond backend hardening. + +## Worktree Parallelization Strategy + +### Dependency table + +| Step | Modules touched | Depends on | +| --- | --- | --- | +| 1. Define async shell bootstrap contract | `crates/shell/src/execution/platform_world/`, `crates/shell/src/execution/routing/dispatch/` | — | +| 2. Refactor macOS shared readiness internals | `crates/world-mac-lima/` | 1 | +| 3. Harden WSL parity internals | `crates/world-windows-wsl/` | 1 | +| 4. Migrate persistent-session caller to async seam | `crates/shell/src/execution/routing/dispatch/`, `crates/shell/src/repl/` | 1, 2 | +| 5. Add sync-regression coverage | `crates/shell/tests/`, `crates/shell/src/execution/routing/` | 4 | +| 6. Update docs and smoke notes | `docs/`, `llm-last-mile/` | 4, 5 | + +### Parallel lanes + +Lane A: Step 1 -> Step 4 -> Step 5 +Reason: shared shell modules, sequential changes, one owner + +Lane B: Step 2 +Reason: isolated macOS backend work once Step 1 contract is frozen + +Lane C: Step 3 +Reason: isolated WSL backend hardening once Step 1 contract is frozen + +Lane D: Step 6 +Reason: doc work can start after Step 4 stabilizes, then finish after Step 5 validation lands + +### Execution order + +1. Land Step 1 first. This defines the contract every other lane depends on. +2. Launch Lane B and Lane C in parallel worktrees after Step 1 merges, or after Step 1 is frozen enough that function names and result shapes will not churn. +3. Merge Lane B before Lane A Step 4 if the shell migration depends on the final macOS adapter names. +4. Run Lane A Step 4 after Lane B is stable, because the shell caller migration is the user-visible fix. +5. Run Lane A Step 5 after Step 4, because sync-regression coverage must validate the final shell wiring. +6. Finish Lane D last, after test names and final behavior are known. + +### Conflict flags + +1. Lane A and Lane B share the shell-to-backend contract even though they do not share the same files. Freeze function names and error-shape expectations before parallelizing. +2. Lane A Step 5 touches shell routing modules that may also pick up compile fixes from Step 4. Keep those changes in the same lane to avoid merge churn. +3. Lane B and Lane C are safe in parallel. They touch different backend crates. +4. Do not split Step 4 and Step 5 across different worktrees. Both touch the shell routing surface and will conflict. + +## Validation Commands + +```bash +cargo fmt --all +cargo clippy --workspace --all-targets -- -D warnings +cargo test -p substrate-shell -- --nocapture +cargo test -p world-mac-lima -- --nocapture +cargo test -p world-windows-wsl -- --nocapture +script -q /dev/null zsh -lc 'RUST_BACKTRACE=1 ~/.substrate/bin/substrate' +``` + +If the macOS manual smoke still fails, it must fail as a normal readiness error. A Tokio runtime panic means the slice is not done. + +## Completion Summary + +This plan is ready to implement when the work lands with this end state: + +1. Step 0: scope accepted as-is, no trait rewrite, no protocol rewrite. +2. Architecture: one async persistent-session readiness seam added, sync seam preserved. +3. Code quality: backend-owned readiness logic shared, no shell-side duplication, no generic future-API abstraction. +4. Tests: REPL startup regression, socket override regression, sync surface regressions, macOS adapter coverage, WSL parity coverage. +5. Performance: no duplicate readiness pass, no extra VM warm cycle, no blocking bridge on the async happy path. +6. `NOT in scope`: written. +7. `What already exists`: written. +8. Failure modes: six named production failures accounted for, five of them critical if left untested. +9. Parallelization: four lanes, two backend lanes safely parallel after the shell contract lands. +10. Lake score: the complete fix wins over the hotfix. The repo pays the right engineering cost once instead of paying the bug twice. diff --git a/config/agents/codex.yaml b/config/agents/codex.yaml new file mode 100644 index 000000000..9c7939198 --- /dev/null +++ b/config/agents/codex.yaml @@ -0,0 +1,20 @@ +version: 1 +id: codex +config: + kind: cli + enabled: true + protocol: uaa.agent.session + execution: + scope: host + cli: + binary: codex + mode: persistent + capabilities: + session_start: true + session_resume: true + session_fork: true + session_stop: true + status_snapshot: true + event_stream: true + llm: true + mcp_client: false diff --git a/crates/agent-api-client/src/lib.rs b/crates/agent-api-client/src/lib.rs index e6da1778a..a0777d681 100644 --- a/crates/agent-api-client/src/lib.rs +++ b/crates/agent-api-client/src/lib.rs @@ -8,10 +8,10 @@ use std::sync::Arc; use agent_api_types::{ ApiError, ExecuteCancelRequestV1, ExecuteCancelResponseV1, ExecuteRequest, ExecuteResponse, - GatewayLifecycleRequestV1, GatewayLifecycleResponseV1, PendingDiffClearRequestV1, - PendingDiffClearResponseV1, PendingDiffReconcileRequestV1, PendingDiffReconcileResponseV1, - PendingDiffRecordV1, PendingDiffRequestV1, WorldDoctorReportV1, WorldFsReadRequestV1, - WorldFsReadResponseV1, + GatewayLifecycleRequestV1, GatewayLifecycleResponseV1, MemberTurnSubmitRequestV1, + PendingDiffClearRequestV1, PendingDiffClearResponseV1, PendingDiffReconcileRequestV1, + PendingDiffReconcileResponseV1, PendingDiffRecordV1, PendingDiffRequestV1, WorldDoctorReportV1, + WorldFsReadRequestV1, WorldFsReadResponseV1, }; use anyhow::{anyhow, Context, Result}; use http_body_util::{BodyExt, Full}; @@ -143,6 +143,31 @@ impl AgentClient { self.parse_response(response).await } + /// Submit a follow-up turn to a retained world member session and stream incremental output. + pub async fn submit_member_turn_stream( + &self, + request: MemberTurnSubmitRequestV1, + ) -> Result> { + let response = self + .post("/v1/member_turn/stream", &request) + .await + .context("Failed to initiate member turn submit stream")?; + + if response.status().is_success() { + return Ok(response); + } + + let status = response.status(); + let body_bytes = response + .into_body() + .collect() + .await + .context("Failed to read error body")? + .to_bytes(); + + Err(Self::map_http_error(status, &body_bytes)) + } + /// Get agent capabilities. pub async fn capabilities(&self) -> Result { let response = self diff --git a/crates/agent-api-types/Cargo.toml b/crates/agent-api-types/Cargo.toml index c0ebb16df..31022c6d7 100644 --- a/crates/agent-api-types/Cargo.toml +++ b/crates/agent-api-types/Cargo.toml @@ -8,3 +8,4 @@ serde = { version = "1", features = ["derive"] } serde_json = "1" thiserror = "1" substrate-common = { version = "0.2.7", path = "../common" } +world-api = { version = "0.1.1", path = "../world-api" } diff --git a/crates/agent-api-types/src/lib.rs b/crates/agent-api-types/src/lib.rs index aa597baa4..174b056e7 100644 --- a/crates/agent-api-types/src/lib.rs +++ b/crates/agent-api-types/src/lib.rs @@ -3,12 +3,17 @@ use serde::{Deserialize, Serialize}; use std::collections::HashMap; use std::net::IpAddr; +use std::path::Path; use substrate_common::agent_events::AgentEvent; pub use substrate_common::{ validate_identity_tuple_and_placement_posture, FsDiff, IdentityTuple, PlacementExecution, PlacementPosture, ProcessEvent, ProcessEventType, ProcessEventsStatus, ProcessTelemetry, WorldFsMode, }; +pub use world_api::{ + SharedWorldBindingSnapshot, SharedWorldBindingState, SharedWorldOwnerAction, + SharedWorldOwnerSpec, WorldReuseMode, +}; #[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)] #[serde(rename_all = "snake_case")] @@ -653,7 +658,287 @@ pub struct WorldNetworkRoutingV1 { pub allowed_domains: Vec, } +#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)] +#[serde(try_from = "ResolvedMemberRuntimeDescriptorDef")] +pub struct ResolvedMemberRuntimeDescriptorV1 { + pub backend_kind: MemberRuntimeBackendKindV1, + pub binary_path: String, +} + +#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)] +#[serde(rename_all = "snake_case")] +pub enum MemberRuntimeBackendKindV1 { + Codex, + ClaudeCode, +} + +#[derive(Debug, Clone, Deserialize)] +#[serde(deny_unknown_fields)] +struct ResolvedMemberRuntimeDescriptorDef { + backend_kind: MemberRuntimeBackendKindV1, + binary_path: String, +} + +impl ResolvedMemberRuntimeDescriptorV1 { + pub fn validate(&self) -> Result<(), String> { + validate_non_empty_request_field( + "member_dispatch.resolved_runtime.binary_path", + &self.binary_path, + )?; + if !Path::new(&self.binary_path).is_absolute() { + return Err( + "member_dispatch.resolved_runtime.binary_path must be an absolute path".to_string(), + ); + } + Ok(()) + } +} + +impl TryFrom for ResolvedMemberRuntimeDescriptorV1 { + type Error = String; + + fn try_from(value: ResolvedMemberRuntimeDescriptorDef) -> Result { + let descriptor = Self { + backend_kind: value.backend_kind, + binary_path: value.binary_path, + }; + descriptor.validate()?; + Ok(descriptor) + } +} + +#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)] +#[serde(try_from = "MemberDispatchRequestDef")] +pub struct MemberDispatchRequestV1 { + #[serde(default = "member_dispatch_request_v1_default_schema_version")] + pub schema_version: u32, + pub orchestration_session_id: String, + pub participant_id: String, + pub orchestrator_participant_id: String, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub parent_participant_id: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub resumed_from_participant_id: Option, + pub backend_id: String, + pub protocol: String, + pub run_id: String, + pub world_id: String, + pub world_generation: u64, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub initial_prompt: Option, + pub resolved_runtime: ResolvedMemberRuntimeDescriptorV1, +} + +#[derive(Debug, Clone, Deserialize)] +#[serde(deny_unknown_fields)] +struct MemberDispatchRequestDef { + #[serde(default = "member_dispatch_request_v1_default_schema_version")] + schema_version: u32, + orchestration_session_id: String, + participant_id: String, + orchestrator_participant_id: String, + #[serde(default)] + parent_participant_id: Option, + #[serde(default)] + resumed_from_participant_id: Option, + backend_id: String, + protocol: String, + run_id: String, + world_id: String, + world_generation: u64, + #[serde(default)] + initial_prompt: Option, + resolved_runtime: ResolvedMemberRuntimeDescriptorV1, +} + +fn member_dispatch_request_v1_default_schema_version() -> u32 { + 1 +} + +impl MemberDispatchRequestV1 { + pub fn validate(&self) -> Result<(), String> { + if self.schema_version != 1 { + return Err(format!( + "unsupported member_dispatch.schema_version: {} (expected 1)", + self.schema_version + )); + } + + validate_non_empty_request_field( + "member_dispatch.orchestration_session_id", + &self.orchestration_session_id, + )?; + validate_non_empty_request_field("member_dispatch.participant_id", &self.participant_id)?; + validate_non_empty_request_field( + "member_dispatch.orchestrator_participant_id", + &self.orchestrator_participant_id, + )?; + validate_optional_non_empty_request_field( + "member_dispatch.parent_participant_id", + self.parent_participant_id.as_deref(), + )?; + validate_optional_non_empty_request_field( + "member_dispatch.resumed_from_participant_id", + self.resumed_from_participant_id.as_deref(), + )?; + validate_non_empty_request_field("member_dispatch.backend_id", &self.backend_id)?; + validate_non_empty_request_field("member_dispatch.protocol", &self.protocol)?; + validate_non_empty_request_field("member_dispatch.run_id", &self.run_id)?; + validate_non_empty_request_field("member_dispatch.world_id", &self.world_id)?; + validate_optional_non_empty_request_field( + "member_dispatch.initial_prompt", + self.initial_prompt.as_deref(), + )?; + self.resolved_runtime.validate()?; + + if self.orchestrator_participant_id == self.participant_id { + return Err( + "member_dispatch.orchestrator_participant_id must not equal participant_id" + .to_string(), + ); + } + + if self.parent_participant_id.as_deref() == Some(self.participant_id.as_str()) { + return Err( + "member_dispatch.parent_participant_id must not point to participant_id" + .to_string(), + ); + } + + if self.resumed_from_participant_id.as_deref() == Some(self.participant_id.as_str()) { + return Err( + "member_dispatch.resumed_from_participant_id must not point to participant_id" + .to_string(), + ); + } + + Ok(()) + } +} + +impl TryFrom for MemberDispatchRequestV1 { + type Error = String; + + fn try_from(value: MemberDispatchRequestDef) -> Result { + let request = Self { + schema_version: value.schema_version, + orchestration_session_id: value.orchestration_session_id, + participant_id: value.participant_id, + orchestrator_participant_id: value.orchestrator_participant_id, + parent_participant_id: value.parent_participant_id, + resumed_from_participant_id: value.resumed_from_participant_id, + backend_id: value.backend_id, + protocol: value.protocol, + run_id: value.run_id, + world_id: value.world_id, + world_generation: value.world_generation, + initial_prompt: value.initial_prompt, + resolved_runtime: value.resolved_runtime, + }; + request.validate()?; + Ok(request) + } +} + +#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)] +#[serde(try_from = "MemberTurnSubmitRequestDef")] +pub struct MemberTurnSubmitRequestV1 { + #[serde(default = "member_turn_submit_request_v1_default_schema_version")] + pub schema_version: u32, + pub orchestration_session_id: String, + pub participant_id: String, + pub orchestrator_participant_id: String, + pub backend_id: String, + pub run_id: String, + pub world_id: String, + pub world_generation: u64, + pub prompt: String, +} + +#[derive(Debug, Clone, Deserialize)] +#[serde(deny_unknown_fields)] +struct MemberTurnSubmitRequestDef { + #[serde(default = "member_turn_submit_request_v1_default_schema_version")] + schema_version: u32, + orchestration_session_id: String, + participant_id: String, + orchestrator_participant_id: String, + backend_id: String, + run_id: String, + world_id: String, + world_generation: u64, + prompt: String, +} + +fn member_turn_submit_request_v1_default_schema_version() -> u32 { + 1 +} + +impl MemberTurnSubmitRequestV1 { + pub fn validate(&self) -> Result<(), String> { + if self.schema_version != 1 { + return Err(format!( + "unsupported member_turn_submit.schema_version: {} (expected 1)", + self.schema_version + )); + } + + validate_non_empty_request_field( + "member_turn_submit.orchestration_session_id", + &self.orchestration_session_id, + )?; + validate_non_empty_request_field( + "member_turn_submit.participant_id", + &self.participant_id, + )?; + validate_non_empty_request_field( + "member_turn_submit.orchestrator_participant_id", + &self.orchestrator_participant_id, + )?; + validate_non_empty_request_field("member_turn_submit.backend_id", &self.backend_id)?; + validate_gateway_backend_id_selector(self.backend_id.trim()).map_err(|_| { + format!( + "invalid member_turn_submit.backend_id '{}'; expected :", + self.backend_id.trim() + ) + })?; + validate_non_empty_request_field("member_turn_submit.run_id", &self.run_id)?; + validate_non_empty_request_field("member_turn_submit.world_id", &self.world_id)?; + validate_non_empty_request_field("member_turn_submit.prompt", &self.prompt)?; + + if self.orchestrator_participant_id == self.participant_id { + return Err( + "member_turn_submit.orchestrator_participant_id must not equal participant_id" + .to_string(), + ); + } + + Ok(()) + } +} + +impl TryFrom for MemberTurnSubmitRequestV1 { + type Error = String; + + fn try_from(value: MemberTurnSubmitRequestDef) -> Result { + let request = Self { + schema_version: value.schema_version, + orchestration_session_id: value.orchestration_session_id, + participant_id: value.participant_id, + orchestrator_participant_id: value.orchestrator_participant_id, + backend_id: value.backend_id, + run_id: value.run_id, + world_id: value.world_id, + world_generation: value.world_generation, + prompt: value.prompt, + }; + request.validate()?; + Ok(request) + } +} + #[derive(Debug, Clone, Serialize, Deserialize)] +#[serde(try_from = "ExecuteRequestDef")] pub struct ExecuteRequest { pub profile: Option, pub cmd: String, @@ -664,9 +949,88 @@ pub struct ExecuteRequest { pub budget: Option, pub policy_snapshot: PolicySnapshotV3, #[serde(default, skip_serializing_if = "Option::is_none")] + pub shared_world: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] pub world_network: Option, #[serde(default, skip_serializing_if = "Option::is_none")] pub world_fs_mode: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub member_dispatch: Option, +} + +#[derive(Debug, Clone, Deserialize)] +#[serde(deny_unknown_fields)] +struct ExecuteRequestDef { + profile: Option, + cmd: String, + cwd: Option, + env: Option>, + pty: bool, + agent_id: String, + budget: Option, + policy_snapshot: PolicySnapshotV3, + #[serde(default)] + shared_world: Option, + #[serde(default)] + world_network: Option, + #[serde(default)] + world_fs_mode: Option, + #[serde(default)] + member_dispatch: Option, +} + +impl ExecuteRequest { + pub fn validate(&self) -> Result<(), String> { + let cmd_is_empty = self.cmd.trim().is_empty(); + + match self.member_dispatch.as_ref() { + Some(member_dispatch) => { + member_dispatch.validate()?; + if !cmd_is_empty { + return Err( + "execute request member_dispatch requires cmd.trim().is_empty()" + .to_string(), + ); + } + if self.pty { + return Err("execute request member_dispatch requires pty=false".to_string()); + } + } + None => { + if cmd_is_empty { + return Err( + "execute request process exec requires a non-empty cmd when member_dispatch is absent" + .to_string(), + ); + } + } + } + + Ok(()) + } +} + +impl TryFrom for ExecuteRequest { + type Error = String; + + fn try_from(value: ExecuteRequestDef) -> Result { + let request = Self { + profile: value.profile, + cmd: value.cmd, + cwd: value.cwd, + env: value.env, + pty: value.pty, + agent_id: value.agent_id, + budget: value.budget, + policy_snapshot: value.policy_snapshot, + shared_world: value.shared_world, + world_network: value.world_network, + world_fs_mode: value.world_fs_mode, + member_dispatch: value.member_dispatch, + }; + request.validate()?; + Ok(request) + } } #[derive(Debug, Clone, Serialize, Deserialize)] @@ -678,6 +1042,8 @@ pub struct ExecuteResponse { pub scopes_used: Vec, #[serde(skip_serializing_if = "Option::is_none")] pub fs_diff: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub shared_world: Option, #[serde(flatten, default)] pub process_telemetry: ProcessTelemetry, } @@ -732,6 +1098,23 @@ fn pending_diff_record_v1_default_schema_version() -> u32 { 1 } +fn validate_non_empty_request_field(field: &str, value: &str) -> Result<(), String> { + if value.trim().is_empty() { + return Err(format!("{field} must not be empty")); + } + Ok(()) +} + +fn validate_optional_non_empty_request_field( + field: &str, + value: Option<&str>, +) -> Result<(), String> { + let Some(value) = value else { + return Ok(()); + }; + validate_non_empty_request_field(field, value) +} + #[derive(Debug, Clone, Serialize, Deserialize)] pub struct PendingDiffClearRequestV1 { pub profile: Option, @@ -1961,11 +2344,16 @@ mod tests { agent_id: "tester".into(), budget: None, policy_snapshot: snapshot, + shared_world: Some(SharedWorldOwnerSpec { + orchestration_session_id: "orch_123".into(), + action: SharedWorldOwnerAction::AttachOrCreate, + }), world_network: Some(WorldNetworkRoutingV1 { isolate_network: true, allowed_domains: vec!["github.com".to_string()], }), world_fs_mode: Some(WorldFsMode::ReadOnly), + member_dispatch: None, }; let json = serde_json::to_string(&req).expect("serialize request"); @@ -1979,6 +2367,13 @@ mod tests { ); let back: ExecuteRequest = serde_json::from_str(&json).expect("deserialize request"); assert_eq!(back.world_fs_mode, Some(WorldFsMode::ReadOnly)); + assert_eq!( + back.shared_world, + Some(SharedWorldOwnerSpec { + orchestration_session_id: "orch_123".into(), + action: SharedWorldOwnerAction::AttachOrCreate, + }) + ); assert_eq!(back.policy_snapshot.schema_version, 3); assert_eq!( back.policy_snapshot.net_allowed, @@ -2028,8 +2423,10 @@ mod tests { agent_id: "tester".into(), budget: None, policy_snapshot: snapshot, + shared_world: None, world_network: None, world_fs_mode: None, + member_dispatch: None, }; let json = serde_json::to_string(&req).expect("serialize request"); @@ -2055,6 +2452,42 @@ mod tests { ); } + #[test] + fn execute_response_shared_world_round_trip() { + let response = ExecuteResponse { + exit: 0, + span_id: "spn_123".into(), + stdout_b64: "aGVsbG8=".into(), + stderr_b64: String::new(), + scopes_used: vec!["github.com".into()], + fs_diff: None, + shared_world: Some(SharedWorldBindingSnapshot { + orchestration_session_id: "orch_123".into(), + world_id: "wld_123".into(), + world_generation: 3, + binding_state: SharedWorldBindingState::Active, + }), + process_telemetry: ProcessTelemetry::default(), + }; + + let json = serde_json::to_string(&response).expect("serialize response"); + assert!( + json.contains("\"shared_world\""), + "expected shared_world to serialize" + ); + let back: ExecuteResponse = + serde_json::from_str(&json).expect("deserialize execute response"); + assert_eq!( + back.shared_world, + Some(SharedWorldBindingSnapshot { + orchestration_session_id: "orch_123".into(), + world_id: "wld_123".into(), + world_generation: 3, + binding_state: SharedWorldBindingState::Active, + }) + ); + } + #[test] fn execute_cancel_request_round_trip() { let req = ExecuteCancelRequestV1 { @@ -2068,6 +2501,377 @@ mod tests { assert_eq!(back, req); } + fn test_absolute_binary_path() -> String { + std::env::current_exe() + .expect("current_exe") + .to_string_lossy() + .into_owned() + } + + #[test] + fn execute_request_member_dispatch_round_trip() { + let binary_path = test_absolute_binary_path(); + let snapshot = PolicySnapshotV3 { + schema_version: 3, + net_allowed: Vec::new(), + world_fs: PolicySnapshotWorldFsV3 { + host_visible: true, + fail_closed: PolicySnapshotWorldFsFailClosedV3 { routing: false }, + deny_enforcement: None, + caged_required: false, + discover: None, + read: None, + write: PolicySnapshotWorldFsWriteV3 { + enabled: true, + allow_list: vec![".".to_string()], + deny_list: Vec::new(), + }, + }, + }; + + let req = ExecuteRequest { + profile: None, + cmd: String::new(), + cwd: Some("/tmp".into()), + env: None, + pty: false, + agent_id: "tester".into(), + budget: None, + policy_snapshot: snapshot, + shared_world: None, + world_network: None, + world_fs_mode: None, + member_dispatch: Some(MemberDispatchRequestV1 { + schema_version: 1, + orchestration_session_id: "orch_123".into(), + participant_id: "ash_member_123".into(), + orchestrator_participant_id: "ash_orch_123".into(), + parent_participant_id: Some("ash_parent_123".into()), + resumed_from_participant_id: Some("ash_old_123".into()), + backend_id: "cli:codex".into(), + protocol: "uaa.agent.session".into(), + run_id: "run_123".into(), + world_id: "world_123".into(), + world_generation: 7, + initial_prompt: Some("first turn".into()), + resolved_runtime: ResolvedMemberRuntimeDescriptorV1 { + backend_kind: MemberRuntimeBackendKindV1::Codex, + binary_path: binary_path.clone(), + }, + }), + }; + + let json = serde_json::to_string(&req).expect("serialize request"); + assert!(json.contains("\"member_dispatch\"")); + + let back: ExecuteRequest = serde_json::from_str(&json).expect("deserialize request"); + assert!(back.cmd.is_empty()); + assert_eq!( + back.member_dispatch, + Some(MemberDispatchRequestV1 { + schema_version: 1, + orchestration_session_id: "orch_123".into(), + participant_id: "ash_member_123".into(), + orchestrator_participant_id: "ash_orch_123".into(), + parent_participant_id: Some("ash_parent_123".into()), + resumed_from_participant_id: Some("ash_old_123".into()), + backend_id: "cli:codex".into(), + protocol: "uaa.agent.session".into(), + run_id: "run_123".into(), + world_id: "world_123".into(), + world_generation: 7, + initial_prompt: Some("first turn".into()), + resolved_runtime: ResolvedMemberRuntimeDescriptorV1 { + backend_kind: MemberRuntimeBackendKindV1::Codex, + binary_path, + }, + }) + ); + } + + #[test] + fn member_turn_submit_request_round_trip() { + let req = MemberTurnSubmitRequestV1 { + schema_version: 1, + orchestration_session_id: "orch_123".into(), + participant_id: "ash_member_123".into(), + orchestrator_participant_id: "ash_orch_123".into(), + backend_id: "cli:codex".into(), + run_id: "run_123".into(), + world_id: "world_123".into(), + world_generation: 7, + prompt: "summarize the failure".into(), + }; + + let json = serde_json::to_string(&req).expect("serialize member turn submit request"); + let back: MemberTurnSubmitRequestV1 = + serde_json::from_str(&json).expect("deserialize member turn submit request"); + assert_eq!(back, req); + } + + #[test] + fn member_turn_submit_rejects_invalid_backend_id() { + let err = serde_json::from_value::(serde_json::json!({ + "schema_version": 1, + "orchestration_session_id": "orch_123", + "participant_id": "ash_member_123", + "orchestrator_participant_id": "ash_orch_123", + "backend_id": "codex", + "run_id": "run_123", + "world_id": "world_123", + "world_generation": 7, + "prompt": "resume" + })) + .expect_err("invalid backend id should fail"); + + assert!( + err.to_string() + .contains("invalid member_turn_submit.backend_id"), + "unexpected error: {err}" + ); + } + + #[test] + fn member_turn_submit_rejects_empty_prompt() { + let err = serde_json::from_value::(serde_json::json!({ + "schema_version": 1, + "orchestration_session_id": "orch_123", + "participant_id": "ash_member_123", + "orchestrator_participant_id": "ash_orch_123", + "backend_id": "cli:codex", + "run_id": "run_123", + "world_id": "world_123", + "world_generation": 7, + "prompt": " " + })) + .expect_err("blank prompt should fail"); + + assert!( + err.to_string() + .contains("member_turn_submit.prompt must not be empty"), + "unexpected error: {err}" + ); + } + + #[test] + fn execute_request_rejects_empty_process_exec_cmd_at_boundary() { + let err = serde_json::from_value::(serde_json::json!({ + "profile": null, + "cmd": " ", + "cwd": "/tmp", + "pty": false, + "agent_id": "tester", + "policy_snapshot": { + "schema_version": 3, + "net_allowed": [], + "world_fs": { + "host_visible": true, + "fail_closed": { "routing": false }, + "caged_required": false, + "write": { + "enabled": true, + "allow_list": ["."], + "deny_list": [] + } + } + } + })) + .expect_err("empty process cmd should fail"); + + assert!( + err.to_string().contains("non-empty cmd"), + "unexpected error: {err}" + ); + } + + #[test] + fn execute_request_rejects_member_dispatch_with_non_empty_cmd_at_boundary() { + let binary_path = test_absolute_binary_path(); + let err = serde_json::from_value::(serde_json::json!({ + "cmd": "echo hi", + "cwd": "/tmp", + "pty": false, + "agent_id": "tester", + "policy_snapshot": { + "schema_version": 3, + "net_allowed": [], + "world_fs": { + "host_visible": true, + "fail_closed": { "routing": false }, + "caged_required": false, + "write": { + "enabled": true, + "allow_list": ["."], + "deny_list": [] + } + } + }, + "member_dispatch": { + "schema_version": 1, + "orchestration_session_id": "orch_123", + "participant_id": "ash_member_123", + "orchestrator_participant_id": "ash_orch_123", + "backend_id": "cli:codex", + "protocol": "uaa.agent.session", + "run_id": "run_123", + "world_id": "world_123", + "world_generation": 7, + "resolved_runtime": { + "backend_kind": "codex", + "binary_path": binary_path + } + } + })) + .expect_err("mixed cmd + member dispatch should fail"); + + assert!( + err.to_string() + .contains("member_dispatch requires cmd.trim().is_empty()"), + "unexpected error: {err}" + ); + } + + #[test] + fn execute_request_rejects_member_dispatch_with_pty_at_boundary() { + let binary_path = test_absolute_binary_path(); + let err = serde_json::from_value::(serde_json::json!({ + "cmd": "", + "cwd": "/tmp", + "pty": true, + "agent_id": "tester", + "policy_snapshot": { + "schema_version": 3, + "net_allowed": [], + "world_fs": { + "host_visible": true, + "fail_closed": { "routing": false }, + "caged_required": false, + "write": { + "enabled": true, + "allow_list": ["."], + "deny_list": [] + } + } + }, + "member_dispatch": { + "schema_version": 1, + "orchestration_session_id": "orch_123", + "participant_id": "ash_member_123", + "orchestrator_participant_id": "ash_orch_123", + "backend_id": "cli:codex", + "protocol": "uaa.agent.session", + "run_id": "run_123", + "world_id": "world_123", + "world_generation": 7, + "resolved_runtime": { + "backend_kind": "codex", + "binary_path": binary_path + } + } + })) + .expect_err("pty member dispatch should fail"); + + assert!( + err.to_string() + .contains("member_dispatch requires pty=false"), + "unexpected error: {err}" + ); + } + + #[test] + fn member_dispatch_rejects_self_referential_lineage() { + let binary_path = test_absolute_binary_path(); + let err = serde_json::from_value::(serde_json::json!({ + "schema_version": 1, + "orchestration_session_id": "orch_123", + "participant_id": "ash_member_123", + "orchestrator_participant_id": "ash_member_123", + "backend_id": "cli:codex", + "protocol": "uaa.agent.session", + "run_id": "run_123", + "world_id": "world_123", + "world_generation": 7, + "resolved_runtime": { + "backend_kind": "codex", + "binary_path": binary_path + } + })) + .expect_err("self-referential lineage should fail"); + + assert!( + err.to_string() + .contains("orchestrator_participant_id must not equal participant_id"), + "unexpected error: {err}" + ); + } + + #[test] + fn execute_request_rejects_member_dispatch_without_resolved_runtime_at_boundary() { + let err = serde_json::from_value::(serde_json::json!({ + "cmd": "", + "cwd": "/tmp", + "pty": false, + "agent_id": "tester", + "policy_snapshot": { + "schema_version": 3, + "net_allowed": [], + "world_fs": { + "host_visible": true, + "fail_closed": { "routing": false }, + "caged_required": false, + "write": { + "enabled": true, + "allow_list": ["."], + "deny_list": [] + } + } + }, + "member_dispatch": { + "schema_version": 1, + "orchestration_session_id": "orch_123", + "participant_id": "ash_member_123", + "orchestrator_participant_id": "ash_orch_123", + "backend_id": "cli:codex", + "protocol": "uaa.agent.session", + "run_id": "run_123", + "world_id": "world_123", + "world_generation": 7 + } + })) + .expect_err("member dispatch without resolved_runtime should fail"); + + assert!( + err.to_string().contains("missing field `resolved_runtime`"), + "unexpected error: {err}" + ); + } + + #[test] + fn member_dispatch_rejects_non_absolute_resolved_runtime_binary_path() { + let err = serde_json::from_value::(serde_json::json!({ + "schema_version": 1, + "orchestration_session_id": "orch_123", + "participant_id": "ash_member_123", + "orchestrator_participant_id": "ash_orch_123", + "backend_id": "cli:codex", + "protocol": "uaa.agent.session", + "run_id": "run_123", + "world_id": "world_123", + "world_generation": 7, + "resolved_runtime": { + "backend_kind": "codex", + "binary_path": "codex" + } + })) + .expect_err("relative binary path should fail"); + + assert!( + err.to_string() + .contains("member_dispatch.resolved_runtime.binary_path must be an absolute path"), + "unexpected error: {err}" + ); + } + #[test] fn execute_cancel_response_defaults_schema_version() { let response: ExecuteCancelResponseV1 = serde_json::from_value(serde_json::json!({ diff --git a/crates/broker/src/policy.rs b/crates/broker/src/policy.rs index 728131f08..90f4de443 100644 --- a/crates/broker/src/policy.rs +++ b/crates/broker/src/policy.rs @@ -642,7 +642,7 @@ impl Serialize for SortedMap<'_> { S: serde::Serializer, { let mut entries: Vec<_> = self.0.iter().collect(); - entries.sort_by(|(a_key, _), (b_key, _)| a_key.cmp(b_key)); + entries.sort_by_key(|(key, _)| *key); let mut map = serializer.serialize_map(Some(entries.len()))?; for (key, value) in entries { diff --git a/crates/common/src/agent_events.rs b/crates/common/src/agent_events.rs index 095412bd1..094135b70 100644 --- a/crates/common/src/agent_events.rs +++ b/crates/common/src/agent_events.rs @@ -82,6 +82,12 @@ pub struct AgentEvent { pub run_id: String, #[serde(default, skip_serializing_if = "Option::is_none")] pub parent_run_id: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub participant_id: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub parent_participant_id: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub resumed_from_participant_id: Option, // Attribution + correlation (optional) #[serde(default, skip_serializing_if = "Option::is_none")] @@ -129,6 +135,12 @@ struct AgentEventDef { #[serde(default)] parent_run_id: Option, #[serde(default)] + participant_id: Option, + #[serde(default)] + parent_participant_id: Option, + #[serde(default)] + resumed_from_participant_id: Option, + #[serde(default)] backend_id: Option, #[serde(default)] thread_id: Option, @@ -199,6 +211,9 @@ impl AgentEvent { orchestration_session_id: orchestration_session_id.into(), run_id: run_id.into(), parent_run_id: None, + participant_id: None, + parent_participant_id: None, + resumed_from_participant_id: None, data, backend_id: None, thread_id: None, @@ -365,6 +380,9 @@ impl TryFrom for AgentEvent { orchestration_session_id: value.orchestration_session_id, run_id: value.run_id, parent_run_id: value.parent_run_id, + participant_id: value.participant_id, + parent_participant_id: value.parent_participant_id, + resumed_from_participant_id: value.resumed_from_participant_id, backend_id: value.backend_id, thread_id: value.thread_id, role: value.role, diff --git a/crates/common/src/gateway_auth_bundle.rs b/crates/common/src/gateway_auth_bundle.rs new file mode 100644 index 000000000..e0f52b5f8 --- /dev/null +++ b/crates/common/src/gateway_auth_bundle.rs @@ -0,0 +1,235 @@ +//! Shared read-once auth-bundle contract for integrated gateway startup. +//! +//! ``` +//! use std::collections::HashMap; +//! +//! use substrate_common::{ +//! GatewayAuthBundleV1, SUBSTRATE_LLM_AUTH_BUNDLE_FD, +//! SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCESS_TOKEN, +//! }; +//! +//! let mut fields = HashMap::new(); +//! fields.insert( +//! SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCESS_TOKEN.to_string(), +//! "header.payload.signature".to_string(), +//! ); +//! +//! let bundle = GatewayAuthBundleV1 { +//! schema_version: 1, +//! backend_id: "cli:codex".to_string(), +//! fields, +//! }; +//! +//! bundle.validate().unwrap(); +//! assert_eq!(SUBSTRATE_LLM_AUTH_BUNDLE_FD, "SUBSTRATE_LLM_AUTH_BUNDLE_FD"); +//! ``` + +use serde::{Deserialize, Serialize}; +use std::collections::HashMap; + +pub const SUBSTRATE_LLM_AUTH_BUNDLE_FD: &str = "SUBSTRATE_LLM_AUTH_BUNDLE_FD"; +pub const SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCOUNT_ID: &str = + "SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCOUNT_ID"; +pub const SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCESS_TOKEN: &str = + "SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCESS_TOKEN"; +pub const SUBSTRATE_LLM_BACKEND_AUTH_API_OPENAI_API_KEY: &str = + "SUBSTRATE_LLM_BACKEND_AUTH_API_OPENAI_API_KEY"; + +pub const GATEWAY_AUTH_BUNDLE_SCHEMA_VERSION: u32 = 1; +pub const GATEWAY_AUTH_BUNDLE_BACKEND_CLI_CODEX: &str = "cli:codex"; +pub const GATEWAY_AUTH_BUNDLE_BACKEND_API_OPENAI: &str = "api:openai"; + +pub const CLI_CODEX_GATEWAY_AUTH_ALLOWED_FIELDS: &[&str] = &[ + SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCOUNT_ID, + SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCESS_TOKEN, +]; +pub const CLI_CODEX_GATEWAY_AUTH_REQUIRED_FIELDS: &[&str] = + &[SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCESS_TOKEN]; +pub const API_OPENAI_GATEWAY_AUTH_ALLOWED_FIELDS: &[&str] = + &[SUBSTRATE_LLM_BACKEND_AUTH_API_OPENAI_API_KEY]; +pub const API_OPENAI_GATEWAY_AUTH_REQUIRED_FIELDS: &[&str] = + &[SUBSTRATE_LLM_BACKEND_AUTH_API_OPENAI_API_KEY]; + +#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)] +#[serde(deny_unknown_fields)] +pub struct GatewayAuthBundleV1 { + #[serde(default = "gateway_auth_bundle_schema_version")] + pub schema_version: u32, + pub backend_id: String, + pub fields: HashMap, +} + +impl GatewayAuthBundleV1 { + pub fn validate(&self) -> Result<(), String> { + validate_gateway_auth_bundle(self) + } + + pub fn required_field_names(&self) -> Result<&'static [&'static str], String> { + required_gateway_auth_fields(self.backend_id.as_str()) + } + + pub fn allowed_field_names(&self) -> Result<&'static [&'static str], String> { + allowed_gateway_auth_fields(self.backend_id.as_str()) + } +} + +pub fn gateway_auth_bundle_schema_version() -> u32 { + GATEWAY_AUTH_BUNDLE_SCHEMA_VERSION +} + +pub fn validate_gateway_auth_bundle(bundle: &GatewayAuthBundleV1) -> Result<(), String> { + if bundle.schema_version != GATEWAY_AUTH_BUNDLE_SCHEMA_VERSION { + return Err(format!( + "unsupported gateway auth bundle schema_version: {} (expected {})", + bundle.schema_version, GATEWAY_AUTH_BUNDLE_SCHEMA_VERSION + )); + } + + let backend_id = bundle.backend_id.trim(); + if backend_id != bundle.backend_id { + return Err( + "gateway auth bundle backend_id must not contain surrounding whitespace".into(), + ); + } + if backend_id.is_empty() { + return Err("gateway auth bundle backend_id is required".into()); + } + + let allowed_fields = allowed_gateway_auth_fields(backend_id)?; + let required_fields = required_gateway_auth_fields(backend_id)?; + + if bundle.fields.is_empty() { + return Err(format!( + "gateway auth bundle for '{}' must include at least one field", + backend_id + )); + } + + for (name, value) in &bundle.fields { + if !allowed_fields.contains(&name.as_str()) { + return Err(format!( + "gateway auth bundle for '{}' contains unsupported field '{}'", + backend_id, name + )); + } + if name.trim() != name || name.contains(char::is_whitespace) || name.contains('=') { + return Err(format!( + "gateway auth bundle for '{}' contains invalid field name '{}'", + backend_id, name + )); + } + if value.trim().is_empty() { + return Err(format!( + "gateway auth bundle for '{}' contains blank value for '{}'", + backend_id, name + )); + } + } + + for required_field in required_fields { + if !bundle.fields.contains_key(*required_field) { + return Err(format!( + "gateway auth bundle for '{}' is missing required field '{}'", + backend_id, required_field + )); + } + } + + Ok(()) +} + +pub fn required_gateway_auth_fields(backend_id: &str) -> Result<&'static [&'static str], String> { + match backend_id { + GATEWAY_AUTH_BUNDLE_BACKEND_CLI_CODEX => Ok(CLI_CODEX_GATEWAY_AUTH_REQUIRED_FIELDS), + GATEWAY_AUTH_BUNDLE_BACKEND_API_OPENAI => Ok(API_OPENAI_GATEWAY_AUTH_REQUIRED_FIELDS), + other => Err(format!( + "unsupported gateway auth bundle backend_id '{}'", + other + )), + } +} + +pub fn allowed_gateway_auth_fields(backend_id: &str) -> Result<&'static [&'static str], String> { + match backend_id { + GATEWAY_AUTH_BUNDLE_BACKEND_CLI_CODEX => Ok(CLI_CODEX_GATEWAY_AUTH_ALLOWED_FIELDS), + GATEWAY_AUTH_BUNDLE_BACKEND_API_OPENAI => Ok(API_OPENAI_GATEWAY_AUTH_ALLOWED_FIELDS), + other => Err(format!( + "unsupported gateway auth bundle backend_id '{}'", + other + )), + } +} + +#[cfg(test)] +mod tests { + use super::*; + + fn cli_codex_bundle() -> GatewayAuthBundleV1 { + GatewayAuthBundleV1 { + schema_version: GATEWAY_AUTH_BUNDLE_SCHEMA_VERSION, + backend_id: GATEWAY_AUTH_BUNDLE_BACKEND_CLI_CODEX.to_string(), + fields: HashMap::from([( + SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCESS_TOKEN.to_string(), + "header.payload.signature".to_string(), + )]), + } + } + + #[test] + fn validate_accepts_cli_codex_bundle_with_required_field() { + cli_codex_bundle().validate().unwrap(); + } + + #[test] + fn validate_accepts_api_openai_bundle_with_canonical_key() { + let bundle = GatewayAuthBundleV1 { + schema_version: GATEWAY_AUTH_BUNDLE_SCHEMA_VERSION, + backend_id: GATEWAY_AUTH_BUNDLE_BACKEND_API_OPENAI.to_string(), + fields: HashMap::from([( + SUBSTRATE_LLM_BACKEND_AUTH_API_OPENAI_API_KEY.to_string(), + "sk-test".to_string(), + )]), + }; + + bundle.validate().unwrap(); + } + + #[test] + fn validate_rejects_unknown_backend() { + let mut bundle = cli_codex_bundle(); + bundle.backend_id = "api:anthropic".to_string(); + + let err = bundle.validate().unwrap_err(); + assert!(err.contains("unsupported gateway auth bundle backend_id")); + } + + #[test] + fn validate_rejects_missing_required_field() { + let mut bundle = cli_codex_bundle(); + bundle.fields.clear(); + + let err = bundle.validate().unwrap_err(); + assert!(err.contains("must include at least one field")); + } + + #[test] + fn validate_rejects_unknown_field_for_backend() { + let mut bundle = cli_codex_bundle(); + bundle.fields.insert( + SUBSTRATE_LLM_BACKEND_AUTH_API_OPENAI_API_KEY.to_string(), + "sk-test".to_string(), + ); + + let err = bundle.validate().unwrap_err(); + assert!(err.contains("unsupported field")); + } + + #[test] + fn validate_rejects_wrong_schema_version() { + let mut bundle = cli_codex_bundle(); + bundle.schema_version = 2; + + let err = bundle.validate().unwrap_err(); + assert!(err.contains("unsupported gateway auth bundle schema_version")); + } +} diff --git a/crates/common/src/lib.rs b/crates/common/src/lib.rs index 02515de34..b0a98805c 100644 --- a/crates/common/src/lib.rs +++ b/crates/common/src/lib.rs @@ -6,15 +6,27 @@ use std::collections::{HashMap, HashSet}; pub mod agent_events; pub mod agent_identity; pub mod fs_diff; +pub mod gateway_auth_bundle; pub mod identity; pub mod manager_manifest; pub mod paths; +pub mod seccomp; pub mod settings; pub mod world_exec_guard; pub use agent_events::{AgentEvent, AgentEventKind}; pub use agent_identity::derive_agent_backend_id; pub use fs_diff::FsDiff; +pub use gateway_auth_bundle::{ + allowed_gateway_auth_fields, gateway_auth_bundle_schema_version, required_gateway_auth_fields, + validate_gateway_auth_bundle, GatewayAuthBundleV1, API_OPENAI_GATEWAY_AUTH_ALLOWED_FIELDS, + API_OPENAI_GATEWAY_AUTH_REQUIRED_FIELDS, CLI_CODEX_GATEWAY_AUTH_ALLOWED_FIELDS, + CLI_CODEX_GATEWAY_AUTH_REQUIRED_FIELDS, GATEWAY_AUTH_BUNDLE_BACKEND_API_OPENAI, + GATEWAY_AUTH_BUNDLE_BACKEND_CLI_CODEX, GATEWAY_AUTH_BUNDLE_SCHEMA_VERSION, + SUBSTRATE_LLM_AUTH_BUNDLE_FD, SUBSTRATE_LLM_BACKEND_AUTH_API_OPENAI_API_KEY, + SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCESS_TOKEN, + SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCOUNT_ID, +}; pub use identity::{ validate_identity_tuple_and_placement_posture, IdentityTuple, PlacementExecution, PlacementPosture, @@ -179,6 +191,17 @@ impl Default for ProcessTelemetry { pub mod prelude { pub use crate::agent_events::{AgentEvent, AgentEventKind}; pub use crate::fs_diff::FsDiff; + pub use crate::gateway_auth_bundle::{ + allowed_gateway_auth_fields, gateway_auth_bundle_schema_version, + required_gateway_auth_fields, validate_gateway_auth_bundle, GatewayAuthBundleV1, + API_OPENAI_GATEWAY_AUTH_ALLOWED_FIELDS, API_OPENAI_GATEWAY_AUTH_REQUIRED_FIELDS, + CLI_CODEX_GATEWAY_AUTH_ALLOWED_FIELDS, CLI_CODEX_GATEWAY_AUTH_REQUIRED_FIELDS, + GATEWAY_AUTH_BUNDLE_BACKEND_API_OPENAI, GATEWAY_AUTH_BUNDLE_BACKEND_CLI_CODEX, + GATEWAY_AUTH_BUNDLE_SCHEMA_VERSION, SUBSTRATE_LLM_AUTH_BUNDLE_FD, + SUBSTRATE_LLM_BACKEND_AUTH_API_OPENAI_API_KEY, + SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCESS_TOKEN, + SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCOUNT_ID, + }; pub use crate::log_schema; pub use crate::manager_manifest::{ DetectSpec, GuestSpec, InitSpec, InstallClass, InstallSpec, ManagerManifest, ManagerSpec, diff --git a/crates/common/src/seccomp.rs b/crates/common/src/seccomp.rs new file mode 100644 index 000000000..c9411d990 --- /dev/null +++ b/crates/common/src/seccomp.rs @@ -0,0 +1,236 @@ +use anyhow::Result; + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum SeccompAction { + Log, + Errno(i32), +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct SeccompRule<'a> { + pub syscall: &'a str, + pub action: SeccompAction, +} + +impl<'a> SeccompRule<'a> { + pub const fn log(syscall: &'a str) -> Self { + Self { + syscall, + action: SeccompAction::Log, + } + } + + pub const fn errno(syscall: &'a str, errno: i32) -> Self { + Self { + syscall, + action: SeccompAction::Errno(errno), + } + } +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum SeccompInstallStatus { + Installed, + Unavailable(&'static str), +} + +#[cfg(all(target_os = "linux", not(target_env = "musl")))] +mod imp { + use super::{SeccompAction, SeccompInstallStatus, SeccompRule}; + use anyhow::{anyhow, Context, Result}; + use std::ffi::{c_char, c_int, c_uint, c_void, CStr, CString}; + use std::mem; + + const RTLD_NOW: c_int = 0x0002; + const SCMP_ACT_ERRNO_MASK: u32 = 0x0005_0000; + const SCMP_ACT_LOG: u32 = 0x7ffc_0000; + const SCMP_ACT_ALLOW: u32 = 0x7fff_0000; + const __NR_SCMP_ERROR: c_int = -1; + const LIBSECCOMP_NAMES: [&str; 2] = ["libseccomp.so.2", "libseccomp.so"]; + + unsafe extern "C" { + fn dlopen(filename: *const c_char, flags: c_int) -> *mut c_void; + fn dlclose(handle: *mut c_void) -> c_int; + fn dlsym(handle: *mut c_void, symbol: *const c_char) -> *mut c_void; + fn dlerror() -> *const c_char; + } + + type SeccompInit = unsafe extern "C" fn(u32) -> *mut c_void; + type SeccompRelease = unsafe extern "C" fn(*mut c_void); + type SeccompLoad = unsafe extern "C" fn(*const c_void) -> c_int; + type SeccompSyscallResolveName = unsafe extern "C" fn(*const c_char) -> c_int; + type SeccompRuleAddArray = + unsafe extern "C" fn(*mut c_void, u32, c_int, c_uint, *const c_void) -> c_int; + + pub(super) fn install_allow_filter(rules: &[SeccompRule<'_>]) -> Result { + let Some(runtime) = Runtime::load()? else { + return Ok(SeccompInstallStatus::Unavailable( + "libseccomp shared library not found", + )); + }; + runtime.install_allow_filter(rules)?; + Ok(SeccompInstallStatus::Installed) + } + + struct Runtime { + handle: *mut c_void, + seccomp_init: SeccompInit, + seccomp_release: SeccompRelease, + seccomp_load: SeccompLoad, + seccomp_syscall_resolve_name: SeccompSyscallResolveName, + seccomp_rule_add_array: SeccompRuleAddArray, + } + + impl Runtime { + fn load() -> Result> { + for candidate in LIBSECCOMP_NAMES { + if let Some(runtime) = Self::load_candidate(candidate)? { + return Ok(Some(runtime)); + } + } + Ok(None) + } + + fn load_candidate(candidate: &str) -> Result> { + let name = CString::new(candidate).expect("libseccomp candidate name"); + let handle = unsafe { + clear_dlerror(); + dlopen(name.as_ptr(), RTLD_NOW) + }; + if handle.is_null() { + return Ok(None); + } + + let runtime = unsafe { + Self { + handle, + seccomp_init: load_symbol(handle, b"seccomp_init\0")?, + seccomp_release: load_symbol(handle, b"seccomp_release\0")?, + seccomp_load: load_symbol(handle, b"seccomp_load\0")?, + seccomp_syscall_resolve_name: load_symbol( + handle, + b"seccomp_syscall_resolve_name\0", + )?, + seccomp_rule_add_array: load_symbol(handle, b"seccomp_rule_add_array\0")?, + } + }; + + Ok(Some(runtime)) + } + + fn install_allow_filter(&self, rules: &[SeccompRule<'_>]) -> Result<()> { + let ctx = unsafe { (self.seccomp_init)(SCMP_ACT_ALLOW) }; + if ctx.is_null() { + return Err(anyhow!("seccomp init failed")); + } + let guard = FilterContextGuard { + ctx, + release: self.seccomp_release, + }; + + for rule in rules { + let name = CString::new(rule.syscall) + .with_context(|| format!("invalid seccomp syscall name '{}'", rule.syscall))?; + let syscall = unsafe { (self.seccomp_syscall_resolve_name)(name.as_ptr()) }; + if syscall == __NR_SCMP_ERROR { + continue; + } + + cvt( + unsafe { + (self.seccomp_rule_add_array)( + guard.ctx, + action_to_raw(rule.action), + syscall, + 0, + std::ptr::null(), + ) + }, + &format!("seccomp add_rule({})", rule.syscall), + )?; + } + + cvt( + unsafe { (self.seccomp_load)(guard.ctx.cast_const()) }, + "seccomp load", + ) + } + } + + impl Drop for Runtime { + fn drop(&mut self) { + unsafe { + let _ = dlclose(self.handle); + } + } + } + + struct FilterContextGuard { + ctx: *mut c_void, + release: SeccompRelease, + } + + impl Drop for FilterContextGuard { + fn drop(&mut self) { + unsafe { + (self.release)(self.ctx); + } + } + } + + unsafe fn load_symbol(handle: *mut c_void, symbol: &[u8]) -> Result + where + T: Copy, + { + clear_dlerror(); + let ptr = dlsym(handle, symbol.as_ptr().cast()); + if ptr.is_null() { + return Err(anyhow!(dlerror_message()).context(format!( + "failed to resolve {}", + String::from_utf8_lossy(symbol).trim_end_matches('\0') + ))); + } + Ok(mem::transmute_copy(&ptr)) + } + + unsafe fn clear_dlerror() { + let _ = dlerror(); + } + + unsafe fn dlerror_message() -> String { + let ptr = dlerror(); + if ptr.is_null() { + "dynamic loader error".to_string() + } else { + CStr::from_ptr(ptr).to_string_lossy().into_owned() + } + } + + fn action_to_raw(action: SeccompAction) -> u32 { + match action { + SeccompAction::Log => SCMP_ACT_LOG, + SeccompAction::Errno(errno) => SCMP_ACT_ERRNO_MASK | ((errno as u32) & 0xffff), + } + } + + fn cvt(rc: c_int, operation: &str) -> Result<()> { + if rc == 0 { + return Ok(()); + } + let errno = if rc < 0 { -rc } else { rc }; + let error = std::io::Error::from_raw_os_error(errno); + Err(error).with_context(|| operation.to_string()) + } +} + +#[cfg(all(target_os = "linux", not(target_env = "musl")))] +pub fn install_allow_filter(rules: &[SeccompRule<'_>]) -> Result { + imp::install_allow_filter(rules) +} + +#[cfg(not(all(target_os = "linux", not(target_env = "musl"))))] +pub fn install_allow_filter(_rules: &[SeccompRule<'_>]) -> Result { + Ok(SeccompInstallStatus::Unavailable( + "libseccomp unavailable on this target", + )) +} diff --git a/crates/common/src/settings.rs b/crates/common/src/settings.rs index bbda7273b..d6d17e996 100644 --- a/crates/common/src/settings.rs +++ b/crates/common/src/settings.rs @@ -4,8 +4,9 @@ use std::str::FromStr; /// Selects whether a world filesystem should be writable (overlay/copy-diff) /// or mounted read-only. -#[derive(Debug, Clone, Copy, PartialEq, Eq)] +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] pub enum WorldFsMode { + #[default] Writable, ReadOnly, } @@ -25,12 +26,6 @@ impl WorldFsMode { } } -impl Default for WorldFsMode { - fn default() -> Self { - Self::Writable - } -} - impl FromStr for WorldFsMode { type Err = String; diff --git a/crates/common/tests/agent_hub_event_envelope_schema.rs b/crates/common/tests/agent_hub_event_envelope_schema.rs index 3c9a44c4b..38e25450c 100644 --- a/crates/common/tests/agent_hub_event_envelope_schema.rs +++ b/crates/common/tests/agent_hub_event_envelope_schema.rs @@ -84,6 +84,116 @@ fn parent_run_id_omits_by_field_absence_when_unset() { ); } +#[test] +fn participant_id_roundtrips_when_present() { + let mut value = minimal_valid_envelope_json(); + value + .as_object_mut() + .expect("envelope object") + .insert("participant_id".to_string(), json!("ash_0195f8f1")); + + let event: AgentEvent = serde_json::from_value(value).expect("deserialize AgentEvent"); + let roundtrip = serde_json::to_value(&event).expect("serialize AgentEvent"); + + assert_eq!( + roundtrip.get("participant_id").and_then(Value::as_str), + Some("ash_0195f8f1"), + "expected participant_id to survive roundtrip; got: {roundtrip}" + ); +} + +#[test] +fn participant_id_omits_when_unset() { + let event: AgentEvent = + serde_json::from_value(minimal_valid_envelope_json()).expect("deserialize AgentEvent"); + let roundtrip = serde_json::to_value(&event).expect("serialize AgentEvent"); + + assert!( + roundtrip.get("participant_id").is_none(), + "expected participant_id to omit when unset; got: {roundtrip}" + ); + assert!( + !roundtrip.to_string().contains("\"participant_id\":null"), + "participant_id must not serialize as null when unset: {roundtrip}" + ); +} + +#[test] +fn parent_participant_id_roundtrips_when_present() { + let mut value = minimal_valid_envelope_json(); + value + .as_object_mut() + .expect("envelope object") + .insert("parent_participant_id".to_string(), json!("ash_parent")); + + let event: AgentEvent = serde_json::from_value(value).expect("deserialize AgentEvent"); + let roundtrip = serde_json::to_value(&event).expect("serialize AgentEvent"); + + assert_eq!( + roundtrip + .get("parent_participant_id") + .and_then(Value::as_str), + Some("ash_parent"), + "expected parent_participant_id to survive roundtrip; got: {roundtrip}" + ); +} + +#[test] +fn parent_participant_id_omits_when_unset() { + let event: AgentEvent = + serde_json::from_value(minimal_valid_envelope_json()).expect("deserialize AgentEvent"); + let roundtrip = serde_json::to_value(&event).expect("serialize AgentEvent"); + + assert!( + roundtrip.get("parent_participant_id").is_none(), + "expected parent_participant_id to omit when unset; got: {roundtrip}" + ); + assert!( + !roundtrip + .to_string() + .contains("\"parent_participant_id\":null"), + "parent_participant_id must not serialize as null when unset: {roundtrip}" + ); +} + +#[test] +fn resumed_from_participant_id_roundtrips_when_present() { + let mut value = minimal_valid_envelope_json(); + value.as_object_mut().expect("envelope object").insert( + "resumed_from_participant_id".to_string(), + json!("ash_previous"), + ); + + let event: AgentEvent = serde_json::from_value(value).expect("deserialize AgentEvent"); + let roundtrip = serde_json::to_value(&event).expect("serialize AgentEvent"); + + assert_eq!( + roundtrip + .get("resumed_from_participant_id") + .and_then(Value::as_str), + Some("ash_previous"), + "expected resumed_from_participant_id to survive roundtrip; got: {roundtrip}" + ); +} + +#[test] +fn resumed_from_participant_id_omits_when_unset() { + let event: AgentEvent = + serde_json::from_value(minimal_valid_envelope_json()).expect("deserialize AgentEvent"); + let roundtrip = serde_json::to_value(&event).expect("serialize AgentEvent"); + + assert!( + roundtrip.get("resumed_from_participant_id").is_none(), + "expected resumed_from_participant_id to omit when unset; got: {roundtrip}" + ); + assert!( + !roundtrip + .to_string() + .contains("\"resumed_from_participant_id\":null"), + "resumed_from_participant_id must not serialize as null when unset: {roundtrip}" + ); +} + #[test] fn safe_channel_roundtrips() { let mut value = minimal_valid_envelope_json(); diff --git a/crates/gateway/Cargo.toml b/crates/gateway/Cargo.toml index 3e852926c..b46d2d622 100644 --- a/crates/gateway/Cargo.toml +++ b/crates/gateway/Cargo.toml @@ -70,6 +70,7 @@ rand = "0.8" # Random generation for PKCE chrono = { workspace = true } # Timestamps url = "2" # URL parsing secrecy = "0.8" # Secure secret handling +substrate-common = { path = "../common" } [target.'cfg(unix)'.dependencies] nix = { version = "0.29", features = ["signal"] } # Unix signals diff --git a/crates/gateway/docs/contracts/chatgpt-codex-auth-handoff-contract.md b/crates/gateway/docs/contracts/chatgpt-codex-auth-handoff-contract.md index 11862bc47..40825d7c7 100644 --- a/crates/gateway/docs/contracts/chatgpt-codex-auth-handoff-contract.md +++ b/crates/gateway/docs/contracts/chatgpt-codex-auth-handoff-contract.md @@ -23,9 +23,8 @@ This contract does not own: - Substrate owns policy-gated host credential reads, auth-state validation, and host-to-world delivery for the ChatGPT Codex auth material required by this route. - The gateway bootstrap selects integrated versus standalone auth mode before provider construction; provider code consumes the selected auth source and does not become the owner of host credential reads or trust-boundary decisions. -- Current v1 integrated delivery may place secret-bearing values for the closed `SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_*` field set directly in the in-world gateway/manager process environment. -- That v1 env-based delivery remains subordinate to the Substrate-owned trust boundary above: host credential reads stay policy-gated, values are not authoritative because they are in env vars, and the gateway remains only a consumer of the delivered auth context. -- The preferred additive direction remains a secret-channel payload plus inherited FD/pipe-style auth-bundle delivery so secret values do not live in the in-world process environment by default. +- The landed integrated delivery path is a Substrate-owned auth-bundle payload delivered over inherited FD handoff via `SUBSTRATE_LLM_AUTH_BUNDLE_FD`. +- The gateway must read that bundle once during integrated startup, remove the pointer env after reading it, and consume the decoded auth context without depending on secret-bearing child env vars. - The canonical integrated-mode field identifiers are: - `SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCOUNT_ID` - `SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCESS_TOKEN` @@ -41,7 +40,7 @@ Runtime bootstrap note: - `in_world` selects the integrated auth source for this route and must be paired with an explicit config-path handoff plus either an explicit gateway token-store path or explicit persistence disablement. - `host_only` selects the bounded standalone compatibility source for gateway-local operation and may still use the local convenience defaults under `~/.substrate-gateway/`. -The gateway must treat those field names as the authoritative integrated-mode inputs for this route, whether the current runtime receives their values through v1 env delivery or a later auth-bundle carrier. +The gateway must treat those field names as the authoritative integrated-mode inputs for this route as carried inside the startup-owned auth bundle. ## Standalone Compatibility Mode diff --git a/crates/gateway/src/auth/codex_auth_context.rs b/crates/gateway/src/auth/codex_auth_context.rs index 8720b262d..b01aa3b15 100644 --- a/crates/gateway/src/auth/codex_auth_context.rs +++ b/crates/gateway/src/auth/codex_auth_context.rs @@ -2,8 +2,9 @@ use anyhow::{anyhow, Context, Result}; use base64::{engine::general_purpose::URL_SAFE_NO_PAD, Engine as _}; +use once_cell::sync::Lazy; use secrecy::{ExposeSecret, SecretString}; -use std::{env, path::PathBuf}; +use std::{collections::HashMap, env, path::PathBuf, sync::RwLock}; use super::codex_auth_state::CodexAuthState; @@ -38,6 +39,12 @@ pub struct CodexIntegratedAuthHandoff { pub access_token: SecretString, } +static INTEGRATED_CODEX_AUTH_HANDOFF: Lazy>> = + Lazy::new(|| RwLock::new(None)); +#[cfg(test)] +static INTEGRATED_CODEX_AUTH_TEST_LOCK: Lazy> = + Lazy::new(|| tokio::sync::Mutex::new(())); + impl CodexIntegratedAuthHandoff { pub fn new(account_id: Option, access_token: SecretString) -> Self { Self { @@ -46,6 +53,26 @@ impl CodexIntegratedAuthHandoff { } } + pub fn from_fields(fields: &HashMap) -> Result { + let access_token = fields + .get(SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCESS_TOKEN) + .map(|value| value.trim().to_string()) + .filter(|value| !value.is_empty()) + .ok_or_else(|| { + anyhow!( + "integrated Codex auth handoff is missing required field {}", + SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCESS_TOKEN + ) + })?; + + let account_id = fields + .get(SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCOUNT_ID) + .map(|value| value.trim().to_string()) + .filter(|value| !value.is_empty()); + + Ok(Self::new(account_id, SecretString::new(access_token))) + } + pub fn from_env() -> Result> { let access_token = read_env_trimmed(SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCESS_TOKEN)?; let Some(access_token) = access_token else { @@ -66,6 +93,49 @@ impl CodexIntegratedAuthHandoff { } } +pub(crate) fn install_integrated_codex_auth_handoff( + handoff: Option, +) -> Result<()> { + let mut guard = INTEGRATED_CODEX_AUTH_HANDOFF + .write() + .map_err(|_| anyhow!("integrated Codex auth handoff lock poisoned"))?; + *guard = handoff; + Ok(()) +} + +fn integrated_codex_auth_handoff() -> Result> { + INTEGRATED_CODEX_AUTH_HANDOFF + .read() + .map(|guard| guard.clone()) + .map_err(|_| anyhow!("integrated Codex auth handoff lock poisoned")) +} + +#[cfg(test)] +pub(crate) fn integrated_codex_auth_test_lock() -> &'static tokio::sync::Mutex<()> { + &INTEGRATED_CODEX_AUTH_TEST_LOCK +} + +#[cfg(test)] +pub(crate) struct InstalledHandoffGuard { + previous: Option, +} + +#[cfg(test)] +impl InstalledHandoffGuard { + pub(crate) fn set(next: Option) -> Self { + let previous = integrated_codex_auth_handoff().unwrap(); + install_integrated_codex_auth_handoff(next).unwrap(); + Self { previous } + } +} + +#[cfg(test)] +impl Drop for InstalledHandoffGuard { + fn drop(&mut self) { + install_integrated_codex_auth_handoff(self.previous.take()).unwrap(); + } +} + #[derive(Debug, Clone)] pub enum CodexAuthSource { Integrated, @@ -83,9 +153,9 @@ impl CodexAuthSource { pub fn resolve(&self) -> Result { match self { Self::Integrated => { - let Some(handoff) = CodexIntegratedAuthHandoff::from_env()? else { + let Some(handoff) = integrated_codex_auth_handoff()? else { return Err(anyhow!( - "integrated Codex auth source is unavailable: Substrate-delivered auth handoff is missing" + "integrated Codex auth source is unavailable: startup-owned integrated auth handoff is missing" )); }; @@ -179,12 +249,8 @@ mod tests { use base64::engine::general_purpose::URL_SAFE_NO_PAD; use secrecy::ExposeSecret; use std::fs; - use std::sync::Mutex; use tempfile::TempDir; - static ENV_LOCK: once_cell::sync::Lazy> = - once_cell::sync::Lazy::new(|| Mutex::new(())); - fn codex_access_token(account_id: &str) -> SecretString { let payload = serde_json::json!({ "https://api.openai.com/auth": { @@ -276,8 +342,30 @@ mod tests { #[test] fn integrated_source_uses_canonical_field_names() { - let _env_lock_guard = ENV_LOCK.lock().unwrap(); + let _env_lock_guard = integrated_codex_auth_test_lock().blocking_lock(); + let _handoff_guard = InstalledHandoffGuard::set(Some( + CodexIntegratedAuthHandoff::from_fields(&HashMap::from([ + ( + SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCOUNT_ID.to_string(), + "acct_env_explicit".to_string(), + ), + ( + SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCESS_TOKEN.to_string(), + "header.eyJodHRwczovL2FwaS5vcGVuYWkuY29tL2F1dGgiOnsiY2hhdGdwdF9hY2NvdW50X2lkIjoiYWNjdF9lbnZfand0In19.signature".to_string(), + ), + ])) + .unwrap(), + )); + + let resolved = CodexAuthSource::Integrated.resolve().unwrap(); + assert_eq!(resolved.mode, CodexAuthMode::Integrated); + assert_eq!(resolved.account_id, "acct_env_explicit"); + assert_eq!(resolved.account_id_source, CodexAccountIdSource::Explicit); + } + #[test] + fn from_env_reads_canonical_cli_codex_field_names() { + let _env_lock_guard = integrated_codex_auth_test_lock().blocking_lock(); let _account_id_guard = EnvGuard::set( SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCOUNT_ID, "acct_env_explicit", @@ -287,10 +375,9 @@ mod tests { "header.eyJodHRwczovL2FwaS5vcGVuYWkuY29tL2F1dGgiOnsiY2hhdGdwdF9hY2NvdW50X2lkIjoiYWNjdF9lbnZfand0In19.signature", ); - let resolved = CodexAuthSource::Integrated.resolve().unwrap(); - assert_eq!(resolved.mode, CodexAuthMode::Integrated); - assert_eq!(resolved.account_id, "acct_env_explicit"); - assert_eq!(resolved.account_id_source, CodexAccountIdSource::Explicit); + let handoff = CodexIntegratedAuthHandoff::from_env().unwrap().unwrap(); + assert_eq!(handoff.account_id.as_deref(), Some("acct_env_explicit")); + assert!(handoff.access_token.expose_secret().contains('.')); } struct EnvGuard { @@ -324,27 +411,21 @@ mod tests { #[test] fn integrated_source_requires_substrate_handoff() { - let _env_lock_guard = ENV_LOCK.lock().unwrap(); - - let _account_id_guard = EnvGuard::clear(SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCOUNT_ID); - let _access_token_guard = - EnvGuard::clear(SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCESS_TOKEN); + let _env_lock_guard = integrated_codex_auth_test_lock().blocking_lock(); + let _handoff_guard = InstalledHandoffGuard::set(None); let err = CodexAuthSource::Integrated.resolve().unwrap_err(); assert!( err.to_string() - .contains("Substrate-delivered auth handoff is missing"), + .contains("startup-owned integrated auth handoff is missing"), "unexpected error: {err}" ); } #[test] fn integrated_source_does_not_read_local_auth_files() { - let _env_lock_guard = ENV_LOCK.lock().unwrap(); - - let _account_id_guard = EnvGuard::clear(SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCOUNT_ID); - let _access_token_guard = - EnvGuard::clear(SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCESS_TOKEN); + let _env_lock_guard = integrated_codex_auth_test_lock().blocking_lock(); + let _handoff_guard = InstalledHandoffGuard::set(None); let temp_dir = TempDir::new().unwrap(); let path = temp_dir.path().join("auth.json"); @@ -360,11 +441,42 @@ mod tests { let err = CodexAuthSource::Integrated.resolve().unwrap_err(); assert!( err.to_string() - .contains("Substrate-delivered auth handoff is missing"), + .contains("startup-owned integrated auth handoff is missing"), "unexpected error: {err}" ); } + #[test] + fn integrated_handoff_from_fields_accepts_required_cli_codex_fields() { + let handoff = CodexIntegratedAuthHandoff::from_fields(&HashMap::from([ + ( + SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCESS_TOKEN.to_string(), + codex_access_token("acct_jwt").expose_secret().to_string(), + ), + ( + SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCOUNT_ID.to_string(), + "acct_explicit".to_string(), + ), + ])) + .unwrap(); + + assert_eq!(handoff.account_id.as_deref(), Some("acct_explicit")); + assert!(handoff.access_token.expose_secret().contains('.')); + } + + #[test] + fn integrated_handoff_from_fields_rejects_missing_access_token() { + let err = CodexIntegratedAuthHandoff::from_fields(&HashMap::from([( + SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCOUNT_ID.to_string(), + "acct_explicit".to_string(), + )])) + .unwrap_err(); + + assert!(err + .to_string() + .contains(SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCESS_TOKEN)); + } + #[test] fn standalone_local_source_uses_explicit_path() { let temp_dir = TempDir::new().unwrap(); diff --git a/crates/gateway/src/cli/mod.rs b/crates/gateway/src/cli/mod.rs index f50bdac9e..9fda6cf10 100644 --- a/crates/gateway/src/cli/mod.rs +++ b/crates/gateway/src/cli/mod.rs @@ -1,7 +1,12 @@ +use crate::auth::codex_auth_context::{ + SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCESS_TOKEN, + SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCOUNT_ID, +}; use crate::providers::ProviderConfig; use anyhow::{Context, Result}; use serde::{Deserialize, Serialize}; use std::path::PathBuf; +use substrate_common::SUBSTRATE_LLM_BACKEND_AUTH_API_OPENAI_API_KEY; /// Application configuration #[derive(Debug, Clone, Deserialize, Serialize)] @@ -67,6 +72,12 @@ fn resolve_env_var(value: &str) -> Option { .and_then(|env_var| std::env::var(env_var).ok()) } +fn referenced_env_var(value: &str) -> Option<&str> { + value.strip_prefix('$') +} + +const OPENAI_API_KEY_ENV: &str = "OPENAI_API_KEY"; + impl Default for ServerConfig { fn default() -> Self { Self { @@ -191,7 +202,18 @@ impl AppConfig { } /// Load configuration from a TOML file + #[allow(dead_code)] pub fn from_file(path: &PathBuf) -> Result { + let mut config = Self::parse_file_without_env_resolution(path)?; + + // Resolve environment variables + config.resolve_env_vars()?; + + Ok(config) + } + + /// Load configuration from a TOML file without interpolating environment variables. + pub fn parse_file_without_env_resolution(path: &PathBuf) -> Result { // Check if file exists, if not create a default one if !path.exists() { Self::create_default_config(path)?; @@ -200,12 +222,9 @@ impl AppConfig { let content = std::fs::read_to_string(path) .with_context(|| format!("Failed to read config file: {}", path.display()))?; - let mut config: AppConfig = toml::from_str(&content) + let config: AppConfig = toml::from_str(&content) .with_context(|| format!("Failed to parse config file: {}", path.display()))?; - // Resolve environment variables - config.resolve_env_vars()?; - Ok(config) } @@ -347,9 +366,28 @@ default = "placeholder-model" } /// Resolve environment variables in configuration - fn resolve_env_vars(&mut self) -> Result<()> { + pub(crate) fn resolve_env_vars(&mut self) -> Result<()> { + self.resolve_env_vars_with_blocked_secrets(&[]) + } + + pub(crate) fn resolve_env_vars_for_integrated_mode(&mut self) -> Result<()> { + self.resolve_env_vars_with_blocked_secrets(&[ + SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCOUNT_ID, + SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCESS_TOKEN, + SUBSTRATE_LLM_BACKEND_AUTH_API_OPENAI_API_KEY, + OPENAI_API_KEY_ENV, + ]) + } + + fn resolve_env_vars_with_blocked_secrets(&mut self, blocked_env_vars: &[&str]) -> Result<()> { // Resolve server API key if let Some(ref key) = self.server.api_key { + ensure_env_reference_allowed( + key, + blocked_env_vars, + "server.api_key", + self.server.host.as_str(), + )?; if let Some(resolved) = resolve_env_var(key) { self.server.api_key = Some(resolved); } @@ -364,6 +402,12 @@ default = "placeholder-model" // Only resolve env vars for API key auth if let Some(ref api_key) = provider.api_key { + ensure_env_reference_allowed( + api_key, + blocked_env_vars, + "providers.api_key", + provider.name.as_str(), + )?; if let Some(env_var) = api_key.strip_prefix('$') { if let Some(resolved) = resolve_env_var(api_key) { provider.api_key = Some(resolved); @@ -378,6 +422,12 @@ default = "placeholder-model" } if let Some(ref base_url) = provider.base_url { + ensure_env_reference_allowed( + base_url, + blocked_env_vars, + "providers.base_url", + provider.name.as_str(), + )?; if let Some(env_var) = base_url.strip_prefix('$') { if let Some(resolved) = resolve_env_var(base_url) { provider.base_url = Some(resolved); @@ -396,6 +446,28 @@ default = "placeholder-model" } } +fn ensure_env_reference_allowed( + value: &str, + blocked_env_vars: &[&str], + field_name: &str, + owner_name: &str, +) -> Result<()> { + let Some(env_var) = referenced_env_var(value) else { + return Ok(()); + }; + + if blocked_env_vars.contains(&env_var) { + anyhow::bail!( + "Integrated gateway startup forbids env-based secret delivery via ${} for {} '{}'", + env_var, + field_name, + owner_name + ); + } + + Ok(()) +} + // TODO: Re-enable these tests by adding tempfile to dev-dependencies // #[cfg(test)] // mod tests { diff --git a/crates/gateway/src/main.rs b/crates/gateway/src/main.rs index 604be64af..22e72993c 100644 --- a/crates/gateway/src/main.rs +++ b/crates/gateway/src/main.rs @@ -49,6 +49,8 @@ async fn start_foreground( config: cli::AppConfig, launch: GatewayLaunchContract, ) -> anyhow::Result<()> { + let integrated_auth = server::IntegratedGatewayAuthContext::from_launch_mode(launch.mode)?; + // Write PID file if let Err(e) = pid::write_pid() { eprintln!("Warning: Failed to write PID file: {}", e); @@ -77,7 +79,7 @@ async fn start_foreground( println!(); println!("Press Ctrl+C to stop"); - let result = server::start_server(config, launch).await; + let result = server::start_server(config, launch, integrated_auth).await; let _ = pid::cleanup_pid(); result } @@ -121,7 +123,7 @@ fn resolve_launch_and_config( cli::AppConfig::default_path, auth::token_store::TokenStore::default_path, )?; - let config = cli::AppConfig::from_file(&launch.config_path)?; + let config = cli::AppConfig::parse_file_without_env_resolution(&launch.config_path)?; Ok((launch, config)) } diff --git a/crates/gateway/src/providers/openai.rs b/crates/gateway/src/providers/openai.rs index c87afe72f..920d3880c 100644 --- a/crates/gateway/src/providers/openai.rs +++ b/crates/gateway/src/providers/openai.rs @@ -3650,19 +3650,17 @@ impl super::GatewayProvider for OpenAIProvider { #[cfg(test)] mod tests { use super::*; - use crate::auth::codex_auth_context::{CodexAccountIdSource, CodexAuthMode}; + use crate::auth::codex_auth_context::{ + integrated_codex_auth_test_lock, CodexAccountIdSource, CodexAuthMode, + CodexIntegratedAuthHandoff, InstalledHandoffGuard, + }; use crate::auth::CodexAuthSource; use crate::models::SystemPrompt; use crate::providers::GatewayProvider; use base64::engine::general_purpose::URL_SAFE_NO_PAD; use secrecy::SecretString; use serde::Deserialize; - use std::{env, fs}; - use tempfile::TempDir; - use tokio::sync::Mutex; - - static ENV_LOCK: once_cell::sync::Lazy> = - once_cell::sync::Lazy::new(|| Mutex::new(())); + use std::env; #[derive(Debug, Deserialize)] struct FixtureFile { @@ -3826,58 +3824,18 @@ mod tests { } #[test] - fn codex_auth_resolution_prefers_integrated_env_handoff_before_local_path() { - let _guard = ENV_LOCK.blocking_lock(); - let temp_dir = TempDir::new().unwrap(); - let bogus_home = temp_dir.path().join("home-as-file"); - fs::write(&bogus_home, "not a directory").unwrap(); - - let original_home = env::var_os("HOME"); - let original_account_id = env::var_os( - crate::auth::codex_auth_context::SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCOUNT_ID, - ); - let original_access_token = env::var_os( - crate::auth::codex_auth_context::SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCESS_TOKEN, - ); - - env::set_var("HOME", &bogus_home); - env::set_var( - crate::auth::codex_auth_context::SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCOUNT_ID, - "acct_env_explicit", - ); - env::set_var( - crate::auth::codex_auth_context::SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCESS_TOKEN, - codex_access_token("acct_env_jwt"), - ); + fn codex_auth_resolution_uses_startup_owned_integrated_handoff() { + let _guard = integrated_codex_auth_test_lock().blocking_lock(); + let _handoff_guard = InstalledHandoffGuard::set(Some(CodexIntegratedAuthHandoff::new( + Some("acct_handoff_explicit".to_string()), + SecretString::new(codex_access_token("acct_handoff_jwt")), + ))); let provider = test_oauth_provider(); let resolved = provider.resolve_codex_auth_context().unwrap(); - assert_eq!(resolved.account_id, "acct_env_explicit"); + assert_eq!(resolved.account_id, "acct_handoff_explicit"); assert_eq!(resolved.account_id_source, CodexAccountIdSource::Explicit); - - match original_home { - Some(value) => env::set_var("HOME", value), - None => env::remove_var("HOME"), - } - match original_account_id { - Some(value) => env::set_var( - crate::auth::codex_auth_context::SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCOUNT_ID, - value, - ), - None => env::remove_var( - crate::auth::codex_auth_context::SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCOUNT_ID, - ), - } - match original_access_token { - Some(value) => env::set_var( - crate::auth::codex_auth_context::SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCESS_TOKEN, - value, - ), - None => env::remove_var( - crate::auth::codex_auth_context::SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCESS_TOKEN, - ), - } } fn codex_request_with_tool_choice(tool_choice: serde_json::Value) -> GatewayRequest { @@ -4896,7 +4854,7 @@ mod tests { #[tokio::test(flavor = "current_thread")] async fn send_message_validates_codex_route_before_resolving_integrated_auth() { - let _guard = ENV_LOCK.lock().await; + let _guard = integrated_codex_auth_test_lock().lock().await; let provider = test_oauth_provider(); let original_account_id = env::var_os( crate::auth::codex_auth_context::SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCOUNT_ID, diff --git a/crates/gateway/src/server/mod.rs b/crates/gateway/src/server/mod.rs index cb3c2629c..a122a440c 100644 --- a/crates/gateway/src/server/mod.rs +++ b/crates/gateway/src/server/mod.rs @@ -3,14 +3,17 @@ mod openai_compat; pub mod openai_conformance_test_support; mod openai_responses; +use crate::auth::codex_auth_context::{ + install_integrated_codex_auth_handoff, CodexIntegratedAuthHandoff, +}; use crate::auth::TokenStore; use crate::cli::AppConfig; use crate::core::GatewayRequest; -use crate::launch::{GatewayLaunchContract, TokenStoreStrategy}; +use crate::launch::{GatewayLaunchContract, GatewayMode, TokenStoreStrategy}; use crate::message_tracing::MessageTracer; use crate::models::{AnthropicMessagesRequest, RouteType}; use crate::providers::error::ProviderError; -use crate::providers::ProviderRegistry; +use crate::providers::{AuthType, ProviderRegistry}; use crate::router::Router; use crate::structured_events::{ normalized_events_from_provider_response, AnthropicSseNormalizedEventExtractor, @@ -26,11 +29,76 @@ use axum::{ }; use chrono::Local; use futures::stream::TryStreamExt; +use std::env; +#[cfg(unix)] +use std::io::Read; use std::pin::Pin; use std::sync::Arc; +use substrate_common::{ + GatewayAuthBundleV1, GATEWAY_AUTH_BUNDLE_BACKEND_API_OPENAI, + GATEWAY_AUTH_BUNDLE_BACKEND_CLI_CODEX, SUBSTRATE_LLM_AUTH_BUNDLE_FD, + SUBSTRATE_LLM_BACKEND_AUTH_API_OPENAI_API_KEY, +}; use tokio::net::TcpListener; use tracing::{debug, error, info}; +const OPENAI_API_KEY_ENV: &str = "OPENAI_API_KEY"; + +#[derive(Debug, Clone)] +pub enum IntegratedGatewayAuthContext { + CliCodex(CodexIntegratedAuthHandoff), + ApiOpenAI { api_key: String }, +} + +impl IntegratedGatewayAuthContext { + pub fn from_launch_mode(mode: GatewayMode) -> anyhow::Result> { + match mode { + GatewayMode::HostOnly => Ok(None), + GatewayMode::InWorld => Self::from_auth_bundle_env().map(Some), + } + } + + fn from_auth_bundle_env() -> anyhow::Result { + let bundle = read_gateway_auth_bundle_from_env()?; + match bundle.backend_id.as_str() { + GATEWAY_AUTH_BUNDLE_BACKEND_CLI_CODEX => Ok(Self::CliCodex( + CodexIntegratedAuthHandoff::from_fields(&bundle.fields)?, + )), + GATEWAY_AUTH_BUNDLE_BACKEND_API_OPENAI => { + let api_key = bundle + .fields + .get(SUBSTRATE_LLM_BACKEND_AUTH_API_OPENAI_API_KEY) + .map(|value| value.trim().to_string()) + .filter(|value| !value.is_empty()) + .ok_or_else(|| { + anyhow::anyhow!( + "Integrated gateway startup received invalid gateway auth bundle: gateway auth bundle for '{}' is missing required field '{}'", + GATEWAY_AUTH_BUNDLE_BACKEND_API_OPENAI, + SUBSTRATE_LLM_BACKEND_AUTH_API_OPENAI_API_KEY + ) + })?; + Ok(Self::ApiOpenAI { api_key }) + } + other => Err(anyhow::anyhow!( + "Integrated gateway startup received invalid gateway auth bundle: unsupported gateway auth bundle backend_id '{}'", + other + )), + } + } + + fn apply_to_config(&self, config: &mut AppConfig) -> anyhow::Result<()> { + install_integrated_codex_auth_handoff(None)?; + + match self { + Self::CliCodex(handoff) => { + install_integrated_codex_auth_handoff(Some(handoff.clone()))?; + Ok(()) + } + Self::ApiOpenAI { api_key } => overlay_api_openai_auth(config, api_key), + } + } +} + /// Reloadable components - rebuilt on config reload pub struct ReloadableState { pub config: AppConfig, @@ -85,6 +153,136 @@ fn build_app(state: Arc) -> AxumRouter { .with_state(state) } +fn prepare_startup_config( + config: &mut AppConfig, + mode: GatewayMode, + integrated_auth: Option<&IntegratedGatewayAuthContext>, +) -> anyhow::Result<()> { + install_integrated_codex_auth_handoff(None)?; + + match mode { + GatewayMode::HostOnly => { + if integrated_auth.is_some() { + anyhow::bail!( + "Host-only gateway startup does not accept an integrated auth bundle context" + ); + } + config.resolve_env_vars()?; + } + GatewayMode::InWorld => { + let integrated_auth = integrated_auth.ok_or_else(|| { + anyhow::anyhow!( + "Integrated gateway startup requires a startup-owned auth bundle context before provider initialization" + ) + })?; + integrated_auth.apply_to_config(config)?; + config.resolve_env_vars_for_integrated_mode()?; + } + } + + Ok(()) +} + +fn overlay_api_openai_auth(config: &mut AppConfig, api_key: &str) -> anyhow::Result<()> { + let mut applied = 0usize; + + for provider in &mut config.providers { + if !provider.is_enabled() || provider.auth_type != AuthType::ApiKey { + continue; + } + + match provider.api_key.as_deref() { + Some(value) + if value == format!("${OPENAI_API_KEY_ENV}") + || value == format!("${SUBSTRATE_LLM_BACKEND_AUTH_API_OPENAI_API_KEY}") => + { + provider.api_key = Some(api_key.to_string()); + applied += 1; + } + _ => {} + } + } + + if applied == 0 { + anyhow::bail!( + "Integrated gateway startup could not apply api:openai auth bundle: no provider api_key matched $OPENAI_API_KEY or $SUBSTRATE_LLM_BACKEND_AUTH_API_OPENAI_API_KEY" + ); + } + + Ok(()) +} + +fn take_auth_bundle_fd_env() -> anyhow::Result { + match env::var(SUBSTRATE_LLM_AUTH_BUNDLE_FD) { + Ok(value) => { + env::remove_var(SUBSTRATE_LLM_AUTH_BUNDLE_FD); + Ok(value) + } + Err(env::VarError::NotPresent) => Err(anyhow::anyhow!( + "Integrated gateway startup is missing pointer env {}", + SUBSTRATE_LLM_AUTH_BUNDLE_FD + )), + Err(err) => Err(anyhow::anyhow!( + "Integrated gateway startup could not read pointer env {}: {}", + SUBSTRATE_LLM_AUTH_BUNDLE_FD, + err + )), + } +} + +#[cfg(unix)] +fn read_gateway_auth_bundle_from_env() -> anyhow::Result { + use std::fs::File; + use std::os::fd::{FromRawFd, OwnedFd, RawFd}; + + let raw_fd = take_auth_bundle_fd_env()?; + let fd = raw_fd.trim().parse::().map_err(|err| { + anyhow::anyhow!( + "Integrated gateway startup could not parse {} as a file descriptor: {}", + SUBSTRATE_LLM_AUTH_BUNDLE_FD, + err + ) + })?; + + let mut body = String::new(); + { + let owned_fd = unsafe { OwnedFd::from_raw_fd(fd) }; + let mut file = File::from(owned_fd); + file.read_to_string(&mut body).map_err(|err| { + anyhow::anyhow!( + "Integrated gateway startup could not read auth bundle from {}: {}", + SUBSTRATE_LLM_AUTH_BUNDLE_FD, + err + ) + })?; + } + + let bundle: GatewayAuthBundleV1 = serde_json::from_str(&body).map_err(|err| { + anyhow::anyhow!( + "Integrated gateway startup received malformed gateway auth bundle JSON: {}", + err + ) + })?; + + bundle.validate().map_err(|err| { + anyhow::anyhow!( + "Integrated gateway startup received invalid gateway auth bundle: {}", + err + ) + })?; + + Ok(bundle) +} + +#[cfg(not(unix))] +fn read_gateway_auth_bundle_from_env() -> anyhow::Result { + let _ = take_auth_bundle_fd_env(); + anyhow::bail!( + "Integrated gateway startup via {} is unsupported on this platform", + SUBSTRATE_LLM_AUTH_BUNDLE_FD + ) +} + const RECENT_REQUESTS_WINDOW: usize = 20; #[derive(Debug, Clone, Copy, PartialEq, Eq)] @@ -231,12 +429,18 @@ fn write_routing_info(model: &str, provider: &str, route_type: &RouteType) { } /// Start the HTTP server -pub async fn start_server(config: AppConfig, launch: GatewayLaunchContract) -> anyhow::Result<()> { - let router = Router::new(config.clone()); +pub async fn start_server( + mut config: AppConfig, + launch: GatewayLaunchContract, + integrated_auth: Option, +) -> anyhow::Result<()> { let GatewayLaunchContract { mode, token_store, .. } = launch; + prepare_startup_config(&mut config, mode, integrated_auth.as_ref())?; + let router = Router::new(config.clone()); + // Initialize OAuth token store FIRST (needed by provider registry) let token_store = match token_store { TokenStoreStrategy::Persistent(path) => TokenStore::new(path), diff --git a/crates/gateway/tests/openai_shared_parity.rs b/crates/gateway/tests/openai_shared_parity.rs index 108ae2ba8..705a76ea1 100644 --- a/crates/gateway/tests/openai_shared_parity.rs +++ b/crates/gateway/tests/openai_shared_parity.rs @@ -9,11 +9,23 @@ use serde::Deserialize; use serde_json::{json, Value}; use std::env; use std::fs; +#[cfg(unix)] +use std::io::{Seek, SeekFrom, Write}; +#[cfg(unix)] +use std::os::fd::IntoRawFd; use std::path::PathBuf; +#[cfg(unix)] +use substrate_common::{ + GatewayAuthBundleV1, GATEWAY_AUTH_BUNDLE_BACKEND_API_OPENAI, + GATEWAY_AUTH_BUNDLE_BACKEND_CLI_CODEX, SUBSTRATE_LLM_AUTH_BUNDLE_FD, + SUBSTRATE_LLM_BACKEND_AUTH_API_OPENAI_API_KEY, +}; use substrate_gateway::auth::CodexAuthSource; use substrate_gateway::core::{ GatewayRequest, GatewayResponse, GatewayStreamResponse, GatewayUsage, }; +#[cfg(unix)] +use substrate_gateway::launch::GatewayMode; use substrate_gateway::models::{ContentBlock, MessageContent}; use substrate_gateway::providers::error::ProviderError; use substrate_gateway::providers::{GatewayProvider, ProviderRegistry}; @@ -21,6 +33,8 @@ use substrate_gateway::server::openai_conformance_test_support::{ read_json_fixture, response_text, response_text_response, ConformanceHarness, FixtureNamespace, StubProvider, }; +#[cfg(unix)] +use substrate_gateway::server::IntegratedGatewayAuthContext; use tempfile::TempDir; use tokio::sync::Mutex; @@ -324,6 +338,37 @@ impl Drop for EnvVarGuard { } } +#[cfg(unix)] +struct GatewayAuthBundleFdGuard { + original: Option, +} + +#[cfg(unix)] +impl GatewayAuthBundleFdGuard { + fn install(bundle: GatewayAuthBundleV1) -> Self { + let mut file = tempfile::tempfile().expect("create auth bundle tempfile"); + serde_json::to_writer(&mut file, &bundle).expect("serialize auth bundle"); + file.flush().expect("flush auth bundle"); + file.seek(SeekFrom::Start(0)) + .expect("rewind auth bundle tempfile"); + let raw_fd = file.into_raw_fd(); + + let original = env::var(SUBSTRATE_LLM_AUTH_BUNDLE_FD).ok(); + env::set_var(SUBSTRATE_LLM_AUTH_BUNDLE_FD, raw_fd.to_string()); + Self { original } + } +} + +#[cfg(unix)] +impl Drop for GatewayAuthBundleFdGuard { + fn drop(&mut self) { + match self.original.as_deref() { + Some(value) => env::set_var(SUBSTRATE_LLM_AUTH_BUNDLE_FD, value), + None => env::remove_var(SUBSTRATE_LLM_AUTH_BUNDLE_FD), + } + } +} + #[derive(Debug, Clone)] struct CodexAuthProbeProvider { base_url: String, @@ -948,50 +993,85 @@ async fn provider_auth_failure_maps_to_shared_auth_envelope() { assert_provider_error_shape(&json, "auth", "Authentication failed"); } +#[cfg(unix)] #[tokio::test] -async fn integrated_codex_env_handoff_succeeds_without_local_auth_files_at_route_boundary() { +async fn integrated_gateway_startup_reads_cli_codex_bundle_once_from_pointer_env() { let _guard = ENV_LOCK.lock().await; - let mut server = Server::new_async().await; - let access_token = codex_access_token("acct_env_jwt"); - let upstream = server - .mock("POST", "/backend-api/codex/responses") - .match_header("authorization", format!("Bearer {}", access_token).as_str()) - .match_header("chatgpt-account-id", "acct_env_explicit") - .match_header("content-type", "application/json") - .with_status(200) - .with_header("content-type", "text/event-stream") - .with_body(codex_sync_sse_body("Integrated auth route proof")) - .create_async() - .await; + let _openai = EnvVarGuard::unset("OPENAI_API_KEY"); + let _account_id = EnvVarGuard::unset(SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCOUNT_ID); + let _access_token = EnvVarGuard::unset(SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCESS_TOKEN); + let _bundle = GatewayAuthBundleFdGuard::install(GatewayAuthBundleV1 { + schema_version: 1, + backend_id: GATEWAY_AUTH_BUNDLE_BACKEND_CLI_CODEX.to_string(), + fields: std::collections::HashMap::from([ + ( + SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCOUNT_ID.to_string(), + "acct_bundle_explicit".to_string(), + ), + ( + SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCESS_TOKEN.to_string(), + codex_access_token("acct_bundle_jwt"), + ), + ]), + }); - let bad_home = TempDir::new().unwrap(); - let bad_home_path = bad_home.path().join("home-as-file"); - fs::write(&bad_home_path, "not a directory").unwrap(); + let context = IntegratedGatewayAuthContext::from_launch_mode(GatewayMode::InWorld) + .expect("read cli:codex auth bundle") + .expect("in-world startup should produce integrated auth context"); + match context { + IntegratedGatewayAuthContext::CliCodex(handoff) => { + let expected_access_token = codex_access_token("acct_bundle_jwt"); + assert_eq!(handoff.account_id.as_deref(), Some("acct_bundle_explicit")); + assert_eq!( + handoff.access_token.expose_secret(), + expected_access_token.as_str() + ); + } + other => panic!("unexpected integrated auth context: {other:?}"), + } - let _home = EnvVarGuard::set("HOME", bad_home_path.display().to_string()); - let _account_id = EnvVarGuard::set( - SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCOUNT_ID, - "acct_env_explicit", - ); - let _access_token = EnvVarGuard::set( - SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCESS_TOKEN, - access_token, + assert!( + env::var(SUBSTRATE_LLM_AUTH_BUNDLE_FD).is_err(), + "pointer env should be consumed on first read" ); - let harness = build_codex_oauth_provider_harness( - &server.url(), - "codex-mini-latest", - CodexAuthSource::Integrated, - ); - let response = harness - .invoke_responses(HeaderMap::new(), responses_sync_request()) - .await; + let err = IntegratedGatewayAuthContext::from_launch_mode(GatewayMode::InWorld) + .expect_err("second read should fail after pointer env is consumed"); + assert!(err.to_string().contains("missing pointer env")); +} - let status = response.status(); - let body = response_text(response).await; - assert_eq!(status, StatusCode::OK, "{body}"); - assert!(body.contains("Integrated auth route proof")); - upstream.assert_async().await; +#[cfg(unix)] +#[tokio::test] +async fn integrated_gateway_startup_reads_openai_bundle_without_secret_env_fallback() { + let _guard = ENV_LOCK.lock().await; + let _openai = EnvVarGuard::unset("OPENAI_API_KEY"); + let _bundle = GatewayAuthBundleFdGuard::install(GatewayAuthBundleV1 { + schema_version: 1, + backend_id: GATEWAY_AUTH_BUNDLE_BACKEND_API_OPENAI.to_string(), + fields: std::collections::HashMap::from([( + SUBSTRATE_LLM_BACKEND_AUTH_API_OPENAI_API_KEY.to_string(), + "sk-openai-bundle".to_string(), + )]), + }); + + let context = IntegratedGatewayAuthContext::from_launch_mode(GatewayMode::InWorld) + .expect("read api:openai auth bundle") + .expect("in-world startup should produce integrated auth context"); + match context { + IntegratedGatewayAuthContext::ApiOpenAI { api_key } => { + assert_eq!(api_key, "sk-openai-bundle"); + } + other => panic!("unexpected integrated auth context: {other:?}"), + } + + assert!( + env::var("OPENAI_API_KEY").is_err(), + "integrated startup should not require OPENAI_API_KEY in process env" + ); + assert!( + env::var(SUBSTRATE_LLM_AUTH_BUNDLE_FD).is_err(), + "pointer env should be consumed on first read" + ); } #[tokio::test] diff --git a/crates/replay/src/replay/executor.rs b/crates/replay/src/replay/executor.rs index b83a5c503..6e00741d6 100644 --- a/crates/replay/src/replay/executor.rs +++ b/crates/replay/src/replay/executor.rs @@ -34,6 +34,7 @@ use std::process as stdprocess; use std::sync::OnceLock; #[cfg(target_os = "linux")] use world::{copydiff, overlayfs}; +use world_api::WorldReuseMode; const ANCHOR_MODE_ENV: &str = "SUBSTRATE_ANCHOR_MODE"; const ANCHOR_PATH_ENV: &str = "SUBSTRATE_ANCHOR_PATH"; @@ -141,8 +142,10 @@ fn build_agent_execute_request( agent_id: std::env::var("SUBSTRATE_AGENT_ID").unwrap_or_else(|_| "replay".to_string()), budget: None, policy_snapshot, + shared_world: None, world_network: Some(world_network), world_fs_mode: Some(substrate_broker::world_fs_mode()), + member_dispatch: None, }) } @@ -328,6 +331,7 @@ async fn try_world_backend( let start = Instant::now(); let spec = WorldSpec { reuse_session: true, + reuse_mode: WorldReuseMode::GenericCompatible, isolate_network: true, limits: ResourceLimits::default(), enable_preload: false, @@ -344,6 +348,8 @@ async fn try_world_backend( env: state.env.clone(), pty: false, span_id: Some(state.span_id.clone()), + shared_world: None, + member_dispatch: None, }; match backend.exec(&handle, req) { Ok(res) => { diff --git a/crates/replay/tests/planner_executor.rs b/crates/replay/tests/planner_executor.rs index 54ad46fd2..5b85684bd 100644 --- a/crates/replay/tests/planner_executor.rs +++ b/crates/replay/tests/planner_executor.rs @@ -84,10 +84,9 @@ fn latest_replay_strategy(trace_path: &PathBuf) -> Option { .ok()? .lines() .filter_map(|line| serde_json::from_str::(line).ok()) - .filter(|value| { + .rfind(|value| { value.get("event_type").and_then(|value| value.as_str()) == Some("replay_strategy") }) - .next_back() } #[tokio::test] diff --git a/crates/shell/Cargo.toml b/crates/shell/Cargo.toml index a4705c1a7..ae2d9c371 100644 --- a/crates/shell/Cargo.toml +++ b/crates/shell/Cargo.toml @@ -58,7 +58,7 @@ futures = "0.3" base64 = "0.22" url = "2" rand = "0.8" -agent_api = { package = "unified-agent-api", version = "=0.2.3", default-features = false, features = ["codex", "claude_code"] } +agent_api = { package = "unified-agent-api", version = "=0.3.5", default-features = false, features = ["codex", "claude_code"] } agent-api-client = { version = "0.2.7", path = "../agent-api-client" } agent-api-types = { version = "0.2.7", path = "../agent-api-types" } http-body-util = "0.1" diff --git a/crates/shell/src/execution/agent_dev_support.rs b/crates/shell/src/execution/agent_dev_support.rs new file mode 100644 index 000000000..3e772a26e --- /dev/null +++ b/crates/shell/src/execution/agent_dev_support.rs @@ -0,0 +1,24 @@ +use anyhow::Result; + +use crate::execution::agent_runtime::state_store::{ + AgentRuntimeStateStore, DurableInboxItemKind, DurableInboxItemRecord, +}; + +/// Persists one pending runtime-alert inbox item through the authoritative state-store path. +/// +/// This is reserved for test and validation support. It must not be surfaced through the public +/// `substrate agent ...` grammar. +pub fn persist_runtime_alert_for_dev_support( + orchestration_session_id: &str, + item_id: &str, + message: Option, +) -> Result<()> { + let store = AgentRuntimeStateStore::new()?; + let item = DurableInboxItemRecord::new( + orchestration_session_id, + item_id, + DurableInboxItemKind::RuntimeAlert, + message, + ); + store.persist_inbox_item(&item) +} diff --git a/crates/shell/src/execution/agent_events.rs b/crates/shell/src/execution/agent_events.rs index 41acb4de7..c047ebbc2 100644 --- a/crates/shell/src/execution/agent_events.rs +++ b/crates/shell/src/execution/agent_events.rs @@ -9,7 +9,6 @@ use uuid::Uuid; /// Global sender storage so any component can publish agent events. static AGENT_EVENT_SENDER: OnceLock>>> = OnceLock::new(); -static ORCHESTRATION_SESSION_ID: OnceLock = OnceLock::new(); #[cfg(test)] static EVENT_TEST_GUARD: OnceLock> = OnceLock::new(); @@ -63,13 +62,73 @@ pub(crate) fn clear_agent_event_sender() { } } -pub(crate) fn orchestration_session_id() -> String { - ORCHESTRATION_SESSION_ID - .get_or_init(|| Uuid::now_v7().to_string()) - .clone() +#[derive(Clone, Debug, PartialEq, Eq)] +pub(crate) struct ShellEventEmissionContext { + pub(crate) orchestration_session_id: String, + pub(crate) agent_id: String, + pub(crate) role: Option, + pub(crate) backend_id: Option, + pub(crate) participant_id: Option, + pub(crate) parent_participant_id: Option, + pub(crate) resumed_from_participant_id: Option, + pub(crate) world_id: Option, + pub(crate) world_generation: Option, } -pub(crate) fn publish_command_completion(command: &str, cmd_id: &str, status: &ExitStatus) { +#[derive(Clone, Debug, PartialEq, Eq)] +pub(crate) struct ShellCommandEventContext { + pub(crate) emission: ShellEventEmissionContext, + pub(crate) cmd_id: String, + pub(crate) run_id: Option, + pub(crate) span_id: Option, +} + +impl ShellCommandEventContext { + pub(crate) fn new( + emission: ShellEventEmissionContext, + cmd_id: impl Into, + run_id: Option, + span_id: Option, + ) -> Self { + Self { + emission, + cmd_id: cmd_id.into(), + run_id, + span_id, + } + } +} + +fn build_shell_message_event( + context: &ShellCommandEventContext, + kind: MessageEventKind, + message: String, +) -> Option { + let run_id = context.run_id.as_ref()?; + let mut event = AgentEvent::message( + context.emission.agent_id.clone(), + context.emission.orchestration_session_id.clone(), + run_id.clone(), + kind, + message, + ); + event.cmd_id = Some(context.cmd_id.clone()); + event.span_id = context.span_id.clone(); + event.role = context.emission.role.clone(); + event.backend_id = context.emission.backend_id.clone(); + event.participant_id = context.emission.participant_id.clone(); + event.parent_participant_id = context.emission.parent_participant_id.clone(); + event.resumed_from_participant_id = context.emission.resumed_from_participant_id.clone(); + event.world_id = context.emission.world_id.clone(); + event.world_generation = context.emission.world_generation; + Some(event) +} + +pub(crate) fn publish_command_completion( + context: Option<&ShellCommandEventContext>, + command: &str, + status: &ExitStatus, +) { #[cfg(unix)] { use std::os::unix::process::ExitStatusExt; @@ -85,28 +144,31 @@ pub(crate) fn publish_command_completion(command: &str, cmd_id: &str, status: &E if !enabled { return; } - - let mut event = AgentEvent::message( - "shell", - orchestration_session_id(), - cmd_id.to_string(), - MessageEventKind::TaskEnd, - format!("Command `{command}` completed successfully"), - ); - event.cmd_id = Some(cmd_id.to_string()); + let Some(event) = context.and_then(|context| { + build_shell_message_event( + context, + MessageEventKind::TaskEnd, + format!("Command `{command}` completed successfully"), + ) + }) else { + return; + }; let _ = publish_agent_event(event); return; } - let code = status.code().unwrap_or(-1); - let mut event = AgentEvent::message( - "shell", - orchestration_session_id(), - cmd_id.to_string(), - MessageEventKind::TaskEnd, - format!("Command `{command}` exited with status {code}"), - ); - event.cmd_id = Some(cmd_id.to_string()); + let Some(event) = context.and_then(|context| { + build_shell_message_event( + context, + MessageEventKind::TaskEnd, + format!( + "Command `{command}` exited with status {}", + status.code().unwrap_or(-1) + ), + ) + }) else { + return; + }; let _ = publish_agent_event(event); } @@ -151,7 +213,7 @@ pub(crate) fn schedule_demo_events() { return; } - let orchestration_session_id = orchestration_session_id(); + let orchestration_session_id = Uuid::now_v7().to_string(); let run_id = Uuid::now_v7().to_string(); let mut first = AgentEvent::message( @@ -198,7 +260,7 @@ pub(crate) fn schedule_demo_burst(agent_count: usize, events_per_agent: usize, d let agent_count = agent_count.clamp(1, 16); let events_per_agent = events_per_agent.clamp(1, 10_000); - let orchestration_session_id = orchestration_session_id(); + let orchestration_session_id = Uuid::now_v7().to_string(); let run_id = Uuid::now_v7().to_string(); for agent_idx in 0..agent_count { @@ -261,6 +323,7 @@ mod tests { } #[test] + #[serial_test::serial] fn schedule_demo_burst_emits_expected_events() { let _guard = super::acquire_event_test_guard(); let rt = Runtime::new().expect("runtime"); @@ -293,6 +356,7 @@ mod tests { } #[test] + #[serial_test::serial] fn publish_command_completion_failure_emits_task_end_with_cmd_id() { let _guard = super::acquire_event_test_guard(); let rt = Runtime::new().expect("runtime"); @@ -300,12 +364,36 @@ mod tests { let mut rx = init_event_channel(); let cmd_id = "cmd-failure"; let status = exit_status_from_code(7); + let context = ShellCommandEventContext::new( + ShellEventEmissionContext { + orchestration_session_id: "orch-live".to_string(), + agent_id: "shell".to_string(), + role: Some("orchestrator".to_string()), + backend_id: Some("shell:repl".to_string()), + participant_id: Some("participant-1".to_string()), + parent_participant_id: None, + resumed_from_participant_id: None, + world_id: Some("world-1".to_string()), + world_generation: Some(4), + }, + cmd_id, + Some(cmd_id.to_string()), + Some("spn-1".to_string()), + ); - publish_command_completion("false", cmd_id, &status); + publish_command_completion(Some(&context), "false", &status); let event = rx.recv().await.expect("event"); assert_eq!(event.kind, AgentEventKind::TaskEnd); assert_eq!(event.cmd_id.as_deref(), Some(cmd_id)); + assert_eq!(event.orchestration_session_id, "orch-live"); + assert_eq!(event.run_id, cmd_id); + assert_eq!(event.span_id.as_deref(), Some("spn-1")); + assert_eq!(event.role.as_deref(), Some("orchestrator")); + assert_eq!(event.backend_id.as_deref(), Some("shell:repl")); + assert_eq!(event.participant_id.as_deref(), Some("participant-1")); + assert_eq!(event.world_id.as_deref(), Some("world-1")); + assert_eq!(event.world_generation, Some(4)); assert_eq!( event .data @@ -323,6 +411,26 @@ mod tests { } #[test] + #[serial_test::serial] + fn publish_command_completion_none_emits_no_agent_event() { + let _guard = super::acquire_event_test_guard(); + let rt = Runtime::new().expect("runtime"); + rt.block_on(async { + let mut rx = init_event_channel(); + let status = exit_status_from_code(7); + + publish_command_completion(None, "false", &status); + + assert!( + rx.try_recv().is_err(), + "no-context command completion must not emit an agent event" + ); + }); + clear_agent_event_sender(); + } + + #[test] + #[serial_test::serial] fn publish_command_completion_success_emits_task_end_when_enabled() { let _guard = super::acquire_event_test_guard(); let rt = Runtime::new().expect("runtime"); @@ -331,12 +439,29 @@ mod tests { let cmd_id = "cmd-success"; let status = exit_status_from_code(0); std::env::set_var("SUBSTRATE_COMMAND_SUCCESS_EVENTS", "1"); + let context = ShellCommandEventContext::new( + ShellEventEmissionContext { + orchestration_session_id: "orch-live".to_string(), + agent_id: "shell".to_string(), + role: Some("orchestrator".to_string()), + backend_id: Some("shell:repl".to_string()), + participant_id: Some("participant-1".to_string()), + parent_participant_id: None, + resumed_from_participant_id: None, + world_id: None, + world_generation: None, + }, + cmd_id, + Some(cmd_id.to_string()), + None, + ); - publish_command_completion("true", cmd_id, &status); + publish_command_completion(Some(&context), "true", &status); let event = rx.recv().await.expect("event"); assert_eq!(event.kind, AgentEventKind::TaskEnd); assert_eq!(event.cmd_id.as_deref(), Some(cmd_id)); + assert_eq!(event.orchestration_session_id, "orch-live"); assert_eq!( event .data diff --git a/crates/shell/src/execution/agent_runtime/control.rs b/crates/shell/src/execution/agent_runtime/control.rs new file mode 100644 index 000000000..d5e6ce6f2 --- /dev/null +++ b/crates/shell/src/execution/agent_runtime/control.rs @@ -0,0 +1,2923 @@ +use std::fs; +#[cfg(unix)] +use std::io::Write; +use std::io::{self, Read}; +#[cfg(unix)] +use std::os::unix::net::UnixListener as StdUnixListener; +use std::path::{Path, PathBuf}; +use std::process::ExitStatus; +use std::sync::atomic::{AtomicBool, Ordering}; +use std::sync::{Arc, Mutex}; +use std::thread; +use std::time::Duration; + +#[cfg(target_os = "linux")] +use agent_api_types::{ExecuteStreamFrame, MemberTurnSubmitRequestV1}; +use anyhow::{Context, Result}; +#[cfg(target_os = "linux")] +use base64::engine::general_purpose::STANDARD as BASE64; +#[cfg(target_os = "linux")] +use base64::Engine; +use chrono::Utc; +use futures::StreamExt; +use serde::{Deserialize, Serialize}; +#[cfg(unix)] +use tokio::io::{AsyncBufReadExt, AsyncWriteExt, BufReader}; +#[cfg(unix)] +use tokio::net::{UnixListener, UnixStream}; +use tokio::sync::{mpsc, oneshot}; +use uuid::Uuid; + +#[cfg(unix)] +use crate::execution::agent_events::format_event_line; +use crate::execution::agent_runtime::orchestration_session::StartupPromptStreamState; +#[cfg(target_os = "linux")] +use crate::execution::build_agent_client_and_pending_diff_request; +use crate::execution::config_model::AgentExecutionScope; + +use super::{ + mapping::AgentRuntimeBackendKind, session::AgentRuntimeSessionManifest, + validator::RuntimeSelectionDescriptor, AgentRuntimeSessionState, AgentRuntimeStateStore, + OrchestrationSessionRecord, OrchestrationSessionState, ORCHESTRATOR_ROLE, PURE_AGENT_PROTOCOL, +}; +use substrate_common::agent_events::{AgentEvent, MessageEventKind}; + +pub(crate) const AGENT_API_SESSION_RESUME_V1: &str = "agent_api.session.resume.v1"; +pub(crate) const AGENT_API_TURN_LIFECYCLE_V1: &str = "agent_api.turn.lifecycle.v1"; +pub(crate) const HIDDEN_OWNER_HELPER_SUBCOMMAND: &str = "__owner-helper"; +const OWNER_HELPER_READY_TIMEOUT_ERROR_PREFIX: &str = + "timed out waiting for authoritative owner-helper readiness for orchestration session "; +const OWNER_HELPER_READY_TIMEOUT: Duration = Duration::from_secs(30); +const OWNER_HELPER_READY_POLL_INTERVAL: Duration = Duration::from_millis(100); +#[cfg(unix)] +const PRIVATE_STOP_UNIX_PATH_MAX: usize = 100; +#[cfg(unix)] +const PRIVATE_PROMPT_READY_TIMEOUT: Duration = Duration::from_secs(10); +#[cfg(unix)] +const PRIVATE_PROMPT_READY_POLL_INTERVAL: Duration = Duration::from_millis(100); +#[cfg(unix)] +const STARTUP_PROMPT_STREAM_ACCEPT_TIMEOUT: Duration = Duration::from_secs(10); + +#[allow(dead_code)] +#[derive(Clone, Copy, Debug, Deserialize, Eq, PartialEq, Serialize)] +#[serde(rename_all = "snake_case")] +pub(crate) enum OwnerHelperMode { + Start, + Resume, + ResumeOneTurn, + Fork, +} + +#[allow(dead_code)] +impl OwnerHelperMode { + pub(crate) fn as_str(self) -> &'static str { + match self { + Self::Start => "start", + Self::Resume => "resume", + Self::ResumeOneTurn => "resume_one_turn", + Self::Fork => "fork", + } + } +} + +#[derive(Clone, Copy, Debug, Deserialize, Eq, PartialEq, Serialize)] +#[serde(rename_all = "snake_case")] +pub(crate) enum PublicPromptAction { + Start, + Turn, +} + +impl PublicPromptAction { + #[cfg(unix)] + pub(crate) fn as_str(self) -> &'static str { + match self { + Self::Start => "start", + Self::Turn => "turn", + } + } +} + +#[derive(Clone, Copy, Debug, Deserialize, Eq, PartialEq, Serialize)] +#[serde(rename_all = "snake_case")] +pub(crate) enum PublicSessionPosture { + Active, + DetachedReattachable, + Terminal, +} + +impl PublicSessionPosture { + #[cfg(unix)] + pub(crate) fn as_str(self) -> &'static str { + match self { + Self::Active => "active", + Self::DetachedReattachable => "detached_reattachable", + Self::Terminal => "terminal", + } + } +} + +#[derive(Clone, Debug, Default, Eq, PartialEq)] +pub(crate) struct PublicPromptInput { + pub prompt: Option, + pub prompt_file: Option, +} + +#[derive(Clone, Debug, Eq, PartialEq)] +pub(crate) struct LoadedPublicPrompt { + pub prompt_text: String, +} + +#[derive(Clone, Debug, Eq, PartialEq)] +pub(crate) struct PublicPromptCommandRequest { + pub action: PublicPromptAction, + pub orchestration_session_id: Option, + pub backend_id: String, + pub prompt: LoadedPublicPrompt, + pub json: bool, +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub(crate) struct PersistedWorldBinding { + pub world_id: String, + pub world_generation: u64, +} + +#[derive(Clone)] +pub(crate) struct PromptSubmitRuntime { + pub descriptor: RuntimeSelectionDescriptor, + pub orchestration_session: Arc>, + pub manifest: Arc>, + pub store: AgentRuntimeStateStore, + pub uaa_session_handle_id: String, + pub park_after_turn_tx: Option>, +} + +#[cfg_attr(not(target_os = "linux"), allow(dead_code))] +#[derive(Clone, Debug, PartialEq)] +pub(crate) enum SubmittedPromptStreamEvent { + Agent(Box), + Stdout(String), + Stderr(String), +} + +#[derive(Clone, Debug, Eq, PartialEq)] +pub(crate) struct SubmittedPromptCompletion { + pub exit_code: i32, + pub warning: Option, +} + +#[allow(dead_code)] +#[derive(Clone, Copy, Debug, Deserialize, Eq, PartialEq, Serialize)] +#[serde(rename_all = "snake_case")] +pub(crate) enum ResolvedRuntimeBackendKind { + Codex, + ClaudeCode, +} + +impl From for ResolvedRuntimeBackendKind { + fn from(value: AgentRuntimeBackendKind) -> Self { + match value { + AgentRuntimeBackendKind::Codex => Self::Codex, + AgentRuntimeBackendKind::ClaudeCode => Self::ClaudeCode, + } + } +} + +impl TryFrom for AgentRuntimeBackendKind { + type Error = anyhow::Error; + + fn try_from(value: ResolvedRuntimeBackendKind) -> Result { + Ok(match value { + ResolvedRuntimeBackendKind::Codex => AgentRuntimeBackendKind::Codex, + ResolvedRuntimeBackendKind::ClaudeCode => AgentRuntimeBackendKind::ClaudeCode, + }) + } +} + +#[allow(dead_code)] +#[derive(Clone, Debug, Deserialize, Eq, PartialEq, Serialize)] +pub(crate) struct ResolvedRuntimeDescriptor { + pub agent_id: String, + pub backend_id: String, + pub backend_kind: ResolvedRuntimeBackendKind, + pub protocol: String, + pub execution_scope: AgentExecutionScope, + pub binary_path: String, +} + +impl From<&RuntimeSelectionDescriptor> for ResolvedRuntimeDescriptor { + fn from(value: &RuntimeSelectionDescriptor) -> Self { + Self { + agent_id: value.agent_id.clone(), + backend_id: value.backend_id.clone(), + backend_kind: value.backend_kind.into(), + protocol: value.protocol.clone(), + execution_scope: value.execution_scope, + binary_path: value.binary_path.display().to_string(), + } + } +} + +impl TryFrom<&ResolvedRuntimeDescriptor> for RuntimeSelectionDescriptor { + type Error = anyhow::Error; + + fn try_from(value: &ResolvedRuntimeDescriptor) -> Result { + Ok(Self { + agent_id: value.agent_id.clone(), + backend_id: value.backend_id.clone(), + backend_kind: value.backend_kind.try_into()?, + protocol: value.protocol.clone(), + execution_scope: value.execution_scope, + binary_path: PathBuf::from(&value.binary_path), + }) + } +} + +#[allow(dead_code)] +#[derive(Clone, Debug, Deserialize, Eq, PartialEq, Serialize)] +pub(crate) struct HiddenOwnerHelperSessionPlan { + pub orchestration_session_id: String, + pub shell_trace_session_id: String, + pub workspace_root: String, + pub world_id: Option, + pub world_generation: Option, +} + +#[allow(dead_code)] +#[derive(Clone, Debug, Deserialize, Eq, PartialEq, Serialize)] +pub(crate) struct HiddenOwnerHelperParticipantPlan { + pub participant_id: String, + pub lease_token: String, + pub run_id: String, + pub resumed_from_participant_id: Option, + pub internal_uaa_session_id: Option, +} + +#[allow(dead_code)] +#[derive(Clone, Debug, Deserialize, Eq, PartialEq, Serialize)] +pub(crate) struct HiddenOwnerHelperStartupPromptPlan { + pub prompt_text: String, + pub stream_path: PathBuf, +} + +#[allow(dead_code)] +#[derive(Clone, Debug, Deserialize, Eq, PartialEq, Serialize)] +pub(crate) struct HiddenOwnerHelperLaunchPlan { + pub mode: OwnerHelperMode, + pub descriptor: ResolvedRuntimeDescriptor, + pub session: HiddenOwnerHelperSessionPlan, + pub participant: HiddenOwnerHelperParticipantPlan, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub startup_prompt: Option, + pub source_orchestration_session_id: Option, +} + +#[allow(dead_code)] +impl HiddenOwnerHelperLaunchPlan { + pub(crate) fn orchestration_session_id(&self) -> &str { + &self.session.orchestration_session_id + } + + pub(crate) fn participant_id(&self) -> &str { + &self.participant.participant_id + } + + pub(crate) fn requires_internal_session_id(&self) -> bool { + matches!( + self.mode, + OwnerHelperMode::Resume | OwnerHelperMode::ResumeOneTurn | OwnerHelperMode::Fork + ) + } +} + +#[derive(Clone, Copy, Debug, Deserialize, Eq, PartialEq, Serialize)] +#[serde(rename_all = "snake_case")] +pub(crate) enum PrivateStopOutcome { + Accepted, + AlreadyTerminal, + OwnerUnreachable, + ProtocolError, +} + +#[derive(Debug)] +pub(crate) struct PrivateStopRequest { + pub response_tx: oneshot::Sender, +} + +pub(crate) type PrivateStopRequestReceiver = mpsc::UnboundedReceiver; +pub(crate) type PrivateStopRequestSender = mpsc::UnboundedSender; + +#[derive(Debug)] +pub(crate) struct PrivateStopTransport { + shutdown_tx: Option>, + task: Option>, + path: PathBuf, +} + +#[derive(Debug)] +pub(crate) struct PrivatePromptRequest { + pub action: PublicPromptAction, + pub prompt: String, + pub envelope_tx: mpsc::UnboundedSender, +} + +pub(crate) type PrivatePromptRequestReceiver = mpsc::UnboundedReceiver; +pub(crate) type PrivatePromptRequestSender = mpsc::UnboundedSender; + +#[derive(Debug)] +pub(crate) struct PrivatePromptTransport { + shutdown_tx: Option>, + task: Option>, + path: PathBuf, +} + +#[cfg(unix)] +pub(crate) struct StartupPromptTransportListener { + listener: StdUnixListener, + path: PathBuf, +} + +#[cfg(unix)] +impl StartupPromptTransportListener { + pub(crate) fn path(&self) -> &Path { + &self.path + } + + async fn remove_path(path: &Path) { + let _ = tokio::fs::remove_file(path).await; + } +} + +impl PrivatePromptTransport { + #[allow(dead_code)] + pub(crate) fn path(&self) -> &Path { + &self.path + } + + pub(crate) async fn close(&mut self) { + if let Some(shutdown_tx) = self.shutdown_tx.take() { + let _ = shutdown_tx.send(()); + } + if let Some(task) = self.task.take() { + let _ = task.await; + } + #[cfg(unix)] + { + let _ = tokio::fs::remove_file(&self.path).await; + } + } +} + +#[derive(Clone, Debug, Deserialize, Serialize)] +#[serde(tag = "kind", rename_all = "snake_case")] +pub(crate) enum PublicPromptEnvelope { + Accepted { + version: u8, + action: PublicPromptAction, + orchestration_session_id: String, + backend_id: String, + #[serde(skip_serializing_if = "Option::is_none")] + participant_id: Option, + scope: String, + }, + Event { + version: u8, + event_kind: String, + data: serde_json::Value, + }, + Warning { + version: u8, + message: String, + }, + Completed { + version: u8, + action: PublicPromptAction, + orchestration_session_id: String, + backend_id: String, + #[serde(skip_serializing_if = "Option::is_none")] + participant_id: Option, + turn_outcome: String, + session_posture: PublicSessionPosture, + state: String, + warnings: Vec, + }, + Failed { + version: u8, + terminal: bool, + stage: String, + error_code: String, + message: String, + }, +} + +impl PrivateStopTransport { + #[allow(dead_code)] + pub(crate) fn path(&self) -> &Path { + &self.path + } + + pub(crate) async fn close(&mut self) { + if let Some(shutdown_tx) = self.shutdown_tx.take() { + let _ = shutdown_tx.send(()); + } + if let Some(task) = self.task.take() { + let _ = task.await; + } + #[cfg(unix)] + { + let _ = tokio::fs::remove_file(&self.path).await; + } + } +} + +pub(crate) fn build_session_resume_extension(session_id: &str) -> serde_json::Value { + serde_json::json!({ + "selector": "id", + "id": session_id, + }) +} + +#[allow(dead_code)] +pub(crate) fn hidden_owner_helper_plan_path( + store: &AgentRuntimeStateStore, + orchestration_session_id: &str, + participant_id: &str, +) -> PathBuf { + let session_fragment = compact_stop_transport_fragment(orchestration_session_id); + let participant_fragment = compact_stop_transport_fragment(participant_id); + store + .handles_dir() + .join("owner-helper") + .join(format!("{session_fragment}-{participant_fragment}.json")) +} + +#[cfg(unix)] +pub(crate) fn hidden_owner_helper_startup_prompt_stream_path( + store: &AgentRuntimeStateStore, + orchestration_session_id: &str, + participant_id: &str, +) -> PathBuf { + let session_fragment = compact_stop_transport_fragment(orchestration_session_id); + let participant_fragment = compact_stop_transport_fragment(participant_id); + let socket_name = format!("{session_fragment}-{participant_fragment}.startup.sock"); + let preferred = store.handles_dir().join("startup").join(&socket_name); + if preferred.as_os_str().len() > PRIVATE_STOP_UNIX_PATH_MAX { + return PathBuf::from("/tmp") + .join("substrate-agent-hub-startup") + .join(socket_name); + } + preferred +} + +#[cfg(unix)] +pub(crate) fn register_hidden_owner_helper_startup_prompt_listener( + store: &AgentRuntimeStateStore, + orchestration_session_id: &str, + participant_id: &str, +) -> Result { + let path = hidden_owner_helper_startup_prompt_stream_path( + store, + orchestration_session_id, + participant_id, + ); + let parent = path.parent().ok_or_else(|| { + anyhow::anyhow!( + "startup prompt transport path '{}' is missing a parent directory", + path.display() + ) + })?; + fs::create_dir_all(parent).with_context(|| format!("failed to create {}", parent.display()))?; + remove_existing_stop_transport_path(&path)?; + let listener = StdUnixListener::bind(&path) + .with_context(|| format!("failed to bind startup prompt transport {}", path.display()))?; + listener.set_nonblocking(true).with_context(|| { + format!( + "failed to configure startup prompt transport {}", + path.display() + ) + })?; + Ok(StartupPromptTransportListener { listener, path }) +} + +#[cfg(unix)] +pub(crate) async fn consume_hidden_owner_helper_startup_prompt_stream( + listener: StartupPromptTransportListener, + mut on_envelope: F, +) -> Result +where + F: FnMut(&PublicPromptEnvelope) -> Result<()>, +{ + let StartupPromptTransportListener { listener, path } = listener; + let tokio_listener = tokio::net::UnixListener::from_std(listener).with_context(|| { + format!( + "failed to activate startup prompt transport {}", + path.display() + ) + })?; + let accept = + tokio::time::timeout(STARTUP_PROMPT_STREAM_ACCEPT_TIMEOUT, tokio_listener.accept()) + .await + .map_err(|_| { + anyhow::anyhow!( + "stream_bridge_failed: timed out waiting for hidden owner-helper startup prompt stream {}", + path.display() + ) + })?; + let (stream, _) = accept.with_context(|| { + format!( + "stream_bridge_failed: failed to accept hidden owner-helper startup prompt stream {}", + path.display() + ) + })?; + + let mut reader = BufReader::new(stream); + let mut line = String::new(); + let mut saw_accept = false; + loop { + line.clear(); + let bytes_read = reader.read_line(&mut line).await?; + if bytes_read == 0 { + break; + } + let envelope: PublicPromptEnvelope = + serde_json::from_str(line.trim()).with_context(|| { + format!( + "failed to decode hidden owner-helper startup prompt envelope from {}", + path.display() + ) + })?; + if matches!(envelope, PublicPromptEnvelope::Accepted { .. }) { + saw_accept = true; + } + on_envelope(&envelope)?; + match envelope { + PublicPromptEnvelope::Completed { turn_outcome, .. } => { + StartupPromptTransportListener::remove_path(&path).await; + return Ok(completed_exit_code(turn_outcome.as_str())); + } + PublicPromptEnvelope::Failed { message, .. } => { + StartupPromptTransportListener::remove_path(&path).await; + return Err(anyhow::anyhow!(message)); + } + _ => {} + } + } + StartupPromptTransportListener::remove_path(&path).await; + + if saw_accept { + anyhow::bail!("owner_unreachable: startup prompt stream ended after accepting the request"); + } + anyhow::bail!( + "owner_unreachable: hidden owner-helper startup prompt stream ended before accepting the request" + ); +} + +#[cfg(unix)] +pub(crate) fn run_hidden_owner_helper_startup_prompt_stream( + listener: StartupPromptTransportListener, + json: bool, +) -> Result<()> { + let rt = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .context("failed to initialize startup prompt transport runtime")?; + let mut renderer = PublicPromptRenderer::new(json); + let result = rt.block_on(async { + consume_hidden_owner_helper_startup_prompt_stream(listener, |envelope| { + renderer.render(envelope) + }) + .await + }); + + match result { + Ok(0) => Ok(()), + Ok(code) => Err(anyhow::Error::new(PublicPromptRenderedExit { + exit_code: code, + })), + Err(err) => Err(err), + } +} + +#[allow(dead_code)] +pub(crate) fn persist_hidden_owner_helper_launch_plan( + store: &AgentRuntimeStateStore, + plan: &HiddenOwnerHelperLaunchPlan, +) -> Result { + let path = hidden_owner_helper_plan_path( + store, + plan.orchestration_session_id(), + plan.participant_id(), + ); + let parent = path.parent().ok_or_else(|| { + anyhow::anyhow!( + "hidden owner-helper launch plan path '{}' is missing a parent directory", + path.display() + ) + })?; + fs::create_dir_all(parent).with_context(|| format!("failed to create {}", parent.display()))?; + fs::write(&path, serde_json::to_vec_pretty(plan)?).with_context(|| { + format!( + "failed to write hidden owner-helper launch plan {}", + path.display() + ) + })?; + Ok(path) +} + +#[allow(dead_code)] +pub(crate) fn load_hidden_owner_helper_launch_plan( + path: &Path, +) -> Result { + let bytes = fs::read(path).with_context(|| { + format!( + "failed to read hidden owner-helper launch plan {}", + path.display() + ) + })?; + serde_json::from_slice(&bytes).with_context(|| { + format!( + "failed to decode hidden owner-helper launch plan {}", + path.display() + ) + }) +} + +#[allow(dead_code)] +pub(crate) fn remove_hidden_owner_helper_launch_plan(path: &Path) -> Result<()> { + match fs::remove_file(path) { + Ok(()) => Ok(()), + Err(err) if err.kind() == std::io::ErrorKind::NotFound => Ok(()), + Err(err) => Err(err).with_context(|| { + format!( + "failed to remove hidden owner-helper launch plan {}", + path.display() + ) + }), + } +} + +#[allow(dead_code)] +pub(crate) fn wait_for_hidden_owner_helper_readiness( + store: &AgentRuntimeStateStore, + plan: &HiddenOwnerHelperLaunchPlan, +) -> Result<()> { + let started_at = std::time::Instant::now(); + loop { + let readiness = store.classify_hidden_owner_helper_launch_readiness( + plan.orchestration_session_id(), + plan.participant_id(), + plan.requires_internal_session_id(), + )?; + let startup_prompt_ready = if plan.mode == OwnerHelperMode::Start { + start_launch_startup_prompt_is_accepted_or_terminal(store, plan)? + } else { + true + }; + if startup_prompt_ready + && (readiness == super::state_store::HiddenOwnerHelperLaunchReadiness::ReadyAttached + || (plan.mode == OwnerHelperMode::Start + && matches!( + readiness, + super::state_store::HiddenOwnerHelperLaunchReadiness::ReadyDetached(_) + ))) + { + return Ok(()); + } + if started_at.elapsed() >= OWNER_HELPER_READY_TIMEOUT { + anyhow::bail!( + "{}{}", + OWNER_HELPER_READY_TIMEOUT_ERROR_PREFIX, + plan.orchestration_session_id(), + ); + } + thread::sleep(OWNER_HELPER_READY_POLL_INTERVAL); + } +} + +pub(crate) fn hidden_owner_helper_readiness_timed_out(err: &anyhow::Error) -> bool { + err.to_string() + .starts_with(OWNER_HELPER_READY_TIMEOUT_ERROR_PREFIX) +} + +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub(crate) enum HiddenOwnerHelperStartTimeoutReconciliation { + Success, + FailureMarkedTerminal, + FailureUnchanged, +} + +pub(crate) fn reconcile_hidden_owner_helper_start_timeout( + store: &AgentRuntimeStateStore, + plan: &HiddenOwnerHelperLaunchPlan, +) -> Result { + if plan.mode != OwnerHelperMode::Start { + return Ok(HiddenOwnerHelperStartTimeoutReconciliation::FailureUnchanged); + } + + if start_launch_startup_prompt_is_terminal(store, plan)? { + match store.classify_hidden_owner_helper_launch_readiness( + plan.orchestration_session_id(), + plan.participant_id(), + plan.requires_internal_session_id(), + )? { + super::state_store::HiddenOwnerHelperLaunchReadiness::ReadyAttached + | super::state_store::HiddenOwnerHelperLaunchReadiness::ReadyDetached(_) => { + return Ok(HiddenOwnerHelperStartTimeoutReconciliation::Success); + } + super::state_store::HiddenOwnerHelperLaunchReadiness::Pending => {} + } + } + + let Some(session) = store.load_orchestration_session(plan.orchestration_session_id())? else { + return Ok(HiddenOwnerHelperStartTimeoutReconciliation::FailureUnchanged); + }; + let Some(participant) = store.load_participant(plan.participant_id())? else { + return Ok(HiddenOwnerHelperStartTimeoutReconciliation::FailureUnchanged); + }; + + if startup_prompt_is_terminal_for_participant(&session, plan.participant_id()) { + if let Some((next_session, next_participant)) = build_detached_start_reconciliation( + &session, + &participant, + plan.requires_internal_session_id(), + ) { + if next_session != session { + store.persist_orchestration_session(&next_session)?; + } + if next_participant != participant { + store.persist_participant(&next_participant)?; + } + + match store.classify_hidden_owner_helper_launch_readiness( + plan.orchestration_session_id(), + plan.participant_id(), + plan.requires_internal_session_id(), + )? { + super::state_store::HiddenOwnerHelperLaunchReadiness::ReadyAttached + | super::state_store::HiddenOwnerHelperLaunchReadiness::ReadyDetached(_) => { + return Ok(HiddenOwnerHelperStartTimeoutReconciliation::Success); + } + super::state_store::HiddenOwnerHelperLaunchReadiness::Pending => {} + } + } + } + + if should_mark_start_timeout_terminal_failure(&session, &participant, plan.participant_id()) { + let reason = format!( + "hidden owner-helper readiness timed out during start for orchestration session {}", + plan.orchestration_session_id(), + ); + let mut failed_session = session; + let mut failed_participant = participant; + persist_start_timeout_terminal_failure( + store, + &mut failed_session, + &mut failed_participant, + &reason, + )?; + return Ok(HiddenOwnerHelperStartTimeoutReconciliation::FailureMarkedTerminal); + } + + Ok(HiddenOwnerHelperStartTimeoutReconciliation::FailureUnchanged) +} + +fn start_launch_startup_prompt_is_accepted_or_terminal( + store: &AgentRuntimeStateStore, + plan: &HiddenOwnerHelperLaunchPlan, +) -> Result { + if plan.startup_prompt.is_none() { + return Ok(true); + } + + Ok(matches!( + store + .startup_prompt_replay_state(plan.orchestration_session_id(), plan.participant_id(),)?, + super::StartupPromptReplayState::AcceptedOrTerminal + )) +} + +fn start_launch_startup_prompt_is_terminal( + store: &AgentRuntimeStateStore, + plan: &HiddenOwnerHelperLaunchPlan, +) -> Result { + if plan.startup_prompt.is_none() { + return Ok(true); + } + + let Some(session) = store.load_orchestration_session(plan.orchestration_session_id())? else { + return Ok(false); + }; + Ok(startup_prompt_is_terminal_for_participant( + &session, + plan.participant_id(), + )) +} + +fn startup_prompt_is_terminal_for_participant( + session: &OrchestrationSessionRecord, + participant_id: &str, +) -> bool { + matches!( + session.startup_prompt.as_ref(), + Some(startup_prompt) + if startup_prompt.participant_id == participant_id + && matches!( + startup_prompt.state, + StartupPromptStreamState::Completed | StartupPromptStreamState::Failed + ) + ) +} + +fn build_detached_start_reconciliation( + session: &OrchestrationSessionRecord, + participant: &AgentRuntimeSessionManifest, + require_internal_session_id: bool, +) -> Option<(OrchestrationSessionRecord, AgentRuntimeSessionManifest)> { + if session.state != OrchestrationSessionState::Active { + return None; + } + if session.active_participant_id() != Some(participant.participant_id()) { + return None; + } + if !participant.matches_public_parent_linkage(session) + || !participant.is_host_orchestrator() + || !participant.handle.state.is_live() + { + return None; + } + + let mut detached_session = session.clone(); + let mut detached_participant = participant.clone(); + detached_session.shell_owner_pid = 0; + detached_participant.release_runtime_ownership(); + detached_participant.mark_client_detached("owner detached cleanly"); + detached_participant.internal.shell_owner_pid = 0; + detached_participant.touch_heartbeat(); + detached_session.transition_state(OrchestrationSessionState::Active); + if detached_session.pending_inbox_count > 0 { + detached_session.mark_awaiting_attention(); + } else { + detached_session.mark_parked_resumable("owner detached cleanly"); + } + + super::state_store::valid_detached_host_continuity_posture( + &detached_session, + &detached_participant, + require_internal_session_id, + )?; + Some((detached_session, detached_participant)) +} + +fn should_mark_start_timeout_terminal_failure( + session: &OrchestrationSessionRecord, + participant: &AgentRuntimeSessionManifest, + participant_id: &str, +) -> bool { + if session.state.is_terminal() + || !participant.handle.state.is_live() + || participant.participant_id() != participant_id + || !participant.matches_public_parent_linkage(session) + { + return false; + } + + session.attached_participant_id() == Some(participant_id) + || participant.attached_client_present() + || matches!( + session.startup_prompt.as_ref(), + Some(startup_prompt) if startup_prompt.participant_id == participant_id + ) +} + +fn persist_start_timeout_terminal_failure( + store: &AgentRuntimeStateStore, + session: &mut OrchestrationSessionRecord, + participant: &mut AgentRuntimeSessionManifest, + reason: &str, +) -> Result<()> { + session.mark_startup_prompt_failed(participant.participant_id(), reason.to_string()); + if participant.handle.state.is_live() { + let next_state = if participant.internal_uaa_session_id().is_some() + || participant.handle.state != AgentRuntimeSessionState::Allocating + { + AgentRuntimeSessionState::Invalidated + } else { + AgentRuntimeSessionState::Failed + }; + participant.transition_state(next_state); + } + participant.mark_terminal_state(reason.to_string()); + participant.internal.last_error_bucket = Some("bootstrap_run".to_string()); + participant.internal.last_error_message = Some(reason.to_string()); + participant.touch_heartbeat(); + if !session.state.is_terminal() { + session.transition_state(OrchestrationSessionState::Failed); + } + session.mark_terminal(reason.to_string()); + persist_runtime_snapshots(store, session, participant) +} + +pub(crate) fn private_stop_request_channel( +) -> (PrivateStopRequestSender, PrivateStopRequestReceiver) { + mpsc::unbounded_channel() +} + +pub(crate) fn runtime_is_terminal(manifest: &Arc>) -> bool { + let state = manifest + .lock() + .expect("runtime manifest mutex poisoned") + .handle + .state + .clone(); + !state.is_live() +} + +pub(crate) fn runtime_stop_transport_ids( + manifest: &Arc>, +) -> (String, String) { + let manifest = manifest.lock().expect("runtime manifest mutex poisoned"); + ( + manifest.handle.orchestration_session_id.clone(), + manifest.handle.participant_id.clone(), + ) +} + +pub(crate) fn runtime_controls_parent_session(role: &str) -> bool { + role == ORCHESTRATOR_ROLE +} + +pub(crate) fn mark_orchestration_session_failed( + store: &AgentRuntimeStateStore, + orchestration_session: &Arc>, + message: impl Into, +) { + let message = message.into(); + let snapshot = { + let mut guard = orchestration_session + .lock() + .expect("orchestration session mutex poisoned"); + guard.transition_state(OrchestrationSessionState::Failed); + guard.mark_terminal(message); + guard.clone() + }; + let _ = store.persist_orchestration_session(&snapshot); +} + +pub(crate) fn persist_runtime_snapshots( + store: &AgentRuntimeStateStore, + orchestration_session: &OrchestrationSessionRecord, + manifest: &AgentRuntimeSessionManifest, +) -> Result<()> { + store.persist_orchestration_session(orchestration_session)?; + store.persist_participant(manifest) +} + +pub(crate) fn mark_runtime_startup_failed( + store: &AgentRuntimeStateStore, + orchestration_session: &Arc>, + manifest: &Arc>, + message: &str, +) { + let (orchestration_snapshot, manifest_snapshot) = { + let mut manifest_guard = manifest.lock().expect("runtime manifest mutex poisoned"); + let controls_parent_session = runtime_controls_parent_session(&manifest_guard.handle.role); + if manifest_guard.handle.state == AgentRuntimeSessionState::Allocating { + manifest_guard.transition_state(AgentRuntimeSessionState::Failed); + } + if !manifest_guard.has_valid_ownership() { + manifest_guard.mark_terminal_state(message.to_string()); + } + manifest_guard.internal.last_error_bucket = Some("bootstrap_run".to_string()); + manifest_guard.internal.last_error_message = Some(message.to_string()); + let orchestration_snapshot = { + let mut orchestration_guard = orchestration_session + .lock() + .expect("orchestration session mutex poisoned"); + if controls_parent_session { + orchestration_guard.transition_state(OrchestrationSessionState::Failed); + orchestration_guard.mark_terminal(message.to_string()); + } else { + orchestration_guard.touch_active(); + } + orchestration_guard.clone() + }; + (orchestration_snapshot, manifest_guard.clone()) + }; + let _ = persist_runtime_snapshots(store, &orchestration_snapshot, &manifest_snapshot); +} + +pub(crate) fn persist_world_binding_authority( + store: &AgentRuntimeStateStore, + orchestration_session: &Arc>, + world_binding: Option<&PersistedWorldBinding>, +) -> Result { + let snapshot = { + let mut guard = orchestration_session + .lock() + .expect("orchestration session mutex poisoned"); + match world_binding { + Some(binding) => { + store.set_orchestration_session_world_binding( + &mut guard, + binding.world_id.clone(), + binding.world_generation, + )?; + } + None => { + store.clear_orchestration_session_world_binding(&mut guard)?; + } + }; + guard.clone() + }; + Ok(snapshot) +} + +pub(crate) fn invalidate_stale_world_members_after_binding( + store: &AgentRuntimeStateStore, + orchestration_session_id: &str, + active_generation: u64, +) -> Result> { + store.invalidate_stale_world_members_for_session(orchestration_session_id, active_generation) +} + +pub(crate) fn note_runtime_stop_requested( + store: &AgentRuntimeStateStore, + orchestration_session: &Arc>, + manifest: &Arc>, +) -> Result<()> { + let (orchestration_snapshot, manifest_snapshot) = { + let mut orchestration_guard = orchestration_session + .lock() + .expect("orchestration session mutex poisoned"); + let mut manifest_guard = manifest.lock().expect("runtime manifest mutex poisoned"); + let controls_parent_session = runtime_controls_parent_session(&manifest_guard.handle.role); + if manifest_guard.handle.state.is_live() { + manifest_guard.transition_state(AgentRuntimeSessionState::Stopping); + manifest_guard.touch_heartbeat(); + } + if controls_parent_session && orchestration_guard.state == OrchestrationSessionState::Active + { + orchestration_guard.transition_state(OrchestrationSessionState::Stopping); + } else if orchestration_guard.state == OrchestrationSessionState::Active { + orchestration_guard.touch_active(); + } + (orchestration_guard.clone(), manifest_guard.clone()) + }; + persist_runtime_snapshots(store, &orchestration_snapshot, &manifest_snapshot) +} + +pub(crate) fn apply_runtime_stop_closeout( + orchestration_session: &mut OrchestrationSessionRecord, + manifest: &mut AgentRuntimeSessionManifest, +) { + manifest.transition_state(AgentRuntimeSessionState::Stopped); + manifest.mark_terminal_state("stopped"); + manifest.touch_heartbeat(); + if runtime_controls_parent_session(&manifest.handle.role) { + orchestration_session.transition_state(OrchestrationSessionState::Stopped); + orchestration_session.mark_terminal("stopped"); + } else { + orchestration_session.touch_active(); + } +} + +pub(crate) fn persist_runtime_stop_closeout( + store: &AgentRuntimeStateStore, + orchestration_session: &mut OrchestrationSessionRecord, + manifest: &mut AgentRuntimeSessionManifest, +) -> Result<()> { + apply_runtime_stop_closeout(orchestration_session, manifest); + persist_runtime_snapshots(store, orchestration_session, manifest) +} + +#[cfg(unix)] +pub(crate) fn private_stop_transport_path( + store: &AgentRuntimeStateStore, + orchestration_session_id: &str, + participant_id: &str, +) -> PathBuf { + let session_fragment = compact_stop_transport_fragment(orchestration_session_id); + let participant_fragment = compact_stop_transport_fragment(participant_id); + let socket_name = format!("{session_fragment}-{participant_fragment}.sock"); + let preferred = store.handles_dir().join("stop").join(&socket_name); + #[cfg(unix)] + if preferred.as_os_str().len() > PRIVATE_STOP_UNIX_PATH_MAX { + return PathBuf::from("/tmp") + .join("substrate-agent-hub-stop") + .join(socket_name); + } + preferred +} + +fn compact_stop_transport_fragment(id: &str) -> String { + let normalized = id + .chars() + .filter(|ch| ch.is_ascii_alphanumeric()) + .collect::(); + if normalized.len() <= 12 { + return normalized; + } + + format!( + "{}{}", + &normalized[..6], + &normalized[normalized.len() - 6..] + ) +} + +#[cfg(unix)] +pub(crate) async fn register_private_stop_transport( + store: &AgentRuntimeStateStore, + orchestration_session_id: &str, + participant_id: &str, + stop_tx: PrivateStopRequestSender, +) -> Result { + let path = private_stop_transport_path(store, orchestration_session_id, participant_id); + let parent = path.parent().ok_or_else(|| { + anyhow::anyhow!( + "private stop transport path '{}' is missing a parent directory", + path.display() + ) + })?; + fs::create_dir_all(parent).with_context(|| format!("failed to create {}", parent.display()))?; + remove_existing_stop_transport_path(&path)?; + let listener = UnixListener::bind(&path) + .with_context(|| format!("failed to bind private stop transport {}", path.display()))?; + let (shutdown_tx, mut shutdown_rx) = oneshot::channel(); + let path_for_task = path.clone(); + let task = tokio::spawn(async move { + loop { + tokio::select! { + _ = &mut shutdown_rx => break, + accept = listener.accept() => { + let Ok((stream, _)) = accept else { + break; + }; + let stop_tx = stop_tx.clone(); + tokio::spawn(async move { + let _ = handle_private_stop_connection(stream, stop_tx).await; + }); + } + } + } + let _ = tokio::fs::remove_file(&path_for_task).await; + }); + Ok(PrivateStopTransport { + shutdown_tx: Some(shutdown_tx), + task: Some(task), + path, + }) +} + +#[cfg(not(unix))] +pub(crate) async fn register_private_stop_transport( + _store: &AgentRuntimeStateStore, + _orchestration_session_id: &str, + _participant_id: &str, + _stop_tx: PrivateStopRequestSender, +) -> Result { + Ok(PrivateStopTransport { + shutdown_tx: None, + task: None, + path: PathBuf::new(), + }) +} + +pub(crate) fn spawn_local_private_stop_owner( + store: AgentRuntimeStateStore, + orchestration_session: Arc>, + manifest: Arc>, + shutdown_requested: Arc, + cancel: agent_api::AgentWrapperCancelHandle, + mut stop_rx: PrivateStopRequestReceiver, +) -> tokio::task::JoinHandle<()> { + tokio::spawn(async move { + while let Some(request) = stop_rx.recv().await { + let outcome = if runtime_is_terminal(&manifest) { + PrivateStopOutcome::AlreadyTerminal + } else { + shutdown_requested.store(true, Ordering::SeqCst); + let _ = note_runtime_stop_requested(&store, &orchestration_session, &manifest); + cancel.cancel(); + PrivateStopOutcome::Accepted + }; + let _ = request.response_tx.send(outcome); + } + }) +} + +pub(crate) fn private_prompt_request_channel( +) -> (PrivatePromptRequestSender, PrivatePromptRequestReceiver) { + mpsc::unbounded_channel() +} + +#[cfg(unix)] +pub(crate) fn private_prompt_transport_path( + store: &AgentRuntimeStateStore, + orchestration_session_id: &str, + participant_id: &str, +) -> PathBuf { + let session_fragment = compact_stop_transport_fragment(orchestration_session_id); + let participant_fragment = compact_stop_transport_fragment(participant_id); + let socket_name = format!("{session_fragment}-{participant_fragment}.prompt.sock"); + let preferred = store.handles_dir().join("prompt").join(&socket_name); + #[cfg(unix)] + if preferred.as_os_str().len() > PRIVATE_STOP_UNIX_PATH_MAX { + return PathBuf::from("/tmp") + .join("substrate-agent-hub-prompt") + .join(socket_name); + } + preferred +} + +pub(crate) fn prompt_runtime_from_parts( + descriptor: RuntimeSelectionDescriptor, + orchestration_session: Arc>, + manifest: Arc>, + store: AgentRuntimeStateStore, + uaa_session_handle_id: String, + park_after_turn_tx: Option>, +) -> PromptSubmitRuntime { + PromptSubmitRuntime { + descriptor, + orchestration_session, + manifest, + store, + uaa_session_handle_id, + park_after_turn_tx, + } +} + +pub(crate) async fn submit_host_prompt_turn( + runtime: &PromptSubmitRuntime, + run_id: &str, + prompt: &str, + mut on_event: F, +) -> Result +where + F: FnMut(SubmittedPromptStreamEvent), +{ + let gateway = super::build_gateway_for_descriptor(&runtime.descriptor) + .context("build host targeted-turn gateway")?; + let agent_kind = + agent_api::AgentWrapperKind::new(runtime.descriptor.backend_kind.as_agent_kind_str()) + .map_err(|err| anyhow::anyhow!("substrate: error: {err}"))?; + + let request = agent_api::AgentWrapperRunRequest { + prompt: prompt.to_string(), + working_dir: Some(std::env::current_dir().unwrap_or_else(|_| PathBuf::from("."))), + timeout: None, + env: std::collections::BTreeMap::new(), + extensions: std::collections::BTreeMap::from([( + AGENT_API_SESSION_RESUME_V1.to_string(), + build_session_resume_extension(&runtime.uaa_session_handle_id), + )]), + }; + let control = gateway + .run_control(&agent_kind, request) + .await + .map_err(|err| anyhow::anyhow!("substrate: error: {}", err))?; + let agent_api::AgentWrapperRunControl { handle, cancel: _ } = control; + let agent_api::AgentWrapperRunHandle { + mut events, + completion, + } = handle; + + while let Some(wrapper_event) = events.next().await { + let (orchestration_snapshot, manifest_snapshot, event) = { + let mut orchestration_guard = runtime + .orchestration_session + .lock() + .expect("orchestration session mutex poisoned"); + let mut manifest_guard = runtime + .manifest + .lock() + .expect("runtime manifest mutex poisoned"); + if let Some(session_id) = extract_session_handle_id(wrapper_event.data.as_ref()) { + if manifest_guard.internal.uaa_session_id.as_deref() != Some(session_id) { + manifest_guard.set_uaa_session_id(session_id.to_string()); + } + } + manifest_guard.touch_event(Utc::now()); + orchestration_guard.touch_active(); + let event = translate_prompt_wrapper_event( + &manifest_guard, + &orchestration_guard, + run_id, + wrapper_event, + ); + (orchestration_guard.clone(), manifest_guard.clone(), event) + }; + persist_runtime_snapshots(&runtime.store, &orchestration_snapshot, &manifest_snapshot)?; + on_event(SubmittedPromptStreamEvent::Agent(Box::new(event))); + } + + let completion = completion + .await + .map_err(|err| anyhow::anyhow!("substrate: error: {}", err))?; + if let Some(session_id) = extract_session_handle_id(completion.data.as_ref()) { + let mut manifest_guard = runtime + .manifest + .lock() + .expect("runtime manifest mutex poisoned"); + if manifest_guard.internal.uaa_session_id.as_deref() != Some(session_id) { + manifest_guard.set_uaa_session_id(session_id.to_string()); + } + } + Ok(SubmittedPromptCompletion { + exit_code: completion.status.code().unwrap_or(-1), + warning: warning_for_exit_status(&completion.status), + }) +} + +#[cfg(target_os = "linux")] +pub(crate) async fn submit_world_prompt_turn( + runtime: &PromptSubmitRuntime, + run_id: &str, + prompt: &str, + mut on_event: F, +) -> Result +where + F: FnMut(SubmittedPromptStreamEvent), +{ + use http_body_util::BodyExt as _; + + let request = { + let manifest_guard = runtime + .manifest + .lock() + .expect("runtime manifest mutex poisoned"); + MemberTurnSubmitRequestV1 { + schema_version: 1, + orchestration_session_id: manifest_guard.handle.orchestration_session_id.clone(), + participant_id: manifest_guard.handle.participant_id.clone(), + orchestrator_participant_id: manifest_guard + .handle + .orchestrator_participant_id + .clone() + .ok_or_else(|| { + anyhow::anyhow!( + "substrate: error: retained world-scoped member is missing orchestrator_participant_id" + ) + })?, + backend_id: runtime.descriptor.backend_id.clone(), + run_id: run_id.to_string(), + world_id: manifest_guard.handle.world_id.clone().ok_or_else(|| { + anyhow::anyhow!( + "substrate: error: retained world-scoped member is missing world_id" + ) + })?, + world_generation: manifest_guard.handle.world_generation.ok_or_else(|| { + anyhow::anyhow!( + "substrate: error: retained world-scoped member is missing world_generation" + ) + })?, + prompt: prompt.to_string(), + } + }; + + let (client, _pending_diff_request, _agent_id) = build_agent_client_and_pending_diff_request()?; + let response = client + .submit_member_turn_stream(request) + .await + .map_err(|err| anyhow::anyhow!("substrate: error: {err:#}"))?; + + let mut body = std::pin::pin!(response.into_body()); + let mut buffer = Vec::new(); + let mut observed_exit: Option = None; + while let Some(frame) = body.as_mut().frame().await { + let frame = frame.map_err(|err| anyhow::anyhow!("substrate: error: {err:#}"))?; + let Some(data) = frame.data_ref() else { + continue; + }; + buffer.extend_from_slice(data); + + while let Some(pos) = buffer.iter().position(|&byte| byte == b'\n') { + let line: Vec = buffer.drain(..=pos).collect(); + if line.len() <= 1 { + continue; + } + let payload = &line[..line.len() - 1]; + if payload.is_empty() { + continue; + } + let frame = serde_json::from_slice::(payload) + .map_err(|err| anyhow::anyhow!("substrate: error: {err:#}"))?; + match frame { + ExecuteStreamFrame::Start { .. } => {} + ExecuteStreamFrame::Event { event } => { + on_event(SubmittedPromptStreamEvent::Agent(Box::new(event))); + } + ExecuteStreamFrame::Stdout { chunk_b64 } => { + let decoded = BASE64 + .decode(chunk_b64.as_bytes()) + .map_err(|err| anyhow::anyhow!("substrate: error: {err:#}"))?; + on_event(SubmittedPromptStreamEvent::Stdout( + String::from_utf8_lossy(&decoded).to_string(), + )); + } + ExecuteStreamFrame::Stderr { chunk_b64 } => { + let decoded = BASE64 + .decode(chunk_b64.as_bytes()) + .map_err(|err| anyhow::anyhow!("substrate: error: {err:#}"))?; + on_event(SubmittedPromptStreamEvent::Stderr( + String::from_utf8_lossy(&decoded).to_string(), + )); + } + ExecuteStreamFrame::Exit { exit, .. } => { + observed_exit = Some(exit); + } + ExecuteStreamFrame::Error { message } => { + return Err(anyhow::anyhow!("substrate: error: {message}")); + } + } + } + } + + let exit_code = observed_exit.unwrap_or(0); + Ok(SubmittedPromptCompletion { + exit_code, + warning: warning_for_exit_code(exit_code), + }) +} + +#[cfg(not(target_os = "linux"))] +pub(crate) async fn submit_world_prompt_turn( + _runtime: &PromptSubmitRuntime, + _run_id: &str, + _prompt: &str, + _on_event: F, +) -> Result +where + F: FnMut(SubmittedPromptStreamEvent), +{ + Err(anyhow::anyhow!( + "substrate: error: world-targeted follow-up turns are supported on Linux only" + )) +} + +pub(crate) fn spawn_local_private_prompt_owner( + runtime: PromptSubmitRuntime, + mut prompt_rx: PrivatePromptRequestReceiver, +) -> tokio::task::JoinHandle<()> { + tokio::spawn(async move { + while let Some(request) = prompt_rx.recv().await { + let _ = stream_private_prompt_request(&runtime, request, false).await; + } + }) +} + +#[cfg(target_os = "linux")] +pub(crate) fn spawn_remote_private_prompt_owner( + runtime: PromptSubmitRuntime, + mut prompt_rx: PrivatePromptRequestReceiver, +) -> tokio::task::JoinHandle<()> { + tokio::spawn(async move { + while let Some(request) = prompt_rx.recv().await { + let _ = stream_private_prompt_request(&runtime, request, true).await; + } + }) +} + +#[cfg(not(target_os = "linux"))] +#[allow(dead_code)] +pub(crate) fn spawn_remote_private_prompt_owner( + _runtime: PromptSubmitRuntime, + mut prompt_rx: PrivatePromptRequestReceiver, +) -> tokio::task::JoinHandle<()> { + tokio::spawn(async move { while prompt_rx.recv().await.is_some() {} }) +} + +#[allow(dead_code)] +pub(crate) fn load_public_prompt_source(input: &PublicPromptInput) -> Result { + let raw = match (&input.prompt, &input.prompt_file) { + (Some(_), Some(_)) => anyhow::bail!( + "malformed_prompt_source: provide exactly one of --prompt or --prompt-file" + ), + (Some(prompt), None) => prompt.clone(), + (None, Some(path)) if path == Path::new("-") => { + let mut buffer = String::new(); + io::stdin() + .read_to_string(&mut buffer) + .context("malformed_prompt_source: failed to read prompt from stdin")?; + buffer + } + (None, Some(path)) => fs::read_to_string(path) + .with_context(|| format!("malformed_prompt_source: failed to read {}", path.display()))?, + (None, None) => anyhow::bail!( + "missing_prompt_source: provide --prompt or --prompt-file (use --prompt-file - for stdin)" + ), + }; + + let normalized = raw.trim_end_matches(['\r', '\n']).to_string(); + if normalized.trim().is_empty() { + anyhow::bail!("empty_prompt: prompt input was empty"); + } + Ok(LoadedPublicPrompt { + prompt_text: normalized, + }) +} + +#[allow(dead_code)] +pub(crate) fn run_public_prompt_command( + request: PublicPromptCommandRequest, + _cli_world: bool, + _cli_no_world: bool, +) -> Result<()> { + let store = AgentRuntimeStateStore::new()?; + let (orchestration_session_id, backend_id) = validate_public_prompt_command_request(&request)?; + let participant_id = match request.action { + PublicPromptAction::Start => { + resolve_public_start_prompt_target(&store, orchestration_session_id, backend_id)? + } + PublicPromptAction::Turn => { + let resolved = + store.resolve_public_turn_target(orchestration_session_id, backend_id)?; + if resolved.session_posture != PublicSessionPosture::Active { + anyhow::bail!( + "owner_unreachable: orchestration session {} backend {} is not currently attached to a live retained turn target", + orchestration_session_id, + backend_id + ); + } + resolved.participant.handle.participant_id.clone() + } + }; + + #[cfg(not(unix))] + { + let _ = participant_id; + anyhow::bail!( + "unsupported_platform_or_posture: public prompt submission requires a Unix private owner transport" + ); + } + + #[cfg(unix)] + { + let transport_path = private_prompt_transport_path( + &store, + orchestration_session_id, + participant_id.as_str(), + ); + let rt = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .context("failed to initialize prompt transport runtime")?; + let mut renderer = PublicPromptRenderer::new(request.json); + let mut stream_started = false; + let mut saw_terminal = false; + let result = rt.block_on(async { + wait_for_private_prompt_transport(&transport_path).await?; + request_private_prompt_stream( + &transport_path, + request.action, + &request.prompt.prompt_text, + |envelope| { + stream_started = true; + if matches!( + envelope, + PublicPromptEnvelope::Completed { .. } + | PublicPromptEnvelope::Failed { .. } + ) { + saw_terminal = true; + } + renderer.render(envelope) + }, + ) + .await + }); + + match result { + Ok(0) => Ok(()), + Ok(code) => Err(anyhow::Error::new(PublicPromptRenderedExit { + exit_code: code, + })), + Err(err) if stream_started => { + if !saw_terminal { + renderer.render(&failed_prompt_envelope( + "bridge", + "owner_unreachable", + err.to_string(), + ))?; + } + Err(anyhow::Error::new(PublicPromptRenderedExit { + exit_code: 1, + })) + } + Err(err) => Err(err), + } + } +} + +fn validate_public_prompt_command_request( + request: &PublicPromptCommandRequest, +) -> Result<(&str, &str)> { + let session_error = match request.action { + PublicPromptAction::Start => { + "runtime_start_failed: public start actions require an orchestration session id" + } + PublicPromptAction::Turn => { + "unknown_session: public turn actions require --session " + } + }; + let backend_error = match request.action { + PublicPromptAction::Start => { + "runtime_start_failed: public start actions require an exact backend id" + } + PublicPromptAction::Turn => { + "missing_backend: public turn actions require --backend " + } + }; + + let orchestration_session_id = request + .orchestration_session_id + .as_deref() + .filter(|value| !value.trim().is_empty()) + .ok_or_else(|| anyhow::anyhow!(session_error))?; + let backend_id = (!request.backend_id.trim().is_empty()) + .then_some(request.backend_id.as_str()) + .ok_or_else(|| anyhow::anyhow!(backend_error))?; + + Ok((orchestration_session_id, backend_id)) +} + +#[cfg(unix)] +#[allow(dead_code)] +pub(crate) async fn request_private_stop(path: &Path) -> Result { + let mut stream = UnixStream::connect(path).await.with_context(|| { + format!( + "failed to connect to private stop transport {}", + path.display() + ) + })?; + let request = serde_json::json!({ + "version": 1, + "action": "stop", + }); + stream + .write_all(serde_json::to_string(&request)?.as_bytes()) + .await?; + stream.write_all(b"\n").await?; + stream.flush().await?; + + let mut reader = BufReader::new(stream); + let mut line = String::new(); + reader.read_line(&mut line).await?; + let response: PrivateStopResponse = serde_json::from_str(line.trim()).with_context(|| { + format!( + "failed to decode private stop transport response from {}", + path.display() + ) + })?; + Ok(response.outcome) +} + +#[cfg(unix)] +pub(crate) async fn register_private_prompt_transport( + store: &AgentRuntimeStateStore, + orchestration_session_id: &str, + participant_id: &str, + prompt_tx: PrivatePromptRequestSender, +) -> Result { + let path = private_prompt_transport_path(store, orchestration_session_id, participant_id); + let parent = path.parent().ok_or_else(|| { + anyhow::anyhow!( + "private prompt transport path '{}' is missing a parent directory", + path.display() + ) + })?; + fs::create_dir_all(parent).with_context(|| format!("failed to create {}", parent.display()))?; + remove_existing_stop_transport_path(&path)?; + let listener = UnixListener::bind(&path) + .with_context(|| format!("failed to bind private prompt transport {}", path.display()))?; + let (shutdown_tx, mut shutdown_rx) = oneshot::channel(); + let path_for_task = path.clone(); + let task = tokio::spawn(async move { + loop { + tokio::select! { + _ = &mut shutdown_rx => break, + accept = listener.accept() => { + let Ok((stream, _)) = accept else { + break; + }; + let prompt_tx = prompt_tx.clone(); + tokio::spawn(async move { + let _ = handle_private_prompt_connection(stream, prompt_tx).await; + }); + } + } + } + let _ = tokio::fs::remove_file(&path_for_task).await; + }); + Ok(PrivatePromptTransport { + shutdown_tx: Some(shutdown_tx), + task: Some(task), + path, + }) +} + +#[cfg(not(unix))] +pub(crate) async fn register_private_prompt_transport( + _store: &AgentRuntimeStateStore, + _orchestration_session_id: &str, + _participant_id: &str, + _prompt_tx: PrivatePromptRequestSender, +) -> Result { + Ok(PrivatePromptTransport { + shutdown_tx: None, + task: None, + path: PathBuf::new(), + }) +} + +#[cfg(unix)] +fn remove_existing_stop_transport_path(path: &Path) -> Result<()> { + match fs::symlink_metadata(path) { + Ok(_) => fs::remove_file(path).with_context(|| { + format!( + "failed to remove stale private stop transport {}", + path.display() + ) + }), + Err(err) if err.kind() == io::ErrorKind::NotFound => Ok(()), + Err(err) => Err(err).with_context(|| { + format!( + "failed to inspect existing private stop transport {}", + path.display() + ) + }), + } +} + +#[cfg(unix)] +async fn handle_private_stop_connection( + stream: UnixStream, + stop_tx: PrivateStopRequestSender, +) -> Result<()> { + let mut reader = BufReader::new(stream); + let mut line = String::new(); + let bytes_read = reader.read_line(&mut line).await?; + let outcome = if bytes_read == 0 || parse_private_stop_request(line.trim()).is_err() { + PrivateStopOutcome::ProtocolError + } else { + let (response_tx, response_rx) = oneshot::channel(); + if stop_tx.send(PrivateStopRequest { response_tx }).is_err() { + PrivateStopOutcome::OwnerUnreachable + } else { + match tokio::time::timeout(Duration::from_secs(5), response_rx).await { + Ok(Ok(outcome)) => outcome, + Ok(Err(_)) | Err(_) => PrivateStopOutcome::OwnerUnreachable, + } + } + }; + + let response = PrivateStopResponse { + version: 1, + outcome, + }; + let mut stream = reader.into_inner(); + stream + .write_all(serde_json::to_string(&response)?.as_bytes()) + .await?; + stream.write_all(b"\n").await?; + stream.flush().await?; + Ok(()) +} + +#[cfg(unix)] +fn parse_private_stop_request(payload: &str) -> Result { + let request: PrivateStopRequestV1 = + serde_json::from_str(payload).context("failed to decode private stop request")?; + if request.version != 1 || request.action != "stop" { + anyhow::bail!("unsupported private stop request"); + } + Ok(request) +} + +#[cfg(unix)] +#[derive(Debug, Deserialize)] +#[serde(deny_unknown_fields)] +struct PrivateStopRequestV1 { + version: u8, + action: String, +} + +#[cfg(unix)] +#[derive(Debug, Deserialize, Serialize)] +struct PrivateStopResponse { + version: u8, + outcome: PrivateStopOutcome, +} + +#[cfg(unix)] +#[derive(Debug, Deserialize, Serialize)] +struct PrivatePromptRequestV1 { + version: u8, + action: PublicPromptAction, + prompt: String, +} + +#[cfg(unix)] +async fn request_private_prompt_stream( + path: &Path, + action: PublicPromptAction, + prompt: &str, + mut on_envelope: F, +) -> Result +where + F: FnMut(&PublicPromptEnvelope) -> Result<()>, +{ + let mut stream = UnixStream::connect(path).await.with_context(|| { + format!( + "failed to connect to private prompt transport {}", + path.display() + ) + })?; + let request = PrivatePromptRequestV1 { + version: 1, + action, + prompt: prompt.to_string(), + }; + stream + .write_all(serde_json::to_string(&request)?.as_bytes()) + .await?; + stream.write_all(b"\n").await?; + stream.flush().await?; + + let mut reader = BufReader::new(stream); + let mut line = String::new(); + let mut saw_accept = false; + loop { + line.clear(); + let bytes_read = reader.read_line(&mut line).await?; + if bytes_read == 0 { + break; + } + let envelope: PublicPromptEnvelope = + serde_json::from_str(line.trim()).with_context(|| { + format!( + "failed to decode private prompt transport response from {}", + path.display() + ) + })?; + if matches!(envelope, PublicPromptEnvelope::Accepted { .. }) { + saw_accept = true; + } + on_envelope(&envelope)?; + match envelope { + PublicPromptEnvelope::Completed { turn_outcome, .. } => { + return Ok(completed_exit_code(turn_outcome.as_str())); + } + PublicPromptEnvelope::Failed { message, .. } => return Err(anyhow::anyhow!(message)), + _ => {} + } + } + + if saw_accept { + anyhow::bail!("owner_unreachable: prompt stream ended before terminal envelope"); + } + anyhow::bail!("owner_unreachable: prompt owner closed the stream before accepting the request"); +} + +#[cfg(unix)] +async fn handle_private_prompt_connection( + stream: UnixStream, + prompt_tx: PrivatePromptRequestSender, +) -> Result<()> { + let mut reader = BufReader::new(stream); + let mut line = String::new(); + let bytes_read = reader.read_line(&mut line).await?; + let request = if bytes_read == 0 { + None + } else { + Some(parse_private_prompt_request(line.trim())?) + }; + + let mut stream = reader.into_inner(); + let Some(request) = request else { + let failed = failed_prompt_envelope( + "bridge", + "malformed_prompt_source", + "empty private prompt request", + ); + stream + .write_all(serde_json::to_string(&failed)?.as_bytes()) + .await?; + stream.write_all(b"\n").await?; + stream.flush().await?; + return Ok(()); + }; + + let (envelope_tx, mut envelope_rx) = mpsc::unbounded_channel(); + if prompt_tx + .send(PrivatePromptRequest { + action: request.action, + prompt: request.prompt, + envelope_tx, + }) + .is_err() + { + let failed = failed_prompt_envelope( + "bridge", + "owner_unreachable", + "private prompt owner is no longer available", + ); + stream + .write_all(serde_json::to_string(&failed)?.as_bytes()) + .await?; + stream.write_all(b"\n").await?; + stream.flush().await?; + return Ok(()); + } + + let mut saw_accept = false; + let mut saw_terminal = false; + while let Some(envelope) = envelope_rx.recv().await { + if matches!(envelope, PublicPromptEnvelope::Accepted { .. }) { + saw_accept = true; + } + stream + .write_all(serde_json::to_string(&envelope)?.as_bytes()) + .await?; + stream.write_all(b"\n").await?; + stream.flush().await?; + if matches!( + envelope, + PublicPromptEnvelope::Completed { .. } | PublicPromptEnvelope::Failed { .. } + ) { + saw_terminal = true; + break; + } + } + if saw_accept && !saw_terminal { + let failed = failed_prompt_envelope( + "bridge", + "owner_unreachable", + "private prompt owner closed after accepting the request", + ); + stream + .write_all(serde_json::to_string(&failed)?.as_bytes()) + .await?; + stream.write_all(b"\n").await?; + stream.flush().await?; + } + Ok(()) +} + +#[cfg(unix)] +fn parse_private_prompt_request(payload: &str) -> Result { + let request: PrivatePromptRequestV1 = + serde_json::from_str(payload).context("failed to decode private prompt request")?; + if request.version != 1 { + anyhow::bail!("unsupported private prompt request"); + } + if request.prompt.trim().is_empty() { + anyhow::bail!("private prompt request requires a non-empty prompt"); + } + Ok(request) +} + +async fn stream_private_prompt_request( + runtime: &PromptSubmitRuntime, + request: PrivatePromptRequest, + world_scoped: bool, +) -> Result<()> { + if runtime_is_terminal(&runtime.manifest) { + let _ = request.envelope_tx.send(failed_prompt_envelope( + "runtime", + "owner_unreachable", + "runtime is no longer authoritative-live", + )); + return Ok(()); + } + + let manifest_snapshot = runtime + .manifest + .lock() + .expect("runtime manifest mutex poisoned") + .clone(); + let run_id = Uuid::now_v7().to_string(); + let accepted = PublicPromptEnvelope::Accepted { + version: 1, + action: request.action, + orchestration_session_id: manifest_snapshot.handle.orchestration_session_id.clone(), + backend_id: runtime.descriptor.backend_id.clone(), + participant_id: Some(manifest_snapshot.handle.participant_id.clone()), + scope: scope_label(runtime.descriptor.execution_scope).to_string(), + }; + if request.envelope_tx.send(accepted).is_err() { + return Ok(()); + } + + let submit_result = if world_scoped { + submit_world_prompt_turn(runtime, &run_id, &request.prompt, |event| { + let _ = request + .envelope_tx + .send(event_to_public_prompt_envelope(event)); + }) + .await + } else { + submit_host_prompt_turn(runtime, &run_id, &request.prompt, |event| { + let _ = request + .envelope_tx + .send(event_to_public_prompt_envelope(event)); + }) + .await + }; + + match submit_result { + Ok(completion) => { + let mut warnings = Vec::new(); + if let Some(message) = completion.warning { + warnings.push(message.clone()); + let _ = request.envelope_tx.send(PublicPromptEnvelope::Warning { + version: 1, + message, + }); + } + let (session_posture, state) = prompt_completion_session_state(runtime); + let _ = request.envelope_tx.send(PublicPromptEnvelope::Completed { + version: 1, + action: request.action, + orchestration_session_id: manifest_snapshot.handle.orchestration_session_id, + backend_id: runtime.descriptor.backend_id.clone(), + participant_id: Some(manifest_snapshot.handle.participant_id), + turn_outcome: turn_outcome_label(completion.exit_code).to_string(), + session_posture, + state, + warnings, + }); + request_helper_park_after_turn(runtime, request.action); + } + Err(err) => { + let _ = request.envelope_tx.send(failed_prompt_envelope( + "runtime", + "owner_unreachable", + err.to_string(), + )); + request_helper_park_after_turn(runtime, request.action); + } + } + Ok(()) +} + +fn request_helper_park_after_turn(runtime: &PromptSubmitRuntime, action: PublicPromptAction) { + if action != PublicPromptAction::Turn { + return; + } + if let Some(park_after_turn_tx) = runtime.park_after_turn_tx.as_ref() { + let _ = park_after_turn_tx.send(()); + } +} + +fn event_to_public_prompt_envelope(event: SubmittedPromptStreamEvent) -> PublicPromptEnvelope { + let (event_kind, data) = match event { + SubmittedPromptStreamEvent::Agent(event) => ( + "message".to_string(), + serde_json::to_value(event).unwrap_or_default(), + ), + SubmittedPromptStreamEvent::Stdout(text) => { + ("message".to_string(), serde_json::json!({ "text": text })) + } + SubmittedPromptStreamEvent::Stderr(text) => { + ("stderr".to_string(), serde_json::json!({ "text": text })) + } + }; + PublicPromptEnvelope::Event { + version: 1, + event_kind, + data, + } +} + +fn scope_label(scope: AgentExecutionScope) -> &'static str { + match scope { + AgentExecutionScope::Host => "host", + AgentExecutionScope::World => "world", + } +} + +#[cfg(target_os = "linux")] +fn warning_for_exit_code(exit_code: i32) -> Option { + (exit_code != 0).then(|| format!("Command failed with status: {exit_code}")) +} + +fn turn_outcome_label(exit_code: i32) -> &'static str { + match exit_code { + 0 => "success", + 130 => "cancelled", + _ => "nonzero_exit", + } +} + +#[cfg(unix)] +fn completed_exit_code(turn_outcome: &str) -> i32 { + match turn_outcome { + "success" => 0, + "cancelled" => 130, + _ => 1, + } +} + +fn prompt_completion_session_state( + runtime: &PromptSubmitRuntime, +) -> (PublicSessionPosture, String) { + let session = runtime + .orchestration_session + .lock() + .expect("orchestration session mutex poisoned") + .clone(); + let manifest = runtime + .manifest + .lock() + .expect("runtime manifest mutex poisoned") + .clone(); + + let posture = + if !manifest.handle.state.is_live() || session.state != OrchestrationSessionState::Active { + PublicSessionPosture::Terminal + } else { + PublicSessionPosture::Active + }; + ( + posture, + runtime_state_label(&manifest.handle.state).to_string(), + ) +} + +fn runtime_state_label(state: &AgentRuntimeSessionState) -> &'static str { + match state { + AgentRuntimeSessionState::Allocating + | AgentRuntimeSessionState::Ready + | AgentRuntimeSessionState::Running + | AgentRuntimeSessionState::Restarting + | AgentRuntimeSessionState::Stopping => "active", + AgentRuntimeSessionState::Stopped => "stopped", + AgentRuntimeSessionState::Invalidated => "invalidated", + AgentRuntimeSessionState::Failed => "failed", + } +} + +fn failed_prompt_envelope( + stage: impl Into, + error_code: impl Into, + message: impl Into, +) -> PublicPromptEnvelope { + PublicPromptEnvelope::Failed { + version: 1, + terminal: true, + stage: stage.into(), + error_code: error_code.into(), + message: message.into(), + } +} + +#[derive(Debug)] +pub(crate) struct PublicPromptRenderedExit { + exit_code: i32, +} + +impl std::fmt::Display for PublicPromptRenderedExit { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + write!(f, "public prompt command already rendered terminal output") + } +} + +impl std::error::Error for PublicPromptRenderedExit {} + +pub(crate) fn public_prompt_rendered_exit_code(err: &anyhow::Error) -> Option { + err.downcast_ref::() + .map(|exit| exit.exit_code) +} + +fn resolve_public_start_prompt_target( + store: &AgentRuntimeStateStore, + orchestration_session_id: &str, + backend_id: &str, +) -> Result { + let Some(record) = store.load_orchestration_session(orchestration_session_id)? else { + anyhow::bail!( + "unknown_session: no orchestration session found for '{}'", + orchestration_session_id + ); + }; + if record.state != OrchestrationSessionState::Active { + anyhow::bail!( + "runtime_start_failed: orchestration session {} is not active", + orchestration_session_id + ); + } + let active_participant_id = record.active_participant_id().ok_or_else(|| { + anyhow::anyhow!( + "runtime_start_failed: orchestration session {} is missing an active participant", + orchestration_session_id + ) + })?; + let active_participant = store + .load_participant(active_participant_id)? + .ok_or_else(|| { + anyhow::anyhow!( + "runtime_start_failed: orchestration session {} lost its active participant {}", + orchestration_session_id, + active_participant_id + ) + })?; + if active_participant.handle.backend_id != backend_id { + anyhow::bail!( + "runtime_start_failed: orchestration session {} active backend {} did not match requested backend {}", + orchestration_session_id, + active_participant.handle.backend_id, + backend_id + ); + } + Ok(active_participant.handle.participant_id.clone()) +} + +#[cfg(unix)] +async fn wait_for_private_prompt_transport(path: &Path) -> Result<()> { + let started_at = std::time::Instant::now(); + loop { + match tokio::fs::metadata(path).await { + Ok(_) => return Ok(()), + Err(err) if err.kind() == io::ErrorKind::NotFound => { + if started_at.elapsed() >= PRIVATE_PROMPT_READY_TIMEOUT { + anyhow::bail!( + "stream_bridge_failed: timed out waiting for private prompt transport {}", + path.display() + ); + } + tokio::time::sleep(PRIVATE_PROMPT_READY_POLL_INTERVAL).await; + } + Err(err) => { + return Err(err).with_context(|| { + format!( + "stream_bridge_failed: failed to inspect private prompt transport {}", + path.display() + ) + }); + } + } + } +} + +fn warning_for_exit_status(status: &ExitStatus) -> Option { + if status.success() { + return None; + } + + #[cfg(unix)] + { + use std::os::unix::process::ExitStatusExt; + if let Some(sig) = status.signal() { + return Some(format!("Command terminated by signal {sig}")); + } + } + + Some(format!( + "Command failed with status: {}", + status.code().unwrap_or(-1) + )) +} + +fn extract_session_handle_id(data: Option<&serde_json::Value>) -> Option<&str> { + let value = data?; + if value.get("schema").and_then(serde_json::Value::as_str) + == Some(super::SESSION_HANDLE_SCHEMA_V1) + { + return value + .get("session") + .and_then(serde_json::Value::as_object) + .and_then(|session| session.get("id")) + .and_then(serde_json::Value::as_str) + .filter(|id| !id.trim().is_empty()); + } + + value + .get("type") + .and_then(serde_json::Value::as_str) + .filter(|event_type| matches!(*event_type, "thread.started" | "turn.started"))?; + value + .get("thread_id") + .and_then(serde_json::Value::as_str) + .filter(|id| !id.trim().is_empty()) +} + +fn translate_prompt_wrapper_event( + manifest: &AgentRuntimeSessionManifest, + orchestration_session: &OrchestrationSessionRecord, + run_id: &str, + wrapper_event: agent_api::AgentWrapperEvent, +) -> AgentEvent { + let mut event = match wrapper_event.kind { + agent_api::AgentWrapperEventKind::Status => AgentEvent::message( + manifest.handle.agent_id.clone(), + manifest.handle.orchestration_session_id.clone(), + run_id.to_string(), + MessageEventKind::Status, + wrapper_event + .message + .clone() + .unwrap_or_else(|| "agent runtime status".to_string()), + ), + agent_api::AgentWrapperEventKind::TextOutput => AgentEvent::message( + manifest.handle.agent_id.clone(), + manifest.handle.orchestration_session_id.clone(), + run_id.to_string(), + MessageEventKind::TaskProgress, + wrapper_event + .text + .clone() + .unwrap_or_else(|| "agent runtime output".to_string()), + ), + agent_api::AgentWrapperEventKind::ToolCall + | agent_api::AgentWrapperEventKind::ToolResult => AgentEvent::message( + manifest.handle.agent_id.clone(), + manifest.handle.orchestration_session_id.clone(), + run_id.to_string(), + MessageEventKind::TaskProgress, + wrapper_event + .message + .clone() + .unwrap_or_else(|| "agent runtime tool activity".to_string()), + ), + agent_api::AgentWrapperEventKind::Error => AgentEvent::alert( + manifest.handle.agent_id.clone(), + manifest.handle.orchestration_session_id.clone(), + run_id.to_string(), + "agent_wrapper_error", + wrapper_event + .message + .clone() + .unwrap_or_else(|| "agent runtime error".to_string()), + ), + agent_api::AgentWrapperEventKind::Unknown => AgentEvent::message( + manifest.handle.agent_id.clone(), + manifest.handle.orchestration_session_id.clone(), + run_id.to_string(), + MessageEventKind::TaskProgress, + "agent runtime emitted an unknown event".to_string(), + ), + }; + + event.role = Some(manifest.handle.role.clone()); + event.backend_id = Some(manifest.handle.backend_id.clone()); + event.set_pure_agent_telemetry_identity(manifest.handle.agent_id.clone()); + event.set_channel(wrapper_event.channel.clone()); + event.world_id = orchestration_session.world_id.clone(); + event.world_generation = orchestration_session.world_generation; + event.participant_id = Some(manifest.handle.participant_id.clone()); + event.parent_participant_id = manifest.handle.parent_participant_id.clone(); + event.resumed_from_participant_id = manifest.handle.resumed_from_participant_id.clone(); + + if let Some(data) = wrapper_event.data { + if let Some(obj) = event.data.as_object_mut() { + obj.insert("uaa_event".to_string(), data); + obj.insert( + "protocol".to_string(), + serde_json::json!(PURE_AGENT_PROTOCOL), + ); + } + } + + event +} + +#[cfg(unix)] +struct PublicPromptRenderer { + json: bool, +} + +#[cfg(unix)] +impl PublicPromptRenderer { + fn new(json: bool) -> Self { + Self { json } + } + + fn render(&mut self, envelope: &PublicPromptEnvelope) -> Result<()> { + if self.json { + let stdout = io::stdout(); + let mut lock = stdout.lock(); + writeln!(lock, "{}", serde_json::to_string(envelope)?) + .context("failed to render prompt envelope")?; + let _ = lock.flush(); + return Ok(()); + } + + match envelope { + PublicPromptEnvelope::Accepted { .. } => {} + PublicPromptEnvelope::Completed { + action, + orchestration_session_id, + backend_id, + participant_id, + turn_outcome, + session_posture, + .. + } => { + let stdout = io::stdout(); + let mut lock = stdout.lock(); + let _ = writeln!( + lock, + "action={} orchestration_session_id={} backend_id={} participant_id={} turn_outcome={} session_posture={}", + action.as_str(), + orchestration_session_id, + backend_id, + participant_id.as_deref().unwrap_or("-"), + turn_outcome, + session_posture.as_str() + ); + let _ = lock.flush(); + } + PublicPromptEnvelope::Warning { message, .. } + | PublicPromptEnvelope::Failed { message, .. } => { + let stderr = io::stderr(); + let mut lock = stderr.lock(); + let _ = writeln!(lock, "{message}"); + let _ = lock.flush(); + } + PublicPromptEnvelope::Event { + event_kind, data, .. + } => { + if event_kind == "stderr" { + let stderr = io::stderr(); + let mut lock = stderr.lock(); + let _ = lock.write_all(prompt_event_text(data).as_bytes()); + let _ = lock.flush(); + } else if let Ok(event) = serde_json::from_value::(data.clone()) { + let stdout = io::stdout(); + let mut lock = stdout.lock(); + let _ = writeln!(lock, "{}", format_event_line(&event)); + let _ = lock.flush(); + } else { + let stdout = io::stdout(); + let mut lock = stdout.lock(); + let _ = lock.write_all(prompt_event_text(data).as_bytes()); + let _ = lock.flush(); + } + } + } + Ok(()) + } +} + +#[cfg(unix)] +fn prompt_event_text(data: &serde_json::Value) -> String { + data.get("text") + .and_then(serde_json::Value::as_str) + .unwrap_or_default() + .to_string() +} + +#[cfg(test)] +mod tests { + use super::{ + apply_runtime_stop_closeout, reconcile_hidden_owner_helper_start_timeout, + validate_public_prompt_command_request, HiddenOwnerHelperLaunchPlan, + HiddenOwnerHelperParticipantPlan, HiddenOwnerHelperSessionPlan, + HiddenOwnerHelperStartTimeoutReconciliation, HiddenOwnerHelperStartupPromptPlan, + LoadedPublicPrompt, OwnerHelperMode, PrivateStopOutcome, PublicPromptAction, + PublicPromptCommandRequest, ResolvedRuntimeBackendKind, ResolvedRuntimeDescriptor, + PURE_AGENT_PROTOCOL, + }; + #[cfg(unix)] + use super::{ + handle_private_prompt_connection, private_prompt_request_channel, PublicPromptEnvelope, + }; + use crate::execution::agent_runtime::{ + mapping::AgentRuntimeBackendKind, + orchestration_session::{ + OrchestrationSessionPosture, OrchestrationSessionRecord, OrchestrationSessionState, + StartupPromptStreamState, + }, + session::{AgentRuntimeParticipantRecord, AgentRuntimeSessionState}, + validator::RuntimeSelectionDescriptor, + AgentRuntimeStateStore, + }; + use crate::execution::config_model::AgentExecutionScope; + use std::path::PathBuf; + use tempfile::TempDir; + #[cfg(unix)] + use tokio::io::{AsyncBufReadExt, AsyncWriteExt, BufReader}; + #[cfg(unix)] + use tokio::net::UnixStream; + + fn with_store(test: impl FnOnce(&AgentRuntimeStateStore)) { + let temp = TempDir::new().expect("tempdir"); + std::env::set_var("SUBSTRATE_HOME", temp.path()); + let store = AgentRuntimeStateStore::new().expect("state store"); + test(&store); + std::env::remove_var("SUBSTRATE_HOME"); + } + + fn test_plan( + orchestration_session_id: &str, + participant_id: &str, + ) -> HiddenOwnerHelperLaunchPlan { + HiddenOwnerHelperLaunchPlan { + mode: OwnerHelperMode::Start, + descriptor: ResolvedRuntimeDescriptor { + agent_id: "codex".to_string(), + backend_id: "cli:codex".to_string(), + backend_kind: ResolvedRuntimeBackendKind::Codex, + protocol: PURE_AGENT_PROTOCOL.to_string(), + execution_scope: AgentExecutionScope::Host, + binary_path: "/usr/bin/codex".to_string(), + }, + session: HiddenOwnerHelperSessionPlan { + orchestration_session_id: orchestration_session_id.to_string(), + shell_trace_session_id: "trace_session".to_string(), + workspace_root: "/workspace".to_string(), + world_id: None, + world_generation: None, + }, + participant: HiddenOwnerHelperParticipantPlan { + participant_id: participant_id.to_string(), + lease_token: format!("lease_{participant_id}"), + run_id: "run_start".to_string(), + resumed_from_participant_id: None, + internal_uaa_session_id: None, + }, + startup_prompt: Some(HiddenOwnerHelperStartupPromptPlan { + prompt_text: "hello".to_string(), + stream_path: PathBuf::from("/tmp/startup.sock"), + }), + source_orchestration_session_id: None, + } + } + + #[test] + fn owner_helper_modes_remain_internal_and_exact() { + assert_eq!(OwnerHelperMode::Start.as_str(), "start"); + assert_eq!(OwnerHelperMode::Resume.as_str(), "resume"); + assert_eq!(OwnerHelperMode::ResumeOneTurn.as_str(), "resume_one_turn"); + assert_eq!(OwnerHelperMode::Fork.as_str(), "fork"); + } + + #[test] + fn private_stop_outcomes_are_exact() { + let outcomes = [ + PrivateStopOutcome::Accepted, + PrivateStopOutcome::AlreadyTerminal, + PrivateStopOutcome::OwnerUnreachable, + PrivateStopOutcome::ProtocolError, + ]; + assert_eq!(outcomes.len(), 4); + } + + #[test] + fn stop_closeout_helper_converges_on_stopped_terminal_snapshots() { + let descriptor = RuntimeSelectionDescriptor { + agent_id: "codex".to_string(), + backend_id: "cli:codex".to_string(), + backend_kind: AgentRuntimeBackendKind::Codex, + protocol: PURE_AGENT_PROTOCOL.to_string(), + execution_scope: AgentExecutionScope::Host, + binary_path: PathBuf::from("/usr/bin/codex"), + }; + let mut manifest = AgentRuntimeParticipantRecord::new_orchestrator_participant( + &descriptor, + "sess_stop_helper".to_string(), + "ash_stop_helper".to_string(), + "lease_stop_helper".to_string(), + ) + .expect("orchestrator participant"); + manifest.transition_state(AgentRuntimeSessionState::Ready); + manifest.set_uaa_session_id("uaa_session"); + manifest.mark_runtime_ownership_retained(); + let mut orchestration = OrchestrationSessionRecord::new( + "sess_stop_helper".to_string(), + "trace_session".to_string(), + "/workspace".to_string(), + &manifest, + ); + orchestration.transition_state(OrchestrationSessionState::Active); + orchestration.bind_active_session_handle(manifest.handle.participant_id.clone()); + + apply_runtime_stop_closeout(&mut orchestration, &mut manifest); + + assert_eq!(manifest.handle.state, AgentRuntimeSessionState::Stopped); + assert_eq!( + manifest.internal.termination_reason.as_deref(), + Some("stopped") + ); + assert!(!manifest.internal.resume_eligible); + assert!(!manifest.internal.attached_client_present); + assert_eq!(orchestration.state, OrchestrationSessionState::Stopped); + assert!(orchestration.closed_at.is_some()); + assert_eq!( + orchestration.posture, + crate::execution::agent_runtime::orchestration_session::OrchestrationSessionPosture::Terminal + ); + } + + #[test] + fn public_turn_prompt_requests_require_exact_session_and_backend_contract() { + let prompt = LoadedPublicPrompt { + prompt_text: "hello".to_string(), + }; + let missing_session = PublicPromptCommandRequest { + action: PublicPromptAction::Turn, + orchestration_session_id: None, + backend_id: "cli:codex".to_string(), + prompt: prompt.clone(), + json: false, + }; + let missing_backend = PublicPromptCommandRequest { + action: PublicPromptAction::Turn, + orchestration_session_id: Some("sess_public".to_string()), + backend_id: " ".to_string(), + prompt, + json: false, + }; + + let session_err = validate_public_prompt_command_request(&missing_session) + .expect_err("turn requests must require an orchestration session id"); + assert!(session_err + .to_string() + .contains("public turn actions require --session ")); + + let backend_err = validate_public_prompt_command_request(&missing_backend) + .expect_err("turn requests must require an exact backend id"); + assert!(backend_err + .to_string() + .contains("missing_backend: public turn actions require --backend ")); + } + + #[test] + #[serial_test::serial] + fn start_timeout_reconciliation_parks_stale_attached_truth_into_detached_success() { + with_store(|store| { + let descriptor = RuntimeSelectionDescriptor { + agent_id: "codex".to_string(), + backend_id: "cli:codex".to_string(), + backend_kind: AgentRuntimeBackendKind::Codex, + protocol: PURE_AGENT_PROTOCOL.to_string(), + execution_scope: AgentExecutionScope::Host, + binary_path: PathBuf::from("/usr/bin/codex"), + }; + let mut participant = AgentRuntimeParticipantRecord::new_orchestrator_participant( + &descriptor, + "sess_start_timeout_success".to_string(), + "ash_start_timeout_success".to_string(), + "lease_start_timeout_success".to_string(), + ) + .expect("orchestrator participant"); + participant.transition_state(AgentRuntimeSessionState::Ready); + participant.set_uaa_session_id("uaa_session"); + + let mut orchestration = OrchestrationSessionRecord::new( + "sess_start_timeout_success".to_string(), + "trace_session".to_string(), + "/workspace".to_string(), + &participant, + ); + orchestration.transition_state(OrchestrationSessionState::Active); + orchestration.bind_active_session_handle(participant.handle.participant_id.clone()); + orchestration.initialize_startup_prompt(participant.handle.participant_id.clone()); + orchestration.mark_startup_prompt_completed( + participant.handle.participant_id.as_str(), + "success", + ); + participant.mark_client_detached("owner detached cleanly"); + + store + .persist_orchestration_session(&orchestration) + .expect("persist orchestration"); + store + .persist_participant(&participant) + .expect("persist participant"); + + let reconciliation = reconcile_hidden_owner_helper_start_timeout( + store, + &test_plan("sess_start_timeout_success", "ash_start_timeout_success"), + ) + .expect("reconcile timeout"); + assert_eq!( + reconciliation, + HiddenOwnerHelperStartTimeoutReconciliation::Success + ); + + let persisted = store + .load_orchestration_session("sess_start_timeout_success") + .expect("load orchestration") + .expect("orchestration exists"); + assert_eq!(persisted.state, OrchestrationSessionState::Active); + assert_eq!( + persisted.posture, + OrchestrationSessionPosture::ParkedResumable + ); + assert_eq!(persisted.attached_participant_id(), None); + + let readiness = store + .classify_hidden_owner_helper_launch_readiness( + "sess_start_timeout_success", + "ash_start_timeout_success", + false, + ) + .expect("classify readiness"); + assert_eq!( + readiness, + crate::execution::agent_runtime::state_store::HiddenOwnerHelperLaunchReadiness::ReadyDetached( + OrchestrationSessionPosture::ParkedResumable + ) + ); + }); + } + + #[test] + #[serial_test::serial] + fn helper_readiness_does_not_wait_for_startup_prompt_terminalization_once_attached_live() { + with_store(|store| { + let descriptor = RuntimeSelectionDescriptor { + agent_id: "codex".to_string(), + backend_id: "cli:codex".to_string(), + backend_kind: AgentRuntimeBackendKind::Codex, + protocol: PURE_AGENT_PROTOCOL.to_string(), + execution_scope: AgentExecutionScope::Host, + binary_path: PathBuf::from("/usr/bin/codex"), + }; + let mut participant = AgentRuntimeParticipantRecord::new_orchestrator_participant( + &descriptor, + "sess_start_ready_attached".to_string(), + "ash_start_ready_attached".to_string(), + "lease_start_ready_attached".to_string(), + ) + .expect("orchestrator participant"); + participant.transition_state(AgentRuntimeSessionState::Ready); + participant.set_uaa_session_id("uaa_session"); + participant.mark_runtime_ownership_retained(); + + let mut orchestration = OrchestrationSessionRecord::new( + "sess_start_ready_attached".to_string(), + "trace_session".to_string(), + "/workspace".to_string(), + &participant, + ); + orchestration.transition_state(OrchestrationSessionState::Active); + orchestration.bind_active_session_handle(participant.handle.participant_id.clone()); + orchestration.initialize_startup_prompt(participant.handle.participant_id.clone()); + orchestration.mark_startup_prompt_accepted(participant.handle.participant_id.as_str()); + + store + .persist_orchestration_session(&orchestration) + .expect("persist orchestration"); + store + .persist_participant(&participant) + .expect("persist participant"); + + super::wait_for_hidden_owner_helper_readiness( + store, + &test_plan("sess_start_ready_attached", "ash_start_ready_attached"), + ) + .expect("attached live readiness should not wait for startup prompt completion"); + }); + } + + #[test] + #[serial_test::serial] + fn start_timeout_reconciliation_marks_terminal_failure_when_startup_prompt_is_not_terminal() { + with_store(|store| { + let descriptor = RuntimeSelectionDescriptor { + agent_id: "codex".to_string(), + backend_id: "cli:codex".to_string(), + backend_kind: AgentRuntimeBackendKind::Codex, + protocol: PURE_AGENT_PROTOCOL.to_string(), + execution_scope: AgentExecutionScope::Host, + binary_path: PathBuf::from("/usr/bin/codex"), + }; + let mut participant = AgentRuntimeParticipantRecord::new_orchestrator_participant( + &descriptor, + "sess_start_timeout_failure".to_string(), + "ash_start_timeout_failure".to_string(), + "lease_start_timeout_failure".to_string(), + ) + .expect("orchestrator participant"); + participant.transition_state(AgentRuntimeSessionState::Ready); + participant.set_uaa_session_id("uaa_session"); + + let mut orchestration = OrchestrationSessionRecord::new( + "sess_start_timeout_failure".to_string(), + "trace_session".to_string(), + "/workspace".to_string(), + &participant, + ); + orchestration.transition_state(OrchestrationSessionState::Active); + orchestration.bind_active_session_handle(participant.handle.participant_id.clone()); + orchestration.initialize_startup_prompt(participant.handle.participant_id.clone()); + orchestration.mark_startup_prompt_accepted(participant.handle.participant_id.as_str()); + participant.mark_client_detached("owner detached cleanly"); + + store + .persist_orchestration_session(&orchestration) + .expect("persist orchestration"); + store + .persist_participant(&participant) + .expect("persist participant"); + + let reconciliation = reconcile_hidden_owner_helper_start_timeout( + store, + &test_plan("sess_start_timeout_failure", "ash_start_timeout_failure"), + ) + .expect("reconcile timeout"); + assert_eq!( + reconciliation, + HiddenOwnerHelperStartTimeoutReconciliation::FailureMarkedTerminal + ); + + let persisted = store + .load_orchestration_session("sess_start_timeout_failure") + .expect("load orchestration") + .expect("orchestration exists"); + assert_eq!(persisted.state, OrchestrationSessionState::Failed); + assert_eq!(persisted.posture, OrchestrationSessionPosture::Terminal); + assert_eq!(persisted.attached_participant_id(), None); + assert_eq!( + persisted.startup_prompt_state(), + Some(StartupPromptStreamState::Failed) + ); + + let persisted_participant = store + .load_participant("ash_start_timeout_failure") + .expect("load participant") + .expect("participant exists"); + assert_eq!( + persisted_participant.handle.state, + AgentRuntimeSessionState::Invalidated + ); + assert!(persisted_participant + .internal + .terminal_observed_at + .is_some()); + assert_eq!( + persisted_participant.internal.last_error_bucket.as_deref(), + Some("bootstrap_run") + ); + }); + } + + #[cfg(unix)] + #[tokio::test(flavor = "current_thread")] + async fn private_prompt_bridge_emits_terminal_failed_after_accepted_owner_drop() { + let (client, server) = UnixStream::pair().expect("unix stream pair"); + let (prompt_tx, mut prompt_rx) = private_prompt_request_channel(); + let server_task = tokio::spawn(async move { + handle_private_prompt_connection(server, prompt_tx) + .await + .expect("private prompt bridge should complete"); + }); + let owner_task = tokio::spawn(async move { + let request = prompt_rx.recv().await.expect("prompt request"); + request + .envelope_tx + .send(PublicPromptEnvelope::Accepted { + version: 1, + action: PublicPromptAction::Turn, + orchestration_session_id: "orch-parked".to_string(), + backend_id: "cli:codex".to_string(), + participant_id: Some("ash_resumed".to_string()), + scope: "host".to_string(), + }) + .expect("accepted envelope should send"); + }); + + let mut client = client; + client + .write_all(br#"{"version":1,"action":"turn","prompt":"resume"}"#) + .await + .expect("write request"); + client.write_all(b"\n").await.expect("newline"); + client.flush().await.expect("flush request"); + + let mut reader = BufReader::new(client); + let mut line = String::new(); + let mut envelopes = Vec::new(); + loop { + line.clear(); + let bytes_read = reader.read_line(&mut line).await.expect("read envelope"); + if bytes_read == 0 { + break; + } + envelopes.push( + serde_json::from_str::(line.trim()) + .expect("decode prompt envelope"), + ); + } + + owner_task.await.expect("owner task"); + server_task.await.expect("server task"); + + assert!(matches!( + envelopes.first(), + Some(PublicPromptEnvelope::Accepted { .. }) + )); + assert!(matches!( + envelopes.get(1), + Some(PublicPromptEnvelope::Failed { + error_code, + message, + .. + }) if error_code == "owner_unreachable" + && message.contains("closed after accepting") + )); + } +} diff --git a/crates/shell/src/execution/agent_runtime/mod.rs b/crates/shell/src/execution/agent_runtime/mod.rs index b300b156a..1ba281653 100644 --- a/crates/shell/src/execution/agent_runtime/mod.rs +++ b/crates/shell/src/execution/agent_runtime/mod.rs @@ -1,16 +1,34 @@ +pub(crate) mod control; pub(crate) mod mapping; +pub(crate) mod orchestration_session; pub(crate) mod registry; pub(crate) mod session; pub(crate) mod state_store; pub(crate) mod validator; +#[allow(unused_imports)] +pub(crate) use control::{ + load_public_prompt_source, public_prompt_rendered_exit_code, run_public_prompt_command, + LoadedPublicPrompt, PromptSubmitRuntime, PublicPromptAction, PublicPromptCommandRequest, + PublicPromptInput, PublicPromptRenderedExit, PublicSessionPosture, SubmittedPromptCompletion, + SubmittedPromptStreamEvent, +}; pub(crate) use mapping::{ MEMBER_ROLE, NESTED_ROUTER, ORCHESTRATOR_ROLE, PURE_AGENT_PROTOCOL, PURE_AGENT_ROUTER, SESSION_HANDLE_SCHEMA_V1, }; +pub(crate) use orchestration_session::{OrchestrationSessionRecord, OrchestrationSessionState}; pub(crate) use registry::build_gateway_for_descriptor; -pub(crate) use session::{AgentRuntimeSessionManifest, AgentRuntimeSessionState}; -pub(crate) use state_store::AgentRuntimeStateStore; +#[allow(unused_imports)] +pub(crate) use session::{ + AgentRuntimeOwnershipMode, AgentRuntimeParticipantRecord, AgentRuntimeParticipantWorldBinding, + AgentRuntimeSessionManifest, AgentRuntimeSessionState, +}; +#[allow(unused_imports)] +pub(crate) use state_store::{ + AgentRuntimeSessionRecord, AgentRuntimeStateStore, PublicControlAction, PublicTurnTargetKind, + ResolvedPublicTurnTarget, StartupPromptReplayState, +}; pub(crate) use validator::{ backend_allowed, runtime_realizability_error_exit_code, validate_orchestrator_selection, validate_runtime_realizability, diff --git a/crates/shell/src/execution/agent_runtime/orchestration_session.rs b/crates/shell/src/execution/agent_runtime/orchestration_session.rs new file mode 100644 index 000000000..f6bd9821c --- /dev/null +++ b/crates/shell/src/execution/agent_runtime/orchestration_session.rs @@ -0,0 +1,428 @@ +use chrono::{DateTime, Utc}; +use serde::{Deserialize, Serialize}; + +use super::session::AgentRuntimeSessionManifest; + +#[derive(Clone, Debug, Serialize, Deserialize, PartialEq, Eq)] +#[serde(rename_all = "snake_case")] +pub(crate) enum OrchestrationSessionState { + Allocating, + Active, + Invalidated, + Stopping, + Stopped, + Failed, +} + +impl OrchestrationSessionState { + pub(crate) fn is_active(&self) -> bool { + matches!(self, Self::Active) + } + + pub(crate) fn is_terminal(&self) -> bool { + matches!(self, Self::Invalidated | Self::Stopped | Self::Failed) + } +} + +#[derive(Clone, Copy, Debug, Default, Serialize, Deserialize, PartialEq, Eq)] +#[serde(rename_all = "snake_case")] +pub(crate) enum StartupPromptStreamState { + #[default] + PendingAcceptance, + Accepted, + Completed, + Failed, +} + +impl StartupPromptStreamState { + pub(crate) fn accepted(self) -> bool { + !matches!(self, Self::PendingAcceptance) + } +} + +#[derive(Clone, Debug, Serialize, Deserialize, PartialEq, Eq)] +pub(crate) struct StartupPromptRecord { + pub participant_id: String, + #[serde(default)] + pub state: StartupPromptStreamState, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub accepted_at: Option>, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub terminal_at: Option>, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub turn_outcome: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub error_message: Option, +} + +#[derive(Clone, Copy, Debug, Default, Serialize, Deserialize, PartialEq, Eq)] +#[serde(rename_all = "snake_case")] +pub(crate) enum OrchestrationSessionPosture { + ActiveAttached, + #[default] + ParkedResumable, + AwaitingAttention, + Terminal, +} + +#[derive(Clone, Debug, Serialize, Deserialize, PartialEq, Eq)] +pub(crate) struct OrchestrationSessionRecord { + // This is the public/operator-facing selector for the parent orchestration row. + pub orchestration_session_id: String, + pub shell_trace_session_id: String, + pub workspace_root: String, + pub shell_owner_pid: u32, + pub state: OrchestrationSessionState, + pub opened_at: DateTime, + pub last_active_at: DateTime, + pub orchestrator_agent_id: String, + pub orchestrator_backend_id: String, + pub orchestrator_protocol: String, + // Compatibility storage name retained; the value is the active orchestrator participant_id. + pub active_session_handle_id: Option, + pub latest_run_id: Option, + pub world_id: Option, + pub world_generation: Option, + pub invalidation_reason: Option, + pub closed_at: Option>, + #[serde(default)] + pub posture: OrchestrationSessionPosture, + #[serde(default = "Utc::now")] + pub posture_changed_at: DateTime, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub attached_participant_id: Option, + #[serde(default)] + pub pending_inbox_count: u64, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub last_parked_at: Option>, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub last_attention_at: Option>, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub parked_reason: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub startup_prompt: Option, +} + +impl OrchestrationSessionRecord { + pub(crate) fn new( + orchestration_session_id: String, + shell_trace_session_id: String, + workspace_root: String, + child_manifest: &AgentRuntimeSessionManifest, + ) -> Self { + let now = Utc::now(); + Self { + orchestration_session_id, + shell_trace_session_id, + workspace_root, + shell_owner_pid: child_manifest.internal.shell_owner_pid, + state: OrchestrationSessionState::Allocating, + opened_at: now, + last_active_at: now, + orchestrator_agent_id: child_manifest.handle.agent_id.clone(), + orchestrator_backend_id: child_manifest.handle.backend_id.clone(), + orchestrator_protocol: child_manifest.handle.protocol.clone(), + active_session_handle_id: None, + latest_run_id: child_manifest.internal.latest_run_id.clone(), + world_id: None, + world_generation: None, + invalidation_reason: None, + closed_at: None, + posture: OrchestrationSessionPosture::ActiveAttached, + posture_changed_at: now, + attached_participant_id: Some(child_manifest.handle.participant_id.clone()), + pending_inbox_count: 0, + last_parked_at: None, + last_attention_at: None, + parked_reason: None, + startup_prompt: None, + } + } + + pub(crate) fn transition_state(&mut self, next: OrchestrationSessionState) { + self.state = next; + self.last_active_at = Utc::now(); + if self.state.is_terminal() { + self.apply_posture(OrchestrationSessionPosture::Terminal); + self.attached_participant_id = None; + } else if self.state.is_active() { + self.closed_at = None; + self.invalidation_reason = None; + if self.posture == OrchestrationSessionPosture::Terminal { + self.apply_posture(self.detached_posture_without_attachment()); + } + } + } + + pub(crate) fn touch_active(&mut self) { + self.last_active_at = Utc::now(); + } + + pub(crate) fn active_participant_id(&self) -> Option<&str> { + self.active_session_handle_id.as_deref() + } + + pub(crate) fn attached_participant_id(&self) -> Option<&str> { + self.attached_participant_id.as_deref() + } + + pub(crate) fn has_world_binding(&self) -> bool { + self.world_id.is_some() && self.world_generation.is_some() + } + + pub(crate) fn bind_active_session_handle(&mut self, participant_id: impl Into) { + let participant_id = participant_id.into(); + self.active_session_handle_id = Some(participant_id.clone()); + self.attached_participant_id = Some(participant_id); + self.apply_posture(OrchestrationSessionPosture::ActiveAttached); + self.parked_reason = None; + self.touch_active(); + } + + pub(crate) fn set_world_binding(&mut self, world_id: impl Into, world_generation: u64) { + self.world_id = Some(world_id.into()); + self.world_generation = Some(world_generation); + self.touch_active(); + } + + pub(crate) fn clear_world_binding(&mut self) { + self.world_id = None; + self.world_generation = None; + self.touch_active(); + } + + #[allow(dead_code)] + pub(crate) fn set_pending_inbox_count(&mut self, pending_inbox_count: u64) { + self.pending_inbox_count = pending_inbox_count; + if pending_inbox_count > 0 { + self.last_attention_at = Some(Utc::now()); + } + self.touch_active(); + } + + #[allow(dead_code)] + pub(crate) fn mark_active_attached(&mut self, participant_id: impl Into) { + self.attached_participant_id = Some(participant_id.into()); + self.apply_posture(OrchestrationSessionPosture::ActiveAttached); + self.parked_reason = None; + self.touch_active(); + } + + #[allow(dead_code)] + pub(crate) fn mark_parked_resumable(&mut self, reason: impl Into) { + self.attached_participant_id = None; + self.last_parked_at = Some(Utc::now()); + self.parked_reason = Some(reason.into()); + self.apply_posture(OrchestrationSessionPosture::ParkedResumable); + self.touch_active(); + } + + #[allow(dead_code)] + pub(crate) fn mark_awaiting_attention(&mut self) { + self.attached_participant_id = None; + self.last_attention_at = Some(Utc::now()); + self.apply_posture(OrchestrationSessionPosture::AwaitingAttention); + self.touch_active(); + } + + pub(crate) fn validate_persisted_invariants(&self) -> anyhow::Result<()> { + match self.posture { + OrchestrationSessionPosture::ActiveAttached => { + if self.attached_participant_id.is_none() { + anyhow::bail!("active_attached posture requires attached_participant_id"); + } + } + OrchestrationSessionPosture::ParkedResumable => { + if self.attached_participant_id.is_some() { + anyhow::bail!("parked_resumable posture must clear attached_participant_id"); + } + if self.pending_inbox_count > 0 { + anyhow::bail!("parked_resumable posture cannot retain pending inbox items"); + } + } + OrchestrationSessionPosture::AwaitingAttention => { + if self.attached_participant_id.is_some() { + anyhow::bail!("awaiting_attention posture must clear attached_participant_id"); + } + if self.pending_inbox_count == 0 { + anyhow::bail!("awaiting_attention posture requires pending_inbox_count > 0"); + } + } + OrchestrationSessionPosture::Terminal => { + if self.attached_participant_id.is_some() { + anyhow::bail!("terminal posture must clear attached_participant_id"); + } + } + } + + if self.state.is_terminal() && self.posture != OrchestrationSessionPosture::Terminal { + anyhow::bail!("terminal session state requires terminal posture"); + } + if !self.state.is_terminal() && self.posture == OrchestrationSessionPosture::Terminal { + anyhow::bail!("non-terminal session state cannot advertise terminal posture"); + } + + Ok(()) + } + + pub(crate) fn mark_terminal(&mut self, reason: impl Into) { + let reason = reason.into(); + self.last_active_at = Utc::now(); + self.closed_at = Some(self.last_active_at); + self.invalidation_reason = Some(reason.clone()); + self.attached_participant_id = None; + self.parked_reason = Some(reason); + self.apply_posture(OrchestrationSessionPosture::Terminal); + } + + pub(crate) fn initialize_startup_prompt(&mut self, participant_id: impl Into) { + self.startup_prompt = Some(StartupPromptRecord { + participant_id: participant_id.into(), + state: StartupPromptStreamState::PendingAcceptance, + accepted_at: None, + terminal_at: None, + turn_outcome: None, + error_message: None, + }); + self.touch_active(); + } + + pub(crate) fn startup_prompt_state(&self) -> Option { + self.startup_prompt.as_ref().map(|record| record.state) + } + + pub(crate) fn mark_startup_prompt_accepted(&mut self, participant_id: &str) { + let Some(record) = self.startup_prompt.as_mut() else { + return; + }; + if record.participant_id != participant_id || record.state.accepted() { + return; + } + record.state = StartupPromptStreamState::Accepted; + record.accepted_at = Some(Utc::now()); + record.error_message = None; + self.touch_active(); + } + + pub(crate) fn mark_startup_prompt_completed( + &mut self, + participant_id: &str, + turn_outcome: impl Into, + ) { + let Some(record) = self.startup_prompt.as_mut() else { + return; + }; + if record.participant_id != participant_id { + return; + } + record.state = StartupPromptStreamState::Completed; + record.accepted_at.get_or_insert_with(Utc::now); + record.terminal_at = Some(Utc::now()); + record.turn_outcome = Some(turn_outcome.into()); + record.error_message = None; + self.touch_active(); + } + + pub(crate) fn mark_startup_prompt_failed( + &mut self, + participant_id: &str, + error_message: impl Into, + ) { + let Some(record) = self.startup_prompt.as_mut() else { + return; + }; + if record.participant_id != participant_id { + return; + } + record.state = StartupPromptStreamState::Failed; + record.accepted_at.get_or_insert_with(Utc::now); + record.terminal_at = Some(Utc::now()); + record.error_message = Some(error_message.into()); + self.touch_active(); + } + + fn apply_posture(&mut self, posture: OrchestrationSessionPosture) { + if self.posture != posture { + self.posture = posture; + self.posture_changed_at = Utc::now(); + } + } + + fn detached_posture_without_attachment(&self) -> OrchestrationSessionPosture { + if self.pending_inbox_count > 0 { + OrchestrationSessionPosture::AwaitingAttention + } else { + OrchestrationSessionPosture::ParkedResumable + } + } +} + +#[cfg(test)] +mod tests { + use std::path::PathBuf; + + use super::*; + use crate::execution::agent_runtime::{ + mapping::AgentRuntimeBackendKind, validator::RuntimeSelectionDescriptor, + }; + use crate::execution::config_model::AgentExecutionScope; + + fn manifest() -> AgentRuntimeSessionManifest { + crate::execution::agent_runtime::session::AgentRuntimeParticipantRecord::new_orchestrator_participant( + &RuntimeSelectionDescriptor { + agent_id: "codex".to_string(), + backend_id: "cli:codex".to_string(), + backend_kind: AgentRuntimeBackendKind::Codex, + protocol: "uaa.agent.session".to_string(), + execution_scope: AgentExecutionScope::Host, + binary_path: PathBuf::from("/usr/bin/codex"), + }, + "sess_001".to_string(), + "ash_001".to_string(), + "lease_001".to_string(), + ) + .expect("manifest") + } + + #[test] + fn new_session_starts_active_attached() { + let manifest = manifest(); + let session = OrchestrationSessionRecord::new( + "sess_001".to_string(), + "trace_001".to_string(), + "/workspace".to_string(), + &manifest, + ); + + assert_eq!(session.posture, OrchestrationSessionPosture::ActiveAttached); + assert_eq!(session.attached_participant_id(), Some("ash_001")); + assert_eq!(session.pending_inbox_count, 0); + session + .validate_persisted_invariants() + .expect("new session invariants"); + } + + #[test] + fn detached_postures_enforce_pending_inbox_truth() { + let manifest = manifest(); + let mut session = OrchestrationSessionRecord::new( + "sess_001".to_string(), + "trace_001".to_string(), + "/workspace".to_string(), + &manifest, + ); + + session.mark_parked_resumable("owner detached cleanly"); + session + .validate_persisted_invariants() + .expect("parked invariants"); + + session.set_pending_inbox_count(1); + assert!(session.validate_persisted_invariants().is_err()); + + session.mark_awaiting_attention(); + session + .validate_persisted_invariants() + .expect("attention invariants"); + } +} diff --git a/crates/shell/src/execution/agent_runtime/session.rs b/crates/shell/src/execution/agent_runtime/session.rs index ee1d63789..737fe5844 100644 --- a/crates/shell/src/execution/agent_runtime/session.rs +++ b/crates/shell/src/execution/agent_runtime/session.rs @@ -1,13 +1,14 @@ use chrono::{DateTime, Utc}; -use serde::{Deserialize, Serialize}; +use serde::{Deserialize, Deserializer, Serialize}; use crate::execution::config_model::AgentExecutionScope; -use super::mapping::ORCHESTRATOR_ROLE; +use super::mapping::{MEMBER_ROLE, ORCHESTRATOR_ROLE}; +use super::orchestration_session::OrchestrationSessionRecord; use super::validator::RuntimeSelectionDescriptor; #[derive(Clone, Debug, Serialize, Deserialize, PartialEq, Eq)] -pub(crate) struct AgentRuntimeSessionExecution { +pub(crate) struct AgentRuntimeParticipantExecution { pub scope: AgentExecutionScope, } @@ -34,14 +35,27 @@ impl AgentRuntimeSessionState { } #[derive(Clone, Debug, Serialize, Deserialize, PartialEq, Eq)] -pub(crate) struct AgentRuntimeSessionHandle { +#[serde(rename_all = "snake_case")] +pub(crate) enum AgentRuntimeOwnershipMode { + AttachedControl, + MemberRuntime, + Replaced, +} + +#[derive(Clone, Debug, Serialize, PartialEq, Eq)] +pub(crate) struct AgentRuntimeParticipantHandle { + // Canonical runtime lineage identifier for this participant row. + pub participant_id: String, + // Legacy compatibility alias kept in-memory for existing callers; do not use as canonical + // public terminology in new code. + #[serde(skip)] pub session_handle_id: String, pub orchestration_session_id: String, pub agent_id: String, pub backend_id: String, pub role: String, pub protocol: String, - pub execution: AgentRuntimeSessionExecution, + pub execution: AgentRuntimeParticipantExecution, pub state: AgentRuntimeSessionState, pub opened_at: DateTime, pub last_transition_at: DateTime, @@ -49,10 +63,18 @@ pub(crate) struct AgentRuntimeSessionHandle { pub world_id: Option, #[serde(skip_serializing_if = "Option::is_none")] pub world_generation: Option, + // Lineage links use participant_id canonically. The *_session_handle_id fields remain as + // compatibility mirrors for legacy reads only. #[serde(skip_serializing_if = "Option::is_none")] + pub parent_participant_id: Option, + #[serde(skip)] pub parent_session_handle_id: Option, #[serde(skip_serializing_if = "Option::is_none")] + pub resumed_from_participant_id: Option, + #[serde(skip)] pub resumed_from_session_handle_id: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub orchestrator_participant_id: Option, } #[derive(Clone, Debug, Serialize, Deserialize, PartialEq, Eq)] @@ -61,12 +83,13 @@ pub(crate) struct AgentRuntimeSessionInternal { pub resolved_binary_path: String, pub shell_owner_pid: u32, pub lease_token: String, + // Backend-native upstream runtime handle. Internal only; never the default operator target. #[serde(skip_serializing_if = "Option::is_none")] pub uaa_session_id: Option, #[serde(skip_serializing_if = "Option::is_none")] pub latest_run_id: Option, pub cancel_supported: bool, - // A manifest is only authoritative-live while the REPL still retains the attached + // A participant record is only authoritative-live while the REPL still retains the attached // UAA control boundary: the cancel handle remains owned, the event stream task is // still active, and the completion observer is still retained. #[serde(default)] @@ -76,7 +99,7 @@ pub(crate) struct AgentRuntimeSessionInternal { #[serde(default)] pub completion_observer_retained: bool, #[serde(default, skip_serializing_if = "Option::is_none")] - pub ownership_mode: Option, + pub ownership_mode: Option, #[serde(default)] pub ownership_valid: bool, #[serde(default, skip_serializing_if = "Option::is_none")] @@ -89,68 +112,362 @@ pub(crate) struct AgentRuntimeSessionInternal { pub terminal_observed_at: Option>, #[serde(default, skip_serializing_if = "Option::is_none")] pub termination_reason: Option, + #[serde(default)] + pub attached_client_present: bool, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub last_attached_at: Option>, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub last_detached_at: Option>, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub detach_reason: Option, + #[serde(default)] + pub resume_eligible: bool, #[serde(skip_serializing_if = "Option::is_none")] pub last_error_bucket: Option, #[serde(skip_serializing_if = "Option::is_none")] pub last_error_message: Option, } -#[derive(Clone, Debug, Serialize, Deserialize, PartialEq, Eq)] -pub(crate) struct AgentRuntimeSessionManifest { +#[derive(Clone, Debug, Serialize, PartialEq, Eq)] +pub(crate) struct AgentRuntimeParticipantRecord { #[serde(flatten)] - pub handle: AgentRuntimeSessionHandle, + pub handle: AgentRuntimeParticipantHandle, pub internal: AgentRuntimeSessionInternal, } -impl AgentRuntimeSessionManifest { +impl<'de> Deserialize<'de> for AgentRuntimeParticipantRecord { + fn deserialize(deserializer: D) -> Result + where + D: Deserializer<'de>, + { + let wire = AgentRuntimeParticipantRecordWire::deserialize(deserializer)?; + Self::try_from(wire).map_err(serde::de::Error::custom) + } +} + +impl TryFrom for AgentRuntimeParticipantRecord { + type Error = String; + + fn try_from(wire: AgentRuntimeParticipantRecordWire) -> Result { + let handle = AgentRuntimeParticipantHandle { + session_handle_id: wire.handle.participant_id.clone(), + participant_id: wire.handle.participant_id, + orchestration_session_id: wire.handle.orchestration_session_id, + agent_id: wire.handle.agent_id, + backend_id: wire.handle.backend_id, + role: wire.handle.role, + protocol: wire.handle.protocol, + execution: AgentRuntimeParticipantExecution { + scope: wire.handle.execution.scope, + }, + state: wire.handle.state, + opened_at: wire.handle.opened_at, + last_transition_at: wire.handle.last_transition_at, + world_id: wire.handle.world_id, + world_generation: wire.handle.world_generation, + parent_session_handle_id: wire.handle.parent_participant_id.clone(), + parent_participant_id: wire.handle.parent_participant_id, + resumed_from_session_handle_id: wire.handle.resumed_from_participant_id.clone(), + resumed_from_participant_id: wire.handle.resumed_from_participant_id, + orchestrator_participant_id: wire.handle.orchestrator_participant_id, + }; + let record = Self { + handle, + internal: wire.internal, + }; + record.validate().map_err(|err| err.to_string())?; + Ok(record) + } +} + +struct AgentRuntimeParticipantLineage { + parent_participant_id: Option, + resumed_from_participant_id: Option, + orchestrator_participant_id: Option, +} + +struct AgentRuntimeParticipantInit { + orchestration_session_id: String, + participant_id: String, + role: String, + lease_token: String, + lineage: AgentRuntimeParticipantLineage, + world: Option, + ownership_mode: AgentRuntimeOwnershipMode, +} + +#[allow(dead_code)] +pub(crate) struct AgentRuntimeReplacementParticipantInit { + pub orchestration_session_id: String, + pub participant_id: String, + pub role: String, + pub orchestrator_participant_id: Option, + pub parent_participant_id: Option, + pub resumed_from_participant_id: String, + pub world: Option, + pub lease_token: String, +} + +impl AgentRuntimeParticipantRecord { pub(crate) fn new( descriptor: &RuntimeSelectionDescriptor, orchestration_session_id: String, - session_handle_id: String, + participant_id: String, lease_token: String, ) -> Self { - let now = Utc::now(); - Self { - handle: AgentRuntimeSessionHandle { - session_handle_id, + Self::new_orchestrator_participant( + descriptor, + orchestration_session_id, + participant_id, + lease_token, + ) + .expect("orchestrator participants must satisfy model invariants") + } + + pub(crate) fn new_orchestrator_participant( + descriptor: &RuntimeSelectionDescriptor, + orchestration_session_id: String, + participant_id: String, + lease_token: String, + ) -> anyhow::Result { + Self::build_participant( + descriptor, + AgentRuntimeParticipantInit { orchestration_session_id, + participant_id, + role: ORCHESTRATOR_ROLE.to_string(), + lease_token, + lineage: AgentRuntimeParticipantLineage { + parent_participant_id: None, + resumed_from_participant_id: None, + orchestrator_participant_id: None, + }, + world: None, + ownership_mode: AgentRuntimeOwnershipMode::AttachedControl, + }, + ) + } + + #[allow(dead_code)] + pub(crate) fn new_member_participant( + descriptor: &RuntimeSelectionDescriptor, + orchestration_session_id: String, + participant_id: String, + orchestrator_participant_id: String, + parent_participant_id: Option, + world: Option, + lease_token: String, + ) -> anyhow::Result { + Self::build_participant( + descriptor, + AgentRuntimeParticipantInit { + orchestration_session_id, + participant_id, + role: MEMBER_ROLE.to_string(), + lease_token, + lineage: AgentRuntimeParticipantLineage { + parent_participant_id, + resumed_from_participant_id: None, + orchestrator_participant_id: Some(orchestrator_participant_id), + }, + world, + ownership_mode: AgentRuntimeOwnershipMode::MemberRuntime, + }, + ) + } + + #[allow(dead_code)] + pub(crate) fn new_replacement_participant( + descriptor: &RuntimeSelectionDescriptor, + init: AgentRuntimeReplacementParticipantInit, + ) -> anyhow::Result { + let ownership_mode = if init.role == ORCHESTRATOR_ROLE { + AgentRuntimeOwnershipMode::AttachedControl + } else { + AgentRuntimeOwnershipMode::MemberRuntime + }; + + Self::build_participant( + descriptor, + AgentRuntimeParticipantInit { + orchestration_session_id: init.orchestration_session_id, + participant_id: init.participant_id, + role: init.role, + lease_token: init.lease_token, + lineage: AgentRuntimeParticipantLineage { + parent_participant_id: init.parent_participant_id, + resumed_from_participant_id: Some(init.resumed_from_participant_id), + orchestrator_participant_id: init.orchestrator_participant_id, + }, + world: init.world, + ownership_mode, + }, + ) + } + + fn build_participant( + descriptor: &RuntimeSelectionDescriptor, + init: AgentRuntimeParticipantInit, + ) -> anyhow::Result { + let now = Utc::now(); + let world_id = init.world.as_ref().map(|binding| binding.world_id.clone()); + let world_generation = init.world.map(|binding| binding.world_generation); + let host_attachment_contract = init.role == ORCHESTRATOR_ROLE + && descriptor.execution_scope == AgentExecutionScope::Host; + let record = Self { + handle: AgentRuntimeParticipantHandle { + session_handle_id: init.participant_id.clone(), + participant_id: init.participant_id, + orchestration_session_id: init.orchestration_session_id, agent_id: descriptor.agent_id.clone(), backend_id: descriptor.backend_id.clone(), - role: ORCHESTRATOR_ROLE.to_string(), + role: init.role, protocol: descriptor.protocol.clone(), - execution: AgentRuntimeSessionExecution { + execution: AgentRuntimeParticipantExecution { scope: descriptor.execution_scope, }, state: AgentRuntimeSessionState::Allocating, opened_at: now, last_transition_at: now, - world_id: None, - world_generation: None, - parent_session_handle_id: None, - resumed_from_session_handle_id: None, + world_id, + world_generation, + parent_session_handle_id: init.lineage.parent_participant_id.clone(), + parent_participant_id: init.lineage.parent_participant_id, + resumed_from_session_handle_id: init.lineage.resumed_from_participant_id.clone(), + resumed_from_participant_id: init.lineage.resumed_from_participant_id, + orchestrator_participant_id: init.lineage.orchestrator_participant_id, }, internal: AgentRuntimeSessionInternal { resolved_agent_kind: descriptor.backend_kind.as_agent_kind_str().to_string(), resolved_binary_path: descriptor.binary_path.display().to_string(), shell_owner_pid: std::process::id(), - lease_token, + lease_token: init.lease_token, uaa_session_id: None, latest_run_id: None, cancel_supported: true, control_owner_retained: false, event_stream_active: false, completion_observer_retained: false, - ownership_mode: Some("attached_control".to_string()), + ownership_mode: Some(init.ownership_mode), ownership_valid: false, ownership_verified_at: None, last_heartbeat_at: Some(now), last_event_at: None, terminal_observed_at: None, termination_reason: None, + attached_client_present: host_attachment_contract, + last_attached_at: host_attachment_contract.then_some(now), + last_detached_at: None, + detach_reason: None, + resume_eligible: host_attachment_contract, last_error_bucket: None, last_error_message: None, }, + }; + record.validate()?; + Ok(record) + } + + pub(crate) fn validate(&self) -> anyhow::Result<()> { + if self.handle.participant_id.trim().is_empty() { + anyhow::bail!("participant_id must not be empty"); } + if self.handle.orchestration_session_id.trim().is_empty() { + anyhow::bail!("orchestration_session_id must not be empty"); + } + + let world_fields_present = + self.handle.world_id.is_some() || self.handle.world_generation.is_some(); + let world_fields_complete = + self.handle.world_id.is_some() && self.handle.world_generation.is_some(); + if world_fields_present && !world_fields_complete { + anyhow::bail!( + "world-scoped participant metadata must include both world_id and world_generation" + ); + } + + match self.handle.execution.scope { + AgentExecutionScope::Host if world_fields_present => { + anyhow::bail!("host-scoped participants must omit world_id and world_generation"); + } + AgentExecutionScope::World if !world_fields_complete => { + anyhow::bail!( + "world-scoped participants must include world_id and world_generation" + ); + } + _ => {} + } + + match self.handle.role.as_str() { + ORCHESTRATOR_ROLE => { + if self.handle.execution.scope != AgentExecutionScope::Host { + anyhow::bail!("orchestrator participants must use execution.scope=host"); + } + if self.handle.orchestrator_participant_id.is_some() { + anyhow::bail!( + "orchestrator participants must omit orchestrator_participant_id" + ); + } + } + MEMBER_ROLE => { + let orchestrator_participant_id = self + .handle + .orchestrator_participant_id + .as_deref() + .filter(|value| !value.trim().is_empty()) + .ok_or_else(|| { + anyhow::anyhow!( + "member participants must include orchestrator_participant_id" + ) + })?; + if orchestrator_participant_id == self.handle.participant_id { + anyhow::bail!( + "member participants must not reuse participant_id as orchestrator_participant_id" + ); + } + } + other => anyhow::bail!("unsupported participant role '{other}'"), + } + + let uses_host_attachment_contract = self.is_host_orchestrator(); + let host_attachment_fields_set = self.internal.attached_client_present + || self.internal.last_attached_at.is_some() + || self.internal.last_detached_at.is_some() + || self.internal.detach_reason.is_some() + || self.internal.resume_eligible; + if !uses_host_attachment_contract && host_attachment_fields_set { + anyhow::bail!( + "non-host participants must not set attached_client_present, resume_eligible, or detach timestamps" + ); + } + if uses_host_attachment_contract { + if self.internal.attached_client_present && self.internal.last_attached_at.is_none() { + anyhow::bail!("attached host participants must include internal.last_attached_at"); + } + if self.internal.attached_client_present && !self.internal.resume_eligible { + anyhow::bail!("attached host participants must remain resume_eligible"); + } + if self.internal.resume_eligible && self.internal.last_attached_at.is_none() { + anyhow::bail!( + "resume-eligible host participants must record internal.last_attached_at" + ); + } + if self.internal.terminal_observed_at.is_some() && self.internal.resume_eligible { + anyhow::bail!("terminal host participants must clear resume_eligible"); + } + } + + if self.handle.parent_participant_id.as_deref() == Some(self.handle.participant_id.as_str()) + { + anyhow::bail!("parent_participant_id must not point to the participant itself"); + } + if self.handle.resumed_from_participant_id.as_deref() + == Some(self.handle.participant_id.as_str()) + { + anyhow::bail!("resumed_from_participant_id must not point to the participant itself"); + } + + Ok(()) } pub(crate) fn transition_state(&mut self, next: AgentRuntimeSessionState) { @@ -158,6 +475,16 @@ impl AgentRuntimeSessionManifest { self.handle.last_transition_at = Utc::now(); } + pub(crate) fn invalidate_for_world_generation_rollover(&mut self) -> bool { + if self.handle.state == AgentRuntimeSessionState::Invalidated { + return false; + } + + self.mark_terminal_state("world generation invalidated by replacement binding"); + self.transition_state(AgentRuntimeSessionState::Invalidated); + true + } + pub(crate) fn touch_heartbeat(&mut self) { self.internal.last_heartbeat_at = Some(Utc::now()); } @@ -166,8 +493,34 @@ impl AgentRuntimeSessionManifest { self.internal.last_event_at = Some(ts); } - pub(crate) fn set_uaa_session_id(&mut self, session_id: impl Into) { - self.internal.uaa_session_id = Some(session_id.into()); + pub(crate) fn participant_id(&self) -> &str { + &self.handle.participant_id + } + + pub(crate) fn internal_uaa_session_id(&self) -> Option<&str> { + self.internal.uaa_session_id.as_deref() + } + + pub(crate) fn is_host_orchestrator(&self) -> bool { + self.handle.role == ORCHESTRATOR_ROLE + && self.handle.execution.scope == AgentExecutionScope::Host + } + + pub(crate) fn matches_public_parent_linkage( + &self, + session: &OrchestrationSessionRecord, + ) -> bool { + self.handle.orchestration_session_id == session.orchestration_session_id + && self.handle.agent_id == session.orchestrator_agent_id + && self.is_host_orchestrator() + } + + pub(crate) fn set_uaa_session_id(&mut self, backend_session_id: impl Into) { + self.internal.uaa_session_id = Some(backend_session_id.into()); + if self.is_host_orchestrator() && self.handle.state.is_live() { + self.internal.resume_eligible = true; + self.internal.last_attached_at.get_or_insert_with(Utc::now); + } self.refresh_ownership_validity(); } @@ -201,6 +554,7 @@ impl AgentRuntimeSessionManifest { self.internal.control_owner_retained = true; self.internal.event_stream_active = true; self.internal.completion_observer_retained = true; + self.mark_client_attached(); self.refresh_ownership_validity(); } @@ -212,19 +566,22 @@ impl AgentRuntimeSessionManifest { } pub(crate) fn mark_terminal_state(&mut self, reason: impl Into) { + let reason = reason.into(); let now = Utc::now(); self.release_runtime_ownership(); - self.internal.terminal_observed_at = Some(Utc::now()); - self.internal.termination_reason = Some(reason.into()); + self.internal.terminal_observed_at = Some(now); + self.internal.termination_reason = Some(reason.clone()); + if self.is_host_orchestrator() { + self.internal.attached_client_present = false; + self.internal.resume_eligible = false; + self.internal.last_detached_at = Some(now); + self.internal.detach_reason = Some(reason); + } if self.internal.ownership_verified_at.is_none() { self.internal.ownership_verified_at = Some(now); } } - pub(crate) fn mark_ownership_invalid(&mut self, reason: impl Into) { - self.mark_terminal_state(reason); - } - pub(crate) fn has_valid_ownership(&self) -> bool { self.internal.ownership_valid && self.can_advertise_live() } @@ -233,10 +590,409 @@ impl AgentRuntimeSessionManifest { self.handle.state.is_live() && self.has_valid_ownership() } + pub(crate) fn attached_client_present(&self) -> bool { + self.internal.attached_client_present + } + + pub(crate) fn is_resume_eligible(&self) -> bool { + self.internal.resume_eligible + && self.is_host_orchestrator() + && self.handle.state.is_live() + && self.internal.terminal_observed_at.is_none() + } + + #[allow(dead_code)] + pub(crate) fn mark_client_detached(&mut self, reason: impl Into) { + if !self.is_host_orchestrator() { + return; + } + self.internal.attached_client_present = false; + self.internal.last_detached_at = Some(Utc::now()); + self.internal.detach_reason = Some(reason.into()); + self.internal.resume_eligible = self.handle.state.is_live() + && self.internal.terminal_observed_at.is_none() + && self.internal.uaa_session_id.is_some(); + } + pub(crate) fn last_status_at(&self) -> DateTime { self.internal .last_event_at .or(self.internal.last_heartbeat_at) .unwrap_or(self.handle.last_transition_at) } + + fn mark_client_attached(&mut self) { + if !self.is_host_orchestrator() { + return; + } + self.internal.attached_client_present = true; + self.internal.resume_eligible = true; + self.internal.last_attached_at = Some(Utc::now()); + self.internal.detach_reason = None; + } +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub(crate) struct AgentRuntimeParticipantWorldBinding { + pub world_id: String, + pub world_generation: u64, +} + +#[derive(Deserialize)] +struct AgentRuntimeParticipantRecordWire { + #[serde(flatten)] + handle: AgentRuntimeParticipantHandleWire, + internal: AgentRuntimeSessionInternal, +} + +#[derive(Deserialize)] +struct AgentRuntimeParticipantHandleWire { + // Legacy alias reads remain supported for compatibility; participant_id is canonical. + #[serde(alias = "session_handle_id")] + participant_id: String, + orchestration_session_id: String, + agent_id: String, + backend_id: String, + role: String, + protocol: String, + execution: AgentRuntimeParticipantExecutionWire, + state: AgentRuntimeSessionState, + opened_at: DateTime, + last_transition_at: DateTime, + #[serde(default)] + world_id: Option, + #[serde(default)] + world_generation: Option, + #[serde(default, alias = "parent_session_handle_id")] + parent_participant_id: Option, + #[serde(default, alias = "resumed_from_session_handle_id")] + resumed_from_participant_id: Option, + #[serde(default)] + orchestrator_participant_id: Option, +} + +#[derive(Deserialize)] +struct AgentRuntimeParticipantExecutionWire { + scope: AgentExecutionScope, +} + +#[allow(dead_code)] +pub(crate) type AgentRuntimeSessionExecution = AgentRuntimeParticipantExecution; +#[allow(dead_code)] +pub(crate) type AgentRuntimeSessionHandle = AgentRuntimeParticipantHandle; +pub(crate) type AgentRuntimeSessionManifest = AgentRuntimeParticipantRecord; + +#[cfg(test)] +mod tests { + use std::path::PathBuf; + + use serde_json::json; + + use super::*; + use crate::execution::agent_runtime::mapping::AgentRuntimeBackendKind; + + fn descriptor(scope: AgentExecutionScope) -> RuntimeSelectionDescriptor { + RuntimeSelectionDescriptor { + agent_id: "codex".to_string(), + backend_id: "cli:codex".to_string(), + backend_kind: AgentRuntimeBackendKind::Codex, + protocol: "uaa.agent.session".to_string(), + execution_scope: scope, + binary_path: PathBuf::from("/usr/bin/codex"), + } + } + + #[test] + fn orchestrator_participant_constructor() { + let participant = AgentRuntimeParticipantRecord::new_orchestrator_participant( + &descriptor(AgentExecutionScope::Host), + "sess_001".to_string(), + "ash_001".to_string(), + "lease_001".to_string(), + ) + .expect("orchestrator constructor should succeed"); + + assert_eq!(participant.handle.participant_id, "ash_001"); + assert_eq!(participant.handle.session_handle_id, "ash_001"); + assert_eq!(participant.handle.role, ORCHESTRATOR_ROLE); + assert_eq!( + participant.handle.execution.scope, + AgentExecutionScope::Host + ); + assert!(participant.handle.world_id.is_none()); + assert!(participant.handle.world_generation.is_none()); + assert!(participant.handle.orchestrator_participant_id.is_none()); + assert_eq!( + participant.internal.ownership_mode, + Some(AgentRuntimeOwnershipMode::AttachedControl) + ); + } + + #[test] + fn member_participant_constructor() { + let participant = AgentRuntimeParticipantRecord::new_member_participant( + &descriptor(AgentExecutionScope::World), + "sess_001".to_string(), + "ash_002".to_string(), + "ash_orchestrator".to_string(), + Some("ash_parent".to_string()), + Some(AgentRuntimeParticipantWorldBinding { + world_id: "world-17".to_string(), + world_generation: 3, + }), + "lease_002".to_string(), + ) + .expect("member constructor should succeed"); + + assert_eq!(participant.handle.participant_id, "ash_002"); + assert_eq!(participant.handle.role, MEMBER_ROLE); + assert_eq!( + participant.handle.orchestrator_participant_id.as_deref(), + Some("ash_orchestrator") + ); + assert_eq!( + participant.handle.parent_participant_id.as_deref(), + Some("ash_parent") + ); + assert_eq!(participant.handle.world_id.as_deref(), Some("world-17")); + assert_eq!(participant.handle.world_generation, Some(3)); + assert_eq!( + participant.internal.ownership_mode, + Some(AgentRuntimeOwnershipMode::MemberRuntime) + ); + } + + #[test] + fn replacement_participant_constructor() { + let participant = AgentRuntimeParticipantRecord::new_replacement_participant( + &descriptor(AgentExecutionScope::World), + AgentRuntimeReplacementParticipantInit { + orchestration_session_id: "sess_001".to_string(), + participant_id: "ash_003".to_string(), + role: MEMBER_ROLE.to_string(), + orchestrator_participant_id: Some("ash_orchestrator".to_string()), + parent_participant_id: Some("ash_parent".to_string()), + resumed_from_participant_id: "ash_002".to_string(), + world: Some(AgentRuntimeParticipantWorldBinding { + world_id: "world-18".to_string(), + world_generation: 4, + }), + lease_token: "lease_003".to_string(), + }, + ) + .expect("replacement constructor should succeed"); + + assert_eq!( + participant.handle.resumed_from_participant_id.as_deref(), + Some("ash_002") + ); + assert_eq!( + participant.handle.resumed_from_session_handle_id.as_deref(), + Some("ash_002") + ); + assert_eq!( + participant.internal.ownership_mode, + Some(AgentRuntimeOwnershipMode::MemberRuntime) + ); + } + + #[test] + fn invalidation_helper_marks_world_generation_rollover_tombstone() { + let mut participant = AgentRuntimeParticipantRecord::new_member_participant( + &descriptor(AgentExecutionScope::World), + "sess_001".to_string(), + "ash_002".to_string(), + "ash_orchestrator".to_string(), + None, + Some(AgentRuntimeParticipantWorldBinding { + world_id: "world-17".to_string(), + world_generation: 3, + }), + "lease_002".to_string(), + ) + .expect("member constructor should succeed"); + participant.mark_runtime_ownership_retained(); + participant.set_uaa_session_id("uaa_session"); + participant.transition_state(AgentRuntimeSessionState::Running); + + let changed = participant.invalidate_for_world_generation_rollover(); + + assert!(changed, "invalidation helper must report first transition"); + assert_eq!( + participant.handle.state, + AgentRuntimeSessionState::Invalidated + ); + assert!(!participant.internal.ownership_valid); + assert!(!participant.can_advertise_live()); + assert_eq!( + participant.internal.termination_reason.as_deref(), + Some("world generation invalidated by replacement binding") + ); + assert!( + participant.internal.terminal_observed_at.is_some(), + "invalidation should stamp terminal metadata" + ); + } + + #[test] + fn invalidation_helper_is_idempotent() { + let mut participant = AgentRuntimeParticipantRecord::new_member_participant( + &descriptor(AgentExecutionScope::World), + "sess_001".to_string(), + "ash_002".to_string(), + "ash_orchestrator".to_string(), + None, + Some(AgentRuntimeParticipantWorldBinding { + world_id: "world-17".to_string(), + world_generation: 3, + }), + "lease_002".to_string(), + ) + .expect("member constructor should succeed"); + + assert!(participant.invalidate_for_world_generation_rollover()); + let first_transition_at = participant.handle.last_transition_at; + let first_terminal_observed_at = participant.internal.terminal_observed_at; + + assert!( + !participant.invalidate_for_world_generation_rollover(), + "already invalidated participants must remain untouched" + ); + assert_eq!(participant.handle.last_transition_at, first_transition_at); + assert_eq!( + participant.internal.terminal_observed_at, + first_terminal_observed_at + ); + } + + #[test] + fn host_participant_constructor_tracks_attachment_fields() { + let participant = AgentRuntimeParticipantRecord::new_orchestrator_participant( + &descriptor(AgentExecutionScope::Host), + "sess_001".to_string(), + "ash_001".to_string(), + "lease_001".to_string(), + ) + .expect("host participant"); + + assert!(participant.attached_client_present()); + assert!(participant.is_resume_eligible()); + assert!(participant.internal.last_attached_at.is_some()); + } + + #[test] + fn member_participant_rejects_host_attachment_fields() { + let mut participant = AgentRuntimeParticipantRecord::new_member_participant( + &descriptor(AgentExecutionScope::World), + "sess_001".to_string(), + "member_001".to_string(), + "ash_001".to_string(), + None, + Some(AgentRuntimeParticipantWorldBinding { + world_id: "world-17".to_string(), + world_generation: 2, + }), + "lease_001".to_string(), + ) + .expect("member participant"); + + participant.internal.resume_eligible = true; + let err = participant + .validate() + .expect_err("attachment contract error"); + assert!(err + .to_string() + .contains("non-host participants must not set attached_client_present")); + } + + #[test] + fn legacy_handle_upgrade() { + let participant: AgentRuntimeParticipantRecord = serde_json::from_value(json!({ + "session_handle_id": "ash_legacy", + "orchestration_session_id": "sess_legacy", + "agent_id": "codex", + "backend_id": "cli:codex", + "role": "orchestrator", + "protocol": "uaa.agent.session", + "execution": { "scope": "host" }, + "state": "allocating", + "opened_at": "2026-04-24T18:30:00Z", + "last_transition_at": "2026-04-24T18:30:00Z", + "parent_session_handle_id": "ash_parent", + "resumed_from_session_handle_id": "ash_prev", + "internal": { + "resolved_agent_kind": "codex", + "resolved_binary_path": "/usr/bin/codex", + "shell_owner_pid": 42, + "lease_token": "lease_legacy", + "cancel_supported": true, + "control_owner_retained": false, + "event_stream_active": false, + "completion_observer_retained": false, + "ownership_mode": "attached_control", + "ownership_valid": false + } + })) + .expect("legacy handle JSON should deserialize"); + + assert_eq!(participant.handle.participant_id, "ash_legacy"); + assert_eq!(participant.handle.session_handle_id, "ash_legacy"); + assert_eq!( + participant.handle.parent_participant_id.as_deref(), + Some("ash_parent") + ); + assert_eq!( + participant.handle.resumed_from_participant_id.as_deref(), + Some("ash_prev") + ); + assert!(participant.handle.orchestrator_participant_id.is_none()); + } + + #[test] + fn centralized_role_scope_world_validation() { + let err = AgentRuntimeParticipantRecord::new_member_participant( + &descriptor(AgentExecutionScope::World), + "sess_001".to_string(), + "ash_invalid".to_string(), + "ash_orchestrator".to_string(), + None, + None, + "lease_invalid".to_string(), + ) + .expect_err("world-scoped members without world binding must be rejected"); + assert!(err + .to_string() + .contains("world-scoped participants must include world_id and world_generation")); + + let err = serde_json::from_value::(json!({ + "participant_id": "ash_invalid", + "orchestration_session_id": "sess_001", + "agent_id": "codex", + "backend_id": "cli:codex", + "role": "orchestrator", + "protocol": "uaa.agent.session", + "execution": { "scope": "world" }, + "state": "allocating", + "opened_at": "2026-04-24T18:30:00Z", + "last_transition_at": "2026-04-24T18:30:00Z", + "world_id": "world-17", + "world_generation": 3, + "internal": { + "resolved_agent_kind": "codex", + "resolved_binary_path": "/usr/bin/codex", + "shell_owner_pid": 42, + "lease_token": "lease_invalid", + "cancel_supported": true, + "control_owner_retained": false, + "event_stream_active": false, + "completion_observer_retained": false, + "ownership_mode": "attached_control", + "ownership_valid": false + } + })) + .expect_err("orchestrator world rows must be rejected"); + assert!(err + .to_string() + .contains("orchestrator participants must use execution.scope=host")); + } } diff --git a/crates/shell/src/execution/agent_runtime/state_store.rs b/crates/shell/src/execution/agent_runtime/state_store.rs index 78d600b03..92756a366 100644 --- a/crates/shell/src/execution/agent_runtime/state_store.rs +++ b/crates/shell/src/execution/agent_runtime/state_store.rs @@ -1,13 +1,316 @@ +use std::cmp::Ordering; +use std::collections::{BTreeMap, BTreeSet}; use std::fs; use std::io; use std::path::{Path, PathBuf}; +use std::sync::{Mutex, OnceLock}; use anyhow::{Context, Result}; +use chrono::{DateTime, Utc}; +use serde::{Deserialize, Serialize}; use tempfile::NamedTempFile; use substrate_common::paths as substrate_paths; -use super::session::AgentRuntimeSessionManifest; +use crate::execution::config_model::AgentExecutionScope; + +use super::{ + control::PublicSessionPosture, + mapping::{MEMBER_ROLE, ORCHESTRATOR_ROLE}, + orchestration_session::{ + OrchestrationSessionPosture, OrchestrationSessionRecord, OrchestrationSessionState, + StartupPromptStreamState, + }, + session::{AgentRuntimeParticipantRecord, AgentRuntimeSessionManifest}, +}; + +#[derive(Clone, Debug)] +pub(crate) struct AgentRuntimeSessionRecord { + pub session: OrchestrationSessionRecord, + pub participants: Vec, + #[allow(dead_code)] + pub warnings: Vec, + has_authoritative_parent: bool, + #[allow(dead_code)] + complete: bool, +} + +impl AgentRuntimeSessionRecord { + pub(crate) fn orchestration_session_id(&self) -> &str { + &self.session.orchestration_session_id + } + + #[allow(dead_code)] + pub(crate) fn is_complete(&self) -> bool { + self.complete + } + + pub(crate) fn live_participants(&self) -> Vec { + self.participants + .iter() + .filter(|participant| { + participant.is_authoritative_live() && owner_process_is_alive(participant) + }) + .cloned() + .collect() + } + + pub(crate) fn live_orchestrator(&self) -> Option { + let active_participant_id = self.session.active_participant_id()?; + self.live_participants().into_iter().find(|participant| { + participant.participant_id() == active_participant_id + && participant.matches_public_parent_linkage(&self.session) + }) + } + + pub(crate) fn status_visible_participants(&self) -> Vec { + let mut participants = self.live_participants(); + if let Ok(Some(detached_participant)) = detached_status_visible_participant(self) { + if participants.iter().all(|participant| { + participant.handle.participant_id != detached_participant.handle.participant_id + }) { + participants.push(detached_participant); + } + } + participants.sort_by(|left, right| { + left.handle + .last_transition_at + .cmp(&right.handle.last_transition_at) + .then(left.handle.participant_id.cmp(&right.handle.participant_id)) + }); + participants + } + + #[allow(dead_code)] + pub(crate) fn live_participant_for_agent( + &self, + agent_id: &str, + scope: AgentExecutionScope, + role: &str, + ) -> Option { + self.live_participants().into_iter().find(|participant| { + participant.handle.agent_id == agent_id + && participant.handle.execution.scope == scope + && participant.handle.role == role + }) + } + + #[allow(dead_code)] + pub(crate) fn invalidated_world_members(&self) -> Vec { + self.participants + .iter() + .filter(|participant| { + participant.handle.role == MEMBER_ROLE + && participant.handle.execution.scope == AgentExecutionScope::World + && participant.handle.state + == super::session::AgentRuntimeSessionState::Invalidated + }) + .cloned() + .collect() + } + + pub(crate) fn last_updated_at(&self) -> DateTime { + self.participants + .iter() + .map(AgentRuntimeParticipantRecord::last_status_at) + .max() + .map_or(self.session.last_active_at, |participant_ts| { + participant_ts.max(self.session.last_active_at) + }) + } + + fn has_authoritative_parent(&self) -> bool { + self.has_authoritative_parent + } +} + +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub(crate) enum HiddenOwnerHelperLaunchReadiness { + Pending, + ReadyAttached, + ReadyDetached(OrchestrationSessionPosture), +} + +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub(crate) enum HiddenOwnerHelperLaunchContinuity { + Pending, + AttachedLive, + DetachedReconciled(OrchestrationSessionPosture), + StaleAttachedTruth, +} + +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub(crate) enum StartupPromptReplayState { + NotTracked, + PendingAcceptance, + AcceptedOrTerminal, +} + +impl StartupPromptReplayState { + #[cfg(unix)] + pub(crate) fn replay_safe(self) -> bool { + matches!(self, Self::NotTracked | Self::PendingAcceptance) + } +} + +#[derive(Clone, Copy, Debug, Deserialize, Eq, PartialEq, Serialize)] +#[serde(rename_all = "snake_case")] +pub(crate) enum DurableInboxItemKind { + ApprovalRequired, + CompletionNotice, + FollowUpMessage, + RuntimeAlert, +} + +#[derive(Clone, Copy, Debug, Deserialize, Eq, PartialEq, Serialize)] +#[serde(rename_all = "snake_case")] +pub(crate) enum DurableInboxItemState { + Pending, + Acknowledged, + Dismissed, +} + +impl DurableInboxItemState { + fn is_pending(self) -> bool { + matches!(self, Self::Pending) + } +} + +#[derive(Clone, Debug, Deserialize, Eq, PartialEq, Serialize)] +pub(crate) struct DurableInboxItemRecord { + pub orchestration_session_id: String, + pub item_id: String, + pub kind: DurableInboxItemKind, + pub state: DurableInboxItemState, + pub created_at: DateTime, + pub updated_at: DateTime, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub resolved_at: Option>, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub message: Option, +} + +impl DurableInboxItemRecord { + #[allow(dead_code)] + pub(crate) fn new( + orchestration_session_id: impl Into, + item_id: impl Into, + kind: DurableInboxItemKind, + message: Option, + ) -> Self { + let now = Utc::now(); + Self { + orchestration_session_id: orchestration_session_id.into(), + item_id: item_id.into(), + kind, + state: DurableInboxItemState::Pending, + created_at: now, + updated_at: now, + resolved_at: None, + message, + } + } + + fn validate(&self) -> Result<()> { + if self.orchestration_session_id.trim().is_empty() { + anyhow::bail!("durable inbox item must include orchestration_session_id"); + } + if self.item_id.trim().is_empty() { + anyhow::bail!("durable inbox item must include item_id"); + } + if self.state.is_pending() && self.resolved_at.is_some() { + anyhow::bail!("pending durable inbox items must not include resolved_at"); + } + if !self.state.is_pending() && self.resolved_at.is_none() { + anyhow::bail!("resolved durable inbox items must include resolved_at"); + } + + Ok(()) + } + + fn is_pending(&self) -> bool { + self.state.is_pending() + } + + fn transition_state(&mut self, state: DurableInboxItemState) { + let now = Utc::now(); + self.state = state; + self.updated_at = now; + if state.is_pending() { + self.resolved_at = None; + } else { + self.resolved_at = Some(now); + } + } +} + +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub(crate) enum PublicControlAction { + Resume, + Fork, + Stop, +} + +impl PublicControlAction { + fn requires_internal_session_id(self) -> bool { + matches!(self, Self::Resume | Self::Fork) + } + + fn rejects_live_owner(self) -> bool { + matches!(self, Self::Resume) + } +} + +#[derive(Clone, Debug)] +pub(crate) struct ResolvedPublicControlTarget { + pub session: OrchestrationSessionRecord, + pub active_participant: AgentRuntimeParticipantRecord, + pub session_posture: PublicSessionPosture, +} + +impl ResolvedPublicControlTarget { + #[allow(dead_code)] + pub(crate) fn orchestration_session_id(&self) -> &str { + &self.session.orchestration_session_id + } +} + +#[allow(dead_code)] +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub(crate) enum PublicTurnTargetKind { + Host, + World, +} + +#[allow(dead_code)] +#[derive(Clone, Debug)] +pub(crate) struct ResolvedPublicTurnTarget { + pub session: OrchestrationSessionRecord, + pub participant: AgentRuntimeParticipantRecord, + pub target_kind: PublicTurnTargetKind, + pub session_posture: PublicSessionPosture, +} + +impl ResolvedPublicTurnTarget { + #[allow(dead_code)] + pub(crate) fn orchestration_session_id(&self) -> &str { + &self.session.orchestration_session_id + } +} + +#[derive(Clone, Copy, Debug, Eq, Ord, PartialEq, PartialOrd)] +enum ParticipantRecordSource { + Canonical, + Flat, + Legacy, +} + +#[derive(Clone, Debug)] +struct ResolvedAuthoritativeSessionControl { + session: OrchestrationSessionRecord, + participant: AgentRuntimeParticipantRecord, + session_posture: PublicSessionPosture, +} #[derive(Clone, Debug)] pub(crate) struct AgentRuntimeStateStore { @@ -21,6 +324,13 @@ impl AgentRuntimeStateStore { }) } + pub(crate) fn participants_dir(&self) -> PathBuf { + self.substrate_home + .join("run") + .join("agent-hub") + .join("participants") + } + pub(crate) fn handles_dir(&self) -> PathBuf { self.substrate_home .join("run") @@ -28,145 +338,4002 @@ impl AgentRuntimeStateStore { .join("handles") } - fn ensure_dirs(&self) -> Result<()> { - fs::create_dir_all(self.handles_dir()) - .with_context(|| format!("failed to create {}", self.handles_dir().display())) + pub(crate) fn sessions_dir(&self) -> PathBuf { + self.substrate_home + .join("run") + .join("agent-hub") + .join("sessions") + } + + fn canonical_session_dir(&self, orchestration_session_id: &str) -> PathBuf { + self.sessions_dir().join(orchestration_session_id) } - fn manifest_path(&self, session_handle_id: &str) -> PathBuf { - self.handles_dir().join(format!("{session_handle_id}.json")) + fn canonical_session_path(&self, orchestration_session_id: &str) -> PathBuf { + self.canonical_session_dir(orchestration_session_id) + .join("session.json") } - fn lease_path(&self, session_handle_id: &str) -> PathBuf { - self.handles_dir() - .join(format!("{session_handle_id}.lease")) + fn canonical_participants_dir(&self, orchestration_session_id: &str) -> PathBuf { + self.canonical_session_dir(orchestration_session_id) + .join("participants") } - pub(crate) fn persist_manifest(&self, manifest: &AgentRuntimeSessionManifest) -> Result<()> { - self.ensure_dirs()?; + fn canonical_participant_path( + &self, + orchestration_session_id: &str, + participant_id: &str, + ) -> PathBuf { + self.canonical_participants_dir(orchestration_session_id) + .join(format!("{participant_id}.json")) + } + + fn canonical_leases_dir(&self, orchestration_session_id: &str) -> PathBuf { + self.canonical_session_dir(orchestration_session_id) + .join("leases") + } + + #[allow(dead_code)] + pub(crate) fn canonical_inbox_dir(&self, orchestration_session_id: &str) -> PathBuf { + self.canonical_session_dir(orchestration_session_id) + .join("inbox") + } + + #[allow(dead_code)] + pub(crate) fn canonical_inbox_item_path( + &self, + orchestration_session_id: &str, + item_id: &str, + ) -> PathBuf { + self.canonical_inbox_dir(orchestration_session_id) + .join(format!("{item_id}.json")) + } + + fn canonical_lease_path( + &self, + orchestration_session_id: &str, + participant_id: &str, + ) -> PathBuf { + self.canonical_leases_dir(orchestration_session_id) + .join(format!("{participant_id}.lease")) + } + + fn ensure_participants_dir(&self) -> Result<()> { + fs::create_dir_all(self.participants_dir()) + .with_context(|| format!("failed to create {}", self.participants_dir().display())) + } + + fn ensure_sessions_dir(&self) -> Result<()> { + fs::create_dir_all(self.sessions_dir()) + .with_context(|| format!("failed to create {}", self.sessions_dir().display())) + } + + fn participant_path(&self, participant_id: &str) -> PathBuf { + self.participants_dir() + .join(format!("{participant_id}.json")) + } + + fn orchestration_session_path(&self, orchestration_session_id: &str) -> PathBuf { + self.sessions_dir() + .join(format!("{orchestration_session_id}.json")) + } + + fn lease_path(&self, participant_id: &str) -> PathBuf { + self.participants_dir() + .join(format!("{participant_id}.lease")) + } + + pub(crate) fn persist_participant( + &self, + participant: &AgentRuntimeParticipantRecord, + ) -> Result<()> { + let _write_guard = snapshot_write_lock() + .lock() + .expect("snapshot write mutex poisoned"); + self.validate_participant_record(participant)?; + if let Some(existing) = self.load_participant(&participant.handle.participant_id)? { + if !should_persist_participant_snapshot(&existing, participant) { + return Ok(()); + } + } + self.ensure_participants_dir()?; + write_atomic_json( + &self.participant_path(&participant.handle.participant_id), + participant, + )?; write_atomic_json( - &self.manifest_path(&manifest.handle.session_handle_id), - manifest, + &self.canonical_participant_path( + &participant.handle.orchestration_session_id, + &participant.handle.participant_id, + ), + participant, )?; - self.persist_lease(manifest) + self.persist_lease(participant) } - fn persist_lease(&self, manifest: &AgentRuntimeSessionManifest) -> Result<()> { - let payload = serde_json::json!({ - "session_handle_id": manifest.handle.session_handle_id, - "shell_owner_pid": manifest.internal.shell_owner_pid, - "lease_token": manifest.internal.lease_token, - "state": manifest.handle.state, - "ownership_valid": manifest.internal.ownership_valid, - "last_heartbeat_at": manifest.internal.last_heartbeat_at, - "terminal_observed_at": manifest.internal.terminal_observed_at, - }); - write_atomic_json( - &self.lease_path(&manifest.handle.session_handle_id), - &payload, - ) + pub(crate) fn load_participant( + &self, + participant_id: &str, + ) -> Result> { + Ok(self + .list_participants_across_sources()? + .into_iter() + .find(|participant| participant.handle.participant_id == participant_id)) } - pub(crate) fn list_manifests(&self) -> Result> { - let dir = self.handles_dir(); - let entries = match fs::read_dir(&dir) { - Ok(entries) => entries, - Err(err) if err.kind() == io::ErrorKind::NotFound => return Ok(Vec::new()), - Err(err) => { - return Err(err).with_context(|| format!("failed to read {}", dir.display())); - } - }; + pub(crate) fn list_participants(&self) -> Result> { + self.list_participants_across_sources() + } - let mut manifests = Vec::new(); - for entry in entries { - let entry = entry.with_context(|| format!("failed to read {}", dir.display()))?; - let path = entry.path(); - if path.extension().and_then(|value| value.to_str()) != Some("json") { - continue; - } - let raw = fs::read_to_string(&path) - .with_context(|| format!("failed to read {}", path.display()))?; - let manifest: AgentRuntimeSessionManifest = serde_json::from_str(&raw) - .with_context(|| format!("failed to parse {}", path.display()))?; - manifests.push(manifest); + pub(crate) fn list_live_participants(&self) -> Result> { + Ok(self + .list_participants_across_sources()? + .into_iter() + .filter(|participant| { + participant.is_authoritative_live() && owner_process_is_alive(participant) + }) + .collect()) + } + + #[allow(dead_code)] + pub(crate) fn list_invalidated_participants( + &self, + ) -> Result> { + Ok(self + .read_participant_dir(&self.participants_dir())? + .into_iter() + .filter(|participant| { + participant.handle.state == super::session::AgentRuntimeSessionState::Invalidated + }) + .collect()) + } + + pub(crate) fn list_participants_across_sources( + &self, + ) -> Result> { + let mut participants = BTreeMap::new(); + + for (participant, source) in self.read_canonical_participants()? { + participants.insert( + participant.handle.participant_id.clone(), + (participant, source), + ); + } + + for participant in self.read_participant_dir(&self.participants_dir())? { + participants + .entry(participant.handle.participant_id.clone()) + .or_insert((participant, ParticipantRecordSource::Flat)); + } + + for participant in self.read_participant_dir(&self.handles_dir())? { + participants + .entry(participant.handle.participant_id.clone()) + .or_insert((participant, ParticipantRecordSource::Legacy)); } - manifests.sort_by(|left, right| { + let mut participants = participants + .into_values() + .map(|(participant, _)| participant) + .collect::>(); + participants.sort_by(|left, right| { left.handle .last_transition_at .cmp(&right.handle.last_transition_at) + .then(left.handle.participant_id.cmp(&right.handle.participant_id)) }); - Ok(manifests) + Ok(participants) } - pub(crate) fn list_live_manifests(&self) -> Result> { + pub(crate) fn list_invalidated_participants_across_sources( + &self, + ) -> Result> { Ok(self - .list_manifests()? + .list_participants_across_sources()? .into_iter() - // A persisted live manifest is only authoritative while the REPL still owns the - // retained control boundary and the original shell owner process is still alive. - .filter(|manifest| manifest.is_authoritative_live() && owner_process_is_alive(manifest)) + .filter(|participant| { + participant.handle.state == super::session::AgentRuntimeSessionState::Invalidated + }) .collect()) } - pub(crate) fn find_live_orchestrator( + #[allow(dead_code)] + pub(crate) fn list_live_participants_for_session( &self, - agent_id: &str, - ) -> Result> { - let mut latest: Option = None; - for manifest in self.list_live_manifests()? { - if manifest.handle.agent_id != agent_id { + orchestration_session_id: &str, + ) -> Result> { + Ok(self + .list_live_participants()? + .into_iter() + .filter(|participant| { + participant.handle.orchestration_session_id == orchestration_session_id + }) + .collect()) + } + + pub(crate) fn invalidate_stale_world_members_for_session( + &self, + orchestration_session_id: &str, + active_generation: u64, + ) -> Result> { + let mut invalidated_participant_ids = Vec::new(); + + for mut participant in self.list_participants_across_sources()? { + if participant.handle.orchestration_session_id != orchestration_session_id + || participant.handle.role != MEMBER_ROLE + || participant.handle.execution.scope != AgentExecutionScope::World + || !participant.is_authoritative_live() + || !owner_process_is_alive(&participant) + { continue; } - let replace = match &latest { - Some(existing) => { - manifest.handle.last_transition_at >= existing.handle.last_transition_at - } - None => true, + + let Some(world_generation) = participant.handle.world_generation else { + continue; }; - if replace { - latest = Some(manifest); + if world_generation >= active_generation { + continue; + } + + if participant.invalidate_for_world_generation_rollover() { + invalidated_participant_ids.push(participant.handle.participant_id.clone()); + self.persist_participant(&participant)?; } } - Ok(latest) + + Ok(invalidated_participant_ids) } -} -fn write_atomic_json(path: &Path, value: &impl serde::Serialize) -> Result<()> { - let parent = path - .parent() - .ok_or_else(|| anyhow::anyhow!("{} has no parent directory", path.display()))?; - fs::create_dir_all(parent).with_context(|| format!("failed to create {}", parent.display()))?; + pub(crate) fn load_session( + &self, + orchestration_session_id: &str, + ) -> Result> { + let session = self.load_authoritative_session(orchestration_session_id)?; + let participants = self + .list_participants_across_sources()? + .into_iter() + .filter(|participant| { + participant.handle.orchestration_session_id == orchestration_session_id + }) + .collect::>(); + if session.is_none() && participants.is_empty() { + return Ok(None); + } - let mut tmp = NamedTempFile::new_in(parent) - .with_context(|| format!("failed to create temp file in {}", parent.display()))?; - serde_json::to_writer_pretty(tmp.as_file_mut(), value) - .with_context(|| format!("failed to serialize {}", path.display()))?; - tmp.as_file_mut() - .sync_all() - .with_context(|| format!("failed to flush {}", path.display()))?; - tmp.persist(path) - .map_err(|err| err.error) - .with_context(|| format!("failed to persist {}", path.display()))?; - Ok(()) -} + Ok(Some(self.build_session_record( + orchestration_session_id, + session, + participants, + ))) + } -#[cfg(unix)] -fn owner_process_is_alive(manifest: &AgentRuntimeSessionManifest) -> bool { - let pid = manifest.internal.shell_owner_pid as libc::pid_t; - if pid <= 0 { - return false; + pub(crate) fn list_sessions(&self) -> Result> { + let mut session_ids = BTreeSet::new(); + + for session_id in self.canonical_session_root_ids()? { + session_ids.insert(session_id); + } + for session_id in self.flat_session_ids()? { + session_ids.insert(session_id); + } + for participant in self.list_participants_across_sources()? { + session_ids.insert(participant.handle.orchestration_session_id.clone()); + } + + let mut sessions = Vec::new(); + for session_id in session_ids { + if let Some(record) = self.load_session(&session_id)? { + sessions.push(record); + } + } + + sessions.sort_by(|left, right| { + left.last_updated_at().cmp(&right.last_updated_at()).then( + left.orchestration_session_id() + .cmp(right.orchestration_session_id()), + ) + }); + Ok(sessions) } - let rc = unsafe { libc::kill(pid, 0) }; - if rc == 0 { - return true; + pub(crate) fn list_status_sessions_for_agent( + &self, + orchestrator_agent_id: &str, + ) -> Result> { + let _ = orchestrator_agent_id; + self.list_sessions() } - matches!(io::Error::last_os_error().raw_os_error(), Some(libc::EPERM)) -} + #[allow(dead_code)] + pub(crate) fn list_live_sessions(&self) -> Result> { + Ok(self + .list_sessions()? + .into_iter() + .filter(|record| { + record.is_complete() + && record.session.state == OrchestrationSessionState::Active + && owner_pid_is_alive(record.session.shell_owner_pid) + }) + .collect()) + } -#[cfg(not(unix))] -fn owner_process_is_alive(manifest: &AgentRuntimeSessionManifest) -> bool { - manifest.internal.shell_owner_pid == std::process::id() + pub(crate) fn resolve_single_live_session_for_agent( + &self, + orchestrator_agent_id: &str, + ) -> Result> { + let active_candidates = self + .list_sessions()? + .into_iter() + .filter(|record| { + record.has_authoritative_parent() + && record.session.orchestrator_agent_id == orchestrator_agent_id + && record.session.state == OrchestrationSessionState::Active + && owner_pid_is_alive(record.session.shell_owner_pid) + }) + .collect::>(); + if active_candidates.len() > 1 { + anyhow::bail!( + "multiple active orchestration session candidates found for agent {orchestrator_agent_id}" + ); + } + + let live_host_orchestrators = self + .list_live_participants()? + .into_iter() + .filter(|participant| { + participant.handle.agent_id == orchestrator_agent_id + && participant.handle.role == ORCHESTRATOR_ROLE + && participant.handle.execution.scope == AgentExecutionScope::Host + }) + .collect::>(); + if live_host_orchestrators.len() > 1 { + anyhow::bail!( + "multiple live orchestrator participant candidates found for agent {orchestrator_agent_id}" + ); + } + + let Some(record) = active_candidates.into_iter().next() else { + return if live_host_orchestrators.is_empty() { + Ok(None) + } else { + Err(anyhow::anyhow!( + "live host-scoped orchestrator participant exists for agent {orchestrator_agent_id} without an active parent session" + )) + }; + }; + + let active_participant_id = + record + .session + .active_session_handle_id + .clone() + .ok_or_else(|| { + anyhow::anyhow!( + "active orchestration session {} is missing active_session_handle_id", + record.session.orchestration_session_id + ) + })?; + + let participant = record + .participants + .iter() + .find(|participant| participant.handle.participant_id == active_participant_id) + .cloned() + .ok_or_else(|| { + anyhow::anyhow!( + "active orchestration session {} references missing participant {}", + record.session.orchestration_session_id, + active_participant_id + ) + })?; + + if !participant.is_authoritative_live() || !owner_process_is_alive(&participant) { + anyhow::bail!( + "active orchestration session {} references inactive participant {}", + record.session.orchestration_session_id, + active_participant_id + ); + } + if participant.handle.agent_id != orchestrator_agent_id { + anyhow::bail!( + "active orchestration session {} belongs to agent {} not {}", + record.session.orchestration_session_id, + participant.handle.agent_id, + orchestrator_agent_id + ); + } + if participant.handle.orchestration_session_id != record.session.orchestration_session_id { + anyhow::bail!( + "active orchestration session {} does not match participant {} parent {}", + record.session.orchestration_session_id, + active_participant_id, + participant.handle.orchestration_session_id + ); + } + if participant.handle.role != ORCHESTRATOR_ROLE + || participant.handle.execution.scope != AgentExecutionScope::Host + { + anyhow::bail!( + "active orchestration session {} references non-host orchestrator participant {}", + record.session.orchestration_session_id, + active_participant_id + ); + } + if live_host_orchestrators + .iter() + .any(|candidate| candidate.handle.participant_id != participant.handle.participant_id) + { + anyhow::bail!( + "multiple live orchestrator participant candidates found for agent {orchestrator_agent_id}" + ); + } + + Ok(Some(record)) + } + + pub(crate) fn resolve_public_control_target( + &self, + orchestration_session_id: &str, + action: PublicControlAction, + ) -> Result { + let Some(record) = self.load_session(orchestration_session_id)? else { + return Err(self.public_session_selector_error(orchestration_session_id)); + }; + let resolved = resolve_authoritative_session_control(&record, orchestration_session_id)?; + + if session_requires_linux_first_public_control_posture(&record) + && !cfg!(target_os = "linux") + { + anyhow::bail!( + "unsupported_platform_or_posture: orchestration session {} requires Linux world-sensitive control posture", + orchestration_session_id + ); + } + + if action.rejects_live_owner() && resolved.session_posture == PublicSessionPosture::Active { + anyhow::bail!( + "session_already_owned: orchestration session {} already has a live retained owner", + orchestration_session_id + ); + } + if matches!(action, PublicControlAction::Stop) + && resolved.session_posture == PublicSessionPosture::Terminal + { + anyhow::bail!( + "owner_unreachable: orchestration session {} no longer has a reachable retained owner", + orchestration_session_id + ); + } + if (action.requires_internal_session_id() + || matches!( + (action, resolved.session_posture), + ( + PublicControlAction::Stop, + PublicSessionPosture::DetachedReattachable + ) + )) + && !resolved.participant.is_resume_eligible() + { + anyhow::bail!( + "owner_unreachable: orchestration session {} no longer has a resume-eligible retained owner", + orchestration_session_id + ); + } + if (action.requires_internal_session_id() + || matches!( + (action, resolved.session_posture), + ( + PublicControlAction::Stop, + PublicSessionPosture::DetachedReattachable + ) + )) + && resolved.participant.internal_uaa_session_id().is_none() + { + anyhow::bail!( + "missing_internal_session_id: orchestration session {} active participant {} is missing internal.uaa_session_id", + orchestration_session_id, + resolved.participant.handle.participant_id + ); + } + + Ok(ResolvedPublicControlTarget { + session: resolved.session, + active_participant: resolved.participant, + session_posture: resolved.session_posture, + }) + } + + // Public turn routing stays exact: + // (orchestration_session_id, backend_id) selects one authoritative retained slot, + // or it fails closed without falling back to fuzzy inventory guesses. + #[allow(dead_code)] + pub(crate) fn resolve_public_turn_target( + &self, + orchestration_session_id: &str, + backend_id: &str, + ) -> Result { + if backend_id.trim().is_empty() { + anyhow::bail!("missing_backend: public turn actions require --backend "); + } + + let Some(record) = self.load_session(orchestration_session_id)? else { + return Err(self.public_turn_session_selector_error(orchestration_session_id)); + }; + + if !record.has_authoritative_parent() { + anyhow::bail!( + "missing_active_parent: orchestration session {} is missing authoritative parent metadata", + orchestration_session_id + ); + } + if record.session.state != OrchestrationSessionState::Active { + anyhow::bail!( + "missing_active_parent: orchestration session {} is not active", + orchestration_session_id + ); + } + + let authoritative = + resolve_authoritative_session_control(&record, orchestration_session_id)?; + + let slot_present = public_turn_session_mentions_backend(&record, backend_id); + let mut candidates = public_turn_authoritative_candidates(&record, backend_id); + if candidates.is_empty() { + if slot_present { + anyhow::bail!( + "stale_linkage: orchestration session {} backend {} no longer has an authoritative retained turn target", + orchestration_session_id, + backend_id + ); + } + anyhow::bail!( + "backend_not_in_session: orchestration session {} has no exact backend slot for {}", + orchestration_session_id, + backend_id + ); + } + if candidates.len() > 1 { + let participant_ids = candidates + .iter() + .map(|candidate| candidate.participant.handle.participant_id.as_str()) + .collect::>() + .join(", "); + anyhow::bail!( + "ambiguous_backend_slot: orchestration session {} has multiple authoritative retained turn targets for backend {} ({participant_ids})", + orchestration_session_id, + backend_id + ); + } + + let candidate = candidates.pop().expect("candidate count checked above"); + if candidate.kind == PublicTurnTargetKind::World && !cfg!(target_os = "linux") { + anyhow::bail!( + "unsupported_platform_or_posture: orchestration session {} backend {} requires Linux world-sensitive follow-up posture", + orchestration_session_id, + backend_id + ); + } + + Ok(ResolvedPublicTurnTarget { + session: authoritative.session, + participant: candidate.participant, + target_kind: candidate.kind, + session_posture: authoritative.session_posture, + }) + } + + pub(crate) fn classify_hidden_owner_helper_launch_readiness( + &self, + orchestration_session_id: &str, + participant_id: &str, + require_internal_session_id: bool, + ) -> Result { + Ok( + match self.classify_hidden_owner_helper_launch_continuity( + orchestration_session_id, + participant_id, + require_internal_session_id, + )? { + HiddenOwnerHelperLaunchContinuity::AttachedLive => { + HiddenOwnerHelperLaunchReadiness::ReadyAttached + } + HiddenOwnerHelperLaunchContinuity::DetachedReconciled(posture) => { + HiddenOwnerHelperLaunchReadiness::ReadyDetached(posture) + } + HiddenOwnerHelperLaunchContinuity::Pending + | HiddenOwnerHelperLaunchContinuity::StaleAttachedTruth => { + HiddenOwnerHelperLaunchReadiness::Pending + } + }, + ) + } + + pub(crate) fn classify_hidden_owner_helper_launch_continuity( + &self, + orchestration_session_id: &str, + participant_id: &str, + require_internal_session_id: bool, + ) -> Result { + let Some(record) = self.load_session(orchestration_session_id)? else { + return Ok(HiddenOwnerHelperLaunchContinuity::Pending); + }; + if record.session.state != OrchestrationSessionState::Active { + return Ok(HiddenOwnerHelperLaunchContinuity::Pending); + } + if record.session.active_participant_id() != Some(participant_id) { + return Ok(HiddenOwnerHelperLaunchContinuity::Pending); + } + + let Some(participant) = record + .participants + .iter() + .find(|participant| participant.participant_id() == participant_id) + else { + return Ok(HiddenOwnerHelperLaunchContinuity::Pending); + }; + if !participant.matches_public_parent_linkage(&record.session) { + return Ok(HiddenOwnerHelperLaunchContinuity::Pending); + } + if require_internal_session_id && participant.internal_uaa_session_id().is_none() { + return Ok(HiddenOwnerHelperLaunchContinuity::Pending); + } + + let attached_live = session_attached_to_participant(&record.session, participant) + && participant.attached_client_present() + && participant.is_authoritative_live() + && owner_process_is_alive(participant); + if attached_live { + return Ok(HiddenOwnerHelperLaunchContinuity::AttachedLive); + } + + if let Some(posture) = valid_detached_host_continuity_posture( + &record.session, + participant, + require_internal_session_id, + ) { + return Ok(HiddenOwnerHelperLaunchContinuity::DetachedReconciled( + posture, + )); + } + + if recoverable_stale_host_attachment( + &record, + &record.session, + participant, + require_internal_session_id, + ) { + return Ok(HiddenOwnerHelperLaunchContinuity::StaleAttachedTruth); + } + + Ok(HiddenOwnerHelperLaunchContinuity::Pending) + } + + #[cfg(unix)] + pub(crate) fn resumed_public_turn_detach_posture( + &self, + orchestration_session_id: &str, + participant_id: &str, + ) -> Result> { + let Some(record) = self.load_session(orchestration_session_id)? else { + return Ok(None); + }; + if record.session.state != OrchestrationSessionState::Active { + return Ok(None); + } + if record.session.active_participant_id() != Some(participant_id) { + return Ok(None); + } + + let Some(participant) = record + .participants + .iter() + .find(|participant| participant.participant_id() == participant_id) + else { + return Ok(None); + }; + + let Some(posture) = + valid_detached_host_continuity_posture(&record.session, participant, true) + else { + return Ok(None); + }; + + if record.participants.iter().any(|candidate| { + candidate.participant_id() != participant_id + && candidate.matches_public_parent_linkage(&record.session) + && candidate.is_host_orchestrator() + && candidate.attached_client_present() + && candidate.is_authoritative_live() + && owner_process_is_alive(candidate) + }) { + return Ok(None); + } + + Ok(Some(posture)) + } + + pub(crate) fn startup_prompt_replay_state( + &self, + orchestration_session_id: &str, + participant_id: &str, + ) -> Result { + let Some(session) = self.load_orchestration_session(orchestration_session_id)? else { + return Ok(StartupPromptReplayState::NotTracked); + }; + let Some(startup_prompt) = session.startup_prompt.as_ref() else { + return Ok(StartupPromptReplayState::NotTracked); + }; + if startup_prompt.participant_id != participant_id { + return Ok(StartupPromptReplayState::AcceptedOrTerminal); + } + Ok(match startup_prompt.state { + StartupPromptStreamState::PendingAcceptance => { + StartupPromptReplayState::PendingAcceptance + } + StartupPromptStreamState::Accepted + | StartupPromptStreamState::Completed + | StartupPromptStreamState::Failed => StartupPromptReplayState::AcceptedOrTerminal, + }) + } + + pub(crate) fn resolve_live_orchestrator_participant( + &self, + orchestrator_agent_id: &str, + ) -> Result> { + Ok(self + .resolve_single_live_session_for_agent(orchestrator_agent_id)? + .and_then(|record| { + record + .live_orchestrator() + .map(|participant| (record.session, participant)) + })) + } + + pub(crate) fn validate_participant_record( + &self, + participant: &AgentRuntimeParticipantRecord, + ) -> Result<()> { + participant.validate() + } + + pub(crate) fn validate_session_record( + &self, + session: &OrchestrationSessionRecord, + ) -> Result<()> { + session.validate_persisted_invariants() + } + + fn validate_inbox_item_record(&self, item: &DurableInboxItemRecord) -> Result<()> { + item.validate() + } + + pub(crate) fn persist_orchestration_session( + &self, + session: &OrchestrationSessionRecord, + ) -> Result<()> { + let _write_guard = snapshot_write_lock() + .lock() + .expect("snapshot write mutex poisoned"); + self.validate_session_record(session)?; + if let Some(existing) = + self.load_authoritative_session(&session.orchestration_session_id)? + { + if !should_persist_orchestration_session_snapshot(&existing, session) { + return Ok(()); + } + } + self.persist_parent_session_snapshot(session) + } + + #[allow(dead_code)] + pub(crate) fn persist_inbox_item(&self, item: &DurableInboxItemRecord) -> Result<()> { + let _write_guard = snapshot_write_lock() + .lock() + .expect("snapshot write mutex poisoned"); + self.validate_inbox_item_record(item)?; + + let mut session = self + .load_authoritative_session(&item.orchestration_session_id)? + .ok_or_else(|| { + anyhow::anyhow!( + "missing authoritative parent session {} for durable inbox item {}", + item.orchestration_session_id, + item.item_id + ) + })?; + let existing = self.load_inbox_item(&item.orchestration_session_id, &item.item_id)?; + let next_pending_count = updated_pending_inbox_count( + session.pending_inbox_count, + existing + .as_ref() + .is_some_and(DurableInboxItemRecord::is_pending), + item.is_pending(), + )?; + apply_pending_inbox_count(&mut session, next_pending_count); + + let item_path = + self.canonical_inbox_item_path(&item.orchestration_session_id, &item.item_id); + write_atomic_json(&item_path, item)?; + if let Err(err) = self.persist_parent_session_snapshot(&session) { + rollback_inbox_item_write(&item_path, existing.as_ref())?; + return Err(err); + } + + Ok(()) + } + + #[allow(dead_code)] + pub(crate) fn acknowledge_inbox_item( + &self, + orchestration_session_id: &str, + item_id: &str, + ) -> Result { + self.resolve_inbox_item( + orchestration_session_id, + item_id, + DurableInboxItemState::Acknowledged, + ) + } + + #[allow(dead_code)] + pub(crate) fn dismiss_inbox_item( + &self, + orchestration_session_id: &str, + item_id: &str, + ) -> Result { + self.resolve_inbox_item( + orchestration_session_id, + item_id, + DurableInboxItemState::Dismissed, + ) + } + + #[allow(dead_code)] + pub(crate) fn load_inbox_item( + &self, + orchestration_session_id: &str, + item_id: &str, + ) -> Result> { + let path = self.canonical_inbox_item_path(orchestration_session_id, item_id); + let Some(item) = read_regular_json_if_exists::(&path)? else { + return Ok(None); + }; + self.validate_inbox_item_record(&item) + .with_context(|| format!("invalid durable inbox item in {}", path.display()))?; + if item.orchestration_session_id != orchestration_session_id { + anyhow::bail!( + "durable inbox item {} belongs to session {} not {}", + item_id, + item.orchestration_session_id, + orchestration_session_id + ); + } + if item.item_id != item_id { + anyhow::bail!( + "durable inbox artifact {} stored mismatched item_id {}", + path.display(), + item.item_id + ); + } + + Ok(Some(item)) + } + + #[allow(dead_code)] + pub(crate) fn list_inbox_items( + &self, + orchestration_session_id: &str, + ) -> Result> { + let inbox_dir = self.canonical_inbox_dir(orchestration_session_id); + let Some(entries) = safe_read_dir(&inbox_dir)? else { + return Ok(Vec::new()); + }; + + let mut items = Vec::new(); + for entry in entries { + let entry = entry.with_context(|| format!("failed to read {}", inbox_dir.display()))?; + let path = entry.path(); + if path.extension().and_then(|value| value.to_str()) != Some("json") { + continue; + } + + let Some(item) = read_regular_json_if_exists::(&path)? else { + continue; + }; + self.validate_inbox_item_record(&item) + .with_context(|| format!("invalid durable inbox item in {}", path.display()))?; + if item.orchestration_session_id != orchestration_session_id { + anyhow::bail!( + "durable inbox item {} belongs to session {} not {}", + item.item_id, + item.orchestration_session_id, + orchestration_session_id + ); + } + items.push(item); + } + + items.sort_by(|left, right| { + left.created_at + .cmp(&right.created_at) + .then(left.item_id.cmp(&right.item_id)) + }); + Ok(items) + } + + pub(crate) fn set_orchestration_session_world_binding( + &self, + session: &mut OrchestrationSessionRecord, + world_id: impl Into, + world_generation: u64, + ) -> Result<()> { + let _write_guard = snapshot_write_lock() + .lock() + .expect("snapshot write mutex poisoned"); + session.set_world_binding(world_id, world_generation); + self.persist_parent_session_snapshot(session) + } + + pub(crate) fn clear_orchestration_session_world_binding( + &self, + session: &mut OrchestrationSessionRecord, + ) -> Result<()> { + let _write_guard = snapshot_write_lock() + .lock() + .expect("snapshot write mutex poisoned"); + session.clear_world_binding(); + self.persist_parent_session_snapshot(session) + } + + fn persist_lease(&self, participant: &AgentRuntimeParticipantRecord) -> Result<()> { + let payload = serde_json::json!({ + "participant_id": participant.handle.participant_id, + "session_handle_id": participant.handle.session_handle_id, + "shell_owner_pid": participant.internal.shell_owner_pid, + "lease_token": participant.internal.lease_token, + "state": participant.handle.state, + "ownership_valid": participant.internal.ownership_valid, + "last_heartbeat_at": participant.internal.last_heartbeat_at, + "terminal_observed_at": participant.internal.terminal_observed_at, + }); + write_atomic_json( + &self.lease_path(&participant.handle.participant_id), + &payload, + )?; + write_atomic_json( + &self.canonical_lease_path( + &participant.handle.orchestration_session_id, + &participant.handle.participant_id, + ), + &payload, + ) + } + + fn persist_parent_session_snapshot(&self, session: &OrchestrationSessionRecord) -> Result<()> { + self.validate_session_record(session)?; + self.ensure_sessions_dir()?; + write_atomic_json( + &self.orchestration_session_path(&session.orchestration_session_id), + session, + )?; + write_atomic_json( + &self.canonical_session_path(&session.orchestration_session_id), + session, + ) + } + + fn read_participant_dir(&self, dir: &Path) -> Result> { + let Some(entries) = safe_read_dir(dir)? else { + return Ok(Vec::new()); + }; + let mut participants = Vec::new(); + for entry in entries { + let entry = entry.with_context(|| format!("failed to read {}", dir.display()))?; + let path = entry.path(); + if path.extension().and_then(|value| value.to_str()) != Some("json") { + continue; + } + let Some(participant) = + read_regular_json_if_exists::(&path)? + else { + continue; + }; + self.validate_participant_record(&participant) + .with_context(|| format!("invalid participant record in {}", path.display()))?; + participants.push(participant); + } + + Ok(participants) + } + + fn read_canonical_participants( + &self, + ) -> Result> { + let mut participants = Vec::new(); + + for orchestration_session_id in self.canonical_session_root_ids()? { + let participants_dir = self.canonical_participants_dir(&orchestration_session_id); + let Some(entries) = safe_read_dir(&participants_dir)? else { + continue; + }; + for entry in entries { + let entry = entry + .with_context(|| format!("failed to read {}", participants_dir.display()))?; + let path = entry.path(); + if path.extension().and_then(|value| value.to_str()) != Some("json") { + continue; + } + let Some(participant) = + read_regular_json_if_exists::(&path)? + else { + continue; + }; + self.validate_participant_record(&participant) + .with_context(|| format!("invalid participant record in {}", path.display()))?; + participants.push((participant, ParticipantRecordSource::Canonical)); + } + } + + Ok(participants) + } + + #[allow(dead_code)] + pub(crate) fn load_orchestration_session( + &self, + orchestration_session_id: &str, + ) -> Result> { + self.load_authoritative_session(orchestration_session_id) + } + + #[allow(dead_code)] + pub(crate) fn list_orchestration_sessions(&self) -> Result> { + let mut sessions = Vec::new(); + let mut session_ids = BTreeSet::new(); + for session_id in self.canonical_session_root_ids()? { + session_ids.insert(session_id); + } + for session_id in self.flat_session_ids()? { + session_ids.insert(session_id); + } + + for session_id in session_ids { + if let Some(session) = self.load_authoritative_session(&session_id)? { + sessions.push(session); + } + } + + sessions.sort_by_key(|session| session.last_active_at); + Ok(sessions) + } + + #[allow(dead_code)] + pub(crate) fn find_active_orchestration_session_for_pid( + &self, + pid: u32, + ) -> Result> { + let matches = self + .list_orchestration_sessions()? + .into_iter() + .filter(|session| { + session.shell_owner_pid == pid + && session.state == OrchestrationSessionState::Active + && owner_pid_is_alive(session.shell_owner_pid) + }) + .collect::>(); + match matches.len() { + 0 => Ok(None), + 1 => Ok(matches.into_iter().next()), + _ => Err(anyhow::anyhow!( + "multiple active orchestration sessions found for shell pid {pid}" + )), + } + } + + #[allow(dead_code)] + pub(crate) fn persist_manifest(&self, manifest: &AgentRuntimeSessionManifest) -> Result<()> { + self.persist_participant(manifest) + } + + #[allow(dead_code)] + pub(crate) fn list_manifests(&self) -> Result> { + self.list_participants() + } + + #[allow(dead_code)] + pub(crate) fn list_live_manifests(&self) -> Result> { + self.list_live_participants() + } + + #[allow(dead_code)] + pub(crate) fn find_live_orchestrator( + &self, + agent_id: &str, + ) -> Result> { + Ok(self + .resolve_live_orchestrator_participant(agent_id)? + .map(|(_, participant)| participant)) + } + + #[allow(dead_code)] + pub(crate) fn resolve_live_orchestrator_session( + &self, + agent_id: &str, + ) -> Result> { + self.resolve_live_orchestrator_participant(agent_id) + } + + #[allow(dead_code)] + fn load_manifest(&self, participant_id: &str) -> Result> { + self.load_participant(participant_id) + } + + fn load_authoritative_session( + &self, + orchestration_session_id: &str, + ) -> Result> { + let canonical_dir = self.canonical_session_dir(orchestration_session_id); + if safe_metadata(&canonical_dir)?.is_some_and(|metadata| metadata.is_dir()) { + let canonical_path = self.canonical_session_path(orchestration_session_id); + if let Some(session) = + read_regular_json_if_exists::(&canonical_path)? + { + self.validate_session_record(&session).with_context(|| { + format!("invalid session record in {}", canonical_path.display()) + })?; + return Ok(Some(session)); + } + } + + let flat_path = self.orchestration_session_path(orchestration_session_id); + if let Some(session) = read_regular_json_if_exists(&flat_path)? { + self.validate_session_record(&session) + .with_context(|| format!("invalid session record in {}", flat_path.display()))?; + return Ok(Some(session)); + } + Ok(None) + } + + fn public_session_selector_error(&self, selector: &str) -> anyhow::Error { + if selector.trim().is_empty() { + return anyhow::anyhow!( + "unknown_session: public control actions require --session " + ); + } + + if self + .list_orchestration_sessions() + .map(|sessions| { + sessions + .into_iter() + .any(|session| session.active_participant_id() == Some(selector)) + }) + .unwrap_or(false) + { + return anyhow::anyhow!( + "unknown_session: selector '{}' matched active_session_handle_id; public control actions accept only orchestration_session_id", + selector + ); + } + + if self + .list_participants_across_sources() + .map(|participants| { + participants.into_iter().any(|participant| { + participant.participant_id() == selector + || participant.handle.session_handle_id == selector + }) + }) + .unwrap_or(false) + { + return anyhow::anyhow!( + "unknown_session: selector '{}' matched participant_id/session_handle_id; public control actions accept only orchestration_session_id", + selector + ); + } + + if self + .list_participants_across_sources() + .map(|participants| { + participants + .into_iter() + .any(|participant| participant.internal_uaa_session_id() == Some(selector)) + }) + .unwrap_or(false) + { + return anyhow::anyhow!( + "unknown_session: selector '{}' matched internal.uaa_session_id; public control actions accept only orchestration_session_id", + selector + ); + } + + anyhow::anyhow!( + "unknown_session: no orchestration session found for '{}'", + selector + ) + } + + #[allow(dead_code)] + fn public_turn_session_selector_error(&self, selector: &str) -> anyhow::Error { + if selector.trim().is_empty() { + return anyhow::anyhow!( + "unknown_session: public turn actions require --session " + ); + } + + if self + .list_orchestration_sessions() + .map(|sessions| { + sessions + .into_iter() + .any(|session| session.active_participant_id() == Some(selector)) + }) + .unwrap_or(false) + { + return anyhow::anyhow!( + "noncanonical_session_selector: selector '{}' matched active_session_handle_id; public turn actions accept only orchestration_session_id", + selector + ); + } + + if self + .list_participants_across_sources() + .map(|participants| { + participants.into_iter().any(|participant| { + participant.participant_id() == selector + || participant.handle.session_handle_id == selector + }) + }) + .unwrap_or(false) + { + return anyhow::anyhow!( + "noncanonical_session_selector: selector '{}' matched participant_id/session_handle_id; public turn actions accept only orchestration_session_id", + selector + ); + } + + if self + .list_participants_across_sources() + .map(|participants| { + participants + .into_iter() + .any(|participant| participant.internal_uaa_session_id() == Some(selector)) + }) + .unwrap_or(false) + { + return anyhow::anyhow!( + "noncanonical_session_selector: selector '{}' matched internal.uaa_session_id; public turn actions accept only orchestration_session_id", + selector + ); + } + + anyhow::anyhow!( + "unknown_session: no orchestration session found for '{}'", + selector + ) + } + + fn canonical_session_root_ids(&self) -> Result> { + let Some(entries) = safe_read_dir(&self.sessions_dir())? else { + return Ok(Vec::new()); + }; + + let mut session_ids = Vec::new(); + for entry in entries { + let entry = entry + .with_context(|| format!("failed to read {}", self.sessions_dir().display()))?; + let path = entry.path(); + let Some(metadata) = safe_metadata(&path)? else { + continue; + }; + if metadata.file_type().is_dir() { + let file_name = entry.file_name(); + let Some(session_id) = file_name.to_str() else { + continue; + }; + session_ids.push(session_id.to_string()); + } + } + session_ids.sort(); + Ok(session_ids) + } + + fn flat_session_ids(&self) -> Result> { + let Some(entries) = safe_read_dir(&self.sessions_dir())? else { + return Ok(Vec::new()); + }; + + let mut session_ids = Vec::new(); + for entry in entries { + let entry = entry + .with_context(|| format!("failed to read {}", self.sessions_dir().display()))?; + let path = entry.path(); + let Some(metadata) = safe_metadata(&path)? else { + continue; + }; + if !metadata.is_file() + || path.extension().and_then(|value| value.to_str()) != Some("json") + { + continue; + } + let Some(stem) = path.file_stem().and_then(|value| value.to_str()) else { + continue; + }; + session_ids.push(stem.to_string()); + } + session_ids.sort(); + Ok(session_ids) + } + + fn build_session_record( + &self, + orchestration_session_id: &str, + session: Option, + mut participants: Vec, + ) -> AgentRuntimeSessionRecord { + participants.sort_by(|left, right| { + left.handle + .last_transition_at + .cmp(&right.handle.last_transition_at) + .then(left.handle.participant_id.cmp(&right.handle.participant_id)) + }); + + let has_authoritative_parent = session.is_some(); + let mut warnings = Vec::new(); + if !has_authoritative_parent { + warnings.push(format!( + "orchestration session {orchestration_session_id} is missing authoritative parent session metadata" + )); + } + + let session = session + .unwrap_or_else(|| synthesize_session_record(orchestration_session_id, &participants)); + + let contract_valid = match validate_runtime_contract(&session, &participants) { + Ok(()) => true, + Err(err) => { + warnings.push(format!( + "orchestration session {} violates persisted runtime contract: {err}", + session.orchestration_session_id + )); + false + } + }; + + let complete = if !has_authoritative_parent || !contract_valid { + false + } else if session.state != OrchestrationSessionState::Active { + true + } else { + match session_authoritative_participant_id(&session) { + Some(active_participant_id) => match participants + .iter() + .find(|participant| participant.handle.participant_id == active_participant_id) + { + Some(participant) if participant.matches_public_parent_linkage(&session) => { + if session_attached_to_participant(&session, participant) { + participant.is_authoritative_live() + && owner_process_is_alive(participant) + } else { + valid_detached_host_continuity_posture(&session, participant, true) + .is_some() + } + } + Some(participant) => { + warnings.push(format!( + "active orchestration session {} references incomplete live orchestrator participant {}", + session.orchestration_session_id, participant.handle.participant_id + )); + false + } + None => { + warnings.push(format!( + "active orchestration session {} references missing participant {}", + session.orchestration_session_id, active_participant_id + )); + false + } + }, + None => { + warnings.push(format!( + "active orchestration session {} is missing authoritative orchestrator participant linkage", + session.orchestration_session_id + )); + false + } + } + }; + + AgentRuntimeSessionRecord { + session, + participants, + warnings, + has_authoritative_parent, + complete, + } + } + + fn resolve_inbox_item( + &self, + orchestration_session_id: &str, + item_id: &str, + state: DurableInboxItemState, + ) -> Result { + if state.is_pending() { + anyhow::bail!("durable inbox resolution requires a terminal inbox state"); + } + + let mut item = self + .load_inbox_item(orchestration_session_id, item_id)? + .ok_or_else(|| { + anyhow::anyhow!( + "durable inbox item {} not found in session {}", + item_id, + orchestration_session_id + ) + })?; + item.transition_state(state); + self.persist_inbox_item(&item)?; + Ok(item) + } +} + +fn write_atomic_json(path: &Path, value: &impl serde::Serialize) -> Result<()> { + let parent = path + .parent() + .ok_or_else(|| anyhow::anyhow!("{} has no parent directory", path.display()))?; + fs::create_dir_all(parent).with_context(|| format!("failed to create {}", parent.display()))?; + + let mut tmp = NamedTempFile::new_in(parent) + .with_context(|| format!("failed to create temp file in {}", parent.display()))?; + serde_json::to_writer_pretty(tmp.as_file_mut(), value) + .with_context(|| format!("failed to serialize {}", path.display()))?; + tmp.as_file_mut() + .sync_all() + .with_context(|| format!("failed to flush {}", path.display()))?; + tmp.persist(path) + .map_err(|err| err.error) + .with_context(|| format!("failed to persist {}", path.display()))?; + Ok(()) +} + +fn snapshot_write_lock() -> &'static Mutex<()> { + static SNAPSHOT_WRITE_LOCK: OnceLock> = OnceLock::new(); + SNAPSHOT_WRITE_LOCK.get_or_init(|| Mutex::new(())) +} + +fn updated_pending_inbox_count(current: u64, was_pending: bool, is_pending: bool) -> Result { + match (was_pending, is_pending) { + (false, false) | (true, true) => Ok(current), + (false, true) => current + .checked_add(1) + .ok_or_else(|| anyhow::anyhow!("pending_inbox_count overflow")), + (true, false) => current + .checked_sub(1) + .ok_or_else(|| anyhow::anyhow!("pending_inbox_count underflow")), + } +} + +fn apply_pending_inbox_count(session: &mut OrchestrationSessionRecord, pending_inbox_count: u64) { + let now = Utc::now(); + session.pending_inbox_count = pending_inbox_count; + session.last_active_at = now; + if pending_inbox_count > 0 { + session.last_attention_at = Some(now); + } + + if session.state.is_terminal() { + return; + } + + if session.attached_participant_id().is_some() { + return; + } + + let desired_posture = if pending_inbox_count > 0 { + OrchestrationSessionPosture::AwaitingAttention + } else { + OrchestrationSessionPosture::ParkedResumable + }; + if session.posture != desired_posture { + session.posture = desired_posture; + session.posture_changed_at = now; + } + if desired_posture == OrchestrationSessionPosture::ParkedResumable { + session.last_parked_at = Some(now); + } +} + +fn rollback_inbox_item_write( + item_path: &Path, + previous: Option<&DurableInboxItemRecord>, +) -> Result<()> { + match previous { + Some(previous) => write_atomic_json(item_path, previous), + None => match fs::remove_file(item_path) { + Ok(()) => Ok(()), + Err(err) if err.kind() == io::ErrorKind::NotFound => Ok(()), + Err(err) => { + Err(err).with_context(|| format!("failed to roll back {}", item_path.display())) + } + }, + } +} + +fn participant_snapshot_freshness(participant: &AgentRuntimeParticipantRecord) -> DateTime { + [ + Some(participant.handle.last_transition_at), + participant.internal.last_event_at, + participant.internal.last_heartbeat_at, + participant.internal.ownership_verified_at, + participant.internal.terminal_observed_at, + ] + .into_iter() + .flatten() + .max() + .unwrap_or(participant.handle.last_transition_at) +} + +fn participant_state_rank(state: &super::session::AgentRuntimeSessionState) -> u8 { + match state { + super::session::AgentRuntimeSessionState::Allocating => 0, + super::session::AgentRuntimeSessionState::Ready => 1, + super::session::AgentRuntimeSessionState::Running => 2, + super::session::AgentRuntimeSessionState::Restarting => 3, + super::session::AgentRuntimeSessionState::Stopping => 4, + super::session::AgentRuntimeSessionState::Stopped => 5, + super::session::AgentRuntimeSessionState::Failed => 6, + super::session::AgentRuntimeSessionState::Invalidated => 7, + } +} + +fn should_persist_participant_snapshot( + existing: &AgentRuntimeParticipantRecord, + incoming: &AgentRuntimeParticipantRecord, +) -> bool { + if !existing.handle.state.is_live() && incoming.handle.state.is_live() { + return false; + } + if participant_state_rank(&incoming.handle.state) + < participant_state_rank(&existing.handle.state) + && incoming.handle.last_transition_at <= existing.handle.last_transition_at + { + return false; + } + + match participant_snapshot_freshness(incoming).cmp(&participant_snapshot_freshness(existing)) { + Ordering::Greater => true, + Ordering::Less => false, + Ordering::Equal => { + let incoming_terminal = incoming.internal.terminal_observed_at.is_some(); + let existing_terminal = existing.internal.terminal_observed_at.is_some(); + ( + incoming.handle.last_transition_at, + incoming_terminal, + participant_state_rank(&incoming.handle.state), + ) >= ( + existing.handle.last_transition_at, + existing_terminal, + participant_state_rank(&existing.handle.state), + ) + } + } +} + +fn orchestration_session_freshness(session: &OrchestrationSessionRecord) -> DateTime { + session + .closed_at + .unwrap_or(session.last_active_at) + .max(session.last_active_at) +} + +fn orchestration_session_state_rank(state: &OrchestrationSessionState) -> u8 { + match state { + OrchestrationSessionState::Allocating => 0, + OrchestrationSessionState::Active => 1, + OrchestrationSessionState::Stopping => 2, + OrchestrationSessionState::Stopped => 3, + OrchestrationSessionState::Failed => 4, + OrchestrationSessionState::Invalidated => 5, + } +} + +fn should_persist_orchestration_session_snapshot( + existing: &OrchestrationSessionRecord, + incoming: &OrchestrationSessionRecord, +) -> bool { + if matches!( + existing.state, + OrchestrationSessionState::Stopped + | OrchestrationSessionState::Failed + | OrchestrationSessionState::Invalidated + ) && incoming.state.is_active() + { + return false; + } + if orchestration_session_state_rank(&incoming.state) + < orchestration_session_state_rank(&existing.state) + { + return false; + } + + match orchestration_session_freshness(incoming).cmp(&orchestration_session_freshness(existing)) + { + Ordering::Greater => true, + Ordering::Less => false, + Ordering::Equal => { + ( + incoming.last_active_at, + incoming.closed_at.is_some(), + orchestration_session_state_rank(&incoming.state), + ) >= ( + existing.last_active_at, + existing.closed_at.is_some(), + orchestration_session_state_rank(&existing.state), + ) + } + } +} + +#[cfg(unix)] +fn owner_process_is_alive(participant: &AgentRuntimeParticipantRecord) -> bool { + owner_pid_is_alive(participant.internal.shell_owner_pid) +} + +#[cfg(unix)] +fn owner_pid_is_alive(pid: u32) -> bool { + let pid = pid as libc::pid_t; + if pid <= 0 { + return false; + } + + let rc = unsafe { libc::kill(pid, 0) }; + if rc == 0 { + return true; + } + + matches!(io::Error::last_os_error().raw_os_error(), Some(libc::EPERM)) +} + +#[cfg(not(unix))] +fn owner_process_is_alive(participant: &AgentRuntimeParticipantRecord) -> bool { + owner_pid_is_alive(participant.internal.shell_owner_pid) +} + +#[cfg(not(unix))] +fn owner_pid_is_alive(pid: u32) -> bool { + pid == std::process::id() +} + +fn read_json_if_exists(path: &Path) -> Result> +where + T: serde::de::DeserializeOwned, +{ + match fs::read_to_string(path) { + Ok(raw) => { + Ok(Some(serde_json::from_str(&raw).with_context(|| { + format!("failed to parse {}", path.display()) + })?)) + } + Err(err) if err.kind() == io::ErrorKind::NotFound => Ok(None), + Err(err) => Err(err).with_context(|| format!("failed to read {}", path.display())), + } +} + +fn safe_metadata(path: &Path) -> Result> { + match fs::symlink_metadata(path) { + Ok(metadata) if metadata.file_type().is_symlink() => Ok(None), + Ok(metadata) => Ok(Some(metadata)), + Err(err) if err.kind() == io::ErrorKind::NotFound => Ok(None), + Err(err) => Err(err).with_context(|| format!("failed to stat {}", path.display())), + } +} + +fn safe_read_dir(path: &Path) -> Result> { + let Some(metadata) = safe_metadata(path)? else { + return Ok(None); + }; + if !metadata.is_dir() { + return Ok(None); + } + + let entries = + fs::read_dir(path).with_context(|| format!("failed to read {}", path.display()))?; + Ok(Some(entries)) +} + +fn read_regular_json_if_exists(path: &Path) -> Result> +where + T: serde::de::DeserializeOwned, +{ + let Some(metadata) = safe_metadata(path)? else { + return Ok(None); + }; + if !metadata.is_file() { + return Ok(None); + } + + read_json_if_exists(path) +} + +fn synthesize_session_record( + orchestration_session_id: &str, + participants: &[AgentRuntimeParticipantRecord], +) -> OrchestrationSessionRecord { + let template = participants + .iter() + .find(|participant| { + participant.handle.role == ORCHESTRATOR_ROLE + && participant.handle.execution.scope == AgentExecutionScope::Host + }) + .or_else(|| participants.first()) + .expect("synthetic session record requires at least one participant"); + + let mut session = OrchestrationSessionRecord::new( + orchestration_session_id.to_string(), + "".to_string(), + "".to_string(), + template, + ); + session.opened_at = participants + .iter() + .map(|participant| participant.handle.opened_at) + .min() + .unwrap_or(session.opened_at); + session.last_active_at = participants + .iter() + .map(AgentRuntimeParticipantRecord::last_status_at) + .max() + .unwrap_or(session.last_active_at); + + if let Some(orchestrator) = participants.iter().find(|participant| { + participant.handle.role == ORCHESTRATOR_ROLE + && participant.handle.execution.scope == AgentExecutionScope::Host + }) { + session.orchestrator_agent_id = orchestrator.handle.agent_id.clone(); + session.orchestrator_backend_id = orchestrator.handle.backend_id.clone(); + session.orchestrator_protocol = orchestrator.handle.protocol.clone(); + } + session.active_session_handle_id = participants + .iter() + .find(|participant| { + participant.handle.role == ORCHESTRATOR_ROLE + && participant.handle.execution.scope == AgentExecutionScope::Host + && participant.is_authoritative_live() + && owner_process_is_alive(participant) + }) + .map(|participant| participant.handle.participant_id.clone()); + session.latest_run_id = participants + .iter() + .filter_map(|participant| { + participant + .internal + .latest_run_id + .as_ref() + .map(|run_id| (participant.last_status_at(), run_id.clone())) + }) + .max_by(|left, right| left.0.cmp(&right.0)) + .map(|(_, run_id)| run_id); + session.state = if session.active_session_handle_id.is_some() { + OrchestrationSessionState::Active + } else { + OrchestrationSessionState::Allocating + }; + session +} + +fn session_requires_linux_first_public_control_posture(record: &AgentRuntimeSessionRecord) -> bool { + record.session.has_world_binding() + || record.participants.iter().any(|participant| { + participant.handle.role == MEMBER_ROLE + && participant.handle.execution.scope == AgentExecutionScope::World + && participant.handle.state.is_live() + }) +} + +#[derive(Clone, Debug)] +#[allow(dead_code)] +struct PublicTurnTargetCandidate { + participant: AgentRuntimeParticipantRecord, + kind: PublicTurnTargetKind, +} + +#[allow(dead_code)] +fn public_turn_session_mentions_backend( + record: &AgentRuntimeSessionRecord, + backend_id: &str, +) -> bool { + record.session.orchestrator_backend_id == backend_id + || record + .participants + .iter() + .any(|participant| participant.handle.backend_id == backend_id) +} + +#[allow(dead_code)] +fn public_turn_authoritative_candidates( + record: &AgentRuntimeSessionRecord, + backend_id: &str, +) -> Vec { + let mut candidates = Vec::new(); + let active_participant_id = session_authoritative_participant_id(&record.session); + + if let Some(active_participant_id) = active_participant_id { + if let Some(participant) = record + .participants + .iter() + .find(|participant| participant.participant_id() == active_participant_id) + .filter(|participant| { + participant.handle.backend_id == backend_id + && participant.matches_public_parent_linkage(&record.session) + }) + { + candidates.push(PublicTurnTargetCandidate { + participant: participant.clone(), + kind: PublicTurnTargetKind::Host, + }); + } + } + + let Some(world_id) = record.session.world_id.as_deref() else { + return candidates; + }; + let Some(world_generation) = record.session.world_generation else { + return candidates; + }; + + candidates.extend( + record + .participants + .iter() + .filter(|participant| { + participant.handle.backend_id == backend_id + && participant.handle.orchestration_session_id + == record.session.orchestration_session_id + && participant.handle.role == MEMBER_ROLE + && participant.handle.execution.scope == AgentExecutionScope::World + && participant.handle.orchestrator_participant_id.as_deref() + == active_participant_id + && participant.handle.world_id.as_deref() == Some(world_id) + && participant.handle.world_generation == Some(world_generation) + }) + .cloned() + .map(|participant| PublicTurnTargetCandidate { + participant, + kind: PublicTurnTargetKind::World, + }), + ); + + candidates +} + +fn resolve_authoritative_session_control( + record: &AgentRuntimeSessionRecord, + orchestration_session_id: &str, +) -> Result { + if !record.has_authoritative_parent() { + anyhow::bail!( + "missing_active_parent: orchestration session {} is missing authoritative parent metadata", + orchestration_session_id + ); + } + if record.session.state != OrchestrationSessionState::Active { + anyhow::bail!( + "missing_active_parent: orchestration session {} is not active", + orchestration_session_id + ); + } + + let active_participant_id = session_authoritative_participant_id(&record.session).ok_or_else( + || { + anyhow::anyhow!( + "stale_linkage: orchestration session {} is missing authoritative orchestrator participant linkage", + orchestration_session_id + ) + }, + )?; + let participant = record + .participants + .iter() + .find(|participant| participant.participant_id() == active_participant_id) + .cloned() + .ok_or_else(|| { + anyhow::anyhow!( + "stale_linkage: orchestration session {} references missing participant {}", + orchestration_session_id, + active_participant_id + ) + })?; + + if !participant.handle.state.is_live() { + anyhow::bail!( + "stale_linkage: orchestration session {} references inactive participant {}", + orchestration_session_id, + active_participant_id + ); + } + if !participant.matches_public_parent_linkage(&record.session) { + anyhow::bail!( + "stale_linkage: orchestration session {} active participant {} does not match exact orchestrator linkage", + orchestration_session_id, + active_participant_id + ); + } + + Ok(ResolvedAuthoritativeSessionControl { + session: record.session.clone(), + participant: participant.clone(), + session_posture: classify_public_session_posture(record, &participant), + }) +} + +fn detached_status_visible_participant( + record: &AgentRuntimeSessionRecord, +) -> Result> { + let resolved = + resolve_authoritative_session_control(record, &record.session.orchestration_session_id)?; + Ok( + (resolved.session_posture == PublicSessionPosture::DetachedReattachable) + .then_some(resolved.participant), + ) +} + +#[allow(dead_code)] +fn classify_public_session_posture( + record: &AgentRuntimeSessionRecord, + participant: &AgentRuntimeParticipantRecord, +) -> PublicSessionPosture { + let session = &record.session; + if session.posture == OrchestrationSessionPosture::Terminal || session.state.is_terminal() { + return PublicSessionPosture::Terminal; + } + if session_attached_to_participant(session, participant) + && participant.attached_client_present() + && participant.is_authoritative_live() + && owner_process_is_alive(participant) + { + return PublicSessionPosture::Active; + } + if valid_detached_host_continuity_posture(session, participant, true).is_some() { + return PublicSessionPosture::DetachedReattachable; + } + if recoverable_stale_host_attachment(record, session, participant, true) { + return PublicSessionPosture::DetachedReattachable; + } + PublicSessionPosture::Terminal +} + +// Detached continuity is valid only when persisted session truth and participant truth agree. +pub(crate) fn valid_detached_host_continuity_posture( + session: &OrchestrationSessionRecord, + participant: &AgentRuntimeParticipantRecord, + require_internal_session_id: bool, +) -> Option { + if session.state.is_terminal() || !participant.handle.state.is_live() { + return None; + } + if session.active_participant_id() != Some(participant.participant_id()) { + return None; + } + if !participant.matches_public_parent_linkage(session) { + return None; + } + if session.attached_participant_id().is_some() || participant.attached_client_present() { + return None; + } + if !participant.is_resume_eligible() { + return None; + } + if require_internal_session_id && participant.internal_uaa_session_id().is_none() { + return None; + } + + match session.posture { + OrchestrationSessionPosture::ParkedResumable if session.pending_inbox_count == 0 => { + Some(OrchestrationSessionPosture::ParkedResumable) + } + OrchestrationSessionPosture::AwaitingAttention if session.pending_inbox_count > 0 => { + Some(OrchestrationSessionPosture::AwaitingAttention) + } + _ => None, + } +} + +fn recoverable_stale_host_attachment( + record: &AgentRuntimeSessionRecord, + session: &OrchestrationSessionRecord, + participant: &AgentRuntimeParticipantRecord, + require_internal_session_id: bool, +) -> bool { + if session.state.is_terminal() || !participant.handle.state.is_live() { + return false; + } + if session.posture != OrchestrationSessionPosture::ActiveAttached { + return false; + } + if session.active_participant_id() != Some(participant.participant_id()) { + return false; + } + if !participant.matches_public_parent_linkage(session) { + return false; + } + if !session_attached_to_participant(session, participant) + || !participant.attached_client_present() + { + return false; + } + if owner_process_is_alive(participant) || !participant.is_resume_eligible() { + return false; + } + if record.participants.iter().any(|candidate| { + candidate.participant_id() != participant.participant_id() + && candidate.matches_public_parent_linkage(session) + && candidate.is_host_orchestrator() + && candidate.attached_client_present() + && candidate.is_authoritative_live() + && owner_process_is_alive(candidate) + }) { + return false; + } + + !require_internal_session_id || participant.internal_uaa_session_id().is_some() +} + +fn session_authoritative_participant_id(session: &OrchestrationSessionRecord) -> Option<&str> { + session + .active_participant_id() + .or(session.attached_participant_id()) +} + +fn session_attached_to_participant( + session: &OrchestrationSessionRecord, + participant: &AgentRuntimeParticipantRecord, +) -> bool { + session.attached_participant_id() == Some(participant.participant_id()) +} + +fn validate_runtime_contract( + session: &OrchestrationSessionRecord, + participants: &[AgentRuntimeParticipantRecord], +) -> Result<()> { + session.validate_persisted_invariants()?; + + let Some(authoritative_participant_id) = session_authoritative_participant_id(session) else { + if session.state == OrchestrationSessionState::Active + && session.posture == OrchestrationSessionPosture::ActiveAttached + { + anyhow::bail!("active_attached session is missing authoritative participant linkage"); + } + return Ok(()); + }; + + let participant = participants + .iter() + .find(|participant| participant.participant_id() == authoritative_participant_id) + .ok_or_else(|| { + anyhow::anyhow!( + "authoritative participant {} is missing from the session snapshot", + authoritative_participant_id + ) + })?; + if !participant.matches_public_parent_linkage(session) { + anyhow::bail!( + "authoritative participant {} no longer matches the session linkage", + authoritative_participant_id + ); + } + + match session.posture { + OrchestrationSessionPosture::ActiveAttached => { + if !participant.attached_client_present() { + anyhow::bail!("active_attached session requires attached host participant truth"); + } + } + OrchestrationSessionPosture::ParkedResumable => { + if !participant.is_resume_eligible() { + anyhow::bail!("parked_resumable session requires resume-eligible host participant"); + } + } + OrchestrationSessionPosture::AwaitingAttention => { + if !participant.is_resume_eligible() { + anyhow::bail!( + "awaiting_attention session requires resume-eligible host participant" + ); + } + } + OrchestrationSessionPosture::Terminal => {} + } + + Ok(()) +} + +#[cfg(test)] +mod tests { + use std::path::PathBuf; + + use serde_json::{json, Value}; + use tempfile::TempDir; + + use super::*; + use crate::execution::agent_runtime::{ + mapping::AgentRuntimeBackendKind, session::AgentRuntimeSessionState, + validator::RuntimeSelectionDescriptor, + }; + + fn descriptor(agent_id: &str, scope: AgentExecutionScope) -> RuntimeSelectionDescriptor { + RuntimeSelectionDescriptor { + agent_id: agent_id.to_string(), + backend_id: format!("cli:{agent_id}"), + backend_kind: AgentRuntimeBackendKind::Codex, + protocol: "uaa.agent.session".to_string(), + execution_scope: scope, + binary_path: PathBuf::from("/usr/bin/codex"), + } + } + + fn set_live(participant: &mut AgentRuntimeParticipantRecord) { + participant.transition_state(AgentRuntimeSessionState::Ready); + participant.mark_runtime_ownership_retained(); + participant.set_uaa_session_id("uaa_session"); + } + + fn live_orchestrator( + agent_id: &str, + orchestration_session_id: &str, + participant_id: &str, + ) -> AgentRuntimeParticipantRecord { + let mut participant = AgentRuntimeParticipantRecord::new_orchestrator_participant( + &descriptor(agent_id, AgentExecutionScope::Host), + orchestration_session_id.to_string(), + participant_id.to_string(), + format!("lease_{participant_id}"), + ) + .expect("orchestrator participant"); + set_live(&mut participant); + participant + } + + fn detached_orchestrator( + agent_id: &str, + orchestration_session_id: &str, + participant_id: &str, + ) -> AgentRuntimeParticipantRecord { + let mut participant = AgentRuntimeParticipantRecord::new_orchestrator_participant( + &descriptor(agent_id, AgentExecutionScope::Host), + orchestration_session_id.to_string(), + participant_id.to_string(), + format!("lease_{participant_id}"), + ) + .expect("orchestrator participant"); + participant.transition_state(AgentRuntimeSessionState::Ready); + participant.set_uaa_session_id("uaa_session"); + participant.mark_client_detached("owner detached cleanly"); + participant + } + + fn live_member( + agent_id: &str, + orchestration_session_id: &str, + participant_id: &str, + orchestrator_participant_id: &str, + ) -> AgentRuntimeParticipantRecord { + let mut participant = AgentRuntimeParticipantRecord::new_member_participant( + &descriptor(agent_id, AgentExecutionScope::World), + orchestration_session_id.to_string(), + participant_id.to_string(), + orchestrator_participant_id.to_string(), + None, + Some(super::super::session::AgentRuntimeParticipantWorldBinding { + world_id: "world-17".to_string(), + world_generation: 2, + }), + format!("lease_{participant_id}"), + ) + .expect("member participant"); + set_live(&mut participant); + participant + } + + fn active_parent(participant: &AgentRuntimeParticipantRecord) -> OrchestrationSessionRecord { + let mut parent = OrchestrationSessionRecord::new( + participant.handle.orchestration_session_id.clone(), + "trace_session".to_string(), + "/workspace".to_string(), + participant, + ); + parent.transition_state(OrchestrationSessionState::Active); + parent.bind_active_session_handle(participant.handle.participant_id.clone()); + parent + } + + fn parked_parent(participant: &AgentRuntimeParticipantRecord) -> OrchestrationSessionRecord { + let mut parent = active_parent(participant); + parent.mark_parked_resumable("owner detached cleanly"); + parent + } + + fn pending_inbox_item( + orchestration_session_id: &str, + item_id: &str, + kind: DurableInboxItemKind, + ) -> DurableInboxItemRecord { + DurableInboxItemRecord::new( + orchestration_session_id.to_string(), + item_id.to_string(), + kind, + Some(format!("message for {item_id}")), + ) + } + + fn write_legacy_handle_file( + store: &AgentRuntimeStateStore, + participant_id: &str, + agent_id: &str, + orchestration_session_id: &str, + ownership_valid: bool, + extras: Option, + ) { + fs::create_dir_all(store.handles_dir()).expect("create handles dir"); + let mut payload = json!({ + "session_handle_id": participant_id, + "orchestration_session_id": orchestration_session_id, + "agent_id": agent_id, + "backend_id": format!("cli:{agent_id}"), + "role": "orchestrator", + "protocol": "uaa.agent.session", + "execution": { "scope": "host" }, + "state": "ready", + "opened_at": "2026-04-24T18:30:00Z", + "last_transition_at": "2026-04-24T18:30:00Z", + "parent_session_handle_id": null, + "resumed_from_session_handle_id": null, + "internal": { + "resolved_agent_kind": "codex", + "resolved_binary_path": "/usr/bin/codex", + "shell_owner_pid": std::process::id(), + "lease_token": format!("lease_{participant_id}"), + "uaa_session_id": "uaa_session", + "cancel_supported": true, + "control_owner_retained": ownership_valid, + "event_stream_active": ownership_valid, + "completion_observer_retained": ownership_valid, + "ownership_mode": "attached_control", + "ownership_valid": ownership_valid, + "last_heartbeat_at": "2026-04-24T18:30:00Z" + } + }); + if let Some(extras) = extras { + merge_json(&mut payload, extras); + } + fs::write( + store.handles_dir().join(format!("{participant_id}.json")), + serde_json::to_vec_pretty(&payload).expect("serialize legacy handle"), + ) + .expect("write legacy handle"); + } + + fn write_flat_session_file( + store: &AgentRuntimeStateStore, + session: &OrchestrationSessionRecord, + ) { + fs::create_dir_all(store.sessions_dir()).expect("create sessions dir"); + fs::write( + store.orchestration_session_path(&session.orchestration_session_id), + serde_json::to_vec_pretty(session).expect("serialize session"), + ) + .expect("write flat session"); + } + + fn write_flat_participant_file( + store: &AgentRuntimeStateStore, + participant: &AgentRuntimeParticipantRecord, + ) { + fs::create_dir_all(store.participants_dir()).expect("create participants dir"); + fs::write( + store.participant_path(&participant.handle.participant_id), + serde_json::to_vec_pretty(participant).expect("serialize flat participant"), + ) + .expect("write flat participant"); + } + + fn write_canonical_session_file( + store: &AgentRuntimeStateStore, + session: &OrchestrationSessionRecord, + ) { + fs::create_dir_all(store.canonical_session_dir(&session.orchestration_session_id)) + .expect("create canonical session dir"); + fs::write( + store.canonical_session_path(&session.orchestration_session_id), + serde_json::to_vec_pretty(session).expect("serialize canonical session"), + ) + .expect("write canonical session"); + } + + fn write_canonical_participant_file( + store: &AgentRuntimeStateStore, + participant: &AgentRuntimeParticipantRecord, + ) { + fs::create_dir_all( + store.canonical_participants_dir(&participant.handle.orchestration_session_id), + ) + .expect("create canonical participants dir"); + fs::write( + store.canonical_participant_path( + &participant.handle.orchestration_session_id, + &participant.handle.participant_id, + ), + serde_json::to_vec_pretty(participant).expect("serialize canonical participant"), + ) + .expect("write canonical participant"); + } + + fn merge_json(target: &mut Value, extra: Value) { + match (target, extra) { + (Value::Object(target), Value::Object(extra)) => { + for (key, value) in extra { + merge_json(target.entry(key).or_insert(Value::Null), value); + } + } + (target, extra) => *target = extra, + } + } + + fn with_store(test: impl FnOnce(&AgentRuntimeStateStore)) { + let temp = TempDir::new().expect("tempdir"); + std::env::set_var("SUBSTRATE_HOME", temp.path()); + let store = AgentRuntimeStateStore::new().expect("state store"); + test(&store); + std::env::remove_var("SUBSTRATE_HOME"); + } + + #[test] + #[serial_test::serial] + fn participants_write_load_roundtrip() { + with_store(|store| { + let participant = live_orchestrator("codex", "sess_roundtrip", "ash_roundtrip"); + store + .persist_participant(&participant) + .expect("persist participant"); + + let loaded = store + .load_participant("ash_roundtrip") + .expect("load participant") + .expect("participant should exist"); + assert_eq!(loaded, participant); + assert!(store.participant_path("ash_roundtrip").exists()); + }); + } + + #[test] + #[serial_test::serial] + fn dual_read_prefers_participant_file_for_same_identity() { + with_store(|store| { + let participant = live_orchestrator("codex", "sess_dual", "ash_dual"); + store + .persist_participant(&participant) + .expect("persist participant"); + write_legacy_handle_file(store, "ash_dual", "legacy-agent", "sess_dual", true, None); + + let loaded = store + .load_participant("ash_dual") + .expect("load participant") + .expect("participant should exist"); + assert_eq!(loaded.handle.agent_id, "codex"); + + let participants = store.list_participants().expect("list participants"); + assert_eq!(participants.len(), 1); + assert_eq!(participants[0].handle.agent_id, "codex"); + }); + } + + #[test] + #[serial_test::serial] + fn load_session_prefers_canonical_objects_and_keeps_flat_participant_fallbacks() { + with_store(|store| { + let canonical_orchestrator = + live_orchestrator("codex", "sess_precedence", "ash_primary"); + let mut flat_orchestrator = + live_orchestrator("legacy", "sess_precedence", "ash_primary"); + flat_orchestrator.internal.latest_run_id = Some("run-flat".to_string()); + let flat_member = live_member( + "claude_code", + "sess_precedence", + "ash_member", + "ash_primary", + ); + + let canonical_parent = active_parent(&canonical_orchestrator); + let flat_parent = active_parent(&flat_orchestrator); + + store + .persist_participant(&flat_orchestrator) + .expect("persist flat orchestrator"); + store + .persist_participant(&flat_member) + .expect("persist flat member"); + write_flat_session_file(store, &flat_parent); + write_canonical_participant_file(store, &canonical_orchestrator); + write_canonical_session_file(store, &canonical_parent); + + let session = store + .load_session("sess_precedence") + .expect("load session") + .expect("session exists"); + assert!( + session.warnings.is_empty(), + "complete record should not warn" + ); + assert!( + session.is_complete(), + "canonical parent plus live participant should be complete" + ); + assert_eq!(session.session.orchestrator_agent_id, "codex"); + assert_eq!( + session + .participants + .iter() + .find(|participant| participant.handle.participant_id == "ash_primary") + .expect("orchestrator participant") + .handle + .agent_id, + "codex" + ); + assert!( + session + .participants + .iter() + .any(|participant| participant.handle.participant_id == "ash_member"), + "canonical parent must not erase flat participant compatibility fallback" + ); + }); + } + + #[test] + #[serial_test::serial] + fn persist_writes_canonical_and_flat_compatibility_layouts() { + with_store(|store| { + let participant = live_orchestrator("codex", "sess_dual_write", "ash_dual_write"); + let parent = active_parent(&participant); + + store + .persist_orchestration_session(&parent) + .expect("persist parent"); + store + .persist_participant(&participant) + .expect("persist participant"); + + assert!( + store.canonical_session_path("sess_dual_write").is_file(), + "canonical session.json must be written" + ); + assert!( + store + .orchestration_session_path("sess_dual_write") + .is_file(), + "flat compatibility parent session must remain readable during cutover" + ); + assert!( + store + .canonical_participant_path("sess_dual_write", "ash_dual_write") + .is_file(), + "canonical participant record must be written" + ); + assert!( + store.participant_path("ash_dual_write").is_file(), + "flat compatibility participant record must remain readable during cutover" + ); + assert!( + store + .canonical_lease_path("sess_dual_write", "ash_dual_write") + .is_file(), + "canonical lease must be written" + ); + assert!( + store.lease_path("ash_dual_write").is_file(), + "flat compatibility lease must remain readable during cutover" + ); + }); + } + + #[test] + #[serial_test::serial] + fn persist_inbox_item_updates_detached_pending_count_and_writes_canonical_artifact() { + with_store(|store| { + let participant = + detached_orchestrator("codex", "sess_inbox_pending", "ash_inbox_pending"); + let parent = parked_parent(&participant); + store + .persist_orchestration_session(&parent) + .expect("persist parent"); + store + .persist_participant(&participant) + .expect("persist participant"); + + let item = pending_inbox_item( + "sess_inbox_pending", + "item_approval", + DurableInboxItemKind::ApprovalRequired, + ); + store.persist_inbox_item(&item).expect("persist inbox item"); + + let loaded_session = store + .load_orchestration_session("sess_inbox_pending") + .expect("load orchestration session") + .expect("orchestration session exists"); + assert_eq!(loaded_session.pending_inbox_count, 1); + assert_eq!( + loaded_session.posture, + OrchestrationSessionPosture::AwaitingAttention + ); + assert!( + store + .canonical_inbox_item_path("sess_inbox_pending", "item_approval") + .is_file(), + "durable inbox artifacts must be stored canonically under sessions//inbox" + ); + + let loaded_item = store + .load_inbox_item("sess_inbox_pending", "item_approval") + .expect("load inbox item") + .expect("inbox item exists"); + assert_eq!(loaded_item.kind, DurableInboxItemKind::ApprovalRequired); + assert_eq!(loaded_item.state, DurableInboxItemState::Pending); + assert!(loaded_item.resolved_at.is_none()); + }); + } + + #[test] + #[serial_test::serial] + fn resolving_inbox_item_updates_pending_count_without_deleting_artifact() { + with_store(|store| { + let participant = + detached_orchestrator("codex", "sess_inbox_resolve", "ash_inbox_resolve"); + let parent = parked_parent(&participant); + store + .persist_orchestration_session(&parent) + .expect("persist parent"); + store + .persist_participant(&participant) + .expect("persist participant"); + + for (item_id, kind) in [ + ("item_completion", DurableInboxItemKind::CompletionNotice), + ("item_follow_up", DurableInboxItemKind::FollowUpMessage), + ("item_runtime", DurableInboxItemKind::RuntimeAlert), + ] { + store + .persist_inbox_item(&pending_inbox_item("sess_inbox_resolve", item_id, kind)) + .expect("persist inbox item"); + } + + let acknowledged = store + .acknowledge_inbox_item("sess_inbox_resolve", "item_completion") + .expect("acknowledge inbox item"); + assert_eq!(acknowledged.state, DurableInboxItemState::Acknowledged); + assert!(acknowledged.resolved_at.is_some()); + + let dismissed = store + .dismiss_inbox_item("sess_inbox_resolve", "item_follow_up") + .expect("dismiss inbox item"); + assert_eq!(dismissed.state, DurableInboxItemState::Dismissed); + assert!(dismissed.resolved_at.is_some()); + + let still_pending = store + .load_inbox_item("sess_inbox_resolve", "item_runtime") + .expect("load pending inbox item") + .expect("pending inbox item exists"); + assert_eq!(still_pending.state, DurableInboxItemState::Pending); + + let loaded_session = store + .load_orchestration_session("sess_inbox_resolve") + .expect("load orchestration session") + .expect("orchestration session exists"); + assert_eq!(loaded_session.pending_inbox_count, 1); + assert_eq!( + loaded_session.posture, + OrchestrationSessionPosture::AwaitingAttention + ); + + store + .dismiss_inbox_item("sess_inbox_resolve", "item_runtime") + .expect("dismiss final inbox item"); + let settled_session = store + .load_orchestration_session("sess_inbox_resolve") + .expect("load settled session") + .expect("settled session exists"); + assert_eq!(settled_session.pending_inbox_count, 0); + assert_eq!( + settled_session.posture, + OrchestrationSessionPosture::ParkedResumable + ); + + let items = store + .list_inbox_items("sess_inbox_resolve") + .expect("list inbox items"); + assert_eq!(items.len(), 3); + assert!( + items.iter().all(|item| { + store + .canonical_inbox_item_path("sess_inbox_resolve", &item.item_id) + .is_file() + }), + "resolved inbox items must remain durable artifacts instead of being deleted" + ); + }); + } + + #[test] + #[serial_test::serial] + fn list_live_participants_for_session_preserves_same_agent_siblings() { + with_store(|store| { + let orchestrator = live_orchestrator("codex", "sess_siblings", "ash_orchestrator"); + let member = live_member("codex", "sess_siblings", "ash_member", "ash_orchestrator"); + store + .persist_participant(&orchestrator) + .expect("persist orchestrator"); + store.persist_participant(&member).expect("persist member"); + + let participants = store + .list_live_participants_for_session("sess_siblings") + .expect("list live participants"); + assert_eq!(participants.len(), 2); + assert_eq!( + participants + .iter() + .map(|participant| participant.handle.participant_id.as_str()) + .collect::>(), + vec!["ash_orchestrator", "ash_member"] + ); + }); + } + + #[test] + #[serial_test::serial] + fn list_live_participants_filters_dead_owner_pid_rows() { + with_store(|store| { + let healthy = live_orchestrator("codex", "sess_live", "ash_live"); + let mut dead = live_orchestrator("codex", "sess_dead", "ash_dead"); + dead.internal.shell_owner_pid = u32::MAX; + store + .persist_participant(&healthy) + .expect("persist healthy"); + store.persist_participant(&dead).expect("persist dead"); + + let participants = store + .list_live_participants() + .expect("list live participants"); + assert_eq!(participants.len(), 1); + assert_eq!(participants[0].handle.participant_id, "ash_live"); + }); + } + + #[test] + #[serial_test::serial] + fn list_live_participants_filters_ownership_invalid_rows() { + with_store(|store| { + let healthy = live_orchestrator("codex", "sess_live", "ash_live"); + let mut invalid = live_orchestrator("codex", "sess_invalid", "ash_invalid"); + invalid.internal.ownership_valid = false; + invalid.internal.control_owner_retained = false; + invalid.internal.event_stream_active = false; + invalid.internal.completion_observer_retained = false; + store + .persist_participant(&healthy) + .expect("persist healthy"); + store + .persist_participant(&invalid) + .expect("persist invalid"); + + let participants = store + .list_live_participants() + .expect("list live participants"); + assert_eq!(participants.len(), 1); + assert_eq!(participants[0].handle.participant_id, "ash_live"); + }); + } + + #[test] + #[serial_test::serial] + fn list_sessions_discovers_participant_only_roots_and_excludes_them_from_live_results() { + with_store(|store| { + let live_session_orchestrator = live_orchestrator("codex", "sess_live", "ash_live"); + let live_parent = active_parent(&live_session_orchestrator); + let torn_orchestrator = live_orchestrator("codex", "sess_torn", "ash_torn"); + + store + .persist_participant(&live_session_orchestrator) + .expect("persist live orchestrator"); + store + .persist_orchestration_session(&live_parent) + .expect("persist live parent"); + store + .persist_participant(&torn_orchestrator) + .expect("persist torn participant"); + + let sessions = store.list_sessions().expect("list sessions"); + assert_eq!(sessions.len(), 2); + let torn = sessions + .iter() + .find(|record| record.orchestration_session_id() == "sess_torn") + .expect("participant-only torn root discovered"); + assert!( + !torn.is_complete(), + "participant-only torn roots must remain incomplete" + ); + assert!( + !torn.warnings.is_empty(), + "participant-only torn roots must surface warnings" + ); + + let live_sessions = store.list_live_sessions().expect("list live sessions"); + assert_eq!( + live_sessions + .iter() + .map(|record| record.orchestration_session_id()) + .collect::>(), + vec!["sess_live"] + ); + }); + } + + #[test] + #[serial_test::serial] + fn parent_only_torn_roots_degrade_with_warnings_instead_of_failing_discovery() { + with_store(|store| { + let participant = live_orchestrator("codex", "sess_parent_only", "ash_missing"); + let parent = active_parent(&participant); + store + .persist_orchestration_session(&parent) + .expect("persist parent"); + + let session = store + .load_session("sess_parent_only") + .expect("load parent-only torn root") + .expect("parent-only torn root exists"); + assert!( + !session.is_complete(), + "parent-only torn roots must stay incomplete" + ); + assert!( + !session.warnings.is_empty(), + "parent-only torn roots must surface warnings" + ); + assert!( + store + .list_live_sessions() + .expect("list live sessions") + .into_iter() + .all(|record| record.orchestration_session_id() != "sess_parent_only"), + "parent-only torn roots must not be promoted into live discovery" + ); + }); + } + + #[test] + #[serial_test::serial] + fn invalidate_stale_world_members_for_session_is_session_local_and_world_only() { + with_store(|store| { + let orchestrator = live_orchestrator("claude_code", "sess_live", "ash_orchestrator"); + let stale_member = + live_member("codex", "sess_live", "ash_member_old", "ash_orchestrator"); + let mut current_member = + live_member("codex", "sess_live", "ash_member_new", "ash_orchestrator"); + current_member.handle.world_generation = Some(3); + let mut host_member = AgentRuntimeParticipantRecord::new_member_participant( + &descriptor("codex", AgentExecutionScope::Host), + "sess_live".to_string(), + "ash_host_member".to_string(), + "ash_orchestrator".to_string(), + None, + None, + "lease_host".to_string(), + ) + .expect("host member"); + set_live(&mut host_member); + let stale_other_session = live_member( + "codex", + "sess_other", + "ash_member_other", + "ash_orchestrator", + ); + + store + .persist_participant(&orchestrator) + .expect("persist orchestrator"); + store + .persist_participant(&stale_member) + .expect("persist stale member"); + store + .persist_participant(¤t_member) + .expect("persist current member"); + store + .persist_participant(&host_member) + .expect("persist host member"); + store + .persist_participant(&stale_other_session) + .expect("persist other session member"); + + let invalidated = store + .invalidate_stale_world_members_for_session("sess_live", 3) + .expect("invalidate stale members"); + + assert_eq!(invalidated, vec!["ash_member_old"]); + assert_eq!( + store + .load_participant("ash_member_old") + .expect("load stale member") + .expect("stale member exists") + .handle + .state, + AgentRuntimeSessionState::Invalidated + ); + assert!(store + .load_participant("ash_member_new") + .expect("load current member") + .expect("current member exists") + .is_authoritative_live()); + assert!(store + .load_participant("ash_host_member") + .expect("load host member") + .expect("host member exists") + .is_authoritative_live()); + assert!(store + .load_participant("ash_member_other") + .expect("load other session member") + .expect("other session member exists") + .is_authoritative_live()); + }); + } + + #[test] + #[serial_test::serial] + fn invalidate_stale_world_members_for_session_is_idempotent() { + with_store(|store| { + let orchestrator = live_orchestrator("claude_code", "sess_live", "ash_orchestrator"); + let stale_member = + live_member("codex", "sess_live", "ash_member_old", "ash_orchestrator"); + + store + .persist_participant(&orchestrator) + .expect("persist orchestrator"); + store + .persist_participant(&stale_member) + .expect("persist stale member"); + + let first = store + .invalidate_stale_world_members_for_session("sess_live", 3) + .expect("first invalidation"); + let second = store + .invalidate_stale_world_members_for_session("sess_live", 3) + .expect("second invalidation"); + + assert_eq!(first, vec!["ash_member_old"]); + assert!(second.is_empty(), "second sweep must be a no-op"); + }); + } + + #[test] + #[serial_test::serial] + fn list_invalidated_participants_reads_authoritative_tombstones_only() { + with_store(|store| { + let mut participant = + live_member("codex", "sess_live", "ash_member_old", "ash_orchestrator"); + participant.invalidate_for_world_generation_rollover(); + store + .persist_participant(&participant) + .expect("persist invalidated participant"); + write_legacy_handle_file( + store, + "ash_legacy_only", + "codex", + "sess_live", + false, + Some(json!({ + "role": "member", + "execution": { "scope": "world" }, + "state": "invalidated", + "world_id": "world-17", + "world_generation": 1, + "orchestrator_participant_id": "ash_orchestrator", + "internal": { + "ownership_mode": "member_runtime" + } + })), + ); + + let invalidated = store + .list_invalidated_participants() + .expect("list invalidated participants"); + + assert_eq!(invalidated.len(), 1); + assert_eq!(invalidated[0].handle.participant_id, "ash_member_old"); + }); + } + + #[test] + #[serial_test::serial] + fn list_invalidated_participants_across_sources_includes_legacy_fallback_tombstones() { + with_store(|store| { + let mut participant = + live_member("codex", "sess_live", "ash_member_old", "ash_orchestrator"); + participant.invalidate_for_world_generation_rollover(); + store + .persist_participant(&participant) + .expect("persist invalidated participant"); + write_legacy_handle_file( + store, + "ash_legacy_only", + "codex", + "sess_live", + false, + Some(json!({ + "role": "member", + "execution": { "scope": "world" }, + "state": "invalidated", + "world_id": "world-17", + "world_generation": 1, + "orchestrator_participant_id": "ash_orchestrator", + "internal": { + "ownership_mode": "member_runtime" + } + })), + ); + + let invalidated = store + .list_invalidated_participants_across_sources() + .expect("list invalidated participants across sources"); + + assert_eq!(invalidated.len(), 2); + let mut participant_ids = invalidated + .iter() + .map(|participant| participant.handle.participant_id.as_str()) + .collect::>(); + participant_ids.sort(); + assert_eq!(participant_ids, vec!["ash_legacy_only", "ash_member_old"]); + }); + } + + #[test] + #[serial_test::serial] + fn resolve_live_orchestrator_participant_fails_closed_on_ambiguity() { + with_store(|store| { + let participant_a = live_orchestrator("codex", "sess_a", "ash_a"); + let participant_b = live_orchestrator("codex", "sess_b", "ash_b"); + let parent_a = active_parent(&participant_a); + let parent_b = active_parent(&participant_b); + store + .persist_participant(&participant_a) + .expect("persist participant a"); + store + .persist_participant(&participant_b) + .expect("persist participant b"); + store + .persist_orchestration_session(&parent_a) + .expect("persist parent a"); + store + .persist_orchestration_session(&parent_b) + .expect("persist parent b"); + + let err = store + .resolve_live_orchestrator_participant("codex") + .expect_err("ambiguous orchestrators must fail closed"); + assert!(err.to_string().contains( + "multiple active orchestration session candidates found for agent codex" + )); + }); + } + + #[test] + #[serial_test::serial] + fn resolve_public_control_target_rejects_active_session_handle_selector() { + with_store(|store| { + let participant = live_orchestrator("codex", "sess_public", "ash_selected"); + let parent = active_parent(&participant); + store + .persist_orchestration_session(&parent) + .expect("persist parent"); + store + .persist_participant(&participant) + .expect("persist participant"); + + let err = store + .resolve_public_control_target("ash_selected", PublicControlAction::Stop) + .expect_err("non-canonical active_session_handle_id selectors must be rejected"); + assert!(err.to_string().contains("unknown_session")); + assert!(err.to_string().contains("active_session_handle_id")); + }); + } + + #[test] + #[serial_test::serial] + fn resolve_public_control_target_rejects_internal_uaa_selector() { + with_store(|store| { + let participant = live_orchestrator("codex", "sess_public", "ash_selected"); + let parent = active_parent(&participant); + store + .persist_orchestration_session(&parent) + .expect("persist parent"); + store + .persist_participant(&participant) + .expect("persist participant"); + + let err = store + .resolve_public_control_target("uaa_session", PublicControlAction::Fork) + .expect_err("internal uaa session selectors must be rejected"); + assert!(err.to_string().contains("unknown_session")); + assert!(err.to_string().contains("internal.uaa_session_id")); + }); + } + + #[test] + #[serial_test::serial] + fn resolve_public_control_target_resume_rejects_already_owned_sessions() { + with_store(|store| { + let participant = live_orchestrator("codex", "sess_resume_live", "ash_selected"); + let parent = active_parent(&participant); + store + .persist_orchestration_session(&parent) + .expect("persist parent"); + store + .persist_participant(&participant) + .expect("persist participant"); + + let err = store + .resolve_public_control_target("sess_resume_live", PublicControlAction::Resume) + .expect_err("live retained ownership must reject public resume"); + assert!(err.to_string().contains("session_already_owned")); + }); + } + + #[test] + #[serial_test::serial] + fn resolve_public_control_target_allows_resume_for_parked_session() { + with_store(|store| { + let participant = detached_orchestrator("codex", "sess_resume_parked", "ash_detached"); + let mut parent = active_parent(&participant); + parent.mark_parked_resumable("owner detached cleanly"); + store + .persist_orchestration_session(&parent) + .expect("persist parent"); + store + .persist_participant(&participant) + .expect("persist participant"); + + let target = store + .resolve_public_control_target("sess_resume_parked", PublicControlAction::Resume) + .expect("parked session should remain resumable"); + assert_eq!( + target.active_participant.handle.participant_id, + "ash_detached" + ); + }); + } + + #[test] + #[serial_test::serial] + fn resolve_public_control_target_requires_internal_session_id_for_resume_and_fork() { + with_store(|store| { + let participant = + detached_orchestrator("codex", "sess_missing_internal", "ash_detached"); + let mut participant = participant; + participant.internal.uaa_session_id = None; + participant.internal.resume_eligible = false; + let parent = active_parent(&participant); + store + .persist_orchestration_session(&parent) + .expect("persist parent"); + store + .persist_participant(&participant) + .expect("persist participant"); + + let resume_err = store + .resolve_public_control_target("sess_missing_internal", PublicControlAction::Resume) + .expect_err("resume must require internal.uaa_session_id"); + assert!(resume_err.to_string().contains("owner_unreachable")); + + let fork_err = store + .resolve_public_control_target("sess_missing_internal", PublicControlAction::Fork) + .expect_err("fork must require internal.uaa_session_id"); + assert!(fork_err.to_string().contains("owner_unreachable")); + }); + } + + #[test] + #[serial_test::serial] + fn resolve_public_turn_target_reports_detached_posture_for_parked_host_session() { + with_store(|store| { + let participant = detached_orchestrator("codex", "sess_turn_parked", "ash_detached"); + let mut parent = active_parent(&participant); + parent.mark_parked_resumable("owner detached cleanly"); + store + .persist_orchestration_session(&parent) + .expect("persist parent"); + store + .persist_participant(&participant) + .expect("persist participant"); + + let target = store + .resolve_public_turn_target("sess_turn_parked", "cli:codex") + .expect("parked host target"); + assert_eq!( + target.session_posture, + PublicSessionPosture::DetachedReattachable + ); + }); + } + + #[test] + #[serial_test::serial] + fn resolve_public_turn_target_recovers_stale_attached_host_owner_as_detached() { + with_store(|store| { + let mut participant = live_orchestrator("codex", "sess_turn_stale", "ash_stale"); + participant.internal.shell_owner_pid = 999_999_999; + let parent = active_parent(&participant); + store + .persist_orchestration_session(&parent) + .expect("persist parent"); + store + .persist_participant(&participant) + .expect("persist participant"); + + let target = store + .resolve_public_turn_target("sess_turn_stale", "cli:codex") + .expect("stale attached host target"); + assert_eq!( + target.session_posture, + PublicSessionPosture::DetachedReattachable + ); + }); + } + + #[test] + #[serial_test::serial] + fn resolve_public_turn_target_does_not_recover_stale_attached_owner_when_live_successor_exists() + { + with_store(|store| { + let mut stale = live_orchestrator("codex", "sess_turn_stale_blocked", "ash_stale"); + stale.internal.shell_owner_pid = 999_999_999; + let live_successor = + live_orchestrator("codex", "sess_turn_stale_blocked", "ash_successor"); + let parent = active_parent(&stale); + store + .persist_orchestration_session(&parent) + .expect("persist parent"); + store.persist_participant(&stale).expect("persist stale"); + store + .persist_participant(&live_successor) + .expect("persist live successor"); + + let target = store + .resolve_public_turn_target("sess_turn_stale_blocked", "cli:codex") + .expect("stale attached host target with live successor should still resolve"); + assert_eq!(target.session_posture, PublicSessionPosture::Terminal); + }); + } + + #[test] + #[serial_test::serial] + fn startup_prompt_replay_state_allows_replay_only_before_acceptance() { + with_store(|store| { + let participant = live_orchestrator("codex", "sess_startup_prompt", "ash_start"); + let mut parent = active_parent(&participant); + parent.initialize_startup_prompt("ash_start"); + store + .persist_orchestration_session(&parent) + .expect("persist parent"); + store + .persist_participant(&participant) + .expect("persist participant"); + + assert_eq!( + store + .startup_prompt_replay_state("sess_startup_prompt", "ash_start") + .expect("pending replay state"), + StartupPromptReplayState::PendingAcceptance + ); + + let mut accepted_parent = parent.clone(); + accepted_parent.mark_startup_prompt_accepted("ash_start"); + store + .persist_orchestration_session(&accepted_parent) + .expect("persist accepted parent"); + assert_eq!( + store + .startup_prompt_replay_state("sess_startup_prompt", "ash_start") + .expect("accepted replay state"), + StartupPromptReplayState::AcceptedOrTerminal + ); + }); + } + + #[test] + #[serial_test::serial] + fn hidden_owner_helper_launch_classifier_accepts_detached_attention_needed_session() { + with_store(|store| { + let participant = + detached_orchestrator("codex", "sess_ready_attention", "ash_detached"); + let parent = parked_parent(&participant); + store + .persist_orchestration_session(&parent) + .expect("persist parent"); + store + .persist_participant(&participant) + .expect("persist participant"); + store + .persist_inbox_item(&pending_inbox_item( + "sess_ready_attention", + "item_attention", + DurableInboxItemKind::ApprovalRequired, + )) + .expect("persist pending inbox item"); + + let continuity = store + .classify_hidden_owner_helper_launch_continuity( + "sess_ready_attention", + "ash_detached", + true, + ) + .expect("classify continuity"); + assert_eq!( + continuity, + HiddenOwnerHelperLaunchContinuity::DetachedReconciled( + OrchestrationSessionPosture::AwaitingAttention, + ) + ); + + let readiness = store + .classify_hidden_owner_helper_launch_readiness( + "sess_ready_attention", + "ash_detached", + true, + ) + .expect("classify readiness"); + assert_eq!( + readiness, + HiddenOwnerHelperLaunchReadiness::ReadyDetached( + OrchestrationSessionPosture::AwaitingAttention, + ) + ); + }); + } + + #[test] + #[serial_test::serial] + fn hidden_owner_helper_launch_classifier_rejects_detached_posture_pending_mismatch() { + with_store(|store| { + let participant = detached_orchestrator("codex", "sess_bad_detached", "ash_detached"); + let mut parent = parked_parent(&participant); + parent.pending_inbox_count = 1; + write_canonical_session_file(store, &parent); + write_flat_session_file(store, &parent); + store + .persist_participant(&participant) + .expect("persist participant"); + + let err = store + .classify_hidden_owner_helper_launch_readiness( + "sess_bad_detached", + "ash_detached", + true, + ) + .expect_err("invalid detached posture must fail closed"); + assert!(err.to_string().contains("invalid session record")); + + let loaded = store + .load_session("sess_bad_detached") + .expect_err("invalid detached posture must fail closed during load"); + assert!(loaded.to_string().contains("invalid session record")); + }); + } + + #[test] + #[serial_test::serial] + fn hidden_owner_helper_launch_continuity_reports_stale_attached_truth_separately() { + with_store(|store| { + let mut participant = live_orchestrator("codex", "sess_stale_attached", "ash_stale"); + participant.internal.shell_owner_pid = 999_999_999; + let parent = active_parent(&participant); + store + .persist_orchestration_session(&parent) + .expect("persist parent"); + store + .persist_participant(&participant) + .expect("persist participant"); + + let continuity = store + .classify_hidden_owner_helper_launch_continuity( + "sess_stale_attached", + "ash_stale", + true, + ) + .expect("classify continuity"); + assert_eq!( + continuity, + HiddenOwnerHelperLaunchContinuity::StaleAttachedTruth + ); + + let readiness = store + .classify_hidden_owner_helper_launch_readiness( + "sess_stale_attached", + "ash_stale", + true, + ) + .expect("classify readiness"); + assert_eq!(readiness, HiddenOwnerHelperLaunchReadiness::Pending); + }); + } + + #[test] + #[serial_test::serial] + fn resolve_public_turn_target_rejects_active_session_handle_selector() { + with_store(|store| { + let participant = live_orchestrator("codex", "sess_public_turn", "ash_selected"); + let parent = active_parent(&participant); + store + .persist_orchestration_session(&parent) + .expect("persist parent"); + store + .persist_participant(&participant) + .expect("persist participant"); + + let err = store + .resolve_public_turn_target("ash_selected", "cli:codex") + .expect_err("non-canonical active_session_handle_id selectors must be rejected"); + assert!(err.to_string().contains("noncanonical_session_selector")); + assert!(err.to_string().contains("active_session_handle_id")); + }); + } + + #[test] + #[serial_test::serial] + fn resolve_public_turn_target_rejects_internal_uaa_selector() { + with_store(|store| { + let participant = live_orchestrator("codex", "sess_public_turn", "ash_selected"); + let parent = active_parent(&participant); + store + .persist_orchestration_session(&parent) + .expect("persist parent"); + store + .persist_participant(&participant) + .expect("persist participant"); + + let err = store + .resolve_public_turn_target("uaa_session", "cli:codex") + .expect_err("internal uaa session selectors must be rejected"); + assert!(err.to_string().contains("noncanonical_session_selector")); + assert!(err.to_string().contains("internal.uaa_session_id")); + }); + } + + #[test] + #[serial_test::serial] + fn resolve_public_turn_target_requires_exact_world_member_linkage() { + with_store(|store| { + let orchestrator = live_orchestrator("codex", "sess_world_turn_stale", "ash_selected"); + let mut member = live_member( + "codex", + "sess_world_turn_stale", + "ash_member", + "ash_stale_owner", + ); + member.handle.backend_id = "cli:world-member".to_string(); + let mut parent = active_parent(&orchestrator); + parent.set_world_binding("world-17", 2); + + store + .persist_orchestration_session(&parent) + .expect("persist parent"); + store + .persist_participant(&orchestrator) + .expect("persist orchestrator"); + store.persist_participant(&member).expect("persist member"); + + let err = store + .resolve_public_turn_target("sess_world_turn_stale", "cli:world-member") + .expect_err("stale world-member linkage must fail closed"); + assert!(err.to_string().contains("stale_linkage")); + }); + } + + #[test] + #[serial_test::serial] + fn resolve_public_control_target_stop_allows_detached_resumable_session() { + with_store(|store| { + let mut participant = + detached_orchestrator("codex", "sess_stop_detached", "ash_selected"); + participant.set_uaa_session_id("uaa_session"); + let mut parent = active_parent(&participant); + parent.mark_parked_resumable("owner detached cleanly"); + store + .persist_orchestration_session(&parent) + .expect("persist parent"); + store + .persist_participant(&participant) + .expect("persist participant"); + + let target = store + .resolve_public_control_target("sess_stop_detached", PublicControlAction::Stop) + .expect("stop must remain available for parked resumable sessions"); + assert_eq!( + target.session_posture, + PublicSessionPosture::DetachedReattachable + ); + assert_eq!( + target.active_participant.handle.participant_id, + "ash_selected" + ); + }); + } + + #[test] + #[serial_test::serial] + fn resolve_public_control_target_stop_requires_resume_contract_for_detached_session() { + with_store(|store| { + let mut participant = + detached_orchestrator("codex", "sess_stop_missing_internal", "ash_selected"); + participant.internal.uaa_session_id = None; + participant.internal.resume_eligible = false; + let mut parent = active_parent(&participant); + parent.mark_parked_resumable("owner detached cleanly"); + store + .persist_orchestration_session(&parent) + .expect("persist parent"); + store + .persist_participant(&participant) + .expect("persist participant"); + + let err = store + .resolve_public_control_target( + "sess_stop_missing_internal", + PublicControlAction::Stop, + ) + .expect_err("detached stop must fail closed without a resumable retained owner"); + assert!(err.to_string().contains("owner_unreachable")); + }); + } + + #[test] + #[serial_test::serial] + fn resolve_public_control_target_stop_allows_detached_attention_session() { + with_store(|store| { + let participant = + detached_orchestrator("codex", "sess_stop_attention", "ash_attention"); + let parent = parked_parent(&participant); + store + .persist_orchestration_session(&parent) + .expect("persist parent"); + store + .persist_participant(&participant) + .expect("persist participant"); + store + .persist_inbox_item(&pending_inbox_item( + "sess_stop_attention", + "item_attention", + DurableInboxItemKind::ApprovalRequired, + )) + .expect("persist pending inbox item"); + + let target = store + .resolve_public_control_target("sess_stop_attention", PublicControlAction::Stop) + .expect("stop must remain available for attention-needed durable sessions"); + assert_eq!( + target.session_posture, + PublicSessionPosture::DetachedReattachable + ); + assert_eq!( + target.session.posture, + OrchestrationSessionPosture::AwaitingAttention + ); + assert_eq!(target.session.pending_inbox_count, 1); + assert_eq!( + target.active_participant.handle.participant_id, + "ash_attention" + ); + }); + } + + #[test] + #[serial_test::serial] + fn status_visible_participants_include_detached_authoritative_orchestrator() { + with_store(|store| { + let mut participant = + detached_orchestrator("codex", "sess_status_detached", "ash_detached"); + participant.set_uaa_session_id("uaa_session"); + let mut parent = active_parent(&participant); + parent.mark_parked_resumable("owner detached cleanly"); + store + .persist_orchestration_session(&parent) + .expect("persist parent"); + store + .persist_participant(&participant) + .expect("persist participant"); + + let record = store + .load_session("sess_status_detached") + .expect("load session") + .expect("session record exists"); + let visible = record.status_visible_participants(); + assert_eq!(visible.len(), 1); + assert_eq!(visible[0].handle.participant_id, "ash_detached"); + assert!(!visible[0].attached_client_present()); + }); + } + + #[test] + #[serial_test::serial] + fn status_visible_participants_include_awaiting_attention_orchestrator() { + with_store(|store| { + let participant = + detached_orchestrator("codex", "sess_status_attention", "ash_detached"); + let parent = parked_parent(&participant); + store + .persist_orchestration_session(&parent) + .expect("persist parent"); + store + .persist_participant(&participant) + .expect("persist participant"); + store + .persist_inbox_item(&pending_inbox_item( + "sess_status_attention", + "item_attention", + DurableInboxItemKind::RuntimeAlert, + )) + .expect("persist pending inbox item"); + + let record = store + .load_session("sess_status_attention") + .expect("load session") + .expect("session record exists"); + assert_eq!(record.session.pending_inbox_count, 1); + assert_eq!( + record.session.posture, + OrchestrationSessionPosture::AwaitingAttention + ); + let visible = record.status_visible_participants(); + assert_eq!(visible.len(), 1); + assert_eq!(visible[0].handle.participant_id, "ash_detached"); + }); + } + + #[test] + #[serial_test::serial] + fn resolve_public_control_target_requires_exact_parent_linkage() { + with_store(|store| { + let participant = live_orchestrator("codex", "sess_stale_linkage", "ash_selected"); + let mut parent = active_parent(&participant); + parent.orchestrator_agent_id = "claude_code".to_string(); + store + .persist_orchestration_session(&parent) + .expect("persist parent"); + store + .persist_participant(&participant) + .expect("persist participant"); + + let err = store + .resolve_public_control_target("sess_stale_linkage", PublicControlAction::Stop) + .expect_err("mismatched active parent linkage must fail closed"); + assert!(err.to_string().contains("stale_linkage")); + }); + } + + #[test] + #[serial_test::serial] + fn resolve_public_control_target_enforces_linux_first_world_posture() { + with_store(|store| { + let orchestrator = live_orchestrator("codex", "sess_world_posture", "ash_selected"); + let member = live_member("codex", "sess_world_posture", "ash_member", "ash_selected"); + let mut parent = active_parent(&orchestrator); + parent.set_world_binding("world-17", 2); + store + .persist_orchestration_session(&parent) + .expect("persist parent"); + store + .persist_participant(&orchestrator) + .expect("persist orchestrator"); + store.persist_participant(&member).expect("persist member"); + + let result = store + .resolve_public_control_target("sess_world_posture", PublicControlAction::Stop); + + #[cfg(target_os = "linux")] + { + let resolved = result.expect("linux should accept world-sensitive control posture"); + assert_eq!( + resolved.session.orchestration_session_id, + "sess_world_posture" + ); + } + + #[cfg(not(target_os = "linux"))] + { + let err = result.expect_err( + "non-linux platforms must fail closed for world-sensitive control posture", + ); + assert!(err.to_string().contains("unsupported_platform_or_posture")); + } + }); + } + + #[test] + #[serial_test::serial] + fn resolve_single_live_session_fails_closed_on_stale_active_handle_reference() { + with_store(|store| { + let selected = live_orchestrator("codex", "sess_live", "ash_selected"); + let mut parent = active_parent(&selected); + parent.bind_active_session_handle("ash_missing"); + + store + .persist_orchestration_session(&parent) + .expect("persist parent"); + store + .persist_participant(&selected) + .expect("persist selected participant"); + + let err = store + .resolve_single_live_session_for_agent("codex") + .expect_err("stale active handle references must fail closed"); + assert!(err.to_string().contains( + "active orchestration session sess_live references missing participant ash_missing" + )); + }); + } + + #[test] + #[serial_test::serial] + fn list_status_sessions_for_agent_includes_participant_visible_torn_roots_without_relaxing_strict_resolution( + ) { + with_store(|store| { + let selected = live_orchestrator("codex", "sess_torn_visible", "ash_selected"); + let mut drifted_parent = active_parent(&selected); + drifted_parent.orchestrator_agent_id = "claude_code".to_string(); + + store + .persist_orchestration_session(&drifted_parent) + .expect("persist drifted parent"); + store + .persist_participant(&selected) + .expect("persist selected participant"); + + let status_sessions = store + .list_status_sessions_for_agent("codex") + .expect("list status sessions"); + let torn = status_sessions + .into_iter() + .find(|record| record.orchestration_session_id() == "sess_torn_visible") + .expect("status seam should retain participant-visible torn root"); + + assert!( + !torn.is_complete(), + "status seam must degrade torn roots instead of authorizing control" + ); + assert!( + !torn.warnings.is_empty(), + "status seam must preserve degraded warnings from build_session_record" + ); + + let err = store + .resolve_single_live_session_for_agent("codex") + .expect_err("strict selector must remain fail closed"); + assert!(err.to_string().contains( + "live host-scoped orchestrator participant exists for agent codex without an active parent session" + )); + }); + } + + #[test] + #[serial_test::serial] + fn load_session_prefers_canonical_participant_over_conflicting_legacy_handle_fallback() { + with_store(|store| { + let canonical_orchestrator = + live_orchestrator("codex", "sess_canonical_legacy", "ash_primary"); + let canonical_parent = active_parent(&canonical_orchestrator); + + write_legacy_handle_file( + store, + "ash_primary", + "legacy", + "sess_legacy_conflict", + true, + Some(json!({ + "last_transition_at": "2026-04-24T18:31:00Z", + "internal": { + "latest_run_id": "run-legacy" + } + })), + ); + write_canonical_participant_file(store, &canonical_orchestrator); + write_canonical_session_file(store, &canonical_parent); + + let session = store + .load_session("sess_canonical_legacy") + .expect("load session") + .expect("session exists"); + let selected = session + .participants + .iter() + .find(|participant| participant.handle.participant_id == "ash_primary") + .expect("selected participant"); + + assert_eq!( + selected.handle.agent_id, "codex", + "canonical participant must outrank conflicting legacy-handle fallback" + ); + assert_eq!( + selected.handle.orchestration_session_id, "sess_canonical_legacy", + "canonical participant must keep the canonical parent linkage instead of drifting to the legacy handle" + ); + }); + } + + #[test] + #[serial_test::serial] + fn load_session_prefers_flat_participant_over_conflicting_legacy_handle_fallback() { + with_store(|store| { + let flat_orchestrator = live_orchestrator("codex", "sess_flat_legacy", "ash_primary"); + let flat_parent = active_parent(&flat_orchestrator); + + write_legacy_handle_file( + store, + "ash_primary", + "legacy", + "sess_legacy_conflict", + true, + Some(json!({ + "last_transition_at": "2026-04-24T18:31:00Z", + "internal": { + "latest_run_id": "run-legacy" + } + })), + ); + write_flat_participant_file(store, &flat_orchestrator); + write_flat_session_file(store, &flat_parent); + + let session = store + .load_session("sess_flat_legacy") + .expect("load session") + .expect("session exists"); + let selected = session + .participants + .iter() + .find(|participant| participant.handle.participant_id == "ash_primary") + .expect("selected participant"); + + assert_eq!( + selected.handle.agent_id, "codex", + "flat compatibility participant must outrank conflicting legacy-handle fallback when the canonical child is absent" + ); + assert_eq!( + selected.handle.orchestration_session_id, "sess_flat_legacy", + "flat compatibility participant must keep the flat parent linkage instead of drifting to the legacy handle" + ); + }); + } + + #[test] + #[serial_test::serial] + fn resolve_single_live_session_fails_closed_on_inactive_selected_participant() { + with_store(|store| { + let mut selected = live_orchestrator("codex", "sess_live", "ash_selected"); + selected.internal.ownership_valid = false; + selected.internal.control_owner_retained = false; + selected.internal.event_stream_active = false; + selected.internal.completion_observer_retained = false; + let parent = active_parent(&selected); + + store + .persist_orchestration_session(&parent) + .expect("persist parent"); + store + .persist_participant(&selected) + .expect("persist inactive selected participant"); + + let err = store + .resolve_single_live_session_for_agent("codex") + .expect_err("inactive selected participant must fail closed"); + assert!(err.to_string().contains( + "active orchestration session sess_live references inactive participant ash_selected" + )); + }); + } + + #[test] + #[serial_test::serial] + fn persist_participant_rejects_stale_live_snapshot_after_terminal_snapshot() { + with_store(|store| { + let live = + live_orchestrator("codex", "sess_stale_participant", "ash_stale_participant"); + let mut invalidated = live.clone(); + invalidated.transition_state(AgentRuntimeSessionState::Invalidated); + invalidated.mark_terminal_state("attached control exited"); + + store + .persist_participant(&invalidated) + .expect("persist invalidated participant"); + store + .persist_participant(&live) + .expect("reject stale live participant snapshot"); + + let loaded = store + .load_participant("ash_stale_participant") + .expect("load participant") + .expect("participant exists"); + assert_eq!(loaded.handle.state, AgentRuntimeSessionState::Invalidated); + assert!(loaded.internal.terminal_observed_at.is_some()); + }); + } + + #[test] + #[serial_test::serial] + fn persist_orchestration_session_rejects_stale_active_snapshot_after_terminal_snapshot() { + with_store(|store| { + let participant = live_orchestrator("codex", "sess_stale_parent", "ash_stale_parent"); + let active = active_parent(&participant); + let mut invalidated = active.clone(); + invalidated.transition_state(OrchestrationSessionState::Invalidated); + invalidated.mark_terminal("attached control exited"); + + store + .persist_orchestration_session(&invalidated) + .expect("persist invalidated parent"); + store + .persist_orchestration_session(&active) + .expect("reject stale active parent snapshot"); + + let loaded = store + .load_orchestration_session("sess_stale_parent") + .expect("load orchestration session") + .expect("orchestration session exists"); + assert_eq!(loaded.state, OrchestrationSessionState::Invalidated); + assert!(loaded.closed_at.is_some()); + }); + } + + #[test] + #[serial_test::serial] + fn persist_orchestration_session_rejects_touched_allocating_regression_after_active() { + with_store(|store| { + let participant = + live_orchestrator("codex", "sess_regressed_parent", "ash_regressed_parent"); + let active = active_parent(&participant); + let mut allocating = OrchestrationSessionRecord::new( + "sess_regressed_parent".to_string(), + "trace_session".to_string(), + "/workspace".to_string(), + &participant, + ); + + store + .persist_orchestration_session(&active) + .expect("persist active parent"); + allocating.touch_active(); + store + .persist_orchestration_session(&allocating) + .expect("reject touched allocating regression"); + + let loaded = store + .load_orchestration_session("sess_regressed_parent") + .expect("load orchestration session") + .expect("orchestration session exists"); + assert_eq!(loaded.state, OrchestrationSessionState::Active); + assert_eq!( + loaded.active_session_handle_id.as_deref(), + Some("ash_regressed_parent") + ); + }); + } + + #[cfg(unix)] + #[test] + #[serial_test::serial] + fn path_hardening_ignores_symlinked_and_non_regular_entries() { + use std::os::unix::fs::symlink; + + with_store(|store| { + fs::create_dir_all(store.sessions_dir()).expect("create sessions dir"); + fs::create_dir_all(store.participants_dir()).expect("create participants dir"); + + let symlink_participant = live_orchestrator("codex", "sess_symlink", "ash_symlink"); + let real_root = store.substrate_home.join("real-session-root"); + fs::create_dir_all(real_root.join("participants")).expect("create real session root"); + fs::write( + real_root.join("session.json"), + serde_json::to_vec_pretty(&active_parent(&symlink_participant)) + .expect("serialize symlinked session"), + ) + .expect("write real session json"); + fs::write( + real_root.join("participants/ash_symlink.json"), + serde_json::to_vec_pretty(&symlink_participant) + .expect("serialize symlinked participant"), + ) + .expect("write real participant json"); + symlink(&real_root, store.sessions_dir().join("sess_symlink")) + .expect("symlink canonical root"); + + let external_participant_path = store.substrate_home.join("real-participant.json"); + fs::write( + &external_participant_path, + serde_json::to_vec_pretty(&symlink_participant) + .expect("serialize external participant"), + ) + .expect("write external participant"); + symlink( + &external_participant_path, + store.participants_dir().join("ash_symlink.json"), + ) + .expect("symlink participant"); + + fs::create_dir_all(store.sessions_dir().join("ignored.json")) + .expect("create non-regular session entry"); + fs::create_dir_all(store.participants_dir().join("ignored.json")) + .expect("create non-regular participant entry"); + + assert!( + store + .load_orchestration_session("sess_symlink") + .expect("load symlinked session") + .is_none(), + "symlinked canonical session roots must be ignored" + ); + assert!( + store + .load_participant("ash_symlink") + .expect("load symlinked participant") + .is_none(), + "symlinked participant entries must be ignored" + ); + assert!( + store + .list_sessions() + .expect("list sessions") + .into_iter() + .all(|record| record.orchestration_session_id() != "sess_symlink"), + "symlinked canonical roots must not be promoted into discovery" + ); + }); + } + + #[test] + #[serial_test::serial] + fn validation_helper_rejects_malformed_participant_json() { + with_store(|store| { + fs::create_dir_all(store.participants_dir()).expect("create participants dir"); + let payload = json!({ + "participant_id": "ash_bad", + "orchestration_session_id": "sess_bad", + "agent_id": "codex", + "backend_id": "cli:codex", + "role": "member", + "protocol": "uaa.agent.session", + "execution": { "scope": "world" }, + "state": "ready", + "opened_at": "2026-04-24T18:30:00Z", + "last_transition_at": "2026-04-24T18:30:00Z", + "world_id": "world-17", + "world_generation": 1, + "internal": { + "resolved_agent_kind": "codex", + "resolved_binary_path": "/usr/bin/codex", + "shell_owner_pid": std::process::id(), + "lease_token": "lease_bad", + "uaa_session_id": "uaa_session", + "cancel_supported": true, + "control_owner_retained": true, + "event_stream_active": true, + "completion_observer_retained": true, + "ownership_mode": "member_runtime", + "ownership_valid": true + } + }); + fs::write( + store.participant_path("ash_bad"), + serde_json::to_vec_pretty(&payload).expect("serialize payload"), + ) + .expect("write malformed participant"); + + let err = store + .load_participant("ash_bad") + .expect_err("malformed participant must fail validation"); + assert!(!err.to_string().is_empty()); + }); + } } diff --git a/crates/shell/src/execution/agent_runtime/validator.rs b/crates/shell/src/execution/agent_runtime/validator.rs index 2beeb6a1e..48ed05546 100644 --- a/crates/shell/src/execution/agent_runtime/validator.rs +++ b/crates/shell/src/execution/agent_runtime/validator.rs @@ -27,6 +27,18 @@ pub(crate) struct RuntimeRealizabilityError { pub reason: String, } +#[derive(Clone, Debug)] +pub(crate) struct MemberSelectionError { + pub exit_code: i32, + pub reason: String, +} + +#[derive(Clone, Debug)] +pub(crate) struct ExactBackendSelectionError { + pub exit_code: i32, + pub reason: String, +} + pub(crate) fn validate_orchestrator_selection<'a>( effective_config: &SubstrateConfig, inventory: &'a BTreeMap, @@ -110,7 +122,7 @@ pub(crate) fn validate_runtime_realizability( return Err(RuntimeRealizabilityError { exit_code: 2, reason: format!( - "selected orchestrator '{}' is not runtime-realizable by the shell-owned UAA runtime because config.kind={} is unsupported; only config.kind=cli is supported in v1", + "selected runtime '{}' is not runtime-realizable by the shell-owned UAA runtime because config.kind={} is unsupported; only config.kind=cli is supported in v1", entry.file.id, entry.file.config.kind.as_str() ), @@ -122,7 +134,7 @@ pub(crate) fn validate_runtime_realizability( return Err(RuntimeRealizabilityError { exit_code: 2, reason: format!( - "selected orchestrator '{}' is not runtime-realizable because cli.mode={} is unsupported; only cli.mode=persistent is supported for the first caller path", + "selected runtime '{}' is not runtime-realizable because cli.mode={} is unsupported; only cli.mode=persistent is supported for the first caller path", entry.file.id, match cli_mode { AgentCliMode::Persistent => "persistent", @@ -135,20 +147,24 @@ pub(crate) fn validate_runtime_realizability( let backend_kind = orchestrator_backend_kind(&entry.file.id).map_err(|err| RuntimeRealizabilityError { exit_code: 2, - reason: err.to_string(), + reason: err + .to_string() + .replace("selected orchestrator backend", "selected runtime backend"), })?; - let binary = entry.effective_cli_binary().ok_or_else(|| RuntimeRealizabilityError { - exit_code: 4, - reason: format!( - "selected orchestrator '{}' is not runtime-realizable because config.cli.binary is missing", + let binary = entry + .effective_cli_binary() + .ok_or_else(|| RuntimeRealizabilityError { + exit_code: 4, + reason: format!( + "selected runtime '{}' is not runtime-realizable because config.cli.binary is missing", entry.file.id ), - })?; + })?; let binary_path = which::which(binary).map_err(|err| RuntimeRealizabilityError { exit_code: 4, reason: format!( - "selected orchestrator '{}' is not runtime-realizable because config.cli.binary '{}' did not resolve on the host: {}", + "selected runtime '{}' is not runtime-realizable because config.cli.binary '{}' did not resolve on the host: {}", entry.file.id, binary, err ), })?; @@ -163,9 +179,463 @@ pub(crate) fn validate_runtime_realizability( }) } +pub(crate) fn validate_member_selection( + effective_config: &SubstrateConfig, + inventory: &BTreeMap, +) -> std::result::Result, MemberSelectionError> { + let mut selected = Vec::new(); + + for entry in inventory.values() { + if !entry.file.config.enabled + || entry.effective_scope(effective_config) != AgentExecutionScope::World + { + continue; + } + + if entry.file.config.protocol.as_deref() != Some(PURE_AGENT_PROTOCOL) { + return Err(MemberSelectionError { + exit_code: 2, + reason: format!( + "world-scoped member '{}' is not eligible because it does not advertise protocol '{}'", + entry.file.id, PURE_AGENT_PROTOCOL + ), + }); + } + + if let Some(capability) = + missing_required_orchestrator_capability(&entry.file.config.capabilities) + { + return Err(MemberSelectionError { + exit_code: 2, + reason: format!( + "world-scoped member '{}' is not eligible because it is missing required capability '{}'", + entry.file.id, capability + ), + }); + } + + let descriptor = + validate_runtime_realizability(entry, effective_config).map_err(|err| { + MemberSelectionError { + exit_code: err.exit_code, + reason: err.reason, + } + })?; + selected.push(descriptor); + } + + match selected.len() { + 0 => Ok(None), + 1 => Ok(selected.into_iter().next()), + _ => { + let agent_ids = selected + .iter() + .map(|descriptor| descriptor.agent_id.as_str()) + .collect::>() + .join(", "); + Err(MemberSelectionError { + exit_code: 2, + reason: format!( + "ambiguous world member selection: multiple eligible world-scoped members found ({agent_ids})" + ), + }) + } + } +} + +pub(crate) fn validate_exact_backend_selection( + effective_config: &SubstrateConfig, + inventory: &BTreeMap, + scope: AgentExecutionScope, + backend_id: &str, +) -> std::result::Result, ExactBackendSelectionError> { + let mut matches = inventory + .values() + .filter(|entry| { + entry.file.config.enabled + && entry.effective_scope(effective_config) == scope + && entry.derived_backend_id() == backend_id + }) + .collect::>(); + + match matches.len() { + 0 => Ok(None), + 1 => { + let entry = matches.pop().expect("single exact backend match"); + validate_exact_backend_entry(entry, effective_config, scope, backend_id) + } + _ => { + let agent_ids = matches + .iter() + .map(|entry| entry.file.id.as_str()) + .collect::>() + .join(", "); + Err(ExactBackendSelectionError { + exit_code: 2, + reason: format!( + "ambiguous exact backend selection: multiple {} runtime entries advertise backend '{}' ({agent_ids})", + runtime_scope_label(scope), + backend_id, + ), + }) + } + } +} + +fn validate_exact_backend_entry( + entry: &AgentInventoryEntryV1, + effective_config: &SubstrateConfig, + scope: AgentExecutionScope, + backend_id: &str, +) -> std::result::Result, ExactBackendSelectionError> { + if entry.file.config.protocol.as_deref() != Some(PURE_AGENT_PROTOCOL) { + return Err(ExactBackendSelectionError { + exit_code: 2, + reason: format!( + "selected {} runtime '{}' for backend '{}' does not advertise protocol '{}'", + runtime_scope_label(scope), + entry.file.id, + backend_id, + PURE_AGENT_PROTOCOL, + ), + }); + } + + if let Some(capability) = + missing_required_orchestrator_capability(&entry.file.config.capabilities) + { + return Err(ExactBackendSelectionError { + exit_code: 2, + reason: format!( + "selected {} runtime '{}' for backend '{}' is missing required capability '{}'", + runtime_scope_label(scope), + entry.file.id, + backend_id, + capability, + ), + }); + } + + let descriptor = validate_runtime_realizability(entry, effective_config).map_err(|err| { + ExactBackendSelectionError { + exit_code: err.exit_code, + reason: err.reason, + } + })?; + Ok(Some(descriptor)) +} + +fn runtime_scope_label(scope: AgentExecutionScope) -> &'static str { + match scope { + AgentExecutionScope::Host => "host-scoped", + AgentExecutionScope::World => "world-scoped", + } +} + pub(crate) fn runtime_realizability_error_exit_code(error: &RuntimeRealizabilityError) -> i32 { error.exit_code } +pub(crate) fn member_selection_error_exit_code(error: &MemberSelectionError) -> i32 { + error.exit_code +} + +pub(crate) fn exact_backend_selection_error_exit_code(error: &ExactBackendSelectionError) -> i32 { + error.exit_code +} + #[allow(dead_code)] fn _assert_result_type(_: Result) {} + +#[cfg(test)] +mod tests { + use super::{ + exact_backend_selection_error_exit_code, validate_exact_backend_selection, + validate_member_selection, validate_runtime_realizability, AgentRuntimeBackendKind, + ExactBackendSelectionError, MemberSelectionError, RuntimeSelectionDescriptor, + PURE_AGENT_PROTOCOL, + }; + use crate::execution::agent_inventory::{ + AgentCapabilitiesV1, AgentCliConfigV1, AgentConfigKind, AgentConfigV1, + AgentExecutionConfigV1, AgentFileV1, AgentInventoryEntryV1, + }; + use crate::execution::config_model::{AgentCliMode, AgentExecutionScope, SubstrateConfig}; + use std::collections::BTreeMap; + use std::path::PathBuf; + + fn make_entry( + agent_id: &str, + scope: AgentExecutionScope, + protocol: Option<&str>, + cli_mode: AgentCliMode, + capabilities: AgentCapabilitiesV1, + ) -> AgentInventoryEntryV1 { + AgentInventoryEntryV1 { + path: PathBuf::from(format!("{agent_id}.yaml")), + file: AgentFileV1 { + version: 1, + id: agent_id.to_string(), + config: AgentConfigV1 { + enabled: true, + kind: AgentConfigKind::Cli, + protocol: protocol.map(str::to_string), + execution: AgentExecutionConfigV1 { scope: Some(scope) }, + cli: Some(AgentCliConfigV1 { + binary: "cargo".to_string(), + mode: Some(cli_mode), + }), + api: None, + capabilities, + }, + policy_overlay: None, + }, + } + } + + fn required_capabilities() -> AgentCapabilitiesV1 { + AgentCapabilitiesV1 { + session_start: true, + session_resume: true, + session_fork: true, + session_stop: true, + status_snapshot: true, + event_stream: true, + llm: true, + mcp_client: false, + } + } + + fn assert_selected_descriptor( + result: std::result::Result, MemberSelectionError>, + ) -> RuntimeSelectionDescriptor { + result + .expect("selection should succeed") + .expect("descriptor") + } + + fn assert_exact_selected_descriptor( + result: std::result::Result, ExactBackendSelectionError>, + ) -> RuntimeSelectionDescriptor { + result + .expect("selection should succeed") + .expect("descriptor") + } + + #[test] + fn validate_member_selection_returns_none_when_no_world_members_exist() { + let config = SubstrateConfig::default(); + let mut inventory = BTreeMap::new(); + inventory.insert( + "claude_code".to_string(), + make_entry( + "claude_code", + AgentExecutionScope::Host, + Some(PURE_AGENT_PROTOCOL), + AgentCliMode::Persistent, + required_capabilities(), + ), + ); + + assert!(validate_member_selection(&config, &inventory) + .expect("selection should not fail") + .is_none()); + } + + #[test] + fn validate_member_selection_returns_descriptor_for_unique_world_member() { + let config = SubstrateConfig::default(); + let mut inventory = BTreeMap::new(); + inventory.insert( + "codex".to_string(), + make_entry( + "codex", + AgentExecutionScope::World, + Some(PURE_AGENT_PROTOCOL), + AgentCliMode::Persistent, + required_capabilities(), + ), + ); + + let descriptor = assert_selected_descriptor(validate_member_selection(&config, &inventory)); + assert_eq!(descriptor.agent_id, "codex"); + assert_eq!(descriptor.backend_id, "cli:codex"); + assert_eq!(descriptor.backend_kind, AgentRuntimeBackendKind::Codex); + assert_eq!(descriptor.execution_scope, AgentExecutionScope::World); + } + + #[test] + fn validate_member_selection_fails_closed_on_ambiguity() { + let config = SubstrateConfig::default(); + let mut inventory = BTreeMap::new(); + inventory.insert( + "codex".to_string(), + make_entry( + "codex", + AgentExecutionScope::World, + Some(PURE_AGENT_PROTOCOL), + AgentCliMode::Persistent, + required_capabilities(), + ), + ); + inventory.insert( + "claude_code".to_string(), + make_entry( + "claude_code", + AgentExecutionScope::World, + Some(PURE_AGENT_PROTOCOL), + AgentCliMode::Persistent, + required_capabilities(), + ), + ); + + let error = validate_member_selection(&config, &inventory).expect_err("must fail closed"); + assert_eq!(error.exit_code, 2); + assert!( + error.reason.contains("ambiguous world member selection"), + "unexpected reason: {}", + error.reason + ); + assert!( + error.reason.contains("claude_code, codex"), + "unexpected reason: {}", + error.reason + ); + } + + #[test] + fn validate_member_selection_fails_for_wrong_protocol() { + let config = SubstrateConfig::default(); + let mut inventory = BTreeMap::new(); + inventory.insert( + "codex".to_string(), + make_entry( + "codex", + AgentExecutionScope::World, + Some("other.protocol"), + AgentCliMode::Persistent, + required_capabilities(), + ), + ); + + let error = validate_member_selection(&config, &inventory).expect_err("must fail closed"); + assert_eq!(error.exit_code, 2); + assert!( + error + .reason + .contains("does not advertise protocol 'uaa.agent.session'"), + "unexpected reason: {}", + error.reason + ); + } + + #[test] + fn validate_exact_backend_selection_returns_none_when_backend_is_missing() { + let config = SubstrateConfig::default(); + let mut inventory = BTreeMap::new(); + inventory.insert( + "codex".to_string(), + make_entry( + "codex", + AgentExecutionScope::World, + Some(PURE_AGENT_PROTOCOL), + AgentCliMode::Persistent, + required_capabilities(), + ), + ); + + let selected = validate_exact_backend_selection( + &config, + &inventory, + AgentExecutionScope::World, + "cli:claude-code", + ) + .expect("selection should not fail"); + assert!(selected.is_none()); + } + + #[test] + fn validate_exact_backend_selection_bypasses_world_ambiguity_when_backend_matches_exactly() { + let config = SubstrateConfig::default(); + let mut inventory = BTreeMap::new(); + inventory.insert( + "codex".to_string(), + make_entry( + "codex", + AgentExecutionScope::World, + Some(PURE_AGENT_PROTOCOL), + AgentCliMode::Persistent, + required_capabilities(), + ), + ); + inventory.insert( + "claude_code".to_string(), + make_entry( + "claude_code", + AgentExecutionScope::World, + Some(PURE_AGENT_PROTOCOL), + AgentCliMode::Persistent, + required_capabilities(), + ), + ); + + let descriptor = assert_exact_selected_descriptor(validate_exact_backend_selection( + &config, + &inventory, + AgentExecutionScope::World, + "cli:codex", + )); + assert_eq!(descriptor.agent_id, "codex"); + assert_eq!(descriptor.backend_id, "cli:codex"); + } + + #[test] + fn validate_exact_backend_selection_reports_scope_specific_protocol_error() { + let config = SubstrateConfig::default(); + let mut inventory = BTreeMap::new(); + inventory.insert( + "claude_code".to_string(), + make_entry( + "claude_code", + AgentExecutionScope::Host, + Some("other.protocol"), + AgentCliMode::Persistent, + required_capabilities(), + ), + ); + + let error = validate_exact_backend_selection( + &config, + &inventory, + AgentExecutionScope::Host, + "cli:claude_code", + ) + .expect_err("must fail closed"); + assert_eq!(exact_backend_selection_error_exit_code(&error), 2); + assert!( + error + .reason + .contains("selected host-scoped runtime 'claude_code' for backend 'cli:claude_code' does not advertise protocol 'uaa.agent.session'"), + "unexpected reason: {}", + error.reason + ); + } + + #[test] + fn validate_runtime_realizability_uses_generic_runtime_wording() { + let config = SubstrateConfig::default(); + let entry = make_entry( + "claude_code", + AgentExecutionScope::Host, + Some(PURE_AGENT_PROTOCOL), + AgentCliMode::PerRequest, + required_capabilities(), + ); + + let error = validate_runtime_realizability(&entry, &config).expect_err("must fail"); + assert!( + error.reason.starts_with("selected runtime 'claude_code'"), + "unexpected reason: {}", + error.reason + ); + } +} diff --git a/crates/shell/src/execution/agents_cmd.rs b/crates/shell/src/execution/agents_cmd.rs index 9f652533a..c94e11c85 100644 --- a/crates/shell/src/execution/agents_cmd.rs +++ b/crates/shell/src/execution/agents_cmd.rs @@ -2,14 +2,45 @@ use crate::execution::agent_inventory::{ discover_agent_files, load_effective_agent_inventory, validate_agent_file, AgentInventoryEntryV1, }; +#[cfg(unix)] +use crate::execution::agent_runtime::control::request_private_stop; +use crate::execution::agent_runtime::control::{ + hidden_owner_helper_readiness_timed_out, load_hidden_owner_helper_launch_plan, + load_public_prompt_source, persist_hidden_owner_helper_launch_plan, + persist_runtime_stop_closeout, public_prompt_rendered_exit_code, + reconcile_hidden_owner_helper_start_timeout, remove_hidden_owner_helper_launch_plan, + run_public_prompt_command, wait_for_hidden_owner_helper_readiness, HiddenOwnerHelperLaunchPlan, + HiddenOwnerHelperParticipantPlan, HiddenOwnerHelperSessionPlan, + HiddenOwnerHelperStartTimeoutReconciliation, OwnerHelperMode, PublicPromptAction, + PublicPromptCommandRequest, PublicPromptInput, PublicSessionPosture, + ResolvedRuntimeBackendKind, ResolvedRuntimeDescriptor, HIDDEN_OWNER_HELPER_SUBCOMMAND, +}; +#[cfg(unix)] +use crate::execution::agent_runtime::control::{ + private_stop_transport_path, register_hidden_owner_helper_startup_prompt_listener, + run_hidden_owner_helper_startup_prompt_stream, HiddenOwnerHelperStartupPromptPlan, +}; +use crate::execution::agent_runtime::orchestration_session::{ + OrchestrationSessionPosture, OrchestrationSessionRecord, +}; +#[cfg(unix)] +use crate::execution::agent_runtime::state_store::HiddenOwnerHelperLaunchReadiness; +use crate::execution::agent_runtime::validator::{ + exact_backend_selection_error_exit_code, member_selection_error_exit_code, + validate_exact_backend_selection, validate_member_selection, +}; +#[cfg(unix)] +use crate::execution::agent_runtime::StartupPromptReplayState; use crate::execution::agent_runtime::{ runtime_realizability_error_exit_code, validate_orchestrator_selection, - validate_runtime_realizability, AgentRuntimeSessionManifest, AgentRuntimeStateStore, - MEMBER_ROLE, NESTED_ROUTER, ORCHESTRATOR_ROLE, PURE_AGENT_PROTOCOL, PURE_AGENT_ROUTER, + validate_runtime_realizability, AgentRuntimeParticipantRecord, AgentRuntimeSessionRecord, + AgentRuntimeStateStore, PublicControlAction, PublicTurnTargetKind, MEMBER_ROLE, NESTED_ROUTER, + ORCHESTRATOR_ROLE, PURE_AGENT_PROTOCOL, PURE_AGENT_ROUTER, }; use crate::execution::cli::{ - AgentAction, AgentCmd, AgentDoctorArgs, AgentScopeArg, AgentToolboxAction, AgentToolboxCmd, - AgentToolboxViewArgs, AgentViewArgs, AgentsAction, AgentsCmd, Cli, + AgentAction, AgentCmd, AgentDoctorArgs, AgentOwnerHelperArgs, AgentScopeArg, + AgentSessionControlArgs, AgentStartArgs, AgentToolboxAction, AgentToolboxCmd, + AgentToolboxViewArgs, AgentTurnArgs, AgentViewArgs, AgentsAction, AgentsCmd, Cli, }; use crate::execution::config_model::{ self, AgentExecutionScope, AgentToolboxBindTransport, CliConfigOverrides, SubstrateConfig, @@ -22,11 +53,26 @@ use std::env; use std::fs::File; use std::io::{self, BufRead, BufReader}; use std::path::{Path, PathBuf}; -use std::process::Command; +use std::process::{Command, Stdio}; +use std::thread; +use std::time::Duration; use substrate_broker::Policy; use substrate_common::paths as substrate_paths; use substrate_common::{AgentEvent, PlacementExecution}; +#[cfg(unix)] +use tokio::runtime::Builder as TokioRuntimeBuilder; +use uuid::Uuid; const TOOLBOX_VERSION: u32 = 1; +#[cfg(unix)] +const START_DETACH_NORMALIZATION_TIMEOUT: Duration = Duration::from_secs(10); +#[cfg(unix)] +const START_DETACH_NORMALIZATION_POLL_INTERVAL: Duration = Duration::from_millis(50); +#[cfg(unix)] +const START_ATTACHED_GRACE_TIMEOUT: Duration = Duration::from_millis(250); +#[cfg(unix)] +const TURN_DETACH_NORMALIZATION_TIMEOUT: Duration = Duration::from_secs(10); +#[cfg(unix)] +const TURN_DETACH_NORMALIZATION_POLL_INTERVAL: Duration = Duration::from_millis(50); pub(crate) fn handle_agent_command(cmd: &AgentCmd, cli: &Cli) -> i32 { match &cmd.action { @@ -63,10 +109,73 @@ pub(crate) fn handle_agent_command(cmd: &AgentCmd, cli: &Cli) -> i32 { 1 } }, + AgentAction::Start(args) => render_public_prompt_command_result(run_start(args, cli)), + AgentAction::Turn(args) => render_public_prompt_command_result(run_turn(args, cli)), + AgentAction::Reattach(args) => match run_reattach(args, cli) { + Ok(()) => 0, + Err(err) if config_model::is_user_error(&err) => { + eprintln!("{err}"); + 2 + } + Err(err) => { + eprintln!("{err:#}"); + 1 + } + }, + AgentAction::Fork(args) => match run_fork(args, cli) { + Ok(()) => 0, + Err(err) if config_model::is_user_error(&err) => { + eprintln!("{err}"); + 2 + } + Err(err) => { + eprintln!("{err:#}"); + 1 + } + }, + AgentAction::Stop(args) => match run_stop(args, cli) { + Ok(()) => 0, + Err(err) if config_model::is_user_error(&err) => { + eprintln!("{err}"); + 2 + } + Err(err) => { + eprintln!("{err:#}"); + 1 + } + }, + AgentAction::OwnerHelper(args) => match run_owner_helper(args) { + Ok(code) => code, + Err(err) if config_model::is_user_error(&err) => { + eprintln!("{err}"); + 2 + } + Err(err) => { + eprintln!("{err:#}"); + 1 + } + }, AgentAction::Toolbox(cmd) => handle_agent_toolbox_command(cmd, cli), } } +fn render_public_prompt_command_result(result: Result<()>) -> i32 { + match result { + Ok(()) => 0, + Err(err) => { + if let Some(code) = public_prompt_rendered_exit_code(&err) { + return code; + } + if config_model::is_user_error(&err) { + eprintln!("{err}"); + return 2; + } + eprintln!("{err:#}"); + 1 + } + } +} + fn handle_agent_toolbox_command(cmd: &AgentToolboxCmd, cli: &Cli) -> i32 { let result = match &cmd.action { AgentToolboxAction::Status(args) => run_toolbox_status(args, cli), @@ -138,6 +247,939 @@ fn run_doctor(args: &AgentDoctorArgs, cli: &Cli) -> Result { Ok(exit_code) } +fn run_owner_helper(args: &AgentOwnerHelperArgs) -> Result { + let plan = load_hidden_owner_helper_launch_plan(&args.plan_file)?; + remove_hidden_owner_helper_launch_plan(&args.plan_file)?; + crate::repl::async_repl::run_hidden_owner_helper(plan) +} + +#[allow(dead_code)] +#[derive(Clone, Debug)] +pub(crate) struct HiddenOwnerHelperLaunchReceipt { + pub helper_pid: u32, + pub orchestration_session_id: String, + pub participant_id: String, + pub backend_id: String, +} + +#[allow(dead_code)] +pub(crate) fn launch_hidden_owner_helper( + plan: &HiddenOwnerHelperLaunchPlan, + cli: &Cli, +) -> Result { + let store = AgentRuntimeStateStore::new()?; + let plan_path = persist_hidden_owner_helper_launch_plan(&store, plan)?; + let exe = env::current_exe() + .context("failed to resolve current substrate executable for hidden owner-helper launch")?; + let mut command = Command::new(exe); + if cli.world { + command.arg("--world"); + } else if cli.no_world { + command.arg("--no-world"); + } + command + .args(["agent", HIDDEN_OWNER_HELPER_SUBCOMMAND, "--plan-file"]) + .arg(&plan_path) + .current_dir(&plan.session.workspace_root) + .stdin(Stdio::null()) + .stdout(Stdio::null()) + .stderr(Stdio::null()); + + let mut child = command.spawn().with_context(|| { + format!( + "failed to spawn hidden owner-helper for orchestration session {}", + plan.session.orchestration_session_id + ) + })?; + if let Err(err) = wait_for_hidden_owner_helper_readiness(&store, plan) { + let reconciled = if plan.mode == OwnerHelperMode::Start + && hidden_owner_helper_readiness_timed_out(&err) + { + Some(reconcile_hidden_owner_helper_start_timeout(&store, plan)) + } else { + None + }; + match reconciled { + Some(Ok(HiddenOwnerHelperStartTimeoutReconciliation::Success)) => {} + Some(Ok(HiddenOwnerHelperStartTimeoutReconciliation::FailureMarkedTerminal)) => { + let _ = child.kill(); + let _ = child.wait(); + let _ = remove_hidden_owner_helper_launch_plan(&plan_path); + return Err(anyhow::anyhow!( + "{}; persisted terminal startup failure for orchestration session {}", + err, + plan.orchestration_session_id(), + )); + } + Some(Ok(HiddenOwnerHelperStartTimeoutReconciliation::FailureUnchanged)) | None => { + let _ = child.kill(); + let _ = child.wait(); + let _ = remove_hidden_owner_helper_launch_plan(&plan_path); + return Err(err); + } + Some(Err(reconcile_err)) => { + let _ = child.kill(); + let _ = child.wait(); + let _ = remove_hidden_owner_helper_launch_plan(&plan_path); + return Err(anyhow::anyhow!( + "{}; additionally failed to reconcile persisted startup state: {reconcile_err:#}", + err + )); + } + } + } + #[cfg(unix)] + if let Err(err) = stabilize_hidden_owner_helper_start_return(&store, plan, &mut child) { + let _ = child.kill(); + let _ = child.wait(); + let _ = remove_hidden_owner_helper_launch_plan(&plan_path); + return Err(err); + } + + Ok(HiddenOwnerHelperLaunchReceipt { + helper_pid: child.id(), + orchestration_session_id: plan.session.orchestration_session_id.clone(), + participant_id: plan.participant.participant_id.clone(), + backend_id: plan.descriptor.backend_id.clone(), + }) +} + +const AGENT_CONTROL_STOP_WAIT_TIMEOUT: Duration = Duration::from_secs(10); +const AGENT_CONTROL_STOP_WAIT_POLL_INTERVAL: Duration = Duration::from_millis(100); + +#[derive(Serialize)] +struct AgentControlResultJson<'a> { + action: &'a str, + orchestration_session_id: &'a str, + backend_id: &'a str, + scope: &'a str, + state: &'a str, + warnings: Vec, + #[serde(skip_serializing_if = "Option::is_none")] + participant_id: Option<&'a str>, + #[serde(skip_serializing_if = "Option::is_none")] + source_orchestration_session_id: Option<&'a str>, +} + +fn run_start(args: &AgentStartArgs, cli: &Cli) -> Result<()> { + let prompt = load_public_prompt_source(&PublicPromptInput { + prompt: args.prompt_source.prompt.clone(), + prompt_file: args.prompt_source.prompt_file.clone(), + }) + .map_err(normalize_public_prompt_error)?; + let context = resolve_command_context(cli)?; + let store = AgentRuntimeStateStore::new()?; + let plan = build_start_launch_plan(args, &context)?; + + #[cfg(not(unix))] + { + let _ = store; + let _ = plan; + let _ = prompt; + anyhow::bail!(config_model::user_error( + "unsupported_platform_or_posture: public start prompt streaming requires a Unix launch-time backchannel" + )); + } + + #[cfg(unix)] + { + let mut plan = plan; + let startup_listener = register_hidden_owner_helper_startup_prompt_listener( + &store, + plan.orchestration_session_id(), + plan.participant_id(), + ) + .map_err(runtime_start_error)?; + plan.startup_prompt = Some(HiddenOwnerHelperStartupPromptPlan { + prompt_text: prompt.prompt_text.clone(), + stream_path: startup_listener.path().to_path_buf(), + }); + + let receipt = launch_hidden_owner_helper(&plan, cli).map_err(runtime_start_error)?; + match run_hidden_owner_helper_startup_prompt_stream(startup_listener, args.json) { + Ok(()) => wait_for_start_prompt_completion_normalization( + &store, + &receipt.orchestration_session_id, + &receipt.participant_id, + ) + .map_err(runtime_start_error), + Err(err) + if recoverable_detached_start_retry( + &receipt.orchestration_session_id, + &receipt.participant_id, + receipt.helper_pid, + &err, + )? => + { + let retry_listener = register_hidden_owner_helper_startup_prompt_listener( + &store, + plan.orchestration_session_id(), + plan.participant_id(), + ) + .map_err(runtime_start_error)?; + plan.startup_prompt = Some(HiddenOwnerHelperStartupPromptPlan { + prompt_text: prompt.prompt_text, + stream_path: retry_listener.path().to_path_buf(), + }); + let retry_receipt = + launch_hidden_owner_helper(&plan, cli).map_err(runtime_start_error)?; + run_hidden_owner_helper_startup_prompt_stream(retry_listener, args.json) + .map_err(normalize_public_prompt_error)?; + wait_for_start_prompt_completion_normalization( + &store, + &retry_receipt.orchestration_session_id, + &retry_receipt.participant_id, + ) + .map_err(runtime_start_error) + } + Err(err) => Err(normalize_public_prompt_error(err)), + } + } +} + +#[cfg(unix)] +fn stabilize_hidden_owner_helper_start_return( + store: &AgentRuntimeStateStore, + plan: &HiddenOwnerHelperLaunchPlan, + child: &mut std::process::Child, +) -> Result<()> { + if plan.mode != OwnerHelperMode::Start { + return Ok(()); + } + + let grace_started_at = std::time::Instant::now(); + loop { + if matches!( + store.classify_hidden_owner_helper_launch_readiness( + plan.orchestration_session_id(), + plan.participant_id(), + plan.requires_internal_session_id(), + )?, + HiddenOwnerHelperLaunchReadiness::ReadyDetached(_) + ) { + return Ok(()); + } + if child + .try_wait() + .context("failed to poll hidden owner-helper exit status")? + .is_some() + { + break; + } + if grace_started_at.elapsed() >= START_ATTACHED_GRACE_TIMEOUT { + return Ok(()); + } + thread::sleep(START_DETACH_NORMALIZATION_POLL_INTERVAL); + } + + let normalization_started_at = std::time::Instant::now(); + loop { + if matches!( + store.classify_hidden_owner_helper_launch_readiness( + plan.orchestration_session_id(), + plan.participant_id(), + plan.requires_internal_session_id(), + )?, + HiddenOwnerHelperLaunchReadiness::ReadyDetached(_) + ) { + return Ok(()); + } + + if normalization_started_at.elapsed() >= START_DETACH_NORMALIZATION_TIMEOUT { + let snapshot_summary = store + .load_orchestration_session(plan.orchestration_session_id())? + .map(|session| { + format!( + "state={:?}, posture={:?}, attached_participant_id={:?}, shell_owner_pid={}", + session.state, + session.posture, + session.attached_participant_id, + session.shell_owner_pid, + ) + }) + .unwrap_or_else(|| "session_missing".to_string()); + anyhow::bail!( + "timed out waiting for detached start normalization for orchestration session {} after hidden owner-helper {} exited ({snapshot_summary})", + plan.orchestration_session_id(), + child.id(), + ); + } + + thread::sleep(START_DETACH_NORMALIZATION_POLL_INTERVAL); + } +} + +#[cfg(unix)] +fn wait_for_start_prompt_completion_normalization( + store: &AgentRuntimeStateStore, + orchestration_session_id: &str, + participant_id: &str, +) -> Result<()> { + let normalization_started_at = std::time::Instant::now(); + let mut attached_grace_started_at: Option = None; + loop { + match store.classify_hidden_owner_helper_launch_readiness( + orchestration_session_id, + participant_id, + true, + )? { + HiddenOwnerHelperLaunchReadiness::ReadyDetached(_) => return Ok(()), + HiddenOwnerHelperLaunchReadiness::ReadyAttached => { + let grace_started_at = + attached_grace_started_at.get_or_insert_with(std::time::Instant::now); + if grace_started_at.elapsed() >= START_ATTACHED_GRACE_TIMEOUT { + return Ok(()); + } + } + HiddenOwnerHelperLaunchReadiness::Pending => { + attached_grace_started_at = None; + } + } + + if normalization_started_at.elapsed() >= START_DETACH_NORMALIZATION_TIMEOUT { + let snapshot_summary = store + .load_orchestration_session(orchestration_session_id)? + .map(|session| { + format!( + "state={:?}, posture={:?}, attached_participant_id={:?}, shell_owner_pid={}", + session.state, + session.posture, + session.attached_participant_id, + session.shell_owner_pid, + ) + }) + .unwrap_or_else(|| "session_missing".to_string()); + anyhow::bail!( + "timed out waiting for detached start normalization after startup prompt completion for orchestration session {} ({snapshot_summary})", + orchestration_session_id, + ); + } + + thread::sleep(START_DETACH_NORMALIZATION_POLL_INTERVAL); + } +} + +#[cfg(unix)] +fn recoverable_detached_start_retry( + orchestration_session_id: &str, + participant_id: &str, + helper_pid: u32, + err: &anyhow::Error, +) -> Result { + if public_prompt_rendered_exit_code(err).is_some() { + return Ok(false); + } + + let message = err.to_string(); + let prompt_transport_failed = message.contains("startup prompt stream") + || message.starts_with("stream_bridge_failed:") + || message.contains("hidden owner-helper startup prompt stream ended"); + if !prompt_transport_failed { + return Ok(false); + } + + let store = AgentRuntimeStateStore::new()?; + let replay_state = + store.startup_prompt_replay_state(orchestration_session_id, participant_id)?; + if !replay_state.replay_safe() { + return Ok(false); + } + #[cfg(unix)] + if pid_is_alive(helper_pid) { + return Ok(false); + } + Ok(matches!( + replay_state, + StartupPromptReplayState::NotTracked | StartupPromptReplayState::PendingAcceptance + )) +} + +#[cfg(unix)] +fn pid_is_alive(pid: u32) -> bool { + let rc = unsafe { libc::kill(pid as libc::pid_t, 0) }; + rc == 0 + || matches!( + std::io::Error::last_os_error().raw_os_error(), + Some(libc::EPERM) + ) +} + +fn run_turn(args: &AgentTurnArgs, cli: &Cli) -> Result<()> { + let prompt = load_public_prompt_source(&PublicPromptInput { + prompt: args.prompt_source.prompt.clone(), + prompt_file: args.prompt_source.prompt_file.clone(), + }) + .map_err(normalize_public_prompt_error)?; + let store = AgentRuntimeStateStore::new()?; + let target = store + .resolve_public_turn_target(&args.session, &args.backend) + .map_err(|err| config_model::user_error(err.to_string()))?; + let orchestration_session_id = target.session.orchestration_session_id.clone(); + let backend_id = target.participant.handle.backend_id.clone(); + + let resumed_receipt = match target.session_posture { + PublicSessionPosture::Active => None, + PublicSessionPosture::DetachedReattachable => match target.target_kind { + PublicTurnTargetKind::Host => { + let plan = build_successor_launch_plan( + cli, + &args.session, + OwnerHelperMode::ResumeOneTurn, + )?; + if plan.descriptor.backend_id != backend_id { + anyhow::bail!(config_model::user_error(format!( + "backend_not_in_session: orchestration session {} has no exact backend slot for {}", + args.session, + args.backend + ))); + } + Some(launch_hidden_owner_helper(&plan, cli).map_err(runtime_start_error)?) + } + PublicTurnTargetKind::World => { + anyhow::bail!(config_model::user_error(format!( + "unsupported_platform_or_posture: orchestration session {} backend {} is detached and cannot be recovered through the retained world-member seam without an active host owner; run `substrate agent reattach --session {}` first", + args.session, + args.backend, + args.session + ))); + } + }, + PublicSessionPosture::Terminal => { + anyhow::bail!(config_model::user_error(format!( + "owner_unreachable: orchestration session {} backend {} is terminal and cannot accept follow-up turns", + args.session, + args.backend + ))); + } + }; + #[cfg(not(unix))] + let _ = &resumed_receipt; + + run_public_prompt_command( + PublicPromptCommandRequest { + action: PublicPromptAction::Turn, + orchestration_session_id: Some(orchestration_session_id), + backend_id, + prompt, + json: args.json, + }, + cli.world, + cli.no_world, + ) + .map_err(normalize_public_prompt_error)?; + + #[cfg(unix)] + if let Some(receipt) = resumed_receipt.as_ref() { + wait_for_resumed_public_turn_detach( + &store, + &receipt.orchestration_session_id, + &receipt.participant_id, + ) + .map_err(runtime_start_error)?; + } + + Ok(()) +} + +fn run_reattach(args: &AgentSessionControlArgs, cli: &Cli) -> Result<()> { + let plan = build_successor_launch_plan(cli, &args.session, OwnerHelperMode::Resume)?; + let receipt = launch_hidden_owner_helper(&plan, cli).map_err(runtime_start_error)?; + if receipt.orchestration_session_id != args.session { + anyhow::bail!(runtime_start_error(anyhow::anyhow!( + "hidden owner-helper resumed orchestration session {} instead of requested session {}", + receipt.orchestration_session_id, + args.session + ))); + } + + let store = AgentRuntimeStateStore::new()?; + let target = store + .resolve_public_control_target(&receipt.orchestration_session_id, PublicControlAction::Stop) + .map_err(runtime_start_error)?; + if target.session_posture != PublicSessionPosture::Active + || target.active_participant.handle.participant_id != receipt.participant_id + { + anyhow::bail!(runtime_start_error(anyhow::anyhow!( + "orchestration session {} did not reach durable active_attached ownership after reattach", + receipt.orchestration_session_id + ))); + } + + render_agent_control_result( + args.json, + &AgentControlResultJson { + action: "reattach", + orchestration_session_id: &receipt.orchestration_session_id, + backend_id: &receipt.backend_id, + scope: "host", + state: "active", + warnings: Vec::new(), + participant_id: Some(&receipt.participant_id), + source_orchestration_session_id: None, + }, + ) +} + +fn normalize_public_prompt_error(err: anyhow::Error) -> anyhow::Error { + if config_model::is_user_error(&err) || public_prompt_rendered_exit_code(&err).is_some() { + return err; + } + + let message = err.to_string(); + const USER_PREFIXES: &[&str] = &[ + "missing_backend:", + "unknown_backend:", + "ambiguous_backend:", + "missing_prompt_source:", + "malformed_prompt_source:", + "empty_prompt:", + "unknown_session:", + "noncanonical_session_selector:", + "backend_not_in_session:", + "ambiguous_backend_slot:", + "missing_active_parent:", + "stale_linkage:", + "missing_internal_session_id:", + "session_already_owned:", + "owner_unreachable:", + "unsupported_platform_or_posture:", + "policy_disallow:", + "stream_bridge_failed:", + "runtime_start_failed:", + ]; + + if USER_PREFIXES + .iter() + .any(|prefix| message.starts_with(prefix)) + { + return config_model::user_error(message); + } + + err +} + +fn runtime_start_error(err: anyhow::Error) -> anyhow::Error { + if config_model::is_user_error(&err) { + return err; + } + config_model::user_error(format!("runtime_start_failed: {err}")) +} + +fn run_fork(args: &AgentSessionControlArgs, cli: &Cli) -> Result<()> { + let plan = build_successor_launch_plan(cli, &args.session, OwnerHelperMode::Fork)?; + let receipt = launch_hidden_owner_helper(&plan, cli)?; + render_agent_control_result( + args.json, + &AgentControlResultJson { + action: "fork", + orchestration_session_id: &receipt.orchestration_session_id, + backend_id: &receipt.backend_id, + scope: "host", + state: "active", + warnings: Vec::new(), + participant_id: Some(&receipt.participant_id), + source_orchestration_session_id: plan.source_orchestration_session_id.as_deref(), + }, + ) +} + +fn run_stop(args: &AgentSessionControlArgs, _cli: &Cli) -> Result<()> { + let store = AgentRuntimeStateStore::new()?; + let target = store + .resolve_public_control_target(&args.session, PublicControlAction::Stop) + .map_err(|err| config_model::user_error(err.to_string()))?; + let participant_id = target.active_participant.handle.participant_id.clone(); + let backend_id = target.active_participant.handle.backend_id.clone(); + let orchestration_session_id = target.session.orchestration_session_id.clone(); + + if target.session_posture == PublicSessionPosture::DetachedReattachable { + let mut session = target.session.clone(); + let mut manifest = target.active_participant.clone(); + persist_runtime_stop_closeout(&store, &mut session, &mut manifest) + .context("failed to persist detached durable stop closeout")?; + + let final_state = + wait_for_terminal_session_state(&store, orchestration_session_id.as_str())? + .ok_or_else(|| { + config_model::user_error(format!( + "owner_unreachable: timed out waiting for orchestration session {} to reach a terminal state", + orchestration_session_id + )) + })?; + return render_agent_control_result( + args.json, + &AgentControlResultJson { + action: "stop", + orchestration_session_id: orchestration_session_id.as_str(), + backend_id: &backend_id, + scope: "host", + state: final_state.as_str(), + warnings: Vec::new(), + participant_id: Some(&participant_id), + source_orchestration_session_id: None, + }, + ); + } + + #[cfg(not(unix))] + { + let _ = store; + let _ = target; + anyhow::bail!(config_model::user_error( + "unsupported_platform_or_posture: public stop requires a Unix private owner transport" + )); + } + + #[cfg(unix)] + { + let transport_path = private_stop_transport_path( + &store, + orchestration_session_id.as_str(), + participant_id.as_str(), + ); + let rt = TokioRuntimeBuilder::new_current_thread() + .enable_all() + .build() + .context("failed to initialize stop transport runtime")?; + let outcome = rt + .block_on(async { request_private_stop(&transport_path).await }) + .map_err(|err| config_model::user_error(format!("owner_unreachable: {err}")))?; + match outcome { + crate::execution::agent_runtime::control::PrivateStopOutcome::Accepted + | crate::execution::agent_runtime::control::PrivateStopOutcome::AlreadyTerminal => {} + crate::execution::agent_runtime::control::PrivateStopOutcome::OwnerUnreachable => { + anyhow::bail!(config_model::user_error(format!( + "owner_unreachable: orchestration session {} no longer has a reachable live owner", + args.session + ))); + } + crate::execution::agent_runtime::control::PrivateStopOutcome::ProtocolError => { + anyhow::bail!(config_model::user_error(format!( + "owner_unreachable: private owner transport for orchestration session {} returned a protocol error", + args.session + ))); + } + } + + let final_state = wait_for_terminal_session_state(&store, &args.session)?.ok_or_else(|| { + config_model::user_error(format!( + "owner_unreachable: timed out waiting for orchestration session {} to reach a terminal state", + args.session + )) + })?; + render_agent_control_result( + args.json, + &AgentControlResultJson { + action: "stop", + orchestration_session_id: &args.session, + backend_id: &backend_id, + scope: "host", + state: final_state.as_str(), + warnings: Vec::new(), + participant_id: Some(&participant_id), + source_orchestration_session_id: None, + }, + ) + } +} + +fn build_start_launch_plan( + args: &AgentStartArgs, + context: &AgentCommandContext, +) -> Result { + let backend_id = args.backend.trim(); + if backend_id.is_empty() { + anyhow::bail!(config_model::user_error( + "unknown_backend: public start requires --backend " + )); + } + + let descriptor = match validate_exact_backend_selection( + &context.effective_config, + &context.inventory, + AgentExecutionScope::Host, + backend_id, + ) { + Ok(Some(descriptor)) => descriptor, + Ok(None) => { + if validate_exact_backend_selection( + &context.effective_config, + &context.inventory, + AgentExecutionScope::World, + backend_id, + ) + .ok() + .flatten() + .is_some() + { + anyhow::bail!(config_model::user_error(format!( + "unsupported_platform_or_posture: backend '{}' resolves only to a world-scoped runtime; public root start is host-only in v1", + backend_id + ))); + } + anyhow::bail!(config_model::user_error(format!( + "unknown_backend: no exact host-scoped backend match found for '{}'", + backend_id + ))); + } + Err(err) => { + let prefix = if err.reason.contains("ambiguous exact backend selection") { + "ambiguous_backend" + } else { + "unknown_backend" + }; + let _ = exact_backend_selection_error_exit_code(&err); + anyhow::bail!(config_model::user_error(format!( + "{prefix}: {}", + err.reason + ))); + } + }; + if !backend_allowed(&context.base_policy, &descriptor.backend_id) { + anyhow::bail!(config_model::user_error(format!( + "policy_disallow: selected orchestrator backend '{}' is not allowlisted by effective policy agents.allowed_backends", + descriptor.backend_id + ))); + } + + let orchestration_session_id = Uuid::now_v7().to_string(); + Ok(HiddenOwnerHelperLaunchPlan { + mode: OwnerHelperMode::Start, + descriptor: (&descriptor).into(), + session: HiddenOwnerHelperSessionPlan { + orchestration_session_id, + shell_trace_session_id: Uuid::now_v7().to_string(), + workspace_root: current_dir().display().to_string(), + world_id: None, + world_generation: None, + }, + participant: HiddenOwnerHelperParticipantPlan { + participant_id: format!("ash_{}", Uuid::now_v7()), + lease_token: Uuid::now_v7().to_string(), + run_id: Uuid::now_v7().to_string(), + resumed_from_participant_id: None, + internal_uaa_session_id: None, + }, + startup_prompt: None, + source_orchestration_session_id: None, + }) +} + +fn build_successor_launch_plan( + _cli: &Cli, + orchestration_session_id: &str, + mode: OwnerHelperMode, +) -> Result { + let store = AgentRuntimeStateStore::new()?; + let action = match mode { + OwnerHelperMode::Resume | OwnerHelperMode::ResumeOneTurn => PublicControlAction::Resume, + OwnerHelperMode::Fork => PublicControlAction::Fork, + OwnerHelperMode::Start => anyhow::bail!("invalid successor launch mode"), + }; + let target = store + .resolve_public_control_target(orchestration_session_id, action) + .map_err(|err| config_model::user_error(err.to_string()))?; + let descriptor = descriptor_from_participant(&target.active_participant)?; + let target_session_id = if mode == OwnerHelperMode::Fork { + Uuid::now_v7().to_string() + } else { + target.session.orchestration_session_id.clone() + }; + let trace_session_id = if mode == OwnerHelperMode::Fork { + Uuid::now_v7().to_string() + } else { + target.session.shell_trace_session_id.clone() + }; + + Ok(HiddenOwnerHelperLaunchPlan { + mode, + descriptor, + session: HiddenOwnerHelperSessionPlan { + orchestration_session_id: target_session_id, + shell_trace_session_id: trace_session_id, + workspace_root: target.session.workspace_root.clone(), + world_id: target.session.world_id.clone(), + world_generation: target.session.world_generation, + }, + participant: HiddenOwnerHelperParticipantPlan { + participant_id: format!("ash_{}", Uuid::now_v7()), + lease_token: Uuid::now_v7().to_string(), + run_id: Uuid::now_v7().to_string(), + resumed_from_participant_id: Some( + target.active_participant.handle.participant_id.clone(), + ), + internal_uaa_session_id: target + .active_participant + .internal_uaa_session_id() + .map(ToOwned::to_owned), + }, + startup_prompt: None, + source_orchestration_session_id: (mode == OwnerHelperMode::Fork) + .then(|| target.session.orchestration_session_id.clone()), + }) +} + +#[cfg(unix)] +fn wait_for_resumed_public_turn_detach( + store: &AgentRuntimeStateStore, + orchestration_session_id: &str, + resumed_participant_id: &str, +) -> Result<()> { + let started_at = std::time::Instant::now(); + loop { + if store + .resumed_public_turn_detach_posture(orchestration_session_id, resumed_participant_id)? + .is_some() + { + return Ok(()); + } + + let Some(record) = store.load_session(orchestration_session_id)? else { + anyhow::bail!( + "owner_unreachable: orchestration session {} disappeared before the resumed public turn could detach cleanly", + orchestration_session_id + ); + }; + if record.session.state.is_terminal() + || record.session.posture == OrchestrationSessionPosture::Terminal + { + anyhow::bail!( + "owner_unreachable: orchestration session {} became terminal before resumed participant {} could repark", + orchestration_session_id, + resumed_participant_id + ); + } + if record + .session + .active_participant_id() + .is_some_and(|participant_id| participant_id != resumed_participant_id) + { + anyhow::bail!( + "owner_unreachable: orchestration session {} reassigned active ownership to {} before resumed participant {} could repark", + orchestration_session_id, + record.session.active_participant_id().unwrap_or_default(), + resumed_participant_id + ); + } + + if started_at.elapsed() >= TURN_DETACH_NORMALIZATION_TIMEOUT { + anyhow::bail!( + "owner_unreachable: timed out waiting for orchestration session {} resumed participant {} to converge back to detached durable continuity", + orchestration_session_id, + resumed_participant_id + ); + } + + thread::sleep(TURN_DETACH_NORMALIZATION_POLL_INTERVAL); + } +} + +fn descriptor_from_participant( + participant: &AgentRuntimeParticipantRecord, +) -> Result { + let backend_kind = match participant.internal.resolved_agent_kind.as_str() { + "codex" => ResolvedRuntimeBackendKind::Codex, + "claude_code" => ResolvedRuntimeBackendKind::ClaudeCode, + other => { + anyhow::bail!(config_model::user_error(format!( + "unsupported_platform_or_posture: unsupported hidden owner-helper backend kind '{}'", + other + ))); + } + }; + Ok(ResolvedRuntimeDescriptor { + agent_id: participant.handle.agent_id.clone(), + backend_id: participant.handle.backend_id.clone(), + backend_kind, + protocol: participant.handle.protocol.clone(), + execution_scope: participant.handle.execution.scope, + binary_path: participant.internal.resolved_binary_path.clone(), + }) +} + +fn wait_for_terminal_session_state( + store: &AgentRuntimeStateStore, + orchestration_session_id: &str, +) -> Result> { + let started_at = std::time::Instant::now(); + loop { + if let Some(state) = terminal_session_state(store, orchestration_session_id)? { + return Ok(Some(state)); + } + if started_at.elapsed() >= AGENT_CONTROL_STOP_WAIT_TIMEOUT { + return Ok(None); + } + thread::sleep(AGENT_CONTROL_STOP_WAIT_POLL_INTERVAL); + } +} + +fn terminal_session_state( + store: &AgentRuntimeStateStore, + orchestration_session_id: &str, +) -> Result> { + Ok(store + .list_sessions()? + .into_iter() + .find(|record| record.session.orchestration_session_id == orchestration_session_id) + .and_then(|record| { + OrchestrationSessionStateLabel::from_session_state(&record.session.state) + })) +} + +#[derive(Clone, Copy)] +enum OrchestrationSessionStateLabel { + Stopped, + Invalidated, + Failed, +} + +impl OrchestrationSessionStateLabel { + fn as_str(self) -> &'static str { + match self { + Self::Stopped => "stopped", + Self::Invalidated => "invalidated", + Self::Failed => "failed", + } + } + + fn from_session_state( + state: &crate::execution::agent_runtime::OrchestrationSessionState, + ) -> Option { + match state { + crate::execution::agent_runtime::OrchestrationSessionState::Stopped => { + Some(Self::Stopped) + } + crate::execution::agent_runtime::OrchestrationSessionState::Invalidated => { + Some(Self::Invalidated) + } + crate::execution::agent_runtime::OrchestrationSessionState::Failed => { + Some(Self::Failed) + } + _ => None, + } + } +} + +fn render_agent_control_result(json_mode: bool, result: &AgentControlResultJson<'_>) -> Result<()> { + if json_mode { + println!("{}", serde_json::to_string_pretty(result)?); + return Ok(()); + } + + let mut line = format!( + "{} {} [{}] -> {}", + result.action, result.orchestration_session_id, result.backend_id, result.state + ); + if let Some(participant_id) = result.participant_id { + line.push_str(&format!(" participant={participant_id}")); + } + if let Some(source) = result.source_orchestration_session_id { + line.push_str(&format!(" source={source}")); + } + println!("{line}"); + Ok(()) +} + struct AgentCommandContext { effective_config: SubstrateConfig, base_policy: Policy, @@ -324,13 +1366,19 @@ fn render_list_report(report: &ListReportJson<'_>, json_mode: bool) -> Result<() #[derive(Clone, Serialize)] struct StatusSessionJson { orchestration_session_id: String, + #[serde(skip_serializing_if = "Option::is_none")] + participant_id: Option, agent_id: String, + source_kind: &'static str, backend_id: String, client: String, router: String, protocol: String, execution: ExecutionScopeJson<'static>, role: Option, + posture: Option, + attached_participant_id: Option, + pending_inbox_count: Option, last_event_at: String, #[serde(skip_serializing_if = "Option::is_none")] world_id: Option, @@ -341,6 +1389,8 @@ struct StatusSessionJson { #[derive(Clone, Serialize)] struct NestedParentJson { orchestration_session_id: String, + #[serde(skip_serializing_if = "Option::is_none")] + participant_id: Option, agent_id: String, } @@ -362,6 +1412,7 @@ struct StatusReportJson<'a> { scope_filter: &'a str, role_filter: Option<&'a str>, orchestrator_agent_id: String, + warnings: Vec, sessions: Vec, nested_llm_records: Vec, } @@ -375,15 +1426,39 @@ struct SessionProjection { #[derive(Clone)] struct SessionProjectionSource { - orchestration_session_id: String, - agent_id: String, - run_id: String, + identity: StatusIdentityKey, + run_id: Option, ts: DateTime, is_world_scoped: bool, has_top_level_world_id: bool, has_top_level_world_generation: bool, } +#[derive(Clone)] +struct SelectedParentRun { + participant_id: Option, + run_id: String, +} + +#[derive(Clone, Debug, Eq, Ord, PartialEq, PartialOrd)] +struct StatusIdentityKey { + orchestration_session_id: String, + agent_id: String, + execution_scope: &'static str, + participant_id: Option, +} + +impl StatusIdentityKey { + fn coarse(&self) -> Self { + Self { + orchestration_session_id: self.orchestration_session_id.clone(), + agent_id: self.agent_id.clone(), + execution_scope: self.execution_scope, + participant_id: None, + } + } +} + #[derive(Clone)] struct NestedProjection { sort_key: (String, String, String), @@ -397,10 +1472,10 @@ struct NestedProjection { #[derive(Clone)] struct NestedProjectionSource { - orchestration_session_id: String, - agent_id: String, + parent_identity: StatusIdentityKey, run_id: String, parent_run_id: Option, + parent_participant_id: Option, } fn build_status_report<'a>( @@ -421,16 +1496,23 @@ fn build_status_report<'a>( scope_filter: args.scope.as_str(), role_filter, orchestrator_agent_id, + warnings: Vec::new(), sessions: Vec::new(), nested_llm_records: Vec::new(), }); } let events = read_trace_agent_events()?; - let live_manifests = AgentRuntimeStateStore::new()?.list_live_manifests()?; - let mut sessions = BTreeMap::<(String, String), SessionProjection>::new(); + let state_store = AgentRuntimeStateStore::new()?; + let status_sessions = state_store.list_status_sessions_for_agent(&orchestrator_agent_id)?; + let mut warnings = status_sessions + .iter() + .flat_map(|session| session.warnings.iter().cloned()) + .collect::>(); + let mut sessions = BTreeMap::::new(); let mut nested = BTreeMap::<(String, String, String), NestedProjection>::new(); - let mut historical_parent_runs = BTreeMap::<(String, String), BTreeSet>::new(); + let mut historical_parent_runs_exact = BTreeMap::>::new(); + let mut historical_parent_runs_coarse = BTreeMap::>::new(); for event in events { let Some(entry) = context.inventory.get(&event.agent_id) else { @@ -451,20 +1533,25 @@ fn build_status_report<'a>( &context.effective_config, ); - if let Some(session_key) = pure_session_key(&event) { - historical_parent_runs - .entry(session_key.clone()) + if let Some(identity) = pure_session_key(&event, scope) { + historical_parent_runs_exact + .entry(identity.clone()) + .or_default() + .insert(event.run_id.clone()); + historical_parent_runs_coarse + .entry(identity.coarse()) .or_default() .insert(event.run_id.clone()); - let orchestration_session_id = session_key.0.clone(); let world_id = event.world_id.clone(); let world_generation = event.world_generation; let projection = SessionProjection { last_event_ts: event.ts, session: StatusSessionJson { - orchestration_session_id: orchestration_session_id.clone(), + orchestration_session_id: identity.orchestration_session_id.clone(), + participant_id: identity.participant_id.clone(), agent_id: entry.file.id.clone(), + source_kind: "trace_fallback", backend_id: entry.derived_backend_id(), client: entry.file.id.clone(), router: PURE_AGENT_ROUTER.to_string(), @@ -473,6 +1560,9 @@ fn build_status_report<'a>( scope: scope.as_str(), }, role: role.map(ToOwned::to_owned), + posture: None, + attached_participant_id: None, + pending_inbox_count: None, last_event_at: event.ts.to_rfc3339(), world_id: if scope == AgentExecutionScope::World { world_id.clone() @@ -486,9 +1576,8 @@ fn build_status_report<'a>( }, }, source: SessionProjectionSource { - orchestration_session_id, - agent_id: entry.file.id.clone(), - run_id: event.run_id.clone(), + identity: identity.clone(), + run_id: Some(event.run_id.clone()), ts: event.ts, is_world_scoped: scope == AgentExecutionScope::World, has_top_level_world_id: world_id.is_some(), @@ -496,16 +1585,16 @@ fn build_status_report<'a>( }, }; - let should_replace = match sessions.get(&session_key) { + let should_replace = match sessions.get(&identity) { Some(existing) => projection.last_event_ts >= existing.last_event_ts, None => true, }; if should_replace { - sessions.insert(session_key, projection); + sessions.insert(identity, projection); } } - if let Some(projection) = nested_projection(&event, entry) { + if let Some(projection) = nested_projection(&event, entry, scope) { nested.insert(projection.sort_key.clone(), projection); } } @@ -526,9 +1615,9 @@ fn build_status_report<'a>( { return Err(config_model::user_error(format!( "malformed world identity on newest selected world-scoped pure-agent status event: agent_id={} orchestration_session_id={} run_id={} ts={} requires top-level world_id and world_generation", - projection.source.agent_id, - projection.source.orchestration_session_id, - projection.source.run_id, + projection.source.identity.agent_id, + projection.source.identity.orchestration_session_id, + projection.source.run_id.as_deref().unwrap_or(""), projection.source.ts.to_rfc3339(), ))); } @@ -536,37 +1625,131 @@ fn build_status_report<'a>( filtered_session_projections.push(projection); } - let selected_parent_runs: BTreeMap<(String, String), String> = filtered_session_projections + let mut selected_session_projections = Vec::new(); + for status_session in &status_sessions { + for projection in live_session_status_projections(status_session) { + if !matches_scope( + scope_from_label(projection.session.execution.scope), + args.scope, + ) || !matches_role(projection.session.role.as_deref(), role_filter) + { + continue; + } + selected_session_projections.push(projection); + } + } + + let live_fallback_suppression_exact = selected_session_projections .iter() - .map(|projection| { - ( - ( - projection.source.orchestration_session_id.clone(), - projection.source.agent_id.clone(), - ), - projection.source.run_id.clone(), - ) + .map(session_fallback_suppression_key) + .collect::>(); + let live_fallback_suppression_coarse = live_fallback_suppression_exact + .iter() + .map(StatusIdentityKey::coarse) + .collect::>(); + let invalidated_fallback_suppression_exact = state_store + .list_invalidated_participants_across_sources()? + .into_iter() + .filter(|participant| { + participant.handle.role == MEMBER_ROLE + && participant.handle.execution.scope == AgentExecutionScope::World }) - .collect(); + .map(participant_fallback_suppression_key) + .collect::>(); + let invalidated_fallback_suppression_coarse = invalidated_fallback_suppression_exact + .iter() + .map(StatusIdentityKey::coarse) + .collect::>(); + let coarse_trace_identities = filtered_session_projections + .iter() + .filter(|projection| projection.source.identity.participant_id.is_none()) + .map(|projection| projection.source.identity.coarse()) + .collect::>(); + for coarse_identity in coarse_trace_identities { + let participant_scoped_count = filtered_session_projections + .iter() + .filter(|projection| projection.source.identity.coarse() == coarse_identity) + .filter(|projection| projection.source.identity.participant_id.is_some()) + .count(); + if participant_scoped_count > 0 { + warnings.insert(format!( + "Status fallback for agent {} in orchestration session {} includes participant-less trace rows, so fallback selection remains coarse until participant-scoped records are available.", + coarse_identity.agent_id, + coarse_identity.orchestration_session_id, + )); + } + } + selected_session_projections.extend(filtered_session_projections.into_iter().filter( + |projection| { + let suppression_key = session_fallback_suppression_key(projection); + if suppression_key.participant_id.is_some() { + !live_fallback_suppression_exact.contains(&suppression_key) + && !invalidated_fallback_suppression_exact.contains(&suppression_key) + } else { + let coarse_key = suppression_key.coarse(); + !live_fallback_suppression_coarse.contains(&coarse_key) + && !invalidated_fallback_suppression_coarse.contains(&coarse_key) + } + }, + )); + + let mut selected_parent_runs_exact = + BTreeMap::>::new(); + let mut selected_parent_runs_coarse = + BTreeMap::>::new(); + for projection in &selected_session_projections { + let Some(run_id) = projection.source.run_id.as_ref() else { + continue; + }; + let selected_parent_run = SelectedParentRun { + participant_id: projection.source.identity.participant_id.clone(), + run_id: run_id.clone(), + }; + selected_parent_runs_exact + .entry(projection.source.identity.clone()) + .or_default() + .push(selected_parent_run.clone()); + selected_parent_runs_coarse + .entry(projection.source.identity.coarse()) + .or_default() + .push(selected_parent_run); + } let mut filtered_nested = Vec::new(); for projection in nested.into_values() { - let parent_key = ( - projection.source.orchestration_session_id.clone(), - projection.source.agent_id.clone(), + let selected_parent_runs = selected_parent_run_candidates( + &projection, + &selected_parent_runs_exact, + &selected_parent_runs_coarse, ); - let Some(selected_parent_run_id) = selected_parent_runs.get(&parent_key) else { + if selected_parent_runs.is_empty() { continue; - }; + } + if projection.source.parent_participant_id.is_none() + && selected_parent_runs + .iter() + .filter_map(|candidate| candidate.participant_id.as_deref()) + .collect::>() + .len() + > 1 + { + warnings.insert(format!( + "Nested status correlation for agent {} in orchestration session {} is using coarse parent matching because parent_participant_id is absent.", + projection.source.parent_identity.agent_id, + projection.source.parent_identity.orchestration_session_id, + )); + } let parent_run_id = projection.source.parent_run_id.as_deref(); - let parent_run_matches_selected = parent_run_id == Some(selected_parent_run_id.as_str()); - if parent_run_matches_selected { + if let Some(selected_parent_run) = selected_parent_runs + .iter() + .find(|candidate| parent_run_id == Some(candidate.run_id.as_str())) + { let missing_fields = missing_required_nested_fields(&projection); if !missing_fields.is_empty() { return Err(config_model::user_error(format!( "malformed nested tuple on selected status surface: agent_id={} orchestration_session_id={} run_id={} missing_fields={} requires provider and auth_authority on selected nested substrate_gateway status rows", - projection.source.agent_id, - projection.source.orchestration_session_id, + projection.source.parent_identity.agent_id, + projection.source.parent_identity.orchestration_session_id, projection.source.run_id, missing_fields.join(","), ))); @@ -574,8 +1757,13 @@ fn build_status_report<'a>( filtered_nested.push(NestedLlmRecordJson { parent: NestedParentJson { - orchestration_session_id: projection.source.orchestration_session_id.clone(), - agent_id: projection.source.agent_id.clone(), + orchestration_session_id: projection + .source + .parent_identity + .orchestration_session_id + .clone(), + participant_id: selected_parent_run.participant_id.clone(), + agent_id: projection.source.parent_identity.agent_id.clone(), }, run_id: projection.source.run_id.clone(), backend_id: projection.backend_id, @@ -593,57 +1781,40 @@ fn build_status_report<'a>( } let invalid_parent_run_id = format_invalid_parent_run_id(parent_run_id); - let historical_match = parent_run_id.is_some_and(|candidate| { - historical_parent_runs - .get(&parent_key) - .is_some_and(|runs| runs.contains(candidate)) - }); + let historical_match = historical_parent_run_matches( + &projection.source.parent_identity, + parent_run_id, + &historical_parent_runs_exact, + &historical_parent_runs_coarse, + ); if historical_match { continue; } return Err(config_model::user_error(format!( "malformed nested parent correlation on selected status surface: agent_id={} orchestration_session_id={} run_id={} parent_run_id={} requires parent_run_id to match the winning selected pure-agent run or a known historical pure-agent run for the same session", - projection.source.agent_id, - projection.source.orchestration_session_id, + projection.source.parent_identity.agent_id, + projection.source.parent_identity.orchestration_session_id, projection.source.run_id, invalid_parent_run_id, ))); } - let mut live_sessions_by_agent = BTreeMap::, StatusSessionJson)>::new(); - for manifest in live_manifests { - let session = live_manifest_status_session(&manifest); - if !matches_scope(scope_from_label(session.execution.scope), args.scope) - || !matches_role(session.role.as_deref(), role_filter) - { - continue; - } - let last_status_at = manifest.last_status_at(); - let should_replace = match live_sessions_by_agent.get(&session.agent_id) { - Some((existing_ts, _)) => last_status_at >= *existing_ts, - None => true, - }; - if should_replace { - live_sessions_by_agent.insert(session.agent_id.clone(), (last_status_at, session)); - } - } - let live_agent_ids: BTreeSet = live_sessions_by_agent.keys().cloned().collect(); - let mut filtered_sessions: Vec = live_sessions_by_agent - .into_values() - .map(|(_, session)| session) + let mut filtered_sessions: Vec = selected_session_projections + .into_iter() + .map(|projection| projection.session) .collect(); - for projection in filtered_session_projections { - if live_agent_ids.contains(&projection.session.agent_id) { - continue; - } - filtered_sessions.push(projection.session); - } filtered_sessions.sort_by(|left, right| { - left.agent_id.cmp(&right.agent_id).then( - left.orchestration_session_id - .cmp(&right.orchestration_session_id), - ) + left.orchestration_session_id + .cmp(&right.orchestration_session_id) + .then( + left.participant_id + .as_deref() + .unwrap_or("") + .cmp(right.participant_id.as_deref().unwrap_or("")), + ) + .then(left.agent_id.cmp(&right.agent_id)) + .then(left.source_kind.cmp(right.source_kind)) }); Ok(StatusReportJson { @@ -651,6 +1822,7 @@ fn build_status_report<'a>( scope_filter: args.scope.as_str(), role_filter, orchestrator_agent_id, + warnings: warnings.into_iter().collect(), sessions: filtered_sessions, nested_llm_records: filtered_nested, }) @@ -668,6 +1840,12 @@ fn render_status_report(report: &StatusReportJson<'_>, json_mode: bool) -> Resul if let Some(role_filter) = report.role_filter { println!("role_filter: {role_filter}"); } + if !report.warnings.is_empty() { + println!("warnings:"); + for warning in &report.warnings { + println!(" - {warning}"); + } + } println!(); println!("orchestrator"); @@ -681,13 +1859,33 @@ fn render_status_report(report: &StatusReportJson<'_>, json_mode: bool) -> Resul "orchestration_session_id={}", session.orchestration_session_id ), + format!( + "participant_id={}", + session.participant_id.as_deref().unwrap_or("") + ), format!("agent_id={}", session.agent_id), + format!("source_kind={}", session.source_kind), format!("backend_id={}", session.backend_id), format!("client={}", session.client), format!("router={}", session.router), format!("protocol={}", session.protocol), format!("execution.scope={}", session.execution.scope), format!("role={}", session.role.as_deref().unwrap_or("")), + format!( + "posture={}", + human_status_option_str(session.posture.as_deref(), session.source_kind) + ), + format!( + "attached_participant_id={}", + human_status_option_str( + session.attached_participant_id.as_deref(), + session.source_kind + ) + ), + format!( + "pending_inbox_count={}", + human_status_option_u64(session.pending_inbox_count, session.source_kind) + ), format!("last_event_at={}", session.last_event_at), ]; if let (Some(world_id), Some(world_generation)) = @@ -704,8 +1902,9 @@ fn render_status_report(report: &StatusReportJson<'_>, json_mode: bool) -> Resul println!("nested_llm_records"); for record in &report.nested_llm_records { println!( - " parent.orchestration_session_id={} | parent.agent_id={} | run_id={} | backend_id={} | client={} | router={} | provider={} | auth_authority={} | protocol={}", + " parent.orchestration_session_id={} | parent.participant_id={} | parent.agent_id={} | run_id={} | backend_id={} | client={} | router={} | provider={} | auth_authority={} | protocol={}", record.parent.orchestration_session_id, + record.parent.participant_id.as_deref().unwrap_or(""), record.parent.agent_id, record.run_id, record.backend_id, @@ -801,6 +2000,12 @@ struct ToolboxEligibilityJson { reason: Option, } +#[derive(Clone, Serialize)] +struct ToolboxActiveWorldBindingJson { + world_id: String, + world_generation: u64, +} + #[derive(Clone, Serialize)] struct ToolboxStatusReportJson<'a> { toolbox_enabled: bool, @@ -811,6 +2016,8 @@ struct ToolboxStatusReportJson<'a> { #[serde(skip_serializing_if = "Option::is_none")] endpoint_template: Option, active_orchestration_session_id: Option, + #[serde(skip_serializing_if = "Option::is_none")] + active_world_binding: Option, eligibility: ToolboxEligibilityJson, #[serde(skip_serializing_if = "Option::is_none")] orchestrator: Option>, @@ -837,6 +2044,7 @@ fn build_toolbox_status_report<'a>( endpoint: None, endpoint_template: None, active_orchestration_session_id: None, + active_world_binding: None, eligibility: ToolboxEligibilityJson { state: "disabled".to_string(), reason: Some("agents are disabled by effective config".to_string()), @@ -863,6 +2071,7 @@ fn build_toolbox_status_report<'a>( endpoint: None, endpoint_template: None, active_orchestration_session_id: None, + active_world_binding: None, eligibility: ToolboxEligibilityJson { state: "disabled".to_string(), reason: Some("agents.toolbox.enabled is false in the effective config".to_string()), @@ -879,6 +2088,7 @@ fn build_toolbox_status_report<'a>( endpoint: None, endpoint_template: None, active_orchestration_session_id: None, + active_world_binding: None, eligibility: ToolboxEligibilityJson { state: "denied".to_string(), reason: Some(format!( @@ -898,6 +2108,7 @@ fn build_toolbox_status_report<'a>( endpoint: None, endpoint_template: None, active_orchestration_session_id: None, + active_world_binding: None, eligibility: ToolboxEligibilityJson { state: "unsupported".to_string(), reason: Some( @@ -909,19 +2120,23 @@ fn build_toolbox_status_report<'a>( }), AgentToolboxBindTransport::Uds => { let endpoint_template = Some(toolbox_uds_endpoint_template()?); - let latest_session = - AgentRuntimeStateStore::new()?.find_live_orchestrator(&orchestrator.file.id)?; + let latest_session = AgentRuntimeStateStore::new()? + .resolve_single_live_session_for_agent(&orchestrator.file.id) + .map_err(|err| config_model::user_error(err.to_string()))?; match latest_session { - Some(session) => Ok(ToolboxStatusReportJson { + Some(session_record) => Ok(ToolboxStatusReportJson { toolbox_enabled: true, toolbox_version: TOOLBOX_VERSION, transport, - endpoint: Some(toolbox_uds_endpoint(&session.handle.orchestration_session_id)?), + endpoint: Some(toolbox_uds_endpoint( + session_record.orchestration_session_id(), + )?), endpoint_template, active_orchestration_session_id: Some( - session.handle.orchestration_session_id, + session_record.orchestration_session_id().to_string(), ), + active_world_binding: toolbox_active_world_binding(&session_record.session), eligibility: ToolboxEligibilityJson { state: "allowed".to_string(), reason: None, @@ -935,10 +2150,11 @@ fn build_toolbox_status_report<'a>( endpoint: None, endpoint_template, active_orchestration_session_id: None, + active_world_binding: None, eligibility: ToolboxEligibilityJson { state: "dependency_unavailable".to_string(), reason: Some( - "no active pure-agent orchestrator session found in authoritative live manifests for the selected orchestrator" + "no live host-scoped orchestrator participant found for the selected orchestrator" .to_string(), ), }, @@ -977,6 +2193,12 @@ fn render_toolbox_status_report( if let Some(endpoint_template) = &report.endpoint_template { println!("endpoint_template: {endpoint_template}"); } + if let Some(binding) = &report.active_world_binding { + println!( + "active_world_binding: world_id={} | world_generation={}", + binding.world_id, binding.world_generation + ); + } if let Some(orchestrator) = &report.orchestrator { println!( "orchestrator: agent_id={} | backend_id={} | role={} | execution.scope={}", @@ -990,6 +2212,21 @@ fn render_toolbox_status_report( Ok(()) } +fn toolbox_active_world_binding( + session: &crate::execution::agent_runtime::OrchestrationSessionRecord, +) -> Option { + let world_id = session.world_id.as_ref()?; + let world_generation = session.world_generation?; + if world_id.trim().is_empty() { + return None; + } + + Some(ToolboxActiveWorldBindingJson { + world_id: world_id.clone(), + world_generation, + }) +} + fn build_toolbox_env_report(report: &ToolboxStatusReportJson<'_>) -> Result { Ok(ToolboxEnvReportJson { substrate_agent_toolbox_endpoint: report @@ -1050,6 +2287,99 @@ fn shell_single_quote(value: &str) -> String { format!("'{}'", value.replace('\'', "'\\''")) } +fn live_participant_status_projection( + session: &OrchestrationSessionRecord, + participant: &AgentRuntimeParticipantRecord, +) -> SessionProjection { + SessionProjection { + last_event_ts: participant.last_status_at(), + session: StatusSessionJson { + orchestration_session_id: participant.handle.orchestration_session_id.clone(), + participant_id: Some(participant.handle.participant_id.clone()), + agent_id: participant.handle.agent_id.clone(), + source_kind: "live_runtime", + backend_id: participant.handle.backend_id.clone(), + client: participant.handle.agent_id.clone(), + router: PURE_AGENT_ROUTER.to_string(), + protocol: participant.handle.protocol.clone(), + execution: ExecutionScopeJson { + scope: match participant.handle.execution.scope { + AgentExecutionScope::Host => "host", + AgentExecutionScope::World => "world", + }, + }, + role: Some(participant.handle.role.clone()), + posture: Some(orchestration_session_posture_label(session.posture).to_string()), + attached_participant_id: session.attached_participant_id.clone(), + pending_inbox_count: Some(session.pending_inbox_count), + last_event_at: participant.last_status_at().to_rfc3339(), + world_id: participant.handle.world_id.clone(), + world_generation: participant.handle.world_generation, + }, + source: SessionProjectionSource { + identity: StatusIdentityKey { + orchestration_session_id: participant.handle.orchestration_session_id.clone(), + agent_id: participant.handle.agent_id.clone(), + execution_scope: execution_scope_label(participant.handle.execution.scope), + participant_id: Some(participant.handle.participant_id.clone()), + }, + run_id: participant.internal.latest_run_id.clone(), + ts: participant.last_status_at(), + is_world_scoped: participant.handle.execution.scope == AgentExecutionScope::World, + has_top_level_world_id: participant.handle.world_id.is_some(), + has_top_level_world_generation: participant.handle.world_generation.is_some(), + }, + } +} + +fn live_session_status_projections(session: &AgentRuntimeSessionRecord) -> Vec { + session + .status_visible_participants() + .into_iter() + .map(|participant| live_participant_status_projection(&session.session, &participant)) + .collect() +} + +fn orchestration_session_posture_label(posture: OrchestrationSessionPosture) -> &'static str { + match posture { + OrchestrationSessionPosture::ActiveAttached => "active_attached", + OrchestrationSessionPosture::ParkedResumable => "parked_resumable", + OrchestrationSessionPosture::AwaitingAttention => "awaiting_attention", + OrchestrationSessionPosture::Terminal => "terminal", + } +} + +fn human_status_option_str(value: Option<&str>, source_kind: &str) -> String { + match value { + Some(value) => value.to_string(), + None if source_kind == "trace_fallback" => "".to_string(), + None => "".to_string(), + } +} + +fn human_status_option_u64(value: Option, source_kind: &str) -> String { + match value { + Some(value) => value.to_string(), + None if source_kind == "trace_fallback" => "".to_string(), + None => "".to_string(), + } +} + +fn session_fallback_suppression_key(projection: &SessionProjection) -> StatusIdentityKey { + projection.source.identity.clone() +} + +fn participant_fallback_suppression_key( + participant: AgentRuntimeParticipantRecord, +) -> StatusIdentityKey { + StatusIdentityKey { + orchestration_session_id: participant.handle.orchestration_session_id, + agent_id: participant.handle.agent_id, + execution_scope: execution_scope_label(participant.handle.execution.scope), + participant_id: Some(participant.handle.participant_id), + } +} + fn read_trace_agent_events() -> Result> { let trace_path = trace_log_path()?; let file = match File::open(&trace_path) { @@ -1096,20 +2426,23 @@ fn trace_log_path() -> Result { Ok(substrate_paths::substrate_home()?.join("trace.jsonl")) } -fn pure_session_key(event: &AgentEvent) -> Option<(String, String)> { +fn pure_session_key(event: &AgentEvent, scope: AgentExecutionScope) -> Option { let tuple = event.identity_tuple.as_ref()?; if tuple.router != PURE_AGENT_ROUTER || tuple.protocol != PURE_AGENT_PROTOCOL { return None; } - Some(( - event.orchestration_session_id.clone(), - event.agent_id.clone(), - )) + Some(StatusIdentityKey { + orchestration_session_id: event.orchestration_session_id.clone(), + agent_id: event.agent_id.clone(), + execution_scope: execution_scope_label(scope), + participant_id: event.participant_id.clone(), + }) } fn nested_projection( event: &AgentEvent, entry: &AgentInventoryEntryV1, + scope: AgentExecutionScope, ) -> Option { let tuple = event.identity_tuple.as_ref()?; if tuple.router != NESTED_ROUTER { @@ -1124,10 +2457,15 @@ fn nested_projection( Some(NestedProjection { sort_key, source: NestedProjectionSource { - orchestration_session_id: event.orchestration_session_id.clone(), - agent_id: event.agent_id.clone(), + parent_identity: StatusIdentityKey { + orchestration_session_id: event.orchestration_session_id.clone(), + agent_id: event.agent_id.clone(), + execution_scope: execution_scope_label(scope), + participant_id: event.parent_participant_id.clone(), + }, run_id: event.run_id.clone(), parent_run_id: event.parent_run_id.clone(), + parent_participant_id: event.parent_participant_id.clone(), }, backend_id: entry.derived_backend_id(), client: event.agent_id.clone(), @@ -1156,6 +2494,44 @@ fn format_invalid_parent_run_id(parent_run_id: Option<&str>) -> String { } } +fn selected_parent_run_candidates<'a>( + projection: &NestedProjection, + exact: &'a BTreeMap>, + coarse: &'a BTreeMap>, +) -> Vec<&'a SelectedParentRun> { + if let Some(selected) = exact.get(&projection.source.parent_identity) { + return selected.iter().collect(); + } + + coarse + .get(&projection.source.parent_identity.coarse()) + .map(|selected| selected.iter().collect()) + .unwrap_or_default() +} + +fn historical_parent_run_matches( + parent_identity: &StatusIdentityKey, + parent_run_id: Option<&str>, + exact: &BTreeMap>, + coarse: &BTreeMap>, +) -> bool { + parent_run_id.is_some_and(|candidate| { + exact + .get(parent_identity) + .is_some_and(|runs| runs.contains(candidate)) + || coarse + .get(&parent_identity.coarse()) + .is_some_and(|runs| runs.contains(candidate)) + }) +} + +fn execution_scope_label(scope: AgentExecutionScope) -> &'static str { + match scope { + AgentExecutionScope::Host => "host", + AgentExecutionScope::World => "world", + } +} + fn scope_for_event( event: &AgentEvent, entry: &AgentInventoryEntryV1, @@ -1228,16 +2604,6 @@ fn backend_allowed(policy: &Policy, backend_id: &str) -> bool { .any(|allowed| allowed == backend_id) } -fn enabled_world_member_exists( - inventory: &BTreeMap, - effective_config: &SubstrateConfig, -) -> bool { - inventory.values().any(|entry| { - entry.file.config.enabled - && entry.effective_scope(effective_config) == AgentExecutionScope::World - }) -} - #[derive(Serialize)] struct DoctorOrchestratorJson<'a> { agent_id: String, @@ -1263,6 +2629,8 @@ struct DoctorReportJson<'a> { world_boundary_exit_code: Option, #[serde(skip)] runtime_realizability_exit_code: Option, + #[serde(skip)] + member_selection_exit_code: Option, } enum RequiredWorldBoundaryState { @@ -1349,6 +2717,7 @@ fn build_doctor_report(cli: &Cli) -> Result> { checks, world_boundary_exit_code: None, runtime_realizability_exit_code: None, + member_selection_exit_code: None, }); } }; @@ -1381,12 +2750,80 @@ fn build_doctor_report(cli: &Cli) -> Result> { checks, world_boundary_exit_code: None, runtime_realizability_exit_code: Some(exit_code), + member_selection_exit_code: None, }); } }; let _ = runtime_descriptor; - if let Some(reason) = policy_allowlist_failure(&effective_config, &inventory, &base_policy) { + match validate_passive_participant_store() { + Ok(()) => { + checks.push(DoctorCheckJson { + check: "participant_store".to_string(), + status: "pass".to_string(), + reason: None, + }); + } + Err(err) => { + checks.push(DoctorCheckJson { + check: "participant_store".to_string(), + status: "fail".to_string(), + reason: Some(err.to_string()), + }); + return Ok(DoctorReportJson { + healthy: false, + fail_closed: true, + orchestrator: Some(orchestrator), + checks, + world_boundary_exit_code: None, + runtime_realizability_exit_code: None, + member_selection_exit_code: None, + }); + } + } + + let selected_member = match validate_member_selection(&effective_config, &inventory) { + Ok(Some(descriptor)) => { + checks.push(DoctorCheckJson { + check: "member_selection".to_string(), + status: "pass".to_string(), + reason: None, + }); + Some(descriptor) + } + Ok(None) => { + checks.push(DoctorCheckJson { + check: "member_selection".to_string(), + status: "not_applicable".to_string(), + reason: None, + }); + None + } + Err(err) => { + let exit_code = member_selection_error_exit_code(&err); + checks.push(DoctorCheckJson { + check: "member_selection".to_string(), + status: "fail".to_string(), + reason: Some(err.reason), + }); + return Ok(DoctorReportJson { + healthy: false, + fail_closed: true, + orchestrator: Some(orchestrator), + checks, + world_boundary_exit_code: None, + runtime_realizability_exit_code: None, + member_selection_exit_code: Some(exit_code), + }); + } + }; + + if let Some(reason) = policy_allowlist_failure( + &effective_config, + &inventory, + &base_policy, + selected_member.as_ref(), + ) { checks.push(DoctorCheckJson { check: "policy_allowlist".to_string(), status: "fail".to_string(), @@ -1399,6 +2836,7 @@ fn build_doctor_report(cli: &Cli) -> Result> { checks, world_boundary_exit_code: None, runtime_realizability_exit_code: None, + member_selection_exit_code: None, }); } checks.push(DoctorCheckJson { @@ -1407,7 +2845,7 @@ fn build_doctor_report(cli: &Cli) -> Result> { reason: None, }); - if enabled_world_member_exists(&inventory, &effective_config) { + if selected_member.is_some() { if !effective_config.world.enabled { checks.push(DoctorCheckJson { check: "world_boundary".to_string(), @@ -1424,6 +2862,7 @@ fn build_doctor_report(cli: &Cli) -> Result> { checks, world_boundary_exit_code: Some(3), runtime_realizability_exit_code: None, + member_selection_exit_code: None, }); } @@ -1448,6 +2887,7 @@ fn build_doctor_report(cli: &Cli) -> Result> { checks, world_boundary_exit_code: Some(exit_code), runtime_realizability_exit_code: None, + member_selection_exit_code: None, }); } } @@ -1466,6 +2906,7 @@ fn build_doctor_report(cli: &Cli) -> Result> { checks, world_boundary_exit_code: None, runtime_realizability_exit_code: None, + member_selection_exit_code: None, }) } @@ -1485,6 +2926,7 @@ fn failed_doctor_report( }], world_boundary_exit_code: None, runtime_realizability_exit_code: None, + member_selection_exit_code: None, } } @@ -1623,6 +3065,9 @@ fn policy_allowlist_failure( effective_config: &SubstrateConfig, inventory: &BTreeMap, base_policy: &Policy, + selected_member: Option< + &crate::execution::agent_runtime::validator::RuntimeSelectionDescriptor, + >, ) -> Option { let orchestrator = validate_orchestrator_selection(effective_config, inventory).ok()?; let orchestrator_backend_id = orchestrator.derived_backend_id(); @@ -1633,17 +3078,11 @@ fn policy_allowlist_failure( )); } - for entry in inventory.values() { - if !entry.file.config.enabled - || entry.effective_scope(effective_config) != AgentExecutionScope::World - { - continue; - } - let backend_id = entry.derived_backend_id(); - if !backend_allowed(base_policy, &backend_id) { + if let Some(selected_member) = selected_member { + if !backend_allowed(base_policy, &selected_member.backend_id) { return Some(format!( "required world-scoped member backend '{}' is not allowlisted by effective policy agents.allowed_backends", - backend_id + selected_member.backend_id )); } } @@ -1669,30 +3108,16 @@ fn doctor_exit_code(report: &DoctorReportJson<'_>) -> i32 { "policy_allowlist" => 5, "world_boundary" => report.world_boundary_exit_code.unwrap_or(3), "runtime_realizability" => report.runtime_realizability_exit_code.unwrap_or(2), - "inventory_scan" | "orchestrator_selection" => 2, + "member_selection" => report.member_selection_exit_code.unwrap_or(2), + "inventory_scan" | "orchestrator_selection" | "participant_store" => 2, _ => 1, } } -fn live_manifest_status_session(manifest: &AgentRuntimeSessionManifest) -> StatusSessionJson { - StatusSessionJson { - orchestration_session_id: manifest.handle.orchestration_session_id.clone(), - agent_id: manifest.handle.agent_id.clone(), - backend_id: manifest.handle.backend_id.clone(), - client: manifest.handle.agent_id.clone(), - router: PURE_AGENT_ROUTER.to_string(), - protocol: manifest.handle.protocol.clone(), - execution: ExecutionScopeJson { - scope: match manifest.handle.execution.scope { - AgentExecutionScope::Host => "host", - AgentExecutionScope::World => "world", - }, - }, - role: Some(manifest.handle.role.clone()), - last_event_at: manifest.last_status_at().to_rfc3339(), - world_id: manifest.handle.world_id.clone(), - world_generation: manifest.handle.world_generation, - } +fn validate_passive_participant_store() -> Result<()> { + let state_store = AgentRuntimeStateStore::new()?; + state_store.list_participants()?; + Ok(()) } fn render_doctor_report(report: &DoctorReportJson<'_>, json_mode: bool) -> Result<()> { diff --git a/crates/shell/src/execution/cli.rs b/crates/shell/src/execution/cli.rs index a9501d9a0..1c6fa52f0 100644 --- a/crates/shell/src/execution/cli.rs +++ b/crates/shell/src/execution/cli.rs @@ -402,11 +402,25 @@ pub struct AgentCmd { pub action: AgentAction, } -#[derive(Copy, Clone, Debug, ValueEnum, PartialEq, Eq)] +#[derive(Args, Debug, Default)] +#[group(id = "public_prompt_source", required = true, multiple = false)] +pub struct PublicPromptArgs { + #[arg(long, group = "public_prompt_source", value_name = "TEXT")] + pub prompt: Option, + #[arg( + long = "prompt-file", + group = "public_prompt_source", + value_name = "PATH" + )] + pub prompt_file: Option, +} + +#[derive(Copy, Clone, Debug, Default, ValueEnum, PartialEq, Eq)] #[value(rename_all = "snake_case")] pub enum AgentScopeArg { Host, World, + #[default] Any, } @@ -420,12 +434,6 @@ impl AgentScopeArg { } } -impl Default for AgentScopeArg { - fn default() -> Self { - Self::Any - } -} - #[derive(Args, Debug, Default)] pub struct AgentViewArgs { /// Emit JSON instead of human-readable output @@ -446,6 +454,42 @@ pub struct AgentDoctorArgs { pub json: bool, } +#[derive(Args, Debug)] +pub struct AgentOwnerHelperArgs { + #[arg(long = "plan-file", value_name = "PATH")] + pub plan_file: PathBuf, +} + +#[derive(Args, Debug, Default)] +pub struct AgentStartArgs { + #[arg(long = "backend", value_name = "BACKEND_ID")] + pub backend: String, + #[command(flatten)] + pub prompt_source: PublicPromptArgs, + #[arg(long)] + pub json: bool, +} + +#[derive(Args, Debug, Default)] +pub struct AgentSessionControlArgs { + #[arg(long = "session", value_name = "ORCHESTRATION_SESSION_ID")] + pub session: String, + #[arg(long)] + pub json: bool, +} + +#[derive(Args, Debug, Default)] +pub struct AgentTurnArgs { + #[arg(long = "session", value_name = "ORCHESTRATION_SESSION_ID")] + pub session: String, + #[arg(long = "backend", value_name = "BACKEND_ID")] + pub backend: String, + #[command(flatten)] + pub prompt_source: PublicPromptArgs, + #[arg(long)] + pub json: bool, +} + #[derive(Args, Debug)] pub struct AgentToolboxCmd { #[command(subcommand)] @@ -475,6 +519,19 @@ pub enum AgentAction { Status(AgentViewArgs), /// Validate deterministic startability of the agent control plane Doctor(AgentDoctorArgs), + /// Start a new host-scoped orchestration session from an exact backend id + Start(AgentStartArgs), + /// Submit a follow-up prompt to the exact backend in an orchestration session + Turn(AgentTurnArgs), + /// Reattach a retained owner loop to the exact orchestration session + #[command(name = "reattach", alias = "resume")] + Reattach(AgentSessionControlArgs), + /// Fork a new orchestration session from the exact orchestration session + Fork(AgentSessionControlArgs), + /// Stop the exact orchestration session through the private owner transport + Stop(AgentSessionControlArgs), + #[command(name = "__owner-helper", hide = true)] + OwnerHelper(AgentOwnerHelperArgs), /// Inspect the internal orchestration toolbox surface Toolbox(AgentToolboxCmd), } diff --git a/crates/shell/src/execution/config_model.rs b/crates/shell/src/execution/config_model.rs index aabd0cd4e..25e9bb254 100644 --- a/crates/shell/src/execution/config_model.rs +++ b/crates/shell/src/execution/config_model.rs @@ -107,19 +107,14 @@ pub(crate) struct SubstrateConfig { pub agents: AgentsConfig, } -#[derive(Debug, Clone, Copy, Serialize, PartialEq, Eq)] +#[derive(Debug, Clone, Copy, Default, Serialize, PartialEq, Eq)] #[serde(rename_all = "snake_case")] pub(crate) enum ReplExitCwdMode { + #[default] Entered, LastWorld, } -impl Default for ReplExitCwdMode { - fn default() -> Self { - Self::Entered - } -} - impl ReplExitCwdMode { pub(crate) fn parse_insensitive(raw: &str) -> Option { match raw.trim().to_ascii_lowercase().as_str() { @@ -187,19 +182,14 @@ pub(crate) struct LlmGatewayConfig { pub mode: LlmGatewayMode, } -#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)] +#[derive(Debug, Clone, Copy, Default, Serialize, Deserialize, PartialEq, Eq)] #[serde(rename_all = "snake_case")] pub(crate) enum LlmGatewayMode { + #[default] InWorld, HostOnly, } -impl Default for LlmGatewayMode { - fn default() -> Self { - Self::InWorld - } -} - impl LlmGatewayMode { pub(crate) fn parse_insensitive(raw: &str) -> Option { match raw.trim().to_ascii_lowercase().as_str() { @@ -262,19 +252,14 @@ pub(crate) struct WorldDepsConfig { pub builtins: WorldDepsBuiltinsMode, } -#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)] +#[derive(Debug, Clone, Copy, Default, Serialize, Deserialize, PartialEq, Eq)] #[serde(rename_all = "snake_case")] pub(crate) enum WorldDepsInventoryMode { + #[default] Merged, WorkspaceOnly, } -impl Default for WorldDepsInventoryMode { - fn default() -> Self { - Self::Merged - } -} - impl WorldDepsInventoryMode { pub(crate) fn parse_insensitive(raw: &str) -> Option { match raw.trim().to_ascii_lowercase().as_str() { @@ -285,19 +270,14 @@ impl WorldDepsInventoryMode { } } -#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)] +#[derive(Debug, Clone, Copy, Default, Serialize, Deserialize, PartialEq, Eq)] #[serde(rename_all = "snake_case")] pub(crate) enum WorldDepsBuiltinsMode { + #[default] Enabled, Disabled, } -impl Default for WorldDepsBuiltinsMode { - fn default() -> Self { - Self::Enabled - } -} - impl WorldDepsBuiltinsMode { pub(crate) fn parse_insensitive(raw: &str) -> Option { match raw.trim().to_ascii_lowercase().as_str() { @@ -308,20 +288,15 @@ impl WorldDepsBuiltinsMode { } } -#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)] +#[derive(Debug, Clone, Copy, Default, Serialize, Deserialize, PartialEq, Eq)] #[serde(rename_all = "kebab-case")] pub(crate) enum PolicyMode { Disabled, + #[default] Observe, Enforce, } -impl Default for PolicyMode { - fn default() -> Self { - Self::Observe - } -} - impl PolicyMode { pub(crate) fn parse_insensitive(raw: &str) -> Option { match raw.trim().to_ascii_lowercase().as_str() { @@ -347,20 +322,15 @@ pub(crate) struct PolicyConfig { pub mode: PolicyMode, } -#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)] +#[derive(Debug, Clone, Copy, Default, Serialize, Deserialize, PartialEq, Eq)] #[serde(rename_all = "snake_case")] pub(crate) enum SyncDirection { + #[default] FromWorld, FromHost, Both, } -impl Default for SyncDirection { - fn default() -> Self { - Self::FromWorld - } -} - impl SyncDirection { pub(crate) fn parse_insensitive(raw: &str) -> Option { match raw.trim().to_ascii_lowercase().as_str() { @@ -372,20 +342,15 @@ impl SyncDirection { } } -#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)] +#[derive(Debug, Clone, Copy, Default, Serialize, Deserialize, PartialEq, Eq)] #[serde(rename_all = "snake_case")] pub(crate) enum SyncConflictPolicy { + #[default] PreferHost, PreferWorld, Abort, } -impl Default for SyncConflictPolicy { - fn default() -> Self { - Self::PreferHost - } -} - impl SyncConflictPolicy { pub(crate) fn parse_insensitive(raw: &str) -> Option { match raw.trim().to_ascii_lowercase().as_str() { @@ -428,19 +393,14 @@ pub(crate) struct AgentDefaultsExecutionConfig { pub scope: AgentExecutionScope, } -#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)] +#[derive(Debug, Clone, Copy, Default, Serialize, Deserialize, PartialEq, Eq)] #[serde(rename_all = "snake_case")] pub(crate) enum AgentExecutionScope { Host, + #[default] World, } -impl Default for AgentExecutionScope { - fn default() -> Self { - Self::World - } -} - impl AgentExecutionScope { pub(crate) fn parse_insensitive(raw: &str) -> Option { match raw.trim().to_ascii_lowercase().as_str() { @@ -457,19 +417,14 @@ pub(crate) struct AgentDefaultsCliConfig { pub mode: AgentCliMode, } -#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)] +#[derive(Debug, Clone, Copy, Default, Serialize, Deserialize, PartialEq, Eq)] #[serde(rename_all = "snake_case")] pub(crate) enum AgentCliMode { + #[default] Persistent, PerRequest, } -impl Default for AgentCliMode { - fn default() -> Self { - Self::Persistent - } -} - impl AgentCliMode { pub(crate) fn parse_insensitive(raw: &str) -> Option { match raw.trim().to_ascii_lowercase().as_str() { @@ -493,19 +448,14 @@ pub(crate) struct AgentToolboxBindConfig { pub transport: AgentToolboxBindTransport, } -#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)] +#[derive(Debug, Clone, Copy, Default, Serialize, Deserialize, PartialEq, Eq)] #[serde(rename_all = "snake_case")] pub(crate) enum AgentToolboxBindTransport { + #[default] Uds, Tcp, } -impl Default for AgentToolboxBindTransport { - fn default() -> Self { - Self::Uds - } -} - impl AgentToolboxBindTransport { pub(crate) fn parse_insensitive(raw: &str) -> Option { match raw.trim().to_ascii_lowercase().as_str() { @@ -529,19 +479,14 @@ pub(crate) struct AgentHubWorldRestartConfig { pub on_drift: WorldRestartOnDriftMode, } -#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)] +#[derive(Debug, Clone, Copy, Default, Serialize, Deserialize, PartialEq, Eq)] #[serde(rename_all = "snake_case")] pub(crate) enum WorldRestartOnDriftMode { + #[default] AutoRestart, FailClosed, } -impl Default for WorldRestartOnDriftMode { - fn default() -> Self { - Self::AutoRestart - } -} - impl WorldRestartOnDriftMode { pub(crate) fn parse_insensitive(raw: &str) -> Option { match raw.trim().to_ascii_lowercase().as_str() { diff --git a/crates/shell/src/execution/invocation/runtime.rs b/crates/shell/src/execution/invocation/runtime.rs index 4f4ce8349..f0acfc787 100644 --- a/crates/shell/src/execution/invocation/runtime.rs +++ b/crates/shell/src/execution/invocation/runtime.rs @@ -80,7 +80,7 @@ pub(crate) fn run_wrap_mode(config: &ShellConfig, command: &str) -> Result // Set up signal handlers for wrap mode to properly handle signals like SIGTERM setup_signal_handlers(running_child_pid.clone())?; - let status = execute_command(config, command, &cmd_id, running_child_pid)?; + let status = execute_command(config, command, &cmd_id, running_child_pid, None)?; Ok(exit_code(status)) } @@ -274,7 +274,7 @@ pub(crate) fn run_pipe_mode(config: &ShellConfig) -> Result { } let cmd_id = Uuid::now_v7().to_string(); - match execute_command(config, &line, &cmd_id, no_signal_handler.clone()) { + match execute_command(config, &line, &cmd_id, no_signal_handler.clone(), None) { Ok(status) => { last_status = exit_code(status); if !status.success() && config.ci_mode && !config.no_exit_on_error { diff --git a/crates/shell/src/execution/invocation/tests.rs b/crates/shell/src/execution/invocation/tests.rs index bb383cffb..6ff02fd50 100644 --- a/crates/shell/src/execution/invocation/tests.rs +++ b/crates/shell/src/execution/invocation/tests.rs @@ -23,6 +23,24 @@ fn restore_env(key: &str, previous: Option) { } } +struct CurrentDirGuard { + previous: PathBuf, +} + +impl CurrentDirGuard { + fn change_to(path: &std::path::Path) -> Self { + let previous = std::env::current_dir().expect("capture cwd"); + std::env::set_current_dir(path).expect("set cwd"); + Self { previous } + } +} + +impl Drop for CurrentDirGuard { + fn drop(&mut self) { + let _ = std::env::set_current_dir(&self.previous); + } +} + #[test] #[serial] fn wrap_mode_uses_cli_shell_and_shimmed_path() { @@ -32,6 +50,7 @@ fn wrap_mode_uses_cli_shell_and_shimmed_path() { let home = temp.path().join("home"); let substrate_home = home.join(".substrate"); fs::create_dir_all(substrate_home.join("shims")).unwrap(); + let _cwd_guard = CurrentDirGuard::change_to(temp.path()); let home_str = home.display().to_string(); let substrate_home_str = substrate_home.display().to_string(); @@ -92,6 +111,7 @@ fn skip_shims_and_no_world_disable_shimmed_path() { let home = temp.path().join("home"); let substrate_home = home.join(".substrate"); fs::create_dir_all(substrate_home.join("shims")).unwrap(); + let _cwd_guard = CurrentDirGuard::change_to(temp.path()); let home_str = home.display().to_string(); let substrate_home_str = substrate_home.display().to_string(); diff --git a/crates/shell/src/execution/mod.rs b/crates/shell/src/execution/mod.rs index 900515a9a..4ea8c7aaf 100644 --- a/crates/shell/src/execution/mod.rs +++ b/crates/shell/src/execution/mod.rs @@ -1,3 +1,5 @@ +#[doc(hidden)] +pub mod agent_dev_support; pub mod agent_events; pub(crate) mod agent_inventory; pub(crate) mod agent_runtime; diff --git a/crates/shell/src/execution/platform/linux.rs b/crates/shell/src/execution/platform/linux.rs index 593b79d7e..93a916375 100644 --- a/crates/shell/src/execution/platform/linux.rs +++ b/crates/shell/src/execution/platform/linux.rs @@ -820,8 +820,10 @@ async fn legacy_world_doctor_report_v1_via_execute( agent_id: "doctor-world-probe".to_string(), budget: None, policy_snapshot, + shared_world: None, world_network: Some(world_network), world_fs_mode: Some(WorldFsMode::Writable), + member_dispatch: None, }; let resp = client.execute(req).await?; diff --git a/crates/shell/src/execution/platform_world/mod.rs b/crates/shell/src/execution/platform_world/mod.rs index 15074e663..861e1acca 100644 --- a/crates/shell/src/execution/platform_world/mod.rs +++ b/crates/shell/src/execution/platform_world/mod.rs @@ -11,9 +11,14 @@ pub fn detect() -> Result { use crate::execution::policy_snapshot::bootstrap_world_spec; #[cfg(not(target_os = "windows"))] use crate::execution::settings; +use agent_api_types::SharedWorldOwnerSpec; +#[cfg(target_os = "linux")] +use anyhow::Context; use anyhow::Result; use std::fmt; +use std::future::Future; use std::path::PathBuf; +use std::pin::Pin; use std::sync::Arc; use std::sync::OnceLock; use world_api::WorldBackend; @@ -53,10 +58,74 @@ pub struct PlatformWorldContext { #[allow(dead_code)] pub socket_path: PathBuf, pub ensure_ready: Box anyhow::Result<()> + Send + Sync>, + #[allow(dead_code)] + pub ensure_persistent_session_ready_async: Box, } +pub type PersistentSessionReadyFuture = + Pin> + Send + 'static>>; +pub type PersistentSessionReadyFn = dyn Fn() -> PersistentSessionReadyFuture + Send + Sync; + static GLOBAL_CTX: OnceLock> = OnceLock::new(); +fn validate_shared_owner_request_support( + request: Option<&SharedWorldOwnerSpec>, + operation: &str, +) -> Result<()> { + let Some(request) = request else { + return Ok(()); + }; + + #[cfg(target_os = "linux")] + { + let _ = operation; + let _ = request; + return Ok(()); + } + + #[cfg(target_os = "macos")] + { + if std::env::var_os("SUBSTRATE_WORLD_SOCKET").is_some() { + anyhow::bail!( + "{} rejects explicit shared-owner world reuse when SUBSTRATE_WORLD_SOCKET overrides the Lima-backed transport (orchestration_session_id={})", + operation, + request.orchestration_session_id + ); + } + + Ok(()) + } + + #[cfg(not(any(target_os = "linux", target_os = "macos")))] + { + anyhow::bail!( + "{} rejects explicit shared-owner world reuse on this platform (orchestration_session_id={})", + operation, + request.orchestration_session_id + ); + } +} + +#[cfg(target_os = "macos")] +pub(crate) fn reject_non_linux_shared_owner_request( + request: Option<&SharedWorldOwnerSpec>, + operation: &str, +) -> Result<()> { + validate_shared_owner_request_support(request, operation) +} + +pub(crate) fn with_supported_shared_world_request( + request: Option<&SharedWorldOwnerSpec>, + operation: &str, + on_supported: F, +) -> Result +where + F: FnOnce() -> Result, +{ + validate_shared_owner_request_support(request, operation)?; + on_supported() +} + pub fn store_context_globally(ctx: PlatformWorldContext) { let _ = GLOBAL_CTX.set(Arc::new(ctx)); } @@ -65,6 +134,11 @@ pub fn get_context() -> Option> { GLOBAL_CTX.get().cloned() } +#[allow(dead_code)] +pub async fn ensure_persistent_session_ready_async(ctx: &PlatformWorldContext) -> Result<()> { + (ctx.ensure_persistent_session_ready_async.as_ref())().await +} + #[cfg(target_os = "macos")] pub fn detect() -> Result { use world_mac_lima::MacLimaBackend; @@ -90,6 +164,7 @@ pub fn detect() -> Result { let backend = Arc::new(MacLimaBackend::new()?); let ensure_ready_backend = backend.clone(); + let ensure_persistent_session_ready_async_backend = backend.clone(); let ensure_ready = Box::new(move || { use world_api::WorldBackend as _; let spec = bootstrap_world_spec( @@ -98,12 +173,18 @@ pub fn detect() -> Result { ); ensure_ready_backend.ensure_session(&spec).map(|_| ()) }); + let ensure_persistent_session_ready_async = Box::new(move || { + let backend = ensure_persistent_session_ready_async_backend.clone(); + Box::pin(async move { backend.ensure_persistent_session_ready_async().await }) + as PersistentSessionReadyFuture + }); Ok(PlatformWorldContext { backend, transport, socket_path: default_sock, ensure_ready, + ensure_persistent_session_ready_async, }) } @@ -115,6 +196,7 @@ pub fn detect() -> Result { let backend = Arc::new(LinuxLocalBackend::new()); let ensure_ready_backend = backend.clone(); + let ensure_persistent_session_ready_async_backend = backend.clone(); let ensure_ready = Box::new(move || { let spec = bootstrap_world_spec( settings::world_root_from_env().path, @@ -122,6 +204,21 @@ pub fn detect() -> Result { ); ensure_ready_backend.ensure_session(&spec).map(|_| ()) }); + let ensure_persistent_session_ready_async = Box::new(move || { + let backend = ensure_persistent_session_ready_async_backend.clone(); + Box::pin(async move { + tokio::task::spawn_blocking(move || { + use world_api::WorldBackend as _; + let spec = bootstrap_world_spec( + settings::world_root_from_env().path, + substrate_broker::world_fs_mode(), + ); + backend.ensure_session(&spec).map(|_| ()) + }) + .await + .context("persistent-session readiness join failure")? + }) as PersistentSessionReadyFuture + }); // Native Linux agent socket path let sock = PathBuf::from("/run/substrate.sock"); @@ -132,6 +229,7 @@ pub fn detect() -> Result { transport, socket_path: sock, ensure_ready, + ensure_persistent_session_ready_async, }) } diff --git a/crates/shell/src/execution/platform_world/windows.rs b/crates/shell/src/execution/platform_world/windows.rs index f5e545b1e..549fce925 100644 --- a/crates/shell/src/execution/platform_world/windows.rs +++ b/crates/shell/src/execution/platform_world/windows.rs @@ -4,11 +4,11 @@ use crate::execution::settings; #[cfg(test)] use crate::execution::world_env_guard; use agent_api_client::{AgentClient, Transport}; -use anyhow::Result; +use anyhow::{Context, Result}; use std::path::PathBuf; use std::sync::{Arc, OnceLock}; use substrate_broker::world_fs_mode; -use world_api::{WorldBackend, WorldSpec}; +use world_api::{SharedWorldOwnerSpec, WorldBackend, WorldSpec}; use world_windows_wsl::WindowsWslBackend; struct WindowsContext { @@ -54,10 +54,14 @@ fn socket_path_from_transport(transport: &Transport) -> PathBuf { } pub fn ensure_world_ready_with_state(no_world: bool) -> Result> { - ensure_world_ready_impl(no_world, get_backend) + ensure_world_ready_impl(no_world, None, get_backend) } -fn ensure_world_ready_impl(no_world: bool, backend_provider: F) -> Result> +fn ensure_world_ready_impl( + no_world: bool, + shared_world: Option<&SharedWorldOwnerSpec>, + backend_provider: F, +) -> Result> where F: FnOnce() -> Result>, { @@ -68,16 +72,18 @@ where #[cfg(test)] let _env_guard = world_env_guard(); - let backend = backend_provider()?; - let spec = bootstrap_world_spec(); - match backend.ensure_session(&spec) { - Ok(handle) => { - std::env::set_var("SUBSTRATE_WORLD", "enabled"); - std::env::set_var("SUBSTRATE_WORLD_ID", &handle.id); - Ok(Some(handle.id)) + super::with_supported_shared_world_request(shared_world, "windows world bootstrap", || { + let backend = backend_provider()?; + let spec = bootstrap_world_spec(); + match backend.ensure_session(&spec) { + Ok(handle) => { + std::env::set_var("SUBSTRATE_WORLD", "enabled"); + std::env::set_var("SUBSTRATE_WORLD_ID", &handle.id); + Ok(Some(handle.id)) + } + Err(_err) => Ok(None), } - Err(_err) => Ok(None), - } + }) } pub fn get_backend() -> Result> { @@ -95,16 +101,29 @@ pub fn detect() -> Result { let transport = to_world_transport(&transport_config); let socket_path = socket_path_from_transport(&transport_config); let ensure_backend = ctx.backend_trait.clone(); + let ensure_persistent_session_ready_async_backend = ctx.backend_trait.clone(); let ensure_ready = Box::new(move || { let spec = bootstrap_world_spec(); ensure_backend.ensure_session(&spec).map(|_| ()) }); + let ensure_persistent_session_ready_async = Box::new(move || { + let backend = ensure_persistent_session_ready_async_backend.clone(); + Box::pin(async move { + tokio::task::spawn_blocking(move || { + let spec = bootstrap_world_spec(); + backend.ensure_session(&spec).map(|_| ()) + }) + .await + .context("persistent-session readiness join failure")? + }) as super::PersistentSessionReadyFuture + }); Ok(PlatformWorldContext { backend, transport, socket_path, ensure_ready, + ensure_persistent_session_ready_async, }) } @@ -165,7 +184,10 @@ mod tests { impl StubBackend { fn new(id: &str, ensure_calls: Arc) -> Self { Self { - handle: WorldHandle { id: id.to_string() }, + handle: WorldHandle { + id: id.to_string(), + shared_binding: None, + }, ensure_calls, } } @@ -199,7 +221,7 @@ mod tests { let calls = Arc::new(AtomicUsize::new(0)); let backend = Arc::new(StubBackend::new("wld_test", calls.clone())); - let result = ensure_world_ready_impl(false, || Ok(backend.clone())).unwrap(); + let result = ensure_world_ready_impl(false, None, || Ok(backend.clone())).unwrap(); assert_eq!(result.as_deref(), Some("wld_test")); assert_eq!(std::env::var("SUBSTRATE_WORLD").unwrap(), "enabled"); assert_eq!(std::env::var("SUBSTRATE_WORLD_ID").unwrap(), "wld_test"); @@ -218,7 +240,7 @@ mod tests { let calls = Arc::new(AtomicUsize::new(0)); let backend = Arc::new(StubBackend::new("wld_forced", calls.clone())); - let result = ensure_world_ready_impl(false, || Ok(backend.clone())).unwrap(); + let result = ensure_world_ready_impl(false, None, || Ok(backend.clone())).unwrap(); assert_eq!(result.as_deref(), Some("wld_forced")); assert_eq!(std::env::var("SUBSTRATE_WORLD").unwrap(), "enabled"); assert_eq!(std::env::var("SUBSTRATE_WORLD_ID").unwrap(), "wld_forced"); @@ -237,7 +259,7 @@ mod tests { let calls = Arc::new(AtomicUsize::new(0)); let backend = Arc::new(StubBackend::new("wld_test", calls.clone())); - let result = ensure_world_ready_impl(true, || Ok(backend.clone())).unwrap(); + let result = ensure_world_ready_impl(true, None, || Ok(backend.clone())).unwrap(); assert!(result.is_none()); assert_eq!(calls.load(Ordering::SeqCst), 0); diff --git a/crates/shell/src/execution/policy_cmd.rs b/crates/shell/src/execution/policy_cmd.rs index f3d18893f..183798e29 100644 --- a/crates/shell/src/execution/policy_cmd.rs +++ b/crates/shell/src/execution/policy_cmd.rs @@ -398,7 +398,7 @@ impl Serialize for SortedMetadata<'_> { S: serde::Serializer, { let mut entries: Vec<_> = self.0.iter().collect(); - entries.sort_by(|(a_key, _), (b_key, _)| a_key.cmp(b_key)); + entries.sort_by_key(|(key, _)| *key); let mut map = serializer.serialize_map(Some(entries.len()))?; for (key, value) in entries { diff --git a/crates/shell/src/execution/policy_snapshot.rs b/crates/shell/src/execution/policy_snapshot.rs index 967cc48e7..ffe9ea05a 100644 --- a/crates/shell/src/execution/policy_snapshot.rs +++ b/crates/shell/src/execution/policy_snapshot.rs @@ -14,7 +14,7 @@ use std::path::{Path, PathBuf}; use std::sync::{Mutex, OnceLock}; use std::time::SystemTime; use substrate_common::WorldFsMode; -use world_api::{ResourceLimits, WorldSpec}; +use world_api::{ResourceLimits, WorldReuseMode, WorldSpec}; const WORLD_FS_ENFORCEMENT_PLAN_B64_ENV: &str = "SUBSTRATE_WORLD_FS_ENFORCEMENT_PLAN_B64"; @@ -171,6 +171,7 @@ pub(crate) fn resolve_world_network_policy_for_snapshot( pub(crate) fn bootstrap_world_spec(project_dir: PathBuf, fs_mode: WorldFsMode) -> WorldSpec { WorldSpec { reuse_session: true, + reuse_mode: WorldReuseMode::GenericCompatible, isolate_network: false, limits: ResourceLimits::default(), enable_preload: false, @@ -198,6 +199,7 @@ pub(crate) fn world_spec_for_network_policy( ) -> WorldSpec { WorldSpec { reuse_session: true, + reuse_mode: WorldReuseMode::GenericCompatible, isolate_network: network_policy.isolate_network, limits: ResourceLimits::default(), enable_preload: false, diff --git a/crates/shell/src/execution/repl_persistent_session.rs b/crates/shell/src/execution/repl_persistent_session.rs index 2bef46db6..e7200ad07 100644 --- a/crates/shell/src/execution/repl_persistent_session.rs +++ b/crates/shell/src/execution/repl_persistent_session.rs @@ -8,6 +8,10 @@ use base64::engine::general_purpose::STANDARD as BASE64; use base64::Engine; use serde::Deserialize; +use world_api::{ + SharedWorldBindingSnapshot, SharedWorldBindingState, SharedWorldOwnerAction, + SharedWorldOwnerSpec, +}; #[derive(Debug, Clone)] pub struct PersistentSessionProtocolError { @@ -41,6 +45,7 @@ pub struct PersistentSessionClientCore { phase: Phase, shutdown_initiated: bool, latched_fatal: Option, + requested_shared_world: Option, } #[derive(Debug, Default)] @@ -64,6 +69,8 @@ enum ServerFrame { world_id: String, cwd: String, protocol_version: u32, + #[serde(default)] + shared_world: Option, }, Stdout { data_b64: String, @@ -94,9 +101,17 @@ impl PersistentSessionClientCore { } pub fn note_start_session_sent(&mut self) { + self.note_start_session_sent_with_shared_world(None); + } + + pub fn note_start_session_sent_with_shared_world( + &mut self, + requested_shared_world: Option, + ) { if self.latched_fatal.is_some() { return; } + self.requested_shared_world = requested_shared_world; self.phase = Phase::WaitingReady; } @@ -156,6 +171,7 @@ impl PersistentSessionClientCore { world_id, cwd, protocol_version, + shared_world, } => { if !matches!(self.phase, Phase::WaitingReady) { return Err(self.latch_fatal( @@ -182,6 +198,13 @@ impl PersistentSessionClientCore { "protocol error: ready.cwd must be an absolute world path: {cwd}" ))); } + validate_shared_world_echo( + self.requested_shared_world.as_ref(), + shared_world.as_ref(), + "ready.shared_world", + Some(world_id.as_str()), + ) + .map_err(|message| self.latch_fatal(format!("protocol error: {message}")))?; self.phase = Phase::Ready; Ok(()) } @@ -282,6 +305,75 @@ fn redact_token(token: &str) -> String { format!("{}…{}", &token[..4], &token[token.len() - 4..]) } +pub fn validate_shared_world_echo( + requested: Option<&SharedWorldOwnerSpec>, + echoed: Option<&SharedWorldBindingSnapshot>, + context: &str, + expected_world_id: Option<&str>, +) -> Result, String> { + let Some(binding) = echoed else { + if let Some(requested) = requested { + return Err(format!( + "{context} missing authoritative proof for orchestration session {}", + requested.orchestration_session_id + )); + } + return Ok(None); + }; + + let Some(requested) = requested else { + return Err(format!( + "{context} must be absent when no explicit shared-owner request was sent" + )); + }; + + if binding.orchestration_session_id.trim().is_empty() { + return Err(format!( + "{context}.orchestration_session_id must be a non-empty string" + )); + } + if binding.orchestration_session_id != requested.orchestration_session_id { + return Err(format!( + "{context}.orchestration_session_id mismatch (expected {}, got {})", + requested.orchestration_session_id, binding.orchestration_session_id + )); + } + if binding.world_id.trim().is_empty() { + return Err(format!("{context}.world_id must be a non-empty string")); + } + if let Some(expected_world_id) = expected_world_id { + if binding.world_id != expected_world_id { + return Err(format!( + "{context}.world_id mismatch (expected {}, got {})", + expected_world_id, binding.world_id + )); + } + } + if binding.binding_state != SharedWorldBindingState::Active { + return Err(format!( + "{context}.binding_state must be active, got {:?}", + binding.binding_state + )); + } + + match requested.action { + SharedWorldOwnerAction::AttachOrCreate => {} + SharedWorldOwnerAction::ReplaceExpectedGeneration { + expected_generation, + .. + } => { + if binding.world_generation <= expected_generation { + return Err(format!( + "{context}.world_generation must advance past expected_generation={} (got {})", + expected_generation, binding.world_generation + )); + } + } + } + + Ok(Some(binding.clone())) +} + #[cfg(test)] mod tests { use super::*; diff --git a/crates/shell/src/execution/routing.rs b/crates/shell/src/execution/routing.rs index 0f90eb624..09b414024 100644 --- a/crates/shell/src/execution/routing.rs +++ b/crates/shell/src/execution/routing.rs @@ -37,9 +37,10 @@ pub(crate) use self::world::initialize_world; pub(crate) use dispatch::world_persistent_session::{ ReplPersistentSessionClient, ReplSessionStartParams, ReplStdinMode, }; +#[allow(unused_imports)] pub(crate) use dispatch::{ - build_agent_client_and_pending_diff_request, build_agent_client_and_request, execute_command, - needs_pty, + build_agent_client_and_member_dispatch_request, build_agent_client_and_pending_diff_request, + build_agent_client_and_request, execute_command, needs_pty, MemberDispatchTransportRequest, }; #[cfg(target_os = "linux")] use nix::sys::termios::{ diff --git a/crates/shell/src/execution/routing/dispatch/exec.rs b/crates/shell/src/execution/routing/dispatch/exec.rs index f83557fcb..9cf3467f7 100644 --- a/crates/shell/src/execution/routing/dispatch/exec.rs +++ b/crates/shell/src/execution/routing/dispatch/exec.rs @@ -11,6 +11,7 @@ use super::world_ops::WorldFsStrategyUnavailableError; use super::world_ops::{ collect_world_telemetry, emit_stream_chunk, stream_non_pty_via_agent, AgentStreamOutcome, }; +use crate::execution::agent_events::ShellCommandEventContext; use crate::execution::config_model; use crate::execution::config_model::CliConfigOverrides; use crate::execution::config_model::PolicyMode; @@ -170,6 +171,7 @@ pub(crate) fn execute_command( command: &str, cmd_id: &str, running_child_pid: Arc, + command_event_context: Option, ) -> Result { let trimmed = command.trim(); @@ -984,6 +986,7 @@ pub(crate) fn execute_command( &agent_command, span_id_for_cmd_events.as_deref(), Some(cmd_id), + command_event_context.clone(), ) { Ok(outcome) => { if let Some(active_span) = span.as_mut() { @@ -1065,6 +1068,7 @@ pub(crate) fn execute_command( &agent_command, span_id_for_cmd_events.as_deref(), Some(cmd_id), + command_event_context.clone(), ) { Ok(outcome) => { if let Some(active_span) = span.as_mut() { @@ -1177,8 +1181,12 @@ pub(crate) fn execute_command( socket_activation::socket_activation_report().is_socket_activated(), ), }; - match stream_non_pty_via_agent(trimmed, span_id_for_cmd_events.as_deref(), Some(cmd_id)) - { + match stream_non_pty_via_agent( + trimmed, + span_id_for_cmd_events.as_deref(), + Some(cmd_id), + command_event_context.clone(), + ) { Ok(outcome) => { if let Some(active_span) = span.as_mut() { active_span.set_execution_origin(ExecutionOrigin::World); @@ -1381,7 +1389,18 @@ pub(crate) fn execute_command( } // Execute external command through shell for complex commands or when no builtin matched. - let status = execute_external(config, trimmed, running_child_pid, cmd_id)?; + let mut host_stream_context = command_event_context; + if let Some(context) = host_stream_context.as_mut() { + context.span_id = context.span_id.clone().or(span_id_for_cmd_events.clone()); + } + + let status = execute_external( + config, + trimmed, + running_child_pid, + cmd_id, + host_stream_context, + )?; // Log command completion with redacted command let duration = start_time.elapsed(); @@ -1447,6 +1466,7 @@ fn execute_external( command: &str, running_child_pid: Arc, cmd_id: &str, + command_event_context: Option, ) -> Result { let shell = &config.shell_path; let mut command = command.to_string(); @@ -1544,14 +1564,22 @@ fn execute_external( .spawn() .with_context(|| format!("Failed to execute: {command}"))?; - let stdout_thread = child - .stdout - .take() - .map(|pipe| spawn_host_stream_thread(pipe, false, SHELL_AGENT_ID.to_string())); - let stderr_thread = child - .stderr - .take() - .map(|pipe| spawn_host_stream_thread(pipe, true, SHELL_AGENT_ID.to_string())); + let stdout_thread = child.stdout.take().map(|pipe| { + spawn_host_stream_thread( + pipe, + false, + SHELL_AGENT_ID.to_string(), + command_event_context.clone(), + ) + }); + let stderr_thread = child.stderr.take().map(|pipe| { + spawn_host_stream_thread( + pipe, + true, + SHELL_AGENT_ID.to_string(), + command_event_context.clone(), + ) + }); let child_pid = child.id() as i32; running_child_pid.store(child_pid, Ordering::SeqCst); @@ -1584,6 +1612,7 @@ fn spawn_host_stream_thread( mut reader: R, is_stderr: bool, agent_label: String, + command_event_context: Option, ) -> std::thread::JoinHandle> where R: std::io::Read + Send + 'static, @@ -1594,7 +1623,26 @@ where match std::io::Read::read(&mut reader, &mut buf) { Ok(0) => break, Ok(n) => { - emit_stream_chunk(&agent_label, "unknown", None, &buf[..n], is_stderr); + let (orchestration_session_id, run_id, span_id) = + match command_event_context.as_ref() { + Some(context) => match context.run_id.as_deref() { + Some(run_id) => ( + Some(context.emission.orchestration_session_id.as_str()), + run_id, + context.span_id.as_deref(), + ), + None => (None, "", None), + }, + None => (None, "", None), + }; + emit_stream_chunk( + &agent_label, + orchestration_session_id, + run_id, + span_id, + &buf[..n], + is_stderr, + ); } Err(e) if e.kind() == std::io::ErrorKind::Interrupted => continue, Err(e) => return Err(anyhow::anyhow!("pipe read failed: {}", e)), diff --git a/crates/shell/src/execution/routing/dispatch/prelude.rs b/crates/shell/src/execution/routing/dispatch/prelude.rs index d201d489e..0e9126e39 100644 --- a/crates/shell/src/execution/routing/dispatch/prelude.rs +++ b/crates/shell/src/execution/routing/dispatch/prelude.rs @@ -13,8 +13,9 @@ pub(crate) use super::shim_ops::wrap_with_anchor_guard; pub(crate) use super::world_ops::init_linux_world; #[allow(unused_imports)] pub(crate) use super::world_ops::{ - build_agent_client_and_pending_diff_request, build_agent_client_and_request, - consume_agent_stream_buffer, stream_non_pty_via_agent, AgentStreamOutcome, + build_agent_client_and_member_dispatch_request, build_agent_client_and_pending_diff_request, + build_agent_client_and_request, consume_agent_stream_buffer, stream_non_pty_via_agent, + AgentStreamOutcome, MemberDispatchTransportRequest, }; #[cfg(all(test, target_os = "linux"))] pub(crate) use super::world_ops::{init_linux_world_with_probe, LinuxWorldInit}; diff --git a/crates/shell/src/execution/routing/dispatch/tests/host_replay.rs b/crates/shell/src/execution/routing/dispatch/tests/host_replay.rs index 8f46e11aa..4f22f97ce 100644 --- a/crates/shell/src/execution/routing/dispatch/tests/host_replay.rs +++ b/crates/shell/src/execution/routing/dispatch/tests/host_replay.rs @@ -56,6 +56,7 @@ fn async_repl_host_commands_record_replay_context() { "printf async-repl-host > repl-host.log", "async-repl-host", Arc::new(AtomicI32::new(0)), + None, ) .expect("execute async repl command"); diff --git a/crates/shell/src/execution/routing/dispatch/tests/repl_persistent_session_client_fail_closed.rs b/crates/shell/src/execution/routing/dispatch/tests/repl_persistent_session_client_fail_closed.rs index 48729c2b6..39ebca81e 100644 --- a/crates/shell/src/execution/routing/dispatch/tests/repl_persistent_session_client_fail_closed.rs +++ b/crates/shell/src/execution/routing/dispatch/tests/repl_persistent_session_client_fail_closed.rs @@ -6,6 +6,10 @@ //! - docs/project_management/next/world-first-repl-persistent-pty/requirements_traceability.md use serde_json::json; +use world_api::{ + SharedWorldBindingSnapshot, SharedWorldBindingState, SharedWorldOwnerAction, + SharedWorldOwnerSpec, +}; use crate::execution::repl_persistent_session::{ PersistentSessionClientCore, PersistentSessionProtocolError, @@ -21,6 +25,17 @@ fn ready_frame(protocol_version: u32) -> serde_json::Value { }) } +fn ready_frame_with_shared_world( + protocol_version: u32, + shared_world: Option, +) -> serde_json::Value { + let mut frame = ready_frame(protocol_version); + if let Some(shared_world) = shared_world { + frame["shared_world"] = serde_json::to_value(shared_world).expect("serialize shared_world"); + } + frame +} + fn exit_frame(code: i32) -> serde_json::Value { json!({ "type": "exit", @@ -179,3 +194,67 @@ fn ready_empty_world_id_is_fatal_protocol_error() { .expect_err("ready.world_id must be non-empty"); assert!(err.is_fatal(), "empty world_id must be fatal: {err:#}"); } + +#[test] +fn ready_missing_shared_world_for_explicit_owner_request_is_fatal() { + let mut client = PersistentSessionClientCore::new(); + client.note_start_session_sent_with_shared_world(Some(SharedWorldOwnerSpec { + orchestration_session_id: "orch-test".to_string(), + action: SharedWorldOwnerAction::AttachOrCreate, + })); + + let err = client + .on_server_frame(ready_frame_with_shared_world(1, None)) + .expect_err("explicit owner-bound start_session must require echoed proof"); + assert!( + err.is_fatal(), + "missing shared_world must be fatal: {err:#}" + ); +} + +#[test] +fn ready_unexpected_shared_world_for_generic_request_is_fatal() { + let mut client = PersistentSessionClientCore::new(); + client.note_start_session_sent(); + + let err = client + .on_server_frame(ready_frame_with_shared_world( + 1, + Some(SharedWorldBindingSnapshot { + orchestration_session_id: "orch-test".to_string(), + world_id: "wld_test".to_string(), + world_generation: 0, + binding_state: SharedWorldBindingState::Active, + }), + )) + .expect_err("generic start_session must reject unexpected shared_world proof"); + assert!( + err.is_fatal(), + "unexpected shared_world must be fatal: {err:#}" + ); +} + +#[test] +fn ready_replace_expected_generation_requires_advanced_generation() { + let mut client = PersistentSessionClientCore::new(); + client.note_start_session_sent_with_shared_world(Some(SharedWorldOwnerSpec { + orchestration_session_id: "orch-test".to_string(), + action: SharedWorldOwnerAction::ReplaceExpectedGeneration { + expected_generation: 4, + reason: "restart".to_string(), + }, + })); + + let err = client + .on_server_frame(ready_frame_with_shared_world( + 1, + Some(SharedWorldBindingSnapshot { + orchestration_session_id: "orch-test".to_string(), + world_id: "wld_test".to_string(), + world_generation: 4, + binding_state: SharedWorldBindingState::Active, + }), + )) + .expect_err("replace requests must reject stale echoed generations"); + assert!(err.is_fatal(), "stale generation must be fatal: {err:#}"); +} diff --git a/crates/shell/src/execution/routing/dispatch/tests/telemetry.rs b/crates/shell/src/execution/routing/dispatch/tests/telemetry.rs index 2e3d78082..38475f480 100644 --- a/crates/shell/src/execution/routing/dispatch/tests/telemetry.rs +++ b/crates/shell/src/execution/routing/dispatch/tests/telemetry.rs @@ -7,13 +7,13 @@ use agent_api_types::{ use base64::engine::general_purpose::STANDARD as BASE64; use base64::Engine; use serde_json::Value as JsonValue; -use substrate_common::agent_events::AgentEventKind; use substrate_common::FsDiff; use tokio::runtime::Runtime; // Telemetry stream handling #[test] -fn consume_agent_stream_buffer_emits_agent_events() { +#[serial_test::serial] +fn consume_agent_stream_buffer_without_context_suppresses_agent_events() { let _guard = agent_events::acquire_event_test_guard(); let rt = Runtime::new().expect("runtime"); rt.block_on(async { @@ -55,15 +55,10 @@ fn consume_agent_stream_buffer_emits_agent_events() { ) .expect("consume stream"); - let stdout_event = rx.recv().await.expect("stdout event"); - assert_eq!(stdout_event.kind, AgentEventKind::PtyData); - assert_eq!(stdout_event.data["chunk"], "hello"); - assert_eq!(stdout_event.data["stream"], "stdout"); - - let stderr_event = rx.recv().await.expect("stderr event"); - assert_eq!(stderr_event.kind, AgentEventKind::PtyData); - assert_eq!(stderr_event.data["chunk"], "oops"); - assert_eq!(stderr_event.data["stream"], "stderr"); + assert!( + rx.try_recv().is_err(), + "no-context stream parsing must not synthesize orchestration-scoped agent events" + ); assert_eq!(exit_code, Some(0)); assert_eq!(scopes_used, vec!["scope:a".to_string()]); diff --git a/crates/shell/src/execution/routing/dispatch/world_ops.rs b/crates/shell/src/execution/routing/dispatch/world_ops.rs index cb11f31d6..be5710586 100644 --- a/crates/shell/src/execution/routing/dispatch/world_ops.rs +++ b/crates/shell/src/execution/routing/dispatch/world_ops.rs @@ -2,6 +2,7 @@ use super::shim_ops::build_world_env_map_for_cwd; use crate::execution::agent_events::publish_agent_event; +use crate::execution::agent_events::ShellCommandEventContext; #[cfg(target_os = "windows")] use crate::execution::policy_snapshot::world_spec_for_network_policy; use crate::execution::policy_snapshot::{ @@ -19,7 +20,10 @@ use crate::execution::{policy_snapshot::bootstrap_world_spec, socket_activation} use agent_api_client::AgentClient; #[cfg(not(target_os = "windows"))] use agent_api_types::ExecuteCancelRequestV1; -use agent_api_types::{ExecuteRequest, ExecuteStreamFrame, ProcessTelemetry, WorldFsMode}; +use agent_api_types::{ + ExecuteRequest, ExecuteStreamFrame, MemberDispatchRequestV1, MemberRuntimeBackendKindV1, + ProcessTelemetry, ResolvedMemberRuntimeDescriptorV1, WorldFsMode, +}; use base64::engine::general_purpose::STANDARD as BASE64; use base64::Engine; use std::env; @@ -167,6 +171,7 @@ pub(super) fn collect_world_telemetry( let backend = LinuxLocalBackend::new(); let handle = world_api::WorldHandle { id: world_id.clone(), + shared_binding: None, }; // Try to get filesystem diff @@ -193,6 +198,77 @@ pub(crate) struct PtyWorldOutcome { pub(crate) process_telemetry: ProcessTelemetry, } +#[allow(dead_code)] +#[derive(Debug, Clone)] +pub(crate) struct MemberDispatchTransportRequest { + pub orchestration_session_id: String, + pub participant_id: String, + pub orchestrator_participant_id: String, + pub parent_participant_id: Option, + pub resumed_from_participant_id: Option, + pub backend_id: String, + pub protocol: String, + pub run_id: String, + pub world_id: String, + pub world_generation: u64, + pub initial_prompt: Option, + pub backend_kind: MemberRuntimeBackendKindV1, + pub binary_path: String, +} + +fn build_execute_request(input: ExecuteRequestInput) -> ExecuteRequest { + ExecuteRequest { + profile: input.profile, + cmd: input.cmd, + cwd: Some(input.cwd), + env: Some(input.env_map), + pty: false, + agent_id: input.agent_id, + budget: None, + policy_snapshot: input.policy_snapshot, + shared_world: None, + world_network: Some(input.world_network), + world_fs_mode: Some(input.world_fs_mode), + member_dispatch: input.member_dispatch, + } +} + +struct ExecuteRequestInput { + profile: Option, + cmd: String, + cwd: String, + env_map: std::collections::HashMap, + agent_id: String, + policy_snapshot: agent_api_types::PolicySnapshotV3, + world_network: agent_api_types::WorldNetworkRoutingV1, + world_fs_mode: WorldFsMode, + member_dispatch: Option, +} + +#[allow(dead_code)] +fn build_member_dispatch_payload( + request: &MemberDispatchTransportRequest, +) -> MemberDispatchRequestV1 { + MemberDispatchRequestV1 { + schema_version: 1, + orchestration_session_id: request.orchestration_session_id.clone(), + participant_id: request.participant_id.clone(), + orchestrator_participant_id: request.orchestrator_participant_id.clone(), + parent_participant_id: request.parent_participant_id.clone(), + resumed_from_participant_id: request.resumed_from_participant_id.clone(), + backend_id: request.backend_id.clone(), + protocol: request.protocol.clone(), + run_id: request.run_id.clone(), + world_id: request.world_id.clone(), + world_generation: request.world_generation, + initial_prompt: request.initial_prompt.clone(), + resolved_runtime: ResolvedMemberRuntimeDescriptorV1 { + backend_kind: request.backend_kind, + binary_path: request.binary_path.clone(), + }, + } +} + #[cfg(target_os = "linux")] pub(super) fn execute_world_pty_over_ws( cmd: &str, @@ -998,6 +1074,17 @@ pub(crate) fn build_agent_client_and_request_with_trace_metadata( build_agent_client_and_request_impl(cmd, parent_span_id, parent_cmd_id) } +#[allow(dead_code)] +pub(crate) fn build_agent_client_and_member_dispatch_request( + request: &MemberDispatchTransportRequest, +) -> anyhow::Result<( + agent_api_client::AgentClient, + agent_api_types::ExecuteRequest, + String, +)> { + build_agent_client_and_member_dispatch_request_impl(request) +} + pub(crate) fn build_agent_client_and_pending_diff_request() -> anyhow::Result<( agent_api_client::AgentClient, agent_api_types::PendingDiffRequestV1, @@ -1021,6 +1108,21 @@ fn current_world_request_profile() -> Option { .filter(|value| !RESERVED_WORLD_REQUEST_PROFILES.contains(&value.as_str())) } +#[cfg(target_os = "windows")] +fn validate_execute_response_shared_world( + requested: Option<&agent_api_types::SharedWorldOwnerSpec>, + response: &agent_api_types::ExecuteResponse, +) -> anyhow::Result<()> { + crate::execution::repl_persistent_session::validate_shared_world_echo( + requested, + response.shared_world.as_ref(), + "execute_response.shared_world", + None, + ) + .map(|_| ()) + .map_err(|message| anyhow::anyhow!("protocol error: {message}")) +} + #[cfg(target_os = "linux")] fn build_agent_client_and_request_impl( cmd: &str, @@ -1056,18 +1158,62 @@ fn build_agent_client_and_request_impl( preserve_world_project_dir_override(&mut env_map); inject_process_trace_env(&mut env_map, parent_span_id, parent_cmd_id); - let request = ExecuteRequest { + let request = build_execute_request(ExecuteRequestInput { profile: current_world_request_profile(), cmd: cmd.to_string(), - cwd: Some(cwd), - env: Some(env_map), - pty: false, + cwd, + env_map, agent_id: agent_id.clone(), - budget: None, policy_snapshot, - world_network: Some(world_network), - world_fs_mode: Some(current_world_fs_mode()), - }; + world_network, + world_fs_mode: current_world_fs_mode(), + member_dispatch: None, + }); + + Ok((client, request, agent_id)) +} + +#[allow(dead_code)] +#[cfg(target_os = "linux")] +fn build_agent_client_and_member_dispatch_request_impl( + dispatch: &MemberDispatchTransportRequest, +) -> anyhow::Result<( + agent_api_client::AgentClient, + agent_api_types::ExecuteRequest, + String, +)> { + let socket_path = std::env::var_os("SUBSTRATE_WORLD_SOCKET") + .map(std::path::PathBuf::from) + .unwrap_or_else(|| std::path::PathBuf::from("/run/substrate.sock")); + + let client = AgentClient::unix_socket(&socket_path)?; + let cwd_path = std::env::current_dir().unwrap_or_else(|_| std::path::PathBuf::from(".")); + let cwd = cwd_path.display().to_string(); + let agent_id = std::env::var("SUBSTRATE_AGENT_ID").unwrap_or_else(|_| "human".to_string()); + let network_policy = resolve_world_network_policy_for_cwd(&cwd_path)?; + let world_network = request_world_network_routing(&network_policy); + let policy_snapshot = network_policy.snapshot; + let (mut env_map, inherit_from_host) = build_world_env_map_for_cwd(&cwd_path)?; + if inherit_from_host { + eprintln!("substrate: warning: world env is forwarding selected host env vars (world.env.inherit_from_host=true)"); + } + crate::execution::policy_snapshot::inject_world_fs_enforcement_plan_env( + &policy_snapshot, + &mut env_map, + )?; + ensure_world_deps_bin_on_path(&mut env_map); + preserve_world_project_dir_override(&mut env_map); + let request = build_execute_request(ExecuteRequestInput { + profile: current_world_request_profile(), + cmd: String::new(), + cwd, + env_map, + agent_id: agent_id.clone(), + policy_snapshot, + world_network, + world_fs_mode: current_world_fs_mode(), + member_dispatch: Some(build_member_dispatch_payload(dispatch)), + }); Ok((client, request, agent_id)) } @@ -1152,18 +1298,17 @@ fn build_agent_client_and_request_impl( )?; inject_process_trace_env(&mut env_map, parent_span_id, parent_cmd_id); - let request = ExecuteRequest { + let request = build_execute_request(ExecuteRequestInput { profile: current_world_request_profile(), cmd: cmd.to_string(), - cwd: Some(cwd), - env: Some(env_map), - pty: false, + cwd, + env_map, agent_id: agent_id.clone(), - budget: None, policy_snapshot, - world_network: Some(world_network), - world_fs_mode: Some(current_world_fs_mode()), - }; + world_network, + world_fs_mode: current_world_fs_mode(), + member_dispatch: None, + }); return Ok((client, request, agent_id)); } @@ -1205,18 +1350,108 @@ fn build_agent_client_and_request_impl( )?; inject_process_trace_env(&mut env_map, parent_span_id, parent_cmd_id); - let request = ExecuteRequest { + let request = build_execute_request(ExecuteRequestInput { profile: current_world_request_profile(), cmd: cmd.to_string(), - cwd: Some(cwd), - env: Some(env_map), - pty: false, + cwd, + env_map, agent_id: agent_id.clone(), - budget: None, policy_snapshot, - world_network: Some(world_network), - world_fs_mode: Some(current_world_fs_mode()), + world_network, + world_fs_mode: current_world_fs_mode(), + member_dispatch: None, + }); + + Ok((client, request, agent_id)) +} + +#[allow(dead_code)] +#[cfg(target_os = "macos")] +fn build_agent_client_and_member_dispatch_request_impl( + dispatch: &MemberDispatchTransportRequest, +) -> anyhow::Result<( + agent_api_client::AgentClient, + agent_api_types::ExecuteRequest, + String, +)> { + if let Some(socket_path) = std::env::var_os("SUBSTRATE_WORLD_SOCKET") { + let socket_path = std::path::PathBuf::from(socket_path); + let client = AgentClient::unix_socket(&socket_path)?; + let cwd_path = std::env::current_dir().unwrap_or_else(|_| std::path::PathBuf::from(".")); + let cwd = cwd_path.display().to_string(); + let (mut env_map, inherit_from_host) = build_world_env_map_for_cwd(&cwd_path)?; + if inherit_from_host { + eprintln!("substrate: warning: world env is forwarding selected host env vars (world.env.inherit_from_host=true)"); + } + normalize_env_for_linux_guest(&mut env_map); + ensure_world_deps_bin_on_path(&mut env_map); + let agent_id = std::env::var("SUBSTRATE_AGENT_ID").unwrap_or_else(|_| "human".to_string()); + let network_policy = resolve_world_network_policy_for_cwd(&cwd_path)?; + let world_network = request_world_network_routing(&network_policy); + let policy_snapshot = network_policy.snapshot; + crate::execution::policy_snapshot::inject_world_fs_enforcement_plan_env( + &policy_snapshot, + &mut env_map, + )?; + let request = build_execute_request(ExecuteRequestInput { + profile: current_world_request_profile(), + cmd: String::new(), + cwd, + env_map, + agent_id: agent_id.clone(), + policy_snapshot, + world_network, + world_fs_mode: current_world_fs_mode(), + member_dispatch: Some(build_member_dispatch_payload(dispatch)), + }); + + return Ok((client, request, agent_id)); + } + + let ctx = match pw::get_context() { + Some(ctx) => ctx, + None => { + let detected = + pw::detect().map_err(|e| anyhow::anyhow!("platform world detect failed: {e:#}"))?; + pw::store_context_globally(detected); + pw::get_context().ok_or_else(|| anyhow::anyhow!("no platform world context"))? + } }; + (ctx.ensure_ready.as_ref())()?; + + let client = match &ctx.transport { + pw::WorldTransport::Unix(path) => AgentClient::unix_socket(path), + pw::WorldTransport::Tcp { host, port } => AgentClient::tcp(host, *port), + pw::WorldTransport::Vsock { port } => AgentClient::tcp("127.0.0.1", *port), + }?; + + let cwd_path = std::env::current_dir().unwrap_or_else(|_| std::path::PathBuf::from(".")); + let cwd = cwd_path.display().to_string(); + let (mut env_map, inherit_from_host) = build_world_env_map_for_cwd(&cwd_path)?; + if inherit_from_host { + eprintln!("substrate: warning: world env is forwarding selected host env vars (world.env.inherit_from_host=true)"); + } + normalize_env_for_linux_guest(&mut env_map); + ensure_world_deps_bin_on_path(&mut env_map); + let agent_id = std::env::var("SUBSTRATE_AGENT_ID").unwrap_or_else(|_| "human".to_string()); + let network_policy = resolve_world_network_policy_for_cwd(&cwd_path)?; + let world_network = request_world_network_routing(&network_policy); + let policy_snapshot = network_policy.snapshot; + crate::execution::policy_snapshot::inject_world_fs_enforcement_plan_env( + &policy_snapshot, + &mut env_map, + )?; + let request = build_execute_request(ExecuteRequestInput { + profile: current_world_request_profile(), + cmd: String::new(), + cwd, + env_map, + agent_id: agent_id.clone(), + policy_snapshot, + world_network, + world_fs_mode: current_world_fs_mode(), + member_dispatch: Some(build_member_dispatch_payload(dispatch)), + }); Ok((client, request, agent_id)) } @@ -1354,18 +1589,82 @@ fn build_agent_client_and_request_impl( )?; inject_process_trace_env(&mut env_map, parent_span_id, parent_cmd_id); - let request = ExecuteRequest { + let request = build_execute_request(ExecuteRequestInput { profile, cmd: cmd.to_string(), - cwd: Some(cwd), - env: Some(env_map), - pty: false, + cwd, + env_map, agent_id: agent_id.clone(), - budget: None, policy_snapshot, - world_network: Some(world_network), - world_fs_mode: Some(current_world_fs_mode()), - }; + world_network, + world_fs_mode: current_world_fs_mode(), + member_dispatch: None, + }); + + Ok((client, request, agent_id)) +} + +#[allow(dead_code)] +#[cfg(target_os = "windows")] +fn build_agent_client_and_member_dispatch_request_impl( + dispatch: &MemberDispatchTransportRequest, +) -> anyhow::Result<( + agent_api_client::AgentClient, + agent_api_types::ExecuteRequest, + String, +)> { + use crate::execution::platform_world::windows; + let backend = windows::get_backend()?; + #[cfg(test)] + let _env_guard = world_env_guard(); + + let client = windows::build_agent_client()?; + let cwd = windows::current_dir_wsl()?; + let host_cwd = std::env::current_dir().unwrap_or_else(|_| std::path::PathBuf::from(".")); + let network_policy = resolve_world_network_policy_for_cwd(&host_cwd)?; + let spec = world_spec_for_network_policy( + crate::execution::settings::world_root_from_env().path, + world_fs_mode(), + &network_policy, + ); + let handle = backend.ensure_session(&spec)?; + + std::env::set_var("SUBSTRATE_WORLD", "enabled"); + std::env::set_var("SUBSTRATE_WORLD_ID", &handle.id); + + let profile = current_world_request_profile(); + let (mut env_map, inherit_from_host) = build_world_env_map_for_cwd(&host_cwd)?; + if inherit_from_host { + eprintln!("substrate: warning: world env is forwarding selected host env vars (world.env.inherit_from_host=true)"); + } + normalize_env_for_linux_guest(&mut env_map); + if profile.as_deref() == Some("world-deps-provision") { + env_map.retain(|k, _| { + k == "PATH" + || k == "HOME" + || k.starts_with("SUBSTRATE_") + || k.starts_with("WORLD_") + || k.starts_with("SHIM_") + }); + } + let agent_id = std::env::var("SUBSTRATE_AGENT_ID").unwrap_or_else(|_| "human".to_string()); + let world_network = request_world_network_routing(&network_policy); + let policy_snapshot = network_policy.snapshot; + crate::execution::policy_snapshot::inject_world_fs_enforcement_plan_env( + &policy_snapshot, + &mut env_map, + )?; + let request = build_execute_request(ExecuteRequestInput { + profile, + cmd: String::new(), + cwd, + env_map, + agent_id: agent_id.clone(), + policy_snapshot, + world_network, + world_fs_mode: current_world_fs_mode(), + member_dispatch: Some(build_member_dispatch_payload(dispatch)), + }); Ok((client, request, agent_id)) } @@ -1417,11 +1716,10 @@ pub(crate) fn stream_non_pty_via_agent( command: &str, parent_span_id: Option<&str>, parent_cmd_id: Option<&str>, + command_event_context: Option, ) -> anyhow::Result { let (client, request, agent_id) = build_agent_client_and_request_with_trace_metadata(command, parent_span_id, parent_cmd_id)?; - let run_id = parent_cmd_id.unwrap_or("unknown").to_string(); - let parent_span_id = parent_span_id.map(|s| s.to_string()); let host_visible = request.policy_snapshot.world_fs.host_visible; let empty_env: std::collections::HashMap = std::collections::HashMap::new(); @@ -1435,10 +1733,10 @@ pub(crate) fn stream_non_pty_via_agent( substrate_common::world_exec_guard::check_command(&request.cmd, cwd, env_map, host_visible) { let message = substrate_common::world_exec_guard::deny_message(&deny); - emit_stream_chunk( + emit_stream_chunk_with_context( &agent_id, - &run_id, - parent_span_id.as_deref(), + command_event_context.as_ref(), + None, message.as_bytes(), true, ); @@ -1466,6 +1764,7 @@ pub(crate) fn stream_non_pty_via_agent( let timeout = parse_timeout_ms("SUBSTRATE_WSL_AGENT_EXEC_TIMEOUT_MS") .unwrap_or_else(|| std::time::Duration::from_secs(120)); + let requested_shared_world = request.shared_world.clone(); let response = tokio::time::timeout(timeout, async { client @@ -1481,14 +1780,27 @@ pub(crate) fn stream_non_pty_via_agent( client.transport().description() ) })??; + validate_execute_response_shared_world(requested_shared_world.as_ref(), &response)?; let stdout = BASE64 .decode(response.stdout_b64.as_bytes()) .unwrap_or_else(|_| response.stdout_b64.clone().into_bytes()); let stderr = BASE64 .decode(response.stderr_b64.as_bytes()) .unwrap_or_else(|_| response.stderr_b64.clone().into_bytes()); - emit_stream_chunk(&agent_id, &run_id, parent_span_id.as_deref(), &stdout, false); - emit_stream_chunk(&agent_id, &run_id, parent_span_id.as_deref(), &stderr, true); + emit_stream_chunk_with_context( + &agent_id, + command_event_context.as_ref(), + None, + &stdout, + false, + ); + emit_stream_chunk_with_context( + &agent_id, + command_event_context.as_ref(), + None, + &stderr, + true, + ); Ok(AgentStreamOutcome { exit_code: response.exit, @@ -1535,6 +1847,7 @@ pub(crate) fn stream_non_pty_via_agent( let result = process_agent_stream( response.into_body(), agent_id, + command_event_context, &mut sigint_rx, |span_id, sig| async { client @@ -1554,19 +1867,28 @@ pub(crate) fn stream_non_pty_via_agent( async fn process_agent_stream( body: hyper::body::Incoming, agent_label: String, + command_event_context: Option, sigint_rx: &mut tokio::sync::mpsc::UnboundedReceiver<()>, cancel: impl FnMut(String, String) -> Fut, ) -> anyhow::Result where Fut: std::future::Future>, { - process_agent_stream_body(body, agent_label, sigint_rx, cancel).await + process_agent_stream_body( + body, + agent_label, + command_event_context.as_ref(), + sigint_rx, + cancel, + ) + .await } #[cfg(not(target_os = "windows"))] async fn process_agent_stream_body( body: B, agent_label: String, + command_event_context: Option<&ShellCommandEventContext>, sigint_rx: &mut tokio::sync::mpsc::UnboundedReceiver<()>, mut cancel: impl FnMut(String, String) -> Fut, ) -> anyhow::Result @@ -1612,8 +1934,9 @@ where let frame = frame.map_err(|e| anyhow::anyhow!("stream frame error: {}", e))?; if let Some(data) = frame.data_ref() { buffer.extend_from_slice(data); - consume_agent_stream_buffer_with_meta( + consume_agent_stream_buffer_with_context( &agent_label, + command_event_context, &mut buffer, &mut active_span_id, &mut exit_code, @@ -1629,8 +1952,9 @@ where } if exit_code.is_none() && !buffer.is_empty() { - consume_agent_stream_buffer_with_meta( + consume_agent_stream_buffer_with_context( &agent_label, + command_event_context, &mut buffer, &mut active_span_id, &mut exit_code, @@ -1685,6 +2009,31 @@ pub(crate) fn consume_agent_stream_buffer_with_meta( fs_diff: &mut Option, fs_strategy: &mut Option, process_telemetry: &mut ProcessTelemetry, +) -> anyhow::Result<()> { + consume_agent_stream_buffer_with_context( + agent_label, + None, + buffer, + active_span_id, + exit_code, + scopes_used, + fs_diff, + fs_strategy, + process_telemetry, + ) +} + +#[allow(clippy::too_many_arguments)] +fn consume_agent_stream_buffer_with_context( + agent_label: &str, + command_event_context: Option<&ShellCommandEventContext>, + buffer: &mut Vec, + active_span_id: &mut Option, + exit_code: &mut Option, + scopes_used: &mut Vec, + fs_diff: &mut Option, + fs_strategy: &mut Option, + process_telemetry: &mut ProcessTelemetry, ) -> anyhow::Result<()> { use anyhow::Context as _; @@ -1713,10 +2062,9 @@ pub(crate) fn consume_agent_stream_buffer_with_meta( let bytes = BASE64 .decode(chunk_b64.as_bytes()) .map_err(|e| anyhow::anyhow!("invalid stdout chunk: {}", e))?; - let run_id = active_span_id.as_deref().unwrap_or("unknown"); - emit_stream_chunk( + emit_stream_chunk_with_context( agent_label, - run_id, + command_event_context, active_span_id.as_deref(), &bytes, false, @@ -1726,8 +2074,13 @@ pub(crate) fn consume_agent_stream_buffer_with_meta( let bytes = BASE64 .decode(chunk_b64.as_bytes()) .map_err(|e| anyhow::anyhow!("invalid stderr chunk: {}", e))?; - let run_id = active_span_id.as_deref().unwrap_or("unknown"); - emit_stream_chunk(agent_label, run_id, active_span_id.as_deref(), &bytes, true); + emit_stream_chunk_with_context( + agent_label, + command_event_context, + active_span_id.as_deref(), + &bytes, + true, + ); } ExecuteStreamFrame::Event { event } => { if let (Some(primary), Some(final_strategy), Some(reason)) = ( @@ -1783,8 +2136,38 @@ pub(crate) fn consume_agent_stream_buffer_with_meta( Ok(()) } +fn emit_stream_chunk_with_context( + agent_label: &str, + command_event_context: Option<&ShellCommandEventContext>, + span_id: Option<&str>, + data: &[u8], + is_stderr: bool, +) { + let (orchestration_session_id, run_id, effective_span_id) = match command_event_context { + Some(context) => match context.run_id.as_deref() { + Some(run_id) => ( + Some(context.emission.orchestration_session_id.as_str()), + run_id, + span_id.or(context.span_id.as_deref()), + ), + None => (None, "", None), + }, + None => (None, "", None), + }; + + emit_stream_chunk( + agent_label, + orchestration_session_id, + run_id, + effective_span_id, + data, + is_stderr, + ); +} + pub(super) fn emit_stream_chunk( agent_label: &str, + orchestration_session_id: Option<&str>, run_id: &str, span_id: Option<&str>, data: &[u8], @@ -1802,10 +2185,13 @@ pub(super) fn emit_stream_chunk( let _ = stdout.flush(); } + let Some(orchestration_session_id) = orchestration_session_id else { + return; + }; let text = String::from_utf8_lossy(data); let mut event = AgentEvent::stream_chunk( agent_label, - crate::execution::agent_events::orchestration_session_id(), + orchestration_session_id.to_string(), run_id.to_string(), is_stderr, text.to_string(), @@ -1817,17 +2203,28 @@ pub(super) fn emit_stream_chunk( #[cfg(all(test, any(target_os = "linux", target_os = "macos")))] mod tests { use super::{ - current_world_request_profile, ensure_world_deps_bin_on_path, - extract_process_telemetry_from_ws_exit, preserve_world_project_dir_override, - process_agent_stream_body, WORLD_PROJECT_DIR_OVERRIDE_ENV, + build_execute_request, build_member_dispatch_payload, current_world_request_profile, + emit_stream_chunk, ensure_world_deps_bin_on_path, extract_process_telemetry_from_ws_exit, + preserve_world_project_dir_override, process_agent_stream_body, ExecuteRequestInput, + MemberDispatchTransportRequest, BASE64, WORLD_PROJECT_DIR_OVERRIDE_ENV, + }; + use crate::execution::agent_events::{ + acquire_event_test_guard, clear_agent_event_sender, init_event_channel, + ShellCommandEventContext, ShellEventEmissionContext, }; - use agent_api_types::ExecuteStreamFrame; + use agent_api_types::{ + ExecuteStreamFrame, MemberRuntimeBackendKindV1, PolicySnapshotV3, + PolicySnapshotWorldFsFailClosedV3, PolicySnapshotWorldFsV3, PolicySnapshotWorldFsWriteV3, + ResolvedMemberRuntimeDescriptorV1, WorldFsMode, WorldNetworkRoutingV1, + }; + use base64::Engine; use futures::stream; use http_body_util::StreamBody; use serde_json::json; use std::convert::Infallible; use std::sync::{Arc, Mutex, OnceLock}; use std::time::Duration; + use substrate_common::agent_events::AgentEventKind; fn with_env_var(key: &str, value: &str, f: impl FnOnce() -> T) -> T { let _guard = test_env_lock().lock().expect("test env mutex poisoned"); @@ -1945,6 +2342,122 @@ mod tests { ); } + #[test] + fn member_dispatch_payload_preserves_frozen_lineage_and_world_fields() { + let payload = build_member_dispatch_payload(&MemberDispatchTransportRequest { + orchestration_session_id: "orch_123".to_string(), + participant_id: "ash_member_123".to_string(), + orchestrator_participant_id: "ash_orch_123".to_string(), + parent_participant_id: Some("ash_parent_123".to_string()), + resumed_from_participant_id: Some("ash_prev_123".to_string()), + backend_id: "cli:codex".to_string(), + protocol: "uaa.agent.session".to_string(), + run_id: "run_123".to_string(), + world_id: "world_123".to_string(), + world_generation: 9, + initial_prompt: Some("first turn".to_string()), + backend_kind: MemberRuntimeBackendKindV1::Codex, + binary_path: "/usr/bin/codex".to_string(), + }); + + assert_eq!(payload.schema_version, 1); + assert_eq!(payload.orchestration_session_id, "orch_123"); + assert_eq!(payload.participant_id, "ash_member_123"); + assert_eq!(payload.orchestrator_participant_id, "ash_orch_123"); + assert_eq!( + payload.parent_participant_id.as_deref(), + Some("ash_parent_123") + ); + assert_eq!( + payload.resumed_from_participant_id.as_deref(), + Some("ash_prev_123") + ); + assert_eq!(payload.backend_id, "cli:codex"); + assert_eq!(payload.protocol, "uaa.agent.session"); + assert_eq!(payload.run_id, "run_123"); + assert_eq!(payload.world_id, "world_123"); + assert_eq!(payload.world_generation, 9); + assert_eq!(payload.initial_prompt.as_deref(), Some("first turn")); + assert_eq!( + payload.resolved_runtime, + ResolvedMemberRuntimeDescriptorV1 { + backend_kind: MemberRuntimeBackendKindV1::Codex, + binary_path: "/usr/bin/codex".to_string(), + } + ); + } + + #[test] + fn build_execute_request_supports_typed_member_dispatch_shape() { + let request = build_execute_request(ExecuteRequestInput { + profile: Some("world-member-dispatch".to_string()), + cmd: String::new(), + cwd: "/tmp/worktree".to_string(), + env_map: std::collections::HashMap::new(), + agent_id: "tester".to_string(), + policy_snapshot: PolicySnapshotV3 { + schema_version: 3, + net_allowed: Vec::new(), + world_fs: PolicySnapshotWorldFsV3 { + host_visible: true, + fail_closed: PolicySnapshotWorldFsFailClosedV3 { routing: false }, + deny_enforcement: None, + caged_required: false, + discover: None, + read: None, + write: PolicySnapshotWorldFsWriteV3 { + enabled: true, + allow_list: vec![".".to_string()], + deny_list: Vec::new(), + }, + }, + }, + world_network: WorldNetworkRoutingV1 { + isolate_network: false, + allowed_domains: Vec::new(), + }, + world_fs_mode: WorldFsMode::Writable, + member_dispatch: Some(build_member_dispatch_payload( + &MemberDispatchTransportRequest { + orchestration_session_id: "orch_123".to_string(), + participant_id: "ash_member_123".to_string(), + orchestrator_participant_id: "ash_orch_123".to_string(), + parent_participant_id: None, + resumed_from_participant_id: None, + backend_id: "cli:codex".to_string(), + protocol: "uaa.agent.session".to_string(), + run_id: "run_123".to_string(), + world_id: "world_123".to_string(), + world_generation: 9, + initial_prompt: None, + backend_kind: MemberRuntimeBackendKindV1::Codex, + binary_path: "/usr/bin/codex".to_string(), + }, + )), + }); + + assert!(request.cmd.is_empty()); + assert!(!request.pty); + assert_eq!(request.agent_id, "tester"); + assert_eq!( + request + .member_dispatch + .as_ref() + .map(|dispatch| dispatch.run_id.as_str()), + Some("run_123") + ); + assert_eq!( + request + .member_dispatch + .as_ref() + .map(|dispatch| dispatch.resolved_runtime.binary_path.as_str()), + Some("/usr/bin/codex") + ); + request + .validate() + .expect("typed member dispatch request validates"); + } + #[test] fn current_world_request_profile_rejects_reserved_world_deps_profiles() { for reserved in ["world-deps-provision", "world-deps-probe"] { @@ -2005,6 +2518,7 @@ mod tests { let outcome = process_agent_stream_body( body, "agent".to_string(), + None, &mut sigint_rx, move |span_id, sig| { let cancels = Arc::clone(&cancels_for_cancel); @@ -2025,6 +2539,109 @@ mod tests { }); } + #[test] + #[serial_test::serial] + fn emit_stream_chunk_some_emits_orchestration_scoped_agent_event() { + let _guard = acquire_event_test_guard(); + let mut rx = init_event_channel(); + + emit_stream_chunk( + "agent", + Some("orch-live"), + "run-1", + Some("spn-1"), + b"hello stdout", + false, + ); + + let event = rx.try_recv().expect("stream chunk event"); + assert_eq!(event.kind, AgentEventKind::PtyData); + assert_eq!(event.orchestration_session_id, "orch-live"); + assert_eq!(event.run_id, "run-1"); + assert_eq!(event.span_id.as_deref(), Some("spn-1")); + clear_agent_event_sender(); + } + + #[test] + #[serial_test::serial] + fn emit_stream_chunk_none_emits_no_orchestration_scoped_agent_event() { + let _guard = acquire_event_test_guard(); + let mut rx = init_event_channel(); + + emit_stream_chunk("agent", None, "run-1", Some("spn-1"), b"hello stderr", true); + + assert!( + rx.try_recv().is_err(), + "stream chunk without orchestration context must not emit an agent event" + ); + clear_agent_event_sender(); + } + + #[test] + #[serial_test::serial] + fn process_agent_stream_body_uses_launch_owned_run_id_for_stream_rows() { + let _guard = acquire_event_test_guard(); + let rt = tokio::runtime::Runtime::new().expect("runtime"); + rt.block_on(async { + let mut rx = init_event_channel(); + let frames = vec![ + encode_stream_frame(ExecuteStreamFrame::Start { + span_id: "spn-world".to_string(), + }), + encode_stream_frame(ExecuteStreamFrame::Stdout { + chunk_b64: BASE64.encode(b"hello world"), + }), + encode_stream_frame(ExecuteStreamFrame::Exit { + exit: 0, + span_id: "spn-world".to_string(), + scopes_used: Vec::new(), + fs_diff: None, + process_telemetry: agent_api_types::ProcessTelemetry::default(), + }), + ]; + let body = StreamBody::new(stream::iter( + frames + .into_iter() + .map(|frame| Ok::<_, Infallible>(hyper::body::Frame::data(frame))), + )); + let mut sigint_rx = tokio::sync::mpsc::unbounded_channel().1; + let context = ShellCommandEventContext::new( + ShellEventEmissionContext { + orchestration_session_id: "orch-live".to_string(), + agent_id: "shell".to_string(), + role: Some("orchestrator".to_string()), + backend_id: Some("shell:repl".to_string()), + participant_id: Some("participant-1".to_string()), + parent_participant_id: None, + resumed_from_participant_id: None, + world_id: Some("world-1".to_string()), + world_generation: Some(4), + }, + "cmd-123", + Some("cmd-123".to_string()), + Some("spn-shell".to_string()), + ); + + let outcome = process_agent_stream_body( + body, + "agent".to_string(), + Some(&context), + &mut sigint_rx, + |_span_id, _sig| async { Ok(()) }, + ) + .await + .expect("process stream"); + + assert_eq!(outcome.exit_code, 0); + let event = rx.try_recv().expect("stream chunk event"); + assert_eq!(event.kind, AgentEventKind::PtyData); + assert_eq!(event.orchestration_session_id, "orch-live"); + assert_eq!(event.run_id, "cmd-123"); + assert_eq!(event.span_id.as_deref(), Some("spn-world")); + }); + clear_agent_event_sender(); + } + #[test] fn extract_process_telemetry_from_ws_exit_preserves_ptrace_not_permitted_reason() { let exit = json!({ diff --git a/crates/shell/src/execution/routing/dispatch/world_persistent_session.rs b/crates/shell/src/execution/routing/dispatch/world_persistent_session.rs index b1805d00c..c9eaf3dc7 100644 --- a/crates/shell/src/execution/routing/dispatch/world_persistent_session.rs +++ b/crates/shell/src/execution/routing/dispatch/world_persistent_session.rs @@ -62,6 +62,7 @@ mod imp { enum SessionState { Starting { ready_tx: tokio::sync::oneshot::Sender, + requested_shared_world: Option, }, Ready { next_seq: u64, @@ -81,6 +82,7 @@ mod imp { pub(crate) world_id: String, pub(crate) cwd: String, pub(crate) protocol_version: u32, + pub(crate) shared_world: Option, } #[derive(Debug, Clone)] @@ -88,6 +90,7 @@ mod imp { pub(crate) cwd: String, pub(crate) env: HashMap, pub(crate) policy_snapshot: agent_api_types::PolicySnapshotV3, + pub(crate) shared_world: Option, pub(crate) world_network: agent_api_types::WorldNetworkRoutingV1, pub(crate) cols: u16, pub(crate) rows: u16, @@ -107,6 +110,7 @@ mod imp { cwd, env, policy_snapshot, + shared_world: None, world_network, cols, rows, @@ -123,12 +127,16 @@ mod imp { start: ReplSessionStartParams, on_stdout: StdoutCallback, ) -> Result { + let requested_shared_world = start.shared_world.clone(); let (ws, start_frame) = build_ws_and_start_session_frame(start).await?; let (sink, stream) = ws.split(); let sink = Arc::new(Mutex::new(sink)); let (ready_tx, ready_rx) = tokio::sync::oneshot::channel(); - let state = Arc::new(Mutex::new(SessionState::Starting { ready_tx })); + let state = Arc::new(Mutex::new(SessionState::Starting { + ready_tx, + requested_shared_world, + })); let ready_cell = OnceCell::new(); let (fatal_tx, fatal_rx) = tokio::sync::watch::channel::>(None); @@ -217,6 +225,7 @@ mod imp { cwd, env: env_map, policy_snapshot: network_policy.snapshot, + shared_world: None, world_network, cols, rows, @@ -368,6 +377,8 @@ mod imp { cwd: String, env: HashMap, policy_snapshot: Box, + #[serde(skip_serializing_if = "Option::is_none")] + shared_world: Option, world_network: agent_api_types::WorldNetworkRoutingV1, cols: u16, rows: u16, @@ -400,6 +411,8 @@ mod imp { world_id: String, cwd: String, protocol_version: u32, + #[serde(default)] + shared_world: Option, }, Stdout { data_b64: String, @@ -533,21 +546,42 @@ mod imp { world_id, cwd, protocol_version, + shared_world, } => { if world_id.trim().is_empty() { return Err(anyhow!( "protocol error: ready.world_id must be a non-empty string" )); } + let mut guard = state.lock().await; + let requested_shared_world = match &*guard { + SessionState::Starting { + requested_shared_world, + .. + } => requested_shared_world.clone(), + _ => { + return Err(anyhow!( + "protocol error: unexpected ready frame after session start" + )) + } + }; + let validated_shared_world = + crate::execution::repl_persistent_session::validate_shared_world_echo( + requested_shared_world.as_ref(), + shared_world.as_ref(), + "ready.shared_world", + Some(world_id.as_str()), + ) + .map_err(|message| anyhow!("protocol error: {message}"))?; let ready = ReadyFrame { session_nonce, world_id, cwd, protocol_version, + shared_world: validated_shared_world, }; - let mut guard = state.lock().await; match std::mem::replace(&mut *guard, SessionState::Closed) { - SessionState::Starting { ready_tx } => { + SessionState::Starting { ready_tx, .. } => { *guard = SessionState::Ready { next_seq: 1 }; let _ = ready_tx.send(ready); Ok(()) @@ -668,6 +702,7 @@ mod imp { cwd, env, policy_snapshot, + shared_world, world_network, cols, rows, @@ -676,6 +711,7 @@ mod imp { cwd, env, policy_snapshot: Box::new(policy_snapshot), + shared_world, world_network, cols, rows, @@ -692,10 +728,15 @@ mod imp { cwd, mut env, policy_snapshot, + shared_world, world_network, cols, rows, } = start; + pw::reject_non_linux_shared_owner_request( + shared_world.as_ref(), + "persistent world session bootstrap", + )?; super::super::world_ops::normalize_env_for_linux_guest(&mut env); // Allow explicit socket overrides (used by tests/fixtures and advanced setups). @@ -713,6 +754,7 @@ mod imp { cwd, env, policy_snapshot: Box::new(policy_snapshot), + shared_world: shared_world.clone(), world_network: world_network.clone(), cols, rows, @@ -730,7 +772,7 @@ mod imp { pw::get_context().ok_or_else(|| anyhow!("no platform world context"))? } }; - (ctx.ensure_ready.as_ref())()?; + pw::ensure_persistent_session_ready_async(&ctx).await?; let (ws, _resp) = match &ctx.transport { pw::WorldTransport::Unix(path) => { @@ -768,6 +810,7 @@ mod imp { cwd, env, policy_snapshot: Box::new(policy_snapshot), + shared_world, world_network, cols, rows, @@ -813,7 +856,7 @@ mod imp { let mut guard = state.lock().await; let old = std::mem::replace(&mut *guard, SessionState::Closed); match old { - SessionState::Starting { ready_tx } => { + SessionState::Starting { ready_tx, .. } => { drop(ready_tx); } SessionState::InFlight { complete_tx, .. } => { @@ -826,6 +869,26 @@ mod imp { #[cfg(test)] mod tests { use super::*; + #[cfg(target_os = "macos")] + use anyhow::Result; + #[cfg(target_os = "macos")] + use futures::{SinkExt, StreamExt}; + #[cfg(target_os = "macos")] + use serial_test::serial; + #[cfg(target_os = "macos")] + use std::path::{Path, PathBuf}; + #[cfg(target_os = "macos")] + use std::sync::atomic::{AtomicUsize, Ordering}; + #[cfg(target_os = "macos")] + use std::sync::{Arc, Mutex, OnceLock}; + #[cfg(target_os = "macos")] + use tempfile::tempdir; + #[cfg(target_os = "macos")] + use tokio::net::UnixListener; + #[cfg(target_os = "macos")] + use tokio_tungstenite::accept_async; + #[cfg(target_os = "macos")] + use world_api::{ExecRequest, ExecResult, FsDiff, WorldBackend, WorldHandle, WorldSpec}; #[test] fn signal_frame_serializes_sig_field() { @@ -869,6 +932,10 @@ mod imp { }, }, }), + shared_world: Some(agent_api_types::SharedWorldOwnerSpec { + orchestration_session_id: "orch-test".to_string(), + action: agent_api_types::SharedWorldOwnerAction::AttachOrCreate, + }), world_network: agent_api_types::WorldNetworkRoutingV1 { isolate_network: true, allowed_domains: vec!["example.com".to_string()], @@ -883,6 +950,206 @@ mod imp { value["world_network"]["allowed_domains"], serde_json::json!(["example.com"]) ); + assert_eq!( + value["shared_world"], + serde_json::json!({ + "orchestration_session_id": "orch-test", + "action": "attach_or_create", + }) + ); + } + + #[cfg(target_os = "macos")] + static TEST_SYNC_READY_CALLS: AtomicUsize = AtomicUsize::new(0); + #[cfg(target_os = "macos")] + static TEST_ASYNC_READY_CALLS: AtomicUsize = AtomicUsize::new(0); + + #[cfg(target_os = "macos")] + fn test_env_lock() -> &'static Mutex<()> { + static LOCK: OnceLock> = OnceLock::new(); + LOCK.get_or_init(|| Mutex::new(())) + } + + #[cfg(target_os = "macos")] + #[derive(Clone)] + struct StubWorldBackend; + + #[cfg(target_os = "macos")] + impl WorldBackend for StubWorldBackend { + fn ensure_session(&self, _spec: &WorldSpec) -> Result { + Ok(WorldHandle { + id: "wld_test".to_string(), + shared_binding: None, + }) + } + + fn exec(&self, _world: &WorldHandle, _req: ExecRequest) -> Result { + anyhow::bail!("exec not implemented in test backend") + } + + fn fs_diff(&self, _world: &WorldHandle, _span_id: &str) -> Result { + Ok(FsDiff::default()) + } + + fn apply_policy(&self, _world: &WorldHandle, _spec: &WorldSpec) -> Result<()> { + Ok(()) + } + } + + #[cfg(target_os = "macos")] + fn test_start_params() -> ReplSessionStartParams { + ReplSessionStartParams { + cwd: "/tmp/project".to_string(), + env: HashMap::new(), + policy_snapshot: agent_api_types::PolicySnapshotV3 { + schema_version: 3, + net_allowed: Vec::new(), + world_fs: agent_api_types::PolicySnapshotWorldFsV3 { + host_visible: true, + fail_closed: agent_api_types::PolicySnapshotWorldFsFailClosedV3 { + routing: false, + }, + deny_enforcement: None, + caged_required: false, + discover: None, + read: None, + write: agent_api_types::PolicySnapshotWorldFsWriteV3 { + enabled: true, + allow_list: vec![".".to_string()], + deny_list: Vec::new(), + }, + }, + }, + shared_world: None, + world_network: agent_api_types::WorldNetworkRoutingV1 { + isolate_network: false, + allowed_domains: Vec::new(), + }, + cols: 80, + rows: 24, + } + } + + #[cfg(target_os = "macos")] + async fn spawn_ready_server(socket_path: &Path) -> tokio::task::JoinHandle> { + if let Some(parent) = socket_path.parent() { + tokio::fs::create_dir_all(parent) + .await + .expect("create socket parent"); + } + let _ = tokio::fs::remove_file(socket_path).await; + let listener = UnixListener::bind(socket_path).expect("bind ready socket"); + tokio::spawn(async move { + let (stream, _) = listener.accept().await.expect("accept start_session"); + let mut ws = accept_async(stream).await.expect("accept websocket"); + let message = ws.next().await.expect("start frame").expect("frame"); + let text = message.into_text().expect("text frame"); + assert!( + text.contains("\"type\":\"start_session\""), + "expected start_session frame, got: {text}" + ); + let ready = serde_json::json!({ + "type": "ready", + "session_nonce": "0123456789abcdef0123456789abcdef", + "world_id": "wld_test", + "cwd": "/tmp/project", + "protocol_version": 1, + }); + ws.send(tungs::tungstenite::Message::Text(ready.to_string())) + .await + .expect("send ready"); + Ok(()) + }) + } + + #[cfg(target_os = "macos")] + fn install_test_platform_context_once(socket_path: PathBuf) { + if pw::get_context().is_some() { + return; + } + + let backend: Arc = Arc::new(StubWorldBackend); + let transport = pw::WorldTransport::Unix(socket_path.clone()); + let ensure_ready = Box::new(|| { + TEST_SYNC_READY_CALLS.fetch_add(1, Ordering::SeqCst); + tokio::task::block_in_place(|| Ok(())) + }); + let ensure_persistent_session_ready_async = Box::new(|| { + TEST_ASYNC_READY_CALLS.fetch_add(1, Ordering::SeqCst); + Box::pin(async { Ok(()) }) as pw::PersistentSessionReadyFuture + }); + + pw::store_context_globally(pw::PlatformWorldContext { + backend, + transport, + socket_path, + ensure_ready, + ensure_persistent_session_ready_async, + }); + } + + #[cfg(target_os = "macos")] + #[test] + #[serial] + fn macos_no_override_current_thread_start_uses_async_readiness_without_panic() { + let _guard = test_env_lock().lock().expect("env lock"); + TEST_SYNC_READY_CALLS.store(0, Ordering::SeqCst); + TEST_ASYNC_READY_CALLS.store(0, Ordering::SeqCst); + std::env::remove_var("SUBSTRATE_WORLD_SOCKET"); + + let temp = tempdir().expect("tempdir"); + let socket_path = temp.path().join("world.sock"); + install_test_platform_context_once(socket_path.clone()); + + let runtime = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .expect("current-thread runtime"); + + runtime.block_on(async { + let server = spawn_ready_server(&socket_path).await; + let client = + ReplPersistentSessionClient::start_with(test_start_params(), Arc::new(|_| {})) + .await + .expect("start persistent session"); + client.close().await.expect("close client"); + server.await.expect("server join").expect("server result"); + }); + + assert_eq!(TEST_SYNC_READY_CALLS.load(Ordering::SeqCst), 0); + assert_eq!(TEST_ASYNC_READY_CALLS.load(Ordering::SeqCst), 1); + } + + #[cfg(target_os = "macos")] + #[test] + #[serial] + fn macos_socket_override_bypasses_platform_async_readiness() { + let _guard = test_env_lock().lock().expect("env lock"); + TEST_SYNC_READY_CALLS.store(0, Ordering::SeqCst); + TEST_ASYNC_READY_CALLS.store(0, Ordering::SeqCst); + + let temp = tempdir().expect("tempdir"); + let socket_path = temp.path().join("override.sock"); + + let runtime = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .expect("current-thread runtime"); + + runtime.block_on(async { + let server = spawn_ready_server(&socket_path).await; + std::env::set_var("SUBSTRATE_WORLD_SOCKET", &socket_path); + let client = + ReplPersistentSessionClient::start_with(test_start_params(), Arc::new(|_| {})) + .await + .expect("start persistent session with override"); + std::env::remove_var("SUBSTRATE_WORLD_SOCKET"); + client.close().await.expect("close client"); + server.await.expect("server join").expect("server result"); + }); + + assert_eq!(TEST_SYNC_READY_CALLS.load(Ordering::SeqCst), 0); + assert_eq!(TEST_ASYNC_READY_CALLS.load(Ordering::SeqCst), 0); } } } @@ -915,6 +1182,7 @@ mod imp { pub(crate) world_id: String, pub(crate) cwd: String, pub(crate) protocol_version: u32, + pub(crate) shared_world: Option, } #[derive(Debug, Clone)] @@ -922,6 +1190,7 @@ mod imp { pub(crate) cwd: String, pub(crate) env: HashMap, pub(crate) policy_snapshot: agent_api_types::PolicySnapshotV3, + pub(crate) shared_world: Option, pub(crate) world_network: agent_api_types::WorldNetworkRoutingV1, pub(crate) cols: u16, pub(crate) rows: u16, @@ -939,6 +1208,7 @@ mod imp { cwd, env: HashMap::new(), policy_snapshot, + shared_world: None, world_network, cols: 80, rows: 24, diff --git a/crates/shell/src/execution/routing/test_utils.rs b/crates/shell/src/execution/routing/test_utils.rs index 6a5430e4b..50a33503a 100644 --- a/crates/shell/src/execution/routing/test_utils.rs +++ b/crates/shell/src/execution/routing/test_utils.rs @@ -70,15 +70,18 @@ pub(crate) fn restore_env(key: &str, previous: Option) { pub(crate) struct DirGuard { original: PathBuf, + _env_guard: ReentrantMutexGuard<'static, ()>, _lock: ReentrantMutexGuard<'static, ()>, } impl DirGuard { pub(crate) fn new() -> Self { + let env_guard = world_env_guard(); let lock = cwd_lock().lock(); let original = env::current_dir().expect("capture cwd"); Self { original, + _env_guard: env_guard, _lock: lock, } } diff --git a/crates/shell/src/execution/routing/world.rs b/crates/shell/src/execution/routing/world.rs index cd6ec87db..9fad89118 100644 --- a/crates/shell/src/execution/routing/world.rs +++ b/crates/shell/src/execution/routing/world.rs @@ -5,6 +5,8 @@ use crate::execution::policy_snapshot::bootstrap_world_spec; #[cfg(all(test, any(target_os = "windows", target_os = "macos")))] use crate::execution::world_env_guard; use crate::execution::ShellConfig; +#[cfg(any(target_os = "macos", target_os = "windows"))] +use agent_api_types::SharedWorldOwnerSpec; #[cfg(target_os = "linux")] use super::dispatch::init_linux_world; @@ -28,6 +30,18 @@ pub(crate) fn initialize_world(config: &ShellConfig) { init_linux_world_default(config); } +#[cfg(any(target_os = "macos", target_os = "windows"))] +fn bootstrap_platform_world( + operation: &str, + shared_world: Option<&SharedWorldOwnerSpec>, + bootstrap: F, +) -> anyhow::Result<()> +where + F: FnOnce() -> anyhow::Result<()>, +{ + pw::with_supported_shared_world_request(shared_world, operation, bootstrap) +} + #[cfg(target_os = "windows")] fn init_windows_world(config: &ShellConfig) { if world_disabled(config) { @@ -44,21 +58,24 @@ fn init_windows_world(config: &ShellConfig) { return; } - match pw::detect() { - Ok(ctx) => { - if (ctx.ensure_ready)().is_ok() { - std::env::set_var("SUBSTRATE_WORLD", "enabled"); - if let Ok(handle) = ctx - .backend - .ensure_session(&pw::windows::bootstrap_world_spec()) - { - std::env::set_var("SUBSTRATE_WORLD_ID", handle.id); + let _ = bootstrap_platform_world("windows world bootstrap", None, || { + match pw::detect() { + Ok(ctx) => { + if (ctx.ensure_ready)().is_ok() { + std::env::set_var("SUBSTRATE_WORLD", "enabled"); + if let Ok(handle) = ctx + .backend + .ensure_session(&pw::windows::bootstrap_world_spec()) + { + std::env::set_var("SUBSTRATE_WORLD_ID", handle.id); + } } + pw::store_context_globally(ctx); } - pw::store_context_globally(ctx); + Err(_e) => {} } - Err(_e) => {} - } + Ok(()) + }); } #[cfg(target_os = "macos")] @@ -80,23 +97,26 @@ fn init_macos_world(config: &ShellConfig) { return; } - match pw::detect() { - Ok(ctx) => { - if (ctx.ensure_ready)().is_ok() { - // Set parity with Linux: world enabled + ID only - std::env::set_var("SUBSTRATE_WORLD", "enabled"); - - // Attempt to retrieve world id - let spec = - bootstrap_world_spec(config.world_root.effective_root(), world_fs_mode()); - if let Ok(handle) = ctx.backend.ensure_session(&spec) { - std::env::set_var("SUBSTRATE_WORLD_ID", handle.id); + let _ = bootstrap_platform_world("macos world bootstrap", None, || { + match pw::detect() { + Ok(ctx) => { + if (ctx.ensure_ready)().is_ok() { + // Set parity with Linux: world enabled + ID only + std::env::set_var("SUBSTRATE_WORLD", "enabled"); + + // Attempt to retrieve world id + let spec = + bootstrap_world_spec(config.world_root.effective_root(), world_fs_mode()); + if let Ok(handle) = ctx.backend.ensure_session(&spec) { + std::env::set_var("SUBSTRATE_WORLD_ID", handle.id); + } } + pw::store_context_globally(ctx); } - pw::store_context_globally(ctx); + Err(_e) => {} } - Err(_e) => {} - } + Ok(()) + }); } #[cfg(target_os = "linux")] @@ -104,3 +124,90 @@ fn init_linux_world_default(config: &ShellConfig) { let disabled = world_disabled(config); let _ = init_linux_world(disabled); } + +#[cfg(all(test, any(target_os = "macos", target_os = "windows")))] +mod tests { + use super::bootstrap_platform_world; + use agent_api_types::{SharedWorldOwnerAction, SharedWorldOwnerSpec}; + use std::sync::atomic::{AtomicUsize, Ordering}; + + #[cfg(target_os = "macos")] + #[test] + fn shared_owner_macos_allows_lima_backed_bootstrap() { + let calls = AtomicUsize::new(0); + let _env_guard = crate::execution::world_env_guard(); + let request = SharedWorldOwnerSpec { + orchestration_session_id: "orch-test".to_string(), + action: SharedWorldOwnerAction::AttachOrCreate, + }; + + std::env::remove_var("SUBSTRATE_WORLD_SOCKET"); + + bootstrap_platform_world("routing world bootstrap test", Some(&request), || { + calls.fetch_add(1, Ordering::SeqCst); + Ok(()) + }) + .expect("macOS shared-owner bootstrap should use the Lima-backed path"); + + assert_eq!( + calls.load(Ordering::SeqCst), + 1, + "bootstrap callback must run when macOS can use the Lima-backed path" + ); + } + + #[cfg(target_os = "macos")] + #[test] + fn shared_owner_macos_rejects_socket_override_before_bootstrap() { + let calls = AtomicUsize::new(0); + let _env_guard = crate::execution::world_env_guard(); + let request = SharedWorldOwnerSpec { + orchestration_session_id: "orch-test".to_string(), + action: SharedWorldOwnerAction::AttachOrCreate, + }; + + std::env::set_var("SUBSTRATE_WORLD_SOCKET", "/tmp/substrate-test.sock"); + let err = bootstrap_platform_world("routing world bootstrap test", Some(&request), || { + calls.fetch_add(1, Ordering::SeqCst); + Ok(()) + }) + .expect_err("explicit shared-owner bootstrap must reject socket overrides on macOS"); + + assert_eq!( + calls.load(Ordering::SeqCst), + 0, + "bootstrap callback must not run after macOS shared-owner override rejection" + ); + assert!( + err.to_string().contains("Lima-backed transport"), + "error should explain that the override bypasses the forwarded path: {err:#}" + ); + } + + #[cfg(target_os = "windows")] + #[test] + fn shared_owner_windows_rejects_before_bootstrap() { + let calls = AtomicUsize::new(0); + let request = SharedWorldOwnerSpec { + orchestration_session_id: "orch-test".to_string(), + action: SharedWorldOwnerAction::AttachOrCreate, + }; + + let err = bootstrap_platform_world("routing world bootstrap test", Some(&request), || { + calls.fetch_add(1, Ordering::SeqCst); + Ok(()) + }) + .expect_err("explicit shared-owner requests must reject before bootstrap fallback"); + + assert_eq!( + calls.load(Ordering::SeqCst), + 0, + "bootstrap callback must not run after early reject" + ); + assert!( + err.to_string() + .contains("explicit shared-owner world reuse"), + "error should explain early unsupported-platform rejection: {err:#}" + ); + } +} diff --git a/crates/shell/src/repl/async_repl.rs b/crates/shell/src/repl/async_repl.rs index 3f0b649f1..9794a061e 100644 --- a/crates/shell/src/repl/async_repl.rs +++ b/crates/shell/src/repl/async_repl.rs @@ -1,4 +1,5 @@ use std::collections::{BTreeMap, HashMap}; +use std::env; use std::io::{self, IsTerminal, Write}; use std::path::{Path, PathBuf}; use std::process::{Command, ExitStatus, Stdio}; @@ -7,29 +8,76 @@ use std::sync::{Arc, Mutex}; use std::thread; use std::time::Duration; +#[cfg(any(target_os = "linux", target_os = "macos"))] +use agent_api_client::AgentClient; +#[cfg(any(target_os = "linux", target_os = "macos"))] +use agent_api_types::{ + ExecuteCancelRequestV1, ExecuteStreamFrame, MemberRuntimeBackendKindV1, + MemberTurnSubmitRequestV1, +}; use anyhow::{anyhow, Context, Result}; use futures::{pin_mut, FutureExt, StreamExt}; use reedline::{ExternalPrinter, Prompt, Reedline, Signal}; +use serde::Deserialize; use tokio::runtime::Builder as TokioRuntimeBuilder; use tokio::sync::mpsc::{self, UnboundedReceiver, UnboundedSender}; use tokio::task; use uuid::Uuid; use crate::execution::agent_events::{ - clear_agent_event_sender, format_event_line, init_event_channel, orchestration_session_id, - publish_agent_event, publish_command_completion, schedule_demo_burst, schedule_demo_events, + clear_agent_event_sender, format_event_line, init_event_channel, publish_agent_event, + publish_command_completion, schedule_demo_burst, schedule_demo_events, + ShellCommandEventContext, ShellEventEmissionContext, +}; +use crate::execution::agent_inventory::{load_effective_agent_inventory, AgentInventoryEntryV1}; +#[cfg(any(target_os = "linux", target_os = "macos"))] +use crate::execution::agent_runtime::control::spawn_remote_private_prompt_owner; +use crate::execution::agent_runtime::control::{ + apply_runtime_stop_closeout, build_session_resume_extension, + invalidate_stale_world_members_after_binding, mark_orchestration_session_failed, + mark_runtime_startup_failed, note_runtime_stop_requested, persist_runtime_snapshots, + persist_world_binding_authority, private_prompt_request_channel, private_stop_request_channel, + prompt_runtime_from_parts, register_private_prompt_transport, register_private_stop_transport, + runtime_controls_parent_session, runtime_is_terminal, runtime_stop_transport_ids, + spawn_local_private_prompt_owner, spawn_local_private_stop_owner, submit_host_prompt_turn, + HiddenOwnerHelperLaunchPlan, OwnerHelperMode, PersistedWorldBinding, PrivatePromptTransport, + PrivateStopOutcome, PrivateStopRequestReceiver, PrivateStopTransport, PublicPromptAction, + PublicPromptEnvelope, PublicSessionPosture, ResolvedRuntimeDescriptor, + SubmittedPromptStreamEvent, AGENT_API_SESSION_RESUME_V1, AGENT_API_TURN_LIFECYCLE_V1, +}; +#[cfg(any(target_os = "linux", target_os = "macos"))] +use crate::execution::agent_runtime::mapping::AgentRuntimeBackendKind; +use crate::execution::agent_runtime::orchestration_session::{ + OrchestrationSessionPosture, StartupPromptStreamState, +}; +use crate::execution::agent_runtime::session::AgentRuntimeReplacementParticipantInit; +use crate::execution::agent_runtime::state_store::valid_detached_host_continuity_posture; +use crate::execution::agent_runtime::validator::RuntimeSelectionDescriptor; +use crate::execution::agent_runtime::validator::{ + exact_backend_selection_error_exit_code, validate_exact_backend_selection, }; -use crate::execution::agent_inventory::load_effective_agent_inventory; +#[cfg(any(target_os = "linux", target_os = "macos"))] +use crate::execution::agent_runtime::validator::{ + member_selection_error_exit_code, validate_member_selection, +}; +#[cfg(any(target_os = "linux", target_os = "macos"))] +use crate::execution::agent_runtime::AgentRuntimeParticipantWorldBinding; use crate::execution::agent_runtime::{ backend_allowed, build_gateway_for_descriptor, runtime_realizability_error_exit_code, - validate_orchestrator_selection, validate_runtime_realizability, AgentRuntimeSessionManifest, - AgentRuntimeSessionState, AgentRuntimeStateStore, PURE_AGENT_PROTOCOL, - SESSION_HANDLE_SCHEMA_V1, + validate_orchestrator_selection, validate_runtime_realizability, AgentRuntimeParticipantRecord, + AgentRuntimeSessionManifest, AgentRuntimeSessionState, AgentRuntimeStateStore, + OrchestrationSessionRecord, OrchestrationSessionState, MEMBER_ROLE, ORCHESTRATOR_ROLE, + PURE_AGENT_PROTOCOL, SESSION_HANDLE_SCHEMA_V1, }; #[cfg(unix)] use crate::execution::get_terminal_size; use crate::execution::ReplSessionTelemetry; use crate::execution::WorldRootSettings; +#[cfg(any(target_os = "linux", target_os = "macos"))] +use crate::execution::{ + build_agent_client_and_member_dispatch_request, build_agent_client_and_pending_diff_request, + MemberDispatchTransportRequest, +}; use crate::execution::{ canonicalize_or, enforce_caged_destination, execute_command, find_workspace_root, is_shell_stream_event, needs_pty, policy_snapshot, resolve_world_root, setup_signal_handlers, @@ -37,8 +85,12 @@ use crate::execution::{ ShellConfig, PTY_ACTIVE, }; use crate::repl::editor; +#[cfg(any(target_os = "linux", target_os = "macos"))] +use substrate_broker::Policy; use substrate_broker::{detect_profile, world_fs_policy}; use substrate_common::agent_events::{AgentEvent, MessageEventKind}; +use substrate_common::paths as substrate_paths; +use substrate_common::WorldRootMode; #[derive(Clone)] enum ReplPrinter { @@ -353,6 +405,12 @@ enum PromptWorkerErrorDisposition { GenericError, } +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +enum HostRuntimeShutdownMode { + Stop, + ParkIfResumable, +} + fn classify_prompt_worker_error( is_reedline: bool, err: &anyhow::Error, @@ -432,16 +490,44 @@ pub(crate) fn run_async_repl(config: &ShellConfig) -> Result { let prompt_active = Arc::new(AtomicBool::new(false)); let stdout_cb = make_world_stdout_callback(prompt_active.clone(), stdout_printer); + let resolved_host_bootstrap = match resolve_host_orchestrator_bootstrap(&shared_config) { + Ok(resolved) => resolved, + Err(failure) => { + agent_printer.print(failure.message.clone()); + write_best_effort_stderr_line(&failure.message); + tokio::time::sleep(Duration::from_millis(100)).await; + return Ok(failure.exit_code); + } + }; + let (prepared_runtime, mut dormant_host_bootstrap) = match resolved_host_bootstrap { + Some(resolved) => (None, Some(resolved)), + None => (None, None), + }; + let mut startup_context = None; + let mut world_session = if !shared_config.no_world { let requested = std::env::current_dir() .unwrap_or_else(|_| PathBuf::from(".")) .display() .to_string(); - match start_world_session(requested, stdout_cb.clone(), &agent_printer, &mut telemetry) - .await + match start_world_session( + requested, + startup_context.as_ref(), + stdout_cb.clone(), + &agent_printer, + &mut telemetry, + ) + .await { Ok(session) => Some(session), Err(err) => { + if let Some(startup_context) = startup_context.as_ref() { + mark_orchestration_session_failed( + &startup_context.store, + &startup_context.orchestration_session, + format!("failed to start persistent world session: {err:#}"), + ); + } let exit_code = if is_world_restart_required_error(&err) { 3 } else { @@ -463,8 +549,13 @@ pub(crate) fn run_async_repl(config: &ShellConfig) -> Result { } else { None }; - let mut agent_runtime = match start_host_orchestrator_runtime( - &shared_config, + let initial_world_binding = world_session.as_ref().map(|session| PersistedWorldBinding { + world_id: session.world_id.clone(), + world_generation: session.world_generation, + }); + let mut agent_runtime = match start_host_orchestrator_runtime_with_prepared( + prepared_runtime, + initial_world_binding.as_ref(), &agent_printer, &mut telemetry, ) @@ -472,16 +563,25 @@ pub(crate) fn run_async_repl(config: &ShellConfig) -> Result { { Ok(runtime) => runtime, Err(failure) => { + finalize_runtime_startup_failure( + startup_context.as_ref(), + &mut world_session, + &failure.message, + ) + .await; agent_printer.print(failure.message.clone()); write_best_effort_stderr_line(&failure.message); tokio::time::sleep(Duration::from_millis(100)).await; return Ok(failure.exit_code); } }; + let mut member_runtimes = RetainedMemberRuntimeMap::new(); + let mut pending_member_replacements = PendingMemberReplacementMap::new(); let mut should_exit = false; let mut termination_cause = ReplTerminationCause::NormalExit; - while !should_exit { + let mut fatal_runtime_error: Option = None; + 'repl_loop: while !should_exit { prompt_active.store(true, Ordering::SeqCst); if let Err(err) = prompt_worker.request_prompt() { termination_cause = ReplTerminationCause::AbnormalTerminalLoss; @@ -604,6 +704,46 @@ pub(crate) fn run_async_repl(config: &ShellConfig) -> Result { let cmd_id = Uuid::now_v7().to_string(); + if trimmed.starts_with("::") { + let Some(targeted_turn) = parse_targeted_turn(trimmed) else { + agent_printer.print( + "substrate: error: targeted follow-up turns require exact syntax ':: ' on a single line", + ); + continue; + }; + + let result = match dispatch_targeted_follow_up_turn( + targeted_turn, + TargetedTurnDispatchContext { + startup_context: &mut startup_context, + dormant_host_bootstrap: &mut dormant_host_bootstrap, + agent_runtime: &mut agent_runtime, + world_session: &mut world_session, + member_runtimes: &mut member_runtimes, + pending_member_replacements: &mut pending_member_replacements, + agent_printer: &agent_printer, + telemetry: &mut telemetry, + }, + ) + .await + { + Ok(TargetedTurnDispatchStatus::Submitted) => Ok(()), + Ok(TargetedTurnDispatchStatus::Rejected(failure)) => { + agent_printer + .print(format!("substrate: error: {}", failure.message)); + continue; + } + Err(err) => Err(err), + }; + + if let Err(err) = result { + fatal_runtime_error = Some(err); + should_exit = true; + continue 'repl_loop; + } + continue; + } + if !has_embedded_newlines(&command) { if trimmed == ":host" { agent_printer.print("substrate: error: :host requires a command"); @@ -633,6 +773,12 @@ pub(crate) fn run_async_repl(config: &ShellConfig) -> Result { agent_printer: &agent_printer, max_pty_buffered_lines, }; + let command_event_context = build_repl_shell_command_event_context( + startup_context.as_ref(), + agent_runtime.as_ref(), + world_session.as_ref(), + &cmd_id, + ); let exit_code = exec_host_line( shared_config.as_ref(), &mut host_state, @@ -642,10 +788,22 @@ pub(crate) fn run_async_repl(config: &ShellConfig) -> Result { world_session.as_ref().map(|s| &s.client), &mut io_ctx, ) - .await?; + .await; + let exit_code = match exit_code { + Ok(exit_code) => exit_code, + Err(err) => { + fatal_runtime_error = Some(err); + should_exit = true; + continue 'repl_loop; + } + }; let status = exit_status_from_code(exit_code); report_nonzero_status(&status); - publish_command_completion(&trimmed_owned, &cmd_id, &status); + publish_command_completion( + command_event_context.as_ref(), + &trimmed_owned, + &status, + ); telemetry.record_command(); continue; } @@ -658,12 +816,52 @@ pub(crate) fn run_async_repl(config: &ShellConfig) -> Result { } if world_session.is_some() { - ensure_no_policy_drift( + let drift_check = ensure_no_policy_drift( &mut world_session, + startup_context.as_ref(), + &agent_printer, + &mut telemetry, + ) + .await; + if let Err(err) = drift_check { + fatal_runtime_error = Some(err); + should_exit = true; + continue 'repl_loop; + } + if let Err(err) = reconcile_member_runtime_generation( + world_session.as_ref(), + &mut member_runtimes, + &mut pending_member_replacements, + &agent_printer, + &mut telemetry, + ) + .await + { + fatal_runtime_error = Some(err); + should_exit = true; + continue 'repl_loop; + } + if let Err(err) = ensure_member_runtime_ready( + startup_context.as_ref(), + world_session.as_ref(), + &mut member_runtimes, + &mut pending_member_replacements, &agent_printer, &mut telemetry, ) - .await?; + .await + { + fatal_runtime_error = Some(err); + should_exit = true; + continue 'repl_loop; + } + let command_event_context = + build_repl_shell_command_event_context( + startup_context.as_ref(), + agent_runtime.as_ref(), + world_session.as_ref(), + &cmd_id, + ); let exit_code = { let session = world_session .as_mut() @@ -676,17 +874,49 @@ pub(crate) fn run_async_repl(config: &ShellConfig) -> Result { agent_printer: &agent_printer, max_pty_buffered_lines, }; - exec_world_pty(session, pty_cmd, &cmd_id, &mut io_ctx).await? + match exec_world_pty(session, pty_cmd, &cmd_id, &mut io_ctx) + .await + { + Ok(exit_code) => exit_code, + Err(err) => { + fatal_runtime_error = Some(err); + should_exit = true; + continue 'repl_loop; + } + } }; - ensure_no_policy_drift( + let drift_check = ensure_no_policy_drift( &mut world_session, + startup_context.as_ref(), + &agent_printer, + &mut telemetry, + ) + .await; + if let Err(err) = drift_check { + fatal_runtime_error = Some(err); + should_exit = true; + continue 'repl_loop; + } + if let Err(err) = reconcile_member_runtime_generation( + world_session.as_ref(), + &mut member_runtimes, + &mut pending_member_replacements, &agent_printer, &mut telemetry, ) - .await?; + .await + { + fatal_runtime_error = Some(err); + should_exit = true; + continue 'repl_loop; + } let status = exit_status_from_code(exit_code); report_nonzero_status(&status); - publish_command_completion(&trimmed_owned, &cmd_id, &status); + publish_command_completion( + command_event_context.as_ref(), + &trimmed_owned, + &status, + ); telemetry.record_command(); continue; } @@ -694,12 +924,51 @@ pub(crate) fn run_async_repl(config: &ShellConfig) -> Result { } if world_session.is_some() { - ensure_no_policy_drift( + let drift_check = ensure_no_policy_drift( &mut world_session, + startup_context.as_ref(), + &agent_printer, + &mut telemetry, + ) + .await; + if let Err(err) = drift_check { + fatal_runtime_error = Some(err); + should_exit = true; + continue 'repl_loop; + } + if let Err(err) = reconcile_member_runtime_generation( + world_session.as_ref(), + &mut member_runtimes, + &mut pending_member_replacements, + &agent_printer, + &mut telemetry, + ) + .await + { + fatal_runtime_error = Some(err); + should_exit = true; + continue 'repl_loop; + } + if let Err(err) = ensure_member_runtime_ready( + startup_context.as_ref(), + world_session.as_ref(), + &mut member_runtimes, + &mut pending_member_replacements, &agent_printer, &mut telemetry, ) - .await?; + .await + { + fatal_runtime_error = Some(err); + should_exit = true; + continue 'repl_loop; + } + let command_event_context = build_repl_shell_command_event_context( + startup_context.as_ref(), + agent_runtime.as_ref(), + world_session.as_ref(), + &cmd_id, + ); let pty = needs_pty(trimmed); let exit_code = { let session = world_session @@ -714,20 +983,61 @@ pub(crate) fn run_async_repl(config: &ShellConfig) -> Result { max_pty_buffered_lines, }; if pty { - exec_world_pty(session, &command, &cmd_id, &mut io_ctx).await? + match exec_world_pty(session, &command, &cmd_id, &mut io_ctx) + .await + { + Ok(exit_code) => exit_code, + Err(err) => { + fatal_runtime_error = Some(err); + should_exit = true; + continue 'repl_loop; + } + } } else { - exec_world_line(session, &command, &cmd_id, &mut io_ctx).await? + match exec_world_line(session, &command, &cmd_id, &mut io_ctx) + .await + { + Ok(exit_code) => exit_code, + Err(err) => { + fatal_runtime_error = Some(err); + should_exit = true; + continue 'repl_loop; + } + } } }; - ensure_no_policy_drift( + let drift_check = ensure_no_policy_drift( &mut world_session, + startup_context.as_ref(), + &agent_printer, + &mut telemetry, + ) + .await; + if let Err(err) = drift_check { + fatal_runtime_error = Some(err); + should_exit = true; + continue 'repl_loop; + } + if let Err(err) = reconcile_member_runtime_generation( + world_session.as_ref(), + &mut member_runtimes, + &mut pending_member_replacements, &agent_printer, &mut telemetry, ) - .await?; + .await + { + fatal_runtime_error = Some(err); + should_exit = true; + continue 'repl_loop; + } let status = exit_status_from_code(exit_code); report_nonzero_status(&status); - publish_command_completion(&trimmed_owned, &cmd_id, &status); + publish_command_completion( + command_event_context.as_ref(), + &trimmed_owned, + &status, + ); telemetry.record_command(); continue; } @@ -738,12 +1048,20 @@ pub(crate) fn run_async_repl(config: &ShellConfig) -> Result { let running_clone = running_child_pid.clone(); let command_for_exec = command.clone(); let cmd_id_for_exec = cmd_id.clone(); + let command_event_context = build_repl_shell_command_event_context( + startup_context.as_ref(), + agent_runtime.as_ref(), + world_session.as_ref(), + &cmd_id, + ); + let command_event_context_for_exec = command_event_context.clone(); let command_fut = task::spawn_blocking(move || { execute_command( &config_clone, &command_for_exec, &cmd_id_for_exec, running_clone, + command_event_context_for_exec, ) }) .map(|res: Result, tokio::task::JoinError>| { @@ -759,7 +1077,15 @@ pub(crate) fn run_async_repl(config: &ShellConfig) -> Result { let status = loop { tokio::select! { - res = &mut command_fut => break res?, + res = &mut command_fut => { + match res { + Ok(status) => break Some(status), + Err(err) => { + fatal_runtime_error = Some(err); + break None; + } + } + } maybe_event = agent_rx.recv() => { if let Some(event) = maybe_event { if is_shell_stream_event(&event) { @@ -785,6 +1111,10 @@ pub(crate) fn run_async_repl(config: &ShellConfig) -> Result { _maybe_sigint = sigint_rx.recv() => {} } }; + let Some(status) = status else { + should_exit = true; + continue 'repl_loop; + }; for line in buffered_structured_lines { agent_printer.print(line); @@ -801,7 +1131,11 @@ pub(crate) fn run_async_repl(config: &ShellConfig) -> Result { } report_nonzero_status(&status); - publish_command_completion(&trimmed_owned, &cmd_id, &status); + publish_command_completion( + command_event_context.as_ref(), + &trimmed_owned, + &status, + ); telemetry.record_command(); } PromptWorkerResponse::CtrlC => { @@ -822,8 +1156,16 @@ pub(crate) fn run_async_repl(config: &ShellConfig) -> Result { write_best_effort_stderr_line( "substrate: warning: prompt backend degraded (cursor query timeout); falling back to plain stdin reader" ); - prompt_worker = PromptWorker::spawn_stdio(shared_config.clone()) - .context("failed to start plain prompt worker")?; + prompt_worker = match PromptWorker::spawn_stdio(shared_config.clone()) + .context("failed to start plain prompt worker") + { + Ok(worker) => worker, + Err(err) => { + fatal_runtime_error = Some(err); + should_exit = true; + continue 'repl_loop; + } + }; agent_printer = prompt_worker.printer_handle(); prompt_responses = prompt_worker.take_response_receiver(); continue; @@ -895,12 +1237,26 @@ pub(crate) fn run_async_repl(config: &ShellConfig) -> Result { } }; + shutdown_all_member_runtimes(&mut member_runtimes, &agent_printer, &mut telemetry).await; + if let Some(runtime) = agent_runtime.take() { + let shutdown_mode = if termination_cause == ReplTerminationCause::AbnormalTerminalLoss + { + HostRuntimeShutdownMode::ParkIfResumable + } else { + HostRuntimeShutdownMode::Stop + }; + shutdown_host_orchestrator_runtime_with_mode( + runtime, + shutdown_mode, + &agent_printer, + &mut telemetry, + ) + .await; + } + drain_pending_agent_events(&mut agent_rx, &mut telemetry, &agent_printer); prompt_worker.shutdown_with_disposition(shutdown_disposition_for_termination_cause( termination_cause, )); - if let Some(runtime) = agent_runtime.take() { - shutdown_host_orchestrator_runtime(runtime, &agent_printer, &mut telemetry).await; - } clear_agent_event_sender(); let auto_sync_exit_code: i32 = { @@ -951,6 +1307,14 @@ pub(crate) fn run_async_repl(config: &ShellConfig) -> Result { let _ = session.client.close().await; } + if let Some(err) = fatal_runtime_error.as_ref() { + let (exit_code, message) = runtime_loop_exit(err); + agent_printer.print(message.clone()); + write_best_effort_stderr_line(&message); + tokio::time::sleep(Duration::from_millis(100)).await; + return Ok(exit_code); + } + let exit_code = match termination_cause { ReplTerminationCause::NormalExit => auto_sync_exit_code, ReplTerminationCause::AbnormalTerminalLoss => 1, @@ -973,10 +1337,16 @@ struct PromptWorker { impl PromptWorker { fn spawn(config: Arc) -> Result { + // Tests that intentionally exercise Reedline can force that path even when they also skip + // shims for bootstrap simplicity. We still require a real terminal below. + let force_reedline = std::env::var_os("SUBSTRATE_FORCE_REEDLINE").is_some(); // CI runners often drive Substrate through PTY harnesses like `script` where Reedline's // cursor position query can consume the piped input stream. Prefer a plain stdin-backed - // prompt in CI to keep smoke runs deterministic. + // prompt in CI to keep smoke runs deterministic. Do the same when shims are explicitly + // skipped, since that mode is primarily used by tests and diagnostic harnesses that + // emulate a TTY but do not always satisfy Reedline's terminal capability probes in time. if config.ci_mode + || (config.skip_shims && !force_reedline) || std::env::var_os("CI").is_some() || std::env::var_os("GITHUB_ACTIONS").is_some() { @@ -1227,95 +1597,633 @@ fn handle_agent_event( agent_printer.print(format_event_line(&event)); } +fn drain_pending_agent_events( + agent_rx: &mut tokio::sync::mpsc::UnboundedReceiver, + telemetry: &mut ReplSessionTelemetry, + agent_printer: &ReplPrinter, +) { + while let Ok(event) = agent_rx.try_recv() { + handle_agent_event(event, telemetry, agent_printer); + } +} + #[derive(Debug)] struct RuntimeBootstrapFailure { exit_code: i32, message: String, } +#[derive(Clone, Debug, PartialEq, Eq)] +struct TargetedTurn<'a> { + backend_id: &'a str, + prompt: &'a str, +} + +enum TargetedTurnRoute { + Host, + World(RuntimeSelectionDescriptor), +} + +enum TargetedTurnDispatchStatus { + Submitted, + Rejected(RuntimeBootstrapFailure), +} + +struct TargetedTurnDispatchContext<'a> { + startup_context: &'a mut Option, + dormant_host_bootstrap: &'a mut Option, + agent_runtime: &'a mut Option, + world_session: &'a mut Option, + member_runtimes: &'a mut RetainedMemberRuntimeMap, + pending_member_replacements: &'a mut PendingMemberReplacementMap, + agent_printer: &'a ReplPrinter, + telemetry: &'a mut ReplSessionTelemetry, +} + +#[derive(Clone)] +struct RuntimeOrchestrationContext { + store: AgentRuntimeStateStore, + orchestration_session: Arc>, + effective_config: crate::execution::config_model::SubstrateConfig, + #[cfg(any(target_os = "linux", target_os = "macos"))] + base_policy: Policy, + inventory: BTreeMap, +} + +impl RuntimeOrchestrationContext { + fn orchestration_session_id(&self) -> String { + self.orchestration_session + .lock() + .expect("orchestration session mutex poisoned") + .orchestration_session_id + .clone() + } + + fn snapshot(&self) -> OrchestrationSessionRecord { + self.orchestration_session + .lock() + .expect("orchestration session mutex poisoned") + .clone() + } +} + +struct PreparedAgentRuntime { + descriptor: RuntimeSelectionDescriptor, + gateway: agent_api::AgentWrapperGateway, + agent_kind: agent_api::AgentWrapperKind, + startup_context: RuntimeOrchestrationContext, + manifest: Arc>, + run_id: String, + startup_extensions: BTreeMap, +} + +struct ResolvedHostOrchestratorBootstrap { + cwd: PathBuf, + shell_session_id: String, + descriptor: RuntimeSelectionDescriptor, + gateway: agent_api::AgentWrapperGateway, + agent_kind: agent_api::AgentWrapperKind, + state_store: AgentRuntimeStateStore, + effective_config: crate::execution::config_model::SubstrateConfig, + #[cfg(any(target_os = "linux", target_os = "macos"))] + base_policy: Policy, + inventory: BTreeMap, +} + enum RuntimeStartupSignal { Running, Failed(String), } +#[derive(Clone, Debug)] +enum InitialExecPromptPlan { + Replace(String), + StartupPrompt { + prompt: String, + stream_path: PathBuf, + }, +} + +#[derive(Clone)] +struct StartupPromptBackchannel { + envelope_tx: UnboundedSender, + terminal_sent: Arc, +} + +impl StartupPromptBackchannel { + fn send(&self, envelope: PublicPromptEnvelope) { + if self.terminal_sent.load(Ordering::SeqCst) { + return; + } + let terminal = matches!( + envelope, + PublicPromptEnvelope::Completed { .. } | PublicPromptEnvelope::Failed { .. } + ); + if terminal { + self.terminal_sent.store(true, Ordering::SeqCst); + } + let _ = self.envelope_tx.send(envelope); + } + + fn send_event(&self, event: &AgentEvent) { + self.send(PublicPromptEnvelope::Event { + version: 1, + event_kind: "message".to_string(), + data: serde_json::to_value(event).unwrap_or_default(), + }); + } +} + // The REPL retains live UAA runtime ownership via the cancel handle plus the two // long-lived tasks that own the non-clonable `run_control.handle` facets. A manifest // may only advertise a live orchestrator session while all three remain retained. -struct RetainedRunControl { +struct LocalRetainedRunControl { cancel: agent_api::AgentWrapperCancelHandle, event_task: Option>, completion_task: Option>, } +#[cfg(any(target_os = "linux", target_os = "macos"))] +struct RemoteRetainedRunControl { + client: Arc, + span_id: String, + observe_task: Option>, +} + +enum RetainedRunControl { + Local(LocalRetainedRunControl), + #[cfg(any(target_os = "linux", target_os = "macos"))] + Remote(RemoteRetainedRunControl), +} + +const LOCAL_RETAINED_STOP_COMPLETION_TIMEOUT: Duration = Duration::from_secs(5); + struct AsyncReplAgentRuntime { + descriptor: RuntimeSelectionDescriptor, + orchestration_session: Arc>, manifest: Arc>, store: AgentRuntimeStateStore, uaa_session_handle_id: String, retained_control: RetainedRunControl, shutdown_requested: Arc, + auto_park_rx: Option>, + private_stop_rx: Option, + stop_transport: Option, + stop_owner_task: Option>, + prompt_transport: Option, + prompt_owner_task: Option>, heartbeat_stop_tx: Option>, heartbeat_task: Option>, } -async fn start_host_orchestrator_runtime( - config: &Arc, - agent_printer: &ReplPrinter, - telemetry: &mut ReplSessionTelemetry, -) -> std::result::Result, RuntimeBootstrapFailure> { - let cwd = std::env::current_dir().unwrap_or_else(|_| PathBuf::from(".")); - let effective_config = crate::execution::config_model::resolve_effective_config( - &cwd, - &crate::execution::config_model::CliConfigOverrides { - world_enabled: if config.cli_world { - Some(true) - } else if config.cli_no_world { - Some(false) - } else { - None - }, - anchor_mode: config.cli_anchor_mode, - anchor_path: config - .cli_anchor_path - .as_ref() - .map(|path| path.to_string_lossy().to_string()), - caged: config.cli_caged, - }, - ) - .map_err(runtime_bootstrap_failure_from_anyhow)?; - if !effective_config.agents.enabled { - return Ok(None); +type RetainedMemberRuntimeMap = BTreeMap; +type PendingMemberReplacementMap = BTreeMap; + +fn runtime_registered_message(role: &str) -> &'static str { + if role == MEMBER_ROLE { + "world-scoped member session handle allocated" + } else { + "shell-owned orchestrator session handle allocated" } +} - let (base_policy, _) = substrate_broker::resolve_effective_policy_with_explain(&cwd, false) - .map_err(|err| RuntimeBootstrapFailure { - exit_code: 2, - message: err.to_string(), - })?; - let inventory = load_effective_agent_inventory(&cwd, &base_policy) - .map_err(runtime_bootstrap_failure_from_anyhow)?; - let orchestrator = - validate_orchestrator_selection(&effective_config, &inventory).map_err(|reason| { - RuntimeBootstrapFailure { - exit_code: 2, - message: reason, - } - })?; +fn runtime_task_start_message(role: &str) -> &'static str { + if role == MEMBER_ROLE { + "starting long-lived world-scoped member control turn" + } else { + "starting long-lived shell-owned orchestrator control turn" + } +} - if !backend_allowed(&base_policy, &orchestrator.derived_backend_id()) { - return Err(RuntimeBootstrapFailure { - exit_code: 5, - message: format!( - "selected orchestrator backend '{}' is not allowlisted by effective policy agents.allowed_backends", - orchestrator.derived_backend_id() - ), - }); +fn runtime_bootstrap_prompt(role: &str) -> &'static str { + if role == MEMBER_ROLE { + "Enter persistent Substrate world-scoped member mode. Keep this control session attached for the lifetime of the parent REPL session and do not exit until the client cancels the run." + } else { + "Enter persistent Substrate host orchestrator mode. Keep this control session attached for the lifetime of the host REPL and do not exit until the client cancels the run." } +} - let descriptor = - validate_runtime_realizability(orchestrator, &effective_config).map_err(|err| { - RuntimeBootstrapFailure { - exit_code: runtime_realizability_error_exit_code(&err), - message: err.reason, +fn runtime_ready_message(role: &str) -> &'static str { + if role == MEMBER_ROLE { + "world-scoped member session is ready via retained attached control ownership" + } else { + "shell-owned orchestrator session is ready via retained attached control ownership" + } +} + +#[derive(Debug, Deserialize)] +struct TurnLifecycleEventV1 { + schema: String, + turn: TurnLifecyclePayloadV1, +} + +#[derive(Debug, Deserialize)] +struct TurnLifecyclePayloadV1 { + thread_id: String, + turn_id: String, + phase: String, +} + +fn turn_lifecycle_phase(data: Option<&serde_json::Value>) -> Option<&str> { + let value = data?; + let parsed: TurnLifecycleEventV1 = serde_json::from_value(value.clone()).ok()?; + if parsed.schema != AGENT_API_TURN_LIFECYCLE_V1 { + return None; + } + let _ = (&parsed.turn.thread_id, &parsed.turn.turn_id); + Some(match parsed.turn.phase.as_str() { + "completed" => "completed", + "failed" => "failed", + _ => return None, + }) +} + +fn infer_startup_turn_phase_fallback( + event: &agent_api::AgentWrapperEvent, + saw_substantive_output: bool, +) -> Option<&'static str> { + match event.kind { + agent_api::AgentWrapperEventKind::Status => { + match event + .message + .as_deref() + .map(str::trim) + .filter(|message| !message.is_empty()) + { + Some("turn failed") => Some("failed"), + Some(_) => None, + None if event.data.is_none() && saw_substantive_output => Some("completed"), + None => None, + } + } + _ => None, + } +} + +#[cfg(unix)] +async fn connect_startup_prompt_backchannel( + stream_path: &Path, +) -> std::result::Result { + use tokio::io::AsyncWriteExt; + use tokio::net::UnixStream; + + let stream = UnixStream::connect(stream_path) + .await + .map_err(|err| RuntimeBootstrapFailure { + exit_code: 4, + message: format!( + "failed to connect hidden owner-helper startup prompt stream {}: {err}", + stream_path.display() + ), + })?; + let (envelope_tx, mut envelope_rx) = mpsc::unbounded_channel::(); + let terminal_sent = Arc::new(AtomicBool::new(false)); + let terminal_sent_for_task = Arc::clone(&terminal_sent); + tokio::spawn(async move { + let mut stream = stream; + while let Some(envelope) = envelope_rx.recv().await { + if stream + .write_all( + format!( + "{}\n", + serde_json::to_string(&envelope).unwrap_or_else(|_| "{}".to_string()) + ) + .as_bytes(), + ) + .await + .is_err() + { + break; + } + let _ = stream.flush().await; + if matches!( + envelope, + PublicPromptEnvelope::Completed { .. } | PublicPromptEnvelope::Failed { .. } + ) { + break; + } + } + terminal_sent_for_task.store(true, Ordering::SeqCst); + }); + + Ok(StartupPromptBackchannel { + envelope_tx, + terminal_sent, + }) +} + +#[cfg(not(unix))] +async fn connect_startup_prompt_backchannel( + _stream_path: &Path, +) -> std::result::Result { + Err(RuntimeBootstrapFailure { + exit_code: 4, + message: "hidden owner-helper startup prompt streaming requires Unix transports" + .to_string(), + }) +} + +fn startup_prompt_state_label(state: &AgentRuntimeSessionState) -> &'static str { + match state { + AgentRuntimeSessionState::Allocating + | AgentRuntimeSessionState::Ready + | AgentRuntimeSessionState::Running + | AgentRuntimeSessionState::Restarting + | AgentRuntimeSessionState::Stopping => "active", + AgentRuntimeSessionState::Stopped => "stopped", + AgentRuntimeSessionState::Failed => "failed", + AgentRuntimeSessionState::Invalidated => "invalidated", + } +} + +fn startup_prompt_posture( + session: &OrchestrationSessionRecord, + manifest: &AgentRuntimeSessionManifest, +) -> PublicSessionPosture { + if session.state == OrchestrationSessionState::Active + && manifest.is_authoritative_live() + && session.attached_participant_id() == Some(manifest.handle.participant_id.as_str()) + { + PublicSessionPosture::Active + } else { + PublicSessionPosture::Terminal + } +} + +fn startup_prompt_completed_envelope( + session: &OrchestrationSessionRecord, + manifest: &AgentRuntimeSessionManifest, +) -> PublicPromptEnvelope { + PublicPromptEnvelope::Completed { + version: 1, + action: PublicPromptAction::Start, + orchestration_session_id: session.orchestration_session_id.clone(), + backend_id: manifest.handle.backend_id.clone(), + participant_id: Some(manifest.handle.participant_id.clone()), + turn_outcome: "success".to_string(), + session_posture: startup_prompt_posture(session, manifest), + state: startup_prompt_state_label(&manifest.handle.state).to_string(), + warnings: Vec::new(), + } +} + +fn startup_prompt_failed_envelope(message: impl Into) -> PublicPromptEnvelope { + PublicPromptEnvelope::Failed { + version: 1, + terminal: true, + stage: "runtime".to_string(), + error_code: "owner_unreachable".to_string(), + message: message.into(), + } +} + +fn terminalize_startup_prompt_failure( + orchestration_session: &mut OrchestrationSessionRecord, + manifest: &AgentRuntimeSessionManifest, + message: impl Into, +) -> Option { + if !matches!( + orchestration_session.startup_prompt_state(), + Some(StartupPromptStreamState::PendingAcceptance | StartupPromptStreamState::Accepted) + ) { + return None; + } + + let message = message.into(); + orchestration_session + .mark_startup_prompt_failed(manifest.handle.participant_id.as_str(), message.clone()); + Some(message) +} + +fn runtime_stopping_message(role: &str, uaa_session_handle_id: &str) -> String { + if role == MEMBER_ROLE { + format!( + "world-scoped member session stopping (uaa_session_handle_id={uaa_session_handle_id})" + ) + } else { + format!("shell-owned orchestrator session stopping (uaa_session_handle_id={uaa_session_handle_id})") + } +} + +fn runtime_stopped_message(role: &str) -> &'static str { + if role == MEMBER_ROLE { + "world-scoped member session stopped" + } else { + "shell-owned orchestrator session stopped" + } +} + +fn runtime_detached_message(role: &str) -> &'static str { + if role == MEMBER_ROLE { + "world-scoped member session detached cleanly" + } else { + "shell-owned orchestrator detached cleanly; session parked for resume" + } +} + +fn runtime_stream_closed_alert_code(role: &str) -> &'static str { + if role == MEMBER_ROLE { + "member_runtime_stream_closed" + } else { + "orchestrator_runtime_stream_closed" + } +} + +fn runtime_invalidated_alert_code(role: &str) -> &'static str { + if role == MEMBER_ROLE { + "member_runtime_invalidated" + } else { + "orchestrator_runtime_invalidated" + } +} + +fn build_repl_shell_event_emission_context( + startup_context: Option<&RuntimeOrchestrationContext>, + agent_runtime: Option<&AsyncReplAgentRuntime>, + world_session: Option<&WorldSession>, +) -> Option { + let startup_context = startup_context?; + let agent_runtime = agent_runtime?; + let orchestration_snapshot = startup_context.snapshot(); + let manifest_snapshot = agent_runtime + .manifest + .lock() + .expect("runtime manifest mutex poisoned") + .clone(); + + Some(ShellEventEmissionContext { + orchestration_session_id: orchestration_snapshot.orchestration_session_id, + agent_id: "shell".to_string(), + role: Some("orchestrator".to_string()), + backend_id: Some("shell:repl".to_string()), + participant_id: Some(manifest_snapshot.handle.participant_id), + parent_participant_id: manifest_snapshot.handle.parent_participant_id, + resumed_from_participant_id: manifest_snapshot.handle.resumed_from_participant_id, + world_id: world_session + .map(|session| session.world_id.clone()) + .or(orchestration_snapshot.world_id), + world_generation: world_session + .map(|session| session.world_generation) + .or(orchestration_snapshot.world_generation), + }) +} + +fn build_repl_shell_command_event_context( + startup_context: Option<&RuntimeOrchestrationContext>, + agent_runtime: Option<&AsyncReplAgentRuntime>, + world_session: Option<&WorldSession>, + cmd_id: &str, +) -> Option { + let emission = + build_repl_shell_event_emission_context(startup_context, agent_runtime, world_session)?; + Some(ShellCommandEventContext::new( + emission, + cmd_id, + Some(cmd_id.to_string()), + None, + )) +} + +#[allow(dead_code)] +async fn start_host_orchestrator_runtime( + config: &Arc, + agent_printer: &ReplPrinter, + telemetry: &mut ReplSessionTelemetry, +) -> std::result::Result, RuntimeBootstrapFailure> { + let prepared = prepare_host_orchestrator_runtime_startup(config)?; + start_host_orchestrator_runtime_with_prepared(prepared, None, agent_printer, telemetry).await +} + +fn prepare_host_orchestrator_runtime_startup( + config: &Arc, +) -> std::result::Result, RuntimeBootstrapFailure> { + let Some(resolved) = resolve_host_orchestrator_bootstrap(config)? else { + return Ok(None); + }; + Ok(Some(prepare_host_orchestrator_runtime_from_resolved( + resolved, + )?)) +} + +fn prepare_host_orchestrator_runtime_from_resolved( + resolved: ResolvedHostOrchestratorBootstrap, +) -> std::result::Result { + let ResolvedHostOrchestratorBootstrap { + cwd, + shell_session_id, + descriptor, + gateway, + agent_kind, + state_store, + effective_config, + #[cfg(any(target_os = "linux", target_os = "macos"))] + base_policy, + inventory, + } = resolved; + + let participant_id = format!("ash_{}", Uuid::now_v7()); + let lease_token = Uuid::now_v7().to_string(); + let run_id = Uuid::now_v7().to_string(); + let orchestration_session_id = Uuid::now_v7().to_string(); + let mut manifest = AgentRuntimeSessionManifest::new( + &descriptor, + orchestration_session_id.clone(), + participant_id, + lease_token, + ); + manifest.internal.latest_run_id = Some(run_id.clone()); + let orchestration_session = Arc::new(Mutex::new(OrchestrationSessionRecord::new( + orchestration_session_id, + shell_session_id, + cwd.display().to_string(), + &manifest, + ))); + state_store + .persist_orchestration_session( + &orchestration_session + .lock() + .expect("orchestration session mutex poisoned"), + ) + .map_err(|err| RuntimeBootstrapFailure { + exit_code: 1, + message: format!("failed to persist orchestration session record: {err:#}"), + })?; + + Ok(PreparedAgentRuntime { + descriptor, + gateway, + agent_kind, + startup_context: RuntimeOrchestrationContext { + store: state_store, + orchestration_session, + effective_config, + #[cfg(any(target_os = "linux", target_os = "macos"))] + base_policy, + inventory, + }, + manifest: Arc::new(Mutex::new(manifest)), + run_id, + startup_extensions: BTreeMap::new(), + }) +} + +fn resolve_host_orchestrator_bootstrap( + config: &Arc, +) -> std::result::Result, RuntimeBootstrapFailure> { + let cwd = std::env::current_dir().unwrap_or_else(|_| PathBuf::from(".")); + let effective_config = crate::execution::config_model::resolve_effective_config( + &cwd, + &crate::execution::config_model::CliConfigOverrides { + world_enabled: if config.cli_world { + Some(true) + } else if config.cli_no_world { + Some(false) + } else { + None + }, + anchor_mode: config.cli_anchor_mode, + anchor_path: config + .cli_anchor_path + .as_ref() + .map(|path| path.to_string_lossy().to_string()), + caged: config.cli_caged, + }, + ) + .map_err(runtime_bootstrap_failure_from_anyhow)?; + if !effective_config.agents.enabled { + return Ok(None); + } + + let (base_policy, _) = substrate_broker::resolve_effective_policy_with_explain(&cwd, false) + .map_err(|err| RuntimeBootstrapFailure { + exit_code: 2, + message: err.to_string(), + })?; + let inventory = load_effective_agent_inventory(&cwd, &base_policy) + .map_err(runtime_bootstrap_failure_from_anyhow)?; + let orchestrator = + validate_orchestrator_selection(&effective_config, &inventory).map_err(|reason| { + RuntimeBootstrapFailure { + exit_code: 2, + message: reason, + } + })?; + + if !backend_allowed(&base_policy, &orchestrator.derived_backend_id()) { + return Err(RuntimeBootstrapFailure { + exit_code: 5, + message: format!( + "selected orchestrator backend '{}' is not allowlisted by effective policy agents.allowed_backends", + orchestrator.derived_backend_id() + ), + }); + } + + let descriptor = + validate_runtime_realizability(orchestrator, &effective_config).map_err(|err| { + RuntimeBootstrapFailure { + exit_code: runtime_realizability_error_exit_code(&err), + message: err.reason, } })?; let state_store = AgentRuntimeStateStore::new().map_err(|err| RuntimeBootstrapFailure { @@ -1333,608 +2241,3845 @@ async fn start_host_orchestrator_runtime( message: format!("failed to resolve runtime backend kind: {err}"), })?; - let session_handle_id = format!("ash_{}", Uuid::now_v7()); - let lease_token = Uuid::now_v7().to_string(); - let run_id = Uuid::now_v7().to_string(); - let mut manifest = AgentRuntimeSessionManifest::new( - &descriptor, - orchestration_session_id(), - session_handle_id, - lease_token, - ); - manifest.internal.latest_run_id = Some(run_id.clone()); - state_store - .persist_manifest(&manifest) + Ok(Some(ResolvedHostOrchestratorBootstrap { + cwd, + shell_session_id: config.session_id.clone(), + descriptor, + gateway, + agent_kind, + state_store, + effective_config, + #[cfg(any(target_os = "linux", target_os = "macos"))] + base_policy, + inventory, + })) +} + +fn owner_helper_shell_config(plan: &HiddenOwnerHelperLaunchPlan) -> Result { + let cwd = + std::env::current_dir().unwrap_or_else(|_| PathBuf::from(&plan.session.workspace_root)); + let substrate_home = substrate_paths::substrate_home()?; + let trace_log_file = env::var("SHIM_TRACE_LOG") + .map(PathBuf::from) + .unwrap_or_else(|_| substrate_home.join("trace.jsonl")); + + Ok(ShellConfig { + mode: crate::execution::ShellMode::Interactive { use_pty: false }, + session_id: plan.session.shell_trace_session_id.clone(), + trace_log_file, + original_path: env::var("PATH").unwrap_or_default(), + shim_dir: substrate_home.join("shim-bin"), + shell_path: env::var("SHELL").unwrap_or_else(|_| "/bin/sh".to_string()), + ci_mode: false, + no_exit_on_error: false, + skip_shims: true, + no_world: false, + cli_world: false, + cli_no_world: false, + cli_anchor_mode: None, + cli_anchor_path: None, + cli_caged: Some(true), + world_root: WorldRootSettings { + mode: WorldRootMode::Project, + path: cwd, + caged: true, + }, + async_repl: false, + repl_host_escape: false, + env_vars: HashMap::new(), + manager_init_path: substrate_home.join("manager_init.sh"), + manager_env_path: substrate_home.join("manager_env.sh"), + shimmed_path: None, + host_bash_env: None, + bash_preexec_path: substrate_home.join(".substrate_preexec"), + preexec_available: false, + }) +} + +fn owner_helper_runtime_descriptor( + descriptor: &ResolvedRuntimeDescriptor, +) -> std::result::Result { + descriptor + .try_into() .map_err(|err| RuntimeBootstrapFailure { exit_code: 1, - message: format!("failed to persist agent runtime manifest: {err:#}"), - })?; + message: format!("failed to decode hidden owner-helper runtime descriptor: {err:#}"), + }) +} - emit_runtime_event( - build_runtime_message_event( - &manifest, - run_id.clone(), - MessageEventKind::Registered, - "shell-owned orchestrator session handle allocated", - ), - telemetry, - agent_printer, - ); - emit_runtime_event( - build_runtime_message_event( - &manifest, - run_id.clone(), - MessageEventKind::TaskStart, - "starting long-lived shell-owned orchestrator control turn", +fn owner_helper_startup_extensions( + plan: &HiddenOwnerHelperLaunchPlan, +) -> std::result::Result, RuntimeBootstrapFailure> { + match plan.mode { + OwnerHelperMode::Start => Ok(BTreeMap::new()), + OwnerHelperMode::Resume | OwnerHelperMode::ResumeOneTurn | OwnerHelperMode::Fork => { + let session_id = plan + .participant + .internal_uaa_session_id + .as_deref() + .ok_or_else(|| RuntimeBootstrapFailure { + exit_code: 1, + message: format!( + "hidden owner-helper {} requires internal.uaa_session_id", + plan.mode.as_str() + ), + })?; + Ok(BTreeMap::from([( + AGENT_API_SESSION_RESUME_V1.to_string(), + build_session_resume_extension(session_id), + )])) + } + } +} + +fn owner_helper_manifest( + descriptor: &RuntimeSelectionDescriptor, + plan: &HiddenOwnerHelperLaunchPlan, +) -> std::result::Result { + let participant_id = plan.participant.participant_id.clone(); + let lease_token = plan.participant.lease_token.clone(); + let manifest = match plan.mode { + OwnerHelperMode::Start => AgentRuntimeSessionManifest::new_orchestrator_participant( + descriptor, + plan.session.orchestration_session_id.clone(), + participant_id, + lease_token, ), - telemetry, - agent_printer, + OwnerHelperMode::Resume | OwnerHelperMode::ResumeOneTurn | OwnerHelperMode::Fork => { + let resumed_from_participant_id = plan + .participant + .resumed_from_participant_id + .clone() + .ok_or_else(|| RuntimeBootstrapFailure { + exit_code: 1, + message: format!( + "hidden owner-helper {} requires resumed_from_participant_id", + plan.mode.as_str() + ), + })?; + AgentRuntimeSessionManifest::new_replacement_participant( + descriptor, + AgentRuntimeReplacementParticipantInit { + orchestration_session_id: plan.session.orchestration_session_id.clone(), + participant_id, + role: ORCHESTRATOR_ROLE.to_string(), + orchestrator_participant_id: None, + parent_participant_id: None, + resumed_from_participant_id, + world: None, + lease_token, + }, + ) + } + } + .map_err(|err| RuntimeBootstrapFailure { + exit_code: 1, + message: format!("failed to construct hidden owner-helper manifest: {err:#}"), + })?; + + let mut manifest = manifest; + manifest.internal.latest_run_id = Some(plan.participant.run_id.clone()); + Ok(manifest) +} + +fn owner_helper_orchestration_session( + plan: &HiddenOwnerHelperLaunchPlan, + manifest: &AgentRuntimeSessionManifest, +) -> OrchestrationSessionRecord { + let mut session = OrchestrationSessionRecord::new( + plan.session.orchestration_session_id.clone(), + plan.session.shell_trace_session_id.clone(), + plan.session.workspace_root.clone(), + manifest, ); + session.latest_run_id = Some(plan.participant.run_id.clone()); + session.active_session_handle_id = None; + session.shell_owner_pid = std::process::id(); + if let (Some(world_id), Some(world_generation)) = + (&plan.session.world_id, plan.session.world_generation) + { + session.world_id = Some(world_id.clone()); + session.world_generation = Some(world_generation); + } + if plan.startup_prompt.is_some() { + session.initialize_startup_prompt(manifest.handle.participant_id.clone()); + } + session +} - let request = agent_api::AgentWrapperRunRequest { - prompt: "Enter persistent Substrate host orchestrator mode. Keep this control session attached for the lifetime of the host REPL and do not exit until the client cancels the run.".to_string(), - working_dir: Some(cwd), - timeout: None, - env: BTreeMap::new(), - extensions: BTreeMap::new(), - }; - let control = gateway - .run_control(&agent_kind, request) - .await - .map_err(runtime_bootstrap_failure_from_wrapper_error)?; - let agent_api::AgentWrapperRunControl { handle, cancel } = control; - let agent_api::AgentWrapperRunHandle { events, completion } = handle; - let manifest = Arc::new(Mutex::new(manifest)); - let shutdown_requested = Arc::new(AtomicBool::new(false)); - let (startup_tx, startup_rx) = tokio::sync::oneshot::channel::(); - let startup_signal = Arc::new(Mutex::new(Some(startup_tx))); - let agent_id = { - manifest - .lock() - .expect("runtime manifest mutex poisoned") - .handle - .agent_id - .clone() +fn prepare_hidden_owner_helper_runtime( + config: &Arc, + plan: &HiddenOwnerHelperLaunchPlan, +) -> std::result::Result { + let Some(resolved) = resolve_host_orchestrator_bootstrap(config)? else { + return Err(RuntimeBootstrapFailure { + exit_code: 2, + message: "hidden owner-helper requires agents.enabled=true".to_string(), + }); }; - let run_id_for_tasks = run_id.clone(); - let event_store = state_store.clone(); - let event_manifest = Arc::clone(&manifest); - let startup_signal_for_events = Arc::clone(&startup_signal); - let shutdown_for_events = Arc::clone(&shutdown_requested); - let mut events = events; - let event_task = tokio::spawn(async move { - while let Some(wrapper_event) = events.next().await { - let mut startup_became_live = false; - let (snapshot, event) = { - let mut guard = event_manifest - .lock() - .expect("runtime manifest mutex poisoned"); - if let Some(session_id) = extract_session_handle_id(wrapper_event.data.as_ref()) { - if guard.internal.uaa_session_id.as_deref() != Some(session_id) { - guard.set_uaa_session_id(session_id.to_string()); - } - if guard.handle.state == AgentRuntimeSessionState::Allocating - && guard.can_advertise_live() - { - guard.transition_state(AgentRuntimeSessionState::Ready); - guard.touch_heartbeat(); - startup_became_live = true; - } - } else if guard.handle.state == AgentRuntimeSessionState::Ready - && guard.can_advertise_live() - { - guard.transition_state(AgentRuntimeSessionState::Running); + let ResolvedHostOrchestratorBootstrap { + cwd: _cwd, + shell_session_id: _shell_session_id, + descriptor: validated_descriptor, + gateway: _validated_gateway, + agent_kind: _validated_agent_kind, + state_store, + effective_config, + #[cfg(any(target_os = "linux", target_os = "macos"))] + base_policy, + inventory, + } = resolved; + let descriptor = owner_helper_runtime_descriptor(&plan.descriptor)?; + if validated_descriptor.backend_id != descriptor.backend_id { + return Err(RuntimeBootstrapFailure { + exit_code: 2, + message: format!( + "hidden owner-helper descriptor mismatch: validated backend '{}' does not match resolved backend '{}'", + validated_descriptor.backend_id, descriptor.backend_id + ), + }); + } + + let manifest = owner_helper_manifest(&descriptor, plan)?; + let orchestration_session = Arc::new(Mutex::new(owner_helper_orchestration_session( + plan, &manifest, + ))); + state_store + .persist_orchestration_session( + &orchestration_session + .lock() + .expect("hidden owner-helper orchestration session mutex poisoned"), + ) + .map_err(|err| RuntimeBootstrapFailure { + exit_code: 1, + message: format!("failed to persist hidden owner-helper session record: {err:#}"), + })?; + + let gateway = + build_gateway_for_descriptor(&descriptor).map_err(|err| RuntimeBootstrapFailure { + exit_code: 1, + message: format!("failed to build hidden owner-helper gateway: {err:#}"), + })?; + let agent_kind = agent_api::AgentWrapperKind::new(descriptor.backend_kind.as_agent_kind_str()) + .map_err(|err| RuntimeBootstrapFailure { + exit_code: 2, + message: format!("failed to resolve hidden owner-helper backend kind: {err}"), + })?; + Ok(PreparedAgentRuntime { + descriptor, + gateway, + agent_kind, + startup_context: RuntimeOrchestrationContext { + store: state_store, + orchestration_session, + effective_config, + #[cfg(any(target_os = "linux", target_os = "macos"))] + base_policy, + inventory, + }, + manifest: Arc::new(Mutex::new(manifest)), + run_id: plan.participant.run_id.clone(), + startup_extensions: owner_helper_startup_extensions(plan)?, + }) +} + +async fn wait_for_hidden_owner_helper_completion( + runtime: AsyncReplAgentRuntime, + agent_printer: &ReplPrinter, + telemetry: &mut ReplSessionTelemetry, +) -> Result { + let AsyncReplAgentRuntime { + descriptor, + orchestration_session, + manifest, + store, + uaa_session_handle_id, + mut retained_control, + shutdown_requested, + mut auto_park_rx, + mut private_stop_rx, + mut stop_transport, + mut stop_owner_task, + mut prompt_transport, + mut prompt_owner_task, + mut heartbeat_stop_tx, + mut heartbeat_task, + } = runtime; + + let mut join_failed = false; + + match &mut retained_control { + RetainedRunControl::Local(retained_control) => { + let outcome = wait_for_hidden_owner_helper_local_runtime( + HiddenOwnerHelperLocalRuntimeContext { + descriptor: descriptor.clone(), + store: &store, + orchestration_session: &orchestration_session, + manifest: &manifest, + uaa_session_handle_id: uaa_session_handle_id.as_str(), + shutdown_requested: &shutdown_requested, + auto_park_rx: &mut auto_park_rx, + private_stop_rx: &mut private_stop_rx, + stop_transport: &mut stop_transport, + stop_owner_task: &mut stop_owner_task, + prompt_transport: &mut prompt_transport, + prompt_owner_task: &mut prompt_owner_task, + heartbeat_stop_tx: &mut heartbeat_stop_tx, + heartbeat_task: &mut heartbeat_task, + }, + retained_control, + agent_printer, + telemetry, + ) + .await; + match outcome { + HiddenOwnerHelperLocalRuntimeOutcome::Joined { + join_failed: local_failed, + } => { + join_failed |= local_failed; } - let event = translate_wrapper_event(&guard, &run_id_for_tasks, wrapper_event); - guard.touch_event(event.ts); - if guard.handle.state == AgentRuntimeSessionState::Ready - && guard.can_advertise_live() - && !startup_became_live - { - guard.transition_state(AgentRuntimeSessionState::Running); + HiddenOwnerHelperLocalRuntimeOutcome::AutoParked => { + return Ok(hidden_owner_helper_exit_code(&manifest)); } - (guard.clone(), event) - }; - let _ = event_store.persist_manifest(&snapshot); - let _ = publish_agent_event(event); - - if startup_became_live { - let _ = publish_agent_event(build_runtime_message_event( - &snapshot, - run_id_for_tasks.clone(), - MessageEventKind::Status, - "shell-owned orchestrator session is ready via retained attached control ownership", - )); - signal_runtime_startup(&startup_signal_for_events, RuntimeStartupSignal::Running); } } - - let mut publish_events = Vec::new(); - let mut startup_failure: Option = None; - let snapshot = { - let mut guard = event_manifest - .lock() - .expect("runtime manifest mutex poisoned"); - let was_allocating = guard.handle.state == AgentRuntimeSessionState::Allocating; - let was_live = guard.is_authoritative_live(); - guard.set_event_stream_active(false); - if was_allocating { - let reason = - "attached control turn ended before ownership could be established".to_string(); - guard.transition_state(AgentRuntimeSessionState::Failed); - guard.mark_terminal_state(reason.clone()); - guard.internal.last_error_bucket = Some("bootstrap_run".to_string()); - guard.internal.last_error_message = Some(reason.clone()); - startup_failure = Some(reason); - } else if !shutdown_for_events.load(Ordering::SeqCst) && was_live { - let reason = - "shell-owned orchestrator control stream ended before completion observation" - .to_string(); - guard.internal.last_error_bucket = Some("runtime_lifecycle".to_string()); - guard.internal.last_error_message = Some(reason.clone()); - publish_events.push(AgentEvent::alert( - guard.handle.agent_id.clone(), - guard.handle.orchestration_session_id.clone(), - run_id_for_tasks.clone(), - "orchestrator_runtime_stream_closed", - reason, - )); + #[cfg(any(target_os = "linux", target_os = "macos"))] + RetainedRunControl::Remote(retained_control) => { + if let Some(mut auto_park_requests) = auto_park_rx.take() { + if let Some(mut observe_task) = retained_control.observe_task.take() { + tokio::select! { + result = &mut observe_task => { + join_failed |= result.is_err(); + } + maybe_auto_park = auto_park_requests.recv() => { + if maybe_auto_park.is_some() { + retained_control.observe_task = Some(observe_task); + shutdown_host_orchestrator_runtime_with_mode( + AsyncReplAgentRuntime { + descriptor: descriptor.clone(), + orchestration_session: Arc::clone(&orchestration_session), + manifest: Arc::clone(&manifest), + store: store.clone(), + uaa_session_handle_id: uaa_session_handle_id.clone(), + retained_control: RetainedRunControl::Remote(RemoteRetainedRunControl { + client: Arc::clone(&retained_control.client), + span_id: retained_control.span_id.clone(), + observe_task: retained_control.observe_task.take(), + }), + shutdown_requested: Arc::clone(&shutdown_requested), + auto_park_rx: Some(auto_park_requests), + private_stop_rx, + stop_transport, + stop_owner_task, + prompt_transport, + prompt_owner_task, + heartbeat_stop_tx, + heartbeat_task, + }, + HostRuntimeShutdownMode::ParkIfResumable, + agent_printer, + telemetry, + ) + .await; + return Ok(hidden_owner_helper_exit_code(&manifest)); + } + join_failed |= observe_task.await.is_err(); + } + } + } + } else if let Some(task) = retained_control.observe_task.take() { + join_failed |= task.await.is_err(); } - guard.clone() - }; - let _ = event_store.persist_manifest(&snapshot); - for event in publish_events { - let _ = publish_agent_event(event); - } - if let Some(message) = startup_failure { - signal_runtime_startup( - &startup_signal_for_events, - RuntimeStartupSignal::Failed(message), - ); } - }); + } - let completion_store = state_store.clone(); - let completion_manifest = Arc::clone(&manifest); - let startup_signal_for_completion = Arc::clone(&startup_signal); - let shutdown_for_completion = Arc::clone(&shutdown_requested); - let run_id_for_completion = run_id.clone(); - let completion_task = tokio::spawn(async move { - let completion = completion.await; - let shutdown_requested = shutdown_for_completion.load(Ordering::SeqCst); - let mut startup_failure: Option = None; - let mut publish_events = Vec::new(); + if let Some(mut stop_transport) = stop_transport.take() { + stop_transport.close().await; + } + if let Some(task) = stop_owner_task.take() { + let _ = task.await; + } + if let Some(mut prompt_transport) = prompt_transport.take() { + prompt_transport.close().await; + } + if let Some(task) = prompt_owner_task.take() { + let _ = task.await; + } + if let Some(stop_tx) = heartbeat_stop_tx.take() { + let _ = stop_tx.send(()); + } + if let Some(task) = heartbeat_task.take() { + let _ = task.await; + } - let snapshot = { - let mut guard = completion_manifest - .lock() - .expect("runtime manifest mutex poisoned"); + Ok(if join_failed { + 1 + } else { + hidden_owner_helper_exit_code(&manifest) + }) +} - match completion { - Ok(completion) => { - if guard.internal.uaa_session_id.is_none() { - if let Some(session_id) = - extract_session_handle_id(completion.data.as_ref()) - { - guard.set_uaa_session_id(session_id.to_string()); - } - } +fn hidden_owner_helper_exit_code(manifest: &Arc>) -> i32 { + let state = manifest + .lock() + .expect("hidden owner-helper manifest mutex poisoned") + .handle + .state + .clone(); + if matches!( + state, + AgentRuntimeSessionState::Failed | AgentRuntimeSessionState::Invalidated + ) { + 1 + } else { + 0 + } +} - if guard.handle.state == AgentRuntimeSessionState::Allocating { - let reason = if completion.status.success() { - "attached control turn ended before ownership could be established" - .to_string() - } else { - format!( - "attached control turn exited with status {} before ownership was established", - completion.status.code().unwrap_or(-1) - ) - }; - guard.transition_state(AgentRuntimeSessionState::Failed); - guard.mark_terminal_state(reason.clone()); - guard.internal.last_error_bucket = Some("bootstrap_run".to_string()); - guard.internal.last_error_message = Some(reason.clone()); - startup_failure = Some(reason); - } else if shutdown_requested - && guard.handle.state == AgentRuntimeSessionState::Stopping - { - guard.transition_state(AgentRuntimeSessionState::Stopped); - guard.mark_terminal_state("stopped"); - guard.touch_heartbeat(); - } else if shutdown_requested - && guard.handle.state == AgentRuntimeSessionState::Failed - { - // Preserve bootstrap failure when startup cleanup observes completion. - } else if guard.handle.state == AgentRuntimeSessionState::Stopping { - guard.transition_state(AgentRuntimeSessionState::Stopped); - guard.mark_terminal_state("stopped"); - guard.touch_heartbeat(); - } else { - let reason = format!( - "attached control turn exited with status {}", - completion.status.code().unwrap_or(-1) - ); - guard.transition_state(AgentRuntimeSessionState::Invalidated); - guard.mark_terminal_state(reason.clone()); - guard.internal.last_error_bucket = Some("runtime_lifecycle".to_string()); - guard.internal.last_error_message = Some(reason.clone()); - publish_events.push(AgentEvent::alert( - guard.handle.agent_id.clone(), - guard.handle.orchestration_session_id.clone(), - run_id_for_completion.clone(), - "orchestrator_runtime_invalidated", - reason, - )); - } - } - Err(err) => { - let reason = match &err { - agent_api::AgentWrapperError::Backend { message } => message.clone(), - other => other.to_string(), - }; - if guard.handle.state == AgentRuntimeSessionState::Allocating { - let reason = - format!("failed to establish attached control ownership: {reason}"); - guard.transition_state(AgentRuntimeSessionState::Failed); - guard.mark_terminal_state(reason.clone()); - guard.internal.last_error_bucket = Some("bootstrap_run".to_string()); - guard.internal.last_error_message = Some(reason.clone()); - startup_failure = Some(reason); - } else if shutdown_requested - && guard.handle.state == AgentRuntimeSessionState::Stopping - && reason == "cancelled" - { - guard.transition_state(AgentRuntimeSessionState::Stopped); - guard.mark_terminal_state("stopped"); - guard.touch_heartbeat(); - } else if shutdown_requested - && guard.handle.state == AgentRuntimeSessionState::Failed - { - // Preserve bootstrap failure when startup cleanup observes completion. - } else if shutdown_requested { - let reason = format!("failed to stop attached control: {reason}"); - guard.transition_state(AgentRuntimeSessionState::Failed); - guard.mark_terminal_state(reason.clone()); - guard.internal.last_error_bucket = Some("runtime_shutdown".to_string()); - guard.internal.last_error_message = Some(reason); - } else { - let reason = format!("attached control turn ended unexpectedly: {reason}"); - guard.transition_state(AgentRuntimeSessionState::Invalidated); - guard.mark_terminal_state(reason.clone()); - guard.internal.last_error_bucket = Some("runtime_lifecycle".to_string()); - guard.internal.last_error_message = Some(reason.clone()); - publish_events.push(AgentEvent::alert( - guard.handle.agent_id.clone(), - guard.handle.orchestration_session_id.clone(), - run_id_for_completion.clone(), - "orchestrator_runtime_invalidated", - reason, - )); - } +enum HiddenOwnerHelperLocalRuntimeOutcome { + Joined { join_failed: bool }, + AutoParked, +} + +struct HiddenOwnerHelperLocalRuntimeContext<'a> { + descriptor: RuntimeSelectionDescriptor, + store: &'a AgentRuntimeStateStore, + orchestration_session: &'a Arc>, + manifest: &'a Arc>, + uaa_session_handle_id: &'a str, + shutdown_requested: &'a Arc, + auto_park_rx: &'a mut Option>, + private_stop_rx: &'a mut Option, + stop_transport: &'a mut Option, + stop_owner_task: &'a mut Option>, + prompt_transport: &'a mut Option, + prompt_owner_task: &'a mut Option>, + heartbeat_stop_tx: &'a mut Option>, + heartbeat_task: &'a mut Option>, +} + +async fn wait_for_hidden_owner_helper_local_runtime( + context: HiddenOwnerHelperLocalRuntimeContext<'_>, + retained_control: &mut LocalRetainedRunControl, + agent_printer: &ReplPrinter, + telemetry: &mut ReplSessionTelemetry, +) -> HiddenOwnerHelperLocalRuntimeOutcome { + let HiddenOwnerHelperLocalRuntimeContext { + descriptor, + store, + orchestration_session, + manifest, + uaa_session_handle_id, + shutdown_requested, + auto_park_rx, + private_stop_rx, + stop_transport, + stop_owner_task, + prompt_transport, + prompt_owner_task, + heartbeat_stop_tx, + heartbeat_task, + } = context; + + let mut join_failed = false; + let Some(mut completion_task) = retained_control.completion_task.take() else { + return HiddenOwnerHelperLocalRuntimeOutcome::Joined { join_failed: true }; + }; + + let outcome = tokio::select! { + result = &mut completion_task => { + join_failed |= result.is_err(); + HiddenOwnerHelperLocalRuntimeOutcome::Joined { join_failed } + } + maybe_request = async { + match private_stop_rx.as_mut() { + Some(stop_rx) => stop_rx.recv().await, + None => std::future::pending().await, + } + } => { + let request = match maybe_request { + Some(request) => request, + None => { + join_failed |= completion_task.await.is_err(); + return HiddenOwnerHelperLocalRuntimeOutcome::Joined { join_failed }; } + }; + + if runtime_is_terminal(manifest) { + let _ = request.response_tx.send(PrivateStopOutcome::AlreadyTerminal); + join_failed |= completion_task.await.is_err(); + return HiddenOwnerHelperLocalRuntimeOutcome::Joined { join_failed }; } - guard.clone() - }; + if let Err(err) = note_runtime_stop_requested(store, orchestration_session, manifest) { + persist_hidden_owner_helper_stop_failure( + store, + orchestration_session, + manifest, + format!("failed to persist stop request before shutdown: {err:#}"), + ); + let _ = request.response_tx.send(PrivateStopOutcome::ProtocolError); + shutdown_requested.store(true, Ordering::SeqCst); + retained_control.cancel.cancel(); + join_failed |= completion_task.await.is_err(); + return HiddenOwnerHelperLocalRuntimeOutcome::Joined { join_failed }; + } - let _ = completion_store.persist_manifest(&snapshot); - for event in publish_events { - let _ = publish_agent_event(event); - } - if let Some(message) = startup_failure { - signal_runtime_startup( - &startup_signal_for_completion, - RuntimeStartupSignal::Failed(message), - ); - } - }); + let _ = request.response_tx.send(PrivateStopOutcome::Accepted); + if let Some(mut owned_stop_transport) = stop_transport.take() { + owned_stop_transport.close().await; + } + if let Some(mut owned_prompt_transport) = prompt_transport.take() { + owned_prompt_transport.close().await; + } + if let Some(task) = prompt_owner_task.take() { + let _ = task.await; + } + shutdown_requested.store(true, Ordering::SeqCst); + if let Some(stop_tx) = heartbeat_stop_tx.take() { + let _ = stop_tx.send(()); + } + retained_control.cancel.cancel(); - let (heartbeat_stop_tx, mut heartbeat_stop_rx) = tokio::sync::oneshot::channel(); - let heartbeat_store = state_store.clone(); - let heartbeat_manifest = Arc::clone(&manifest); - let heartbeat_task = tokio::spawn(async move { - let mut interval = tokio::time::interval(Duration::from_secs(5)); - interval.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Delay); - loop { + let mut completion_observed = false; + let mut stop_failed = false; tokio::select! { - _ = interval.tick() => { - let next = { - let mut guard = heartbeat_manifest - .lock() - .expect("runtime manifest mutex poisoned"); - if !guard.is_authoritative_live() { - None - } else { - guard.touch_heartbeat(); - Some(guard.clone()) - } - }; - let Some(next) = next else { - break; - }; - let _ = heartbeat_store.persist_manifest(&next); + result = &mut completion_task => { + completion_observed = true; + join_failed |= result.is_err(); + stop_failed |= result.is_err(); + } + _ = tokio::time::sleep(LOCAL_RETAINED_STOP_COMPLETION_TIMEOUT) => { + completion_task.abort(); + let _ = completion_task.await; + stop_failed = true; } - _ = &mut heartbeat_stop_rx => break, + } + + if let Some(task) = retained_control.event_task.take() { + match tokio::time::timeout(LOCAL_RETAINED_STOP_COMPLETION_TIMEOUT, task).await { + Ok(_) => {} + Err(_) => join_failed = true, + } + } + if let Some(task) = heartbeat_task.take() { + let _ = task.await; + } + + if stop_failed || !completion_observed { + persist_hidden_owner_helper_stop_failure( + store, + orchestration_session, + manifest, + "hidden owner-helper stop did not produce authoritative terminal completion" + .to_string(), + ); + } + HiddenOwnerHelperLocalRuntimeOutcome::Joined { join_failed } + } + maybe_auto_park = async { + match auto_park_rx.as_mut() { + Some(requests) => requests.recv().await, + None => std::future::pending().await, + } + } => { + if maybe_auto_park.is_some() { + retained_control.completion_task = Some(completion_task); + shutdown_host_orchestrator_runtime_with_mode( + AsyncReplAgentRuntime { + descriptor, + orchestration_session: Arc::clone(orchestration_session), + manifest: Arc::clone(manifest), + store: store.clone(), + uaa_session_handle_id: uaa_session_handle_id.to_string(), + retained_control: RetainedRunControl::Local(LocalRetainedRunControl { + cancel: retained_control.cancel.clone(), + event_task: retained_control.event_task.take(), + completion_task: retained_control.completion_task.take(), + }), + shutdown_requested: Arc::clone(shutdown_requested), + auto_park_rx: auto_park_rx.take(), + private_stop_rx: private_stop_rx.take(), + stop_transport: stop_transport.take(), + stop_owner_task: stop_owner_task.take(), + prompt_transport: prompt_transport.take(), + prompt_owner_task: prompt_owner_task.take(), + heartbeat_stop_tx: heartbeat_stop_tx.take(), + heartbeat_task: heartbeat_task.take(), + }, + HostRuntimeShutdownMode::ParkIfResumable, + agent_printer, + telemetry, + ) + .await; + HiddenOwnerHelperLocalRuntimeOutcome::AutoParked + } else { + join_failed |= completion_task.await.is_err(); + HiddenOwnerHelperLocalRuntimeOutcome::Joined { join_failed } } } - }); - let retained_snapshot = { - let mut guard = manifest.lock().expect("runtime manifest mutex poisoned"); - guard.mark_runtime_ownership_retained(); - guard.clone() - }; - state_store - .persist_manifest(&retained_snapshot) - .map_err(|err| RuntimeBootstrapFailure { - exit_code: 1, - message: format!("failed to persist retained runtime ownership: {err:#}"), - })?; - let mut retained_control = RetainedRunControl { - cancel, - event_task: Some(event_task), - completion_task: Some(completion_task), }; - let mut heartbeat_stop_tx = Some(heartbeat_stop_tx); - let mut heartbeat_task = Some(heartbeat_task); - match tokio::time::timeout(Duration::from_secs(10), startup_rx).await { - Ok(Ok(RuntimeStartupSignal::Running)) => {} - Ok(Ok(RuntimeStartupSignal::Failed(message))) => { - abort_bootstrap_runtime( - &shutdown_requested, - &mut retained_control, - &mut heartbeat_stop_tx, - &mut heartbeat_task, - ) - .await; - return Err(RuntimeBootstrapFailure { - exit_code: 4, - message, - }); - } - Ok(Err(_)) => { - abort_bootstrap_runtime( - &shutdown_requested, - &mut retained_control, - &mut heartbeat_stop_tx, - &mut heartbeat_task, - ) - .await; - mark_runtime_startup_failed( - &state_store, - &manifest, - "failed to establish attached control ownership", - ); - return Err(RuntimeBootstrapFailure { - exit_code: 1, - message: "failed to establish attached control ownership".to_string(), - }); - } - Err(_) => { - abort_bootstrap_runtime( - &shutdown_requested, - &mut retained_control, - &mut heartbeat_stop_tx, - &mut heartbeat_task, - ) - .await; - let message = format!( - "timed out waiting for shell-owned orchestrator control ownership: {agent_id}" - ); - mark_runtime_startup_failed(&state_store, &manifest, &message); - return Err(RuntimeBootstrapFailure { - exit_code: 4, - message, - }); + if let HiddenOwnerHelperLocalRuntimeOutcome::Joined { .. } = outcome { + if let Some(task) = retained_control.event_task.take() { + let _ = task.await; } } - let uaa_session_handle_id = manifest - .lock() - .expect("runtime manifest mutex poisoned") - .internal - .uaa_session_id - .clone() - .ok_or_else(|| RuntimeBootstrapFailure { - exit_code: 1, - message: "runtime startup signalled ready without a surfaced UAA session handle" - .to_string(), - })?; - Ok(Some(AsyncReplAgentRuntime { - manifest, - store: state_store, - uaa_session_handle_id, - retained_control, - shutdown_requested, - heartbeat_stop_tx, - heartbeat_task, - })) + outcome } -async fn shutdown_host_orchestrator_runtime( - mut runtime: AsyncReplAgentRuntime, - agent_printer: &ReplPrinter, - telemetry: &mut ReplSessionTelemetry, +fn persist_hidden_owner_helper_stop_failure( + store: &AgentRuntimeStateStore, + orchestration_session: &Arc>, + manifest: &Arc>, + reason: String, ) { - let (run_id, should_attempt_stop) = { - let guard = runtime - .manifest + let (orchestration_snapshot, manifest_snapshot) = { + let mut orchestration_guard = orchestration_session .lock() - .expect("runtime manifest mutex poisoned"); - ( - guard - .internal - .latest_run_id - .clone() - .unwrap_or_else(|| Uuid::now_v7().to_string()), - guard.internal.control_owner_retained || guard.internal.completion_observer_retained, - ) + .expect("orchestration session mutex poisoned"); + let mut manifest_guard = manifest.lock().expect("runtime manifest mutex poisoned"); + + if !manifest_guard.handle.state.is_live() { + return; + } + + manifest_guard.transition_state(AgentRuntimeSessionState::Failed); + manifest_guard.mark_terminal_state(reason.clone()); + manifest_guard.internal.last_error_bucket = Some("runtime_shutdown".to_string()); + manifest_guard.internal.last_error_message = Some(reason.clone()); + let _ = terminalize_startup_prompt_failure( + &mut orchestration_guard, + &manifest_guard, + reason.clone(), + ); + if runtime_controls_parent_session(&manifest_guard.handle.role) { + orchestration_guard.transition_state(OrchestrationSessionState::Failed); + orchestration_guard.mark_terminal(reason); + } else { + orchestration_guard.touch_active(); + } + (orchestration_guard.clone(), manifest_guard.clone()) }; - if should_attempt_stop { - let manifest = { - let mut guard = runtime - .manifest - .lock() - .expect("runtime manifest mutex poisoned"); - if guard.handle.state.is_live() { - guard.transition_state(AgentRuntimeSessionState::Stopping); - guard.touch_heartbeat(); + + let _ = persist_runtime_snapshots(store, &orchestration_snapshot, &manifest_snapshot); +} + +pub(crate) fn run_hidden_owner_helper(plan: HiddenOwnerHelperLaunchPlan) -> Result { + let config = Arc::new(owner_helper_shell_config(&plan)?); + let running_child_pid = Arc::new(AtomicI32::new(0)); + setup_signal_handlers(running_child_pid)?; + let rt = TokioRuntimeBuilder::new_current_thread() + .enable_all() + .build() + .context("failed to initialize hidden owner-helper runtime")?; + + rt.block_on(async move { + let mut telemetry = ReplSessionTelemetry::new(config.clone(), "agent_owner_helper"); + let prepared = prepare_hidden_owner_helper_runtime(&config, &plan) + .map_err(|failure| anyhow!(failure.message))?; + let initial_prompt = plan.startup_prompt.as_ref().map(|startup_prompt| { + InitialExecPromptPlan::StartupPrompt { + prompt: startup_prompt.prompt_text.clone(), + stream_path: startup_prompt.stream_path.clone(), } - guard.clone() + }); + let runtime = start_host_orchestrator_runtime_with_prepared_prompt( + Some(prepared), + None, + initial_prompt, + true, + matches!(plan.mode, OwnerHelperMode::ResumeOneTurn), + &ReplPrinter::Stdout, + &mut telemetry, + ) + .await + .map_err(|failure| anyhow!(failure.message))?; + let Some(runtime) = runtime else { + return Ok(0); }; - let _ = runtime.store.persist_manifest(&manifest); - emit_runtime_event( - build_runtime_message_event( - &manifest, - run_id.clone(), - MessageEventKind::Status, - format!( - "shell-owned orchestrator session stopping (uaa_session_handle_id={})", - runtime.uaa_session_handle_id - ), - ), - telemetry, - agent_printer, - ); - } + wait_for_hidden_owner_helper_completion(runtime, &ReplPrinter::Stdout, &mut telemetry).await + }) +} - runtime.shutdown_requested.store(true, Ordering::SeqCst); - if let Some(stop_tx) = runtime.heartbeat_stop_tx.take() { - let _ = stop_tx.send(()); - } - runtime.retained_control.cancel.cancel(); +async fn start_host_orchestrator_runtime_with_prepared( + prepared: Option, + initial_world_binding: Option<&PersistedWorldBinding>, + agent_printer: &ReplPrinter, + telemetry: &mut ReplSessionTelemetry, +) -> std::result::Result, RuntimeBootstrapFailure> { + start_host_orchestrator_runtime_with_prepared_prompt( + prepared, + initial_world_binding, + None, + false, + false, + agent_printer, + telemetry, + ) + .await +} - let mut stop_failed = false; - let mut completion_observed = false; - if let Some(task) = runtime.retained_control.completion_task.take() { - match tokio::time::timeout(Duration::from_secs(5), task).await { - Ok(Ok(())) => completion_observed = true, - Ok(Err(_)) | Err(_) => stop_failed = true, - } - } - if let Some(task) = runtime.retained_control.event_task.take() { - let _ = tokio::time::timeout(Duration::from_secs(5), task).await; - } - if let Some(task) = runtime.heartbeat_task.take() { - let _ = task.await; +async fn start_host_orchestrator_runtime_with_prepared_prompt( + prepared: Option, + initial_world_binding: Option<&PersistedWorldBinding>, + initial_prompt: Option, + runtime_owns_private_stop: bool, + auto_park_after_public_turn: bool, + agent_printer: &ReplPrinter, + telemetry: &mut ReplSessionTelemetry, +) -> std::result::Result, RuntimeBootstrapFailure> { + let Some(prepared) = prepared else { + return Ok(None); + }; + let PreparedAgentRuntime { + descriptor, + gateway, + agent_kind, + startup_context, + manifest, + run_id, + startup_extensions, + } = prepared; + let runtime_role = { + manifest + .lock() + .expect("runtime manifest mutex poisoned") + .handle + .role + .clone() + }; + let controls_parent_session = runtime_controls_parent_session(&runtime_role); + if let Err(err) = persist_world_binding_authority( + &startup_context.store, + &startup_context.orchestration_session, + initial_world_binding, + ) { + mark_orchestration_session_failed( + &startup_context.store, + &startup_context.orchestration_session, + format!("failed to persist startup world binding: {err:#}"), + ); + return Err(RuntimeBootstrapFailure { + exit_code: 1, + message: format!("failed to persist startup world binding: {err:#}"), + }); } - if stop_failed || (should_attempt_stop && !completion_observed) { - let reason = - "shell-owned orchestrator cancel did not produce authoritative terminal completion" - .to_string(); - let manifest = { - let mut guard = runtime - .manifest - .lock() - .expect("runtime manifest mutex poisoned"); - guard.transition_state(AgentRuntimeSessionState::Failed); - guard.mark_terminal_state(reason.clone()); - guard.internal.last_error_bucket = Some("runtime_shutdown".to_string()); - guard.internal.last_error_message = Some(reason.clone()); - guard.clone() - }; - let _ = runtime.store.persist_manifest(&manifest); - emit_runtime_event( - AgentEvent::alert( - manifest.handle.agent_id.clone(), - manifest.handle.orchestration_session_id.clone(), - run_id, - "orchestrator_runtime_invalidated", - reason, - ), - telemetry, - agent_printer, + let persist_participant_result = { + let manifest_guard = manifest.lock().expect("runtime manifest mutex poisoned"); + startup_context.store.persist_participant(&manifest_guard) + }; + if let Err(err) = persist_participant_result { + mark_orchestration_session_failed( + &startup_context.store, + &startup_context.orchestration_session, + format!("failed to persist agent runtime participant record: {err:#}"), ); - return; + return Err(RuntimeBootstrapFailure { + exit_code: 1, + message: format!("failed to persist agent runtime participant record: {err:#}"), + }); } - - let manifest = runtime - .manifest + let orchestration_snapshot = startup_context + .orchestration_session .lock() - .expect("runtime manifest mutex poisoned") + .expect("orchestration session mutex poisoned") .clone(); - if should_attempt_stop && manifest.handle.state == AgentRuntimeSessionState::Stopped { - emit_runtime_event( - build_runtime_message_event( - &manifest, - run_id, - MessageEventKind::Status, - "shell-owned orchestrator session stopped", - ), - telemetry, - agent_printer, - ); - } -} - -async fn abort_bootstrap_runtime( - shutdown_requested: &Arc, - retained_control: &mut RetainedRunControl, - heartbeat_stop_tx: &mut Option>, - heartbeat_task: &mut Option>, -) { - shutdown_requested.store(true, Ordering::SeqCst); - if let Some(stop_tx) = heartbeat_stop_tx.take() { - let _ = stop_tx.send(()); - } - retained_control.cancel.cancel(); - if let Some(task) = retained_control.completion_task.take() { - let _ = task.await; - } - if let Some(task) = retained_control.event_task.take() { - let _ = task.await; - } - if let Some(task) = heartbeat_task.take() { - let _ = task.await; - } -} - -fn signal_runtime_startup( - signal: &Arc>>>, - value: RuntimeStartupSignal, -) { - if let Some(tx) = signal + let manifest_snapshot = manifest .lock() - .expect("runtime startup signal mutex poisoned") - .take() - { - let _ = tx.send(value); - } -} + .expect("runtime manifest mutex poisoned") + .clone(); + emit_runtime_event( + build_runtime_message_event( + &manifest_snapshot, + &orchestration_snapshot, + run_id.clone(), + MessageEventKind::Registered, + runtime_registered_message(&runtime_role), + ), + telemetry, + agent_printer, + ); + emit_runtime_event( + build_runtime_message_event( + &manifest_snapshot, + &orchestration_snapshot, + run_id.clone(), + MessageEventKind::TaskStart, + runtime_task_start_message(&runtime_role), + ), + telemetry, + agent_printer, + ); -fn mark_runtime_startup_failed( - store: &AgentRuntimeStateStore, - manifest: &Arc>, - message: &str, -) { - let snapshot = { - let mut guard = manifest.lock().expect("runtime manifest mutex poisoned"); - if guard.handle.state == AgentRuntimeSessionState::Allocating { - guard.transition_state(AgentRuntimeSessionState::Failed); - } - if !guard.has_valid_ownership() { - guard.mark_ownership_invalid(message.to_string()); - } - guard.internal.last_error_bucket = Some("bootstrap_run".to_string()); - guard.internal.last_error_message = Some(message.to_string()); - guard.clone() + let request = agent_api::AgentWrapperRunRequest { + prompt: match initial_prompt.as_ref() { + Some(InitialExecPromptPlan::Replace(prompt)) => prompt.clone(), + Some(InitialExecPromptPlan::StartupPrompt { prompt, .. }) => prompt.clone(), + None => runtime_bootstrap_prompt(&runtime_role).to_string(), + }, + working_dir: Some(std::env::current_dir().unwrap_or_else(|_| PathBuf::from("."))), + timeout: None, + env: BTreeMap::new(), + extensions: startup_extensions, }; - let _ = store.persist_manifest(&snapshot); -} - -fn runtime_bootstrap_failure_from_anyhow(err: anyhow::Error) -> RuntimeBootstrapFailure { - if crate::execution::config_model::is_user_error(&err) { - RuntimeBootstrapFailure { - exit_code: 2, - message: err.to_string(), + let startup_backchannel = match initial_prompt.as_ref() { + Some(InitialExecPromptPlan::StartupPrompt { stream_path, .. }) => { + Some(connect_startup_prompt_backchannel(stream_path).await?) } - } else { - RuntimeBootstrapFailure { + _ => None, + }; + if let Some(backchannel) = startup_backchannel.as_ref() { + let (orchestration_snapshot, manifest_snapshot) = { + let mut orchestration_guard = startup_context + .orchestration_session + .lock() + .expect("orchestration session mutex poisoned"); + let manifest_guard = manifest.lock().expect("runtime manifest mutex poisoned"); + orchestration_guard + .mark_startup_prompt_accepted(manifest_guard.handle.participant_id.as_str()); + (orchestration_guard.clone(), manifest_guard.clone()) + }; + persist_runtime_snapshots( + &startup_context.store, + &orchestration_snapshot, + &manifest_snapshot, + ) + .map_err(|err| RuntimeBootstrapFailure { exit_code: 1, - message: format!("{err:#}"), - } + message: format!("failed to persist startup prompt acceptance: {err:#}"), + })?; + backchannel.send(PublicPromptEnvelope::Accepted { + version: 1, + action: PublicPromptAction::Start, + orchestration_session_id: orchestration_snapshot.orchestration_session_id.clone(), + backend_id: manifest_snapshot.handle.backend_id.clone(), + participant_id: Some(manifest_snapshot.handle.participant_id.clone()), + scope: "host".to_string(), + }); } -} + let control = match gateway.run_control(&agent_kind, request).await { + Ok(control) => control, + Err(err) => { + let failure = runtime_bootstrap_failure_from_wrapper_error(err); + if let Some(backchannel) = startup_backchannel.as_ref() { + let (orchestration_snapshot, manifest_snapshot) = { + let mut orchestration_guard = startup_context + .orchestration_session + .lock() + .expect("orchestration session mutex poisoned"); + let manifest_guard = manifest.lock().expect("runtime manifest mutex poisoned"); + orchestration_guard.mark_startup_prompt_failed( + manifest_guard.handle.participant_id.as_str(), + failure.message.clone(), + ); + (orchestration_guard.clone(), manifest_guard.clone()) + }; + let _ = persist_runtime_snapshots( + &startup_context.store, + &orchestration_snapshot, + &manifest_snapshot, + ); + backchannel.send(startup_prompt_failed_envelope(failure.message.clone())); + } + mark_runtime_startup_failed( + &startup_context.store, + &startup_context.orchestration_session, + &manifest, + &failure.message, + ); + return Err(failure); + } + }; + let agent_api::AgentWrapperRunControl { handle, cancel } = control; + let agent_api::AgentWrapperRunHandle { events, completion } = handle; + let shutdown_requested = Arc::new(AtomicBool::new(false)); + let (startup_tx, startup_rx) = tokio::sync::oneshot::channel::(); + let startup_signal = Arc::new(Mutex::new(Some(startup_tx))); + let agent_id = { + manifest + .lock() + .expect("runtime manifest mutex poisoned") + .handle + .agent_id + .clone() + }; + let run_id_for_tasks = run_id.clone(); + let event_store = startup_context.store.clone(); + let event_orchestration_session = Arc::clone(&startup_context.orchestration_session); + let event_manifest = Arc::clone(&manifest); + let startup_signal_for_events = Arc::clone(&startup_signal); + let startup_backchannel_for_events = startup_backchannel.clone(); + let shutdown_for_events = Arc::clone(&shutdown_requested); + let runtime_role_for_events = runtime_role.clone(); + let mut events = events; + let event_task = tokio::spawn(async move { + let controls_parent_session = runtime_controls_parent_session(&runtime_role_for_events); + let mut startup_saw_substantive_output = false; + while let Some(wrapper_event) = events.next().await { + let startup_turn_phase = turn_lifecycle_phase(wrapper_event.data.as_ref()) + .map(ToOwned::to_owned) + .or_else(|| { + infer_startup_turn_phase_fallback( + &wrapper_event, + startup_saw_substantive_output, + ) + .map(ToOwned::to_owned) + }); + let startup_failure_message = wrapper_event.message.clone(); + if matches!( + wrapper_event.kind, + agent_api::AgentWrapperEventKind::TextOutput + | agent_api::AgentWrapperEventKind::ToolCall + | agent_api::AgentWrapperEventKind::ToolResult + | agent_api::AgentWrapperEventKind::Error + ) { + startup_saw_substantive_output = true; + } + let mut startup_became_live = false; + let (orchestration_snapshot, manifest_snapshot, event) = { + let mut orchestration_guard = event_orchestration_session + .lock() + .expect("orchestration session mutex poisoned"); + let mut manifest_guard = event_manifest + .lock() + .expect("runtime manifest mutex poisoned"); + if let Some(session_id) = extract_session_handle_id(wrapper_event.data.as_ref()) { + if manifest_guard.internal.uaa_session_id.as_deref() != Some(session_id) { + manifest_guard.set_uaa_session_id(session_id.to_string()); + } + if manifest_guard.handle.state == AgentRuntimeSessionState::Allocating + && manifest_guard.can_advertise_live() + { + manifest_guard.transition_state(AgentRuntimeSessionState::Ready); + manifest_guard.touch_heartbeat(); + if controls_parent_session { + orchestration_guard.bind_active_session_handle( + manifest_guard.handle.participant_id.clone(), + ); + orchestration_guard.transition_state(OrchestrationSessionState::Active); + } else if orchestration_guard.state == OrchestrationSessionState::Active { + orchestration_guard.touch_active(); + } + startup_became_live = true; + } + } else if manifest_guard.handle.state == AgentRuntimeSessionState::Ready + && manifest_guard.can_advertise_live() + { + manifest_guard.transition_state(AgentRuntimeSessionState::Running); + orchestration_guard.touch_active(); + } + let event = translate_wrapper_event( + &manifest_guard, + &orchestration_guard, + &run_id_for_tasks, + wrapper_event, + ); + manifest_guard.touch_event(event.ts); + if orchestration_guard.state == OrchestrationSessionState::Active { + orchestration_guard.touch_active(); + } + if manifest_guard.handle.state == AgentRuntimeSessionState::Ready + && manifest_guard.can_advertise_live() + && !startup_became_live + { + manifest_guard.transition_state(AgentRuntimeSessionState::Running); + orchestration_guard.touch_active(); + } + if let Some(phase) = startup_turn_phase.as_deref() { + match phase { + "completed" => orchestration_guard.mark_startup_prompt_completed( + manifest_guard.handle.participant_id.as_str(), + "success", + ), + "failed" => orchestration_guard.mark_startup_prompt_failed( + manifest_guard.handle.participant_id.as_str(), + startup_failure_message + .clone() + .unwrap_or_else(|| "startup prompt turn failed".to_string()), + ), + _ => {} + } + } + (orchestration_guard.clone(), manifest_guard.clone(), event) + }; + let _ = persist_runtime_snapshots( + &event_store, + &orchestration_snapshot, + &manifest_snapshot, + ); + if let Some(backchannel) = startup_backchannel_for_events.as_ref() { + backchannel.send_event(&event); + if let Some(phase) = startup_turn_phase.as_deref() { + match phase { + "completed" => { + backchannel.send(startup_prompt_completed_envelope( + &orchestration_snapshot, + &manifest_snapshot, + )); + } + "failed" => backchannel.send(startup_prompt_failed_envelope( + startup_failure_message + .clone() + .unwrap_or_else(|| "startup prompt turn failed".to_string()), + )), + _ => {} + } + } + } + let _ = publish_agent_event(event); + + if startup_became_live { + let _ = publish_agent_event(build_runtime_message_event( + &manifest_snapshot, + &orchestration_snapshot, + run_id_for_tasks.clone(), + MessageEventKind::Status, + runtime_ready_message(&runtime_role_for_events), + )); + signal_runtime_startup(&startup_signal_for_events, RuntimeStartupSignal::Running); + } + } + + let mut publish_events = Vec::new(); + let mut startup_failure: Option = None; + let mut startup_backchannel_failure: Option = None; + let (orchestration_snapshot, manifest_snapshot) = { + let mut orchestration_guard = event_orchestration_session + .lock() + .expect("orchestration session mutex poisoned"); + let mut manifest_guard = event_manifest + .lock() + .expect("runtime manifest mutex poisoned"); + let was_allocating = + manifest_guard.handle.state == AgentRuntimeSessionState::Allocating; + let was_live = manifest_guard.is_authoritative_live(); + manifest_guard.set_event_stream_active(false); + if was_allocating { + let reason = + "attached control turn ended before ownership could be established".to_string(); + manifest_guard.transition_state(AgentRuntimeSessionState::Failed); + manifest_guard.mark_terminal_state(reason.clone()); + manifest_guard.internal.last_error_bucket = Some("bootstrap_run".to_string()); + manifest_guard.internal.last_error_message = Some(reason.clone()); + if controls_parent_session { + orchestration_guard.transition_state(OrchestrationSessionState::Failed); + orchestration_guard.mark_terminal(reason.clone()); + } else { + orchestration_guard.touch_active(); + } + startup_failure = Some(reason); + } else if !shutdown_for_events.load(Ordering::SeqCst) && was_live { + if can_park_host_runtime_after_detach( + &event_store, + controls_parent_session, + &orchestration_guard, + &manifest_guard, + runtime_owns_private_stop, + false, + ) { + apply_parked_host_runtime_snapshots( + &mut orchestration_guard, + &mut manifest_guard, + "owner detached cleanly", + ) + .expect("prevalidated host runtime parking should satisfy continuity"); + publish_events.push(build_runtime_message_event( + &manifest_guard, + &orchestration_guard, + run_id_for_tasks.clone(), + MessageEventKind::Status, + runtime_detached_message(&runtime_role_for_events), + )); + } else if runtime_owns_private_stop && controls_parent_session { + // Hidden owner-helper lifetimes are resolved by the retained completion + // observer at helper-exit handoff. Losing the event stream first is not, + // by itself, an invalidation condition for the durable parent session. + orchestration_guard.touch_active(); + } else { + let reason = if runtime_role_for_events == MEMBER_ROLE { + "world-scoped member control stream ended before completion observation" + } else { + "shell-owned orchestrator control stream ended before completion observation" + } + .to_string(); + manifest_guard.transition_state(AgentRuntimeSessionState::Invalidated); + manifest_guard.mark_terminal_state(reason.clone()); + manifest_guard.internal.last_error_bucket = + Some("runtime_lifecycle".to_string()); + manifest_guard.internal.last_error_message = Some(reason.clone()); + if controls_parent_session { + orchestration_guard + .transition_state(OrchestrationSessionState::Invalidated); + orchestration_guard.mark_terminal(reason.clone()); + } else { + orchestration_guard.touch_active(); + } + let mut event = AgentEvent::alert( + manifest_guard.handle.agent_id.clone(), + manifest_guard.handle.orchestration_session_id.clone(), + run_id_for_tasks.clone(), + runtime_stream_closed_alert_code(&runtime_role_for_events), + reason, + ); + apply_runtime_participant_lineage(&mut event, &manifest_guard); + publish_events.push(event); + } + } + if startup_backchannel_for_events.is_some() { + startup_backchannel_failure = match orchestration_guard.startup_prompt_state() { + Some(StartupPromptStreamState::PendingAcceptance) => { + terminalize_startup_prompt_failure( + &mut orchestration_guard, + &manifest_guard, + startup_failure.clone().unwrap_or_else(|| { + "startup prompt stream closed before acceptance".to_string() + }), + ) + } + Some(StartupPromptStreamState::Accepted) => terminalize_startup_prompt_failure( + &mut orchestration_guard, + &manifest_guard, + startup_failure.clone().unwrap_or_else(|| { + "startup prompt stream ended before terminal completion".to_string() + }), + ), + _ => None, + }; + } + (orchestration_guard.clone(), manifest_guard.clone()) + }; + let _ = + persist_runtime_snapshots(&event_store, &orchestration_snapshot, &manifest_snapshot); + for event in publish_events { + let _ = publish_agent_event(event); + } + if let Some(backchannel) = startup_backchannel_for_events.as_ref() { + if let Some(message) = startup_backchannel_failure { + backchannel.send(startup_prompt_failed_envelope(message)); + } + } + if let Some(message) = startup_failure { + signal_runtime_startup( + &startup_signal_for_events, + RuntimeStartupSignal::Failed(message), + ); + } + }); + + let completion_store = startup_context.store.clone(); + let completion_orchestration_session = Arc::clone(&startup_context.orchestration_session); + let completion_manifest = Arc::clone(&manifest); + let startup_signal_for_completion = Arc::clone(&startup_signal); + let startup_backchannel_for_completion = startup_backchannel.clone(); + let shutdown_for_completion = Arc::clone(&shutdown_requested); + let run_id_for_completion = run_id.clone(); + let runtime_role_for_completion = runtime_role.clone(); + let completion_task = tokio::spawn(async move { + let controls_parent_session = runtime_controls_parent_session(&runtime_role_for_completion); + let completion = completion.await; + let shutdown_requested = shutdown_for_completion.load(Ordering::SeqCst); + let mut startup_failure: Option = None; + let mut publish_events = Vec::new(); + let mut startup_backchannel_failure: Option = None; + + let (orchestration_snapshot, manifest_snapshot) = { + let mut orchestration_guard = completion_orchestration_session + .lock() + .expect("orchestration session mutex poisoned"); + let mut manifest_guard = completion_manifest + .lock() + .expect("runtime manifest mutex poisoned"); + + match completion { + Ok(completion) => { + if manifest_guard.internal.uaa_session_id.is_none() { + if let Some(session_id) = + extract_session_handle_id(completion.data.as_ref()) + { + manifest_guard.set_uaa_session_id(session_id.to_string()); + } + } + + if manifest_guard.handle.state == AgentRuntimeSessionState::Allocating { + let reason = if completion.status.success() { + "attached control turn ended before ownership could be established" + .to_string() + } else { + format!( + "attached control turn exited with status {} before ownership was established", + completion.status.code().unwrap_or(-1) + ) + }; + manifest_guard.transition_state(AgentRuntimeSessionState::Failed); + manifest_guard.mark_terminal_state(reason.clone()); + manifest_guard.internal.last_error_bucket = + Some("bootstrap_run".to_string()); + manifest_guard.internal.last_error_message = Some(reason.clone()); + if controls_parent_session { + orchestration_guard.transition_state(OrchestrationSessionState::Failed); + orchestration_guard.mark_terminal(reason.clone()); + } else { + orchestration_guard.touch_active(); + } + startup_failure = Some(reason); + } else if shutdown_requested + && manifest_guard.handle.state == AgentRuntimeSessionState::Stopping + { + apply_runtime_stop_closeout(&mut orchestration_guard, &mut manifest_guard); + } else if shutdown_requested + && manifest_guard.handle.state == AgentRuntimeSessionState::Failed + { + // Preserve bootstrap failure when startup cleanup observes completion. + } else if manifest_guard.handle.state == AgentRuntimeSessionState::Failed { + // Preserve bootstrap failure even if completion wins the race before + // startup cleanup flips the shutdown_requested flag. + } else if manifest_guard.handle.state == AgentRuntimeSessionState::Invalidated { + // Preserve invalidation emitted when the attached control boundary ended. + } else if manifest_guard.handle.state == AgentRuntimeSessionState::Stopping { + apply_runtime_stop_closeout(&mut orchestration_guard, &mut manifest_guard); + } else if can_park_host_runtime_after_detach( + &completion_store, + controls_parent_session, + &orchestration_guard, + &manifest_guard, + runtime_owns_private_stop, + true, + ) { + apply_parked_host_runtime_snapshots( + &mut orchestration_guard, + &mut manifest_guard, + "owner detached cleanly", + ) + .expect("prevalidated host runtime parking should satisfy continuity"); + publish_events.push(build_runtime_message_event( + &manifest_guard, + &orchestration_guard, + run_id_for_completion.clone(), + MessageEventKind::Status, + runtime_detached_message(&runtime_role_for_completion), + )); + } else { + let reason = format!( + "attached control turn exited with status {}", + completion.status.code().unwrap_or(-1) + ); + manifest_guard.transition_state(AgentRuntimeSessionState::Invalidated); + manifest_guard.mark_terminal_state(reason.clone()); + manifest_guard.internal.last_error_bucket = + Some("runtime_lifecycle".to_string()); + manifest_guard.internal.last_error_message = Some(reason.clone()); + if controls_parent_session { + orchestration_guard + .transition_state(OrchestrationSessionState::Invalidated); + orchestration_guard.mark_terminal(reason.clone()); + } else { + orchestration_guard.touch_active(); + } + let mut event = AgentEvent::alert( + manifest_guard.handle.agent_id.clone(), + manifest_guard.handle.orchestration_session_id.clone(), + run_id_for_completion.clone(), + runtime_invalidated_alert_code(&runtime_role_for_completion), + reason, + ); + apply_runtime_participant_lineage(&mut event, &manifest_guard); + publish_events.push(event); + } + } + Err(err) => { + let reason = match &err { + agent_api::AgentWrapperError::Backend { message } => message.clone(), + other => other.to_string(), + }; + if manifest_guard.handle.state == AgentRuntimeSessionState::Allocating { + let reason = + format!("failed to establish attached control ownership: {reason}"); + manifest_guard.transition_state(AgentRuntimeSessionState::Failed); + manifest_guard.mark_terminal_state(reason.clone()); + manifest_guard.internal.last_error_bucket = + Some("bootstrap_run".to_string()); + manifest_guard.internal.last_error_message = Some(reason.clone()); + if controls_parent_session { + orchestration_guard.transition_state(OrchestrationSessionState::Failed); + orchestration_guard.mark_terminal(reason.clone()); + } else { + orchestration_guard.touch_active(); + } + startup_failure = Some(reason); + } else if shutdown_requested + && manifest_guard.handle.state == AgentRuntimeSessionState::Stopping + && reason == "cancelled" + { + apply_runtime_stop_closeout(&mut orchestration_guard, &mut manifest_guard); + } else if shutdown_requested + && manifest_guard.handle.state == AgentRuntimeSessionState::Failed + { + // Preserve bootstrap failure when startup cleanup observes completion. + } else if manifest_guard.handle.state == AgentRuntimeSessionState::Failed { + // Preserve bootstrap failure even if completion wins the race before + // startup cleanup flips the shutdown_requested flag. + } else if manifest_guard.handle.state == AgentRuntimeSessionState::Invalidated { + // Preserve invalidation emitted when the attached control boundary ended. + } else if shutdown_requested { + let reason = format!("failed to stop attached control: {reason}"); + manifest_guard.transition_state(AgentRuntimeSessionState::Failed); + manifest_guard.mark_terminal_state(reason.clone()); + manifest_guard.internal.last_error_bucket = + Some("runtime_shutdown".to_string()); + manifest_guard.internal.last_error_message = Some(reason.clone()); + if controls_parent_session { + orchestration_guard.transition_state(OrchestrationSessionState::Failed); + orchestration_guard.mark_terminal(reason); + } else { + orchestration_guard.touch_active(); + } + } else if can_park_host_runtime_after_detach( + &completion_store, + controls_parent_session, + &orchestration_guard, + &manifest_guard, + runtime_owns_private_stop, + true, + ) { + apply_parked_host_runtime_snapshots( + &mut orchestration_guard, + &mut manifest_guard, + "owner detached cleanly", + ) + .expect("prevalidated host runtime parking should satisfy continuity"); + publish_events.push(build_runtime_message_event( + &manifest_guard, + &orchestration_guard, + run_id_for_completion.clone(), + MessageEventKind::Status, + runtime_detached_message(&runtime_role_for_completion), + )); + } else { + let reason = format!("attached control turn ended unexpectedly: {reason}"); + manifest_guard.transition_state(AgentRuntimeSessionState::Invalidated); + manifest_guard.mark_terminal_state(reason.clone()); + manifest_guard.internal.last_error_bucket = + Some("runtime_lifecycle".to_string()); + manifest_guard.internal.last_error_message = Some(reason.clone()); + if controls_parent_session { + orchestration_guard + .transition_state(OrchestrationSessionState::Invalidated); + orchestration_guard.mark_terminal(reason.clone()); + } else { + orchestration_guard.touch_active(); + } + let mut event = AgentEvent::alert( + manifest_guard.handle.agent_id.clone(), + manifest_guard.handle.orchestration_session_id.clone(), + run_id_for_completion.clone(), + runtime_invalidated_alert_code(&runtime_role_for_completion), + reason, + ); + apply_runtime_participant_lineage(&mut event, &manifest_guard); + publish_events.push(event); + } + } + } + if startup_backchannel_for_completion.is_some() { + startup_backchannel_failure = terminalize_startup_prompt_failure( + &mut orchestration_guard, + &manifest_guard, + startup_failure.clone().unwrap_or_else(|| { + "startup prompt completion did not reach a terminal turn outcome" + .to_string() + }), + ); + } + + (orchestration_guard.clone(), manifest_guard.clone()) + }; + + let _ = persist_runtime_snapshots( + &completion_store, + &orchestration_snapshot, + &manifest_snapshot, + ); + for event in publish_events { + let _ = publish_agent_event(event); + } + if let Some(backchannel) = startup_backchannel_for_completion.as_ref() { + if let Some(message) = startup_backchannel_failure { + backchannel.send(startup_prompt_failed_envelope(message)); + } + } + if let Some(message) = startup_failure { + signal_runtime_startup( + &startup_signal_for_completion, + RuntimeStartupSignal::Failed(message), + ); + } + }); + + let (heartbeat_stop_tx, mut heartbeat_stop_rx) = tokio::sync::oneshot::channel(); + let heartbeat_store = startup_context.store.clone(); + let heartbeat_orchestration_session = Arc::clone(&startup_context.orchestration_session); + let heartbeat_manifest = Arc::clone(&manifest); + let heartbeat_task = tokio::spawn(async move { + let mut interval = tokio::time::interval(Duration::from_secs(5)); + interval.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Delay); + loop { + tokio::select! { + _ = interval.tick() => { + let next = { + let mut orchestration_guard = heartbeat_orchestration_session + .lock() + .expect("orchestration session mutex poisoned"); + let mut manifest_guard = heartbeat_manifest + .lock() + .expect("runtime manifest mutex poisoned"); + if !manifest_guard.is_authoritative_live() { + None + } else { + manifest_guard.touch_heartbeat(); + if orchestration_guard.state == OrchestrationSessionState::Active { + orchestration_guard.touch_active(); + } + Some((orchestration_guard.clone(), manifest_guard.clone())) + } + }; + let Some(next) = next else { + break; + }; + let _ = persist_runtime_snapshots( + &heartbeat_store, + &next.0, + &next.1, + ); + } + _ = &mut heartbeat_stop_rx => break, + } + } + }); + let (retained_orchestration_snapshot, retained_manifest_snapshot) = { + let mut orchestration_guard = startup_context + .orchestration_session + .lock() + .expect("orchestration session mutex poisoned"); + let mut manifest_guard = manifest.lock().expect("runtime manifest mutex poisoned"); + manifest_guard.mark_runtime_ownership_retained(); + if orchestration_guard.state == OrchestrationSessionState::Active || controls_parent_session + { + orchestration_guard.touch_active(); + } + (orchestration_guard.clone(), manifest_guard.clone()) + }; + persist_runtime_snapshots( + &startup_context.store, + &retained_orchestration_snapshot, + &retained_manifest_snapshot, + ) + .map_err(|err| RuntimeBootstrapFailure { + exit_code: 1, + message: format!("failed to persist retained runtime ownership: {err:#}"), + })?; + let mut retained_control = LocalRetainedRunControl { + cancel, + event_task: Some(event_task), + completion_task: Some(completion_task), + }; + let mut heartbeat_stop_tx = Some(heartbeat_stop_tx); + let mut heartbeat_task = Some(heartbeat_task); + + match tokio::time::timeout(Duration::from_secs(10), startup_rx).await { + Ok(Ok(RuntimeStartupSignal::Running)) => {} + Ok(Ok(RuntimeStartupSignal::Failed(message))) => { + abort_bootstrap_runtime( + &shutdown_requested, + &mut retained_control, + &mut heartbeat_stop_tx, + &mut heartbeat_task, + ) + .await; + return Err(RuntimeBootstrapFailure { + exit_code: 4, + message, + }); + } + Ok(Err(_)) => { + abort_bootstrap_runtime( + &shutdown_requested, + &mut retained_control, + &mut heartbeat_stop_tx, + &mut heartbeat_task, + ) + .await; + mark_runtime_startup_failed( + &startup_context.store, + &startup_context.orchestration_session, + &manifest, + "failed to establish attached control ownership", + ); + return Err(RuntimeBootstrapFailure { + exit_code: 1, + message: "failed to establish attached control ownership".to_string(), + }); + } + Err(_) => { + abort_bootstrap_runtime( + &shutdown_requested, + &mut retained_control, + &mut heartbeat_stop_tx, + &mut heartbeat_task, + ) + .await; + let message = format!( + "timed out waiting for {} control ownership: {agent_id}", + if runtime_role == MEMBER_ROLE { + "world-scoped member" + } else { + "shell-owned orchestrator" + } + ); + mark_runtime_startup_failed( + &startup_context.store, + &startup_context.orchestration_session, + &manifest, + &message, + ); + return Err(RuntimeBootstrapFailure { + exit_code: 4, + message, + }); + } + } + let surfaced_uaa_session_handle_id = { + manifest + .lock() + .expect("runtime manifest mutex poisoned") + .internal + .uaa_session_id + .clone() + }; + let uaa_session_handle_id = match surfaced_uaa_session_handle_id { + Some(session_handle_id) => session_handle_id, + None => { + abort_bootstrap_runtime( + &shutdown_requested, + &mut retained_control, + &mut heartbeat_stop_tx, + &mut heartbeat_task, + ) + .await; + let message = + "runtime startup signalled ready without a surfaced UAA session handle".to_string(); + mark_runtime_startup_failed( + &startup_context.store, + &startup_context.orchestration_session, + &manifest, + &message, + ); + return Err(RuntimeBootstrapFailure { + exit_code: 1, + message, + }); + } + }; + let (stop_tx, stop_rx) = private_stop_request_channel(); + let (stop_orchestration_session_id, stop_participant_id) = + runtime_stop_transport_ids(&manifest); + let stop_transport = match register_private_stop_transport( + &startup_context.store, + &stop_orchestration_session_id, + &stop_participant_id, + stop_tx, + ) + .await + { + Ok(stop_transport) => stop_transport, + Err(err) => { + abort_bootstrap_runtime( + &shutdown_requested, + &mut retained_control, + &mut heartbeat_stop_tx, + &mut heartbeat_task, + ) + .await; + let message = format!("failed to register private stop transport: {err:#}"); + mark_runtime_startup_failed( + &startup_context.store, + &startup_context.orchestration_session, + &manifest, + &message, + ); + return Err(RuntimeBootstrapFailure { + exit_code: 1, + message, + }); + } + }; + let (private_stop_rx, stop_owner_task) = if runtime_owns_private_stop { + (Some(stop_rx), None) + } else { + ( + None, + Some(spawn_local_private_stop_owner( + startup_context.store.clone(), + Arc::clone(&startup_context.orchestration_session), + Arc::clone(&manifest), + Arc::clone(&shutdown_requested), + retained_control.cancel.clone(), + stop_rx, + )), + ) + }; + let (prompt_tx, prompt_rx) = private_prompt_request_channel(); + let (auto_park_tx, auto_park_rx) = if auto_park_after_public_turn { + let (tx, rx) = mpsc::unbounded_channel(); + (Some(tx), Some(rx)) + } else { + (None, None) + }; + let prompt_transport = match register_private_prompt_transport( + &startup_context.store, + &stop_orchestration_session_id, + &stop_participant_id, + prompt_tx, + ) + .await + { + Ok(prompt_transport) => prompt_transport, + Err(err) => { + abort_bootstrap_runtime( + &shutdown_requested, + &mut retained_control, + &mut heartbeat_stop_tx, + &mut heartbeat_task, + ) + .await; + let message = format!("failed to register private prompt transport: {err:#}"); + mark_runtime_startup_failed( + &startup_context.store, + &startup_context.orchestration_session, + &manifest, + &message, + ); + return Err(RuntimeBootstrapFailure { + exit_code: 1, + message, + }); + } + }; + let prompt_owner_task = spawn_local_private_prompt_owner( + prompt_runtime_from_parts( + descriptor.clone(), + Arc::clone(&startup_context.orchestration_session), + Arc::clone(&manifest), + startup_context.store.clone(), + uaa_session_handle_id.clone(), + auto_park_tx.clone(), + ), + prompt_rx, + ); + + Ok(Some(AsyncReplAgentRuntime { + descriptor, + orchestration_session: startup_context.orchestration_session, + manifest, + store: startup_context.store, + uaa_session_handle_id, + retained_control: RetainedRunControl::Local(retained_control), + shutdown_requested, + auto_park_rx, + private_stop_rx, + stop_transport: Some(stop_transport), + stop_owner_task, + prompt_transport: Some(prompt_transport), + prompt_owner_task: Some(prompt_owner_task), + heartbeat_stop_tx, + heartbeat_task, + })) +} + +fn runtime_manifest_snapshot(runtime: &AsyncReplAgentRuntime) -> AgentRuntimeParticipantRecord { + runtime + .manifest + .lock() + .expect("runtime manifest mutex poisoned") + .clone() +} + +fn runtime_backend_id(runtime: &AsyncReplAgentRuntime) -> String { + runtime + .manifest + .lock() + .expect("runtime manifest mutex poisoned") + .handle + .backend_id + .clone() +} + +fn runtime_prompt_submit_runtime( + runtime: &AsyncReplAgentRuntime, +) -> crate::execution::agent_runtime::control::PromptSubmitRuntime { + prompt_runtime_from_parts( + runtime.descriptor.clone(), + Arc::clone(&runtime.orchestration_session), + Arc::clone(&runtime.manifest), + runtime.store.clone(), + runtime.uaa_session_handle_id.clone(), + None, + ) +} + +#[cfg(any(target_os = "linux", target_os = "macos"))] +fn select_member_runtime_descriptor( + startup_context: &RuntimeOrchestrationContext, +) -> std::result::Result, RuntimeBootstrapFailure> { + validate_member_selection( + &startup_context.effective_config, + &startup_context.inventory, + ) + .map_err(|err| RuntimeBootstrapFailure { + exit_code: member_selection_error_exit_code(&err), + message: err.reason, + }) +} + +fn select_exact_runtime_descriptor( + startup_context: &RuntimeOrchestrationContext, + scope: crate::execution::config_model::AgentExecutionScope, + backend_id: &str, +) -> std::result::Result, RuntimeBootstrapFailure> { + validate_exact_backend_selection( + &startup_context.effective_config, + &startup_context.inventory, + scope, + backend_id, + ) + .map_err(|err| RuntimeBootstrapFailure { + exit_code: exact_backend_selection_error_exit_code(&err), + message: err.reason, + }) +} + +fn select_member_runtime_descriptor_for_backend( + startup_context: &RuntimeOrchestrationContext, + backend_id: &str, +) -> std::result::Result, RuntimeBootstrapFailure> { + select_exact_runtime_descriptor( + startup_context, + crate::execution::config_model::AgentExecutionScope::World, + backend_id, + ) +} + +fn select_host_runtime_descriptor_for_backend( + startup_context: &RuntimeOrchestrationContext, + backend_id: &str, +) -> std::result::Result, RuntimeBootstrapFailure> { + select_exact_runtime_descriptor( + startup_context, + crate::execution::config_model::AgentExecutionScope::Host, + backend_id, + ) +} + +fn active_orchestrator_backend_id(runtime: &AsyncReplAgentRuntime) -> String { + runtime_backend_id(runtime) +} + +fn resolve_targeted_turn_route( + startup_context: Option<&RuntimeOrchestrationContext>, + dormant_host_bootstrap: Option<&ResolvedHostOrchestratorBootstrap>, + agent_runtime: Option<&AsyncReplAgentRuntime>, + backend_id: &str, +) -> std::result::Result { + if let Some(runtime) = agent_runtime { + let active_backend_id = active_orchestrator_backend_id(runtime); + if active_backend_id == backend_id { + return Ok(TargetedTurnRoute::Host); + } + } + + if let Some(dormant_host_bootstrap) = dormant_host_bootstrap { + if dormant_host_bootstrap.descriptor.backend_id == backend_id { + return Ok(TargetedTurnRoute::Host); + } + } + + if let Some(startup_context) = startup_context { + if select_host_runtime_descriptor_for_backend(startup_context, backend_id)?.is_some() { + let expected = agent_runtime + .map(active_orchestrator_backend_id) + .or_else(|| { + dormant_host_bootstrap.map(|bootstrap| bootstrap.descriptor.backend_id.clone()) + }) + .unwrap_or_else(|| "".to_string()); + return Err(RuntimeBootstrapFailure { + exit_code: 2, + message: format!( + "targeted host follow-up turns may only target the active orchestrator backend for this REPL session (expected '{}', got '{}')", + expected, backend_id + ), + }); + } + + if let Some(descriptor) = + select_member_runtime_descriptor_for_backend(startup_context, backend_id)? + { + return Ok(TargetedTurnRoute::World(descriptor)); + } + } + + Err(RuntimeBootstrapFailure { + exit_code: 2, + message: format!("no exact targeted-turn backend match for '{backend_id}'"), + }) +} + +async fn dispatch_targeted_follow_up_turn( + targeted_turn: TargetedTurn<'_>, + context: TargetedTurnDispatchContext<'_>, +) -> std::result::Result { + let TargetedTurnDispatchContext { + startup_context, + dormant_host_bootstrap, + agent_runtime, + world_session, + member_runtimes, + pending_member_replacements, + agent_printer, + telemetry, + } = context; + + let route = match resolve_targeted_turn_route( + startup_context.as_ref(), + dormant_host_bootstrap.as_ref(), + agent_runtime.as_ref(), + targeted_turn.backend_id, + ) { + Ok(route) => route, + Err(failure) => return Ok(TargetedTurnDispatchStatus::Rejected(failure)), + }; + + match route { + TargetedTurnRoute::Host => { + if agent_runtime.is_none() { + let resolved = dormant_host_bootstrap.take().ok_or_else(|| { + anyhow!( + "substrate: error: no active or dormant orchestrator runtime is available for targeted follow-up turns" + ) + })?; + let prepared = prepare_host_orchestrator_runtime_from_resolved(resolved) + .map_err(|failure| anyhow!("substrate: error: {}", failure.message))?; + let prepared_startup_context = prepared.startup_context.clone(); + let initial_world_binding = + world_session.as_ref().map(|session| PersistedWorldBinding { + world_id: session.world_id.clone(), + world_generation: session.world_generation, + }); + let runtime = match start_host_orchestrator_runtime_with_prepared_prompt( + Some(prepared), + initial_world_binding.as_ref(), + Some(InitialExecPromptPlan::Replace( + targeted_turn.prompt.to_string(), + )), + false, + false, + agent_printer, + telemetry, + ) + .await + { + Ok(Some(runtime)) => runtime, + Ok(None) => { + return Err(anyhow!( + "substrate: error: targeted orchestrator launch did not produce a runtime" + )); + } + Err(failure) => { + finalize_runtime_startup_failure( + Some(&prepared_startup_context), + world_session, + &failure.message, + ) + .await; + return Err(anyhow!("substrate: error: {}", failure.message)); + } + }; + *startup_context = Some(prepared_startup_context); + *agent_runtime = Some(runtime); + } else { + let runtime = agent_runtime.as_mut().ok_or_else(|| { + anyhow!( + "substrate: error: no active orchestrator runtime is available for targeted follow-up turns" + ) + })?; + submit_host_targeted_turn(runtime, targeted_turn.prompt, agent_printer, telemetry) + .await?; + } + } + TargetedTurnRoute::World(descriptor) => { + ensure_no_policy_drift( + world_session, + startup_context.as_ref(), + agent_printer, + telemetry, + ) + .await?; + reconcile_member_runtime_generation( + world_session.as_ref(), + member_runtimes, + pending_member_replacements, + agent_printer, + telemetry, + ) + .await?; + let launched_from_targeted_prompt = ensure_member_runtime_ready_for_descriptor( + EnsureMemberRuntimeReadyContext { + startup_context: startup_context.as_ref(), + world_session: world_session.as_ref(), + agent_printer, + telemetry, + }, + &descriptor, + Some(targeted_turn.prompt), + member_runtimes, + pending_member_replacements, + ) + .await?; + if launched_from_targeted_prompt { + return Ok(TargetedTurnDispatchStatus::Submitted); + } + let runtime = member_runtimes + .get_mut(targeted_turn.backend_id) + .ok_or_else(|| { + anyhow!( + "substrate: error: world-scoped member runtime is unavailable for targeted follow-up turns for backend '{}'", + targeted_turn.backend_id + ) + })?; + submit_world_targeted_turn(runtime, targeted_turn.prompt, agent_printer, telemetry) + .await?; + } + } + + Ok(TargetedTurnDispatchStatus::Submitted) +} + +#[cfg(any(target_os = "linux", target_os = "macos"))] +fn ensure_member_backend_allowed( + startup_context: &RuntimeOrchestrationContext, + descriptor: &RuntimeSelectionDescriptor, +) -> std::result::Result<(), RuntimeBootstrapFailure> { + if backend_allowed(&startup_context.base_policy, &descriptor.backend_id) { + Ok(()) + } else { + Err(RuntimeBootstrapFailure { + exit_code: 5, + message: format!( + "required world-scoped member backend '{}' is not allowlisted by effective policy agents.allowed_backends", + descriptor.backend_id + ), + }) + } +} + +#[cfg(any(target_os = "linux", target_os = "macos"))] +fn resolve_live_member_parent( + startup_context: &RuntimeOrchestrationContext, +) -> std::result::Result { + let orchestration_snapshot = startup_context.snapshot(); + let (parent_session, parent_participant) = startup_context + .store + .resolve_live_orchestrator_participant(&orchestration_snapshot.orchestrator_agent_id) + .map_err(|err| RuntimeBootstrapFailure { + exit_code: 1, + message: format!( + "failed to resolve the live orchestrator parent required for member launch: {err:#}" + ), + })? + .ok_or_else(|| RuntimeBootstrapFailure { + exit_code: 1, + message: + "member launch requires exactly one live orchestrator parent, but none is active" + .to_string(), + })?; + if parent_session.orchestration_session_id != orchestration_snapshot.orchestration_session_id { + return Err(RuntimeBootstrapFailure { + exit_code: 1, + message: format!( + "member launch resolved a live orchestrator parent from a different orchestration session (expected {}, got {})", + orchestration_snapshot.orchestration_session_id, + parent_session.orchestration_session_id + ), + }); + } + Ok(parent_participant) +} + +#[cfg(any(target_os = "linux", target_os = "macos"))] +fn authoritative_member_world_binding( + startup_context: &RuntimeOrchestrationContext, + world_binding: &PersistedWorldBinding, +) -> std::result::Result { + let orchestration_snapshot = startup_context.snapshot(); + let Some(authoritative_world_id) = orchestration_snapshot.world_id.clone() else { + return Err(RuntimeBootstrapFailure { + exit_code: 1, + message: + "member launch requires an authoritative world binding, but world_id is missing" + .to_string(), + }); + }; + let Some(authoritative_world_generation) = orchestration_snapshot.world_generation else { + return Err(RuntimeBootstrapFailure { + exit_code: 1, + message: + "member launch requires an authoritative world binding, but world_generation is missing" + .to_string(), + }); + }; + if authoritative_world_id != world_binding.world_id + || authoritative_world_generation != world_binding.world_generation + { + return Err(RuntimeBootstrapFailure { + exit_code: 1, + message: format!( + "member launch requires the authoritative parent world binding to match the active world session (authoritative={},{} active={},{} )", + authoritative_world_id, + authoritative_world_generation, + world_binding.world_id, + world_binding.world_generation, + ), + }); + } + Ok(AgentRuntimeParticipantWorldBinding { + world_id: authoritative_world_id, + world_generation: authoritative_world_generation, + }) +} + +#[cfg(any(target_os = "linux", target_os = "macos"))] +fn live_member_for_generation( + startup_context: &RuntimeOrchestrationContext, + descriptor: &RuntimeSelectionDescriptor, + world_generation: u64, +) -> std::result::Result, RuntimeBootstrapFailure> { + let orchestration_session_id = startup_context.orchestration_session_id(); + let mut live_members = startup_context + .store + .list_live_participants_for_session(&orchestration_session_id) + .map_err(|err| RuntimeBootstrapFailure { + exit_code: 1, + message: format!( + "failed to inspect live member participants for the current session: {err:#}" + ), + })? + .into_iter() + .filter(|participant| { + participant.handle.role == MEMBER_ROLE + && participant.handle.backend_id == descriptor.backend_id + && participant.handle.execution.scope + == crate::execution::config_model::AgentExecutionScope::World + && participant.handle.world_generation == Some(world_generation) + }) + .collect::>(); + match live_members.len() { + 0 => Ok(None), + 1 => Ok(live_members.pop()), + _ => Err(RuntimeBootstrapFailure { + exit_code: 1, + message: format!( + "member launch found multiple live world-scoped member participants for backend '{}' in world generation {} within the current orchestration session", + descriptor.backend_id, world_generation + ), + }), + } +} + +#[cfg(any(target_os = "linux", target_os = "macos"))] +fn prepare_member_runtime_startup_for_descriptor( + startup_context: &RuntimeOrchestrationContext, + descriptor: RuntimeSelectionDescriptor, + world_binding: &PersistedWorldBinding, + resumed_from: Option<&AgentRuntimeParticipantRecord>, +) -> std::result::Result { + ensure_member_backend_allowed(startup_context, &descriptor)?; + let parent_participant = resolve_live_member_parent(startup_context)?; + let authoritative_world = authoritative_member_world_binding(startup_context, world_binding)?; + + if live_member_for_generation( + startup_context, + &descriptor, + authoritative_world.world_generation, + )? + .is_some() + { + return Err(RuntimeBootstrapFailure { + exit_code: 1, + message: format!( + "member launch found an existing authoritative-live world-scoped member participant for backend '{}' in world generation {} without a retained runtime handle in the current REPL", + descriptor.backend_id, authoritative_world.world_generation + ), + }); + } + + let participant_id = format!("ash_{}", Uuid::now_v7()); + let lease_token = Uuid::now_v7().to_string(); + let run_id = Uuid::now_v7().to_string(); + let manifest = if let Some(previous_participant) = resumed_from { + AgentRuntimeSessionManifest::new_replacement_participant( + &descriptor, + AgentRuntimeReplacementParticipantInit { + orchestration_session_id: startup_context.orchestration_session_id(), + participant_id, + role: MEMBER_ROLE.to_string(), + orchestrator_participant_id: Some(parent_participant.handle.participant_id.clone()), + parent_participant_id: None, + resumed_from_participant_id: previous_participant.handle.participant_id.clone(), + world: Some(authoritative_world), + lease_token, + }, + ) + } else { + AgentRuntimeSessionManifest::new_member_participant( + &descriptor, + startup_context.orchestration_session_id(), + participant_id, + parent_participant.handle.participant_id.clone(), + None, + Some(authoritative_world), + lease_token, + ) + } + .map_err(|err| RuntimeBootstrapFailure { + exit_code: 1, + message: format!("failed to construct world-scoped member participant state: {err:#}"), + })?; + let mut manifest = manifest; + manifest.internal.latest_run_id = Some(run_id.clone()); + + let gateway = + build_gateway_for_descriptor(&descriptor).map_err(|err| RuntimeBootstrapFailure { + exit_code: 1, + message: format!("failed to build world-scoped member UAA runtime registry: {err:#}"), + })?; + let agent_kind = agent_api::AgentWrapperKind::new(descriptor.backend_kind.as_agent_kind_str()) + .map_err(|err| RuntimeBootstrapFailure { + exit_code: 2, + message: format!("failed to resolve member runtime backend kind: {err}"), + })?; + + Ok(PreparedAgentRuntime { + descriptor, + gateway, + agent_kind, + startup_context: startup_context.clone(), + manifest: Arc::new(Mutex::new(manifest)), + run_id, + startup_extensions: BTreeMap::new(), + }) +} + +#[cfg(all(test, unix))] +async fn start_member_runtime_with_prepared( + prepared: Option, + agent_printer: &ReplPrinter, + telemetry: &mut ReplSessionTelemetry, +) -> std::result::Result, RuntimeBootstrapFailure> { + start_host_orchestrator_runtime_with_prepared(prepared, None, agent_printer, telemetry).await +} + +#[cfg(any(target_os = "linux", target_os = "macos"))] +fn member_runtime_backend_kind( + descriptor: &RuntimeSelectionDescriptor, +) -> MemberRuntimeBackendKindV1 { + match descriptor.backend_kind { + AgentRuntimeBackendKind::Codex => MemberRuntimeBackendKindV1::Codex, + AgentRuntimeBackendKind::ClaudeCode => MemberRuntimeBackendKindV1::ClaudeCode, + } +} + +#[cfg(any(target_os = "linux", target_os = "macos"))] +fn build_member_dispatch_transport_request( + prepared: &PreparedAgentRuntime, + initial_prompt: Option, +) -> std::result::Result { + let manifest = prepared + .manifest + .lock() + .expect("runtime manifest mutex poisoned") + .clone(); + let world_id = manifest + .handle + .world_id + .clone() + .ok_or_else(|| RuntimeBootstrapFailure { + exit_code: 1, + message: "member dispatch requires an authoritative world_id".to_string(), + })?; + let world_generation = + manifest + .handle + .world_generation + .ok_or_else(|| RuntimeBootstrapFailure { + exit_code: 1, + message: "member dispatch requires an authoritative world_generation".to_string(), + })?; + let orchestrator_participant_id = manifest + .handle + .orchestrator_participant_id + .clone() + .ok_or_else(|| RuntimeBootstrapFailure { + exit_code: 1, + message: "member dispatch requires orchestrator_participant_id".to_string(), + })?; + + Ok(MemberDispatchTransportRequest { + orchestration_session_id: manifest.handle.orchestration_session_id.clone(), + participant_id: manifest.handle.participant_id.clone(), + orchestrator_participant_id, + parent_participant_id: manifest.handle.parent_participant_id.clone(), + resumed_from_participant_id: manifest.handle.resumed_from_participant_id.clone(), + backend_id: prepared.descriptor.backend_id.clone(), + protocol: prepared.descriptor.protocol.clone(), + run_id: prepared.run_id.clone(), + world_id, + world_generation, + initial_prompt, + backend_kind: member_runtime_backend_kind(&prepared.descriptor), + binary_path: prepared.descriptor.binary_path.display().to_string(), + }) +} + +#[cfg(any(target_os = "linux", target_os = "macos"))] +async fn abort_remote_member_bootstrap_runtime( + shutdown_requested: &Arc, + client: &AgentClient, + span_id: &Arc>>, + observe_task: &mut Option>, +) { + shutdown_requested.store(true, Ordering::SeqCst); + let span_id = span_id + .lock() + .expect("remote member span mutex poisoned") + .clone(); + if let Some(span_id) = span_id { + let _ = client + .cancel_execute(ExecuteCancelRequestV1 { + span_id, + sig: "INT".to_string(), + }) + .await; + } + if let Some(task) = observe_task.take() { + let _ = task.await; + } +} + +#[cfg(any(target_os = "linux", target_os = "macos"))] +fn spawn_remote_private_stop_owner( + manifest: Arc>, + shutdown_requested: Arc, + client: Arc, + span_id: String, + mut stop_rx: PrivateStopRequestReceiver, +) -> tokio::task::JoinHandle<()> { + tokio::spawn(async move { + while let Some(request) = stop_rx.recv().await { + let outcome = if runtime_is_terminal(&manifest) { + PrivateStopOutcome::AlreadyTerminal + } else { + shutdown_requested.store(true, Ordering::SeqCst); + match client + .cancel_execute(ExecuteCancelRequestV1 { + span_id: span_id.clone(), + sig: "INT".to_string(), + }) + .await + { + Ok(_) => PrivateStopOutcome::Accepted, + Err(_) => { + shutdown_requested.store(false, Ordering::SeqCst); + PrivateStopOutcome::ProtocolError + } + } + }; + let _ = request.response_tx.send(outcome); + } + }) +} + +#[cfg(any(target_os = "linux", target_os = "macos"))] +async fn start_remote_member_runtime_with_prepared( + prepared: Option, + initial_prompt: Option, + agent_printer: &ReplPrinter, + telemetry: &mut ReplSessionTelemetry, +) -> std::result::Result, RuntimeBootstrapFailure> { + use http_body_util::BodyExt as _; + + let Some(prepared) = prepared else { + return Ok(None); + }; + let transport_request = build_member_dispatch_transport_request(&prepared, initial_prompt)?; + let PreparedAgentRuntime { + descriptor, + gateway: _gateway, + agent_kind: _agent_kind, + startup_context, + manifest, + run_id, + startup_extensions: _startup_extensions, + } = prepared; + + let runtime_role = { + manifest + .lock() + .expect("runtime manifest mutex poisoned") + .handle + .role + .clone() + }; + let persist_participant_result = { + let manifest_guard = manifest.lock().expect("runtime manifest mutex poisoned"); + startup_context.store.persist_participant(&manifest_guard) + }; + if let Err(err) = persist_participant_result { + mark_runtime_startup_failed( + &startup_context.store, + &startup_context.orchestration_session, + &manifest, + &format!("failed to persist agent runtime participant record: {err:#}"), + ); + return Err(RuntimeBootstrapFailure { + exit_code: 1, + message: format!("failed to persist agent runtime participant record: {err:#}"), + }); + } + + let orchestration_snapshot = startup_context + .orchestration_session + .lock() + .expect("orchestration session mutex poisoned") + .clone(); + let manifest_snapshot = manifest + .lock() + .expect("runtime manifest mutex poisoned") + .clone(); + emit_runtime_event( + build_runtime_message_event( + &manifest_snapshot, + &orchestration_snapshot, + run_id.clone(), + MessageEventKind::Registered, + runtime_registered_message(&runtime_role), + ), + telemetry, + agent_printer, + ); + emit_runtime_event( + build_runtime_message_event( + &manifest_snapshot, + &orchestration_snapshot, + run_id.clone(), + MessageEventKind::TaskStart, + runtime_task_start_message(&runtime_role), + ), + telemetry, + agent_printer, + ); + + let (client, request, _agent_id) = + build_agent_client_and_member_dispatch_request(&transport_request) + .map_err(runtime_bootstrap_failure_from_anyhow)?; + let response = client + .execute_stream(request) + .await + .map_err(runtime_bootstrap_failure_from_anyhow)?; + + let (retained_orchestration_snapshot, retained_manifest_snapshot) = { + let mut orchestration_guard = startup_context + .orchestration_session + .lock() + .expect("orchestration session mutex poisoned"); + let mut manifest_guard = manifest.lock().expect("runtime manifest mutex poisoned"); + manifest_guard.mark_runtime_ownership_retained(); + orchestration_guard.touch_active(); + (orchestration_guard.clone(), manifest_guard.clone()) + }; + persist_runtime_snapshots( + &startup_context.store, + &retained_orchestration_snapshot, + &retained_manifest_snapshot, + ) + .map_err(|err| RuntimeBootstrapFailure { + exit_code: 1, + message: format!("failed to persist retained runtime ownership: {err:#}"), + })?; + + let shutdown_requested = Arc::new(AtomicBool::new(false)); + let (startup_tx, startup_rx) = tokio::sync::oneshot::channel::(); + let startup_signal = Arc::new(Mutex::new(Some(startup_tx))); + let span_id = Arc::new(Mutex::new(None::)); + + let event_store = startup_context.store.clone(); + let event_orchestration_session = Arc::clone(&startup_context.orchestration_session); + let event_manifest = Arc::clone(&manifest); + let startup_signal_for_events = Arc::clone(&startup_signal); + let shutdown_for_events = Arc::clone(&shutdown_requested); + let runtime_role_for_events = runtime_role.clone(); + let run_id_for_events = run_id.clone(); + let span_id_for_events = Arc::clone(&span_id); + + let observe_task = tokio::spawn(async move { + let mut body = std::pin::pin!(response.into_body()); + let mut buffer = Vec::new(); + let mut saw_terminal = false; + + while let Some(frame) = body.as_mut().frame().await { + let Ok(frame) = frame else { + break; + }; + let Some(data) = frame.data_ref() else { + continue; + }; + buffer.extend_from_slice(data); + + while let Some(pos) = buffer.iter().position(|&b| b == b'\n') { + let line: Vec = buffer.drain(..=pos).collect(); + if line.len() <= 1 { + continue; + } + let payload = &line[..line.len() - 1]; + if payload.is_empty() { + continue; + } + let Ok(frame) = serde_json::from_slice::(payload) else { + continue; + }; + + match frame { + ExecuteStreamFrame::Start { + span_id: stream_span_id, + } => { + *span_id_for_events + .lock() + .expect("remote member span mutex poisoned") = Some(stream_span_id); + } + ExecuteStreamFrame::Event { event } => { + let mut startup_became_live = false; + let (orchestration_snapshot, manifest_snapshot) = { + let mut orchestration_guard = event_orchestration_session + .lock() + .expect("orchestration session mutex poisoned"); + let mut manifest_guard = event_manifest + .lock() + .expect("runtime manifest mutex poisoned"); + if let Some(session_id) = extract_session_handle_id(Some(&event.data)) { + if manifest_guard.internal.uaa_session_id.as_deref() + != Some(session_id) + { + manifest_guard.set_uaa_session_id(session_id.to_string()); + } + if manifest_guard.handle.state + == AgentRuntimeSessionState::Allocating + && manifest_guard.can_advertise_live() + { + manifest_guard + .transition_state(AgentRuntimeSessionState::Ready); + manifest_guard.touch_heartbeat(); + orchestration_guard.touch_active(); + startup_became_live = true; + } + } + manifest_guard.touch_event(event.ts); + orchestration_guard.touch_active(); + (orchestration_guard.clone(), manifest_guard.clone()) + }; + let _ = persist_runtime_snapshots( + &event_store, + &orchestration_snapshot, + &manifest_snapshot, + ); + let _ = publish_agent_event(event); + if startup_became_live { + let _ = publish_agent_event(build_runtime_message_event( + &manifest_snapshot, + &orchestration_snapshot, + run_id_for_events.clone(), + MessageEventKind::Status, + runtime_ready_message(&runtime_role_for_events), + )); + signal_runtime_startup( + &startup_signal_for_events, + RuntimeStartupSignal::Running, + ); + } + } + ExecuteStreamFrame::Exit { exit, .. } => { + saw_terminal = true; + let mut startup_failure = None; + let (orchestration_snapshot, manifest_snapshot) = { + let mut orchestration_guard = event_orchestration_session + .lock() + .expect("orchestration session mutex poisoned"); + let mut manifest_guard = event_manifest + .lock() + .expect("runtime manifest mutex poisoned"); + if manifest_guard.handle.state == AgentRuntimeSessionState::Allocating { + let reason = format!( + "world-scoped member runtime exited with status {exit} before ownership could be established" + ); + manifest_guard.transition_state(AgentRuntimeSessionState::Failed); + manifest_guard.mark_terminal_state(reason.clone()); + manifest_guard.internal.last_error_bucket = + Some("bootstrap_run".to_string()); + manifest_guard.internal.last_error_message = Some(reason.clone()); + orchestration_guard.touch_active(); + startup_failure = Some(reason); + } else if shutdown_for_events.load(Ordering::SeqCst) + || matches!(exit, 0 | 129 | 130 | 131 | 143) + { + if manifest_guard.handle.state + == AgentRuntimeSessionState::Invalidated + { + orchestration_guard.touch_active(); + } else { + manifest_guard + .transition_state(AgentRuntimeSessionState::Stopped); + manifest_guard + .mark_terminal_state("world-scoped member session stopped"); + orchestration_guard.touch_active(); + } + } else { + let reason = format!( + "world-scoped member session exited with status {exit}" + ); + manifest_guard.transition_state(AgentRuntimeSessionState::Failed); + manifest_guard.mark_terminal_state(reason.clone()); + manifest_guard.internal.last_error_bucket = + Some("runtime_lifecycle".to_string()); + manifest_guard.internal.last_error_message = Some(reason.clone()); + orchestration_guard.touch_active(); + } + (orchestration_guard.clone(), manifest_guard.clone()) + }; + let _ = persist_runtime_snapshots( + &event_store, + &orchestration_snapshot, + &manifest_snapshot, + ); + if let Some(reason) = startup_failure { + signal_runtime_startup( + &startup_signal_for_events, + RuntimeStartupSignal::Failed(reason), + ); + } + break; + } + ExecuteStreamFrame::Error { message } => { + saw_terminal = true; + let mut startup_failure = None; + let (orchestration_snapshot, manifest_snapshot, alert) = { + let mut orchestration_guard = event_orchestration_session + .lock() + .expect("orchestration session mutex poisoned"); + let mut manifest_guard = event_manifest + .lock() + .expect("runtime manifest mutex poisoned"); + let reason = format!("remote member runtime failed: {message}"); + if manifest_guard.handle.state == AgentRuntimeSessionState::Allocating { + manifest_guard.transition_state(AgentRuntimeSessionState::Failed); + startup_failure = Some(reason.clone()); + } else { + manifest_guard.transition_state(AgentRuntimeSessionState::Failed); + } + manifest_guard.mark_terminal_state(reason.clone()); + manifest_guard.internal.last_error_bucket = + Some("runtime_lifecycle".to_string()); + manifest_guard.internal.last_error_message = Some(reason.clone()); + orchestration_guard.touch_active(); + let mut event = AgentEvent::alert( + manifest_guard.handle.agent_id.clone(), + manifest_guard.handle.orchestration_session_id.clone(), + run_id_for_events.clone(), + runtime_invalidated_alert_code(&runtime_role_for_events), + reason, + ); + apply_runtime_participant_lineage(&mut event, &manifest_guard); + (orchestration_guard.clone(), manifest_guard.clone(), event) + }; + let _ = persist_runtime_snapshots( + &event_store, + &orchestration_snapshot, + &manifest_snapshot, + ); + let _ = publish_agent_event(alert); + if let Some(reason) = startup_failure { + signal_runtime_startup( + &startup_signal_for_events, + RuntimeStartupSignal::Failed(reason), + ); + } + break; + } + ExecuteStreamFrame::Stdout { .. } | ExecuteStreamFrame::Stderr { .. } => {} + } + } + } + + if saw_terminal { + return; + } + + let mut publish_events = Vec::new(); + let mut startup_failure = None; + let (orchestration_snapshot, manifest_snapshot) = { + let mut orchestration_guard = event_orchestration_session + .lock() + .expect("orchestration session mutex poisoned"); + let mut manifest_guard = event_manifest + .lock() + .expect("runtime manifest mutex poisoned"); + let was_allocating = + manifest_guard.handle.state == AgentRuntimeSessionState::Allocating; + let was_live = manifest_guard.is_authoritative_live(); + manifest_guard.set_event_stream_active(false); + if was_allocating { + let reason = + "attached control turn ended before ownership could be established".to_string(); + manifest_guard.transition_state(AgentRuntimeSessionState::Failed); + manifest_guard.mark_terminal_state(reason.clone()); + manifest_guard.internal.last_error_bucket = Some("bootstrap_run".to_string()); + manifest_guard.internal.last_error_message = Some(reason.clone()); + orchestration_guard.touch_active(); + startup_failure = Some(reason); + } else if !shutdown_for_events.load(Ordering::SeqCst) && was_live { + let reason = + "world-scoped member control stream ended before completion observation" + .to_string(); + manifest_guard.transition_state(AgentRuntimeSessionState::Invalidated); + manifest_guard.mark_terminal_state(reason.clone()); + manifest_guard.internal.last_error_bucket = Some("runtime_lifecycle".to_string()); + manifest_guard.internal.last_error_message = Some(reason.clone()); + orchestration_guard.touch_active(); + let mut event = AgentEvent::alert( + manifest_guard.handle.agent_id.clone(), + manifest_guard.handle.orchestration_session_id.clone(), + run_id_for_events.clone(), + runtime_stream_closed_alert_code(&runtime_role_for_events), + reason, + ); + apply_runtime_participant_lineage(&mut event, &manifest_guard); + publish_events.push(event); + } + (orchestration_guard.clone(), manifest_guard.clone()) + }; + let _ = + persist_runtime_snapshots(&event_store, &orchestration_snapshot, &manifest_snapshot); + for event in publish_events { + let _ = publish_agent_event(event); + } + if let Some(reason) = startup_failure { + signal_runtime_startup( + &startup_signal_for_events, + RuntimeStartupSignal::Failed(reason), + ); + } + }); + + let mut observe_task = Some(observe_task); + match tokio::time::timeout(Duration::from_secs(10), startup_rx).await { + Ok(Ok(RuntimeStartupSignal::Running)) => {} + Ok(Ok(RuntimeStartupSignal::Failed(message))) => { + abort_remote_member_bootstrap_runtime( + &shutdown_requested, + &client, + &span_id, + &mut observe_task, + ) + .await; + return Err(RuntimeBootstrapFailure { + exit_code: 4, + message, + }); + } + Ok(Err(_)) => { + abort_remote_member_bootstrap_runtime( + &shutdown_requested, + &client, + &span_id, + &mut observe_task, + ) + .await; + let message = "failed to establish attached control ownership".to_string(); + mark_runtime_startup_failed( + &startup_context.store, + &startup_context.orchestration_session, + &manifest, + &message, + ); + return Err(RuntimeBootstrapFailure { + exit_code: 1, + message, + }); + } + Err(_) => { + abort_remote_member_bootstrap_runtime( + &shutdown_requested, + &client, + &span_id, + &mut observe_task, + ) + .await; + let message = "timed out waiting for world-scoped member control ownership".to_string(); + mark_runtime_startup_failed( + &startup_context.store, + &startup_context.orchestration_session, + &manifest, + &message, + ); + return Err(RuntimeBootstrapFailure { + exit_code: 4, + message, + }); + } + } + + let surfaced_uaa_session_handle_id = { + manifest + .lock() + .expect("runtime manifest mutex poisoned") + .internal + .uaa_session_id + .clone() + }; + let uaa_session_handle_id = + surfaced_uaa_session_handle_id.ok_or_else(|| RuntimeBootstrapFailure { + exit_code: 1, + message: "runtime startup signalled ready without a surfaced UAA session handle" + .to_string(), + })?; + let resolved_span_id = span_id + .lock() + .expect("remote member span mutex poisoned") + .clone() + .ok_or_else(|| RuntimeBootstrapFailure { + exit_code: 1, + message: "runtime startup signalled ready without a streamed execute span_id" + .to_string(), + })?; + let (stop_tx, stop_rx) = private_stop_request_channel(); + let (stop_orchestration_session_id, stop_participant_id) = + runtime_stop_transport_ids(&manifest); + let stop_transport = match register_private_stop_transport( + &startup_context.store, + &stop_orchestration_session_id, + &stop_participant_id, + stop_tx, + ) + .await + { + Ok(stop_transport) => stop_transport, + Err(err) => { + abort_remote_member_bootstrap_runtime( + &shutdown_requested, + &client, + &span_id, + &mut observe_task, + ) + .await; + let message = format!("failed to register private stop transport: {err:#}"); + mark_runtime_startup_failed( + &startup_context.store, + &startup_context.orchestration_session, + &manifest, + &message, + ); + return Err(RuntimeBootstrapFailure { + exit_code: 1, + message, + }); + } + }; + let client = Arc::new(client); + let stop_owner_task = spawn_remote_private_stop_owner( + Arc::clone(&manifest), + Arc::clone(&shutdown_requested), + Arc::clone(&client), + resolved_span_id.clone(), + stop_rx, + ); + let (prompt_tx, prompt_rx) = private_prompt_request_channel(); + let prompt_transport = match register_private_prompt_transport( + &startup_context.store, + &stop_orchestration_session_id, + &stop_participant_id, + prompt_tx, + ) + .await + { + Ok(prompt_transport) => prompt_transport, + Err(err) => { + abort_remote_member_bootstrap_runtime( + &shutdown_requested, + &client, + &span_id, + &mut observe_task, + ) + .await; + let message = format!("failed to register private prompt transport: {err:#}"); + mark_runtime_startup_failed( + &startup_context.store, + &startup_context.orchestration_session, + &manifest, + &message, + ); + return Err(RuntimeBootstrapFailure { + exit_code: 1, + message, + }); + } + }; + let prompt_owner_task = spawn_remote_private_prompt_owner( + prompt_runtime_from_parts( + descriptor.clone(), + Arc::clone(&startup_context.orchestration_session), + Arc::clone(&manifest), + startup_context.store.clone(), + uaa_session_handle_id.clone(), + None, + ), + prompt_rx, + ); + + Ok(Some(AsyncReplAgentRuntime { + descriptor, + orchestration_session: startup_context.orchestration_session, + manifest, + store: startup_context.store, + uaa_session_handle_id, + retained_control: RetainedRunControl::Remote(RemoteRetainedRunControl { + client, + span_id: resolved_span_id, + observe_task, + }), + shutdown_requested, + auto_park_rx: None, + private_stop_rx: None, + stop_transport: Some(stop_transport), + stop_owner_task: Some(stop_owner_task), + prompt_transport: Some(prompt_transport), + prompt_owner_task: Some(prompt_owner_task), + heartbeat_stop_tx: None, + heartbeat_task: None, + })) +} + +async fn reconcile_member_runtime_generation( + world_session: Option<&WorldSession>, + member_runtimes: &mut RetainedMemberRuntimeMap, + pending_member_replacements: &mut PendingMemberReplacementMap, + agent_printer: &ReplPrinter, + telemetry: &mut ReplSessionTelemetry, +) -> Result<()> { + let Some(world_session) = world_session else { + return Ok(()); + }; + let stale_backends = member_runtimes + .iter() + .filter_map(|(backend_id, runtime)| { + let manifest_snapshot = runtime_manifest_snapshot(runtime); + (manifest_snapshot.handle.role == MEMBER_ROLE + && manifest_snapshot.handle.world_generation + != Some(world_session.world_generation)) + .then_some(backend_id.clone()) + }) + .collect::>(); + + for backend_id in stale_backends { + let Some(runtime) = member_runtimes.remove(&backend_id) else { + continue; + }; + let (orchestration_snapshot, invalidated_manifest) = { + let mut manifest_guard = runtime + .manifest + .lock() + .expect("runtime manifest mutex poisoned"); + if manifest_guard.handle.state != AgentRuntimeSessionState::Invalidated { + let _ = manifest_guard.invalidate_for_world_generation_rollover(); + } + let orchestration_snapshot = runtime + .orchestration_session + .lock() + .expect("orchestration session mutex poisoned") + .clone(); + (orchestration_snapshot, manifest_guard.clone()) + }; + persist_runtime_snapshots( + &runtime.store, + &orchestration_snapshot, + &invalidated_manifest, + ) + .with_context(|| { + format!( + "persist invalidated stale member runtime state after world generation rollover for backend '{}'", + backend_id + ) + })?; + pending_member_replacements.insert(backend_id, invalidated_manifest.clone()); + shutdown_host_orchestrator_runtime(runtime, agent_printer, telemetry).await; + } + Ok(()) +} + +#[cfg(any(target_os = "linux", target_os = "macos"))] +struct EnsureMemberRuntimeReadyContext<'a> { + startup_context: Option<&'a RuntimeOrchestrationContext>, + world_session: Option<&'a WorldSession>, + agent_printer: &'a ReplPrinter, + telemetry: &'a mut ReplSessionTelemetry, +} + +#[cfg(any(target_os = "linux", target_os = "macos"))] +async fn ensure_member_runtime_ready_for_descriptor( + context: EnsureMemberRuntimeReadyContext<'_>, + descriptor: &RuntimeSelectionDescriptor, + initial_prompt: Option<&str>, + member_runtimes: &mut RetainedMemberRuntimeMap, + pending_member_replacements: &mut PendingMemberReplacementMap, +) -> Result { + let EnsureMemberRuntimeReadyContext { + startup_context, + world_session, + agent_printer, + telemetry, + } = context; + let Some(startup_context) = startup_context else { + return Ok(false); + }; + let Some(world_session) = world_session else { + return Ok(false); + }; + ensure_member_backend_allowed(startup_context, descriptor) + .map_err(|failure| anyhow!("substrate: error: {}", failure.message))?; + + let exact_live_member = + live_member_for_generation(startup_context, descriptor, world_session.world_generation) + .map_err(|failure| anyhow!("substrate: error: {}", failure.message))?; + + if let Some(runtime) = member_runtimes.get(&descriptor.backend_id) { + let manifest_snapshot = runtime_manifest_snapshot(runtime); + if manifest_snapshot.handle.role == MEMBER_ROLE + && manifest_snapshot.handle.orchestration_session_id + == startup_context.orchestration_session_id() + && manifest_snapshot.handle.backend_id == descriptor.backend_id + && manifest_snapshot.handle.world_generation == Some(world_session.world_generation) + && manifest_snapshot.is_authoritative_live() + && exact_live_member.as_ref().is_some_and(|participant| { + participant.handle.participant_id == manifest_snapshot.handle.participant_id + }) + { + return Ok(false); + } + } + + if let Some(runtime) = member_runtimes.remove(&descriptor.backend_id) { + shutdown_host_orchestrator_runtime(runtime, agent_printer, telemetry).await; + } + + let resumed_from = pending_member_replacements + .get(&descriptor.backend_id) + .filter(|participant| { + participant.handle.orchestration_session_id + == startup_context.orchestration_session_id() + && participant.handle.backend_id == descriptor.backend_id + }); + let prepared = prepare_member_runtime_startup_for_descriptor( + startup_context, + descriptor.clone(), + &PersistedWorldBinding { + world_id: world_session.world_id.clone(), + world_generation: world_session.world_generation, + }, + resumed_from, + ) + .map_err(|failure| anyhow!("substrate: error: {}", failure.message))?; + + let runtime = match start_remote_member_runtime_with_prepared( + Some(prepared), + initial_prompt.map(str::to_string), + agent_printer, + telemetry, + ) + .await + { + Ok(runtime) => runtime, + Err(failure) => return Err(anyhow!("substrate: error: {}", failure.message)), + }; + if let Some(runtime) = runtime { + pending_member_replacements.remove(&descriptor.backend_id); + member_runtimes.insert(runtime_backend_id(&runtime), runtime); + return Ok(true); + } + Ok(false) +} + +#[cfg(not(any(target_os = "linux", target_os = "macos")))] +struct EnsureMemberRuntimeReadyContext<'a> { + startup_context: Option<&'a RuntimeOrchestrationContext>, + world_session: Option<&'a WorldSession>, + agent_printer: &'a ReplPrinter, + telemetry: &'a mut ReplSessionTelemetry, +} + +#[cfg(not(any(target_os = "linux", target_os = "macos")))] +async fn ensure_member_runtime_ready_for_descriptor( + context: EnsureMemberRuntimeReadyContext<'_>, + _descriptor: &RuntimeSelectionDescriptor, + _initial_prompt: Option<&str>, + _member_runtimes: &mut RetainedMemberRuntimeMap, + _pending_member_replacements: &mut PendingMemberReplacementMap, +) -> Result { + let EnsureMemberRuntimeReadyContext { + startup_context, + world_session, + agent_printer: _agent_printer, + telemetry: _telemetry, + } = context; + let Some(_startup_context) = startup_context else { + return Ok(false); + }; + let Some(_world_session) = world_session else { + return Ok(false); + }; + + Err(anyhow!( + "substrate: error: world-scoped member runtime dispatch is supported on Linux only" + )) +} + +#[cfg(any(target_os = "linux", target_os = "macos"))] +async fn ensure_member_runtime_ready( + startup_context: Option<&RuntimeOrchestrationContext>, + world_session: Option<&WorldSession>, + member_runtimes: &mut RetainedMemberRuntimeMap, + pending_member_replacements: &mut PendingMemberReplacementMap, + agent_printer: &ReplPrinter, + telemetry: &mut ReplSessionTelemetry, +) -> Result<()> { + let Some(startup_context) = startup_context else { + return Ok(()); + }; + let Some(descriptor) = select_member_runtime_descriptor(startup_context) + .map_err(|failure| anyhow!("substrate: error: {}", failure.message))? + else { + return Ok(()); + }; + + ensure_member_runtime_ready_for_descriptor( + EnsureMemberRuntimeReadyContext { + startup_context: Some(startup_context), + world_session, + agent_printer, + telemetry, + }, + &descriptor, + None, + member_runtimes, + pending_member_replacements, + ) + .await + .map(|_| ()) +} + +#[cfg(not(any(target_os = "linux", target_os = "macos")))] +async fn ensure_member_runtime_ready( + startup_context: Option<&RuntimeOrchestrationContext>, + world_session: Option<&WorldSession>, + _member_runtimes: &mut RetainedMemberRuntimeMap, + _pending_member_replacements: &mut PendingMemberReplacementMap, + _agent_printer: &ReplPrinter, + _telemetry: &mut ReplSessionTelemetry, +) -> Result<()> { + let Some(_startup_context) = startup_context else { + return Ok(()); + }; + let Some(_world_session) = world_session else { + return Ok(()); + }; + + Err(anyhow!( + "substrate: error: world-scoped member runtime dispatch is supported on Linux only" + )) +} + +async fn shutdown_all_member_runtimes( + member_runtimes: &mut RetainedMemberRuntimeMap, + agent_printer: &ReplPrinter, + telemetry: &mut ReplSessionTelemetry, +) { + while let Some((_, runtime)) = member_runtimes.pop_first() { + shutdown_host_orchestrator_runtime(runtime, agent_printer, telemetry).await; + } +} + +fn note_submitted_turn_started( + runtime: &AsyncReplAgentRuntime, + run_id: &str, + message: &str, + telemetry: &mut ReplSessionTelemetry, + agent_printer: &ReplPrinter, +) -> Result<()> { + let (orchestration_snapshot, manifest_snapshot, event) = { + let mut orchestration_guard = runtime + .orchestration_session + .lock() + .expect("orchestration session mutex poisoned"); + let mut manifest_guard = runtime + .manifest + .lock() + .expect("runtime manifest mutex poisoned"); + manifest_guard.internal.latest_run_id = Some(run_id.to_string()); + orchestration_guard.touch_active(); + let event = build_runtime_message_event( + &manifest_guard, + &orchestration_guard, + run_id.to_string(), + MessageEventKind::TaskStart, + message, + ); + (orchestration_guard.clone(), manifest_guard.clone(), event) + }; + persist_runtime_snapshots(&runtime.store, &orchestration_snapshot, &manifest_snapshot)?; + emit_runtime_event(event, telemetry, agent_printer); + Ok(()) +} + +fn note_submitted_turn_completed( + runtime: &AsyncReplAgentRuntime, + run_id: &str, + message: impl Into, + telemetry: &mut ReplSessionTelemetry, + agent_printer: &ReplPrinter, +) -> Result<()> { + let (orchestration_snapshot, manifest_snapshot, event) = { + let mut orchestration_guard = runtime + .orchestration_session + .lock() + .expect("orchestration session mutex poisoned"); + let manifest_guard = runtime + .manifest + .lock() + .expect("runtime manifest mutex poisoned"); + orchestration_guard.touch_active(); + let event = build_runtime_message_event( + &manifest_guard, + &orchestration_guard, + run_id.to_string(), + MessageEventKind::Status, + message, + ); + (orchestration_guard.clone(), manifest_guard.clone(), event) + }; + persist_runtime_snapshots(&runtime.store, &orchestration_snapshot, &manifest_snapshot)?; + emit_runtime_event(event, telemetry, agent_printer); + Ok(()) +} + +async fn submit_host_targeted_turn( + runtime: &mut AsyncReplAgentRuntime, + prompt: &str, + agent_printer: &ReplPrinter, + telemetry: &mut ReplSessionTelemetry, +) -> Result<()> { + let run_id = Uuid::now_v7().to_string(); + note_submitted_turn_started( + runtime, + &run_id, + format!( + "submitted targeted follow-up turn to {}", + runtime.descriptor.backend_id + ) + .as_str(), + telemetry, + agent_printer, + )?; + let completion = submit_host_prompt_turn( + &runtime_prompt_submit_runtime(runtime), + &run_id, + prompt, + |event| match event { + SubmittedPromptStreamEvent::Agent(event) => { + emit_runtime_event(*event, telemetry, agent_printer); + } + SubmittedPromptStreamEvent::Stdout(text) | SubmittedPromptStreamEvent::Stderr(text) => { + agent_printer.print(text); + } + }, + ) + .await?; + if let Some(message) = completion.warning.as_deref() { + write_best_effort_stderr_line(message); + } + note_submitted_turn_completed( + runtime, + &run_id, + if completion.exit_code == 0 { + format!( + "targeted follow-up turn completed for {}", + runtime.descriptor.backend_id + ) + } else { + format!( + "targeted follow-up turn exited with status {} for {}", + completion.exit_code, runtime.descriptor.backend_id + ) + }, + telemetry, + agent_printer, + )?; + Ok(()) +} + +#[cfg(any(target_os = "linux", target_os = "macos"))] +async fn submit_world_targeted_turn( + runtime: &mut AsyncReplAgentRuntime, + prompt: &str, + agent_printer: &ReplPrinter, + telemetry: &mut ReplSessionTelemetry, +) -> Result<()> { + use base64::engine::general_purpose::STANDARD as BASE64; + use base64::Engine; + use http_body_util::BodyExt as _; + + let run_id = Uuid::now_v7().to_string(); + note_submitted_turn_started( + runtime, + &run_id, + format!( + "submitted targeted follow-up turn to {}", + runtime.descriptor.backend_id + ) + .as_str(), + telemetry, + agent_printer, + )?; + let request = { + let manifest_guard = runtime + .manifest + .lock() + .expect("runtime manifest mutex poisoned"); + MemberTurnSubmitRequestV1 { + schema_version: 1, + orchestration_session_id: manifest_guard.handle.orchestration_session_id.clone(), + participant_id: manifest_guard.handle.participant_id.clone(), + orchestrator_participant_id: manifest_guard + .handle + .orchestrator_participant_id + .clone() + .ok_or_else(|| { + anyhow!( + "substrate: error: retained world-scoped member is missing orchestrator_participant_id" + ) + })?, + backend_id: runtime.descriptor.backend_id.clone(), + run_id: run_id.clone(), + world_id: manifest_guard.handle.world_id.clone().ok_or_else(|| { + anyhow!("substrate: error: retained world-scoped member is missing world_id") + })?, + world_generation: manifest_guard.handle.world_generation.ok_or_else(|| { + anyhow!( + "substrate: error: retained world-scoped member is missing world_generation" + ) + })?, + prompt: prompt.to_string(), + } + }; + let (client, _pending_diff_request, _agent_id) = build_agent_client_and_pending_diff_request()?; + let response = client + .submit_member_turn_stream(request) + .await + .map_err(|err| anyhow!("substrate: error: {err:#}"))?; + + let mut body = std::pin::pin!(response.into_body()); + let mut buffer = Vec::new(); + let mut exit_code = 0; + while let Some(frame) = body.as_mut().frame().await { + let frame = frame.map_err(|err| anyhow!("substrate: error: {err:#}"))?; + let Some(data) = frame.data_ref() else { + continue; + }; + buffer.extend_from_slice(data); + + while let Some(pos) = buffer.iter().position(|&byte| byte == b'\n') { + let line: Vec = buffer.drain(..=pos).collect(); + if line.len() <= 1 { + continue; + } + let payload = &line[..line.len() - 1]; + if payload.is_empty() { + continue; + } + let frame = serde_json::from_slice::(payload) + .map_err(|err| anyhow!("substrate: error: {err:#}"))?; + match frame { + ExecuteStreamFrame::Start { .. } => {} + ExecuteStreamFrame::Event { event } => { + handle_agent_event(event, telemetry, agent_printer); + } + ExecuteStreamFrame::Stdout { chunk_b64 } => { + let decoded = BASE64 + .decode(chunk_b64.as_bytes()) + .map_err(|err| anyhow!("substrate: error: {err:#}"))?; + agent_printer.print(String::from_utf8_lossy(&decoded).to_string()); + } + ExecuteStreamFrame::Stderr { chunk_b64 } => { + let decoded = BASE64 + .decode(chunk_b64.as_bytes()) + .map_err(|err| anyhow!("substrate: error: {err:#}"))?; + agent_printer.print(String::from_utf8_lossy(&decoded).to_string()); + } + ExecuteStreamFrame::Exit { exit, .. } => { + exit_code = exit; + } + ExecuteStreamFrame::Error { message } => { + return Err(anyhow!("substrate: error: {message}")); + } + } + } + } + if exit_code != 0 { + write_best_effort_stderr_line(&format!("Command failed with status: {exit_code}")); + } + note_submitted_turn_completed( + runtime, + &run_id, + if exit_code == 0 { + format!( + "targeted follow-up turn completed for {}", + runtime.descriptor.backend_id + ) + } else { + format!( + "targeted follow-up turn exited with status {} for {}", + exit_code, runtime.descriptor.backend_id + ) + }, + telemetry, + agent_printer, + )?; + Ok(()) +} + +#[cfg(not(any(target_os = "linux", target_os = "macos")))] +async fn submit_world_targeted_turn( + _runtime: &mut AsyncReplAgentRuntime, + _prompt: &str, + _agent_printer: &ReplPrinter, + _telemetry: &mut ReplSessionTelemetry, +) -> Result<()> { + Err(anyhow!( + "substrate: error: world-targeted follow-up turns are supported on Linux and macOS only" + )) +} + +async fn shutdown_host_orchestrator_runtime( + runtime: AsyncReplAgentRuntime, + agent_printer: &ReplPrinter, + telemetry: &mut ReplSessionTelemetry, +) { + shutdown_host_orchestrator_runtime_with_mode( + runtime, + HostRuntimeShutdownMode::Stop, + agent_printer, + telemetry, + ) + .await; +} + +async fn shutdown_host_orchestrator_runtime_with_mode( + mut runtime: AsyncReplAgentRuntime, + mode: HostRuntimeShutdownMode, + agent_printer: &ReplPrinter, + telemetry: &mut ReplSessionTelemetry, +) { + let (run_id, should_attempt_stop, runtime_role) = { + let guard = runtime + .manifest + .lock() + .expect("runtime manifest mutex poisoned"); + ( + guard + .internal + .latest_run_id + .clone() + .unwrap_or_else(|| Uuid::now_v7().to_string()), + guard.internal.control_owner_retained || guard.internal.completion_observer_retained, + guard.handle.role.clone(), + ) + }; + let controls_parent_session = runtime_controls_parent_session(&runtime_role); + if mode == HostRuntimeShutdownMode::ParkIfResumable + && park_host_orchestrator_runtime( + &mut runtime, + controls_parent_session, + &run_id, + &runtime_role, + agent_printer, + telemetry, + ) + .await + { + return; + } + + if should_attempt_stop { + let (orchestration_session, manifest) = { + let mut orchestration_guard = runtime + .orchestration_session + .lock() + .expect("orchestration session mutex poisoned"); + let mut manifest_guard = runtime + .manifest + .lock() + .expect("runtime manifest mutex poisoned"); + if manifest_guard.handle.state.is_live() { + manifest_guard.transition_state(AgentRuntimeSessionState::Stopping); + manifest_guard.touch_heartbeat(); + } + if controls_parent_session + && orchestration_guard.state == OrchestrationSessionState::Active + { + orchestration_guard.transition_state(OrchestrationSessionState::Stopping); + } + (orchestration_guard.clone(), manifest_guard.clone()) + }; + let _ = persist_runtime_snapshots(&runtime.store, &orchestration_session, &manifest); + emit_runtime_event( + build_runtime_message_event( + &manifest, + &orchestration_session, + run_id.clone(), + MessageEventKind::Status, + runtime_stopping_message(&runtime_role, &runtime.uaa_session_handle_id), + ), + telemetry, + agent_printer, + ); + } + + if let Some(mut stop_transport) = runtime.stop_transport.take() { + stop_transport.close().await; + } + if let Some(task) = runtime.stop_owner_task.take() { + let _ = task.await; + } + if let Some(mut prompt_transport) = runtime.prompt_transport.take() { + prompt_transport.close().await; + } + if let Some(task) = runtime.prompt_owner_task.take() { + let _ = task.await; + } + runtime.shutdown_requested.store(true, Ordering::SeqCst); + if let Some(stop_tx) = runtime.heartbeat_stop_tx.take() { + let _ = stop_tx.send(()); + } + let mut stop_failed = false; + let mut completion_observed = false; + match &mut runtime.retained_control { + RetainedRunControl::Local(retained_control) => { + retained_control.cancel.cancel(); + if let Some(task) = retained_control.completion_task.take() { + match tokio::time::timeout(Duration::from_secs(5), task).await { + Ok(Ok(())) => completion_observed = true, + Ok(Err(_)) | Err(_) => stop_failed = true, + } + } + if let Some(task) = retained_control.event_task.take() { + let _ = tokio::time::timeout(Duration::from_secs(5), task).await; + } + } + #[cfg(any(target_os = "linux", target_os = "macos"))] + RetainedRunControl::Remote(retained_control) => { + if retained_control + .client + .cancel_execute(ExecuteCancelRequestV1 { + span_id: retained_control.span_id.clone(), + sig: "INT".to_string(), + }) + .await + .is_err() + { + stop_failed = true; + } + if let Some(task) = retained_control.observe_task.take() { + match tokio::time::timeout(Duration::from_secs(5), task).await { + Ok(Ok(())) => completion_observed = true, + Ok(Err(_)) | Err(_) => stop_failed = true, + } + } + } + } + if let Some(task) = runtime.heartbeat_task.take() { + let _ = task.await; + } + if stop_failed || (should_attempt_stop && !completion_observed) { + let reason = + "shell-owned orchestrator cancel did not produce authoritative terminal completion" + .to_string(); + let (orchestration_session, manifest) = { + let mut orchestration_guard = runtime + .orchestration_session + .lock() + .expect("orchestration session mutex poisoned"); + let mut manifest_guard = runtime + .manifest + .lock() + .expect("runtime manifest mutex poisoned"); + manifest_guard.transition_state(AgentRuntimeSessionState::Failed); + manifest_guard.mark_terminal_state(reason.clone()); + manifest_guard.internal.last_error_bucket = Some("runtime_shutdown".to_string()); + manifest_guard.internal.last_error_message = Some(reason.clone()); + if controls_parent_session { + orchestration_guard.transition_state(OrchestrationSessionState::Failed); + orchestration_guard.mark_terminal(reason.clone()); + } else { + orchestration_guard.touch_active(); + } + (orchestration_guard.clone(), manifest_guard.clone()) + }; + let _ = persist_runtime_snapshots(&runtime.store, &orchestration_session, &manifest); + let mut event = AgentEvent::alert( + manifest.handle.agent_id.clone(), + manifest.handle.orchestration_session_id.clone(), + run_id, + runtime_invalidated_alert_code(&runtime_role), + reason, + ); + apply_runtime_participant_lineage(&mut event, &manifest); + emit_runtime_event(event, telemetry, agent_printer); + return; + } + + let manifest = runtime + .manifest + .lock() + .expect("runtime manifest mutex poisoned") + .clone(); + if should_attempt_stop && manifest.handle.state == AgentRuntimeSessionState::Stopped { + let orchestration_session = runtime + .orchestration_session + .lock() + .expect("orchestration session mutex poisoned") + .clone(); + emit_runtime_event( + build_runtime_message_event( + &manifest, + &orchestration_session, + run_id, + MessageEventKind::Status, + runtime_stopped_message(&runtime_role), + ), + telemetry, + agent_printer, + ); + } +} + +async fn park_host_orchestrator_runtime( + runtime: &mut AsyncReplAgentRuntime, + controls_parent_session: bool, + run_id: &str, + runtime_role: &str, + agent_printer: &ReplPrinter, + telemetry: &mut ReplSessionTelemetry, +) -> bool { + let can_park = { + let manifest = runtime + .manifest + .lock() + .expect("runtime manifest mutex poisoned"); + controls_parent_session + && manifest.is_host_orchestrator() + && manifest.handle.state.is_live() + && manifest.internal.uaa_session_id.is_some() + }; + if !can_park { + return false; + } + + if let Some(mut stop_transport) = runtime.stop_transport.take() { + stop_transport.close().await; + } + if let Some(task) = runtime.stop_owner_task.take() { + task.abort(); + let _ = task.await; + } + if let Some(mut prompt_transport) = runtime.prompt_transport.take() { + prompt_transport.close().await; + } + if let Some(task) = runtime.prompt_owner_task.take() { + task.abort(); + let _ = task.await; + } + runtime.shutdown_requested.store(true, Ordering::SeqCst); + if let Some(stop_tx) = runtime.heartbeat_stop_tx.take() { + let _ = stop_tx.send(()); + } + if let Some(task) = runtime.heartbeat_task.take() { + task.abort(); + let _ = task.await; + } + + match &mut runtime.retained_control { + RetainedRunControl::Local(retained_control) => { + if let Some(task) = retained_control.completion_task.take() { + task.abort(); + let _ = task.await; + } + if let Some(task) = retained_control.event_task.take() { + task.abort(); + let _ = task.await; + } + } + #[cfg(any(target_os = "linux", target_os = "macos"))] + RetainedRunControl::Remote(retained_control) => { + if let Some(task) = retained_control.observe_task.take() { + task.abort(); + let _ = task.await; + } + } + } + + let reason = "owner detached cleanly".to_string(); + let Some((orchestration_session, manifest)) = ({ + let mut orchestration_guard = runtime + .orchestration_session + .lock() + .expect("orchestration session mutex poisoned"); + let mut manifest_guard = runtime + .manifest + .lock() + .expect("runtime manifest mutex poisoned"); + apply_parked_host_runtime_snapshots(&mut orchestration_guard, &mut manifest_guard, &reason) + }) else { + return false; + }; + let _ = persist_runtime_snapshots(&runtime.store, &orchestration_session, &manifest); + emit_runtime_event( + build_runtime_message_event( + &manifest, + &orchestration_session, + run_id.to_string(), + MessageEventKind::Status, + runtime_detached_message(runtime_role), + ), + telemetry, + agent_printer, + ); + true +} + +fn can_park_host_runtime_after_detach( + store: &AgentRuntimeStateStore, + controls_parent_session: bool, + orchestration_session: &OrchestrationSessionRecord, + manifest: &AgentRuntimeSessionManifest, + runtime_owns_private_stop: bool, + owner_helper_exited: bool, +) -> bool { + if !controls_parent_session + || !manifest.is_host_orchestrator() + || !manifest.handle.state.is_live() + || manifest.internal.uaa_session_id.is_none() + { + return false; + } + + if !runtime_owns_private_stop { + return true; + } + if !owner_helper_exited { + return false; + } + + let Ok(Some(record)) = store.load_session(&orchestration_session.orchestration_session_id) + else { + return false; + }; + let participant_id = manifest.handle.participant_id.as_str(); + if record.session.state != OrchestrationSessionState::Active + || record.session.posture != OrchestrationSessionPosture::ActiveAttached + || record.session.active_participant_id() != Some(participant_id) + || record.session.attached_participant_id() != Some(participant_id) + { + return false; + } + + !record.participants.iter().any(|participant| { + participant.participant_id() != participant_id + && participant.matches_public_parent_linkage(&record.session) + && participant.is_host_orchestrator() + && participant.attached_client_present() + && participant.is_authoritative_live() + && shell_owner_pid_is_alive(participant.internal.shell_owner_pid) + }) +} + +fn build_parked_host_runtime_snapshots( + orchestration_guard: &OrchestrationSessionRecord, + manifest_guard: &AgentRuntimeSessionManifest, + reason: &str, +) -> Option<(OrchestrationSessionRecord, AgentRuntimeSessionManifest)> { + let mut parked_orchestration = orchestration_guard.clone(); + let mut parked_manifest = manifest_guard.clone(); + parked_orchestration.shell_owner_pid = 0; + parked_manifest.release_runtime_ownership(); + parked_manifest.mark_client_detached(reason.to_string()); + parked_manifest.internal.shell_owner_pid = 0; + parked_manifest.touch_heartbeat(); + parked_orchestration.transition_state(OrchestrationSessionState::Active); + match if parked_orchestration.pending_inbox_count > 0 { + OrchestrationSessionPosture::AwaitingAttention + } else { + OrchestrationSessionPosture::ParkedResumable + } { + OrchestrationSessionPosture::ParkedResumable => { + parked_orchestration.mark_parked_resumable(reason.to_string()); + } + OrchestrationSessionPosture::AwaitingAttention => { + parked_orchestration.mark_awaiting_attention(); + } + OrchestrationSessionPosture::ActiveAttached | OrchestrationSessionPosture::Terminal => { + unreachable!("detached host parking only normalizes to parked or attention posture") + } + } + + valid_detached_host_continuity_posture(&parked_orchestration, &parked_manifest, true)?; + Some((parked_orchestration, parked_manifest)) +} + +fn apply_parked_host_runtime_snapshots( + orchestration_guard: &mut OrchestrationSessionRecord, + manifest_guard: &mut AgentRuntimeSessionManifest, + reason: &str, +) -> Option<(OrchestrationSessionRecord, AgentRuntimeSessionManifest)> { + let (next_orchestration, next_manifest) = + build_parked_host_runtime_snapshots(orchestration_guard, manifest_guard, reason)?; + *orchestration_guard = next_orchestration.clone(); + *manifest_guard = next_manifest.clone(); + Some((next_orchestration, next_manifest)) +} + +#[cfg(unix)] +fn shell_owner_pid_is_alive(pid: u32) -> bool { + let pid = pid as libc::pid_t; + if pid <= 0 { + return false; + } + + let rc = unsafe { libc::kill(pid, 0) }; + if rc == 0 { + return true; + } + + matches!(io::Error::last_os_error().raw_os_error(), Some(libc::EPERM)) +} + +#[cfg(not(unix))] +fn shell_owner_pid_is_alive(pid: u32) -> bool { + pid == std::process::id() +} + +async fn abort_bootstrap_runtime( + shutdown_requested: &Arc, + retained_control: &mut LocalRetainedRunControl, + heartbeat_stop_tx: &mut Option>, + heartbeat_task: &mut Option>, +) { + shutdown_requested.store(true, Ordering::SeqCst); + if let Some(stop_tx) = heartbeat_stop_tx.take() { + let _ = stop_tx.send(()); + } + retained_control.cancel.cancel(); + if let Some(task) = retained_control.completion_task.take() { + let _ = task.await; + } + if let Some(task) = retained_control.event_task.take() { + let _ = task.await; + } + if let Some(task) = heartbeat_task.take() { + let _ = task.await; + } +} + +fn signal_runtime_startup( + signal: &Arc>>>, + value: RuntimeStartupSignal, +) { + if let Some(tx) = signal + .lock() + .expect("runtime startup signal mutex poisoned") + .take() + { + let _ = tx.send(value); + } +} + +async fn finalize_runtime_startup_failure( + startup_context: Option<&RuntimeOrchestrationContext>, + world_session: &mut Option, + failure_message: &str, +) { + let mut close_succeeded = false; + if let Some(session) = world_session.take() { + close_succeeded = session.client.close().await.is_ok(); + } + + let Some(startup_context) = startup_context else { + return; + }; + + mark_orchestration_session_failed( + &startup_context.store, + &startup_context.orchestration_session, + failure_message.to_string(), + ); + if close_succeeded { + let _ = persist_world_binding_authority( + &startup_context.store, + &startup_context.orchestration_session, + None, + ); + } +} + +fn runtime_loop_exit(err: &anyhow::Error) -> (i32, String) { + if is_world_restart_required_error(err) { + return (3, err.to_string()); + } + + let message = format!("{err:#}"); + if message.starts_with("substrate: error:") { + (1, message) + } else { + (1, format!("substrate: error: {message}")) + } +} + +fn runtime_bootstrap_failure_from_anyhow(err: anyhow::Error) -> RuntimeBootstrapFailure { + if crate::execution::config_model::is_user_error(&err) { + RuntimeBootstrapFailure { + exit_code: 2, + message: err.to_string(), + } + } else { + RuntimeBootstrapFailure { + exit_code: 1, + message: format!("{err:#}"), + } + } +} fn runtime_bootstrap_failure_from_wrapper_error( err: agent_api::AgentWrapperError, @@ -1966,6 +6111,7 @@ fn runtime_bootstrap_failure_from_wrapper_error( fn translate_wrapper_event( manifest: &AgentRuntimeSessionManifest, + orchestration_session: &OrchestrationSessionRecord, run_id: &str, wrapper_event: agent_api::AgentWrapperEvent, ) -> AgentEvent { @@ -2024,6 +6170,9 @@ fn translate_wrapper_event( event.backend_id = Some(manifest.handle.backend_id.clone()); event.set_pure_agent_telemetry_identity(manifest.handle.agent_id.clone()); event.set_channel(wrapper_event.channel.clone()); + event.world_id = orchestration_session.world_id.clone(); + event.world_generation = orchestration_session.world_generation; + apply_runtime_participant_lineage(&mut event, manifest); if let Some(data) = wrapper_event.data { if let Some(obj) = event.data.as_object_mut() { @@ -2038,8 +6187,18 @@ fn translate_wrapper_event( event } +fn apply_runtime_participant_lineage( + event: &mut AgentEvent, + manifest: &AgentRuntimeSessionManifest, +) { + event.participant_id = Some(manifest.handle.participant_id.clone()); + event.parent_participant_id = manifest.handle.parent_participant_id.clone(); + event.resumed_from_participant_id = manifest.handle.resumed_from_participant_id.clone(); +} + fn build_runtime_message_event( manifest: &AgentRuntimeSessionManifest, + orchestration_session: &OrchestrationSessionRecord, run_id: String, kind: MessageEventKind, message: impl Into, @@ -2054,6 +6213,9 @@ fn build_runtime_message_event( event.role = Some(manifest.handle.role.clone()); event.backend_id = Some(manifest.handle.backend_id.clone()); event.set_pure_agent_telemetry_identity(manifest.handle.agent_id.clone()); + event.world_id = orchestration_session.world_id.clone(); + event.world_generation = orchestration_session.world_generation; + apply_runtime_participant_lineage(&mut event, manifest); event } @@ -2067,13 +6229,21 @@ fn emit_runtime_event( fn extract_session_handle_id(data: Option<&serde_json::Value>) -> Option<&str> { let value = data?; - if value.get("schema").and_then(serde_json::Value::as_str) != Some(SESSION_HANDLE_SCHEMA_V1) { - return None; + if value.get("schema").and_then(serde_json::Value::as_str) == Some(SESSION_HANDLE_SCHEMA_V1) { + return value + .get("session") + .and_then(serde_json::Value::as_object) + .and_then(|session| session.get("id")) + .and_then(serde_json::Value::as_str) + .filter(|id| !id.trim().is_empty()); } + + value + .get("type") + .and_then(serde_json::Value::as_str) + .filter(|event_type| matches!(*event_type, "thread.started" | "turn.started"))?; value - .get("session") - .and_then(serde_json::Value::as_object) - .and_then(|session| session.get("id")) + .get("thread_id") .and_then(serde_json::Value::as_str) .filter(|id| !id.trim().is_empty()) } @@ -2098,6 +6268,20 @@ fn report_nonzero_status(status: &ExitStatus) { )); } +fn parse_targeted_turn(input: &str) -> Option> { + if has_embedded_newlines(input) { + return None; + } + + let rest = input.strip_prefix("::")?; + let (backend_id, prompt) = rest.split_once(' ')?; + if backend_id.is_empty() || prompt.is_empty() { + return None; + } + + Some(TargetedTurn { backend_id, prompt }) +} + fn parse_demo_burst(input: &str) -> Option<(usize, usize, u64)> { let rest = input.strip_prefix(":demo-burst")?.trim(); if rest.is_empty() { @@ -2249,15 +6433,19 @@ fn classify_world_restart_reason( } fn emit_world_restarted_alert( + orchestration_session_id: Option<&str>, previous_world_id: &str, previous_world_generation: u64, new_world_id: &str, new_world_generation: u64, reason: WorldRestartReason, ) { + let Some(orchestration_session_id) = orchestration_session_id else { + return; + }; let mut event = AgentEvent::alert( "shell", - orchestration_session_id(), + orchestration_session_id.to_string(), Uuid::now_v7().to_string(), "world_restarted", reason.message(), @@ -2290,13 +6478,15 @@ fn emit_world_restarted_alert( } fn build_world_restart_required_alert( + orchestration_session_id: Option<&str>, current_world_id: &str, current_world_generation: u64, reason: WorldRestartReason, -) -> AgentEvent { +) -> Option { + let orchestration_session_id = orchestration_session_id?; let mut event = AgentEvent::alert( "shell", - orchestration_session_id(), + orchestration_session_id.to_string(), Uuid::now_v7().to_string(), "world_restart_required", reason.restart_required_message(), @@ -2316,7 +6506,7 @@ fn build_world_restart_required_alert( data.insert("on_drift".to_string(), serde_json::json!("fail_closed")); } - event + Some(event) } fn resolve_world_restart_on_drift( @@ -2334,6 +6524,8 @@ fn resolve_world_restart_on_drift( struct WorldDriftRequest<'a> { requested_cwd: String, + startup_context: Option<&'a RuntimeOrchestrationContext>, + live_runtime_established: bool, policy_snapshot: agent_api_types::PolicySnapshotV3, snapshot_hash: String, workspace_root: Option, @@ -2348,6 +6540,8 @@ async fn handle_detected_world_drift( ) -> Result { let WorldDriftRequest { requested_cwd, + startup_context, + live_runtime_established, policy_snapshot, snapshot_hash, workspace_root, @@ -2360,30 +6554,80 @@ async fn handle_detected_world_drift( crate::execution::config_model::WorldRestartOnDriftMode::AutoRestart => { restart_world_session( old_session, - requested_cwd, - policy_snapshot, - snapshot_hash, - workspace_root, - agent_printer, - reason, + WorldDriftRequest { + requested_cwd, + startup_context, + live_runtime_established, + policy_snapshot, + snapshot_hash, + workspace_root, + agent_printer, + telemetry, + reason, + }, ) .await } crate::execution::config_model::WorldRestartOnDriftMode::FailClosed => { - let alert = build_world_restart_required_alert( - &old_session.world_id, - old_session.world_generation, + let persisted_snapshot = if let Some(startup_context) = startup_context { + let current_binding = PersistedWorldBinding { + world_id: old_session.world_id.clone(), + world_generation: old_session.world_generation, + }; + Some(persist_world_binding_authority( + &startup_context.store, + &startup_context.orchestration_session, + Some(¤t_binding), + )?) + } else { + None + }; + let startup_orchestration_session_id = + startup_context.map(RuntimeOrchestrationContext::orchestration_session_id); + let orchestration_session_id = persisted_snapshot + .as_ref() + .map(|snapshot| snapshot.orchestration_session_id.as_str()) + .or(startup_orchestration_session_id.as_deref()); + let current_world_id = persisted_snapshot + .as_ref() + .and_then(|snapshot| snapshot.world_id.as_deref()) + .unwrap_or(old_session.world_id.as_str()); + let current_world_generation = persisted_snapshot + .as_ref() + .and_then(|snapshot| snapshot.world_generation) + .unwrap_or(old_session.world_generation); + if let Some(alert) = build_world_restart_required_alert( + orchestration_session_id, + current_world_id, + current_world_generation, reason, - ); - telemetry.persist_agent_event(&alert); - telemetry.record_agent_event(); - agent_printer.print(format_event_line(&alert)); - let _ = old_session.client.close().await; + ) { + telemetry.persist_agent_event(&alert); + telemetry.record_agent_event(); + // This path exits immediately after surfacing the alert. Reedline's external + // printer can drop queued lines during that shutdown, so write directly to stdout. + write_best_effort_stdout_line(&format_event_line(&alert)); + } else { + write_best_effort_stdout_line(&format!( + "[shell] {}", + reason.restart_required_message() + )); + } + let close_succeeded = old_session.client.close().await.is_ok(); + if !live_runtime_established && close_succeeded { + if let Some(startup_context) = startup_context { + let _ = persist_world_binding_authority( + &startup_context.store, + &startup_context.orchestration_session, + None, + ); + } + } Err(anyhow!(WorldRestartRequiredError::new(format!( "substrate: error: world restart required before continuing ({}, world_id={}, generation={})", reason.code(), - old_session.world_id, - old_session.world_generation, + current_world_id, + current_world_generation, )))) } } @@ -2393,6 +6637,7 @@ struct OpenWorldSessionRequest<'a> { requested_cwd: String, requested_path: &'a Path, resolved_policy_snapshot: agent_api_types::PolicySnapshotV3, + shared_world_request: Option, snapshot_hash: String, workspace_root: Option, on_stdout: StdoutCallback, @@ -2406,6 +6651,7 @@ async fn open_world_session(request: OpenWorldSessionRequest<'_>) -> Result) -> Result) -> Result, - agent_printer: &ReplPrinter, - reason: WorldRestartReason, + request: WorldDriftRequest<'_>, ) -> Result { + let WorldDriftRequest { + requested_cwd, + startup_context, + live_runtime_established: _live_runtime_established, + policy_snapshot, + snapshot_hash, + workspace_root, + agent_printer, + telemetry: _telemetry, + reason, + } = request; let requested_path = PathBuf::from(&requested_cwd); let on_stdout = old_session.on_stdout.clone(); let previous_world_id = old_session.world_id.clone(); let previous_world_generation = old_session.world_generation; + let orchestration_session_id = + startup_context.map(RuntimeOrchestrationContext::orchestration_session_id); + let shared_world_request = + orchestration_session_id + .clone() + .map(|id| agent_api_types::SharedWorldOwnerSpec { + orchestration_session_id: id, + action: agent_api_types::SharedWorldOwnerAction::ReplaceExpectedGeneration { + expected_generation: previous_world_generation, + reason: reason.message().to_string(), + }, + }); old_session.client.close().await?; @@ -2471,6 +6741,7 @@ async fn restart_world_session( requested_cwd, requested_path: requested_path.as_path(), resolved_policy_snapshot: policy_snapshot, + shared_world_request, snapshot_hash, workspace_root, on_stdout, @@ -2480,11 +6751,42 @@ async fn restart_world_session( }) .await?; + let mut authoritative_world_id = new_session.world_id.clone(); + let mut authoritative_world_generation = new_session.world_generation; + if let Some(startup_context) = startup_context { + let next_binding = PersistedWorldBinding { + world_id: new_session.world_id.clone(), + world_generation: new_session.world_generation, + }; + let persisted_snapshot = persist_world_binding_authority( + &startup_context.store, + &startup_context.orchestration_session, + Some(&next_binding), + )?; + if let Some(world_id) = persisted_snapshot.world_id { + authoritative_world_id = world_id; + } + if let Some(world_generation) = persisted_snapshot.world_generation { + authoritative_world_generation = world_generation; + } + + if let Err(err) = invalidate_stale_world_members_after_binding( + &startup_context.store, + &startup_context.orchestration_session_id(), + authoritative_world_generation, + ) { + let _ = new_session.client.close().await; + return Err(err).context( + "failed to invalidate stale world-scoped participants after replacement binding persistence", + ); + } + } emit_world_restarted_alert( + orchestration_session_id.as_deref(), &previous_world_id, previous_world_generation, - &new_session.world_id, - new_session.world_generation, + &authoritative_world_id, + authoritative_world_generation, reason, ); @@ -2632,6 +6934,7 @@ fn spawn_resize_task(resize_tx: UnboundedSender<(u16, u16)>) { async fn start_world_session( requested_cwd: String, + startup_context: Option<&RuntimeOrchestrationContext>, on_stdout: StdoutCallback, agent_printer: &ReplPrinter, telemetry: &mut ReplSessionTelemetry, @@ -2641,10 +6944,16 @@ async fn start_world_session( .context("policy snapshot (start)")?; let start_hash = resolved_start.snapshot_hash.clone(); let start_workspace_root = find_workspace_root(requested_path); + let shared_world_request = + startup_context.map(|context| agent_api_types::SharedWorldOwnerSpec { + orchestration_session_id: context.orchestration_session_id(), + action: agent_api_types::SharedWorldOwnerAction::AttachOrCreate, + }); let session = open_world_session(OpenWorldSessionRequest { requested_cwd: requested_cwd.clone(), requested_path, resolved_policy_snapshot: resolved_start.snapshot, + shared_world_request, snapshot_hash: start_hash.clone(), workspace_root: start_workspace_root.clone(), on_stdout, @@ -2681,6 +6990,8 @@ async fn start_world_session( session, WorldDriftRequest { requested_cwd: ready_cwd, + startup_context, + live_runtime_established: false, policy_snapshot: resolved_ready.snapshot, snapshot_hash: ready_hash, workspace_root: ready_workspace_root, @@ -2697,6 +7008,7 @@ async fn start_world_session( async fn ensure_no_policy_drift( world_session: &mut Option, + startup_context: Option<&RuntimeOrchestrationContext>, agent_printer: &ReplPrinter, telemetry: &mut ReplSessionTelemetry, ) -> Result<()> { @@ -2739,6 +7051,8 @@ async fn ensure_no_policy_drift( old, WorldDriftRequest { requested_cwd: requested, + startup_context, + live_runtime_established: true, policy_snapshot: resolved.snapshot, snapshot_hash: resolved.snapshot_hash, workspace_root, @@ -2951,703 +7265,1448 @@ async fn exec_world_pty( let _ = session.client.send_resize(cols, rows).await; } } - maybe_sigint = io.sigint_rx.recv() => { - if maybe_sigint.is_some() { - let _ = session.client.send_signal("INT").await; + maybe_sigint = io.sigint_rx.recv() => { + if maybe_sigint.is_some() { + let _ = session.client.send_signal("INT").await; + } + } + } + } + }; + + stdin_done.store(true, Ordering::Relaxed); + let _ = stdin_thread.join(); + + for line in buffered_structured_lines { + io.agent_printer.print(line); + } + if dropped_structured_event_lines > 0 { + io.telemetry.persist_warning_pty_structured_event_drops( + dropped_structured_event_lines, + max_pty_buffered_lines, + Some(cmd_id), + ); + io.agent_printer.print(format!( + "substrate: warning: dropped {dropped_structured_event_lines} structured agent event line(s) during PTY passthrough (cap={max_pty_buffered_lines})" + )); + } + + let mut next_cwd = cwd; + if exit == 0 { + if let Some(predicted) = predict_cd_next_cwd(&prev_cwd, program) { + if next_cwd == prev_cwd { + let (predicted, warning) = + apply_caged_predicted_cwd_from_config(&prev_cwd, predicted); + let warning = + suppress_redundant_caged_prediction_warning(&prev_cwd, &predicted, warning); + if let Some(message) = warning { + io.agent_printer.print(message); + } + next_cwd = predicted; + } + } + } + session.world_cwd = next_cwd; + Ok(exit) +} + +#[cfg(test)] +mod caged_prediction_tests { + use super::*; + use substrate_common::WorldRootMode; + use tempfile::tempdir; + + #[test] + fn apply_caged_predicted_cwd_bounces_outside_anchor() { + let temp = tempdir().unwrap(); + let root = temp.path().join("root"); + let outside = temp.path().join("outside"); + std::fs::create_dir_all(&root).unwrap(); + std::fs::create_dir_all(&outside).unwrap(); + + let settings = WorldRootSettings { + mode: WorldRootMode::Project, + path: std::fs::canonicalize(&root).unwrap(), + caged: true, + }; + + let current = std::fs::canonicalize(&root).unwrap(); + let predicted = std::fs::canonicalize(&outside).unwrap(); + let current_s = current.to_string_lossy().to_string(); + let predicted_s = predicted.to_string_lossy().to_string(); + let (out, warning) = apply_caged_predicted_cwd(&settings, ¤t_s, predicted_s); + assert_eq!(out, settings.path.to_string_lossy().to_string()); + assert!(warning.is_some()); + } + + #[test] + fn apply_caged_predicted_cwd_allows_inside_anchor() { + let temp = tempdir().unwrap(); + let root = temp.path().join("root"); + let inside = root.join("inside"); + std::fs::create_dir_all(&inside).unwrap(); + + let settings = WorldRootSettings { + mode: WorldRootMode::Project, + path: std::fs::canonicalize(&root).unwrap(), + caged: true, + }; + + let current = std::fs::canonicalize(&root).unwrap(); + let predicted = std::fs::canonicalize(&inside).unwrap(); + let current_s = current.to_string_lossy().to_string(); + let predicted_s = predicted.to_string_lossy().to_string(); + let (out, warning) = apply_caged_predicted_cwd(&settings, ¤t_s, predicted_s.clone()); + assert_eq!(out, predicted_s); + assert!(warning.is_none()); + } + + #[test] + fn suppress_redundant_caged_prediction_warning_drops_when_cwd_unchanged() { + let temp = tempdir().unwrap(); + let root = temp.path().join("root"); + std::fs::create_dir_all(&root).unwrap(); + let prev = format!("{}/.", root.display()); + let predicted = root.to_string_lossy().to_string(); + let warning = Some("substrate: info: caged root guard: blocked cd to /tmp/outside (outside /tmp/root); returning to /tmp/root".to_string()); + + let out = suppress_redundant_caged_prediction_warning(&prev, &predicted, warning); + assert!(out.is_none()); + } +} + +fn spawn_passthrough_stdin_thread( + stdin_tx: UnboundedSender>, + done: Arc, + cmd_id: &str, +) -> thread::JoinHandle<()> { + #[cfg(unix)] + { + use nix::sys::select::{select, FdSet}; + use nix::sys::time::TimeVal; + use std::io::Read; + use std::os::unix::io::{AsFd, AsRawFd}; + + let cmd_id = cmd_id.to_string(); + thread::spawn(move || { + let mut stdin = io::stdin(); + let mut buffer = vec![0u8; 4096]; + while !done.load(Ordering::Relaxed) { + let stdin_fd = stdin.as_raw_fd(); + let stdin_borrowed = stdin.as_fd(); + let mut read_fds = FdSet::new(); + read_fds.insert(stdin_borrowed); + let mut timeout = TimeVal::new(0, 100_000); + let result = select( + stdin_fd + 1, + Some(&mut read_fds), + None, + None, + Some(&mut timeout), + ); + match result { + Ok(0) => continue, + Ok(_) if read_fds.contains(stdin_borrowed) => match stdin.read(&mut buffer) { + Ok(0) => break, + Ok(n) => { + let _ = stdin_tx.send(buffer[..n].to_vec()); + } + Err(e) => { + log::warn!("[{cmd_id}] passthrough stdin read failed: {e}"); + break; + } + }, + Ok(_) => continue, + Err(e) => { + if e != nix::errno::Errno::EINTR { + log::warn!("[{cmd_id}] passthrough select() failed: {e}"); + break; + } + } + } + } + }) + } + + #[cfg(not(unix))] + { + let _ = (stdin_tx, done, cmd_id); + thread::spawn(|| {}) + } +} + +async fn exec_host_line( + config: &ShellConfig, + host_state: &mut HostState, + line: &str, + cmd_id: &str, + running_child_pid: Arc, + world_client: Option<&ReplPersistentSessionClient>, + io: &mut ReplIo<'_>, +) -> Result { + if let Some(code) = try_run_host_builtin(config, host_state, line, io.agent_printer)? { + return Ok(code); + } + + let shell_path = config.shell_path.clone(); + let cwd = host_state.cwd.clone(); + let env = host_state.env.clone(); + let line = line.to_string(); + let cmd_id = cmd_id.to_string(); + let command_fut = task::spawn_blocking(move || { + spawn_host_shell_command(&shell_path, &cwd, &env, &line, &cmd_id, running_child_pid) + }) + .map( + |res: Result, tokio::task::JoinError>| match res { + Ok(inner) => inner, + Err(err) => Err(anyhow!(err)), + }, + ); + pin_mut!(command_fut); + + let status = loop { + tokio::select! { + res = &mut command_fut => break res?, + maybe_event = io.agent_rx.recv() => { + if let Some(event) = maybe_event { + handle_agent_event(event, io.telemetry, io.agent_printer); + } + } + maybe_resize = io.resize_rx.recv() => { + if let Some((cols, rows)) = maybe_resize { + if let Some(client) = world_client { + let _ = client.send_resize(cols, rows).await; + } + } + } + _maybe_sigint = io.sigint_rx.recv() => { + // Host signals are forwarded by global handlers; drain to avoid leaking to world. + } + } + }; + + Ok(exit_code_from_status(status)) +} + +fn try_run_host_builtin( + config: &ShellConfig, + host_state: &mut HostState, + line: &str, + agent_printer: &ReplPrinter, +) -> Result> { + let tokens = shell_words::split(line) + .unwrap_or_else(|_| line.split_whitespace().map(|s| s.to_string()).collect()); + if tokens.is_empty() { + return Ok(Some(0)); + } + + match tokens[0].as_str() { + "pwd" => { + agent_printer.print(format!("{}", host_state.cwd.display())); + Ok(Some(0)) + } + "cd" => { + let target = tokens.get(1).map(String::as_str).unwrap_or("~"); + let expanded = shellexpand::tilde(target).to_string(); + let prev = host_state.cwd.clone(); + let candidate = PathBuf::from(&expanded); + let absolute = if candidate.is_absolute() { + candidate + } else { + prev.join(candidate) + }; + let requested = match std::fs::canonicalize(&absolute) { + Ok(path) => path, + Err(_) => { + agent_printer.print(format!( + "substrate: error: :host cd: not a directory: {}", + absolute.display() + )); + return Ok(Some(1)); + } + }; + + let world_root = resolve_world_root( + config.cli_anchor_mode, + config.cli_anchor_path.clone(), + config.cli_caged, + &prev, + )?; + let (destination, warning) = enforce_caged_destination(&world_root, &prev, requested); + if let Some(message) = warning { + agent_printer.print(message); + } + + host_state.cwd = destination; + host_state + .env + .insert("PWD".to_string(), host_state.cwd.display().to_string()); + Ok(Some(0)) + } + "export" => { + for arg in tokens.iter().skip(1) { + if let Some((k, v)) = arg.split_once('=') { + host_state.env.insert(k.to_string(), v.to_string()); + } else { + let val = host_state + .env + .get(arg) + .cloned() + .or_else(|| std::env::var(arg).ok()) + .unwrap_or_default(); + host_state.env.insert(arg.to_string(), val); } } + Ok(Some(0)) + } + "unset" => { + for arg in tokens.iter().skip(1) { + host_state.env.remove(arg); } + Ok(Some(0)) } - }; + _ => Ok(None), + } +} - stdin_done.store(true, Ordering::Relaxed); - let _ = stdin_thread.join(); +fn spawn_host_shell_command( + shell_path: &str, + cwd: &Path, + env: &HashMap, + line: &str, + cmd_id: &str, + running_child_pid: Arc, +) -> Result { + let mut cmd = Command::new(shell_path); + cmd.arg("-c") + .arg(line) + .current_dir(cwd) + .env_clear() + .envs(env) + .env("SHIM_PARENT_CMD_ID", cmd_id) + .stdin(Stdio::inherit()) + .stdout(Stdio::inherit()) + .stderr(Stdio::inherit()); - for line in buffered_structured_lines { - io.agent_printer.print(line); + #[cfg(unix)] + { + use std::os::unix::process::CommandExt; + unsafe { + cmd.pre_exec(|| { + libc::setpgid(0, 0); + Ok(()) + }); + } } - if dropped_structured_event_lines > 0 { - io.telemetry.persist_warning_pty_structured_event_drops( - dropped_structured_event_lines, - max_pty_buffered_lines, - Some(cmd_id), - ); - io.agent_printer.print(format!( - "substrate: warning: dropped {dropped_structured_event_lines} structured agent event line(s) during PTY passthrough (cap={max_pty_buffered_lines})" - )); + + let mut child = cmd.spawn().context("spawn :host command")?; + running_child_pid.store(child.id() as i32, Ordering::SeqCst); + let status = child.wait().context("wait :host command")?; + running_child_pid.store(0, Ordering::SeqCst); + Ok(status) +} + +fn exit_code_from_status(status: ExitStatus) -> i32 { + #[cfg(unix)] + { + use std::os::unix::process::ExitStatusExt; + status + .code() + .unwrap_or_else(|| 128 + status.signal().unwrap_or(1)) } + #[cfg(not(unix))] + status.code().unwrap_or(1) +} - let mut next_cwd = cwd; - if exit == 0 { - if let Some(predicted) = predict_cd_next_cwd(&prev_cwd, program) { - if next_cwd == prev_cwd { - let (predicted, warning) = - apply_caged_predicted_cwd_from_config(&prev_cwd, predicted); - let warning = - suppress_redundant_caged_prediction_warning(&prev_cwd, &predicted, warning); - if let Some(message) = warning { - io.agent_printer.print(message); - } - next_cwd = predicted; - } - } +struct PtyActiveResetGuard; + +impl PtyActiveResetGuard { + fn new() -> Self { + PTY_ACTIVE.store(true, Ordering::SeqCst); + Self + } +} + +impl Drop for PtyActiveResetGuard { + fn drop(&mut self) { + PTY_ACTIVE.store(false, Ordering::SeqCst); + } +} + +fn exit_status_from_code(code: i32) -> ExitStatus { + #[cfg(unix)] + { + use std::os::unix::process::ExitStatusExt; + let clamped = code.clamp(0, 255); + ExitStatus::from_raw(clamped << 8) + } + #[cfg(windows)] + { + use std::os::windows::process::ExitStatusExt; + ExitStatus::from_raw(code as u32) + } + #[cfg(not(any(unix, windows)))] + { + panic!("exit_status_from_code is unsupported on this platform (code={code})"); } - session.world_cwd = next_cwd; - Ok(exit) } #[cfg(test)] -mod caged_prediction_tests { +mod tests { use super::*; - use substrate_common::WorldRootMode; - use tempfile::tempdir; + use crate::execution::agent_events::{ + acquire_event_test_guard, clear_agent_event_sender, init_event_channel, + }; + #[cfg(unix)] + use crate::execution::agent_runtime::orchestration_session::OrchestrationSessionPosture; + #[cfg(unix)] + use crate::execution::config_model::AgentExecutionScope; + #[cfg(unix)] + use crate::execution::ShellMode; + #[cfg(unix)] + use crate::execution::WorldRootSettings; + use std::cell::Cell; + #[cfg(unix)] + use std::cell::RefCell; + #[cfg(unix)] + use std::collections::HashMap; + #[cfg(unix)] + use std::fs; + use substrate_common::agent_events::AgentEventKind; + #[cfg(unix)] + use tempfile::TempDir; - #[test] - fn apply_caged_predicted_cwd_bounces_outside_anchor() { - let temp = tempdir().unwrap(); - let root = temp.path().join("root"); - let outside = temp.path().join("outside"); - std::fs::create_dir_all(&root).unwrap(); - std::fs::create_dir_all(&outside).unwrap(); + #[cfg(unix)] + fn reedline_terminal_loss_error() -> anyhow::Error { + anyhow!(io::Error::from_raw_os_error(libc::ENOTTY)) + } - let settings = WorldRootSettings { - mode: WorldRootMode::Project, - path: std::fs::canonicalize(&root).unwrap(), - caged: true, - }; + #[cfg(not(unix))] + fn reedline_terminal_loss_error() -> anyhow::Error { + anyhow!("terminal invalid") + } - let current = std::fs::canonicalize(&root).unwrap(); - let predicted = std::fs::canonicalize(&outside).unwrap(); - let current_s = current.to_string_lossy().to_string(); - let predicted_s = predicted.to_string_lossy().to_string(); - let (out, warning) = apply_caged_predicted_cwd(&settings, ¤t_s, predicted_s); - assert_eq!(out, settings.path.to_string_lossy().to_string()); - assert!(warning.is_some()); + #[cfg(unix)] + fn common_terminal_loss_errors() -> Vec { + vec![ + anyhow!(io::Error::from_raw_os_error(libc::ENOTTY)), + anyhow!(io::Error::from_raw_os_error(libc::EIO)), + anyhow!(io::Error::from_raw_os_error(libc::EBADF)), + anyhow!(io::Error::new(io::ErrorKind::BrokenPipe, "broken pipe")), + anyhow!(io::Error::new(io::ErrorKind::NotConnected, "not connected")), + anyhow!(io::Error::new( + io::ErrorKind::UnexpectedEof, + "unexpected eof" + )), + ] + } + + #[cfg(not(unix))] + fn common_terminal_loss_errors() -> Vec { + vec![ + anyhow!("terminal invalid"), + anyhow!("input/output error"), + anyhow!("bad file descriptor"), + anyhow!(io::Error::new(io::ErrorKind::BrokenPipe, "broken pipe")), + anyhow!(io::Error::new(io::ErrorKind::NotConnected, "not connected")), + anyhow!(io::Error::new( + io::ErrorKind::UnexpectedEof, + "unexpected eof" + )), + ] } #[test] - fn apply_caged_predicted_cwd_allows_inside_anchor() { - let temp = tempdir().unwrap(); - let root = temp.path().join("root"); - let inside = root.join("inside"); - std::fs::create_dir_all(&inside).unwrap(); + fn classify_prompt_worker_error_falls_back_on_cursor_timeout() { + let err = anyhow!("cursor position could not be read within a normal duration"); - let settings = WorldRootSettings { - mode: WorldRootMode::Project, - path: std::fs::canonicalize(&root).unwrap(), - caged: true, - }; + assert_eq!( + classify_prompt_worker_error(true, &err), + PromptWorkerErrorDisposition::FallbackToStdio + ); + } - let current = std::fs::canonicalize(&root).unwrap(); - let predicted = std::fs::canonicalize(&inside).unwrap(); - let current_s = current.to_string_lossy().to_string(); - let predicted_s = predicted.to_string_lossy().to_string(); - let (out, warning) = apply_caged_predicted_cwd(&settings, ¤t_s, predicted_s.clone()); - assert_eq!(out, predicted_s); - assert!(warning.is_none()); + #[test] + fn classify_prompt_worker_error_treats_reedline_failures_as_abnormal_terminal_loss() { + let err = reedline_terminal_loss_error(); + + assert_eq!( + classify_prompt_worker_error(true, &err), + PromptWorkerErrorDisposition::AbnormalTerminalLoss + ); } #[test] - fn suppress_redundant_caged_prediction_warning_drops_when_cwd_unchanged() { - let temp = tempdir().unwrap(); - let root = temp.path().join("root"); - std::fs::create_dir_all(&root).unwrap(); - let prev = format!("{}/.", root.display()); - let predicted = root.to_string_lossy().to_string(); - let warning = Some("substrate: info: caged root guard: blocked cd to /tmp/outside (outside /tmp/root); returning to /tmp/root".to_string()); + fn classify_prompt_worker_error_treats_common_terminal_loss_errors_as_abnormal() { + let cases = common_terminal_loss_errors(); - let out = suppress_redundant_caged_prediction_warning(&prev, &predicted, warning); - assert!(out.is_none()); + for err in &cases { + assert_eq!( + classify_prompt_worker_error(true, err), + PromptWorkerErrorDisposition::AbnormalTerminalLoss + ); + } } -} -fn spawn_passthrough_stdin_thread( - stdin_tx: UnboundedSender>, - done: Arc, - cmd_id: &str, -) -> thread::JoinHandle<()> { - #[cfg(unix)] - { - use nix::sys::select::{select, FdSet}; - use nix::sys::time::TimeVal; - use std::io::Read; - use std::os::unix::io::{AsFd, AsRawFd}; + #[test] + fn classify_prompt_worker_error_keeps_unrelated_reedline_errors_generic() { + let err = anyhow!("completion menu rendering failed"); - let cmd_id = cmd_id.to_string(); - thread::spawn(move || { - let mut stdin = io::stdin(); - let mut buffer = vec![0u8; 4096]; - while !done.load(Ordering::Relaxed) { - let stdin_fd = stdin.as_raw_fd(); - let stdin_borrowed = stdin.as_fd(); - let mut read_fds = FdSet::new(); - read_fds.insert(stdin_borrowed); - let mut timeout = TimeVal::new(0, 100_000); - let result = select( - stdin_fd + 1, - Some(&mut read_fds), - None, - None, - Some(&mut timeout), - ); - match result { - Ok(0) => continue, - Ok(_) if read_fds.contains(stdin_borrowed) => match stdin.read(&mut buffer) { - Ok(0) => break, - Ok(n) => { - let _ = stdin_tx.send(buffer[..n].to_vec()); - } - Err(e) => { - log::warn!("[{cmd_id}] passthrough stdin read failed: {e}"); - break; - } - }, - Ok(_) => continue, - Err(e) => { - if e != nix::errno::Errno::EINTR { - log::warn!("[{cmd_id}] passthrough select() failed: {e}"); - break; - } - } - } - } - }) + assert_eq!( + classify_prompt_worker_error(true, &err), + PromptWorkerErrorDisposition::GenericError + ); } - #[cfg(not(unix))] - { - let _ = (stdin_tx, done, cmd_id); - thread::spawn(|| {}) + #[test] + fn shutdown_disposition_tracks_termination_cause() { + assert_eq!( + shutdown_disposition_for_termination_cause(ReplTerminationCause::NormalExit), + PromptWorkerShutdownDisposition::Graceful + ); + assert_eq!( + shutdown_disposition_for_termination_cause(ReplTerminationCause::AbnormalTerminalLoss), + PromptWorkerShutdownDisposition::Abandon + ); } -} -async fn exec_host_line( - config: &ShellConfig, - host_state: &mut HostState, - line: &str, - cmd_id: &str, - running_child_pid: Arc, - world_client: Option<&ReplPersistentSessionClient>, - io: &mut ReplIo<'_>, -) -> Result { - if let Some(code) = try_run_host_builtin(config, host_state, line, io.agent_printer)? { - return Ok(code); + #[test] + fn build_world_restart_required_alert_only_builds_with_orchestration_context() { + let event = build_world_restart_required_alert( + Some("orch-live"), + "world-1", + 4, + WorldRestartReason::PolicySnapshotChanged, + ) + .expect("alert should be built"); + assert_eq!(event.kind, AgentEventKind::Alert); + assert_eq!(event.orchestration_session_id, "orch-live"); + assert_eq!(event.world_id.as_deref(), Some("world-1")); + assert_eq!(event.world_generation, Some(4)); + assert!( + build_world_restart_required_alert( + None, + "world-1", + 4, + WorldRestartReason::PolicySnapshotChanged, + ) + .is_none(), + "missing orchestration context must suppress the alert event" + ); } - let shell_path = config.shell_path.clone(); - let cwd = host_state.cwd.clone(); - let env = host_state.env.clone(); - let line = line.to_string(); - let cmd_id = cmd_id.to_string(); - let command_fut = task::spawn_blocking(move || { - spawn_host_shell_command(&shell_path, &cwd, &env, &line, &cmd_id, running_child_pid) - }) - .map( - |res: Result, tokio::task::JoinError>| match res { - Ok(inner) => inner, - Err(err) => Err(anyhow!(err)), - }, - ); - pin_mut!(command_fut); + #[test] + #[serial_test::serial] + fn emit_world_restarted_alert_only_emits_with_orchestration_context() { + let _guard = acquire_event_test_guard(); + let mut rx = init_event_channel(); + + emit_world_restarted_alert( + Some("orch-live"), + "world-old", + 1, + "world-new", + 2, + WorldRestartReason::WorkspaceRootChanged, + ); - let status = loop { - tokio::select! { - res = &mut command_fut => break res?, - maybe_event = io.agent_rx.recv() => { - if let Some(event) = maybe_event { - handle_agent_event(event, io.telemetry, io.agent_printer); - } - } - maybe_resize = io.resize_rx.recv() => { - if let Some((cols, rows)) = maybe_resize { - if let Some(client) = world_client { - let _ = client.send_resize(cols, rows).await; - } - } - } - _maybe_sigint = io.sigint_rx.recv() => { - // Host signals are forwarded by global handlers; drain to avoid leaking to world. - } - } - }; + let event = rx.try_recv().expect("world restarted event"); + assert_eq!(event.kind, AgentEventKind::Alert); + assert_eq!(event.orchestration_session_id, "orch-live"); + assert_eq!(event.world_id.as_deref(), Some("world-new")); + assert_eq!(event.world_generation, Some(2)); + + emit_world_restarted_alert( + None, + "world-old", + 1, + "world-new", + 2, + WorldRestartReason::WorkspaceRootChanged, + ); + assert!( + rx.try_recv().is_err(), + "missing orchestration context must suppress the restarted alert event" + ); + clear_agent_event_sender(); + } - Ok(exit_code_from_status(status)) -} + #[test] + fn resolve_reedline_ctrl_d_terminal_loss_returns_pre_recorded_message() { + let detected = AtomicBool::new(true); + let message = Mutex::new(Some("controlling terminal became invalid".to_string())); + let detect_calls = Cell::new(0usize); -fn try_run_host_builtin( - config: &ShellConfig, - host_state: &mut HostState, - line: &str, - agent_printer: &ReplPrinter, -) -> Result> { - let tokens = shell_words::split(line) - .unwrap_or_else(|_| line.split_whitespace().map(|s| s.to_string()).collect()); - if tokens.is_empty() { - return Ok(Some(0)); - } + let err = resolve_reedline_ctrl_d_terminal_loss_with( + &detected, + &message, + || { + detect_calls.set(detect_calls.get() + 1); + None + }, + || panic!("should not sleep when terminal loss is already recorded"), + ) + .expect("expected recorded terminal loss"); - match tokens[0].as_str() { - "pwd" => { - agent_printer.print(format!("{}", host_state.cwd.display())); - Ok(Some(0)) - } - "cd" => { - let target = tokens.get(1).map(String::as_str).unwrap_or("~"); - let expanded = shellexpand::tilde(target).to_string(); - let prev = host_state.cwd.clone(); - let candidate = PathBuf::from(&expanded); - let absolute = if candidate.is_absolute() { - candidate - } else { - prev.join(candidate) - }; - let requested = match std::fs::canonicalize(&absolute) { - Ok(path) => path, - Err(_) => { - agent_printer.print(format!( - "substrate: error: :host cd: not a directory: {}", - absolute.display() - )); - return Ok(Some(1)); - } - }; + assert_eq!(detect_calls.get(), 0); + assert!(err + .to_string() + .contains("controlling terminal became invalid")); + } - let world_root = resolve_world_root( - config.cli_anchor_mode, - config.cli_anchor_path.clone(), - config.cli_caged, - &prev, - )?; - let (destination, warning) = enforce_caged_destination(&world_root, &prev, requested); - if let Some(message) = warning { - agent_printer.print(message); - } + #[test] + fn resolve_reedline_ctrl_d_terminal_loss_retries_detector_within_window() { + let detected = AtomicBool::new(false); + let message = Mutex::new(None::); + let detect_calls = Cell::new(0usize); + let sleep_calls = Cell::new(0usize); - host_state.cwd = destination; - host_state - .env - .insert("PWD".to_string(), host_state.cwd.display().to_string()); - Ok(Some(0)) - } - "export" => { - for arg in tokens.iter().skip(1) { - if let Some((k, v)) = arg.split_once('=') { - host_state.env.insert(k.to_string(), v.to_string()); + let err = resolve_reedline_ctrl_d_terminal_loss_with( + &detected, + &message, + || { + let next = detect_calls.get() + 1; + detect_calls.set(next); + if next == 3 { + Some(anyhow!("delayed terminal loss")) } else { - let val = host_state - .env - .get(arg) - .cloned() - .or_else(|| std::env::var(arg).ok()) - .unwrap_or_default(); - host_state.env.insert(arg.to_string(), val); + None + } + }, + || sleep_calls.set(sleep_calls.get() + 1), + ) + .expect("expected delayed terminal loss"); + + assert_eq!(detect_calls.get(), 3); + assert_eq!(sleep_calls.get(), 2); + assert!(err.to_string().contains("delayed terminal loss")); + } + + #[test] + fn resolve_reedline_ctrl_d_terminal_loss_observes_monitor_update_on_retry() { + let detected = AtomicBool::new(false); + let message = Mutex::new(None::); + let sleep_calls = Cell::new(0usize); + + let err = resolve_reedline_ctrl_d_terminal_loss_with( + &detected, + &message, + || None, + || { + let next = sleep_calls.get() + 1; + sleep_calls.set(next); + if next == 1 { + detected.store(true, Ordering::SeqCst); + *message.lock().expect("message mutex poisoned") = + Some("monitor reported terminal loss".to_string()); } - } - Ok(Some(0)) - } - "unset" => { - for arg in tokens.iter().skip(1) { - host_state.env.remove(arg); - } - Ok(Some(0)) - } - _ => Ok(None), + }, + ) + .expect("expected terminal loss from monitor update"); + + assert_eq!(sleep_calls.get(), 1); + assert!(err.to_string().contains("monitor reported terminal loss")); } -} -fn spawn_host_shell_command( - shell_path: &str, - cwd: &Path, - env: &HashMap, - line: &str, - cmd_id: &str, - running_child_pid: Arc, -) -> Result { - let mut cmd = Command::new(shell_path); - cmd.arg("-c") - .arg(line) - .current_dir(cwd) - .env_clear() - .envs(env) - .env("SHIM_PARENT_CMD_ID", cmd_id) - .stdin(Stdio::inherit()) - .stdout(Stdio::inherit()) - .stderr(Stdio::inherit()); + #[test] + fn resolve_reedline_ctrl_d_terminal_loss_returns_none_without_signal() { + let detected = AtomicBool::new(false); + let message = Mutex::new(None::); + let detect_calls = Cell::new(0usize); + let sleep_calls = Cell::new(0usize); + + let err = resolve_reedline_ctrl_d_terminal_loss_with( + &detected, + &message, + || { + detect_calls.set(detect_calls.get() + 1); + None + }, + || sleep_calls.set(sleep_calls.get() + 1), + ); + + assert!(err.is_none()); + assert_eq!(detect_calls.get(), REEDLINE_CTRLD_TERMINAL_LOSS_RECHECKS); + assert_eq!(sleep_calls.get(), REEDLINE_CTRLD_TERMINAL_LOSS_RECHECKS - 1); + } #[cfg(unix)] - { - use std::os::unix::process::CommandExt; - unsafe { - cmd.pre_exec(|| { - libc::setpgid(0, 0); - Ok(()) - }); + fn test_runtime_selection_descriptor() -> RuntimeSelectionDescriptor { + RuntimeSelectionDescriptor { + agent_id: "codex".to_string(), + backend_id: "cli:codex".to_string(), + backend_kind: AgentRuntimeBackendKind::Codex, + protocol: PURE_AGENT_PROTOCOL.to_string(), + execution_scope: AgentExecutionScope::Host, + binary_path: PathBuf::from("/bin/sh"), } } - let mut child = cmd.spawn().context("spawn :host command")?; - running_child_pid.store(child.id() as i32, Ordering::SeqCst); - let status = child.wait().context("wait :host command")?; - running_child_pid.store(0, Ordering::SeqCst); - Ok(status) -} - -fn exit_code_from_status(status: ExitStatus) -> i32 { #[cfg(unix)] - { - use std::os::unix::process::ExitStatusExt; - status - .code() - .unwrap_or_else(|| 128 + status.signal().unwrap_or(1)) - } - #[cfg(not(unix))] - status.code().unwrap_or(1) -} + #[test] + fn terminalize_startup_prompt_failure_marks_accepted_prompt_failed() { + let descriptor = test_runtime_selection_descriptor(); + let manifest = AgentRuntimeSessionManifest::new( + &descriptor, + "orch-test".to_string(), + "ash_test".to_string(), + "lease-test".to_string(), + ); + let mut session = OrchestrationSessionRecord::new( + "orch-test".to_string(), + "shell-test".to_string(), + "/tmp".to_string(), + &manifest, + ); + session.initialize_startup_prompt(manifest.handle.participant_id.clone()); + session.mark_startup_prompt_accepted(manifest.handle.participant_id.as_str()); -struct PtyActiveResetGuard; + let message = terminalize_startup_prompt_failure( + &mut session, + &manifest, + "startup prompt stream ended before terminal completion", + ) + .expect("accepted startup prompt should be terminalized"); -impl PtyActiveResetGuard { - fn new() -> Self { - PTY_ACTIVE.store(true, Ordering::SeqCst); - Self + assert_eq!( + message, + "startup prompt stream ended before terminal completion" + ); + let record = session + .startup_prompt + .as_ref() + .expect("startup prompt should remain present"); + assert_eq!(record.state, StartupPromptStreamState::Failed); + assert!(record.accepted_at.is_some()); + assert!(record.terminal_at.is_some()); + assert_eq!( + record.error_message.as_deref(), + Some("startup prompt stream ended before terminal completion") + ); } -} -impl Drop for PtyActiveResetGuard { - fn drop(&mut self) { - PTY_ACTIVE.store(false, Ordering::SeqCst); + #[cfg(unix)] + #[test] + fn terminalize_startup_prompt_failure_leaves_completed_prompt_unchanged() { + let descriptor = test_runtime_selection_descriptor(); + let manifest = AgentRuntimeSessionManifest::new( + &descriptor, + "orch-test".to_string(), + "ash_test".to_string(), + "lease-test".to_string(), + ); + let mut session = OrchestrationSessionRecord::new( + "orch-test".to_string(), + "shell-test".to_string(), + "/tmp".to_string(), + &manifest, + ); + session.initialize_startup_prompt(manifest.handle.participant_id.clone()); + session.mark_startup_prompt_completed(manifest.handle.participant_id.as_str(), "success"); + + let result = terminalize_startup_prompt_failure( + &mut session, + &manifest, + "startup prompt completion did not reach a terminal turn outcome", + ); + + assert!(result.is_none()); + let record = session + .startup_prompt + .as_ref() + .expect("startup prompt should remain present"); + assert_eq!(record.state, StartupPromptStreamState::Completed); + assert_eq!(record.turn_outcome.as_deref(), Some("success")); + assert!(record.error_message.is_none()); } -} -fn exit_status_from_code(code: i32) -> ExitStatus { #[cfg(unix)] - { - use std::os::unix::process::ExitStatusExt; - let clamped = code.clamp(0, 255); - ExitStatus::from_raw(clamped << 8) - } - #[cfg(windows)] - { - use std::os::windows::process::ExitStatusExt; - ExitStatus::from_raw(code as u32) - } - #[cfg(not(any(unix, windows)))] - { - panic!("exit_status_from_code is unsupported on this platform (code={code})"); + struct CurrentDirGuard { + original: PathBuf, } -} -#[cfg(test)] -mod tests { - use super::*; - use crate::execution::ShellMode; - use crate::execution::WorldRootSettings; - use std::cell::Cell; #[cfg(unix)] - use std::cell::RefCell; - use std::collections::HashMap; - use std::fs; - use tempfile::TempDir; + impl CurrentDirGuard { + fn change_to(path: &Path) -> Self { + let original = std::env::current_dir().expect("current dir should resolve"); + std::env::set_current_dir(path).expect("set current dir"); + Self { original } + } + } #[cfg(unix)] - fn reedline_terminal_loss_error() -> anyhow::Error { - anyhow!(io::Error::from_raw_os_error(libc::ENOTTY)) + impl Drop for CurrentDirGuard { + fn drop(&mut self) { + let _ = std::env::set_current_dir(&self.original); + } } - #[cfg(not(unix))] - fn reedline_terminal_loss_error() -> anyhow::Error { - anyhow!("terminal invalid") + #[cfg(unix)] + fn write_fake_codex_script(temp: &TempDir, keep_alive: bool) -> PathBuf { + let path = temp.path().join("fake-codex.sh"); + let body = if keep_alive { + "#!/bin/sh\ntrap 'exit 0' INT TERM\nprintf '{\"type\":\"thread.started\",\"thread_id\":\"thread-test\"}\\r\\n'\nprintf '{\"type\":\"turn.started\",\"thread_id\":\"thread-test\",\"turn_id\":\"turn-1\"}\\r\\n'\nwhile :; do sleep 1; done\n" + } else { + "#!/bin/sh\nprintf '{\"type\":\"thread.started\",\"thread_id\":\"thread-test\"}\\r\\n'\nprintf '{\"type\":\"turn.started\",\"thread_id\":\"thread-test\",\"turn_id\":\"turn-1\"}\\r\\n'\n" + }; + fs::write(&path, body).expect("write fake codex script"); + let mut perms = fs::metadata(&path) + .expect("fake codex metadata") + .permissions(); + use std::os::unix::fs::PermissionsExt; + perms.set_mode(0o755); + fs::set_permissions(&path, perms).expect("set fake codex permissions"); + path } #[cfg(unix)] - fn common_terminal_loss_errors() -> Vec { - vec![ - anyhow!(io::Error::from_raw_os_error(libc::ENOTTY)), - anyhow!(io::Error::from_raw_os_error(libc::EIO)), - anyhow!(io::Error::from_raw_os_error(libc::EBADF)), - anyhow!(io::Error::new(io::ErrorKind::BrokenPipe, "broken pipe")), - anyhow!(io::Error::new(io::ErrorKind::NotConnected, "not connected")), - anyhow!(io::Error::new( - io::ErrorKind::UnexpectedEof, - "unexpected eof" - )), - ] + fn write_fake_codex_script_without_session_handle(temp: &TempDir) -> PathBuf { + let path = temp.path().join("fake-codex-no-session.sh"); + let body = "#!/bin/sh\nprintf 'bootstrap-without-session-handle\\n'\n"; + fs::write(&path, body).expect("write fake codex script without session handle"); + let mut perms = fs::metadata(&path) + .expect("fake codex metadata") + .permissions(); + use std::os::unix::fs::PermissionsExt; + perms.set_mode(0o755); + fs::set_permissions(&path, perms).expect("set fake codex permissions"); + path } - #[cfg(not(unix))] - fn common_terminal_loss_errors() -> Vec { - vec![ - anyhow!("terminal invalid"), - anyhow!("input/output error"), - anyhow!("bad file descriptor"), - anyhow!(io::Error::new(io::ErrorKind::BrokenPipe, "broken pipe")), - anyhow!(io::Error::new(io::ErrorKind::NotConnected, "not connected")), - anyhow!(io::Error::new( - io::ErrorKind::UnexpectedEof, - "unexpected eof" - )), - ] + #[cfg(unix)] + fn write_fake_codex_script_with_shutdown_delay(temp: &TempDir, delay_seconds: u64) -> PathBuf { + let path = temp.path().join("fake-codex-shutdown-delay.sh"); + let body = format!( + "#!/bin/sh\ntrap 'sleep {}; exit 0' INT TERM\nprintf '{{\"type\":\"thread.started\",\"thread_id\":\"thread-test\"}}\\r\\n'\nprintf '{{\"type\":\"turn.started\",\"thread_id\":\"thread-test\",\"turn_id\":\"turn-1\"}}\\r\\n'\nwhile :; do sleep 1; done\n", + delay_seconds + ); + fs::write(&path, body).expect("write fake codex shutdown delay script"); + let mut perms = fs::metadata(&path) + .expect("fake codex metadata") + .permissions(); + use std::os::unix::fs::PermissionsExt; + perms.set_mode(0o755); + fs::set_permissions(&path, perms).expect("set fake codex permissions"); + path } - #[test] - fn classify_prompt_worker_error_falls_back_on_cursor_timeout() { - let err = anyhow!("cursor position could not be read within a normal duration"); + #[cfg(unix)] + fn write_fake_codex_script_with_pid_file(temp: &TempDir, pid_file: &Path) -> PathBuf { + let path = temp.path().join("fake-codex-with-pid.sh"); + let body = format!( + "#!/bin/sh\ntrap 'exit 0' INT TERM\nprintf '%s\\n' $$ > {}\nprintf '{{\"type\":\"thread.started\",\"thread_id\":\"thread-test\"}}\\r\\n'\nprintf '{{\"type\":\"turn.started\",\"thread_id\":\"thread-test\",\"turn_id\":\"turn-1\"}}\\r\\n'\nwhile :; do sleep 1; done\n", + pid_file.display() + ); + fs::write(&path, body).expect("write fake codex pid script"); + let mut perms = fs::metadata(&path) + .expect("fake codex metadata") + .permissions(); + use std::os::unix::fs::PermissionsExt; + perms.set_mode(0o755); + fs::set_permissions(&path, perms).expect("set fake codex permissions"); + path + } - assert_eq!( - classify_prompt_worker_error(true, &err), - PromptWorkerErrorDisposition::FallbackToStdio + #[cfg(unix)] + fn write_fake_codex_script_with_running_and_shutdown_delay( + temp: &TempDir, + running_delay_seconds: u64, + shutdown_delay_seconds: u64, + ) -> PathBuf { + let path = temp.path().join("fake-codex-running-and-shutdown-delay.sh"); + let body = format!( + "#!/bin/sh\ntrap 'sleep {}; exit 0' INT TERM\nprintf '{{\"type\":\"thread.started\",\"thread_id\":\"thread-test\"}}\\r\\n'\nsleep {}\nprintf '{{\"type\":\"turn.started\",\"thread_id\":\"thread-test\",\"turn_id\":\"turn-1\"}}\\r\\n'\nwhile :; do sleep 1; done\n", + shutdown_delay_seconds, running_delay_seconds ); + fs::write(&path, body).expect("write fake codex delayed script"); + let mut perms = fs::metadata(&path) + .expect("fake codex metadata") + .permissions(); + use std::os::unix::fs::PermissionsExt; + perms.set_mode(0o755); + fs::set_permissions(&path, perms).expect("set fake codex permissions"); + path } - #[test] - fn classify_prompt_worker_error_treats_reedline_failures_as_abnormal_terminal_loss() { - let err = reedline_terminal_loss_error(); - - assert_eq!( - classify_prompt_worker_error(true, &err), - PromptWorkerErrorDisposition::AbnormalTerminalLoss - ); + #[cfg(unix)] + fn participant_snapshot_path(store: &AgentRuntimeStateStore, participant_id: &str) -> PathBuf { + store + .participants_dir() + .join(format!("{participant_id}.json")) } - #[test] - fn classify_prompt_worker_error_treats_common_terminal_loss_errors_as_abnormal() { - let cases = common_terminal_loss_errors(); + #[cfg(unix)] + fn legacy_handle_snapshot_path( + store: &AgentRuntimeStateStore, + participant_id: &str, + ) -> PathBuf { + store.handles_dir().join(format!("{participant_id}.json")) + } - for err in &cases { - assert_eq!( - classify_prompt_worker_error(true, err), - PromptWorkerErrorDisposition::AbnormalTerminalLoss - ); + #[cfg(unix)] + fn session_state_name(state: &AgentRuntimeSessionState) -> &'static str { + match state { + AgentRuntimeSessionState::Allocating => "allocating", + AgentRuntimeSessionState::Ready => "ready", + AgentRuntimeSessionState::Running => "running", + AgentRuntimeSessionState::Restarting => "restarting", + AgentRuntimeSessionState::Stopping => "stopping", + AgentRuntimeSessionState::Stopped => "stopped", + AgentRuntimeSessionState::Failed => "failed", + AgentRuntimeSessionState::Invalidated => "invalidated", } } - #[test] - fn classify_prompt_worker_error_keeps_unrelated_reedline_errors_generic() { - let err = anyhow!("completion menu rendering failed"); + #[cfg(unix)] + fn read_persisted_participant_snapshot( + store: &AgentRuntimeStateStore, + participant_id: &str, + ) -> serde_json::Value { + let participant_path = participant_snapshot_path(store, participant_id); + assert!( + participant_path.exists(), + "participant snapshot should exist at {}", + participant_path.display() + ); - assert_eq!( - classify_prompt_worker_error(true, &err), - PromptWorkerErrorDisposition::GenericError + let legacy_handle_path = legacy_handle_snapshot_path(store, participant_id); + assert!( + !legacy_handle_path.exists(), + "writer cutover must not create legacy handle snapshots at {}", + legacy_handle_path.display() ); - } - #[test] - fn shutdown_disposition_tracks_termination_cause() { + let payload: serde_json::Value = serde_json::from_slice( + &fs::read(&participant_path).expect("read persisted participant snapshot"), + ) + .expect("parse persisted participant snapshot"); assert_eq!( - shutdown_disposition_for_termination_cause(ReplTerminationCause::NormalExit), - PromptWorkerShutdownDisposition::Graceful + payload + .get("participant_id") + .and_then(serde_json::Value::as_str), + Some(participant_id) ); + assert!( + payload.get("session_handle_id").is_none(), + "participant snapshot should not serialize legacy session_handle_id" + ); + payload + } + + #[cfg(unix)] + fn assert_persisted_participant_snapshot( + store: &AgentRuntimeStateStore, + participant_id: &str, + expected_state: &AgentRuntimeSessionState, + ) -> serde_json::Value { + let payload = read_persisted_participant_snapshot(store, participant_id); assert_eq!( - shutdown_disposition_for_termination_cause(ReplTerminationCause::AbnormalTerminalLoss), - PromptWorkerShutdownDisposition::Abandon + payload.get("state").and_then(serde_json::Value::as_str), + Some(session_state_name(expected_state)) ); + payload } - #[test] - fn resolve_reedline_ctrl_d_terminal_loss_returns_pre_recorded_message() { - let detected = AtomicBool::new(true); - let message = Mutex::new(Some("controlling terminal became invalid".to_string())); - let detect_calls = Cell::new(0usize); + #[cfg(unix)] + async fn wait_for_persisted_participant_snapshot( + store: &AgentRuntimeStateStore, + participant_id: &str, + expected_state: AgentRuntimeSessionState, + ) -> serde_json::Value { + tokio::time::timeout(Duration::from_secs(3), async { + loop { + let payload = read_persisted_participant_snapshot(store, participant_id); + if payload.get("state").and_then(serde_json::Value::as_str) + == Some(session_state_name(&expected_state)) + { + break payload; + } + tokio::time::sleep(Duration::from_millis(25)).await; + } + }) + .await + .expect("timed out waiting for participant snapshot state") + } - let err = resolve_reedline_ctrl_d_terminal_loss_with( - &detected, - &message, - || { - detect_calls.set(detect_calls.get() + 1); - None + #[cfg(unix)] + fn test_shell_config(workspace_root: &Path, substrate_home: &Path) -> ShellConfig { + ShellConfig { + mode: ShellMode::Interactive { use_pty: false }, + session_id: Uuid::now_v7().to_string(), + trace_log_file: substrate_home.join("trace.jsonl"), + original_path: std::env::var("PATH").unwrap_or_default(), + shim_dir: substrate_home.join("shims"), + shell_path: "/bin/sh".to_string(), + ci_mode: false, + no_exit_on_error: false, + skip_shims: true, + no_world: true, + cli_world: false, + cli_no_world: true, + cli_anchor_mode: None, + cli_anchor_path: None, + cli_caged: None, + world_root: WorldRootSettings { + mode: substrate_common::WorldRootMode::Project, + path: workspace_root.to_path_buf(), + caged: true, }, - || panic!("should not sleep when terminal loss is already recorded"), + async_repl: true, + repl_host_escape: false, + env_vars: HashMap::new(), + manager_init_path: substrate_home.join("manager-init.sh"), + manager_env_path: substrate_home.join("manager-env.sh"), + shimmed_path: None, + host_bash_env: None, + bash_preexec_path: substrate_home.join("bash-preexec.sh"), + preexec_available: false, + } + } + + #[cfg(unix)] + fn write_runtime_inventory_with_world_member( + substrate_home: &Path, + orchestrator_binary: &Path, + member_binary: &Path, + ) { + fs::write( + substrate_home.join("config.yaml"), + "agents:\n enabled: true\n hub:\n orchestrator_agent_id: claude_code\n", ) - .expect("expected recorded terminal loss"); + .expect("write config"); + fs::write( + substrate_home.join("policy.yaml"), + "agents:\n allowed_backends:\n - cli:claude_code\n - cli:codex\n", + ) + .expect("write policy"); + let agents_dir = substrate_home.join("agents"); + fs::create_dir_all(&agents_dir).expect("agents dir"); + fs::write( + agents_dir.join("claude_code.yaml"), + format!( + "version: 1\nid: claude_code\nconfig:\n kind: cli\n enabled: true\n protocol: {PURE_AGENT_PROTOCOL}\n execution:\n scope: host\n cli:\n binary: {}\n mode: persistent\n capabilities:\n session_start: true\n session_resume: true\n session_fork: true\n session_stop: true\n status_snapshot: true\n event_stream: true\n llm: true\n mcp_client: false\n", + orchestrator_binary.display() + ), + ) + .expect("write claude_code agent file"); + fs::write( + agents_dir.join("codex.yaml"), + format!( + "version: 1\nid: codex\nconfig:\n kind: cli\n enabled: true\n protocol: {PURE_AGENT_PROTOCOL}\n execution:\n scope: world\n cli:\n binary: {}\n mode: persistent\n capabilities:\n session_start: true\n session_resume: true\n session_fork: true\n session_stop: true\n status_snapshot: true\n event_stream: true\n llm: true\n mcp_client: false\n", + member_binary.display() + ), + ) + .expect("write codex agent file"); + } - assert_eq!(detect_calls.get(), 0); - assert!(err - .to_string() - .contains("controlling terminal became invalid")); + #[cfg(unix)] + fn seed_live_orchestrator_parent( + startup_context: &RuntimeOrchestrationContext, + manifest: &Arc>, + world_binding: &PersistedWorldBinding, + ) { + persist_world_binding_authority( + &startup_context.store, + &startup_context.orchestration_session, + Some(world_binding), + ) + .expect("persist parent world binding"); + let (orchestration_snapshot, manifest_snapshot) = { + let mut orchestration_guard = startup_context + .orchestration_session + .lock() + .expect("orchestration session mutex poisoned"); + let mut manifest_guard = manifest.lock().expect("runtime manifest mutex poisoned"); + manifest_guard.set_uaa_session_id("thread-parent".to_string()); + manifest_guard.mark_runtime_ownership_retained(); + manifest_guard.transition_state(AgentRuntimeSessionState::Ready); + manifest_guard.touch_heartbeat(); + orchestration_guard + .bind_active_session_handle(manifest_guard.handle.participant_id.clone()); + orchestration_guard.transition_state(OrchestrationSessionState::Active); + (orchestration_guard.clone(), manifest_guard.clone()) + }; + persist_runtime_snapshots( + &startup_context.store, + &orchestration_snapshot, + &manifest_snapshot, + ) + .expect("persist live orchestrator parent"); } + #[cfg(unix)] #[test] - fn resolve_reedline_ctrl_d_terminal_loss_retries_detector_within_window() { - let detected = AtomicBool::new(false); - let message = Mutex::new(None::); - let detect_calls = Cell::new(0usize); - let sleep_calls = Cell::new(0usize); + #[serial_test::serial] + fn start_host_orchestrator_runtime_persists_participant_snapshots_across_lifecycle_states() { + let _world_env_guard = crate::execution::world_env_guard(); + let temp = TempDir::new().expect("tempdir"); + let workspace_root = temp.path().join("workspace"); + let substrate_home = temp.path().join("substrate-home"); + fs::create_dir_all(&workspace_root).expect("workspace root"); + fs::create_dir_all(&substrate_home).expect("substrate home"); + let _cwd_guard = CurrentDirGuard::change_to(&workspace_root); + let fake_codex = write_fake_codex_script_with_running_and_shutdown_delay(&temp, 1, 1); - let err = resolve_reedline_ctrl_d_terminal_loss_with( - &detected, - &message, - || { - let next = detect_calls.get() + 1; - detect_calls.set(next); - if next == 3 { - Some(anyhow!("delayed terminal loss")) - } else { - None - } - }, - || sleep_calls.set(sleep_calls.get() + 1), + std::env::set_var("SUBSTRATE_HOME", &substrate_home); + fs::write( + substrate_home.join("config.yaml"), + "agents:\n enabled: true\n hub:\n orchestrator_agent_id: codex\n", ) - .expect("expected delayed terminal loss"); + .expect("write config"); + fs::write( + substrate_home.join("policy.yaml"), + "agents:\n allowed_backends:\n - cli:codex\n", + ) + .expect("write policy"); + let agents_dir = substrate_home.join("agents"); + fs::create_dir_all(&agents_dir).expect("agents dir"); + fs::write( + agents_dir.join("codex.yaml"), + format!( + "version: 1\nid: codex\nconfig:\n kind: cli\n enabled: true\n protocol: {PURE_AGENT_PROTOCOL}\n execution:\n scope: host\n cli:\n binary: {}\n mode: persistent\n capabilities:\n session_start: true\n session_resume: true\n session_fork: true\n session_stop: true\n status_snapshot: true\n event_stream: true\n llm: true\n mcp_client: false\n", + fake_codex.display() + ), + ) + .expect("write codex agent file"); - assert_eq!(detect_calls.get(), 3); - assert_eq!(sleep_calls.get(), 2); - assert!(err.to_string().contains("delayed terminal loss")); - } + let config = Arc::new(test_shell_config(&workspace_root, &substrate_home)); + let rt = tokio::runtime::Runtime::new().expect("tokio runtime"); + rt.block_on(async { + let mut telemetry = ReplSessionTelemetry::new(config.clone(), "async-test"); + let runtime = + start_host_orchestrator_runtime(&config, &ReplPrinter::Stdout, &mut telemetry) + .await + .expect("bootstrap runtime should succeed") + .expect("agents enabled should create a runtime"); + let store = runtime.store.clone(); - #[test] - fn resolve_reedline_ctrl_d_terminal_loss_observes_monitor_update_on_retry() { - let detected = AtomicBool::new(false); - let message = Mutex::new(None::); - let sleep_calls = Cell::new(0usize); + let (parent, live) = runtime + .store + .resolve_live_orchestrator_session("codex") + .expect("resolve live orchestrator session") + .expect("live orchestration session should exist"); + let participant_id = live.handle.participant_id.clone(); + assert_eq!(parent.state, OrchestrationSessionState::Active); + assert_eq!( + parent.active_session_handle_id.as_deref(), + Some(participant_id.as_str()) + ); + assert_eq!(parent.shell_trace_session_id, config.session_id); + assert_eq!( + fs::canonicalize(&parent.workspace_root).expect("canonicalize parent workspace"), + fs::canonicalize(&workspace_root).expect("canonicalize workspace root") + ); + assert_eq!( + runtime + .store + .find_active_orchestration_session_for_pid(std::process::id()) + .expect("find active orchestration session for pid") + .expect("active orchestration session should exist") + .orchestration_session_id, + parent.orchestration_session_id + ); + assert_eq!(live.handle.state, AgentRuntimeSessionState::Ready); + assert_eq!(live.internal.uaa_session_id.as_deref(), Some("thread-test")); + assert!(live.internal.ownership_valid); + assert!(live.internal.control_owner_retained); + assert!(live.internal.event_stream_active); + assert!(live.internal.completion_observer_retained); + assert_eq!(live.handle.protocol, PURE_AGENT_PROTOCOL); + assert_persisted_participant_snapshot( + &store, + &participant_id, + &AgentRuntimeSessionState::Ready, + ); - let err = resolve_reedline_ctrl_d_terminal_loss_with( - &detected, - &message, - || None, - || { - let next = sleep_calls.get() + 1; - sleep_calls.set(next); - if next == 1 { - detected.store(true, Ordering::SeqCst); - *message.lock().expect("message mutex poisoned") = - Some("monitor reported terminal loss".to_string()); - } - }, - ) - .expect("expected terminal loss from monitor update"); + wait_for_persisted_participant_snapshot( + &store, + &participant_id, + AgentRuntimeSessionState::Running, + ) + .await; - assert_eq!(sleep_calls.get(), 1); - assert!(err.to_string().contains("monitor reported terminal loss")); + let shutdown_config = config.clone(); + let shutdown_task = tokio::spawn(async move { + let mut shutdown_telemetry = + ReplSessionTelemetry::new(shutdown_config, "async-test-shutdown"); + shutdown_host_orchestrator_runtime( + runtime, + &ReplPrinter::Stdout, + &mut shutdown_telemetry, + ) + .await; + }); + + wait_for_persisted_participant_snapshot( + &store, + &participant_id, + AgentRuntimeSessionState::Stopping, + ) + .await; + let stopping_parent = store + .load_orchestration_session(&parent.orchestration_session_id) + .expect("load stopping orchestration session") + .expect("stopping orchestration session should exist"); + assert_eq!(stopping_parent.state, OrchestrationSessionState::Stopping); + assert_eq!( + stopping_parent.active_session_handle_id.as_deref(), + Some(participant_id.as_str()) + ); + + shutdown_task + .await + .expect("shutdown task should complete cleanly"); + + let manifests = store.list_manifests().expect("list manifests"); + let stopped = manifests + .into_iter() + .find(|manifest| manifest.handle.agent_id == "codex") + .expect("stopped manifest should exist"); + assert_eq!(stopped.handle.state, AgentRuntimeSessionState::Stopped); + assert!(!stopped.internal.ownership_valid); + assert!(!stopped.internal.control_owner_retained); + assert!(!stopped.internal.event_stream_active); + assert!(!stopped.internal.completion_observer_retained); + assert_persisted_participant_snapshot( + &store, + &participant_id, + &AgentRuntimeSessionState::Stopped, + ); + let stopped_parent = store + .load_orchestration_session(&stopped.handle.orchestration_session_id) + .expect("load stopped orchestration session") + .expect("stopped orchestration session should exist"); + assert_eq!(stopped_parent.state, OrchestrationSessionState::Stopped); + assert_eq!( + stopped_parent.active_session_handle_id.as_deref(), + Some(participant_id.as_str()) + ); + }); + std::env::remove_var("SUBSTRATE_HOME"); } + #[cfg(unix)] #[test] - fn resolve_reedline_ctrl_d_terminal_loss_returns_none_without_signal() { - let detected = AtomicBool::new(false); - let message = Mutex::new(None::); - let detect_calls = Cell::new(0usize); - let sleep_calls = Cell::new(0usize); - - let err = resolve_reedline_ctrl_d_terminal_loss_with( - &detected, - &message, - || { - detect_calls.set(detect_calls.get() + 1); - None - }, - || sleep_calls.set(sleep_calls.get() + 1), - ); + #[serial_test::serial] + fn start_host_orchestrator_runtime_parks_when_attached_control_exits() { + let _world_env_guard = crate::execution::world_env_guard(); + let temp = TempDir::new().expect("tempdir"); + let workspace_root = temp.path().join("workspace"); + let substrate_home = temp.path().join("substrate-home"); + fs::create_dir_all(&workspace_root).expect("workspace root"); + fs::create_dir_all(&substrate_home).expect("substrate home"); + let _cwd_guard = CurrentDirGuard::change_to(&workspace_root); + let fake_codex = write_fake_codex_script(&temp, false); - assert!(err.is_none()); - assert_eq!(detect_calls.get(), REEDLINE_CTRLD_TERMINAL_LOSS_RECHECKS); - assert_eq!(sleep_calls.get(), REEDLINE_CTRLD_TERMINAL_LOSS_RECHECKS - 1); - } + std::env::set_var("SUBSTRATE_HOME", &substrate_home); + fs::write( + substrate_home.join("config.yaml"), + "agents:\n enabled: true\n hub:\n orchestrator_agent_id: codex\n", + ) + .expect("write config"); + fs::write( + substrate_home.join("policy.yaml"), + "agents:\n allowed_backends:\n - cli:codex\n", + ) + .expect("write policy"); + let agents_dir = substrate_home.join("agents"); + fs::create_dir_all(&agents_dir).expect("agents dir"); + fs::write( + agents_dir.join("codex.yaml"), + format!( + "version: 1\nid: codex\nconfig:\n kind: cli\n enabled: true\n protocol: {PURE_AGENT_PROTOCOL}\n execution:\n scope: host\n cli:\n binary: {}\n mode: persistent\n capabilities:\n session_start: true\n session_resume: true\n session_fork: true\n session_stop: true\n status_snapshot: true\n event_stream: true\n llm: true\n mcp_client: false\n", + fake_codex.display() + ), + ) + .expect("write codex agent file"); - struct CurrentDirGuard { - original: PathBuf, - } + let config = Arc::new(test_shell_config(&workspace_root, &substrate_home)); + let rt = tokio::runtime::Runtime::new().expect("tokio runtime"); + rt.block_on(async { + let mut telemetry = ReplSessionTelemetry::new(config.clone(), "async-test"); + let runtime = + start_host_orchestrator_runtime(&config, &ReplPrinter::Stdout, &mut telemetry) + .await + .expect("runtime start should still observe attached ownership briefly") + .expect("agents enabled should create a runtime"); - impl CurrentDirGuard { - fn change_to(path: &Path) -> Self { - let original = std::env::current_dir().expect("current dir should resolve"); - std::env::set_current_dir(path).expect("set current dir"); - Self { original } - } - } + tokio::time::timeout(Duration::from_secs(2), async { + loop { + let store = AgentRuntimeStateStore::new().expect("state store"); + let manifests = store.list_manifests().expect("list manifests"); + let manifest = manifests + .into_iter() + .find(|manifest| manifest.handle.agent_id == "codex") + .expect("runtime manifest should exist"); + let parent = store + .load_orchestration_session(&manifest.handle.orchestration_session_id) + .expect("load orchestration session") + .expect("runtime orchestration session should exist"); + if parent.posture == OrchestrationSessionPosture::ParkedResumable + && !manifest.internal.ownership_valid + && !manifest.internal.control_owner_retained + && !manifest.internal.completion_observer_retained + && !manifest.attached_client_present() + && manifest.is_resume_eligible() + { + assert_eq!(parent.state, OrchestrationSessionState::Active); + assert_eq!(parent.attached_participant_id.as_deref(), None); + assert_eq!( + parent.active_session_handle_id.as_deref(), + Some(manifest.handle.participant_id.as_str()) + ); + assert_eq!(parent.shell_owner_pid, 0); + assert!(parent.closed_at.is_none()); + assert!(!manifest.internal.ownership_valid); + assert!(!manifest.internal.control_owner_retained); + assert!(!manifest.internal.completion_observer_retained); + assert!(!manifest.attached_client_present()); + assert!(manifest.is_resume_eligible()); + assert_eq!(manifest.internal.shell_owner_pid, 0); + assert_eq!( + manifest.internal.uaa_session_id.as_deref(), + Some("thread-test") + ); + break; + } + tokio::time::sleep(Duration::from_millis(25)).await; + } + }) + .await + .expect("runtime should park promptly after attached control exits"); - impl Drop for CurrentDirGuard { - fn drop(&mut self) { - let _ = std::env::set_current_dir(&self.original); - } + let live_orchestrator = AgentRuntimeStateStore::new() + .expect("state store") + .find_live_orchestrator("codex"); + assert!( + match &live_orchestrator { + Ok(None) => true, + Err(err) => err.to_string().contains("references inactive participant"), + Ok(Some(_)) => false, + }, + "parked detached control must disappear from authoritative live lookups: {live_orchestrator:?}" + ); + let live_session = AgentRuntimeStateStore::new() + .expect("state store") + .resolve_live_orchestrator_session("codex"); + assert!( + match &live_session { + Ok(None) => true, + Err(err) => err.to_string().contains("references inactive participant"), + Ok(Some(_)) => false, + }, + "parked detached control must disappear from parent-gated live resolution: {live_session:?}" + ); + shutdown_host_orchestrator_runtime(runtime, &ReplPrinter::Stdout, &mut telemetry).await; + }); + std::env::remove_var("SUBSTRATE_HOME"); } #[cfg(unix)] - fn write_fake_codex_script(temp: &TempDir, keep_alive: bool) -> PathBuf { - let path = temp.path().join("fake-codex.sh"); - let body = if keep_alive { - "#!/bin/sh\ntrap 'exit 0' INT TERM\nprintf '{\"type\":\"thread.started\",\"thread_id\":\"thread-test\"}\\r\\n'\nprintf '{\"type\":\"turn.started\",\"thread_id\":\"thread-test\",\"turn_id\":\"turn-1\"}\\r\\n'\nwhile :; do sleep 1; done\n" - } else { - "#!/bin/sh\nprintf '{\"type\":\"thread.started\",\"thread_id\":\"thread-test\"}\\r\\n'\nprintf '{\"type\":\"turn.started\",\"thread_id\":\"thread-test\",\"turn_id\":\"turn-1\"}\\r\\n'\n" - }; - fs::write(&path, body).expect("write fake codex script"); - let mut perms = fs::metadata(&path) - .expect("fake codex metadata") - .permissions(); - use std::os::unix::fs::PermissionsExt; - perms.set_mode(0o755); - fs::set_permissions(&path, perms).expect("set fake codex permissions"); - path - } + #[test] + #[serial_test::serial] + fn start_host_orchestrator_runtime_does_not_persist_live_manifest_without_session_handle() { + let _world_env_guard = crate::execution::world_env_guard(); + let temp = TempDir::new().expect("tempdir"); + let workspace_root = temp.path().join("workspace"); + let substrate_home = temp.path().join("substrate-home"); + fs::create_dir_all(&workspace_root).expect("workspace root"); + fs::create_dir_all(&substrate_home).expect("substrate home"); + let _cwd_guard = CurrentDirGuard::change_to(&workspace_root); + let fake_codex = write_fake_codex_script_without_session_handle(&temp); - #[cfg(unix)] - fn write_fake_codex_script_without_session_handle(temp: &TempDir) -> PathBuf { - let path = temp.path().join("fake-codex-no-session.sh"); - let body = "#!/bin/sh\nprintf 'bootstrap-without-session-handle\\n'\n"; - fs::write(&path, body).expect("write fake codex script without session handle"); - let mut perms = fs::metadata(&path) - .expect("fake codex metadata") - .permissions(); - use std::os::unix::fs::PermissionsExt; - perms.set_mode(0o755); - fs::set_permissions(&path, perms).expect("set fake codex permissions"); - path - } + std::env::set_var("SUBSTRATE_HOME", &substrate_home); + fs::write( + substrate_home.join("config.yaml"), + "agents:\n enabled: true\n hub:\n orchestrator_agent_id: codex\n", + ) + .expect("write config"); + fs::write( + substrate_home.join("policy.yaml"), + "agents:\n allowed_backends:\n - cli:codex\n", + ) + .expect("write policy"); + let agents_dir = substrate_home.join("agents"); + fs::create_dir_all(&agents_dir).expect("agents dir"); + fs::write( + agents_dir.join("codex.yaml"), + format!( + "version: 1\nid: codex\nconfig:\n kind: cli\n enabled: true\n protocol: {PURE_AGENT_PROTOCOL}\n execution:\n scope: host\n cli:\n binary: {}\n mode: persistent\n capabilities:\n session_start: true\n session_resume: true\n session_fork: true\n session_stop: true\n status_snapshot: true\n event_stream: true\n llm: true\n mcp_client: false\n", + fake_codex.display() + ), + ) + .expect("write codex agent file"); - #[cfg(unix)] - fn write_fake_codex_script_with_shutdown_delay(temp: &TempDir, delay_seconds: u64) -> PathBuf { - let path = temp.path().join("fake-codex-shutdown-delay.sh"); - let body = format!( - "#!/bin/sh\ntrap 'sleep {}; exit 0' INT TERM\nprintf '{{\"type\":\"thread.started\",\"thread_id\":\"thread-test\"}}\\r\\n'\nprintf '{{\"type\":\"turn.started\",\"thread_id\":\"thread-test\",\"turn_id\":\"turn-1\"}}\\r\\n'\nwhile :; do sleep 1; done\n", - delay_seconds - ); - fs::write(&path, body).expect("write fake codex shutdown delay script"); - let mut perms = fs::metadata(&path) - .expect("fake codex metadata") - .permissions(); - use std::os::unix::fs::PermissionsExt; - perms.set_mode(0o755); - fs::set_permissions(&path, perms).expect("set fake codex permissions"); - path - } + let config = Arc::new(test_shell_config(&workspace_root, &substrate_home)); + let rt = tokio::runtime::Runtime::new().expect("tokio runtime"); + rt.block_on(async { + let mut telemetry = ReplSessionTelemetry::new(config.clone(), "async-test"); + let failure = match start_host_orchestrator_runtime( + &config, + &ReplPrinter::Stdout, + &mut telemetry, + ) + .await + { + Ok(_) => panic!("bootstrap without a surfaced session handle must fail"), + Err(failure) => failure, + }; + assert!( + failure + .message + .contains("failed to establish attached control ownership") + || failure + .message + .contains("ended before ownership could be established"), + "bootstrap failure should explain the missing durable ownership boundary: {failure:?}" + ); - #[cfg(unix)] - fn test_shell_config(workspace_root: &Path, substrate_home: &Path) -> ShellConfig { - ShellConfig { - mode: ShellMode::Interactive { use_pty: false }, - session_id: Uuid::now_v7().to_string(), - trace_log_file: substrate_home.join("trace.jsonl"), - original_path: std::env::var("PATH").unwrap_or_default(), - shim_dir: substrate_home.join("shims"), - shell_path: "/bin/sh".to_string(), - ci_mode: false, - no_exit_on_error: false, - skip_shims: true, - no_world: true, - cli_world: false, - cli_no_world: true, - cli_anchor_mode: None, - cli_anchor_path: None, - cli_caged: None, - world_root: WorldRootSettings { - mode: substrate_common::WorldRootMode::Project, - path: workspace_root.to_path_buf(), - caged: true, - }, - async_repl: true, - repl_host_escape: false, - env_vars: HashMap::new(), - manager_init_path: substrate_home.join("manager-init.sh"), - manager_env_path: substrate_home.join("manager-env.sh"), - shimmed_path: None, - host_bash_env: None, - bash_preexec_path: substrate_home.join("bash-preexec.sh"), - preexec_available: false, - } + assert!( + AgentRuntimeStateStore::new() + .expect("state store") + .find_live_orchestrator("codex") + .expect("load live orchestrator") + .is_none(), + "bootstrap failure before session handle ownership must not leave a live manifest" + ); + assert!( + AgentRuntimeStateStore::new() + .expect("state store") + .resolve_live_orchestrator_session("codex") + .expect("resolve live orchestrator session") + .is_none(), + "bootstrap failure before session handle ownership must not resolve a live parent session" + ); + + let manifest = AgentRuntimeStateStore::new() + .expect("state store") + .list_manifests() + .expect("list manifests") + .into_iter() + .find(|manifest| manifest.handle.agent_id == "codex") + .expect("failed manifest should exist"); + assert_eq!(manifest.handle.state, AgentRuntimeSessionState::Failed); + assert!(!manifest.internal.ownership_valid); + assert_persisted_participant_snapshot( + &AgentRuntimeStateStore::new().expect("state store"), + &manifest.handle.participant_id, + &AgentRuntimeSessionState::Failed, + ); + let parent = AgentRuntimeStateStore::new() + .expect("state store") + .load_orchestration_session(&manifest.handle.orchestration_session_id) + .expect("load failed orchestration session") + .expect("failed orchestration session should exist"); + assert_eq!(parent.state, OrchestrationSessionState::Failed); + assert!(parent.active_session_handle_id.is_none()); + }); + std::env::remove_var("SUBSTRATE_HOME"); } #[cfg(unix)] #[test] #[serial_test::serial] - fn start_host_orchestrator_runtime_bootstraps_and_persists_a_live_manifest() { + fn shutdown_host_orchestrator_runtime_waits_for_cancel_completion_before_stopping() { let _world_env_guard = crate::execution::world_env_guard(); let temp = TempDir::new().expect("tempdir"); let workspace_root = temp.path().join("workspace"); @@ -3655,7 +8714,7 @@ mod tests { fs::create_dir_all(&workspace_root).expect("workspace root"); fs::create_dir_all(&substrate_home).expect("substrate home"); let _cwd_guard = CurrentDirGuard::change_to(&workspace_root); - let fake_codex = write_fake_codex_script(&temp, true); + let fake_codex = write_fake_codex_script_with_shutdown_delay(&temp, 1); std::env::set_var("SUBSTRATE_HOME", &substrate_home); fs::write( @@ -3688,38 +8747,39 @@ mod tests { .await .expect("bootstrap runtime should succeed") .expect("agents enabled should create a runtime"); + let store = runtime.store.clone(); + let shutdown_config = config.clone(); + let shutdown_task = tokio::spawn(async move { + let mut shutdown_telemetry = + ReplSessionTelemetry::new(shutdown_config, "async-test-shutdown"); + shutdown_host_orchestrator_runtime( + runtime, + &ReplPrinter::Stdout, + &mut shutdown_telemetry, + ) + .await; + }); - let live = runtime - .store - .find_live_orchestrator("codex") - .expect("load live orchestrator") - .expect("live orchestrator manifest should exist"); - assert!(matches!( - live.handle.state, - AgentRuntimeSessionState::Ready | AgentRuntimeSessionState::Running - )); - assert_eq!(live.internal.uaa_session_id.as_deref(), Some("thread-test")); - assert!(live.internal.ownership_valid); - assert!(live.internal.control_owner_retained); - assert!(live.internal.event_stream_active); - assert!(live.internal.completion_observer_retained); - assert_eq!(live.handle.protocol, PURE_AGENT_PROTOCOL); - - shutdown_host_orchestrator_runtime(runtime, &ReplPrinter::Stdout, &mut telemetry).await; + tokio::time::sleep(Duration::from_millis(200)).await; + assert!( + !shutdown_task.is_finished(), + "shutdown must stay blocked until the retained completion path resolves" + ); + shutdown_task + .await + .expect("shutdown task should complete cleanly"); - let manifests = AgentRuntimeStateStore::new() - .expect("state store") + let manifest = store .list_manifests() - .expect("list manifests"); - let stopped = manifests + .expect("list manifests") .into_iter() .find(|manifest| manifest.handle.agent_id == "codex") .expect("stopped manifest should exist"); - assert_eq!(stopped.handle.state, AgentRuntimeSessionState::Stopped); - assert!(!stopped.internal.ownership_valid); - assert!(!stopped.internal.control_owner_retained); - assert!(!stopped.internal.event_stream_active); - assert!(!stopped.internal.completion_observer_retained); + assert_eq!(manifest.handle.state, AgentRuntimeSessionState::Stopped); + assert_eq!( + manifest.internal.termination_reason.as_deref(), + Some("stopped") + ); }); std::env::remove_var("SUBSTRATE_HOME"); } @@ -3727,7 +8787,7 @@ mod tests { #[cfg(unix)] #[test] #[serial_test::serial] - fn start_host_orchestrator_runtime_invalidates_when_attached_control_exits() { + fn hidden_owner_private_stop_fails_closed_when_completion_never_resolves() { let _world_env_guard = crate::execution::world_env_guard(); let temp = TempDir::new().expect("tempdir"); let workspace_root = temp.path().join("workspace"); @@ -3735,7 +8795,7 @@ mod tests { fs::create_dir_all(&workspace_root).expect("workspace root"); fs::create_dir_all(&substrate_home).expect("substrate home"); let _cwd_guard = CurrentDirGuard::change_to(&workspace_root); - let fake_codex = write_fake_codex_script(&temp, false); + let fake_codex = write_fake_codex_script(&temp, true); std::env::set_var("SUBSTRATE_HOME", &substrate_home); fs::write( @@ -3762,55 +8822,110 @@ mod tests { let config = Arc::new(test_shell_config(&workspace_root, &substrate_home)); let rt = tokio::runtime::Runtime::new().expect("tokio runtime"); rt.block_on(async { + let prepared = prepare_host_orchestrator_runtime_startup(&config) + .expect("prepare host runtime should succeed") + .expect("host runtime should be configured"); let mut telemetry = ReplSessionTelemetry::new(config.clone(), "async-test"); - let runtime = - start_host_orchestrator_runtime(&config, &ReplPrinter::Stdout, &mut telemetry) - .await - .expect("runtime start should still observe attached ownership briefly") - .expect("agents enabled should create a runtime"); - - tokio::time::timeout(Duration::from_secs(2), async { - loop { - let manifests = AgentRuntimeStateStore::new() - .expect("state store") - .list_manifests() - .expect("list manifests"); - let manifest = manifests - .into_iter() - .find(|manifest| manifest.handle.agent_id == "codex") - .expect("runtime manifest should exist"); - if manifest.handle.state == AgentRuntimeSessionState::Invalidated { - assert!(!manifest.internal.ownership_valid); - assert!(!manifest.internal.control_owner_retained); - assert!(!manifest.internal.completion_observer_retained); - assert!(manifest.internal.terminal_observed_at.is_some()); - break; + let mut runtime = start_host_orchestrator_runtime_with_prepared_prompt( + Some(prepared), + None, + None, + true, + false, + &ReplPrinter::Stdout, + &mut telemetry, + ) + .await + .expect("hidden helper runtime should start") + .expect("runtime"); + let store = runtime.store.clone(); + let orchestration_session_id = runtime + .orchestration_session + .lock() + .expect("orchestration session mutex poisoned") + .orchestration_session_id + .clone(); + let participant_id = runtime + .manifest + .lock() + .expect("runtime manifest mutex poisoned") + .handle + .participant_id + .clone(); + + match &mut runtime.retained_control { + RetainedRunControl::Local(retained_control) => { + if let Some(task) = retained_control.completion_task.take() { + task.abort(); + let _ = task.await; } + retained_control.completion_task = + Some(tokio::spawn(std::future::pending::<()>())); + } + #[cfg(any(target_os = "linux", target_os = "macos"))] + RetainedRunControl::Remote(_) => { + panic!("host helper runtime should stay on the local retained-control path") + } + } + + let helper_config = Arc::clone(&config); + let helper_task = tokio::spawn(async move { + let mut helper_telemetry = + ReplSessionTelemetry::new(helper_config, "async-test-helper"); + wait_for_hidden_owner_helper_completion( + runtime, + &ReplPrinter::Stdout, + &mut helper_telemetry, + ) + .await + }); + let stop_transport_path = + crate::execution::agent_runtime::control::private_stop_transport_path( + &store, + &orchestration_session_id, + &participant_id, + ); + tokio::time::timeout(Duration::from_secs(3), async { + while !stop_transport_path.exists() { tokio::time::sleep(Duration::from_millis(25)).await; } }) .await - .expect("runtime should invalidate promptly after attached control exits"); - - assert!( - AgentRuntimeStateStore::new() - .expect("state store") - .find_live_orchestrator("codex") - .expect("load live orchestrator") - .is_none(), - "invalidated attached control must disappear from authoritative live lookups" - ); + .expect("hidden helper stop transport should be published"); - shutdown_host_orchestrator_runtime(runtime, &ReplPrinter::Stdout, &mut telemetry).await; + let outcome = crate::execution::agent_runtime::control::request_private_stop( + &stop_transport_path, + ) + .await + .expect("private stop request should connect"); + assert_eq!(outcome, PrivateStopOutcome::Accepted); - let manifest = AgentRuntimeStateStore::new() - .expect("state store") + let exit_code = helper_task + .await + .expect("hidden helper wait task should join") + .expect("hidden helper wait should return an exit code"); + assert_eq!(exit_code, 1); + + let parent = store + .load_orchestration_session(&orchestration_session_id) + .expect("load orchestration session") + .expect("terminal orchestration session should exist"); + assert_eq!(parent.state, OrchestrationSessionState::Failed); + assert_eq!(parent.posture, OrchestrationSessionPosture::Terminal); + assert!(parent.attached_participant_id.is_none()); + assert!(parent.closed_at.is_some()); + + let manifest = store .list_manifests() .expect("list manifests") .into_iter() - .find(|manifest| manifest.handle.agent_id == "codex") - .expect("runtime manifest should still exist"); - assert_eq!(manifest.handle.state, AgentRuntimeSessionState::Invalidated); + .find(|manifest| manifest.handle.participant_id == participant_id) + .expect("failed manifest should exist"); + assert_eq!(manifest.handle.state, AgentRuntimeSessionState::Failed); + assert_eq!( + manifest.internal.last_error_bucket.as_deref(), + Some("runtime_shutdown") + ); }); std::env::remove_var("SUBSTRATE_HOME"); } @@ -3818,15 +8933,16 @@ mod tests { #[cfg(unix)] #[test] #[serial_test::serial] - fn start_host_orchestrator_runtime_does_not_persist_live_manifest_without_session_handle() { + fn shutdown_host_orchestrator_runtime_parks_resumable_host_session_on_detach() { let _world_env_guard = crate::execution::world_env_guard(); let temp = TempDir::new().expect("tempdir"); let workspace_root = temp.path().join("workspace"); let substrate_home = temp.path().join("substrate-home"); + let pid_file = temp.path().join("fake-codex.pid"); fs::create_dir_all(&workspace_root).expect("workspace root"); fs::create_dir_all(&substrate_home).expect("substrate home"); let _cwd_guard = CurrentDirGuard::change_to(&workspace_root); - let fake_codex = write_fake_codex_script_without_session_handle(&temp); + let fake_codex = write_fake_codex_script_with_pid_file(&temp, &pid_file); std::env::set_var("SUBSTRATE_HOME", &substrate_home); fs::write( @@ -3854,63 +8970,84 @@ mod tests { let rt = tokio::runtime::Runtime::new().expect("tokio runtime"); rt.block_on(async { let mut telemetry = ReplSessionTelemetry::new(config.clone(), "async-test"); - let failure = match start_host_orchestrator_runtime( - &config, + let runtime = + start_host_orchestrator_runtime(&config, &ReplPrinter::Stdout, &mut telemetry) + .await + .expect("bootstrap runtime should succeed") + .expect("agents enabled should create a runtime"); + let store = runtime.store.clone(); + let session_id = runtime + .orchestration_session + .lock() + .expect("orchestration session mutex poisoned") + .orchestration_session_id + .clone(); + + shutdown_host_orchestrator_runtime_with_mode( + runtime, + HostRuntimeShutdownMode::ParkIfResumable, &ReplPrinter::Stdout, &mut telemetry, ) - .await - { - Ok(_) => panic!("bootstrap without a surfaced session handle must fail"), - Err(failure) => failure, - }; - assert!( - failure - .message - .contains("failed to establish attached control ownership") - || failure - .message - .contains("ended before ownership could be established"), - "bootstrap failure should explain the missing durable ownership boundary: {failure:?}" - ); + .await; + let target = store + .resolve_public_control_target( + &session_id, + crate::execution::agent_runtime::PublicControlAction::Resume, + ) + .expect("parked session should stay resume-eligible"); + assert_eq!(target.session.state, OrchestrationSessionState::Active); + assert_eq!( + target.session.posture, + OrchestrationSessionPosture::ParkedResumable + ); + assert!(target.session.attached_participant_id.is_none()); + assert_eq!( + target.session.active_session_handle_id.as_deref(), + Some(target.active_participant.handle.participant_id.as_str()) + ); + assert!(target.active_participant.handle.state.is_live()); + assert!(!target.active_participant.attached_client_present()); + assert!(target.active_participant.is_resume_eligible()); + assert!(!target.active_participant.internal.control_owner_retained); assert!( - AgentRuntimeStateStore::new() - .expect("state store") - .find_live_orchestrator("codex") - .expect("load live orchestrator") - .is_none(), - "bootstrap failure before session handle ownership must not leave a live manifest" + !target + .active_participant + .internal + .completion_observer_retained + ); + assert!(!target.active_participant.internal.event_stream_active); + assert_eq!( + target.active_participant.internal.detach_reason.as_deref(), + Some("owner detached cleanly") ); - - let manifest = AgentRuntimeStateStore::new() - .expect("state store") - .list_manifests() - .expect("list manifests") - .into_iter() - .find(|manifest| manifest.handle.agent_id == "codex") - .expect("failed manifest should exist"); - assert!(matches!( - manifest.handle.state, - AgentRuntimeSessionState::Failed | AgentRuntimeSessionState::Invalidated - )); - assert!(!manifest.internal.ownership_valid); }); + + let pid = fs::read_to_string(&pid_file) + .expect("pid file") + .trim() + .parse::() + .expect("pid should parse"); + unsafe { + libc::kill(pid, libc::SIGTERM); + } std::env::remove_var("SUBSTRATE_HOME"); } #[cfg(unix)] #[test] #[serial_test::serial] - fn shutdown_host_orchestrator_runtime_waits_for_cancel_completion_before_stopping() { + fn shutdown_host_orchestrator_runtime_fails_closed_when_detached_continuity_breaks() { let _world_env_guard = crate::execution::world_env_guard(); let temp = TempDir::new().expect("tempdir"); let workspace_root = temp.path().join("workspace"); let substrate_home = temp.path().join("substrate-home"); + let pid_file = temp.path().join("fake-codex.pid"); fs::create_dir_all(&workspace_root).expect("workspace root"); fs::create_dir_all(&substrate_home).expect("substrate home"); let _cwd_guard = CurrentDirGuard::change_to(&workspace_root); - let fake_codex = write_fake_codex_script_with_shutdown_delay(&temp, 1); + let fake_codex = write_fake_codex_script_with_pid_file(&temp, &pid_file); std::env::set_var("SUBSTRATE_HOME", &substrate_home); fs::write( @@ -3943,26 +9080,272 @@ mod tests { .await .expect("bootstrap runtime should succeed") .expect("agents enabled should create a runtime"); + let store = runtime.store.clone(); + let session_id = runtime + .orchestration_session + .lock() + .expect("orchestration session mutex poisoned") + .orchestration_session_id + .clone(); - let started_at = std::time::Instant::now(); - shutdown_host_orchestrator_runtime(runtime, &ReplPrinter::Stdout, &mut telemetry).await; + runtime + .orchestration_session + .lock() + .expect("orchestration session mutex poisoned") + .active_session_handle_id = Some("ash_wrong".to_string()); + + shutdown_host_orchestrator_runtime_with_mode( + runtime, + HostRuntimeShutdownMode::ParkIfResumable, + &ReplPrinter::Stdout, + &mut telemetry, + ) + .await; + + let session = store + .load_orchestration_session(&session_id) + .expect("load orchestration session") + .expect("terminal orchestration session should exist"); + assert!(session.state.is_terminal()); + assert_eq!(session.posture, OrchestrationSessionPosture::Terminal); + assert!(session.closed_at.is_some()); assert!( - started_at.elapsed() >= Duration::from_secs(1), - "shutdown must wait for the retained completion path before returning" + store + .resolve_public_control_target( + &session_id, + crate::execution::agent_runtime::PublicControlAction::Resume, + ) + .is_err(), + "invalid detached continuity must fail closed instead of parking" ); + }); - let manifest = AgentRuntimeStateStore::new() - .expect("state store") - .list_manifests() - .expect("list manifests") + let pid = fs::read_to_string(&pid_file) + .expect("pid file") + .trim() + .parse::() + .expect("pid should parse"); + unsafe { + libc::kill(pid, libc::SIGTERM); + } + std::env::remove_var("SUBSTRATE_HOME"); + } + + #[cfg(unix)] + #[test] + #[serial_test::serial] + fn start_member_runtime_reuses_parent_session_and_persists_world_binding() { + let _world_env_guard = crate::execution::world_env_guard(); + let temp = TempDir::new().expect("tempdir"); + let workspace_root = temp.path().join("workspace"); + let substrate_home = temp.path().join("substrate-home"); + fs::create_dir_all(&workspace_root).expect("workspace root"); + fs::create_dir_all(&substrate_home).expect("substrate home"); + let _cwd_guard = CurrentDirGuard::change_to(&workspace_root); + let fake_orchestrator = + write_fake_codex_script_with_running_and_shutdown_delay(&temp, 1, 1); + let fake_member = write_fake_codex_script_with_running_and_shutdown_delay(&temp, 1, 1); + + std::env::set_var("SUBSTRATE_HOME", &substrate_home); + write_runtime_inventory_with_world_member( + &substrate_home, + &fake_orchestrator, + &fake_member, + ); + + let config = Arc::new(test_shell_config(&workspace_root, &substrate_home)); + let rt = tokio::runtime::Runtime::new().expect("tokio runtime"); + rt.block_on(async { + let prepared = prepare_host_orchestrator_runtime_startup(&config) + .expect("prepare host runtime should succeed") + .expect("host runtime should be configured"); + let startup_context = prepared.startup_context.clone(); + let host_manifest = prepared.manifest.clone(); + let initial_world_binding = PersistedWorldBinding { + world_id: "wld_member_test".to_string(), + world_generation: 7, + }; + let mut telemetry = ReplSessionTelemetry::new(config.clone(), "async-test"); + seed_live_orchestrator_parent(&startup_context, &host_manifest, &initial_world_binding); + + let selected_descriptor = select_member_runtime_descriptor(&startup_context) + .expect("member selection should succeed") + .expect("one world-scoped member should be selected"); + let member_prepared = prepare_member_runtime_startup_for_descriptor( + &startup_context, + selected_descriptor, + &initial_world_binding, + None, + ) + .expect("member runtime prepare should succeed"); + let member_runtime = start_member_runtime_with_prepared( + Some(member_prepared), + &ReplPrinter::Stdout, + &mut telemetry, + ) + .await + .expect("member runtime start should succeed") + .expect("member runtime"); + + let parent = startup_context.snapshot(); + let live_members = startup_context + .store + .list_live_participants_for_session(&parent.orchestration_session_id) + .expect("list live participants") .into_iter() - .find(|manifest| manifest.handle.agent_id == "codex") - .expect("stopped manifest should exist"); - assert_eq!(manifest.handle.state, AgentRuntimeSessionState::Stopped); + .filter(|participant| participant.handle.role == MEMBER_ROLE) + .collect::>(); assert_eq!( - manifest.internal.termination_reason.as_deref(), - Some("stopped") + live_members.len(), + 1, + "expected exactly one live member runtime" + ); + let member = &live_members[0]; + assert_eq!(member.handle.agent_id, "codex"); + assert_eq!(member.handle.world_id.as_deref(), Some("wld_member_test")); + assert_eq!(member.handle.world_generation, Some(7)); + assert_eq!( + parent.active_session_handle_id.as_deref(), + Some( + host_manifest + .lock() + .expect("host runtime manifest") + .handle + .participant_id + .as_str() + ), + "member launch must not replace the authoritative orchestrator participant" + ); + + let member_snapshot = runtime_manifest_snapshot(&member_runtime); + assert_eq!(member_snapshot.handle.role, MEMBER_ROLE); + assert_eq!(member_snapshot.handle.world_generation, Some(7)); + assert!(member_snapshot.is_authoritative_live()); + + shutdown_host_orchestrator_runtime( + member_runtime, + &ReplPrinter::Stdout, + &mut telemetry, + ) + .await; + }); + std::env::remove_var("SUBSTRATE_HOME"); + } + + #[cfg(unix)] + #[test] + #[serial_test::serial] + fn prepare_member_replacement_runtime_preserves_resumed_from_lineage() { + let _world_env_guard = crate::execution::world_env_guard(); + let temp = TempDir::new().expect("tempdir"); + let workspace_root = temp.path().join("workspace"); + let substrate_home = temp.path().join("substrate-home"); + fs::create_dir_all(&workspace_root).expect("workspace root"); + fs::create_dir_all(&substrate_home).expect("substrate home"); + let _cwd_guard = CurrentDirGuard::change_to(&workspace_root); + let fake_orchestrator = + write_fake_codex_script_with_running_and_shutdown_delay(&temp, 1, 1); + let fake_member = write_fake_codex_script_with_running_and_shutdown_delay(&temp, 1, 1); + + std::env::set_var("SUBSTRATE_HOME", &substrate_home); + write_runtime_inventory_with_world_member( + &substrate_home, + &fake_orchestrator, + &fake_member, + ); + + let config = Arc::new(test_shell_config(&workspace_root, &substrate_home)); + let rt = tokio::runtime::Runtime::new().expect("tokio runtime"); + rt.block_on(async { + let prepared = prepare_host_orchestrator_runtime_startup(&config) + .expect("prepare host runtime should succeed") + .expect("host runtime should be configured"); + let startup_context = prepared.startup_context.clone(); + let host_manifest = prepared.manifest.clone(); + let initial_world_binding = PersistedWorldBinding { + world_id: "wld_member_old".to_string(), + world_generation: 2, + }; + let mut telemetry = ReplSessionTelemetry::new(config.clone(), "async-test"); + seed_live_orchestrator_parent(&startup_context, &host_manifest, &initial_world_binding); + + let selected_descriptor = select_member_runtime_descriptor(&startup_context) + .expect("member selection should succeed") + .expect("one world-scoped member should be selected"); + let member_prepared = prepare_member_runtime_startup_for_descriptor( + &startup_context, + selected_descriptor.clone(), + &initial_world_binding, + None, + ) + .expect("member runtime prepare should succeed"); + let member_runtime = start_member_runtime_with_prepared( + Some(member_prepared), + &ReplPrinter::Stdout, + &mut telemetry, + ) + .await + .expect("member runtime start should succeed") + .expect("member runtime"); + + let previous_member = runtime_manifest_snapshot(&member_runtime); + let replacement_binding = PersistedWorldBinding { + world_id: "wld_member_new".to_string(), + world_generation: 3, + }; + persist_world_binding_authority( + &startup_context.store, + &startup_context.orchestration_session, + Some(&replacement_binding), + ) + .expect("persist replacement binding"); + startup_context + .store + .invalidate_stale_world_members_for_session( + &startup_context.orchestration_session_id(), + replacement_binding.world_generation, + ) + .expect("invalidate stale world members"); + + let replacement_prepared = prepare_member_runtime_startup_for_descriptor( + &startup_context, + selected_descriptor, + &replacement_binding, + Some(&previous_member), + ) + .expect("replacement prepare should succeed"); + let replacement_manifest = replacement_prepared + .manifest + .lock() + .expect("replacement manifest") + .clone(); + assert_eq!(replacement_manifest.handle.role, MEMBER_ROLE); + assert_eq!( + replacement_manifest + .handle + .resumed_from_participant_id + .as_deref(), + Some(previous_member.handle.participant_id.as_str()) + ); + assert_ne!( + replacement_manifest.handle.participant_id, + previous_member.handle.participant_id + ); + assert_eq!( + replacement_manifest.handle.world_generation, + Some(replacement_binding.world_generation) + ); + assert_eq!( + replacement_manifest.handle.world_id.as_deref(), + Some(replacement_binding.world_id.as_str()) ); + + shutdown_host_orchestrator_runtime( + member_runtime, + &ReplPrinter::Stdout, + &mut telemetry, + ) + .await; }); std::env::remove_var("SUBSTRATE_HOME"); } diff --git a/crates/shell/tests/agent_hub_trace_persistence.rs b/crates/shell/tests/agent_hub_trace_persistence.rs index 504884cff..bd718cded 100644 --- a/crates/shell/tests/agent_hub_trace_persistence.rs +++ b/crates/shell/tests/agent_hub_trace_persistence.rs @@ -56,6 +56,42 @@ metadata: {} fs::write(project_dir.join(".substrate-profile"), profile).expect("write .substrate-profile"); } +fn write_host_only_profile(project_dir: &Path) { + let profile = r#"id: test-policy +name: Test Policy +world_fs: + host_visible: true + fail_closed: + routing: false + write: + enabled: true +net_allowed: [] +cmd_allowed: [] +cmd_denied: [] +cmd_isolated: [] +require_approval: false +allow_shell_operators: true +limits: + max_memory_mb: null + max_cpu_percent: null + max_runtime_ms: null + max_egress_bytes: null +metadata: {} +"#; + fs::write(project_dir.join(".substrate-profile"), profile).expect("write host-only profile"); +} + +fn write_workspace_world_disabled(project_dir: &Path) { + let workspace_dir = project_dir.join(".substrate"); + fs::create_dir_all(&workspace_dir).expect("create workspace config dir"); + fs::write( + workspace_dir.join("workspace.yaml"), + "world:\n enabled: false\n", + ) + .expect("write workspace.yaml"); +} + +#[allow(dead_code)] fn write_policy(home_substrate: &Path) { fs::create_dir_all(home_substrate).expect("create SUBSTRATE_HOME"); let policy = r#"id: test-global-policy @@ -82,6 +118,181 @@ metadata: {} fs::write(home_substrate.join("policy.yaml"), policy).expect("write policy.yaml"); } +fn write_host_only_policy(home_substrate: &Path) { + fs::create_dir_all(home_substrate).expect("create SUBSTRATE_HOME"); + let policy = r#"id: test-global-policy +name: Test Global Policy +world_fs: + host_visible: true + fail_closed: + routing: false + write: + enabled: true +net_allowed: [] +cmd_allowed: [] +cmd_denied: [] +cmd_isolated: [] +require_approval: false +allow_shell_operators: true +limits: + max_memory_mb: null + max_cpu_percent: null + max_runtime_ms: null + max_egress_bytes: null +metadata: {} +"#; + fs::write(home_substrate.join("policy.yaml"), policy).expect("write host-only policy.yaml"); +} + +fn write_orchestrator_runtime_config(home_substrate: &Path, fake_codex: &Path) { + fs::create_dir_all(home_substrate.join("agents")).expect("create agents dir"); + fs::write( + home_substrate.join("config.yaml"), + "agents:\n enabled: true\n hub:\n orchestrator_agent_id: codex\n", + ) + .expect("write config.yaml"); + fs::write( + home_substrate.join("policy.yaml"), + "id: test-global-policy\nname: Test Global Policy\nworld_fs:\n host_visible: true\n fail_closed:\n routing: true\n write:\n enabled: true\nnet_allowed: []\ncmd_allowed: []\ncmd_denied: []\ncmd_isolated: []\nrequire_approval: false\nallow_shell_operators: true\nlimits:\n max_memory_mb: null\n max_cpu_percent: null\n max_runtime_ms: null\n max_egress_bytes: null\nmetadata: {}\nagents:\n allowed_backends:\n - cli:codex\n", + ) + .expect("write agent runtime policy"); + fs::write( + home_substrate.join("agents/codex.yaml"), + format!( + "version: 1\nid: codex\nconfig:\n kind: cli\n enabled: true\n protocol: uaa.agent.session\n execution:\n scope: host\n cli:\n binary: {}\n mode: persistent\n capabilities:\n session_start: true\n session_resume: true\n session_fork: true\n session_stop: true\n status_snapshot: true\n event_stream: true\n llm: true\n mcp_client: false\n", + fake_codex.display() + ), + ) + .expect("write codex agent file"); +} + +#[cfg(unix)] +fn write_fake_codex_script(temp: &Path) -> PathBuf { + let path = temp.join("fake-codex.sh"); + let body = "#!/bin/sh\ntrap 'exit 0' INT TERM\nprintf '{\"type\":\"thread.started\",\"thread_id\":\"thread-test\"}\\r\\n'\nprintf '{\"type\":\"turn.started\",\"thread_id\":\"thread-test\",\"turn_id\":\"turn-1\"}\\r\\n'\nwhile :; do sleep 1; done\n"; + fs::write(&path, body).expect("write fake codex script"); + let mut perms = fs::metadata(&path) + .expect("fake codex metadata") + .permissions(); + use std::os::unix::fs::PermissionsExt; + perms.set_mode(0o755); + fs::set_permissions(&path, perms).expect("set fake codex permissions"); + path +} + +fn extract_session_id(output: &[u8]) -> String { + let text = String::from_utf8_lossy(output); + text.lines() + .find_map(|line| line.strip_prefix("Session ID: ").map(str::to_string)) + .expect("session id line") +} + +fn read_trace_events(trace_path: &Path) -> Vec { + fs::read_to_string(trace_path) + .expect("read trace") + .lines() + .filter(|line| !line.trim().is_empty()) + .map(|line| serde_json::from_str(line).expect("parse trace line")) + .collect() +} + +fn wait_for_trace_output(trace_path: &Path, needle: &str, timeout: Duration) -> Option { + let deadline = Instant::now() + timeout; + while Instant::now() < deadline { + if let Ok(contents) = fs::read_to_string(trace_path) { + if let Some(index) = contents.find(needle) { + return Some(index); + } + } + std::thread::sleep(Duration::from_millis(25)); + } + None +} + +fn sessions_dir(substrate_home: &Path) -> PathBuf { + substrate_home.join("run/agent-hub/sessions") +} + +fn canonical_session_path(substrate_home: &Path, orchestration_session_id: &str) -> PathBuf { + sessions_dir(substrate_home) + .join(orchestration_session_id) + .join("session.json") +} + +fn canonical_participant_path( + substrate_home: &Path, + orchestration_session_id: &str, + participant_id: &str, +) -> PathBuf { + sessions_dir(substrate_home) + .join(orchestration_session_id) + .join("participants") + .join(format!("{participant_id}.json")) +} + +fn canonical_lease_path( + substrate_home: &Path, + orchestration_session_id: &str, + participant_id: &str, +) -> PathBuf { + sessions_dir(substrate_home) + .join(orchestration_session_id) + .join("leases") + .join(format!("{participant_id}.lease")) +} + +fn load_single_session_record(substrate_home: &Path) -> Value { + let sessions_dir = sessions_dir(substrate_home); + let mut canonical_entries = fs::read_dir(&sessions_dir) + .expect("read orchestration session dir") + .filter_map(Result::ok) + .map(|entry| entry.path()) + .filter(|path| path.is_dir()) + .map(|path| path.join("session.json")) + .filter(|path| path.is_file()) + .collect::>(); + canonical_entries.sort(); + if let Some(session_path) = canonical_entries.into_iter().next() { + return serde_json::from_str::( + &fs::read_to_string(session_path).expect("read canonical session file"), + ) + .expect("parse canonical session file"); + } + + let sessions_dir = substrate_home.join("run/agent-hub/sessions"); + let mut entries = fs::read_dir(&sessions_dir) + .expect("read orchestration session dir") + .filter_map(Result::ok) + .map(|entry| entry.path()) + .filter(|path| path.extension().and_then(|ext| ext.to_str()) == Some("json")) + .collect::>(); + entries.sort(); + let session_path = entries + .into_iter() + .next() + .expect("orchestration session file"); + serde_json::from_str::( + &fs::read_to_string(session_path).expect("read flat session file"), + ) + .expect("parse flat session file") +} + +fn load_single_orchestration_session_id(substrate_home: &Path) -> String { + load_single_session_record(substrate_home) + .get("orchestration_session_id") + .and_then(Value::as_str) + .expect("session orchestration_session_id") + .to_string() +} + +fn load_single_active_participant_id(substrate_home: &Path) -> String { + load_single_session_record(substrate_home) + .get("active_session_handle_id") + .and_then(Value::as_str) + .expect("session active_session_handle_id") + .to_string() +} + struct PtyRepl { child: Box, master: Option>, @@ -116,6 +327,16 @@ impl PtyRepl { home_dir: &Path, substrate_home: &Path, trace_path: &Path, + ) -> Self { + Self::spawn_with_env(project_dir, home_dir, substrate_home, trace_path, &[]) + } + + fn spawn_with_env( + project_dir: &Path, + home_dir: &Path, + substrate_home: &Path, + trace_path: &Path, + extra_env: &[(&str, &str)], ) -> Self { ensure_substrate_built(); @@ -151,6 +372,9 @@ impl PtyRepl { cmd.env_remove("SUBSTRATE_WORLD_ENABLED"); cmd.env_remove("SUBSTRATE_WORLD_ID"); cmd.env("SHELL", "/bin/bash"); + for (key, value) in extra_env { + cmd.env(key, value); + } cmd.arg("--async-repl"); cmd.arg("--shim-skip"); @@ -165,6 +389,7 @@ impl PtyRepl { let writer_for_thread = Arc::downgrade(&writer); let reader_handle = std::thread::spawn(move || { let mut buf = [0u8; 4096]; + let mut carry = Vec::::new(); loop { if stop_for_thread.load(Ordering::Relaxed) { break; @@ -187,7 +412,10 @@ impl PtyRepl { let n: usize = rc as usize; let chunk = &buf[..n]; - if chunk.windows(4).any(|w| w == b"\x1b[6n") { + let mut probe = carry.clone(); + probe.extend_from_slice(chunk); + + if probe.windows(4).any(|w| w == b"\x1b[6n") { if let Some(writer) = writer_for_thread.upgrade() { if let Ok(mut w) = writer.lock() { let _ = w.write_all(b"\x1b[1;1R"); @@ -195,7 +423,7 @@ impl PtyRepl { } } } - if chunk.windows(5).any(|w| w == b"\x1b[18t") { + if probe.windows(5).any(|w| w == b"\x1b[18t") { if let Some(writer) = writer_for_thread.upgrade() { if let Ok(mut w) = writer.lock() { let _ = w.write_all(b"\x1b[8;24;80t"); @@ -204,6 +432,14 @@ impl PtyRepl { } } + carry.clear(); + let keep = 8usize; + if probe.len() > keep { + carry.extend_from_slice(&probe[probe.len() - keep..]); + } else { + carry.extend_from_slice(&probe); + } + if let Ok(mut guard) = output_for_thread.lock() { guard.extend_from_slice(chunk); } @@ -232,16 +468,18 @@ impl PtyRepl { writer.flush().expect("flush"); } + fn output_string(&self) -> String { + let out = self.output.lock().expect("output lock"); + String::from_utf8_lossy(&out).into_owned() + } + fn wait_for_output(&self, needle: &str, timeout: Duration) -> Option { let start = Instant::now(); while start.elapsed() < timeout { - let out = self.output.lock().expect("output lock"); - if let Ok(text) = std::str::from_utf8(&out) { - if let Some(pos) = text.find(needle) { - return Some(pos); - } + let text = self.output_string(); + if let Some(pos) = text.find(needle) { + return Some(pos); } - drop(out); std::thread::sleep(Duration::from_millis(10)); } None @@ -292,6 +530,7 @@ fn agent_events_append_flattened_agent_event_records_with_join_keys() { fs::create_dir_all(&substrate_home).expect("create substrate home"); fs::write(substrate_home.join("trace.jsonl"), "").expect("seed trace"); write_profile(&project); + write_workspace_world_disabled(&project); write_policy(&substrate_home); let trace_path = substrate_home.join("trace.jsonl"); @@ -299,11 +538,17 @@ fn agent_events_append_flattened_agent_event_records_with_join_keys() { repl.wait_for_output("Substrate v", Duration::from_secs(2)) .expect("banner"); + assert!( + repl.wait_for_output("substrate>", Duration::from_secs(5)) + .is_some(), + "initial prompt; output:\n{}", + repl.output_string() + ); repl.send_line(":demo-burst 1 1 0"); repl.wait_for_output( "scheduled burst: agents=1, events_per_agent=1, delay_ms=0", - Duration::from_secs(2), + Duration::from_secs(5), ) .expect("demo burst ack"); repl.wait_for_output("chunk #00000", Duration::from_secs(3)) @@ -378,6 +623,228 @@ fn agent_events_append_flattened_agent_event_records_with_join_keys() { } } +#[test] +#[serial] +fn runtime_owned_agent_event_rows_retain_shell_session_and_real_orchestration_session() { + let temp = temp_dir("substrate-runtime-agent-hub-trace-"); + let home = temp.path().join("home"); + let project = temp.path().join("project"); + let substrate_home = home.join(".substrate"); + fs::create_dir_all(substrate_home.join("shims")).expect("create shims dir"); + fs::create_dir_all(&project).expect("create project dir"); + fs::create_dir_all(&substrate_home).expect("create substrate home"); + fs::write(substrate_home.join("trace.jsonl"), "").expect("seed trace"); + write_profile(&project); + write_workspace_world_disabled(&project); + let fake_codex = write_fake_codex_script(temp.path()); + write_orchestrator_runtime_config(&substrate_home, &fake_codex); + + let trace_path = substrate_home.join("trace.jsonl"); + let mut repl = PtyRepl::spawn(&project, &home, &substrate_home, &trace_path); + + repl.wait_for_output("Substrate v", Duration::from_secs(2)) + .expect("banner"); + assert!( + repl.wait_for_output("substrate>", Duration::from_secs(5)) + .is_some(), + "initial prompt; output:\n{}", + repl.output_string() + ); + repl.send_line("::cli:codex trace owned runtime"); + let runtime_ready = repl.wait_for_output( + "shell-owned orchestrator session is ready via retained attached control ownership", + Duration::from_secs(5), + ); + if runtime_ready.is_none() { + let trace_ready = wait_for_trace_output( + &trace_path, + "shell-owned orchestrator session is ready via retained attached control ownership", + Duration::from_secs(5), + ); + assert!( + trace_ready.is_some(), + "runtime ready trace event; output:\n{}\ntrace:\n{}", + repl.output_string(), + fs::read_to_string(&trace_path).unwrap_or_default() + ); + } + + repl.send_line("exit"); + let (code, out) = repl.shutdown_graceful(Duration::from_secs(5)); + assert_eq!( + code, + 0, + "expected clean exit; output:\n{}", + String::from_utf8_lossy(&out) + ); + + let shell_session_id = extract_session_id(&out); + let orchestration_session_id = load_single_orchestration_session_id(&substrate_home); + assert_ne!( + shell_session_id, orchestration_session_id, + "shell trace session_id and orchestration_session_id must remain distinct identities" + ); + let participant_id = load_single_active_participant_id(&substrate_home); + let canonical_session = canonical_session_path(&substrate_home, &orchestration_session_id); + let canonical_participant = + canonical_participant_path(&substrate_home, &orchestration_session_id, &participant_id); + let canonical_lease = + canonical_lease_path(&substrate_home, &orchestration_session_id, &participant_id); + let legacy_handle = substrate_home + .join("run/agent-hub/handles") + .join(format!("{participant_id}.json")); + let events = read_trace_events(&trace_path); + let runtime_records = events + .iter() + .filter(|event| event.get("event_type").and_then(Value::as_str) == Some("agent_event")) + .filter(|event| event.get("agent_id").and_then(Value::as_str) == Some("codex")) + .collect::>(); + + assert!( + !runtime_records.is_empty(), + "expected runtime-owned codex agent_event rows in trace; got: {events:?}" + ); + assert!( + canonical_session.is_file(), + "runtime session persistence must write canonical session roots: {}", + canonical_session.display() + ); + assert!( + canonical_participant.is_file(), + "runtime session persistence must write canonical participant roots: {}", + canonical_participant.display() + ); + assert!( + canonical_lease.is_file(), + "runtime session persistence must write canonical lease roots: {}", + canonical_lease.display() + ); + assert!( + !legacy_handle.exists(), + "runtime session persistence must not resurrect handles/*.json authority: {}", + legacy_handle.display() + ); + + for record in runtime_records { + assert_eq!( + record.get("session_id").and_then(Value::as_str), + Some(shell_session_id.as_str()), + "runtime-owned agent_event row must retain the shell trace session_id: {record:?}" + ); + assert_eq!( + record + .get("orchestration_session_id") + .and_then(Value::as_str), + Some(orchestration_session_id.as_str()), + "runtime-owned agent_event row must retain the authoritative orchestration_session_id: {record:?}" + ); + assert_ne!( + record.get("orchestration_session_id").and_then(Value::as_str), + record.get("session_id").and_then(Value::as_str), + "runtime-owned agent_event row must not synthesize orchestration_session_id from session_id: {record:?}" + ); + assert_eq!( + record.get("participant_id").and_then(Value::as_str), + Some(participant_id.as_str()), + "runtime-owned agent_event row must retain participant_id from the live runtime manifest: {record:?}" + ); + assert!( + record.get("parent_participant_id").is_none(), + "runtime-owned orchestrator rows with no parent lineage must omit parent_participant_id: {record:?}" + ); + assert!( + record.get("resumed_from_participant_id").is_none(), + "runtime-owned orchestrator rows with no resume lineage must omit resumed_from_participant_id: {record:?}" + ); + } +} + +#[test] +#[serial] +fn no_context_shell_command_completion_does_not_synthesize_agent_event_trace_row() { + let temp = temp_dir("substrate-no-context-command-trace-"); + let home = temp.path().join("home"); + let project = temp.path().join("project"); + let substrate_home = home.join(".substrate"); + fs::create_dir_all(substrate_home.join("shims")).expect("create shims dir"); + fs::create_dir_all(&project).expect("create project dir"); + fs::create_dir_all(&substrate_home).expect("create substrate home"); + fs::write(substrate_home.join("trace.jsonl"), "").expect("seed trace"); + write_host_only_profile(&project); + write_workspace_world_disabled(&project); + write_host_only_policy(&substrate_home); + + let trace_path = substrate_home.join("trace.jsonl"); + let mut repl = PtyRepl::spawn_with_env( + &project, + &home, + &substrate_home, + &trace_path, + &[("SUBSTRATE_REPL_HOST_ESCAPE", "1")], + ); + + repl.wait_for_output("Substrate v", Duration::from_secs(2)) + .expect("banner"); + assert!( + repl.wait_for_output("substrate>", Duration::from_secs(5)) + .is_some(), + "initial prompt; output:\n{}", + repl.output_string() + ); + repl.send_line(":host false"); + repl.wait_for_output("Command failed with status: 1", Duration::from_secs(5)) + .expect("host failure output"); + repl.send_line("exit"); + let (code, out) = repl.shutdown_graceful(Duration::from_secs(5)); + assert_eq!( + code, + 0, + "expected clean exit; output:\n{}", + String::from_utf8_lossy(&out) + ); + + let shell_session_id = extract_session_id(&out); + let events = read_trace_events(&trace_path); + let repl_status_stop_records = events + .iter() + .filter(|event| event.get("event_type").and_then(Value::as_str) == Some("repl_status")) + .filter(|event| event.get("component").and_then(Value::as_str) == Some("shell")) + .filter(|event| { + event.get("session_id").and_then(Value::as_str) == Some(shell_session_id.as_str()) + }) + .filter(|event| event.get("stage").and_then(Value::as_str) == Some("stop")) + .collect::>(); + let shell_completion_records = events + .iter() + .filter(|event| event.get("event_type").and_then(Value::as_str) == Some("agent_event")) + .filter(|event| event.get("agent_id").and_then(Value::as_str) == Some("shell")) + .filter(|event| event.get("kind").and_then(Value::as_str) == Some("task_end")) + .collect::>(); + + assert!( + !repl_status_stop_records.is_empty(), + "suppressing a shell-owned orchestration agent_event row must remain additive; expected non-agent shell trace records to continue: {events:?}" + ); + for record in repl_status_stop_records { + assert_eq!( + record + .get("metrics") + .and_then(|metrics| metrics.get("commands_executed")) + .and_then(Value::as_u64), + Some(1), + "no-context shell completion should still contribute to persisted shell metrics even when the orchestration-scoped agent_event row is suppressed: {record:?}" + ); + assert!( + record.get("orchestration_session_id").is_none(), + "shell-owned non-agent trace records without orchestration context must not grow heuristic orchestration_session_id fields: {record:?}" + ); + } + assert!( + shell_completion_records.is_empty(), + "no-context shell completion must not synthesize an orchestration-scoped agent_event row: {shell_completion_records:?}" + ); +} + #[test] fn flattened_agent_event_records_retain_parent_run_id_when_present() { let mut event = AgentEvent::message( @@ -412,3 +879,299 @@ fn flattened_agent_event_records_retain_parent_run_id_when_present() { "flattened agent_event records must retain parent_run_id when present: {record:?}" ); } + +#[test] +fn flattened_agent_event_records_retain_participant_lineage_when_present() { + let mut event = AgentEvent::message( + "runtime-agent", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6f12", + "0195f8f1-7a35-7b7f-9c4d-9a7c2f5d6f14", + MessageEventKind::Status, + "runtime participant event", + ); + event.participant_id = Some("ash_live".to_string()); + event.parent_participant_id = Some("ash_parent".to_string()); + event.resumed_from_participant_id = Some("ash_previous".to_string()); + + let mut record = event.to_trace_record().expect("flatten agent event"); + let obj = record + .as_object_mut() + .expect("agent event trace record should be an object"); + obj.insert( + "event_type".to_string(), + Value::String("agent_event".to_string()), + ); + obj.insert( + "session_id".to_string(), + Value::String("ses_agent_hub".to_string()), + ); + obj.insert( + "component".to_string(), + Value::String("agent-hub".to_string()), + ); + + assert_eq!( + record.get("participant_id").and_then(Value::as_str), + Some("ash_live"), + "flattened agent_event records must retain participant_id when present: {record:?}" + ); + assert_eq!( + record.get("parent_participant_id").and_then(Value::as_str), + Some("ash_parent"), + "flattened agent_event records must retain parent_participant_id when present: {record:?}" + ); + assert_eq!( + record + .get("resumed_from_participant_id") + .and_then(Value::as_str), + Some("ash_previous"), + "flattened agent_event records must retain resumed_from_participant_id when present: {record:?}" + ); +} + +#[test] +fn world_member_registered_status_and_terminal_trace_rows_preserve_top_level_world_identity() { + let mut registered = AgentEvent::message( + "runtime-agent", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6f12", + "0195f8f1-7a35-7b7f-9c4d-9a7c2f5d6f15", + MessageEventKind::Registered, + "world member registered", + ); + registered.participant_id = Some("ash_member_live".to_string()); + registered.parent_participant_id = Some("ash_orchestrator_live".to_string()); + registered.role = Some("member".to_string()); + registered.world_id = Some("wld_trace_member_0001".to_string()); + registered.world_generation = Some(8); + + let mut status = AgentEvent::message( + "runtime-agent", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6f12", + "0195f8f1-7a35-7b7f-9c4d-9a7c2f5d6f16", + MessageEventKind::Status, + "world member still live", + ); + status.participant_id = Some("ash_member_live".to_string()); + status.parent_participant_id = Some("ash_orchestrator_live".to_string()); + status.role = Some("member".to_string()); + status.world_id = Some("wld_trace_member_0001".to_string()); + status.world_generation = Some(8); + + let mut terminal = AgentEvent::message( + "runtime-agent", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6f12", + "0195f8f1-7a35-7b7f-9c4d-9a7c2f5d6f17", + MessageEventKind::TaskEnd, + "world member exited cleanly", + ); + terminal.participant_id = Some("ash_member_live".to_string()); + terminal.parent_participant_id = Some("ash_orchestrator_live".to_string()); + terminal.role = Some("member".to_string()); + terminal.world_id = Some("wld_trace_member_0001".to_string()); + terminal.world_generation = Some(8); + + for (event, expected_kind) in [ + (registered, "registered"), + (status, "status"), + (terminal, "task_end"), + ] { + let mut record = event.to_trace_record().expect("flatten agent event"); + let obj = record + .as_object_mut() + .expect("agent event trace record should be an object"); + obj.insert( + "event_type".to_string(), + Value::String("agent_event".to_string()), + ); + obj.insert( + "session_id".to_string(), + Value::String("ses_agent_hub".to_string()), + ); + obj.insert( + "component".to_string(), + Value::String("agent-hub".to_string()), + ); + + assert_eq!( + record.get("kind").and_then(Value::as_str), + Some(expected_kind), + "member trace rows must preserve their event kind: {record:?}" + ); + assert_eq!( + record.get("participant_id").and_then(Value::as_str), + Some("ash_member_live") + ); + assert_eq!( + record.get("parent_participant_id").and_then(Value::as_str), + Some("ash_orchestrator_live") + ); + assert_eq!(record.get("role").and_then(Value::as_str), Some("member")); + assert_eq!( + record.get("world_id").and_then(Value::as_str), + Some("wld_trace_member_0001"), + "world-scoped member trace rows must keep world_id at the top level: {record:?}" + ); + assert_eq!( + record.get("world_generation").and_then(Value::as_u64), + Some(8), + "world-scoped member trace rows must keep world_generation at the top level: {record:?}" + ); + } +} + +#[test] +fn replacement_member_trace_rows_preserve_lineage_while_terminal_predecessors_stay_auditable() { + let mut predecessor_terminal = AgentEvent::message( + "runtime-agent", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6f12", + "0195f8f1-7a35-7b7f-9c4d-9a7c2f5d6f18", + MessageEventKind::TaskEnd, + "old world member terminated before replacement", + ); + predecessor_terminal.participant_id = Some("ash_member_old".to_string()); + predecessor_terminal.parent_participant_id = Some("ash_orchestrator_live".to_string()); + predecessor_terminal.role = Some("member".to_string()); + predecessor_terminal.world_id = Some("wld_trace_member_old".to_string()); + predecessor_terminal.world_generation = Some(7); + + let mut replacement_registered = AgentEvent::message( + "runtime-agent", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6f12", + "0195f8f1-7a35-7b7f-9c4d-9a7c2f5d6f19", + MessageEventKind::Registered, + "replacement world member registered", + ); + replacement_registered.participant_id = Some("ash_member_new".to_string()); + replacement_registered.parent_participant_id = Some("ash_orchestrator_live".to_string()); + replacement_registered.resumed_from_participant_id = Some("ash_member_old".to_string()); + replacement_registered.role = Some("member".to_string()); + replacement_registered.world_id = Some("wld_trace_member_new".to_string()); + replacement_registered.world_generation = Some(8); + + let mut terminal_record = predecessor_terminal + .to_trace_record() + .expect("flatten predecessor terminal event"); + let terminal_obj = terminal_record + .as_object_mut() + .expect("terminal trace record should be an object"); + terminal_obj.insert( + "event_type".to_string(), + Value::String("agent_event".to_string()), + ); + terminal_obj.insert( + "session_id".to_string(), + Value::String("ses_agent_hub".to_string()), + ); + terminal_obj.insert( + "component".to_string(), + Value::String("agent-hub".to_string()), + ); + + let mut replacement_record = replacement_registered + .to_trace_record() + .expect("flatten replacement registered event"); + let replacement_obj = replacement_record + .as_object_mut() + .expect("replacement trace record should be an object"); + replacement_obj.insert( + "event_type".to_string(), + Value::String("agent_event".to_string()), + ); + replacement_obj.insert( + "session_id".to_string(), + Value::String("ses_agent_hub".to_string()), + ); + replacement_obj.insert( + "component".to_string(), + Value::String("agent-hub".to_string()), + ); + + assert_eq!( + terminal_record.get("kind").and_then(Value::as_str), + Some("task_end"), + "stale terminal rows must remain auditable as terminal events: {terminal_record:?}" + ); + assert_eq!( + terminal_record + .get("participant_id") + .and_then(Value::as_str), + Some("ash_member_old") + ); + assert_eq!( + terminal_record.get("world_id").and_then(Value::as_str), + Some("wld_trace_member_old") + ); + assert!( + terminal_record.get("resumed_from_participant_id").is_none(), + "terminal predecessor rows must not be rewritten with replacement lineage: {terminal_record:?}" + ); + + assert_eq!( + replacement_record.get("kind").and_then(Value::as_str), + Some("registered") + ); + assert_eq!( + replacement_record + .get("participant_id") + .and_then(Value::as_str), + Some("ash_member_new") + ); + assert_eq!( + replacement_record + .get("resumed_from_participant_id") + .and_then(Value::as_str), + Some("ash_member_old"), + "replacement trace rows must preserve resumed_from_participant_id lineage at the top level: {replacement_record:?}" + ); + assert_eq!( + replacement_record.get("world_id").and_then(Value::as_str), + Some("wld_trace_member_new") + ); + assert_eq!( + replacement_record + .get("world_generation") + .and_then(Value::as_u64), + Some(8) + ); +} + +#[test] +fn world_alert_rows_omit_synthesized_participant_lineage_without_runtime_context() { + let event = AgentEvent::alert( + "shell", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6f12", + "0195f8f1-7a35-7b7f-9c4d-9a7c2f5d6f14", + "world_restart_required", + "world restart required before continuing", + ); + + let mut record = event.to_trace_record().expect("flatten agent event"); + let obj = record + .as_object_mut() + .expect("agent event trace record should be an object"); + obj.insert( + "event_type".to_string(), + Value::String("agent_event".to_string()), + ); + obj.insert( + "session_id".to_string(), + Value::String("ses_agent_hub".to_string()), + ); + obj.insert( + "component".to_string(), + Value::String("agent-hub".to_string()), + ); + + assert!( + record.get("participant_id").is_none(), + "world alert rows without runtime context must not synthesize participant_id: {record:?}" + ); + assert!( + record.get("parent_participant_id").is_none(), + "world alert rows without runtime context must not synthesize parent_participant_id: {record:?}" + ); + assert!( + record.get("resumed_from_participant_id").is_none(), + "world alert rows without runtime context must not synthesize resumed_from_participant_id: {record:?}" + ); +} diff --git a/crates/shell/tests/agent_public_control_surface_v1.rs b/crates/shell/tests/agent_public_control_surface_v1.rs new file mode 100644 index 000000000..8410d5029 --- /dev/null +++ b/crates/shell/tests/agent_public_control_surface_v1.rs @@ -0,0 +1,3494 @@ +#![cfg(any(target_os = "linux", target_os = "macos"))] + +#[path = "support/mod.rs"] +mod support; + +use portable_pty::{native_pty_system, CommandBuilder, PtySize}; +use serde_json::{json, Value}; +use serial_test::serial; +use std::fs; +use std::io::{BufRead, Write}; +#[cfg(unix)] +use std::os::unix::fs::PermissionsExt; +use std::path::{Path, PathBuf}; +use std::process::{Child, Output, Stdio}; +use std::sync::atomic::{AtomicBool, Ordering}; +use std::sync::{Arc, Mutex}; +use std::time::{Duration, Instant}; +use support::{ + binary_path, ensure_substrate_built, persist_runtime_alert_for_substrate_home, + substrate_shell_driver, +}; +#[cfg(target_os = "linux")] +use support::{MemberDispatchStreamScript, ReplWorldAgentStub, StreamBehavior}; +use tempfile::TempDir; + +const PURE_AGENT_PROTOCOL: &str = "uaa.agent.session"; +#[cfg(unix)] +const PRIVATE_STOP_UNIX_PATH_MAX: usize = 100; + +#[cfg(unix)] +fn set_fd_nonblocking(fd: i32) { + unsafe { + let flags = libc::fcntl(fd, libc::F_GETFL); + if flags < 0 { + return; + } + let _ = libc::fcntl(fd, libc::F_SETFL, flags | libc::O_NONBLOCK); + } +} + +fn manager_manifest_path() -> PathBuf { + PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("../../config/manager_hooks.yaml") +} + +struct AgentControlFixture { + _temp: TempDir, + home: PathBuf, + substrate_home: PathBuf, + workspace_root: PathBuf, + fake_codex: PathBuf, +} + +impl AgentControlFixture { + fn new() -> Self { + Self::new_with_fake_codex(write_fake_codex_script) + } + + fn new_with_fake_codex(script_writer: fn(&Path) -> PathBuf) -> Self { + let temp = tempfile::Builder::new() + .prefix("sac-") + .tempdir_in("/tmp") + .expect("allocate short temp dir"); + let home = temp.path().join("h"); + let substrate_home = temp.path().join("s"); + let workspace_root = temp.path().join("w"); + fs::create_dir_all(&home).expect("create HOME"); + fs::create_dir_all(&substrate_home).expect("create SUBSTRATE_HOME"); + fs::create_dir_all(substrate_home.join("shims")).expect("create shims dir"); + fs::create_dir_all(&workspace_root).expect("create workspace root"); + fs::write(substrate_home.join("trace.jsonl"), "").expect("seed trace"); + let fake_codex = script_writer(temp.path()); + Self { + _temp: temp, + home, + substrate_home, + workspace_root, + fake_codex, + } + } + + fn command(&self) -> assert_cmd::Command { + let mut cmd = substrate_shell_driver(); + cmd.env("HOME", &self.home) + .env("USERPROFILE", &self.home) + .env("SUBSTRATE_HOME", &self.substrate_home) + .env("SUBSTRATE_MANAGER_MANIFEST", manager_manifest_path()) + .env("SHIM_TRACE_LOG", self.trace_path()); + cmd + } + + fn init_workspace(&self) { + let output = self + .command() + .arg("workspace") + .arg("init") + .arg(&self.workspace_root) + .arg("--force") + .output() + .expect("run workspace init"); + assert!( + output.status.success(), + "workspace init should succeed: {output:?}" + ); + } + + fn write_runtime_inventory(&self, include_world_backend: bool) { + fs::create_dir_all(self.substrate_home.join("agents")).expect("create agents dir"); + let allowed_backends = if include_world_backend { + " - cli:codex\n - cli:claude_code\n" + } else { + " - cli:codex\n" + }; + fs::write( + self.substrate_home.join("config.yaml"), + "agents:\n enabled: true\n hub:\n orchestrator_agent_id: codex\n toolbox:\n enabled: true\n bind:\n transport: uds\n", + ) + .expect("write config.yaml"); + fs::write( + self.substrate_home.join("policy.yaml"), + format!( + "id: test-global-policy\nname: Test Global Policy\nworld_fs:\n host_visible: true\n fail_closed:\n routing: true\n write:\n enabled: true\nnet_allowed: []\ncmd_allowed: []\ncmd_denied: []\ncmd_isolated: []\nrequire_approval: false\nallow_shell_operators: true\nlimits:\n max_memory_mb: null\n max_cpu_percent: null\n max_runtime_ms: null\n max_egress_bytes: null\nmetadata: {{}}\nagents:\n allowed_backends:\n{allowed_backends}", + ), + ) + .expect("write policy.yaml"); + fs::write( + self.workspace_root.join(".substrate-profile"), + "id: test-policy\nname: Test Policy\nworld_fs:\n host_visible: true\n fail_closed:\n routing: true\n write:\n enabled: true\nnet_allowed: []\ncmd_allowed: []\ncmd_denied: []\ncmd_isolated: []\nrequire_approval: false\nallow_shell_operators: true\nlimits:\n max_memory_mb: null\n max_cpu_percent: null\n max_runtime_ms: null\n max_egress_bytes: null\nmetadata: {}\n", + ) + .expect("write .substrate-profile"); + fs::write( + self.substrate_home.join("agents/codex.yaml"), + cli_agent_file("codex", "host", &self.fake_codex), + ) + .expect("write codex agent file"); + if include_world_backend { + fs::write( + self.substrate_home.join("agents/claude_code.yaml"), + cli_agent_file("claude_code", "world", &self.fake_codex), + ) + .expect("write claude_code agent file"); + } + } + + fn run(&self, args: &[&str]) -> Output { + self.command() + .current_dir(&self.workspace_root) + .args(args) + .output() + .expect("run substrate command") + } + + fn spawn(&self, args: &[&str]) -> Child { + ensure_substrate_built(); + + std::process::Command::new(binary_path()) + .env("HOME", &self.home) + .env("USERPROFILE", &self.home) + .env("SUBSTRATE_HOME", &self.substrate_home) + .env("SUBSTRATE_MANAGER_MANIFEST", manager_manifest_path()) + .env("SHIM_TRACE_LOG", self.trace_path()) + .current_dir(&self.workspace_root) + .args(args) + .stdout(Stdio::piped()) + .stderr(Stdio::piped()) + .spawn() + .expect("spawn substrate command") + } + + fn trace_path(&self) -> PathBuf { + self.substrate_home.join("trace.jsonl") + } + + fn load_orchestration_session(&self, orchestration_session_id: &str) -> Value { + read_json_file(&canonical_orchestration_session_path( + &self.substrate_home, + orchestration_session_id, + )) + } + + fn load_participant(&self, orchestration_session_id: &str, participant_id: &str) -> Value { + read_json_file(&canonical_participant_manifest_path( + &self.substrate_home, + orchestration_session_id, + participant_id, + )) + } + + fn reset_fake_codex_state(&self) { + let state_path = self + .fake_codex + .parent() + .expect("fake codex parent") + .join("fake-codex.count"); + let _ = fs::remove_file(state_path); + } + + fn fake_codex_args_path(&self, invocation: u32) -> PathBuf { + self.fake_codex + .parent() + .expect("fake codex parent") + .join(format!("fake-codex-{invocation}.args")) + } + + fn fake_codex_stdin_path(&self, invocation: u32) -> PathBuf { + self.fake_codex + .parent() + .expect("fake codex parent") + .join(format!("fake-codex-{invocation}.stdin")) + } + + fn read_fake_codex_args(&self, invocation: u32) -> Vec { + fs::read_to_string(self.fake_codex_args_path(invocation)) + .unwrap_or_else(|err| panic!("read fake codex args {invocation}: {err}")) + .lines() + .map(|line| line.to_string()) + .collect() + } + + fn read_fake_codex_stdin(&self, invocation: u32) -> String { + fs::read_to_string(self.fake_codex_stdin_path(invocation)) + .unwrap_or_else(|err| panic!("read fake codex stdin {invocation}: {err}")) + } +} + +fn cli_agent_file(agent_id: &str, scope: &str, binary: &Path) -> String { + format!( + "version: 1\nid: {agent_id}\nconfig:\n kind: cli\n enabled: true\n protocol: {PURE_AGENT_PROTOCOL}\n execution:\n scope: {scope}\n cli:\n binary: {}\n mode: persistent\n capabilities:\n session_start: true\n session_resume: true\n session_fork: true\n session_stop: true\n status_snapshot: true\n event_stream: true\n llm: true\n mcp_client: false\n", + binary.display() + ) +} + +fn write_fake_codex_script(dir: &Path) -> PathBuf { + let path = dir.join("fake-codex.sh"); + let count_path = dir.join("fake-codex.count"); + let body = format!( + "#!/bin/sh\nSTATE_FILE='{}'\nSCRIPT_DIR='{}'\ncount=0\nif [ -f \"$STATE_FILE\" ]; then\n count=$(cat \"$STATE_FILE\")\nfi\ncount=$((count + 1))\nprintf '%s' \"$count\" > \"$STATE_FILE\"\nprintf '%s\\n' \"$@\" > \"$SCRIPT_DIR/fake-codex-$count.args\"\ncat > \"$SCRIPT_DIR/fake-codex-$count.stdin\"\nif [ \"$count\" -eq 1 ]; then\n trap 'exit 0' INT TERM\n printf '{{\"type\":\"thread.started\",\"thread_id\":\"thread-test\"}}\\r\\n'\n printf '{{\"type\":\"turn.started\",\"thread_id\":\"thread-test\",\"turn_id\":\"turn-1\"}}\\r\\n'\n printf '{{\"type\":\"item.completed\",\"thread_id\":\"thread-test\",\"turn_id\":\"turn-1\",\"item_id\":\"msg-1\",\"status\":\"completed\",\"item_type\":\"agent_message\",\"content\":{{\"text\":\"startup prompt success\"}}}}\\r\\n'\n printf '{{\"type\":\"turn.completed\",\"thread_id\":\"thread-test\",\"turn_id\":\"turn-1\"}}\\r\\n'\n while :; do sleep 1; done\nfi\nprintf '{{\"type\":\"thread.resumed\",\"thread_id\":\"thread-test\"}}\\r\\n'\nprintf '{{\"type\":\"turn.started\",\"thread_id\":\"thread-test\",\"turn_id\":\"turn-%s\"}}\\r\\n' \"$count\"\nprintf '{{\"type\":\"item.completed\",\"thread_id\":\"thread-test\",\"turn_id\":\"turn-%s\",\"item_id\":\"msg-%s\",\"status\":\"completed\",\"item_type\":\"agent_message\",\"content\":{{\"text\":\"follow-up prompt success\"}}}}\\r\\n' \"$count\" \"$count\"\nprintf '{{\"type\":\"turn.completed\",\"thread_id\":\"thread-test\",\"turn_id\":\"turn-%s\"}}\\r\\n' \"$count\"\nexit 0\n", + count_path.display() + , + dir.display() + ); + fs::write(&path, body).expect("write fake codex script"); + let mut perms = fs::metadata(&path) + .expect("fake codex metadata") + .permissions(); + perms.set_mode(0o755); + fs::set_permissions(&path, perms).expect("set fake codex permissions"); + path +} + +fn write_fake_codex_script_exit_after_startup_prompt(dir: &Path) -> PathBuf { + let path = dir.join("fake-codex-exit-after-startup-prompt.sh"); + let count_path = dir.join("fake-codex-exit-after-startup-prompt.count"); + let body = format!( + "#!/bin/sh\nSTATE_FILE='{}'\nSCRIPT_DIR='{}'\ncount=0\nif [ -f \"$STATE_FILE\" ]; then\n count=$(cat \"$STATE_FILE\")\nfi\ncount=$((count + 1))\nprintf '%s' \"$count\" > \"$STATE_FILE\"\nprintf '%s\\n' \"$@\" > \"$SCRIPT_DIR/fake-codex-$count.args\"\ncat > \"$SCRIPT_DIR/fake-codex-$count.stdin\"\nif [ \"$count\" -eq 1 ]; then\n printf '{{\"type\":\"thread.started\",\"thread_id\":\"thread-test\"}}\\r\\n'\n printf '{{\"type\":\"turn.started\",\"thread_id\":\"thread-test\",\"turn_id\":\"turn-1\"}}\\r\\n'\n printf '{{\"type\":\"item.completed\",\"thread_id\":\"thread-test\",\"turn_id\":\"turn-1\",\"item_id\":\"msg-1\",\"status\":\"completed\",\"item_type\":\"agent_message\",\"content\":{{\"text\":\"startup prompt success\"}}}}\\r\\n'\n printf '{{\"type\":\"turn.completed\",\"thread_id\":\"thread-test\",\"turn_id\":\"turn-1\"}}\\r\\n'\n exit 0\nfi\nprintf '{{\"type\":\"thread.resumed\",\"thread_id\":\"thread-test\"}}\\r\\n'\nprintf '{{\"type\":\"turn.started\",\"thread_id\":\"thread-test\",\"turn_id\":\"turn-%s\"}}\\r\\n' \"$count\"\nprintf '{{\"type\":\"item.completed\",\"thread_id\":\"thread-test\",\"turn_id\":\"turn-%s\",\"item_id\":\"msg-%s\",\"status\":\"completed\",\"item_type\":\"agent_message\",\"content\":{{\"text\":\"follow-up prompt success\"}}}}\\r\\n' \"$count\" \"$count\"\nprintf '{{\"type\":\"turn.completed\",\"thread_id\":\"thread-test\",\"turn_id\":\"turn-%s\"}}\\r\\n' \"$count\"\nexit 0\n", + count_path.display(), + dir.display() + ); + fs::write(&path, body).expect("write exit-after-startup fake codex script"); + let mut perms = fs::metadata(&path) + .expect("exit-after-startup fake codex metadata") + .permissions(); + perms.set_mode(0o755); + fs::set_permissions(&path, perms).expect("set exit-after-startup fake codex permissions"); + path +} + +fn write_fake_codex_script_turn_requires_helper_cancel_after_prompt(dir: &Path) -> PathBuf { + let path = dir.join("fake-codex-turn-requires-helper-cancel.sh"); + let count_path = dir.join("fake-codex-turn-requires-helper-cancel.count"); + let body = format!( + "#!/bin/sh\nSTATE_FILE='{}'\nSCRIPT_DIR='{}'\ncount=0\nif [ -f \"$STATE_FILE\" ]; then\n count=$(cat \"$STATE_FILE\")\nfi\ncount=$((count + 1))\nprintf '%s' \"$count\" > \"$STATE_FILE\"\nprintf '%s\\n' \"$@\" > \"$SCRIPT_DIR/fake-codex-$count.args\"\ncat > \"$SCRIPT_DIR/fake-codex-$count.stdin\"\nif [ \"$count\" -eq 1 ]; then\n trap 'exit 0' INT TERM\n printf '{{\"type\":\"thread.resumed\",\"thread_id\":\"thread-test\"}}\\r\\n'\n printf '{{\"type\":\"turn.started\",\"thread_id\":\"thread-test\",\"turn_id\":\"turn-bootstrap\"}}\\r\\n'\n while :; do sleep 1; done\nfi\nif [ \"$count\" -eq 2 ]; then\n printf '{{\"type\":\"thread.resumed\",\"thread_id\":\"thread-test\"}}\\r\\n'\n printf '{{\"type\":\"turn.started\",\"thread_id\":\"thread-test\",\"turn_id\":\"turn-%s\"}}\\r\\n' \"$count\"\n printf '{{\"type\":\"item.completed\",\"thread_id\":\"thread-test\",\"turn_id\":\"turn-%s\",\"item_id\":\"msg-%s\",\"status\":\"completed\",\"item_type\":\"agent_message\",\"content\":{{\"text\":\"follow-up prompt success\"}}}}\\r\\n' \"$count\" \"$count\"\n printf '{{\"type\":\"turn.completed\",\"thread_id\":\"thread-test\",\"turn_id\":\"turn-%s\"}}\\r\\n' \"$count\"\n exit 0\nfi\ntrap 'exit 0' INT TERM\nprintf '{{\"type\":\"thread.resumed\",\"thread_id\":\"thread-test\"}}\\r\\n'\nprintf '{{\"type\":\"turn.started\",\"thread_id\":\"thread-test\",\"turn_id\":\"turn-%s\"}}\\r\\n' \"$count\"\nwhile :; do sleep 1; done\n", + count_path.display(), + dir.display(), + ); + fs::write(&path, body).expect("write helper-cancel fake codex script"); + let mut perms = fs::metadata(&path) + .expect("helper-cancel fake codex metadata") + .permissions(); + perms.set_mode(0o755); + fs::set_permissions(&path, perms).expect("set helper-cancel fake codex permissions"); + path +} + +fn write_fake_codex_script_clean_bootstrap_then_prompt_resume(dir: &Path) -> PathBuf { + let path = dir.join("fake-codex-clean-bootstrap-then-prompt-resume.sh"); + let count_path = dir.join("fake-codex-clean-bootstrap-then-prompt-resume.count"); + let start_ready_path = dir.join("fake-codex-clean-bootstrap-start.ready"); + let turn_ready_path = dir.join("fake-codex-clean-bootstrap-turn.ready"); + let body = format!( + "#!/bin/sh\nSTATE_FILE='{}'\nSTART_READY='{}'\nTURN_READY='{}'\ncount=0\nif [ -f \"$STATE_FILE\" ]; then\n count=$(cat \"$STATE_FILE\")\nfi\ncount=$((count + 1))\nprintf '%s' \"$count\" > \"$STATE_FILE\"\nif [ \"$count\" -eq 1 ]; then\n printf '{{\"type\":\"thread.started\",\"thread_id\":\"thread-test\"}}\\r\\n'\n printf '{{\"type\":\"turn.started\",\"thread_id\":\"thread-test\",\"turn_id\":\"turn-1\"}}\\r\\n'\n exit 0\nfi\nif [ \"$count\" -eq 2 ]; then\n trap 'exit 0' INT TERM\n rm -f \"$START_READY\"\n printf '{{\"type\":\"thread.started\",\"thread_id\":\"thread-test\"}}\\r\\n'\n printf '{{\"type\":\"turn.started\",\"thread_id\":\"thread-test\",\"turn_id\":\"turn-2\"}}\\r\\n'\n while [ ! -f \"$START_READY\" ]; do sleep 0.1; done\n exit 0\nfi\nif [ \"$count\" -eq 3 ]; then\n : > \"$START_READY\"\n printf 'start prompt success\\n'\n exit 0\nfi\nif [ \"$count\" -eq 4 ]; then\n trap 'exit 0' INT TERM\n rm -f \"$TURN_READY\"\n printf '{{\"type\":\"thread.started\",\"thread_id\":\"thread-test\"}}\\r\\n'\n printf '{{\"type\":\"turn.started\",\"thread_id\":\"thread-test\",\"turn_id\":\"turn-4\"}}\\r\\n'\n while [ ! -f \"$TURN_READY\" ]; do sleep 0.1; done\n exit 0\nfi\nif [ \"$count\" -eq 5 ]; then\n : > \"$TURN_READY\"\n printf 'turn prompt success\\n'\n exit 0\nfi\ntrap 'exit 0' INT TERM\nprintf '{{\"type\":\"thread.started\",\"thread_id\":\"thread-test\"}}\\r\\n'\nprintf '{{\"type\":\"turn.started\",\"thread_id\":\"thread-test\",\"turn_id\":\"turn-%s\"}}\\r\\n' \"$count\"\nwhile :; do sleep 1; done\n", + count_path.display(), + start_ready_path.display(), + turn_ready_path.display() + ); + fs::write(&path, body).expect("write clean bootstrap fake codex script"); + let mut perms = fs::metadata(&path) + .expect("clean bootstrap fake codex metadata") + .permissions(); + perms.set_mode(0o755); + fs::set_permissions(&path, perms).expect("set clean bootstrap fake codex permissions"); + path +} + +fn write_fake_codex_script_without_session_handle(dir: &Path) -> PathBuf { + let path = dir.join("fake-codex-no-session-handle.sh"); + let body = "#!/bin/sh\nprintf 'bootstrap-without-session-handle\\n'\n"; + fs::write(&path, body).expect("write fake codex script without session handle"); + let mut perms = fs::metadata(&path) + .expect("fake codex metadata") + .permissions(); + perms.set_mode(0o755); + fs::set_permissions(&path, perms).expect("set fake codex permissions"); + path +} + +fn write_fake_codex_script_with_delayed_prompt_completion(dir: &Path) -> PathBuf { + let path = dir.join("fake-codex-delayed-prompt-completion.sh"); + let count_path = dir.join("fake-codex-delayed-prompt-completion.count"); + let body = format!( + "#!/bin/sh\nSTATE_FILE='{}'\ncount=0\nif [ -f \"$STATE_FILE\" ]; then\n count=$(cat \"$STATE_FILE\")\nfi\ncount=$((count + 1))\nprintf '%s' \"$count\" > \"$STATE_FILE\"\nif [ \"$count\" -eq 1 ]; then\n trap 'exit 0' INT TERM\n printf '{{\"type\":\"thread.started\",\"thread_id\":\"thread-test\"}}\\r\\n'\n printf '{{\"type\":\"turn.started\",\"thread_id\":\"thread-test\",\"turn_id\":\"turn-1\"}}\\r\\n'\n while :; do sleep 1; done\nfi\nsleep 5\nexit 0\n", + count_path.display() + ); + fs::write(&path, body).expect("write delayed prompt fake codex script"); + let mut perms = fs::metadata(&path) + .expect("delayed prompt fake codex metadata") + .permissions(); + perms.set_mode(0o755); + fs::set_permissions(&path, perms).expect("set delayed prompt fake codex permissions"); + path +} + +fn write_fake_codex_script_with_slow_startup_prompt_completion(dir: &Path) -> PathBuf { + let path = dir.join("fake-codex-slow-startup-prompt-completion.sh"); + let count_path = dir.join("fake-codex-slow-startup-prompt-completion.count"); + let body = format!( + "#!/bin/sh\nSTATE_FILE='{}'\nSCRIPT_DIR='{}'\ncount=0\nif [ -f \"$STATE_FILE\" ]; then\n count=$(cat \"$STATE_FILE\")\nfi\ncount=$((count + 1))\nprintf '%s' \"$count\" > \"$STATE_FILE\"\nprintf '%s\\n' \"$@\" > \"$SCRIPT_DIR/fake-codex-$count.args\"\ncat > \"$SCRIPT_DIR/fake-codex-$count.stdin\"\nif [ \"$count\" -eq 1 ]; then\n printf '{{\"type\":\"thread.started\",\"thread_id\":\"thread-test\"}}\\r\\n'\n printf '{{\"type\":\"turn.started\",\"thread_id\":\"thread-test\",\"turn_id\":\"turn-1\"}}\\r\\n'\n sleep 11\n printf '{{\"type\":\"item.completed\",\"thread_id\":\"thread-test\",\"turn_id\":\"turn-1\",\"item_id\":\"msg-1\",\"status\":\"completed\",\"item_type\":\"agent_message\",\"content\":{{\"text\":\"slow startup prompt success\"}}}}\\r\\n'\n printf '{{\"type\":\"turn.completed\",\"thread_id\":\"thread-test\",\"turn_id\":\"turn-1\"}}\\r\\n'\n exit 0\nfi\nprintf '{{\"type\":\"thread.resumed\",\"thread_id\":\"thread-test\"}}\\r\\n'\nprintf '{{\"type\":\"turn.started\",\"thread_id\":\"thread-test\",\"turn_id\":\"turn-%s\"}}\\r\\n' \"$count\"\nprintf '{{\"type\":\"item.completed\",\"thread_id\":\"thread-test\",\"turn_id\":\"turn-%s\",\"item_id\":\"msg-%s\",\"status\":\"completed\",\"item_type\":\"agent_message\",\"content\":{{\"text\":\"follow-up prompt success\"}}}}\\r\\n' \"$count\" \"$count\"\nprintf '{{\"type\":\"turn.completed\",\"thread_id\":\"thread-test\",\"turn_id\":\"turn-%s\"}}\\r\\n' \"$count\"\nexit 0\n", + count_path.display(), + dir.display() + ); + fs::write(&path, body).expect("write slow-startup fake codex script"); + let mut perms = fs::metadata(&path) + .expect("slow-startup fake codex metadata") + .permissions(); + perms.set_mode(0o755); + fs::set_permissions(&path, perms).expect("set slow-startup fake codex permissions"); + path +} + +fn parse_json_output(output: &Output) -> Value { + serde_json::from_slice(&output.stdout).expect("stdout should be valid JSON") +} + +fn parse_ndjson_output(output: &Output) -> Vec { + String::from_utf8_lossy(&output.stdout) + .lines() + .filter(|line| !line.trim().is_empty()) + .map(|line| serde_json::from_str(line).expect("stdout should be valid NDJSON")) + .collect() +} + +fn find_ndjson_record<'a>(records: &'a [Value], kind: &str) -> &'a Value { + records + .iter() + .find(|record| record.get("kind").and_then(Value::as_str) == Some(kind)) + .unwrap_or_else(|| panic!("missing NDJSON record kind={kind}: {records:?}")) +} + +fn status_sessions(json: &Value) -> &[Value] { + json["sessions"] + .as_array() + .map(Vec::as_slice) + .expect("sessions should be an array") +} + +fn find_status_session_by_orchestration_session_id<'a>( + sessions: &'a [Value], + orchestration_session_id: &str, +) -> &'a Value { + sessions + .iter() + .find(|session| { + session + .get("orchestration_session_id") + .and_then(Value::as_str) + == Some(orchestration_session_id) + }) + .unwrap_or_else(|| { + panic!( + "missing status row orchestration_session_id={orchestration_session_id}: {sessions:?}" + ) + }) +} + +fn stderr_text(output: &Output) -> String { + String::from_utf8_lossy(&output.stderr).to_string() +} + +fn assert_empty_warnings(json: &Value) { + assert_eq!( + json.get("warnings").and_then(Value::as_array).map(Vec::len), + Some(0), + "control success output must keep warnings empty: {json}" + ); +} + +fn canonical_orchestration_session_path( + substrate_home: &Path, + orchestration_session_id: &str, +) -> PathBuf { + substrate_home + .join("run/agent-hub/sessions") + .join(orchestration_session_id) + .join("session.json") +} + +fn canonical_participant_manifest_path( + substrate_home: &Path, + orchestration_session_id: &str, + participant_id: &str, +) -> PathBuf { + substrate_home + .join("run/agent-hub/sessions") + .join(orchestration_session_id) + .join("participants") + .join(format!("{participant_id}.json")) +} + +#[cfg(target_os = "linux")] +fn canonical_participants_dir(substrate_home: &Path, orchestration_session_id: &str) -> PathBuf { + substrate_home + .join("run/agent-hub/sessions") + .join(orchestration_session_id) + .join("participants") +} + +fn read_json_file(path: &Path) -> Value { + serde_json::from_str( + &fs::read_to_string(path) + .unwrap_or_else(|err| panic!("failed to read {}: {err}", path.display())), + ) + .unwrap_or_else(|err| panic!("failed to parse {}: {err}", path.display())) +} + +fn compact_stop_transport_fragment(id: &str) -> String { + let normalized = id + .chars() + .filter(|ch| ch.is_ascii_alphanumeric()) + .collect::(); + if normalized.len() <= 12 { + return normalized; + } + format!( + "{}{}", + &normalized[..6], + &normalized[normalized.len() - 6..] + ) +} + +fn stop_transport_path( + fixture: &AgentControlFixture, + orchestration_session_id: &str, + participant_id: &str, +) -> PathBuf { + let socket_name = format!( + "{}-{}.sock", + compact_stop_transport_fragment(orchestration_session_id), + compact_stop_transport_fragment(participant_id) + ); + let preferred = fixture + .substrate_home + .join("run/agent-hub/handles/stop") + .join(&socket_name); + if preferred.as_os_str().len() > PRIVATE_STOP_UNIX_PATH_MAX { + return PathBuf::from("/tmp") + .join("substrate-agent-hub-stop") + .join(socket_name); + } + preferred +} + +fn wait_for_path(path: &Path, timeout: Duration) -> bool { + let start = Instant::now(); + while start.elapsed() < timeout { + if path.exists() { + return true; + } + std::thread::sleep(Duration::from_millis(25)); + } + false +} + +#[cfg(unix)] +fn pid_is_alive(pid: u32) -> bool { + let rc = unsafe { libc::kill(pid as libc::pid_t, 0) }; + rc == 0 + || matches!( + std::io::Error::last_os_error().raw_os_error(), + Some(libc::EPERM) + ) +} + +#[cfg(unix)] +fn terminate_pid(pid: u32) { + let pid = pid as libc::pid_t; + unsafe { + let _ = libc::kill(pid, libc::SIGTERM); + } + let start = Instant::now(); + while start.elapsed() < Duration::from_secs(2) { + if !pid_is_alive(pid as u32) { + return; + } + std::thread::sleep(Duration::from_millis(25)); + } + unsafe { + let _ = libc::kill(pid, libc::SIGKILL); + } +} + +fn wait_for_pid_exit(pid: u32, timeout: Duration) -> bool { + let start = Instant::now(); + while start.elapsed() < timeout { + if !pid_is_alive(pid) { + return true; + } + std::thread::sleep(Duration::from_millis(25)); + } + false +} + +fn wait_for_single_active_session( + fixture: &AgentControlFixture, + timeout: Duration, +) -> (String, String) { + let start = Instant::now(); + while start.elapsed() < timeout { + let sessions_dir = fixture.substrate_home.join("run/agent-hub/sessions"); + if let Ok(entries) = fs::read_dir(&sessions_dir) { + let mut dirs = entries + .filter_map(Result::ok) + .map(|entry| entry.path()) + .filter(|path| path.is_dir()) + .collect::>(); + dirs.sort(); + if let Some(dir) = dirs.into_iter().next() { + let session = read_json_file(&dir.join("session.json")); + if session.get("state").and_then(Value::as_str) == Some("active") { + if let Some(participant_id) = session + .get("active_session_handle_id") + .and_then(Value::as_str) + .map(str::to_string) + { + if read_json_file( + &dir.join("participants") + .join(format!("{participant_id}.json")), + ) + .get("state") + .and_then(Value::as_str) + .is_some_and(|state| matches!(state, "ready" | "running")) + { + return ( + session["orchestration_session_id"] + .as_str() + .expect("session id") + .to_string(), + participant_id, + ); + } + } + } + } + } + std::thread::sleep(Duration::from_millis(25)); + } + panic!("timed out waiting for a single active orchestration session"); +} + +fn wait_for_session_posture( + fixture: &AgentControlFixture, + orchestration_session_id: &str, + expected_posture: &str, + timeout: Duration, +) -> Value { + let start = Instant::now(); + while start.elapsed() < timeout { + let session = fixture.load_orchestration_session(orchestration_session_id); + if session.get("posture").and_then(Value::as_str) == Some(expected_posture) { + return session; + } + std::thread::sleep(Duration::from_millis(25)); + } + panic!( + "timed out waiting for session {} posture {}", + orchestration_session_id, expected_posture + ); +} + +fn wait_for_detached_resume_eligible_participant( + fixture: &AgentControlFixture, + orchestration_session_id: &str, + participant_id: &str, + timeout: Duration, +) -> Value { + let start = Instant::now(); + while start.elapsed() < timeout { + let participant = fixture.load_participant(orchestration_session_id, participant_id); + let resume_eligible = participant + .pointer("/internal/resume_eligible") + .and_then(Value::as_bool); + let attached_client_present = participant + .pointer("/internal/attached_client_present") + .and_then(Value::as_bool); + if resume_eligible == Some(true) && attached_client_present == Some(false) { + return participant; + } + std::thread::sleep(Duration::from_millis(25)); + } + panic!( + "timed out waiting for participant {} in session {} to become detached and resume-eligible", + participant_id, orchestration_session_id + ); +} + +fn wait_for_successor_owner_pid( + fixture: &AgentControlFixture, + orchestration_session_id: &str, + previous_participant_id: &str, + timeout: Duration, +) -> u32 { + let start = Instant::now(); + while start.elapsed() < timeout { + let session = fixture.load_orchestration_session(orchestration_session_id); + let participant_changed = session + .get("active_session_handle_id") + .and_then(Value::as_str) + .is_some_and(|participant_id| participant_id != previous_participant_id); + let shell_owner_pid = session + .get("shell_owner_pid") + .and_then(Value::as_u64) + .map(|pid| pid as u32); + if participant_changed && shell_owner_pid.is_some_and(pid_is_alive) { + return shell_owner_pid.expect("owner pid"); + } + std::thread::sleep(Duration::from_millis(25)); + } + panic!( + "timed out waiting for successor owner pid in orchestration session {}", + orchestration_session_id + ); +} + +#[cfg(target_os = "linux")] +fn session_participant_manifests( + substrate_home: &Path, + orchestration_session_id: &str, +) -> Vec { + let participants_dir = canonical_participants_dir(substrate_home, orchestration_session_id); + let Ok(entries) = fs::read_dir(participants_dir) else { + return Vec::new(); + }; + let mut manifests = entries + .filter_map(Result::ok) + .map(|entry| read_json_file(&entry.path())) + .collect::>(); + manifests.sort_by(|left, right| { + left.get("participant_id") + .and_then(Value::as_str) + .cmp(&right.get("participant_id").and_then(Value::as_str)) + }); + manifests +} + +#[cfg(target_os = "linux")] +fn participant_is_authoritative_live(manifest: &Value) -> bool { + let Some(state) = manifest.get("state").and_then(Value::as_str) else { + return false; + }; + let live_state = matches!( + state, + "allocating" | "ready" | "running" | "restarting" | "stopping" + ); + live_state + && manifest + .pointer("/internal/uaa_session_id") + .and_then(Value::as_str) + .is_some() + && manifest + .pointer("/internal/control_owner_retained") + .and_then(Value::as_bool) + == Some(true) + && manifest + .pointer("/internal/event_stream_active") + .and_then(Value::as_bool) + == Some(true) + && manifest + .pointer("/internal/completion_observer_retained") + .and_then(Value::as_bool) + == Some(true) + && manifest + .pointer("/internal/ownership_valid") + .and_then(Value::as_bool) + == Some(true) + && manifest + .pointer("/internal/terminal_observed_at") + .is_none_or(Value::is_null) +} + +#[cfg(target_os = "linux")] +fn authoritative_live_world_member_manifests_for_session( + substrate_home: &Path, + orchestration_session_id: &str, +) -> Vec { + session_participant_manifests(substrate_home, orchestration_session_id) + .into_iter() + .filter(|manifest| manifest.get("role").and_then(Value::as_str) == Some("member")) + .filter(|manifest| { + manifest.pointer("/execution/scope").and_then(Value::as_str) == Some("world") + }) + .filter(participant_is_authoritative_live) + .collect() +} + +#[cfg(target_os = "linux")] +fn wait_for_live_world_member_count( + fixture: &AgentControlFixture, + orchestration_session_id: &str, + expected_count: usize, + timeout: Duration, +) -> Vec { + let deadline = Instant::now() + timeout; + while Instant::now() < deadline { + let members = authoritative_live_world_member_manifests_for_session( + &fixture.substrate_home, + orchestration_session_id, + ); + if members.len() == expected_count { + return members; + } + std::thread::sleep(Duration::from_millis(25)); + } + + panic!( + "timed out waiting for authoritative live world member count == {expected_count}; got {:?}", + authoritative_live_world_member_manifests_for_session( + &fixture.substrate_home, + orchestration_session_id + ), + ); +} + +fn write_json_file(path: &Path, value: &Value) { + let parent = path.parent().expect("fixture json path should have parent"); + fs::create_dir_all(parent) + .unwrap_or_else(|err| panic!("failed to create {}: {err}", parent.display())); + fs::write( + path, + serde_json::to_vec_pretty(value).expect("serialize fixture json"), + ) + .unwrap_or_else(|err| panic!("failed to write {}: {err}", path.display())); +} + +fn write_active_orchestration_session( + fixture: &AgentControlFixture, + agent_id: &str, + orchestration_session_id: &str, + active_session_handle_id: &str, + ts: &str, +) { + write_json_file( + &canonical_orchestration_session_path(&fixture.substrate_home, orchestration_session_id), + &json!({ + "orchestration_session_id": orchestration_session_id, + "shell_trace_session_id": "ses_agent_control", + "workspace_root": fixture.workspace_root.display().to_string(), + "shell_owner_pid": std::process::id(), + "state": "active", + "posture": "active_attached", + "posture_changed_at": ts, + "opened_at": ts, + "last_active_at": ts, + "orchestrator_agent_id": agent_id, + "orchestrator_backend_id": format!("cli:{agent_id}"), + "orchestrator_protocol": PURE_AGENT_PROTOCOL, + "active_session_handle_id": active_session_handle_id, + "attached_participant_id": active_session_handle_id, + "pending_inbox_count": 0, + "last_parked_at": Value::Null, + "last_attention_at": Value::Null, + "parked_reason": Value::Null, + "latest_run_id": "0195f8f1-7a35-7b7f-9c4d-9a7c2f5d6fab", + "world_id": Value::Null, + "world_generation": Value::Null, + "invalidation_reason": Value::Null, + "closed_at": Value::Null + }), + ); +} + +fn write_parked_orchestration_session( + fixture: &AgentControlFixture, + agent_id: &str, + orchestration_session_id: &str, + active_session_handle_id: &str, + ts: &str, +) { + write_json_file( + &canonical_orchestration_session_path(&fixture.substrate_home, orchestration_session_id), + &json!({ + "orchestration_session_id": orchestration_session_id, + "shell_trace_session_id": "ses_agent_control", + "workspace_root": fixture.workspace_root.display().to_string(), + "shell_owner_pid": std::process::id(), + "state": "active", + "posture": "parked_resumable", + "posture_changed_at": ts, + "opened_at": ts, + "last_active_at": ts, + "orchestrator_agent_id": agent_id, + "orchestrator_backend_id": format!("cli:{agent_id}"), + "orchestrator_protocol": PURE_AGENT_PROTOCOL, + "active_session_handle_id": active_session_handle_id, + "attached_participant_id": Value::Null, + "pending_inbox_count": 0, + "last_parked_at": ts, + "last_attention_at": Value::Null, + "parked_reason": "owner detached cleanly", + "latest_run_id": "0195f8f1-7a35-7b7f-9c4d-9a7c2f5d6fab", + "world_id": Value::Null, + "world_generation": Value::Null, + "invalidation_reason": Value::Null, + "closed_at": Value::Null + }), + ); +} + +#[cfg(target_os = "linux")] +fn write_parked_world_orchestration_session( + fixture: &AgentControlFixture, + agent_id: &str, + orchestration_session_id: &str, + active_session_handle_id: &str, + world_id: &str, + world_generation: u64, + ts: &str, +) { + write_json_file( + &canonical_orchestration_session_path(&fixture.substrate_home, orchestration_session_id), + &json!({ + "orchestration_session_id": orchestration_session_id, + "shell_trace_session_id": "ses_agent_control", + "workspace_root": fixture.workspace_root.display().to_string(), + "shell_owner_pid": std::process::id(), + "state": "active", + "posture": "parked_resumable", + "posture_changed_at": ts, + "opened_at": ts, + "last_active_at": ts, + "orchestrator_agent_id": agent_id, + "orchestrator_backend_id": format!("cli:{agent_id}"), + "orchestrator_protocol": PURE_AGENT_PROTOCOL, + "active_session_handle_id": active_session_handle_id, + "attached_participant_id": Value::Null, + "pending_inbox_count": 0, + "last_parked_at": ts, + "last_attention_at": Value::Null, + "parked_reason": "owner detached cleanly", + "latest_run_id": "0195f8f1-7a35-7b7f-9c4d-9a7c2f5d6fab", + "world_id": world_id, + "world_generation": world_generation, + "invalidation_reason": Value::Null, + "closed_at": Value::Null + }), + ); +} + +#[allow(clippy::too_many_arguments)] +fn write_orchestration_session( + fixture: &AgentControlFixture, + agent_id: &str, + orchestration_session_id: &str, + active_session_handle_id: Option<&str>, + state: &str, + world_id: Option<&str>, + world_generation: Option, + ts: &str, +) { + let (posture, attached_participant_id, last_parked_at, parked_reason, closed_at) = match state { + "active" => ( + "active_attached", + active_session_handle_id + .map(Value::from) + .unwrap_or(Value::Null), + Value::Null, + Value::Null, + Value::Null, + ), + "stopped" | "failed" | "invalidated" => ( + "terminal", + Value::Null, + Value::Null, + Value::Null, + Value::String(ts.to_string()), + ), + _ => ( + "active_attached", + active_session_handle_id + .map(Value::from) + .unwrap_or(Value::Null), + Value::Null, + Value::Null, + Value::Null, + ), + }; + write_json_file( + &canonical_orchestration_session_path(&fixture.substrate_home, orchestration_session_id), + &json!({ + "orchestration_session_id": orchestration_session_id, + "shell_trace_session_id": "ses_agent_control", + "workspace_root": fixture.workspace_root.display().to_string(), + "shell_owner_pid": std::process::id(), + "state": state, + "posture": posture, + "posture_changed_at": ts, + "opened_at": ts, + "last_active_at": ts, + "orchestrator_agent_id": agent_id, + "orchestrator_backend_id": format!("cli:{agent_id}"), + "orchestrator_protocol": PURE_AGENT_PROTOCOL, + "active_session_handle_id": active_session_handle_id, + "attached_participant_id": attached_participant_id, + "pending_inbox_count": 0, + "last_parked_at": last_parked_at, + "last_attention_at": Value::Null, + "parked_reason": parked_reason, + "latest_run_id": "0195f8f1-7a35-7b7f-9c4d-9a7c2f5d6fab", + "world_id": world_id, + "world_generation": world_generation, + "invalidation_reason": Value::Null, + "closed_at": closed_at + }), + ); +} + +#[allow(clippy::too_many_arguments)] +fn write_runtime_participant( + fixture: &AgentControlFixture, + participant_id: &str, + agent_id: &str, + orchestration_session_id: &str, + state: &str, + ownership_valid: bool, + uaa_session_id: Option<&str>, + resumed_from_participant_id: Option<&str>, + ts: &str, +) { + let attached_client_present = ownership_valid; + let resume_eligible = uaa_session_id.is_some() && state != "invalidated"; + let last_attached_at = if resume_eligible { + Value::String(ts.to_string()) + } else { + Value::Null + }; + let last_detached_at = if !attached_client_present && resume_eligible { + Value::String(ts.to_string()) + } else { + Value::Null + }; + let detach_reason = if !attached_client_present && resume_eligible { + Value::String("owner detached cleanly".to_string()) + } else { + Value::Null + }; + write_json_file( + &canonical_participant_manifest_path( + &fixture.substrate_home, + orchestration_session_id, + participant_id, + ), + &json!({ + "participant_id": participant_id, + "orchestration_session_id": orchestration_session_id, + "agent_id": agent_id, + "backend_id": format!("cli:{agent_id}"), + "role": "orchestrator", + "protocol": PURE_AGENT_PROTOCOL, + "execution": { "scope": "host" }, + "state": state, + "opened_at": ts, + "last_transition_at": ts, + "resumed_from_participant_id": resumed_from_participant_id, + "internal": { + "resolved_agent_kind": agent_id, + "resolved_binary_path": fixture.fake_codex.display().to_string(), + "shell_owner_pid": std::process::id(), + "lease_token": format!("lease-{participant_id}"), + "uaa_session_id": uaa_session_id, + "latest_run_id": "0195f8f1-7a35-7b7f-9c4d-9a7c2f5d6fab", + "cancel_supported": true, + "control_owner_retained": ownership_valid, + "event_stream_active": ownership_valid, + "completion_observer_retained": ownership_valid, + "ownership_mode": "attached_control", + "ownership_valid": ownership_valid, + "ownership_verified_at": ts, + "last_heartbeat_at": ts, + "last_event_at": ts, + "attached_client_present": attached_client_present, + "last_attached_at": last_attached_at, + "last_detached_at": last_detached_at, + "detach_reason": detach_reason, + "resume_eligible": resume_eligible, + "terminal_observed_at": Value::Null, + "termination_reason": Value::Null, + "last_error_bucket": Value::Null, + "last_error_message": Value::Null + } + }), + ); +} + +#[cfg(target_os = "linux")] +#[allow(clippy::too_many_arguments)] +fn write_world_member_participant( + fixture: &AgentControlFixture, + participant_id: &str, + agent_id: &str, + orchestration_session_id: &str, + orchestrator_participant_id: &str, + world_id: &str, + world_generation: u64, + state: &str, + ownership_valid: bool, + uaa_session_id: Option<&str>, + ts: &str, +) { + write_json_file( + &canonical_participant_manifest_path( + &fixture.substrate_home, + orchestration_session_id, + participant_id, + ), + &json!({ + "participant_id": participant_id, + "orchestration_session_id": orchestration_session_id, + "agent_id": agent_id, + "backend_id": format!("cli:{agent_id}"), + "role": "member", + "protocol": PURE_AGENT_PROTOCOL, + "execution": { "scope": "world" }, + "state": state, + "opened_at": ts, + "last_transition_at": ts, + "parent_session_handle_id": Value::Null, + "resumed_from_session_handle_id": Value::Null, + "world_id": world_id, + "world_generation": world_generation, + "orchestrator_participant_id": orchestrator_participant_id, + "internal": { + "resolved_agent_kind": agent_id, + "resolved_binary_path": fixture.fake_codex.display().to_string(), + "shell_owner_pid": std::process::id(), + "lease_token": format!("lease-{participant_id}"), + "uaa_session_id": uaa_session_id, + "latest_run_id": "0195f8f1-7a35-7b7f-9c4d-9a7c2f5d6fab", + "cancel_supported": true, + "control_owner_retained": ownership_valid, + "event_stream_active": ownership_valid, + "completion_observer_retained": ownership_valid, + "ownership_mode": "member_runtime", + "ownership_valid": ownership_valid, + "ownership_verified_at": ts, + "last_heartbeat_at": ts, + "last_event_at": ts, + "terminal_observed_at": Value::Null, + "termination_reason": Value::Null, + "last_error_bucket": Value::Null, + "last_error_message": Value::Null + } + }), + ); +} + +struct PtyRepl { + child: Box, + master: Option>, + waited: Option, + writer: Arc>>, + output: Arc>>, + reader_handle: Option>, + stop_reader: Arc, +} + +impl Drop for PtyRepl { + fn drop(&mut self) { + self.stop_reader.store(true, Ordering::Relaxed); + self.master.take(); + if self.waited.is_none() { + let _ = self.child.kill(); + let _ = self.child.try_wait().ok().flatten().map(|status| { + self.waited = Some(status); + }); + } + if let Some(handle) = self.reader_handle.take() { + let _ = handle.join(); + } + } +} + +impl PtyRepl { + fn spawn(fixture: &AgentControlFixture) -> Self { + Self::spawn_with_options(fixture, None) + } + + #[cfg(target_os = "linux")] + fn spawn_with_world_socket(fixture: &AgentControlFixture, socket_path: &Path) -> Self { + Self::spawn_with_options(fixture, Some(socket_path)) + } + + fn spawn_with_options(fixture: &AgentControlFixture, socket_path: Option<&Path>) -> Self { + ensure_substrate_built(); + + let pty_system = native_pty_system(); + let pair = pty_system + .openpty(PtySize { + rows: 24, + cols: 80, + pixel_width: 0, + pixel_height: 0, + }) + .expect("openpty"); + let master = pair.master; + + #[cfg(unix)] + let master_fd = master.as_raw_fd(); + #[cfg(unix)] + if let Some(fd) = master_fd { + set_fd_nonblocking(fd); + } + + let mut cmd = CommandBuilder::new(binary_path()); + cmd.cwd(&fixture.workspace_root); + cmd.env("HOME", &fixture.home); + cmd.env("USERPROFILE", &fixture.home); + cmd.env("SUBSTRATE_HOME", &fixture.substrate_home); + cmd.env("SUBSTRATE_MANAGER_MANIFEST", manager_manifest_path()); + cmd.env("SHIM_TRACE_LOG", fixture.trace_path()); + cmd.env_remove("SHIM_ORIGINAL_PATH"); + cmd.env_remove("SUBSTRATE_WORLD"); + cmd.env_remove("SUBSTRATE_WORLD_ENABLED"); + cmd.env_remove("SUBSTRATE_WORLD_ID"); + match socket_path { + Some(socket_path) => { + cmd.env("SUBSTRATE_WORLD_SOCKET", socket_path); + cmd.env("SUBSTRATE_OVERRIDE_WORLD", "enabled"); + cmd.arg("--world"); + } + None => { + cmd.env("SUBSTRATE_OVERRIDE_WORLD", "disabled"); + } + } + cmd.env("SHELL", "/bin/bash"); + cmd.arg("--async-repl"); + cmd.arg("--shim-skip"); + + let child = pair.slave.spawn_command(cmd).expect("spawn substrate repl"); + let writer: Arc>> = + Arc::new(Mutex::new(master.take_writer().expect("take writer"))); + let output = Arc::new(Mutex::new(Vec::new())); + let stop_reader = Arc::new(AtomicBool::new(false)); + let output_for_thread = output.clone(); + let stop_for_thread = stop_reader.clone(); + let writer_for_thread = Arc::downgrade(&writer); + let reader_handle = std::thread::spawn(move || { + let mut buf = [0u8; 4096]; + loop { + if stop_for_thread.load(Ordering::Relaxed) { + break; + } + + #[cfg(unix)] + if let Some(fd) = master_fd { + let rc = unsafe { libc::read(fd, buf.as_mut_ptr().cast(), buf.len()) }; + if rc == 0 { + break; + } + if rc < 0 { + let err = std::io::Error::last_os_error(); + if err.kind() == std::io::ErrorKind::WouldBlock { + std::thread::sleep(Duration::from_millis(25)); + continue; + } + break; + } + + let n = rc as usize; + let chunk = &buf[..n]; + if chunk.windows(4).any(|window| window == b"\x1b[6n") { + if let Some(writer) = writer_for_thread.upgrade() { + if let Ok(mut handle) = writer.lock() { + let _ = handle.write_all(b"\x1b[1;1R"); + let _ = handle.flush(); + } + } + } + if chunk.windows(5).any(|window| window == b"\x1b[18t") { + if let Some(writer) = writer_for_thread.upgrade() { + if let Ok(mut handle) = writer.lock() { + let _ = handle.write_all(b"\x1b[8;24;80t"); + let _ = handle.flush(); + } + } + } + + if let Ok(mut guard) = output_for_thread.lock() { + guard.extend_from_slice(chunk); + } + continue; + } + + std::thread::sleep(Duration::from_millis(25)); + } + }); + + Self { + child, + master: Some(master), + waited: None, + writer, + output, + reader_handle: Some(reader_handle), + stop_reader, + } + } + + fn send_line(&mut self, line: &str) { + let mut writer = self.writer.lock().expect("pty writer"); + writer.write_all(line.as_bytes()).expect("write line"); + writer.write_all(b"\n").expect("write newline"); + writer.flush().expect("flush line"); + } + + fn wait_for_output(&self, needle: &str, timeout: Duration) -> Option { + let start = Instant::now(); + while start.elapsed() < timeout { + let output = self.output.lock().expect("output lock"); + if let Ok(text) = std::str::from_utf8(&output) { + if let Some(pos) = text.find(needle) { + return Some(pos); + } + } + drop(output); + std::thread::sleep(Duration::from_millis(10)); + } + None + } + + fn shutdown_graceful(mut self, timeout: Duration) -> (i32, Vec) { + let start = Instant::now(); + while start.elapsed() < timeout { + if let Ok(Some(status)) = self.child.try_wait() { + self.waited = Some(status); + break; + } + std::thread::sleep(Duration::from_millis(10)); + } + if self.waited.is_none() { + let _ = self.child.kill(); + if let Ok(Some(status)) = self.child.try_wait() { + self.waited = Some(status); + } + } + + self.stop_reader.store(true, Ordering::Relaxed); + self.master.take(); + if let Some(handle) = self.reader_handle.take() { + let _ = handle.join(); + } + + let code = self + .waited + .as_ref() + .map(|status| status.exit_code() as i32) + .unwrap_or(-1); + let output = self.output.lock().expect("output lock").clone(); + (code, output) + } +} + +#[test] +#[serial] +fn public_start_turn_and_stop_emit_streaming_ndjson_and_authoritative_state() { + let fixture = AgentControlFixture::new(); + fixture.init_workspace(); + fixture.write_runtime_inventory(false); + + let start_output = fixture.run(&[ + "agent", + "start", + "--backend", + "cli:codex", + "--prompt", + "hello from start", + "--json", + ]); + assert!( + start_output.status.success(), + "public start should succeed: {start_output:?}" + ); + let start_records = parse_ndjson_output(&start_output); + let start_accepted = find_ndjson_record(&start_records, "accepted"); + assert_eq!( + start_records + .first() + .and_then(|record| record.get("kind")) + .and_then(Value::as_str), + Some("accepted"), + "public start must stream acceptance before terminal completion: {start_records:?}" + ); + let start_json = find_ndjson_record(&start_records, "completed"); + assert_eq!( + start_json.get("action").and_then(Value::as_str), + Some("start") + ); + assert_eq!( + start_json.get("backend_id").and_then(Value::as_str), + Some("cli:codex") + ); + assert_eq!( + start_accepted.get("scope").and_then(Value::as_str), + Some("host") + ); + assert_eq!( + start_json.get("turn_outcome").and_then(Value::as_str), + Some("success") + ); + assert_eq!( + start_json.get("session_posture").and_then(Value::as_str), + Some("active") + ); + assert_eq!( + start_json.get("state").and_then(Value::as_str), + Some("active") + ); + assert!( + start_json.get("source_orchestration_session_id").is_none(), + "start must not surface source_orchestration_session_id: {start_json}" + ); + assert_empty_warnings(start_json); + + let orchestration_session_id = start_json["orchestration_session_id"] + .as_str() + .expect("start session id") + .to_string(); + let participant_id = start_json["participant_id"] + .as_str() + .expect("start participant id") + .to_string(); + let persisted_session = fixture.load_orchestration_session(&orchestration_session_id); + assert_eq!( + persisted_session + .get("active_session_handle_id") + .and_then(Value::as_str), + Some(participant_id.as_str()), + "public start must not report readiness before the state store points at the active participant" + ); + assert_eq!( + persisted_session + .pointer("/startup_prompt/state") + .and_then(Value::as_str), + Some("completed"), + "public start must persist startup prompt completion for replay gating" + ); + let owner_pid = persisted_session["shell_owner_pid"] + .as_u64() + .expect("shell owner pid") as u32; + assert!( + pid_is_alive(owner_pid), + "public start must leave the hidden owner-helper process alive" + ); + assert!( + wait_for_path( + &stop_transport_path(&fixture, &orchestration_session_id, &participant_id), + Duration::from_secs(5), + ), + "public start must materialize a per-session private stop transport" + ); + let persisted_participant = + fixture.load_participant(&orchestration_session_id, &participant_id); + assert_eq!( + persisted_participant + .pointer("/internal/uaa_session_id") + .and_then(Value::as_str), + Some("thread-test"), + "public start must persist the retained internal.uaa_session_id from the single startup exec" + ); + let start_args = fixture.read_fake_codex_args(1); + assert!( + start_args.iter().any(|arg| arg == "exec"), + "startup invocation must use codex exec: {start_args:?}" + ); + assert!( + start_args.iter().any(|arg| arg == "--json"), + "startup invocation must use codex exec --json: {start_args:?}" + ); + assert!( + !start_args.iter().any(|arg| arg == "resume"), + "start prompt must not use resume on the initial retained exec: {start_args:?}" + ); + let start_stdin = fixture.read_fake_codex_stdin(1); + assert!( + start_stdin.contains("hello from start"), + "the first visible start prompt must ride the startup exec stdin payload: {start_stdin:?}" + ); + assert!( + !start_stdin.contains("Enter persistent Substrate host orchestrator mode."), + "start prompt must not be wrapped in hidden host bootstrap instructions: {start_stdin:?}" + ); + assert!( + !start_stdin.contains("First visible operator request:"), + "start prompt must not be rewritten into a bootstrap+visible composed payload: {start_stdin:?}" + ); + assert!( + !fixture.fake_codex_args_path(2).exists(), + "agent start must cause exactly one Codex exec launch before any follow-up turn" + ); + + let turn_output = fixture.run(&[ + "agent", + "turn", + "--session", + &orchestration_session_id, + "--backend", + "cli:codex", + "--prompt", + "hello from turn", + "--json", + ]); + assert!( + turn_output.status.success(), + "public turn should succeed: {turn_output:?}" + ); + let turn_records = parse_ndjson_output(&turn_output); + let turn_accepted = find_ndjson_record(&turn_records, "accepted"); + assert_eq!( + turn_records + .first() + .and_then(|record| record.get("kind")) + .and_then(Value::as_str), + Some("accepted"), + "public turn must stream acceptance before terminal completion: {turn_records:?}" + ); + let turn_json = find_ndjson_record(&turn_records, "completed"); + assert_eq!( + turn_json.get("action").and_then(Value::as_str), + Some("turn") + ); + assert_eq!( + turn_json + .get("orchestration_session_id") + .and_then(Value::as_str), + Some(orchestration_session_id.as_str()) + ); + assert_eq!( + turn_json.get("backend_id").and_then(Value::as_str), + Some("cli:codex") + ); + assert_eq!( + turn_accepted.get("scope").and_then(Value::as_str), + Some("host") + ); + assert_eq!( + turn_json.get("turn_outcome").and_then(Value::as_str), + Some("success") + ); + assert_eq!( + turn_json.get("session_posture").and_then(Value::as_str), + Some("active") + ); + assert_eq!( + turn_json.get("state").and_then(Value::as_str), + Some("active") + ); + assert_empty_warnings(turn_json); + let turn_args = fixture.read_fake_codex_args(2); + assert!( + turn_args.iter().any(|arg| arg == "resume"), + "the first follow-up turn must be the first resume-backed invocation: {turn_args:?}" + ); + let turn_stdin = fixture.read_fake_codex_stdin(2); + assert!( + turn_stdin.contains("hello from turn"), + "resume-backed follow-up turns must continue to send the prompt on stdin: {turn_stdin:?}" + ); + + let stop_output = fixture.run(&[ + "agent", + "stop", + "--session", + &orchestration_session_id, + "--json", + ]); + assert!( + stop_output.status.success(), + "public stop should succeed: {stop_output:?}" + ); + let stop_json = parse_json_output(&stop_output); + assert_eq!( + stop_json.get("action").and_then(Value::as_str), + Some("stop") + ); + assert_eq!( + stop_json + .get("orchestration_session_id") + .and_then(Value::as_str), + Some(orchestration_session_id.as_str()) + ); + assert_eq!( + stop_json.get("participant_id").and_then(Value::as_str), + Some(participant_id.as_str()) + ); + assert_eq!( + stop_json.get("backend_id").and_then(Value::as_str), + Some("cli:codex") + ); + assert!( + matches!( + stop_json.get("state").and_then(Value::as_str), + Some("stopped") | Some("invalidated") + ), + "public stop must wait for a terminal parent state: {stop_json}" + ); + assert_empty_warnings(&stop_json); + + let final_session = fixture.load_orchestration_session(&orchestration_session_id); + assert_eq!( + final_session.get("state").and_then(Value::as_str), + Some("stopped"), + "host-scoped public stop should persist a stopped parent session on clean shutdown" + ); +} + +#[test] +#[serial] +fn public_reattach_and_fork_preserve_exact_session_and_lineage_contracts() { + let fixture = AgentControlFixture::new(); + fixture.init_workspace(); + fixture.write_runtime_inventory(false); + + let ts = "2026-05-05T00:00:00Z"; + write_parked_orchestration_session(&fixture, "codex", "sess_resume_source", "ash_source", ts); + write_runtime_participant( + &fixture, + "ash_source", + "codex", + "sess_resume_source", + "running", + false, + Some("uaa-detached-1"), + None, + ts, + ); + + let reattach_output = fixture.run(&[ + "agent", + "reattach", + "--session", + "sess_resume_source", + "--json", + ]); + assert!( + reattach_output.status.success(), + "public reattach should succeed for an orphaned authoritative session: {reattach_output:?}" + ); + let resume_json = parse_json_output(&reattach_output); + assert_eq!( + resume_json.get("action").and_then(Value::as_str), + Some("reattach") + ); + assert_eq!( + resume_json + .get("orchestration_session_id") + .and_then(Value::as_str), + Some("sess_resume_source") + ); + assert_eq!( + resume_json.get("backend_id").and_then(Value::as_str), + Some("cli:codex") + ); + assert_eq!( + resume_json.get("scope").and_then(Value::as_str), + Some("host") + ); + assert_eq!( + resume_json.get("state").and_then(Value::as_str), + Some("active") + ); + assert!( + resume_json.get("source_orchestration_session_id").is_none(), + "reattach must stay inside the source orchestration session: {resume_json}" + ); + assert_empty_warnings(&resume_json); + + let resumed_participant_id = resume_json["participant_id"] + .as_str() + .expect("resume participant id") + .to_string(); + assert_ne!( + resumed_participant_id, "ash_source", + "reattach must allocate a successor participant" + ); + let resumed_participant = + fixture.load_participant("sess_resume_source", &resumed_participant_id); + assert_eq!( + resumed_participant + .get("resumed_from_participant_id") + .and_then(Value::as_str), + Some("ash_source"), + "reattach successor persistence must retain resumed_from_participant_id lineage" + ); + let resumed_owner_pid = fixture.load_orchestration_session("sess_resume_source") + ["shell_owner_pid"] + .as_u64() + .expect("resume owner pid") as u32; + assert!( + pid_is_alive(resumed_owner_pid), + "reattach must leave a live owner loop" + ); + + fixture.reset_fake_codex_state(); + let fork_output = fixture.run(&["agent", "fork", "--session", "sess_resume_source", "--json"]); + assert!( + fork_output.status.success(), + "public fork should succeed from the active resumed session: {fork_output:?}" + ); + let fork_json = parse_json_output(&fork_output); + assert_eq!( + fork_json.get("action").and_then(Value::as_str), + Some("fork") + ); + assert_eq!( + fork_json + .get("source_orchestration_session_id") + .and_then(Value::as_str), + Some("sess_resume_source") + ); + assert_eq!( + fork_json.get("backend_id").and_then(Value::as_str), + Some("cli:codex") + ); + assert_eq!(fork_json.get("scope").and_then(Value::as_str), Some("host")); + assert_eq!( + fork_json.get("state").and_then(Value::as_str), + Some("active") + ); + assert_empty_warnings(&fork_json); + + let fork_session_id = fork_json["orchestration_session_id"] + .as_str() + .expect("fork session id") + .to_string(); + assert_ne!( + fork_session_id, "sess_resume_source", + "fork must allocate a new orchestration session id" + ); + let fork_participant_id = fork_json["participant_id"] + .as_str() + .expect("fork participant id") + .to_string(); + let fork_participant = fixture.load_participant(&fork_session_id, &fork_participant_id); + assert_eq!( + fork_participant + .get("resumed_from_participant_id") + .and_then(Value::as_str), + Some(resumed_participant_id.as_str()), + "fork successor persistence must point lineage at the exact live source participant" + ); + let fork_owner_pid = fixture.load_orchestration_session(&fork_session_id)["shell_owner_pid"] + .as_u64() + .expect("fork owner pid") as u32; + assert!( + pid_is_alive(fork_owner_pid), + "fork must leave a live owner loop" + ); + + terminate_pid(resumed_owner_pid); + terminate_pid(fork_owner_pid); +} + +#[test] +#[serial] +fn public_stop_cleanly_closes_same_durable_session_after_reattach() { + let fixture = AgentControlFixture::new(); + fixture.init_workspace(); + fixture.write_runtime_inventory(false); + + let ts = "2026-05-05T00:00:00Z"; + write_parked_orchestration_session(&fixture, "codex", "sess_resume_stop", "ash_source", ts); + write_runtime_participant( + &fixture, + "ash_source", + "codex", + "sess_resume_stop", + "running", + false, + Some("uaa-detached-stop-1"), + None, + ts, + ); + + let reattach_output = fixture.run(&[ + "agent", + "reattach", + "--session", + "sess_resume_stop", + "--json", + ]); + assert!( + reattach_output.status.success(), + "public reattach should succeed for a parked durable session: {reattach_output:?}" + ); + let reattach_json = parse_json_output(&reattach_output); + let resumed_participant_id = reattach_json["participant_id"] + .as_str() + .expect("reattach participant id") + .to_string(); + let resumed_owner_pid = wait_for_successor_owner_pid( + &fixture, + "sess_resume_stop", + "ash_source", + Duration::from_secs(5), + ); + let reattached_session = wait_for_session_posture( + &fixture, + "sess_resume_stop", + "active_attached", + Duration::from_secs(5), + ); + assert_eq!( + reattached_session + .get("attached_participant_id") + .and_then(Value::as_str), + Some(resumed_participant_id.as_str()) + ); + assert!(pid_is_alive(resumed_owner_pid)); + assert!( + wait_for_path( + &stop_transport_path(&fixture, "sess_resume_stop", &resumed_participant_id), + Duration::from_secs(5), + ), + "reattach must materialize the per-session private stop transport before stop is exercised" + ); + + let stop_output = fixture.run(&["agent", "stop", "--session", "sess_resume_stop", "--json"]); + assert!( + stop_output.status.success(), + "public stop should close the same durable session after reattach: {stop_output:?}" + ); + let stop_json = parse_json_output(&stop_output); + assert_eq!( + stop_json.get("action").and_then(Value::as_str), + Some("stop") + ); + assert_eq!( + stop_json + .get("orchestration_session_id") + .and_then(Value::as_str), + Some("sess_resume_stop") + ); + assert_eq!( + stop_json.get("state").and_then(Value::as_str), + Some("stopped") + ); + + let stopped_session = fixture.load_orchestration_session("sess_resume_stop"); + assert_eq!( + stopped_session.get("state").and_then(Value::as_str), + Some("stopped") + ); + assert_eq!( + stopped_session.get("posture").and_then(Value::as_str), + Some("terminal") + ); + assert!(stopped_session + .get("attached_participant_id") + .is_none_or(Value::is_null)); + assert!(stopped_session + .get("closed_at") + .and_then(Value::as_str) + .is_some()); + assert!( + wait_for_pid_exit(resumed_owner_pid, Duration::from_secs(5)), + "reattach stop should leave no live hidden owner-helper" + ); + + let repeated_stop_output = + fixture.run(&["agent", "stop", "--session", "sess_resume_stop", "--json"]); + assert_eq!( + repeated_stop_output.status.code(), + Some(2), + "repeated stop must no longer see a routable active parent after terminal convergence: {repeated_stop_output:?}" + ); + assert!( + stderr_text(&repeated_stop_output).contains("missing_active_parent"), + "repeated stop must fail closed as non-active after terminal convergence: {repeated_stop_output:?}" + ); +} + +#[test] +#[serial] +fn public_turn_resumes_parked_host_session_and_preserves_exact_session_selector_contracts() { + let fixture = AgentControlFixture::new(); + fixture.init_workspace(); + fixture.write_runtime_inventory(false); + + let ts = "2026-05-05T00:00:00Z"; + write_parked_orchestration_session(&fixture, "codex", "sess_turn_parked", "ash_parked", ts); + write_runtime_participant( + &fixture, + "ash_parked", + "codex", + "sess_turn_parked", + "running", + false, + Some("uaa-detached-turn"), + None, + ts, + ); + + let turn_output = fixture.run(&[ + "agent", + "turn", + "--session", + "sess_turn_parked", + "--backend", + "cli:codex", + "--prompt", + "resume parked host turn", + "--json", + ]); + assert!( + turn_output.status.success(), + "public turn should resume a parked authoritative session: {turn_output:?}" + ); + let turn_records = parse_ndjson_output(&turn_output); + let turn_accepted = find_ndjson_record(&turn_records, "accepted"); + let turn_json = find_ndjson_record(&turn_records, "completed"); + assert_eq!( + turn_records + .first() + .and_then(|record| record.get("kind")) + .and_then(Value::as_str), + Some("accepted"), + "parked host turn must still emit acceptance before terminal completion: {turn_records:?}" + ); + assert_eq!( + turn_accepted.get("scope").and_then(Value::as_str), + Some("host") + ); + assert_eq!( + turn_json.get("action").and_then(Value::as_str), + Some("turn") + ); + assert_eq!( + turn_json + .get("orchestration_session_id") + .and_then(Value::as_str), + Some("sess_turn_parked") + ); + assert_eq!( + turn_json.get("backend_id").and_then(Value::as_str), + Some("cli:codex") + ); + assert_eq!( + turn_json.get("session_posture").and_then(Value::as_str), + Some("active") + ); + assert_eq!( + turn_json.get("state").and_then(Value::as_str), + Some("active") + ); + assert!( + turn_json.get("source_orchestration_session_id").is_none(), + "parked host turn must stay inside the selected orchestration session: {turn_json}" + ); + assert_empty_warnings(turn_json); + + let resumed_participant_id = turn_json["participant_id"] + .as_str() + .expect("resumed participant id") + .to_string(); + let resumed_participant = fixture.load_participant("sess_turn_parked", &resumed_participant_id); + assert_eq!( + resumed_participant + .get("resumed_from_participant_id") + .and_then(Value::as_str), + Some("ash_parked") + ); +} + +#[test] +#[serial] +fn public_same_session_parked_status_turn_reattach_and_stop_stay_on_one_orchestration_session_id() { + let fixture = AgentControlFixture::new_with_fake_codex( + write_fake_codex_script_turn_requires_helper_cancel_after_prompt, + ); + fixture.init_workspace(); + fixture.write_runtime_inventory(false); + + let ts = "2026-05-05T00:00:00Z"; + let orchestration_session_id = "sess_turn_reattach_stop"; + write_parked_orchestration_session( + &fixture, + "codex", + orchestration_session_id, + "ash_turn_source", + ts, + ); + write_runtime_participant( + &fixture, + "ash_turn_source", + "codex", + orchestration_session_id, + "running", + false, + Some("uaa-turn-reattach-1"), + None, + ts, + ); + + let parked_status_output = fixture.run(&["agent", "status", "--json"]); + assert!( + parked_status_output.status.success(), + "parked status must succeed before same-session follow-up control flow: {parked_status_output:?}" + ); + let parked_status_json = parse_json_output(&parked_status_output); + let parked_status_row = find_status_session_by_orchestration_session_id( + status_sessions(&parked_status_json), + orchestration_session_id, + ); + assert_eq!( + parked_status_row.get("source_kind").and_then(Value::as_str), + Some("live_runtime") + ); + assert_eq!( + parked_status_row.get("posture").and_then(Value::as_str), + Some("parked_resumable") + ); + assert!( + parked_status_row + .get("attached_participant_id") + .is_some_and(Value::is_null), + "parked status rows must preserve detached ownership as explicit null: {parked_status_row}" + ); + assert_eq!( + parked_status_row + .get("pending_inbox_count") + .and_then(Value::as_u64), + Some(0) + ); + + let turn_output = fixture.run(&[ + "agent", + "turn", + "--session", + orchestration_session_id, + "--backend", + "cli:codex", + "--prompt", + "resume parked host turn and exit", + "--json", + ]); + assert!( + turn_output.status.success(), + "parked turn should succeed before reattach on the same durable session: {turn_output:?}" + ); + let turn_records = parse_ndjson_output(&turn_output); + let turn_accepted = find_ndjson_record(&turn_records, "accepted"); + let turn_json = find_ndjson_record(&turn_records, "completed"); + assert_eq!( + turn_records + .first() + .and_then(|record| record.get("kind")) + .and_then(Value::as_str), + Some("accepted"), + "parked same-session turn must stream acceptance before completion: {turn_records:?}" + ); + assert_eq!( + turn_records + .iter() + .filter(|record| { + matches!( + record.get("kind").and_then(Value::as_str), + Some("completed" | "failed") + ) + }) + .count(), + 1, + "same-session public turn must emit exactly one terminal envelope after accepted: {turn_records:?}" + ); + assert_eq!( + turn_accepted.get("scope").and_then(Value::as_str), + Some("host") + ); + assert_eq!( + turn_json.get("action").and_then(Value::as_str), + Some("turn") + ); + assert_eq!( + turn_json + .get("orchestration_session_id") + .and_then(Value::as_str), + Some(orchestration_session_id) + ); + assert_eq!( + turn_json.get("backend_id").and_then(Value::as_str), + Some("cli:codex") + ); + assert_eq!( + turn_json.get("session_posture").and_then(Value::as_str), + Some("active") + ); + assert_eq!( + turn_json.get("state").and_then(Value::as_str), + Some("active") + ); + assert!( + turn_json.get("source_orchestration_session_id").is_none(), + "turn must remain on the selected orchestration session id: {turn_json}" + ); + assert_empty_warnings(turn_json); + + let turned_participant_id = turn_json["participant_id"] + .as_str() + .expect("turned participant id") + .to_string(); + let turned_participant = + fixture.load_participant(orchestration_session_id, &turned_participant_id); + assert_eq!( + turned_participant + .get("resumed_from_participant_id") + .and_then(Value::as_str), + Some("ash_turn_source") + ); + + let reparked_session = wait_for_session_posture( + &fixture, + orchestration_session_id, + "parked_resumable", + Duration::from_secs(5), + ); + assert_eq!( + reparked_session + .get("active_session_handle_id") + .and_then(Value::as_str), + Some(turned_participant_id.as_str()) + ); + assert!(reparked_session + .get("attached_participant_id") + .is_none_or(Value::is_null)); + assert_eq!( + reparked_session + .get("pending_inbox_count") + .and_then(Value::as_u64), + Some(0) + ); + let reparked_status_output = fixture.run(&["agent", "status", "--json"]); + assert!( + reparked_status_output.status.success(), + "status should reflect detached durable truth immediately after same-session turn repark: {reparked_status_output:?}" + ); + let reparked_status_json = parse_json_output(&reparked_status_output); + let reparked_status_row = find_status_session_by_orchestration_session_id( + status_sessions(&reparked_status_json), + orchestration_session_id, + ); + assert_eq!( + reparked_status_row.get("posture").and_then(Value::as_str), + Some("parked_resumable") + ); + assert!( + reparked_status_row + .get("attached_participant_id") + .is_some_and(Value::is_null), + "status must report the detached parked participant as explicit null after turn: {reparked_status_row}" + ); + assert_eq!( + reparked_status_row + .get("pending_inbox_count") + .and_then(Value::as_u64), + Some(0) + ); + + let reattach_output = fixture.run(&[ + "agent", + "reattach", + "--session", + orchestration_session_id, + "--json", + ]); + assert!( + reattach_output.status.success(), + "reattach should recover the same durable session after the follow-up turn exits: {reattach_output:?}" + ); + let reattach_json = parse_json_output(&reattach_output); + let reattached_participant_id = reattach_json["participant_id"] + .as_str() + .expect("reattach participant id") + .to_string(); + let resumed_owner_pid = wait_for_successor_owner_pid( + &fixture, + orchestration_session_id, + &turned_participant_id, + Duration::from_secs(5), + ); + let reattached_session = wait_for_session_posture( + &fixture, + orchestration_session_id, + "active_attached", + Duration::from_secs(5), + ); + assert_eq!( + reattached_session + .get("attached_participant_id") + .and_then(Value::as_str), + Some(reattached_participant_id.as_str()) + ); + assert!(pid_is_alive(resumed_owner_pid)); + assert!( + wait_for_path( + &stop_transport_path( + &fixture, + orchestration_session_id, + &reattached_participant_id, + ), + Duration::from_secs(5), + ), + "reattach must restore the private stop transport before the same-session stop" + ); + + let stop_output = fixture.run(&[ + "agent", + "stop", + "--session", + orchestration_session_id, + "--json", + ]); + assert!( + stop_output.status.success(), + "stop should close the same orchestration session after parked turn plus reattach: {stop_output:?}" + ); + let stop_json = parse_json_output(&stop_output); + assert_eq!( + stop_json.get("action").and_then(Value::as_str), + Some("stop") + ); + assert_eq!( + stop_json + .get("orchestration_session_id") + .and_then(Value::as_str), + Some(orchestration_session_id) + ); + assert_eq!( + stop_json.get("state").and_then(Value::as_str), + Some("stopped") + ); + + let stopped_session = fixture.load_orchestration_session(orchestration_session_id); + assert_eq!( + stopped_session.get("state").and_then(Value::as_str), + Some("stopped") + ); + assert_eq!( + stopped_session.get("posture").and_then(Value::as_str), + Some("terminal") + ); + assert!( + wait_for_pid_exit(resumed_owner_pid, Duration::from_secs(5)), + "same-session stop should leave no live reattached owner helper" + ); +} + +#[test] +#[serial] +fn public_start_split_bootstrap_retry_timeout_emits_single_terminal_failure() { + let fixture = AgentControlFixture::new_with_fake_codex( + write_fake_codex_script_clean_bootstrap_then_prompt_resume, + ); + fixture.init_workspace(); + fixture.write_runtime_inventory(false); + + let start_output = fixture.run(&[ + "agent", + "start", + "--backend", + "cli:codex", + "--prompt", + "hello from clean bootstrap", + "--json", + ]); + assert_eq!( + start_output.status.code(), + Some(1), + "split bootstrap start must surface the accepted-then-terminal failure contract when readiness reconciliation times out: {start_output:?}" + ); + let records = parse_ndjson_output(&start_output); + assert_eq!( + records + .first() + .and_then(|record| record.get("kind")) + .and_then(Value::as_str), + Some("accepted"), + "split bootstrap retry timeout must preserve the accepted envelope before terminal failure: {records:?}" + ); + assert_eq!( + records + .iter() + .filter(|record| { + matches!( + record.get("kind").and_then(Value::as_str), + Some("completed" | "failed") + ) + }) + .count(), + 1, + "split bootstrap retry timeout must emit exactly one terminal envelope after accepted: {records:?}" + ); + let failed = find_ndjson_record(&records, "failed"); + assert_eq!(failed.get("terminal").and_then(Value::as_bool), Some(true)); + assert_eq!(failed.get("stage").and_then(Value::as_str), Some("runtime")); + assert!( + failed + .get("error_code") + .and_then(Value::as_str) + .is_some_and(|code| code == "owner_unreachable"), + "split bootstrap retry timeout must keep the startup-stream owner_unreachable classifier: {failed}" + ); + assert!( + failed + .get("message") + .and_then(Value::as_str) + .is_some_and(|message| { + message.contains("startup prompt stream ended before terminal completion") + }), + "split bootstrap retry timeout must explain the terminalized startup prompt stream failure: {failed}" + ); + assert!( + records + .iter() + .all(|record| record.get("kind").and_then(Value::as_str) != Some("completed")), + "split bootstrap retry timeout must not also emit completed: {records:?}" + ); + let orchestration_session_id = records + .iter() + .find_map(|record| { + record + .get("orchestration_session_id") + .and_then(Value::as_str) + .map(str::to_string) + }) + .expect("split bootstrap accepted session id"); + let reconciled_session = wait_for_session_posture( + &fixture, + &orchestration_session_id, + "parked_resumable", + Duration::from_secs(5), + ); + assert_eq!( + reconciled_session.get("state").and_then(Value::as_str), + Some("active"), + "readiness timeout must reconcile the failed start onto detached live session truth" + ); + assert_ne!( + reconciled_session.get("posture").and_then(Value::as_str), + Some("active_attached"), + "failed start must not leave behind stale active_attached posture" + ); + assert_eq!( + reconciled_session + .pointer("/startup_prompt/state") + .and_then(Value::as_str), + Some("failed"), + "readiness-timeout fallback failure must terminalize startup prompt tracking" + ); + assert_eq!( + reconciled_session + .pointer("/startup_prompt/participant_id") + .and_then(Value::as_str), + reconciled_session + .get("active_session_handle_id") + .and_then(Value::as_str), + "terminalized startup prompt must remain pinned to the failed authoritative participant" + ); + assert!( + reconciled_session + .get("attached_participant_id") + .is_none_or(Value::is_null), + "readiness-timeout reconciliation must clear attached ownership once the helper detaches: {reconciled_session}" + ); + let stderr = stderr_text(&start_output); + assert!( + stderr.contains("startup prompt stream ended before terminal completion"), + "split bootstrap retry timeout stderr should match the terminal failed envelope message: {start_output:?}" + ); +} + +#[test] +#[serial] +fn public_start_persists_detached_session_when_hidden_owner_helper_exits() { + let fixture = + AgentControlFixture::new_with_fake_codex(write_fake_codex_script_exit_after_startup_prompt); + fixture.init_workspace(); + fixture.write_runtime_inventory(false); + + let start_output = fixture.run(&[ + "agent", + "start", + "--backend", + "cli:codex", + "--prompt", + "hello from parked start", + "--json", + ]); + assert!( + start_output.status.success(), + "public start should succeed even if the helper exits immediately after the startup prompt: {start_output:?}" + ); + let start_records = parse_ndjson_output(&start_output); + let start_json = find_ndjson_record(&start_records, "completed"); + assert_empty_warnings(start_json); + + let orchestration_session_id = start_json["orchestration_session_id"] + .as_str() + .expect("start session id") + .to_string(); + let participant_id = start_json["participant_id"] + .as_str() + .expect("start participant id") + .to_string(); + + let persisted_session = fixture.load_orchestration_session(&orchestration_session_id); + assert_eq!( + persisted_session.get("state").and_then(Value::as_str), + Some("active") + ); + assert_eq!( + persisted_session.get("posture").and_then(Value::as_str), + Some("parked_resumable"), + "stored session must already be detached-normalized before any later reattach" + ); + assert_eq!( + persisted_session + .get("active_session_handle_id") + .and_then(Value::as_str), + Some(participant_id.as_str()) + ); + assert!(persisted_session + .get("attached_participant_id") + .is_none_or(Value::is_null)); + assert_eq!( + persisted_session + .get("pending_inbox_count") + .and_then(Value::as_u64), + Some(0) + ); + assert_eq!( + persisted_session + .get("shell_owner_pid") + .and_then(Value::as_u64), + Some(0), + "detached normalization must clear the dead helper pid from persisted session truth" + ); + assert!(persisted_session + .get("closed_at") + .is_none_or(Value::is_null)); + + let persisted_participant = + fixture.load_participant(&orchestration_session_id, &participant_id); + assert_eq!( + persisted_participant + .pointer("/internal/uaa_session_id") + .and_then(Value::as_str), + Some("thread-test"), + "detached normalization must preserve the durable internal.uaa_session_id" + ); + assert_eq!( + persisted_participant + .pointer("/internal/shell_owner_pid") + .and_then(Value::as_u64), + Some(0), + "detached normalization must clear the dead helper pid from participant truth" + ); + assert_eq!( + persisted_participant + .pointer("/internal/attached_client_present") + .and_then(Value::as_bool), + Some(false) + ); + assert_eq!( + persisted_participant + .pointer("/internal/resume_eligible") + .and_then(Value::as_bool), + Some(true) + ); +} + +#[test] +#[serial] +fn detached_pending_inbox_normalizes_to_awaiting_attention() { + let fixture = AgentControlFixture::new(); + fixture.init_workspace(); + fixture.write_runtime_inventory(false); + + let ts = "2026-05-05T00:00:00Z"; + let orchestration_session_id = "sess_attention".to_string(); + let original_participant_id = "ash_attention".to_string(); + write_parked_orchestration_session( + &fixture, + "codex", + &orchestration_session_id, + &original_participant_id, + ts, + ); + write_runtime_participant( + &fixture, + &original_participant_id, + "codex", + &orchestration_session_id, + "running", + false, + Some("uaa-attention-1"), + None, + ts, + ); + + let parked_session = wait_for_session_posture( + &fixture, + &orchestration_session_id, + "parked_resumable", + Duration::from_secs(5), + ); + let parked_handle_id = parked_session + .get("active_session_handle_id") + .and_then(Value::as_str) + .expect("parked authoritative handle id") + .to_string(); + assert_eq!(parked_handle_id, original_participant_id); + assert!(parked_session + .get("attached_participant_id") + .is_none_or(Value::is_null)); + + persist_runtime_alert_for_substrate_home( + &fixture.substrate_home, + &orchestration_session_id, + "item_attention", + Some("attention needed for item_attention".to_string()), + ); + let attention_session = wait_for_session_posture( + &fixture, + &orchestration_session_id, + "awaiting_attention", + Duration::from_secs(5), + ); + assert_eq!( + attention_session + .get("orchestration_session_id") + .and_then(Value::as_str), + Some(orchestration_session_id.as_str()) + ); + assert_eq!( + attention_session.get("state").and_then(Value::as_str), + Some("active") + ); + assert_eq!( + attention_session.get("posture").and_then(Value::as_str), + Some("awaiting_attention") + ); + assert!(attention_session + .get("attached_participant_id") + .is_none_or(Value::is_null)); + assert_eq!( + attention_session + .get("active_session_handle_id") + .and_then(Value::as_str), + Some(parked_handle_id.as_str()) + ); + assert_eq!( + attention_session + .get("pending_inbox_count") + .and_then(Value::as_u64), + Some(1) + ); + let detached_participant = wait_for_detached_resume_eligible_participant( + &fixture, + &orchestration_session_id, + &original_participant_id, + Duration::from_secs(5), + ); + assert_eq!( + detached_participant + .pointer("/internal/resume_eligible") + .and_then(Value::as_bool), + Some(true), + "detached awaiting-attention normalization must keep the authoritative host participant resumable" + ); + assert_eq!( + detached_participant + .pointer("/internal/attached_client_present") + .and_then(Value::as_bool), + Some(false), + "detached awaiting-attention normalization must keep the authoritative host participant detached" + ); + + let status_output = fixture.run(&["agent", "status", "--json"]); + assert!( + status_output.status.success(), + "status must keep awaiting-attention durable rows visible after detached normalization: {status_output:?}" + ); + let status_json = parse_json_output(&status_output); + let attention_status_row = find_status_session_by_orchestration_session_id( + status_sessions(&status_json), + &orchestration_session_id, + ); + assert_eq!( + attention_status_row + .get("source_kind") + .and_then(Value::as_str), + Some("live_runtime") + ); + assert_eq!( + attention_status_row.get("posture").and_then(Value::as_str), + Some("awaiting_attention") + ); + assert!( + attention_status_row + .get("attached_participant_id") + .is_some_and(Value::is_null), + "awaiting-attention rows must preserve detached ownership as explicit null: {attention_status_row}" + ); + assert_eq!( + attention_status_row + .get("pending_inbox_count") + .and_then(Value::as_u64), + Some(1) + ); +} + +#[test] +#[serial] +fn public_start_reports_runtime_start_failed_for_missing_bootstrap_handle() { + let fixture = + AgentControlFixture::new_with_fake_codex(write_fake_codex_script_without_session_handle); + fixture.init_workspace(); + fixture.write_runtime_inventory(false); + + let start_output = fixture.run(&[ + "agent", + "start", + "--backend", + "cli:codex", + "--prompt", + "hello from broken bootstrap", + "--json", + ]); + assert_eq!( + start_output.status.code(), + Some(2), + "broken bootstrap must fail closed instead of reporting a successful parked session: {start_output:?}" + ); + assert!( + start_output.stdout.is_empty(), + "broken bootstrap must not emit public prompt envelopes when startup ownership never becomes authoritative: {start_output:?}" + ); + let stderr = stderr_text(&start_output); + assert!( + stderr.contains("runtime_start_failed"), + "broken bootstrap must keep the runtime_start_failed taxonomy: {start_output:?}" + ); + assert!( + stderr.contains("failed to establish attached control ownership") + || stderr.contains("timed out waiting for authoritative owner-helper readiness") + || stderr.contains("missing an active participant") + || stderr.contains("owner_unreachable"), + "broken bootstrap failure should explain the missing ownership/bootstrap continuity: {start_output:?}" + ); +} + +#[test] +#[serial] +fn public_start_survives_slow_startup_prompt_completion_after_bootstrap_readiness() { + let fixture = AgentControlFixture::new_with_fake_codex( + write_fake_codex_script_with_slow_startup_prompt_completion, + ); + fixture.init_workspace(); + fixture.write_runtime_inventory(false); + + let start_output = fixture.run(&[ + "agent", + "start", + "--backend", + "cli:codex", + "--prompt", + "hello from slow startup prompt", + "--json", + ]); + assert!( + start_output.status.success(), + "slow startup prompt completion must not trip the helper readiness timeout once bootstrap continuity is established: {start_output:?}" + ); + let records = parse_ndjson_output(&start_output); + assert_eq!( + records + .iter() + .filter(|record| record.get("kind").and_then(Value::as_str) == Some("accepted")) + .count(), + 1, + "slow startup prompt completion must still emit exactly one accepted envelope: {records:?}" + ); + assert_eq!( + records + .iter() + .filter(|record| { + matches!( + record.get("kind").and_then(Value::as_str), + Some("completed" | "failed") + ) + }) + .count(), + 1, + "slow startup prompt completion must still emit exactly one terminal envelope: {records:?}" + ); + let completed = find_ndjson_record(&records, "completed"); + let orchestration_session_id = completed["orchestration_session_id"] + .as_str() + .expect("slow start session id"); + let persisted_session = fixture.load_orchestration_session(orchestration_session_id); + assert_eq!( + persisted_session + .pointer("/startup_prompt/state") + .and_then(Value::as_str), + Some("completed"), + "slow startup prompt completion must persist completed startup prompt state" + ); + assert_eq!( + persisted_session.get("state").and_then(Value::as_str), + Some("active"), + "slow startup prompt completion must still produce a live durable session" + ); + assert_eq!( + persisted_session.get("posture").and_then(Value::as_str), + Some("parked_resumable"), + "slow startup prompt completion must still return a parked host session" + ); +} + +#[test] +#[serial] +fn public_turn_emits_explicit_failed_after_accepted_owner_drop() { + let fixture = AgentControlFixture::new_with_fake_codex( + write_fake_codex_script_with_delayed_prompt_completion, + ); + fixture.init_workspace(); + fixture.write_runtime_inventory(false); + + let ts = "2026-05-05T00:00:00Z"; + write_parked_orchestration_session( + &fixture, + "codex", + "sess_late_drop", + "ash_late_drop_source", + ts, + ); + write_runtime_participant( + &fixture, + "ash_late_drop_source", + "codex", + "sess_late_drop", + "running", + false, + Some("uaa-late-drop-1"), + None, + ts, + ); + + let mut child = fixture.spawn(&[ + "agent", + "turn", + "--session", + "sess_late_drop", + "--backend", + "cli:codex", + "--prompt", + "force late owner loss", + "--json", + ]); + let stdout = child.stdout.take().expect("stdout pipe"); + let mut reader = std::io::BufReader::new(stdout); + let mut line = String::new(); + let mut records = Vec::new(); + let mut killed_owner = false; + loop { + line.clear(); + let bytes_read = reader.read_line(&mut line).expect("read stdout line"); + if bytes_read == 0 { + break; + } + let trimmed = line.trim(); + if trimmed.is_empty() { + continue; + } + let record: Value = serde_json::from_str(trimmed).expect("stdout should be valid NDJSON"); + if !killed_owner && record.get("kind").and_then(Value::as_str) == Some("accepted") { + let owner_pid = wait_for_successor_owner_pid( + &fixture, + "sess_late_drop", + "ash_late_drop_source", + Duration::from_secs(5), + ); + terminate_pid(owner_pid); + killed_owner = true; + } + records.push(record); + } + + let status = child.wait().expect("wait for child"); + let stderr = { + let mut stderr = String::new(); + let mut handle = child.stderr.take().expect("stderr pipe"); + std::io::Read::read_to_string(&mut handle, &mut stderr).expect("read stderr"); + stderr + }; + + assert_eq!( + status.code(), + Some(1), + "owner loss after accepted must fail the public turn command: {records:?}\nstderr={stderr}" + ); + assert_eq!( + records + .first() + .and_then(|record| record.get("kind")) + .and_then(Value::as_str), + Some("accepted"), + "the late-drop prompt path must emit accepted before the owner is terminated: {records:?}" + ); + let failed = find_ndjson_record(&records, "failed"); + assert_eq!(failed.get("terminal").and_then(Value::as_bool), Some(true)); + assert_eq!( + failed.get("error_code").and_then(Value::as_str), + Some("owner_unreachable") + ); + assert!( + records + .iter() + .all(|record| record.get("kind").and_then(Value::as_str) != Some("completed")), + "owner drop after accepted must not silently produce completed: {records:?}" + ); +} + +#[test] +#[serial] +fn public_control_rejects_non_orchestration_session_selectors() { + let fixture = AgentControlFixture::new(); + fixture.init_workspace(); + fixture.write_runtime_inventory(false); + + let ts = "2026-05-05T00:00:00Z"; + write_parked_orchestration_session(&fixture, "codex", "sess_selector", "ash_live", ts); + write_runtime_participant( + &fixture, + "ash_live", + "codex", + "sess_selector", + "running", + false, + Some("uaa-live-1"), + None, + ts, + ); + write_runtime_participant( + &fixture, + "ash_previous", + "codex", + "sess_selector", + "invalidated", + false, + Some("uaa-old-1"), + Some("ash_older"), + ts, + ); + + let active_handle_output = + fixture.run(&["agent", "reattach", "--session", "ash_live", "--json"]); + assert_eq!( + active_handle_output.status.code(), + Some(2), + "active participant selectors must fail closed: {active_handle_output:?}" + ); + assert!( + stderr_text(&active_handle_output).contains("matched active_session_handle_id"), + "public control must explain active_session_handle_id rejection: {active_handle_output:?}" + ); + + let participant_output = + fixture.run(&["agent", "reattach", "--session", "ash_previous", "--json"]); + assert_eq!( + participant_output.status.code(), + Some(2), + "non-canonical participant selectors must fail closed: {participant_output:?}" + ); + assert!( + stderr_text(&participant_output).contains("matched participant_id/session_handle_id"), + "public control must explain participant/session-handle rejection: {participant_output:?}" + ); + + let internal_output = fixture.run(&["agent", "reattach", "--session", "uaa-live-1", "--json"]); + assert_eq!( + internal_output.status.code(), + Some(2), + "internal uaa session ids must fail closed as public selectors: {internal_output:?}" + ); + assert!( + stderr_text(&internal_output).contains("matched internal.uaa_session_id"), + "public control must explain internal.uaa_session_id rejection: {internal_output:?}" + ); + + let turn_selector_output = fixture.run(&[ + "agent", + "turn", + "--session", + "ash_live", + "--backend", + "cli:codex", + "--prompt", + "next", + "--json", + ]); + assert_eq!( + turn_selector_output.status.code(), + Some(2), + "public turn must reject non-canonical active handle selectors: {turn_selector_output:?}" + ); + assert!( + stderr_text(&turn_selector_output).contains("matched active_session_handle_id"), + "public turn must explain active_session_handle_id rejection: {turn_selector_output:?}" + ); + + let turn_participant_output = fixture.run(&[ + "agent", + "turn", + "--session", + "ash_previous", + "--backend", + "cli:codex", + "--prompt", + "next", + "--json", + ]); + assert_eq!( + turn_participant_output.status.code(), + Some(2), + "public turn must reject participant/session-handle selectors: {turn_participant_output:?}" + ); + assert!( + stderr_text(&turn_participant_output).contains("matched participant_id/session_handle_id"), + "public turn must explain participant/session-handle rejection: {turn_participant_output:?}" + ); + + let turn_internal_output = fixture.run(&[ + "agent", + "turn", + "--session", + "uaa-live-1", + "--backend", + "cli:codex", + "--prompt", + "next", + "--json", + ]); + assert_eq!( + turn_internal_output.status.code(), + Some(2), + "public turn must reject internal uaa session selectors: {turn_internal_output:?}" + ); + assert!( + stderr_text(&turn_internal_output).contains("matched internal.uaa_session_id"), + "public turn must explain internal uaa selector rejection: {turn_internal_output:?}" + ); +} + +#[test] +#[serial] +fn public_turn_fail_closed_taxonomy_is_explicit_for_missing_backend_unknown_session_and_parent_slot_errors( +) { + let fixture = AgentControlFixture::new(); + fixture.init_workspace(); + fixture.write_runtime_inventory(true); + + let ts = "2026-05-05T00:00:00Z"; + write_active_orchestration_session(&fixture, "codex", "sess_host_only", "ash_live", ts); + write_runtime_participant( + &fixture, + "ash_live", + "codex", + "sess_host_only", + "running", + true, + Some("uaa-live-1"), + None, + ts, + ); + + let missing_backend_output = fixture.run(&[ + "agent", + "turn", + "--session", + "sess_host_only", + "--backend", + "", + "--prompt", + "next", + "--json", + ]); + assert_eq!( + missing_backend_output.status.code(), + Some(2), + "public turn must fail closed when --backend is missing: {missing_backend_output:?}" + ); + assert!( + stderr_text(&missing_backend_output).contains("missing_backend"), + "missing backend must stay classified explicitly: {missing_backend_output:?}" + ); + + let unknown_session_output = fixture.run(&[ + "agent", + "turn", + "--session", + "sess_missing", + "--backend", + "cli:codex", + "--prompt", + "next", + "--json", + ]); + assert_eq!( + unknown_session_output.status.code(), + Some(2), + "unknown orchestration sessions must fail closed: {unknown_session_output:?}" + ); + assert!( + stderr_text(&unknown_session_output).contains("unknown_session"), + "unknown orchestration sessions must keep the frozen classifier: {unknown_session_output:?}" + ); + + write_orchestration_session( + &fixture, + "codex", + "sess_stopped", + Some("ash_stopped"), + "stopped", + None, + None, + ts, + ); + write_runtime_participant( + &fixture, + "ash_stopped", + "codex", + "sess_stopped", + "running", + true, + Some("uaa-live-stopped"), + None, + ts, + ); + let missing_parent_output = fixture.run(&[ + "agent", + "turn", + "--session", + "sess_stopped", + "--backend", + "cli:codex", + "--prompt", + "next", + "--json", + ]); + assert_eq!( + missing_parent_output.status.code(), + Some(2), + "inactive parents must fail closed for follow-up turns: {missing_parent_output:?}" + ); + assert!( + stderr_text(&missing_parent_output).contains("missing_active_parent"), + "inactive parents must keep the missing_active_parent classifier: {missing_parent_output:?}" + ); + + let backend_not_in_session_output = fixture.run(&[ + "agent", + "turn", + "--session", + "sess_host_only", + "--backend", + "cli:claude_code", + "--prompt", + "next", + "--json", + ]); + assert_eq!( + backend_not_in_session_output.status.code(), + Some(2), + "public turn must fail closed when the backend is not present in the orchestration session: {backend_not_in_session_output:?}" + ); + assert!( + stderr_text(&backend_not_in_session_output).contains("backend_not_in_session"), + "backend-not-in-session must keep the frozen classifier: {backend_not_in_session_output:?}" + ); +} + +#[cfg(target_os = "linux")] +#[test] +#[serial] +fn public_turn_fail_closed_taxonomy_is_explicit_for_world_linkage_ambiguity_and_detached_rejection() +{ + let fixture = AgentControlFixture::new(); + fixture.init_workspace(); + fixture.write_runtime_inventory(true); + + let ts = "2026-05-05T00:00:00Z"; + + write_orchestration_session( + &fixture, + "codex", + "sess_world_stale", + Some("ash_owner_stale"), + "active", + Some("world-17"), + Some(2), + ts, + ); + write_runtime_participant( + &fixture, + "ash_owner_stale", + "codex", + "sess_world_stale", + "running", + true, + Some("uaa-owner-1"), + None, + ts, + ); + write_world_member_participant( + &fixture, + "ash_member_stale", + "claude_code", + "sess_world_stale", + "ash_stale_owner", + "world-17", + 2, + "ready", + true, + Some("uaa-member-stale"), + ts, + ); + + let stale_linkage_output = fixture.run(&[ + "agent", + "turn", + "--session", + "sess_world_stale", + "--backend", + "cli:claude_code", + "--prompt", + "next", + "--json", + ]); + assert_eq!( + stale_linkage_output.status.code(), + Some(2), + "world follow-up must fail closed when retained linkage drifts: {stale_linkage_output:?}" + ); + assert!( + stderr_text(&stale_linkage_output).contains("stale_linkage"), + "world linkage drift must keep the stale_linkage classifier: {stale_linkage_output:?}" + ); + + write_orchestration_session( + &fixture, + "codex", + "sess_world_ambiguous", + Some("ash_owner_ambiguous"), + "active", + Some("world-18"), + Some(3), + ts, + ); + write_runtime_participant( + &fixture, + "ash_owner_ambiguous", + "codex", + "sess_world_ambiguous", + "running", + true, + Some("uaa-owner-2"), + None, + ts, + ); + write_world_member_participant( + &fixture, + "ash_member_a", + "claude_code", + "sess_world_ambiguous", + "ash_owner_ambiguous", + "world-18", + 3, + "ready", + true, + Some("uaa-member-a"), + ts, + ); + write_world_member_participant( + &fixture, + "ash_member_b", + "claude_code", + "sess_world_ambiguous", + "ash_owner_ambiguous", + "world-18", + 3, + "ready", + true, + Some("uaa-member-b"), + ts, + ); + + let ambiguous_output = fixture.run(&[ + "agent", + "turn", + "--session", + "sess_world_ambiguous", + "--backend", + "cli:claude_code", + "--prompt", + "next", + "--json", + ]); + assert_eq!( + ambiguous_output.status.code(), + Some(2), + "world follow-up must fail closed when multiple authoritative slots match the same backend: {ambiguous_output:?}" + ); + assert!( + stderr_text(&ambiguous_output).contains("ambiguous_backend_slot"), + "ambiguous world member slots must keep the frozen classifier: {ambiguous_output:?}" + ); + + write_parked_world_orchestration_session( + &fixture, + "codex", + "sess_world_detached", + "ash_owner_detached", + "world-19", + 4, + ts, + ); + write_runtime_participant( + &fixture, + "ash_owner_detached", + "codex", + "sess_world_detached", + "running", + false, + Some("uaa-owner-3"), + None, + ts, + ); + write_world_member_participant( + &fixture, + "ash_member_detached", + "claude_code", + "sess_world_detached", + "ash_owner_detached", + "world-19", + 4, + "ready", + false, + Some("uaa-member-detached"), + ts, + ); + + let detached_world_output = fixture.run(&[ + "agent", + "turn", + "--session", + "sess_world_detached", + "--backend", + "cli:claude_code", + "--prompt", + "next", + "--json", + ]); + assert_eq!( + detached_world_output.status.code(), + Some(2), + "detached world follow-up must fail closed until the parent owner is reattached: {detached_world_output:?}" + ); + let detached_world_stderr = stderr_text(&detached_world_output); + assert!( + detached_world_stderr.contains("unsupported_platform_or_posture"), + "detached world follow-up must keep the detached-world posture classifier: {detached_world_output:?}" + ); + assert!( + detached_world_stderr.contains("substrate agent reattach --session sess_world_detached"), + "detached world rejection must direct callers through reattach before world follow-up resumes: {detached_world_output:?}" + ); +} + +#[cfg(target_os = "linux")] +#[test] +#[serial] +fn public_turn_routes_linux_world_member_follow_up_through_typed_submit_path() { + let fixture = AgentControlFixture::new(); + fixture.init_workspace(); + fixture.write_runtime_inventory(true); + + let socket_home = tempfile::Builder::new() + .prefix("sac-world-submit-") + .tempdir_in("/tmp") + .expect("socket tempdir"); + let socket_path = socket_home.path().join("world.sock"); + let server = ReplWorldAgentStub::start_with_member_dispatch_scripts( + &socket_path, + StreamBehavior::Normal, + vec![MemberDispatchStreamScript::ReadyAndHoldUntilCancel { + session_handle_id: "session-public-world-turn".to_string(), + exit_code_on_cancel: 130, + }], + ); + let records = server.records(); + let mut repl = PtyRepl::spawn_with_world_socket(&fixture, &socket_path); + repl.wait_for_output("Substrate v", Duration::from_secs(6)) + .expect("banner"); + repl.wait_for_output("substrate>", Duration::from_secs(2)) + .expect("prompt"); + + repl.send_line("::cli:codex start retained host runtime"); + repl.wait_for_output( + "shell-owned orchestrator session is ready via retained attached control ownership", + Duration::from_secs(5), + ) + .expect("host runtime ready"); + + repl.send_line("::cli:claude_code member targeted first turn"); + repl.wait_for_output("substrate>", Duration::from_secs(5)) + .expect("prompt after initial world turn"); + + let (orchestration_session_id, owner_participant_id) = + wait_for_single_active_session(&fixture, Duration::from_secs(5)); + let owner_pid = fixture.load_orchestration_session(&orchestration_session_id)["shell_owner_pid"] + .as_u64() + .expect("owner pid") as u32; + let live_members = wait_for_live_world_member_count( + &fixture, + &orchestration_session_id, + 1, + Duration::from_secs(5), + ); + let member = &live_members[0]; + let member_participant_id = member + .get("participant_id") + .and_then(Value::as_str) + .expect("member participant_id") + .to_string(); + let member_orchestrator_participant_id = member + .get("orchestrator_participant_id") + .and_then(Value::as_str) + .expect("member orchestrator_participant_id") + .to_string(); + assert_eq!( + member_orchestrator_participant_id, owner_participant_id, + "the retained world member must stay linked to the exact authoritative owner participant" + ); + let world_id = member + .get("world_id") + .and_then(Value::as_str) + .expect("member world_id") + .to_string(); + let world_generation = member + .get("world_generation") + .and_then(Value::as_u64) + .expect("member world_generation"); + + let turn_output = fixture + .command() + .current_dir(&fixture.workspace_root) + .env("SUBSTRATE_WORLD_SOCKET", &socket_path) + .args([ + "agent", + "turn", + "--session", + &orchestration_session_id, + "--backend", + "cli:claude_code", + "--prompt", + "continue in world", + "--json", + ]) + .output() + .expect("run public world turn"); + assert!( + turn_output.status.success(), + "public world turn should succeed on Linux: {turn_output:?}" + ); + let turn_records = parse_ndjson_output(&turn_output); + let turn_json = find_ndjson_record(&turn_records, "completed"); + assert_eq!( + turn_json.get("action").and_then(Value::as_str), + Some("turn") + ); + assert_eq!( + turn_json + .get("orchestration_session_id") + .and_then(Value::as_str), + Some(orchestration_session_id.as_str()) + ); + assert_eq!( + turn_json.get("backend_id").and_then(Value::as_str), + Some("cli:claude_code") + ); + assert_eq!( + turn_json.get("turn_outcome").and_then(Value::as_str), + Some("success") + ); + assert_eq!( + turn_json.get("session_posture").and_then(Value::as_str), + Some("active") + ); + + let guard = records.lock().expect("lock world-agent records"); + assert_eq!( + guard.member_turn_submit_requests.len(), + 1, + "public world follow-up must submit exactly one typed member turn request: {guard:#?}" + ); + let submit = &guard.member_turn_submit_requests[0]; + assert_eq!(submit.orchestration_session_id, orchestration_session_id); + assert_eq!(submit.participant_id, member_participant_id); + assert_eq!(submit.orchestrator_participant_id, owner_participant_id); + assert_eq!(submit.backend_id, "cli:claude_code"); + assert_eq!(submit.world_id, world_id); + assert_eq!(submit.world_generation, world_generation); + assert_eq!(submit.prompt, "continue in world"); + drop(guard); + + assert!( + String::from_utf8_lossy(&turn_output.stdout) + .contains("__MEMBER_TURN_SUBMIT_STUB__ continue in world"), + "public world follow-up must surface typed submit output: {turn_output:?}" + ); + + repl.send_line("exit"); + let (_code, _out) = repl.shutdown_graceful(Duration::from_secs(3)); + terminate_pid(owner_pid); +} + +#[test] +#[serial] +fn public_root_start_rejects_world_scoped_backends_in_v1() { + let fixture = AgentControlFixture::new(); + fixture.init_workspace(); + fixture.write_runtime_inventory(true); + + let output = fixture.run(&[ + "agent", + "start", + "--backend", + "cli:claude_code", + "--prompt", + "hello", + "--json", + ]); + assert_eq!( + output.status.code(), + Some(2), + "world-scoped root start must fail closed: {output:?}" + ); + let stderr = stderr_text(&output); + assert!( + stderr.contains("unsupported_platform_or_posture"), + "world-scoped root start must classify the posture failure exactly: {stderr}" + ); + assert!( + stderr.contains("public root start is host-only in v1"), + "world-scoped root start failure must explain the Linux-first host-only contract: {stderr}" + ); +} + +#[test] +#[serial] +fn public_command_mode_remains_shell_wrap_not_agent_prompt() { + let fixture = AgentControlFixture::new(); + fixture.init_workspace(); + fs::write( + fixture.workspace_root.join(".substrate/workspace.yaml"), + "world:\n enabled: false\n", + ) + .expect("disable world in workspace config"); + fixture.write_runtime_inventory(false); + fs::write( + fixture.workspace_root.join(".substrate-profile"), + "id: test-policy\nname: Test Policy\nworld_fs:\n host_visible: true\n fail_closed:\n routing: false\n write:\n enabled: true\nnet_allowed: []\ncmd_allowed: []\ncmd_denied: []\ncmd_isolated: []\nrequire_approval: false\nallow_shell_operators: true\nlimits:\n max_memory_mb: null\n max_cpu_percent: null\n max_runtime_ms: null\n max_egress_bytes: null\nmetadata: {}\n", + ) + .expect("write host-only profile"); + fs::write( + fixture.substrate_home.join("policy.yaml"), + "id: test-global-policy\nname: Test Global Policy\nworld_fs:\n host_visible: true\n fail_closed:\n routing: false\n write:\n enabled: true\nnet_allowed: []\ncmd_allowed: []\ncmd_denied: []\ncmd_isolated: []\nrequire_approval: false\nallow_shell_operators: true\nlimits:\n max_memory_mb: null\n max_cpu_percent: null\n max_runtime_ms: null\n max_egress_bytes: null\nmetadata: {}\nagents:\n allowed_backends:\n - cli:codex\n", + ) + .expect("write host-only policy"); + + let output = fixture.run(&["-c", "printf shell-wrap"]); + assert!( + output.status.success(), + "substrate -c must remain ordinary shell wrap mode: {output:?}" + ); + assert_eq!(String::from_utf8_lossy(&output.stdout), "shell-wrap"); + assert!( + !stderr_text(&output).contains("missing_prompt_source"), + "substrate -c must not be reinterpreted as an agent prompt surface: {output:?}" + ); +} + +#[test] +#[serial] +fn public_stop_reaches_repl_owned_sessions_through_the_same_private_owner_plane() { + let fixture = AgentControlFixture::new(); + fixture.write_runtime_inventory(false); + + let mut repl = PtyRepl::spawn(&fixture); + repl.wait_for_output("Substrate v", Duration::from_secs(2)) + .expect("repl banner"); + repl.wait_for_output("substrate>", Duration::from_secs(2)) + .expect("initial prompt"); + repl.send_line("::cli:codex start retained host runtime"); + repl.wait_for_output( + "shell-owned orchestrator session is ready via retained attached control ownership", + Duration::from_secs(5), + ) + .expect("runtime ready"); + let (orchestration_session_id, participant_id) = + wait_for_single_active_session(&fixture, Duration::from_secs(5)); + assert!( + wait_for_path( + &stop_transport_path(&fixture, &orchestration_session_id, &participant_id), + Duration::from_secs(5), + ), + "repl-owned sessions must publish the same per-session private stop transport" + ); + let stop_output = fixture.run(&[ + "agent", + "stop", + "--session", + &orchestration_session_id, + "--json", + ]); + assert!( + stop_output.status.success(), + "public stop should succeed against a REPL-owned owner plane: {stop_output:?}" + ); + let stop_json = parse_json_output(&stop_output); + assert_eq!( + stop_json.get("action").and_then(Value::as_str), + Some("stop") + ); + assert_eq!( + stop_json + .get("orchestration_session_id") + .and_then(Value::as_str), + Some(orchestration_session_id.as_str()) + ); + assert_eq!( + fixture + .load_orchestration_session(&orchestration_session_id) + .get("state") + .and_then(Value::as_str), + Some("stopped"), + "REPL-owned public stop must drive the same authoritative terminal session state" + ); + + repl.send_line("exit"); + let (code, output) = repl.shutdown_graceful(Duration::from_secs(5)); + assert_eq!( + code, + 0, + "repl should still exit cleanly after public stop:\n{}", + String::from_utf8_lossy(&output) + ); +} diff --git a/crates/shell/tests/agent_successor_contract_ahcsitc0.rs b/crates/shell/tests/agent_successor_contract_ahcsitc0.rs index 2f7000afb..b33f58fc7 100644 --- a/crates/shell/tests/agent_successor_contract_ahcsitc0.rs +++ b/crates/shell/tests/agent_successor_contract_ahcsitc0.rs @@ -2,12 +2,12 @@ mod common; -use common::{substrate_shell_driver, temp_dir}; +use common::substrate_shell_driver; use serde_json::{json, Value}; use std::fs; -use std::path::PathBuf; +use std::path::{Path, PathBuf}; use std::process::Output; -use tempfile::TempDir; +use tempfile::{Builder, TempDir}; const PURE_AGENT_PROTOCOL: &str = "uaa.agent.session"; @@ -45,7 +45,10 @@ struct AgentSuccessorFixture { impl AgentSuccessorFixture { fn new() -> Self { - let temp = temp_dir("substrate-agent-successor-"); + let temp = Builder::new() + .prefix("substrate-agent-successor-") + .tempdir_in("/tmp") + .expect("failed to allocate integration test temp dir"); let home = temp.path().join("home"); fs::create_dir_all(&home).expect("failed to create HOME fixture"); let substrate_home = temp.path().join("substrate-home"); @@ -264,17 +267,15 @@ fn write_live_runtime_manifest( fixture: &AgentSuccessorFixture, agent_id: &str, orchestration_session_id: &str, - session_handle_id: &str, + participant_id: &str, ts: &str, ) { - write_runtime_manifest( + write_runtime_participant( fixture, + participant_id, agent_id, orchestration_session_id, - session_handle_id, - "running", - true, - ts, + RuntimeParticipantOptions::host_orchestrator("running", true, ts), ); } @@ -282,75 +283,612 @@ fn write_invalidated_runtime_manifest( fixture: &AgentSuccessorFixture, agent_id: &str, orchestration_session_id: &str, - session_handle_id: &str, + participant_id: &str, ts: &str, ) { - write_runtime_manifest( + write_runtime_participant( fixture, + participant_id, agent_id, orchestration_session_id, - session_handle_id, - "invalidated", - false, - ts, + RuntimeParticipantOptions::host_orchestrator("invalidated", false, ts), ); } -fn write_runtime_manifest( +#[allow(clippy::too_many_arguments)] +fn write_invalidated_world_member_manifest( fixture: &AgentSuccessorFixture, agent_id: &str, orchestration_session_id: &str, - session_handle_id: &str, - state: &str, - ownership_valid: bool, + participant_id: &str, + orchestrator_participant_id: &str, + world_id: &str, + world_generation: u64, + resumed_from_participant_id: Option<&str>, ts: &str, ) { - let handles_dir = fixture - .substrate_home - .join("run") - .join("agent-hub") - .join("handles"); - fs::create_dir_all(&handles_dir).expect("failed to create runtime handles dir"); - let manifest = json!({ - "session_handle_id": session_handle_id, - "orchestration_session_id": orchestration_session_id, - "agent_id": agent_id, - "backend_id": format!("cli:{agent_id}"), - "role": "orchestrator", - "protocol": PURE_AGENT_PROTOCOL, - "execution": { "scope": "host" }, - "state": state, - "opened_at": ts, - "last_transition_at": ts, - "parent_session_handle_id": null, - "resumed_from_session_handle_id": null, - "internal": { + write_runtime_participant( + fixture, + participant_id, + agent_id, + orchestration_session_id, + RuntimeParticipantOptions::world_member( + "invalidated", + false, + ts, + world_id, + world_generation, + orchestrator_participant_id, + ) + .with_resumed_from_participant_id(resumed_from_participant_id), + ); +} + +#[allow(clippy::too_many_arguments)] +fn write_replacement_world_member_manifest( + fixture: &AgentSuccessorFixture, + agent_id: &str, + orchestration_session_id: &str, + participant_id: &str, + orchestrator_participant_id: &str, + world_id: &str, + world_generation: u64, + resumed_from_participant_id: &str, + ts: &str, +) { + write_runtime_participant( + fixture, + participant_id, + agent_id, + orchestration_session_id, + RuntimeParticipantOptions::world_member( + "ready", + true, + ts, + world_id, + world_generation, + orchestrator_participant_id, + ) + .with_resumed_from_participant_id(Some(resumed_from_participant_id)), + ); +} + +#[derive(Clone, Copy)] +struct RuntimeParticipantOptions<'a> { + role: &'a str, + scope: &'a str, + state: &'a str, + ownership_mode: &'a str, + ownership_valid: bool, + ts: &'a str, + world_binding: Option<(&'a str, u64)>, + parent_participant_id: Option<&'a str>, + resumed_from_participant_id: Option<&'a str>, + orchestrator_participant_id: Option<&'a str>, +} + +impl<'a> RuntimeParticipantOptions<'a> { + fn host_orchestrator(state: &'a str, ownership_valid: bool, ts: &'a str) -> Self { + Self { + role: "orchestrator", + scope: "host", + state, + ownership_mode: "attached_control", + ownership_valid, + ts, + world_binding: None, + parent_participant_id: None, + resumed_from_participant_id: None, + orchestrator_participant_id: None, + } + } + + fn world_member( + state: &'a str, + ownership_valid: bool, + ts: &'a str, + world_id: &'a str, + world_generation: u64, + orchestrator_participant_id: &'a str, + ) -> Self { + Self { + role: "member", + scope: "world", + state, + ownership_mode: "member_runtime", + ownership_valid, + ts, + world_binding: Some((world_id, world_generation)), + parent_participant_id: None, + resumed_from_participant_id: None, + orchestrator_participant_id: Some(orchestrator_participant_id), + } + } + + fn with_resumed_from_participant_id( + mut self, + resumed_from_participant_id: Option<&'a str>, + ) -> Self { + self.resumed_from_participant_id = resumed_from_participant_id; + self + } +} + +fn write_runtime_participant( + fixture: &AgentSuccessorFixture, + participant_id: &str, + agent_id: &str, + orchestration_session_id: &str, + options: RuntimeParticipantOptions<'_>, +) { + write_json_file( + &canonical_participant_manifest_path(fixture, orchestration_session_id, participant_id), + &runtime_participant_manifest( + fixture, + participant_id, + agent_id, + orchestration_session_id, + options, + ), + ); +} + +fn write_flat_runtime_participant_compatibility( + fixture: &AgentSuccessorFixture, + participant_id: &str, + agent_id: &str, + orchestration_session_id: &str, + options: RuntimeParticipantOptions<'_>, +) { + write_json_file( + &flat_participant_manifest_path(fixture, participant_id), + &runtime_participant_manifest( + fixture, + participant_id, + agent_id, + orchestration_session_id, + options, + ), + ); +} + +fn runtime_participant_manifest( + _fixture: &AgentSuccessorFixture, + participant_id: &str, + agent_id: &str, + orchestration_session_id: &str, + options: RuntimeParticipantOptions<'_>, +) -> Value { + let is_host_orchestrator = options.role == "orchestrator" && options.scope == "host"; + let attached_client_present = is_host_orchestrator && options.ownership_valid; + let resume_eligible = is_host_orchestrator + && options.state != "invalidated" + && options.state != "stopped" + && options.state != "failed"; + let mut manifest = serde_json::Map::new(); + manifest.insert("participant_id".to_string(), json!(participant_id)); + manifest.insert( + "orchestration_session_id".to_string(), + json!(orchestration_session_id), + ); + manifest.insert("agent_id".to_string(), json!(agent_id)); + manifest.insert("backend_id".to_string(), json!(format!("cli:{agent_id}"))); + manifest.insert("role".to_string(), json!(options.role)); + manifest.insert("protocol".to_string(), json!(PURE_AGENT_PROTOCOL)); + manifest.insert("execution".to_string(), json!({ "scope": options.scope })); + manifest.insert("state".to_string(), json!(options.state)); + manifest.insert("opened_at".to_string(), json!(options.ts)); + manifest.insert("last_transition_at".to_string(), json!(options.ts)); + if let Some((world_id, world_generation)) = options.world_binding { + manifest.insert("world_id".to_string(), json!(world_id)); + manifest.insert("world_generation".to_string(), json!(world_generation)); + } + if let Some(parent_participant_id) = options.parent_participant_id { + manifest.insert( + "parent_participant_id".to_string(), + json!(parent_participant_id), + ); + } + if let Some(resumed_from_participant_id) = options.resumed_from_participant_id { + manifest.insert( + "resumed_from_participant_id".to_string(), + json!(resumed_from_participant_id), + ); + } + if let Some(orchestrator_participant_id) = options.orchestrator_participant_id { + manifest.insert( + "orchestrator_participant_id".to_string(), + json!(orchestrator_participant_id), + ); + } + manifest.insert( + "internal".to_string(), + json!({ "resolved_agent_kind": agent_id, "resolved_binary_path": "sh", "shell_owner_pid": std::process::id(), - "lease_token": "lease-test", - "uaa_session_id": "external-session-1", + "lease_token": format!("lease-{participant_id}"), + "uaa_session_id": if options.ownership_valid || resume_eligible { Value::String("external-session-1".to_string()) } else { Value::Null }, "latest_run_id": "0195f8f1-7a35-7b7f-9c4d-9a7c2f5d6fab", "cancel_supported": true, - "control_owner_retained": ownership_valid, - "event_stream_active": ownership_valid, - "completion_observer_retained": ownership_valid, - "ownership_mode": "attached_control", - "ownership_valid": ownership_valid, - "ownership_verified_at": ts, - "last_heartbeat_at": ts, - "last_event_at": ts, - "terminal_observed_at": if ownership_valid { Value::Null } else { json!(ts) }, - "termination_reason": if ownership_valid { Value::Null } else { json!("attached control exited") }, + "control_owner_retained": options.ownership_valid, + "event_stream_active": options.ownership_valid, + "completion_observer_retained": options.ownership_valid, + "ownership_mode": options.ownership_mode, + "ownership_valid": options.ownership_valid, + "ownership_verified_at": options.ts, + "last_heartbeat_at": options.ts, + "last_event_at": options.ts, + "terminal_observed_at": if options.state == "invalidated" || options.state == "stopped" || options.state == "failed" { json!(options.ts) } else { Value::Null }, + "termination_reason": if options.state == "invalidated" || options.state == "stopped" || options.state == "failed" { json!("attached control exited") } else { Value::Null }, + "attached_client_present": attached_client_present, + "last_attached_at": if is_host_orchestrator { json!(options.ts) } else { Value::Null }, + "last_detached_at": if is_host_orchestrator && !attached_client_present && resume_eligible { json!(options.ts) } else { Value::Null }, + "detach_reason": if is_host_orchestrator && !attached_client_present && resume_eligible { json!("owner detached cleanly") } else { Value::Null }, + "resume_eligible": resume_eligible, "last_error_bucket": null, "last_error_message": null - } - }); + }), + ); + Value::Object(manifest) +} + +fn participant_manifest_path(fixture: &AgentSuccessorFixture, participant_id: &str) -> PathBuf { + let participant_dirs = fixture + .substrate_home + .join("run") + .join("agent-hub") + .join("sessions"); + fs::read_dir(&participant_dirs) + .expect("canonical sessions dir should exist") + .filter_map(|entry| entry.ok()) + .map(|entry| entry.path()) + .find_map(|path| { + let candidate = path + .join("participants") + .join(format!("{participant_id}.json")); + candidate.is_file().then_some(candidate) + }) + .unwrap_or_else(|| { + panic!("expected canonical participant manifest for `{participant_id}` to exist") + }) +} + +fn canonical_participant_manifest_path( + fixture: &AgentSuccessorFixture, + orchestration_session_id: &str, + participant_id: &str, +) -> PathBuf { + fixture + .substrate_home + .join("run") + .join("agent-hub") + .join("sessions") + .join(orchestration_session_id) + .join("participants") + .join(format!("{participant_id}.json")) +} + +fn flat_participant_manifest_path( + fixture: &AgentSuccessorFixture, + participant_id: &str, +) -> PathBuf { + fixture + .substrate_home + .join("run") + .join("agent-hub") + .join("participants") + .join(format!("{participant_id}.json")) +} + +fn canonical_orchestration_session_path( + fixture: &AgentSuccessorFixture, + orchestration_session_id: &str, +) -> PathBuf { + fixture + .substrate_home + .join("run") + .join("agent-hub") + .join("sessions") + .join(orchestration_session_id) + .join("session.json") +} + +fn flat_orchestration_session_path( + fixture: &AgentSuccessorFixture, + orchestration_session_id: &str, +) -> PathBuf { + fixture + .substrate_home + .join("run") + .join("agent-hub") + .join("sessions") + .join(format!("{orchestration_session_id}.json")) +} + +fn write_active_orchestration_session( + fixture: &AgentSuccessorFixture, + agent_id: &str, + orchestration_session_id: &str, + active_session_handle_id: Option<&str>, + ts: &str, +) { + write_orchestration_session( + fixture, + agent_id, + orchestration_session_id, + active_session_handle_id, + "active", + ts, + ); +} + +fn write_active_orchestration_session_with_world_binding( + fixture: &AgentSuccessorFixture, + agent_id: &str, + orchestration_session_id: &str, + active_session_handle_id: Option<&str>, + ts: &str, + world_id: &str, + world_generation: u64, +) { + write_orchestration_session_with_world_binding( + fixture, + agent_id, + orchestration_session_id, + active_session_handle_id, + "active", + ts, + (world_id, world_generation), + ); +} + +fn write_inactive_orchestration_session( + fixture: &AgentSuccessorFixture, + agent_id: &str, + orchestration_session_id: &str, + active_session_handle_id: Option<&str>, + ts: &str, +) { + write_orchestration_session( + fixture, + agent_id, + orchestration_session_id, + active_session_handle_id, + "stopped", + ts, + ); +} + +fn write_orchestration_session_with_world_binding( + fixture: &AgentSuccessorFixture, + agent_id: &str, + orchestration_session_id: &str, + active_session_handle_id: Option<&str>, + state: &str, + ts: &str, + world_binding: (&str, u64), +) { + write_orchestration_session_impl( + fixture, + agent_id, + orchestration_session_id, + active_session_handle_id, + state, + ts, + Some(world_binding), + ); +} + +fn write_orchestration_session( + fixture: &AgentSuccessorFixture, + agent_id: &str, + orchestration_session_id: &str, + active_session_handle_id: Option<&str>, + state: &str, + ts: &str, +) { + write_orchestration_session_impl( + fixture, + agent_id, + orchestration_session_id, + active_session_handle_id, + state, + ts, + None, + ); +} + +fn write_orchestration_session_impl( + fixture: &AgentSuccessorFixture, + agent_id: &str, + orchestration_session_id: &str, + active_session_handle_id: Option<&str>, + state: &str, + ts: &str, + world_binding: Option<(&str, u64)>, +) { + write_orchestration_session_with_manifest_options( + fixture, + OrchestrationSessionManifestSpec { + agent_id, + orchestration_session_id, + active_session_handle_id, + state, + ts, + world_binding, + }, + OrchestrationSessionManifestOptions::default(), + ); +} + +#[derive(Clone, Copy)] +struct OrchestrationSessionManifestSpec<'a> { + agent_id: &'a str, + orchestration_session_id: &'a str, + active_session_handle_id: Option<&'a str>, + state: &'a str, + ts: &'a str, + world_binding: Option<(&'a str, u64)>, +} + +#[derive(Clone, Copy, Default)] +struct OrchestrationSessionManifestOptions<'a> { + posture: Option<&'a str>, + attached_participant_id: Option>, + pending_inbox_count: Option, + last_parked_at: Option>, + last_attention_at: Option>, + parked_reason: Option>, +} + +fn write_orchestration_session_with_manifest_options( + fixture: &AgentSuccessorFixture, + spec: OrchestrationSessionManifestSpec<'_>, + options: OrchestrationSessionManifestOptions<'_>, +) { + write_json_file( + &canonical_orchestration_session_path(fixture, spec.orchestration_session_id), + &orchestration_session_manifest_with_options(fixture, spec, options), + ); +} + +fn write_flat_orchestration_session_compatibility( + fixture: &AgentSuccessorFixture, + agent_id: &str, + orchestration_session_id: &str, + active_session_handle_id: Option<&str>, + state: &str, + ts: &str, + world_binding: Option<(&str, u64)>, +) { + write_json_file( + &flat_orchestration_session_path(fixture, orchestration_session_id), + &orchestration_session_manifest( + fixture, + OrchestrationSessionManifestSpec { + agent_id, + orchestration_session_id, + active_session_handle_id, + state, + ts, + world_binding, + }, + ), + ); +} + +fn orchestration_session_manifest( + fixture: &AgentSuccessorFixture, + spec: OrchestrationSessionManifestSpec<'_>, +) -> Value { + orchestration_session_manifest_with_options( + fixture, + spec, + OrchestrationSessionManifestOptions::default(), + ) +} + +fn orchestration_session_manifest_with_options( + fixture: &AgentSuccessorFixture, + spec: OrchestrationSessionManifestSpec<'_>, + options: OrchestrationSessionManifestOptions<'_>, +) -> Value { + let OrchestrationSessionManifestSpec { + agent_id, + orchestration_session_id, + active_session_handle_id, + state, + ts, + world_binding, + } = spec; + let (world_id, world_generation) = match world_binding { + Some((world_id, world_generation)) => (json!(world_id), json!(world_generation)), + None => (Value::Null, Value::Null), + }; + let (default_posture, default_attached_participant_id, closed_at) = + match (state, active_session_handle_id) { + ("active", Some(active_session_handle_id)) => ( + "active_attached", + Value::String(active_session_handle_id.to_string()), + Value::Null, + ), + ("active", None) => ("parked_resumable", Value::Null, Value::Null), + ("stopped", _) | ("failed", _) | ("invalidated", _) => { + ("terminal", Value::Null, json!(ts)) + } + _ => ( + "active_attached", + active_session_handle_id + .map(Value::from) + .unwrap_or(Value::Null), + if state == "active" { + Value::Null + } else { + json!(ts) + }, + ), + }; + let posture = options.posture.unwrap_or(default_posture); + let attached_participant_id = match options.attached_participant_id { + Some(Some(participant_id)) => Value::String(participant_id.to_string()), + Some(None) => Value::Null, + None => default_attached_participant_id, + }; + let pending_inbox_count = options.pending_inbox_count.unwrap_or(0); + let last_parked_at = match options.last_parked_at { + Some(Some(value)) => json!(value), + Some(None) => Value::Null, + None => Value::Null, + }; + let last_attention_at = match options.last_attention_at { + Some(Some(value)) => json!(value), + Some(None) => Value::Null, + None => Value::Null, + }; + let parked_reason = match options.parked_reason { + Some(Some(value)) => json!(value), + Some(None) => Value::Null, + None => Value::Null, + }; + json!({ + "orchestration_session_id": orchestration_session_id, + "shell_trace_session_id": "ses_agent_hub", + "workspace_root": fixture.workspace_root.display().to_string(), + "shell_owner_pid": std::process::id(), + "state": state, + "posture": posture, + "posture_changed_at": ts, + "opened_at": ts, + "last_active_at": ts, + "orchestrator_agent_id": agent_id, + "orchestrator_backend_id": format!("cli:{agent_id}"), + "orchestrator_protocol": PURE_AGENT_PROTOCOL, + "active_session_handle_id": active_session_handle_id, + "attached_participant_id": attached_participant_id, + "pending_inbox_count": pending_inbox_count, + "last_parked_at": last_parked_at, + "last_attention_at": last_attention_at, + "parked_reason": parked_reason, + "latest_run_id": "0195f8f1-7a35-7b7f-9c4d-9a7c2f5d6fab", + "world_id": world_id, + "world_generation": world_generation, + "invalidation_reason": if state == "active" { Value::Null } else { json!("fixture stopped parent") }, + "closed_at": closed_at + }) +} + +fn write_json_file(path: &Path, value: &Value) { + let parent = path + .parent() + .expect("json fixture path should have a parent"); + fs::create_dir_all(parent) + .unwrap_or_else(|err| panic!("failed to create fixture dir {}: {err}", parent.display())); fs::write( - handles_dir.join(format!("{session_handle_id}.json")), - serde_json::to_vec_pretty(&manifest).expect("serialize runtime manifest"), + path, + serde_json::to_vec_pretty(value).expect("serialize fixture json"), ) - .expect("write runtime manifest"); + .unwrap_or_else(|err| panic!("failed to write fixture json {}: {err}", path.display())); } fn repo_root() -> PathBuf { @@ -365,6 +903,35 @@ fn read_repo_file(path: &str) -> String { .unwrap_or_else(|err| panic!("failed to read {path}: {err}")) } +fn read_production_shell_source_without_inline_tests(path: &Path) -> String { + let source = fs::read_to_string(path) + .unwrap_or_else(|err| panic!("failed to read production source {}: {err}", path.display())); + let cutoff = source.find("#[cfg(test)]").unwrap_or(source.len()); + source[..cutoff].to_string() +} + +fn collect_rust_files(root: &Path) -> Vec { + let mut files = Vec::new(); + let mut stack = vec![root.to_path_buf()]; + while let Some(path) = stack.pop() { + let entries = fs::read_dir(&path) + .unwrap_or_else(|err| panic!("failed to read directory {}: {err}", path.display())); + for entry in entries { + let entry = entry.unwrap_or_else(|err| { + panic!("failed to enumerate directory {}: {err}", path.display()) + }); + let entry_path = entry.path(); + if entry_path.is_dir() { + stack.push(entry_path); + } else if entry_path.extension().and_then(|ext| ext.to_str()) == Some("rs") { + files.push(entry_path); + } + } + } + files.sort(); + files +} + fn parse_json_output(output: &Output) -> Value { serde_json::from_slice(&output.stdout).expect("stdout should be valid JSON") } @@ -508,31 +1075,139 @@ fn assert_doctor_fails_at_orchestrator_selection(output: &Output, expected_reaso ); } -fn find_session_by_agent<'a>(sessions: &'a [Value], agent_id: &str) -> &'a Value { - sessions - .iter() - .find(|session| session.pointer("/agent_id").and_then(Value::as_str) == Some(agent_id)) - .unwrap_or_else(|| panic!("expected session row for agent `{agent_id}`")) -} - -fn seed_nested_gateway_status_fixture(fixture: &AgentSuccessorFixture) { - fixture.init_workspace(); - fixture.write_global_config_patch( - r#"agents: - enabled: true - hub: - orchestrator_agent_id: claude_code -"#, +fn assert_parent_resolution_fail_closed( + output: &Output, + command: &str, + expected_stderr_fragments: &[&str], +) { + assert_eq!( + output.status.code(), + Some(2), + "{command} must fail closed on broken orchestrator parent resolution: {output:?}" + ); + assert!( + String::from_utf8_lossy(&output.stdout).trim().is_empty(), + "{command} failures must not emit stdout: {output:?}" ); - fixture.write_global_policy_patch( - r#"id: "ahcsitc2-policy" -name: "ahcsitc2-policy" -world_fs: - host_visible: true - fail_closed: - routing: true - write: + let stderr = String::from_utf8_lossy(&output.stderr); + for needle in expected_stderr_fragments { + assert!( + stderr.contains(needle), + "{command} stderr must contain `{needle}`: {stderr}" + ); + } +} + +fn assert_toolbox_parent_resolution_fail_closed_across_operator_surfaces( + fixture: &AgentSuccessorFixture, + expected_stderr_fragments: &[&str], +) { + for args in [ + vec!["agent", "toolbox", "status", "--json"], + vec!["agent", "toolbox", "env", "--json"], + ] { + let output = fixture.run(&args); + assert_parent_resolution_fail_closed(&output, &args.join(" "), expected_stderr_fragments); + } +} + +fn assert_status_succeeds_with_expected_warnings( + output: &Output, + expected_warning_fragments: &[&str], +) -> Value { + assert!( + output.status.success(), + "agent status should stay readable on degraded parent/session state: {output:?}" + ); + let json = parse_json_output(output); + let warnings = json["warnings"] + .as_array() + .expect("warnings should be an array"); + for needle in expected_warning_fragments { + assert!( + warnings + .iter() + .filter_map(Value::as_str) + .any(|warning| warning.contains(needle)), + "status warnings must contain `{needle}`: {json}" + ); + } + json +} + +fn find_session_by_agent<'a>(sessions: &'a [Value], agent_id: &str) -> &'a Value { + sessions + .iter() + .find(|session| session.pointer("/agent_id").and_then(Value::as_str) == Some(agent_id)) + .unwrap_or_else(|| panic!("expected session row for agent `{agent_id}`")) +} + +fn find_session_by_agent_and_orchestration_session<'a>( + sessions: &'a [Value], + agent_id: &str, + orchestration_session_id: &str, +) -> &'a Value { + sessions + .iter() + .find(|session| { + session.pointer("/agent_id").and_then(Value::as_str) == Some(agent_id) + && session + .pointer("/orchestration_session_id") + .and_then(Value::as_str) + == Some(orchestration_session_id) + }) + .unwrap_or_else(|| { + panic!( + "expected session row for agent `{agent_id}` in orchestration session `{orchestration_session_id}`" + ) + }) +} + +fn find_session_by_participant<'a>(sessions: &'a [Value], participant_id: &str) -> &'a Value { + sessions + .iter() + .find(|session| { + session.pointer("/participant_id").and_then(Value::as_str) == Some(participant_id) + }) + .unwrap_or_else(|| panic!("expected session row for participant `{participant_id}`")) +} + +fn find_text_session_line<'a>(stdout: &'a str, agent_id: &str) -> &'a str { + stdout + .lines() + .find(|line| line.contains(&format!("agent_id={agent_id}"))) + .unwrap_or_else(|| panic!("expected text session row for agent `{agent_id}`")) +} + +fn assert_substrings_in_order(haystack: &str, needles: &[&str]) { + let mut search_from = 0; + for needle in needles { + let relative = haystack[search_from..] + .find(needle) + .unwrap_or_else(|| panic!("expected `{needle}` in order within `{haystack}`")); + search_from += relative + needle.len(); + } +} + +fn seed_nested_gateway_status_fixture(fixture: &AgentSuccessorFixture) { + fixture.init_workspace(); + fixture.write_global_config_patch( + r#"agents: + enabled: true + hub: + orchestrator_agent_id: claude_code +"#, + ); + fixture.write_global_policy_patch( + r#"id: "ahcsitc2-policy" +name: "ahcsitc2-policy" + +world_fs: + host_visible: true + fail_closed: + routing: true + write: enabled: true agents: @@ -814,6 +1489,10 @@ fn agent_toolbox_status_json_reports_template_when_no_active_orchestrator_sessio .and_then(Value::as_str), Some("host") ); + assert!( + json.get("active_world_binding").is_none(), + "toolbox status must omit active_world_binding when no active session is live: {json}" + ); } #[test] @@ -835,9 +1514,7 @@ fn agent_toolbox_env_requires_an_active_orchestrator_session() { let stderr = String::from_utf8_lossy(&output.stderr); assert!( - stderr.contains( - "no active pure-agent orchestrator session found in authoritative live manifests" - ), + stderr.contains("no live host-scoped orchestrator participant found"), "stderr must explain the missing active session: {stderr}" ); } @@ -891,7 +1568,7 @@ fn agent_toolbox_env_trace_history_does_not_authorize_active_session() { let stderr = String::from_utf8_lossy(&output.stderr); assert!( - stderr.contains("authoritative live manifests"), + stderr.contains("no live host-scoped orchestrator participant found"), "trace-only active history must fail closed at the authoritative live manifest boundary: {stderr}" ); } @@ -908,6 +1585,13 @@ fn agent_toolbox_env_prefers_live_manifest_over_trace_fallback() { "ash_live_toolbox", "2026-04-06T00:00:02Z", ); + write_active_orchestration_session( + &fixture, + "claude_code", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6faa", + Some("ash_live_toolbox"), + "2026-04-06T00:00:02Z", + ); fixture.write_trace_events(&[json!({ "ts": "2026-04-06T00:00:03Z", "event_type": "agent_event", @@ -945,238 +1629,2052 @@ fn agent_toolbox_env_prefers_live_manifest_over_trace_fallback() { } #[test] -fn agent_toolbox_env_invalidated_manifest_and_trace_still_fail_closed() { +fn agent_status_degrades_but_toolbox_fails_closed_when_live_orchestrator_child_has_no_parent_session( +) { let fixture = AgentSuccessorFixture::new(); fixture.init_workspace(); fixture.seed_inventory_for_toolbox_contracts("uds"); - write_invalidated_runtime_manifest( + write_live_runtime_manifest( &fixture, "claude_code", - "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6faa", - "ash_invalidated_toolbox", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fac", + "ash_missing_parent", "2026-04-06T00:00:02Z", ); - fixture.write_trace_events(&[json!({ - "ts": "2026-04-06T00:00:03Z", - "event_type": "agent_event", - "session_id": "ses_agent_hub", - "component": "agent-hub", - "kind": "status", - "agent_id": "claude_code", - "orchestration_session_id": "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fab", - "run_id": "0195f8f1-7a35-7b7f-9c4d-9a7c2f5d6fac", - "backend_id": "cli:claude_code", - "client": "claude_code", - "router": "agent_hub", - "protocol": "uaa.agent.session", - "role": "orchestrator", - "data": { "message": "historical trace must not resurrect an invalidated session" } - })]); - let output = fixture.run(&["agent", "toolbox", "env", "--json"]); + let status = fixture.run(&["agent", "status", "--json"]); + let json = assert_status_succeeds_with_expected_warnings( + &status, + &["missing authoritative parent session metadata"], + ); + let sessions = json["sessions"] + .as_array() + .expect("sessions should be an array"); assert_eq!( - output.status.code(), - Some(3), - "invalidated manifests must not be resurrected by trace fallback: {output:?}" + sessions.len(), + 1, + "status should keep the visible participant: {json}" ); - assert!( - String::from_utf8_lossy(&output.stderr).contains("authoritative live manifests"), - "invalidated manifests must keep toolbox env behind the authoritative live-manifest contract" + assert_eq!( + find_session_by_agent(sessions, "claude_code") + .pointer("/participant_id") + .and_then(Value::as_str), + Some("ash_missing_parent") + ); + + assert_toolbox_parent_resolution_fail_closed_across_operator_surfaces( + &fixture, + &["live host-scoped orchestrator participant exists for agent claude_code without an active parent session"], ); } #[test] -fn agent_toolbox_status_json_reports_tcp_as_unsupported_pre_runtime() { +fn agent_status_stays_readable_but_toolbox_fails_closed_when_live_orchestrator_parent_is_inactive() +{ let fixture = AgentSuccessorFixture::new(); fixture.init_workspace(); - fixture.seed_inventory_for_toolbox_contracts("tcp"); + fixture.seed_inventory_for_toolbox_contracts("uds"); + write_live_runtime_manifest( + &fixture, + "claude_code", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fad", + "ash_inactive_parent", + "2026-04-06T00:00:02Z", + ); + write_inactive_orchestration_session( + &fixture, + "claude_code", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fad", + Some("ash_inactive_parent"), + "2026-04-06T00:00:02Z", + ); - let output = fixture.run(&["agent", "toolbox", "status", "--json"]); + let status = fixture.run(&["agent", "status", "--json"]); + let json = parse_json_output(&status); assert!( - output.status.success(), - "toolbox status should stay readable when tcp transport is selected: {output:?}" + status.status.success(), + "status should stay readable when only an inactive parent session remains: {status:?}" ); - - let json = parse_json_output(&output); assert_eq!( - json.pointer("/transport").and_then(Value::as_str), - Some("tcp") + json["warnings"].as_array().map(Vec::len), + Some(0), + "inactive parent records should not invent a degraded warning on agent status: {json}" ); + let sessions = json["sessions"] + .as_array() + .expect("sessions should be an array"); assert_eq!( - json.pointer("/eligibility/state").and_then(Value::as_str), - Some("unsupported") + sessions.len(), + 1, + "status should keep the visible participant: {json}" ); - let reason = json - .pointer("/eligibility/reason") - .and_then(Value::as_str) - .expect("unsupported tcp posture must publish a reason"); - assert!( - reason.contains("deterministic pre-runtime loopback port contract"), - "unsupported tcp status must explain the pre-runtime port-allocation gap: {json}" + assert_eq!( + find_session_by_agent(sessions, "claude_code") + .pointer("/participant_id") + .and_then(Value::as_str), + Some("ash_inactive_parent") + ); + + assert_toolbox_parent_resolution_fail_closed_across_operator_surfaces( + &fixture, + &["live host-scoped orchestrator participant exists for agent claude_code without an active parent session"], ); } #[test] -fn agent_status_preserves_member_roles_and_filters_them_by_contract_label() { +fn agent_status_degrades_but_toolbox_fails_closed_when_active_parent_omits_active_session_handle_id( +) { let fixture = AgentSuccessorFixture::new(); fixture.init_workspace(); - fixture.seed_inventory_for_list_and_status_contracts(); - fixture.write_trace_events(&[ - json!({ - "ts": "2026-04-05T00:00:00Z", - "event_type": "agent_event", - "session_id": "ses_agent_hub", - "component": "agent-hub", - "kind": "status", - "agent_id": "claude_code", - "orchestration_session_id": "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6f12", - "run_id": "0195f8f1-7a35-7b7f-9c4d-9a7c2f5d6f13", - "backend_id": "cli:claude_code", - "client": "claude_code", - "router": "agent_hub", - "protocol": "uaa.agent.session", - "role": "orchestrator", - "data": { "message": "orchestrator session is live" } - }), - json!({ - "ts": "2026-04-05T00:00:01Z", - "event_type": "agent_event", - "session_id": "ses_agent_hub", - "component": "agent-hub", - "kind": "status", - "agent_id": "codex", - "orchestration_session_id": "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6f12", - "run_id": "0195f8f1-7a35-7b7f-9c4d-9a7c2f5d6f14", - "backend_id": "cli:codex", - "client": "codex", - "router": "agent_hub", - "protocol": "uaa.agent.session", - "role": "member", - "world_id": "wld_active_0002", - "world_generation": 7, - "data": { "message": "member session is live" } - }), - ]); - - let output = fixture.run(&["agent", "status", "--json"]); - assert!( - output.status.success(), - "unfiltered agent status should succeed for member-role fixtures: {output:?}" + fixture.seed_inventory_for_toolbox_contracts("uds"); + write_live_runtime_manifest( + &fixture, + "claude_code", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fae", + "ash_missing_active_handle", + "2026-04-06T00:00:02Z", + ); + write_active_orchestration_session( + &fixture, + "claude_code", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fae", + None, + "2026-04-06T00:00:02Z", ); - let json = parse_json_output(&output); - assert!( - json.pointer("/role_filter") - .is_some_and(serde_json::Value::is_null), - "unfiltered status must publish a null role_filter: {json}" + let status = fixture.run(&["agent", "status", "--json"]); + let json = assert_status_succeeds_with_expected_warnings( + &status, + &["is missing authoritative orchestrator participant linkage"], ); let sessions = json["sessions"] .as_array() .expect("sessions should be an array"); assert_eq!( sessions.len(), - 2, - "unfiltered status should keep both orchestrator and member sessions: {json}" + 1, + "status should keep the visible participant: {json}" ); - - let orchestrator = find_session_by_agent(sessions, "claude_code"); assert_eq!( - orchestrator.pointer("/role").and_then(Value::as_str), - Some("orchestrator"), - "orchestrator session must preserve the orchestrator role label: {json}" + find_session_by_agent(sessions, "claude_code") + .pointer("/participant_id") + .and_then(Value::as_str), + Some("ash_missing_active_handle") ); - let member = find_session_by_agent(sessions, "codex"); - assert_eq!( - member.pointer("/role").and_then(Value::as_str), - Some("member"), - "member session must preserve the member role label: {json}" + assert_toolbox_parent_resolution_fail_closed_across_operator_surfaces( + &fixture, + &[ + "active orchestration session 0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fae is missing active_session_handle_id", + ], ); - assert_eq!( - member.pointer("/world_id").and_then(Value::as_str), - Some("wld_active_0002"), - "world-scoped member status rows must keep world_id: {json}" +} + +#[test] +fn agent_status_degrades_but_toolbox_fails_closed_when_active_parent_points_to_different_live_handle( +) { + let fixture = AgentSuccessorFixture::new(); + fixture.init_workspace(); + fixture.seed_inventory_for_toolbox_contracts("uds"); + write_live_runtime_manifest( + &fixture, + "claude_code", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6faf", + "ash_live_handle", + "2026-04-06T00:00:02Z", ); - assert_eq!( - member.pointer("/world_generation").and_then(Value::as_u64), - Some(7), - "world-scoped member status rows must keep world_generation: {json}" + write_active_orchestration_session( + &fixture, + "claude_code", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6faf", + Some("ash_other_handle"), + "2026-04-06T00:00:02Z", ); - let member_output = fixture.run(&["agent", "status", "--role", "member", "--json"]); - assert!( - member_output.status.success(), - "member-filtered agent status should succeed: {member_output:?}" + let status = fixture.run(&["agent", "status", "--json"]); + let json = assert_status_succeeds_with_expected_warnings( + &status, + &["authoritative participant ash_other_handle is missing from the session snapshot"], ); - let member_json = parse_json_output(&member_output); + let sessions = json["sessions"] + .as_array() + .expect("sessions should be an array"); + assert_eq!( + sessions.len(), + 1, + "status should keep the visible participant: {json}" + ); + assert_eq!( + find_session_by_agent(sessions, "claude_code") + .pointer("/participant_id") + .and_then(Value::as_str), + Some("ash_live_handle") + ); + + assert_toolbox_parent_resolution_fail_closed_across_operator_surfaces( + &fixture, + &[ + "active orchestration session 0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6faf references missing participant ash_other_handle", + ], + ); +} + +#[test] +fn agent_status_degrades_but_toolbox_fails_closed_when_active_parent_selects_inactive_participant() +{ + let fixture = AgentSuccessorFixture::new(); + fixture.init_workspace(); + fixture.seed_inventory_for_toolbox_contracts("uds"); + write_invalidated_runtime_manifest( + &fixture, + "claude_code", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fb0", + "ash_inactive_selected", + "2026-04-06T00:00:02Z", + ); + write_active_orchestration_session( + &fixture, + "claude_code", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fb0", + Some("ash_inactive_selected"), + "2026-04-06T00:00:02Z", + ); + + let status = fixture.run(&["agent", "status", "--json"]); + let json = assert_status_succeeds_with_expected_warnings( + &status, + &["active_attached session requires attached host participant truth"], + ); + let sessions = json["sessions"] + .as_array() + .expect("sessions should be an array"); + assert!( + sessions.is_empty(), + "status should omit incomplete orchestrator rows while surfacing the degraded warning: {json}" + ); + + assert_toolbox_parent_resolution_fail_closed_across_operator_surfaces( + &fixture, + &[ "active orchestration session 0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fb0 references inactive participant ash_inactive_selected" ], + ); +} + +#[test] +fn agent_status_keeps_multiple_active_parent_candidates_visible_but_toolbox_fails_closed() { + let fixture = AgentSuccessorFixture::new(); + fixture.init_workspace(); + fixture.seed_inventory_for_toolbox_contracts("uds"); + write_live_runtime_manifest( + &fixture, + "claude_code", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fb0", + "ash_parent_one", + "2026-04-06T00:00:02Z", + ); + write_active_orchestration_session( + &fixture, + "claude_code", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fb0", + Some("ash_parent_one"), + "2026-04-06T00:00:02Z", + ); + write_live_runtime_manifest( + &fixture, + "claude_code", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fb1", + "ash_parent_two", + "2026-04-06T00:00:03Z", + ); + write_active_orchestration_session( + &fixture, + "claude_code", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fb1", + Some("ash_parent_two"), + "2026-04-06T00:00:03Z", + ); + + let status = fixture.run(&["agent", "status", "--json"]); + assert!( + status.status.success(), + "status should stay readable when multiple active parent candidates exist: {status:?}" + ); + let json = parse_json_output(&status); + let sessions = json["sessions"] + .as_array() + .expect("sessions should be an array"); + let claude_rows: Vec<&Value> = sessions + .iter() + .filter(|session| { + session.pointer("/agent_id").and_then(Value::as_str) == Some("claude_code") + }) + .collect(); + assert_eq!( + claude_rows.len(), + 2, + "status should keep both active parent candidates visible on the read surface: {json}" + ); + assert_eq!( + find_session_by_participant(sessions, "ash_parent_one") + .pointer("/orchestration_session_id") + .and_then(Value::as_str), + Some("0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fb0") + ); + assert_eq!( + find_session_by_participant(sessions, "ash_parent_two") + .pointer("/orchestration_session_id") + .and_then(Value::as_str), + Some("0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fb1") + ); + + assert_toolbox_parent_resolution_fail_closed_across_operator_surfaces( + &fixture, + &["multiple active orchestration session candidates found for agent claude_code"], + ); +} + +#[test] +fn agent_toolbox_env_invalidated_manifest_and_trace_still_fail_closed() { + let fixture = AgentSuccessorFixture::new(); + fixture.init_workspace(); + fixture.seed_inventory_for_toolbox_contracts("uds"); + write_invalidated_runtime_manifest( + &fixture, + "claude_code", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6faa", + "ash_invalidated_toolbox", + "2026-04-06T00:00:02Z", + ); + fixture.write_trace_events(&[json!({ + "ts": "2026-04-06T00:00:03Z", + "event_type": "agent_event", + "session_id": "ses_agent_hub", + "component": "agent-hub", + "kind": "status", + "agent_id": "claude_code", + "orchestration_session_id": "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fab", + "run_id": "0195f8f1-7a35-7b7f-9c4d-9a7c2f5d6fac", + "backend_id": "cli:claude_code", + "client": "claude_code", + "router": "agent_hub", + "protocol": "uaa.agent.session", + "role": "orchestrator", + "data": { "message": "historical trace must not resurrect an invalidated session" } + })]); + + let output = fixture.run(&["agent", "toolbox", "env", "--json"]); + assert_eq!( + output.status.code(), + Some(3), + "invalidated manifests must not be resurrected by trace fallback: {output:?}" + ); + assert!( + String::from_utf8_lossy(&output.stderr) + .contains("no live host-scoped orchestrator participant found"), + "invalidated manifests must keep toolbox env behind the authoritative live-manifest contract" + ); +} + +#[test] +fn agent_toolbox_status_json_omits_active_world_binding_non_fatally_for_live_host_session() { + let fixture = AgentSuccessorFixture::new(); + fixture.init_workspace(); + fixture.seed_inventory_for_toolbox_contracts("uds"); + write_live_runtime_manifest( + &fixture, + "claude_code", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fb2", + "ash_live_no_binding", + "2026-04-06T00:00:02Z", + ); + write_active_orchestration_session( + &fixture, + "claude_code", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fb2", + Some("ash_live_no_binding"), + "2026-04-06T00:00:02Z", + ); + + let output = fixture.run(&["agent", "toolbox", "status", "--json"]); + assert!( + output.status.success(), + "toolbox status should remain readable when the live host session has no binding proof: {output:?}" + ); + + let json = parse_json_output(&output); + let expected = format!( + "unix://{}/run/agent-toolbox/0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fb2.sock", + fixture.substrate_home.display() + ); + assert_eq!( + json.pointer("/eligibility/state").and_then(Value::as_str), + Some("allowed") + ); + assert_eq!( + json.pointer("/active_orchestration_session_id") + .and_then(Value::as_str), + Some("0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fb2") + ); + assert_eq!( + json.pointer("/endpoint").and_then(Value::as_str), + Some(expected.as_str()) + ); + assert!( + json.get("active_world_binding").is_none(), + "toolbox status must keep active_world_binding optional for live host sessions without projected binding state: {json}" + ); +} + +#[test] +fn agent_toolbox_status_json_reports_active_world_binding_from_live_parent_session() { + let fixture = AgentSuccessorFixture::new(); + fixture.init_workspace(); + fixture.seed_inventory_for_toolbox_contracts("uds"); + write_live_runtime_manifest( + &fixture, + "claude_code", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fb3", + "ash_live_with_binding", + "2026-04-06T00:00:02Z", + ); + write_active_orchestration_session_with_world_binding( + &fixture, + "claude_code", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fb3", + Some("ash_live_with_binding"), + "2026-04-06T00:00:02Z", + "wld_active_0002", + 7, + ); + + let toolbox_output = fixture.run(&["agent", "toolbox", "status", "--json"]); + assert!( + toolbox_output.status.success(), + "toolbox status should surface the optional active binding proof without changing transport semantics: {toolbox_output:?}" + ); + + let toolbox_json = parse_json_output(&toolbox_output); + assert_eq!( + toolbox_json + .pointer("/active_world_binding/world_id") + .and_then(Value::as_str), + Some("wld_active_0002"), + "toolbox status must publish the active world_id proof from the authoritative parent session: {toolbox_json}" + ); + assert_eq!( + toolbox_json + .pointer("/active_world_binding/world_generation") + .and_then(Value::as_u64), + Some(7), + "toolbox status must publish the active world_generation proof from the authoritative parent session: {toolbox_json}" + ); +} + +#[test] +fn agent_toolbox_surfaces_prefer_canonical_session_roots_over_flat_compatibility_files() { + let fixture = AgentSuccessorFixture::new(); + fixture.init_workspace(); + fixture.seed_inventory_for_toolbox_contracts("uds"); + write_live_runtime_manifest( + &fixture, + "claude_code", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fb4", + "ash_canonical_toolbox", + "2026-04-06T00:00:02Z", + ); + write_active_orchestration_session( + &fixture, + "claude_code", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fb4", + Some("ash_canonical_toolbox"), + "2026-04-06T00:00:02Z", + ); + write_flat_runtime_participant_compatibility( + &fixture, + "ash_canonical_toolbox", + "claude_code", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6f99", + RuntimeParticipantOptions::host_orchestrator("invalidated", false, "2026-04-06T00:00:01Z"), + ); + write_flat_orchestration_session_compatibility( + &fixture, + "claude_code", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fb4", + Some("ash_flat_conflict"), + "stopped", + "2026-04-06T00:00:01Z", + None, + ); + + let status_output = fixture.run(&["agent", "toolbox", "status", "--json"]); + assert!( + status_output.status.success(), + "toolbox status must prefer the canonical session-root record over conflicting flat compatibility files: {status_output:?}" + ); + let status_json = parse_json_output(&status_output); + assert_eq!( + status_json + .pointer("/active_orchestration_session_id") + .and_then(Value::as_str), + Some("0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fb4") + ); + + let env_output = fixture.run(&["agent", "toolbox", "env", "--json"]); + assert!( + env_output.status.success(), + "toolbox env must resolve from the same canonical session-root record chosen by toolbox status: {env_output:?}" + ); + let env_json = parse_json_output(&env_output); + let expected_endpoint = format!( + "unix://{}/run/agent-toolbox/0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fb4.sock", + fixture.substrate_home.display() + ); + assert_eq!( + env_json + .pointer("/SUBSTRATE_AGENT_TOOLBOX_ENDPOINT") + .and_then(Value::as_str), + Some(expected_endpoint.as_str()), + "canonical participant and parent records must outrank conflicting flat compatibility files on both toolbox surfaces: {env_json}" + ); +} + +#[test] +fn agent_toolbox_surfaces_fall_back_to_flat_participant_when_canonical_root_is_incomplete() { + let fixture = AgentSuccessorFixture::new(); + fixture.init_workspace(); + fixture.seed_inventory_for_toolbox_contracts("uds"); + write_active_orchestration_session( + &fixture, + "claude_code", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fb5", + Some("ash_flat_fallback"), + "2026-04-06T00:00:02Z", + ); + write_flat_runtime_participant_compatibility( + &fixture, + "ash_flat_fallback", + "claude_code", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fb5", + RuntimeParticipantOptions::host_orchestrator("running", true, "2026-04-06T00:00:02Z"), + ); + + let status_output = fixture.run(&["agent", "toolbox", "status", "--json"]); + assert!( + status_output.status.success(), + "toolbox status must keep the flat compatibility participant fallback when the canonical session root is parent-only: {status_output:?}" + ); + let status_json = parse_json_output(&status_output); + assert_eq!( + status_json + .pointer("/active_orchestration_session_id") + .and_then(Value::as_str), + Some("0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fb5") + ); + + let env_output = fixture.run(&["agent", "toolbox", "env", "--json"]); + assert!( + env_output.status.success(), + "toolbox env must share the same flat compatibility fallback when the canonical participant is missing: {env_output:?}" + ); + let env_json = parse_json_output(&env_output); + let expected_endpoint = format!( + "unix://{}/run/agent-toolbox/0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fb5.sock", + fixture.substrate_home.display() + ); + assert_eq!( + env_json + .pointer("/SUBSTRATE_AGENT_TOOLBOX_ENDPOINT") + .and_then(Value::as_str), + Some(expected_endpoint.as_str()), + "flat compatibility participant fallback must stay readable when the canonical root is incomplete: {env_json}" + ); +} + +#[test] +fn agent_toolbox_surfaces_stay_orchestrator_anchored_when_live_member_exists() { + let fixture = AgentSuccessorFixture::new(); + fixture.init_workspace(); + fixture.seed_inventory_for_toolbox_contracts("uds"); + write_live_runtime_manifest( + &fixture, + "claude_code", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fb7", + "ash_toolbox_parent_live", + "2026-04-06T00:00:02Z", + ); + write_active_orchestration_session_with_world_binding( + &fixture, + "claude_code", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fb7", + Some("ash_toolbox_parent_live"), + "2026-04-06T00:00:02Z", + "wld_parent_binding_0007", + 12, + ); + write_runtime_participant( + &fixture, + "ash_toolbox_member_live", + "codex", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fb7", + RuntimeParticipantOptions::world_member( + "running", + true, + "2026-04-06T00:00:03Z", + "wld_member_runtime_9999", + 2, + "ash_toolbox_parent_live", + ), + ); + + let status_output = fixture.run(&["agent", "toolbox", "status", "--json"]); + assert!( + status_output.status.success(), + "toolbox status must stay readable when a live world member exists beside the orchestrator: {status_output:?}" + ); + let status_json = parse_json_output(&status_output); + assert_eq!( + status_json + .pointer("/active_orchestration_session_id") + .and_then(Value::as_str), + Some("0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fb7") + ); + assert_eq!( + status_json + .pointer("/orchestrator/agent_id") + .and_then(Value::as_str), + Some("claude_code"), + "toolbox status must stay anchored to the selected host orchestrator even when a member is live: {status_json}" + ); + assert_eq!( + status_json + .pointer("/orchestrator/execution/scope") + .and_then(Value::as_str), + Some("host") + ); + assert_eq!( + status_json + .pointer("/active_world_binding/world_id") + .and_then(Value::as_str), + Some("wld_parent_binding_0007"), + "toolbox status must derive active world binding from the parent orchestration session, not the member runtime row: {status_json}" + ); + assert_eq!( + status_json + .pointer("/active_world_binding/world_generation") + .and_then(Value::as_u64), + Some(12) + ); + + let env_output = fixture.run(&["agent", "toolbox", "env", "--json"]); + assert!( + env_output.status.success(), + "toolbox env must remain anchored to the orchestrator session when a live member exists: {env_output:?}" + ); + let env_json = parse_json_output(&env_output); + let expected_endpoint = format!( + "unix://{}/run/agent-toolbox/0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fb7.sock", + fixture.substrate_home.display() + ); + assert_eq!( + env_json + .pointer("/SUBSTRATE_AGENT_TOOLBOX_ENDPOINT") + .and_then(Value::as_str), + Some(expected_endpoint.as_str()), + "toolbox env must keep using the orchestrator-scoped endpoint contract even when the member runtime is live: {env_json}" + ); + let env_body = serde_json::to_string(&env_json).expect("serialize toolbox env json"); + assert!( + !env_body.contains("ash_toolbox_member_live") + && !env_body.contains("wld_member_runtime_9999"), + "toolbox env must not pivot to member-scoped participant or world identity fields: {env_json}" + ); +} + +#[test] +fn agent_status_selected_host_row_stays_unchanged_when_parent_session_has_world_binding() { + let fixture = AgentSuccessorFixture::new(); + fixture.init_workspace(); + fixture.seed_inventory_for_toolbox_contracts("uds"); + write_live_runtime_manifest( + &fixture, + "claude_code", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fb3", + "ash_live_with_binding", + "2026-04-06T00:00:02Z", + ); + write_active_orchestration_session_with_world_binding( + &fixture, + "claude_code", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fb3", + Some("ash_live_with_binding"), + "2026-04-06T00:00:02Z", + "wld_active_0002", + 7, + ); + + let status_output = fixture.run(&["agent", "status", "--json"]); + assert!( + status_output.status.success(), + "selected host status rows must remain readable when the parent session has world binding state: {status_output:?}" + ); + + let status_json = parse_json_output(&status_output); + let sessions = status_json["sessions"] + .as_array() + .expect("sessions should be an array"); + let orchestrator = find_session_by_agent(sessions, "claude_code"); + assert_eq!( + orchestrator.pointer("/execution/scope").and_then(Value::as_str), + Some("host"), + "selected orchestrator rows must stay host-scoped even when the parent session carries world binding state: {status_json}" + ); + assert!( + orchestrator.get("world_id").is_none(), + "selected host status rows must keep omitting world_id: {status_json}" + ); + assert!( + orchestrator.get("world_generation").is_none(), + "selected host status rows must keep omitting world_generation: {status_json}" + ); +} + +#[test] +fn agent_toolbox_status_json_reports_tcp_as_unsupported_pre_runtime() { + let fixture = AgentSuccessorFixture::new(); + fixture.init_workspace(); + fixture.seed_inventory_for_toolbox_contracts("tcp"); + + let output = fixture.run(&["agent", "toolbox", "status", "--json"]); + assert!( + output.status.success(), + "toolbox status should stay readable when tcp transport is selected: {output:?}" + ); + + let json = parse_json_output(&output); + assert_eq!( + json.pointer("/transport").and_then(Value::as_str), + Some("tcp") + ); + assert_eq!( + json.pointer("/eligibility/state").and_then(Value::as_str), + Some("unsupported") + ); + let reason = json + .pointer("/eligibility/reason") + .and_then(Value::as_str) + .expect("unsupported tcp posture must publish a reason"); + assert!( + reason.contains("deterministic pre-runtime loopback port contract"), + "unsupported tcp status must explain the pre-runtime port-allocation gap: {json}" + ); +} + +#[test] +fn agent_status_preserves_member_roles_and_filters_them_by_contract_label() { + let fixture = AgentSuccessorFixture::new(); + fixture.init_workspace(); + fixture.seed_inventory_for_list_and_status_contracts(); + fixture.write_trace_events(&[ + json!({ + "ts": "2026-04-05T00:00:00Z", + "event_type": "agent_event", + "session_id": "ses_agent_hub", + "component": "agent-hub", + "kind": "status", + "agent_id": "claude_code", + "orchestration_session_id": "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6f12", + "run_id": "0195f8f1-7a35-7b7f-9c4d-9a7c2f5d6f13", + "backend_id": "cli:claude_code", + "client": "claude_code", + "router": "agent_hub", + "protocol": "uaa.agent.session", + "role": "orchestrator", + "data": { "message": "orchestrator session is live" } + }), + json!({ + "ts": "2026-04-05T00:00:01Z", + "event_type": "agent_event", + "session_id": "ses_agent_hub", + "component": "agent-hub", + "kind": "status", + "agent_id": "codex", + "orchestration_session_id": "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6f12", + "run_id": "0195f8f1-7a35-7b7f-9c4d-9a7c2f5d6f14", + "backend_id": "cli:codex", + "client": "codex", + "router": "agent_hub", + "protocol": "uaa.agent.session", + "role": "member", + "world_id": "wld_active_0002", + "world_generation": 7, + "data": { "message": "member session is live" } + }), + ]); + + let output = fixture.run(&["agent", "status", "--json"]); + assert!( + output.status.success(), + "unfiltered agent status should succeed for member-role fixtures: {output:?}" + ); + + let json = parse_json_output(&output); + assert!( + json.pointer("/role_filter") + .is_some_and(serde_json::Value::is_null), + "unfiltered status must publish a null role_filter: {json}" + ); + let sessions = json["sessions"] + .as_array() + .expect("sessions should be an array"); + assert_eq!( + sessions.len(), + 2, + "unfiltered status should keep both orchestrator and member sessions: {json}" + ); + + let orchestrator = find_session_by_agent(sessions, "claude_code"); + assert_eq!( + orchestrator.pointer("/role").and_then(Value::as_str), + Some("orchestrator"), + "orchestrator session must preserve the orchestrator role label: {json}" + ); + + let member = find_session_by_agent(sessions, "codex"); + assert_eq!( + member.pointer("/role").and_then(Value::as_str), + Some("member"), + "member session must preserve the member role label: {json}" + ); + assert_eq!( + member.pointer("/world_id").and_then(Value::as_str), + Some("wld_active_0002"), + "world-scoped member status rows must keep world_id: {json}" + ); + assert_eq!( + member.pointer("/world_generation").and_then(Value::as_u64), + Some(7), + "world-scoped member status rows must keep world_generation: {json}" + ); + + let member_output = fixture.run(&["agent", "status", "--role", "member", "--json"]); + assert!( + member_output.status.success(), + "member-filtered agent status should succeed: {member_output:?}" + ); + let member_json = parse_json_output(&member_output); assert_eq!( member_json.pointer("/role_filter").and_then(Value::as_str), Some("member"), "member-filtered status must report role_filter=member: {member_json}" ); - let member_sessions = member_json["sessions"] + let member_sessions = member_json["sessions"] + .as_array() + .expect("sessions should be an array"); + assert_eq!( + member_sessions.len(), + 1, + "--role member should return exactly one member session: {member_json}" + ); + let member_only = find_session_by_agent(member_sessions, "codex"); + assert_eq!( + member_only.pointer("/role").and_then(Value::as_str), + Some("member") + ); + assert_eq!( + member_only.pointer("/world_id").and_then(Value::as_str), + Some("wld_active_0002") + ); + assert_eq!( + member_only + .pointer("/world_generation") + .and_then(Value::as_u64), + Some(7) + ); + + let orchestrator_output = fixture.run(&["agent", "status", "--role", "orchestrator", "--json"]); + assert!( + orchestrator_output.status.success(), + "orchestrator-filtered agent status should succeed: {orchestrator_output:?}" + ); + let orchestrator_json = parse_json_output(&orchestrator_output); + assert_eq!( + orchestrator_json + .pointer("/role_filter") + .and_then(Value::as_str), + Some("orchestrator"), + "orchestrator-filtered status must report role_filter=orchestrator: {orchestrator_json}" + ); + let orchestrator_sessions = orchestrator_json["sessions"] + .as_array() + .expect("sessions should be an array"); + assert_eq!( + orchestrator_sessions.len(), + 1, + "--role orchestrator should return exactly one orchestrator session: {orchestrator_json}" + ); + let orchestrator_only = find_session_by_agent(orchestrator_sessions, "claude_code"); + assert_eq!( + orchestrator_only.pointer("/role").and_then(Value::as_str), + Some("orchestrator") + ); +} + +#[test] +fn agent_status_prefers_live_manifest_over_trace_fallback_for_selected_orchestrator() { + let fixture = AgentSuccessorFixture::new(); + fixture.init_workspace(); + fixture.seed_inventory_for_list_and_status_contracts(); + write_live_runtime_manifest( + &fixture, + "claude_code", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6faa", + "ash_live_status", + "2026-04-05T00:00:02Z", + ); + write_active_orchestration_session( + &fixture, + "claude_code", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6faa", + Some("ash_live_status"), + "2026-04-05T00:00:02Z", + ); + fixture.write_trace_events(&[json!({ + "ts": "2026-04-05T00:00:03Z", + "event_type": "agent_event", + "session_id": "ses_agent_hub", + "component": "agent-hub", + "kind": "status", + "agent_id": "claude_code", + "orchestration_session_id": "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fab", + "run_id": "0195f8f1-7a35-7b7f-9c4d-9a7c2f5d6fac", + "backend_id": "cli:claude_code", + "client": "claude_code", + "router": "agent_hub", + "protocol": "uaa.agent.session", + "role": "orchestrator", + "data": { "message": "trace fallback should lose to live manifest" } + })]); + + let output = fixture.run(&["agent", "status", "--json"]); + assert!( + output.status.success(), + "status should prefer the live manifest: {output:?}" + ); + + let json = parse_json_output(&output); + let sessions = json["sessions"] + .as_array() + .expect("sessions should be an array"); + let orchestrator = find_session_by_agent(sessions, "claude_code"); + assert_eq!( + orchestrator + .pointer("/orchestration_session_id") + .and_then(Value::as_str), + Some("0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6faa"), + "status must prefer live manifest state over trace fallback for the selected orchestrator: {json}" + ); +} + +#[test] +fn agent_status_json_surfaces_durable_session_fields_for_live_runtime_rows() { + let fixture = AgentSuccessorFixture::new(); + fixture.init_workspace(); + fixture.seed_inventory_for_list_and_status_contracts(); + write_live_runtime_manifest( + &fixture, + "claude_code", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fb9", + "ash_live_runtime_fields", + "2026-04-05T00:00:02Z", + ); + write_active_orchestration_session( + &fixture, + "claude_code", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fb9", + Some("ash_live_runtime_fields"), + "2026-04-05T00:00:02Z", + ); + + let output = fixture.run(&["agent", "status", "--json"]); + assert!( + output.status.success(), + "agent status should surface live runtime durable session fields: {output:?}" + ); + + let json = parse_json_output(&output); + let sessions = json["sessions"] + .as_array() + .expect("sessions should be an array"); + let live_row = find_session_by_participant(sessions, "ash_live_runtime_fields"); + assert_eq!( + live_row.pointer("/source_kind").and_then(Value::as_str), + Some("live_runtime") + ); + assert_eq!( + live_row.pointer("/posture").and_then(Value::as_str), + Some("active_attached") + ); + assert_eq!( + live_row + .pointer("/attached_participant_id") + .and_then(Value::as_str), + Some("ash_live_runtime_fields") + ); + assert_eq!( + live_row + .pointer("/pending_inbox_count") + .and_then(Value::as_u64), + Some(0) + ); +} + +#[test] +fn agent_status_json_surfaces_parked_resumable_fields_from_parent_session_truth() { + let fixture = AgentSuccessorFixture::new(); + fixture.init_workspace(); + fixture.seed_inventory_for_list_and_status_contracts(); + write_runtime_participant( + &fixture, + "ash_parked_detached", + "claude_code", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fba", + RuntimeParticipantOptions::host_orchestrator("running", false, "2026-04-05T00:00:02Z"), + ); + write_orchestration_session_with_manifest_options( + &fixture, + OrchestrationSessionManifestSpec { + agent_id: "claude_code", + orchestration_session_id: "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fba", + active_session_handle_id: Some("ash_parked_detached"), + state: "active", + ts: "2026-04-05T00:00:02Z", + world_binding: None, + }, + OrchestrationSessionManifestOptions { + posture: Some("parked_resumable"), + attached_participant_id: Some(None), + pending_inbox_count: Some(0), + last_parked_at: Some(Some("2026-04-05T00:00:02Z")), + last_attention_at: Some(None), + parked_reason: Some(Some("owner detached cleanly")), + }, + ); + + let output = fixture.run(&["agent", "status", "--json"]); + assert!( + output.status.success(), + "agent status should keep detached parked sessions visible: {output:?}" + ); + + let json = parse_json_output(&output); + let sessions = json["sessions"] + .as_array() + .expect("sessions should be an array"); + let parked = find_session_by_participant(sessions, "ash_parked_detached"); + assert_eq!( + parked.pointer("/source_kind").and_then(Value::as_str), + Some("live_runtime") + ); + assert_eq!( + parked.pointer("/posture").and_then(Value::as_str), + Some("parked_resumable") + ); + assert!( + parked + .pointer("/attached_participant_id") + .is_some_and(Value::is_null), + "parked detached rows must preserve null attached_participant_id from parent truth: {parked}" + ); + assert_eq!( + parked + .pointer("/pending_inbox_count") + .and_then(Value::as_u64), + Some(0) + ); +} + +#[test] +fn agent_status_json_surfaces_awaiting_attention_fields_from_parent_session_truth() { + let fixture = AgentSuccessorFixture::new(); + fixture.init_workspace(); + fixture.seed_inventory_for_list_and_status_contracts(); + write_runtime_participant( + &fixture, + "ash_attention_detached", + "claude_code", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fbb", + RuntimeParticipantOptions::host_orchestrator("running", false, "2026-04-05T00:00:03Z"), + ); + write_orchestration_session_with_manifest_options( + &fixture, + OrchestrationSessionManifestSpec { + agent_id: "claude_code", + orchestration_session_id: "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fbb", + active_session_handle_id: Some("ash_attention_detached"), + state: "active", + ts: "2026-04-05T00:00:03Z", + world_binding: None, + }, + OrchestrationSessionManifestOptions { + posture: Some("awaiting_attention"), + attached_participant_id: Some(None), + pending_inbox_count: Some(2), + last_parked_at: Some(None), + last_attention_at: Some(Some("2026-04-05T00:00:03Z")), + parked_reason: Some(None), + }, + ); + + let output = fixture.run(&["agent", "status", "--json"]); + assert!( + output.status.success(), + "agent status should keep detached awaiting-attention sessions visible: {output:?}" + ); + + let json = parse_json_output(&output); + let sessions = json["sessions"] + .as_array() + .expect("sessions should be an array"); + let attention = find_session_by_participant(sessions, "ash_attention_detached"); + assert_eq!( + attention.pointer("/source_kind").and_then(Value::as_str), + Some("live_runtime") + ); + assert_eq!( + attention.pointer("/posture").and_then(Value::as_str), + Some("awaiting_attention") + ); + assert!( + attention + .pointer("/attached_participant_id") + .is_some_and(Value::is_null), + "awaiting-attention rows must preserve null attached_participant_id from parent truth: {attention}" + ); + assert_eq!( + attention + .pointer("/pending_inbox_count") + .and_then(Value::as_u64), + Some(2) + ); +} + +#[test] +fn agent_status_json_trace_fallback_rows_emit_explicit_null_posture_fields() { + let fixture = AgentSuccessorFixture::new(); + fixture.init_workspace(); + fixture.seed_inventory_for_list_and_status_contracts(); + fixture.write_trace_events(&[json!({ + "ts": "2026-04-05T00:00:04Z", + "event_type": "agent_event", + "session_id": "ses_agent_hub", + "component": "agent-hub", + "kind": "status", + "agent_id": "claude_code", + "orchestration_session_id": "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fbc", + "participant_id": "ash_trace_only", + "run_id": "0195f8f1-7a35-7b7f-9c4d-9a7c2f5d6fbd", + "backend_id": "cli:claude_code", + "client": "claude_code", + "router": "agent_hub", + "protocol": "uaa.agent.session", + "role": "orchestrator", + "data": { "message": "trace fallback row should keep durable posture fields null" } + })]); + + let output = fixture.run(&["agent", "status", "--json"]); + assert!( + output.status.success(), + "agent status should keep trace-fallback rows readable: {output:?}" + ); + + let json = parse_json_output(&output); + let sessions = json["sessions"] + .as_array() + .expect("sessions should be an array"); + let fallback = find_session_by_participant(sessions, "ash_trace_only"); + assert_eq!( + fallback.pointer("/source_kind").and_then(Value::as_str), + Some("trace_fallback") + ); + assert!( + fallback.pointer("/posture").is_some_and(Value::is_null), + "trace fallback rows must publish posture as explicit null: {fallback}" + ); + assert!( + fallback + .pointer("/attached_participant_id") + .is_some_and(Value::is_null), + "trace fallback rows must publish attached_participant_id as explicit null: {fallback}" + ); + assert!( + fallback + .pointer("/pending_inbox_count") + .is_some_and(Value::is_null), + "trace fallback rows must publish pending_inbox_count as explicit null: {fallback}" + ); +} + +#[test] +fn agent_status_human_output_includes_durable_session_fields_for_live_rows() { + let fixture = AgentSuccessorFixture::new(); + fixture.init_workspace(); + fixture.seed_inventory_for_list_and_status_contracts(); + write_live_runtime_manifest( + &fixture, + "claude_code", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fbe", + "ash_text_live", + "2026-04-05T00:00:05Z", + ); + write_active_orchestration_session( + &fixture, + "claude_code", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fbe", + Some("ash_text_live"), + "2026-04-05T00:00:05Z", + ); + + let output = fixture.run(&["agent", "status"]); + assert!( + output.status.success(), + "agent status text output should succeed for live runtime rows: {output:?}" + ); + + let stdout = String::from_utf8_lossy(&output.stdout); + let session_line = find_text_session_line(&stdout, "claude_code"); + assert_substrings_in_order( + session_line, + &[ + "orchestration_session_id=0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fbe", + "participant_id=ash_text_live", + "agent_id=claude_code", + "source_kind=live_runtime", + "backend_id=cli:claude_code", + "client=claude_code", + "router=agent_hub", + "protocol=uaa.agent.session", + "execution.scope=host", + "role=orchestrator", + "posture=active_attached", + "attached_participant_id=ash_text_live", + "pending_inbox_count=0", + "last_event_at=2026-04-05T00:00:05+00:00", + ], + ); +} + +#[test] +fn agent_status_human_output_marks_fallback_rows_unknown_for_durable_session_fields() { + let fixture = AgentSuccessorFixture::new(); + fixture.init_workspace(); + fixture.seed_inventory_for_list_and_status_contracts(); + fixture.write_trace_events(&[json!({ + "ts": "2026-04-05T00:00:06Z", + "event_type": "agent_event", + "session_id": "ses_agent_hub", + "component": "agent-hub", + "kind": "status", + "agent_id": "claude_code", + "orchestration_session_id": "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fbf", + "participant_id": "ash_text_trace", + "run_id": "0195f8f1-7a35-7b7f-9c4d-9a7c2f5d6fc0", + "backend_id": "cli:claude_code", + "client": "claude_code", + "router": "agent_hub", + "protocol": "uaa.agent.session", + "role": "orchestrator", + "data": { "message": "trace fallback row should keep text durable posture fields unknown" } + })]); + + let output = fixture.run(&["agent", "status"]); + assert!( + output.status.success(), + "agent status text output should succeed for trace fallback rows: {output:?}" + ); + + let stdout = String::from_utf8_lossy(&output.stdout); + let session_line = find_text_session_line(&stdout, "claude_code"); + assert_substrings_in_order( + session_line, + &[ + "orchestration_session_id=0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fbf", + "participant_id=ash_text_trace", + "agent_id=claude_code", + "source_kind=trace_fallback", + "backend_id=cli:claude_code", + "client=claude_code", + "router=agent_hub", + "protocol=uaa.agent.session", + "execution.scope=host", + "role=orchestrator", + "posture=", + "attached_participant_id=", + "pending_inbox_count=", + "last_event_at=2026-04-05T00:00:06+00:00", + ], + ); +} + +#[test] +fn agent_status_tombstone_suppression_beats_stale_trace_fallback_for_world_member() { + let fixture = AgentSuccessorFixture::new(); + fixture.init_workspace(); + fixture.seed_inventory_for_list_and_status_contracts(); + write_invalidated_world_member_manifest( + &fixture, + "codex", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fb0", + "ash_codex_old", + "ash_orchestrator", + "wld_old_0001", + 6, + None, + "2026-04-05T00:00:02Z", + ); + fixture.write_trace_events(&[json!({ + "ts": "2026-04-05T00:00:03Z", + "event_type": "agent_event", + "session_id": "ses_agent_hub", + "component": "agent-hub", + "kind": "status", + "agent_id": "codex", + "orchestration_session_id": "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fb0", + "run_id": "0195f8f1-7a35-7b7f-9c4d-9a7c2f5d6fb1", + "backend_id": "cli:codex", + "client": "codex", + "router": "agent_hub", + "protocol": "uaa.agent.session", + "role": "member", + "world_id": "wld_old_0001", + "world_generation": 6, + "data": { "message": "stale trace row must not beat the authoritative participant tombstone" } + })]); + + let output = fixture.run(&["agent", "status", "--scope", "world", "--json"]); + assert!( + output.status.success(), + "world status should succeed when tombstones suppress stale trace fallback: {output:?}" + ); + + let json = parse_json_output(&output); + let sessions = json["sessions"] + .as_array() + .expect("sessions should be an array"); + assert!( + sessions.is_empty(), + "authoritative participant tombstones must suppress stale trace fallback for the same (orchestration_session_id, agent_id, execution.scope) tuple: {json}" + ); +} + +#[test] +fn agent_status_omits_invalidated_world_member_until_replacement_persists() { + let fixture = AgentSuccessorFixture::new(); + fixture.init_workspace(); + fixture.seed_inventory_for_list_and_status_contracts(); + write_live_runtime_manifest( + &fixture, + "claude_code", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fb2", + "ash_orchestrator_live", + "2026-04-05T00:00:01Z", + ); + write_active_orchestration_session( + &fixture, + "claude_code", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fb2", + Some("ash_orchestrator_live"), + "2026-04-05T00:00:01Z", + ); + write_invalidated_world_member_manifest( + &fixture, + "codex", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fb2", + "ash_codex_missing_replacement", + "ash_orchestrator_live", + "wld_old_0002", + 6, + None, + "2026-04-05T00:00:02Z", + ); + + let output = fixture.run(&["agent", "status", "--json"]); + assert!( + output.status.success(), + "status should succeed when a replacement participant has not been persisted yet: {output:?}" + ); + + let json = parse_json_output(&output); + let sessions = json["sessions"] + .as_array() + .expect("sessions should be an array"); + assert_eq!( + sessions.len(), + 1, + "missing replacements must leave omission rather than stale liveness on the selected status surface: {json}" + ); + let orchestrator = find_session_by_agent(sessions, "claude_code"); + assert_eq!( + orchestrator + .pointer("/participant_id") + .and_then(Value::as_str), + Some("ash_orchestrator_live") + ); + assert!( + sessions + .iter() + .all(|session| session.pointer("/agent_id").and_then(Value::as_str) != Some("codex")), + "invalidated members must stay absent until a replacement participant is persisted: {json}" + ); +} + +#[test] +fn agent_status_keeps_same_agent_concurrent_sessions_visible_across_orchestration_sessions() { + let fixture = AgentSuccessorFixture::new(); + fixture.init_workspace(); + fixture.seed_inventory_for_list_and_status_contracts(); + write_live_runtime_manifest( + &fixture, + "claude_code", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6faa", + "ash_live_status_suppression", + "2026-04-05T00:00:02Z", + ); + write_active_orchestration_session( + &fixture, + "claude_code", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6faa", + Some("ash_live_status_suppression"), + "2026-04-05T00:00:02Z", + ); + fixture.write_trace_events(&[json!({ + "ts": "2026-04-05T00:00:01Z", + "event_type": "agent_event", + "session_id": "ses_agent_hub", + "component": "agent-hub", + "kind": "status", + "agent_id": "claude_code", + "orchestration_session_id": "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fab", + "run_id": "0195f8f1-7a35-7b7f-9c4d-9a7c2f5d6fac", + "backend_id": "cli:claude_code", + "client": "claude_code", + "router": "agent_hub", + "protocol": "uaa.agent.session", + "role": "orchestrator", + "data": { "message": "distinct orchestration_session_id values must remain independently visible" } + })]); + + let output = fixture.run(&["agent", "status", "--json"]); + assert!( + output.status.success(), + "status should succeed when the same agent is visible in concurrent sessions: {output:?}" + ); + + let json = parse_json_output(&output); + let sessions = json["sessions"] + .as_array() + .expect("sessions should be an array"); + let claude_rows: Vec<&Value> = sessions + .iter() + .filter(|session| { + session.pointer("/agent_id").and_then(Value::as_str) == Some("claude_code") + }) + .collect(); + assert_eq!( + claude_rows.len(), + 2, + "suppression identity must stay scoped to (orchestration_session_id, agent_id, execution.scope) so same-agent concurrent sessions remain independently visible: {json}" + ); + assert_eq!( + find_session_by_agent_and_orchestration_session( + sessions, + "claude_code", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6faa", + ) + .pointer("/orchestration_session_id") + .and_then(Value::as_str), + Some("0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6faa") + ); + assert_eq!( + find_session_by_agent_and_orchestration_session( + sessions, + "claude_code", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fab", + ) + .pointer("/orchestration_session_id") + .and_then(Value::as_str), + Some("0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fab") + ); +} + +#[test] +fn agent_status_keeps_same_agent_concurrent_live_sessions_visible_from_session_records() { + let fixture = AgentSuccessorFixture::new(); + fixture.init_workspace(); + fixture.seed_inventory_for_list_and_status_contracts(); + write_live_runtime_manifest( + &fixture, + "claude_code", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fc1", + "ash_orchestrator_session_one", + "2026-04-05T00:00:02Z", + ); + write_active_orchestration_session( + &fixture, + "claude_code", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fc1", + Some("ash_orchestrator_session_one"), + "2026-04-05T00:00:02Z", + ); + write_runtime_participant( + &fixture, + "ash_codex_session_one", + "codex", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fc1", + RuntimeParticipantOptions::world_member( + "running", + true, + "2026-04-05T00:00:02Z", + "wld_live_member_0001", + 7, + "ash_orchestrator_session_one", + ), + ); + write_live_runtime_manifest( + &fixture, + "claude_code", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fc2", + "ash_orchestrator_session_two", + "2026-04-05T00:00:03Z", + ); + write_active_orchestration_session( + &fixture, + "claude_code", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fc2", + Some("ash_orchestrator_session_two"), + "2026-04-05T00:00:03Z", + ); + write_runtime_participant( + &fixture, + "ash_codex_session_two", + "codex", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fc2", + RuntimeParticipantOptions::world_member( + "running", + true, + "2026-04-05T00:00:03Z", + "wld_live_member_0002", + 8, + "ash_orchestrator_session_two", + ), + ); + + let output = fixture.run(&["agent", "status", "--scope", "world", "--json"]); + assert!( + output.status.success(), + "agent status must keep same-agent concurrent live sessions visible when both rows come from store-owned session records: {output:?}" + ); + + let json = parse_json_output(&output); + let sessions = json["sessions"] + .as_array() + .expect("sessions should be an array"); + let codex_rows: Vec<&Value> = sessions + .iter() + .filter(|session| session.pointer("/agent_id").and_then(Value::as_str) == Some("codex")) + .collect(); + assert_eq!( + codex_rows.len(), + 2, + "same-agent concurrent visibility must survive the session-record regrouping cutover when both parent sessions are selected candidates: {json}" + ); + assert_eq!( + find_session_by_agent_and_orchestration_session( + sessions, + "codex", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fc1", + ) + .pointer("/participant_id") + .and_then(Value::as_str), + Some("ash_codex_session_one") + ); + assert_eq!( + find_session_by_agent_and_orchestration_session( + sessions, + "codex", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fc2", + ) + .pointer("/participant_id") + .and_then(Value::as_str), + Some("ash_codex_session_two") + ); +} + +#[test] +fn agent_status_trace_only_participant_aware_fallback_keeps_same_agent_siblings_visible() { + let fixture = AgentSuccessorFixture::new(); + fixture.init_workspace(); + fixture.seed_inventory_for_list_and_status_contracts(); + fixture.write_trace_events(&[ + json!({ + "ts": "2026-04-05T00:00:00Z", + "event_type": "agent_event", + "session_id": "ses_agent_hub", + "component": "agent-hub", + "kind": "status", + "agent_id": "codex", + "orchestration_session_id": "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fd1", + "participant_id": "ash_trace_codex_one", + "run_id": "0195f8f1-7a35-7b7f-9c4d-9a7c2f5d6fd2", + "backend_id": "cli:codex", + "client": "codex", + "router": "agent_hub", + "protocol": "uaa.agent.session", + "role": "member", + "world_id": "wld_trace_0001", + "world_generation": 9, + "data": { "message": "first trace-only sibling remains visible" } + }), + json!({ + "ts": "2026-04-05T00:00:01Z", + "event_type": "agent_event", + "session_id": "ses_agent_hub", + "component": "agent-hub", + "kind": "status", + "agent_id": "codex", + "orchestration_session_id": "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fd1", + "participant_id": "ash_trace_codex_two", + "run_id": "0195f8f1-7a35-7b7f-9c4d-9a7c2f5d6fd3", + "backend_id": "cli:codex", + "client": "codex", + "router": "agent_hub", + "protocol": "uaa.agent.session", + "role": "member", + "world_id": "wld_trace_0001", + "world_generation": 9, + "data": { "message": "second trace-only sibling remains visible" } + }), + ]); + + let output = fixture.run(&["agent", "status", "--scope", "world", "--json"]); + assert!( + output.status.success(), + "world status should preserve participant-scoped trace siblings: {output:?}" + ); + + let json = parse_json_output(&output); + let sessions = json["sessions"] .as_array() .expect("sessions should be an array"); + let codex_rows: Vec<&Value> = sessions + .iter() + .filter(|session| session.pointer("/agent_id").and_then(Value::as_str) == Some("codex")) + .collect(); assert_eq!( - member_sessions.len(), + codex_rows.len(), + 2, + "participant-aware trace fallback must keep same-agent siblings in one orchestration session distinct: {json}" + ); + assert_eq!( + find_session_by_participant(sessions, "ash_trace_codex_one") + .pointer("/participant_id") + .and_then(Value::as_str), + Some("ash_trace_codex_one") + ); + assert_eq!( + find_session_by_participant(sessions, "ash_trace_codex_two") + .pointer("/participant_id") + .and_then(Value::as_str), + Some("ash_trace_codex_two") + ); +} + +#[test] +fn agent_status_sibling_specific_suppression_keeps_other_trace_participants_visible() { + let fixture = AgentSuccessorFixture::new(); + fixture.init_workspace(); + fixture.seed_inventory_for_list_and_status_contracts(); + write_invalidated_world_member_manifest( + &fixture, + "codex", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fd4", + "ash_trace_suppressed", + "ash_orchestrator", + "wld_trace_0002", + 10, + None, + "2026-04-05T00:00:02Z", + ); + fixture.write_trace_events(&[ + json!({ + "ts": "2026-04-05T00:00:03Z", + "event_type": "agent_event", + "session_id": "ses_agent_hub", + "component": "agent-hub", + "kind": "status", + "agent_id": "codex", + "orchestration_session_id": "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fd4", + "participant_id": "ash_trace_suppressed", + "run_id": "0195f8f1-7a35-7b7f-9c4d-9a7c2f5d6fd5", + "backend_id": "cli:codex", + "client": "codex", + "router": "agent_hub", + "protocol": "uaa.agent.session", + "role": "member", + "world_id": "wld_trace_0002", + "world_generation": 10, + "data": { "message": "matching tombstoned participant must be suppressed" } + }), + json!({ + "ts": "2026-04-05T00:00:04Z", + "event_type": "agent_event", + "session_id": "ses_agent_hub", + "component": "agent-hub", + "kind": "status", + "agent_id": "codex", + "orchestration_session_id": "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fd4", + "participant_id": "ash_trace_survives", + "run_id": "0195f8f1-7a35-7b7f-9c4d-9a7c2f5d6fd6", + "backend_id": "cli:codex", + "client": "codex", + "router": "agent_hub", + "protocol": "uaa.agent.session", + "role": "member", + "world_id": "wld_trace_0002", + "world_generation": 10, + "data": { "message": "sibling participant must remain visible" } + }), + ]); + + let output = fixture.run(&["agent", "status", "--scope", "world", "--json"]); + assert!( + output.status.success(), + "world status should keep non-suppressed sibling trace participants visible: {output:?}" + ); + + let json = parse_json_output(&output); + let sessions = json["sessions"] + .as_array() + .expect("sessions should be an array"); + assert_eq!( + sessions.len(), 1, - "--role member should return exactly one member session: {member_json}" + "participant-specific suppression must not collapse unrelated siblings: {json}" ); - let member_only = find_session_by_agent(member_sessions, "codex"); assert_eq!( - member_only.pointer("/role").and_then(Value::as_str), - Some("member") + find_session_by_participant(sessions, "ash_trace_survives") + .pointer("/participant_id") + .and_then(Value::as_str), + Some("ash_trace_survives") + ); +} + +#[test] +fn agent_status_nested_parent_correlation_prefers_parent_participant_id_when_siblings_exist() { + let fixture = AgentSuccessorFixture::new(); + seed_nested_gateway_status_fixture(&fixture); + let orchestration_session_id = "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fd7"; + fixture.write_trace_events(&[ + json!({ + "ts": "2026-04-05T00:00:00Z", + "event_type": "agent_event", + "session_id": "ses_agent_hub", + "component": "agent-hub", + "kind": "status", + "agent_id": "claude_code", + "orchestration_session_id": orchestration_session_id, + "participant_id": "ash_parent_one", + "run_id": "0195f8f1-7a35-7b7f-9c4d-9a7c2f5d6fd8", + "backend_id": "cli:claude_code", + "client": "claude_code", + "router": "agent_hub", + "protocol": "uaa.agent.session", + "role": "orchestrator", + "world_id": "wld_nested_0001", + "world_generation": 11, + "data": { "message": "first pure-agent sibling is visible" } + }), + json!({ + "ts": "2026-04-05T00:00:01Z", + "event_type": "agent_event", + "session_id": "ses_agent_hub", + "component": "agent-hub", + "kind": "status", + "agent_id": "claude_code", + "orchestration_session_id": orchestration_session_id, + "participant_id": "ash_parent_two", + "run_id": "0195f8f1-7a35-7b7f-9c4d-9a7c2f5d6fd9", + "backend_id": "cli:claude_code", + "client": "claude_code", + "router": "agent_hub", + "protocol": "uaa.agent.session", + "role": "orchestrator", + "world_id": "wld_nested_0001", + "world_generation": 11, + "data": { "message": "second pure-agent sibling is visible" } + }), + json!({ + "ts": "2026-04-05T00:00:02Z", + "event_type": "agent_event", + "session_id": "ses_agent_hub", + "component": "agent-hub", + "kind": "status", + "agent_id": "claude_code", + "orchestration_session_id": orchestration_session_id, + "run_id": "0195f8f1-7a35-7b7f-9c4d-9a7c2f5d6fda", + "parent_run_id": "0195f8f1-7a35-7b7f-9c4d-9a7c2f5d6fd8", + "parent_participant_id": "ash_parent_one", + "backend_id": "cli:claude_code", + "client": "claude_code", + "router": "substrate_gateway", + "protocol": "openai.responses", + "provider": "openai", + "auth_authority": "codex_subscription", + "data": { "summary": "nested record binds to the first sibling parent" } + }), + json!({ + "ts": "2026-04-05T00:00:03Z", + "event_type": "agent_event", + "session_id": "ses_agent_hub", + "component": "agent-hub", + "kind": "status", + "agent_id": "claude_code", + "orchestration_session_id": orchestration_session_id, + "run_id": "0195f8f1-7a35-7b7f-9c4d-9a7c2f5d6fdb", + "parent_run_id": "0195f8f1-7a35-7b7f-9c4d-9a7c2f5d6fd9", + "parent_participant_id": "ash_parent_two", + "backend_id": "cli:claude_code", + "client": "claude_code", + "router": "substrate_gateway", + "protocol": "openai.responses", + "provider": "openai", + "auth_authority": "codex_subscription", + "data": { "summary": "nested record binds to the second sibling parent" } + }), + ]); + + let output = fixture.run(&["agent", "status", "--json"]); + assert!( + output.status.success(), + "status should correlate nested rows by parent_participant_id when sibling parents exist: {output:?}" ); + + let json = parse_json_output(&output); assert_eq!( - member_only.pointer("/world_id").and_then(Value::as_str), - Some("wld_active_0002") + json["warnings"].as_array().map(Vec::len), + Some(0), + "participant-aware nested correlation should avoid coarse-match warnings: {json}" + ); + let nested = json["nested_llm_records"] + .as_array() + .expect("nested_llm_records should be an array"); + assert_eq!( + nested.len(), + 2, + "both sibling-bound nested rows should survive: {json}" ); assert_eq!( - member_only + nested[0] + .pointer("/parent/participant_id") + .and_then(Value::as_str), + Some("ash_parent_one") + ); + assert_eq!( + nested[1] + .pointer("/parent/participant_id") + .and_then(Value::as_str), + Some("ash_parent_two") + ); +} + +#[test] +fn agent_status_persists_resumed_from_participant_id_for_replacement_members() { + let fixture = AgentSuccessorFixture::new(); + fixture.init_workspace(); + fixture.seed_inventory_for_list_and_status_contracts(); + write_live_runtime_manifest( + &fixture, + "claude_code", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fb3", + "ash_orchestrator_lineage", + "2026-04-05T00:00:01Z", + ); + write_active_orchestration_session( + &fixture, + "claude_code", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fb3", + Some("ash_orchestrator_lineage"), + "2026-04-05T00:00:01Z", + ); + write_invalidated_world_member_manifest( + &fixture, + "codex", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fb3", + "ash_codex_old_lineage", + "ash_orchestrator_lineage", + "wld_old_0003", + 6, + None, + "2026-04-05T00:00:02Z", + ); + write_replacement_world_member_manifest( + &fixture, + "codex", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fb3", + "ash_codex_new_lineage", + "ash_orchestrator_lineage", + "wld_new_0003", + 7, + "ash_codex_old_lineage", + "2026-04-05T00:00:03Z", + ); + + let output = fixture.run(&["agent", "status", "--scope", "world", "--json"]); + assert!( + output.status.success(), + "world status should succeed for replacement lineage fixtures: {output:?}" + ); + + let json = parse_json_output(&output); + let sessions = json["sessions"] + .as_array() + .expect("sessions should be an array"); + assert_eq!( + sessions.len(), + 1, + "replacement member should win world status selection: {json}" + ); + let replacement = find_session_by_agent(sessions, "codex"); + assert_eq!( + replacement + .pointer("/participant_id") + .and_then(Value::as_str), + Some("ash_codex_new_lineage") + ); + assert_eq!( + replacement .pointer("/world_generation") .and_then(Value::as_u64), Some(7) ); - let orchestrator_output = fixture.run(&["agent", "status", "--role", "orchestrator", "--json"]); - assert!( - orchestrator_output.status.success(), - "orchestrator-filtered agent status should succeed: {orchestrator_output:?}" - ); - let orchestrator_json = parse_json_output(&orchestrator_output); + let persisted = serde_json::from_str::( + &fs::read_to_string(participant_manifest_path(&fixture, "ash_codex_new_lineage")) + .expect("replacement participant manifest should be readable"), + ) + .expect("replacement participant manifest should be valid JSON"); assert_eq!( - orchestrator_json - .pointer("/role_filter") + persisted + .pointer("/resumed_from_participant_id") .and_then(Value::as_str), - Some("orchestrator"), - "orchestrator-filtered status must report role_filter=orchestrator: {orchestrator_json}" + Some("ash_codex_old_lineage"), + "replacement participant persistence must keep resumed_from_participant_id as the lineage field name" + ); + assert!( + persisted.get("resumed_from_session_handle_id").is_none(), + "replacement participant persistence must not revert to the legacy resumed_from_session_handle_id field name" + ); +} + +#[test] +fn agent_status_surfaces_host_successor_lineage_for_active_orchestrator_sessions() { + let fixture = AgentSuccessorFixture::new(); + fixture.init_workspace(); + fixture.seed_inventory_for_list_and_status_contracts(); + write_active_orchestration_session( + &fixture, + "claude_code", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fb4", + Some("ash_orchestrator_new_lineage"), + "2026-04-05T00:00:03Z", + ); + write_invalidated_runtime_manifest( + &fixture, + "claude_code", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fb4", + "ash_orchestrator_old_lineage", + "2026-04-05T00:00:01Z", + ); + write_runtime_participant( + &fixture, + "ash_orchestrator_new_lineage", + "claude_code", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fb4", + RuntimeParticipantOptions::host_orchestrator("running", true, "2026-04-05T00:00:03Z") + .with_resumed_from_participant_id(Some("ash_orchestrator_old_lineage")), + ); + + let output = fixture.run(&["agent", "status", "--json"]); + assert!( + output.status.success(), + "agent status should succeed for host successor lineage fixtures: {output:?}" ); - let orchestrator_sessions = orchestrator_json["sessions"] + + let json = parse_json_output(&output); + let sessions = json["sessions"] .as_array() .expect("sessions should be an array"); + let successor = find_session_by_participant(sessions, "ash_orchestrator_new_lineage"); assert_eq!( - orchestrator_sessions.len(), - 1, - "--role orchestrator should return exactly one orchestrator session: {orchestrator_json}" + successor.pointer("/role").and_then(Value::as_str), + Some("orchestrator") ); - let orchestrator_only = find_session_by_agent(orchestrator_sessions, "claude_code"); assert_eq!( - orchestrator_only.pointer("/role").and_then(Value::as_str), - Some("orchestrator") + successor.pointer("/participant_id").and_then(Value::as_str), + Some("ash_orchestrator_new_lineage"), + "status must surface the live successor participant after public control successor flows" + ); + + let persisted = serde_json::from_str::( + &fs::read_to_string(participant_manifest_path( + &fixture, + "ash_orchestrator_new_lineage", + )) + .expect("host successor manifest should be readable"), + ) + .expect("host successor manifest should be valid JSON"); + assert_eq!( + persisted + .pointer("/resumed_from_participant_id") + .and_then(Value::as_str), + Some("ash_orchestrator_old_lineage") + ); + assert!( + persisted.get("resumed_from_session_handle_id").is_none(), + "host successor persistence must keep participant lineage canonical" ); } #[test] -fn agent_status_prefers_live_manifest_over_trace_fallback_for_selected_orchestrator() { +fn agent_status_json_prefers_live_runtime_member_manifest_with_top_level_world_identity() { let fixture = AgentSuccessorFixture::new(); fixture.init_workspace(); fixture.seed_inventory_for_list_and_status_contracts(); write_live_runtime_manifest( &fixture, "claude_code", - "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6faa", - "ash_live_status", - "2026-04-05T00:00:02Z", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fb6", + "ash_orchestrator_runtime_member", + "2026-04-05T00:00:01Z", + ); + write_active_orchestration_session( + &fixture, + "claude_code", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fb6", + Some("ash_orchestrator_runtime_member"), + "2026-04-05T00:00:01Z", + ); + write_runtime_participant( + &fixture, + "ash_codex_runtime_member", + "codex", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fb6", + RuntimeParticipantOptions::world_member( + "running", + true, + "2026-04-05T00:00:02Z", + "wld_runtime_member_0004", + 9, + "ash_orchestrator_runtime_member", + ), ); fixture.write_trace_events(&[json!({ "ts": "2026-04-05T00:00:03Z", @@ -1184,92 +3682,169 @@ fn agent_status_prefers_live_manifest_over_trace_fallback_for_selected_orchestra "session_id": "ses_agent_hub", "component": "agent-hub", "kind": "status", - "agent_id": "claude_code", - "orchestration_session_id": "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fab", - "run_id": "0195f8f1-7a35-7b7f-9c4d-9a7c2f5d6fac", - "backend_id": "cli:claude_code", - "client": "claude_code", + "agent_id": "codex", + "orchestration_session_id": "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fb6", + "run_id": "0195f8f1-7a35-7b7f-9c4d-9a7c2f5d6fb7", + "participant_id": "ash_trace_member_should_lose", + "backend_id": "cli:codex", + "client": "codex", "router": "agent_hub", "protocol": "uaa.agent.session", - "role": "orchestrator", - "data": { "message": "trace fallback should lose to live manifest" } + "role": "member", + "world_id": "wld_trace_member_should_lose", + "world_generation": 3, + "data": { "message": "trace fallback should not override the live runtime member manifest" } })]); let output = fixture.run(&["agent", "status", "--json"]); assert!( output.status.success(), - "status should prefer the live manifest: {output:?}" + "agent status --json should surface the live runtime member from authoritative state: {output:?}" ); let json = parse_json_output(&output); let sessions = json["sessions"] .as_array() .expect("sessions should be an array"); - let orchestrator = find_session_by_agent(sessions, "claude_code"); + let member = find_session_by_agent(sessions, "codex"); assert_eq!( - orchestrator - .pointer("/orchestration_session_id") + member.pointer("/source_kind").and_then(Value::as_str), + Some("live_runtime") + ); + assert_eq!( + member.pointer("/participant_id").and_then(Value::as_str), + Some("ash_codex_runtime_member"), + "the selected live member must come from runtime state rather than trace fallback: {json}" + ); + assert_eq!( + member.pointer("/execution/scope").and_then(Value::as_str), + Some("world") + ); + assert_eq!( + member.pointer("/world_id").and_then(Value::as_str), + Some("wld_runtime_member_0004"), + "runtime-owned world members must publish world_id at the top level on agent status --json: {json}" + ); + assert_eq!( + member.pointer("/world_generation").and_then(Value::as_u64), + Some(9), + "runtime-owned world members must publish world_generation at the top level on agent status --json: {json}" + ); + assert_eq!( + member.pointer("/posture").and_then(Value::as_str), + Some("active_attached") + ); + assert_eq!( + member + .pointer("/attached_participant_id") .and_then(Value::as_str), - Some("0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6faa"), - "status must prefer live manifest state over trace fallback for the selected orchestrator: {json}" + Some("ash_orchestrator_runtime_member") + ); + assert_ne!( + member.pointer("/attached_participant_id").and_then(Value::as_str), + member.pointer("/participant_id").and_then(Value::as_str), + "member rows must source attached_participant_id from the parent session rather than the member participant id: {json}" + ); + assert_eq!( + member + .pointer("/pending_inbox_count") + .and_then(Value::as_u64), + Some(0) ); } #[test] -fn agent_status_suppresses_trace_fallback_rows_for_live_pure_agent_identity() { +fn agent_status_keeps_stale_terminal_member_trace_rows_auditable_without_reviving_them() { let fixture = AgentSuccessorFixture::new(); fixture.init_workspace(); fixture.seed_inventory_for_list_and_status_contracts(); write_live_runtime_manifest( &fixture, "claude_code", - "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6faa", - "ash_live_status_suppression", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fb8", + "ash_orchestrator_terminal_audit", + "2026-04-05T00:00:01Z", + ); + write_active_orchestration_session( + &fixture, + "claude_code", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fb8", + Some("ash_orchestrator_terminal_audit"), + "2026-04-05T00:00:01Z", + ); + write_invalidated_world_member_manifest( + &fixture, + "codex", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fb8", + "ash_codex_terminal_old", + "ash_orchestrator_terminal_audit", + "wld_terminal_old_0008", + 5, + None, "2026-04-05T00:00:02Z", ); + write_replacement_world_member_manifest( + &fixture, + "codex", + "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fb8", + "ash_codex_terminal_new", + "ash_orchestrator_terminal_audit", + "wld_terminal_new_0008", + 6, + "ash_codex_terminal_old", + "2026-04-05T00:00:03Z", + ); fixture.write_trace_events(&[json!({ - "ts": "2026-04-05T00:00:01Z", + "ts": "2026-04-05T00:00:04Z", "event_type": "agent_event", "session_id": "ses_agent_hub", "component": "agent-hub", - "kind": "status", - "agent_id": "claude_code", - "orchestration_session_id": "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fab", - "run_id": "0195f8f1-7a35-7b7f-9c4d-9a7c2f5d6fac", - "backend_id": "cli:claude_code", - "client": "claude_code", + "kind": "task_end", + "agent_id": "codex", + "orchestration_session_id": "0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6fb8", + "run_id": "0195f8f1-7a35-7b7f-9c4d-9a7c2f5d6fb9", + "participant_id": "ash_codex_terminal_old", + "backend_id": "cli:codex", + "client": "codex", "router": "agent_hub", "protocol": "uaa.agent.session", - "role": "orchestrator", - "data": { "message": "trace fallback should be suppressed for the same pure-agent identity" } + "role": "member", + "world_id": "wld_terminal_old_0008", + "world_generation": 5, + "data": { "message": "historical terminal row remains auditable but must not become live again" } })]); - let output = fixture.run(&["agent", "status", "--json"]); + let output = fixture.run(&["agent", "status", "--scope", "world", "--json"]); assert!( output.status.success(), - "status should succeed when a live pure-agent manifest suppresses trace fallback rows: {output:?}" + "world status should suppress stale terminal trace rows once a replacement member is live: {output:?}" ); let json = parse_json_output(&output); let sessions = json["sessions"] .as_array() .expect("sessions should be an array"); - let claude_rows: Vec<&Value> = sessions - .iter() - .filter(|session| { - session.pointer("/agent_id").and_then(Value::as_str) == Some("claude_code") - }) - .collect(); assert_eq!( - claude_rows.len(), + sessions.len(), 1, - "live pure-agent status identity must suppress trace fallback duplicates for the same agent_id: {json}" + "stale terminal trace rows must remain auditable without reviving an old live member: {json}" ); + let replacement = find_session_by_agent(sessions, "codex"); assert_eq!( - claude_rows[0] - .pointer("/orchestration_session_id") + replacement + .pointer("/participant_id") .and_then(Value::as_str), - Some("0195f8f1-7a34-7b7f-9c4d-9a7c2f5d6faa") + Some("ash_codex_terminal_new") + ); + assert_eq!( + replacement.pointer("/world_id").and_then(Value::as_str), + Some("wld_terminal_new_0008") + ); + assert_eq!( + replacement + .pointer("/world_generation") + .and_then(Value::as_u64), + Some(6) ); } @@ -1791,6 +4366,8 @@ fn agent_doctor_json_locks_field_names_omissions_and_check_order() { "inventory_scan", "orchestrator_selection", "runtime_realizability", + "participant_store", + "member_selection", "policy_allowlist", "world_boundary", ], @@ -1808,7 +4385,15 @@ fn agent_doctor_json_locks_field_names_omissions_and_check_order() { .collect(); assert_eq!( statuses, - vec!["pass", "pass", "pass", "pass", "not_applicable"], + vec![ + "pass", + "pass", + "pass", + "pass", + "not_applicable", + "pass", + "not_applicable", + ], "host-only doctor fixture should report a not_applicable world boundary after the four required passes: {json}" ); } @@ -2041,7 +4626,7 @@ fn agent_doctor_fails_at_runtime_realizability_when_selected_cli_mode_is_per_req assert_eq!( checks[2].pointer("/reason").and_then(Value::as_str), Some( - "selected orchestrator 'claude_code' is not runtime-realizable because cli.mode=per_request is unsupported; only cli.mode=persistent is supported for the first caller path" + "selected runtime 'claude_code' is not runtime-realizable because cli.mode=per_request is unsupported; only cli.mode=persistent is supported for the first caller path" ) ); } @@ -2073,12 +4658,23 @@ fn docs_usage_and_repo_boundary_match_the_successor_contract() { "substrate agents list", "substrate agents status", "substrate agents doctor", + "live session discovery is backed by persisted manifests under `~/.substrate/run/agent-hub/handles/`", ] { assert!( !usage.contains(forbidden), "docs/USAGE.md must not advertise plural successor aliases: {forbidden}" ); } + for required in [ + "`~/.substrate/run/agent-hub/sessions//session.json`", + "`~/.substrate/run/agent-hub/sessions//participants/.json`", + "`~/.substrate/run/agent-hub/handles/*.json` remains compatibility input only", + ] { + assert!( + usage.contains(required), + "docs/USAGE.md must describe the session-root live-state authority boundary `{required}`" + ); + } assert!( !repo_root().join("crates/agent-hub").exists(), @@ -2167,6 +4763,39 @@ fn ahcsitc3_configuration_doc_locks_successor_config_surface() { } } +#[test] +fn closeout_docs_keep_session_and_participant_terms_aligned_to_runtime_truth() { + let matrix = read_repo_file("AGENT_ORCHESTRATION_GAP_MATRIX.md"); + for required in [ + "Status ambiguity handling", + "Trace-only participant-aware fallback", + "orchestration_session_id", + "participant_id", + ] { + assert!( + matrix.contains(required), + "gap matrix must retain closeout terminology `{required}`" + ); + } + + let packet = read_repo_file("llm-last-mile/README.md"); + for required in [ + "`~/.substrate/run/agent-hub/sessions//session.json`", + "`~/.substrate/run/agent-hub/sessions//participants/.json`", + "trace.jsonl` remains audit history, not live state authority", + "read-only status can degrade to warnings", + ] { + assert!( + packet.contains(required), + "llm-last-mile README must document the runtime truth `{required}`" + ); + } + assert!( + !packet.contains("`~/.substrate/run/agent-sessions/.json`"), + "llm-last-mile README must not keep the superseded agent-sessions path" + ); +} + #[test] fn ahcsitc3_trace_doc_locks_tuple_compatible_fields() { let trace = read_repo_file("docs/TRACE.md"); @@ -2179,6 +4808,12 @@ fn ahcsitc3_trace_doc_locks_tuple_compatible_fields() { "`auth_authority`", "`world_id`", "`world_generation`", + "`~/.substrate/run/agent-hub/sessions//session.json`", + "`~/.substrate/run/agent-hub/sessions//participants/.json`", + "`~/.substrate/run/agent-hub/sessions/.json`", + "`~/.substrate/run/agent-hub/participants/*.json`", + "`~/.substrate/run/agent-hub/handles/*.json` remains legacy compatibility input only", + "Invalidated participant tombstones in canonical or flat compatibility participant records beat stale trace fallback rows", ] { assert!( trace.contains(required), @@ -2187,6 +4822,46 @@ fn ahcsitc3_trace_doc_locks_tuple_compatible_fields() { } } +#[test] +fn production_runtime_snapshot_writes_remain_centralized_in_state_store() { + let src_root = repo_root().join("crates/shell/src"); + let forbidden_markers = [ + "join(\"agent-hub\")", + "join(\"handles\")", + "join(\"participants\")", + "join(\"leases\")", + "handles_dir()", + "participants_dir()", + "sessions_dir()", + ]; + + let offenders = collect_rust_files(&src_root) + .into_iter() + .filter(|path| path.file_name().and_then(|name| name.to_str()) != Some("state_store.rs")) + .filter_map(|path| { + let source = read_production_shell_source_without_inline_tests(&path); + let hits = forbidden_markers + .iter() + .copied() + .filter(|marker| { + if path.file_name().and_then(|name| name.to_str()) == Some("control.rs") { + return *marker != "handles_dir()" && *marker != "join(\"handles\")"; + } + true + }) + .filter(|marker| source.contains(marker)) + .collect::>(); + (!hits.is_empty()).then_some(format!("{} => {}", path.display(), hits.join(", "))) + }) + .collect::>(); + + assert!( + offenders.is_empty(), + "production callers outside state_store.rs must not reach flat compatibility parent/participant/lease or legacy handle paths directly: {}", + offenders.join(" | ") + ); +} + #[test] fn agent_doctor_fails_closed_on_world_member_allowlist_before_world_boundary() { let fixture = AgentSuccessorFixture::new(); @@ -2270,12 +4945,14 @@ metadata: {} "inventory_scan", "orchestrator_selection", "runtime_realizability", + "participant_store", + "member_selection", "policy_allowlist", ], "fail-closed routing must stop at the member backend allowlist before world_boundary: {json}" ); assert_eq!( - checks[3].pointer("/reason").and_then(Value::as_str), + checks[5].pointer("/reason").and_then(Value::as_str), Some( "required world-scoped member backend 'cli:codex' is not allowlisted by effective policy agents.allowed_backends" ), @@ -2283,6 +4960,181 @@ metadata: {} ); } +#[test] +fn agent_doctor_host_only_inventory_keeps_zero_world_member_candidates_not_applicable() { + let fixture = AgentSuccessorFixture::new(); + fixture.init_workspace(); + fixture.seed_inventory_for_doctor_contract(); + + let output = fixture.run(&["agent", "doctor", "--json"]); + assert!( + output.status.success(), + "doctor should preserve host-only behavior when zero eligible world members exist: {output:?}" + ); + + let json = parse_json_output(&output); + let checks = json["checks"] + .as_array() + .expect("checks should be an array"); + assert_eq!( + checks[4].pointer("/check").and_then(Value::as_str), + Some("member_selection") + ); + assert_eq!( + checks[4].pointer("/status").and_then(Value::as_str), + Some("not_applicable"), + "zero eligible world members must preserve a not_applicable member-selection posture: {json}" + ); + assert_eq!( + checks[6].pointer("/check").and_then(Value::as_str), + Some("world_boundary") + ); + assert_eq!( + checks[6].pointer("/status").and_then(Value::as_str), + Some("not_applicable"), + "zero eligible world members must keep world-boundary checks dormant: {json}" + ); +} + +#[test] +fn agent_doctor_exactly_one_world_member_candidate_continues_into_world_boundary_checks() { + let fixture = AgentSuccessorFixture::new(); + fixture.init_workspace(); + fixture.write_global_config_patch( + r#"world: + enabled: false +agents: + enabled: true + hub: + orchestrator_agent_id: claude_code +"#, + ); + fixture.write_global_policy_patch( + r#"id: "ahcsitc2-policy" +name: "ahcsitc2-policy" + +world_fs: + host_visible: true + fail_closed: + routing: true + write: + enabled: true + +agents: + allowed_backends: + - "cli:claude_code" + - "cli:codex" + +net_allowed: [] +cmd_allowed: [] +cmd_denied: [] +cmd_isolated: [] + +require_approval: false +allow_shell_operators: true + +limits: + max_memory_mb: null + max_cpu_percent: null + max_runtime_ms: null + max_egress_bytes: null + +metadata: {} +"#, + ); + fixture.write_agent_file( + "claude_code.yaml", + &cli_agent_file("claude_code", "host", true, true, true), + ); + fixture.write_agent_file( + "codex.yaml", + &cli_agent_file("codex", "world", true, false, true), + ); + + let output = fixture.run(&["agent", "doctor", "--json"]); + assert_eq!( + output.status.code(), + Some(3), + "exactly one eligible world member must continue through world-boundary validation: {output:?}" + ); + + let json = parse_json_output(&output); + let checks = json["checks"] + .as_array() + .expect("checks should be an array"); + let observed: Vec<&str> = checks + .iter() + .map(|check| { + check + .pointer("/check") + .and_then(Value::as_str) + .expect("check id should be a string") + }) + .collect(); + assert_eq!( + observed, + vec![ + "inventory_scan", + "orchestrator_selection", + "runtime_realizability", + "participant_store", + "member_selection", + "policy_allowlist", + "world_boundary", + ], + "a unique world member candidate must advance through member_selection and policy_allowlist before world_boundary: {json}" + ); + assert_eq!( + checks[4].pointer("/status").and_then(Value::as_str), + Some("pass"), + "exactly one eligible world member must satisfy member_selection: {json}" + ); + assert_eq!( + checks[5].pointer("/status").and_then(Value::as_str), + Some("pass"), + "the unique selected member must continue through policy_allowlist: {json}" + ); + assert_eq!( + checks[6].pointer("/status").and_then(Value::as_str), + Some("fail") + ); + assert_eq!( + checks[6].pointer("/reason").and_then(Value::as_str), + Some("world-scoped member posture requires world isolation but world is disabled"), + "world-boundary failure must be reported only after unique member selection succeeds: {json}" + ); +} + +#[test] +fn doctor_member_selection_contract_source_locks_ambiguous_fail_closed_path() { + let validator = read_repo_file("crates/shell/src/execution/agent_runtime/validator.rs"); + assert!( + validator.contains( + "ambiguous world member selection: multiple eligible world-scoped members found" + ), + "validator.rs must retain the shared ambiguous world-member fail-closed reason" + ); + + let agents_cmd = read_repo_file("crates/shell/src/execution/agents_cmd.rs"); + let member_selection = agents_cmd + .find("check: \"member_selection\"") + .expect("agents_cmd.rs must report a doctor member_selection check"); + let policy_allowlist = agents_cmd + .find("check: \"policy_allowlist\"") + .expect("agents_cmd.rs must report a doctor policy_allowlist check"); + let world_boundary = agents_cmd + .find("check: \"world_boundary\"") + .expect("agents_cmd.rs must report a doctor world_boundary check"); + assert!( + member_selection < policy_allowlist && policy_allowlist < world_boundary, + "doctor must keep member_selection ahead of policy_allowlist and world_boundary in source order" + ); + assert!( + agents_cmd.contains("validate_member_selection(&effective_config, &inventory)"), + "doctor must reuse the shared runtime member-selection helper rather than drifting to a separate truth source" + ); +} + #[test] fn agent_doctor_does_not_treat_trace_records_as_control_plane_authorization() { let fixture = AgentSuccessorFixture::new(); diff --git a/crates/shell/tests/agents_validate.rs b/crates/shell/tests/agents_validate.rs index dd0484f09..98e8d9698 100644 --- a/crates/shell/tests/agents_validate.rs +++ b/crates/shell/tests/agents_validate.rs @@ -3,10 +3,10 @@ mod common; use assert_cmd::Command; -use common::{substrate_shell_driver, temp_dir}; +use common::substrate_shell_driver; use std::fs; use std::path::PathBuf; -use tempfile::TempDir; +use tempfile::{Builder, TempDir}; struct AgentsValidateFixture { _temp: TempDir, @@ -17,7 +17,10 @@ struct AgentsValidateFixture { impl AgentsValidateFixture { fn new() -> Self { - let temp = temp_dir("substrate-agents-validate-"); + let temp = Builder::new() + .prefix("substrate-agents-validate-") + .tempdir_in("/tmp") + .expect("failed to allocate integration test temp dir"); let home = temp.path().join("home"); fs::create_dir_all(&home).expect("failed to create HOME fixture"); let substrate_home = temp.path().join("substrate-home"); diff --git a/crates/shell/tests/authoritative_inbox_dev_support.rs b/crates/shell/tests/authoritative_inbox_dev_support.rs new file mode 100644 index 000000000..529ff300e --- /dev/null +++ b/crates/shell/tests/authoritative_inbox_dev_support.rs @@ -0,0 +1,27 @@ +#![cfg(any(target_os = "linux", target_os = "macos"))] + +#[path = "support/mod.rs"] +mod support; + +use std::env; +use std::path::PathBuf; + +#[test] +#[ignore = "dev-support helper invoked explicitly by validation scripts"] +fn authoritative_inbox_dev_support_persists_pending_item_from_env() { + let substrate_home = PathBuf::from( + env::var("SUBSTRATE_HOME").expect("SUBSTRATE_HOME must be set for dev-support helper"), + ); + let orchestration_session_id = env::var("SUBSTRATE_DEV_SUPPORT_SESSION_ID") + .expect("SUBSTRATE_DEV_SUPPORT_SESSION_ID must be set"); + let item_id = env::var("SUBSTRATE_DEV_SUPPORT_ITEM_ID") + .expect("SUBSTRATE_DEV_SUPPORT_ITEM_ID must be set"); + let message = env::var("SUBSTRATE_DEV_SUPPORT_MESSAGE").ok(); + + support::persist_runtime_alert_for_substrate_home( + &substrate_home, + &orchestration_session_id, + &item_id, + message, + ); +} diff --git a/crates/shell/tests/cmd_denied_host_only.rs b/crates/shell/tests/cmd_denied_host_only.rs index 39a4c2366..dadda3a74 100644 --- a/crates/shell/tests/cmd_denied_host_only.rs +++ b/crates/shell/tests/cmd_denied_host_only.rs @@ -26,6 +26,8 @@ impl CmdDeniedHostOnlyFixture { let socket_path = socket_dir.path().join("substrate.sock"); let project_dir = shell.home().join("project"); fs::create_dir_all(&project_dir).expect("failed to create project dir"); + fs::create_dir_all(project_dir.join(".substrate")) + .expect("failed to create workspace .substrate"); Self { shell, _socket_dir: socket_dir, @@ -38,6 +40,10 @@ impl CmdDeniedHostOnlyFixture { self.shell.home().join(".substrate") } + fn workspace_config_path(&self) -> PathBuf { + self.project_dir.join(".substrate").join("workspace.yaml") + } + fn socket_path(&self) -> &Path { &self.socket_path } @@ -50,6 +56,8 @@ impl CmdDeniedHostOnlyFixture { "world:\n enabled: true\n anchor_mode: follow-cwd\n anchor_path: \"\"\n caged: false\n\npolicy:\n mode: enforce\n\nsync:\n auto_sync: false\n direction: from_world\n conflict_policy: prefer_host\n exclude: []\n", ) .expect("failed to write config.yaml"); + fs::write(self.workspace_config_path(), "world:\n enabled: true\n") + .expect("failed to write workspace.yaml"); } fn write_global_policy_patch_cmd_denied(&self) { diff --git a/crates/shell/tests/common.rs b/crates/shell/tests/common.rs index 3f749267a..f48e86cdf 100644 --- a/crates/shell/tests/common.rs +++ b/crates/shell/tests/common.rs @@ -38,11 +38,12 @@ pub fn shared_tmpdir() -> &'static Path { }) } -/// Allocate a temporary directory with a descriptive prefix under the shared root. +/// Allocate a temporary directory with a descriptive prefix under `/tmp` so +/// integration fixtures never become nested descendants of the repo workspace. pub fn temp_dir(prefix: &str) -> TempDir { Builder::new() .prefix(prefix) - .tempdir_in(shared_tmpdir()) + .tempdir_in("/tmp") .expect("failed to allocate integration test temp dir") } diff --git a/crates/shell/tests/config_set.rs b/crates/shell/tests/config_set.rs index c9d02c8db..d65f43f3f 100644 --- a/crates/shell/tests/config_set.rs +++ b/crates/shell/tests/config_set.rs @@ -3,12 +3,12 @@ mod common; use assert_cmd::Command; -use common::{substrate_shell_driver, temp_dir}; +use common::substrate_shell_driver; use serde_json::Value as JsonValue; use serde_yaml::Value as YamlValue; use std::fs; use std::path::{Path, PathBuf}; -use tempfile::TempDir; +use tempfile::{Builder, TempDir}; struct ConfigSetFixture { _temp: TempDir, @@ -19,7 +19,10 @@ struct ConfigSetFixture { impl ConfigSetFixture { fn new() -> Self { - let temp = temp_dir("substrate-config-set-"); + let temp = Builder::new() + .prefix("substrate-config-set-") + .tempdir_in("/tmp") + .expect("failed to allocate integration test temp dir"); let home = temp.path().join("home"); fs::create_dir_all(&home).expect("failed to create HOME fixture"); let substrate_home = temp.path().join("substrate-home"); diff --git a/crates/shell/tests/config_show.rs b/crates/shell/tests/config_show.rs index 09d11fa1b..c98045bc9 100644 --- a/crates/shell/tests/config_show.rs +++ b/crates/shell/tests/config_show.rs @@ -3,12 +3,12 @@ mod common; use assert_cmd::Command; -use common::{substrate_shell_driver, temp_dir}; +use common::substrate_shell_driver; use serde_json::Value as JsonValue; use serde_yaml::Value as YamlValue; use std::fs; use std::path::{Path, PathBuf}; -use tempfile::TempDir; +use tempfile::{Builder, TempDir}; struct ConfigShowFixture { _temp: TempDir, @@ -19,7 +19,10 @@ struct ConfigShowFixture { impl ConfigShowFixture { fn new() -> Self { - let temp = temp_dir("substrate-config-show-"); + let temp = Builder::new() + .prefix("substrate-config-show-") + .tempdir_in("/tmp") + .expect("failed to allocate integration test temp dir"); let home = temp.path().join("home"); fs::create_dir_all(&home).expect("failed to create HOME fixture"); let substrate_home = temp.path().join("substrate-home"); diff --git a/crates/shell/tests/config_world_deps_phase_b.rs b/crates/shell/tests/config_world_deps_phase_b.rs index 38358a32d..4621d1d7a 100644 --- a/crates/shell/tests/config_world_deps_phase_b.rs +++ b/crates/shell/tests/config_world_deps_phase_b.rs @@ -3,12 +3,12 @@ mod common; use assert_cmd::Command; -use common::{substrate_shell_driver, temp_dir}; +use common::substrate_shell_driver; use serde_json::Value as JsonValue; use serde_yaml::Value as YamlValue; use std::fs; use std::path::{Path, PathBuf}; -use tempfile::TempDir; +use tempfile::{Builder, TempDir}; struct WorldDepsConfigFixture { _temp: TempDir, @@ -19,7 +19,10 @@ struct WorldDepsConfigFixture { impl WorldDepsConfigFixture { fn new() -> Self { - let temp = temp_dir("substrate-config-world-deps-"); + let temp = Builder::new() + .prefix("substrate-config-world-deps-") + .tempdir_in("/tmp") + .expect("failed to allocate integration test temp dir"); let home = temp.path().join("home"); fs::create_dir_all(&home).expect("failed to create HOME fixture"); let substrate_home = temp.path().join("substrate-home"); diff --git a/crates/shell/tests/ev0_override_split.rs b/crates/shell/tests/ev0_override_split.rs index 61b4216b0..858bf9b0b 100644 --- a/crates/shell/tests/ev0_override_split.rs +++ b/crates/shell/tests/ev0_override_split.rs @@ -3,11 +3,11 @@ mod common; use assert_cmd::Command; -use common::{substrate_shell_driver, temp_dir}; +use common::substrate_shell_driver; use serde_json::Value as JsonValue; use std::fs; use std::path::{Path, PathBuf}; -use tempfile::TempDir; +use tempfile::{Builder, TempDir}; struct OverrideSplitFixture { _temp: TempDir, @@ -19,7 +19,10 @@ struct OverrideSplitFixture { impl OverrideSplitFixture { fn new(prefix: &str) -> Self { - let temp = temp_dir(prefix); + let temp = Builder::new() + .prefix(prefix) + .tempdir_in("/tmp") + .expect("failed to allocate integration test temp dir"); let home = temp.path().join("home"); fs::create_dir_all(&home).expect("create HOME"); let substrate_home = temp.path().join("substrate-home"); diff --git a/crates/shell/tests/fail_closed_semantics.rs b/crates/shell/tests/fail_closed_semantics.rs index ac163033d..3cd308c86 100644 --- a/crates/shell/tests/fail_closed_semantics.rs +++ b/crates/shell/tests/fail_closed_semantics.rs @@ -3,9 +3,10 @@ #[path = "common.rs"] mod common; -use common::{substrate_shell_driver, temp_dir}; +use common::substrate_shell_driver; use std::fs; use std::path::{Path, PathBuf}; +use tempfile::Builder; fn manager_manifest_path() -> PathBuf { PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("../../config/manager_hooks.yaml") @@ -82,7 +83,10 @@ fn base_env_cmd( #[test] fn fail_closed_routing_true_exits_3_when_world_socket_unavailable() { - let temp = temp_dir("substrate-i1-fail-closed-routing-"); + let temp = Builder::new() + .prefix("substrate-i1-fail-closed-routing-") + .tempdir_in("/tmp") + .expect("failed to allocate integration test temp dir"); let home = temp.path().join("home"); let project = temp.path().join("project"); fs::create_dir_all(&home).expect("create home"); @@ -124,7 +128,10 @@ fn fail_closed_routing_true_exits_3_when_world_socket_unavailable() { #[test] fn require_world_false_warns_once_and_falls_back_to_host_when_world_socket_unavailable() { - let temp = temp_dir("substrate-i1-optional-world-"); + let temp = Builder::new() + .prefix("substrate-i1-optional-world-") + .tempdir_in("/tmp") + .expect("failed to allocate integration test temp dir"); let home = temp.path().join("home"); let project = temp.path().join("project"); fs::create_dir_all(&home).expect("create home"); diff --git a/crates/shell/tests/persistent_session_client_v1.rs b/crates/shell/tests/persistent_session_client_v1.rs index d4d7ea08f..53c68d14f 100644 --- a/crates/shell/tests/persistent_session_client_v1.rs +++ b/crates/shell/tests/persistent_session_client_v1.rs @@ -1,3 +1,4 @@ +use agent_api_types::{SharedWorldOwnerAction, SharedWorldOwnerSpec}; use serde_json::json; use substrate_shell::execution::repl_persistent_session::PersistentSessionClientCore; @@ -11,6 +12,20 @@ fn ready_frame(protocol_version: u32) -> serde_json::Value { }) } +fn ready_frame_with_shared_world( + protocol_version: u32, + shared_world: serde_json::Value, +) -> serde_json::Value { + json!({ + "type": "ready", + "session_nonce": "0123456789abcdef0123456789abcdef", + "world_id": "wld_test", + "cwd": "/", + "protocol_version": protocol_version, + "shared_world": shared_world, + }) +} + fn exit_frame(code: i32) -> serde_json::Value { json!({ "type": "exit", @@ -35,6 +50,19 @@ fn command_complete_frame(seq: u64, token_hex: &str) -> serde_json::Value { }) } +fn shared_world_binding( + world_id: &str, + orchestration_session_id: &str, + world_generation: u64, +) -> serde_json::Value { + json!({ + "orchestration_session_id": orchestration_session_id, + "world_id": world_id, + "world_generation": world_generation, + "binding_state": "active", + }) +} + #[test] fn persistent_session_client_v1_fail_closed_core_contract() { // Version mismatch must fail closed. @@ -101,3 +129,58 @@ fn persistent_session_client_v1_fail_closed_core_contract() { client.note_shutdown_initiated(); client.on_server_frame(exit_frame(0)).unwrap(); } + +#[test] +fn persistent_session_client_v1_accepts_shared_world_attach_create_ready_proof() { + let mut client = PersistentSessionClientCore::new(); + client.note_start_session_sent_with_shared_world(Some(SharedWorldOwnerSpec { + orchestration_session_id: "orch-test".to_string(), + action: SharedWorldOwnerAction::AttachOrCreate, + })); + + client + .on_server_frame(ready_frame_with_shared_world( + 1, + shared_world_binding("wld_test", "orch-test", 0), + )) + .expect("matching attach/create proof should be accepted"); +} + +#[test] +fn persistent_session_client_v1_accepts_replacement_ready_when_generation_advances() { + let mut client = PersistentSessionClientCore::new(); + client.note_start_session_sent_with_shared_world(Some(SharedWorldOwnerSpec { + orchestration_session_id: "orch-test".to_string(), + action: SharedWorldOwnerAction::ReplaceExpectedGeneration { + expected_generation: 41, + reason: "test replacement".to_string(), + }, + })); + + client + .on_server_frame(ready_frame_with_shared_world( + 1, + shared_world_binding("wld_test", "orch-test", 42), + )) + .expect("replacement proof should require a strictly newer world generation"); +} + +#[test] +fn persistent_session_client_v1_rejects_invalid_shared_world_ready_proof() { + let mut client = PersistentSessionClientCore::new(); + client.note_start_session_sent_with_shared_world(Some(SharedWorldOwnerSpec { + orchestration_session_id: "orch-test".to_string(), + action: SharedWorldOwnerAction::AttachOrCreate, + })); + + let err = client + .on_server_frame(ready_frame_with_shared_world( + 1, + shared_world_binding("wld_other", "orch-test", 0), + )) + .expect_err("world_id mismatch must fail closed"); + assert!(err.is_fatal()); + assert!(err + .to_string() + .contains("ready.shared_world.world_id mismatch")); +} diff --git a/crates/shell/tests/policy_discovery.rs b/crates/shell/tests/policy_discovery.rs index c1f91722e..953aed63f 100644 --- a/crates/shell/tests/policy_discovery.rs +++ b/crates/shell/tests/policy_discovery.rs @@ -3,12 +3,12 @@ mod common; use assert_cmd::Command; -use common::{substrate_shell_driver, temp_dir}; +use common::substrate_shell_driver; use serde_json::Value as JsonValue; use serde_yaml::Value as YamlValue; use std::fs; use std::path::PathBuf; -use tempfile::TempDir; +use tempfile::{Builder, TempDir}; struct PolicyFixture { _temp: TempDir, @@ -19,7 +19,10 @@ struct PolicyFixture { impl PolicyFixture { fn new() -> Self { - let temp = temp_dir("substrate-policy-"); + let temp = Builder::new() + .prefix("substrate-policy-") + .tempdir_in("/tmp") + .expect("failed to allocate integration test temp dir"); let home = temp.path().join("home"); fs::create_dir_all(&home).expect("create HOME fixture"); let substrate_home = temp.path().join("substrate-home"); diff --git a/crates/shell/tests/policy_routing_semantics.rs b/crates/shell/tests/policy_routing_semantics.rs index f8698300a..11c424682 100644 --- a/crates/shell/tests/policy_routing_semantics.rs +++ b/crates/shell/tests/policy_routing_semantics.rs @@ -3,11 +3,11 @@ mod common; use assert_cmd::Command; -use common::{substrate_shell_driver, temp_dir}; +use common::substrate_shell_driver; use serde_json::Value as JsonValue; use std::fs; use std::path::{Path, PathBuf}; -use tempfile::TempDir; +use tempfile::{Builder, TempDir}; struct Pcm2Fixture { _temp: TempDir, @@ -19,7 +19,10 @@ struct Pcm2Fixture { impl Pcm2Fixture { fn new(prefix: &str) -> Self { - let temp = temp_dir(prefix); + let temp = Builder::new() + .prefix(prefix) + .tempdir_in("/tmp") + .expect("failed to allocate integration test temp dir"); let home = temp.path().join("home"); fs::create_dir_all(&home).expect("create HOME"); let substrate_home = temp.path().join("substrate-home"); diff --git a/crates/shell/tests/repl_tty_disconnect_linux.rs b/crates/shell/tests/repl_tty_disconnect_linux.rs index a41cd8db3..7a3c71a7f 100644 --- a/crates/shell/tests/repl_tty_disconnect_linux.rs +++ b/crates/shell/tests/repl_tty_disconnect_linux.rs @@ -148,7 +148,6 @@ impl PtyRepl { cmd.env_remove("CI"); cmd.env_remove("GITHUB_ACTIONS"); cmd.env("SHELL", "/bin/bash"); - cmd.arg("--shim-skip"); let child = pair.slave.spawn_command(cmd).expect("spawn substrate"); diff --git a/crates/shell/tests/repl_tty_disconnect_macos.rs b/crates/shell/tests/repl_tty_disconnect_macos.rs index 2d8d26041..215f6706f 100644 --- a/crates/shell/tests/repl_tty_disconnect_macos.rs +++ b/crates/shell/tests/repl_tty_disconnect_macos.rs @@ -175,6 +175,7 @@ impl PtyRepl { cmd.env_remove("SUBSTRATE_WORLD_ID"); cmd.env_remove("CI"); cmd.env_remove("GITHUB_ACTIONS"); + cmd.env("SUBSTRATE_FORCE_REEDLINE", "1"); cmd.env("SHELL", "/bin/bash"); cmd.arg("--shim-skip"); diff --git a/crates/shell/tests/repl_world_first_routing_v1.rs b/crates/shell/tests/repl_world_first_routing_v1.rs index fda56ad87..768eeb655 100644 --- a/crates/shell/tests/repl_world_first_routing_v1.rs +++ b/crates/shell/tests/repl_world_first_routing_v1.rs @@ -13,6 +13,8 @@ use std::sync::{Arc, Mutex}; use std::time::{Duration, Instant}; use portable_pty::{native_pty_system, CommandBuilder, PtySize}; +#[cfg(any(target_os = "linux", target_os = "macos"))] +use support::MemberDispatchStreamScript; use support::{binary_path, ensure_substrate_built, temp_dir, ReplWorldAgentStub, StreamBehavior}; use tempfile::TempDir; @@ -168,6 +170,30 @@ fn wait_for_min_start_sessions( ); } +fn wait_for_min_start_sessions_with_output( + repl: &PtyRepl, + records: &Arc>, + min_starts: usize, + timeout: Duration, +) { + let deadline = Instant::now() + timeout; + while Instant::now() < deadline { + let guard = records.lock().expect("lock records"); + if guard.persistent_start_sessions.len() >= min_starts { + return; + } + drop(guard); + std::thread::sleep(Duration::from_millis(25)); + } + + let guard = records.lock().expect("lock records"); + panic!( + "timed out waiting for persistent start sessions >= {min_starts}; got {}; output:\n{}\nrecords: {guard:#?}", + guard.persistent_start_sessions.len(), + repl.output_string(), + ); +} + fn wait_for_min_records( records: &Arc>, min_execs: usize, @@ -194,6 +220,74 @@ fn wait_for_min_records( ); } +#[cfg(any(target_os = "linux", target_os = "macos"))] +fn wait_for_min_member_dispatch_requests( + records: &Arc>, + min_requests: usize, + timeout: Duration, +) { + let deadline = Instant::now() + timeout; + while Instant::now() < deadline { + let guard = records.lock().expect("lock records"); + if guard.member_dispatch_requests.len() >= min_requests { + return; + } + drop(guard); + std::thread::sleep(Duration::from_millis(25)); + } + + let guard = records.lock().expect("lock records"); + panic!( + "timed out waiting for member_dispatch requests >= {min_requests}; got {}; records: {guard:#?}", + guard.member_dispatch_requests.len(), + ); +} + +#[cfg(any(target_os = "linux", target_os = "macos"))] +fn wait_for_min_member_turn_submit_requests( + records: &Arc>, + min_requests: usize, + timeout: Duration, +) { + let deadline = Instant::now() + timeout; + while Instant::now() < deadline { + let guard = records.lock().expect("lock records"); + if guard.member_turn_submit_requests.len() >= min_requests { + return; + } + drop(guard); + std::thread::sleep(Duration::from_millis(25)); + } + + let guard = records.lock().expect("lock records"); + panic!( + "timed out waiting for member_turn_submit requests >= {min_requests}; got {}; records: {guard:#?}", + guard.member_turn_submit_requests.len(), + ); +} + +fn wait_for_min_execute_cancel_requests( + records: &Arc>, + min_requests: usize, + timeout: Duration, +) { + let deadline = Instant::now() + timeout; + while Instant::now() < deadline { + let guard = records.lock().expect("lock records"); + if guard.execute_cancel_requests.len() >= min_requests { + return; + } + drop(guard); + std::thread::sleep(Duration::from_millis(25)); + } + + let guard = records.lock().expect("lock records"); + panic!( + "timed out waiting for execute/cancel requests >= {min_requests}; got {}; records: {guard:#?}", + guard.execute_cancel_requests.len(), + ); +} + fn read_trace(trace_path: &Path) -> Vec { fs::read_to_string(trace_path) .expect("read trace") @@ -203,96 +297,119 @@ fn read_trace(trace_path: &Path) -> Vec { .collect() } -fn assert_world_network_payload( - start: &support::PersistentStartSessionRecord, - expected_isolate_network: bool, - expected_allowed_domains: &[&str], +#[cfg(target_os = "linux")] +fn write_orchestrator_runtime_world_config( + home_substrate: &Path, + fake_codex: &Path, + on_drift: &str, ) { - let isolate_network = start - .world_network - .get("isolate_network") - .and_then(|value| value.as_bool()) - .expect("world_network.isolate_network bool"); - assert_eq!( - isolate_network, expected_isolate_network, - "unexpected world_network.isolate_network" - ); - - let allowed_domains = start - .world_network - .get("allowed_domains") - .and_then(|value| value.as_array()) - .expect("world_network.allowed_domains array"); - let allowed_domains: Vec<&str> = allowed_domains - .iter() - .map(|value| value.as_str().expect("allowed_domains string")) - .collect(); - assert_eq!( - allowed_domains, expected_allowed_domains, - "unexpected world_network.allowed_domains" + fs::create_dir_all(home_substrate.join("agents")).expect("create agents dir"); + let config = format!( + r#"world: + enabled: true + anchor_mode: workspace + anchor_path: '' + caged: false + net: + filter: false +policy: + mode: observe +sync: + auto_sync: false + direction: from_world + conflict_policy: prefer_host + exclude: [] +agents: + enabled: true + hub: + orchestrator_agent_id: codex + world_restart: + on_drift: {on_drift} +"# ); + fs::write(home_substrate.join("config.yaml"), config).expect("write config.yaml"); + fs::write( + home_substrate.join("policy.yaml"), + "id: test-global-policy\nname: Test Global Policy\nworld_fs:\n host_visible: true\n fail_closed:\n routing: true\n write:\n enabled: true\nnet_allowed: []\ncmd_allowed: []\ncmd_denied: []\ncmd_isolated: []\nrequire_approval: false\nallow_shell_operators: true\nlimits:\n max_memory_mb: null\n max_cpu_percent: null\n max_runtime_ms: null\n max_egress_bytes: null\nmetadata: {}\nagents:\n allowed_backends:\n - cli:codex\n", + ) + .expect("write agent runtime policy"); + fs::write( + home_substrate.join("agents/codex.yaml"), + format!( + "version: 1\nid: codex\nconfig:\n kind: cli\n enabled: true\n protocol: uaa.agent.session\n execution:\n scope: host\n cli:\n binary: {}\n mode: persistent\n capabilities:\n session_start: true\n session_resume: true\n session_fork: true\n session_stop: true\n status_snapshot: true\n event_stream: true\n llm: true\n mcp_client: false\n", + fake_codex.display() + ), + ) + .expect("write codex agent file"); } -fn run_repl_routing_case(case: &ReplRoutingCase<'_>) { - let temp = temp_dir("substrate-repl-net-allowed-"); - let home = temp.path().join("home"); - let project = temp.path().join("project"); - let substrate_home = home.join(".substrate"); - fs::create_dir_all(&home).expect("create home"); - fs::create_dir_all(&project).expect("create project"); - fs::create_dir_all(&substrate_home).expect("create substrate home"); - fs::write(home.join(".substrate/trace.jsonl"), "").expect("seed trace"); - write_profile(&project); - write_policy_with_net_allowed(&substrate_home, true, case.net_allowed_yaml); - write_config(&substrate_home, case.world_net_filter); - - let sock_temp = short_socket_dir("sub-c3ws-routing-"); - let sock = sock_temp.path().join("world.sock"); - let server = ReplWorldAgentStub::start(&sock, StreamBehavior::Normal); - let records = server.records(); - - let mut repl = PtyRepl::spawn(&project, &home, &substrate_home, &sock, &[], &["--world"]); - repl.wait_for_output("Substrate v", Duration::from_secs(2)) - .expect("banner"); - wait_for_min_start_sessions(&records, 1, Duration::from_secs(3)); - repl.send_line("exit"); - - let (_code, _out) = repl.shutdown_graceful(Duration::from_secs(2)); - - let guard = records.lock().expect("lock records"); - let start = guard - .persistent_start_sessions - .first() - .expect("persistent start session"); - let net_allowed = start - .policy_snapshot - .get("net_allowed") - .and_then(|value| value.as_array()) - .expect("policy_snapshot.net_allowed array"); - let net_allowed: Vec<&str> = net_allowed - .iter() - .map(|value| value.as_str().expect("net_allowed string")) - .collect(); - assert_eq!( - net_allowed, case.expected_net_allowed, - "{}: unexpected canonical policy_snapshot.net_allowed", - case.name - ); - assert_world_network_payload( - start, - case.expected_isolate_network, - case.expected_allowed_domains, +#[cfg(any(target_os = "linux", target_os = "macos"))] +fn write_orchestrator_and_world_member_runtime_world_config( + home_substrate: &Path, + fake_orchestrator: &Path, + fake_member: &Path, + on_drift: &str, +) { + fs::create_dir_all(home_substrate.join("agents")).expect("create agents dir"); + let config = format!( + r#"world: + enabled: true + anchor_mode: workspace + anchor_path: '' + caged: false + net: + filter: false +policy: + mode: observe +sync: + auto_sync: false + direction: from_world + conflict_policy: prefer_host + exclude: [] +agents: + enabled: true + hub: + orchestrator_agent_id: claude_code + world_restart: + on_drift: {on_drift} +"# ); + fs::write(home_substrate.join("config.yaml"), config).expect("write config.yaml"); + write_member_runtime_policy(home_substrate, true); + fs::write( + home_substrate.join("agents/claude_code.yaml"), + format!( + "version: 1\nid: claude_code\nconfig:\n kind: cli\n enabled: true\n protocol: uaa.agent.session\n execution:\n scope: host\n cli:\n binary: {}\n mode: persistent\n capabilities:\n session_start: true\n session_resume: true\n session_fork: true\n session_stop: true\n status_snapshot: true\n event_stream: true\n llm: true\n mcp_client: false\n", + fake_orchestrator.display() + ), + ) + .expect("write claude_code agent file"); + fs::write( + home_substrate.join("agents/codex.yaml"), + format!( + "version: 1\nid: codex\nconfig:\n kind: cli\n enabled: true\n protocol: uaa.agent.session\n execution:\n scope: world\n cli:\n binary: {}\n mode: persistent\n capabilities:\n session_start: true\n session_resume: true\n session_fork: true\n session_stop: true\n status_snapshot: true\n event_stream: true\n llm: true\n mcp_client: false\n", + fake_member.display() + ), + ) + .expect("write codex agent file"); } -fn write_workspace_marker(workspace_root: &Path) { - let dir = workspace_root.join(".substrate"); - fs::create_dir_all(&dir).expect("create .substrate"); - let cfg = r#"world: +#[cfg(any(target_os = "linux", target_os = "macos"))] +fn write_dual_host_runtime_world_config( + home_substrate: &Path, + fake_orchestrator: &Path, + fake_secondary_host: &Path, + on_drift: &str, +) { + fs::create_dir_all(home_substrate.join("agents")).expect("create agents dir"); + let config = format!( + r#"world: enabled: true anchor_mode: workspace anchor_path: '' caged: false + net: + filter: false policy: mode: observe sync: @@ -300,366 +417,3291 @@ sync: direction: from_world conflict_policy: prefer_host exclude: [] -"#; - fs::write(dir.join("workspace.yaml"), cfg).expect("write workspace.yaml"); +agents: + enabled: true + hub: + orchestrator_agent_id: claude_code + world_restart: + on_drift: {on_drift} +"# + ); + fs::write(home_substrate.join("config.yaml"), config).expect("write config.yaml"); + write_member_runtime_policy(home_substrate, true); + fs::write( + home_substrate.join("agents/claude_code.yaml"), + format!( + "version: 1\nid: claude_code\nconfig:\n kind: cli\n enabled: true\n protocol: uaa.agent.session\n execution:\n scope: host\n cli:\n binary: {}\n mode: persistent\n capabilities:\n session_start: true\n session_resume: true\n session_fork: true\n session_stop: true\n status_snapshot: true\n event_stream: true\n llm: true\n mcp_client: false\n", + fake_orchestrator.display() + ), + ) + .expect("write claude_code agent file"); + fs::write( + home_substrate.join("agents/codex.yaml"), + format!( + "version: 1\nid: codex\nconfig:\n kind: cli\n enabled: true\n protocol: uaa.agent.session\n execution:\n scope: host\n cli:\n binary: {}\n mode: persistent\n capabilities:\n session_start: true\n session_resume: true\n session_fork: true\n session_stop: true\n status_snapshot: true\n event_stream: true\n llm: true\n mcp_client: false\n", + fake_secondary_host.display() + ), + ) + .expect("write codex agent file"); } -struct PtyRepl { - child: Box, - master: Option>, - waited: Option, - writer: Arc>>, - output: Arc>>, - reader_handle: Option>, - stop_reader: Arc, +#[cfg(any(target_os = "linux", target_os = "macos"))] +fn write_member_runtime_policy(home_substrate: &Path, require_world: bool) { + fs::create_dir_all(home_substrate).expect("create SUBSTRATE_HOME"); + let require_world = if require_world { "true" } else { "false" }; + let policy = format!( + r#"id: test-global-policy +name: Test Global Policy +world_fs: + host_visible: true + fail_closed: + routing: {require_world} + write: + enabled: true +net_allowed: [] +cmd_allowed: [] +cmd_denied: [] +cmd_isolated: [] +require_approval: false +allow_shell_operators: true +limits: + max_memory_mb: null + max_cpu_percent: null + max_runtime_ms: null + max_egress_bytes: null +metadata: {{}} +agents: + allowed_backends: + - cli:claude_code + - cli:codex +"# + ); + fs::write(home_substrate.join("policy.yaml"), policy).expect("write policy.yaml"); } -impl PtyRepl { - fn spawn( - project_dir: &Path, - home_dir: &Path, - substrate_home: &Path, - socket_path: &Path, - extra_env: &[(&str, &str)], - args: &[&str], - ) -> Self { - ensure_substrate_built(); +#[cfg(any(target_os = "linux", target_os = "macos"))] +fn write_fake_codex_script(temp: &Path) -> PathBuf { + let path = temp.join("fake-codex.sh"); + let body = "#!/bin/sh\ntrap 'exit 0' INT TERM\nprintf '{\"type\":\"thread.started\",\"thread_id\":\"thread-test\"}\\r\\n'\nprintf '{\"type\":\"turn.started\",\"thread_id\":\"thread-test\",\"turn_id\":\"turn-1\"}\\r\\n'\nwhile :; do sleep 1; done\n"; + fs::write(&path, body).expect("write fake codex script"); + let mut perms = fs::metadata(&path) + .expect("fake codex metadata") + .permissions(); + use std::os::unix::fs::PermissionsExt; + perms.set_mode(0o755); + fs::set_permissions(&path, perms).expect("set fake codex permissions"); + path +} - let pty_system = native_pty_system(); - let pair = pty_system - .openpty(PtySize { - rows: 24, - cols: 80, - pixel_width: 0, - pixel_height: 0, - }) - .expect("openpty"); +#[cfg(any(target_os = "linux", target_os = "macos"))] +fn write_fake_codex_script_with_invocation_log_and_stdio_capture( + temp: &Path, +) -> (PathBuf, PathBuf, PathBuf, PathBuf) { + let path = temp.join("fake-codex-with-stdio-capture.sh"); + let count_path = temp.join("fake-codex-with-stdio-capture.count"); + let args_dir = temp.join("fake-codex-captured-args"); + let stdin_dir = temp.join("fake-codex-captured-stdin"); + fs::create_dir_all(&args_dir).expect("create fake codex args dir"); + fs::create_dir_all(&stdin_dir).expect("create fake codex stdin dir"); + let body = format!( + "#!/bin/sh\nSTATE_FILE='{}'\nARGS_DIR='{}'\nSTDIN_DIR='{}'\ncount=0\nif [ -f \"$STATE_FILE\" ]; then\n count=$(cat \"$STATE_FILE\")\nfi\ncount=$((count + 1))\nprintf '%s' \"$count\" > \"$STATE_FILE\"\nprintf '%s\\n' \"$@\" > \"$ARGS_DIR/$count.args\"\ncat > \"$STDIN_DIR/$count.stdin\"\ntrap 'exit 0' INT TERM\nprintf '{{\"type\":\"thread.started\",\"thread_id\":\"thread-test\"}}\\r\\n'\nprintf '{{\"type\":\"turn.started\",\"thread_id\":\"thread-test\",\"turn_id\":\"turn-1\"}}\\r\\n'\nwhile :; do sleep 1; done\n", + count_path.display(), + args_dir.display(), + stdin_dir.display() + ); + fs::write(&path, body).expect("write fake codex script with stdio capture"); + let mut perms = fs::metadata(&path) + .expect("fake codex metadata") + .permissions(); + use std::os::unix::fs::PermissionsExt; + perms.set_mode(0o755); + fs::set_permissions(&path, perms).expect("set fake codex permissions"); + (path, count_path, args_dir, stdin_dir) +} - let master = pair.master; +#[cfg(any(target_os = "linux", target_os = "macos"))] +fn write_fake_claude_script(temp: &Path) -> PathBuf { + let path = temp.join("fake-claude.sh"); + let body = "#!/bin/sh\ntrap 'exit 0' INT TERM\nprintf '{\"type\":\"system\",\"subtype\":\"init\",\"session_id\":\"session-test\"}\\r\\n'\nwhile :; do sleep 1; done\n"; + fs::write(&path, body).expect("write fake claude script"); + let mut perms = fs::metadata(&path) + .expect("fake claude metadata") + .permissions(); + use std::os::unix::fs::PermissionsExt; + perms.set_mode(0o755); + fs::set_permissions(&path, perms).expect("set fake claude permissions"); + path +} - #[cfg(unix)] - let master_fd = master.as_raw_fd(); +#[cfg(any(target_os = "linux", target_os = "macos"))] +fn write_fake_claude_script_first_session_then_exit(temp: &Path) -> PathBuf { + let path = temp.join("fake-claude-first-session-then-exit.sh"); + let state_path = temp.join("fake-claude-first-session-then-exit.count"); + let body = format!( + "#!/bin/sh\nSTATE_FILE='{}'\ncount=0\nif [ -f \"$STATE_FILE\" ]; then\n count=$(cat \"$STATE_FILE\")\nfi\ncount=$((count + 1))\nprintf '%s' \"$count\" > \"$STATE_FILE\"\nif [ \"$count\" -eq 1 ]; then\n trap 'exit 0' INT TERM\n printf '{{\"type\":\"system\",\"subtype\":\"init\",\"session_id\":\"session-test\"}}\\r\\n'\n while :; do sleep 1; done\nfi\nexit 0\n", + state_path.display() + ); + fs::write(&path, body).expect("write fake claude first-session-then-exit script"); + let mut perms = fs::metadata(&path) + .expect("fake claude first-session-then-exit metadata") + .permissions(); + use std::os::unix::fs::PermissionsExt; + perms.set_mode(0o755); + fs::set_permissions(&path, perms).expect("set fake claude first-session-then-exit permissions"); + path +} - #[cfg(unix)] - if let Some(fd) = master_fd { - set_fd_nonblocking(fd); - } +#[cfg(any(target_os = "linux", target_os = "macos"))] +fn write_host_orchestrator_runtime_world_config( + home_substrate: &Path, + fake_codex: &Path, + on_drift: &str, +) { + fs::create_dir_all(home_substrate.join("agents")).expect("create agents dir"); + let config = format!( + r#"world: + enabled: true + anchor_mode: workspace + anchor_path: '' + caged: false + net: + filter: false +policy: + mode: observe +sync: + auto_sync: false + direction: from_world + conflict_policy: prefer_host + exclude: [] +agents: + enabled: true + hub: + orchestrator_agent_id: codex + world_restart: + on_drift: {on_drift} +"# + ); + fs::write(home_substrate.join("config.yaml"), config).expect("write config.yaml"); + fs::write( + home_substrate.join("policy.yaml"), + "id: test-global-policy\nname: Test Global Policy\nworld_fs:\n host_visible: true\n fail_closed:\n routing: true\n write:\n enabled: true\nnet_allowed: []\ncmd_allowed: []\ncmd_denied: []\ncmd_isolated: []\nrequire_approval: false\nallow_shell_operators: true\nlimits:\n max_memory_mb: null\n max_cpu_percent: null\n max_runtime_ms: null\n max_egress_bytes: null\nmetadata: {}\nagents:\n allowed_backends:\n - cli:codex\n", + ) + .expect("write agent runtime policy"); + fs::write( + home_substrate.join("agents/codex.yaml"), + format!( + "version: 1\nid: codex\nconfig:\n kind: cli\n enabled: true\n protocol: uaa.agent.session\n execution:\n scope: host\n cli:\n binary: {}\n mode: persistent\n capabilities:\n session_start: true\n session_resume: true\n session_fork: true\n session_stop: true\n status_snapshot: true\n event_stream: true\n llm: true\n mcp_client: false\n", + fake_codex.display() + ), + ) + .expect("write codex agent file"); +} - let mut cmd = CommandBuilder::new(binary_path()); - cmd.args(args); - cmd.cwd(project_dir); - cmd.env("HOME", home_dir); - cmd.env("USERPROFILE", home_dir); - cmd.env("SUBSTRATE_HOME", substrate_home); - cmd.env("SUBSTRATE_MANAGER_MANIFEST", manager_manifest_path()); - cmd.env("SUBSTRATE_CAGED", "0"); - cmd.arg("--uncaged"); - cmd.env("SHIM_TRACE_LOG", home_dir.join(".substrate/trace.jsonl")); - cmd.env("SUBSTRATE_WORLD_SOCKET", socket_path); - cmd.env("SUBSTRATE_OVERRIDE_WORLD", "enabled"); - cmd.env_remove("SHIM_ORIGINAL_PATH"); - cmd.env_remove("SUBSTRATE_WORLD"); - cmd.env_remove("SUBSTRATE_WORLD_ENABLED"); - cmd.env_remove("SUBSTRATE_WORLD_ID"); - cmd.env("SHELL", "/bin/bash"); - cmd.arg("--shim-skip"); - for (k, v) in extra_env { - cmd.env(k, v); +#[cfg(target_os = "linux")] +fn write_fake_codex_script_first_success_then_fail_without_session_handle(temp: &Path) -> PathBuf { + let path = temp.join("fake-codex-first-success-then-fail.sh"); + let state_path = temp.join("fake-codex-first-success-then-fail.count"); + let body = format!( + "#!/bin/sh\nSTATE_FILE='{}'\ncount=0\nif [ -f \"$STATE_FILE\" ]; then\n count=$(cat \"$STATE_FILE\")\nfi\ncount=$((count + 1))\nprintf '%s' \"$count\" > \"$STATE_FILE\"\nif [ \"$count\" -eq 1 ]; then\n trap 'exit 0' INT TERM\n printf '{{\"type\":\"thread.started\",\"thread_id\":\"thread-test\"}}\\r\\n'\n printf '{{\"type\":\"turn.started\",\"thread_id\":\"thread-test\",\"turn_id\":\"turn-1\"}}\\r\\n'\n while :; do sleep 1; done\nfi\nprintf 'bootstrap-without-session-handle\\n'\n", + state_path.display() + ); + fs::write(&path, body).expect("write fake codex replacement failure script"); + let mut perms = fs::metadata(&path) + .expect("fake codex metadata") + .permissions(); + use std::os::unix::fs::PermissionsExt; + perms.set_mode(0o755); + fs::set_permissions(&path, perms).expect("set fake codex permissions"); + path +} + +#[cfg(target_os = "linux")] +fn write_fake_codex_script_without_session_handle(temp: &Path) -> PathBuf { + let path = temp.join("fake-codex-no-session-handle.sh"); + let body = "#!/bin/sh\nprintf 'bootstrap-without-session-handle\\n'\n"; + fs::write(&path, body).expect("write fake codex script without session handle"); + let mut perms = fs::metadata(&path) + .expect("fake codex metadata") + .permissions(); + use std::os::unix::fs::PermissionsExt; + perms.set_mode(0o755); + fs::set_permissions(&path, perms).expect("set fake codex permissions"); + path +} + +#[cfg(any(target_os = "linux", target_os = "macos"))] +fn read_invocation_count(path: &Path) -> usize { + fs::read_to_string(path) + .ok() + .and_then(|value| value.trim().parse::().ok()) + .unwrap_or(0) +} + +#[cfg(any(target_os = "linux", target_os = "macos"))] +fn load_single_orchestration_session_id(substrate_home: &Path) -> String { + let sessions_dir = sessions_dir(substrate_home); + let mut canonical_entries = fs::read_dir(&sessions_dir) + .expect("read orchestration session dir") + .filter_map(Result::ok) + .map(|entry| entry.path()) + .filter(|path| path.is_dir()) + .map(|path| path.join("session.json")) + .filter(|path| path.is_file()) + .collect::>(); + canonical_entries.sort(); + if let Some(session_path) = canonical_entries.into_iter().next() { + return serde_json::from_str::( + &fs::read_to_string(session_path).expect("read canonical session file"), + ) + .expect("parse canonical session file") + .get("orchestration_session_id") + .and_then(Value::as_str) + .expect("session orchestration_session_id") + .to_string(); + } + + let mut entries = fs::read_dir(&sessions_dir) + .expect("read orchestration session dir") + .filter_map(Result::ok) + .map(|entry| entry.path()) + .filter(|path| path.extension().and_then(|ext| ext.to_str()) == Some("json")) + .collect::>(); + entries.sort(); + let session_path = entries + .into_iter() + .next() + .expect("orchestration session file"); + serde_json::from_str::(&fs::read_to_string(session_path).expect("read session file")) + .expect("parse session file") + .get("orchestration_session_id") + .and_then(Value::as_str) + .expect("session orchestration_session_id") + .to_string() +} + +#[cfg(any(target_os = "linux", target_os = "macos"))] +fn sessions_dir(substrate_home: &Path) -> PathBuf { + substrate_home.join("run/agent-hub/sessions") +} + +#[cfg(any(target_os = "linux", target_os = "macos"))] +fn has_any_persisted_orchestration_session(substrate_home: &Path) -> bool { + let sessions_dir = sessions_dir(substrate_home); + fs::read_dir(&sessions_dir) + .ok() + .into_iter() + .flat_map(|entries| entries.filter_map(Result::ok)) + .map(|entry| entry.path()) + .any(|path| { + if path.is_dir() { + path.join("session.json").is_file() + } else { + path.extension().and_then(|ext| ext.to_str()) == Some("json") + } + }) +} + +#[cfg(any(target_os = "linux", target_os = "macos"))] +fn orchestration_session_path(substrate_home: &Path, orchestration_session_id: &str) -> PathBuf { + sessions_dir(substrate_home) + .join(orchestration_session_id) + .join("session.json") +} + +#[cfg(any(target_os = "linux", target_os = "macos"))] +fn canonical_participants_dir(substrate_home: &Path, orchestration_session_id: &str) -> PathBuf { + sessions_dir(substrate_home) + .join(orchestration_session_id) + .join("participants") +} + +#[cfg(target_os = "linux")] +fn canonical_participant_path( + substrate_home: &Path, + orchestration_session_id: &str, + participant_id: &str, +) -> PathBuf { + canonical_participants_dir(substrate_home, orchestration_session_id) + .join(format!("{participant_id}.json")) +} + +#[cfg(any(target_os = "linux", target_os = "macos"))] +fn read_orchestration_session(session_path: &Path) -> Value { + serde_json::from_str(&fs::read_to_string(session_path).expect("read orchestration session")) + .expect("parse orchestration session") +} + +#[cfg(target_os = "linux")] +fn assert_session_world_binding( + session: &Value, + expected_world_id: Option<&str>, + expected_world_generation: Option, +) { + assert_eq!( + session.pointer("/world_id").and_then(Value::as_str), + expected_world_id, + "unexpected persisted parent world_id: {session:?}" + ); + assert_eq!( + session.pointer("/world_generation").and_then(Value::as_u64), + expected_world_generation, + "unexpected persisted parent world_generation: {session:?}" + ); +} + +#[cfg(any(target_os = "linux", target_os = "macos"))] +fn flat_participants_dir(substrate_home: &Path) -> PathBuf { + substrate_home.join("run/agent-hub/participants") +} + +#[cfg(target_os = "linux")] +fn flat_participant_manifest_path(substrate_home: &Path, participant_id: &str) -> PathBuf { + flat_participants_dir(substrate_home).join(format!("{participant_id}.json")) +} + +#[cfg(target_os = "linux")] +fn read_trace_lenient(trace_path: &Path) -> Vec { + fs::read_to_string(trace_path) + .unwrap_or_default() + .lines() + .filter(|line| !line.trim().is_empty()) + .filter_map(|line| serde_json::from_str(line).ok()) + .collect() +} + +#[cfg(target_os = "linux")] +fn read_participant_manifest(substrate_home: &Path, participant_id: &str) -> Value { + let canonical_path = fs::read_dir(sessions_dir(substrate_home)) + .ok() + .into_iter() + .flat_map(|entries| entries.filter_map(Result::ok)) + .map(|entry| entry.path()) + .filter(|path| path.is_dir()) + .map(|path| { + path.join("participants") + .join(format!("{participant_id}.json")) + }) + .find(|path| path.is_file()); + let path = canonical_path + .unwrap_or_else(|| flat_participant_manifest_path(substrate_home, participant_id)); + serde_json::from_str(&fs::read_to_string(path).expect("read participant manifest")) + .expect("parse participant manifest") +} + +#[cfg(target_os = "linux")] +fn write_live_world_member_manifest( + substrate_home: &Path, + orchestration_session_id: &str, + participant_id: &str, + orchestrator_participant_id: &str, + world_id: &str, + world_generation: u64, +) { + fs::create_dir_all(canonical_participants_dir( + substrate_home, + orchestration_session_id, + )) + .expect("create canonical participants dir"); + let ts = "2026-04-30T12:00:00Z"; + let payload = serde_json::json!({ + "participant_id": participant_id, + "orchestration_session_id": orchestration_session_id, + "agent_id": "codex", + "backend_id": "cli:codex", + "role": "member", + "protocol": "uaa.agent.session", + "execution": { "scope": "world" }, + "state": "ready", + "opened_at": ts, + "last_transition_at": ts, + "world_id": world_id, + "world_generation": world_generation, + "parent_session_handle_id": Value::Null, + "resumed_from_session_handle_id": Value::Null, + "orchestrator_participant_id": orchestrator_participant_id, + "internal": { + "resolved_agent_kind": "codex", + "resolved_binary_path": "/usr/bin/codex", + "shell_owner_pid": std::process::id(), + "lease_token": format!("lease_{participant_id}"), + "uaa_session_id": "uaa_session", + "latest_run_id": Value::Null, + "cancel_supported": true, + "control_owner_retained": true, + "event_stream_active": true, + "completion_observer_retained": true, + "ownership_mode": "member_runtime", + "ownership_valid": true, + "ownership_verified_at": ts, + "last_heartbeat_at": ts, + "last_event_at": ts, + "terminal_observed_at": Value::Null, + "termination_reason": Value::Null, + "last_error_bucket": Value::Null, + "last_error_message": Value::Null } + }); + fs::write( + canonical_participant_path(substrate_home, orchestration_session_id, participant_id), + serde_json::to_vec_pretty(&payload).expect("serialize participant manifest"), + ) + .expect("write participant manifest"); +} - let child = pair.slave.spawn_command(cmd).expect("spawn substrate"); - let writer: Arc>> = - Arc::new(Mutex::new(master.take_writer().expect("take writer"))); +#[cfg(any(target_os = "linux", target_os = "macos"))] +fn session_participant_manifests( + substrate_home: &Path, + orchestration_session_id: &str, +) -> Vec { + use std::collections::BTreeMap; + + let mut manifests = BTreeMap::new(); + + let canonical_dir = canonical_participants_dir(substrate_home, orchestration_session_id); + for path in fs::read_dir(canonical_dir) + .ok() + .into_iter() + .flat_map(|entries| entries.filter_map(Result::ok)) + .map(|entry| entry.path()) + .filter(|path| path.extension().and_then(|ext| ext.to_str()) == Some("json")) + { + let manifest = serde_json::from_str::( + &fs::read_to_string(path).expect("read canonical participant file"), + ) + .expect("parse canonical participant file"); + let Some(participant_id) = manifest + .get("participant_id") + .and_then(Value::as_str) + .map(str::to_string) + else { + continue; + }; + manifests.insert(participant_id, manifest); + } - let output = Arc::new(Mutex::new(Vec::new())); - let stop_reader = Arc::new(AtomicBool::new(false)); - let output_for_thread = output.clone(); - let writer_for_thread = writer.clone(); - let stop_for_thread = stop_reader.clone(); - let reader_handle = std::thread::spawn(move || { - let mut buf = [0u8; 4096]; - let mut carry = Vec::::new(); - loop { - if stop_for_thread.load(Ordering::Relaxed) { - break; - } + for manifest in fs::read_dir(flat_participants_dir(substrate_home)) + .ok() + .into_iter() + .flat_map(|entries| entries.filter_map(Result::ok)) + .map(|entry| entry.path()) + .filter(|path| path.extension().and_then(|ext| ext.to_str()) == Some("json")) + .map(|path| { + serde_json::from_str::( + &fs::read_to_string(path).expect("read flat participant file"), + ) + .expect("parse flat participant file") + }) + .filter(|manifest| { + manifest + .get("orchestration_session_id") + .and_then(Value::as_str) + == Some(orchestration_session_id) + }) + { + let Some(participant_id) = manifest + .get("participant_id") + .and_then(Value::as_str) + .map(str::to_string) + else { + continue; + }; + manifests.entry(participant_id).or_insert(manifest); + } - #[cfg(unix)] - if let Some(fd) = master_fd { - let rc = unsafe { libc::read(fd, buf.as_mut_ptr().cast(), buf.len()) }; - if rc == 0 { - break; - } - if rc < 0 { - let e = std::io::Error::last_os_error(); - if e.kind() == std::io::ErrorKind::WouldBlock { - std::thread::sleep(Duration::from_millis(25)); - continue; - } - break; - } + manifests.into_values().collect() +} - let n: usize = rc as usize; - // Reedline/crossterm may emit terminal queries which require a response - // from the terminal emulator. When running under a raw PTY in tests, - // provide minimal responses so the REPL can make progress. - // - // - DSR (cursor position): ESC [ 6 n → ESC [ 1 ; 1 R - // - Window size request: ESC [ 18 t → ESC [ 8 ; rows ; cols t - let chunk = &buf[..n]; - // Some terminals split these query bytes across reads. Use a small - // rolling carry buffer to detect queries across chunk boundaries. - let mut probe = carry.clone(); - probe.extend_from_slice(chunk); +#[cfg(any(target_os = "linux", target_os = "macos"))] +fn participant_is_authoritative_live(manifest: &Value) -> bool { + let Some(state) = manifest.get("state").and_then(Value::as_str) else { + return false; + }; + let live_state = matches!( + state, + "allocating" | "ready" | "running" | "restarting" | "stopping" + ); + live_state + && manifest + .pointer("/internal/uaa_session_id") + .and_then(Value::as_str) + .is_some() + && manifest + .pointer("/internal/control_owner_retained") + .and_then(Value::as_bool) + == Some(true) + && manifest + .pointer("/internal/event_stream_active") + .and_then(Value::as_bool) + == Some(true) + && manifest + .pointer("/internal/completion_observer_retained") + .and_then(Value::as_bool) + == Some(true) + && manifest + .pointer("/internal/ownership_valid") + .and_then(Value::as_bool) + == Some(true) + && manifest + .pointer("/internal/terminal_observed_at") + .is_none_or(Value::is_null) +} - if probe.windows(4).any(|w| w == b"\x1b[6n") { - if let Ok(mut w) = writer_for_thread.lock() { - let _ = w.write_all(b"\x1b[1;1R"); - let _ = w.flush(); - } - } - if probe.windows(5).any(|w| w == b"\x1b[18t") { - if let Ok(mut w) = writer_for_thread.lock() { - let _ = w.write_all(b"\x1b[8;24;80t"); - let _ = w.flush(); - } - } +#[cfg(any(target_os = "linux", target_os = "macos"))] +fn has_any_authoritative_live_host_participant(substrate_home: &Path) -> bool { + fs::read_dir(flat_participants_dir(substrate_home)) + .ok() + .into_iter() + .flat_map(|entries| entries.filter_map(Result::ok)) + .map(|entry| entry.path()) + .filter(|path| path.extension().and_then(|ext| ext.to_str()) == Some("json")) + .map(|path| { + serde_json::from_str::(&fs::read_to_string(path).expect("read participant file")) + .expect("parse participant file") + }) + .any(|manifest| { + manifest.get("role").and_then(Value::as_str) == Some("orchestrator") + && manifest.pointer("/execution/scope").and_then(Value::as_str) == Some("host") + && participant_is_authoritative_live(&manifest) + }) +} - carry.clear(); - let keep = 8usize; - if probe.len() > keep { - carry.extend_from_slice(&probe[probe.len() - keep..]); - } else { - carry.extend_from_slice(&probe); - } +#[cfg(any(target_os = "linux", target_os = "macos"))] +fn assert_no_persisted_host_orchestrator_state_for_interval( + substrate_home: &Path, + interval: Duration, +) { + let deadline = Instant::now() + interval; + while Instant::now() < deadline { + assert!( + !has_any_persisted_orchestration_session(substrate_home), + "orchestration session must remain absent before the first targeted host turn" + ); + assert!( + !has_any_authoritative_live_host_participant(substrate_home), + "authoritative-live host participant must remain absent before the first targeted host turn" + ); + std::thread::sleep(Duration::from_millis(25)); + } +} - if let Ok(mut guard) = output_for_thread.lock() { - guard.extend_from_slice(chunk); - } - continue; - } +#[cfg(target_os = "linux")] +fn live_world_member_generations_for_session( + substrate_home: &Path, + orchestration_session_id: &str, +) -> Vec<(String, u64)> { + let mut live = session_participant_manifests(substrate_home, orchestration_session_id) + .into_iter() + .filter(|manifest| manifest.get("role").and_then(Value::as_str) == Some("member")) + .filter(|manifest| { + manifest.pointer("/execution/scope").and_then(Value::as_str) == Some("world") + }) + .filter(participant_is_authoritative_live) + .filter_map(|manifest| { + Some(( + manifest.get("participant_id")?.as_str()?.to_string(), + manifest.get("world_generation")?.as_u64()?, + )) + }) + .collect::>(); + live.sort(); + live +} - std::thread::sleep(Duration::from_millis(25)); - } - }); +#[cfg(any(target_os = "linux", target_os = "macos"))] +fn world_member_manifests_for_session( + substrate_home: &Path, + orchestration_session_id: &str, +) -> Vec { + let mut manifests = session_participant_manifests(substrate_home, orchestration_session_id) + .into_iter() + .filter(|manifest| manifest.get("role").and_then(Value::as_str) == Some("member")) + .filter(|manifest| { + manifest.pointer("/execution/scope").and_then(Value::as_str) == Some("world") + }) + .collect::>(); + manifests.sort_by(|left, right| { + left.get("participant_id") + .and_then(Value::as_str) + .cmp(&right.get("participant_id").and_then(Value::as_str)) + }); + manifests +} - Self { - child, - master: Some(master), - waited: None, - writer, - output, - reader_handle: Some(reader_handle), - stop_reader, +#[cfg(any(target_os = "linux", target_os = "macos"))] +fn authoritative_live_participant_manifests_for_session( + substrate_home: &Path, + orchestration_session_id: &str, +) -> Vec { + let mut manifests = session_participant_manifests(substrate_home, orchestration_session_id) + .into_iter() + .filter(participant_is_authoritative_live) + .collect::>(); + manifests.sort_by(|left, right| { + left.get("backend_id") + .and_then(Value::as_str) + .cmp(&right.get("backend_id").and_then(Value::as_str)) + }); + manifests +} + +#[cfg(any(target_os = "linux", target_os = "macos"))] +fn authoritative_live_participant_manifest_for_backend<'a>( + manifests: &'a [Value], + backend_id: &str, +) -> &'a Value { + manifests + .iter() + .find(|manifest| manifest.get("backend_id").and_then(Value::as_str) == Some(backend_id)) + .unwrap_or_else(|| panic!("missing authoritative-live participant for {backend_id}")) +} + +#[cfg(any(target_os = "linux", target_os = "macos"))] +fn authoritative_live_world_member_manifests_for_session( + substrate_home: &Path, + orchestration_session_id: &str, +) -> Vec { + world_member_manifests_for_session(substrate_home, orchestration_session_id) + .into_iter() + .filter(participant_is_authoritative_live) + .collect() +} + +#[cfg(any(target_os = "linux", target_os = "macos"))] +fn assert_world_member_absent_for_interval( + substrate_home: &Path, + orchestration_session_id: &str, + interval: Duration, +) { + let deadline = Instant::now() + interval; + while Instant::now() < deadline { + let members = authoritative_live_world_member_manifests_for_session( + substrate_home, + orchestration_session_id, + ); + assert!( + members.is_empty(), + "world member must remain absent before the first world-backed command: {members:?}" + ); + std::thread::sleep(Duration::from_millis(25)); + } +} + +#[cfg(any(target_os = "linux", target_os = "macos"))] +fn wait_for_live_world_member_count( + substrate_home: &Path, + orchestration_session_id: &str, + expected_count: usize, + timeout: Duration, +) -> Vec { + let deadline = Instant::now() + timeout; + while Instant::now() < deadline { + let members = authoritative_live_world_member_manifests_for_session( + substrate_home, + orchestration_session_id, + ); + if members.len() == expected_count { + return members; } + std::thread::sleep(Duration::from_millis(25)); } - fn send_line(&mut self, line: &str) { - if let Ok(mut w) = self.writer.lock() { - let _ = w.write_all(line.as_bytes()); - // Use LF for Enter so both the stdio prompt worker (read_line) and - // Reedline-backed prompt worker reliably consume the line under PTY - // harnesses in CI. - let _ = w.write_all(b"\n"); - let _ = w.flush(); + panic!( + "timed out waiting for authoritative live world member count == {expected_count}; got {:?}", + authoritative_live_world_member_manifests_for_session( + substrate_home, + orchestration_session_id + ), + ); +} + +#[cfg(target_os = "linux")] +fn wait_for_world_restarted_alert_without_stale_liveness( + trace_path: &Path, + substrate_home: &Path, + orchestration_session_id: &str, + stale_generation: u64, + timeout: Duration, +) -> Value { + let deadline = Instant::now() + timeout; + while Instant::now() < deadline { + let events = read_trace_lenient(trace_path); + let alert = events.into_iter().find(|event| { + event.get("event_type").and_then(Value::as_str) == Some("agent_event") + && event.get("kind").and_then(Value::as_str) == Some("alert") + && event.pointer("/data/code").and_then(Value::as_str) == Some("world_restarted") + && event + .get("orchestration_session_id") + .and_then(Value::as_str) + == Some(orchestration_session_id) + }); + let stale_live = + live_world_member_generations_for_session(substrate_home, orchestration_session_id) + .into_iter() + .filter(|(_, generation)| *generation == stale_generation) + .collect::>(); + if let Some(alert) = alert { + assert!( + stale_live.is_empty(), + "world_restarted published before stale generation {stale_generation} invalidated: alert={alert:?} stale_live={stale_live:?}" + ); + return alert; } + std::thread::sleep(Duration::from_millis(10)); } - fn output_string(&self) -> String { - let guard = self.output.lock().expect("lock output"); - String::from_utf8_lossy(&guard).into_owned() - } + panic!( + "timed out waiting for world_restarted without stale live generation {stale_generation}; live={:?}; trace={:?}", + live_world_member_generations_for_session(substrate_home, orchestration_session_id), + read_trace_lenient(trace_path), + ); +} + +fn alert_rows_by_code<'a>(events: &'a [Value], code: &str) -> Vec<&'a Value> { + events + .iter() + .filter(|event| event.get("event_type").and_then(Value::as_str) == Some("agent_event")) + .filter(|event| event.get("kind").and_then(Value::as_str) == Some("alert")) + .filter(|event| event.pointer("/data/code").and_then(Value::as_str) == Some(code)) + .collect() +} + +fn assert_no_alert_rows_by_code(events: &[Value], code: &str, scenario: &str) { + let alerts = alert_rows_by_code(events, code); + assert!( + alerts.is_empty(), + "{scenario}: explicit-context-only suppression must prevent `{code}` agent_event rows when no shared-world context exists: {alerts:?}" + ); +} + +fn assert_start_sessions_have_no_shared_world_context( + records: &support::ReplWorldAgentRecords, + scenario: &str, +) { + assert!( + records + .persistent_start_sessions + .iter() + .all(|start| start.shared_world.is_none()), + "{scenario}: harness only exposes explicit orchestration context through start_session.shared_world; unexpected shared-world requests: {:#?}", + records.persistent_start_sessions + ); +} + +fn assert_world_network_payload( + start: &support::PersistentStartSessionRecord, + expected_isolate_network: bool, + expected_allowed_domains: &[&str], +) { + let isolate_network = start + .world_network + .get("isolate_network") + .and_then(|value| value.as_bool()) + .expect("world_network.isolate_network bool"); + assert_eq!( + isolate_network, expected_isolate_network, + "unexpected world_network.isolate_network" + ); + + let allowed_domains = start + .world_network + .get("allowed_domains") + .and_then(|value| value.as_array()) + .expect("world_network.allowed_domains array"); + let allowed_domains: Vec<&str> = allowed_domains + .iter() + .map(|value| value.as_str().expect("allowed_domains string")) + .collect(); + assert_eq!( + allowed_domains, expected_allowed_domains, + "unexpected world_network.allowed_domains" + ); +} + +fn run_repl_routing_case(case: &ReplRoutingCase<'_>) { + let temp = temp_dir("substrate-repl-net-allowed-"); + let home = temp.path().join("home"); + let project = temp.path().join("project"); + let substrate_home = home.join(".substrate"); + fs::create_dir_all(&home).expect("create home"); + fs::create_dir_all(&project).expect("create project"); + fs::create_dir_all(&substrate_home).expect("create substrate home"); + fs::write(home.join(".substrate/trace.jsonl"), "").expect("seed trace"); + write_profile(&project); + write_policy_with_net_allowed(&substrate_home, true, case.net_allowed_yaml); + write_config(&substrate_home, case.world_net_filter); + + let sock_temp = short_socket_dir("sub-c3ws-routing-"); + let sock = sock_temp.path().join("world.sock"); + let server = ReplWorldAgentStub::start(&sock, StreamBehavior::Normal); + let records = server.records(); + + let mut repl = PtyRepl::spawn(&project, &home, &substrate_home, &sock, &[], &["--world"]); + repl.wait_for_output("Substrate v", Duration::from_secs(6)) + .expect("banner"); + wait_for_min_start_sessions_with_output(&repl, &records, 1, Duration::from_secs(3)); + repl.send_line("exit"); + + let (_code, _out) = repl.shutdown_graceful(Duration::from_secs(2)); + + let guard = records.lock().expect("lock records"); + assert_start_sessions_have_no_shared_world_context(&guard, case.name); + let start = guard + .persistent_start_sessions + .first() + .expect("persistent start session"); + let net_allowed = start + .policy_snapshot + .get("net_allowed") + .and_then(|value| value.as_array()) + .expect("policy_snapshot.net_allowed array"); + let net_allowed: Vec<&str> = net_allowed + .iter() + .map(|value| value.as_str().expect("net_allowed string")) + .collect(); + assert_eq!( + net_allowed, case.expected_net_allowed, + "{}: unexpected canonical policy_snapshot.net_allowed", + case.name + ); + assert_world_network_payload( + start, + case.expected_isolate_network, + case.expected_allowed_domains, + ); +} + +fn write_workspace_marker(workspace_root: &Path) { + let dir = workspace_root.join(".substrate"); + fs::create_dir_all(&dir).expect("create .substrate"); + let cfg = r#"world: + enabled: true + anchor_mode: workspace + anchor_path: '' + caged: false +policy: + mode: observe +sync: + auto_sync: false + direction: from_world + conflict_policy: prefer_host + exclude: [] +"#; + fs::write(dir.join("workspace.yaml"), cfg).expect("write workspace.yaml"); +} + +struct PtyRepl { + child: Box, + master: Option>, + waited: Option, + writer: Arc>>, + output: Arc>>, + reader_handle: Option>, + stop_reader: Arc, +} + +impl PtyRepl { + fn spawn( + project_dir: &Path, + home_dir: &Path, + substrate_home: &Path, + socket_path: &Path, + extra_env: &[(&str, &str)], + args: &[&str], + ) -> Self { + Self::spawn_inner( + project_dir, + home_dir, + substrate_home, + socket_path, + extra_env, + args, + None, + ) + } + + fn spawn_inner( + project_dir: &Path, + home_dir: &Path, + substrate_home: &Path, + socket_path: &Path, + extra_env: &[(&str, &str)], + args: &[&str], + startup_delay: Option, + ) -> Self { + ensure_substrate_built(); + + let pty_system = native_pty_system(); + let pair = pty_system + .openpty(PtySize { + rows: 24, + cols: 80, + pixel_width: 0, + pixel_height: 0, + }) + .expect("openpty"); + + let master = pair.master; + + #[cfg(unix)] + let master_fd = master.as_raw_fd(); + + #[cfg(unix)] + if let Some(fd) = master_fd { + set_fd_nonblocking(fd); + } + + let substrate_binary = binary_path(); + let mut cmd = if let Some(startup_delay) = startup_delay { + let mut cmd = CommandBuilder::new("bash"); + cmd.arg("-lc"); + cmd.arg(format!( + "sleep {:.3}; exec \"$0\" \"$@\"", + startup_delay.as_secs_f64() + )); + cmd.arg(substrate_binary.clone()); + cmd + } else { + CommandBuilder::new(substrate_binary) + }; + cmd.args(args); + cmd.cwd(project_dir); + cmd.env("HOME", home_dir); + cmd.env("USERPROFILE", home_dir); + cmd.env("SUBSTRATE_HOME", substrate_home); + cmd.env("SUBSTRATE_MANAGER_MANIFEST", manager_manifest_path()); + cmd.env("SUBSTRATE_CAGED", "0"); + cmd.arg("--uncaged"); + cmd.env("SHIM_TRACE_LOG", home_dir.join(".substrate/trace.jsonl")); + cmd.env("SUBSTRATE_WORLD_SOCKET", socket_path); + cmd.env("SUBSTRATE_OVERRIDE_WORLD", "enabled"); + cmd.env_remove("SHIM_ORIGINAL_PATH"); + cmd.env_remove("SUBSTRATE_WORLD"); + cmd.env_remove("SUBSTRATE_WORLD_ENABLED"); + cmd.env_remove("SUBSTRATE_WORLD_ID"); + cmd.env("SHELL", "/bin/bash"); + cmd.arg("--shim-skip"); + for (k, v) in extra_env { + cmd.env(k, v); + } + + let child = pair.slave.spawn_command(cmd).expect("spawn substrate"); + let writer: Arc>> = + Arc::new(Mutex::new(master.take_writer().expect("take writer"))); + + let output = Arc::new(Mutex::new(Vec::new())); + let stop_reader = Arc::new(AtomicBool::new(false)); + let output_for_thread = output.clone(); + let writer_for_thread = writer.clone(); + let stop_for_thread = stop_reader.clone(); + let reader_handle = std::thread::spawn(move || { + let mut buf = [0u8; 4096]; + let mut carry = Vec::::new(); + loop { + if stop_for_thread.load(Ordering::Relaxed) { + break; + } + + #[cfg(unix)] + if let Some(fd) = master_fd { + let rc = unsafe { libc::read(fd, buf.as_mut_ptr().cast(), buf.len()) }; + if rc == 0 { + break; + } + if rc < 0 { + let e = std::io::Error::last_os_error(); + if e.kind() == std::io::ErrorKind::WouldBlock { + std::thread::sleep(Duration::from_millis(25)); + continue; + } + break; + } + + let n: usize = rc as usize; + // Reedline/crossterm may emit terminal queries which require a response + // from the terminal emulator. When running under a raw PTY in tests, + // provide minimal responses so the REPL can make progress. + // + // - DSR (cursor position): ESC [ 6 n → ESC [ 1 ; 1 R + // - Window size request: ESC [ 18 t → ESC [ 8 ; rows ; cols t + let chunk = &buf[..n]; + // Some terminals split these query bytes across reads. Use a small + // rolling carry buffer to detect queries across chunk boundaries. + let mut probe = carry.clone(); + probe.extend_from_slice(chunk); + + if probe.windows(4).any(|w| w == b"\x1b[6n") { + if let Ok(mut w) = writer_for_thread.lock() { + let _ = w.write_all(b"\x1b[1;1R"); + let _ = w.flush(); + } + } + if probe.windows(5).any(|w| w == b"\x1b[18t") { + if let Ok(mut w) = writer_for_thread.lock() { + let _ = w.write_all(b"\x1b[8;24;80t"); + let _ = w.flush(); + } + } + + carry.clear(); + let keep = 8usize; + if probe.len() > keep { + carry.extend_from_slice(&probe[probe.len() - keep..]); + } else { + carry.extend_from_slice(&probe); + } + + if let Ok(mut guard) = output_for_thread.lock() { + guard.extend_from_slice(chunk); + } + continue; + } + + std::thread::sleep(Duration::from_millis(25)); + } + }); + + Self { + child, + master: Some(master), + waited: None, + writer, + output, + reader_handle: Some(reader_handle), + stop_reader, + } + } + + fn send_line(&mut self, line: &str) { + if let Ok(mut w) = self.writer.lock() { + let _ = w.write_all(line.as_bytes()); + // Use LF for Enter so both the stdio prompt worker (read_line) and + // Reedline-backed prompt worker reliably consume the line under PTY + // harnesses in CI. + let _ = w.write_all(b"\n"); + let _ = w.flush(); + } + } + + fn output_string(&self) -> String { + let guard = self.output.lock().expect("lock output"); + String::from_utf8_lossy(&guard).into_owned() + } + + fn wait_for_output(&self, needle: &str, timeout: Duration) -> anyhow::Result<()> { + let deadline = Instant::now() + timeout; + while Instant::now() < deadline { + let out = self.output_string(); + if out.contains(needle) { + return Ok(()); + } + std::thread::sleep(Duration::from_millis(25)); + } + anyhow::bail!( + "timed out waiting for output containing `{}`; output so far:\n{}", + needle, + self.output_string() + ) + } + + fn wait_for_prompt(&self, timeout: Duration) -> anyhow::Result<()> { + self.wait_for_output("substrate> ", timeout) + } + + fn try_wait(&mut self) -> anyhow::Result { + if self.waited.is_some() { + return Ok(true); + } + if let Some(status) = self.child.try_wait()? { + self.waited = Some(status); + return Ok(true); + } + Ok(false) + } + + fn shutdown_graceful(mut self, timeout: Duration) -> (i32, String) { + let deadline = Instant::now() + timeout; + while self.waited.is_none() && Instant::now() < deadline { + match self.try_wait() { + Ok(true) => break, + Ok(false) => std::thread::sleep(Duration::from_millis(25)), + Err(_) => break, + } + } + self.shutdown() + } + + fn shutdown(mut self) -> (i32, String) { + self.stop_reader.store(true, Ordering::Relaxed); + self.master.take(); + if self.waited.is_none() { + let _ = self.child.kill(); + let status = self.child.wait().expect("wait child"); + self.waited = Some(status); + } + if let Some(handle) = self.reader_handle.take() { + let _ = handle.join(); + } + let code = self + .waited + .as_ref() + .map(|s| s.exit_code() as i32) + .unwrap_or(-1); + (code, self.output_string()) + } +} + +#[cfg(any(target_os = "linux", target_os = "macos"))] +fn launch_host_runtime_via_targeted_turn(repl: &mut PtyRepl, backend_id: &str) { + repl.wait_for_prompt(Duration::from_secs(2)) + .expect("prompt before targeted host launch"); + repl.send_line(&format!("::{backend_id} start retained host runtime")); + repl.wait_for_output( + "shell-owned orchestrator session is ready via retained attached control ownership", + Duration::from_secs(5), + ) + .expect("runtime ready event"); + repl.wait_for_prompt(Duration::from_secs(2)) + .expect("prompt after targeted host launch"); +} + +#[test] +#[serial] +fn c3_host_directive_is_gated_disabled_by_default() { + let temp = temp_dir("substrate-c3-host-gate-off-"); + let home = temp.path().join("home"); + let project = temp.path().join("project"); + let substrate_home = home.join(".substrate"); + fs::create_dir_all(&home).expect("create home"); + fs::create_dir_all(&project).expect("create project"); + fs::create_dir_all(&substrate_home).expect("create substrate home"); + fs::write(home.join(".substrate/trace.jsonl"), "").expect("seed trace"); + write_profile(&project); + write_policy(&substrate_home, true); + + let sock_temp = short_socket_dir("sub-c3ws-host-off-"); + let sock = sock_temp.path().join("world.sock"); + let _server = ReplWorldAgentStub::start(&sock, StreamBehavior::Normal); + + let mut repl = PtyRepl::spawn(&project, &home, &substrate_home, &sock, &[], &["--world"]); + + repl.wait_for_output("Substrate v", Duration::from_secs(6)) + .expect("banner"); + repl.wait_for_prompt(Duration::from_secs(2)) + .expect("prompt"); + + repl.send_line(":host pwd"); + repl.wait_for_output("host escape", Duration::from_secs(2)) + .expect("host-escape gating message"); + repl.send_line("exit"); + + let (_code, out) = repl.shutdown_graceful(Duration::from_secs(2)); + + assert!( + out.to_ascii_lowercase().contains("host escape"), + "expected a host-escape gating error, got output:\n{out}" + ); +} + +#[test] +#[serial] +fn c3_host_directive_executes_on_host_when_enabled() { + let temp = temp_dir("substrate-c3-host-gate-on-"); + let home = temp.path().join("home"); + let project = temp.path().join("project"); + let substrate_home = home.join(".substrate"); + fs::create_dir_all(&home).expect("create home"); + fs::create_dir_all(&project).expect("create project"); + fs::create_dir_all(&substrate_home).expect("create substrate home"); + fs::write(home.join(".substrate/trace.jsonl"), "").expect("seed trace"); + write_profile(&project); + write_policy(&substrate_home, true); + + let sock_temp = short_socket_dir("sub-c3ws-host-on-"); + let sock = sock_temp.path().join("world.sock"); + let _server = ReplWorldAgentStub::start(&sock, StreamBehavior::Normal); + + let mut repl = PtyRepl::spawn( + &project, + &home, + &substrate_home, + &sock, + &[("SUBSTRATE_REPL_HOST_ESCAPE", "1")], + &["--world"], + ); + + repl.wait_for_output("Substrate v", Duration::from_secs(6)) + .expect("banner"); + repl.wait_for_prompt(Duration::from_secs(2)) + .expect("prompt"); + + let project = fs::canonicalize(&project).unwrap_or(project); + let project_str = project.to_string_lossy().into_owned(); + repl.send_line(":host pwd"); + repl.wait_for_output(project_str.as_ref(), Duration::from_secs(2)) + .expect("host pwd output"); + repl.send_line("exit"); + + let (_code, out) = repl.shutdown_graceful(Duration::from_secs(2)); + assert!( + out.contains(project_str.as_str()), + "expected :host pwd to print the host cwd ({project_str}), got output:\n{out}" + ); +} + +#[test] +#[serial] +fn c3_pty_directive_routes_to_persistent_session_when_world_enabled() { + let temp = temp_dir("substrate-c3-pty-world-"); + let home = temp.path().join("home"); + let project = temp.path().join("project"); + let substrate_home = home.join(".substrate"); + fs::create_dir_all(&home).expect("create home"); + fs::create_dir_all(&project).expect("create project"); + fs::create_dir_all(&substrate_home).expect("create substrate home"); + fs::write(home.join(".substrate/trace.jsonl"), "").expect("seed trace"); + write_profile(&project); + write_policy(&substrate_home, true); + + let sock_temp = short_socket_dir("sub-c3ws-pty-world-"); + let sock = sock_temp.path().join("world.sock"); + let server = ReplWorldAgentStub::start(&sock, StreamBehavior::Normal); + let records = server.records(); + + let mut repl = PtyRepl::spawn(&project, &home, &substrate_home, &sock, &[], &["--world"]); + + repl.wait_for_output("Substrate v", Duration::from_secs(6)) + .expect("banner"); + + repl.send_line(":pty echo hello"); + repl.send_line("exit"); + + let (_code, _out) = repl.shutdown_graceful(Duration::from_secs(2)); + + let guard = records.lock().expect("lock records"); + assert!( + guard + .persistent_execs + .iter() + .any(|rec| rec.stdin_mode == "passthrough" && rec.program_utf8.trim() == "echo hello"), + "expected :pty to use persistent-session exec passthrough with stripped prefix; records: {guard:#?}" + ); +} + +#[test] +#[serial] +fn c3_persistent_session_start_carries_canonical_net_allowed_snapshot() { + let temp = temp_dir("substrate-c3-net-allowed-pty-"); + let home = temp.path().join("home"); + let project = temp.path().join("project"); + let substrate_home = home.join(".substrate"); + fs::create_dir_all(&home).expect("create home"); + fs::create_dir_all(&project).expect("create project"); + fs::create_dir_all(&substrate_home).expect("create substrate home"); + fs::write(home.join(".substrate/trace.jsonl"), "").expect("seed trace"); + write_profile(&project); + write_policy_with_net_allowed( + &substrate_home, + true, + "[\" Example.COM. \", \"example.com\", \"Api.Example.com.\"]", + ); + write_config(&substrate_home, true); + + let sock_temp = short_socket_dir("sub-c3ws-net-allowed-"); + let sock = sock_temp.path().join("world.sock"); + let server = ReplWorldAgentStub::start(&sock, StreamBehavior::Normal); + let records = server.records(); + + let mut repl = PtyRepl::spawn(&project, &home, &substrate_home, &sock, &[], &["--world"]); + + repl.wait_for_output("Substrate v", Duration::from_secs(6)) + .expect("banner"); + repl.send_line("echo hello"); + repl.wait_for_output("hello", Duration::from_secs(3)) + .expect("command output"); + repl.send_line("exit"); + + let (_code, _out) = repl.shutdown_graceful(Duration::from_secs(2)); + + let guard = records.lock().expect("lock records"); + let start = guard + .persistent_start_sessions + .first() + .expect("persistent start session"); + let net_allowed = start + .policy_snapshot + .get("net_allowed") + .and_then(|value| value.as_array()) + .expect("policy_snapshot.net_allowed array"); + let net_allowed: Vec<&str> = net_allowed + .iter() + .map(|value| value.as_str().expect("net_allowed string")) + .collect(); + assert_eq!(net_allowed, vec!["example.com", "api.example.com"]); + assert_world_network_payload(start, true, &["example.com", "api.example.com"]); +} + +#[test] +#[serial] +fn c3_persistent_session_start_obeys_net_allowed_routing_matrix() { + let cases = [ + ReplRoutingCase { + name: "gate off plus restrictive policy stays allow-all at routing layer", + net_allowed_yaml: "[\" Example.COM. \", \"example.com\", \"Api.Example.com.\"]", + world_net_filter: false, + expected_net_allowed: &["example.com", "api.example.com"], + expected_isolate_network: false, + expected_allowed_domains: &[], + }, + ReplRoutingCase { + name: "gate on plus allow-all singleton does not request isolation", + net_allowed_yaml: "[\" * \"]", + world_net_filter: true, + expected_net_allowed: &["*"], + expected_isolate_network: false, + expected_allowed_domains: &[], + }, + ReplRoutingCase { + name: "gate on plus deny-all requests isolation with empty allowlist", + net_allowed_yaml: "[]", + world_net_filter: true, + expected_net_allowed: &[], + expected_isolate_network: true, + expected_allowed_domains: &[], + }, + ReplRoutingCase { + name: "gate on plus restrictive allowlist requests isolation with canonical domains", + net_allowed_yaml: "[\" Example.COM. \", \"example.com\", \"Api.Example.com.\"]", + world_net_filter: true, + expected_net_allowed: &["example.com", "api.example.com"], + expected_isolate_network: true, + expected_allowed_domains: &["example.com", "api.example.com"], + }, + ]; + + for case in &cases { + run_repl_routing_case(case); + } +} + +#[cfg(target_os = "linux")] +#[test] +#[serial] +fn c3_first_start_shared_world_attach_create_is_owner_bound() { + let temp = temp_dir("substrate-c3-world-owner-bound-"); + let home = temp.path().join("home"); + let project = temp.path().join("project"); + let substrate_home = home.join(".substrate"); + fs::create_dir_all(&home).expect("create home"); + fs::create_dir_all(&project).expect("create project"); + fs::create_dir_all(&substrate_home).expect("create substrate home"); + fs::write(home.join(".substrate/trace.jsonl"), "").expect("seed trace"); + write_profile(&project); + let fake_codex = write_fake_codex_script(temp.path()); + write_orchestrator_runtime_world_config(&substrate_home, &fake_codex, "auto_restart"); + + let sock_temp = short_socket_dir("sub-c3ws-owner-bound-"); + let sock = sock_temp.path().join("world.sock"); + let server = ReplWorldAgentStub::start(&sock, StreamBehavior::Normal); + let records = server.records(); + + let mut repl = PtyRepl::spawn(&project, &home, &substrate_home, &sock, &[], &["--world"]); + + repl.wait_for_output("Substrate v", Duration::from_secs(6)) + .expect("banner"); + repl.wait_for_prompt(Duration::from_secs(2)) + .expect("initial prompt"); + launch_host_runtime_via_targeted_turn(&mut repl, "cli:codex"); + repl.send_line("exit"); + + let (_code, _out) = repl.shutdown_graceful(Duration::from_secs(2)); + let orchestration_session_id = load_single_orchestration_session_id(&substrate_home); + + let guard = records.lock().expect("lock records"); + assert_eq!( + guard.persistent_start_sessions.len(), + 1, + "lazy host launch must reuse the existing REPL-owned world session instead of creating a second shared-world start: {guard:#?}" + ); + let session = read_orchestration_session(&orchestration_session_path( + &substrate_home, + &orchestration_session_id, + )); + assert_eq!( + session + .get("orchestration_session_id") + .and_then(Value::as_str), + Some(orchestration_session_id.as_str()) + ); +} + +#[cfg(any(target_os = "linux", target_os = "macos"))] +#[test] +#[serial] +fn c3_host_orchestrator_remains_dormant_until_first_targeted_turn() { + let temp = temp_dir("substrate-c3-host-dormant-until-target-"); + let home = temp.path().join("home"); + let project = temp.path().join("project"); + let substrate_home = home.join(".substrate"); + fs::create_dir_all(&home).expect("create home"); + fs::create_dir_all(&project).expect("create project"); + fs::create_dir_all(&substrate_home).expect("create substrate home"); + fs::write(home.join(".substrate/trace.jsonl"), "").expect("seed trace"); + write_profile(&project); + let (fake_codex, invocation_count_path, captured_args_dir, captured_stdin_dir) = + write_fake_codex_script_with_invocation_log_and_stdio_capture(temp.path()); + write_host_orchestrator_runtime_world_config(&substrate_home, &fake_codex, "auto_restart"); + + let sock_temp = short_socket_dir("sub-c3ws-host-dormant-target-"); + let sock = sock_temp.path().join("world.sock"); + let _server = ReplWorldAgentStub::start(&sock, StreamBehavior::Normal); + + let mut repl = PtyRepl::spawn( + &project, + &home, + &substrate_home, + &sock, + &[], + &["--no-world"], + ); + repl.wait_for_output("Substrate v", Duration::from_secs(6)) + .expect("banner"); + repl.wait_for_prompt(Duration::from_secs(2)) + .expect("initial prompt"); + std::thread::sleep(Duration::from_millis(150)); + + let out_before_target = repl.output_string(); + assert_eq!( + read_invocation_count(&invocation_count_path), + 0, + "orchestrator backend must not be invoked before the first explicit targeted turn; output:\n{out_before_target}" + ); + assert_no_persisted_host_orchestrator_state_for_interval( + &substrate_home, + Duration::from_millis(150), + ); + + repl.send_line("::cli:codex launch on demand"); + repl.wait_for_output( + "shell-owned orchestrator session is ready via retained attached control ownership", + Duration::from_secs(3), + ) + .expect("runtime ready after explicit targeted turn"); + assert_eq!( + read_invocation_count(&invocation_count_path), + 1, + "first explicit targeted turn should launch the retained host runtime exactly once" + ); + let orchestration_session_id = load_single_orchestration_session_id(&substrate_home); + let live_participants = authoritative_live_participant_manifests_for_session( + &substrate_home, + &orchestration_session_id, + ); + let live_host = live_participants + .iter() + .filter(|manifest| manifest.get("role").and_then(Value::as_str) == Some("orchestrator")) + .filter(|manifest| { + manifest.pointer("/execution/scope").and_then(Value::as_str) == Some("host") + }) + .collect::>(); + assert_eq!( + live_host.len(), + 1, + "first targeted host turn must materialize exactly one authoritative-live host participant: {live_participants:?}" + ); + let first_args = + fs::read_to_string(captured_args_dir.join("1.args")).expect("read first captured args"); + assert!( + first_args.lines().any(|line| line == "exec"), + "first targeted host launch must use the initial exec path: {first_args:?}" + ); + let first_stdin = + fs::read_to_string(captured_stdin_dir.join("1.stdin")).expect("read first captured stdin"); + assert!( + first_stdin.contains("launch on demand"), + "first targeted host launch must consume the user prompt as stdin: {first_stdin:?}" + ); + assert!( + !first_stdin.contains("Enter persistent Substrate host orchestrator mode."), + "first targeted host launch must not consume the hidden bootstrap prompt: {first_stdin:?}" + ); + assert!( + !first_stdin.contains("First visible operator request:"), + "first targeted host launch must not compose bootstrap and visible prompt text: {first_stdin:?}" + ); + + repl.send_line("exit"); + let (_code, _out) = repl.shutdown_graceful(Duration::from_secs(3)); +} + +#[cfg(any(target_os = "linux", target_os = "macos"))] +#[test] +#[serial] +fn c3_first_targeted_world_turn_uses_initial_prompt_in_member_dispatch() { + let temp = temp_dir("substrate-c3-targeted-world-initial-prompt-"); + let home = temp.path().join("home"); + let project = temp.path().join("project"); + let substrate_home = home.join(".substrate"); + fs::create_dir_all(&home).expect("create home"); + fs::create_dir_all(&project).expect("create project"); + fs::create_dir_all(&substrate_home).expect("create substrate home"); + fs::write(home.join(".substrate/trace.jsonl"), "").expect("seed trace"); + write_profile(&project); + let fake_orchestrator = write_fake_claude_script(temp.path()); + let fake_member = write_fake_codex_script(temp.path()); + write_orchestrator_and_world_member_runtime_world_config( + &substrate_home, + &fake_orchestrator, + &fake_member, + "auto_restart", + ); + + let sock_temp = short_socket_dir("sub-c3ws-targeted-world-initial-prompt-"); + let sock = sock_temp.path().join("world.sock"); + let server = ReplWorldAgentStub::start_with_member_dispatch_scripts( + &sock, + StreamBehavior::Normal, + vec![MemberDispatchStreamScript::ReadyAndHoldUntilCancel { + session_handle_id: "session-targeted-world-first-turn".to_string(), + exit_code_on_cancel: 130, + }], + ); + let records = server.records(); + + let mut repl = PtyRepl::spawn(&project, &home, &substrate_home, &sock, &[], &["--world"]); + repl.wait_for_output("Substrate v", Duration::from_secs(6)) + .expect("banner"); + repl.wait_for_prompt(Duration::from_secs(2)) + .expect("initial prompt"); + + repl.send_line("::cli:claude_code start host runtime"); + repl.wait_for_output( + "shell-owned orchestrator session is ready via retained attached control ownership", + Duration::from_secs(3), + ) + .expect("host runtime ready"); + + repl.send_line("::cli:codex member targeted first turn"); + wait_for_min_member_dispatch_requests(&records, 1, Duration::from_secs(3)); + repl.wait_for_prompt(Duration::from_secs(2)) + .expect("prompt after first targeted world turn"); + std::thread::sleep(Duration::from_millis(100)); + + let guard = records.lock().expect("lock records"); + assert_eq!( + guard.member_turn_submit_requests.len(), + 0, + "first targeted world turn must ride launch-time initial_prompt instead of the typed submit route: {guard:#?}" + ); + let dispatch = guard + .member_dispatch_requests + .first() + .and_then(|request| request.member_dispatch.as_ref()) + .expect("captured member dispatch request"); + assert_eq!( + dispatch.initial_prompt.as_deref(), + Some("member targeted first turn") + ); + drop(guard); + + repl.send_line("exit"); + let (_code, _out) = repl.shutdown_graceful(Duration::from_secs(3)); + wait_for_min_execute_cancel_requests(&records, 1, Duration::from_secs(3)); + + let guard = records.lock().expect("lock records"); + assert_eq!( + guard.execute_cancel_requests.len(), + 1, + "member launch shutdown must route retained member cancel through /v1/execute/cancel: {guard:#?}" + ); + assert!( + guard.execute_cancel_requests[0] + .span_id + .starts_with("member-span-"), + "member shutdown cancel must target the guest member_dispatch span: {guard:#?}" + ); +} + +#[cfg(any(target_os = "linux", target_os = "macos"))] +#[test] +#[serial] +fn c3_first_world_backed_command_lazily_launches_member_runtime() { + let temp = temp_dir("substrate-c3-first-world-command-lazy-member-"); + let home = temp.path().join("home"); + let project = temp.path().join("project"); + let substrate_home = home.join(".substrate"); + fs::create_dir_all(&home).expect("create home"); + fs::create_dir_all(&project).expect("create project"); + fs::create_dir_all(&substrate_home).expect("create substrate home"); + fs::write(home.join(".substrate/trace.jsonl"), "").expect("seed trace"); + write_profile(&project); + let fake_orchestrator = write_fake_claude_script(temp.path()); + let fake_member = write_fake_codex_script(temp.path()); + write_orchestrator_and_world_member_runtime_world_config( + &substrate_home, + &fake_orchestrator, + &fake_member, + "auto_restart", + ); + + let sock_temp = short_socket_dir("sub-c3ws-lazy-member-"); + let sock = sock_temp.path().join("world.sock"); + let server = ReplWorldAgentStub::start_with_member_dispatch_scripts( + &sock, + StreamBehavior::Normal, + vec![MemberDispatchStreamScript::ReadyAndHoldUntilCancel { + session_handle_id: "session-lazy-member".to_string(), + exit_code_on_cancel: 130, + }], + ); + let records = server.records(); + + let mut repl = PtyRepl::spawn(&project, &home, &substrate_home, &sock, &[], &["--world"]); + repl.wait_for_output("Substrate v", Duration::from_secs(6)) + .expect("banner"); + repl.wait_for_prompt(Duration::from_secs(2)) + .expect("initial prompt"); + launch_host_runtime_via_targeted_turn(&mut repl, "cli:claude_code"); + + let orchestration_session_id = load_single_orchestration_session_id(&substrate_home); + let initial_session = read_orchestration_session(&orchestration_session_path( + &substrate_home, + &orchestration_session_id, + )); + assert_world_member_absent_for_interval( + &substrate_home, + &orchestration_session_id, + Duration::from_millis(250), + ); + + repl.send_line("echo first"); + wait_for_min_records(&records, 1, 1, Duration::from_secs(3)); + repl.wait_for_output("first", Duration::from_secs(3)) + .expect("first command output"); + + let live_members = wait_for_live_world_member_count( + &substrate_home, + &orchestration_session_id, + 1, + Duration::from_secs(5), + ); + let member = &live_members[0]; + assert_eq!( + member.get("agent_id").and_then(Value::as_str), + Some("codex") + ); + assert!( + matches!( + member.get("state").and_then(Value::as_str), + Some("ready" | "running") + ), + "lazy member launch must become authoritative-live: {member:?}" + ); + assert_eq!( + member.get("world_generation").and_then(Value::as_u64), + Some(0) + ); + assert_eq!( + member.get("world_id").and_then(Value::as_str), + initial_session.get("world_id").and_then(Value::as_str), + "lazy member launch must bind to the current authoritative world" + ); + assert_eq!( + member + .get("orchestrator_participant_id") + .and_then(Value::as_str), + initial_session + .get("active_session_handle_id") + .and_then(Value::as_str), + "lazy member launch must retain the live orchestrator seam" + ); + assert!( + member + .get("resumed_from_participant_id") + .is_none_or(Value::is_null), + "first member launch must not claim replacement lineage: {member:?}" + ); + + repl.send_line("exit"); + let (_code, _out) = repl.shutdown_graceful(Duration::from_secs(3)); +} + +#[cfg(target_os = "linux")] +#[test] +#[serial] +fn c3_targeted_turn_requires_exact_double_colon_grammar_before_shell_fallback() { + let temp = temp_dir("substrate-c3-targeted-turn-grammar-"); + let home = temp.path().join("home"); + let project = temp.path().join("project"); + let substrate_home = home.join(".substrate"); + fs::create_dir_all(&home).expect("create home"); + fs::create_dir_all(&project).expect("create project"); + fs::create_dir_all(&substrate_home).expect("create substrate home"); + fs::write(home.join(".substrate/trace.jsonl"), "").expect("seed trace"); + write_profile(&project); + let fake_orchestrator = write_fake_claude_script(temp.path()); + let fake_member = write_fake_codex_script(temp.path()); + write_orchestrator_and_world_member_runtime_world_config( + &substrate_home, + &fake_orchestrator, + &fake_member, + "auto_restart", + ); + + let sock_temp = short_socket_dir("sub-c3ws-targeted-turn-grammar-"); + let sock = sock_temp.path().join("world.sock"); + let server = ReplWorldAgentStub::start(&sock, StreamBehavior::Normal); + let records = server.records(); + + let mut repl = PtyRepl::spawn(&project, &home, &substrate_home, &sock, &[], &["--world"]); + repl.wait_for_output("Substrate v", Duration::from_secs(6)) + .expect("banner"); + repl.wait_for_prompt(Duration::from_secs(2)) + .expect("initial prompt"); + + repl.send_line("::cli:codex"); + repl.wait_for_output( + "substrate: error: targeted follow-up turns require exact syntax ':: ' on a single line", + Duration::from_secs(3), + ) + .expect("malformed syntax rejection"); + repl.wait_for_prompt(Duration::from_secs(2)) + .expect("prompt after malformed syntax"); + std::thread::sleep(Duration::from_millis(100)); + + let guard = records.lock().expect("lock records"); + assert!( + guard.persistent_execs.is_empty(), + "malformed targeted syntax must fail before shell fallback: {guard:#?}" + ); + assert!( + guard.member_dispatch_requests.is_empty(), + "malformed targeted syntax must not lazy-launch a member runtime: {guard:#?}" + ); + assert!( + guard.member_turn_submit_requests.is_empty(), + "malformed targeted syntax must not hit the typed submit route: {guard:#?}" + ); + drop(guard); + + repl.send_line("exit"); + let (_code, _out) = repl.shutdown_graceful(Duration::from_secs(3)); +} + +#[cfg(any(target_os = "linux", target_os = "macos"))] +#[test] +#[serial] +fn c3_targeted_world_turn_uses_typed_submit_route_without_relaunching_member() { + let temp = temp_dir("substrate-c3-targeted-world-submit-"); + let home = temp.path().join("home"); + let project = temp.path().join("project"); + let substrate_home = home.join(".substrate"); + fs::create_dir_all(&home).expect("create home"); + fs::create_dir_all(&project).expect("create project"); + fs::create_dir_all(&substrate_home).expect("create substrate home"); + fs::write(home.join(".substrate/trace.jsonl"), "").expect("seed trace"); + write_profile(&project); + let fake_orchestrator = write_fake_claude_script(temp.path()); + let fake_member = write_fake_codex_script(temp.path()); + write_orchestrator_and_world_member_runtime_world_config( + &substrate_home, + &fake_orchestrator, + &fake_member, + "auto_restart", + ); + + let sock_temp = short_socket_dir("sub-c3ws-targeted-world-submit-"); + let sock = sock_temp.path().join("world.sock"); + let server = ReplWorldAgentStub::start_with_member_dispatch_scripts( + &sock, + StreamBehavior::Normal, + vec![MemberDispatchStreamScript::ReadyAndHoldUntilCancel { + session_handle_id: "session-targeted-world".to_string(), + exit_code_on_cancel: 130, + }], + ); + let records = server.records(); + + let mut repl = PtyRepl::spawn(&project, &home, &substrate_home, &sock, &[], &["--world"]); + repl.wait_for_output("Substrate v", Duration::from_secs(6)) + .expect("banner"); + repl.wait_for_prompt(Duration::from_secs(2)) + .expect("initial prompt"); + launch_host_runtime_via_targeted_turn(&mut repl, "cli:claude_code"); + + let orchestration_session_id = load_single_orchestration_session_id(&substrate_home); + + repl.send_line("echo first"); + wait_for_min_records(&records, 1, 1, Duration::from_secs(3)); + wait_for_min_member_dispatch_requests(&records, 1, Duration::from_secs(3)); + repl.wait_for_output("first", Duration::from_secs(3)) + .expect("first command output"); + + let orchestration_session = read_orchestration_session(&orchestration_session_path( + &substrate_home, + &orchestration_session_id, + )); + let live_participants = authoritative_live_participant_manifests_for_session( + &substrate_home, + &orchestration_session_id, + ); + assert_eq!( + live_participants.len(), + 2, + "first world-backed command must leave both the host orchestrator and world member authoritative-live: {live_participants:?}" + ); + assert_eq!( + live_participants + .iter() + .map(|manifest| manifest.get("backend_id").and_then(Value::as_str)) + .collect::>(), + vec![Some("cli:claude_code"), Some("cli:codex")], + "first world-backed command must establish authoritative-live coexistence for exactly cli:claude_code and cli:codex" + ); + let orchestrator = + authoritative_live_participant_manifest_for_backend(&live_participants, "cli:claude_code"); + let orchestrator_participant_id = orchestrator + .get("participant_id") + .and_then(Value::as_str) + .expect("orchestrator participant_id") + .to_string(); + assert_eq!( + orchestrator.get("role").and_then(Value::as_str), + Some("orchestrator") + ); + assert_eq!( + orchestrator + .pointer("/execution/scope") + .and_then(Value::as_str), + Some("host") + ); + assert_eq!( + Some(orchestrator_participant_id.as_str()), + orchestration_session + .get("active_session_handle_id") + .and_then(Value::as_str), + "the active orchestration seam must remain owned by cli:claude_code after cli:codex becomes live" + ); + + let live_members = wait_for_live_world_member_count( + &substrate_home, + &orchestration_session_id, + 1, + Duration::from_secs(5), + ); + let member = &live_members[0]; + let member_participant_id = member + .get("participant_id") + .and_then(Value::as_str) + .expect("member participant_id") + .to_string(); + let member_orchestrator_participant_id = member + .get("orchestrator_participant_id") + .and_then(Value::as_str) + .expect("member orchestrator_participant_id") + .to_string(); + assert_eq!( + member_orchestrator_participant_id, + orchestrator_participant_id, + "the retained cli:codex world member must stay linked to the authoritative cli:claude_code orchestrator participant" + ); + let world_id = member + .get("world_id") + .and_then(Value::as_str) + .expect("member world_id") + .to_string(); + let world_generation = member + .get("world_generation") + .and_then(Value::as_u64) + .expect("member world_generation"); + + repl.send_line("::cli:codex second"); + wait_for_min_member_turn_submit_requests(&records, 1, Duration::from_secs(3)); + repl.wait_for_output("__MEMBER_TURN_SUBMIT_STUB__ second", Duration::from_secs(3)) + .expect("typed submit route output"); + + let guard = records.lock().expect("lock records"); + assert_eq!( + guard.member_dispatch_requests.len(), + 1, + "targeted follow-up turn must reuse the retained member instead of relaunching it: {guard:#?}" + ); + let submit = guard + .member_turn_submit_requests + .first() + .expect("member turn submit request"); + assert_eq!(submit.orchestration_session_id, orchestration_session_id); + assert_eq!(submit.participant_id, member_participant_id); + assert_eq!( + submit.orchestrator_participant_id, + member_orchestrator_participant_id + ); + assert_eq!(submit.backend_id, "cli:codex"); + assert_eq!(submit.world_id, world_id); + assert_eq!(submit.world_generation, world_generation); + assert_eq!(submit.prompt, "second"); + drop(guard); + + let live_members_after = wait_for_live_world_member_count( + &substrate_home, + &orchestration_session_id, + 1, + Duration::from_secs(3), + ); + assert_eq!( + live_members_after[0] + .get("participant_id") + .and_then(Value::as_str), + Some(member_participant_id.as_str()), + "targeted submit must keep one retained world member rather than swap or duplicate it" + ); + let live_participants_after = authoritative_live_participant_manifests_for_session( + &substrate_home, + &orchestration_session_id, + ); + assert_eq!( + live_participants_after.len(), + 2, + "targeted follow-up turns must keep the same two authoritative-live backends after coexistence is established: {live_participants_after:?}" + ); + assert_eq!( + live_participants_after + .iter() + .map(|manifest| manifest.get("backend_id").and_then(Value::as_str)) + .collect::>(), + vec![Some("cli:claude_code"), Some("cli:codex")], + "targeted follow-up turns must preserve authoritative-live coexistence for exactly cli:claude_code and cli:codex" + ); + let orchestrator_after = authoritative_live_participant_manifest_for_backend( + &live_participants_after, + "cli:claude_code", + ); + assert_eq!( + orchestrator_after + .get("participant_id") + .and_then(Value::as_str), + Some(orchestrator_participant_id.as_str()), + "cli:claude_code targeted coexistence must reuse the original orchestrator participant" + ); + let member_after = + authoritative_live_participant_manifest_for_backend(&live_participants_after, "cli:codex"); + assert_eq!( + member_after.get("participant_id").and_then(Value::as_str), + Some(member_participant_id.as_str()), + "cli:codex targeted coexistence must reuse the original world member participant" + ); + + repl.send_line("exit"); + let (_code, _out) = repl.shutdown_graceful(Duration::from_secs(3)); +} + +#[cfg(target_os = "linux")] +#[test] +#[serial] +fn c3_targeted_world_turn_relaunches_exact_backend_after_world_restart() { + let temp = temp_dir("substrate-c3-targeted-world-relaunch-"); + let home = temp.path().join("home"); + let project = temp.path().join("project"); + let substrate_home = home.join(".substrate"); + let trace_path = home.join(".substrate/trace.jsonl"); + fs::create_dir_all(&home).expect("create home"); + fs::create_dir_all(&project).expect("create project"); + fs::create_dir_all(&substrate_home).expect("create substrate home"); + fs::write(&trace_path, "").expect("seed trace"); + write_profile(&project); + let fake_orchestrator = write_fake_claude_script(temp.path()); + let fake_member = write_fake_codex_script(temp.path()); + write_orchestrator_and_world_member_runtime_world_config( + &substrate_home, + &fake_orchestrator, + &fake_member, + "auto_restart", + ); + + let sock_temp = short_socket_dir("sub-c3ws-targeted-world-relaunch-"); + let sock = sock_temp.path().join("world.sock"); + let server = ReplWorldAgentStub::start_with_member_dispatch_scripts( + &sock, + StreamBehavior::Normal, + vec![ + MemberDispatchStreamScript::ReadyAndHoldUntilCancel { + session_handle_id: "session-first-member".to_string(), + exit_code_on_cancel: 130, + }, + MemberDispatchStreamScript::ReadyAndHoldUntilCancel { + session_handle_id: "session-replacement-member".to_string(), + exit_code_on_cancel: 130, + }, + ], + ); + let records = server.records(); + + let mut repl = PtyRepl::spawn(&project, &home, &substrate_home, &sock, &[], &["--world"]); + repl.wait_for_output("Substrate v", Duration::from_secs(6)) + .expect("banner"); + repl.wait_for_prompt(Duration::from_secs(2)) + .expect("initial prompt"); + launch_host_runtime_via_targeted_turn(&mut repl, "cli:claude_code"); + let orchestration_session_id = load_single_orchestration_session_id(&substrate_home); + let session_path = orchestration_session_path(&substrate_home, &orchestration_session_id); + + repl.send_line("echo first"); + wait_for_min_records(&records, 1, 1, Duration::from_secs(3)); + repl.wait_for_output("first", Duration::from_secs(3)) + .expect("first command output"); + let first_live_members = wait_for_live_world_member_count( + &substrate_home, + &orchestration_session_id, + 1, + Duration::from_secs(5), + ); + let first_member = &first_live_members[0]; + let first_member_id = first_member + .get("participant_id") + .and_then(Value::as_str) + .expect("first member participant_id") + .to_string(); + let orchestrator_participant_id = first_member + .get("orchestrator_participant_id") + .and_then(Value::as_str) + .expect("member orchestrator_participant_id") + .to_string(); + + write_member_runtime_policy(&substrate_home, false); + std::thread::sleep(Duration::from_millis(25)); + + repl.send_line("::cli:codex second"); + let alert = wait_for_world_restarted_alert_without_stale_liveness( + &trace_path, + &substrate_home, + &orchestration_session_id, + 0, + Duration::from_secs(15), + ); + wait_for_min_member_dispatch_requests(&records, 2, Duration::from_secs(3)); + repl.wait_for_prompt(Duration::from_secs(2)) + .expect("prompt after replacement launch turn"); + + let replacement_live_members = wait_for_live_world_member_count( + &substrate_home, + &orchestration_session_id, + 1, + Duration::from_secs(5), + ); + let replacement = &replacement_live_members[0]; + let replacement_id = replacement + .get("participant_id") + .and_then(Value::as_str) + .expect("replacement participant_id") + .to_string(); + assert_ne!( + replacement_id, first_member_id, + "targeted follow-up after restart must relaunch a distinct exact-backend member runtime" + ); + assert_eq!( + replacement.get("world_generation").and_then(Value::as_u64), + Some(1), + "replacement member must bind to the new world generation" + ); + assert_eq!( + replacement.get("world_id").and_then(Value::as_str), + alert.get("world_id").and_then(Value::as_str), + "replacement member must bind to the replacement world id" + ); + assert_eq!( + replacement + .get("orchestrator_participant_id") + .and_then(Value::as_str), + Some(orchestrator_participant_id.as_str()), + "replacement member must preserve the retained-control seam" + ); + assert_eq!( + replacement + .get("resumed_from_participant_id") + .and_then(Value::as_str), + Some(first_member_id.as_str()), + "replacement member must preserve lineage to the stale exact-backend runtime" + ); + + let guard = records.lock().expect("lock records"); + assert_eq!( + guard.persistent_execs.len(), + 1, + "targeted world follow-up must not fall back to a second shell/persistent exec after restart: {guard:#?}" + ); + assert_eq!( + guard.member_dispatch_requests.len(), + 2, + "targeted world follow-up after restart must relaunch the exact backend slot once: {guard:#?}" + ); + assert!( + guard.member_turn_submit_requests.is_empty(), + "replacement targeted turn should ride launch-time initial_prompt instead of a second typed submit: {guard:#?}" + ); + let replacement_dispatch = guard + .member_dispatch_requests + .get(1) + .and_then(|request| request.member_dispatch.as_ref()) + .expect("replacement member dispatch request"); + assert_eq!( + replacement_dispatch.initial_prompt.as_deref(), + Some("second") + ); + drop(guard); + + let stale = read_participant_manifest(&substrate_home, &first_member_id); + assert_eq!( + stale.get("state").and_then(Value::as_str), + Some("invalidated"), + "stale generation member must remain invalidated after targeted relaunch submit: {stale:?}" + ); + let persisted = read_orchestration_session(&session_path); + assert_session_world_binding(&persisted, Some("wld_stub_0002"), Some(1)); + + repl.send_line("exit"); + let (_code, _out) = repl.shutdown_graceful(Duration::from_secs(3)); +} + +#[cfg(any(target_os = "linux", target_os = "macos"))] +#[test] +#[serial] +fn c3_targeted_host_turn_resumes_active_orchestrator_backend() { + let temp = temp_dir("substrate-c3-targeted-host-active-resume-"); + let home = temp.path().join("home"); + let project = temp.path().join("project"); + let substrate_home = home.join(".substrate"); + fs::create_dir_all(&home).expect("create home"); + fs::create_dir_all(&project).expect("create project"); + fs::create_dir_all(&substrate_home).expect("create substrate home"); + fs::write(home.join(".substrate/trace.jsonl"), "").expect("seed trace"); + write_profile(&project); + let fake_orchestrator = write_fake_claude_script_first_session_then_exit(temp.path()); + let fake_secondary_host = write_fake_codex_script(temp.path()); + write_dual_host_runtime_world_config( + &substrate_home, + &fake_orchestrator, + &fake_secondary_host, + "auto_restart", + ); + + let sock_temp = short_socket_dir("sub-c3ws-targeted-host-active-resume-"); + let sock = sock_temp.path().join("world.sock"); + let server = ReplWorldAgentStub::start(&sock, StreamBehavior::Normal); + let records = server.records(); + + let mut repl = PtyRepl::spawn(&project, &home, &substrate_home, &sock, &[], &["--world"]); + repl.wait_for_output("Substrate v", Duration::from_secs(6)) + .expect("banner"); + repl.wait_for_prompt(Duration::from_secs(2)) + .expect("initial prompt"); + launch_host_runtime_via_targeted_turn(&mut repl, "cli:claude_code"); + + repl.send_line("::cli:claude_code resume active host"); + repl.wait_for_output( + "submitted targeted follow-up turn to cli:claude_code", + Duration::from_secs(3), + ) + .expect("targeted host submit started"); + repl.wait_for_output( + "targeted follow-up turn completed for cli:claude_code", + Duration::from_secs(3), + ) + .expect("targeted host submit completion"); + repl.wait_for_prompt(Duration::from_secs(2)) + .expect("prompt after targeted host submit"); + + let guard = records.lock().expect("lock records"); + assert!( + guard.member_dispatch_requests.is_empty(), + "targeted host follow-up must not launch a world member runtime: {guard:#?}" + ); + assert!( + guard.member_turn_submit_requests.is_empty(), + "targeted host follow-up must not hit the typed world submit route: {guard:#?}" + ); + drop(guard); + + repl.send_line("exit"); + let (_code, _out) = repl.shutdown_graceful(Duration::from_secs(3)); +} + +#[cfg(any(target_os = "linux", target_os = "macos"))] +#[test] +#[serial] +fn c3_targeted_host_turn_rejects_non_active_orchestrator_backend() { + let temp = temp_dir("substrate-c3-targeted-host-rejects-nonactive-"); + let home = temp.path().join("home"); + let project = temp.path().join("project"); + let substrate_home = home.join(".substrate"); + fs::create_dir_all(&home).expect("create home"); + fs::create_dir_all(&project).expect("create project"); + fs::create_dir_all(&substrate_home).expect("create substrate home"); + fs::write(home.join(".substrate/trace.jsonl"), "").expect("seed trace"); + write_profile(&project); + let fake_orchestrator = write_fake_claude_script(temp.path()); + let fake_secondary_host = write_fake_codex_script(temp.path()); + write_dual_host_runtime_world_config( + &substrate_home, + &fake_orchestrator, + &fake_secondary_host, + "auto_restart", + ); + + let sock_temp = short_socket_dir("sub-c3ws-targeted-host-rejects-nonactive-"); + let sock = sock_temp.path().join("world.sock"); + let server = ReplWorldAgentStub::start(&sock, StreamBehavior::Normal); + let records = server.records(); + + let mut repl = PtyRepl::spawn(&project, &home, &substrate_home, &sock, &[], &["--world"]); + repl.wait_for_output("Substrate v", Duration::from_secs(6)) + .expect("banner"); + repl.wait_for_prompt(Duration::from_secs(2)) + .expect("initial prompt"); + launch_host_runtime_via_targeted_turn(&mut repl, "cli:claude_code"); + + repl.send_line("::cli:codex should fail"); + repl.wait_for_output( + "substrate: error: targeted host follow-up turns may only target the active orchestrator backend for this REPL session", + Duration::from_secs(3), + ) + .expect("non-active host backend rejection"); + repl.wait_for_prompt(Duration::from_secs(2)) + .expect("prompt after non-active host backend rejection"); + std::thread::sleep(Duration::from_millis(100)); + + let guard = records.lock().expect("lock records"); + assert!( + guard.persistent_execs.is_empty(), + "rejected host targeted turn must not fall back to shell execution: {guard:#?}" + ); + assert!( + guard.member_dispatch_requests.is_empty(), + "rejected host targeted turn must not launch a world member runtime: {guard:#?}" + ); + assert!( + guard.member_turn_submit_requests.is_empty(), + "rejected host targeted turn must not hit the typed world submit route: {guard:#?}" + ); + drop(guard); + + repl.send_line("exit"); + let (_code, _out) = repl.shutdown_graceful(Duration::from_secs(3)); +} + +#[cfg(target_os = "linux")] +#[test] +#[serial] +fn c3_same_generation_world_command_reuses_live_member_runtime() { + let temp = temp_dir("substrate-c3-same-generation-member-reuse-"); + let home = temp.path().join("home"); + let project = temp.path().join("project"); + let substrate_home = home.join(".substrate"); + fs::create_dir_all(&home).expect("create home"); + fs::create_dir_all(&project).expect("create project"); + fs::create_dir_all(&substrate_home).expect("create substrate home"); + fs::write(home.join(".substrate/trace.jsonl"), "").expect("seed trace"); + write_profile(&project); + let fake_orchestrator = write_fake_claude_script(temp.path()); + let fake_member = write_fake_codex_script(temp.path()); + write_orchestrator_and_world_member_runtime_world_config( + &substrate_home, + &fake_orchestrator, + &fake_member, + "auto_restart", + ); + + let sock_temp = short_socket_dir("sub-c3ws-member-reuse-"); + let sock = sock_temp.path().join("world.sock"); + let server = ReplWorldAgentStub::start_with_member_dispatch_scripts( + &sock, + StreamBehavior::Normal, + vec![ + MemberDispatchStreamScript::ReadyAndHoldUntilCancel { + session_handle_id: "session-first-member".to_string(), + exit_code_on_cancel: 130, + }, + MemberDispatchStreamScript::ExitWithoutReady { exit_code: 1 }, + ], + ); + let records = server.records(); + + let mut repl = PtyRepl::spawn(&project, &home, &substrate_home, &sock, &[], &["--world"]); + repl.wait_for_output("Substrate v", Duration::from_secs(6)) + .expect("banner"); + repl.wait_for_prompt(Duration::from_secs(2)) + .expect("initial prompt"); + launch_host_runtime_via_targeted_turn(&mut repl, "cli:claude_code"); + let orchestration_session_id = load_single_orchestration_session_id(&substrate_home); + repl.send_line("echo first"); + wait_for_min_records(&records, 1, 1, Duration::from_secs(3)); + repl.wait_for_output("first", Duration::from_secs(3)) + .expect("first command output"); + let first_live_members = wait_for_live_world_member_count( + &substrate_home, + &orchestration_session_id, + 1, + Duration::from_secs(5), + ); + let first_member_id = first_live_members[0] + .get("participant_id") + .and_then(Value::as_str) + .expect("first member participant_id") + .to_string(); + + repl.send_line("echo second"); + wait_for_min_records(&records, 2, 1, Duration::from_secs(3)); + repl.wait_for_output("second", Duration::from_secs(3)) + .expect("second command output"); + std::thread::sleep(Duration::from_millis(100)); + + let members = world_member_manifests_for_session(&substrate_home, &orchestration_session_id); + assert_eq!( + members.len(), + 1, + "same-generation second command must reuse the existing member instead of creating a sibling: {members:?}" + ); + let live_members = wait_for_live_world_member_count( + &substrate_home, + &orchestration_session_id, + 1, + Duration::from_secs(2), + ); + let member = &live_members[0]; + assert_eq!( + member.get("participant_id").and_then(Value::as_str), + Some(first_member_id.as_str()), + "same-generation second command must reuse the live member" + ); + assert_eq!( + member.get("world_generation").and_then(Value::as_u64), + Some(0) + ); + + repl.send_line("exit"); + let (_code, _out) = repl.shutdown_graceful(Duration::from_secs(3)); +} + +#[cfg(target_os = "linux")] +#[test] +#[serial] +fn c3_startup_drift_before_first_command_retains_persisted_startup_context() { + let temp = temp_dir("substrate-c3-startup-drift-context-"); + let home = temp.path().join("home"); + let project = temp.path().join("project"); + let child = project.join("child"); + let substrate_home = home.join(".substrate"); + fs::create_dir_all(&home).expect("create home"); + fs::create_dir_all(&child).expect("create project child"); + fs::create_dir_all(&substrate_home).expect("create substrate home"); + fs::write(home.join(".substrate/trace.jsonl"), "").expect("seed trace"); + write_workspace_marker(&project); + write_profile(&project); + let fake_codex = write_fake_codex_script(temp.path()); + write_orchestrator_runtime_world_config(&substrate_home, &fake_codex, "auto_restart"); + + let parent_cwd = temp.path().to_string_lossy().into_owned(); + + let sock_temp = short_socket_dir("sub-c3ws-startup-context-"); + let sock = sock_temp.path().join("world.sock"); + let server = ReplWorldAgentStub::start_with_first_ready_cwd_override( + &sock, + StreamBehavior::Normal, + parent_cwd, + ); + let records = server.records(); + + let mut repl = PtyRepl::spawn(&child, &home, &substrate_home, &sock, &[], &["--world"]); + + repl.wait_for_output("Substrate v", Duration::from_secs(6)) + .expect("banner"); + repl.wait_for_prompt(Duration::from_secs(2)) + .expect("initial prompt"); + launch_host_runtime_via_targeted_turn(&mut repl, "cli:codex"); + wait_for_min_start_sessions_with_output(&repl, &records, 2, Duration::from_secs(3)); + repl.send_line("exit"); + + let (_code, _out) = repl.shutdown_graceful(Duration::from_secs(3)); + let guard = records.lock().expect("lock records"); + assert_eq!( + guard.persistent_start_sessions.len(), + 2, + "startup drift before the first command should only restart the REPL-owned world session once: {guard:#?}" + ); + assert_start_sessions_have_no_shared_world_context(&guard, "workspace root drift"); +} + +#[cfg(target_os = "linux")] +#[test] +#[serial] +fn c3_parent_binding_persists_before_world_restarted_publishes() { + let temp = temp_dir("substrate-c3-world-restarted-binding-"); + let home = temp.path().join("home"); + let project = temp.path().join("project"); + let substrate_home = home.join(".substrate"); + let trace_path = home.join(".substrate/trace.jsonl"); + fs::create_dir_all(&home).expect("create home"); + fs::create_dir_all(&project).expect("create project"); + fs::create_dir_all(&substrate_home).expect("create substrate home"); + fs::write(&trace_path, "").expect("seed trace"); + write_profile(&project); + let fake_codex = write_fake_codex_script(temp.path()); + write_orchestrator_runtime_world_config(&substrate_home, &fake_codex, "auto_restart"); + + let sock_temp = short_socket_dir("sub-c3ws-world-restarted-binding-"); + let sock = sock_temp.path().join("world.sock"); + let server = ReplWorldAgentStub::start(&sock, StreamBehavior::Normal); + let records = server.records(); + + let mut repl = PtyRepl::spawn(&project, &home, &substrate_home, &sock, &[], &["--world"]); + + repl.wait_for_output("Substrate v", Duration::from_secs(6)) + .expect("banner"); + repl.wait_for_prompt(Duration::from_secs(2)) + .expect("initial prompt"); + launch_host_runtime_via_targeted_turn(&mut repl, "cli:codex"); + let orchestration_session_id = load_single_orchestration_session_id(&substrate_home); + let session_path = orchestration_session_path(&substrate_home, &orchestration_session_id); + repl.send_line("echo first"); + wait_for_min_records(&records, 1, 1, Duration::from_secs(3)); + repl.wait_for_output("first", Duration::from_secs(3)) + .expect("first command output"); + + write_policy(&substrate_home, false); + std::thread::sleep(Duration::from_millis(25)); + + repl.send_line("echo second"); + repl.wait_for_output( + "[shell] world restarted due to policy snapshot drift", + Duration::from_secs(5), + ) + .expect("world_restarted alert output"); + + let persisted = read_orchestration_session(&session_path); + assert_session_world_binding(&persisted, Some("wld_stub_0002"), Some(1)); + + repl.wait_for_output("second", Duration::from_secs(3)) + .expect("second command output"); + repl.send_line("exit"); + let (_code, _out) = repl.shutdown_graceful(Duration::from_secs(3)); + + let events = read_trace(&trace_path); + let alerts = alert_rows_by_code(&events, "world_restarted"); + assert_eq!( + alerts.len(), + 1, + "expected one world_restarted alert: {alerts:?}" + ); + let alert = alerts[0]; + assert_eq!( + alert + .get("orchestration_session_id") + .and_then(Value::as_str), + Some(orchestration_session_id.as_str()), + "world_restarted alert must retain the authoritative orchestration_session_id: {alert:?}" + ); + assert_eq!( + alert.get("world_id").and_then(Value::as_str), + Some("wld_stub_0002"), + "world_restarted alert must publish the replacement world binding: {alert:?}" + ); + assert_eq!( + alert.get("world_generation").and_then(Value::as_u64), + Some(1), + "world_restarted alert must publish the replacement world generation: {alert:?}" + ); + assert_eq!( + alert + .pointer("/data/previous_world_id") + .and_then(Value::as_str), + Some("wld_stub_0001"), + "world_restarted alert must retain the previous world_id: {alert:?}" + ); + assert_eq!( + alert.pointer("/data/new_world_id").and_then(Value::as_str), + Some("wld_stub_0002"), + "world_restarted alert must retain the replacement world_id: {alert:?}" + ); +} + +#[cfg(target_os = "linux")] +#[test] +#[serial] +fn c3_fail_closed_drift_repersists_binding_before_world_restart_required_publishes() { + let temp = temp_dir("substrate-c3-world-restart-required-binding-"); + let home = temp.path().join("home"); + let project = temp.path().join("project"); + let substrate_home = home.join(".substrate"); + let trace_path = home.join(".substrate/trace.jsonl"); + fs::create_dir_all(&home).expect("create home"); + fs::create_dir_all(&project).expect("create project"); + fs::create_dir_all(&substrate_home).expect("create substrate home"); + fs::write(&trace_path, "").expect("seed trace"); + write_profile(&project); + let fake_codex = write_fake_codex_script(temp.path()); + write_orchestrator_runtime_world_config(&substrate_home, &fake_codex, "fail_closed"); + + let sock_temp = short_socket_dir("sub-c3ws-world-restart-required-binding-"); + let sock = sock_temp.path().join("world.sock"); + let server = ReplWorldAgentStub::start(&sock, StreamBehavior::Normal); + let records = server.records(); + + let mut repl = PtyRepl::spawn(&project, &home, &substrate_home, &sock, &[], &["--world"]); + + repl.wait_for_output("Substrate v", Duration::from_secs(6)) + .expect("banner"); + repl.wait_for_prompt(Duration::from_secs(2)) + .expect("initial prompt"); + launch_host_runtime_via_targeted_turn(&mut repl, "cli:codex"); + let orchestration_session_id = load_single_orchestration_session_id(&substrate_home); + let session_path = orchestration_session_path(&substrate_home, &orchestration_session_id); + repl.send_line("echo first"); + wait_for_min_records(&records, 1, 1, Duration::from_secs(3)); + repl.wait_for_output("first", Duration::from_secs(3)) + .expect("first command output"); + + write_policy(&substrate_home, false); + std::thread::sleep(Duration::from_millis(25)); + + repl.send_line("echo second"); + repl.wait_for_output( + "[shell] world restart required due to policy snapshot drift", + Duration::from_secs(5), + ) + .expect("world_restart_required alert output"); + + let persisted = read_orchestration_session(&session_path); + assert_session_world_binding(&persisted, Some("wld_stub_0001"), Some(0)); + + let start = Instant::now(); + while start.elapsed() < Duration::from_secs(3) { + if repl.try_wait().expect("try_wait") { + break; + } + std::thread::sleep(Duration::from_millis(50)); + } + let (code, _out) = repl.shutdown(); + assert_eq!(code, 3, "expected fail-closed drift exit code 3"); + + let events = read_trace(&trace_path); + let alerts = alert_rows_by_code(&events, "world_restart_required"); + assert_eq!( + alerts.len(), + 1, + "expected one world_restart_required alert: {alerts:?}" + ); + let alert = alerts[0]; + assert_eq!( + alert.get("orchestration_session_id").and_then(Value::as_str), + Some(orchestration_session_id.as_str()), + "world_restart_required alert must retain the authoritative orchestration_session_id: {alert:?}" + ); + assert_eq!( + alert.get("world_id").and_then(Value::as_str), + Some("wld_stub_0001"), + "world_restart_required alert must publish the current authoritative world binding: {alert:?}" + ); + assert_eq!( + alert.get("world_generation").and_then(Value::as_u64), + Some(0), + "world_restart_required alert must publish the current authoritative world generation: {alert:?}" + ); +} + +#[cfg(target_os = "linux")] +#[test] +#[serial] +fn c3_world_restart_invalidates_stale_member_generation_before_publish() { + let temp = temp_dir("substrate-c3-world-restart-invalidates-stale-members-"); + let home = temp.path().join("home"); + let project = temp.path().join("project"); + let substrate_home = home.join(".substrate"); + let trace_path = home.join(".substrate/trace.jsonl"); + fs::create_dir_all(&home).expect("create home"); + fs::create_dir_all(&project).expect("create project"); + fs::create_dir_all(&substrate_home).expect("create substrate home"); + fs::write(&trace_path, "").expect("seed trace"); + write_profile(&project); + let fake_codex = write_fake_codex_script(temp.path()); + write_orchestrator_runtime_world_config(&substrate_home, &fake_codex, "auto_restart"); + + let sock_temp = short_socket_dir("sub-c3ws-world-restart-invalidates-stale-members-"); + let sock = sock_temp.path().join("world.sock"); + let server = ReplWorldAgentStub::start(&sock, StreamBehavior::Normal); + let records = server.records(); + + let mut repl = PtyRepl::spawn(&project, &home, &substrate_home, &sock, &[], &["--world"]); + repl.wait_for_output("Substrate v", Duration::from_secs(6)) + .expect("banner"); + repl.wait_for_prompt(Duration::from_secs(2)) + .expect("initial prompt"); + launch_host_runtime_via_targeted_turn(&mut repl, "cli:codex"); + repl.send_line("echo first"); + wait_for_min_records(&records, 1, 1, Duration::from_secs(3)); + repl.wait_for_output("first", Duration::from_secs(3)) + .expect("first command output"); + + let orchestration_session_id = load_single_orchestration_session_id(&substrate_home); + let session_path = orchestration_session_path(&substrate_home, &orchestration_session_id); + let initial_session = read_orchestration_session(&session_path); + let world_id = initial_session + .get("world_id") + .and_then(Value::as_str) + .expect("initial world_id") + .to_string(); + let orchestrator_participant_id = initial_session + .get("active_session_handle_id") + .and_then(Value::as_str) + .expect("active_session_handle_id") + .to_string(); + + let stale_member_ids = (0..256) + .map(|idx| format!("ash_member_stale_{idx:03}")) + .collect::>(); + for participant_id in &stale_member_ids { + write_live_world_member_manifest( + &substrate_home, + &orchestration_session_id, + participant_id, + &orchestrator_participant_id, + &world_id, + 0, + ); + } + + write_policy(&substrate_home, false); + std::thread::sleep(Duration::from_millis(25)); + + repl.send_line("echo second"); + let alert = wait_for_world_restarted_alert_without_stale_liveness( + &trace_path, + &substrate_home, + &orchestration_session_id, + 0, + Duration::from_secs(15), + ); + assert_eq!( + alert.get("world_generation").and_then(Value::as_u64), + Some(1), + "world_restarted alert must publish replacement generation: {alert:?}" + ); + + repl.wait_for_output("second", Duration::from_secs(3)) + .expect("second command output"); + repl.send_line("exit"); + let (_code, _out) = repl.shutdown_graceful(Duration::from_secs(3)); + + let persisted = read_orchestration_session(&session_path); + assert_session_world_binding(&persisted, Some("wld_stub_0002"), Some(1)); + assert!( + live_world_member_generations_for_session(&substrate_home, &orchestration_session_id) + .is_empty(), + "stale generation members must not remain authoritative-live after restart" + ); + for participant_id in &stale_member_ids { + let manifest = read_participant_manifest(&substrate_home, participant_id); + assert_eq!( + manifest.get("state").and_then(Value::as_str), + Some("invalidated"), + "stale member must be invalidated: {manifest:?}" + ); + assert_eq!( + manifest + .pointer("/internal/termination_reason") + .and_then(Value::as_str), + Some("world generation invalidated by replacement binding"), + "stale member invalidation reason must explain generation rollover: {manifest:?}" + ); + } +} + +#[cfg(target_os = "linux")] +#[test] +#[serial] +fn c3_world_restart_launches_live_member_replacement_on_new_generation() { + let temp = temp_dir("substrate-c3-world-restart-live-member-replacement-"); + let home = temp.path().join("home"); + let project = temp.path().join("project"); + let substrate_home = home.join(".substrate"); + let trace_path = home.join(".substrate/trace.jsonl"); + fs::create_dir_all(&home).expect("create home"); + fs::create_dir_all(&project).expect("create project"); + fs::create_dir_all(&substrate_home).expect("create substrate home"); + fs::write(&trace_path, "").expect("seed trace"); + write_profile(&project); + let fake_orchestrator = write_fake_claude_script(temp.path()); + let fake_member = write_fake_codex_script(temp.path()); + write_orchestrator_and_world_member_runtime_world_config( + &substrate_home, + &fake_orchestrator, + &fake_member, + "auto_restart", + ); + + let sock_temp = short_socket_dir("sub-c3ws-live-member-replacement-"); + let sock = sock_temp.path().join("world.sock"); + let server = ReplWorldAgentStub::start_with_member_dispatch_scripts( + &sock, + StreamBehavior::Normal, + vec![ + MemberDispatchStreamScript::ReadyAndHoldUntilCancel { + session_handle_id: "session-first-member".to_string(), + exit_code_on_cancel: 130, + }, + MemberDispatchStreamScript::ReadyAndHoldUntilCancel { + session_handle_id: "session-replacement-member".to_string(), + exit_code_on_cancel: 130, + }, + ], + ); + let records = server.records(); + + let mut repl = PtyRepl::spawn(&project, &home, &substrate_home, &sock, &[], &["--world"]); + repl.wait_for_output("Substrate v", Duration::from_secs(6)) + .expect("banner"); + repl.wait_for_prompt(Duration::from_secs(2)) + .expect("initial prompt"); + launch_host_runtime_via_targeted_turn(&mut repl, "cli:claude_code"); + let orchestration_session_id = load_single_orchestration_session_id(&substrate_home); + let session_path = orchestration_session_path(&substrate_home, &orchestration_session_id); + + repl.send_line("echo first"); + wait_for_min_records(&records, 1, 1, Duration::from_secs(3)); + repl.wait_for_output("first", Duration::from_secs(3)) + .expect("first command output"); + let first_live_members = wait_for_live_world_member_count( + &substrate_home, + &orchestration_session_id, + 1, + Duration::from_secs(5), + ); + let first_member = &first_live_members[0]; + let first_member_id = first_member + .get("participant_id") + .and_then(Value::as_str) + .expect("first member participant_id") + .to_string(); + let orchestrator_participant_id = first_member + .get("orchestrator_participant_id") + .and_then(Value::as_str) + .expect("member orchestrator_participant_id") + .to_string(); + + write_member_runtime_policy(&substrate_home, false); + std::thread::sleep(Duration::from_millis(25)); + + repl.send_line("echo second"); + let alert = wait_for_world_restarted_alert_without_stale_liveness( + &trace_path, + &substrate_home, + &orchestration_session_id, + 0, + Duration::from_secs(15), + ); + wait_for_min_records(&records, 2, 1, Duration::from_secs(3)); + repl.wait_for_output("second", Duration::from_secs(3)) + .expect("second command output"); + + let replacement_live_members = wait_for_live_world_member_count( + &substrate_home, + &orchestration_session_id, + 1, + Duration::from_secs(5), + ); + let replacement = &replacement_live_members[0]; + let replacement_id = replacement + .get("participant_id") + .and_then(Value::as_str) + .expect("replacement participant_id"); + assert_ne!( + replacement_id, first_member_id, + "restart must create a distinct replacement member runtime" + ); + assert_eq!( + replacement.get("world_generation").and_then(Value::as_u64), + Some(1), + "replacement member must bind to the new world generation" + ); + assert_eq!( + replacement.get("world_id").and_then(Value::as_str), + alert.get("world_id").and_then(Value::as_str), + "replacement member must bind to the replacement world id" + ); + assert_eq!( + replacement + .get("orchestrator_participant_id") + .and_then(Value::as_str), + Some(orchestrator_participant_id.as_str()), + "replacement member must preserve the retained-control seam" + ); + assert_eq!( + replacement + .get("resumed_from_participant_id") + .and_then(Value::as_str), + Some(first_member_id.as_str()), + "replacement member must retain explicit lineage to the previous generation" + ); + + let stale = read_participant_manifest(&substrate_home, &first_member_id); + assert_eq!( + stale.get("state").and_then(Value::as_str), + Some("invalidated"), + "previous generation member must be invalidated after replacement launch: {stale:?}" + ); + + let persisted = read_orchestration_session(&session_path); + assert_session_world_binding(&persisted, Some("wld_stub_0002"), Some(1)); + + repl.send_line("exit"); + let (_code, _out) = repl.shutdown_graceful(Duration::from_secs(3)); +} + +#[cfg(target_os = "linux")] +#[test] +#[serial] +fn c3_world_restart_failed_member_replacement_leaves_honest_absence() { + let temp = temp_dir("substrate-c3-world-restart-failed-member-replacement-"); + let home = temp.path().join("home"); + let project = temp.path().join("project"); + let substrate_home = home.join(".substrate"); + let trace_path = home.join(".substrate/trace.jsonl"); + fs::create_dir_all(&home).expect("create home"); + fs::create_dir_all(&project).expect("create project"); + fs::create_dir_all(&substrate_home).expect("create substrate home"); + fs::write(&trace_path, "").expect("seed trace"); + write_profile(&project); + let fake_orchestrator = write_fake_claude_script(temp.path()); + let fake_member = + write_fake_codex_script_first_success_then_fail_without_session_handle(temp.path()); + write_orchestrator_and_world_member_runtime_world_config( + &substrate_home, + &fake_orchestrator, + &fake_member, + "auto_restart", + ); + + let sock_temp = short_socket_dir("sub-c3ws-failed-member-replacement-"); + let sock = sock_temp.path().join("world.sock"); + let server = ReplWorldAgentStub::start_with_member_dispatch_scripts( + &sock, + StreamBehavior::Normal, + vec![ + MemberDispatchStreamScript::ReadyAndHoldUntilCancel { + session_handle_id: "session-first-member".to_string(), + exit_code_on_cancel: 130, + }, + MemberDispatchStreamScript::ExitWithoutReady { exit_code: 1 }, + ], + ); + let records = server.records(); - fn wait_for_output(&self, needle: &str, timeout: Duration) -> anyhow::Result<()> { - let deadline = Instant::now() + timeout; - while Instant::now() < deadline { - let out = self.output_string(); - if out.contains(needle) { - return Ok(()); - } - std::thread::sleep(Duration::from_millis(25)); - } - anyhow::bail!( - "timed out waiting for output containing `{}`; output so far:\n{}", - needle, - self.output_string() - ) - } + let mut repl = PtyRepl::spawn(&project, &home, &substrate_home, &sock, &[], &["--world"]); + repl.wait_for_output("Substrate v", Duration::from_secs(6)) + .expect("banner"); + repl.wait_for_prompt(Duration::from_secs(2)) + .expect("initial prompt"); + launch_host_runtime_via_targeted_turn(&mut repl, "cli:claude_code"); + let orchestration_session_id = load_single_orchestration_session_id(&substrate_home); + let session_path = orchestration_session_path(&substrate_home, &orchestration_session_id); - fn wait_for_prompt(&self, timeout: Duration) -> anyhow::Result<()> { - self.wait_for_output("substrate> ", timeout) - } + repl.send_line("echo first"); + wait_for_min_records(&records, 1, 1, Duration::from_secs(3)); + repl.wait_for_output("first", Duration::from_secs(3)) + .expect("first command output"); + let first_live_members = wait_for_live_world_member_count( + &substrate_home, + &orchestration_session_id, + 1, + Duration::from_secs(5), + ); + let first_member_id = first_live_members[0] + .get("participant_id") + .and_then(Value::as_str) + .expect("first member participant_id") + .to_string(); - fn try_wait(&mut self) -> anyhow::Result { - if self.waited.is_some() { - return Ok(true); - } - if let Some(status) = self.child.try_wait()? { - self.waited = Some(status); - return Ok(true); - } - Ok(false) - } + write_member_runtime_policy(&substrate_home, false); + std::thread::sleep(Duration::from_millis(25)); - fn shutdown_graceful(mut self, timeout: Duration) -> (i32, String) { - let deadline = Instant::now() + timeout; - while self.waited.is_none() && Instant::now() < deadline { - match self.try_wait() { - Ok(true) => break, - Ok(false) => std::thread::sleep(Duration::from_millis(25)), - Err(_) => break, - } + repl.send_line("echo second"); + let _alert = wait_for_world_restarted_alert_without_stale_liveness( + &trace_path, + &substrate_home, + &orchestration_session_id, + 0, + Duration::from_secs(15), + ); + let start = Instant::now(); + while start.elapsed() < Duration::from_secs(3) { + if repl.try_wait().expect("try_wait") { + break; } - self.shutdown() + std::thread::sleep(Duration::from_millis(50)); } - fn shutdown(mut self) -> (i32, String) { - self.stop_reader.store(true, Ordering::Relaxed); - self.master.take(); - if self.waited.is_none() { - let _ = self.child.kill(); - let status = self.child.wait().expect("wait child"); - self.waited = Some(status); - } - if let Some(handle) = self.reader_handle.take() { - let _ = handle.join(); - } - let code = self - .waited - .as_ref() - .map(|s| s.exit_code() as i32) - .unwrap_or(-1); - (code, self.output_string()) - } + let (code, out) = repl.shutdown(); + assert_eq!( + code, 1, + "replacement startup failure must fail closed instead of continuing; output:\n{out}" + ); + assert!( + out.contains("world-scoped member runtime exited with status 1 before ownership could be established"), + "replacement startup failure must surface the member bootstrap error; output:\n{out}" + ); + assert!( + !out.contains("__PERSISTENT_EXEC_STUB__ eof echo second"), + "replacement startup failure must block the second command from executing; output:\n{out}" + ); + + let guard = records.lock().expect("lock records"); + assert_eq!( + guard.persistent_execs.len(), + 1, + "fail-closed member startup must not fall through to a host-local or restarted-world second exec; records: {guard:#?}" + ); + assert!( + guard.persistent_start_sessions.len() >= 2, + "world restart should still allocate the replacement world before the member bootstrap failure; records: {guard:#?}" + ); + drop(guard); + + let live_members = wait_for_live_world_member_count( + &substrate_home, + &orchestration_session_id, + 0, + Duration::from_secs(5), + ); + assert!( + live_members.is_empty(), + "replacement failure must leave no authoritative-live world member" + ); + let stale = read_participant_manifest(&substrate_home, &first_member_id); + assert_eq!( + stale.get("state").and_then(Value::as_str), + Some("invalidated"), + "replacement failure must not resurrect the stale member: {stale:?}" + ); + assert!( + world_member_manifests_for_session(&substrate_home, &orchestration_session_id) + .into_iter() + .all(|manifest| !participant_is_authoritative_live(&manifest)), + "replacement failure must leave honest absence rather than any authoritative-live member" + ); + let persisted = read_orchestration_session(&session_path); + assert_session_world_binding(&persisted, Some("wld_stub_0002"), Some(1)); } +#[cfg(target_os = "linux")] #[test] #[serial] -fn c3_host_directive_is_gated_disabled_by_default() { - let temp = temp_dir("substrate-c3-host-gate-off-"); +fn c3_world_restart_missing_member_replacement_leaves_absence() { + let temp = temp_dir("substrate-c3-world-restart-member-absence-"); let home = temp.path().join("home"); let project = temp.path().join("project"); let substrate_home = home.join(".substrate"); + let trace_path = home.join(".substrate/trace.jsonl"); fs::create_dir_all(&home).expect("create home"); fs::create_dir_all(&project).expect("create project"); fs::create_dir_all(&substrate_home).expect("create substrate home"); - fs::write(home.join(".substrate/trace.jsonl"), "").expect("seed trace"); + fs::write(&trace_path, "").expect("seed trace"); write_profile(&project); - write_policy(&substrate_home, true); + let fake_codex = write_fake_codex_script(temp.path()); + write_orchestrator_runtime_world_config(&substrate_home, &fake_codex, "auto_restart"); - let sock_temp = short_socket_dir("sub-c3ws-host-off-"); + let sock_temp = short_socket_dir("sub-c3ws-world-restart-member-absence-"); let sock = sock_temp.path().join("world.sock"); - let _server = ReplWorldAgentStub::start(&sock, StreamBehavior::Normal); + let server = ReplWorldAgentStub::start(&sock, StreamBehavior::Normal); + let records = server.records(); let mut repl = PtyRepl::spawn(&project, &home, &substrate_home, &sock, &[], &["--world"]); - - repl.wait_for_output("Substrate v", Duration::from_secs(2)) + repl.wait_for_output("Substrate v", Duration::from_secs(6)) .expect("banner"); repl.wait_for_prompt(Duration::from_secs(2)) - .expect("prompt"); + .expect("initial prompt"); + launch_host_runtime_via_targeted_turn(&mut repl, "cli:codex"); + repl.send_line("echo first"); + wait_for_min_records(&records, 1, 1, Duration::from_secs(3)); + repl.wait_for_output("first", Duration::from_secs(3)) + .expect("first command output"); - repl.send_line(":host pwd"); - repl.wait_for_output("host escape", Duration::from_secs(2)) - .expect("host-escape gating message"); - repl.send_line("exit"); + let orchestration_session_id = load_single_orchestration_session_id(&substrate_home); + let session_path = orchestration_session_path(&substrate_home, &orchestration_session_id); + let initial_session = read_orchestration_session(&session_path); + write_live_world_member_manifest( + &substrate_home, + &orchestration_session_id, + "ash_member_stale_only", + initial_session + .get("active_session_handle_id") + .and_then(Value::as_str) + .expect("active_session_handle_id"), + initial_session + .get("world_id") + .and_then(Value::as_str) + .expect("initial world_id"), + 0, + ); - let (_code, out) = repl.shutdown_graceful(Duration::from_secs(2)); + write_policy(&substrate_home, false); + std::thread::sleep(Duration::from_millis(25)); + repl.send_line("echo second"); + let _alert = wait_for_world_restarted_alert_without_stale_liveness( + &trace_path, + &substrate_home, + &orchestration_session_id, + 0, + Duration::from_secs(5), + ); assert!( - out.to_ascii_lowercase().contains("host escape"), - "expected a host-escape gating error, got output:\n{out}" + live_world_member_generations_for_session(&substrate_home, &orchestration_session_id) + .is_empty(), + "missing member replacement must leave absence rather than stale liveness" + ); + + repl.wait_for_output("second", Duration::from_secs(3)) + .expect("second command output"); + repl.send_line("exit"); + let (_code, _out) = repl.shutdown_graceful(Duration::from_secs(3)); + + let stale = read_participant_manifest(&substrate_home, "ash_member_stale_only"); + assert_eq!( + stale.get("state").and_then(Value::as_str), + Some("invalidated"), + "stale member must remain invalidated when no replacement is created: {stale:?}" ); + let persisted = read_orchestration_session(&session_path); + assert_session_world_binding(&persisted, Some("wld_stub_0002"), Some(1)); } +#[cfg(target_os = "linux")] #[test] #[serial] -fn c3_host_directive_executes_on_host_when_enabled() { - let temp = temp_dir("substrate-c3-host-gate-on-"); +fn c3_world_restart_replacement_generation_becomes_only_live_generation() { + let temp = temp_dir("substrate-c3-world-restart-member-replacement-"); let home = temp.path().join("home"); let project = temp.path().join("project"); let substrate_home = home.join(".substrate"); + let trace_path = home.join(".substrate/trace.jsonl"); fs::create_dir_all(&home).expect("create home"); fs::create_dir_all(&project).expect("create project"); fs::create_dir_all(&substrate_home).expect("create substrate home"); - fs::write(home.join(".substrate/trace.jsonl"), "").expect("seed trace"); + fs::write(&trace_path, "").expect("seed trace"); write_profile(&project); - write_policy(&substrate_home, true); + let fake_codex = write_fake_codex_script(temp.path()); + write_orchestrator_runtime_world_config(&substrate_home, &fake_codex, "auto_restart"); - let sock_temp = short_socket_dir("sub-c3ws-host-on-"); + let sock_temp = short_socket_dir("sub-c3ws-world-restart-member-replacement-"); let sock = sock_temp.path().join("world.sock"); - let _server = ReplWorldAgentStub::start(&sock, StreamBehavior::Normal); + let server = ReplWorldAgentStub::start(&sock, StreamBehavior::Normal); + let records = server.records(); - let mut repl = PtyRepl::spawn( - &project, - &home, + let mut repl = PtyRepl::spawn(&project, &home, &substrate_home, &sock, &[], &["--world"]); + repl.wait_for_output("Substrate v", Duration::from_secs(6)) + .expect("banner"); + repl.wait_for_prompt(Duration::from_secs(2)) + .expect("initial prompt"); + launch_host_runtime_via_targeted_turn(&mut repl, "cli:codex"); + repl.send_line("echo first"); + wait_for_min_records(&records, 1, 1, Duration::from_secs(3)); + repl.wait_for_output("first", Duration::from_secs(3)) + .expect("first command output"); + + let orchestration_session_id = load_single_orchestration_session_id(&substrate_home); + let session_path = orchestration_session_path(&substrate_home, &orchestration_session_id); + let initial_session = read_orchestration_session(&session_path); + let orchestrator_participant_id = initial_session + .get("active_session_handle_id") + .and_then(Value::as_str) + .expect("active_session_handle_id") + .to_string(); + write_live_world_member_manifest( &substrate_home, - &sock, - &[("SUBSTRATE_REPL_HOST_ESCAPE", "1")], - &["--world"], + &orchestration_session_id, + "ash_member_old_generation", + &orchestrator_participant_id, + initial_session + .get("world_id") + .and_then(Value::as_str) + .expect("initial world_id"), + 0, ); - repl.wait_for_output("Substrate v", Duration::from_secs(2)) - .expect("banner"); - repl.wait_for_prompt(Duration::from_secs(2)) - .expect("prompt"); + write_policy(&substrate_home, false); + std::thread::sleep(Duration::from_millis(25)); - let project = fs::canonicalize(&project).unwrap_or(project); - let project_str = project.to_string_lossy().into_owned(); - repl.send_line(":host pwd"); - repl.wait_for_output(project_str.as_ref(), Duration::from_secs(2)) - .expect("host pwd output"); + repl.send_line("echo second"); + let alert = wait_for_world_restarted_alert_without_stale_liveness( + &trace_path, + &substrate_home, + &orchestration_session_id, + 0, + Duration::from_secs(5), + ); + let replacement_world_id = alert + .get("world_id") + .and_then(Value::as_str) + .expect("replacement world_id") + .to_string(); + let replacement_generation = alert + .get("world_generation") + .and_then(Value::as_u64) + .expect("replacement world_generation"); + write_live_world_member_manifest( + &substrate_home, + &orchestration_session_id, + "ash_member_new_generation", + &orchestrator_participant_id, + &replacement_world_id, + replacement_generation, + ); + assert_eq!( + live_world_member_generations_for_session(&substrate_home, &orchestration_session_id), + vec![("ash_member_new_generation".to_string(), 1)], + "replacement generation must become the only authoritative-live world member generation" + ); + + repl.wait_for_output("second", Duration::from_secs(3)) + .expect("second command output"); repl.send_line("exit"); + let (_code, _out) = repl.shutdown_graceful(Duration::from_secs(3)); - let (_code, out) = repl.shutdown_graceful(Duration::from_secs(2)); - assert!( - out.contains(project_str.as_str()), - "expected :host pwd to print the host cwd ({project_str}), got output:\n{out}" + let stale = read_participant_manifest(&substrate_home, "ash_member_old_generation"); + assert_eq!( + stale.get("state").and_then(Value::as_str), + Some("invalidated"), + "stale generation must not remain live after replacement is present: {stale:?}" + ); + let persisted = read_orchestration_session(&session_path); + assert_session_world_binding( + &persisted, + Some(replacement_world_id.as_str()), + Some(replacement_generation), ); } +#[cfg(target_os = "linux")] #[test] #[serial] -fn c3_pty_directive_routes_to_persistent_session_when_world_enabled() { - let temp = temp_dir("substrate-c3-pty-world-"); +fn c3_world_restart_keeps_same_agent_members_in_other_sessions_isolated() { + let temp = temp_dir("substrate-c3-world-restart-cross-session-isolation-"); let home = temp.path().join("home"); let project = temp.path().join("project"); let substrate_home = home.join(".substrate"); + let trace_path = home.join(".substrate/trace.jsonl"); fs::create_dir_all(&home).expect("create home"); fs::create_dir_all(&project).expect("create project"); fs::create_dir_all(&substrate_home).expect("create substrate home"); - fs::write(home.join(".substrate/trace.jsonl"), "").expect("seed trace"); + fs::write(&trace_path, "").expect("seed trace"); write_profile(&project); - write_policy(&substrate_home, true); + let fake_codex = write_fake_codex_script(temp.path()); + write_orchestrator_runtime_world_config(&substrate_home, &fake_codex, "auto_restart"); - let sock_temp = short_socket_dir("sub-c3ws-pty-world-"); + let sock_temp = short_socket_dir("sub-c3ws-cross-session-isolation-"); let sock = sock_temp.path().join("world.sock"); let server = ReplWorldAgentStub::start(&sock, StreamBehavior::Normal); let records = server.records(); let mut repl = PtyRepl::spawn(&project, &home, &substrate_home, &sock, &[], &["--world"]); - - repl.wait_for_output("Substrate v", Duration::from_secs(2)) + repl.wait_for_output("Substrate v", Duration::from_secs(6)) .expect("banner"); + repl.wait_for_prompt(Duration::from_secs(2)) + .expect("initial prompt"); + launch_host_runtime_via_targeted_turn(&mut repl, "cli:codex"); + repl.send_line("echo first"); + wait_for_min_records(&records, 1, 1, Duration::from_secs(3)); + repl.wait_for_output("first", Duration::from_secs(3)) + .expect("first command output"); - repl.send_line(":pty echo hello"); + let orchestration_session_id = load_single_orchestration_session_id(&substrate_home); + let session_path = orchestration_session_path(&substrate_home, &orchestration_session_id); + let initial_session = read_orchestration_session(&session_path); + let orchestrator_participant_id = initial_session + .get("active_session_handle_id") + .and_then(Value::as_str) + .expect("active_session_handle_id") + .to_string(); + let initial_world_id = initial_session + .get("world_id") + .and_then(Value::as_str) + .expect("initial world_id") + .to_string(); + + write_live_world_member_manifest( + &substrate_home, + &orchestration_session_id, + "ash_member_current_session", + &orchestrator_participant_id, + &initial_world_id, + 0, + ); + + let other_orchestration_session_id = "orch_other_same_agent"; + write_live_world_member_manifest( + &substrate_home, + other_orchestration_session_id, + "ash_member_other_session", + "ash_other_orchestrator", + "wld_other_0001", + 0, + ); + assert_eq!( + live_world_member_generations_for_session(&substrate_home, other_orchestration_session_id), + vec![("ash_member_other_session".to_string(), 0)], + "fixture must start with an independently live same-agent member in the other session" + ); + + write_policy(&substrate_home, false); + std::thread::sleep(Duration::from_millis(25)); + + repl.send_line("echo second"); + let alert = wait_for_world_restarted_alert_without_stale_liveness( + &trace_path, + &substrate_home, + &orchestration_session_id, + 0, + Duration::from_secs(5), + ); + assert_eq!( + alert.get("world_generation").and_then(Value::as_u64), + Some(1), + "restart must still advance the active session generation: {alert:?}" + ); + + assert_eq!( + live_world_member_generations_for_session(&substrate_home, &orchestration_session_id), + Vec::<(String, u64)>::new(), + "current-session stale members must be suppressed after restart" + ); + assert_eq!( + live_world_member_generations_for_session(&substrate_home, other_orchestration_session_id), + vec![("ash_member_other_session".to_string(), 0)], + "same-agent members in other orchestration sessions must stay isolated from current-session invalidation" + ); + + repl.wait_for_output("second", Duration::from_secs(3)) + .expect("second command output"); repl.send_line("exit"); + let (_code, _out) = repl.shutdown_graceful(Duration::from_secs(3)); - let (_code, _out) = repl.shutdown_graceful(Duration::from_secs(2)); + let current = read_participant_manifest(&substrate_home, "ash_member_current_session"); + assert_eq!( + current.get("state").and_then(Value::as_str), + Some("invalidated"), + "current-session stale member must be invalidated by the restart: {current:?}" + ); - let guard = records.lock().expect("lock records"); - assert!( - guard - .persistent_execs - .iter() - .any(|rec| rec.stdin_mode == "passthrough" && rec.program_utf8.trim() == "echo hello"), - "expected :pty to use persistent-session exec passthrough with stripped prefix; records: {guard:#?}" + let other = read_participant_manifest(&substrate_home, "ash_member_other_session"); + assert_eq!( + other.get("state").and_then(Value::as_str), + Some("ready"), + "same-agent member from another orchestration session must stay live: {other:?}" ); } +#[cfg(target_os = "linux")] #[test] #[serial] -fn c3_persistent_session_start_carries_canonical_net_allowed_snapshot() { - let temp = temp_dir("substrate-c3-net-allowed-pty-"); +fn c3_bootstrap_failure_after_attach_cleans_up_world_and_parent_session_state() { + let temp = temp_dir("substrate-c3-startup-bootstrap-cleanup-"); let home = temp.path().join("home"); let project = temp.path().join("project"); let substrate_home = home.join(".substrate"); @@ -668,93 +3710,61 @@ fn c3_persistent_session_start_carries_canonical_net_allowed_snapshot() { fs::create_dir_all(&substrate_home).expect("create substrate home"); fs::write(home.join(".substrate/trace.jsonl"), "").expect("seed trace"); write_profile(&project); - write_policy_with_net_allowed( - &substrate_home, - true, - "[\" Example.COM. \", \"example.com\", \"Api.Example.com.\"]", - ); - write_config(&substrate_home, true); + let fake_codex = write_fake_codex_script_without_session_handle(temp.path()); + write_orchestrator_runtime_world_config(&substrate_home, &fake_codex, "fail_closed"); - let sock_temp = short_socket_dir("sub-c3ws-net-allowed-"); + let sock_temp = short_socket_dir("sub-c3ws-startup-bootstrap-cleanup-"); let sock = sock_temp.path().join("world.sock"); let server = ReplWorldAgentStub::start(&sock, StreamBehavior::Normal); let records = server.records(); let mut repl = PtyRepl::spawn(&project, &home, &substrate_home, &sock, &[], &["--world"]); - repl.wait_for_output("Substrate v", Duration::from_secs(2)) + repl.wait_for_output("Substrate v", Duration::from_secs(6)) .expect("banner"); - repl.send_line("echo hello"); - repl.wait_for_output("hello", Duration::from_secs(3)) - .expect("command output"); - repl.send_line("exit"); - - let (_code, _out) = repl.shutdown_graceful(Duration::from_secs(2)); + repl.wait_for_prompt(Duration::from_secs(2)) + .expect("initial prompt"); + repl.send_line("::cli:codex bootstrap should fail"); + wait_for_min_start_sessions_with_output(&repl, &records, 1, Duration::from_secs(3)); - let guard = records.lock().expect("lock records"); - let start = guard - .persistent_start_sessions - .first() - .expect("persistent start session"); - let net_allowed = start - .policy_snapshot - .get("net_allowed") - .and_then(|value| value.as_array()) - .expect("policy_snapshot.net_allowed array"); - let net_allowed: Vec<&str> = net_allowed - .iter() - .map(|value| value.as_str().expect("net_allowed string")) - .collect(); - assert_eq!(net_allowed, vec!["example.com", "api.example.com"]); - assert_world_network_payload(start, true, &["example.com", "api.example.com"]); -} + let start = Instant::now(); + while start.elapsed() < Duration::from_secs(3) { + if repl.try_wait().expect("try_wait") { + break; + } + std::thread::sleep(Duration::from_millis(50)); + } -#[test] -#[serial] -fn c3_persistent_session_start_obeys_net_allowed_routing_matrix() { - let cases = [ - ReplRoutingCase { - name: "gate off plus restrictive policy stays allow-all at routing layer", - net_allowed_yaml: "[\" Example.COM. \", \"example.com\", \"Api.Example.com.\"]", - world_net_filter: false, - expected_net_allowed: &["example.com", "api.example.com"], - expected_isolate_network: false, - expected_allowed_domains: &[], - }, - ReplRoutingCase { - name: "gate on plus allow-all singleton does not request isolation", - net_allowed_yaml: "[\" * \"]", - world_net_filter: true, - expected_net_allowed: &["*"], - expected_isolate_network: false, - expected_allowed_domains: &[], - }, - ReplRoutingCase { - name: "gate on plus deny-all requests isolation with empty allowlist", - net_allowed_yaml: "[]", - world_net_filter: true, - expected_net_allowed: &[], - expected_isolate_network: true, - expected_allowed_domains: &[], - }, - ReplRoutingCase { - name: "gate on plus restrictive allowlist requests isolation with canonical domains", - net_allowed_yaml: "[\" Example.COM. \", \"example.com\", \"Api.Example.com.\"]", - world_net_filter: true, - expected_net_allowed: &["example.com", "api.example.com"], - expected_isolate_network: true, - expected_allowed_domains: &["example.com", "api.example.com"], - }, - ]; + let (code, out) = repl.shutdown(); + assert_eq!( + code, 1, + "expected targeted orchestrator bootstrap failure exit code 1, got output:\n{out}" + ); + assert!( + out.contains("attached control turn ended before ownership could be established") + || out.contains("failed to establish attached control ownership"), + "expected bootstrap failure output after world attach; output:\n{out}" + ); - for case in &cases { - run_repl_routing_case(case); + let session_count = fs::read_dir(sessions_dir(&substrate_home)) + .ok() + .into_iter() + .flat_map(|entries| entries.filter_map(Result::ok)) + .count(); + if session_count == 0 { + return; } + + let orchestration_session_id = load_single_orchestration_session_id(&substrate_home); + let session_path = orchestration_session_path(&substrate_home, &orchestration_session_id); + let persisted = read_orchestration_session(&session_path); + assert_session_world_binding(&persisted, None, None); } #[test] #[serial] -fn c3_drift_restart_restarts_session_and_emits_message() { +fn c3_drift_restart_restarts_session_and_suppresses_world_restarted_agent_event_without_explicit_context( +) { let temp = temp_dir("substrate-c3-drift-restart-"); let home = temp.path().join("home"); let project = temp.path().join("project"); @@ -774,7 +3784,7 @@ fn c3_drift_restart_restarts_session_and_emits_message() { let mut repl = PtyRepl::spawn(&project, &home, &substrate_home, &sock, &[], &["--world"]); - repl.wait_for_output("Substrate v", Duration::from_secs(2)) + repl.wait_for_output("Substrate v", Duration::from_secs(6)) .expect("banner"); repl.send_line("echo first"); @@ -804,94 +3814,23 @@ fn c3_drift_restart_restarts_session_and_emits_message() { "expected an operator-visible drift-restart message, got output:\n{out}" ); + assert_start_sessions_have_no_shared_world_context( + &guard, + "drift restart without explicit orchestration context", + ); drop(guard); let events = read_trace(&trace_path); - let alerts: Vec<&Value> = events - .iter() - .filter(|event| event.get("event_type").and_then(Value::as_str) == Some("agent_event")) - .filter(|event| event.get("kind").and_then(Value::as_str) == Some("alert")) - .filter(|event| { - event.pointer("/data/code").and_then(Value::as_str) == Some("world_restarted") - }) - .collect(); - assert_eq!( - alerts.len(), - 1, - "expected exactly one world_restarted alert record, got: {alerts:?}" - ); - - let alert = alerts[0]; - assert_eq!( - alert.get("agent_id").and_then(Value::as_str), - Some("shell"), - "restart alert must attribute to shell: {alert:?}" - ); - assert_eq!( - alert.get("backend_id").and_then(Value::as_str), - Some("shell:repl"), - "restart alert must preserve backend attribution: {alert:?}" - ); - assert_eq!( - alert.get("role").and_then(Value::as_str), - Some("orchestrator"), - "restart alert must carry role=orchestrator: {alert:?}" - ); - assert_eq!( - alert.get("world_id").and_then(Value::as_str), - Some("wld_stub_0002"), - "top-level world_id must point at the active replacement world: {alert:?}" - ); - assert_eq!( - alert.get("world_generation").and_then(Value::as_u64), - Some(1), - "top-level world_generation must point at the active replacement generation: {alert:?}" - ); - assert_eq!( - alert.pointer("/data/reason").and_then(Value::as_str), - Some("policy_snapshot_changed"), - "policy drift restart should classify as policy_snapshot_changed: {alert:?}" - ); - assert_eq!( - alert.pointer("/data/on_drift").and_then(Value::as_str), - Some("auto_restart"), - "restart alert must record on_drift=auto_restart: {alert:?}" - ); - assert_eq!( - alert - .pointer("/data/previous_world_id") - .and_then(Value::as_str), - Some("wld_stub_0001"), - "restart alert must capture previous world_id: {alert:?}" - ); - assert_eq!( - alert.pointer("/data/new_world_id").and_then(Value::as_str), - Some("wld_stub_0002"), - "restart alert must capture new world_id: {alert:?}" - ); - assert_eq!( - alert - .pointer("/data/previous_world_generation") - .and_then(Value::as_u64), - Some(0), - "restart alert must capture previous generation 0: {alert:?}" - ); - assert_eq!( - alert - .pointer("/data/new_world_generation") - .and_then(Value::as_u64), - Some(1), - "restart alert must capture new generation 1: {alert:?}" - ); - assert!( - alert.pointer("/data/world_id").is_none() && alert.pointer("/data/world_generation").is_none(), - "restart alerts must keep active world identity at the top level, not duplicate it under data: {alert:?}" + assert_no_alert_rows_by_code( + &events, + "world_restarted", + "drift restart without explicit orchestration context", ); } #[test] #[serial] -fn c3_startup_drift_restart_emits_world_restarted_alert() { +fn c3_startup_drift_restart_suppresses_world_restarted_agent_event_without_explicit_context() { let temp = temp_dir("substrate-c3-startup-drift-restart-"); let home = temp.path().join("home"); let project = temp.path().join("project"); @@ -918,7 +3857,7 @@ fn c3_startup_drift_restart_emits_world_restarted_alert() { let records = server.records(); let mut repl = PtyRepl::spawn(&child, &home, &substrate_home, &sock, &[], &["--world"]); - repl.wait_for_output("Substrate v", Duration::from_secs(2)) + repl.wait_for_output("Substrate v", Duration::from_secs(6)) .expect("banner"); wait_for_min_start_sessions(&records, 2, Duration::from_secs(3)); repl.send_line("exit"); @@ -931,72 +3870,28 @@ fn c3_startup_drift_restart_emits_world_restarted_alert() { "expected startup drift restart note, got output:\n{out}" ); - let events = read_trace(&trace_path); - let alerts: Vec<&Value> = events - .iter() - .filter(|event| event.get("event_type").and_then(Value::as_str) == Some("agent_event")) - .filter(|event| event.get("kind").and_then(Value::as_str) == Some("alert")) - .filter(|event| { - event.pointer("/data/code").and_then(Value::as_str) == Some("world_restarted") - }) - .collect(); - assert_eq!( - alerts.len(), - 1, - "expected exactly one startup world_restarted alert record, got: {alerts:?}" - ); - - let alert = alerts[0]; - assert_eq!( - alert.get("world_id").and_then(Value::as_str), - Some("wld_stub_0002"), - "startup restart alert must publish the active replacement world at the top level: {alert:?}" - ); - assert_eq!( - alert.get("world_generation").and_then(Value::as_u64), - Some(1), - "startup restart alert must publish the replacement generation at the top level: {alert:?}" - ); - assert_eq!( - alert.pointer("/data/reason").and_then(Value::as_str), - Some("workspace_root_changed"), - "startup drift restart should classify as workspace_root_changed: {alert:?}" - ); - assert_eq!( - alert - .pointer("/data/previous_world_id") - .and_then(Value::as_str), - Some("wld_stub_0001"), - "startup drift alert must capture the first world id: {alert:?}" - ); - assert_eq!( - alert.pointer("/data/new_world_id").and_then(Value::as_str), - Some("wld_stub_0002"), - "startup drift alert must capture the restarted world id: {alert:?}" - ); - assert_eq!( - alert - .pointer("/data/previous_world_generation") - .and_then(Value::as_u64), - Some(0), - "startup drift alert must start from generation 0: {alert:?}" + let guard = records.lock().expect("lock records"); + assert!( + guard.persistent_start_sessions.len() >= 2, + "expected startup drift to restart the session before prompt; records: {guard:#?}" ); - assert_eq!( - alert - .pointer("/data/new_world_generation") - .and_then(Value::as_u64), - Some(1), - "startup drift alert must increment to generation 1: {alert:?}" + assert_start_sessions_have_no_shared_world_context( + &guard, + "startup drift restart without explicit orchestration context", ); - assert!( - alert.pointer("/data/world_id").is_none() && alert.pointer("/data/world_generation").is_none(), - "startup restart alerts must keep active world identity at the top level, not under data: {alert:?}" + drop(guard); + + let events = read_trace(&trace_path); + assert_no_alert_rows_by_code( + &events, + "world_restarted", + "startup drift restart without explicit orchestration context", ); } #[test] #[serial] -fn c3_drift_fail_closed_emits_world_restart_required_alert() { +fn c3_drift_fail_closed_suppresses_world_restart_required_agent_event_without_explicit_context() { let temp = temp_dir("substrate-c3-drift-fail-closed-"); let home = temp.path().join("home"); let project = temp.path().join("project"); @@ -1016,7 +3911,7 @@ fn c3_drift_fail_closed_emits_world_restart_required_alert() { let records = server.records(); let mut repl = PtyRepl::spawn(&project, &home, &substrate_home, &sock, &[], &["--world"]); - repl.wait_for_output("Substrate v", Duration::from_secs(2)) + repl.wait_for_output("Substrate v", Duration::from_secs(6)) .expect("banner"); repl.send_line("echo first"); @@ -1064,75 +3959,24 @@ fn c3_drift_fail_closed_emits_world_restart_required_alert() { 1, "fail-closed drift must stop before the second exec reaches world-agent; records: {guard:#?}" ); + assert_start_sessions_have_no_shared_world_context( + &guard, + "fail-closed drift without explicit orchestration context", + ); drop(guard); let events = read_trace(&trace_path); - let alerts: Vec<&Value> = events - .iter() - .filter(|event| event.get("event_type").and_then(Value::as_str) == Some("agent_event")) - .filter(|event| event.get("kind").and_then(Value::as_str) == Some("alert")) - .filter(|event| { - event.pointer("/data/code").and_then(Value::as_str) == Some("world_restart_required") - }) - .collect(); - assert_eq!( - alerts.len(), - 1, - "expected exactly one world_restart_required alert record, got: {alerts:?}" - ); - - let alert = alerts[0]; - assert_eq!( - alert.get("agent_id").and_then(Value::as_str), - Some("shell"), - "restart-required alert must attribute to shell: {alert:?}" - ); - assert_eq!( - alert.get("backend_id").and_then(Value::as_str), - Some("shell:repl"), - "restart-required alert must preserve backend attribution: {alert:?}" - ); - assert_eq!( - alert.get("role").and_then(Value::as_str), - Some("orchestrator"), - "restart-required alert must carry role=orchestrator: {alert:?}" - ); - assert_eq!( - alert.get("world_id").and_then(Value::as_str), - Some("wld_stub_0001"), - "top-level world_id must point at the current world: {alert:?}" - ); - assert_eq!( - alert.get("world_generation").and_then(Value::as_u64), - Some(0), - "top-level world_generation must publish the current generation for fail-closed restart-required alerts: {alert:?}" - ); - assert_eq!( - alert.pointer("/data/reason").and_then(Value::as_str), - Some("policy_snapshot_changed"), - "policy drift fail-closed alert should classify as policy_snapshot_changed: {alert:?}" - ); - assert_eq!( - alert.pointer("/data/on_drift").and_then(Value::as_str), - Some("fail_closed"), - "restart-required alert must record on_drift=fail_closed: {alert:?}" - ); - assert_eq!( - alert - .pointer("/data/required_action") - .and_then(Value::as_str), - Some("restart_world"), - "restart-required alert must record required_action=restart_world: {alert:?}" - ); - assert!( - alert.pointer("/data/world_id").is_none() && alert.pointer("/data/world_generation").is_none(), - "restart-required alerts must keep current world identity at the top level when known: {alert:?}" + assert_no_alert_rows_by_code( + &events, + "world_restart_required", + "fail-closed drift without explicit orchestration context", ); } #[test] #[serial] -fn c3_startup_drift_fail_closed_emits_world_restart_required_alert() { +fn c3_startup_drift_fail_closed_suppresses_world_restart_required_agent_event_without_explicit_context( +) { let temp = temp_dir("substrate-c3-startup-drift-fail-closed-"); let home = temp.path().join("home"); let project = temp.path().join("project"); @@ -1160,7 +4004,7 @@ fn c3_startup_drift_fail_closed_emits_world_restart_required_alert() { let records = server.records(); let mut repl = PtyRepl::spawn(&child, &home, &substrate_home, &sock, &[], &["--world"]); - repl.wait_for_output("Substrate v", Duration::from_secs(2)) + repl.wait_for_output("Substrate v", Duration::from_secs(6)) .expect("banner"); let start = Instant::now(); @@ -1189,54 +4033,17 @@ fn c3_startup_drift_fail_closed_emits_world_restart_required_alert() { 1, "startup fail-closed drift must not auto-restart the world session; records: {guard:#?}" ); + assert_start_sessions_have_no_shared_world_context( + &guard, + "startup fail-closed drift without explicit orchestration context", + ); drop(guard); let events = read_trace(&trace_path); - let alerts: Vec<&Value> = events - .iter() - .filter(|event| event.get("event_type").and_then(Value::as_str) == Some("agent_event")) - .filter(|event| event.get("kind").and_then(Value::as_str) == Some("alert")) - .filter(|event| { - event.pointer("/data/code").and_then(Value::as_str) == Some("world_restart_required") - }) - .collect(); - assert_eq!( - alerts.len(), - 1, - "expected exactly one startup world_restart_required alert record, got: {alerts:?}" - ); - - let alert = alerts[0]; - assert_eq!( - alert.get("world_id").and_then(Value::as_str), - Some("wld_stub_0001"), - "startup fail-closed alert must publish the current world id at the top level: {alert:?}" - ); - assert_eq!( - alert.get("world_generation").and_then(Value::as_u64), - Some(0), - "startup fail-closed alert must publish the current generation at the top level: {alert:?}" - ); - assert_eq!( - alert.pointer("/data/reason").and_then(Value::as_str), - Some("workspace_root_changed"), - "startup fail-closed drift should classify as workspace_root_changed: {alert:?}" - ); - assert_eq!( - alert.pointer("/data/on_drift").and_then(Value::as_str), - Some("fail_closed"), - "startup fail-closed alert must record on_drift=fail_closed: {alert:?}" - ); - assert_eq!( - alert - .pointer("/data/required_action") - .and_then(Value::as_str), - Some("restart_world"), - "startup fail-closed alert must record required_action=restart_world: {alert:?}" - ); - assert!( - alert.pointer("/data/world_id").is_none() && alert.pointer("/data/world_generation").is_none(), - "startup fail-closed alerts must keep current world identity at the top level when known: {alert:?}" + assert_no_alert_rows_by_code( + &events, + "world_restart_required", + "startup fail-closed drift without explicit orchestration context", ); } @@ -1323,7 +4130,10 @@ fn c3_startup_surfaces_fatal_persistent_session_error_before_ready() { #[test] #[serial] fn c3_drift_restart_refreshes_anchor_env_for_new_cwd() { - let temp = temp_dir("substrate-c3-anchor-drift-"); + let temp = tempfile::Builder::new() + .prefix("substrate-c3-anchor-drift-") + .tempdir_in("/tmp") + .expect("create anchor drift tempdir in /tmp"); let home = temp.path().join("home"); let project = temp.path().join("project"); let child = project.join("child"); @@ -1344,7 +4154,7 @@ fn c3_drift_restart_refreshes_anchor_env_for_new_cwd() { let records = server.records(); let mut repl = PtyRepl::spawn(&child, &home, &substrate_home, &sock, &[], &["--world"]); - repl.wait_for_output("Substrate v", Duration::from_secs(2)) + repl.wait_for_output("Substrate v", Duration::from_secs(6)) .expect("banner"); let project_canon = project.canonicalize().unwrap_or(project.clone()); @@ -1407,11 +4217,12 @@ fn c3_drift_restart_refreshes_world_network_routing() { let home = temp.path().join("home"); let project = temp.path().join("project"); let substrate_home = home.join(".substrate"); + let trace_path = home.join(".substrate/trace.jsonl"); fs::create_dir_all(&home).expect("create home"); fs::create_dir_all(&project).expect("create project"); fs::create_dir_all(&substrate_home).expect("create substrate home"); - fs::write(home.join(".substrate/trace.jsonl"), "").expect("seed trace"); + fs::write(&trace_path, "").expect("seed trace"); write_profile(&project); write_policy_with_net_allowed(&substrate_home, true, "[\"*\"]"); @@ -1423,7 +4234,7 @@ fn c3_drift_restart_refreshes_world_network_routing() { let records = server.records(); let mut repl = PtyRepl::spawn(&project, &home, &substrate_home, &sock, &[], &["--world"]); - repl.wait_for_output("Substrate v", Duration::from_secs(2)) + repl.wait_for_output("Substrate v", Duration::from_secs(6)) .expect("banner"); repl.send_line("echo first"); @@ -1456,6 +4267,10 @@ fn c3_drift_restart_refreshes_world_network_routing() { assert_world_network_payload(first, false, &[]); assert_world_network_payload(second, true, &["example.com", "api.example.com"]); + assert_start_sessions_have_no_shared_world_context( + &guard, + "network-routing drift restart without explicit orchestration context", + ); let second_net_allowed = second .policy_snapshot @@ -1467,4 +4282,13 @@ fn c3_drift_restart_refreshes_world_network_routing() { .map(|value| value.as_str().expect("net_allowed string")) .collect(); assert_eq!(second_net_allowed, vec!["example.com", "api.example.com"]); + + drop(guard); + + let events = read_trace(&trace_path); + assert_no_alert_rows_by_code( + &events, + "world_restarted", + "network-routing drift restart without explicit orchestration context", + ); } diff --git a/crates/shell/tests/support/mod.rs b/crates/shell/tests/support/mod.rs index e44a1c548..cae0e3baa 100644 --- a/crates/shell/tests/support/mod.rs +++ b/crates/shell/tests/support/mod.rs @@ -5,9 +5,10 @@ pub mod common; use assert_cmd::Command; +use std::env; use std::fs; use std::path::{Path, PathBuf}; -use tempfile::TempDir; +use tempfile::{Builder, TempDir}; pub use common::{binary_path, ensure_substrate_built, substrate_shell_driver, temp_dir}; pub use substrate_common::dedupe_path; @@ -15,8 +16,8 @@ mod socket; pub use socket::{AgentSocket, PendingDiffAckError, PendingDiffAckState, SocketResponse}; mod repl_world_agent; pub use repl_world_agent::{ - PersistentExecRecord, PersistentExecStdoutOverride, PersistentStartSessionRecord, - ReplWorldAgentRecords, ReplWorldAgentStub, StreamBehavior, + MemberDispatchStreamScript, PersistentExecRecord, PersistentExecStdoutOverride, + PersistentStartSessionRecord, ReplWorldAgentRecords, ReplWorldAgentStub, StreamBehavior, }; pub const PAYLOAD_MARKER: &str = "__SUBSTRATE_PAYLOAD__"; @@ -32,7 +33,10 @@ pub struct ShellEnvFixture { impl ShellEnvFixture { pub fn new() -> Self { - let temp = temp_dir("substrate-test-"); + let temp = Builder::new() + .prefix("substrate-test-") + .tempdir_in("/tmp") + .expect("failed to allocate integration test temp dir"); let home = temp.path().join("home"); fs::create_dir_all(home.join(".substrate/shims")) .expect("failed to create shims directory"); @@ -116,3 +120,24 @@ pub fn payload_lines(stdout: &[u8]) -> Vec { ); lines } + +pub fn persist_runtime_alert_for_substrate_home( + substrate_home: &Path, + orchestration_session_id: &str, + item_id: &str, + message: Option, +) { + let previous_substrate_home = env::var_os("SUBSTRATE_HOME"); + env::set_var("SUBSTRATE_HOME", substrate_home); + let result = + substrate_shell::execution::agent_dev_support::persist_runtime_alert_for_dev_support( + orchestration_session_id, + item_id, + message, + ); + match previous_substrate_home { + Some(previous) => env::set_var("SUBSTRATE_HOME", previous), + None => env::remove_var("SUBSTRATE_HOME"), + } + result.expect("persist runtime alert through authoritative state store"); +} diff --git a/crates/shell/tests/support/repl_world_agent.rs b/crates/shell/tests/support/repl_world_agent.rs index 2acbd566c..9d73bab8b 100644 --- a/crates/shell/tests/support/repl_world_agent.rs +++ b/crates/shell/tests/support/repl_world_agent.rs @@ -5,7 +5,7 @@ use base64::engine::general_purpose::STANDARD as BASE64; use base64::Engine; use futures::{SinkExt, StreamExt}; use serde_json::Value as JsonValue; -use std::collections::HashMap; +use std::collections::{HashMap, VecDeque}; use std::io::{Read, Write}; use std::path::{Path, PathBuf}; use std::sync::{ @@ -20,6 +20,7 @@ pub struct PersistentStartSessionRecord { pub cwd: String, pub env: HashMap, pub policy_snapshot: JsonValue, + pub shared_world: Option, pub world_network: JsonValue, pub cols: u16, pub rows: u16, @@ -51,6 +52,27 @@ pub struct ReplWorldAgentRecords { pub persistent_stdin: Vec>, pub persistent_signals: Vec, pub legacy_pty_starts: Vec, + pub member_dispatch_requests: Vec, + pub member_turn_submit_requests: Vec, + pub execute_cancel_requests: Vec, +} + +#[derive(Debug, Clone)] +pub enum MemberDispatchStreamScript { + ReadyAndExit { + session_handle_id: String, + exit_code: i32, + }, + ReadyAndHoldUntilCancel { + session_handle_id: String, + exit_code_on_cancel: i32, + }, + ExitWithoutReady { + exit_code: i32, + }, + ErrorBeforeReady { + message: String, + }, } #[derive(Clone, Copy, Debug, PartialEq, Eq)] @@ -81,12 +103,20 @@ pub struct ReplWorldAgentStub { connections: Arc, records: Arc>, persistent_exec_stdout_override: Option, + member_dispatch_scripts: Arc>>, handle: Option>, } +fn assert_member_dispatch_capture(dispatch: &agent_api_types::MemberDispatchRequestV1) { + assert!( + Path::new(&dispatch.resolved_runtime.binary_path).is_absolute(), + "captured member dispatch binary_path must remain absolute" + ); +} + impl ReplWorldAgentStub { pub fn start(path: &Path, behavior: StreamBehavior) -> Self { - Self::start_with_overrides(path, behavior, None, None) + Self::start_with_overrides(path, behavior, None, None, Vec::new()) } pub fn start_with_persistent_exec_stdout_override( @@ -105,6 +135,7 @@ impl ReplWorldAgentStub { out_of_band_after_complete: None, }), None, + Vec::new(), ) } @@ -112,7 +143,7 @@ impl ReplWorldAgentStub { path: &Path, script: PersistentExecStdoutOverride, ) -> Self { - Self::start_with_overrides(path, StreamBehavior::Normal, Some(script), None) + Self::start_with_overrides(path, StreamBehavior::Normal, Some(script), None, Vec::new()) } pub fn start_with_first_ready_cwd_override( @@ -120,7 +151,15 @@ impl ReplWorldAgentStub { behavior: StreamBehavior, ready_cwd: impl Into, ) -> Self { - Self::start_with_overrides(path, behavior, None, Some(ready_cwd.into())) + Self::start_with_overrides(path, behavior, None, Some(ready_cwd.into()), Vec::new()) + } + + pub fn start_with_member_dispatch_scripts( + path: &Path, + behavior: StreamBehavior, + scripts: Vec, + ) -> Self { + Self::start_with_overrides(path, behavior, None, None, scripts) } fn start_with_overrides( @@ -128,6 +167,7 @@ impl ReplWorldAgentStub { behavior: StreamBehavior, persistent_exec_stdout_override: Option, first_ready_cwd_override: Option, + member_dispatch_scripts: Vec, ) -> Self { let _ = std::fs::remove_file(path); if let Some(parent) = path.parent() { @@ -143,7 +183,9 @@ impl ReplWorldAgentStub { let records = Arc::new(Mutex::new(ReplWorldAgentRecords::default())); let records_for_thread = records.clone(); let persistent_exec_stdout_override_for_thread = persistent_exec_stdout_override.clone(); - let mut first_ready_cwd_override = first_ready_cwd_override; + let member_dispatch_scripts = Arc::new(Mutex::new(VecDeque::from(member_dispatch_scripts))); + let member_dispatch_scripts_for_thread = member_dispatch_scripts.clone(); + let first_ready_cwd_override = Arc::new(Mutex::new(first_ready_cwd_override)); let handle = thread::spawn(move || { let rt = tokio::runtime::Builder::new_current_thread() @@ -159,13 +201,6 @@ impl ReplWorldAgentStub { use tokio_tungstenite as tungs; use tungs::tungstenite::protocol::Message; - #[derive(serde::Deserialize)] - struct ExecuteRequestStub { - cmd: String, - cwd: Option, - env: Option>, - } - async fn read_http_request( stream: &mut tokio::net::UnixStream, ) -> Option<(String, Vec)> { @@ -282,8 +317,74 @@ impl ReplWorldAgentStub { let _ = stream.shutdown().await; } + async fn write_http_stream_start(stream: &mut tokio::net::UnixStream) { + let resp = "HTTP/1.1 200 OK\r\nContent-Type: application/x-ndjson\r\nTransfer-Encoding: chunked\r\nConnection: close\r\n\r\n"; + let _ = stream.write_all(resp.as_bytes()).await; + let _ = stream.flush().await; + } + + async fn write_chunked_frame( + stream: &mut tokio::net::UnixStream, + frame: &agent_api_types::ExecuteStreamFrame, + ) { + let mut payload = serde_json::to_vec(frame).expect("serialize stream frame"); + payload.push(b'\n'); + let header = format!("{:X}\r\n", payload.len()); + let _ = stream.write_all(header.as_bytes()).await; + let _ = stream.write_all(&payload).await; + let _ = stream.write_all(b"\r\n").await; + let _ = stream.flush().await; + } + + async fn finish_chunked_stream(stream: &mut tokio::net::UnixStream) { + let _ = stream.write_all(b"0\r\n\r\n").await; + let _ = stream.flush().await; + let _ = stream.shutdown().await; + } + + fn build_member_dispatch_ready_event( + request: &agent_api_types::ExecuteRequest, + dispatch: &agent_api_types::MemberDispatchRequestV1, + span_id: &str, + session_handle_id: &str, + ) -> agent_api_types::ExecuteStreamFrame { + agent_api_types::ExecuteStreamFrame::Event { + event: substrate_common::agent_events::AgentEvent { + ts: chrono::Utc::now(), + agent_id: request.agent_id.clone(), + kind: substrate_common::agent_events::AgentEventKind::Registered, + orchestration_session_id: dispatch.orchestration_session_id.clone(), + run_id: dispatch.run_id.clone(), + parent_run_id: None, + participant_id: Some(dispatch.participant_id.clone()), + parent_participant_id: dispatch.parent_participant_id.clone(), + resumed_from_participant_id: dispatch.resumed_from_participant_id.clone(), + backend_id: Some(dispatch.backend_id.clone()), + thread_id: None, + role: Some("member".to_string()), + world_id: Some(dispatch.world_id.clone()), + world_generation: Some(dispatch.world_generation), + cmd_id: None, + span_id: Some(span_id.to_string()), + channel: None, + identity_tuple: None, + placement_posture: None, + project: None, + data: serde_json::json!({ + "schema": "agent_api.session.handle.v1", + "session": { + "id": session_handle_id, + } + }), + }, + } + } + let listener = UnixListener::bind(&path_buf).expect("bind stub socket"); - let mut next_world_id: u64 = 1; + let next_world_id = Arc::new(Mutex::new(1u64)); + let active_member_dispatch_cancels: Arc< + Mutex>>, + > = Arc::new(Mutex::new(HashMap::new())); while !shutdown_for_thread.load(Ordering::SeqCst) { let accept = tokio::time::timeout( @@ -310,7 +411,8 @@ impl ReplWorldAgentStub { } if first_line.starts_with("POST /v1/execute/stream ") { - let parsed: ExecuteRequestStub = match serde_json::from_slice(&body) { + let parsed: agent_api_types::ExecuteRequest = + match serde_json::from_slice(&body) { Ok(p) => p, Err(_) => { write_http_json(&mut stream, "400 Bad Request", r#"{"error":"bad_request","message":"invalid json"}"#).await; @@ -318,6 +420,142 @@ impl ReplWorldAgentStub { } }; + if let Some(dispatch) = parsed.member_dispatch.clone() { + assert_member_dispatch_capture(&dispatch); + if let Ok(mut guard) = records_for_thread.lock() { + guard.member_dispatch_requests.push(parsed.clone()); + } + + let span_id = format!("member-span-{}", dispatch.participant_id); + let script = member_dispatch_scripts_for_thread + .lock() + .ok() + .and_then(|mut guard| guard.pop_front()) + .unwrap_or(MemberDispatchStreamScript::ReadyAndExit { + session_handle_id: format!("session-{}", dispatch.participant_id), + exit_code: 0, + }); + + let shutdown_for_member_dispatch = shutdown_for_thread.clone(); + let active_member_dispatch_cancels = + active_member_dispatch_cancels.clone(); + tokio::spawn(async move { + write_http_stream_start(&mut stream).await; + write_chunked_frame( + &mut stream, + &agent_api_types::ExecuteStreamFrame::Start { + span_id: span_id.clone(), + }, + ) + .await; + + match script { + MemberDispatchStreamScript::ReadyAndExit { + session_handle_id, + exit_code, + } => { + write_chunked_frame( + &mut stream, + &build_member_dispatch_ready_event( + &parsed, + &dispatch, + &span_id, + &session_handle_id, + ), + ) + .await; + write_chunked_frame( + &mut stream, + &agent_api_types::ExecuteStreamFrame::Exit { + exit: exit_code, + span_id: span_id.clone(), + scopes_used: Vec::new(), + fs_diff: None, + process_telemetry: + agent_api_types::ProcessTelemetry::default(), + }, + ) + .await; + finish_chunked_stream(&mut stream).await; + } + MemberDispatchStreamScript::ReadyAndHoldUntilCancel { + session_handle_id, + exit_code_on_cancel, + } => { + let cancelled = Arc::new(AtomicBool::new(false)); + if let Ok(mut guard) = + active_member_dispatch_cancels.lock() + { + guard.insert(span_id.clone(), cancelled.clone()); + } + write_chunked_frame( + &mut stream, + &build_member_dispatch_ready_event( + &parsed, + &dispatch, + &span_id, + &session_handle_id, + ), + ) + .await; + while !shutdown_for_member_dispatch + .load(Ordering::SeqCst) + && !cancelled.load(Ordering::SeqCst) + { + tokio::time::sleep( + std::time::Duration::from_millis(25), + ) + .await; + } + if let Ok(mut guard) = + active_member_dispatch_cancels.lock() + { + guard.remove(&span_id); + } + write_chunked_frame( + &mut stream, + &agent_api_types::ExecuteStreamFrame::Exit { + exit: exit_code_on_cancel, + span_id: span_id.clone(), + scopes_used: Vec::new(), + fs_diff: None, + process_telemetry: + agent_api_types::ProcessTelemetry::default(), + }, + ) + .await; + finish_chunked_stream(&mut stream).await; + } + MemberDispatchStreamScript::ExitWithoutReady { exit_code } => { + write_chunked_frame( + &mut stream, + &agent_api_types::ExecuteStreamFrame::Exit { + exit: exit_code, + span_id: span_id.clone(), + scopes_used: Vec::new(), + fs_diff: None, + process_telemetry: + agent_api_types::ProcessTelemetry::default(), + }, + ) + .await; + finish_chunked_stream(&mut stream).await; + } + MemberDispatchStreamScript::ErrorBeforeReady { message } => { + write_chunked_frame( + &mut stream, + &agent_api_types::ExecuteStreamFrame::Error { + message, + }, + ) + .await; + finish_chunked_stream(&mut stream).await; + } + } + }); + continue; + } + let mut cmd = std::process::Command::new("bash"); cmd.arg("-lc").arg(&parsed.cmd); if let Some(cwd) = &parsed.cwd { @@ -376,8 +614,84 @@ impl ReplWorldAgentStub { continue; } + if first_line.starts_with("POST /v1/execute/cancel ") { + let parsed: agent_api_types::ExecuteCancelRequestV1 = + match serde_json::from_slice(&body) { + Ok(p) => p, + Err(_) => { + write_http_json(&mut stream, "400 Bad Request", r#"{"error":"bad_request","message":"invalid json"}"#).await; + continue; + } + }; + if let Ok(mut guard) = records_for_thread.lock() { + guard.execute_cancel_requests.push(parsed.clone()); + } + let delivered = active_member_dispatch_cancels + .lock() + .ok() + .and_then(|guard| guard.get(&parsed.span_id).cloned()) + .map(|flag| { + flag.store(true, Ordering::SeqCst); + true + }) + .unwrap_or(false); + let response = serde_json::json!({ + "schema_version": 1, + "delivered": delivered, + }) + .to_string(); + write_http_json(&mut stream, "200 OK", &response).await; + continue; + } + + if first_line.starts_with("POST /v1/member_turn/stream ") { + let parsed: agent_api_types::MemberTurnSubmitRequestV1 = + match serde_json::from_slice(&body) { + Ok(p) => p, + Err(_) => { + write_http_json(&mut stream, "400 Bad Request", r#"{"error":"bad_request","message":"invalid json"}"#).await; + continue; + } + }; + if let Ok(mut guard) = records_for_thread.lock() { + guard.member_turn_submit_requests.push(parsed.clone()); + } + + let span_id = format!("member-turn-span-{}", parsed.participant_id); + let stdout = format!("__MEMBER_TURN_SUBMIT_STUB__ {}\n", parsed.prompt); + write_http_stream_start(&mut stream).await; + write_chunked_frame( + &mut stream, + &agent_api_types::ExecuteStreamFrame::Start { + span_id: span_id.clone(), + }, + ) + .await; + write_chunked_frame( + &mut stream, + &agent_api_types::ExecuteStreamFrame::Stdout { + chunk_b64: BASE64.encode(stdout.as_bytes()), + }, + ) + .await; + write_chunked_frame( + &mut stream, + &agent_api_types::ExecuteStreamFrame::Exit { + exit: 0, + span_id, + scopes_used: Vec::new(), + fs_diff: None, + process_telemetry: agent_api_types::ProcessTelemetry::default(), + }, + ) + .await; + finish_chunked_stream(&mut stream).await; + continue; + } + if first_line.starts_with("POST /v1/execute ") { - let parsed: ExecuteRequestStub = match serde_json::from_slice(&body) { + let parsed: agent_api_types::ExecuteRequest = + match serde_json::from_slice(&body) { Ok(p) => p, Err(_) => { write_http_json(&mut stream, "400 Bad Request", r#"{"error":"bad_request","message":"invalid json"}"#).await; @@ -394,6 +708,21 @@ impl ReplWorldAgentStub { cmd.envs(env); } let output = cmd.output().expect("run host command"); + let shared_world = parsed.shared_world.as_ref().map(|request| { + let generation = match request.action { + agent_api_types::SharedWorldOwnerAction::AttachOrCreate => 0, + agent_api_types::SharedWorldOwnerAction::ReplaceExpectedGeneration { + expected_generation, + .. + } => expected_generation.saturating_add(1), + }; + serde_json::json!({ + "orchestration_session_id": request.orchestration_session_id, + "world_id": "wld_execute_stub", + "world_generation": generation, + "binding_state": "active", + }) + }); let resp = serde_json::json!({ "exit": output.status.code().unwrap_or(-1), "span_id": "agent-span", @@ -401,6 +730,7 @@ impl ReplWorldAgentStub { "stderr_b64": BASE64.encode(&output.stderr), "scopes_used": [], "fs_diff": serde_json::Value::Null, + "shared_world": shared_world, "process_events": [], "process_events_status": "unavailable", "process_events_reason": "backend_disabled", @@ -415,38 +745,148 @@ impl ReplWorldAgentStub { continue; } - // The request bytes were already read. Replay them into the websocket acceptor. - let raw = header.into_bytes(); - let replay = ReplayStream { - prefix: std::io::Cursor::new(raw), - inner: stream, - }; + let records_for_ws = records_for_thread.clone(); + let persistent_exec_stdout_override_for_ws = + persistent_exec_stdout_override_for_thread.clone(); + let first_ready_cwd_override = first_ready_cwd_override.clone(); + let next_world_id = next_world_id.clone(); + tokio::spawn(async move { + // The request bytes were already read. Replay them into the websocket acceptor. + let raw = header.into_bytes(); + let replay = ReplayStream { + prefix: std::io::Cursor::new(raw), + inner: stream, + }; - let ws = match tungs::accept_async(replay).await { - Ok(ws) => ws, - Err(_) => continue, - }; - let (mut sink, mut ws_stream) = ws.split(); + let ws = match tungs::accept_async(replay).await { + Ok(ws) => ws, + Err(_) => return, + }; + let (mut sink, mut ws_stream) = ws.split(); - let first = ws_stream.next().await; - let Some(Ok(Message::Text(first_text))) = first else { - continue; - }; + let first = ws_stream.next().await; + let Some(Ok(Message::Text(first_text))) = first else { + return; + }; - let Ok(first_json) = serde_json::from_str::(&first_text) else { - let _ = sink.send(Message::Close(None)).await; - continue; - }; + let Ok(first_json) = serde_json::from_str::(&first_text) else { + let _ = sink.send(Message::Close(None)).await; + return; + }; - let ty = first_json.get("type").and_then(|v| v.as_str()).unwrap_or(""); + let ty = first_json.get("type").and_then(|v| v.as_str()).unwrap_or(""); - if ty == "start" { - // Legacy per-command PTY protocol. - let cmd = first_json - .get("cmd") - .and_then(|v| v.as_str()) - .unwrap_or("") + if ty == "start" { + let cmd = first_json + .get("cmd") + .and_then(|v| v.as_str()) + .unwrap_or("") + .to_string(); + let cwd = first_json + .get("cwd") + .and_then(|v| v.as_str()) + .unwrap_or("") + .to_string(); + let env = first_json + .get("env") + .and_then(|v| v.as_object()) + .map(|m| { + m.iter() + .filter_map(|(k, v)| { + Some((k.clone(), v.as_str()?.to_string())) + }) + .collect::>() + }) + .unwrap_or_default(); + let env = env + .into_iter() + .map(|(k, v)| { + let keep = matches!( + k.as_str(), + "SUBSTRATE_ANCHOR_MODE" + | "SUBSTRATE_ANCHOR_PATH" + | "SUBSTRATE_CAGED" + | "SUBSTRATE_WORLD_DEPS_GUEST_BIN_DIR" + | "PATH" + | "HOME" + | "TERM" + | "XDG_CACHE_HOME" + | "XDG_CONFIG_HOME" + | "XDG_DATA_HOME" + ); + if keep { + (k, v) + } else { + (k, "".to_string()) + } + }) + .collect::>(); + let span_id = first_json + .get("span_id") + .and_then(|v| v.as_str()) + .unwrap_or("") + .to_string(); + let cols = + first_json.get("cols").and_then(|v| v.as_u64()).unwrap_or(80) + as u16; + let rows = + first_json.get("rows").and_then(|v| v.as_u64()).unwrap_or(24) + as u16; + + if let Ok(mut guard) = records_for_ws.lock() { + guard.legacy_pty_starts.push(LegacyPtyStartRecord { + cmd: cmd.clone(), + cwd, + env, + span_id, + cols, + rows, + }); + } + + let stdout = format!("__LEGACY_PTY_STUB__ {cmd}\n"); + let out = serde_json::json!({ + "type": "stdout", + "data_b64": BASE64.encode(stdout.as_bytes()), + }) .to_string(); + let _ = sink.send(Message::Text(out)).await; + + let exit = serde_json::json!({ + "type": "exit", + "code": 0, + "world_fs_strategy_primary": "overlay", + "world_fs_strategy_final": "overlay", + "world_fs_strategy_fallback_reason": "none", + }) + .to_string(); + let _ = sink.send(Message::Text(exit)).await; + let _ = sink.send(Message::Close(None)).await; + return; + } + + if ty != "start_session" { + let _ = sink.send(Message::Close(None)).await; + return; + } + + if behavior == StreamBehavior::CloseBeforeReady { + let _ = sink.send(Message::Close(None)).await; + return; + } + if behavior == StreamBehavior::FatalBeforeReady { + let err = serde_json::json!({ + "type": "error", + "code": "simulated_start_failure", + "message": "simulated persistent start failure", + "fatal": true, + }) + .to_string(); + let _ = sink.send(Message::Text(err)).await; + let _ = sink.send(Message::Close(None)).await; + return; + } + let cwd = first_json .get("cwd") .and_then(|v| v.as_str()) @@ -461,8 +901,6 @@ impl ReplWorldAgentStub { .collect::>() }) .unwrap_or_default(); - // Avoid leaking host secrets into test failure output: keep a small allowlist - // of routing/env keys, redact everything else. let env = env .into_iter() .map(|(k, v)| { @@ -486,353 +924,289 @@ impl ReplWorldAgentStub { } }) .collect::>(); - let span_id = first_json - .get("span_id") - .and_then(|v| v.as_str()) - .unwrap_or("") - .to_string(); - let cols = first_json.get("cols").and_then(|v| v.as_u64()).unwrap_or(80) as u16; - let rows = first_json.get("rows").and_then(|v| v.as_u64()).unwrap_or(24) as u16; - - if let Ok(mut guard) = records_for_thread.lock() { - guard.legacy_pty_starts.push(LegacyPtyStartRecord { - cmd: cmd.clone(), - cwd, + let policy_snapshot = first_json + .get("policy_snapshot") + .cloned() + .unwrap_or(JsonValue::Null); + let shared_world = first_json + .get("shared_world") + .cloned() + .and_then(|value| serde_json::from_value(value).ok()); + let world_network = first_json + .get("world_network") + .cloned() + .unwrap_or(JsonValue::Null); + let cols = + first_json.get("cols").and_then(|v| v.as_u64()).unwrap_or(80) as u16; + let rows = + first_json.get("rows").and_then(|v| v.as_u64()).unwrap_or(24) as u16; + + if let Ok(mut guard) = records_for_ws.lock() { + guard.persistent_start_sessions.push(PersistentStartSessionRecord { + cwd: cwd.clone(), env, - span_id, + policy_snapshot, + shared_world: shared_world.clone(), + world_network, cols, rows, }); } - // Minimal happy-path: produce one stdout frame and then exit. - let stdout = format!("__LEGACY_PTY_STUB__ {cmd}\n"); - let out = serde_json::json!({ - "type": "stdout", - "data_b64": BASE64.encode(stdout.as_bytes()), - }) - .to_string(); - let _ = sink.send(Message::Text(out)).await; - - let exit = serde_json::json!({ - "type": "exit", - "code": 0, - "world_fs_strategy_primary": "overlay", - "world_fs_strategy_final": "overlay", - "world_fs_strategy_fallback_reason": "none", - }) - .to_string(); - let _ = sink.send(Message::Text(exit)).await; - let _ = sink.send(Message::Close(None)).await; - continue; - } - - if ty != "start_session" { - let _ = sink.send(Message::Close(None)).await; - continue; - } + let mut session_cwd = if let Some(override_cwd) = first_ready_cwd_override + .lock() + .expect("first ready cwd override mutex poisoned") + .take() + { + override_cwd + } else if cwd.trim().is_empty() { + "/".to_string() + } else { + cwd + }; - if behavior == StreamBehavior::CloseBeforeReady { - let _ = sink.send(Message::Close(None)).await; - continue; - } - if behavior == StreamBehavior::FatalBeforeReady { - let err = serde_json::json!({ - "type": "error", - "code": "simulated_start_failure", - "message": "simulated persistent start failure", - "fatal": true, + let world_id = { + let mut guard = next_world_id + .lock() + .expect("next world id mutex poisoned"); + let world_id = format!("wld_stub_{:04}", *guard); + *guard = guard.saturating_add(1); + world_id + }; + let ready_world_id = world_id.clone(); + let ready = serde_json::json!({ + "type": "ready", + "session_nonce": "0123456789abcdef0123456789abcdef", + "world_id": world_id, + "cwd": session_cwd, + "protocol_version": 1, + "shared_world": shared_world.as_ref().map(|request| { + let world_generation = match request.action { + agent_api_types::SharedWorldOwnerAction::AttachOrCreate => 0, + agent_api_types::SharedWorldOwnerAction::ReplaceExpectedGeneration { + expected_generation, + .. + } => expected_generation.saturating_add(1), + }; + serde_json::json!({ + "orchestration_session_id": request.orchestration_session_id, + "world_id": ready_world_id, + "world_generation": world_generation, + "binding_state": "active", + }) + }), }) .to_string(); - let _ = sink.send(Message::Text(err)).await; - let _ = sink.send(Message::Close(None)).await; - continue; - } - - // Persistent session protocol v1. - let cwd = first_json - .get("cwd") - .and_then(|v| v.as_str()) - .unwrap_or("") - .to_string(); - let env = first_json - .get("env") - .and_then(|v| v.as_object()) - .map(|m| { - m.iter() - .filter_map(|(k, v)| Some((k.clone(), v.as_str()?.to_string()))) - .collect::>() - }) - .unwrap_or_default(); - // Avoid leaking host secrets into test failure output: keep a small allowlist - // of REPL routing/env keys, redact everything else. - let env = env - .into_iter() - .map(|(k, v)| { - let keep = matches!( - k.as_str(), - "SUBSTRATE_ANCHOR_MODE" - | "SUBSTRATE_ANCHOR_PATH" - | "SUBSTRATE_CAGED" - | "SUBSTRATE_WORLD_DEPS_GUEST_BIN_DIR" - | "PATH" - | "HOME" - | "TERM" - | "XDG_CACHE_HOME" - | "XDG_CONFIG_HOME" - | "XDG_DATA_HOME" - ); - if keep { - (k, v) - } else { - (k, "".to_string()) - } - }) - .collect::>(); - let policy_snapshot = first_json - .get("policy_snapshot") - .cloned() - .unwrap_or(JsonValue::Null); - let world_network = first_json - .get("world_network") - .cloned() - .unwrap_or(JsonValue::Null); - let cols = first_json.get("cols").and_then(|v| v.as_u64()).unwrap_or(80) as u16; - let rows = first_json.get("rows").and_then(|v| v.as_u64()).unwrap_or(24) as u16; - - if let Ok(mut guard) = records_for_thread.lock() { - guard.persistent_start_sessions.push(PersistentStartSessionRecord { - cwd: cwd.clone(), - env, - policy_snapshot, - world_network, - cols, - rows, - }); - } - - let mut session_cwd = if let Some(override_cwd) = first_ready_cwd_override.take() - { - override_cwd - } else if cwd.trim().is_empty() { - "/".to_string() - } else { - cwd - }; + let _ = sink.send(Message::Text(ready)).await; - // Respond with a deterministic ready. - let world_id = format!("wld_stub_{next_world_id:04}"); - next_world_id = next_world_id.saturating_add(1); - let ready = serde_json::json!({ - "type": "ready", - "session_nonce": "0123456789abcdef0123456789abcdef", - "world_id": world_id, - "cwd": session_cwd, - "protocol_version": 1, - }) - .to_string(); - let _ = sink.send(Message::Text(ready)).await; - - while let Some(next) = ws_stream.next().await { - let Ok(msg) = next else { - break; - }; - let Message::Text(text) = msg else { - if msg.is_close() { + while let Some(next) = ws_stream.next().await { + let Ok(msg) = next else { break; - } - continue; - }; + }; + let Message::Text(text) = msg else { + if msg.is_close() { + break; + } + continue; + }; - let Ok(frame) = serde_json::from_str::(&text) else { - break; - }; - let fty = frame.get("type").and_then(|v| v.as_str()).unwrap_or(""); - - match fty { - "exec" => { - let seq = frame.get("seq").and_then(|v| v.as_u64()).unwrap_or(0); - let token_hex = frame - .get("token_hex") - .and_then(|v| v.as_str()) - .unwrap_or("") - .to_string(); - let cmd_id = frame - .get("cmd_id") - .and_then(|v| v.as_str()) - .unwrap_or("") - .to_string(); - let stdin_mode = frame - .get("stdin_mode") - .and_then(|v| v.as_str()) - .unwrap_or("") - .to_string(); - let program_b64 = frame - .get("program_b64") - .and_then(|v| v.as_str()) - .unwrap_or(""); - let program_utf8 = BASE64 - .decode(program_b64) - .ok() - .and_then(|b| String::from_utf8(b).ok()) - .unwrap_or_default(); - - // Minimal cwd tracking: support basic `cd ` commands so REPL - // drift restarts can be tested against real host paths. - let trimmed = program_utf8.trim(); - if let Some(rest) = trimmed.strip_prefix("cd ") { - let arg = rest.trim(); - if !arg.is_empty() { - let next = if arg.starts_with('/') { - PathBuf::from(arg) - } else { - PathBuf::from(&session_cwd).join(arg) - }; - let mut normalized = PathBuf::new(); - for comp in next.components() { - match comp { - std::path::Component::RootDir => normalized.push("/"), - std::path::Component::CurDir => {} - std::path::Component::ParentDir => { - normalized.pop(); - } - std::path::Component::Normal(seg) => { - normalized.push(seg); + let Ok(frame) = serde_json::from_str::(&text) else { + break; + }; + let fty = frame.get("type").and_then(|v| v.as_str()).unwrap_or(""); + + match fty { + "exec" => { + let seq = + frame.get("seq").and_then(|v| v.as_u64()).unwrap_or(0); + let token_hex = frame + .get("token_hex") + .and_then(|v| v.as_str()) + .unwrap_or("") + .to_string(); + let cmd_id = frame + .get("cmd_id") + .and_then(|v| v.as_str()) + .unwrap_or("") + .to_string(); + let stdin_mode = frame + .get("stdin_mode") + .and_then(|v| v.as_str()) + .unwrap_or("") + .to_string(); + let program_b64 = frame + .get("program_b64") + .and_then(|v| v.as_str()) + .unwrap_or(""); + let program_utf8 = BASE64 + .decode(program_b64) + .ok() + .and_then(|b| String::from_utf8(b).ok()) + .unwrap_or_default(); + + let trimmed = program_utf8.trim(); + if let Some(rest) = trimmed.strip_prefix("cd ") { + let arg = rest.trim(); + if !arg.is_empty() { + let next = if arg.starts_with('/') { + PathBuf::from(arg) + } else { + PathBuf::from(&session_cwd).join(arg) + }; + let mut normalized = PathBuf::new(); + for comp in next.components() { + match comp { + std::path::Component::RootDir => { + normalized.push("/") + } + std::path::Component::CurDir => {} + std::path::Component::ParentDir => { + normalized.pop(); + } + std::path::Component::Normal(seg) => { + normalized.push(seg); + } + std::path::Component::Prefix(_) => {} } - std::path::Component::Prefix(_) => {} } + if normalized.as_os_str().is_empty() { + normalized.push("/"); + } + session_cwd = + normalized.to_string_lossy().to_string(); } - if normalized.as_os_str().is_empty() { - normalized.push("/"); - } - session_cwd = normalized.to_string_lossy().to_string(); } - } - if let Ok(mut guard) = records_for_thread.lock() { - guard.persistent_execs.push(PersistentExecRecord { - seq, - token_hex: token_hex.clone(), - cmd_id, - stdin_mode: stdin_mode.clone(), - program_utf8: program_utf8.clone(), - }); - } + if let Ok(mut guard) = records_for_ws.lock() { + guard.persistent_execs.push(PersistentExecRecord { + seq, + token_hex: token_hex.clone(), + cmd_id, + stdin_mode: stdin_mode.clone(), + program_utf8: program_utf8.clone(), + }); + } - // Minimal stdout for observability (optionally overridden for tests). - let mut stdout_bytes = None; - if let Some(ov) = &persistent_exec_stdout_override_for_thread { - if !ov.marker.is_empty() && program_utf8.contains(&ov.marker) { - stdout_bytes = Some(ov.bytes.clone()); + let mut stdout_bytes = None; + if let Some(ov) = &persistent_exec_stdout_override_for_ws { + if !ov.marker.is_empty() + && program_utf8.contains(&ov.marker) + { + stdout_bytes = Some(ov.bytes.clone()); + } } - } - let stdout_bytes = stdout_bytes.unwrap_or_else(|| { - format!( - "__PERSISTENT_EXEC_STUB__ {stdin_mode} {program_utf8}\n" - ) - .into_bytes() - }); + let stdout_bytes = stdout_bytes.unwrap_or_else(|| { + format!( + "__PERSISTENT_EXEC_STUB__ {stdin_mode} {program_utf8}\n" + ) + .into_bytes() + }); - // Allow testing output buffering by splitting the stream. - if stdout_bytes.len() > 3 { - let split_at = stdout_bytes.len() / 2; - for chunk in [&stdout_bytes[..split_at], &stdout_bytes[split_at..]] - { + if stdout_bytes.len() > 3 { + let split_at = stdout_bytes.len() / 2; + for chunk in + [&stdout_bytes[..split_at], &stdout_bytes[split_at..]] + { + let stdout = serde_json::json!({ + "type": "stdout", + "data_b64": BASE64.encode(chunk), + }) + .to_string(); + let _ = sink.send(Message::Text(stdout)).await; + } + } else { let stdout = serde_json::json!({ "type": "stdout", - "data_b64": BASE64.encode(chunk), + "data_b64": BASE64.encode(&stdout_bytes), }) .to_string(); let _ = sink.send(Message::Text(stdout)).await; } - } else { - let stdout = serde_json::json!({ - "type": "stdout", - "data_b64": BASE64.encode(&stdout_bytes), + + if let Some(ov) = &persistent_exec_stdout_override_for_ws { + if !ov.marker.is_empty() + && program_utf8.contains(&ov.marker) + { + if let (Some(delay_ms), Some(suffix)) = + (ov.delay_before_suffix_ms, ov.suffix_bytes.as_ref()) + { + tokio::time::sleep( + std::time::Duration::from_millis(delay_ms), + ) + .await; + let stdout = serde_json::json!({ + "type": "stdout", + "data_b64": BASE64.encode(suffix), + }) + .to_string(); + let _ = sink.send(Message::Text(stdout)).await; + } + } + } + + let complete = serde_json::json!({ + "type": "command_complete", + "seq": seq, + "token_hex": token_hex, + "exit": 0, + "cwd": session_cwd, }) .to_string(); - let _ = sink.send(Message::Text(stdout)).await; - } + let _ = sink.send(Message::Text(complete)).await; - if let Some(ov) = &persistent_exec_stdout_override_for_thread { - if !ov.marker.is_empty() && program_utf8.contains(&ov.marker) { - if let (Some(delay_ms), Some(suffix)) = - (ov.delay_before_suffix_ms, ov.suffix_bytes.as_ref()) + if let Some(ov) = &persistent_exec_stdout_override_for_ws { + if ov.marker.is_empty() + || !program_utf8.contains(&ov.marker) + { + } else if let Some((delay_ms, bytes)) = + ov.out_of_band_after_complete.as_ref() { - tokio::time::sleep(std::time::Duration::from_millis( - delay_ms, - )) + tokio::time::sleep( + std::time::Duration::from_millis(*delay_ms), + ) .await; let stdout = serde_json::json!({ "type": "stdout", - "data_b64": BASE64.encode(suffix), + "data_b64": BASE64.encode(bytes), }) .to_string(); let _ = sink.send(Message::Text(stdout)).await; } } } - - let complete = serde_json::json!({ - "type": "command_complete", - "seq": seq, - "token_hex": token_hex, - "exit": 0, - "cwd": session_cwd, - }) - .to_string(); - let _ = sink.send(Message::Text(complete)).await; - - if let Some(ov) = &persistent_exec_stdout_override_for_thread { - if ov.marker.is_empty() || !program_utf8.contains(&ov.marker) { - // no-op - } else if let Some((delay_ms, bytes)) = - ov.out_of_band_after_complete.as_ref() - { - tokio::time::sleep(std::time::Duration::from_millis( - *delay_ms, - )) - .await; - let stdout = serde_json::json!({ - "type": "stdout", - "data_b64": BASE64.encode(bytes), - }) - .to_string(); - let _ = sink.send(Message::Text(stdout)).await; + "stdin" => { + let data_b64 = frame + .get("data_b64") + .and_then(|v| v.as_str()) + .unwrap_or(""); + if let Ok(bytes) = BASE64.decode(data_b64.as_bytes()) { + if let Ok(mut guard) = records_for_ws.lock() { + guard.persistent_stdin.push(bytes); + } } } - } - "stdin" => { - let data_b64 = frame - .get("data_b64") - .and_then(|v| v.as_str()) - .unwrap_or(""); - if let Ok(bytes) = BASE64.decode(data_b64.as_bytes()) { - if let Ok(mut guard) = records_for_thread.lock() { - guard.persistent_stdin.push(bytes); + "signal" => { + let signal = frame + .get("sig") + .and_then(|v| v.as_str()) + .unwrap_or("") + .to_string(); + if let Ok(mut guard) = records_for_ws.lock() { + guard.persistent_signals.push(signal); } } - } - "signal" => { - let signal = frame - .get("sig") - .and_then(|v| v.as_str()) - .unwrap_or("") - .to_string(); - if let Ok(mut guard) = records_for_thread.lock() { - guard.persistent_signals.push(signal); + "close" => { + let exit = + serde_json::json!({ "type": "exit", "code": 0 }).to_string(); + let _ = sink.send(Message::Text(exit)).await; + let _ = sink.send(Message::Close(None)).await; + break; } + _ => {} } - "close" => { - let exit = serde_json::json!({ "type": "exit", "code": 0 }).to_string(); - let _ = sink.send(Message::Text(exit)).await; - let _ = sink.send(Message::Close(None)).await; - break; - } - _ => {} } - } + }); } }); }); @@ -867,6 +1241,7 @@ impl ReplWorldAgentStub { connections, records, persistent_exec_stdout_override, + member_dispatch_scripts, handle: Some(handle), } } diff --git a/crates/shell/tests/support/socket.rs b/crates/shell/tests/support/socket.rs index c00f755d5..7c0ef4e8f 100644 --- a/crates/shell/tests/support/socket.rs +++ b/crates/shell/tests/support/socket.rs @@ -996,6 +996,20 @@ fn record_execute_request(records: &Arc>>, request: &HttpRe } } +pub fn decode_recorded_execute_requests( + records: &Arc>>, +) -> anyhow::Result> { + let guard = records + .lock() + .map_err(|_| anyhow::anyhow!("recorded execute request mutex poisoned"))?; + guard + .iter() + .cloned() + .map(serde_json::from_value) + .collect::, _>>() + .map_err(|err| anyhow::anyhow!("invalid recorded execute request: {err}")) +} + fn read_http_request(stream: &mut UnixStream) -> std::io::Result { let mut buf = Vec::new(); let mut tmp = [0u8; 4096]; @@ -1103,6 +1117,18 @@ struct ExecuteRequestStub { cmd: String, cwd: Option, env: Option>, + #[serde(default)] + member_dispatch: Option, +} + +fn assert_member_dispatch_capture( + dispatch: &agent_api_types::MemberDispatchRequestV1, +) -> anyhow::Result<()> { + anyhow::ensure!( + Path::new(&dispatch.resolved_runtime.binary_path).is_absolute(), + "captured member dispatch binary_path must remain absolute" + ); + Ok(()) } fn handle_host_execute(request: &HttpRequest, scopes: &[String]) -> anyhow::Result { @@ -1142,6 +1168,15 @@ struct HostCommandOutput { fn run_host_command(request: &ExecuteRequestStub) -> anyhow::Result { use std::process::Command; + if let Some(dispatch) = request.member_dispatch.as_ref() { + assert_member_dispatch_capture(dispatch)?; + return Ok(HostCommandOutput { + exit: 0, + stdout: Vec::new(), + stderr: Vec::new(), + }); + } + let mut cmd = Command::new("bash"); cmd.arg("-c").arg(&request.cmd); if let Some(cwd) = &request.cwd { diff --git a/crates/shell/tests/wfgadax1_routing_fail_closed.rs b/crates/shell/tests/wfgadax1_routing_fail_closed.rs index 985eee4b6..28a2875a3 100644 --- a/crates/shell/tests/wfgadax1_routing_fail_closed.rs +++ b/crates/shell/tests/wfgadax1_routing_fail_closed.rs @@ -89,7 +89,10 @@ fn wfgadax1_fail_closed_routing_true_world_disabled_hard_errors_exit_2() { #[test] fn wfgadax1_runtime_routing_fail_closed_missing_socket_maps_to_exit_3() { - let temp = temp_dir("substrate-wfgadax1-world-missing-"); + let temp = Builder::new() + .prefix("substrate-wfgadax1-world-missing-") + .tempdir_in("/tmp") + .expect("create missing-socket tempdir"); let home = temp.path().join("home"); let project = temp.path().join("project"); fs::create_dir_all(home.join(".substrate")).expect("create SUBSTRATE_HOME"); diff --git a/crates/shell/tests/wfgadax2_caging_required.rs b/crates/shell/tests/wfgadax2_caging_required.rs index 4cb9920e0..fd9bdd768 100644 --- a/crates/shell/tests/wfgadax2_caging_required.rs +++ b/crates/shell/tests/wfgadax2_caging_required.rs @@ -65,7 +65,10 @@ fn base_env_cmd( #[test] fn wfgadax2_control_caged_required_allows_caged_workspace_execution() { - let temp = temp_dir("substrate-wfgadax2-control-"); + let temp = Builder::new() + .prefix("substrate-wfgadax2-control-") + .tempdir_in("/tmp") + .expect("create control tempdir"); let home = temp.path().join("home"); let project = temp.path().join("project"); let substrate_home = home.join(".substrate"); diff --git a/crates/shell/tests/wfgadax3_repl_exit_transparency.rs b/crates/shell/tests/wfgadax3_repl_exit_transparency.rs index c8d1e05a7..4f016e392 100644 --- a/crates/shell/tests/wfgadax3_repl_exit_transparency.rs +++ b/crates/shell/tests/wfgadax3_repl_exit_transparency.rs @@ -349,7 +349,7 @@ fn wfgadax3_prints_exit_note_when_world_cwd_differs_default_target_is_entered_cw let _server = ReplWorldAgentStub::start(&sock, StreamBehavior::Normal); let mut repl = PtyRepl::spawn(&project, &home, &substrate_home, &sock, &[], &["--world"]); - repl.wait_for_output_or_exit("Substrate v", Duration::from_secs(2)) + repl.wait_for_output_or_exit("Substrate v", Duration::from_secs(6)) .expect("banner"); repl.send_line("cd exit-target"); @@ -399,7 +399,7 @@ fn wfgadax3_repl_exit_cwd_last_world_selects_world_cwd_as_exit_target_when_repre let _server = ReplWorldAgentStub::start(&sock, StreamBehavior::Normal); let mut repl = PtyRepl::spawn(&project, &home, &substrate_home, &sock, &[], &["--world"]); - repl.wait_for_output_or_exit("Substrate v", Duration::from_secs(2)) + repl.wait_for_output_or_exit("Substrate v", Duration::from_secs(6)) .expect("banner"); repl.send_line("cd exit-target"); diff --git a/crates/shell/tests/workspace_auto_sync_ws3.rs b/crates/shell/tests/workspace_auto_sync_ws3.rs index 458553e02..7917f6695 100644 --- a/crates/shell/tests/workspace_auto_sync_ws3.rs +++ b/crates/shell/tests/workspace_auto_sync_ws3.rs @@ -6,7 +6,7 @@ use serde_json::json; use std::fs; use std::path::PathBuf; use std::sync::{Arc, Mutex}; -use support::{substrate_shell_driver, temp_dir, AgentSocket, PendingDiffAckState, SocketResponse}; +use support::{substrate_shell_driver, AgentSocket, PendingDiffAckState, SocketResponse}; use tempfile::{Builder, TempDir}; struct WorkspaceAutoSyncFixture { @@ -18,7 +18,10 @@ struct WorkspaceAutoSyncFixture { impl WorkspaceAutoSyncFixture { fn new() -> Self { - let temp = temp_dir("substrate-workspace-auto-sync-ws3-"); + let temp = Builder::new() + .prefix("substrate-workspace-auto-sync-ws3-") + .tempdir_in("/tmp") + .expect("failed to allocate ws3 temp dir"); let home = temp.path().join("home"); fs::create_dir_all(&home).expect("failed to create HOME fixture"); let substrate_home = temp.path().join("substrate-home"); diff --git a/crates/shell/tests/workspace_checkpoint_ws6.rs b/crates/shell/tests/workspace_checkpoint_ws6.rs index 94f290010..f970cdadf 100644 --- a/crates/shell/tests/workspace_checkpoint_ws6.rs +++ b/crates/shell/tests/workspace_checkpoint_ws6.rs @@ -5,8 +5,8 @@ mod support; use std::fs; use std::path::PathBuf; use std::process::Command as StdCommand; -use support::{substrate_shell_driver, temp_dir}; -use tempfile::TempDir; +use support::substrate_shell_driver; +use tempfile::{Builder, TempDir}; struct WorkspaceCheckpointFixture { _temp: TempDir, @@ -17,7 +17,10 @@ struct WorkspaceCheckpointFixture { impl WorkspaceCheckpointFixture { fn new() -> Self { - let temp = temp_dir("substrate-workspace-checkpoint-"); + let temp = Builder::new() + .prefix("substrate-workspace-checkpoint-") + .tempdir_in("/tmp") + .expect("failed to allocate ws6 temp dir"); let home = temp.path().join("home"); fs::create_dir_all(&home).expect("create HOME fixture"); let substrate_home = temp.path().join("substrate-home"); diff --git a/crates/shell/tests/world_overlayfs_enumeration_wo0.rs b/crates/shell/tests/world_overlayfs_enumeration_wo0.rs index 947dfdc4f..a11f54fcb 100644 --- a/crates/shell/tests/world_overlayfs_enumeration_wo0.rs +++ b/crates/shell/tests/world_overlayfs_enumeration_wo0.rs @@ -49,6 +49,7 @@ fn world_doctor_json_includes_strategy_keys_per_adr_0004() { let _socket = AgentSocket::start(&socket_path, SocketResponse::Capabilities); let output = get_substrate_binary() + .current_dir(temp.path()) .env("HOME", &home) .env("USERPROFILE", &home) .env("SUBSTRATE_HOME", home.join(".substrate")) @@ -155,6 +156,7 @@ fn trace_command_complete_includes_world_fs_strategy_fields_and_enums() { fs::write(&trace_path, "").expect("seed trace log"); get_substrate_binary() + .current_dir(temp.path()) .env("HOME", &home) .env("USERPROFILE", &home) .env("SUBSTRATE_HOME", home.join(".substrate")) diff --git a/crates/shell/tests/world_process_exec_tracing_parity_wpep1.rs b/crates/shell/tests/world_process_exec_tracing_parity_wpep1.rs index d0ee15639..508f962f2 100644 --- a/crates/shell/tests/world_process_exec_tracing_parity_wpep1.rs +++ b/crates/shell/tests/world_process_exec_tracing_parity_wpep1.rs @@ -65,6 +65,7 @@ fn world_command_complete_defaults_process_event_diagnostics_when_capture_unavai get_substrate_binary() .env("HOME", &home) .env("USERPROFILE", &home) + .current_dir(temp.path()) .env("SUBSTRATE_HOME", home.join(".substrate")) .env("SHIM_TRACE_LOG", &trace_path) .env("SUBSTRATE_OVERRIDE_WORLD", "enabled") @@ -155,6 +156,7 @@ fn world_process_events_append_before_command_complete_summary() { get_substrate_binary() .env("HOME", &home) .env("USERPROFILE", &home) + .current_dir(temp.path()) .env("SUBSTRATE_HOME", home.join(".substrate")) .env("SHIM_TRACE_LOG", &trace_path) .env("SUBSTRATE_OVERRIDE_WORLD", "enabled") diff --git a/crates/shell/tests/world_process_exec_tracing_parity_wpep2.rs b/crates/shell/tests/world_process_exec_tracing_parity_wpep2.rs index 5cf40d174..8a3c285e6 100644 --- a/crates/shell/tests/world_process_exec_tracing_parity_wpep2.rs +++ b/crates/shell/tests/world_process_exec_tracing_parity_wpep2.rs @@ -69,6 +69,7 @@ fn world_command_complete_preserves_ptrace_not_permitted_diagnostics() { get_substrate_binary() .env("HOME", &home) .env("USERPROFILE", &home) + .current_dir(temp.path()) .env("SUBSTRATE_HOME", home.join(".substrate")) .env("SHIM_TRACE_LOG", &trace_path) .env("SUBSTRATE_OVERRIDE_WORLD", "enabled") @@ -159,6 +160,7 @@ fn world_process_events_preserve_linux_capture_fields_and_explicit_argv_omission get_substrate_binary() .env("HOME", &home) .env("USERPROFILE", &home) + .current_dir(temp.path()) .env("SUBSTRATE_HOME", home.join(".substrate")) .env("SHIM_TRACE_LOG", &trace_path) .env("SUBSTRATE_OVERRIDE_WORLD", "enabled") diff --git a/crates/shell/tests/world_process_exec_tracing_parity_wpep3.rs b/crates/shell/tests/world_process_exec_tracing_parity_wpep3.rs index 1f50ab371..aca73b74d 100644 --- a/crates/shell/tests/world_process_exec_tracing_parity_wpep3.rs +++ b/crates/shell/tests/world_process_exec_tracing_parity_wpep3.rs @@ -100,6 +100,7 @@ fn world_process_events_preserve_linux_capture_fields_and_redacted_argv() { get_substrate_binary() .env("HOME", &home) .env("USERPROFILE", &home) + .current_dir(temp.path()) .env("SUBSTRATE_HOME", home.join(".substrate")) .env("SHIM_TRACE_LOG", &trace_path) .env("SUBSTRATE_OVERRIDE_WORLD", "enabled") diff --git a/crates/shim/src/exec/logging.rs b/crates/shim/src/exec/logging.rs index f0fea1e5d..bd27d7747 100644 --- a/crates/shim/src/exec/logging.rs +++ b/crates/shim/src/exec/logging.rs @@ -52,6 +52,7 @@ pub(crate) fn collect_world_telemetry(span_id: &str) -> (Vec, Option, } +#[derive(Debug, Clone)] +pub(crate) struct LinuxWorldPlacementContext { + pub working_dir: PathBuf, + pub cgroup_path: PathBuf, + pub require_cgroup_attach: bool, +} + +impl LinuxWorldPlacementContext { + pub(crate) fn cgroup_procs_path(&self) -> PathBuf { + self.cgroup_path.join("cgroup.procs") + } +} + +impl From<&GatewayRuntimeStartContext> for LinuxWorldPlacementContext { + fn from(value: &GatewayRuntimeStartContext) -> Self { + Self { + working_dir: value.project_dir.clone(), + cgroup_path: value.cgroup_path.clone(), + require_cgroup_attach: value.require_cgroup_attach, + } + } +} + +#[derive(Debug, Clone)] +pub(crate) struct PreparedLinuxWorldEntryLauncher { + pub launcher_path: PathBuf, + pub env: Vec<(String, String)>, +} + #[derive(Debug, Clone, Serialize, Deserialize)] struct GatewayRuntimeManifest { world_id: String, @@ -632,6 +678,8 @@ fn start_runtime( ctx: GatewayRuntimeStartContext, ) -> Result { validate_binding_capabilities(ctx.binding)?; + let placement = LinuxWorldPlacementContext::from(&ctx); + validate_linux_world_placement_context(&placement)?; let port = pick_free_port().map_err(|err| GatewayRuntimeFailure::transient(err.to_string()))?; let runtime_dir = runtime_dir_for_world(&ctx.world_id, ctx.binding.backend_id); let home_dir = runtime_dir.join("home"); @@ -667,7 +715,8 @@ fn start_runtime( .with_context(|| format!("failed to create {}", stderr_log.display())) .map_err(|err| GatewayRuntimeFailure::transient(err.to_string()))?; - let auth = resolve_integrated_auth_handoff(ctx.binding, ctx.integrated_auth)?; + let auth_bundle_handoff = + prepare_gateway_auth_bundle_handoff(ctx.binding, ctx.integrated_auth)?; let mut command = Command::new(&binary_path); command @@ -678,24 +727,25 @@ fn start_runtime( .env("HOME", &home_dir) .env(GATEWAY_LAUNCH_MODE_ENV, GATEWAY_MODE_IN_WORLD) .env(GATEWAY_LAUNCH_CONFIG_PATH_ENV, &config_path) - .env(GATEWAY_LAUNCH_DISABLE_TOKEN_PERSISTENCE_ENV, "1"); + .env(GATEWAY_LAUNCH_DISABLE_TOKEN_PERSISTENCE_ENV, "1") + .env_remove(SUBSTRATE_LLM_AUTH_BUNDLE_FD) + .env( + SUBSTRATE_LLM_AUTH_BUNDLE_FD, + &auth_bundle_handoff.fd_env_value, + ); for env_key in KNOWN_GATEWAY_AUTH_ENV_VARS { command.env_remove(env_key); } - for (env_key, value) in auth.env_vars { - command.env(env_key, value); - } append_gateway_start_args(&mut command, &config_path); let mut child = command .spawn() .with_context(|| format!("failed to spawn {}", binary_path.display())) .map_err(|err| GatewayRuntimeFailure::transient(err.to_string()))?; + drop(auth_bundle_handoff); - if let Err(err) = - attach_child_to_cgroup(child.id(), &ctx.cgroup_path, ctx.require_cgroup_attach) - { + if let Err(err) = attach_pid_to_linux_world_cgroup(child.id(), &placement) { let _ = kill_child_process(&mut child); return Err(err); } @@ -735,15 +785,35 @@ fn stop_runtime(runtime: ManagedGatewayRuntime) -> Result<()> { stop_process(&mut process) } -fn attach_child_to_cgroup( +pub(crate) fn validate_linux_world_placement_context( + placement: &LinuxWorldPlacementContext, +) -> Result<(), GatewayRuntimeFailure> { + if !placement.working_dir.is_dir() { + return Err(GatewayRuntimeFailure::transient(format!( + "world entry working_dir is missing or not a directory: {}", + placement.working_dir.display() + ))); + } + + let cgroup_procs = placement.cgroup_procs_path(); + if placement.require_cgroup_attach && !cgroup_procs.is_file() { + return Err(GatewayRuntimeFailure::transient(format!( + "world entry cgroup attach target is missing: {}", + cgroup_procs.display() + ))); + } + + Ok(()) +} + +pub(crate) fn attach_pid_to_linux_world_cgroup( pid: u32, - cgroup_path: &Path, - required: bool, + placement: &LinuxWorldPlacementContext, ) -> Result<(), GatewayRuntimeFailure> { - let cgroup_procs = cgroup_path.join("cgroup.procs"); + let cgroup_procs = placement.cgroup_procs_path(); match fs::write(&cgroup_procs, pid.to_string()) { Ok(()) => Ok(()), - Err(_err) if !required => Ok(()), + Err(_err) if !placement.require_cgroup_attach => Ok(()), Err(err) => Err(GatewayRuntimeFailure::transient(format!( "failed to attach gateway pid {} to {}: {}", pid, @@ -753,6 +823,113 @@ fn attach_child_to_cgroup( } } +pub(crate) fn prepare_linux_world_entry_launcher( + launcher_dir: &Path, + actual_binary_path: &Path, + placement: &LinuxWorldPlacementContext, +) -> Result { + validate_linux_world_placement_context(placement)?; + + if !actual_binary_path.is_file() { + return Err(GatewayRuntimeFailure::invalid_integration(format!( + "world entry binary does not exist or is not a file: {}", + actual_binary_path.display() + ))); + } + + ensure_directory_with_mode(launcher_dir, GATEWAY_RUNTIME_DIR_MODE) + .with_context(|| { + format!( + "failed to create world entry launcher dir {}", + launcher_dir.display() + ) + }) + .map_err(|err| GatewayRuntimeFailure::transient(err.to_string()))?; + + let launcher_path = launcher_dir.join("world-entry.sh"); + write_file_with_mode( + &launcher_path, + render_linux_world_entry_wrapper().as_bytes(), + WORLD_ENTRY_WRAPPER_MODE, + ) + .with_context(|| format!("failed to write {}", launcher_path.display())) + .map_err(|err| GatewayRuntimeFailure::transient(err.to_string()))?; + + Ok(PreparedLinuxWorldEntryLauncher { + launcher_path, + env: vec![ + ( + WORLD_ENTRY_BINARY_ENV.to_string(), + actual_binary_path.display().to_string(), + ), + ( + WORLD_ENTRY_WORKING_DIR_ENV.to_string(), + placement.working_dir.display().to_string(), + ), + ( + WORLD_ENTRY_CGROUP_PROCS_ENV.to_string(), + placement.cgroup_procs_path().display().to_string(), + ), + ( + WORLD_ENTRY_REQUIRE_CGROUP_ATTACH_ENV.to_string(), + if placement.require_cgroup_attach { + "1".to_string() + } else { + "0".to_string() + }, + ), + ], + }) +} + +fn render_linux_world_entry_wrapper() -> &'static str { + r#"#!/bin/sh +set -eu + +binary="${SUBSTRATE_WORLD_ENTRY_BINARY:-}" +working_dir="${SUBSTRATE_WORLD_ENTRY_WORKING_DIR:-}" +cgroup_procs_path="${SUBSTRATE_WORLD_ENTRY_CGROUP_PROCS_PATH:-}" +require_cgroup_attach="${SUBSTRATE_WORLD_ENTRY_REQUIRE_CGROUP_ATTACH:-1}" + +if [ -z "$binary" ]; then + echo "substrate: error: world entry missing binary path" >&2 + exit 125 +fi +if [ ! -f "$binary" ]; then + echo "substrate: error: world entry binary missing: $binary" >&2 + exit 125 +fi +if [ -z "$working_dir" ]; then + echo "substrate: error: world entry missing working_dir" >&2 + exit 125 +fi +if [ ! -d "$working_dir" ]; then + echo "substrate: error: world entry working_dir missing: $working_dir" >&2 + exit 125 +fi +cd "$working_dir" + +if [ "$require_cgroup_attach" = "1" ]; then + if [ -z "$cgroup_procs_path" ]; then + echo "substrate: error: world entry missing cgroup attach target" >&2 + exit 125 + fi + if [ ! -e "$cgroup_procs_path" ]; then + echo "substrate: error: world entry cgroup attach target does not exist: $cgroup_procs_path" >&2 + exit 125 + fi + if ! printf '%s\n' "$$" > "$cgroup_procs_path"; then + echo "substrate: error: world entry cgroup attach failed: $cgroup_procs_path" >&2 + exit 125 + fi +elif [ -n "$cgroup_procs_path" ] && [ -e "$cgroup_procs_path" ]; then + printf '%s\n' "$$" > "$cgroup_procs_path" 2>/dev/null || true +fi + +exec "$binary" "$@" +"# +} + fn append_gateway_start_args(command: &mut Command, config_path: &Path) { command.arg("--config").arg(config_path).arg("start"); } @@ -927,14 +1104,74 @@ fn render_provider_auth_config(binding: &GatewayBackendBinding) -> String { GatewayProviderAuthConfig::OAuth { oauth_provider } => { format!("auth_type = \"oauth\"\noauth_provider = \"{oauth_provider}\"") } - GatewayProviderAuthConfig::ApiKey { env_var } => { + GatewayProviderAuthConfig::ApiKey { env_var, .. } => { format!("auth_type = \"apikey\"\napi_key = \"${env_var}\"") } } } struct ResolvedGatewayAuthHandoff { - env_vars: Vec<(&'static str, String)>, + bundle: GatewayAuthBundleV1, +} + +struct GatewayAuthBundleHandoff { + _read_fd: OwnedFd, + fd_env_value: String, +} + +fn prepare_gateway_auth_bundle_handoff( + binding: &GatewayBackendBinding, + auth: Option, +) -> Result { + let bundle = resolve_integrated_auth_handoff(binding, auth)?.bundle; + bundle.validate().map_err(|err| { + GatewayRuntimeFailure::transient(format!("invalid gateway auth bundle: {err}")) + })?; + let encoded = serde_json::to_vec(&bundle) + .context("failed to encode gateway auth bundle") + .map_err(|err| GatewayRuntimeFailure::transient(err.to_string()))?; + let (read_fd, mut write_file) = create_inherited_auth_bundle_pipe() + .map_err(|err| GatewayRuntimeFailure::transient(err.to_string()))?; + write_file + .write_all(&encoded) + .context("failed to write gateway auth bundle") + .map_err(|err| GatewayRuntimeFailure::transient(err.to_string()))?; + drop(write_file); + + Ok(GatewayAuthBundleHandoff { + fd_env_value: read_fd.as_raw_fd().to_string(), + _read_fd: read_fd, + }) +} + +fn create_inherited_auth_bundle_pipe() -> Result<(OwnedFd, fs::File)> { + let mut fds = [-1; 2]; + let rc = unsafe { libc::pipe2(fds.as_mut_ptr(), libc::O_CLOEXEC) }; + if rc != 0 { + return Err(std::io::Error::last_os_error()).context("failed to create auth bundle pipe"); + } + + let read_fd = unsafe { OwnedFd::from_raw_fd(fds[0]) }; + let write_fd = unsafe { OwnedFd::from_raw_fd(fds[1]) }; + clear_close_on_exec(read_fd.as_raw_fd())?; + + Ok((read_fd, fs::File::from(write_fd))) +} + +fn clear_close_on_exec(fd: i32) -> Result<()> { + let flags = unsafe { libc::fcntl(fd, libc::F_GETFD) }; + if flags < 0 { + return Err(std::io::Error::last_os_error()) + .context("failed to inspect auth bundle fd flags"); + } + + let rc = unsafe { libc::fcntl(fd, libc::F_SETFD, flags & !libc::FD_CLOEXEC) }; + if rc != 0 { + return Err(std::io::Error::last_os_error()) + .context("failed to mark auth bundle fd inheritable"); + } + + Ok(()) } fn resolve_integrated_auth_handoff( @@ -975,7 +1212,7 @@ fn resolve_codex_auth_handoff( if access_token.is_empty() { return Err(GatewayRuntimeFailure::invalid_integration(format!( "request-provided {} is empty", - CODEX_ACCESS_TOKEN_ENV + SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCESS_TOKEN ))); } @@ -984,19 +1221,35 @@ fn resolve_codex_auth_handoff( .map(|value| value.trim().to_string()) .filter(|value| !value.is_empty()); - let mut env_vars = vec![(CODEX_ACCESS_TOKEN_ENV, access_token)]; + let mut fields = HashMap::from([( + SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCESS_TOKEN.to_string(), + access_token, + )]); if let Some(account_id) = account_id { - env_vars.push((CODEX_ACCOUNT_ID_ENV, account_id)); + fields.insert( + SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCOUNT_ID.to_string(), + account_id, + ); } - Ok(ResolvedGatewayAuthHandoff { env_vars }) + Ok(ResolvedGatewayAuthHandoff { + bundle: GatewayAuthBundleV1 { + schema_version: GATEWAY_AUTH_BUNDLE_SCHEMA_VERSION, + backend_id: DEFAULT_BACKEND.to_string(), + fields, + }, + }) } fn resolve_api_env_auth_handoff( binding: &GatewayBackendBinding, auth: &GatewayIntegratedAuthPayloadV1, ) -> Result { - let GatewayProviderAuthConfig::ApiKey { env_var } = binding.provider_auth else { + let GatewayProviderAuthConfig::ApiKey { + env_var, + bundle_field, + } = binding.provider_auth + else { return Err(GatewayRuntimeFailure::invalid_integration(format!( "backend '{}' is not configured for api env auth", binding.backend_id @@ -1025,7 +1278,11 @@ fn resolve_api_env_auth_handoff( } Ok(ResolvedGatewayAuthHandoff { - env_vars: vec![(env_var, value)], + bundle: GatewayAuthBundleV1 { + schema_version: GATEWAY_AUTH_BUNDLE_SCHEMA_VERSION, + backend_id: binding.backend_id.to_string(), + fields: HashMap::from([(bundle_field.to_string(), value)]), + }, }) } @@ -1312,8 +1569,36 @@ mod tests { } fn start_context(project_dir: &Path, world_id: &str) -> GatewayRuntimeStartContext { + start_context_with_codex_auth( + project_dir, + world_id, + "header.payload.signature", + Some("acct_test"), + ) + } + + fn start_context_with_codex_auth( + project_dir: &Path, + world_id: &str, + access_token: &str, + account_id: Option<&str>, + ) -> GatewayRuntimeStartContext { let binding = resolve_gateway_backend_binding(DEFAULT_BACKEND).expect("codex binding"); - start_context_with_binding(project_dir, world_id, binding) + GatewayRuntimeStartContext { + world_id: world_id.to_string(), + project_dir: project_dir.to_path_buf(), + cgroup_path: project_dir.join("missing-cgroup"), + require_cgroup_attach: false, + binding, + integrated_auth: Some(GatewayIntegratedAuthPayloadV1 { + backend_id: binding.backend_id.to_string(), + cli_codex: Some(GatewayCliCodexIntegratedAuthV1 { + account_id: account_id.map(str::to_string), + access_token: access_token.to_string(), + }), + api_env: None, + }), + } } fn start_context_with_binding( @@ -1358,9 +1643,17 @@ mod tests { } } + fn auth_bundle_snapshot_path(temp_dir: &TempDir, launch: u32) -> PathBuf { + temp_dir + .path() + .join("auth-bundles") + .join(format!("{launch}.json")) + } + fn delayed_gateway_binary(temp_dir: &TempDir, delay_ms: u64) -> (PathBuf, PathBuf, PathBuf) { let path = temp_dir.path().join("delayed-gateway.sh"); let pid_dir = temp_dir.path().join("pids"); + let bundle_dir = temp_dir.path().join("auth-bundles"); let launch_count_path = temp_dir.path().join("launch-count.txt"); fs::write( &path, @@ -1388,8 +1681,8 @@ if [ -z "$config" ]; then exit 64 fi -if [ -z "${{SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCESS_TOKEN:-}}" ]; then - echo "missing Codex access token env" >&2 +if [ -z "${{SUBSTRATE_LLM_AUTH_BUNDLE_FD:-}}" ]; then + echo "missing gateway auth bundle fd env" >&2 exit 65 fi @@ -1406,6 +1699,35 @@ PY mkdir -p "{pid_dir}" printf '%s\n' "$$" >"{pid_dir}/$launch.pid" +python3 - "$launch" "{bundle_dir}" <<'PY' +import json +import os +import pathlib +import sys + +launch = sys.argv[1] +bundle_dir = pathlib.Path(sys.argv[2]) +bundle_dir.mkdir(parents=True, exist_ok=True) +for forbidden in ( + "SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCOUNT_ID", + "SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCESS_TOKEN", + "SUBSTRATE_LLM_BACKEND_AUTH_API_OPENAI_API_KEY", + "OPENAI_API_KEY", +): + if os.environ.get(forbidden): + raise SystemExit("forbidden secret env present: " + forbidden) +fd_raw = os.environ["SUBSTRATE_LLM_AUTH_BUNDLE_FD"] +with os.fdopen(int(fd_raw), "rb") as handle: + payload = handle.read() +bundle = json.loads(payload) +(bundle_dir / (launch + ".json")).write_bytes(payload) +if bundle.get("backend_id") != "cli:codex": + raise SystemExit("unexpected backend_id") +fields = bundle.get("fields") or dict() +if "SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCESS_TOKEN" not in fields: + raise SystemExit("missing Codex token field") +PY + port="$(python3 - "$config" <<'PY' import re import sys @@ -1424,6 +1746,7 @@ printf 'ok' >"$root/health" exec python3 -m http.server "$port" --bind 127.0.0.1 --directory "$root" "#, launch_count_path = launch_count_path.display(), + bundle_dir = bundle_dir.display(), pid_dir = pid_dir.display(), delay_s = format_delay_seconds(delay_ms), ), @@ -1438,6 +1761,7 @@ exec python3 -m http.server "$port" --bind 127.0.0.1 --directory "$root" fn first_launch_hangs_second_ready_binary(temp_dir: &TempDir) -> (PathBuf, PathBuf, PathBuf) { let path = temp_dir.path().join("phased-gateway.sh"); let pid_dir = temp_dir.path().join("pids"); + let bundle_dir = temp_dir.path().join("auth-bundles"); let launch_count_path = temp_dir.path().join("launch-count.txt"); fs::write( &path, @@ -1465,8 +1789,8 @@ if [ -z "$config" ]; then exit 64 fi -if [ -z "${{SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCESS_TOKEN:-}}" ]; then - echo "missing Codex access token env" >&2 +if [ -z "${{SUBSTRATE_LLM_AUTH_BUNDLE_FD:-}}" ]; then + echo "missing gateway auth bundle fd env" >&2 exit 65 fi @@ -1483,6 +1807,35 @@ PY mkdir -p "{pid_dir}" printf '%s\n' "$$" >"{pid_dir}/$launch.pid" +python3 - "$launch" "{bundle_dir}" <<'PY' +import json +import os +import pathlib +import sys + +launch = sys.argv[1] +bundle_dir = pathlib.Path(sys.argv[2]) +bundle_dir.mkdir(parents=True, exist_ok=True) +for forbidden in ( + "SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCOUNT_ID", + "SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCESS_TOKEN", + "SUBSTRATE_LLM_BACKEND_AUTH_API_OPENAI_API_KEY", + "OPENAI_API_KEY", +): + if os.environ.get(forbidden): + raise SystemExit("forbidden secret env present: " + forbidden) +fd_raw = os.environ["SUBSTRATE_LLM_AUTH_BUNDLE_FD"] +with os.fdopen(int(fd_raw), "rb") as handle: + payload = handle.read() +bundle = json.loads(payload) +(bundle_dir / (launch + ".json")).write_bytes(payload) +if bundle.get("backend_id") != "cli:codex": + raise SystemExit("unexpected backend_id") +fields = bundle.get("fields") or dict() +if "SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCESS_TOKEN" not in fields: + raise SystemExit("missing Codex token field") +PY + if [ "$launch" = "1" ]; then sleep 30 exit 0 @@ -1504,6 +1857,7 @@ printf 'ok' >"$root/health" exec python3 -m http.server "$port" --bind 127.0.0.1 --directory "$root" "#, launch_count_path = launch_count_path.display(), + bundle_dir = bundle_dir.display(), pid_dir = pid_dir.display(), ), ) @@ -1537,6 +1891,23 @@ exec python3 -m http.server "$port" --bind 127.0.0.1 --directory "$root" } } + fn wait_for_auth_bundle_snapshot(temp_dir: &TempDir, launch: u32) -> GatewayAuthBundleV1 { + let path = auth_bundle_snapshot_path(temp_dir, launch); + let deadline = Instant::now() + Duration::from_secs(2); + loop { + if let Ok(raw) = fs::read(&path) { + return serde_json::from_slice(&raw) + .unwrap_or_else(|err| panic!("invalid {}: {err}", path.display())); + } + assert!( + Instant::now() < deadline, + "timed out waiting for {}", + path.display() + ); + std::thread::sleep(Duration::from_millis(25)); + } + } + fn read_launch_count(path: &Path) -> u32 { fs::read_to_string(path) .ok() @@ -1725,8 +2096,11 @@ exec python3 -m http.server "$port" --bind 127.0.0.1 --directory "$root" .expect("openai auth handoff"); assert_eq!( - auth.env_vars, - vec![(OPENAI_API_KEY_ENV, "sk-openai-proof".to_string())] + auth.bundle.fields, + HashMap::from([( + SUBSTRATE_LLM_BACKEND_AUTH_API_OPENAI_API_KEY.to_string(), + "sk-openai-proof".to_string(), + )]) ); } @@ -1809,6 +2183,20 @@ exec python3 -m http.server "$port" --bind 127.0.0.1 --directory "$root" GATEWAY_RUNTIME_FILE_MODE, ); assert_mode(&runtime.manifest_path, GATEWAY_RUNTIME_FILE_MODE); + let bundle = wait_for_auth_bundle_snapshot(&temp_dir, 1); + assert_eq!(bundle.backend_id, DEFAULT_BACKEND); + assert_eq!( + bundle + .fields + .get(SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCESS_TOKEN), + Some(&"header.payload.signature".to_string()) + ); + assert_eq!( + bundle + .fields + .get(SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCOUNT_ID), + Some(&"acct_test".to_string()) + ); stop_runtime(runtime).expect("stop runtime"); } @@ -1973,4 +2361,59 @@ exec python3 -m http.server "$port" --bind 127.0.0.1 --directory "$root" let response = restart.await.unwrap().expect("restart should finish"); assert_eq!(response.status, GatewayStatusV1::Available); } + + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn restart_redelivers_a_fresh_auth_bundle() { + let _env_lock = ENV_LOCK.lock().await; + let temp_dir = TempDir::new().unwrap(); + let runtime_root = temp_dir.path().join("runtime-root"); + let _runtime_root_guard = EnvGuard::set("SUBSTRATE_GATEWAY_RUNTIME_ROOT", &runtime_root); + let (binary, _pid_dir, launch_count_path) = delayed_gateway_binary(&temp_dir, 0); + let _binary_guard = EnvGuard::set(GATEWAY_BINARY_OVERRIDE_ENV, binary); + let manager = Arc::new(GatewayRuntimeManager::new()); + + let first = start_context_with_codex_auth( + temp_dir.path(), + "bundle-rotate", + "token-one", + Some("acct-one"), + ); + manager + .sync_with_timeout(first, Duration::from_secs(3)) + .await + .expect("initial sync"); + let first_bundle = wait_for_auth_bundle_snapshot(&temp_dir, 1); + assert_eq!( + first_bundle + .fields + .get(SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCESS_TOKEN), + Some(&"token-one".to_string()) + ); + + let second = start_context_with_codex_auth( + temp_dir.path(), + "bundle-rotate", + "token-two", + Some("acct-two"), + ); + manager + .restart(second) + .await + .expect("restart should redeliver auth"); + + let second_bundle = wait_for_auth_bundle_snapshot(&temp_dir, 2); + assert_eq!(read_launch_count(&launch_count_path), 2); + assert_eq!( + second_bundle + .fields + .get(SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCESS_TOKEN), + Some(&"token-two".to_string()) + ); + assert_eq!( + second_bundle + .fields + .get(SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCOUNT_ID), + Some(&"acct-two".to_string()) + ); + } } diff --git a/crates/world-agent/src/handlers.rs b/crates/world-agent/src/handlers.rs index 5571ab113..bdea1e35c 100644 --- a/crates/world-agent/src/handlers.rs +++ b/crates/world-agent/src/handlers.rs @@ -3,10 +3,11 @@ use crate::service::WorldAgentService; use agent_api_types::{ ApiError, ExecuteCancelRequestV1, ExecuteCancelResponseV1, ExecuteRequest, ExecuteResponse, - GatewayLifecycleRequestV1, GatewayLifecycleResponseV1, PendingDiffClearRequestV1, - PendingDiffClearResponseV1, PendingDiffReconcileRequestV1, PendingDiffReconcileResponseV1, - PendingDiffRecordV1, PendingDiffRequestV1, WorldDoctorLandlockV1, WorldDoctorNetfilterStatusV1, - WorldDoctorReportV1, WorldDoctorWorldFsStrategyKindV1, WorldDoctorWorldFsStrategyProbeResultV1, + GatewayLifecycleRequestV1, GatewayLifecycleResponseV1, MemberTurnSubmitRequestV1, + PendingDiffClearRequestV1, PendingDiffClearResponseV1, PendingDiffReconcileRequestV1, + PendingDiffReconcileResponseV1, PendingDiffRecordV1, PendingDiffRequestV1, + WorldDoctorLandlockV1, WorldDoctorNetfilterStatusV1, WorldDoctorReportV1, + WorldDoctorWorldFsStrategyKindV1, WorldDoctorWorldFsStrategyProbeResultV1, WorldDoctorWorldFsStrategyProbeV1, WorldDoctorWorldFsStrategyV1, WorldFsReadRequestV1, WorldFsReadResponseV1, }; @@ -349,6 +350,24 @@ pub async fn execute_stream( }) } +/// Submit a follow-up turn to a retained world member and stream the response. +pub async fn member_turn_stream( + State(service): State, + body: Bytes, +) -> Result { + let payload: Value = serde_json::from_slice(&body) + .map_err(|e| ApiErrorResponse(ApiError::BadRequest(format!("Invalid JSON: {e}"))))?; + let req: MemberTurnSubmitRequestV1 = serde_json::from_value(payload) + .map_err(|e| ApiErrorResponse(ApiError::BadRequest(format!("Invalid JSON: {e}"))))?; + service.submit_member_turn_stream(req).await.map_err(|e| { + if let Some(bad) = e.downcast_ref::() { + ApiErrorResponse(ApiError::BadRequest(bad.message().to_string())) + } else { + ApiErrorResponse(ApiError::Internal(e.to_string())) + } + }) +} + /// Send a signal to an active streamed execution. pub async fn execute_cancel( State(service): State, diff --git a/crates/world-agent/src/internal_exec.rs b/crates/world-agent/src/internal_exec.rs index 3e811e517..809cfe12e 100644 --- a/crates/world-agent/src/internal_exec.rs +++ b/crates/world-agent/src/internal_exec.rs @@ -1065,36 +1065,23 @@ fn drop_caps_for_deny_enforcement(plan: &enforcement_plan::EnforcementPlanV1) -> #[cfg(all(target_os = "linux", not(target_env = "musl")))] fn install_strict_deny_seccomp() -> Result<()> { - use libseccomp::{ScmpAction, ScmpFilterContext, ScmpSyscall}; - - let mut ctx = ScmpFilterContext::new_filter(ScmpAction::Allow) - .map_err(|e| anyhow::anyhow!("seccomp init failed: {e}"))?; - - let syscalls = [ - "mount", - "umount2", - "pivot_root", - "open_tree", - "move_mount", - "fsopen", - "fsmount", - "fspick", + let rules = [ + substrate_common::seccomp::SeccompRule::errno("mount", libc::EPERM), + substrate_common::seccomp::SeccompRule::errno("umount2", libc::EPERM), + substrate_common::seccomp::SeccompRule::errno("pivot_root", libc::EPERM), + substrate_common::seccomp::SeccompRule::errno("open_tree", libc::EPERM), + substrate_common::seccomp::SeccompRule::errno("move_mount", libc::EPERM), + substrate_common::seccomp::SeccompRule::errno("fsopen", libc::EPERM), + substrate_common::seccomp::SeccompRule::errno("fsmount", libc::EPERM), + substrate_common::seccomp::SeccompRule::errno("fspick", libc::EPERM), ]; - for name in syscalls { - let syscall = match ScmpSyscall::from_name(name) { - Ok(v) => v, - Err(_) => continue, // syscall not present on this kernel/arch (N/A) - }; - - ctx.add_rule(ScmpAction::Errno(libc::EPERM), syscall) - .map_err(|e| anyhow::anyhow!("seccomp add_rule({name}) failed: {e}"))?; + match substrate_common::seccomp::install_allow_filter(&rules)? { + substrate_common::seccomp::SeccompInstallStatus::Installed => Ok(()), + substrate_common::seccomp::SeccompInstallStatus::Unavailable(reason) => { + anyhow::bail!("strict deny enforcement requires seccomp, but {reason}") + } } - - ctx.load() - .map_err(|e| anyhow::anyhow!("seccomp load failed: {e}"))?; - - Ok(()) } #[cfg(all(target_os = "linux", target_env = "musl"))] diff --git a/crates/world-agent/src/lib.rs b/crates/world-agent/src/lib.rs index 8e43c4ddd..19e5bf73c 100644 --- a/crates/world-agent/src/lib.rs +++ b/crates/world-agent/src/lib.rs @@ -6,6 +6,8 @@ mod gateway_runtime; pub mod gc; pub mod handlers; pub mod internal_exec; +#[cfg(target_os = "linux")] +mod member_runtime; pub mod pty; mod request_routing; pub mod service; @@ -266,6 +268,7 @@ fn build_router(service: WorldAgentService) -> Router { .route("/v1/gateway/sync", post(handlers::gateway_sync)) .route("/v1/gateway/restart", post(handlers::gateway_restart)) .route("/v1/execute/stream", post(handlers::execute_stream)) + .route("/v1/member_turn/stream", post(handlers::member_turn_stream)) .route("/v1/stream", get(handlers::stream)) .route("/v1/trace/:span_id", get(handlers::get_trace)) .route("/v1/request_scopes", post(handlers::request_scopes)) diff --git a/crates/world-agent/src/member_runtime.rs b/crates/world-agent/src/member_runtime.rs new file mode 100644 index 000000000..453266f94 --- /dev/null +++ b/crates/world-agent/src/member_runtime.rs @@ -0,0 +1,1275 @@ +use agent_api::{ + backends::{ + claude_code::{ClaudeCodeBackend, ClaudeCodeBackendConfig}, + codex::{CodexBackend, CodexBackendConfig}, + }, + AgentWrapperCancelHandle, AgentWrapperCompletion, AgentWrapperError, AgentWrapperEvent, + AgentWrapperEventKind, AgentWrapperGateway, AgentWrapperKind, AgentWrapperRunControl, + AgentWrapperRunRequest, +}; +use agent_api_types::{ + ExecuteStreamFrame, MemberDispatchRequestV1, MemberRuntimeBackendKindV1, + MemberTurnSubmitRequestV1, ProcessTelemetry, +}; +use anyhow::{anyhow, Result}; +use axum::{ + body::{boxed, Bytes, StreamBody}, + http::StatusCode, + response::Response, +}; +use futures_util::StreamExt; +use serde_json::json; +use std::{ + collections::{BTreeMap, HashMap}, + convert::Infallible, + fs, + path::Path, + sync::{Arc, Mutex, RwLock}, +}; +use substrate_common::agent_events::{AgentEvent, AgentEventKind, MessageEventKind}; +use tokio_stream::wrappers::UnboundedReceiverStream; +use world_api::SharedWorldBindingSnapshot; + +use crate::gateway_runtime::{prepare_linux_world_entry_launcher, LinuxWorldPlacementContext}; + +const MEMBER_ROLE: &str = "member"; +const SESSION_HANDLE_SCHEMA_V1: &str = "agent_api.session.handle.v1"; +const CANCELLED_MESSAGE: &str = "cancelled"; +const SESSION_RESUME_EXTENSION_V1: &str = "agent_api.session.resume.v1"; + +#[derive(Clone, Default)] +pub(crate) struct MemberRuntimeManager { + active_members: Arc>, + active_turns_by_span_id: Arc>>>, +} + +#[derive(Default)] +struct ActiveMemberRegistry { + by_participant_id: HashMap>, + by_retained_key: HashMap, +} + +struct ActiveMemberRuntime { + agent_id: String, + participant_id: String, + orchestration_session_id: String, + orchestrator_participant_id: String, + parent_participant_id: Option, + resumed_from_participant_id: Option, + backend_id: String, + backend_kind: MemberRuntimeBackendKindV1, + binary_path: std::path::PathBuf, + working_dir: std::path::PathBuf, + env: BTreeMap, + binding: SharedWorldBindingSnapshot, + protocol: serde_json::Value, + bootstrap_span_id: String, + bootstrap_cancel: AgentWrapperCancelHandle, + bootstrap_last_signal: Mutex>, + active_turn_span_id: Mutex>, + uaa_session_id: Mutex>, + launcher_dir: Option, +} + +struct ActiveSubmittedTurn { + participant_id: String, + cancel: AgentWrapperCancelHandle, + last_signal: Mutex>, +} + +#[derive(Clone)] +struct MemberStreamContext { + orchestration_session_id: String, + run_id: String, + participant_id: String, + parent_participant_id: Option, + resumed_from_participant_id: Option, + backend_id: String, + protocol: serde_json::Value, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +enum MemberStreamMode { + Bootstrap, + SubmittedTurn, +} + +#[derive(Clone, Debug, Eq, Hash, PartialEq)] +struct RetainedMemberKey { + orchestration_session_id: String, + world_generation: u64, + backend_id: String, +} + +impl MemberRuntimeManager { + pub(crate) fn new() -> Self { + Self::default() + } + + pub(crate) async fn launch( + &self, + agent_id: String, + env: HashMap, + span_id: String, + dispatch: MemberDispatchRequestV1, + binding: SharedWorldBindingSnapshot, + placement: LinuxWorldPlacementContext, + ) -> Result { + let actual_binary = validate_member_runtime_binary(&dispatch)?; + let prepared_launcher = prepare_member_runtime_launcher(&actual_binary, &placement)?; + let runtime_env = env + .into_iter() + .chain(prepared_launcher.env.iter().cloned()) + .collect::>(); + + let (gateway, agent_kind) = build_gateway_for_backend( + &dispatch.resolved_runtime.backend_kind, + prepared_launcher.launcher_path.clone(), + )?; + let initial_prompt = dispatch + .initial_prompt + .clone() + .unwrap_or_else(|| runtime_bootstrap_prompt().to_string()); + let AgentWrapperRunControl { handle, cancel } = match gateway + .run_control( + &agent_kind, + AgentWrapperRunRequest { + prompt: initial_prompt, + working_dir: Some(placement.working_dir.clone()), + timeout: None, + env: runtime_env.clone(), + extensions: BTreeMap::new(), + }, + ) + .await + { + Ok(control) => control, + Err(err) => { + let _ = fs::remove_dir_all(&prepared_launcher.launcher_dir); + return Err(map_wrapper_error(err)); + } + }; + + let active = Arc::new(ActiveMemberRuntime { + agent_id, + participant_id: dispatch.participant_id.clone(), + orchestration_session_id: dispatch.orchestration_session_id.clone(), + orchestrator_participant_id: dispatch.orchestrator_participant_id.clone(), + parent_participant_id: dispatch.parent_participant_id.clone(), + resumed_from_participant_id: dispatch.resumed_from_participant_id.clone(), + backend_id: dispatch.backend_id.clone(), + backend_kind: dispatch.resolved_runtime.backend_kind, + binary_path: actual_binary, + working_dir: placement.working_dir.clone(), + env: runtime_env, + binding: binding.clone(), + protocol: json!(dispatch.protocol), + bootstrap_span_id: span_id.clone(), + bootstrap_cancel: cancel, + bootstrap_last_signal: Mutex::new(None), + active_turn_span_id: Mutex::new(None), + uaa_session_id: Mutex::new(None), + launcher_dir: Some(prepared_launcher.launcher_dir), + }); + if let Err(err) = self.register_member(active.clone()) { + active.bootstrap_cancel.cancel(); + if let Some(launcher_dir) = active.launcher_dir.as_ref() { + let _ = fs::remove_dir_all(launcher_dir); + } + return Err(err); + } + + let (tx, rx) = tokio::sync::mpsc::unbounded_channel::(); + let _ = tx.send(ExecuteStreamFrame::Start { + span_id: span_id.clone(), + }); + + let manager = self.clone(); + let participant_id = dispatch.participant_id.clone(); + let context = MemberStreamContext { + orchestration_session_id: dispatch.orchestration_session_id.clone(), + run_id: dispatch.run_id.clone(), + participant_id: dispatch.participant_id.clone(), + parent_participant_id: dispatch.parent_participant_id.clone(), + resumed_from_participant_id: dispatch.resumed_from_participant_id.clone(), + backend_id: dispatch.backend_id.clone(), + protocol: json!(dispatch.protocol), + }; + tokio::spawn(async move { + let mut events = handle.events; + let completion = handle.completion; + let mut emitted_registered = false; + + while let Some(wrapper_event) = events.next().await { + if let Some(session_id) = + surfaced_uaa_session_id_from_data(wrapper_event.data.as_ref()) + { + manager.remember_uaa_session_id(&participant_id, session_id); + } + if let Some(frame) = frame_from_wrapper_event( + &context, + &binding, + &span_id, + wrapper_event, + &mut emitted_registered, + MemberStreamMode::Bootstrap, + active.agent_id.as_str(), + ) { + let _ = tx.send(frame); + } + } + + let completion = completion.await; + if let Ok(ref completion) = completion { + if let Some(session_id) = + surfaced_uaa_session_id_from_data(completion.data.as_ref()) + { + manager.remember_uaa_session_id(&participant_id, session_id); + } + } + for frame in frames_from_completion( + &context, + &binding, + &span_id, + completion, + active.bootstrap_last_signal(), + &mut emitted_registered, + MemberStreamMode::Bootstrap, + active.agent_id.as_str(), + ) { + let _ = tx.send(frame); + } + + manager.unregister_member(&participant_id); + }); + + stream_response(rx) + } + + pub(crate) async fn submit_turn(&self, req: MemberTurnSubmitRequestV1) -> Result { + let active = self.find_submit_target(&req)?; + validate_submit_turn_request(&req, RetainedMemberIdentity::from_active(active.as_ref()))?; + self.validate_submit_target_slot(&req)?; + let uaa_session_id = active.uaa_session_id().ok_or_else(|| { + crate::service::BadRequestError::new(format!( + "member_turn_submit.participant_id {} has no surfaced uaa_session_id", + req.participant_id + )) + })?; + + let span_id = format!("spn_{}", uuid::Uuid::now_v7()); + self.reserve_turn_slot(&active, &span_id)?; + + let (gateway, agent_kind) = + build_gateway_for_backend(&active.backend_kind, active.binary_path.clone())?; + let mut extensions = BTreeMap::new(); + extensions.insert( + SESSION_RESUME_EXTENSION_V1.to_string(), + json!({ + "selector": "id", + "id": uaa_session_id, + }), + ); + + let AgentWrapperRunControl { handle, cancel } = match gateway + .run_control( + &agent_kind, + AgentWrapperRunRequest { + prompt: req.prompt.clone(), + working_dir: Some(active.working_dir.clone()), + timeout: None, + env: active.env.clone(), + extensions, + }, + ) + .await + { + Ok(control) => control, + Err(err) => { + self.clear_reserved_turn_slot(&active, &span_id); + return Err(map_wrapper_error(err)); + } + }; + + let turn = Arc::new(ActiveSubmittedTurn { + participant_id: active.participant_id.clone(), + cancel, + last_signal: Mutex::new(None), + }); + self.register_turn(span_id.clone(), turn.clone()); + + let (tx, rx) = tokio::sync::mpsc::unbounded_channel::(); + let _ = tx.send(ExecuteStreamFrame::Start { + span_id: span_id.clone(), + }); + + let manager = self.clone(); + let context = active.submit_context(req.run_id.clone()); + let binding = active.binding.clone(); + tokio::spawn(async move { + let mut events = handle.events; + let completion = handle.completion; + let mut emitted_registered = false; + + while let Some(wrapper_event) = events.next().await { + if let Some(session_id) = + surfaced_uaa_session_id_from_data(wrapper_event.data.as_ref()) + { + manager.remember_uaa_session_id(&turn.participant_id, session_id); + } + if let Some(frame) = frame_from_wrapper_event( + &context, + &binding, + &span_id, + wrapper_event, + &mut emitted_registered, + MemberStreamMode::SubmittedTurn, + active.agent_id.as_str(), + ) { + let _ = tx.send(frame); + } + } + + let completion = completion.await; + if let Ok(ref completion) = completion { + if let Some(session_id) = + surfaced_uaa_session_id_from_data(completion.data.as_ref()) + { + manager.remember_uaa_session_id(&turn.participant_id, session_id); + } + } + for frame in frames_from_completion( + &context, + &binding, + &span_id, + completion, + turn.last_signal(), + &mut emitted_registered, + MemberStreamMode::SubmittedTurn, + active.agent_id.as_str(), + ) { + let _ = tx.send(frame); + } + + manager.unregister_turn(&span_id); + }); + + stream_response(rx) + } + + pub(crate) fn cancel(&self, span_id: &str, sig: &str) -> Result { + validate_cancel_signal(sig)?; + + let submitted_turn = self + .active_turns_by_span_id + .read() + .expect("member runtime registry lock poisoned") + .get(span_id) + .cloned(); + if let Some(submitted_turn) = submitted_turn { + if let Ok(mut guard) = submitted_turn.last_signal.lock() { + *guard = Some(sig.trim().to_ascii_uppercase()); + } + submitted_turn.cancel.cancel(); + return Ok(true); + } + + let bootstrap = self + .active_members + .read() + .expect("member runtime registry lock poisoned") + .by_participant_id + .values() + .find(|active| active.bootstrap_span_id == span_id) + .cloned(); + let Some(bootstrap) = bootstrap else { + return Ok(false); + }; + + if let Ok(mut guard) = bootstrap.bootstrap_last_signal.lock() { + *guard = Some(sig.trim().to_ascii_uppercase()); + } + bootstrap.bootstrap_cancel.cancel(); + Ok(true) + } + + fn register_member(&self, active: Arc) -> Result<()> { + let retained_key = RetainedMemberKey::from_active(active.as_ref()); + let mut guard = self + .active_members + .write() + .expect("member runtime registry lock poisoned"); + if guard.by_participant_id.contains_key(&active.participant_id) { + return Err(crate::service::BadRequestError::new(format!( + "member_dispatch.participant_id {} is already retained", + active.participant_id + )) + .into()); + } + if let Some(existing_participant_id) = guard.by_retained_key.get(&retained_key) { + return Err( + duplicate_retained_member_error(&retained_key, existing_participant_id).into(), + ); + } + + guard + .by_retained_key + .insert(retained_key, active.participant_id.clone()); + guard + .by_participant_id + .insert(active.participant_id.clone(), active); + Ok(()) + } + + fn unregister_member(&self, participant_id: &str) { + if let Ok(mut guard) = self.active_members.write() { + if let Some(active) = guard.by_participant_id.remove(participant_id) { + let retained_key = RetainedMemberKey::from_active(active.as_ref()); + guard.by_retained_key.remove(&retained_key); + if let Some(launcher_dir) = active.launcher_dir.as_ref() { + let _ = fs::remove_dir_all(launcher_dir); + } + } + } + } + + fn register_turn(&self, span_id: String, turn: Arc) { + self.active_turns_by_span_id + .write() + .expect("member runtime registry lock poisoned") + .insert(span_id, turn); + } + + fn unregister_turn(&self, span_id: &str) { + if let Ok(mut guard) = self.active_turns_by_span_id.write() { + if let Some(turn) = guard.remove(span_id) { + if let Some(active) = + self.active_members.read().ok().and_then(|active| { + active.by_participant_id.get(&turn.participant_id).cloned() + }) + { + self.clear_reserved_turn_slot(&active, span_id); + } + } + } + } + + fn reserve_turn_slot(&self, active: &Arc, span_id: &str) -> Result<()> { + let mut guard = active + .active_turn_span_id + .lock() + .map_err(|_| anyhow!("member runtime turn slot lock poisoned"))?; + if let Some(existing) = guard.as_ref() { + return Err(crate::service::BadRequestError::new(format!( + "member_turn_submit.participant_id {} already has an active submitted turn ({existing})", + active.participant_id + )) + .into()); + } + *guard = Some(span_id.to_string()); + Ok(()) + } + + fn clear_reserved_turn_slot(&self, active: &Arc, span_id: &str) { + if let Ok(mut guard) = active.active_turn_span_id.lock() { + if guard.as_deref() == Some(span_id) { + *guard = None; + } + } + } + + fn remember_uaa_session_id(&self, participant_id: &str, session_id: String) { + if let Some(active) = self + .active_members + .read() + .ok() + .and_then(|guard| guard.by_participant_id.get(participant_id).cloned()) + { + active.remember_uaa_session_id(session_id); + } + } + + fn find_submit_target( + &self, + req: &MemberTurnSubmitRequestV1, + ) -> Result> { + let retained_key = RetainedMemberKey::from_submit(req); + let guard = self + .active_members + .read() + .expect("member runtime registry lock poisoned"); + if let Some(active) = guard.by_participant_id.get(&req.participant_id).cloned() { + return Ok(active); + } + + if let Some(existing_participant_id) = guard.by_retained_key.get(&retained_key) { + return Err(retained_slot_owner_mismatch_error( + &retained_key, + existing_participant_id, + &req.participant_id, + ) + .into()); + } + + Err(crate::service::BadRequestError::new(format!( + "member_turn_submit.participant_id {} is not retained", + req.participant_id + )) + .into()) + } + + fn validate_submit_target_slot(&self, req: &MemberTurnSubmitRequestV1) -> Result<()> { + let retained_key = RetainedMemberKey::from_submit(req); + let guard = self + .active_members + .read() + .expect("member runtime registry lock poisoned"); + match guard.by_retained_key.get(&retained_key) { + Some(participant_id) if participant_id == &req.participant_id => Ok(()), + Some(participant_id) => Err(retained_slot_owner_mismatch_error( + &retained_key, + participant_id, + &req.participant_id, + ) + .into()), + None => Err(missing_retained_slot_error(&retained_key).into()), + } + } +} + +fn runtime_bootstrap_prompt() -> &'static str { + "Enter persistent Substrate world-scoped member mode. Keep this control session attached for the lifetime of the parent REPL session and do not exit until the client cancels the run." +} + +fn validate_member_runtime_binary( + dispatch: &MemberDispatchRequestV1, +) -> Result { + let path = Path::new(&dispatch.resolved_runtime.binary_path); + if !path.is_file() { + return Err(anyhow!( + "member_dispatch.resolved_runtime.binary_path does not exist or is not a file: {}", + dispatch.resolved_runtime.binary_path + )); + } + Ok(path.to_path_buf()) +} + +fn build_gateway_for_backend( + backend_kind: &MemberRuntimeBackendKindV1, + binary_path: std::path::PathBuf, +) -> Result<(AgentWrapperGateway, AgentWrapperKind)> { + let mut gateway = AgentWrapperGateway::new(); + let binary_path = Some(binary_path); + + let agent_kind = match backend_kind { + MemberRuntimeBackendKindV1::Codex => { + gateway + .register(Arc::new(CodexBackend::new(CodexBackendConfig { + binary: binary_path, + ..Default::default() + }))) + .map_err(map_wrapper_error)?; + AgentWrapperKind::new("codex").map_err(map_wrapper_error)? + } + MemberRuntimeBackendKindV1::ClaudeCode => { + gateway + .register(Arc::new(ClaudeCodeBackend::new(ClaudeCodeBackendConfig { + binary: binary_path, + ..Default::default() + }))) + .map_err(map_wrapper_error)?; + AgentWrapperKind::new("claude_code").map_err(map_wrapper_error)? + } + }; + + Ok((gateway, agent_kind)) +} + +struct PreparedMemberRuntimeLauncher { + launcher_path: std::path::PathBuf, + launcher_dir: std::path::PathBuf, + env: Vec<(String, String)>, +} + +fn prepare_member_runtime_launcher( + actual_binary_path: &Path, + placement: &LinuxWorldPlacementContext, +) -> Result { + let launcher_dir = std::env::temp_dir().join(format!( + "substrate-member-runtime-entry-{}", + uuid::Uuid::now_v7() + )); + let launcher = prepare_linux_world_entry_launcher(&launcher_dir, actual_binary_path, placement) + .map_err(|err| anyhow!(err.to_string()))?; + + Ok(PreparedMemberRuntimeLauncher { + launcher_path: launcher.launcher_path, + launcher_dir, + env: launcher.env, + }) +} + +fn frame_from_wrapper_event( + context: &MemberStreamContext, + binding: &SharedWorldBindingSnapshot, + span_id: &str, + wrapper_event: AgentWrapperEvent, + emitted_registered: &mut bool, + mode: MemberStreamMode, + agent_id: &str, +) -> Option { + Some(ExecuteStreamFrame::Event { + event: agent_event_from_wrapper_event( + context, + binding, + span_id, + wrapper_event, + emitted_registered, + mode, + agent_id, + )?, + }) +} + +#[allow(clippy::too_many_arguments)] +fn frames_from_completion( + context: &MemberStreamContext, + binding: &SharedWorldBindingSnapshot, + span_id: &str, + completion: std::result::Result, + cancel_signal: Option, + emitted_registered: &mut bool, + mode: MemberStreamMode, + agent_id: &str, +) -> Vec { + match completion { + Ok(completion) => { + let mut frames = Vec::new(); + if mode == MemberStreamMode::Bootstrap && !*emitted_registered { + if let Some(event) = registered_event_from_data( + context, + binding, + span_id, + completion.data.as_ref(), + agent_id, + ) { + *emitted_registered = true; + frames.push(ExecuteStreamFrame::Event { event }); + } + } + + frames.push(ExecuteStreamFrame::Exit { + exit: exit_code_from_status(&completion.status), + span_id: span_id.to_string(), + scopes_used: Vec::new(), + fs_diff: None, + process_telemetry: ProcessTelemetry::default(), + }); + frames + } + Err(AgentWrapperError::Backend { message }) if message == CANCELLED_MESSAGE => { + vec![ExecuteStreamFrame::Exit { + exit: cancel_exit_code(cancel_signal.as_deref()), + span_id: span_id.to_string(), + scopes_used: Vec::new(), + fs_diff: None, + process_telemetry: ProcessTelemetry::default(), + }] + } + Err(err) => vec![ExecuteStreamFrame::Error { + message: format!("member runtime failed: {err}"), + }], + } +} + +fn agent_event_from_wrapper_event( + context: &MemberStreamContext, + binding: &SharedWorldBindingSnapshot, + span_id: &str, + wrapper_event: AgentWrapperEvent, + emitted_registered: &mut bool, + mode: MemberStreamMode, + agent_id: &str, +) -> Option { + if mode == MemberStreamMode::Bootstrap { + if let Some(event) = registered_event_from_data( + context, + binding, + span_id, + wrapper_event.data.as_ref(), + agent_id, + ) { + *emitted_registered = true; + return Some(event); + } + } + + let mut event = match wrapper_event.kind { + AgentWrapperEventKind::Status => AgentEvent::message( + agent_id, + context.orchestration_session_id.clone(), + context.run_id.clone(), + MessageEventKind::Status, + wrapper_event + .message + .clone() + .unwrap_or_else(|| "member runtime status".to_string()), + ), + AgentWrapperEventKind::TextOutput => AgentEvent::message( + agent_id, + context.orchestration_session_id.clone(), + context.run_id.clone(), + MessageEventKind::TaskProgress, + wrapper_event + .text + .clone() + .unwrap_or_else(|| "member runtime output".to_string()), + ), + AgentWrapperEventKind::ToolCall | AgentWrapperEventKind::ToolResult => AgentEvent::message( + agent_id, + context.orchestration_session_id.clone(), + context.run_id.clone(), + MessageEventKind::TaskProgress, + wrapper_event + .message + .clone() + .unwrap_or_else(|| "member runtime tool activity".to_string()), + ), + AgentWrapperEventKind::Error => AgentEvent::alert( + agent_id, + context.orchestration_session_id.clone(), + context.run_id.clone(), + "agent_wrapper_error", + wrapper_event + .message + .clone() + .unwrap_or_else(|| "member runtime error".to_string()), + ), + AgentWrapperEventKind::Unknown => AgentEvent::message( + agent_id, + context.orchestration_session_id.clone(), + context.run_id.clone(), + MessageEventKind::TaskProgress, + "member runtime emitted an unknown event".to_string(), + ), + }; + + stamp_event_identity( + &mut event, + agent_id, + context, + binding, + span_id, + wrapper_event.channel, + ); + + if let Some(data) = wrapper_event.data { + if let Some(obj) = event.data.as_object_mut() { + obj.insert("uaa_event".to_string(), data); + obj.insert("protocol".to_string(), context.protocol.clone()); + } + } + + Some(event) +} + +fn registered_event_from_data( + context: &MemberStreamContext, + binding: &SharedWorldBindingSnapshot, + span_id: &str, + data: Option<&serde_json::Value>, + agent_id: &str, +) -> Option { + let data = data?; + if data.get("schema").and_then(serde_json::Value::as_str) != Some(SESSION_HANDLE_SCHEMA_V1) { + return None; + } + + let mut event = AgentEvent { + ts: chrono::Utc::now(), + kind: AgentEventKind::Registered, + data: data.clone(), + agent_id: agent_id.to_string(), + orchestration_session_id: context.orchestration_session_id.clone(), + run_id: context.run_id.clone(), + parent_run_id: None, + participant_id: Some(context.participant_id.clone()), + parent_participant_id: context.parent_participant_id.clone(), + resumed_from_participant_id: context.resumed_from_participant_id.clone(), + backend_id: Some(context.backend_id.clone()), + thread_id: None, + role: Some(MEMBER_ROLE.to_string()), + world_id: Some(binding.world_id.clone()), + world_generation: Some(binding.world_generation), + cmd_id: None, + span_id: Some(span_id.to_string()), + channel: None, + identity_tuple: None, + placement_posture: None, + project: None, + }; + event.set_pure_agent_telemetry_identity(agent_id.to_string()); + Some(event) +} + +fn stamp_event_identity( + event: &mut AgentEvent, + agent_id: &str, + context: &MemberStreamContext, + binding: &SharedWorldBindingSnapshot, + span_id: &str, + channel: Option, +) { + event.role = Some(MEMBER_ROLE.to_string()); + event.backend_id = Some(context.backend_id.clone()); + event.participant_id = Some(context.participant_id.clone()); + event.parent_participant_id = context.parent_participant_id.clone(); + event.resumed_from_participant_id = context.resumed_from_participant_id.clone(); + event.world_id = Some(binding.world_id.clone()); + event.world_generation = Some(binding.world_generation); + event.span_id = Some(span_id.to_string()); + event.set_channel(channel); + event.set_pure_agent_telemetry_identity(agent_id.to_string()); +} + +fn surfaced_uaa_session_id_from_data(data: Option<&serde_json::Value>) -> Option { + let data = data?; + for pointer in ["/internal/uaa_session_id", "/session/id"] { + if let Some(session_id) = data.pointer(pointer).and_then(serde_json::Value::as_str) { + let trimmed = session_id.trim(); + if !trimmed.is_empty() { + return Some(trimmed.to_string()); + } + } + } + None +} + +#[derive(Clone, Copy)] +struct RetainedMemberIdentity<'a> { + orchestration_session_id: &'a str, + orchestrator_participant_id: &'a str, + backend_id: &'a str, + world_id: &'a str, + world_generation: u64, +} + +impl<'a> RetainedMemberIdentity<'a> { + fn from_active(active: &'a ActiveMemberRuntime) -> Self { + Self { + orchestration_session_id: &active.orchestration_session_id, + orchestrator_participant_id: &active.orchestrator_participant_id, + backend_id: &active.backend_id, + world_id: &active.binding.world_id, + world_generation: active.binding.world_generation, + } + } +} + +fn validate_submit_turn_request( + req: &MemberTurnSubmitRequestV1, + retained: RetainedMemberIdentity<'_>, +) -> Result<()> { + if retained.orchestration_session_id != req.orchestration_session_id { + return Err(crate::service::BadRequestError::new(format!( + "member_turn_submit.orchestration_session_id mismatch (expected {}, got {})", + retained.orchestration_session_id, req.orchestration_session_id + )) + .into()); + } + if retained.orchestrator_participant_id != req.orchestrator_participant_id { + return Err(crate::service::BadRequestError::new(format!( + "member_turn_submit.orchestrator_participant_id mismatch (expected {}, got {})", + retained.orchestrator_participant_id, req.orchestrator_participant_id + )) + .into()); + } + if retained.backend_id != req.backend_id { + return Err(crate::service::BadRequestError::new(format!( + "member_turn_submit.backend_id mismatch (expected {}, got {})", + retained.backend_id, req.backend_id + )) + .into()); + } + if retained.world_id != req.world_id { + return Err(crate::service::BadRequestError::new(format!( + "member_turn_submit.world_id mismatch (expected {}, got {})", + retained.world_id, req.world_id + )) + .into()); + } + if retained.world_generation != req.world_generation { + return Err(crate::service::BadRequestError::new(format!( + "member_turn_submit.world_generation mismatch (expected {}, got {})", + retained.world_generation, req.world_generation + )) + .into()); + } + Ok(()) +} + +fn stream_response( + rx: tokio::sync::mpsc::UnboundedReceiver, +) -> Result { + let stream = UnboundedReceiverStream::new(rx).map(|frame| { + let mut payload = serde_json::to_vec(&frame).expect("serialize member runtime frame"); + payload.push(b'\n'); + Ok::(Bytes::from(payload)) + }); + + Response::builder() + .status(StatusCode::OK) + .header("content-type", "application/x-ndjson") + .body(boxed(StreamBody::new(stream))) + .map_err(|err| anyhow!("failed to build member runtime stream response: {err}")) +} + +fn validate_cancel_signal(sig: &str) -> Result<()> { + match sig.trim().to_ascii_uppercase().as_str() { + "INT" | "SIGINT" | "TERM" | "SIGTERM" | "HUP" | "SIGHUP" | "QUIT" | "SIGQUIT" => Ok(()), + _ => Err(anyhow!("unsupported execute cancellation signal: {sig}")), + } +} + +fn cancel_exit_code(sig: Option<&str>) -> i32 { + match sig.unwrap_or("INT").trim().to_ascii_uppercase().as_str() { + "HUP" | "SIGHUP" => 129, + "QUIT" | "SIGQUIT" => 131, + "TERM" | "SIGTERM" => 143, + _ => 130, + } +} + +fn exit_code_from_status(status: &std::process::ExitStatus) -> i32 { + #[cfg(unix)] + { + use std::os::unix::process::ExitStatusExt; + + if let Some(code) = status.code() { + return code; + } + if let Some(sig) = status.signal() { + return 128 + sig; + } + } + + status.code().unwrap_or(1) +} + +fn map_wrapper_error(err: AgentWrapperError) -> anyhow::Error { + match err { + AgentWrapperError::UnknownBackend { agent_kind } => { + anyhow!("unsupported member runtime backend: {agent_kind}") + } + AgentWrapperError::UnsupportedCapability { + agent_kind, + capability, + } => anyhow!("member runtime backend {agent_kind} does not support {capability}"), + AgentWrapperError::InvalidAgentKind { message } + | AgentWrapperError::InvalidRequest { message } + | AgentWrapperError::Backend { message } => anyhow!(message), + } +} + +impl ActiveMemberRuntime { + fn remember_uaa_session_id(&self, session_id: String) { + if let Ok(mut guard) = self.uaa_session_id.lock() { + *guard = Some(session_id); + } + } + + fn uaa_session_id(&self) -> Option { + self.uaa_session_id + .lock() + .ok() + .and_then(|guard| guard.clone()) + } + + fn bootstrap_last_signal(&self) -> Option { + self.bootstrap_last_signal + .lock() + .ok() + .and_then(|guard| guard.clone()) + } + + fn submit_context(&self, run_id: String) -> MemberStreamContext { + MemberStreamContext { + orchestration_session_id: self.orchestration_session_id.clone(), + run_id, + participant_id: self.participant_id.clone(), + parent_participant_id: self.parent_participant_id.clone(), + resumed_from_participant_id: self.resumed_from_participant_id.clone(), + backend_id: self.backend_id.clone(), + protocol: self.protocol.clone(), + } + } +} + +impl ActiveSubmittedTurn { + fn last_signal(&self) -> Option { + self.last_signal.lock().ok().and_then(|guard| guard.clone()) + } +} + +impl RetainedMemberKey { + fn from_active(active: &ActiveMemberRuntime) -> Self { + Self { + orchestration_session_id: active.orchestration_session_id.clone(), + world_generation: active.binding.world_generation, + backend_id: active.backend_id.clone(), + } + } + + fn from_submit(req: &MemberTurnSubmitRequestV1) -> Self { + Self { + orchestration_session_id: req.orchestration_session_id.clone(), + world_generation: req.world_generation, + backend_id: req.backend_id.clone(), + } + } +} + +fn duplicate_retained_member_error( + retained_key: &RetainedMemberKey, + existing_participant_id: &str, +) -> crate::service::BadRequestError { + crate::service::BadRequestError::new(format!( + "a retained world member is already active for orchestration_session_id {} world_generation {} backend_id {} (participant_id {})", + retained_key.orchestration_session_id, + retained_key.world_generation, + retained_key.backend_id, + existing_participant_id, + )) +} + +fn retained_slot_owner_mismatch_error( + retained_key: &RetainedMemberKey, + expected_participant_id: &str, + actual_participant_id: &str, +) -> crate::service::BadRequestError { + crate::service::BadRequestError::new(format!( + "member_turn_submit.participant_id mismatch for retained member orchestration_session_id {} world_generation {} backend_id {} (expected {}, got {})", + retained_key.orchestration_session_id, + retained_key.world_generation, + retained_key.backend_id, + expected_participant_id, + actual_participant_id, + )) +} + +fn missing_retained_slot_error( + retained_key: &RetainedMemberKey, +) -> crate::service::BadRequestError { + crate::service::BadRequestError::new(format!( + "member_turn_submit retained member is not active for orchestration_session_id {} world_generation {} backend_id {}", + retained_key.orchestration_session_id, + retained_key.world_generation, + retained_key.backend_id, + )) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn sample_submit_turn_request() -> MemberTurnSubmitRequestV1 { + MemberTurnSubmitRequestV1 { + schema_version: 1, + orchestration_session_id: "orch_123".to_string(), + participant_id: "ash_member".to_string(), + orchestrator_participant_id: "ash_orchestrator".to_string(), + backend_id: "cli:codex".to_string(), + run_id: "run_turn".to_string(), + world_id: "world_123".to_string(), + world_generation: 7, + prompt: "continue".to_string(), + } + } + + fn sample_retained_identity() -> RetainedMemberIdentity<'static> { + RetainedMemberIdentity { + orchestration_session_id: "orch_123", + orchestrator_participant_id: "ash_orchestrator", + backend_id: "cli:codex", + world_id: "world_123", + world_generation: 7, + } + } + + #[test] + fn surfaced_uaa_session_id_prefers_internal_then_session_id() { + let payload = json!({ + "internal": { + "uaa_session_id": "uaa_internal" + }, + "session": { + "id": "uaa_session" + } + }); + assert_eq!( + surfaced_uaa_session_id_from_data(Some(&payload)).as_deref(), + Some("uaa_internal") + ); + + let payload = json!({ + "session": { + "id": "uaa_session" + } + }); + assert_eq!( + surfaced_uaa_session_id_from_data(Some(&payload)).as_deref(), + Some("uaa_session") + ); + } + + #[test] + fn validate_submit_turn_request_accepts_matching_retained_identity() { + validate_submit_turn_request(&sample_submit_turn_request(), sample_retained_identity()) + .expect("matching retained member identity should validate"); + } + + type SubmitTurnDriftCase = ( + &'static str, + fn(&mut MemberTurnSubmitRequestV1), + &'static str, + ); + + #[test] + fn validate_submit_turn_request_rejects_retained_identity_drift() { + let cases: [SubmitTurnDriftCase; 5] = [ + ( + "orchestration_session_id", + |req| req.orchestration_session_id = "orch_999".to_string(), + "member_turn_submit.orchestration_session_id mismatch", + ), + ( + "orchestrator_participant_id", + |req| req.orchestrator_participant_id = "ash_orchestrator_other".to_string(), + "member_turn_submit.orchestrator_participant_id mismatch", + ), + ( + "backend_id", + |req| req.backend_id = "cli:anthropic".to_string(), + "member_turn_submit.backend_id mismatch", + ), + ( + "world_id", + |req| req.world_id = "world_999".to_string(), + "member_turn_submit.world_id mismatch", + ), + ( + "world_generation", + |req| req.world_generation = 9, + "member_turn_submit.world_generation mismatch", + ), + ]; + + for (field, mutate, expected) in cases { + let mut req = sample_submit_turn_request(); + mutate(&mut req); + + let err = match validate_submit_turn_request(&req, sample_retained_identity()) { + Ok(()) => panic!("expected {field} drift to be rejected"), + Err(err) => err, + }; + assert!( + err.to_string().contains(expected), + "expected {field} drift to mention {expected}, got: {err}" + ); + } + } + + #[test] + fn retained_member_key_uses_session_generation_and_backend_id() { + let req = sample_submit_turn_request(); + + assert_eq!( + RetainedMemberKey::from_submit(&req), + RetainedMemberKey { + orchestration_session_id: "orch_123".to_string(), + world_generation: 7, + backend_id: "cli:codex".to_string(), + } + ); + } + + #[test] + fn duplicate_retained_member_error_mentions_backend_slot_identity() { + let err = duplicate_retained_member_error( + &RetainedMemberKey { + orchestration_session_id: "orch_123".to_string(), + world_generation: 7, + backend_id: "cli:codex".to_string(), + }, + "ash_member_existing", + ); + + assert!( + err.to_string().contains("backend_id cli:codex"), + "unexpected error: {err}" + ); + assert!( + err.to_string() + .contains("participant_id ash_member_existing"), + "unexpected error: {err}" + ); + } + + #[test] + fn retained_slot_owner_mismatch_error_mentions_expected_participant() { + let err = retained_slot_owner_mismatch_error( + &RetainedMemberKey { + orchestration_session_id: "orch_123".to_string(), + world_generation: 7, + backend_id: "cli:codex".to_string(), + }, + "ash_member_existing", + "ash_member_other", + ); + + assert!( + err.to_string() + .contains("member_turn_submit.participant_id mismatch"), + "unexpected error: {err}" + ); + assert!( + err.to_string() + .contains("expected ash_member_existing, got ash_member_other"), + "unexpected error: {err}" + ); + } + + #[test] + fn find_submit_target_rejects_participant_id_drift_for_retained_slot() { + let manager = MemberRuntimeManager::new(); + let retained_key = RetainedMemberKey { + orchestration_session_id: "orch_123".to_string(), + world_generation: 7, + backend_id: "cli:codex".to_string(), + }; + manager + .active_members + .write() + .expect("member runtime registry lock poisoned") + .by_retained_key + .insert(retained_key, "ash_member_existing".to_string()); + + let mut req = sample_submit_turn_request(); + req.participant_id = "ash_member_other".to_string(); + + let err = match manager.find_submit_target(&req) { + Ok(_) => panic!("participant drift should be rejected"), + Err(err) => err, + }; + assert!( + err.to_string() + .contains("member_turn_submit.participant_id mismatch"), + "unexpected error: {err}" + ); + assert!( + err.to_string() + .contains("expected ash_member_existing, got ash_member_other"), + "unexpected error: {err}" + ); + } +} diff --git a/crates/world-agent/src/pty.rs b/crates/world-agent/src/pty.rs index ba92b28a6..6c1529b1a 100644 --- a/crates/world-agent/src/pty.rs +++ b/crates/world-agent/src/pty.rs @@ -7,13 +7,14 @@ use crate::request_routing::resolve_snapshot_routing; use crate::service::WorldAgentService; #[cfg(target_os = "linux")] use crate::service::{ - apply_full_isolation_helper_env, is_full_isolation, resolve_landlock_allowlist_paths, - resolve_project_dir, resolve_project_write_allowlist_prefixes, WORLD_FS_MODE_ENV, + apply_full_isolation_helper_env, build_world_spec, is_full_isolation, + resolve_landlock_allowlist_paths, resolve_project_dir, + resolve_project_write_allowlist_prefixes, resolve_shared_world_binding, WORLD_FS_MODE_ENV, WORLD_FS_WRITE_ALLOWLIST_ENV, }; #[cfg(target_os = "linux")] use agent_api_types::PolicyResolutionModeV1; -use agent_api_types::{PolicySnapshotV3, WorldNetworkRoutingV1}; +use agent_api_types::{PolicySnapshotV3, SharedWorldOwnerSpec, WorldNetworkRoutingV1}; use axum::extract::ws::{Message, WebSocket}; #[cfg(target_os = "linux")] use base64::{engine::general_purpose::STANDARD as BASE64, Engine}; @@ -109,12 +110,14 @@ pub enum ClientMessage { #[serde(rename = "start")] Start { cmd: String, - cwd: PathBuf, - env: HashMap, + cwd: Box, + env: Box>, #[serde(default)] policy_snapshot: Box>, #[serde(default)] - world_network: Option, + shared_world: Box>, + #[serde(default)] + world_network: Box>, span_id: Option, cols: u16, rows: u16, @@ -167,6 +170,8 @@ enum PersistentClientMessage { env: HashMap, policy_snapshot: Box, #[serde(default)] + shared_world: Option, + #[serde(default)] world_network: Option, cols: u16, rows: u16, @@ -208,6 +213,8 @@ enum PersistentServerMessage { world_id: String, cwd: PathBuf, protocol_version: u32, + #[serde(skip_serializing_if = "Option::is_none")] + shared_world: Option, }, #[cfg(target_os = "linux")] Stdout { data_b64: String }, @@ -799,6 +806,7 @@ async fn handle_persistent_session( cwd, env, policy_snapshot, + shared_world, world_network, cols, rows, @@ -807,6 +815,7 @@ async fn handle_persistent_session( cwd, env, policy_snapshot, + shared_world, world_network, cols, rows, @@ -846,6 +855,7 @@ async fn handle_persistent_session( requested_cwd, mut session_env, policy_snapshot, + shared_world, world_network, cols, rows, @@ -956,6 +966,7 @@ async fn handle_persistent_session( &session_env, &ready_cwd, &policy_snapshot, + shared_world.as_ref(), world_network.as_ref(), ) { Ok(world) => world, @@ -995,6 +1006,7 @@ async fn handle_persistent_session( world_id: world.world_id.clone(), cwd: ready_cwd.clone(), protocol_version: 1, + shared_world: world.shared_world.clone(), }) .is_err() { @@ -1190,6 +1202,7 @@ async fn handle_persistent_session( let world_handle = world_api::WorldHandle { id: world.world_id.clone(), + shared_binding: None, }; if let Err(e) = service.refresh_session_network_filter(&world_handle) @@ -1447,6 +1460,7 @@ struct PersistentWorldContext { isolate_network: bool, netns_name: Option, cgroup_path: Option, + shared_world: Option, base_env: HashMap, } @@ -1560,10 +1574,9 @@ fn prepare_persistent_world_context( session_env: &HashMap, ready_cwd: &std::path::Path, policy_snapshot: &PolicySnapshotV3, + shared_world: Option<&SharedWorldOwnerSpec>, world_network: Option<&WorldNetworkRoutingV1>, ) -> Result { - use world_api::{ResourceLimits, WorldSpec}; - let project_dir = resolve_project_dir(Some(session_env), Some(ready_cwd)).map_err(|e| e.to_string())?; @@ -1578,22 +1591,22 @@ fn prepare_persistent_world_context( isolate_network, ); - let spec = WorldSpec { - reuse_session: true, - isolate_network, - limits: ResourceLimits::default(), - enable_preload: false, - allowed_domains: resolved.world_network.allowed_domains.clone(), - project_dir: project_dir.clone(), - always_isolate: true, + let spec = build_world_spec( + project_dir.clone(), + true, fs_mode, - }; + isolate_network, + resolved.world_network.allowed_domains.clone(), + shared_world, + ); let (world, merged_dir) = service.ensure_session_overlay_root(&spec).map_err(|e| { service.record_last_netfilter_failure_for_error(isolate_network, &e); format!("Failed to prepare world overlay: {e}") })?; service.clear_last_netfilter_failure_on_success(isolate_network); + let shared_world = resolve_shared_world_binding(shared_world, &world) + .map_err(|e| format!("Failed to validate shared world proof: {e}"))?; let ns_name = format!("substrate-{}", world.id); let ns_path = format!("/var/run/netns/{ns_name}"); @@ -1632,6 +1645,7 @@ fn prepare_persistent_world_context( isolate_network, netns_name, cgroup_path, + shared_world, base_env, }) } @@ -2256,12 +2270,16 @@ async fn handle_legacy_start( cwd, env, policy_snapshot, + shared_world, world_network, span_id, cols, rows, }) => { - let mut env = env; + let cwd = *cwd; + let mut env = *env; + let shared_world = *shared_world; + let world_network = *world_network; ensure_xdg_dirs(&mut env); let policy_snapshot = *policy_snapshot; info!( @@ -2277,6 +2295,7 @@ async fn handle_legacy_start( cwd, env, policy_snapshot, + shared_world, world_network, span_id, cols, @@ -2305,7 +2324,8 @@ async fn handle_legacy_start( } }; - let (cmd, cwd, env_map, policy_snapshot, _world_network, _span_id, cols, rows) = start_msg; + let (cmd, cwd, env_map, policy_snapshot, shared_world, _world_network, _span_id, cols, rows) = + start_msg; #[cfg(not(target_os = "linux"))] let _policy_snapshot = policy_snapshot; #[cfg(target_os = "linux")] @@ -2351,7 +2371,18 @@ async fn handle_legacy_start( let fs_strategy_meta: Option; #[cfg(target_os = "linux")] { - use world_api::{ResourceLimits, WorldSpec}; + if shared_world.is_some() { + let _ = send_ws_message( + &tx, + &ServerMessage::Error { + message: "Legacy PTY start does not support shared_world; use start_session" + .to_string(), + }, + ) + .await; + return; + } + let project_dir = match resolve_project_dir(Some(&env), Some(&cwd)) { Ok(dir) => dir, Err(err) => { @@ -2457,16 +2488,14 @@ async fn handle_legacy_start( } } - let spec = WorldSpec { - reuse_session: true, + let spec = build_world_spec( + project_dir.clone(), + true, + fs_mode, isolate_network, - limits: ResourceLimits::default(), - enable_preload: false, allowed_domains, - project_dir: project_dir.clone(), - always_isolate: true, - fs_mode, - }; + None, + ); let merged_dir: PathBuf; match service.ensure_session_overlay_root(&spec) { @@ -2863,10 +2892,11 @@ mod tests { fn test_client_message_start_serialization() { let msg = ClientMessage::Start { cmd: "echo hi".into(), - cwd: std::env::current_dir().unwrap(), - env: HashMap::new(), + cwd: Box::new(std::env::current_dir().unwrap()), + env: Box::new(HashMap::new()), policy_snapshot: Box::new(None), - world_network: None, + shared_world: Box::new(None), + world_network: Box::new(None), span_id: Some("spn_test".into()), cols: 80, rows: 24, diff --git a/crates/world-agent/src/service.rs b/crates/world-agent/src/service.rs index 9417618c9..953d2d3a5 100644 --- a/crates/world-agent/src/service.rs +++ b/crates/world-agent/src/service.rs @@ -10,10 +10,10 @@ use agent_api_types::PendingDiffBucketV1; use agent_api_types::WorldFsEntryTypeV1; use agent_api_types::{ Budget, ExecuteCancelRequestV1, ExecuteCancelResponseV1, ExecuteRequest, ExecuteResponse, - GatewayLifecycleRequestV1, GatewayLifecycleResponseV1, PendingDiffClearRequestV1, - PendingDiffClearResponseV1, PendingDiffReconcileRequestV1, PendingDiffReconcileResponseV1, - PendingDiffRecordV1, PendingDiffRequestV1, ProcessTelemetry, WorldFsReadRequestV1, - WorldFsReadResponseV1, WorldNetworkRoutingV1, + GatewayLifecycleRequestV1, GatewayLifecycleResponseV1, MemberTurnSubmitRequestV1, + PendingDiffClearRequestV1, PendingDiffClearResponseV1, PendingDiffReconcileRequestV1, + PendingDiffReconcileResponseV1, PendingDiffRecordV1, PendingDiffRequestV1, ProcessTelemetry, + WorldFsReadRequestV1, WorldFsReadResponseV1, WorldNetworkRoutingV1, }; #[cfg(target_os = "linux")] use anyhow::Context; @@ -51,15 +51,23 @@ use tokio::task; use tokio_stream::wrappers::UnboundedReceiverStream; #[cfg(target_os = "linux")] use world::stream::{install_stream_sink, StreamKind, StreamSink}; -use world_api::{WorldBackend, WorldHandle, WorldSpec}; +use world_api::{ + MemberDispatchRequestV1 as BackendMemberDispatchRequestV1, + MemberRuntimeBackendKindV1 as BackendMemberRuntimeBackendKindV1, + ResolvedMemberRuntimeDescriptorV1 as BackendResolvedMemberRuntimeDescriptorV1, + SharedWorldBindingSnapshot, SharedWorldBindingState, SharedWorldOwnerSpec, WorldBackend, + WorldHandle, WorldReuseMode, WorldSpec, +}; use crate::enforcement_plan; #[cfg(target_os = "linux")] use crate::gateway_runtime::{ resolve_gateway_backend_binding, unavailable_response as gateway_unavailable_response_impl, GatewayControlSettings, GatewayRuntimeFailure, GatewayRuntimeManager, - GatewayRuntimeStartContext, + GatewayRuntimeStartContext, LinuxWorldPlacementContext, }; +#[cfg(target_os = "linux")] +use crate::member_runtime::MemberRuntimeManager; use crate::request_routing::resolve_snapshot_routing; pub(crate) const ANCHOR_MODE_ENV: &str = "SUBSTRATE_ANCHOR_MODE"; @@ -165,6 +173,13 @@ fn resolve_landlock_helper_src() -> Option { } } + #[cfg(test)] + if let Some(exe) = exe.as_ref() { + if exe.is_file() { + return Some(exe.display().to_string()); + } + } + let exe = exe?; resolve_landlock_helper_src_from_exe(&exe).map(|p| p.display().to_string()) } @@ -178,6 +193,8 @@ pub struct WorldAgentService { #[cfg(target_os = "linux")] gateway_runtime: Arc, #[cfg(target_os = "linux")] + member_runtime: Arc, + #[cfg(target_os = "linux")] pending_diff_origin: Arc>>, #[allow(dead_code)] worlds: Arc>>, @@ -247,6 +264,7 @@ impl WorldAgentService { backend, linux_backend, gateway_runtime: Arc::new(GatewayRuntimeManager::new()), + member_runtime: Arc::new(MemberRuntimeManager::new()), pending_diff_origin: Arc::new(RwLock::new(HashMap::new())), worlds: Arc::new(RwLock::new(HashMap::new())), budgets: Arc::new(RwLock::new(HashMap::new())), @@ -410,6 +428,7 @@ impl WorldAgentService { pub(crate) fn note_pty_pending_diff(&self, world_id: &str) { let world = WorldHandle { id: world_id.to_string(), + shared_binding: None, }; let (_started_at, diff) = match self.linux_backend.pending_diff(&world) { Ok(v) => v, @@ -430,6 +449,13 @@ impl WorldAgentService { /// Execute a command with budget tracking. pub async fn execute(&self, req: ExecuteRequest) -> Result { + if req.member_dispatch.is_some() { + return Err(BadRequestError::new( + "member_dispatch requires POST /v1/execute/stream".to_string(), + ) + .into()); + } + // Validate agent_id if req.agent_id.is_empty() { anyhow::bail!("agent_id is required for API calls"); @@ -484,21 +510,8 @@ impl WorldAgentService { let host_visible = !isolation_full; let empty_env: HashMap = HashMap::new(); let guard_env = req.env.as_ref().unwrap_or(&empty_env); - if let Some(deny) = - crate::world_exec_guard::check_command(&req.cmd, &cwd, guard_env, host_visible) - { - let span_id = format!("spn_{}", uuid::Uuid::now_v7()); - let message = crate::world_exec_guard::deny_message(&deny); - return Ok(ExecuteResponse { - exit: 5, - span_id, - stdout_b64: BASE64.encode(b""), - stderr_b64: BASE64.encode(message.as_bytes()), - scopes_used: Vec::new(), - fs_diff: None, - process_telemetry: ProcessTelemetry::default(), - }); - } + let guard_deny = + crate::world_exec_guard::check_command(&req.cmd, &cwd, guard_env, host_visible); // Create world spec from request let spec = build_world_spec( @@ -507,6 +520,7 @@ impl WorldAgentService { fs_mode, isolate_network, allowed_domains, + req.shared_world.as_ref(), ); // Ensure world exists @@ -518,6 +532,22 @@ impl WorldAgentService { return Err(anyhow::anyhow!("Failed to ensure session world")); } }; + let shared_world = resolve_shared_world_binding(req.shared_world.as_ref(), &world)?; + + if let Some(deny) = guard_deny { + let span_id = format!("spn_{}", uuid::Uuid::now_v7()); + let message = crate::world_exec_guard::deny_message(&deny); + return Ok(ExecuteResponse { + exit: 5, + span_id, + stdout_b64: BASE64.encode(b""), + stderr_b64: BASE64.encode(message.as_bytes()), + scopes_used: Vec::new(), + fs_diff: None, + shared_world, + process_telemetry: ProcessTelemetry::default(), + }); + } // Prepare execution request let mut env_map = req.env.unwrap_or_default(); @@ -552,6 +582,11 @@ impl WorldAgentService { env: env_map, pty: req.pty, span_id: None, + shared_world: req.shared_world.clone(), + member_dispatch: req + .member_dispatch + .as_ref() + .map(convert_member_dispatch_request), }; // Execute command @@ -588,6 +623,7 @@ impl WorldAgentService { stderr_b64: BASE64.encode(result.stderr), scopes_used: result.scopes_used, fs_diff: result.fs_diff, + shared_world, process_telemetry: result.process_telemetry, }) } @@ -623,6 +659,7 @@ impl WorldAgentService { fs_mode, isolate_network, allowed_domains, + None, ); let world = match self.backend.ensure_session(&spec) { @@ -730,6 +767,7 @@ impl WorldAgentService { fs_mode, isolate_network, allowed_domains, + None, ); let world = match self.backend.ensure_session(&spec) { @@ -808,6 +846,7 @@ impl WorldAgentService { fs_mode, isolate_network, allowed_domains, + None, ); let world = match self.backend.ensure_session(&spec) { @@ -948,6 +987,7 @@ impl WorldAgentService { fs_mode, isolate_network, allowed_domains, + None, ); let world = match self.backend.ensure_session(&spec) { @@ -1184,12 +1224,70 @@ impl WorldAgentService { self.record_doctor_request_context(policy_resolution_mode, isolate_network); let always_isolate = should_always_isolate(&req); + let shared_owner_spec = requested_shared_world_owner_spec(&req); + + let spec = build_world_spec( + project_dir.clone(), + always_isolate, + fs_mode, + isolate_network, + allowed_domains, + shared_owner_spec.as_ref(), + ); + + let world = match self.backend.ensure_session(&spec) { + Ok(w) => w, + Err(e) => { + self.record_last_netfilter_failure_for_error(isolate_network, &e); + tracing::error!(error = %e, error_debug = ?e, "ensure_session failed"); + anyhow::bail!("Failed to ensure session world"); + } + }; + let shared_world = resolve_shared_world_binding(shared_owner_spec.as_ref(), &world)?; + + if let Some(dispatch) = req.member_dispatch.clone() { + let placement = self.resolve_authoritative_member_placement_context( + &dispatch, + world, + shared_world.as_ref(), + &project_dir, + &cwd, + isolation_full, + )?; + let env_map = build_member_runtime_launch_env( + req.env.clone().unwrap_or_default(), + MemberRuntimeLaunchEnvContext { + placement_root: &placement.placement_root, + isolation_full, + write_allowlist_prefixes: &write_allowlist_prefixes, + landlock_discover_paths: &landlock_discover_paths, + landlock_read_paths: &landlock_read_paths, + landlock_write_paths: &landlock_write_paths, + enforcement_plan_b64: enforcement_plan_b64.as_deref(), + }, + ); + let span_id = format!("spn_{}", uuid::Uuid::now_v7()); + let launch_placement = placement.launch_placement(); + return self + .member_runtime + .launch( + req.agent_id.clone(), + env_map, + span_id, + dispatch, + placement.binding, + launch_placement, + ) + .await; + } + let host_visible = !isolation_full; let empty_env: HashMap = HashMap::new(); let guard_env = req.env.as_ref().unwrap_or(&empty_env); - if let Some(deny) = - crate::world_exec_guard::check_command(&req.cmd, &cwd, guard_env, host_visible) - { + let guard_deny = + crate::world_exec_guard::check_command(&req.cmd, &cwd, guard_env, host_visible); + + if let Some(deny) = guard_deny { let span_id = format!("spn_{}", uuid::Uuid::now_v7()); let message = crate::world_exec_guard::deny_message(&deny); let frames = vec![ @@ -1223,23 +1321,6 @@ impl WorldAgentService { return Ok(response); } - let spec = build_world_spec( - project_dir, - always_isolate, - fs_mode, - isolate_network, - allowed_domains, - ); - - let world = match self.backend.ensure_session(&spec) { - Ok(w) => w, - Err(e) => { - self.record_last_netfilter_failure_for_error(isolate_network, &e); - tracing::error!(error = %e, error_debug = ?e, "ensure_session failed"); - anyhow::bail!("Failed to ensure session world"); - } - }; - let mut exec_req = world_api::ExecRequest { cmd: req.cmd.clone(), cwd: cwd.clone(), @@ -1273,6 +1354,11 @@ impl WorldAgentService { }, pty: false, span_id: None, + shared_world: shared_owner_spec.clone(), + member_dispatch: req + .member_dispatch + .as_ref() + .map(convert_member_dispatch_request), }; let span_id = format!("spn_{}", uuid::Uuid::now_v7()); @@ -1288,6 +1374,7 @@ impl WorldAgentService { #[cfg(target_os = "linux")] let service = self.clone(); let agent_id = req.agent_id.clone(); + let shared_world_for_events = shared_world.clone(); let span_id_for_cleanup = span_id.clone(); task::spawn_blocking(move || { let sink = Arc::new(StreamingSink::new(tx.clone())); @@ -1318,14 +1405,27 @@ impl WorldAgentService { ts: chrono::Utc::now(), agent_id: agent_id.clone(), kind: AgentEventKind::Status, - orchestration_session_id: span_id.clone(), + orchestration_session_id: shared_world_for_events + .as_ref() + .map(|binding| binding.orchestration_session_id.clone()) + .unwrap_or_else(|| span_id.clone()), run_id: span_id.clone(), parent_run_id: None, + participant_id: None, + parent_participant_id: None, + resumed_from_participant_id: None, backend_id: None, thread_id: None, role: None, - world_id: Some(world.id.clone()), - world_generation: None, + world_id: Some( + shared_world_for_events + .as_ref() + .map(|binding| binding.world_id.clone()) + .unwrap_or_else(|| world.id.clone()), + ), + world_generation: shared_world_for_events + .as_ref() + .map(|binding| binding.world_generation), cmd_id: None, span_id: Some(span_id.clone()), channel: None, @@ -1381,6 +1481,23 @@ impl WorldAgentService { anyhow::bail!("World agent streaming is only supported on Linux"); } + #[cfg(target_os = "linux")] + pub async fn submit_member_turn_stream( + &self, + req: MemberTurnSubmitRequestV1, + ) -> Result { + req.validate().map_err(BadRequestError::new)?; + self.member_runtime.submit_turn(req).await + } + + #[cfg(not(target_os = "linux"))] + pub async fn submit_member_turn_stream( + &self, + _req: MemberTurnSubmitRequestV1, + ) -> Result { + anyhow::bail!("World member turn submission is only supported on Linux"); + } + #[cfg(target_os = "linux")] pub async fn execute_cancel( &self, @@ -1390,6 +1507,17 @@ impl WorldAgentService { return Err(BadRequestError::new("span_id is required".to_string()).into()); } + let delivered = self + .member_runtime + .cancel(&req.span_id, &req.sig) + .with_context(|| format!("failed to cancel member runtime span {}", req.span_id))?; + if delivered { + return Ok(ExecuteCancelResponseV1 { + schema_version: 1, + delivered: true, + }); + } + let mut delivered = false; for _ in 0..80 { delivered = @@ -1461,6 +1589,7 @@ impl WorldAgentService { policy_inputs.fs_mode, policy_inputs.isolate_network, policy_inputs.allowed_domains, + None, ); let control = GatewayControlSettings::from_request_env(env_ref).map_err(gateway_runtime_error)?; @@ -1485,6 +1614,39 @@ impl WorldAgentService { }) } + #[cfg(target_os = "linux")] + fn resolve_authoritative_member_placement_context( + &self, + dispatch: &agent_api_types::MemberDispatchRequestV1, + world: WorldHandle, + authoritative_binding: Option<&SharedWorldBindingSnapshot>, + project_dir: &Path, + cwd: &Path, + isolation_full: bool, + ) -> Result { + let binding = validate_member_dispatch_binding(dispatch, &world, authoritative_binding)?; + let cgroup_path = self + .session_cgroup_path(&world) + .with_context(|| format!("failed to resolve session cgroup for world {}", world.id))?; + let placement_root = if isolation_full { + self.linux_backend + .ensure_overlay_root(&world) + .with_context(|| format!("failed to resolve overlay root for world {}", world.id))? + } else { + project_dir.to_path_buf() + }; + let effective_cwd = + resolve_member_runtime_effective_cwd(project_dir, &placement_root, cwd)?; + + Ok(AuthoritativeMemberPlacementContext { + world, + binding, + placement_root, + effective_cwd, + cgroup_path, + }) + } + #[cfg(target_os = "linux")] fn resolve_gateway_runtime_binding( &self, @@ -1620,6 +1782,28 @@ struct PreparedGatewayRuntimeRequest { integrated_auth: Option, } +#[cfg(target_os = "linux")] +#[derive(Debug, Clone)] +struct AuthoritativeMemberPlacementContext { + #[allow(dead_code)] + world: WorldHandle, + binding: SharedWorldBindingSnapshot, + placement_root: PathBuf, + effective_cwd: PathBuf, + cgroup_path: PathBuf, +} + +#[cfg(target_os = "linux")] +impl AuthoritativeMemberPlacementContext { + fn launch_placement(&self) -> LinuxWorldPlacementContext { + LinuxWorldPlacementContext { + working_dir: self.effective_cwd.clone(), + cgroup_path: self.cgroup_path.clone(), + require_cgroup_attach: true, + } + } +} + #[cfg(target_os = "linux")] #[derive(Clone, Copy)] enum GatewayRuntimeBindingMode { @@ -1724,6 +1908,90 @@ fn gateway_runtime_error(err: GatewayRuntimeFailure) -> anyhow::Error { anyhow!(err.to_string()) } +#[cfg(target_os = "linux")] +struct MemberRuntimeLaunchEnvContext<'a> { + placement_root: &'a Path, + isolation_full: bool, + write_allowlist_prefixes: &'a [String], + landlock_discover_paths: &'a [String], + landlock_read_paths: &'a [String], + landlock_write_paths: &'a [String], + enforcement_plan_b64: Option<&'a str>, +} + +#[cfg(target_os = "linux")] +fn build_member_runtime_launch_env( + mut env_map: HashMap, + ctx: MemberRuntimeLaunchEnvContext<'_>, +) -> HashMap { + env_map.insert( + WORLD_PROJECT_DIR_OVERRIDE_ENV.to_string(), + ctx.placement_root.display().to_string(), + ); + env_map.insert( + WORLD_FS_ISOLATION_ENV.to_string(), + if ctx.isolation_full { + "full".to_string() + } else { + "workspace".to_string() + }, + ); + if ctx.isolation_full && !ctx.write_allowlist_prefixes.is_empty() { + env_map.insert( + WORLD_FS_WRITE_ALLOWLIST_ENV.to_string(), + ctx.write_allowlist_prefixes.join("\n"), + ); + } + + let landlock_supported = world::landlock::detect_support().supported; + if ctx.isolation_full { + apply_full_isolation_helper_env( + &mut env_map, + landlock_supported, + ctx.landlock_discover_paths, + ctx.landlock_read_paths, + ctx.landlock_write_paths, + ctx.enforcement_plan_b64, + ); + } + + env_map +} + +#[cfg(target_os = "linux")] +fn resolve_member_runtime_effective_cwd( + project_dir: &Path, + placement_root: &Path, + requested_cwd: &Path, +) -> Result { + let requested_cwd = + std::fs::canonicalize(requested_cwd).unwrap_or_else(|_| requested_cwd.to_path_buf()); + let project_dir = + std::fs::canonicalize(project_dir).unwrap_or_else(|_| project_dir.to_path_buf()); + + if requested_cwd == project_dir { + return Ok(placement_root.to_path_buf()); + } + + let relative = requested_cwd.strip_prefix(&project_dir).map_err(|_| { + anyhow!( + "member runtime cwd {} is outside the authoritative project root {}", + requested_cwd.display(), + project_dir.display() + ) + })?; + let effective_cwd = placement_root.join(relative); + if !effective_cwd.is_dir() { + anyhow::bail!( + "member runtime cwd {} is missing from authoritative placement root {}", + requested_cwd.display(), + placement_root.display() + ); + } + + Ok(effective_cwd) +} + #[cfg(test)] mod pending_diff_id_tests { use super::WorldAgentService; @@ -1890,6 +2158,7 @@ mod gateway_runtime_binding_tests { WorldFsMode::Writable, false, vec!["b.example.com".to_string(), "a.example.com".to_string()], + None, ); let first = non_isolated_gateway_runtime_id(&project_dir, &world_spec, "cli:codex"); world_spec.allowed_domains.reverse(); @@ -1906,6 +2175,7 @@ mod gateway_runtime_binding_tests { WorldFsMode::Writable, false, vec!["example.com".to_string()], + None, ); let changed_project = non_isolated_gateway_runtime_id(Path::new("/tmp/other"), &base, "cli:codex"); @@ -1917,6 +2187,7 @@ mod gateway_runtime_binding_tests { WorldFsMode::Writable, false, vec!["example.com".to_string()], + None, ), "cli:codex", ); @@ -1928,6 +2199,7 @@ mod gateway_runtime_binding_tests { WorldFsMode::Writable, false, vec!["other.example.com".to_string()], + None, ), "cli:codex", ); @@ -2266,15 +2538,20 @@ fn resolve_policy_inputs( )) } -fn build_world_spec( +pub(crate) fn build_world_spec( project_dir: PathBuf, always_isolate: bool, fs_mode: WorldFsMode, isolate_network: bool, allowed_domains: Vec, + shared_world: Option<&SharedWorldOwnerSpec>, ) -> WorldSpec { WorldSpec { reuse_session: true, + reuse_mode: shared_world + .cloned() + .map(WorldReuseMode::SharedOrchestration) + .unwrap_or(WorldReuseMode::GenericCompatible), isolate_network, limits: world_api::ResourceLimits::default(), enable_preload: false, @@ -2285,6 +2562,156 @@ fn build_world_spec( } } +#[cfg(target_os = "linux")] +fn requested_shared_world_owner_spec(req: &ExecuteRequest) -> Option { + req.shared_world.clone().or_else(|| { + req.member_dispatch + .as_ref() + .map(|dispatch| SharedWorldOwnerSpec { + orchestration_session_id: dispatch.orchestration_session_id.clone(), + action: world_api::SharedWorldOwnerAction::AttachOrCreate, + }) + }) +} + +fn convert_member_dispatch_request( + dispatch: &agent_api_types::MemberDispatchRequestV1, +) -> BackendMemberDispatchRequestV1 { + BackendMemberDispatchRequestV1 { + schema_version: dispatch.schema_version, + orchestration_session_id: dispatch.orchestration_session_id.clone(), + participant_id: dispatch.participant_id.clone(), + orchestrator_participant_id: dispatch.orchestrator_participant_id.clone(), + parent_participant_id: dispatch.parent_participant_id.clone(), + resumed_from_participant_id: dispatch.resumed_from_participant_id.clone(), + backend_id: dispatch.backend_id.clone(), + protocol: dispatch.protocol.clone(), + run_id: dispatch.run_id.clone(), + world_id: dispatch.world_id.clone(), + world_generation: dispatch.world_generation, + initial_prompt: dispatch.initial_prompt.clone(), + resolved_runtime: BackendResolvedMemberRuntimeDescriptorV1 { + backend_kind: convert_member_runtime_backend_kind( + dispatch.resolved_runtime.backend_kind, + ), + binary_path: dispatch.resolved_runtime.binary_path.clone(), + }, + } +} + +fn convert_member_runtime_backend_kind( + backend_kind: agent_api_types::MemberRuntimeBackendKindV1, +) -> BackendMemberRuntimeBackendKindV1 { + match backend_kind { + agent_api_types::MemberRuntimeBackendKindV1::Codex => { + BackendMemberRuntimeBackendKindV1::Codex + } + agent_api_types::MemberRuntimeBackendKindV1::ClaudeCode => { + BackendMemberRuntimeBackendKindV1::ClaudeCode + } + } +} + +#[cfg(target_os = "linux")] +fn validate_member_dispatch_binding( + dispatch: &agent_api_types::MemberDispatchRequestV1, + world: &WorldHandle, + authoritative_binding: Option<&SharedWorldBindingSnapshot>, +) -> Result { + let binding = authoritative_binding + .cloned() + .or_else(|| world.shared_binding.clone()) + .ok_or_else(|| { + BadRequestError::new(format!( + "member_dispatch requires authoritative shared world binding for orchestration session {}", + dispatch.orchestration_session_id + )) + })?; + + if binding.orchestration_session_id != dispatch.orchestration_session_id { + return Err(BadRequestError::new(format!( + "member_dispatch.orchestration_session_id mismatch (expected {}, got {})", + binding.orchestration_session_id, dispatch.orchestration_session_id + )) + .into()); + } + if binding.world_id != dispatch.world_id { + return Err(BadRequestError::new(format!( + "member_dispatch.world_id mismatch (expected {}, got {})", + binding.world_id, dispatch.world_id + )) + .into()); + } + if binding.world_generation != dispatch.world_generation { + return Err(BadRequestError::new(format!( + "member_dispatch.world_generation mismatch (expected {}, got {})", + binding.world_generation, dispatch.world_generation + )) + .into()); + } + if binding.binding_state != SharedWorldBindingState::Active { + return Err(BadRequestError::new(format!( + "member_dispatch requires active shared world binding, got {:?}", + binding.binding_state + )) + .into()); + } + if world.id != dispatch.world_id { + return Err(BadRequestError::new(format!( + "member_dispatch.world_id mismatch (expected {}, got {})", + world.id, dispatch.world_id + )) + .into()); + } + + Ok(binding) +} + +pub(crate) fn resolve_shared_world_binding( + requested: Option<&SharedWorldOwnerSpec>, + world: &WorldHandle, +) -> Result> { + let Some(owner_spec) = requested else { + return Ok(world.shared_binding.clone()); + }; + + let binding = world.shared_binding.clone().ok_or_else(|| { + anyhow!( + "missing authoritative shared world proof for orchestration session {}", + owner_spec.orchestration_session_id + ) + })?; + + if binding.orchestration_session_id.trim().is_empty() { + anyhow::bail!("malformed shared world proof: orchestration_session_id is empty"); + } + if binding.orchestration_session_id != owner_spec.orchestration_session_id { + anyhow::bail!( + "malformed shared world proof: orchestration_session_id mismatch (expected {}, got {})", + owner_spec.orchestration_session_id, + binding.orchestration_session_id + ); + } + if binding.world_id.trim().is_empty() { + anyhow::bail!("malformed shared world proof: world_id is empty"); + } + if binding.world_id != world.id { + anyhow::bail!( + "malformed shared world proof: world_id mismatch (expected {}, got {})", + world.id, + binding.world_id + ); + } + if binding.binding_state != SharedWorldBindingState::Active { + anyhow::bail!( + "malformed shared world proof: binding_state must be active, got {:?}", + binding.binding_state + ); + } + + Ok(Some(binding)) +} + pub(crate) fn apply_full_isolation_helper_env( env_map: &mut HashMap, landlock_supported: bool, @@ -2602,6 +3029,7 @@ pub(crate) fn resolve_landlock_allowlist_paths( #[cfg(test)] mod tests { use super::*; + use world_api::{SharedWorldOwnerAction, WorldHandle}; #[test] fn landlock_helper_src_accepts_substrate_world_agent_name() { @@ -2650,6 +3078,7 @@ mod tests { display_path: None, summary: None, }), + shared_world: None, process_telemetry: ProcessTelemetry { process_events: vec![substrate_common::ProcessEvent { event_type: substrate_common::ProcessEventType::WorldProcessStart, @@ -2703,6 +3132,49 @@ mod tests { assert!(fd.deletes.is_empty()); } + #[test] + fn build_world_spec_uses_shared_orchestration_reuse_mode() { + let owner = SharedWorldOwnerSpec { + orchestration_session_id: "orch_test".to_string(), + action: SharedWorldOwnerAction::AttachOrCreate, + }; + + let spec = build_world_spec( + PathBuf::from("/tmp/project"), + true, + WorldFsMode::Writable, + false, + vec!["example.com".to_string()], + Some(&owner), + ); + + assert_eq!(spec.reuse_mode, WorldReuseMode::SharedOrchestration(owner)); + } + + #[test] + fn resolve_shared_world_binding_rejects_mismatched_owner_proof() { + let requested = SharedWorldOwnerSpec { + orchestration_session_id: "orch_expected".to_string(), + action: SharedWorldOwnerAction::AttachOrCreate, + }; + let world = WorldHandle { + id: "wld_test".to_string(), + shared_binding: Some(SharedWorldBindingSnapshot { + orchestration_session_id: "orch_other".to_string(), + world_id: "wld_test".to_string(), + world_generation: 0, + binding_state: SharedWorldBindingState::Active, + }), + }; + + let err = resolve_shared_world_binding(Some(&requested), &world).expect_err("mismatch"); + assert!( + err.to_string() + .contains("orchestration_session_id mismatch"), + "unexpected error: {err:#}" + ); + } + #[test] fn process_telemetry_redaction_helpers_redact_common_secret_shapes() { let argv = vec![ diff --git a/crates/world-agent/tests/fs_mode.rs b/crates/world-agent/tests/fs_mode.rs index 042908a4d..135a5150d 100644 --- a/crates/world-agent/tests/fs_mode.rs +++ b/crates/world-agent/tests/fs_mode.rs @@ -90,8 +90,10 @@ fn non_pty_read_only_mode_blocks_writes() { agent_id: "fs-mode-test".to_string(), budget: None, policy_snapshot: policy_snapshot_for_mode(WorldFsMode::ReadOnly), + shared_world: None, world_network: None, world_fs_mode: Some(WorldFsMode::ReadOnly), + member_dispatch: None, }; let rt = Runtime::new().expect("runtime"); @@ -154,8 +156,10 @@ fn non_pty_read_only_mode_blocks_absolute_project_writes() { agent_id: "fs-mode-test".to_string(), budget: None, policy_snapshot: policy_snapshot_for_mode(WorldFsMode::ReadOnly), + shared_world: None, world_network: None, world_fs_mode: Some(WorldFsMode::ReadOnly), + member_dispatch: None, }; let rt = Runtime::new().expect("runtime"); @@ -208,8 +212,10 @@ fn non_pty_writable_mode_records_diffs_for_writes() { agent_id: "fs-mode-test".to_string(), budget: None, policy_snapshot: policy_snapshot_for_mode(WorldFsMode::Writable), + shared_world: None, world_network: None, world_fs_mode: Some(WorldFsMode::Writable), + member_dispatch: None, }; let rt = Runtime::new().expect("runtime"); @@ -456,8 +462,10 @@ async fn pty_writable_mode_keeps_writes_in_overlay() { agent_id: "fs-mode-test".to_string(), budget: None, policy_snapshot: policy_snapshot_for_mode(WorldFsMode::Writable), + shared_world: None, world_network: None, world_fs_mode: Some(WorldFsMode::Writable), + member_dispatch: None, }; match service.execute(verify_req).await { @@ -514,8 +522,10 @@ async fn pty_and_non_pty_share_overlay_state_across_mode_switch() { agent_id: "fs-mode-test".to_string(), budget: None, policy_snapshot: policy_snapshot_for_mode(WorldFsMode::Writable), + shared_world: None, world_network: None, world_fs_mode: Some(WorldFsMode::Writable), + member_dispatch: None, }; match service.execute(write_non_pty).await { @@ -615,8 +625,10 @@ async fn pty_and_non_pty_share_overlay_state_across_mode_switch() { agent_id: "fs-mode-test".to_string(), budget: None, policy_snapshot: policy_snapshot_for_mode(WorldFsMode::ReadOnly), + shared_world: None, world_network: None, world_fs_mode: Some(WorldFsMode::ReadOnly), + member_dispatch: None, }; match service.execute(verify_ro).await { diff --git a/crates/world-agent/tests/full_isolation_nonpty.rs b/crates/world-agent/tests/full_isolation_nonpty.rs index 1ae354fb2..cb2959b7b 100644 --- a/crates/world-agent/tests/full_isolation_nonpty.rs +++ b/crates/world-agent/tests/full_isolation_nonpty.rs @@ -114,8 +114,10 @@ fn execute_non_pty( agent_id: "full-isolation-nonpty-test".to_string(), budget: None, policy_snapshot, + shared_world: None, world_network: None, world_fs_mode: Some(world_fs_mode), + member_dispatch: None, }; let rt = Runtime::new().expect("runtime"); diff --git a/crates/world-agent/tests/gateway_runtime_parity.rs b/crates/world-agent/tests/gateway_runtime_parity.rs index 7221bf069..710ba9945 100644 --- a/crates/world-agent/tests/gateway_runtime_parity.rs +++ b/crates/world-agent/tests/gateway_runtime_parity.rs @@ -11,6 +11,12 @@ use std::fs; use std::os::unix::fs::PermissionsExt; use std::path::{Path, PathBuf}; use std::time::{Duration, Instant}; +use substrate_common::{ + GatewayAuthBundleV1, GATEWAY_AUTH_BUNDLE_BACKEND_API_OPENAI, + GATEWAY_AUTH_BUNDLE_BACKEND_CLI_CODEX, SUBSTRATE_LLM_BACKEND_AUTH_API_OPENAI_API_KEY, + SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCESS_TOKEN, + SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCOUNT_ID, +}; use tempfile::TempDir; use tokio::sync::Mutex; use world_agent::WorldAgentService; @@ -120,11 +126,79 @@ fn service_or_skip() -> Option { } } +fn gateway_auth_bundle_check_snippet() -> &'static str { + r#"if [ -z "${SUBSTRATE_LLM_AUTH_BUNDLE_FD:-}" ]; then + echo "missing gateway auth bundle fd env" >&2 + exit 65 +fi + +python3 - "$config" <<'PY' +import json +import os +import pathlib +import sys + +config = pathlib.Path(sys.argv[1]).read_text(encoding='utf-8') +for forbidden in ( + "SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCOUNT_ID", + "SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCESS_TOKEN", + "SUBSTRATE_LLM_BACKEND_AUTH_API_OPENAI_API_KEY", + "OPENAI_API_KEY", +): + if os.environ.get(forbidden): + raise SystemExit("forbidden secret env present: " + forbidden) + +fd_raw = os.environ["SUBSTRATE_LLM_AUTH_BUNDLE_FD"] +with os.fdopen(int(fd_raw), "rb") as handle: + payload = handle.read() +bundle = json.loads(payload) +backend = bundle.get("backend_id") +fields = bundle.get("fields") or {} + +if 'api_key = "$OPENAI_API_KEY"' in config: + if backend != "api:openai": + raise SystemExit("unexpected backend_id: " + repr(backend)) + if "SUBSTRATE_LLM_BACKEND_AUTH_API_OPENAI_API_KEY" not in fields: + raise SystemExit("missing OpenAI API key field") +else: + if backend != "cli:codex": + raise SystemExit("unexpected backend_id: " + repr(backend)) + if "SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCESS_TOKEN" not in fields: + raise SystemExit("missing Codex access token field") +PY +"# +} + +fn auth_bundle_snapshot_path(temp_dir: &TempDir, launch: u32) -> PathBuf { + temp_dir + .path() + .join("auth-bundles") + .join(format!("{launch}.json")) +} + +fn wait_for_auth_bundle_snapshot(temp_dir: &TempDir, launch: u32) -> GatewayAuthBundleV1 { + let path = auth_bundle_snapshot_path(temp_dir, launch); + let deadline = Instant::now() + Duration::from_secs(2); + loop { + if let Ok(raw) = fs::read(&path) { + return serde_json::from_slice(&raw) + .unwrap_or_else(|err| panic!("invalid {}: {err}", path.display())); + } + assert!( + Instant::now() < deadline, + "timed out waiting for {}", + path.display() + ); + std::thread::sleep(Duration::from_millis(25)); + } +} + fn fake_gateway_binary(temp_dir: &TempDir) -> PathBuf { let path = temp_dir.path().join("fake-gateway.sh"); fs::write( &path, - r#"#!/bin/sh + format!( + r#"#!/bin/sh set -eu config="" while [ "$#" -gt 0 ]; do @@ -147,15 +221,7 @@ if [ -z "$config" ]; then exit 64 fi -if grep -q 'api_key = "\$OPENAI_API_KEY"' "$config"; then - if [ -z "${OPENAI_API_KEY:-}" ]; then - echo "missing OpenAI API key env" >&2 - exit 65 - fi -elif [ -z "${SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCESS_TOKEN:-}" ]; then - echo "missing Codex access token env" >&2 - exit 65 -fi +{bundle_check} port="$(python3 - "$config" <<'PY' import re @@ -173,6 +239,8 @@ mkdir -p "$root" printf 'ok' >"$root/health" exec python3 -m http.server "$port" --bind 127.0.0.1 --directory "$root" "#, + bundle_check = gateway_auth_bundle_check_snippet(), + ), ) .unwrap(); let mut perms = fs::metadata(&path).unwrap().permissions(); @@ -253,15 +321,7 @@ if [ -z "$config" ]; then exit 64 fi -if grep -q 'api_key = "\$OPENAI_API_KEY"' "$config"; then - if [ -z "${{OPENAI_API_KEY:-}}" ]; then - echo "missing OpenAI API key env" >&2 - exit 65 - fi -elif [ -z "${{SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCESS_TOKEN:-}}" ]; then - echo "missing Codex access token env" >&2 - exit 65 -fi +{bundle_check} port="$(python3 - "$config" <<'PY' import re @@ -276,6 +336,7 @@ PY exec python3 "{server_path}" "$port" "#, + bundle_check = gateway_auth_bundle_check_snippet(), server_path = server_path.display(), ), ) @@ -315,15 +376,7 @@ if [ -z "$config" ]; then exit 64 fi -if grep -q 'api_key = "\$OPENAI_API_KEY"' "$config"; then - if [ -z "${{OPENAI_API_KEY:-}}" ]; then - echo "missing OpenAI API key env" >&2 - exit 65 - fi -elif [ -z "${{SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCESS_TOKEN:-}}" ]; then - echo "missing Codex access token env" >&2 - exit 65 -fi +{bundle_check} printf '%s\n' "$$" >"{pid_path}" port="$(python3 - "$config" <<'PY' @@ -343,6 +396,7 @@ mkdir -p "$root" printf 'ok' >"$root/health" exec python3 -m http.server "$port" --bind 127.0.0.1 --directory "$root" "#, + bundle_check = gateway_auth_bundle_check_snippet(), pid_path = pid_path.display(), delay_s = delay_ms as f64 / 1000.0, ), @@ -361,6 +415,7 @@ fn tracking_gateway_binary( ) -> (PathBuf, PathBuf, PathBuf) { let path = temp_dir.path().join(format!("{name}-tracking-gateway.sh")); let pid_dir = temp_dir.path().join(format!("{name}-pids")); + let bundle_dir = temp_dir.path().join("auth-bundles"); let launch_count_path = temp_dir.path().join(format!("{name}-launch-count.txt")); fs::write( &path, @@ -388,13 +443,8 @@ if [ -z "$config" ]; then exit 64 fi -if grep -q 'api_key = "\$OPENAI_API_KEY"' "$config"; then - if [ -z "${{OPENAI_API_KEY:-}}" ]; then - echo "missing OpenAI API key env" >&2 - exit 65 - fi -elif [ -z "${{SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCESS_TOKEN:-}}" ]; then - echo "missing Codex access token env" >&2 +if [ -z "${{SUBSTRATE_LLM_AUTH_BUNDLE_FD:-}}" ]; then + echo "missing gateway auth bundle fd env" >&2 exit 65 fi @@ -411,6 +461,46 @@ PY mkdir -p "{pid_dir}" printf '%s\n' "$$" >"{pid_dir}/$launch.pid" +python3 - "$config" "$launch" "{bundle_dir}" <<'PY' +import json +import os +import pathlib +import sys + +config = pathlib.Path(sys.argv[1]).read_text(encoding='utf-8') +launch = sys.argv[2] +bundle_dir = pathlib.Path(sys.argv[3]) +bundle_dir.mkdir(parents=True, exist_ok=True) + +for forbidden in ( + "SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCOUNT_ID", + "SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCESS_TOKEN", + "SUBSTRATE_LLM_BACKEND_AUTH_API_OPENAI_API_KEY", + "OPENAI_API_KEY", +): + if os.environ.get(forbidden): + raise SystemExit("forbidden secret env present: " + forbidden) + +fd_raw = os.environ["SUBSTRATE_LLM_AUTH_BUNDLE_FD"] +with os.fdopen(int(fd_raw), "rb") as handle: + payload = handle.read() +bundle = json.loads(payload) +fields = bundle.get("fields") or {{}} + +if 'api_key = "$OPENAI_API_KEY"' in config: + if bundle.get("backend_id") != "api:openai": + raise SystemExit("unexpected backend_id") + if "SUBSTRATE_LLM_BACKEND_AUTH_API_OPENAI_API_KEY" not in fields: + raise SystemExit("missing OpenAI API key field") +else: + if bundle.get("backend_id") != "cli:codex": + raise SystemExit("unexpected backend_id") + if "SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCESS_TOKEN" not in fields: + raise SystemExit("missing Codex access token field") + +(bundle_dir / (launch + ".json")).write_bytes(payload) +PY + port="$(python3 - "$config" <<'PY' import re import sys @@ -428,6 +518,7 @@ mkdir -p "$root" printf 'ok' >"$root/health" exec python3 -m http.server "$port" --bind 127.0.0.1 --directory "$root" "#, + bundle_dir = bundle_dir.display(), launch_count_path = launch_count_path.display(), pid_dir = pid_dir.display(), delay_s = delay_ms as f64 / 1000.0, @@ -472,18 +563,11 @@ while [ "$#" -gt 0 ]; do ;; esac done -if grep -q 'api_key = "\$OPENAI_API_KEY"' "$config"; then - if [ -z "${{OPENAI_API_KEY:-}}" ]; then - echo "missing OpenAI API key env" >&2 - exit 65 - fi -elif [ -z "${{SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCESS_TOKEN:-}}" ]; then - echo "missing Codex access token env" >&2 - exit 65 -fi +{bundle_check} printf '%s\n' "$$" >"{pid_path}" sleep 30 "#, + bundle_check = gateway_auth_bundle_check_snippet(), pid_path = pid_path.display(), ), ) @@ -668,6 +752,27 @@ async fn gateway_sync_makes_status_available_and_is_idempotent() { ); assert_eq!(read_launch_count(&launch_count_path), 1); assert_eq!(first_pid, wait_for_pid(&pid_dir, 1)); + assert_eq!( + wait_for_auth_bundle_snapshot(&temp_dir, 1), + GatewayAuthBundleV1 { + schema_version: 1, + backend_id: GATEWAY_AUTH_BUNDLE_BACKEND_CLI_CODEX.to_string(), + fields: std::collections::HashMap::from([ + ( + SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCOUNT_ID.to_string(), + "acct_test".to_string(), + ), + ( + SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCESS_TOKEN.to_string(), + "header.payload.signature".to_string(), + ), + ]), + } + ); + assert!( + !auth_bundle_snapshot_path(&temp_dir, 2).exists(), + "idempotent sync should not relaunch or reread the auth bundle" + ); } #[tokio::test] @@ -708,6 +813,21 @@ async fn gateway_openai_sync_makes_status_available_and_is_idempotent() { assert_eq!(second_sync.status, GatewayStatusV1::Available); assert_eq!(read_launch_count(&launch_count_path), 1); assert_eq!(first_pid, wait_for_pid(&pid_dir, 1)); + assert_eq!( + wait_for_auth_bundle_snapshot(&temp_dir, 1), + GatewayAuthBundleV1 { + schema_version: 1, + backend_id: GATEWAY_AUTH_BUNDLE_BACKEND_API_OPENAI.to_string(), + fields: std::collections::HashMap::from([( + SUBSTRATE_LLM_BACKEND_AUTH_API_OPENAI_API_KEY.to_string(), + "sk-openai-proof".to_string(), + )]), + } + ); + assert!( + !auth_bundle_snapshot_path(&temp_dir, 2).exists(), + "idempotent sync should not relaunch or reread the auth bundle" + ); } #[tokio::test] @@ -770,6 +890,80 @@ async fn gateway_restart_recycles_the_runtime() { assert_process_exited(initial_pid); } +#[tokio::test] +async fn gateway_restart_uses_fresh_codex_bundle_after_rotation() { + let _env_lock = ENV_LOCK.lock().await; + let temp_dir = TempDir::new().unwrap(); + let (binary, pid_dir, launch_count_path) = + tracking_gateway_binary(&temp_dir, "codex-rotation", 0); + let _binary_guard = EnvGuard::set("SUBSTRATE_GATEWAY_BINARY", binary); + let Some(service) = service_or_skip() else { + return; + }; + let mut request = gateway_request(temp_dir.path()); + request.integrated_auth = Some(GatewayIntegratedAuthPayloadV1 { + backend_id: REGRESSION_FLOOR_BACKEND_ID.to_string(), + cli_codex: Some(GatewayCliCodexIntegratedAuthV1 { + account_id: Some("acct_first".to_string()), + access_token: "token-first".to_string(), + }), + api_env: None, + }); + + service + .gateway_sync(request.clone()) + .await + .expect("initial gateway sync"); + let initial_pid = wait_for_pid(&pid_dir, 1); + let first_bundle = wait_for_auth_bundle_snapshot(&temp_dir, 1); + assert_eq!( + first_bundle + .fields + .get("SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCOUNT_ID"), + Some(&"acct_first".to_string()) + ); + assert_eq!( + first_bundle + .fields + .get(SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCESS_TOKEN), + Some(&"token-first".to_string()) + ); + + request.integrated_auth = Some(GatewayIntegratedAuthPayloadV1 { + backend_id: REGRESSION_FLOOR_BACKEND_ID.to_string(), + cli_codex: Some(GatewayCliCodexIntegratedAuthV1 { + account_id: Some("acct_second".to_string()), + access_token: "token-second".to_string(), + }), + api_env: None, + }); + + let restart_response = service + .gateway_restart(request) + .await + .expect("gateway restart"); + assert_eq!(restart_response.status, GatewayStatusV1::Available); + + let restarted_pid = wait_for_pid(&pid_dir, 2); + let second_bundle = wait_for_auth_bundle_snapshot(&temp_dir, 2); + assert_ne!(initial_pid, restarted_pid); + assert_eq!(read_launch_count(&launch_count_path), 2); + assert_process_exited(initial_pid); + assert_eq!( + second_bundle + .fields + .get("SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCOUNT_ID"), + Some(&"acct_second".to_string()) + ); + assert_eq!( + second_bundle + .fields + .get(SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCESS_TOKEN), + Some(&"token-second".to_string()) + ); + assert_ne!(first_bundle, second_bundle); +} + #[tokio::test] async fn gateway_openai_restart_recycles_the_runtime() { let _env_lock = ENV_LOCK.lock().await; @@ -800,6 +994,72 @@ async fn gateway_openai_restart_recycles_the_runtime() { assert_process_exited(initial_pid); } +#[tokio::test] +async fn gateway_openai_restart_uses_fresh_bundle_after_rotation() { + let _env_lock = ENV_LOCK.lock().await; + let temp_dir = TempDir::new().unwrap(); + let (binary, pid_dir, launch_count_path) = + tracking_gateway_binary(&temp_dir, "openai-rotation", 0); + let _binary_guard = EnvGuard::set("SUBSTRATE_GATEWAY_BINARY", binary); + let Some(service) = service_or_skip() else { + return; + }; + let mut request = gateway_request_for_backend(temp_dir.path(), FIRST_ADDITIONAL_BACKEND_ID); + request.integrated_auth = Some(GatewayIntegratedAuthPayloadV1 { + backend_id: FIRST_ADDITIONAL_BACKEND_ID.to_string(), + cli_codex: None, + api_env: Some(GatewayApiEnvIntegratedAuthV1 { + env: std::collections::HashMap::from([( + "OPENAI_API_KEY".to_string(), + "sk-openai-first".to_string(), + )]), + }), + }); + + service + .gateway_sync(request.clone()) + .await + .expect("initial gateway sync"); + let initial_pid = wait_for_pid(&pid_dir, 1); + let first_bundle = wait_for_auth_bundle_snapshot(&temp_dir, 1); + assert_eq!( + first_bundle + .fields + .get(SUBSTRATE_LLM_BACKEND_AUTH_API_OPENAI_API_KEY), + Some(&"sk-openai-first".to_string()) + ); + + request.integrated_auth = Some(GatewayIntegratedAuthPayloadV1 { + backend_id: FIRST_ADDITIONAL_BACKEND_ID.to_string(), + cli_codex: None, + api_env: Some(GatewayApiEnvIntegratedAuthV1 { + env: std::collections::HashMap::from([( + "OPENAI_API_KEY".to_string(), + "sk-openai-second".to_string(), + )]), + }), + }); + + let restart_response = service + .gateway_restart(request) + .await + .expect("gateway restart"); + assert_eq!(restart_response.status, GatewayStatusV1::Available); + + let restarted_pid = wait_for_pid(&pid_dir, 2); + let second_bundle = wait_for_auth_bundle_snapshot(&temp_dir, 2); + assert_ne!(initial_pid, restarted_pid); + assert_eq!(read_launch_count(&launch_count_path), 2); + assert_process_exited(initial_pid); + assert_eq!( + second_bundle + .fields + .get(SUBSTRATE_LLM_BACKEND_AUTH_API_OPENAI_API_KEY), + Some(&"sk-openai-second".to_string()) + ); + assert_ne!(first_bundle, second_bundle); +} + #[tokio::test] async fn gateway_unbound_lifecycle_actions_do_not_fall_back_to_running_codex_runtime() { let _env_lock = ENV_LOCK.lock().await; diff --git a/crates/world-agent/tests/member_runtime_world_placement_v1.rs b/crates/world-agent/tests/member_runtime_world_placement_v1.rs new file mode 100644 index 000000000..f37ca5741 --- /dev/null +++ b/crates/world-agent/tests/member_runtime_world_placement_v1.rs @@ -0,0 +1,327 @@ +#![cfg(all(unix, target_os = "linux"))] + +use agent_api_types::{ + ExecuteCancelRequestV1, ExecuteRequest, MemberDispatchRequestV1, MemberRuntimeBackendKindV1, + PolicySnapshotV3, PolicySnapshotWorldFsFailClosedV3, PolicySnapshotWorldFsV3, + PolicySnapshotWorldFsWriteV3, ResolvedMemberRuntimeDescriptorV1, +}; +use hyper::body::HttpBody; +use serde_json::Value; +use std::collections::HashMap; +use std::fs; +use std::path::{Path, PathBuf}; +use std::time::Duration; +use tempfile::tempdir; +use tokio::time::timeout; +use world_agent::WorldAgentService; +use world_api::{SharedWorldOwnerAction, SharedWorldOwnerSpec, WorldReuseMode, WorldSpec}; + +const WORLD_PROJECT_DIR_OVERRIDE_ENV: &str = "SUBSTRATE_WORLD_PROJECT_DIR"; + +fn full_isolation_policy_snapshot() -> PolicySnapshotV3 { + PolicySnapshotV3 { + schema_version: 3, + net_allowed: Vec::new(), + world_fs: PolicySnapshotWorldFsV3 { + host_visible: false, + fail_closed: PolicySnapshotWorldFsFailClosedV3 { routing: false }, + deny_enforcement: None, + caged_required: false, + discover: None, + read: None, + write: PolicySnapshotWorldFsWriteV3 { + enabled: true, + allow_list: vec![".".to_string()], + deny_list: Vec::new(), + }, + }, + } +} + +fn make_member_dispatch_request( + cwd: &Path, + binary_path: &Path, + world_id: &str, + world_generation: u64, + env: HashMap, +) -> ExecuteRequest { + ExecuteRequest { + profile: None, + cmd: String::new(), + cwd: Some(cwd.display().to_string()), + env: Some(env), + pty: false, + agent_id: "member-runtime-world-placement-test".to_string(), + budget: None, + policy_snapshot: full_isolation_policy_snapshot(), + shared_world: None, + world_network: None, + world_fs_mode: None, + member_dispatch: Some(MemberDispatchRequestV1 { + schema_version: 1, + orchestration_session_id: "orch-member-runtime-world-placement".to_string(), + participant_id: "ash_member_world_placement_test".to_string(), + orchestrator_participant_id: "ash_orchestrator_world_placement_test".to_string(), + parent_participant_id: None, + resumed_from_participant_id: None, + backend_id: "cli:codex".to_string(), + protocol: "uaa.agent.session".to_string(), + run_id: "run-member-world-placement-test".to_string(), + world_id: world_id.to_string(), + world_generation, + initial_prompt: None, + resolved_runtime: ResolvedMemberRuntimeDescriptorV1 { + backend_kind: MemberRuntimeBackendKindV1::Codex, + binary_path: binary_path.display().to_string(), + }, + }), + } +} + +fn write_placement_member_runtime(temp: &Path) -> PathBuf { + let path = temp.join("fake-member-runtime-placement.sh"); + let body = r#"#!/bin/sh +set -eu + +proof_path="${PLACEMENT_PROOF_PATH:?}" +cgroup_procs="${EXPECTED_CGROUP_PROCS:?}" +attached=0 +attempt=0 +while [ "$attempt" -lt 100 ]; do + if [ -f "$cgroup_procs" ] && grep -qx "$$" "$cgroup_procs"; then + attached=1 + break + fi + attempt=$((attempt + 1)) + sleep 0.05 +done + +{ + printf 'cwd=%s\n' "$(pwd)" + printf 'attached=%s\n' "$attached" +} > "$proof_path" + +printf '{"type":"thread.started","thread_id":"thread-member-world-placement"}\r\n' +printf '{"type":"turn.started","thread_id":"thread-member-world-placement","turn_id":"turn-member-world-placement"}\r\n' +while :; do sleep 1; done +"#; + fs::write(&path, body).expect("write fake member runtime"); + let mut perms = fs::metadata(&path) + .expect("fake member runtime metadata") + .permissions(); + use std::os::unix::fs::PermissionsExt; + perms.set_mode(0o755); + fs::set_permissions(&path, perms).expect("set fake member runtime permissions"); + path +} + +async fn next_stream_frame_value(body: &mut B, buffer: &mut Vec) -> Value +where + B: HttpBody + Unpin, + B::Error: std::fmt::Debug + std::fmt::Display, +{ + loop { + if let Some(pos) = buffer.iter().position(|&b| b == b'\n') { + let line: Vec = buffer.drain(..=pos).collect(); + let payload = &line[..line.len() - 1]; + return serde_json::from_slice(payload).expect("valid stream frame json value"); + } + + let chunk = timeout(Duration::from_secs(5), body.data()) + .await + .expect("timed out waiting for stream chunk") + .expect("stream ended unexpectedly") + .expect("stream chunk error"); + buffer.extend_from_slice(&chunk); + } +} + +fn frame_start_span_id(frame: &Value) -> Option<&str> { + if frame.get("type")?.as_str() == Some("start") { + return frame.get("span_id")?.as_str(); + } + frame.get("Start")?.get("span_id")?.as_str() +} + +fn frame_exit(frame: &Value) -> Option<(i32, &str)> { + if frame.get("type")?.as_str() == Some("exit") { + let exit = frame.get("exit")?.as_i64()?; + let span_id = frame.get("span_id")?.as_str()?; + return Some((exit as i32, span_id)); + } + let exit = frame.get("Exit")?.get("exit")?.as_i64()?; + let span_id = frame.get("Exit")?.get("span_id")?.as_str()?; + Some((exit as i32, span_id)) +} + +fn frame_error_message(frame: &Value) -> Option<&str> { + if frame.get("type")?.as_str() == Some("error") { + return frame.get("message")?.as_str(); + } + frame.get("Error")?.get("message")?.as_str() +} + +fn is_event_frame(frame: &Value) -> bool { + frame.get("type").and_then(Value::as_str) == Some("event") || frame.get("Event").is_some() +} + +fn is_stream_chunk_frame(frame: &Value) -> bool { + matches!( + frame.get("type").and_then(Value::as_str), + Some("stdout" | "stderr") + ) || frame.get("Stdout").is_some() + || frame.get("Stderr").is_some() +} + +async fn wait_for_proof_file(path: &Path) -> String { + timeout(Duration::from_secs(5), async { + loop { + match fs::read_to_string(path) { + Ok(content) if !content.trim().is_empty() => return content, + _ => tokio::time::sleep(Duration::from_millis(50)).await, + } + } + }) + .await + .expect("timed out waiting for placement proof") +} + +fn parse_proof_value<'a>(proof: &'a str, key: &str) -> Option<&'a str> { + proof.lines().find_map(|line| { + let (found_key, value) = line.split_once('=')?; + (found_key == key).then_some(value) + }) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn member_runtime_launches_inside_authoritative_overlay_and_cgroup() { + let service = match WorldAgentService::new() { + Ok(svc) => svc, + Err(err) => { + eprintln!("skipping member placement proof test: service init failed: {err}"); + return; + } + }; + + let tmp = tempdir().expect("tempdir"); + let nested = tmp.path().join("nested"); + fs::create_dir_all(&nested).expect("create nested cwd"); + let member_binary = write_placement_member_runtime(tmp.path()); + + let world_spec = WorldSpec { + reuse_session: true, + reuse_mode: WorldReuseMode::SharedOrchestration(SharedWorldOwnerSpec { + orchestration_session_id: "orch-member-runtime-world-placement".to_string(), + action: SharedWorldOwnerAction::AttachOrCreate, + }), + isolate_network: false, + limits: world_api::ResourceLimits::default(), + enable_preload: false, + allowed_domains: Vec::new(), + project_dir: tmp.path().to_path_buf(), + always_isolate: true, + fs_mode: substrate_common::WorldFsMode::Writable, + }; + let (world, overlay_root) = match service.ensure_session_overlay_root(&world_spec) { + Ok(result) => result, + Err(err) => { + eprintln!( + "skipping member placement proof test: failed to prepare overlay root: {err}" + ); + return; + } + }; + let binding = world + .shared_binding + .clone() + .expect("shared world binding should exist once authoritative world is prepared"); + let cgroup_procs = service + .session_cgroup_path(&world) + .expect("session cgroup path should resolve once authoritative world is prepared") + .join("cgroup.procs"); + let proof_path = tmp.path().join("placement-proof.txt"); + + let mut env = HashMap::new(); + env.insert( + WORLD_PROJECT_DIR_OVERRIDE_ENV.to_string(), + tmp.path().display().to_string(), + ); + env.insert( + "PLACEMENT_PROOF_PATH".to_string(), + proof_path.display().to_string(), + ); + env.insert( + "EXPECTED_CGROUP_PROCS".to_string(), + cgroup_procs.display().to_string(), + ); + let request = make_member_dispatch_request( + &nested, + &member_binary, + &binding.world_id, + binding.world_generation, + env, + ); + + let response = match service.execute_stream(request).await { + Ok(response) => response, + Err(err) => { + panic!( + "member placement proof test failed to establish streamed execute placement: {err}" + ); + } + }; + + let mut body = response.into_body(); + let mut buffer = Vec::new(); + + let start = next_stream_frame_value(&mut body, &mut buffer).await; + let span_id = frame_start_span_id(&start) + .map(ToOwned::to_owned) + .unwrap_or_else(|| panic!("expected start frame, got {start:?}")); + + let proof = wait_for_proof_file(&proof_path).await; + let expected_cwd = overlay_root.join("nested"); + let expected_cwd_display = expected_cwd.display().to_string(); + assert_ne!( + expected_cwd, nested, + "placement proof requires an overlay-root cwd distinct from the host cwd" + ); + assert_eq!( + parse_proof_value(&proof, "cwd"), + Some(expected_cwd_display.as_str()), + "member runtime should observe the authoritative overlay cwd" + ); + assert_eq!( + parse_proof_value(&proof, "attached"), + Some("1"), + "member runtime pid should be attached to the authoritative session cgroup" + ); + + let cancel = service + .execute_cancel(ExecuteCancelRequestV1 { + span_id: span_id.clone(), + sig: "INT".to_string(), + }) + .await + .expect("member execute_cancel should succeed"); + assert!(cancel.delivered, "expected member cancel delivery"); + + loop { + let frame = next_stream_frame_value(&mut body, &mut buffer).await; + if is_event_frame(&frame) + || is_stream_chunk_frame(&frame) + || frame_start_span_id(&frame).is_some() + { + continue; + } + if let Some((exit, exit_span)) = frame_exit(&frame) { + assert_eq!(exit_span, span_id); + assert_eq!(exit, 130, "SIGINT exit should follow shell convention"); + break; + } + if let Some(message) = frame_error_message(&frame) { + panic!("unexpected member streamed error: {message}"); + } + panic!("unexpected member streamed frame: {frame:?}"); + } +} diff --git a/crates/world-agent/tests/overlayfs_enumeration.rs b/crates/world-agent/tests/overlayfs_enumeration.rs index 018ec0bdf..dfbdbdbe7 100644 --- a/crates/world-agent/tests/overlayfs_enumeration.rs +++ b/crates/world-agent/tests/overlayfs_enumeration.rs @@ -74,8 +74,10 @@ fn execute_non_pty( agent_id: "overlayfs-enumeration-test".to_string(), budget: None, policy_snapshot, + shared_world: None, world_network: None, world_fs_mode: Some(world_fs_mode), + member_dispatch: None, }; let rt = Runtime::new().expect("runtime"); diff --git a/crates/world-agent/tests/repl_persistent_session_exec_v1.rs b/crates/world-agent/tests/repl_persistent_session_exec_v1.rs index 34546d3d9..9fe45c2ce 100644 --- a/crates/world-agent/tests/repl_persistent_session_exec_v1.rs +++ b/crates/world-agent/tests/repl_persistent_session_exec_v1.rs @@ -97,6 +97,20 @@ async fn recv_json(ws: &mut Ws) -> Value { serde_json::from_str(&text).expect("server ws message is valid JSON") } +async fn assert_ws_closes_after_fatal(ws: &mut Ws) { + let msg = timeout(Duration::from_secs(2), ws.next()).await; + match msg { + Ok(None) => {} + Ok(Some(Ok(Message::Close(_)))) => {} + Ok(Some(Ok(other))) => panic!("expected ws close after fatal error, got: {other:?}"), + Ok(Some(Err(_))) => { + // Some implementations drop/reset the TCP connection without a close handshake. + // This still satisfies "close the WebSocket connection" semantics. + } + Err(_) => panic!("timed out waiting for ws close after fatal error"), + } +} + fn looks_like_missing_world_prereqs(frame: &Value) -> bool { if frame.get("type").and_then(Value::as_str) != Some("error") { return false; @@ -412,16 +426,33 @@ async fn exec_while_busy_is_fatal_protocol_error() { .await .expect("send exec 2 while exec 1 in-flight"); - let frame = recv_json(&mut ws).await; - if frame.get("type").and_then(Value::as_str) == Some("stdout") - && decode_stdout_frame(&frame) - .as_deref() - .is_some_and(stdout_indicates_missing_world_prereqs) - { - eprintln!( - "skipping exec-while-busy test: world prereqs missing during exec: {}", - String::from_utf8_lossy(&decode_stdout_frame(&frame).unwrap_or_default()) - ); + let mut stdout = Vec::new(); + let frame = loop { + let frame = recv_json(&mut ws).await; + match frame.get("type").and_then(Value::as_str) { + Some("stdout") => { + let bytes = decode_stdout_frame(&frame).unwrap_or_default(); + stdout.extend_from_slice(&bytes); + if stdout_indicates_missing_world_prereqs(&stdout) { + eprintln!( + "skipping exec-while-busy test: world prereqs missing during exec: {}", + String::from_utf8_lossy(&stdout) + ); + drop(ws); + stop_server(shutdown, server).await; + return; + } + } + Some("error") => break frame, + Some(other) => panic!( + "unexpected server frame type before fatal busy error: {other:?} frame={frame}" + ), + None => panic!("server frame missing type: {frame}"), + } + }; + + if looks_like_missing_world_prereqs(&frame) { + eprintln!("skipping exec-while-busy test: world prereqs missing during exec: {frame}"); drop(ws); stop_server(shutdown, server).await; return; @@ -433,6 +464,7 @@ async fn exec_while_busy_is_fatal_protocol_error() { ); assert_eq!(frame.get("fatal").and_then(Value::as_bool), Some(true)); assert_eq!(frame.get("seq").and_then(Value::as_u64), Some(2)); + assert_ws_closes_after_fatal(&mut ws).await; drop(ws); stop_server(shutdown, server).await; diff --git a/crates/world-agent/tests/streamed_execute_cancel_v1.rs b/crates/world-agent/tests/streamed_execute_cancel_v1.rs index cba9cbfb2..cd06391cf 100644 --- a/crates/world-agent/tests/streamed_execute_cancel_v1.rs +++ b/crates/world-agent/tests/streamed_execute_cancel_v1.rs @@ -1,16 +1,20 @@ #![cfg(all(unix, target_os = "linux"))] use agent_api_types::{ - ExecuteCancelRequestV1, ExecuteRequest, ExecuteStreamFrame, PolicySnapshotV3, - PolicySnapshotWorldFsFailClosedV3, PolicySnapshotWorldFsV3, PolicySnapshotWorldFsWriteV3, + ExecuteCancelRequestV1, ExecuteRequest, MemberDispatchRequestV1, MemberRuntimeBackendKindV1, + PolicySnapshotV3, PolicySnapshotWorldFsFailClosedV3, PolicySnapshotWorldFsV3, + PolicySnapshotWorldFsWriteV3, ResolvedMemberRuntimeDescriptorV1, }; use hyper::body::HttpBody; +use serde_json::Value; use std::collections::HashMap; +use std::fs; use std::path::Path; use std::time::Duration; use tempfile::tempdir; use tokio::time::timeout; use world_agent::WorldAgentService; +use world_api::{SharedWorldOwnerAction, SharedWorldOwnerSpec, WorldReuseMode, WorldSpec}; fn minimal_policy_snapshot() -> PolicySnapshotV3 { PolicySnapshotV3 { @@ -47,12 +51,93 @@ fn make_request(cwd: &Path, cmd: &str) -> ExecuteRequest { agent_id: "streamed-execute-cancel-test".to_string(), budget: None, policy_snapshot: minimal_policy_snapshot(), + shared_world: None, world_network: None, world_fs_mode: None, + member_dispatch: None, } } -async fn next_stream_frame(body: &mut B, buffer: &mut Vec) -> ExecuteStreamFrame +fn make_member_dispatch_request( + cwd: &Path, + binary_path: &Path, + world_id: &str, + world_generation: u64, +) -> ExecuteRequest { + make_member_dispatch_request_with_backend( + cwd, + binary_path, + world_id, + world_generation, + "orch-streamed-member-cancel", + "ash_member_cancel_test", + "run-member-cancel-test", + "cli:codex", + MemberRuntimeBackendKindV1::Codex, + ) +} + +#[allow(clippy::too_many_arguments)] +fn make_member_dispatch_request_with_backend( + cwd: &Path, + binary_path: &Path, + world_id: &str, + world_generation: u64, + orchestration_session_id: &str, + participant_id: &str, + run_id: &str, + backend_id: &str, + backend_kind: MemberRuntimeBackendKindV1, +) -> ExecuteRequest { + let mut request = make_request(cwd, ""); + request.member_dispatch = Some(MemberDispatchRequestV1 { + schema_version: 1, + orchestration_session_id: orchestration_session_id.to_string(), + participant_id: participant_id.to_string(), + orchestrator_participant_id: "ash_orchestrator_cancel_test".to_string(), + parent_participant_id: None, + resumed_from_participant_id: None, + backend_id: backend_id.to_string(), + protocol: "uaa.agent.session".to_string(), + run_id: run_id.to_string(), + world_id: world_id.to_string(), + world_generation, + initial_prompt: None, + resolved_runtime: ResolvedMemberRuntimeDescriptorV1 { + backend_kind, + binary_path: binary_path.display().to_string(), + }, + }); + request +} + +fn write_fake_member_runtime(temp: &Path) -> std::path::PathBuf { + let path = temp.join("fake-member-runtime.sh"); + let body = "#!/bin/sh\ntrap 'exit 0' INT TERM HUP QUIT\nprintf '{\"type\":\"thread.started\",\"thread_id\":\"thread-member-cancel\"}\\r\\n'\nprintf '{\"type\":\"turn.started\",\"thread_id\":\"thread-member-cancel\",\"turn_id\":\"turn-member-cancel\"}\\r\\n'\nwhile :; do sleep 1; done\n"; + fs::write(&path, body).expect("write fake member runtime"); + let mut perms = fs::metadata(&path) + .expect("fake member runtime metadata") + .permissions(); + use std::os::unix::fs::PermissionsExt; + perms.set_mode(0o755); + fs::set_permissions(&path, perms).expect("set fake member runtime permissions"); + path +} + +fn write_fake_claude_member_runtime(temp: &Path) -> std::path::PathBuf { + let path = temp.join("fake-member-runtime-claude.sh"); + let body = "#!/bin/sh\ntrap 'exit 0' INT TERM HUP QUIT\nprintf '{\"type\":\"system\",\"subtype\":\"init\",\"session_id\":\"sess-member-cancel\"}\\n'\nwhile :; do sleep 1; done\n"; + fs::write(&path, body).expect("write fake claude member runtime"); + let mut perms = fs::metadata(&path) + .expect("fake claude member runtime metadata") + .permissions(); + use std::os::unix::fs::PermissionsExt; + perms.set_mode(0o755); + fs::set_permissions(&path, perms).expect("set fake claude member runtime permissions"); + path +} + +async fn next_stream_frame_value(body: &mut B, buffer: &mut Vec) -> Value where B: HttpBody + Unpin, B::Error: std::fmt::Debug + std::fmt::Display, @@ -61,7 +146,7 @@ where if let Some(pos) = buffer.iter().position(|&b| b == b'\n') { let line: Vec = buffer.drain(..=pos).collect(); let payload = &line[..line.len() - 1]; - return serde_json::from_slice(payload).expect("valid stream frame json"); + return serde_json::from_slice(payload).expect("valid stream frame json value"); } let chunk = timeout(Duration::from_secs(5), body.data()) @@ -73,6 +158,141 @@ where } } +fn frame_start_span_id(frame: &Value) -> Option<&str> { + if frame.get("type")?.as_str() == Some("start") { + return frame.get("span_id")?.as_str(); + } + frame.get("Start")?.get("span_id")?.as_str() +} + +fn frame_exit(frame: &Value) -> Option<(i32, &str)> { + if frame.get("type")?.as_str() == Some("exit") { + let exit = frame.get("exit")?.as_i64()?; + let span_id = frame.get("span_id")?.as_str()?; + return Some((exit as i32, span_id)); + } + let exit = frame.get("Exit")?.get("exit")?.as_i64()?; + let span_id = frame.get("Exit")?.get("span_id")?.as_str()?; + Some((exit as i32, span_id)) +} + +fn frame_error_message(frame: &Value) -> Option<&str> { + if frame.get("type")?.as_str() == Some("error") { + return frame.get("message")?.as_str(); + } + frame.get("Error")?.get("message")?.as_str() +} + +fn frame_event(frame: &Value) -> Option<&Value> { + if frame.get("type").and_then(Value::as_str) == Some("event") { + return frame.get("event"); + } + frame.get("Event")?.get("event") +} + +fn assert_registered_event( + frame: &Value, + expected_participant_id: &str, + expected_backend_id: &str, + expected_world_id: &str, + expected_world_generation: u64, + expected_span_id: &str, +) { + let event = frame + .get("event") + .or_else(|| frame.pointer("/Event/event")) + .unwrap_or_else(|| panic!("expected member registered event, got {frame:?}")); + assert_eq!( + event.get("kind").and_then(Value::as_str), + Some("registered") + ); + assert_eq!( + event.pointer("/data/schema").and_then(Value::as_str), + Some("agent_api.session.handle.v1") + ); + assert_eq!( + event.get("participant_id").and_then(Value::as_str), + Some(expected_participant_id) + ); + assert_eq!( + event.get("backend_id").and_then(Value::as_str), + Some(expected_backend_id) + ); + assert_eq!( + event.get("world_id").and_then(Value::as_str), + Some(expected_world_id) + ); + assert_eq!( + event.get("world_generation").and_then(Value::as_u64), + Some(expected_world_generation) + ); + assert_eq!( + event.get("span_id").and_then(Value::as_str), + Some(expected_span_id) + ); +} + +fn is_stream_chunk_frame(frame: &Value) -> bool { + matches!( + frame.get("type").and_then(Value::as_str), + Some("stdout" | "stderr") + ) || frame.get("Stdout").is_some() + || frame.get("Stderr").is_some() +} + +async fn next_registered_frame(body: &mut B, buffer: &mut Vec, span_id: &str) -> Value +where + B: HttpBody + Unpin, + B::Error: std::fmt::Debug + std::fmt::Display, +{ + loop { + let frame = next_stream_frame_value(body, buffer).await; + if let Some(event) = frame_event(&frame) { + if event.get("kind").and_then(Value::as_str) == Some("registered") { + return frame; + } + continue; + } + if is_stream_chunk_frame(&frame) || frame_start_span_id(&frame).is_some() { + continue; + } + if let Some((exit, exit_span)) = frame_exit(&frame) { + panic!( + "expected member registered event before exit, got exit={exit} span_id={exit_span} expected_span_id={span_id}" + ); + } + if let Some(message) = frame_error_message(&frame) { + panic!("unexpected streamed execute error before registered event: {message}"); + } + panic!("unexpected streamed execute frame before registered event: {frame:?}"); + } +} + +async fn assert_stream_exit_for_span(body: &mut B, buffer: &mut Vec, span_id: &str) +where + B: HttpBody + Unpin, + B::Error: std::fmt::Debug + std::fmt::Display, +{ + loop { + let frame = next_stream_frame_value(body, buffer).await; + if is_stream_chunk_frame(&frame) + || frame_event(&frame).is_some() + || frame_start_span_id(&frame).is_some() + { + continue; + } + if let Some((exit, exit_span)) = frame_exit(&frame) { + assert_eq!(exit_span, span_id); + assert_eq!(exit, 130, "SIGINT exit should follow shell convention"); + return; + } + if let Some(message) = frame_error_message(&frame) { + panic!("unexpected streamed execute error: {message}"); + } + panic!("unexpected streamed execute frame: {frame:?}"); + } +} + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] async fn execute_stream_cancel_interrupts_live_non_pty_command() { let service = match WorldAgentService::new() { @@ -96,11 +316,10 @@ async fn execute_stream_cancel_interrupts_live_non_pty_command() { let mut body = response.into_body(); let mut buffer = Vec::new(); - let start = next_stream_frame(&mut body, &mut buffer).await; - let span_id = match start { - ExecuteStreamFrame::Start { span_id } => span_id, - other => panic!("expected start frame, got {other:?}"), - }; + let start = next_stream_frame_value(&mut body, &mut buffer).await; + let span_id = frame_start_span_id(&start) + .map(ToOwned::to_owned) + .unwrap_or_else(|| panic!("expected start frame, got {start:?}")); tokio::time::sleep(Duration::from_millis(500)).await; @@ -114,26 +333,257 @@ async fn execute_stream_cancel_interrupts_live_non_pty_command() { assert!(cancel.delivered, "expected cancel signal delivery"); loop { - match next_stream_frame(&mut body, &mut buffer).await { - ExecuteStreamFrame::Stdout { .. } - | ExecuteStreamFrame::Stderr { .. } - | ExecuteStreamFrame::Event { .. } => continue, - ExecuteStreamFrame::Exit { - exit, - span_id: exit_span, - process_telemetry: _, - .. - } => { - assert_eq!(exit_span, span_id); - assert_eq!(exit, 130, "SIGINT exit should follow shell convention"); - break; - } - ExecuteStreamFrame::Error { message } => { - panic!("unexpected streamed execute error: {message}"); - } - ExecuteStreamFrame::Start { .. } => { - panic!("unexpected duplicate start frame"); - } + let frame = next_stream_frame_value(&mut body, &mut buffer).await; + if is_stream_chunk_frame(&frame) + || frame_event(&frame).is_some() + || frame_start_span_id(&frame).is_some() + { + continue; + } + if let Some((exit, exit_span)) = frame_exit(&frame) { + assert_eq!(exit_span, span_id); + assert_eq!(exit, 130, "SIGINT exit should follow shell convention"); + break; + } + if let Some(message) = frame_error_message(&frame) { + panic!("unexpected streamed execute error: {message}"); } + panic!("unexpected streamed execute frame: {frame:?}"); } } + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn execute_stream_cancel_interrupts_live_member_runtime() { + let service = match WorldAgentService::new() { + Ok(svc) => svc, + Err(err) => { + eprintln!("skipping member streamed cancel test: service init failed: {err}"); + return; + } + }; + + let tmp = tempdir().expect("tempdir"); + let codex_member_binary = write_fake_member_runtime(tmp.path()); + let world_spec = WorldSpec { + reuse_session: true, + reuse_mode: WorldReuseMode::SharedOrchestration(SharedWorldOwnerSpec { + orchestration_session_id: "orch-streamed-member-cancel".to_string(), + action: SharedWorldOwnerAction::AttachOrCreate, + }), + isolate_network: false, + limits: world_api::ResourceLimits::default(), + enable_preload: false, + allowed_domains: Vec::new(), + project_dir: tmp.path().to_path_buf(), + always_isolate: true, + fs_mode: substrate_common::WorldFsMode::Writable, + }; + let world = match service.ensure_session_world(&world_spec) { + Ok(world) => world, + Err(err) => { + eprintln!("skipping member streamed cancel test: failed to ensure shared world: {err}"); + return; + } + }; + let Some(binding) = world.shared_binding.clone() else { + eprintln!("skipping member streamed cancel test: shared world binding missing"); + return; + }; + + let request = make_member_dispatch_request( + tmp.path(), + &codex_member_binary, + &binding.world_id, + binding.world_generation, + ); + let response = match service.execute_stream(request).await { + Ok(response) => response, + Err(err) => { + eprintln!("skipping member streamed cancel test: execute_stream failed: {err}"); + return; + } + }; + + let mut body = response.into_body(); + let mut buffer = Vec::new(); + + let start = next_stream_frame_value(&mut body, &mut buffer).await; + let span_id = frame_start_span_id(&start) + .map(ToOwned::to_owned) + .unwrap_or_else(|| panic!("expected start frame, got {start:?}")); + + let ready = next_registered_frame(&mut body, &mut buffer, &span_id).await; + assert_registered_event( + &ready, + "ash_member_cancel_test", + "cli:codex", + &binding.world_id, + binding.world_generation, + &span_id, + ); + + let cancel = service + .execute_cancel(ExecuteCancelRequestV1 { + span_id: span_id.clone(), + sig: "INT".to_string(), + }) + .await + .expect("member execute_cancel should succeed"); + assert!(cancel.delivered, "expected member cancel delivery"); + + assert_stream_exit_for_span(&mut body, &mut buffer, &span_id).await; +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn member_runtime_backend_slots_allow_distinct_backends_and_reject_duplicates() { + let service = match WorldAgentService::new() { + Ok(svc) => svc, + Err(err) => { + eprintln!("skipping member backend slot test: service init failed: {err}"); + return; + } + }; + + let tmp = tempdir().expect("tempdir"); + let codex_member_binary = write_fake_member_runtime(tmp.path()); + let claude_member_binary = write_fake_claude_member_runtime(tmp.path()); + let orchestration_session_id = "orch-streamed-member-backend-slots"; + let world_spec = WorldSpec { + reuse_session: true, + reuse_mode: WorldReuseMode::SharedOrchestration(SharedWorldOwnerSpec { + orchestration_session_id: orchestration_session_id.to_string(), + action: SharedWorldOwnerAction::AttachOrCreate, + }), + isolate_network: false, + limits: world_api::ResourceLimits::default(), + enable_preload: false, + allowed_domains: Vec::new(), + project_dir: tmp.path().to_path_buf(), + always_isolate: true, + fs_mode: substrate_common::WorldFsMode::Writable, + }; + let world = match service.ensure_session_world(&world_spec) { + Ok(world) => world, + Err(err) => { + eprintln!("skipping member backend slot test: failed to ensure shared world: {err}"); + return; + } + }; + let Some(binding) = world.shared_binding.clone() else { + eprintln!("skipping member backend slot test: shared world binding missing"); + return; + }; + + let codex_request = make_member_dispatch_request_with_backend( + tmp.path(), + &codex_member_binary, + &binding.world_id, + binding.world_generation, + orchestration_session_id, + "ash_member_codex_slot_test", + "run-member-codex-slot-test", + "cli:codex", + MemberRuntimeBackendKindV1::Codex, + ); + let codex_response = service + .execute_stream(codex_request) + .await + .expect("codex member launch should succeed"); + let mut codex_body = codex_response.into_body(); + let mut codex_buffer = Vec::new(); + let codex_start = next_stream_frame_value(&mut codex_body, &mut codex_buffer).await; + let codex_span_id = frame_start_span_id(&codex_start) + .map(ToOwned::to_owned) + .unwrap_or_else(|| panic!("expected start frame, got {codex_start:?}")); + let codex_ready = + next_registered_frame(&mut codex_body, &mut codex_buffer, &codex_span_id).await; + assert_registered_event( + &codex_ready, + "ash_member_codex_slot_test", + "cli:codex", + &binding.world_id, + binding.world_generation, + &codex_span_id, + ); + + let claude_request = make_member_dispatch_request_with_backend( + tmp.path(), + &claude_member_binary, + &binding.world_id, + binding.world_generation, + orchestration_session_id, + "ash_member_claude_slot_test", + "run-member-claude-slot-test", + "cli:claude_code", + MemberRuntimeBackendKindV1::ClaudeCode, + ); + let claude_response = service + .execute_stream(claude_request) + .await + .expect("claude_code member launch should succeed"); + let mut claude_body = claude_response.into_body(); + let mut claude_buffer = Vec::new(); + let claude_start = next_stream_frame_value(&mut claude_body, &mut claude_buffer).await; + let claude_span_id = frame_start_span_id(&claude_start) + .map(ToOwned::to_owned) + .unwrap_or_else(|| panic!("expected start frame, got {claude_start:?}")); + let claude_ready = + next_registered_frame(&mut claude_body, &mut claude_buffer, &claude_span_id).await; + assert_registered_event( + &claude_ready, + "ash_member_claude_slot_test", + "cli:claude_code", + &binding.world_id, + binding.world_generation, + &claude_span_id, + ); + + let duplicate_request = make_member_dispatch_request_with_backend( + tmp.path(), + &codex_member_binary, + &binding.world_id, + binding.world_generation, + orchestration_session_id, + "ash_member_codex_duplicate_slot_test", + "run-member-codex-duplicate-slot-test", + "cli:codex", + MemberRuntimeBackendKindV1::Codex, + ); + let duplicate_err = service + .execute_stream(duplicate_request) + .await + .expect_err("duplicate cli:codex slot should fail closed"); + let duplicate_message = duplicate_err.to_string(); + assert!( + duplicate_message.contains("a retained world member is already active"), + "unexpected duplicate error: {duplicate_message}" + ); + assert!( + duplicate_message.contains("backend_id cli:codex"), + "unexpected duplicate error: {duplicate_message}" + ); + + let codex_cancel = service + .execute_cancel(ExecuteCancelRequestV1 { + span_id: codex_span_id.clone(), + sig: "INT".to_string(), + }) + .await + .expect("codex execute_cancel should succeed"); + assert!(codex_cancel.delivered, "expected codex cancel delivery"); + + let claude_cancel = service + .execute_cancel(ExecuteCancelRequestV1 { + span_id: claude_span_id.clone(), + sig: "INT".to_string(), + }) + .await + .expect("claude_code execute_cancel should succeed"); + assert!( + claude_cancel.delivered, + "expected claude_code cancel delivery" + ); + + assert_stream_exit_for_span(&mut codex_body, &mut codex_buffer, &codex_span_id).await; + assert_stream_exit_for_span(&mut claude_body, &mut claude_buffer, &claude_span_id).await; +} diff --git a/crates/world-agent/tests/wfgad3_wildcard_deny_symlink_handling.rs b/crates/world-agent/tests/wfgad3_wildcard_deny_symlink_handling.rs index fbb4a5bd7..480bddf99 100644 --- a/crates/world-agent/tests/wfgad3_wildcard_deny_symlink_handling.rs +++ b/crates/world-agent/tests/wfgad3_wildcard_deny_symlink_handling.rs @@ -100,8 +100,10 @@ fn execute_non_pty( agent_id: "wfgad3-test".to_string(), budget: None, policy_snapshot, + shared_world: None, world_network: None, world_fs_mode: None, + member_dispatch: None, }; let rt = Runtime::new().expect("runtime"); diff --git a/crates/world-agent/tests/wfgad5_strict_deny_lockdown.rs b/crates/world-agent/tests/wfgad5_strict_deny_lockdown.rs index 18e9adc24..5edbe839f 100644 --- a/crates/world-agent/tests/wfgad5_strict_deny_lockdown.rs +++ b/crates/world-agent/tests/wfgad5_strict_deny_lockdown.rs @@ -74,8 +74,10 @@ fn execute_non_pty( agent_id: "wfgad5-test".to_string(), budget: None, policy_snapshot, + shared_world: None, world_network: None, world_fs_mode: None, + member_dispatch: None, }; let rt = Runtime::new().expect("runtime"); diff --git a/crates/world-api/src/lib.rs b/crates/world-api/src/lib.rs index 963517369..a2f1159ad 100644 --- a/crates/world-api/src/lib.rs +++ b/crates/world-api/src/lib.rs @@ -9,11 +9,100 @@ use serde::{Deserialize, Serialize}; use std::collections::HashMap; use std::path::PathBuf; +/// Reuse semantics for a world allocation request. +#[derive(Clone, Debug, Serialize, Deserialize, PartialEq, Eq, Default)] +#[serde(rename_all = "snake_case")] +pub enum WorldReuseMode { + /// Use the legacy compatibility lookup rules for generic reusable worlds. + #[default] + GenericCompatible, + /// Use the explicit shared-world ownership contract. + SharedOrchestration(SharedWorldOwnerSpec), +} + +impl WorldReuseMode { + pub fn shared_owner(&self) -> Option<&SharedWorldOwnerSpec> { + match self { + Self::GenericCompatible => None, + Self::SharedOrchestration(owner) => Some(owner), + } + } +} + +#[derive(Clone, Debug, Serialize, Deserialize, PartialEq, Eq)] +pub struct SharedWorldOwnerSpec { + pub orchestration_session_id: String, + pub action: SharedWorldOwnerAction, +} + +#[derive(Clone, Debug, Serialize, Deserialize, PartialEq, Eq)] +#[serde(rename_all = "snake_case")] +pub enum SharedWorldOwnerAction { + AttachOrCreate, + ReplaceExpectedGeneration { + expected_generation: u64, + reason: String, + }, +} + +#[derive(Clone, Debug, Serialize, Deserialize, PartialEq, Eq)] +#[serde(rename_all = "snake_case")] +pub enum SharedWorldBindingState { + Active, + Replacing, + Replaced, + Abandoned, +} + +#[derive(Clone, Debug, Serialize, Deserialize, PartialEq, Eq)] +pub struct SharedWorldBindingSnapshot { + pub orchestration_session_id: String, + pub world_id: String, + pub world_generation: u64, + pub binding_state: SharedWorldBindingState, +} + +#[derive(Clone, Debug, Serialize, Deserialize, PartialEq, Eq)] +#[serde(rename_all = "snake_case")] +pub enum MemberRuntimeBackendKindV1 { + Codex, + ClaudeCode, +} + +#[derive(Clone, Debug, Serialize, Deserialize, PartialEq, Eq)] +pub struct ResolvedMemberRuntimeDescriptorV1 { + pub backend_kind: MemberRuntimeBackendKindV1, + pub binary_path: String, +} + +#[derive(Clone, Debug, Serialize, Deserialize, PartialEq, Eq)] +pub struct MemberDispatchRequestV1 { + pub schema_version: u32, + pub orchestration_session_id: String, + pub participant_id: String, + pub orchestrator_participant_id: String, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub parent_participant_id: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub resumed_from_participant_id: Option, + pub backend_id: String, + pub protocol: String, + pub run_id: String, + pub world_id: String, + pub world_generation: u64, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub initial_prompt: Option, + pub resolved_runtime: ResolvedMemberRuntimeDescriptorV1, +} + /// Configuration for a world execution environment. #[derive(Clone, Debug, Serialize, Deserialize)] pub struct WorldSpec { /// Whether to reuse an existing session world. pub reuse_session: bool, + /// Whether to use generic compatibility reuse or explicit shared-world ownership. + #[serde(default)] + pub reuse_mode: WorldReuseMode, /// Whether to isolate network access. pub isolate_network: bool, /// Resource limits for the world. @@ -35,6 +124,7 @@ impl Default for WorldSpec { fn default() -> Self { Self { reuse_session: true, + reuse_mode: WorldReuseMode::GenericCompatible, isolate_network: true, limits: ResourceLimits::default(), enable_preload: false, @@ -74,6 +164,9 @@ impl Default for ResourceLimits { pub struct WorldHandle { /// Unique identifier for this world instance. pub id: String, + /// Authoritative shared-world binding proof when explicit owner mode is active. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub shared_binding: Option, } /// Request to execute a command in a world. @@ -90,6 +183,12 @@ pub struct ExecRequest { /// Optional span identifier to correlate fs_diff and telemetry #[serde(skip_serializing_if = "Option::is_none")] pub span_id: Option, + /// Optional shared-world ownership intent for orchestration-aware backends. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub shared_world: Option, + /// Optional member dispatch payload for orchestration-aware backends. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub member_dispatch: Option, } /// Result of command execution. @@ -183,6 +282,7 @@ mod tests { fn test_world_spec_default() { let spec = WorldSpec::default(); assert!(spec.reuse_session); + assert_eq!(spec.reuse_mode, WorldReuseMode::GenericCompatible); assert!(spec.isolate_network); assert!(!spec.enable_preload); assert!(spec.allowed_domains.contains(&"github.com".to_string())); @@ -210,4 +310,106 @@ mod tests { _ => panic!("Unexpected default backend"), } } + + #[test] + fn shared_world_contract_round_trips_with_canonical_shape() { + let mode = WorldReuseMode::SharedOrchestration(SharedWorldOwnerSpec { + orchestration_session_id: "orch_123".into(), + action: SharedWorldOwnerAction::ReplaceExpectedGeneration { + expected_generation: 7, + reason: "restart".into(), + }, + }); + let json = serde_json::to_string(&mode).expect("serialize reuse mode"); + assert_eq!( + json, + r#"{"shared_orchestration":{"orchestration_session_id":"orch_123","action":{"replace_expected_generation":{"expected_generation":7,"reason":"restart"}}}}"# + ); + + let binding = SharedWorldBindingSnapshot { + orchestration_session_id: "orch_123".into(), + world_id: "wld_123".into(), + world_generation: 8, + binding_state: SharedWorldBindingState::Active, + }; + let binding_json = serde_json::to_string(&binding).expect("serialize binding"); + assert_eq!( + binding_json, + r#"{"orchestration_session_id":"orch_123","world_id":"wld_123","world_generation":8,"binding_state":"active"}"# + ); + + let spec = WorldSpec { + reuse_mode: mode.clone(), + ..WorldSpec::default() + }; + let decoded: WorldSpec = + serde_json::from_str(&serde_json::to_string(&spec).expect("serialize spec")) + .expect("deserialize spec"); + assert_eq!(decoded.reuse_mode, mode); + + let handle = WorldHandle { + id: "wld_123".into(), + shared_binding: Some(binding.clone()), + }; + let handle_back: WorldHandle = + serde_json::from_str(&serde_json::to_string(&handle).expect("serialize handle")) + .expect("deserialize handle"); + assert_eq!( + handle_back + .shared_binding + .expect("shared binding should deserialize"), + binding + ); + + let exec_request = ExecRequest { + cmd: "echo hi".into(), + cwd: PathBuf::from("/tmp"), + env: HashMap::new(), + pty: false, + span_id: Some("spn_123".into()), + shared_world: Some(SharedWorldOwnerSpec { + orchestration_session_id: "orch_123".into(), + action: SharedWorldOwnerAction::AttachOrCreate, + }), + member_dispatch: Some(MemberDispatchRequestV1 { + schema_version: 1, + orchestration_session_id: "orch_123".into(), + participant_id: "participant_123".into(), + orchestrator_participant_id: "participant_root".into(), + parent_participant_id: Some("participant_parent".into()), + resumed_from_participant_id: None, + backend_id: "backend_123".into(), + protocol: "stdio".into(), + run_id: "run_123".into(), + world_id: "wld_123".into(), + world_generation: 8, + initial_prompt: Some("Continue".into()), + resolved_runtime: ResolvedMemberRuntimeDescriptorV1 { + backend_kind: MemberRuntimeBackendKindV1::Codex, + binary_path: "/usr/bin/env".into(), + }, + }), + }; + let exec_back: ExecRequest = serde_json::from_str( + &serde_json::to_string(&exec_request).expect("serialize exec request"), + ) + .expect("deserialize exec request"); + assert_eq!( + exec_back + .shared_world + .expect("shared world should deserialize"), + SharedWorldOwnerSpec { + orchestration_session_id: "orch_123".into(), + action: SharedWorldOwnerAction::AttachOrCreate, + } + ); + assert_eq!( + exec_back + .member_dispatch + .expect("member dispatch should deserialize"), + exec_request + .member_dispatch + .expect("original member dispatch should exist") + ); + } } diff --git a/crates/world-mac-lima/examples/mac_backend_smoke.rs b/crates/world-mac-lima/examples/mac_backend_smoke.rs index 6b5011222..b3cf02f18 100644 --- a/crates/world-mac-lima/examples/mac_backend_smoke.rs +++ b/crates/world-mac-lima/examples/mac_backend_smoke.rs @@ -29,6 +29,8 @@ fn main() -> Result<()> { env: std::env::vars().collect(), pty: false, span_id: None, + shared_world: None, + member_dispatch: None, }; let res = backend.exec(&handle, req)?; diff --git a/crates/world-mac-lima/src/forwarding.rs b/crates/world-mac-lima/src/forwarding.rs index b874a33c2..ca0e60d54 100644 --- a/crates/world-mac-lima/src/forwarding.rs +++ b/crates/world-mac-lima/src/forwarding.rs @@ -313,7 +313,24 @@ fn create_ssh_uds_forwarding(vm_name: &str) -> Result { std::thread::sleep(Duration::from_millis(500)); } - // Best-effort: if ssh is still running but socket never appeared, capture any stderr already emitted. + // Timed out waiting for the forwarded socket to become healthy. Terminate the ssh helper before + // draining stderr so we do not block forever on an otherwise-live tunnel process. + match child.try_wait() { + Ok(None) => { + if let Err(err) = child.kill() { + warn!("Failed to kill timed out SSH forwarding process: {err}"); + } + if let Err(err) = child.wait() { + warn!("Failed to wait on timed out SSH forwarding process: {err}"); + } + } + Ok(Some(_)) => {} + Err(err) => { + warn!("Failed to poll timed out SSH forwarding process: {err}"); + } + } + + // Best-effort: capture any stderr now that the child has terminated. if let Some(mut stderr) = child.stderr.take() { use std::io::Read as _; let _ = stderr.read_to_string(&mut stderr_buf); diff --git a/crates/world-mac-lima/src/lib.rs b/crates/world-mac-lima/src/lib.rs index a801f3bd5..3d1c18e71 100644 --- a/crates/world-mac-lima/src/lib.rs +++ b/crates/world-mac-lima/src/lib.rs @@ -6,16 +6,20 @@ use agent_api_client::AgentClient; use agent_api_types::{ - ExecuteRequest, ExecuteResponse, PolicySnapshotV3, PolicySnapshotWorldFsDimensionV3, - PolicySnapshotWorldFsFailClosedV3, PolicySnapshotWorldFsV3, PolicySnapshotWorldFsWriteV3, - WorldFsMode, + ExecuteRequest, ExecuteResponse, MemberDispatchRequestV1, + MemberRuntimeBackendKindV1 as AgentMemberRuntimeBackendKindV1, PolicySnapshotV3, + PolicySnapshotWorldFsDimensionV3, PolicySnapshotWorldFsFailClosedV3, PolicySnapshotWorldFsV3, + PolicySnapshotWorldFsWriteV3, ResolvedMemberRuntimeDescriptorV1, WorldFsMode, }; use anyhow::{Context, Result}; use serde::Deserialize; use std::path::PathBuf; use substrate_common::FsDiff; use tokio::runtime::Runtime; -use world_api::{ExecRequest, ExecResult, WorldBackend, WorldHandle, WorldSpec}; +use world_api::{ + ExecRequest, ExecResult, SharedWorldBindingSnapshot, SharedWorldBindingState, + SharedWorldOwnerAction, WorldBackend, WorldHandle, WorldSpec, +}; pub mod forwarding; mod limactl; @@ -34,7 +38,11 @@ pub struct MacLimaBackend { runtime: Option, forwarding: std::sync::Mutex>, session_cache: std::sync::Mutex>, + shared_owner_cache: std::sync::Mutex>, + shared_owner_mutex: std::sync::Mutex<()>, fs_mode: std::sync::Mutex, + #[cfg(test)] + session_setup_override: Option>, } impl MacLimaBackend { @@ -59,7 +67,11 @@ impl MacLimaBackend { runtime: Some(runtime), forwarding: std::sync::Mutex::new(None), session_cache: std::sync::Mutex::new(None), + shared_owner_cache: std::sync::Mutex::new(std::collections::HashMap::new()), + shared_owner_mutex: std::sync::Mutex::new(()), fs_mode: std::sync::Mutex::new(WorldFsMode::Writable), + #[cfg(test)] + session_setup_override: None, }) } @@ -93,6 +105,15 @@ impl MacLimaBackend { Ok(backend) } + #[cfg(test)] + pub(crate) fn with_session_setup_mock( + session_setup_override: std::sync::Arc, + ) -> Result { + let mut backend = Self::new()?; + backend.session_setup_override = Some(session_setup_override); + Ok(backend) + } + fn ensure_vm_running(&self) -> Result<()> { tracing::debug!("Checking if Lima VM '{}' is running", self.vm_name); @@ -250,6 +271,126 @@ impl MacLimaBackend { Ok(()) } + fn ensure_session_setup(&self) -> Result<()> { + #[cfg(test)] + if let Some(override_impl) = &self.session_setup_override { + return override_impl.ensure_setup(); + } + + self.ensure_vm_running()?; + self.ensure_forwarding()?; + Ok(()) + } + + async fn verify_agent_ready(&self) -> Result<()> { + #[cfg(test)] + if let Some(override_impl) = &self.session_setup_override { + return override_impl.verify_ready(); + } + + let client = self.build_agent_client()?; + let caps = client + .capabilities() + .await + .context("Failed to verify agent connectivity")?; + + tracing::info!("Agent connectivity verified: {:?}", caps); + Ok(()) + } + + pub async fn ensure_persistent_session_ready_async(&self) -> Result<()> { + self.ensure_session_setup()?; + self.verify_agent_ready().await + } + + fn ensure_agent_ready(&self) -> Result<()> { + self.block_on_compat(self.ensure_persistent_session_ready_async()) + } + + fn shared_world_id(&self, orchestration_session_id: &str, world_generation: u64) -> String { + format!( + "vm:{}:{}:{}", + self.vm_name, orchestration_session_id, world_generation + ) + } + + fn ensure_shared_owner_session( + &self, + owner: &world_api::SharedWorldOwnerSpec, + ) -> Result { + let _guard = self + .shared_owner_mutex + .lock() + .map_err(|e| anyhow::anyhow!("Lock poisoned: {}", e))?; + let mut cache = self + .shared_owner_cache + .lock() + .map_err(|e| anyhow::anyhow!("Lock poisoned: {}", e))?; + + match &owner.action { + SharedWorldOwnerAction::AttachOrCreate => { + if let Some(handle) = cache.get(&owner.orchestration_session_id) { + return Ok(handle.clone()); + } + + let world_id = self.shared_world_id(&owner.orchestration_session_id, 0); + let handle = WorldHandle { + id: world_id.clone(), + shared_binding: Some(SharedWorldBindingSnapshot { + orchestration_session_id: owner.orchestration_session_id.clone(), + world_id, + world_generation: 0, + binding_state: SharedWorldBindingState::Active, + }), + }; + cache.insert(owner.orchestration_session_id.clone(), handle.clone()); + Ok(handle) + } + SharedWorldOwnerAction::ReplaceExpectedGeneration { + expected_generation, + reason: _, + } => { + let current = cache + .get(&owner.orchestration_session_id) + .cloned() + .ok_or_else(|| { + anyhow::anyhow!( + "no active shared world found for orchestration session {}", + owner.orchestration_session_id + ) + })?; + let current_generation = current + .shared_binding + .as_ref() + .map(|binding| binding.world_generation) + .ok_or_else(|| anyhow::anyhow!("active shared world missing binding proof"))?; + if current_generation != *expected_generation { + anyhow::bail!( + "shared world generation conflict for {}: expected {}, found {}", + owner.orchestration_session_id, + expected_generation, + current_generation + ); + } + + let next_generation = expected_generation + 1; + let world_id = + self.shared_world_id(&owner.orchestration_session_id, next_generation); + let handle = WorldHandle { + id: world_id.clone(), + shared_binding: Some(SharedWorldBindingSnapshot { + orchestration_session_id: owner.orchestration_session_id.clone(), + world_id, + world_generation: next_generation, + binding_state: SharedWorldBindingState::Active, + }), + }; + cache.insert(owner.orchestration_session_id.clone(), handle.clone()); + Ok(handle) + } + } + } + fn get_agent_endpoint(&self) -> Result { let forwarding = self .forwarding @@ -307,8 +448,10 @@ impl MacLimaBackend { agent_id: "world-mac-lima".to_string(), budget: None, policy_snapshot, + shared_world: req.shared_world.clone(), world_network: None, world_fs_mode: Some(fs_mode), + member_dispatch: req.member_dispatch.as_ref().map(convert_member_dispatch), } } @@ -385,11 +528,14 @@ impl Drop for MacLimaBackend { impl WorldBackend for MacLimaBackend { fn ensure_session(&self, spec: &WorldSpec) -> Result { - self.ensure_vm_running()?; - self.ensure_forwarding()?; + self.ensure_agent_ready()?; self.store_fs_mode(spec.fs_mode)?; + if let Some(owner) = spec.reuse_mode.shared_owner() { + return self.ensure_shared_owner_session(owner); + } + // Cache session if requested if spec.reuse_session { let cache = self @@ -402,18 +548,13 @@ impl WorldBackend for MacLimaBackend { } } - // Verify connectivity via agent client - let client = self.build_agent_client()?; - let caps = self - .block_on_compat(async { client.capabilities().await }) - .context("Failed to verify agent connectivity")?; - - tracing::info!("Agent connectivity verified: {:?}", caps); - // Generate world ID let world_id = format!("vm:{}", self.vm_name); - let handle = WorldHandle { id: world_id }; + let handle = WorldHandle { + id: world_id.clone(), + shared_binding: None, + }; if spec.reuse_session { let mut cache = self @@ -469,6 +610,48 @@ impl WorldBackend for MacLimaBackend { } } +fn convert_member_dispatch( + dispatch: &world_api::MemberDispatchRequestV1, +) -> MemberDispatchRequestV1 { + MemberDispatchRequestV1 { + schema_version: dispatch.schema_version, + orchestration_session_id: dispatch.orchestration_session_id.clone(), + participant_id: dispatch.participant_id.clone(), + orchestrator_participant_id: dispatch.orchestrator_participant_id.clone(), + parent_participant_id: dispatch.parent_participant_id.clone(), + resumed_from_participant_id: dispatch.resumed_from_participant_id.clone(), + backend_id: dispatch.backend_id.clone(), + protocol: dispatch.protocol.clone(), + run_id: dispatch.run_id.clone(), + world_id: dispatch.world_id.clone(), + world_generation: dispatch.world_generation, + initial_prompt: dispatch.initial_prompt.clone(), + resolved_runtime: ResolvedMemberRuntimeDescriptorV1 { + backend_kind: convert_member_runtime_backend_kind( + dispatch.resolved_runtime.backend_kind.clone(), + ), + binary_path: dispatch.resolved_runtime.binary_path.clone(), + }, + } +} + +fn convert_member_runtime_backend_kind( + backend_kind: world_api::MemberRuntimeBackendKindV1, +) -> AgentMemberRuntimeBackendKindV1 { + match backend_kind { + world_api::MemberRuntimeBackendKindV1::Codex => AgentMemberRuntimeBackendKindV1::Codex, + world_api::MemberRuntimeBackendKindV1::ClaudeCode => { + AgentMemberRuntimeBackendKindV1::ClaudeCode + } + } +} + +#[cfg(test)] +pub(crate) trait SessionSetupMock: Send + Sync { + fn ensure_setup(&self) -> Result<()>; + fn verify_ready(&self) -> Result<()>; +} + #[cfg(test)] mod test_util { use std::sync::{LazyLock, Mutex, MutexGuard}; @@ -483,6 +666,34 @@ mod test_util { #[cfg(test)] mod tests { use super::*; + use std::sync::atomic::{AtomicUsize, Ordering}; + use std::sync::Arc; + + struct AlwaysReadySessionSetup { + setup_calls: AtomicUsize, + verify_calls: AtomicUsize, + } + + impl AlwaysReadySessionSetup { + fn new() -> Self { + Self { + setup_calls: AtomicUsize::new(0), + verify_calls: AtomicUsize::new(0), + } + } + } + + impl SessionSetupMock for AlwaysReadySessionSetup { + fn ensure_setup(&self) -> Result<()> { + self.setup_calls.fetch_add(1, Ordering::SeqCst); + Ok(()) + } + + fn verify_ready(&self) -> Result<()> { + self.verify_calls.fetch_add(1, Ordering::SeqCst); + Ok(()) + } + } #[test] fn test_backend_creation() { @@ -561,6 +772,28 @@ mod tests { env: std::collections::HashMap::new(), pty: false, span_id: None, + shared_world: Some(world_api::SharedWorldOwnerSpec { + orchestration_session_id: "orch_123".to_string(), + action: world_api::SharedWorldOwnerAction::AttachOrCreate, + }), + member_dispatch: Some(world_api::MemberDispatchRequestV1 { + schema_version: 1, + orchestration_session_id: "orch_123".to_string(), + participant_id: "participant_123".to_string(), + orchestrator_participant_id: "participant_root".to_string(), + parent_participant_id: None, + resumed_from_participant_id: None, + backend_id: "backend_123".to_string(), + protocol: "stdio".to_string(), + run_id: "run_123".to_string(), + world_id: "wld_123".to_string(), + world_generation: 0, + initial_prompt: Some("prompt".to_string()), + resolved_runtime: world_api::ResolvedMemberRuntimeDescriptorV1 { + backend_kind: world_api::MemberRuntimeBackendKindV1::Codex, + binary_path: "/usr/bin/env".to_string(), + }, + }), }; let fs_mode = backend.effective_fs_mode().expect("fs_mode"); let agent_req = backend.convert_exec_request(&req, fs_mode); @@ -569,6 +802,25 @@ mod tests { Some(WorldFsMode::ReadOnly), "mac backend should pass through env-derived fs mode" ); + assert_eq!(agent_req.shared_world, req.shared_world); + assert_eq!( + agent_req.member_dispatch.as_ref().map(|dispatch| ( + dispatch.orchestration_session_id.as_str(), + dispatch.participant_id.as_str(), + dispatch.orchestrator_participant_id.as_str(), + dispatch.backend_id.as_str(), + dispatch.world_id.as_str(), + dispatch.world_generation, + )), + Some(( + "orch_123", + "participant_123", + "participant_root", + "backend_123", + "wld_123", + 0, + )) + ); } match prev { @@ -576,4 +828,104 @@ mod tests { None => std::env::remove_var("SUBSTRATE_WORLD_FS_MODE"), } } + + #[test] + fn ensure_session_owner_mode_sets_authoritative_shared_binding() { + let _env_guard = crate::test_util::lock_env(); + let session_setup = Arc::new(AlwaysReadySessionSetup::new()); + let backend = + MacLimaBackend::with_session_setup_mock(session_setup.clone()).expect("backend"); + + let attach = WorldSpec { + reuse_mode: world_api::WorldReuseMode::SharedOrchestration( + world_api::SharedWorldOwnerSpec { + orchestration_session_id: "orch_123".to_string(), + action: world_api::SharedWorldOwnerAction::AttachOrCreate, + }, + ), + ..WorldSpec::default() + }; + let attached = backend + .ensure_session(&attach) + .expect("attach/create session"); + let attached_binding = attached.shared_binding.expect("attach/create binding"); + assert_eq!(attached_binding.orchestration_session_id, "orch_123"); + assert_eq!(attached_binding.world_id, attached.id); + assert_eq!(attached_binding.world_generation, 0); + assert_eq!( + attached_binding.binding_state, + SharedWorldBindingState::Active + ); + + let replace = WorldSpec { + reuse_mode: world_api::WorldReuseMode::SharedOrchestration( + world_api::SharedWorldOwnerSpec { + orchestration_session_id: "orch_123".to_string(), + action: world_api::SharedWorldOwnerAction::ReplaceExpectedGeneration { + expected_generation: attached_binding.world_generation, + reason: "restart".to_string(), + }, + }, + ), + ..WorldSpec::default() + }; + + let replaced = backend + .ensure_session(&replace) + .expect("replacement session"); + let binding = replaced.shared_binding.expect("replacement shared binding"); + assert_eq!(binding.orchestration_session_id, "orch_123"); + assert_eq!(binding.world_id, replaced.id); + assert_eq!(binding.world_generation, 1); + assert_eq!(binding.binding_state, SharedWorldBindingState::Active); + assert_ne!(binding.world_id, attached.id); + assert_eq!(session_setup.setup_calls.load(Ordering::SeqCst), 2); + assert_eq!(session_setup.verify_calls.load(Ordering::SeqCst), 2); + } + + #[test] + fn ensure_persistent_session_ready_async_runs_setup_and_verify() { + let _env_guard = crate::test_util::lock_env(); + let session_setup = Arc::new(AlwaysReadySessionSetup::new()); + let backend = + MacLimaBackend::with_session_setup_mock(session_setup.clone()).expect("backend"); + let runtime = MacLimaBackend::new_runtime().expect("runtime"); + + runtime + .block_on(backend.ensure_persistent_session_ready_async()) + .expect("async readiness"); + + assert_eq!(session_setup.setup_calls.load(Ordering::SeqCst), 1); + assert_eq!(session_setup.verify_calls.load(Ordering::SeqCst), 1); + } + + #[test] + fn ensure_session_owner_mode_replace_requires_existing_generation() { + let _env_guard = crate::test_util::lock_env(); + let backend = + MacLimaBackend::with_session_setup_mock(Arc::new(AlwaysReadySessionSetup::new())) + .expect("backend"); + + let replace = WorldSpec { + reuse_mode: world_api::WorldReuseMode::SharedOrchestration( + world_api::SharedWorldOwnerSpec { + orchestration_session_id: "orch_123".to_string(), + action: world_api::SharedWorldOwnerAction::ReplaceExpectedGeneration { + expected_generation: 4, + reason: "restart".to_string(), + }, + }, + ), + ..WorldSpec::default() + }; + + let err = backend + .ensure_session(&replace) + .expect_err("replace should fail closed"); + assert!( + err.to_string() + .contains("no active shared world found for orchestration session orch_123"), + "unexpected error: {err:#}" + ); + } } diff --git a/crates/world-windows-wsl/src/backend.rs b/crates/world-windows-wsl/src/backend.rs index d6f67ff71..20db89971 100644 --- a/crates/world-windows-wsl/src/backend.rs +++ b/crates/world-windows-wsl/src/backend.rs @@ -293,6 +293,14 @@ impl WindowsWslBackend { } } + pub(crate) fn ensure_ready(&self) -> Result<()> { + self.ensure_agent_ready() + } + + pub(crate) fn ensure_persistent_session_ready(&self) -> Result<()> { + self.ensure_agent_ready() + } + fn convert_exec_request(&self, req: &ExecRequest) -> Result { let cwd = to_wsl_path(&self.project_path, &req.cwd)?; let env = if req.env.is_empty() { @@ -336,8 +344,10 @@ impl WindowsWslBackend { agent_id: self.agent_id.clone(), budget: None, policy_snapshot, + shared_world: None, world_network: None, world_fs_mode: Some(fs_mode), + member_dispatch: None, }) } @@ -369,6 +379,7 @@ impl WindowsWslBackend { fn generate_world_handle(&self) -> WorldHandle { WorldHandle { id: format!("wsl:{}:{}", self.distro, Uuid::now_v7()), + shared_binding: None, } } @@ -389,12 +400,12 @@ impl WorldBackend for WindowsWslBackend { .expect("session cache poisoned") .clone() { - self.ensure_agent_ready()?; + self.ensure_persistent_session_ready()?; return Ok(handle); } } - self.ensure_agent_ready()?; + self.ensure_ready()?; let handle = self.generate_world_handle(); if spec.reuse_session { diff --git a/crates/world-windows-wsl/src/tests.rs b/crates/world-windows-wsl/src/tests.rs index fe95e2573..3fb00dc12 100644 --- a/crates/world-windows-wsl/src/tests.rs +++ b/crates/world-windows-wsl/src/tests.rs @@ -75,6 +75,7 @@ impl AgentApiMock for MockAgent { stderr_b64: BASE64_STANDARD.encode(b""), scopes_used: vec![], fs_diff: None, + shared_world: None, process_telemetry: ProcessTelemetry::not_supported_platform(), }) }) @@ -135,6 +136,28 @@ fn ensure_session_runs_warm_on_failure() { assert_eq!(warm_invocations.load(Ordering::SeqCst), 1); } +#[test] +fn ensure_ready_runs_warm_on_failure() { + let (backend, agent, warm_invocations) = test_backend_with_agent(); + agent.push_capabilities(Err(anyhow!("pipe missing"))); + agent.push_capabilities(Ok(json!({"ok": true}))); + + backend.ensure_ready().expect("ready after warm"); + assert_eq!(warm_invocations.load(Ordering::SeqCst), 1); +} + +#[test] +fn ensure_persistent_session_ready_runs_warm_on_failure() { + let (backend, agent, warm_invocations) = test_backend_with_agent(); + agent.push_capabilities(Err(anyhow!("pipe missing"))); + agent.push_capabilities(Ok(json!({"ok": true}))); + + backend + .ensure_persistent_session_ready() + .expect("persistent session ready after warm"); + assert_eq!(warm_invocations.load(Ordering::SeqCst), 1); +} + #[test] fn exec_routes_to_agent() { let (backend, agent, _) = test_backend_with_agent(); @@ -149,6 +172,7 @@ fn exec_routes_to_agent() { writes: vec![PathBuf::from("/mnt/c/repo/new.txt")], ..Default::default() }), + shared_world: None, process_telemetry: ProcessTelemetry::not_supported_platform(), })); @@ -160,6 +184,8 @@ fn exec_routes_to_agent() { env: std::iter::once(("KEY".to_string(), "VALUE".to_string())).collect(), pty: false, span_id: Some("span-123".to_string()), + shared_world: None, + member_dispatch: None, }; let result = backend.exec(&world, req.clone()).expect("exec result"); diff --git a/crates/world/Cargo.toml b/crates/world/Cargo.toml index 9f28b4896..41b6438b4 100644 --- a/crates/world/Cargo.toml +++ b/crates/world/Cargo.toml @@ -30,7 +30,6 @@ linux-raw-sys = { version = "0.4.15", features = ["landlock"] } [target.'cfg(all(target_os = "linux", not(target_env = "musl")))'.dependencies] nix = { version = "0.27", features = ["mount", "sched", "fs", "user"] } prctl = "1" -libseccomp = "0.3" which = "6" [target.'cfg(all(target_os = "linux", target_env = "musl"))'.dependencies] diff --git a/crates/world/src/isolation.rs b/crates/world/src/isolation.rs index be45d3cfe..591c17adb 100644 --- a/crates/world/src/isolation.rs +++ b/crates/world/src/isolation.rs @@ -282,31 +282,21 @@ impl LinuxIsolation { #[cfg(all(target_os = "linux", not(target_env = "musl")))] fn apply_seccomp_baseline(&self) -> Result<()> { - // Baseline seccomp using libseccomp: default allow; log risky syscalls - use libseccomp::{ScmpAction, ScmpFilterContext, ScmpSyscall}; - - let mut ctx = ScmpFilterContext::new_filter(ScmpAction::Allow) - .map_err(|e| anyhow::anyhow!("seccomp init failed: {}", e))?; - - // Mark dangerous syscalls to be logged (kernel must support SCMP_ACT_LOG) let dangerous = [ - "mount", - "umount2", - "pivot_root", - "keyctl", - "perf_event_open", - "bpf", + substrate_common::seccomp::SeccompRule::log("mount"), + substrate_common::seccomp::SeccompRule::log("umount2"), + substrate_common::seccomp::SeccompRule::log("pivot_root"), + substrate_common::seccomp::SeccompRule::log("keyctl"), + substrate_common::seccomp::SeccompRule::log("perf_event_open"), + substrate_common::seccomp::SeccompRule::log("bpf"), ]; - for name in dangerous { - if let Ok(num) = ScmpSyscall::from_name(name) { - ctx.add_rule(ScmpAction::Log, num) - .map_err(|e| anyhow::anyhow!("seccomp add_rule failed: {}", e))?; + match substrate_common::seccomp::install_allow_filter(&dangerous)? { + substrate_common::seccomp::SeccompInstallStatus::Installed => {} + substrate_common::seccomp::SeccompInstallStatus::Unavailable(reason) => { + tracing::warn!("seccomp baseline skipped: {reason}"); } } - - ctx.load() - .map_err(|e| anyhow::anyhow!("seccomp load failed: {}", e))?; Ok(()) } diff --git a/crates/world/src/lib.rs b/crates/world/src/lib.rs index 30a530842..acd7b327c 100644 --- a/crates/world/src/lib.rs +++ b/crates/world/src/lib.rs @@ -4,7 +4,10 @@ //! using Linux namespaces, cgroups v2, nftables, and other native security features. use anyhow::{Context, Result}; -use world_api::{ExecRequest, ExecResult, FsDiff, WorldBackend, WorldHandle, WorldSpec}; +use world_api::{ + ExecRequest, ExecResult, FsDiff, SharedWorldOwnerAction, SharedWorldOwnerSpec, WorldBackend, + WorldHandle, WorldSpec, +}; pub mod cgroups; pub mod copydiff; @@ -28,6 +31,7 @@ pub use session::SessionWorld; #[derive(Default)] pub struct LinuxLocalBackend { session_cache: std::sync::RwLock>, + shared_owner_mutex: std::sync::Mutex<()>, } impl LinuxLocalBackend { @@ -40,39 +44,371 @@ impl LinuxLocalBackend { self.find_compatible_session_from_root(&SessionWorld::shared_root_dir(), spec, false) } + fn world_handle(world: &SessionWorld) -> WorldHandle { + WorldHandle { + id: world.id.clone(), + shared_binding: world.shared_binding(), + } + } + fn find_compatible_session_from_root( &self, root_dir: &std::path::Path, spec: &WorldSpec, update_fs_mode: bool, + ) -> Result> { + if let Some(owner_spec) = spec.reuse_mode.shared_owner() { + return self.find_shared_owner_session_from_root( + root_dir, + spec, + owner_spec, + update_fs_mode, + ); + } + + if !spec.reuse_session { + return Ok(None); + } + + self.find_generic_session_from_root(root_dir, spec, update_fs_mode) + } + + fn find_generic_session_from_root( + &self, + root_dir: &std::path::Path, + spec: &WorldSpec, + update_fs_mode: bool, ) -> Result> { let mut cache = self .session_cache .write() .map_err(|e| anyhow::anyhow!("Failed to acquire session cache write lock: {}", e))?; - if let Some(world) = cache.values_mut().find(|world| world.compatible_with(spec)) { + if let Some(world) = cache + .values_mut() + .find(|world| world.is_generic_reusable_with(spec)) + { if update_fs_mode { world.spec.fs_mode = spec.fs_mode; } - return Ok(Some(WorldHandle { - id: world.id.clone(), - })); + return Ok(Some(Self::world_handle(world))); } - let Some(mut world) = SessionWorld::recover_compatible_from_root(root_dir, spec)? else { + let Some(mut world) = SessionWorld::recover_generic_compatible_from_root(root_dir, spec)? + else { return Ok(None); }; if update_fs_mode { world.spec.fs_mode = spec.fs_mode; } - let handle = WorldHandle { - id: world.id.clone(), + let handle = Self::world_handle(&world); + cache.insert(world.id.clone(), world); + Ok(Some(handle)) + } + + fn find_shared_owner_session_from_root( + &self, + root_dir: &std::path::Path, + spec: &WorldSpec, + owner_spec: &SharedWorldOwnerSpec, + update_fs_mode: bool, + ) -> Result> { + let mut cache = self + .session_cache + .write() + .map_err(|e| anyhow::anyhow!("Failed to acquire session cache write lock: {}", e))?; + + let matching_ids = cache + .iter() + .filter_map(|(world_id, world)| { + world + .is_shared_owner_reusable_with(spec, owner_spec) + .then_some(world_id.clone()) + }) + .collect::>(); + match matching_ids.as_slice() { + [] => {} + [world_id] => { + let world = cache + .get_mut(world_id) + .context("shared world missing from cache during reuse")?; + if update_fs_mode { + world.spec.fs_mode = spec.fs_mode; + } + return Ok(Some(Self::world_handle(world))); + } + _ => { + anyhow::bail!( + "multiple active shared worlds found for orchestration session {}", + owner_spec.orchestration_session_id + ); + } + } + + let Some(mut world) = + SessionWorld::recover_shared_active_from_root(root_dir, spec, owner_spec)? + else { + return Ok(None); }; + if update_fs_mode { + world.spec.fs_mode = spec.fs_mode; + } + let handle = Self::world_handle(&world); cache.insert(world.id.clone(), world); Ok(Some(handle)) } + fn create_generic_session(&self, spec: &WorldSpec) -> Result { + self.create_generic_session_in_root(&SessionWorld::shared_root_dir(), spec) + } + + fn create_generic_session_in_root( + &self, + root_dir: &std::path::Path, + spec: &WorldSpec, + ) -> Result { + let world = SessionWorld::ensure_started_in_root(spec.clone(), root_dir.to_path_buf()) + .context("Failed to create session world")?; + let handle = Self::world_handle(&world); + let mut cache = self + .session_cache + .write() + .map_err(|e| anyhow::anyhow!("Failed to acquire session cache write lock: {}", e))?; + cache.insert(world.id.clone(), world); + Ok(handle) + } + + fn create_shared_owner_session_in_root( + &self, + root_dir: &std::path::Path, + spec: &WorldSpec, + owner_spec: &SharedWorldOwnerSpec, + world_generation: u64, + last_restart_reason: Option, + ) -> Result { + let world = SessionWorld::ensure_started_for_shared_owner_at_root( + root_dir.to_path_buf(), + spec.clone(), + owner_spec.orchestration_session_id.clone(), + world_generation, + last_restart_reason, + ) + .context("Failed to create shared session world")?; + let handle = Self::world_handle(&world); + let mut cache = self + .session_cache + .write() + .map_err(|e| anyhow::anyhow!("Failed to acquire session cache write lock: {}", e))?; + cache.insert(world.id.clone(), world); + Ok(handle) + } + + fn create_shared_owner_session_in_root_with_world_id( + &self, + root_dir: &std::path::Path, + spec: &WorldSpec, + owner_spec: &SharedWorldOwnerSpec, + world_generation: u64, + last_restart_reason: Option, + world_id: String, + ) -> Result { + let world = SessionWorld::ensure_started_for_shared_owner_at_root_with_world_id( + root_dir.to_path_buf(), + spec.clone(), + owner_spec.orchestration_session_id.clone(), + world_generation, + last_restart_reason, + world_id, + ) + .context("Failed to create shared session world")?; + let handle = Self::world_handle(&world); + let mut cache = self + .session_cache + .write() + .map_err(|e| anyhow::anyhow!("Failed to acquire session cache write lock: {}", e))?; + cache.insert(world.id.clone(), world); + Ok(handle) + } + + fn replace_shared_owner_session_from_root( + &self, + root_dir: &std::path::Path, + spec: &WorldSpec, + owner_spec: &SharedWorldOwnerSpec, + expected_generation: u64, + reason: String, + ) -> Result { + let replacement_reason = reason.clone(); + self.replace_shared_owner_session_from_root_with_creator( + root_dir, + spec, + owner_spec, + expected_generation, + reason, + |replacement_world_id| { + self.create_shared_owner_session_in_root_with_world_id( + root_dir, + spec, + owner_spec, + expected_generation + 1, + Some(replacement_reason), + replacement_world_id, + ) + }, + ) + } + + fn replace_shared_owner_session_from_root_with_creator( + &self, + root_dir: &std::path::Path, + spec: &WorldSpec, + owner_spec: &SharedWorldOwnerSpec, + expected_generation: u64, + reason: String, + create_replacement: F, + ) -> Result + where + F: FnOnce(String) -> Result, + { + let handle = self + .find_shared_owner_session_from_root(root_dir, spec, owner_spec, true)? + .ok_or_else(|| { + anyhow::anyhow!( + "no active shared world found for orchestration session {}", + owner_spec.orchestration_session_id + ) + })?; + let current_generation = handle + .shared_binding + .as_ref() + .map(|binding| binding.world_generation) + .ok_or_else(|| anyhow::anyhow!("active shared world missing binding proof"))?; + if current_generation != expected_generation { + anyhow::bail!( + "shared world generation conflict for {}: expected {}, found {}", + owner_spec.orchestration_session_id, + expected_generation, + current_generation + ); + } + + let replacement_world_id = format!("wld_{}", uuid::Uuid::now_v7()); + + { + let mut cache = self.session_cache.write().map_err(|e| { + anyhow::anyhow!("Failed to acquire session cache write lock: {}", e) + })?; + let world = cache + .get_mut(&handle.id) + .context("replaced shared world missing from cache")?; + world.set_shared_binding_state( + world_api::SharedWorldBindingState::Replacing, + Some(reason.clone()), + )?; + } + + let replacement_handle = match create_replacement(replacement_world_id.clone()) { + Ok(handle) => handle, + Err(create_err) => { + let rollback_err = { + let mut cache = self.session_cache.write().map_err(|e| { + anyhow::anyhow!("Failed to acquire session cache write lock: {}", e) + })?; + let world = cache + .get_mut(&handle.id) + .context("rollback shared world missing from cache")?; + world.set_shared_binding_state(world_api::SharedWorldBindingState::Active, None) + }; + + let cleanup_err = match rollback_err { + Ok(()) => { + self.cleanup_partial_shared_world_root(root_dir, &replacement_world_id) + } + Err(_) => Ok(()), + }; + + let mut message = format!("failed to create replacement world: {create_err:#}"); + if let Err(err) = rollback_err { + message.push_str(&format!("; rollback failed: {err:#}")); + } + if let Err(err) = cleanup_err { + message.push_str(&format!("; cleanup failed: {err:#}")); + } + return Err(anyhow::anyhow!(message)); + } + }; + + let finalize_result = { + let mut cache = self.session_cache.write().map_err(|e| { + anyhow::anyhow!("Failed to acquire session cache write lock: {}", e) + })?; + let world = cache + .get_mut(&handle.id) + .context("finalized shared world missing from cache")?; + world.set_shared_binding_state( + world_api::SharedWorldBindingState::Replaced, + Some(reason), + ) + }; + if let Err(err) = finalize_result { + tracing::warn!( + error = %err, + world_id = %handle.id, + "shared world replacement committed but old world finalize failed" + ); + } + + Ok(replacement_handle) + } + + fn cleanup_partial_shared_world_root( + &self, + root_dir: &std::path::Path, + world_id: &str, + ) -> Result<()> { + let partial_root = root_dir.join(world_id); + match std::fs::remove_dir_all(&partial_root) { + Ok(()) => Ok(()), + Err(err) if err.kind() == std::io::ErrorKind::NotFound => Ok(()), + Err(err) => { + Err(err).with_context(|| format!("failed to remove {}", partial_root.display())) + } + } + } + + fn ensure_shared_owner_session_from_root( + &self, + root_dir: &std::path::Path, + spec: &WorldSpec, + owner_spec: &SharedWorldOwnerSpec, + ) -> Result { + let _guard = self + .shared_owner_mutex + .lock() + .map_err(|e| anyhow::anyhow!("Failed to acquire shared owner mutex: {}", e))?; + + match &owner_spec.action { + SharedWorldOwnerAction::AttachOrCreate => { + if let Some(handle) = + self.find_shared_owner_session_from_root(root_dir, spec, owner_spec, true)? + { + return Ok(handle); + } + self.create_shared_owner_session_in_root(root_dir, spec, owner_spec, 0, None) + } + SharedWorldOwnerAction::ReplaceExpectedGeneration { + expected_generation, + reason, + } => self.replace_shared_owner_session_from_root( + root_dir, + spec, + owner_spec, + *expected_generation, + reason.clone(), + ), + } + } + /// Ensure the overlay for a world is mounted and return its merged root. pub fn ensure_overlay_root(&self, world: &WorldHandle) -> Result { let mut cache = self @@ -160,31 +496,26 @@ impl WorldBackend for LinuxLocalBackend { fn ensure_session(&self, spec: &WorldSpec) -> Result { self.check_platform()?; - if spec.reuse_session { - if let Some(handle) = self.find_compatible_session_from_root( + match spec.reuse_mode.shared_owner() { + Some(owner_spec) => self.ensure_shared_owner_session_from_root( &SessionWorld::shared_root_dir(), spec, - true, - )? { - return Ok(handle); + owner_spec, + ), + None => { + if spec.reuse_session { + if let Some(handle) = self.find_compatible_session_from_root( + &SessionWorld::shared_root_dir(), + spec, + true, + )? { + return Ok(handle); + } + } + + self.create_generic_session(spec) } } - - // Create new session world - let world = - SessionWorld::ensure_started(spec.clone()).context("Failed to create session world")?; - - let handle = WorldHandle { - id: world.id.clone(), - }; - - let mut cache = self - .session_cache - .write() - .map_err(|e| anyhow::anyhow!("Failed to acquire session cache write lock: {}", e))?; - cache.insert(world.id.clone(), world); - - Ok(handle) } fn exec(&self, world: &WorldHandle, req: ExecRequest) -> Result { @@ -233,9 +564,33 @@ mod tests { use std::collections::HashMap; #[cfg(target_os = "linux")] use std::sync::RwLock; - #[cfg(target_os = "linux")] + use std::sync::{mpsc, Arc}; + use std::time::Duration; use tempfile::tempdir; + fn shared_owner_spec(action: SharedWorldOwnerAction) -> SharedWorldOwnerSpec { + SharedWorldOwnerSpec { + orchestration_session_id: "orch_123".into(), + action, + } + } + + fn shared_world_spec( + project_dir: &std::path::Path, + action: SharedWorldOwnerAction, + ) -> WorldSpec { + WorldSpec { + reuse_session: true, + reuse_mode: world_api::WorldReuseMode::SharedOrchestration(shared_owner_spec(action)), + isolate_network: false, + allowed_domains: vec!["example.com".into()], + project_dir: project_dir.to_path_buf(), + always_isolate: false, + fs_mode: world_api::WorldFsMode::Writable, + ..WorldSpec::default() + } + } + #[test] fn test_backend_creation() { let backend = LinuxLocalBackend::new(); @@ -256,7 +611,6 @@ mod tests { assert!(backend.check_platform().is_ok()); } - #[cfg(target_os = "linux")] #[test] fn cache_miss_with_valid_metadata_repopulates_backend_cache() { let temp = tempdir().unwrap(); @@ -266,8 +620,7 @@ mod tests { std::fs::create_dir_all(&root_dir).unwrap(); std::fs::create_dir_all(&project_dir).unwrap(); std::fs::create_dir_all(&cgroup_path).unwrap(); - let metadata_dir = root_dir.join("wld_recovered"); - std::fs::create_dir_all(&metadata_dir).unwrap(); + std::fs::create_dir_all(root_dir.join("wld_recovered")).unwrap(); let spec = WorldSpec { reuse_session: true, @@ -279,20 +632,17 @@ mod tests { ..WorldSpec::default() }; std::fs::write( - metadata_dir.join("session.json"), - format!( - r#"{{ - "world_id": "wld_recovered", - "project_dir": "{}", - "isolate_network": false, - "always_isolate": false, - "allowed_domains": ["example.com"], - "cgroup_path": "{}", - "started_at_unix_millis": 5000 -}}"#, - project_dir.display(), - cgroup_path.display() - ), + root_dir.join("wld_recovered").join("session.json"), + serde_json::to_vec_pretty(&serde_json::json!({ + "world_id": "wld_recovered", + "project_dir": project_dir, + "isolate_network": false, + "always_isolate": false, + "allowed_domains": ["example.com"], + "cgroup_path": cgroup_path, + "started_at_unix_millis": 5000, + })) + .unwrap(), ) .unwrap(); @@ -309,6 +659,181 @@ mod tests { .contains_key(&handle.id)); } + #[test] + fn replace_success_commits_new_active_and_finalizes_old_world() { + let temp = tempdir().unwrap(); + let root_dir = temp.path().join("world-root"); + let project_dir = temp.path().join("project"); + std::fs::create_dir_all(&root_dir).unwrap(); + std::fs::create_dir_all(&project_dir).unwrap(); + + let owner_spec = shared_owner_spec(SharedWorldOwnerAction::AttachOrCreate); + let spec = shared_world_spec(&project_dir, owner_spec.action.clone()); + let original = SessionWorld::ensure_started_for_shared_owner_at_root_with_world_id( + root_dir.clone(), + spec.clone(), + owner_spec.orchestration_session_id.clone(), + 0, + None, + "wld_original".into(), + ) + .unwrap(); + original.persist_metadata().unwrap(); + + let backend = LinuxLocalBackend::new(); + let handle = backend + .replace_shared_owner_session_from_root( + &root_dir, + &spec, + &owner_spec, + 0, + "restart".into(), + ) + .unwrap(); + + let binding = handle.shared_binding.expect("replacement shared binding"); + assert_eq!( + binding.binding_state, + world_api::SharedWorldBindingState::Active + ); + assert_eq!(binding.world_generation, 1); + assert_ne!(binding.world_id, "wld_original"); + + let recovered = + SessionWorld::recover_shared_active_from_root(&root_dir, &spec, &owner_spec) + .unwrap() + .expect("active world should recover"); + assert_eq!( + recovered.shared_binding().unwrap().world_generation, + 1, + "recovery should prefer the committed replacement" + ); + + let previous_metadata = serde_json::from_slice::( + &std::fs::read(root_dir.join("wld_original").join("session.json")).unwrap(), + ) + .unwrap(); + assert_eq!(previous_metadata["binding_state"], "replaced"); + assert_eq!(previous_metadata["world_generation"], 0); + } + + #[test] + fn replace_failure_rolls_back_old_world_and_cleans_partial_root() { + let temp = tempdir().unwrap(); + let root_dir = temp.path().join("world-root"); + let project_dir = temp.path().join("project"); + std::fs::create_dir_all(&root_dir).unwrap(); + std::fs::create_dir_all(&project_dir).unwrap(); + + let owner_spec = shared_owner_spec(SharedWorldOwnerAction::AttachOrCreate); + let spec = shared_world_spec(&project_dir, owner_spec.action.clone()); + let original = SessionWorld::ensure_started_for_shared_owner_at_root_with_world_id( + root_dir.clone(), + spec.clone(), + owner_spec.orchestration_session_id.clone(), + 0, + None, + "wld_original".into(), + ) + .unwrap(); + original.persist_metadata().unwrap(); + + let backend = LinuxLocalBackend::new(); + let err = backend + .replace_shared_owner_session_from_root_with_creator( + &root_dir, + &spec, + &owner_spec, + 0, + "restart".into(), + |replacement_world_id| { + std::fs::create_dir_all(root_dir.join(&replacement_world_id)).unwrap(); + anyhow::bail!("boom") + }, + ) + .unwrap_err(); + assert!( + err.to_string() + .contains("failed to create replacement world: boom"), + "unexpected error: {err:#}" + ); + + let recovered = + SessionWorld::recover_shared_active_from_root(&root_dir, &spec, &owner_spec) + .unwrap() + .expect("original world should still recover"); + let binding = recovered.shared_binding().unwrap(); + assert_eq!(binding.world_id, "wld_original"); + assert_eq!(binding.world_generation, 0); + assert_eq!( + binding.binding_state, + world_api::SharedWorldBindingState::Active + ); + + let previous_metadata = serde_json::from_slice::( + &std::fs::read(root_dir.join("wld_original").join("session.json")).unwrap(), + ) + .unwrap(); + assert_eq!(previous_metadata["binding_state"], "active"); + assert!(previous_metadata["last_restart_reason"].is_null()); + + let replacement_roots = std::fs::read_dir(&root_dir) + .unwrap() + .filter_map(|entry| entry.ok()) + .map(|entry| entry.file_name().to_string_lossy().into_owned()) + .filter(|name| name != "wld_original") + .collect::>(); + assert!( + replacement_roots.is_empty(), + "partial replacement roots should be cleaned up: {replacement_roots:?}" + ); + } + + #[test] + fn shared_owner_branch_waits_on_backend_mutex() { + let temp = tempdir().unwrap(); + let root_dir = temp.path().join("world-root"); + let project_dir = temp.path().join("project"); + std::fs::create_dir_all(&root_dir).unwrap(); + std::fs::create_dir_all(&project_dir).unwrap(); + + let spec = shared_world_spec(&project_dir, SharedWorldOwnerAction::AttachOrCreate); + let backend = Arc::new(LinuxLocalBackend::new()); + let guard = backend.shared_owner_mutex.lock().unwrap(); + let (tx, rx) = mpsc::channel(); + let owner_spec = spec.reuse_mode.shared_owner().unwrap().clone(); + + let backend_for_thread = Arc::clone(&backend); + let root_for_thread = root_dir.clone(); + let spec_for_thread = spec.clone(); + let owner_spec_for_thread = owner_spec.clone(); + let worker = std::thread::spawn(move || { + let result = backend_for_thread.ensure_shared_owner_session_from_root( + &root_for_thread, + &spec_for_thread, + &owner_spec_for_thread, + ); + tx.send(result).unwrap(); + }); + + assert!( + rx.recv_timeout(Duration::from_millis(100)).is_err(), + "shared-owner request should block while the backend mutex is held" + ); + drop(guard); + + let handle = rx + .recv_timeout(Duration::from_secs(5)) + .expect("worker should complete once the mutex is released") + .unwrap(); + let second = backend + .ensure_shared_owner_session_from_root(&root_dir, &spec, &owner_spec) + .unwrap(); + assert_eq!(handle.id, second.id); + + worker.join().unwrap(); + } + #[cfg(target_os = "linux")] fn poison_cache(cache: &RwLock>) { std::thread::scope(|scope| { @@ -322,13 +847,14 @@ mod tests { }); } - #[cfg(target_os = "linux")] #[test] + #[cfg(target_os = "linux")] fn poisoned_cache_returns_error_in_fs_diff() { let backend = LinuxLocalBackend::new(); poison_cache(&backend.session_cache); let handle = WorldHandle { id: "missing".to_string(), + shared_binding: None, }; let result = std::panic::catch_unwind(|| backend.fs_diff(&handle, "span")); diff --git a/crates/world/src/session.rs b/crates/world/src/session.rs index dcbf6ff4f..5491fc2b3 100644 --- a/crates/world/src/session.rs +++ b/crates/world/src/session.rs @@ -8,10 +8,21 @@ use std::fs; use std::path::{Path, PathBuf}; use std::process::ExitStatus; use std::time::{Duration, SystemTime, UNIX_EPOCH}; -use world_api::{ExecResult, FsDiff, WorldFsMode, WorldSpec}; +use world_api::{ + ExecResult, FsDiff, SharedWorldBindingSnapshot, SharedWorldBindingState, SharedWorldOwnerSpec, + WorldFsMode, WorldSpec, +}; const SESSION_METADATA_FILE_NAME: &str = "session.json"; +#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq, Default)] +#[serde(rename_all = "snake_case")] +enum SessionWorldOwnerMode { + #[default] + Generic, + SharedOrchestration, +} + #[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)] struct SessionWorldMetadata { world_id: String, @@ -21,6 +32,26 @@ struct SessionWorldMetadata { allowed_domains: Vec, cgroup_path: PathBuf, started_at_unix_millis: u64, + #[serde(default)] + owner_mode: SessionWorldOwnerMode, + #[serde(default, skip_serializing_if = "Option::is_none")] + orchestration_session_id: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + world_generation: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + binding_state: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + policy_snapshot_hash: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + world_fs_mode: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + last_restart_reason: Option, +} + +#[derive(Debug, Clone, PartialEq, Eq)] +enum SessionWorldOwnership { + Generic, + Shared(SharedWorldBindingSnapshot), } /// A reusable Linux world with proper isolation. @@ -34,6 +65,9 @@ pub struct SessionWorld { pub started_at: SystemTime, pub network_filter: Option, pub fs_by_span: HashMap, + shared_binding: Option, + policy_snapshot_hash: Option, + last_restart_reason: Option, /// Persistent overlay mount for this session (writable or read-only). overlay: Option, overlay_mode: Option, @@ -56,11 +90,76 @@ impl SessionWorld { /// Ensure a session world is started and return it. pub fn ensure_started(spec: WorldSpec) -> Result { - // Create new session world - let world_id = format!("wld_{}", uuid::Uuid::now_v7()); + Self::ensure_started_in_root(spec, Self::shared_root_dir()) + } + + pub(crate) fn ensure_started_in_root(spec: WorldSpec, root_dir: PathBuf) -> Result { + Self::ensure_started_with_binding_at_root(spec, None, None, root_dir, None) + } + + pub(crate) fn ensure_started_for_shared_owner_at_root( + root_dir: PathBuf, + spec: WorldSpec, + orchestration_session_id: String, + world_generation: u64, + last_restart_reason: Option, + ) -> Result { + let shared_binding = SharedWorldBindingSnapshot { + orchestration_session_id, + world_id: String::new(), + world_generation, + binding_state: SharedWorldBindingState::Active, + }; + Self::ensure_started_with_binding_at_root( + spec, + Some(shared_binding), + last_restart_reason, + root_dir, + None, + ) + } + + pub(crate) fn ensure_started_for_shared_owner_at_root_with_world_id( + root_dir: PathBuf, + spec: WorldSpec, + orchestration_session_id: String, + world_generation: u64, + last_restart_reason: Option, + world_id: String, + ) -> Result { + let shared_binding = SharedWorldBindingSnapshot { + orchestration_session_id, + world_id: String::new(), + world_generation, + binding_state: SharedWorldBindingState::Active, + }; + Self::ensure_started_with_binding_at_root( + spec, + Some(shared_binding), + last_restart_reason, + root_dir, + Some(world_id), + ) + } + + fn ensure_started_with_binding_at_root( + spec: WorldSpec, + world_binding: Option, + last_restart_reason: Option, + root_dir: PathBuf, + world_id: Option, + ) -> Result { + let world_id = world_id.unwrap_or_else(|| format!("wld_{}", uuid::Uuid::now_v7())); + let shared_binding = match world_binding { + Some(mut binding) => { + binding.world_id = world_id.clone(); + Some(binding) + } + None => None, + }; let mut world = Self { id: world_id.clone(), - root_dir: Self::shared_root_dir(), + root_dir, project_dir: spec.project_dir.clone(), cgroup_path: PathBuf::from("/sys/fs/cgroup/substrate").join(&world_id), net_namespace: None, @@ -68,12 +167,15 @@ impl SessionWorld { started_at: SystemTime::now(), network_filter: None, fs_by_span: HashMap::new(), + shared_binding, + policy_snapshot_hash: None, + last_restart_reason, overlay: None, overlay_mode: None, }; world.setup()?; - if world.spec.reuse_session { + if world.should_persist_metadata() { world.persist_metadata()?; } Ok(world) @@ -87,7 +189,65 @@ impl SessionWorld { && self.spec.allowed_domains == spec.allowed_domains } - pub(crate) fn recover_compatible_from_root( + pub(crate) fn is_generic_reusable_with(&self, spec: &WorldSpec) -> bool { + self.shared_binding.is_none() && self.compatible_with(spec) + } + + pub(crate) fn is_shared_owner_reusable_with( + &self, + spec: &WorldSpec, + owner_spec: &SharedWorldOwnerSpec, + ) -> bool { + self.compatible_with(spec) + && self.shared_binding.as_ref().is_some_and(|binding| { + binding.binding_state == SharedWorldBindingState::Active + && binding.orchestration_session_id == owner_spec.orchestration_session_id + }) + } + + pub(crate) fn shared_binding(&self) -> Option { + self.shared_binding.clone() + } + + pub(crate) fn set_shared_binding_state( + &mut self, + binding_state: SharedWorldBindingState, + last_restart_reason: Option, + ) -> Result<()> { + let binding = self + .shared_binding + .as_mut() + .ok_or_else(|| anyhow!("shared binding missing for world {}", self.id))?; + + let current = binding.binding_state.clone(); + let transition_allowed = matches!( + (¤t, &binding_state), + ( + SharedWorldBindingState::Active, + SharedWorldBindingState::Replacing + ) | ( + SharedWorldBindingState::Replacing, + SharedWorldBindingState::Active + ) | ( + SharedWorldBindingState::Replacing, + SharedWorldBindingState::Replaced + ) + ); + if !transition_allowed { + anyhow::bail!( + "invalid shared binding state transition for world {}: {:?} -> {:?}", + self.id, + current, + binding_state + ); + } + + binding.binding_state = binding_state; + self.last_restart_reason = last_restart_reason; + self.persist_metadata() + } + + pub(crate) fn recover_generic_compatible_from_root( root_dir: &Path, spec: &WorldSpec, ) -> Result> { @@ -95,6 +255,124 @@ impl SessionWorld { return Ok(None); } + Self::recover_first_from_root(root_dir, |metadata_path| { + Self::recover_generic_from_metadata_path(metadata_path, root_dir, spec) + }) + } + + #[cfg(test)] + pub(crate) fn recover_compatible_from_root( + root_dir: &Path, + spec: &WorldSpec, + ) -> Result> { + Self::recover_generic_compatible_from_root(root_dir, spec) + } + + pub(crate) fn recover_shared_active_from_root( + root_dir: &Path, + spec: &WorldSpec, + owner_spec: &SharedWorldOwnerSpec, + ) -> Result> { + if !root_dir.is_dir() { + return Ok(None); + } + + let mut active_worlds = Vec::new(); + let mut replacing_worlds = Vec::new(); + for entry in fs::read_dir(root_dir) + .with_context(|| format!("failed to read session root {}", root_dir.display()))? + { + let entry = match entry { + Ok(entry) => entry, + Err(err) => { + tracing::warn!(error = %err, root = %root_dir.display(), "failed to inspect session metadata entry"); + continue; + } + }; + let metadata_path = entry.path().join(SESSION_METADATA_FILE_NAME); + if !metadata_path.is_file() { + continue; + } + + match Self::recover_shared_from_metadata_path( + &metadata_path, + root_dir, + spec, + owner_spec, + ) { + Ok(Some(world)) => match world + .shared_binding + .as_ref() + .map(|binding| binding.binding_state.clone()) + { + Some(SharedWorldBindingState::Active) => active_worlds.push(world), + Some(SharedWorldBindingState::Replacing) => replacing_worlds.push(world), + Some( + SharedWorldBindingState::Replaced | SharedWorldBindingState::Abandoned, + ) => {} + None => tracing::warn!( + metadata = %metadata_path.display(), + "shared recovery candidate missing binding proof after parsing" + ), + }, + Ok(None) => {} + Err(err) => { + tracing::warn!( + error = %err, + metadata = %metadata_path.display(), + "ignoring invalid shared session metadata without deleting it" + ); + } + } + } + + if active_worlds.len() > 1 { + anyhow::bail!( + "multiple active shared worlds found for orchestration session {}", + owner_spec.orchestration_session_id + ); + } + + if let Some(active_world) = active_worlds.pop() { + let active_generation = active_world + .shared_binding + .as_ref() + .map(|binding| binding.world_generation) + .ok_or_else(|| anyhow!("active shared world missing binding proof"))?; + if replacing_worlds.iter().any(|world| { + world + .shared_binding + .as_ref() + .is_some_and(|binding| binding.world_generation >= active_generation) + }) { + anyhow::bail!( + "ambiguous shared world recovery state for orchestration session {}", + owner_spec.orchestration_session_id + ); + } + return Ok(Some(active_world)); + } + + match replacing_worlds.len() { + 0 => Ok(None), + 1 => { + let mut world = replacing_worlds + .pop() + .ok_or_else(|| anyhow!("replacing shared world missing from recovery set"))?; + world.set_shared_binding_state(SharedWorldBindingState::Active, None)?; + Ok(Some(world)) + } + _ => anyhow::bail!( + "ambiguous replacing shared worlds found for orchestration session {}", + owner_spec.orchestration_session_id + ), + } + } + + fn recover_first_from_root(root_dir: &Path, mut recover: F) -> Result> + where + F: FnMut(&Path) -> Result>, + { for entry in fs::read_dir(root_dir) .with_context(|| format!("failed to read session root {}", root_dir.display()))? { @@ -110,7 +388,7 @@ impl SessionWorld { continue; } - match Self::recover_from_metadata_path(&metadata_path, root_dir, spec) { + match recover(&metadata_path) { Ok(Some(world)) => return Ok(Some(world)), Ok(None) => {} Err(err) => { @@ -127,7 +405,7 @@ impl SessionWorld { Ok(None) } - fn recover_from_metadata_path( + fn recover_generic_from_metadata_path( metadata_path: &Path, root_dir: &Path, spec: &WorldSpec, @@ -139,7 +417,44 @@ impl SessionWorld { if !Self::metadata_is_usable(&metadata, metadata_path) { return Ok(None); } - Ok(Some(Self::from_metadata(root_dir, spec, metadata))) + match Self::ownership_from_metadata(&metadata)? { + SessionWorldOwnership::Generic => { + let world = Self::from_metadata(root_dir, spec, metadata, None); + world.ensure_cgroup_attach_target().with_context(|| { + format!("failed to prepare cgroup attach target for {}", world.id) + })?; + Ok(Some(world)) + } + SessionWorldOwnership::Shared(_) => Ok(None), + } + } + + fn recover_shared_from_metadata_path( + metadata_path: &Path, + root_dir: &Path, + spec: &WorldSpec, + owner_spec: &SharedWorldOwnerSpec, + ) -> Result> { + let metadata = Self::read_metadata(metadata_path)?; + if !Self::metadata_matches_spec(&metadata, spec) { + return Ok(None); + } + if !Self::metadata_is_usable(&metadata, metadata_path) { + return Ok(None); + } + match Self::ownership_from_metadata(&metadata)? { + SessionWorldOwnership::Generic => Ok(None), + SessionWorldOwnership::Shared(binding) + if binding.orchestration_session_id == owner_spec.orchestration_session_id => + { + let world = Self::from_metadata(root_dir, spec, metadata, Some(binding)); + world.ensure_cgroup_attach_target().with_context(|| { + format!("failed to prepare cgroup attach target for {}", world.id) + })?; + Ok(Some(world)) + } + SessionWorldOwnership::Shared(_) => Ok(None), + } } fn metadata_matches_spec(metadata: &SessionWorldMetadata, spec: &WorldSpec) -> bool { @@ -155,7 +470,60 @@ impl SessionWorld { && metadata.cgroup_path.is_dir() } - fn from_metadata(root_dir: &Path, spec: &WorldSpec, metadata: SessionWorldMetadata) -> Self { + fn ownership_from_metadata(metadata: &SessionWorldMetadata) -> Result { + let owner_fields_present = metadata.orchestration_session_id.is_some() + || metadata.world_generation.is_some() + || metadata.binding_state.is_some(); + match metadata.owner_mode { + SessionWorldOwnerMode::Generic => { + if owner_fields_present { + Err(anyhow!( + "generic session metadata {} includes shared owner fields", + metadata.world_id + )) + } else { + Ok(SessionWorldOwnership::Generic) + } + } + SessionWorldOwnerMode::SharedOrchestration => { + let orchestration_session_id = metadata + .orchestration_session_id + .clone() + .filter(|id| !id.is_empty()) + .ok_or_else(|| { + anyhow!( + "shared session metadata {} missing orchestration_session_id", + metadata.world_id + ) + })?; + let world_generation = metadata.world_generation.ok_or_else(|| { + anyhow!( + "shared session metadata {} missing world_generation", + metadata.world_id + ) + })?; + let binding_state = metadata.binding_state.clone().ok_or_else(|| { + anyhow!( + "shared session metadata {} missing binding_state", + metadata.world_id + ) + })?; + Ok(SessionWorldOwnership::Shared(SharedWorldBindingSnapshot { + orchestration_session_id, + world_id: metadata.world_id.clone(), + world_generation, + binding_state, + })) + } + } + } + + fn from_metadata( + root_dir: &Path, + spec: &WorldSpec, + metadata: SessionWorldMetadata, + shared_binding: Option, + ) -> Self { Self { id: metadata.world_id, root_dir: root_dir.to_path_buf(), @@ -166,6 +534,9 @@ impl SessionWorld { started_at: UNIX_EPOCH + Duration::from_millis(metadata.started_at_unix_millis), network_filter: None, fs_by_span: HashMap::new(), + shared_binding, + policy_snapshot_hash: metadata.policy_snapshot_hash, + last_restart_reason: metadata.last_restart_reason, overlay: None, overlay_mode: None, } @@ -187,6 +558,16 @@ impl SessionWorld { .as_millis() .try_into() .context("session start time exceeds u64 millis")?; + let (owner_mode, orchestration_session_id, world_generation, binding_state) = + match self.shared_binding.as_ref() { + Some(binding) => ( + SessionWorldOwnerMode::SharedOrchestration, + Some(binding.orchestration_session_id.clone()), + Some(binding.world_generation), + Some(binding.binding_state.clone()), + ), + None => (SessionWorldOwnerMode::Generic, None, None, None), + }; Ok(SessionWorldMetadata { world_id: self.id.clone(), project_dir: self.project_dir.clone(), @@ -195,6 +576,13 @@ impl SessionWorld { allowed_domains: self.spec.allowed_domains.clone(), cgroup_path: self.cgroup_path.clone(), started_at_unix_millis, + owner_mode, + orchestration_session_id, + world_generation, + binding_state, + policy_snapshot_hash: self.policy_snapshot_hash.clone(), + world_fs_mode: Some(self.spec.fs_mode), + last_restart_reason: self.last_restart_reason.clone(), }) } @@ -202,11 +590,64 @@ impl SessionWorld { let metadata_dir = Self::metadata_dir(&self.root_dir, &self.id); fs::create_dir_all(&metadata_dir) .with_context(|| format!("failed to create {}", metadata_dir.display()))?; - let encoded = - serde_json::to_vec_pretty(&self.to_metadata()?).context("encode session metadata")?; let metadata_path = Self::metadata_path(&self.root_dir, &self.id); - fs::write(&metadata_path, encoded) - .with_context(|| format!("failed to write {}", metadata_path.display())) + let temp_path = metadata_dir.join(format!( + ".{}.{}.tmp", + SESSION_METADATA_FILE_NAME, + uuid::Uuid::now_v7() + )); + let metadata = self.to_metadata()?; + let mut temp_file = match fs::File::options() + .create_new(true) + .write(true) + .open(&temp_path) + { + Ok(file) => file, + Err(err) => { + return Err(err) + .with_context(|| format!("failed to create {}", temp_path.display())); + } + }; + + if let Err(err) = serde_json::to_writer_pretty(&mut temp_file, &metadata) + .with_context(|| format!("failed to serialize {}", metadata_path.display())) + { + let _ = fs::remove_file(&temp_path); + return Err(err); + } + + if let Err(err) = temp_file + .sync_all() + .with_context(|| format!("failed to flush {}", temp_path.display())) + { + let _ = fs::remove_file(&temp_path); + return Err(err); + } + + if let Err(err) = fs::rename(&temp_path, &metadata_path) + .with_context(|| format!("failed to persist {}", metadata_path.display())) + { + let _ = fs::remove_file(&temp_path); + return Err(err); + } + + #[cfg(unix)] + if let Some(parent) = metadata_path.parent() { + match fs::File::open(parent).and_then(|dir| dir.sync_all()) { + Ok(()) => {} + Err(err) => tracing::warn!( + error = %err, + path = %parent.display(), + "failed to sync metadata directory after atomic persist" + ), + } + } + + Ok(()) + } + + fn should_persist_metadata(&self) -> bool { + self.spec.reuse_session || self.shared_binding.is_some() } fn read_metadata(path: &Path) -> Result { @@ -318,6 +759,7 @@ impl SessionWorld { std::fs::create_dir_all(&fallback) .context("Failed to create fallback cgroup directory")?; self.cgroup_path = fallback; + self.ensure_cgroup_attach_target()?; return Ok(()); } tracing::error!( @@ -327,6 +769,28 @@ impl SessionWorld { ); return Err(e).context("Failed to create cgroup directory"); } + self.ensure_cgroup_attach_target()?; + Ok(()) + } + + fn ensure_cgroup_attach_target(&self) -> Result<()> { + if self.cgroup_path.starts_with(Path::new("/sys/fs/cgroup")) { + return Ok(()); + } + + let cgroup_procs = self.cgroup_path.join("cgroup.procs"); + if cgroup_procs.exists() { + return Ok(()); + } + + // Unprivileged fallback directories are ordinary host paths rather than kernel cgroups, + // so materialize a writable surrogate that the launchers can use for placement proofing. + fs::File::options() + .create(true) + .truncate(false) + .write(true) + .open(&cgroup_procs) + .with_context(|| format!("failed to create {}", cgroup_procs.display()))?; Ok(()) } @@ -747,11 +1211,14 @@ impl Drop for SessionWorld { #[cfg(test)] mod tests { use super::*; + #[cfg(unix)] + use std::os::unix::fs::PermissionsExt; #[cfg(target_os = "linux")] use std::sync::Mutex; use tempfile::tempdir; #[cfg(target_os = "linux")] use tempfile::TempDir; + use world_api::SharedWorldOwnerAction; #[cfg(target_os = "linux")] static ENV_LOCK: Mutex<()> = Mutex::new(()); @@ -809,11 +1276,82 @@ mod tests { started_at: std::time::SystemTime::UNIX_EPOCH, network_filter: None, fs_by_span: HashMap::new(), + shared_binding: None, + policy_snapshot_hash: None, + last_restart_reason: None, overlay: None, overlay_mode: None, } } + #[test] + #[cfg(target_os = "linux")] + fn create_directories_materializes_surrogate_cgroup_procs_for_fallback_paths() { + let temp = tempdir().unwrap(); + let fallback_cgroup_path = temp.path().join("fallback-cgroup").join("wld_test"); + let mut world = test_world(&temp, false); + world.cgroup_path = fallback_cgroup_path.clone(); + + world.create_directories().unwrap(); + + assert!(fallback_cgroup_path.is_dir()); + assert!( + fallback_cgroup_path.join("cgroup.procs").is_file(), + "expected fallback cgroup paths to materialize a writable cgroup.procs surrogate" + ); + } + + #[test] + #[cfg(target_os = "linux")] + fn recovered_fallback_world_materializes_surrogate_cgroup_procs() { + let temp = tempdir().unwrap(); + let root_dir = temp.path().join("world-root"); + let project_dir = temp.path().join("project"); + let cgroup_path = temp.path().join("fallback-cgroup").join("wld_recovered"); + std::fs::create_dir_all(&root_dir).unwrap(); + std::fs::create_dir_all(&project_dir).unwrap(); + std::fs::create_dir_all(&cgroup_path).unwrap(); + + let spec = WorldSpec { + reuse_session: true, + isolate_network: false, + project_dir: project_dir.clone(), + fs_mode: WorldFsMode::Writable, + ..WorldSpec::default() + }; + let world = SessionWorld { + id: "wld_recovered".into(), + root_dir: root_dir.clone(), + project_dir: project_dir.clone(), + cgroup_path: cgroup_path.clone(), + net_namespace: None, + spec: spec.clone(), + started_at: UNIX_EPOCH + Duration::from_millis(1_234), + network_filter: None, + fs_by_span: HashMap::new(), + shared_binding: None, + policy_snapshot_hash: None, + last_restart_reason: None, + overlay: None, + overlay_mode: None, + }; + + world.persist_metadata().unwrap(); + assert!( + !cgroup_path.join("cgroup.procs").exists(), + "test precondition should reflect pre-patch recovered fallback worlds" + ); + + let recovered = SessionWorld::recover_compatible_from_root(&root_dir, &spec) + .unwrap() + .expect("metadata should recover"); + assert_eq!(recovered.cgroup_path, cgroup_path); + assert!( + recovered.cgroup_path.join("cgroup.procs").is_file(), + "expected recovered fallback worlds to self-heal their cgroup attach target" + ); + } + #[test] fn test_session_world_creation() { let spec = WorldSpec::default(); @@ -836,6 +1374,7 @@ mod tests { fn session_compatibility_respects_core_spec_fields() { let base_spec = WorldSpec { reuse_session: true, + reuse_mode: world_api::WorldReuseMode::GenericCompatible, isolate_network: true, limits: world_api::ResourceLimits::default(), enable_preload: false, @@ -854,6 +1393,9 @@ mod tests { started_at: std::time::SystemTime::UNIX_EPOCH, network_filter: None, fs_by_span: HashMap::new(), + shared_binding: None, + policy_snapshot_hash: None, + last_restart_reason: None, overlay: None, overlay_mode: None, }; @@ -913,6 +1455,9 @@ mod tests { started_at: UNIX_EPOCH + Duration::from_millis(1_234), network_filter: None, fs_by_span: HashMap::new(), + shared_binding: None, + policy_snapshot_hash: None, + last_restart_reason: None, overlay: None, overlay_mode: None, }; @@ -927,6 +1472,299 @@ mod tests { assert_eq!(recovered.cgroup_path, world.cgroup_path); assert_eq!(recovered.started_at, world.started_at); assert!(recovered.compatible_with(&spec)); + assert_eq!(recovered.shared_binding(), None); + } + + #[test] + fn shared_metadata_round_trips_for_recovery() { + let temp = tempdir().unwrap(); + let root_dir = temp.path().join("world-root"); + let project_dir = temp.path().join("project"); + let cgroup_path = temp.path().join("cgroup").join("wld_shared"); + std::fs::create_dir_all(&root_dir).unwrap(); + std::fs::create_dir_all(&project_dir).unwrap(); + std::fs::create_dir_all(&cgroup_path).unwrap(); + + let owner_spec = SharedWorldOwnerSpec { + orchestration_session_id: "orch_123".into(), + action: SharedWorldOwnerAction::AttachOrCreate, + }; + let spec = WorldSpec { + reuse_session: true, + reuse_mode: world_api::WorldReuseMode::SharedOrchestration(owner_spec.clone()), + isolate_network: false, + allowed_domains: vec!["example.com".into()], + project_dir: project_dir.clone(), + always_isolate: false, + fs_mode: WorldFsMode::Writable, + ..WorldSpec::default() + }; + let world = SessionWorld { + id: "wld_shared".into(), + root_dir: root_dir.clone(), + project_dir: project_dir.clone(), + cgroup_path: cgroup_path.clone(), + net_namespace: None, + spec: spec.clone(), + started_at: UNIX_EPOCH + Duration::from_millis(1_234), + network_filter: None, + fs_by_span: HashMap::new(), + shared_binding: Some(SharedWorldBindingSnapshot { + orchestration_session_id: owner_spec.orchestration_session_id.clone(), + world_id: "wld_shared".into(), + world_generation: 0, + binding_state: SharedWorldBindingState::Active, + }), + policy_snapshot_hash: None, + last_restart_reason: None, + overlay: None, + overlay_mode: None, + }; + + world.persist_metadata().unwrap(); + + let recovered = + SessionWorld::recover_shared_active_from_root(&root_dir, &spec, &owner_spec) + .unwrap() + .expect("shared metadata should recover"); + assert_eq!( + recovered.shared_binding(), + Some(SharedWorldBindingSnapshot { + orchestration_session_id: "orch_123".into(), + world_id: "wld_shared".into(), + world_generation: 0, + binding_state: SharedWorldBindingState::Active, + }) + ); + assert_eq!(recovered.last_restart_reason, None); + } + + #[test] + fn shared_binding_state_transitions_persist_and_reject_invalid_edges() { + let temp = tempdir().unwrap(); + let root_dir = temp.path().join("world-root"); + let project_dir = temp.path().join("project"); + let cgroup_path = temp.path().join("cgroup").join("wld_shared"); + std::fs::create_dir_all(&root_dir).unwrap(); + std::fs::create_dir_all(&project_dir).unwrap(); + std::fs::create_dir_all(&cgroup_path).unwrap(); + + let owner_spec = SharedWorldOwnerSpec { + orchestration_session_id: "orch_123".into(), + action: SharedWorldOwnerAction::AttachOrCreate, + }; + let spec = WorldSpec { + reuse_session: true, + reuse_mode: world_api::WorldReuseMode::SharedOrchestration(owner_spec.clone()), + isolate_network: false, + allowed_domains: vec!["example.com".into()], + project_dir: project_dir.clone(), + always_isolate: false, + fs_mode: WorldFsMode::Writable, + ..WorldSpec::default() + }; + let mut world = SessionWorld { + id: "wld_shared".into(), + root_dir: root_dir.clone(), + project_dir, + cgroup_path, + net_namespace: None, + spec, + started_at: UNIX_EPOCH + Duration::from_millis(1_234), + network_filter: None, + fs_by_span: HashMap::new(), + shared_binding: Some(SharedWorldBindingSnapshot { + orchestration_session_id: owner_spec.orchestration_session_id, + world_id: "wld_shared".into(), + world_generation: 7, + binding_state: SharedWorldBindingState::Active, + }), + policy_snapshot_hash: None, + last_restart_reason: None, + overlay: None, + overlay_mode: None, + }; + + world.persist_metadata().unwrap(); + world + .set_shared_binding_state(SharedWorldBindingState::Replacing, Some("restart".into())) + .unwrap(); + assert_eq!( + world.shared_binding().unwrap().binding_state, + SharedWorldBindingState::Replacing + ); + assert_eq!(world.last_restart_reason.as_deref(), Some("restart")); + + world + .set_shared_binding_state(SharedWorldBindingState::Active, None) + .unwrap(); + assert_eq!( + world.shared_binding().unwrap().binding_state, + SharedWorldBindingState::Active + ); + assert_eq!(world.last_restart_reason, None); + + world + .set_shared_binding_state(SharedWorldBindingState::Replacing, Some("restart".into())) + .unwrap(); + world + .set_shared_binding_state(SharedWorldBindingState::Replaced, Some("restart".into())) + .unwrap(); + assert_eq!( + world.shared_binding().unwrap().binding_state, + SharedWorldBindingState::Replaced + ); + assert_eq!(world.last_restart_reason.as_deref(), Some("restart")); + + let err = world + .set_shared_binding_state(SharedWorldBindingState::Active, None) + .unwrap_err(); + assert!( + err.to_string() + .contains("invalid shared binding state transition"), + "unexpected error: {err:#}" + ); + } + + #[test] + fn lone_replacing_world_recovers_back_to_active() { + let temp = tempdir().unwrap(); + let root_dir = temp.path().join("world-root"); + let project_dir = temp.path().join("project"); + let cgroup_path = temp.path().join("cgroup").join("wld_shared"); + std::fs::create_dir_all(&root_dir).unwrap(); + std::fs::create_dir_all(&project_dir).unwrap(); + std::fs::create_dir_all(&cgroup_path).unwrap(); + + let owner_spec = SharedWorldOwnerSpec { + orchestration_session_id: "orch_123".into(), + action: SharedWorldOwnerAction::AttachOrCreate, + }; + let spec = WorldSpec { + reuse_session: true, + reuse_mode: world_api::WorldReuseMode::SharedOrchestration(owner_spec.clone()), + isolate_network: false, + allowed_domains: vec!["example.com".into()], + project_dir: project_dir.clone(), + always_isolate: false, + fs_mode: WorldFsMode::Writable, + ..WorldSpec::default() + }; + let world = SessionWorld { + id: "wld_shared".into(), + root_dir: root_dir.clone(), + project_dir, + cgroup_path, + net_namespace: None, + spec: spec.clone(), + started_at: UNIX_EPOCH + Duration::from_millis(1_234), + network_filter: None, + fs_by_span: HashMap::new(), + shared_binding: Some(SharedWorldBindingSnapshot { + orchestration_session_id: owner_spec.orchestration_session_id.clone(), + world_id: "wld_shared".into(), + world_generation: 3, + binding_state: SharedWorldBindingState::Replacing, + }), + policy_snapshot_hash: None, + last_restart_reason: Some("restart".into()), + overlay: None, + overlay_mode: None, + }; + + world.persist_metadata().unwrap(); + + let recovered = + SessionWorld::recover_shared_active_from_root(&root_dir, &spec, &owner_spec) + .unwrap() + .expect("replacing metadata should recover"); + let binding = recovered.shared_binding().expect("shared binding"); + assert_eq!(binding.binding_state, SharedWorldBindingState::Active); + assert_eq!(binding.world_generation, 3); + assert_eq!(recovered.last_restart_reason, None); + } + + #[test] + fn shared_recovery_prefers_newer_active_over_older_replacing() { + let temp = tempdir().unwrap(); + let root_dir = temp.path().join("world-root"); + let project_dir = temp.path().join("project"); + let old_cgroup_path = temp.path().join("cgroup").join("wld_old"); + let new_cgroup_path = temp.path().join("cgroup").join("wld_new"); + std::fs::create_dir_all(&root_dir).unwrap(); + std::fs::create_dir_all(&project_dir).unwrap(); + std::fs::create_dir_all(&old_cgroup_path).unwrap(); + std::fs::create_dir_all(&new_cgroup_path).unwrap(); + + let owner_spec = SharedWorldOwnerSpec { + orchestration_session_id: "orch_123".into(), + action: SharedWorldOwnerAction::AttachOrCreate, + }; + let spec = WorldSpec { + reuse_session: true, + reuse_mode: world_api::WorldReuseMode::SharedOrchestration(owner_spec.clone()), + isolate_network: false, + allowed_domains: vec!["example.com".into()], + project_dir: project_dir.clone(), + always_isolate: false, + fs_mode: WorldFsMode::Writable, + ..WorldSpec::default() + }; + + let old_world = SessionWorld { + id: "wld_old".into(), + root_dir: root_dir.clone(), + project_dir: project_dir.clone(), + cgroup_path: old_cgroup_path, + net_namespace: None, + spec: spec.clone(), + started_at: UNIX_EPOCH + Duration::from_millis(1_000), + network_filter: None, + fs_by_span: HashMap::new(), + shared_binding: Some(SharedWorldBindingSnapshot { + orchestration_session_id: owner_spec.orchestration_session_id.clone(), + world_id: "wld_old".into(), + world_generation: 4, + binding_state: SharedWorldBindingState::Replacing, + }), + policy_snapshot_hash: None, + last_restart_reason: Some("restart".into()), + overlay: None, + overlay_mode: None, + }; + old_world.persist_metadata().unwrap(); + + let new_world = SessionWorld { + id: "wld_new".into(), + root_dir: root_dir.clone(), + project_dir, + cgroup_path: new_cgroup_path, + net_namespace: None, + spec: spec.clone(), + started_at: UNIX_EPOCH + Duration::from_millis(2_000), + network_filter: None, + fs_by_span: HashMap::new(), + shared_binding: Some(SharedWorldBindingSnapshot { + orchestration_session_id: owner_spec.orchestration_session_id.clone(), + world_id: "wld_new".into(), + world_generation: 5, + binding_state: SharedWorldBindingState::Active, + }), + policy_snapshot_hash: None, + last_restart_reason: Some("restart".into()), + overlay: None, + overlay_mode: None, + }; + new_world.persist_metadata().unwrap(); + + let recovered = + SessionWorld::recover_shared_active_from_root(&root_dir, &spec, &owner_spec) + .unwrap() + .expect("newer active metadata should recover"); + let binding = recovered.shared_binding().expect("shared binding"); + assert_eq!(binding.world_id, "wld_new"); + assert_eq!(binding.world_generation, 5); + assert_eq!(binding.binding_state, SharedWorldBindingState::Active); } #[test] @@ -951,6 +1789,13 @@ mod tests { allowed_domains: vec!["example.com".into()], cgroup_path: temp.path().join("missing-cgroup"), started_at_unix_millis: 42, + owner_mode: SessionWorldOwnerMode::Generic, + orchestration_session_id: None, + world_generation: None, + binding_state: None, + policy_snapshot_hash: None, + world_fs_mode: None, + last_restart_reason: None, }; std::fs::write( stale_dir.join(SESSION_METADATA_FILE_NAME), @@ -972,6 +1817,251 @@ mod tests { assert!(recovered.is_none(), "stale metadata should be ignored"); } + #[test] + fn shared_recovery_rejects_ownerless_legacy_metadata() { + let temp = tempdir().unwrap(); + let root_dir = temp.path().join("world-root"); + let project_dir = temp.path().join("project"); + let cgroup_path = temp.path().join("cgroup").join("wld_legacy"); + std::fs::create_dir_all(&root_dir).unwrap(); + std::fs::create_dir_all(&project_dir).unwrap(); + std::fs::create_dir_all(&cgroup_path).unwrap(); + let metadata_dir = root_dir.join("wld_legacy"); + std::fs::create_dir_all(&metadata_dir).unwrap(); + + std::fs::write( + metadata_dir.join(SESSION_METADATA_FILE_NAME), + format!( + r#"{{ + "world_id": "wld_legacy", + "project_dir": "{}", + "isolate_network": false, + "always_isolate": false, + "allowed_domains": ["example.com"], + "cgroup_path": "{}", + "started_at_unix_millis": 5000 +}}"#, + project_dir.display(), + cgroup_path.display() + ), + ) + .unwrap(); + + let owner_spec = SharedWorldOwnerSpec { + orchestration_session_id: "orch_123".into(), + action: SharedWorldOwnerAction::AttachOrCreate, + }; + let spec = WorldSpec { + reuse_session: true, + reuse_mode: world_api::WorldReuseMode::SharedOrchestration(owner_spec.clone()), + isolate_network: false, + allowed_domains: vec!["example.com".into()], + project_dir, + always_isolate: false, + fs_mode: WorldFsMode::Writable, + ..WorldSpec::default() + }; + + let recovered = + SessionWorld::recover_shared_active_from_root(&root_dir, &spec, &owner_spec).unwrap(); + assert!( + recovered.is_none(), + "legacy ownerless metadata must not be reused for shared-owner mode" + ); + } + + #[test] + fn shared_recovery_rejects_cross_owned_or_inactive_metadata() { + let temp = tempdir().unwrap(); + let root_dir = temp.path().join("world-root"); + let project_dir = temp.path().join("project"); + let cgroup_path = temp.path().join("cgroup").join("wld_shared"); + std::fs::create_dir_all(&root_dir).unwrap(); + std::fs::create_dir_all(&project_dir).unwrap(); + std::fs::create_dir_all(&cgroup_path).unwrap(); + + let metadata_dir = root_dir.join("wld_shared"); + std::fs::create_dir_all(&metadata_dir).unwrap(); + let metadata = SessionWorldMetadata { + world_id: "wld_shared".into(), + project_dir: project_dir.clone(), + isolate_network: false, + always_isolate: false, + allowed_domains: vec!["example.com".into()], + cgroup_path: cgroup_path.clone(), + started_at_unix_millis: 42, + owner_mode: SessionWorldOwnerMode::SharedOrchestration, + orchestration_session_id: Some("orch_other".into()), + world_generation: Some(0), + binding_state: Some(SharedWorldBindingState::Replaced), + policy_snapshot_hash: None, + world_fs_mode: Some(WorldFsMode::Writable), + last_restart_reason: Some("restart".into()), + }; + std::fs::write( + metadata_dir.join(SESSION_METADATA_FILE_NAME), + serde_json::to_vec(&metadata).unwrap(), + ) + .unwrap(); + + let owner_spec = SharedWorldOwnerSpec { + orchestration_session_id: "orch_123".into(), + action: SharedWorldOwnerAction::AttachOrCreate, + }; + let spec = WorldSpec { + reuse_session: true, + reuse_mode: world_api::WorldReuseMode::SharedOrchestration(owner_spec.clone()), + isolate_network: false, + allowed_domains: vec!["example.com".into()], + project_dir, + always_isolate: false, + fs_mode: WorldFsMode::Writable, + ..WorldSpec::default() + }; + + let recovered = + SessionWorld::recover_shared_active_from_root(&root_dir, &spec, &owner_spec).unwrap(); + assert!( + recovered.is_none(), + "cross-owned or inactive metadata must not be reused" + ); + } + + #[test] + fn shared_recovery_ignores_partial_owner_metadata() { + let temp = tempdir().unwrap(); + let root_dir = temp.path().join("world-root"); + let project_dir = temp.path().join("project"); + let cgroup_path = temp.path().join("cgroup").join("wld_partial"); + std::fs::create_dir_all(&root_dir).unwrap(); + std::fs::create_dir_all(&project_dir).unwrap(); + std::fs::create_dir_all(&cgroup_path).unwrap(); + + let metadata_dir = root_dir.join("wld_partial"); + std::fs::create_dir_all(&metadata_dir).unwrap(); + let partial = SessionWorldMetadata { + world_id: "wld_partial".into(), + project_dir: project_dir.clone(), + isolate_network: false, + always_isolate: false, + allowed_domains: vec!["example.com".into()], + cgroup_path, + started_at_unix_millis: 42, + owner_mode: SessionWorldOwnerMode::SharedOrchestration, + orchestration_session_id: Some("orch_123".into()), + world_generation: Some(0), + binding_state: None, + policy_snapshot_hash: None, + world_fs_mode: Some(WorldFsMode::Writable), + last_restart_reason: None, + }; + std::fs::write( + metadata_dir.join(SESSION_METADATA_FILE_NAME), + serde_json::to_vec(&partial).unwrap(), + ) + .unwrap(); + + let owner_spec = SharedWorldOwnerSpec { + orchestration_session_id: "orch_123".into(), + action: SharedWorldOwnerAction::AttachOrCreate, + }; + let spec = WorldSpec { + reuse_session: true, + reuse_mode: world_api::WorldReuseMode::SharedOrchestration(owner_spec.clone()), + isolate_network: false, + allowed_domains: vec!["example.com".into()], + project_dir, + always_isolate: false, + fs_mode: WorldFsMode::Writable, + ..WorldSpec::default() + }; + + let recovered = + SessionWorld::recover_shared_active_from_root(&root_dir, &spec, &owner_spec).unwrap(); + assert!( + recovered.is_none(), + "partial owner metadata must be treated as non-reusable" + ); + assert!( + metadata_dir.join(SESSION_METADATA_FILE_NAME).is_file(), + "shared recovery must retain malformed owner metadata on disk" + ); + } + + #[test] + #[cfg(unix)] + fn atomic_persist_failure_preserves_prior_metadata_bytes() { + if current_uid() == 0 { + return; + } + + let temp = tempdir().unwrap(); + let root_dir = temp.path().join("world-root"); + let project_dir = temp.path().join("project"); + let cgroup_path = temp.path().join("cgroup").join("wld_shared"); + std::fs::create_dir_all(&root_dir).unwrap(); + std::fs::create_dir_all(&project_dir).unwrap(); + std::fs::create_dir_all(&cgroup_path).unwrap(); + + let owner_spec = SharedWorldOwnerSpec { + orchestration_session_id: "orch_123".into(), + action: SharedWorldOwnerAction::AttachOrCreate, + }; + let spec = WorldSpec { + reuse_session: true, + reuse_mode: world_api::WorldReuseMode::SharedOrchestration(owner_spec.clone()), + isolate_network: false, + allowed_domains: vec!["example.com".into()], + project_dir, + always_isolate: false, + fs_mode: WorldFsMode::Writable, + ..WorldSpec::default() + }; + let mut world = SessionWorld { + id: "wld_shared".into(), + root_dir: root_dir.clone(), + project_dir: temp.path().join("project"), + cgroup_path, + net_namespace: None, + spec, + started_at: UNIX_EPOCH + Duration::from_millis(1_234), + network_filter: None, + fs_by_span: HashMap::new(), + shared_binding: Some(SharedWorldBindingSnapshot { + orchestration_session_id: owner_spec.orchestration_session_id, + world_id: "wld_shared".into(), + world_generation: 0, + binding_state: SharedWorldBindingState::Active, + }), + policy_snapshot_hash: None, + last_restart_reason: None, + overlay: None, + overlay_mode: None, + }; + + world.persist_metadata().unwrap(); + let metadata_path = root_dir.join("wld_shared").join(SESSION_METADATA_FILE_NAME); + let original = std::fs::read(&metadata_path).unwrap(); + + let metadata_dir = root_dir.join("wld_shared"); + let original_permissions = std::fs::metadata(&metadata_dir).unwrap().permissions(); + let mut read_only_permissions = original_permissions.clone(); + read_only_permissions.set_mode(0o555); + std::fs::set_permissions(&metadata_dir, read_only_permissions).unwrap(); + + let err = world + .set_shared_binding_state(SharedWorldBindingState::Replacing, Some("restart".into())) + .unwrap_err(); + assert!( + err.to_string().contains("failed to create") + || err.to_string().contains("Permission denied"), + "unexpected error: {err:#}" + ); + + std::fs::set_permissions(&metadata_dir, original_permissions).unwrap(); + assert_eq!(std::fs::read(&metadata_path).unwrap(), original); + } + #[test] #[cfg(target_os = "linux")] fn setup_fails_when_requested_isolation_cannot_install_netfilter() { @@ -995,6 +2085,9 @@ mod tests { started_at: std::time::SystemTime::UNIX_EPOCH, network_filter: None, fs_by_span: HashMap::new(), + shared_binding: None, + policy_snapshot_hash: None, + last_restart_reason: None, overlay: None, overlay_mode: None, }; diff --git a/docs/BACKLOG.md b/docs/BACKLOG.md index 3422470ac..3bf954d37 100644 --- a/docs/BACKLOG.md +++ b/docs/BACKLOG.md @@ -302,6 +302,28 @@ Keep concise, actionable, and security-focused. ## Hardening / Quality +- **P1 – Revisit and likely retire the remaining hidden `runtime_bootstrap_prompt(...)` host-orchestrator bootstrap path** + - Context: public prompt-taking flows no longer use the hidden bootstrap prompt. + - Public `substrate agent start` loads the caller’s `--prompt` into `plan.startup_prompt.prompt_text` in `crates/shell/src/execution/agents_cmd.rs`, then launches the owner-helper with `InitialExecPromptPlan::StartupPrompt { prompt, .. }` in `crates/shell/src/repl/async_repl.rs`. + - Public `substrate agent turn` sends the caller’s follow-up prompt text through the private owner transport via `run_public_prompt_command()` / `request_private_prompt_stream()` / `submit_host_prompt_turn()` in `crates/shell/src/execution/agent_runtime/control.rs`. + - The strongest in-repo proof today is `crates/shell/tests/agent_public_control_surface_v1.rs::public_start_turn_and_stop_emit_streaming_ndjson_and_authoritative_state`, which captures the fake backend stdin and asserts invocation 1 contains `"hello from start"` and invocation 2 contains `"hello from turn"`, while the startup invocation is `exec` rather than `resume`. + - Remaining hidden path: `runtime_bootstrap_prompt(...)` is still reachable when `start_host_orchestrator_runtime_with_prepared_prompt(...)` is called with `initial_prompt == None` in `crates/shell/src/repl/async_repl.rs`. + - This still happens in the normal async REPL host-orchestrator bootstrap path via `start_host_orchestrator_runtime_with_prepared(...)` / `start_host_orchestrator_runtime(...)`. + - It is therefore not dead code and still injects an internal hidden prompt on non-public REPL bootstrap, even though the public agent-control paths have been de-bootstrapified. + - Why revisit soon: + - The codebase now has two materially different startup semantics: + - public agent-control start/turn = operator prompt text is the real prompt, + - legacy/internal REPL bootstrap = hidden bootstrap prompt when no explicit initial prompt is provided. + - This split is easy for humans and future agents to misread, especially when tracing regressions around startup ordering, ownership establishment, or prompt provenance. + - Keeping the hidden path around may be correct for now, but it should be a deliberate contract, not accidental drift. + - Work: + - Confirm whether the REPL still needs `runtime_bootstrap_prompt(...)` at all, or whether it should move to the same explicit-prompt model as public agent control. + - If it must remain, document the contract clearly in `docs/USAGE.md` / `docs/WORLD.md` / agent-control docs: which entrypoints still synthesize a hidden bootstrap prompt, and why. + - If it can be removed, delete the fallback branch in `start_host_orchestrator_runtime_with_prepared_prompt(...)` and update tests to prove there is no hidden prompt injection anywhere on host public-control paths or REPL startup. + - Add or tighten regression coverage around prompt provenance so future changes cannot silently reintroduce bootstrap-composed prompts into public `agent start` / `agent turn`. + - Acceptance: + - Future readers can answer, from docs and tests, whether any shipped entrypoint still uses a hidden bootstrap prompt. + - Either the hidden fallback is removed, or it remains with explicit documentation and tests proving its exact reachability boundary. - Document backend divergence - Clarify why shell uses world-agent and replay uses LinuxLocalBackend; outline future convergence plan. - Redaction hardening for “command body” tracing (preexec + future trace surfaces) diff --git a/docs/CONFIGURATION.md b/docs/CONFIGURATION.md index 1a3cdd3a6..037c91d0b 100644 --- a/docs/CONFIGURATION.md +++ b/docs/CONFIGURATION.md @@ -245,6 +245,10 @@ Unknown keys and extra tables are preserved for future expansion. - `substrate world enable` overwrites `install:` after provisioning succeeds and repairs malformed metadata. - Legacy installs that still have `config.json` are read automatically, but new writes use `config.yaml`. - The generated `~/.substrate/manager_env.sh` exports derived `SUBSTRATE_*` state so shims and subprocesses observe a consistent view of the effective config. +- Supported Linux/macOS provisioning helpers also wire `SUBSTRATE_HOME` into the + `substrate-world-agent` systemd unit and keep that exact path writable via + `ReadWritePaths`. WSL helper scripts are intentionally fail-closed in this slice + and do not claim that placement contract yet. ### World filesystem mode diff --git a/docs/DEVELOPMENT.md b/docs/DEVELOPMENT.md index a06c8ec82..fde290fd4 100644 --- a/docs/DEVELOPMENT.md +++ b/docs/DEVELOPMENT.md @@ -114,13 +114,13 @@ cargo test --workspace -- --nocapture ```bash # Dynamically link to a system Kuzu (fast if installed) -cargo build -p substrate-graph --features kuzu-dylib +cargo build --manifest-path crates/substrate-graph/Cargo.toml --features kuzu-dylib # Statically build Kuzu (slow; requires cmake and a long native build) -cargo build -p substrate-graph --features kuzu-static +cargo build --manifest-path crates/substrate-graph/Cargo.toml --features kuzu-static ``` -Use `cargo tree -p substrate-graph -e features` to inspect which features are active. +Use `cargo tree --manifest-path crates/substrate-graph/Cargo.toml -e features` to inspect which features are active. ### Documentation diff --git a/docs/GRAPH.md b/docs/GRAPH.md index 55f7bd114..76d29d749 100644 --- a/docs/GRAPH.md +++ b/docs/GRAPH.md @@ -92,10 +92,11 @@ CREATE REL TABLE PARENT_OF(FROM Span TO Span); ```bash # Default (mock facade) -cargo build -p substrate-graph +cargo build --manifest-path crates/substrate-graph/Cargo.toml # With Kuzu (planned): requires cmake, slow build -# cargo build -p substrate-graph --features kuzu-static +# cargo build --manifest-path crates/substrate-graph/Cargo.toml --features kuzu-static +``` ## CLI Usage (mock backend) diff --git a/docs/INSTALLATION.md b/docs/INSTALLATION.md index f26652d75..e7e9f1d4c 100644 --- a/docs/INSTALLATION.md +++ b/docs/INSTALLATION.md @@ -1,26 +1,30 @@ # Installation Guide (v0.2.0-beta) -Substrate ships release bundles with a cross-platform installer that deploys the -CLI, shim launcher, and world backend in one step. The same script is used by -our `curl | bash` flow and the offline bundles published via -GitHub Releases (`https://github.com/atomize-hq/substrate/releases`). +Substrate ships release bundles with installer flows for the CLI, shims, and +supported world backends. In this slice, Linux host-native and macOS Lima-backed +provisioning are supported; the WSL helper path is intentionally fail-closed +until it matches the Linux-first placement contract. ## Supported Platforms - **Linux**: systemd-based distributions with `sudo`, `curl`, `tar`, and `jq` - available. The world backend runs via the `substrate-world-agent.service` + + available. When world provisioning is enabled, the installer also provisions + the host `libseccomp` runtime package needed for seccomp-backed hardening. The + world backend runs via the `substrate-world-agent.service` + `substrate-world-agent.socket` units. -- **Windows 11 / 10 (22H2+) with WSL2 + systemd**: install via the bundled PowerShell script (`scripts/windows/install-substrate.ps1`), which provisions the `substrate-wsl` distro after enabling systemd in `/etc/wsl.conf`. The WSL backend is functional but experimental—expect ongoing updates. - **macOS 14+ (arm64)**: requires Apple Virtualization Framework and Lima (the installer verifies both). +- **Windows / WSL**: world provisioning is intentionally fail-closed in this + slice. The owned WSL helper scripts exit explicitly instead of mutating the + guest because their socket/group and `SUBSTRATE_HOME` placement are not yet + aligned with Linux/macOS. -> ℹ️ PowerShell automation is available—use the Windows instructions below to -> install from the host. Manual installation from inside WSL remains an option -> if you prefer the Linux script. +> ℹ️ If you need a CLI-only install inside WSL, use the Linux installer with +> `--no-world`. Do not use the WSL provisioning helpers in this slice. ## Quick Install (Release Bundles) -### Linux / WSL (systemd) +### Linux (systemd host-native) ```bash curl -fsSL https://raw.githubusercontent.com/atomize-hq/substrate/main/scripts/substrate/install.sh | bash @@ -84,7 +88,9 @@ The installer will: `root:substrate` with `0750` permissions, `/run/substrate.sock` is recreated as `root:substrate` with `0660` permissions, and managed gateway runtime artifacts under `/run/substrate/substrate-gateway-runtime/` stay group-readable (`0750` - directories, `0640` files). The installer prints + directories, `0640` files). The service unit also exports `SUBSTRATE_HOME=` + and keeps that path writable in `ReadWritePaths`, alongside `/var/lib/substrate`, + `/run`, `/run/substrate`, `/sys/fs/cgroup`, and `/tmp`. The installer prints `loginctl enable-linger ` guidance so socket activation survives logout/reboots. @@ -94,7 +100,8 @@ rerun the installer (or `scripts/linux/world-provision.sh` / `scripts/mac/lima-w By default the installer adds `/bin` (default: `~/.substrate/bin`) to your shell PATH by appending a small, idempotent snippet to your rc files (bash/zsh/fish). Set `SUBSTRATE_INSTALL_NO_PATH=1` to skip this behavior and -invoke `~/.substrate/bin/substrate` directly instead. Supplying `--no-world` skips step 6, writes +invoke `~/.substrate/bin/substrate` directly instead. Supplying `--no-world` skips world provisioning, +writes `~/.substrate/config.yaml` with `install.world_enabled: false`, and prints the exact `substrate world enable` command to run when you are ready to provision the backend. You can still force a single world-isolated run later @@ -124,8 +131,7 @@ with `substrate --world ...` without changing the stored metadata. ### Prerequisites -- PID 1 must be `systemd` (`ps -p 1 -o comm=`). On WSL, enable systemd by adding - `boot.systemd=true` under `[boot]` in `/etc/wsl.conf`, then `wsl --shutdown`. +- PID 1 must be `systemd` (`ps -p 1 -o comm=`). - `sudo`, `curl`, `tar`, and `jq` must be available on the host. During installation the script: @@ -142,7 +148,9 @@ During installation the script: units so `/run/substrate` is `root:substrate 0750`, `/run/substrate.sock` is owned by `root:substrate` with `0660` permissions, and managed gateway runtime logs/config/manifests under `/run/substrate/substrate-gateway-runtime/` are - group-readable (`0750` directories, `0640` files). The script reports the + group-readable (`0750` directories, `0640` files). The generated service unit + includes `CAP_CHOWN` plus the existing namespace/network capabilities and keeps + `SUBSTRATE_HOME=` writable through `ReadWritePaths`. The script reports the current `loginctl` lingering status and reminds you to run `loginctl enable-linger ` so socket activation stays live after logout or reboot. @@ -175,6 +183,9 @@ The macOS flow mirrors the Linux installer but additionally: - Requires `envsubst` (install via `brew install gettext` to provide it) - Provisions the Lima VM (`scripts/mac/lima-warm.sh`) and copies the Linux `world-agent` and `substrate-gateway` binaries into the guest +- Writes the guest systemd unit with `SUBSTRATE_HOME=/.substrate` + and keeps that path writable in `ReadWritePaths`, matching the Linux placement + contract for runtime state and manager/config files **Manual Lima preparation** is documented in `docs/WORLD.md`. @@ -184,23 +195,15 @@ The macOS flow mirrors the Linux installer but additionally: pwsh -File scripts/windows/install-substrate.ps1 ``` -- Flags mirror the Unix installer: `-Version`, `-Prefix`, `-Archive`, - `-NoWorld`, `-NoShims`, `-DryRun`, `-DistroName`. -- Defaults to `$env:LOCALAPPDATA\Substrate` and provisions the - `substrate-wsl` distro unless `-NoWorld` is supplied. -- Hosted one-liner: - - ```powershell - irm https://raw.githubusercontent.com/atomize-hq/substrate/main/scripts/windows/install-substrate.ps1 | iex - ``` - -- Requires WSL2 (with systemd enabled inside the distro) and PowerShell 7+. +- The owned WSL provisioning helpers are intentionally fail-closed in this slice. +- Expect the WSL warm/provision path to exit explicitly instead of mutating the guest. +- Use Linux or macOS for supported world provisioning, or install CLI-only flows with `--no-world`. ## Post-Install Checks After the script completes: -### macOS / Linux / WSL +### macOS / Linux ```bash substrate --version @@ -227,7 +230,7 @@ substrate.exe world doctor --json | ConvertFrom-Json | Select-Object schema_vers ``` If you installed with `--no-world`, run `substrate world enable` once you are -ready to provision the backend (macOS Lima VM, Linux namespaces, or WSL). If +ready to provision the backend (macOS Lima VM or Linux namespaces). If either doctor surfaces failures, consult `docs/WORLD.md` and the troubleshooting appendices for the relevant platform. @@ -326,7 +329,7 @@ It also reports whether `loginctl enable-linger ` still needs to be run. - **Doctor failures**: capture `substrate shim doctor --json` and `substrate world doctor --json`; attach both to bug reports so we can spot PATH vs kernel/virtualization gaps quickly. -- **World agent inactive (Linux/WSL)**: confirm `systemctl status +- **World agent inactive (Linux)**: confirm `systemctl status substrate-world-agent.socket` reports `listening`, `systemctl status substrate-world-agent.service` reports `active` (or restarts cleanly), and that `/run/substrate.sock` exists as `root substrate 0660` (`sudo ls -l /run/substrate.sock`). If the socket @@ -338,9 +341,9 @@ It also reports whether `loginctl enable-linger ` still needs to be run. `id -nG "$USER"`—or lingering is still disabled (`loginctl enable-linger "$USER"`). `substrate world doctor --json | jq '.host.world_socket'` and `substrate --shim-status` both spell out whether socket activation is healthy. -- **WSL systemd disabled**: edit `/etc/wsl.conf`, set `[boot] -systemd=true`, run - `wsl --shutdown`, and reopen the distribution. +- **WSL provisioning request**: the owned WSL helpers now fail closed on purpose. + Use a CLI-only `--no-world` install inside WSL or move the world backend to a + supported Linux/macOS host until WSL placement alignment lands. - **macOS virtualization disabled**: enable "Virtualization" in System Settings → Privacy & Security, then rerun the installer or `substrate world enable`. diff --git a/docs/TRACE.md b/docs/TRACE.md index 4fd7b9e2a..f35fb56b5 100644 --- a/docs/TRACE.md +++ b/docs/TRACE.md @@ -59,7 +59,13 @@ Phase 8 introduces/locks additional cross-feature correlation fields (e.g., `orc Operator note (non-negotiable): - Do not rely on heuristic joins. Prefer explicit join keys (`session_id`, `orchestration_session_id`, `run_id`, explicit cause refs) as defined in ADR-0028/Phase 8 contracts. - Trace is safe-by-default: do not mirror raw third-party JSONL/NDJSON agent logs into `trace.jsonl` by default. Treat raw wrapper logs and any payloads that may contain secrets as per-session artifacts, and apply redaction/caps rules per ADR-0028 and the Phase 8 secrets rubric. -- Live shell-owned orchestrator session ownership is persisted separately from trace under `~/.substrate/run/agent-hub/handles/*.json`. Those manifests reflect a REPL-owned session only while the shell still retains the attached UAA control boundary (cancel ownership plus active event/completion observation). Trace remains the canonical historical event log; the live manifests only provide cross-invocation session discovery and precedence for current-session status/toolbox surfaces. +- Live shell-owned orchestrator session ownership is persisted separately from trace under `~/.substrate/run/agent-hub/sessions//session.json` plus `~/.substrate/run/agent-hub/sessions//participants/.json`. Those canonical session-root records are the live-state authority boundary for agent-hub operator surfaces. +- `~/.substrate/run/agent-hub/sessions/.json` plus `~/.substrate/run/agent-hub/participants/*.json` remain flat compatibility bridge input/output only during the cutover. They may fill torn-root gaps on read, but they do not outrank canonical session-root records and must not be treated as live-state authority. +- Legacy `~/.substrate/run/agent-hub/handles/*.json` remains legacy compatibility input only and is last-resort compatibility input only. Handle files never authorize live state when canonical or flat compatibility session records exist. +- The child participant record is authoritative-live only while the shell still retains the attached UAA control boundary, and production status/toolbox discovery must resolve through the parent record plus the live runtime participant snapshot instead of treating flat compatibility files or legacy handle files as truth. +- Invalidated participant tombstones in canonical or flat compatibility participant records beat stale trace fallback rows for live-state selection. If no replacement participant has been persisted yet for the same `(orchestration_session_id, agent_id, execution.scope)` tuple, the selected live surface omits that tuple instead of resurfacing stale liveness from trace. +- Runtime participant lineage persists under `resumed_from_participant_id`; any `resumed_from_session_handle_id` input is compatibility-only aliasing on read. +- Trace remains the canonical historical event log and historical fallback for `substrate agent status` gaps only. Trace never authorizes current-session toolbox state or `substrate agent toolbox env`. ### Command Span Schema (`command_start` / `command_complete`) @@ -150,6 +156,12 @@ These are canonical cross-feature correlation identifiers. Details and required/ - `backend_id`: backend identifier in `:` form (e.g., `cli:codex`, `api:openai`) when a specific backend is involved. - `world_id`: world boundary identity; required on in-world telemetry families (e.g., `world_process_*`) and any record that describes an in-world boundary/session. +Emission rule: +- `AgentEvent` keeps backward-compatible additive lineage fields: `participant_id`, `parent_participant_id`, and `resumed_from_participant_id`. +- Runtime-owned producers must emit a real `orchestration_session_id` or suppress the agent-event row entirely; they must not synthesize a process-global fallback id. +- Shell-owned command-completion and stream emitters follow the same real-id-or-suppress rule for orchestration-scoped `agent_event` rows. They may append the row only when live orchestration context already carries the authoritative `orchestration_session_id`; they must not backfill from `session_id`, PID lookup, synthetic run correlation, or any other heuristic. +- Legacy trace rows may omit `participant_id`, `parent_participant_id`, and `resumed_from_participant_id`; consumers that ignore these additive fields continue to work unchanged. + ### Agent Identity-Tuple Fields Agent-hub successor telemetry keeps adapter identity separate from semantic identity: @@ -162,7 +174,12 @@ Agent-hub successor telemetry keeps adapter identity separate from semantic iden - `auth_authority`: nested gateway-backed auth authority only; pure-agent records omit it. - `parent_run_id`: nested gateway-backed trace correlation only; points at the parent pure-agent `run_id`. - `world_id`: world boundary identifier for world-scoped pure-agent records and in-world telemetry families. -- `world_generation`: generation counter for the active world-scoped pure-agent session or world-backed execution. +- `world_generation`: generation counter for the active world-scoped pure-agent session or world-backed execution when an authoritative shared-world binding proof exists. + +Boundary note: +- The trace can carry `orchestration_session_id`, `world_id`, and `world_generation` for explicit shared-owner world executions, but trace remains historical audit output only. +- Projection of the active shared-world binding into shell-owned runtime state remains PLAN-04. +- Replacement/invalidation semantics for prior generations remain PLAN-05; consumers must not infer global invalidation from trace fields alone. `uaa.agent.session` is currently a Substrate-local normalized protocol-family id, not an automatic claim of upstream Unified Agent API wire or API compatibility. In the current repo, pure-agent records are stamped with that label by [agent_events.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/common/src/agent_events.rs:17), and `substrate agent status` / orchestrator-selection logic consumes the same label in [agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:25). @@ -174,6 +191,8 @@ The shell-owned UAA runtime translates external `agent_api` wrapper events into Bootstrap and lifecycle rows for the first host orchestrator caller path are emitted through the same canonical `agent_event` family; raw wrapper output stays outside `trace.jsonl`. +Runtime-owned shell rows follow the same rule. Host stream chunks, shell command-completion events, and world-restart alerts emit orchestration-scoped `agent_event` rows only when a live parent orchestration session exists and supplies the real `orchestration_session_id`; otherwise stdout/stderr, `command_*` trace spans, and operator-facing terminal messaging continue without appending an orchestration-scoped `agent_event` row. Suppression here is additive only: missing orchestration context suppresses the shell-owned `agent_event` row, but it does not authorize heuristic recovery or synthetic correlation. + Operator-facing omission rules: - Pure-agent records keep `client`, `router`, and `protocol`, and omit `provider` plus `auth_authority`. - Nested gateway-backed records may add `provider` and `auth_authority`, but they must omit `world_id` and `world_generation`. @@ -248,6 +267,10 @@ These fields may appear on non-span records even when command spans remain uncha Agent Hub emits structured alert events to make world session reuse and restart behavior operator-verifiable. These alerts are appended to `trace.jsonl` as structured agent events (not command spans). +Current scope boundary: +- These alerts are observational trace records, not the authoritative live-state registry for shared-world bindings. +- A `world_generation` value in trace or on an alert does not by itself define replacement/invalidation semantics for prior-generation participants; that contract remains PLAN-05. + Key machine-detectable alert codes: - `data.code="world_restarted"`: emitted when the hub auto-restarts the world (e.g., due to world-relevant drift). - `data.code="world_restart_required"`: emitted when drift is detected under a fail-closed posture (no implicit restart; operator action required). diff --git a/docs/USAGE.md b/docs/USAGE.md index 78f9adeb1..442616af4 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -81,14 +81,49 @@ Operator-visible identity rules on these surfaces: - `backend_id` always renders as `:`. - Pure-agent list rows omit `provider`, `auth_authority`, `world_id`, and `world_generation`. - Pure-agent status rows omit `provider` and `auth_authority`. +- Pure-agent `source_kind=live_runtime` status rows carry authoritative `posture`, `attached_participant_id`, and `pending_inbox_count` from the session-root parent record. +- Pure-agent `source_kind=trace_fallback` status rows keep `posture`, `attached_participant_id`, and `pending_inbox_count` as explicit `null` in `--json`; human-readable output marks those fields as ``. - `world_id` and `world_generation` render only for world-scoped pure-agent session rows. - Nested gateway-backed status rows stay separate from pure-agent rows and are the only rows that publish `provider` and `auth_authority`. - Nested gateway-backed status rows depend on valid trace-side `parent_run_id` correlation; stale historical nested rows are ignored, and malformed selected-surface rows fail closed. - `substrate agent doctor` now includes a `runtime_realizability` check after `orchestrator_selection`. For the selected orchestrator, it fail-closes on unsupported `config.kind`, unsupported `cli.mode`, unsupported shell-owned backend mapping, or an unresolved `config.cli.binary`. - `substrate agent toolbox status` is a pre-runtime introspection surface: it projects the effective toolbox posture, the selected orchestrator identity, and either the active per-session UDS endpoint or the deterministic endpoint template when no orchestrator session is active yet. -- `substrate agent toolbox env` emits `SUBSTRATE_AGENT_TOOLBOX_ENDPOINT` and `SUBSTRATE_AGENT_TOOLBOX_VERSION` only when a current pure-agent orchestrator session is present; otherwise it fails closed with a specific exit code. +- `substrate agent toolbox env` emits `SUBSTRATE_AGENT_TOOLBOX_ENDPOINT` and `SUBSTRATE_AGENT_TOOLBOX_VERSION` only when a current live host-scoped orchestrator session is present; otherwise it fails closed with exit `3`. -When the async REPL owns a shell-scoped orchestrator session, live session discovery is backed by persisted manifests under `~/.substrate/run/agent-hub/handles/`. `substrate agent status` and `substrate agent toolbox ...` prefer those live manifests before falling back to trace-derived session discovery. +When the async REPL owns a shell-scoped orchestrator session, live session discovery is backed by the store-owned session root: +- `~/.substrate/run/agent-hub/sessions//session.json` +- `~/.substrate/run/agent-hub/sessions//participants/.json` + +`substrate agent status` and `substrate agent toolbox ...` resolve live state from those session records first. Those canonical session-root parent plus participant records are the live-state authority boundary for live-runtime pure-agent rows. + +`~/.substrate/run/agent-hub/sessions/.json` and `~/.substrate/run/agent-hub/participants/*.json` remain flat compatibility bridge input/output only during the cutover. Legacy `~/.substrate/run/agent-hub/handles/*.json` remains compatibility input only and is last-resort compatibility input only. None of those files outrank the canonical session-root records. + +Trace is historical fallback only for `substrate agent status` gaps after live-state filtering. It can still contribute `source_kind=trace_fallback` pure-agent rows and trace-correlated `nested_llm_records`, but it never authorizes live posture truth for the three durable-session fields and never authorizes current-session toolbox state or `substrate agent toolbox env`. + +### Public Session Control + +The public prompt-taking and control surface is intentionally narrow: + +```bash +substrate agent start --backend --prompt "hello" --json +substrate agent turn --session --backend --prompt "next" --json +substrate agent reattach --session --json +substrate agent fork --session --json +substrate agent stop --session --json +``` + +- `substrate agent start` is the canonical public root prompt-taking surface and remains host-only in v1. +- `substrate agent turn` is the canonical public follow-up surface and requires the exact pair `(--session , --backend )`. +- `substrate agent reattach` is attached-owner recovery only for the same durable session; it does not submit a prompt. +- `substrate agent stop` is the canonical closeout path for attached and parked durable host sessions. +- `substrate agent status --json` is the authoritative parked-session read surface for live-runtime `posture`, `attached_participant_id`, and `pending_inbox_count`. +- Public follow-up never falls back to `participant_id`, legacy `session_handle_id`, `active_session_handle_id`, or `internal.uaa_session_id`; those selector shapes fail closed. +- There is still no default-agent routing and there is still no public world-root `start`. +- On Linux, exact world-member follow-up reuses the retained member slot and submits through the typed `/v1/member_turn/stream` path. +- Detached host recovery stays on `substrate agent reattach --session `. +- Detached world follow-up fails closed until `reattach` restores an active host owner. +- Durable inbox behavior is intentionally narrow: persistence exists, pending work can normalize posture into `awaiting_attention`, internal ack/dismiss plus dev-support/test ingress exist, and no public inbox command surface or automatic resume-from-inbox workflow is shipped. +- `substrate -c`, `--command`, and piped stdin remain shell execution surfaces rather than agent-prompt aliases. ## PTY Support diff --git a/docs/WORLD.md b/docs/WORLD.md index bf6543c85..9495bf9de 100644 --- a/docs/WORLD.md +++ b/docs/WORLD.md @@ -2,7 +2,7 @@ This document describes the world execution model, transport topology, and validation evidence used by Substrate. It is descriptive context for `docs/contracts/substrate-gateway-runtime-parity.md`, `docs/contracts/substrate-gateway-operator-contract.md`, and `docs/contracts/substrate-gateway-status-schema.md`; it does not redefine those operator contracts. -Status: Linux and macOS default to "always-in-world" execution. Windows uses the same gateway semantics through WSL-backed transport, with permitted backend divergence in the hidden transport layer. +Status: Linux host-native and macOS Lima-backed worlds are the supported provisioning paths in this slice. Windows/WSL helper scripts are intentionally fail-closed until their placement contract matches the Linux-first runtime contract. --- @@ -22,7 +22,7 @@ On Linux the agent runs directly on the host. On macOS the agent runs inside a L Helper scripts (`scripts/mac/lima-*.sh`, `scripts/mac/smoke.sh`) keep the Lima environment reproducible. -`/tmp` is included in the guest unit’s `ReadWritePaths` list so replay and shim flows can surface temp‑file diffs on both platforms. The provisioning script embeds this setting in the unit automatically—no manual tuning needed. +`/tmp` is included in the guest unit’s `ReadWritePaths` list so replay and shim flows can surface temp‑file diffs on both platforms. The provisioning scripts also wire `SUBSTRATE_HOME` into the service unit and keep that path writable under `ReadWritePaths`, so manager/config/runtime state lands in the same canonical home on Linux host-native and macOS Lima guest paths. ### Transport boundary and multi-user posture @@ -51,12 +51,62 @@ Always-on by default (unless disabled via `SUBSTRATE_WORLD=disabled`): - REPL integration - The Substrate REPL can force PTY per‑line using the `:pty ` prefix. The shell strips this prefix before sending the command to the agent. + - Persistent-session startup keeps a caller-shape split: sync shell/bootstrap callers continue to + use sync `PlatformWorldContext.ensure_ready()`, while the macOS async REPL startup path now + uses a dedicated async readiness seam that delegates back into backend-owned Lima readiness. + - `SUBSTRATE_WORLD_SOCKET` keeps exact bypass semantics for persistent-session startup. When that + override is set, the shell connects directly to the requested socket instead of invoking the + platform readiness seam first. - Fallback - If `world_fs.require_world=false` and the agent/socket is unavailable (or a transport handshake fails), the shell prints exactly one warning and runs on the host path for that command. Subsequent commands continue to attempt world routing. - If `world_fs.require_world=true`, Substrate fails closed when the world backend is unavailable (or disabled via `--no-world`/`SUBSTRATE_WORLD=disabled`) instead of falling back to host execution. -Windows (WSL backend) follows the same operator-facing lifecycle/status meaning; only the transport/bootstrap implementation differs. When the world backend is unavailable, the same `world_fs.require_world` rules apply (fallback only when `false`). +Windows/WSL helper support is intentionally fail-closed in this slice. The older WSL bootstrap path is not documented here as a supported operator flow because its socket/group and `SUBSTRATE_HOME` placement do not yet match the Linux/macOS contract. + +### Shared-owner world reuse (Linux-source contract, macOS/Lima parity) + +Substrate now has two reuse modes in the shared `world-api` contract: + +- `reuse_mode=generic_compatible` is the legacy/default reusable-world lookup. +- `reuse_mode=shared_orchestration` activates explicit shared-owner reuse with: + - `orchestration_session_id` + - `action=attach_or_create` + - `action=replace_expected_generation { expected_generation, reason }` + +When explicit shared-owner reuse is active, the authoritative proof surface is `WorldHandle.shared_binding`: + +- `orchestration_session_id` +- `world_id` +- `world_generation` +- `binding_state` + +Current enforcement boundary: + +- Linux remains the source-of-truth implementation for owner-bound shared-world reuse semantics, and those semantics live in `crates/world`. +- On Linux, the persisted authority is `SessionWorldMetadata` in `crates/world/src/session.rs`; there is no shell-owned co-authoritative binding store in this slice. +- `set_shared_binding_state()` is the only shared-binding mutation path, and `persist_metadata()` writes `session.json` atomically. +- Reuse requires both exact owner match and compatible world inputs; generic compatibility alone is not enough. +- Same-owner Linux shared-world ensure/replace paths are serialized by a backend-local mutex in `crates/world` so attach/create and replace do not race each other inside one backend instance. +- When a caller explicitly requests shared-owner reuse, the proof echoed back to request/PTY callers must describe the same `orchestration_session_id` and `world_id`, and the proof must be in `binding_state=active`. `active` remains the only binding proof downstream surfaces may expose or accept. +- On macOS, the supported Lima-backed transport path now preserves the same explicit shared-owner proof contract: the shell forwards `SharedWorldOwnerSpec`, requires `ready.shared_world` / `WorldHandle.shared_binding` proof, and keeps replacement fail-closed on the forwarded guest path. +- On macOS, explicit shared-owner requests still reject when callers bypass the Lima-backed path with `SUBSTRATE_WORLD_SOCKET`, because that override skips the authoritative forwarded transport. +- Windows still rejects explicit shared-owner requests before bootstrap/fallback logic runs in this slice. +- Windows/WSL readiness split work in this slice is internal-only parity hardening. It does not + create a supported Windows persistent-session shell caller. + +Replacement and recovery guarantees on Linux: + +- `replace_expected_generation` runs in four stages inside `crates/world`: pre-commit the current world to `binding_state=replacing`, commit the replacement world as `binding_state=active` with `world_generation + 1`, roll the old world back to `active` if replacement creation fails, then finalize the old world to `binding_state=replaced`. +- The commit window is intentionally two-phase: recovery may observe either one `active` world or one lone `replacing` world for an owner, but never treat `replacing` as a downstream proof state. +- Recovery reconciles a lone `replacing` world back to `active`, prefers a newer `active` world over an older `replacing` world, and fails closed on ambiguous same-owner states or malformed owner metadata. +- Ownerless legacy metadata, partial owner metadata, and `replaced`/`abandoned` bindings are never reusable for shared-owner flows. + +Deliberate boundary for later lanes: + +- Runtime-state projection of the active binding into shell-owned live state remains PLAN-04. +- Invalidation/replacement registry semantics for generation changes and non-`active` binding states remain PLAN-05. +- This document does not claim restart persistence or cross-generation invalidation behavior beyond the current Linux backend reuse proof. --- @@ -66,6 +116,10 @@ Windows (WSL backend) follows the same operator-facing lifecycle/status meaning; world-agent under `/usr/local/bin`, write the `.service` **and** `.socket` units, and enable socket activation. The script uses `sudo` for filesystem and systemd operations and will prompt if elevated credentials are required. +- The generated service unit exports `SUBSTRATE_HOME` (default: `/.substrate`, + or the explicit `SUBSTRATE_HOME` you pass to the helper) and keeps that path in + `ReadWritePaths` alongside `/var/lib/substrate`, `/run`, `/run/substrate`, `/sys/fs/cgroup`, + and `/tmp`. - The helper ensures the Linux `substrate` group exists, adds the invoking user when possible, and rewrites the socket/service units so `/run/substrate` is recreated as `root:substrate 0750`, `/run/substrate.sock` is created as `root:substrate 0660`, @@ -132,7 +186,7 @@ Substrate on macOS uses a Lima VM (“substrate”) to host the world-agent. The Hosted installer behavior coverage on macOS flows through this Lima-backed Linux guest/world-agent path; package-manager selection itself remains Linux-only and does not define native macOS package-manager selection. - Provisioning & lifecycle -- `scripts/mac/lima-warm.sh` starts or creates the VM from `scripts/mac/lima/substrate.yaml`, installs required packages, and ensures the systemd unit writes to `/run/substrate.sock` and managed gateway runtime artifacts under `/run/substrate/substrate-gateway-runtime/` with the same `substrate`-group boundary inside the guest, with `/tmp` included in `ReadWritePaths`. +- `scripts/mac/lima-warm.sh` starts or creates the VM from `scripts/mac/lima/substrate.yaml`, installs required packages, and ensures the systemd unit writes to `/run/substrate.sock` and managed gateway runtime artifacts under `/run/substrate/substrate-gateway-runtime/` with the same `substrate`-group boundary inside the guest, exports `SUBSTRATE_HOME=/.substrate`, and keeps that path plus `/tmp` in `ReadWritePaths`. - `scripts/mac/lima-stop.sh` shuts the VM down cleanly; `scripts/mac/lima-doctor.sh` reports health (virtualization, agent socket, service status, forwarding tools). - The helper scripts substitute the active project path so `/src` inside the VM mirrors the host repo checkout. - If full isolation writable allowlists fail with `EPERM` in the guest, confirm the guest service has `cap_chown`: @@ -143,6 +197,9 @@ Hosted installer behavior coverage on macOS flows through this Lima-backed Linux 2. SSH Unix domain socket forwarding (`~/.substrate/sock/agent.sock`) 3. SSH TCP forwarding (`127.0.0.1:`) - The backend attempts transports in that order; failure logs include remediation hints and the shell degrades to host execution after a single warning if all transports fail. + - For async persistent-session startup on macOS, the shell now awaits the backend-owned async + readiness path before opening `/v1/stream` unless `SUBSTRATE_WORLD_SOCKET` is explicitly + overriding the transport. - Logs & diagnostics - Agent logs live in the guest: `substrate sudo journalctl -u substrate-world-agent -n 200` (the CLI shells into Lima automatically) or manually via `limactl shell substrate sudo journalctl -u substrate-world-agent -n 200`. @@ -150,8 +207,9 @@ Hosted installer behavior coverage on macOS flows through this Lima-backed Linux - Validation - `scripts/mac/smoke.sh` exercises non‑PTY, PTY, and replay flows on macOS and asserts that the replay `fs_diff` contains project paths. + - `scripts/mac/orchestration-smoke.sh` warms Lima, runs the live `world-mac-lima` backend smoke example, and then runs the macOS-targeted orchestration regression tests that cover shared-owner attach/create, replacement, lazy member launch, targeted follow-up reuse, guest-owned cancel, and shared-world mismatch rejection. - `scripts/mac/smoke.sh --bedpm-installer-conformance` runs the BEDPM Linux smoke wrapper through the same Lima-backed guest path so hosted installer verification reuses the authoritative Linux harness instead of implying native macOS package-manager selection. - - `scripts/linux/agent-hub-isolation-verify.sh` verifies `world_fs.mode=read_only` and `world_fs.isolation=full` enforcement (on macOS it drives the Lima-backed world; on Windows, use WSL-specific tooling instead). + - `scripts/linux/agent-hub-isolation-verify.sh` verifies `world_fs.mode=read_only` and `world_fs.isolation=full` enforcement (on macOS it drives the Lima-backed world). WSL-specific provisioning helpers are intentionally disabled in this slice. ## 4) Isolation Details (Linux) @@ -185,24 +243,36 @@ Socket: `/run/substrate.sock` - `GET /v1/doctor/world` - World enforcement readiness report (guest-kernel + agent view; used by `substrate world doctor`) - `POST /v1/execute` (non‑PTY) - - Body: `{ cmd, cwd, env, pty: false, agent_id, budget?, profile? }` - - Returns: `{ exit, span_id, stdout_b64, stderr_b64, scopes_used }` + - Body: `{ cmd, cwd, env, pty: false, agent_id, budget?, profile?, policy_snapshot, shared_world?, world_network?, world_fs_mode? }` + - `shared_world` request shape: `{ orchestration_session_id, action }` + - Returns: `{ exit, span_id, stdout_b64, stderr_b64, scopes_used, fs_diff?, shared_world?, ...process_telemetry }` + - `shared_world` response shape is the authoritative echoed proof: + `{ orchestration_session_id, world_id, world_generation, binding_state }` - `GET /v1/stream` (WebSocket, PTY) - Client → Server frames (text JSON): - `{"type":"start","cmd":"bash -lc ''","cwd":"/path","env":{...},"span_id":"spn_...","cols":,"rows":}` + - Legacy `start` does not support `shared_world`. + - Persistent session start: + `{"type":"start_session","protocol_version":1,"cwd":"/path","env":{...},"policy_snapshot":{...},"shared_world?":{...},"world_network?":{...},"cols":,"rows":}` - `{"type":"stdin","data_b64":"..."}` - `{"type":"resize","cols":,"rows":}` - `{"type":"signal","sig":"INT|TERM|HUP|QUIT"}` - Server → Client frames: + - Persistent-session ready: + `{"type":"ready","session_nonce":"","world_id":"wld_...","cwd":"/path","protocol_version":1,"shared_world?":{...}}` - `{"type":"stdout","data_b64":"..."}` + - Persistent-session command completion: + `{"type":"command_complete","seq":,"token_hex":"","exit":0,"cwd":"/path"}` - `{"type":"exit","code":0}` - - `{"type":"error","message":"..."}` + - `{"type":"error","code":"...","message":"...","fatal":true,"seq?":}` - `GET /v1/trace/:span_id` (placeholder) - `POST /v1/request_scopes` (placeholder) - `POST /v1/gc` (netns GC; see §10) Notes - Protocol is stable; no TLS by design (UDS only). +- The shared-world proof is additive. Generic callers omit `shared_world` entirely and do not receive owner-bound proof fields. +- PTY shared-world requests must use `start_session`; the agent rejects `shared_world` on legacy `start`. - `profile` is an advanced request field used for backend-specific execution posture. For example, `substrate world enable --provision-deps` sets `profile=world-deps-provision` so provisioning can run with the expected world-agent behavior. On guest Linux agents, that profile is executed via a @@ -220,21 +290,21 @@ Notes ## 6) Shell Behavior -- Default‑on world (Linux, macOS, Windows) +- Default‑on world (Linux, macOS) - On startup, the shell ensures a session world and sets `SUBSTRATE_WORLD=enabled` plus `SUBSTRATE_WORLD_ID`. - macOS builds warm the Lima VM, establish forwarding, and reuse the same backend factory used on Linux. - - Windows hosts call `platform_world::windows::ensure_world_ready`, which provisions/warms the `substrate-wsl` distro (via the PowerShell helpers) and keeps the world agent reachable through the forwarder named pipe. - The gateway lifecycle/status meaning is owned by the runtime parity contract; this section only records how the shell reaches the agent on each platform. - Routing - - Non‑PTY: POST to `/v1/execute` over UDS (Linux) or the forwarded socket/port (macOS/Windows). + - Non‑PTY: POST to `/v1/execute` over UDS (Linux) or the forwarded socket/port (macOS). - PTY: use WS to `/v1/stream` over the active transport; host fallback only occurs when `world_fs.require_world=false`. + - Explicit shared-owner requests keep Linux as the source contract, but the supported macOS/Lima path now forwards the same attach/create and replacement proof flow instead of rejecting before bootstrap. Windows still rejects explicit shared-owner requests before bootstrap/fallback rather than silently degrading to generic reuse. - Prompt safety - The REPL wraps PTY runs in `reedline::suspend_guard()` to avoid prompt corruption during external output. - Readiness & auto‑spawn - The shell probes `/v1/capabilities`; if stale socket is found, it removes it. - If the agent isn’t running, the shell attempts to spawn it (Linux dev flow: `target/debug/world-agent`). - macOS invokes the Lima backend ensure path to boot the VM and wire up its tunnel. - - Windows triggers the forwarder warm routine; see `docs/cross-platform/wsl_world_setup.md` for the underlying PowerShell flow. + - Windows/WSL helper flows are intentionally fail-closed in this slice; see `docs/cross-platform/wsl_world_setup.md`. - Fallback - With `world_fs.require_world=false`, exactly one warning is printed if the world cannot be reached; execution continues on the host in that situation. - With `world_fs.require_world=true`, world routing failures are treated as hard errors (no host fallback). @@ -322,6 +392,7 @@ Legacy `world-deps.yaml` overlay plumbing and `SUBSTRATE_WORLD_DEPS_MANIFEST` ar - macOS quick validation - `scripts/mac/lima-doctor.sh` - `PATH="$(pwd)/target/debug:$PATH" scripts/mac/smoke.sh` (non‑PTY, PTY, replay + fs_diff assertion) + - `PATH="$(pwd)/target/debug:$PATH" scripts/mac/orchestration-smoke.sh` (Lima warm + live backend reachability + shared-owner/member-runtime orchestration contract regressions) - `substrate sudo journalctl -u substrate-world-agent -n 200` (or `limactl shell substrate sudo journalctl -u substrate-world-agent -n 200`) to review guest logs --- @@ -367,7 +438,7 @@ Implemented features: - The host CLI exposes the same inventory via `substrate world cleanup`. Without flags it reports idle/active namespaces, cgroups, and host-level nft tables plus the exact manual commands needed to purge them. - Add `--purge` (and run as root/CAP_NET_ADMIN) to delete idle `substrate-` netns entries, their nft tables, and matching `/sys/fs/cgroup/substrate/` directories. -- macOS + Lima: run the helper inside the guest (`limactl shell substrate sudo substrate world cleanup --purge`). WSL follows the same pattern (`wsl -d substrate-wsl -- sudo substrate world cleanup --purge`). +- macOS + Lima: run the helper inside the guest (`limactl shell substrate sudo substrate world cleanup --purge`). - When purge isn't available, follow the printed instructions (`sudo ip netns exec ... nft delete table inet substrate_`, `sudo ip netns delete ...`, `sudo rm -rf /sys/fs/cgroup/substrate/`). ### Isolation fallback diagnostics @@ -384,7 +455,7 @@ Implemented features: ## 12) Limitations & Next Steps - PTY overlay fs_diff is intentionally deferred; non‑PTY continues to provide `fs_diff`. -- macOS/Windows support is "observe‑only" for worlds. +- Windows/WSL provisioning helpers remain unsupported in this slice and exit fail-closed instead of mutating the guest. - Next steps - Consider PTY overlay or post‑exit diff per span as a follow‑up phase diff --git a/docs/contracts/substrate-gateway-policy-evaluation.md b/docs/contracts/substrate-gateway-policy-evaluation.md index dd13901db..5ef46492a 100644 --- a/docs/contracts/substrate-gateway-policy-evaluation.md +++ b/docs/contracts/substrate-gateway-policy-evaluation.md @@ -30,7 +30,7 @@ Concrete rules: - Host credential file reads are permitted only as a host-side fallback when the required env auth handoff is absent; file material must not override, supplement, or merge with an already-present env handoff. - If env auth material is present but blocked by `llm.secrets.env_allowed`, the result is policy denial; host credential file fallback must not be used to bypass that denial. - If partial env auth material is present for the selected backend, the request is invalid integration and must fail closed rather than backfilling missing fields from a host credential file read. -- Those precedence rules govern the content of the integrated handoff, not the durable host-to-world carrier. Current integrated delivery may still use env-based transport for the closed auth field set, while the preferred additive direction is a Substrate-owned secret-channel payload plus inherited FD/pipe-style auth-bundle delivery so secret values do not live in the in-world process environment by default. +- Those precedence rules govern the content of the integrated handoff, not the durable host-to-world carrier. The landed integrated carrier is a Substrate-owned secret-channel auth bundle delivered over an inherited FD (`SUBSTRATE_LLM_AUTH_BUNDLE_FD`), and the in-world gateway must consume that pointer once without depending on secret-bearing child env vars. - Backend-specific auth renderers may differ, but they must preserve the same precedence and fail-closed rules above; carrier choice does not change policy authorization. - Gateway-local config, admin mutation surfaces, and token persistence remain implementation details of `substrate-gateway`; they do not become trusted policy inputs. @@ -51,6 +51,6 @@ Not defined here: - host secret sourcing and delivery remain policy-gated and Substrate-owned - treat complete allowlisted env auth material as primary and host credential files as fallback-only when env auth is absent - do not merge partial env auth material with host credential file reads; fail as invalid integration instead -- keep auth-source precedence separate from carrier choice; current env delivery is compatible with this rule, but the preferred additive transport remains a secret-channel FD/auth-bundle path +- keep auth-source precedence separate from carrier choice; the landed integrated transport is the secret-channel FD/auth-bundle path, and carrier choice does not relax precedence or fail-closed rules - policy evaluation remains part of the Substrate operator surface - keep invalid integration state distinct from dependency unavailability and policy denial diff --git a/docs/cross-platform/wsl_world_setup.md b/docs/cross-platform/wsl_world_setup.md index 03dac57a4..dc075157b 100644 --- a/docs/cross-platform/wsl_world_setup.md +++ b/docs/cross-platform/wsl_world_setup.md @@ -1,257 +1,70 @@ - # Windows WSL World Setup Guide -This guide explains how to provision and maintain the `substrate-wsl` -distribution that powers the Windows Always World stack. The WSL backend is -functional but experimental—expect ongoing updates and occasional rough edges. -Follow these steps so every `substrate` command executes inside WSL with full -telemetry and isolation. - -## Prerequisites - -### Host requirements +The owned WSL provisioning helpers are intentionally fail-closed in this slice. -1. **Windows build**: Windows 11 22H2 (or later) or Windows 10 22H2 with WSL2. +Current posture: -1. **Virtualization**: Firmware virtualization must be enabled. Verify with: +- `scripts/windows/wsl-warm.ps1` exits explicitly instead of importing or mutating a WSL distro. +- `scripts/wsl/provision.sh` exits explicitly instead of installing packages or writing systemd units. +- This is deliberate: the WSL path is not yet aligned with the Linux/macOS placement contract for: + - `SUBSTRATE_HOME` placement + - `/run/substrate.sock` ownership and group access + - managed runtime artifact access under `/run/substrate/substrate-gateway-runtime/` -```powershell -systeminfo | Select-String "Virtualization" -``` +## What To Do Instead - Each reported capability should end with `Yes`. +- For supported world provisioning, use: + - Linux host-native: `scripts/linux/world-provision.sh` + - macOS Lima guest: `scripts/mac/lima-warm.sh` +- For a CLI-only workflow inside WSL, install with `--no-world` and do not run the WSL warm/provision helpers. -1. **Optional features**: Enable WSL and VirtualMachinePlatform. Check with: +Example CLI-only install inside WSL: -```powershell -Get-WindowsOptionalFeature -Online -FeatureName Microsoft-Windows-Subsystem-Linux -Get-WindowsOptionalFeature -Online -FeatureName VirtualMachinePlatform +```bash +curl -fsSL https://raw.githubusercontent.com/atomize-hq/substrate/main/scripts/substrate/install.sh | bash -s -- --no-world ``` - Both commands should report `State : Enabled`. If not, enable and reboot. - -1. **WSL status**: Confirm WSL is healthy and the default version is 2: +Example dev install inside WSL: -```powershell -wsl --status +```bash +scripts/substrate/dev-install-substrate.sh --no-world ``` -1. **PowerShell 7**: Required for the helper scripts. Install if `pwsh` is - missing: +## Expected Failure Mode -```powershell -winget install --id Microsoft.PowerShell -e --scope user -``` +If you run the WSL helpers anyway, they should fail early with an explicit unsupported message. -1. **Developer toolchain**: Ensure Git, Rust, and Python are present: +Expected commands: ```powershell -winget install --id Git.Git -e -winget install --id Rustlang.Rustup -e -winget install --id Python.Python.3.12 -e +pwsh -File scripts/windows/wsl-warm.ps1 -DistroName substrate-wsl -ProjectPath (Resolve-Path .) ``` -1. **Execution policy**: Allow local scripts for the current user: - -```powershell -Set-ExecutionPolicy -Scope CurrentUser -ExecutionPolicy RemoteSigned +```bash +scripts/wsl/provision.sh ``` -### Repository location - -Clone `substrate` into a path without spaces (the plan references -`C:\workspace\substrate`). Run helper scripts from the repo root or pass -`-ProjectPath` when using a different path. - -## First-time setup - -### Step 1: Warm the environment - -The warm script performs these tasks: - -- Downloads the latest Ubuntu 24.04 `.wsl` image from Canonical. -- Imports (or upgrades) the `substrate-wsl` distribution as WSL2. -- Executes `scripts/wsl/provision.sh` inside the distro to install packages - and configure the `substrate-world-agent` service. -- Builds `substrate-forwarder` and `world-agent` if needed, copies the agent - into WSL, and restarts the service. -- Starts the Windows forwarder and probes `\.\pipe\substrate-agent` with a - client connect. - -Run the script from the repo root: - -```powershell -pwsh -File scripts/windows/wsl-warm.ps1 ` - -DistroName substrate-wsl ` - -ProjectPath (Resolve-Path .) -``` - -Sample success excerpt: - -```text -[INFO] Starting wsl-warm for distro 'substrate-wsl' -[INFO] Project path: C:\workspace\substrate -[INFO] Importing distro 'substrate-wsl' -[INFO] Downloading Ubuntu WSL image (noble-wsl-amd64.wsl) -[INFO] Probing forwarder pipe \.\pipe\substrate-agent -[INFO] Forwarder pipe accepted probe in 420 ms -[INFO] Forwarder pipe ready -[INFO] Warm complete -``` - -> Tip: Pass `-WhatIf` for dry runs. It validates prerequisites without -> downloading or provisioning. - -### Step 2: Validate the distro - -Ensure the distro exists after warm finishes: - -```powershell -wsl -l -v | Select-String substrate-wsl -``` - -The state should be `Running` immediately after warm or `Stopped` later on. - -### Step 3: Confirm agent and forwarder - -Check both sides of the bridge: +Expected result: -```powershell -# inside WSL -wsl -d substrate-wsl -- bash -lc 'systemctl status substrate-world-agent.socket && systemctl status substrate-world-agent.service' -wsl -d substrate-wsl -- bash -lc "substrate host doctor --json | jq '.host.world_socket'" - -# on Windows -Get-Content "$env:LOCALAPPDATA\Substrate\logs\forwarder.log" -Tail 50 -# Logs rotate daily (5 files, 10 MB each) -``` +- Exit code `4` +- No package-manager mutation +- No systemd unit rewrite +- No WSL distro import or reprovisioning side effects -## Updating the agent binary +## Verification -When `world-agent` changes, rebuild and rerun warm so the new binary is copied -into WSL: +Use these checks to confirm you stayed on the fail-closed path: ```powershell -cargo build -p world-agent --release -pwsh -File scripts/windows/wsl-warm.ps1 ` - -DistroName substrate-wsl ` - -ProjectPath (Resolve-Path .) +$LASTEXITCODE ``` -A future plan step adds `-SkipProvision`; until then, rerunning the full warm -script is safe and idempotent. - -## Running the doctor script - -Use the doctor script to verify host prerequisites, distro health, forwarder, -and agent endpoints: - ```powershell -pwsh -File scripts/windows/wsl-doctor.ps1 -DistroName substrate-wsl -pwsh -File scripts/windows/wsl-doctor.ps1 -DistroName substrate-wsl -Json ` - | ConvertFrom-Json | Format-Table -``` - -Expected PASS output (Appendix I): - -```text -[substrate/windows doctor] -Virtualization: PASS (Enabled) -WSL status: PASS (Default version 2, kernel 5.15.133.1) -Distro substrate-wsl: PASS (Running) -Forwarder pipe: PASS (\.\pipe\substrate-agent) -Agent socket: PASS (/run/substrate.sock exists) -Agent capabilities: PASS (features: execute, pty_streaming, trace_retrieval) -Nftables: PASS -Logs: PASS (no errors in last 100 lines) +wsl -l -v ``` -If a check fails, read the remediation column in the table and consult -`docs/dev/wsl_world_troubleshooting.md` for detailed guidance. - -## Smoke test - -After Phase 5 scripts are in place, run the smoke suite for end-to-end -validation: - ```powershell -pwsh -File scripts/windows/wsl-smoke.ps1 ` - -DistroName substrate-wsl ` - -ProjectPath (Resolve-Path .) +Get-ChildItem "$env:LOCALAPPDATA\\Substrate" -ErrorAction SilentlyContinue ``` -The script performs: - -1. Warm (unless `-SkipWarm` is supplied). -1. Doctor checks. -1. Non-PTY span verification to ensure the latest trace entry includes - `world_id` and `fs_diff`. -1. PTY execution test through ConPTY and WebSocket. -1. Replay of the most recent span. -1. Forwarder restart resilience. - -Record the console output in the evidence log after each run. - -## Common operations - -- **Stop distro and forwarder** - - ```powershell - pwsh -File scripts/windows/wsl-stop.ps1 -DistroName substrate-wsl - ``` - -- **Launch an interactive shell** - - ```powershell - wsl -d substrate-wsl -- bash - ``` - -- **View agent logs** - - ```powershell - wsl -d substrate-wsl -- bash -lc 'journalctl -u substrate-world-agent -n 200' - ``` - -- **Tail forwarder log:** - - ```powershell - Get-Content "$env:LOCALAPPDATA\Substrate\logs\forwarder.log" -Tail 200 -Wait - ``` - -- **Run forwarder with timeout** (manual debugging) - - ```powershell - pwsh -File scripts/windows/start-forwarder.ps1 -DistroName substrate-wsl - ``` - -## Troubleshooting - -A dedicated catalogue lives at `docs/dev/wsl_world_troubleshooting.md`. It -covers scenarios such as: - -- Virtualization disabled or blocked by VBS. -- Optional feature enable failures. -- WSL import errors or stale `.wsl` images. -- Forwarder pipe permissions or orphaned PID files. -- Agent provisioning failures within the distro. -- ConPTY or WebSocket connectivity issues. -- Windows/WSL path translation mismatches. - -Use the catalogue when doctor or smoke checks fail, and include the relevant -entry ID in the evidence log when documenting remediation. - -## Maintenance checklist - -1. Keep the helper scripts aligned with the plan appendices. -1. Run the doctor script after OS, firmware, or WSL updates. -1. Re-run the smoke script after modifying `world-agent` or forwarder code. -1. For releases, capture doctor and smoke outputs and attach them to the - Windows evidence log. - -## Support - -- Team channel: `#substrate-windows` -- Plan owner: Substrate Core - -Follow the Phase 5 guardrails: execute steps in order, capture evidence after -Each check, and stop immediately if a command fails. +You should see the helper exit with `4`, and no new WSL world provisioning state should appear as a consequence of these owned scripts. diff --git a/docs/internals/repl/persistent_session.md b/docs/internals/repl/persistent_session.md index d958ecbaa..d66f69008 100644 --- a/docs/internals/repl/persistent_session.md +++ b/docs/internals/repl/persistent_session.md @@ -54,3 +54,23 @@ When uncaged traversal moves outside the workspace: - a drift restart can be expected (new cwd ⇒ potentially different workspace root and policy), and - the REPL should attempt best-effort cwd continuity on restart (see DR-17 in the project docs). +## Host Readiness Split + +Persistent-session startup now preserves a caller-shape split between sync bootstrap code and the +async REPL startup path. + +- Sync callers still use `PlatformWorldContext.ensure_ready()`. That remains the shell-facing sync + bridge for request builders and bootstrap flows that are not already inside async startup. +- The macOS async persistent-session startup path no longer calls that sync bridge. When + `build_ws_and_start_session_frame(...)` needs to bring the Lima-backed world path online without a + `SUBSTRATE_WORLD_SOCKET` override, it now awaits + `PlatformWorldContext.ensure_persistent_session_ready_async()`, which delegates to backend-owned + readiness. +- On macOS, the backend-owned async path is `MacLimaBackend::ensure_persistent_session_ready_async`. + The shell does not duplicate VM startup, forwarding, or readiness verification logic. +- `SUBSTRATE_WORLD_SOCKET` keeps exact bypass semantics. If callers override the socket path, the + persistent-session client connects directly to that socket and does not invoke platform readiness + helpers first. +- The Windows/WSL backend now mirrors the same split internally for parity, but this slice does not + ship a Windows persistent-session shell caller. The parity work only keeps backend-owned + readiness helpers aligned with the macOS/Linux contract. diff --git a/docs/project_management/adrs/draft/ADR-0021-substrate-workflow-engine.md b/docs/project_management/adrs/draft/ADR-0021-substrate-workflow-engine.md index 56a6790d9..8ac9672d5 100644 --- a/docs/project_management/adrs/draft/ADR-0021-substrate-workflow-engine.md +++ b/docs/project_management/adrs/draft/ADR-0021-substrate-workflow-engine.md @@ -27,6 +27,7 @@ - Trace + event foundations: `docs/project_management/adrs/draft/ADR-0028-in-world-process-execution-tracing-parity.md` - Output routing + attribution: `docs/project_management/adrs/draft/ADR-0017-agent-hub-concurrent-execution-and-output-routing.md` - Router daemon (workflow triggers, queues, cross-workspace): `docs/project_management/adrs/draft/ADR-0029-host-event-bus-and-router-daemon.md` + - Host orchestration durable session and inbox contract: `docs/project_management/adrs/draft/ADR-0047-host-orchestrator-durable-session-and-parked-resumable-ownership.md` - Config/policy surface (no new roots): `docs/project_management/adrs/draft/ADR-0027-llm-and-agent-config-policy-surface.md` ## Executive Summary (Operator) @@ -164,6 +165,7 @@ ADR_BODY_SHA256: 0e7a75f2761d8bab7aa43ea95c05e6ab5eed4da516d7c1fdcce7c489a0e305f - human-readable streaming progress, - workflow result summary (success/failure and per-node statuses), - trace spans written to `~/.substrate/trace.jsonl` (default path) with graph linkage. + - when a workflow run or node produces host-attention, approval, completion, or follow-up work for an active host orchestration session, that outcome must be representable as an ADR-0047-compatible session-local inbox item rather than a workflow-only side artifact. ## Sequencing / Dependencies - Sequencing entry: this ADR must add a `workflow-engine` entry to `docs/project_management/packs/sequencing.json` before it can be marked `Accepted`. @@ -171,6 +173,7 @@ ADR_BODY_SHA256: 0e7a75f2761d8bab7aa43ea95c05e6ab5eed4da516d7c1fdcce7c489a0e305f - Dependencies: - Reuse existing Agent API request/response + streaming frame model (`crates/agent-api-*`) for tool/script execution nodes. - Reuse `substrate-trace` for span persistence and replay association. + - Any workflow correlation fields that surface outside the workflow runtime (`workflow_id`, `workflow_run_id`, `workflow_node_id`, and adjacent cause/join keys) must remain additive and joinable with ADR-0047 session-local inbox items and ADR-0029 router records. ## Security / Safety Posture - Fail-closed rules: diff --git a/docs/project_management/adrs/draft/ADR-0029-host-event-bus-and-router-daemon.md b/docs/project_management/adrs/draft/ADR-0029-host-event-bus-and-router-daemon.md index 757dbc12d..43dae7163 100644 --- a/docs/project_management/adrs/draft/ADR-0029-host-event-bus-and-router-daemon.md +++ b/docs/project_management/adrs/draft/ADR-0029-host-event-bus-and-router-daemon.md @@ -18,6 +18,8 @@ - Trace/event foundations: - `docs/project_management/adrs/draft/ADR-0028-in-world-process-execution-tracing-parity.md` - `docs/project_management/adrs/draft/ADR-0017-agent-hub-concurrent-execution-and-output-routing.md` +- Host orchestration session compatibility: + - `docs/project_management/adrs/draft/ADR-0047-host-orchestrator-durable-session-and-parked-resumable-ownership.md` - FS path semantics & allow/deny matching: - `docs/project_management/adrs/implemented/ADR-0018-world-fs-granular-allow-deny-and-strict-deny.md` - Config/policy layering model: @@ -158,6 +160,11 @@ Only an explicit allowlist of event families is triggerable. v1 supports: - The workflow router uses durable JSONL queues under `SUBSTRATE_HOME/workflow/`: - `inbox.jsonl` stores **requests** (durable “intent to act”). - `work_queue.jsonl` stores **actions** derived from requests after routing/policy evaluation. +- Compatibility boundary: + - These workflow-router queues are not the same store as the orchestration-session-local inbox defined by ADR-0047. + - `workflow/inbox.jsonl` and `workflow/work_queue.jsonl` are router-owned request/action queues scoped to routing and execution policy. + - ADR-0047 inbox items are session-local durable retained events scoped to one host orchestration session. + - The stores have different identities and lifecycles, but they MUST use compatible correlation/join vocabulary so a routed request, workflow run, and session-local host-attention item can be joined without heuristic reconstruction. - Processing semantics: - Handling is at-least-once; duplicate processing MUST be bounded via dedupe keys. - Each request/action MUST have a stable idempotency key derived from: diff --git a/docs/project_management/adrs/draft/ADR-0045-orchestration-toolbox-internal-mcp-identity-trace-contract.md b/docs/project_management/adrs/draft/ADR-0045-orchestration-toolbox-internal-mcp-identity-trace-contract.md index 723d6a23e..f9e085445 100644 --- a/docs/project_management/adrs/draft/ADR-0045-orchestration-toolbox-internal-mcp-identity-trace-contract.md +++ b/docs/project_management/adrs/draft/ADR-0045-orchestration-toolbox-internal-mcp-identity-trace-contract.md @@ -195,6 +195,7 @@ Operator rule: - `component: "agent-toolbox"` - `session_id` - `orchestration_session_id` + - `participant_id` - `run_id` - `agent_id` - `client` @@ -229,6 +230,7 @@ Operator rule: - `component=agent-toolbox` - `session_id=` - `orchestration_session_id=` + - `participant_id=` - `run_id=` - `agent_id=` - `client=claude_code` diff --git a/docs/project_management/adrs/draft/ADR-0047-host-orchestrator-durable-session-and-parked-resumable-ownership.md b/docs/project_management/adrs/draft/ADR-0047-host-orchestrator-durable-session-and-parked-resumable-ownership.md new file mode 100644 index 000000000..321527005 --- /dev/null +++ b/docs/project_management/adrs/draft/ADR-0047-host-orchestrator-durable-session-and-parked-resumable-ownership.md @@ -0,0 +1,421 @@ +# ADR-0047 — Host Orchestrator Durable Session and Parked-Resumable Ownership + +## Status +- Status: Draft +- Date (UTC): 2026-05-09 +- Owner(s): Spenser McConnell (Substrate) + +## Scope +- Feature directory: `docs/project_management/packs/draft/host-orchestrator-durable-session-and-parked-resumable-ownership/` +- Sequencing spine: `docs/project_management/packs/sequencing.json` +- Standards: + - `docs/project_management/system/standards/adr/EXECUTIVE_SUMMARY_STANDARD.md` + +## Related Docs (links only) + +This ADR is a lifecycle correction for host orchestration. It does not replace the existing public caller surfaces; it clarifies what those surfaces bind to and what runtime durability they require. + +- Foundational runtime / ownership ADRs: + - `docs/project_management/adrs/draft/ADR-0040-substrate-gateway-boundary-and-runtime-ownership.md` + - `docs/project_management/adrs/draft/ADR-0042-llm-and-agent-identity-tuple-and-deployment-posture.md` +- Public caller-surface and execution-plan context: + - `llm-last-mile/PLAN-22.md` + - `llm-last-mile/ORCH_PLAN-22.md` + - `llm-last-mile/22-broaden-caller-surfaces-from-repl-first-to-public-session-member-turns.md` +- Existing runtime hotspots and observed failure seam: + - `crates/shell/src/execution/agents_cmd.rs` + - `crates/shell/src/execution/agent_runtime/control.rs` + - `crates/shell/src/execution/agent_runtime/session.rs` + - `crates/shell/src/execution/agent_runtime/state_store.rs` + - `crates/shell/src/repl/async_repl.rs` +- Future-compatible workflow/router correlation context: + - `docs/project_management/adrs/draft/ADR-0029-host-event-bus-and-router-daemon.md` + - `docs/project_management/adrs/draft/ADR-0021-substrate-workflow-engine.md` + - `docs/project_management/packs/PHASE_8_CROSS_CUTTING_DECISION_REGISTRY.md` +- Existing proof surfaces: + - `crates/shell/tests/agent_public_control_surface_v1.rs` + - `crates/shell/tests/repl_world_first_routing_v1.rs` + +## Executive Summary (Operator) + +ADR_BODY_SHA256: ebae56fd591e1cffea1d3e8262ed660fcf9123ef5d1574b7a3a98783525154cc + +### Changes (operator-facing) +- Clarify that a host orchestration session is durable even when the current Codex wrapper process is not + - Existing: host orchestration readiness and liveness are treated as if a transient helper process must remain continuously attached for the orchestration session to stay valid. + - New: the durable unit is the Substrate-owned orchestration session plus its inbox/task state. A Codex session is an attachable execution client, not the durable orchestration authority itself. + - Why: the intended `codex exec`-style launch model is prompt-driven and lazy. Requiring a continuously running attached client reintroduces the old bootstrap-first assumption that the current public surface intentionally moved away from. + - Links: + - `crates/shell/src/execution/agent_runtime/session.rs` + - `crates/shell/src/repl/async_repl.rs` +- Preserve the current public CLI surface while redefining the meaning of host session persistence + - Existing: `substrate agent start` and `substrate agent turn` are already the canonical public prompt-taking surfaces, but the runtime still tends to interpret clean helper exit as owner loss or invalidation. + - New: `start`, `turn`, and `reattach` remain the public surface, but they bind to durable orchestration state with explicit attached, parked, and attention-needed postures. + - Why: this keeps the public contract stable while correcting the underlying ownership model. + - Links: + - `llm-last-mile/PLAN-22.md` + - `crates/shell/src/execution/agent_runtime/control.rs` +- Tighten the prompt-stream delivery contract after `Accepted` + - Existing: the bridge can emit `Accepted` and later surface EOF because the owner-side control stream ended before a terminal envelope was emitted. + - New: once `Accepted` is emitted, the request must end with an explicit terminal envelope (`Completed` or `Failed`) on every path. + - Why: `Accepted` without terminal-delivery guarantees is an unstable public prompt contract and makes orchestrator parking indistinguishable from owner death. + - Links: + - `crates/shell/src/execution/agent_runtime/control.rs` + +## Problem / Context +- The current host orchestrator path still carries an assumption from an older bootstrap model: + - if the helper process becomes authoritative-live and later its control stream ends, the orchestration session is at risk of being invalidated as though durable ownership was lost. +- That assumption conflicts with the intended launch posture: + - REPL launch should not bootstrap a fake backend session with a dummy prompt. + - a host backend session should be created only when the user actually runs a prompt-taking action such as `::cli:codex ...` or `substrate agent start --backend ... --prompt ...`. + - the resulting backend process may behave like `codex exec`, meaning it is naturally prompt-driven and not guaranteed to remain alive indefinitely after the prompt completes. +- The deeper requirement is not "keep the Codex process alive forever." The real requirement is: + - the host orchestration session must remain durable, + - world-launched work must still be able to report end messages, approval needs, and follow-up state back into Substrate, + - and Substrate must be able to resume or reattach an execution client later without treating the orchestration session itself as lost. +- This implies a missing architectural boundary: + - Substrate needs a durable orchestration session and delivery surface that outlives any one Codex wrapper process. + - world-originated messages cannot depend on a particular attached client process still being alive. + +## Goals +- Make the Substrate-owned orchestration session the durable authority for host orchestration. +- Treat Codex or equivalent backend sessions as attachable execution clients rather than the durable orchestrator identity. +- Define an explicit parked/resumable host posture that is valid and routable, not an implicit invalidation state. +- Preserve the existing public CLI contract: + - `substrate agent start` remains the canonical public root prompt-taking surface. + - `substrate agent turn` remains the canonical public follow-up surface. + - `substrate agent reattach` remains the canonical explicit reattachment surface. +- Require post-`Accepted` terminal-delivery guarantees for the public prompt bridge. +- Preserve strict world posture rules: + - root `start` remains host-only in v1, + - world-only root start stays fail-closed, + - detached-world follow-up remains fail-closed until a valid host owner is re-established through the sanctioned path. +- Create an architecture shape that will remain compatible with a future Substrate-native context/prompt builder. + +## Non-Goals +- Changing the public selector contract for `turn`; exact `(orchestration_session_id, backend_id)` remains authoritative. +- Introducing fuzzy routing, default routing, or new public selector types. +- Changing Linux world-member follow-up away from `MemberTurnSubmitRequestV1` and `/v1/member_turn/stream`. +- Replacing the current world follow-up ownership rules with direct world-to-world public routing. +- Designing the full native prompt builder in this ADR. +- Defining a remote multi-tenant orchestration control plane. + +## User Contract (Authoritative) + +### CLI +- Commands: + - `substrate agent start --backend --prompt ...` remains the canonical public root prompt-taking surface. + - `substrate agent turn --session --backend --prompt ...` remains the canonical public follow-up surface. + - `substrate agent reattach --session ` remains the canonical explicit host reattachment surface. + - `substrate agent stop --session ` remains the canonical explicit shutdown surface. +- Root prompt rules: + - root `start` remains host-only in v1. + - world-only backends must fail closed as root-start targets. +- Follow-up rules: + - public `turn` requires exact `--session ` and exact `--backend `. + - no fuzzy session routing, no fuzzy backend routing, and no latest-session fallback are introduced here. +- Delivery guarantee: + - once a public prompt request emits `Accepted`, it MUST terminate with an explicit terminal envelope from the Substrate-owned bridge. + - EOF or stream disappearance without a terminal envelope is a runtime bug, not an operator-facing steady-state contract. + +### Session posture contract +- `active_attached` + - A host execution client is currently attached to the orchestration session and can actively receive prompt traffic and interactive work. +- `parked_resumable` + - The orchestration session remains valid. + - No host execution client is currently attached. + - Substrate retains durable routing state, inbox state, and enough session identity to resume host execution later. + - This is the normal state after a clean prompt-driven host client exit when orchestration remains live. +- `awaiting_attention` + - The orchestration session remains valid but has pending world-originated work, completion messages, or approval requirements that require host-side review or resumption. +- `terminal` + - The orchestration session is closed, invalidated, or fully completed and can no longer accept prompt or orchestration traffic. + +### Detached-host versus detached-world rules +- Host: + - a parked host session is valid and resumable. + - clean host client exit after valid session establishment must not be treated as equivalent to fatal owner loss. +- World: + - detached-world follow-up remains fail-closed. + - world follow-up must not broaden into self-sustaining public continuity without a valid host orchestration session that remains routable through Substrate. + +### Event delivery contract +- World-originated approvals, completion notices, and other orchestration-relevant events must land in a durable Substrate-owned inbox, queue, or task ledger that outlives any attached Codex client process. +- A live attached client may consume from that inbox in real time. +- If no client is attached, Substrate must retain the event and expose the need for resume/reattach rather than losing or invalidating the orchestration session. + +## Internal Runtime-State Schema (Authoritative, internal) + +This section defines the minimum retained runtime-state schema required to implement the durable host-orchestration contract in this ADR. + +This is an internal persisted runtime-state contract. It is not a config/policy surface, does not add operator-settable keys, and does not change the file-family ownership defined by ADR-0027. + +### Canonical filesystem layout +- Canonical live-state authority remains under: + - `~/.substrate/run/agent-hub/sessions//session.json` + - `~/.substrate/run/agent-hub/sessions//participants/.json` + - `~/.substrate/run/agent-hub/sessions//leases/.lease` +- This ADR adds a canonical durable inbox path under the same session root: + - `~/.substrate/run/agent-hub/sessions//inbox/.json` +- The session-root files above are the sole live-state authority from first write. + +### Orchestration session record +- Existing `session.json` fields remain authoritative: + - `orchestration_session_id` + - `shell_trace_session_id` + - `workspace_root` + - `shell_owner_pid` + - `state` + - `opened_at` + - `last_active_at` + - `orchestrator_agent_id` + - `orchestrator_backend_id` + - `orchestrator_protocol` + - `active_session_handle_id` + - `latest_run_id` + - `world_id` + - `world_generation` + - `invalidation_reason` + - `closed_at` +- Required additive fields: + - `posture: active_attached|parked_resumable|awaiting_attention|terminal` + - `posture_changed_at: ` + - `attached_participant_id: |null` + - `pending_inbox_count: ` + - `last_parked_at: |null` + - `last_attention_at: |null` + - `parked_reason: |null` +- Required semantics: + - `state` remains the lifecycle state machine for allocation, active execution, stopping, failure, invalidation, and terminal completion. + - `posture` is the attachability and attention summary. It must not be inferred solely from `state`. + - `active_session_handle_id` retains its existing compatibility meaning: the authoritative orchestrator participant for the orchestration session. It is not proof that a host client is currently attached. + - `attached_participant_id` is the authoritative pointer to the currently attached host execution client. It must be `null` whenever `posture` is `parked_resumable`, `awaiting_attention`, or `terminal`. + - `pending_inbox_count` counts unresolved inbox items for the orchestration session. + - `posture=awaiting_attention` is required when the session remains non-terminal, `attached_participant_id=null`, and `pending_inbox_count>0`. + - `posture=parked_resumable` is required when the session remains non-terminal, `attached_participant_id=null`, `pending_inbox_count=0`, and at least one authoritative host participant remains `resume_eligible=true`. + - `posture=terminal` must align with a non-routable session state such as `Invalidated`, `Stopped`, or `Failed`. + - `posture` is explicit persisted truth and must not be reconstructed heuristically from attachment diagnostics or legacy attachment flags. + +### Participant record +- Existing participant fields remain authoritative: + - handle fields such as `participant_id`, `orchestration_session_id`, `backend_id`, `role`, `protocol`, `state`, lineage links, `world_id`, and `world_generation` + - internal fields such as `uaa_session_id`, `latest_run_id`, `cancel_supported`, `ownership_mode`, `ownership_valid`, `last_heartbeat_at`, `last_event_at`, `terminal_observed_at`, `termination_reason`, `last_error_bucket`, and `last_error_message` +- Required additive internal fields for host-orchestrator participants: + - `attached_client_present: ` + - `last_attached_at: |null` + - `last_detached_at: |null` + - `detach_reason: |null` + - `resume_eligible: ` +- Required semantics: + - `uaa_session_id` remaining populated after clean client exit is valid and expected for a parked-resumable host session. + - `resume_eligible=true` with `attached_client_present=false` and a non-terminal participant `state` is a valid parked host posture, not a failure condition. + - `control_owner_retained`, `event_stream_active`, and `completion_observer_retained` remain attachment diagnostics. They must no longer be treated as the sole proof that the orchestration session itself is valid. + - These new attachment fields are required for host-orchestrator participants. Member-runtime participants may leave them unset or at safe defaults when the semantics do not apply. + - `uaa_session_id` is an identifier and correlation field, not proof of attachment, liveness, or resumability on its own. + +### Durable inbox item +- Each unresolved or retained orchestration event must be persisted as one file under: + - `sessions//inbox/.json` +- Minimum item schema: + - `orchestration_session_id: ` + - `item_id: ` + - `kind: approval_required|completion_notice|follow_up_message|runtime_alert` + - `state: pending|acknowledged|dismissed` + - `created_at: ` + - `updated_at: ` + - `resolved_at: |null` + - `message: |null` +- Required semantics: + - `state=pending` items contribute to `pending_inbox_count`. + - Lack of an attached host client must never delete, skip, or silently consume a pending item. + - A live attached host client may observe and acknowledge items in real time, but the persisted inbox item remains the durable source of truth until it is resolved. + - Resolved items may be compacted later, but only after they have transitioned out of `pending`. + - `item_id` is the session-local durable inbox record identifier. It is not a router `request_id`, workflow run identifier, or trace span identifier. + - The currently landed schema is intentionally minimal. Richer correlation or payload envelopes remain additive follow-on work rather than implied current-state truth. + +### Lease file +- Existing lease payloads may remain minimal and additive. +- If additive attachment fields are mirrored into lease files for fast-path checks, `session.json` and `participants/.json` remain authoritative. + +### Greenfield runtime-state adoption rules +- This runtime-state schema is greenfield. +- Older flat session, participant, or lease layouts are not part of the contract. +- New runtime-state writers MUST write the session-root layout and required posture/attachment fields from first write. +- Readers MUST treat the persisted fields in this section as authoritative rather than reconstructing posture from attachment heuristics. +- No config schema change and no policy schema change is required by this runtime-state schema. + +## Resolved Design Decisions + +These decisions are no longer open in this ADR. They are pinned here so the durable-session contract stays canonical and retained. + +### `reattach` semantics are attached-owner recovery only +- `substrate agent reattach --session ` is the explicit attached-owner recovery verb. +- `reattach` MUST restore a live attached owner loop for the selected host orchestration session when recovery metadata is intact. +- `reattach` MUST NOT submit a prompt, consume inbox work implicitly, or act as a one-shot follow-up-turn shortcut. +- One-shot prompt-taking resume stays on: + - `substrate agent turn --session --backend --prompt ...` +- Rationale: + - this matches the already-landed public caller-surface contract, + - preserves the distinction between operational ownership recovery and foreground turn submission, + - and avoids reintroducing ambiguous `resume` semantics into the public lifecycle surface. + +### Resolved inbox items remain retained for audit, then compacted later +- Resolving an inbox item MUST remove it from authoritative pending/live counts immediately. +- Resolving an inbox item MUST NOT immediately delete the persisted inbox artifact. +- Resolved inbox items SHOULD be retained until: + - the orchestration session has reached terminal state, + - the item is no longer `pending`, + - the item has aged past a bounded retention floor, and + - no unresolved item still depends on it for causation/debug correlation. +- Compaction is a maintenance step, not the acknowledgement primitive. +- This ADR intentionally fixes compaction eligibility rules but does not fix a numeric retention duration. +- The exact numeric retention floor is an implementation/operations follow-on and should be aligned later with adjacent router/workflow retention policy rather than guessed here. +- Rationale: + - the broader event-plane and trace contracts in this repo are append-only and audit-oriented, + - live-state authority and historical retention are intentionally distinct, + - and immediate deletion would make detached-host review and postmortem analysis weaker without providing a meaningful contract benefit. + +### Future inbox kinds remain additive within one canonical envelope +- The durable inbox outer envelope pinned by this ADR is canonical: + - `schema_version` + - `item_id` + - `orchestration_session_id` + - `kind` + - `state` + - `created_at` + - `resolved_at` + - `correlation` + - `payload_schema` + - `payload` +- Future inbox item kinds are allowed only as additive expansions of the `kind` enum. +- Future kinds MUST NOT introduce new ad hoc top-level envelope fields as their primary shape. +- Kind-specific structure MUST live under the versioned `payload_schema` plus `payload` contract. +- Any future cross-cutting join/correlation additions MUST remain additive and defer to ADR-0028 / Phase 8 for canonical naming and required/optional classification rather than creating inbox-only correlation dialects. +- Rationale: + - this preserves one durable session-local inbox envelope across approvals, completions, follow-up work, and later workflow-aware attention items, + - keeps inbox growth compatible with ADR-0029 router records and ADR-0021 workflow correlation, + - and avoids top-level envelope drift as future kinds are introduced. + +### Non-host attached-client generalization is out of scope for this ADR +- The attached-client metadata pinned by this ADR is intentionally host-oriented. +- No additional generalized attachment metadata is required in this ADR for hypothetical future non-host attached clients. +- If Substrate later introduces a new attached-client class that is not host-rooted, that work MUST land as an additive follow-on contract that: + - states the new attached-client type explicitly, + - defines any new persisted metadata fields explicitly, + - and proves that the broader attachment model does not weaken the current host-rooted fail-closed posture. +- Rationale: + - the current orchestration model is explicitly host-rooted, + - public root start remains host-only in v1, + - and broadening the attachment taxonomy now would speculate beyond the runtime model this ADR is actually standardizing. + +## Architecture Shape +- Components: + - `crates/shell` runtime/session/state layers: + - own the durable orchestration session identity, posture state, and routing gates. + - Substrate-owned durable inbox / task ledger: + - owns world-to-host message durability when no client is attached. + - attachable execution client: + - a Codex session or equivalent backend client that may attach, run, exit, and later resume against the same orchestration session. + - world-agent and world-member submit path: + - continue to own world-member turn execution and typed request submission for Linux world follow-up. +- End-to-end flow: + - Inputs: + - public `start`, `turn`, `reattach`, and `stop` + - world-originated completion, approval, or update events + - exact `(orchestration_session_id, backend_id)` routing inputs + - Derived state: + - orchestration session posture + - attached-client presence or absence + - pending durable inbox items + - whether follow-up is host-routable, world-routable, or fail-closed + - Actions: + - `start` creates or binds the durable orchestration session and runs the initial prompt + - if the prompt-driven host client exits cleanly, Substrate transitions to `parked_resumable` instead of invalidating + - world-originated messages land in the durable inbox regardless of client attachment + - `turn` or `reattach` resumes a valid host orchestration session as needed + - `Accepted` is emitted only under a bridge contract that guarantees terminal completion signaling + - Outputs: + - stable orchestration session state + - explicit attached/parked/attention-needed posture + - explicit terminal envelopes for accepted prompt requests + +## Sequencing / Dependencies +- Sequencing entry: `docs/project_management/packs/sequencing.json` → `host-orchestrator-durable-session-and-parked-resumable-ownership` or the next available orchestration-runtime slot +- Prerequisite integration task IDs: + - the public caller-surface hardening from `PLAN-22` remains prerequisite context + - the exact host/world posture and identity-tuple rules from ADR-0042 remain prerequisite context + - the detailed inbox/ledger runtime contract is expected as a follow-on contract/spec, not fully defined here + +## Work Lift (discovery estimate) + + +```json +{ + "model_version": 1, + "touch": { + "create_files": 1, + "edit_files": 8, + "delete_files": 0, + "deprecate_files": 0, + "crates_touched": 2, + "boundary_crossings": 2 + }, + "contract": { + "cli_flags": 0, + "config_keys": 0, + "exit_codes": 0, + "file_formats": 0, + "behavior_deltas": 1 + }, + "qa": { "new_test_files": 0, "new_test_cases": 10 }, + "docs": { "new_docs_files": 2 }, + "ops": { "new_smoke_steps": 1, "ci_changes": 0 }, + "risk": { + "cross_platform": true, + "security_sensitive": false, + "concurrency_or_ordering": true, + "migration_or_backfill": true, + "unknowns_high": true + }, + "notes": "Primary lift is runtime ownership/state semantics, retained internal session/participant/inbox schema updates, and accepted-to-terminal delivery guarantees. Config/policy schema is unchanged." +} +``` + + +## Security / Safety Posture +- Fail-closed rules: + - world-only root start remains rejected. + - detached-world public follow-up remains rejected until a sanctioned host path re-establishes routable ownership. + - public `turn` remains exact-selector only. + - world-originated messages must not bypass Substrate-owned routing and durability surfaces. +- Protected paths/invariants: + - a clean prompt-driven host client exit after session establishment must not silently degrade into session invalidation. + - accepted prompt requests must not end in silent stream loss. + - durable orchestration state must remain authoritative over any one attached client process. + +## Validation Plan (Authoritative) + +### Tests +- Unit tests: + - host session posture transitions: + - `active_attached` -> `parked_resumable` + - `parked_resumable` -> `active_attached` + - `parked_resumable` -> `awaiting_attention` + - legal transition to `terminal` + - accepted prompt bridge: + - once `Accepted` is emitted, the bridge always emits a terminal envelope on completion, cancellation, owner loss, or runtime invalidation +- Integration tests: + - host `start` creates a valid orchestration session and allows clean client exit without invalidation + - host `turn` succeeds against a valid parked host session + - `reattach` restores attached host posture from a valid parked host session + - world-originated completion or approval work is retained when no host client is attached + - detached-world follow-up remains fail-closed + - `repl_world_first_routing_v1.rs` remains green as a non-regression surface + +### Manual validation +- Re-run host public manual flow outside the integration harness: + - `substrate agent start --backend --prompt "hello" --json` + - `substrate agent turn --session --backend --prompt "next" --json` + - `substrate agent reattach --session --json` +- Confirm the host session parks cleanly when the prompt-driven client exits and can later resume. +- Confirm no post-`Accepted` stream ends without `Completed` or `Failed`. diff --git a/docs/project_management/packs/draft/agent-hub-core-successor-identity-tuple-compatible/agent-hub-session-protocol-spec.md b/docs/project_management/packs/draft/agent-hub-core-successor-identity-tuple-compatible/agent-hub-session-protocol-spec.md index 1b1435dec..8aadd4695 100644 --- a/docs/project_management/packs/draft/agent-hub-core-successor-identity-tuple-compatible/agent-hub-session-protocol-spec.md +++ b/docs/project_management/packs/draft/agent-hub-core-successor-identity-tuple-compatible/agent-hub-session-protocol-spec.md @@ -1,6 +1,6 @@ # agent-hub-session-protocol-spec -This file is the single authoritative schema inventory for ADR-0044 session semantics inside this feature pack. It defines the backend capability descriptor, the hub-owned session-handle object, the lifecycle state machine, the host-orchestrator and world-member routing rules, the shared-world reuse boundary, and the exact machine-readable object shapes projected into `substrate agent list --json` and `substrate agent status --json`. +This file is the single authoritative schema inventory for ADR-0044 session semantics inside this feature pack. It defines the backend capability descriptor, the hub-owned session participant record, the lifecycle state machine, the host-orchestrator and world-member routing rules, the shared-world reuse boundary, and the exact machine-readable object shapes projected into `substrate agent list --json` and `substrate agent status --json`. Authoritative inputs: - `docs/project_management/adrs/draft/ADR-0044-agent-hub-core-successor-identity-tuple-compatible.md` @@ -17,12 +17,16 @@ Authoritative inputs: - This file is authoritative for: - the canonical backend capability descriptor consumed by the hub - - the canonical hub-owned session-handle schema + - the canonical hub-owned session participant schema - the lifecycle state names and legal transitions for agent sessions - the exact host-orchestrator eligibility rule - the exact world-scoped member routing, world reuse, and world restart invalidation rules - the exact machine-readable status objects reused by `substrate agent list --json` and `substrate agent status --json` - the exact structured event envelope reused between agent backends and the hub +- Live runtime authority is exactly `~/.substrate/run/agent-hub/sessions/*.json` plus `~/.substrate/run/agent-hub/participants/*.json`. +- `~/.substrate/run/agent-hub/handles/*.json` is compatibility-only input and never outranks the canonical participant path when both exist. +- Invalidated participant tombstones in `participants/*.json` beat stale trace fallback for live-state selection on the same `(orchestration_session_id, agent_id, execution.scope)` tuple. +- If a replacement participant for that tuple has not been persisted yet, the live status surface omits the tuple instead of projecting stale liveness from trace. - `contract.md` remains authoritative for operator-facing command behavior, exit codes, and human-readable render rules. - `policy-spec.md` remains authoritative for ordered deny evaluation and fail-closed control-plane outcomes. - `telemetry-spec.md` remains authoritative for trace placement, nested-record publication, and redaction rules after the hub projects protocol objects into canonical trace. @@ -85,15 +89,15 @@ Projection rule: - `substrate agent list --json` projects each effective descriptor into the `agents[*]` shape defined in `contract.md`. - The list surface is a projection. It does not widen, rename, or reinterpret the underlying descriptor fields defined here. -## Session-handle schema +## Session participant schema -The hub owns the session handle. Backends do not invent or persist an alternate handle format. +The hub owns the session participant record. Backends do not invent or persist an alternate child-record format. -### `AgentSessionHandleV1` +### `AgentSessionParticipantV1` ```json { - "session_handle_id": "ash_001", + "participant_id": "ash_001", "orchestration_session_id": "sess_001", "agent_id": "codex", "backend_id": "cli:codex", @@ -107,24 +111,27 @@ The hub owns the session handle. Backends do not invent or persist an alternate "last_transition_at": "2026-04-24T18:30:00Z", "world_id": "world-17", "world_generation": 3, - "parent_session_handle_id": null, - "resumed_from_session_handle_id": null + "parent_participant_id": null, + "resumed_from_participant_id": null, + "orchestrator_participant_id": "ash_orch_001" } ``` Field rules: -- `session_handle_id` is hub-assigned and unique within the repository trace domain. +- `participant_id` is hub-assigned and unique within the repository trace domain. - `orchestration_session_id` groups the orchestrator session plus every member session opened under the same control-plane session. -- `agent_id`, `backend_id`, `protocol`, and `execution.scope` are copied from the canonical capability descriptor and never drift during the life of one handle. +- `agent_id`, `backend_id`, `protocol`, and `execution.scope` are copied from the canonical capability descriptor and never drift during the life of one participant record. - `role` is exactly `orchestrator` or `member`. - `state` is one of the lifecycle state names defined below. -- `opened_at` is the first successful allocation time for this handle. +- `opened_at` is the first successful allocation time for this participant record. - `last_transition_at` is updated on every state transition. - `world_id` and `world_generation` are both present or both absent. - `world_id` and `world_generation` are required when `execution.scope=world`. - `world_id` and `world_generation` are omitted when `execution.scope=host`. -- `parent_session_handle_id` is non-null only for a forked child handle. -- `resumed_from_session_handle_id` is non-null only when a new handle replaced an invalidated world-scoped handle after a hub-driven restart. +- `parent_participant_id` is non-null only for a forked or explicitly derived child participant. +- `resumed_from_participant_id` is the persisted lineage field name and is non-null only when a new participant replaced an invalidated participant after a hub-driven restart. +- `orchestrator_participant_id` is omitted on the orchestrator participant and required on member participants. +- The parent orchestration-session record keeps its serialized field name `active_session_handle_id` during `PLAN-02`, but its value now points at the active `participant_id`. ## Lifecycle state machine @@ -133,19 +140,19 @@ Field rules: - `allocating` - The hub accepted the request and is waiting for backend or world allocation to complete. - `ready` - - The handle is live and ready to accept work. No task is currently executing. + - The participant is live and ready to accept work. No task is currently executing. - `running` - - The handle is actively executing or streaming work. + - The participant is actively executing or streaming work. - `restarting` - - The hub is replacing a world-scoped handle because the shared world restarted. + - The hub is replacing a world-scoped participant because the shared world restarted. - `stopping` - The hub issued a stop and is waiting for terminal confirmation. - `stopped` - Terminal state after a successful stop. - `failed` - - Terminal state after an execution or lifecycle failure that leaves the handle unusable. + - Terminal state after an execution or lifecycle failure that leaves the participant unusable. - `invalidated` - - Terminal state after world restart or orchestrator teardown made the handle unusable without an explicit stop. + - Terminal state after world restart or orchestrator teardown made the participant unusable without an explicit stop. ### Legal transitions @@ -158,7 +165,7 @@ Field rules: | `running` | `ready` | work dispatch completes successfully | | `ready` | `restarting` | shared world drift requires restart | | `running` | `restarting` | shared world drift requires restart after the in-flight unit ends or is cancelled | -| `restarting` | `invalidated` | old world-scoped handle is retired | +| `restarting` | `invalidated` | old world-scoped participant is retired | | `restarting` | `failed` | restart attempt fails closed | | `ready` | `stopping` | `stop_session` begins | | `running` | `stopping` | `stop_session` begins while work is active | @@ -167,12 +174,12 @@ Field rules: | `running` | `failed` | backend execution fails irrecoverably | Non-negotiable transition rules: -- A world restart invalidates every live world-scoped member handle tied to the previous `world_id`. -- A replacement handle created after restart MUST use a new `session_handle_id`. -- The replacement handle MUST copy `orchestration_session_id`, `agent_id`, `backend_id`, `role`, and `protocol`. -- The replacement handle MUST publish `resumed_from_session_handle_id=`. -- The replacement handle MUST publish the new `world_id` plus `world_generation = previous_world_generation + 1`. -- Host-scoped handles never transition through `restarting`. +- A world restart invalidates every live world-scoped member participant tied to the previous `world_id`. +- A replacement participant created after restart MUST use a new `participant_id`. +- The replacement participant MUST copy `orchestration_session_id`, `agent_id`, `backend_id`, `role`, and `protocol`. +- The replacement participant MUST publish `resumed_from_participant_id=`. +- The replacement participant MUST publish the new `world_id` plus `world_generation = previous_world_generation + 1`. +- Host-scoped participants never transition through `restarting`. ## Host-orchestrator eligibility @@ -208,8 +215,8 @@ Failure rules: ## Shared-world reuse and restart contract -- The v1 default is exactly one shared `world_id` per `orchestration_session_id` for all world-scoped member handles. -- Host-scoped orchestrator handles do not join that shared world and never publish `world_id` or `world_generation`. +- The v1 default is exactly one shared `world_id` per `orchestration_session_id` for all world-scoped member participants. +- Host-scoped orchestrator participants do not join that shared world and never publish `world_id` or `world_generation`. - `world_generation` starts at `0` when the first shared world is allocated for an orchestration session. - The shared world is reused while these inputs remain unchanged: - workspace root @@ -221,11 +228,11 @@ Failure rules: - When `agents.hub.world_restart.on_drift=auto_restart`: - the hub MUST allocate a new shared world - the hub MUST increment `world_generation` by exactly `1` - - the hub MUST invalidate every live world-scoped handle bound to the previous world - - the hub MUST create replacement handles before more work is dispatched + - the hub MUST invalidate every live world-scoped participant bound to the previous world + - the hub MUST create replacement participants before more work is dispatched - When `agents.hub.world_restart.on_drift=fail_closed`: - the hub MUST NOT restart implicitly - - the hub MUST leave the prior world-scoped handles in `invalidated` or `failed` + - the hub MUST leave the prior world-scoped participants in `invalidated` or `failed` - the hub MUST require an explicit restart path before more world-scoped work begins ## Status exchange objects @@ -268,6 +275,7 @@ This is the exact object shape for each pure-agent entry in `substrate agent sta ```json { + "participant_id": "ash_001", "orchestration_session_id": "sess_001", "agent_id": "codex", "backend_id": "cli:codex", @@ -285,11 +293,12 @@ This is the exact object shape for each pure-agent entry in `substrate agent sta ``` Field rules: +- `participant_id` is the canonical child identity for status, toolbox, and participant-aware trace correlation. - `client` is exactly the emitting session `agent_id`. - `router` is exactly `agent_hub`. - `protocol` is exactly `uaa.agent.session`. - `role` is `orchestrator` or `member`. -- `last_event_at` is the latest accepted structured event timestamp for that handle. +- `last_event_at` is the latest accepted structured event timestamp for that participant. - `world_id` and `world_generation` are both required when `execution.scope=world`. - `world_id` and `world_generation` are both omitted when `execution.scope=host`. - `provider` and `auth_authority` never appear on `AgentSessionStatusV1`. @@ -302,6 +311,7 @@ This is the exact object shape for each nested record in `substrate agent status { "parent": { "orchestration_session_id": "sess_001", + "participant_id": "ash_001", "agent_id": "codex" }, "run_id": "run_nested_001", @@ -315,7 +325,7 @@ This is the exact object shape for each nested record in `substrate agent status ``` Field rules: -- `parent.orchestration_session_id` and `parent.agent_id` identify the pure-agent session that triggered the nested request. +- `parent.orchestration_session_id`, `parent.participant_id`, and `parent.agent_id` identify the pure-agent participant that triggered the nested request. - `run_id` is the nested request correlation id and is required on every nested row. - `backend_id` remains the parent agent backend id. - `client` remains the parent agent id. @@ -347,7 +357,7 @@ Field rules: - `role_filter` is `null`, `orchestrator`, or `member`. - `sessions[*]` uses `AgentSessionStatusV1` exactly. - `nested_llm_records[*]` uses `NestedLlmStatusRecordV1` exactly. -- `sessions` sort by `orchestration_session_id`, then `agent_id`, ascending byte order. +- `sessions` sort by `orchestration_session_id`, then `participant_id`, then `agent_id`, ascending byte order. - `nested_llm_records` sort by `parent.orchestration_session_id`, then `parent.agent_id`, then `run_id` ascending byte order. ## Structured event exchange @@ -360,12 +370,13 @@ Required envelope fields for pure-agent session events: - `data` - `agent_id` - `orchestration_session_id` +- `participant_id` - `run_id` Required or conditional fields: - `backend_id` is required for v1 agent-hub session events because the emitting backend is always known. - `role` is required on orchestrator-emitted control-plane alerts and optional on member progress events. -- `world_id` is required when the emitting handle uses `execution.scope=world`. +- `world_id` is required when the emitting participant uses `execution.scope=world`. - `identity_tuple` is required on pure-agent session events and MUST be: - `client=` - `router=agent_hub` @@ -383,22 +394,22 @@ Event-kind rules: Ordering rules: - The hub does not promise one global total order across all agents. -- For a single `session_handle_id`, events are processed in emission order. -- A `registered` event, if emitted, precedes every later event for the same handle. +- For a single `participant_id`, events are processed in emission order. +- A `registered` event, if emitted, precedes every later event for the same participant. - A terminal `task_end` or `alert` that closes an in-flight unit precedes any transition back to `ready`. Retry rules: - Capability discovery is retryable and side-effect free. -- `start_session` is retryable only while no terminal handle exists for the same `(orchestration_session_id, agent_id, role)` tuple. -- `stop_session` is retryable and returns the same terminal state when the handle is already `stopped`, `failed`, or `invalidated`. -- `fork_session` is intentionally non-idempotent. Each successful fork creates a new `session_handle_id`. +- `start_session` is retryable only while no terminal participant exists for the same requested participant lineage tuple. +- `stop_session` is retryable and returns the same terminal state when the participant is already `stopped`, `failed`, or `invalidated`. +- `fork_session` is intentionally non-idempotent. Each successful fork creates a new `participant_id`. Idempotency rules: -- Repeating `start_session` for a live `(orchestration_session_id, agent_id, role)` tuple returns the existing live handle instead of creating a second live handle. -- Repeating `resume_session` for the same live `session_handle_id` returns that same handle. -- Repeating `stop_session` for a terminal handle returns the same terminal handle and does not create new events other than an at-most-once confirmation. +- Repeating `start_session` for the same live requested participant lineage tuple returns the existing live participant instead of creating a second live participant. +- Repeating `resume_session` for the same live `participant_id` returns that same participant. +- Repeating `stop_session` for a terminal participant returns the same terminal participant and does not create new events other than an at-most-once confirmation. Timeout rules: - Session allocation timeout is evaluated in `allocating`. - Dispatch timeout is evaluated in `running`. -- A timeout transitions the handle to `failed` unless the world restart path first moved it to `restarting`. +- A timeout transitions the participant to `failed` unless the world restart path first moved it to `restarting`. diff --git a/docs/project_management/packs/draft/agent-hub-core-successor-identity-tuple-compatible/contract.md b/docs/project_management/packs/draft/agent-hub-core-successor-identity-tuple-compatible/contract.md index 510644ba7..5ee3549cb 100644 --- a/docs/project_management/packs/draft/agent-hub-core-successor-identity-tuple-compatible/contract.md +++ b/docs/project_management/packs/draft/agent-hub-core-successor-identity-tuple-compatible/contract.md @@ -24,7 +24,7 @@ Authoritative inputs: - `docs/project_management/packs/implemented/llm_and_agent_config_policy_surface/contract.md` remains authoritative for config and policy file families, precedence, and the deny-by-default allowlist posture. - `docs/project_management/packs/implemented/llm_and_agent_config_policy_surface/SCHEMA.md` remains authoritative for the existing key families `agents.hub.orchestrator_agent_id`, `agents.hub.world_restart.on_drift`, `agents.allowed_backends`, and `llm.allowed_backends`. - `docs/project_management/packs/draft/llm-and-agent-identity-tuple-and-deployment-posture/contract.md` remains authoritative for the normalized meanings of `client`, `router`, `provider`, `auth_authority`, and `protocol`. -- The future `agent-hub-session-protocol-spec.md` remains authoritative for capability-descriptor and session-handle schemas. `C-01` only fixes the command-surface projection of those schemas. +- The future `agent-hub-session-protocol-spec.md` remains authoritative for capability-descriptor and session participant schemas. `C-01` only fixes the command-surface projection of those schemas. - The future `policy-spec.md` remains authoritative for ordered deny evaluation. - The future `telemetry-spec.md` remains authoritative for structured event and trace field placement. - If any later feature-local document conflicts with `C-01` on operator command behavior, `C-01` wins. @@ -161,8 +161,8 @@ JSON rules: #### Behavior -- `substrate agent status` reports the live successor view for the current process. -- It returns the selected orchestrator identity plus every active pure-agent session row that survives the requested filters. +- `substrate agent status` reports the selected successor status surface for the current process. +- It returns the selected orchestrator identity plus every selected pure-agent session row that survives the requested filters after live-runtime-first selection and trace fallback for gaps. - It renders nested gateway-backed LLM activity as separate correlated rows. #### Human-readable sections @@ -175,19 +175,29 @@ Human-readable status output renders: Pure-agent session render rules: - Each session row renders these fields in this order: 1. `orchestration_session_id` - 2. `agent_id` - 3. `backend_id` - 4. `client` - 5. `router` - 6. `protocol` - 7. `execution.scope` - 8. `role` - 9. `last_event_at` - 10. `world_id` - 11. `world_generation` + 2. `participant_id` + 3. `agent_id` + 4. `source_kind` + 5. `backend_id` + 6. `client` + 7. `router` + 8. `protocol` + 9. `execution.scope` + 10. `role` + 11. `posture` + 12. `attached_participant_id` + 13. `pending_inbox_count` + 14. `last_event_at` + 15. `world_id` + 16. `world_generation` - `client` renders exactly as the executing session's `agent_id`. - `router` renders `agent_hub`. - `protocol` renders `uaa.agent.session`. +- `participant_id` renders the canonical child identity for the live participant row. +- `source_kind` renders `live_runtime` or `trace_fallback`. +- `posture`, `attached_participant_id`, and `pending_inbox_count` render authoritative live session-root values only on `source_kind=live_runtime` pure-agent rows. +- Human-readable `source_kind=trace_fallback` rows render `posture=`, `attached_participant_id=`, and `pending_inbox_count=`. +- Human-readable `source_kind=live_runtime` rows render `attached_participant_id=` when the authoritative parent session has no current attachment. - `world_id` and `world_generation` render only when `execution.scope=world`. - Host-scoped rows omit `world_id` and `world_generation`. - Pure-agent session rows omit `provider` and `auth_authority`. @@ -195,14 +205,15 @@ Pure-agent session render rules: Nested LLM render rules: - Each nested row renders: 1. `parent.orchestration_session_id` - 2. `parent.agent_id` - 3. `run_id` - 4. `backend_id` - 5. `client` - 6. `router` - 7. `provider` - 8. `auth_authority` - 9. `protocol` + 2. `parent.participant_id` + 3. `parent.agent_id` + 4. `run_id` + 5. `backend_id` + 6. `client` + 7. `router` + 8. `provider` + 9. `auth_authority` + 10. `protocol` - `router` renders `substrate_gateway`. - `client` renders the parent agent session's `agent_id`. - `run_id` renders the nested request correlation id. @@ -220,7 +231,9 @@ Nested LLM render rules: "sessions": [ { "orchestration_session_id": "sess_001", + "participant_id": "ash_001", "agent_id": "codex", + "source_kind": "live_runtime", "backend_id": "cli:codex", "client": "codex", "router": "agent_hub", @@ -229,6 +242,9 @@ Nested LLM render rules: "scope": "world" }, "role": null, + "posture": "active_attached", + "attached_participant_id": "ash_orch_001", + "pending_inbox_count": 0, "last_event_at": "2026-04-24T18:30:00Z", "world_id": "world-17", "world_generation": 3 @@ -238,6 +254,7 @@ Nested LLM render rules: { "parent": { "orchestration_session_id": "sess_001", + "participant_id": "ash_001", "agent_id": "codex" }, "run_id": "run_nested_001", @@ -254,11 +271,16 @@ Nested LLM render rules: JSON rules: - `orchestrator_agent_id` is the selected inventory id, not the derived `backend_id`. +- Each pure-agent session object includes `participant_id` as the canonical child identity. +- Each pure-agent session object includes `source_kind`. - Each pure-agent session object includes `client`, `router`, and `protocol`. - Each pure-agent session object omits `provider` and `auth_authority`. +- Each pure-agent `source_kind=live_runtime` session object includes authoritative `posture`, `attached_participant_id`, and `pending_inbox_count` from the session-root parent record. +- Each pure-agent `source_kind=trace_fallback` session object renders `posture`, `attached_participant_id`, and `pending_inbox_count` as explicit `null`. - `world_id` and `world_generation` are both present or both absent. - `world_generation` is an integer that starts at `0` for a fresh world allocation and increments by `1` on each hub-driven restart of that orchestration session's world. - Each nested record includes `run_id` as the nested request correlation id. +- Each nested record includes `parent.participant_id`. - Each nested record includes `provider` and `auth_authority`. - Each nested record omits `world_id` and `world_generation`. diff --git a/docs/project_management/system/schemas/proving_run_closeout.schema.json b/docs/project_management/system/schemas/proving_run_closeout.schema.json new file mode 100644 index 000000000..8e221f366 --- /dev/null +++ b/docs/project_management/system/schemas/proving_run_closeout.schema.json @@ -0,0 +1,208 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://substrate.dev/docs/project_management/system/schemas/proving_run_closeout.schema.json", + "title": "Proving Run Closeout", + "type": "object", + "additionalProperties": false, + "required": [ + "schema_version", + "artifact_type", + "prepared_at", + "mode", + "preparation", + "machine_owned", + "human_owned", + "handoff" + ], + "properties": { + "schema_version": { + "const": 1 + }, + "artifact_type": { + "const": "proving_run_closeout" + }, + "prepared_at": { + "type": "string" + }, + "mode": { + "const": "create" + }, + "preparation": { + "type": "object", + "additionalProperties": false, + "required": [ + "script", + "transaction_id", + "inputs" + ], + "properties": { + "script": { + "type": "string" + }, + "transaction_id": { + "type": "string" + }, + "inputs": { + "type": "array", + "minItems": 1, + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "kind", + "path", + "sha256" + ], + "properties": { + "kind": { + "enum": [ + "lifecycle_facts", + "human_inputs" + ] + }, + "path": { + "type": "string" + }, + "sha256": { + "type": "string", + "pattern": "^[a-f0-9]{64}$" + } + } + } + } + } + }, + "machine_owned": { + "type": "object", + "additionalProperties": false, + "required": [ + "run_id", + "lifecycle", + "publication", + "facts_snapshot" + ], + "properties": { + "run_id": { + "type": "string" + }, + "lifecycle": { + "type": "object", + "additionalProperties": false, + "required": [ + "source_state", + "target_state" + ], + "properties": { + "source_state": { + "const": "published_baseline" + }, + "target_state": { + "const": "closed_baseline" + } + } + }, + "publication": { + "type": "object", + "required": [ + "status", + "published_at", + "artifact_path" + ], + "properties": { + "status": { + "const": "green" + }, + "published_at": { + "type": "string" + }, + "artifact_path": { + "type": "string" + }, + "evidence_refs": { + "type": "array", + "items": { + "type": "string" + } + } + }, + "additionalProperties": true + }, + "facts_snapshot": { + "type": "object", + "required": [ + "schema_version", + "run_id", + "mode", + "lifecycle", + "publication" + ], + "additionalProperties": true + } + } + }, + "human_owned": { + "type": "object", + "additionalProperties": false, + "required": [ + "residual_friction", + "manual_edits", + "operator_notes", + "follow_ups" + ], + "properties": { + "residual_friction": { + "type": [ + "string", + "null" + ] + }, + "manual_edits": { + "type": [ + "array", + "null" + ], + "items": { + "type": "string" + } + }, + "operator_notes": { + "type": [ + "string", + "null" + ] + }, + "follow_ups": { + "type": "array", + "items": { + "type": "string" + } + } + } + }, + "handoff": { + "type": "object", + "additionalProperties": false, + "required": [ + "status", + "required_human_fields" + ], + "properties": { + "status": { + "enum": [ + "awaiting_human_inputs", + "ready_to_close" + ] + }, + "required_human_fields": { + "type": "array", + "items": { + "enum": [ + "human_owned.residual_friction", + "human_owned.manual_edits" + ] + } + } + } + } + } +} diff --git a/docs/project_management/system/scripts/execution/prepare_proving_run_closeout.py b/docs/project_management/system/scripts/execution/prepare_proving_run_closeout.py new file mode 100644 index 000000000..dfe49ab12 --- /dev/null +++ b/docs/project_management/system/scripts/execution/prepare_proving_run_closeout.py @@ -0,0 +1,267 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import argparse +import hashlib +import json +import os +import sys +import tempfile +import uuid +from copy import deepcopy +from datetime import datetime, timezone +from pathlib import Path +from typing import Any + + +SCRIPT_PATH = "docs/project_management/system/scripts/execution/prepare_proving_run_closeout.py" +REQUIRED_SOURCE_STATE = "published_baseline" +TARGET_STATE = "closed_baseline" +REQUIRED_HUMAN_FIELDS = ("residual_friction", "manual_edits") +ALLOWED_HUMAN_FIELDS = set(REQUIRED_HUMAN_FIELDS) | {"operator_notes", "follow_ups"} + + +class PreparationError(RuntimeError): + pass + + +def utc_now() -> str: + return datetime.now(timezone.utc).replace(microsecond=0).isoformat().replace("+00:00", "Z") + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser( + description=( + "Prepare a repo-owned proving-run closeout draft from machine-known lifecycle/publication facts " + "plus optional human inputs." + ) + ) + parser.add_argument("--facts", required=True, help="Path to the machine-known lifecycle/publication facts JSON.") + parser.add_argument( + "--human-inputs", + help=( + "Optional path to human-owned closeout inputs JSON. Allowed keys: residual_friction, manual_edits, " + "operator_notes, follow_ups." + ), + ) + parser.add_argument( + "--output", + default="proving-run-closeout.json", + help="Output path for the prepared closeout JSON. Defaults to proving-run-closeout.json.", + ) + parser.add_argument("--force", action="store_true", help="Overwrite the output path if it already exists.") + return parser.parse_args() + + +def read_json(path: Path) -> dict[str, Any]: + try: + with path.open("r", encoding="utf-8") as handle: + payload = json.load(handle) + except FileNotFoundError as exc: + raise PreparationError(f"missing required input file: {path}") from exc + except json.JSONDecodeError as exc: + raise PreparationError(f"invalid JSON in {path}: {exc}") from exc + if not isinstance(payload, dict): + raise PreparationError(f"expected top-level JSON object in {path}") + return payload + + +def sha256_file(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as handle: + for chunk in iter(lambda: handle.read(1024 * 1024), b""): + digest.update(chunk) + return digest.hexdigest() + + +def validate_facts(facts: dict[str, Any], source_path: Path) -> dict[str, Any]: + if facts.get("schema_version") != 1: + raise PreparationError( + f"{source_path}: facts.schema_version must be 1 for prepare_proving_run_closeout.py" + ) + if facts.get("mode") != "create": + raise PreparationError(f"{source_path}: facts.mode must be 'create'") + + run_id = facts.get("run_id") + if not isinstance(run_id, str) or not run_id.strip(): + raise PreparationError(f"{source_path}: facts.run_id must be a non-empty string") + + lifecycle = facts.get("lifecycle") + if not isinstance(lifecycle, dict): + raise PreparationError(f"{source_path}: facts.lifecycle must be an object") + current_state = lifecycle.get("current_state") + if current_state != REQUIRED_SOURCE_STATE: + raise PreparationError( + f"{source_path}: facts.lifecycle.current_state must be '{REQUIRED_SOURCE_STATE}', got {current_state!r}" + ) + + publication = facts.get("publication") + if not isinstance(publication, dict): + raise PreparationError(f"{source_path}: facts.publication must be an object") + status = publication.get("status") + if status != "green": + raise PreparationError(f"{source_path}: facts.publication.status must be 'green', got {status!r}") + + published_at = publication.get("published_at") + if not isinstance(published_at, str) or not published_at.strip(): + raise PreparationError(f"{source_path}: facts.publication.published_at must be a non-empty string") + + artifact_path = publication.get("artifact_path") + if not isinstance(artifact_path, str) or not artifact_path.strip(): + raise PreparationError(f"{source_path}: facts.publication.artifact_path must be a non-empty string") + + evidence_refs = publication.get("evidence_refs") + if evidence_refs is not None: + if not isinstance(evidence_refs, list) or not all(isinstance(item, str) for item in evidence_refs): + raise PreparationError(f"{source_path}: facts.publication.evidence_refs must be an array of strings") + + return facts + + +def validate_human_inputs(human_inputs: dict[str, Any], source_path: Path) -> dict[str, Any]: + unknown = sorted(set(human_inputs) - ALLOWED_HUMAN_FIELDS) + if unknown: + raise PreparationError( + f"{source_path}: unexpected human-owned keys: {', '.join(unknown)}" + ) + + if "residual_friction" in human_inputs: + value = human_inputs["residual_friction"] + if value is not None and not isinstance(value, str): + raise PreparationError(f"{source_path}: residual_friction must be a string or null") + + if "manual_edits" in human_inputs: + value = human_inputs["manual_edits"] + if value is not None: + if not isinstance(value, list) or not all(isinstance(item, str) for item in value): + raise PreparationError(f"{source_path}: manual_edits must be an array of strings or null") + + if "operator_notes" in human_inputs: + value = human_inputs["operator_notes"] + if value is not None and not isinstance(value, str): + raise PreparationError(f"{source_path}: operator_notes must be a string or null") + + if "follow_ups" in human_inputs: + value = human_inputs["follow_ups"] + if not isinstance(value, list) or not all(isinstance(item, str) for item in value): + raise PreparationError(f"{source_path}: follow_ups must be an array of strings") + + return human_inputs + + +def build_human_owned_section(human_inputs: dict[str, Any] | None) -> tuple[dict[str, Any], list[str]]: + human_owned: dict[str, Any] = { + "residual_friction": None, + "manual_edits": None, + "operator_notes": None, + "follow_ups": [], + } + if human_inputs: + for key, value in human_inputs.items(): + human_owned[key] = deepcopy(value) + + missing: list[str] = [] + for key in REQUIRED_HUMAN_FIELDS: + if human_owned.get(key) is None: + missing.append(f"human_owned.{key}") + + return human_owned, missing + + +def build_closeout( + facts: dict[str, Any], + facts_path: Path, + human_inputs: dict[str, Any] | None, + human_inputs_path: Path | None, +) -> dict[str, Any]: + prepared_at = utc_now() + transaction_id = str(uuid.uuid4()) + human_owned, missing_fields = build_human_owned_section(human_inputs) + + preparation_inputs: list[dict[str, str]] = [ + { + "kind": "lifecycle_facts", + "path": str(facts_path), + "sha256": sha256_file(facts_path), + } + ] + if human_inputs_path is not None: + preparation_inputs.append( + { + "kind": "human_inputs", + "path": str(human_inputs_path), + "sha256": sha256_file(human_inputs_path), + } + ) + + return { + "schema_version": 1, + "artifact_type": "proving_run_closeout", + "prepared_at": prepared_at, + "mode": "create", + "preparation": { + "script": SCRIPT_PATH, + "transaction_id": transaction_id, + "inputs": preparation_inputs, + }, + "machine_owned": { + "run_id": facts["run_id"], + "lifecycle": { + "source_state": facts["lifecycle"]["current_state"], + "target_state": TARGET_STATE, + }, + "publication": deepcopy(facts["publication"]), + "facts_snapshot": deepcopy(facts), + }, + "human_owned": human_owned, + "handoff": { + "status": "awaiting_human_inputs" if missing_fields else "ready_to_close", + "required_human_fields": missing_fields, + }, + } + + +def atomic_write_json(path: Path, payload: dict[str, Any], force: bool) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + if path.exists() and not force: + raise PreparationError(f"refusing to overwrite existing output without --force: {path}") + + rendered = json.dumps(payload, indent=2, sort_keys=True) + "\n" + fd, temp_path_raw = tempfile.mkstemp(prefix=f".{path.name}.", suffix=".tmp", dir=path.parent) + temp_path = Path(temp_path_raw) + try: + with os.fdopen(fd, "w", encoding="utf-8") as handle: + handle.write(rendered) + handle.flush() + os.fsync(handle.fileno()) + temp_path.replace(path) + finally: + try: + temp_path.unlink() + except FileNotFoundError: + pass + + +def main() -> int: + args = parse_args() + facts_path = Path(args.facts).resolve() + output_path = Path(args.output).resolve() + human_inputs_path = Path(args.human_inputs).resolve() if args.human_inputs else None + + try: + facts = validate_facts(read_json(facts_path), facts_path) + human_inputs = None + if human_inputs_path is not None: + human_inputs = validate_human_inputs(read_json(human_inputs_path), human_inputs_path) + closeout = build_closeout(facts, facts_path, human_inputs, human_inputs_path) + atomic_write_json(output_path, closeout, force=args.force) + except PreparationError as exc: + print(f"ERROR: {exc}", file=sys.stderr) + return 2 + + print(output_path) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/docs/project_management/system/scripts/execution/tests/test_prepare_proving_run_closeout.py b/docs/project_management/system/scripts/execution/tests/test_prepare_proving_run_closeout.py new file mode 100644 index 000000000..7dd13e5f5 --- /dev/null +++ b/docs/project_management/system/scripts/execution/tests/test_prepare_proving_run_closeout.py @@ -0,0 +1,126 @@ +from __future__ import annotations + +import json +import subprocess +import sys +import tempfile +import unittest +from pathlib import Path + + +class PrepareProvingRunCloseoutTests(unittest.TestCase): + @classmethod + def setUpClass(cls) -> None: + cls.repo_root = Path(__file__).resolve().parents[6] + cls.script = cls.repo_root / "docs" / "project_management" / "system" / "scripts" / "execution" / "prepare_proving_run_closeout.py" + if not cls.script.is_file(): + raise unittest.SkipTest("prepare_proving_run_closeout.py not found at expected canonical path") + + def setUp(self) -> None: + self.temp_dir = Path(tempfile.mkdtemp(prefix="prepare_proving_run_closeout_")) + + def _write_json(self, relative_path: str, payload: dict) -> Path: + path = self.temp_dir / relative_path + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(json.dumps(payload, indent=2) + "\n", encoding="utf-8") + return path + + def _run(self, *args: str) -> subprocess.CompletedProcess[str]: + return subprocess.run( + [sys.executable, str(self.script), *args], + cwd=self.repo_root, + text=True, + capture_output=True, + check=False, + ) + + def _baseline_facts(self) -> dict: + return { + "schema_version": 1, + "run_id": "create-run-001", + "mode": "create", + "lifecycle": { + "current_state": "published_baseline", + }, + "publication": { + "status": "green", + "published_at": "2026-05-04T12:00:00Z", + "artifact_path": "artifacts/publication.json", + "evidence_refs": [ + "artifacts/publication.json", + "artifacts/summary.md", + ], + }, + } + + def test_generates_scaffold_when_human_inputs_are_missing(self) -> None: + facts_path = self._write_json("facts.json", self._baseline_facts()) + output_path = self.temp_dir / "proving-run-closeout.json" + + result = self._run("--facts", str(facts_path), "--output", str(output_path)) + + self.assertEqual(result.returncode, 0, result.stderr) + payload = json.loads(output_path.read_text(encoding="utf-8")) + self.assertEqual(payload["machine_owned"]["run_id"], "create-run-001") + self.assertEqual(payload["machine_owned"]["lifecycle"]["source_state"], "published_baseline") + self.assertEqual(payload["machine_owned"]["lifecycle"]["target_state"], "closed_baseline") + self.assertEqual(payload["handoff"]["status"], "awaiting_human_inputs") + self.assertEqual( + payload["handoff"]["required_human_fields"], + ["human_owned.residual_friction", "human_owned.manual_edits"], + ) + self.assertIsNone(payload["human_owned"]["residual_friction"]) + self.assertIsNone(payload["human_owned"]["manual_edits"]) + self.assertEqual(len(payload["preparation"]["inputs"]), 1) + + def test_marks_closeout_ready_when_required_human_inputs_are_present(self) -> None: + facts_path = self._write_json("facts.json", self._baseline_facts()) + human_inputs_path = self._write_json( + "human-inputs.json", + { + "residual_friction": "low: one manual review pass", + "manual_edits": [], + "operator_notes": "publication diff already reconciled", + }, + ) + output_path = self.temp_dir / "proving-run-closeout.json" + + result = self._run( + "--facts", + str(facts_path), + "--human-inputs", + str(human_inputs_path), + "--output", + str(output_path), + ) + + self.assertEqual(result.returncode, 0, result.stderr) + payload = json.loads(output_path.read_text(encoding="utf-8")) + self.assertEqual(payload["handoff"]["status"], "ready_to_close") + self.assertEqual(payload["handoff"]["required_human_fields"], []) + self.assertEqual(payload["human_owned"]["manual_edits"], []) + self.assertEqual(len(payload["preparation"]["inputs"]), 2) + + def test_rejects_non_published_baseline_source_state(self) -> None: + facts = self._baseline_facts() + facts["lifecycle"]["current_state"] = "draft_baseline" + facts_path = self._write_json("facts.json", facts) + + result = self._run("--facts", str(facts_path), "--output", str(self.temp_dir / "closeout.json")) + + self.assertEqual(result.returncode, 2) + self.assertIn("facts.lifecycle.current_state must be 'published_baseline'", result.stderr) + + def test_rejects_non_green_publication(self) -> None: + facts = self._baseline_facts() + facts["publication"]["status"] = "failed" + facts_path = self._write_json("facts.json", facts) + + result = self._run("--facts", str(facts_path), "--output", str(self.temp_dir / "closeout.json")) + + self.assertEqual(result.returncode, 2) + self.assertIn("facts.publication.status must be 'green'", result.stderr) + + +if __name__ == "__main__": + unittest.main() diff --git a/docs/project_management/system/standards/MANIFEST.yaml b/docs/project_management/system/standards/MANIFEST.yaml index 46feb32b4..d54ee61e7 100644 --- a/docs/project_management/system/standards/MANIFEST.yaml +++ b/docs/project_management/system/standards/MANIFEST.yaml @@ -41,6 +41,10 @@ standards: category: execution path: docs/project_management/system/standards/execution/SLICE_CLOSEOUT_GATE_STANDARD.md description: "Slice closeout gate standard." + - id: proving_run_closeout_preparation_standard + category: execution + path: docs/project_management/system/standards/execution/PROVING_RUN_CLOSEOUT_PREPARATION_STANDARD.md + description: "Repo-owned proving-run closeout preparation standard." - id: task_triads_and_feature_setup category: triad diff --git a/docs/project_management/system/standards/README.md b/docs/project_management/system/standards/README.md index cafda1d89..a619388fd 100644 --- a/docs/project_management/system/standards/README.md +++ b/docs/project_management/system/standards/README.md @@ -25,6 +25,7 @@ This directory contains **standards only** (normative rules). Prompt payloads li - `docs/project_management/system/standards/execution/EXECUTION_PREFLIGHT_GATE_STANDARD.md` - `docs/project_management/system/standards/execution/SLICE_CLOSEOUT_GATE_STANDARD.md` +- `docs/project_management/system/standards/execution/PROVING_RUN_CLOSEOUT_PREPARATION_STANDARD.md` ### Triad @@ -76,6 +77,7 @@ Templates: Schema: - Tasks schema: `docs/project_management/system/schemas/tasks.schema.json` +- Proving-run closeout schema: `docs/project_management/system/schemas/proving_run_closeout.schema.json` Machine-readable map: diff --git a/docs/project_management/system/standards/execution/PROVING_RUN_CLOSEOUT_PREPARATION_STANDARD.md b/docs/project_management/system/standards/execution/PROVING_RUN_CLOSEOUT_PREPARATION_STANDARD.md new file mode 100644 index 000000000..030806bd4 --- /dev/null +++ b/docs/project_management/system/standards/execution/PROVING_RUN_CLOSEOUT_PREPARATION_STANDARD.md @@ -0,0 +1,48 @@ +# Proving-Run Closeout Preparation Standard + +Goal: +- Eliminate freehand create-lane closeout authoring by generating a repo-owned `proving-run-closeout.json` draft from lifecycle and publication truth, then leaving only the minimum human-owned fields for completion. + +## Canonical lifecycle states +- `published_baseline` is the only legal source state for closeout preparation. +- `closed_baseline` is the terminal target state after closeout is completed. +- A create-mode proving run must not skip directly from green publication to `closed_baseline` without a prepared closeout artifact. + +## Source-of-truth inputs +- Machine-owned lifecycle/publication facts live in one JSON object with: + - `schema_version = 1` + - `mode = "create"` + - `run_id` + - `lifecycle.current_state = "published_baseline"` + - `publication.status = "green"` + - `publication.published_at` + - `publication.artifact_path` +- Optional human inputs may be supplied separately and merged during preparation. + +## Ownership split +- Machine-owned fields: + - lifecycle source state and target state + - publication status and publication timestamps + - publication artifact path and evidence refs + - the full input facts snapshot + - preparation transaction metadata and input file hashes +- Human-owned fields: + - `human_owned.residual_friction` + - `human_owned.manual_edits` + - `human_owned.operator_notes` + - `human_owned.follow_ups` + +## Preparation flow +- Run `docs/project_management/system/scripts/execution/prepare_proving_run_closeout.py`. +- The script must: + - fail closed unless publication is green and lifecycle state is `published_baseline` + - write `proving-run-closeout.json` atomically + - preserve machine-owned truth exactly + - scaffold missing human-owned fields as `null` or empty arrays + - record `handoff.status = "awaiting_human_inputs"` until the required human fields are present + - record `handoff.status = "ready_to_close"` only when `residual_friction` and `manual_edits` are explicitly supplied + +## Output contract +- Canonical schema: `docs/project_management/system/schemas/proving_run_closeout.schema.json` +- Canonical script: `docs/project_management/system/scripts/execution/prepare_proving_run_closeout.py` +- Default output filename: `proving-run-closeout.json` diff --git a/llm-last-mile/03-shared-world-ownership-linux-first.md b/llm-last-mile/03-shared-world-ownership-linux-first.md index 0e61f5974..62e6e4dca 100644 --- a/llm-last-mile/03-shared-world-ownership-linux-first.md +++ b/llm-last-mile/03-shared-world-ownership-linux-first.md @@ -1,439 +1,74 @@ # SOW: Shared-World Ownership Explicitness, Linux-First -## Objective +Status: historical draft. This file records what changed after merge so readers do not treat the abandoned shell-authoritative design as current behavior. -Make shared-world ownership explicit for agent-hub world-scoped execution on Linux by binding exactly one active `world_id` to exactly one `orchestration_session_id`, with monotonic `world_generation` tracking starting at `0` and incrementing only on hub-driven world replacement. +Authoritative current references: -This slice is a prerequisite for the ADR-0044 / successor agent-hub contract already reflected in: +- [PLAN-03](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-03.md) +- [docs/WORLD.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/WORLD.md) -- `docs/project_management/adrs/draft/ADR-0044-agent-hub-core-successor-identity-tuple-compatible.md` -- `docs/project_management/packs/draft/agent-hub-core-successor-identity-tuple-compatible/agent-hub-session-protocol-spec.md` -- `docs/project_management/packs/draft/agent-hub-core-successor-identity-tuple-compatible/telemetry-spec.md` -- `docs/project_management/packs/draft/agent-hub-core-successor-identity-tuple-compatible/platform-parity-spec.md` +## Historical Drift Correction -The implementation target is Linux first. macOS and Windows must accept the additive model surface introduced here, but they are explicitly constrained from inventing independent shared-world ownership behavior during this slice. +This draft originally proposed a shell-owned binding store under `~/.substrate/run/agent-hub/world-bindings/` with Linux `session.json` acting as a mirrored copy. -## Scope +That proposal is stale and was not the merged design. -In scope: +The merged Linux-first implementation instead makes `crates/world` authoritative: -- establish an explicit shared-world binding contract for Linux world-scoped member sessions -- persist the binding so world reuse is keyed by `orchestration_session_id`, not only `WorldSpec` compatibility -- track `world_generation` as authoritative runtime state -- thread the binding through shell runtime state, world backend state, and world-agent request handling -- make status/telemetry surfaces capable of reporting the bound `world_id` and `world_generation` +- persisted authority lives in `SessionWorldMetadata` in `crates/world/src/session.rs` +- `set_shared_binding_state()` is the only shared-binding mutation path +- `persist_metadata()` atomically writes owner-bearing metadata +- same-owner shared-world ensure and replace paths are serialized by a backend-local mutex in `crates/world/src/lib.rs` +- downstream surfaces may expose or accept only `binding_state=Active` -Out of scope: +## Final Linux-First Behavior -- redesigning the full agent-hub control plane -- implementing all drift-restart triggers end-to-end for every platform -- changing host-scoped orchestrator semantics -- broad reuse-policy changes for non-agent shell flows that do not opt into explicit shared-world ownership +### Ownership proof -## Current State and Blockers +Explicit shared-owner reuse is keyed by persisted owner metadata: -### 1. Linux world reuse is spec-compatible, not owner-compatible - -Current Linux reuse is driven by `WorldSpec` compatibility and on-disk `session.json` discovery: - -- `crates/world/src/lib.rs` -- `crates/world/src/session.rs` - -`LinuxLocalBackend::ensure_session()` currently reuses a world when `project_dir`, network flags, and allowed domains match. `SessionWorldMetadata` does not record `orchestration_session_id` or `world_generation`. That means the current backend can reuse the same world across unrelated orchestrations if they share the same `WorldSpec`. - -This is the primary blocker. The agent-hub contract requires one shared world per orchestration session, not one shared world per compatible spec. - -### 2. Shell runtime manifests already expose world fields, but they are not authoritative yet - -The shell-side live-session manifest model already carries optional world identity: - -- `crates/shell/src/execution/agent_runtime/session.rs` -- `crates/shell/src/execution/agent_runtime/state_store.rs` - -`AgentRuntimeSessionHandle` includes `world_id` and `world_generation`, and `substrate agent status` already expects world-scoped rows to have both: - -- `crates/shell/src/execution/agents_cmd.rs` - -However, there is no authoritative store that binds those fields to a Linux world allocation. The manifest can report these fields, but the backend currently has no matching ownership record to prove them. - -### 3. Current orchestration identity is too incidental for shared-world ownership - -`crates/shell/src/execution/agent_events.rs` currently mints a process-global `orchestration_session_id` via `OnceLock`. That is acceptable for lightweight event correlation, but it is not sufficient as the ownership anchor for a reusable shared world that must survive multiple commands and explicit restart/generation transitions. - -The shared-world binding must be driven by an explicit runtime/session owner, not by opportunistic event emission. - -### 4. World-agent request paths do not carry explicit shared-world ownership intent - -The relevant execution paths: - -- `crates/agent-api-types/src/lib.rs` -- `crates/world-agent/src/service.rs` -- `crates/world-agent/src/pty.rs` -- `crates/shell/src/execution/routing/dispatch/world_persistent_session.rs` - -Today, `ExecuteRequest` and the persistent-session `start_session` message carry policy, cwd, env, and world network data, but not an explicit shared-world owner binding. The world-agent therefore cannot distinguish: - -- generic world reuse for ordinary shell execution -- explicit agent-hub shared-world reuse for one orchestration session - -### 5. Placeholder world attribution exists in event paths - -`WorldAgentService` currently emits some event data with placeholder correlation behavior, including using `span_id` as `orchestration_session_id` in one execution-stream path: - -- `crates/world-agent/src/service.rs` - -That is not acceptable once shared-world ownership is explicit. The shared-world binding must be sourced from the orchestrator/session model, not reconstructed from span-level execution identifiers. - -### 6. Cross-platform bootstrap code can allocate worlds without agent-hub ownership semantics - -Platform bootstrapping currently calls `ensure_session()` for generic world readiness: - -- `crates/shell/src/execution/routing/world.rs` -- `crates/shell/src/execution/platform_world/mod.rs` -- `crates/shell/src/execution/platform_world/windows.rs` - -That startup path must not accidentally become the source of truth for agent-hub shared-world ownership. This slice must keep explicit shared-world ownership opt-in and scoped to world-scoped member execution. - -## Target Outcome - -After this slice: - -- Linux world-scoped member sessions have an explicit shared-world binding keyed by `orchestration_session_id`. -- Exactly one active `world_id` is bound to one `orchestration_session_id` at a time. -- `world_generation` is persisted and starts at `0` for first allocation. -- Reuse on Linux for agent-hub shared worlds is by exact owner match plus compatible world inputs, not by spec compatibility alone. -- The shell live-manifest and status surfaces read from data that the backend can prove. -- macOS and Windows accept additive schema/plumbing changes but do not introduce independent shared-world ownership logic in this slice. - -## Proposed Design - -### Authoritative Binding Model - -Introduce a Linux-first shared-world binding record owned by the shell runtime and mirrored into Linux world metadata. - -Authoritative shell-side binding path: - -- `~/.substrate/run/agent-hub/world-bindings/.json` -- implementation owner: new helper under `crates/shell/src/execution/agent_runtime/` - -Linux world-side mirrored metadata: - -- `/tmp/substrate-worlds//session.json` -- implementation owner: `crates/world/src/session.rs` - -Required binding fields: - -- `schema_version` -- `orchestration_session_id` -- `current_world_id` -- `world_generation` -- `workspace_root` -- `policy_snapshot_hash` -- `world_fs_mode` -- `isolate_network` -- `allowed_domains_hash` -- `state` -- `created_at` -- `last_bound_at` -- `last_restart_reason` - -Rules: - -- exactly one authoritative binding file exists per live `orchestration_session_id` -- exactly one active `current_world_id` exists in that binding at a time -- `world_generation` is monotonic per `orchestration_session_id` -- Linux world metadata must mirror the owning `orchestration_session_id` and `world_generation` -- a world with missing or conflicting ownership metadata is not reusable for agent-hub shared-world flows - -## Backend and Model Changes - -### 1. `world-api` contract becomes ownership-aware - -Primary files: - -- `crates/world-api/src/lib.rs` -- `crates/agent-api-types/src/lib.rs` - -Add additive fields so explicit shared-world ownership can be requested without overloading generic world reuse: - -- `WorldSpec` gains an optional shared-world ownership request, for example: - - `shared_owner.orchestration_session_id` - - `shared_owner.expected_generation` optional for restart/validation paths - - `shared_owner.mode = shared_orchestration_session` -- `WorldHandle` gains additive ownership metadata, at minimum: - - `generation: Option` - - `orchestration_session_id: Option` -- `ExecuteRequest` gains additive shared-world ownership fields for non-PTY paths -- persistent-session `start_session` gains the same additive ownership fields for PTY/session paths in `crates/world-agent/src/pty.rs` - -Constraint: - -- the new fields are optional so legacy/non-agent callers remain unchanged -- explicit shared-world behavior only activates when ownership fields are present - -### 2. Linux backend distinguishes generic reuse from shared-world reuse - -Primary files: - -- `crates/world/src/lib.rs` -- `crates/world/src/session.rs` - -Behavior change: - -- keep existing spec-compatible reuse for generic non-agent flows with no explicit owner -- add a separate exact-owner reuse path when `shared_owner.orchestration_session_id` is present - -Required Linux semantics: - -- first look up an existing world by bound `orchestration_session_id` -- verify ownership metadata, generation, and world-input compatibility -- reuse only when both ownership and compatible world inputs match -- create a new world when no valid binding exists -- never reuse a world already owned by a different `orchestration_session_id` - -### 3. Session metadata becomes ownership-bearing - -Primary file: - -- `crates/world/src/session.rs` - -Extend `SessionWorldMetadata` with: - -- `owner_mode` +- `owner_mode=shared_orchestration` - `orchestration_session_id` - `world_generation` - `binding_state` -Required changes: - -- `metadata_matches_spec()` must also validate owner identity when shared-world ownership is requested -- `recover_compatible_from_root()` must reject stale or cross-owned worlds for shared-world flows -- `persist_metadata()` must atomically write the new ownership fields - -## Shell Runtime and Persistence Changes - -### 1. Add a dedicated shared-world binding store - -Primary owner directory: - -- `crates/shell/src/execution/agent_runtime/` - -Suggested new surface: - -- `shared_world_store.rs` - -Responsibilities: - -- allocate the first binding for an `orchestration_session_id` -- read/update the authoritative `current_world_id` -- persist `world_generation` -- invalidate old bindings during restart -- expose a narrow API used by agent runtime/session code - -This store should use the same atomic-write posture already used in: - -- `crates/shell/src/execution/agent_runtime/state_store.rs` - -### 2. Make live session manifests authoritative consumers of the binding - -Primary files: - -- `crates/shell/src/execution/agent_runtime/session.rs` -- `crates/shell/src/execution/agent_runtime/state_store.rs` -- `crates/shell/src/execution/agents_cmd.rs` - -Required behavior: - -- world-scoped manifests must populate `world_id` and `world_generation` from the shared-world binding path, not from ad hoc environment data -- restart replacement handles must set: - - new `session_handle_id` - - same `orchestration_session_id` - - `resumed_from_session_handle_id = ` - - new `world_id` - - incremented `world_generation` -- host-scoped manifests continue to omit both fields - -## World-Agent and Routing Changes - -Primary files: - -- `crates/world-agent/src/service.rs` -- `crates/world-agent/src/pty.rs` -- `crates/shell/src/execution/routing/dispatch/world_ops.rs` -- `crates/shell/src/execution/routing/dispatch/world_persistent_session.rs` - -Required changes: - -- thread explicit shared-world ownership from shell dispatch into world-agent requests -- build `WorldSpec` for Linux shared-world flows with ownership fields attached -- return the bound `world_id` and generation-aware handle data to the shell runtime -- stop using placeholder execution identifiers as substitutes for `orchestration_session_id` - -Constraint: - -- do not widen unrelated shell command execution paths -- only world-scoped member/agent-hub execution should use the explicit shared-world binding path in this slice - -## Telemetry and Status Changes - -Primary files: - -- `crates/common/src/agent_events.rs` -- `crates/shell/src/execution/agent_events.rs` -- `crates/shell/src/execution/agents_cmd.rs` -- `docs/TRACE.md` - -Required outcome: - -- world-scoped pure-agent records can publish top-level `world_id` and `world_generation` from explicit ownership state -- nested gateway-backed records continue omitting those fields -- status output remains fail-closed if a world-scoped session lacks both fields - -This slice does not need to finish every future drift alert, but it must establish the data path that later `world_restarted` and `world_restart_required` alerts will consume. - -## Linux-First Platform Constraints - -### Linux - -Linux is the only platform that changes world ownership behavior in this slice. - -Implementation authority: - -- `crates/world` -- `crates/world-agent` -- shell-side agent runtime persistence - -### macOS - -Primary files likely touched later for additive compatibility only: - -- `crates/world-mac-lima/src/lib.rs` -- `crates/shell/src/execution/platform_world/mod.rs` -- `crates/shell/src/execution/routing/world.rs` - -Constraints for this slice: - -- accept additive request/model fields needed for shared-world ownership -- preserve existing Lima-backed world realization -- do not implement a separate shared-world ownership store in the macOS host path -- do not claim platform-complete shared-world restart/generation behavior beyond the Linux-backed contract - -### Windows - -Primary files likely touched later for additive compatibility only: - -- `crates/shell/src/execution/platform_world/windows.rs` -- `crates/world-windows-wsl/` - -Constraints for this slice: - -- accept additive request/model fields needed for shared-world ownership -- preserve existing WSL-backed world realization -- do not implement an independent Windows-native shared-world ownership algorithm -- fail closed at the agent-hub/shared-world boundary if explicit Linux-first ownership behavior is required but not yet supported - -## Sequencing - -1. Freeze the ownership contract and file layout. - - finalize shared-world binding schema - - finalize Linux `session.json` additions - - finalize additive `WorldSpec` / request-shape fields - -2. Land model changes. - - update `crates/world-api/src/lib.rs` - - update `crates/agent-api-types/src/lib.rs` - - update shell/runtime structs that hold `world_id` and `world_generation` - -3. Land Linux persistence and reuse behavior. - - implement binding store in shell runtime - - extend `SessionWorldMetadata` - - teach `LinuxLocalBackend` to resolve by explicit owner binding - -4. Thread request plumbing. - - shell dispatch passes explicit shared-world owner data - - world-agent builds ownership-aware `WorldSpec` - - shell receives authoritative `world_id` and generation - -5. Publish runtime state. - - persist live manifest fields from authoritative binding data - - remove placeholder correlation where it conflicts with explicit ownership - -6. Add validation coverage and doc updates. - - targeted Linux tests first - - additive status/trace checks - - update `docs/WORLD.md` and `docs/TRACE.md` after implementation lands - -## Acceptance Criteria - -- Linux world-scoped member allocation with a given `orchestration_session_id` creates one shared world with `world_generation = 0`. -- A second world-scoped member allocation for the same `orchestration_session_id` reuses the same `world_id` when world inputs are unchanged. -- A different `orchestration_session_id` never reuses that world, even if `WorldSpec` inputs are otherwise identical. -- Linux persisted world metadata includes `orchestration_session_id` and `world_generation`. -- Shell authoritative runtime state includes one binding file per orchestration session and one active `current_world_id`. -- World-scoped `AgentRuntimeSessionManifest` rows publish both `world_id` and `world_generation`. -- Host-scoped rows omit both fields. -- `substrate agent status` continues to fail closed if the selected newest world-scoped session lacks either field. -- macOS and Windows compile with the additive model changes and do not invent independent shared-world ownership semantics in this slice. +Generic reusable worlds remain separate and do not carry shared-owner proof. -## Validation and Testing Suggestions +Shared-owner reuse requires: -Targeted unit/integration coverage: +- exact `orchestration_session_id` match +- compatible world inputs +- `binding_state=Active` -- `crates/world/src/session.rs` - - persist and recover owned metadata - - reject reuse when `orchestration_session_id` mismatches - - reject reuse when generation metadata is missing for explicit shared-world flows -- `crates/world/src/lib.rs` - - exact-owner lookup wins over generic spec-compatible reuse for explicit shared-world requests - - generic callers without ownership fields preserve current behavior -- `crates/world-agent/src/pty.rs` - - persistent-session start accepts explicit shared-world binding fields - - ready frame returns the correct bound `world_id` -- `crates/world-agent/src/service.rs` - - non-PTY execute path uses ownership-aware `WorldSpec` - - no placeholder `span_id -> orchestration_session_id` behavior remains on shared-world paths -- `crates/shell/src/execution/agent_runtime/*` - - binding-store atomic write/read behavior - - live-manifest projection of `world_id` and `world_generation` - - restart replacement handle semantics -- `crates/shell/src/execution/agents_cmd.rs` - - world-scoped status rows render both fields - - malformed selected rows still fail closed +Ownerless legacy metadata, partial owner metadata, and inactive bindings are never reusable for shared-owner flows. -Suggested command validation once implemented: +### Replace window -- `cargo fmt --all` -- `cargo clippy --workspace --all-targets -- -D warnings` -- `cargo test --workspace -- --nocapture` -- targeted Linux-first runs for `crates/world`, `crates/world-agent`, and shell agent-runtime/status tests +`replace_shared_owner_session()` now follows this ordering: -Manual validation suggestions: +1. pre-commit the current world to `binding_state=Replacing` +2. commit a fresh replacement world with generation `expected_generation + 1` and `binding_state=Active` +3. roll the old world back to `Active` if replacement creation fails, then clean up the partial replacement root +4. finalize the old world to `binding_state=Replaced` after the replacement is committed -- start one orchestration session with a world-scoped member, record `world_id` and `world_generation` -- start a second world-scoped member in the same orchestration, verify the same `world_id` -- start a different orchestration session with identical workspace/policy inputs, verify a different `world_id` -- force a hub-driven restart path and verify generation increments exactly by `1` -- verify `substrate agent status --json` shows world fields only for world-scoped rows +`Replacing` is an internal transition state only. It is not a valid downstream proof state. -## Risks +### Recovery guarantees -- `WorldHandle` shape changes will fan out across Linux, macOS, and Windows call sites. -- Generic shell startup currently allocates worlds in several places; if the explicit binding boundary is not kept narrow, unrelated execution paths may change behavior. -- There is a risk of dual truth if shell binding files and Linux `session.json` can diverge. The implementation should define one authority and one mirrored copy with explicit conflict handling. -- Existing persisted `session.json` files under `/tmp/substrate-worlds` may lack the new ownership fields. Shared-world reuse must fail closed for explicit ownership flows rather than silently adopting legacy worlds. -- The current global `orchestration_session_id` helper in `crates/shell/src/execution/agent_events.rs` may remain useful for simple event flows, but it must not remain the authority for shared-world ownership. +Linux recovery now guarantees: -## Open Questions +- a lone `Replacing` world for an owner is reconciled back to `Active` +- a newer `Active` world is preferred over an older `Replacing` world for the same owner +- ambiguous same-owner recovery state fails closed +- malformed or partial owner metadata is ignored without being adopted +- `Replaced` and `Abandoned` worlds are never reused for shared-owner flows -1. Should the authoritative binding store live only under `~/.substrate/run/agent-hub/world-bindings/`, or should Linux `session.json` be treated as co-authoritative for recovery after shell restart? -2. Should `world_generation` live on `WorldHandle` directly, or should the shell read it from a separate binding snapshot after `ensure_session()` returns? -3. What is the exact restart API boundary for the later drift slice: a new backend method, or an ownership-aware `ensure_session()` with an explicit replacement flag? -4. How should shell startup worlds created by `initialize_world()` be prevented from colliding with later agent-hub shared-world bindings when both target the same workspace? -5. For non-Linux platforms in this slice, should explicit shared-world requests fail at shell routing before backend invocation, or should backends return a uniform unsupported/shared-world-unavailable error? +## What This File No Longer Claims -## Recommended Decision Posture for This Slice +Do not use this historical draft as authority for any of the following: -- Treat shell-side binding files as the primary authority for orchestration ownership. -- Treat Linux `session.json` as mirrored proof required for backend reuse and crash recovery. -- Keep explicit shared-world ownership opt-in and limited to world-scoped agent-hub execution. -- Preserve legacy spec-compatible reuse for non-agent callers until a later cleanup slice intentionally retires it. +- shell runtime persistence as the primary shared-world authority +- a co-authoritative or mirrored shell binding registry +- slice-03 ownership of runtime-manifest projection +- slice-03 ownership of participant invalidation or replacement registry behavior +- downstream acceptance of non-`Active` binding states diff --git a/llm-last-mile/04-thread-world-binding-into-runtime-state.md b/llm-last-mile/04-thread-world-binding-into-runtime-state.md index d2e8fd176..b9a838d5d 100644 --- a/llm-last-mile/04-thread-world-binding-into-runtime-state.md +++ b/llm-last-mile/04-thread-world-binding-into-runtime-state.md @@ -2,26 +2,33 @@ ## Objective -Make the shell-owned agent runtime state authoritative for world binding by persisting the active -`world_id` and `world_generation` in live runtime state, not only in trace/event records. +Make the shell-owned agent runtime state authoritative for the live projection of shared-world +binding by persisting the active `world_id` and `world_generation` in live runtime state, not only +in trace/event records. This prerequisite is complete when the runtime manifest/state store tracks the currently bound -world identity across initial startup, drift-triggered restarts, and fail-closed +world identity across initial startup, backend-accepted generation changes, and fail-closed restart-required cases, and when downstream live-state consumers can trust the persisted runtime -record instead of reconstructing world binding from trace history. +record instead of reconstructing world binding from trace history or shell-local inference. ## Why This Exists -The repo already has two parallel truths: +`PLAN-03` establishes the missing authority seam. Linux shared-world ownership and +`world_generation` now come from the world-api/Linux backend/world-agent contract, not from a +shell-local invention path. + +This slice starts after that contract exists. The repo still has two truths that need to be +joined: - Live runtime authority for orchestrator session ownership: - `crates/shell/src/execution/agent_runtime/session.rs` - `crates/shell/src/execution/agent_runtime/state_store.rs` - `crates/shell/src/execution/agents_cmd.rs` -- World identity authority for the active shared world: +- Backend-authoritative shared-world binding that now arrives through shell-visible transport: - in-memory `WorldSession` state in `crates/shell/src/repl/async_repl.rs` - persistent-session ready frames in `crates/shell/src/execution/repl_persistent_session.rs` - - trace/event records written via `crates/shell/src/execution/agent_events.rs` and + - non-PTY execute responses and trace/event records written via + `crates/shell/src/execution/agent_events.rs` and `crates/shell/src/execution/routing/telemetry.rs` Today those truths are not joined. The runtime manifest schema already has top-level @@ -40,16 +47,17 @@ Today those truths are not joined. The runtime manifest schema already has top-l - `AgentRuntimeStateStore::persist_manifest(...)` writes the full manifest JSON, but the lease sidecar in `persist_lease(...)` does not currently include world binding. -### World binding source of truth +### Authoritative world-binding ingress - `WorldSession` in `crates/shell/src/repl/async_repl.rs` carries: - `world_id` - `world_generation` - restart sequencing -- `open_world_session(...)` gets `ready.world_id` from the persistent-session protocol. -- `start_world_session(...)` seeds `world_generation: 0`. -- `restart_world_session(...)` increments `world_generation` and emits a `world_restarted` - alert. +- `open_world_session(...)` gets the ready-frame binding from the persistent-session protocol. +- `start_world_session(...)` and `restart_world_session(...)` already sit on the path where the + shell learns the backend-authoritative binding snapshot. +- In this slice, `WorldSession` is the shell ingress for that snapshot. It is not the place that + defines shared-world owner authority or generic Linux reuse rules. - `handle_detected_world_drift(...)` emits `world_restart_required` alerts in fail-closed mode. ### Status and authorization surfaces @@ -83,8 +91,9 @@ Today those truths are not joined. The runtime manifest schema already has top-l 1. `AgentRuntimeSessionManifest` has world-binding fields but no mutation API, so runtime state cannot reflect the active world even though the schema allows it. -2. The only durable place that tracks world generation changes today is trace/event history. - `WorldSession` knows the active generation in memory, but live runtime manifests do not. +2. Even after `PLAN-03`, the only durable shell-visible place that tracks authoritative world + generation changes is still trace/event history. `WorldSession` knows the active generation in + memory, but live runtime manifests do not. 3. Drift restart alerts (`world_restarted`, `world_restart_required`) publish top-level world identity, but they do not synchronize the runtime manifest before or with the event. @@ -105,10 +114,11 @@ This prerequisite is limited to the shell runtime state path. ### In scope -- Persisting active world binding into the shell runtime manifest and lease store. +- Persisting the active backend-authoritative world binding into the shell runtime manifest and + lease store. - Seeding that binding from the active `WorldSession` at orchestrator runtime startup. -- Updating that binding when the world restarts and when a fail-closed restart-required posture - is detected. +- Updating that projection when the backend advances the shared-world generation and when a + fail-closed restart-required posture is detected. - Making runtime event producers consume the persisted binding so runtime state and emitted events stay aligned. - Preserving current `agent status` and toolbox live-authority behavior while upgrading the @@ -117,6 +127,8 @@ This prerequisite is limited to the shell runtime state path. ### Out of scope - Relaxing host-only orchestrator selection. +- Defining the shared-world owner tuple, generic Linux world reuse behavior, or backend allocation + rules already owned by `PLAN-03`. - Adding a general multi-member live registry or `/v1/agents` service. - Changing `substrate world gateway status --json` or the gateway status schema contracts. - Redefining whether host-scoped selected orchestrator rows should display world binding in @@ -156,15 +168,15 @@ Required changes: - Thread the current `WorldSession` binding into `start_host_orchestrator_runtime(...)`. - `run_async_repl(...)` already starts the world session before starting the host orchestrator runtime; use that ordering. -- Seed the newly created runtime manifest with the active world binding before the runtime is - considered ready/live. +- Seed the newly created runtime manifest with the active authoritative world binding before the + runtime is considered ready/live. - Persist the bound manifest before emitting runtime events that should reflect live world attachment. Recommended implementation approach: -- Introduce a small helper in `async_repl.rs` that applies the `WorldSession` binding to the - in-memory manifest and persists it through `AgentRuntimeStateStore`. +- Introduce a small helper in `async_repl.rs` that applies the `WorldSession` binding snapshot to + the in-memory manifest and persists it through `AgentRuntimeStateStore`. - Use that helper: - once during initial orchestrator runtime bootstrap if `world_session.is_some()` - again after any world-session replacement caused by `restart_world_session(...)` @@ -179,18 +191,20 @@ Primary file: Required changes: -- Synchronize runtime manifest state whenever `WorldSession` changes. +- Synchronize runtime manifest state whenever the shell receives a new authoritative + `WorldSession` binding. - The ordering requirement is important: - update manifest binding - persist manifest - then emit the corresponding alert/event -- Ensure `world_generation` remains consistent with existing REPL semantics: - - startup generation: `0` - - each restart: `previous + 1` +- Ensure `world_generation` remains consistent with backend-authoritative restart semantics: + - initial shared-world allocation returns the first authoritative generation + - each replacement persists the next generation the backend accepted Specific seams to touch: - `start_world_session(...)` +- `open_world_session(...)` - `restart_world_session(...)` - `handle_detected_world_drift(...)` - any call sites that replace `world_session` after drift handling @@ -207,7 +221,7 @@ Primary files: Required changes: - Runtime-originated pure-agent events should derive world binding from the runtime manifest once - it is available, not from ad hoc call-site copies. + it is available, not from ad hoc call-site copies or shell-side regeneration. - World lifecycle alerts should remain top-level-field compatible with the existing trace/event contract, but the persisted runtime state must be updated first. - No event schema expansion is required for this prerequisite; use the existing top-level @@ -288,8 +302,8 @@ as the consistency model. - A live runtime manifest under `~/.substrate/run/agent-hub/handles/*.json` persists the active `world_id` and `world_generation` whenever the REPL has an active world session. - The paired `*.lease` file also includes the same world-binding fields. -- On world restart, the live manifest updates from the old binding to the new binding with the - incremented generation. +- On world restart, the live manifest updates from the old binding to the new + backend-authoritative binding with the incremented generation. - On fail-closed restart-required drift, the manifest still records the current authoritative binding instead of dropping back to `None`. @@ -404,6 +418,6 @@ that owns: ## Deliverable -Implement the runtime-state plumbing so that world binding is persisted in live shell runtime -state and remains aligned with event emission, while preserving the current host-scoped selected -orchestrator status contract. +Implement the runtime-state plumbing so that the backend-authoritative shared-world binding is +persisted in live shell runtime state and remains aligned with event emission, while preserving the +current host-scoped selected orchestrator status contract. diff --git a/llm-last-mile/05-restart-invalidation-semantics.md b/llm-last-mile/05-restart-invalidation-semantics.md index 3d187f5cb..5c4e81fa1 100644 --- a/llm-last-mile/05-restart-invalidation-semantics.md +++ b/llm-last-mile/05-restart-invalidation-semantics.md @@ -2,11 +2,16 @@ ## Objective -Define and land the live-state and registry semantics that make shared-world restarts authoritative for world-scoped member sessions. When `world_generation` changes for an `orchestration_session_id`, every prior-generation world-scoped member session must stop being live in the registry immediately, and no status or toolbox surface may resurrect those stale sessions from trace history. +Define and land the live-state and registry semantics that consume already-authoritative +shared-world restarts for world-scoped member sessions. Once `world_generation` changes for an +`orchestration_session_id`, every prior-generation world-scoped member session must stop being +live in the registry immediately, and no status or toolbox surface may resurrect those stale +sessions from trace history. This prerequisite exists to close the gap between: - the current authoritative-live manifest model in [`crates/shell/src/execution/agent_runtime/session.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs) and [`crates/shell/src/execution/agent_runtime/state_store.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs), +- the owner/generation authority established by [`PLAN-03.md`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-03.md) and the runtime-state projection established by [`04-thread-world-binding-into-runtime-state.md`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/04-thread-world-binding-into-runtime-state.md), - the current restart signaling in [`crates/shell/src/repl/async_repl.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs), - and the successor contract already written in [`docs/project_management/packs/draft/agent-hub-core-successor-identity-tuple-compatible/agent-hub-session-protocol-spec.md`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/project_management/packs/draft/agent-hub-core-successor-identity-tuple-compatible/agent-hub-session-protocol-spec.md). @@ -14,11 +19,18 @@ This prerequisite exists to close the gap between: The repo already has most of the pieces, but they are not yet locked together for restart invalidation: +- `PLAN-03` now owns the authoritative shared-world owner tuple and generation transition + contract. +- `PLAN-04` now owns persisting that authoritative binding into shell runtime state. +- This slice starts only after those two contracts hold. It does not define owner authority, + generic Linux world reuse, or generation assignment. + - `AgentRuntimeSessionState` already includes `Restarting` and `Invalidated`, and `is_authoritative_live()` already excludes invalidated manifests. - Source: [`crates/shell/src/execution/agent_runtime/session.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs) - `AgentRuntimeStateStore::list_live_manifests()` already treats persisted manifests as the authoritative live source and explicitly separates them from historical trace. - Source: [`crates/shell/src/execution/agent_runtime/state_store.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) -- REPL world drift already increments `world_generation` and emits `world_restarted` or `world_restart_required`. +- REPL world drift already emits `world_restarted` or `world_restart_required`, and the shell now + receives the active generation through the upstream binding contract before invalidation runs. - Source: [`crates/shell/src/repl/async_repl.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) - `substrate agent status` already merges authoritative live manifests with historical trace, preferring live manifests when present. - Source: [`crates/shell/src/execution/agents_cmd.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs) @@ -28,7 +40,9 @@ The repo already has most of the pieces, but they are not yet locked together fo - [`docs/project_management/packs/draft/agent-hub-core-successor-identity-tuple-compatible/telemetry-spec.md`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/project_management/packs/draft/agent-hub-core-successor-identity-tuple-compatible/telemetry-spec.md) - [`docs/project_management/packs/draft/agent-hub-core-successor-identity-tuple-compatible/policy-spec.md`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/project_management/packs/draft/agent-hub-core-successor-identity-tuple-compatible/policy-spec.md) -The missing prerequisite is the live-state rule that ties generation transitions to registry invalidation. Without that rule, a prior-generation member can remain selectable or be re-surfaced from trace fallback after the shared world has moved on. +The missing prerequisite is the live-state rule that ties an already-authoritative generation +transition to registry invalidation. Without that rule, a prior-generation member can remain +selectable or be re-surfaced from trace fallback after the shared world has moved on. ## Current Repo Seams @@ -39,6 +53,8 @@ The missing prerequisite is the live-state rule that ties generation transitions - [`crates/shell/src/execution/agent_runtime/state_store.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) - Owns persistence under `~/.substrate/run/agent-hub/handles/*.json`. - Currently lists live manifests but does not expose generation-aware invalidation helpers or an active-generation index. + - In this slice, the store consumes `world_generation` as an already-authoritative value; it + does not assign or prove that value. ### Status and operator surfaces @@ -52,7 +68,8 @@ The missing prerequisite is the live-state rule that ties generation transitions - [`crates/shell/src/repl/async_repl.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) - Owns `classify_world_restart_reason()`, `emit_world_restarted_alert()`, `build_world_restart_required_alert()`, and restart sequencing. - - Already increments `world_generation` on auto-restart. + - Receives or projects the active shared-world generation into shell-visible runtime state before + this slice runs its invalidation rules. - Already uses `Invalidated` for shell-owned runtime ownership loss, which is the closest existing state precedent. ### Existing contract and tests that this prerequisite must satisfy @@ -72,12 +89,15 @@ The missing prerequisite is the live-state rule that ties generation transitions - Define the required registry semantics for marking old-generation members non-live. - Define how `substrate agent status` must select, suppress, or omit stale member sessions. - Define how trace/history must remain auditable without being allowed to re-authorize stale sessions. -- Define sequencing expectations for restart commit ordering. +- Define sequencing expectations after owner binding and generation have already advanced. - Define concrete acceptance criteria and tests. ## Out of Scope - Shipping the full multi-member runtime if it does not already exist. +- Defining shared-world owner authority, generic Linux world reuse, or backend generation + assignment behavior already owned by `PLAN-03`. +- Re-defining runtime-state projection/persistence of the active binding already owned by `PLAN-04`. - Redesigning trace schema wholesale. - Changing orchestrator selection rules. - Replacing the current manifest storage layout under `~/.substrate/run/agent-hub/handles`. @@ -108,7 +128,8 @@ The missing prerequisite is the live-state rule that ties generation transitions ### 1. Authoritative identity and generation boundary -For world-scoped member sessions, authoritative live identity is not just `agent_id`. It is: +For world-scoped member sessions, authoritative live identity is not just `agent_id`. Given the +already-authoritative owner tuple and active world binding from `PLAN-03`/`PLAN-04`, it is: - `orchestration_session_id` - `agent_id` @@ -124,7 +145,8 @@ Rule: ### 2. Generation change is a hard invalidation barrier -When the shared world for an `orchestration_session_id` advances from generation `G` to `G+1`: +When the shared world for an `orchestration_session_id` advances from generation `G` to `G+1` and +that advancement is already authoritative: - every live world-scoped member manifest bound to generation `G` must be transitioned out of the live set, - every such manifest must end in `state=invalidated` unless a stricter terminal failure state is required, @@ -144,13 +166,14 @@ Non-negotiable invariant: ### 3. Commit ordering -Restart handling must use commit ordering that never leaves the registry in an ambiguous live state. +Restart handling must use commit ordering that never leaves the registry in an ambiguous live +state once the binding has advanced. Required sequence: 1. Detect drift or explicit restart need. -2. Allocate or prepare the replacement shared world. -3. Persist the new active world binding for the orchestration session. +2. `PLAN-03` commits or returns the replacement shared-world binding for generation `G+1`. +3. `PLAN-04` persists that authoritative active world binding for the orchestration session. 4. Persist replacement member manifests for the new generation. 5. Mark prior-generation member manifests `invalidated`. 6. Only after steps 3-5 succeed, emit operator-facing success reporting such as `world_restarted`. @@ -158,7 +181,8 @@ Required sequence: Implementation rule: - The alert is historical reporting. -- The manifest/registry update is the live-state authority. +- The manifest/registry update is the live-state authority after the upstream binding change is + already committed. - If those disagree, the manifests win for current-state reads. ### 4. Fail-closed posture when replacement is not ready @@ -167,7 +191,7 @@ If restart is required but replacement member handles are not ready yet: - prior-generation world-scoped member handles must still leave the authoritative live set, - status must not continue to show them as active, -- trace history must not resurrect them, +- invalidation tombstones must beat trace fallback and status selection, - policy evaluation for new world-scoped work must fail closed with a reason equivalent to the current successor draft: - “prior handle is invalidated and no replacement handle is ready”. @@ -235,7 +259,8 @@ Practical consequence: - Make status projection generation-aware. - Ensure invalidated/superseded registry state suppresses trace fallback for stale members. - [`crates/shell/src/repl/async_repl.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) - - Ensure restart alert emission happens after live-state commit. + - Ensure restart alert emission happens after the authoritative binding change has been projected + into live state and the invalidation commit has completed. - Provide the future restart handoff contract member runtimes will follow. ### Test and contract surfaces @@ -272,7 +297,9 @@ Practical consequence: ### Phase 3: Wire restart commit order -- Update restart handling so generation advancement, replacement-handle registration, and old-handle invalidation are committed before success alert publication. +- Update restart handling so the authoritative generation advancement from `PLAN-03`, the + runtime-state projection from `PLAN-04`, replacement-handle registration, and old-handle + invalidation are committed before success alert publication. - Ensure fail-closed posture does not leave stale prior-generation handles visible as live. ### Phase 4: Update status/reporting selection @@ -367,4 +394,6 @@ Practical consequence: ## Deliverable -A generation-aware live-state contract and implementation path that makes shared-world restart semantics deterministic for the registry, keeps status/toolbox surfaces fail-closed, and preserves trace as historical audit rather than live authorization. +A generation-aware live-state contract and implementation path that reacts to already-authoritative +shared-world restart semantics, keeps status/toolbox surfaces fail-closed, and preserves trace as +historical audit rather than live authorization. diff --git a/llm-last-mile/07-world-replacement-ordering-rollback-atomic-metadata.md b/llm-last-mile/07-world-replacement-ordering-rollback-atomic-metadata.md new file mode 100644 index 000000000..b73e7054e --- /dev/null +++ b/llm-last-mile/07-world-replacement-ordering-rollback-atomic-metadata.md @@ -0,0 +1,255 @@ +# SOW: World Replacement Ordering, Rollback, and Atomic Metadata Writes + +Status: implementation reference. This document records the exact backend hardening scope for Linux shared-world replacement ordering, rollback, and atomic metadata persistence so follow-on planning can rely on the actual contract that was required in code. + +## Objective + +Close the Linux shared-world replacement correctness gap so a shared-owner world restart: + +- never leaves an orchestration session with zero valid active world because replacement creation failed mid-transition, +- persists owner-bearing `session.json` metadata atomically, +- rolls the old world back to `Active` when replacement creation fails after the old world has already entered a pre-commit transition state, +- cleans up partial replacement roots before returning failure, and +- preserves a single authoritative downstream proof state: `binding_state=Active`. + +This scope is backend-first. It hardens the Linux shared-world authority layer in `crates/world` that later shell/runtime slices consume. + +## Problem Statement + +The explicit shared-world ownership contract was already landed at the world API and backend layer, but two correctness risks remained in the Linux backend: + +1. replacement ordering could transition the old world out of `Active` before the replacement world was durably committed, creating a crash/failure window where no valid active world remained for the owner, +2. Linux `session.json` metadata persistence was weaker than the shell runtime store’s atomic-write posture, making owner-bearing metadata more exposed to partial-write corruption or torn replacement-state transitions. + +This problem sits under the already-landed explicit shared-owner world model in: + +- [crates/world-api/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-api/src/lib.rs) +- [crates/world/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world/src/lib.rs) +- [crates/world/src/session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world/src/session.rs) +- [crates/world-agent/src/service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs) +- [crates/world-agent/src/pty.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/pty.rs) + +It is not a shell-runtime projection problem. Projection of active binding into shell state belongs to later `llm-last-mile` slices. + +## Current Repo Seams + +### Shared-owner contract surface + +- [crates/world-api/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-api/src/lib.rs) + - `WorldReuseMode` + - `SharedWorldOwnerSpec` + - `SharedWorldOwnerAction` + - `SharedWorldBindingSnapshot` + - `SharedWorldBindingState` + +- [crates/agent-api-types/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/agent-api-types/src/lib.rs) + - `ExecuteRequest.shared_world` + - `ExecuteResponse.shared_world` + +### Linux backend replacement path + +- [crates/world/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world/src/lib.rs) + - `LinuxLocalBackend::ensure_shared_owner_session_from_root(...)` + - `LinuxLocalBackend::replace_shared_owner_session_from_root(...)` + - `LinuxLocalBackend::replace_shared_owner_session_from_root_with_creator(...)` + - backend-local `shared_owner_mutex` + +### Linux world metadata authority + +- [crates/world/src/session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world/src/session.rs) + - `SessionWorldMetadata` + - `SessionWorld::to_metadata()` + - `SessionWorld::persist_metadata()` + - `SessionWorld::set_shared_binding_state(...)` + - `SessionWorld::recover_shared_active_from_root(...)` + +### Downstream proof consumers + +- [crates/world-agent/src/service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs) + - `build_world_spec(...)` + - `resolve_shared_world_binding(...)` + +- [crates/world-agent/src/pty.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/pty.rs) + - persistent-session ready-frame shared-world proof plumbing + +- [crates/shell/src/execution/repl_persistent_session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/repl_persistent_session.rs) + - fail-closed client proof validation + +## In Scope + +- Linux shared-owner replacement ordering in `crates/world` +- explicit pre-commit replacement state for the old world +- rollback to `Active` if replacement creation fails +- cleanup of partial replacement roots +- atomic `session.json` metadata persistence for Linux session worlds +- recovery rules for `Replacing` vs `Active` vs `Replaced` +- targeted backend tests for success, rollback, cleanup, and recovery + +## Out of Scope + +- shell runtime projection of `world_id` / `world_generation` +- participant invalidation or replacement-member creation +- toolbox/status selection logic +- non-Linux independent shared-world semantics +- broad trace/event schema work + +## Required Data / State Contract + +Linux session metadata for shared-owner worlds must remain owner-bearing and explicit: + +- `owner_mode=shared_orchestration` +- `orchestration_session_id` +- `world_generation` +- `binding_state` +- `last_restart_reason` +- `policy_snapshot_hash` +- `world_fs_mode` + +Only `binding_state=Active` is a valid downstream reusable proof state. `Replacing` is an internal transition state only. + +## Required Replacement Ordering + +`ReplaceExpectedGeneration` must follow this contract: + +1. Resolve the currently active shared-owner world for the requested `orchestration_session_id`. +2. Verify the current world’s generation matches `expected_generation`. +3. Allocate a fresh replacement `world_id`. +4. Pre-commit the old world to `binding_state=Replacing` and persist that metadata atomically. +5. Create the replacement world with: + - the same `orchestration_session_id` + - `world_generation = expected_generation + 1` + - `binding_state=Active` +6. If replacement creation fails: + - roll the old world back to `binding_state=Active`, + - persist rollback metadata atomically, + - remove any partial replacement root, + - return a single failure that includes create, rollback, and cleanup errors when present. +7. If replacement creation succeeds: + - finalize the old world to `binding_state=Replaced`, + - persist finalization metadata atomically, + - return the new active world handle even if old-world finalization logs a warning. + +This ordering keeps the only unsafe window on the side of “old world temporarily marked `Replacing` but still recoverable,” not “old world gone before replacement exists.” + +## Failure / Rollback Semantics + +### Create failure after pre-commit + +If the backend has already moved the old world to `Replacing` and replacement creation fails: + +- the old world must be restored to `Active`, +- recovery after the failure must still find the original world as the active reusable world, +- a partial replacement root must not remain reusable or recoverable, +- the returned error must preserve root cause detail. + +### Finalize-old-world failure after replacement commit + +If replacement creation succeeds but old-world finalization to `Replaced` fails: + +- the new world remains the committed active world, +- the backend may log a warning, +- recovery must prefer the new active world, +- downstream consumers must still only observe the new `Active` binding as reusable. + +### Recovery posture + +Recovery from persisted metadata must satisfy: + +- a lone `Replacing` world reconciles back to `Active`, +- an `Active` replacement outranks an older `Replacing` predecessor for the same owner, +- `Replaced` and `Abandoned` worlds are never reused, +- malformed partial owner metadata is ignored rather than adopted, +- ambiguous same-owner active states fail closed. + +## Required Atomic Metadata Persistence + +`SessionWorld::persist_metadata()` must use an atomic-write posture comparable to the shell runtime store: + +1. serialize metadata to a temp file inside the destination world directory, +2. `sync_all()` the temp file, +3. rename temp file onto `session.json`, +4. on Unix, best-effort `sync_all()` the containing directory, +5. remove temp files on failure paths. + +Target implementation seam: + +- [crates/world/src/session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world/src/session.rs) + +This must apply to: + +- initial shared-owner world creation, +- pre-commit `Replacing`, +- rollback to `Active`, +- finalize to `Replaced`, +- generic reusable world metadata where persistence is enabled. + +## Exact Code Areas To Touch + +### Primary implementation + +- [crates/world/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world/src/lib.rs) + - `replace_shared_owner_session_from_root(...)` + - `replace_shared_owner_session_from_root_with_creator(...)` + - `cleanup_partial_shared_world_root(...)` + - `ensure_shared_owner_session_from_root(...)` + +- [crates/world/src/session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world/src/session.rs) + - `persist_metadata()` + - `set_shared_binding_state(...)` + - `recover_shared_active_from_root(...)` + - metadata read/write helpers + +### Proof-surface validation context + +- [crates/world-agent/src/service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs) +- [crates/world-agent/src/pty.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/pty.rs) +- [crates/shell/src/execution/repl_persistent_session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/repl_persistent_session.rs) + +These are not the main edit surface, but they are required context because downstream consumers must continue to accept only `binding_state=Active`. + +## Testing Requirements + +At minimum, land targeted tests for: + +1. successful replacement commits a new `Active` world and finalizes the old world to `Replaced` +2. replacement creation failure rolls the old world back to `Active` +3. replacement creation failure cleans any partial replacement root +4. recovery prefers the committed new `Active` world over an older `Replacing` world +5. a lone `Replacing` world reconciles back to `Active` +6. malformed or partial owner metadata is rejected during recovery +7. atomic metadata persistence does not leave stray temp files after normal success + +Primary test anchors: + +- [crates/world/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world/src/lib.rs) +- [crates/world/src/session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world/src/session.rs) +- [crates/world-agent/src/service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs) + +Recommended verification commands: + +```bash +cargo test -p world-api shared_world_contract_round_trips_with_canonical_shape +cargo test -p agent-api-types execute_response_shared_world_round_trip +cargo test -p world +cargo test -p world-agent resolve_shared_world_binding_rejects_mismatched_owner_proof +``` + +## Acceptance Criteria + +- shared-owner replacement never leaves an owner with zero recoverable active world because replacement creation failed, +- replacement create failure restores the original world to `binding_state=Active`, +- partial replacement roots are cleaned on failure, +- `session.json` writes for Linux worlds are atomic, +- downstream proof consumers continue to observe only `binding_state=Active`, +- recovery semantics are deterministic and fail closed on ambiguity, +- targeted tests cover success, rollback, cleanup, and recovery. + +## Relationship To Other `llm-last-mile` Work + +This scope hardens the backend authority layer that later slices consume: + +- it is upstream of runtime-state projection work in [04-thread-world-binding-into-runtime-state.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/04-thread-world-binding-into-runtime-state.md), +- upstream of member invalidation work in [05-restart-invalidation-semantics.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/05-restart-invalidation-semantics.md), +- and upstream of session-centric runtime-store work in [06-session-centric-state-store.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/06-session-centric-state-store.md). + +It must not absorb participant replacement, shell registry mutation, or live status/toolbox UX changes. diff --git a/llm-last-mile/08-explicit-orchestration-authority-event-emission.md b/llm-last-mile/08-explicit-orchestration-authority-event-emission.md new file mode 100644 index 000000000..ed17608d3 --- /dev/null +++ b/llm-last-mile/08-explicit-orchestration-authority-event-emission.md @@ -0,0 +1,375 @@ +# SOW: Explicit Orchestration Authority for Event Emission + +## Objective + +Remove ambient PID and process-state lookup from shell-owned orchestration-scoped event emission paths, and replace it with explicit runtime-owned context threaded from session/bootstrap state into every emitter that can append `agent_event` rows. + +The required outcome is narrow and implementation-oriented: + +- no production event-emission path may recover `orchestration_session_id` by scanning runtime state for `shell_owner_pid`, +- shell-owned emitters must accept explicit runtime ownership/context from the caller, +- emitters that do not receive explicit orchestration context must suppress the orchestration-scoped `agent_event` row instead of reconstructing identity from ambient state, +- terminal stdout/stderr behavior and trace span emission must continue even when orchestration-scoped `agent_event` emission is suppressed. + +This slice is about authority and plumbing on event emission paths. It is not a redesign of agent-event schema, runtime-store layout, or operator-facing status selection. + +## Why This Is Needed + +The repo already states the intended contract in [`docs/TRACE.md`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/TRACE.md): + +- runtime-owned producers must emit a real `orchestration_session_id` or suppress the row, +- they must not synthesize a process-global fallback id, +- trace joins are supposed to use explicit keys, not heuristics. + +Current production code still violates that contract on several shell-generated paths by doing live PID-based recovery at emit time: + +- [`crates/shell/src/repl/async_repl.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) + - `resolve_active_orchestration_session_id()` + - REPL command-completion call sites +- [`crates/shell/src/execution/routing/dispatch/exec.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/exec.rs) + - `resolve_active_orchestration_session_id()` + - host external command stream emission +- [`crates/shell/src/execution/routing/dispatch/world_ops.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs) + - `resolve_active_orchestration_session_id()` + - non-PTY world stream emission + +That lookup shape is too weak for an authority boundary: + +- it depends on mutable runtime-store state and liveness scans during emission, +- it couples event identity to `shell_owner_pid` instead of the already-allocated orchestration session, +- it fails only after the emitter has already chosen to reach for ambient state, +- it is duplicated in multiple modules, so drift is likely, +- it is inconsistent with shell runtime paths that already have explicit manifest/session ownership in hand. + +The repo already has the correct source of truth for most shell-owned orchestrator activity. This SOW exists to require that the emitters consume it directly. + +## Current Repo Seams + +### Explicit runtime/session authority already exists + +The async REPL bootstrap path already allocates and persists explicit orchestration ownership: + +- [`crates/shell/src/repl/async_repl.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) + - `RuntimeOrchestrationContext` + - `PreparedHostOrchestratorRuntime` + - `start_host_orchestrator_runtime_with_prepared(...)` + - `translate_wrapper_event(...)` + - `build_runtime_message_event(...)` +- [`crates/shell/src/execution/agent_runtime/orchestration_session.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/orchestration_session.rs) + - `OrchestrationSessionRecord` +- [`crates/shell/src/execution/agent_runtime/session.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs) + - `AgentRuntimeParticipantRecord` + - participant lineage fields +- [`crates/shell/src/execution/agent_runtime/state_store.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) + - canonical session-root persistence under `run/agent-hub/sessions//...` + +Those paths already prove the repo does not need ambient PID lookup to know who owns the orchestration session. They already have that information in memory. + +### Event helpers are currently too weak + +[`crates/shell/src/execution/agent_events.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_events.rs) currently exposes low-level helpers: + +- `publish_agent_event(...)` +- `publish_command_completion(orchestration_session_id: Option<&str>, ...)` + +That API surface only accepts an optional session id string. It does not model runtime-owned orchestration authority as a first-class input, so callers are free to recover it ad hoc. + +### Ambient lookup is still used in production emitters + +1. REPL command completion + +- [`crates/shell/src/repl/async_repl.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) + - host escape completion + - world PTY completion + - world line completion + - host-only completion + +All currently call `publish_command_completion(...)` after `resolve_active_orchestration_session_id()`. + +2. Host external command stream chunks + +- [`crates/shell/src/execution/routing/dispatch/exec.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/exec.rs) + - `execute_external(...)` + - `spawn_host_stream_thread(...)` + +This path resolves orchestration identity once per process launch from PID-owned runtime state, then emits stream rows from background threads using that recovered value. + +3. World dispatch shell-generated stream chunks + +- [`crates/shell/src/execution/routing/dispatch/world_ops.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs) + - `stream_non_pty_via_agent(...)` + - `process_agent_stream_body(...)` + - `emit_stream_chunk(...)` + +This path also recovers orchestration identity via PID/state-store lookup before emitting shell-owned chunk rows. + +### Restart alert paths are partially explicit today + +[`crates/shell/src/repl/async_repl.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) already threads `startup_context` into: + +- `build_world_restart_required_alert(...)` +- `emit_world_restarted_alert(...)` +- `handle_detected_world_drift(...)` +- `restart_world_session(...)` + +Those paths already suppress rows when orchestration context is absent. They are the right behavioral model for the rest of this slice. + +## In Scope + +- introducing one explicit shell-local event-emission context model for orchestration-scoped shell events, +- threading that context through REPL host/world command execution paths, +- updating shell-owned command-completion and stream-chunk emission to consume explicit context, +- removing production use of PID-based orchestration lookup on event emission paths, +- tightening docs/tests so the explicit-context requirement is enforced, +- leaving pass-through wrapper/world-agent events untouched unless a shell-owned event wrapper is still deriving identity ambiently. + +## Out of Scope + +- redesigning `AgentEvent` wire schema, +- changing `substrate agent status` or toolbox session selection behavior, +- changing session-root or participant-root filesystem layout, +- changing Linux shared-world ownership semantics, +- broad trace replay or router/workflow schema changes, +- reworking agent wrapper event translation in paths that already have explicit manifest/session authority, +- changing historical trace rows beyond additive clarification in docs/tests. + +## Current Blockers and Gaps + +1. Production emitters can still reach for ambient authority. + +- `find_active_orchestration_session_for_pid(std::process::id())` is currently used as an event-time fallback in multiple modules. +- That is the wrong dependency direction: emission should consume already-owned context, not rediscover it from persisted state. + +2. Helper signatures do not force correct call-site behavior. + +- `publish_command_completion(...)` takes `Option<&str>`. +- `emit_stream_chunk(...)` takes raw optional strings plus caller-chosen run ids. +- Neither API makes “this came from the active shell-owned orchestration runtime” explicit. + +3. Host command execution does not currently accept runtime context. + +- [`execute_command(...)` in `dispatch/exec.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/exec.rs) only accepts `config`, `command`, `cmd_id`, and `running_child_pid`. +- That forces the callee to rediscover orchestration identity itself. + +4. Some shell-generated event paths still synthesize weak per-event correlation. + +- host stream threads currently emit with `run_id="unknown"`, +- world dispatch stream helpers still permit `run_id` values that are not explicitly tied to a caller-owned context object. + +This SOW does not need to redesign all run semantics, but it must stop orchestration-scoped shell rows from being emitted with ambiently recovered session identity plus synthetic fallback correlation. + +5. The state store currently exposes the exact ambient authority helper this slice wants to retire from production code. + +- [`AgentRuntimeStateStore::find_active_orchestration_session_for_pid(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) + +That method may remain for diagnostics/tests if still needed, but it must stop being part of production event-emission control flow. + +## Required Semantics and Invariants + +### 1. Explicit authority only + +Every shell-owned emitter that appends an orchestration-scoped `AgentEvent` must receive explicit orchestration authority from its caller. + +Minimum required authority payload: + +- `orchestration_session_id` +- event producer identity for the shell-owned row (`agent_id`, `role`, `backend_id`) when the helper sets those fields +- participant lineage when the row is logically attached to a persisted participant +- `world_id` and `world_generation` when the caller already has authoritative values + +Exact Rust type names are flexible, but the implementation must use one dedicated context type rather than more unrelated `Option<&str>` parameters. + +### 2. No PID-based recovery on emit paths + +The following production pattern must be eliminated from event-emission control flow: + +```rust +AgentRuntimeStateStore::new()? + .find_active_orchestration_session_for_pid(std::process::id()) +``` + +Specifically: + +- no REPL command-completion path may do this, +- no host external command stream path may do this, +- no world dispatch stream path may do this, +- no restart alert helper may reintroduce it as a fallback. + +### 3. Suppress, do not guess + +If an emitter does not receive explicit orchestration authority: + +- it must still write terminal stdout/stderr normally, +- it must still write ordinary command spans/trace telemetry normally, +- it must not append an orchestration-scoped `agent_event`, +- it must not scan runtime state, shell globals, PID tables, environment variables, or trace history to guess the orchestration session. + +This is the same fail-closed posture already used by: + +- [`publish_command_completion(None, ...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_events.rs) +- [`build_world_restart_required_alert(None, ...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) +- [`emit_stream_chunk(..., None, ...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs) + +### 4. Caller-owned run/cmd correlation + +For shell-owned orchestration-scoped rows emitted during command execution, the caller must also provide the per-command correlation needed for that row. + +At minimum: + +- `cmd_id` for command-completion rows, +- `run_id` for stream-chunk rows, +- `span_id` when the event is tied to a known command span. + +Required rule: + +- orchestration-scoped shell rows must not emit with synthetic fallback values like `"unknown"` once explicit context plumbing is available. +- if the caller cannot provide a real run identity for an orchestration-scoped stream row, suppress the row instead of inventing one. + +### 5. Existing explicit runtime translation remains the model + +Paths that already translate wrapper/runtime events from explicit manifest/session context must remain explicit and unchanged in principle: + +- `translate_wrapper_event(...)` +- `build_runtime_message_event(...)` +- world-restart alert construction from `startup_context` + +This slice should align the weaker host/world shell emitters with that existing model rather than invent a second authority path. + +### 6. Trace doc contract must become implementation contract + +[`docs/TRACE.md`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/TRACE.md) must continue to say: + +- runtime-owned producers emit a real `orchestration_session_id` or suppress the row, +- no process-global fallback id is allowed. + +After this slice, production code must actually satisfy that statement across shell-owned event emitters. + +## Exact Code Areas + +### Primary implementation + +- [`crates/shell/src/execution/agent_events.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_events.rs) + - introduce the explicit event-emission context type + - update `publish_command_completion(...)` to consume explicit context rather than a bare optional session id + - keep suppression semantics when context is absent + +- [`crates/shell/src/execution/routing/dispatch/exec.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/exec.rs) + - remove `resolve_active_orchestration_session_id()` + - thread optional explicit emission context into `execute_command(...)` + - thread command/run context into `execute_external(...)` + - update `spawn_host_stream_thread(...)` so it receives caller-owned context instead of ambiently recovered session identity + +- [`crates/shell/src/execution/routing/dispatch/world_ops.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs) + - remove `resolve_active_orchestration_session_id()` + - thread explicit emission context into `stream_non_pty_via_agent(...)` + - update `process_agent_stream_body(...)` and `emit_stream_chunk(...)` so shell-generated chunk rows use caller-owned context only + +- [`crates/shell/src/repl/async_repl.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) + - stop calling `resolve_active_orchestration_session_id()` for command completion + - derive event-emission context from `RuntimeOrchestrationContext` / active manifest snapshots + - pass that context into host/world execution helpers + - preserve current explicit restart-alert behavior + +### Related authority and compatibility seams + +- [`crates/shell/src/execution/agent_runtime/state_store.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) + - production emitters must stop depending on `find_active_orchestration_session_for_pid(...)` + - keeping the helper for tests/diagnostics is acceptable if production paths no longer call it + +- [`crates/common/src/agent_events.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/common/src/agent_events.rs) + - no schema redesign required, but this file is the field contract the new explicit shell-local context must populate correctly + +- [`docs/TRACE.md`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/TRACE.md) + - update wording if needed so shell-owned command/stream emitters are explicitly covered by the no-fallback rule + +## Testing Requirements + +At minimum, land or update tests for: + +1. command completion emits only when explicit orchestration context is supplied + +- anchor: + - [`crates/shell/src/execution/agent_events.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_events.rs) + +2. host stream chunk emission emits only when explicit orchestration context is supplied, and does not emit an orchestration-scoped row when context is absent + +- anchor: + - [`crates/shell/src/execution/routing/dispatch/world_ops.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs) + +3. REPL host and world command-completion paths no longer read orchestration identity from PID-owned runtime state + +- anchors: + - [`crates/shell/src/repl/async_repl.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) + - [`crates/shell/src/execution/routing/dispatch/exec.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/exec.rs) + +4. shell-owned stream rows do not emit with synthetic fallback run identity once explicit context plumbing exists + +- anchors: + - [`crates/shell/src/execution/routing/dispatch/exec.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/exec.rs) + - [`crates/shell/src/execution/routing/dispatch/world_ops.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs) + +5. world restart alert behavior remains explicit-context-only + +- anchor: + - [`crates/shell/src/repl/async_repl.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) + +6. repo-wide guard that production code no longer uses PID lookup for event emission + +- acceptable forms: + - direct tests for the new context plumbing, + - or a bounded repo assertion that only tests/diagnostics still reference `find_active_orchestration_session_for_pid(...)`. + +Recommended verification commands: + +```bash +cargo test -p shell agent_events +cargo test -p shell async_repl +cargo test -p shell routing::dispatch::world_ops +cargo test -p shell routing::dispatch::tests::host_replay +``` + +If module-filtered targets are not stable in this crate layout, run: + +```bash +cargo test -p shell -- --nocapture +``` + +## Acceptance Criteria + +- no production event-emission path in `async_repl`, `dispatch/exec`, or `dispatch/world_ops` looks up orchestration identity from `shell_owner_pid`, +- shell-owned command-completion events consume explicit caller-provided orchestration context, +- shell-owned stream-chunk events consume explicit caller-provided orchestration and run context, +- missing orchestration context suppresses the orchestration-scoped `agent_event` row without suppressing stdout/stderr or command spans, +- existing explicit runtime event translation paths remain explicit and continue to work, +- docs and tests reflect the “real id or suppress” contract, +- the remaining state-store PID lookup helper, if retained, is no longer part of production emission control flow. + +## Recommended Execution Order + +1. Define the explicit shell event-emission context type and update `publish_command_completion(...)`. + +2. Thread optional context from `RuntimeOrchestrationContext` / active manifest state through REPL command execution entry points. + +3. Update `dispatch/exec.rs` host execution helpers to accept that context and remove ambient PID lookup. + +4. Update `dispatch/world_ops.rs` non-PTY/stream helpers to accept explicit context and remove ambient PID lookup. + +5. Tighten tests around suppression, explicit context, and no-synthetic-fallback behavior. + +6. Update [`docs/TRACE.md`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/TRACE.md) so the documented contract matches the landed implementation. + +## Relationship To Existing `llm-last-mile` Work + +This slice is downstream of the earlier identity/runtime-state work: + +- [`01-orchestration-session-identity.md`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/01-orchestration-session-identity.md) +- [`02-session-participant-record.md`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/02-session-participant-record.md) +- [`06-session-centric-state-store.md`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/06-session-centric-state-store.md) + +It relies on those slices having created real session/participant authority that can be threaded directly. + +It should stay separate from: + +- shared-world backend correctness work in [`07-world-replacement-ordering-rollback-atomic-metadata.md`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/07-world-replacement-ordering-rollback-atomic-metadata.md), +- operator surface selection and status aggregation work, +- any larger trace-schema or workflow-router/toolbox redesign. diff --git a/llm-last-mile/09-live-state-authority-and-compatibility-cutover.md b/llm-last-mile/09-live-state-authority-and-compatibility-cutover.md new file mode 100644 index 000000000..7a00505f8 --- /dev/null +++ b/llm-last-mile/09-live-state-authority-and-compatibility-cutover.md @@ -0,0 +1,421 @@ +# SOW: Live-State Authority Boundary and Compatibility Cutover + +Status: implementation-oriented scope. This document freezes the production contract for live-state authority across canonical session-root records, flat compatibility files, and trace fallback so the remaining cleanup work does not reintroduce split authority now that shell-owned trace emission already follows the explicit real-id-or-suppress contract landed in `PLAN-08`. + +## Objective + +Make one boundary authoritative for live agent-hub state: + +- canonical parent session records under `sessions//session.json`, +- canonical participant records under `sessions//participants/.json`, +- store-owned live-session resolution for operator surfaces, +- bounded compatibility reads and writes only where required during cutover, +- trace fallback limited to historical gap-filling and never elevated into control-plane authority. + +This SOW exists to remove the last ambiguity around "what is truth" for: + +- `substrate agent status` +- `substrate agent toolbox status` +- `substrate agent toolbox env` +- the cutover posture for flat compatibility files and legacy `handles/*.json` + +## Why Needed + +The repo now contains the right pieces, but the final authority contract is still easy to misread because it is spread across runtime code, tests, and docs: + +- the store already prefers canonical session-root objects and degrades torn roots with warnings: + [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) +- `status` already overlays live session records ahead of trace fallback and suppresses stale trace rows behind tombstones: + [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs) +- `toolbox status|env` already resolves one live session through the store and fails closed on ambiguity or broken parent/child linkage: + [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs) +- shell-owned command-completion, stream, and restart-alert `agent_event` rows now already obey the explicit real-id-or-suppress trace contract instead of doing ambient PID-based recovery: + [docs/TRACE.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/TRACE.md) +- docs now state that canonical session-root records outrank flat compatibility files and trace, but the write-side retirement decision is not yet captured in one execution-ready SOW: + [docs/TRACE.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/TRACE.md) + [docs/USAGE.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/USAGE.md) + +Without one explicit cutover contract, follow-on cleanup work can easily regress into one of these bad states: + +- flat compatibility files regaining live-state authority, +- trace rows being used to authorize current toolbox sessions, +- ambiguous multi-session orchestrator state silently picking "latest", +- dual-write being removed too early without proving no compatibility readers still need it, +- torn-root crash windows turning into accidental operator-facing false positives. + +## Current Repo Seams + +### Live-state store and precedence + +Primary owner: + +- [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) + +Key seams: + +- `load_authoritative_session(...)` +- `build_session_record(...)` +- `list_sessions()` +- `list_live_sessions()` +- `resolve_single_live_session_for_agent(...)` +- `list_invalidated_participants_across_sources()` +- `persist_orchestration_session(...)` +- `persist_participant(...)` +- `persist_parent_session_snapshot(...)` + +This file already encodes the real precedence rule: + +1. canonical session-root parent record, +2. flat compatibility parent record only if canonical parent is absent, +3. canonical participant record, +4. flat compatibility participant record, +5. legacy `handles/*.json` only as last-resort compatibility input. + +### Parent session model + +- [crates/shell/src/execution/agent_runtime/orchestration_session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/orchestration_session.rs) + +This file owns the parent-session state used by live discovery: + +- `state` +- `active_session_handle_id` +- `world_id` +- `world_generation` +- `shell_owner_pid` + +### Operator surfaces + +- [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs) + +Key seams: + +- `build_status_report(...)` +- `live_session_status_projections(...)` +- `build_toolbox_status_report(...)` +- `build_toolbox_env_report(...)` +- trace fallback suppression helpers + +### Runtime writer choke point + +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) + +Key seam: + +- `persist_runtime_snapshots(...)` + +This is where parent and participant snapshots enter the store and where any write-side compatibility bridge must remain centralized. + +### Existing contract tests + +Primary regression anchor: + +- [crates/shell/tests/agent_successor_contract_ahcsitc0.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_successor_contract_ahcsitc0.rs) + +This suite already locks most of the intended behavior for: + +- canonical-over-flat precedence, +- flat fallback when canonical roots are incomplete, +- trace fallback suppression, +- tombstone suppression, +- fail-closed ambiguity and parent/child mismatch handling, +- doc wording around live-state authority. + +## In Scope + +- freezing the live-state authority order across canonical records, flat compatibility files, and trace fallback +- deciding whether dual-write remains intentional during the cutover and where it is allowed to live +- defining exact ambiguity and torn-root behavior for `status` and `toolbox` +- tightening doc cleanup so active docs describe the same authority boundary as runtime code +- preserving `PLAN-05` tombstone suppression while cleaning up compatibility behavior +- defining the removal gates for flat compatibility writes and legacy `handles/*.json` reads + +## Out of Scope + +- redesigning the parent session schema +- changing the public JSON fields on `substrate agent status` or `substrate agent toolbox status` +- changing trace schema or adding new trace families +- introducing a new transactional runtime registry +- re-opening shell-owned event-emission authority cleanup already landed by [PLAN-08.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-08.md) +- reworking world replacement semantics from [05-restart-invalidation-semantics.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/05-restart-invalidation-semantics.md) +- removing compatibility reads or writes in the same step that merely documents the contract + +## Blockers And Gaps + +### 1. Flat-file retirement is not yet proven safe + +The repo still intentionally reads: + +- `~/.substrate/run/agent-hub/sessions/.json` +- `~/.substrate/run/agent-hub/participants/*.json` +- `~/.substrate/run/agent-hub/handles/*.json` + +That means immediate compatibility-write removal is not justified until: + +- in-repo fixtures stop depending on flat fallback except where explicitly testing migration behavior, +- active docs stop describing flat files as anything other than compatibility input, +- an owner confirms there are no required off-repo consumers of flat session or participant files. + +### 2. Operator semantics are spread across tests and code paths + +The runtime already fails closed on the right ambiguity classes, but the rules currently live in: + +- store error strings, +- `toolbox env` exit-code routing, +- status suppression logic, +- individual contract tests. + +This SOW must unify those into one execution order so later cleanup does not relax them accidentally. + +### 3. Torn-root behavior is intentionally degraded, not transactional + +Current code tolerates: + +- parent-only canonical roots, +- participant-only canonical roots, +- flat compatibility participant fallback when the canonical parent exists but the canonical child is missing. + +That is the correct current posture, but it must stay explicit. The cutover does not authorize pretending these states are impossible. + +## Required Semantics And Invariants + +### 1. Authority order + +For live-state operator surfaces, truth is ordered exactly like this: + +1. canonical parent session record: + `~/.substrate/run/agent-hub/sessions//session.json` +2. canonical participant record: + `~/.substrate/run/agent-hub/sessions//participants/.json` +3. flat compatibility parent and participant files only when needed to fill an incomplete canonical root on read +4. legacy `handles/*.json` only as compatibility input of last resort +5. trace fallback only for historical status projection gaps, never for current-session authority + +Non-negotiable rules: + +- canonical objects outrank conflicting flat compatibility objects +- flat compatibility input may fill a torn-root gap but may not overrule a present canonical object +- legacy `handles/*.json` is never authoritative live-state input +- trace never outranks canonical or flat runtime state + +### 2. Dual-write decision + +Dual-write remains intentional during this cutover, but only in a bounded form: + +- it is allowed only inside [state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) +- it may write flat compatibility copies for: + - `sessions/.json` + - `participants/.json` + - flat lease files +- it must not be duplicated in callers +- it must not reintroduce new write ownership for `handles/*.json` + +Required posture: + +- canonical session-root writes are the real write target +- flat compatibility writes are a temporary migration bridge only +- `handles/*.json` remains read-only compatibility input and should be removed last + +Removal gate: + +- remove flat dual-write only after compatibility reads and fixtures are proven unnecessary outside explicit migration tests + +### 3. `substrate agent status` semantics + +`status` must: + +- build authoritative live rows from `list_live_sessions()` +- preserve one row per live participant, not one row per `agent_id` +- use trace fallback only for tuples not already covered by a live record or invalidated tombstone +- suppress stale trace fallback when `list_invalidated_participants_across_sources()` finds an invalidated tuple for the same `(orchestration_session_id, agent_id, execution.scope)` + +`status` must not: + +- silently collapse concurrent sessions for the same orchestrator agent +- resurrect invalidated world members from trace +- treat historical trace rows as proof that a toolbox/control-plane session is currently live + +### 4. `substrate agent toolbox status|env` semantics + +`toolbox status` and `toolbox env` must resolve through `resolve_single_live_session_for_agent(...)`. + +Required behavior: + +- exactly one valid live host-scoped orchestrator session: + - `toolbox status` reports concrete endpoint, `active_orchestration_session_id`, and optional parent `world_id` / `world_generation` + - `toolbox env` succeeds +- zero valid live host-scoped orchestrator sessions: + - `toolbox status` returns `dependency_unavailable` + - `toolbox env` exits `3` +- more than one valid candidate or any broken parent/child linkage: + - fail closed + - keep the error operator-readable + - never pick the newest candidate heuristically + +Ambiguity or corruption classes that must fail closed: + +- multiple active parent sessions for one orchestrator agent +- multiple live host-scoped orchestrator participants for one orchestrator agent +- live child exists without an active parent +- active parent missing `active_session_handle_id` +- active parent points at a missing participant +- active parent points at an inactive participant +- parent and participant disagree on `orchestration_session_id` +- parent points at a non-host or non-orchestrator participant + +### 5. Torn-root behavior + +Torn roots must degrade, not self-authorize: + +- participant-only roots may be discovered by `list_sessions()`, but remain incomplete and non-live +- parent-only roots may be discovered by `load_session()`, but remain incomplete and non-live +- warnings must be preserved on incomplete session records +- incomplete roots must not be promoted by `list_live_sessions()` +- flat compatibility participant fallback remains allowed when the canonical parent exists but the canonical child is absent +- trace fallback must not hide torn-root warnings by pretending current liveness exists + +### 6. Trace boundary + +Trace remains the historical event log, not the live registry. + +Required rules: + +- shell-owned trace rows are assumed to have already satisfied the explicit real-id-or-suppress contract from [PLAN-08.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-08.md); this slice does not reintroduce heuristic event identity recovery +- trace may fill status gaps only after live runtime state and tombstones have had first priority +- trace must not authorize `toolbox env` +- trace must not authorize orchestrator control-plane health or selection +- trace must not resurrect invalidated participants + +## Exact Code And Doc Areas + +### Primary implementation surfaces + +- [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) + - precedence + - torn-root completeness rules + - dual-write ownership + - legacy-read retirement sequencing +- [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs) + - status live-row selection + - trace fallback suppression + - toolbox ambiguity handling + - exit-code mapping for unavailable versus unsupported versus denied +- [crates/shell/src/execution/agent_runtime/orchestration_session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/orchestration_session.rs) + - parent fields that define active-session authority +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) + - `persist_runtime_snapshots(...)` + - ensuring store-owned canonical writes remain the only write choke point + +### Primary docs to align + +- [docs/TRACE.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/TRACE.md) + - canonical live-state authority wording + - trace fallback and tombstone suppression wording +- [docs/USAGE.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/USAGE.md) + - operator-facing live discovery wording + - toolbox `dependency_unavailable` posture + +### Secondary doc audit if active wording still exists + +- [docs/project_management/packs/draft/agent-hub-core-successor-identity-tuple-compatible/compatibility-spec.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/project_management/packs/draft/agent-hub-core-successor-identity-tuple-compatible/compatibility-spec.md) +- [docs/project_management/packs/draft/agent-hub-core-successor-identity-tuple-compatible/manual_testing_playbook.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/project_management/packs/draft/agent-hub-core-successor-identity-tuple-compatible/manual_testing_playbook.md) + +These are not the primary authority files for runtime discovery, but they should not contradict the cutover decision. + +## Testing Requirements + +At minimum, preserve or extend regression coverage for: + +1. canonical session-root records outrank conflicting flat compatibility files +2. flat compatibility participant fallback remains readable when the canonical parent exists but the canonical child is missing +3. participant-only and parent-only torn roots stay incomplete and never become live +4. `toolbox env` fails closed when trace is the only evidence of liveness +5. `toolbox status|env` fail closed on all ambiguity and parent/child mismatch classes +6. `status` prefers live runtime state over newer trace fallback +7. invalidated tombstones suppress stale trace rows for the same tuple +8. dual-write continues to produce canonical roots plus flat compatibility copies while the bridge remains intentional +9. no new test or fixture depends on `handles/*.json` as the primary write target + +Primary anchors: + +- [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) + - `persist_writes_canonical_and_flat_compatibility_layouts` + - `list_sessions_discovers_participant_only_roots_and_excludes_them_from_live_results` + - `parent_only_torn_roots_degrade_with_warnings_instead_of_failing_discovery` +- [crates/shell/tests/agent_successor_contract_ahcsitc0.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_successor_contract_ahcsitc0.rs) + - `agent_toolbox_env_trace_history_does_not_authorize_active_session` + - `agent_toolbox_surfaces_prefer_canonical_session_roots_over_flat_compatibility_files` + - `agent_toolbox_surfaces_fall_back_to_flat_participant_when_canonical_root_is_incomplete` + - `operator_surfaces_fail_closed_when_multiple_active_parent_candidates_exist` + - `agent_status_prefers_live_manifest_over_trace_fallback_for_selected_orchestrator` + - `agent_status_tombstone_suppression_beats_stale_trace_fallback_for_world_member` + +Recommended verification commands: + +```bash +cargo test -p shell agent_runtime::state_store -- --nocapture +cargo test -p shell agent_successor_contract_ahcsitc0 -- --nocapture +``` + +## Acceptance Criteria + +- canonical session-root parent and participant records are the documented and implemented live-state authority +- flat `sessions/*.json`, `participants/*.json`, and flat lease files are explicitly temporary compatibility input/output only during the bridge +- `handles/*.json` is never described or treated as the authoritative live-state registry +- dual-write, while it exists, is store-owned only and does not leak into callers +- `substrate agent status` keeps concurrent sessions distinct and does not resurrect invalidated tuples from trace +- `substrate agent toolbox status|env` fail closed on ambiguity, missing parent linkage, or inactive selected participants +- torn roots remain discoverable with warnings but do not become live +- active docs describe the same authority boundary as code and tests + +## Recommended Execution Order + +### 1. Freeze the boundary first + +Lock the precedence and dual-write decision in the store-owned contract: + +- canonical root is truth +- flat compatibility data is bounded bridge input/output +- `handles/*.json` is legacy input only +- trace is historical fallback only + +### 2. Keep operator surfaces strict + +Audit `status` and `toolbox` against that boundary: + +- no caller-owned regrouping +- no heuristic "latest" selection +- no trace-based control-plane authorization +- no promotion of incomplete roots into live discovery + +### 3. Finish doc cleanup while behavior is still fresh + +Update active docs so operators see the same rule set that tests enforce: + +- one authority statement in `TRACE.md` +- one operator statement in `USAGE.md` +- secondary compatibility-pack wording only if it contradicts the runtime contract + +### 4. Retire the bridge last + +Only after the above is green: + +- stop relying on flat compatibility reads except in explicit migration tests +- remove flat dual-write +- remove `handles/*.json` reads last + +This sequence keeps the repo safe under concurrent development because it removes stale wording and hidden heuristics before it removes compatibility scaffolding. + +## Relationship To Other `llm-last-mile` Work + +This scope is downstream of: + +- [08-explicit-orchestration-authority-event-emission.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/08-explicit-orchestration-authority-event-emission.md) +- [05-restart-invalidation-semantics.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/05-restart-invalidation-semantics.md) +- [06-session-centric-state-store.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/06-session-centric-state-store.md) + +It should be read as the cutover and cleanup contract on top of those earlier slices: + +- `08` removes ambient event-identity recovery and leaves trace rows on the explicit real-id-or-suppress footing this slice now assumes +- `05` defines invalidation and tombstone meaning +- `06` defines session-centric store structure +- this file defines which sources remain authoritative during the migration bridge and how operator surfaces must behave when those sources disagree or are incomplete diff --git a/llm-last-mile/10-member-runtime-launch-seam.md b/llm-last-mile/10-member-runtime-launch-seam.md new file mode 100644 index 000000000..7bda57d8c --- /dev/null +++ b/llm-last-mile/10-member-runtime-launch-seam.md @@ -0,0 +1,402 @@ +# SOW: Production Member Runtime Launch Seam + +Status: implementation-oriented draft. This document defines the missing production seam for launching and tracking world-scoped member runtimes inside an existing orchestration session. It is intentionally bounded to launch and lifecycle plumbing. It consumes the authority and cutover contracts already frozen by earlier `llm-last-mile` slices and does not reopen shared-world authority, replacement ordering, event-emission authority, or live-state compatibility-cutover design. + +## Objective + +Land one real production launch seam for world-scoped member participants so the shell can: + +- select an allowed member backend, +- launch it against the already-authoritative active shared world, +- persist a real live `member` participant record, +- stream and observe its lifecycle with the same rigor as the host orchestrator path, +- and surface that member through existing runtime store and status machinery. + +The required outcome is not "member-shaped JSON exists." The required outcome is that production code in the shell actually creates, starts, persists, updates, and retires member participants under a live `orchestration_session_id`. + +## Why This Is Needed + +The repository now has strong runtime modeling, but the launch seam is still missing. + +What already exists: + +- participant-capable runtime records in [session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs) +- session-centric runtime storage in [state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) +- canonical session-root parent plus participant records are now the frozen live-state authority boundary, with flat compatibility files and legacy handles explicitly demoted to bounded bridge/input roles by [PLAN-09.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-09.md) +- shared-world authority and generation plumbing from [PLAN-03.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-03.md), [04-thread-world-binding-into-runtime-state.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/04-thread-world-binding-into-runtime-state.md), and [07-world-replacement-ordering-rollback-atomic-metadata.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/07-world-replacement-ordering-rollback-atomic-metadata.md) +- stale-generation invalidation rules in [05-restart-invalidation-semantics.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/05-restart-invalidation-semantics.md) +- shell-owned orchestration-scoped trace rows now already obey the explicit real-id-or-suppress event contract from [PLAN-08.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-08.md) +- live status/toolbox projections that can already consume member participants once real producers exist in [agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs) + +What is still missing: + +- the only production runtime bootstrap path today is the host orchestrator path in [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) +- the only production `AgentWrapperGateway.run_control(...)` call today is also in that host orchestrator path +- `new_member_participant(...)` and `new_replacement_participant(...)` exist, but there is no production caller outside tests and store fixtures +- invalidation and status suppression logic therefore consume a future contract rather than a live producer + +That mismatch is now the bottleneck. The repo can describe member participants honestly, but it still cannot launch one in production. + +## Current Repo Seams + +### Production runtime path that exists today + +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) + - `prepare_host_orchestrator_runtime_startup(...)` + - `start_host_orchestrator_runtime_with_prepared(...)` + - `shutdown_host_orchestrator_runtime(...)` + - `persist_world_binding_authority(...)` + - `invalidate_stale_world_members_after_binding(...)` +- That file proves the current runtime is orchestrator-first: + - it allocates one host-scoped orchestrator participant, + - persists one parent orchestration session, + - and calls `gateway.run_control(...)` only for that host runtime. + +### Runtime descriptor and registry surfaces + +- [crates/shell/src/execution/agent_runtime/validator.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/validator.rs) + - validates only the selected orchestrator path today + - `validate_runtime_realizability(...)` is written around "selected orchestrator" and the first caller path +- [crates/shell/src/execution/agent_runtime/registry.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/registry.rs) + - builds a gateway from one `RuntimeSelectionDescriptor` + - registers exactly one backend implementation for that descriptor +- [crates/shell/src/execution/agent_runtime/mod.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/mod.rs) + - exports orchestrator-oriented helpers today + - does not yet expose a production member-launch entrypoint + +### Runtime state that is ready to consume a real member producer + +- [crates/shell/src/execution/agent_runtime/session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs) + - already models `role=member` + - already models required `orchestrator_participant_id` + - already models `world_id` and `world_generation` + - already models replacement lineage through `resumed_from_participant_id` +- [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) + - already persists participants under the session-centric runtime tree + - already groups participants into `AgentRuntimeSessionRecord` + - already has `invalidate_stale_world_members_for_session(...)` + - already suppresses invalidated participants in live reads + +### Status and toolbox consumer posture + +- [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs) + - `build_status_report(...)` already merges live runtime participants with trace-derived history + - invalidated world members already suppress stale trace fallback + - `build_toolbox_status_report(...)` already anchors toolbox selection to the live orchestrator session, not a member runtime +- This means status/toolbox mostly need producer validation, not a redesign. + +### Existing tests that define the seam boundary + +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) + - current unit coverage is for host orchestrator startup/shutdown and invalidation +- [crates/shell/tests/agent_successor_contract_ahcsitc0.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_successor_contract_ahcsitc0.rs) + - contract tests for status, toolbox, inventory, and runtime-state surfaces +- [crates/shell/tests/repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs) + - shared-world startup, restart, and persisted binding behavior +- [crates/shell/tests/agent_hub_trace_persistence.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_hub_trace_persistence.rs) + - persisted runtime and trace evidence for the current orchestrator path + +### Important non-seam + +- [crates/shell/src/builtins/world_gateway.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/builtins/world_gateway.rs) + - owns nested LLM gateway lifecycle + - is not the member runtime launch surface + - must not be repurposed as the source of truth for member liveness or session identity + +## In Scope + +- one production launch path for a world-scoped member runtime under an existing live orchestration session +- member descriptor selection, runtime realizability checks, and backend allowlist enforcement +- member participant allocation, persistence, and lifecycle transitions +- integration with already-authoritative world binding and already-defined stale-generation invalidation +- member runtime event publication and trace/state alignment +- status/reporting updates required to validate the new live producer +- targeted runtime, status, and world-restart tests + +## Out of Scope + +- redefining shared-world owner authority, generation assignment, or Linux reuse behavior already owned by [PLAN-03.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-03.md) +- replacement ordering, rollback, or backend atomic metadata work already owned by [07-world-replacement-ordering-rollback-atomic-metadata.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/07-world-replacement-ordering-rollback-atomic-metadata.md) +- broader authority/cutover cleanup slices +- reopening the live-state authority ladder or compatibility-bridge posture already frozen by [PLAN-09.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-09.md) +- reopening shell-owned event-emission authority cleanup already landed by [PLAN-08.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-08.md) +- a generalized `/v1/agents` service or new top-level agent-hub daemon +- public dispatch UX, scheduler design, or speculative multi-member productization +- repurposing toolbox or gateway lifecycle as a substitute for member session ownership +- host-scoped member product work unless strictly needed for shared helper reuse + +## Blockers And Gaps + +1. There is no production member selection or realizability seam. + - `validate_orchestrator_selection(...)` and `validate_runtime_realizability(...)` are orchestrator-oriented today. + - The doctor surface already checks whether required world-scoped member backends are allowlisted and whether a world boundary exists, but runtime startup does not yet consume that posture as a member launch preflight. + +2. There is no production runtime registry shape for member launches. + - `build_gateway_for_descriptor(...)` can build a gateway for one descriptor, but there is no shell-owned member runtime handle object, gateway cache, or launch helper that binds descriptor plus world placement plus orchestration lineage together. + +3. Constructors are not a production launch seam. + - `new_member_participant(...)` and `new_replacement_participant(...)` in [session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs) are valuable model helpers, but they currently prove only that JSON can be shaped correctly. + - They do not start a backend, own an event stream, persist lifecycle transitions, or produce authoritative live state by themselves. + +4. `async_repl.rs` has no sibling member bootstrap path. + - There is no `prepare_member_runtime_startup(...)`, `start_member_runtime(...)`, or `shutdown_member_runtime(...)`. + - The only long-lived runtime control loop today is the host orchestrator control turn. + +5. The status/store contract is ahead of the real runtime producer. + - `list_live_sessions()`, invalidation helpers, and status fallback suppression already assume world-scoped member participants can be created and retired authoritatively. + - Until production launch exists, those paths are only partially exercised through tests and fixtures. + +6. Restart invalidation exists, but replacement launch does not. + - [05-restart-invalidation-semantics.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/05-restart-invalidation-semantics.md) defines how stale members must stop being live. + - This slice must supply the missing replacement producer that launches the successor participant on the current generation. + +## Required Semantics And Invariants + +### 1. Launch preconditions + +A world-scoped member launch must fail closed unless all of the following are true: + +- an active `OrchestrationSessionRecord` exists +- the parent live orchestrator participant exists and is authoritative-live +- the parent session has authoritative `world_id` plus `world_generation` +- the selected member inventory item resolves to `execution.scope=world` +- the selected member inventory item resolves to `protocol=uaa.agent.session` +- the selected member runtime is realizable for the current platform/build +- the selected member derived `backend_id` is allowlisted by effective policy + +Missing world binding, missing live parent orchestrator, or denied backend must prevent launch. No synthetic fallback to host execution is allowed. + +### 2. Member identity is session-local and world-bound + +Every launched member participant must publish: + +- the same `orchestration_session_id` as its parent session +- a fresh hub-assigned `participant_id` +- `role=member` +- `execution.scope=world` +- the selected member `agent_id` +- the selected member `backend_id` +- `orchestrator_participant_id=` +- `world_id` plus `world_generation` copied from the already-authoritative parent session binding +- `ownership_mode=MemberRuntime` + +For replacement after restart: + +- the replacement participant must use a fresh `participant_id` +- `resumed_from_participant_id` must point at the retired participant +- the replacement participant must bind to the new active `world_generation` + +### 3. Constructor output is not live authority + +Creating a `member` participant record is not enough to advertise liveness. + +Required rule: + +- a member participant may be persisted in `allocating`, but it does not become authoritative-live until the runtime has acquired a usable backend session handle, retained ownership, and started the event/completion observers required by the runtime contract + +This must match the rigor already used by the host orchestrator path. + +### 4. Lifecycle parity with the host runtime + +The member seam must own the same core lifecycle responsibilities already implemented for the host orchestrator: + +- state transition from `allocating` to `ready` +- transition to `running` while work is active if the member protocol needs it +- heartbeat and last-event updates +- terminal `failed`, `stopped`, or `invalidated` transitions +- persisted snapshots on every meaningful lifecycle edge +- publication of runtime events that reflect the persisted participant state + +This does not require the member path to duplicate the host orchestrator UX. It does require the same persistence and state-machine discipline. + +### 5. Shared-world generation is consumed, not invented + +This slice must not generate or reinterpret `world_generation`. + +Required rule: + +- member launch always consumes the current authoritative binding already persisted on the orchestration session +- member replacement after a restart consumes the latest binding after the world replacement is already committed +- stale-generation invalidation continues to use the existing state-store invalidation helper and restart semantics already defined upstream + +### 6. Status and trace contracts + +Once a real member runtime exists: + +- live member participants must appear in `substrate agent status` through the runtime store, not only through trace replay +- world-scoped member rows must publish top-level `world_id` plus `world_generation` +- invalidated members must not be resurrected through trace fallback +- toolbox selection remains anchored to the live orchestrator session and does not switch to a member runtime + +### 7. Persistence and authority boundary + +The member-launch seam must consume the live-state authority and write-ownership rules already frozen upstream. + +Required rules: + +- member launch persists parent plus participant state only through store-owned helpers and the existing snapshot writer choke points +- member launch must treat canonical session-root parent plus participant records as the only authoritative live-state target +- member launch must not directly write flat compatibility parent, participant, or lease files from callers +- member launch must not directly write or depend on legacy `handles/*.json` +- compatibility bridge behavior, if still present in the store, remains an implementation detail of store-owned persistence rather than a contract of the member-launch seam + +### 8. Gateway boundary remains separate + +Nested gateway lifecycle and member runtime lifecycle are different contracts. + +Required rule: + +- `substrate_gateway` records remain nested runtime evidence +- they do not establish member participant liveness +- a launched member runtime must emit and persist its own pure-agent participant identity regardless of whether nested gateway usage occurs later + +## Recommended Implementation Shape + +The first landed production slice should be a bounded shell-owned launch primitive, not a scheduler. + +Recommended shape: + +1. Add a member-selection helper next to existing runtime validation in [validator.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/validator.rs). +2. Add a shell-owned member runtime launch helper in [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) or a narrowly scoped sibling module under `crates/shell/src/execution/agent_runtime/`. +3. Reuse the existing Unified Agent API control contract already used by the host orchestrator path. + - The member seam should launch through the same gateway/control abstraction family as the orchestrator seam. + - Do not invent a second member-only backend protocol. +4. Make that helper accept: + - a `RuntimeOrchestrationContext` + - the selected member `RuntimeSelectionDescriptor` + - the live parent orchestrator participant identity + - the active authoritative world binding +5. Make the helper return a member runtime handle object that owns: + - participant record mutex or snapshot path + - cancel handle + - event stream task + - completion observer task + - enough metadata to support restart replacement and explicit shutdown + +This keeps the seam concrete without forcing a speculative agent-hub service or public dispatch API in the same slice. + +## Exact Code Areas + +### Primary implementation surfaces + +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) + - add the production member runtime bootstrap and teardown seam + - wire it to the existing world-binding and restart flow + - ensure replacement launches consume the already-updated parent binding + +- [crates/shell/src/execution/agent_runtime/validator.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/validator.rs) + - add member-specific realizability and capability validation + - keep orchestrator validation separate from member validation instead of overloading one function + +- [crates/shell/src/execution/agent_runtime/registry.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/registry.rs) + - extend gateway/registry creation so production member launch can resolve a backend from a member descriptor, not only the selected host orchestrator + +- [crates/shell/src/execution/agent_runtime/session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs) + - keep `member` constructor and replacement helpers authoritative for record shape + - add any small lifecycle helpers needed by the new runtime seam + - remove the practical dead-code status by giving these constructors a real production caller + +- [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) + - add any lookup helpers needed by the member launch path + - preserve session-centric persistence as the live authority boundary + - preserve store-owned compatibility bridging as an internal persistence concern instead of a caller-owned write path + - keep invalidation session-local and generation-aware + +### Consumer validation surfaces + +- [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs) + - confirm live status projections render real member participants correctly + - confirm toolbox remains orchestrator-local + - keep doctor checks aligned with the real launch preconditions + +### Context surfaces that should not absorb this slice + +- [crates/shell/src/builtins/world_gateway.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/builtins/world_gateway.rs) + - do not use as the member runtime lifecycle owner +- [crates/world/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world/src/lib.rs) +- [crates/world-agent/src/service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs) +- [crates/world-agent/src/pty.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/pty.rs) + - these remain upstream authority/provider seams; this SOW consumes them rather than redesigning them + +## Testing Requirements + +At minimum, land tests that prove real production member launch rather than fixture-only shape. + +### Runtime/unit coverage + +- add `async_repl.rs` unit coverage for: + - successful member bootstrap under an active orchestrator session + - failed member bootstrap when no world binding exists + - failed member bootstrap when the parent orchestrator is missing or not live + - replacement member bootstrap after generation change using fresh lineage + - member shutdown or unexpected completion persisting terminal state correctly + +### Store/status integration coverage + +- extend [crates/shell/tests/agent_successor_contract_ahcsitc0.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_successor_contract_ahcsitc0.rs) so that: + - a real live member participant shows in `agent status` + - world-scoped member rows publish `world_id` plus `world_generation` + - invalidated member tombstones still suppress trace fallback + - toolbox still resolves only through the orchestrator session + +### World/restart integration coverage + +- extend [crates/shell/tests/repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs) so that: + - a startup path with shared-world binding can also launch a member runtime + - world restart produces a replacement member bound to the new generation + - old-generation members are not left live after replacement + - fail-closed restart posture refuses new member launch when replacement is not ready + +### Trace persistence coverage + +- extend [crates/shell/tests/agent_hub_trace_persistence.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_hub_trace_persistence.rs) so that: + - member runtime events persist with pure-agent identity + - member rows preserve world binding and participant lineage + - member terminal transitions remain auditable without becoming authoritative-live again + +### Recommended verification commands + +```bash +cargo test -p shell agent_successor_contract_ahcsitc0 -- --nocapture +cargo test -p shell repl_world_first_routing_v1 -- --nocapture +cargo test -p shell agent_hub_trace_persistence -- --nocapture +cargo test -p shell async_repl -- --nocapture +``` + +## Acceptance Criteria + +- production shell code can launch at least one world-scoped member runtime under a live orchestration session +- launch requires a live host orchestrator participant plus authoritative parent world binding +- the launched member writes an authoritative participant record under the session-centric runtime store +- member liveness depends on a real runtime ownership boundary, not only a constructor call +- `substrate agent status` can surface the live member from runtime state with `world_id` plus `world_generation` +- invalidated old-generation members remain non-live after replacement +- toolbox behavior stays orchestrator-scoped and unchanged except for consuming the same parent session authority +- no part of the implementation redefines world ownership, restart ordering, or backend rollback semantics already owned by earlier slices + +## Recommended Staged Execution Order + +1. Member preflight and descriptor resolution. + - Add member-specific validation for scope, protocol, runtime realizability, and policy allowlisting. + +2. Runtime launch abstraction. + - Introduce the shell-owned member runtime handle and gateway/descriptor plumbing needed to start a real backend. + +3. Participant bootstrap and persistence. + - Allocate the member participant, persist `allocating`, bind it to the active orchestrator and world generation, and transition to authoritative-live only after ownership is retained. + +4. Lifecycle and replacement integration. + - Add shutdown, unexpected-exit, and restart-replacement handling. + - Consume existing invalidation helpers rather than redefining cutover semantics here. + +5. Status and trace validation. + - Prove that live status, tombstone suppression, and trace persistence all work with a real member producer. + +6. Integration hardening. + - Add world-restart coverage, replacement lineage coverage, and fail-closed launch coverage. + +## Main Contract Decisions + +- The repo currently launches only the host orchestrator runtime in production; member constructors and store helpers do not count as a real launch seam. +- The first required capability expansion is one bounded shell-owned world-member launch primitive, not a generalized orchestration service. +- Member launch must consume already-authoritative world/session state and must stay isolated from authority, rollback, event-emission-authority, and cleanup/cutover slices. diff --git a/llm-last-mile/11-in-world-member-dispatch-over-existing-host-world-transport.md b/llm-last-mile/11-in-world-member-dispatch-over-existing-host-world-transport.md new file mode 100644 index 000000000..7b038aef0 --- /dev/null +++ b/llm-last-mile/11-in-world-member-dispatch-over-existing-host-world-transport.md @@ -0,0 +1,518 @@ +# SOW: In-World Member Dispatch Over Existing Host<->World Transport + +Status: implementation-oriented draft. This document defines the remaining member-runtime placement slice after the shell-side launch, persistence, status, and replacement semantics from [10-member-runtime-launch-seam.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/10-member-runtime-launch-seam.md) are already present in production code. It is intentionally bounded to moving world-scoped member execution from the shell process into `world-agent` over the existing host<->world execute/stream transport. It does not reopen shared-world ownership, world-generation authority, session-centric live-state authority, shell-side replacement semantics, or broad UAA adoption. + +## Objective + +Land one Linux-first production path where a world-scoped member runtime is started inside `world-agent`, not inside the shell process, while preserving the already-landed shell authority model: + +- the shell remains the authoritative owner of orchestration session state, +- the shell still persists the canonical parent plus participant records, +- the shell still applies invalidation and replacement semantics, +- but the actual member `AgentWrapperGateway.run_control(...)` call moves into `world-agent`, +- and the shell drives that runtime through the existing `/v1/execute/stream` plus `/v1/execute/cancel` transport. + +The required outcome is not a second agent hub. The required outcome is that the existing world-scoped member participant model becomes honest about placement: member execution actually runs inside the active world boundary that its participant record claims to be bound to. + +## Why This Is Needed + +The main launch/lifecycle seam from [10-member-runtime-launch-seam.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/10-member-runtime-launch-seam.md) is now effectively landed in the shell runtime: + +- shell-owned UAA orchestrator startup is real, +- session-centric runtime state is real, +- authoritative `world_id` plus `world_generation` plumbing is real, +- stale-generation invalidation is real, +- replacement-member lineage across world rollover is real, +- and `substrate agent status` already consumes real member participants. + +The remaining mismatch is narrower: + +- `prepare_member_runtime_startup_for_descriptor(...)` and `start_member_runtime_with_prepared(...)` exist in [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs), +- but `start_member_runtime_with_prepared(...)` still delegates to `start_host_orchestrator_runtime_with_prepared(...)`, +- which still calls `gateway.run_control(...)` inside the shell process, +- so the member is only correlated to a world binding in state, +- not actually launched inside `world-agent` over the host<->world transport. + +That is now the last major placement gap called out in [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md). + +## Relationship To Landed Slices + +This slice depends on, and must consume without redesign: + +- [03-shared-world-ownership-linux-first.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/03-shared-world-ownership-linux-first.md) +- [04-thread-world-binding-into-runtime-state.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/04-thread-world-binding-into-runtime-state.md) +- [05-restart-invalidation-semantics.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/05-restart-invalidation-semantics.md) +- [07-world-replacement-ordering-rollback-atomic-metadata.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/07-world-replacement-ordering-rollback-atomic-metadata.md) +- [08-explicit-orchestration-authority-event-emission.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/08-explicit-orchestration-authority-event-emission.md) +- [09-live-state-authority-and-compatibility-cutover.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/09-live-state-authority-and-compatibility-cutover.md) +- [10-member-runtime-launch-seam.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/10-member-runtime-launch-seam.md) + +Non-goal reminder: + +- this slice does not re-decide member selection, +- does not re-decide replacement ordering, +- does not re-decide who owns `world_generation`, +- and does not broaden the architecture into a new public `agent` service surface. + +## Current Repo Seams And Exact Code Surfaces + +### Shell-side member runtime seam already exists + +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) + - `prepare_member_runtime_startup_for_descriptor(...)` + - `start_member_runtime_with_prepared(...)` + - `ensure_member_runtime_ready(...)` + - `reconcile_member_runtime_generation(...)` + - `shutdown_host_orchestrator_runtime(...)` +- The critical current-reality fact is that: + - `start_member_runtime_with_prepared(...)` currently calls `start_host_orchestrator_runtime_with_prepared(...)` + - and `start_host_orchestrator_runtime_with_prepared(...)` is where `gateway.run_control(...)` happens + - so member launch still uses the shell-owned local UAA runtime path. + +### Shell authority and state seams already exist + +- [crates/shell/src/execution/agent_runtime/session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs) + - `new_member_participant(...)` + - `new_replacement_participant(...)` + - `can_advertise_live()` + - `is_authoritative_live()` + - `mark_runtime_ownership_retained()` +- [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) + - `persist_participant(...)` + - `persist_orchestration_session(...)` + - `resolve_live_orchestrator_participant(...)` + - `list_live_participants_for_session(...)` + - `invalidate_stale_world_members_for_session(...)` +- [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs) + - `build_status_report(...)` + - `build_toolbox_status_report(...)` + +These are not missing. This slice must keep them authoritative. + +### Existing host<->world transport already exists + +- [crates/agent-api-types/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/agent-api-types/src/lib.rs) + - `ExecuteRequest` + - `ExecuteStreamFrame` + - `ExecuteCancelRequestV1` + - `ExecuteCancelResponseV1` +- [crates/agent-api-client/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/agent-api-client/src/lib.rs) + - `execute_stream(...)` + - `cancel_execute(...)` +- [crates/shell/src/execution/routing/dispatch/world_ops.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs) + - `build_agent_client_and_request_with_trace_metadata(...)` + - `stream_non_pty_via_agent(...)` + - `process_agent_stream_body(...)` +- [crates/world-agent/src/handlers.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/handlers.rs) + - `execute_stream(...)` + - `execute_cancel(...)` +- [crates/world-agent/src/service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs) + - `execute_stream(...)` + - `execute_cancel(...)` + +This slice must reuse that transport rather than creating a separate top-level control API. + +### World-agent already has one relevant runtime-management pattern + +- [crates/world-agent/src/gateway_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/gateway_runtime.rs) + - `GatewayRuntimeManager` + - `GatewayRuntimeStartContext` + - runtime manifest persistence + - pid rediscovery + - cgroup attach + - ready/restart lifecycle +- [crates/world-agent/src/service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs) + - `prepare_gateway_runtime_request(...)` + - `resolve_gateway_runtime_binding(...)` + +That code is not the member-launch surface, but it is a strong implementation reference for Linux-first world-owned long-lived runtime management. + +## In Scope + +- moving production world-scoped member execution into `world-agent` +- reusing `/v1/execute/stream` plus `/v1/execute/cancel` as the transport seam +- adding the typed request metadata needed to tell `world-agent` to launch a member runtime instead of a shell command +- forwarding member lifecycle and wrapper events back to the shell over existing stream frames +- keeping the shell as the authoritative writer of session-root live state +- preserving already-landed replacement-member lineage and invalidation behavior while moving the actual replacement launch into the world transport path +- Linux-first implementation and validation + +## Out Of Scope + +- new shared-world ownership or rollback logic +- new session-root schema or new compatibility-cutover rules +- reworking the host orchestrator runtime into a world-owned service +- broad `substrate agent start|resume|fork|stop` CLI productization +- a new `/v1/member/*` or `/v1/agents/*` public API family +- toolbox redesign +- macOS and Windows parity beyond fail-closed posture +- changing the already-landed selection rule except where transport serialization needs a stable descriptor shape + +## Blockers And Gaps + +1. Member runtime placement is still wrong. + - The shell allocates correct member participants, but the actual UAA control turn still lives in the shell process. + +2. The transport has no typed member-dispatch request today. + - `ExecuteRequest` only models process execution. + - There is no explicit payload for "start this world-scoped member runtime under attached control." + +3. `world-agent` has no member-runtime manager today. + - There is no world-owned equivalent of the shell path that builds a UAA gateway, starts `run_control(...)`, retains cancel ownership, forwards events, and observes completion for a member session. + +4. `/v1/execute/cancel` only knows how to cancel process exec spans today. + - Member dispatch needs the same cancel endpoint to reach a retained world-owned UAA session, not just a process group. + +5. The shell member path is structurally coupled to the local orchestrator launcher. + - Reusing `start_host_orchestrator_runtime_with_prepared(...)` for member launch made sense for the first shell-only slice. + - It is now the exact coupling that must be broken. + +6. Current tests prove state semantics, not placement semantics. + - Existing shell tests prove member records, replacement lineage, and status. + - They do not yet prove that the member binary actually starts inside `world-agent` over the transport boundary. + +## Required Semantics And Invariants + +### 1. Shell authority remains authoritative + +This slice does not move live-state authority into `world-agent`. + +Required rule: + +- `world-agent` owns in-world member process execution and in-world UAA control retention, +- the shell owns canonical `AgentRuntimeStateStore` writes, +- the shell remains the authority for participant state transitions exposed through `substrate agent status`, +- and the shell remains the owner of stale-generation invalidation and replacement decisions. + +`world-agent` may emit events and terminal outcomes; it must not become the writer of canonical session-root state. + +### 2. No host-side fallback for world-scoped members + +Once a selected runtime resolves to `execution.scope=world`, the shell must not silently fall back to local `gateway.run_control(...)`. + +Required rule: + +- if world transport member dispatch cannot be established, member startup fails closed, +- the participant remains non-live or transitions terminally failed, +- and the shell must not "helpfully" launch the member locally. + +### 3. Launch must consume the authoritative active world binding + +The shell must dispatch against the already-authoritative active shared world from the orchestration session snapshot. + +Required data carried into member dispatch: + +- `orchestration_session_id` +- `participant_id` +- `orchestrator_participant_id` +- `resumed_from_participant_id` when replacement applies +- `agent_id` +- `backend_id` +- backend/runtime selector sufficient for `world-agent` to build the correct UAA backend +- authoritative `world_id` +- authoritative `world_generation` +- caller-owned `run_id` + +The world side must consume those values. It must not invent a different `world_id`, `world_generation`, or lineage. + +### 4. The existing transport endpoints remain the seam + +Linux-first member dispatch must ride: + +- `POST /v1/execute/stream` for launch plus attached-control streaming +- `POST /v1/execute/cancel` for cancellation +- existing `ExecuteStreamFrame::Start`, `Event`, `Exit`, and `Error` frames for control-plane feedback + +The implementation may extend request payloads and may add additive event contents, but it must not require a new top-level world-agent endpoint for the first shipped slice. + +### 5. Readiness must remain strict + +The shell may persist a member participant in `allocating`, but it may only advertise it authoritative-live after all of the following have occurred: + +- `world-agent` has successfully started the member backend inside the active world, +- a real UAA session handle has surfaced, +- remote cancel ownership is retained, +- remote event streaming is active, +- remote completion observation is retained, +- and the shell has persisted the updated participant snapshot. + +This is the same live-authority bar already enforced by `can_advertise_live()` and `is_authoritative_live()`. + +### 6. World replacement semantics stay the same; only placement changes + +When `world_generation` rolls forward: + +- stale member participants still invalidate in the shell store, +- the predecessor still remains auditable, +- the replacement participant still receives a fresh `participant_id`, +- `resumed_from_participant_id` still points at the predecessor, +- but the replacement runtime must now be launched through `world-agent`, not in the shell. + +### 7. Event and trace identity must stay participant-correct + +Remote member lifecycle events forwarded from `world-agent` must preserve: + +- `orchestration_session_id` +- `participant_id` +- `parent_participant_id` when present +- `resumed_from_participant_id` when present +- `role=member` +- `agent_id` +- `backend_id` +- top-level `world_id` +- top-level `world_generation` +- caller-owned `run_id` + +The transport must not flatten those into anonymous shell command rows. + +### 8. Cancel and terminal handling must remain fail-closed + +If the shell asks `world-agent` to cancel a member dispatch span and delivery fails: + +- the shell must not keep advertising the participant as safely live forever, +- shutdown must surface failure, +- and the participant must converge to `stopped`, `failed`, or `invalidated` based on what is actually observed. + +### 9. Linux-first posture is explicit + +For the first shipped slice: + +- Linux is the required implementation target, +- non-Linux world-scoped member dispatch may deny as unavailable, +- and non-Linux paths must fail closed instead of routing the member locally. + +## Recommended Implementation Shape + +### 1. Split local orchestrator launch from remote world-member launch + +Keep the current shell-owned orchestrator path intact. + +Change the member path so that: + +- `prepare_member_runtime_startup_for_descriptor(...)` returns a member-dispatch preparation object, not a local `AgentWrapperGateway`, +- `start_member_runtime_with_prepared(...)` opens a long-lived world-agent execute stream, +- and the shell uses that stream as the retained attached-control boundary for the member runtime. + +The shell should still allocate and persist the participant record before remote launch so the existing authority model remains intact. + +### 2. Extend `ExecuteRequest` with a typed member-dispatch payload + +Recommended transport shape: + +- keep `/v1/execute/stream`, +- keep `ExecuteStreamFrame`, +- add one additive typed request payload such as `member_dispatch: Option` under `ExecuteRequest`, +- and branch inside `world-agent` on that field. + +Do not encode member launch as a magic shell command string. + +Minimum member-dispatch payload should include: + +- orchestration and participant identity +- member backend selection data +- lineage data +- run correlation data +- authoritative world binding data + +The top-level `ExecuteRequest` can continue to carry: + +- `cwd` +- `env` +- `policy_snapshot` +- `shared_world` +- `world_network` + +so the request still uses the existing world-routing and policy-input machinery. + +### 3. Implement a Linux-first world-agent member runtime manager + +Add a dedicated world-agent runtime manager modeled after the rigor of `gateway_runtime.rs`, but specialized for pure-agent member control: + +- build the UAA backend inside `world-agent` +- start `run_control(...)` +- retain cancel ownership +- forward wrapper events as `ExecuteStreamFrame::Event` +- emit terminal `Exit` or `Error` +- keep cancellation addressable by the transport `span_id` +- and tear down cleanly on completion or cancel + +This manager should be internal to `world-agent`. It does not need a public status API in the first slice. + +### 4. Reuse the existing shell lifecycle helpers wherever possible + +The shell already knows how to: + +- persist `allocating` +- mark retained ownership +- translate runtime events into canonical `AgentEvent` +- publish ready/status/invalidated alerts +- and persist heartbeat snapshots + +Do not duplicate that logic in a second shell-local state machine. Parameterize or extract the generic pieces from [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) so both: + +- the local orchestrator path, and +- the remote in-world member path + +share the same persistence and event-translation discipline. + +### 5. Reuse `/v1/execute/cancel` rather than inventing a second cancel plane + +The world-agent side should register member dispatch spans in a cancel registry that `/v1/execute/cancel` can reach. + +Recommended shape: + +- keep process-exec cancellation behavior as-is, +- add a member-dispatch cancellation branch keyed by the same `span_id`, +- and let the shell continue treating cancel as one transport operation. + +### 6. Keep the first slice intentionally narrow + +The first landed slice should support: + +- one shell-owned orchestrator +- one active world-scoped member per selected descriptor per active world generation +- replacement on generation rollover +- cancel/stop through the same transport + +Do not combine this with multi-member scheduling or public session-control UX work. + +## Exact Code Areas To Change + +### Shell + +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) + - stop routing member launch through `start_host_orchestrator_runtime_with_prepared(...)` + - introduce a remote member dispatch startup path + - adapt shutdown and replacement handling to the remote transport-backed control boundary + - reuse existing lifecycle persistence and runtime-event helpers + +- [crates/shell/src/execution/routing/dispatch/world_ops.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs) + - add a request-builder/helper for member dispatch over existing world transport + - reuse existing world client creation, policy snapshot injection, and shared-world routing inputs + - add any stream-consumption helper needed for non-command member lifecycle frames + +- [crates/shell/src/execution/agent_runtime/session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs) + - only if needed for additive remote-control metadata + - do not redesign role, lineage, or world-binding fields + +- [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs) + - validate that status/doctor output remains honest for the remote member path + - add no new selection semantics here + +### Transport + +- [crates/agent-api-types/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/agent-api-types/src/lib.rs) + - add the typed member-dispatch request payload carried over `ExecuteRequest` + - add any additive validation helpers needed for that payload + - keep `ExecuteStreamFrame` compatible with existing process-exec callers + +- [crates/agent-api-client/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/agent-api-client/src/lib.rs) + - only additive request-construction helpers if needed + - no new transport family required + +### World agent + +- [crates/world-agent/src/service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs) + - branch `execute_stream(...)` for typed member dispatch + - wire the member runtime manager + - integrate cancel delivery for remote member spans + +- [crates/world-agent/src/handlers.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/handlers.rs) + - input validation only as required by the additive request schema + +- [crates/world-agent/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/lib.rs) + - register any new internal module wiring only + +- recommended new module: + - [crates/world-agent/src/member_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs) + - purpose: + - UAA backend build/start + - retained cancel/event/completion ownership + - stream-frame emission + - cancel integration + - Linux-first lifecycle cleanup + +### Tests + +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) + - update unit coverage for member startup and replacement to assert remote transport usage rather than local gateway reuse + +- [crates/shell/tests/repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs) + - assert member launch occurs through world transport on Linux-first world-backed paths + +- [crates/shell/tests/agent_hub_trace_persistence.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_hub_trace_persistence.rs) + - assert remote member rows still preserve top-level world identity and replacement lineage + +- [crates/shell/tests/agent_successor_contract_ahcsitc0.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_successor_contract_ahcsitc0.rs) + - assert status/doctor behavior remains fail-closed and participant-aware + +- Linux-first world-agent tests: + - [crates/world-agent/src/service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs) + - [crates/world-agent/tests/](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/tests/) + - add focused tests for member dispatch startup, event forwarding, cancellation, and terminal cleanup + +## Testing, Acceptance Criteria, And Failure Cases + +### Acceptance criteria + +1. With one live shell-owned orchestrator and one eligible world-scoped member, the first world-backed REPL command causes the member to be launched through `world-agent`, not through local shell `gateway.run_control(...)`. +2. The member participant is first persisted in `allocating`, then becomes authoritative-live only after a real UAA session handle is surfaced from the world-owned runtime. +3. `substrate agent status` shows the live member with correct `participant_id`, `role=member`, `world_id`, and `world_generation`. +4. After world-generation rollover, the prior member becomes invalidated and a replacement member is launched through the same world transport with correct `resumed_from_participant_id`. +5. Shutting down the REPL or cancelling the member dispatch uses `/v1/execute/cancel` and leaves the member in a terminal non-live state. +6. Trace rows and canonical `AgentEvent` rows for the remote member preserve top-level `world_id` and `world_generation`. + +### Required failure cases + +1. Ambiguous or missing eligible member selection must fail before any transport call is issued. +2. Backend deny by policy must fail before any transport call is issued. +3. Missing authoritative world binding must fail before any transport call is issued. +4. If the world-agent-side UAA backend cannot start, the participant must end failed and must never be advertised live. +5. If the remote stream closes before ownership is established, the participant must end failed, not live. +6. If the remote stream closes after readiness but before clean completion, the participant must end invalidated unless a normal stop path already won the race. +7. If cancel delivery cannot be made, shutdown must surface failure and the participant must not remain authoritatively live indefinitely. +8. On non-Linux or unavailable world transport, world-scoped member startup must fail closed rather than running locally. + +### Validation commands + +Minimum validation for implementation work on this slice should include: + +```bash +cargo test -p world-agent -- --nocapture +cargo test -p shell --test repl_world_first_routing_v1 -- --nocapture +cargo test -p shell --test agent_hub_trace_persistence -- --nocapture +cargo test -p shell --test agent_successor_contract_ahcsitc0 -- --nocapture +``` + +If the implementation touches shared-world restart behavior or cancellation internals, add the relevant Linux doctor/smoke evidence already required by the repo guidelines. + +## Sequencing And Rollout Notes + +1. Land transport typing first. + - Add the typed member-dispatch payload and its validation without changing the orchestrator path. + +2. Land Linux-first world-agent member runtime management next. + - Keep it internal to `world-agent`. + - Reuse existing execute-stream and cancel entrypoints. + +3. Switch the shell member-launch path after the world-agent path exists. + - The shell should stop building a local member gateway at that point. + +4. Keep the host orchestrator path unchanged in this slice. + - The orchestrator remains shell-owned. + +5. Treat macOS and Windows as follow-on parity work. + - The first slice may deny world-scoped member dispatch there if the in-world runtime path is not yet implemented. + +## Open Choices Left By Current Repo Reality + +1. Whether to factor UAA backend registration into a shared helper used by both shell and `world-agent`, or to keep two thin backend builders with identical supported-backend coverage. + - Shared extraction is cleaner. + - A duplicated first slice may still be acceptable if tightly bounded and tested. + +2. Whether the member-dispatch extension hangs directly off `ExecuteRequest` or uses a tagged execute-target enum. + - The first option is smaller. + - The second is structurally cleaner if more transport variants are expected soon. + +3. Whether the world-agent-side cancel registry should extend the existing exec-span registry or live beside it as a transport-span multiplexer. + - Either is valid as long as `/v1/execute/cancel` remains the only caller-facing cancel seam for this slice. diff --git a/llm-last-mile/13-member-runtime-world-placement-gap-sow.md b/llm-last-mile/13-member-runtime-world-placement-gap-sow.md new file mode 100644 index 000000000..ae18cd3d1 --- /dev/null +++ b/llm-last-mile/13-member-runtime-world-placement-gap-sow.md @@ -0,0 +1,357 @@ +# SOW: Finish True In-World Placement For World-Scoped Member Dispatch + +Status: implementation-oriented draft. This SOW covers the remaining correctness gap after the transport cutover work from [11-in-world-member-dispatch-over-existing-host-world-transport.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/11-in-world-member-dispatch-over-existing-host-world-transport.md). The shell now launches world-scoped members through `world-agent` over the existing host<->world transport, but the launched member runtime is still not actually placed into the authoritative per-session world boundary. + +## Objective + +Land one Linux-first production path where a world-scoped member runtime is: + +- selected and persisted by the shell as it is today, +- launched by `world-agent` over the existing `/v1/execute/stream` seam as it is today, +- and actually executed inside the active session world rather than merely correlated to that world in metadata. + +The required outcome is not "transport-owned member runtime." The required outcome is "transport-owned and world-placed member runtime." + +## Problem Statement + +The current implementation moved ownership of member startup, event forwarding, and cancellation into `world-agent`, but it stopped short of real world placement. + +Today the flow is: + +1. The shell chooses a world-scoped member, prepares lineage and authoritative world binding, and builds a typed member-dispatch request in [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:3394) and [crates/shell/src/execution/routing/dispatch/world_ops.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs:203). +2. The shell sends that request over the existing world socket in [crates/shell/src/execution/routing/dispatch/world_ops.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs:1176). +3. `world-agent` ensures the authoritative shared world and validates that the request matches the current `world_id` and `world_generation` in [crates/world-agent/src/service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs:1223) and [crates/world-agent/src/service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs:1233). +4. `world-agent` then launches the member by calling `gateway.run_control(...)` directly from `MemberRuntimeManager::launch(...)` in [crates/world-agent/src/member_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs:50). + +The missing step is that `MemberRuntimeManager::launch(...)` never re-enters the session world: + +- it does not execute through `backend.exec(&world, ...)`, +- it does not use the session overlay root, +- it does not attach the child to the session cgroup, +- it does not establish the same world placement guarantees as normal world exec or the isolated gateway-runtime path. + +That means the member runtime is: + +- authoritative in shell state, +- remote-owned for startup and cancel, +- but still not honestly placed inside the world boundary it claims to be bound to. + +## Why This Matters + +This is a correctness issue, not just a documentation issue. + +The current code can truthfully say: + +- the member belongs to `world_id=X`, +- the member belongs to `world_generation=Y`, +- `world-agent` owns the retained control handle, +- `/v1/execute/cancel` can reach it. + +But it cannot yet truthfully say: + +- the member process is actually executing under the same per-session world isolation contract as world commands. + +This is also the remaining unclosed gap called out in [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md:110), and it sits directly on top of the ordered SOW packet in [llm-last-mile/README.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/README.md:23). + +That mismatch undermines: + +- filesystem isolation expectations, +- cgroup accounting expectations, +- network posture expectations, +- replay/trace/operator trust in `world_id` and `world_generation`, +- and the intent captured in [11-in-world-member-dispatch-over-existing-host-world-transport.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/11-in-world-member-dispatch-over-existing-host-world-transport.md:182). + +## Current Relevant Code Surfaces + +### Shell request construction and shell-owned authority + +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:3353) + - `ensure_member_runtime_ready(...)` + - `prepare_member_runtime_startup_for_descriptor(...)` + - `start_remote_member_runtime_with_prepared(...)` +- [crates/shell/src/execution/routing/dispatch/world_ops.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs:203) + - `MemberDispatchTransportRequest` + - `build_member_dispatch_payload(...)` + - `build_agent_client_and_member_dispatch_request_impl(...)` + +These surfaces are already doing the right transport and state-authority work. This SOW does not redesign member selection, lineage, or canonical persistence. + +### World-agent request routing and current gap + +- [crates/world-agent/src/service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs:1233) + - `execute_stream(...)` branches to member dispatch after ensuring the authoritative world. +- [crates/world-agent/src/member_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs:50) + - `MemberRuntimeManager::launch(...)` directly calls `gateway.run_control(...)`. +- [crates/world-agent/src/service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs:1446) + - `execute_cancel(...)` already reaches member runtime spans. + +### World placement reference implementations + +- Normal non-PTY world execution goes through the backend in [crates/world-agent/src/service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs:1290) and [crates/world-agent/src/service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs:1343). +- Gateway runtime uses an explicit isolated runtime binding and session cgroup attach in [crates/world-agent/src/service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs:1580), [crates/world-agent/src/service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs:1601), and [crates/world-agent/src/gateway_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/gateway_runtime.rs:696). + +These are the reference patterns the member-runtime placement fix must reuse or mirror. + +## In Scope + +- make world-scoped member runtime launch actually enter the authoritative session world +- preserve the existing `/v1/execute/stream` and `/v1/execute/cancel` transport seam +- preserve shell ownership of canonical participant/session state +- preserve current replacement, invalidation, and lineage semantics +- preserve current typed member-dispatch request shape unless one additive field is required for placement +- update installer/provisioning paths so shipped/dev-installed `world-agent` behavior matches the new runtime contract +- update docs and targeted tests to prove true placement rather than metadata-only correlation + +## Out Of Scope + +- redesigning orchestrator ownership +- moving runtime selection into `world-agent` +- public `substrate agent start|resume|fork|stop` productization +- cross-platform feature parity beyond explicit Linux-first fail-closed posture +- secret-carrier changes for gateway integrated auth +- toolbox work + +## Required Semantics + +1. Shell authority stays authoritative. +The shell must remain the canonical writer of orchestration-session and participant state. + +2. No host-side fallback for world-scoped members. +If world placement for a selected world-scoped member cannot be established, startup must fail closed. The shell must not silently fall back to a host-local retained-control launch. + +3. World placement must be real, not inferred. +The launched member runtime must receive the same effective world placement guarantees that make `world_id` and `world_generation` meaningful operator-visible facts. + +4. Transport ownership stays in `world-agent`. +The fix must not regress back to a shell-owned member runtime. + +5. Replacement semantics remain shell-owned. +Generation rollover still invalidates stale members and launches replacements via shell authority, but the actual replacement runtime must now be placed into the replacement world. + +6. "Unavailable member runtime" must not silently become "member placement skipped." +The current Linux shell path still prints `world-scoped member runtime unavailable` and returns control to the REPL loop in [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:3405). The final behavior for a selected world-scoped member must be fail-closed once real placement is required. + +## Core Implementation Work + +### 1. Introduce a real world-placement launch path for member runtime + +`MemberRuntimeManager::launch(...)` in [crates/world-agent/src/member_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs:50) must stop treating `gateway.run_control(...)` as sufficient by itself. + +The fix must add one honest placement mechanism for the spawned member runtime that covers at least: + +- authoritative session-world resolution +- session cgroup attachment +- correct world-root / overlay / filesystem view +- correct network posture +- correct retained-control ownership and cancellation retention + +This does not require reusing the exact normal `backend.exec(&world, ...)` API if that API is a poor fit for a long-lived retained-control runtime. It does require equivalent world-placement guarantees. + +### 2. Decide and implement the member-runtime world-entry strategy + +The implementation must choose one sanctioned strategy and document it in code and docs. Acceptable direction: + +- extend `world-agent` with a placement-aware runtime launcher for long-lived retained-control member processes +- reuse the same world-session discovery and cgroup/overlay preparation rules that back the normal exec path +- keep `agent_api` backend construction in `world-agent`, but wrap the actual child launch in world-entry logic rather than direct daemon-local spawn + +The implementation must not leave placement implicit. + +### 3. Thread the required placement context into member runtime launch + +Today `service.rs` validates the world and passes only `cwd`, `env`, `dispatch`, and `binding` into the member-runtime lane in [crates/world-agent/src/service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs:1233). + +That lane will likely need additive context such as: + +- the resolved `WorldHandle` +- the session cgroup path +- overlay root or equivalent resolved world-root placement facts +- any explicit placement mode needed by the launcher + +The exact carrier can be a new internal context struct. The SOW does not require a particular struct name. + +### 4. Keep cancel, event, and completion semantics intact + +The new placement-aware lane must preserve: + +- `ExecuteStreamFrame::Start` +- streamed `Event` +- surfaced `agent_api.session.handle.v1` +- terminal `Exit` and `Error` +- `/v1/execute/cancel` delivery through the same `span_id` + +Nothing in the placement fix should regress the already-landed remote cancel path in [crates/world-agent/src/service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs:1446). + +### 5. Keep shell lifecycle logic unchanged except where placement failures become stricter + +The shell side in [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:3353) should continue to: + +- prepare the member participant +- persist `allocating` +- wait for surfaced session handle +- mark runtime ownership retained +- persist ready/failed/invalidation transitions + +The only intended shell behavior change is that world-scoped member launch should now fail when true world placement cannot be established, instead of merely succeeding with metadata correlation. + +## Installer And Provisioning Work + +This slice is not just a code-path change. It changes the runtime contract of `world-agent`, so install and provisioning flows must be aligned. + +### Why installer work is required + +There are multiple independent authors of the effective `world-agent` runtime environment: + +- dev install on Linux/macOS in [scripts/substrate/dev-install-substrate.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/substrate/dev-install-substrate.sh:1482) +- production install on Linux in [scripts/substrate/install-substrate.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/substrate/install-substrate.sh:1932) +- Linux provisioning helper in [scripts/linux/world-provision.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/linux/world-provision.sh:535) +- macOS Lima guest provisioning in [scripts/mac/lima-warm.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/lima-warm.sh:659) + +Any new world-agent placement dependency that only lands in one of those paths will produce drift between: + +- source checkout dev installs, +- release installs, +- direct Linux provisioning, +- and macOS guest provisioning. + +There is already observable drift today: + +- the release installer unit in [scripts/substrate/install-substrate.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/substrate/install-substrate.sh:1977) includes `SUBSTRATE_HOME` and a home-path `ReadWritePaths` entry, +- while the Linux provisioning helper in [scripts/linux/world-provision.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/linux/world-provision.sh:564) includes `CAP_CHOWN`, +- and the WSL provisioning chain still writes `Group=root` and `SocketGroup=root` in [scripts/wsl/provision.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/wsl/provision.sh:37). + +This SOW therefore treats installer alignment as part of the runtime correctness work, not as optional cleanup. + +### Minimum installer/provisioning update set + +The SOW requires review and update of both user-named install paths: + +- [scripts/substrate/dev-install-substrate.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/substrate/dev-install-substrate.sh:1482) +- [scripts/substrate/install-substrate.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/substrate/install-substrate.sh:1932) + +And it requires lockstep review of the underlying provisioning writers they depend on: + +- [scripts/linux/world-provision.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/linux/world-provision.sh:535) +- [scripts/mac/lima-warm.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/lima-warm.sh:659) + +And it requires an explicit decision about the WSL path: + +- [scripts/windows/wsl-warm.ps1](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/windows/wsl-warm.ps1:248) +- [scripts/wsl/provision.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/wsl/provision.sh:37) + +The WSL lane does not need Linux-first feature parity for this slice, but it does need an explicit posture. If the placement fix changes unit/runtime requirements, either: + +- WSL is updated in the same change, +- or the implementation documents and enforces a fail-closed unsupported posture there until the WSL provisioning chain is aligned. + +At minimum, those paths must be checked and updated for any new: + +- installed binary or helper artifact needed by member placement +- systemd unit env vars +- `ReadWritePaths` +- `CapabilityBoundingSet` +- `AmbientCapabilities` +- runtime directories +- staging/caching of Linux guest binaries under dev-install + +Even if the final implementation does not need a new binary, the service-unit duplication still requires explicit review so the runtime assumptions remain aligned. + +### Specific installer concerns to cover in the implementation + +1. `dev-install-substrate.sh` +- must keep the staged Linux guest bundle honest if the member-placement fix changes the Linux `world-agent` runtime requirements or artifacts +- must keep Linux provisioning and macOS Lima provisioning in sync +- must continue to disable world mode fail-closed when provisioning cannot establish the required runtime + +2. `install-substrate.sh` +- must ship any new runtime artifact required for member placement +- must keep the generated `substrate-world-agent.service` and `.socket` consistent with the placement fix +- must not preserve an older capability set if the new path needs additional sanctioned world-placement privileges + +3. `world-provision.sh` and `lima-warm.sh` +- must mirror the same unit-level runtime contract as the two higher-level installers +- must remain the authoritative service-unit templates for Linux host and macOS guest deployment + +4. WSL provisioning +- must not be left silently stale if the final member-placement fix depends on service-unit contract changes +- must either adopt the same relevant runtime contract or be explicitly gated out by product posture and documentation + +## Testing And Validation + +### Existing tests that should remain green + +- [crates/world-agent/tests/streamed_execute_cancel_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/tests/streamed_execute_cancel_v1.rs:186) +- [crates/shell/tests/repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs:1576) +- [crates/shell/tests/repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs:1682) +- [crates/shell/tests/repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs:2124) +- [crates/shell/tests/repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs:2268) +- [crates/shell/tests/agent_successor_contract_ahcsitc0.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_successor_contract_ahcsitc0.rs:3907) +- [crates/shell/tests/agent_hub_trace_persistence.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_hub_trace_persistence.rs:941) + +These already cover lazy launch, same-generation reuse, replacement launch, replacement failure, remote cancellation semantics, and the higher-level status/trace surfaces that consume member-runtime truth. + +### New tests required for this SOW + +Add focused proof that the member runtime is actually placed into the world, not just tagged with world metadata. + +At least one Linux-first test should prove one or more of: + +- the spawned member process is attached to the authoritative session cgroup +- the spawned member process observes the session overlay/rootfs view rather than the daemon host view +- the spawned member process is constrained by the session network posture when isolation is enabled +- the runtime fails closed when placement cannot be established + +The test should verify placement through observable runtime facts, not by inferring from participant metadata alone. + +Support harnesses likely involved: + +- [crates/shell/tests/support/repl_world_agent.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/support/repl_world_agent.rs:156) +- [crates/shell/tests/support/socket.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/support/socket.rs:1124) + +### Suggested validation commands + +```bash +cargo test -p shell --lib -- --nocapture +cargo test -p world-agent --test streamed_execute_cancel_v1 --no-run +cargo test -p shell --test repl_world_first_routing_v1 --no-run +cargo test -p world-agent --test streamed_execute_cancel_v1 -- --nocapture +cargo test -p shell --test repl_world_first_routing_v1 -- --nocapture +cargo test -p shell --test agent_successor_contract_ahcsitc0 -- --nocapture +cargo test -p shell --test agent_hub_trace_persistence -- --nocapture +``` + +If the placement fix adds a dedicated Linux-only proof test, that command must be added to this section and to the final implementation PR. + +### Installer/provisioning verification + +The implementation closeout should include at least: + +- one Linux dev-install verification pass +- one Linux production-install or `world-provision.sh` verification pass +- one macOS Lima provisioning verification pass if the guest service-unit contract changes + +Relevant reference docs: + +- [docs/WORLD.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/WORLD.md:105) +- [docs/INSTALLATION.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/INSTALLATION.md:79) +- [docs/manual_verification/linux_world_socket.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/manual_verification/linux_world_socket.md:1) +- [docs/cross-platform/wsl_world_setup.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/cross-platform/wsl_world_setup.md:1) +- [docs/cross-platform/mac_world_setup.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/cross-platform/mac_world_setup.md:1) + +## Acceptance Criteria + +1. A selected world-scoped member still launches through `world-agent` over `/v1/execute/stream`. +2. The launched member runtime is actually placed inside the authoritative session world rather than merely labeled with `world_id` and `world_generation`. +3. `/v1/execute/cancel` still cancels the live member runtime by transport `span_id`. +4. Shell-owned participant persistence, readiness gating, invalidation, and replacement semantics remain authoritative and unchanged in shape. +5. Same-generation reuse and world-generation replacement continue to work. +6. World-scoped member startup fails closed when true world placement cannot be established. +7. Both [scripts/substrate/dev-install-substrate.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/substrate/dev-install-substrate.sh:1482) and [scripts/substrate/install-substrate.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/substrate/install-substrate.sh:1932) are updated as needed, and their downstream provisioning paths are kept in sync with the final runtime contract. +8. Any WSL/macOS service-unit or provisioning drift introduced by the placement fix is either resolved in the same slice or explicitly documented and fail-closed. + +## Non-Negotiable Guardrails + +- Do not move runtime selection into `world-agent`. +- Do not reintroduce shell-local host fallback for a world-scoped member. +- Do not let participant metadata claim a world placement guarantee that the process launch path does not actually establish. +- Do not update only one installer/provisioning surface when the runtime contract changes. diff --git a/llm-last-mile/14-secret-handoff-into-the-world-gateway.md b/llm-last-mile/14-secret-handoff-into-the-world-gateway.md new file mode 100644 index 000000000..a86849469 --- /dev/null +++ b/llm-last-mile/14-secret-handoff-into-the-world-gateway.md @@ -0,0 +1,338 @@ +# SOW: Secret Handoff Into The World Gateway + +Status: implementation-oriented draft. This SOW covers the remaining gateway secret-delivery blocker called out in [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md:119). Host-side policy gating, auth sourcing, and typed `integrated_auth` request payloads already exist. The missing work is the carrier swap inside the world-owned gateway lifecycle so secret values stop living in the in-world gateway process environment by default. + +## Objective + +Land one production path where Substrate continues to: + +- source auth material on the host under existing policy gates, +- send typed auth material to `world-agent` over the existing lifecycle request seam, +- and launch the in-world gateway under Substrate-owned lifecycle control, + +but changes the final handoff from: + +- secret-bearing child env vars, + +to: + +- a read-once auth bundle delivered through an inherited one-time FD/pipe channel, with only a non-secret pointer env var exposed to the gateway process. + +The required outcome is not a new gateway control plane. The required outcome is that integrated gateway auth delivery becomes honest about the trust boundary: Substrate still owns secret delivery, but the in-world gateway no longer receives secret values in its process environment by default. + +## Why This Is Needed + +The current integrated gateway path is already substantially landed: + +- the shell resolves allowlisted auth material under the existing config/policy surface in [crates/shell/src/builtins/world_gateway.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/builtins/world_gateway.rs:640), +- lifecycle requests already carry typed `integrated_auth` payloads in [crates/shell/src/builtins/world_gateway.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/builtins/world_gateway.rs:354), +- and `world-agent` already validates backend-specific auth expectations in [crates/world-agent/src/gateway_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/gateway_runtime.rs:1101). + +The remaining mismatch is narrower and more security-sensitive: + +1. `world-agent` resolves the integrated auth payload into concrete secret env vars in [crates/world-agent/src/gateway_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/gateway_runtime.rs:1097). +2. The managed gateway launch path then injects those values directly into the child process environment in [crates/world-agent/src/gateway_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/gateway_runtime.rs:706). +3. The fake managed-gateway binaries used in runtime tests still assert the presence of secret-bearing env vars in [crates/world-agent/src/gateway_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/gateway_runtime.rs:1552). + +That means the integrated path currently satisfies policy and lifecycle requirements, but not the preferred secret-channel posture already selected in the governing docs: + +- [ADR-0023](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/project_management/adrs/draft/ADR-0023-in-world-llm-gateway-front-door.md:72) records the additive upgrade from legacy env injection to FD/pipe auth-bundle delivery. +- [ADR-0040](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/project_management/adrs/draft/ADR-0040-substrate-gateway-boundary-and-runtime-ownership.md:82) keeps host-to-world secret delivery explicitly Substrate-owned. +- [Substrate Gateway Policy Evaluation](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/contracts/substrate-gateway-policy-evaluation.md:33) already says the carrier should move away from env-based delivery by default. +- [DR-0018](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/project_management/_archived/next/llm_gateway_in_world/decision_register.md:683) explicitly selects the auth-bundle FD/pipe direction. +- [Secrets Delivery Channel Rubric](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/project_management/system/standards/shared/SECRETS_DELIVERY_CHANNEL_RUBRIC.md:36) says FD/pipe is the default when Substrate spawns both endpoints. + +This is now the remaining security-critical last-mile gap for the in-world gateway path. + +## Relationship To Existing Decisions + +This SOW is intentionally bounded. It consumes existing decisions without reopening them: + +- `ADR-0027` remains the source of truth for: + - `llm.secrets.env_allowed` + - `agents.host_credentials.read.allowed_backends` + - backend allowlists + - host-side secret sourcing precedence +- `ADR-0040` remains the ownership split: + - Substrate owns host-to-world secret delivery + - `substrate-gateway` owns in-world runtime internals +- `ADR-0046` remains the integrated runtime realization seam: + - selected backend resolution + - adapter binding + - typed `integrated_auth` payloads +- `DR-0018` and the shared secrets rubric remain the source of truth for: + - why FD/pipe is preferred, + - why secret-bearing env vars should stop being the default, + - and why a non-secret pointer env var is allowed. + +Non-goal reminder: + +- this slice does not redesign backend selection, +- does not redesign host-side auth sourcing, +- does not widen the operator command family, +- and does not broaden the request contract into a new public auth API. + +## Current Relevant Code Surfaces + +### Host-side auth sourcing and lifecycle request construction + +- [crates/shell/src/builtins/world_gateway.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/builtins/world_gateway.rs:339) + - resolves the selected backend + - derives `integrated_auth` + - builds `GatewayLifecycleRequestV1` +- [crates/shell/src/builtins/world_gateway.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/builtins/world_gateway.rs:640) + - `resolve_integrated_auth_payload(...)` + - `resolve_cli_codex_integrated_auth(...)` + - `resolve_api_env_integrated_auth(...)` + +These are already doing the right policy-gated host sourcing work. This SOW does not redesign them. + +### World-agent request preparation and current secret-delivery gap + +- [crates/world-agent/src/service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs:1558) + - validates the typed gateway lifecycle request and carries `integrated_auth` toward the runtime manager +- [crates/world-agent/src/gateway_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/gateway_runtime.rs:706) + - launches the managed gateway child + - currently sets secret-bearing env vars on that child +- [crates/world-agent/src/gateway_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/gateway_runtime.rs:1097) + - `ResolvedGatewayAuthHandoff` + - `resolve_integrated_auth_handoff(...)` + - `resolve_codex_auth_handoff(...)` + - `resolve_api_env_auth_handoff(...)` + +This is the exact seam that must change. + +### Operator and policy contracts that must remain true + +- [docs/contracts/substrate-gateway-operator-contract.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/contracts/substrate-gateway-operator-contract.md:7) +- [docs/contracts/substrate-gateway-policy-evaluation.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/contracts/substrate-gateway-policy-evaluation.md:25) + +This SOW must preserve those contracts while changing only the durable secret carrier between `world-agent` and the in-world gateway process. + +## In Scope + +- replace secret-bearing gateway child env injection with auth-bundle FD/pipe delivery +- preserve the existing typed `integrated_auth` lifecycle request seam from shell to `world-agent` +- keep host-side secret sourcing and policy gating unchanged +- define the non-secret pointer contract used to tell the gateway which FD/pipe to read +- update runtime launch, restart, and rotation semantics so re-sync/restart delivers a fresh read-once bundle +- update tests and docs so the integrated path proves absence of secret-bearing env vars in the gateway process by default +- Linux-first implementation and validation, with explicit fail-closed posture where equivalent support is not yet available + +## Out Of Scope + +- redesigning `GatewayLifecycleRequestV1` into a new public API family +- tuple-policy work from `ADR-0042` / `ADR-0043` +- backend-selection redesign from `ADR-0046` +- gateway-local admin/config redesign +- changing the operator command family (`sync|status|restart`) +- changing host-side secret precedence rules +- changing member-runtime placement or toolbox auth delivery + +## Required Semantics And Invariants + +### 1. Host-side policy and sourcing rules must not change + +The auth-bundle slice must preserve the existing rules already encoded in shell policy evaluation: + +- allowlisted env auth remains primary when complete, +- host credential file reads remain fallback-only where permitted, +- partial env auth remains invalid integration, +- and blocked env reads remain policy denial. + +Carrier choice must not change authorization semantics. + +### 2. The existing typed `integrated_auth` request remains the host→world seam + +The shell already sends typed `integrated_auth` payloads to `world-agent`. This SOW should treat that existing lifecycle request as the Substrate-owned secret-channel payload. + +Required rule: + +- do not invent a second public request family just to carry the same auth material. + +### 3. Secret values must not be present in the in-world gateway process environment by default + +Once this slice lands, the normal integrated gateway path must stop placing values for: + +- `SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_*` +- `SUBSTRATE_LLM_BACKEND_AUTH_API_*` +- or raw provider env vars such as `OPENAI_API_KEY` + +into the managed gateway process environment by default. + +### 4. A non-secret pointer env var is allowed + +The gateway process may receive one non-secret pointer env var such as: + +- `SUBSTRATE_LLM_AUTH_BUNDLE_FD` + +Required rule: + +- the pointer env var may identify the inherited FD number, +- but it must never contain secret material itself, +- and it must be safe to print/log if surfaced by diagnostics. + +### 5. The auth bundle must be read-once and closed promptly + +The managed gateway contract must follow the shared secrets rubric: + +- `world-agent` writes the bundle once, +- the gateway reads it once, +- the FD/pipe is closed promptly, +- and it is not forwarded to child processes unless an owning spec explicitly requires and documents that propagation. + +### 6. Canonical auth field names must remain stable + +The auth bundle payload must continue using the canonical `SUBSTRATE_LLM_BACKEND_AUTH_*` field names for the secret-bearing keys even when the values no longer travel as env vars. + +That keeps redaction/caps rules uniform across: + +- legacy env-based compatibility paths, +- auth-bundle delivery, +- and any later in-world child propagation rules. + +### 7. No silent fallback to env injection on the production integrated path + +If the selected integrated gateway runtime cannot consume the auth bundle, the integrated lifecycle must fail closed instead of silently reintroducing secret-bearing env vars as the default path. + +Legacy env injection may remain only as an explicit compatibility path if deliberately documented and gated. It must not be the default once this slice lands. + +### 8. Restart/rotation must re-deliver a fresh bundle + +`substrate world gateway restart` and any `sync`-driven replacement path must deliver a fresh auth bundle for the new process instance. + +No persisted auth bundle files are permitted. + +## Core Implementation Work + +### 1. Replace env-var-oriented auth handoff with an auth-bundle-oriented internal model + +`ResolvedGatewayAuthHandoff` in [crates/world-agent/src/gateway_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/gateway_runtime.rs:1097) is currently modeled as `env_vars`. + +This slice should replace that internal model with something closer to: + +- canonical auth field names plus secret values, +- suitable for serialization into a read-once bundle payload, +- without assuming the final carrier is env injection. + +The exact type name is implementation-defined. The important change is semantic: `world-agent` should resolve auth material into a bundle payload, not into process env assignments. + +### 2. Add bundle creation and inherited-FD spawn plumbing in `GatewayRuntimeManager` + +The gateway launch path in [crates/world-agent/src/gateway_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/gateway_runtime.rs:706) must: + +- create a one-time pipe or equivalent FD-backed channel, +- serialize the auth bundle payload into the write end, +- pass the read end to the gateway child process, +- export only the non-secret pointer env var, +- and avoid setting secret-bearing env vars on the child. + +The implementation must ensure: + +- correct close discipline on both sides, +- correct error handling if the bundle cannot be created or written, +- and cleanup on spawn failure so no dangling open FDs remain. + +### 3. Define the managed gateway consumption contract clearly + +This repo owns the Substrate side of the lifecycle seam, but the gateway must still know how to consume the bundle. + +Required contract: + +- if `SUBSTRATE_LLM_AUTH_BUNDLE_FD` is present, the managed gateway must read the auth bundle from that FD, +- load the values into memory only, +- and not require the corresponding secret-bearing env vars to be present. + +This SOW does not require a second gateway control API. It requires one clear startup contract. + +### 4. Preserve the existing shell-side secret sourcing boundary + +The shell-side code in [crates/shell/src/builtins/world_gateway.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/builtins/world_gateway.rs:640) should continue to: + +- read allowed env values, +- read allowed credential files where applicable, +- fail closed on incomplete/blocked inputs, +- and send typed `integrated_auth` to `world-agent`. + +This slice should not move secret sourcing into the gateway or into gateway-local config. + +### 5. Preserve invalid-integration, dependency-unavailable, and policy-denial distinctions + +The current gateway policy contract already requires those buckets to remain distinct in [docs/contracts/substrate-gateway-policy-evaluation.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/contracts/substrate-gateway-policy-evaluation.md:38). + +The auth-bundle work must preserve that separation for cases such as: + +- missing host-side secret material, +- blocked env or credential-file reads, +- bundle creation/write failure, +- gateway process that does not honor the pointer contract, +- and transient launch/readiness failures. + +### 6. Keep gateway runtime artifacts non-secret + +Managed logs, status surfaces, and diagnostics may continue to expose: + +- runtime log paths, +- backend id, +- lifecycle state, +- and the non-secret pointer contract when necessary. + +They must not expose: + +- bundle contents, +- secret-bearing field values, +- or secret-bearing env vars that should no longer exist on the child process. + +### 7. Update fake gateway launchers and parity tests to prove the new contract + +The managed-gateway test launchers in [crates/world-agent/src/gateway_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/gateway_runtime.rs:1540) currently assert secret env presence. + +Those tests must be rewritten to prove the new posture: + +- the pointer env var is present, +- the auth bundle can be read exactly once, +- secret-bearing env vars are absent by default, +- restart delivers a fresh bundle, +- and failure to read the bundle fails the lifecycle cleanly. + +## Required Validation Work + +### Unit and integration tests + +At minimum this slice should add or update tests for: + +- shell-side auth resolution still honoring current precedence and policy gates +- `world-agent` auth-bundle rendering for `cli:codex` and `api:*` +- managed gateway launch proving the absence of secret-bearing env vars by default +- managed gateway launch proving the presence of `SUBSTRATE_LLM_AUTH_BUNDLE_FD` +- read-once behavior and prompt close of the bundle FD +- restart/sync rotation delivering a fresh bundle +- fail-closed behavior when the gateway does not consume the bundle or when the bundle cannot be created + +Primary test surfaces: + +- [crates/shell/tests/world_gateway.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/world_gateway.rs) +- [crates/world-agent/tests/gateway_runtime_parity.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/tests/gateway_runtime_parity.rs) +- [crates/world-agent/src/gateway_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/gateway_runtime.rs) + +### Documentation and contract updates + +This slice should update the docs that currently describe the preferred auth-bundle posture so they match the landed runtime: + +- `AGENT_ORCHESTRATION_GAP_MATRIX.md` should stop treating this item as missing once the carrier swap is real. +- any gateway contract docs that still describe env-based integrated delivery as the only concrete behavior should be updated to reflect auth-bundle-by-default. +- test fixture comments that assume `SUBSTRATE_LLM_BACKEND_AUTH_*` values appear in the gateway env must be rewritten. + +## Exit Criteria + +This slice is complete when all of the following are true on the primary Linux production path: + +1. The shell still resolves and sends typed `integrated_auth` under existing policy gates. +2. `world-agent` no longer resolves that auth into child env assignments as the default managed-gateway path. +3. The managed gateway receives auth through a read-once FD/pipe bundle with only a non-secret pointer env var exposed. +4. The managed gateway process environment no longer contains secret-bearing auth values by default. +5. Restart/sync rotation re-delivers fresh auth material without persistence. +6. Tests prove both the happy path and fail-closed posture. +7. The matrix row for `Secret handoff into the world gateway` can be marked landed. + +At that point, the remaining v1 gaps are no longer about secure gateway secret delivery. They move back to operator/control-plane follow-ons such as status ambiguity handling and public session controls. diff --git a/llm-last-mile/15-targeted-repl-agent-turns-linux-first.md b/llm-last-mile/15-targeted-repl-agent-turns-linux-first.md new file mode 100644 index 000000000..09288724e --- /dev/null +++ b/llm-last-mile/15-targeted-repl-agent-turns-linux-first.md @@ -0,0 +1,444 @@ +# SOW: Targeted REPL Agent Turns, Linux-First + +Status: implementation-oriented draft. This SOW defines the next landing after the Linux-first member-runtime placement and gateway-auth slices. It is intentionally narrow: add one explicit REPL caller grammar for targeted agent turns, route those turns by named `backend_id`, submit a real user prompt into the already-live UAA session for the selected backend, and stream the result back through the REPL. It does not redesign `substrate -c`, and it does not productize a broader `substrate agent start|resume|fork|stop` command family. + +## Objective + +Land one Linux-first production path where the interactive REPL can: + +- parse an explicit targeted agent-turn syntax, +- resolve that syntax to a named backend such as `cli:codex`, +- require routing by `backend_id` instead of "the one eligible member," +- submit the user’s prompt into the already-active retained-control UAA session for the selected runtime, +- and stream the resulting agent output back through the REPL without collapsing the input into normal shell execution. + +The preferred syntax decision for this slice is: + +- `::cli:codex ` +- generalized as `:: ` + +This slice stays REPL-only. Non-interactive `substrate -c` remains shell wrap mode and is intentionally out of scope. + +## Why This Is Needed + +The repo now has most of the runtime control plane needed for a first targeted-agent caller path, but the operator-facing turn-submission seam is still missing. + +What already exists: + +- the REPL starts a shell-owned orchestrator runtime through UAA in [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:1729) and [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:1848) +- that startup path persists authoritative orchestration and participant state in [state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:100) +- backend inventory already derives canonical `backend_id` values such as `cli:codex` in [agent_inventory.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_inventory.rs:105) +- member-runtime selection, realizability checks, and policy gating already exist in [validator.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/validator.rs:30) +- Linux member runtimes already launch through `world-agent` over the host<->world seam in [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:3364), [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:2840), and [member_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs:54) +- for active world-scoped members, the retained control handle is owned inside `world-agent`, not the shell, because `MemberRuntimeManager` stores the active cancel handle in its own registry in [member_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs:38) through [member_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs:157) +- the gap matrix now explicitly records that invocation grammar, user-turn submission, and explicit backend targeting are still open in [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md:86), [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md:106), [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md:112), and [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md:113) + +What is still missing: + +- the REPL only recognizes shell directives like `:host` and `:pty`; normal input still falls through to shell execution in [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:669) through [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:930) +- both orchestrator and member runtimes are currently started with a fixed bootstrap prompt rather than a real user turn in [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:1622), [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:1932), and [member_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs:68) +- there is no repo-visible path after startup that submits an arbitrary user prompt into the active retained UAA session; current `agent_api` usage in the shell and `world-agent` is bootstrap-only `run_control(...)` +- the current shell<->world seam for active members is limited to remote launch, streamed events/output, and cancel: + - the shell starts a member through `execute_stream(...)` in [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:2921) + - `world-agent` branches that request into `member_runtime.launch(...)` in [service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs:1233) through [service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs:1266) + - later cancellation goes back through `execute_cancel(...)` in [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:2827) and [service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs:1465) + - there is no follow-on request seam today for "submit this user turn into the already-live member session" +- world-member selection still fails closed when more than one eligible member exists instead of routing by a named backend in [validator.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/validator.rs:161) through [validator.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/validator.rs:235) +- `substrate -c` is still explicitly planned and tested as shell wrap mode in [cli.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/cli.rs:27), [plan.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/invocation/plan.rs:29), [plan.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/invocation/plan.rs:616), and [tests.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/invocation/tests.rs:56) +- the public CLI still exposes `agent list|status|doctor|toolbox` only; there is no first-class `start|resume|fork|stop` namespace in [cli.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/cli.rs:399) through [cli.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/cli.rs:480) + +That makes the current runtime honest about startup ownership, but not yet useful as a user-facing targeted turn surface. + +## Relationship To Existing Slices + +This SOW consumes the already-landed runtime slices and does not reopen them: + +- [10-member-runtime-launch-seam.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/10-member-runtime-launch-seam.md) established the need for a real member runtime lifecycle seam +- [13-member-runtime-world-placement-gap-sow.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/13-member-runtime-world-placement-gap-sow.md) bounded the Linux-first in-world placement hardening +- [14-secret-handoff-into-the-world-gateway.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/14-secret-handoff-into-the-world-gateway.md) bounded the gateway auth-carrier fix and is not a substitute for pure-agent turn submission +- [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md:170) already recommends the next runtime slice as explicit caller grammar plus real user-turn submission + +This slice should be treated as the first operator-facing caller seam on top of the landed REPL-owned orchestrator and Linux-first world-member runtime. + +## Current Relevant Code Surfaces + +### REPL parsing and current shell-first behavior + +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:669) + - rejects bare `:host` and `:pty` + - handles `:host ` + - handles `:pty ` + - otherwise executes the input as a shell command in host or world scope +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:4095) + - already centralizes the "embedded newlines disable directive parsing" rule + +This is the insertion point for explicit agent-turn grammar. + +### Runtime startup and current bootstrap-only UAA ownership + +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:1729) + - prepares orchestrator selection, inventory, policy gating, and state-store setup +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:1932) + - starts the orchestrator with `AgentWrapperGateway.run_control(...)` and the fixed bootstrap prompt +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:2663) + - prepares a member runtime manifest with authoritative world binding +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:2840) + - starts the Linux member runtime over the remote member-dispatch seam +- [crates/world-agent/src/member_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs:68) + - also uses `run_control(...)` with the same fixed member bootstrap prompt + +These paths prove retained-control runtime ownership exists already. They do not provide a user-turn submission call after startup, and for world-scoped members the retained control owner is `world-agent`, not the shell. + +### Current member control seam is launch, stream, and cancel only + +- [crates/world-agent/src/service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs:1233) + - receives `member_dispatch` on `execute_stream(...)` + - launches the active member runtime and returns an NDJSON event stream +- [crates/world-agent/src/service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs:1465) + - receives `execute_cancel(...)` and routes it to `member_runtime.cancel(...)` +- [crates/world-agent/src/member_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs:91) + - stores the active runtime in the `world-agent` process, including the cancel handle + +That means Linux world-member targeted turns cannot be solved by a shell-only submission helper. The shell does not own the live member control handle after launch, and there is no current request seam for follow-on turn submission into that already-live member session. + +### Backend identity, routing, and current ambiguity posture + +- [crates/shell/src/execution/agent_inventory.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_inventory.rs:105) + - derives canonical backend ids from inventory entries +- [crates/shell/src/execution/agent_runtime/validator.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/validator.rs:12) + - already carries `backend_id` on `RuntimeSelectionDescriptor` +- [crates/shell/src/execution/agent_runtime/validator.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/validator.rs:161) + - selects world members by "collect every eligible world-scoped member" +- [crates/shell/src/execution/agent_runtime/validator.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/validator.rs:229) + - fails closed when more than one eligible member exists +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:2518) + - still calls the ambiguity-based member selector rather than a named-backend selector + +This is the exact seam that must change from eligibility-only routing to explicit `backend_id` routing. + +### Active-session state and public status surface + +- [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:433) + - already has `resolve_single_live_session_for_agent(...)` +- [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:421) + - status preflights `resolve_single_live_session_for_agent(...)` +- [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:971) + - toolbox also depends on a single live orchestrator session + +Those surfaces matter for context, but this SOW does not require a full status redesign. It only needs enough runtime identity to send a targeted REPL turn into the selected active session. + +### `substrate -c` and `substrate agent` posture that stays out of scope + +- [crates/shell/src/execution/cli.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/cli.rs:27) + - `-c/--command` is documented as "Execute a single command" +- [crates/shell/src/execution/invocation/plan.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/invocation/plan.rs:29) + - `ShellMode::Wrap(String)` is still the `-c` mode +- [crates/shell/src/execution/invocation/plan.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/invocation/plan.rs:619) + - `cli.command` maps directly to `ShellMode::Wrap(cmd)` +- [crates/shell/src/execution/invocation/tests.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/invocation/tests.rs:56) + - existing tests enforce wrap-mode semantics +- [crates/shell/src/execution/cli.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/cli.rs:471) + - `agent` subcommands still stop at `list|status|doctor|toolbox` + +This slice should reference those facts, not fight them. + +## In Scope + +- add explicit REPL grammar for targeted agent turns +- prefer `::cli:codex ` and support generalized `:: ` +- route targeted turns by named `backend_id` +- add one real prompt-submission path into the active retained UAA session for the selected backend +- stream the submitted turn’s result back through the REPL +- keep the first landing Linux-first for world-scoped member turns +- allow host-scoped targeted turns only if they reuse the already-live shell-owned orchestrator runtime without widening product scope +- add one additive shell<->world-agent turn-submission seam for already-live Linux world-member runtimes +- add the minimum runtime/session bookkeeping needed to correlate a submitted turn with the selected active session +- add focused tests for grammar parsing, named-backend routing, and fail-closed behavior + +## Out Of Scope + +- redesigning `substrate -c` or changing `ShellMode::Wrap` +- adding non-interactive agent invocation syntax +- productizing `substrate agent start|resume|fork|stop` +- inventing default-agent implicit routing for plain REPL input +- replacing the shell/world transport seam for member runtimes +- toolbox work +- gateway auth-carrier work +- cross-platform parity promises beyond an explicit Linux-first implementation +- broad session-resume or session-fork UX beyond the minimum context this document needs to mention +- toolbox, gateway, or nested-LLM paths as substitutes for pure-agent targeted turns + +## Required Semantics And Invariants + +### 1. Agent turns must be explicit in the REPL + +Normal REPL input must remain shell execution. Agent turns must require explicit syntax. + +Required rule: + +- only inputs matching `:: ` enter the targeted-agent path +- plain shell input must keep its current shell/world execution behavior +- `:host` and `:pty` keep their existing meanings + +This avoids silently changing the semantics of an existing shell-first REPL. + +### 2. Backend routing must be by named `backend_id`, not by eligible-member heuristics + +The targeted-agent caller surface must resolve exactly the backend the user named. + +Required rule: + +- the parser extracts a concrete `backend_id` +- runtime selection uses inventory lookup plus policy validation for that backend +- the implementation must not reuse `validate_member_selection(...)` as the final routing rule for explicit agent turns + +When multiple world-scoped members are configured, the targeted-turn path must not ask the runtime to "pick one." It must route to the named backend or fail closed. + +### 3. Host vs world placement still follows inventory scope + +Named backend routing does not erase the existing placement model. + +Required rule: + +- if the selected backend resolves to `execution.scope=host`, route to the active host-scoped orchestrator-side runtime only if that runtime matches the targeted backend contract for this slice +- if the selected backend resolves to `execution.scope=world`, require the Linux world-member runtime path and authoritative world binding already established by the REPL session +- no host fallback is allowed for a targeted world-scoped backend + +This keeps the new caller grammar aligned with the repo’s existing execution-scope model. + +### 4. A real user turn must be submitted after bootstrap + +This slice must stop at least one path from being bootstrap-prompt-only. + +Required rule: + +- the targeted REPL path must send the user’s actual prompt into the active selected UAA session after startup +- the submitted prompt must not be represented by re-running the runtime bootstrap prompt +- the shell must preserve the existing bootstrap prompt only for startup ownership, not for operator turns + +This is the central functional gap the slice must close. + +### 5. REPL streaming must stay attached and operator-visible + +The targeted turn must stream output back through the interactive REPL rather than waiting for a detached snapshot. + +Required rule: + +- the operator sees streamed response events in the REPL +- streamed output must be correlated to the targeted backend and active session +- cancellation and terminal completion must surface through the same REPL control lane + +The first pass does not need a polished transcript model, but it must be a real streamed interaction. + +### 6. Host-scoped and world-scoped submission ownership must stay distinct + +The submission path is not symmetric across scopes in the current repo. + +Required rule: + +- host-scoped targeted turns may use a shell-local submission lane because the shell owns the live host orchestrator runtime after startup +- world-scoped targeted turns must use a remote submission lane owned by `world-agent` because `world-agent` owns the active retained member runtime after launch +- the Linux world-member solution must add an additive shell<->world-agent request seam for turn submission into an already-live member session +- this seam must stay bounded to active member-turn submission and must not broaden into a generic new hub service + +### 7. Runtime state must stay authoritative + +The new targeted-turn seam must consume the current runtime authority model rather than inventing a parallel one. + +Required rule: + +- live session/participant identity stays in the existing runtime store under `~/.substrate/run/agent-hub/sessions/...` +- the targeted-turn path resolves the selected active session from authoritative live state +- no new ad hoc in-memory-only session registry becomes the source of truth + +### 8. `substrate -c` remains out of scope + +This slice must not quietly broaden into non-interactive agent execution. + +Required rule: + +- `substrate -c` remains shell wrap mode exactly as it is today +- no reuse of `-c` for agent prompts in this landing +- any later non-interactive agent surface must be handled by a separate slice + +### 9. Public agent-command productization remains deferred + +This slice can mention future `start|resume|fork|stop` surfaces for context, but it must not block on them. + +Required rule: + +- do not require a new public `substrate agent start|resume|fork|stop` family to land targeted REPL turns +- do not redesign the `agent` CLI namespace as part of this work + +## Recommended Implementation Shape + +### 1. Add a dedicated REPL parse step for targeted agent turns + +Add a small explicit parser in [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) near the existing directive handling. + +Recommended behavior: + +- run before shell execution fallback +- ignore multi-line inputs for the first pass, matching the existing directive posture around [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:4095) +- parse `:: ` +- reject missing backend ids +- reject missing prompts +- produce clear REPL-facing errors for malformed syntax + +The parser does not need to support alternate spellings in this slice. The preferred contract should be explicit and narrow. + +### 2. Add explicit backend-id selection helpers + +Introduce a selector next to the current validator helpers that resolves one named backend from effective inventory and policy. + +The helper should: + +- validate the requested `backend_id` format +- resolve the matching inventory entry +- confirm the derived backend id exactly matches the requested backend id +- enforce `PURE_AGENT_PROTOCOL` +- reuse realizability checks from [validator.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/validator.rs:112) +- enforce policy allowlisting through `backend_allowed(...)` +- return a `RuntimeSelectionDescriptor` + +This should be additive, not a rewrite of the current ambiguity-based helper. Existing startup flows can keep using `validate_member_selection(...)` until a later slice says otherwise. + +### 3. Separate "runtime startup" from "turn submission" + +The current code path conflates runtime usefulness with bootstrap ownership. This slice should introduce a second operation after startup: submit a turn into an already-live selected runtime. + +Recommended shape: + +- keep current orchestrator/member startup flows for retained ownership +- add one shell-local submission function for host-scoped targeted turns that accepts: + - selected `RuntimeSelectionDescriptor` + - authoritative live participant/session identity + - prompt text + - REPL printer / telemetry handles +- add one additive shell<->world-agent submission request for Linux world-scoped targeted turns that addresses an already-live member runtime rather than launching a new one +- route both paths through already-live runtime ownership rather than by spawning a fresh runtime per prompt + +The repo does not currently show any post-bootstrap active-session submission usage of `agent_api`; it shows bootstrap `run_control(...)` usage only. This slice must therefore add the narrowest wrappers and request seams needed to surface real post-bootstrap turn submission rather than implying that the capability is already exercised here. + +### 4. Add runtime resolution by backend and scope + +The targeted-turn lane needs to map a named backend to the active runtime that owns that backend in the current REPL session. + +Recommended behavior: + +- for host scope: + - ensure the active runtime exists + - ensure it matches the requested backend id + - submit the turn through a shell-local active-runtime lane + - fail closed if the selected backend is not the active host runtime for this REPL session +- for Linux world scope: + - ensure authoritative world binding exists + - ensure the member runtime for that backend is ready for the active `world_generation` + - start it through the existing Linux member path if needed + - submit the turn through a narrow shell<->world-agent request addressed to the already-live member runtime that `world-agent` owns + - fail closed on non-Linux platforms + +This keeps routing honest without requiring a full multi-backend scheduler. + +### 5. Add a narrow remote turn-submission seam for active Linux member runtimes + +The Linux world-member path needs one additive control request beyond today’s launch/event/cancel contract. + +Recommended shape: + +- add one internal shell<->world-agent request that means "submit this user turn into the already-live member runtime identified by the current orchestration session, participant/runtime identity, and backend" +- keep the request bounded to active member-turn submission; do not broaden it into a generic new agent hub or reusable multi-purpose runtime service +- reuse the existing remote ownership model in `world-agent` rather than copying member session control back into the shell +- return streamed output over the same narrow request lane so the REPL can render the response live + +This is the missing Linux-first world-member seam. Without it, explicit backend parsing still cannot drive a real targeted user turn into an already-live in-world member session. + +### 6. Stream targeted-turn output through the REPL + +The submission lane should reuse the existing event/printing discipline already used for runtime startup and world-command passthrough where possible. + +Minimum expected behavior: + +- print backend-targeted progress/errors to the REPL +- stream agent output incrementally +- surface completion or failure distinctly from shell command completion +- keep runtime-owned cancellation semantics intact + +This slice does not need a final transcript persistence design, but it should not reduce the interaction to a one-shot buffered response if the active UAA session can stream. + +### 7. Keep world-member routing Linux-first and fail-closed elsewhere + +The repo already encodes Linux-only member runtime readiness in [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:3430). + +Required posture: + +- world-scoped targeted turns are supported only on Linux in this slice +- non-Linux targeted world-member turns fail closed with a clear error +- the SOW must not promise macOS or Windows parity + +## Concrete Work Breakdown + +1. Add a targeted-turn parser and handler in [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) before shell fallback execution. + +2. Add named-backend resolution helpers in [crates/shell/src/execution/agent_runtime/validator.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/validator.rs) so explicit REPL turns can select one backend deterministically. + +3. Add a shell-side active-runtime resolution layer that maps the named backend plus scope to: + - the active host runtime, or + - the active Linux member runtime for the current `world_generation` + +4. Add a shell-local user-turn submission primitive for the active host-scoped runtime. + +5. Add one additive shell<->world-agent turn-submission seam for already-live Linux member runtimes. + +6. Add REPL streaming for submitted turn output and terminal completion across both the shell-local host path and the remote world-member path. + +7. Add focused tests for: + - grammar parsing + - malformed syntax + - unknown backend ids + - denied backends + - multiple eligible members with explicit backend routing + - proof that the world-member path uses a remote submission seam rather than a shell-only helper + - Linux-only world-member targeting + - proof that plain REPL input still executes as shell input + - proof that `substrate -c` still stays in wrap mode + +## Validation Expectations + +At minimum, this slice should add or update tests that prove: + +- `:host` and `:pty` still behave as they do today +- `::cli:codex hello` is not treated as a shell command +- targeted routing uses the named backend id rather than the ambiguity-based member selector +- the selected active host runtime receives a real user prompt after bootstrap through the shell-local submission lane +- the selected active Linux world-member runtime receives a real user prompt after bootstrap through the additive shell<->world-agent submission seam +- Linux world-member targeted turns fail closed when authoritative world binding or active member runtime cannot be established +- non-Linux world-member targeted turns fail closed with an explicit Linux-only error +- `substrate -c` still maps to `ShellMode::Wrap` + +If the implementation needs a fake CLI backend fixture to prove post-bootstrap submission, it should add the narrowest test harness necessary rather than widening the product surface. + +## Open Questions To Resolve During Implementation + +1. Which exact post-bootstrap turn-submission primitive should the repo standardize on for the host path, given that current repo usage only shows bootstrap `run_control(...)` and does not yet exercise active-session submission? + - This must be resolved explicitly in code rather than implied by the SOW. + +2. Should host-scoped targeted turns be limited to the selected orchestrator backend only in the first pass? + - Current repo reality only guarantees one live host-scoped orchestrator runtime per REPL session, so widening to arbitrary additional host backends should not be assumed. + +3. What is the smallest request/response shape the repo should add between the shell and `world-agent` for active member-turn submission without turning it into a generic new runtime service? + - The existing Linux member seam is launch, stream, and cancel only, so this needs one explicit additive contract. + +4. Should the first pass reject multi-line `::` prompts entirely? + - The current REPL directive model already treats embedded newlines specially, so explicit rejection is safer than silent partial parsing. + +5. How much new trace or `AgentEvent` structure is actually required for submitted user turns? + - This SOW expects real streamed interaction, but it does not require a final message-history schema if existing event machinery is sufficient for the first landing. + +## Non-Goals Repeated Explicitly + +- no `substrate -c` redesign +- no broader public `substrate agent start|resume|fork|stop` productization +- no toolbox substitution for targeted agent turns +- no cross-platform parity commitment beyond Linux-first world-member support +- no implicit "default backend" shell-to-agent conversion for ordinary REPL input diff --git a/llm-last-mile/16-explicit-backend-selection-by-backend-id.md b/llm-last-mile/16-explicit-backend-selection-by-backend-id.md new file mode 100644 index 000000000..c91228d63 --- /dev/null +++ b/llm-last-mile/16-explicit-backend-selection-by-backend-id.md @@ -0,0 +1,221 @@ +# SOW A: Complete Explicit Backend Selection by Backend ID + +Status: implementation-oriented follow-on draft. This SOW does not introduce backend-id targeting from scratch. The repo already has a partially landed exact-backend path for targeted REPL follow-up turns through `validate_exact_backend_selection(...)` in [validator.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/validator.rs:244) and `resolve_targeted_turn_route(...)` in [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:2702). This document isolates the remaining work needed to harden and complete that model beyond the currently landed targeted-turn route: align retained-member reuse to `backend_id`, remove leftover singleton assumptions in both the shell and `world-agent`, keep fail-closed behavior precise, and prove that multiple eligible world members can coexist as simultaneously retained live runtimes without ambiguity. + +## Objective + +Complete the already-landed explicit backend-targeting model for Linux-first member runtime orchestration by: + +- hardening inventory lookup and fail-closed exact selection around derived `backend_id`, +- moving retained world-member reuse from leftover singleton or `agent_id`-based heuristics to `backend_id`, +- ensuring launch and reuse paths consistently target the named backend beyond the current targeted-turn route, +- and adding regression coverage showing that multiple eligible world members in one orchestration session/world generation can coexist as simultaneously retained live runtimes and still be selected unambiguously by backend id. + +This SOW is about finishing and hardening backend identity and retained-runtime semantics. It is not the initial caller-grammar landing, and it is not the broader prompt-submission product slice. + +## Why This Is Needed + +The repo already has the core selector for explicit backend targeting, but that selector is only fully exercised along the currently landed targeted-turn path. The remaining implementation still spans two different eras: + +- inventory entries derive canonical backend ids such as `cli:codex` in [agent_inventory.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_inventory.rs) +- exact backend lookup already exists in [validator.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/validator.rs), alongside the older singleton-style `validate_member_selection(...)` +- REPL targeted-turn routing already uses exact backend selection in [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) +- member dispatch and follow-up submit requests already carry `backend_id` over the shell<->world transport in [lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/agent-api-types/src/lib.rs) and [world_ops.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs) +- Linux world-member follow-up validation already checks that a submitted turn’s `backend_id` matches the retained runtime in [member_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs) + +What is still inconsistent: + +- the gap matrix still records explicit backend targeting as only partially landed, with broader productization still open in [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md) +- older selection logic still exists for non-exact/member-singleton flows and still fails on ambiguity in [validator.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/validator.rs) +- the shell REPL still models world-member retention as a single slot via `let mut member_runtime: Option = None` in [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:553), and readiness/reuse still flows through that singleton slot in [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:3577) +- live-member reuse in the shell path still keys off `agent_id` plus `world_generation` in `live_member_for_generation(...)` in [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:2837), which is narrower and less precise than "reuse the retained runtime for backend `cli:codex`" +- `world-agent` still enforces a single retained world member globally via `active_members_by_participant_id` and rejects coexistence outright in [member_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs:391) + +That leaves the repo with a real exact-backend targeted-turn route, but not yet a fully coherent backend-id ownership and reuse model across simultaneously retained world members. This slice closes that narrower gap. + +## Relationship To Existing Slices + +- [15-targeted-repl-agent-turns-linux-first.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/15-targeted-repl-agent-turns-linux-first.md) is the parent combined SOW. This document carves out its backend-selection and retained-member identity work as SOW A. +- This SOW depends on the already-landed Linux member-runtime placement and world dispatch plumbing referenced in [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md). +- A later SOW can focus on caller grammar, prompt submission UX, and non-interactive surfaces. Those are intentionally not reopened here. + +## Current Relevant Code Surfaces + +### Inventory identity and derived backend ids + +- [crates/shell/src/execution/agent_inventory.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_inventory.rs) + - `AgentFileV1::derived_backend_id()` + - `AgentInventoryEntryV1::derived_backend_id()` + - effective inventory loading and validation entry points + +### Selection, scope checks, and currently landed exact targeting + +- [crates/shell/src/execution/agent_runtime/validator.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/validator.rs) + - `validate_runtime_realizability(...)` + - `validate_member_selection(...)` + - `validate_exact_backend_selection(...)` + - `RuntimeSelectionDescriptor { backend_id, execution_scope, binary_path, ... }` + +### REPL runtime selection and world-member launch + +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:553) + - single retained member slot: `member_runtime: Option` + - targeted route resolution by backend id in [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:2702) + - `live_member_for_generation(...)` in [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:2837) + - singleton readiness/reuse path in `ensure_member_runtime_ready_for_descriptor(...)` in [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:3577) + - Linux follow-up submission via `submit_world_targeted_turn(...)` in [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:3891) + +### Member dispatch transport contract + +- [crates/agent-api-types/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/agent-api-types/src/lib.rs) + - `MemberDispatchRequestV1` + - `MemberTurnSubmitRequestV1` + - `ResolvedMemberRuntimeDescriptorV1` + +- [crates/shell/src/execution/routing/dispatch/world_ops.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs) + - `MemberDispatchTransportRequest` + - request building for member dispatch payloads + +### Linux retained member ownership and backend checks + +- [crates/world-agent/src/member_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs) + - retained member registry + - submit-turn request validation against retained identity + - single-retained-member constraint that currently blocks coexistence + +## In Scope + +- hardening inventory lookup by derived `backend_id` for explicit backend-targeted runtime paths +- completing exact backend selection rollout beyond the currently landed targeted-turn route +- fail-closed behavior for: + - missing backend + - duplicate effective inventory matches for the same `backend_id` + - wrong execution scope + - disallowed backend under `agents.allowed_backends` + - runtime-unrealizable backend selections +- retained-member reuse keyed by backend identity rather than singleton member heuristics +- launch-path changes needed so the named backend is what gets started or reused +- Linux-first shell and `world-agent` cardinality changes needed to allow multiple retained eligible members to coexist as simultaneously retained live runtimes +- focused regression tests for ambiguity removal and coexistence + +## Out Of Scope + +- changing REPL grammar +- redesigning `substrate -c` +- broader `substrate agent start|resume|fork|stop` productization +- macOS or Windows parity +- transcript UX or output formatting changes +- gateway auth-carrier work +- toolbox work +- replacing the host<->world transport schema with a new service family + +## Required Invariants + +### 1. `backend_id` remains the canonical selector + +All explicit backend-targeting paths must continue to resolve by the effective inventory entry’s derived `backend_id`, not by `agent_id`, binary path, backend kind alone, or "the only eligible member." + +### 2. Exact selection must fail closed + +If a named backend is absent, duplicated, disabled, wrong-scope, not policy-allowed, missing required protocol/capabilities, or not runtime-realizable, the path must stop with a concrete error. There must be no fallback to another backend and no fallback to legacy singleton selection. + +### 3. Scope is part of the contract + +The same `backend_id` request must only match entries in the requested scope. A host-targeted path cannot silently select a world-scoped runtime, and a world-targeted path cannot silently select a host runtime. + +### 4. Retained world-member reuse must key by backend id + +Within one orchestration session and authoritative world generation, retained runtime lookup must be by the named backend. Reuse must mean "reuse the retained runtime for `cli:codex`" rather than "reuse whichever member exists." + +### 5. Multiple eligible world members must be able to coexist + +For this SOW, coexistence means more than inventory eligibility. If effective inventory contains more than one valid world-scoped backend, the system must be able to retain and reuse distinct live runtimes for those backends simultaneously within the same orchestration session and authoritative world generation without ambiguity. + +### 6. Transport identity must stay end-to-end + +`backend_id` must remain explicit in member dispatch, retained-runtime state, follow-up submit requests, and validation of those follow-up requests. World-member submission must never target a retained runtime whose backend identity does not exactly match the request. + +## Recommended Implementation Shape + +1. Keep `validate_exact_backend_selection(...)` in [validator.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/validator.rs) as the canonical selector for explicit backend-targeted paths, and demote `validate_member_selection(...)` to legacy/non-targeted use only. +2. Normalize inventory lookup around derived `backend_id` so all explicit runtime callers consume one selector contract and one family of errors. +3. Replace shell-side singleton reuse and readiness in [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:553) and [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:3577) with backend-aware retained-runtime management keyed by: + - `orchestration_session_id` + - `world_generation` + - `backend_id` +4. Widen the shell REPL and Linux `world-agent` retained-member cardinality together: + - the shell can no longer model retained world members as one `Option` + - `world-agent` can no longer model retained world members as a global singleton registry + - both sides must support backend-aware lookup and validation for more than one simultaneously retained live member +5. Keep follow-up submit validation strict: the submitted turn must still match the retained participant, orchestrator participant, world binding, and `backend_id`. + +## Concrete Work Breakdown + +### A1. Harden the canonical exact-backend selector + +- Audit explicit backend-targeted paths in [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) and adjacent runtime helpers so the already-landed exact selector stays authoritative anywhere a named backend is supplied. +- Ensure error wording distinguishes: + - backend missing from effective inventory + - backend present but wrong-scope + - backend present but disallowed by policy + - backend present but unrealizable + - backend duplicated across effective entries in the same scope + +### A2. Retained-member lookup and reuse by backend id + +- Replace `live_member_for_generation(...)` semantics in [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:2837) so lookup is keyed by `backend_id` plus authoritative world generation, not by `agent_id` alone. +- Preserve fail-closed behavior if more than one live participant claims the same backend key for the same orchestration session and world generation. + +### A3. Replace shell-side singleton retained-member state + +- Replace `member_runtime: Option` in [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:553) with a backend-aware retained-member collection that can hold more than one live world member at once. +- Refactor `ensure_member_runtime_ready_for_descriptor(...)` in [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:3577) and `reconcile_member_runtime_generation(...)` in [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:3527) so readiness, invalidation, replacement, and reuse operate on the targeted backend entry instead of a single global member slot. +- Ensure member startup preparation consumes the exact selected `RuntimeSelectionDescriptor`. +- Keep policy allowlist enforcement against `descriptor.backend_id`. + +### A4. Replace `world-agent` singleton retained-member state + +- Refactor [member_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs:391) so retained runtime ownership is not globally singleton. +- Add a backend-aware retained lookup that supports coexistence of `cli:codex` and `cli:claude-code` in the same live world. +- Retain strict duplicate detection for the same backend key. + +### A5. Regression coverage + +- Extend validator tests in [validator.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/validator.rs) for exact-backend failure cases and scope-specific errors, without regressing the already-landed exact-match path. +- Add REPL/runtime tests in [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) that prove: + - multiple eligible world members can be retained live simultaneously within one orchestration session/world generation without ambiguity + - explicit launch for `cli:codex` reuses only the retained `cli:codex` runtime + - explicit launch for `cli:claude-code` reuses only the retained `cli:claude-code` runtime and does not collide with `cli:codex` + - duplicate live retained entries for the same backend still fail closed +- Add Linux `world-agent` tests in [member_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs) covering multi-member retention and backend-id submit validation. + +## Validation Expectations + +At minimum, this slice should ship with: + +- `cargo test -p shell validate_exact_backend_selection` +- targeted REPL/runtime tests covering backend-id keyed reuse and coexistence +- `cargo test -p world-agent member_runtime` + +Expected assertions: + +- the currently landed exact targeted-turn route continues to resolve `cli:codex` and `cli:claude-code` by backend id +- inventory with both `cli:codex` and `cli:claude-code` world members no longer fails just because more than one eligible member exists in adjacent launch/reuse paths +- both backends can remain simultaneously retained live runtimes within one orchestration session/world generation, and each targeted launch/reuse path selects the correct retained runtime by `backend_id` +- explicit backend targeting still fails closed when the named backend is absent or wrong-scope +- explicit backend targeting still fails closed when policy disallows the selected backend +- retained member follow-up submission is rejected if the request `backend_id` does not match the retained runtime +- two simultaneously retained live members with distinct backend ids can coexist; two retained members with the same backend key cannot + +## Explicit Non-Goals + +- Do not introduce implicit default-agent routing. +- Do not add a backend aliasing layer beyond existing derived `backend_id` rules. +- Do not relax policy gating in order to make backend selection more permissive. +- Do not broaden this slice into session-resume UX, transcript modeling, or non-interactive agent callers. + +## Open Risks + +- The repo already has partially landed exact-backend behavior, so the main risk is leaving mixed identity models in place: exact selection in the targeted-turn route, but singleton or `agent_id`-based reuse deeper in the runtime. +- The largest behavioral shift in this slice is cardinality on both sides of the seam: the shell REPL no longer holding one member runtime slot, and `world-agent` no longer holding one retained member globally. +- If any persisted participant records or live-state helpers implicitly assume one world member at a time, those assumptions will need to be surfaced and tested before the slice is considered closed. diff --git a/llm-last-mile/17-turn-submit-reuse-for-selected-member.md b/llm-last-mile/17-turn-submit-reuse-for-selected-member.md new file mode 100644 index 000000000..4f97fa026 --- /dev/null +++ b/llm-last-mile/17-turn-submit-reuse-for-selected-member.md @@ -0,0 +1,250 @@ +# SOW B: Finish Turn Submit + Reuse For Selected Member + +Status: implementation-oriented split draft. This document narrows the second half of [15-targeted-repl-agent-turns-linux-first.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/15-targeted-repl-agent-turns-linux-first.md:1). It does not re-open the already-landed REPL grammar or claim user-turn submit is absent. Instead, it defines the remaining work needed to finish, harden, and deliberately surface selected-backend follow-up turn submission and reuse for the current REPL-first caller path, while explicitly keeping new CLI submit surfaces out of scope. + +## Objective + +Keep the current Linux-first REPL targeted-turn path intact, then finish the remaining product work so that selected-backend follow-up turns: + +- submit real user prompts into the retained active session for the selected backend, +- reuse the retained world member when that member is still valid for the selected backend and world generation, +- relaunch only when the retained runtime is unavailable or stale, +- keep one coherent submit/reuse contract across the current host-targeted and world-targeted REPL paths, +- and fail closed with clear operator-facing errors and docs. + +This SOW is intentionally narrow. It is about finishing submit/reuse semantics for the selected member, not redesigning orchestration wholesale. + +## Why This Is Needed + +The repo is already past the "can we submit turns at all?" stage. + +What is already landed: + +- the REPL recognizes exact single-line targeted syntax and routes it before shell fallback in [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:683) and [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:4550) +- host follow-up turns already resume the retained active orchestrator session in [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:3781) +- Linux world follow-up turns already go through a typed submit route in [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:3889), [service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs:1472), and [member_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs:232) +- the world submit request shape and identity contract already exist in [lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/agent-api-types/src/lib.rs:836) +- current Linux tests already prove grammar rejection, typed world submit, host fail-closed routing, and same-generation member reuse in [repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs:1776), [repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs:1841), [repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs:1964), and [repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs:2029) + +What is still incomplete: + +- the product surface is still REPL-first; the gap matrix still records non-interactive caller work as open in [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md:106) +- exact backend targeting is only partially productized; the matrix still describes it as REPL-first in [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md:114) +- `substrate -c` is still shell wrap mode in [cli.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/cli.rs:27) and [plan.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/invocation/plan.rs:620), so this slice keeps CLI submit parity out of scope rather than reopening command-mode semantics +- world-targeted follow-up turns still fail closed off Linux in [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:4014), and the gap matrix still tracks macOS/Lima parity as open in [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md:118) +- active-session resolution outside the current REPL path still assumes a single live orchestrator session in [state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:433), and operator surfaces such as `agent status` / toolbox still preflight that single-session model in [agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:421) and [agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:972) +- the operator-facing contract is spread across code paths and tests, but not yet fully tightened as a deliberate user-facing submit/reuse surface + +This SOW exists to finish that remaining layer without pretending the current Linux-first REPL path does not already work. + +## Relationship To Existing Docs + +- [15-targeted-repl-agent-turns-linux-first.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/15-targeted-repl-agent-turns-linux-first.md:1) is the broader combined slice +- [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md:105) now records the repo truth that targeted REPL grammar and Linux-first turn submission are already landed, while broader caller-surface and parity work remains open + +This document should be read as "SOW B after the initial Linux-first REPL submit path landed." + +## Current Relevant Code Surfaces + +### Targeted REPL parse and route entry points + +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:683) + - recognizes `::...` input before shell fallback +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:4550) + - enforces exact `:: ` parsing on a single line +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:2704) + - resolves targeted turns by exact backend id and fails closed when the requested host backend is not the active orchestrator backend + +### Current submit paths + +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:3781) + - host follow-up submit path using session-resume extensions against the retained host runtime +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:3889) + - Linux world follow-up submit path shaping `MemberTurnSubmitRequestV1` +- [crates/world-agent/src/service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs:1472) + - typed `/v1/member_turn/stream` handoff into `MemberRuntimeManager` +- [crates/world-agent/src/member_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs:232) + - retained-member submit path that resumes the surfaced UAA session id + +### Reuse and relaunch decision points + +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:3527) + - generation reconciliation for stale retained members +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:3577) + - ensure-ready path that reuses an existing member runtime when the descriptor and world binding are still valid +- [crates/world-agent/src/member_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs:444) + - rejects concurrent submitted turns against one retained member slot + +### Request shape and identity validation + +- [crates/agent-api-types/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/agent-api-types/src/lib.rs:836) + - `MemberTurnSubmitRequestV1` +- [crates/world-agent/src/member_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs:803) + - retained identity drift checks for orchestration session id, participant ids, backend id, world id, and world generation + +### Caller surfaces that still constrain the remaining work + +- [crates/shell/src/execution/cli.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/cli.rs:27) + - `-c/--command` still means shell command execution +- [crates/shell/src/execution/invocation/plan.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/invocation/plan.rs:620) + - non-interactive command mode still maps to `ShellMode::Wrap` +- [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:433) + - current single-live-session resolution helper +- [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:421) + - status still preflights a single live session +- [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:972) + - toolbox status also assumes one selected live orchestrator session + +## In Scope + +- finish and harden selected-backend follow-up turn submission on top of the already-landed REPL Linux-first path +- define exact reuse-vs-relaunch behavior for follow-up turns to the selected world member +- make submit/reuse semantics explicit and consistent across the current REPL host-targeted and world-targeted paths +- tighten operator-facing errors for malformed syntax, bad backend selection, stale/missing retained state, unsupported platform paths, and concurrent submit collisions +- update the operator docs that describe the contract, current Linux-first limitations, and any new submit surface + - for this slice, "new submit surface" means the already-landed REPL targeted-turn path plus its operator docs and help text, not a new CLI caller + +## Out Of Scope + +- redesigning plain REPL input into implicit default-agent routing +- changing `substrate -c` semantics by implication +- broader `substrate agent start|resume|fork|stop` productization +- replacing the retained-runtime ownership split between shell-owned host control and world-owned member control +- full toolbox product work +- broad status-surface redesign unrelated to submit/reuse +- cross-platform parity beyond what this slice needs to document and fail closed around + +## Required Semantics And Invariants + +### 1. Do not regress the landed REPL contract + +The current exact grammar and shell-first fallback posture must remain true. + +- `:: ` stays the only targeted follow-up grammar for the REPL +- malformed targeted syntax must fail before shell fallback +- plain REPL input remains shell execution + +### 2. Follow-up submit must reuse retained ownership when valid + +For a selected backend that already has a valid retained runtime: + +- host follow-up turns must resume the retained host orchestrator session rather than relaunching the orchestrator +- Linux world follow-up turns must submit into the retained selected member rather than relaunching a sibling member +- successful follow-up submit must not swap participant identity or create duplicate live members + +### 3. Relaunch is allowed only for real invalidation or drift + +Relaunch is correct only when the retained runtime is no longer valid for the requested turn. + +- if the retained member is missing, invalidated, or stale for the current `world_generation`, the shell may re-ready the member before submit +- same-generation follow-up turns must prefer reuse +- submit itself must not be the trigger that silently creates unnecessary replacement members + +### 4. Exact backend selection remains mandatory + +- backend routing must remain by exact `backend_id` +- host follow-up turns may only target the active orchestrator backend for the current REPL session +- world follow-up turns must not revert to "pick the one eligible member" heuristics once a backend was explicitly named + +### 5. State authority stays in the existing runtime store and retained handles + +- active orchestration and participant identity stays in the existing state store under `~/.substrate/run/agent-hub/...` +- host follow-up submit continues to use the retained surfaced UAA session handle +- world follow-up submit continues to use the persisted participant/world identity plus the retained surfaced UAA session id inside `world-agent` + +### 6. No new CLI caller surface is introduced in this slice + +- the submit/reuse hardening work in this SOW applies to the current REPL targeted-turn paths only +- `substrate -c` remains shell wrap mode +- any later non-REPL caller must be a separate slice with its own explicit session-selection contract when more than one live orchestration session exists + +### 7. Operator-facing failures must be deliberate and explainable + +Errors should stay precise and fail closed. Relevant existing examples already exist in code: + +- malformed targeted syntax in [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:685) +- wrong active host backend in [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:2724) +- non-Linux world-targeted follow-up rejection in [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:4021) +- missing retained member / missing surfaced session id / active submitted turn collision in [member_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs:241), [member_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs:249), and [member_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs:444) + +This slice should standardize how those errors surface to operators across the current REPL targeted-turn paths rather than leaving them as implementation accidents. + +## Recommended Implementation Shape + +1. Treat the current REPL Linux-first path as the reference contract, not as throwaway scaffolding. + +2. Factor the shared selected-backend submit/reuse logic behind one internal helper seam so the host-targeted and world-targeted REPL paths do not fork their own route-selection, readiness, and submit behavior. + +3. Preserve the current ownership split: + - host follow-up submits stay shell-owned and session-resume based + - world follow-up submits stay `world-agent`-owned and use the typed member-turn route + +4. Keep CLI parity out of scope for this slice instead of changing `-c` by stealth. + - `substrate -c` stays wrap mode. + - If a later slice introduces a CLI submit surface, it should be explicit and should require a deterministic active-session selector when ambiguity exists. + +5. Tighten reuse-vs-relaunch bookkeeping around the already-landed ready/reconcile paths. + - reuse the retained member when descriptor, backend id, world id, and world generation still match + - relaunch only after invalidation, drift, or missing runtime state + - keep submitted-turn collision behavior explicit instead of letting overlapping calls race + +6. Update docs and help text so operators can discover: + - the exact targeted-turn grammar + - the Linux-first scope of world-member follow-up submit/reuse + - the explicit error cases and what action the operator should take next + +## Concrete Work Breakdown + +1. Codify the current selected-turn contract. + - Document the already-landed REPL behavior and Linux-first retained-member submit path. + - Remove any remaining stale doc language that still implies submit is wholly missing. + +2. Consolidate shared submit/reuse orchestration. + - Centralize exact backend routing, retained-runtime readiness checks, and host-vs-world submit branching. + - Keep REPL wiring thin around that shared core. + +3. Tighten the REPL-only contract instead of widening the caller set. + - Keep `-c` unchanged. + - Remove stale doc language that implies a new CLI submit surface is part of this landing. + +4. Harden reuse and collision behavior. + - Make reuse the default for same-generation follow-up turns. + - Preserve fail-closed behavior on identity drift, missing retained session ids, and concurrent submitted turns. + +5. Finish operator-facing docs and errors. + - Update [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md:105) and any help text or operator docs touched by the new surface. + - Ensure surfaced errors describe whether the operator should retry, target a different backend, restart the REPL/session, or switch to Linux for world follow-up turns. + +## Validation Expectations + +Existing regression coverage that must keep passing: + +- [repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs:1776) + - malformed targeted syntax fails before shell fallback +- [repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs:1841) + - world targeted follow-up turns use typed submit without relaunching the member +- [repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs:1964) + - host targeted follow-up turns reject non-active host backends +- [repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs:2029) + - same-generation world commands reuse the retained member runtime + +Additional validation expected for this slice: + +- tests that operator-visible errors remain fail-closed and do not fall back to shell execution +- regression coverage for concurrent submitted-turn rejection and identity-drift rejection +- coverage that non-Linux paths keep the explicit fail-closed posture for world-targeted follow-up turns until parity is intentionally landed + +Recommended commands: + +```bash +cargo test -p shell --test repl_world_first_routing_v1 -- --nocapture +cargo test -p world-agent -- --nocapture +cargo test -p agent-api-types -- --nocapture +``` + +## Explicit Non-Goals + +- Do not claim the repo still lacks targeted REPL follow-up submit or Linux retained-member reuse. +- Do not silently broaden this slice into a general orchestration control-plane redesign. +- Do not convert shell command surfaces into agent caller surfaces without an explicit contract change. +- Do not promise non-Linux world-member follow-up turn parity before the underlying backend path is actually wired. diff --git a/llm-last-mile/18-status-surface-and-session-handle-hardening.md b/llm-last-mile/18-status-surface-and-session-handle-hardening.md new file mode 100644 index 000000000..19fec861f --- /dev/null +++ b/llm-last-mile/18-status-surface-and-session-handle-hardening.md @@ -0,0 +1,279 @@ +# SOW: Harden Agent Status and Session-Handle Contracts For Public Control Surfaces + +Status: implementation-oriented follow-on draft. This SOW does not claim that `substrate agent status` or session persistence are missing. Live runtime state, canonical session/participant snapshots, participant-aware producer-side trace rows, and nested gateway tuple validation are already landed. The remaining gap is narrower and operator-facing: stop `agent status` from aborting wholesale on strict live-session selection failures, make trace-only fallback stop collapsing sibling participants, and clean up the local-vs-upstream session-handle contract enough that future public `start|resume|fork|stop` surfaces do not inherit ambiguous identifiers by default. + +## Objective + +Harden only the status/addressability layer needed before public control-plane work is safe: + +- make `substrate agent status` degrade cleanly when parent-session selection is ambiguous or stale, +- make trace-only fallback participant-aware so same-agent sibling participants inside one orchestration session do not collapse, +- and lock one minimal public addressability contract: + - public parent-session selectors use `orchestration_session_id` + - `participant_id` remains a Substrate runtime/lineage identifier, not the primary public session selector + - surfaced backend-native session ids remain internal implementation detail for now + +This SOW does not add public `start|resume|fork|stop`. It only hardens the read-side and naming contract those later surfaces would rely on. + +## Why This Is Needed + +The repo already has the core primitives this slice should reuse rather than replace: + +- the gap matrix now records live runtime persistence, trace production, and status/live fallback as real, not hypothetical, in [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md:76) +- the same matrix also calls out the remaining open rows this SOW should close: + - `Session handles` in [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md:112) + - `Status ambiguity handling` in [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md:125) + - `Trace-only participant-aware fallback` in [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md:126) +- `AgentRuntimeStateStore` already has a permissive session-record model that can preserve warnings instead of panicking in [state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:858) +- live runtime status rows already preserve `participant_id` in [agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:1140) +- same-agent concurrent live world rows from store-owned session records are already supposed to remain visible, as pinned in [agent_successor_contract_ahcsitc0.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_successor_contract_ahcsitc0.rs:2528) + +What is still wrong is a narrower mismatch between those landed primitives and the selected read-side/operator contract: + +1. `build_status_report(...)` still hard-preflights `resolve_single_live_session_for_agent(...)` in [agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:421), [agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:446), and [state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:433). +2. That strict helper is correct for active-control authorization, but it intentionally fails closed on: + - multiple active parent sessions, + - missing `active_session_handle_id`, + - stale active-handle references, + - inactive selected participants, + - or live children without an active parent, + as covered by tests in [state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:1936) and [agent_successor_contract_ahcsitc0.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_successor_contract_ahcsitc0.rs:1468). +3. Reusing that same strict helper for `agent status` means a read-only surface aborts wholesale even though the store can already enumerate partial records and warnings. +4. The trace fallback path is still keyed too coarsely: + - pure-agent trace rows use `(orchestration_session_id, agent_id)` in `pure_session_key(...)` at [agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:1252) + - nested parent-run selection is keyed by `(orchestration_session_id, agent_id)` in [agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:595) + - fallback suppression is only `(orchestration_session_id, agent_id, execution.scope)` in [agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:1185) +5. That coarse key shape is enough for many rows, but it is not enough once one orchestration session can legitimately contain sibling participants for the same `agent_id`. The gap matrix already names this as open in [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md:126). +6. The persisted session-handle vocabulary is still overloaded: + - `OrchestrationSessionRecord.active_session_handle_id` stores the selected participant id in [orchestration_session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/orchestration_session.rs:35) + - `AgentRuntimeParticipantHandle.session_handle_id` is skipped from serialization and mirrors `participant_id` in [session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs:45) + - deserialization still aliases legacy `session_handle_id`, `parent_session_handle_id`, and `resumed_from_session_handle_id` onto participant ids in [session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs:528) + - the real surfaced upstream backend handle lives separately as `internal.uaa_session_id` in [session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs:75) + +That identity mix is survivable for internal plumbing, but it is the wrong base contract for a public control plane. + +## Relationship To Existing Slices + +- This SOW is the narrow read-side follow-on to the current gap-matrix recommendation to "make `substrate agent status` degrade cleanly on ambiguity/stale parent linkage and make its trace fallback participant-aware" in [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md:179). +- It intentionally does not reopen backend targeting, turn submit/reuse, or the broader public caller-surface design. It only fixes the status/addressability substrate those later slices need. + +## Current Relevant Code Surfaces + +### Strict live-session selection and the current read-side mismatch + +- [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:433) + - `resolve_single_live_session_for_agent(...)` + - intentionally strict control-plane selector +- [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:858) + - `build_session_record(...)` + - already preserves warnings and incompleteness instead of crashing +- [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:421) + - `build_status_report(...)` + - currently aborts status rendering by preflighting the strict single-live-session selector + +### Trace fallback and selected-row correlation + +- [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:473) + - pure-agent trace projection +- [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:595) + - selected parent-run correlation for nested rows +- [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:1185) + - fallback suppression key +- [crates/shell/tests/agent_successor_contract_ahcsitc0.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_successor_contract_ahcsitc0.rs:2263) + - invalidated tombstones already beat stale trace fallback +- [crates/shell/tests/agent_successor_contract_ahcsitc0.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_successor_contract_ahcsitc0.rs:2528) + - store-owned same-agent concurrent rows already must remain visible + +### Local identity, participant lineage, and surfaced upstream handles + +- [crates/shell/src/execution/agent_runtime/orchestration_session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/orchestration_session.rs:24) + - authoritative parent session record, including `orchestration_session_id` and `active_session_handle_id` +- [crates/shell/src/execution/agent_runtime/session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs:45) + - local participant handle fields +- [crates/shell/src/execution/agent_runtime/session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs:75) + - `internal.uaa_session_id` +- [crates/shell/src/execution/agent_runtime/session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs:528) + - legacy wire aliases that currently conflate "session handle" and participant lineage + +## In Scope + +- remove the strict single-live-session preflight from `substrate agent status` +- degrade ambiguous or stale parent linkage into readable status output instead of a whole-surface abort +- make trace fallback participant-aware for pure-agent selection, nested parent-run correlation, and fallback suppression +- define one minimal public addressability contract: + - later public control surfaces address orchestrations by `orchestration_session_id` + - `participant_id` remains visible for diagnostics/lineage + - `internal.uaa_session_id` stays internal and is not the public control-plane handle +- preserve backward-compatible reads for legacy `session_handle_id` aliases +- add focused regression coverage for the above and nothing broader + +## Out Of Scope + +- adding `substrate agent start|resume|fork|stop` +- redesigning backend selection or targeted-turn grammar +- changing the runtime authority model for host orchestrator or world member ownership +- widening toolbox into a mutation surface +- changing macOS/Lima parity status +- rewriting trace schema producers beyond the minimal identity fields already present in `AgentEvent` +- replacing existing persistence directories or compatibility dual-write layout + +## Required Semantics And Invariants + +### 1. Read-only status must degrade; control selectors must stay strict + +`substrate agent status` should no longer depend on `resolve_single_live_session_for_agent(...)` as a precondition for rendering any output. + +At the same time: + +- helpers used to authorize active control should keep failing closed on ambiguity or stale linkage +- `toolbox env`, future mutating agent commands, and any public `resume|fork|stop` path should continue to require explicit authoritative selection +- trace history must not back-authorize control operations, as already pinned in [agent_successor_contract_ahcsitc0.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_successor_contract_ahcsitc0.rs:4129) + +### 2. Live runtime state stays authoritative over trace fallback + +The current precedence remains correct and must stay true: + +- live authoritative participant/session records win +- invalidated authoritative participant tombstones suppress stale trace rows +- trace fallback only fills gaps the store cannot fill + +This is already the intended behavior in [agent_successor_contract_ahcsitc0.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_successor_contract_ahcsitc0.rs:2208) and [agent_successor_contract_ahcsitc0.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_successor_contract_ahcsitc0.rs:2263). This SOW should harden, not invert, that ordering. + +### 3. Participant identity must survive trace-only fallback + +Where producer-side trace rows already include `participant_id`, `parent_participant_id`, and `resumed_from_participant_id`, the fallback reader must use those fields as part of its identity model instead of collapsing by only `(orchestration_session_id, agent_id)`. + +Required rule: + +- two sibling participants for the same agent within one orchestration session must be able to remain distinct in fallback selection and suppression if the trace evidence distinguishes them. + +### 4. Public addressability must stop conflating three identities + +For public/operator-facing contracts after this slice: + +- `orchestration_session_id` is the only required public control-plane selector +- `participant_id` is a subordinate runtime identity for status, lineage, and debugging +- surfaced backend session identity remains separate and internal, backed today by `internal.uaa_session_id` + +Required rule: + +- do not expose `participant_id` under the public name `session_handle_id` +- do not ask operators to target `uaa_session_id` directly on public control surfaces + +### 5. Compatibility reads may stay; public docs and response shapes should use canonical names + +The repo already supports reading older wire names through serde aliases in [session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs:528). + +That compatibility may remain, but: + +- new public docs and new JSON/text status contracts should prefer canonical names +- later public control-plane surfaces should not require users to understand legacy alias names + +### 6. Do not weaken malformed-row validation + +This SOW is about ambiguity/linkage degradation, not about accepting corrupt selected rows. + +Current fail-closed validation should remain for: + +- malformed world identity on selected world-scoped status rows in [agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:544) +- malformed nested tuple fields in [agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:626) +- malformed nested parent correlation in [agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:668) + +The degradation target is session-selection ambiguity and stale parent linkage, not malformed selected tuples. + +## Recommended Implementation Shape + +1. Split selection logic cleanly: + - strict active-session selection remains for control/authorization + - permissive enumeration/projection is used for `agent status` + +2. Rework `build_status_report(...)` around: + - `list_live_sessions()` + - session-record warnings/incompleteness + - authoritative live participants + - participant-aware trace fallback + +3. Freeze the public naming contract now: + - public parent-session handle: `orchestration_session_id` + - diagnostic/runtime row id: `participant_id` + - backend-native session id: internal only + +4. Keep compatibility at the serde boundary only. + - Read legacy `session_handle_id` aliases. + - Stop treating them as canonical semantics in new reporting/control code. + +## Concrete Work Breakdown + +### 1. Separate status rendering from active-control preflight + +- Remove the unconditional `resolve_single_live_session_for_agent(...)` preflight from `build_status_report(...)` in [agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:446). +- Replace it with status-specific logic that: + - lists authoritative live sessions and participants, + - preserves rows that are internally valid, + - and degrades broken parent-selection state into non-fatal warnings or omitted affected rows instead of aborting all status output. +- Keep the strict helper in place for `toolbox env`, future start/resume/fork/stop surfaces, and any other operation that needs to authorize a live control target. + +### 2. Make trace fallback participant-aware + +- Change the pure-agent fallback key in [agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:1252) so it incorporates `participant_id` when trace evidence supplies it. +- Change fallback suppression keys in [agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:1185) so one live or invalidated participant suppresses only its matching fallback row, not every sibling row for the same `(orchestration_session_id, agent_id, execution.scope)`. +- Make nested parent-run selection in [agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:595) participant-aware when selected parent rows carry participant identity, while preserving current run-id-based fail-closed validation. + +### 3. Clean up the public addressability contract + +- Canonicalize the runtime/status vocabulary around: + - `orchestration_session_id` + - `participant_id` + - internal backend-native session id +- Stop treating `active_session_handle_id` and `session_handle_id` as public-facing semantics when they are really storing participant identity today. +- Keep backward-compatible reads for: + - `session_handle_id` + - `parent_session_handle_id` + - `resumed_from_session_handle_id` + but shift comments, docs, and any new response shapes toward canonical names. + +### 4. Add the minimum status/read-side reporting needed for degraded cases + +- Extend `agent status` output so degraded records are still intelligible. +- The exact text/JSON warning shape is implementation-defined, but the surface must make it clear when: + - a row came from authoritative live runtime state + - a row came from trace fallback + - or some candidate rows were omitted due to stale/incomplete parent linkage + +### 5. Lock the contract with focused regression tests + +- Add status tests proving that ambiguity or stale active-parent linkage no longer aborts the entire `agent status` surface. +- Add trace-only tests proving that sibling participants for the same `agent_id` inside one orchestration session stay distinct when participant-aware trace fields are present. +- Add compatibility tests proving legacy `session_handle_id` aliases still deserialize, while canonical reporting uses participant/session terminology instead. + +## Validation Expectations + +At minimum, this slice should ship with targeted coverage in: + +- [crates/shell/tests/agent_successor_contract_ahcsitc0.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_successor_contract_ahcsitc0.rs:2208) +- [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:1936) +- [crates/shell/src/execution/agent_runtime/session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs:564) + +Expected assertions: + +- `substrate agent status --json` still prefers live authoritative runtime rows over trace fallback. +- Invalidated authoritative tombstones still suppress stale trace rows. +- Same-agent concurrent live rows from authoritative session records remain visible. +- Trace-only fallback no longer collapses same-agent sibling participants when participant-aware lineage is available. +- Ambiguous or stale active-parent selection no longer aborts all `agent status` output. +- strict active-session selector tests still fail closed on ambiguity or stale linkage. +- Legacy `session_handle_id`-style snapshots still deserialize. +- Canonical status/control reporting treats `orchestration_session_id` as the public handle and `participant_id` as subordinate runtime identity. + +## Explicit Non-Goals + +- Do not use this slice to sneak in public `start|resume|fork|stop`. +- Do not relax active-control authorization by allowing trace-only history to pick a control target. +- Do not redesign on-disk storage layout or remove compatibility aliases yet. +- Do not broaden this into toolbox mutation work, submit/reuse behavior, or non-interactive invocation design. + +## Open Risks + +- The biggest risk is relaxing the wrong helper. If `resolve_single_live_session_for_agent(...)` is weakened instead of factoring status onto a different read path, future control surfaces will inherit unsafe ambiguity. +- The second risk is only partially fixing trace fallback, for example making pure-agent projections participant-aware but leaving suppression or nested-parent correlation keyed too coarsely. +- The session-handle cleanup must avoid a flag day. The repo already reads legacy names, and the safe path is compatibility reads plus canonical public naming, not storage churn for its own sake. diff --git a/llm-last-mile/19-public-agent-control-surfaces.md b/llm-last-mile/19-public-agent-control-surfaces.md new file mode 100644 index 000000000..6cf9c6afa --- /dev/null +++ b/llm-last-mile/19-public-agent-control-surfaces.md @@ -0,0 +1,202 @@ +# SOW: Public Agent Control Surfaces + +Status: implementation-oriented follow-on draft. This SOW is the narrow public control-plane follow-on after [18-status-surface-and-session-handle-hardening.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/18-status-surface-and-session-handle-hardening.md:3). It defines the minimal public `substrate agent start|resume|fork|stop` family, the selector contract those commands must use, and the fail-closed posture for Linux-first world-bound work. It does not absorb the preceding status/session-handle hardening slice, it does not redesign `substrate -c`, and it does not redesign the REPL `:: ` grammar. + +## Objective + +Expose one narrow public control surface under `substrate agent` that: + +- makes session creation and shutdown first-class instead of REPL-only side effects, +- reserves explicit public entry points for `resume` and `fork` instead of leaving them implied, +- uses `orchestration_session_id` as the only public session handle, +- keeps backend-native session ids internal, +- and stays fail-closed anywhere the current runtime cannot prove ownership, linkage, platform posture, or policy allowability. + +This slice is about public control-surface shape and bounded productization of already-landed runtime pieces. It is not a daemonization project and not a general hub redesign. + +## Current Repo Truth + +The current repo already proves most of the internal control-plane substrate: + +- the public CLI still stops at inspection surfaces. `AgentAction` exposes `List`, `Status`, `Doctor`, and `Toolbox` only in [crates/shell/src/execution/cli.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/cli.rs:471) +- host-runtime startup already exists internally in `start_host_orchestrator_runtime(...)` and its prepared variant in [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:1802) and [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:1930) +- internal shutdown already exists in `shutdown_host_orchestrator_runtime(...)` in [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:4090) +- session-resume extension shaping already exists in `AGENT_API_SESSION_RESUME_V1` and `build_session_resume_extension(...)` in [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:1572) and [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:3776) +- inventory capability flags already model `session_start`, `session_resume`, `session_fork`, and `session_stop` in [crates/shell/src/execution/agent_inventory.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_inventory.rs:89) and [crates/shell/src/execution/agent_runtime/validator.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/validator.rs:99) +- authoritative session persistence already uses `orchestration_session_id` at the parent-session layer in [crates/shell/src/execution/agent_runtime/orchestration_session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/orchestration_session.rs:25) +- surfaced backend-native session ids are still stored separately as internal `uaa_session_id` in [crates/shell/src/execution/agent_runtime/session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs:81) +- the gap matrix already says the surface is not yet public and should be treated as follow-on work after the current status/session-handle hardening concerns in [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md:104), [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md:110), [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md:112), and [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md:179) + +The honest summary is: + +- `start` and `stop` are partially landed internally because startup, attached control retention, persistence, and shutdown plumbing already exist inside the REPL-owned runtime, +- `resume` and `fork` have partial ingredients such as capability flags and session-resume extension wiring, but they are not yet public Substrate control-plane actions, +- and the current control owner is still the long-lived shell process rather than a reusable daemon, as the gap matrix explicitly notes in [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md:108). + +## Scope + +This SOW defines only the minimal public command family: + +```text +substrate agent start --backend [--json] +substrate agent resume --session [--json] +substrate agent fork --session [--json] +substrate agent stop --session [--json] +``` + +The command family is intentionally narrow: + +- `start` creates one orchestration session for one exact backend +- `resume` attempts to re-enter one existing orchestration session +- `fork` creates a successor orchestration session from one existing orchestration session +- `stop` requests authoritative shutdown of one existing orchestration session + +No command in this slice takes a prompt payload. Prompt submission remains the existing REPL targeted-turn surface from [15-targeted-repl-agent-turns-linux-first.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/15-targeted-repl-agent-turns-linux-first.md:3) and its follow-on hardening work, not this public control-plane slice. + +## Public Selector Contract + +The public selector contract must stay boring and exact. + +### 1. `orchestration_session_id` is the only public session handle + +For any command that targets an existing session, the only accepted public selector is: + +- `--session ` + +That id already exists as the parent-session authority in [orchestration_session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/orchestration_session.rs:25) and is the cleanest durable Substrate-owned handle. + +This slice must not accept any of these as the primary public selector: + +- `participant_id` +- `active_session_handle_id` +- `session_handle_id` +- `uaa_session_id` +- raw backend-native thread or run ids + +### 2. `uaa_session_id` stays internal + +The surfaced backend-native session id remains an internal runtime detail: + +- it is persisted as `internal.uaa_session_id` in [session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs:81) +- it is already used internally when the REPL submits a resumed host follow-up turn through `build_session_resume_extension(...)` in [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:3776) + +Public control commands must resolve from `orchestration_session_id` to any needed participant record or internal `uaa_session_id` through the state store. They must not ask operators to provide backend-native handles. + +### 3. `start` uses exact `backend_id` + +`start` must require: + +- `--backend ` + +and that selector must be an exact derived `backend_id`, not `agent_id`, not a display label, and not "the default agent." + +This follows the already-landed exact backend targeting posture from [16-explicit-backend-selection-by-backend-id.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/16-explicit-backend-selection-by-backend-id.md:14) and avoids reopening the unresolved broader routing/default-selection question called out in [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md:90). + +### 4. No fuzzy or "latest" resolution + +This slice must not introduce: + +- prefix matching on session ids, +- newest-session wins behavior, +- backend inference from agent id, +- or automatic fallthrough to the "only active session I found." + +Every selector path must be exact or fail closed. + +## Landing Posture + +Implementation order should follow the repo truth rather than pretend all four verbs are equally ready. + +### Phase A: publicize the already-landed internals for `start` and `stop` + +This phase should: + +- add public CLI entries for `start` and `stop` under `AgentAction` +- reuse the existing startup and shutdown plumbing in [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:1802) and [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:4090) +- emit `orchestration_session_id` as the public handle on success +- keep stop tied to authoritative-live ownership only + +This is public productization of internal start/stop, not net-new runtime invention. + +### Phase B: expose `resume` and `fork` on the same selector contract + +This phase should: + +- add public CLI entries for `resume` and `fork` +- resolve the selected session through the authoritative state store, not through trace history +- reuse internal session-resume wiring only behind the `orchestration_session_id` lookup boundary +- fail closed anywhere the runtime cannot prove that the selected session still has a valid active parent, a valid retained control owner, and any required world linkage + +This phase must stay honest that `resume` and `fork` are not public today and must not be papered over with "best effort" heuristics. + +## Linux-First World Posture + +Where the selected session is purely host-scoped orchestrator work, the command family can follow the existing host runtime path. + +Where the selected session depends on world-member posture, this slice stays Linux-first: + +- if a command requires authoritative shared-world ownership, current `world_generation`, or retained world-member reuse, it must only advertise success on the Linux path that already persists and validates those bindings +- macOS/Lima and Windows/WSL must fail closed for any control action that would require the not-yet-equivalent shared-world/member-runtime contract described as still missing in [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md:115) and [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md:117) +- public help text and error messages must say `unsupported_platform_or_posture` instead of implying parity that does not exist + +This slice must not weaken the existing Linux-first truth just to make the public CLI look symmetrical across platforms. + +## Fail-Closed Rules + +The public control surface must fail closed in these cases: + +1. Ambiguity + - more than one candidate session matches + - more than one candidate backend matches + - or a selected session contains conflicting live ownership state +2. Stale linkage + - the selected parent points at a missing or inactive selected participant + - the selected participant points at an inactive or missing parent + - or the selected world member is tied to stale world binding or stale world generation +3. Missing active parent + - the chosen `orchestration_session_id` exists historically but is no longer active enough to authorize control-plane actions +4. Unsupported platform or posture + - the selected action would require world-member control semantics that are not implemented on the current platform +5. World-boundary unavailable + - the action requires current world ownership, socket reachability, or generation continuity and the shell cannot prove it +6. Policy disallow + - the broker or runtime validator rejects the backend, scope, or action even though inventory advertises the capability bit + +The preceding status/session-handle slice already points at the right read-side authority helpers for this: `list_live_sessions()` and `resolve_single_live_session_for_agent(...)` in [state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:421) and [state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:433). This slice should consume that hardened authority model, not redefine it. + +## Non-Goals + +This SOW explicitly does not include: + +- absorbing or partially redoing [18-status-surface-and-session-handle-hardening.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/18-status-surface-and-session-handle-hardening.md:3) +- redesigning `substrate -c` or `--command`; that remains shell wrap mode per the gap matrix in [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md:106) +- redesigning the REPL targeted-turn grammar or broadening it beyond the current `:: ` path +- adding a default-agent surface or any heuristic backend/session picker +- making `uaa_session_id` public API +- promoting the REPL-owned attached-control runtime into a general daemon or toolbox mutation service + +## Acceptance Criteria + +This slice is done when all of the following are true: + +1. `substrate agent` publicly exposes `start`, `resume`, `fork`, and `stop` in the CLI surface. +2. `start` requires exact `--backend ` and returns `orchestration_session_id` on success. +3. `resume`, `fork`, and `stop` require exact `--session `. +4. No public command accepts or documents `uaa_session_id`. +5. Host-only success paths reuse existing internal runtime start/stop plumbing rather than reimplementing ownership logic. +6. World-sensitive paths remain Linux-first and fail closed elsewhere. +7. All ambiguity, stale-linkage, missing-parent, unsupported-platform/posture, world-boundary-unavailable, and policy-disallow cases return explicit operator-facing errors rather than fallback selection. +8. This slice lands without changing `substrate -c` semantics and without changing REPL targeted-turn grammar. + +## Testing Expectations + +At minimum, this slice should add targeted coverage for: + +- CLI parse coverage proving the four new subcommands exist and require the exact selectors above +- exact `backend_id` requirement for `start` +- exact `orchestration_session_id` requirement for `resume`, `fork`, and `stop` +- rejection of `uaa_session_id` as a public selector +- fail-closed behavior on ambiguity, stale linkage, missing active parent, unsupported platform/posture, unavailable world boundary, and policy disallow +- Linux-first success and non-Linux fail-closed cases for world-sensitive actions + +The right goal is not broad surface area. The right goal is one narrow public namespace that tells the truth about what the runtime can already prove. diff --git a/llm-last-mile/20-public-non-interactive-agent-caller-surface.md b/llm-last-mile/20-public-non-interactive-agent-caller-surface.md new file mode 100644 index 000000000..ceb730c30 --- /dev/null +++ b/llm-last-mile/20-public-non-interactive-agent-caller-surface.md @@ -0,0 +1,688 @@ +# SOW: Public Non-Interactive Agent Caller Surface + +Status: implementation-oriented follow-on draft. This SOW defines the next caller-surface +slice after the already-landed public control namespace and the already-landed REPL +targeted-turn path. + +It changes the public command model to: + +- start = new session + first real prompt +- turn = existing session + next turn +- reattach = restore retained ownership to an existing session + +This slice is intentionally narrow: make prompt-taking explicit on public start, keep +substrate -c / --command as shell wrap mode, require exact target selectors, and return +real streamed output to the caller. It does not redesign REPL grammar, does not widen +default-agent routing, and does not implicitly repurpose shell command mode. + +## Objective + +Expose one explicit non-interactive public prompt-taking model: + +substrate agent start --backend (--prompt | --prompt-file | +--prompt-file -) [--json] +substrate agent turn --session --backend (--prompt + | --prompt-file | --prompt-file -) [--json] + +This slice must: + +- keep substrate -c / --command unchanged as shell wrap mode, +- require exact backend_id targeting for every prompt-taking call, +- require exact orchestration_session_id plus exact backend_id for follow-up turns, +- make start create a new host-scoped orchestration session and submit the provided prompt + as the inaugural real turn, +- make turn submit one foreground follow-up turn into the exact retained backend inside the + exact orchestration session, +- stream caller-visible output during execution, +- fail malformed or empty prompt input before any runtime launch, +- forbid synthetic bootstrap text from being injected as public agent input on either start + or turn, +- preserve explicit lifecycle commands as non-prompt-taking operational surfaces, +- and reserve the word resume for a possible future true interactive/conversational resume + UX rather than owner-loop recovery. + +## Current Repo Truth + +The repo has already landed more than the older control-surface drafts assumed, but the +current public start shape is no longer acceptable as the product contract. + +What is already landed: + +- substrate agent start|resume|fork|stop are already public in ../crates/shell/src/ + execution/cli.rs:493 and handled in ../crates/shell/src/execution/agents_cmd.rs:291. +- REPL targeted follow-up turns are already public as exact :: syntax + in ../crates/shell/src/repl/async_repl.rs:710. +- Exact REPL turn dispatch already distinguishes host vs world follow-up routing in + dispatch_targeted_follow_up_turn(...) at ../crates/shell/src/repl/async_repl.rs:3180. +- Host follow-up turn submission is already real in submit_host_targeted_turn(...) at ../ + crates/shell/src/repl/async_repl.rs:4451. +- Linux world-member follow-up turn submission is already real in + submit_world_targeted_turn(...) at ../crates/shell/src/repl/async_repl.rs:4559, + submit_member_turn_stream(...) at ../crates/world-agent/src/service.rs:1472, and + submit_turn(...) at ../crates/world-agent/src/member_runtime.rs:249. +- Public control selection already fails closed on non-canonical selectors, stale linkage, + missing internal session ids, missing owners, and non-Linux world-sensitive posture in + resolve_public_control_target(...) at ../crates/shell/src/execution/agent_runtime/ + state_store.rs:589, with regression coverage at ../crates/shell/src/execution/ + agent_runtime/state_store.rs:2194. +- Public root agent start --backend ... is intentionally host-only in v1, with failure + coverage for world-only backends in ../crates/shell/tests/ + agent_public_control_surface_v1.rs:962. +- substrate -c is still shell wrap mode by contract in ../crates/shell/src/execution/ + cli.rs:27, ../crates/shell/src/execution/invocation/plan.rs:29, ../crates/shell/src/ + execution/invocation/plan.rs:619, and ../crates/shell/src/execution/invocation/ + tests.rs:26. + +What is wrong with the current public start shape: + +- AgentStartArgs currently takes only --backend plus --json, with no public prompt source, + in ../crates/shell/src/execution/cli.rs:456. +- The current public start path launches a detached hidden owner-helper with stdin, stdout, + and stderr all set to null in ../crates/shell/src/execution/agents_cmd.rs:231. +- The current retained-owner runtime also has an internal synthetic bootstrap prompt used + to establish long-lived control ownership in runtime_bootstrap_prompt(...) at ../crates/ + shell/src/repl/async_repl.rs:1758. + +That current shape means public start can create a live background session without a real +user prompt and rely on internal retained-control bootstrap semantics. Given the current +Codex integration posture, that is not acceptable as the product model: it pays context +cost up front for an artificial turn and preserves hidden-message behavior we now want to +eliminate from the public contract. + +What is true about the current public resume shape: + +- today’s resume is not conversational resume, +- it does not submit a turn, +- it does not reopen an interactive REPL surface, +- it reattaches a retained owner loop to an already-existing orchestration session. + +This SOW therefore uses reattach as the target public term for that lifecycle behavior and +treats resume as reserved for a possible future true interactive/conversational resume +surface. + +What remains open: + +- there is still no first-class non-interactive public prompt-taking caller under substrate + agent, +- public start still means detached host-only retained ownership rather than “first real + prompt,” +- turn still needs a public exact-target follow-up contract for sessions that may contain + multiple retained backends, +- the current public control resolver is session-centric and is not by itself sufficient + for exact follow-up targeting when a session may contain multiple retained backends, +- and no public non-interactive surface yet defines a caller-visible streaming return path. + +The gap matrix already calls the non-interactive caller surface out directly in ../ +AGENT_ORCHESTRATION_GAP_MATRIX.md:107 and recommends deliberate new caller contracts rather +than implicit -c changes in ../AGENT_ORCHESTRATION_GAP_MATRIX.md:179. + +## Relationship To Existing Slices + +This SOW consumes already-landed work and should not reopen it: + +- ./15-targeted-repl-agent-turns-linux-first.md established the REPL-first exact targeted- + turn contract, and that contract is now live. +- ./18-status-surface-and-session-handle-hardening.md hardened the status/addressability + substrate this slice should reuse. +- ./19-public-agent-control-surfaces.md scoped the public control plane; most of that + namespace is now landed, but this slice changes the public meaning of start, defines turn + as follow-up only, and renames the current public resume concept to reattach. + +## Scope + +This slice defines one public prompt-taking command family: + +substrate agent start --backend (--prompt | --prompt-file | +--prompt-file -) [--json] +substrate agent turn --session --backend (--prompt + | --prompt-file | --prompt-file -) [--json] + +It also renames one lifecycle concept in the target public contract: + +substrate agent reattach --session [--json] + +The command family is intentionally narrow: + +- start --backend ... means create one new host-scoped orchestration session for that exact + backend and submit exactly one inaugural real turn in the foreground. +- turn --session ... --backend ... means submit exactly one foreground follow-up turn into + that exact backend inside that exact orchestration session. +- reattach --session ... means restore retained ownership to an existing orchestration + session and does not submit a prompt. +- backend_id is always required for prompt-taking calls. +- orchestration_session_id is required for follow-up turns and lifecycle recovery calls. +- exactly one prompt source is required for both start and turn. +- prompt text is always agent input, never shell input. + +## Out Of Scope + +This slice does not include: + +- preserving no-prompt public start as a valid product surface, +- changing substrate -c / --command, +- changing REPL :: grammar, +- adding default-agent or implicit routing, +- fuzzy selector resolution, +- public world-root start, +- member-level public selectors independent of orchestration_session_id, +- daemonizing the control plane, +- promising macOS/Lima parity for world-sensitive follow-up turns, +- preserving the current detached hidden owner-helper bootstrap semantics as the public + start contract, +- redefining reattach as a prompt-taking or conversational command, +- introducing a future true interactive resume surface, +- or redesigning agent status semantics beyond preserving coherence after start, turn, and + reattach. + +If future operational lifecycle workflows are needed, they should be separate explicitly +operational commands, not preserved by overloading public start. + +## Public Selector Contract + +### 1. backend_id is always required for prompt-taking calls + +Every prompt-taking call must name an exact backend_id. + +Required rules: + +- no agent_id routing, +- no default backend, +- no “latest backend in this session” behavior, +- no prefix matching. + +### 2. Public start requires exact backend_id plus exact prompt source + +The root selector model for v1 is: + +substrate agent start --backend (--prompt | --prompt-file | +--prompt-file -) + +Required rules: + +- start without --backend is invalid, +- exactly one prompt source is required, +- malformed or empty prompt input must fail before any runtime launch, +- start must not create an empty live session with no real prompt, +- start must not inject synthetic bootstrap text as agent input, +- root start remains host-only in v1. + +### 3. Follow-up turns require both orchestration_session_id and backend_id + +The follow-up selector model for v1 is: + +substrate agent turn --session --backend ... + +Required rules: + +- --session alone is invalid in follow-up mode, +- --session selects the parent orchestration session, +- --backend selects the exact retained backend inside that session, +- follow-up targeting must use a new or extended authoritative state-store resolver for + exact (orchestration_session_id, backend_id) selection, +- the current session-centric resolve_public_control_target(...) helper is not sufficient + by itself for this selector contract. + +That new or extended resolver must fail closed if: + +- the selected session contains no matching backend slot, +- the selected session contains more than one authoritative candidate for that backend, +- or only stale, inactive, or non-authoritative candidates exist. + +### 4. Canonical public session handle remains orchestration_session_id + +For follow-up targeting and lifecycle recovery, the only accepted public session handle +remains orchestration_session_id. + +The surface must reject these as public session selectors: + +- participant_id +- active_session_handle_id +- session_handle_id +- internal.uaa_session_id + +That posture already exists in the public control/state-store contract and must not +regress. + +### 5. reattach remains session-only and non-prompt-taking + +The lifecycle recovery model for v1 is: + +substrate agent reattach --session [--json] + +Required rules: + +- reattach does not accept a prompt, +- reattach does not create a new conversation turn, +- reattach restores retained ownership to an existing orchestration session when + authoritative recovery metadata is intact, +- reattach must fail closed if the session is already live-owned, lacks required internal + session metadata, or is otherwise not recoverable. + +## Execution Model + +Public start and turn are foreground one-shot caller surfaces. + +That means: + +- the invoking CLI process remains attached until the submitted turn completes or fails, +- the caller receives streamed output while the turn is running, +- the command exits after exactly one submitted turn reaches a terminal result, +- and any retained owner or session state that survives past command exit is an explicit + postcondition, not an implicit side effect. + +This slice must not blur together detached control-plane ownership and foreground caller +experience. + +### Product distinction between start, turn, and reattach + +The product distinction must stay obvious: + +- substrate agent start --backend ... --prompt ... creates a new session and submits the + first real user prompt. +- substrate agent turn --session ... --backend ... --prompt ... submits the next real user + prompt into an exact existing session/backend target. +- substrate agent reattach --session ... restores retained ownership to an existing session + and does not submit a turn. +- none of these commands may inject synthetic bootstrap text as public agent input. + +Root start --backend may legitimately complete in any of the three session_posture states +described below, depending on backend capability and runtime outcome. active is not the +only expected result. + +## Session Lifetime Postconditions + +A successful start or turn must report the resulting session posture explicitly. + +The minimum postcondition enum is: + +- active +- detached_reattachable +- terminal + +These meanings are mandatory, but they are authoritative only at command completion time. +Later calls remain subject to fresh reachability, policy, posture, and state-store checks. + +- active + - at command completion time, a live retained owner is still attached + - substrate agent status must surface the session coherently as live at completion time + - later turn --session ... --backend ..., stop, reattach, and fork remain subject to + fresh reachability, policy, and posture checks +- detached_reattachable + - at command completion time, the session is not live-owned, but authoritative recovery + metadata remains intact + - substrate agent status must surface the detached session coherently at completion + time + - later reattach, fork, or follow-up turn --session ... --backend ... remain subject to + fresh reachability, policy, and posture checks +- terminal + - at command completion time, the session is already completed, failed, invalidated, or + stopped + - substrate agent status must surface the final state coherently at completion time + - later turn --session ... --backend ..., reattach, fork, and stop remain subject to + fresh reachability, policy, and posture checks and may fail closed + +### Startup prompt constraint from current repo truth + +The repo already has an honest prompt-ownership rule in the REPL path: the first explicit +operator prompt must ride the launch-time initial exec prompt instead of being hidden behind +an internal bootstrap composition. + +This slice must respect that distinction: + +- start --backend must not rewrite the first visible operator prompt into a bootstrap-plus- + visible composed payload, +- it must not claim that the resulting session will remain live-owned beyond completion unless + that state is actually proven and surfaced, +- the actual resulting session_posture must be surfaced explicitly in the terminal output + contract. + +## Streaming Return Path + +The public prompt-taking surface must have a real streamed caller-visible return path. + +This requirement is mandatory regardless of internal mechanism. + +Allowed internal patterns include: + +- parent CLI process streaming frames directly from a local runtime or control handle, +- parent CLI process bridging a hidden owner-helper stream over inherited pipes, +- parent CLI process bridging a private owner transport that supports streamed turn output, +- or another equivalent streamed return path. + +What is not allowed: + +- buffering the whole turn and printing only at the end, +- writing output only to persistent trace or status surfaces and asking the caller to poll, +- adding a private request enum with no caller-visible streaming channel, +- or swapping real prompt input for synthetic bootstrap text before streaming begins. + +### Human-mode streaming rules + +Default human mode must: + +- stream user-visible agent output to stdout while the turn is running, +- reserve stderr for CLI-local usage, configuration errors, and pre-stream setup failures, +- and print a terminal summary that includes at least orchestration_session_id, backend_id, + participant_id when available, final turn outcome, and resulting session_posture. + +### JSON streaming rules + +--json should be a line-delimited streaming contract on stdout. + +Recommended minimum record shape: + +{"version":1,"kind":"accepted",...} +{"version":1,"kind":"event",...} +{"version":1,"kind":"warning",...} +{"version":1,"kind":"completed",...} + +Required rules: + +- one JSON object per line, +- stable integer version, +- stable string kind, +- accepted must surface target identity early, +- completed must surface final turn outcome plus resulting session_posture, +- after the first stdout JSON record is emitted, subsequent runtime or turn errors must be + represented in-stream rather than only on stderr. + +For setup or bridge failures that occur after stream start, emit one explicit terminal +failure record on stdout such as: + +{ +"version": 1, +"kind": "failed", +"terminal": true, +"stage": "setup|bridge|runtime", +"error_code": "owner_unreachable|stream_bridge_failed|runtime_start_failed|...", +"message": "..." +} + +The exact event payload may reuse translated runtime and agent event shapes where +practical. + +## Required Semantics And Invariants + +### 1. substrate -c semantics must not change + +This slice must not reinterpret -c, pipe mode, or plain stdin as agent prompting. + +### 2. Exact selectors only + +Every start or turn path must resolve by exact backend_id, and follow-up turn paths must +additionally resolve by exact orchestration_session_id. + +No fallback to: + +- latest session, +- only live session, +- only live backend, +- agent id, +- or trace-derived history. + +### 3. Prompt validation happens before runtime launch + +Both start and turn must validate prompt presence and prompt source shape before launching +or resuming any runtime. + +Required rules: + +- malformed prompt source is a pre-launch failure, +- empty effective prompt is a pre-launch failure, +- and neither command may replace invalid or missing prompt input with synthetic bootstrap + text. + +### 4. Follow-up ownership must remain single-writer + +A follow-up turn must not create competing retained owners for the same exact backend slot +inside the same exact orchestration session. + +Required rule: + +- if a live owner already exists for the exact session/backend target, submit through that + owner or fail closed, +- do not silently create a second retained owner for the same live target. + +### 5. Host and world follow-up paths must stay distinct + +The public non-interactive surface must preserve current runtime truth: + +- host follow-up may use local retained ownership paths, +- Linux world follow-up must continue to rely on the exact retained member-turn seam or an + equivalent owner path that ultimately submits through the same typed world-member + contract, +- non-Linux world-sensitive follow-up must fail closed. + +### 6. Live runtime state remains authoritative + +Follow-up authorization must be derived from authoritative persisted runtime state, not +trace fallback. + +### 7. Policy and posture must remain fail-closed + +The start, turn, and reattach surfaces must preserve current policy and posture rules: + +- root world-only backend rejection remains explicit, +- policy drift remains a hard error, +- non-Linux world-sensitive follow-up remains an explicit posture error, +- detached sessions remain reattachable only when authoritative recovery metadata is + intact, +- and strict selector surfaces stay fail-closed. + +## Recommended Implementation Shape + +1. Extend AgentStartArgs so public start requires exact backend_id plus one exact prompt + source. +2. Add AgentAction::Turn under the existing public agent namespace. +3. Rename the target public meaning of the current resume lifecycle command to reattach. +4. Make backend_id mandatory for all turn calls. +5. Make orchestration_session_id mandatory for follow-up turn calls and reattach. +6. Reuse the existing authoritative session registry and exact public control selectors + where they still fit. +7. Add a new or extended authoritative state-store resolver for exact + (orchestration_session_id, backend_id) follow-up targeting. +8. Reuse the landed host and world turn-submission behavior, but do not hard-code this SOW + to any one startup helper or helper-specific call graph. +9. Add whatever private streamed bridge is necessary so the foreground caller sees live + output while the exact retained target handles the turn. +10. Surface resulting session_posture explicitly so session-lifetime truth at completion + time is visible and testable. + +## Concrete Work Breakdown + +### 1. Change the public start CLI shape + +Update public start in ../crates/shell/src/execution/cli.rs:456 so it requires: + +- --backend +- exactly one of: + - --prompt + - --prompt-file + - --prompt-file - +- optional --json + +### 2. Implement prompt-required root start as a foreground call + +For start --backend ...: + +- validate exact host-scoped backend selection, +- validate exact prompt source before runtime launch, +- reject world-only backends with the same posture classification already used by public + root start, +- create a new orchestration session, +- submit exactly one inaugural real prompt, +- stream output to the caller, +- return the new orchestration_session_id, +- and surface the resulting session_posture explicitly. + +This slice requires the behavior, not one particular helper chain. + +### 3. Remove the root turn --backend form + +Public turn becomes follow-up only. + +There is no supported public root form: + +substrate agent turn --backend ... + +That root prompt-taking role now belongs to start. + +### 4. Implement exact follow-up targeting by session plus backend + +For turn --session ... --backend ...: + +- resolve the exact orchestration session through authoritative state, +- resolve the exact backend target within that session through the new or extended backend- + aware authoritative resolver, +- validate exact prompt source before runtime launch or resume, +- submit one follow-up turn through the current retained target if live, +- otherwise use the existing exact public recovery posture only if the target remains + reattachable, +- otherwise fail closed. + +### 5. Rename and preserve the operational recovery command as reattach + +For reattach --session ...: + +- preserve the current operational recovery behavior of the existing public resume path, +- keep the command non-prompt-taking, +- keep it session-scoped, +- keep it fail-closed when the selected session is already live-owned, lacks required + internal session metadata, or is otherwise not recoverable, +- and update the public terminology and docs so this behavior is no longer described as + conversational “resume.” + +### 6. Add a real streamed bridge + +If the exact start or turn target is not directly owned by the invoking CLI process, add a +streamed return path that bridges runtime output back to the caller. + +This is mandatory for both human mode and JSON mode. + +### 7. Preserve status and read-side coherence + +After any successful or failed start, turn, or reattach, the resulting session must remain +coherent in substrate agent status: + +- new active sessions appear as live, +- detached reattachable sessions appear as detached and coherent rather than disappearing, +- terminal sessions appear with truthful final state, +- strict selector surfaces remain fail-closed. + +## Failure Posture + +The surface must fail closed in these cases: + +1. Unknown backend +2. Ambiguous backend +3. World-only backend passed to root start --backend +4. Missing --backend +5. Missing prompt source +6. Malformed prompt source +7. Empty effective prompt +8. Follow-up mode invoked without --session +9. Unknown session in follow-up mode +10. Non-canonical follow-up session selector +11. Exact backend not present in the selected follow-up session +12. Exact backend slot stale or ambiguous inside the selected follow-up session +13. Missing active parent +14. Missing internal session id for reattachable follow-up recovery +15. Live owner unreachable +16. Unsupported platform or posture for world-sensitive follow-up +17. Policy disallow or policy drift +18. Stream bridge establishment failure + +No failure case may fall back to shell execution, implicit REPL grammar, best-effort +session discovery, or synthetic bootstrap text as agent input. + +## Acceptance Criteria + +This slice is done when all of the following are true: + +1. substrate agent start is a public prompt-taking surface. +2. start requires exact --backend plus exactly one prompt source. +3. start creates a new host-scoped orchestration session and submits the provided prompt as + the inaugural real turn. +4. start streams caller-visible output in the foreground. +5. start returns the new orchestration_session_id. +6. start does not create an empty live session with no real prompt. +7. start does not inject synthetic bootstrap text as public agent input. +8. start remains host-only in v1 and rejects world-only backends explicitly. +9. substrate agent turn is follow-up only. +10. turn requires both --session and --backend plus + exactly one prompt source. +11. turn uses a new or extended authoritative backend-aware resolver rather than session- + only control resolution. +12. turn can target exact retained backends within an existing orchestration session + without relying on session id alone. +13. substrate agent reattach is the public lifecycle recovery surface for restoring + retained ownership to an existing session. +14. reattach does not take a prompt and does not submit a conversation turn. +15. The caller receives real streamed output during start and turn execution. +16. --json is a line-delimited streaming contract with stable version and kind. +17. Setup or bridge failures after JSON stream start emit one terminal kind="failed" record + in-stream. +18. Every successful start or turn reports resulting session_posture as active, + detached_reattachable, or terminal, authoritative at command completion time. +19. substrate agent status surfaces the resulting session coherently after start, turn, and + reattach operations. +20. Strict control and read-side selector contracts do not regress: non-canonical selectors + still fail closed. +21. Linux world-sensitive follow-up can succeed only through the exact landed world follow- + up posture. +22. Non-Linux world-sensitive follow-up still fails closed. +23. Human-mode streaming and JSON-mode streaming both preserve the -c non-agent contract by + construction because prompt-taking remains available only through substrate agent start + and substrate agent turn. + +## Testing Expectations + +At minimum, this slice should add targeted regression coverage for: + +- CLI parse coverage for prompt-required agent start +- CLI parse coverage for follow-up-only agent turn +- CLI parse or alias coverage for agent reattach +- exact required backend_id +- exact required prompt source +- rejection of missing, malformed, or empty prompt input before runtime launch +- rejection of world-only root backends +- host root start success with streamed output +- exact follow-up success into a live host-owned backend +- exact follow-up success into a Linux world-retained backend when authoritative posture is + valid +- fail-closed non-Linux world-sensitive follow-up +- explicit session_posture postconditions at command completion time +- agent status coherence after successful and failed start, turn, and reattach operations +- no regression to strict fail-closed selector behavior on reattach|fork|stop|toolbox +- no regression to -c wrap-mode semantics, with both human-mode and JSON-mode start and + turn paths remaining structurally separate from shell command mode +- no synthetic bootstrap text being used as public agent input on start or turn + +Validation expectations should also include the repo’s required operator surfaces on +touched platforms and postures: + +substrate agent doctor --json +substrate shim doctor --json +substrate world doctor --json +substrate health --json + +Where the slice touches world-sensitive follow-up behavior, capture Linux-first validation +evidence and ensure these surfaces remain coherent with the resulting session and state +truth. + +## Summary + +The missing thin slice is not “teach -c to talk to agents.” The missing thin slice is one +explicit public prompt-taking model where: + +- start means new session plus first real prompt, +- turn means existing session plus next turn, +- reattach means restore retained ownership to an existing session, +- exact backend_id targeting is mandatory for prompt-taking calls, +- exact orchestration_session_id is mandatory for follow-up turns and recovery calls, +- no default backend, fuzzy matching, or implicit latest-session behavior exists, +- malformed or empty prompt input fails before runtime launch, +- synthetic bootstrap text is not accepted as public agent input, +- the current owner-loop recovery behavior is no longer mislabeled as conversational + resume, +- and the repo’s current fail-closed control and status posture remains intact. diff --git a/llm-last-mile/21-macos-lima-shared-owner-and-member-runtime-parity.md b/llm-last-mile/21-macos-lima-shared-owner-and-member-runtime-parity.md new file mode 100644 index 000000000..c4ea02d15 --- /dev/null +++ b/llm-last-mile/21-macos-lima-shared-owner-and-member-runtime-parity.md @@ -0,0 +1,336 @@ +# SOW: macOS/Lima shared-owner and member-runtime parity + +Status: implementation-oriented draft. This SOW covers item `#1` from the recommended v1 runtime slice in [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md:175): bring macOS/Lima onto the same shared-owner/member-runtime path as Linux before widening scope to a different non-Linux backend model. The goal is to extend the already-landed Linux contract across the existing Lima forwarded backend seam, not to invent a different orchestration model. This remains a Linux-first / REPL-first parity slice; it does not broaden non-REPL caller surfaces, but it does include the broader backend-trait and backend-handle work needed to keep macOS/Lima honest with Linux instead of preserving a second-class seam indefinitely. + +## Objective + +Land one honest macOS/Lima path where: + +- the shell remains the owner of the host orchestrator runtime and the authoritative writer of orchestration session and participant state, +- shared-world ownership continues to use the existing `SharedWorldOwnerSpec` / `SharedWorldBindingSnapshot` contract on the REPL persistent-session seam, +- world-scoped member launch, reuse, targeted follow-up turns, and cancel continue to use the existing Linux `member_dispatch`, `/v1/member_turn/stream`, and `/v1/execute/cancel` contract, +- and the macOS host reaches that same contract through the forwarded Lima guest `world-agent` rather than through a lossy backend trait seam or a host-local fallback. + +The required outcome is parity with the Linux shared-owner/member-runtime model. The required outcome is not “a mac-specific orchestration model.” + +## Implementation Shape Choice + +This SOW chooses the broader parity slice, landed in layers but scoped as one coherent slice. + +### Chosen shape: broader parity slice with layered landing + +In scope: + +- macOS REPL persistent-session shared-owner parity over the forwarded `/v1/stream` transport, +- macOS shell parity for world-member bootstrap, follow-up turns, and cancel over the forwarded guest `world-agent` transport, +- shell-side `#[cfg(target_os = "linux")]` parity work needed to make those paths compile and run on macOS, +- expanding `crates/world-api` so backend-level execution and handles can carry the Linux orchestration-sensitive ownership and dispatch contract instead of silently dropping it, +- teaching the generic `WorldBackend::exec(ExecRequest)` path and related backend APIs to preserve Linux orchestration-sensitive request fields end to end, +- making `WorldHandle.shared_binding` parity part of the required backend contract for the Lima backend, +- live Lima validation and doc updates. + +### Landing posture + +This work should still land in layers for risk control, but those layers belong to one SOW and one parity target: + +1. remove macOS host-side shared-owner pre-bootstrap rejection and enable forwarded persistent-session shared-owner proof, +2. widen shell-side member-runtime code paths beyond Linux-only cfg gates, +3. expand `crates/world-api` / backend execution contracts so `world-mac-lima` can preserve the same ownership and dispatch information Linux already relies on, +4. bring backend-level `WorldHandle.shared_binding` and execution-field parity up to the same standard, +5. validate the whole stack together with reproducible Lima orchestration smoke. + +## Current Repo Truth + +### Linux is already the baseline contract + +- The gap matrix already says the intended v1 slice is to keep the existing shell-owned orchestrator runtime, reuse shared-world owner binding and world-generation invalidation, keep Linux world-owned member launch over the existing host↔world seam, and bring macOS/Lima onto that same path before widening scope elsewhere in [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md:171). +- The shell already requests explicit shared-world ownership when opening and replacing REPL world sessions: + - attach/create in [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:5804) + - replace-expected-generation in [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:5575) +- The persistent-session protocol already validates echoed shared-world proof fail-closed, including generation advancement on replacement, in [repl_persistent_session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/repl_persistent_session.rs:308). +- Linux world-member bootstrap already builds a typed `MemberDispatchTransportRequest` carrying `orchestration_session_id`, participant lineage, exact `backend_id`, `world_id`, and `world_generation`, then sends it over `/v1/execute/stream` in [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:3541) and [world_ops.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs:203). +- Linux `world-agent` already validates authoritative shared-world binding for `member_dispatch`, resolves placement context, launches the retained member runtime, exposes `/v1/member_turn/stream`, and delivers `/v1/execute/cancel` in [service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs:1238), [service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs:1471), and [service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs:1488). +- The retained world-member identity model is already explicit in `world-agent`: retained members are keyed by `(orchestration_session_id, world_generation, backend_id)` and follow-up turns validate exact orchestration/backend/world identity in [member_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs:97), [member_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs:869), and [member_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs:1014). + +### The macOS/Lima forwarded transport is already partly there + +- macOS already has a forwarded typed `member_dispatch` request builder in [world_ops.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs:1369). The typed request shape itself is not missing on the host. +- Inference from current code: because Lima forwards into a Linux guest `world-agent`, the guest-side Linux routes are already the intended execution target. The missing work is host-side parity and forwarded-seam honesty, not a second mac-native `world-agent` design. + +### The real macOS shared-owner blocker is host-side pre-bootstrap rejection + +- The REPL builds `SharedWorldOwnerSpec`, but macOS persistent-session bootstrap rejects explicit shared-owner requests before guest bootstrap in [world_persistent_session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_persistent_session.rs:723). +- That rejection goes through the common non-Linux guard in [platform_world/mod.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/platform_world/mod.rs:61). +- `docs/WORLD.md` already states the same truth twice: + - non-Linux shared-owner requests are rejected before bootstrap in [docs/WORLD.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/WORLD.md:86) + - and again in the transport/API section in [docs/WORLD.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/WORLD.md:286) + +### The shell member-runtime path is still Linux-compiled + +- `async_repl` imports `AgentClient`, `ExecuteCancelRequestV1`, `ExecuteStreamFrame`, `MemberTurnSubmitRequestV1`, and the member-dispatch helper surface only on Linux in [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:11) and [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:71). +- Transport request construction for member bootstrap is Linux-only in [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:3540). +- Non-Linux member launch still hard-errors in [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:4297). +- Non-Linux world-targeted follow-up turns still hard-error in [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:4683). +- Bootstrap cancel and remote retained-control shutdown are still Linux-only in [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:3592), [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:3988), and [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:4776). + +### The backend trait seam is lossy today and is part of this slice’s fix target + +- `crates/world-api` `WorldBackend::exec` takes `ExecRequest`, and that type carries only `cmd`, `cwd`, `env`, `pty`, and optional `span_id` in [world-api/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-api/src/lib.rs:139). It does not carry `shared_world` or `member_dispatch`. +- That is why `world-mac-lima` converts ordinary backend exec requests to agent requests with `shared_world: None` and `member_dispatch: None` in [world-mac-lima/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/lib.rs:301). +- `world-mac-lima` also returns `WorldHandle { shared_binding: None }` in [world-mac-lima/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/lib.rs:388). +- If parity with Linux is the real target, this lossy backend seam cannot remain a permanent exception. This draft therefore treats `crates/world-api`, `WorldBackend::exec`, `ExecRequest`, and `WorldHandle.shared_binding` expansion as in-scope work rather than a separate follow-on. + +## Scope + +- Remove the host-side pre-bootstrap rejection for macOS REPL persistent-session shared-owner requests and route them through the forwarded guest `world-agent` transport. +- Preserve the existing `SharedWorldOwnerSpec` request and `ready.shared_world` proof contract on the persistent-session seam. +- Extend shell-side member-runtime code paths so the existing Linux `member_dispatch`, `/v1/member_turn/stream`, and `/v1/execute/cancel` flow compiles and runs on macOS. +- Reuse the existing forwarded typed request builders in `world_ops.rs` rather than inventing a second request model. +- Expand `crates/world-api` so backend-level execution requests and results can preserve Linux orchestration-sensitive ownership and member-dispatch data instead of silently dropping it. +- Update the generic `WorldBackend` execution path and the Lima backend so `shared_world`, `member_dispatch`, and authoritative shared binding can survive the backend seam. +- Make backend-level `WorldHandle.shared_binding` parity a required part of the Lima contract, not an optional nice-to-have. +- Keep member identity, reuse, replacement, and fail-closed validation exactly aligned with the existing guest-Linux `world-agent` contract. +- Add reproducible macOS/Lima orchestration smoke coverage that explicitly exercises attach/create, replacement, targeted turns, and member cancel. +- Update docs, including [docs/WORLD.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/WORLD.md), to stop claiming non-Linux rejection once this slice lands. + +## Non-Goals + +- Replacing the shell-owned orchestrator runtime. +- Redesigning `substrate -c` or creating a new one-shot agent caller surface. +- Broadening or redesigning the public `substrate agent start|resume|fork|stop` contract from [19-public-agent-control-surfaces.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/19-public-agent-control-surfaces.md:3). +- Weakening exact `backend_id`, `orchestration_session_id`, `world_id`, or `world_generation` checks. +- Adding Windows/WSL parity in the same slice. +- Introducing host-local fallback for a selected world-scoped member when forwarded world dispatch is unavailable. + +## Main Code Seams + +### Persistent-session shared-owner proof seam + +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:5804) + - shell-owned attach/create and replace-expected-generation request construction +- [crates/shell/src/execution/repl_persistent_session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/repl_persistent_session.rs:308) + - fail-closed validation for echoed `ready.shared_world` +- [crates/shell/src/execution/routing/dispatch/world_persistent_session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_persistent_session.rs:723) + - actual macOS host-side blocker: rejects explicit shared-owner request before guest bootstrap +- [crates/shell/src/execution/platform_world/mod.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/platform_world/mod.rs:61) + - common non-Linux shared-owner rejection helper +- [docs/WORLD.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/WORLD.md:61) + - documents the current Linux-first shared-owner rule and non-Linux rejection posture + +### Member dispatch / member turn / cancel seam + +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:11) + - Linux-only imports and compile gates that currently block macOS parity +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:3540) + - Linux-only member transport request construction +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:4297) + - non-Linux member launch hard-error +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:4683) + - non-Linux follow-up hard-error +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:3592) + - Linux-only bootstrap cancel support +- [crates/shell/src/execution/routing/dispatch/world_ops.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs:1369) + - existing macOS forwarded typed `member_dispatch` request builder +- [crates/world-agent/src/service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs:1238) + - guest-Linux `member_dispatch` entrypoint and authoritative binding validation +- [crates/world-agent/src/service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs:1471) + - guest-Linux `/v1/member_turn/stream` +- [crates/world-agent/src/service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs:1488) + - guest-Linux `/v1/execute/cancel` +- [crates/world-agent/src/member_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs:869) + - guest-Linux exact identity validation for retained-member follow-up turns + +### Backend trait / backend handle parity seam + +- [crates/world-api/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-api/src/lib.rs:139) + - current trait/type boundary that does not carry orchestration-sensitive request fields and must be widened +- [crates/world-mac-lima/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/lib.rs:301) + - ordinary backend exec conversion that currently drops `shared_world` and `member_dispatch` +- [crates/world-mac-lima/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/lib.rs:388) + - `WorldHandle.shared_binding` remains `None` today and must gain parity + +## Required Semantics + +### 1. Linux remains the contract source of truth + +macOS/Lima parity must match the Linux request/response contract and failure posture already in the repo. This slice must not create a second orchestration truth. + +### 2. Shared-world ownership stays explicit and authoritative on the persistent-session seam + +Where an orchestration session is opening or replacing a shared world through the REPL persistent-session flow, the request must continue to use `SharedWorldOwnerSpec`, the reply must continue to surface `ready.shared_world`, and missing or malformed proof must continue to fail closed. + +### 3. No host-side pre-bootstrap rejection on macOS for this supported path + +The current non-Linux rejection path must be removed for the supported macOS/Lima persistent-session shared-owner flow. After that change, macOS must either complete the forwarded shared-owner handshake or fail closed based on the guest response. It must not be rejected before bootstrap solely because the host OS is macOS. + +### 4. No host-local fallback for world-scoped members + +If the selected runtime is world-scoped and the Lima-forwarded guest contract cannot be established, startup and follow-up turns must fail closed. The shell must not degrade to a host-local retained member runtime. + +### 5. Exact identity and reuse rules must stay the same + +macOS/Lima world-member reuse and follow-up turns must remain keyed by the same identity tuple Linux already enforces: + +- `orchestration_session_id` +- retained participant lineage +- exact `backend_id` +- authoritative `world_id` +- authoritative `world_generation` + +### 6. Targeted follow-up turns must reuse the existing route + +macOS/Lima follow-up turns must go through the existing typed `/v1/member_turn/stream` path and retained `internal.uaa_session_id` flow. This slice must not invent a different resume transport for macOS. + +### 7. Cancel must remain transport-owned in the guest + +Bootstrap cancel and submitted-turn cancel must continue to route through `/v1/execute/cancel` and the retained guest-side control owner. This slice must not move world-member cancel ownership back into the shell. + +### 8. Replacement semantics must remain shell-owned and fail-closed + +On world rollover, the shell must continue to own replacement intent, persisted authoritative binding, stale-member invalidation, and replacement-participant lineage. macOS/Lima must match Linux on: + +- `ReplaceExpectedGeneration` request semantics, +- world-generation advancement, +- invalidation of stale retained members, +- exact-backend relaunch into the replacement world, +- and rollback/fail-closed behavior if the replacement proof or replacement launch cannot be proven. + +### 9. Backend-level parity must stop dropping Linux contract fields + +The generic backend execution seam must no longer silently erase orchestration-sensitive Linux fields on macOS/Lima. If backend-level execution or handle surfaces participate in a shared-owner or member-runtime path, they must carry: + +- explicit shared-world ownership input, +- member-dispatch input, +- and authoritative shared binding output + +with the same fail-closed posture Linux already expects. + +## Recommended Implementation Shape + +### 1. Use the forwarded-client path first, but remove the lossy backend exception in the same slice + +Route shared-owner REPL bootstrap and member-runtime orchestration through the existing forwarded host→guest client paths that already speak the Linux guest `world-agent` contract. But do not leave the generic backend seam as a permanent parity hole; widen `crates/world-api` and the Lima backend so backend-level surfaces can preserve the same ownership and dispatch data instead of hardcoding `None`. + +### 2. Remove the macOS persistent-session shared-owner rejection + +Replace the guard at [world_persistent_session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_persistent_session.rs:736) for the Lima-backed path so macOS can send `StartSession { shared_world: ... }` over the forwarded transport and consume `ready.shared_world` through the existing fail-closed validator in [repl_persistent_session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/repl_persistent_session.rs:308). + +### 3. Extend shell-side member-runtime code past Linux-only cfg gates + +Make the shell’s world-member orchestration path compile and run on macOS by widening the currently Linux-only import, request-building, launch, follow-up, and cancel code in [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:11), [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:3540), [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:4297), and [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:4683). + +### 4. Reuse the existing macOS forwarded typed request builder + +The macOS forwarded `member_dispatch` request builder already exists in [world_ops.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs:1369). Reuse it rather than introducing a second macOS request shape. + +### 5. Reuse guest-side Linux `world-agent` semantics unchanged + +The guest Linux `world-agent` already validates binding and owns retained member runtime behavior in [service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs:1238), [service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs:1471), [service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs:1488), and [member_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs:869). The macOS/Lima slice should consume that contract, not fork it. + +### 6. Expand backend-level world contracts to preserve Linux orchestration semantics + +Update [world-api/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-api/src/lib.rs:139) and downstream backend implementations so the generic backend path can carry: + +- explicit shared-world ownership input, +- member-dispatch input, +- and authoritative shared binding output + +without requiring Linux-only side channels or lossy conversion at the Lima seam. + +## Acceptance Criteria + +### A. Persistent-session shared-owner proof / restart proof + +1. On a macOS host with a provisioned Lima guest, REPL world startup with an orchestration context sends `StartSession { shared_world: Some(AttachOrCreate) }` over the forwarded transport instead of failing at the host-side non-Linux rejection gate. +2. The macOS REPL consumes `ready.shared_world` through the existing validator and persists authoritative `world_id` / `world_generation` into `WorldSession` and shell runtime state. +3. macOS world restart sends `ReplaceExpectedGeneration` over the same persistent-session seam and accepts only a reply whose echoed proof advances `world_generation`. +4. Missing, malformed, mismatched, or non-`active` `ready.shared_world` proof still fails closed on macOS exactly as it does today on Linux. + +### B. Member dispatch / member-turn / cancel parity + +1. macOS/Lima world-member lazy launch no longer errors with “supported on Linux only” and instead sends the same typed `member_dispatch` payload Linux already uses. +2. A retained macOS/Lima world member can accept targeted REPL follow-up turns through `/v1/member_turn/stream` using exact `backend_id`, `world_id`, and `world_generation`. +3. macOS/Lima bootstrap cancel reaches `/v1/execute/cancel` for a forwarded retained member bootstrap span. +4. macOS/Lima submitted-turn cancel reaches `/v1/execute/cancel` for a forwarded retained member submitted-turn span. +5. Stale or mismatched `orchestration_session_id`, `backend_id`, `world_id`, or `world_generation` still fail closed on macOS/Lima exactly as they do on Linux. +6. World rollover on macOS/Lima invalidates stale retained members and relaunches the exact replacement backend into the new authoritative world generation. +7. No macOS/Lima world-sensitive path silently falls back to host-local member runtime ownership. + +### C. Backend trait parity + +1. `crates/world-api` grows a backend-level request/handle contract that can preserve the Linux shared-owner/member-dispatch shape rather than forcing `world-mac-lima` to zero those fields. +2. `WorldBackend::exec` callers that participate in orchestration-sensitive flows can carry the required shared-owner/member-dispatch data without an out-of-band workaround. +3. `world-mac-lima` no longer hardcodes `shared_world: None` and `member_dispatch: None` for the relevant parity paths. +4. `WorldHandle.shared_binding` is populated for the Lima-backed parity path when explicit shared-owner mode is active. +5. Docs and tests do not describe backend-level Linux parity as optional once this slice lands. + +## Testing Expectations + +### Required automated coverage + +- Add or extend shell tests around [world_persistent_session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_persistent_session.rs:723), [repl_persistent_session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/repl_persistent_session.rs:308), and [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:4297) to prove: + - macOS persistent-session shared-owner requests are forwarded instead of pre-rejected, + - replacement proof must advance generation, + - member launch/follow-up/cancel no longer fail at non-Linux cfg stubs. +- Add or extend backend-level tests around [world-api/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-api/src/lib.rs:139) and [world-mac-lima/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/lib.rs:301) to prove: + - backend request/handle types preserve shared-owner/member-dispatch semantics, + - Lima no longer zeroes those fields for parity paths, + - `WorldHandle.shared_binding` is populated when explicit shared-owner mode is active. +- Keep `world-agent` member-runtime tests in [member_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs:1071) and service tests around [service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs:1238) and [service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs:1471) passing without relaxing identity validation. These are host-side regression checks; authoritative macOS/Lima behavior must still be proven by the Lima-backed orchestration smoke below. +- Run at minimum: + - `cargo test -p world-agent member_runtime` + - `cargo test -p shell` + - `cargo test -p world-api` + - `cargo test -p world-mac-lima` + - `cargo test --workspace -- --nocapture` + +### Required guest-Linux behavior validation + +The guest `world-agent` behavior that macOS consumes must be validated in the guest-backed path, not assumed from host-only unit tests. At minimum this means exercising the forwarded macOS path against the real Lima guest `world-agent` service that owns: + +- shared-owner persistent-session replies, +- `member_dispatch`, +- `/v1/member_turn/stream`, +- and `/v1/execute/cancel`. + +### Required reproducible macOS/Lima orchestration smoke + +Existing [scripts/mac/smoke.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/smoke.sh) is not sufficient because [docs/WORLD.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/WORLD.md:196) only claims non-PTY, PTY, and replay coverage today. + +This slice requires one new reproducible harness: [scripts/mac/orchestration-smoke.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/orchestration-smoke.sh). + +It must explicitly cover: + +1. shared-owner attach/create over the persistent-session seam, +2. shared-owner replacement with generation advancement, +3. world-member lazy launch through `member_dispatch`, +4. targeted `:: ` follow-up turn through `/v1/member_turn/stream`, +5. member bootstrap or submitted-turn cancel through `/v1/execute/cancel`, +6. and at least one fail-closed mismatch case for stale `world_generation` or wrong `backend_id`. + +The smoke must be runnable from the current checkout on a macOS host with Lima provisioned, without relying on ad hoc manual steps. + +## Validation Playback + +The implementation is not done until one reproducible macOS/Lima orchestration playback proves the full seam end to end. + +1. Warm the guest from the current checkout with [scripts/mac/lima-warm.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/lima-warm.sh). +2. Start a REPL orchestration session on the macOS host and enter a world-backed path that requests shared-world attach/create. +3. Verify the request is not rejected before bootstrap and that the returned ready state carries authoritative `ready.shared_world`. +4. Trigger lazy world-member launch for one exact backend and verify the retained member is keyed to the authoritative generation. +5. Submit one targeted follow-up turn as `:: ` and verify it routes through the retained world member rather than spawning a new host-local runtime. +6. Deliver cancel once during bootstrap or a submitted turn and verify the forwarded guest-side owner receives it. +7. Force world restart or drift so replacement uses `ReplaceExpectedGeneration`, then verify generation advancement, stale-member invalidation, and exact-backend relaunch. +8. Repeat a targeted follow-up turn after rollover and verify stale-generation submission is rejected while the replacement generation succeeds. +9. Exercise one failure case where shared-world proof or generation continuity cannot be established and verify the macOS host fails closed instead of silently downgrading behavior. + +## Documentation Expectations + +- Update [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md) so the macOS/Lima row no longer claims the parity gap once this slice lands. +- Update [docs/WORLD.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/WORLD.md) to remove the current statement that macOS rejects explicit shared-owner requests before bootstrap in [docs/WORLD.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/WORLD.md:86) and [docs/WORLD.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/WORLD.md:286), and to document the new orchestration validation harness. +- If any caller-facing help, smoke instructions, or control-surface docs mention Linux-only world-member follow-up semantics, update them to reflect the new Lima-backed parity. +- Update any backend/world contract docs that describe `WorldBackend::exec` / `ExecRequest` / `WorldHandle.shared_binding` as insufficient for Linux-parity orchestration semantics, because this slice now includes that rewrite. + +## Done Means + +This slice is done when a macOS host can use the forwarded Lima guest path to participate in the same shared-owner persistent-session proof flow and the same guest-owned retained member-runtime flow that Linux already uses, and when the backend-level world contract no longer forces Lima to zero the ownership and dispatch fields Linux depends on. Completion requires reproducible attach/create, replacement, targeted turns, cancel, fail-closed mismatch validation, and backend-level shared-binding / request-field parity. Until then, the repo should continue to describe world-sensitive agent orchestration as Linux-first. diff --git a/llm-last-mile/22-broaden-caller-surfaces-from-repl-first-to-public-session-member-turns.md b/llm-last-mile/22-broaden-caller-surfaces-from-repl-first-to-public-session-member-turns.md new file mode 100644 index 000000000..8f1668d57 --- /dev/null +++ b/llm-last-mile/22-broaden-caller-surfaces-from-repl-first-to-public-session-member-turns.md @@ -0,0 +1,217 @@ +# SOW: Broaden Caller Surfaces From REPL-First To Public Session/Member Turns + +Status: implementation-oriented draft. This slice is not about inventing a new prompt surface. The thin v1 path is already real in two places: exact REPL targeted turns via `:: ` in [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:5124), and public prompt-taking via `substrate agent start` / `substrate agent turn` in [cli.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/cli.rs:520) and [agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:301). The remaining work is to harden and explicitly validate the public runtime path for session/member turns, especially the Linux-first world-member follow-up path. + +## Objective + +Make `substrate agent start` and `substrate agent turn` the deliberately validated v1 public caller surfaces for prompt-taking orchestration by: + +- validating the end-to-end public runtime path from CLI parsing to prompt streaming to retained target resolution to owner transport or world-member submit, +- locking exact `(orchestration_session_id, backend_id)` routing and fail-closed behavior on the public turn path, +- proving the retained world-member follow-up contract from public selector resolution to typed `MemberTurnSubmitRequestV1` submission into `world-agent`, +- and keeping recovery posture explicit: detached host follow-up may recover through owner reattach, detached world follow-up must fail closed until `substrate agent reattach --session ...` restores an active host owner. + +## Why This Is Needed + +The public surface exists, but the behavior boundary is spread across several layers and the missing work is mostly runtime hardening and coverage, not invention: + +- `run_turn(...)` in [agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:324) loads prompt input, resolves the public target, branches on active versus detached posture, performs detached-host recovery, and explicitly rejects detached-world follow-up. +- `resolve_public_turn_target(...)` in [state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:711) enforces exact `orchestration_session_id` plus exact `backend_id`, rejects noncanonical selectors, and resolves one authoritative retained target slot. +- `run_public_prompt_command(...)` in [control.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/control.rs:1121) revalidates active posture before private prompt transport submission and is therefore part of the public “must be live now” contract. +- Linux world-member follow-up is not just “session plus backend.” The shell resolves that public selector into retained-member identity and submits a typed `MemberTurnSubmitRequestV1` carrying `participant_id`, `orchestrator_participant_id`, `backend_id`, `world_id`, and `world_generation`; `world-agent` validates and resumes the retained member through `submit_member_turn_stream(...)` in [service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs:1485) and `submit_turn(...)` in [member_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs:249). +- Coverage already exists for public `start`, public `turn`, selector rejection, and `-c` preservation in [agent_public_control_surface_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_public_control_surface_v1.rs:674), but the public world-member turn path and several fail-closed cases still need explicit end-to-end tests. + +## Current Repo Truth + +- `substrate agent start --backend ...` is already public and host-only in v1. `run_start(...)` launches the hidden owner helper, then submits the initial prompt through the same public prompt plane used by follow-up turns in [agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:301). +- `substrate agent turn --session --backend ...` is already public and exact in [cli.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/cli.rs:487). +- Public turn selectors already reject `active_session_handle_id`, `participant_id` / legacy `session_handle_id`, and `internal.uaa_session_id` as noncanonical in [state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:1142). +- Linux world-member follow-up submission is already typed and retained. `MemberTurnSubmitRequestV1` in [lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/agent-api-types/src/lib.rs:845) is the host-to-world submit contract, and `validate_submit_turn_request(...)` in [member_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs:869) proves that tuple identity, not backend id alone, is authoritative. +- `substrate -c` remains shell wrap mode and is already protected by test in [agent_public_control_surface_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_public_control_surface_v1.rs:1129). + +## Scope + +In scope: + +- harden and validate the end-to-end public runtime path for `agent start` and `agent turn`, +- treat `reattach` as required recovery for detached world-follow-up posture, but not as the primary feature under test, +- explicitly test the world-member public turn path from public `(session, backend)` resolution to `/v1/member_turn/stream` submission, +- add missing fail-closed coverage for exact public turn error cases, +- update operator-facing docs only where needed to reflect the hardened runtime contract. + +## Non-Goals + +Out of scope: + +- redesigning `substrate -c`, +- changing REPL grammar, +- default-agent or fuzzy routing, +- public world-root start, +- macOS or Windows parity, +- making `fork` or `stop` primary acceptance gates for this slice, +- broader status-surface redesign. + +## Required Runtime Contract + +### Public start + +- `substrate agent start` remains the canonical public root prompt-taking surface. +- It requires exact `--backend `. +- Root start remains host-only in v1; world-only backends must fail closed. + +### Public turn + +- `substrate agent turn` remains the canonical public follow-up surface. +- It requires exact `--session ` and exact `--backend `. +- Exact selector resolution is authoritative, but the full behavior contract spans: + - `run_turn(...)` for posture branching and detached recovery or rejection, + - `resolve_public_turn_target(...)` for exact retained-slot selection, + - `run_public_prompt_command(...)` for active owner transport submission, + - and Linux `MemberTurnSubmitRequestV1` submission plus `world-agent` validation for world-member turns. + +### Hidden engineering contract for world follow-up + +For world-sensitive public follow-up turns, the public selector pair `(session, backend)` must be translated into retained-member identity before submission. That translation must preserve: + +- `participant_id`, +- `orchestrator_participant_id`, +- `backend_id`, +- `world_id`, +- `world_generation`. + +The shell must submit those fields through `MemberTurnSubmitRequestV1`, and `world-agent` must reject identity drift before resuming the retained member session. + +### Fail-closed rules + +Public turn actions must fail closed for: + +- `missing_backend`, +- `unknown_session`, +- all `noncanonical_session_selector` variants, +- `missing_active_parent`, +- `backend_not_in_session`, +- `stale_linkage`, +- `ambiguous_backend_slot`, +- `unsupported_platform_or_posture`, +- `owner_unreachable`. + +No fallback is allowed to a fuzzy session, fuzzy backend, REPL state, or `-c` command mode. + +## Concrete Work Breakdown + +### 1. Runtime hardening and end-to-end validation + +- Audit the public prompt-taking path across [agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs), [state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs), [control.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/control.rs), [lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/agent-api-types/src/lib.rs), [service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs), and [member_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs). +- Tighten any mismatch where public turn routing, detached posture handling, private owner transport submission, and world-member submit identity do not form one coherent contract. + +### 2. Public world-member turn coverage + +- Add explicit end-to-end tests showing that a public `agent turn --session ... --backend ` on Linux resolves the exact retained member slot, shapes `MemberTurnSubmitRequestV1`, and reaches the retained member submit path successfully. +- Prove that the resolved public selector is not merely “backend selected,” but is translated into the retained member tuple required by `world-agent`. + +### 3. Missing fail-closed coverage for public turn + +Add explicit tests for: + +- `missing_backend`, +- `unknown_session`, +- `missing_active_parent`, +- `backend_not_in_session`, +- `stale_linkage`, +- `ambiguous_backend_slot`, +- noncanonical selector via `active_session_handle_id`, +- noncanonical selector via `participant_id` / legacy `session_handle_id`, +- noncanonical selector via `internal.uaa_session_id`, +- detached-world follow-up rejection with reattach guidance, +- `owner_unreachable`. + +### 4. Retained-member identity-drift rejection + +- Add at least one explicit negative test at the `MemberTurnSubmitRequestV1` / `submit_turn(...)` boundary proving that world-member follow-up is rejected when the retained identity tuple drifts. +- The minimum acceptable case is one mismatch in retained-member identity, such as `backend_id`, `world_generation`, `world_id`, `participant_id`, or `orchestrator_participant_id`, as validated by `validate_submit_turn_request(...)` in [member_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs:869). +- This test should make the hidden contract explicit: successful public world follow-up depends on exact retained-member tuple continuity, not just exact public selector resolution. + +### 5. Keep recovery posture exact + +- Keep detached-host recovery in `run_turn(...)` as-is in principle, but validate it with tests as part of the public contract. +- Keep detached-world follow-up fail-closed until `reattach` restores an active host owner; do not broaden world follow-up recovery implicitly. + +### 6. Focused docs and planning cleanup + +If runtime hardening changes or clarifies operator-visible behavior, update only these files: + +- [docs/USAGE.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/USAGE.md) +- [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md) +- the new `llm-last-mile` SOW file created from this draft + +This is secondary to runtime hardening and coverage. + +## Acceptance Criteria + +This slice is done when all of the following are true: + +1. `substrate agent start` still succeeds only for exact host-scoped backends and still uses the public prompt plane end to end. +2. `substrate agent turn` is covered end to end for both: + - host follow-up through the private owner prompt transport, + - Linux world-member follow-up through `MemberTurnSubmitRequestV1` to `/v1/member_turn/stream` to `submit_turn(...)`. +3. The Linux world-member public turn path proves that `(orchestration_session_id, backend_id)` is translated into retained-member identity including `participant_id`, `orchestrator_participant_id`, `world_id`, and `world_generation`. +4. Public turn rejects `missing_backend` with an explicit test. +5. Public turn rejects `unknown_session` with an explicit test. +6. Public turn rejects `missing_active_parent` with an explicit test. +7. Public turn rejects `backend_not_in_session` with an explicit test. +8. Public turn rejects `stale_linkage` with an explicit test. +9. Public turn rejects `ambiguous_backend_slot` with an explicit test. +10. Public turn rejects all noncanonical selector variants with explicit tests: + - `active_session_handle_id`, + - `participant_id` / legacy `session_handle_id`, + - `internal.uaa_session_id`. +11. Detached host follow-up recovery and detached world follow-up rejection are both covered as part of the public contract. +12. Public turn reaches `owner_unreachable` with an explicit test. +13. The world-member submit boundary has at least one explicit retained-member identity-drift rejection test. +14. `substrate -c` still remains shell wrap mode. +15. Any operator-facing doc changes are limited to `docs/USAGE.md`, `AGENT_ORCHESTRATION_GAP_MATRIX.md`, and the new `llm-last-mile` file for this slice. + +## Testing Expectations + +Primary test targets: + +- [crates/shell/tests/agent_public_control_surface_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_public_control_surface_v1.rs) +- [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) +- [crates/world-agent/src/member_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs) +- [crates/agent-api-types/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/agent-api-types/src/lib.rs) + +Required assertions: + +- public `start` streams acceptance and terminal completion and leaves authoritative active state behind it, +- public host `turn` succeeds end to end on the exact session/backend pair, +- public Linux world-member `turn` succeeds end to end on the exact session/backend pair, +- world-member public turn carries the retained identity tuple required by `MemberTurnSubmitRequestV1`, +- `missing_backend` fails closed, +- `unknown_session` fails closed, +- `missing_active_parent` fails closed, +- `backend_not_in_session` fails closed, +- `stale_linkage` fails closed, +- `ambiguous_backend_slot` fails closed, +- each noncanonical selector variant fails closed with the expected classifier, +- detached host recovery is exercised and verified, +- detached world follow-up fails closed and instructs the operator to reattach, +- `owner_unreachable` fails closed, +- one retained-member identity-drift case is rejected at the `MemberTurnSubmitRequestV1` / `submit_turn(...)` boundary, +- root start for world-only backends still fails closed, +- `substrate -c` is not reinterpreted as an agent prompt surface. + +Recommended commands: + +```bash +cargo test -p shell --test agent_public_control_surface_v1 -- --nocapture +cargo test -p shell agent_runtime::state_store -- --nocapture +cargo test -p world-agent member_runtime -- --nocapture +cargo test -p agent-api-types member_turn_submit -- --nocapture +``` + +## Explicit Non-Goals For Review + +- Do not widen scope into `fork` or `stop` product work. +- Do not redesign prompt grammar or add default routing. +- Do not repurpose `substrate -c`. +- Do not promise non-Linux world-follow-up parity before the backend path exists. diff --git a/llm-last-mile/23-host-orchestrator-durable-session-and-parked-resumable-ownership.md b/llm-last-mile/23-host-orchestrator-durable-session-and-parked-resumable-ownership.md new file mode 100644 index 000000000..86a9f83f3 --- /dev/null +++ b/llm-last-mile/23-host-orchestrator-durable-session-and-parked-resumable-ownership.md @@ -0,0 +1,552 @@ +# SOW: Host Orchestrator Durable Session And Parked-Resumable Ownership + +Status: implementation-oriented follow-on draft. This SOW expands and operationalizes [ADR-0047 — Host Orchestrator Durable Session and Parked-Resumable Ownership](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/project_management/adrs/draft/ADR-0047-host-orchestrator-durable-session-and-parked-resumable-ownership.md) after the public caller surfaces landed in [20-public-non-interactive-agent-caller-surface.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/20-public-non-interactive-agent-caller-surface.md) and the public turn path was hardened in [22-broaden-caller-surfaces-from-repl-first-to-public-session-member-turns.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/22-broaden-caller-surfaces-from-repl-first-to-public-session-member-turns.md). + +The public verbs are already the right verbs: `substrate agent start`, `substrate agent turn`, `substrate agent reattach`, and `substrate agent stop`. The missing work is not new routing syntax. The missing work is durable lifecycle truth: Substrate must own the orchestration session as the durable authority, while a Codex or equivalent backend process is only an attachable execution client that may exit cleanly without invalidating the session. + +This SOW deliberately covers more than posture naming. It pulls in the ADR's full runtime-state contract, durable inbox requirements, resolved `reattach` and compaction decisions, greenfield state-adoption rules, and the operator/test obligations required to make the new lifecycle model real. + +## Objective + +Make host orchestration durable across clean prompt-driven backend exits by: + +- preserving the public `start|turn|reattach|stop` contract exactly as already landed, +- making the Substrate-owned orchestration session, not the attached client process, the durable authority, +- introducing explicit host postures for `active_attached`, `parked_resumable`, `awaiting_attention`, and `terminal`, +- persisting world-originated approvals, completion notices, follow-up messages, and runtime alerts in a session-local durable inbox, +- making `turn` and `reattach` resume valid parked host sessions without fuzzy recovery, +- tightening the public prompt bridge so every request that emits `Accepted` also emits an explicit `Completed` or `Failed`, +- and standardizing the internal runtime-state schema and filesystem layout needed to make those guarantees authoritative from first write. + +## Why This Is Needed + +The public surface already looks durable, but the runtime still behaves too much like attached process retention is the orchestration authority itself. + +- Public prompt-taking is already wired through `run_start(...)`, `run_turn(...)`, `run_reattach(...)`, and `run_stop(...)` in [agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:301). +- Exact public follow-up targeting is already enforced by `resolve_public_turn_target(...)` in [state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:711), which resolves one authoritative retained slot from exact `(orchestration_session_id, backend_id)`. +- The public prompt bridge already exposes `Accepted`, `Completed`, and `Failed` envelopes and submits prompt work through `run_public_prompt_command(...)` in [control.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/control.rs:1063). +- `OrchestrationSessionRecord` still only models `Allocating | Active | Invalidated | Stopping | Stopped | Failed` in [orchestration_session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/orchestration_session.rs:7), which is insufficient to express “valid but detached and resumable.” +- Participant liveness still leans on attachment diagnostics such as `control_owner_retained`, `event_stream_active`, `completion_observer_retained`, and `terminal_observed_at` in [session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs:81). +- The existing REPL test `start_host_orchestrator_runtime_invalidates_when_attached_control_exits()` in [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:7335) documents the current lifecycle bias directly: clean attached-client exit still tends to collapse into invalidation rather than a valid parked session. +- World follow-up already depends on resumption plumbing such as `build_session_resume_extension(...)` in [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:35), but there is still no first-class Substrate-owned durable inbox or parked-host posture separating “no client attached right now” from “session is dead.” + +The result is a contract mismatch: + +- `start` and `turn` already look like durable orchestration commands, +- but clean attached-client exit can still be interpreted as lost ownership, +- and world-originated follow-up pressure still lacks a durable retained surface when no host client is currently attached. + +## Relationship To Existing Slices + +This SOW consumes and narrows follow-on work after existing `llm-last-mile` slices. + +- [19-public-agent-control-surfaces.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/19-public-agent-control-surfaces.md) made the public lifecycle namespace explicit. +- [20-public-non-interactive-agent-caller-surface.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/20-public-non-interactive-agent-caller-surface.md) made `start`, `turn`, and `reattach` the public prompt-taking and lifecycle surface. +- [22-broaden-caller-surfaces-from-repl-first-to-public-session-member-turns.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/22-broaden-caller-surfaces-from-repl-first-to-public-session-member-turns.md) hardened exact public turn targeting and the world-member follow-up path. + +This slice must not reopen: + +- exact `(orchestration_session_id, backend_id)` public turn targeting, +- host-only root `start` in v1, +- detached-world fail-closed posture, +- or the Linux-first `MemberTurnSubmitRequestV1` world follow-up contract. + +It only clarifies what those commands bind to, how host continuity survives clean detach, and what persisted runtime state must exist to make that durable model true. + +## Scope + +In scope: + +- define a durable host orchestration session model that survives clean attached-client exit, +- persist explicit host posture independently from raw lifecycle state, +- add a canonical session-local durable inbox under the session root, +- standardize additive session and participant metadata required for parked/resumable ownership, +- make `turn` and `reattach` resume valid parked host sessions without fuzzy routing, +- preserve detached-world fail-closed behavior while broadening host durability, +- make resolved inbox retention and later compaction explicit rather than implicit, +- require greenfield runtime-state writers to adopt the new session-root contract from first write, +- and update tests and operator-facing docs so the new lifecycle truth is externally provable. + +## Out Of Scope + +This slice does not include: + +- changing the public selector contract for `turn`, +- introducing fuzzy routing, default routing, or new public selector types, +- redesigning Linux world-member follow-up away from `MemberTurnSubmitRequestV1`, +- making detached-world follow-up self-sustaining without a valid host orchestration session, +- redesigning REPL grammar, +- designing the full Substrate-native prompt/context builder, +- defining a remote multi-tenant orchestration control plane, +- generalizing the new attachment schema to hypothetical non-host attached clients, +- or turning `reattach` into a prompt-taking shortcut. + +## Required Runtime Contract + +### 1. Public verbs stay stable + +The public contract remains: + +- `substrate agent start --backend --prompt ...` +- `substrate agent turn --session --backend --prompt ...` +- `substrate agent reattach --session ` +- `substrate agent stop --session ` + +This slice must not rename, broaden, or overload those verbs. + +### 2. The durable authority is the orchestration session, not the attached client + +After a session is created successfully, clean exit of the prompt-driven host client must not automatically mean orchestration loss. The durable authority is the Substrate-owned session record, participant state, and durable inbox/task state. + +An attached Codex or equivalent backend process is: + +- a valid execution client while attached, +- resumable if recovery metadata remains intact, +- and not the durable authority for whether the orchestration session itself remains valid. + +### 3. Host posture must be explicit + +The runtime must expose semantics equivalent to: + +- `active_attached` + - a host execution client is attached and can receive prompt traffic immediately +- `parked_resumable` + - the orchestration session remains valid, but no host execution client is currently attached +- `awaiting_attention` + - the orchestration session remains valid and has pending approvals, completion messages, follow-up messages, or runtime alerts that require host-side review or resumption +- `terminal` + - the orchestration session is no longer routable + +`awaiting_attention` may be persisted directly or derived from `pending_inbox_count > 0`, but the operator-visible behavior must exist and the persisted authority must be unambiguous. + +### 4. Detached-host and detached-world rules stay distinct + +- A parked host session is valid and resumable. +- Detached-world follow-up remains fail-closed. +- World follow-up must continue to route through a valid host orchestration session owned by Substrate. + +This slice must not use parked-host durability as a reason to loosen detached-world fail-closed posture. + +### 5. World-originated delivery must be durable without an attached client + +Approvals, completion notices, follow-up messages, and runtime alerts from world-side work must land in a Substrate-owned durable inbox, queue, or equivalent session-local task ledger. A live attached client may consume those events immediately, but the absence of an attached client must not: + +- silently drop them, +- silently consume them, +- or invalidate the session just because no client is currently attached. + +### 6. Accepted implies explicit terminal delivery + +Once a public prompt request emits `Accepted`, the prompt bridge must always terminate with an explicit `Completed` or `Failed` envelope. Stream disappearance, EOF, or helper exit without a terminal envelope is a runtime bug and must not be treated as the operator-facing steady state for parking. + +### 7. `reattach` semantics are owner recovery only + +`substrate agent reattach --session ` is the explicit attached-owner recovery verb. + +Required rules: + +- `reattach` restores a live attached owner loop when recovery metadata is intact, +- `reattach` does not submit a prompt, +- `reattach` does not implicitly consume inbox work, +- `reattach` does not act as a one-shot follow-up-turn shortcut, +- one-shot prompt-taking resume remains on `substrate agent turn --session ... --backend ... --prompt ...`. + +## Internal Runtime-State Contract + +This section is implementation-binding for live-state authority. It is internal runtime state, not a new operator config or policy surface. + +### Canonical filesystem layout + +Canonical live-state authority remains under: + +- `~/.substrate/run/agent-hub/sessions//session.json` +- `~/.substrate/run/agent-hub/sessions//participants/.json` +- `~/.substrate/run/agent-hub/sessions//leases/.lease` + +This slice adds a canonical durable inbox path under the same session root: + +- `~/.substrate/run/agent-hub/sessions//inbox/.json` + +These session-root files are the sole live-state authority from first write. + +### Orchestration session record + +Existing `session.json` identity and lifecycle fields remain authoritative, including: + +- `orchestration_session_id` +- `shell_trace_session_id` +- `workspace_root` +- `shell_owner_pid` +- `state` +- `opened_at` +- `last_active_at` +- `orchestrator_agent_id` +- `orchestrator_backend_id` +- `orchestrator_protocol` +- `active_session_handle_id` +- `latest_run_id` +- `world_id` +- `world_generation` +- `invalidation_reason` +- `closed_at` + +Required additive fields: + +- `posture: active_attached|parked_resumable|awaiting_attention|terminal` +- `posture_changed_at: ` +- `attached_participant_id: |null` +- `pending_inbox_count: ` +- `last_parked_at: |null` +- `last_attention_at: |null` +- `parked_reason: |null` + +Required semantics: + +- `state` remains the lifecycle state machine for allocation, active execution, stopping, failure, invalidation, and terminal completion. +- `posture` is the attachability and attention summary. It must not be inferred solely from `state`. +- `active_session_handle_id` retains its existing compatibility meaning as the authoritative orchestrator participant for the session. It is not proof that a host client is currently attached. +- `attached_participant_id` is the authoritative pointer to the currently attached host execution client. It must be `null` whenever `posture` is `parked_resumable`, `awaiting_attention`, or `terminal`. +- `pending_inbox_count` counts unresolved inbox items for the session. +- `posture=awaiting_attention` is required when the session remains non-terminal, `attached_participant_id=null`, and `pending_inbox_count>0`. +- `posture=parked_resumable` is required when the session remains non-terminal, `attached_participant_id=null`, `pending_inbox_count=0`, and at least one authoritative host participant remains `resume_eligible=true`. +- `posture=terminal` must align with a non-routable session state such as `Invalidated`, `Stopped`, or `Failed`. +- `posture` is explicit persisted truth and must not be reconstructed heuristically from attachment diagnostics or legacy attachment flags. + +### Participant record + +Existing participant fields remain authoritative, including: + +- handle and lineage fields such as `participant_id`, `orchestration_session_id`, `backend_id`, `role`, `protocol`, `state`, `world_id`, and `world_generation` +- internal fields such as `uaa_session_id`, `latest_run_id`, `cancel_supported`, `ownership_mode`, `ownership_valid`, `last_heartbeat_at`, `last_event_at`, `terminal_observed_at`, `termination_reason`, `last_error_bucket`, and `last_error_message` + +Required additive internal fields for host-orchestrator participants: + +- `attached_client_present: ` +- `last_attached_at: |null` +- `last_detached_at: |null` +- `detach_reason: |null` +- `resume_eligible: ` + +Required semantics: + +- `uaa_session_id` remaining populated after clean client exit is valid and expected for a parked-resumable host session. +- `resume_eligible=true` with `attached_client_present=false` and a non-terminal participant `state` is a valid parked host posture, not a failure condition. +- `control_owner_retained`, `event_stream_active`, and `completion_observer_retained` remain attachment diagnostics. They must no longer be treated as the sole proof that the orchestration session itself is valid. +- These new fields are required for host-orchestrator participants. Member-runtime participants may leave them unset or at safe defaults when the semantics do not apply. +- `uaa_session_id` is an identifier and correlation field, not proof of attachment, liveness, or resumability on its own. + +### Durable inbox item + +Each unresolved or retained orchestration event must be persisted as one file under: + +- `sessions//inbox/.json` + +Minimum durable inbox schema: + +- `schema_version: ` +- `item_id: ` +- `orchestration_session_id: ` +- `kind: approval_required|completion_notice|follow_up_message|runtime_alert` +- `state: pending|acknowledged|dismissed` +- `created_at: ` +- `resolved_at: |null` +- `correlation: ` +- `payload_schema: ` +- `payload: ` + +Minimum correlation envelope: + +- `source_event_type: |null` +- `source_span_id: |null` +- `source_cmd_id: |null` +- `source_trace_session_id: |null` +- `origin_participant_id: |null` +- `origin_backend_id: |null` +- `origin_run_id: |null` +- `caused_by_turn_id: |null` +- `workflow_id: |null` +- `workflow_run_id: |null` +- `workflow_node_id: |null` +- `request_id: |null` +- `idempotency_key: |null` + +Required semantics: + +- `state=pending` items contribute to `pending_inbox_count`. +- Lack of an attached host client must never delete, skip, or silently consume a pending item. +- A live attached host client may observe and acknowledge items in real time, but the persisted inbox item remains the durable source of truth until it is resolved. +- Resolved items may be compacted later, but only after they have transitioned out of `pending`. +- `item_id` is the session-local durable inbox record identifier. It is not a router `request_id`, workflow run identifier, or trace span identifier. +- The correlation envelope is additive and nullable by design; fields may be unset when the source family does not define them. +- Correlation fields are join keys for trace, router, workflow, and orchestration analysis. They do not themselves grant delivery, liveness, or resumability semantics. +- The canonical naming and required/optional classification of cross-cutting correlation fields remain owned by ADR-0028 and the Phase 8 registry. This slice only requires that inbox items carry an explicit compatible correlation envelope and must not rely on heuristic joins. + +### Resolved inbox retention and compaction contract + +Resolved inbox items remain retained for audit and later compaction. + +Required rules: + +- resolving an inbox item removes it from authoritative pending/live counts immediately, +- resolving an inbox item must not immediately delete the persisted inbox artifact, +- resolved items should be retained until the session is terminal, the item is no longer pending, the item has aged past a bounded retention floor, and no unresolved item still depends on it for causation or debug correlation, +- compaction is a maintenance step, not the acknowledgement primitive, +- this slice fixes compaction eligibility rules but does not choose a numeric retention duration. + +### Future inbox-kind growth stays additive + +The durable inbox outer envelope above is canonical. + +Required rules: + +- future inbox item kinds may expand the `kind` enum only additively, +- future kinds must not introduce new ad hoc top-level envelope fields as their primary shape, +- kind-specific structure must live under `payload_schema` plus `payload`, +- future cross-cutting join/correlation additions must remain additive and defer to ADR-0028 and the Phase 8 registry instead of creating inbox-only dialects. + +### Lease-file authority + +Existing lease payloads may remain minimal and additive. If fast-path attachment fields are mirrored into lease files, `session.json` and `participants/.json` remain authoritative. + +### Greenfield runtime-state adoption rules + +This runtime-state schema is greenfield. + +Required rules: + +- new runtime-state writers must write the session-root layout and required posture/attachment fields from first write, +- readers must treat the persisted fields in this SOW as authoritative rather than reconstructing posture from heuristics, +- no config schema change and no policy schema change is required by this runtime-state contract, +- older flat session, participant, or lease layouts are not part of this contract. + +## Architecture Shape + +Components: + +- `crates/shell` runtime/session/state layers own durable orchestration-session identity, posture state, and routing gates. +- A Substrate-owned durable inbox or task ledger owns world-to-host message durability when no client is attached. +- An attachable execution client, such as a Codex session, may attach, run, exit, and later resume against the same orchestration session. +- `world-agent` and the world-member submit path continue to own world-member execution and typed request submission for Linux world follow-up. + +End-to-end shape: + +- public `start`, `turn`, `reattach`, and `stop` operate on durable orchestration-session state, +- world-originated events are translated into durable inbox items whether or not a host client is attached, +- posture is derived from explicit persisted session and participant truth, not transient attachment heuristics, +- `turn` and `reattach` resume a valid parked host session through sanctioned host paths only, +- and prompt-bridge `Accepted` responses are emitted only under a runtime invariant that guarantees explicit terminal delivery. + +## Concrete Work Breakdown + +### 1. Add explicit session-posture state to the live-state authority + +Update the orchestration session persistence layer in [orchestration_session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/orchestration_session.rs) and the state-store/session writers in [state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) and [session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs) so the persisted session record gains: + +- explicit `posture`, +- `posture_changed_at`, +- `attached_participant_id`, +- `pending_inbox_count`, +- `last_parked_at`, +- `last_attention_at`, +- `parked_reason`. + +This work must make the persisted session record authoritative for operator-visible posture and must not keep posture as an inferred view over legacy attachment booleans. + +### 2. Add host attachment metadata to participant records + +Update host-orchestrator participant persistence so participant records gain: + +- `attached_client_present`, +- `last_attached_at`, +- `last_detached_at`, +- `detach_reason`, +- `resume_eligible`. + +This is the bridge between raw process-level attachment changes and durable parked/resume semantics. It must make clean detach a first-class valid condition rather than a failure-shaped absence. + +### 3. Stop treating clean attached-client exit as automatic invalidation + +Adjust the runtime and REPL lifecycle handling in [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) so that: + +- clean prompt-driven host client exit transitions a valid host session into `parked_resumable`, +- a detached parked session with pending inbox work transitions into `awaiting_attention`, +- explicit `stop`, fatal invalidation, or terminal failure still transition to `terminal`, +- retained world/member linkage is preserved when the session remains valid, +- and legacy attachment diagnostics remain diagnostics rather than sole validity authority. + +The existing invalidation-on-clean-exit behavior must be replaced, not just hidden behind docs. + +### 4. Add the canonical durable inbox under the session root + +Introduce the session-local inbox path and persistence contract under: + +- `~/.substrate/run/agent-hub/sessions//inbox/.json` + +Minimum implementation responsibilities: + +- define the canonical inbox item envelope, +- write pending items durably for world-originated approvals, completions, follow-up messages, and runtime alerts, +- maintain `pending_inbox_count` as authoritative live state, +- preserve correlation metadata needed for trace/router/workflow joins, +- and ensure the absence of an attached client never causes event loss. + +### 5. Preserve resolved-item audit retention and explicit compaction rules + +The inbox implementation must distinguish: + +- acknowledgement or dismissal of an item, +- removal of the item from authoritative pending/live counts, +- and later physical compaction of historical artifacts. + +Resolved items must remain inspectable until compaction eligibility rules are met. Immediate deletion is not acceptable because it weakens detached-host review and postmortem analysis. + +### 6. Make `turn` and `reattach` resume valid parked host sessions + +Update the public control path across [agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs), [state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs), [control.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/control.rs), and [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) so that: + +- `turn` can resume prompt-taking against a valid parked host session, +- `reattach` can explicitly restore attached host ownership without submitting a prompt, +- exact `(orchestration_session_id, backend_id)` routing remains authoritative, +- and no fuzzy session or backend recovery is introduced. + +### 7. Harden the public prompt bridge terminal contract + +`run_public_prompt_command(...)` already models `Accepted`, `Completed`, and `Failed`. This slice must convert that into a pinned invariant: + +- every path after `Accepted` ends in `Completed` or `Failed`, +- parked-host transitions do not masquerade as owner disappearance, +- helper/client exit is surfaced through explicit terminal delivery rather than silent stream loss, +- and EOF after `Accepted` remains a failure path, not a valid parked steady state. + +### 8. Preserve detached-world fail-closed posture while broadening host durability + +Nothing in this slice may weaken: + +- root host-only `start` in v1, +- exact public `turn` targeting, +- detached-world follow-up rejection, +- or the existing Linux-first world-member contract. + +The only broadened continuity is parked host continuity. + +### 9. Adopt the new runtime-state schema as greenfield live-state truth + +Writers must move directly to the session-root layout and additive fields from first write. Readers must trust the persisted fields rather than reconstructing posture or resumability heuristically. This work should not introduce configuration toggles or policy schema changes to stage correctness. + +### 10. Update focused docs and operator surfaces + +Update only the docs needed to tell the truth about: + +- durable host orchestration ownership, +- parked versus attached versus attention-needed posture, +- the durable inbox/task ledger, +- and the accepted-to-terminal delivery guarantee. + +This SOW is itself one of those truth surfaces and should remain aligned with the ADR. + +## Sequencing / Dependencies + +Prerequisite context that remains binding: + +- the public caller-surface hardening from `PLAN-22`, +- the host/world posture and identity-tuple rules from ADR-0042, +- the workflow/router correlation posture from ADR-0021, ADR-0028, ADR-0029, and the Phase 8 registry. + +This slice should assume those decisions are already pinned and should only add the durable host-session semantics that sit on top of them. + +## Security / Safety Posture + +Fail-closed rules that must remain true: + +- world-only root `start` remains rejected, +- detached-world public follow-up remains rejected until a sanctioned host path restores routable ownership, +- public `turn` remains exact-selector only, +- world-originated messages must not bypass Substrate-owned routing and durability surfaces. + +Protected invariants: + +- clean prompt-driven host client exit after session establishment must not silently degrade into session invalidation, +- accepted prompt requests must not end in silent stream loss, +- durable orchestration state must remain authoritative over any one attached client process, +- and persisted inbox correlation data must not be mistaken for new routing authority. + +## Acceptance Criteria + +This slice is done when all of the following are true: + +1. `substrate agent start`, `turn`, `reattach`, and `stop` keep the same public verb and selector contract. +2. A successfully established host orchestration session can survive clean attached-client exit without being invalidated automatically. +3. The runtime persists an explicit valid parked-resumable host posture instead of forcing detached host into terminal or invalidation semantics. +4. The runtime persists an explicit `awaiting_attention` posture directly or derives it from authoritative persisted pending-inbox state. +5. `session.json` persists the additive posture fields required by this SOW. +6. Host-orchestrator participant records persist the additive attachment and resumability fields required by this SOW. +7. The canonical durable inbox path exists under the session root and stores one persisted artifact per unresolved or retained orchestration event. +8. World-originated approvals, completion notices, follow-up messages, and runtime alerts are retained durably when no host client is attached. +9. A live attached host client may observe or acknowledge inbox items in real time, but unresolved items remain durable source-of-truth artifacts until resolved. +10. Resolving an inbox item removes it from pending/live counts immediately but does not immediately delete the persisted inbox artifact. +11. Future inbox-kind growth stays additive under the canonical envelope rather than creating ad hoc top-level variants. +12. `turn` can resume prompt-taking against a valid parked host session using exact `(orchestration_session_id, backend_id)` targeting. +13. `reattach` can restore attached host ownership for a valid parked host session without submitting a prompt. +14. Detached-world follow-up remains fail-closed until a sanctioned host path re-establishes routable ownership. +15. Host-only root `start` remains fail-closed for world-only backends. +16. Once a public prompt request emits `Accepted`, it always ends with explicit `Completed` or `Failed`. +17. Prompt-stream EOF or silent helper exit after `Accepted` is treated as a bug or failure path, not an operator-facing steady state. +18. Greenfield runtime-state writers write the new session-root layout and additive posture/attachment fields from first write. +19. Readers treat persisted posture and attachment fields as authoritative rather than reconstructing them heuristically from legacy diagnostics. +20. Existing public world-member follow-up coverage from the `PLAN-22` slice remains green as a non-regression constraint. + +## Testing Expectations + +Primary coverage areas: + +- orchestration session posture and state transitions in [orchestration_session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/orchestration_session.rs) +- participant attachment and resumability state in [session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs) +- state-store persistence and exact public turn resolution in [state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) +- prompt-bridge terminal-delivery invariants in [control.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/control.rs) +- REPL/runtime lifecycle handling in [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) +- public caller-surface behavior in [agent_public_control_surface_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_public_control_surface_v1.rs) +- world-first routing non-regression in [repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs) + +Required assertions: + +- `active_attached -> parked_resumable` is covered explicitly, +- `parked_resumable -> active_attached` via `reattach` is covered explicitly, +- `parked_resumable -> awaiting_attention` is covered explicitly or proven through authoritative pending-inbox derivation, +- legal transitions to `terminal` are covered explicitly, +- host `start` creates a valid orchestration session and clean host client exit does not invalidate it, +- host `turn` succeeds against a valid parked session, +- `reattach` succeeds against a valid parked session and does not submit a prompt, +- world-originated retained work survives detached host periods, +- pending/live counts update correctly when inbox items are acknowledged or dismissed, +- resolved items remain retained until compaction eligibility is met, +- detached-world follow-up still fails closed, +- existing exact public turn routing remains authoritative, +- and every accepted prompt request ends in `Completed` or `Failed`. + +Recommended commands: + +```bash +cargo test -p shell --test agent_public_control_surface_v1 -- --nocapture +cargo test -p shell agent_runtime::control -- --nocapture +cargo test -p shell agent_runtime::state_store -- --nocapture +cargo test -p shell async_repl -- --nocapture +cargo test -p shell --test repl_world_first_routing_v1 -- --nocapture +``` + +## Manual Validation + +Re-run the public host flow outside the integration harness: + +```bash +substrate agent start --backend --prompt "hello" --json +substrate agent turn --session --backend --prompt "next" --json +substrate agent reattach --session --json +``` + +Confirm all of the following manually: + +- the host session parks cleanly when the prompt-driven client exits, +- the parked session remains resumable rather than invalidated, +- world-originated messages create durable inbox items even while no client is attached, +- `reattach` restores attached ownership without consuming inbox work implicitly, +- and no post-`Accepted` stream ends without `Completed` or `Failed`. diff --git a/llm-last-mile/24-fix-host-bootstrap-readiness-and-clean-detach-parking.md b/llm-last-mile/24-fix-host-bootstrap-readiness-and-clean-detach-parking.md new file mode 100644 index 000000000..416cc7a1f --- /dev/null +++ b/llm-last-mile/24-fix-host-bootstrap-readiness-and-clean-detach-parking.md @@ -0,0 +1,313 @@ +# SOW: Fix Host Bootstrap Readiness And Clean-Detach Parking + +Status: corrective follow-on draft. This SOW narrows and sequences the remaining work after [23-host-orchestrator-durable-session-and-parked-resumable-ownership.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/23-host-orchestrator-durable-session-and-parked-resumable-ownership.md) so the original manual smoke failure from [PLAN-22.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-22.md) is actually fixed in the runtime, not only modeled in persisted state and tests built from synthetic parked sessions. + +This slice does not back out the new posture, inbox, or `reattach` semantics. It keeps those features and finishes the missing runtime seams: + +- hidden owner-helper startup readiness must stop requiring a continuously attached live owner when a valid resumable host session already exists, +- clean bootstrap/control-stream exit must park a valid host session instead of invalidating it, +- and the public prompt bridge must keep its terminal-delivery guarantee without pretending the deeper startup contract is already durable. + +## Objective + +Fix the original host smoke-failure path while keeping the new durable-session features. + +This slice is done only when all of the following are true: + +1. A host `substrate agent start --backend --prompt ...` can succeed even when the prompt-driven backend behaves like `codex exec`: emit a valid session handle, become resumable, then let the bootstrap control stream end cleanly. +2. The resulting orchestration session parks deterministically: + - `parked_resumable` when the session remains non-terminal, `attached_participant_id == null`, `pending_inbox_count == 0`, and at least one authoritative host participant remains `resume_eligible == true`; + - `awaiting_attention` when the session remains non-terminal, `attached_participant_id == null`, and `pending_inbox_count > 0`; + - and never collapses to `invalidated` / `terminal` solely because the bootstrap control stream ended cleanly after continuity was established. +3. `substrate agent turn --session ... --backend ... --prompt ...` and `substrate agent reattach --session ...` continue to work against that legitimately parked session. +4. The new session posture, participant attachment fields, and session-local durable inbox from slice `23` remain intact. +5. The public prompt bridge still guarantees explicit `Completed` or `Failed` after `Accepted`, but the runtime no longer depends on that bridge-level fallback to mask a startup-contract defect. + +## Why This Follow-On Exists + +Slice `23` landed important scaffolding: + +- explicit session posture, +- additive host attachment metadata, +- a canonical session-local durable inbox, +- resumed parked-session tests, +- and a bridge-level synthetic terminal `Failed` when a private prompt owner drops after `Accepted`. + +Those are all useful and should remain. + +But the original manual failure path still survives because the runtime seams that caused it were not actually changed: + +- `run_start(...)` still waits for hidden owner-helper readiness before public prompt submission in [agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:301). +- `wait_for_hidden_owner_helper_readiness(...)` still delegates to `hidden_owner_helper_launch_ready(...)` in [control.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/control.rs:466). +- `hidden_owner_helper_launch_ready(...)` still requires `participant.is_authoritative_live()` and `owner_process_is_alive(participant)` in [state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:924), which is the old attached-live bootstrap model. +- when the attached control stream ends after live ownership, the event-task teardown path still invalidates the orchestrator in [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:2608). +- the new parking path currently lives in explicit shutdown handling rather than the hidden-owner-helper bootstrap teardown seam in [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:4807). + +That means the implementation can now model and resume synthetic parked sessions, but it still cannot reliably *create* them through the original host bootstrap path that failed in manual testing. + +## Relationship To Slice 23 + +This SOW is a completion and correction slice for `23`, not a restart. + +Keep from `23`: + +- explicit `active_attached | parked_resumable | awaiting_attention | terminal` posture, +- additive host participant attachment and `resume_eligible` fields, +- session-local durable inbox under `sessions//inbox/`, +- `reattach` as attached-owner recovery only, +- resolved-item retention and later compaction eligibility, +- greenfield session-root state authority, +- and additive inbox correlation compatibility. + +Fix after `23`: + +- readiness still tied to attached-live ownership, +- invalidation-on-clean-bootstrap-exit still active, +- manual host `start` clean-exit path still failing, +- and test coverage still centered on synthetic parked state rather than the original failing bootstrap seam. + +This slice must not remove the new features just because the first implementation missed the root seam. + +## Exact Remaining Defect Statement + +The original bug is still: + +1. Substrate assumes the hidden owner-helper bootstrap run must remain a durable attached-control session until explicit cancel. +2. A `codex exec`-style backend can emit a valid session handle and become resumable, then end the bootstrap control stream cleanly. +3. The runtime still interprets that control-stream end as failure or invalidation instead of valid parked continuity. + +The secondary weakness is still: + +1. The prompt bridge emits `Accepted` before real prompt completion is durably guaranteed. +2. A later owner drop is rendered as an explicit failure now, which is better than EOF. +3. But the lifecycle contract still needs to stop manufacturing that failure in the first place when the session is actually valid and parked. + +## Scope + +In scope: + +- change hidden owner-helper startup readiness so a valid parked-resumable host session counts as ready enough for `agent start` to continue, +- replace invalidation with parking on the exact clean bootstrap/control-stream-end seam when host session continuity is valid, +- preserve invalidation only for truly broken or unrecoverable startup cases, +- align startup completion handling with the new parked-session contract, +- add a true regression test for the manual smoke path, +- and tighten docs/tests around the distinction between: + - successful parked bootstrap, + - real startup failure, + - and post-`Accepted` terminal failure. + +Out of scope: + +- new public verbs or selector changes, +- world-root broadening or detached-world recovery broadening, +- inbox product UX such as listing or browsing inbox items, +- changing the durable inbox outer envelope again, +- changing compaction policy, +- or backing out the new posture/participant/inbox fields from `23`. + +## Required Runtime Contract + +### 1. Hidden owner-helper readiness must accept valid parked host continuity + +For host orchestrator startup, readiness must no longer mean only: + +- session is `Active`, +- exact authoritative participant is selected, +- participant is authoritative-live, +- owner PID is alive, +- internal session id exists when required. + +It must also allow the new valid detached host posture when all of the following are true: + +- the orchestration session remains non-terminal, +- `active_session_handle_id` still points at the authoritative host participant selected for this orchestration session, +- `attached_participant_id == null`, +- the persisted session posture is already normalized to either: + - `parked_resumable` when `pending_inbox_count == 0`, or + - `awaiting_attention` when `pending_inbox_count > 0`, +- the authoritative host participant remains `resume_eligible == true`, +- the authoritative host participant has `attached_client_present == false`, +- a valid `uaa_session_id` exists when required by the backend resume path, +- and no participant or session lifecycle field marks the session invalidated, stopped, failed, or otherwise terminal. + +This is the runtime seam that lets a `codex exec`-style bootstrap succeed without requiring a permanently attached owner process. + +The minimum persisted resumability proof for this slice is therefore: + +- session `state` is non-terminal, +- session `posture` is `parked_resumable` or `awaiting_attention`, +- session `attached_participant_id == null`, +- session `active_session_handle_id` still matches the authoritative host participant, +- session `pending_inbox_count` agrees with the normalized posture, +- participant `resume_eligible == true`, +- participant `attached_client_present == false`, +- participant `uaa_session_id` is populated when the backend resume path requires it. + +Anything weaker than that is not enough to treat clean bootstrap exit as resumable continuity. + +### 2. Clean bootstrap control-stream end must park, not invalidate + +When the hidden owner-helper bootstrap path has already established valid host session continuity and the control stream ends cleanly: + +- the participant must release attachment diagnostics, +- the session must remain `state == Active`, +- the session posture must normalize deterministically: + - `parked_resumable` when `pending_inbox_count == 0`, + - `awaiting_attention` when `pending_inbox_count > 0`, +- and the participant must remain `resume_eligible == true`. + +This path must not: + +- transition the participant or session to `Invalidated`, +- mark the session `terminal`, +- or emit startup failure solely because the owner process is no longer attached after continuity was established. + +### 3. Invalidations remain for truly broken startup cases + +The runtime must still fail closed when: + +- ownership was never established, +- the session handle or `uaa_session_id` required for resume was never surfaced, +- `active_session_handle_id` does not match the authoritative host participant, +- the authoritative host participant record is missing, terminal, or not host-routable, +- `resume_eligible == false`, +- `attached_participant_id != null` after clean detach normalization should already have happened, +- `pending_inbox_count` and `posture` disagree with the persisted invariants from ADR-0047 and `PLAN.md`, +- the session `state` is `Invalidated`, `Stopped`, `Failed`, or otherwise terminal, +- the selected target is world-only or detached-world follow-up would be required, +- or the control stream ends with concrete evidence that the session is not safely resumable under the persisted contract. + +This slice is not “always park on stream end.” It is “park only when the new durable-session contract says the host session is already valid.” + +### 4. The public prompt bridge keeps explicit terminal delivery + +The post-`Accepted` bridge rule from `23` remains: + +- once `Accepted` is emitted, the request must terminate with `Completed` or `Failed`. + +But the runtime should now hit the `Failed` path less often for clean host bootstrap behavior because more valid clean-detach cases are classified as parked continuity rather than owner death. + +### 5. `turn` and `reattach` semantics stay as already corrected + +This slice must preserve: + +- `turn` as prompt-taking resume against exact `(orchestration_session_id, backend_id)`, +- `reattach` as attached-owner recovery only, +- and detached-world follow-up as fail closed. + +## Concrete Work Breakdown + +### 1. Rework hidden owner-helper readiness classification + +Update the startup/readiness path across: + +- [agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs) +- [control.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/control.rs) +- [state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) + +Required change: + +- `hidden_owner_helper_launch_ready(...)` must stop treating attached-live ownership as the only successful bootstrap readiness state. +- It must accept a valid parked/resume-eligible host session when the authoritative linkage and required internal session id are already in place. + +### 2. Rewrite the bootstrap event/completion teardown seam + +Update the hidden owner-helper lifecycle teardown in: + +- [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) + +Required change: + +- the event-task branch that currently invalidates on `shell-owned orchestrator control stream ended before completion observation` must instead route through a “park if continuity already established” decision for host orchestrators. +- the completion-side startup failure path must align with that same decision so event/completion race ordering cannot reintroduce terminal invalidation for resumable host sessions. + +### 3. Preserve explicit shutdown parking but do not rely on it as the only parking seam + +The explicit shutdown parking path added in `23` stays. This slice extends the same semantics to the hidden-owner-helper bootstrap lifecycle seam instead of leaving the manual smoke path on old invalidation logic. + +### 4. Align status/projection logic with the real startup contract + +Any status-session completeness or readiness projection that still assumes attached-live owner truth as the only valid active host continuity must be updated to respect the new parked bootstrap outcomes. + +Required projection rule: + +- posture is recomputed from authoritative session + participant + pending-inbox truth, not command-local heuristics; +- `attached_participant_id != null` normalizes to `active_attached`; +- `attached_participant_id == null` plus `pending_inbox_count > 0` normalizes to `awaiting_attention`; +- `attached_participant_id == null` plus `pending_inbox_count == 0` plus at least one authoritative host participant with `resume_eligible == true` normalizes to `parked_resumable`; +- terminal lifecycle state normalizes to `terminal`. + +This slice should not leave any read path free to infer parked versus attention-needed versus terminal from `owner_process_is_alive(...)` alone. + +### 5. Add the missing regression tests + +Required tests: + +- a true `agent start` / hidden-owner-helper bootstrap regression proving that: + - a fake host backend can emit a valid session handle, + - then end the bootstrap control stream cleanly, + - and the resulting session persists as: + - `parked_resumable` when `pending_inbox_count == 0`, + - `awaiting_attention` when detached pending inbox work exists, + - and never `invalidated` solely because the bootstrap stream ended cleanly. +- a follow-up test proving that exact public `turn` succeeds against the session created by that bootstrap path. +- a follow-up test proving `reattach` restores attached ownership for that same session. +- a guard test proving that genuinely broken startup without resumability still fails closed. + +Synthetic prewritten parked-session tests from `23` remain useful, but they are not sufficient on their own. + +## Acceptance Criteria + +This slice is done only when all of the following are true: + +1. The original manual smoke path that motivated this work succeeds: + - `substrate agent start --backend --prompt ...` can create a valid host session even if the bootstrap control stream ends cleanly after session establishment. +2. That session persists with the exact normalized posture required by ADR-0047 and `PLAN.md`: + - `parked_resumable` when detached and `pending_inbox_count == 0`, + - `awaiting_attention` when detached and `pending_inbox_count > 0`, + - never `invalidated` or `terminal` solely because bootstrap ended cleanly. +3. `substrate agent turn --session ... --backend ... --prompt ...` succeeds against that exact parked session. +4. `substrate agent reattach --session ...` succeeds against that exact parked session and restores attached host ownership without submitting a prompt. +5. Truly broken startup without resumability still fails closed as `runtime_start_failed`. +6. Detached-world follow-up remains fail closed. +7. The prompt bridge still guarantees explicit terminal delivery after `Accepted`. +8. The durable posture, participant, and inbox features from `23` remain intact. + +## Testing Expectations + +Required coverage: + +- startup readiness classification for hidden owner-helper launch +- bootstrap event-stream teardown classification +- bootstrap completion race alignment +- posture normalization from authoritative session + participant + pending-inbox truth +- public host `start` with a clean-exit fake backend +- public host `turn` after parked bootstrap +- public host `reattach` after parked bootstrap +- fail-closed startup when resumability prerequisites are absent +- existing parked-session synthetic tests as non-regression support + +Recommended commands: + +```bash +cargo test -p shell async_repl -- --nocapture +cargo test -p shell agent_runtime::state_store -- --nocapture +cargo test -p shell agent_runtime::control -- --nocapture +cargo test -p shell --test agent_public_control_surface_v1 -- --nocapture +``` + +Manual validation should re-run the real CLI host flow, not only harness tests: + +```bash +substrate agent start --backend --prompt "hello" --json +substrate agent turn --session --backend --prompt "next" --json +substrate agent reattach --session --json +``` + +The manual result is green only when the original failure mode is gone: + +- no `runtime_start_failed` on clean bootstrap continuity, +- no `shell-owned orchestrator control stream ended before completion observation` terminal collapse for a resumable session, +- detached clean-exit with no pending inbox normalizes to `parked_resumable`, +- detached clean-exit with pending inbox work normalizes to `awaiting_attention`, +- and no need to rely on synthetic fixtures to claim the bootstrap path works. diff --git a/llm-last-mile/25-host-durable-session-closeout-and-qa-hardening.md b/llm-last-mile/25-host-durable-session-closeout-and-qa-hardening.md new file mode 100644 index 000000000..37db21f58 --- /dev/null +++ b/llm-last-mile/25-host-durable-session-closeout-and-qa-hardening.md @@ -0,0 +1,279 @@ +# SOW: Host Durable Session Closeout And QA Hardening + +Status: truth-aligned remaining-work draft. This SOW closes the next narrow slice after [23-host-orchestrator-durable-session-and-parked-resumable-ownership.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/23-host-orchestrator-durable-session-and-parked-resumable-ownership.md) and [24-fix-host-bootstrap-readiness-and-clean-detach-parking.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/24-fix-host-bootstrap-readiness-and-clean-detach-parking.md). It is anchored to [ADR-0047 — Host Orchestrator Durable Session and Parked-Resumable Ownership](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/project_management/adrs/draft/ADR-0047-host-orchestrator-durable-session-and-parked-resumable-ownership.md), [HOST_ORCHESTRATOR_INTENDED_BEHAVIOR_TRUTH.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/HOST_ORCHESTRATOR_INTENDED_BEHAVIOR_TRUTH.md), and [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md). + +This is not a redesign slice. The durable session model already exists. The public verbs already exist. Parked sessions already persist. `substrate agent status --json` already projects the key durable-session fields. Canonical inbox item storage already exists. The remaining work is to regression-proof the already chosen contract and keep docs, tests, and operator guidance aligned to it. + +## Frozen Public Contract + +This slice assumes and preserves the following contract: + +1. `substrate agent start --backend --prompt ...` creates or binds the durable host orchestration session and uses the user prompt as the true initial backend prompt. +2. `substrate agent turn --session --backend --prompt ...` is prompt-taking follow-up on that same durable session. +3. `substrate agent reattach --session ` is attached-owner recovery only for that same durable session. +4. `substrate agent stop --session ` is the canonical closeout path for attached and parked durable host sessions. +5. `substrate agent status --json` is the authoritative parked-session read surface for live-runtime `posture`, `attached_participant_id`, and `pending_inbox_count`. +6. Detached-world follow-up stays fail-closed until `reattach` restores an active host owner. +7. Durable inbox behavior stays narrow: persistence exists, posture normalization into `awaiting_attention` exists, internal ack/dismiss support exists, and dev-support/test ingress exists, but no public inbox command surface or automatic resume-from-inbox workflow is shipped. + +## Objective + +Finish the durable host-session closeout slice by making the frozen contract above regression-proof in code, tests, and docs. + +This slice is done only when all of the following are true: + +1. The repo stops describing parked-session recovery semantics as open design. +2. `stop` keeps working correctly for the same durable session whether it is `active_attached`, `parked_resumable`, or `awaiting_attention`. +3. The already-landed `status` projection is treated as a hardening seam, not an open feature seam. +4. Detached-world follow-up continues to fail closed until `reattach` restores an active host owner. +5. The repo stops implying that persisted inbox scaffolding means a public inbox workflow or automatic runtime resume mechanism. + +## Already Landed And Assumed + +This SOW assumes the following are already true and are not being redesigned here: + +- `substrate agent start`, `turn`, `reattach`, `fork`, and `stop` already exist as public verbs in [cli.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/cli.rs:520). +- Public follow-up targeting remains exact `(orchestration_session_id, backend_id)`. +- Durable parked host postures already exist in [orchestration_session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/orchestration_session.rs:89). +- `substrate agent status --json` already publishes authoritative `posture`, `attached_participant_id`, and `pending_inbox_count` for live-runtime rows, and trace fallback rows keep those fields unset as documented in [docs/USAGE.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/USAGE.md:84). +- Detached authoritative parked sessions already remain status-visible from canonical session truth via [state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:66). +- Detached posture classification already comes from persisted session plus participant truth through [classify_public_session_posture(...)](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:2191) and [valid_detached_host_continuity_posture(...)](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:2216). +- Session-local inbox persistence already exists through [persist_inbox_item(...)](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:1039), along with load/list/ack/dismiss support and posture updates. + +## Current Repo Truth + +### `status` is closed as a product contract + +This slice must not describe `status` as a primary missing feature. + +What is already true: + +- `build_status_report(...)` reads live session rows from `state_store.list_status_sessions_for_agent(...)` before trace fallback in [agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:1333). +- parked and awaiting-attention sessions remain visible from authoritative session-root truth in [state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:3695). +- trace fallback does not invent durable-session posture truth in [agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:1389). + +The remaining work on `status` is only: + +- protect that behavior from regression, +- prove parked sessions stay correctly visible as control and readiness code evolves, +- and keep docs accurate about what those fields do and do not mean. + +### Inbox persistence is real and intentionally narrow + +What is clearly real today: + +- canonical inbox item persistence under the session root, +- `pending_inbox_count` as durable state, +- posture normalization from parked to `awaiting_attention` when pending work exists, +- internal ack/dismiss support for retained items, +- and dev-support/test helpers proving those mechanics exist. + +What this slice must keep explicit: + +- no public inbox command surface is shipped, +- no public inbox-driven automatic resume path is shipped, +- and docs must not imply broader productization than the code currently implements. + +### Detached-world follow-up remains fail-closed + +The current supported rule is narrow: + +- detached host recovery uses `reattach`, +- host prompt-taking follow-up uses `turn`, +- and detached-world follow-up remains fail-closed until `reattach` restores an active host owner. + +This slice must preserve that fail-closed rule rather than soften it. + +## In Scope + +- keep the `turn` versus `reattach` contract frozen and documented truthfully, +- harden `reattach` so success means real durable attached truth for the named session, +- keep `stop` first-class for valid parked sessions, +- regression-proof the already-landed `status` projection, +- keep detached-world follow-up fail-closed until `reattach`, +- and freeze the minimum supported inbox contract to persistence, posture normalization, internal ack/dismiss, and dev-support/test ingress only. + +## Out Of Scope + +This slice does not include: + +- fuzzy routing or default-agent routing, +- new root selector forms, +- public world-root `start`, +- a broad new inbox UX, +- a public inbox command surface, +- automatic resume from inbox items, +- detached-world follow-up without `reattach`, +- durable inbox schema redesign, +- a daemon/router redesign, +- or another durable-session model rewrite. + +## Concrete Work Breakdown + +### 1. Keep the public parked-session contract frozen + +Primary anchors: + +- `run_turn(...)` and `run_reattach(...)` in [agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:492) +- target resolution in [state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:892) +- intended behavior notes in [HOST_ORCHESTRATOR_INTENDED_BEHAVIOR_TRUTH.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/HOST_ORCHESTRATOR_INTENDED_BEHAVIOR_TRUTH.md) + +Required outcome: + +- docs and operator guidance consistently say `turn` is prompt-taking follow-up on the same durable session, +- `reattach` remains attached-owner recovery only, +- and no packet or truth doc reopens that contract as a product-design choice. + +### 2. `reattach` success must mean attached truth is actually restored + +Primary anchors: + +- [run_reattach(...)](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:565) +- [wait_for_hidden_owner_helper_readiness(...)](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/control.rs:645) +- detached continuity validation in [state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:2216) + +Required outcome: + +- `reattach` does not report success until the durable session has truly converged back to `active_attached`, +- the same orchestration session remains the winning session before and after reattach, +- and immediate follow-up `status`, `turn`, and `stop` work against that same session without requiring a second recovery step. + +### 3. Durable `stop` must remain first-class for parked sessions + +Primary anchors: + +- [run_stop(...)](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:667) +- stop closeout handling in [control.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/control.rs:867) +- public target resolution in [state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:765) + +Required outcome: + +- `stop` works for `active_attached`, `parked_resumable`, and `awaiting_attention`, +- the detached parked path does not require a currently live attached-owner PID when durable truth already says the session is valid, +- and the durable parent session closes terminally under the same orchestration session id. + +### 4. Keep the inbox contract narrow + +Primary anchors: + +- canonical inbox persistence in [state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:1039) +- posture updates and pending-count normalization in [state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:1726) +- current dev-support seam in [agent_dev_support.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_dev_support.rs:1) + +Required outcome: + +- the repo explicitly states that shipped inbox behavior is limited to persistence, posture normalization, internal ack/dismiss support, and dev-support/test ingress, +- no doc implies a public inbox command surface, +- and no doc implies automatic resume or public workflow productization from inbox items. + +### 5. Keep detached-world follow-up fail-closed until `reattach` + +Primary anchors: + +- detached-world guidance in [agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:535) +- detached continuity validation in [state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:2216) +- current intended behavior notes in [HOST_ORCHESTRATOR_INTENDED_BEHAVIOR_TRUTH.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/HOST_ORCHESTRATOR_INTENDED_BEHAVIOR_TRUTH.md) + +Required outcome: + +- the repo keeps one explicit current-state rule for detached-world follow-up, +- that rule stays fail-closed until `reattach` restores an active host owner, +- and no doc softens that rule into implicit resume or auto-reattach semantics. + +### 6. Treat `status` as hardening, not greenfield + +Primary anchors: + +- [build_status_report(...)](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:1270) +- [status_visible_participants(...)](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:66) + +Required outcome: + +- the current authoritative parked-session fields remain correct, +- parked sessions stay visible from canonical session truth, +- and later control/readiness changes do not reintroduce attached-live-only assumptions. + +## Required Test Additions Or Tightening + +### Integration coverage in [agent_public_control_surface_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_public_control_surface_v1.rs) + +Required scenarios: + +- parked `start` -> `status --json` still shows the same orchestration session as parked, +- parked `start` -> `turn` -> parked again stays on the same orchestration session id, +- parked session with pending inbox work -> `status --json` shows `awaiting_attention`, +- parked session -> `reattach` succeeds only when durable `active_attached` truth is actually restored, +- a successful `reattach` leaves immediate `turn` and `stop` usable against the same session, +- parked session -> `stop` succeeds without a live attached-owner plane, +- awaiting-attention parked session -> `stop` also succeeds, +- detached-world follow-up stays fail-closed until `reattach`, +- and parked retained inbox state does not imply a public inbox workflow or automatic resume path. + +### Store-level coverage in [state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) + +Required scenarios: + +- detached host continuity remains a valid public control target only when the persisted resume contract is intact, +- detached parked participants remain status-visible from authoritative session truth when owner PID liveness is absent, +- inbox persistence and resolution keep `pending_inbox_count` and posture synchronized under the supported item kinds, +- and contradictory session/participant linkage still fails closed for control while remaining readable on status when appropriate. + +### Runtime lifecycle coverage in [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) + +Required scenarios: + +- detached parking normalization preserves the exact fields later `turn`, `reattach`, `stop`, and `status` depend on, +- pending inbox escalation keeps the same session in `awaiting_attention` without invalidation, +- and closeout paths do not regress parked durable sessions back into attached-live-only assumptions. + +### Inbox realism coverage + +Required scenarios: + +- at least one supported non-test runtime path proves how a real inbox item is created for a parked session, +- that proof stays limited to persistence and posture normalization unless broader public behavior is intentionally shipped in a later slice, +- and tests do not use dev-support helpers to imply a public inbox workflow or automatic runtime resume story that does not exist. + +## Acceptance Criteria + +- the repo keeps one explicit public parked-session recovery contract: `turn` for prompt-taking follow-up and `reattach` for attached-owner recovery. +- `reattach` success always corresponds to real durable `active_attached` truth for the same orchestration session. +- `stop` works for valid parked host sessions without requiring a live attached owner process. +- the already-landed `status` projection remains authoritative and non-regressive for parked sessions. +- detached-world follow-up stays fail-closed until `reattach` restores an active host owner. +- the minimum supported inbox contract is explicit and limited to persistence, posture normalization, internal ack/dismiss support, and dev-support/test ingress. +- docs and tests stop implying a public inbox workflow, automatic inbox-driven resume, or any detached-world follow-up path that bypasses `reattach`. + +## Validation Expectations + +- add or tighten targeted tests in the files above for each acceptance path, +- run the touched shell test coverage and full workspace tests: + - `cargo test --workspace -- --nocapture` +- manual validation for this slice must explicitly exercise: + - parked `start`, + - parked `status`, + - parked `turn`, + - parked `reattach`, + - parked `stop`, + - and detached-world fail-closed follow-up before and after `reattach`. + +## Docs And Truth Sync + +When this slice is closed, the truth docs must stop advertising uncertainty that the code has already resolved and must stop implying inbox behavior that is not actually supported. + +At minimum: + +- update [HOST_ORCHESTRATOR_INTENDED_BEHAVIOR_TRUTH.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/HOST_ORCHESTRATOR_INTENDED_BEHAVIOR_TRUTH.md) to reflect the frozen parked-session recovery contract, +- update [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md) so its remaining-gap wording reflects the true post-closeout state, +- and update [docs/USAGE.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/USAGE.md) so operator guidance matches the frozen semantics for parked recovery, `stop`, detached-world fail-closed behavior, and narrow inbox state. + +## Done Shape + +This slice is complete when the repo no longer hand-waves any of these points: + +- `turn` is prompt-taking follow-up on the same durable session, +- `reattach` is attached-owner recovery only, +- `stop` is the canonical closeout path for attached and parked durable host sessions, +- `status --json` is the authoritative parked-session read surface, +- detached-world follow-up stays fail-closed until `reattach`, +- and inbox behavior is documented as narrow retained state rather than as a public inbox product or automatic resume mechanism. diff --git a/llm-last-mile/26-async-persistent-session-bootstrap-readiness.md b/llm-last-mile/26-async-persistent-session-bootstrap-readiness.md new file mode 100644 index 000000000..3a71242ab --- /dev/null +++ b/llm-last-mile/26-async-persistent-session-bootstrap-readiness.md @@ -0,0 +1,560 @@ +# SOW: Async Persistent-Session Bootstrap Readiness Split + +Status: implementation-oriented corrective draft. This SOW defines the medium-scope fix for the non-Linux persistent-session startup seam after the macOS/Lima interactive startup panic traced through [world-mac-lima/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/lib.rs:78), [platform_world/mod.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/platform_world/mod.rs:51), [world_persistent_session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_persistent_session.rs:730), and [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:443). It is a follow-on to the macOS/Lima parity work in [21-macos-lima-shared-owner-and-member-runtime-parity.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/21-macos-lima-shared-owner-and-member-runtime-parity.md), but it intentionally does not reopen the larger `WorldBackend` abstraction or the persistent-session protocol itself. + +This slice exists because the repo now has a real async caller for persistent world-session startup, but that caller still routes through a synchronous readiness hook built for older sync-oriented bootstrap surfaces. + +## Objective + +Land one explicit split between: + +- synchronous world-readiness used by legacy/bootstrap/sync request-building code, and +- async world-readiness used by the already-async persistent-session startup path. + +This slice is done only when all of the following are true: + +1. Persistent-session startup no longer calls the synchronous `ctx.ensure_ready` closure from inside the async REPL runtime. +2. macOS/Lima interactive startup no longer panics when launched under the current-thread Tokio runtime used by the REPL. +3. The sync `PlatformWorldContext.ensure_ready` surface remains available for the existing synchronous bootstrap and request-builder call sites that still rely on it. +4. The new async startup seam reuses the same fail-closed transport/bootstrap rules as the sync path instead of inventing a second backend model. +5. The fix is scoped below the `WorldBackend` trait boundary. `WorldBackend` remains synchronous in this slice. +6. Tests and docs make the split explicit so future async callers do not regress back through the synchronous bridge. + +## Why This Follow-On Exists + +The crash is the symptom of a real design mismatch, not just an unlucky Tokio call. + +### Current startup path + +- The interactive REPL starts on a current-thread Tokio runtime in [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:452). +- REPL startup immediately opens a world session through [start_world_session(...)](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:6792) and [open_world_session(...)](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:6506). +- `open_world_session(...)` calls [ReplPersistentSessionClient::start_with(...)](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_persistent_session.rs:125), which calls [build_ws_and_start_session_frame(...)](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_persistent_session.rs:730). +- That async bootstrap helper still uses `(ctx.ensure_ready.as_ref())()?` in [world_persistent_session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_persistent_session.rs:775). +- On macOS, `ctx.ensure_ready` is the closure installed in [platform_world/mod.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/platform_world/mod.rs:151), which calls `backend.ensure_session(&spec)`. +- On Lima, `ensure_session(...)` immediately calls `ensure_agent_ready()` in [world-mac-lima/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/lib.rs:512). +- `ensure_agent_ready()` performs a capabilities probe via `block_on_compat(...)` in [world-mac-lima/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/lib.rs:274). +- `block_on_compat(...)` currently assumes any active Tokio runtime can use `block_in_place(...)` in [world-mac-lima/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/lib.rs:79), which is false for the REPL’s current-thread runtime and causes the panic. + +### Why the sync seam was probably intentional + +This is not good evidence that the sync hook was a mistake when introduced. The repo still has real synchronous consumers for it: + +- platform bootstrap in [routing/world.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/world.rs:100), +- agent/world request builders in [world_ops.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs:1316), [world_ops.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs:1411), and [world_ops.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs:1496), +- and world-dependent workspace sync in [workspace_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/workspace_cmd.rs:557). + +The design error is narrower: + +- an async startup caller is now real, +- but the async caller still crosses the old sync bridge, +- and that bridge reaches backend-local sync-over-async logic that is only safe under some runtime flavors. + +## Chosen Implementation Shape + +This SOW chooses the medium-scope fix. + +### Keep in this slice + +- keep `WorldBackend` synchronous in [world-api/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-api/src/lib.rs:263), +- keep `PlatformWorldContext.ensure_ready` for sync callers, +- keep the persistent-session websocket protocol and `ready` validation contract, +- keep platform-specific backend ownership of VM/forwarder/agent readiness, +- and keep fail-closed behavior when readiness, forwarding, or agent connectivity cannot be proven. + +### Add in this slice + +- an explicit async readiness/bootstrap seam for persistent-session startup, +- shared internal readiness logic that can be consumed from both sync and async entrypoints, +- backend-specific async readiness support for non-Linux forwarded backends, +- and tests that prove the async path no longer routes through the sync bridge. + +### Deliberately avoid in this slice + +- converting `WorldBackend` into an async trait, +- widening every backend call site to async, +- rewriting LinuxLocalBackend, +- redesigning world bootstrap for every command surface, +- or collapsing all readiness into one immediate grand refactor. + +## Sizing And Rationale + +This is a medium slice, not a tiny hotfix and not a trait-level rewrite. + +The sizing rationale is: + +- GitNexus shows the async REPL seam itself is relatively contained: + - [ReplPersistentSessionClient::start_with(...)](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_persistent_session.rs:125) has low upstream fan-in and is primarily reached from [open_world_session(...)](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:6506). + - [open_world_session(...)](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:6506) is primarily consumed by REPL world start/restart flows. +- The broader sync readiness surface is clearly shared across bootstrap and request-building seams, so deleting it outright would have a larger cross-cutting blast radius. +- `WorldBackend` itself is implemented by Linux, macOS/Lima, Windows/WSL, and the shell test stub, so widening that trait now would force a much larger migration than this fix needs. + +The right scope is therefore: + +- split the async caller off the sync readiness hook, +- share backend-local readiness internals where possible, +- preserve the sync API for existing sync callers, +- and defer any full trait conversion until there is a separately justified backend-API slice. + +## Current Repo Truth + +### 1. The async caller is already real + +The repo is not planning for an async persistent-session caller in the abstract. It already has one: + +- the REPL runs on Tokio in [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:452), +- world session startup is awaited in [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:527), +- persistent-session websocket bootstrap is async in [world_persistent_session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_persistent_session.rs:125), +- and the websocket connect/handshake path is already async through `UnixStream::connect`, `TcpStream::connect`, and `tungs::client_async` in [world_persistent_session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_persistent_session.rs:742). + +This means the sync bridge in that path is no longer merely a compatibility convenience. It is now an architectural mismatch. + +### 2. The sync readiness surface is shared and still needed + +The sync readiness hook is not only a macOS artifact: + +- `PlatformWorldContext` stores synchronous `ensure_ready` at [platform_world/mod.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/platform_world/mod.rs:51). +- macOS world bootstrap uses it in [routing/world.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/world.rs:103). +- sync request builders use it in [world_ops.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs:1327), [world_ops.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs:1420), and [world_ops.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs:1505). + +This SOW must not break those call sites in the name of fixing the REPL panic. + +### 3. The backend-local readiness logic already exists, but only behind sync bridges + +macOS/Lima already owns the right sub-steps: + +- VM boot/verification, +- forwarding setup, +- client construction, +- and agent capabilities probe + +through [ensure_vm_running(...)](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/lib.rs:117), [ensure_forwarding(...)](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/lib.rs:260), [build_agent_client(...)](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/lib.rs:485), and [ensure_agent_ready(...)](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/lib.rs:274). + +Windows/WSL mirrors the same overall pattern through [ensure_agent_ready(...)](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-windows-wsl/src/backend.rs:251) and its own sync-over-async helper in [backend.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-windows-wsl/src/backend.rs:159). + +This is why the chosen fix should extract/shared-split readiness logic rather than inventing a shell-side second implementation of VM/forwarder bootstrap. + +### 4. The protocol itself is not the problem + +The persistent-session protocol contract in [docs/internals/repl/persistent_session.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/internals/repl/persistent_session.md) remains valid. The problem is not: + +- the `StartSession` frame, +- the `ready` handshake, +- cwd/env persistence, +- or shared-world semantics. + +The problem is the readiness seam used immediately before opening that websocket. + +## In Scope + +- add an async readiness/bootstrap seam for persistent-session startup, +- route [build_ws_and_start_session_frame(...)](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_persistent_session.rs:730) through that async seam instead of `ctx.ensure_ready`, +- refactor macOS/Lima backend readiness internals so the async path and sync path share the same VM/forwarding/client/bootstrap rules, +- provide Windows/WSL parity for the same async readiness seam if that path can reach the same sync-over-async bridge, +- keep the existing sync `ensure_ready` closure for synchronous callers, +- keep websocket transport selection, shared-world request construction, and `ready` validation behavior unchanged, +- and add regression coverage and docs proving the split. + +## Out Of Scope + +This slice does not include: + +- converting `WorldBackend` methods to async, +- changing LinuxLocalBackend semantics, +- redesigning `substrate world enable` or bootstrap CLI flows, +- replacing `PlatformWorldContext` wholesale, +- changing `ready.shared_world` or world-generation semantics, +- changing member-runtime placement or durable-session posture logic, +- or broadening unrelated public agent control surfaces. + +## Main Code Seams + +### Async caller seam + +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:443) + - creates the current-thread runtime and starts the world session during REPL startup +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:6506) + - `open_world_session(...)` +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:6792) + - `start_world_session(...)` +- [crates/shell/src/execution/routing/dispatch/world_persistent_session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_persistent_session.rs:125) + - `ReplPersistentSessionClient::start_with(...)` +- [crates/shell/src/execution/routing/dispatch/world_persistent_session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_persistent_session.rs:730) + - `build_ws_and_start_session_frame(...)` + +### Shared sync readiness seam + +- [crates/shell/src/execution/platform_world/mod.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/platform_world/mod.rs:51) + - `PlatformWorldContext` +- [crates/shell/src/execution/platform_world/mod.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/platform_world/mod.rs:151) + - macOS `ensure_ready` closure +- [crates/shell/src/execution/routing/world.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/world.rs:100) + - sync world bootstrap +- [crates/shell/src/execution/routing/dispatch/world_ops.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs:1327) + - sync world execute request builder +- [crates/shell/src/execution/workspace_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/workspace_cmd.rs:557) + - separate sync Tokio runtime used for workspace sync capabilities checks + +### Backend-local readiness seam + +- [crates/world-mac-lima/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/lib.rs:78) + - `block_on_compat(...)` +- [crates/world-mac-lima/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/lib.rs:274) + - `ensure_agent_ready(...)` +- [crates/world-mac-lima/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/lib.rs:511) + - `WorldBackend::ensure_session(...)` +- [crates/world-windows-wsl/src/backend.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-windows-wsl/src/backend.rs:159) + - `block_on(...)` +- [crates/world-windows-wsl/src/backend.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-windows-wsl/src/backend.rs:251) + - `ensure_agent_ready(...)` + +## Required Semantics + +### 1. The async startup path must stop crossing the sync bridge + +For persistent-session startup: + +- `build_ws_and_start_session_frame(...)` must no longer call `(ctx.ensure_ready.as_ref())()?`, +- it must instead call an async readiness/bootstrap seam that is safe to await from the current-thread runtime, +- and that seam must return only after platform readiness is sufficiently established to open the websocket transport. + +### 2. The sync startup path must remain supported + +The existing sync world bootstrap and sync request-builder surfaces must continue to work: + +- `PlatformWorldContext.ensure_ready` stays available, +- sync callers do not become async in this slice, +- and the repo does not require a Tokio runtime just to use those existing sync APIs. + +### 3. Readiness logic must stay shared in substance + +The async path and sync path must not drift into two separately maintained definitions of readiness. + +At minimum, both paths must preserve the same platform-specific rules for: + +- VM existence/running checks, +- forwarder establishment, +- agent endpoint discovery, +- capabilities verification, +- and fail-closed error behavior. + +The preferred design is to share backend-local readiness internals and expose: + +- one sync adapter for existing sync callers, +- and one async adapter for async callers. + +### 4. The fix must preserve fail-closed behavior + +The new async path must not weaken failure handling. It must still fail closed when: + +- the VM cannot be started, +- forwarding cannot be established, +- the agent transport is unavailable, +- the capabilities probe fails, +- the socket path or TCP endpoint cannot be reached, +- or the websocket handshake fails. + +This SOW is about runtime-correct readiness, not about making startup more permissive. + +### 5. The fix must not require widening `WorldBackend` + +In this slice, `WorldBackend` in [world-api/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-api/src/lib.rs:263) remains synchronous. + +That means: + +- no `async_trait` conversion, +- no blanket migration of Linux/macOS/Windows backend interfaces, +- and no trait-level API churn outside the chosen async readiness seam. + +### 6. Windows parity must be considered, even if macOS is the reproducer + +The same sync-over-async helper pattern exists in WSL: + +- `block_on(...)` uses `block_in_place(...)` in [backend.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-windows-wsl/src/backend.rs:163). + +This slice should either: + +- migrate Windows persistent-session startup to the same async readiness seam at the same time, or +- explicitly prove that the Windows persistent-session path cannot hit the same issue and leave a documented follow-on if not fixed here. + +The default expectation for this SOW is parity, not a macOS-only special case. + +## Recommended Implementation Shape + +### 1. Split platform-world readiness by caller shape, not by protocol + +Do not move readiness logic into `async_repl.rs` or hardcode Lima-specific startup inside `world_persistent_session.rs`. + +Instead: + +- keep readiness owned by the platform-world/backend seam, +- add an async entrypoint specifically for async callers, +- and let persistent-session bootstrap consume that entrypoint. + +Possible acceptable shapes include: + +- adding an async helper function in `platform_world/mod.rs` that resolves/bootstraps an already-detected context for async use, +- adding backend-specific async readiness methods and dispatching them from platform-world code, +- or adding an async-ready transport/bootstrap descriptor returned from the platform-world layer. + +The design must not depend on storing an async closure directly inside `PlatformWorldContext` unless that shape stays ergonomic and testable across platforms. A free async helper plus the existing sync closure is likely simpler. + +### 2. Extract backend-local readiness internals so sync and async adapters share them + +For macOS/Lima, refactor the internals around: + +- [ensure_vm_running(...)](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/lib.rs:117), +- [ensure_forwarding(...)](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/lib.rs:260), +- [build_agent_client(...)](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/lib.rs:485), +- and the capabilities verification part of [ensure_agent_ready(...)](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/lib.rs:274) + +so that: + +- the sync adapter can still implement `ensure_session(...)`, +- and the async adapter can await readiness without routing through `block_on_compat(...)`. + +The same principle applies to Windows/WSL readiness internals. + +### 3. Persistent-session bootstrap should prepare transport asynchronously, then open websocket + +`build_ws_and_start_session_frame(...)` should continue to own: + +- env normalization, +- explicit socket override handling, +- transport selection for Unix/TCP/VSock-forwarded endpoints, +- websocket handshake, +- and `StartSession` frame creation. + +But before those steps it should obtain readiness through the new async seam rather than the sync closure. + +This preserves the existing protocol logic while fixing the readiness boundary. + +### 4. Keep sync request builders on the old surface for now + +The sync builders in `world_ops.rs` and similar surfaces should continue to use sync readiness in this slice. If they later become async, that should be a separate refactor. + +### 5. Do not paper over the issue by only special-casing runtime flavor + +A runtime-flavor guard inside `block_on_compat(...)` would be a useful tactical guardrail, but by itself it does not solve the design bug that an async caller is using the sync bridge. + +This SOW is complete only when the async persistent-session caller has its own async readiness path. + +It is acceptable to also harden `block_on_compat(...)` and the WSL equivalent as defense-in-depth, but that is not the primary success criterion. + +## Concrete Work Breakdown + +### 1. Add an async platform-world readiness/bootstrap seam + +Primary anchors: + +- [crates/shell/src/execution/platform_world/mod.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/platform_world/mod.rs:51) +- [crates/shell/src/execution/routing/dispatch/world_persistent_session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_persistent_session.rs:730) + +Required outcome: + +- there is one explicit async path for preparing a non-Linux forwarded world session transport for persistent-session startup, +- that path is callable from async REPL/bootstrap code without going through `ctx.ensure_ready`, +- and it composes cleanly with existing `pw::detect()` / `pw::get_context()` behavior. + +### 2. Refactor macOS/Lima readiness internals behind sync + async adapters + +Primary anchors: + +- [crates/world-mac-lima/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/lib.rs:117) +- [crates/world-mac-lima/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/lib.rs:260) +- [crates/world-mac-lima/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/lib.rs:274) +- [crates/world-mac-lima/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/lib.rs:511) + +Required outcome: + +- VM boot, forwarding, client construction, and capabilities verification are shared internal steps, +- the sync adapter still satisfies `ensure_session(...)`, +- the async adapter is await-safe under the REPL runtime, +- and persistent-session startup no longer depends on `block_on_compat(...)`. + +### 3. Decide and implement Windows/WSL parity for the same seam + +Primary anchors: + +- [crates/world-windows-wsl/src/backend.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-windows-wsl/src/backend.rs:159) +- [crates/world-windows-wsl/src/backend.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-windows-wsl/src/backend.rs:251) + +Required outcome: + +- the repo either adopts the same async readiness split for WSL persistent-session bootstrap now, +- or proves with tests and comments that the current async path cannot hit the same sync-over-async bridge and documents the follow-on. + +The preferred outcome for this SOW is to implement the parity split now. + +### 4. Rewrite persistent-session bootstrap to consume the async seam + +Primary anchors: + +- [crates/shell/src/execution/routing/dispatch/world_persistent_session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_persistent_session.rs:730) +- [crates/shell/src/execution/routing/dispatch/world_persistent_session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_persistent_session.rs:766) + +Required outcome: + +- the async bootstrap helper gets the context, +- ensures platform readiness through the new async seam, +- then opens the chosen transport and websocket, +- with no call to `ctx.ensure_ready`. + +### 5. Preserve sync readiness call sites unchanged in behavior + +Primary anchors: + +- [crates/shell/src/execution/routing/world.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/world.rs:100) +- [crates/shell/src/execution/routing/dispatch/world_ops.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs:1327) +- [crates/shell/src/execution/workspace_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/workspace_cmd.rs:557) + +Required outcome: + +- sync bootstrap and sync request-builder paths continue to work, +- no caller is forced async accidentally, +- and the only behavior change they see is any intentional backend-local hardening that falls out of sharing readiness internals. + +### 6. Add defense-in-depth runtime hardening + +Primary anchors: + +- [crates/world-mac-lima/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/lib.rs:78) +- [crates/world-windows-wsl/src/backend.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-windows-wsl/src/backend.rs:159) + +Required outcome: + +- backend-local sync-over-async helpers are hardened so they do not panic if another async caller accidentally routes through them in the future, +- but the main architecture still keeps persistent-session bootstrap off those helpers. + +This is defense-in-depth, not the primary fix target. + +### 7. Update docs to explain the split + +Primary anchors: + +- [docs/internals/repl/persistent_session.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/internals/repl/persistent_session.md) +- [docs/WORLD.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/WORLD.md) + +Required outcome: + +- repo docs explain that persistent-session startup is an async bootstrap path with its own readiness adapter, +- docs stop implying that all world readiness always flows through the sync closure, +- and any platform-specific startup notes reflect the real Lima/WSL bootstrap split. + +## Required Test Additions Or Tightening + +### 1. REPL startup regression proving no current-thread panic + +Required scenario: + +- launch the interactive startup path under the same current-thread Tokio runtime shape used by [run_async_repl(...)](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:443), +- force the persistent-session bootstrap path to execute, +- and prove startup no longer panics when platform readiness is established. + +The test does not need to use a real Lima VM if a targeted platform/backing stub can exercise the same async seam honestly. + +### 2. Persistent-session bootstrap test proving the async seam is used + +Required scenarios: + +- async bootstrap succeeds through the new readiness path and reaches websocket connect, +- explicit socket overrides still bypass platform bootstrap as before, +- and bootstrap failures in readiness propagate as ordinary `Result` errors instead of Tokio runtime panics. + +### 3. Sync readiness regression tests + +Required scenarios: + +- sync bootstrap in `routing/world.rs` still works, +- sync request-builders in `world_ops.rs` still work, +- and sharing internal readiness logic does not remove or weaken the existing sync surface. + +### 4. macOS/Lima-focused backend tests + +Required scenarios: + +- forwarded readiness can be established through the async adapter, +- capabilities probe failures fail closed, +- sync adapter still satisfies `ensure_session(...)`, +- and caching/shared-owner behavior is not regressed by the readiness refactor. + +Use or extend the existing macOS test seams around the `session_setup_override` path in [world-mac-lima/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/lib.rs:44) rather than inventing a less honest test seam. + +### 5. Windows/WSL parity tests if included + +Required scenarios: + +- async bootstrap does not route through the sync `block_on(...)` helper, +- warm-script and capabilities failure behavior remains fail closed, +- and sync `ensure_session(...)` behavior is preserved. + +### 6. Manual smoke expectation + +At minimum, the manual repro used during investigation must stop failing on macOS: + +```bash +script -q /dev/null zsh -lc 'RUST_BACKTRACE=1 ~/.substrate/bin/substrate' +``` + +Expected outcome: + +- no `can call blocking only when running on the multi-threaded runtime` panic, +- REPL reaches normal startup or a normal fail-closed readiness error, +- and any failure is a plain user-facing readiness failure rather than a Tokio runtime invariant panic. + +## Acceptance Criteria + +### A. Async startup correctness + +1. Persistent-session startup no longer calls `ctx.ensure_ready` from [world_persistent_session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_persistent_session.rs:775). +2. The REPL current-thread runtime can start a world session on macOS/Lima without the runtime panic observed during investigation. +3. Readiness/connectivity failures surface as normal `Result` failures rather than panic unwinds. + +### B. Sync surface preservation + +1. `PlatformWorldContext.ensure_ready` remains available for sync callers. +2. Sync bootstrap in [routing/world.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/world.rs:100) still behaves as before. +3. Sync request-builders in [world_ops.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs:1316) still behave as before. + +### C. Backend-local readiness sharing + +1. macOS/Lima readiness logic is not duplicated between shell and backend layers. +2. Sync and async readiness adapters share the same platform rules for VM/forwarding/client/capabilities setup. +3. Defense-in-depth hardening prevents future runtime-flavor panics even if another async caller accidentally hits the sync bridge. + +### D. Windows/WSL parity decision + +1. The same readiness-split decision is applied to WSL, or +2. the repo explicitly documents and proves why WSL cannot hit the same failure path and records the follow-on if parity is deferred. + +### E. Documentation + +1. [docs/internals/repl/persistent_session.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/internals/repl/persistent_session.md) reflects the async bootstrap split. +2. Any affected world/bootstrap docs describe the real caller-shape split instead of implying one universal sync readiness path. + +## Validation Checklist + +- `cargo fmt --all` +- `cargo clippy --workspace --all-targets -- -D warnings` +- `cargo test -p substrate-shell -- --nocapture` +- `cargo test -p world-mac-lima -- --nocapture` +- `cargo test -p world-windows-wsl -- --nocapture` +- targeted REPL/persistent-session tests covering startup and restart flows +- macOS manual repro smoke under a pseudo-TTY +- if Windows parity lands in this slice, run the relevant WSL smoke coverage too + +## Important Related Docs And Prior Slices + +- [21-macos-lima-shared-owner-and-member-runtime-parity.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/21-macos-lima-shared-owner-and-member-runtime-parity.md) +- [24-fix-host-bootstrap-readiness-and-clean-detach-parking.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/24-fix-host-bootstrap-readiness-and-clean-detach-parking.md) +- [25-host-durable-session-closeout-and-qa-hardening.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/25-host-durable-session-closeout-and-qa-hardening.md) +- [docs/internals/repl/persistent_session.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/internals/repl/persistent_session.md) +- [docs/WORLD.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/WORLD.md) +- [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md) + +## Explicit Non-Decision + +This SOW intentionally does not answer whether `WorldBackend` should eventually become async. It only freezes the medium-scope near-term answer: + +- async persistent-session bootstrap gets its own async readiness seam now, +- sync world bootstrap and sync request-builders keep their sync seam now, +- and any full backend-trait migration is deferred to a later, separately justified slice. diff --git a/llm-last-mile/ORCH_PLAN-04.md b/llm-last-mile/ORCH_PLAN-04.md new file mode 100644 index 000000000..585977341 --- /dev/null +++ b/llm-last-mile/ORCH_PLAN-04.md @@ -0,0 +1,745 @@ +# ORCH_PLAN-04: Thread World Binding Into Runtime State + +Branch: `feat/thread-world-binding` +Plan source: [PLAN-04.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-04.md) +Execution type: backend-only orchestration plan, no UI scope + +## Summary + +This document is the execution control artifact for `PLAN-04`, not a summary of it. + +The parent agent owns the contract, the single-writer runtime integration path, and the final branch state. Worker lanes exist only where the repo seams permit safe parallelism without corrupting the startup and restart ordering centered in [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs). + +The critical path is: + +1. persist a pending parent [OrchestrationSessionRecord](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/orchestration_session.rs) before the first shared-world attach/create request, +2. ensure that first request is owner-bound, +3. make the persisted parent-session record the live binding authority for `world_id/world_generation`, +4. reorder alerts and host runtime events so they publish only after parent binding persistence succeeds, +5. expose an optional proof in `substrate agent toolbox status --json` while keeping the selected host row in `substrate agent status --json` unchanged. + +This slice must stop if implementation drifts into lease-sidecar expansion, host participant world binding, UI changes, or `PLAN-05` invalidation semantics. + +## Orchestration State Surfaces + +These are run artifacts for the orchestration control plane. They are derived process surfaces, not product source. + +### Canonical run state + +Single local source of truth for the run: + +- `.runs/plan-04/run-state.json` + +Parent-only writes to this file. It tracks: + +- current phase, +- active task IDs, +- branch/worktree assignment, +- worker ownership, +- gate status, +- blocked or green terminal state, +- final closeout pointer. + +If a worker report conflicts with `run-state.json`, the parent trusts `run-state.json` until it explicitly reconciles the discrepancy. + +### Derived run artifacts + +The parent may maintain these local derived surfaces under the repo root: + +- `.runs/plan-04/queue.json` + - ordered execution queue and dispatch metadata +- `.runs/plan-04/session.log` + - append-only parent log for decisions, gate results, merge outcomes, and blockers +- `.runs/plan-04/sentinels/task-m04-a1-preflight.ok` +- `.runs/plan-04/sentinels/task-m04-a2-foundation.ok` +- `.runs/plan-04/sentinels/task-m04-b1-record-authority.ok` +- `.runs/plan-04/sentinels/task-m04-b2-toolbox-proof.ok` +- `.runs/plan-04/sentinels/task-m04-c1-runtime-integration.ok` +- `.runs/plan-04/sentinels/task-m04-d1-world-tests.ok` +- `.runs/plan-04/sentinels/task-m04-d2-status-tests-docs.ok` +- `.runs/plan-04/sentinels/task-m04-e1-closeout.ok` +- `.runs/plan-04/blocked.json` + - present only on blocked termination +- `.runs/plan-04/closeout.md` + - final successful-run closeout + +Sentinel rules: + +1. `.ok` means the parent validated the task output and advanced the run. +2. Absence of a sentinel means the task is not accepted. +3. Blocked termination writes `blocked.json` and does not write downstream `.ok` sentinels. + +## Concurrency Policy + +1. The parent remains the only integrator. +2. The parent remains the only writer of final branch state on `feat/thread-world-binding`. +3. Exact concurrency cap: `2` active worker lanes. +4. Reason for the cap: + - [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) is the dominant overlap surface, + - startup/restart ordering is single-writer work, + - more than two worker lanes here would create merge churn faster than throughput. +5. The parent may run `0`, `1`, or `2` workers depending on phase stability. +6. `async_repl.rs` is single-writer by default. No worker lane edits it while foundational runtime integration is still moving. +7. Test worktrees are launched only after parent runtime integration stabilizes. + +## Hard Guards + +### Locked invariants + +1. The parent-session record is the live binding authority. The host participant manifest is not. +2. The first shared-world `AttachOrCreate` request must be owner-bound and must occur only after a persisted pending parent session record exists. +3. `world_restarted`, `world_restart_required`, and host/orchestrator runtime events that claim a world binding must publish only after parent binding persistence succeeds. +4. `toolbox status --json` may gain an optional proof object; the selected host row in `agent status --json` must remain unchanged. +5. `--no-world` is an explicit bypass path. It must preserve host-only startup ordering and must not attempt world-binding persistence. +6. Lease-sidecar expansion is deferred. No worker may broaden slice scope to make lease files authoritative. +7. Integration stops immediately if `PLAN-04.md`, [llm-last-mile/README.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/README.md), or [05-restart-invalidation-semantics.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/05-restart-invalidation-semantics.md) drift in a way that contradicts these rules. + +### File-level boundaries + +Parent-owned critical overlap surface: + +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) + +Parent-reviewed shared contract surfaces: + +- [crates/shell/src/execution/agent_runtime/orchestration_session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/orchestration_session.rs) +- [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) +- [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs) +- [crates/shell/src/execution/repl_persistent_session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/repl_persistent_session.rs) + +Validation and documentation surfaces: + +- [crates/shell/tests/repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs) +- [crates/shell/tests/agent_successor_contract_ahcsitc0.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_successor_contract_ahcsitc0.rs) +- [llm-last-mile/README.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/README.md) +- [llm-last-mile/05-restart-invalidation-semantics.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/05-restart-invalidation-semantics.md) + +### Non-negotiable stop conditions + +Stop the orchestration and return to planning if any of these occur: + +1. A worker proposes writing `world_id/world_generation` into host-scoped participant manifests. +2. The first attach/create request cannot be owner-bound without changing `PLAN-03` semantics. +3. `toolbox status --json` proof requires exposing `orchestration_session_id` publicly. +4. The selected host status row changes shape or gains world fields. +5. Runtime ordering changes cannot be proven with test coverage in the real shell test files. +6. Two workstreams need concurrent edits to `async_repl.rs`. + +## Approval And Gate Model + +There are no true human approval gates defined for this run. + +Replacement control mechanism: + +1. Parent validation gates at every phase transition. +2. Blocked-path recording in `.runs/plan-04/blocked.json`. +3. Green-path recording in `.runs/plan-04/closeout.md`. +4. No phase transition without a parent-written sentinel for the prior task. + +Green-path rule: + +- The parent writes the task sentinel, updates `run-state.json`, appends `session.log`, and opens the next task set. + +Blocked-path rule: + +- The parent writes `blocked.json`, updates `run-state.json` to `blocked`, appends `session.log`, and stops dispatch. + +## Workstream Plan + +### Orchestration topology + +Parent checkout: + +- current checkout on `feat/thread-world-binding` + +Child worktrees and branches: + +1. `../substrate-m04-record-authority` + - branch: `codex/feat-thread-world-binding-m04-record-authority` +2. `../substrate-m04-toolbox-proof` + - branch: `codex/feat-thread-world-binding-m04-toolbox-proof` +3. `../substrate-m04-world-tests` + - branch: `codex/feat-thread-world-binding-m04-world-tests` +4. `../substrate-m04-status-tests-docs` + - branch: `codex/feat-thread-world-binding-m04-status-tests-docs` + +The parent remains the only integrator. Subagents do not merge each other’s work. They return patches, exact file lists, test evidence, and blockers to the parent. + +### Task index + +Parent-only serialized tasks: + +- `task/m04-a1-preflight` +- `task/m04-a2-foundation` +- `task/m04-c1-runtime-integration` +- `task/m04-e1-closeout` + +Worker-owned tasks: + +- `task/m04-b1-record-authority` +- `task/m04-b2-toolbox-proof` +- `task/m04-d1-world-tests` +- `task/m04-d2-status-tests-docs` + +### task/m04-a1-preflight + +Ownership: + +- parent only + +Primary objective: + +- establish the run, freeze scope, and create the control-plane surfaces + +Scope: + +1. Re-read [PLAN-04.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-04.md), [README.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/README.md), and [05-restart-invalidation-semantics.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/05-restart-invalidation-semantics.md). +2. Freeze the invariant list into the worker prompt packet. +3. Confirm the slice remains backend-only. +4. Confirm the active seams still center on: + - `start_world_session(...)` + - `start_host_orchestrator_runtime(...)` + - `handle_detected_world_drift(...)` + - `restart_world_session(...)` + - `resolve_live_orchestrator_session(...)` + - `build_toolbox_status_report(...)` +5. Initialize: + - `.runs/plan-04/run-state.json` + - `.runs/plan-04/queue.json` + - `.runs/plan-04/session.log` + +Acceptance: + +1. Parent can name the persisted-authority barrier. +2. Parent can name the proof surface and the selected-row non-change contract. +3. `run-state.json` records the initial phase and queue. + +Green-path output: + +- `.runs/plan-04/sentinels/task-m04-a1-preflight.ok` + +Blocked-path output: + +- `.runs/plan-04/blocked.json` + +### Parent validation gate A + +Required before `task/m04-a2-foundation` starts: + +1. No packet contradiction is unresolved. +2. The parent can state why host participant world binding is forbidden in this slice. +3. The parent can state why `toolbox status --json` is the proof surface and `agent status --json` is not. + +### task/m04-a2-foundation + +Ownership: + +- parent only + +Why serialized: + +- [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) owns startup, runtime bootstrap, restart ordering, fail-closed behavior, and alert emission. Splitting first-start context plumbing across workers would be false parallelism. + +Primary files: + +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) +- [crates/shell/src/execution/repl_persistent_session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/repl_persistent_session.rs) only if startup request plumbing requires aligned validation work + +Scope: + +1. Introduce durable startup context before first world start. +2. Persist a pending parent session record in `Allocating` before the first shared-world request. +3. Thread that context through: + - first startup, + - startup drift, + - auto-restart, + - fail-closed restart-required handling. +4. Make `--no-world` the explicit bypass path. +5. Lock the first `AttachOrCreate` request contract so workers can safely code against it. + +Acceptance: + +1. A persisted parent session record exists before any first-start shared-world attach/create request leaves the shell. +2. The first shared-world request uses the persisted `orchestration_session_id`. +3. Pre-live startup drift handling no longer depends on `resolve_active_orchestration_session_id()`. +4. `--no-world` still preserves host-only startup. + +Green-path output: + +- `.runs/plan-04/sentinels/task-m04-a2-foundation.ok` + +Blocked-path output: + +- `.runs/plan-04/blocked.json` if a second authority store, `PLAN-03` semantic drift, or `--no-world` regression is required + +### Parallel window B + +This is the first real worker window. It opens only after `task/m04-a2-foundation` is accepted. + +### task/m04-b1-record-authority + +Ownership: + +- worker-owned +- parent-reviewed + +Worktree: + +- `../substrate-m04-record-authority` +- `codex/feat-thread-world-binding-m04-record-authority` + +Allowed files: + +- [crates/shell/src/execution/agent_runtime/orchestration_session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/orchestration_session.rs) +- [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) + +Scope: + +1. Add explicit parent-session mutators: + - `set_world_binding(...)` + - `clear_world_binding()` +2. Keep parent-session persistence as the binding write and read authority. +3. Keep participant and lease persistence outside the binding barrier. + +Must not do: + +1. No host participant world fields. +2. No lease-sidecar expansion. +3. No slice `06` registry-shape work. + +Acceptance: + +1. The parent can call a single explicit parent-binding mutator path from `async_repl.rs`. +2. The worker output does not broaden state ownership. +3. The worker returns the legal clear points for slice `04`. + +Green-path output: + +- `.runs/plan-04/sentinels/task-m04-b1-record-authority.ok` after parent review + +Blocked-path output: + +- no sentinel +- session-log rejection entry if the patch couples binding truth to participant or lease writes + +### task/m04-b2-toolbox-proof + +Ownership: + +- worker-owned +- parent-reviewed + +Worktree: + +- `../substrate-m04-toolbox-proof` +- `codex/feat-thread-world-binding-m04-toolbox-proof` + +Allowed files: + +- [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs) + +Scope: + +1. Extend `substrate agent toolbox status --json` with optional: + +```json +"active_world_binding": { + "world_id": "wld_active_0002", + "world_generation": 7 +} +``` + +2. Read the proof from `resolve_live_orchestrator_session(...)`. +3. Keep omission non-fatal when live authoritative resolution is unavailable. +4. Preserve the selected host row contract in `agent status --json`. + +Must not do: + +1. No selected-row schema broadening. +2. No new transport semantics. +3. No exposure of `orchestration_session_id` in the proof object. + +Acceptance: + +1. JSON shape is locked for tests before merge. +2. The new proof field is cleanly optional. +3. No selected-row behavior changes are introduced. + +Green-path output: + +- `.runs/plan-04/sentinels/task-m04-b2-toolbox-proof.ok` after parent review + +Blocked-path output: + +- no sentinel +- session-log rejection entry if proof requires public correlation-key exposure or hidden `agent status` changes + +### Parent validation gate B + +Required before `task/m04-c1-runtime-integration` starts: + +1. `task/m04-b1-record-authority` and `task/m04-b2-toolbox-proof` are both parent-reviewed. +2. Neither worker output changes the authority surface. +3. Neither worker output changes the selected-row contract. +4. `run-state.json` records which worker outputs were accepted. + +### task/m04-c1-runtime-integration + +Ownership: + +- parent only + +Primary files: + +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) +- [crates/shell/src/execution/repl_persistent_session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/repl_persistent_session.rs) if needed for request-path alignment +- [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs) only for final proof-field integration + +Scope: + +1. Integrate `task/m04-b1-record-authority`. +2. Add the parent-binding persistence barrier in `async_repl.rs`. +3. Reorder startup runtime events so `registered` and `task_start` emit only after binding persistence succeeds when a world session exists. +4. Reorder drift alerts: + - `world_restarted` only after replacement binding persists + - `world_restart_required` only after current authoritative binding re-persists +5. Make bootstrap failure after world attach close the world, mark the parent terminal, and clear binding only after close succeeds. +6. Stamp host and orchestrator runtime events from the persisted parent snapshot, not transient pre-persist state. +7. Integrate `task/m04-b2-toolbox-proof` only after runtime truth sourcing is stable. + +Acceptance: + +1. No alert or host/orchestrator runtime event can claim a world binding not already persisted in the parent record. +2. Bootstrap failure after world attach cannot leave a live orphaned world tied to a never-live runtime. +3. `resolve_active_orchestration_session_id()` is no longer the deciding source for pre-live startup or restart paths. +4. Selected host status rows still omit world fields. + +Green-path output: + +- `.runs/plan-04/sentinels/task-m04-c1-runtime-integration.ok` + +Blocked-path output: + +- `.runs/plan-04/blocked.json` if runtime truth sourcing remains ambiguous after integration + +### Parent validation gate C + +This gate must pass before any test worktree is launched. + +Checks: + +1. Foundational runtime integration is complete in the parent checkout. +2. No further structural API churn is expected for worker test lanes. +3. `async_repl.rs` is no longer under active structural edit. + +Green-path output: + +- session log entry: `gate-c accepted; launch test worktrees` + +Blocked-path output: + +- no test worktrees launched + +### task/m04-d1-world-tests + +Ownership: + +- worker-owned +- parent-reviewed + +Worktree: + +- `../substrate-m04-world-tests` +- `codex/feat-thread-world-binding-m04-world-tests` + +Allowed files: + +- [crates/shell/tests/repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs) + +Scope: + +1. Prove first-start shared-world attach/create is owner-bound. +2. Prove startup drift before first command retains the persisted startup context. +3. Prove parent binding persistence happens before `world_restarted`. +4. Prove fail-closed drift re-persists the current authoritative binding before `world_restart_required`. +5. Prove bootstrap failure after attach cleans up the world and parent-session state. + +Must not do: + +1. No production-code edits unless the parent explicitly reopens the task as a red/green repair. +2. No contract broadening beyond slice `04`. + +Acceptance: + +1. Integration tests in the real REPL shell test file fail if persist-before-publish regresses. +2. Worker provides exact tests added and commands run. + +Green-path output: + +- `.runs/plan-04/sentinels/task-m04-d1-world-tests.ok` after parent review + +Blocked-path output: + +- no sentinel +- escalation back to parent if runtime behavior, not just tests, must change + +### task/m04-d2-status-tests-docs + +Ownership: + +- worker-owned +- parent-reviewed + +Worktree: + +- `../substrate-m04-status-tests-docs` +- `codex/feat-thread-world-binding-m04-status-tests-docs` + +Allowed files: + +- [crates/shell/tests/agent_successor_contract_ahcsitc0.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_successor_contract_ahcsitc0.rs) +- [llm-last-mile/README.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/README.md) + +Read-for-validation only: + +- [llm-last-mile/05-restart-invalidation-semantics.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/05-restart-invalidation-semantics.md) + +Scope: + +1. Add contract coverage for: + - `toolbox status --json` `active_world_binding` + - proof omission remaining non-fatal + - selected host status rows remaining unchanged +2. Update `llm-last-mile/README.md` so slice `04` is documented as the runtime-state bridge into slice `05` and slice `06`. +3. Flag downstream contract drift against `05-restart-invalidation-semantics.md`. + +Must not do: + +1. No silent rewrite of `05-restart-invalidation-semantics.md`. +2. No selected-status contract rewrite. + +Acceptance: + +1. Contract tests prove the new proof field while preserving selected-row behavior. +2. Docs reflect the bridge role of slice `04` without reordering the packet. + +Green-path output: + +- `.runs/plan-04/sentinels/task-m04-d2-status-tests-docs.ok` after parent review + +Blocked-path output: + +- no sentinel +- escalation if code and downstream packet assumptions no longer agree + +## Integration Procedure + +This is a parent-agent procedure, not general merge advice. + +### Integration order + +1. Complete `task/m04-a1-preflight`. +2. Complete `task/m04-a2-foundation`. +3. Launch at most two worker lanes: + - `task/m04-b1-record-authority` + - `task/m04-b2-toolbox-proof` +4. Review both worker outputs and pass Parent validation gate B. +5. Perform `task/m04-c1-runtime-integration` in the parent checkout. +6. Run parent post-integration validation before launching any test worktrees. +7. Launch test worktrees only after gate C passes: + - `task/m04-d1-world-tests` + - `task/m04-d2-status-tests-docs` +8. Integrate test and docs outputs. +9. Run final validation stack. +10. Complete `task/m04-e1-closeout`. + +### Parent post-merge validation ordering + +After `task/m04-c1-runtime-integration`, run in this order from the parent checkout: + +```bash +cargo test -p substrate-shell start_host_orchestrator_runtime -- --nocapture +cargo test -p substrate-shell repl_world_first_routing_v1 -- --nocapture +cargo test -p substrate-shell agent_successor_contract_ahcsitc0 -- --nocapture +``` + +Only if these pass should the parent launch the Phase D test worktrees. This prevents dispatching workers against unstable runtime integration. + +After integrating `task/m04-d1-world-tests` and `task/m04-d2-status-tests-docs`, run in this order: + +```bash +cargo test -p substrate-shell -- --nocapture +cargo fmt --all -- --check +cargo clippy --workspace --all-targets -- -D warnings +cargo test --workspace -- --nocapture +``` + +Integration refusal rule: + +- If two workstreams produce outputs that require different interpretations of `PLAN-04.md`, neither merges until the parent resolves the contract mismatch explicitly and records it in `.runs/plan-04/session.log`. + +## Context-Control Rules + +1. The parent keeps only the invariant set, merge state, gates, and open blockers in working memory. +2. Worker prompts must include the exact task ID, allowed files, forbidden files, acceptance criteria, and expected sentinel name. +3. Workers return: + - touched files, + - concise rationale, + - tests run, + - blockers or uncertainty. +4. Workers do not paste large file dumps back to the parent. +5. The parent re-reads [PLAN-04.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-04.md) before merging any lane that affects runtime ordering or status surfaces. +6. The parent re-reads [05-restart-invalidation-semantics.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/05-restart-invalidation-semantics.md) before closing the slice to confirm that `PLAN-05` can still consume the parent-session binding model unchanged. +7. `async_repl.rs` is single-writer. Workers may read it for context, but only the parent changes it unless the parent explicitly delegates a bounded repair after main integration. +8. Test workers do not “fix” production behavior by changing assertions to match drift. +9. Parent updates `run-state.json` and `session.log` at every gate transition or blocked termination. + +## Tests And Acceptance + +### Required targeted commands + +These commands map to the real files and contracts in this repo. + +1. Runtime and bootstrap sequencing in [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs): + +```bash +cargo test -p substrate-shell start_host_orchestrator_runtime -- --nocapture +``` + +2. World-start, startup-drift, and fail-closed validation in [repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs): + +```bash +cargo test -p substrate-shell repl_world_first_routing_v1 -- --nocapture +``` + +3. Status and toolbox successor contract validation in [agent_successor_contract_ahcsitc0.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_successor_contract_ahcsitc0.rs): + +```bash +cargo test -p substrate-shell agent_successor_contract_ahcsitc0 -- --nocapture +``` + +4. Full shell crate regression: + +```bash +cargo test -p substrate-shell -- --nocapture +``` + +5. Workspace formatting and lint gates: + +```bash +cargo fmt --all -- --check +cargo clippy --workspace --all-targets -- -D warnings +``` + +6. Final workspace regression: + +```bash +cargo test --workspace -- --nocapture +``` + +### Acceptance matrix + +| Gate | Required proof | Primary files | +| --- | --- | --- | +| Startup authority gate | first attach/create request is owner-bound and preceded by a persisted pending parent session record | `async_repl.rs`, `repl_persistent_session.rs`, `repl_world_first_routing_v1.rs` | +| Binding authority gate | parent session record persists active `world_id/world_generation`; host participant stays world-empty | `orchestration_session.rs`, `state_store.rs`, `async_repl.rs` | +| Publish ordering gate | alerts and host runtime events publish only after parent binding persistence succeeds | `async_repl.rs`, `repl_world_first_routing_v1.rs` | +| Bootstrap cleanup gate | post-attach bootstrap failure closes world, marks parent terminal, clears binding at the right point | `async_repl.rs`, `repl_world_first_routing_v1.rs` | +| Proof surface gate | `toolbox status --json` shows optional `active_world_binding` without changing selected host status rows | `agents_cmd.rs`, `agent_successor_contract_ahcsitc0.rs` | +| Explicit bypass gate | `--no-world` path does not require world-binding persistence and keeps host-only ordering | `async_repl.rs`, targeted runtime tests | +| Contract preservation gate | `agent status --json` selected host row remains unchanged | `agents_cmd.rs`, `agent_successor_contract_ahcsitc0.rs` | + +### Runtime validation rules + +1. The parent does not accept “code looks right” as evidence for ordering-sensitive paths. +2. Any regression involving `world_restarted`, `world_restart_required`, or startup host-runtime events must be backed by test assertions in the real shell test files. +3. If targeted tests pass but full `substrate-shell` fails on related runtime-state surfaces, the slice stays open. +4. If `clippy` or workspace tests fail for reasons caused by this slice, the slice stays open. + +### Final acceptance checklist + +1. Pending parent session is persisted before first shared-world attach/create. +2. First shared-world attach/create is owner-bound. +3. Parent session record is the live world-binding authority. +4. Alerts and host runtime events publish only after binding persistence succeeds. +5. Host participant manifests remain world-empty. +6. `toolbox status --json` gets the optional proof field. +7. `agent status --json` selected host row remains unchanged. +8. Lease-sidecar expansion remains deferred. +9. `--no-world` remains an explicit bypass case. +10. `PLAN-05` prerequisites in [05-restart-invalidation-semantics.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/05-restart-invalidation-semantics.md) still hold after merge. + +## Run Exit Criteria + +This defines DONE for the orchestration run itself, not just for the code slice. + +### Successful run + +The run is complete only when all of these exist or are true: + +1. Updated production code across the intended backend surfaces: + - `async_repl.rs` + - `orchestration_session.rs` + - `state_store.rs` + - `agents_cmd.rs` + - `repl_persistent_session.rs` only if needed by the final implementation +2. Updated tests in: + - `repl_world_first_routing_v1.rs` + - `agent_successor_contract_ahcsitc0.rs` +3. Updated packet documentation in: + - `llm-last-mile/README.md` +4. Parent validation commands have passed in the defined order. +5. `.runs/plan-04/closeout.md` exists and summarizes: + - accepted task IDs, + - integrated worktrees and branches, + - final validation commands and outcomes, + - intentional deferrals to `PLAN-05` or slice `06`. +6. `.runs/plan-04/sentinels/task-m04-e1-closeout.ok` exists. +7. `.runs/plan-04/run-state.json` marks the run `completed`. + +### Blocked termination + +If the run cannot proceed without violating a hard guard: + +1. parent writes `.runs/plan-04/blocked.json`, +2. parent records the blocking task ID, invariant, and unresolved conflict in `.runs/plan-04/session.log`, +3. parent marks `run-state.json` as `blocked`, +4. parent does not write `task-m04-e1-closeout.ok`. + +### task/m04-e1-closeout + +Ownership: + +- parent only + +Scope: + +1. Verify all accepted task sentinels are present. +2. Verify final validation commands passed. +3. Confirm no blocked artifact remains active for the final run state. +4. Write `.runs/plan-04/closeout.md`. +5. Mark `run-state.json` as `completed`. + +Acceptance: + +1. The code slice is complete by the checklist above. +2. The orchestration artifacts clearly show green termination rather than a partial stop. +3. The closeout explains what was deliberately deferred. + +Green-path output: + +- `.runs/plan-04/closeout.md` +- `.runs/plan-04/sentinels/task-m04-e1-closeout.ok` + +Blocked-path output: + +- `.runs/plan-04/blocked.json` + +## Assumptions + +1. `PLAN-03` owner-binding and shared-world generation semantics are already landed and are not being redesigned in this slice. +2. The current implementation branch for this work is `feat/thread-world-binding`, and subagent branches layer on top of it rather than forking a new product direction. +3. This slice remains backend-only. No UI or UX redesign work is required. +4. The live proof surface for this slice is `substrate agent toolbox status --json`; TCP transport remains unsupported and unchanged. +5. [llm-last-mile/README.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/README.md) is updated only to reflect the runtime-state bridge role of slice `04`, not to reorder the packet. +6. [05-restart-invalidation-semantics.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/05-restart-invalidation-semantics.md) is a downstream contract consumer for validation. If it conflicts with the finished slice, the correct action is to stop and reconcile the packet, not to silently stretch slice `04`. diff --git a/llm-last-mile/ORCH_PLAN-05.md b/llm-last-mile/ORCH_PLAN-05.md new file mode 100644 index 000000000..bcf01eba9 --- /dev/null +++ b/llm-last-mile/ORCH_PLAN-05.md @@ -0,0 +1,818 @@ +# ORCH_PLAN-05: Restart Invalidation Semantics for Live State + +Branch: `feat/restart-invalidation-semantics` +Plan source: [PLAN-05.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-05.md) +Execution type: backend-only orchestration plan, no UI scope + +## Summary + +This run executes `PLAN-05` on the current branch `feat/restart-invalidation-semantics` with an exact concurrency cap of `2` worker lanes. The parent is the only integrator and the only final branch writer. The canonical run-state source of truth is `.runs/plan-05/run-state.json`. + +Worktree set for the run: + +- `../substrate-m05-status-suppression` on `codex/feat-restart-invalidation-semantics-m05-status-suppression` +- `../substrate-m05-contract-tests-docs` on `codex/feat-restart-invalidation-semantics-m05-contract-tests-docs` +- `../substrate-m05-restart-tests` on `codex/feat-restart-invalidation-semantics-m05-restart-tests` + +The critical path stays parent-owned through these phases: + +1. `task/m05-a1-preflight` +2. `task/m05-a2-foundation` +3. `task/m05-b2-restart-integration` +4. `task/m05-d1-integrate-and-validate` +5. `task/m05-d2-closeout` + +This is deliberate. The real overlap surfaces are [crates/shell/src/execution/agent_runtime/session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs), [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs), [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs), and [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs). False concurrency across those files would create merge churn and contract drift faster than it would create throughput. + +The parent-owned execution brief is: + +1. freeze the authority and invalidation contract from `PLAN-05`, +2. land the bounded invalidation primitive and session-local sweep without pre-spending `PLAN-06`, +3. integrate restart ordering so stale generation invalidates before replacement publication becomes live, +4. accept a bounded worker change for status suppression only after foundation is frozen, +5. launch tests and docs only after production behavior stabilizes, +6. integrate all worker outputs in one parent-owned validation phase, +7. close the run only if all sentinels are present and the full validation stack is green. + +## Orchestration State Surfaces + +### Canonical run state + +Single local source of truth for the run: + +- `.runs/plan-05/run-state.json` + +Parent-only writes to this file. It tracks: + +- current phase, +- active task IDs, +- branch and worktree assignment, +- accepted worker outputs, +- gate status, +- blocked or completed terminal state, +- final closeout pointer. + +If a worker report conflicts with `run-state.json`, the parent trusts `run-state.json` until it explicitly reconciles the discrepancy. + +### Derived run artifacts + +The parent may maintain these local artifacts: + +- `.runs/plan-05/queue.json` +- `.runs/plan-05/session.log` +- `.runs/plan-05/sentinels/task-m05-a1-preflight.ok` +- `.runs/plan-05/sentinels/task-m05-a2-foundation.ok` +- `.runs/plan-05/sentinels/task-m05-b1-status-suppression.ok` +- `.runs/plan-05/sentinels/task-m05-b2-restart-integration.ok` +- `.runs/plan-05/sentinels/task-m05-c1-contract-tests-docs.ok` +- `.runs/plan-05/sentinels/task-m05-c2-restart-tests.ok` +- `.runs/plan-05/sentinels/task-m05-d1-integrate-and-validate.ok` +- `.runs/plan-05/sentinels/task-m05-d2-closeout.ok` +- `.runs/plan-05/blocked.json` +- `.runs/plan-05/closeout.md` + +Sentinel rules: + +1. `.ok` means the parent validated the task output and advanced the run. +2. Missing sentinel means the task is not accepted. +3. `blocked.json` is written only on blocked termination. +4. `closeout.md` is written only on successful completion. + +## Concurrency Policy + +1. The parent is the only integrator. +2. The parent is the only writer of final branch state on `feat/restart-invalidation-semantics`. +3. Exact worker cap: `2` active worker lanes. +4. The parent owns the critical overlap surfaces first: + - [crates/shell/src/execution/agent_runtime/session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs) + - [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) + - [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) +5. [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs) does not open as a worker lane until the parent has frozen the invalidation helper and state-store contract. +6. No two lanes may edit `async_repl.rs`, `agents_cmd.rs`, or `state_store.rs` concurrently. +7. Test and docs worktrees launch only after production integration stabilizes in the parent checkout. +8. Safe parallel windows: + - Window B: parent restart integration plus one worker on `agents_cmd.rs` + - Window C: two worker lanes on tests and docs after parent production validation passes + +## Hard Guards + +### Locked invariants + +1. `participants/*.json` and `sessions/*.json` are the authoritative runtime store. +2. `handles/*.json` remains compatibility input only; this slice must not restore it as runtime authority. +3. The active generation source is the parent `OrchestrationSessionRecord.world_generation` from `PLAN-04`. +4. `PLAN-05` consumes that generation and must not assign it from any new store. +5. Only `role=member` plus `execution.scope=world` participants are invalidated by generation rollover. +6. Restart ordering must prefer fail-closed absence over stale presence. +7. `Invalidated` rows are tombstones for live surfaces and must suppress trace fallback. +8. Trace fallback suppression identity is `(orchestration_session_id, agent_id, execution.scope)`. +9. Persisted lineage naming stays `resumed_from_participant_id`. +10. `PLAN-05` must not pre-spend `PLAN-06` with a grouped registry rewrite or new active-generation index file. +11. `status` remains a live-state surface, not a historical reconstruction surface. +12. Docs named in `PLAN-05` are part of the slice. + +### File-level boundaries + +Parent-owned critical overlap surfaces: + +- [crates/shell/src/execution/agent_runtime/session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs) +- [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) + +Worker-safe implementation surface after foundation freeze: + +- [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs) + +Validation and docs surfaces: + +- [crates/shell/tests/agent_successor_contract_ahcsitc0.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_successor_contract_ahcsitc0.rs) +- [crates/shell/tests/repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs) +- [docs/TRACE.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/TRACE.md) +- [docs/project_management/packs/draft/agent-hub-core-successor-identity-tuple-compatible/agent-hub-session-protocol-spec.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/project_management/packs/draft/agent-hub-core-successor-identity-tuple-compatible/agent-hub-session-protocol-spec.md) + +Read-for-validation only: + +- [llm-last-mile/PLAN-05.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-05.md) +- [llm-last-mile/ORCH_PLAN-04.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/ORCH_PLAN-04.md) + +### Non-negotiable stop conditions + +Stop the run and write `blocked.json` if any of these occur: + +1. A task requires a new session-grouped registry layout under `run/agent-hub/sessions/...`. +2. A task requires a new active-generation index file to make `PLAN-05` work. +3. A task makes `handles/*.json` or trace authoritative for live state. +4. A task preserves `(agent_id, role)` as the suppression boundary. +5. A task cannot invalidate stale generation before replacement publication without redesigning the slice. +6. A task keeps stale generation visible "for continuity" during replacement failure. +7. A task needs concurrent edits to `async_repl.rs` and `agents_cmd.rs` before the parent freezes the shared contract. +8. A task changes public status JSON shape beyond `PLAN-05` requirements. +9. A task renames persisted lineage fields away from `resumed_from_participant_id`. + +## Approval And Gate Model + +There are no human approval gates defined for this run. + +Replacement control mechanism: + +1. parent validation gates, +2. parent-written sentinels, +3. `session.log` for acceptance and rejection rationale, +4. `blocked.json` for hard-stop termination, +5. `closeout.md` for successful completion. + +### Gate A: Packet freeze + +Required before implementation starts: + +- parent re-reads `PLAN-05` and `ORCH_PLAN-04`, +- parent records the invariant list, +- parent records the exact files that belong to the slice. + +### Gate B: Foundation freeze + +Required before any worker edits `agents_cmd.rs`: + +- invalidation helper contract is fixed, +- session-local sweep contract is fixed, +- parent has proven no `PLAN-06` storage work is needed, +- worker prompts contain exact allowed files and stop conditions. + +### Gate C: Production stabilization + +Required before test/docs worktrees launch: + +- parent production integration is complete, +- no structural churn remains in `session.rs`, `state_store.rs`, `async_repl.rs`, or `agents_cmd.rs`, +- parent targeted validation passes in the defined order. + +### Gate D: Final acceptance + +Required before closeout: + +- targeted tests pass, +- shell crate regression passes, +- formatting and clippy gates pass, +- workspace regression passes, +- docs reflect the final contract. + +## Workstream Plan + +### Worktree topology + +Common naming pattern: + +- relative worktree path: `../substrate-m05-` +- worker branch name: `codex/feat-restart-invalidation-semantics-m05-` + +Concrete worktrees for this run: + +- `../substrate-m05-status-suppression` on `codex/feat-restart-invalidation-semantics-m05-status-suppression` +- `../substrate-m05-contract-tests-docs` on `codex/feat-restart-invalidation-semantics-m05-contract-tests-docs` +- `../substrate-m05-restart-tests` on `codex/feat-restart-invalidation-semantics-m05-restart-tests` + +Parent checkout: + +- current checkout on `feat/restart-invalidation-semantics` + +Subagents do not merge each other’s work. They return patches, touched files, tests run, and blockers to the parent. + +### Task index + +Parent-only serialized tasks: + +- `task/m05-a1-preflight` +- `task/m05-a2-foundation` +- `task/m05-b2-restart-integration` +- `task/m05-d1-integrate-and-validate` +- `task/m05-d2-closeout` + +Worker-owned tasks: + +- `task/m05-b1-status-suppression` +- `task/m05-c1-contract-tests-docs` +- `task/m05-c2-restart-tests` + +### task/m05-a1-preflight + +Ownership: + +- parent only + +Scope: + +1. Re-read [PLAN-05.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-05.md) and [ORCH_PLAN-04.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/ORCH_PLAN-04.md). +2. Freeze the authoritative-store rules, restart-ordering rule, suppression identity, and `PLAN-06` deferrals into the worker prompt packet. +3. Initialize: + - `.runs/plan-05/run-state.json` + - `.runs/plan-05/queue.json` + - `.runs/plan-05/session.log` +4. Confirm the slice remains backend-only. +5. Confirm the primary seam list: + - invalidation helper in `session.rs` + - session-local sweep in `state_store.rs` + - restart ordering in `async_repl.rs` + - status suppression in `agents_cmd.rs` + - tests and docs named in `PLAN-05` + +Acceptance: + +1. Parent can state the active-generation authority source. +2. Parent can state the exact suppression tuple. +3. Parent can state why grouped-registry rewrite is forbidden in this slice. +4. `run-state.json` records the initial phase and queue. + +Green-path output: + +- `.runs/plan-05/sentinels/task-m05-a1-preflight.ok` + +Blocked-path output: + +- `.runs/plan-05/blocked.json` + +### Parent validation gate A + +Required before `task/m05-a2-foundation` starts: + +1. No invariant contradiction remains unresolved. +2. The parent can explain why fail-closed absence is safer than stale presence. +3. The parent can explain why `PLAN-04` remains the only active-generation authority. + +### task/m05-a2-foundation + +Ownership: + +- parent only + +Why serialized: + +- `session.rs`, `state_store.rs`, and `async_repl.rs` share the invalidation contract. Freezing the helper and sweep semantics in the parent avoids false concurrency and merge churn. + +Allowed files: + +- [crates/shell/src/execution/agent_runtime/session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs) +- [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) + +Scope: + +1. Add one explicit participant invalidation helper using existing `Invalidated`. +2. Add one bounded session-local invalidation sweep keyed by: + - `orchestration_session_id` + - `role=member` + - `execution.scope=world` + - `world_generation < active_generation` + - currently live state only +3. Keep the sweep idempotent. +4. Keep host-scoped and already-invalidated rows untouched. +5. Freeze any state-store read helpers needed later by `agents_cmd.rs` and `async_repl.rs`. +6. Add or update unit coverage in the touched source files for: + - invalidation metadata stamping, + - session-local scope, + - host-row exclusion, + - idempotence. + +Must not do: + +1. No new state enum variant. +2. No new index file. +3. No directory-layout rewrite. +4. No changes to `list_live_manifests()` semantics beyond consuming existing authoritative-live behavior. + +Acceptance: + +1. A single invalidation helper exists for generation rollover semantics. +2. A single session-local sweep exists and returns invalidated participant IDs for logging and assertions. +3. Re-running the sweep is harmless. +4. The parent can now hand workers a frozen contract for status and restart integration. + +Green-path output: + +- `.runs/plan-05/sentinels/task-m05-a2-foundation.ok` + +Blocked-path output: + +- `.runs/plan-05/blocked.json` + +### Parent validation gate B + +Required before `task/m05-b1-status-suppression` opens: + +1. Foundation code is integrated in the parent checkout. +2. Helper signatures are stable enough for worker consumption. +3. No `PLAN-06` work was introduced. +4. `run-state.json` records the frozen foundation contract. + +### Parallel window B + +This is the first real concurrency window. It opens only after `task/m05-a2-foundation` is accepted. + +### task/m05-b1-status-suppression + +Ownership: + +- worker-owned +- parent-reviewed + +Worktree: + +- `../substrate-m05-status-suppression` +- `codex/feat-restart-invalidation-semantics-m05-status-suppression` + +Allowed files: + +- [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs) + +Scope: + +1. Replace the trace-fallback suppression key with `(orchestration_session_id, agent_id, execution.scope)`. +2. Build suppression identities from authoritative live rows first. +3. Build suppression identities from invalidated world-member tombstones on disk second. +4. Suppress trace fallback on either match. +5. Keep host-orchestrator selected-row behavior unchanged. +6. Keep public status JSON shape unchanged. + +Must not do: + +1. No edits to `state_store.rs` or `async_repl.rs`. +2. No public shape redesign for `agent status --json`. +3. No fallback to `(agent_id, role)`. +4. If the worker discovers it needs new helper signatures, broader state-store API changes, or additional authority-surface changes outside `agents_cmd.rs`, it must stop immediately, report the required delta, and bounce the work back to the parent instead of expanding scope. + +Acceptance: + +1. Worker output stays inside `agents_cmd.rs`. +2. Suppression semantics are session-aware and world-scope-aware. +3. The selected host row contract remains unchanged. +4. Any discovered dependency on new helper or API shape is escalated back to the parent rather than worked around in-lane. + +Green-path output: + +- `.runs/plan-05/sentinels/task-m05-b1-status-suppression.ok` after parent review + +Blocked-path output: + +- no sentinel +- rejection entry in `.runs/plan-05/session.log` if the patch requires broader authority changes + +### task/m05-b2-restart-integration + +Ownership: + +- parent only + +Allowed files: + +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) +- [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) only for bounded call-site glue already implied by `task/m05-a2-foundation` + +Scope: + +1. Keep the `PLAN-04` parent-binding barrier. +2. After generation `G+1` is accepted and persisted on the parent, invalidate stale generation `G` members before replacement publication is considered complete. +3. Allow temporary absence. +4. Forbid temporary stale-live visibility. +5. Emit restart success only after: + - parent binding persists, + - stale member generation is invalidated, + - replacement publication succeeds, or the path intentionally remains fail closed. +6. Keep trace non-authoritative for live state. + +Must not do: + +1. No replacement success publication before stale generation is dead. +2. No "continuity" path that leaves stale members live while replacement is missing. +3. No new authority store. + +Acceptance: + +1. Crash or injected failure after invalidation but before replacement leaves absence, not stale liveness. +2. Dual-live generations are impossible on the happy path. +3. Parent-owned restart sequencing is now stable enough for test worktrees. + +Green-path output: + +- `.runs/plan-05/sentinels/task-m05-b2-restart-integration.ok` + +Blocked-path output: + +- `.runs/plan-05/blocked.json` + +### Parent validation gate C + +Required before any test/docs worktree launches: + +1. `task/m05-b1-status-suppression` is accepted. +2. `task/m05-b2-restart-integration` is complete in the parent checkout. +3. No further structural API churn is expected in production files. +4. Parent targeted validation passes in the defined order. + +### Parallel window C + +This window opens only after gate C passes. + +### task/m05-c1-contract-tests-docs + +Ownership: + +- worker-owned +- parent-reviewed + +Worktree: + +- `../substrate-m05-contract-tests-docs` +- `codex/feat-restart-invalidation-semantics-m05-contract-tests-docs` + +Allowed files: + +- [crates/shell/tests/agent_successor_contract_ahcsitc0.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_successor_contract_ahcsitc0.rs) +- [docs/TRACE.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/TRACE.md) +- [docs/project_management/packs/draft/agent-hub-core-successor-identity-tuple-compatible/agent-hub-session-protocol-spec.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/project_management/packs/draft/agent-hub-core-successor-identity-tuple-compatible/agent-hub-session-protocol-spec.md) + +Read-for-validation only: + +- [llm-last-mile/PLAN-05.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-05.md) + +Scope: + +1. Add contract coverage for: + - tombstone suppression beating stale trace fallback, + - no-replacement-yet omission from live status, + - same-agent concurrent sessions remaining independently visible, + - `resumed_from_participant_id` persisted lineage assertions. +2. Update docs so they say: + - `participants/*.json` is the authority path, + - tombstones beat trace for live-state selection, + - persisted lineage uses `resumed_from_participant_id`. +3. Keep compatibility-path references bounded and clearly secondary. + +Must not do: + +1. No silent rewrite of `PLAN-05`. +2. No new public status fields. +3. No shift back to `handles/*.json`-first fixtures. + +Acceptance: + +1. Contract tests cover the suppression tuple and tombstone rules. +2. Docs match the final runtime contract. +3. Worker stays within the allowed files. +4. New fixtures remain `participants/*.json`-first. +5. Any remaining `handles/*.json` references are compatibility-only and are not used as the primary authority path in new assertions. + +Green-path output: + +- `.runs/plan-05/sentinels/task-m05-c1-contract-tests-docs.ok` after parent review + +Blocked-path output: + +- no sentinel +- escalation if docs and finished runtime behavior disagree + +### task/m05-c2-restart-tests + +Ownership: + +- worker-owned +- parent-reviewed + +Worktree: + +- `../substrate-m05-restart-tests` +- `codex/feat-restart-invalidation-semantics-m05-restart-tests` + +Allowed files: + +- [crates/shell/tests/repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs) + +Scope: + +1. Add integration coverage proving stale member generation invalidates before restart success publication. +2. Add fail-closed coverage proving injected failure between invalidation and replacement leaves the member absent. +3. Add replacement coverage proving the new generation becomes the only live generation after full restart. +4. Keep assertions tied to the real runtime ordering, not mocked abstractions. + +Must not do: + +1. No production-code edits unless the parent explicitly reopens the task as a repair lane. +2. No contract broadening beyond `PLAN-05`. + +Acceptance: + +1. Integration tests fail if publish-before-invalidate regresses. +2. Integration tests fail if a missing replacement still leaves stale liveness visible. +3. Worker provides exact tests added and exact commands run. + +Green-path output: + +- `.runs/plan-05/sentinels/task-m05-c2-restart-tests.ok` after parent review + +Blocked-path output: + +- no sentinel +- escalation back to parent if production behavior, not only tests, must change + +### task/m05-d1-integrate-and-validate + +Ownership: + +- parent only + +Scope: + +1. Merge accepted foundation, production, status, test, and docs outputs into the parent checkout in this order: + - parent-owned `task/m05-a2-foundation` + - accepted worker output from `task/m05-b1-status-suppression` + - parent-owned `task/m05-b2-restart-integration` + - accepted worker output from `task/m05-c1-contract-tests-docs` + - accepted worker output from `task/m05-c2-restart-tests` +2. Reconcile any patch overlap in the parent checkout without delegating merge resolution back to workers. +3. Run the exact validation stack in the required order. +4. Record accepted worker branches, rejected worker branches, and any manual parent reconciliation in `.runs/plan-05/session.log`. +5. Refuse integration if competing outputs imply different interpretations of `PLAN-05`. + +Merge refusal rules: + +1. If a worker output implies `handles/*.json` is authoritative, do not merge it. +2. If a worker output requires new helper signatures or broader state-store changes not frozen at gate B, do not merge it; reopen parent planning instead. +3. If worker outputs disagree about the suppression tuple, do not merge either interpretation until the parent resolves the contract explicitly. +4. If test assertions only pass by weakening fail-closed behavior or tolerating stale liveness, do not merge them. +5. If docs describe a different authority path or lineage field than the code now enforces, stop and reconcile before merging. + +Exact validation command ordering: + +```bash +cargo test -p substrate-shell agent_runtime::state_store -- --nocapture +cargo test -p substrate-shell agent_successor_contract_ahcsitc0 -- --nocapture +cargo test -p substrate-shell repl_world_first_routing_v1 -- --nocapture +cargo test -p substrate-shell -- --nocapture +cargo fmt --all -- --check +cargo clippy --workspace --all-targets -- -D warnings +cargo test --workspace -- --nocapture +``` + +Acceptance: + +1. All accepted worker outputs are integrated or explicitly rejected by parent decision. +2. The validation stack passes in the exact order above. +3. The parent can state one coherent `PLAN-05` interpretation across runtime, status, tests, and docs. +4. No active `blocked.json` remains. + +Green-path output: + +- `.runs/plan-05/sentinels/task-m05-d1-integrate-and-validate.ok` + +Blocked-path output: + +- `.runs/plan-05/blocked.json` +- rejection details appended to `.runs/plan-05/session.log` + +### task/m05-d2-closeout + +Ownership: + +- parent only + +Scope: + +1. Verify all accepted task sentinels are present. +2. Verify `task/m05-d1-integrate-and-validate` completed and its sentinel exists. +3. Confirm no blocked artifact remains active. +4. Write `.runs/plan-05/closeout.md`. +5. Mark `run-state.json` as `completed`. + +Acceptance: + +1. The slice is complete by the final acceptance checklist. +2. The orchestration artifacts show green termination rather than partial progress. +3. The closeout names any intentional deferrals to `PLAN-06`. + +Green-path output: + +- `.runs/plan-05/closeout.md` +- `.runs/plan-05/sentinels/task-m05-d2-closeout.ok` + +Blocked-path output: + +- `.runs/plan-05/blocked.json` + +## Integration Procedure + +### Integration order + +1. Complete `task/m05-a1-preflight`. +2. Complete `task/m05-a2-foundation`. +3. Launch `task/m05-b1-status-suppression`. +4. In parallel, complete `task/m05-b2-restart-integration` in the parent checkout. +5. Review and accept or reject `task/m05-b1-status-suppression`. +6. Pass gate C. +7. Launch in parallel: + - `task/m05-c1-contract-tests-docs` + - `task/m05-c2-restart-tests` +8. Review and accept or reject both worker outputs. +9. Complete `task/m05-d1-integrate-and-validate`. +10. Complete `task/m05-d2-closeout`. + +### Parent post-integration validation ordering + +The parent does not treat procedural integration as complete until `task/m05-d1-integrate-and-validate` succeeds. + +Validation order inside that task is fixed: + +```bash +cargo test -p substrate-shell agent_runtime::state_store -- --nocapture +cargo test -p substrate-shell agent_successor_contract_ahcsitc0 -- --nocapture +cargo test -p substrate-shell repl_world_first_routing_v1 -- --nocapture +cargo test -p substrate-shell -- --nocapture +cargo fmt --all -- --check +cargo clippy --workspace --all-targets -- -D warnings +cargo test --workspace -- --nocapture +``` + +Integration refusal rule: + +- If two outputs require different interpretations of `PLAN-05`, neither merges until the parent resolves the contract mismatch and records the decision in `.runs/plan-05/session.log`. + +## Context-Control Rules + +1. The parent keeps only invariants, open tasks, gates, and blockers in working memory. +2. Every worker prompt must include: + - exact task ID, + - allowed files, + - forbidden files, + - acceptance criteria, + - stop conditions, + - expected sentinel name. +3. Workers return: + - touched files, + - concise rationale, + - tests run, + - blockers or uncertainty. +4. Workers do not paste large file dumps back to the parent. +5. The parent re-reads [PLAN-05.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-05.md) before merging any lane that affects restart ordering or status suppression. +6. `async_repl.rs`, `state_store.rs`, and `session.rs` are single-writer parent-owned during production integration. +7. `agents_cmd.rs` is single-writer worker-owned only after gate B; the parent does not concurrently reshape its contract. +8. Test workers do not relax assertions to match drift. +9. Parent updates `run-state.json` and `session.log` at every gate transition, task acceptance, rejection, integration decision, or blocked termination. + +## Tests And Acceptance + +### Required targeted commands + +These commands map directly to the files and behaviors named in `PLAN-05`. + +1. Foundation sweep and runtime-state unit coverage in [state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs): + +```bash +cargo test -p substrate-shell agent_runtime::state_store -- --nocapture +``` + +2. Status and successor contract validation in [agent_successor_contract_ahcsitc0.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_successor_contract_ahcsitc0.rs): + +```bash +cargo test -p substrate-shell agent_successor_contract_ahcsitc0 -- --nocapture +``` + +3. Restart ordering validation in [repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs): + +```bash +cargo test -p substrate-shell repl_world_first_routing_v1 -- --nocapture +``` + +4. Full shell crate regression: + +```bash +cargo test -p substrate-shell -- --nocapture +``` + +5. Repo gates: + +```bash +cargo fmt --all -- --check +cargo clippy --workspace --all-targets -- -D warnings +``` + +6. Final workspace regression: + +```bash +cargo test --workspace -- --nocapture +``` + +### Acceptance matrix + +| Gate | Required proof | Primary files | +| --- | --- | --- | +| Authority gate | `participants/*.json` and `sessions/*.json` remain authoritative; no `PLAN-06` storage work leaks in | `session.rs`, `state_store.rs` | +| Invalidation gate | stale world members in the same orchestration session become `Invalidated` and stay tombstoned | `session.rs`, `state_store.rs` | +| Suppression gate | trace fallback suppresses by `(orchestration_session_id, agent_id, execution.scope)` using live rows and tombstones | `agents_cmd.rs`, `agent_successor_contract_ahcsitc0.rs` | +| Isolation gate | same `agent_id` across concurrent sessions does not cross-suppress | `agents_cmd.rs`, `agent_successor_contract_ahcsitc0.rs` | +| Restart ordering gate | parent binding persists, stale generation invalidates, then replacement publication becomes visible | `async_repl.rs`, `repl_world_first_routing_v1.rs` | +| Fail-closed gate | missing replacement yields absence, not stale liveness | `async_repl.rs`, `repl_world_first_routing_v1.rs` | +| Lineage gate | persisted lineage is asserted as `resumed_from_participant_id` | `agent_successor_contract_ahcsitc0.rs`, successor protocol doc | +| Docs gate | trace and protocol docs describe tombstone-beats-trace and authority-path terminology | `docs/TRACE.md`, successor protocol doc | + +### Final acceptance checklist + +1. `PLAN-04` parent session `world_generation` remains the only active-generation source. +2. Every older world-scoped member in the same orchestration session becomes `Invalidated` after rollover. +3. Live reads no longer surface stale-generation world members after invalidation commits. +4. `agent status --json` suppresses stale trace fallback when a matching tombstone exists. +5. Same-agent concurrent sessions remain independently visible. +6. Restart success cannot surface before stale generation is dead. +7. Replacement failure leaves absence, not stale presence. +8. `resumed_from_participant_id` remains the persisted lineage field name. +9. `PLAN-06` storage work remains deferred. +10. The docs named in `PLAN-05` match the final code contract. + +## Run Exit Criteria + +### Successful run + +The run is complete only when all of these are true: + +1. Updated production code exists only across the intended slice surfaces: + - `session.rs` + - `state_store.rs` + - `agents_cmd.rs` + - `async_repl.rs` +2. Updated tests exist in: + - `agent_successor_contract_ahcsitc0.rs` + - `repl_world_first_routing_v1.rs` + - source-adjacent unit tests where foundation code was changed +3. Updated docs exist in: + - `docs/TRACE.md` + - successor protocol doc named in `PLAN-05` +4. All worker sentinels are present: + - `task-m05-b1-status-suppression.ok` + - `task-m05-c1-contract-tests-docs.ok` + - `task-m05-c2-restart-tests.ok` +5. The parent integration sentinel is present: + - `task-m05-d1-integrate-and-validate.ok` +6. The closeout sentinel is present: + - `task-m05-d2-closeout.ok` +7. The full validation stack is green in the required order. +8. No active `.runs/plan-05/blocked.json` remains. +9. `.runs/plan-05/closeout.md` exists and summarizes: + - accepted task IDs, + - integrated worktrees and branches, + - final validation commands and outcomes, + - intentional deferrals to `PLAN-06`. +10. `.runs/plan-05/run-state.json` marks the run `completed`. + +### Blocked termination + +If the run cannot proceed without violating a hard guard: + +1. parent writes `.runs/plan-05/blocked.json`, +2. parent records the blocking task ID, invariant, and unresolved conflict in `.runs/plan-05/session.log`, +3. parent marks `run-state.json` as `blocked`, +4. parent does not write `task-m05-d1-integrate-and-validate.ok`, +5. parent does not write `task-m05-d2-closeout.ok`. + +## Assumptions + +1. `PLAN-04` is already landed on this branch or otherwise available as the runtime authority model consumed by this slice. +2. The implementation branch for this work is `feat/restart-invalidation-semantics`, and worker branches layer on top of it rather than redefining scope. +3. This slice remains backend-only. No UI or transport redesign work is required. +4. `participants/*.json` and `sessions/*.json` are stable enough to carry the invalidation contract without a storage migration. +5. The docs named in `PLAN-05` are the only required documentation updates for this slice unless implementation reveals a direct contradiction. +6. If finished code conflicts with `PLAN-05` on any locked invariant, the correct action is to stop and reconcile the packet rather than stretch the slice into `PLAN-06`. diff --git a/llm-last-mile/ORCH_PLAN-06.md b/llm-last-mile/ORCH_PLAN-06.md new file mode 100644 index 000000000..c07b6ee23 --- /dev/null +++ b/llm-last-mile/ORCH_PLAN-06.md @@ -0,0 +1,737 @@ +# ORCH_PLAN-06: Session-Centric Runtime State Store + +Branch: `feat/session-centric-state-store` +Plan source: [PLAN-06.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-06.md) +Execution type: backend and operator-CLI orchestration plan, no UI scope + +## Summary + +This run executes `PLAN-06` on the current branch `feat/session-centric-state-store` with an exact worker cap of `2`. The parent is the only integrator, the only final branch writer, and the only agent allowed to mutate the overlapping production seams in [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs), [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs), and [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs). The canonical run-state source of truth is `.runs/plan-06/run-state.json`. +The worker cap is exactly `2` because the only honest parallelism in this slice is the late tests/docs window after Gate C, and there are intentionally zero production-code workers before that because the parent owns every overlapping production seam. + +Worktree set for the run: + +- `../substrate-m06-contract-tests-docs` on `codex/feat-session-centric-state-store-m06-contract-tests-docs` +- `../substrate-m06-persistence-routing-tests` on `codex/feat-session-centric-state-store-m06-persistence-routing-tests` + +The critical path stays parent-owned through these phases: + +1. `task/m06-a1-preflight` +2. `task/m06-a2-projection-foundation` +3. `task/m06-b1-consumer-invalidation-cutover` +4. `task/m06-c1-canonical-writer-cutover` +5. `task/m06-e1-integrate-and-validate` +6. `task/m06-e2-closeout` + +This is deliberate. `PLAN-06`'s sequencing truth is correct, but the repo seams mean Phase A, Phase B, and Phase C are not real parallel lanes here. The session-record API, tombstone walker, toolbox/session resolution, and canonical writer helpers all converge in `state_store.rs`, and the writer cutover shares a choke point with `persist_runtime_snapshots()` in `async_repl.rs`. Opening workers on those files before the parent freezes each contract would create merge churn and contract drift faster than it would create throughput. + +The parent-owned execution brief is: + +1. freeze `PLAN-05` semantics as a non-negotiable acceptance gate, +2. land the store-owned session projection and hardened source walkers first, +3. cut `agent status`, `toolbox status`, `toolbox env`, tombstone reads, and invalidation over to the new store contract second, +4. move canonical session-root writes under the existing lifecycle ordering third, +5. launch worker lanes only after production code stabilizes in the parent checkout, +6. integrate worker outputs in one parent-owned validation pass, +7. close the run only if the shell-targeted validation stack and final repo gates are green. + +## Hard Guards + +### Locked invariants + +1. `PLAN-05` semantics are a hard guard for `PLAN-06`, not a best-effort carry-forward. +2. The suppression identity remains `(orchestration_session_id, agent_id, execution.scope)`. +3. `OrchestrationSessionRecord.world_generation` remains the only active-generation authority. +4. `handles/*.json` remains compatibility input only until the parent explicitly removes those reads last. +5. Session grouping becomes store-owned; caller-owned regrouping in `agents_cmd.rs` is forbidden once the cutover lands. +6. `substrate agent status` must preserve one row per live participant, not collapse back to one row per `agent_id`. +7. `substrate agent toolbox status|env` must resolve exactly one live session record and fail closed on ambiguity or incomplete live state. +8. `persist_runtime_snapshots()` remains the write choke point. This slice does not authorize lifecycle redesign or a new transactional layer. +9. Any temporary dual-write is allowed only inside the store-owned writer path, not in callers or tests. +10. Canonical session-root traversal must harden the local trust boundary: no symlink-following, no non-regular file promotion, no implicit nested traversal. +11. `docs/USAGE.md` currently contains stale wording that still claims live discovery is backed by `handles/`; correcting that wording is required for slice completion. +12. No new selector UX such as `--orchestration-session-id` is allowed in this slice. +13. No new cache file, index file, or second authority store is allowed. +14. Crash-window tolerance is part of the contract. Parent-only roots, participant-only roots, and stale active-handle references must degrade safely instead of failing whole commands. + +### File-level boundaries + +Parent-owned critical overlap surfaces: + +- [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) +- [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs) +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) + +Worker-safe late-phase validation and docs surfaces: + +- [crates/shell/tests/agent_successor_contract_ahcsitc0.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_successor_contract_ahcsitc0.rs) +- [crates/shell/tests/agent_hub_trace_persistence.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_hub_trace_persistence.rs) +- [crates/shell/tests/repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs) +- [docs/TRACE.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/TRACE.md) +- [docs/USAGE.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/USAGE.md) + +Read-for-truth only: + +- [llm-last-mile/PLAN-06.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-06.md) +- [llm-last-mile/ORCH_PLAN-05.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/ORCH_PLAN-05.md) +- [llm-last-mile/ORCH_PLAN-04.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/ORCH_PLAN-04.md) +- [.runs/plan-05/closeout.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/plan-05/closeout.md) + +### Non-negotiable stop conditions + +Stop the run and write `.runs/plan-06/blocked.json` if any of these occur: + +1. A task requires new selector UX or any public CLI expansion beyond `PLAN-06`. +2. A task requires a new index file, cache file, or second authority store to make session-centric reads work. +3. A task restores `handles/*.json` or trace rows as live-state authority. +4. A task cannot preserve the `PLAN-05` suppression tuple or invalidation semantics after regrouping. +5. A task requires concurrent edits to `state_store.rs`, `agents_cmd.rs`, or `async_repl.rs`. +6. A task requires moving write ownership outside `persist_runtime_snapshots()` or inventing a transactional layer. +7. A task leaves `docs/USAGE.md` claiming `handles/` is the authority for live discovery. +8. A task turns normal torn-root states into hard command failures on `status` or `toolbox`. +9. A worker lane needs to touch any parent-owned production file to complete its assignment. + +## Orchestration State Surfaces + +### Canonical run state + +Single local source of truth for the run: + +- `.runs/plan-06/run-state.json` + +Parent-only writes to this file. It tracks: + +- current phase, +- active task IDs, +- branch and worktree assignment, +- gate status, +- accepted and rejected worker outputs, +- blocked or completed terminal state, +- final closeout pointer, +- frozen `PLAN-05` guard contract, +- frozen projection/read contract, +- frozen writer-cutover contract. + +If a worker report conflicts with `run-state.json`, the parent trusts `run-state.json` until it explicitly reconciles the discrepancy. + +### Derived run artifacts + +The parent may maintain these local artifacts: + +- `.runs/plan-06/queue.json` +- `.runs/plan-06/session.log` +- `.runs/plan-06/sentinels/task-m06-a1-preflight.ok` +- `.runs/plan-06/sentinels/task-m06-a2-projection-foundation.ok` +- `.runs/plan-06/sentinels/task-m06-b1-consumer-invalidation-cutover.ok` +- `.runs/plan-06/sentinels/task-m06-c1-canonical-writer-cutover.ok` +- `.runs/plan-06/sentinels/task-m06-d1-contract-tests-docs.ok` +- `.runs/plan-06/sentinels/task-m06-d2-persistence-routing-tests.ok` +- `.runs/plan-06/sentinels/task-m06-e1-integrate-and-validate.ok` +- `.runs/plan-06/sentinels/task-m06-e2-closeout.ok` +- `.runs/plan-06/blocked.json` +- `.runs/plan-06/closeout.md` + +Sentinel rules: + +1. `.ok` means the parent validated the task output and advanced the run. +2. Missing sentinel means the task is not accepted. +3. `blocked.json` is written only on blocked termination. +4. `closeout.md` is written only on successful completion. +5. Worker-generated notes never replace parent-written sentinels. + +## Concurrency Policy + +1. The parent is the only integrator. +2. The parent is the only writer of final branch state on `feat/session-centric-state-store`. +3. Exact worker cap: `2` active worker lanes. +4. There are no worker lanes during the Phase A, Phase B, or Phase C production cutovers. +5. No two lanes may edit `state_store.rs`, `agents_cmd.rs`, or `async_repl.rs` concurrently. +6. Worker lanes open only after the parent has stabilized the production diff and seeded each worktree from that exact state. +7. `task/m06-d1-contract-tests-docs` and `task/m06-d2-persistence-routing-tests` are the only parallel window in this run. +8. If worker tasks reveal missing support changes in `state_store.rs`, `agents_cmd.rs`, `async_repl.rs`, or any shared helper outside their allowed files, the worker stops and hands the change back to the parent instead of widening scope. + +## Approval And Gate Model + +There are no human approval gates defined for this run. + +Replacement control mechanism: + +1. parent validation gates, +2. parent-written sentinels, +3. `session.log` for acceptance and rejection rationale, +4. `blocked.json` for hard-stop termination, +5. `closeout.md` for successful completion. + +### Gate A: Packet freeze + +Required before implementation starts: + +- parent re-reads `PLAN-06`, `ORCH_PLAN-05`, `ORCH_PLAN-04`, and `PLAN-05` closeout, +- parent records the exact sequencing truth `A -> B -> C -> D`, +- parent records the authoritative overlap surfaces and worker-safe late-phase files, +- parent records the package-name normalization: use `shell`, never `substrate-shell`. + +### Gate B: Projection freeze + +Required before consumer cutover starts: + +- the `state_store.rs` session-record API is frozen in the parent checkout, +- object-level precedence and torn-root warning behavior are frozen, +- path-hardening behavior is frozen, +- the parent has validated the projection unit stack before touching consumers. + +### Gate C: Production stabilization + +Required before worker lanes launch: + +- Phase B and Phase C production integration are complete in the parent checkout, +- no structural churn remains in `state_store.rs`, `agents_cmd.rs`, or `async_repl.rs`, +- the parent has seeded worker worktrees from the stabilized production state, +- the parent has completed the pre-worker compile/proof checks defined below. + +### Gate D: Final acceptance + +Required before closeout: + +- both worker outputs are accepted or deliberately rejected and replaced by parent work, +- the exact validation command order passes, +- docs reflect the final authority boundary, +- `PLAN-05` invalidation and tombstone semantics still hold under the canonical session-root layout. + +## Workstream Plan + +### Worktree topology + +Parent checkout: + +- current checkout on `feat/session-centric-state-store` + +Child worktrees and branches: + +- `../substrate-m06-contract-tests-docs` + - `codex/feat-session-centric-state-store-m06-contract-tests-docs` +- `../substrate-m06-persistence-routing-tests` + - `codex/feat-session-centric-state-store-m06-persistence-routing-tests` + +Subagents do not merge each other’s work. They return patches, touched files, tests run, and blockers to the parent. + +### Task graph + +Execution graph for the run: + +1. `task/m06-a1-preflight` +2. `task/m06-a2-projection-foundation` +3. `task/m06-b1-consumer-invalidation-cutover` +4. `task/m06-c1-canonical-writer-cutover` +5. `task/m06-d1-contract-tests-docs` and `task/m06-d2-persistence-routing-tests` in parallel +6. `task/m06-e1-integrate-and-validate` +7. `task/m06-e2-closeout` + +Parent-only serialized tasks: + +- `task/m06-a1-preflight` +- `task/m06-a2-projection-foundation` +- `task/m06-b1-consumer-invalidation-cutover` +- `task/m06-c1-canonical-writer-cutover` +- `task/m06-e1-integrate-and-validate` +- `task/m06-e2-closeout` + +Worker-owned tasks: + +- `task/m06-d1-contract-tests-docs` +- `task/m06-d2-persistence-routing-tests` + +## Parallel Window D + +This is the only worker window in the run. + +It opens only after Gate C passes and after the parent has finished the full production-code path through `task/m06-c1-canonical-writer-cutover`. There are zero production-code workers before Gate C because the parent owns the overlapping seams in `state_store.rs`, `agents_cmd.rs`, and `async_repl.rs`, and splitting those files earlier would create merge churn rather than throughput. + +### task/m06-a1-preflight + +Ownership: + +- parent only + +Scope: + +1. Re-read [PLAN-06.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-06.md), [ORCH_PLAN-05.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/ORCH_PLAN-05.md), [ORCH_PLAN-04.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/ORCH_PLAN-04.md), and [.runs/plan-05/closeout.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/plan-05/closeout.md). +2. Confirm the run executes from `feat/session-centric-state-store`. +3. Freeze the `PLAN-05` invariants, the `PLAN-06` phase order, the parent-only seam list, and the late worker-safe seam list into the run packet. +4. Initialize: + - `.runs/plan-06/run-state.json` + - `.runs/plan-06/queue.json` + - `.runs/plan-06/session.log` +5. Record the repo-truth validation rule that all package-targeted commands use `-p shell`. + +Acceptance: + +1. The parent can explain why Phase A, Phase B, and Phase C stay serialized in this repo. +2. The parent can state the `PLAN-05` suppression tuple and active-generation authority without ambiguity. +3. The parent can name the stale `docs/USAGE.md` authority wording that must be removed. +4. `run-state.json` records the initial phase and queue. + +Green-path output: + +- `.runs/plan-06/sentinels/task-m06-a1-preflight.ok` + +Blocked-path output: + +- `.runs/plan-06/blocked.json` + +### Parent validation gate A + +Required before `task/m06-a2-projection-foundation` starts: + +1. No invariant contradiction remains unresolved. +2. The parent can explain why caller-owned regrouping is forbidden after the session-record API lands. +3. The parent can explain why `PLAN-05` remains an acceptance gate for `PLAN-06`, not a separate slice. + +### task/m06-a2-projection-foundation + +Ownership: + +- parent only + +Why serialized: + +- [state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) owns the projection contract, source precedence, hardening, torn-root warnings, and discovery rules. There is no safe throughput gain from splitting that file before its API is frozen. + +Allowed files: + +- [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) + +Scope: + +1. Add canonical session-root helpers for: + - `sessions//session.json` + - `sessions//participants/.json` + - `sessions//leases/.lease` +2. Add one hardened participant walker that merges canonical, flat-current, and legacy participant objects by object precedence. +3. Add parent-session loading that prefers canonical `session.json` over flat `sessions/.json`. +4. Add: + - `load_session()` + - `list_sessions()` + - `list_live_sessions()` + - `resolve_single_live_session_for_agent()` + - compatibility wrappers only if they reduce diff size +5. Add torn-root warnings and incomplete-root handling. +6. Add session discovery from the union of canonical session roots, flat session files, and participant-derived `orchestration_session_id` values. +7. Add path hardening with `symlink_metadata` and explicit rejection of symlinked or non-regular entries. +8. Add unit coverage for precedence, torn roots, path hardening, and participant-only discovery. + +Must not do: + +1. No `agents_cmd.rs` consumer cutover yet. +2. No canonical writer cutover yet. +3. No new cache or index file. +4. No caller-owned merge helpers outside the store. + +Acceptance: + +1. The parent can name one store-owned session-record API surface for live discovery. +2. `list_sessions()` can discover participant-only torn roots. +3. `list_live_sessions()` excludes incomplete roots. +4. Canonical, flat, and legacy precedence is proven in source-local tests. +5. Path-hardening tests prove symlinked or non-regular entries are ignored. + +Green-path output: + +- `.runs/plan-06/sentinels/task-m06-a2-projection-foundation.ok` + +Blocked-path output: + +- `.runs/plan-06/blocked.json` + +### Parent validation gate B + +Required before `task/m06-b1-consumer-invalidation-cutover` starts: + +1. `cargo test -p shell agent_runtime::state_store -- --nocapture` passes. +2. The projection/read contract is frozen into `run-state.json`. +3. The parent can state the object-level precedence contract without referring back to callers. + +### task/m06-b1-consumer-invalidation-cutover + +Ownership: + +- parent only + +Why serialized: + +- [agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs) currently depends on `list_live_manifests()`, `resolve_live_orchestrator_session()`, and flat tombstone scans that are all being replaced by the same evolving store contract in [state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs). Splitting this phase across workers would create API churn without real parallelism. + +Allowed files: + +- [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) +- [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs) + +Scope: + +1. Move `build_status_report()` onto `list_live_sessions()`. +2. Preserve one selected status row per live participant. +3. Preserve same-agent concurrent-session visibility instead of collapsing rows by `agent_id`. +4. Move tombstone suppression onto `list_invalidated_participants_across_sources()`. +5. Move invalidation reads onto the same shared participant walker by updating `invalidate_stale_world_members_for_session()`. +6. Restrict trace fallback to gaps that are not already covered by a live record or tombstone. +7. Move `build_toolbox_status_report()` and `build_toolbox_env_report()` onto `resolve_single_live_session_for_agent()`. +8. Preserve the current toolbox JSON fields: + - `active_orchestration_session_id` + - `active_world_binding` + - `dependency_unavailable` +9. Preserve fail-closed ambiguity with operator-readable errors. + +Must not do: + +1. No canonical writer cutover yet. +2. No `async_repl.rs` lifecycle edits. +3. No public JSON reshaping beyond the plan contract. +4. No regression to `PLAN-05` suppression semantics. + +Acceptance: + +1. No authoritative live status selection depends on `list_live_manifests()`. +2. No tombstone or invalidation path assumes only flat `participants/*.json`. +3. Toolbox resolution fails closed via session-record selection. +4. Same-agent concurrent sessions stay distinct in status output. +5. The parent can prove the suppression tuple still matches `PLAN-05`. + +Green-path output: + +- `.runs/plan-06/sentinels/task-m06-b1-consumer-invalidation-cutover.ok` + +Blocked-path output: + +- `.runs/plan-06/blocked.json` + +### task/m06-c1-canonical-writer-cutover + +Ownership: + +- parent only + +Why serialized: + +- The writer cutover spans [state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) and [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) across the same persistence choke point. That is single-writer work. + +Allowed files: + +- [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) + +Scope: + +1. Keep current bootstrap and lifecycle ordering semantics intact. +2. Move parent, participant, and lease writes onto store-owned canonical helpers. +3. Keep `persist_runtime_snapshots()` as the write choke point. +4. Keep any temporary dual-write inside the store only. +5. Preserve mixed-layout read correctness while canonical and flat files coexist. +6. Add crash-window coverage for: + - parent written, participant missing, + - participant written, parent missing, + - active handle points at stale participant. +7. Update any helper readers in `async_repl.rs` that still assume only flat participant layout. + +Must not do: + +1. No transactional redesign. +2. No scope expansion into new operator UX. +3. No worker dispatch before the parent validates the stabilized production diff. + +Acceptance: + +1. Canonical writes land under `sessions//...` through store-owned helpers. +2. Crash-window reads still degrade safely. +3. Mixed canonical and compatibility layouts remain readable by the store. +4. No lifecycle ordering change escapes the write path. + +Green-path output: + +- `.runs/plan-06/sentinels/task-m06-c1-canonical-writer-cutover.ok` + +Blocked-path output: + +- `.runs/plan-06/blocked.json` + +### Parent validation gate C + +Required before the worker window opens: + +1. `cargo test -p shell agent_runtime::state_store -- --nocapture` passes again after the writer cutover. +2. `cargo test -p shell --tests --no-run` passes so the late worker packet starts from a compiling production state. +3. The parent seeds both worker worktrees from the exact post-`task/m06-c1` tree. +4. The parent records any still-allowed compatibility bridge in `run-state.json`. + +### task/m06-d1-contract-tests-docs + +Ownership: + +- worker-owned +- parent-reviewed + +Worktree: + +- `../substrate-m06-contract-tests-docs` +- `codex/feat-session-centric-state-store-m06-contract-tests-docs` + +Allowed files: + +- [crates/shell/tests/agent_successor_contract_ahcsitc0.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_successor_contract_ahcsitc0.rs) +- [docs/TRACE.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/TRACE.md) +- [docs/USAGE.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/USAGE.md) + +Scope: + +1. Update contract fixtures and assertions to default to canonical session-root trees. +2. Add contract coverage for: + - toolbox status resolution from session records, + - toolbox env resolution from the same selected session, + - same-agent concurrent session visibility, + - fail-closed ambiguity, + - canonical participant precedence over flat compatibility data, + - flat compatibility fallback when canonical roots are incomplete. +3. Correct the stale `docs/USAGE.md` wording that still says live discovery is backed by `handles/`. +4. Update `docs/TRACE.md` so the live-state authority description matches the final session-root truth plus any bounded compatibility bridge. + +Must not do: + +1. No edits to `state_store.rs`, `agents_cmd.rs`, or `async_repl.rs`. +2. No edits to any other tests. +3. No new scope beyond the session-centric store slice. + +Commands: + +1. `cargo test -p shell --test agent_successor_contract_ahcsitc0 -- --nocapture` + +Acceptance: + +1. `docs/USAGE.md` no longer names `handles/` as the live authority. +2. Contract tests assert session-record-driven status and toolbox behavior. +3. The worker report includes exact tests run and any remaining doc ambiguities. + +Green-path output: + +- `.runs/plan-06/sentinels/task-m06-d1-contract-tests-docs.ok` + +Blocked-path output: + +- `.runs/plan-06/blocked.json` if the worker needs parent-owned files + +### task/m06-d2-persistence-routing-tests + +Ownership: + +- worker-owned +- parent-reviewed + +Worktree: + +- `../substrate-m06-persistence-routing-tests` +- `codex/feat-session-centric-state-store-m06-persistence-routing-tests` + +Allowed files: + +- [crates/shell/tests/agent_hub_trace_persistence.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_hub_trace_persistence.rs) +- [crates/shell/tests/repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs) + +Scope: + +1. Add integration coverage for canonical session-root persistence. +2. Add coverage that `PLAN-05` invalidation still works after participant location changes. +3. Add coverage that tombstone suppression still wins after the consumer regrouping cutover. +4. Add coverage that same-agent concurrent world rows stay isolated across session ids. +5. Add externally visible crash-window coverage where the named integration tests can prove safe degradation. + +Must not do: + +1. No edits to helper readers in `async_repl.rs`. +2. No edits to `state_store.rs` or `agents_cmd.rs`. +3. No expansion into unrelated integration suites. + +Commands: + +1. `cargo test -p shell --test agent_hub_trace_persistence -- --nocapture` +2. `cargo test -p shell --test repl_world_first_routing_v1 -- --nocapture` + +Acceptance: + +1. The named integration suites cover persistence and routing behavior under the canonical layout. +2. The worker report makes any unmet helper dependency explicit instead of silently widening scope. +3. `PLAN-05` invalidation expectations are preserved in the updated tests. + +Green-path output: + +- `.runs/plan-06/sentinels/task-m06-d2-persistence-routing-tests.ok` + +Blocked-path output: + +- `.runs/plan-06/blocked.json` if the worker needs parent-owned files + +### task/m06-e1-integrate-and-validate + +Ownership: + +- parent only + +Scope: + +1. Review both worker outputs against the frozen contracts. +2. Reject any worker patch that touches unowned files, assumes stale authority, or omits test evidence. +3. Integrate accepted worker outputs into the parent checkout. +4. If worker outputs expose missing parent-owned support changes, make those changes in the parent checkout and rerun the affected targeted validation steps. +5. Run the final validation stack in the exact order defined below. + +Acceptance: + +1. Every accepted worker output has a matching parent-written sentinel. +2. No stale `handles/` authority wording remains in docs. +3. The full validation order passes. +4. `run-state.json` records accepted and rejected outputs explicitly. + +Green-path output: + +- `.runs/plan-06/sentinels/task-m06-e1-integrate-and-validate.ok` + +Blocked-path output: + +- `.runs/plan-06/blocked.json` + +### task/m06-e2-closeout + +Ownership: + +- parent only + +Scope: + +1. Mark the run complete in `.runs/plan-06/run-state.json`. +2. Write `.runs/plan-06/closeout.md` with: + - accepted tasks, + - integrated worktrees and branches, + - validation commands and outcomes, + - any compatibility behavior intentionally retained, + - any explicit removals of legacy `handles/*.json` reads. +3. Append the final acceptance rationale to `.runs/plan-06/session.log`. + +Acceptance: + +1. `closeout.md` is present and matches the final branch state. +2. All sentinels through `task/m06-e2-closeout` exist. +3. No blocked-path artifact exists for a successful run. + +Green-path output: + +- `.runs/plan-06/sentinels/task-m06-e2-closeout.ok` + +## Context-Control Rules + +1. The parent owns `.runs/plan-06/*`. Workers do not edit run-state, sentinels, queue, session log, blocked state, or closeout artifacts. +2. Worker packets include only: + - the task ID, + - allowed files, + - frozen invariants, + - stop conditions, + - exact tests to run. +3. Workers must stop immediately if they need to touch any file outside their allowed list. +4. Workers report: + - touched files, + - tests run, + - unresolved assumptions, + - blockers, + - whether they observed any stale authority wording or compatibility assumptions. +5. Workers do not rebase, merge, or integrate each other’s work. +6. The parent rejects any worker patch that silently broadens scope or makes compatibility policy decisions not already frozen in `run-state.json`. +7. If worker outputs conflict with current parent truth, the parent re-derives the correct result from production code and rewrites the patch locally instead of negotiating a blended state across worktrees. + +## Tests And Acceptance + +### Pre-worker proof order + +Run these parent-owned checks before dispatching the worker window: + +1. `cargo test -p shell agent_runtime::state_store -- --nocapture` +2. `cargo test -p shell --tests --no-run` + +### Final validation order + +Run these commands in this exact order during `task/m06-e1-integrate-and-validate`: + +1. `cargo test -p shell agent_runtime::state_store -- --nocapture` +2. `cargo test -p shell --test agent_successor_contract_ahcsitc0 -- --nocapture` +3. `cargo test -p shell --test agent_hub_trace_persistence -- --nocapture` +4. `cargo test -p shell --test repl_world_first_routing_v1 -- --nocapture` +5. `cargo test -p shell -- --nocapture` +6. `cargo fmt --all -- --check` +7. `cargo clippy -p shell --all-targets -- -D warnings` +8. `cargo clippy --workspace --all-targets -- -D warnings` +9. `cargo test --workspace -- --nocapture` + +### Acceptance checklist + +The slice is accepted only if all of these are true: + +1. The store can answer `load_session()` and `list_live_sessions()` directly. +2. `agent status` reads live rows from store-owned session records. +3. `toolbox status|env` resolves exactly one live session record and fails closed on ambiguity. +4. `PLAN-05` invalidation and tombstone suppression remain correct under the canonical session-root layout. +5. Crash-window reads remain safe. +6. Docs reflect the current authority boundary. +7. No caller-owned regrouping or stale `handles/` authority survives the cutover. + +## Merge Refusal Rules + +The parent refuses to merge a worker output if any of these are true: + +1. The patch edits a file outside the task’s allowed file list. +2. The patch assumes `handles/*.json` is authoritative for live discovery. +3. The patch weakens `PLAN-05` suppression, invalidation, or same-agent concurrent-session visibility. +4. The patch requires concurrent parent edits to `state_store.rs`, `agents_cmd.rs`, or `async_repl.rs` to become intelligible. +5. The patch omits test evidence for the behavior it claims to cover. +6. The patch leaves doc wording or fixture assumptions in a state that contradicts the final production code. + +## Run Exit Criteria + +### Successful run + +The run is successful only if all of these are true: + +1. `.runs/plan-06/sentinels/task-m06-a1-preflight.ok` exists. +2. `.runs/plan-06/sentinels/task-m06-a2-projection-foundation.ok` exists. +3. `.runs/plan-06/sentinels/task-m06-b1-consumer-invalidation-cutover.ok` exists. +4. `.runs/plan-06/sentinels/task-m06-c1-canonical-writer-cutover.ok` exists. +5. `.runs/plan-06/sentinels/task-m06-d1-contract-tests-docs.ok` exists. +6. `.runs/plan-06/sentinels/task-m06-d2-persistence-routing-tests.ok` exists. +7. `.runs/plan-06/sentinels/task-m06-e1-integrate-and-validate.ok` exists. +8. `.runs/plan-06/sentinels/task-m06-e2-closeout.ok` exists. +9. `.runs/plan-06/run-state.json` exists and records a completed terminal state. +10. `.runs/plan-06/queue.json` and `.runs/plan-06/session.log` exist. +11. `.runs/plan-06/closeout.md` exists and matches the final accepted branch state. +12. `.runs/plan-06/blocked.json` does not exist. + +### Blocked termination + +The run terminates blocked only if all of these are true: + +1. `.runs/plan-06/blocked.json` exists and records the blocking reason. +2. `.runs/plan-06/run-state.json` exists and records `blocked` as the terminal state. +3. `session.log` contains the parent rationale for the stop. +4. No downstream task sentinel may be written after the blocking point. +5. `.runs/plan-06/sentinels/task-m06-e2-closeout.ok` must not exist. +6. `.runs/plan-06/closeout.md` must not exist. + +## Closeout + +Successful closeout records: + +1. the final frozen contract actually shipped, +2. the exact validation order and outcomes, +3. whether compatibility reads from `handles/*.json` remain and why, +4. confirmation that the stale `docs/USAGE.md` authority wording is gone, +5. confirmation that `PLAN-05` still governs invalidation and tombstone semantics after the session-centric cutover. + +## Assumptions + +1. The parent run starts from the current branch baseline `feat/session-centric-state-store`. +2. `PLAN-05` is already the upstream invalidation and tombstone contract and must be preserved rather than redesigned here. +3. The Rust package name `shell` is authoritative for validation commands in this repository. +4. A bounded read-compatibility bridge around `handles/*.json` may remain temporarily during migration, but only as compatibility input inside the store and never as live-state authority. diff --git a/llm-last-mile/ORCH_PLAN-08.md b/llm-last-mile/ORCH_PLAN-08.md new file mode 100644 index 000000000..a4e866546 --- /dev/null +++ b/llm-last-mile/ORCH_PLAN-08.md @@ -0,0 +1,729 @@ +# ORCH_PLAN-08: Explicit Orchestration Authority For Event Emission + +Branch: `feat/session-centric-state-store` +Plan source: [PLAN-08.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-08.md) +Execution type: shell/runtime authority cleanup orchestration plan, no UI scope + +## Summary + +This run executes `PLAN-08` on the current branch `feat/session-centric-state-store` with an exact worker cap of `2`. The parent remains the only integrator, the only final branch writer, and the only agent allowed to mutate the coupled production seam across [crates/shell/src/execution/agent_events.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_events.rs), [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs), [crates/shell/src/execution/routing/dispatch/exec.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/exec.rs), and [crates/shell/src/execution/routing/dispatch/world_ops.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs). The canonical run-state source of truth is `.runs/plan-08/run-state.json`. + +The worker cap is exactly `2` because the only honest parallelism in this slice is the late regression/doc window after the shell event context contract and launch-boundary authority/correlation mapping are frozen. There are intentionally zero safe production-code worker lanes before that point. `agent_events.rs`, `async_repl.rs`, `exec.rs`, and `world_ops.rs` are one authority seam, and `execute_command(...)` signature fallout into [crates/shell/src/execution/invocation/runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/invocation/runtime.rs) is part of that same parent-owned path, not an independent lane. + +Worktree set for the run: + +- `../substrate-m08-trace-contract-docs` on `codex/feat-session-centric-state-store-m08-trace-contract-docs` +- `../substrate-m08-repl-world-regressions` on `codex/feat-session-centric-state-store-m08-repl-world-regressions` + +The critical path stays parent-owned through these phases: + +1. `task/m08-a1-preflight` +2. `task/m08-a2-context-and-launch-freeze` +3. `task/m08-b1-host-plumbing-and-signature-fallout` +4. `task/m08-c1-world-stream-plumbing-and-helper-retirement` +5. `task/m08-e1-integrate-and-validate` +6. `task/m08-e2-closeout` + +This is deliberate. `PLAN-08` is truthful that most production work is sequential until the explicit shell context contract and launch-boundary authority/correlation mapping are frozen. Fake parallelism across `agent_events.rs`, `async_repl.rs`, `exec.rs`, and `world_ops.rs` would create merge churn and contract drift faster than it would create throughput. The parent-owned execution brief is: + +1. freeze the explicit shell event context shape and the no-PID-lookup rule first, +2. freeze the REPL launch-boundary authority and correlation mapping second, +3. thread that frozen contract through host execution and `execute_command(...)` fallout third, +4. thread the same frozen contract through world non-PTY streaming and retire production helper lookups fourth, +5. open workers only after the production seam is stable in the parent checkout, +6. integrate worker outputs in one parent-owned validation pass, +7. close the run only if the shell-targeted validation stack and final repo gates are green. + +## Hard Guards + +### Locked invariants + +1. `PLAN-08` is a shell/runtime authority cleanup slice only. No UI work is authorized. +2. The parent is the only integrator and the only final branch writer. +3. Shell-owned orchestration-scoped rows must use explicit caller-provided authority; ambient PID recovery on production emission paths is forbidden. +4. Shell-owned stream rows must not emit with synthetic correlation such as `run_id="unknown"`. +5. Missing orchestration context suppresses only the orchestration-scoped `agent_event` row; stdout, stderr, and trace spans must keep working. +6. `RuntimeOrchestrationContext` in `async_repl.rs` remains the authority source for REPL-owned orchestration context. +7. [crates/shell/src/execution/invocation/runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/invocation/runtime.rs) remains a non-orchestrator caller in this slice and must pass `None` explicitly after `execute_command(...)` changes. +8. [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) is a helper-retirement seam only; this slice does not authorize live-state redesign. +9. [docs/TRACE.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/TRACE.md) is a contract lock seam; it must reflect the shipped suppression and correlation rules. +10. No new cache, registry, service, selector UX, or infrastructure is allowed. +11. No human approval gate is required or invented for this run. +12. Validation commands must use actual package names in this repo, especially `cargo test -p shell`. + +### File-level boundaries + +Parent-owned serialized production surfaces: + +- [crates/shell/src/execution/agent_events.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_events.rs) +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) +- [crates/shell/src/execution/routing/dispatch/exec.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/exec.rs) +- [crates/shell/src/execution/routing/dispatch/world_ops.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs) +- [crates/shell/src/execution/invocation/runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/invocation/runtime.rs) +- [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) + +Worker-safe late-phase validation and doc surfaces: + +- [crates/shell/tests/agent_hub_trace_persistence.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_hub_trace_persistence.rs) +- [crates/shell/tests/repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs) +- [docs/TRACE.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/TRACE.md) + +Read-for-truth only: + +- [llm-last-mile/PLAN-08.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-08.md) +- [llm-last-mile/ORCH_PLAN-06.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/ORCH_PLAN-06.md) +- [llm-last-mile/ORCH_PLAN-05.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/ORCH_PLAN-05.md) + +### Non-negotiable stop conditions + +Stop the run and write `.runs/plan-08/blocked.json` if any of these occur: + +1. A task requires parallel edits across `agent_events.rs`, `async_repl.rs`, `exec.rs`, or `world_ops.rs` before the context and launch contract is frozen. +2. A task requires `invocation/runtime.rs` to derive orchestration context instead of explicitly passing `None`. +3. A task restores `find_active_orchestration_session_for_pid(...)` or any renamed equivalent to production event-emission control flow. +4. A task emits orchestration-scoped shell stream rows with synthetic fallback correlation. +5. A task blocks stdout, stderr, or trace span emission when orchestration-scoped rows are suppressed. +6. A worker lane needs to touch any parent-owned production file or any source-local unit test inside those files. +7. A task expands into `PLAN-09` live-state authority work, restart semantics redesign, or schema redesign. +8. A task requires new infrastructure or operator approval not already present in the plan. + +## Orchestration State Surfaces + +### Canonical run state + +Single local source of truth for the run: + +- `.runs/plan-08/run-state.json` + +Parent-only writes to this file. It tracks: + +- current phase, +- active task IDs, +- branch and worktree assignment, +- gate status, +- frozen shell event context contract, +- frozen launch-boundary authority/correlation mapping, +- accepted and rejected worker outputs, +- blocked or completed terminal state, +- final closeout pointer. + +If a worker report conflicts with `run-state.json`, the parent trusts `run-state.json` until it explicitly reconciles the discrepancy. + +### Derived run artifacts + +The parent may maintain these local artifacts: + +- `.runs/plan-08/queue.json` +- `.runs/plan-08/session.log` +- `.runs/plan-08/sentinels/task-m08-a1-preflight.ok` +- `.runs/plan-08/sentinels/task-m08-a2-context-and-launch-freeze.ok` +- `.runs/plan-08/sentinels/task-m08-b1-host-plumbing-and-signature-fallout.ok` +- `.runs/plan-08/sentinels/task-m08-c1-world-stream-plumbing-and-helper-retirement.ok` +- `.runs/plan-08/sentinels/task-m08-d1-trace-contract-docs.ok` +- `.runs/plan-08/sentinels/task-m08-d2-repl-world-regressions.ok` +- `.runs/plan-08/sentinels/task-m08-e1-integrate-and-validate.ok` +- `.runs/plan-08/sentinels/task-m08-e2-closeout.ok` +- `.runs/plan-08/blocked.json` +- `.runs/plan-08/closeout.md` + +Sentinel rules: + +1. `.ok` means the parent validated the task output and advanced the run. +2. Missing sentinel means the task is not accepted. +3. `blocked.json` is written only on blocked termination. +4. `closeout.md` is written only on successful completion. +5. Worker-generated notes never replace parent-written sentinels or run-state artifacts. + +## Concurrency Policy + +1. The parent is the only integrator. +2. The parent is the only writer of final branch state on `feat/session-centric-state-store`. +3. Exact worker cap: `2` active worker lanes. +4. There are zero worker lanes during `task/m08-a2`, `task/m08-b1`, or `task/m08-c1`. +5. No two lanes may edit `agent_events.rs`, `async_repl.rs`, `exec.rs`, `world_ops.rs`, `invocation/runtime.rs`, or `state_store.rs` concurrently. +6. Worker lanes open only after the parent has stabilized the production seam and seeded both worktrees from that exact state. +7. `task/m08-d1-trace-contract-docs` and `task/m08-d2-repl-world-regressions` are the only parallel window in this run. +8. If worker tasks expose missing parent-owned support work, the worker stops and hands the change back to the parent instead of widening scope. + +## Approval And Gate Model + +There are no human approval gates defined for this run. + +Replacement control mechanism: + +1. parent validation gates, +2. parent-written sentinels, +3. `session.log` for acceptance and rejection rationale, +4. `blocked.json` for hard-stop termination, +5. `closeout.md` for successful completion. + +### Gate A: Packet freeze + +Required before implementation starts: + +- parent re-reads `PLAN-08`, `ORCH_PLAN-06`, and `ORCH_PLAN-05`, +- parent records that early production work stays parent-owned because the authority seam is coupled across REPL launch, host execution, and world streaming, +- parent records the package-name normalization: use `shell`, never `substrate-shell`. + +### Gate B: Context and launch freeze + +Required before host plumbing starts: + +- the shell event context type is frozen, +- the REPL launch-boundary mapping for `cmd_id`, `run_id`, and `span_id` is frozen, +- the parent can explain why non-REPL callers in `invocation/runtime.rs` must explicitly pass `None`. + +### Gate C: Host-path freeze + +Required before world plumbing starts: + +- `execute_command(...)` signature fallout is complete, +- host stream threads capture immutable explicit context before background I/O starts, +- no host production path still emits `run_id="unknown"` for orchestration-scoped rows. + +### Gate D: Production stabilization + +Required before the worker window opens: + +- production integration is complete in the parent checkout, +- no structural churn remains in `agent_events.rs`, `async_repl.rs`, `exec.rs`, `world_ops.rs`, `invocation/runtime.rs`, or `state_store.rs`, +- the parent has completed the pre-worker compile/proof checks defined below, +- both worker worktrees are seeded from the exact post-`task/m08-c1` tree. + +### Gate E: Final acceptance + +Required before closeout: + +- both worker outputs are accepted or deliberately rejected and replaced by parent work, +- the final validation order passes, +- docs reflect the final authority boundary, +- production emitters no longer depend on PID-based orchestration recovery. + +## Workstream Plan + +### Worktree topology + +Parent checkout: + +- current checkout on `feat/session-centric-state-store` + +Child worktrees and branches: + +- `../substrate-m08-trace-contract-docs` + - `codex/feat-session-centric-state-store-m08-trace-contract-docs` +- `../substrate-m08-repl-world-regressions` + - `codex/feat-session-centric-state-store-m08-repl-world-regressions` + +Subagents do not merge each other’s work. They return patches, touched files, tests run, and blockers to the parent. + +### Task graph + +Execution graph for the run: + +1. `task/m08-a1-preflight` +2. `task/m08-a2-context-and-launch-freeze` +3. `task/m08-b1-host-plumbing-and-signature-fallout` +4. `task/m08-c1-world-stream-plumbing-and-helper-retirement` +5. `task/m08-d1-trace-contract-docs` and `task/m08-d2-repl-world-regressions` in parallel +6. `task/m08-e1-integrate-and-validate` +7. `task/m08-e2-closeout` + +Parent-only serialized tasks: + +- `task/m08-a1-preflight` +- `task/m08-a2-context-and-launch-freeze` +- `task/m08-b1-host-plumbing-and-signature-fallout` +- `task/m08-c1-world-stream-plumbing-and-helper-retirement` +- `task/m08-e1-integrate-and-validate` +- `task/m08-e2-closeout` + +Worker-owned tasks: + +- `task/m08-d1-trace-contract-docs` +- `task/m08-d2-repl-world-regressions` + +## Parallel Window D + +This is the only worker window in the run. + +It opens only after Gate D passes and after the parent has finished the full production path through `task/m08-c1-world-stream-plumbing-and-helper-retirement`. There are zero safe production-code workers before that because the launch-boundary authority contract and the host/world streaming contract are one coupled seam. `agent_events.rs`, `async_repl.rs`, `exec.rs`, and `world_ops.rs` cannot be split honestly until the parent has frozen the exact context shape and correlation rules. + +### task/m08-a1-preflight + +Ownership: + +- parent only + +Scope: + +1. Re-read [PLAN-08.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-08.md), [ORCH_PLAN-06.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/ORCH_PLAN-06.md), and [ORCH_PLAN-05.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/ORCH_PLAN-05.md). +2. Confirm the run executes from `feat/session-centric-state-store`. +3. Freeze the no-UI scope, parent-only critical seam list, and honest concurrency limits into the run packet. +4. Initialize: + - `.runs/plan-08/run-state.json` + - `.runs/plan-08/queue.json` + - `.runs/plan-08/session.log` +5. Record the repo-truth validation rule that all package-targeted commands use `-p shell`. + +Acceptance: + +1. The parent can explain why early production work stays parent-owned. +2. The parent can name the execute-command fallout into `invocation/runtime.rs`. +3. `run-state.json` records the initial phase and queue. + +Green-path output: + +- `.runs/plan-08/sentinels/task-m08-a1-preflight.ok` + +Blocked-path output: + +- `.runs/plan-08/blocked.json` + +### Parent validation gate A + +Required before `task/m08-a2-context-and-launch-freeze` starts: + +1. No invariant contradiction remains unresolved. +2. The parent can explain why fake parallelism across `agent_events.rs`, `async_repl.rs`, `exec.rs`, and `world_ops.rs` is disallowed. +3. The parent can restate the suppression rule as "real id or suppress." + +### task/m08-a2-context-and-launch-freeze + +Ownership: + +- parent only + +Why serialized: + +- [crates/shell/src/execution/agent_events.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_events.rs) and [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) define the contract that every downstream host/world emitter must consume. There is no honest throughput gain from splitting downstream work before this mapping is frozen. + +Allowed files: + +- [crates/shell/src/execution/agent_events.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_events.rs) +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) + +Scope: + +1. Add the explicit shell event context type and any adjacent shell-local helpers. +2. Move command-completion emission onto the explicit context contract. +3. Freeze how REPL launch points derive authority from `RuntimeOrchestrationContext` plus the live manifest/world snapshot. +4. Freeze how REPL launch points choose `cmd_id`, `run_id`, and `span_id` for shell-owned rows. +5. Remove REPL production dependence on `resolve_active_orchestration_session_id()`. +6. Keep source-local regression additions in these files parent-owned. + +Must not do: + +1. No host or world stream plumbing yet. +2. No `invocation/runtime.rs` fallback logic. +3. No new shared schema or cross-crate contract redesign. + +Acceptance: + +1. `publish_command_completion(...)` consumes the frozen shell context contract. +2. REPL command-completion paths no longer recover orchestration identity from PID-owned runtime state. +3. The parent can state exactly when `run_id` is authoritative and when rows must be suppressed. + +Green-path output: + +- `.runs/plan-08/sentinels/task-m08-a2-context-and-launch-freeze.ok` + +Blocked-path output: + +- `.runs/plan-08/blocked.json` + +### Parent validation gate B + +Required before `task/m08-b1-host-plumbing-and-signature-fallout` starts: + +1. `cargo test -p shell publish_command_completion -- --nocapture` passes. +2. `cargo test -p shell build_world_restart_required_alert_only_builds_with_orchestration_context -- --nocapture` passes. +3. The frozen shell context contract and launch-boundary mapping are recorded in `run-state.json`. + +### task/m08-b1-host-plumbing-and-signature-fallout + +Ownership: + +- parent only + +Why serialized: + +- [crates/shell/src/execution/routing/dispatch/exec.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/exec.rs) owns `execute_command(...)`, `execute_external(...)`, and host stream thread setup. The signature fallout into [crates/shell/src/execution/invocation/runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/invocation/runtime.rs) is mechanical but coupled to the same change. + +Allowed files: + +- [crates/shell/src/execution/routing/dispatch/exec.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/exec.rs) +- [crates/shell/src/execution/invocation/runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/invocation/runtime.rs) +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) for direct call-site fallout only + +Scope: + +1. Thread optional explicit shell event context into `execute_command(...)` and `execute_external(...)`. +2. Capture immutable host-stream authority and stable run correlation before background stream threads start. +3. Remove host-path PID lookup and stop emitting orchestration-scoped host stream rows with `run_id="unknown"`. +4. Update `invocation/runtime.rs` callers to pass `None` explicitly. +5. Preserve host-only execution behavior when orchestration context is absent. + +Must not do: + +1. No world non-PTY frame-loop changes yet. +2. No attempt to infer orchestration context inside `invocation/runtime.rs`. +3. No worker dispatch before the parent proves the host path compiles and suppresses correctly. + +Acceptance: + +1. Non-REPL host invocation callers compile with explicit `None`. +2. Host stream emission uses launch-owned correlation only. +3. No host production path still depends on `resolve_active_orchestration_session_id()`. + +Green-path output: + +- `.runs/plan-08/sentinels/task-m08-b1-host-plumbing-and-signature-fallout.ok` + +Blocked-path output: + +- `.runs/plan-08/blocked.json` + +### Parent validation gate C + +Required before `task/m08-c1-world-stream-plumbing-and-helper-retirement` starts: + +1. `cargo test -p shell emit_stream_chunk -- --nocapture` passes. +2. `cargo test -p shell --tests --no-run` passes so the parent is carrying a compiling host-path diff forward. +3. The parent can explain why `invocation/runtime.rs` passing `None` is a contract requirement, not an omission. + +### task/m08-c1-world-stream-plumbing-and-helper-retirement + +Ownership: + +- parent only + +Why serialized: + +- [crates/shell/src/execution/routing/dispatch/world_ops.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs) consumes the same frozen authority and correlation contract as the host path, and [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) is only being narrowed as a helper-retirement seam. + +Allowed files: + +- [crates/shell/src/execution/routing/dispatch/world_ops.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs) +- [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) for parent-owned fallout only + +Scope: + +1. Thread explicit event context into `stream_non_pty_via_agent(...)`, `process_agent_stream_body(...)`, and `emit_stream_chunk(...)`. +2. Remove world-path PID lookup and suppress orchestration-scoped stream rows until both context and real run correlation exist. +3. Preserve stdout/stderr passthrough and deny-path terminal behavior when rows are suppressed. +4. Leave `find_active_orchestration_session_for_pid(...)` available only for tests or diagnostics if still needed. +5. Keep any source-local regression additions in these parent-owned files. + +Must not do: + +1. No new state-store authority behavior. +2. No synthetic run-correlation fallback from `active_span_id` alone. +3. No widening of worker-safe scope into source-local tests. + +Acceptance: + +1. No production event-emission path in `exec.rs`, `world_ops.rs`, or `async_repl.rs` resolves orchestration identity from PID-owned runtime state. +2. World non-PTY rows no longer emit with synthetic correlation. +3. Suppression preserves terminal output and trace span behavior. + +Green-path output: + +- `.runs/plan-08/sentinels/task-m08-c1-world-stream-plumbing-and-helper-retirement.ok` + +Blocked-path output: + +- `.runs/plan-08/blocked.json` + +### Parent validation gate D + +Required before the worker window opens: + +1. `cargo test -p shell emit_world_restarted_alert_only_emits_with_orchestration_context -- --nocapture` passes. +2. `cargo test -p shell start_host_orchestrator_runtime_persists_participant_snapshots_across_lifecycle_states -- --nocapture` passes. +3. `cargo test -p shell --tests --no-run` passes from the stabilized parent checkout. +4. The parent seeds both worker worktrees from the exact post-`task/m08-c1` tree. + +### task/m08-d1-trace-contract-docs + +Ownership: + +- worker-owned +- parent-reviewed + +Worktree: + +- `../substrate-m08-trace-contract-docs` +- `codex/feat-session-centric-state-store-m08-trace-contract-docs` + +Allowed files: + +- [crates/shell/tests/agent_hub_trace_persistence.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_hub_trace_persistence.rs) +- [docs/TRACE.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/TRACE.md) + +Scope: + +1. Update `docs/TRACE.md` so shell-owned command completion and shell-owned stream emitters are explicit examples of the "real id or suppress" rule. +2. Add or tighten trace assertions for: + - no-context shell completion suppressing orchestration-scoped `agent_event` rows, + - runtime-owned rows retaining real orchestration session identity, + - shell-owned omission behavior staying additive rather than heuristic. +3. Add a bounded guard in the allowed integration surface if needed, but only if it does not require touching production files. + +Must not do: + +1. No edits to `agent_events.rs`, `async_repl.rs`, `exec.rs`, `world_ops.rs`, `invocation/runtime.rs`, or `state_store.rs`. +2. No edits to `repl_world_first_routing_v1.rs`. +3. No source-local unit-test edits inside production files. + +Commands: + +1. `cargo test -p shell no_context_shell_command_completion_does_not_synthesize_agent_event_trace_row -- --nocapture` +2. `cargo test -p shell runtime_owned_agent_event_rows_retain_shell_session_and_real_orchestration_session -- --nocapture` +3. `cargo test -p shell --test agent_hub_trace_persistence -- --nocapture` + +Acceptance: + +1. `docs/TRACE.md` reflects the final shell-owned suppression and correlation contract. +2. The worker report includes exact tests run and any remaining contract ambiguities. +3. No worker patch widens into parent-owned source files. + +Green-path output: + +- `.runs/plan-08/sentinels/task-m08-d1-trace-contract-docs.ok` + +Blocked-path output: + +- `.runs/plan-08/blocked.json` if the worker needs parent-owned files + +### task/m08-d2-repl-world-regressions + +Ownership: + +- worker-owned +- parent-reviewed + +Worktree: + +- `../substrate-m08-repl-world-regressions` +- `codex/feat-session-centric-state-store-m08-repl-world-regressions` + +Allowed files: + +- [crates/shell/tests/repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs) + +Scope: + +1. Lock restart-alert suppression behavior so the refactor does not weaken the existing explicit-context-only posture. +2. Add or tighten world-routing regression coverage where the existing integration harness can prove suppression-only behavior without touching production code. +3. Make unmet helper dependencies explicit instead of silently requesting production edits. + +Must not do: + +1. No edits to any production file. +2. No edits to `agent_hub_trace_persistence.rs` or `docs/TRACE.md`. +3. No new integration harness outside the allowed file. + +Commands: + +1. `cargo test -p shell --test repl_world_first_routing_v1 -- --nocapture` + +Acceptance: + +1. Restart-alert suppression tests still pass after the production seam changes. +2. The worker report clearly identifies any remaining world-stream coverage gaps that require parent-owned work. +3. The lane stays inside the single allowed file. + +Green-path output: + +- `.runs/plan-08/sentinels/task-m08-d2-repl-world-regressions.ok` + +Blocked-path output: + +- `.runs/plan-08/blocked.json` if the worker needs parent-owned files + +### task/m08-e1-integrate-and-validate + +Ownership: + +- parent only + +Scope: + +1. Review both worker outputs against the frozen authority and correlation contracts. +2. Reject any worker patch that touches unowned files, assumes ambient lookup, or omits test evidence. +3. Integrate accepted worker outputs into the parent checkout. +4. If worker outputs expose missing parent-owned support changes, make those changes in the parent checkout and rerun the affected targeted validation steps. +5. Run the final validation stack in the exact order defined below. + +Acceptance: + +1. Every accepted worker output has a matching parent-written sentinel. +2. No production emitter depends on PID-based orchestration recovery. +3. The full validation order passes. +4. `run-state.json` records accepted and rejected outputs explicitly. + +Green-path output: + +- `.runs/plan-08/sentinels/task-m08-e1-integrate-and-validate.ok` + +Blocked-path output: + +- `.runs/plan-08/blocked.json` + +### task/m08-e2-closeout + +Ownership: + +- parent only + +Scope: + +1. Mark the run complete in `.runs/plan-08/run-state.json`. +2. Write `.runs/plan-08/closeout.md` with: + - accepted tasks, + - integrated worktrees and branches, + - validation commands and outcomes, + - any retained diagnostic-only use of helper APIs, + - confirmation that production emitters no longer consult PID-based session lookup. +3. Append the final acceptance rationale to `.runs/plan-08/session.log`. + +Acceptance: + +1. `closeout.md` is present and matches the final branch state. +2. All sentinels through `task/m08-e2-closeout` exist. +3. No blocked-path artifact exists for a successful run. + +Green-path output: + +- `.runs/plan-08/sentinels/task-m08-e2-closeout.ok` + +## Context-Control Rules + +1. The parent owns `.runs/plan-08/*`. Workers do not edit run-state, sentinels, queue, session log, blocked state, or closeout artifacts. +2. Worker packets include only: + - the task ID, + - allowed files, + - frozen invariants, + - stop conditions, + - exact tests to run. +3. Workers must stop immediately if they need to touch any file outside their allowed list. +4. Workers report: + - touched files, + - tests run, + - unresolved assumptions, + - blockers, + - whether they observed any remaining ambient-lookup or synthetic-correlation assumptions. +5. Workers do not rebase, merge, or integrate each other’s work. +6. The parent rejects any worker patch that silently broadens scope or makes authority decisions not already frozen in `run-state.json`. +7. If worker outputs conflict with current parent truth, the parent re-derives the correct result from production code and rewrites the patch locally instead of negotiating a blended state across worktrees. + +## Tests And Acceptance + +### Pre-worker proof order + +Run these parent-owned checks before dispatching the worker window: + +1. `cargo test -p shell publish_command_completion -- --nocapture` +2. `cargo test -p shell emit_stream_chunk -- --nocapture` +3. `cargo test -p shell build_world_restart_required_alert_only_builds_with_orchestration_context -- --nocapture` +4. `cargo test -p shell emit_world_restarted_alert_only_emits_with_orchestration_context -- --nocapture` +5. `cargo test -p shell start_host_orchestrator_runtime_persists_participant_snapshots_across_lifecycle_states -- --nocapture` +6. `cargo test -p shell --tests --no-run` + +### Final validation order + +Run these commands in this exact order during `task/m08-e1-integrate-and-validate`: + +1. `cargo test -p shell publish_command_completion -- --nocapture` +2. `cargo test -p shell emit_stream_chunk -- --nocapture` +3. `cargo test -p shell build_world_restart_required_alert_only_builds_with_orchestration_context -- --nocapture` +4. `cargo test -p shell emit_world_restarted_alert_only_emits_with_orchestration_context -- --nocapture` +5. `cargo test -p shell start_host_orchestrator_runtime_persists_participant_snapshots_across_lifecycle_states -- --nocapture` +6. `cargo test -p shell no_context_shell_command_completion_does_not_synthesize_agent_event_trace_row -- --nocapture` +7. `cargo test -p shell runtime_owned_agent_event_rows_retain_shell_session_and_real_orchestration_session -- --nocapture` +8. `cargo test -p shell --test agent_hub_trace_persistence -- --nocapture` +9. `cargo test -p shell --test repl_world_first_routing_v1 -- --nocapture` +10. `cargo test -p shell -- --nocapture` +11. `cargo fmt --all -- --check` +12. `cargo clippy -p shell --all-targets -- -D warnings` +13. `cargo clippy --workspace --all-targets -- -D warnings` +14. `cargo test --workspace -- --nocapture` + +Conditional blast-radius checks: + +1. Run `cargo test -p world-agent -- --nocapture` +2. Run `cargo test -p world-api -- --nocapture` +3. Run `cargo test -p agent-api-types -- --nocapture` + +Only require that conditional block if the implementation escapes shell-local authority plumbing and changes shared stream-frame, request, or cross-crate event-shape assumptions. + +### Acceptance checklist + +The slice is accepted only if all of these are true: + +1. No production event-emission path in `async_repl.rs`, `exec.rs`, or `world_ops.rs` resolves orchestration identity from `shell_owner_pid`. +2. Shell-owned command-completion rows consume explicit caller-provided authority. +3. Shell-owned stream rows consume explicit caller-provided authority and real run correlation. +4. Non-REPL callers in `invocation/runtime.rs` pass `None` explicitly instead of re-deriving runtime context. +5. Missing context suppresses only orchestration-scoped `agent_event` rows and does not suppress stdout, stderr, or trace spans. +6. `docs/TRACE.md` reflects the final "real id or suppress" rule. +7. `find_active_orchestration_session_for_pid(...)`, if retained, is no longer part of production emission control flow. + +## Merge Refusal Rules + +The parent refuses to merge a worker output if any of these are true: + +1. The patch edits a file outside the task’s allowed file list. +2. The patch assumes PID-based recovery or synthetic run correlation is acceptable for production emitters. +3. The patch requires concurrent parent edits to `agent_events.rs`, `async_repl.rs`, `exec.rs`, `world_ops.rs`, `invocation/runtime.rs`, or `state_store.rs` to become intelligible. +4. The patch omits test evidence for the behavior it claims to cover. +5. The patch leaves `docs/TRACE.md` or the targeted regression surfaces contradicting the final production code. + +## Run Exit Criteria + +### Successful run + +The run is successful only if all of these are true: + +1. `.runs/plan-08/sentinels/task-m08-a1-preflight.ok` exists. +2. `.runs/plan-08/sentinels/task-m08-a2-context-and-launch-freeze.ok` exists. +3. `.runs/plan-08/sentinels/task-m08-b1-host-plumbing-and-signature-fallout.ok` exists. +4. `.runs/plan-08/sentinels/task-m08-c1-world-stream-plumbing-and-helper-retirement.ok` exists. +5. `.runs/plan-08/sentinels/task-m08-d1-trace-contract-docs.ok` exists. +6. `.runs/plan-08/sentinels/task-m08-d2-repl-world-regressions.ok` exists. +7. `.runs/plan-08/sentinels/task-m08-e1-integrate-and-validate.ok` exists. +8. `.runs/plan-08/sentinels/task-m08-e2-closeout.ok` exists. +9. `.runs/plan-08/run-state.json` exists and records a completed terminal state. +10. `.runs/plan-08/queue.json` and `.runs/plan-08/session.log` exist. +11. `.runs/plan-08/closeout.md` exists and matches the final accepted branch state. +12. `.runs/plan-08/blocked.json` does not exist. + +### Blocked termination + +The run terminates blocked only if all of these are true: + +1. `.runs/plan-08/blocked.json` exists and records the blocking reason. +2. `.runs/plan-08/run-state.json` exists and records `blocked` as the terminal state. +3. `session.log` contains the parent rationale for the stop. +4. No downstream task sentinel may be written after the blocking point. +5. `.runs/plan-08/sentinels/task-m08-e2-closeout.ok` must not exist. +6. `.runs/plan-08/closeout.md` must not exist. + +## Closeout + +Successful closeout records: + +1. the final frozen shell event context contract that shipped, +2. the final launch-boundary authority/correlation mapping, +3. the exact validation order and outcomes, +4. confirmation that `invocation/runtime.rs` remained an explicit `None` caller path, +5. confirmation that production emitters no longer consult PID-based session lookup, +6. confirmation that `docs/TRACE.md` reflects the final suppression contract. + +## Assumptions + +1. The parent run starts from the current branch baseline `feat/session-centric-state-store`. +2. `PLAN-08` remains a shell-local authority-plumbing slice and does not expand into live-state redesign. +3. The Rust package name `shell` is authoritative for validation commands in this repository. +4. Source-local unit tests that live inside parent-owned production files remain parent-owned even after the production seam stabilizes. diff --git a/llm-last-mile/ORCH_PLAN-09.md b/llm-last-mile/ORCH_PLAN-09.md new file mode 100644 index 000000000..a36c960be --- /dev/null +++ b/llm-last-mile/ORCH_PLAN-09.md @@ -0,0 +1,679 @@ +# ORCH_PLAN-09: Live-State Authority and Compatibility Cutover + +Branch: `feat/session-centric-state-store` +Plan source: [PLAN-09.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-09.md) +Execution type: shell/runtime authority-cutover orchestration plan, no UI scope, strong DX/doc contract scope + +## Summary + +This run executes `PLAN-09` on the current branch `feat/session-centric-state-store` with an exact active worker cap of `2`. The parent remains the only integrator, the only final branch writer, and the only agent allowed to mutate the parent-owned production seam in [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) and [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs). The canonical run-state source of truth is `.runs/plan-09/run-state.json`. + +This slice is not an event-emission cleanup, a host-runtime lifecycle redesign, or a bridge-removal run. It is a contract-freeze slice around live-state authority and compatibility cutover. The only honest concurrency in this repo is: + +1. parent-only preflight and store authority freeze first, +2. one worker lane for operator-surface tightening and contract regressions second, +3. one worker lane for active-doc authority wording second, in parallel with the operator lane, +4. parent-only integration and final validation last. + +The active worker cap is exactly `2` because only two child worktrees are needed for the only real parallel window in this slice. The parent checkout itself carries all parent-only tasks. That means the practical execution surface is exactly three checkouts: + +- parent checkout on `feat/session-centric-state-store` for `task/m09-a1`, `task/m09-a2`, `task/m09-e1`, and `task/m09-e2` +- `../substrate-m09-operator-contracts` on `codex/feat-session-centric-state-store-m09-operator-contracts` +- `../substrate-m09-authority-docs` on `codex/feat-session-centric-state-store-m09-authority-docs` + +The critical path stays parent-owned through these phases: + +1. `task/m09-a1-preflight` +2. `task/m09-a2-store-authority-freeze` +3. `task/m09-e1-integrate-and-validate` +4. `task/m09-e2-closeout` + +This is deliberate. `PLAN-09` is already truthful about the repo topology: the authority seam must freeze first, then operator tightening and docs can proceed in parallel, then final validation closes the run. `state_store.rs` owns the live-state authority ladder, torn-root posture, selected-participant linkage, and fail-closed session resolution. Splitting workers before that file is frozen would create merge churn and contract drift rather than throughput. + +The parent-owned execution brief is: + +1. freeze the live-state authority ladder and retained compatibility bridge in the store first, +2. prove the missing precedence and inactive-selected-participant regressions at the store seam second, +3. seed both worker worktrees from that exact frozen parent tree, +4. let one worker tighten operator-surface behavior and contract coverage without touching parent-owned production files, +5. let one worker tighten active docs and only audit secondary docs if contradiction remains, +6. integrate accepted worker outputs in the parent checkout, +7. perform one exact validation stack centered on the `PLAN-09` seam, +8. close the run only if the authority contract, contract tests, shell suite, and repo gates are all green. + +## Hard Guards + +### Locked invariants + +1. Canonical session-root parent and participant records are the live-state authority boundary for operator surfaces. +2. Flat compatibility parent, participant, and lease files are bridge input/output only during cutover. +3. Legacy `handles/*.json` is last-resort compatibility input only and never outranks canonical or flat records. +4. Trace is historical fallback only for `status` gaps and never current-session toolbox authorization. +5. `substrate agent status` and `substrate agent toolbox ...` fail closed on ambiguity, corruption, broken parent/child linkage, missing selected participant, or inactive selected participant. +6. `persist_runtime_snapshots(...)` remains the caller choke point, but `PLAN-09` does not authorize worker-lane production edits in [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs). +7. No caller outside [state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) may directly write flat compatibility parent, participant, lease, or handle artifacts. +8. Dual-write ownership stays localized to store helpers in [state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs). +9. Bridge removal is gated and explicitly not part of this slice. +10. `PLAN-06` session-centric authority semantics and `PLAN-08` trace/event authority semantics are upstream constraints, not reopenable scope. +11. No new selector UX, cache file, registry, or transactional persistence layer is allowed. +12. Active docs must describe the same authority ladder as runtime behavior. +13. Validation commands must use the real package name in this repo: `cargo ... -p shell`. + +### File-level boundaries + +Parent-owned serialized production surfaces: + +- [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) + +Worker-safe post-freeze operator surface: + +- [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs) +- [crates/shell/tests/agent_successor_contract_ahcsitc0.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_successor_contract_ahcsitc0.rs) + +Worker-safe post-freeze docs surface: + +- [docs/TRACE.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/TRACE.md) +- [docs/USAGE.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/USAGE.md) + +Conditional docs-only audit surface: + +- [compatibility-spec.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/project_management/packs/draft/agent-hub-core-successor-identity-tuple-compatible/compatibility-spec.md) +- [manual_testing_playbook.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/project_management/packs/draft/agent-hub-core-successor-identity-tuple-compatible/manual_testing_playbook.md) + +Read-for-truth only: + +- [PLAN-09.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-09.md) +- [ORCH_PLAN-08.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/ORCH_PLAN-08.md) +- [ORCH_PLAN-06.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/ORCH_PLAN-06.md) + +### Non-negotiable stop conditions + +Stop the run and write `.runs/plan-09/blocked.json` if any of these occur: + +1. A task requires concurrent edits to `state_store.rs` and worker-lane files before the authority freeze lands. +2. A task restores flat compatibility files, legacy handles, or trace rows as live-state authority. +3. A task weakens fail-closed behavior into heuristic newest-session or trace-first recovery. +4. A task removes flat compatibility reads or writes instead of merely freezing their bridge posture. +5. A task spreads flat compatibility write ownership outside the store. +6. A worker lane needs to touch [state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) or [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) to complete its assignment. +7. A docs lane needs to touch test or production files to make its wording true. +8. A task introduces new selector UX, a new registry, or a new persistence layer. +9. A task leaves inactive selected-participant behavior ambiguous instead of fail-closed. + +## Orchestration State Surfaces + +### Canonical run state + +Single local source of truth for the run: + +- `.runs/plan-09/run-state.json` + +Parent-only writes to this file. It tracks: + +- current phase, +- active task IDs, +- branch and worktree assignment, +- gate status, +- frozen live-state authority ladder, +- frozen fail-closed operator-surface contract, +- frozen retained-bridge and bridge-removal-gate posture, +- accepted and rejected worker outputs, +- blocked or completed terminal state, +- final closeout pointer. + +If a worker report conflicts with `run-state.json`, the parent trusts `run-state.json` until it explicitly reconciles the discrepancy. + +### Derived run artifacts + +The parent may maintain these local artifacts: + +- `.runs/plan-09/queue.json` +- `.runs/plan-09/session.log` +- `.runs/plan-09/sentinels/task-m09-a1-preflight.ok` +- `.runs/plan-09/sentinels/task-m09-a2-store-authority-freeze.ok` +- `.runs/plan-09/sentinels/task-m09-b1-operator-contracts.ok` +- `.runs/plan-09/sentinels/task-m09-c1-authority-docs.ok` +- `.runs/plan-09/sentinels/task-m09-e1-integrate-and-validate.ok` +- `.runs/plan-09/sentinels/task-m09-e2-closeout.ok` +- `.runs/plan-09/blocked.json` +- `.runs/plan-09/closeout.md` + +Sentinel rules: + +1. `.ok` means the parent validated the task output and advanced the run. +2. Missing sentinel means the task is not accepted. +3. `blocked.json` is written only on blocked termination. +4. `closeout.md` is written only on successful completion. +5. Worker-generated notes never replace parent-written sentinels or run-state artifacts. + +## Concurrency Policy + +1. The parent is the only integrator. +2. The parent is the only writer of final branch state on `feat/session-centric-state-store`. +3. Exact active worker cap: `2`. +4. There are zero worker lanes during `task/m09-a1-preflight` and `task/m09-a2-store-authority-freeze`. +5. Only two child worktrees exist in this run because only two worker-safe late-phase lanes exist in this slice. +6. No worker may edit [state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) or [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs). +7. Worker lanes open only after the parent has stabilized the store authority diff and seeded each worktree from that exact state. +8. `task/m09-b1-operator-contracts` and `task/m09-c1-authority-docs` are the only parallel window in this run. +9. If `task/m09-b1` proves that the write-ownership drift guard requires production changes in `async_repl.rs`, the worker stops and hands the issue back to the parent instead of widening scope. +10. If `task/m09-c1` proves that a doc contradiction requires code or contract-test changes, the worker stops and hands the issue back to the parent instead of widening scope. + +## Approval And Gate Model + +There are no human approval gates defined for this run. + +Replacement control mechanism: + +1. parent validation gates, +2. parent-written sentinels, +3. `session.log` for acceptance and rejection rationale, +4. `blocked.json` for hard-stop termination, +5. `closeout.md` for successful completion. + +### Gate A: Scope And Topology Lock + +Required before implementation starts: + +- parent re-reads `PLAN-09`, `ORCH_PLAN-08`, and `ORCH_PLAN-06`, +- parent records the honest run shape: authority freeze first, operator/docs parallel second, final validation last, +- parent records the exact parent-owned files and the exact worker-safe files, +- parent records the package-name normalization: use `shell`, never `substrate-shell`. + +### Gate B: Authority Freeze + +Required before the worker window opens: + +- the store-owned authority ladder is frozen in the parent checkout, +- canonical-over-flat-over-legacy precedence is frozen, +- inactive selected-participant fail-closed behavior is frozen, +- retained-bridge and bridge-removal-gate posture is frozen, +- the parent has completed the pre-worker proof order defined below, +- both worker worktrees are seeded from the exact post-`task/m09-a2` tree. + +### Gate C: Final Acceptance + +Required before closeout: + +- both worker outputs are accepted or deliberately rejected and replaced by parent work, +- the final validation order passes, +- active docs reflect the final authority ladder and toolbox fail-closed posture, +- no caller-owned flat compatibility writes exist outside the store. + +## Workstream Plan + +### Worktree topology + +Parent checkout: + +- current checkout on `feat/session-centric-state-store` + +Child worktrees and branches: + +- `../substrate-m09-operator-contracts` + - `codex/feat-session-centric-state-store-m09-operator-contracts` +- `../substrate-m09-authority-docs` + - `codex/feat-session-centric-state-store-m09-authority-docs` + +Execution topology: + +1. The parent checkout serves every parent-only lane in this run. +2. Exactly two child worktrees exist because exactly two worker-safe tasks exist after Gate B. +3. No third child worktree is authorized because there is no third honest parallel seam in `PLAN-09`. + +Subagents do not merge each other’s work. They return patches, touched files, tests run, and blockers to the parent. + +### Task graph + +Execution graph for the run: + +1. `task/m09-a1-preflight` +2. `task/m09-a2-store-authority-freeze` +3. `task/m09-b1-operator-contracts` and `task/m09-c1-authority-docs` in parallel +4. `task/m09-e1-integrate-and-validate` +5. `task/m09-e2-closeout` + +Parent-only serialized tasks: + +- `task/m09-a1-preflight` +- `task/m09-a2-store-authority-freeze` +- `task/m09-e1-integrate-and-validate` +- `task/m09-e2-closeout` + +Worker-owned tasks: + +- `task/m09-b1-operator-contracts` +- `task/m09-c1-authority-docs` + +## Parallel Window BC + +This is the only worker window in the run. + +It opens only after Gate B passes and after the parent has completed the authority freeze in `task/m09-a2-store-authority-freeze`. There are zero honest production-code workers before that because [state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) owns the authority ladder that both operator surfaces and docs must consume. + +The parallel window is optimal rather than merely acceptable because the two late tasks are materially independent once the store contract is frozen: + +- the operator lane owns [agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs) plus the contract suite, +- the docs lane owns active authority wording plus contradiction-only secondary-doc audit, +- neither worker needs the other’s files to make progress, +- both workers can validate against the same frozen parent tree without touching parent-owned production seams. + +### task/m09-a1-preflight + +Ownership: + +- parent only + +Scope: + +1. Re-read [PLAN-09.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-09.md), [ORCH_PLAN-08.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/ORCH_PLAN-08.md), and [ORCH_PLAN-06.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/ORCH_PLAN-06.md). +2. Confirm the run executes from `feat/session-centric-state-store`. +3. Freeze the no-UI scope, exact `2`-worker cap, parent-only production boundaries, and retained-bridge posture into the run packet. +4. Initialize: + - `.runs/plan-09/run-state.json` + - `.runs/plan-09/queue.json` + - `.runs/plan-09/session.log` +5. Record the repo-truth validation rule that all package-targeted commands use `-p shell`. + +Acceptance: + +1. The parent can explain why `state_store.rs` must freeze before any worker lane opens. +2. The parent can restate the live-state authority ladder without ambiguity. +3. The parent can name the required missing regression work for this slice: + - legacy-handle precedence never outranking canonical or flat records, + - inactive selected participant fail-closed coverage, + - write-ownership drift guard proving no caller outside the store writes flat compatibility artifacts directly. +4. `run-state.json` records the initial phase and queue. + +Green-path output: + +- `.runs/plan-09/sentinels/task-m09-a1-preflight.ok` + +Blocked-path output: + +- `.runs/plan-09/blocked.json` + +### Parent validation gate A + +Required before `task/m09-a2-store-authority-freeze` starts: + +1. No invariant contradiction remains unresolved. +2. The parent can explain why bridge removal is explicitly deferred. +3. The parent can explain why only two child worktrees are needed for this run. + +### task/m09-a2-store-authority-freeze + +Ownership: + +- parent only + +Why serialized: + +- [state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) owns the precedence contract, torn-root posture, live-session selection, selected-participant linkage, and last-resort legacy-handle fallback. There is no honest throughput gain from splitting downstream work before those rules are frozen. + +Allowed files: + +- [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) + +Scope: + +1. Freeze the read order in `load_authoritative_session(...)` and adjacent participant/session readers: + - canonical parent, + - flat parent only if canonical parent is absent, + - canonical participant, + - flat participant only if canonical participant is absent, + - legacy handle alias last. +2. Keep `list_live_sessions()` and `resolve_single_live_session_for_agent(...)` strict about: + - active parent state, + - owner PID liveness, + - selected-participant presence, + - selected-participant activity, + - parent/child linkage. +3. Add or retain direct store-level regressions for: + - canonical participant beating conflicting legacy-handle fallback, + - flat participant beating conflicting legacy-handle fallback when the canonical child is absent, + - selected participant present but inactive failing closed. +4. Keep dual-write ownership localized to the store helpers. Do not move any write ownership into callers. +5. Do not touch `agents_cmd.rs`, `async_repl.rs`, docs, or contract tests yet. + +Must not do: + +1. No operator-surface tightening yet. +2. No docs edits yet. +3. No bridge removal or dual-write retirement. +4. No new helper layer, cache, or registry to “clarify” the authority ladder. + +Acceptance: + +1. The parent can point to one frozen store-owned authority ladder for live discovery. +2. Store-level tests prove canonical and flat participant records outrank legacy handles. +3. Store-level tests prove an inactive selected participant fails closed. +4. No authority decision depends on trace or caller-owned precedence logic. + +Green-path output: + +- `.runs/plan-09/sentinels/task-m09-a2-store-authority-freeze.ok` + +Blocked-path output: + +- `.runs/plan-09/blocked.json` + +### Parent validation gate B + +Required before the worker window opens: + +1. `cargo test -p shell agent_runtime::state_store -- --nocapture` passes. +2. `cargo test -p shell --test agent_successor_contract_ahcsitc0 -- --no-run` passes. +3. `cargo test -p shell --tests --no-run` passes from the stabilized parent checkout. +4. The parent records the frozen authority ladder, fail-closed operator contract, retained bridge posture, and bridge-removal gates in `run-state.json`. +5. The parent seeds both worker worktrees from the exact post-`task/m09-a2` tree. + +### task/m09-b1-operator-contracts + +Ownership: + +- worker-owned +- parent-reviewed + +Worktree: + +- `../substrate-m09-operator-contracts` +- `codex/feat-session-centric-state-store-m09-operator-contracts` + +Allowed files: + +- [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs) +- [crates/shell/tests/agent_successor_contract_ahcsitc0.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_successor_contract_ahcsitc0.rs) + +Scope: + +1. Keep `build_status_report(...)` live-first and trace-second: + - live session projection first, + - tombstone suppression next, + - trace fallback only for still-uncovered status tuples. +2. Keep `build_toolbox_status_report(...)` and `build_toolbox_env_report(...)` resolving through `resolve_single_live_session_for_agent(...)` with no local precedence or newest-session heuristics. +3. Add CLI-surface regression coverage for an active parent whose selected participant exists but is inactive. +4. Add a bounded write-ownership drift guard in the contract suite proving that no production caller outside the store writes flat compatibility parent, participant, lease, or handle artifacts directly. +5. Preserve current toolbox JSON fields and exit-code posture, especially exit `3` for `dependency_unavailable`. + +Must not do: + +1. No edits to [state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs). +2. No edits to [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs). +3. No docs edits. +4. No bridge removal or flat dual-write retirement. +5. No heuristic “latest session” or trace-only live-session recovery. +6. No new persistence owner outside the store. + +Commands: + +1. `cargo test -p shell --test agent_successor_contract_ahcsitc0 -- --nocapture` + +Execution packet for the worker: + +1. Read [PLAN-09.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-09.md). +2. Read [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs). +3. Read [crates/shell/tests/agent_successor_contract_ahcsitc0.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_successor_contract_ahcsitc0.rs). +4. Make only allowed-file changes. +5. Run the exact command listed above. +6. Stop immediately if the drift guard requires production edits in [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs). + +Acceptance: + +1. Contract coverage proves inactive selected-participant resolution fails closed across operator surfaces. +2. Contract coverage proves no caller outside the store owns direct flat compatibility writes. +3. No patch reintroduces trace-authorized toolbox state or direct caller-owned compatibility writes. +4. The worker report includes touched files, exact command run, and whether `async_repl.rs` follow-up is needed from the parent. +5. The lane stays inside its allowed files. + +Green-path output: + +- `.runs/plan-09/sentinels/task-m09-b1-operator-contracts.ok` + +Blocked-path output: + +- `.runs/plan-09/blocked.json` if the worker needs parent-owned production files + +### task/m09-c1-authority-docs + +Ownership: + +- worker-owned +- parent-reviewed + +Worktree: + +- `../substrate-m09-authority-docs` +- `codex/feat-session-centric-state-store-m09-authority-docs` + +Allowed files: + +- [docs/TRACE.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/TRACE.md) +- [docs/USAGE.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/USAGE.md) + +Conditional allowed files only if active docs would otherwise remain contradictory: + +- [compatibility-spec.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/project_management/packs/draft/agent-hub-core-successor-identity-tuple-compatible/compatibility-spec.md) +- [manual_testing_playbook.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/project_management/packs/draft/agent-hub-core-successor-identity-tuple-compatible/manual_testing_playbook.md) + +Scope: + +1. Keep one clear authority statement: canonical session-root parent plus participant records are the live-state authority boundary. +2. Keep one clear bridge statement: flat compatibility files are bridge input/output only during cutover. +3. Keep one clear legacy statement: `handles/*.json` remains last-resort compatibility input only. +4. Keep one clear trace statement: trace is historical fallback for `status` gaps only and never toolbox authorization. +5. Keep one clear toolbox statement: `substrate agent toolbox env` emits variables only for a current live host-scoped orchestrator session and otherwise fails closed with exit `3`. +6. Touch secondary docs only if they contradict the active runtime contract, and then only change the contradictory sentence or bullet. + +Must not do: + +1. No edits to code or tests. +2. No bridge-removal edits disguised as doc cleanup. +3. No assertion changes in contract suites from this lane. + +Commands: + +1. `cargo test -p shell --test agent_successor_contract_ahcsitc0 -- --nocapture` + +Execution packet for the worker: + +1. Read [PLAN-09.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-09.md). +2. Read [docs/TRACE.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/TRACE.md). +3. Read [docs/USAGE.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/USAGE.md). +4. Audit the two secondary docs only if active-doc contradictions remain after the primary edits. +5. Make only allowed-file changes. +6. Run the exact command listed above. +7. Stop immediately if any required fix would need code or contract-test edits. + +Acceptance: + +1. `TRACE.md` and `USAGE.md` describe the same authority ladder as the runtime. +2. No active doc implies that flat compatibility files, legacy handles, or trace history are current truth. +3. Any secondary-doc edits are minimal and contradiction-driven. +4. The worker report includes touched files, exact command run, and whether any unresolved contradiction remains outside docs. +5. The lane stays inside its allowed files. + +Green-path output: + +- `.runs/plan-09/sentinels/task-m09-c1-authority-docs.ok` + +Blocked-path output: + +- `.runs/plan-09/blocked.json` if the worker needs code or test edits + +### task/m09-e1-integrate-and-validate + +Ownership: + +- parent only + +Scope: + +1. Review both worker outputs against the frozen authority and fail-closed contracts. +2. Reject any worker patch that touches unowned files, assumes stale authority, or omits test evidence. +3. Integrate accepted worker outputs into the parent checkout. +4. If the operator worker proves that the write-ownership drift guard requires bounded parent-owned production changes in [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs), make those changes in the parent checkout and rerun the affected validation steps. +5. Run the final validation stack in the exact order defined below. + +Acceptance: + +1. Every accepted worker output has a matching parent-written sentinel. +2. No flat compatibility or legacy-handle source outranks canonical session-root authority. +3. No operator surface authorizes current liveness from trace. +4. No direct caller-owned flat compatibility writes exist outside the store. +5. `run-state.json` records accepted and rejected outputs explicitly. + +Green-path output: + +- `.runs/plan-09/sentinels/task-m09-e1-integrate-and-validate.ok` + +Blocked-path output: + +- `.runs/plan-09/blocked.json` + +### task/m09-e2-closeout + +Ownership: + +- parent only + +Scope: + +1. Mark the run complete in `.runs/plan-09/run-state.json`. +2. Write `.runs/plan-09/closeout.md` with: + - accepted tasks, + - integrated worktrees and branches, + - validation commands and outcomes, + - retained bridge behavior and explicit removal gates, + - confirmation that live-state authority remains canonical session-root parent plus participant records, + - confirmation that no caller outside the store writes flat compatibility artifacts directly. +3. Append the final acceptance rationale to `.runs/plan-09/session.log`. + +Acceptance: + +1. `closeout.md` is present and matches the final branch state. +2. All sentinels through `task/m09-e2-closeout` exist. +3. No blocked-path artifact exists for a successful run. + +Green-path output: + +- `.runs/plan-09/sentinels/task-m09-e2-closeout.ok` + +## Context-Control Rules + +1. The parent owns `.runs/plan-09/*`. Workers do not edit run-state, sentinels, queue, session log, blocked state, or closeout artifacts. +2. Worker packets include only: + - the task ID, + - allowed files, + - frozen invariants, + - stop conditions, + - exact commands to run. +3. Workers must stop immediately if they need to touch any file outside their allowed list. +4. Workers report: + - touched files, + - commands run, + - unresolved assumptions, + - blockers, + - whether they observed any stale authority wording, trace-authority assumptions, or write-ownership drift that requires parent-owned follow-up. +5. Workers do not rebase, merge, or integrate each other’s work. +6. The parent rejects any worker patch that silently broadens scope or makes authority decisions not already frozen in `run-state.json`. +7. If worker outputs conflict with current parent truth, the parent re-derives the correct result from production code and rewrites the patch locally instead of negotiating a blended state across worktrees. + +## Tests And Acceptance + +### Pre-worker proof order + +Run these parent-owned checks before dispatching the worker window: + +1. `cargo test -p shell agent_runtime::state_store -- --nocapture` +2. `cargo test -p shell --test agent_successor_contract_ahcsitc0 -- --no-run` +3. `cargo test -p shell --tests --no-run` + +### Final validation order + +Run these commands in this exact order during `task/m09-e1-integrate-and-validate`: + +1. `cargo test -p shell agent_runtime::state_store -- --nocapture` +2. `cargo test -p shell --test agent_successor_contract_ahcsitc0 -- --nocapture` +3. `cargo test -p shell -- --nocapture` +4. `cargo fmt --all -- --check` +5. `cargo clippy -p shell --all-targets -- -D warnings` +6. `cargo clippy --workspace --all-targets -- -D warnings` +7. `cargo test --workspace -- --nocapture` + +### Acceptance checklist + +The slice is accepted only if all of these are true: + +1. Canonical session-root parent and participant records are the implemented and documented live-state authority. +2. Flat compatibility parent, participant, and lease files remain bridge input/output only during cutover. +3. Legacy `handles/*.json` remains documented and tested as last-resort compatibility input only. +4. Canonical or flat participant records never lose precedence to legacy-handle fallback. +5. `status` uses live state first and trace only as bounded historical fill. +6. `toolbox status|env` never authorize current liveness from trace and fail closed on ambiguity or broken linkage. +7. An inactive selected participant fails closed at both store and operator-surface levels. +8. No caller-owned flat compatibility write path exists outside the store. +9. Active docs reflect the current authority ladder and retained bridge posture. +10. Bridge removal remains deferred behind explicit later gates. + +## Merge Refusal Rules + +The parent refuses to merge a worker output if any of these are true: + +1. The patch edits a file outside the task’s allowed file list. +2. The patch assumes flat compatibility files, legacy handles, or trace rows are authoritative for live discovery. +3. The patch weakens fail-closed behavior for ambiguity, broken linkage, or inactive selected participants. +4. The patch removes bridge behavior or dual-write without a separate gated slice. +5. The patch requires concurrent parent edits to `state_store.rs` or `async_repl.rs` to become intelligible. +6. The patch omits test evidence for the behavior it claims to cover. +7. The patch leaves docs or fixtures contradicting the final runtime contract. +8. The patch introduces or preserves direct flat compatibility writes outside the store. + +## Run Exit Criteria + +### Successful run + +The run is successful only if all of these are true: + +1. `.runs/plan-09/sentinels/task-m09-a1-preflight.ok` exists. +2. `.runs/plan-09/sentinels/task-m09-a2-store-authority-freeze.ok` exists. +3. `.runs/plan-09/sentinels/task-m09-b1-operator-contracts.ok` exists. +4. `.runs/plan-09/sentinels/task-m09-c1-authority-docs.ok` exists. +5. `.runs/plan-09/sentinels/task-m09-e1-integrate-and-validate.ok` exists. +6. `.runs/plan-09/sentinels/task-m09-e2-closeout.ok` exists. +7. `.runs/plan-09/run-state.json` exists and records a completed terminal state. +8. `.runs/plan-09/queue.json` and `.runs/plan-09/session.log` exist. +9. `.runs/plan-09/closeout.md` exists and matches the final accepted branch state. +10. `.runs/plan-09/blocked.json` does not exist. + +### Blocked termination + +The run terminates blocked only if all of these are true: + +1. `.runs/plan-09/blocked.json` exists and records the blocking reason. +2. `.runs/plan-09/run-state.json` exists and records `blocked` as the terminal state. +3. `session.log` contains the parent rationale for the stop. +4. No downstream task sentinel may be written after the blocking point. +5. `.runs/plan-09/sentinels/task-m09-e2-closeout.ok` must not exist. +6. `.runs/plan-09/closeout.md` must not exist. + +## Closeout + +Successful closeout records: + +1. the final frozen live-state authority ladder that shipped, +2. the final fail-closed operator-surface contract, +3. the exact validation order and outcomes, +4. confirmation that flat compatibility files remain bridge input/output only, +5. confirmation that legacy handles remain last-resort compatibility input only, +6. confirmation that no caller outside the store writes flat compatibility artifacts directly, +7. confirmation that active docs reflect the final authority boundary. + +## Assumptions + +1. The parent run starts from the current branch baseline `feat/session-centric-state-store`. +2. `PLAN-06` and `PLAN-08` already define upstream runtime and trace constraints that this slice must preserve. +3. The Rust package name `shell` is authoritative for validation commands in this repository. +4. The bridge around flat compatibility files remains intentionally present during this slice. +5. Any secondary doc drift can be corrected with minimal docs-only edits and does not require reopening runtime authority design. diff --git a/llm-last-mile/ORCH_PLAN-10.md b/llm-last-mile/ORCH_PLAN-10.md new file mode 100644 index 000000000..d552ac925 --- /dev/null +++ b/llm-last-mile/ORCH_PLAN-10.md @@ -0,0 +1,985 @@ +# ORCH_PLAN-10: Production World-Scoped Member Runtime Launch + +Branch: `feat/session-centric-state-store` +Plan source: [PLAN-10.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-10.md) +Reference style source: [ORCH_PLAN-09.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/ORCH_PLAN-09.md) +Execution type: shell/runtime launch orchestration plan, no UI scope, strong DX/runtime/status/trace scope + +## Summary + +This run executes `PLAN-10` on `feat/session-centric-state-store` with an exact active worker cap of +`2`. The parent remains the only integrator, the only final branch writer, and the only agent +allowed to mutate the production ownership seam in +[crates/shell/src/execution/agent_runtime/validator.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/validator.rs), +[crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs), +and +[crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs). +The canonical run-state source of truth is `.runs/plan-10/run-state.json`. + +This slice is not a scheduler, not a multi-member selector product, not a new persistence layer, +and not a docs-first cleanup run. It is one bounded production seam: + +1. freeze the shared world-member selection contract in `validator.rs`, +2. align `agent doctor` to that same contract in `agents_cmd.rs`, +3. land lazy world-backed member launch in the REPL command branch, +4. land restart replacement on that same retained-control seam, +5. close the runtime/status/trace test wall, +6. touch docs only if the landed runtime behavior actually changed wording. + +The honest concurrency shape is narrower than the abstract lane count in `PLAN-10`. +`PLAN-10` is explicit about implementation order: validator first, doctor alignment as required +production work, lazy launch next, restart replacement on that same seam, then the test wall, then +optional docs. That means the parent owns all production-code steps serially through restart +replacement. The only real worker window opens after that seam is frozen: + +1. parent-only preflight, +2. parent-only validator contract, +3. parent-only doctor alignment, +4. parent-only lazy launch seam, +5. parent-only restart replacement, +6. one worker lane for operator/trace contract tests, +7. one worker lane for world-routing/restart integration tests, +8. parent-only integration, final validation, optional docs review, and closeout. + +This is deliberate. `validator.rs`, `agents_cmd.rs`, and `async_repl.rs` together define one +selection-and-launch contract. Parallelizing those production edits would create selection drift, +error-posture drift, and merge churn around the same retained-control seam. After the runtime seam +lands, the test wall becomes the only honest place to split work without forking production truth. + +Worker-model execution policy for this run: + +1. The parent remains the only integrator and the only final branch writer. +2. Every child worker uses `GPT-5.4` with `reasoning_effort=high`. +3. The active worker cap remains exactly `2`. +4. Worker coordination uses sentinels and long waits. Tight polling loops are not allowed. + +## Hard Guards + +### Locked invariants + +1. V1 member selection means: zero eligible world members preserves host-only behavior, exactly one + eligible world member is required for launch, and more than one eligible world member fails + closed. +2. Doctor and runtime launch must share the same selection truth. No boolean "any world member + exists" shortcut is allowed once `PLAN-10` lands. +3. The first production caller stays in the world-backed command branch in + [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) + after `ensure_no_policy_drift(...)` and before `exec_world_pty(...)` or `exec_world_line(...)`. +4. A new member participant is persisted in `allocating` first and does not become + authoritative-live until retained UAA control, active event stream, and completion observer are + all present. +5. Restart replacement must reuse the same retained-control lifecycle seam and + `new_replacement_participant(...)`. No second lifecycle definition is allowed. +6. `substrate agent status --json` must surface the live member from runtime state; stale trace + rows must never resurrect current liveness. +7. `substrate agent toolbox status|env` remains orchestrator-scoped for this slice even when a + world member is live. +8. The store remains the only persistence owner. No new direct caller-owned canonical, + compatibility, lease, or handle writes are allowed. +9. `registry.rs`, `state_store.rs`, and `world_gateway.rs` remain default no-change surfaces. If a + real launch gap forces edits there, the parent owns them and must record the reason in + `run-state.json` before touching the file. +10. No selector UX, member scheduler, gateway cache, runtime manager, or public `/v1/agents` + surface is allowed in this run. +11. The test wall must be green before any docs edits are considered. +12. Package-targeted cargo commands use the real package name in this repo: `-p shell`. +13. The parent is the only writer of `.runs/plan-10/*`. + +### File-level boundaries + +Parent-owned serialized production surfaces: + +- [crates/shell/src/execution/agent_runtime/validator.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/validator.rs) +- [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs) +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) + +Parent-owned escalation-only production surfaces: + +- [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) +- [crates/shell/src/execution/agent_runtime/registry.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/registry.rs) +- [crates/shell/src/builtins/world_gateway.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/builtins/world_gateway.rs) + +Worker-safe test surface after the runtime seam is frozen: + +- [crates/shell/tests/agent_successor_contract_ahcsitc0.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_successor_contract_ahcsitc0.rs) +- [crates/shell/tests/agent_hub_trace_persistence.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_hub_trace_persistence.rs) +- [crates/shell/tests/repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs) + +Optional parent-owned docs surface only after the test wall is green: + +- [docs/TRACE.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/TRACE.md) +- [docs/USAGE.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/USAGE.md) + +Read-for-truth only: + +- [PLAN-10.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-10.md) +- [ORCH_PLAN-09.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/ORCH_PLAN-09.md) + +### Non-negotiable stop conditions + +Stop the run and write `.runs/plan-10/blocked.json` if any of these occur: + +1. A task introduces a second member-selection rule or preserves a doctor/runtime mismatch. +2. A task guesses among multiple world-member candidates instead of failing closed. +3. A task falls back to host execution after a hard member-launch preflight failure other than the + explicit zero-eligible-member no-op case. +4. A task advertises a member live before retained UAA control and the full liveness boundary are + proven. +5. A task reopens store ownership by writing canonical or compatibility state directly from the + REPL or doctor path. +6. A worker lane needs to touch `validator.rs`, `agents_cmd.rs`, `async_repl.rs`, `state_store.rs`, + `registry.rs`, or `world_gateway.rs` to complete its assignment. +7. A task lets stale generation state regain liveness through trace fallback after restart. +8. A task requires docs edits to explain behavior that the runtime and tests still do not prove. +9. A task broadens scope into scheduler, selector UX, or multi-member control-plane work. + +## Orchestration State Surfaces + +### Canonical run state + +The only canonical source of truth for run orchestration state: + +- `.runs/plan-10/run-state.json` + +Parent-only writes to this file. It tracks: + +- current phase, +- active task IDs, +- branch and worktree assignment, +- gate status, +- frozen selection contract, +- frozen doctor/runtime alignment contract, +- frozen lazy-launch and restart-replacement contract, +- accepted and rejected worker outputs, +- blocked or completed terminal state, +- final closeout pointer. + +If a worker report conflicts with `run-state.json`, the parent treats `run-state.json` as +authoritative and ignores the worker report until the parent explicitly reconciles the discrepancy. + +### Derived run artifacts + +The parent maintains these local artifacts: + +- `.runs/plan-10/queue.json` +- `.runs/plan-10/session.log` +- `.runs/plan-10/sentinels/task-m10-a1-preflight.ok` +- `.runs/plan-10/sentinels/task-m10-a2-validator-contract.ok` +- `.runs/plan-10/sentinels/task-m10-a3-doctor-alignment.ok` +- `.runs/plan-10/sentinels/task-m10-a4-member-launch-seam.ok` +- `.runs/plan-10/sentinels/task-m10-a5-restart-replacement.ok` +- `.runs/plan-10/sentinels/task-m10-b1-runtime-contract-tests.ok` +- `.runs/plan-10/sentinels/task-m10-b2-world-routing-restart-tests.ok` +- `.runs/plan-10/sentinels/task-m10-e1-integrate-and-validate.ok` +- `.runs/plan-10/sentinels/task-m10-e2-docs-review.ok` +- `.runs/plan-10/sentinels/task-m10-e3-closeout.ok` +- `.runs/plan-10/blocked.json` +- `.runs/plan-10/closeout.md` + +Sentinel rules: + +1. `.ok` means the parent validated the task output and advanced the run. +2. Missing sentinel means the task is not accepted. +3. `blocked.json` is written only on blocked termination. +4. `closeout.md` is written only on successful completion. +5. Worker-generated notes never replace parent-written sentinels or run-state artifacts. + +## Concurrency Policy + +1. The parent is the only integrator. +2. The parent is the only writer of final branch state on `feat/session-centric-state-store`. +3. Exact active worker cap: `2`. +4. There are zero worker lanes during `task/m10-a1` through `task/m10-a5`. +5. The only honest parallel window is the test wall after the shared launch seam is frozen. +6. No worker may edit `validator.rs`, `agents_cmd.rs`, `async_repl.rs`, `state_store.rs`, + `registry.rs`, `world_gateway.rs`, or docs. +7. The parent seeds both child worktrees from the exact post-`task/m10-a5` tree. +8. `task/m10-b1` and `task/m10-b2` are the only parallel tasks in this run. +9. Optional docs stay parent-owned and sequential because they depend on the final accepted + runtime behavior rather than generating independent throughput. +10. If a worker proves a missing production change is required, the worker stops and hands the + issue back to the parent instead of widening scope. +11. Worker coordination uses sentinel files and long waits. Tight polling against run-state or + branch state is forbidden. + +## Approval And Gate Model + +There are no human approval gates defined for this run. + +Replacement control mechanism: + +1. parent validation gates, +2. parent-written sentinels, +3. `session.log` for acceptance and rejection rationale, +4. `blocked.json` for hard-stop termination, +5. `closeout.md` for successful completion. + +### Gate A: Scope And Topology Lock + +Required before implementation starts: + +- parent re-reads `PLAN-10.md` and `ORCH_PLAN-09.md`, +- parent records the exact serialized production order: validator, doctor, lazy launch, restart + replacement, then the test wall, +- parent records the exact `2`-worker cap and why the worker window opens only after + `task/m10-a5`, +- parent records parent-owned and worker-safe files, +- parent records the package-name normalization: use `shell`, never `substrate-shell`. + +### Gate B: Shared Selection Freeze + +Required before doctor alignment proceeds: + +- `validate_member_selection(...)` is frozen in `validator.rs`, +- zero vs one vs many eligible world-member outcomes are explicit and tested, +- helper wording is neutral enough for both doctor and runtime call sites, +- no REPL or doctor behavior change ships through a private selection rule. + +### Gate C: Runtime Seam Freeze + +Required before the worker window opens: + +- doctor and runtime share one frozen selection contract, +- the REPL world-backed command branch owns lazy launch before `exec_world_pty(...)` / + `exec_world_line(...)`, +- restart replacement is frozen on that same retained-control seam, +- any escalation into `state_store.rs` or `registry.rs` is already integrated and justified, +- the parent has completed the pre-worker proof order defined below, +- both worker worktrees are seeded from the exact post-`task/m10-a5` tree. + +### Gate D: Final Acceptance + +Required before closeout: + +- both worker outputs are accepted or explicitly rejected and replaced by parent work, +- the final validation order passes, +- docs review is complete and either produced minimal aligned edits or an explicit no-change + decision, +- `run-state.json` records the final selection, doctor, launch, restart, and test contracts. + +## Workstream Plan + +### Worktree topology + +Parent checkout: + +- current checkout on `feat/session-centric-state-store` + +Child worktrees and branches: + +- `../substrate-m10-runtime-contract-tests` + - `codex/feat-session-centric-state-store-m10-runtime-contract-tests` +- `../substrate-m10-world-routing-restart` + - `codex/feat-session-centric-state-store-m10-world-routing-restart` + +Suggested creation commands: + +```bash +git worktree add ../substrate-m10-runtime-contract-tests \ + -b codex/feat-session-centric-state-store-m10-runtime-contract-tests \ + feat/session-centric-state-store +git worktree add ../substrate-m10-world-routing-restart \ + -b codex/feat-session-centric-state-store-m10-world-routing-restart \ + feat/session-centric-state-store +``` + +Execution topology: + +1. The parent checkout serves every production task and every final integration task. +2. Exactly two child worktrees exist because exactly two worker-safe late test tasks exist in this + slice. +3. No third child worktree is authorized because no earlier production seam can be parallelized + honestly without forking selection or retained-control truth. + +Subagents do not merge each other’s work. They return only changed files, exact commands run, and +blockers to the parent. + +### Task graph + +Execution graph for the run: + +1. `task/m10-a1-preflight` +2. `task/m10-a2-validator-contract` +3. `task/m10-a3-doctor-alignment` +4. `task/m10-a4-member-launch-seam` +5. `task/m10-a5-restart-replacement` +6. `task/m10-b1-runtime-contract-tests` and `task/m10-b2-world-routing-restart-tests` in parallel +7. `task/m10-e1-integrate-and-validate` +8. `task/m10-e2-docs-review` +9. `task/m10-e3-closeout` + +Parent-only serialized tasks: + +- `task/m10-a1-preflight` +- `task/m10-a2-validator-contract` +- `task/m10-a3-doctor-alignment` +- `task/m10-a4-member-launch-seam` +- `task/m10-a5-restart-replacement` +- `task/m10-e1-integrate-and-validate` +- `task/m10-e2-docs-review` +- `task/m10-e3-closeout` + +Worker-owned tasks: + +- `task/m10-b1-runtime-contract-tests` +- `task/m10-b2-world-routing-restart-tests` + +## Parallel Window B + +This is the only worker window in the run. + +It opens only after Gate C passes and after the parent has completed the production launch seam in +`task/m10-a5-restart-replacement`. There is no earlier honest worker window because: + +1. `validator.rs` freezes the selection contract that both doctor and runtime must consume, +2. `PLAN-10` requires doctor alignment as required production work before the slice is considered + coherent, +3. `async_repl.rs` owns both lazy launch and restart replacement on the same retained-control seam, +4. splitting those production edits across worktrees would create contract drift faster than it + creates throughput. + +The test wall does parallelize cleanly after the runtime seam lands: + +- `task/m10-b1` covers operator/trace contract suites in files disjoint from the restart-routing + integration file, +- `task/m10-b2` covers the world-first routing and restart integration suite, +- both workers validate against the same frozen production tree, +- both workers must stop if their assertions require parent-owned production edits. + +### task/m10-a1-preflight + +Ownership: + +- parent only + +Scope: + +1. Re-read [PLAN-10.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-10.md) + and [ORCH_PLAN-09.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/ORCH_PLAN-09.md). +2. Confirm the run executes from `feat/session-centric-state-store`. +3. Freeze the no-UI scope, exact `2`-worker cap, serialized production order, and parent-only + production ownership into the run packet. +4. Initialize: + - `.runs/plan-10/run-state.json` + - `.runs/plan-10/queue.json` + - `.runs/plan-10/session.log` +5. Record the repo-truth validation rule that package-targeted commands use `-p shell`. + +Commands: + +1. `git branch --show-current` +2. `mkdir -p .runs/plan-10/sentinels` + +Acceptance: + +1. The parent can explain why production work stays serial through restart replacement. +2. The parent can restate the v1 selection rule without ambiguity. +3. The parent can name the only honest worker window in the slice: the test wall after + `task/m10-a5`. +4. `run-state.json` records the initial phase and queue. + +Green-path output: + +- `.runs/plan-10/sentinels/task-m10-a1-preflight.ok` + +Blocked-path output: + +- `.runs/plan-10/blocked.json` + +### Parent validation gate A + +Required before `task/m10-a2-validator-contract` starts: + +1. No invariant contradiction remains unresolved. +2. The parent can explain why zero eligible world members is the only case that preserves host-only + behavior. +3. The parent can explain why docs are explicitly deferred until after the test wall. + +### task/m10-a2-validator-contract + +Ownership: + +- parent only + +Allowed files: + +- [crates/shell/src/execution/agent_runtime/validator.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/validator.rs) + +Scope: + +1. Land the shared member-selection contract in `validator.rs`. +2. Add `validate_member_selection(...)` returning a generic `RuntimeSelectionDescriptor`. +3. Make zero, one, and many eligible world-member outcomes explicit. +4. Reuse `validate_runtime_realizability(...)` through neutral wording rather than forking a second + realizability contract. +5. Add unit coverage proving the selection contract without changing doctor or REPL behavior yet. + +Must not do: + +1. No edits to `agents_cmd.rs`. +2. No edits to `async_repl.rs`. +3. No launch ownership in the store. +4. No selector UX, config surface, or heuristic fallback. + +Commands: + +1. `cargo test -p shell --lib -- --nocapture` +2. `cargo test -p shell --tests --no-run` + +Acceptance: + +1. The repo has one shared member-selection truth. +2. Zero, one, and many eligible world-member cases are tested. +3. Runtime descriptor shape stays generic. +4. Doctor and REPL behavior remain unchanged until later tasks. + +Green-path output: + +- `.runs/plan-10/sentinels/task-m10-a2-validator-contract.ok` + +Blocked-path output: + +- `.runs/plan-10/blocked.json` + +### Parent validation gate B + +Required before `task/m10-a3-doctor-alignment` starts: + +1. `cargo test -p shell --lib -- --nocapture` passes. +2. The parent records the frozen zero/one/many selection contract in `run-state.json`. +3. No downstream task is allowed to fork its own selection logic. + +### task/m10-a3-doctor-alignment + +Ownership: + +- parent only + +Allowed files: + +- [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs) + +Scope: + +1. Align `build_doctor_report(...)` to the same selection contract as runtime launch preflight. +2. Preserve host-only success when zero eligible world members exist. +3. Fail closed on ambiguous world-member selection before any world-member boundary claim is made. +4. Keep allowlist and world-boundary checks scoped to the unique selected member path. +5. Do not redesign `status` or `toolbox` here; this task is doctor alignment only. + +Must not do: + +1. No boolean "any world member exists" launch truth. +2. No new selection logic outside the helper contract already landed in `task/m10-a2`. +3. No edits to `async_repl.rs`. +4. No docs or test-wall edits yet. + +Commands: + +1. `cargo test -p shell --lib -- --nocapture` +2. `cargo test -p shell --test agent_successor_contract_ahcsitc0 -- --no-run` + +Acceptance: + +1. Doctor and runtime share one selection truth. +2. `agent doctor` can distinguish zero vs one vs many eligible world-member candidates. +3. Ambiguous selection fails closed. +4. No runtime-launch logic has landed yet. + +Green-path output: + +- `.runs/plan-10/sentinels/task-m10-a3-doctor-alignment.ok` + +Blocked-path output: + +- `.runs/plan-10/blocked.json` + +### task/m10-a4-member-launch-seam + +Ownership: + +- parent only + +Allowed files: + +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) + +Conditional parent-only escalation files only if a concrete launch gap forces it: + +- [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) +- [crates/shell/src/execution/agent_runtime/registry.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/registry.rs) + +Scope: + +1. Add lazy `ensure_member_runtime_ready(...)` in the world-backed command branch after + `ensure_no_policy_drift(...)` and before `exec_world_pty(...)` / `exec_world_line(...)`. +2. Reuse the shared selection contract and authoritative parent world binding from + `RuntimeOrchestrationContext`. +3. Reuse `AsyncReplAgentRuntime` and retained-control lifecycle machinery rather than copying a + second member lifecycle. +4. Persist the member in `allocating` first and advertise it live only after retained UAA control, + active event stream, and completion observer all exist. +5. Add bounded inline or local runtime tests in `async_repl.rs` proving launch-state progression. + +Must not do: + +1. No startup-time auto-launch. +2. No second member-only runtime holder unless reuse proves impossible and the run is blocked. +3. No direct writes to canonical or compatibility state outside store helpers. +4. No restart replacement yet beyond bounded scaffolding required by the same seam. + +Commands: + +1. `cargo test -p shell async_repl -- --nocapture` +2. `cargo test -p shell --test repl_world_first_routing_v1 -- --no-run` + +Acceptance: + +1. The first world-backed command can lazily launch the selected member. +2. The second command on the same generation can reuse the live member. +3. Failed preflight exits without a half-live participant. +4. Any escalation into `state_store.rs` or `registry.rs` is justified and recorded. + +Green-path output: + +- `.runs/plan-10/sentinels/task-m10-a4-member-launch-seam.ok` + +Blocked-path output: + +- `.runs/plan-10/blocked.json` + +### task/m10-a5-restart-replacement + +Ownership: + +- parent only + +Allowed files: + +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) + +Conditional parent-only escalation files only if a concrete seam gap forces it: + +- [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) +- [crates/shell/src/execution/agent_runtime/registry.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/registry.rs) + +Scope: + +1. Land restart replacement on the same retained-control seam as lazy launch. +2. After authoritative world binding persists and stale members are invalidated, either launch a + replacement participant for the new generation or leave honest absence. +3. Use `new_replacement_participant(...)` with fresh `participant_id` and + `resumed_from_participant_id=`. +4. Keep stale generation state invalidated and non-live even when terminal trace rows persist. +5. Finalize bounded inline/runtime tests around replacement lifecycle in `async_repl.rs`. + +Must not do: + +1. No second restart-specific lifecycle definition. +2. No trace-authorized resurrection of stale members. +3. No worker dispatch before restart replacement is frozen. +4. No docs edits. + +Commands: + +1. `cargo test -p shell async_repl -- --nocapture` +2. `cargo test -p shell --test repl_world_first_routing_v1 -- --no-run` +3. `cargo test -p shell --test agent_hub_trace_persistence -- --no-run` + +Acceptance: + +1. Restart with a live member yields either a live replacement on the new generation or a clear + failed replacement outcome. +2. Stale generation never appears live again. +3. The retained-control seam still defines liveness for both first launch and replacement. + +Green-path output: + +- `.runs/plan-10/sentinels/task-m10-a5-restart-replacement.ok` + +Blocked-path output: + +- `.runs/plan-10/blocked.json` + +### Parent validation gate C + +Required before the worker window opens: + +1. `cargo test -p shell async_repl -- --nocapture` passes. +2. `cargo test -p shell --test agent_successor_contract_ahcsitc0 -- --no-run` passes. +3. `cargo test -p shell --test repl_world_first_routing_v1 -- --no-run` passes. +4. `cargo test -p shell --test agent_hub_trace_persistence -- --no-run` passes. +5. `run-state.json` records the frozen selection, doctor, lazy-launch, and restart-replacement + contracts. +6. The parent seeds both worker worktrees from the exact post-`task/m10-a5` tree. + +### task/m10-b1-runtime-contract-tests + +Ownership: + +- worker-owned +- parent-reviewed + +Worktree: + +- `../substrate-m10-runtime-contract-tests` +- `codex/feat-session-centric-state-store-m10-runtime-contract-tests` + +Allowed files: + +- [crates/shell/tests/agent_successor_contract_ahcsitc0.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_successor_contract_ahcsitc0.rs) +- [crates/shell/tests/agent_hub_trace_persistence.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_hub_trace_persistence.rs) + +Scope: + +1. Add doctor contract cases for zero vs one vs many world-member candidates, especially + ambiguous-selection fail-closed behavior. +2. Add `agent status --json` contract coverage proving the live launched member comes from runtime + state with top-level `world_id` and `world_generation`. +3. Add contract coverage proving `toolbox status|env` stays orchestrator-anchored even when the + member is live. +4. Add trace persistence coverage for member `Registered`, `Status`, and terminal events plus + replacement lineage fields. +5. Keep stale terminal rows auditable without making them live again. + +Must not do: + +1. No edits to `validator.rs`, `agents_cmd.rs`, or `async_repl.rs`. +2. No docs edits. +3. No fresh selection logic in tests that the production seam does not own. +4. No weakening of orchestrator-only toolbox anchoring. + +Commands: + +1. `cargo test -p shell --test agent_successor_contract_ahcsitc0 -- --nocapture` +2. `cargo test -p shell --test agent_hub_trace_persistence -- --nocapture` + +Execution packet for the worker: + +1. Read [PLAN-10.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-10.md). +2. Read [crates/shell/tests/agent_successor_contract_ahcsitc0.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_successor_contract_ahcsitc0.rs). +3. Read [crates/shell/tests/agent_hub_trace_persistence.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_hub_trace_persistence.rs). +4. Make only allowed-file changes. +5. Run the exact commands listed above. +6. Stop immediately if the assertions require production edits in parent-owned files. + +Acceptance: + +1. Contract coverage proves doctor, status, toolbox, and trace all consume the frozen launch truth. +2. Trace lineage stays auditable without becoming current liveness. +3. The worker report includes only changed files, exact commands run, and blockers. +4. The lane stays inside its allowed files. + +Green-path output: + +- `.runs/plan-10/sentinels/task-m10-b1-runtime-contract-tests.ok` + +Blocked-path output: + +- `.runs/plan-10/blocked.json` if the worker needs parent-owned production files + +### task/m10-b2-world-routing-restart-tests + +Ownership: + +- worker-owned +- parent-reviewed + +Worktree: + +- `../substrate-m10-world-routing-restart` +- `codex/feat-session-centric-state-store-m10-world-routing-restart` + +Allowed files: + +- [crates/shell/tests/repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs) + +Scope: + +1. Add integration coverage for first world-backed command causing lazy member launch. +2. Add coverage for same-generation reuse on a second command. +3. Add restart integration coverage proving a live replacement member can come up on the new + generation. +4. Add replacement-failure coverage proving honest absence instead of stale-liveness resurrection. +5. Keep the integration suite anchored to the frozen parent-owned production seam rather than + inventing new hooks. + +Must not do: + +1. No edits to `async_repl.rs`. +2. No edits to any other test file. +3. No docs edits. +4. No workarounds that bypass the real world-backed command branch. + +Commands: + +1. `cargo test -p shell --test repl_world_first_routing_v1 -- --nocapture` + +Execution packet for the worker: + +1. Read [PLAN-10.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-10.md). +2. Read [crates/shell/tests/repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs). +3. Make only allowed-file changes. +4. Run the exact command listed above. +5. Stop immediately if the assertions require production edits in parent-owned files. + +Acceptance: + +1. The integration suite proves lazy launch, same-generation reuse, restart replacement, and + honest absence on replacement failure. +2. The worker report includes only changed files, exact commands run, and blockers. +3. The lane stays inside its allowed file. + +Green-path output: + +- `.runs/plan-10/sentinels/task-m10-b2-world-routing-restart-tests.ok` + +Blocked-path output: + +- `.runs/plan-10/blocked.json` if the worker needs parent-owned production files + +### task/m10-e1-integrate-and-validate + +Ownership: + +- parent only + +Scope: + +1. Review both worker outputs against the frozen selection, doctor, launch, and restart + contracts. +2. Reject any worker patch that touches unowned files, assumes stale trace authority, or omits test + evidence. +3. Integrate accepted worker outputs into the parent checkout. +4. If a worker proves bounded production fallout is required, make those changes in the parent + checkout and rerun the affected validation steps before acceptance. +5. Run the final validation stack in the exact order defined below. + +Acceptance: + +1. Every accepted worker output has a matching parent-written sentinel. +2. No accepted patch forks selection truth or weakens fail-closed launch behavior. +3. No accepted patch lets stale trace rows authorize current liveness. +4. `run-state.json` records accepted and rejected outputs explicitly. + +Green-path output: + +- `.runs/plan-10/sentinels/task-m10-e1-integrate-and-validate.ok` + +Blocked-path output: + +- `.runs/plan-10/blocked.json` + +### task/m10-e2-docs-review + +Ownership: + +- parent only + +Allowed files: + +- [docs/TRACE.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/TRACE.md) +- [docs/USAGE.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/USAGE.md) + +Scope: + +1. Review whether the landed runtime behavior changed user-visible wording or trace examples. +2. If no wording drift exists, record an explicit no-change decision in `run-state.json` and + `session.log`. +3. If wording drift exists, make the smallest truthful docs edits needed after the test wall is + already green. +4. Keep docs aligned with the shipped behavior: + - explicit unique-member selection rule, + - lazy world-backed launch on first need, + - orchestrator-anchored toolbox posture, + - restart replacement or honest absence, + - no stale-trace resurrection. + +Must not do: + +1. No docs-first speculation. +2. No docs edits to justify unproven runtime behavior. +3. No reopening of runtime design through docs. + +Acceptance: + +1. Docs are either unchanged with an explicit recorded reason or minimally updated to match shipped + behavior. +2. No doc implies scheduler, selector UX, or trace-authorized liveness. + +Green-path output: + +- `.runs/plan-10/sentinels/task-m10-e2-docs-review.ok` + +Blocked-path output: + +- `.runs/plan-10/blocked.json` + +### task/m10-e3-closeout + +Ownership: + +- parent only + +Scope: + +1. Mark the run complete in `.runs/plan-10/run-state.json`. +2. Write `.runs/plan-10/closeout.md` with: + - accepted tasks, + - integrated worktrees and branches, + - validation commands and outcomes, + - final shipped selection rule, + - final doctor/runtime alignment statement, + - final lazy-launch and restart-replacement behavior, + - explicit docs-reviewed outcome, + - confirmation that the parent remained the only final branch writer. +3. Append the final acceptance rationale to `.runs/plan-10/session.log`. + +Acceptance: + +1. `closeout.md` is present and matches the final branch state. +2. All sentinels through `task/m10-e3-closeout` exist. +3. No blocked-path artifact exists for a successful run. + +Green-path output: + +- `.runs/plan-10/sentinels/task-m10-e3-closeout.ok` + +## Context-Control Rules + +1. The parent owns `.runs/plan-10/*`. Workers do not edit run-state, sentinels, queue, session + log, blocked state, or closeout artifacts. +2. Worker packets include only: + - the task ID, + - allowed files, + - frozen invariants, + - stop conditions, + - exact commands to run. +3. Workers must stop immediately if they need to touch any file outside their allowed list. +4. Workers report: + - changed files, + - exact commands run, + - blockers. +5. Workers do not rebase, merge, or integrate each other’s work. +6. The parent rejects any worker patch that silently broadens scope or re-derives selection or + liveness rules outside the frozen production contract. +7. If worker outputs conflict with current parent truth, the parent re-derives the correct result + from production code and rewrites the patch locally instead of negotiating blended semantics + across worktrees. + +## Tests And Acceptance + +### Pre-worker proof order + +Run these parent-owned checks before dispatching the worker window: + +1. `cargo test -p shell --lib -- --nocapture` +2. `cargo test -p shell async_repl -- --nocapture` +3. `cargo test -p shell --test agent_successor_contract_ahcsitc0 -- --no-run` +4. `cargo test -p shell --test repl_world_first_routing_v1 -- --no-run` +5. `cargo test -p shell --test agent_hub_trace_persistence -- --no-run` + +### Final validation order + +Run these commands in this exact order during `task/m10-e1-integrate-and-validate`: + +1. `cargo test -p shell async_repl -- --nocapture` +2. `cargo test -p shell --test agent_successor_contract_ahcsitc0 -- --nocapture` +3. `cargo test -p shell --test repl_world_first_routing_v1 -- --nocapture` +4. `cargo test -p shell --test agent_hub_trace_persistence -- --nocapture` +5. `cargo test -p shell -- --nocapture` +6. `cargo fmt --all -- --check` +7. `cargo clippy -p shell --all-targets -- -D warnings` +8. `cargo clippy --workspace --all-targets -- -D warnings` +9. `cargo test --workspace -- --nocapture` + +### Acceptance checklist + +The slice is accepted only if all of these are true: + +1. One unique eligible world-scoped member can be selected deterministically from inventory. +2. Zero eligible world members preserves host-only behavior without inventing fake member state. +3. Ambiguous world-member selection fails closed in both doctor and runtime launch. +4. The first world-backed REPL command can lazily launch the selected member under a live + orchestrator session. +5. The launched member is persisted in `allocating` first and only becomes authoritative-live after + retained UAA ownership is proven. +6. `substrate agent status --json` can surface the live launched member from runtime state with + top-level `world_id` and `world_generation`. +7. `substrate agent toolbox status|env` remains orchestrator-scoped. +8. World restart invalidates the stale member and can launch a replacement participant on the new + generation. +9. Failed replacement leaves honest absence, not stale liveness. +10. Trace persistence remains auditable without becoming current liveness authority. +11. The targeted shell commands and workspace gates above pass. +12. Docs were touched only if the runtime wording actually changed, and otherwise the no-change + decision was recorded explicitly. + +## Merge Refusal Rules + +The parent refuses to merge a worker output if any of these are true: + +1. The patch edits a file outside the task’s allowed file list. +2. The patch invents or preserves a selection rule not frozen in `validator.rs`. +3. The patch authorizes current liveness from trace rows. +4. The patch weakens fail-closed behavior for ambiguity, missing parent binding, or replacement + failure. +5. The patch requires concurrent parent edits to production files to become intelligible. +6. The patch omits test evidence for the behavior it claims to cover. +7. The patch implies docs changes are required to explain behavior the runtime still does not prove. +8. The patch broadens scope into scheduler, selector UX, or public control-plane work. + +## Run Exit Criteria + +### Successful run + +The run is successful only if all of these are true: + +1. `.runs/plan-10/sentinels/task-m10-a1-preflight.ok` exists. +2. `.runs/plan-10/sentinels/task-m10-a2-validator-contract.ok` exists. +3. `.runs/plan-10/sentinels/task-m10-a3-doctor-alignment.ok` exists. +4. `.runs/plan-10/sentinels/task-m10-a4-member-launch-seam.ok` exists. +5. `.runs/plan-10/sentinels/task-m10-a5-restart-replacement.ok` exists. +6. `.runs/plan-10/sentinels/task-m10-b1-runtime-contract-tests.ok` exists. +7. `.runs/plan-10/sentinels/task-m10-b2-world-routing-restart-tests.ok` exists. +8. `.runs/plan-10/sentinels/task-m10-e1-integrate-and-validate.ok` exists. +9. `.runs/plan-10/sentinels/task-m10-e2-docs-review.ok` exists. +10. `.runs/plan-10/sentinels/task-m10-e3-closeout.ok` exists. +11. `.runs/plan-10/run-state.json` exists and records a completed terminal state. +12. `.runs/plan-10/queue.json` and `.runs/plan-10/session.log` exist. +13. `.runs/plan-10/closeout.md` exists and matches the final accepted branch state. +14. `.runs/plan-10/blocked.json` does not exist. + +### Blocked termination + +The run terminates blocked only if all of these are true: + +1. `.runs/plan-10/blocked.json` exists and records the blocking reason. +2. `.runs/plan-10/run-state.json` exists and records `blocked` as the terminal state. +3. `session.log` contains the parent rationale for the stop. +4. No downstream task sentinel may be written after the blocking point. +5. `.runs/plan-10/sentinels/task-m10-e3-closeout.ok` must not exist. +6. `.runs/plan-10/closeout.md` must not exist. + +## Closeout + +Successful closeout records: + +1. the final shipped member-selection rule, +2. the final doctor/runtime alignment contract, +3. the final lazy-launch and restart-replacement behavior, +4. the exact validation order and outcomes, +5. whether docs were changed or explicitly left untouched, +6. confirmation that the parent remained the only final branch writer, +7. confirmation that worker lanes stayed inside their file boundaries. + +## Assumptions + +1. The parent run starts from the current branch baseline `feat/session-centric-state-store`. +2. `PLAN-09` already froze the store-owned session authority ladder that `PLAN-10` must consume + rather than redesign. +3. The Rust package name `shell` is authoritative for validation commands in this repository. +4. `state_store.rs`, `registry.rs`, and `world_gateway.rs` can remain untouched unless a concrete + runtime seam gap proves otherwise. +5. Any user-visible wording drift can be corrected with minimal post-test docs edits and does not + require reopening runtime architecture. diff --git a/llm-last-mile/ORCH_PLAN-11.md b/llm-last-mile/ORCH_PLAN-11.md new file mode 100644 index 000000000..b7ae6eb60 --- /dev/null +++ b/llm-last-mile/ORCH_PLAN-11.md @@ -0,0 +1,1095 @@ +# ORCH_PLAN-11: Refreeze Member Dispatch Runtime Descriptor And Complete In-World Cutover + +Branch: `feat/session-centric-state-store` +Plan source: [PLAN-11.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-11.md) +Reference style source: [ORCH_PLAN-10.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/ORCH_PLAN-10.md) +Blocked-run evidence: [.runs/plan-11/run-state.json](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/plan-11/run-state.json), [.runs/plan-11/blocked.json](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/plan-11/blocked.json) +Execution type: shell/world-agent transport-backed member-placement cutover orchestration plan, Linux-first, no UI scope, strong runtime/status/trace scope + +## Summary + +This run replaces the blocked `ORCH_PLAN-11` execution with a fresh parent-runner plan that starts +from the actual blocked state, explicitly invalidates the old Gate B freeze, refreezes +`MemberDispatchRequestV1` with the required `resolved_runtime` descriptor, refreezes the shell +builder and harness seam including the `dispatch/prelude.rs` export, then opens exactly one real +parallel runtime window with an exact active worker cap of `2`. + +This slice is not a second resolver, not a new public API family, not a scheduler, not a +shared-crate extraction project, and not a docs-first cleanup run. It is one bounded production +cutover: + +1. import the blocked-run truth and supersede the old frozen request contract, +2. refreeze the typed `ExecuteRequest.member_dispatch` contract in + [crates/agent-api-types/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/agent-api-types/src/lib.rs) + by adding required `resolved_runtime`, +3. refreeze the shell builder and harness seam in + [crates/shell/src/execution/routing/dispatch/world_ops.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs), + [crates/shell/src/execution/routing/dispatch/prelude.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/prelude.rs), + [crates/shell/tests/support/socket.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/support/socket.rs), + and + [crates/shell/tests/support/repl_world_agent.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/support/repl_world_agent.rs), +4. run one honest two-lane runtime window: + - lane `B1` implements in-world UAA member startup in `world-agent`, including a direct + `unified-agent-api` dependency, + - lane `B2` cuts the shell member path over to explicit remote retained control and the frozen + typed builder, +5. close the parent-only regression wall for cancel, replacement, status, and trace correctness, +6. review docs only after the runtime and test wall are green. + +The prior run blocked because Gate B froze too early. The frozen request lacked +`resolved_runtime`, so `world-agent` could not honestly reconstruct the shell-owned UAA startup +path without inventing a second resolver, and the shell lane also lacked the parent-fixable +builder export seam through `dispatch/prelude.rs`. This plan prevents that failure mode by making +both fixes mandatory in the parent-only refreeze lane before any worker starts. + +Parent-owned critical path: + +1. `task/m11-a1-preflight-and-blocked-import` +2. `task/m11-a2-member-dispatch-contract-refreeze` +3. `task/m11-a3-builder-export-and-harness-refreeze` +4. `task/m11-c1-integrate-and-regression-wall` +5. `task/m11-c2-docs-review` +6. `task/m11-c3-closeout` + +Every child worker uses `GPT-5.4` with `reasoning_effort=high`. + +## Hard Guards + +### Locked invariants + +1. `PLAN-11` remains a transport-backed placement-honesty slice only. No UI work is authorized. +2. `POST /v1/execute/stream` and `POST /v1/execute/cancel` remain the only transport seam. +3. `ExecuteRequest.member_dispatch` remains additive, typed, and internal. No magic command + string or new target enum is authorized. +4. The contract refreeze stays on `MemberDispatchRequestV1`. No `V2` rename is authorized for this + unshipped internal seam. +5. `MemberDispatchRequestV1` must carry required + `resolved_runtime: ResolvedMemberRuntimeDescriptorV1`. +6. `ResolvedMemberRuntimeDescriptorV1` contains exactly: + - `backend_kind` + - `binary_path` +7. `backend_kind` is explicit. No inference from `backend_id` is allowed. +8. `binary_path` is the already-resolved absolute path selected by the shell. `world-agent` must + consume it and must not re-resolve from inventory, config, or a second selector. +9. Request-boundary validation lives in + [crates/agent-api-types/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/agent-api-types/src/lib.rs), + not in deep handler code. +10. For typed member dispatch, `cmd.trim().is_empty()` must be true and `pty` must be false. For + ordinary process exec, `member_dispatch` must be `None`. +11. The top-level `agent_id` remains authoritative for budgets, traces, and diagnostics. + `member_dispatch` must not duplicate it. +12. The shell remains the only authority for canonical session-root writes, persisted participant + state, `status`, `doctor`, and toolbox surfaces. +13. `world-agent` owns in-world member execution, retained remote cancel delivery, remote event + streaming, and completion observation only. +14. `world-agent` must take a direct `unified-agent-api` dependency in + [crates/world-agent/Cargo.toml](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/Cargo.toml). + No new shared crate extraction is authorized in this slice. +15. The shell must represent retained control explicitly as local vs remote. Remote member control + must not be stored as a local `RetainedRunControl`. +16. The shell must use an explicit `PreparedMemberDispatch` or equivalent dedicated remote launch + shape. Host orchestrator startup stays on the existing local path. +17. The frozen builder must be re-exported through + [crates/shell/src/execution/routing/dispatch/prelude.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/prelude.rs) + before the shell runtime lane opens. +18. A world-scoped member launch fails closed on remote dispatch failure. Host fallback is + forbidden. +19. The authoritative `world_id` and `world_generation` supplied by the shell must be validated + remotely and rejected on mismatch. +20. `ExecuteStreamFrame::{Start,Event,Exit,Error}` remain the only stream families. +21. Remote readiness remains gated by the existing session-handle event contract unless the parent + explicitly refreezes it before worker lanes open. +22. A member persists as `Allocating` before remote launch and may advertise live only after + session-handle evidence, retained ownership, active event stream, and completion observation + all exist. +23. `substrate agent status --json` and trace rows must remain producer-backed and + participant-correct after cancel and replacement. Stale liveness must not revive. +24. Linux-first is explicit. Non-Linux member dispatch paths fail closed in this slice. +25. Docs are late and optional only after the regression wall is green. +26. Package-targeted cargo commands use the actual package names in this repo: + `agent-api-types`, `world-agent`, and `shell`. +27. The parent is the only writer of `.runs/plan-11/*`. + +### File-level boundaries + +Parent-owned serialized refreeze surfaces: + +- [crates/agent-api-types/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/agent-api-types/src/lib.rs) +- [crates/shell/src/execution/routing/dispatch/world_ops.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs) +- [crates/shell/src/execution/routing/dispatch/prelude.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/prelude.rs) +- [crates/shell/tests/support/socket.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/support/socket.rs) +- [crates/shell/tests/support/repl_world_agent.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/support/repl_world_agent.rs) + +Worker-safe world-agent lane after the refreeze: + +- [crates/world-agent/Cargo.toml](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/Cargo.toml) +- [crates/world-agent/src/service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs) +- [crates/world-agent/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/lib.rs) +- new [crates/world-agent/src/member_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs) +- [crates/world-agent/tests/streamed_execute_cancel_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/tests/streamed_execute_cancel_v1.rs) + +Worker-safe shell runtime lane after the refreeze: + +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) +- [crates/shell/tests/repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs) + +Parent-owned late regression and drift-closure surfaces: + +- [crates/shell/tests/agent_successor_contract_ahcsitc0.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_successor_contract_ahcsitc0.rs) +- [crates/shell/tests/agent_hub_trace_persistence.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_hub_trace_persistence.rs) + +Parent-owned escalation-only production surfaces: + +- [crates/shell/src/execution/agent_runtime/session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs) +- [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) +- [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs) + +Read-for-truth only: + +- [PLAN-11.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-11.md) +- [ORCH_PLAN-10.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/ORCH_PLAN-10.md) +- [.runs/plan-11/run-state.json](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/plan-11/run-state.json) +- [.runs/plan-11/blocked.json](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/plan-11/blocked.json) + +Optional parent-owned docs surface only after the regression wall is green: + +- [docs/TRACE.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/TRACE.md) +- [docs/USAGE.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/USAGE.md) + +### Non-negotiable stop conditions + +Stop the run and write `.runs/plan-11/blocked.json` if any of these occur: + +1. A worker lane opens before the parent lands the `resolved_runtime` refreeze and the + `dispatch/prelude.rs` builder export. +2. Any task requires reopening the refrozen request contract after either worker lane has started. +3. Any task requires `world-agent` to infer runtime selection from `backend_id`, protocol, agent + inventory, or shell-private selectors instead of consuming `resolved_runtime`. +4. Any task requires extracting a new shared startup crate to finish `PLAN-11`. +5. Any task falls back to host launch after remote world-member dispatch fails. +6. Any task keeps the shell member path on local `gateway.run_control(...)` while claiming world + placement. +7. Any task treats remote control state as if it were a local `RetainedRunControl`. +8. Any task introduces a second lifecycle model instead of reusing the existing + manifest/store/liveness contract. +9. Any worker lane needs to touch a parent-owned refreeze file, late regression file, docs, or + `.runs/plan-11/*` to finish its assignment. +10. The world-agent lane needs to rename or redesign the session-bearing readiness `Event` + contract after the shell lane starts. +11. Any task proves `status` or trace correctness depends on heuristic trace-first liveness + recovery. +12. Any task requires early docs edits to explain behavior that runtime and tests still do not + prove. + +## Orchestration State Surfaces + +### Canonical run state + +The only canonical source of truth for run orchestration state: + +- `.runs/plan-11/run-state.json` + +Parent-only writes to this file. It tracks: + +- current phase, +- active task IDs, +- branch and worktree assignment, +- gate status, +- the prior blocked freeze as superseded evidence, +- the refrozen `member_dispatch` request contract, +- the refrozen builder-export and harness contract, +- the refrozen session-handle event contract, +- the refrozen local-vs-remote retained-control contract, +- accepted and rejected worker outputs, +- escalation-file usage, +- blocked or completed terminal state, +- final closeout pointer. + +If a worker report conflicts with `run-state.json`, the parent treats `run-state.json` as +authoritative until it explicitly reconciles the discrepancy. + +### Derived run artifacts + +The parent may maintain these local artifacts: + +- `.runs/plan-11/queue.json` +- `.runs/plan-11/session.log` +- `.runs/plan-11/sentinels/task-m11-a1-preflight-and-blocked-import.ok` +- `.runs/plan-11/sentinels/task-m11-a2-member-dispatch-contract-refreeze.ok` +- `.runs/plan-11/sentinels/task-m11-a3-builder-export-and-harness-refreeze.ok` +- `.runs/plan-11/sentinels/task-m11-b1-world-agent-member-manager.ok` +- `.runs/plan-11/sentinels/task-m11-b2-shell-remote-member-cutover.ok` +- `.runs/plan-11/sentinels/task-m11-c1-integrate-and-regression-wall.ok` +- `.runs/plan-11/sentinels/task-m11-c2-docs-review.ok` +- `.runs/plan-11/sentinels/task-m11-c3-closeout.ok` +- `.runs/plan-11/blocked.json` +- `.runs/plan-11/closeout.md` + +Sentinel rules: + +1. `.ok` means the parent validated the task output and advanced the run. +2. Missing sentinel means the task is not accepted. +3. `blocked.json` is written only on blocked termination. +4. `closeout.md` is written only on successful completion. +5. Worker notes, commits, or branch state never replace parent-written sentinels or + `run-state.json`. + +## Concurrency Policy + +1. The parent is the only integrator. +2. The parent is the only writer of final branch state on `feat/session-centric-state-store`. +3. Exact active worker cap: `2`. +4. There are zero worker lanes during the refreeze lane: + - `task/m11-a1-preflight-and-blocked-import` + - `task/m11-a2-member-dispatch-contract-refreeze` + - `task/m11-a3-builder-export-and-harness-refreeze` +5. The only honest parallel window is: + - `task/m11-b1-world-agent-member-manager` + - `task/m11-b2-shell-remote-member-cutover` +6. The regression wall is parent-only again after both runtime lanes return. +7. No worker may edit any parent-owned refreeze file, any late regression file, docs, or + `.runs/plan-11/*`. +8. Both worker worktrees are seeded from the exact post-`task/m11-a3` tree. +9. If either worker proves a missing contract change is required in a frozen parent-owned file, + that worker stops and hands the gap back to the parent. +10. Worker coordination uses sentinels and long waits. Tight polling loops against git state or + run-state are forbidden. + +### Why the worker cap remains exactly `2` + +The worker cap remains exactly `2` because `PLAN-11` has only two honest runtime seams after the +parent-owned refreeze and no third independent lane before the regression wall: + +1. the request contract, builder export, and harness/event seam are one coupled parent-owned + freeze, +2. there is one real `world-agent` runtime lane after that freeze, +3. there is one real shell remote-cutover lane after that freeze, +4. status, trace, replacement, and operator-surface correctness depend on integrated behavior of + those two runtime halves, +5. a third worker before `task/m11-c1` would only create contract churn, merge noise, and a + higher chance of repeating the blocked refreeze failure. + +## Approval And Gate Model + +There are no human approval gates defined for this run. + +Replacement control mechanism: + +1. parent validation gates, +2. parent-written sentinels, +3. `session.log` for acceptance and rejection rationale, +4. `blocked.json` for hard-stop termination, +5. `closeout.md` for successful completion. + +### Gate A: Blocked-State Import And Topology Lock + +Required before implementation starts: + +- parent re-reads [PLAN-11.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-11.md), + [ORCH_PLAN-10.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/ORCH_PLAN-10.md), + [.runs/plan-11/run-state.json](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/plan-11/run-state.json), + and [.runs/plan-11/blocked.json](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/plan-11/blocked.json), +- parent records the exact execution shape: parent-only refreeze lane, one real parallel runtime + window, parent-only regression wall, +- parent records the exact worker cap of `2`, +- parent records the actual blocked reason: missing `resolved_runtime` and missing prelude export + seam, +- parent records the package-name normalization: `agent-api-types`, `world-agent`, `shell`. + +### Gate B: Request, Builder, And Harness Refreeze + +Required before either worker lane opens: + +- `MemberDispatchRequestV1` is refrozen with required `resolved_runtime`, +- request-boundary validation rules are refrozen, +- `world_ops.rs` carries the refrozen `resolved_runtime` fields, +- `dispatch/prelude.rs` re-exports the frozen builder, +- `socket.rs` and `repl_world_agent.rs` assert the refrozen payload shape, +- the session-bearing remote readiness event remains frozen well enough for + `extract_session_handle_id(...)` or an explicitly parent-approved equivalent, +- the parent has completed the pre-worker proof order, +- both worker worktrees are seeded from the exact post-`task/m11-a3` tree. + +### Gate C: Runtime Seam Integration + +Required before the final regression wall starts: + +- both worker outputs are integrated by the parent, +- any approved escalation into + [crates/shell/src/execution/agent_runtime/session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs), + [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs), + or + [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs) + is already complete and justified in `run-state.json`, +- host orchestrator startup remains unchanged, +- no production path still routes a world-member startup through local `gateway.run_control(...)`, +- `world-agent` consumes the transport-carried `resolved_runtime` instead of any second resolver. + +### Gate D: Final Acceptance + +Required before closeout: + +- the integrated runtime uses typed execute-stream member dispatch with `resolved_runtime`, +- cancel, replacement, status, and trace regression tests are green in the required order, +- docs review is complete and either produced aligned late edits or an explicit no-change + decision, +- `run-state.json` records the final request, builder, remote-control, replacement, and + regression contracts. + +## Workstream Plan + +### Worktree topology + +Parent checkout: + +- current checkout on `feat/session-centric-state-store` + +Child worktrees and branches: + +- `../substrate-m11-world-agent-member-manager` + - `codex/feat-session-centric-state-store-m11-world-agent-member-manager` +- `../substrate-m11-shell-remote-member-cutover` + - `codex/feat-session-centric-state-store-m11-shell-remote-member-cutover` + +Suggested creation commands: + +```bash +git worktree add ../substrate-m11-world-agent-member-manager \ + -b codex/feat-session-centric-state-store-m11-world-agent-member-manager \ + feat/session-centric-state-store +git worktree add ../substrate-m11-shell-remote-member-cutover \ + -b codex/feat-session-centric-state-store-m11-shell-remote-member-cutover \ + feat/session-centric-state-store +``` + +Execution topology: + +1. The parent checkout serves every refreeze, integration, and final validation task. +2. Exactly two child worktrees exist because exactly two worker-safe runtime seams exist after the + refreeze. +3. No third child worktree is authorized because no earlier seam can be parallelized honestly + without repeating the blocked contract failure. + +Subagents do not merge each other’s work. They return only changed files, exact commands run, test +results, and blockers to the parent. + +### Task graph + +Execution graph for the run: + +1. `task/m11-a1-preflight-and-blocked-import` +2. `task/m11-a2-member-dispatch-contract-refreeze` +3. `task/m11-a3-builder-export-and-harness-refreeze` +4. `task/m11-b1-world-agent-member-manager` and `task/m11-b2-shell-remote-member-cutover` in + parallel +5. `task/m11-c1-integrate-and-regression-wall` +6. `task/m11-c2-docs-review` +7. `task/m11-c3-closeout` + +Parent-only serialized tasks: + +- `task/m11-a1-preflight-and-blocked-import` +- `task/m11-a2-member-dispatch-contract-refreeze` +- `task/m11-a3-builder-export-and-harness-refreeze` +- `task/m11-c1-integrate-and-regression-wall` +- `task/m11-c2-docs-review` +- `task/m11-c3-closeout` + +Worker-owned tasks: + +- `task/m11-b1-world-agent-member-manager` +- `task/m11-b2-shell-remote-member-cutover` + +## Parallel Window B + +This is the only worker window in the run. + +It opens only after Gate B passes and after the parent has landed the refreeze that the prior run +was missing. There is no earlier honest worker window because: + +1. [crates/agent-api-types/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/agent-api-types/src/lib.rs) + defines the request contract the runtime lanes share, +2. [crates/shell/src/execution/routing/dispatch/world_ops.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs) + and + [crates/shell/src/execution/routing/dispatch/prelude.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/prelude.rs) + define the only allowed builder seam for the shell lane, +3. [crates/shell/tests/support/socket.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/support/socket.rs) + and + [crates/shell/tests/support/repl_world_agent.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/support/repl_world_agent.rs) + define what the runtime lanes can safely assume about captured payloads and readiness events, +4. the blocked run already proved that splitting before this refreeze creates fake progress and a + guaranteed re-open. + +After the refreeze, the runtime work splits honestly: + +- lane `B1` owns `world-agent` execution, UAA startup, span registration, cancel delivery, and + streaming, +- lane `B2` owns shell retained-control representation, remote launch cutover, same-generation + reuse, and replacement convergence, +- neither lane may reopen the typed request, builder, or readiness-event contract. + +### task/m11-a1-preflight-and-blocked-import + +Ownership: + +- parent only + +Scope: + +1. Re-read [PLAN-11.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-11.md), + [ORCH_PLAN-10.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/ORCH_PLAN-10.md), + [.runs/plan-11/run-state.json](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/plan-11/run-state.json), + and [.runs/plan-11/blocked.json](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/plan-11/blocked.json). +2. Confirm the run executes on `feat/session-centric-state-store`. +3. Import the blocked-run truth into the new run packet and mark the old Gate B freeze as + superseded, not reusable. +4. Initialize or refresh: + - `.runs/plan-11/run-state.json` + - `.runs/plan-11/queue.json` + - `.runs/plan-11/session.log` +5. Record the no-fallback, shell-authority, Linux-first, docs-late, and exact `2`-worker posture. +6. Record the parent-owned, worker-owned, and escalation-only file boundaries. + +Commands: + +1. `git branch --show-current` +2. `mkdir -p .runs/plan-11/sentinels` + +Acceptance: + +1. The parent can restate the exact blocked reason without ambiguity: + - missing `resolved_runtime` + - missing `dispatch/prelude.rs` export seam +2. `run-state.json` records that the old request freeze is superseded and Gate B must be rerun. +3. The parent can name the only honest worker window: after `task/m11-a3`. +4. Gate A passes. + +Green-path output: + +- `.runs/plan-11/sentinels/task-m11-a1-preflight-and-blocked-import.ok` + +Blocked-path output: + +- `.runs/plan-11/blocked.json` + +### Parent validation gate A + +Required before `task/m11-a2-member-dispatch-contract-refreeze` starts: + +1. No invariant contradiction remains unresolved. +2. The parent can explain why the prior Gate B freeze is invalid and cannot be reused. +3. The parent can explain why docs stay deferred until after the runtime and regression wall. + +### task/m11-a2-member-dispatch-contract-refreeze + +Ownership: + +- parent only + +Allowed files: + +- [crates/agent-api-types/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/agent-api-types/src/lib.rs) + +Scope: + +1. Refreeze additive `MemberDispatchRequestV1` and `ExecuteRequest.member_dispatch` by adding: + - required `resolved_runtime: ResolvedMemberRuntimeDescriptorV1` + - `ResolvedMemberRuntimeDescriptorV1.backend_kind` + - `ResolvedMemberRuntimeDescriptorV1.binary_path` +2. Keep the internal type name `MemberDispatchRequestV1`. +3. Implement parse-boundary validation for `backend_kind`, `binary_path`, and the existing + mutually exclusive `cmd` / `pty` / `member_dispatch` rules. +4. Preserve the authoritative identity tuple carried by member dispatch: + - orchestration session, + - participant lineage, + - backend identity, + - protocol, + - run id, + - world id, + - world generation, + - resolved runtime. +5. Update round-trip and invalid-shape tests so ordinary process exec remains unchanged and missing + `resolved_runtime` or invalid mixed shapes fail at parse time. + +Must not do: + +1. No `V2` transport family. +2. No generic execute target enum. +3. No shell or `world-agent` handler logic in this task. + +Commands: + +1. `cargo test -p agent-api-types -- --nocapture` +2. `cargo test -p world-agent --test streamed_execute_cancel_v1 -- --no-run` +3. `cargo test -p shell --test repl_world_first_routing_v1 -- --no-run` +4. `cargo test -p shell --test agent_successor_contract_ahcsitc0 -- --no-run` +5. `cargo test -p shell --test agent_hub_trace_persistence -- --no-run` + +Acceptance: + +1. The request contract now carries the exact resolved launch facts `world-agent` needs. +2. Invalid `cmd` plus `member_dispatch` shapes fail at the request boundary. +3. Missing or malformed `resolved_runtime` fails at the request boundary. +4. Downstream packages still compile against the additive request shape. +5. The parent records the accepted refrozen schema in `run-state.json`. + +Green-path output: + +- `.runs/plan-11/sentinels/task-m11-a2-member-dispatch-contract-refreeze.ok` + +Blocked-path output: + +- `.runs/plan-11/blocked.json` + +### Parent validation gate B1 + +Required before `task/m11-a3-builder-export-and-harness-refreeze` starts: + +1. `cargo test -p agent-api-types -- --nocapture` passes. +2. `run-state.json` records the refrozen `resolved_runtime` contract. +3. No downstream task may remove or re-interpret `resolved_runtime`. + +### task/m11-a3-builder-export-and-harness-refreeze + +Ownership: + +- parent only + +Allowed files: + +- [crates/shell/src/execution/routing/dispatch/world_ops.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs) +- [crates/shell/src/execution/routing/dispatch/prelude.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/prelude.rs) +- [crates/shell/tests/support/socket.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/support/socket.rs) +- [crates/shell/tests/support/repl_world_agent.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/support/repl_world_agent.rs) + +Scope: + +1. Refreeze the dedicated member-dispatch builder in + [crates/shell/src/execution/routing/dispatch/world_ops.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs) + so it carries `resolved_runtime` and keeps existing env and trace plumbing. +2. Re-export `build_agent_client_and_member_dispatch_request(...)` through + [crates/shell/src/execution/routing/dispatch/prelude.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/prelude.rs). +3. Extend [crates/shell/tests/support/socket.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/support/socket.rs) + so recorded execute-stream payloads deserialize and assert nested `resolved_runtime`. +4. Extend + [crates/shell/tests/support/repl_world_agent.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/support/repl_world_agent.rs) + so typed member-dispatch payloads capture nested `resolved_runtime` while keeping accepted + ready, cancel, success, and failure scripting contracts. +5. Freeze the remote-ready event compatibility required by + `extract_session_handle_id(...)` in + [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs). + +Must not do: + +1. No runtime lifecycle logic in `world_ops.rs`. +2. No widening of the public routing surface beyond the one builder export the shell lane needs. +3. No change to the ready-event schema unless the parent explicitly records a replacement + contract before worker lanes open. + +Commands: + +1. `cargo test -p agent-api-types -- --nocapture` +2. `cargo test -p shell --lib -- --nocapture` +3. `cargo test -p world-agent --test streamed_execute_cancel_v1 -- --no-run` +4. `cargo test -p shell --test repl_world_first_routing_v1 -- --no-run` +5. `cargo test -p shell --test agent_successor_contract_ahcsitc0 -- --no-run` +6. `cargo test -p shell --test agent_hub_trace_persistence -- --no-run` + +Acceptance: + +1. The shell lane can import the frozen builder through the normal routing seam. +2. Tests can inspect typed member-dispatch payloads with nested `resolved_runtime`. +3. The builder contract is frozen for both worker lanes. +4. The ready-event contract remains frozen and testable. +5. Gate B passes. + +Green-path output: + +- `.runs/plan-11/sentinels/task-m11-a3-builder-export-and-harness-refreeze.ok` + +Blocked-path output: + +- `.runs/plan-11/blocked.json` + +### Parent validation gate B2 + +Required before the worker window opens: + +1. The full pre-worker proof order passes. +2. `run-state.json` records the refrozen request, builder export, and readiness-event contracts. +3. The parent seeds both worker worktrees from the exact post-`task/m11-a3` tree. + +### task/m11-b1-world-agent-member-manager + +Ownership: + +- worker lane `B1` only in `../substrate-m11-world-agent-member-manager` +- worker model: `GPT-5.4`, `reasoning_effort=high` + +Allowed files: + +- [crates/world-agent/Cargo.toml](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/Cargo.toml) +- [crates/world-agent/src/service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs) +- [crates/world-agent/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/lib.rs) +- new [crates/world-agent/src/member_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs) +- [crates/world-agent/tests/streamed_execute_cancel_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/tests/streamed_execute_cancel_v1.rs) + +Forbidden touch surfaces: + +- every parent-owned refreeze file, +- every shell runtime-lane file, +- every parent-owned late regression file, +- docs and `.runs/plan-11/*`. + +Scope: + +1. Add direct `unified-agent-api` dependency in + [crates/world-agent/Cargo.toml](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/Cargo.toml). +2. Branch + [crates/world-agent/src/service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs) + on `member_dispatch` while preserving ordinary process execution unchanged. +3. Add new internal + [crates/world-agent/src/member_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs) + that: + - validates `resolved_runtime`, + - validates authoritative world binding, + - builds the UAA backend from `backend_kind` plus `binary_path`, + - starts in-world retained `run_control(...)`, + - retains cancel, event, and completion ownership, + - emits `Start`, session-bearing `Event`, and terminal `Exit` or `Error`, + - cleans up terminal state honestly. +4. Wire minimal module exposure in + [crates/world-agent/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/lib.rs). +5. Extend `/v1/execute/cancel` so it can deliver to member-dispatch spans with bounded wait + behavior. +6. Extend + [crates/world-agent/tests/streamed_execute_cancel_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/tests/streamed_execute_cancel_v1.rs) + for: + - success, + - unsupported backend kind, + - missing binary path, + - binding mismatch, + - cancel, + - bootstrap failure, + - abnormal termination. + +Must not do: + +1. No backend-specific stdout parsing as readiness truth. +2. No runtime re-resolution from inventory or config. +3. No shell-private helper dependency. +4. No canonical session-state writes in `world-agent`. + +Minimum worker test commands before handoff: + +1. `cargo test -p world-agent --test streamed_execute_cancel_v1 -- --nocapture` +2. `cargo test -p world-agent -- --no-run` + +Acceptance: + +1. Typed member-dispatch startup and cancel work at the `world-agent` layer without the shell REPL + in the loop. +2. `world-agent` consumes the transport-carried `resolved_runtime` and does not invent a second + resolver. +3. Mismatched world binding and unsupported backend fail closed. +4. Ordinary process streaming remains green. +5. Worker returns no contract-change request against a frozen parent-owned file. + +Stop-back conditions for lane `B1`: + +1. Any required change to `agent-api-types`, `world_ops.rs`, `dispatch/prelude.rs`, `socket.rs`, + or `repl_world_agent.rs`. +2. Any required rename of the readiness-bearing `Event` contract. +3. Any need to extract a shared startup crate in this slice. +4. Any need to move canonical session-state writes into `world-agent`. + +Green-path output: + +- `.runs/plan-11/sentinels/task-m11-b1-world-agent-member-manager.ok` after parent acceptance + +### task/m11-b2-shell-remote-member-cutover + +Ownership: + +- worker lane `B2` only in `../substrate-m11-shell-remote-member-cutover` +- worker model: `GPT-5.4`, `reasoning_effort=high` + +Allowed files: + +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) +- [crates/shell/tests/repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs) + +Forbidden touch surfaces: + +- every parent-owned refreeze file, +- every world-agent lane file, +- every parent-owned late regression file, +- docs and `.runs/plan-11/*`. + +Scope: + +1. Replace the member launch path in + [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) + so the member startup path uses the refrozen typed builder through `dispatch/prelude.rs` + instead of host orchestrator startup. +2. Introduce explicit `PreparedMemberDispatch` and explicit local-vs-remote retained-control + representation. +3. Keep host orchestrator startup unchanged. +4. Persist member participants in `Allocating` before remote launch, then drive `Ready` and + `Running` from remote stream events plus retained ownership. +5. Route remote cancel through `/v1/execute/cancel` using the retained remote span id. +6. Reuse existing manifest, store, replacement, invalidation, and liveness helpers rather than + inventing a second state machine. +7. Extend shell tests in + [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) + and + [crates/shell/tests/repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs) + for: + - first world-backed launch over typed execute-stream, + - same-generation reuse, + - preflight failure before transport, + - replacement launch, + - failed replacement leaving honest absence, + - non-live convergence on cancel or failure. + +Must not do: + +1. No local `gateway.run_control(...)` for world-member startup. +2. No treating remote control as a local cancel handle. +3. No redesign of store semantics. +4. No broadening into doctor/toolbox work before the regression wall. + +Minimum worker test commands before handoff: + +1. `cargo test -p shell async_repl -- --nocapture` +2. `cargo test -p shell --test repl_world_first_routing_v1 -- --nocapture` +3. `cargo test -p shell --test agent_successor_contract_ahcsitc0 -- --no-run` + +Acceptance: + +1. The first world-backed command launches the member through typed `/v1/execute/stream`. +2. Same-generation reuse remains intact. +3. Missing parent, binding, or selection still fails before any transport call. +4. Replacement launch crosses the same remote transport seam. +5. Worker returns no contract-change request against a frozen parent-owned file. + +Stop-back conditions for lane `B2`: + +1. Any need to reopen `agent-api-types`, `world_ops.rs`, `dispatch/prelude.rs`, `socket.rs`, or + `repl_world_agent.rs`. +2. Any need to redesign store semantics. +3. Any need to treat remote control as a local cancel handle. +4. Any need to broaden scope into doctor/toolbox behavior before the regression wall. + +Green-path output: + +- `.runs/plan-11/sentinels/task-m11-b2-shell-remote-member-cutover.ok` after parent acceptance + +### task/m11-c1-integrate-and-regression-wall + +Ownership: + +- parent only + +Scope: + +1. Integrate accepted outputs from lanes `B1` and `B2` in the parent checkout. +2. Resolve conflicts only in the parent branch. +3. Land the final status/trace/replacement regression wall in: + - [crates/shell/tests/agent_successor_contract_ahcsitc0.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_successor_contract_ahcsitc0.rs) + - [crates/shell/tests/agent_hub_trace_persistence.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_hub_trace_persistence.rs) +4. Touch + [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs) + only if a real remotely launched member exposes operator-surface drift not already covered by + existing logic. +5. Touch + [crates/shell/src/execution/agent_runtime/session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs) + only if one additive ownership marker is truly required by the remote-control split. +6. Keep + [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) + no-semantics-change by default. + +Required final validation order: + +1. `cargo test -p agent-api-types -- --nocapture` +2. `cargo test -p world-agent --test streamed_execute_cancel_v1 -- --nocapture` +3. `cargo test -p shell async_repl -- --nocapture` +4. `cargo test -p shell --test repl_world_first_routing_v1 -- --nocapture` +5. `cargo test -p shell --test agent_successor_contract_ahcsitc0 -- --nocapture` +6. `cargo test -p shell --test agent_hub_trace_persistence -- --nocapture` +7. `cargo test -p world-agent -- --nocapture` +8. `cargo test -p shell -- --nocapture` +9. `cargo fmt --all -- --check` +10. `cargo clippy --workspace --all-targets -- -D warnings` + +Acceptance: + +1. Cancel reaches real member-dispatch spans. +2. Replacement launch preserves fresh `participant_id` and correct + `resumed_from_participant_id`. +3. Status remains correct for a real remotely launched member. +4. Trace rows remain participant-correct and world-correct without reviving stale liveness. +5. Gate C passes. + +Green-path output: + +- `.runs/plan-11/sentinels/task-m11-c1-integrate-and-regression-wall.ok` + +Blocked-path output: + +- `.runs/plan-11/blocked.json` + +### task/m11-c2-docs-review + +Ownership: + +- parent only + +Scope: + +1. Review [docs/TRACE.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/TRACE.md) + and [docs/USAGE.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/USAGE.md) + only after the full regression wall is green. +2. Apply minimal wording updates only if shipped operator-visible transport behavior or trace + examples changed. +3. Otherwise record an explicit no-change decision in `run-state.json` and `session.log`. + +Acceptance: + +1. Docs are either updated minimally and truthfully or explicitly left unchanged. +2. No docs edit compensates for missing runtime or test proof. + +Green-path output: + +- `.runs/plan-11/sentinels/task-m11-c2-docs-review.ok` + +Blocked-path output: + +- `.runs/plan-11/blocked.json` + +### task/m11-c3-closeout + +Ownership: + +- parent only + +Scope: + +1. Confirm all required sentinels exist and `blocked.json` does not. +2. Finalize `run-state.json` with accepted decisions, superseded blocked-run notes, escalation + notes, and validation outcomes. +3. Write `.runs/plan-11/closeout.md` with branch state, worker-output disposition, tests run, and + remaining deferred work limited to post-`PLAN-11` items. +4. Confirm no unresolved scope creep remains into shared-crate extraction, scheduler work, public + control-plane APIs, auth-bundle redesign, or non-Linux parity. + +Acceptance: + +1. Gate D passes. +2. `.runs/plan-11/closeout.md` exists. +3. The final run-state records the refrozen contract that avoided the prior blocked failure. + +Green-path output: + +- `.runs/plan-11/sentinels/task-m11-c3-closeout.ok` + +## Context-Control Rules + +1. The parent owns `.runs/plan-11/*`. Workers do not edit `run-state.json`, sentinels, + `queue.json`, `session.log`, `blocked.json`, or `closeout.md`. +2. The parent keeps only the following live in working context: + - current task ID and gate state, + - the refrozen `member_dispatch` request contract, + - the refrozen builder-export contract, + - the refrozen readiness-event contract used by `extract_session_handle_id(...)`, + - the exact allowed and forbidden files for each worker lane, + - the latest accepted worker summaries, narrow diffs, and blockers, + - the required pre-worker and final validation orders. +3. Worker packets contain only: + - task ID, + - worktree path and branch name, + - allowed files, + - forbidden files, + - frozen invariants, + - stop-back conditions, + - exact commands to run, + - exact handoff format. +4. Each worker prompt must enumerate the exact files it may touch and the exact files it must not + touch. +5. Workers must stop immediately if they need to touch any file outside their allowed list or if + they need to reinterpret the frozen request, builder, or event contract. +6. Each worker must return: + - short result summary, + - touched files, + - exact commands run, + - test outcomes, + - blockers or unresolved assumptions, + - narrow diff summary tied only to the touched files, + - explicit statement of whether parent follow-up is required. +7. Workers do not rebase, merge, integrate each other, or update parent run artifacts. +8. The parent reviews worker summaries plus narrow diffs, not broad repo restatements. +9. The parent merges accepted work locally, reruns required validations, and then closes the lane. +10. If worker outputs conflict with current parent truth, the parent re-derives the correct result + from production code and rewrites the patch locally instead of negotiating blended semantics + across worktrees. + +## Tests And Acceptance + +### Pre-worker proof order + +Run these parent-owned checks before dispatching the worker window: + +1. `cargo test -p agent-api-types -- --nocapture` +2. `cargo test -p shell --lib -- --nocapture` +3. `cargo test -p world-agent --test streamed_execute_cancel_v1 -- --no-run` +4. `cargo test -p shell --test repl_world_first_routing_v1 -- --no-run` +5. `cargo test -p shell --test agent_successor_contract_ahcsitc0 -- --no-run` +6. `cargo test -p shell --test agent_hub_trace_persistence -- --no-run` + +### Final validation order + +Run these commands in this exact order during `task/m11-c1-integrate-and-regression-wall`: + +1. `cargo test -p agent-api-types -- --nocapture` +2. `cargo test -p world-agent --test streamed_execute_cancel_v1 -- --nocapture` +3. `cargo test -p shell async_repl -- --nocapture` +4. `cargo test -p shell --test repl_world_first_routing_v1 -- --nocapture` +5. `cargo test -p shell --test agent_successor_contract_ahcsitc0 -- --nocapture` +6. `cargo test -p shell --test agent_hub_trace_persistence -- --nocapture` +7. `cargo test -p world-agent -- --nocapture` +8. `cargo test -p shell -- --nocapture` +9. `cargo fmt --all -- --check` +10. `cargo clippy --workspace --all-targets -- -D warnings` + +### Acceptance checklist + +#### Refrozen request contract + +1. `ExecuteRequest.member_dispatch` remains additive, typed, and validated at the request + boundary. +2. `MemberDispatchRequestV1` now requires `resolved_runtime`. +3. `ResolvedMemberRuntimeDescriptorV1` carries explicit `backend_kind` and absolute + `binary_path`. +4. Ordinary process exec still uses non-empty `cmd` with `member_dispatch=None`. +5. Typed member dispatch requires empty `cmd`, `pty=false`, and a fully formed identity plus + resolved-runtime tuple. +6. Invalid mixed shapes and invalid `resolved_runtime` fail at parse time rather than in deep + runtime handlers. +7. Top-level `agent_id` remains authoritative and is not duplicated inside `member_dispatch`. + +#### Builder export and harness refreeze + +1. `world_ops.rs` builds the typed member-dispatch request without inventing a second transport + stack. +2. `dispatch/prelude.rs` re-exports the frozen builder required by the shell lane. +3. `socket.rs` can deserialize and inspect recorded `member_dispatch` payloads including nested + `resolved_runtime`. +4. `repl_world_agent.rs` can capture typed payloads and script ready, cancel, success, and + failure flows. +5. Worker lanes begin only after these conditions are proven by the pre-worker proof order. + +#### World-agent runtime lane + +1. `world-agent` branches `execute_stream(...)` on `member_dispatch` while preserving ordinary + process streaming. +2. `world-agent` depends directly on `unified-agent-api`. +3. The new member runtime manager validates binding and `resolved_runtime` before startup. +4. Remote member startup emits `Start`, a session-bearing `Event`, and terminal `Exit` or + `Error`. +5. `/v1/execute/cancel` can deliver to a live member-dispatch span. +6. Bootstrap failure and abnormal termination clean up honestly and do not leave a live claim + behind. + +#### Shell remote cutover lane + +1. The member path in `async_repl.rs` no longer reuses local host orchestrator startup. +2. The shell uses an explicit local-vs-remote retained-control split. +3. The shell uses an explicit `PreparedMemberDispatch` or equivalent dedicated remote-prepared + shape. +4. The shell persists `Allocating` before launch and only transitions to `Ready` and `Running` + after remote ownership is retained. +5. Same-generation reuse remains intact and preflight failures still occur before any transport + call. + +#### Regression wall + +1. Cancel, replacement, status, and trace correctness are proven after both runtime halves are + integrated. +2. Replacement launch on world-generation rollover uses the same typed transport seam. +3. Replacement preserves fresh `participant_id` and correct `resumed_from_participant_id`. +4. Failed replacement leaves honest absence rather than stale liveness. +5. Trace remains auditable without becoming current liveness authority. + +#### Workspace and scope boundary + +1. No new public API family is introduced. +2. No scheduler, selector UX, shared-crate extraction, auth-bundle redesign, or non-Linux + implementation is introduced. +3. Docs remain late and optional. +4. `.runs/plan-11/*` remains parent-owned only. +5. All work stays inside the `PLAN-11` architecture contract and the listed file boundaries. + +## Merge Refusal Rules + +The parent refuses to merge a worker output if any of these are true: + +1. The patch edits a file outside the task’s allowed file list. +2. The patch reopens the refrozen request, builder, or readiness-event contract. +3. The patch authorizes host fallback for a world-scoped member. +4. The patch treats remote control state as if it were a local cancel handle. +5. The patch reintroduces a second runtime resolver or avoids the direct `unified-agent-api` + dependency by falling back to backend-specific CLIs. +6. The patch weakens fail-closed behavior for binding mismatch, missing parent, cancel failure, or + replacement failure. +7. The patch requires concurrent edits to parent-owned refreeze files or late regression files to + become intelligible. +8. The patch omits test evidence for the behavior it claims to cover. +9. The patch broadens scope into shared-crate extraction, public APIs, scheduler work, early docs + work, or non-Linux parity. + +## Assumptions + +1. The branch already contains the accepted carryover transport and harness work referenced in + [PLAN-11.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-11.md), + even though the old request freeze is superseded. +2. The existing readiness contract based on the session-bearing `Event` remains acceptable for + this slice; no new stream frame family is required. +3. The resolved binary path chosen by the shell is valid input for Linux-world startup, and + failure to use it in-world is a runtime error to surface and test, not a reason to add a + second selector. +4. The current branch can accept a direct `unified-agent-api` dependency in `world-agent` without + forcing a larger dependency architecture change. +5. Parent-owned `.runs/plan-11/*` artifacts remain the execution truth even if the repo has + unrelated concurrent work elsewhere. +6. Operator surfaces should need at most additive parent-owned adjustments in + [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs); + a broader redesign would be out of scope and should block the run. + +## Run Exit Criteria + +### Successful run + +The run is successful only if all of these are true: + +1. `.runs/plan-11/sentinels/task-m11-a1-preflight-and-blocked-import.ok` exists. +2. `.runs/plan-11/sentinels/task-m11-a2-member-dispatch-contract-refreeze.ok` exists. +3. `.runs/plan-11/sentinels/task-m11-a3-builder-export-and-harness-refreeze.ok` exists. +4. `.runs/plan-11/sentinels/task-m11-b1-world-agent-member-manager.ok` exists. +5. `.runs/plan-11/sentinels/task-m11-b2-shell-remote-member-cutover.ok` exists. +6. `.runs/plan-11/sentinels/task-m11-c1-integrate-and-regression-wall.ok` exists. +7. `.runs/plan-11/sentinels/task-m11-c2-docs-review.ok` exists. +8. `.runs/plan-11/sentinels/task-m11-c3-closeout.ok` exists. +9. `.runs/plan-11/run-state.json` exists and records a completed terminal state. +10. `.runs/plan-11/queue.json` and `.runs/plan-11/session.log` exist. +11. `.runs/plan-11/closeout.md` exists and matches the final accepted branch state. +12. `.runs/plan-11/blocked.json` does not exist. + +### Blocked termination + +The run terminates as blocked only if all of these are true: + +1. `.runs/plan-11/blocked.json` exists with the triggering stop condition and current task ID. +2. No later-phase sentinel is written after the blocking condition is detected. +3. `run-state.json` records the blocked terminal state. +4. Partial worker outputs are either rejected or explicitly quarantined in `session.log`. +5. No fake completion signal is written to `.runs/plan-11/closeout.md`. diff --git a/llm-last-mile/ORCH_PLAN-11_5.md b/llm-last-mile/ORCH_PLAN-11_5.md new file mode 100644 index 000000000..c7a43c42b --- /dev/null +++ b/llm-last-mile/ORCH_PLAN-11_5.md @@ -0,0 +1,573 @@ +# ORCH_PLAN-11_5: Execute PLAN-11_5 From The Accepted Gate A/B Carryover + +Branch: `feat/session-centric-state-store` +Plan source: [PLAN-11_5.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-11_5.md) +Prior blocked-run evidence only: [.runs/plan-11/run-state.json](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/plan-11/run-state.json), [.runs/plan-11/blocked.json](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/plan-11/blocked.json) +Execution type: continuation orchestration plan, Linux-first, backend-only, status/trace truth required + +## Summary + +This document is the execution control artifact for `PLAN-11_5`. + +The continuation truth is fixed: + +1. Gate A and Gate B carryover are already accepted. +2. Those carryover files become read-only after the parent lands the missing crate-surface export in [crates/shell/src/execution/routing.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing.rs). +3. Only then may the run open the single honest parallel window: + - one `world-agent` lane, + - one shell lane. +4. The parent then performs the only integration pass, the only regression wall, and the only closeout decision. + +Fresh run control lives under `.runs/plan-11_5/*`. The old `.runs/plan-11/*` artifacts remain read-only evidence and never become active truth for this continuation. + +Canonical task IDs: + +- `task/m11_5-p0-carryover-import-and-unblock` +- `task/m11_5-l1-world-agent-member-runtime` +- `task/m11_5-l2-shell-remote-member-cutover` +- `task/m11_5-g1-worker-launch-gate` +- `task/m11_5-g2-integration-gate` +- `task/m11_5-p1-parent-integration-and-regression-wall` +- `task/m11_5-g3-closeout-gate` +- `task/m11_5-p2-closeout` + +The parent remains the only integrator, the only writer of accepted branch state on `feat/session-centric-state-store`, and the only writer of `.runs/plan-11_5/*`. + +## Hard Guards + +### Locked scope + +1. This run executes only the blocked remainder of `PLAN-11_5`. It does not restart `PLAN-11`. +2. Gate A/B carryover is accepted and frozen after `task/m11_5-p0-carryover-import-and-unblock`. +3. The parent-owned unblock seam is exactly [crates/shell/src/execution/routing.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing.rs). +4. No worker may reopen the accepted Gate A/B files: + - [crates/agent-api-types/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/agent-api-types/src/lib.rs) + - [crates/shell/src/execution/routing/dispatch/world_ops.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs) + - [crates/shell/src/execution/routing/dispatch/prelude.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/prelude.rs) + - [crates/shell/tests/support/socket.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/support/socket.rs) + - [crates/shell/tests/support/repl_world_agent.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/support/repl_world_agent.rs) +5. `POST /v1/execute/stream` and `POST /v1/execute/cancel` remain the only transport seam. +6. `ExecuteStreamFrame::{Start,Event,Exit,Error}` remain the only stream families. +7. `world-agent` owns in-world member execution, remote cancel delivery, event streaming, and completion observation only. +8. The shell remains the only canonical writer of orchestration session state, participant state, status, and trace-facing producer truth. +9. World-scoped member launch fails closed. Host fallback is forbidden. +10. `world-agent` must validate shell-supplied `world_id` and `world_generation` and reject mismatches. +11. The shell must represent retained control explicitly as local vs remote and must not treat remote member control as local `RetainedRunControl`. +12. The shell must use the exported member-dispatch builder through `crate::execution::*`; no private-module reach-in and no duplicate request assembly is allowed. +13. Linux-first remains explicit. Non-Linux member-dispatch behavior stays fail-closed in this slice. +14. No new API family, no `V2` request rename, no shared startup-crate extraction, no extra worker lane, and no status/doctor redesign are allowed in this run. +15. Workers use `GPT-5.4` with `reasoning_effort=high`. + +### File boundaries + +Parent-only: + +- [crates/shell/src/execution/routing.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing.rs) +- [crates/shell/tests/agent_successor_contract_ahcsitc0.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_successor_contract_ahcsitc0.rs) +- [crates/shell/tests/agent_hub_trace_persistence.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_hub_trace_persistence.rs) +- `.runs/plan-11_5/*` +- `.runs/task-m11_5-*/**` + +World-agent lane only: + +- [crates/world-agent/Cargo.toml](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/Cargo.toml) +- [crates/world-agent/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/lib.rs) +- [crates/world-agent/src/service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs) +- [crates/world-agent/src/member_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs) +- [crates/world-agent/tests/streamed_execute_cancel_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/tests/streamed_execute_cancel_v1.rs) + +Shell lane only: + +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) +- [crates/shell/tests/repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs) + +Escalation-only, parent-owned if the regression wall proves the planned blast radius false: + +- [crates/shell/src/execution/agent_runtime/session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs) +- [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) +- [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs) + +### Stop conditions + +Stop the run, write `.runs/plan-11_5/blocked.json`, and do not advance if any of these occur: + +1. `task/m11_5-p0-carryover-import-and-unblock` needs any file besides `routing.rs` to land the visibility bridge. +2. Either worker lane needs to edit any frozen Gate A/B file. +3. Either worker lane needs to edit the other lane's owned files. +4. The shell lane cannot consume `build_agent_client_and_member_dispatch_request(...)` through `crate::execution::*`. +5. The `world-agent` lane needs a second runtime selector, backend inference from `backend_id`, or host-side fallback to finish. +6. Status or trace truth requires unplanned production changes outside the defined escalation surfaces. +7. A third independent worker lane appears necessary before the regression wall. +8. Integration cannot preserve the parent as the only integrator. + +## Workstream Plan + +### Fresh-run state surfaces + +Canonical parent-owned state: + +- `.runs/plan-11_5/run-state.json` +- `.runs/plan-11_5/tasks.json` +- `.runs/plan-11_5/session-log.md` + +`run-state.json` is the single source of truth for: + +- `plan_id: "PLAN-11_5"` +- `plan_source: "llm-last-mile/PLAN-11_5.md"` +- `orchestration_plan_source: "llm-last-mile/ORCH_PLAN-11_5.md"` +- `branch: "feat/session-centric-state-store"` +- `current_phase` +- `active_task_ids` +- `worker_cap: 2` +- `terminal_state` +- `prior_run_evidence` +- `gate_status` +- `frozen_file_boundaries` +- `worktrees` +- `accepted_worker_outputs` +- `rejected_worker_outputs` +- `blocked_worker_outputs` +- `escalation_usage` +- `final_validation` +- `closeout_pointer` + +`tasks.json` is the ordered execution registry for: + +- task IDs, +- owner, +- worktree path, +- branch, +- allowed files, +- command gates, +- expected artifacts, +- sentinel name, +- current status. + +`session-log.md` is the append-only parent log for: + +- gate decisions, +- worker launch and return records, +- acceptance or rejection rationale, +- quarantine rationale, +- merge refusal rationale, +- final closeout or blocked termination summary. + +Required parent-owned sentinels and terminal artifacts: + +- `.runs/plan-11_5/sentinels/task-m11_5-p0-carryover-import-and-unblock.ok` +- `.runs/plan-11_5/sentinels/task-m11_5-g1-worker-launch-gate.ok` +- `.runs/plan-11_5/sentinels/task-m11_5-l1-world-agent-member-runtime.ok` +- `.runs/plan-11_5/sentinels/task-m11_5-l2-shell-remote-member-cutover.ok` +- `.runs/plan-11_5/sentinels/task-m11_5-g2-integration-gate.ok` +- `.runs/plan-11_5/sentinels/task-m11_5-p1-parent-integration-and-regression-wall.ok` +- `.runs/plan-11_5/sentinels/task-m11_5-g3-closeout-gate.ok` +- `.runs/plan-11_5/sentinels/task-m11_5-p2-closeout.ok` +- `.runs/plan-11_5/blocked.json` +- `.runs/plan-11_5/closeout.md` + +Per-task artifact directories: + +- `.runs/task-m11_5-p0-carryover-import-and-unblock/` +- `.runs/task-m11_5-g1-worker-launch-gate/` +- `.runs/task-m11_5-l1-world-agent-member-runtime/` +- `.runs/task-m11_5-l2-shell-remote-member-cutover/` +- `.runs/task-m11_5-g2-integration-gate/` +- `.runs/task-m11_5-p1-parent-integration-and-regression-wall/` +- `.runs/task-m11_5-g3-closeout-gate/` +- `.runs/task-m11_5-p2-closeout/` + +Each per-task directory contains: + +- `task.json` + - task metadata, command gates, status, and parent decision +- `summary.md` + - concise execution notes and result +- `commands.txt` + - exact commands run for that task +- `artifacts/` + - captured test output snippets or generated evidence + +Worker task directories also contain: + +- `worker-output.patch` + - the returned file-scoped patch +- `worker-report.md` + - touched files, rationale, tests, blockers + +Accepted worker outputs additionally contain: + +- `accepted.json` + - parent acceptance timestamp, integrated commit/tree reference, accepted files + +Rejected worker outputs additionally contain: + +- `rejected.json` + - parent rejection reason, violated guard or mismatch, non-integrated status +- `quarantine/` + - preserved patch and evidence retained without integration + +Blocked worker outputs additionally contain: + +- `blocked.json` + - blocking condition, required escalation, and stop instruction + +Sentinel semantics: + +1. A `.ok` sentinel means the parent validated and accepted that task or gate. +2. Missing sentinel means the task is not accepted, even if code exists in a worker tree. +3. `.runs/plan-11/*` stays read-only evidence and is never rewritten by this continuation. +4. `blocked.json` exists only for blocked termination of the fresh run. +5. `closeout.md` exists only for successful completion. + +### Worktrees and branches + +Fresh worktree root: + +- `/Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-11_5` + +Worker worktrees: + +- `/Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-11_5/world-agent-member-runtime` +- `/Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-11_5/shell-remote-member-cutover` + +Worker branches: + +- `codex/feat-session-centric-state-store-m11_5-world-agent-member-runtime` +- `codex/feat-session-centric-state-store-m11_5-shell-remote-member-cutover` + +Exact setup commands: + +```bash +mkdir -p /Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-11_5 +git worktree add /Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-11_5/world-agent-member-runtime -b codex/feat-session-centric-state-store-m11_5-world-agent-member-runtime feat/session-centric-state-store +git worktree add /Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-11_5/shell-remote-member-cutover -b codex/feat-session-centric-state-store-m11_5-shell-remote-member-cutover feat/session-centric-state-store +``` + +### Topology and concurrency + +Parent checkout: + +- current checkout on `feat/session-centric-state-store` + +Concurrency rules: + +1. Exact worker cap is `2`. +2. `task/m11_5-p0-carryover-import-and-unblock` is parent-only and must finish first. +3. The only real parallel window is: + - `task/m11_5-l1-world-agent-member-runtime` + - `task/m11_5-l2-shell-remote-member-cutover` +4. `task/m11_5-p1-parent-integration-and-regression-wall` is parent-only after both worker lanes return. +5. `task/m11_5-p2-closeout` is parent-only. +6. No additional parallelism is honest in this continuation because status/trace truth depends on integrated behavior of the two runtime halves. + +### Gate checkpoints + +`task/m11_5-g1-worker-launch-gate` + +- parent confirms `p0` is accepted, +- `routing.rs` export is landed, +- Gate A/B carryover files are still unchanged, +- both worker worktrees are seeded from the exact post-`p0` tree, +- `.runs/plan-11_5/sentinels/task-m11_5-g1-worker-launch-gate.ok` is written before any worker edits begin. + +`task/m11_5-g2-integration-gate` + +- parent confirms both worker lanes have returned, +- each worker output is classified as accepted, rejected, or blocked before merge, +- no frozen-file violation, cross-lane ownership violation, or escalation drift is unresolved, +- `.runs/plan-11_5/sentinels/task-m11_5-g2-integration-gate.ok` is written before parent integration starts. + +`task/m11_5-g3-closeout-gate` + +- parent confirms the full regression wall passed in order, +- final validation results are recorded in `run-state.json`, +- no active blocked artifact exists for the fresh run, +- `.runs/plan-11_5/sentinels/task-m11_5-g3-closeout-gate.ok` is written before closeout starts. + +### Task order + +#### `task/m11_5-p0-carryover-import-and-unblock` + +Owner: + +- parent only + +Scope: + +1. Initialize `.runs/plan-11_5/*` and the per-task artifact directories. +2. Import `.runs/plan-11/run-state.json` and `.runs/plan-11/blocked.json` as historical evidence only. +3. Freeze the accepted Gate A/B carryover files as read-only for the rest of the run. +4. Add `build_agent_client_and_member_dispatch_request` to the crate-surface re-export block in `routing.rs`. +5. Record the post-`routing.rs` tree state as the reseed point for both workers. + +Command gate: + +```bash +cargo test -p shell --lib -- --nocapture +``` + +Acceptance: + +1. Only `routing.rs` changed in this task. +2. Gate A/B carryover files are unchanged and marked frozen in `run-state.json`. +3. The command gate passes. +4. Parent writes `.runs/plan-11_5/sentinels/task-m11_5-p0-carryover-import-and-unblock.ok`. + +#### `task/m11_5-l1-world-agent-member-runtime` + +Owner: + +- worker only + +Worktree: + +- `/Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-11_5/world-agent-member-runtime` + +Branch: + +- `codex/feat-session-centric-state-store-m11_5-world-agent-member-runtime` + +Scope: + +1. Add Linux-only member runtime management in `world-agent`. +2. Route typed `member_dispatch` requests from `service.rs`. +3. Register member spans for retained control and cancel targeting. +4. Extend cancel handling to reach member spans truthfully. +5. Fail closed on binding mismatch, unsupported backend, missing runtime facts, or missing binary. + +Command gates: + +```bash +cargo test -p world-agent --test streamed_execute_cancel_v1 --no-run +cargo test -p world-agent --test streamed_execute_cancel_v1 -- --nocapture +``` + +Acceptance: + +1. The lane touches only its allowed files. +2. Both command gates pass. +3. The worker returns `worker-output.patch`, `worker-report.md`, touched files, and exact command results. +4. The lane does not request edits to frozen Gate A/B files. + +#### `task/m11_5-l2-shell-remote-member-cutover` + +Owner: + +- worker only + +Worktree: + +- `/Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-11_5/shell-remote-member-cutover` + +Branch: + +- `codex/feat-session-centric-state-store-m11_5-shell-remote-member-cutover` + +Scope: + +1. Split host-orchestrator launch from remote-member launch in `async_repl.rs`. +2. Consume the exported builder through `crate::execution::*`. +3. Model remote retained control explicitly. +4. Route startup, readiness, cancel, reuse, and replacement through the remote member path. +5. Preserve same-generation reuse and fail-closed replacement behavior. + +Command gates: + +```bash +cargo test -p shell --test repl_world_first_routing_v1 --no-run +cargo test -p shell --test repl_world_first_routing_v1 -- --nocapture +``` + +Acceptance: + +1. The lane touches only its allowed files. +2. Ready/Running remains gated on session-handle evidence, not any arbitrary stream event. +3. Both command gates pass. +4. The worker returns `worker-output.patch`, `worker-report.md`, touched files, and exact command results. +5. The lane does not request edits to frozen Gate A/B files. + +#### `task/m11_5-p1-parent-integration-and-regression-wall` + +Owner: + +- parent only + +Scope: + +1. Classify each returned worker output as accepted, rejected, or blocked. +2. Integrate accepted `l1` output first. +3. Integrate accepted `l2` output second. +4. Land status truth assertions in `agent_successor_contract_ahcsitc0.rs`. +5. Land trace truth assertions in `agent_hub_trace_persistence.rs`. +6. Prove stale generation never revives after a failed replacement. +7. Record any escalation usage explicitly in `.runs/plan-11_5/run-state.json`. + +Command gates: + +```bash +cargo test -p shell --lib -- --nocapture +cargo test -p world-agent --test streamed_execute_cancel_v1 --no-run +cargo test -p shell --test repl_world_first_routing_v1 --no-run +cargo test -p world-agent --test streamed_execute_cancel_v1 -- --nocapture +cargo test -p shell --test repl_world_first_routing_v1 -- --nocapture +cargo test -p shell --test agent_successor_contract_ahcsitc0 -- --nocapture +cargo test -p shell --test agent_hub_trace_persistence -- --nocapture +``` + +Acceptance: + +1. Parent remains the only integrator. +2. Status and trace truth are green from producer-backed evidence, not inferred state alone. +3. All command gates pass in order. +4. Parent writes `.runs/plan-11_5/sentinels/task-m11_5-p1-parent-integration-and-regression-wall.ok`. + +#### `task/m11_5-p2-closeout` + +Owner: + +- parent only + +Scope: + +1. Verify all required sentinels exist. +2. Verify final validation results are recorded in `run-state.json`. +3. Confirm `.runs/plan-11_5/blocked.json` is absent for the green path. +4. Write `.runs/plan-11_5/closeout.md`. +5. Mark `.runs/plan-11_5/run-state.json` as `completed`. + +Command gate: + +```bash +test -f .runs/plan-11_5/sentinels/task-m11_5-g3-closeout-gate.ok +``` + +Acceptance: + +1. The run ends with either a complete closeout or an earlier blocked termination, never a silent partial state. +2. Parent writes `.runs/plan-11_5/sentinels/task-m11_5-p2-closeout.ok` only on green completion. + +### Worker output handling + +Accepted worker output: + +1. Parent records the output in `accepted_worker_outputs` in `run-state.json`. +2. Parent writes the task sentinel. +3. Parent stores the integrated patch, evidence, and `accepted.json` in the task directory. +4. Only accepted output may be merged into the parent checkout. + +Rejected worker output: + +1. Parent records the output in `rejected_worker_outputs` in `run-state.json`. +2. Parent does not write the task sentinel. +3. Parent preserves the patch, report, and evidence under `quarantine/` in the task directory. +4. Rejected output is never integrated, even if parts look plausible. + +Blocked worker output: + +1. Parent records the output in `blocked_worker_outputs` in `run-state.json`. +2. Parent writes task-local `blocked.json` in the task directory. +3. Parent preserves the patch and report without integration. +4. If the block violates a hard guard, the full run terminates with `.runs/plan-11_5/blocked.json`. + +### Merge refusal rules + +The parent refuses merge and blocks the run if integrating either lane requires: + +1. reopening any frozen Gate A/B carryover file, +2. cross-lane ownership violations, +3. edits to escalation-only surfaces that were not explicitly approved before merge, +4. reclassifying rejected or blocked output as accepted without new command-gate evidence, +5. any change that breaks the parent-only integrator rule. + +## Context-Control Rules + +1. The parent keeps only the active phase, frozen invariants, open blockers, and merge state in working memory. +2. Worker prompts must include the exact task ID, worktree, branch, allowed files, forbidden files, command gates, acceptance criteria, and sentinel name. +3. Every worker prompt must specify `model: GPT-5.4` and `reasoning_effort: high`. +4. Workers may read frozen files for context but may not edit them. +5. Workers do not update `.runs/plan-11_5/*` or `.runs/task-m11_5-*/**`. +6. Workers do not paste large file dumps; they return concise rationale and exact evidence only. +7. The parent updates `run-state.json`, `tasks.json`, and `session-log.md` at every gate transition, acceptance, rejection, quarantine, or blocked stop. +8. The parent re-reads [PLAN-11_5.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-11_5.md) before accepting any escalation outside the planned blast radius. +9. Tests may tighten assertions, but no lane may "fix" behavior by weakening status or trace truth claims. +10. The old `.runs/plan-11/*` artifacts are historical inputs only and never become active truth for the fresh run. + +## Tests And Acceptance + +### Task-scoped command gates + +`task/m11_5-p0-carryover-import-and-unblock` + +```bash +cargo test -p shell --lib -- --nocapture +``` + +`task/m11_5-l1-world-agent-member-runtime` + +```bash +cargo test -p world-agent --test streamed_execute_cancel_v1 --no-run +cargo test -p world-agent --test streamed_execute_cancel_v1 -- --nocapture +``` + +`task/m11_5-l2-shell-remote-member-cutover` + +```bash +cargo test -p shell --test repl_world_first_routing_v1 --no-run +cargo test -p shell --test repl_world_first_routing_v1 -- --nocapture +``` + +`task/m11_5-p1-parent-integration-and-regression-wall` + +```bash +1. cargo test -p shell --lib -- --nocapture +2. cargo test -p world-agent --test streamed_execute_cancel_v1 --no-run +3. cargo test -p shell --test repl_world_first_routing_v1 --no-run +4. cargo test -p world-agent --test streamed_execute_cancel_v1 -- --nocapture +5. cargo test -p shell --test repl_world_first_routing_v1 -- --nocapture +6. cargo test -p shell --test agent_successor_contract_ahcsitc0 -- --nocapture +7. cargo test -p shell --test agent_hub_trace_persistence -- --nocapture +``` + +### Acceptance matrix + +| Gate | Required proof | Primary surfaces | +| --- | --- | --- | +| Carryover freeze gate | Gate A/B files remain unchanged after `p0` | `agent-api-types`, `world_ops.rs`, `prelude.rs`, socket/repl support tests | +| Worker launch gate | parent exported the builder, froze carryover, seeded both worktrees from the same post-`p0` tree | `routing.rs`, worktree records, `g1` sentinel | +| World-agent runtime gate | remote launch and cancel work from shell-resolved runtime facts | `world-agent` sources, `streamed_execute_cancel_v1.rs` | +| Shell cutover gate | world-backed member launch no longer uses the host-local member path | `async_repl.rs`, `repl_world_first_routing_v1.rs` | +| Integration gate | both worker outputs are classified and safe to integrate | task directories, `tasks.json`, `g2` sentinel | +| Status truth gate | `substrate agent status` reflects the real remote producer | `agent_successor_contract_ahcsitc0.rs` | +| Trace truth gate | trace rows remain participant-correct after remote launch, cancel, and replacement | `agent_hub_trace_persistence.rs` | +| Closeout gate | regression wall passed and no active blocked artifact remains | `run-state.json`, `g3` sentinel | + +### Run exit criteria + +Successful completion requires all of the following: + +1. `routing.rs` exports the builder at the crate surface. +2. The frozen Gate A/B carryover files remain unchanged in this continuation. +3. `world-agent` owns remote member retained control and cancel for world-scoped members. +4. `async_repl.rs` launches world-scoped members through the remote path only. +5. Same-generation reuse still avoids redundant relaunch. +6. Replacement preserves lineage and fails closed honestly. +7. Status and trace show the real remote producer. +8. All required command gates pass in order. +9. `.runs/plan-11_5/closeout.md` exists and `.runs/plan-11_5/run-state.json` is `completed`. + +Blocked completion requires all of the following: + +1. `.runs/plan-11_5/blocked.json` exists. +2. `.runs/plan-11_5/run-state.json` is `blocked`. +3. `.runs/plan-11_5/session-log.md` records the exact violated guard and stopping task. +4. Rejected or blocked worker output remains preserved but not integrated. +5. No green closeout sentinel is written. + +## Assumptions + +1. `PLAN-11_5` is correct that Gate A and Gate B carryover are accepted and should not be reopened unless a new regression proves the continuation assumptions false. +2. The current implementation branch for this work remains `feat/session-centric-state-store`. +3. The missing crate-surface export in `routing.rs` is the only parent-owned unblock required before the worker window reopens. +4. The two honest runtime seams after that unblock are exactly the `world-agent` lane and the shell lane; no third independent lane exists before the regression wall. +5. The parent can seed both workers from the exact post-`p0` tree and remain the only integrator for the rest of the run. +6. The command gates listed above remain the correct execution wall for this continuation. diff --git a/llm-last-mile/ORCH_PLAN-12.md b/llm-last-mile/ORCH_PLAN-12.md new file mode 100644 index 000000000..983133489 --- /dev/null +++ b/llm-last-mile/ORCH_PLAN-12.md @@ -0,0 +1,709 @@ +# ORCH_PLAN-12: Execute PLAN-12 From The Frozen Payload Contract And Parent-Owned Bridge Step + +Branch: `feat/session-centric-state-store` +Plan source: [PLAN-12.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-12.md) +Prior blocked-run evidence only: [.runs/plan-11_5/run-state.json](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/plan-11_5/run-state.json), [.runs/plan-11_5/blocked.json](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/plan-11_5/blocked.json), [.runs/task-m11_5-l2-shell-remote-member-cutover/blocked.json](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/task-m11_5-l2-shell-remote-member-cutover/blocked.json) +Execution type: fresh continuation orchestration plan, Linux-first, backend-only, status/trace truth required + +## Summary + +This document is the execution control artifact for `PLAN-12`. + +The orchestration truth is fixed: + +1. Step 1 is parent-only. The parent initializes the fresh run, lands the crate-surface bridge, and freezes the bridge seam before any worker starts. +2. The run then opens exactly one honest parallel window: + - one `world-agent` lane, + - one shell lane. +3. The parent remains the only integrator, the only writer of accepted branch truth on `feat/session-centric-state-store`, and the only writer of `.runs/plan-12/*`. +4. The parent performs the only integration pass, the only regression wall, and the only closeout decision after both worker lanes return. + +Canonical task IDs: + +- `task/m12-p1-parent-bridge-and-run-init` +- `task/m12-g1-worker-launch-gate` +- `task/m12-l1-world-agent-member-runtime` +- `task/m12-l2-shell-remote-member-cutover` +- `task/m12-g2-integration-gate` +- `task/m12-p2-parent-integration-and-regression-wall` +- `task/m12-g3-closeout-gate` +- `task/m12-p3-closeout` + +Fresh run control lives under `.runs/plan-12/*`. The old `.runs/plan-11_5/*` artifacts remain read-only evidence and never become active truth for this run. + +## Hard Guards + +### Frozen bridge truth + +This planning truth is mandatory and preserved exactly: + +> the payload contract stays frozen, but the parent is now explicitly authorized to thaw the crate-surface request bridge before reopening lanes. The plan chooses a preferred fix, direct re-export of MemberDispatchTransportRequest through the allowed crate surface, and pre-authorizes one fallback, a sanctioned adapter helper, so the run cannot block again on the same boundary mistake. + +Operational meaning: + +1. `MemberDispatchRequestV1` remains frozen. +2. `resolved_runtime` remains frozen. +3. `resolved_runtime.binary_path` stays absolute. +4. `resolved_runtime.backend_kind` stays explicit. +5. The top-level `agent_id` remains authoritative. +6. `protocol` remains part of the transport identity contract. +7. Step 1 uses the preferred bridge first: + - re-export `MemberDispatchTransportRequest` through [crates/shell/src/execution/routing/dispatch/prelude.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/prelude.rs) + - re-export it through [crates/shell/src/execution/routing.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing.rs) +8. If code review rejects direct type export, the parent may use one sanctioned adapter helper in [crates/shell/src/execution/routing/dispatch/world_ops.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs), then re-export that helper through `prelude.rs` and `routing.rs`. +9. No worker may reopen the bridge seam after Step 1 lands, regardless of whether the preferred bridge or the sanctioned fallback was used. + +### Locked scope + +1. This run executes only `PLAN-12`. It does not restart `PLAN-11_5`. +2. The parent-owned bridge seam is resolved in Step 1 only. +3. Workers may read bridge files for context but may not edit them after `task/m12-p1-parent-bridge-and-run-init` completes. +4. `POST /v1/execute/stream` and `POST /v1/execute/cancel` remain the only transport seam. +5. `ExecuteStreamFrame::{Start,Event,Exit,Error}` remain the only stream families. +6. `world-agent` owns in-world member execution, remote cancel delivery, event streaming, and completion observation only. +7. The shell remains the only canonical writer of orchestration session state, participant state, status, and trace-facing producer truth. +8. World-scoped member launch fails closed. Host fallback is forbidden. +9. `world-agent` must validate shell-supplied `world_id` and `world_generation` and reject mismatches. +10. The shell must represent retained control explicitly as local vs remote and must not model remote member control as local `RetainedRunControl`. +11. The shell must consume the sanctioned bridge through `crate::execution::*`; no private-module reach-in and no duplicate request assembly is allowed. +12. Linux-first remains explicit. Non-Linux member-dispatch behavior stays fail-closed in this slice. +13. No new API family, no `V2` request rename, no shared startup-crate extraction, no status/doctor redesign, and no third worker lane are allowed in this run. +14. Workers use `GPT-5.4` with `reasoning_effort=high`. + +### File boundaries + +Parent-only bridge files in Step 1, then frozen for the rest of the run: + +- [crates/shell/src/execution/routing.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing.rs) +- [crates/shell/src/execution/routing/dispatch/prelude.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/prelude.rs) +- [crates/shell/src/execution/routing/dispatch/world_ops.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs) + +Parent-only, entire run: + +- [crates/agent-api-types/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/agent-api-types/src/lib.rs) +- [crates/shell/tests/agent_successor_contract_ahcsitc0.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_successor_contract_ahcsitc0.rs) +- [crates/shell/tests/agent_hub_trace_persistence.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_hub_trace_persistence.rs) +- `.runs/plan-12/*` +- `.runs/task-m12-*/**` + +World-agent lane only: + +- [crates/world-agent/Cargo.toml](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/Cargo.toml) +- [crates/world-agent/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/lib.rs) +- [crates/world-agent/src/service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs) +- [crates/world-agent/src/member_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs) +- [crates/world-agent/tests/streamed_execute_cancel_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/tests/streamed_execute_cancel_v1.rs) + +Shell lane only: + +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) +- [crates/shell/tests/repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs) + +Escalation-only, parent-owned if the regression wall proves the planned blast radius false: + +- [crates/shell/src/execution/agent_runtime/session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs) +- [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) +- [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs) + +### Stop conditions + +Stop the run, write `.runs/plan-12/blocked.json`, and do not advance if any of these occur: + +1. Step 1 requires changing [crates/agent-api-types/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/agent-api-types/src/lib.rs). +2. Step 1 requires changing the serialized `member_dispatch` payload. +3. The shell lane still cannot construct the request through `crate::execution::*` after the Step 1 bridge lands. +4. Either worker lane needs to edit any bridge file after Step 1. +5. Either worker lane needs to edit the other lane's owned files. +6. The `world-agent` lane needs a second runtime selector, backend inference from `backend_id`, or host-side fallback. +7. Status or trace truth requires unplanned production edits outside the escalation-only surfaces. +8. A third independent worker lane appears necessary before the regression wall. +9. Integration would break the parent-only integrator rule. + +## Workstream Plan + +### Fresh-run state surfaces + +Canonical parent-owned state: + +- `.runs/plan-12/run-state.json` +- `.runs/plan-12/tasks.json` +- `.runs/plan-12/session-log.md` + +`run-state.json` is the single source of truth for: + +- `plan_id: "PLAN-12"` +- `plan_source: "llm-last-mile/PLAN-12.md"` +- `orchestration_plan_source: "llm-last-mile/ORCH_PLAN-12.md"` +- `branch: "feat/session-centric-state-store"` +- `current_phase` +- `active_task_ids` +- `worker_cap: 2` +- `terminal_state` +- `prior_run_evidence` +- `bridge_choice` +- `bridge_files_frozen_after_task` +- `gate_status` +- `worktrees` +- `accepted_worker_outputs` +- `rejected_worker_outputs` +- `blocked_worker_outputs` +- `quarantined_worker_outputs` +- `attempt_counts` +- `retry_budget_by_lane` +- `escalation_usage` +- `final_validation` +- `closeout_pointer` +- `read_only_seed_evidence` + +`tasks.json` is the ordered execution registry for: + +- task IDs +- owner +- worktree path +- branch +- allowed files +- forbidden files +- command gates +- expected artifacts +- sentinel name +- attempt number +- retry eligibility +- current status + +`session-log.md` is the append-only parent log for: + +- gate decisions +- bridge choice rationale +- prior-run evidence consulted +- worker launch and return records +- retry authorization or refusal rationale +- acceptance, rejection, quarantine, or blocked rationale +- merge refusal rationale +- final closeout or blocked termination summary + +Required parent-owned sentinels and terminal artifacts: + +- `.runs/plan-12/sentinels/task-m12-p1-parent-bridge-and-run-init.ok` +- `.runs/plan-12/sentinels/task-m12-g1-worker-launch-gate.ok` +- `.runs/plan-12/sentinels/task-m12-l1-world-agent-member-runtime.ok` +- `.runs/plan-12/sentinels/task-m12-l2-shell-remote-member-cutover.ok` +- `.runs/plan-12/sentinels/task-m12-g2-integration-gate.ok` +- `.runs/plan-12/sentinels/task-m12-p2-parent-integration-and-regression-wall.ok` +- `.runs/plan-12/sentinels/task-m12-g3-closeout-gate.ok` +- `.runs/plan-12/sentinels/task-m12-p3-closeout.ok` +- `.runs/plan-12/blocked.json` +- `.runs/plan-12/closeout.md` + +Per-task artifact directories: + +- `.runs/task-m12-p1-parent-bridge-and-run-init/` +- `.runs/task-m12-g1-worker-launch-gate/` +- `.runs/task-m12-l1-world-agent-member-runtime/` +- `.runs/task-m12-l2-shell-remote-member-cutover/` +- `.runs/task-m12-g2-integration-gate/` +- `.runs/task-m12-p2-parent-integration-and-regression-wall/` +- `.runs/task-m12-g3-closeout-gate/` +- `.runs/task-m12-p3-closeout/` + +Each per-task directory contains: + +- `task.json` +- `summary.md` +- `commands.txt` +- `artifacts/` + +Worker task directories also contain: + +- `worker-output.patch` +- `worker-report.md` + +Accepted worker outputs additionally contain: + +- `accepted.json` + +Rejected worker outputs additionally contain: + +- `rejected.json` +- `quarantine/` + +Blocked worker outputs additionally contain: + +- `blocked.json` + +Sentinel semantics: + +1. A `.ok` sentinel means the parent validated and accepted that task or gate. +2. Missing sentinel means the task is not accepted, even if code exists in a worker tree. +3. `.runs/plan-11_5/*` stays read-only evidence and is never rewritten by this run. +4. `blocked.json` exists only for blocked termination of the fresh run. +5. `closeout.md` exists only for successful completion. + +Read-only seed evidence policy: + +1. The parent and workers may consult preserved `m11_5` artifacts as read-only seed context only. +2. Allowed read-only seed inputs include: + - [.runs/task-m11_5-l1-world-agent-member-runtime/worker-output.patch](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/task-m11_5-l1-world-agent-member-runtime/worker-output.patch) + - [.runs/task-m11_5-l1-world-agent-member-runtime/worker-report.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/task-m11_5-l1-world-agent-member-runtime/worker-report.md) + - [.runs/task-m11_5-l2-shell-remote-member-cutover/worker-report.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/task-m11_5-l2-shell-remote-member-cutover/worker-report.md) + - [.runs/task-m11_5-l2-shell-remote-member-cutover/blocked.json](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/task-m11_5-l2-shell-remote-member-cutover/blocked.json) +3. These artifacts are never accepted branch truth. +4. These artifacts are never applied blindly. +5. `PLAN-12` uses them to mine logic, not hunks. + +### Worktrees and branches + +Fresh worktree root: + +- `/Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-12` + +Worker worktrees: + +- `/Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-12/world-agent-member-runtime` +- `/Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-12/shell-remote-member-cutover` + +Worker branches: + +- `codex/feat-session-centric-state-store-m12-world-agent-member-runtime` +- `codex/feat-session-centric-state-store-m12-shell-remote-member-cutover` + +Exact setup commands: + +```bash +mkdir -p /Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-12 +git worktree add /Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-12/world-agent-member-runtime -b codex/feat-session-centric-state-store-m12-world-agent-member-runtime feat/session-centric-state-store +git worktree add /Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-12/shell-remote-member-cutover -b codex/feat-session-centric-state-store-m12-shell-remote-member-cutover feat/session-centric-state-store +``` + +### Topology and concurrency + +Parent checkout: + +- current checkout on `feat/session-centric-state-store` + +Concurrency rules: + +1. Exact worker cap is `2`. +2. `task/m12-p1-parent-bridge-and-run-init` is parent-only and must finish first. +3. After `g1`, the only real parallel window is: + - `task/m12-l1-world-agent-member-runtime` + - `task/m12-l2-shell-remote-member-cutover` +4. `task/m12-p2-parent-integration-and-regression-wall` is parent-only and starts only after both worker lanes are classified at `g2`. +5. `task/m12-p3-closeout` is parent-only. +6. No extra parallelism is honest because status and trace truth depend on integrated behavior of both runtime halves. + +## Parent Task Breakdown + +### `task/m12-p1-parent-bridge-and-run-init` + +Owner: + +- parent only + +Scope: + +1. Initialize `.runs/plan-12/*` and the per-task artifact directories. +2. Import `.runs/plan-11_5/*` evidence into `run-state.json` as historical inputs only. +3. Freeze the payload contract and bridge seam ownership in `run-state.json`. +4. Land the preferred direct re-export of `MemberDispatchTransportRequest` through `prelude.rs` and `routing.rs`. +5. If code review rejects the direct re-export, land exactly one sanctioned adapter helper in `world_ops.rs`, then re-export it through `prelude.rs` and `routing.rs`. +6. Record the chosen bridge mode in `bridge_choice`. +7. Record the post-bridge tree state as the reseed point for both workers. +8. Freeze [crates/shell/src/execution/routing.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing.rs), [crates/shell/src/execution/routing/dispatch/prelude.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/prelude.rs), and [crates/shell/src/execution/routing/dispatch/world_ops.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs) for the rest of the run. + +Command gate: + +```bash +cargo test -p shell --lib -- --nocapture +``` + +Acceptance: + +1. Production-code edits in this task are limited to the parent-owned bridge seam files only. +2. `.runs/plan-12/*` and `.runs/task-m12-p1-parent-bridge-and-run-init/*` are expected parent-owned run artifacts for this task and do not count as production-code scope expansion. +3. The payload contract file remains unchanged. +4. `bridge_choice` is recorded as either `direct-re-export` or `sanctioned-adapter-helper`. +5. The command gate passes. +6. The parent writes `.runs/plan-12/sentinels/task-m12-p1-parent-bridge-and-run-init.ok`. + +### `task/m12-g1-worker-launch-gate` + +Owner: + +- parent only + +Checks: + +1. `p1` is accepted. +2. The bridge seam is frozen and recorded. +3. Both worker worktrees are seeded from the exact post-`p1` tree. +4. Worker prompts name the frozen bridge files as forbidden edits. +5. Worker prompts name the allowed read-only `m11_5` evidence files they may consult. +6. `run-state.json`, `tasks.json`, and `session-log.md` reflect the launch state. + +Acceptance: + +1. No worker starts before the gate is accepted. +2. The parent writes `.runs/plan-12/sentinels/task-m12-g1-worker-launch-gate.ok`. + +### `task/m12-g2-integration-gate` + +Owner: + +- parent only + +Checks: + +1. Both worker lanes returned. +2. Each worker output is classified as `accepted`, `rejected`, or `blocked` before integration. +3. No bridge-file reopen, cross-lane ownership violation, or escalation drift is unresolved. +4. Any rejected output is quarantined before the parent integrates anything. +5. Any blocked output is preserved and may terminate the run. +6. `g2` may go green only if both worker lanes are classified `accepted`. + +Acceptance: + +1. The parent writes `.runs/plan-12/sentinels/task-m12-g2-integration-gate.ok` only if both worker lanes are accepted and the run can legally proceed to parent integration. + +### `task/m12-p2-parent-integration-and-regression-wall` + +Owner: + +- parent only + +Scope: + +1. Integrate accepted `l1` output first. +2. Integrate accepted `l2` output second. +3. Land status truth assertions in [crates/shell/tests/agent_successor_contract_ahcsitc0.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_successor_contract_ahcsitc0.rs). +4. Land trace truth assertions in [crates/shell/tests/agent_hub_trace_persistence.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_hub_trace_persistence.rs). +5. Prove stale generation never revives after failed replacement. +6. Record any escalation usage explicitly in `.runs/plan-12/run-state.json`. + +Command gates: + +```bash +cargo test -p shell --lib -- --nocapture +cargo test -p world-agent --test streamed_execute_cancel_v1 --no-run +cargo test -p shell --test repl_world_first_routing_v1 --no-run +cargo test -p world-agent --test streamed_execute_cancel_v1 -- --nocapture +cargo test -p shell --test repl_world_first_routing_v1 -- --nocapture +cargo test -p shell --test agent_successor_contract_ahcsitc0 -- --nocapture +cargo test -p shell --test agent_hub_trace_persistence -- --nocapture +``` + +Acceptance: + +1. The parent remains the only integrator. +2. Status and trace truth are green from producer-backed evidence, not inferred state alone. +3. All command gates pass in order. +4. The parent writes `.runs/plan-12/sentinels/task-m12-p2-parent-integration-and-regression-wall.ok`. + +### `task/m12-g3-closeout-gate` + +Owner: + +- parent only + +Checks: + +1. The full regression wall passed in order. +2. Final validation results are recorded in `run-state.json`. +3. No active blocked artifact exists for the fresh run. + +Acceptance: + +1. The parent writes `.runs/plan-12/sentinels/task-m12-g3-closeout-gate.ok` before closeout starts. + +### `task/m12-p3-closeout` + +Owner: + +- parent only + +Scope: + +1. Verify all required sentinels exist. +2. Verify final validation results are recorded in `run-state.json`. +3. Confirm `.runs/plan-12/blocked.json` is absent for the green path. +4. Write `.runs/plan-12/closeout.md`. +5. Mark `.runs/plan-12/run-state.json` as `completed`. + +Command gate: + +```bash +test -f .runs/plan-12/sentinels/task-m12-g3-closeout-gate.ok +``` + +Acceptance: + +1. The run ends with either a complete closeout or an earlier blocked termination, never a silent partial state. +2. The parent writes `.runs/plan-12/sentinels/task-m12-p3-closeout.ok` only on green completion. + +## Worker Task Breakdown + +### `task/m12-l1-world-agent-member-runtime` + +Owner: + +- worker only + +Worktree: + +- `/Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-12/world-agent-member-runtime` + +Branch: + +- `codex/feat-session-centric-state-store-m12-world-agent-member-runtime` + +Scope: + +1. Add Linux-only member runtime management in `world-agent`. +2. Route typed `member_dispatch` requests from `service.rs`. +3. Register member spans for retained control and cancel targeting. +4. Extend cancel handling to reach member spans truthfully. +5. Fail closed on binding mismatch, unsupported backend, missing runtime facts, or missing binary. + +Forbidden edits: + +- all Step 1 bridge files +- all shell-lane files +- all `.runs/plan-12/*` and `.runs/task-m12-*/**` + +Command gates: + +```bash +cargo test -p world-agent --test streamed_execute_cancel_v1 --no-run +cargo test -p world-agent --test streamed_execute_cancel_v1 -- --nocapture +``` + +Acceptance: + +1. The lane touches only its allowed files. +2. Both command gates pass. +3. The worker may consult the allowed `m11_5` evidence files read-only and may not apply preserved hunks blindly. +4. The worker returns `worker-output.patch`, `worker-report.md`, touched files, and exact command results. +5. The lane does not request edits to frozen payload or bridge files. + +### `task/m12-l2-shell-remote-member-cutover` + +Owner: + +- worker only + +Worktree: + +- `/Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-12/shell-remote-member-cutover` + +Branch: + +- `codex/feat-session-centric-state-store-m12-shell-remote-member-cutover` + +Scope: + +1. Split host-orchestrator launch from remote-member launch in `async_repl.rs`. +2. Consume the sanctioned bridge through `crate::execution::*`. +3. Model remote retained control explicitly. +4. Route startup, readiness, cancel, reuse, and replacement through the remote member path. +5. Preserve same-generation reuse and fail-closed replacement behavior. + +Forbidden edits: + +- all Step 1 bridge files +- all world-agent lane files +- all `.runs/plan-12/*` and `.runs/task-m12-*/**` + +Command gates: + +```bash +cargo test -p shell --test repl_world_first_routing_v1 --no-run +cargo test -p shell --test repl_world_first_routing_v1 -- --nocapture +``` + +Acceptance: + +1. The lane touches only its allowed files. +2. `Ready` and `Running` remain gated on session-handle evidence, not any arbitrary stream event. +3. Both command gates pass. +4. The worker may consult the allowed `m11_5` evidence files read-only and may not apply preserved hunks blindly. +5. The worker returns `worker-output.patch`, `worker-report.md`, touched files, and exact command results. +6. The lane does not request edits to frozen payload or bridge files. + +## Gates + +### Gate sequencing + +1. `task/m12-p1-parent-bridge-and-run-init` +2. `task/m12-g1-worker-launch-gate` +3. `task/m12-l1-world-agent-member-runtime` and `task/m12-l2-shell-remote-member-cutover` in parallel +4. `task/m12-g2-integration-gate` +5. `task/m12-p2-parent-integration-and-regression-wall` +6. `task/m12-g3-closeout-gate` +7. `task/m12-p3-closeout` + +### Retry and redrive policy + +1. Each worker lane has a retry budget of `1` additional attempt after the first attempt. +2. The exact worker cap remains `2`; retries reuse the same lane slot and do not open a third lane. +3. The parent must classify the first attempt as `accepted`, `rejected`, or `blocked` before authorizing any retry. +4. A retry is allowed only for a lane-local failure that stayed within the lane's owned file boundary. +5. A retry is allowed only if the first attempt did not violate a hard guard. +6. A retry is allowed only if the first attempt did not drift into cross-lane ownership. +7. A retry is allowed only if the first attempt did not require new blast-radius authority. +8. Retries must stay inside the same owned file boundary and the same lane contract. +9. The parent records each attempt in `run-state.json`, `tasks.json`, and `session-log.md` with attempt count and retry rationale. +10. A retry may reuse the same task ID with `attempt=2` metadata, or use a retry suffix recorded by the parent, but the lane identity does not change. +11. If a lane exhausts its retry budget without an accepted result, the run blocks. +12. Hard-guard violation, cross-lane ownership drift, bridge-file reopen, or need for new blast-radius authority makes the lane immediately non-retryable and may block the run at once. + +### Worker prompt contract + +Each worker prompt must include: + +1. exact task ID +2. attempt number and whether it is first attempt or retry +3. worktree and branch +4. allowed files +5. forbidden files +6. command gates +7. acceptance criteria +8. the specific prior-run evidence files the worker may consult read-only + +### Worker return contract + +Each worker return must include: + +1. changed files +2. commands run with exit codes +3. blocker or unresolved assumption list +4. whether the attempt is `clean`, `retryable`, or `blocked` +5. `worker-output.patch` and `worker-report.md` + +Accepted worker output: + +1. The parent records the output in `accepted_worker_outputs` in `run-state.json`. +2. The parent writes the task sentinel. +3. The parent stores the integrated patch, evidence, and `accepted.json` in the task directory. +4. Only accepted output may be merged into the parent checkout. + +Rejected worker output: + +1. The parent records the output in `rejected_worker_outputs` and `quarantined_worker_outputs`. +2. The parent does not write the task sentinel. +3. The parent preserves the patch, report, and evidence under `quarantine/` in the task directory. +4. Rejected output is never integrated. +5. The parent classifies the rejection as retryable or non-retryable before any redrive decision. +6. A retryable rejection prevents `g2` from going green until the parent-authorized retry returns. +7. A non-retryable rejection, hard-guard violation, or exhausted retry budget blocks the full run. + +Blocked worker output: + +1. The parent records the output in `blocked_worker_outputs`. +2. The parent writes task-local `blocked.json`. +3. The parent preserves the patch and report without integration. +4. If the block violates a hard guard or prevents the regression wall from starting, the full run terminates with `.runs/plan-12/blocked.json`. + +### Merge refusal rules + +The parent refuses merge and blocks the run if integrating either lane requires: + +1. reopening any frozen payload or bridge file, +2. cross-lane ownership violations, +3. edits to escalation-only surfaces that were not explicitly approved before merge, +4. reclassifying rejected or blocked output as accepted without new command-gate evidence, +5. weakening status or trace truth assertions to force green, +6. breaking the parent-only integrator rule. + +## Context-Control Rules + +1. The parent keeps only the active phase, frozen invariants, bridge choice, open blockers, and merge state in working memory. +2. Worker prompts must include the exact task ID, attempt number, worktree, branch, allowed files, forbidden files, command gates, acceptance criteria, sentinel name, and the specific read-only prior-run evidence files allowed for that attempt. +3. Every worker prompt must specify `model: GPT-5.4` and `reasoning_effort: high`. +4. Workers may read frozen files for context but may not edit them. +5. Workers do not update `.runs/plan-12/*` or `.runs/task-m12-*/**`. +6. Workers may consult the allowed `m11_5` patch and blocked-run artifacts read-only as seed context; they are never accepted truth and never applied blindly. +7. Workers do not paste large file dumps; they return concise rationale and exact evidence only. +8. The parent updates `run-state.json`, `tasks.json`, and `session-log.md` at every gate transition, attempt launch, attempt return, acceptance, rejection, retry authorization, quarantine, or blocked stop. +9. The parent re-reads [PLAN-12.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-12.md) before approving any escalation outside the planned blast radius. +10. Tests may tighten assertions, but no lane may "fix" behavior by weakening status or trace truth claims. +11. The blocked `.runs/plan-11_5/*` artifacts remain historical inputs only and never become active truth for this run. + +## Tests And Acceptance + +### Task-scoped command gates + +`task/m12-p1-parent-bridge-and-run-init` + +```bash +cargo test -p shell --lib -- --nocapture +``` + +`task/m12-l1-world-agent-member-runtime` + +```bash +cargo test -p world-agent --test streamed_execute_cancel_v1 --no-run +cargo test -p world-agent --test streamed_execute_cancel_v1 -- --nocapture +``` + +`task/m12-l2-shell-remote-member-cutover` + +```bash +cargo test -p shell --test repl_world_first_routing_v1 --no-run +cargo test -p shell --test repl_world_first_routing_v1 -- --nocapture +``` + +`task/m12-p2-parent-integration-and-regression-wall` + +```bash +1. cargo test -p shell --lib -- --nocapture +2. cargo test -p world-agent --test streamed_execute_cancel_v1 --no-run +3. cargo test -p shell --test repl_world_first_routing_v1 --no-run +4. cargo test -p world-agent --test streamed_execute_cancel_v1 -- --nocapture +5. cargo test -p shell --test repl_world_first_routing_v1 -- --nocapture +6. cargo test -p shell --test agent_successor_contract_ahcsitc0 -- --nocapture +7. cargo test -p shell --test agent_hub_trace_persistence -- --nocapture +``` + +### Acceptance matrix + +| Gate | Required proof | Primary surfaces | +| --- | --- | --- | +| Parent bridge gate | the sanctioned crate-surface bridge is landed without payload changes | `routing.rs`, `prelude.rs`, optional `world_ops.rs`, `p1` sentinel | +| Worker launch gate | bridge files are frozen and both workers are seeded from the same post-`p1` tree | worktree records, `run-state.json`, `g1` sentinel | +| World-agent runtime gate | remote launch and cancel work from shell-resolved runtime facts | `world-agent` sources, `streamed_execute_cancel_v1.rs` | +| Shell cutover gate | world-backed member launch no longer uses the host-local member path and consumes the sanctioned bridge only | `async_repl.rs`, `repl_world_first_routing_v1.rs` | +| Integration gate | both worker outputs are classified and safe to integrate | task directories, `tasks.json`, `g2` sentinel | +| Status truth gate | `substrate agent status` reflects the real remote producer | `agent_successor_contract_ahcsitc0.rs` | +| Trace truth gate | trace rows remain participant-correct after remote launch, cancel, and replacement | `agent_hub_trace_persistence.rs` | +| Closeout gate | regression wall passed and no active blocked artifact remains | `run-state.json`, `g3` sentinel | + +### Run exit criteria + +Successful completion requires all of the following: + +1. The parent landed a sanctioned Step 1 bridge without changing the payload contract. +2. The bridge files remain frozen after Step 1. +3. `world-agent` owns remote member retained control and cancel for world-scoped members. +4. `async_repl.rs` launches world-scoped members through the remote path only. +5. Same-generation reuse still avoids redundant relaunch. +6. Replacement preserves lineage and fails closed honestly. +7. Status and trace show the real remote producer. +8. All required command gates pass in order. +9. `.runs/plan-12/closeout.md` exists and `.runs/plan-12/run-state.json` is `completed`. + +Blocked completion requires all of the following: + +1. `.runs/plan-12/blocked.json` exists. +2. `.runs/plan-12/run-state.json` is `blocked`. +3. `.runs/plan-12/session-log.md` records the exact violated guard and stopping task. +4. Rejected or blocked worker output remains preserved and quarantined without integration. +5. No green closeout sentinel is written. + +## Assumptions + +1. [PLAN-12.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-12.md) is the authoritative execution contract for this run. +2. The current implementation branch for this work remains `feat/session-centric-state-store`. +3. The only honest Step 1 parent-owned unblock is the sanctioned crate-surface bridge described in `PLAN-12`. +4. The two honest worker lanes after Step 1 are exactly the `world-agent` lane and the shell lane. +5. The parent can seed both workers from the exact post-bridge tree and remain the only integrator for the rest of the run. +6. The preserved `m11_5` worker patch and blocked shell worker artifacts are valid read-only seed context for parent and workers, but never accepted branch truth and never blindly applied. +7. The command gates listed above remain the correct execution wall for this run. diff --git a/llm-last-mile/ORCH_PLAN-13.md b/llm-last-mile/ORCH_PLAN-13.md new file mode 100644 index 000000000..e5088b94a --- /dev/null +++ b/llm-last-mile/ORCH_PLAN-13.md @@ -0,0 +1,880 @@ +# ORCH_PLAN-13: Execute PLAN-13 Through Parent-Owned Placement Preflight And Three Frozen Lanes + +Branch: `feat/session-centric-state-store` +Plan source: [PLAN-13.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-13.md) +Source SOW: [13-member-runtime-world-placement-gap-sow.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/13-member-runtime-world-placement-gap-sow.md) +Execution type: fresh orchestration plan, Linux-first, backend-only, installer-sensitive, platform-posture closeout required + +## Summary + +This document is the execution control artifact for `PLAN-13`. + +The orchestration truth is fixed: + +1. `task/m13-p1-parent-preflight-and-run-init` is parent-only and must complete before any worker starts. +2. The payload contract stays frozen for the whole run. +3. The parent is explicitly authorized to thaw the crate-surface request bridge during preflight only if a rebase reveals that the shell can no longer legally consume the frozen request builder through `crate::execution::*`. +4. The preferred bridge repair remains a direct re-export of `MemberDispatchTransportRequest` through the allowed crate surface. +5. The only pre-authorized fallback remains one sanctioned adapter helper. +6. After preflight, exactly three implementation lanes may run in parallel: + - `L1` world-agent placement core + - `L2` shell fail-closed and status/trace truth + - `L3` installer, provisioning, and docs alignment +7. The parent remains the sole integrator, the sole writer of accepted branch truth on `feat/session-centric-state-store`, and the sole writer of `.runs/plan-13/*`. +8. The run is not complete until the parent finishes the proof wall and platform-posture closeout. +9. There are no human approval gates in `PLAN-13`. The only serialized pauses are parent-owned gates and blocked termination conditions. + +Canonical task IDs: + +- `task/m13-p1-parent-preflight-and-run-init` +- `task/m13-g1-worker-launch-gate` +- `task/m13-l1-world-agent-placement-core` +- `task/m13-l2-shell-fail-closed-status-trace` +- `task/m13-l3-installer-provisioning-docs-alignment` +- `task/m13-g2-integration-gate` +- `task/m13-p2-parent-integration-and-proof-wall` +- `task/m13-g3-platform-posture-closeout-gate` +- `task/m13-p3-closeout` + +## Hard Guards + +### Frozen Payload And Bridge Truth + +1. `MemberDispatchRequestV1` stays frozen. +2. No serialized `member_dispatch` payload change is allowed. +3. No second request-construction path may be added in [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs). +4. Runtime selection stays in the shell. `world-agent` must not infer or take over runtime selection. +5. `resolved_runtime.backend_kind` remains explicit and shell-authored. +6. `resolved_runtime.binary_path` remains absolute and shell-authored. +7. `POST /v1/execute/stream` and `POST /v1/execute/cancel` remain the only transport seam. +8. `ExecuteStreamFrame::{Start,Event,Exit,Error}` remain the only stream families. +9. The shell remains the only canonical writer of orchestration session and participant state. +10. `world-agent` remains the transport owner for startup, event streaming, cancel, and terminal observation. +11. If preflight finds bridge drift, the parent may thaw only these files and only in `task/m13-p1-parent-preflight-and-run-init`: + - [routing.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing.rs) + - [prelude.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/prelude.rs) + - [world_ops.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs) +12. Preflight must use this bridge repair order: + - preferred: direct re-export of `MemberDispatchTransportRequest` through `prelude.rs` and `routing.rs` + - only fallback: one sanctioned adapter helper on the existing routing surface, then re-exported through `prelude.rs` and `routing.rs` +13. After `task/m13-p1-parent-preflight-and-run-init` is accepted, the bridge files are frozen for the rest of the run. + +### Placement Truth And Platform Posture + +1. Linux-first placement correctness is required in this slice. +2. World-scoped member launch has only two allowed outcomes: + - observably placed inside the authoritative session world + - fail-closed before the member is advertised as live +3. Host-local fallback for a selected world-scoped member is forbidden. +4. Metadata-only world binding is not acceptable proof of placement. +5. The new placement path must preserve retained control, event streaming, and cancel-by-`span_id`. +6. Same-generation reuse and replacement semantics remain shell-owned. +7. macOS Lima verification is required if the guest service contract or staged runtime changes. +8. WSL verification is required only if this run claims WSL alignment. +9. If WSL is not aligned in-slice, the required proof is explicit fail-closed posture plus docs closeout. Silent incompatibility is forbidden. +10. Worker cap is `3`. +11. All worker lanes run with `model: GPT-5.4` and `reasoning_effort: high`. + +### Run Artifacts And Serialized Gate Model + +1. `.runs/plan-13/*` and `.runs/task-m13-*/**` are run artifacts for orchestration control and evidence capture. +2. These run artifacts are not assumed git-tracked deliverables. +3. Only the parent writes `.runs/plan-13/*` and `.runs/task-m13-*/**`. +4. There are no human approval gates in this run. +5. The only serialized pauses are: + - parent-owned gates + - blocked termination conditions +6. Workers do not wait on discretionary approval once launched. They either return accepted evidence, retryable evidence, or blocked evidence. + +### Parent-Owned And Lane-Owned Boundaries + +Parent-only for the entire run: + +- [crates/agent-api-types/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/agent-api-types/src/lib.rs) +- `.runs/plan-13/*` +- `.runs/task-m13-*/**` + +Parent-only during preflight, then frozen: + +- [crates/shell/src/execution/routing.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing.rs) +- [crates/shell/src/execution/routing/dispatch/prelude.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/prelude.rs) +- [crates/shell/src/execution/routing/dispatch/world_ops.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs) + +Escalation-only, parent-owned if the proof wall reveals unplanned blast radius: + +- [crates/shell/src/execution/agent_runtime/session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs) +- [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) +- [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs) + +### Stop Conditions + +Stop the run, write `.runs/plan-13/blocked.json`, and do not advance if any of these occur: + +1. Preflight requires changing [crates/agent-api-types/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/agent-api-types/src/lib.rs). +2. Preflight requires payload-shape or serialized-payload changes. +3. Any lane requires duplicate request construction in [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs). +4. Any lane requires moving runtime selection into `world-agent`. +5. Any worker edits a frozen bridge file after preflight. +6. Any worker edits another lane's owned files. +7. `L1` cannot produce an observable Linux placement proof. +8. `L2` still leaves a warning-and-continue path for selected world-scoped members. +9. `L3` claims installer or platform alignment without updating every relevant contract author in its owned set. +10. WSL is neither aligned nor explicitly fail-closed in docs and closeout state. +11. A fourth workstream or new blast-radius authority is required before the parent proof wall. +12. The parent-only integrator rule would be violated. + +## Worktrees And Branches + +Fresh worktree root: + +- `/Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-13` + +Worker worktrees: + +- `/Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-13/world-agent-placement-core` +- `/Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-13/shell-fail-closed-status-trace` +- `/Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-13/installer-provisioning-docs-alignment` + +Worker branches: + +- `codex/feat-session-centric-state-store-m13-world-agent-placement-core` +- `codex/feat-session-centric-state-store-m13-shell-fail-closed-status-trace` +- `codex/feat-session-centric-state-store-m13-installer-provisioning-docs-alignment` + +Exact setup commands: + +```bash +mkdir -p /Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-13 +git worktree add /Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-13/world-agent-placement-core -b codex/feat-session-centric-state-store-m13-world-agent-placement-core feat/session-centric-state-store +git worktree add /Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-13/shell-fail-closed-status-trace -b codex/feat-session-centric-state-store-m13-shell-fail-closed-status-trace feat/session-centric-state-store +git worktree add /Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-13/installer-provisioning-docs-alignment -b codex/feat-session-centric-state-store-m13-installer-provisioning-docs-alignment feat/session-centric-state-store +``` + +## Topology And Concurrency + +Parent checkout: + +- current checkout on `feat/session-centric-state-store` + +Concurrency rules: + +1. Exact worker cap is `3`. +2. `task/m13-p1-parent-preflight-and-run-init` is parent-only and must finish first. +3. `task/m13-g1-worker-launch-gate` is parent-only and must accept before any worker starts. +4. The only honest parallel window is: + - `task/m13-l1-world-agent-placement-core` + - `task/m13-l2-shell-fail-closed-status-trace` + - `task/m13-l3-installer-provisioning-docs-alignment` +5. `task/m13-g2-integration-gate` is parent-only and classifies all worker outputs before any integration starts. +6. `task/m13-p2-parent-integration-and-proof-wall` is parent-only and integrates accepted output in this order: + - `L1` first + - `L2` second + - `L3` third +7. `L2` and `L3` must rebase on the accepted `L1` reality if integration reveals drift against the accepted placement-core implementation. +8. `task/m13-g3-platform-posture-closeout-gate` is parent-only. +9. `task/m13-p3-closeout` is parent-only. +10. No additional concurrency is honest because platform posture and final truth depend on integrated runtime behavior. + +## Task Registry And State Surfaces + +Canonical parent-owned state: + +- `.runs/plan-13/run-state.json` +- `.runs/plan-13/tasks.json` +- `.runs/plan-13/session-log.md` + +`run-state.json` is the single source of truth for: + +- `plan_id: "PLAN-13"` +- `plan_source: "llm-last-mile/PLAN-13.md"` +- `orchestration_plan_source: "llm-last-mile/ORCH_PLAN-13.md"` +- `sow_source: "llm-last-mile/13-member-runtime-world-placement-gap-sow.md"` +- `branch: "feat/session-centric-state-store"` +- `current_phase` +- `active_task_ids` +- `worker_cap: 3` +- `bridge_status` +- `bridge_choice` +- `bridge_files_frozen_after_task` +- `payload_contract_frozen: true` +- `placement_carrier_shape` +- `placement_helper_reuse_path` +- `wsl_posture` +- `platform_verification_required` +- `worktrees` +- `accepted_worker_outputs` +- `rejected_worker_outputs` +- `blocked_worker_outputs` +- `quarantined_worker_outputs` +- `attempt_counts` +- `retry_budget_by_lane` +- `gate_status` +- `escalation_usage` +- `final_validation` +- `platform_closeout` +- `terminal_state` + +`tasks.json` is the ordered execution registry for: + +- task ID +- owner +- worktree path +- branch +- allowed files +- forbidden files +- exact PLAN-13 excerpt pointer for the worker +- command gates +- expected artifacts +- sentinel name +- attempt number +- retry eligibility +- merge order +- current status + +`session-log.md` is the append-only parent log for: + +- preflight decisions +- bridge-thaw rationale if used +- worker launch records +- worker returns +- acceptance, rejection, quarantine, or blocked rationale +- integration order +- proof-wall results +- platform-posture closeout decision +- final completion or blocked termination + +Required sentinels: + +- `.runs/plan-13/sentinels/task-m13-p1-parent-preflight-and-run-init.ok` +- `.runs/plan-13/sentinels/task-m13-g1-worker-launch-gate.ok` +- `.runs/plan-13/sentinels/task-m13-l1-world-agent-placement-core.ok` +- `.runs/plan-13/sentinels/task-m13-l2-shell-fail-closed-status-trace.ok` +- `.runs/plan-13/sentinels/task-m13-l3-installer-provisioning-docs-alignment.ok` +- `.runs/plan-13/sentinels/task-m13-g2-integration-gate.ok` +- `.runs/plan-13/sentinels/task-m13-p2-parent-integration-and-proof-wall.ok` +- `.runs/plan-13/sentinels/task-m13-g3-platform-posture-closeout-gate.ok` +- `.runs/plan-13/sentinels/task-m13-p3-closeout.ok` + +Required terminal artifacts: + +- `.runs/plan-13/blocked.json` +- `.runs/plan-13/closeout.md` + +Per-task artifact directories: + +- `.runs/task-m13-p1-parent-preflight-and-run-init/` +- `.runs/task-m13-g1-worker-launch-gate/` +- `.runs/task-m13-l1-world-agent-placement-core/` +- `.runs/task-m13-l2-shell-fail-closed-status-trace/` +- `.runs/task-m13-l3-installer-provisioning-docs-alignment/` +- `.runs/task-m13-g2-integration-gate/` +- `.runs/task-m13-p2-parent-integration-and-proof-wall/` +- `.runs/task-m13-g3-platform-posture-closeout-gate/` +- `.runs/task-m13-p3-closeout/` + +Each task directory contains: + +- `task.json` +- `summary.md` +- `commands.txt` +- `artifacts/` + +Each worker task directory also contains: + +- `worker-output.patch` +- `worker-report.md` + +Accepted worker outputs additionally contain: + +- `accepted.json` + +Rejected worker outputs additionally contain: + +- `rejected.json` +- `quarantine/` + +Blocked worker outputs additionally contain: + +- `blocked.json` + +Sentinel semantics: + +1. A `.ok` sentinel means the parent validated and accepted the task. +2. Missing sentinel means the task is not accepted, even if code exists in a worker tree. +3. Only the parent writes `.runs/plan-13/*` and `.runs/task-m13-*/**`. +4. `blocked.json` exists only for blocked termination. +5. `closeout.md` exists only for successful completion. + +## Parent Workstreams + +### `task/m13-p1-parent-preflight-and-run-init` + +Owner: + +- parent only + +Scope: + +1. Initialize `.runs/plan-13/*` and all per-task directories. +2. Record the frozen payload contract, bridge authority, placement strategy, and platform-posture rules in `run-state.json`. +3. Confirm the shell can still legally consume the frozen request builder through `crate::execution::*`. +4. If a rebase broke that bridge, thaw and repair it using the sanctioned order: + - preferred: direct re-export of `MemberDispatchTransportRequest` + - only fallback: one sanctioned adapter helper +5. Freeze the bridge files after the repair or confirmation. +6. Record one agreed internal placement carrier shape and one agreed helper-reuse direction for `L1`. +7. Record the WSL decision as either `aligned-in-slice` or `fail-closed-docs-posture`. +8. Seed all worker worktrees from the exact post-preflight tree. + +Command gate: + +```bash +cargo test -p shell --lib -- --nocapture +``` + +Acceptance: + +1. No payload or serialized transport change occurred. +2. `bridge_choice` is recorded as `direct-re-export`, `sanctioned-adapter-helper`, or `bridge-unchanged`. +3. The bridge files are frozen after this task. +4. `wsl_posture` is recorded explicitly. +5. The parent writes `.runs/plan-13/sentinels/task-m13-p1-parent-preflight-and-run-init.ok`. + +### `task/m13-g1-worker-launch-gate` + +Owner: + +- parent only + +Checks: + +1. `p1` is accepted. +2. The bridge seam is frozen and recorded. +3. All three worker worktrees are seeded from the same post-`p1` tree. +4. Every worker prompt names only the worker’s owned file set, the exact `PLAN-13` excerpt needed, command gates, forbidden files, sentinel name, and retry budget. +5. Every worker prompt specifies `model: GPT-5.4` and `reasoning_effort: high`. +6. `run-state.json`, `tasks.json`, and `session-log.md` reflect launch state. + +Acceptance: + +1. No worker starts before this gate is accepted. +2. The parent writes `.runs/plan-13/sentinels/task-m13-g1-worker-launch-gate.ok`. + +## Worker Workstreams + +### `task/m13-l1-world-agent-placement-core` + +Owner: + +- worker only + +Worktree: + +- `/Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-13/world-agent-placement-core` + +Branch: + +- `codex/feat-session-centric-state-store-m13-world-agent-placement-core` + +Owned files: + +- [crates/world-agent/src/service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs) +- [crates/world-agent/src/member_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs) +- [crates/world-agent/src/gateway_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/gateway_runtime.rs) +- [crates/world-agent/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/lib.rs) +- [crates/world-agent/Cargo.toml](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/Cargo.toml) +- [crates/world-agent/tests/streamed_execute_cancel_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/tests/streamed_execute_cancel_v1.rs) +- one new Linux-only world-placement proof test in `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/tests/` + +Forbidden files: + +- all parent-only files +- all frozen bridge files +- all `L2` owned files +- all `L3` owned files +- `.runs/plan-13/*` +- `.runs/task-m13-*/**` + +Scope: + +1. Resolve authoritative placement facts after world validation. +2. Add a placement-aware member launcher for long-lived retained-control runtime. +3. Reuse or extract the existing world-entry and cgroup-binding logic from the gateway-runtime path instead of creating a second isolation truth. +4. Preserve `Start`, `Event`, `Exit`, `Error`, retained control, and cancel-by-`span_id`. +5. Fail closed on world mismatch, missing binary, unsupported backend, missing placement facts, or failed world entry. +6. Add one Linux-only observable placement proof test. + +Command gates: + +```bash +cargo test -p world-agent --test streamed_execute_cancel_v1 --no-run +cargo test -p world-agent --test --no-run +cargo test -p world-agent --test streamed_execute_cancel_v1 -- --nocapture +cargo test -p world-agent --test -- --nocapture +``` + +Acceptance: + +1. The lane touches only its owned files. +2. Linux placement is proven by observable runtime facts, not metadata. +3. Cancel still reaches the live placed runtime. +4. All command gates pass. +5. The worker returns changed files, exact commands run, exit codes, `worker-output.patch`, and `worker-report.md`. +6. The lane does not request payload churn, bridge reopening, or shell-owned runtime selection. + +### `task/m13-l2-shell-fail-closed-status-trace` + +Owner: + +- worker only + +Worktree: + +- `/Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-13/shell-fail-closed-status-trace` + +Branch: + +- `codex/feat-session-centric-state-store-m13-shell-fail-closed-status-trace` + +Owned files: + +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) +- [crates/shell/tests/repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs) +- [crates/shell/tests/agent_successor_contract_ahcsitc0.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_successor_contract_ahcsitc0.rs) +- [crates/shell/tests/agent_hub_trace_persistence.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_hub_trace_persistence.rs) + +Forbidden files: + +- all parent-only files +- all frozen bridge files +- all `L1` owned files +- all `L3` owned files +- `.runs/plan-13/*` +- `.runs/task-m13-*/**` + +Scope: + +1. Replace the warning-and-continue path with explicit startup failure for selected world-scoped members. +2. Preserve `Allocating -> Ready` gating on session-handle evidence only. +3. Preserve same-generation reuse behavior. +4. Preserve replacement lineage and invalidation semantics. +5. Make status and trace assertions reflect actual placed-runtime truth. +6. Forbid fallback to host-local retained control after world-scoped selection. + +Command gates: + +```bash +cargo test -p shell --test repl_world_first_routing_v1 --no-run +cargo test -p shell --test agent_successor_contract_ahcsitc0 --no-run +cargo test -p shell --test agent_hub_trace_persistence --no-run +cargo test -p shell --test repl_world_first_routing_v1 -- --nocapture +cargo test -p shell --test agent_successor_contract_ahcsitc0 -- --nocapture +cargo test -p shell --test agent_hub_trace_persistence -- --nocapture +``` + +Acceptance: + +1. The lane touches only its owned files. +2. Placement failure never yields authoritative-live status. +3. `Ready` and `Running` remain remote-evidence-backed. +4. Same-generation reuse stays a no-op. +5. Failed replacement never revives stale generation truth. +6. All command gates pass. +7. The worker returns changed files, exact commands run, exit codes, `worker-output.patch`, and `worker-report.md`. + +### `task/m13-l3-installer-provisioning-docs-alignment` + +Owner: + +- worker only + +Worktree: + +- `/Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-13/installer-provisioning-docs-alignment` + +Branch: + +- `codex/feat-session-centric-state-store-m13-installer-provisioning-docs-alignment` + +Owned files: + +- [scripts/substrate/dev-install-substrate.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/substrate/dev-install-substrate.sh) +- [scripts/substrate/install-substrate.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/substrate/install-substrate.sh) +- [scripts/linux/world-provision.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/linux/world-provision.sh) +- [scripts/mac/lima-warm.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/lima-warm.sh) +- [scripts/wsl/provision.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/wsl/provision.sh) +- [scripts/windows/wsl-warm.ps1](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/windows/wsl-warm.ps1) +- [docs/WORLD.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/WORLD.md) +- [docs/INSTALLATION.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/INSTALLATION.md) +- [docs/CONFIGURATION.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/CONFIGURATION.md) +- [docs/cross-platform/wsl_world_setup.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/cross-platform/wsl_world_setup.md) + +Forbidden files: + +- all parent-only files +- all frozen bridge files +- all `L1` owned files +- all `L2` owned files +- `.runs/plan-13/*` +- `.runs/task-m13-*/**` + +Scope: + +1. Align Linux dev-install, Linux release install, direct Linux provision, and macOS Lima guest provisioning with the final placement contract. +2. Update capability, env, RW path, socket/group, and binary-staging assumptions where required. +3. Either align WSL in-slice or make WSL explicitly fail-closed with docs and closeout posture. +4. Update operator-facing docs so the placement contract and verification steps match the shipped runtime contract. +5. Do not introduce a new installer templating system in this slice. + +Command gates: + +```bash +bash -n scripts/substrate/dev-install-substrate.sh +bash -n scripts/substrate/install-substrate.sh +bash -n scripts/linux/world-provision.sh +bash -n scripts/mac/lima-warm.sh +bash -n scripts/wsl/provision.sh +pwsh -NoProfile -Command "[void][System.Management.Automation.Language.Parser]::ParseFile('scripts/windows/wsl-warm.ps1',[ref]$null,[ref]$null)" +``` + +Acceptance: + +1. The lane touches only its owned files. +2. Every runtime-contract author in the owned set is reviewed and updated if required by the final placement contract. +3. WSL posture is explicit, not implicit. +4. All command gates pass. +5. The worker returns changed files, exact commands run, exit codes, `worker-output.patch`, and `worker-report.md`. + +## Integration And Proof Wall + +### `task/m13-g2-integration-gate` + +Owner: + +- parent only + +Checks: + +1. All three worker lanes returned. +2. Each worker output is classified as `accepted`, `rejected`, or `blocked` before integration. +3. Any rejected output is quarantined before the parent integrates anything. +4. Any blocked output is preserved and may terminate the run. +5. No cross-lane ownership violation, bridge reopen, or stop-condition drift is unresolved. +6. `g2` goes green only if `L1`, `L2`, and `L3` are all classified `accepted`. + +Acceptance: + +1. The parent writes `.runs/plan-13/sentinels/task-m13-g2-integration-gate.ok` only if all three lanes are accepted. + +### `task/m13-p2-parent-integration-and-proof-wall` + +Owner: + +- parent only + +Scope: + +1. Integrate only accepted outputs. +2. Integrate accepted `L1` output first. +3. If accepted `L2` or `L3` output conflicts with the accepted `L1` placement-core reality, bounce that lane back for rebase or apply `PLAN-13` literally. Do not invent a hybrid. +4. Re-run `L1` lane-local command gates on the parent tree if context moved during integration. +5. Integrate accepted `L2` output second, rebased on accepted `L1` reality if needed. +6. Re-run `L2` lane-local command gates on the parent tree if context moved during integration. +7. Integrate accepted `L3` output third, rebased on accepted `L1` reality if needed. +8. If accepted lane outputs disagree on placement-carrier shape, WSL posture, or runtime-contract assumptions, do not resolve the disagreement creatively in parent integration. Either: + - bounce the disagreement back to the owning lane + - or apply `PLAN-13` literally where it already specifies the answer +9. Run the full Rust proof wall in order. +10. Run Linux platform verification. +11. Run macOS Lima verification if `L3` changed guest service contract or staged runtime behavior. +12. Run WSL verification only if the run claims WSL alignment; otherwise record explicit fail-closed docs posture and verify that docs landed. +13. Record final validation and platform posture in `run-state.json` and `session-log.md`. + +Recommended verification command wall, executed in this order: + +```bash +cargo test -p world-agent --test streamed_execute_cancel_v1 --no-run +cargo test -p world-agent --test --no-run +cargo test -p shell --test repl_world_first_routing_v1 --no-run +cargo test -p shell --test agent_successor_contract_ahcsitc0 --no-run +cargo test -p shell --test agent_hub_trace_persistence --no-run +cargo test -p world-agent --test streamed_execute_cancel_v1 -- --nocapture +cargo test -p world-agent --test -- --nocapture +cargo test -p shell --test repl_world_first_routing_v1 -- --nocapture +cargo test -p shell --test agent_successor_contract_ahcsitc0 -- --nocapture +cargo test -p shell --test agent_hub_trace_persistence -- --nocapture +scripts/linux/world-provision.sh --profile release +scripts/mac/lima-warm.sh +pwsh -File scripts/windows/wsl-warm.ps1 -DistroName substrate-wsl -ProjectPath (Resolve-Path .) +``` + +Platform execution rule for that wall: + +1. `scripts/linux/world-provision.sh --profile release` is always required. +2. `scripts/mac/lima-warm.sh` is required only if the guest service contract or staged runtime changed. +3. `pwsh -File scripts/windows/wsl-warm.ps1 ...` is required only if the run claims WSL alignment. +4. If WSL alignment is not claimed, the parent must not mark WSL green from non-WSL evidence; the proof is explicit fail-closed posture plus docs closeout. + +Acceptance: + +1. The parent remains the sole integrator. +2. The full Rust proof wall passes in order. +3. Linux verification is recorded as passed. +4. macOS Lima verification is recorded as passed or not-required with rationale. +5. WSL is recorded as passed or explicit fail-closed with rationale. +6. The parent writes `.runs/plan-13/sentinels/task-m13-p2-parent-integration-and-proof-wall.ok`. + +## Context-Control Rules + +1. The parent keeps only the minimal live artifacts in working context: + - current task and gate state + - frozen invariants + - accepted lane summaries + - narrow diffs under review + - active blockers + - final proof-wall status +2. The parent does not keep full worker transcripts in live working context. +3. Worker prompts include only: + - owned file set + - exact `PLAN-13` excerpt needed for that lane + - command gates + - forbidden files + - sentinel name + - retry budget +4. Worker prompts do not include unrelated plan sections, full repository dumps, or cross-lane speculative design. +5. Workers return only: + - changed files + - commands with exit codes + - blockers or unresolved assumptions + - `worker-output.patch` + - `worker-report.md` +6. Workers do not return long transcripts, exploratory notes, or unused alternatives. +7. The parent reviews summaries plus narrow diffs, not full transcripts. +8. The parent reads only the exact diff and evidence needed to accept, reject, quarantine, or bounce a lane. +9. After a lane is merged or quarantined, the parent closes that worker immediately and removes it from active context. +10. Workers never update `.runs/plan-13/*` or `.runs/task-m13-*/**`. +11. If a lane needs new authority outside its owned boundary, the worker stops and returns blocked evidence instead of improvising. +12. The parent updates `run-state.json`, `tasks.json`, and `session-log.md` at every gate transition, launch, return, retry, classification, integration, and closeout step. + +### Worker Prompt Contract + +Every worker prompt must include: + +1. exact task ID +2. attempt number +3. worktree and branch +4. owned files only +5. forbidden files +6. exact `PLAN-13` excerpt needed for that lane +7. exact command gates +8. sentinel name +9. retry budget +10. `model: GPT-5.4` +11. `reasoning_effort: high` + +### Worker Return Contract + +Every worker return must include: + +1. changed files +2. commands run with exit codes +3. blockers or unresolved assumptions +4. whether the attempt is `clean`, `retryable`, or `blocked` +5. `worker-output.patch` +6. `worker-report.md` + +### Merge Refusal Rules + +The parent refuses merge and blocks or bounces the run if integrating a lane would require: + +1. reopening frozen bridge files after preflight +2. cross-lane ownership drift +3. new blast-radius authority not already authorized by this plan +4. weakening placement truth, status truth, or trace truth to force green +5. creative conflict resolution that contradicts `PLAN-13` +6. inventing a hybrid when lane outputs disagree on placement-carrier shape, WSL posture, or runtime-contract assumptions +7. integrating rejected or blocked output +8. bypassing the parent-only integrator rule + +## Tests And Acceptance + +### Task-Scoped Command Gates + +`task/m13-p1-parent-preflight-and-run-init` + +```bash +cargo test -p shell --lib -- --nocapture +``` + +`task/m13-l1-world-agent-placement-core` + +```bash +cargo test -p world-agent --test streamed_execute_cancel_v1 --no-run +cargo test -p world-agent --test --no-run +cargo test -p world-agent --test streamed_execute_cancel_v1 -- --nocapture +cargo test -p world-agent --test -- --nocapture +``` + +`task/m13-l2-shell-fail-closed-status-trace` + +```bash +cargo test -p shell --test repl_world_first_routing_v1 --no-run +cargo test -p shell --test agent_successor_contract_ahcsitc0 --no-run +cargo test -p shell --test agent_hub_trace_persistence --no-run +cargo test -p shell --test repl_world_first_routing_v1 -- --nocapture +cargo test -p shell --test agent_successor_contract_ahcsitc0 -- --nocapture +cargo test -p shell --test agent_hub_trace_persistence -- --nocapture +``` + +`task/m13-l3-installer-provisioning-docs-alignment` + +```bash +bash -n scripts/substrate/dev-install-substrate.sh +bash -n scripts/substrate/install-substrate.sh +bash -n scripts/linux/world-provision.sh +bash -n scripts/mac/lima-warm.sh +bash -n scripts/wsl/provision.sh +pwsh -NoProfile -Command "[void][System.Management.Automation.Language.Parser]::ParseFile('scripts/windows/wsl-warm.ps1',[ref]$null,[ref]$null)" +``` + +`task/m13-p2-parent-integration-and-proof-wall` + +```bash +1. cargo test -p world-agent --test streamed_execute_cancel_v1 --no-run +2. cargo test -p world-agent --test --no-run +3. cargo test -p shell --test repl_world_first_routing_v1 --no-run +4. cargo test -p shell --test agent_successor_contract_ahcsitc0 --no-run +5. cargo test -p shell --test agent_hub_trace_persistence --no-run +6. cargo test -p world-agent --test streamed_execute_cancel_v1 -- --nocapture +7. cargo test -p world-agent --test -- --nocapture +8. cargo test -p shell --test repl_world_first_routing_v1 -- --nocapture +9. cargo test -p shell --test agent_successor_contract_ahcsitc0 -- --nocapture +10. cargo test -p shell --test agent_hub_trace_persistence -- --nocapture +11. scripts/linux/world-provision.sh --profile release +12. scripts/mac/lima-warm.sh +13. pwsh -File scripts/windows/wsl-warm.ps1 -DistroName substrate-wsl -ProjectPath (Resolve-Path .) +``` + +### Acceptance Matrix + +| Gate | Required proof | Primary surfaces | +| --- | --- | --- | +| Preflight gate | payload frozen, bridge legal through `crate::execution::*`, WSL posture recorded | bridge files, `run-state.json`, `p1` sentinel | +| Worker launch gate | all three workers seeded from the same frozen post-`p1` tree | worktree records, `tasks.json`, `g1` sentinel | +| World-agent gate | live member runtime is observably world-placed and cancel still works | `world-agent` sources, `streamed_execute_cancel_v1.rs`, one new Linux-only world-placement proof test in `crates/world-agent/tests/` | +| Shell gate | placement failure is fail-closed and status/trace truth stays evidence-backed | `async_repl.rs`, shell tests | +| Installer gate | all contract authors align or WSL is explicitly fail-closed | scripts, docs, `L3` report | +| Integration gate | all worker outputs are accepted and safe to integrate | task directories, `g2` sentinel | +| Proof wall | Rust test wall and required platform verification pass in order | parent checkout, `p2` sentinel | +| Platform closeout gate | Linux required, macOS conditional, WSL conditional or explicit fail-closed | `run-state.json`, `g3` sentinel | +| Closeout | all required sentinels exist and terminal state is complete | `.runs/plan-13/closeout.md` | + +### Run Exit Criteria + +Successful completion requires all of the following: + +1. The payload contract remained frozen. +2. Any bridge thaw happened only in preflight and used only the sanctioned repair path. +3. Linux placement truth is proven by runtime facts. +4. Selected world-scoped member launch fails closed if placement cannot be established. +5. Same-generation reuse remains intact. +6. Failed replacement never revives stale generation truth. +7. Status and trace surfaces remain truthful. +8. Linux verification passed. +9. macOS Lima verification passed if required. +10. WSL is either aligned and verified or explicitly fail-closed with docs. +11. `.runs/plan-13/closeout.md` exists and `run-state.json` is `completed`. + +Blocked completion requires all of the following: + +1. `.runs/plan-13/blocked.json` exists. +2. `run-state.json` is `blocked`. +3. `session-log.md` records the exact violated guard and stopping task. +4. Rejected or blocked worker output is preserved without integration. +5. No green closeout sentinel is written. + +## Blocked And Closeout Artifacts + +### Retry And Redrive Policy + +1. Each worker lane has retry budget `1`. +2. Retries reuse the same lane slot and do not increase concurrency beyond `3`. +3. The parent must classify the first attempt as `accepted`, `rejected`, or `blocked` before authorizing any retry. +4. A retry is allowed only for lane-local failure inside the lane's owned files. +5. Hard-guard violation, cross-lane drift, bridge reopen, or new blast-radius authority makes the lane non-retryable. +6. The parent records every attempt, retry rationale, and final classification in `run-state.json`, `tasks.json`, and `session-log.md`. +7. If a lane exhausts retry budget without acceptance, the run blocks. + +### Acceptance, Rejection, Quarantine, And Block Rules + +Accepted worker output: + +1. The parent records the output in `accepted_worker_outputs`. +2. The parent writes the task sentinel. +3. Only accepted output may be integrated. + +Rejected worker output: + +1. The parent records the output in `rejected_worker_outputs` and `quarantined_worker_outputs`. +2. The parent writes `rejected.json` and preserves evidence under `quarantine/`. +3. Rejected output is never integrated unless a later retry returns accepted evidence. + +Blocked worker output: + +1. The parent records the output in `blocked_worker_outputs`. +2. The parent writes task-local `blocked.json`. +3. If the block prevents the proof wall or violates a hard guard, the full run terminates with `.runs/plan-13/blocked.json`. + +### `task/m13-g3-platform-posture-closeout-gate` + +Owner: + +- parent only + +Checks: + +1. `p2` is accepted. +2. Linux verification is recorded as passed. +3. macOS Lima status is recorded as passed or not-required with concrete rationale. +4. WSL status is recorded as passed or explicit fail-closed with concrete rationale. +5. No active blocked artifact remains. + +Acceptance: + +1. The parent writes `.runs/plan-13/sentinels/task-m13-g3-platform-posture-closeout-gate.ok`. + +### `task/m13-p3-closeout` + +Owner: + +- parent only + +Scope: + +1. Verify all required sentinels exist. +2. Verify `run-state.json` includes final validation and platform posture. +3. Confirm `.runs/plan-13/blocked.json` is absent on the green path. +4. Write `.runs/plan-13/closeout.md`. +5. Mark `run-state.json` as `completed`. + +Command gate: + +```bash +test -f .runs/plan-13/sentinels/task-m13-g3-platform-posture-closeout-gate.ok +``` + +Acceptance: + +1. The run ends with either complete closeout or earlier blocked termination. +2. The parent writes `.runs/plan-13/sentinels/task-m13-p3-closeout.ok` only on green completion. + +## Assumptions + +1. [PLAN-13.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-13.md) is the authoritative execution contract for this run. +2. [13-member-runtime-world-placement-gap-sow.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/13-member-runtime-world-placement-gap-sow.md) is the authoritative implementation context for the placement gap and installer-sensitive blast radius. +3. The implementation can stay inside the three planned worker lanes after parent preflight. +4. The sanctioned bridge thaw, if needed, is sufficient to restore legal consumption through `crate::execution::*` without payload churn. +5. One new Linux-only world-placement proof test in `crates/world-agent/tests/` is enough to establish real placement truth for this slice. +6. The parent can integrate `L1` first and reconcile `L2` and `L3` without reopening frozen bridge files. +7. Platform closeout can honestly distinguish `required`, `not-required`, and `explicit fail-closed` postures without inventing unsupported parity claims. +8. `.runs/plan-13/*` remain orchestration artifacts whether or not they are committed in git. diff --git a/llm-last-mile/ORCH_PLAN-14.md b/llm-last-mile/ORCH_PLAN-14.md new file mode 100644 index 000000000..e808dc4f7 --- /dev/null +++ b/llm-last-mile/ORCH_PLAN-14.md @@ -0,0 +1,901 @@ +# ORCH_PLAN-14: Execute PLAN-14 Through A Parent-Frozen Auth-Bundle Contract And Honest Two-Lane Parallelism + +Branch: `feat/session-centric-state-store` +Plan source: [PLAN-14.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-14.md) +Source SOW: [14-secret-handoff-into-the-world-gateway.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/14-secret-handoff-into-the-world-gateway.md) +Gap matrix anchor: [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md) +Reference style source: [ORCH_PLAN-13.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/ORCH_PLAN-13.md) +Execution type: fresh orchestration plan, Linux-first, backend-only, cross-crate gateway hardening with one real parallel window and parent-owned contract freeze + +## Summary + +This document is the execution control artifact for `PLAN-14`. + +The orchestration truth is fixed: + +1. `task/m14-p1-parent-contract-freeze-and-run-init` is parent-only and must complete before any worker starts. +2. The parent is the only integrator and the only writer of orchestration run-state artifacts. +3. The parent is the sole writer of accepted branch truth on `feat/session-centric-state-store`. +4. The shared auth-bundle contract is frozen before worker launch and is not reopened during lane execution. +5. The only honest parallel window is exactly two lanes after that freeze: + - `L1` world-agent auth-bundle producer + - `L2` gateway startup auth-bundle consumer +6. The regression/docs lane does not run in parallel with `L1` or `L2`. It starts only after both code lanes are accepted and integrated. +7. `GatewayLifecycleRequestV1.integrated_auth` remains unchanged for the whole run. +8. Host-side policy gating, host-side auth sourcing, and request-shape semantics remain unchanged for the whole run. +9. The only new stable env surface is `SUBSTRATE_LLM_AUTH_BUNDLE_FD`. +10. The parent freezes the exact Linux manual env-proof method during `p1` and `p3` must execute that frozen method verbatim. +11. The run is not complete until the parent finishes the full validation wall and the Linux manual proof for `sync`, `status`, and `restart`. + +Canonical task IDs: + +- `task/m14-p1-parent-contract-freeze-and-run-init` +- `task/m14-g1-worker-launch-gate` +- `task/m14-l1-world-agent-auth-bundle-producer` +- `task/m14-l2-gateway-startup-auth-bundle-consumer` +- `task/m14-g2-code-lane-integration-gate` +- `task/m14-p2-parent-code-lane-integration` +- `task/m14-g3-regression-launch-gate` +- `task/m14-l3-regression-docs-closeout` +- `task/m14-g4-validation-wall-gate` +- `task/m14-p3-parent-validation-wall-and-manual-proof` +- `task/m14-p4-closeout` + +## Hard Guards + +### Frozen Contract Truth + +1. `GatewayLifecycleRequestV1.integrated_auth` stays frozen. +2. No host policy source or precedence change is allowed. +3. No change to [crates/agent-api-types/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/agent-api-types/src/lib.rs) is allowed. +4. No change to [crates/shell/src/builtins/world_gateway.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/builtins/world_gateway.rs) is allowed. +5. The shared schema owner is [crates/common/src/gateway_auth_bundle.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/common/src/gateway_auth_bundle.rs), not `agent-api-types`. +6. The only stable pointer env name is `SUBSTRATE_LLM_AUTH_BUNDLE_FD`. +7. No second stable env pointer, alias, or duplicate constant is allowed. +8. The integrated path must not inject `SUBSTRATE_LLM_BACKEND_AUTH_*`, `OPENAI_API_KEY`, or equivalent secret-bearing values into the managed gateway process environment by default. +9. `host_only` remains a bounded compatibility path and is not redefined. +10. The gateway integrated startup order is frozen as: + - parse config without secret interpolation + - if `in_world`, read `GatewayAuthBundleV1` + - overlay integrated auth in memory + - resolve remaining non-secret env placeholders + - build `ProviderRegistry` + - start serving traffic +11. Bundle failures remain transient integrated-startup failures, not policy-denial reclassification. +12. `sync` and `restart` must redeliver a fresh bundle every time. +13. The bundle is write-once, read-once, no-disk, and closed promptly. +14. The parent-owned Linux env-proof method is frozen during `p1`; `p3` must not improvise a replacement proof method. + +### Parent-Owned And Lane-Owned Boundaries + +Parent-only for the entire run: + +- [crates/agent-api-types/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/agent-api-types/src/lib.rs) +- [crates/shell/src/builtins/world_gateway.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/builtins/world_gateway.rs) +- `.runs/plan-14/*` +- `.runs/task-m14-*/**` + +Parent-only during `p1`, then frozen: + +- [crates/common/src/gateway_auth_bundle.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/common/src/gateway_auth_bundle.rs) +- [crates/common/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/common/src/lib.rs) +- [crates/common/Cargo.toml](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/common/Cargo.toml) +- [crates/gateway/Cargo.toml](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/gateway/Cargo.toml) + +Lane-owned after `p1`: + +- `L1`: world-agent producer surfaces only +- `L2`: gateway consumer surfaces only +- `L3`: regression, shell tests, contract docs, and gap-matrix only + +### Stop Conditions + +Stop the run, write `.runs/plan-14/blocked.json`, and do not advance if any of these occur: + +1. Preflight requires changing `agent-api-types`. +2. Preflight requires changing shell host policy or auth-sourcing logic. +3. The shared auth-bundle schema spreads into `agent-api-types`. +4. A worker introduces a second stable env surface beyond `SUBSTRATE_LLM_AUTH_BUNDLE_FD`. +5. A worker reintroduces secret-env fallback on the default integrated path. +6. A worker edits another lane’s owned files. +7. A worker updates `.runs/plan-14/*` or `.runs/task-m14-*/**`. +8. `L3` attempts to encode intermediate env-based behavior as final truth. +9. The gap-matrix row is marked closed before tests, docs, and final integrated code all agree on bundle-based default delivery. +10. Shell tests are changed to encode carrier mechanics instead of policy/failure-taxonomy truth. +11. The managed gateway process environment still contains secret-bearing auth values after the integrated path is green. +12. The parent-only integrator rule would be violated. +13. More than two simultaneous workers are required before the regression/docs phase. +14. The gateway startup order would need to move auth overlay after provider construction. +15. `p1` cannot freeze an exact Linux proof method for PID discovery, env inspection, and restart fresh-instance proof. + +## Worktrees And Branches + +Fresh worktree root: + +- `/Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-14` + +Worker worktrees: + +- `/Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-14/world-agent-auth-bundle-producer` +- `/Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-14/gateway-startup-auth-bundle-consumer` +- `/Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-14/regression-docs-closeout` + +Worker branches: + +- `codex/feat-session-centric-state-store-m14-world-agent-auth-bundle-producer` +- `codex/feat-session-centric-state-store-m14-gateway-startup-auth-bundle-consumer` +- `codex/feat-session-centric-state-store-m14-regression-docs-closeout` + +Exact setup commands: + +```bash +mkdir -p /Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-14 +git worktree add /Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-14/world-agent-auth-bundle-producer -b codex/feat-session-centric-state-store-m14-world-agent-auth-bundle-producer feat/session-centric-state-store +git worktree add /Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-14/gateway-startup-auth-bundle-consumer -b codex/feat-session-centric-state-store-m14-gateway-startup-auth-bundle-consumer feat/session-centric-state-store +git worktree add /Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-14/regression-docs-closeout -b codex/feat-session-centric-state-store-m14-regression-docs-closeout feat/session-centric-state-store +``` + +### Parent Integration Surface + +Integration happens on the current checkout on `feat/session-centric-state-store`. + +No dedicated parent integration worktree is used for this run because: + +1. there is exactly one serialized integrator +2. accepted branch truth must land on the authoritative branch checkout the parent already owns +3. `.runs/plan-14/*` artifacts and accepted integration state are parent-only and are easiest to keep coherent on that authoritative checkout +4. there is no honest parallel parent-merge activity in this plan + +The parent must not apply speculative multi-lane combinations on that checkout before `g2` classification is complete. + +## Topology And Concurrency + +Parent checkout: + +- current checkout on `feat/session-centric-state-store` + +Concurrency rules: + +1. Worker cap is `2`. +2. `task/m14-p1-parent-contract-freeze-and-run-init` is parent-only and must finish first. +3. `task/m14-g1-worker-launch-gate` is parent-only and must accept before any worker starts. +4. The only real parallel window is: + - `task/m14-l1-world-agent-auth-bundle-producer` + - `task/m14-l2-gateway-startup-auth-bundle-consumer` +5. `task/m14-g2-code-lane-integration-gate` is parent-only and classifies `L1` and `L2` before integration. +6. `task/m14-p2-parent-code-lane-integration` is parent-only and integrates accepted code lanes in this order: + - `L1` first + - `L2` second +7. `task/m14-g3-regression-launch-gate` is parent-only and starts only after `p2` is accepted. +8. `task/m14-l3-regression-docs-closeout` runs alone. +9. `task/m14-g4-validation-wall-gate`, `task/m14-p3-parent-validation-wall-and-manual-proof`, and `task/m14-p4-closeout` are parent-only. +10. No other concurrency is honest because the regression floor and docs must encode the final merged contract, not intermediate lane reality. + +### Why The Worker Cap Stays `2` + +1. `PLAN-14` itself defines one safe parallel window only after the parent freezes the shared contract. +2. The producer and consumer code lanes are the only independently executable slices after that freeze. +3. The regression/docs lane depends on final integrated behavior and would create fake concurrency if started earlier. +4. Inflating to `3` would either duplicate shared-contract ownership or let tests/docs lock intermediate behavior, both of which are forbidden. + +## PLAN-14 Step Mapping + +| Orchestration task | PLAN-14 implementation step alignment | +| --- | --- | +| `task/m14-p1-parent-contract-freeze-and-run-init` | Step 1: freeze shared auth-bundle contract | +| `task/m14-l1-world-agent-auth-bundle-producer` | Step 2: replace world-agent env injection with bundle delivery | +| `task/m14-l2-gateway-startup-auth-bundle-consumer` | Step 3: refactor gateway startup to consume the bundle before provider construction | +| `task/m14-l3-regression-docs-closeout` | Step 4: rewrite regression floor and close the docs gap | +| `task/m14-p3-parent-validation-wall-and-manual-proof` and `task/m14-p4-closeout` | Step 5 and Definition of Done: validation wall, manual proof, final closeout | + +## Task Registry And State Surfaces + +Canonical parent-owned state: + +- `.runs/plan-14/run-state.json` +- `.runs/plan-14/tasks.json` +- `.runs/plan-14/session-log.md` +- `.runs/plan-14/frozen-contract.json` +- `.runs/plan-14/frozen-proof-method.json` +- `.runs/plan-14/validation-wall.md` + +`frozen-contract.json` is the single source of truth for: + +- `plan_id: "PLAN-14"` +- `plan_source: "llm-last-mile/PLAN-14.md"` +- `orchestration_plan_source: "llm-last-mile/ORCH_PLAN-14.md"` +- `branch: "feat/session-centric-state-store"` +- `schema_owner_file` +- `pointer_env_name` +- `canonical_fields_by_backend` +- `forbidden_child_env_keys` +- `startup_order` +- `failure_taxonomy` +- `host_policy_source_surface_frozen: true` +- `request_shape_surface_frozen: true` +- `host_only_compatibility_path_frozen: true` + +`frozen-proof-method.json` is the single source of truth for the Linux manual proof method: + +- `pid_discovery_command` +- `env_dump_command_template` +- `restart_fresh_instance_assertion_command_template` +- `expected_present_env_keys` +- `expected_absent_env_keys` +- `artifact_filenames` +- `proof_owner_task: "task/m14-p1-parent-contract-freeze-and-run-init"` +- `consuming_task: "task/m14-p3-parent-validation-wall-and-manual-proof"` + +Rules for `frozen-proof-method.json`: + +1. `pid_discovery_command` must be an exact parent-authored command string that returns the managed gateway PID for the integrated runtime from authoritative runtime-owned surfaces. +2. `env_dump_command_template` must be an exact parent-authored command string that inspects the managed process environment from the host side and writes it to artifacts without relying on gateway self-reporting. +3. `restart_fresh_instance_assertion_command_template` must be an exact parent-authored command string that proves pre-restart and post-restart process identity differ. +4. `p3` must execute these exact frozen commands or templates verbatim, with only the recorded PID substitution allowed where the template requires it. +5. If the proof method must change after `p1`, the run blocks and returns to planning. + +`run-state.json` is the single source of truth for: + +- `current_phase` +- `active_task_ids` +- `worker_cap: 2` +- `contract_freeze_status` +- `proof_method_freeze_status` +- `lane_status` +- `integration_order` +- `accepted_worker_outputs` +- `rejected_worker_outputs` +- `blocked_worker_outputs` +- `quarantined_worker_outputs` +- `attempt_counts` +- `retry_budget_by_lane` +- `validation_wall_status` +- `manual_proof_status` +- `terminal_state` + +`tasks.json` is the ordered execution registry for: + +- task ID +- owner +- worktree path +- branch +- allowed files +- forbidden files +- command gates +- expected artifacts +- sentinel name +- attempt number +- retry eligibility +- merge order +- current status + +Required sentinels: + +- `.runs/plan-14/sentinels/task-m14-p1-parent-contract-freeze-and-run-init.ok` +- `.runs/plan-14/sentinels/task-m14-g1-worker-launch-gate.ok` +- `.runs/plan-14/sentinels/task-m14-l1-world-agent-auth-bundle-producer.ok` +- `.runs/plan-14/sentinels/task-m14-l2-gateway-startup-auth-bundle-consumer.ok` +- `.runs/plan-14/sentinels/task-m14-g2-code-lane-integration-gate.ok` +- `.runs/plan-14/sentinels/task-m14-p2-parent-code-lane-integration.ok` +- `.runs/plan-14/sentinels/task-m14-g3-regression-launch-gate.ok` +- `.runs/plan-14/sentinels/task-m14-l3-regression-docs-closeout.ok` +- `.runs/plan-14/sentinels/task-m14-g4-validation-wall-gate.ok` +- `.runs/plan-14/sentinels/task-m14-p3-parent-validation-wall-and-manual-proof.ok` +- `.runs/plan-14/sentinels/task-m14-p4-closeout.ok` + +Per-task artifact directories: + +- `.runs/task-m14-p1-parent-contract-freeze-and-run-init/` +- `.runs/task-m14-g1-worker-launch-gate/` +- `.runs/task-m14-l1-world-agent-auth-bundle-producer/` +- `.runs/task-m14-l2-gateway-startup-auth-bundle-consumer/` +- `.runs/task-m14-g2-code-lane-integration-gate/` +- `.runs/task-m14-p2-parent-code-lane-integration/` +- `.runs/task-m14-g3-regression-launch-gate/` +- `.runs/task-m14-l3-regression-docs-closeout/` +- `.runs/task-m14-g4-validation-wall-gate/` +- `.runs/task-m14-p3-parent-validation-wall-and-manual-proof/` +- `.runs/task-m14-p4-closeout/` + +Each task directory contains: + +- `task.json` +- `summary.md` +- `commands.txt` +- `artifacts/` + +Each worker task directory also contains: + +- `worker-output.patch` +- `worker-report.md` + +## Parent Workstreams + +### `task/m14-p1-parent-contract-freeze-and-run-init` + +Owner: + +- parent only + +Scope: + +1. Initialize all `.runs/plan-14/*` and `.runs/task-m14-*/**` artifacts. +2. Record the frozen contract in `frozen-contract.json`. +3. Record the frozen Linux proof method in `frozen-proof-method.json`. +4. Author the shared schema location in `substrate-common` and export it through [crates/common/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/common/src/lib.rs). +5. Add the `substrate-common` dependency edge to [crates/gateway/Cargo.toml](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/gateway/Cargo.toml) if absent. +6. Freeze exactly one canonical field map: + - `cli:codex` -> canonical `SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_*` + - `api:openai` -> `SUBSTRATE_LLM_BACKEND_AUTH_API_OPENAI_API_KEY` +7. Freeze exactly one pointer env name: `SUBSTRATE_LLM_AUTH_BUNDLE_FD`. +8. Freeze the gateway startup order and failure taxonomy. +9. Freeze the exact Linux manual proof method by recording: + - the exact PID discovery command + - the exact env dump command template + - the exact fresh-instance assertion command template + - the exact env keys expected present and absent + - the exact artifact filenames `p3` must write +10. Seed all worker worktrees from the exact post-`p1` tree. + +Command gate: + +```bash +cargo test -p substrate-common --lib -- --nocapture +cargo test -p world-agent gateway_runtime --no-run +cargo test -p substrate-gateway --no-run +``` + +Acceptance: + +1. No change to shell host policy/auth-sourcing surfaces occurred. +2. No change to `agent-api-types` occurred. +3. `substrate-common` is the only schema owner. +4. The shared contract is written once and frozen. +5. The Linux proof method is written once and frozen in `.runs/plan-14/frozen-proof-method.json`. +6. The parent writes `.runs/plan-14/sentinels/task-m14-p1-parent-contract-freeze-and-run-init.ok`. + +### `task/m14-g1-worker-launch-gate` + +Owner: + +- parent only + +Checks: + +1. `p1` is accepted. +2. All worker worktrees were seeded from the same post-`p1` tree. +3. Every worker prompt names only its owned file set, exact command gates, forbidden files, sentinel, retry budget, and narrow output contract. +4. Every worker prompt specifies `model: GPT-5.4` and `reasoning_effort: high`. +5. `run-state.json`, `tasks.json`, and `session-log.md` reflect launch state. + +Acceptance: + +1. No worker starts before this gate is accepted. +2. The parent writes `.runs/plan-14/sentinels/task-m14-g1-worker-launch-gate.ok`. + +## Worker Workstreams + +### `task/m14-l1-world-agent-auth-bundle-producer` + +Owner: + +- worker only + +Worktree: + +- `/Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-14/world-agent-auth-bundle-producer` + +Branch: + +- `codex/feat-session-centric-state-store-m14-world-agent-auth-bundle-producer` + +Owned files: + +- [crates/world-agent/src/gateway_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/gateway_runtime.rs) +- [crates/world-agent/src/service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs) +- [crates/world-agent/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/lib.rs) + +Forbidden files: + +- all parent-only files +- all `L2` owned files +- all `L3` owned files +- `.runs/plan-14/*` +- `.runs/task-m14-*/**` + +Scope: + +1. Replace env-oriented integrated handoff modeling with `GatewayAuthBundleV1` consumption from `substrate-common`. +2. Create the inherited FD/pipe channel during runtime start. +3. Serialize the auth bundle once and pass only the read end to the managed gateway child. +4. Export only `SUBSTRATE_LLM_AUTH_BUNDLE_FD`. +5. Remove default integrated secret-env injection from the child environment. +6. Preserve `host_only` bounded compatibility behavior. +7. Preserve lifecycle semantics, restart/rotation behavior, and failure taxonomy. + +Command gates: + +```bash +cargo test -p world-agent gateway_runtime --no-run +cargo test -p world-agent gateway_runtime -- --nocapture +``` + +Acceptance: + +1. The lane touches only its owned files. +2. The default integrated path no longer injects secret-bearing env values into the child. +3. `sync` and `restart` both flow through fresh bundle creation. +4. The lane does not alter request shape, shell policy, or schema ownership. +5. All command gates pass. +6. The worker returns changed files, commands run with exit codes, blockers or unresolved assumptions, `worker-output.patch`, and `worker-report.md`. + +### `task/m14-l2-gateway-startup-auth-bundle-consumer` + +Owner: + +- worker only + +Worktree: + +- `/Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-14/gateway-startup-auth-bundle-consumer` + +Branch: + +- `codex/feat-session-centric-state-store-m14-gateway-startup-auth-bundle-consumer` + +Owned files: + +- [crates/gateway/src/main.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/gateway/src/main.rs) +- [crates/gateway/src/cli/mod.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/gateway/src/cli/mod.rs) +- [crates/gateway/src/auth/codex_auth_context.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/gateway/src/auth/codex_auth_context.rs) +- [crates/gateway/src/launch.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/gateway/src/launch.rs) +- [crates/gateway/src/server/mod.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/gateway/src/server/mod.rs) +- [crates/gateway/src/providers/registry.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/gateway/src/providers/registry.rs) + +Forbidden files: + +- all parent-only files +- all `L1` owned files +- all `L3` owned files +- `.runs/plan-14/*` +- `.runs/task-m14-*/**` + +Scope: + +1. Refactor integrated startup into the frozen two-phase order. +2. Read `GatewayAuthBundleV1` exactly once from `SUBSTRATE_LLM_AUTH_BUNDLE_FD`. +3. Close the FD promptly on success and failure. +4. Build a startup-owned in-memory integrated auth context before provider construction. +5. Overlay `cli:codex` and `api:openai` integrated auth in memory. +6. Preserve `host_only` behavior and non-secret env resolution behavior. +7. Fail closed for missing pointer env, unreadable FD, malformed JSON, or missing required fields. +8. Forbid silent fallback to env-based secret delivery in integrated mode. + +Command gates: + +```bash +cargo test -p substrate-gateway --no-run +cargo test -p substrate-gateway codex_auth_context -- --nocapture +``` + +Acceptance: + +1. The lane touches only its owned files. +2. Integrated `cli:codex` no longer depends on process env. +3. Integrated `api:openai` no longer depends on secret env interpolation during config load. +4. `host_only` remains unchanged. +5. No secret-env compatibility fallback is left on by default. +6. All command gates pass. +7. The worker returns changed files, commands run with exit codes, blockers or unresolved assumptions, `worker-output.patch`, and `worker-report.md`. + +## Code-Lane Integration + +### `task/m14-g2-code-lane-integration-gate` + +Owner: + +- parent only + +Checks: + +1. `L1` and `L2` both returned. +2. Each output is classified as `accepted`, `rejected`, or `blocked` before integration. +3. Any rejected output is quarantined before integration. +4. Any blocked output is preserved and may terminate the run. +5. No lane violated contract freeze, schema ownership, or file ownership. +6. `g2` goes green only if both code lanes are classified `accepted`. + +Acceptance: + +1. The parent writes `.runs/plan-14/sentinels/task-m14-g2-code-lane-integration-gate.ok` only if both code lanes are accepted. + +### `task/m14-p2-parent-code-lane-integration` + +Owner: + +- parent only + +Scope: + +1. Integrate only accepted outputs. +2. Integrate `L1` first. +3. Re-run `L1` lane-local command gates on the parent tree. +4. Integrate `L2` second. +5. Re-run `L2` lane-local command gates on the parent tree. +6. If the later-integrated lane merges mechanically but reveals semantic drift against the frozen contract, the parent must not keep that combined state as tentative truth. +7. For semantic drift on schema ownership, startup order, or fallback posture, the parent must do exactly one of: + - apply the frozen `PLAN-14` contract literally if one lane is already correct and the other lane is plainly non-conforming + - quarantine and bounce the offending lane back to its worker slot with an explicit semantic-drift report +8. The parent must not invent hybrid truth during integration. +9. If `L1` and `L2` each look locally correct but together reveal contradictory semantics, the parent quarantines the later-integrated lane first, records the semantic conflict in `.runs/task-m14-p2-parent-code-lane-integration/artifacts/semantic-drift.md`, and bounces that lane for retry. +10. If the bounced lane returns and the semantic drift remains, the run blocks instead of synthesizing a compromise. +11. If either lane attempts to move schema ownership out of `substrate-common`, move auth overlay after provider construction, or leave default integrated secret-env fallback enabled, the parent bounces or blocks that lane instead of merging around it. +12. Record accepted code-lane truth in `run-state.json` and `session-log.md`. + +Command gate: + +```bash +cargo test -p world-agent gateway_runtime --no-run +cargo test -p substrate-gateway --no-run +cargo test -p substrate-gateway codex_auth_context -- --nocapture +``` + +Acceptance: + +1. The parent remains the sole integrator. +2. Integrated code-lane truth matches the frozen contract. +3. No hybrid truth was invented during integration. +4. The parent writes `.runs/plan-14/sentinels/task-m14-p2-parent-code-lane-integration.ok`. + +## Regression And Docs Lane + +### `task/m14-g3-regression-launch-gate` + +Owner: + +- parent only + +Checks: + +1. `p2` is accepted. +2. The regression/docs worktree is reseeded or rebased to the exact post-`p2` tree. +3. The worker prompt names only final regression/docs ownership, not intermediate behavior. +4. The worker prompt forbids encoding env-based secret delivery as final truth. + +Acceptance: + +1. The parent writes `.runs/plan-14/sentinels/task-m14-g3-regression-launch-gate.ok`. + +### `task/m14-l3-regression-docs-closeout` + +Owner: + +- worker only + +Worktree: + +- `/Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-14/regression-docs-closeout` + +Branch: + +- `codex/feat-session-centric-state-store-m14-regression-docs-closeout` + +Owned files: + +- [crates/world-agent/tests/gateway_runtime_parity.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/tests/gateway_runtime_parity.rs) +- [crates/gateway/tests/openai_shared_parity.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/gateway/tests/openai_shared_parity.rs) +- [crates/shell/tests/world_gateway.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/world_gateway.rs) +- [docs/contracts/substrate-gateway-policy-evaluation.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/contracts/substrate-gateway-policy-evaluation.md) +- [crates/gateway/docs/contracts/chatgpt-codex-auth-handoff-contract.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/gateway/docs/contracts/chatgpt-codex-auth-handoff-contract.md) +- [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md) + +Forbidden files: + +- all parent-only files +- all `L1` owned files +- all `L2` owned files +- `.runs/plan-14/*` +- `.runs/task-m14-*/**` + +Scope: + +1. Rewrite parity tests to assert pointer-env presence and secret-env absence. +2. Prove read-once behavior and fresh-bundle restart/rotation behavior. +3. Keep shell precedence and failure-taxonomy tests green without changing shell production semantics. +4. Shell tests may be updated only to preserve policy/failure-taxonomy truth or user-visible lifecycle classification truth. They must not encode carrier mechanics on the shell side. +5. Update docs so auth-bundle delivery is the landed default integrated path. +6. The gap-matrix row may be marked closed only if final integrated code, final tests, and final docs all agree on bundle-based default delivery. + +Command gates: + +```bash +cargo test -p world-agent --test gateway_runtime_parity --no-run +cargo test -p substrate-gateway openai_shared_parity --no-run +cargo test -p shell --test world_gateway --no-run +cargo test -p world-agent --test gateway_runtime_parity -- --nocapture +cargo test -p substrate-gateway openai_shared_parity -- --nocapture +cargo test -p shell --test world_gateway -- --nocapture +``` + +Acceptance: + +1. The lane touches only its owned files. +2. Tests assert final bundle-based behavior only. +3. Shell tests preserve policy/failure-taxonomy truth and do not encode carrier mechanics. +4. Docs no longer describe env-based secret injection as the current default integrated path. +5. The [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md) row is not marked closed unless tests, docs, and final integrated code all agree. +6. All command gates pass. +7. The worker returns changed files, commands run with exit codes, blockers or unresolved assumptions, `worker-output.patch`, and `worker-report.md`. + +## Validation Wall And Closeout + +### `task/m14-g4-validation-wall-gate` + +Owner: + +- parent only + +Checks: + +1. `L3` returned and is classified before final validation. +2. `L3` is `accepted`. +3. No blocked or quarantined output remains unresolved. +4. The final command wall and manual proof plan are written to `validation-wall.md`. +5. `validation-wall.md` references the exact frozen proof method from `.runs/plan-14/frozen-proof-method.json`. + +Acceptance: + +1. The parent writes `.runs/plan-14/sentinels/task-m14-g4-validation-wall-gate.ok`. + +### `task/m14-p3-parent-validation-wall-and-manual-proof` + +Owner: + +- parent only + +Scope: + +1. Integrate only accepted `L3` output. +2. Run the full validation wall in exact order. +3. Run the Linux manual proof for `sync`, `status`, and `restart`. +4. Execute the exact proof method frozen during `p1`; do not improvise or replace it. +5. Capture proof artifacts for env absence and fresh restart behavior using the artifact filenames frozen in `.runs/plan-14/frozen-proof-method.json`. +6. Record final results in `run-state.json`, `validation-wall.md`, and `session-log.md`. + +Required validation commands, executed in this order: + +```bash +cargo test -p world-agent gateway_runtime -- --nocapture +cargo test -p world-agent --test gateway_runtime_parity -- --nocapture +cargo test -p substrate-gateway codex_auth_context -- --nocapture +cargo test -p substrate-gateway openai_shared_parity -- --nocapture +cargo test -p shell --test world_gateway -- --nocapture +substrate world gateway sync +substrate world gateway status --json +substrate world gateway restart +substrate world gateway status --json +``` + +Required manual proof artifacts under `.runs/task-m14-p3-parent-validation-wall-and-manual-proof/artifacts/`: + +- `status-after-sync.json` +- `status-after-restart.json` +- `pid-before-restart.txt` +- `pid-after-restart.txt` +- `env-before-restart.txt` +- `env-after-restart.txt` +- `manual-proof-summary.md` + +Required manual proof execution rules: + +1. Run `pid_discovery_command` from `.runs/plan-14/frozen-proof-method.json` after `sync` and write `pid-before-restart.txt`. +2. Run `env_dump_command_template` from `.runs/plan-14/frozen-proof-method.json` against that PID and write `env-before-restart.txt`. +3. Run `substrate world gateway restart`. +4. Run `pid_discovery_command` again and write `pid-after-restart.txt`. +5. Run `env_dump_command_template` against the post-restart PID and write `env-after-restart.txt`. +6. Run `restart_fresh_instance_assertion_command_template` exactly as frozen and record the result in `manual-proof-summary.md`. +7. If any exact frozen proof command cannot be executed as recorded, the run blocks instead of substituting a different proof. + +Required manual assertions: + +1. `status --json` reaches `available` after `sync`. +2. `status --json` reaches `available` after `restart`. +3. The managed gateway runtime exposes `SUBSTRATE_LLM_AUTH_BUNDLE_FD`. +4. The managed gateway runtime does not expose: + - `SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCOUNT_ID` + - `SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCESS_TOKEN` + - `SUBSTRATE_LLM_BACKEND_AUTH_API_OPENAI_API_KEY` + - `OPENAI_API_KEY` +5. Restart produces a fresh process instance with fresh auth-bundle delivery. + +Acceptance: + +1. The full validation wall passes in order. +2. The Linux manual proof is recorded and green. +3. The manual proof used the exact frozen proof method from `p1`. +4. The parent writes `.runs/plan-14/sentinels/task-m14-p3-parent-validation-wall-and-manual-proof.ok`. + +### `task/m14-p4-closeout` + +Owner: + +- parent only + +Scope: + +1. Verify all required sentinels exist. +2. Verify `run-state.json` includes final validation and manual proof status. +3. Verify `.runs/plan-14/blocked.json` is absent on the green path. +4. Verify the gap-matrix closeout state matches final code, tests, and docs. +5. Write `.runs/plan-14/closeout.md`. +6. Mark `run-state.json` as `completed`. + +Command gate: + +```bash +test -f .runs/plan-14/sentinels/task-m14-p3-parent-validation-wall-and-manual-proof.ok +``` + +Acceptance: + +1. The run ends with either complete closeout or earlier blocked termination. +2. The parent writes `.runs/plan-14/sentinels/task-m14-p4-closeout.ok` only on green completion. + +## Context-Control Rules + +1. The parent keeps only the minimal live artifacts in working context: + - frozen contract + - frozen proof method + - active task state + - accepted lane summaries + - narrow diffs + - blockers + - validation-wall status +2. The parent does not keep full worker transcripts in live context. +3. Worker prompts include only: + - task ID + - attempt number + - worktree and branch + - owned files + - forbidden files + - exact command gates + - sentinel name + - retry budget + - `model: GPT-5.4` + - `reasoning_effort: high` +4. Workers return only: + - changed files + - commands run with exit codes + - blockers or unresolved assumptions + - attempt classification: `clean`, `retryable`, or `blocked` + - `worker-output.patch` + - `worker-report.md` +5. The parent reviews summaries plus narrow diffs, not full transcripts. +6. After a lane is merged or quarantined, the parent closes that worker immediately. +7. Workers never update run-state artifacts. +8. If a lane needs authority outside its boundary, it stops and returns blocked evidence. + +## Merge Refusal Rules + +The parent refuses merge and blocks or bounces the run if integrating a lane would require: + +1. changing `agent-api-types` +2. changing shell host policy or host auth sourcing +3. reopening parent-frozen shared schema files after `p1` +4. cross-lane ownership drift +5. a second stable env surface +6. secret-env fallback on the default integrated path +7. docs or tests encoding intermediate behavior as final truth +8. shell tests encoding carrier mechanics instead of policy/failure-taxonomy truth +9. creative conflict resolution that contradicts `PLAN-14` +10. hybrid truth on schema ownership, startup order, or fallback posture +11. integrating rejected or blocked output +12. bypassing the parent-only integrator rule + +## Retry And Redrive Policy + +1. Each worker lane has retry budget `1`. +2. Retries reuse the same lane slot and do not increase concurrency beyond `2`. +3. The parent must classify the first attempt before authorizing a retry. +4. A retry is allowed only for lane-local failure within owned files. +5. Contract-freeze violations, ownership drift, schema drift, or secret-env fallback are non-retryable. +6. If any lane exhausts retry budget without acceptance, the run blocks. + +## Tests And Acceptance Matrix + +### Task-Scoped Command Gates + +`task/m14-p1-parent-contract-freeze-and-run-init` + +```bash +cargo test -p substrate-common --lib -- --nocapture +cargo test -p world-agent gateway_runtime --no-run +cargo test -p substrate-gateway --no-run +``` + +`task/m14-l1-world-agent-auth-bundle-producer` + +```bash +cargo test -p world-agent gateway_runtime --no-run +cargo test -p world-agent gateway_runtime -- --nocapture +``` + +`task/m14-l2-gateway-startup-auth-bundle-consumer` + +```bash +cargo test -p substrate-gateway --no-run +cargo test -p substrate-gateway codex_auth_context -- --nocapture +``` + +`task/m14-l3-regression-docs-closeout` + +```bash +cargo test -p world-agent --test gateway_runtime_parity --no-run +cargo test -p substrate-gateway openai_shared_parity --no-run +cargo test -p shell --test world_gateway --no-run +cargo test -p world-agent --test gateway_runtime_parity -- --nocapture +cargo test -p substrate-gateway openai_shared_parity -- --nocapture +cargo test -p shell --test world_gateway -- --nocapture +``` + +`task/m14-p3-parent-validation-wall-and-manual-proof` + +```bash +cargo test -p world-agent gateway_runtime -- --nocapture +cargo test -p world-agent --test gateway_runtime_parity -- --nocapture +cargo test -p substrate-gateway codex_auth_context -- --nocapture +cargo test -p substrate-gateway openai_shared_parity -- --nocapture +cargo test -p shell --test world_gateway -- --nocapture +substrate world gateway sync +substrate world gateway status --json +substrate world gateway restart +substrate world gateway status --json +``` + +### Acceptance Matrix + +| Gate | Required proof | Primary surfaces | +| --- | --- | --- | +| Contract freeze | one schema owner, one pointer env name, one startup order, frozen request/policy surfaces, frozen Linux proof method | `substrate-common`, `frozen-contract.json`, `frozen-proof-method.json`, `p1` sentinel | +| Worker launch | both code lanes seeded from same frozen tree | worktrees, `tasks.json`, `g1` sentinel | +| Producer lane | `world-agent` writes bundle, passes only pointer env, preserves restart semantics | `gateway_runtime.rs`, `L1` report | +| Consumer lane | gateway reads bundle once before provider construction and does not depend on secret env | gateway startup surfaces, `L2` report | +| Code integration | both code lanes accepted and parent-integrated in order without hybrid truth | parent checkout, `g2` and `p2` sentinels | +| Regression/docs launch | regression lane starts only after merged code truth exists | `g3` sentinel | +| Regression/docs lane | parity tests and docs encode final bundle-based truth only; shell tests stay policy/failure-taxonomy-only | tests, docs, gap matrix, `L3` report | +| Validation wall | required command wall passes in exact order | parent checkout, `g4` and `p3` sentinels | +| Manual proof | frozen proof method executed verbatim; `sync`, `status`, `restart`, env absence, fresh restart proof recorded | manual artifacts, `p3` sentinel | +| Closeout | all required sentinels exist and run-state is complete | `.runs/plan-14/closeout.md` | + +## Run Exit Criteria + +Successful completion requires all of the following: + +1. Host policy and request-shape surfaces remained frozen. +2. `substrate-common` remained the only schema owner. +3. `world-agent` stopped using default secret-bearing child env injection on the integrated path. +4. `substrate-gateway` adopted the frozen startup order. +5. Integrated `cli:codex` and `api:openai` no longer depend on secret-bearing process env in the managed gateway. +6. The managed gateway process environment no longer contains secret-bearing auth values by default. +7. `sync` and `restart` redeliver fresh bundles. +8. Tests and docs describe the same final behavior. +9. The gap-matrix closeout state matches final code, tests, and docs. +10. The parent completed the validation wall and Linux manual proof using the exact frozen proof method. +11. `.runs/plan-14/closeout.md` exists and `run-state.json` is `completed`. + +Blocked completion requires all of the following: + +1. `.runs/plan-14/blocked.json` exists. +2. `run-state.json` is `blocked`. +3. `session-log.md` records the exact violated guard and stopping task. +4. Rejected or blocked worker output is preserved without integration. +5. No green closeout sentinel is written. + +## Assumptions + +1. [PLAN-14.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-14.md) is the authoritative execution contract. +2. The safe parallel window begins only after the parent freezes the shared contract. +3. The shared schema can be fully owned inside `substrate-common` without reopening public transport contracts. +4. The gateway consumer lane can implement the frozen startup order without needing new request surfaces. +5. The Linux proof method can be frozen concretely during `p1` from runtime-owned surfaces already present in the repo’s managed gateway lifecycle. +6. The regression/docs lane can stay fully serialized after code-lane integration without slowing overall completion materially. diff --git a/llm-last-mile/ORCH_PLAN-15.md b/llm-last-mile/ORCH_PLAN-15.md new file mode 100644 index 000000000..53d62106c --- /dev/null +++ b/llm-last-mile/ORCH_PLAN-15.md @@ -0,0 +1,939 @@ +# ORCH_PLAN-15: Execute PLAN-15 Through A Parent-Frozen Submit-Turn Contract And One Honest Two-Lane Parallel Window + +Branch: `feat/session-centric-state-store` +Plan source: [PLAN-15.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-15.md) +Gap matrix anchor: [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md) +Execution type: fresh orchestration plan, Linux-first, REPL-targeted agent turns, shell + world-agent + typed local transport, parent-owned contract freeze with one real parallel window + +## Summary + +This document is the execution controller for `PLAN-15`, not a restatement of it. + +The run shape is fixed before any worker starts: + +1. `task/m15-p1-parent-contract-freeze-and-run-init` is parent-only and freezes the submit-turn contract, launch-only boundaries, run-state surface, and worker ownership. +2. `task/m15-g1-worker-launch-gate` is parent-only and is the only gate that may launch parallel workers. +3. The only honest parallel window is exactly two code lanes after `p1`: + - `task/m15-l1-shell-parser-selector-host-submit` + - `task/m15-l2-world-submit-route-and-member-registry` +4. `task/m15-g2-code-lane-integration-gate` and `task/m15-p2-parent-code-lane-integration` are parent-only. +5. Tests, stubs, docs, and gap-matrix closeout do not run in parallel with the code lanes. They start only after `p2` lands final merged code truth. +6. `task/m15-l3-tests-stubs-docs-closeout` is a single sequential worker lane on top of the integrated code. +7. `task/m15-g4-validation-wall-gate` and `task/m15-p3-parent-validation-wall-and-closeout` are parent-only and finish the run. + +Canonical task IDs: + +- `task/m15-p1-parent-contract-freeze-and-run-init` +- `task/m15-g1-worker-launch-gate` +- `task/m15-l1-shell-parser-selector-host-submit` +- `task/m15-l2-world-submit-route-and-member-registry` +- `task/m15-g2-code-lane-integration-gate` +- `task/m15-p2-parent-code-lane-integration` +- `task/m15-g3-closeout-launch-gate` +- `task/m15-l3-tests-stubs-docs-closeout` +- `task/m15-g4-validation-wall-gate` +- `task/m15-p3-parent-validation-wall-and-closeout` + +## Assumptions + +1. `PLAN-15.md` remains the authoritative dependency graph for this run. +2. The parent already owns the authoritative checkout on `feat/session-centric-state-store`. +3. The new submit-turn request and client method can be frozen in `p1` without requiring shell or world-agent file edits in the same phase. +4. Linux is the only world-target platform that must go green in this run; non-Linux world targeting may fail closed. +5. Existing Rust test harnesses named in `PLAN-15` are still the correct validation seams for parser, world submit routing, member registry behavior, and wrap regression. +6. The shell continues to own orchestration session truth; `world-agent` continues to own retained world runtime control after bootstrap. + +## Immutable Run Shape + +### Frozen Contract Truth + +These are run-stopping invariants, not preferences: + +1. The only targeted-turn grammar is `:: `. +2. `:host` and `:pty` retain their existing meanings. +3. Plain REPL input remains shell execution. +4. `substrate -c` remains `ShellMode::Wrap`. +5. `ExecuteRequest.member_dispatch` remains launch-only for world-member bootstrap. +6. World follow-up turns use a new typed submit route and do not overload `member_dispatch`. +7. Host follow-up turns may target only the active orchestrator backend for the current REPL session. +8. World follow-up turns are Linux-first and must use the new typed submit route. +9. One retained world member at a time is the product boundary for this run. +10. Bootstrap spans and submitted-turn spans remain distinct and cancel independently. +11. Routing is by exact `backend_id`, never by "the one eligible member". +12. Submitted turns reuse surfaced `internal.uaa_session_id`; they do not replay bootstrap prompts. + +### Parent-Only Versus Worker-Owned Authority + +Parent-only for the full run: + +- `.runs/plan-15/*` +- `.runs/task-m15-*/**` +- [crates/agent-api-types/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/agent-api-types/src/lib.rs) +- [crates/agent-api-client/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/agent-api-client/src/lib.rs) + +Parent-only during `p1`, then frozen by artifact and branch seed: + +- request shape for `MemberTurnSubmitRequestV1` +- client surface for `submit_member_turn_stream(...)` +- launch-only posture of `ExecuteRequest.member_dispatch` +- lane ownership map +- validation-wall command list + +Worker-owned after `p1`: + +- `L1`: shell parser, exact selector, host submit path, Linux-first REPL world submit wiring +- `L2`: world-agent submit route, member registry refactor, submit-span cancel semantics +- `L3`: tests, stubs, wrap regression, gap-matrix closeout + +No worker may write parent-owned run-state. No worker may change another lane's file set. No worker may reopen the frozen contract. + +### Stop Conditions + +Stop the run, write `.runs/plan-15/blocked.json`, and do not advance if any of these occur: + +1. `p1` cannot freeze the new submit-turn request without changing the grammar or launch-only `member_dispatch` boundary. +2. Any implementation requires a second targeted-turn grammar spelling. +3. Any implementation changes `substrate -c` away from wrap mode. +4. Any worker edits [crates/agent-api-types/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/agent-api-types/src/lib.rs) or [crates/agent-api-client/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/agent-api-client/src/lib.rs) after `p1`. +5. Any worker reuses `ExecuteRequest.member_dispatch` for submitted turns. +6. Any worker broadens host targeting beyond the active orchestrator backend for the current REPL session. +7. Any worker broadens world targeting beyond Linux-first behavior. +8. Any worker attempts to retain more than one world member at once. +9. Any worker collapses bootstrap spans and submitted-turn spans into one cancel class. +10. A lane needs an unowned file to proceed and the parent cannot re-plan without breaking the dependency graph. +11. `L3` starts before `p2` accepts final code truth. +12. The gap-matrix row is marked closed before code, tests, and final docs agree on the same landed behavior. +13. The parent would need to invent hybrid semantics during integration to make two lanes appear compatible. +14. The validation wall cannot prove exact grammar, wrap regression, launch-only `member_dispatch`, distinct span classes, and one-retained-world-member behavior. + +## Fresh Worktrees And Branches + +Fresh worktree root: + +- `/Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-15` + +Worker worktrees: + +- `/Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-15/shell-parser-selector-host-submit` +- `/Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-15/world-submit-route-and-member-registry` +- `/Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-15/tests-stubs-docs-closeout` + +Worker branches: + +- `codex/feat-session-centric-state-store-m15-shell-parser-selector-host-submit` +- `codex/feat-session-centric-state-store-m15-world-submit-route-and-member-registry` +- `codex/feat-session-centric-state-store-m15-tests-stubs-docs-closeout` + +Exact setup commands: + +```bash +mkdir -p /Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-15 +git worktree add /Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-15/shell-parser-selector-host-submit -b codex/feat-session-centric-state-store-m15-shell-parser-selector-host-submit feat/session-centric-state-store +git worktree add /Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-15/world-submit-route-and-member-registry -b codex/feat-session-centric-state-store-m15-world-submit-route-and-member-registry feat/session-centric-state-store +git worktree add /Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-15/tests-stubs-docs-closeout -b codex/feat-session-centric-state-store-m15-tests-stubs-docs-closeout feat/session-centric-state-store +``` + +### Parent Integration Surface + +The parent integrates on the authoritative checkout already on `feat/session-centric-state-store`. + +No parent integration worktree is introduced for this run because: + +1. `PLAN-15` has one serialized integrator. +2. accepted branch truth must land on the authoritative branch context named in the plan. +3. `.runs/plan-15/*` is parent-owned state and is easiest to keep coherent on that checkout. +4. there is no honest parallel parent merge activity in this plan. + +## Phase Graph And Concurrency + +Parent checkout: + +- current checkout on `feat/session-centric-state-store` + +Concurrency rules: + +1. Worker cap is `2` until `g2` completes. +2. `p1` must finish before any worker starts. +3. `g1` must accept before any worker starts. +4. The only real parallel window is: + - `task/m15-l1-shell-parser-selector-host-submit` + - `task/m15-l2-world-submit-route-and-member-registry` +5. `g2` classifies both code-lane outputs before integration. +6. `p2` integrates accepted code lanes in this order: + - `L2` first + - `L1` second +7. `g3` starts only after `p2` is green. +8. `L3` runs alone on the exact post-`p2` tree. +9. `g4` and `p3` are parent-only. +10. No third concurrent worker is honest because test stubs and docs must encode final merged route semantics, not intermediate lane guesses. + +### Why `L2` Integrates Before `L1` + +1. `L2` lands the world-agent server truth that the new REPL world submit path depends on. +2. `L1` is the user-facing shell surface. Integrating it second avoids landing outward REPL affordances before the backend submit route exists on the parent tree. +3. If the combined state reveals semantic drift, the later-integrated shell lane is the first quarantine target instead of mutating world-agent ownership rules under active submit semantics. + +## PLAN-15 Step Mapping + +| Orchestration task | PLAN-15 step alignment | +| --- | --- | +| `task/m15-p1-parent-contract-freeze-and-run-init` | Step 1: freeze the submit-turn wire contract | +| `task/m15-l1-shell-parser-selector-host-submit` | Step 2, Step 3, and Step 5 shell-side work | +| `task/m15-l2-world-submit-route-and-member-registry` | Step 4 world-agent ownership and typed submit route | +| `task/m15-l3-tests-stubs-docs-closeout` | Step 6 regression floor and repo truth | +| `task/m15-p3-parent-validation-wall-and-closeout` | Recommended verification commands plus Definition of Done enforcement | + +## Parent-Owned Run-State Surface + +Canonical parent-owned state under `.runs/plan-15/`: + +- `run-state.json` +- `tasks.json` +- `session-log.md` +- `frozen-contract.json` +- `lane-ownership.json` +- `validation-wall.md` +- `quarantine/` +- `blocked.json` on failure only +- `sentinels/` + +`frozen-contract.json` is the single source of truth for: + +- `plan_id: "PLAN-15"` +- `plan_source: "llm-last-mile/PLAN-15.md"` +- `orchestration_plan_source: "llm-last-mile/ORCH_PLAN-15.md"` +- `branch: "feat/session-centric-state-store"` +- `accepted_targeted_grammar: ":: "` +- `launch_only_member_dispatch: true` +- `host_target_rule: "active_orchestrator_backend_only"` +- `world_target_rule: "linux_first_typed_submit_route_only"` +- `world_retention_rule: "single_retained_member"` +- `span_classes: ["bootstrap", "submitted_turn"]` +- `required_stable_fields` +- `failure_taxonomy` +- `validation_commands` + +`lane-ownership.json` is the single source of truth for: + +- task ID +- owner +- worktree path +- branch +- allowed files +- forbidden files +- command gates +- retry budget +- merge order +- sentinel filename + +`run-state.json` is the single source of truth for: + +- `current_phase` +- `active_task_ids` +- `worker_cap` +- `contract_freeze_status` +- `lane_status` +- `accepted_outputs` +- `rejected_outputs` +- `quarantined_outputs` +- `blocked_outputs` +- `retry_budget_by_lane` +- `attempt_counts` +- `integration_order` +- `gate_status` +- `validation_wall_status` +- `termination_reason` +- `terminal_state` + +Required sentinels: + +- `.runs/plan-15/sentinels/task-m15-p1-parent-contract-freeze-and-run-init.ok` +- `.runs/plan-15/sentinels/task-m15-g1-worker-launch-gate.ok` +- `.runs/plan-15/sentinels/task-m15-l1-shell-parser-selector-host-submit.ok` +- `.runs/plan-15/sentinels/task-m15-l2-world-submit-route-and-member-registry.ok` +- `.runs/plan-15/sentinels/task-m15-g2-code-lane-integration-gate.ok` +- `.runs/plan-15/sentinels/task-m15-p2-parent-code-lane-integration.ok` +- `.runs/plan-15/sentinels/task-m15-g3-closeout-launch-gate.ok` +- `.runs/plan-15/sentinels/task-m15-l3-tests-stubs-docs-closeout.ok` +- `.runs/plan-15/sentinels/task-m15-g4-validation-wall-gate.ok` +- `.runs/plan-15/sentinels/task-m15-p3-parent-validation-wall-and-closeout.ok` + +Per-task artifact directories: + +- `.runs/task-m15-p1-parent-contract-freeze-and-run-init/` +- `.runs/task-m15-g1-worker-launch-gate/` +- `.runs/task-m15-l1-shell-parser-selector-host-submit/` +- `.runs/task-m15-l2-world-submit-route-and-member-registry/` +- `.runs/task-m15-g2-code-lane-integration-gate/` +- `.runs/task-m15-p2-parent-code-lane-integration/` +- `.runs/task-m15-g3-closeout-launch-gate/` +- `.runs/task-m15-l3-tests-stubs-docs-closeout/` +- `.runs/task-m15-g4-validation-wall-gate/` +- `.runs/task-m15-p3-parent-validation-wall-and-closeout/` + +Each task directory contains: + +- `task.json` +- `summary.md` +- `commands.txt` +- `artifacts/` +- `evidence-manifest.json` + +Each worker task directory also contains: + +- `worker-output.patch` +- `worker-report.md` +- `rejected.json` on rejection only +- `blocked.json` on blocked lane return only +- `quarantine/` when the parent quarantines the output + +Blocked termination surfaces, parent-owned: + +- `.runs/plan-15/blocked.json` +- `.runs/plan-15/quarantine//` +- `.runs/task-m15-*/blocked.json` when a worker explicitly returns blocked evidence + +Blocked termination minimum contents: + +1. task or gate where execution stopped +2. classification: `rejected`, `blocked`, or `merge_refused` +3. exact contract clause or ownership rule that stopped the run +4. whether retry remained available +5. artifact paths for patch, report, command output, and semantic-drift evidence +6. explicit statement that no blocked output was integrated + +## Parent Phase Details + +### `task/m15-p1-parent-contract-freeze-and-run-init` + +Owner: + +- parent only + +Scope: + +1. Initialize `.runs/plan-15/*` and `.runs/task-m15-*/**`. +2. Freeze the exact targeted-turn grammar and stop conditions into `frozen-contract.json`. +3. Add `MemberTurnSubmitRequestV1` only on the parent branch. +4. Add `submit_member_turn_stream(...)` only on the parent branch. +5. Preserve launch-only `ExecuteRequest.member_dispatch` behavior and boundary tests. +6. Freeze the validation-wall commands the parent will later execute. +7. Seed all worker worktrees from the exact post-`p1` tree. + +Owned files: + +- [crates/agent-api-types/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/agent-api-types/src/lib.rs) +- [crates/agent-api-client/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/agent-api-client/src/lib.rs) +- `.runs/plan-15/*` +- `.runs/task-m15-p1-parent-contract-freeze-and-run-init/**` + +Command gates: + +```bash +cargo test -p agent-api-types --lib -- --nocapture +cargo test -p agent-api-client --lib --no-run +``` + +Acceptance: + +1. The request and client contract compile on the parent tree. +2. `ExecuteRequest.member_dispatch` is still launch-only after `p1`. +3. No shell or world-agent implementation file is needed to define the shared contract. +4. `frozen-contract.json`, `lane-ownership.json`, `run-state.json`, and `tasks.json` exist. +5. The parent writes `.runs/plan-15/sentinels/task-m15-p1-parent-contract-freeze-and-run-init.ok`. + +### `task/m15-g1-worker-launch-gate` + +Owner: + +- parent only + +Checks: + +1. `p1` is accepted. +2. All worker worktrees were seeded from the exact same post-`p1` tree. +3. Worker prompts name only owned files, forbidden files, command gates, sentinel, and retry budget. +4. Worker prompts explicitly repeat the frozen contract clauses that apply to each lane. +5. `run-state.json`, `tasks.json`, and `session-log.md` reflect launch state. + +Acceptance: + +1. No worker starts before this gate is green. +2. The parent writes `.runs/plan-15/sentinels/task-m15-g1-worker-launch-gate.ok`. + +## Worker Lanes + +### `task/m15-l1-shell-parser-selector-host-submit` + +Owner: + +- worker only + +Worktree: + +- `/Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-15/shell-parser-selector-host-submit` + +Branch: + +- `codex/feat-session-centric-state-store-m15-shell-parser-selector-host-submit` + +Owned files: + +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) +- [crates/shell/src/execution/agent_runtime/validator.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/validator.rs) +- [crates/shell/src/execution/agent_runtime/registry.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/registry.rs) + +Forbidden files: + +- all parent-owned files +- all `L2` owned files +- all `L3` owned files +- `.runs/plan-15/*` +- `.runs/task-m15-*/**` + +Scope: + +1. Add exact targeted-turn parsing for `:: `. +2. Route targeted turns before shell fallback and only on single-line input. +3. Add exact `backend_id` selection without reusing ambiguity-driven member selection. +4. Implement the shell-local host resume path against the stored surfaced `uaa_session_id`. +5. Enforce that host-targeted turns only target the active orchestrator backend for the current REPL session. +6. Wire Linux-first REPL world submit usage through the frozen typed client method. +7. Preserve one-retained-world-member behavior in REPL control flow. +8. Preserve `:host`, `:pty`, and plain shell behavior. + +Command gates: + +```bash +cargo test -p shell --lib -- --nocapture +cargo test -p shell --test agents_validate -- --nocapture +cargo test -p shell --test repl_world_first_routing_v1 --no-run +``` + +Acceptance: + +1. The lane touches only its owned files. +2. Malformed `::` syntax fails before shell fallback. +3. Host-targeted turns fail closed when the requested backend is not the active orchestrator backend. +4. The shell lane does not broaden world retention beyond one member at a time. +5. `substrate -c` semantics are not changed by this lane. +6. All command gates pass. +7. Acceptance evidence exists before the parent marks the lane green: + - `worker-output.patch` + - `worker-report.md` + - `commands.txt` with exit codes + - `evidence-manifest.json` +8. The worker returns changed files, commands run with exit codes, unresolved assumptions, `worker-output.patch`, and `worker-report.md`. + +### `task/m15-l2-world-submit-route-and-member-registry` + +Owner: + +- worker only + +Worktree: + +- `/Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-15/world-submit-route-and-member-registry` + +Branch: + +- `codex/feat-session-centric-state-store-m15-world-submit-route-and-member-registry` + +Owned files: + +- [crates/world-agent/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/lib.rs) +- [crates/world-agent/src/handlers.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/handlers.rs) +- [crates/world-agent/src/service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs) +- [crates/world-agent/src/member_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs) + +Forbidden files: + +- all parent-owned files +- all `L1` owned files +- all `L3` owned files +- `.runs/plan-15/*` +- `.runs/task-m15-*/**` + +Scope: + +1. Add the new typed `/v1/member_turn/stream` server route on world-agent surfaces frozen in `p1`. +2. Refactor retained-member ownership to stable `participant_id` identity. +3. Keep submitted-turn cancel handles in a separate `span_id` registry. +4. Validate `backend_id`, `orchestration_session_id`, `world_id`, and `world_generation` against the retained runtime. +5. Reject submit when no surfaced `uaa_session_id` is retained. +6. Reject concurrent submitted turns for one participant. +7. Preserve bootstrap cancel semantics while adding submitted-turn cancel semantics. +8. Keep `ExecuteRequest.member_dispatch` launch-only and separate from follow-up submit routing. + +Command gates: + +```bash +cargo test -p world-agent --lib -- --nocapture +cargo test -p world-agent --test member_runtime_world_placement_v1 -- --nocapture +cargo test -p world-agent --test streamed_execute_cancel_v1 --no-run +``` + +Acceptance: + +1. The lane touches only its owned files. +2. The retained registry is keyed by participant identity for active members and by submit span for active turns. +3. Submitted-turn cancel does not tear down the retained bootstrap runtime. +4. The lane does not overload `member_dispatch`. +5. All command gates pass. +6. Acceptance evidence exists before the parent marks the lane green: + - `worker-output.patch` + - `worker-report.md` + - `commands.txt` with exit codes + - `evidence-manifest.json` +7. The worker returns changed files, commands run with exit codes, unresolved assumptions, `worker-output.patch`, and `worker-report.md`. + +## Parent Integration Phases + +### `task/m15-g2-code-lane-integration-gate` + +Owner: + +- parent only + +Checks: + +1. `L1` and `L2` both returned. +2. Each output is classified as `accepted`, `rejected`, or `blocked` before integration. +3. No lane violated file ownership, contract freeze, or stop conditions. +4. `L2` proves typed submit route separation from `member_dispatch`. +5. `L1` proves grammar freeze and active-orchestrator-only host targeting. +6. `g2` goes green only if both code lanes are `accepted`. + +Acceptance: + +1. The parent writes `.runs/plan-15/sentinels/task-m15-g2-code-lane-integration-gate.ok` only if both code lanes are accepted. + +### `task/m15-p2-parent-code-lane-integration` + +Owner: + +- parent only + +Scope: + +1. Integrate only accepted outputs. +2. Integrate `L2` first and rerun its lane-local gates on the parent tree. +3. Integrate `L1` second and rerun its lane-local gates on the parent tree. +4. If `L1` depends on reopening contract files from `p1`, quarantine it and bounce it back instead of editing around the contract freeze. +5. If the combined state blurs bootstrap spans and submitted-turn spans, quarantine the later-integrated lane and record semantic drift. +6. If the combined state broadens host targeting beyond the active orchestrator backend, quarantine `L1`. +7. If the combined state broadens world ownership beyond one retained member or reuses `member_dispatch` for submit, quarantine `L2`. +8. Record accepted integration truth in `run-state.json` and `session-log.md`. +9. If an accepted worker patch merges mechanically but contradicts `frozen-contract.json`, refuse merge, write `merge-refusal.md`, quarantine the output, and either redrive or block. The parent does not hand-edit around the contradiction. + +Command gates: + +```bash +cargo test -p world-agent --lib -- --nocapture +cargo test -p world-agent --test member_runtime_world_placement_v1 -- --nocapture +cargo test -p shell --lib -- --nocapture +cargo test -p shell --test agents_validate -- --nocapture +cargo test -p shell --test repl_world_first_routing_v1 --no-run +``` + +Acceptance: + +1. The parent remains the sole integrator. +2. The integrated parent tree still matches the frozen contract. +3. No hybrid truth was invented during integration. +4. The parent writes `.runs/plan-15/sentinels/task-m15-p2-parent-code-lane-integration.ok`. + +## Sequential Closeout Lane + +### `task/m15-g3-closeout-launch-gate` + +Owner: + +- parent only + +Checks: + +1. `p2` is accepted. +2. The closeout worktree is reseeded or rebased to the exact post-`p2` tree. +3. The worker prompt names only final tests, stubs, wrap regression, and docs ownership. +4. The worker prompt explicitly forbids changing code-lane production files. + +Acceptance: + +1. The parent writes `.runs/plan-15/sentinels/task-m15-g3-closeout-launch-gate.ok`. + +### `task/m15-l3-tests-stubs-docs-closeout` + +Owner: + +- worker only + +Why this stays worker-owned instead of parent-owned: + +1. The lane is sequential, but still bounded and mechanically distinct. +2. The file set is broad enough to benefit from a dedicated closeout worker once final code truth exists. +3. Keeping it worker-owned prevents the parent from mixing final validation with test-authoring edits in the same phase. + +Worktree: + +- `/Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-15/tests-stubs-docs-closeout` + +Branch: + +- `codex/feat-session-centric-state-store-m15-tests-stubs-docs-closeout` + +Owned files: + +- [crates/shell/tests/repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs) +- [crates/shell/tests/support/repl_world_agent.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/support/repl_world_agent.rs) +- [crates/shell/tests/agents_validate.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agents_validate.rs) +- [crates/shell/src/execution/invocation/tests.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/invocation/tests.rs) +- [crates/world-agent/tests/streamed_execute_cancel_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/tests/streamed_execute_cancel_v1.rs) +- [crates/world-agent/tests/member_runtime_world_placement_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/tests/member_runtime_world_placement_v1.rs) +- [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md) + +Forbidden files: + +- all parent-owned files +- all `L1` owned files +- all `L2` owned files +- `.runs/plan-15/*` +- `.runs/task-m15-*/**` + +Scope: + +1. Prove the exact grammar acceptance and rejection rules in shell tests. +2. Extend the REPL world-agent stub to script `/v1/member_turn/stream`. +3. Prove host-targeted turns resume only the active orchestrator backend. +4. Prove world-targeted turns go through the new typed submit route. +5. Prove cancel semantics distinguish bootstrap spans from submitted-turn spans. +6. Prove one retained world member at a time and explicit backend switching behavior. +7. Add the mandatory `substrate -c` wrap-mode regression. +8. Update the gap matrix so repo truth matches landed behavior and still leaves broader productization open. + +Command gates: + +```bash +cargo test -p shell --test repl_world_first_routing_v1 -- --nocapture +cargo test -p shell --test agents_validate -- --nocapture +cargo test -p shell --lib -- --nocapture +cargo test -p world-agent --test streamed_execute_cancel_v1 -- --nocapture +cargo test -p world-agent --test member_runtime_world_placement_v1 -- --nocapture +``` + +Acceptance: + +1. The lane touches only its owned files. +2. Tests assert the final typed submit route and final grammar only. +3. Wrap-mode regression is explicit and green. +4. The gap-matrix row is not marked closed unless code, tests, and docs agree on the same shipped truth. +5. All command gates pass. +6. Acceptance evidence exists before the parent marks the lane green: + - `worker-output.patch` + - `worker-report.md` + - `commands.txt` with exit codes + - `evidence-manifest.json` +7. The worker returns changed files, commands run with exit codes, unresolved assumptions, `worker-output.patch`, and `worker-report.md`. + +## Validation Wall And Final Closeout + +### `task/m15-g4-validation-wall-gate` + +Owner: + +- parent only + +Checks: + +1. `L3` returned and is classified before final validation. +2. `L3` is `accepted`. +3. No quarantined or blocked output remains unresolved. +4. `validation-wall.md` names the exact final command order. +5. `validation-wall.md` includes contract assertions for grammar, wrap mode, typed world submit route, launch-only `member_dispatch`, single retained world member, and distinct span classes. + +Acceptance: + +1. The parent writes `.runs/plan-15/sentinels/task-m15-g4-validation-wall-gate.ok`. + +### `task/m15-p3-parent-validation-wall-and-closeout` + +Owner: + +- parent only + +Scope: + +1. Integrate only accepted `L3` output. +2. Run the full validation wall in exact order. +3. Capture doctor outputs because this run touches shell and world-agent logic. +4. Record final command results and acceptance evidence in `.runs/task-m15-p3-parent-validation-wall-and-closeout/artifacts/`. +5. Mark the run complete only if the validation wall proves the frozen contract rather than merely compiling. + +Required validation commands, executed in this order: + +```bash +cargo fmt --all -- --check +cargo clippy --workspace --all-targets -- -D warnings +cargo test -p agent-api-types --lib -- --nocapture +cargo test -p shell --lib -- --nocapture +cargo test -p shell --test repl_world_first_routing_v1 -- --nocapture +cargo test -p shell --test agents_validate -- --nocapture +cargo test -p world-agent --test streamed_execute_cancel_v1 -- --nocapture +cargo test -p world-agent --test member_runtime_world_placement_v1 -- --nocapture +substrate world doctor --json +substrate shim doctor --json +substrate health --json +``` + +Required artifacts under `.runs/task-m15-p3-parent-validation-wall-and-closeout/artifacts/`: + +- `fmt.txt` +- `clippy.txt` +- `agent-api-types-tests.txt` +- `shell-lib-tests.txt` +- `repl-world-first-routing.txt` +- `agents-validate.txt` +- `world-agent-streamed-execute-cancel.txt` +- `world-agent-member-runtime-placement.txt` +- `world-doctor.json` +- `shim-doctor.json` +- `health.json` +- `contract-audit.md` +- `closeout.md` + +Required contract assertions in `contract-audit.md`: + +1. The only accepted targeted syntax is `:: `. +2. `substrate -c` still resolves to wrap mode. +3. `ExecuteRequest.member_dispatch` is still launch-only. +4. Host-targeted turns are valid only for the active orchestrator backend in the current REPL session. +5. World-targeted turns are Linux-first and use the typed submit route. +6. One retained world member at a time remains the shipped behavior. +7. Bootstrap spans and submitted-turn spans are distinct in tests and cancel semantics. + +Acceptance: + +1. The full validation wall passes in order. +2. The doctor outputs are captured. +3. The artifact audit proves the frozen contract rather than approximate behavior. +4. The parent writes `.runs/plan-15/sentinels/task-m15-p3-parent-validation-wall-and-closeout.ok`. + +## Gates + +Gate sequencing is serialized and parent-owned: + +1. `task/m15-p1-parent-contract-freeze-and-run-init` +2. `task/m15-g1-worker-launch-gate` +3. parallel launch of `task/m15-l1-shell-parser-selector-host-submit` and `task/m15-l2-world-submit-route-and-member-registry` +4. `task/m15-g2-code-lane-integration-gate` +5. `task/m15-p2-parent-code-lane-integration` +6. `task/m15-g3-closeout-launch-gate` +7. `task/m15-l3-tests-stubs-docs-closeout` +8. `task/m15-g4-validation-wall-gate` +9. `task/m15-p3-parent-validation-wall-and-closeout` + +Gate rules: + +1. The parent updates `run-state.json`, `tasks.json`, and `session-log.md` at every gate transition, worker launch, worker return, retry authorization, quarantine decision, merge refusal, and terminal closeout. +2. A gate is green only when all prerequisite sentinels and evidence for the prior phase exist. +3. A gate cannot go green on narrative progress alone; it requires command evidence, artifact presence, and ownership compliance. +4. `g2` and `g4` are classification gates, not integration gates. They may accept, reject, quarantine, or block; they do not mutate production files. + +### Retry And Redrive Policy + +1. Each worker lane has retry budget `1`. +2. The parent must classify the first attempt before authorizing any retry. +3. A retry is allowed only for lane-local failure inside the lane's owned files. +4. A retry is allowed only if the first attempt did not violate a hard guard, frozen contract clause, or cross-lane ownership rule. +5. Contract-freeze drift, `member_dispatch` submit reuse, host-target broadening, world-retention broadening, or wrap-mode regression are non-retryable. +6. The lane identity does not change on retry; only attempt metadata changes. +7. If a lane exhausts retry budget without acceptance, the run blocks. + +### Merge Refusal Rules + +1. The parent refuses merge for any lane whose patch requires parent-only file edits after `p1`. +2. The parent refuses merge for any lane that is locally green but semantically contradicts `frozen-contract.json`. +3. The parent refuses merge for any lane that broadens scope beyond its owned files, even if the code compiles. +4. The parent refuses merge for any lane that encodes intermediate behavior as final repo truth. +5. Merge refusal writes: + - `.runs/task-/artifacts/merge-refusal.md` + - `.runs/task-/rejected.json` + - `.runs/plan-15/quarantine//` + +### Quarantine, Rejection, And Blocked Termination + +1. Rejected output is never integrated. +2. The parent records rejected output in `rejected_outputs` and `quarantined_outputs`. +3. Quarantine preserves `worker-output.patch`, `worker-report.md`, `commands.txt`, and any semantic-drift notes under `.runs/plan-15/quarantine//`. +4. A rejection is `retryable` only if the lane stayed inside owned files and did not violate frozen contract truth. +5. A blocked worker return, non-retryable rejection, exhausted retry budget, or failed parent gate writes `.runs/plan-15/blocked.json` and stops the run. +6. Blocked termination is explicit. The run ends either `completed` or `blocked`, never in a silent partial state. + +## Worker Interfaces + +### Worker Prompt Contract + +Every worker prompt sent by the parent must include exactly: + +1. task ID +2. attempt number +3. worktree path +4. branch name +5. owned files +6. forbidden files +7. exact frozen contract clauses relevant to that lane +8. command gates +9. retry budget +10. required return artifacts +11. sentinel name + +The parent must not send: + +1. discretionary broad goals without file boundaries +2. permission to edit parent-owned or cross-lane files +3. instructions that reopen `p1` contract decisions +4. instructions that imply hidden concurrency beyond the one honest two-lane window + +### Worker Return Contract + +Every worker must return exactly: + +1. changed files list +2. commands run, each with exit code +3. explicit attempt classification: `clean`, `retryable`, or `blocked` +4. unresolved assumptions or blockers +5. `worker-output.patch` +6. `worker-report.md` +7. `evidence-manifest.json` naming every artifact the parent should inspect + +For a return to be eligible for acceptance, the parent must be able to verify: + +1. the patch stays within owned files +2. command gates actually ran +3. exit codes are present +4. the attempt classification matches the evidence +5. no frozen contract clause was reopened + +## Tests And Acceptance + +### Task-Scoped Command Gates + +`task/m15-p1-parent-contract-freeze-and-run-init` + +```bash +cargo test -p agent-api-types --lib -- --nocapture +cargo test -p agent-api-client --lib --no-run +``` + +`task/m15-l1-shell-parser-selector-host-submit` + +```bash +cargo test -p shell --lib -- --nocapture +cargo test -p shell --test agents_validate -- --nocapture +cargo test -p shell --test repl_world_first_routing_v1 --no-run +``` + +`task/m15-l2-world-submit-route-and-member-registry` + +```bash +cargo test -p world-agent --lib -- --nocapture +cargo test -p world-agent --test member_runtime_world_placement_v1 -- --nocapture +cargo test -p world-agent --test streamed_execute_cancel_v1 --no-run +``` + +`task/m15-p2-parent-code-lane-integration` + +```bash +cargo test -p world-agent --lib -- --nocapture +cargo test -p world-agent --test member_runtime_world_placement_v1 -- --nocapture +cargo test -p shell --lib -- --nocapture +cargo test -p shell --test agents_validate -- --nocapture +cargo test -p shell --test repl_world_first_routing_v1 --no-run +``` + +`task/m15-l3-tests-stubs-docs-closeout` + +```bash +cargo test -p shell --test repl_world_first_routing_v1 -- --nocapture +cargo test -p shell --test agents_validate -- --nocapture +cargo test -p shell --lib -- --nocapture +cargo test -p world-agent --test streamed_execute_cancel_v1 -- --nocapture +cargo test -p world-agent --test member_runtime_world_placement_v1 -- --nocapture +``` + +`task/m15-p3-parent-validation-wall-and-closeout` + +```bash +cargo fmt --all -- --check +cargo clippy --workspace --all-targets -- -D warnings +cargo test -p agent-api-types --lib -- --nocapture +cargo test -p shell --lib -- --nocapture +cargo test -p shell --test repl_world_first_routing_v1 -- --nocapture +cargo test -p shell --test agents_validate -- --nocapture +cargo test -p world-agent --test streamed_execute_cancel_v1 -- --nocapture +cargo test -p world-agent --test member_runtime_world_placement_v1 -- --nocapture +substrate world doctor --json +substrate shim doctor --json +substrate health --json +``` + +### Acceptance Matrix + +| Phase | Required proof | Refuse / block trigger | +| --- | --- | --- | +| Parent freeze `p1` | `MemberTurnSubmitRequestV1` and `submit_member_turn_stream(...)` frozen on parent branch; `member_dispatch` still launch-only; run-state artifacts exist | contract requires shell/world edits in `p1`, or `member_dispatch` loses launch-only posture | +| `L1` shell lane | exact grammar parse path, active-orchestrator-only host targeting, no wrap regression, no cross-lane edits | shell broadens host targeting, changes wrap behavior, or reopens contract files | +| `L2` world lane | typed submit route exists, retained registry split is explicit, submit cancel does not kill bootstrap runtime | submit path overloads `member_dispatch`, broadens retention beyond one member, or blurs span classes | +| Parent integration `p2` | accepted `L2` then accepted `L1`, no hybrid truth, semantic drift resolved by acceptance or quarantine | merge requires parent hand-edit around contradiction, or combined state violates frozen contract | +| `L3` closeout lane | tests and stubs prove final route and grammar, wrap regression explicit, gap matrix matches landed truth | tests encode intermediate behavior or docs close a gap earlier than code truth | +| Final validation wall `p3` | full command wall green, doctor evidence captured, contract audit proves exact frozen clauses | any final assertion fails, any blocked output remains unresolved, or audit cannot prove frozen truth | + +### Run Exit Criteria + +Success requires all of: + +1. every required sentinel exists +2. no blocked artifact exists under `.runs/plan-15/` +3. accepted outputs are integrated in the prescribed order only +4. final validation commands pass +5. `contract-audit.md` proves grammar freeze, launch-only `member_dispatch`, active-orchestrator-only host targeting, Linux-first typed world submit, one retained world member, and distinct span classes + +Blocked termination requires any of: + +1. hard-guard violation +2. non-retryable rejection +3. exhausted retry budget +4. merge refusal with no legal redrive path +5. failed validation wall or doctor evidence + +On blocked termination the parent must write: + +1. `.runs/plan-15/blocked.json` +2. terminal state and rationale in `run-state.json` +3. gate and failure summary in `session-log.md` +4. preserved evidence under `.runs/plan-15/quarantine/` and the relevant task artifact directory + +## Acceptance Criteria + +The run is complete only if all of these are true: + +1. The parent froze the contract before worker launch and never reopened it. +2. The only parallel window was `L1` plus `L2`. +3. `L3` started only after code-lane integration was complete. +4. No file ownership overlap occurred outside parent integration and final closeout takeover. +5. The final code path uses exact `backend_id` targeting for targeted turns. +6. Host turns are constrained to the active orchestrator backend. +7. World turns are Linux-first and use the typed submit route. +8. `member_dispatch` remains launch-only. +9. One retained world member at a time remains true. +10. Bootstrap and submitted-turn span classes remain distinct. +11. `substrate -c` stays wrap mode. +12. Gap-matrix truth matches code and tests. + +## Context-Control Rules + +1. The parent keeps only frozen contract artifacts, active task state, worker reports, narrow diffs, blockers, and validation status in live context. +2. The parent does not keep full worker transcripts in live context. +3. Worker prompts include only task ID, attempt number, worktree, branch, owned files, forbidden files, contract clauses, command gates, and output contract. +4. If a worker discovers it needs an unowned file, it must stop and report that need instead of silently expanding scope. +5. If a lane returns code that is locally green but semantically drifts from `frozen-contract.json`, the parent quarantines it instead of integrating around it. diff --git a/llm-last-mile/ORCH_PLAN-16.md b/llm-last-mile/ORCH_PLAN-16.md new file mode 100644 index 000000000..adba13f45 --- /dev/null +++ b/llm-last-mile/ORCH_PLAN-16.md @@ -0,0 +1,927 @@ +# ORCH_PLAN-16: Execute PLAN-16 Through A Parent-Frozen Backend-Identity Contract And One Honest Two-Lane Parallel Window + +Branch: `feat/session-centric-state-store` +Plan source: [PLAN-16.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-16.md) +Style reference: [ORCH_PLAN-15.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/ORCH_PLAN-15.md) +Structure reference: [M26 Orchestration Plan](/Users/spensermcconnell/__Active_Code/atomize-hq/spec/docs/m26_orchestration_kickoff_prompt.md) +Gap matrix anchor: [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md) +Execution type: fresh orchestration plan, Linux-first, exact backend-id selection, retained world-member coexistence, shell + world-agent + tests/docs closeout, parent-frozen identity contract, parent-only integration and approval +Worker model: GPT-5.4 workers with `reasoning_effort=high` +Max concurrent code workers before integration: `2` + +## Summary + +This document is the execution controller for `PLAN-16`, not a restatement of it. + +The run shape is fixed before any worker starts: + +1. `task/m16-p1-parent-identity-freeze-and-run-init` is parent-only and freezes the retained-member identity contract, exact failure taxonomy, lane ownership, merge order, and validation wall. +2. `task/m16-g1-worker-launch-gate` is parent-only and is the only gate that may launch parallel code workers. +3. The only honest parallel window is exactly two code lanes after `p1`: + - `task/m16-l1-shell-retained-backend-map` + - `task/m16-l2-world-agent-retained-coexistence` +4. `task/m16-g2-code-lane-integration-gate` and `task/m16-p2-parent-code-lane-integration` are parent-only. +5. Tests, docs, and gap-matrix closeout do not run in parallel with the code lanes. They start only after `p2` lands final merged code truth. +6. `task/m16-l3-tests-docs-gap-closeout` is a single sequential worker lane on top of the integrated code. +7. `task/m16-g4-validation-wall-gate` and `task/m16-p3-parent-validation-wall-and-closeout` are parent-only and finish the run. + +Canonical task IDs: + +- `task/m16-p1-parent-identity-freeze-and-run-init` +- `task/m16-g1-worker-launch-gate` +- `task/m16-l1-shell-retained-backend-map` +- `task/m16-l2-world-agent-retained-coexistence` +- `task/m16-g2-code-lane-integration-gate` +- `task/m16-p2-parent-code-lane-integration` +- `task/m16-g3-closeout-launch-gate` +- `task/m16-l3-tests-docs-gap-closeout` +- `task/m16-g4-validation-wall-gate` +- `task/m16-p3-parent-validation-wall-and-closeout` + +## Assumptions + +1. [PLAN-16.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-16.md) remains the authoritative dependency graph for this run. +2. The authoritative checkout for integration remains on `feat/session-centric-state-store`. +3. The already-landed typed follow-up submit contract is sufficient and does not require new transport-surface work in `p1`. +4. Linux is the only platform that must go green for retained world-member coexistence in this run. +5. The current shell and world-agent tests named in `PLAN-16` remain the correct validation seams for coexistence, duplicate detection, targeted routing, and unchanged regressions. + +## Immutable Run Shape + +### Frozen contract truth + +These are run-stopping invariants, not preferences: + +1. `backend_id` remains the canonical selector for every explicit backend-targeted path. +2. The only targeted-turn grammar remains `:: `. +3. Exact backend selection continues to fail closed for missing, wrong-scope, denied, unrealizable, and duplicate-inventory cases. +4. Retained world-member reuse is keyed by `orchestration_session_id + world_generation + backend_id`. +5. Distinct backend ids may coexist as retained live world members in the same orchestration session and authoritative world generation. +6. Duplicate retained live members for the same backend key remain a hard error. +7. `MemberTurnSubmitRequestV1` remains the submit contract. No second submit transport is introduced. +8. No new CLI surface is introduced. +9. No grammar redesign is introduced. +10. No macOS parity work is introduced. +11. The parent agent is the only integrator and the only approval authority for worker outputs. +12. The tests/docs lane never runs concurrently with the two code lanes. + +### Parent-only versus worker-owned authority + +Parent-only for the full run: + +- `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/plan-16/*` +- `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/task-m16-*/**` +- worker launch, output classification, merge approval, quarantine, retry approval, and final validation +- any temporary shared-hotspot takeover if a lane boundary proves wrong + +Parent-only during `p1`, then frozen by artifact: + +- retained-member identity contract +- exact explicit-backend failure taxonomy +- worker cap and lane ownership map +- merge order +- validation-wall command list +- stop conditions + +Worker-owned after `p1`: + +- `L1`: shell retained-member collection, backend-aware reuse, exact shell-side failure preservation +- `L2`: world-agent coexistence, duplicate same-backend detection, exact world-side lookup and submit preservation +- `L3`: integrated tests, test support, docs, and gap-matrix truth + +No worker may edit parent-owned run-state. No worker may edit another lane's owned files. No worker may reopen the frozen contract. + +### Stop conditions + +Stop the run, write `.runs/plan-16/blocked.json`, and do not advance if any of these occur: + +1. The retained-member identity contract cannot be frozen without changing grammar, CLI surface, or transport shape. +2. Any implementation reintroduces `agent_id` or singleton-member heuristics as the reusable selector for explicit backend-targeted world reuse. +3. Any implementation allows duplicate same-backend retained members to coexist without failing closed. +4. Any worker changes the `:: ` grammar. +5. Any worker introduces a new CLI surface. +6. Any worker broadens scope into macOS parity work. +7. Any worker edits another lane's owned files or any parent-owned run-state. +8. `L3` starts before `p2` accepts final code truth. +9. The parent would need to invent hybrid semantics during integration to make the two code lanes appear compatible. +10. The validation wall cannot prove coexistence of distinct backend ids, duplicate same-backend fail-closed behavior, unchanged exact targeted-turn routing, and unchanged existing regressions. +11. The gap matrix is marked closed before code, tests, and final docs agree on the same landed behavior. + +## Fresh Worktrees And Branches + +Fresh worktree root: + +- `/Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-16` + +Worker worktrees: + +- `/Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-16/shell-retained-backend-map` +- `/Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-16/world-agent-retained-coexistence` +- `/Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-16/tests-docs-gap-closeout` + +Worker branches: + +- `codex/feat-session-centric-state-store-m16-shell-retained-backend-map` +- `codex/feat-session-centric-state-store-m16-world-agent-retained-coexistence` +- `codex/feat-session-centric-state-store-m16-tests-docs-gap-closeout` + +Exact setup commands: + +```bash +mkdir -p /Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-16 +git worktree add /Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-16/shell-retained-backend-map -b codex/feat-session-centric-state-store-m16-shell-retained-backend-map feat/session-centric-state-store +git worktree add /Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-16/world-agent-retained-coexistence -b codex/feat-session-centric-state-store-m16-world-agent-retained-coexistence feat/session-centric-state-store +git worktree add /Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-16/tests-docs-gap-closeout -b codex/feat-session-centric-state-store-m16-tests-docs-gap-closeout feat/session-centric-state-store +``` + +### Parent integration surface + +The parent integrates on the authoritative checkout already on `feat/session-centric-state-store`. + +No separate parent integration worktree is introduced because: + +1. `PLAN-16` has one serialized integrator. +2. `.runs/plan-16/*` is parent-owned state and should stay co-located with the authoritative branch context. +3. The only true merge activity is parent-owned and serialized. + +## Phase Graph And Concurrency + +Parent checkout: + +- current authoritative checkout on `feat/session-centric-state-store` + +Concurrency rules: + +1. Worker cap is `2` until `g2` completes. +2. `p1` must finish before any worker starts. +3. `g1` must accept before any worker starts. +4. The only real parallel window is: + - `task/m16-l1-shell-retained-backend-map` + - `task/m16-l2-world-agent-retained-coexistence` +5. `g2` classifies both code-lane outputs before integration. +6. `p2` integrates accepted code lanes in this order: + - `L2` first + - `L1` second +7. `g3` starts only after `p2` is green. +8. `L3` runs alone on the exact post-`p2` tree. +9. `g4` and `p3` are parent-only. +10. No third concurrent worker is honest because tests, support stubs, and docs must encode final merged coexistence semantics, not lane-local guesses. + +### Why `L2` integrates before `L1` + +1. `L2` lands the backend-aware retained-member truth inside `world-agent`, which the shell must reuse honestly. +2. Integrating shell second avoids landing outward coexistence behavior before the world registry can actually honor it. +3. If the combined state reveals semantic drift, the later shell lane is the first quarantine target rather than weakening world-side duplicate detection. + +## PLAN-16 Step Mapping + +| Orchestration task | PLAN-16 step alignment | +| --- | --- | +| `task/m16-p1-parent-identity-freeze-and-run-init` | Step 1: freeze the retained-member identity contract | +| `task/m16-l1-shell-retained-backend-map` | Step 2 and the shell-owned portion of Step 4 | +| `task/m16-l2-world-agent-retained-coexistence` | Step 3 | +| `task/m16-p2-parent-code-lane-integration` | End-to-end Step 4 enforcement across merged shell + world truth | +| `task/m16-l3-tests-docs-gap-closeout` | Step 5 | +| `task/m16-p3-parent-validation-wall-and-closeout` | Recommended verification commands plus Definition of Done enforcement | + +## Parent-Owned Run-State Surface + +Canonical parent-owned state under `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/plan-16/`: + +- `run-state.json` +- `tasks.json` +- `session-log.md` +- `identity-freeze.json` +- `lane-ownership.json` +- `merge-order.json` +- `validation-wall.md` +- `blocked.json` on failure only +- `quarantine/` +- `sentinels/` + +`identity-freeze.json` is the single source of truth for: + +- `plan_id: "PLAN-16"` +- `plan_source: "llm-last-mile/PLAN-16.md"` +- `orchestration_plan_source: "llm-last-mile/ORCH_PLAN-16.md"` +- `branch: "feat/session-centric-state-store"` +- `selector_authority: "validate_exact_backend_selection"` +- `retained_member_key: "orchestration_session_id + world_generation + backend_id"` +- `allow_distinct_backend_coexistence: true` +- `duplicate_same_backend_policy: "fail_closed"` +- `grammar: ":: "` +- `new_cli_surface_allowed: false` +- `macos_parity_allowed: false` +- `validation_commands` +- `stop_conditions` + +`lane-ownership.json` is the single source of truth for: + +- task ID +- owner +- worktree path +- branch +- allowed files +- forbidden files +- command gates +- retry budget +- merge order +- sentinel filename + +`run-state.json` is the single source of truth for: + +- `current_phase` +- `active_task_ids` +- `worker_cap` +- `contract_freeze_status` +- `lane_status` +- `accepted_outputs` +- `rejected_outputs` +- `quarantined_outputs` +- `blocked_outputs` +- `retry_budget_by_lane` +- `attempt_counts` +- `integration_order` +- `gate_status` +- `validation_wall_status` +- `termination_reason` +- `terminal_state` + +Required sentinels: + +- `.runs/plan-16/sentinels/task-m16-p1-parent-identity-freeze-and-run-init.ok` +- `.runs/plan-16/sentinels/task-m16-g1-worker-launch-gate.ok` +- `.runs/plan-16/sentinels/task-m16-l1-shell-retained-backend-map.ok` +- `.runs/plan-16/sentinels/task-m16-l2-world-agent-retained-coexistence.ok` +- `.runs/plan-16/sentinels/task-m16-g2-code-lane-integration-gate.ok` +- `.runs/plan-16/sentinels/task-m16-p2-parent-code-lane-integration.ok` +- `.runs/plan-16/sentinels/task-m16-g3-closeout-launch-gate.ok` +- `.runs/plan-16/sentinels/task-m16-l3-tests-docs-gap-closeout.ok` +- `.runs/plan-16/sentinels/task-m16-g4-validation-wall-gate.ok` +- `.runs/plan-16/sentinels/task-m16-p3-parent-validation-wall-and-closeout.ok` + +Per-task artifact directories: + +- `.runs/task-m16-p1-parent-identity-freeze-and-run-init/` +- `.runs/task-m16-g1-worker-launch-gate/` +- `.runs/task-m16-l1-shell-retained-backend-map/` +- `.runs/task-m16-l2-world-agent-retained-coexistence/` +- `.runs/task-m16-g2-code-lane-integration-gate/` +- `.runs/task-m16-p2-parent-code-lane-integration/` +- `.runs/task-m16-g3-closeout-launch-gate/` +- `.runs/task-m16-l3-tests-docs-gap-closeout/` +- `.runs/task-m16-g4-validation-wall-gate/` +- `.runs/task-m16-p3-parent-validation-wall-and-closeout/` + +Each task directory contains: + +- `task.json` +- `summary.md` +- `commands.txt` +- `artifacts/` +- `evidence-manifest.json` + +Each worker task directory also contains: + +- `worker-output.patch` +- `worker-report.md` +- `rejected.json` on rejection only +- `blocked.json` on blocked lane return only +- `quarantine/` when the parent quarantines the output + +Blocked termination minimum contents: + +1. task or gate where execution stopped +2. classification: `rejected`, `blocked`, or `merge_refused` +3. exact contract clause or ownership rule that stopped the run +4. whether retry remained available +5. artifact paths for patch, report, command output, and semantic-drift evidence +6. explicit statement that no blocked output was integrated + +## Parent Phase Details + +### `task/m16-p1-parent-identity-freeze-and-run-init` + +Owner: + +- parent only + +Scope: + +1. Initialize `.runs/plan-16/*` and `.runs/task-m16-*/**`. +2. Freeze the canonical retained-member key and exact explicit-backend failure taxonomy into `identity-freeze.json`. +3. Freeze worker file ownership, merge order, retry budget, and stop conditions. +4. Freeze the exact validation-wall commands the parent will later execute. +5. Seed all worker worktrees from the exact same post-`p1` tree. + +Expected source files reviewed in `p1` but not lane-opened for shared mutation: + +- [crates/shell/src/execution/agent_runtime/validator.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/validator.rs) +- [crates/agent-api-types/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/agent-api-types/src/lib.rs) +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) +- [crates/world-agent/src/member_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs) + +Command gates: + +```bash +cargo test -p shell validate_exact_backend_selection -- --nocapture +cargo test -p world-agent member_runtime -- --nocapture +``` + +Acceptance: + +1. `identity-freeze.json`, `lane-ownership.json`, `merge-order.json`, `run-state.json`, and `tasks.json` exist. +2. The frozen contract explicitly records exact backend-id selection, backend-aware coexistence, duplicate same-backend fail-closed behavior, and the no-new-surface constraints. +3. No production-file edits are integrated in `p1` unless the parent takes explicit hotspot ownership and reseeds all worker branches afterward. +4. The parent writes `.runs/plan-16/sentinels/task-m16-p1-parent-identity-freeze-and-run-init.ok`. + +### `task/m16-g1-worker-launch-gate` + +Owner: + +- parent only + +Checks: + +1. `p1` is accepted. +2. All worker worktrees were seeded from the exact same post-`p1` tree. +3. Worker prompts name only owned files, forbidden files, command gates, sentinel, and retry budget. +4. Worker prompts explicitly repeat the frozen contract clauses that apply to each lane. +5. `run-state.json`, `tasks.json`, and `session-log.md` reflect launch state. + +Acceptance: + +1. No worker starts before this gate is green. +2. The parent writes `.runs/plan-16/sentinels/task-m16-g1-worker-launch-gate.ok`. + +## Worker Lanes + +### `task/m16-l1-shell-retained-backend-map` + +Owner: + +- worker only + +Worktree: + +- `/Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-16/shell-retained-backend-map` + +Branch: + +- `codex/feat-session-centric-state-store-m16-shell-retained-backend-map` + +Owned files: + +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) +- [crates/shell/src/execution/agent_runtime/validator.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/validator.rs) + +Forbidden files: + +- all parent-owned files +- all `L2` owned files +- all `L3` owned files +- `.runs/plan-16/*` +- `.runs/task-m16-*/**` + +Scope: + +1. Replace shell singleton retained-member state with a backend-aware retained-member collection. +2. Replace shell reuse lookup keyed by `agent_id` or singleton-slot assumptions with exact lookup by `backend_id` within the frozen retained-member key. +3. Update readiness logic so: + - exact retained backend match is reused + - missing exact retained backend launches that backend + - duplicate same-backend retained entries fail closed +4. Preserve exact backend-specific failure reporting for missing, wrong-scope, denied, unrealizable, and duplicate cases. +5. Preserve targeted-turn grammar and the existing exact route boundary. +6. Preserve world-generation rollover invalidation and widen it to all retained backends for the stale generation. + +Command gates: + +```bash +cargo test -p shell --lib -- --nocapture +cargo test -p shell validate_exact_backend_selection -- --nocapture +cargo test -p shell --test repl_world_first_routing_v1 --no-run +``` + +Acceptance: + +1. The lane touches only its owned files. +2. The shell can retain more than one world member by backend id without fallback to singleton replacement behavior. +3. Exact backend-targeted shell reuse is by `backend_id`, not `agent_id`. +4. Duplicate same-backend retained state fails closed. +5. No grammar, CLI, or macOS scope drift is introduced. +6. All command gates pass. +7. Acceptance evidence exists before the parent marks the lane green: + - `worker-output.patch` + - `worker-report.md` + - `commands.txt` with exit codes + - `evidence-manifest.json` + +### `task/m16-l2-world-agent-retained-coexistence` + +Owner: + +- worker only + +Worktree: + +- `/Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-16/world-agent-retained-coexistence` + +Branch: + +- `codex/feat-session-centric-state-store-m16-world-agent-retained-coexistence` + +Owned files: + +- [crates/world-agent/src/member_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs) +- [crates/world-agent/src/service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs) + +Forbidden files: + +- all parent-owned files +- all `L1` owned files +- all `L3` owned files +- `.runs/plan-16/*` +- `.runs/task-m16-*/**` + +Scope: + +1. Remove the effective global single-retained-member restriction inside `world-agent`. +2. Add exact retained-member lookup by `orchestration_session_id + world_generation + backend_id`. +3. Allow distinct backend ids to coexist in one session and generation. +4. Reject duplicate same-backend retained members for the same session and generation. +5. Preserve strict submit validation against retained backend identity. +6. Preserve submitted-turn collision behavior and do not widen transport or CLI surface. + +Command gates: + +```bash +cargo test -p world-agent --lib -- --nocapture +cargo test -p world-agent member_runtime -- --nocapture +cargo test -p world-agent --test streamed_execute_cancel_v1 --no-run +cargo test -p world-agent --test member_runtime_world_placement_v1 -- --nocapture +``` + +Acceptance: + +1. The lane touches only its owned files. +2. Distinct backend ids can coexist in one session and generation. +3. Duplicate same-backend retained members fail closed. +4. Submit validation remains exact about retained backend identity. +5. No grammar, CLI, or macOS scope drift is introduced. +6. All command gates pass. +7. Acceptance evidence exists before the parent marks the lane green: + - `worker-output.patch` + - `worker-report.md` + - `commands.txt` with exit codes + - `evidence-manifest.json` + +## Parent Integration Phases + +### `task/m16-g2-code-lane-integration-gate` + +Owner: + +- parent only + +Checks: + +1. `L1` and `L2` both returned. +2. Each output is classified as `accepted`, `rejected`, or `blocked` before integration. +3. No lane violated file ownership, contract freeze, or stop conditions. +4. `L2` proves coexistence for distinct backend ids without weakening duplicate detection. +5. `L1` proves backend-aware retained lookup and preserves exact failure handling. +6. `g2` goes green only if both code lanes are `accepted`. + +Acceptance: + +1. The parent writes `.runs/plan-16/sentinels/task-m16-g2-code-lane-integration-gate.ok` only if both code lanes are accepted. + +### `task/m16-p2-parent-code-lane-integration` + +Owner: + +- parent only + +Scope: + +1. Integrate only accepted outputs. +2. Integrate `L2` first and rerun its lane-local gates on the parent tree. +3. Integrate `L1` second and rerun its lane-local gates on the parent tree. +4. If the shell lane depends on reopening the retained-member contract, quarantine it and bounce it back instead of editing around the freeze. +5. If the combined state allows coexistence but weakens duplicate same-backend failure, quarantine the later-integrated lane. +6. If the combined state preserves exact routing only by implicit singleton behavior, quarantine the shell lane. +7. Record accepted integration truth in `run-state.json` and `session-log.md`. +8. If an accepted worker patch merges mechanically but contradicts `identity-freeze.json`, refuse merge, write `merge-refusal.md`, quarantine the output, and either redrive or block. + +Command gates: + +```bash +cargo test -p world-agent --lib -- --nocapture +cargo test -p world-agent member_runtime -- --nocapture +cargo test -p shell --lib -- --nocapture +cargo test -p shell validate_exact_backend_selection -- --nocapture +cargo test -p shell --test repl_world_first_routing_v1 --no-run +``` + +Acceptance: + +1. The parent remains the sole integrator. +2. The integrated parent tree still matches the frozen retained-member contract. +3. No hybrid truth was invented during integration. +4. The parent writes `.runs/plan-16/sentinels/task-m16-p2-parent-code-lane-integration.ok`. + +## Sequential Closeout Lane + +### `task/m16-g3-closeout-launch-gate` + +Owner: + +- parent only + +Checks: + +1. `p2` is accepted. +2. The closeout worktree is reseeded or rebased to the exact post-`p2` tree. +3. The worker prompt names only final tests, support stubs, and docs ownership. +4. The worker prompt explicitly forbids changing code-lane production files. + +Acceptance: + +1. The parent writes `.runs/plan-16/sentinels/task-m16-g3-closeout-launch-gate.ok`. + +### `task/m16-l3-tests-docs-gap-closeout` + +Owner: + +- worker only + +Worktree: + +- `/Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-16/tests-docs-gap-closeout` + +Branch: + +- `codex/feat-session-centric-state-store-m16-tests-docs-gap-closeout` + +Owned files: + +- [crates/shell/tests/repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs) +- [crates/shell/tests/support/repl_world_agent.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/support/repl_world_agent.rs) +- [crates/shell/tests/agents_validate.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agents_validate.rs) +- [crates/world-agent/tests/streamed_execute_cancel_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/tests/streamed_execute_cancel_v1.rs) +- [crates/world-agent/tests/member_runtime_world_placement_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/tests/member_runtime_world_placement_v1.rs) +- [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md) + +Forbidden files: + +- all parent-owned files +- all `L1` owned files +- all `L2` owned files +- `.runs/plan-16/*` +- `.runs/task-m16-*/**` + +Scope: + +1. Prove coexistence of `cli:codex` and `cli:claude-code` in one session and generation. +2. Prove exact backend reuse for each backend after both are live. +3. Prove duplicate same-backend retained state fails closed. +4. Preserve the current same-generation same-backend reuse proof. +5. Preserve exact targeted-turn routing and grammar regressions. +6. Extend shell/world-agent test support only as needed to model two retained backends honestly. +7. Update the gap matrix so repo truth matches landed behavior and still leaves broader productization out of scope. + +Command gates: + +```bash +cargo test -p shell --test repl_world_first_routing_v1 -- --nocapture +cargo test -p shell --test agents_validate -- --nocapture +cargo test -p world-agent --test streamed_execute_cancel_v1 -- --nocapture +cargo test -p world-agent --test member_runtime_world_placement_v1 -- --nocapture +``` + +Acceptance: + +1. The lane touches only its owned files. +2. Tests assert coexistence of distinct backend ids and fail-closed duplicate same-backend behavior on final merged code, not on lane-local assumptions. +3. Exact targeted-turn routing remains unchanged and explicitly covered. +4. Existing regressions remain green and explicit. +5. The gap-matrix row is not marked closed unless code, tests, and docs agree on the same shipped truth. +6. All command gates pass. +7. Acceptance evidence exists before the parent marks the lane green: + - `worker-output.patch` + - `worker-report.md` + - `commands.txt` with exit codes + - `evidence-manifest.json` + +## Validation Wall And Final Closeout + +### `task/m16-g4-validation-wall-gate` + +Owner: + +- parent only + +Checks: + +1. `L3` returned and is classified before final validation. +2. `L3` is `accepted`. +3. No quarantined or blocked output remains unresolved. +4. `validation-wall.md` names the exact final command order. +5. `validation-wall.md` includes contract assertions for: + - exact backend-id selection + - coexistence of distinct backend ids + - duplicate same-backend fail-closed behavior + - unchanged targeted-turn grammar and routing + - no new CLI surface + - no macOS parity broadening + +Acceptance: + +1. The parent writes `.runs/plan-16/sentinels/task-m16-g4-validation-wall-gate.ok`. + +### `task/m16-p3-parent-validation-wall-and-closeout` + +Owner: + +- parent only + +Scope: + +1. Integrate only accepted `L3` output. +2. Run the full validation wall in exact order. +3. Record final command results and acceptance evidence in `.runs/task-m16-p3-parent-validation-wall-and-closeout/artifacts/`. +4. Mark the run complete only if the validation wall proves the frozen contract rather than merely compiling. + +Required validation commands, executed in this order: + +```bash +cargo fmt --all -- --check +cargo clippy --workspace --all-targets -- -D warnings +cargo test -p shell --lib -- --nocapture +cargo test -p shell validate_exact_backend_selection -- --nocapture +cargo test -p shell --test agents_validate -- --nocapture +cargo test -p shell --test repl_world_first_routing_v1 -- --nocapture +cargo test -p world-agent --lib -- --nocapture +cargo test -p world-agent member_runtime -- --nocapture +cargo test -p world-agent --test member_runtime_world_placement_v1 -- --nocapture +cargo test -p world-agent --test streamed_execute_cancel_v1 -- --nocapture +substrate world doctor --json +substrate shim doctor --json +substrate health --json +``` + +Required artifacts under `.runs/task-m16-p3-parent-validation-wall-and-closeout/artifacts/`: + +- `fmt.txt` +- `clippy.txt` +- `shell-lib-tests.txt` +- `validate-exact-backend-selection.txt` +- `agents-validate.txt` +- `repl-world-first-routing.txt` +- `world-agent-lib-tests.txt` +- `world-agent-member-runtime.txt` +- `world-agent-member-runtime-placement.txt` +- `world-agent-streamed-execute-cancel.txt` +- `world-doctor.json` +- `shim-doctor.json` +- `health.json` +- `contract-audit.md` +- `closeout.md` + +Required contract assertions in `contract-audit.md`: + +1. Every explicit backend-targeted path still resolves by exact derived `backend_id`. +2. Wrong-scope, denied, unrealizable, duplicate, and missing backend failures remain exact and fail closed. +3. Distinct backend ids can coexist as retained live world members in one session and generation. +4. Duplicate same-backend retained live members still fail closed. +5. The shell reuses retained world members by exact backend id, not by `agent_id`. +6. Existing targeted-turn routing and grammar behavior remain unchanged. +7. Existing same-generation same-backend reuse remains green. +8. No new CLI surface was introduced. +9. No macOS parity work was introduced. +10. The gap matrix matches shipped reality. + +Acceptance: + +1. The full validation wall passes in order. +2. The doctor outputs are captured. +3. The artifact audit proves coexistence, duplicate-fail-closed behavior, unchanged exact routing, and unchanged regressions. +4. The parent writes `.runs/plan-16/sentinels/task-m16-p3-parent-validation-wall-and-closeout.ok`. + +## Gates + +Gate sequencing is serialized and parent-owned: + +1. `task/m16-p1-parent-identity-freeze-and-run-init` +2. `task/m16-g1-worker-launch-gate` +3. parallel launch of `task/m16-l1-shell-retained-backend-map` and `task/m16-l2-world-agent-retained-coexistence` +4. `task/m16-g2-code-lane-integration-gate` +5. `task/m16-p2-parent-code-lane-integration` +6. `task/m16-g3-closeout-launch-gate` +7. `task/m16-l3-tests-docs-gap-closeout` +8. `task/m16-g4-validation-wall-gate` +9. `task/m16-p3-parent-validation-wall-and-closeout` + +Gate rules: + +1. The parent updates `run-state.json`, `tasks.json`, and `session-log.md` at every gate transition, worker launch, worker return, retry authorization, quarantine decision, merge refusal, and terminal closeout. +2. A gate is green only when all prerequisite sentinels and evidence for the prior phase exist. +3. A gate cannot go green on narrative progress alone. It requires command evidence, artifact presence, and ownership compliance. +4. `g2` and `g4` are classification gates, not integration gates. They may accept, reject, quarantine, or block. They do not mutate production files. + +### Retry and redrive policy + +1. Each worker lane has retry budget `1`. +2. The parent must classify the first attempt before authorizing any retry. +3. A retry is allowed only for lane-local failure inside the lane's owned files. +4. A retry is allowed only if the first attempt did not violate a hard guard, frozen contract clause, or cross-lane ownership rule. +5. Grammar drift, CLI-surface broadening, macOS scope broadening, duplicate-same-backend permissiveness, or selector-key regression are non-retryable. +6. If a lane exhausts retry budget without acceptance, the run blocks. + +### Merge refusal rules + +1. The parent refuses merge for any lane whose patch requires parent-only or cross-lane file edits. +2. The parent refuses merge for any lane that is locally green but semantically contradicts `identity-freeze.json`. +3. The parent refuses merge for any lane that encodes intermediate behavior as final repo truth. +4. Merge refusal writes: + - `.runs/task-/artifacts/merge-refusal.md` + - `.runs/task-/rejected.json` + - `.runs/plan-16/quarantine//` + +## Worker Interfaces + +### Worker prompt contract + +Every worker prompt sent by the parent must include exactly: + +1. task ID +2. attempt number +3. worktree path +4. branch name +5. owned files +6. forbidden files +7. exact frozen contract clauses relevant to that lane +8. command gates +9. retry budget +10. required return artifacts +11. sentinel name + +The parent must not send: + +1. discretionary broad goals without file boundaries +2. permission to edit parent-owned or cross-lane files +3. instructions that reopen `p1` contract decisions +4. instructions that imply hidden concurrency beyond the one honest two-lane window + +### Worker return contract + +Every worker must return exactly: + +1. changed files list +2. commands run, each with exit code +3. explicit attempt classification: `clean`, `retryable`, or `blocked` +4. unresolved assumptions or blockers +5. `worker-output.patch` +6. `worker-report.md` +7. `evidence-manifest.json` + +For a return to be eligible for acceptance, the parent must be able to verify: + +1. the patch stays within owned files +2. command gates actually ran +3. exit codes are present +4. the attempt classification matches the evidence +5. no frozen contract clause was reopened + +## Tests And Acceptance + +### Task-scoped command gates + +`task/m16-p1-parent-identity-freeze-and-run-init` + +```bash +cargo test -p shell validate_exact_backend_selection -- --nocapture +cargo test -p world-agent member_runtime -- --nocapture +``` + +`task/m16-l1-shell-retained-backend-map` + +```bash +cargo test -p shell --lib -- --nocapture +cargo test -p shell validate_exact_backend_selection -- --nocapture +cargo test -p shell --test repl_world_first_routing_v1 --no-run +``` + +`task/m16-l2-world-agent-retained-coexistence` + +```bash +cargo test -p world-agent --lib -- --nocapture +cargo test -p world-agent member_runtime -- --nocapture +cargo test -p world-agent --test member_runtime_world_placement_v1 -- --nocapture +cargo test -p world-agent --test streamed_execute_cancel_v1 --no-run +``` + +`task/m16-p2-parent-code-lane-integration` + +```bash +cargo test -p world-agent --lib -- --nocapture +cargo test -p world-agent member_runtime -- --nocapture +cargo test -p shell --lib -- --nocapture +cargo test -p shell validate_exact_backend_selection -- --nocapture +cargo test -p shell --test repl_world_first_routing_v1 --no-run +``` + +`task/m16-l3-tests-docs-gap-closeout` + +```bash +cargo test -p shell --test repl_world_first_routing_v1 -- --nocapture +cargo test -p shell --test agents_validate -- --nocapture +cargo test -p world-agent --test member_runtime_world_placement_v1 -- --nocapture +cargo test -p world-agent --test streamed_execute_cancel_v1 -- --nocapture +``` + +`task/m16-p3-parent-validation-wall-and-closeout` + +```bash +cargo fmt --all -- --check +cargo clippy --workspace --all-targets -- -D warnings +cargo test -p shell --lib -- --nocapture +cargo test -p shell validate_exact_backend_selection -- --nocapture +cargo test -p shell --test agents_validate -- --nocapture +cargo test -p shell --test repl_world_first_routing_v1 -- --nocapture +cargo test -p world-agent --lib -- --nocapture +cargo test -p world-agent member_runtime -- --nocapture +cargo test -p world-agent --test member_runtime_world_placement_v1 -- --nocapture +cargo test -p world-agent --test streamed_execute_cancel_v1 -- --nocapture +substrate world doctor --json +substrate shim doctor --json +substrate health --json +``` + +### Acceptance matrix + +| Phase | Required proof | Refuse / block trigger | +| --- | --- | --- | +| Parent freeze `p1` | retained-member key, exact failure taxonomy, worker ownership, and validation wall are frozen in artifacts | contract freeze requires grammar, CLI, or macOS scope changes | +| `L1` shell lane | backend-aware retained-member collection, exact backend-id reuse, duplicate fail-closed, exact selector preserved | singleton fallback remains, `agent_id` remains reuse key, or grammar drifts | +| `L2` world lane | distinct backend coexistence lands, duplicate same-backend fail-closed lands, exact submit validation preserved | world-agent still acts singleton, duplicate same-backend is permissive, or transport surface widens | +| Parent integration `p2` | accepted `L2` then accepted `L1`, no hybrid truth, frozen contract preserved | merge requires parent hand-edit around contradiction or combined state weakens duplicate detection | +| `L3` closeout lane | coexistence proof, duplicate-fail-closed proof, unchanged routing regressions, truthful gap matrix | tests encode lane-local assumptions or docs overclaim landed behavior | +| Final validation wall `p3` | full command wall green, doctor evidence captured, contract audit proves coexistence and unchanged regressions | any final assertion fails or audit cannot prove the frozen truth | + +## Acceptance Criteria + +The run is complete only if all of these are true: + +1. The parent froze the retained-member identity contract before worker launch and never reopened it. +2. The only parallel window was `L1` plus `L2`. +3. `L3` started only after code-lane integration was complete. +4. No file ownership overlap occurred outside parent integration and final closeout takeover. +5. Distinct backend ids coexist as retained live world members in one session and generation. +6. Duplicate same-backend retained live members still fail closed. +7. Shell reuse is by exact backend id, not by `agent_id`. +8. Existing exact targeted-turn routing and grammar behavior remain green. +9. Existing same-generation same-backend reuse remains green. +10. No new CLI surface was introduced. +11. No macOS parity work was introduced. +12. The gap matrix is updated to match shipped reality. + +## Final State + +Success requires all of: + +1. every required sentinel exists +2. no blocked artifact exists under `.runs/plan-16/` +3. accepted outputs are integrated in the prescribed order only +4. final validation commands pass +5. `contract-audit.md` proves exact backend-id coexistence, duplicate same-backend fail-closed behavior, unchanged exact targeted-turn routing, and unchanged existing regressions + +Blocked termination requires any of: + +1. hard-guard violation +2. non-retryable rejection +3. exhausted retry budget +4. merge refusal with no legal redrive path +5. failed validation wall or doctor evidence + +On blocked termination the parent must write: + +1. `.runs/plan-16/blocked.json` +2. terminal state and rationale in `run-state.json` +3. gate and failure summary in `session-log.md` +4. preserved evidence under `.runs/plan-16/quarantine/` and the relevant task artifact directory + +## Context-Control Rules + +1. The parent keeps only frozen contract artifacts, active task state, worker reports, narrow diffs, blockers, and validation status in live context. +2. The parent does not keep full worker transcripts in live context. +3. Each worker prompt contains only: + - its owned file set + - the exact relevant `PLAN-16` excerpt + - required commands + - forbidden touch surfaces + - the frozen contract clauses it must honor +4. Workers do not write `.runs/plan-16/*`. +5. If a worker discovers it needs an unowned file, it must stop and report that need instead of silently expanding scope. +6. Close each worker immediately after merge or rejection. diff --git a/llm-last-mile/ORCH_PLAN-17.md b/llm-last-mile/ORCH_PLAN-17.md new file mode 100644 index 000000000..e1e1c1977 --- /dev/null +++ b/llm-last-mile/ORCH_PLAN-17.md @@ -0,0 +1,769 @@ +# ORCH_PLAN-17: Execute PLAN-17 Through A Parent-Frozen Selected-Follow-Up Contract And One Honest Serialized Shell Lane + +Branch: `feat/session-centric-state-store` +Plan source: [PLAN-17.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-17.md) +Style reference: [ORCH_PLAN-15.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/ORCH_PLAN-15.md) +Style reference: [ORCH_PLAN-16.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/ORCH_PLAN-16.md) +Structure reference: [M26 Orchestration Plan](/Users/spensermcconnell/__Active_Code/atomize-hq/spec/docs/m26_orchestration_kickoff_prompt.md) +Gap matrix anchor: [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md) +Execution type: fresh orchestration plan, Linux-first, REPL-targeted selected-member submit/reuse contract hardening, shell-centric helper extraction plus regression-floor completion, parent-frozen scope, parent-only integration and gates +Worker model: GPT-5.4 workers with `reasoning_effort=high` +Max concurrent code workers before integration: `1` + +## Summary + +This document is the execution controller for `PLAN-17`, not a restatement of it. + +The current repo truth is narrower than `PLAN-17`'s initial parallelization sketch: + +1. targeted follow-up submit already exists +2. exact backend-id selection already exists +3. retained world-member coexistence by distinct backend id already exists from `PLAN-16` +4. the real remaining work is contract centralization in [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) plus regression-floor completion and repo-truth docs + +That means this run is intentionally serialized: + +1. `task/m17-p1-parent-contract-freeze-and-run-init` is parent-only and freezes the selected-follow-up contract, stop conditions, lane ownership, merge order, and validation wall. +2. `task/m17-g1-shell-lane-launch-gate` is parent-only and is the only gate that may launch the shell lane. +3. `task/m17-l1-shell-contract-centralization-and-regression-floor` is the single implementation lane. It owns the helper extraction, any selector-level pinning needed by that seam, and the REPL integration test floor in one coherent file set. +4. `task/m17-g2-shell-lane-integration-gate` and `task/m17-p2-parent-shell-lane-integration` are parent-only. +5. `task/m17-l2-docs-gap-matrix-closeout` starts only after `p2` lands merged code truth. +6. `task/m17-g4-validation-wall-gate` and `task/m17-p3-parent-validation-wall-and-closeout` are parent-only and finish the run. + +Canonical task IDs: + +- `task/m17-p1-parent-contract-freeze-and-run-init` +- `task/m17-g1-shell-lane-launch-gate` +- `task/m17-l1-shell-contract-centralization-and-regression-floor` +- `task/m17-g2-shell-lane-integration-gate` +- `task/m17-p2-parent-shell-lane-integration` +- `task/m17-g3-doc-closeout-launch-gate` +- `task/m17-l2-docs-gap-matrix-closeout` +- `task/m17-g4-validation-wall-gate` +- `task/m17-p3-parent-validation-wall-and-closeout` + +## Assumptions + +1. [PLAN-17.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-17.md) remains the authoritative dependency graph for this run. +2. The authoritative integration checkout remains the current workspace checkout on `feat/session-centric-state-store`. +3. `PLAN-16` already landed backend-id-scoped retained-member coexistence, so this run must not reopen `world-agent` cardinality or reintroduce singleton logic. +4. The remaining production hotspot is [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs); the remaining proof hotspot is [crates/shell/tests/repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs) plus its stub support. +5. Linux is the only world-target platform that must go green in this run; non-Linux world follow-up behavior remains fail-closed. +6. Repo-truth docs must move last, after validated merged behavior exists. + +## Immutable Run Shape + +### Frozen contract truth + +These are run-stopping invariants, not preferences: + +1. `:: ` remains the only targeted follow-up grammar. +2. Plain REPL input remains shell execution. +3. `validate_exact_backend_selection(...)` remains the canonical explicit selector. +4. Host follow-up turns may target only the active orchestrator backend for the current REPL session. +5. Linux world follow-up turns continue to submit through `MemberTurnSubmitRequestV1` and `/v1/member_turn/stream`. +6. World-member reuse remains keyed by the exact retained backend slot for the current orchestration session and authoritative `world_generation`. +7. Relaunch remains a readiness concern inside `ensure_member_runtime_ready_for_descriptor(...)`, not a submit concern. +8. Duplicate retained members for the same backend slot remain fail-closed. This run does not reopen `PLAN-16`. +9. `substrate -c` remains shell wrap mode. +10. This run may not widen into status/toolbox redesign, non-REPL caller surfaces, public control-plane productization, or macOS parity. +11. The parent agent is the only integrator and the only approval authority for worker outputs. + +### Parent-only versus worker-owned authority + +Parent-only for the full run: + +- `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/plan-17/*` +- `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/task-m17-*/**` +- worker launch, output classification, merge approval, quarantine, retry approval, and final validation +- any decision to stop the run instead of inventing cross-lane or cross-scope semantics + +Parent-only during `p1`, then frozen by artifact: + +- selected-follow-up failure taxonomy +- helper-seam ownership contract +- lane ownership map +- merge order +- validation-wall command list +- stop conditions + +Worker-owned after `p1`: + +- `L1`: shell helper extraction, selector-dependency pinning, REPL integration tests, stub support +- `L2`: docs and planning-index closeout only + +No worker may edit parent-owned run-state. No worker may edit another lane's owned files. No worker may reopen the frozen scope boundary. + +### Stop conditions + +Stop the run, write `.runs/plan-17/blocked.json`, and do not advance if any of these occur: + +1. The selected-follow-up contract cannot be frozen without redesigning grammar, caller surface, or transport shape. +2. Any implementation requires edits in [crates/world-agent/src/member_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs) to rework retained-member cardinality rather than merely validating current behavior. +3. Any worker broadens scope into `state_store.rs`, `agents_cmd.rs`, CLI command-mode semantics, toolbox surfaces, or macOS/Lima parity. +4. Any worker changes the `:: ` grammar. +5. Any worker attempts to add or imply a new non-REPL caller surface. +6. Any worker edits another lane's owned files or any parent-owned run-state. +7. `L2` starts before `p2` accepts final merged code truth. +8. The parent would need to invent hybrid semantics during integration to make test expectations or doc truth fit behavior that the merged code does not actually implement. +9. The validation wall cannot prove helper centralization, positive host follow-up proof, stale-or-missing world relaunch proof, unchanged exact selector behavior, and unchanged fail-closed non-Linux behavior. +10. The gap matrix is marked closed beyond what the integrated tests and validation wall prove. + +## Fresh Worktrees And Branches + +Fresh worktree root: + +- `/Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-17` + +Worker worktrees: + +- `/Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-17/shell-contract-centralization-and-regression-floor` +- `/Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-17/docs-gap-matrix-closeout` + +Worker branches: + +- `codex/feat-session-centric-state-store-m17-shell-contract-centralization-and-regression-floor` +- `codex/feat-session-centric-state-store-m17-docs-gap-matrix-closeout` + +Exact setup commands: + +```bash +mkdir -p /Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-17 +git worktree add /Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-17/shell-contract-centralization-and-regression-floor -b codex/feat-session-centric-state-store-m17-shell-contract-centralization-and-regression-floor feat/session-centric-state-store +git worktree add /Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-17/docs-gap-matrix-closeout -b codex/feat-session-centric-state-store-m17-docs-gap-matrix-closeout feat/session-centric-state-store +``` + +### Parent integration surface + +The parent integrates on the authoritative checkout already on `feat/session-centric-state-store`. + +No separate parent integration worktree is introduced because: + +1. the run has one serialized integrator +2. `.runs/plan-17/*` is parent-owned state and should stay co-located with the authoritative branch context +3. there is no honest concurrent parent merge activity in this plan + +## Parent-Owned Run-State Surface + +Canonical parent-owned state under `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/plan-17/`: + +- `run-state.json` +- `tasks.json` +- `session-log.md` +- `contract-freeze.json` +- `lane-ownership.json` +- `merge-order.json` +- `validation-wall.md` +- `blocked.json` on failure only +- `quarantine/` +- `sentinels/` + +`contract-freeze.json` is the single source of truth for: + +- `plan_id: "PLAN-17"` +- `plan_source: "llm-last-mile/PLAN-17.md"` +- `orchestration_plan_source: "llm-last-mile/ORCH_PLAN-17.md"` +- `branch: "feat/session-centric-state-store"` +- `selector_authority: "validate_exact_backend_selection"` +- `targeted_grammar: ":: "` +- `host_target_rule: "active_orchestrator_backend_only"` +- `world_target_rule: "linux_first_member_turn_submit_only"` +- `reuse_rule: "exact_backend_slot_for_current_world_generation"` +- `relaunch_rule: "readiness_only_when_missing_invalidated_or_stale"` +- `plan_16_cardinality_reopened: false` +- `new_caller_surface_allowed: false` +- `validation_commands` +- `stop_conditions` + +`lane-ownership.json` is the single source of truth for: + +- task ID +- owner +- worktree path +- branch +- allowed files +- forbidden files +- command gates +- retry budget +- merge order +- sentinel filename + +`run-state.json` is the single source of truth for: + +- `current_phase` +- `active_task_ids` +- `worker_cap` +- `contract_freeze_status` +- `lane_status` +- `accepted_outputs` +- `rejected_outputs` +- `quarantined_outputs` +- `blocked_outputs` +- `retry_budget_by_lane` +- `attempt_counts` +- `integration_order` +- `gate_status` +- `validation_wall_status` +- `termination_reason` +- `terminal_state` + +### Kickoff initialization order + +The parent initializes the run in this exact order before any worker prompt is written: + +1. Create `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/plan-17/`, `.runs/plan-17/sentinels/`, `.runs/plan-17/quarantine/`, and every `.runs/task-m17-*/` directory. +2. Write `tasks.json` as the canonical queue and ledger for the entire run. +3. Write `run-state.json` with: + - `current_phase: "kickoff"` + - `worker_cap: 1` + - every task id in `pending` + - empty accepted/rejected/quarantined output arrays +4. Write `contract-freeze.json`, `lane-ownership.json`, `merge-order.json`, and `validation-wall.md`. +5. Write `session-log.md` with the kickoff timestamp, authoritative branch, worktree roots, and the explicit statement that max honest worker concurrency is `1`. +6. Write each per-task `task.json`, `commands.txt`, and `summary.md` stub before launch packets are generated. +7. Seed worker worktrees only after the above artifacts exist and `task-m17-p1-parent-contract-freeze-and-run-init` is marked complete. + +`tasks.json` is both the launch queue and the execution ledger. Each row must include at least: + +- `task_id` +- `title` +- `owner` +- `status` +- `phase_class` +- `depends_on` +- `worktree_path` when applicable +- `branch` when applicable +- `allowed_files` +- `forbidden_files` +- `command_gates` +- `acceptance_checks` +- `sentinel_path` +- `artifact_dir` + +Gate-proof artifacts are parent-owned and mandatory: + +- each gate task directory contains `gate-checklist.md` +- each gate task directory contains `gate-result.json` +- each gate result records: + - checked prerequisites + - artifact paths inspected + - pass/fail classification + - approval timestamp + - approver: `parent` + +Required sentinels: + +- `.runs/plan-17/sentinels/task-m17-p1-parent-contract-freeze-and-run-init.ok` +- `.runs/plan-17/sentinels/task-m17-g1-shell-lane-launch-gate.ok` +- `.runs/plan-17/sentinels/task-m17-l1-shell-contract-centralization-and-regression-floor.ok` +- `.runs/plan-17/sentinels/task-m17-g2-shell-lane-integration-gate.ok` +- `.runs/plan-17/sentinels/task-m17-p2-parent-shell-lane-integration.ok` +- `.runs/plan-17/sentinels/task-m17-g3-doc-closeout-launch-gate.ok` +- `.runs/plan-17/sentinels/task-m17-l2-docs-gap-matrix-closeout.ok` +- `.runs/plan-17/sentinels/task-m17-g4-validation-wall-gate.ok` +- `.runs/plan-17/sentinels/task-m17-p3-parent-validation-wall-and-closeout.ok` + +Per-task artifact directories: + +- `.runs/task-m17-p1-parent-contract-freeze-and-run-init/` +- `.runs/task-m17-g1-shell-lane-launch-gate/` +- `.runs/task-m17-l1-shell-contract-centralization-and-regression-floor/` +- `.runs/task-m17-g2-shell-lane-integration-gate/` +- `.runs/task-m17-p2-parent-shell-lane-integration/` +- `.runs/task-m17-g3-doc-closeout-launch-gate/` +- `.runs/task-m17-l2-docs-gap-matrix-closeout/` +- `.runs/task-m17-g4-validation-wall-gate/` +- `.runs/task-m17-p3-parent-validation-wall-and-closeout/` + +Each task directory contains: + +- `task.json` +- `summary.md` +- `commands.txt` +- `artifacts/` +- `evidence-manifest.json` + +Each gate task directory also contains: + +- `gate-checklist.md` +- `gate-result.json` + +Each worker task directory also contains: + +- `worker-output.patch` +- `worker-report.md` +- `touched-files.txt` +- `command-results.md` +- `rejected.json` on rejection only +- `blocked.json` on blocked lane return only +- `quarantine/` when the parent quarantines the output + +Blocked termination minimum contents: + +1. task or gate where execution stopped +2. classification: `rejected`, `blocked`, or `merge_refused` +3. exact contract clause or ownership rule that stopped the run +4. whether retry remained available +5. artifact paths for patch, report, command output, and semantic-drift evidence +6. explicit statement that no blocked output was integrated + +## Phase Graph And Concurrency + +Parent checkout: + +- current authoritative checkout on `feat/session-centric-state-store` + +Concurrency rules: + +1. Worker cap is `1` for the entire run. +2. `p1` must finish before any worker starts. +3. `g1` must accept before `L1` starts. +4. `g2` classifies `L1` before any docs work starts. +5. `p2` integrates `L1` on the authoritative checkout. +6. `g3` starts only after `p2` is green. +7. `L2` runs alone on the exact post-`p2` tree. +8. `g4` and `p3` are parent-only. +9. No parallel worker window is allowed because the real remaining scope converges on one shared shell/test seam and docs must encode merged, validated truth rather than lane-local guesses. + +### Why this run is serialized + +1. [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) is the production hotspot for contract centralization and the only honest place to own any follow-on readiness clarification. +2. [crates/shell/tests/repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs) and [crates/shell/tests/support/repl_world_agent.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/support/repl_world_agent.rs) depend on the exact merged helper behavior, so splitting them into a concurrent lane would create patch ping-pong rather than speed. +3. Docs can only close after code and tests agree on the same narrowed contract. +4. The earlier `PLAN-17` “two parallel implementation lanes” sketch is not honest against the current repo truth and should not drive execution. + +## Kickoff And Execution Hygiene + +### Parent working context + +The parent keeps only the minimum live context needed to run the controller accurately: + +- [PLAN-17.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-17.md) +- [ORCH_PLAN-17.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/ORCH_PLAN-17.md) +- `.runs/plan-17/tasks.json` +- `.runs/plan-17/run-state.json` +- `.runs/plan-17/contract-freeze.json` +- the current gate's `gate-checklist.md` and latest accepted `gate-result.json` +- the latest integration diff summary once `p2` begins + +The parent should not keep broad exploratory repo context live after `p1`. If a later decision depends on fresh repo evidence outside the frozen contract, the parent re-reads that evidence deliberately and records it in the relevant task artifact directory. + +### Worker prompt contents + +Every worker launch packet must be written by the parent under the worker task artifact directory before the worker starts. Each launch packet must contain: + +- the exact task id and worker branch/worktree path +- the owned file set +- the forbidden file set +- the exact `PLAN-17` step excerpt relevant to that lane +- the frozen contract clauses that lane must preserve +- the command gates the worker must run +- the acceptance checks the worker must satisfy +- the sentinel path the worker is expected to cause the parent to mark after acceptance +- the retry budget +- the explicit blocked-run rule for out-of-scope discoveries + +### Worker return contract + +Every worker must return a bounded artifact package, not freeform commentary. The required return surface is: + +- `worker-output.patch` +- `worker-report.md` +- `touched-files.txt` +- `command-results.md` +- `evidence-manifest.json` + +`worker-report.md` must contain: + +- summary of what changed +- exact commands run +- pass/fail results +- unresolved risks +- explicit statement whether the lane stayed within scope +- explicit statement whether any blocked condition was discovered + +`command-results.md` must include enough output detail for the parent to classify the lane without rerunning guesswork. + +### Prompt and artifact hygiene rules + +1. Worker prompts may quote only the narrow `PLAN-17` and `ORCH_PLAN-17` excerpts needed for that lane. +2. Worker prompts may not paste unrelated repo docs or reopen already-frozen scope debates. +3. The parent is the only actor that updates `tasks.json`, `run-state.json`, sentinels, gate artifacts, or quarantine directories. +4. Workers return patches and evidence only. They do not self-approve gates. +5. If a worker finds a genuine out-of-scope blocker, it returns `blocked.json` and stops instead of proposing a broadened implementation. + +## PLAN-17 Step Mapping + +| Orchestration task | PLAN-17 step alignment | +| --- | --- | +| `task/m17-p1-parent-contract-freeze-and-run-init` | Step 1: freeze the targeted-follow-up contract | +| `task/m17-l1-shell-contract-centralization-and-regression-floor` | Step 2, Step 3, Step 4, and Step 5 | +| `task/m17-l2-docs-gap-matrix-closeout` | Step 6 | +| `task/m17-p3-parent-validation-wall-and-closeout` | Recommended verification commands plus Definition of Done enforcement | + +## Lane Ownership By File Set + +| Lane | Allowed files | Forbidden escalation surfaces | +| --- | --- | --- | +| `L1` | [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs), [crates/shell/src/execution/agent_runtime/validator.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/validator.rs), [crates/shell/tests/repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs), [crates/shell/tests/support/repl_world_agent.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/support/repl_world_agent.rs) | any `crates/world-agent/**` production code, `crates/agent-api-types/**`, `crates/shell/src/execution/agent_runtime/state_store.rs`, `crates/shell/src/execution/agents_cmd.rs`, CLI command-mode files, docs | +| `L2` | [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md), [llm-last-mile/README.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/README.md) if needed | every Rust file and every runtime/control-plane surface | + +## Parent Phase Details + +### `task/m17-p1-parent-contract-freeze-and-run-init` + +Owner: + +- parent only + +Scope: + +1. Initialize `.runs/plan-17/*` and `.runs/task-m17-*/**`. +2. Freeze the exact failure taxonomy, helper-seam responsibility, no-new-surface boundary, and no-`PLAN-16`-redo rule into `contract-freeze.json`. +3. Freeze worker file ownership, merge order, retry budget, and stop conditions. +4. Freeze the exact validation-wall commands the parent will later execute. +5. Seed all worker worktrees from the exact same post-`p1` tree. + +Expected source files reviewed in `p1` but not lane-opened for shared mutation: + +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) +- [crates/shell/src/execution/agent_runtime/validator.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/validator.rs) +- [crates/shell/tests/repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs) +- [crates/shell/tests/support/repl_world_agent.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/support/repl_world_agent.rs) +- [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md) + +Command gates: + +```bash +cargo test -p shell validate_exact_backend_selection -- --nocapture +cargo test -p shell --test repl_world_first_routing_v1 -- --nocapture +``` + +Acceptance: + +1. `contract-freeze.json`, `lane-ownership.json`, `merge-order.json`, `run-state.json`, and `tasks.json` exist. +2. The frozen contract explicitly records that this is a contract-centralization and regression-floor run, not a transport or retained-cardinality run. +3. No production-file edits are integrated in `p1` unless the parent takes explicit hotspot ownership and reseeds all worker branches afterward. +4. The parent writes `.runs/plan-17/sentinels/task-m17-p1-parent-contract-freeze-and-run-init.ok`. + +### `task/m17-g1-shell-lane-launch-gate` + +Owner: + +- parent only + +Checks: + +1. `p1` is accepted. +2. The `L1` worktree was seeded from the exact same post-`p1` tree the parent will integrate against. +3. The `L1` prompt names only owned files, forbidden surfaces, command gates, sentinel, and retry budget. +4. The prompt explicitly repeats that `PLAN-16` retained-member coexistence is already landed and may not be reopened. + +Acceptance: + +1. No worker starts before this gate is green. +2. The parent writes `.runs/plan-17/sentinels/task-m17-g1-shell-lane-launch-gate.ok`. + +## Worker Lanes + +### `task/m17-l1-shell-contract-centralization-and-regression-floor` + +Owner: + +- single worker on `codex/feat-session-centric-state-store-m17-shell-contract-centralization-and-regression-floor` + +Owned files: + +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) +- [crates/shell/src/execution/agent_runtime/validator.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/validator.rs) +- [crates/shell/tests/repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs) +- [crates/shell/tests/support/repl_world_agent.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/support/repl_world_agent.rs) + +Scope: + +1. Extract one explicit targeted-follow-up dispatch seam in `async_repl.rs`. +2. Keep `submit_host_targeted_turn(...)` and `submit_world_targeted_turn(...)` as distinct transport implementations. +3. Keep reuse-vs-relaunch inside `ensure_member_runtime_ready_for_descriptor(...)`. +4. Add positive host follow-up proof for the active orchestrator backend. +5. Add stale-or-missing world-runtime proof that exact-backend relaunch happens before submit. +6. Add selector-level tests only where the new seam depends on exactness not already pinned. + +Command gates: + +```bash +cargo test -p shell validate_exact_backend_selection -- --nocapture +cargo test -p shell --test repl_world_first_routing_v1 -- --nocapture +``` + +Acceptance: + +1. One obvious helper seam owns exact backend selection result handling, host-vs-world classification, host activity check, Linux-only world gate, readiness call, and final submit choice. +2. The positive host targeted-follow-up path is proven, not inferred. +3. Same-generation world-member reuse remains proven and unchanged. +4. Stale-or-missing retained world state is proven to relaunch the exact backend slot before submit rather than fall back to sibling or shell behavior. +5. Any selector exactness relied on by the seam is pinned in `validator.rs` tests. +6. No owned patch broadens into status/toolbox, CLI caller surfaces, macOS parity, or `PLAN-16` cardinality work. +7. The worker writes `.runs/plan-17/sentinels/task-m17-l1-shell-contract-centralization-and-regression-floor.ok`. + +### `task/m17-g2-shell-lane-integration-gate` + +Owner: + +- parent only + +Checks: + +1. `L1` returned a patch, report, command transcript, and evidence manifest. +2. Every touched file is inside the `L1` ownership boundary. +3. No patch reopens `world-agent` retained-member ownership, selector grammar, or caller-surface shape. +4. The parent can describe the merged selected-follow-up contract without inventing semantics not present in code and tests. + +Acceptance: + +1. Accepted `L1` output is marked in `run-state.json`. +2. Rejected or quarantined output is recorded explicitly if `L1` drifted. +3. The parent writes `.runs/plan-17/sentinels/task-m17-g2-shell-lane-integration-gate.ok`. + +### `task/m17-p2-parent-shell-lane-integration` + +Owner: + +- parent only + +Scope: + +1. Integrate the accepted `L1` patch onto the authoritative checkout. +2. Resolve only straightforward merge mechanics within `L1`-owned surfaces. +3. Re-run the `L1` command gates on the authoritative checkout. +4. Freeze the merged code truth before any doc edits begin. + +Acceptance: + +1. The authoritative checkout now contains the helper seam and the merged regression floor. +2. No post-merge edit widens scope beyond `L1` ownership without stopping and re-planning. +3. The parent writes `.runs/plan-17/sentinels/task-m17-p2-parent-shell-lane-integration.ok`. + +### `task/m17-g3-doc-closeout-launch-gate` + +Owner: + +- parent only + +Checks: + +1. `p2` is green. +2. The doc worker is seeded from the exact post-`p2` tree. +3. The doc prompt names only the allowed doc files and repeats that docs must describe the runtime as Linux-first and REPL-first, without implying broader caller-surface completion. + +Acceptance: + +1. No doc worker starts before this gate is green. +2. The parent writes `.runs/plan-17/sentinels/task-m17-g3-doc-closeout-launch-gate.ok`. + +### `task/m17-l2-docs-gap-matrix-closeout` + +Owner: + +- single worker on `codex/feat-session-centric-state-store-m17-docs-gap-matrix-closeout` + +Owned files: + +- [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md) +- [llm-last-mile/README.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/README.md) if the README still needs an index update after merged truth review + +Scope: + +1. Describe targeted follow-up submit and reuse as landed, Linux-first, and REPL-first. +2. Describe remaining open work as broader non-REPL caller surface, status/toolbox productization, and macOS parity. +3. Remove stale language that still implies selected-member follow-up submit is broadly unimplemented. + +Command gates: + +```bash +rg -n "REPL-first|Linux-first|targeted follow-up|member_turn|substrate -c" AGENT_ORCHESTRATION_GAP_MATRIX.md llm-last-mile/README.md +``` + +Acceptance: + +1. The docs do not claim a new caller surface or any macOS parity that the merged code does not prove. +2. The gap matrix and optional README index agree with the merged shell/test truth. +3. The worker writes `.runs/plan-17/sentinels/task-m17-l2-docs-gap-matrix-closeout.ok`. + +## Quarantine, Retry, And Blocked-Run Posture + +1. `L1` gets one retry only for mechanical defects inside its owned files. +2. If `L1` returns evidence that a correct fix would require `world-agent` production changes, state-store/status changes, CLI caller-surface changes, or macOS work, the parent does not repurpose the lane. The run stops as blocked. +3. `L2` gets one retry only for stale phrasing or overclaiming inside owned docs. +4. If `L2` can only make docs “pass” by broadening claims beyond the validated runtime behavior, the parent rejects the lane and stops the run instead of publishing optimistic docs. +5. The parent never hand-merges a hybrid contract out of conflicting worker guesses. When semantics drift, quarantine the output and either retry within the same owned file set or stop. + +## Tests And Acceptance + +This is the parent’s operator checklist for deciding whether the run can advance and whether the final state is acceptable. + +### A. Contract-surface acceptance + +- The targeted-follow-up grammar remains exactly `:: `. +Proof source: +- merged [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) +- `cargo test -p shell --test repl_world_first_routing_v1 -- --nocapture` + +- `validate_exact_backend_selection(...)` remains the canonical selector. +Proof source: +- merged [validator.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/validator.rs) +- `cargo test -p shell validate_exact_backend_selection -- --nocapture` + +- No new non-REPL caller surface is introduced. +Proof source: +- touched-file manifest for `L1` +- absence of edits in CLI/status/toolbox surfaces + +### B. Shell implementation acceptance + +- One explicit helper seam now owns route classification, host/world gating, readiness dispatch, and final submit choice. +Proof source: +- merged [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) +- `L1` report plus parent integration review + +- `submit_host_targeted_turn(...)` and `submit_world_targeted_turn(...)` remain separate transport implementations. +Proof source: +- merged [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) + +- `ensure_member_runtime_ready_for_descriptor(...)` remains the relaunch authority. +Proof source: +- merged [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) +- stale-or-missing world-runtime regression + +### C. Host-path acceptance + +- Positive host targeted follow-up for the active orchestrator backend is proven. +Proof source: +- merged [repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs) +- `cargo test -p shell --test repl_world_first_routing_v1 -- --nocapture` + +- Non-active host backend rejection remains fail-closed with no shell fallback and no world-member launch. +Proof source: +- merged [repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs) +- same integration test command + +### D. World-path acceptance + +- Same-generation exact world-member reuse stays green. +Proof source: +- merged [repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs) +- same integration test command + +- Stale-or-missing retained world state is proven to relaunch the exact backend slot before submit. +Proof source: +- merged [repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs) +- any required stub support in [repl_world_agent.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/support/repl_world_agent.rs) +- same integration test command + +- Non-Linux world-target behavior remains explicit fail-closed rather than silently routing elsewhere. +Proof source: +- merged [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) +- validation-wall review and unchanged `world-agent` tests + +### E. Regression-floor acceptance + +- `cargo fmt --all -- --check` is green. +- `cargo clippy --workspace --all-targets -- -D warnings` is green. +- `cargo test -p shell validate_exact_backend_selection -- --nocapture` is green. +- `cargo test -p shell --test repl_world_first_routing_v1 -- --nocapture` is green. +- `cargo test -p world-agent member_runtime -- --nocapture` is green. + +### F. Docs-truth acceptance + +- [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md) says selected-member follow-up submit/reuse is landed, Linux-first, and REPL-first. +- [llm-last-mile/README.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/README.md) is updated only if needed to keep the packet index truthful. +- Docs do not claim status/toolbox productization, non-REPL caller completion, world-agent cardinality redesign, or macOS parity. + +### G. Run-control acceptance + +- Every completed task has its sentinel. +- Every gate has a `gate-result.json`. +- `tasks.json` and `run-state.json` match the actual accepted/rejected/quarantined outcomes. +- No quarantined or blocked output was partially integrated. + +## Validation Wall + +Parent-owned validation commands, executed only after `L2` is integrated: + +```bash +cargo fmt --all -- --check +cargo clippy --workspace --all-targets -- -D warnings +cargo test -p shell validate_exact_backend_selection -- --nocapture +cargo test -p shell --test repl_world_first_routing_v1 -- --nocapture +cargo test -p world-agent member_runtime -- --nocapture +``` + +Validation-wall requirements: + +1. formatting and clippy stay green +2. exact backend selector regressions stay green +3. REPL integration tests prove malformed syntax rejection, positive host follow-up, host mismatch rejection, same-generation world reuse, and stale-or-missing world relaunch behavior +4. `world-agent` member-runtime tests stay green to prove no regression in the already-landed retained-member contract +5. docs do not overclaim beyond what the green runtime/tests prove + +### `task/m17-g4-validation-wall-gate` + +Owner: + +- parent only + +Checks: + +1. `L2` has been integrated or explicitly skipped because no README index change was needed. +2. All required sentinels through `L2` exist. +3. The parent can enumerate every definition-of-done clause and the command or artifact that proves it. + +Acceptance: + +1. The validation wall is permitted to run exactly once on the final merged tree. +2. The parent writes `.runs/plan-17/sentinels/task-m17-g4-validation-wall-gate.ok`. + +### `task/m17-p3-parent-validation-wall-and-closeout` + +Owner: + +- parent only + +Scope: + +1. Run the full validation wall on the final merged tree. +2. Record command outputs and artifact paths in the final task directory. +3. Confirm the gap matrix and optional README index match the validated runtime truth. +4. Mark the run complete only if every definition-of-done clause is proven. + +Acceptance: + +1. All validation commands succeed on the authoritative checkout. +2. `run-state.json` records a successful terminal state. +3. The parent writes `.runs/plan-17/sentinels/task-m17-p3-parent-validation-wall-and-closeout.ok`. + +## Definition-Of-Done Enforcement + +The run is complete only if all of these are true: + +1. `:: ` remains the only targeted follow-up grammar. +2. exact backend selection still resolves through `validate_exact_backend_selection(...)`. +3. one explicit helper seam in `async_repl.rs` owns the selected-follow-up orchestration contract. +4. host targeted follow-up turns resume only the active orchestrator backend and now have positive proof. +5. Linux world targeted follow-up turns still submit through `MemberTurnSubmitRequestV1`. +6. same-generation exact world-member reuse remains green. +7. stale or missing retained world state causes exact-backend relaunch before submit, not sibling fallback. +8. missing-active-host-runtime and non-Linux world-targeting failures remain explicit and fail closed. +9. no new non-REPL caller surface is introduced. +10. repo docs say REPL-first and Linux-first plainly without implying broader completion. + +## Execution-To-Completion Notes + +This controller is intentionally end-to-end, not a kickoff memo: + +1. the parent freezes the contract before workers start +2. one worker lands the helper seam plus regression floor +3. the parent integrates and rechecks that code truth +4. one worker updates repo-truth docs after the merged code truth exists +5. the parent runs the full validation wall and closes the run only on green proof + +No additional parallel lane should be introduced mid-run unless the parent rewrites this controller first. diff --git a/llm-last-mile/ORCH_PLAN-18.md b/llm-last-mile/ORCH_PLAN-18.md new file mode 100644 index 000000000..49f91b0b6 --- /dev/null +++ b/llm-last-mile/ORCH_PLAN-18.md @@ -0,0 +1,1011 @@ +# ORCH_PLAN-18: Execute PLAN-18 Through A Parent-Frozen Degraded-Status Contract And One Honest Two-Lane Status Split + +Branch: `feat/session-centric-state-store` +Plan source: [PLAN-18.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-18.md) +Style reference: [ORCH_PLAN-17.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/ORCH_PLAN-17.md) +Style reference: [ORCH_PLAN-16.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/ORCH_PLAN-16.md) +Structure reference: [M26 Orchestration Plan](/Users/spensermcconnell/__Active_Code/atomize-hq/spec/docs/m26_orchestration_kickoff_prompt.md) +Gap matrix anchor: [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md) +Execution type: fresh orchestration plan, Linux-first, status/read-side hardening, state-store plus `agents_cmd` split, tests/docs closeout after integration, parent-frozen contract, parent-only integration and approval +Worker model: GPT-5.4 workers with `reasoning_effort=high` +Max concurrent code workers before integration: `2` + +## Summary + +This document is the execution controller for `PLAN-18`, not a restatement of it. + +The run shape is frozen up front: + +1. `task/m18-p1-parent-contract-freeze-and-run-init` is parent-only and freezes the degraded-status contract, the naming contract, the participant-aware identity contract, lane ownership, merge order, retry budget, stop conditions, and the validation wall. +2. `task/m18-g1-implementation-lane-launch-gate` is parent-only and is the only gate that may launch the implementation workers. +3. The only honest parallel implementation window is exactly two lanes after `p1`: + - `task/m18-l1-status-store-enumeration-and-naming-cleanup` + - `task/m18-l2-status-rendering-fallback-and-nested-correlation` +4. `task/m18-g2-code-lane-integration-gate` and `task/m18-p2-parent-code-lane-integration` are parent-only. +5. Tests/docs closeout is not parallel with code lanes. It starts only after `p2` lands merged code truth: + - `task/m18-g3-closeout-launch-gate` + - `task/m18-l3-tests-docs-gap-closeout` +6. `task/m18-g4-validation-wall-gate` and `task/m18-p3-parent-validation-wall-and-closeout` are parent-only and finish the run. + +Canonical task IDs: + +- `task/m18-p1-parent-contract-freeze-and-run-init` +- `task/m18-g1-implementation-lane-launch-gate` +- `task/m18-l1-status-store-enumeration-and-naming-cleanup` +- `task/m18-l2-status-rendering-fallback-and-nested-correlation` +- `task/m18-g2-code-lane-integration-gate` +- `task/m18-p2-parent-code-lane-integration` +- `task/m18-g3-closeout-launch-gate` +- `task/m18-l3-tests-docs-gap-closeout` +- `task/m18-g4-validation-wall-gate` +- `task/m18-p3-parent-validation-wall-and-closeout` + +## Assumptions + +1. [PLAN-18.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-18.md) remains the authoritative dependency graph for this run. +2. The authoritative integration checkout remains the current workspace checkout on `feat/session-centric-state-store`. +3. The authoritative production hotspots are [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) and [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs). +4. The tests/docs hotspot is [crates/shell/tests/agent_successor_contract_ahcsitc0.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_successor_contract_ahcsitc0.rs) plus [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md) and [llm-last-mile/README.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/README.md). +5. Linux-first status and naming cohesion is the only platform goal for this run. No macOS/Lima parity work is honest here. +6. Docs move last, after integrated code and green regressions exist. +7. Lane B is allowed to depend on a parent-frozen seam contract, but it is not allowed to invent a duplicate status enumeration path in `agents_cmd.rs`. + +## Immutable Run Shape + +### Frozen contract truth + +These are run-stopping invariants, not preferences: + +1. `substrate agent status` becomes permissive in rendering only. +2. `substrate agent doctor`, `substrate agent toolbox status`, `substrate agent toolbox env`, and any future public `start|resume|fork|stop` surface remain strict and fail closed. +3. `StatusSessionJson.source_kind` is mandatory and exact. Allowed values are `live_runtime` and `trace_fallback`. +4. `StatusReportJson.warnings` is mandatory as `Vec`. +5. Warning strings are sorted, deduplicated, and human-readable before render. +6. Record-local degradation warnings come from `AgentRuntimeSessionRecord.warnings`. +7. Cross-record ambiguity warnings are emitted by `build_status_report(...)`. +8. One and only one identity family is allowed for status selection and suppression: + +```text +StatusIdentityKey { + orchestration_session_id: String, + agent_id: String, + execution_scope: "host" | "world", + participant_id: Option, +} +``` + +9. The same `StatusIdentityKey` contract is used for pure-agent trace selection, live/tombstone fallback suppression, and nested selected-parent bucketing. +10. Legacy coarse grouping is allowed only for rows that truly lack `participant_id`. +11. The canonical state-store seam is exactly `list_status_sessions_for_agent(&self, orchestrator_agent_id: &str) -> Result>`. +12. Live authoritative rows and invalidated tombstones continue to win over trace fallback. +13. Public/operator-facing naming stays frozen: + - public selector: `orchestration_session_id` + - runtime lineage id: `participant_id` + - internal-only backend-native handle: `internal.uaa_session_id` +14. Legacy compatibility reads for `session_handle_id`, `parent_session_handle_id`, and `resumed_from_session_handle_id` remain supported. +15. This run must not widen into: + - public `start|resume|fork|stop` surfaces + - `substrate -c` redesign + - world-member follow-up or replacement semantics + - weakening doctor/toolbox fail-closed behavior + - trace producer schema expansion without concrete proof + - storage flag-day rename away from `active_session_handle_id` + - macOS/Lima parity +16. The parent agent is the only integrator and the only approval authority. + +### Parent-only versus worker-owned authority + +Parent-only for the full run: + +- `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/plan-18/*` +- `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/task-m18-*/**` +- worker launch, output classification, merge approval, quarantine, retry approval, final validation, and blocked-run termination +- any decision to stop rather than invent cross-lane semantics + +Parent-only during `p1`, then frozen by artifact: + +- degraded-status contract +- naming contract +- `StatusIdentityKey` contract +- the exact `list_status_sessions_for_agent(...)` seam +- worker cap and lane ownership +- merge order +- validation-wall command list +- stop conditions +- whether a compile-only seam scaffold is needed before worker seeding + +Worker-owned after `p1`: + +- `L1` / Lane A: state-store enumeration seam, Linux-first naming/comment cleanup, and any inline unit-test changes inside its owned runtime files only if required by that work +- `L2` / Lane B: `agents_cmd.rs` status rendering, warnings wiring, participant-aware fallback suppression, and nested parent correlation only +- `L3` / Lane C: integration regressions in `agent_successor_contract_ahcsitc0.rs` and repo-truth docs only + +No worker may edit parent-owned run-state. No worker may edit another lane’s owned files. No worker may reopen the frozen contract. + +### Stop conditions + +Stop the run, write `.runs/plan-18/blocked.json`, and do not advance if any of these occur: + +1. The degraded-status contract cannot be frozen without changing public control-surface shape. +2. Lane B can only work by duplicating status enumeration logic locally instead of consuming the frozen `list_status_sessions_for_agent(...)` seam. +3. Any implementation weakens `resolve_single_live_session_for_agent(...)` for doctor, toolbox, or future mutating control paths. +4. Any implementation broadens into `start|resume|fork|stop`, `substrate -c`, world-member follow-up semantics, storage rename, or macOS/Lima work. +5. Any implementation requires trace producer schema expansion without a concrete missing-field regression proving the need. +6. Any worker edits another lane’s owned files or any parent-owned run-state. +7. `L3` starts before `p2` freezes merged code truth. +8. The parent would need to invent hybrid warning, suppression, or naming semantics during integration. +9. The validation wall cannot prove the permissive-vs-strict split, `source_kind`, `warnings`, sorted/dedup warnings, participant-aware suppression, and strict doctor/toolbox non-regression. +10. The gap matrix or README would need to overclaim beyond what the integrated tests and final validation wall actually prove. + +## Fresh Worktrees And Branches + +Fresh worktree root: + +- `/Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-18` + +Worker worktrees: + +- `/Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-18/status-store-enumeration-and-naming-cleanup` +- `/Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-18/status-rendering-fallback-and-nested-correlation` +- `/Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-18/tests-docs-gap-closeout` + +Worker branches: + +- `codex/feat-session-centric-state-store-m18-status-store-enumeration-and-naming-cleanup` +- `codex/feat-session-centric-state-store-m18-status-rendering-fallback-and-nested-correlation` +- `codex/feat-session-centric-state-store-m18-tests-docs-gap-closeout` + +Exact setup commands: + +```bash +mkdir -p /Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-18 +git worktree add /Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-18/status-store-enumeration-and-naming-cleanup -b codex/feat-session-centric-state-store-m18-status-store-enumeration-and-naming-cleanup feat/session-centric-state-store +git worktree add /Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-18/status-rendering-fallback-and-nested-correlation -b codex/feat-session-centric-state-store-m18-status-rendering-fallback-and-nested-correlation feat/session-centric-state-store +git worktree add /Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-18/tests-docs-gap-closeout -b codex/feat-session-centric-state-store-m18-tests-docs-gap-closeout feat/session-centric-state-store +``` + +### Parent integration surface + +The parent integrates on the authoritative checkout already on `feat/session-centric-state-store`. + +No separate parent integration worktree is introduced because: + +1. this run has one serialized integrator +2. `.runs/plan-18/*` is parent-owned state and should stay co-located with the authoritative branch context +3. the only true merge activity is parent-owned and serialized + +## Phase Graph And Concurrency + +Concurrency rules: + +1. Worker cap is `2` until `g2` completes. +2. `p1` must finish before any worker starts. +3. `g1` must accept before any worker starts. +4. The only real parallel window is: + - `task/m18-l1-status-store-enumeration-and-naming-cleanup` + - `task/m18-l2-status-rendering-fallback-and-nested-correlation` +5. `g2` classifies both code-lane outputs before integration. +6. `p2` integrates accepted code lanes in this order: + - Lane A / `L1` first + - Lane B / `L2` second +7. `g3` starts only after `p2` is green. +8. `L3` runs alone on the exact post-`p2` tree. +9. `g4` and `p3` are parent-only. +10. No third concurrent implementation worker is honest because tests/docs must describe the integrated code truth, not lane-local guesses. + +### Why Lane A integrates before Lane B + +1. Lane A owns the canonical seam `list_status_sessions_for_agent(...)` that Lane B must consume. +2. Lane A also owns the Linux-first naming/comment cleanup in the runtime files that define the public-vs-internal naming split B must preserve. +3. Integrating Lane B first would force either a duplicate enumeration path inside `agents_cmd.rs` or parent-invented temporary semantics. Both violate the frozen contract. +4. If the combined state reveals semantic drift, Lane B is the correct quarantine target because it is the downstream consumer of the frozen seam, warning split, and identity-key contract. + +## PLAN-18 Step Mapping + +| Orchestration task | PLAN-18 step alignment | +| --- | --- | +| `task/m18-p1-parent-contract-freeze-and-run-init` | Step 1: freeze the degraded-status, naming, warning, and identity contracts | +| `task/m18-l1-status-store-enumeration-and-naming-cleanup` | Step 2 plus the runtime-file naming/comment portion of Step 1 | +| `task/m18-l2-status-rendering-fallback-and-nested-correlation` | Step 3, Step 4, and Step 5 | +| `task/m18-p2-parent-code-lane-integration` | enforce the end-to-end contract across merged Lane A plus Lane B | +| `task/m18-l3-tests-docs-gap-closeout` | Step 6 and Step 7 | +| `task/m18-p3-parent-validation-wall-and-closeout` | recommended verification commands plus Definition of Done enforcement | + +## Parent-Owned Run-State Surface + +Canonical parent-owned state under `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/plan-18/`: + +- `run-state.json` +- `tasks.json` +- `session-log.md` +- `contract-freeze.json` +- `lane-ownership.json` +- `merge-order.json` +- `validation-wall.md` +- `blocked.json` on failure only +- `quarantine/` +- `sentinels/` + +`contract-freeze.json` is the single source of truth for: + +- `plan_id: "PLAN-18"` +- `plan_source: "llm-last-mile/PLAN-18.md"` +- `orchestration_plan_source: "llm-last-mile/ORCH_PLAN-18.md"` +- `branch: "feat/session-centric-state-store"` +- `status_surface_mode: "permissive_rendering_only"` +- `strict_surfaces: ["agent doctor", "agent toolbox status", "agent toolbox env", "future agent start|resume|fork|stop"]` +- `status_session_source_kind_field: "StatusSessionJson.source_kind"` +- `status_report_warnings_field: "StatusReportJson.warnings"` +- `warning_ordering: "sorted_deduplicated_human_readable"` +- `record_warning_owner: "AgentRuntimeSessionRecord.warnings"` +- `set_warning_owner: "build_status_report"` +- `status_identity_key: "orchestration_session_id + agent_id + execution_scope + participant_id?"` +- `status_list_seam: "AgentRuntimeStateStore::list_status_sessions_for_agent"` +- `trace_schema_expansion_allowed: false` +- `public_control_surfaces_allowed: false` +- `substrate_c_redesign_allowed: false` +- `world_follow_up_semantics_reopened: false` +- `doctor_toolbox_relaxation_allowed: false` +- `storage_flag_day_rename_allowed: false` +- `macos_parity_allowed: false` +- `validation_commands` +- `manual_spot_checks` +- `stop_conditions` + +`lane-ownership.json` is the single source of truth for: + +- task ID +- owner +- worktree path +- branch +- allowed files +- forbidden files +- command gates +- retry budget +- merge order +- sentinel filename + +`merge-order.json` is a frozen artifact, not a narrative note. It records: + +- `plan_id` +- `integration_order: ["task/m18-l1-status-store-enumeration-and-naming-cleanup", "task/m18-l2-status-rendering-fallback-and-nested-correlation", "task/m18-l3-tests-docs-gap-closeout"]` +- `lane_b_acceptance_basis: "accepted_lane_a_tree_only"` +- `lane_b_must_consume_seam: "AgentRuntimeStateStore::list_status_sessions_for_agent"` +- `replay_required_before_acceptance: true` +- `quarantine_on_branch_local_assumption: true` + +`run-state.json` is the single source of truth for: + +- `current_phase` +- `active_task_ids` +- `worker_cap` +- `contract_freeze_status` +- `lane_status` +- `accepted_outputs` +- `rejected_outputs` +- `quarantined_outputs` +- `blocked_outputs` +- `retry_budget_by_lane` +- `attempt_counts` +- `integration_order` +- `gate_status` +- `validation_wall_status` +- `termination_reason` +- `terminal_state` + +### `tasks.json` required row fields + +`tasks.json` is both the launch queue and the execution ledger for the entire run. + +Each task row must include at least: + +- `task_id` +- `title` +- `owner` +- `status` +- `phase_class` +- `depends_on` +- `worktree_path` +- `branch` +- `allowed_files` +- `forbidden_files` +- `command_gates` +- `acceptance_checks` +- `sentinel_path` +- `artifact_dir` + +Recommended status vocabulary: + +- `pending` +- `ready` +- `running` +- `accepted` +- `rejected` +- `quarantined` +- `blocked` +- `completed` + +Recommended phase classes: + +- `parent_freeze` +- `gate` +- `worker_lane` +- `parent_integration` +- `validation` +- `closeout` + +Required sentinels: + +- `.runs/plan-18/sentinels/task-m18-p1-parent-contract-freeze-and-run-init.ok` +- `.runs/plan-18/sentinels/task-m18-g1-implementation-lane-launch-gate.ok` +- `.runs/plan-18/sentinels/task-m18-l1-status-store-enumeration-and-naming-cleanup.ok` +- `.runs/plan-18/sentinels/task-m18-l2-status-rendering-fallback-and-nested-correlation.ok` +- `.runs/plan-18/sentinels/task-m18-g2-code-lane-integration-gate.ok` +- `.runs/plan-18/sentinels/task-m18-p2-parent-code-lane-integration.ok` +- `.runs/plan-18/sentinels/task-m18-g3-closeout-launch-gate.ok` +- `.runs/plan-18/sentinels/task-m18-l3-tests-docs-gap-closeout.ok` +- `.runs/plan-18/sentinels/task-m18-g4-validation-wall-gate.ok` +- `.runs/plan-18/sentinels/task-m18-p3-parent-validation-wall-and-closeout.ok` + +### Per-task artifact directories + +Required task directories: + +- `.runs/task-m18-p1-parent-contract-freeze-and-run-init/` +- `.runs/task-m18-g1-implementation-lane-launch-gate/` +- `.runs/task-m18-l1-status-store-enumeration-and-naming-cleanup/` +- `.runs/task-m18-l2-status-rendering-fallback-and-nested-correlation/` +- `.runs/task-m18-g2-code-lane-integration-gate/` +- `.runs/task-m18-p2-parent-code-lane-integration/` +- `.runs/task-m18-g3-closeout-launch-gate/` +- `.runs/task-m18-l3-tests-docs-gap-closeout/` +- `.runs/task-m18-g4-validation-wall-gate/` +- `.runs/task-m18-p3-parent-validation-wall-and-closeout/` + +Each task directory must contain at least: + +- `task.json` +- `commands.txt` +- `summary.md` + +Each gate task directory must also contain: + +- `gate-checklist.md` +- `gate-result.json` + +Each worker task directory must also contain: + +- `worker-report.md` +- `worker-output.patch` +- `evidence-manifest.json` + +Each task directory may also contain: + +- `artifacts/` +- `rejected.json` on rejection only +- `blocked.json` on blocked task return only +- `quarantine/` when the parent quarantines that task’s output + +### Gate artifact semantics + +Every gate task must produce `gate-result.json` with at least: + +- `task_id` +- `checked_prerequisites` +- `artifacts_inspected` +- `classification` +- `timestamp` +- `approver: "parent"` + +`classification` must be one of: + +- `pass` +- `fail` + +A gate is green only when: + +1. all prerequisite sentinels exist +2. all required artifacts for the previous phase exist +3. ownership compliance is verified +4. pass/fail is recorded explicitly in `gate-result.json` + +## Kickoff Initialization Order + +The parent initializes the run in this exact order before any worker prompt is written: + +1. Create `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/plan-18/`, `.runs/plan-18/sentinels/`, `.runs/plan-18/quarantine/`, and every `.runs/task-m18-*/` directory. +2. Inside each `.runs/task-m18-*/` directory, create: + - `task.json` + - `commands.txt` + - `summary.md` + - `artifacts/` +3. For each gate task, also create: + - `gate-checklist.md` + - `gate-result.json` +4. For each worker task, also create placeholders for: + - `worker-report.md` + - `worker-output.patch` + - `evidence-manifest.json` +5. Write `tasks.json` as the canonical launch queue and execution ledger for the whole run. +6. Write `run-state.json` with `current_phase: "kickoff"`, `worker_cap: 2`, every task in `pending`, and empty accepted/rejected/quarantined output arrays. +7. Write `contract-freeze.json`, `lane-ownership.json`, `merge-order.json`, and `validation-wall.md`. +8. Review the frozen hotspots: + - [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) + - [crates/shell/src/execution/agent_runtime/orchestration_session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/orchestration_session.rs) + - [crates/shell/src/execution/agent_runtime/session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs) + - [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs) +9. If the exact seam `list_status_sessions_for_agent(...)` does not already exist, the parent may take one narrow shared-seam scaffold in `state_store.rs` before worker launch so Lane B can compile against the frozen signature. That scaffold must be signature-first, must not weaken any strict selector behavior, and must be recorded in `contract-freeze.json`. If the parent uses this scaffold, all worker branches must be reseeded from that exact post-scaffold commit. +10. Seed worker worktrees only after the above artifacts and any allowed scaffold are in place. +11. Write `session-log.md` with kickoff timestamp, authoritative branch, worktree roots, worker cap, and the explicit statement that the only honest parallel window is Lane A plus Lane B. +12. Mark `task/m18-p1-parent-contract-freeze-and-run-init` complete only after the frozen contract and seeded worker basis are identical. + +## Task Ledger + +| Task ID | Owner | Depends on | Worktree / branch | Deliverable | +| --- | --- | --- | --- | --- | +| `task/m18-p1-parent-contract-freeze-and-run-init` | parent | — | authoritative checkout | frozen run artifacts and seeded worker basis | +| `task/m18-g1-implementation-lane-launch-gate` | parent | `p1` | authoritative checkout | launch approval for Lane A and Lane B | +| `task/m18-l1-status-store-enumeration-and-naming-cleanup` | worker | `g1` | `status-store-enumeration-and-naming-cleanup` / `codex/feat-session-centric-state-store-m18-status-store-enumeration-and-naming-cleanup` | canonical status-store seam and naming/comment cleanup | +| `task/m18-l2-status-rendering-fallback-and-nested-correlation` | worker | `g1` | `status-rendering-fallback-and-nested-correlation` / `codex/feat-session-centric-state-store-m18-status-rendering-fallback-and-nested-correlation` | status rendering refactor, warnings, participant-aware fallback, nested correlation | +| `task/m18-g2-code-lane-integration-gate` | parent | `l1`, `l2` | authoritative checkout | acceptance/quarantine decision for both code lanes | +| `task/m18-p2-parent-code-lane-integration` | parent | `g2` | authoritative checkout | merged Lane A then Lane B code truth | +| `task/m18-g3-closeout-launch-gate` | parent | `p2` | authoritative checkout | launch approval for tests/docs closeout | +| `task/m18-l3-tests-docs-gap-closeout` | worker | `g3` | `tests-docs-gap-closeout` / `codex/feat-session-centric-state-store-m18-tests-docs-gap-closeout` | regressions and repo-truth docs | +| `task/m18-g4-validation-wall-gate` | parent | `l3` | authoritative checkout | permission to run final command wall | +| `task/m18-p3-parent-validation-wall-and-closeout` | parent | `g4` | authoritative checkout | final validation, closeout, terminal state | + +## Gate Tasks And Parent-Only Tasks + +Gate tasks are parent-only classification or launch-control phases. They do not mutate production files: + +- `task/m18-g1-implementation-lane-launch-gate` +- `task/m18-g2-code-lane-integration-gate` +- `task/m18-g3-closeout-launch-gate` +- `task/m18-g4-validation-wall-gate` + +Parent-only tasks are the only phases allowed to freeze contracts, integrate, approve, quarantine, or terminate the run: + +- `task/m18-p1-parent-contract-freeze-and-run-init` +- `task/m18-p2-parent-code-lane-integration` +- `task/m18-p3-parent-validation-wall-and-closeout` + +## Lane Ownership By File Set + +| Lane | Allowed files | Forbidden escalation surfaces | +| --- | --- | --- | +| Lane A / `L1` | [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs), [crates/shell/src/execution/agent_runtime/orchestration_session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/orchestration_session.rs), [crates/shell/src/execution/agent_runtime/session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs) | [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs), [crates/common/src/agent_events.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/common/src/agent_events.rs), public control surfaces, docs, `agent_successor_contract_ahcsitc0.rs`, `.runs/**` | +| Lane B / `L2` | [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs) | every runtime storage file, every docs file, [crates/common/src/agent_events.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/common/src/agent_events.rs), toolbox/doctor contract broadening, `.runs/**` | +| Lane C / `L3` | [crates/shell/tests/agent_successor_contract_ahcsitc0.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_successor_contract_ahcsitc0.rs), [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md), [llm-last-mile/README.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/README.md) if needed | every production Rust file, `.runs/**`, any public control-surface doc overclaim | + +Rule for inline unit tests: + +- If Lane A needs unit-test adjustments inside `state_store.rs` or `session.rs` to keep the naming or seam contract local, those edits remain Lane A-owned and must stay minimal. +- Lane C does not reopen runtime production files just to move proofs around. + +## Merge-Order Artifact + +`merge-order.json` is frozen during `p1` and governs integration behavior. + +It must state: + +- Lane A integrates first +- Lane B is evaluated only against the accepted Lane A seam +- Lane B is not accepted against its own branch-local assumption set +- Lane C starts only after final parent-integrated code truth exists + +Operational rule: + +- If Lane B’s patch merges mechanically but assumes a different seam signature, different warning ownership split, or different identity-key contract than the accepted Lane A tree, the parent quarantines Lane B rather than editing around the mismatch. + +## Worker Interfaces + +### Worker prompt contract + +Every worker prompt sent by the parent must include: + +1. task ID +2. attempt number +3. worktree path +4. branch name +5. owned files +6. forbidden files +7. exact frozen contract clauses relevant to that lane +8. command gates +9. retry budget +10. required return artifacts +11. sentinel name + +### Worker return contract + +Every worker must return exactly: + +1. changed files list +2. commands run, each with exit code +3. explicit attempt classification: `clean`, `retryable`, or `blocked` +4. unresolved assumptions or blockers +5. `worker-output.patch` +6. `worker-report.md` +7. `evidence-manifest.json` + +The parent may accept a worker return only if it can verify: + +1. the patch stays within owned files +2. command gates actually ran +3. exit codes are present +4. the attempt classification matches the evidence +5. no frozen contract clause was reopened + +## Parent Phase Details + +### `task/m18-p1-parent-contract-freeze-and-run-init` + +Owner: + +- parent only + +Scope: + +1. Initialize `.runs/plan-18/*` and `.runs/task-m18-*/**`. +2. Freeze the permissive-vs-strict surface split, warning ownership split, exact `source_kind` and `warnings` fields, `StatusIdentityKey`, and the exact `list_status_sessions_for_agent(...)` seam into `contract-freeze.json`. +3. Freeze worker file ownership, merge order, retry budget, and stop conditions. +4. Freeze the exact validation-wall commands and manual spot checks. +5. If needed, take the one narrow shared-seam scaffold in `state_store.rs` before worker seeding and record the scaffold commit. +6. Seed all worker worktrees from the exact same post-`p1` tree. + +Command gates: + +```bash +cargo test -p shell --lib -- --nocapture +cargo test -p shell --test agent_successor_contract_ahcsitc0 --no-run +``` + +Acceptance: + +1. `contract-freeze.json`, `lane-ownership.json`, `merge-order.json`, `run-state.json`, and `tasks.json` exist. +2. The frozen contract explicitly records that this is a status/read-side hardening run, not a public control-surface run. +3. Any allowed shared-seam scaffold is recorded and reseeded before worker launch. +4. The parent writes `.runs/plan-18/sentinels/task-m18-p1-parent-contract-freeze-and-run-init.ok`. + +### `task/m18-g1-implementation-lane-launch-gate` + +Owner: + +- parent only + +Checks: + +1. `p1` is accepted. +2. Both implementation worktrees were seeded from the exact same post-`p1` tree. +3. Lane A and Lane B prompts name only owned files, forbidden surfaces, command gates, sentinel, and retry budget. +4. Lane B’s prompt explicitly forbids inventing a duplicate enumeration path inside `agents_cmd.rs`. +5. The prompt explicitly repeats the frozen prohibitions on public control-surface work, `substrate -c`, world-member semantics, trace schema expansion, storage rename, and macOS parity. + +Acceptance: + +1. No worker starts before this gate is green. +2. The parent writes `.runs/plan-18/sentinels/task-m18-g1-implementation-lane-launch-gate.ok`. + +### `task/m18-l1-status-store-enumeration-and-naming-cleanup` + +Owner: + +- single worker on `codex/feat-session-centric-state-store-m18-status-store-enumeration-and-naming-cleanup` + +Owned files: + +- [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) +- [crates/shell/src/execution/agent_runtime/orchestration_session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/orchestration_session.rs) +- [crates/shell/src/execution/agent_runtime/session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs) + +Scope: + +1. Land `list_status_sessions_for_agent(...)` as the canonical permissive status enumeration seam. +2. Keep `resolve_single_live_session_for_agent(...)` strict and unchanged in meaning. +3. Route the new seam through `build_session_record(...)`; do not create a second record-construction path. +4. Preserve `warnings`, `complete`, and `has_authoritative_parent`. +5. Add only the naming/comment cleanup needed to make `orchestration_session_id`, `participant_id`, and `internal.uaa_session_id` responsibilities explicit. +6. Preserve compatibility alias reads. + +Command gates: + +```bash +cargo test -p shell --lib -- --nocapture +cargo test -p shell --test agent_successor_contract_ahcsitc0 --no-run +``` + +Acceptance: + +1. The lane touches only its owned files. +2. The new seam exists with the exact frozen signature. +3. The seam enumerates incomplete-but-readable status records without authorizing control. +4. Strict selector behavior for doctor/toolbox remains unchanged. +5. Naming/comment cleanup does not perform a storage rename. +6. No trace schema, world-member, `substrate -c`, or public control-surface drift is introduced. +7. The worker writes `.runs/plan-18/sentinels/task-m18-l1-status-store-enumeration-and-naming-cleanup.ok`. + +### `task/m18-l2-status-rendering-fallback-and-nested-correlation` + +Owner: + +- single worker on `codex/feat-session-centric-state-store-m18-status-rendering-fallback-and-nested-correlation` + +Owned files: + +- [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs) + +Scope: + +1. Remove the unconditional strict preflight from `build_status_report(...)`. +2. Consume the frozen `list_status_sessions_for_agent(...)` seam rather than re-filtering raw session storage locally. +3. Add `StatusSessionJson.source_kind`. +4. Add `StatusReportJson.warnings`. +5. Populate warnings as the sorted, deduplicated union of record-local warnings and cross-record ambiguity warnings. +6. Replace coarse fallback keys with `StatusIdentityKey`. +7. Preserve coarse grouping only where `participant_id` is truly absent. +8. Make nested parent correlation participant-aware without weakening malformed-tuple fail-closed behavior. + +Command gates: + +```bash +cargo fmt --all -- --check +cargo test -p shell --lib -- --nocapture +cargo test -p shell --test agent_successor_contract_ahcsitc0 --no-run +``` + +Acceptance: + +1. The lane touches only its owned file. +2. `source_kind` and `warnings` land exactly as frozen. +3. Lane B does not duplicate status enumeration logic. +4. Same-agent sibling participants stay distinct when lineage exists. +5. Live/tombstone suppression remains more specific, not looser. +6. Doctor/toolbox behavior is not relaxed as collateral damage. +7. The worker writes `.runs/plan-18/sentinels/task-m18-l2-status-rendering-fallback-and-nested-correlation.ok`. + +### `task/m18-g2-code-lane-integration-gate` + +Owner: + +- parent only + +Checks: + +1. `L1` and `L2` both returned a patch, report, command transcript, and evidence manifest. +2. Every touched file is inside the lane’s ownership boundary. +3. Lane A landed the exact seam and did not weaken strict selector semantics. +4. Lane B landed the exact status fields and key contract and did not introduce a duplicate enumeration path. +5. Lane B is not marked `accepted` until the parent proves it replays cleanly against the accepted Lane A seam on a parent-owned A-applied tree. + +Acceptance: + +1. Accepted, rejected, or quarantined status for each code lane is recorded explicitly in `run-state.json`. +2. The parent writes `.runs/plan-18/sentinels/task-m18-g2-code-lane-integration-gate.ok` only after both code lanes are classified. + +### `task/m18-p2-parent-code-lane-integration` + +Owner: + +- parent only + +Scope: + +1. Integrate accepted Lane A output first. +2. Re-run Lane A command gates on the authoritative checkout. +3. Replay Lane B output onto the accepted-A tree. If Lane B assumed a different seam signature, invented local filtering, or otherwise drifted from the frozen contract, quarantine Lane B instead of hand-editing around it. +4. Integrate accepted Lane B output second. +5. Re-run combined code-lane gates on the authoritative checkout. +6. Freeze the merged code truth before any tests/docs lane starts. + +Command gates: + +```bash +cargo test -p shell --lib -- --nocapture +cargo test -p shell --test agent_successor_contract_ahcsitc0 --no-run +``` + +Acceptance: + +1. The parent remains the sole integrator. +2. The authoritative tree now contains the canonical seam, the status-report refactor, and the participant-aware key contract. +3. No hybrid truth was invented during integration. +4. The parent writes `.runs/plan-18/sentinels/task-m18-p2-parent-code-lane-integration.ok`. + +### `task/m18-g3-closeout-launch-gate` + +Owner: + +- parent only + +Checks: + +1. `p2` is green. +2. The closeout worktree is reseeded or rebased to the exact post-`p2` tree. +3. The worker prompt names only the allowed test and doc files. +4. The worker prompt explicitly forbids reopening production runtime files. + +Acceptance: + +1. No tests/docs worker starts before this gate is green. +2. The parent writes `.runs/plan-18/sentinels/task-m18-g3-closeout-launch-gate.ok`. + +### `task/m18-l3-tests-docs-gap-closeout` + +Owner: + +- single worker on `codex/feat-session-centric-state-store-m18-tests-docs-gap-closeout` + +Owned files: + +- [crates/shell/tests/agent_successor_contract_ahcsitc0.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_successor_contract_ahcsitc0.rs) +- [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md) +- [llm-last-mile/README.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/README.md) if needed + +Scope: + +1. Add degraded-status regressions for missing `active_session_handle_id`. +2. Add degraded-status regressions for multiple active parent candidates. +3. Add trace-only sibling-participant visibility regressions. +4. Add sibling-specific suppression regressions. +5. Add parent-participant-aware nested correlation regressions. +6. Preserve strict doctor/toolbox fail-closed regressions. +7. Update repo-truth docs only after the integrated code and regressions agree on the same shipped truth. + +Command gates: + +```bash +cargo test -p shell --test agent_successor_contract_ahcsitc0 -- --nocapture +rg -n "Status ambiguity handling|Trace-only participant-aware fallback|orchestration_session_id|participant_id" AGENT_ORCHESTRATION_GAP_MATRIX.md llm-last-mile/README.md +``` + +Acceptance: + +1. The lane touches only its owned files. +2. Tests prove degraded status succeeds with warnings while doctor/toolbox remain strict. +3. Docs say `orchestration_session_id` and `participant_id` plainly and do not imply public control-surface completion. +4. The gap matrix does not mark anything closed beyond what the integrated tests and final validation wall prove. +5. The worker writes `.runs/plan-18/sentinels/task-m18-l3-tests-docs-gap-closeout.ok`. + +## Quarantine, Retry, And Blocked-Run Posture + +1. Each worker lane has retry budget `1`. +2. Retry is allowed only for lane-local defects inside owned files. +3. Non-retryable violations include: + - strict-surface relaxation + - duplicate enumeration logic in `agents_cmd.rs` + - trace producer schema expansion without proof + - public control-surface broadening + - `substrate -c` redesign + - world-member semantic drift + - storage rename + - macOS/Lima broadening +4. If Lane A changes the seam signature or weakens `resolve_single_live_session_for_agent(...)`, quarantine it immediately. +5. If Lane B can only pass by broadening ownership into runtime storage files or by recreating status enumeration logic locally, quarantine it immediately. +6. If Lane C can only make docs pass by overclaiming landed behavior, reject Lane C and stop the run instead of publishing optimistic docs. +7. The parent never hand-merges a hybrid contract out of conflicting worker guesses. When semantics drift, quarantine the output and either redrive within the same lane or stop. + +### Quarantine artifact handling + +When a lane is quarantined, the parent must preserve the returned materials in both places: + +1. keep the original files in the lane task directory: + - `.runs/task-/worker-output.patch` + - `.runs/task-/worker-report.md` + - `.runs/task-/evidence-manifest.json` +2. copy them into the plan-level quarantine surface: + - `.runs/plan-18/quarantine//worker-output.patch` + - `.runs/plan-18/quarantine//worker-report.md` + - `.runs/plan-18/quarantine//evidence-manifest.json` + - `.runs/plan-18/quarantine//commands.txt` + - `.runs/plan-18/quarantine//summary.md` + - `.runs/plan-18/quarantine//quarantine-reason.json` + +`quarantine-reason.json` must record: + +- `task_id` +- `attempt` +- `classification` +- `frozen_contract_clause_violated` +- `ownership_violation` if any +- `merge_replay_failure` if any +- `retry_available` +- `timestamp` +- `approver: "parent"` + +Blocked termination minimum contents: + +1. task or gate where execution stopped +2. classification: `rejected`, `blocked`, `quarantined`, or `merge_refused` +3. exact contract clause or ownership rule that stopped the run +4. whether retry remained available +5. artifact paths for patch, report, command output, and semantic-drift evidence +6. explicit statement that no blocked output was integrated + +## Context-Control Rules + +1. The parent keeps only a bounded live context: + - [PLAN-18.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-18.md) + - this orchestration plan + - `tasks.json` + - `run-state.json` + - `contract-freeze.json` + - `merge-order.json` + - latest integration diff summary +2. Worker prompts contain only: + - owned file set + - exact frozen contract excerpts relevant to that lane + - required commands + - forbidden touch surfaces + - the recorded merge-order rule when relevant +3. Workers return summaries and artifacts only. They do not become independent approval or truth surfaces. +4. Workers do not write `.runs/plan-18/*`. +5. The parent reviews summaries plus narrow diffs only. It does not ingest full worker transcripts into the main run context. +6. Each worker is closed after accept/reject/quarantine to keep context bounded. + +## Tests And Acceptance + +### A. Frozen-contract acceptance + +- The permissive-vs-strict split is frozen before worker launch. +- `source_kind`, `warnings`, warning ordering, warning ownership split, `StatusIdentityKey`, and the exact seam signature are all recorded in `contract-freeze.json`. +- No worker prompt reopens those decisions. + +### B. Lane A acceptance + +- `list_status_sessions_for_agent(...)` exists with the exact frozen signature. +- The seam routes through `build_session_record(...)`. +- The seam preserves warnings and completeness truth without authorizing control. +- Strict selector behavior remains fail closed for doctor/toolbox. + +### C. Lane B acceptance + +- `StatusSessionJson.source_kind` is present and exact. +- `StatusReportJson.warnings` is present and exact. +- Warning output is sorted and deduplicated. +- Trace-only fallback, live/tombstone suppression, and nested parent correlation all use the same `StatusIdentityKey` family. +- Lane B does not invent a second status-record construction path. + +### D. Integration acceptance + +- Lane A integrates first and remains green. +- Lane B is replayed on top of the accepted Lane A seam before acceptance. +- The merged tree preserves strict doctor/toolbox behavior while making `agent status` permissive. +- No parent-only hybrid edits are required to explain the merged behavior. + +### E. Lane C acceptance + +- `agent_successor_contract_ahcsitc0.rs` proves degraded status with warnings, sibling visibility, sibling-specific suppression, and nested participant-aware correlation. +- Existing strict doctor/toolbox fail-closed regressions remain green. +- Docs reflect the landed Linux-first status and naming truth and leave public control-surface work deferred. + +### F. Run-control acceptance + +- Every completed task has its sentinel. +- Every gate has a `gate-result.json`. +- `tasks.json` and `run-state.json` match the actual accepted/rejected/quarantined outcomes. +- No quarantined or blocked output was partially integrated. + +## Validation Wall + +Parent-owned validation commands, executed only after `L3` is integrated: + +```bash +cargo fmt --all -- --check +cargo clippy --workspace --all-targets -- -D warnings +cargo test -p shell --lib -- --nocapture +cargo test -p shell --test agent_successor_contract_ahcsitc0 -- --nocapture +``` + +Manual spot checks after the command wall is green: + +```bash +substrate agent status --json +substrate agent doctor --json +substrate agent toolbox status --json +substrate agent toolbox env --json +``` + +Validation-wall requirements: + +1. formatting and clippy stay green +2. shell library tests stay green +3. the targeted successor/status integration suite proves degraded status success with warnings and strict doctor/toolbox non-regression +4. manual spot checks confirm the same operator contract on real CLI output: + - status reports `source_kind` + - status reports `warnings` + - doctor/toolbox remain strict +5. docs do not overclaim beyond what the green runtime and tests prove + +### `task/m18-g4-validation-wall-gate` + +Owner: + +- parent only + +Checks: + +1. `L3` returned and is classified before final validation. +2. `L3` is `accepted`. +3. No quarantined or blocked output remains unresolved. +4. `validation-wall.md` names the exact final command order and manual spot checks. +5. The parent can enumerate every `PLAN-18` definition-of-done clause and the command or artifact that proves it. + +Acceptance: + +1. The parent writes `.runs/plan-18/sentinels/task-m18-g4-validation-wall-gate.ok`. +2. The validation wall is permitted to run exactly once on the final merged tree. + +### `task/m18-p3-parent-validation-wall-and-closeout` + +Owner: + +- parent only + +Scope: + +1. Integrate only accepted `L3` output. +2. Run the full validation wall in exact order. +3. Record final command results and artifact paths in `.runs/task-m18-p3-parent-validation-wall-and-closeout/artifacts/`. +4. Confirm the gap matrix and optional README index match the validated runtime truth. +5. Mark the run complete only if the validation wall proves the frozen contract, not merely compilation. + +Required final artifacts under `.runs/task-m18-p3-parent-validation-wall-and-closeout/artifacts/`: + +- `fmt.txt` +- `clippy.txt` +- `shell-lib-tests.txt` +- `agent-successor-contract.txt` +- `status-json-spot-check.txt` +- `doctor-json-spot-check.txt` +- `toolbox-status-json-spot-check.txt` +- `toolbox-env-json-spot-check.txt` +- `contract-audit.md` +- `closeout.md` + +Acceptance: + +1. All validation commands succeed on the authoritative checkout. +2. Manual spot checks are captured with expected operator-visible outcomes. +3. `run-state.json` records a successful terminal state. +4. The parent writes `.runs/plan-18/sentinels/task-m18-p3-parent-validation-wall-and-closeout.ok`. + +## Completion Criteria Tied To PLAN-18 Definition Of Done + +The run is complete only if all of these are true: + +1. `agent status` no longer aborts on strict parent-selection ambiguity or stale linkage that can be rendered truthfully with warnings. +2. doctor and toolbox surfaces remain strict and fail closed. +3. status rows explicitly identify whether they come from `live_runtime` or `trace_fallback`. +4. same-agent sibling participants remain distinct on the status surface when lineage evidence exists. +5. live/tombstone suppression only suppresses matching participant-aware fallback rows. +6. nested parent correlation respects `parent_participant_id` when present and still rejects malformed tuples. +7. canonical naming in docs and status/report surfaces uses `orchestration_session_id` and `participant_id`. +8. legacy alias reads remain green. +9. no new public `start|resume|fork|stop` surface is introduced. +10. repo-truth docs reflect the landed behavior and nothing broader. + +## Final State + +Success requires all of: + +1. every required sentinel exists +2. no blocked artifact exists under `.runs/plan-18/` +3. accepted outputs were integrated in the prescribed order only +4. final validation commands pass +5. manual spot checks confirm the same operator contract the tests prove +6. `contract-freeze.json` and the final merged tree still agree on the frozen `PLAN-18` contract + +Blocked termination requires any of: + +1. hard-guard violation +2. non-retryable rejection +3. exhausted retry budget +4. merge refusal with no legal redrive path +5. failed validation wall +6. docs requiring overclaim to appear complete + +On blocked termination the parent must write: + +1. `.runs/plan-18/blocked.json` +2. terminal state and rationale in `run-state.json` +3. gate and failure summary in `session-log.md` +4. preserved evidence under `.runs/plan-18/quarantine/` and the relevant task artifact directory diff --git a/llm-last-mile/ORCH_PLAN-19.md b/llm-last-mile/ORCH_PLAN-19.md new file mode 100644 index 000000000..d5294b7d3 --- /dev/null +++ b/llm-last-mile/ORCH_PLAN-19.md @@ -0,0 +1,1003 @@ +# ORCH_PLAN-19: Execute PLAN-19 Through A Parent-Frozen Owner Plane, Exact Session Selectors, And Linux-First Public Control Productization + +Branch: `feat/session-centric-state-store` +Plan source: [PLAN-19.md](/home/spenser/__Active_code/substrate/llm-last-mile/PLAN-19.md) +Style reference: [ORCH_PLAN-18.md](/home/spenser/__Active_code/substrate/llm-last-mile/ORCH_PLAN-18.md) +Structure reference: historical external packet only; not present in this checkout +Packet index: [README.md](/home/spenser/__Active_code/substrate/llm-last-mile/README.md) +Gap matrix anchor: [AGENT_ORCHESTRATION_GAP_MATRIX.md](/home/spenser/__Active_code/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md) +Execution type: mid-run orchestration revision from accepted `p2` truth, Linux-first, public control-plane productization, parent-frozen contract, parent-only integration and approval +Worker model: GPT-5.4 workers with `reasoning_effort=high` +Max concurrent code workers before integration: `2` + +## Summary + +This document is the execution controller for `PLAN-19`, not a restatement of it. It is also the authoritative mid-run revision after the blocked first `L3` attempt proved that the original `p2 -> public CLI` seam was frozen too early. + +- Execute from the current branch `feat/session-centric-state-store`, because that is the authoritative integration checkout in this workspace. +- The live absolute roots for this run are `/home/spenser/__Active_code/substrate` and `/home/spenser/__Active_code/.worktrees/substrate-plan-19/`. Any earlier `/Users/spensermcconnell/...` packet literals are superseded by these live workspace paths. +- Keep the critical path local to the parent agent for contract freeze, owner-model freeze, worker launch gates, both integration windows, the validation wall, and final closeout. +- Preserve accepted Window A work as-is. The landed post-`p2` runtime files are: + - `crates/shell/src/execution/agent_runtime/control.rs` + - `crates/shell/src/execution/agent_runtime/mod.rs` + - `crates/shell/src/execution/agent_runtime/state_store.rs` + - `crates/shell/src/execution/agent_runtime/orchestration_session.rs` + - `crates/shell/src/execution/agent_runtime/session.rs` + - `crates/shell/src/repl/async_repl.rs` +- The old `task/m19-l3-public-cli-and-handler-wiring` is preserved as historical blocked evidence and superseded by this revision. It is not treated as an implementation failure in the accepted tree. +- Use dedicated worker worktrees under `/home/spenser/__Active_code/.worktrees/substrate-plan-19/`. +- Use GPT-5.4 with `reasoning_effort=high` for all workers and cap true code concurrency at `2`. +- Keep orchestration state in one local parent-owned source of truth: + - queue and task ledger: `/home/spenser/__Active_code/substrate/.runs/plan-19/tasks.json` + - run state: `/home/spenser/__Active_code/substrate/.runs/plan-19/run-state.json` + - session log: `/home/spenser/__Active_code/substrate/.runs/plan-19/session-log.md` + - sentinels: `/home/spenser/__Active_code/substrate/.runs/plan-19/sentinels/` + - historical blocked evidence: `/home/spenser/__Active_code/substrate/.runs/plan-19/history/` + +The revised run shape after accepted `p2` is frozen as: + +1. `task/m19-p1-parent-contract-freeze-owner-model-and-run-init` is parent-only and freezes the public verb contract, exact-selector contract, hidden owner-helper contract, private stop transport contract, strict-vs-permissive surface split, Linux-first posture, lane ownership, merge order, retry budget, stop conditions, and validation wall. +2. `task/m19-g1-implementation-window-a-launch-gate` is parent-only and is the only gate that may launch the first implementation window. +3. The only honest initial parallel window is exactly two lanes: + - `task/m19-l1-strict-resolver-and-naming-guardrails` + - `task/m19-l2-owner-plane-and-repl-integration` +4. `task/m19-g2-window-a-integration-gate` and `task/m19-p2-parent-window-a-integration` are parent-only. +5. The blocked first CLI attempt proved that `p2` truth still lacked the helper-launch seam. The parent therefore inserts a new Window B before any public verb wiring: + - `task/m19-g3-helper-launch-seam-gate` + - `task/m19-l3a-hidden-helper-and-launch-plan-handoff` + - `task/m19-g4-helper-launch-integration-gate` + - `task/m19-p3-helper-launch-integration` +6. Public CLI wiring is relaunched only after accepted helper-launch truth exists: + - `task/m19-g5-public-cli-launch-gate` + - `task/m19-l3b-public-cli-and-handler-wiring` + - `task/m19-g6-cli-integration-gate` + - `task/m19-p4-parent-cli-integration` +7. Tests/docs closeout is not parallel with code lanes. It starts only after `p4` lands merged code truth: + - `task/m19-g7-closeout-launch-gate` + - `task/m19-l4-tests-docs-gap-matrix-and-qa-closeout` +8. `task/m19-g8-validation-wall-gate` and `task/m19-p5-parent-validation-wall-and-closeout` are parent-only and finish the run. + +Canonical task IDs: + +- `task/m19-p1-parent-contract-freeze-owner-model-and-run-init` +- `task/m19-g1-implementation-window-a-launch-gate` +- `task/m19-l1-strict-resolver-and-naming-guardrails` +- `task/m19-l2-owner-plane-and-repl-integration` +- `task/m19-g2-window-a-integration-gate` +- `task/m19-p2-parent-window-a-integration` +- `task/m19-g3-helper-launch-seam-gate` +- `task/m19-l3a-hidden-helper-and-launch-plan-handoff` +- `task/m19-g4-helper-launch-integration-gate` +- `task/m19-p3-helper-launch-integration` +- `task/m19-g5-public-cli-launch-gate` +- `task/m19-l3b-public-cli-and-handler-wiring` +- `task/m19-g6-cli-integration-gate` +- `task/m19-p4-parent-cli-integration` +- `task/m19-g7-closeout-launch-gate` +- `task/m19-l4-tests-docs-gap-matrix-and-qa-closeout` +- `task/m19-g8-validation-wall-gate` +- `task/m19-p5-parent-validation-wall-and-closeout` + +## Hard Guards + +These are run-stopping invariants, not preferences: + +1. The authoritative integration checkout remains the current workspace checkout on `feat/session-centric-state-store`. +2. The parent agent is the only integrator, the only approval authority, and the only writer of run-state artifacts under `/home/spenser/__Active_code/substrate/.runs/plan-19/`. +3. Public verbs are frozen to exactly: + - `substrate agent start --backend [--json]` + - `substrate agent resume --session [--json]` + - `substrate agent fork --session [--json]` + - `substrate agent stop --session [--json]` +4. Public existing-session targeting accepts only `--session `. +5. Public root-session creation accepts only `--backend `. +6. No public command accepts `participant_id`, `session_handle_id`, `active_session_handle_id`, or `internal.uaa_session_id` as input. +7. `start`, `resume`, and `fork` are short-lived launch commands and must return only after authoritative readiness is visible in the state store. +8. `stop` must reach the live owner loop through the private owner transport and must not fake success through `session.json` mutation, toolbox mutation, or PID-only signaling. +9. The only new ownership surface allowed in this run is one per-session owner loop plus one private per-session stop transport. No general daemon, no shared multi-session broker, no global listener. +10. Root public session creation is host-orchestrator only in v1. World-scoped root start is an explicit rejection path. +11. World-sensitive reuse and stop posture is Linux-first. If exact posture cannot be proven on the current platform, return `unsupported_platform_or_posture`. +12. `status` remains the only permissive public surface in this family. `doctor`, `toolbox status`, `toolbox env`, `start`, `resume`, `fork`, and `stop` remain strict and fail closed. +13. `resume` stays in the same parent orchestration session. `fork` allocates a new parent orchestration session. +14. Public success and failure output is frozen before worker launch. Lane workers may not invent new fields or vague error names. +15. No worker may widen into prompt submission, `substrate -c` redesign, member-level public selectors, root world-session start, toolbox mutation tools, macOS/Lima parity, or Windows/WSL parity. +16. No worker may edit `.runs/**`. +17. If the frozen owner-helper and stop-transport contract still cannot be expressed after the helper-launch seam lane, the run stops again and writes `/home/spenser/__Active_code/substrate/.runs/plan-19/blocked.json`. + +## Fresh Worktrees And Branches + +Fresh worktree root: + +- `/home/spenser/__Active_code/.worktrees/substrate-plan-19` + +Worker worktrees: + +- `/home/spenser/__Active_code/.worktrees/substrate-plan-19/strict-resolver-and-naming-guardrails` +- `/home/spenser/__Active_code/.worktrees/substrate-plan-19/owner-plane-and-repl-integration` +- `/home/spenser/__Active_code/.worktrees/substrate-plan-19/helper-launch-seam` +- `/home/spenser/__Active_code/.worktrees/substrate-plan-19/public-cli-and-handler-wiring-relaunch` +- `/home/spenser/__Active_code/.worktrees/substrate-plan-19/tests-docs-gap-matrix-and-qa-closeout` + +Historical preserved blocked-attempt worktree: + +- `/home/spenser/__Active_code/.worktrees/substrate-plan-19/public-cli-and-handler-wiring` + +Worker branches: + +- `codex/feat-session-centric-state-store-m19-strict-resolver-and-naming-guardrails` +- `codex/feat-session-centric-state-store-m19-owner-plane-and-repl-integration` +- `codex/feat-session-centric-state-store-m19-helper-launch-seam` +- `codex/feat-session-centric-state-store-m19-public-cli-and-handler-wiring-relaunch` +- `codex/feat-session-centric-state-store-m19-tests-docs-gap-matrix-and-qa-closeout` + +Continuation setup commands after accepted `p2`: + +```bash +mkdir -p /home/spenser/__Active_code/.worktrees/substrate-plan-19 +git worktree add /home/spenser/__Active_code/.worktrees/substrate-plan-19/helper-launch-seam -b codex/feat-session-centric-state-store-m19-helper-launch-seam feat/session-centric-state-store +git worktree add /home/spenser/__Active_code/.worktrees/substrate-plan-19/public-cli-and-handler-wiring-relaunch -b codex/feat-session-centric-state-store-m19-public-cli-and-handler-wiring-relaunch feat/session-centric-state-store +``` + +Parent integration surface: + +- The parent integrates only on `/home/spenser/__Active_code/substrate` on `feat/session-centric-state-store`. +- No separate parent integration worktree is introduced. + +## Parent-Owned Run-State Surface + +Canonical parent-owned state under `/home/spenser/__Active_code/substrate/.runs/plan-19/`: + +- `run-state.json` +- `tasks.json` +- `session-log.md` +- `contract-freeze.json` +- `owner-model-freeze.json` +- `lane-ownership.json` +- `merge-order.json` +- `validation-wall.md` +- `blocked.json` on failure only +- `history/` for superseded blocked evidence only +- `quarantine/` +- `sentinels/` + +Canonical per-task artifact roots: + +- `/home/spenser/__Active_code/substrate/.runs/task-m19-p1-parent-contract-freeze-owner-model-and-run-init/` +- `/home/spenser/__Active_code/substrate/.runs/task-m19-g1-implementation-window-a-launch-gate/` +- `/home/spenser/__Active_code/substrate/.runs/task-m19-l1-strict-resolver-and-naming-guardrails/` +- `/home/spenser/__Active_code/substrate/.runs/task-m19-l2-owner-plane-and-repl-integration/` +- `/home/spenser/__Active_code/substrate/.runs/task-m19-g2-window-a-integration-gate/` +- `/home/spenser/__Active_code/substrate/.runs/task-m19-p2-parent-window-a-integration/` +- `/home/spenser/__Active_code/substrate/.runs/task-m19-g3-helper-launch-seam-gate/` +- `/home/spenser/__Active_code/substrate/.runs/task-m19-l3a-hidden-helper-and-launch-plan-handoff/` +- `/home/spenser/__Active_code/substrate/.runs/task-m19-g4-helper-launch-integration-gate/` +- `/home/spenser/__Active_code/substrate/.runs/task-m19-p3-helper-launch-integration/` +- `/home/spenser/__Active_code/substrate/.runs/task-m19-g5-public-cli-launch-gate/` +- `/home/spenser/__Active_code/substrate/.runs/task-m19-l3b-public-cli-and-handler-wiring/` +- `/home/spenser/__Active_code/substrate/.runs/task-m19-g6-cli-integration-gate/` +- `/home/spenser/__Active_code/substrate/.runs/task-m19-p4-parent-cli-integration/` +- `/home/spenser/__Active_code/substrate/.runs/task-m19-g7-closeout-launch-gate/` +- `/home/spenser/__Active_code/substrate/.runs/task-m19-l4-tests-docs-gap-matrix-and-qa-closeout/` +- `/home/spenser/__Active_code/substrate/.runs/task-m19-g8-validation-wall-gate/` +- `/home/spenser/__Active_code/substrate/.runs/task-m19-p5-parent-validation-wall-and-closeout/` +- historical blocked-attempt evidence remains under `/home/spenser/__Active_code/substrate/.runs/task-m19-l3-public-cli-and-handler-wiring/` + +Each task directory must contain at least: + +- `task.json` +- `commands.txt` +- `summary.md` + +Each gate task directory must also contain: + +- `gate-checklist.md` +- `gate-result.json` + +Each worker task directory must also contain: + +- `worker-report.md` +- `worker-output.patch` +- `evidence-manifest.json` + +Sentinel convention: + +- `/home/spenser/__Active_code/substrate/.runs/plan-19/sentinels/.ok` + +`contract-freeze.json` is the single source of truth for: + +- frozen public verb list +- success JSON fields +- failure taxonomy +- strict-vs-permissive surface split +- exact selector rules +- host-root-only start rule +- Linux-first world-sensitive rejection rule +- no-prompt-surface rule +- validation commands +- stop conditions + +`owner-model-freeze.json` is the single source of truth for: + +- hidden owner-helper modes: `start`, `resume`, `fork` +- required helper inputs +- resolved launch-plan handoff contract for `start|resume|fork` +- per-session private stop transport contract +- `{"version":1,"action":"stop"}` request shape +- accepted stop outcomes: `accepted`, `already_terminal`, `owner_unreachable`, `protocol_error` +- readiness wait requirements +- parent-session state machine +- helper-launch seam acceptance checklist +- REPL-owned and helper-owned owner-plane parity rule + +## Task Ledger + +| Task ID | Owner | Depends on | Worktree / branch | Deliverable | +| --- | --- | --- | --- | --- | +| `task/m19-p1-parent-contract-freeze-owner-model-and-run-init` | parent | — | authoritative checkout | frozen contract, owner-model freeze, run artifacts, seeded worker basis | +| `task/m19-g1-implementation-window-a-launch-gate` | parent | `p1` | authoritative checkout | launch approval for Window A | +| `task/m19-l1-strict-resolver-and-naming-guardrails` | worker | `g1` | `strict-resolver-and-naming-guardrails` / `codex/feat-session-centric-state-store-m19-strict-resolver-and-naming-guardrails` | exact public-control resolver and canonical naming guardrails | +| `task/m19-l2-owner-plane-and-repl-integration` | worker | `g1` | `owner-plane-and-repl-integration` / `codex/feat-session-centric-state-store-m19-owner-plane-and-repl-integration` | shared control module, owner-plane primitives, private stop transport, and REPL parity up to but not including the callable helper-launch seam | +| `task/m19-g2-window-a-integration-gate` | parent | `l1`, `l2` | authoritative checkout | acceptance or quarantine decision for Window A | +| `task/m19-p2-parent-window-a-integration` | parent | `g2` | authoritative checkout | merged resolver plus owner-plane truth | +| `task/m19-g3-helper-launch-seam-gate` | parent | `p2` | authoritative checkout | launch approval for the missing helper-launch seam | +| `task/m19-l3a-hidden-helper-and-launch-plan-handoff` | worker | `g3` | `helper-launch-seam` / `codex/feat-session-centric-state-store-m19-helper-launch-seam` | callable hidden helper entrypoint, typed resolved launch plan, authoritative readiness wait, REPL/helper stop-owner parity | +| `task/m19-g4-helper-launch-integration-gate` | parent | `l3a` | authoritative checkout | acceptance or quarantine decision for the helper-launch seam | +| `task/m19-p3-helper-launch-integration` | parent | `g4` | authoritative checkout | merged helper-launch truth that the public CLI can call honestly | +| `task/m19-g5-public-cli-launch-gate` | parent | `p3` | authoritative checkout | relaunch approval for the public CLI lane | +| `task/m19-l3b-public-cli-and-handler-wiring` | worker | `g5` | `public-cli-and-handler-wiring-relaunch` / `codex/feat-session-centric-state-store-m19-public-cli-and-handler-wiring-relaunch` | public CLI surface and strict command handling over the accepted helper-launch seam | +| `task/m19-g6-cli-integration-gate` | parent | `l3b` | authoritative checkout | acceptance or quarantine decision for the public CLI lane | +| `task/m19-p4-parent-cli-integration` | parent | `g6` | authoritative checkout | merged public command surface | +| `task/m19-g7-closeout-launch-gate` | parent | `p4` | authoritative checkout | launch approval for tests/docs closeout | +| `task/m19-l4-tests-docs-gap-matrix-and-qa-closeout` | worker | `g7` | `tests-docs-gap-matrix-and-qa-closeout` / `codex/feat-session-centric-state-store-m19-tests-docs-gap-matrix-and-qa-closeout` | public-control integration tests, doc truth, QA artifact | +| `task/m19-g8-validation-wall-gate` | parent | `l4` | authoritative checkout | permission to run final command wall | +| `task/m19-p5-parent-validation-wall-and-closeout` | parent | `g8` | authoritative checkout | final validation, closeout, terminal state | + +## Merge Order + +`merge-order.json` is frozen during `p1` and governs integration behavior. + +It must state: + +1. `L1` integrates before `L2` acceptance is finalized if `L2` consumed resolver seams. +2. `L2` is replayed on top of the accepted `L1` tree before acceptance. +3. `L3a` starts only after accepted `L1` and accepted `L2` are integrated. +4. `L3b` starts only after accepted `L3a` is integrated through `p3`. +5. `L4` starts only after accepted `L3b` is integrated through `p4`. +6. The parent never hand-merges a hybrid owner model out of conflicting worker assumptions. + +Operational rule: + +- If `L2` assumes a different exact resolver contract than the accepted `L1` tree, quarantine `L2`. +- If `L3a` tries to bypass a retained helper launch by inventing an in-process fake helper, raw fuzzy selectors, or readiness-free success, quarantine `L3a`. +- If `L3b` invents output fields, selector behavior, or failure semantics beyond the frozen contract, quarantine `L3b`. + +## Kickoff Initialization Order + +The parent initializes the run in this exact order: + +1. Create `/home/spenser/__Active_code/substrate/.runs/plan-19/`, `/home/spenser/__Active_code/substrate/.runs/plan-19/sentinels/`, `/home/spenser/__Active_code/substrate/.runs/plan-19/quarantine/`, and every `.runs/task-m19-*/` directory. +2. Inside each `.runs/task-m19-*/` directory, create `task.json`, `commands.txt`, `summary.md`, and `artifacts/`. +3. For each gate task, also create `gate-checklist.md` and `gate-result.json`. +4. For each worker task, also create placeholders for `worker-report.md`, `worker-output.patch`, and `evidence-manifest.json`. +5. Write `tasks.json` as the canonical launch queue and execution ledger for the whole run. +6. Write `run-state.json` with `current_phase: "kickoff"`, `worker_cap: 2`, every task in `pending`, and empty accepted, rejected, quarantined, and blocked arrays. +7. Write `contract-freeze.json`, `owner-model-freeze.json`, `lane-ownership.json`, `merge-order.json`, and `validation-wall.md`. +8. Review the frozen hotspots: + - [crates/shell/src/execution/cli.rs](/home/spenser/__Active_code/substrate/crates/shell/src/execution/cli.rs) + - [crates/shell/src/execution/agents_cmd.rs](/home/spenser/__Active_code/substrate/crates/shell/src/execution/agents_cmd.rs) + - [crates/shell/src/execution/agent_runtime/state_store.rs](/home/spenser/__Active_code/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) + - [crates/shell/src/execution/agent_runtime/orchestration_session.rs](/home/spenser/__Active_code/substrate/crates/shell/src/execution/agent_runtime/orchestration_session.rs) + - [crates/shell/src/execution/agent_runtime/session.rs](/home/spenser/__Active_code/substrate/crates/shell/src/execution/agent_runtime/session.rs) + - [crates/shell/src/execution/agent_runtime/mod.rs](/home/spenser/__Active_code/substrate/crates/shell/src/execution/agent_runtime/mod.rs) + - [crates/shell/src/repl/async_repl.rs](/home/spenser/__Active_code/substrate/crates/shell/src/repl/async_repl.rs) +9. If needed, take one narrow compile-first scaffold before worker launch in: + - `/home/spenser/__Active_code/substrate/crates/shell/src/execution/agent_runtime/control.rs` + - `/home/spenser/__Active_code/substrate/crates/shell/src/execution/agent_runtime/mod.rs` + - `/home/spenser/__Active_code/substrate/crates/shell/src/execution/agent_runtime/state_store.rs` +10. The allowed scaffold may freeze signatures, types, and internal entrypoints only. It must not weaken strict selector behavior, must not expose public help, and must be recorded in both freeze artifacts. +11. Seed worker worktrees only after the above artifacts and any allowed scaffold are in place. +12. Write `session-log.md` with kickoff timestamp, authoritative branch, worktree roots, worker cap, and the explicit statement that the only honest initial parallel window is `L1` plus `L2`. + +## Lane Ownership By File Set + +| Lane | Allowed files | Forbidden escalation surfaces | +| --- | --- | --- | +| `L1` / strict resolver | [crates/shell/src/execution/agent_runtime/state_store.rs](/home/spenser/__Active_code/substrate/crates/shell/src/execution/agent_runtime/state_store.rs), [crates/shell/src/execution/agent_runtime/orchestration_session.rs](/home/spenser/__Active_code/substrate/crates/shell/src/execution/agent_runtime/orchestration_session.rs), [crates/shell/src/execution/agent_runtime/session.rs](/home/spenser/__Active_code/substrate/crates/shell/src/execution/agent_runtime/session.rs) | `cli.rs`, `agents_cmd.rs`, `async_repl.rs`, docs, `.runs/**`, public output schema changes | +| `L2` / owner plane | `/home/spenser/__Active_code/substrate/crates/shell/src/execution/agent_runtime/control.rs`, [crates/shell/src/execution/agent_runtime/mod.rs](/home/spenser/__Active_code/substrate/crates/shell/src/execution/agent_runtime/mod.rs), [crates/shell/src/repl/async_repl.rs](/home/spenser/__Active_code/substrate/crates/shell/src/repl/async_repl.rs) | `cli.rs`, `agents_cmd.rs`, docs, `.runs/**`, toolbox mutation paths, global daemon work | +| `L3a` / helper-launch seam | [crates/shell/src/execution/agent_runtime/control.rs](/home/spenser/__Active_code/substrate/crates/shell/src/execution/agent_runtime/control.rs), [crates/shell/src/execution/agent_runtime/mod.rs](/home/spenser/__Active_code/substrate/crates/shell/src/execution/agent_runtime/mod.rs), [crates/shell/src/repl/async_repl.rs](/home/spenser/__Active_code/substrate/crates/shell/src/repl/async_repl.rs), [crates/shell/src/execution/agents_cmd.rs](/home/spenser/__Active_code/substrate/crates/shell/src/execution/agents_cmd.rs), [crates/shell/src/execution/cli.rs](/home/spenser/__Active_code/substrate/crates/shell/src/execution/cli.rs) only for hidden/internal helper wiring, [crates/shell/src/execution/agent_runtime/state_store.rs](/home/spenser/__Active_code/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) only for narrow support edits if required | public help text for `start|resume|fork|stop`, tests, docs, `.runs/**`, prompt-taking surfaces, daemon widening, fake in-process helper wrappers | +| `L3b` / public CLI | [crates/shell/src/execution/cli.rs](/home/spenser/__Active_code/substrate/crates/shell/src/execution/cli.rs), [crates/shell/src/execution/agents_cmd.rs](/home/spenser/__Active_code/substrate/crates/shell/src/execution/agents_cmd.rs) | runtime storage files, `control.rs`, `async_repl.rs`, tests, docs, `.runs/**`, prompt-taking surfaces | +| `L4` / tests docs closeout | `/home/spenser/__Active_code/substrate/crates/shell/tests/agent_public_control_surface_v1.rs`, [crates/shell/tests/agent_successor_contract_ahcsitc0.rs](/home/spenser/__Active_code/substrate/crates/shell/tests/agent_successor_contract_ahcsitc0.rs), [AGENT_ORCHESTRATION_GAP_MATRIX.md](/home/spenser/__Active_code/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md), [llm-last-mile/README.md](/home/spenser/__Active_code/substrate/llm-last-mile/README.md) | all production Rust files, `.runs/**`, overclaiming docs | + +## Workstream Plan + +### `task/m19-p1-parent-contract-freeze-owner-model-and-run-init` + +Owner: + +- parent only + +Scope: + +1. Freeze the public verb contract, exact-selector contract, success JSON fields, and failure taxonomy. +2. Freeze the owner model: + - hidden owner-helper is internal only + - modes are `start`, `resume`, and `fork` + - one private per-session stop transport exists + - REPL-owned and helper-owned live sessions both register the same stop transport contract +3. Freeze the parent-session state machine and readiness rules. +4. Freeze the strict-vs-permissive split: + - permissive: `substrate agent status` + - strict: `doctor`, `toolbox status`, `toolbox env`, `start`, `resume`, `fork`, `stop` +5. Freeze lane ownership, merge order, retry budget, and stop conditions. +6. If needed, take the narrow compile-first scaffold described above and reseed workers from that exact commit. + +Command gates: + +```bash +cargo test -p shell --lib -- --nocapture +cargo test -p shell --test agent_successor_contract_ahcsitc0 --no-run +``` + +Acceptance: + +1. `contract-freeze.json`, `owner-model-freeze.json`, `lane-ownership.json`, `merge-order.json`, `tasks.json`, and `run-state.json` exist. +2. The freeze artifacts record the hidden owner-helper, private stop transport, exact selector rules, Linux-first posture, and strict-vs-permissive split explicitly. +3. Any allowed scaffold is signature-first only and compile-clean. +4. The parent writes `/home/spenser/__Active_code/substrate/.runs/plan-19/sentinels/task-m19-p1-parent-contract-freeze-owner-model-and-run-init.ok`. + +### `task/m19-g1-implementation-window-a-launch-gate` + +Owner: + +- parent only + +Checks: + +1. `p1` is accepted. +2. `L1` and `L2` worktrees were seeded from the exact same post-`p1` tree. +3. Worker prompts name only owned files, forbidden surfaces, command gates, retry budget, and sentinel paths. +4. `L1` prompt explicitly forbids status degradation, trace fallback selection, or non-canonical selectors. +5. `L2` prompt explicitly forbids global-daemon work, toolbox mutation, shared multi-session broker work, or public CLI exposure. + +Acceptance: + +1. No worker starts before this gate is green. +2. The parent writes `/home/spenser/__Active_code/substrate/.runs/plan-19/sentinels/task-m19-g1-implementation-window-a-launch-gate.ok`. + +### `task/m19-l1-strict-resolver-and-naming-guardrails` + +Owner: + +- single worker on `codex/feat-session-centric-state-store-m19-strict-resolver-and-naming-guardrails` + +Scope: + +1. Add the exact public-control resolver on authoritative state-store reads. +2. Enforce exact `orchestration_session_id` resolution for `resume`, `fork`, and `stop`. +3. Reject `participant_id`, `session_handle_id`, and `internal.uaa_session_id` as public selectors. +4. Require exact active participant linkage where action semantics require it. +5. Require `internal.uaa_session_id` presence for `resume` and `fork`. +6. Preserve `orchestration_session_id` as the only public parent handle and keep compatibility alias reads intact. +7. Keep `status` degradation logic untouched and fail closed for public control actions. + +Command gates: + +```bash +cargo test -p shell --lib -- --nocapture +cargo test -p shell --test agent_successor_contract_ahcsitc0 --no-run +``` + +Acceptance: + +1. The lane touches only its owned files. +2. Resolver behavior is exact and never consults fuzzy or trace-history selectors. +3. Canonical naming remains `orchestration_session_id` public, `participant_id` lineage/debug, `internal.uaa_session_id` internal only. +4. Non-Linux or unprovable world-sensitive posture yields `unsupported_platform_or_posture`. +5. The worker writes `/home/spenser/__Active_code/substrate/.runs/plan-19/sentinels/task-m19-l1-strict-resolver-and-naming-guardrails.ok`. + +### `task/m19-l2-owner-plane-and-repl-integration` + +Owner: + +- single worker on `codex/feat-session-centric-state-store-m19-owner-plane-and-repl-integration` + +Scope: + +1. Extract the shared control module under `crates/shell/src/execution/agent_runtime/control.rs`. +2. Move shared host lifecycle logic out of `async_repl.rs` into the shared control module. +3. Add exact resume and fork extension shaping with the existing UAA selector grammar. +4. Freeze the hidden owner-helper modes and internal contract pieces needed for a later helper-launch seam, without yet wiring the callable helper-launch path. +5. Add one private per-session stop transport with exactly one v1 mutation request: + - `{"version":1,"action":"stop"}` +6. Ensure REPL-owned sessions register the same stop transport that helper-owned sessions register. +7. Preserve authoritative shutdown through the existing owner path. Do not add JSON mutation shortcuts. + +Command gates: + +```bash +cargo test -p shell --lib -- --nocapture +cargo test -p shell async_repl --no-run +``` + +Acceptance: + +1. The lane touches only its owned files. +2. No general daemon or global listener is introduced. +3. Owner-helper modes stay hidden from public help and consume resolved internal inputs only where the internal control surface already exists. +4. Stop transport is per-session, exact, private, and owner-mediated. +5. REPL startup, targeted-turn resume wiring, and shutdown semantics remain green in principle and testable. +6. The worker writes `/home/spenser/__Active_code/substrate/.runs/plan-19/sentinels/task-m19-l2-owner-plane-and-repl-integration.ok`. + +### `task/m19-g2-window-a-integration-gate` + +Owner: + +- parent only + +Checks: + +1. `L1` and `L2` both returned patch, report, command transcript, and evidence manifest. +2. Every touched file is inside the lane’s ownership boundary. +3. `L1` landed the exact resolver and naming contract without reopening status semantics. +4. `L2` landed the owner plane without introducing a global broker, toolbox mutation, or public exposure. +5. `L2` is not accepted until the parent proves it replays cleanly against the accepted `L1` tree if it consumed resolver seams. + +Acceptance: + +1. Accepted, rejected, or quarantined status for both Window A lanes is recorded in `run-state.json`. +2. The parent writes `/home/spenser/__Active_code/substrate/.runs/plan-19/sentinels/task-m19-g2-window-a-integration-gate.ok`. + +### `task/m19-p2-parent-window-a-integration` + +Owner: + +- parent only + +Scope: + +1. Integrate accepted `L1` output first. +2. Re-run `L1` command gates on the authoritative checkout. +3. Replay `L2` on top of accepted `L1`. If `L2` assumed a different resolver shape or naming contract, quarantine `L2` instead of hand-editing around it. +4. Integrate accepted `L2` output second. +5. Re-run combined Window A gates on the authoritative checkout. +6. Freeze the merged resolver plus owner-plane truth before CLI work begins. + +Command gates: + +```bash +cargo test -p shell --lib -- --nocapture +cargo test -p shell async_repl --no-run +``` + +Acceptance: + +1. The parent remains the sole integrator. +2. The authoritative tree now contains the exact resolver, canonical naming guardrails, shared control module, owner-plane primitives, and private stop transport contract, but not yet the callable helper-launch seam. +3. No hybrid truth was invented during integration. +4. The parent writes `/home/spenser/__Active_code/substrate/.runs/plan-19/sentinels/task-m19-p2-parent-window-a-integration.ok`. + +### `task/m19-g3-helper-launch-seam-gate` + +Owner: + +- parent only + +Checks: + +1. `p2` is green and the old blocked `task/m19-l3-public-cli-and-handler-wiring` evidence is archived as historical context, not active truth. +2. The helper-launch worktree is seeded from the exact accepted post-`p2` tree. +3. The worker prompt names only: + - `crates/shell/src/execution/agent_runtime/control.rs` + - `crates/shell/src/execution/agent_runtime/mod.rs` + - `crates/shell/src/repl/async_repl.rs` + - `crates/shell/src/execution/agents_cmd.rs` + - `crates/shell/src/execution/cli.rs` only for hidden/internal helper wiring + - `crates/shell/src/execution/agent_runtime/state_store.rs` only for narrow supporting edits if needed +4. The prompt explicitly records the current repo truth: + - `control.rs` already has `OwnerHelperMode`, resume-extension shaping, and private stop transport plumbing + - `mod.rs` does not yet expose a callable helper-launch seam + - `cli.rs` does not yet expose public `start|resume|fork|stop` + - `agents_cmd.rs` does not yet own a parent-side launcher that can spawn a retained helper and wait for readiness + +Acceptance: + +1. No helper-launch worker starts before this gate is green. +2. The parent writes `/home/spenser/__Active_code/substrate/.runs/plan-19/sentinels/task-m19-g3-helper-launch-seam-gate.ok`. + +### `task/m19-l3a-hidden-helper-and-launch-plan-handoff` + +Owner: + +- single worker on `codex/feat-session-centric-state-store-m19-helper-launch-seam` + +Scope: + +1. Add an internal helper mode or hidden subcommand that is callable by the parent path but absent from public help text. +2. Introduce a typed resolved launch-plan handoff object for `start`, `resume`, and `fork`. The helper must consume resolved internal data, not raw fuzzy selectors. +3. Add parent-side launcher logic that can spawn the helper process from the future public command path and wait for authoritative readiness before success returns. +4. Preserve helper-mode stop-owner registration parity with the REPL owner path. +5. Keep `state_store.rs` edits, if any, narrow and seam-local to readiness visibility or helper parity support. +6. Do not add the public verbs in this lane. This lane lands only the missing runtime-to-CLI seam. + +Command gates: + +```bash +cargo test -p shell --lib -- --nocapture +cargo test -p shell async_repl -- --nocapture +cargo test -p shell --test agent_successor_contract_ahcsitc0 --no-run +``` + +Acceptance: + +1. The lane touches only its owned files. +2. The hidden helper is callable internally. +3. The helper takes a resolved launch plan, not raw fuzzy selectors. +4. The helper can establish authoritative ownership for `start`. +5. The helper can establish authoritative ownership for `resume`. +6. The helper can establish authoritative ownership for `fork`. +7. Helper success is gated on store-visible readiness. +8. No public help text exposes the helper. +9. REPL-owned and helper-owned live sessions still register the same private stop transport and route stop through the same owner path. +10. The worker writes `/home/spenser/__Active_code/substrate/.runs/plan-19/sentinels/task-m19-l3a-hidden-helper-and-launch-plan-handoff.ok`. + +### `task/m19-g4-helper-launch-integration-gate` + +Owner: + +- parent only + +Checks: + +1. `L3a` returned a patch, report, command transcript, and evidence manifest. +2. Every touched file is inside `L3a` ownership. +3. `L3a` consumed the accepted resolver and owner-plane truth and did not invent a fake in-process helper, daemon, or raw-selector bypass. +4. The new seam-specific proof wall is green on the authoritative checkout: + - `cargo test -p shell --lib -- --nocapture` + - `cargo test -p shell async_repl -- --nocapture` + - `cargo test -p shell --test agent_successor_contract_ahcsitc0 --no-run` + +Acceptance: + +1. Accepted, rejected, or quarantined status for the helper-launch seam is recorded in `run-state.json`. +2. The parent writes `/home/spenser/__Active_code/substrate/.runs/plan-19/sentinels/task-m19-g4-helper-launch-integration-gate.ok`. + +### `task/m19-p3-helper-launch-integration` + +Owner: + +- parent only + +Scope: + +1. Integrate accepted `L3a` output onto the authoritative checkout. +2. Re-run the helper-launch command gates on the authoritative checkout. +3. Freeze the merged helper-launch seam before any public CLI work begins. + +Command gates: + +```bash +cargo test -p shell --lib -- --nocapture +cargo test -p shell async_repl -- --nocapture +cargo test -p shell --test agent_successor_contract_ahcsitc0 --no-run +``` + +Acceptance: + +1. The authoritative tree now contains a callable hidden helper entrypoint, a typed resolved launch plan, parent-side helper spawn logic, and authoritative readiness waiting. +2. The merged tree still reflects the exact-selector contract, strict-vs-permissive split, Linux-first posture, and owner-plane contract. +3. The parent writes `/home/spenser/__Active_code/substrate/.runs/plan-19/sentinels/task-m19-p3-helper-launch-integration.ok`. + +### `task/m19-g5-public-cli-launch-gate` + +Owner: + +- parent only + +Checks: + +1. `p3` is green. +2. The relaunch CLI worktree is reseeded or rebased to the exact post-`p3` tree. +3. The worker prompt names only `cli.rs` and `agents_cmd.rs`. +4. The prompt explicitly freezes: + - public `Start|Resume|Fork|Stop` + - success JSON fields + - failure taxonomy + - no prompt-taking surface + - host-root-only start + - exact `--session` selector semantics + - the rule that all public launch verbs must map into the already-landed resolved launch-plan seam + +Acceptance: + +1. No public CLI worker starts before this gate is green. +2. The parent writes `/home/spenser/__Active_code/substrate/.runs/plan-19/sentinels/task-m19-g5-public-cli-launch-gate.ok`. + +### `task/m19-l3b-public-cli-and-handler-wiring` + +Owner: + +- single worker on `codex/feat-session-centric-state-store-m19-public-cli-and-handler-wiring-relaunch` + +Scope: + +1. Add public `Start`, `Resume`, `Fork`, and `Stop` actions plus exact arg structs. +2. Map flags into the accepted resolved launch-plan seam only. Do not reopen runtime files or raw-selection behavior. +3. Emit stable JSON and deterministic text output. +4. Keep `start` host-backend exact and reject world-scoped root start explicitly. +5. Keep `resume` same-parent and `fork` new-parent semantics explicit. +6. Keep `stop` waiting for terminal parent state, not mere request acceptance. + +Command gates: + +```bash +cargo fmt --all -- --check +cargo test -p shell --lib -- --nocapture +cargo test -p shell --test agent_successor_contract_ahcsitc0 --no-run +``` + +Acceptance: + +1. The lane touches only its owned files. +2. Public help exposes only the four frozen public verbs. +3. No public selector broadening occurs. +4. No prompt-taking public surface is added. +5. Output fields and error names match the frozen contract exactly. +6. The worker writes `/home/spenser/__Active_code/substrate/.runs/plan-19/sentinels/task-m19-l3b-public-cli-and-handler-wiring.ok`. + +### `task/m19-g6-cli-integration-gate` + +Owner: + +- parent only + +Checks: + +1. `L3b` returned a patch, report, command transcript, and evidence manifest. +2. Every touched file is inside `L3b` ownership. +3. `L3b` consumed the accepted helper-launch seam, control-module truth, and resolver truth and did not invent branch-local semantics. +4. `L3b` did not widen into prompt submission, `substrate -c`, or member-level selectors. + +Acceptance: + +1. Accepted, rejected, or quarantined status for the public CLI lane is recorded in `run-state.json`. +2. The parent writes `/home/spenser/__Active_code/substrate/.runs/plan-19/sentinels/task-m19-g6-cli-integration-gate.ok`. + +### `task/m19-p4-parent-cli-integration` + +Owner: + +- parent only + +Scope: + +1. Integrate accepted `L3b` output onto the authoritative checkout. +2. Re-run `L3b` command gates on the authoritative checkout. +3. Freeze the merged public command surface before tests/docs begin. + +Command gates: + +```bash +cargo fmt --all -- --check +cargo test -p shell --lib -- --nocapture +``` + +Acceptance: + +1. The authoritative tree now contains the full public command surface. +2. The merged tree still reflects the exact-selector contract, strict-vs-permissive split, Linux-first posture, and owner-plane contract. +3. The parent writes `/home/spenser/__Active_code/substrate/.runs/plan-19/sentinels/task-m19-p4-parent-cli-integration.ok`. + +### `task/m19-g7-closeout-launch-gate` + +Owner: + +- parent only + +Checks: + +1. `p4` is green. +2. The closeout worktree is reseeded or rebased to the exact post-`p4` tree. +3. The worker prompt names only the allowed test and doc files. +4. The worker prompt explicitly forbids reopening production runtime files. + +Acceptance: + +1. No tests/docs worker starts before this gate is green. +2. The parent writes `/home/spenser/__Active_code/substrate/.runs/plan-19/sentinels/task-m19-g7-closeout-launch-gate.ok`. + +### `task/m19-l4-tests-docs-gap-matrix-and-qa-closeout` + +Owner: + +- single worker on `codex/feat-session-centric-state-store-m19-tests-docs-gap-matrix-and-qa-closeout` + +Scope: + +1. Add or complete `agent_public_control_surface_v1.rs` as the dedicated public-control integration suite. +2. Extend `agent_successor_contract_ahcsitc0.rs` only for non-regression proof where the public-control slice touches prior status and lineage truth. +3. Cover: + - public host `start` + - public `resume` of an orphaned session + - public `fork` creating a new orchestration session + - public `stop` through the owner transport + - selector rejection for wrong handle types + - world-sensitive rejection on unsupported platform or posture + - REPL-owned session stoppability through the same private owner plane +4. Update the gap matrix to mark only the shipped host-orchestrator control surface as landed. +5. Update the packet README index if needed so `PLAN-19` and `ORCH_PLAN-19` are discoverable. +6. Write the QA-facing artifact to the frozen pattern: + - `~/.gstack/projects//-feat-session-centric-state-store-eng-review-test-plan-.md` + +Command gates: + +```bash +cargo test -p shell --test agent_public_control_surface_v1 -- --nocapture +cargo test -p shell --test agent_successor_contract_ahcsitc0 -- --nocapture +rg -n "start|resume|fork|stop|orchestration_session_id|participant_id|Linux-first" /home/spenser/__Active_code/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md /home/spenser/__Active_code/substrate/llm-last-mile/README.md +``` + +Acceptance: + +1. The lane touches only its owned files. +2. Tests prove all four public verbs and the rejection matrix directly. +3. Docs reflect host-orchestrator public control landed, Linux-first world-sensitive posture, and no prompt-taking caller surface. +4. The gap matrix does not overclaim world-root start, macOS parity, Windows parity, or toolbox mutation. +5. The worker writes `/home/spenser/__Active_code/substrate/.runs/plan-19/sentinels/task-m19-l4-tests-docs-gap-matrix-and-qa-closeout.ok`. + +### `task/m19-g8-validation-wall-gate` + +Owner: + +- parent only + +Checks: + +1. `L4` returned and is classified. +2. `L4` is accepted. +3. No quarantined or blocked output remains unresolved. +4. `validation-wall.md` names the exact final command order and manual spot checks. +5. The parent can enumerate every `PLAN-19` definition-of-done clause and the command or artifact that proves it. + +Acceptance: + +1. The parent writes `/home/spenser/__Active_code/substrate/.runs/plan-19/sentinels/task-m19-g8-validation-wall-gate.ok`. +2. The validation wall is permitted to run exactly once on the final merged tree. + +### `task/m19-p5-parent-validation-wall-and-closeout` + +Owner: + +- parent only + +Scope: + +1. Integrate only accepted `L4` output. +2. Run the full validation wall in exact order. +3. Record final command results and artifact paths in the task artifact directory. +4. Confirm the gap matrix and README match the validated runtime truth. +5. Mark the run complete only if the validation wall proves owner-plane honesty, exact selectors, Linux-first fail-closed posture, and strict-vs-permissive surface discipline. + +Required final artifacts under `/home/spenser/__Active_code/substrate/.runs/task-m19-p5-parent-validation-wall-and-closeout/artifacts/`: + +- `fmt.txt` +- `clippy.txt` +- `shell-lib-tests.txt` +- `agent-public-control-surface-v1.txt` +- `agent-successor-contract.txt` +- `async-repl-tests.txt` +- `agent-start-spot-check.txt` +- `agent-resume-spot-check.txt` +- `agent-fork-spot-check.txt` +- `agent-stop-spot-check.txt` +- `agent-status-spot-check.txt` +- `agent-doctor-spot-check.txt` +- `agent-toolbox-status-spot-check.txt` +- `agent-toolbox-env-spot-check.txt` +- `contract-audit.md` +- `closeout.md` + +Acceptance: + +1. All validation commands succeed on the authoritative checkout. +2. Manual spot checks are captured with expected operator-visible outcomes. +3. `run-state.json` records a successful terminal state. +4. The parent writes `/home/spenser/__Active_code/substrate/.runs/plan-19/sentinels/task-m19-p5-parent-validation-wall-and-closeout.ok`. + +## Quarantine, Retry, And Blocked-Run Posture + +1. Each worker lane has retry budget `1`. +2. Retry is allowed only for lane-local defects inside owned files. +3. Non-retryable violations include: + - public selector broadening + - status-style degradation on strict control surfaces + - world-root public start + - JSON mutation or toolbox mutation pretending to be stop + - hidden owner-helper becoming public help + - global daemon or shared broker introduction + - macOS/Lima or Windows/WSL parity broadening + - prompt-taking public surface addition +4. If `L1` cannot keep exact selector discipline or canonical naming, quarantine it immediately. +5. If `L2` can only work by adding a general daemon or global listener, quarantine it immediately. +6. If `L3a` can only pass by inventing a fake in-process helper, mutating session JSON, or returning success before store-visible readiness, quarantine it immediately. +7. If `L3b` can only pass by inventing output fields or fuzzy selection, quarantine it immediately. +8. If `L4` can only make docs pass by overclaiming landed behavior, reject `L4` and stop the run. + +Blocked termination minimum contents: + +1. task or gate where execution stopped +2. classification: `rejected`, `blocked`, `quarantined`, or `merge_refused` +3. exact contract clause or ownership rule that stopped the run +4. whether retry remained available +5. artifact paths for patch, report, command output, and semantic-drift evidence +6. explicit statement that no blocked output was integrated + +## Context-Control Rules + +1. The parent keeps only a bounded live context: + - [PLAN-19.md](/home/spenser/__Active_code/substrate/llm-last-mile/PLAN-19.md) + - this orchestration plan + - `tasks.json` + - `run-state.json` + - `contract-freeze.json` + - `owner-model-freeze.json` + - `merge-order.json` + - latest integration diff summary +2. Worker prompts contain only: + - owned file set + - exact frozen contract excerpts relevant to that lane + - for `L3a`, the explicit statement that `cli.rs` currently lacks public `Start|Resume|Fork|Stop` and that the lane may only add hidden/internal helper wiring there + - required commands + - forbidden touch surfaces + - retry budget + - sentinel path +3. Workers return summaries and artifacts only. They do not become independent approval or truth surfaces. +4. Workers do not write `/home/spenser/__Active_code/substrate/.runs/plan-19/*`. +5. The parent reviews summaries plus narrow diffs only and does not ingest full worker transcripts into the main run context. +6. Each worker is closed after accept, reject, or quarantine to keep context bounded. +7. No external human approval gate is assumed in this run. Every gate here is parent-only. If a true unresolved product decision appears, the run stops rather than inventing one. + +## Tests And Acceptance + +### A. Frozen-contract acceptance + +- `contract-freeze.json` records the public verbs, exact selectors, success JSON fields, failure taxonomy, strict-vs-permissive split, and host-root-only start rule. +- `owner-model-freeze.json` records the hidden owner-helper modes, required inputs, resolved launch-plan handoff, private stop transport request and outcome contract, readiness rules, seam acceptance checklist, and state machine. +- No worker prompt reopens those decisions. + +### B. `L1` acceptance + +- Exact `orchestration_session_id` resolution exists for public control. +- `participant_id`, `session_handle_id`, and `internal.uaa_session_id` are rejected as public selectors. +- `internal.uaa_session_id` remains readable internally for exact resume and fork. +- Linux-first world-sensitive rejection is explicit and fail closed. +- Status degradation semantics remain untouched. + +### C. `L2` acceptance + +- Shared lifecycle logic is extracted to `control.rs`. +- Hidden owner-helper exists only as an internal entrypoint. +- REPL-owned and helper-owned live sessions both register the same stop transport contract. +- Stop remains owner-mediated and authoritative. +- No general daemon, shared broker, or toolbox mutation plane exists. + +### D. `L3a` acceptance + +- A callable hidden helper exists and remains absent from public help text. +- A typed resolved launch-plan handoff exists for `start`, `resume`, and `fork`. +- The parent can spawn the helper and wait for authoritative readiness before success returns. +- Helper-owned live sessions register the same stop-owner contract as REPL-owned live sessions. +- No fake in-process helper, daemon, or raw-selector bypass was introduced. + +### E. `L3b` acceptance + +- `substrate agent` exposes exactly `start`, `resume`, `fork`, and `stop`. +- `start` accepts only exact host `backend_id`. +- `resume`, `fork`, and `stop` accept only exact `orchestration_session_id`. +- JSON output is stable and scriptable. +- No prompt-taking public caller surface exists. + +### F. `L4` acceptance + +- The dedicated public-control integration suite proves success and failure paths for all four public verbs. +- Existing strict doctor/toolbox regressions remain green. +- REPL targeted-turn resume behavior remains green. +- Docs reflect only landed host-orchestrator public control with Linux-first world-sensitive posture. +- The QA-facing artifact exists. + +### G. Integration acceptance + +- `L1` integrates first. +- `L2` is replayed on top of accepted `L1` truth before acceptance if needed. +- `L3a` starts only after accepted `L1` and `L2` are integrated. +- `L3b` starts only after accepted `L3a` is integrated through `p3`. +- No parent-only hybrid edits are required to explain merged behavior. +- No quarantined or blocked output is partially integrated. + +## Validation Wall + +Parent-owned validation commands, executed only after `L4` is integrated: + +```bash +cargo fmt --all -- --check +cargo clippy --workspace --all-targets -- -D warnings +cargo test -p shell --lib -- --nocapture +cargo test -p shell --test agent_public_control_surface_v1 -- --nocapture +cargo test -p shell --test agent_successor_contract_ahcsitc0 -- --nocapture +cargo test -p shell async_repl -- --nocapture +``` + +Manual spot checks after the command wall is green: + +```bash +substrate agent start --backend --json +substrate agent resume --session --json +substrate agent fork --session --json +substrate agent stop --session --json +substrate agent status --json +substrate agent doctor --json +substrate agent toolbox status --json +substrate agent toolbox env --json +``` + +Validation-wall requirements: + +1. formatting and clippy stay green +2. shell library tests stay green +3. the dedicated public-control suite proves success and rejection behavior for all four verbs +4. targeted successor and REPL tests prove no regression in prior runtime truth +5. manual spot checks confirm the same operator contract the tests prove +6. docs do not overclaim beyond what the green runtime and tests prove + +## Completion Criteria Tied To PLAN-19 Definition Of Done + +The run is complete only if all of these are true: + +1. `substrate agent` publicly exposes `start`, `resume`, `fork`, and `stop`. +2. `start` accepts only exact host-scoped `backend_id`. +3. `resume`, `fork`, and `stop` accept only exact `orchestration_session_id`. +4. No public command accepts or emits `internal.uaa_session_id` as a selector. +5. `start`, `resume`, and `fork` return only after authoritative readiness is visible in the store. +6. `stop` routes through the live owner and reaches a terminal parent-session state. +7. `resume` rebinds the same parent session and `fork` creates a new parent session. +8. REPL-owned live sessions expose the same private owner plane for public stop. +9. Strict `doctor` and `toolbox` behavior remain fail closed. +10. Root world-session start is rejected explicitly. +11. World-sensitive control remains Linux-first and fail closed elsewhere. +12. Repo-truth docs reflect landed behavior and nothing broader. + +## Final State + +Success requires all of: + +1. every required sentinel exists +2. no active blocked artifact exists under `/home/spenser/__Active_code/substrate/.runs/plan-19/` +3. accepted outputs were integrated in the prescribed order only +4. final validation commands pass +5. manual spot checks confirm the same operator contract the tests prove +6. `contract-freeze.json`, `owner-model-freeze.json`, and the final merged tree still agree on the frozen `PLAN-19` contract + +Blocked termination requires any of: + +1. hard-guard violation +2. non-retryable rejection +3. exhausted retry budget +4. merge refusal with no legal redrive path +5. failed validation wall +6. docs requiring overclaim to appear complete + +On blocked termination the parent must write: + +1. `/home/spenser/__Active_code/substrate/.runs/plan-19/blocked.json` +2. terminal state and rationale in `run-state.json` +3. gate and failure summary in `session-log.md` +4. preserved evidence under `/home/spenser/__Active_code/substrate/.runs/plan-19/quarantine/`, `/home/spenser/__Active_code/substrate/.runs/plan-19/history/` for superseded blocked attempts, and the relevant task artifact directory + +## Assumptions + +1. [PLAN-19.md](/home/spenser/__Active_code/substrate/llm-last-mile/PLAN-19.md) remains the authoritative dependency graph for this run. +2. The authoritative integration checkout remains `/home/spenser/__Active_code/substrate` on `feat/session-centric-state-store`. +3. The core production hotspots are `control.rs`, `state_store.rs`, `async_repl.rs`, `cli.rs`, `agents_cmd.rs`, and `agent_runtime/mod.rs`. +4. Linux-first public control posture is the only honest platform goal for this run. +5. Docs move last, after integrated code truth and green regressions exist. +6. The missing post-`p2` helper-launch seam can be expressed without introducing a new daemon or new public mutation surface. +7. If the compile-first scaffold is unnecessary, `p1` records that explicitly and workers still branch from the exact same post-`p1` commit. diff --git a/llm-last-mile/ORCH_PLAN-20.md b/llm-last-mile/ORCH_PLAN-20.md new file mode 100644 index 000000000..ab93489ce --- /dev/null +++ b/llm-last-mile/ORCH_PLAN-20.md @@ -0,0 +1,599 @@ +# ORCH_PLAN-20: Execute PLAN-20 Through Exact Prompt-Taking Verbs, Backend-Aware Turn Resolution, And A Helper-Owned Stream Bridge + +Branch: `feat/session-centric-state-store` +Plan source: [PLAN-20.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-20.md) +Style reference: [ORCH_PLAN-19.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/ORCH_PLAN-19.md) +Structure reference: M26 orchestration example provided in chat +Packet index: [README.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/README.md) +Gap matrix anchor: [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md) +Execution type: fresh orchestration controller for prompt-taking public agent surface, Linux-first retained follow-up, parent-frozen contract, parent-only integration and approval +Live root: `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate` +Worktree root: `/Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-20` +Worker model: `GPT-5.4` with `reasoning_effort=high` +Max concurrent code workers before integration: `3` + +## Summary + +This document is the execution controller for `PLAN-20`, not a restatement of it. + +The run lands one honest non-interactive public caller surface under `substrate agent` with: + +- `start` as `new session + first real prompt` +- `turn` as exact follow-up prompt submission for an existing `(orchestration_session_id, backend_id)` +- `reattach` as the canonical lifecycle recovery verb +- helper-owned streaming for prompt-taking calls +- authoritative completion-time `session_posture` +- Linux-first world follow-up through the existing retained-member seam +- explicit fail-closed behavior for non-Linux world-sensitive follow-up +- unchanged `substrate -c` + +True code concurrency is capped at `3` because there are exactly three disjoint implementation seams before closeout: + +1. public CLI grammar and command-handler surface +2. backend-aware exact turn resolver and posture classification +3. shared prompt-submit extraction plus helper-owned bridge + +A fourth code lane would immediately collide with these surfaces or reopen tests/docs too early. + +Frozen run shape: + +1. `task/m20-p1-parent-contract-freeze-run-init-and-api-scaffold` +2. `task/m20-g1-implementation-launch-gate` +3. parallel Window A + - `task/m20-l1-public-cli-grammar-and-canonical-verb-surface` + - `task/m20-l2-backend-aware-turn-resolver-and-posture` + - `task/m20-l3-shared-prompt-submit-and-helper-stream-bridge` +4. `task/m20-g2-window-a-integration-gate` +5. `task/m20-p2-parent-window-a-integration` +6. `task/m20-g3-closeout-launch-gate` +7. `task/m20-l4-tests-docs-gap-matrix-and-qa-closeout` +8. `task/m20-g4-validation-wall-gate` +9. `task/m20-p3-parent-validation-wall-and-closeout` + +## Hard Guards + +1. The authoritative integration checkout remains `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate` on `feat/session-centric-state-store`. +2. The parent agent is the only integrator, the only approval authority, and the only writer of `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/plan-20/**`. +3. Public prompt-taking grammar is frozen to exactly: + - `substrate agent start --backend (--prompt | --prompt-file | --prompt-file -) [--json]` + - `substrate agent turn --session --backend (--prompt | --prompt-file | --prompt-file -) [--json]` + - `substrate agent reattach --session [--json]` + - `substrate agent fork --session [--json]` + - `substrate agent stop --session [--json]` +4. `resume` is never the canonical public term. If retained, it is a hidden deprecated alias only. All docs, help, human output, and JSON `action` values say `reattach`. +5. `start` and `turn` require exactly one prompt source. Missing, malformed, unreadable, invalid UTF-8, or effectively empty prompt input is a hard pre-launch failure. +6. No public prompt-taking path may inject or fall back to synthetic `runtime_bootstrap_prompt` text as public user input. +7. `substrate -c`, `--command`, pipe mode, and plain stdin remain shell-wrap semantics. This run must not reinterpret them as agent prompting. +8. Every public `turn` names both exact `orchestration_session_id` and exact `backend_id`. No fuzzy routing, no latest-session routing, no agent-id routing. +9. No public prompt-taking or recovery surface accepts `participant_id`, `active_session_handle_id`, `session_handle_id`, or `internal.uaa_session_id` as input. +10. Noncanonical handles remain explicit rejection paths. No worker may soften that contract. +11. Public `start` and `turn` stream real output while the submitted turn is running. No full-output buffering before completion. +12. `session_posture` is authoritative only at command completion time and is frozen to `active`, `detached_reattachable`, or `terminal`. +13. Host and world follow-up paths stay distinct. Linux world follow-up continues through the retained-member seam. Non-Linux world-sensitive follow-up fails closed with `unsupported_platform_or_posture`. +14. Root `start` remains exact-backend host-first only. World-only root start is an explicit rejection path in this slice. +15. `fork` and `stop` stay lifecycle surfaces, not prompt-taking surfaces. +16. No docs may claim non-Linux parity for world-sensitive follow-up or claim world-root public start shipped. +17. No worker may widen into daemon work, default-agent routing, toolbox mutation, macOS parity claims, or Windows parity claims. +18. No worker may edit `.runs/**`. +19. If the frozen NDJSON envelope, prompt-source contract, `session_posture`, or selector rules become disputed during implementation, the run stops and the parent writes `blocked.json`. + +## Fresh Worktrees And Branches + +Fresh worktree root: + +- `/Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-20` + +Worker worktrees: + +- `/Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-20/public-cli-grammar-and-canonical-verb-surface` +- `/Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-20/backend-aware-turn-resolver-and-posture` +- `/Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-20/shared-prompt-submit-and-helper-stream-bridge` +- `/Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-20/tests-docs-gap-matrix-and-qa-closeout` + +Worker branches: + +- `codex/feat-session-centric-state-store-m20-public-cli-grammar-and-canonical-verb-surface` +- `codex/feat-session-centric-state-store-m20-backend-aware-turn-resolver-and-posture` +- `codex/feat-session-centric-state-store-m20-shared-prompt-submit-and-helper-stream-bridge` +- `codex/feat-session-centric-state-store-m20-tests-docs-gap-matrix-and-qa-closeout` + +Worktree creation commands: + +```bash +mkdir -p /Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-20 + +git worktree add /Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-20/public-cli-grammar-and-canonical-verb-surface \ + -b codex/feat-session-centric-state-store-m20-public-cli-grammar-and-canonical-verb-surface \ + feat/session-centric-state-store + +git worktree add /Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-20/backend-aware-turn-resolver-and-posture \ + -b codex/feat-session-centric-state-store-m20-backend-aware-turn-resolver-and-posture \ + feat/session-centric-state-store + +git worktree add /Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-20/shared-prompt-submit-and-helper-stream-bridge \ + -b codex/feat-session-centric-state-store-m20-shared-prompt-submit-and-helper-stream-bridge \ + feat/session-centric-state-store + +git worktree add /Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-20/tests-docs-gap-matrix-and-qa-closeout \ + -b codex/feat-session-centric-state-store-m20-tests-docs-gap-matrix-and-qa-closeout \ + feat/session-centric-state-store +``` + +Parent integration surface: + +- `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate` on `feat/session-centric-state-store` +- no separate parent integration worktree + +## Parent-Owned Run-State Surface + +Canonical parent-owned state under `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/plan-20/`: + +- `run-state.json` +- `tasks.json` +- `session-log.md` +- `contract-freeze.json` +- `api-scaffold-freeze.json` +- `lane-ownership.json` +- `merge-order.json` +- `validation-wall.md` +- `blocked.json` on stop only +- `quarantine/` +- `sentinels/` + +Canonical per-task roots: + +- `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/task-m20-p1-parent-contract-freeze-run-init-and-api-scaffold/` +- `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/task-m20-g1-implementation-launch-gate/` +- `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/task-m20-l1-public-cli-grammar-and-canonical-verb-surface/` +- `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/task-m20-l2-backend-aware-turn-resolver-and-posture/` +- `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/task-m20-l3-shared-prompt-submit-and-helper-stream-bridge/` +- `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/task-m20-g2-window-a-integration-gate/` +- `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/task-m20-p2-parent-window-a-integration/` +- `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/task-m20-g3-closeout-launch-gate/` +- `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/task-m20-l4-tests-docs-gap-matrix-and-qa-closeout/` +- `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/task-m20-g4-validation-wall-gate/` +- `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/task-m20-p3-parent-validation-wall-and-closeout/` + +Each task root must contain: + +- `task.json` +- `commands.txt` +- `summary.md` + +Each gate task also contains: + +- `gate-checklist.md` +- `gate-result.json` + +Each worker task also contains: + +- `worker-report.md` +- `worker-output.patch` +- `evidence-manifest.json` + +Sentinel convention: + +- `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/plan-20/sentinels/.ok` + +`contract-freeze.json` is the single source of truth for: + +- exact public verbs +- prompt-source rules +- canonical `reattach` naming +- NDJSON envelope +- human terminal summary fields +- `session_posture` meanings +- failure taxonomy +- Linux-first fail-closed rule +- `-c` non-regression rule +- doc truth boundaries +- stop conditions + +`api-scaffold-freeze.json` is the single source of truth for: + +- frozen shared types and function signatures the parent may scaffold before worker launch +- ownership of `resolve_public_turn_target(...)` +- ownership of prompt-source loader and stream-bridge entrypoints +- ownership of extracted shared submit helpers +- ownership of public handler entrypoints in `agents_cmd.rs` + +## Task Ledger + +| Task ID | Owner | Depends on | Worktree / branch | Deliverable | +| --- | --- | --- | --- | --- | +| `task/m20-p1-parent-contract-freeze-run-init-and-api-scaffold` | parent | — | authoritative checkout | frozen contract, run artifacts, narrow signature scaffold | +| `task/m20-g1-implementation-launch-gate` | parent | `p1` | authoritative checkout | launch approval for Window A | +| `task/m20-l1-public-cli-grammar-and-canonical-verb-surface` | worker | `g1` | `public-cli-grammar-and-canonical-verb-surface` / `codex/feat-session-centric-state-store-m20-public-cli-grammar-and-canonical-verb-surface` | public args, canonical verbs, JSON/human output wiring over frozen seams | +| `task/m20-l2-backend-aware-turn-resolver-and-posture` | worker | `g1` | `backend-aware-turn-resolver-and-posture` / `codex/feat-session-centric-state-store-m20-backend-aware-turn-resolver-and-posture` | exact `(session, backend)` resolver plus posture classification inputs | +| `task/m20-l3-shared-prompt-submit-and-helper-stream-bridge` | worker | `g1` | `shared-prompt-submit-and-helper-stream-bridge` / `codex/feat-session-centric-state-store-m20-shared-prompt-submit-and-helper-stream-bridge` | shared prompt-source loader, extracted submit seam, helper-owned stream bridge | +| `task/m20-g2-window-a-integration-gate` | parent | `l1`, `l2`, `l3` | authoritative checkout | acceptance or quarantine decision for Window A | +| `task/m20-p2-parent-window-a-integration` | parent | `g2` | authoritative checkout | merged code truth for CLI, resolver, and bridge | +| `task/m20-g3-closeout-launch-gate` | parent | `p2` | authoritative checkout | launch approval for tests/docs closeout | +| `task/m20-l4-tests-docs-gap-matrix-and-qa-closeout` | worker | `g3` | `tests-docs-gap-matrix-and-qa-closeout` / `codex/feat-session-centric-state-store-m20-tests-docs-gap-matrix-and-qa-closeout` | integration tests, REPL non-regression checks, gap matrix and packet index updates, QA artifact | +| `task/m20-g4-validation-wall-gate` | parent | `l4` | authoritative checkout | permission to run final wall | +| `task/m20-p3-parent-validation-wall-and-closeout` | parent | `g4` | authoritative checkout | final validation, operator checks, terminal run state | + +## Merge Order + +`merge-order.json` is frozen during `p1` and governs integration. + +1. The parent may take one narrow compile-first scaffold before worker launch in: + - `crates/shell/src/execution/agent_runtime/control.rs` + - `crates/shell/src/execution/agent_runtime/mod.rs` + - `crates/shell/src/execution/agent_runtime/state_store.rs` +2. The scaffold is signature-first only. It may freeze types and function names, not semantics. +3. `L2` integrates before `L3` acceptance is finalized if `L3` consumed target-resolution types from the scaffold. +4. `L3` integrates before `L1` acceptance is finalized if `L1` consumed bridge or prompt-source entrypoints from landed runtime truth. +5. `L1` is replayed on top of the accepted `L2 + L3` tree before final acceptance if handler assumptions drifted from landed runtime truth. +6. `L4` starts only after the merged `p2` tree is green. +7. The parent never hand-merges contradictory `session_posture`, failure-code, selector, or stream-envelope semantics out of worker-local assumptions. + +Quarantine rules: + +- Quarantine `L1` if it touches `async_repl.rs`, state-store files, or invents runtime semantics. +- Quarantine `L2` if it touches CLI, docs, or tests, or broadens handle acceptance. +- Quarantine `L3` if it touches CLI parse surfaces, docs, or tests, duplicates REPL submit logic, or buffers full output. +- Quarantine `L4` if it reopens any production Rust file. +- Quarantine any lane that invents non-Linux parity claims or world-root public start claims. + +## Launch/Integration Gates + +### `task/m20-p1-parent-contract-freeze-run-init-and-api-scaffold` + +Parent only. + +Scope: + +1. Create `.runs/plan-20` and every per-task root. +2. Write `tasks.json`, `run-state.json`, `session-log.md`, `contract-freeze.json`, `api-scaffold-freeze.json`, `lane-ownership.json`, `merge-order.json`, and `validation-wall.md`. +3. Freeze the exact NDJSON envelope: + - `accepted` + - `event` + - `warning` + - `completed` + - `failed` +4. Freeze operator-facing error codes from `PLAN-20`. +5. Freeze `session_posture` meanings and canonical `reattach` naming. +6. If needed, add compile-first placeholder signatures for: + - prompt-source loader + - public start/turn entrypoints + - exact backend-aware turn resolver + - shared stream sink / bridge entrypoints + - shared host/world prompt-submit helpers + +Acceptance: + +- all freeze artifacts exist +- any scaffold is compile-clean and signature-only +- `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/plan-20/sentinels/task-m20-p1-parent-contract-freeze-run-init-and-api-scaffold.ok` exists + +### `task/m20-g1-implementation-launch-gate` + +Parent only. + +Checks: + +1. `p1` is accepted. +2. All three worktrees are seeded from the exact same post-`p1` tree. +3. Worker prompts name only owned files, frozen contracts, command gates, retry budget, and sentinel paths. +4. `L1` prompt explicitly forbids touching `async_repl.rs`, `state_store.rs`, `session.rs`, and `orchestration_session.rs`. +5. `L2` prompt explicitly forbids touching `cli.rs`, `agents_cmd.rs`, docs, and tests. +6. `L3` prompt explicitly forbids touching `cli.rs`, docs, and tests and records that it owns the shared helper bridge and extraction seam. + +Acceptance: + +- no worker starts before this gate is green +- `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/plan-20/sentinels/task-m20-g1-implementation-launch-gate.ok` exists + +### `task/m20-g2-window-a-integration-gate` + +Parent only. + +Checks: + +1. `L1`, `L2`, and `L3` each returned patch, report, command transcript, and evidence manifest. +2. Every touched file stays within lane ownership. +3. `L2` preserved exact selector rules and canonical internal/public handle split. +4. `L3` preserved the existing REPL host/world submit behavior while extracting shared helpers. +5. `L3` preserved Linux-first world follow-up and explicit non-Linux fail-closed behavior. +6. `L1` kept `-c` unchanged and emitted canonical `reattach`. +7. If `L1` assumed APIs that differ from accepted `L2 + L3`, the parent replays `L1` after those lanes integrate or quarantines `L1`. + +Acceptance: + +- accepted, rejected, or quarantined status is recorded for all three lanes +- `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/plan-20/sentinels/task-m20-g2-window-a-integration-gate.ok` exists + +### `task/m20-p2-parent-window-a-integration` + +Parent only. + +Integration sequence and commands: + +1. Integrate accepted `L2` first. + +```bash +git cherry-pick +cargo test -p shell --lib -- --nocapture +cargo test -p shell --test agent_successor_contract_ahcsitc0 --no-run +``` + +2. Integrate accepted `L3` second on top of `L2`. + +```bash +git cherry-pick +cargo test -p shell --lib -- --nocapture +cargo test -p shell --test repl_world_first_routing_v1 --no-run +``` + +3. Compare frozen scaffold signatures versus `L1` assumptions. + - If no drift, integrate `L1`. + - If drift exists, replay `L1` worktree on top of the accepted `L2 + L3` tree first, regenerate its patch, then integrate or quarantine. + +4. Integrate accepted or replayed `L1` last. + +```bash +git cherry-pick +cargo fmt --all -- --check +cargo test -p shell --lib -- --nocapture +cargo test -p shell --test agent_successor_contract_ahcsitc0 --no-run +``` + +5. Run merged Window A proof wall. + +```bash +cargo test -p shell --test repl_world_first_routing_v1 --no-run +cargo clippy --workspace --all-targets -- -D warnings +``` + +Operational rules: + +- The parent remains the only integrator. +- No hand-edited hybrid semantics for `session_posture`, selectors, or NDJSON fields. +- If `L1` needs replay due to signature drift, replay happens before any final `L1` acceptance marker is written. +- If replay cannot be completed cleanly, quarantine `L1` and stop the run. + +Acceptance: + +- the authoritative tree now contains exact public grammar, exact backend-aware turn resolution, shared prompt-submit truth, helper-owned bridge, canonical `reattach`, and unchanged `-c` +- no hybrid or hand-invented semantics were introduced +- `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/plan-20/sentinels/task-m20-p2-parent-window-a-integration.ok` exists + +### `task/m20-g3-closeout-launch-gate` + +Parent only. + +Checks: + +1. `p2` is green. +2. The closeout worktree is seeded from exact post-`p2` truth. +3. The worker prompt names only allowed test and doc files. +4. The worker prompt explicitly forbids reopening production runtime files. + +Acceptance: + +- no closeout worker starts before this gate is green +- `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/plan-20/sentinels/task-m20-g3-closeout-launch-gate.ok` exists + +### `task/m20-g4-validation-wall-gate` + +Parent only. + +Checks: + +1. `L4` returned and is classified. +2. `L4` is accepted. +3. No quarantined or blocked output remains unresolved. +4. `validation-wall.md` names exact final command order and manual spot checks. +5. The parent can map every `PLAN-20` done clause to a command, test, or artifact. + +Acceptance: + +- `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/plan-20/sentinels/task-m20-g4-validation-wall-gate.ok` exists + +## Context-Control Rules + +1. Workers receive only the contract fragment, owned-file list, forbidden surfaces, and current accepted-tree assumptions they need. +2. The parent does all cross-lane reasoning, merge-order decisions, and quarantine decisions. +3. No worker prompt may say “use best judgment” for selector semantics, `session_posture`, prompt-source rules, or NDJSON shape. +4. `L1` owns only: + - `crates/shell/src/execution/cli.rs` + - `crates/shell/src/execution/agents_cmd.rs` +5. `L1` must not touch: + - `crates/shell/src/repl/async_repl.rs` + - `crates/shell/src/execution/agent_runtime/state_store.rs` + - `crates/shell/src/execution/agent_runtime/session.rs` + - `crates/shell/src/execution/agent_runtime/orchestration_session.rs` + - docs + - tests +6. `L2` owns only: + - `crates/shell/src/execution/agent_runtime/state_store.rs` + - `crates/shell/src/execution/agent_runtime/session.rs` if needed for posture helpers + - `crates/shell/src/execution/agent_runtime/orchestration_session.rs` if needed for authoritative metadata comments or narrow support +7. `L2` must not touch: + - `crates/shell/src/execution/cli.rs` + - `crates/shell/src/execution/agents_cmd.rs` + - `crates/shell/src/repl/async_repl.rs` + - docs + - tests +8. `L3` owns only: + - `crates/shell/src/execution/agent_runtime/control.rs` + - `crates/shell/src/execution/agent_runtime/mod.rs` + - `crates/shell/src/repl/async_repl.rs` +9. `L3` owns: + - shared prompt-source loader + - extracted shared host/world prompt-submit seam + - helper-owned stream bridge + - bridge-friendly event sink surface for public `start` and `turn` +10. `L3` must not touch: + - `crates/shell/src/execution/cli.rs` + - `crates/shell/src/execution/agents_cmd.rs` + - docs + - tests +11. `L4` owns only: + - `crates/shell/tests/agent_public_control_surface_v1.rs` + - `crates/shell/tests/agent_successor_contract_ahcsitc0.rs` + - `crates/shell/tests/repl_world_first_routing_v1.rs` + - `AGENT_ORCHESTRATION_GAP_MATRIX.md` + - `llm-last-mile/README.md` +12. `L4` must not reopen any production code. +13. `L4` also writes the QA artifact to: + - `~/.gstack/projects//-feat-session-centric-state-store-eng-review-test-plan-.md` +14. Workers may not edit `.runs/**`, create new execution planes, or claim unsupported platform parity. +15. If a lane hits a blocked dependency on another lane’s unfrozen semantics, it stops and requests parent clarification instead of coding around the gap. + +## Tests And Acceptance + +### Lane command gates + +`L1`: + +```bash +cargo fmt --all -- --check +cargo test -p shell --lib -- --nocapture +cargo test -p shell --test agent_successor_contract_ahcsitc0 --no-run +``` + +`L2`: + +```bash +cargo test -p shell --lib -- --nocapture +cargo test -p shell --test agent_successor_contract_ahcsitc0 --no-run +``` + +`L3`: + +```bash +cargo test -p shell --lib -- --nocapture +cargo test -p shell --test repl_world_first_routing_v1 --no-run +``` + +`L4`: + +```bash +cargo test -p shell --test agent_public_control_surface_v1 -- --nocapture +cargo test -p shell --test agent_successor_contract_ahcsitc0 -- --nocapture +cargo test -p shell --test repl_world_first_routing_v1 -- --nocapture +rg -n "start|turn|reattach|session_posture|Linux-first|-c|world-root" /Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md /Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/README.md +``` + +### Window A acceptance requirements + +`L1` must prove: + +- exact one-of prompt-source parse rules for `start` and `turn` +- canonical `reattach` public surface +- canonical `action` strings in JSON +- no public reinterpretation of `-c` + +`L2` must prove: + +- exact `(orchestration_session_id, backend_id)` resolution +- `backend_not_in_session` and `ambiguous_backend_slot` rejection +- live vs detached vs terminal posture inputs +- non-canonical handle rejection remains strict +- non-Linux world-sensitive follow-up remains fail-closed + +`L3` must prove: + +- one shared prompt-source loader +- no synthetic bootstrap prompt enters public input +- extracted host/world submit helpers keep REPL behavior green +- helper-owned stream bridge emits incremental output +- JSON mode emits only NDJSON after stream start +- post-stream bridge failures surface as terminal in-stream `failed` + +### L4 closeout acceptance + +`L4` must prove: + +- `agent_public_control_surface_v1.rs` directly covers `start`, `turn`, `reattach`, `fork`, and `stop` for the shipped public surface +- at least one streaming assertion exists for incremental prompt-taking output +- `session_posture` outcomes are asserted explicitly +- noncanonical handle rejection remains covered +- `substrate -c` non-regression remains covered +- REPL world-first routing non-regression remains green +- `AGENT_ORCHESTRATION_GAP_MATRIX.md` says only what shipped: + - public non-interactive caller surface landed + - Linux-first world follow-up posture + - non-Linux fail-closed posture + - no claim of world-root public start + - no claim of macOS or Windows parity +- `llm-last-mile/README.md` is updated so `PLAN-20` and `ORCH_PLAN-20` are discoverable +- the QA artifact exists and includes: + - root `start` happy path with `--prompt` + - root `start` stdin path with `--prompt-file -` + - exact host `turn` + - exact Linux world `turn` + - detached recoverable `turn` + - `reattach` + - wrong-handle rejection + - `-c` spot check + +### Final validation wall + +Run in this exact order on `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate`: + +```bash +cargo fmt --all -- --check +cargo clippy --workspace --all-targets -- -D warnings +cargo test -p shell --lib -- --nocapture +cargo test -p shell --test agent_public_control_surface_v1 -- --nocapture +cargo test -p shell --test agent_successor_contract_ahcsitc0 -- --nocapture +cargo test -p shell --test repl_world_first_routing_v1 -- --nocapture +``` + +If world-sensitive follow-up logic changed materially, then run: + +```bash +cargo test -p world-agent --test member_runtime_world_placement_v1 -- --nocapture +``` + +Operator-surface checks after test green: + +```bash +substrate agent doctor --json +substrate shim doctor --json +substrate world doctor --json +substrate health --json +``` + +Manual spot checks after operator-surface checks: + +```bash +substrate agent start --backend --prompt "hello" --json +printf 'hello from stdin\n' | substrate agent start --backend --prompt-file - --json +substrate agent turn --session --backend --prompt "next" --json +substrate agent reattach --session --json +substrate -c "echo hi" +``` + +Manual check expectations: + +- `start` emits `accepted` before completion and ends with `completed` carrying `session_posture` +- stdin prompt is consumed once and only once +- `turn` targets the exact `(session, backend)` pair and does not guess +- `reattach` is canonical in output even if a hidden `resume` alias still exists +- `-c` remains shell-wrap behavior with no agent prompt interpretation + +### Stop conditions and quarantine / retry rules + +1. Stop the run if `-c` semantics change or become ambiguous. +2. Stop the run if canonical `reattach` naming is not representable without public `resume` drift. +3. Stop the run if exact `(session, backend)` targeting cannot be implemented without fuzzy fallback. +4. Stop the run if the extracted shared submit seam regresses REPL host/world routing. +5. Stop the run if non-Linux world follow-up stops failing closed. +6. Stop the run if any prompt-taking path injects `runtime_bootstrap_prompt`. +7. Quarantine any lane that touches files outside ownership, invents undocumented JSON fields, accepts noncanonical handles, or buffers full output before completion. +8. Retry budget is one targeted retry per quarantined worker lane after the parent writes explicit correction notes into that task root. +9. If the same lane fails twice on contract fidelity, the parent closes the run as blocked instead of launching a third attempt. + +## Assumptions + +1. The existing retained follow-up machinery in `async_repl.rs` and `world-agent` is the source of truth and is reused rather than rewritten. +2. A narrow signature-first scaffold is enough to let the three code lanes proceed in parallel without speculative API drift. +3. `agent_public_control_surface_v1.rs`, `agent_successor_contract_ahcsitc0.rs`, and `repl_world_first_routing_v1.rs` remain the right primary proof surfaces for this slice. +4. The public `start` implementation may remain host-first for root session creation while still allowing Linux world follow-up through retained-member routing. +5. Any temporary hidden `resume` alias is compatibility-only and does not change the documented surface or emitted `action` values. +6. The parent can truthfully classify `session_posture` at command completion without promising future reachability. +7. Repo-truth docs update only after merged code and tests are green. The gap matrix must not overclaim world-root start, macOS parity, or Windows parity. diff --git a/llm-last-mile/ORCH_PLAN-21.md b/llm-last-mile/ORCH_PLAN-21.md new file mode 100644 index 000000000..cb9b9770d --- /dev/null +++ b/llm-last-mile/ORCH_PLAN-21.md @@ -0,0 +1,914 @@ +# ORCH_PLAN-21: Execute PLAN-21 Through A Parent-Frozen Lima Parity Contract And One Honest Two-Lane Opening + +Branch: `feat/macos-lima-shared-owner-member-runtime-parity` +Plan source: [PLAN-21.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-21.md) +Style reference: [ORCH_PLAN-18.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/ORCH_PLAN-18.md) +Style reference: [ORCH_PLAN-19.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/ORCH_PLAN-19.md) +Structure reference: user M26 orchestration-plan example +Gap matrix anchor: [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md:171) +Execution type: fresh orchestration plan, Linux-contract parity rollout for macOS/Lima, parent-frozen contract, parent-only gates and integrations, docs/smoke closeout only after merged code truth +Worker model: GPT-5.4 workers with `reasoning_effort=high` +Max concurrent code workers before integration: `2` + +## Summary + +- Authoritative branch: `feat/macos-lima-shared-owner-member-runtime-parity` on `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate`. +- Critical path stays local to the parent for contract freeze, launch gates, both integration windows, the validation wall, and final closeout. +- Worktree root: `/Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-21/`. +- Worker branches: + - `codex/feat-session-centric-state-store-m21-backend-contract-parity` + - `codex/feat-session-centric-state-store-m21-shared-owner-bootstrap-parity` + - `codex/feat-session-centric-state-store-m21-shell-member-runtime-parity` + - `codex/feat-session-centric-state-store-m21-macos-validation-docs-closeout` +- Worker policy: every child worker uses GPT-5.4 with `reasoning_effort=high`. +- Parent-owned run-state surfaces live under `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/plan-21/`. +- `.runs/plan-21/*` and `.runs/task-m21-*/**` are run artifacts for orchestration state and evidence, not assumed git-tracked deliverables. +- The only honest opening parallelism is `L1` plus `L2`; member-runtime parity waits for merged Window A truth; smoke/docs/gap-matrix closeout waits for merged Window B truth. + +## Assumptions + +1. [PLAN-21.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-21.md) remains the authoritative dependency graph for this run. +2. The authoritative integration checkout remains the current workspace checkout on `feat/macos-lima-shared-owner-member-runtime-parity`. +3. Linux already defines the correct shared-owner and retained member-runtime contract. macOS/Lima must converge to that contract rather than define a sibling contract. +4. The supported macOS posture is specifically the Lima-forwarded guest path. Unsupported non-Lima postures may continue to reject explicitly. +5. The exact identity tuple is frozen and must not change: `orchestration_session_id`, `orchestrator_participant_id`, `backend_id`, `world_id`, `world_generation`. +6. The guest remains authoritative for world-sensitive member binding validation and cancel ownership. +7. Shared-owner bootstrap parity and backend contract widening can begin in parallel because they touch disjoint primary seams when ownership is enforced strictly. +8. Shell member-runtime parity depends on both merged bootstrap truth and merged backend field-preservation truth. +9. Scripts, docs, and gap-matrix truth must wait for merged runtime behavior. Early documentation edits are not honest. +10. The parent agent is the only integrator, the only approval authority, and the only writer of run-state artifacts. + +## Execution Shape + +This document is the execution controller for `PLAN-21`, not a restatement of it. + +The run shape is frozen up front: + +1. `task/m21-p1-parent-contract-freeze-and-run-init` is parent-only and freezes the Linux-source-of-truth contract, explicit shared-owner contract, exact identity checks, lane ownership, merge order, retry budget, blocked-run posture, and validation wall. +2. `task/m21-g1-implementation-window-a-launch-gate` is parent-only and is the only gate that may launch the first implementation window. +3. The only honest initial parallel window is exactly two lanes: + - `task/m21-l1-backend-contract-parity` + - `task/m21-l2-shared-owner-bootstrap-parity` +4. `task/m21-g2-window-a-integration-gate` and `task/m21-p2-parent-window-a-integration` are parent-only. +5. `task/m21-g3-member-runtime-launch-gate` starts only after merged Window A truth exists. +6. `task/m21-l3-shell-member-runtime-parity` runs alone on the accepted post-`p2` tree. +7. `task/m21-g4-member-runtime-integration-gate` and `task/m21-p3-parent-member-runtime-integration` are parent-only. +8. `task/m21-g5-closeout-launch-gate` starts only after merged member-runtime truth exists. +9. `task/m21-l4-macos-validation-docs-closeout` runs alone on the accepted post-`p3` tree. +10. `task/m21-g6-validation-wall-gate` is parent-only and permits one final validation wall. +11. `task/m21-p4-parent-validation-wall` runs the exact command wall. +12. `task/m21-p5-parent-closeout-phase` writes terminal artifacts and closes the run. + +Canonical task IDs: + +- `task/m21-p1-parent-contract-freeze-and-run-init` +- `task/m21-g1-implementation-window-a-launch-gate` +- `task/m21-l1-backend-contract-parity` +- `task/m21-l2-shared-owner-bootstrap-parity` +- `task/m21-g2-window-a-integration-gate` +- `task/m21-p2-parent-window-a-integration` +- `task/m21-g3-member-runtime-launch-gate` +- `task/m21-l3-shell-member-runtime-parity` +- `task/m21-g4-member-runtime-integration-gate` +- `task/m21-p3-parent-member-runtime-integration` +- `task/m21-g5-closeout-launch-gate` +- `task/m21-l4-macos-validation-docs-closeout` +- `task/m21-g6-validation-wall-gate` +- `task/m21-p4-parent-validation-wall` +- `task/m21-p5-parent-closeout-phase` + +## Hard Guards + +These are run-stopping invariants, not preferences: + +1. Linux remains the source-of-truth orchestration contract. +2. Shared-world ownership stays explicit through `SharedWorldOwnerSpec` and `SharedWorldBindingSnapshot`. +3. macOS must stop rejecting the supported shared-owner path before guest bootstrap. +4. World-scoped member launch, targeted follow-up, and cancel must fail closed if the forwarded guest contract cannot be established. +5. No host-local fallback is allowed for world-scoped member runtime ownership. +6. Exact identity checks stay unchanged for `orchestration_session_id`, `orchestrator_participant_id`, `backend_id`, `world_id`, and `world_generation`. +7. Targeted follow-up keeps using `/v1/member_turn/stream`. +8. Cancel stays guest-owned through `/v1/execute/cancel`. +9. Replacement stays shell-owned and fail-closed through `ReplaceExpectedGeneration`. +10. Backend-level execution and handles must stop silently dropping Linux orchestration fields. +11. No new orchestration model, daemon, transport family, or mac-native authority service is authorized. +12. No worker may weaken validation by duplicating proof validators or relaxing guest-side binding rules. +13. Docs and gap-matrix work are late-only. They may not start before merged code truth exists. +14. The parent is the only integrator, the only approval authority, and the only writer of `.runs/plan-21/**`. +15. The authoritative integration checkout remains `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate` on `feat/macos-lima-shared-owner-member-runtime-parity`. + +Stop the run, write `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/plan-21/blocked.json`, and do not advance if any of these occur: + +1. A lane requires a second orchestration model instead of Linux-parity widening. +2. A lane requires host-local fallback for world-scoped member runtime. +3. A lane requires changing endpoint ownership away from `/v1/member_turn/stream` or `/v1/execute/cancel`. +4. A lane requires relaxing any frozen identity check. +5. A lane requires parent invention of hybrid semantics across conflicting worker assumptions. +6. A worker touches files outside its frozen ownership surface. +7. `L3` starts before `p2` is accepted. +8. `L4` starts before `p3` is accepted. +9. The final validation wall cannot prove attach/create, replacement, lazy launch, targeted follow-up, cancel, backend field preservation, and doc truth on the same merged tree. +10. The gap matrix or `docs/WORLD.md` would need to overclaim beyond what the final validation wall proves. + +### Blocked-Run Record Contract + +`blocked.json` is parent-written only, exactly once, at the moment the parent decides the run cannot advance. + +Required fields in `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/plan-21/blocked.json`: + +- `run_id` +- `branch` +- `plan_source` +- `timestamp` +- `current_task_id` +- `gate_state` +- `stop_condition_id` +- `summary` +- `blocking_files` +- `worker_lane` +- `accepted_sentinels` +- `rejected_or_quarantined_outputs` +- `next_required_parent_action` + +## Fresh Worktrees And Branches + +Fresh worktree root: + +- `/Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-21/` + +Worker worktrees: + +- `/Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-21/backend-contract-parity` +- `/Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-21/shared-owner-bootstrap-parity` +- `/Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-21/shell-member-runtime-parity` +- `/Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-21/macos-validation-docs-closeout` + +Worker branches: + +- `codex/feat-session-centric-state-store-m21-backend-contract-parity` +- `codex/feat-session-centric-state-store-m21-shared-owner-bootstrap-parity` +- `codex/feat-session-centric-state-store-m21-shell-member-runtime-parity` +- `codex/feat-session-centric-state-store-m21-macos-validation-docs-closeout` + +Exact setup commands: + +```bash +mkdir -p /Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-21 +git worktree add /Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-21/backend-contract-parity -b codex/feat-session-centric-state-store-m21-backend-contract-parity feat/macos-lima-shared-owner-member-runtime-parity +git worktree add /Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-21/shared-owner-bootstrap-parity -b codex/feat-session-centric-state-store-m21-shared-owner-bootstrap-parity feat/macos-lima-shared-owner-member-runtime-parity +git worktree add /Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-21/shell-member-runtime-parity -b codex/feat-session-centric-state-store-m21-shell-member-runtime-parity feat/macos-lima-shared-owner-member-runtime-parity +git worktree add /Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-21/macos-validation-docs-closeout -b codex/feat-session-centric-state-store-m21-macos-validation-docs-closeout feat/macos-lima-shared-owner-member-runtime-parity +``` + +No separate parent integration worktree is introduced. The parent integrates only on `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate`. + +## Parent-Owned Run-State Surface + +Canonical parent-owned state under `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/plan-21/`: + +- `run-state.json` +- `tasks.json` +- `session-log.md` +- `contract-freeze.json` +- `lane-ownership.json` +- `merge-order.json` +- `validation-wall.md` +- `blocked.json` on failure only +- `quarantine/` +- `sentinels/` + +These are orchestration artifacts and evidence surfaces. They are not assumed repository deliverables and are not part of the feature’s product contract unless a later human explicitly chooses to commit selected outputs. + +Required per-task artifact roots: + +- `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/task-m21-p1-parent-contract-freeze-and-run-init/` +- `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/task-m21-g1-implementation-window-a-launch-gate/` +- `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/task-m21-l1-backend-contract-parity/` +- `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/task-m21-l2-shared-owner-bootstrap-parity/` +- `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/task-m21-g2-window-a-integration-gate/` +- `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/task-m21-p2-parent-window-a-integration/` +- `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/task-m21-g3-member-runtime-launch-gate/` +- `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/task-m21-l3-shell-member-runtime-parity/` +- `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/task-m21-g4-member-runtime-integration-gate/` +- `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/task-m21-p3-parent-member-runtime-integration/` +- `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/task-m21-g5-closeout-launch-gate/` +- `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/task-m21-l4-macos-validation-docs-closeout/` +- `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/task-m21-g6-validation-wall-gate/` +- `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/task-m21-p4-parent-validation-wall/` +- `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/task-m21-p5-parent-closeout-phase/` + +Each task directory must contain: + +- `task.json` +- `commands.txt` +- `summary.md` + +Each gate task directory must also contain: + +- `gate-checklist.md` +- `gate-result.json` + +Each worker task directory must also contain: + +- `worker-report.md` +- `worker-output.patch` +- `evidence-manifest.json` + +Required sentinels: + +- `.runs/plan-21/sentinels/task-m21-p1-parent-contract-freeze-and-run-init.ok` +- `.runs/plan-21/sentinels/task-m21-g1-implementation-window-a-launch-gate.ok` +- `.runs/plan-21/sentinels/task-m21-l1-backend-contract-parity.ok` +- `.runs/plan-21/sentinels/task-m21-l2-shared-owner-bootstrap-parity.ok` +- `.runs/plan-21/sentinels/task-m21-g2-window-a-integration-gate.ok` +- `.runs/plan-21/sentinels/task-m21-p2-parent-window-a-integration.ok` +- `.runs/plan-21/sentinels/task-m21-g3-member-runtime-launch-gate.ok` +- `.runs/plan-21/sentinels/task-m21-l3-shell-member-runtime-parity.ok` +- `.runs/plan-21/sentinels/task-m21-g4-member-runtime-integration-gate.ok` +- `.runs/plan-21/sentinels/task-m21-p3-parent-member-runtime-integration.ok` +- `.runs/plan-21/sentinels/task-m21-g5-closeout-launch-gate.ok` +- `.runs/plan-21/sentinels/task-m21-l4-macos-validation-docs-closeout.ok` +- `.runs/plan-21/sentinels/task-m21-g6-validation-wall-gate.ok` +- `.runs/plan-21/sentinels/task-m21-p4-parent-validation-wall.ok` +- `.runs/plan-21/sentinels/task-m21-p5-parent-closeout-phase.ok` + +`merge-order.json` is frozen during `p1` and must record: + +- `integration_order: ["task/m21-l1-backend-contract-parity", "task/m21-l2-shared-owner-bootstrap-parity", "task/m21-l3-shell-member-runtime-parity", "task/m21-l4-macos-validation-docs-closeout"]` +- `l2_acceptance_basis: "accepted_l1_tree_only"` +- `l3_acceptance_basis: "accepted_p2_tree_only"` +- `l4_acceptance_basis: "accepted_p3_tree_only"` +- `replay_required_before_acceptance: true` +- `quarantine_on_branch_local_assumption: true` + +## PLAN-21 Workstream Mapping + +| PLAN-21 workstream | Orchestration tasks | Why this mapping is exact | +| --- | --- | --- | +| Persistent-session shared-owner parity | `task/m21-l2-shared-owner-bootstrap-parity`, `task/m21-g2-window-a-integration-gate`, `task/m21-p2-parent-window-a-integration` | This lane owns host-side pre-bootstrap rejection removal, forwarded attach/create, replacement proof, and shell-side fail-closed shared-world echo validation. | +| Backend contract parity | `task/m21-l1-backend-contract-parity`, `task/m21-g2-window-a-integration-gate`, `task/m21-p2-parent-window-a-integration` | This lane owns additive `ExecRequest` widening, backend preservation, Lima field propagation, and authoritative `shared_binding` surfacing. | +| Shell member-runtime parity on macOS | `task/m21-g3-member-runtime-launch-gate`, `task/m21-l3-shell-member-runtime-parity`, `task/m21-g4-member-runtime-integration-gate`, `task/m21-p3-parent-member-runtime-integration` | This window owns lazy launch, targeted follow-up reuse, and cancel on supported macOS/Lima, on top of merged backend and bootstrap truth. | +| Validation and documentation | `task/m21-g5-closeout-launch-gate`, `task/m21-l4-macos-validation-docs-closeout`, `task/m21-g6-validation-wall-gate`, `task/m21-p4-parent-validation-wall`, `task/m21-p5-parent-closeout-phase` | This closeout window owns smoke harness, macOS docs truth, gap-matrix truth, full validation wall, and final run closure. | + +## Concurrency And Merge Order + +Concurrency rules: + +1. Worker cap is exactly `2` until `g2` completes. +2. `p1` must finish before any worker starts. +3. `g1` must be green before any worker starts. +4. The only honest initial parallel window is `L1` plus `L2`. +5. `L3` waits for accepted and integrated `L1` plus `L2`. +6. `L4` waits for accepted and integrated `L3`. +7. The validation wall runs exactly once on the final merged tree. +8. No third concurrent code lane is authorized because docs/smoke must describe integrated truth, not branch-local assumptions. + +Why `L1` integrates before `L2`: + +1. `L1` widens the upstream backend contract at [crates/world-api/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-api/src/lib.rs), [crates/world-agent/src/service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs), and [crates/world-mac-lima/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/lib.rs). +2. `L3` depends on authoritative backend field preservation and authoritative Lima `shared_binding` surfacing. That truth must be frozen before shell follow-up parity starts. +3. `L2` is a narrower shell consumer of the forwarded bootstrap path and can be replayed cleanly on the accepted backend seam. +4. Integrating `L2` first risks proving shell behavior against branch-local Lima semantics that still erase orchestration fields underneath it. + +## Kickoff Initialization Order + +The parent initializes the run in this exact order: + +1. Create `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/plan-21/`, `.runs/plan-21/sentinels/`, `.runs/plan-21/quarantine/`, and every `.runs/task-m21-*/` directory. +2. Create `task.json`, `commands.txt`, `summary.md`, and `artifacts/` in every task directory. +3. Create `gate-checklist.md` and `gate-result.json` in every gate task directory. +4. Create placeholder `worker-report.md`, `worker-output.patch`, and `evidence-manifest.json` in every worker task directory. +5. Write `tasks.json` as the canonical launch queue and execution ledger. +6. Write `run-state.json` with `current_phase: "kickoff"`, `worker_cap: 2`, every task in `pending`, and empty accepted, rejected, quarantined, and blocked arrays. +7. Write `contract-freeze.json`, `lane-ownership.json`, `merge-order.json`, and `validation-wall.md`. +8. Freeze the exact `ExecRequest` constructor callsite list with parent search and record it in `lane-ownership.json`. `L1` may touch only that frozen constructor list in addition to its primary files. +9. Review the frozen hotspots: + - [crates/world-api/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-api/src/lib.rs) + - [crates/world-mac-lima/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/lib.rs) + - [crates/world-agent/src/service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs) + - [crates/shell/src/execution/platform_world/mod.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/platform_world/mod.rs) + - [crates/shell/src/execution/routing/dispatch/world_persistent_session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_persistent_session.rs) + - [crates/shell/src/execution/repl_persistent_session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/repl_persistent_session.rs) + - [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) + - [crates/shell/src/execution/routing/dispatch/world_ops.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs) +10. No pre-launch production scaffold is authorized. If either initial lane needs parent edits before it can start, block the run rather than invent a hybrid seed state. +11. Seed `L1` and `L2` worktrees from the exact same post-`p1` tree. +12. Write `session-log.md` with kickoff timestamp, branch, worktree roots, worker cap, and the explicit statement that the only honest initial parallel window is `L1` plus `L2`. + +## Task Ledger + +| Task ID | Owner | Depends on | Worktree / branch | Deliverable | +| --- | --- | --- | --- | --- | +| `task/m21-p1-parent-contract-freeze-and-run-init` | parent | — | authoritative checkout | frozen contract and seeded worker basis | +| `task/m21-g1-implementation-window-a-launch-gate` | parent | `p1` | authoritative checkout | launch approval for `L1` and `L2` | +| `task/m21-l1-backend-contract-parity` | worker | `g1` | `backend-contract-parity` / `codex/feat-session-centric-state-store-m21-backend-contract-parity` | additive backend contract widening and Lima field preservation | +| `task/m21-l2-shared-owner-bootstrap-parity` | worker | `g1` | `shared-owner-bootstrap-parity` / `codex/feat-session-centric-state-store-m21-shared-owner-bootstrap-parity` | macOS shared-owner bootstrap and replacement parity | +| `task/m21-g2-window-a-integration-gate` | parent | `l1`, `l2` | authoritative checkout | acceptance, rejection, or quarantine for Window A | +| `task/m21-p2-parent-window-a-integration` | parent | `g2` | authoritative checkout | merged backend parity then bootstrap parity | +| `task/m21-g3-member-runtime-launch-gate` | parent | `p2` | authoritative checkout | launch approval for `L3` | +| `task/m21-l3-shell-member-runtime-parity` | worker | `g3` | `shell-member-runtime-parity` / `codex/feat-session-centric-state-store-m21-shell-member-runtime-parity` | macOS lazy launch, targeted follow-up, and cancel parity | +| `task/m21-g4-member-runtime-integration-gate` | parent | `l3` | authoritative checkout | acceptance, rejection, or quarantine for `L3` | +| `task/m21-p3-parent-member-runtime-integration` | parent | `g4` | authoritative checkout | merged member-runtime parity truth | +| `task/m21-g5-closeout-launch-gate` | parent | `p3` | authoritative checkout | launch approval for `L4` | +| `task/m21-l4-macos-validation-docs-closeout` | worker | `g5` | `macos-validation-docs-closeout` / `codex/feat-session-centric-state-store-m21-macos-validation-docs-closeout` | smoke harness, docs, and gap-matrix truth | +| `task/m21-g6-validation-wall-gate` | parent | `l4` | authoritative checkout | permission to run the final validation wall | +| `task/m21-p4-parent-validation-wall` | parent | `g6` | authoritative checkout | exact PLAN-21 validation wall results | +| `task/m21-p5-parent-closeout-phase` | parent | `p4` | authoritative checkout | terminal run-state, closeout, and artifact audit | + +## Lane Ownership By File Set + +| Lane | Allowed files | Forbidden touch surfaces | +| --- | --- | --- | +| `L1` / backend contract parity | [crates/world-api/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-api/src/lib.rs), [crates/world-mac-lima/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/lib.rs), [crates/world-agent/src/service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs), explicit `ExecRequest` constructor callsites frozen by parent into `lane-ownership.json`, adjacent `world-agent`, `world-api`, and `world-mac-lima` tests only if required by this contract widening | `async_repl.rs`, `world_persistent_session.rs`, `repl_persistent_session.rs`, `world_ops.rs`, `scripts/mac/**`, `docs/**`, `AGENT_ORCHESTRATION_GAP_MATRIX.md`, `.runs/**` | +| `L2` / shared-owner bootstrap parity | [crates/shell/src/execution/platform_world/mod.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/platform_world/mod.rs), [crates/shell/src/execution/routing/dispatch/world_persistent_session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_persistent_session.rs), [crates/shell/src/execution/repl_persistent_session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/repl_persistent_session.rs), shell tests directly proving these seams only | `world-api`, `world-agent`, `world-mac-lima`, `async_repl.rs`, `world_ops.rs`, `scripts/mac/**`, `docs/**`, `.runs/**` | +| `L3` / shell member-runtime parity | [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs), [crates/shell/src/execution/routing/dispatch/world_ops.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs), [crates/shell/tests/repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs), [crates/shell/tests/support/repl_world_agent.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/support/repl_world_agent.rs) | `platform_world/mod.rs`, `world_persistent_session.rs`, `repl_persistent_session.rs`, `world-api`, `world-agent`, `world-mac-lima`, `scripts/mac/**`, `docs/**`, `.runs/**` | +| `L4` / macOS validation docs closeout | [scripts/mac/lima-warm.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/lima-warm.sh), [scripts/mac/smoke.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/smoke.sh), `scripts/mac/orchestration-smoke.sh`, [docs/WORLD.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/WORLD.md), [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md) | every production Rust file, `.runs/**`, any doc claim beyond validated parity scope | + +## Worker Interfaces + +Every worker prompt must include: + +1. task ID +2. attempt number +3. worktree path +4. branch +5. allowed files +6. forbidden files +7. frozen contract clauses relevant to that lane +8. required commands +9. retry budget +10. required return artifacts +11. sentinel path + +Every worker return must include: + +1. changed files list +2. commands run with exit codes +3. explicit attempt classification: `clean`, `retryable`, or `blocked` +4. unresolved assumptions or blockers +5. `worker-output.patch` +6. `worker-report.md` +7. `evidence-manifest.json` + +## Parent Phases And Worker Packets + +### `task/m21-p1-parent-contract-freeze-and-run-init` + +Owner: + +- parent only + +Scope: + +1. Freeze the Linux-source-of-truth parity contract in `contract-freeze.json`. +2. Freeze explicit ownership invariants for `SharedWorldOwnerSpec`, `SharedWorldBindingSnapshot`, `ReplaceExpectedGeneration`, `/v1/member_turn/stream`, and `/v1/execute/cancel`. +3. Freeze exact file ownership, merge order, retry budget, stop conditions, and validation wall. +4. Freeze the exact `ExecRequest` constructor list that `L1` may edit. +5. Seed `L1` and `L2` from the same post-`p1` tree. + +Command gates: + +```bash +cargo test -p world-api --no-run +cargo test -p world-mac-lima --no-run +cargo test -p shell --no-run +``` + +Acceptance: + +1. `contract-freeze.json`, `lane-ownership.json`, `merge-order.json`, `tasks.json`, and `run-state.json` exist. +2. The freeze artifact explicitly records that this is parity work, not a new orchestration model. +3. The frozen contract records no host-local fallback, unchanged exact identity checks, unchanged endpoints, and parent-only replacement ownership. +4. The parent writes `.runs/plan-21/sentinels/task-m21-p1-parent-contract-freeze-and-run-init.ok`. + +### `task/m21-g1-implementation-window-a-launch-gate` + +Owner: + +- parent only + +Checks: + +1. `p1` is accepted. +2. `L1` and `L2` were seeded from the exact same post-`p1` tree. +3. `L1` prompt explicitly forbids shell follow-up or docs work. +4. `L2` prompt explicitly forbids backend contract edits and member-runtime work. +5. Both prompts repeat the frozen prohibitions on host-local fallback, identity relaxation, endpoint changes, and orchestration-model widening. + +Acceptance: + +1. No worker starts before this gate is green. +2. The parent writes `.runs/plan-21/sentinels/task-m21-g1-implementation-window-a-launch-gate.ok`. + +### `task/m21-l1-backend-contract-parity` + +Owner: + +- single worker on `codex/feat-session-centric-state-store-m21-backend-contract-parity` + +Packet fields: + +- Owned files: + - [crates/world-api/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-api/src/lib.rs) + - [crates/world-mac-lima/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/lib.rs) + - [crates/world-agent/src/service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs) + - Frozen `ExecRequest` constructor callsites recorded by parent in `lane-ownership.json` + - Adjacent tests only when required by this exact widening +- Forbidden touch surfaces: + - [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) + - [crates/shell/src/execution/routing/dispatch/world_ops.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs) + - [crates/shell/src/execution/platform_world/mod.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/platform_world/mod.rs) + - `scripts/mac/**` + - `docs/**` + - `.runs/**` +- Exact scope: + 1. Add optional `shared_world` and `member_dispatch` fields to `ExecRequest`. + 2. Update every frozen `ExecRequest` constructor to be explicit about absence or presence. + 3. Pass those fields through [crates/world-agent/src/service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs) when backend execution is orchestration-sensitive. + 4. Make [crates/world-mac-lima/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/lib.rs) preserve those fields instead of zeroing them. + 5. Make Lima `WorldHandle.shared_binding` authoritative in owner mode. + 6. Keep Linux and Windows compile-clean without changing their semantics. +- Exact required commands: + +```bash +cargo test -p world-agent member_runtime -- --nocapture +cargo test -p world-api -- --nocapture +cargo test -p world-mac-lima -- --nocapture +``` + +- Exact acceptance: + 1. The lane touches only its owned files and frozen constructor callsites. + 2. `ExecRequest` widening is additive and optional. + 3. `world-agent` forwards orchestration fields into backend execution when relevant. + 4. Lima no longer zeros `shared_world` or `member_dispatch`. + 5. Owner-mode Lima handles surface authoritative `shared_binding`. + 6. Linux and Windows remain compile-clean without semantic widening. + 7. The worker writes `.runs/plan-21/sentinels/task-m21-l1-backend-contract-parity.ok`. + +### `task/m21-l2-shared-owner-bootstrap-parity` + +Owner: + +- single worker on `codex/feat-session-centric-state-store-m21-shared-owner-bootstrap-parity` + +Packet fields: + +- Owned files: + - [crates/shell/src/execution/platform_world/mod.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/platform_world/mod.rs) + - [crates/shell/src/execution/routing/dispatch/world_persistent_session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_persistent_session.rs) + - [crates/shell/src/execution/repl_persistent_session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/repl_persistent_session.rs) + - Shell tests directly proving these seams only +- Forbidden touch surfaces: + - `world-api` + - `world-agent` + - `world-mac-lima` + - [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) + - [crates/shell/src/execution/routing/dispatch/world_ops.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs) + - `scripts/mac/**` + - `docs/**` + - `.runs/**` +- Exact scope: + 1. Replace blanket non-Linux shared-owner rejection with capability-aware forwarding that allows the Lima-backed path and still rejects unsupported platforms. + 2. Keep one forwarded persistent-session request path. + 3. Keep [validate_shared_world_echo(...)](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/repl_persistent_session.rs:308) as the only shell-side proof validator. + 4. Preserve replacement semantics through `ReplaceExpectedGeneration`. + 5. Add or update shell tests for attach/create, replacement advancement, and invalid proof rejection on the macOS forwarded path. +- Exact required commands: + +```bash +cargo test -p shell --lib -- --nocapture +cargo test -p shell --test repl_world_first_routing_v1 --no-run +``` + +- Exact acceptance: + 1. The lane touches only its owned files. + 2. macOS shared-owner attach/create reaches the forwarded guest path. + 3. macOS replacement still requires generation advancement. + 4. Missing, malformed, stale, or inactive proof still fails closed. + 5. Unsupported non-Lima paths still reject explicitly. + 6. The worker writes `.runs/plan-21/sentinels/task-m21-l2-shared-owner-bootstrap-parity.ok`. + +### `task/m21-g2-window-a-integration-gate` + +Owner: + +- parent only + +Checks: + +1. `L1` and `L2` both returned patch, report, command transcript, and evidence manifest. +2. Every touched file is inside the lane’s ownership boundary. +3. `L1` landed additive backend contract widening without semantic drift. +4. `L2` landed forwarded bootstrap parity without reopening member-runtime or backend field preservation work. +5. `L2` is replayed only against the accepted `L1` tree. + +Quarantine and retry behavior: + +1. If `L1` broadened beyond additive backend seam work, quarantine `L1` immediately. +2. If `L2` depended on branch-local backend semantics instead of accepted `L1` truth, quarantine `L2`. +3. Retry budget remains `1` per lane and is available only for lane-local defects inside owned files. +4. If either lane violated ownership or frozen endpoint/identity constraints, mark non-retryable and quarantine rather than redrive. + +Acceptance: + +1. Accepted, rejected, or quarantined status for both lanes is recorded in `run-state.json`. +2. The parent writes `.runs/plan-21/sentinels/task-m21-g2-window-a-integration-gate.ok`. + +### `task/m21-p2-parent-window-a-integration` + +Owner: + +- parent only + +Scope: + +1. Integrate accepted `L1` output first. +2. Re-run `L1` command gates on the authoritative checkout. +3. Replay `L2` on top of accepted `L1`. If `L2` assumed different backend field semantics, quarantine `L2` instead of hand-editing around the mismatch. +4. Integrate accepted `L2` output second. +5. Freeze the merged backend-plus-bootstrap truth before any member-runtime work starts. + +Command gates: + +```bash +cargo test -p world-agent member_runtime -- --nocapture +cargo test -p world-api -- --nocapture +cargo test -p world-mac-lima -- --nocapture +cargo test -p shell --no-run +``` + +Acceptance: + +1. The parent remains the sole integrator. +2. The authoritative tree now preserves backend orchestration fields and allows supported macOS shared-owner bootstrap. +3. No hybrid contract was invented during integration. +4. The parent writes `.runs/plan-21/sentinels/task-m21-p2-parent-window-a-integration.ok`. + +### `task/m21-g3-member-runtime-launch-gate` + +Owner: + +- parent only + +Checks: + +1. `p2` is green. +2. `L3` worktree is seeded from the exact accepted post-`p2` tree. +3. The prompt names only `async_repl.rs`, `world_ops.rs`, and the frozen shell test files. +4. The prompt explicitly forbids host-local fallback, backend contract edits, and docs work. + +Acceptance: + +1. No `L3` worker starts before this gate is green. +2. The parent writes `.runs/plan-21/sentinels/task-m21-g3-member-runtime-launch-gate.ok`. + +### `task/m21-l3-shell-member-runtime-parity` + +Owner: + +- single worker on `codex/feat-session-centric-state-store-m21-shell-member-runtime-parity` + +Packet fields: + +- Owned files: + - [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) + - [crates/shell/src/execution/routing/dispatch/world_ops.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs) + - [crates/shell/tests/repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs) + - [crates/shell/tests/support/repl_world_agent.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/support/repl_world_agent.rs) +- Forbidden touch surfaces: + - [crates/shell/src/execution/platform_world/mod.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/platform_world/mod.rs) + - [crates/shell/src/execution/routing/dispatch/world_persistent_session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_persistent_session.rs) + - [crates/shell/src/execution/repl_persistent_session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/repl_persistent_session.rs) + - `world-api` + - `world-agent` + - `world-mac-lima` + - `scripts/mac/**` + - `docs/**` + - `.runs/**` +- Exact scope: + 1. Widen shell member-runtime orchestration so supported macOS/Lima no longer hard-errors on non-Linux cfg stubs. + 2. Reuse the existing `member_dispatch` request builder in [world_ops.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs). + 3. Route lazy launch through guest `member_dispatch`. + 4. Route targeted follow-up turns through `/v1/member_turn/stream`. + 5. Route bootstrap and submitted-turn cancel through `/v1/execute/cancel`. + 6. Preserve exact fail-closed behavior for wrong backend, stale generation, wrong world, and missing authoritative binding. + 7. Add or update shell tests proving reuse, mismatch rejection, and cancel routing on the macOS path. +- Exact required commands: + +```bash +cargo test -p shell --lib -- --nocapture +cargo test -p shell --test repl_world_first_routing_v1 -- --nocapture +``` + +- Exact acceptance: + 1. The lane touches only its owned files. + 2. Supported macOS/Lima no longer hits Linux-only hard-error stubs for world-scoped orchestration. + 3. No host-local fallback is introduced. + 4. Follow-up uses `/v1/member_turn/stream` and cancel uses `/v1/execute/cancel`. + 5. Exact identity mismatch paths still fail closed. + 6. The worker writes `.runs/plan-21/sentinels/task-m21-l3-shell-member-runtime-parity.ok`. + +### `task/m21-g4-member-runtime-integration-gate` + +Owner: + +- parent only + +Checks: + +1. `L3` returned patch, report, command transcript, and evidence manifest. +2. Every touched file is inside `L3` ownership. +3. `L3` consumed accepted `p2` truth and did not invent endpoint drift or local fallback. +4. `L3` proof commands are green on the authoritative checkout. + +Quarantine and retry behavior: + +1. If `L3` changed endpoint ownership, introduced host-local fallback, or touched forbidden surfaces, quarantine `L3` immediately as non-retryable. +2. If `L3` failed only for lane-local defects within owned files, retry budget `1` remains available. +3. The parent must not hand-edit around mismatched assumptions. It either redrives the lane or blocks the run. + +Acceptance: + +1. Accepted, rejected, or quarantined status for `L3` is recorded in `run-state.json`. +2. The parent writes `.runs/plan-21/sentinels/task-m21-g4-member-runtime-integration-gate.ok`. + +### `task/m21-p3-parent-member-runtime-integration` + +Owner: + +- parent only + +Scope: + +1. Integrate accepted `L3` output. +2. Re-run `L3` command gates on the authoritative checkout. +3. Freeze the merged shell parity truth before smoke/docs start. + +Command gates: + +```bash +cargo test -p shell -- --nocapture +``` + +Acceptance: + +1. The authoritative tree now supports shared-owner bootstrap, member lazy launch, targeted follow-up, and cancel on the supported macOS/Lima contract. +2. The merged tree still preserves frozen Linux-owned semantics and fail-closed identity checks. +3. The parent writes `.runs/plan-21/sentinels/task-m21-p3-parent-member-runtime-integration.ok`. + +### `task/m21-g5-closeout-launch-gate` + +Owner: + +- parent only + +Checks: + +1. `p3` is green. +2. `L4` worktree is seeded from the exact accepted post-`p3` tree. +3. The prompt names only `scripts/mac/**`, `docs/WORLD.md`, and `AGENT_ORCHESTRATION_GAP_MATRIX.md`. +4. The prompt explicitly forbids reopening production Rust files or overclaiming parity beyond validated support. + +Acceptance: + +1. No `L4` worker starts before this gate is green. +2. The parent writes `.runs/plan-21/sentinels/task-m21-g5-closeout-launch-gate.ok`. + +### `task/m21-l4-macos-validation-docs-closeout` + +Owner: + +- single worker on `codex/feat-session-centric-state-store-m21-macos-validation-docs-closeout` + +Packet fields: + +- Owned files: + - [scripts/mac/lima-warm.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/lima-warm.sh) + - [scripts/mac/smoke.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/smoke.sh) + - `scripts/mac/orchestration-smoke.sh` + - [docs/WORLD.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/WORLD.md) + - [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md) +- Forbidden touch surfaces: + - every production Rust file + - `.runs/**` + - any doc claim beyond validated parity scope +- Exact scope: + 1. Add `scripts/mac/orchestration-smoke.sh`. + 2. Keep [scripts/mac/lima-warm.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/lima-warm.sh) and [scripts/mac/smoke.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/smoke.sh) aligned with the new orchestration smoke path. + 3. Make the smoke harness prove attach/create, replacement, lazy launch, targeted follow-up, cancel, and mismatch rejection. + 4. Update [docs/WORLD.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/WORLD.md) to remove stale pre-bootstrap rejection claims for the supported macOS path. + 5. Update [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md) to reflect validated parity and any remaining out-of-scope gaps only. +- Exact required commands: + +```bash +scripts/mac/lima-warm.sh +scripts/mac/orchestration-smoke.sh +rg -n "shared-owner|member_turn/stream|execute/cancel|macOS|Lima" docs/WORLD.md AGENT_ORCHESTRATION_GAP_MATRIX.md scripts/mac/orchestration-smoke.sh scripts/mac/smoke.sh +``` + +- Exact acceptance: + 1. The lane touches only its owned files. + 2. There is one reproducible macOS/Lima orchestration smoke path. + 3. `docs/WORLD.md` matches the integrated runtime truth. + 4. The gap matrix does not overclaim Windows parity, host-local fallback, or a new orchestration model. + 5. The worker writes `.runs/plan-21/sentinels/task-m21-l4-macos-validation-docs-closeout.ok`. + +### `task/m21-g6-validation-wall-gate` + +Owner: + +- parent only + +Checks: + +1. `L4` returned and is accepted. +2. No quarantined or blocked output remains unresolved. +3. `validation-wall.md` names the exact final command order. +4. The parent can map every frozen completion promise to a command or artifact in the merged tree. + +Quarantine and retry behavior: + +1. If `L4` overclaimed beyond validated behavior, quarantine `L4` instead of editing docs by hand. +2. If `L4` failed only within owned files and stayed within validated scope, retry budget `1` remains available. +3. No validation wall starts until `L4` is either accepted or the run is blocked. + +Acceptance: + +1. The parent writes `.runs/plan-21/sentinels/task-m21-g6-validation-wall-gate.ok`. +2. The validation wall is permitted to run exactly once. + +### `task/m21-p4-parent-validation-wall` + +Owner: + +- parent only + +Scope: + +1. Integrate only accepted `L4` output. +2. Run the exact PLAN-21 validation wall on the authoritative checkout in this order. +3. Record command results and artifact paths. +4. Confirm docs and gap matrix match the validated runtime truth. + +Validation wall commands: + +```bash +cargo test -p world-agent member_runtime +cargo test -p shell +cargo test -p world-api +cargo test -p world-mac-lima +cargo test --workspace -- --nocapture +scripts/mac/lima-warm.sh +scripts/mac/orchestration-smoke.sh +``` + +Required artifacts under `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/task-m21-p4-parent-validation-wall/artifacts/`: + +- `world-agent-member-runtime.txt` +- `shell.txt` +- `world-api.txt` +- `world-mac-lima.txt` +- `workspace.txt` +- `lima-warm.txt` +- `orchestration-smoke.txt` +- `contract-audit.md` + +Acceptance: + +1. All validation wall commands succeed in order. +2. The validation wall proves attach/create, replacement, lazy launch, targeted follow-up, cancel, backend field preservation, and doc truth on the same merged tree. +3. The parent writes `.runs/plan-21/sentinels/task-m21-p4-parent-validation-wall.ok`. + +### `task/m21-p5-parent-closeout-phase` + +Owner: + +- parent only + +Scope: + +1. Confirm all required sentinels exist and `blocked.json` does not. +2. Confirm `tasks.json` and `run-state.json` match actual accepted, rejected, and quarantined outcomes. +3. Confirm no quarantined or blocked output was partially integrated. +4. Write terminal `closeout.md`. +5. Mark the run complete only if the final validated state matches the frozen parity contract. + +Required artifacts under `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/task-m21-p5-parent-closeout-phase/artifacts/`: + +- `closeout.md` +- `final-run-state.json` +- `final-task-ledger.json` +- `final-sentinel-audit.md` + +Acceptance: + +1. `run-state.json` records a successful terminal state. +2. `closeout.md` states exactly what parity landed and what remains out of scope. +3. The parent writes `.runs/plan-21/sentinels/task-m21-p5-parent-closeout-phase.ok`. + +## Quarantine, Retry, And Blocked-Run Posture + +1. Each worker lane has retry budget `1`. +2. Retry is allowed only for lane-local defects inside owned files. +3. Non-retryable violations include host-local fallback, endpoint drift, identity-check relaxation, orchestration-model widening, early docs work, and any cross-lane file touch. +4. If `L1` cannot stay additive at the backend seam, quarantine it immediately. +5. If `L2` can only work by bypassing the existing shared-world echo validator or by editing backend files, quarantine it immediately. +6. If `L3` can only pass by adding host-local fallback or changing `/v1/member_turn/stream` or `/v1/execute/cancel`, quarantine it immediately. +7. If `L4` overclaims parity beyond the smoke harness and final validation wall, quarantine it immediately. +8. The parent never hand-merges a hybrid truth from conflicting worker guesses. + +When a lane is quarantined, the parent must preserve the returned materials in both places: + +1. The original task artifact directory. +2. `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/plan-21/quarantine//`. + +`quarantine//quarantine-reason.json` must record: + +- `task_id` +- `classification` +- `summary` +- `files_touched` +- `frozen_contract_clause_violated` +- `retry_available` +- `next_parent_action` + +## Validation Wall + +The parent may run the final validation wall exactly once, only after `g6` is green. + +The wall is green only when all of these are true: + +1. Backend contract widening is present and additive. +2. Supported macOS shared-owner bootstrap no longer pre-rejects. +3. Supported macOS replacement remains generation-advancing and fail-closed. +4. Supported macOS member lazy launch uses forwarded guest `member_dispatch`. +5. Targeted follow-up uses `/v1/member_turn/stream`. +6. Cancel uses `/v1/execute/cancel`. +7. No supported macOS path silently falls back to host-local member ownership. +8. `docs/WORLD.md` and `AGENT_ORCHESTRATION_GAP_MATRIX.md` match the validated runtime truth. +9. No quarantined or blocked output remains unresolved. + +## Tests And Acceptance + +### Backend Contract + +Acceptance requires all of these to be true: + +- `ExecRequest` carries optional `shared_world` and `member_dispatch` fields without changing absence semantics for existing callers. +- Every frozen `ExecRequest` constructor compiles and is explicit about presence or absence. +- `world-agent` passes orchestration-sensitive fields into backend execution. +- `world-mac-lima` preserves those fields instead of zeroing them. +- Owner-mode Lima returns authoritative `WorldHandle.shared_binding`. +- `cargo test -p world-agent member_runtime`, `cargo test -p world-api`, and `cargo test -p world-mac-lima` are green. + +### Bootstrap Parity + +Acceptance requires all of these to be true: + +- Supported macOS/Lima shared-owner attach/create reaches the forwarded guest path. +- Supported macOS/Lima replacement remains shell-owned and requires generation advancement. +- Missing, malformed, stale, or inactive shared-world proof fails closed. +- Unsupported non-Lima paths still reject explicitly. +- `cargo test -p shell --lib -- --nocapture` and the relevant bootstrap coverage in shell tests are green. + +### Member-Runtime Parity + +Acceptance requires all of these to be true: + +- Supported macOS/Lima no longer hard-errors on the Linux-only member-runtime stub path. +- Lazy launch uses guest `member_dispatch`. +- Targeted follow-up uses `/v1/member_turn/stream`. +- Cancel uses `/v1/execute/cancel`. +- Wrong backend, wrong world, stale generation, and missing authoritative binding all fail closed. +- `cargo test -p shell` proves reuse, mismatch rejection, and cancel routing on the merged tree. + +### Smoke, Docs, And Gap-Matrix Truth + +Acceptance requires all of these to be true: + +- `scripts/mac/orchestration-smoke.sh` exists and is the single reproducible macOS/Lima orchestration smoke path. +- `scripts/mac/lima-warm.sh` and `scripts/mac/smoke.sh` align with the orchestration smoke path. +- `scripts/mac/orchestration-smoke.sh` proves attach/create, replacement, lazy launch, targeted follow-up, cancel, and mismatch rejection. +- `docs/WORLD.md` no longer claims pre-bootstrap rejection for the supported macOS/Lima path. +- `AGENT_ORCHESTRATION_GAP_MATRIX.md` reflects validated parity and only remaining out-of-scope gaps. + +### Operator Flow And Run-State Artifacts + +Acceptance requires all of these to be true: + +- The parent is the sole writer of `.runs/plan-21/**` and `.runs/task-m21-*/**`. +- `tasks.json` and `run-state.json` accurately reflect accepted, rejected, quarantined, and blocked outcomes. +- Every required sentinel exists on the green path. +- `blocked.json` is absent on the green path. +- Final validation artifacts exist under the `p4` and `p5` task directories. +- `closeout.md` states exactly what landed and what remains out of scope without overclaim. + +## Closeout Phase + +The run is complete only when: + +1. Every required sentinel exists. +2. `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/plan-21/blocked.json` does not exist. +3. The exact validation wall is green. +4. `closeout.md` names the landed parity contract and any remaining out-of-scope gaps without overclaim. +5. The final merged tree reflects parity work, not a new orchestration model. diff --git a/llm-last-mile/ORCH_PLAN-22.md b/llm-last-mile/ORCH_PLAN-22.md new file mode 100644 index 000000000..d249a7385 --- /dev/null +++ b/llm-last-mile/ORCH_PLAN-22.md @@ -0,0 +1,889 @@ +# ORCH_PLAN-22: Execute PLAN-22 Through A Parent-Frozen Public Turn Hardening Contract, Linux World-Member Follow-Up Proof, And Explicit Fail-Closed Coverage + +Live workspace branch: `feat/macos-lima-shared-owner-member-runtime-parity` +Recorded branch in [PLAN-22.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-22.md): `feat/broaden-caller-surfaces-from-repl` +Authoritative execution branch for this run: `feat/macos-lima-shared-owner-member-runtime-parity` +Plan source: [PLAN-22.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-22.md) +Source SOW: [22-broaden-caller-surfaces-from-repl-first-to-public-session-member-turns.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/22-broaden-caller-surfaces-from-repl-first-to-public-session-member-turns.md) +Style references: [ORCH_PLAN-20.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/ORCH_PLAN-20.md), [ORCH_PLAN-21.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/ORCH_PLAN-21.md) +Packet index: [README.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/README.md) +Gap matrix anchor: [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md:107), [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md:112) +Execution type: fresh orchestration controller for public caller-surface hardening, Linux-first world-member follow-up proof, parent-frozen contract, parent-only gates and integration, docs closeout only after merged code truth +Live root: `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate` +Worktree root: `/Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-22` +Worker model: `GPT-5.4` with `reasoning_effort=high` +Max concurrent code workers before integration: `2` + +## Summary + +This document is the execution controller for `PLAN-22`, not a restatement of it. + +The branch recorded inside `PLAN-22.md` is stale relative to the live workspace. The current checkout on `feat/macos-lima-shared-owner-member-runtime-parity` already contains later merged truth from the prompt-taking public surface and the macOS/Lima parity follow-on. Rewinding execution to `feat/broaden-caller-surfaces-from-repl` would force speculative backporting and would no longer represent repo truth. This run therefore treats the current live checkout as authoritative and records the `PLAN-22` branch as historical planning context only. + +The slice is honest only if it lands all of these on the same merged tree: + +- `substrate agent start` remains the canonical public root prompt-taking surface. +- `substrate agent turn` remains the canonical public follow-up surface. +- Linux world-member public follow-up is directly proven from exact selector resolution through typed `MemberTurnSubmitRequestV1` submission into `world-agent`. +- detached host follow-up recovery and detached world follow-up rejection are both explicit parts of the public contract. +- the fail-closed public-turn taxonomy is pinned by concrete tests rather than implied by implementation. +- `crates/shell/tests/repl_world_first_routing_v1.rs` remains an explicit non-regression gate for the already-landed REPL-first exact targeted follow-up contract. +- docs and gap-matrix truth are updated only after merged code and tests prove the shipped contract. + +The only honest opening parallelism is `2` lanes: + +1. shell public-turn contract hardening over the existing prompt-taking and selector seams +2. world-agent retained-member identity-drift proof hardening at the submit boundary + +A third concurrent code lane would either collide with the same shell control-suite files or would start docs and operator truth before merged runtime behavior exists. + +Frozen run shape: + +1. `task/m22-p1-parent-contract-freeze-and-run-init` +2. `task/m22-g1-window-a-launch-gate` +3. parallel Window A + - `task/m22-l1-shell-public-turn-contract-hardening` + - `task/m22-l2-world-agent-retained-member-proof` +4. `task/m22-g2-window-a-integration-gate` +5. `task/m22-p2-parent-window-a-integration` +6. `task/m22-g3-closeout-launch-gate` +7. `task/m22-l3-public-surface-tests-docs-closeout` +8. `task/m22-g4-validation-wall-gate` +9. `task/m22-p3-parent-validation-wall` +10. `task/m22-p4-parent-closeout-phase` + +## Hard Guards + +These are run-stopping invariants, not preferences. + +1. The authoritative integration checkout remains `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate` on `feat/macos-lima-shared-owner-member-runtime-parity`. +2. The branch recorded in [PLAN-22.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-22.md) is preserved only as historical plan metadata. No worker may switch the run back to `feat/broaden-caller-surfaces-from-repl`. +3. The parent agent is the only integrator, the only approval authority, and the only writer of `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/plan-22/**`. +4. `substrate agent start` remains the canonical public root prompt-taking surface. +5. `substrate agent turn` remains the canonical public follow-up prompt-taking surface. +6. Root `start` remains host-only in v1. Public world-root start stays fail-closed. +7. Public `turn` requires exact `--session ` and exact `--backend `. +8. No fuzzy session routing, no fuzzy backend routing, no latest-session fallback, and no acceptance of `participant_id`, `active_session_handle_id`, `session_handle_id`, or `internal.uaa_session_id` as public inputs. +9. Linux world-sensitive public follow-up must continue to use `MemberTurnSubmitRequestV1` and `/v1/member_turn/stream`. +10. Detached host follow-up may recover only through the existing detached-host posture path. Detached world follow-up must fail closed until `substrate agent reattach --session ...` restores an active host owner. +11. `substrate -c`, `--command`, pipe mode, and plain stdin remain shell-wrap semantics. This run must not reinterpret them as agent prompting. +12. `crates/shell/tests/repl_world_first_routing_v1.rs` remains a frozen non-regression surface for the exact REPL targeted-turn contract and must stay green on the merged tree. +13. The public-turn fail-closed taxonomy is frozen and must be explicitly tested for: + - `missing_backend` + - `unknown_session` + - `noncanonical_session_selector` via `active_session_handle_id` + - `noncanonical_session_selector` via `participant_id` / legacy `session_handle_id` + - `noncanonical_session_selector` via `internal.uaa_session_id` + - `missing_active_parent` + - `backend_not_in_session` + - `stale_linkage` + - `ambiguous_backend_slot` + - `unsupported_platform_or_posture` + - `owner_unreachable` +14. Docs closeout is late-only. No worker may edit docs or gap-matrix truth before merged code truth exists. +15. No worker may widen into default-agent routing, new public selector types, new prompt verbs, toolbox mutation, daemon work, or non-Linux parity claims. +16. No worker may edit `.runs/**`. +17. If the authoritative branch choice, exact selector contract, Linux member-turn endpoint, detached-world fail-closed posture, or REPL-first non-regression contract becomes disputed during implementation, the run stops and the parent writes `blocked.json`. + +Stop the run, write `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/plan-22/blocked.json`, and do not advance if any of these occur: + +1. A lane requires execution on the older `feat/broaden-caller-surfaces-from-repl` branch instead of the live authoritative checkout. +2. A lane requires a new public selector surface beyond exact `(orchestration_session_id, backend_id)`. +3. A lane requires changing away from `MemberTurnSubmitRequestV1` or `/v1/member_turn/stream`. +4. A lane requires detached-world follow-up success without `reattach`. +5. A lane requires weakening or renaming the frozen failure taxonomy. +6. A lane requires changing `substrate -c` semantics. +7. A lane requires relaxing the exact REPL-first routing contract covered by `repl_world_first_routing_v1.rs`. +8. A worker touches files outside its frozen ownership surface. +9. `L3` starts before merged Window A truth exists. +10. The final validation wall cannot prove Linux world-member follow-up success, detached-world rejection, fail-closed taxonomy coverage, and REPL-first non-regression on the same merged tree. +11. Docs would need to claim world-root public start, default routing, Windows parity, or any unsupported non-Linux world-follow-up behavior. + +### Blocked-Run Record Contract + +`blocked.json` is parent-written only, exactly once, at the moment the parent decides the run cannot advance. + +Required fields in `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/plan-22/blocked.json`: + +- `run_id` +- `authoritative_branch` +- `recorded_plan_branch` +- `plan_source` +- `timestamp` +- `current_task_id` +- `gate_state` +- `stop_condition_id` +- `summary` +- `blocking_files` +- `worker_lane` +- `accepted_sentinels` +- `rejected_or_quarantined_outputs` +- `next_required_parent_action` + +## Fresh Worktrees And Branches + +Fresh worktree root: + +- `/Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-22` + +Worker worktrees: + +- `/Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-22/shell-public-turn-contract-hardening` +- `/Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-22/world-agent-retained-member-proof` +- `/Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-22/public-surface-tests-docs-closeout` + +Worker branches: + +- `codex/feat-macos-lima-shared-owner-member-runtime-parity-m22-shell-public-turn-contract-hardening` +- `codex/feat-macos-lima-shared-owner-member-runtime-parity-m22-world-agent-retained-member-proof` +- `codex/feat-macos-lima-shared-owner-member-runtime-parity-m22-public-surface-tests-docs-closeout` + +Exact setup commands: + +```bash +mkdir -p /Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-22 + +git worktree add /Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-22/shell-public-turn-contract-hardening \ + -b codex/feat-macos-lima-shared-owner-member-runtime-parity-m22-shell-public-turn-contract-hardening \ + feat/macos-lima-shared-owner-member-runtime-parity + +git worktree add /Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-22/world-agent-retained-member-proof \ + -b codex/feat-macos-lima-shared-owner-member-runtime-parity-m22-world-agent-retained-member-proof \ + feat/macos-lima-shared-owner-member-runtime-parity + +git worktree add /Users/spensermcconnell/__Active_Code/atomize-hq/.worktrees/substrate-plan-22/public-surface-tests-docs-closeout \ + -b codex/feat-macos-lima-shared-owner-member-runtime-parity-m22-public-surface-tests-docs-closeout \ + feat/macos-lima-shared-owner-member-runtime-parity +``` + +No separate parent integration worktree is introduced. The parent integrates only on `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate`. + +## Parent-Owned Run-State Surface + +Canonical parent-owned state under `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/plan-22/`: + +- `run-state.json` +- `tasks.json` +- `session-log.md` +- `contract-freeze.json` +- `lane-ownership.json` +- `merge-order.json` +- `validation-wall.md` +- `blocked.json` on failure only +- `quarantine/` +- `sentinels/` + +Required per-task artifact roots: + +- `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/task-m22-p1-parent-contract-freeze-and-run-init/` +- `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/task-m22-g1-window-a-launch-gate/` +- `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/task-m22-l1-shell-public-turn-contract-hardening/` +- `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/task-m22-l2-world-agent-retained-member-proof/` +- `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/task-m22-g2-window-a-integration-gate/` +- `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/task-m22-p2-parent-window-a-integration/` +- `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/task-m22-g3-closeout-launch-gate/` +- `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/task-m22-l3-public-surface-tests-docs-closeout/` +- `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/task-m22-g4-validation-wall-gate/` +- `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/task-m22-p3-parent-validation-wall/` +- `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/task-m22-p4-parent-closeout-phase/` + +Each task directory must contain: + +- `task.json` +- `commands.txt` +- `summary.md` + +Each gate task directory must also contain: + +- `gate-checklist.md` +- `gate-result.json` + +Each worker task directory must also contain: + +- `worker-report.md` +- `worker-output.patch` +- `evidence-manifest.json` + +Required sentinels: + +- `.runs/plan-22/sentinels/task-m22-p1-parent-contract-freeze-and-run-init.ok` +- `.runs/plan-22/sentinels/task-m22-g1-window-a-launch-gate.ok` +- `.runs/plan-22/sentinels/task-m22-l1-shell-public-turn-contract-hardening.ok` +- `.runs/plan-22/sentinels/task-m22-l2-world-agent-retained-member-proof.ok` +- `.runs/plan-22/sentinels/task-m22-g2-window-a-integration-gate.ok` +- `.runs/plan-22/sentinels/task-m22-p2-parent-window-a-integration.ok` +- `.runs/plan-22/sentinels/task-m22-g3-closeout-launch-gate.ok` +- `.runs/plan-22/sentinels/task-m22-l3-public-surface-tests-docs-closeout.ok` +- `.runs/plan-22/sentinels/task-m22-g4-validation-wall-gate.ok` +- `.runs/plan-22/sentinels/task-m22-p3-parent-validation-wall.ok` +- `.runs/plan-22/sentinels/task-m22-p4-parent-closeout-phase.ok` + +`merge-order.json` is frozen during `p1` and must record: + +- `authoritative_branch: "feat/macos-lima-shared-owner-member-runtime-parity"` +- `recorded_plan_branch: "feat/broaden-caller-surfaces-from-repl"` +- `integration_order: ["task/m22-l1-shell-public-turn-contract-hardening", "task/m22-l2-world-agent-retained-member-proof", "task/m22-l3-public-surface-tests-docs-closeout"]` +- `l2_acceptance_basis: "accepted_l1_tree_if_replay_needed_otherwise_original_post_p1_tree"` +- `l3_acceptance_basis: "accepted_p2_tree_only"` +- `replay_required_before_acceptance: true` +- `quarantine_on_branch_local_assumption: true` + +## PLAN-22 Workstream Mapping + +| PLAN-22 workstream | Orchestration tasks | Why this mapping is exact | +| --- | --- | --- | +| Tighten and prove shell-side exact public turn contract | `task/m22-l1-shell-public-turn-contract-hardening`, `task/m22-g2-window-a-integration-gate`, `task/m22-p2-parent-window-a-integration` | This lane owns the public `turn` posture logic, exact selector resolver seam, and shared prompt bridge contract in production code. | +| Widen retained-member drift and world-boundary proof | `task/m22-l2-world-agent-retained-member-proof`, `task/m22-g2-window-a-integration-gate`, `task/m22-p2-parent-window-a-integration` | This lane owns explicit retained-member identity-drift rejection proof at the `world-agent` submit boundary without changing the request schema. | +| Add explicit Linux world-follow-up proof, fail-closed coverage, REPL-first non-regression proof, and late docs truth | `task/m22-g3-closeout-launch-gate`, `task/m22-l3-public-surface-tests-docs-closeout`, `task/m22-g4-validation-wall-gate`, `task/m22-p3-parent-validation-wall`, `task/m22-p4-parent-closeout-phase` | This closeout lane owns the public control suite, `repl_world_first_routing_v1.rs` non-regression, Linux world-member follow-up proof, explicit fail-closed classifier coverage, detached host/world posture proof, eng-review test-plan artifact, and repo-truth docs after merged code truth exists. | + +## Concurrency And Merge Order + +Concurrency rules: + +1. Worker cap is exactly `2` until `g2` completes. +2. `p1` must finish before any worker starts. +3. `g1` must be green before any worker starts. +4. The only honest initial parallel window is `L1` plus `L2`. +5. `L3` waits for accepted and integrated `L1` plus `L2`. +6. The validation wall runs exactly once on the final merged tree. +7. No docs work starts before `p2`. +8. No extra shell test lane is authorized because the decisive proof surfaces are one shared public control suite plus one shared REPL non-regression suite, not disjoint test islands. + +Why `L1` integrates before `L2`: + +1. `L1` freezes the dominant public-shell semantics that `L3` must prove: exact selector resolution, detached posture handling, and prompt-submit routing. +2. `L2` is intentionally narrower and should remain a boundary-proof lane at `world-agent`; it can replay cleanly if it accidentally assumed branch-local shell behavior. +3. `L3` depends much more on accepted shell public-contract truth than on branch-local world-agent assumptions. +4. Integrating `L2` first would not unblock `L3`; integrating `L1` first does. + +## Task Ledger + +| Task ID | Owner | Depends on | Worktree / branch | Deliverable | +| --- | --- | --- | --- | --- | +| `task/m22-p1-parent-contract-freeze-and-run-init` | parent | — | authoritative checkout | frozen contract, run artifacts, branch-resolution record | +| `task/m22-g1-window-a-launch-gate` | parent | `p1` | authoritative checkout | launch approval for `L1` and `L2` | +| `task/m22-l1-shell-public-turn-contract-hardening` | worker | `g1` | `shell-public-turn-contract-hardening` / `codex/feat-macos-lima-shared-owner-member-runtime-parity-m22-shell-public-turn-contract-hardening` | production shell hardening for public `turn` contract only | +| `task/m22-l2-world-agent-retained-member-proof` | worker | `g1` | `world-agent-retained-member-proof` / `codex/feat-macos-lima-shared-owner-member-runtime-parity-m22-world-agent-retained-member-proof` | explicit retained-member identity-drift proof at the world-agent boundary | +| `task/m22-g2-window-a-integration-gate` | parent | `l1`, `l2` | authoritative checkout | acceptance, rejection, or quarantine for Window A | +| `task/m22-p2-parent-window-a-integration` | parent | `g2` | authoritative checkout | merged shell contract truth then merged world-agent boundary proof | +| `task/m22-g3-closeout-launch-gate` | parent | `p2` | authoritative checkout | launch approval for `L3` | +| `task/m22-l3-public-surface-tests-docs-closeout` | worker | `g3` | `public-surface-tests-docs-closeout` / `codex/feat-macos-lima-shared-owner-member-runtime-parity-m22-public-surface-tests-docs-closeout` | public control tests, REPL non-regression proof, Linux world-follow-up proof, docs/gap-matrix closeout, eng-review test-plan artifact | +| `task/m22-g4-validation-wall-gate` | parent | `l3` | authoritative checkout | permission to run the final validation wall | +| `task/m22-p3-parent-validation-wall` | parent | `g4` | authoritative checkout | exact PLAN-22 validation wall results | +| `task/m22-p4-parent-closeout-phase` | parent | `p3` | authoritative checkout | terminal run-state, closeout, and artifact audit | + +## Lane Ownership By File Set + +| Lane | Allowed files | Forbidden touch surfaces | +| --- | --- | --- | +| `L1` / shell public-turn contract hardening | [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs), [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs), [crates/shell/src/execution/agent_runtime/control.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/control.rs), [crates/shell/src/execution/agent_runtime/session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs) only if a narrow posture helper change is required, inline unit tests inside those owned files only | `crates/shell/tests/**`, `crates/world-agent/**`, `crates/agent-api-types/**`, `docs/**`, `AGENT_ORCHESTRATION_GAP_MATRIX.md`, `llm-last-mile/**`, `.runs/**` | +| `L2` / world-agent retained-member proof | [crates/world-agent/src/member_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs), [crates/world-agent/src/service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs) only if a test hook or boundary assertion cannot live solely in `member_runtime.rs`, inline tests in those owned files only | `crates/shell/**`, `crates/agent-api-types/**`, `docs/**`, `AGENT_ORCHESTRATION_GAP_MATRIX.md`, `llm-last-mile/**`, `.runs/**` | +| `L3` / public-surface tests docs closeout | [crates/shell/tests/agent_public_control_surface_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_public_control_surface_v1.rs), [crates/shell/tests/repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs), `crates/shell/tests/support/**` only when directly required by those suites, [docs/USAGE.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/USAGE.md), [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md), [llm-last-mile/README.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/README.md) | every production Rust file, `.runs/**`, any claim of world-root public start, default routing, Windows/WSL parity, or detached-world follow-up success | + +## Worker Interfaces + +Every worker prompt must include: + +1. task ID +2. attempt number +3. worktree path +4. branch +5. allowed files +6. forbidden files +7. frozen contract clauses relevant to that lane +8. exact required commands +9. retry budget +10. required return artifacts +11. sentinel path +12. the branch-mismatch ruling that the live workspace branch is authoritative + +Every worker return must include: + +1. changed files list +2. commands run with exit codes +3. explicit attempt classification: `clean`, `retryable`, or `blocked` +4. unresolved assumptions or blockers +5. `worker-output.patch` +6. `worker-report.md` +7. `evidence-manifest.json` + +## Kickoff Initialization Order + +The parent initializes the run in this exact order: + +1. Create `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/plan-22/`, `.runs/plan-22/sentinels/`, `.runs/plan-22/quarantine/`, and every `.runs/task-m22-*/` directory. +2. Create `task.json`, `commands.txt`, `summary.md`, and `artifacts/` in every task directory. +3. Create `gate-checklist.md` and `gate-result.json` in every gate task directory. +4. Create placeholder `worker-report.md`, `worker-output.patch`, and `evidence-manifest.json` in every worker task directory. +5. Write `tasks.json` as the canonical launch queue and execution ledger. +6. Write `run-state.json` with `current_phase: "kickoff"`, `worker_cap: 2`, `authoritative_branch: "feat/macos-lima-shared-owner-member-runtime-parity"`, `recorded_plan_branch: "feat/broaden-caller-surfaces-from-repl"`, every task in `pending`, and empty accepted, rejected, quarantined, and blocked arrays. +7. Write `contract-freeze.json`, `lane-ownership.json`, `merge-order.json`, and `validation-wall.md`. +8. Freeze the exact public-turn failure taxonomy and the exact detached-host versus detached-world posture rules in `contract-freeze.json`. +9. Freeze the exact docs truth ceiling in `contract-freeze.json`: + - public `start|turn|reattach|fork|stop` exists + - root `start` remains host-only + - public `turn` remains exact `(orchestration_session_id, backend_id)` + - Linux world-member follow-up is source-of-truth and must be proven + - detached world follow-up still requires `reattach` + - no default routing + - no public world-root start + - no Windows/WSL parity claim +10. Freeze the branch-resolution ruling in `contract-freeze.json` and `session-log.md`. +11. Freeze `repl_world_first_routing_v1.rs` as a mandatory non-regression gate in `contract-freeze.json` and `validation-wall.md`. +12. Review the frozen hotspots: + - [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs) + - [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) + - [crates/shell/src/execution/agent_runtime/control.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/control.rs) + - [crates/world-agent/src/member_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs) + - [crates/shell/tests/agent_public_control_surface_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_public_control_surface_v1.rs) + - [crates/shell/tests/repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs) + - [docs/USAGE.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/USAGE.md) +13. No pre-launch production scaffold is authorized. The existing public caller surface and world-agent boundary already exist. If a lane claims it needs parent-seeded API invention, block the run rather than create hybrid semantics. +14. Seed `L1` and `L2` worktrees from the exact same post-`p1` tree. +15. Write `session-log.md` with kickoff timestamp, both branch names, the authoritative branch ruling, the worktree roots, the worker cap, and the statement that the only honest initial parallel window is `L1` plus `L2`. + +## Parent Phases And Worker Packets + +### `task/m22-p1-parent-contract-freeze-and-run-init` + +Owner: + +- parent only + +Scope: + +1. Freeze the branch-mismatch disposition in favor of the live workspace branch. +2. Freeze the public caller-surface contract, REPL-first non-regression contract, and fail-closed taxonomy. +3. Freeze exact file ownership, merge order, retry budget, stop conditions, and validation wall. +4. Seed `L1` and `L2` from the same post-`p1` tree. + +Command gates: + +```bash +cargo test -p shell --no-run +cargo test -p world-agent --no-run +cargo test -p agent-api-types --no-run +``` + +Acceptance: + +1. `contract-freeze.json`, `lane-ownership.json`, `merge-order.json`, `tasks.json`, and `run-state.json` exist. +2. The freeze artifact explicitly records that the current workspace branch is authoritative. +3. The frozen contract records canonical `start`, canonical `turn`, host-only root start, Linux world-member follow-up through `MemberTurnSubmitRequestV1`, detached-world fail-closed posture, unchanged `-c`, mandatory `repl_world_first_routing_v1.rs` non-regression, and late-only docs work. +4. The parent writes `.runs/plan-22/sentinels/task-m22-p1-parent-contract-freeze-and-run-init.ok`. + +### `task/m22-g1-window-a-launch-gate` + +Owner: + +- parent only + +Checks: + +1. `p1` is accepted. +2. `L1` and `L2` were seeded from the exact same post-`p1` tree. +3. `L1` prompt explicitly forbids tests and docs work. +4. `L2` prompt explicitly forbids shell and docs work. +5. Both prompts repeat the frozen prohibitions on branch rewinding, selector widening, endpoint drift, detached-world success broadening, REPL-first contract regression, and `-c` reinterpretation. + +Acceptance: + +1. No worker starts before this gate is green. +2. The parent writes `.runs/plan-22/sentinels/task-m22-g1-window-a-launch-gate.ok`. + +### `task/m22-l1-shell-public-turn-contract-hardening` + +Owner: + +- single worker on `codex/feat-macos-lima-shared-owner-member-runtime-parity-m22-shell-public-turn-contract-hardening` + +Packet fields: + +- Owned files: + - [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs) + - [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) + - [crates/shell/src/execution/agent_runtime/control.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/control.rs) + - [crates/shell/src/execution/agent_runtime/session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs) only if a narrow posture helper change is required + - inline unit tests in those owned files only +- Forbidden touch surfaces: + - `crates/shell/tests/**` + - `crates/world-agent/**` + - `crates/agent-api-types/**` + - `docs/**` + - `AGENT_ORCHESTRATION_GAP_MATRIX.md` + - `llm-last-mile/**` + - `.runs/**` +- Exact scope: + 1. Tighten `run_turn(...)`, `resolve_public_turn_target(...)`, and `run_public_prompt_command(...)` only where repo truth still leaves the public contract implied instead of explicit. + 2. Preserve exact `(orchestration_session_id, backend_id)` routing. + 3. Preserve detached-host recovery semantics. + 4. Preserve detached-world follow-up rejection with explicit reattach posture. + 5. Preserve Linux world-member follow-up through the existing public prompt bridge and retained-member submit path. + 6. Preserve classifier stability for the frozen fail-closed taxonomy. + 7. Preserve the already-landed REPL-first routing contract by not perturbing shared follow-up assumptions. + 8. Do not invent new public verbs, selectors, transport, or platform parity. +- Exact required commands: + +```bash +cargo test -p shell --lib -- --nocapture +cargo test -p shell agent_runtime::state_store -- --nocapture +cargo test -p shell --test agent_public_control_surface_v1 --no-run +cargo test -p shell --test repl_world_first_routing_v1 --no-run +``` + +- Exact acceptance: + 1. The lane touches only its owned files. + 2. Exact selector routing remains authoritative and fail-closed. + 3. Detached host and detached world postures remain explicit and distinct. + 4. Linux world-member follow-up still flows through the existing typed boundary. + 5. No classifier drift, no public selector widening, and no `-c` reinterpretation are introduced. + 6. The lane does not create a regression that blocks `repl_world_first_routing_v1.rs` from remaining green on the merged tree. + 7. The worker writes `.runs/plan-22/sentinels/task-m22-l1-shell-public-turn-contract-hardening.ok`. + +### `task/m22-l2-world-agent-retained-member-proof` + +Owner: + +- single worker on `codex/feat-macos-lima-shared-owner-member-runtime-parity-m22-world-agent-retained-member-proof` + +Packet fields: + +- Owned files: + - [crates/world-agent/src/member_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs) + - [crates/world-agent/src/service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs) only if a boundary assertion or test hook cannot stay inside `member_runtime.rs` + - inline tests in those owned files only +- Forbidden touch surfaces: + - `crates/shell/**` + - `crates/agent-api-types/**` + - `docs/**` + - `AGENT_ORCHESTRATION_GAP_MATRIX.md` + - `llm-last-mile/**` + - `.runs/**` +- Exact scope: + 1. Add explicit retained-member identity-drift rejection proof at the `validate_submit_turn_request(...)` / `submit_turn(...)` boundary. + 2. Make the hidden tuple contract obvious in tests: + - `participant_id` + - `orchestrator_participant_id` + - `backend_id` + - `world_id` + - `world_generation` + 3. Keep the request schema unchanged. + 4. Keep `/v1/member_turn/stream` unchanged. + 5. Do not broaden into shell behavior, docs, or public-surface claims. +- Exact required commands: + +```bash +cargo test -p world-agent member_runtime -- --nocapture +``` + +- Exact acceptance: + 1. The lane touches only its owned files. + 2. At least one explicit retained-member identity-drift case is rejected at the world-agent submit boundary. + 3. The proof makes tuple continuity, not backend id alone, the obvious contract. + 4. No request schema or endpoint changes are introduced. + 5. The worker writes `.runs/plan-22/sentinels/task-m22-l2-world-agent-retained-member-proof.ok`. + +### `task/m22-g2-window-a-integration-gate` + +Owner: + +- parent only + +Checks: + +1. `L1` and `L2` both returned patch, report, command transcript, and evidence manifest. +2. Every touched file is inside the lane’s ownership boundary. +3. `L1` preserved exact selector rules, detached-host recovery, detached-world rejection, Linux world follow-up, classifier stability, and REPL-first non-regression assumptions. +4. `L2` stayed inside world-agent boundary proof and did not widen the host/world contract. +5. If either lane assumed the old `feat/broaden-caller-surfaces-from-repl` branch instead of the authoritative current branch, quarantine that lane immediately. +6. `L2` is replayed only if it accidentally consumed shell-local assumptions from an unaccepted `L1` tree. + +Quarantine and retry behavior: + +1. If `L1` changes public verbs, handle types, `-c` semantics, `repl_world_first_routing_v1` assumptions, or doc surfaces, quarantine `L1` immediately. +2. If `L2` changes request schema, endpoint ownership, or shell files, quarantine `L2` immediately. +3. Retry budget remains `1` per lane and is available only for lane-local defects inside owned files. +4. If either lane violates branch authority, ownership, or frozen endpoint/selector constraints, mark non-retryable and quarantine rather than redrive. + +Acceptance: + +1. Accepted, rejected, or quarantined status for both lanes is recorded in `run-state.json`. +2. The parent writes `.runs/plan-22/sentinels/task-m22-g2-window-a-integration-gate.ok`. + +### `task/m22-p2-parent-window-a-integration` + +Owner: + +- parent only + +Scope: + +1. Integrate accepted `L1` output first. +2. Re-run `L1` command gates on the authoritative checkout. +3. Replay `L2` on top of accepted `L1` only if necessary. If `L2` assumed different shell truth, quarantine `L2` instead of hand-editing around the mismatch. +4. Integrate accepted `L2` output second. +5. Freeze the merged shell-plus-world-boundary truth before public control tests and docs start. + +Command gates: + +```bash +cargo test -p shell --lib -- --nocapture +cargo test -p shell agent_runtime::state_store -- --nocapture +cargo test -p shell --test repl_world_first_routing_v1 -- --nocapture +cargo test -p world-agent member_runtime -- --nocapture +cargo test -p shell --test agent_public_control_surface_v1 --no-run +``` + +Acceptance: + +1. The parent remains the sole integrator. +2. The authoritative tree now contains any required shell contract hardening and explicit world-agent retained-member proof. +3. `repl_world_first_routing_v1.rs` remains green after merged Window A truth. +4. No hybrid contract was invented during integration. +5. The parent writes `.runs/plan-22/sentinels/task-m22-p2-parent-window-a-integration.ok`. + +### `task/m22-g3-closeout-launch-gate` + +Owner: + +- parent only + +Checks: + +1. `p2` is green. +2. `L3` worktree is seeded from the exact accepted post-`p2` tree. +3. The prompt names only the public control test suite, the REPL non-regression suite, test support, `docs/USAGE.md`, `AGENT_ORCHESTRATION_GAP_MATRIX.md`, and `llm-last-mile/README.md`. +4. The prompt explicitly forbids reopening production Rust files or overclaiming branch or platform truth. +5. The prompt explicitly assigns the eng-review test-plan artifact to `L3`. + +Acceptance: + +1. No `L3` worker starts before this gate is green. +2. The parent writes `.runs/plan-22/sentinels/task-m22-g3-closeout-launch-gate.ok`. + +### `task/m22-l3-public-surface-tests-docs-closeout` + +Owner: + +- single worker on `codex/feat-macos-lima-shared-owner-member-runtime-parity-m22-public-surface-tests-docs-closeout` + +Packet fields: + +- Owned files: + - [crates/shell/tests/agent_public_control_surface_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_public_control_surface_v1.rs) + - [crates/shell/tests/repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs) + - `crates/shell/tests/support/**` only when directly required by those suites + - [docs/USAGE.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/USAGE.md) + - [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md) + - [llm-last-mile/README.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/README.md) +- Forbidden touch surfaces: + - every production Rust file + - `.runs/**` + - any claim of world-root public start, default routing, Windows/WSL parity, or detached-world follow-up success +- Exact scope: + 1. Add explicit public Linux world-member follow-up success proof from exact public selector resolution to typed `MemberTurnSubmitRequestV1` submission evidence. + 2. Add explicit fail-closed tests for the frozen taxonomy: + - `missing_backend` + - `unknown_session` + - `missing_active_parent` + - `backend_not_in_session` + - `stale_linkage` + - `ambiguous_backend_slot` + - noncanonical selector via `active_session_handle_id` + - noncanonical selector via `participant_id` / legacy `session_handle_id` + - noncanonical selector via `internal.uaa_session_id` + - `unsupported_platform_or_posture` + - `owner_unreachable` + 3. Add explicit detached host recovery proof and detached world rejection proof with reattach guidance. + 4. Preserve and, if needed, extend `repl_world_first_routing_v1.rs` and its support fixtures as an explicit non-regression surface for the exact REPL targeted-turn contract. + 5. Keep `substrate -c` non-regression explicit. + 6. Update `docs/USAGE.md`, `AGENT_ORCHESTRATION_GAP_MATRIX.md`, and `llm-last-mile/README.md` only to the level proven by the merged tree. + 7. Produce the usual eng-review test-plan artifact at: + - `~/.gstack/projects//-feat-macos-lima-shared-owner-member-runtime-parity-eng-review-test-plan-.md` +- Exact required commands: + +```bash +cargo test -p shell --test agent_public_control_surface_v1 -- --nocapture +cargo test -p shell --test repl_world_first_routing_v1 -- --nocapture +cargo test -p shell agent_runtime::state_store -- --nocapture +cargo test -p world-agent member_runtime -- --nocapture +cargo test -p agent-api-types member_turn_submit -- --nocapture +rg -n "start|turn|reattach|detached|member_turn/stream|world-root|default-agent|-c|PLAN-22|ORCH_PLAN-22" \ + /Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/USAGE.md \ + /Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md \ + /Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/README.md +``` + +- Exact acceptance: + 1. The lane touches only its owned files. + 2. `agent_public_control_surface_v1.rs` explicitly proves Linux world-member public follow-up success. + 3. The frozen fail-closed taxonomy is pinned by explicit tests rather than implied behavior. + 4. Detached host recovery and detached world rejection are both part of the public contract proof. + 5. `repl_world_first_routing_v1.rs` remains an explicit owned non-regression suite and stays green, with any necessary support fixture edits confined to test-support surfaces. + 6. `substrate -c` remains explicitly protected. + 7. `llm-last-mile/README.md` remains discoverable for both `PLAN-22` and `ORCH_PLAN-22`, following the packet-index pattern used by prior orchestration plans. + 8. The eng-review test-plan artifact exists at the required `~/.gstack/projects//...eng-review-test-plan-.md` path and covers: + - host root `start` + - host follow-up `turn` + - Linux world-member follow-up `turn` + - detached host recovery + - detached world rejection + - fail-closed selector taxonomy + - `repl_world_first_routing_v1` non-regression + - `substrate -c` non-regression + 9. Docs and gap-matrix truth do not overclaim beyond the merged validation wall. + 10. The worker writes `.runs/plan-22/sentinels/task-m22-l3-public-surface-tests-docs-closeout.ok`. + +### `task/m22-g4-validation-wall-gate` + +Owner: + +- parent only + +Checks: + +1. `L3` returned and is accepted. +2. No quarantined or blocked output remains unresolved. +3. `validation-wall.md` names the exact final command order. +4. The parent can map every frozen completion promise to a command, test, or artifact in the merged tree. +5. The eng-review test-plan artifact path is recorded in the `L3` evidence manifest. + +Quarantine and retry behavior: + +1. If `L3` overclaims docs truth, drops REPL-first non-regression coverage, or omits the eng-review test-plan artifact, quarantine `L3` instead of editing by hand. +2. If `L3` failed only within owned files and stayed inside validated scope, retry budget `1` remains available. +3. No validation wall starts until `L3` is either accepted or the run is blocked. + +Acceptance: + +1. The parent writes `.runs/plan-22/sentinels/task-m22-g4-validation-wall-gate.ok`. +2. The validation wall is permitted to run exactly once. + +### `task/m22-p3-parent-validation-wall` + +Owner: + +- parent only + +Scope: + +1. Integrate only accepted `L3` output. +2. Run the exact PLAN-22 validation wall on the authoritative checkout in this order. +3. Record command results and artifact paths. +4. Confirm docs, gap matrix, packet index, and test-plan artifact match the validated runtime truth. + +Validation wall commands: + +```bash +cargo fmt --all -- --check +cargo clippy --workspace --all-targets -- -D warnings +cargo test -p shell --lib -- --nocapture +cargo test -p shell agent_runtime::state_store -- --nocapture +cargo test -p shell --test agent_public_control_surface_v1 -- --nocapture +cargo test -p shell --test repl_world_first_routing_v1 -- --nocapture +cargo test -p world-agent member_runtime -- --nocapture +cargo test -p agent-api-types member_turn_submit -- --nocapture +cargo test --workspace -- --nocapture +substrate agent doctor --json +substrate shim doctor --json +substrate world doctor --json +substrate health --json +``` + +Manual spot checks after command wall, on the same merged tree: + +```bash +substrate agent start --backend --prompt "hello" --json +substrate agent turn --session --backend --prompt "next" --json +substrate agent reattach --session --json +substrate -c "echo hi" +``` + +Linux source-of-truth spot check when a retained world member exists: + +```bash +substrate agent turn --session --backend --prompt "continue" --json +``` + +Required artifacts under `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/task-m22-p3-parent-validation-wall/artifacts/`: + +- `fmt.txt` +- `clippy.txt` +- `shell-lib.txt` +- `shell-state-store.txt` +- `agent-public-control-surface-v1.txt` +- `repl-world-first-routing-v1.txt` +- `world-agent-member-runtime.txt` +- `agent-api-types-member-turn-submit.txt` +- `workspace.txt` +- `agent-doctor.txt` +- `shim-doctor.txt` +- `world-doctor.txt` +- `health.txt` +- `manual-spot-checks.md` +- `contract-audit.md` +- `eng-review-test-plan-path.txt` + +Acceptance: + +1. All validation wall commands succeed in order. +2. The validation wall proves Linux world-member public follow-up success, detached host recovery, detached world rejection, explicit fail-closed taxonomy coverage, retained-member identity-drift rejection, unchanged `-c`, and `repl_world_first_routing_v1.rs` non-regression on the same merged tree. +3. The stronger repo-standard wall including `cargo test --workspace -- --nocapture` is green. +4. The parent writes `.runs/plan-22/sentinels/task-m22-p3-parent-validation-wall.ok`. + +### `task/m22-p4-parent-closeout-phase` + +Owner: + +- parent only + +Scope: + +1. Confirm all required sentinels exist and `blocked.json` does not. +2. Confirm `tasks.json` and `run-state.json` match actual accepted, rejected, and quarantined outcomes. +3. Confirm no quarantined or blocked output was partially integrated. +4. Write terminal `closeout.md`. +5. Mark the run complete only if the final validated state matches the frozen public-turn hardening contract and the authoritative branch ruling. + +Required artifacts under `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/task-m22-p4-parent-closeout-phase/artifacts/`: + +- `closeout.md` +- `final-run-state.json` +- `final-task-ledger.json` +- `final-sentinel-audit.md` + +Acceptance: + +1. `run-state.json` records a successful terminal state. +2. `closeout.md` states exactly what public caller-surface hardening landed, that `repl_world_first_routing_v1.rs` stayed green as a non-regression surface, that the eng-review test-plan artifact was produced, and what remains out of scope. +3. The parent writes `.runs/plan-22/sentinels/task-m22-p4-parent-closeout-phase.ok`. + +## Quarantine, Retry, And Blocked-Run Posture + +1. Each worker lane has retry budget `1`. +2. Retry is allowed only for lane-local defects inside owned files. +3. Non-retryable violations include: + - branch-authority reversal + - selector widening + - endpoint drift away from `/v1/member_turn/stream` + - detached-world success broadening + - `-c` reinterpretation + - request schema changes + - REPL-first routing contract regression + - early docs work + - any cross-lane file touch +4. If `L1` cannot stay inside the existing shell public-turn seams, quarantine it immediately. +5. If `L2` can only pass by changing request schema or shell behavior, quarantine it immediately. +6. If `L3` needs production Rust edits to make tests pass, quarantine it and bounce the issue back to the parent rather than reopening runtime code late. +7. The parent never hand-merges a hybrid truth from conflicting worker guesses. + +When a lane is quarantined, the parent must preserve the returned materials in both places: + +1. The original task artifact directory. +2. `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/plan-22/quarantine//` + +`quarantine//quarantine-reason.json` must record: + +- `task_id` +- `classification` +- `summary` +- `files_touched` +- `frozen_contract_clause_violated` +- `retry_available` +- `next_parent_action` + +## Validation Wall + +The parent may run the final validation wall exactly once, only after `g4` is green. + +The wall is green only when all of these are true: + +1. `substrate agent start` still remains the canonical public root prompt-taking surface. +2. `substrate agent turn` still remains the canonical public follow-up surface. +3. Linux world-member public follow-up is directly proven from exact public selectors into typed retained-member submission. +4. Detached host follow-up recovery is proven explicitly. +5. Detached world follow-up rejection with reattach guidance is proven explicitly. +6. The frozen public-turn fail-closed taxonomy is covered explicitly by tests. +7. At least one retained-member identity-drift case is rejected at the world-agent boundary. +8. `substrate -c` remains shell-wrap behavior. +9. `repl_world_first_routing_v1.rs` is green and still proves the exact REPL-first targeted follow-up contract did not regress while public caller surfaces were hardened. +10. `docs/USAGE.md`, `AGENT_ORCHESTRATION_GAP_MATRIX.md`, and `llm-last-mile/README.md` match the validated runtime truth. +11. The eng-review test-plan artifact exists and matches the final proof surfaces. +12. No quarantined or blocked output remains unresolved. + +## Tests And Acceptance + +### Shell Contract Hardening + +Acceptance requires all of these to be true: + +- exact `(orchestration_session_id, backend_id)` resolution stays authoritative +- detached host and detached world postures remain explicit and distinct +- Linux world follow-up still uses the existing retained-member public prompt path +- no public selector widening is introduced +- `cargo test -p shell --lib -- --nocapture` and `cargo test -p shell agent_runtime::state_store -- --nocapture` are green +- `cargo test -p shell --test repl_world_first_routing_v1 -- --nocapture` remains green after integration + +### World-Agent Retained-Member Proof + +Acceptance requires all of these to be true: + +- retained-member identity drift is proven as an explicit submit-boundary rejection +- the test makes the exact tuple continuity contract obvious +- request schema and endpoint ownership stay unchanged +- `cargo test -p world-agent member_runtime -- --nocapture` is green + +### Public Control Suite, REPL Non-Regression, And Docs Truth + +Acceptance requires all of these to be true: + +- `agent_public_control_surface_v1.rs` explicitly covers: + - host-scoped public start + - host follow-up public turn + - Linux world-member public turn success + - detached host recovery + - detached world rejection + - `missing_backend` + - `unknown_session` + - `missing_active_parent` + - `backend_not_in_session` + - `stale_linkage` + - `ambiguous_backend_slot` + - each noncanonical selector variant + - `unsupported_platform_or_posture` + - `owner_unreachable` + - root world-only start rejection + - `substrate -c` non-regression +- `repl_world_first_routing_v1.rs` explicitly remains green as a required non-regression surface, with any fixture updates constrained to test-support files +- the docs say only what the merged tree proves: + - public `start|turn|reattach|fork|stop` exists + - root `start` is still host-only + - follow-up `turn` requires exact `(orchestration_session_id, backend_id)` + - Linux world-member follow-up is proven + - detached world follow-up still requires `reattach` + - no default routing + - no public world-root start + - no Windows/WSL parity claim +- `llm-last-mile/README.md` remains discoverable for both `PLAN-22` and `ORCH_PLAN-22` +- the eng-review test-plan artifact exists under `~/.gstack/projects//...eng-review-test-plan-.md` + +### Operator Flow And Run-State Artifacts + +Acceptance requires all of these to be true: + +- the parent is the sole writer of `.runs/plan-22/**` and `.runs/task-m22-*/**` +- `tasks.json` and `run-state.json` accurately reflect accepted, rejected, quarantined, and blocked outcomes +- every required sentinel exists on the green path +- `blocked.json` is absent on the green path +- final validation artifacts exist under the `p3` and `p4` task directories +- `closeout.md` states exactly what landed and what remains out of scope without overclaim + +## Closeout Phase + +The run is complete only when: + +1. Every required sentinel exists. +2. `/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/plan-22/blocked.json` does not exist. +3. The exact validation wall is green. +4. `closeout.md` names the authoritative branch ruling, the landed public caller-surface hardening contract, the preserved `repl_world_first_routing_v1.rs` non-regression contract, the eng-review test-plan artifact, and any remaining out-of-scope gaps without overclaim. +5. The final merged tree reflects hardening and proof of the already-shipped public surface, not a new orchestration model. diff --git a/llm-last-mile/ORCH_PLAN-25.md b/llm-last-mile/ORCH_PLAN-25.md new file mode 100644 index 000000000..cae412862 --- /dev/null +++ b/llm-last-mile/ORCH_PLAN-25.md @@ -0,0 +1,605 @@ +# ORCH_PLAN-25: Execute PLAN-25 Through Durable Host Session Truth Freeze, Single-Lane Lifecycle QA Hardening, And Late Inbox-Scope Cleanup + +Live workspace branch: `feat/host-orchestrator-durable-session` +Authoritative execution branch for this run: `feat/host-orchestrator-durable-session` +Plan source: [PLAN-25.md](/home/spenser/__Active_code/substrate/llm-last-mile/PLAN-25.md) +Style references: [ORCH_PLAN-20.md](/home/spenser/__Active_code/substrate/llm-last-mile/ORCH_PLAN-20.md), [ORCH_PLAN-22.md](/home/spenser/__Active_code/substrate/llm-last-mile/ORCH_PLAN-22.md) +Packet index: [README.md](/home/spenser/__Active_code/substrate/llm-last-mile/README.md) +Truth anchors: [HOST_ORCHESTRATOR_INTENDED_BEHAVIOR_TRUTH.md](/home/spenser/__Active_code/substrate/HOST_ORCHESTRATOR_INTENDED_BEHAVIOR_TRUTH.md), [AGENT_ORCHESTRATION_GAP_MATRIX.md](/home/spenser/__Active_code/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md), [docs/USAGE.md](/home/spenser/__Active_code/substrate/docs/USAGE.md) +Current root controllers: [PLAN.md](/home/spenser/__Active_code/substrate/PLAN.md), [ORCH_PLAN.md](/home/spenser/__Active_code/substrate/ORCH_PLAN.md) +Execution type: fresh orchestration controller for durable host-session closeout, parent-frozen public contract, parent-only gates and integration, one real code lane, one real docs lane, one late cleanup lane +Live root: `/home/spenser/__Active_code/substrate` +Worktree root: `/home/spenser/__Active_code/.worktrees/substrate-plan-25` +Worker model: `GPT-5.4` with `reasoning_effort=high` +Initial concurrent worker cap: `2` +Total lanes across the full run: `3` + +Post-landing drift note: `PLAN-25` landed at `995e273c` on 2026-05-14 and this controller landed at `8d0367cb` on 2026-05-15. Since then the repo root planning docs moved to slice-26 async persistent-session bootstrap readiness; execute this controller against current HEAD, but do not fold slice-26 work into the `L1` / `L2` / `L3` scope. + +## Summary + +This document is the execution controller for `PLAN-25`, not a restatement of it. + +The public contract is already chosen on `feat/host-orchestrator-durable-session` and is frozen for this run: + +- `turn` is prompt-taking follow-up on the same durable session. +- `reattach` is attached-owner recovery only. +- `stop` is durable closeout for attached and parked host sessions. +- `status --json` is the authoritative parked-session read surface. +- Detached-world follow-up stays fail closed until `reattach` restores an active host owner. + +The orchestration job is to walk that frozen contract to completion without reopening product design. The run is honest only if it lands all of the following on the same merged tree: + +- repo truth docs stop claiming parked `status`, `reattach`, or `stop` are unfinished, +- one same-session lifecycle regression proves parked `status`, parked `turn`, `reattach`, and `stop`, +- explicit command-level parked-status assertions prove `parked_resumable` and `awaiting_attention` rows retain live-runtime fields, +- inbox wording is narrowed to persistence, posture normalization, internal ack/dismiss, and dev-support/test ingress only, +- the final validation wall is published and green. + +Real concurrency is capped at `2` at launch, not `3`, because there is only one real code hotspot: + +1. [crates/shell/tests/agent_public_control_surface_v1.rs](/home/spenser/__Active_code/substrate/crates/shell/tests/agent_public_control_surface_v1.rs) is a single ownership hotspot and must not be split across workers. +2. The docs lane is real and independent. +3. The inbox-comment cleanup lane is real but cannot start until code and docs converge, because it may touch runtime comment surfaces near the same contract seams and depends on the final wording chosen by the merged tree. + +Frozen run shape: + +1. `task/m25-p1-parent-contract-freeze-and-run-init` +2. `task/m25-g1-window-a-launch-gate` +3. parallel Window A + - `task/m25-l1-lifecycle-qa-hardening` + - `task/m25-l2-truth-doc-convergence` +4. `task/m25-g2-window-a-integration-gate` +5. `task/m25-p2-parent-window-a-integration` +6. `task/m25-g3-cleanup-launch-gate` +7. `task/m25-l3-inbox-scope-comment-tightening` +8. `task/m25-g4-validation-wall-gate` +9. `task/m25-p3-parent-validation-wall` +10. `task/m25-p4-parent-closeout-phase` + +## Hard Guards + +These are run-stopping invariants, not preferences. + +1. The authoritative integration checkout remains `/home/spenser/__Active_code/substrate` on `feat/host-orchestrator-durable-session`. +2. The parent agent is the only integrator, the only approval authority, and the only writer of `/home/spenser/__Active_code/substrate/.runs/plan-25/**`. +3. The public contract is frozen exactly to: + - `substrate agent start --backend --prompt ... --json` for root prompt-taking. + - `substrate agent turn --session --backend --prompt ... --json` for same-session prompt-taking follow-up. + - `substrate agent reattach --session --json` for attached-owner recovery only. + - `substrate agent stop --session --json` for durable closeout. + - `substrate agent status --json` for authoritative parked-session truth. +4. `reattach` is never repurposed as a prompt-taking alias. +5. Detached-world follow-up remains fail closed until `reattach` restores an active host owner. No lane may weaken that rule. +6. The durable inbox contract stays narrow: + - persistence exists, + - posture normalization into `awaiting_attention` exists, + - internal ack/dismiss support exists, + - dev-support and test ingress exist, + - no public inbox command surface is shipped, + - no public inbox product workflow may be implied by docs or comments. +7. The implementation sequence is frozen: + - freeze repo truth docs, + - add one same-session lifecycle regression, + - add explicit parked-status assertions, + - tighten inbox comments only if needed, + - publish the validation wall. +8. `agent_public_control_surface_v1.rs` is a single-owner hotspot. No parallel lane may touch it except `L1`. +9. `L1` is the only lane allowed to expose a production-code change, and any such change must be tiny, test-driven, and directly required by the new lifecycle/status assertions. No speculative runtime hardening is allowed. +10. `L2` may not touch Rust files, tests, `.runs/**`, or plan-controller files. +11. `L3` may only touch runtime comments if needed. It may not make semantic Rust changes, may not reopen docs truth, and may not write `.runs/**`. +12. No lane may add, rename, or widen public verbs, selectors, inbox surfaces, or recovery semantics. +13. No lane may claim public inbox productization, automatic resume from inbox items, or detached-world follow-up success without `reattach`. +14. Manual validation is host-only for this slice. `awaiting_attention` visibility and detached-world fail-closed behavior remain automated-only checks. +15. No lane may edit `PLAN-25.md` or this controller during execution. +16. If the frozen public contract, the narrow inbox contract, the single-owner hotspot rule, or the final validation wall becomes disputed, the run stops and the parent writes `blocked.json`. + +Stop the run, write `/home/spenser/__Active_code/substrate/.runs/plan-25/blocked.json`, and do not advance if any of these occur: + +1. A worker needs a second concurrent code lane touching [agent_public_control_surface_v1.rs](/home/spenser/__Active_code/substrate/crates/shell/tests/agent_public_control_surface_v1.rs). +2. `L1` requires a broad runtime refactor instead of a tiny test-exposed fix. +3. `L2` can only make the docs truthful by inventing a public inbox surface or rewriting the recovery contract. +4. `L3` needs to make semantic runtime changes instead of comment tightening. +5. Parked or `awaiting_attention` command-level status assertions cannot be proven deterministically in the existing shell control suite. +6. Detached-world follow-up would need to succeed without `reattach` to keep tests green. +7. The final validation wall cannot prove targeted shell control coverage, targeted state-store coverage, and full workspace health on the same merged tree. + +### Blocked-Run Record Contract + +`blocked.json` is parent-written only, exactly once, at the moment the parent decides the run cannot advance. + +Required fields in `/home/spenser/__Active_code/substrate/.runs/plan-25/blocked.json`: + +- `run_id` +- `authoritative_branch` +- `plan_source` +- `timestamp` +- `current_task_id` +- `gate_state` +- `stop_condition_id` +- `summary` +- `blocking_files` +- `worker_lane` +- `accepted_sentinels` +- `rejected_or_quarantined_outputs` +- `next_required_parent_action` + +## Fresh Worktrees And Branches + +Fresh worktree root: + +- `/home/spenser/__Active_code/.worktrees/substrate-plan-25` + +Worker worktrees: + +- `/home/spenser/__Active_code/.worktrees/substrate-plan-25/lifecycle-qa-hardening` +- `/home/spenser/__Active_code/.worktrees/substrate-plan-25/truth-doc-convergence` +- `/home/spenser/__Active_code/.worktrees/substrate-plan-25/inbox-scope-comment-tightening` + +Worker branches: + +- `codex/feat-host-orchestrator-durable-session-m25-lifecycle-qa-hardening` +- `codex/feat-host-orchestrator-durable-session-m25-truth-doc-convergence` +- `codex/feat-host-orchestrator-durable-session-m25-inbox-scope-comment-tightening` + +Exact setup commands: + +```bash +mkdir -p /home/spenser/__Active_code/.worktrees/substrate-plan-25 + +git worktree add /home/spenser/__Active_code/.worktrees/substrate-plan-25/lifecycle-qa-hardening \ + -b codex/feat-host-orchestrator-durable-session-m25-lifecycle-qa-hardening \ + feat/host-orchestrator-durable-session + +git worktree add /home/spenser/__Active_code/.worktrees/substrate-plan-25/truth-doc-convergence \ + -b codex/feat-host-orchestrator-durable-session-m25-truth-doc-convergence \ + feat/host-orchestrator-durable-session + +git worktree add /home/spenser/__Active_code/.worktrees/substrate-plan-25/inbox-scope-comment-tightening \ + -b codex/feat-host-orchestrator-durable-session-m25-inbox-scope-comment-tightening \ + feat/host-orchestrator-durable-session +``` + +No separate parent integration worktree is introduced. The parent integrates only on `/home/spenser/__Active_code/substrate`. + +## Parent-Owned Run-State Surface + +Canonical parent-owned state under `/home/spenser/__Active_code/substrate/.runs/plan-25/`: + +- `run-state.json` +- `tasks.json` +- `session-log.md` +- `contract-freeze.json` +- `lane-ownership.json` +- `merge-order.json` +- `validation-wall.md` +- `blocked.json` on failure only +- `quarantine/` +- `sentinels/` + +Required per-task artifact roots: + +- `/home/spenser/__Active_code/substrate/.runs/task-m25-p1-parent-contract-freeze-and-run-init/` +- `/home/spenser/__Active_code/substrate/.runs/task-m25-g1-window-a-launch-gate/` +- `/home/spenser/__Active_code/substrate/.runs/task-m25-l1-lifecycle-qa-hardening/` +- `/home/spenser/__Active_code/substrate/.runs/task-m25-l2-truth-doc-convergence/` +- `/home/spenser/__Active_code/substrate/.runs/task-m25-g2-window-a-integration-gate/` +- `/home/spenser/__Active_code/substrate/.runs/task-m25-p2-parent-window-a-integration/` +- `/home/spenser/__Active_code/substrate/.runs/task-m25-g3-cleanup-launch-gate/` +- `/home/spenser/__Active_code/substrate/.runs/task-m25-l3-inbox-scope-comment-tightening/` +- `/home/spenser/__Active_code/substrate/.runs/task-m25-g4-validation-wall-gate/` +- `/home/spenser/__Active_code/substrate/.runs/task-m25-p3-parent-validation-wall/` +- `/home/spenser/__Active_code/substrate/.runs/task-m25-p4-parent-closeout-phase/` + +Each task directory must contain: + +- `task.json` +- `commands.txt` +- `summary.md` + +Each gate task directory must also contain: + +- `gate-checklist.md` +- `gate-result.json` + +Each worker task directory must also contain: + +- `worker-report.md` +- `worker-output.patch` +- `evidence-manifest.json` + +If a worker task is intentionally skipped as `noop`, the parent still writes those three files in that task directory as parent-authored `noop` records, writes the `summary.md` rationale, and then writes the task sentinel. No worker is launched in that branch. + +Required sentinels: + +- `.runs/plan-25/sentinels/task-m25-p1-parent-contract-freeze-and-run-init.ok` +- `.runs/plan-25/sentinels/task-m25-g1-window-a-launch-gate.ok` +- `.runs/plan-25/sentinels/task-m25-l1-lifecycle-qa-hardening.ok` +- `.runs/plan-25/sentinels/task-m25-l2-truth-doc-convergence.ok` +- `.runs/plan-25/sentinels/task-m25-g2-window-a-integration-gate.ok` +- `.runs/plan-25/sentinels/task-m25-p2-parent-window-a-integration.ok` +- `.runs/plan-25/sentinels/task-m25-g3-cleanup-launch-gate.ok` +- `.runs/plan-25/sentinels/task-m25-l3-inbox-scope-comment-tightening.ok` +- `.runs/plan-25/sentinels/task-m25-g4-validation-wall-gate.ok` +- `.runs/plan-25/sentinels/task-m25-p3-parent-validation-wall.ok` +- `.runs/plan-25/sentinels/task-m25-p4-parent-closeout-phase.ok` + +`contract-freeze.json` must record at minimum: + +- `authoritative_branch: "feat/host-orchestrator-durable-session"` +- the frozen `turn` / `reattach` / `stop` / `status --json` contract +- the frozen narrow inbox contract +- the exact initial worker cap of `2` +- the single-owner hotspot file path for `L1` +- the rule that only `L1` may expose a tiny runtime fix +- the host-only manual validation ceiling +- the automated-only status for `awaiting_attention` and detached-world follow-up checks + +`merge-order.json` must record at minimum: + +- `integration_order: ["task/m25-l2-truth-doc-convergence", "task/m25-l1-lifecycle-qa-hardening", "task/m25-l3-inbox-scope-comment-tightening"]` +- `l2_acceptance_basis: "frozen_contract_only"` +- `l1_acceptance_basis: "accepted_l2_tree_if_no_contract_drift_else_replay_after_runtime_fix"` +- `l3_acceptance_basis: "accepted_p2_tree_only"` +- `replay_required_if_l1_changes_contract_relevant_wording: true` +- `quarantine_on_hotspot_collision: true` + +## Workstream Plan + +### PLAN-25 Workstream Mapping + +| PLAN-25 workstream | Orchestration tasks | Why this mapping is exact | +| --- | --- | --- | +| Freeze repo truth docs | `task/m25-l2-truth-doc-convergence`, `task/m25-g2-window-a-integration-gate`, `task/m25-p2-parent-window-a-integration` | The docs lane owns only truth convergence in repo docs and packet docs. It does not invent runtime behavior or change tests. | +| Add one same-session lifecycle regression plus explicit parked-status assertions | `task/m25-l1-lifecycle-qa-hardening`, `task/m25-g2-window-a-integration-gate`, `task/m25-p2-parent-window-a-integration` | The code lane owns the shared shell control-suite hotspot and any tiny runtime fix directly exposed by those assertions. | +| Tighten inbox comments only if needed, then publish final validation wall | `task/m25-g3-cleanup-launch-gate`, `task/m25-l3-inbox-scope-comment-tightening`, `task/m25-g4-validation-wall-gate`, `task/m25-p3-parent-validation-wall`, `task/m25-p4-parent-closeout-phase` | Cleanup must wait for merged code plus docs truth. `L3` owns comments only if needed, and the parent owns all validation planning, execution, and run-state artifacts. | + +### Concurrency And Merge Order + +Concurrency rules: + +1. Worker cap is exactly `2` until `g2` completes. +2. Only `L1` and `L2` may start in parallel. +3. `L3` waits for accepted and integrated `L1` plus `L2`. +4. No second code lane is authorized around [agent_public_control_surface_v1.rs](/home/spenser/__Active_code/substrate/crates/shell/tests/agent_public_control_surface_v1.rs). That file is one hotspot and splitting it would create artificial parallelism, merge noise, and contract drift risk. +5. The docs lane is independent enough to start immediately because the public contract is already frozen. It may not change that contract. +6. The cleanup lane is deliberately late because its runtime comment surfaces may overlap the same contract seams `L1` uses, and it depends on the final merged wording from both code and docs truth. + +Why the merge order is `L2` then `L1` then `L3`: + +1. `PLAN-25` freezes repo truth first, so the docs lane is integrated first when it stays within the frozen contract. +2. `L1` is the only lane that can expose a real defect in the lifecycle/status path. If it does, the runtime/test fix lands after the truth freeze and the parent replays `L2` only if wording must change. +3. `L3` cannot run honestly until the merged tree tells one story about contract wording, lifecycle proof, and inbox boundaries. + +Quarantine rules: + +- Quarantine `L1` if it touches files outside its frozen ownership set, adds new public semantics, or makes a broad runtime refactor. +- Quarantine `L1` if it modifies more than one production Rust file without the new assertions proving both changes are necessary. +- Quarantine `L2` if it touches any Rust file, any test file, `PLAN-25.md`, `ORCH_PLAN-25.md`, or `.runs/**`. +- Quarantine `L2` if it claims public inbox productization, world-originated inbox workflows, or detached-world recovery beyond `reattach`. +- Quarantine `L3` if it touches tests, docs, `.runs/**`, or makes semantic code changes instead of comment-only tightening. +- Quarantine any lane that tries to create a third initial code stream around `agent_public_control_surface_v1.rs`. + +### Task Ledger + +| Task ID | Owner | Depends on | Worktree / branch | Deliverable | +| --- | --- | --- | --- | --- | +| `task/m25-p1-parent-contract-freeze-and-run-init` | parent | — | authoritative checkout | frozen contract, run artifacts, lane ownership, merge order | +| `task/m25-g1-window-a-launch-gate` | parent | `p1` | authoritative checkout | launch approval for `L1` and `L2` only | +| `task/m25-l1-lifecycle-qa-hardening` | worker | `g1` | `lifecycle-qa-hardening` / `codex/feat-host-orchestrator-durable-session-m25-lifecycle-qa-hardening` | same-session lifecycle regression, explicit parked-status assertions, tiny runtime fix only if exposed | +| `task/m25-l2-truth-doc-convergence` | worker | `g1` | `truth-doc-convergence` / `codex/feat-host-orchestrator-durable-session-m25-truth-doc-convergence` | repo truth freeze for public contract and honest inbox wording | +| `task/m25-g2-window-a-integration-gate` | parent | `l1`, `l2` | authoritative checkout | acceptance, rejection, or quarantine for Window A | +| `task/m25-p2-parent-window-a-integration` | parent | `g2` | authoritative checkout | merged docs truth then merged lifecycle/status proof | +| `task/m25-g3-cleanup-launch-gate` | parent | `p2` | authoritative checkout | decide whether inbox-scope comment tightening is needed | +| `task/m25-l3-inbox-scope-comment-tightening` | worker or parent-noop | `g3` | `inbox-scope-comment-tightening` / `codex/feat-host-orchestrator-durable-session-m25-inbox-scope-comment-tightening` | comment-only runtime tightening if launched; otherwise parent-authored `noop` records in the `L3` task artifact root | +| `task/m25-g4-validation-wall-gate` | parent | `g3` | authoritative checkout | permission to run the final validation wall after accepted `L3` or parent-recorded `noop` | +| `task/m25-p3-parent-validation-wall` | parent | `g4` | authoritative checkout | exact automated wall results plus recorded manual host-only smoke evidence | +| `task/m25-p4-parent-closeout-phase` | parent | `p3` | authoritative checkout | final artifact audit, acceptance record, terminal run state | + +### Lane Ownership By File Set + +| Lane | Allowed files | Forbidden touch surfaces | +| --- | --- | --- | +| `L1` / lifecycle QA hardening | [crates/shell/tests/agent_public_control_surface_v1.rs](/home/spenser/__Active_code/substrate/crates/shell/tests/agent_public_control_surface_v1.rs), `crates/shell/tests/support/**` only if directly required by that suite, [crates/shell/src/execution/agents_cmd.rs](/home/spenser/__Active_code/substrate/crates/shell/src/execution/agents_cmd.rs) only for a tiny test-exposed lifecycle/status fix, [crates/shell/src/execution/agent_runtime/state_store.rs](/home/spenser/__Active_code/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) only for a tiny test-exposed lifecycle/status fix | every other Rust file, `docs/**`, repo-root truth docs, `llm-last-mile/**`, `.runs/**` | +| `L2` / truth-doc convergence | [HOST_ORCHESTRATOR_INTENDED_BEHAVIOR_TRUTH.md](/home/spenser/__Active_code/substrate/HOST_ORCHESTRATOR_INTENDED_BEHAVIOR_TRUTH.md), [AGENT_ORCHESTRATION_GAP_MATRIX.md](/home/spenser/__Active_code/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md), [docs/USAGE.md](/home/spenser/__Active_code/substrate/docs/USAGE.md), [llm-last-mile/README.md](/home/spenser/__Active_code/substrate/llm-last-mile/README.md), [llm-last-mile/25-host-durable-session-closeout-and-qa-hardening.md](/home/spenser/__Active_code/substrate/llm-last-mile/25-host-durable-session-closeout-and-qa-hardening.md) | every Rust file, every test file, `PLAN-25.md`, `ORCH_PLAN-25.md`, `.runs/**` | +| `L3` / inbox-scope comment tightening | [crates/shell/src/execution/agent_runtime/state_store.rs](/home/spenser/__Active_code/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) comments only, [crates/shell/src/execution/agent_dev_support.rs](/home/spenser/__Active_code/substrate/crates/shell/src/execution/agent_dev_support.rs) comments only, [crates/shell/src/execution/agents_cmd.rs](/home/spenser/__Active_code/substrate/crates/shell/src/execution/agents_cmd.rs) comments only if needed around the `turn` / `reattach` / `stop` split | every test file, every doc file, all run-state and validation artifacts under `.runs/**`, semantic Rust changes | + +### Kickoff Initialization Order + +The parent initializes the run in this exact order: + +1. Create `/home/spenser/__Active_code/substrate/.runs/plan-25/`, `.runs/plan-25/sentinels/`, `.runs/plan-25/quarantine/`, and every `.runs/task-m25-*/` directory. +2. Create `task.json`, `commands.txt`, and `summary.md` in every task directory. +3. Create `gate-checklist.md` and `gate-result.json` in every gate task directory. +4. Create placeholder `worker-report.md`, `worker-output.patch`, and `evidence-manifest.json` in every worker task directory so a later parent-authored `noop` record can reuse the same artifact contract if a worker is never launched. +5. Write `tasks.json` as the canonical launch queue and execution ledger. +6. Write `run-state.json` with `current_phase: "kickoff"`, `worker_cap: 2`, `authoritative_branch: "feat/host-orchestrator-durable-session"`, every task in `pending`, and empty `accepted`, `rejected`, `quarantined`, and `blocked` arrays. +7. Write `contract-freeze.json`, `lane-ownership.json`, `merge-order.json`, and `validation-wall.md`. +8. Freeze the exact public contract and narrow inbox contract in `contract-freeze.json`. +9. Freeze the rule that [agent_public_control_surface_v1.rs](/home/spenser/__Active_code/substrate/crates/shell/tests/agent_public_control_surface_v1.rs) belongs to `L1` alone. +10. Freeze the host-only manual validation sequence and the automated-only status of `awaiting_attention` and detached-world fail-closed checks. +11. Seed all worktrees from the exact same post-`p1` tree. +12. Record the authoritative branch, worktree paths, lane ownership, and retry budget in `session-log.md`. + +### Launch And Integration Gates + +#### `task/m25-p1-parent-contract-freeze-and-run-init` + +Parent only. + +Scope: + +1. Create the full `.runs/plan-25` tree and every per-task artifact directory. +2. Freeze the exact `turn` / `reattach` / `stop` / `status --json` contract. +3. Freeze the narrow inbox contract and the ban on public inbox product claims. +4. Freeze the real concurrency cap of `2` for the opening window. +5. Freeze that `L1` alone owns the test hotspot and may expose only a tiny test-driven runtime fix. +6. Freeze that `L2` is docs-only and `L3` is comment-only if needed. + +Acceptance: + +- all parent-owned run-state artifacts exist, +- lane ownership is explicit, +- merge order is explicit, +- `/home/spenser/__Active_code/substrate/.runs/plan-25/sentinels/task-m25-p1-parent-contract-freeze-and-run-init.ok` exists. + +#### `task/m25-g1-window-a-launch-gate` + +Parent only. + +Checks: + +1. `p1` is accepted. +2. `L1` and `L2` worktrees are seeded from the exact same post-`p1` tree. +3. `L1` prompt explicitly says: + - one same-session lifecycle regression, + - explicit parked and `awaiting_attention` status assertions, + - detached-world fail-closed must stay green, + - no speculative runtime changes, + - no file ownership beyond the frozen set. +4. `L2` prompt explicitly says: + - freeze docs truth only, + - do not productize inbox, + - do not rewrite the public recovery contract, + - do not touch Rust, tests, or plan-controller files. +5. `L3` is not launched. + +Acceptance: + +- only `L1` and `L2` are authorized to start, +- `/home/spenser/__Active_code/substrate/.runs/plan-25/sentinels/task-m25-g1-window-a-launch-gate.ok` exists. + +#### `task/m25-g2-window-a-integration-gate` + +Parent only. + +Checks: + +1. `L1` and `L2` each returned `worker-report.md`, `worker-output.patch`, `commands.txt`, and `evidence-manifest.json`. +2. `L1` added exactly the lifecycle/status proof asked for: + - one same-session lifecycle regression on one orchestration session id, + - explicit parked `status --json` field assertions, + - explicit `awaiting_attention` `status --json` field assertions. +3. `L1` preserved detached-world fail-closed behavior. +4. Any `L1` production-code change is tiny, directly exposed by the new assertions, and limited to the frozen runtime files. +5. `L2` removed stale "unfinished" wording around parked `status`, `reattach`, and `stop`. +6. `L2` narrowed inbox wording to shipped persistence, posture normalization, internal ack/dismiss, and dev-support/test ingress only. +7. `L2` did not touch Rust files, tests, or plan-controller files. + +Acceptance: + +- each Window A lane is marked `accepted`, `rejected`, or `quarantined`, +- any required `L2` replay is decided here, +- `/home/spenser/__Active_code/substrate/.runs/plan-25/sentinels/task-m25-g2-window-a-integration-gate.ok` exists. + +#### `task/m25-p2-parent-window-a-integration` + +Parent only. + +Integration sequence: + +1. Integrate accepted `L2` first. +2. Integrate accepted `L1` second. +3. If `L1` changed any contract-relevant wording basis, replay `L2` on top of the accepted `L1` tree before closing `p2`. +4. Run the targeted automated checks immediately after `L1` lands. + +Minimum commands: + +```bash +git cherry-pick +git cherry-pick +cargo test -p shell --test agent_public_control_surface_v1 -- --nocapture +cargo test -p shell agent_runtime::state_store -- --nocapture +``` + +Acceptance: + +- docs truth and lifecycle/status proof coexist on one merged tree, +- targeted shell control and state-store coverage are green, +- `/home/spenser/__Active_code/substrate/.runs/plan-25/sentinels/task-m25-p2-parent-window-a-integration.ok` exists. + +#### `task/m25-g3-cleanup-launch-gate` + +Parent only. + +Checks: + +1. `p2` is accepted. +2. The parent decides whether comment tightening is actually needed. +3. If no comment drift exists, the parent does not launch a worker. Instead, the parent writes a `noop` record into `/home/spenser/__Active_code/substrate/.runs/task-m25-l3-inbox-scope-comment-tightening/`: + - `summary.md` with the `noop` rationale, + - `worker-report.md` marked parent-authored `noop`, + - empty `worker-output.patch`, + - `evidence-manifest.json` marked `noop`, + - `.runs/plan-25/sentinels/task-m25-l3-inbox-scope-comment-tightening.ok`. +4. If comment drift exists, `L3` is launched on the accepted `p2` tree with comment-only scope. +5. `L3` prompt explicitly forbids semantic Rust changes and all doc/test changes. + +Acceptance: + +- `L3` is either launched narrowly or marked `noop`, +- `/home/spenser/__Active_code/substrate/.runs/plan-25/sentinels/task-m25-g3-cleanup-launch-gate.ok` exists. + +#### `task/m25-g4-validation-wall-gate` + +Parent only. + +Checks: + +1. `L3` is accepted or recorded as `noop`. +2. `validation-wall.md` lists the exact automated commands and the exact host-only manual smoke path. +3. `validation-wall.md` explicitly records that `awaiting_attention` visibility and detached-world fail-closed remain automated-only checks. +4. No quarantined output remains unresolved. + +Acceptance: + +- the final wall can run once on the fully merged tree, +- `/home/spenser/__Active_code/substrate/.runs/plan-25/sentinels/task-m25-g4-validation-wall-gate.ok` exists. + +#### `task/m25-p3-parent-validation-wall` + +Parent only. + +Scope: + +1. Run the exact automated validation wall. +2. Run and record the host-only manual lifecycle smoke path. +3. Confirm all acceptance checks before closeout. + +Minimum automated commands: + +```bash +cargo test -p shell --test agent_public_control_surface_v1 -- --nocapture +cargo test -p shell agent_runtime::state_store -- --nocapture +cargo test --workspace -- --nocapture +``` + +Acceptance: + +- automated wall is green on the final merged tree, +- manual host-only smoke evidence is recorded, +- `/home/spenser/__Active_code/substrate/.runs/plan-25/sentinels/task-m25-p3-parent-validation-wall.ok` exists. + +#### `task/m25-p4-parent-closeout-phase` + +Parent only. + +Scope: + +1. Audit all sentinels and required artifacts. +2. Confirm `blocked.json` is absent. +3. Record final accepted lanes, quarantines, command outcomes, and any residual follow-up. +4. Mark the run complete only if every acceptance check in this controller is satisfied. + +Acceptance: + +- terminal run state is written, +- accepted scope matches `PLAN-25`, +- `/home/spenser/__Active_code/substrate/.runs/plan-25/sentinels/task-m25-p4-parent-closeout-phase.ok` exists. + +## Context-Control Rules + +1. The parent owns full context. Workers get only the minimum contract excerpt, owned file list, forbidden surface list, exact commands, and artifact return format needed for their lane. +2. The parent never hands `L1` a docs brief and never hands `L2` the shared test hotspot. This preserves the real code/docs split. +3. `L1` receives the frozen lifecycle/status acceptance bullets and the tiny-runtime-fix rule, not the entire repo history. +4. `L2` receives only the frozen public contract, the frozen inbox contract, and the exact truth files it owns. It does not receive latitude to reinterpret product scope. +5. `L3` receives only the accepted `p2` diff summary plus the exact runtime comment targets that still drift. If there is no drift, the parent writes the `noop` record and `L3` is never launched. +6. Every worker prompt must include: + - task ID, + - attempt number, + - worktree path, + - branch, + - allowed files, + - forbidden files, + - frozen contract clauses relevant to that lane, + - exact commands to run, + - retry budget, + - required return artifacts, + - sentinel path. +7. Every launched worker return must include: + - changed files list, + - commands run with exit codes, + - explicit attempt classification: `clean`, `retryable`, or `blocked`, + - unresolved assumptions, + - `worker-output.patch`, + - `worker-report.md`, + - `evidence-manifest.json`. +8. For a `noop` worker task, the parent writes the task-local artifact trio instead of a worker. Those files must explicitly say `parent-authored noop` so the artifact contract stays intact without implying worker-owned output. +9. The parent writes all `.runs/plan-25/**` artifacts before, between, and after worker attempts. Workers never write orchestration state. +10. If a lane proposes widening scope, the parent does not negotiate in-branch. The parent stops the run or reissues a narrower prompt after updating the run-state record. +11. Parent integration notes must call out explicitly whether `L1` exposed a runtime defect or stayed test-only. That distinction matters for final review. + +## Tests And Acceptance + +### Exact Automated Validation Wall + +The final automated wall is frozen to: + +```bash +cargo test -p shell --test agent_public_control_surface_v1 -- --nocapture +cargo test -p shell agent_runtime::state_store -- --nocapture +cargo test --workspace -- --nocapture +``` + +These commands must run on the final merged tree, in that order, unless the parent records a stricter superset in `validation-wall.md`. + +### Host-Only Manual Smoke Path + +The manual smoke path is host-only and must be recorded by the parent. Use one real host backend and one real durable orchestration session: + +```bash +substrate agent start --backend --prompt "first" --json +substrate agent status --json +substrate agent turn --session --backend --prompt "next" --json +substrate agent status --json +substrate agent reattach --session --json +substrate agent stop --session --json +substrate agent status --json +``` + +The parent records success only if all of the following are true: + +1. `start` establishes one durable orchestration session. +2. The first `status --json` shows the session as a live parked session with authoritative runtime fields populated. +3. `turn` reuses the same orchestration-session id and does not require `reattach` first. +4. The second `status --json` still shows that same session with authoritative runtime fields populated. +5. `reattach` restores attached ownership without submitting a prompt. +6. `stop` closes that same durable session terminally. +7. The final `status --json` no longer presents the session as a live non-terminal durable session. + +### Automated-Only Checks + +These remain automated-only for this slice and must not be faked into a manual path: + +1. `awaiting_attention` command-level visibility. +2. Detached-world follow-up fail-closed behavior with `reattach` guidance. + +Reason: + +- there is no public inbox producer surface for creating `awaiting_attention` manually without dev-support/test ingress, +- detached-world follow-up is fixture-heavy and already belongs in automated regression coverage. + +### Acceptance Checklist + +The run is complete only when every item below is true: + +- repo truth docs no longer say parked `status`, `reattach`, or `stop` are unfinished, +- inbox docs say exactly what is shipped today and nothing more, +- one same-session lifecycle regression proves parked `status`, parked `turn`, `reattach`, and `stop` on one orchestration session id, +- command-level parked rows show live-runtime `posture`, `attached_participant_id`, and `pending_inbox_count`, +- command-level `awaiting_attention` rows show the same live-runtime fields, +- detached-world follow-up still fails closed with `reattach` guidance, +- any production-code change made by `L1` is tiny and directly exposed by the new assertions, +- cleanup comments, if any, do not imply a public inbox product surface, +- the exact automated validation wall is green, +- the manual host-only lifecycle validation sequence is recorded, +- all `.runs/plan-25` artifacts and sentinels required by this controller exist. + +## Assumptions + +1. The live branch `feat/host-orchestrator-durable-session` already contains the intended durable-session model; this run is closeout and contract hardening, not architectural expansion. +2. The existing shell control suite can absorb one more lifecycle/status regression without needing a new harness. +3. Any production runtime fix exposed by `L1` will be narrow enough to fit inside [agents_cmd.rs](/home/spenser/__Active_code/substrate/crates/shell/src/execution/agents_cmd.rs) or [state_store.rs](/home/spenser/__Active_code/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) without reopening the product model. +4. Repo truth can be made consistent without editing `PLAN-25.md` or this controller. +5. The parent has a usable host backend available for the manual smoke path. +6. If the parent discovers the docs cannot be made truthful without inventing new product behavior, the correct action is to block the run rather than widen scope. diff --git a/llm-last-mile/PLAN-02.md b/llm-last-mile/PLAN-02.md new file mode 100644 index 000000000..2fc34dbf6 --- /dev/null +++ b/llm-last-mile/PLAN-02.md @@ -0,0 +1,992 @@ + +# PLAN-02: Session Participant Record Cutover + +Source file: [02-session-participant-record.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/02-session-participant-record.md) +Branch: `feat/orchestration-session-identity` +Plan type: backend-only, no UI scope +Review posture: `/autoplan` completeness pass with `/plan-eng-review` structure and rigor +Status: execution-ready + +## What This Plan Does + +`PLAN-01` fixed the parent session authority boundary. The shell now persists a real orchestration session record under [orchestration_session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/orchestration_session.rs:23) and [state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:34). + +The next bottleneck is the child runtime model. The live record is still an orchestrator-shaped manifest in [session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs:36), and the main operator surface still collapses live state by `agent_id` in [agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:617). That means the repo can talk about member sessions, but the authoritative runtime store still cannot represent them honestly. + +This slice replaces that narrow child-handle model with a participant record that can represent: + +1. the host-scoped orchestrator participant, +2. future host-scoped or world-scoped member participants under the same `orchestration_session_id`, +3. replacement lineage after invalidation or restart, and +4. additive participant correlation on status and `AgentEvent` surfaces. + +The goal is concrete: + +1. introduce a canonical participant record and participant store, +2. read legacy `handles/*.json` compatibly during rollout, +3. stop collapsing live state to one row per `agent_id`, +4. keep toolbox authorization host-orchestrator-only and fail-closed, and +5. add participant identity to runtime events without breaking existing tuple consumers. + +This is the participant-model slice. It is not the shared-world owner slice, not the restart-generation slice, and not the grouped session-store rewrite. + +## Scope Challenge + +### Why this is the right second slice + +The repo already has the parent session object from `PLAN-01`. Without this slice, every later world/member feature still has to pretend that one orchestrator-shaped handle is "the runtime." That is the wrong source of truth for multi-participant orchestration. + +What matters is not inventing more machinery. What matters is making the already-existing runtime store honest. Right now the docs can describe members and lineage, but the authoritative live registry still cannot. That mismatch will bite status, toolbox, restart invalidation, and any future `session_fork` or `session_resume` work. + +### What already exists + +| Sub-problem | Existing code | Reuse or replace | +|---|---|---| +| Parent orchestration identity | [orchestration_session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/orchestration_session.rs:23) | Reuse exactly | +| Atomic JSON persistence | [state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:65) | Reuse exactly | +| Live-ownership rules | [session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs:179) | Reuse, but move behind participant semantics | +| Orchestrator bootstrap writer | [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:1374) | Extend | +| Role vocabulary | [mapping.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/mapping.rs:5) | Reuse exactly | +| Status tuple parsing and nested-row correlation | [agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:457) | Extend, do not replace | +| Parent-gated live orchestrator lookup | [state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:225) and [agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:1072) | Rework into participant query helpers | +| Canonical event envelope | [crates/common/src/agent_events.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/common/src/agent_events.rs:71) | Reuse with additive fields only | +| Contract fixtures | [agent_successor_contract_ahcsitc0.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_successor_contract_ahcsitc0.rs:263) | Reuse and widen | + +### Minimum change that achieves the goal + +Do this, and only this: + +- replace the orchestrator-specific manifest type with a participant record type, +- add a `participants/` directory and participant query APIs, +- keep legacy `handles/` as a compatibility read path during rollout, +- convert orchestrator bootstrap to write the participant shape, +- make status/toolbox/doctor consume participant records first, +- add additive participant fields to `AgentEvent`, +- update docs and tests. + +Do not: + +- invent a long-lived agent-hub service, +- redesign nested gateway status rows, +- pull slice `03`, `04`, `05`, or `06` into this plan, +- add a cache layer to the runtime store, +- invent a generic member lifecycle framework before the shell owns a concrete member launch seam. + +### Complexity check + +This slice touches more than 8 files. That is normally a smell. Here it is acceptable because the authority seam already crosses: + +- `crates/common/src/agent_events.rs` +- `crates/shell/src/execution/agent_runtime/` +- `crates/shell/src/repl/async_repl.rs` +- `crates/shell/src/execution/agents_cmd.rs` +- `crates/shell/tests/agent_successor_contract_ahcsitc0.rs` +- `crates/common/tests/agent_hub_event_envelope_schema.rs` +- `docs/USAGE.md` +- `docs/TRACE.md` +- pack/ADR docs + +The guardrail is simple: no new daemon, no new cache, no speculative registry abstraction, no generic "participant manager" service object. Keep it boring. + +### Search-before-building posture + +`[Layer 1]` wins here. The repo already has the right building blocks: + +- atomic JSON writes, +- parent session persistence, +- liveness validation, +- role constants, +- fail-closed operator surfaces, +- tuple-compatible event schema. + +The correct move is to reshape those into a participant model, not to introduce a second persistence engine or a runtime cache. This is one of those classic software moments where the fancy option is worse. A 200-line cache to avoid reading a few small JSON files would be software comedy. + +### Hard non-goals + +- shared-world ownership authority from [03-shared-world-ownership-linux-first.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/03-shared-world-ownership-linux-first.md) +- world-binding promotion from slice `04` +- restart invalidation behavior from slice `05` +- session-centric grouped store layout from slice `06` +- public resume/fork/stop CLI productization +- nested gateway record redesign + +## Architecture Contract + +### No-ambiguity rules + +These are hard rules: + +1. There is exactly one canonical runtime child record type after this slice: a participant record. "Manifest" becomes a compatibility detail, not the public model. +2. `participant_id` is the canonical child identity. No status or toolbox surface may infer identity from `agent_id` alone. +3. Parent orchestration sessions remain the sole authority for the control-plane session. Participants are children, not replacements for the parent record. +4. `role=orchestrator` plus `execution.scope=host` is the only combination that can authorize toolbox publication. +5. World-scoped participants require both `world_id` and `world_generation`. Host-scoped participants omit both. +6. Doctor stays pre-runtime. It may validate participant files when present, but it must not require a live participant registry for a healthy result. +7. Event-schema changes are additive only. Existing consumers that ignore participant fields must keep working. +8. Legacy `handles/*.json` files are read compatibly during rollout, but new writes go to `participants/*.json` only. +9. Every participant lookup that could return more than one valid answer must fail closed. No "latest by agent" heuristics. +10. The absence of a production member-launch seam is not permission to keep the store orchestrator-shaped. It is permission to ship the participant-capable substrate first and wire later slices into it. + +### Target data model + +Replace [AgentRuntimeSessionManifest](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs:98) with a participant-centric shape: + +```rust +#[derive(Clone, Debug, Serialize, Deserialize, PartialEq, Eq)] +pub(crate) struct AgentRuntimeParticipantExecution { + pub scope: AgentExecutionScope, +} + +#[derive(Clone, Debug, Serialize, Deserialize, PartialEq, Eq)] +pub(crate) struct AgentRuntimeParticipantHandle { + pub participant_id: String, + pub orchestration_session_id: String, + pub agent_id: String, + pub backend_id: String, + pub role: String, // orchestrator | member + pub protocol: String, + pub execution: AgentRuntimeParticipantExecution, + pub state: AgentRuntimeSessionState, + pub opened_at: DateTime, + pub last_transition_at: DateTime, + pub world_id: Option, + pub world_generation: Option, + pub parent_participant_id: Option, + pub resumed_from_participant_id: Option, + pub orchestrator_participant_id: Option, +} + +#[derive(Clone, Debug, Serialize, Deserialize, PartialEq, Eq)] +pub(crate) struct AgentRuntimeParticipantRecord { + #[serde(flatten)] + pub handle: AgentRuntimeParticipantHandle, + pub internal: AgentRuntimeSessionInternal, +} +``` + +Implementation rules: + +- `participant_id` may reuse the current `session_handle_id` format (`ash_`) for the first cut. No need to spend an innovation token on ID syntax. +- `AgentRuntimeSessionInternal` stays in place. It already holds the right ephemeral ownership fields. +- `orchestrator_participant_id` is `None` on the orchestrator participant and required on member participants. +- `parent_participant_id` is used only for forked or explicitly derived children. +- `resumed_from_participant_id` is used only for replacement lineage after invalidation or restart. + +### Ownership mode generalization + +Current `ownership_mode` in [session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs:78) is effectively hard-coded to attached orchestrator control. That has to widen: + +```rust +enum AgentRuntimeOwnershipMode { + AttachedControl, + MemberRuntime, + Replaced, +} +``` + +Semantics: + +- `attached_control`: current shell-owned orchestrator lifecycle +- `member_runtime`: live member participant owned by shell/runtime logic +- `replaced`: prior participant retained only for lineage and audit + +Do not use free-form strings forever. Keep the first patch additive if needed, then move to a proper enum before this slice closes. + +### Participant lifecycle contract + +`PLAN-01` already owns the parent session lifecycle. This slice must make participant lifecycle line up with it instead of freelancing its own truth. + +| Moment | Parent state | Participant state | Notes | +|---|---|---|---| +| parent exists, no child yet | `Allocating` | no participant yet | child creation never becomes the thing that legitimizes the parent | +| participant file first persisted | `Allocating` | `Allocating` | bootstrap may write the child record before durable control is proven | +| durable control proven | `Active` | `Ready` | this is the first point where the participant may become authoritative-live | +| command loop actively serving | `Active` | `Running` | status and event emission may surface the participant | +| graceful shutdown requested | `Stopping` | `Stopping` | parent and participant transition in the same shutdown block | +| graceful shutdown completed | `Stopped` | `Stopped` | historical row remains queryable, but never live | +| ownership lost after prior success | `Invalidated` | `Invalidated` | toolbox and live status must fail closed immediately | +| replacement lineage recorded later | `Active` on the replacement parent-child pair | `Replaced` on the old participant | replaced rows remain auditable but never authoritative-live | + +Hard rules: + +- a participant is authoritative-live only when both its own ownership checks pass and its parent session is `Active` +- `Replaced`, `Stopped`, `Failed`, and `Invalidated` participants persist for audit, but they do not suppress a distinct live participant unless the `participant_id` matches exactly +- this slice does not add a participant-specific restart state machine; it adds the identity and lineage fields that later restart work will consume + +### On-disk layout and compatibility rules + +Target directory: + +```text +~/.substrate/run/agent-hub/participants/.json +``` + +Compatibility rules: + +1. Read both: + - `~/.substrate/run/agent-hub/handles/*.json` + - `~/.substrate/run/agent-hub/participants/*.json` +2. Write only: + - `~/.substrate/run/agent-hub/participants/*.json` +3. Upgrade legacy handle files in memory using: + - `session_handle_id -> participant_id` + - `parent_session_handle_id -> parent_participant_id` + - `resumed_from_session_handle_id -> resumed_from_participant_id` + - `orchestrator_participant_id = None` for legacy orchestrator rows +4. When both a legacy handle and a participant record describe the same child identity, prefer the participant file. +5. Do not attempt an eager file-system migration step in this slice. Read-compat is enough. + +### Required store APIs + +Extend [state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:20) with participant-native helpers: + +```rust +pub(crate) fn participants_dir(&self) -> PathBuf; +pub(crate) fn persist_participant(&self, participant: &AgentRuntimeParticipantRecord) -> Result<()>; +pub(crate) fn load_participant(&self, participant_id: &str) -> Result>; +pub(crate) fn list_participants(&self) -> Result>; +pub(crate) fn list_live_participants(&self) -> Result>; +pub(crate) fn list_live_participants_for_session( + &self, + orchestration_session_id: &str, +) -> Result>; +pub(crate) fn resolve_live_orchestrator_participant( + &self, + orchestrator_agent_id: &str, +) -> Result>; +pub(crate) fn validate_participant_record( + &self, + participant: &AgentRuntimeParticipantRecord, +) -> Result<()>; +``` + +Rules: + +- `list_live_participants()` keeps the existing ownership checks from [session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs:224). +- `resolve_live_orchestrator_participant()` filters to `role=orchestrator` and `execution.scope=host`, then parent-gates through the orchestration session record. +- any ambiguous live answer is an error, not a winner selection. +- operator-path helpers may scan the store on demand, but event emission paths must not. + +### Status contract + +This is the core bug. + +Today: + +- trace-derived pure-agent rows are keyed by `(orchestration_session_id, agent_id)` in [agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:434), +- live runtime rows are collapsed by `agent_id` in [agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:617), +- fallback suppression is also keyed by `agent_id` in [agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:648). + +That loses same-agent siblings by construction. + +Required change: + +1. Add additive `participant_id` to every pure-agent status row. +2. Make participant-aware rows key by `(orchestration_session_id, participant_id)`. +3. Keep legacy trace fallback for pre-participant events only, and isolate it from live participant rows. +4. Sort `sessions` by: + - `orchestration_session_id` + - `participant_id` + - `agent_id` +5. Keep nested rows tied to `parent_run_id`, but widen their parent shape to include `parent.participant_id`. + +Recommended status row shape: + +```json +{ + "participant_id": "ash_001", + "orchestration_session_id": "sess_001", + "agent_id": "codex", + "backend_id": "cli:codex", + "client": "codex", + "router": "agent_hub", + "protocol": "uaa.agent.session", + "execution": { "scope": "world" }, + "role": "member", + "last_event_at": "2026-04-29T12:55:04Z", + "world_id": "world-123", + "world_generation": 7 +} +``` + +This is additive. Existing consumers can ignore `participant_id`, but the operator surface stops being lossy. + +### Toolbox contract + +Toolbox stays boring and strict. + +Required changes: + +- replace the manifest-only resolver in [agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:929) with a participant query, +- require: + - `role=orchestrator` + - `execution.scope=host` + - authoritative live ownership + - active parent orchestration session +- keep endpoint derivation on `orchestration_session_id`. + +Member participants never authorize toolbox publication. Not once. Not "if they're the newest." Not if the trace looks convincing. No cute shortcuts. + +### Doctor contract + +Doctor remains pre-runtime. + +Required change: + +- after the current config, policy, orchestrator, and world checks in [agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:1302), add a passive participant-store validation check: + - `pass` when no participant files exist + - `pass` when participant files parse and obey role/scope/world invariants + - `fail` when participant files exist but are malformed or impossible + +Examples of impossible participant state: + +- `role=orchestrator` with `execution.scope=world` +- `role=member` with missing `orchestrator_participant_id` +- `execution.scope=world` with only one of `world_id` or `world_generation` +- `resumed_from_participant_id` pointing at itself + +### Event publication contract + +Additive fields on [AgentEvent](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/common/src/agent_events.rs:74): + +```rust +pub participant_id: Option, +pub parent_participant_id: Option, +pub resumed_from_participant_id: Option, +``` + +Producer rules: + +- [translate_wrapper_event()](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:2187) and [build_runtime_message_event()](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:2261) populate participant fields from the live participant record. +- world restart alerts in [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:2473) and [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:2518) stay additive. If no participant context exists, they omit participant fields instead of inventing them. +- nested gateway-backed records remain separate. They do not become participant rows. + +### Member-writer posture + +There is no concrete production member-launch seam on this branch yet. That is fine. The plan should not invent one. + +What this slice must do: + +- make the participant model capable of representing members, +- add constructors and store APIs for member participants, +- cover member rows in fixtures and contract tests, +- keep the first production writer hookup small and shell-side once slice `03` or `04` provides the concrete member lifecycle. + +What this slice must not do: + +- create a fake generic member runtime service just so the plan can say it wrote "something." + +That would be under-tested complexity pretending to be progress. + +## Architecture Diagrams + +### Runtime store and consumer flow + +```text +CURRENT +======= +Parent session: sessions/.json + │ + └── Child runtime: handles/.json + │ + ├── status live path collapses by agent_id + ├── toolbox resolves one live orchestrator manifest + └── events carry no participant lineage + +TARGET +====== +Parent session: sessions/.json + │ + ├── participant: participants/.json role=orchestrator scope=host + ├── participant: participants/.json role=member scope=host + └── participant: participants/.json role=member scope=world + │ + ├── status resolves by participant_id + ├── toolbox filters to live host orchestrator participant only + ├── doctor validates participant files when present + └── AgentEvent publishes participant_id + lineage fields additively +``` + +### Status and toolbox lookup + +```text +substrate agent status --json + │ + ├── read live participants + │ ├── parent-gate orchestrator row + │ ├── include member rows without collapsing siblings + │ └── suppress legacy trace fallback only when a matching participant_id exists + │ + └── read trace agent events + ├── participant-aware rows key by (session, participant_id) + ├── legacy rows stay fallback-only + └── nested rows correlate by parent_run_id + parent.participant_id + +substrate agent toolbox status|env + │ + └── resolve live orchestrator participant + ├── role=orchestrator + ├── execution.scope=host + ├── parent session active + └── endpoint => .sock +``` + +## Implementation Plan + +### Ordered execution checklist + +1. Lock the participant contract in the pack/ADR docs and status JSON examples. +2. Replace the child manifest model with a participant record plus compatibility parser. +3. Add `participants/` storage and participant query helpers to the runtime store. +4. Convert orchestrator bootstrap to write participant records. +5. Update status, toolbox, and doctor to consume participant records first. +6. Add additive participant lineage to `AgentEvent` and runtime producers. +7. Extend tests and docs. Remove handle-only assumptions once coverage is green. + +### Execution sequencing and merge gates + +This slice only stays boring if the sequencing is strict: + +1. Land the type rename plus compatibility parser first. + Exit gate: legacy `handles/*.json` fixtures still parse into the participant model. +2. Land participant-native store helpers second. + Exit gate: read-path tests prove `participants/` wins over `handles/` for the same child identity, and ambiguous live answers fail closed. +3. Cut over the orchestrator writer third. + Exit gate: bootstrap writes `participants/*.json` only, and parent-child linkage still resolves through the existing parent record. +4. Cut over operator surfaces fourth. + Exit gate: `status`, `toolbox`, and `doctor` all consume participant-native helpers and no remaining live projection deduplicates by `agent_id`. +5. Land event schema and runtime publication after the participant surface is stable. + Exit gate: additive participant lineage fields roundtrip without breaking legacy consumers. +6. Run the contract test and doc sweep last. + Exit gate: status JSON examples, trace docs, and integration fixtures all use participant terminology and sibling-safe semantics. + +Do not merge a later step while it still depends on an unstabilized helper signature from an earlier step. That is how a clean slice turns into a half-renamed swamp. + +### Workstream 1: Participant model and compatibility parser + +Files: + +- [crates/shell/src/execution/agent_runtime/session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs:36) +- [crates/shell/src/execution/agent_runtime/mod.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/mod.rs:1) +- [docs/project_management/packs/draft/agent-hub-core-successor-identity-tuple-compatible/agent-hub-session-protocol-spec.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/project_management/packs/draft/agent-hub-core-successor-identity-tuple-compatible/agent-hub-session-protocol-spec.md) +- [docs/project_management/packs/draft/agent-hub-core-successor-identity-tuple-compatible/contract.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/project_management/packs/draft/agent-hub-core-successor-identity-tuple-compatible/contract.md) + +Do: + +- rename or replace `AgentRuntimeSessionHandle` / `AgentRuntimeSessionManifest` with participant naming, +- keep a compatibility deserializer for legacy handle JSON, +- add constructors: + - `new_orchestrator_participant(...)` + - `new_member_participant(...)` + - `new_replacement_participant(...)` +- codify role/scope/world invariants close to the model. + +### Workstream 2: Participant store and parent-gated queries + +Files: + +- [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:20) + +Do: + +- add `participants_dir()` and participant persistence helpers, +- read `participants/*.json` first, `handles/*.json` second, +- keep parent orchestration-session APIs unchanged, +- replace `find_live_orchestrator()` with participant-native query helpers, +- expose a validation helper reused by doctor and tests. + +### Workstream 3: Orchestrator writer cutover + +Files: + +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:1374) + +Do: + +- allocate `participant_id` instead of `session_handle_id`, +- persist participant records under `participants/`, +- update parent `active_session_handle_id` naming only if the parent record is widened in this slice; otherwise keep the serialized parent field stable and document that it now points to the active participant id, +- thread the participant snapshot through ready/running/stopping/invalidated transitions. + +Recommendation: + +Keep the parent record field name stable for this slice if renaming it would force a bigger cross-packet doc churn. The important thing is the child identity model, not winning a naming purity contest today. + +### Workstream 4: Status, toolbox, and doctor cutover + +Files: + +- [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:406) +- [docs/USAGE.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/USAGE.md) +- [docs/TRACE.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/TRACE.md) +- [docs/project_management/adrs/draft/ADR-0045-orchestration-toolbox-internal-mcp-identity-trace-contract.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/project_management/adrs/draft/ADR-0045-orchestration-toolbox-internal-mcp-identity-trace-contract.md) + +Do: + +- add `participant_id` to status rows, +- stop collapsing live rows by `agent_id`, +- widen nested parent shape to include `parent.participant_id`, +- make toolbox resolve from participant query helpers, +- add passive doctor validation for present participant files. + +### Workstream 5: Event envelope and runtime emission + +Files: + +- [crates/common/src/agent_events.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/common/src/agent_events.rs:71) +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:2187) +- [crates/common/tests/agent_hub_event_envelope_schema.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/common/tests/agent_hub_event_envelope_schema.rs) + +Do: + +- add additive participant fields to `AgentEvent`, +- populate them on pure-agent runtime events, +- keep omission rules strict when lineage does not apply, +- document that legacy traces may omit participant fields. + +### Workstream 6: Contract tests and docs + +Files: + +- [crates/shell/tests/agent_successor_contract_ahcsitc0.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_successor_contract_ahcsitc0.rs) +- [docs/USAGE.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/USAGE.md) +- [docs/TRACE.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/TRACE.md) + +Do: + +- add participant fixtures, +- add sibling-same-agent status coverage, +- add malformed participant-file doctor coverage, +- update status JSON examples and trace docs, +- remove wording that says live discovery is manifest-backed. + +## Architecture Review Findings + +### Finding 1 + +`[P1] (confidence: 10/10) [agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:434) and [agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:617) collapse runtime truth by `(orchestration_session_id, agent_id)` and then by `agent_id` alone.` + +This is the main correctness bug. Two live member participants for the same agent under one orchestration session cannot both survive projection. The plan fixes it by making `participant_id` the canonical child key on both live and trace-backed rows. + +Recommendation: accept this as the primary slice objective and keep the fix explicit. No heuristics. No "latest row wins" fallback for participant-aware data. + +### Finding 2 + +`[P1] (confidence: 9/10) [session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs:106) still constructs every child record as `role=orchestrator` with `ownership_mode=attached_control`.` + +That makes the current model semantically incapable of representing member state. The plan resolves this by renaming the model, adding member constructors, and widening `ownership_mode`. + +Recommendation: do not bolt member fields onto an orchestrator-named type. Rename the thing and move on. + +### Finding 3 + +`[P1] (confidence: 9/10) [state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:202) and [state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:225) still expose orchestrator-specific lookup helpers.` + +The store can validate one active orchestrator, but it cannot answer the real questions the later packet needs: + +- list live participants for one orchestration session, +- load one participant by id, +- validate participant role/scope combinations, +- keep same-agent siblings distinct. + +Recommendation: make the store participant-native now. Do not stack more logic on top of orchestrator-only helpers. + +### Finding 4 + +`[P2] (confidence: 8/10) [AgentEvent](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/common/src/agent_events.rs:74) carries no participant lineage, while runtime producers in [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:2187) already stamp role/backend/world identity.` + +Without `participant_id`, later restart/fork lineage becomes hard to correlate. The plan resolves this with additive fields only. + +Recommendation: add the fields now while the participant model is being introduced. Waiting until slice `05` would force a second schema churn. + +## Code Quality Review + +This slice has a real DRY risk because it is migrating a live type, a store boundary, and operator surfaces at the same time. The plan needs explicit guardrails so the implementation does not leave two truths behind. + +### Finding 1 + +`[P2] (confidence: 9/10) The role/scope/world invariants can easily get duplicated across constructors, store validation, doctor checks, and status projection if the implementation is careless.` + +That is how these migrations rot. One path starts rejecting `role=member` without `orchestrator_participant_id`, another path forgets, and six weeks later the store contains impossible state that only one command notices. + +Recommendation: put participant invariants in one model-level validation path and have store, doctor, and tests call it. Do not hand-roll the same rules three times. + +### Finding 2 + +`[P2] (confidence: 8/10) Legacy-handle compatibility can sprawl if the upgrade logic leaks beyond the read boundary in [state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:20).` + +If `agents_cmd.rs`, `async_repl.rs`, and tests all know how to translate `session_handle_id` into `participant_id`, the repo will carry two mental models for far too long. Not great. + +Recommendation: keep the legacy upgrade shim at the persistence boundary. Everyone above the store should speak participant terminology only. + +### Finding 3 + +`[P2] (confidence: 8/10) Terminology drift will create fake complexity if public APIs keep saying "manifest" or "handle" after the canonical type becomes a participant record.` + +This matters because later slices will read the new helper names as the contract. If the code says `find_live_orchestrator_manifest()` but returns a participant, the type system compiles and the humans still lose. + +Recommendation: reserve `handles/` and `manifest` wording for the compatibility path and on-disk legacy explanation only. New helpers, docs, and test names should say `participant`. + +## Error & Rescue Registry + +These are the failure cases the implementation must rescue explicitly instead of letting them become quiet operator drift. + +| Failure | Detection point | Required rescue behavior | Fail-open forbidden? | +|---|---|---|---| +| malformed `participants/*.json` | participant load path, doctor | reject the file, surface explicit validation failure, keep toolbox unauthorized | yes | +| legacy handle file cannot upgrade cleanly | compatibility parser | omit that record from live selection, return a clear error on validation surfaces, preserve other healthy rows | yes | +| same child identity exists in both `handles/` and `participants/` | store enumeration | prefer `participants/`, ignore legacy duplicate, never merge fields across the two records | yes | +| parent session points at a child id that is missing or non-live | parent-gated lookup | fail closed for toolbox and live status projection, require bootstrap or cleanup to repair | yes | +| multiple live host orchestrator participants survive lookup | orchestrator resolver | return ambiguity, not newest-wins, and block toolbox publication | yes | +| runtime event path lacks a live participant snapshot | runtime event builder | omit additive participant lineage fields instead of inventing or backfilling them from trace history | yes | + +Operator rescue rule: + +- `substrate agent doctor --json` is the place that explains bad participant-store state to the operator +- `substrate agent status --json` may show historical trace rows, but it must never use trace history to re-authorize a broken live participant +- `substrate agent toolbox status|env` stays stricter than status, because a wrong endpoint is worse than no endpoint + +## Test Review + +100% coverage is the goal for the new participant contract. This slice changes identity, lookup, and fail-closed behavior. That is exactly the kind of change that quietly passes the demo path and then burns you later unless the tests are over-complete. + +### Code path coverage + +```text +CODE PATH COVERAGE +=========================== +[+] crates/shell/src/execution/agent_runtime/session.rs + │ + ├── [GAP] Participant constructor: orchestrator record emits participant_id, no world fields on host + ├── [GAP] Participant constructor: member record requires orchestrator_participant_id + ├── [GAP] Compat upgrader: legacy handle JSON maps session_handle_id -> participant_id + ├── [GAP] Validation: world-scoped member rejects missing world_id/world_generation + └── [GAP] Validation: host-scoped orchestrator rejects world fields + +[+] crates/shell/src/execution/agent_runtime/state_store.rs + │ + ├── [GAP] participants/ write + load roundtrip + ├── [GAP] participants/ preferred over legacy handles/ for same child identity + ├── [GAP] list_live_participants filters dead owner PID and invalid ownership + ├── [GAP] resolve_live_orchestrator_participant fails closed on ambiguity + └── [GAP] list_live_participants_for_session preserves same-agent siblings + +[+] crates/shell/src/repl/async_repl.rs + │ + ├── [GAP] host orchestrator bootstrap writes participant record instead of handle manifest + ├── [GAP] ready/running/stopping/stopped transitions persist participant snapshots + └── [GAP] runtime events publish participant_id on orchestrator lifecycle rows + +[+] crates/shell/src/execution/agents_cmd.rs + │ + ├── [GAP] status JSON includes participant_id on live rows + ├── [GAP] same-agent sibling participants both appear in one session + ├── [GAP] legacy trace fallback does not suppress distinct live siblings + ├── [GAP] toolbox status/env accepts only live host orchestrator participant + └── [GAP] doctor passive participant validation fail-closes malformed files + +[+] crates/common/src/agent_events.rs + │ + ├── [GAP] participant_id roundtrips when present + ├── [GAP] parent_participant_id omits by absence when unset + └── [GAP] resumed_from_participant_id omits by absence when unset + +───────────────────────────────── +COVERAGE: 0/20 paths tested (0%) + Code paths: 0/20 +QUALITY: ★★★: 0 ★★: 0 ★: 0 +GAPS: 20 paths need tests +───────────────────────────────── +``` + +### User flow coverage + +```text +USER FLOW COVERAGE +=========================== +[+] substrate agent status --json + │ + ├── [GAP] [→E2E] Live host orchestrator + live world member sibling both render + ├── [GAP] [→E2E] Two same-agent member participants under one orchestration session both render + ├── [GAP] Legacy trace-only session still renders without participant_id + └── [GAP] Malformed participant/world identity fails closed with explicit reason + +[+] substrate agent toolbox status --json + │ + ├── [GAP] Live host orchestrator participant yields endpoint + ├── [GAP] Member-only live state does not authorize endpoint + └── [GAP] Ambiguous live orchestrator participants fail closed + +[+] substrate agent toolbox env --json + │ + ├── [GAP] Allowed orchestrator participant exports endpoint/version + └── [GAP] Trace history alone does not resurrect participant authorization + +[+] substrate agent doctor --json + │ + ├── [GAP] No participant files remains healthy + ├── [GAP] Malformed participant file fails closed + └── [GAP] Impossible role/scope/world combination fails closed + +[+] Runtime event correlation + │ + ├── [GAP] [→EVAL] Participant lineage fields survive schema roundtrip + └── [GAP] Runtime lifecycle rows omit lineage fields cleanly when not applicable + +───────────────────────────────── +COVERAGE: 0/13 flows tested (0%) + User flows: 0/13 +GAPS: 13 flows need tests (2 integration-style, 1 schema/eval-style) +───────────────────────────────── +``` + +### Required test additions by file + +#### `crates/shell/src/execution/agent_runtime/session.rs` + +Add unit tests for: + +- orchestrator participant constructor +- member participant constructor +- legacy handle upgrade +- validation of role/scope/world invariants +- replacement lineage constructor + +#### `crates/shell/src/execution/agent_runtime/state_store.rs` + +Add unit tests for: + +- participants-dir read/write +- dual-read behavior with `participants/` preferred over `handles/` +- live participant enumeration keeping same-agent siblings +- ambiguous orchestrator participant failure +- doctor validation helper on malformed participant JSON + +#### `crates/shell/src/repl/async_repl.rs` + +Extend the existing bootstrap tests to assert: + +- participant file exists under `participants/` +- parent session still points at the active child identity +- runtime lifecycle events publish `participant_id` +- stopped and invalidated orchestrator rows persist participant snapshots, not handle manifests + +#### `crates/shell/tests/agent_successor_contract_ahcsitc0.rs` + +Add integration coverage for: + +- two live participants with same `agent_id` under one `orchestration_session_id` +- `participant_id` presence on status JSON rows +- toolbox authorization from live host orchestrator participant only +- doctor passive validation of malformed participant files +- legacy trace fallback when no participant record exists + +#### `crates/common/tests/agent_hub_event_envelope_schema.rs` + +Add schema tests for: + +- `participant_id` +- `parent_participant_id` +- `resumed_from_participant_id` +- omission rules when unset + +### Test commands + +Run at minimum: + +```bash +cargo test -p substrate-shell agent_successor_contract_ahcsitc0 -- --nocapture +cargo test -p substrate-shell start_host_orchestrator_runtime -- --nocapture +cargo test -p substrate-shell participant -- --nocapture +cargo test -p substrate-common --test agent_hub_event_envelope_schema -- --nocapture +``` + +Then run: + +```bash +cargo test -p substrate-shell -- --nocapture +cargo test --workspace -- --nocapture +``` + +### QA artifact + +Primary QA artifact for follow-up verification: + +[spensermcconnell-feat-orchestration-session-identity-eng-review-test-plan-20260429-125504.md](/Users/spensermcconnell/.gstack/projects/atomize-hq-substrate/spensermcconnell-feat-orchestration-session-identity-eng-review-test-plan-20260429-125504.md) + +## Failure Modes Registry + +| New codepath | Real production failure | Test covers it? | Error handling exists? | User sees clear error? | Critical gap? | +|---|---|---|---|---|---| +| legacy handle upgrade | old `handles/*.json` cannot map to participant fields and disappears from status | planned | yes | yes | no | +| participant write path | orchestrator bootstrap writes participant file but parent still points to stale child id | planned | yes | yes | no | +| same-agent sibling projection | second member silently disappears because status still keys by `agent_id` | planned | yes | yes | yes until fixed | +| toolbox authorization | member-only participant set yields a toolbox endpoint | planned | yes | yes | yes until fixed | +| passive doctor validation | malformed participant file is ignored and doctor reports healthy | planned | yes | yes | yes until fixed | +| event publication | runtime events omit participant lineage after migration | planned | yes | yes | no | +| restart lineage | replacement participant reuses prior id and loses causal chain | planned | yes | yes | no | +| live query ambiguity | two live host orchestrator participants for one agent newest-win silently | planned | yes | yes | yes until fixed | + +Critical gap rule: + +If any operator surface can still collapse distinct live participants or authorize toolbox from a non-orchestrator participant, the slice is not done. + +## Performance Review + +No major performance concern if the implementation stays direct. + +Hard rules: + +- do not scan `participants/` on runtime hot event paths +- operator commands may scan participant files because they are interactive control-plane surfaces, not high-frequency loops +- when a participant already knows its parent `orchestration_session_id`, use direct parent lookup +- do not add a cache just to avoid reading a handful of JSON files + +Footguns to avoid: + +1. Reading both `participants/` and `handles/` on every event emission path. Wrong place. +2. Rebuilding "latest participant by agent" maps after this slice. That reintroduces the exact bug we are fixing. +3. Renaming too many fields in one pass. The cheapest correct move is additive fields plus compatibility parsing, not a flag day. + +## Worktree Parallelization Strategy + +There is a real parallel split once the shared type surface lands. + +### Dependency table + +| Step | Modules touched | Depends on | +|---|---|---| +| Participant model + compat parser | `crates/shell/src/execution/agent_runtime/`, pack docs | - | +| Participant store queries | `crates/shell/src/execution/agent_runtime/` | participant model | +| Orchestrator writer cutover | `crates/shell/src/repl/`, `crates/shell/src/execution/agent_runtime/` | participant model + store | +| Status/toolbox/doctor cutover | `crates/shell/src/execution/`, docs | participant model + store | +| Event schema + runtime publication | `crates/common/src/`, `crates/shell/src/repl/` | participant model | +| Tests + docs cleanup | `crates/shell/tests/`, `crates/common/tests/`, `docs/` | all above | + +### Parallel lanes + +Lane A: participant model + compat parser -> participant store queries -> orchestrator writer cutover +Why: these steps share `agent_runtime/` and define the child identity surface. + +Lane B: status/toolbox/doctor cutover +Why: can branch after Lane A stabilizes participant types and query helpers. + +Lane C: event schema + runtime publication +Why: depends on participant types but is otherwise independent except for one shared touchpoint in `async_repl.rs`. + +Lane D: tests + docs cleanup +Why: goes last so it can validate the final API, not a moving target. + +### Lane ownership rules + +To keep worktrees from stepping on each other: + +- Lane A owns `crates/shell/src/execution/agent_runtime/` and the participant type names. Nobody else edits those files until Lane A merges. +- Lane B owns `crates/shell/src/execution/agents_cmd.rs`, `docs/USAGE.md`, and `docs/TRACE.md` after Lane A merges. +- Lane C owns `crates/common/src/agent_events.rs`, `crates/common/tests/agent_hub_event_envelope_schema.rs`, and the event-emission touchpoints in `async_repl.rs` after Lane A merges. +- Lane D owns integration tests and doc cleanup after B and C merge. + +If a lane needs to cross that boundary, stop and re-split the work. Parallelism is only worth it when ownership is obvious. + +### Execution order + +1. Launch Lane A first. +2. After Lane A lands the participant surface and helper signatures, launch Lane B and Lane C in parallel worktrees. +3. Merge Lane C before Lane B only if `async_repl.rs` stayed isolated to event publication; otherwise rebase whichever lane touched the shared block second. +4. Merge B and C only after both pass their lane-local tests. +5. Run Lane D last. + +Lane-local merge gates: + +- Lane A: participant constructor, compat upgrade, and store tests green +- Lane B: status/toolbox/doctor contract tests green +- Lane C: event envelope schema tests green +- Lane D: full targeted shell/common suite green and docs updated + +### Conflict flags + +- Lane A and Lane C both touch `crates/shell/src/repl/async_repl.rs` +- Lane A and Lane B both touch `crates/shell/src/execution/agent_runtime/state_store.rs` +- Lane B and Lane D both touch `docs/USAGE.md` and `docs/TRACE.md` +- Lane C and Lane D can both touch `crates/common/tests/agent_hub_event_envelope_schema.rs` if test cleanup starts too early + +Safe rule: do not start B or C before A defines the participant type names and store helper signatures. + +## Deferred Work + +There is no `TODOS.md` in this repo root, so deferred items stay here explicitly. + +1. Production member-writer hookup for shared-world members +Why: requires the concrete lifecycle from slices `03` and `04`. + +2. Restart invalidation semantics for member replacement +Why: belongs to slice `05`, even though this slice adds the lineage fields. + +3. Grouped session-centric runtime store layout +Why: belongs to slice `06`; this slice keeps compatibility reads and participant-native queries only. + +## NOT in Scope + +- shared-world ownership lock-in +- world restart generation authority +- public `session_start` / `session_resume` / `session_fork` CLI UX +- nested gateway schema redesign +- toolbox feature expansion or mutating toolbox tools +- background registry service or cache + +## Definition of Done + +The slice is done when all of these are true: + +1. There is one canonical child runtime record type that can represent orchestrator and member participants. +2. New runtime writes go to `~/.substrate/run/agent-hub/participants/*.json`. +3. Legacy `handles/*.json` files are still readable during rollout. +4. `substrate agent status --json` adds `participant_id` and no longer collapses live rows by `agent_id`. +5. Toolbox authorization resolves from a live host-scoped orchestrator participant only. +6. Doctor remains pre-runtime, but malformed participant files fail closed when present. +7. Pure-agent `AgentEvent` rows can publish participant lineage additively. +8. Docs and contract tests reflect participant terminology and sibling-safe status semantics. + +## Completion Summary + +- Step 0: Scope Challenge - scope accepted as-is, but explicitly bounded away from slices `03` through `06` +- Architecture Review: 4 material findings +- Code Quality Review: 3 implementation guardrails around invariant ownership, compat-boundary DRY, and terminology cutover +- Test Review: coverage diagrams produced, 33 concrete gaps/assertions identified +- Performance Review: 0 major issues, 3 direct-lookup/no-cache rules +- Error & Rescue Registry: written +- NOT in scope: written +- What already exists: written +- TODOS.md updates: deferred scope captured in-plan because no `TODOS.md` exists +- Failure modes: 3 critical gaps flagged until the participant-keyed cutover lands +- Outside voice: skipped, no separate codex-review run was performed +- Parallelization: 4 lanes, 2 bounded parallel after the foundation lane +- Lake Score: complete version chosen for every in-slice decision + +## Decision Audit Trail + +| # | Phase | Decision | Classification | Principle | Rationale | Rejected | +|---|---|---|---|---|---|---| +| 1 | Scope | Make `participant_id` the canonical child identity | Mechanical | Completeness | Same-agent siblings are impossible otherwise | Keeping `agent_id` as the lookup key | +| 2 | Scope | Keep parent orchestration session as the top-level authority | Mechanical | Explicit over clever | Parent and child solve different identity layers | Replacing the parent record with participants | +| 3 | Storage | Add `participants/` and keep `handles/` as read-compat only | Mechanical | Pragmatic | Safe rollout without a flag day | In-place destructive migration | +| 4 | Model | Reuse current `ash_` child id format for `participant_id` | Mechanical | Boring by default | Correctness matters more than new ID branding | New bespoke participant-id scheme | +| 5 | Ownership | Widen `ownership_mode` to include member and replaced semantics | Mechanical | Completeness | Member/runtime lineage needs explicit ownership state | Free-form attached-control strings forever | +| 6 | Status | Add `participant_id` to status JSON and stop collapsing by agent | Mechanical | Completeness | Operator truth must not be lossy | Newest-wins per agent heuristics | +| 7 | Toolbox | Keep toolbox authorization host-orchestrator-only | Mechanical | Systems over heroes | Member rows must never become control-plane authority | Authorization from any live participant | +| 8 | Doctor | Add passive participant validation only | Mechanical | Minimal diff | Doctor should stay pre-runtime while still catching bad files | Making healthy doctor depend on live participants | +| 9 | Events | Add participant lineage fields additively to `AgentEvent` now | Mechanical | Boil the lake | Avoid a second schema churn in slice `05` | Delaying lineage fields | +| 10 | Scope | Do not invent a generic production member-launch framework in this slice | Mechanical | Pragmatic | The branch has no concrete member seam yet | Shipping speculative lifecycle infrastructure | + +## GSTACK REVIEW REPORT + +| Review | Trigger | Why | Runs | Status | Findings | +|--------|---------|-----|------|--------|----------| +| CEO Review | `/autoplan` | Scope & strategy | 1 | CLEAR | Correct second slice, explicit non-goals, no accidental absorption of world ownership or grouped-store work | +| Codex Review | `/codex review` | Independent 2nd opinion | 0 | SKIPPED | No separate outside-voice run performed | +| Eng Review | `/plan-eng-review` | Architecture & tests (required) | 1 | CLEAR | Lossy status key fixed at the root, participant store migration bounded, fail-closed operator/test obligations made explicit | +| Design Review | `/plan-design-review` | UI/UX gaps | 0 | SKIPPED | No UI scope | + +**UNRESOLVED:** 0 blocking design decisions remain inside this slice. The only intentional deferrals are later packet slices. + +**VERDICT:** CEO + ENG CLEARED. `PLAN-02` is ready to implement after `PLAN-01`, and before the shared-world, world-binding, restart-generation, and grouped-store follow-ons. diff --git a/llm-last-mile/PLAN-03.md b/llm-last-mile/PLAN-03.md new file mode 100644 index 000000000..bf00e801a --- /dev/null +++ b/llm-last-mile/PLAN-03.md @@ -0,0 +1,988 @@ +# PLAN-03: Shared-World Ownership Contract, Linux-First + +Source file: [03-shared-world-ownership-linux-first.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/03-shared-world-ownership-linux-first.md) +Branch: `feat/shared-world-ownership-contract` +Plan type: backend-only, no UI scope +Review posture: `/autoplan`-style scope tightening with `/plan-eng-review` structure and rigor +Status: execution-ready after slice-boundary correction + +## What This Plan Does + +`PLAN-01` established real orchestration-session identity. `PLAN-02` established a participant-capable runtime model. The next failure point is not "more runtime state." It is that Linux shared worlds still have no authoritative owner contract. + +Today the repo already exposes world identity in a few places, but none of them are the right authority: + +- Linux world reuse still keys off compatible `WorldSpec` inputs in [crates/world/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world/src/lib.rs:39) and [crates/world/src/session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world/src/session.rs:90). +- PTY startup returns only `world_id` from [crates/world-agent/src/pty.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/pty.rs:993), not an owner-proof snapshot. +- Non-PTY execution still builds a plain `WorldSpec` and calls `ensure_session()` in [crates/world-agent/src/service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs:470). +- The REPL already increments `world_generation` locally in [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:2723), which means shell-visible generation exists before the backend can prove it. + +That is the actual problem. + +This plan rewrites slice `03` to do one job only: + +1. establish the authoritative shared-world owner tuple in `world-api`, +2. make Linux reuse owner-aware instead of spec-only, +3. thread that owner tuple through world-agent PTY and non-PTY request/response contracts, +4. fail closed on non-Linux shared-world requests, and +5. stop synthetic world ownership attribution such as `span_id -> orchestration_session_id`. + +This plan explicitly does **not** take ownership of runtime-state projection or restart invalidation. Those are already the point of slices `04` and `05`: + +- [04-thread-world-binding-into-runtime-state.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/04-thread-world-binding-into-runtime-state.md) +- [05-restart-invalidation-semantics.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/05-restart-invalidation-semantics.md) + +That scope cut matters. It prevents slice `03` from fighting the packet instead of enabling it. + +## Scope Challenge + +### Why this is the right third slice + +After `PLAN-02`, the runtime can represent world-scoped participants honestly. What it still cannot do is prove that a Linux world belongs to exactly one orchestration session. + +That ownership proof has to exist before runtime-state projection is meaningful. Otherwise `PLAN-04` would just persist a prettier lie, and `PLAN-05` would invalidate generations that the backend never actually committed. + +The practical bottleneck is simple: + +- generic Linux world reuse is already live, +- world-scoped member semantics need stricter ownership than generic reuse, +- the world-agent transport does not carry that distinction today. + +Fix that seam first. Then the later slices have a real source of truth to consume. + +### What already exists + +| Sub-problem | Existing code | Reuse or replace | +|---|---|---| +| Orchestration identity | [orchestration_session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/orchestration_session.rs:18) | Reuse exactly | +| Participant model with world fields | [session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs:11) | Reuse, but do not make authoritative in this slice | +| Linux world reuse machinery | [crates/world/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world/src/lib.rs:27) | Extend | +| Linux persisted `session.json` metadata | [crates/world/src/session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world/src/session.rs:13) | Extend | +| PTY persistent-session protocol | [crates/world-agent/src/pty.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/pty.rs:163) and [world_persistent_session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_persistent_session.rs:367) | Extend | +| Non-PTY execute transport | [crates/agent-api-types/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/agent-api-types/src/lib.rs:657) and [crates/world-agent/src/service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs:432) | Extend | +| REPL restart/generation signaling | [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:2378) | Reuse later in `PLAN-05`, do not deepen here | +| Fail-closed status expectations for world-scoped rows | [agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:541) | Reuse later in `PLAN-04` | +| Cross-platform world bootstrap | [routing/world.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/world.rs:13), [platform_world/mod.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/platform_world/mod.rs:60), [platform_world/windows.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/platform_world/windows.rs:52) | Keep generic, add fail-closed gate for explicit owner mode | + +### Minimum change that achieves the goal + +Do this, and only this: + +- add an explicit shared-world owner mode to `world-api` and agent API request surfaces, +- return authoritative owner/generation metadata from Linux world allocation, +- persist that metadata in Linux `session.json`, +- make Linux owner-bound reuse a separate code path from generic compatible reuse, +- fail closed for explicit owner-bound requests on non-Linux backends, +- remove synthetic shared-world attribution in world-agent event paths. + +Do not: + +- create a shell-side authoritative world-binding store in this slice, +- project world binding into runtime manifests here, +- redefine participant replacement semantics here, +- redesign `substrate agent status` selection here, +- invent a new agent-hub daemon, +- add a cache to avoid scanning a small number of Linux world metadata files. + +### Complexity check + +This slice crosses more than 8 files. That is acceptable because the authority seam is already spread across: + +- `crates/world-api` +- `crates/agent-api-types` +- `crates/world` +- `crates/world-agent` +- `crates/shell/src/execution/routing/dispatch/world_persistent_session.rs` +- `crates/shell/src/execution/platform_world/*` +- `crates/shell/src/repl/async_repl.rs` +- tests and docs + +The guardrail is strict: + +- no new daemon, +- no new generic registry abstraction, +- no runtime-manifest write path here, +- no speculative non-Linux ownership algorithm, +- no full restart invalidation policy here. + +That keeps the diff large enough to solve the real problem, but still boring. + +### Search-before-building posture + +`[Layer 1]` wins here. + +The repo already has: + +- a portable `world-api` contract, +- Linux persisted world metadata, +- PTY and non-PTY agent transport seams, +- participant/runtime structs that can eventually consume world identity, +- explicit later slices for runtime-state projection and restart invalidation. + +The right move is not to create a new shell authority and mirror it down. That would be a second database disguised as JSON files. Software loves doing that. The pager loves it less. + +The clean path is: + +1. make Linux world metadata and backend allocation authoritative for owner-bound reuse, +2. make transport contracts echo that authority, +3. let later slices project it upward. + +### Hard non-goals + +- runtime-manifest binding persistence from [04-thread-world-binding-into-runtime-state.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/04-thread-world-binding-into-runtime-state.md) +- participant invalidation and generation cutover from [05-restart-invalidation-semantics.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/05-restart-invalidation-semantics.md) +- grouped session-centric state-store reshaping from slice `06` +- non-Linux shared-world parity beyond model acceptance plus fail-closed rejection +- status/toolbox schema redesign +- trace schema redesign beyond removing invalid placeholder attribution + +## Architecture Contract + +### No-ambiguity rules + +These are hard rules: + +1. Slice `03` owns the **world ownership contract**, not runtime-state projection and not restart invalidation. +2. Linux world metadata and backend allocation are the authoritative source of truth for owner-bound shared worlds in this slice. +3. Explicit shared-world requests must be structurally distinct from generic compatible-reuse requests. Missing owner context is an error, not a fallback. +4. Generic shell/world bootstrap behavior remains generic. It must not silently become agent-hub ownership authority. +5. `world_generation` starts at `0` on first owner-bound allocation and changes only when the backend accepts an explicit replacement action. +6. PTY and non-PTY paths must both carry the same owner tuple and receive the same authoritative binding snapshot back. +7. Legacy Linux `session.json` without owner metadata may remain reusable for generic callers, but must be rejected for explicit owner-bound requests. +8. Non-Linux backends may accept additive schema changes, but explicit shared-world owner mode must fail closed before generic reuse can happen. +9. No world-agent event path may synthesize `orchestration_session_id` from `span_id` for shared-world flows. +10. Runtime manifests, selected status rows, and participant invalidation remain follow-on work for `PLAN-04` and `PLAN-05`. + +### Target owner contract + +Add one explicit owner-bound mode to `world-api`. Do not overload "plain `reuse_session=true` plus maybe some optional fields." + +Use this shape: + +```rust +#[derive(Clone, Debug, Serialize, Deserialize, PartialEq, Eq)] +#[serde(rename_all = "snake_case")] +pub enum WorldReuseMode { + GenericCompatible, + SharedOrchestration(SharedWorldOwnerSpec), +} + +#[derive(Clone, Debug, Serialize, Deserialize, PartialEq, Eq)] +pub struct SharedWorldOwnerSpec { + pub orchestration_session_id: String, + pub action: SharedWorldOwnerAction, +} + +#[derive(Clone, Debug, Serialize, Deserialize, PartialEq, Eq)] +#[serde(rename_all = "snake_case")] +pub enum SharedWorldOwnerAction { + AttachOrCreate, + ReplaceExpectedGeneration { + expected_generation: u64, + reason: String, + }, +} + +#[derive(Clone, Debug, Serialize, Deserialize, PartialEq, Eq)] +pub struct SharedWorldBindingSnapshot { + pub orchestration_session_id: String, + pub world_id: String, + pub world_generation: u64, + pub binding_state: SharedWorldBindingState, +} +``` + +Then: + +- `WorldSpec` gains `reuse_mode: WorldReuseMode` +- `WorldHandle` gains `shared_binding: Option` +- non-PTY `ExecuteRequest` gains `shared_world: Option` +- PTY `start_session` gains the same `shared_world` payload +- PTY `ready` and non-PTY `ExecuteResponse` echo the authoritative `SharedWorldBindingSnapshot` + +This is more explicit than the source SOW, and that is intentional. Optional loose fields are how silent downgrade bugs sneak in. + +### Linux metadata becomes the authority seam + +Primary file: + +- [crates/world/src/session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world/src/session.rs:13) + +Extend `SessionWorldMetadata` with a minimal ownership-bearing shape: + +```rust +struct SessionWorldMetadata { + world_id: String, + project_dir: PathBuf, + isolate_network: bool, + always_isolate: bool, + allowed_domains: Vec, + cgroup_path: PathBuf, + started_at_unix_millis: u64, + owner_mode: SessionWorldOwnerMode, + orchestration_session_id: Option, + world_generation: Option, + binding_state: Option, + policy_snapshot_hash: Option, + world_fs_mode: Option, + last_restart_reason: Option, +} +``` + +Rules: + +- generic worlds persist `owner_mode=generic` and omit owner fields +- explicit shared worlds persist `owner_mode=shared_orchestration`, `orchestration_session_id`, `world_generation`, and `binding_state` +- `binding_state` is one of: + - `active` + - `replacing` + - `replaced` + - `abandoned` +- shared-owner reuse is allowed only from `binding_state=active` +- `binding_state=replacing` exists only as an internal pre-commit marker during replacement and recovery; downstream request/PTY/status surfaces may expose or accept only `binding_state=active` +- incomplete or contradictory owner metadata is never reusable for shared-owner flows + +### Linux reuse path splits cleanly + +Primary file: + +- [crates/world/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world/src/lib.rs:39) + +The backend must stop treating shared-world ownership as "generic reuse, but with extra checks later." + +Required behavior: + +1. `WorldReuseMode::GenericCompatible` + - preserve existing compatibility behavior + - legacy `session.json` remains eligible + +2. `WorldReuseMode::SharedOrchestration(AttachOrCreate)` + - scan persisted metadata for `owner_mode=shared_orchestration` + - require exact `orchestration_session_id` match + - require compatible world inputs + - require `binding_state=active` + - reuse matching world if present + - otherwise create a new world with `world_generation=0` + - never reuse a world owned by another orchestration session + +3. `WorldReuseMode::SharedOrchestration(ReplaceExpectedGeneration { expected_generation, reason })` + - resolve the currently active world for that owner + - require exact generation match + - pre-commit the current world to `binding_state=replacing` + - create a fresh world with generation `expected_generation + 1` and committed `binding_state=active` + - if replacement creation fails, roll the prior world back to `binding_state=active` and clean up the partial replacement root + - after the new world is committed, finalize the prior world to `binding_state=replaced` + - return the new authoritative binding snapshot + +4. Any shared-owner request with ownerless legacy metadata + - fail closed for that candidate + - do not silently adopt it + +This split is the whole game. + +### Minimal stale/abandoned world rule + +The source SOW left crash/abandonment too vague. This plan tightens it without stealing slice `05`. + +Required recovery rule in this slice: + +- if a shared-owner world root exists but its metadata is incomplete, unreadable, or references missing critical directories, it is not reusable and recovery leaves it on disk without adopting it +- if recovery finds exactly one `binding_state=replacing` world and no `active` world for the owner, it reconciles that lone `replacing` world back to `active` +- if recovery finds one `active` world and an older `replacing` world for the same owner, it prefers the newer `active` world +- if recovery finds ambiguous same-owner state, such as multiple `active` worlds or a `replacing` world with generation newer than or equal to the `active` world, it fails closed +- `replaced` and `abandoned` worlds are never reused for shared-owner requests + +This prevents the worst case: an interrupted replace leaves metadata on disk, and the next attach accidentally reuses the wrong state because the spec happened to match. + +### Transport contract becomes proof-bearing + +Primary files: + +- [crates/agent-api-types/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/agent-api-types/src/lib.rs:657) +- [crates/world-agent/src/pty.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/pty.rs:163) +- [crates/world-agent/src/service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs:432) +- [crates/shell/src/execution/routing/dispatch/world_persistent_session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_persistent_session.rs:367) + +Required outcome: + +- shared-world owner context is carried into both PTY and non-PTY allocation calls +- Linux returns authoritative owner proof back on both paths +- the shell can validate that proof immediately + +That means: + +- PTY `Ready` must stop being only `{ world_id, cwd, protocol_version }` +- non-PTY `ExecuteResponse` must stop returning only a brand-new span id with no owner proof + +Use this PTY `Ready` shape: + +```rust +Ready { + session_nonce: String, + world_id: String, + cwd: PathBuf, + protocol_version: u32, + shared_world: Option, +} +``` + +Add this to non-PTY `ExecuteResponse`: + +```rust +pub shared_world: Option +``` + +### Shell-side scope in this slice + +Shell changes are intentionally narrow: + +- resolve the active `orchestration_session_id` when building explicit world-scoped member requests +- populate `shared_world` request payloads for shared-owner flows only +- validate returned binding snapshots from PTY/non-PTY responses +- do not persist runtime manifests here +- do not treat REPL-local `world_generation` arithmetic as authoritative here + +In practice: + +- [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:2677) may still track a local generation for in-memory session continuity, but that value must be replaced by the backend-returned generation as soon as the contract exists +- runtime-state persistence remains the responsibility of `PLAN-04` + +### Cross-platform boundary + +Primary files: + +- [crates/shell/src/execution/routing/world.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/world.rs:13) +- [crates/shell/src/execution/platform_world/mod.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/platform_world/mod.rs:60) +- [crates/shell/src/execution/platform_world/windows.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/platform_world/windows.rs:52) +- `crates/world-mac-lima/src/*` +- `crates/world-windows-wsl/*` + +Required rule: + +- explicit shared-owner mode on non-Linux platforms returns one uniform unsupported error before any generic compatible reuse can occur + +That means: + +- macOS and Windows may deserialize the new fields +- compile compatibility is required +- behavior compatibility is **not** the same as shared-owner support + +If a shared-owner request reaches a generic `ensure_session()` path on non-Linux, the slice is wrong. + +## Architecture Diagrams + +### Linux owner-bound allocation flow + +```text +shell world-scoped member request + │ + │ resolve orchestration_session_id + owner action + ▼ +world-agent PTY/non-PTY transport + │ + │ shared_world = { owner, action } + ▼ +world-api::WorldSpec { reuse_mode = SharedOrchestration(...) } + │ + ▼ +LinuxLocalBackend::ensure_session() + │ + ├── scan /tmp/substrate-worlds/*/session.json + │ + ├── if exact owner + compatible inputs + active binding + │ └── reuse existing world + │ + ├── if replace_expected_generation + generation matches + │ └── create new world, old => replaced + │ + └── else + └── create new world generation 0 + ▼ +authoritative SharedWorldBindingSnapshot + │ + ├── PTY ready frame echo + └── non-PTY execute response echo +``` + +### Generic vs shared-owner reuse boundary + +```text +incoming world request + │ + ├── reuse_mode = GenericCompatible + │ └── old compatibility path, legacy metadata allowed + │ + └── reuse_mode = SharedOrchestration(owner) + │ + ├── owner metadata missing on candidate + │ └── reject candidate + │ + ├── owner matches + inputs compatible + active + │ └── reuse + │ + ├── owner differs + │ └── never reuse + │ + ├── replace generation mismatch + │ └── fail closed + │ + └── no valid active match + └── create new owned world +``` + +## Implementation Plan + +### Ordered execution checklist + +1. Narrow slice `03` to owner contract only. +2. Add `WorldReuseMode`, `SharedWorldOwnerSpec`, and `SharedWorldBindingSnapshot`. +3. Extend Linux `session.json` metadata with owner-bearing fields. +4. Split generic reuse from shared-owner reuse in `LinuxLocalBackend`. +5. Extend PTY and non-PTY request/response contracts to carry owner proof. +6. Add fail-closed non-Linux capability gates. +7. Remove placeholder `span_id -> orchestration_session_id` behavior on shared-owner paths. +8. Add Linux-first tests, then docs. + +### Execution sequencing and merge gates + +1. Land the contract types first. + - No backend behavior change yet. + - Merge gate: serde round-trip and compile green. + +2. Land Linux metadata plus owner-aware reuse. + - This is the true behavior change. + - Merge gate: Linux unit/integration coverage for attach/create/replace/reject. + +3. Land PTY and non-PTY transport plumbing. + - The shell must be able to request and validate the new contract. + - Merge gate: world-agent protocol tests green. + +4. Land non-Linux fail-closed routing gates. + - Prevent accidental shared-owner downgrade. + - Merge gate: platform-specific unsupported-path tests green. + +5. Land docs and cleanup. + - Update packet docs to reflect the corrected slice boundary. + +### Workstream 1: Contract types and slice-boundary correction + +Primary files: + +- [llm-last-mile/PLAN-03.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-03.md) +- [crates/world-api/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-api/src/lib.rs:14) +- [crates/agent-api-types/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/agent-api-types/src/lib.rs:650) + +Tasks: + +- add explicit owner-bound reuse mode +- add binding snapshot response type +- extend API payload structs +- update doc comments so generic vs shared-owner behavior is obvious + +### Workstream 2: Linux metadata and exact-owner reuse + +Primary files: + +- [crates/world/src/session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world/src/session.rs:13) +- [crates/world/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world/src/lib.rs:27) + +Tasks: + +- extend `SessionWorldMetadata` +- implement owner-aware metadata matching and rejection rules +- implement shared-owner attach/create logic +- implement replace-expected-generation logic +- mark old world metadata `replaced` + +### Workstream 3: PTY and non-PTY transport plumbing + +Primary files: + +- [crates/world-agent/src/pty.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/pty.rs:163) +- [crates/world-agent/src/service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs:432) +- [crates/shell/src/execution/routing/dispatch/world_persistent_session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_persistent_session.rs:367) + +Tasks: + +- parse `shared_world` on PTY start +- build owner-aware `WorldSpec` +- echo authoritative binding snapshot in PTY `Ready` +- echo authoritative binding snapshot in non-PTY `ExecuteResponse` +- validate missing/malformed snapshots fail closed + +### Workstream 4: Shell routing gates and REPL alignment + +Primary files: + +- [crates/shell/src/execution/routing/world.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/world.rs:13) +- [crates/shell/src/execution/platform_world/mod.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/platform_world/mod.rs:60) +- [crates/shell/src/execution/platform_world/windows.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/platform_world/windows.rs:52) +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:2677) + +Tasks: + +- resolve owner context for shared-world member flows +- reject non-Linux shared-owner mode before backend allocation +- consume backend-returned generation instead of trusting local increment-only state where possible +- keep runtime-manifest persistence out of this slice + +### Workstream 5: Tests and docs + +Primary files: + +- `crates/world/tests/*` +- `crates/world-agent/tests/*` +- [crates/shell/tests/repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs) +- [docs/WORLD.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/WORLD.md) +- [docs/TRACE.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/TRACE.md) +- [llm-last-mile/README.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/README.md) + +Tasks: + +- add missing Linux-first behavior tests +- document shared-owner mode and non-Linux rejection +- document that runtime-state projection is still `PLAN-04` + +## Architecture Review + +This section is the final engineering lock-in for slice `03`. The prior review findings are now resolved into hard implementation requirements, not open recommendations. + +### Locked architecture decisions + +1. **Slice boundary is fixed.** + - `PLAN-03` owns only the shared-world owner contract, Linux reuse semantics, transport proof, and non-Linux fail-closed rejection. + - `PLAN-04` owns runtime-manifest projection of `world_id` and `world_generation`. + - `PLAN-05` owns invalidation and replacement semantics after generation changes. + +2. **Linux backend metadata is the only authority in this slice.** + - The authoritative seam is persisted Linux world metadata plus `crates/world` allocation behavior. + - This plan does not introduce a shell-owned binding file or second authority store under `~/.substrate/run/agent-hub/`. + +3. **Shared-owner reuse is structurally distinct from generic reuse.** + - Generic compatible reuse remains the current path. + - Explicit shared-world reuse must go through `WorldReuseMode::SharedOrchestration(...)`. + - Missing owner context is a hard error, never an implicit downgrade to generic reuse. + +4. **Backend-returned generation is the committed truth.** + - The REPL may stage local restart intent, but it does not commit `world_generation`. + - PTY and non-PTY responses must echo the authoritative `SharedWorldBindingSnapshot`, and that echoed generation is the only value later slices may persist. + +5. **Linux-first is enforced behaviorally, not cosmetically.** + - macOS and Windows may deserialize additive fields. + - They must reject explicit shared-owner requests before any generic `ensure_session()`-style fallback can occur. + +### Architecture acceptance gates + +The plan is only correctly implemented if all five gates pass: + +1. **Contract gate** + - `world-api`, `agent-api-types`, PTY ready frames, and non-PTY execute responses all use the same owner-binding vocabulary. + +2. **Linux authority gate** + - `crates/world` can decide attach/create/replace using persisted owner-bearing metadata alone, without consulting shell-local authority. + +3. **Transport proof gate** + - Both PTY and non-PTY callers receive authoritative owner proof back from the backend, not just `world_id`. + +4. **Fail-closed gate** + - Any malformed, missing, contradictory, or cross-owned metadata path rejects or allocates fresh. It never silently reuses. + +5. **Platform boundary gate** + - Non-Linux explicit shared-owner requests reject uniformly and early. + +## Code Quality Review + +This slice stays boring on purpose. The minimum diff that closes the ownership hole is the right diff. + +### Implementation guardrails + +1. **One place owns reuse compatibility logic.** + - Compatibility hashing, policy fingerprint comparison, and owner-match rules stay in `crates/world`. + - Shell and world-agent code only build requests, pass them through, and validate returned snapshots. + +2. **One snapshot type travels through every layer.** + - `SharedWorldBindingSnapshot` is the reusable proof object. + - `WorldHandle`, PTY `Ready`, and non-PTY `ExecuteResponse` all reuse it instead of inventing adjacent mini-schemas. + +3. **No synthetic ownership attribution.** + - `span_id` remains span correlation only. + - Shared-world ownership fields are populated from explicit owner context or omitted. They are never inferred. + +4. **No second persistence system.** + - No new shell-owned registry, no side cache, no background daemon, no duplicate JSON authority file. + - The implementation extends existing persistence seams instead of adding new ones. + +5. **Provisional REPL state stays clearly provisional.** + - Any local generation arithmetic in [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:2723) must be treated as pre-commit state only. + - Backend echo is the boundary between "requested" and "committed." + +### Minimal-diff rules + +- Reuse existing serde surfaces instead of creating wrapper payloads for one slice. +- Reuse existing Linux metadata recovery paths instead of adding a new world lookup registry. +- Reuse existing platform-world routing seams for fail-closed rejection instead of creating a special shared-world launcher. +- Add comments only where the generic/shared-owner boundary would otherwise be easy to misread. + +## Error & Rescue Registry + +| Failure point | What goes wrong | Expected rescue / fail-closed behavior | +|---|---|---| +| shared-world request missing owner tuple | request enters generic reuse path | reject request before backend allocation | +| same spec, different owner | backend reuses another session’s world | allocate a fresh world; never cross-reuse | +| replace request generation mismatch | stale restart request increments the wrong chain | reject with explicit generation-conflict error | +| owner-bearing PTY request hits non-Linux | backend falls through to generic ensure-session | return uniform unsupported-shared-world-owner error | +| legacy ownerless `session.json` encountered for shared-owner flow | backend silently adopts a pre-owner world | treat candidate as non-reusable for shared-owner mode | +| partial metadata write after crash | future request reuses a half-bound world | ignore the malformed candidate without adopting it; never reuse | +| PTY ready frame omits generation echo | shell invents generation anyway | close session fail closed and surface protocol error | +| world-agent event path uses `span_id` as owner id | trace shows false world lineage | omit owner attribution or use explicit owner context only | + +## Test Review + +100% new-path coverage is the goal. The current branch has partial generic world reuse coverage, but almost none of the owner-bound behavior this slice requires. + +### Code path coverage + +```text +CODE PATH COVERAGE +=========================== +[+] crates/world/src/session.rs + │ + ├── [★★ TESTED] Generic persisted metadata recovery / compatibility path exists today + │ via current recover_compatible_from_root tests + ├── [GAP] Shared-owner metadata parse + persist round-trip + ├── [GAP] Shared-owner candidate rejected when owner mismatches + ├── [GAP] Legacy ownerless metadata rejected for shared-owner flows + └── [GAP] Replaced/abandoned metadata never reused + +[+] crates/world/src/lib.rs + │ + ├── [★★ TESTED] Generic cache miss repopulates backend cache from persisted metadata + ├── [GAP] Shared-owner attach_or_create bypasses generic compatibility lookup + ├── [GAP] Same owner + same inputs reuses same world + ├── [GAP] Different owner + same inputs allocates distinct world + └── [GAP] ReplaceExpectedGeneration increments generation and returns new binding + +[+] crates/world-agent/src/pty.rs + │ + ├── [GAP] start_session accepts shared_world owner payload + ├── [GAP] Ready echoes shared binding snapshot + └── [GAP] missing/invalid shared binding snapshot fails protocol closed + +[+] crates/world-agent/src/service.rs + │ + ├── [GAP] ExecuteRequest shared_world payload builds owner-aware WorldSpec + ├── [GAP] ExecuteResponse echoes shared binding snapshot + └── [GAP] streamed event path never aliases span_id into orchestration ownership + +[+] crates/shell/src/execution/routing/dispatch/world_persistent_session.rs + │ + ├── [GAP] shell validates ready.shared_world for owner-bound sessions + └── [GAP] missing generation echo aborts session instead of trusting local state + +[+] crates/shell/src/execution/platform_world/* + │ + └── [GAP] macOS/Windows explicit shared-owner mode rejects before generic reuse + +───────────────────────────────── +COVERAGE: 2/15 paths tested (13%) + Code paths: 2/15 (13%) +QUALITY: ★★★: 0 ★★: 2 ★: 0 +GAPS: 13 paths need tests +───────────────────────────────── +``` + +### User flow coverage + +```text +USER FLOW COVERAGE +=========================== +[+] Linux shared-world attach/create + │ + ├── [GAP] [→E2E] First member for orchestration session creates generation 0 world + ├── [GAP] [→E2E] Second member same orchestration reuses same world_id + └── [GAP] [→E2E] Different orchestration with identical spec gets a different world_id + +[+] Linux replacement flow + │ + ├── [GAP] [→E2E] ReplaceExpectedGeneration succeeds and returns generation+1 + └── [GAP] ReplaceExpectedGeneration with stale generation fails closed + +[+] Crash / recovery edges + │ + ├── [GAP] Half-written owner metadata is ignored or marked abandoned + └── [GAP] Legacy ownerless world never adopted by shared-owner path + +[+] Cross-platform boundary + │ + ├── [GAP] macOS explicit shared-owner request is rejected before generic allocation + └── [GAP] Windows explicit shared-owner request is rejected before generic allocation + +[+] Transport proof + │ + ├── [GAP] PTY ready frame carries world_id + world_generation + owner echo + └── [GAP] Non-PTY execute response carries same owner proof + +───────────────────────────────── +COVERAGE: 0/10 flows tested (0%) + User flows: 0/10 (0%) +GAPS: 10 flows need tests (5 need E2E-style integration) +───────────────────────────────── +``` + +### Required test additions by file + +#### `crates/world/src/session.rs` + +Add unit coverage for: + +- owned metadata round-trip with generation `0` +- reject shared-owner reuse when `orchestration_session_id` mismatches +- reject ownerless legacy metadata for shared-owner mode +- reject `binding_state=replaced` and `binding_state=abandoned` +- ignore malformed partial owner metadata after crash + +#### `crates/world/src/lib.rs` + +Add Linux backend coverage for: + +- `GenericCompatible` still preserves current behavior +- `SharedOrchestration(AttachOrCreate)` reuses only exact owner match +- different owner with identical inputs gets a new world +- `ReplaceExpectedGeneration` increments generation and returns new binding +- generation mismatch returns explicit error, not fallback reuse + +#### `crates/world-agent/src/pty.rs` + +Add protocol tests for: + +- `start_session` shared-world payload parse +- `Ready.shared_world` echo shape +- protocol fail-closed when owner-bound session gets `Ready` without generation + +#### `crates/world-agent/src/service.rs` + +Add transport tests for: + +- non-PTY `ExecuteRequest.shared_world` to owner-aware `WorldSpec` +- response echo of `SharedWorldBindingSnapshot` +- no `span_id -> orchestration_session_id` fallback on shared-owner streamed events + +#### `crates/shell/src/execution/routing/dispatch/world_persistent_session.rs` + +Add host-side protocol tests for: + +- owner-bound PTY session validates `Ready.shared_world` +- missing or wrong owner echo aborts the session +- backend-returned generation replaces provisional local generation + +#### `crates/shell/src/execution/platform_world/windows.rs` + +Add tests for: + +- explicit shared-owner requests reject before `ensure_session()` generic fallback +- generic bootstraps remain unchanged + +#### `crates/shell/src/execution/platform_world/mod.rs` + +Add tests for: + +- Linux path allows explicit shared-owner mode +- macOS and Windows paths reject it uniformly + +### Test commands + +Run at minimum: + +```bash +cargo test -p world session -- --nocapture +cargo test -p world --lib ensure_session -- --nocapture +cargo test -p world-agent repl_persistent_session_bootstrap_v1 -- --nocapture +cargo test -p world-agent --lib execute_stream -- --nocapture +cargo test -p substrate-shell repl_world_first_routing_v1 -- --nocapture +``` + +Then run: + +```bash +cargo test -p world-agent -- --nocapture +cargo test -p substrate-shell -- --nocapture +cargo test --workspace -- --nocapture +``` + +### QA artifact + +Primary QA artifact for follow-up verification: + +[spensermcconnell-feat-session-participant-record-cutover-eng-review-test-plan-20260429-164041.md](/Users/spensermcconnell/.gstack/projects/atomize-hq-substrate/spensermcconnell-feat-session-participant-record-cutover-eng-review-test-plan-20260429-164041.md) + +## Failure Modes Registry + +| New codepath | Real production failure | Test covers it? | Error handling exists? | User sees clear error? | Critical gap? | +|---|---|---|---|---|---| +| owner-bound attach/create | different orchestration silently reuses same Linux world because spec matched | planned | planned | yes | yes until fixed | +| owner-bound replace | stale expected generation replaces the wrong world chain | planned | planned | yes | yes until fixed | +| PTY ready proof | shell accepts `world_id` without generation and invents authority locally | planned | planned | yes | yes until fixed | +| non-PTY owner proof | API caller cannot tell whether execution used the intended owned world | planned | planned | yes | yes until fixed | +| legacy metadata path | old `session.json` is adopted by owner-bound flow | planned | planned | yes | yes until fixed | +| stale partial metadata | crash leaves a half-bound world that becomes reusable later | planned | planned | yes | yes until fixed | +| non-Linux boundary | explicit shared-owner request downgrades to generic reuse on macOS/Windows | planned | planned | yes | yes until fixed | +| streamed event attribution | trace records fake orchestration ownership from `span_id` | planned | planned | partial | no, but must still be fixed | + +Critical gap rule: + +If any owner-bound request can silently fall back to generic compatible reuse, the slice is not done. + +## Performance Review + +No new cache is justified here. + +Hard rules: + +- owner-bound lookup may scan persisted Linux world metadata because this is a startup/control-plane path, not a hot inner loop +- keep generic and shared-owner lookup codepaths separate, but share compatibility helpers +- do not add a shell-side binding cache to "speed up" status or REPL reads in this slice +- compute compatibility fingerprints once per backend path, not separately in shell and backend + +Footguns to avoid: + +1. Scanning world metadata on every streamed event. Wrong layer. +2. Maintaining both shell-authoritative JSON and Linux-authoritative JSON. Two sources of truth is not performance, it is archaeology. +3. Making non-Linux backends emulate Linux ownership logic in this slice. That spends an innovation token for negative value. + +## Worktree Parallelization Strategy + +There is a clean split after the contract surface lands. + +### Dependency table + +| Step | Modules touched | Depends on | +|---|---|---| +| Contract types | `crates/world-api/`, `crates/agent-api-types/`, `llm-last-mile/` | — | +| Linux metadata + reuse | `crates/world/` | contract types | +| PTY + non-PTY world-agent plumbing | `crates/world-agent/` | contract types, Linux binding snapshot | +| Shell routing gates | `crates/shell/src/execution/platform_world/`, `crates/shell/src/execution/routing/`, `crates/shell/src/repl/` | contract types, world-agent plumbing | +| Tests + docs | `crates/world/tests/`, `crates/world-agent/tests/`, `crates/shell/tests/`, `docs/` | all above | + +### Parallel lanes + +Lane A: contract types -> Linux metadata + reuse +Why: `WorldReuseMode` and `SharedWorldBindingSnapshot` define the semantics every other lane consumes. + +Lane B: PTY + non-PTY world-agent plumbing +Why: can branch once Lane A finalizes the shared binding snapshot shape. + +Lane C: shell routing gates +Why: can branch once Lane B finalizes what PTY/non-PTY responses echo. + +Lane D: tests + docs cleanup +Why: last lane, validates the final public contract instead of a moving target. + +### Lane ownership rules + +- Lane A owns `crates/world-api/`, `crates/agent-api-types/`, and `crates/world/` +- Lane B owns `crates/world-agent/` +- Lane C owns `crates/shell/src/execution/platform_world/`, `crates/shell/src/execution/routing/`, and the minimal response-consumer seams in `async_repl.rs` +- Lane D owns tests and docs after interfaces settle + +### Execution order + +1. Launch Lane A first. +2. After Lane A lands type signatures, launch Lane B. +3. After Lane B lands response shapes, launch Lane C. +4. Merge B before C if shell response validation depends on final transport fields. +5. Run Lane D last. + +### Conflict flags + +- Lane A and Lane B both care about `SharedWorldBindingSnapshot` shape +- Lane B and Lane C both care about PTY `Ready` and non-PTY response fields +- Lane D can step on any lane if started too early + +Safe rule: + +Do not start shell routing validation before world-agent response shapes are frozen. + +## Deferred Work + +There is no `TODOS.md` in this repo root, so deferrals stay here explicitly. + +1. Runtime-manifest persistence of `world_id` and `world_generation` +Why: this is the explicit job of [PLAN-04](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/04-thread-world-binding-into-runtime-state.md). + +2. Participant invalidation and registry cutover on generation change +Why: this is the explicit job of [PLAN-05](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/05-restart-invalidation-semantics.md). + +3. Session-centric grouped store layout and query reshaping +Why: this belongs to slice `06`, not the owner-contract slice. + +4. Non-Linux shared-world ownership semantics beyond fail-closed rejection +Why: Linux-first means Linux-first. Compile support plus explicit rejection is enough here. + +## NOT in Scope + +- runtime-manifest projection of owner binding +- `substrate agent status` live-selection redesign +- participant invalidation or replacement registry semantics +- generalized cleanup daemon for abandoned worlds +- non-Linux owned shared-world support +- gateway/nested status schema redesign + +## Definition of Done + +The slice is done when all of these are true: + +1. `world-api` can represent a structurally explicit shared-owner world request. +2. Linux `session.json` persists owner-bearing metadata plus generation. +3. Generic callers preserve current reuse semantics. +4. Shared-owner attach/create reuses only exact owner match plus compatible inputs. +5. Shared-owner replace increments generation only through an explicit replace action. +6. PTY `Ready` and non-PTY `ExecuteResponse` both echo authoritative shared binding snapshots. +7. Non-Linux explicit shared-owner requests fail closed before generic world reuse. +8. No shared-owner world-agent event path aliases `span_id` into orchestration ownership. +9. Docs make it explicit that runtime-state projection and invalidation remain later slices. + +## Completion Summary + +- Step 0: Scope Challenge - scope reduced from the source SOW; slice `03` now owns owner contract only +- Architecture Review: 5 locked decisions, 5 acceptance gates +- Code Quality Review: 5 implementation guardrails, 4 minimal-diff rules +- Test Review: coverage diagrams produced, 23 concrete gaps/assertions identified +- Performance Review: 0 major issues, 3 no-cache/no-dual-truth rules +- Error & Rescue Registry: written +- NOT in scope: written +- What already exists: written +- TODOS.md updates: deferred scope captured in-plan because no `TODOS.md` exists +- Failure modes: 7 critical gaps flagged until owner-bound reuse is implemented +- Outside voice: ran (`codex exec` strategy + architecture passes), findings incorporated +- Parallelization: 4 lanes, 1 foundational + 2 sequential follow-on lanes + final validation lane +- Lake Score: complete version chosen for every in-slice decision + +## Decision Audit Trail + +| # | Phase | Decision | Classification | Principle | Rationale | Rejected | +|---|---|---|---|---|---|---| +| 1 | Scope | Narrow slice `03` to owner contract only | Mechanical | Explicit over clever | Prevents overlap with slices `04` and `05` | Keeping runtime-state projection and invalidation in this slice | +| 2 | Authority | Linux metadata/backend is the authoritative owner seam | Mechanical | Pragmatic | Reuse decisions already happen there | Shell-side authoritative binding file | +| 3 | API | Add explicit `WorldReuseMode` | Mechanical | Completeness | Missing owner context must be impossible to confuse with generic reuse | Optional loose owner fields on generic requests | +| 4 | Linux reuse | Split generic and shared-owner paths | Mechanical | Explicit over clever | Silent downgrade is the main risk | One codepath with optional owner checks | +| 5 | Generation | Backend-returned generation is the committed truth | Mechanical | Systems over heroes | Prevents REPL-local arithmetic from becoming authority | Trusting shell-local increment only | +| 6 | Legacy | Ownerless legacy metadata remains generic-only | Mechanical | Minimal diff | No destructive migration required | Adopting legacy ownerless worlds into shared-owner flows | +| 7 | Cross-platform | Non-Linux shared-owner mode rejects before generic ensure-session | Mechanical | Fail closed | Linux-first must be behaviorally true | Letting non-Linux fall through to generic reuse | +| 8 | Telemetry | Remove `span_id -> orchestration_session_id` on shared-owner paths | Mechanical | Explicit over clever | Span correlation is not owner identity | Synthetic ownership attribution | +| 9 | Runtime state | Defer manifest/status binding writes to `PLAN-04` | Mechanical | Scope discipline | Packet already has a dedicated slice for it | Writing runtime authority here too | +| 10 | Restart semantics | Defer participant invalidation to `PLAN-05` | Mechanical | Scope discipline | Owner contract must exist before invalidation rules consume it | Defining full invalidation semantics here | + +## GSTACK REVIEW REPORT + +| Review | Trigger | Why | Runs | Status | Findings | +|--------|---------|-----|------|--------|----------| +| CEO Review | `/plan-ceo-review` | Scope & strategy | 0 | SKIPPED | No separate CEO skill run; strategic scope tightening was incorporated directly into this plan | +| Codex Review | `/codex review` | Independent 2nd opinion | 2 | CLEAR | Outside voice forced the slice-boundary correction, rejected shell-authoritative binding, and flagged silent fallback + stale-world rules | +| Eng Review | `/plan-eng-review` | Architecture & tests (required) | 1 | CLEAR | Linux owner authority, explicit owner-mode contract, PTY/non-PTY proof echo, non-Linux fail-closed boundary, and full gap-driven test matrix are now locked in | +| Design Review | `/plan-design-review` | UI/UX gaps | 0 | SKIPPED | No UI scope | + +**CODEX:** Two outside-voice passes materially improved the plan. The first caught dual truth, silent fallback, local-generation authority, and test gaps. The second forced the more important correction: slice `03` should own the Linux owner contract only, while projection and invalidation remain slices `04` and `05`. + +**UNRESOLVED:** 0 blocking design decisions remain inside this slice. Intentional deferrals are explicitly recorded above. + +**VERDICT:** ENG CLEARED. Outside-voice findings incorporated. `PLAN-03` is ready to implement after `PLAN-02`, and before runtime-state projection, restart invalidation, and grouped-store follow-on work. diff --git a/llm-last-mile/PLAN-04.md b/llm-last-mile/PLAN-04.md new file mode 100644 index 000000000..76db58853 --- /dev/null +++ b/llm-last-mile/PLAN-04.md @@ -0,0 +1,877 @@ + + +# PLAN-04: Thread World Binding Into Runtime State + +Source file: [04-thread-world-binding-into-runtime-state.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/04-thread-world-binding-into-runtime-state.md) +Branch: `feat/thread-world-binding` +Plan type: backend-only, no UI scope +Review posture: `/autoplan` consolidation pass with `/plan-eng-review` structure and rigor +Status: execution-ready + +## What This Plan Does + +`PLAN-03` made shared-world ownership and generation backend-authoritative. + +The original slice-04 SOW had the right intent and the wrong host-side storage target. The current participant model still rejects any host-scoped runtime record with `world_id/world_generation` in [crates/shell/src/execution/agent_runtime/session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs:355). So this plan does **not** force world binding onto the host participant manifest. + +Instead, `PLAN-04` does four precise jobs: + +1. persist a pending `OrchestrationSessionRecord` before the first shared-world attach/create request, +2. use that persisted parent session record as the session-scoped bridge authority for active `world_id/world_generation`, +3. require every startup/restart/fail-closed drift path to persist parent binding before publishing alerts or host runtime events, +4. expose one live operator proof point through `substrate agent toolbox status --json` while preserving the existing selected-orchestrator `agent status --json` contract. + +That is the smallest correct bridge between: + +- backend-authoritative shared-world proof from `PLAN-03`, +- member invalidation semantics coming in [05-restart-invalidation-semantics.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/05-restart-invalidation-semantics.md), +- and the future grouped `agent-sessions/.json` layout planned in [llm-last-mile/README.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/README.md:3). + +## Scope Challenge + +### 0A. Premise Challenge + +These are the premises this plan accepts after CEO and engineering review. + +1. **The host participant manifest is not the right authority surface for slice `04`.** + - Accepted. + - The code rejects host-scoped participant records with world fields. Fighting that invariant here would create a bigger, riskier contract change than the slice needs. + +2. **The first shared-world attach/create request must already be owner-bound.** + - Accepted. + - If the shell creates world state before the orchestration session id exists durably, slice `04` never gets authoritative startup proof. + +3. **A persisted pending parent session record is necessary before the first world start, not just an in-memory preallocated id.** + - Accepted. + - Pre-live drift, restart, and bootstrap failure paths all need a durable session-scoped authority anchor before the host runtime becomes active. + +4. **The parent session record is the narrowest bridge authority that matches current code.** + - Accepted. + - It already stores `world_id/world_generation` and can be lifted into slice `06` later. + +5. **A live proof surface should exist, but `agent doctor` is the wrong contract for it.** + - Accepted. + - `toolbox status` is already a live-session-oriented surface. `doctor` is still primarily a static readiness / fail-closed preflight contract. + +### 0B. Existing Code Leverage + +| Sub-problem | Existing code | Reuse or replace | +| --------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------- | +| Session-scoped runtime record already stores world fields | [crates/shell/src/execution/agent_runtime/orchestration_session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/orchestration_session.rs:20) | Reuse, extend with explicit mutators | +| Parent-session persistence already exists | [AgentRuntimeStateStore::persist_orchestration_session(...)](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:271) | Reuse as the binding authority write path | +| Shared-world echo validation already exists | [validate_shared_world_echo(...)](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/repl_persistent_session.rs:314) | Reuse, but make first-start requests actually send owner proof | +| Startup currently opens the world before host runtime bootstrap | [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:433) | Keep the overall shape, change the startup context and persistence ordering | +| Pre-live drift/restart logic already exists | [handle_detected_world_drift(...)](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:2586) and [restart_world_session(...)](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:2707) | Reuse, thread explicit startup context through them | +| Post-start active session lookup already exists | [resolve_active_orchestration_session_id()](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:1255) | Reuse after runtime activation, do not rely on it for first start or pre-live drift | +| Live orchestrator resolution already returns parent + child | [resolve_live_orchestrator_session(...)](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:410) | Reuse for toolbox live authority | +| Status contract already suppresses host world fields | [crates/shell/tests/agent_successor_contract_ahcsitc0.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_successor_contract_ahcsitc0.rs:1850) | Reuse exactly | +| Runtime event schema already supports top-level world fields | [crates/common/src/agent_events.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/common/src/agent_events.rs:66) | Reuse unchanged | + +### 0C. Dream State Mapping + +```text +CURRENT +------- +startup world attach can happen before durable orchestration identity exists + │ + ├── REPL memory knows current world binding + ├── alerts may publish that binding + └── persisted live runtime state can still lag or be absent + +THIS PLAN +--------- +persisted pending OrchestrationSessionRecord + │ + ▼ +first attach/create request includes shared-world owner proof + │ + ▼ +parent session record persists active world binding + │ + ├── restart/fail-closed alerts read persisted parent truth + ├── host runtime events read persisted parent truth + ├── toolbox status can prove current binding live + └── selected status row stays host-scoped + +12-MONTH IDEAL +-------------- +grouped agent-sessions registry + │ + ├── orchestrator and member live records share one lineage model + ├── generation invalidation is explicit and queryable + ├── trace remains historical only + └── slice 04 is an obvious bridge, not dead-end storage debt +``` + +### 0C-bis. Implementation Alternatives + +| Approach | What it does | Effort | Risk | Recommendation | +| ----------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------- | -----: | -------: | -------------------------------------------- | +| A. Keep reconstructing from trace or REPL memory | No durable authority fix | S | High | Reject | +| B. Force world binding onto the host participant manifest | Uses the fields the source SOW pointed at | M | Critical | Reject, violates current host invariant | +| C. Persist binding on the parent session record, create that record before first attach, preserve host participant invariants | Smallest correct bridge | M | Low | Recommended | +| D. Collapse `PLAN-04`, `PLAN-05`, and slice `06` into a thin session-registry vertical slice | Better end-state alignment | L | Medium | Strategic alternative, defer for this packet | + +Recommendation: **C**. + +Approach D is a real strategic alternative. It is also a user-direction change to the packet, not a silent implementation refinement. This plan stays inside the current packet order. + +### 0D. Mode-Specific Analysis + +Mode: `SELECTIVE EXPANSION` + +Scope held: + +- no grouped registry layout yet +- no member invalidation yet +- no selected-status schema change +- no host participant world-field exception + +Expansion accepted inside the blast radius: + +- persist the pending parent session record before first attach/create +- add one live proof field to `toolbox status --json` + +### 0E. Temporal Interrogation + +**Hour 1:** the first world attach/create request is owner-bound and a durable parent runtime record already exists on disk. + +**Hour 6:** after one auto-restart and one fail-closed drift event, the parent session record, emitted alerts, and toolbox status all agree on the current binding. + +**Month 3:** `PLAN-05` invalidates prior-generation member records by consuming the already-persisted active binding instead of inventing a separate startup-recovery rule. + +**Year 1:** slice `06` lifts the same session-scoped truth into `agent-sessions/.json` without undoing host participant rules. + +### 0F. Mode Selection + +`SELECTIVE EXPANSION` is the right mode because the source SOW needed correction, not abandonment: + +- startup identity has to be durable before first attach/create +- the authority surface must be session-scoped, not host-participant-scoped +- the live proof surface should be toolbox status, not doctor or selected status + +## What Already Exists + +- A real backend-authoritative shared-world proof shape from `PLAN-03` +- A parent session record that already stores `world_id/world_generation` +- A live orchestrator resolution path that already joins parent and child runtime state +- A runtime event schema that already carries top-level world fields +- A status contract that already forbids leaking host world fields on the selected orchestrator row + +## Architecture Contract + +### No-ambiguity rules + +1. Host-scoped participant manifests remain world-empty in this slice. +2. The authoritative live binding record for the host orchestrator path is `OrchestrationSessionRecord.world_id/world_generation`. +3. A persisted `Allocating` parent session record must exist before the first world attach/create request. +4. Pre-live drift/restart paths must thread that startup context explicitly. They must not rely on active-session lookup. +5. Binding-only persistence writes only the parent session record. It does not depend on rewriting the host participant or lease sidecar. +6. Alerts and host runtime events that claim world binding must read from a successfully persisted parent session snapshot. +7. `substrate agent status --json` keeps the current selected-orchestrator contract: host scope, no world fields. +8. `substrate agent toolbox status --json` is the live proof surface for current binding, via an optional best-effort `active_world_binding` field. +9. Public proof surfaces omit `orchestration_session_id`. That identifier remains an internal same-user control-plane correlation key in this slice. +10. This slice does not define member invalidation or replacement lineage. `PLAN-05` consumes the persisted active binding later. + +### Target state model + +Keep `OrchestrationSessionRecord.world_id` and `world_generation` as the session-scoped active binding fields. + +Add explicit mutators in [crates/shell/src/execution/agent_runtime/orchestration_session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/orchestration_session.rs): + +- `set_world_binding(world_id: impl Into, world_generation: u64)` +- `clear_world_binding()` + +Clear rules: + +- clear on normal world teardown after the parent session transitions terminal +- clear on bootstrap failure after world attach once close succeeds +- do **not** clear during `world_restart_required` fail-closed handling while the current binding is still authoritative + +Do **not** add host-manifest world-binding mutators in this slice. + +### Startup identity contract + +`run_async_repl(...)` changes from: + +```text +start_world_session() +start_host_orchestrator_runtime() +``` + +to: + +```text +prepare_orchestration_session_context() + ├── generate orchestration_session_id + ├── persist parent session state=Allocating + └── carry startup context for pre-live drift/restart paths + +start_world_session(startup_context) +start_host_orchestrator_runtime(startup_context, initial_world_binding) +``` + +That startup context must carry at least: + +- `orchestration_session_id` +- `shell_trace_session_id` +- `workspace_root` +- persisted parent-record path or resolved handle + +### Ordered synchronization contract + +Add one helper in [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs): + +```text +persist_active_world_binding(store, orchestration_session, world_session) + 1. copy world_id/world_generation from WorldSession to parent session record + 2. persist the parent session record + 3. return the persisted parent snapshot +``` + +Important: + +- this helper is the binding authority barrier +- host participant and lease writes remain separate lifecycle writes, not prerequisites for binding truth +- if later lifecycle transitions also update the participant manifest, those are follow-on writes after the binding barrier, not part of it + +### Bootstrap failure contract + +If world startup succeeds but host runtime bootstrap fails later: + +1. close the world session explicitly, +2. mark the pending parent session record terminal with the failure reason, +3. clear binding only after the close succeeds, +4. exit fail-closed. + +This prevents orphaned worlds owned by a session id that never acquired a durable live runtime. + +### Event and alert contract + +Required changes: + +- `build_runtime_message_event(...)` accepts an optional parent-session binding snapshot for host/orchestrator events +- `translate_wrapper_event(...)` does the same +- startup `registered` / `task_start` runtime events are emitted only after the initial binding barrier succeeds when the world session exists +- `emit_world_restarted_alert(...)` and `build_world_restart_required_alert(...)` run only after `persist_active_world_binding(...)` succeeds + +No `AgentEvent` schema expansion is required. + +### Toolbox and status contract + +Keep status conservative: + +- selected orchestrator row stays host-scoped +- no `world_id/world_generation` on that row + +Add a proof field in `substrate agent toolbox status --json`: + +```json +"active_world_binding": { + "world_id": "wld_active_0002", + "world_generation": 7 +} +``` + +Rules: + +- field appears only when a live parent + child orchestrator resolution succeeds +- field omission is non-fatal +- toolbox status retains its normal eligibility and transport semantics + +### Runtime-mode matrix + +This is the exact behavior matrix. No hidden branches. + +| Runtime condition | Shared-world request shape | Parent-session write requirement | Event / alert rule | +| ----------------- | -------------------------- | -------------------------------- | ------------------ | +| First startup with world enabled | `AttachOrCreate` with persisted `orchestration_session_id` | pending parent record must exist before request; active binding must persist after ready frame | startup `registered` / `task_start` host-runtime events wait until the binding barrier succeeds | +| Restart after drift with auto-restart | `ReplaceExpectedGeneration { expected_generation, reason }` | new authoritative binding must persist before publish | `world_restarted` emits only after persistence succeeds | +| Drift with fail-closed posture | no new request in the failing path | current authoritative binding is re-persisted before fail-closed publish | `world_restart_required` emits only after persistence succeeds, then further world work stops | +| `--no-world` startup | no shared-world request | no world-binding write required | runtime bootstrap keeps the current host-only event ordering | + +## Architecture Diagrams + +### Startup flow + +```text +run_async_repl() + │ + ├── prepare_orchestration_session_context() + │ ├── orchestration_session_id allocated + │ └── parent session persisted as Allocating + │ + ├── start_world_session(startup_context) + │ ├── first AttachOrCreate request carries owner proof + │ └── startup drift paths reuse the same context + │ + └── start_host_orchestrator_runtime(startup_context, initial_world_binding) + │ + ├── create host participant manifest + ├── persist_active_world_binding(...) + ├── emit startup runtime events + └── advertise runtime live +``` + +### Restart and fail-closed drift flow + +```text +world drift detected + │ + ├── AutoRestart + │ ├── restart_world_session(context, old_session) -> new WorldSession + │ ├── persist_active_world_binding(...) + │ └── emit world_restarted + │ + └── FailClosed + ├── persist_active_world_binding(...current authoritative binding...) + ├── emit world_restart_required + └── stop before more world work executes +``` + +### Operator truth surfaces + +```text +backend-authoritative world binding + │ + ▼ +OrchestrationSessionRecord + │ + ├── toolbox status can prove live binding + ├── host runtime events carry top-level world fields + ├── restart alerts carry top-level world fields + └── selected status row stays host-scoped +``` + +## Concrete File Touch Plan + +| File / module | Required change | Must not change | +| ------------- | --------------- | --------------- | +| [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) | Introduce a persisted startup context, thread it through `start_world_session(...)`, `handle_detected_world_drift(...)`, `restart_world_session(...)`, and gate world-bound host-runtime events/alerts behind parent-session persistence | Do not invent a second live-state store or reconstruct binding from trace | +| [crates/shell/src/execution/agent_runtime/orchestration_session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/orchestration_session.rs) | Add explicit `set_world_binding(...)` and `clear_world_binding()` mutators on `OrchestrationSessionRecord` | Do not move world binding onto host participants | +| [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) | Reuse `persist_orchestration_session(...)` as the binding authority write barrier and keep `resolve_live_orchestrator_session(...)` as the live proof reader | Do not expand the lease sidecar in this slice | +| [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs) | Add optional `active_world_binding` to toolbox status JSON using the already-authoritative parent + child live resolution | Do not change selected `agent status --json` rows | +| [crates/shell/src/execution/repl_persistent_session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/repl_persistent_session.rs) | Keep echo validation, but make the first-start request actually carry owner proof | Do not loosen request validation to permit proof-less attach/create | +| [crates/shell/tests/](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/) and [llm-last-mile/README.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/README.md) | Add regression coverage and document slice `04` as the runtime-state bridge to slices `05` and `06` | Do not redefine packet order | + +## Implementation Plan + +### Ordered implementation sequence + +Implement in this order. Each step produces a concrete invariant the next step can trust. + +1. **Persist startup authority before world creation** + - allocate `orchestration_session_id` + - create a pending `OrchestrationSessionRecord` + - persist it before the first `AttachOrCreate` request + +2. **Thread that startup authority through every pre-live path** + - `start_world_session(...)` + - `handle_detected_world_drift(...)` + - `restart_world_session(...)` + - any bootstrap-failure path that runs before active-session lookup can succeed + +3. **Add the parent-only binding barrier** + - copy `world_id/world_generation` from `WorldSession` + - persist the parent session record + - treat that persisted snapshot as the source for subsequent publish work + +4. **Reorder publish points** + - startup host-runtime events publish after the binding barrier when a world session exists + - `world_restarted` publishes after the replacement binding persists + - `world_restart_required` publishes after the current binding re-persists + +5. **Add live proof and regression coverage** + - extend toolbox status JSON with optional `active_world_binding` + - preserve selected status contract + - land tests before calling the slice done + +### Step-by-step acceptance gates + +| Step | Acceptance gate | +| ---- | --------------- | +| 1 | a persisted parent record exists on disk before any first-start shared-world request leaves the shell | +| 2 | startup drift and restart logic can operate without `resolve_active_orchestration_session_id()` | +| 3 | one helper owns the authoritative write order for live binding | +| 4 | no alert or host-runtime event can claim a world binding that is not yet persisted in the parent record | +| 5 | toolbox status proves the live binding when possible and selected status rows remain unchanged | + +### Workstream 1: Persisted startup context + +Primary file: + +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) + +Tasks: + +- introduce `PendingOrchestrationSessionContext` or equivalent +- generate `orchestration_session_id` before world startup +- persist parent session state=`Allocating` before first attach/create +- pass the same context into: + - `start_world_session(...)` + - `handle_detected_world_drift(...)` + - `restart_world_session(...)` +- make `--no-world` the explicit bypass case rather than an implicit "no binding barrier exists" assumption + +### Workstream 2: Parent-only binding authority + +Primary files: + +- [crates/shell/src/execution/agent_runtime/orchestration_session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/orchestration_session.rs) +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) +- [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) + +Tasks: + +- add `set_world_binding` / `clear_world_binding` +- add `persist_active_world_binding(...)` +- keep host participant manifests world-empty +- keep lease-sidecar world binding out of slice `04` unless a same-slice reader appears +- keep `persist_runtime_snapshots(...)` for lifecycle persistence, but do not make participant persistence a prerequisite for binding truth + +### Workstream 3: Bootstrap failure and drift ordering + +Primary file: + +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) + +Tasks: + +- after `restart_world_session(...)`, persist parent binding before `world_restarted` +- in fail-closed drift, re-persist the current authoritative binding before `world_restart_required` +- on bootstrap failure after world attach: + - close the world session + - mark the parent record terminal + - clear binding only after close +- treat any `start_host_orchestrator_runtime(...)` failure after world attach as a cleanup path, including startup timeout, missing UAA handle, and wrapper bootstrap failure + +### Workstream 4: Live proof surface and event stamping + +Primary files: + +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) +- [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs) + +Tasks: + +- derive host/orchestrator event world fields from the persisted parent snapshot +- add `active_world_binding` to `toolbox status --json` +- preserve selected status rows exactly as-is +- make proof-field omission non-fatal when live resolution is ambiguous or unavailable +- keep `active_orchestration_session_id` as the control-plane endpoint input, but omit it from the new proof object + +### Workstream 5: Tests and docs + +Primary files: + +- [crates/shell/tests/repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs) +- [crates/shell/tests/agent_successor_contract_ahcsitc0.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_successor_contract_ahcsitc0.rs) +- [llm-last-mile/README.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/README.md) + +Tasks: + +- add first-start owner-proof coverage +- add startup-drift-before-first-command coverage +- add bootstrap-failure cleanup coverage +- add toolbox proof-surface coverage +- document that slice `04` is the session-scoped binding bridge before slice `06` +- update any packet references that still imply a nonexistent `PLAN-05.md` file + +## Architecture Review + +### Locked architecture decisions + +1. **No host-manifest exception.** + - Host participant invariants stay intact. + +2. **Persist the parent session before the first world attach.** + - In-memory preallocation is not enough for pre-live drift and bootstrap failure. + +3. **Parent-only writes are the binding authority barrier.** + - Binding truth must not depend on unrelated participant/lease rewrites. + +4. **Persist before publish.** + - Alerts and host runtime events that claim binding must read from persisted parent truth. + +5. **`toolbox status` is the visible live proof surface.** + - `doctor` stays a preflight/readiness surface. + +### Architecture acceptance gates + +1. **Startup gate** + - First shared-world attach/create is owner-bound and a pending parent record already exists on disk. + +2. **Pre-live drift gate** + - Startup drift/restart paths retain owner proof and alert attribution before the host runtime becomes active. + +3. **Bootstrap cleanup gate** + - If bootstrap fails after world attach, the world is closed and the parent record is marked terminal. + +4. **Ordering gate** + - `world_restarted` and `world_restart_required` only publish after parent binding persistence succeeds. + +5. **Contract gate** + - Selected status rows remain unchanged while toolbox status gains optional proof. + +## Code Quality Review + +### Implementation guardrails + +1. One helper owns parent binding persistence ordering. +2. Startup context is explicit and durable, not inferred from active-session lookup. +3. Host participant manifests stay world-empty with no special-case hole. +4. Public proof surfaces omit `orchestration_session_id`. +5. No trace reconstruction in live runtime paths. +6. No new registry layout or daemon lands in this slice. + +### Minimal-diff rules + +- reuse `OrchestrationSessionRecord` +- reuse `persist_orchestration_session(...)` for binding authority writes +- reuse `resolve_live_orchestrator_session(...)` for toolbox live authority +- avoid lease-sidecar expansion until there is a reader + +## Error & Rescue Registry + +| Failure point | What goes wrong | Expected rescue / fail-closed behavior | +| ------------------------------------------------------- | ------------------------------------------------------------------------------- | ------------------------------------------------------------ | +| no pending parent record before first attach | startup shared world has no durable authority anchor | hard failure, slice not done | +| bootstrap fails after world attach | orphaned shared world remains bound to a session that never became live | close world, mark parent terminal, clear binding after close | +| startup drift before runtime activation | restart/fail-closed path loses owner proof because no active session exists yet | thread explicit startup context through drift paths | +| host manifest accidentally gets world fields | runtime violates its own invariant | compile/test failure, revert to parent-only storage | +| alert published before parent persist | event and live runtime state disagree | fail test, fix ordering | +| toolbox proof field flakes on ambiguous live resolution | operators lose trust in the proof surface | omit field, keep command non-fatal | + +## Test Review + +100% new-path coverage is the goal. This slice is all lifecycle edge conditions. Those are the bugs that wake you up late. + +### Code path coverage + +```text +CODE PATH COVERAGE +=========================== +[+] crates/shell/src/repl/async_repl.rs + │ + ├── [★★ TESTED] world startup / restart flows already exist + ├── [GAP] startup persists pending parent session before first world attach + ├── [GAP] first shared-world request carries AttachOrCreate owner proof + ├── [GAP] startup drift paths reuse the same startup context + ├── [GAP] persist_active_world_binding persists parent snapshot before alerts/events + ├── [GAP] bootstrap failure after attach closes world and marks parent terminal + └── [GAP] fail-closed re-persists current binding before world_restart_required + +[+] crates/shell/src/execution/agent_runtime/orchestration_session.rs + │ + ├── [GAP] set_world_binding updates only parent-session binding fields + └── [GAP] clear_world_binding is used only at approved clear points + +[+] crates/shell/src/execution/agents_cmd.rs + │ + ├── [★★★ TESTED] selected host orchestrator rows omit world fields today + ├── [GAP] toolbox status JSON shows active_world_binding from live parent session + └── [GAP] proof-field omission is non-fatal on ambiguous live resolution + +[+] crates/shell/src/execution/repl_persistent_session.rs + │ + ├── [★★ TESTED] shared-world echo validator already exists + └── [GAP] first-start startup path no longer relies on “no request means no proof” + +───────────────────────────────── +COVERAGE: 3/13 paths tested (23%) + Code paths: 3/13 (23%) +QUALITY: ★★★: 1 ★★: 2 ★: 0 +GAPS: 10 paths need tests +───────────────────────────────── +``` + +### User flow coverage + +```text +USER FLOW COVERAGE +=========================== +[+] First shared-world startup + │ + ├── [GAP] [→E2E] pending parent session persists before first attach/create + ├── [GAP] active binding is persisted before startup runtime events + └── [GAP] bootstrap failure after attach closes the world and marks parent terminal + +[+] Auto-restart on drift + │ + ├── [GAP] [→E2E] parent binding updates before world_restarted + └── [GAP] subsequent host runtime events carry the new binding + +[+] Fail-closed drift + │ + ├── [GAP] [→E2E] second command is blocked + └── [GAP] toolbox status omits or reports binding consistently without flaking + +[+] Operator surfaces + │ + ├── [GAP] agent status selected row remains host-scoped with no world fields + └── [GAP] toolbox status publishes active_world_binding when live resolution succeeds + +───────────────────────────────── +COVERAGE: 0/9 flows tested (0%) + User flows: 0/9 (0%) +GAPS: 9 flows need tests (3 need integration coverage) +───────────────────────────────── +``` + +### Required test additions by file + +#### `crates/shell/src/repl/async_repl.rs` + +Add focused coverage for: + +- persisting pending startup context before first world attach +- threading that context through startup drift/restart +- persisting parent binding before startup runtime events +- closing the world and marking parent terminal on bootstrap failure after attach +- ordering restart/fail-closed alerts behind parent binding persistence + +#### `crates/shell/src/execution/agent_runtime/orchestration_session.rs` + +Add unit coverage for: + +- `set_world_binding("wld_123", 7)` +- approved `clear_world_binding()` call points + +#### `crates/shell/src/execution/repl_persistent_session.rs` + +Add validation coverage proving: + +- first-start shared-world requests send owner proof +- missing proof on first attach/create is rejected by the startup path + +#### `crates/shell/tests/repl_world_first_routing_v1.rs` + +Add integration coverage for: + +- first-start owner proof +- startup drift before first command still retains owner proof +- parent binding persisted before `world_restarted` +- bootstrap failure cleanup after first attach + +#### `crates/shell/tests/agent_successor_contract_ahcsitc0.rs` + +Add contract coverage for: + +- `toolbox status --json` `active_world_binding` +- selected host status rows still omit world fields +- proof-field omission remains non-fatal on ambiguous live resolution + +### Test commands + +Run at minimum: + +```bash +cargo test -p substrate-shell start_host_orchestrator_runtime -- --nocapture +cargo test -p substrate-shell repl_world_first_routing_v1 -- --nocapture +cargo test -p substrate-shell agent_successor_contract_ahcsitc0 -- --nocapture +``` + +Then run: + +```bash +cargo test -p substrate-shell -- --nocapture +cargo test --workspace -- --nocapture +``` + +### QA artifact + +Primary QA artifact for follow-up verification: + +[spensermcconnell-feat-thread-world-binding-eng-review-test-plan-20260429-232824.md](/Users/spensermcconnell/.gstack/projects/atomize-hq-substrate/spensermcconnell-feat-thread-world-binding-eng-review-test-plan-20260429-232824.md) + +## Failure Modes Registry + +| New codepath | Real production failure | Test covers it? | Error handling exists? | User sees clear error? | Critical gap? | +| ------------------------------ | ------------------------------------------------------------------ | --------------- | ---------------------- | ---------------------- | ------------------------ | +| first-start owner proof | world attaches without durable orchestration ownership proof | planned | planned | partial today | yes until fixed | +| bootstrap failure after attach | world stays bound after host runtime never becomes live | planned | planned | yes | yes until fixed | +| pre-live drift | startup restart/fail-closed loses owner proof or alert attribution | planned | planned | yes | yes until fixed | +| alert ordering | restart alert outruns persisted parent truth | planned | planned | yes | yes until fixed | +| toolbox proof surface | proof field flakes on ambiguous live resolution | planned | planned | partial | no, but trust regression | +| selected status contract | host-scoped row leaks world fields | planned | planned | yes | no, contract regression | + +Critical gap rule: + +If the first shared-world startup still happens without a persisted pending parent session and explicit owner proof, this slice is not done. + +## Performance Review + +No new cache is justified. + +Hard rules: + +- startup context is created once per REPL startup +- parent binding writes happen on startup and restart boundaries, not per wrapper event +- no live trace scans are introduced +- no new registry layout or daemon lands here + +Footguns to avoid: + +1. Making active-session lookup guess pending startup sessions. +2. Rewriting participant + lease files as part of binding-only persistence. +3. Using `doctor` as a live runtime truth surface. + +## Cross-Phase Themes + +These are the concerns that kept repeating across the review passes and are now locked into the plan: + +1. **Persist before publish** + - The main failure mode is not "wrong JSON shape." It is claiming a live binding in alerts or runtime events before the parent session record says the same thing on disk. + +2. **Startup is the sharp edge** + - The dangerous gap is the first attach/create and the pre-live drift window. Once the host runtime is active, existing lookup paths are mostly good enough. Before that, they are not. + +3. **Parent record is the bridge, not the destination** + - This slice should improve truth today without pretending it is the final `agent-sessions/` layout. That keeps the diff boring and keeps slice `06` possible. + +4. **Operator visibility must be safe by default** + - A missing proof field in toolbox status is acceptable. A wrong proof field is not. + +## Worktree Parallelization Strategy + +### Dependency table + +| Step | Modules touched | Depends on | +| ---- | --------------- | ---------- | +| Startup context plumbing | `crates/shell/src/repl/` | — | +| Parent-session mutators and store helpers | `crates/shell/src/execution/agent_runtime/` | — | +| Restart ordering and bootstrap cleanup | `crates/shell/src/repl/` | startup context plumbing, parent-session mutators and store helpers | +| Toolbox proof surface | `crates/shell/src/execution/` | parent-session mutators and store helpers | +| Regression tests and packet docs | `crates/shell/tests/`, `llm-last-mile/` | restart ordering and bootstrap cleanup, toolbox proof surface | + +### Parallel lanes + +Lane A: startup context plumbing -> restart ordering and bootstrap cleanup +Lane B: parent-session mutators and store helpers -> toolbox proof surface +Lane C: regression tests and packet docs + +### Execution order + +1. Launch Lane A step 1 and Lane B step 1 in parallel worktrees. +2. Merge those foundations. +3. Run Lane A step 2 and Lane B step 2 in parallel. +4. Run Lane C last, after the runtime and surface contracts settle. + +### Conflict flags + +- Lane A is fully sequential inside `crates/shell/src/repl/`. Do not split startup plumbing and restart ordering across separate worktrees. +- Lane B is sequential inside `crates/shell/src/execution/agent_runtime/` -> `crates/shell/src/execution/`. The toolbox work depends on the parent-session API shape. +- Lane C will touch test files exercised by both prior lanes. Keep it last to avoid churn and merge noise. + +## Deferred Work + +There is no `TODOS.md` in this repo root, so deferrals stay here explicitly. + +1. Member invalidation and replacement lineage on generation changes + Why: explicit job of [PLAN-05](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/05-restart-invalidation-semantics.md) + +2. Full grouped session-centric registry layout under `agent-sessions/` + Why: explicit job of slice `06` + +3. Any change to selected-orchestrator status JSON + Why: deliberately preserved contract in this slice + +4. Lease-sidecar world-binding expansion + Why: defer until a concrete reader requires it before slice `06` + +5. Broader packet reframe into a single thin session-registry vertical slice + Why: valid strategic alternative, but outside this packet’s current user direction + +## NOT in Scope + +- host participant manifest world-field population +- selected-orchestrator status schema changes +- member invalidation semantics +- grouped session-centric registry layout +- public trace schema changes +- UI work + +## Definition of Done + +This slice is done when all of these are true: + +1. A pending parent session record is persisted before the first shared-world attach/create request. +2. That first attach/create request is owner-bound. +3. Startup drift/restart paths retain that startup context before the host runtime becomes active. +4. `OrchestrationSessionRecord` persists the active `world_id/world_generation`. +5. Host participant manifests remain world-empty. +6. If bootstrap fails after world attach, the world is closed and the parent record is marked terminal. +7. `world_restarted` emits only after the new parent binding is persisted. +8. `world_restart_required` emits only after the current authoritative binding is re-persisted. +9. Host/orchestrator runtime events stamp top-level world fields from the persisted parent snapshot. +10. `toolbox status --json` proves or safely omits the live active binding without flaking. +11. `agent status --json` selected host orchestrator rows remain unchanged. + +## Completion Summary + +- Step 0: Scope Challenge - scope accepted with two corrective expansions: persisted pending startup context and toolbox-status proof surface +- Architecture Review: 5 locked decisions, 5 acceptance gates +- Code Quality Review: 6 implementation guardrails, 4 minimal-diff rules +- Test Review: coverage diagrams produced, 19 concrete gaps/assertions identified +- Performance Review: 0 major issues, 3 no-cache/no-guessing rules +- Cross-Phase Themes: 4 recurring concerns locked into the implementation contract +- Error & Rescue Registry: written +- NOT in scope: written +- What already exists: written +- Dream state delta: written +- TODOS.md updates: deferred scope captured in-plan because no `TODOS.md` exists +- Failure modes: 4 critical gaps flagged until startup proof, bootstrap cleanup, and parent-only binding persistence land +- Outside voice: completed and incorporated +- Parallelization: 3 lanes, 2 foundational lanes in parallel, 1 validation lane last +- Lake Score: complete version chosen for every in-slice decision + +## Decision Audit Trail + +| # | Phase | Decision | Classification | Principle | Rationale | Rejected | +| --- | ---------- | -------------------------------------------------------------------- | -------------- | -------------------- | ------------------------------------------------------------------------ | ------------------------------------------------- | +| 1 | Premise | Reject host-manifest world binding as the slice-04 authority surface | Mechanical | Explicit over clever | Current host participant invariant forbids it | Forcing world fields into host manifests | +| 2 | Startup | Persist a pending parent session before first attach/create | Mechanical | Completeness | Pre-live drift and bootstrap failure need a durable anchor | In-memory preallocation only | +| 3 | Authority | Use `OrchestrationSessionRecord` as the bridge store | Mechanical | Pragmatic | Smallest correct bridge before slice `06` | New registry layout in this slice | +| 4 | Ordering | Make parent-only persistence the binding authority barrier | Mechanical | Systems over heroes | Avoids non-atomic participant/lease rewrites for binding-only truth | Tying binding truth to participant + lease writes | +| 5 | Visibility | Add `toolbox status --json` proof surface | Taste | Bias toward action | Gives operators one live, queryable win without breaking selected status | Keeping the slice entirely invisible | +| 6 | Scope | Defer member invalidation to `PLAN-05` | Mechanical | Scope discipline | Packet already gives it a dedicated slice | Pulling invalidation into `PLAN-04` | +| 7 | Storage | Defer lease-sidecar world-binding expansion until a reader exists | Taste | Minimal diff | Avoids bridge debt with no consumer | Expanding lease sidecars preemptively | + +## CEO DUAL VOICES — CONSENSUS TABLE + +| Dimension | Claude Subagent | Codex | Consensus | +| ------------------------------------ | --------------------------------------------------------- | ----------------------------------------------------------------------- | ------------------------------------------------------------------------------ | +| Premises valid? | flagged host-manifest contradiction and startup-proof gap | flagged same plus packet-end-state tension | CONFIRMED on host-manifest contradiction and startup proof | +| Right problem to solve? | yes, but only with corrected authority layer | yes, but packet-wide vertical-slice alternative exists | CONFIRMED for current packet, strategic alternative deferred | +| Scope calibration correct? | bridge-store approach okay after correction | warned against over-investing in transitional stores | CONFIRMED after narrowing to parent-only bridge and deferring registry reshape | +| Alternatives sufficiently explored? | requested parent-only and no-lease variants | requested thin session-registry vertical slice as strategic alternative | CONFIRMED that alternatives had to be expanded | +| Competitive / operator risk covered? | wanted one visible proof point | wanted a visible proof point but not on status | CONFIRMED via toolbox-status proof surface | +| 6-month trajectory sound? | okay if startup and lifecycle edges are fixed | okay if clearly framed as bridge, not end-state | CONFIRMED after bridge framing | + +## ENG DUAL VOICES — CONSENSUS TABLE + +| Dimension | Claude Subagent | Codex | Consensus | +| ----------------------------------------- | --------------------------------------------------------------- | ------------------------------------------------------------- | ---------------------------------------------------------------- | +| Architecture sound? | yes after shifting to parent-session authority | yes after parent-only authority and persisted startup context | CONFIRMED | +| Test coverage sufficient? | no, bootstrap failure + startup drift gaps missing | no, same gaps missing | CONFIRMED gaps added | +| Performance risks addressed? | mostly yes, but parent+child atomicity was overstated | same | CONFIRMED by parent-only binding writes | +| Security threats covered? | wanted explicit trust-boundary note and less public id exposure | same concern indirectly | CONFIRMED via proof-surface omission of orchestration_session_id | +| Error paths handled? | bootstrap failure, pre-live drift, and clear points missing | same | CONFIRMED gaps added | +| Deployment / operational risk manageable? | yes after cleanup + ordering rules | yes after doctor/toolbox surface correction | CONFIRMED | + +## GSTACK REVIEW REPORT + +| Review | Trigger | Why | Runs | Status | Findings | +| ------------- | --------------------- | ------------------------------- | ---- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| CEO Review | `/plan-ceo-review` | Scope & strategy | 2 | CLEAR | Corrected host-manifest premise, added startup-proof requirement, expanded alternatives, and reframed slice `04` as a bridge to the session-centric registry | +| Codex Review | `/codex review` | Independent 2nd opinion | 2 | CLEAR | Outside voices forced the parent-session authority model, pushed live proof onto toolbox status instead of doctor, and tightened bridge vs end-state framing | +| Eng Review | `/plan-eng-review` | Architecture & tests (required) | 2 | CLEAR | Added persisted pending startup context, pre-live drift threading, bootstrap-failure cleanup, parent-only binding writes, and full regression matrix | +| Design Review | `/plan-design-review` | UI/UX gaps | 0 | SKIPPED | No UI scope | + +**CODEX:** Two Codex passes materially improved this plan. The first rejected host-manifest storage and pushed the bridge-store framing. The second forced the more important lifecycle corrections: persist the pending parent record before first attach, keep binding writes parent-only, and move the proof surface from doctor to toolbox status. + +**CROSS-MODEL:** Both Claude and Codex independently flagged the same four risks: host-manifest contradiction, startup owner-proof gap, lifecycle cleanup on bootstrap failure, and the need for a visible but contract-safe proof surface. That is high-confidence signal. + +**UNRESOLVED:** 0 blocking implementation decisions remain inside slice `04`. The only deferred item is the larger packet-level alternative of collapsing slices `04`/`05`/`06` into one thinner session-registry vertical slice. + +**VERDICT:** CEO + ENG CLEARED. `PLAN-04` is ready to implement after `PLAN-03`, before `PLAN-05`, and as an explicit bridge toward slice `06`. diff --git a/llm-last-mile/PLAN-05.md b/llm-last-mile/PLAN-05.md new file mode 100644 index 000000000..cff7a1493 --- /dev/null +++ b/llm-last-mile/PLAN-05.md @@ -0,0 +1,673 @@ + + +# PLAN-05: Restart Invalidation Semantics for Live State + +Source file: [05-restart-invalidation-semantics.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/05-restart-invalidation-semantics.md) +Branch: `feat/restart-invalidation-semantics` +Plan type: backend-only, no UI scope +Review posture: consolidation pass with `/plan-eng-review` structure and rigor +Status: execution-ready after `PLAN-04`, before `PLAN-06` + +## Objective + +`PLAN-03` made shared-world ownership and generation backend-authoritative. + +`PLAN-04` made the parent orchestration-session record the host-side authority for the active +`world_id` and `world_generation`. + +`PLAN-05` is the next narrow slice. Its job is to make that authoritative generation transition +invalidate stale world-scoped member live state immediately and permanently for live surfaces. + +After this plan lands: + +1. when a parent `orchestration_session_id` advances from generation `G` to `G+1`, every older + world-scoped member for that same session stops being authoritative-live, +2. invalidated member tombstones suppress trace fallback, so old rows stay historical only, +3. restart sequencing fails closed if replacements are not ready, +4. `status` and toolbox surfaces keep using the current live-state authority instead of + reconstructing stale liveness from trace. + +This plan does **not** redesign storage layout, trace schema, or session grouping. That remains +owned by [PLAN-06](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/06-session-centric-state-store.md). + +## Step 0: Scope Challenge + +### 0A. Repo truth that this plan must follow + +The source SOW is directionally right, but the repository has moved since it was written. This +plan is locked to current repo truth: + +1. authoritative participant storage is + [`participants/*.json`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs), + not `handles/*.json`; `handles/*.json` is legacy compatibility input, +2. the authoritative active generation already lives on + [`OrchestrationSessionRecord.world_id/world_generation`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/orchestration_session.rs), +3. [`list_live_manifests()`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) + is already the right low-level live read and stays in place, +4. [`toolbox status --json`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs) + already exposes `active_world_binding` from `PLAN-04`, +5. runtime aliases such as `session_handle_id` and `resumed_from_session_handle_id` still exist, + but the persisted lineage field on disk is `resumed_from_participant_id`. + +If the implementation or tests contradict any of the five rules above, the code is wrong, not the +plan. + +### 0B. Existing code to reuse + +| Sub-problem | Existing code | Plan | +| --- | --- | --- | +| Parent session already stores active world binding | [crates/shell/src/execution/agent_runtime/orchestration_session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/orchestration_session.rs) | Reuse exactly | +| Parent binding persistence already exists | [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) | Reuse exactly | +| Participant state already has `Invalidated` and authoritative-live checks | [crates/shell/src/execution/agent_runtime/session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs) | Reuse, add one explicit invalidation helper | +| Live participant reads already exclude invalidated rows | [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) | Reuse, do not replace | +| Status already overlays live manifests over trace | [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs) | Reuse, narrow the suppression boundary and add tombstones | +| Restart alerts already prove parent binding persistence ordering | [crates/shell/tests/repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs) | Reuse, extend for member invalidation ordering | +| Toolbox already proves live parent binding from the authoritative session record | [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs) | Reuse, do not add a second proof surface | + +### 0C. Minimum viable slice + +The smallest correct slice is: + +1. one record-level invalidation helper on participants, +2. one session-local sweep in the runtime state store, +3. one status suppression fix that uses invalidated tombstones, +4. one restart-ordering rule that prefers fail-closed absence over stale liveness, +5. regression coverage for restart, tombstones, and cross-session same-agent collisions. + +Everything else is extra. + +### 0D. Complexity check + +This plan touches a bounded set of surfaces: + +- runtime state model: `execution/agent_runtime/` +- status surface: `execution/agents_cmd.rs` +- restart orchestration: `repl/async_repl.rs` +- regression suites: `crates/shell/tests/` +- contract docs: `docs/TRACE.md` and the successor protocol draft + +That is one bounded runtime seam, not a rewrite: + +- 0 new crates +- 0 new storage backends +- 0 new public JSON surfaces +- 0 new artifact types or distribution work + +Scope reduction is **not** needed. The slice is already minimal. + +### 0E. Search and built-in check + +No framework or runtime built-in replaces this work. The missing behavior is local repository +policy: + +- how participant invalidation is recorded, +- how live state is selected, +- how trace fallback is suppressed, +- how restart commit ordering avoids dual-live generations. + +This is a repo seam, not a library shopping problem. + +### 0F. What already exists + +- `AgentRuntimeSessionState::Invalidated` already exists. +- `is_authoritative_live()` already excludes non-live participant rows. +- parent sessions already persist `world_id/world_generation`. +- `toolbox status --json` already reports `active_world_binding` when the parent session is live. +- restart alert tests already prove the parent binding persists before alert publication. +- toolbox env already fails closed on invalidated manifests. +- world-scoped member status rows already carry top-level `world_id/world_generation`. + +### 0G. NOT in scope + +- session-grouped registry migration under `run/agent-hub/sessions//...` +- new active-generation index files +- trace schema redesign +- new public `agent status` fields for restart placeholders +- host-orchestrator invalidation from member generation rollover +- toolbox mutating tools +- UI work + +## Architecture Contract + +### No-ambiguity rules + +1. `participants/*.json` plus `sessions/*.json` are the authoritative runtime store. +2. Only `role=member` plus `execution.scope=world` participants are invalidated by shared-world + generation rollover. +3. The active generation source is the parent + `OrchestrationSessionRecord.world_generation` persisted by `PLAN-04`. +4. `PLAN-05` consumes that generation. It does not assign it. +5. A participant in `state=invalidated` is a tombstone for live surfaces. It stays on disk for + audit and compatibility reads, but it may not re-authorize live state. +6. Trace fallback suppression identity is + `(orchestration_session_id, agent_id, execution.scope)`. `(agent_id, role)` is too weak. +7. Persisted replacement lineage uses `resumed_from_participant_id`. +8. Restart ordering prefers fail-closed absence over stale presence. +9. `status` stays a live-session surface, not a historical ledger. +10. `PLAN-06` still owns registry regrouping and migration away from the flat participant tree. + +### Current state to target state + +```text +CURRENT +------- +parent session already persists the active world binding + │ + ├── live participant reads exclude Invalidated + ├── restart alerts publish authoritative generation + └── status still suppresses trace by (agent_id, role) + +TARGET +------ +parent session advances to generation G+1 + │ + ├── state store invalidates generation < G+1 world members for that session + ├── invalidated members remain on disk as tombstones + ├── status suppresses stale trace rows by session-aware identity + └── replacement members appear live only after old generation is already dead + +LATER +----- +PLAN-06 groups the same parent + member lineage under session-centric storage +without changing the invalidation rule +``` + +### Restart ordering contract + +This is the critical sequencing rule: + +```text +backend accepts replacement world binding G+1 + │ + ▼ +PLAN-04 persists parent session binding for G+1 + │ + ▼ +PLAN-05 invalidates every older world-scoped member in that session + │ + ▼ +replacement member manifests persist for G+1 + │ + ▼ +restart success reporting becomes visible +``` + +The order is strict because the bad crash window is "replacement looks live before stale +generation is dead." + +Crash behavior must be: + +- after invalidation, before replacement persistence: member is absent, which is safe +- after replacement persistence, before invalidation: two generations can look live, which is not safe + +### Status and tombstone suppression contract + +`substrate agent status --json` remains a live-state view. It must: + +1. build live projections from authoritative live manifests first, +2. build a second suppression set from invalidated world-scoped member tombstones, +3. suppress trace fallback rows if either a live row or a tombstone owns the same + `(orchestration_session_id, agent_id, execution.scope)` identity, +4. keep host-orchestrator behavior unchanged, +5. keep trace historical. Historical rows may remain on disk forever. They may not come back into + `sessions[]` once a tombstone exists. + +### Architecture diagrams + +#### Generation rollover commit flow + +```text +world restart accepted by backend + │ + ▼ +parent session binding persists generation G+1 + │ + ▼ +invalidate_prior_world_members(session, G+1) + │ + ├── member_a G -> Invalidated tombstone + ├── member_b G -> Invalidated tombstone + └── host orchestrator untouched + │ + ▼ +replacement member manifests persist with generation G+1 + │ + ▼ +world_restarted alert publishes +``` + +#### Status selection and tombstone suppression + +```text +participants/*.json + sessions/*.json + │ + ├── live manifests ----------┐ + ├── invalidated tombstones --┼── suppression set + └── trace.jsonl history -----┘ + │ + ▼ + build_status_report(...) + ├── live rows selected first + ├── trace rows suppressed by live identity + ├── trace rows suppressed by tombstone identity + └── only remaining historical rows may fall back into sessions[] +``` + +#### Cross-session same-agent isolation + +```text +session A / agent codex / world / generation 7 +session B / agent codex / world / generation 2 + │ + ├── invalidate session A generation < 8 + └── do not touch session B + +suppression key = (orchestration_session_id, agent_id, execution.scope) +not (agent_id, role) +``` + +## Concrete File Touch Plan + +### 1. Participant invalidation primitive + +Primary file: +[crates/shell/src/execution/agent_runtime/session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs) + +Add one explicit helper, for example: + +```text +invalidate_for_generation_rollover(reason, error_bucket) + 1. transition_state(Invalidated) + 2. mark_terminal_state(reason) + 3. set last_error_bucket / last_error_message + 4. rely on terminal-state handling to clear ownership validity +``` + +Required behavior: + +- uses existing `Invalidated`, not a new "superseded" state +- records terminal metadata consistently +- records failure reason in one place +- keeps persisted lineage vocabulary `participant_id`-first +- does not alter host/world invariants + +Must not do: + +- invent a new state enum variant +- mutate invalidation fields ad hoc in multiple call sites +- rename persisted lineage fields to match runtime aliases + +### 2. Session-local generation sweep + +Primary file: +[crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) + +Add one bounded helper: + +```text +invalidate_prior_world_members( + orchestration_session_id, + active_generation, + reason +) -> Vec +``` + +Rules: + +- iterate authoritative participant files, not just live projections +- mutate only rows that satisfy all of: + - same `orchestration_session_id` + - `role=member` + - `execution.scope=world` + - `world_generation < active_generation` + - current state is still live +- skip host-scoped participants +- skip already-invalidated rows +- persist each mutated participant back to `participants/*.json` +- return invalidated participant ids for logging and test assertions +- remain idempotent when called twice + +Must not do: + +- add a session-level index file +- rewrite directory layout +- change `list_live_manifests()` semantics + +### 3. Status suppression and trace fail-closed behavior + +Primary file: +[crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs) + +Required changes: + +- replace `session_fallback_suppression_key(...)` +- build suppression keys from live projections +- build suppression keys from invalidated world-member tombstones on disk +- suppress trace fallback on either match +- keep host-orchestrator selected-row behavior unchanged +- keep nested gateway correlation logic unchanged + +Required suppression identity: + +```text +(orchestration_session_id, agent_id, execution.scope) +``` + +Must not do: + +- keep `(agent_id, role)` as the live/trace suppression boundary +- suppress session B because session A used the same agent id +- redesign the public `status` JSON shape + +### 4. Restart ordering and replacement publication + +Primary file: +[crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) + +Required changes: + +- keep the `PLAN-04` parent-binding barrier +- after generation `G+1` is accepted and persisted on the parent, invalidate stale generation + `G` member participants before replacement publication is considered complete +- allow temporary absence +- forbid temporary stale-live visibility +- emit `world_restarted` only after: + - parent binding persists, + - old member generation is invalidated, + - replacement publication succeeds, or the path intentionally remains fail closed + +Must not do: + +- publish replacement success before stale generation is dead +- keep the old generation live "for continuity" +- move authority back into trace + +### 5. Tests and docs + +Primary files: + +- [crates/shell/tests/agent_successor_contract_ahcsitc0.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_successor_contract_ahcsitc0.rs) +- [crates/shell/tests/repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs) +- [docs/TRACE.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/TRACE.md) +- [docs/project_management/packs/draft/agent-hub-core-successor-identity-tuple-compatible/agent-hub-session-protocol-spec.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/project_management/packs/draft/agent-hub-core-successor-identity-tuple-compatible/agent-hub-session-protocol-spec.md) + +Required changes: + +- new regression fixtures prefer `participants/*.json` +- keep at most one legacy `handles/*.json` compatibility test where it still matters +- update doc wording so tombstones beat trace for live-state selection +- update lineage wording so the persisted field name is `resumed_from_participant_id` + +## Error & Rescue Registry + +| Failure point | What goes wrong | Required rescue | +| --- | --- | --- | +| suppression key stays `(agent_id, role)` | one live session suppresses another unrelated concurrent session with the same member agent | change key to `(orchestration_session_id, agent_id, execution.scope)` | +| stale generation invalidates after replacement is surfaced | crash window can leave two generations live | reorder to invalidate old generation before replacement is considered live | +| trace fallback ignores tombstones | dead generation comes back into `sessions[]` | build tombstone suppression from invalidated world-member participants | +| tests still write only `handles/*.json` | regressions exercise the compatibility path, not the authority path | add `participants/*.json`-first fixtures for this slice | +| plan/tests use runtime alias field names | implementation targets the wrong serialized contract | name and assert `resumed_from_participant_id` | + +## Test Review + +100 percent new-path coverage is the target. This slice is mostly stale-state and restart-window +logic. Those are the paths that quietly poison operator trust if they are not tested. + +### Code path coverage + +```text +CODE PATH COVERAGE +=========================== +[+] crates/shell/src/execution/agent_runtime/session.rs + │ + ├── [★★ TESTED] Invalidated already exists and is excluded from live reads + └── [GAP] generation-rollover invalidation helper stamps state + terminal metadata + +[+] crates/shell/src/execution/agent_runtime/state_store.rs + │ + ├── [★★ TESTED] live-manifest reads already filter non-live rows + ├── [GAP] invalidate_prior_world_members(...) sweeps one session only + ├── [GAP] sweep skips host orchestrator rows + ├── [GAP] sweep is idempotent + └── [GAP] sweep and persisted parent generation stay in sync after restart + +[+] crates/shell/src/execution/agents_cmd.rs + │ + ├── [★★★ TESTED] live manifest already beats trace fallback for selected orchestrator cases + ├── [★★ TESTED] invalidated toolbox env already fails closed + ├── [GAP] tombstone suppression beats trace fallback for invalidated members + ├── [GAP] suppression key uses orchestration_session_id + └── [GAP] same-agent concurrent sessions do not suppress each other + +[+] crates/shell/src/repl/async_repl.rs + │ + ├── [★★★ TESTED] parent binding persists before restart alerts publish + ├── [GAP] member invalidation runs before replacement publication is considered complete + └── [GAP] crash-window ordering is fail closed rather than dual-live + +───────────────────────────────── +COVERAGE: 5/13 paths tested (38%) +QUALITY: ★★★: 2 ★★: 3 ★: 0 +GAPS: 8 code paths need tests +───────────────────────────────── +``` + +### User flow coverage + +```text +USER FLOW COVERAGE +=========================== +[+] Auto-restart with replacement ready + │ + ├── [★★ TESTED] parent binding persists before restart alert publishes + ├── [GAP] [→E2E] old generation members disappear before replacement is surfaced live + └── [GAP] replacement lineage persists via resumed_from_participant_id + +[+] Fail-closed restart, replacement not ready + │ + ├── [GAP] [→E2E] member is absent from status, not stale-live + └── [GAP] trace rows remain visible only as history + +[+] Concurrent sessions using the same member agent + │ + ├── [GAP] invalidating session A does not suppress session B + └── [GAP] world-filtered status shows both sessions when both are truly live + +[+] Crash recovery / rerun behavior + │ + ├── [GAP] rerunning the invalidation sweep is idempotent + └── [GAP] partial replacement persistence cannot leave two live generations + +───────────────────────────────── +COVERAGE: 1/8 flows tested (12%) +GAPS: 7 user flows need tests (2 need integration coverage) +───────────────────────────────── +``` + +### Required test additions by file + +#### `crates/shell/src/execution/agent_runtime/session.rs` + +Add unit coverage for: + +- generation-rollover invalidation helper sets: + - `state=Invalidated` + - terminal metadata + - `last_error_bucket` + - `last_error_message` +- replacement constructor assertions continue to validate world lineage correctly + +#### `crates/shell/src/execution/agent_runtime/state_store.rs` + +Add unit coverage for: + +- `invalidate_prior_world_members("sess_a", 8, "...")` only touches session A +- host orchestrator and host-scoped rows remain untouched +- calling the sweep twice is harmless +- already-invalidated rows remain stable + +#### `crates/shell/tests/agent_successor_contract_ahcsitc0.rs` + +Add contract coverage for: + +- invalidated world-member tombstone suppresses stale trace fallback +- generation `8` live replacement wins over generation `7` trace history +- no-replacement-yet case omits the member row entirely +- two concurrent sessions that both use `agent_id=codex` remain independently visible +- persisted lineage is asserted with `resumed_from_participant_id` + +#### `crates/shell/tests/repl_world_first_routing_v1.rs` + +Add integration coverage for: + +- after restart binding persists, stale member generation invalidates before success publication +- crash or injected failure between invalidation and replacement leaves the member absent +- replacement generation becomes the only live generation after the full restart path + +#### Docs + +Add wording checks or examples for: + +- `participants/*.json` as the authority path +- tombstone-beats-trace wording in `docs/TRACE.md` +- persisted lineage field naming in the successor protocol doc + +### Test commands + +Run at minimum: + +```bash +cargo test -p substrate-shell agent_runtime::state_store -- --nocapture +cargo test -p substrate-shell agent_successor_contract_ahcsitc0 -- --nocapture +cargo test -p substrate-shell repl_world_first_routing_v1 -- --nocapture +``` + +Then run: + +```bash +cargo test -p substrate-shell -- --nocapture +cargo test --workspace -- --nocapture +``` + +### QA artifact + +Primary QA handoff artifact: +[spensermcconnell-feat-restart-invalidation-semantics-eng-review-test-plan-20260430-115447.md](/Users/spensermcconnell/.gstack/projects/atomize-hq-substrate/spensermcconnell-feat-restart-invalidation-semantics-eng-review-test-plan-20260430-115447.md) + +## Failure Modes Registry + +| New codepath | Real production failure | Test covers it? | Error handling exists? | User sees clear error? | Critical gap? | +| --- | --- | --- | --- | --- | --- | +| session-local invalidation sweep | stale generation remains live beside the replacement generation | planned | planned | partial today | yes until fixed | +| tombstone-based status suppression | trace history resurrects invalidated member rows | planned | planned | no today | yes until fixed | +| same-agent concurrent sessions | one session suppresses another because `agent_id` matches | planned | no | no | yes until fixed | +| fail-closed no-replacement window | member keeps looking live even though replacement failed | planned | planned | no today | yes until fixed | +| persisted lineage naming | tests assert runtime alias fields instead of serialized participant lineage | planned | yes | yes | no, but contract drift | + +Critical gap rule: + +If generation rollover can still leave two live generations visible, or if invalidated tombstones +do not suppress trace fallback, this slice is not done. + +## Performance Review + +This is a correctness project, not a performance project. + +Still, four rules matter: + +1. the invalidation sweep is bounded to one orchestration session and runs only at restart boundaries, +2. no cache or index file is justified before measurement, +3. `agent status` can afford one extra participant read pass for tombstone suppression at current + scale, +4. if scale later makes the sweep expensive, that is the trigger for `PLAN-06`, not for an ad hoc + cache inside this slice. + +## Worktree Parallelization Strategy + +### Dependency table + +| Step | Modules touched | Depends on | +| --- | --- | --- | +| A. Participant invalidation helper | `crates/shell/src/execution/agent_runtime/` | — | +| B. Status suppression key + tombstone pass | `crates/shell/src/execution/` | A | +| C. Restart ordering | `crates/shell/src/repl/` | A | +| D. Tests and docs | `crates/shell/tests/`, `docs/`, `llm-last-mile/` | B, C | + +### Parallel lanes + +- Lane A: step A +- Lane B: step B +- Lane C: step C +- Lane D: step D + +### Execution order + +1. launch Lane A first, because it defines the shared invalidation primitive and helper signature, +2. after Lane A lands, launch Lane B and Lane C in parallel worktrees, +3. run Lane D last, once status behavior and restart ordering are stable. + +### Conflict flags + +- Lanes B and C both depend on the final helper signature from Lane A. Do not start them early. +- Lane D will touch assertions for both status and restart behavior. Keep it last to reduce merge churn. + +### Parallelization verdict + +This plan has **one foundation lane**, **two safe parallel implementation lanes**, and **one final +validation lane**. + +## Deferred Work + +There is no `TODOS.md` in the repo root, so explicit deferrals stay here: + +1. session-grouped registry layout under `run/agent-hub/sessions//...` + - owned by [PLAN-06](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/06-session-centric-state-store.md) +2. explicit active-generation index file + - defer unless sweep cost proves unacceptable before `PLAN-06` +3. new public status fields for restart placeholders + - this slice keeps status live-only and uses absence plus alerts instead +4. broader member runtime ownership bootstrap productization + - may need follow-on work once real member producers land beyond the current draft and test path +5. trace-schema widening for old/new participant correlation + - useful later, not required for the invalidation contract itself + +## Definition of Done + +This slice is done only when all of the following are true: + +1. parent session `world_generation` from `PLAN-04` is the active-generation source, +2. every older world-scoped member for that session becomes `Invalidated` after generation rollover, +3. `list_live_manifests()` returns zero stale-generation world members after invalidation commits, +4. `substrate agent status --json` suppresses stale trace fallback when a matching tombstone exists, +5. same-agent members in different orchestration sessions do not suppress each other, +6. replacement lineage is persisted and asserted with `resumed_from_participant_id`, +7. restart success reporting becomes visible only after parent binding and member invalidation commit, +8. no-replacement-yet windows remain fail closed and do not keep stale live rows visible, +9. `toolbox status --json` behavior from `PLAN-04` remains unchanged, +10. docs and fixtures use authoritative storage terminology and persisted field names. + +## Completion Summary + +- Step 0: scope accepted as-is after repo-state corrections +- Architecture Review: one bounded invalidation contract, no storage rewrite +- Code Quality Review: one helper, one sweep, one suppression fix, one restart-ordering rule +- Test Review: coverage diagrams produced, 15 concrete gaps or assertions identified +- Performance Review: correctness-first, no new caches or indexes justified +- NOT in scope: written +- What already exists: written +- Failure modes: 4 critical gaps remain until invalidation, suppression, and ordering land +- Parallelization: 4 steps, 2 safe parallel implementation lanes +- Lake Score: complete option chosen for every in-slice decision + +## GSTACK REVIEW REPORT + +| Review | Trigger | Why | Runs | Status | Findings | +| --- | --- | --- | --- | --- | --- | +| CEO Review | `/plan-ceo-review` | Scope and strategy | 0 | SKIPPED | Backend-only slice, no separate CEO pass run | +| Codex Review | `/codex review` | Independent 2nd opinion | 0 | SKIPPED | No separate Codex review run | +| Eng Review | `/plan-eng-review` | Architecture and tests (required) | 1 | CLEAR | Corrected authority-path terminology, tightened restart ordering, fixed suppression boundary, expanded regression matrix | +| Design Review | `/plan-design-review` | UI/UX gaps | 0 | SKIPPED | No UI scope | + +**UNRESOLVED:** 0 blocking design decisions remain inside slice `05`. The main deferred decision is whether a future active-generation index is still needed once `PLAN-06` lands. + +**VERDICT:** ENG CLEARED. `PLAN-05` is ready to implement after `PLAN-04` and before `PLAN-06`. diff --git a/llm-last-mile/PLAN-06.md b/llm-last-mile/PLAN-06.md new file mode 100644 index 000000000..32cc647e0 --- /dev/null +++ b/llm-last-mile/PLAN-06.md @@ -0,0 +1,823 @@ + + +# PLAN-06: Session-Centric Runtime State Store + +Source file: [06-session-centric-state-store.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/06-session-centric-state-store.md) +Branch: `feat/session-centric-state-store` +Plan type: backend and operator-CLI slice, no UI scope +Review posture: prior `/autoplan` and `/plan-eng-review` findings are now folded into one execution contract +Status: execution-ready after [PLAN-05](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-05.md) + +## Objective + +The problem is not directory aesthetics. The problem is that live runtime truth is still partly +reconstructed by callers instead of being served directly by the store. + +The repository already has the right identity boundary: + +- parent orchestration truth in `sessions/*.json` +- participant truth in `participants/*.json` +- compatibility tail in `handles/*.json` +- toolbox socket naming keyed by `orchestration_session_id` +- status suppression keyed by `(orchestration_session_id, agent_id, execution.scope)` + +What is still wrong is the read model. `status`, `toolbox`, and invalidation logic still rely on +flat scans or caller-owned regrouping in places where the store should already be authoritative. + +`PLAN-06` fixes that in one bounded slice: + +1. add a store-owned session projection over the current runtime sources, +2. move every participant enumerator and mutator onto one shared source walker, +3. cut `agent status` and `toolbox status|env` over to session-record reads, +4. then move canonical persistence under one session-root layout so disk shape matches API shape, +5. keep compatibility reads inside the store until upgrade confidence is green. + +After this plan lands: + +1. the store can answer `load_session()` and `list_live_sessions()` directly, +2. `substrate agent status` builds live rows from store-owned session records, not flat manifest regrouping, +3. `substrate agent toolbox status|env` resolves exactly one live session record and fails closed on ambiguity, +4. `PLAN-05` invalidation, tombstones, and same-agent concurrent-session visibility still hold, +5. canonical live runtime state writes under one session root, with flat compatibility reads bounded inside the store. + +This slice does not add explicit selector UX such as `--orchestration-session-id`. Fail-closed +ambiguity is sufficient for this step. + +## Scope Lock + +### Repo truth this plan must follow + +The source SOW is directionally right, but some of its current-state description is stale. This +plan is locked to current repository truth: + +1. authoritative live runtime state is already split between flat parent session snapshots in + [`crates/shell/src/execution/agent_runtime/state_store.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) + and flat participant records in the same module, +2. `handles/*.json` is already legacy compatibility input, not the canonical write target, +3. [`OrchestrationSessionRecord`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/orchestration_session.rs) + already owns session-level truth such as `active_session_handle_id`, `world_id`, and `world_generation`, +4. [`resolve_live_orchestrator_session()`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) + already fails closed on ambiguity and is the correct behavioral baseline, +5. [`build_status_report()`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs) + still reconstructs the live overlay from `list_live_manifests()` plus trace fallback suppression, +6. `PLAN-05` helpers such as `invalidate_stale_world_members_for_session()` and + `list_invalidated_participants()` still enumerate only flat `participants/*.json`, +7. parent-session writes and participant writes are separate atomic rewrites today, so torn roots are a normal runtime condition, not an edge case, +8. test helpers and docs still hardcode flat paths in several places, including + `agent_successor_contract_ahcsitc0.rs`, `agent_hub_trace_persistence.rs`, + `repl_world_first_routing_v1.rs`, `docs/TRACE.md`, and `docs/USAGE.md`. + +If the implementation or tests contradict any of those eight points, the code is wrong, not the +plan. + +### What already exists + +| Sub-problem | Existing code | Plan | +| --- | --- | --- | +| Session-level parent truth | [crates/shell/src/execution/agent_runtime/orchestration_session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/orchestration_session.rs) | Reuse | +| Participant validation, liveness, lineage | [crates/shell/src/execution/agent_runtime/session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs) | Reuse | +| Fail-closed live orchestrator resolution | [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) | Promote into session-record API | +| Parent world-binding authority | [crates/shell/src/execution/agent_runtime/orchestration_session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/orchestration_session.rs) | Reuse | +| `PLAN-05` tombstone invalidation | [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) | Preserve unchanged, move to unified walkers | +| Toolbox JSON shape | [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs) | Keep stable | +| REPL writer already has `orchestration_session_id` | [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) | Reuse, no lifecycle redesign | + +### NOT in scope + +- public schema changes to `AgentRuntimeParticipantRecord` +- trace event schema redesign +- new toolbox mutating tools +- new CLI selector UX such as `--orchestration-session-id` +- new cache or index files +- replay or doctor contract redesign +- UI work + +### Chosen approach + +| Approach | Summary | Effort | Risk | Decision | +| --- | --- | --- | --- | --- | +| A. Projection first, then canonical layout in the same slice | Cut consumers onto store-owned session projections first, then align disk layout | Medium | Medium | **Accepted** | +| B. Layout first | Move files first, then repair consumers | Medium | High | Rejected | +| C. Stay flat forever | Add session projection but never move canonical layout | Small | Medium | Rejected as end state | +| D. Add an index file | Keep flat manifests and add derived session index | Medium | High | Rejected | + +The accepted path is the smallest correct slice. It solves the operator problem first, keeps the +diff boring, and avoids inventing a second source of truth. + +## Execution Contract + +### Problem statement + +The store has to become session-centric by read contract before it becomes session-centric by disk +shape. + +That means: + +- session grouping belongs in the store layer, not in `agents_cmd.rs`, +- compatibility burden belongs in the store layer, not in tests or callers, +- parent and participant precedence must be resolved per object, not per directory root, +- torn writes must be modeled explicitly instead of hand-waved away, +- `PLAN-05` invalidation semantics are part of the acceptance gate, not an adjacent concern. + +### Canonical target layout + +End-state canonical runtime store root: + +```text +$SUBSTRATE_HOME/run/agent-hub/ + sessions/ + / + session.json + participants/ + .json + leases/ + .lease +``` + +Compatibility inputs during migration: + +```text +$SUBSTRATE_HOME/run/agent-hub/ + sessions/.json + participants/.json + handles/.json +``` + +### Required store record + +Add one store-layer session projection type: + +```rust +pub(crate) struct AgentRuntimeSessionRecord { + pub session: OrchestrationSessionRecord, + pub participants: Vec, + pub warnings: Vec, +} +``` + +Required helpers: + +- `orchestration_session_id()` +- `is_complete()` +- `live_participants()` +- `live_orchestrator()` +- `live_participant_for_agent(agent_id, scope, role)` +- `invalidated_world_members()` +- `last_updated_at()` + +`warnings` is the explicit answer to torn writes. This record is internal only. No public JSON +surface changes are required. + +### Required store API + +```rust +impl AgentRuntimeStateStore { + pub(crate) fn load_session( + &self, + orchestration_session_id: &str, + ) -> Result>; + + pub(crate) fn list_sessions(&self) -> Result>; + + pub(crate) fn list_live_sessions(&self) -> Result>; + + pub(crate) fn resolve_single_live_session_for_agent( + &self, + orchestrator_agent_id: &str, + ) -> Result>; + + pub(crate) fn list_participants_across_sources( + &self, + ) -> Result>; + + pub(crate) fn list_invalidated_participants_across_sources( + &self, + ) -> Result>; + + pub(crate) fn invalidate_stale_world_members_for_session( + &self, + orchestration_session_id: &str, + active_generation: u64, + ) -> Result>; +} +``` + +Hard rules: + +1. `list_participants_across_sources()` is the one shared participant source walker. +2. `list_invalidated_participants_across_sources()` is the tombstone source for status suppression. +3. `invalidate_stale_world_members_for_session()` must stop assuming only flat `participants/*.json` exists once canonical roots are introduced. +4. `resolve_live_orchestrator_session()` may remain as a compatibility wrapper over `resolve_single_live_session_for_agent()` if that keeps the diff smaller. + +### Object-level precedence + +The precedence contract is per object, not per root: + +```text +Session object: + canonical sessions//session.json + > flat sessions/.json + +Participant object: + canonical sessions//participants/.json + > flat participants/.json + > legacy handles/.json +``` + +Missing canonical objects do not hide compatibility objects of the same class. That yields the +correct migration behavior: + +- missing canonical parent still allows flat parent fallback, +- missing canonical participant still allows flat or legacy participant fallback, +- canonical parent does not automatically erase flat participants, +- canonical participant does not depend on canonical parent already being present. + +### Torn-root read model + +The runtime already writes parent and participant objects separately. Reads must tolerate all four +states: + +```text +runtime writes parent and participant separately + │ + ├── session.json only ----------------> record + warning + ├── participant only -----------------> record + warning + ├── stale active handle --------------> warning, not whole-command failure + └── complete live pair ---------------> complete session record +``` + +`list_live_sessions()` only returns complete live records. `load_session()` and `list_sessions()` +may return incomplete records with warnings. + +### Session discovery rule + +`list_sessions()` must discover candidate session ids from the union of: + +1. canonical `sessions//` directories, +2. flat `sessions/.json` parent files, +3. merged participant records' `orchestration_session_id` values. + +This is what makes participant-only torn roots observable instead of invisible. + +### Path hardening + +Every canonical session root and nested file is treated as untrusted local input: + +- ignore symlinked session roots, +- ignore non-regular `session.json` entries, +- ignore non-regular participant files, +- ignore unexpected nested entry types, +- do not follow symlinks under canonical roots. + +This is a local trust-boundary hardening requirement, not optional cleanup. + +## File-by-File Contract + +### 1. `crates/shell/src/execution/agent_runtime/state_store.rs` + +This file owns the real change. + +Required changes: + +1. add helpers for canonical session roots, canonical participant paths, and canonical lease paths, +2. add one participant walker that merges canonical, flat-current, and legacy participant objects, +3. add parent-session loading that prefers canonical `session.json` over flat `sessions/.json`, +4. add `load_session()`, `list_sessions()`, `list_live_sessions()`, and `resolve_single_live_session_for_agent()`, +5. move invalidation and invalidated-participant reads onto the unified walker, +6. add torn-root warnings and incomplete-root handling, +7. add path hardening with `symlink_metadata`, +8. keep the implementation in this file unless a tiny internal helper extraction makes the diff materially clearer. + +Anti-goals: + +- no new crate, +- no new cache or index file, +- no parallel store implementation for canonical and compatibility reads, +- no caller-owned merging logic. + +### 2. `crates/shell/src/execution/agents_cmd.rs` + +This file owns the consumer cutover. + +Required changes: + +1. `build_status_report()` must build authoritative live rows from `list_live_sessions()`, +2. `build_status_report()` must preserve one row per live participant, not one row per `agent_id`, +3. tombstone suppression must use `list_invalidated_participants_across_sources()`, +4. trace fallback may only fill gaps not already covered by a live record or tombstone, +5. `build_toolbox_status_report()` and `build_toolbox_env_report()` must resolve one live session record via `resolve_single_live_session_for_agent()`, +6. `toolbox status` keeps the current JSON fields: + - `active_orchestration_session_id` + - `active_world_binding` + - `dependency_unavailable` +7. ambiguity remains fail closed and operator-readable. + +Important repo-truth note: + +`docs/USAGE.md` still says live session discovery is backed by `~/.substrate/run/agent-hub/handles/`. +That wording is stale today and must be corrected as part of this slice. + +### 3. `crates/shell/src/repl/async_repl.rs` + +This file owns writer cutover, not lifecycle redesign. + +Required changes: + +1. keep the current bootstrap and lifecycle ordering semantics, +2. once canonical pathing is added, move parent, participant, and lease writes onto store-owned canonical helpers, +3. keep any temporary dual-write inside the store only, +4. keep `persist_runtime_snapshots()` as the write choke point, +5. add crash-window tests for: + - parent written, participant missing, + - participant written, parent missing, + - active handle points at stale participant. + +This plan does not authorize inventing a new transactional layer. + +### 4. Tests, fixtures, and docs + +Primary touchset: + +- [crates/shell/tests/agent_successor_contract_ahcsitc0.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_successor_contract_ahcsitc0.rs) +- [crates/shell/tests/agent_hub_trace_persistence.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_hub_trace_persistence.rs) +- [crates/shell/tests/repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs) +- helper readers in [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) +- [docs/TRACE.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/TRACE.md) +- [docs/USAGE.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/USAGE.md) + +Required changes: + +1. fixture writers seed canonical session-root trees, +2. flat fixtures remain only where compatibility behavior is being tested explicitly, +3. helper readers stop assuming only `participants/*.json` exists, +4. docs stop claiming `handles/` or flat participant scans are the live authority, +5. test assertions stop depending on incidental pre-cutover path layout. + +## Execution Plan + +### Phase A: Store projection cutover over current roots + +Deliver the store contract before moving writers. + +Scope: + +- `AgentRuntimeSessionRecord` +- canonical path helpers +- merged participant walker +- parent-session loading with precedence +- `load_session()`, `list_sessions()`, `list_live_sessions()` +- torn-root warnings +- path hardening + +Exit criteria: + +1. unit tests prove canonical and compatibility reads merge correctly by object, +2. `list_sessions()` can discover participant-only torn roots, +3. `list_live_sessions()` excludes incomplete roots, +4. no production caller still needs to regroup flat participant lists into sessions. + +### Phase B: Consumer and invalidation cutover + +Move every reader that matters onto the new store contract. + +Scope: + +- `agent status` +- `toolbox status` +- `toolbox env` +- `list_invalidated_participants_across_sources()` +- `invalidate_stale_world_members_for_session()` over the shared participant walker + +Exit criteria: + +1. same-agent concurrent sessions remain distinct in `agent status`, +2. ambiguity in `toolbox status|env` fails closed using session-record resolution, +3. `PLAN-05` tombstone suppression still holds after regrouping, +4. no invalidation or tombstone logic still reads only flat `participants/*.json`. + +### Phase C: Canonical write cutover + +Only after Phase A and Phase B are green: + +- canonical `sessions//session.json` write +- canonical `sessions//participants/.json` write +- canonical `sessions//leases/.lease` write +- bounded dual-write only if needed, only inside the store + +Exit criteria: + +1. runtime writer still tolerates crash windows, +2. store reads remain correct when canonical and flat files coexist, +3. canonical write path is boring and centralized. + +### Phase D: Fixture, doc, and compatibility cleanup + +This is the consolidation pass: + +- fixture updates +- helper-reader updates +- doc corrections +- flat-write removal if the migration bridge is already proven safe +- legacy `handles/*.json` reads removed last, not first + +Exit criteria: + +1. docs reflect current truth, +2. fixtures default to canonical layout, +3. compatibility behavior is either removed or explicitly justified, +4. no stale path assumptions remain in tests. + +### Phase-order diagram + +```text +PHASE A +------- +store-owned session projection over current roots + │ + ├── load_session() + ├── list_sessions() + ├── list_live_sessions() + └── unified participant walker + +PHASE B +------- +consumer + invalidation cutover + │ + ├── status uses live session records + ├── toolbox resolves one session record + └── tombstones/invalidation use same source walker + +PHASE C +------- +canonical writer cutover + │ + ├── session.json write + ├── session-local participant write + └── session-local lease write + +PHASE D +------- +fixture/doc cleanup + compatibility retirement +``` + +## Architecture Diagrams + +### Read-side composition + +```text +load_session("sess_a") + │ + ├── session object + │ canonical sessions/sess_a/session.json + │ > flat sessions/sess_a.json + │ + └── participant objects + canonical sessions/sess_a/participants/ash_123.json + > flat participants/ash_123.json + > legacy handles/ash_123.json +``` + +### Consumer ownership + +```text +AgentRuntimeStateStore + │ + ├── list_live_sessions() -----------------> agent status + ├── resolve_single_live_session_for_agent() -> toolbox status/env + ├── list_invalidated_participants_across_sources() -> status suppression + └── invalidate_stale_world_members_for_session() -> PLAN-05 invalidation path +``` + +### Acceptance boundary + +```text +CURRENT +------- +callers regroup flat manifests + flat tombstone scans + +TARGET +------ +store groups by session +callers consume session records +canonical layout follows once reads are green +``` + +## Test Review + +### Runtime and test framework + +- Runtime: Rust +- Package: `shell` +- Test framework: `cargo test` with unit and integration suites under `crates/shell/tests/` + +### Code path coverage + +```text +CODE PATH COVERAGE +=========================== +[+] crates/shell/src/execution/agent_runtime/state_store.rs + │ + ├── list_participants_across_sources() + │ ├── [GAP] canonical + flat + legacy merge by participant_id + │ ├── [GAP] canonical participant beats flat participant + │ ├── [GAP] flat participant beats legacy handle + │ └── [GAP] symlinked or non-regular files are ignored + │ + ├── load_session() + │ ├── [GAP] canonical parent + canonical participants + │ ├── [GAP] flat parent + flat participants compatibility load + │ ├── [GAP] parent-only torn root returns warning, not failure + │ ├── [GAP] participant-only torn root returns warning, not failure + │ └── [GAP] stale active handle returns warning, not whole-command failure + │ + ├── list_live_sessions() + │ ├── [GAP] incomplete roots are skipped + │ ├── [GAP] two live sessions with same agent stay distinct + │ └── [GAP] invalidated tombstones stay attached to the right session + │ + ├── list_invalidated_participants_across_sources() + │ └── [GAP] canonical session-root tombstones are visible after flat writes stop + │ + └── invalidate_stale_world_members_for_session() + ├── [GAP] canonical session-root participants are invalidated + ├── [GAP] flat compatibility participants are still covered during migration + └── [GAP] idempotence survives the new source walker + +[+] crates/shell/src/execution/agents_cmd.rs + │ + ├── build_toolbox_status_report() + │ ├── [★★ TESTED] current ambiguity already fails closed + │ ├── [GAP] session-record resolution keeps the same behavior + │ └── [GAP] incomplete session records are not promoted live + │ + └── build_status_report() + ├── [GAP] live rows come from session records, not flat live manifests + ├── [GAP] same-agent concurrent sessions stay visible + └── [GAP] tombstone suppression still wins after regrouping + +[+] crates/shell/src/repl/async_repl.rs + │ + ├── bootstrap persistence + │ ├── [GAP] canonical session-root parent write + │ └── [GAP] parent-only crash window remains tolerated by reads + │ + └── lifecycle rewrites + ├── [GAP] canonical session-local participant write + ├── [GAP] participant-only crash window remains tolerated by reads + └── [GAP] invalidation helper still sees moved participants + +───────────────────────────────── +COVERAGE: 1/22 paths partly covered today (5%) +GAPS: 21 code paths need direct tests +───────────────────────────────── +``` + +### User flow coverage + +```text +USER FLOW COVERAGE +=========================== +[+] One live orchestrator session + │ + ├── [GAP] toolbox status resolves one live session record + └── [GAP] toolbox env exports the same endpoint + +[+] Two live orchestrator sessions for the same agent + │ + ├── [★★ TESTED] current ambiguity already fails closed + └── [GAP] session-record ambiguity path keeps the same fail-closed contract + +[+] Agent status with concurrent same-agent world sessions + │ + ├── [GAP] both live rows remain visible + └── [GAP] one tombstone suppresses only its own session + +[+] Upgrade / compatibility window + │ + ├── [GAP] canonical parent beats flat parent + ├── [GAP] canonical participant beats flat participant + ├── [GAP] flat participant beats legacy handle + └── [GAP] missing canonical participant does not hide flat compatibility participant + +[+] Runtime crash windows + │ + ├── [GAP] session.json only does not break status/toolbox + ├── [GAP] participant only does not break status/toolbox + └── [GAP] stale active handle degrades safely + +───────────────────────────────── +COVERAGE: 1/13 flows partly covered today (8%) +GAPS: 12 user flows need tests +───────────────────────────────── +``` + +### Required test additions by file + +#### `crates/shell/src/execution/agent_runtime/state_store.rs` + +Add unit coverage for: + +- object-level precedence across canonical, flat-current, and legacy participant files +- canonical parent vs flat parent precedence +- torn-root warnings: + - parent only + - participant only + - stale active handle +- unified invalidation walker across canonical and flat sources +- symlink and non-regular file rejection + +#### `crates/shell/tests/agent_successor_contract_ahcsitc0.rs` + +Add contract coverage for: + +- toolbox status and env resolving from session records +- ambiguity remaining fail closed with operator-readable text +- status showing two live rows for one `agent_id` across two session ids +- canonical participants overriding flat compatibility participants +- flat compatibility participants still appearing if canonical root is incomplete during migration + +#### `crates/shell/tests/agent_hub_trace_persistence.rs` + +Add integration coverage for: + +- canonical session-root persistence retaining the authoritative orchestration session id +- parent and participant rewrites remaining aligned across lifecycle changes +- incomplete roots degrading safely in reads + +#### `crates/shell/tests/repl_world_first_routing_v1.rs` + +Add integration coverage for: + +- `PLAN-05` invalidation still working after participant location changes +- tombstone suppression still working after flat writes stop +- cross-session same-agent world rows remaining isolated after regrouping + +#### `crates/shell/src/repl/async_repl.rs` + +Add or update helper coverage for: + +- helper readers that currently scan `participants/*.json` +- bootstrap and lifecycle crash windows under the new store contract + +#### Docs + +Update examples or wording in: + +- `docs/TRACE.md` +- `docs/USAGE.md` +- successor protocol draft as needed + +### Test commands + +Run at minimum: + +```bash +cargo test -p shell agent_runtime::state_store -- --nocapture +cargo test -p shell --test agent_successor_contract_ahcsitc0 -- --nocapture +cargo test -p shell --test agent_hub_trace_persistence -- --nocapture +cargo test -p shell --test repl_world_first_routing_v1 -- --nocapture +``` + +Then run: + +```bash +cargo test -p shell -- --nocapture +cargo clippy -p shell --all-targets -- -D warnings +``` + +### QA artifact + +Primary QA handoff artifact: +[spensermcconnell-feat-session-centric-state-store-eng-review-test-plan-20260430-162706.md](/Users/spensermcconnell/.gstack/projects/atomize-hq-substrate/spensermcconnell-feat-session-centric-state-store-eng-review-test-plan-20260430-162706.md) + +## Failure Modes Registry + +| New codepath | Real production failure | Test covers it? | Error handling exists? | User sees clear error? | Critical gap? | +| --- | --- | --- | --- | --- | --- | +| unified participant source walker | invalidation or tombstones stop seeing moved participants | planned | planned | no today | yes until fixed | +| torn-root tolerant reads | normal runtime transitions start failing status/toolbox | planned | planned | partial today | yes until fixed | +| object-level precedence | incomplete canonical roots hide still-valid compatibility objects | planned | planned | no today | yes until fixed | +| status projection cutover | same-agent concurrent sessions collapse back into one row | planned | no today | no today | yes until fixed | +| toolbox session resolution | CLI silently promotes incomplete or ambiguous session state | partly today | yes | yes | yes until session-record path is covered | +| nested directory traversal | symlinked session roots or files poison operator reads | planned | no today | no today | yes until hardened | + +Critical gap rule: + +If invalidation or tombstone suppression can miss canonical session-root participants, or if normal +parent/participant torn writes can now break `status` or `toolbox`, this slice is not done. + +## Performance Review + +This is a correctness slice, not a performance slice. + +Hard rules: + +1. no index file lands before measurement, +2. projection and consumer truth come before micro-optimization, +3. one top-level scan per command is acceptable at current scale, +4. if canonical scans later prove too slow, measure first and add an index later. + +The performance footgun is keeping the flat global scan forever because it already exists. That is +the smaller diff now and the more expensive platform forever. + +## Worktree Parallelization Strategy + +### Dependency table + +| Step | Modules touched | Depends on | +| --- | --- | --- | +| A. Session projection + unified source walker | `crates/shell/src/execution/agent_runtime/` | — | +| B. Status/toolbox consumer cutover | `crates/shell/src/execution/` | A | +| C. Canonical pathing + writer cutover | `crates/shell/src/execution/agent_runtime/`, `crates/shell/src/repl/` | A | +| D. Fixtures, helpers, docs, and compatibility cleanup | `crates/shell/tests/`, `docs/`, `llm-last-mile/` | B, C | + +### Parallel lanes + +- Lane A: Step A +- Lane B: Step B +- Lane C: Step C +- Lane D: Step D + +### Execution order + +1. launch Lane A first, +2. once Lane A lands, launch Lane B and Lane C in parallel worktrees, +3. run Lane D last after both contracts settle. + +### Conflict flags + +- Lanes B and C both depend on the final session-record API shape. +- Lanes A and C both touch `crates/shell/src/execution/agent_runtime/`, so they are not parallel. +- Lane D touches both fixture helpers and operator docs, so keep it last. + +### Parallelization verdict + +This plan has one foundation lane, two safe parallel implementation lanes, and one final +validation lane. + +## Deferred Work + +There is no `TODOS.md` in the repo root, so explicit deferrals stay here: + +1. explicit CLI session selector such as `--orchestration-session-id` + - fail-closed ambiguity is enough for this slice +2. removing flat compatibility reads entirely + - do it after real upgrade confidence exists +3. any measured index or cache file + - only after canonical scans prove too slow +4. broader session history or list-session productization + - this slice only builds the storage and projection boundary those features need + +## Definition of Done + +This slice is done only when all of the following are true: + +1. the store can load one orchestration session and list live sessions directly, +2. `agent status` consumes store-owned live session records, +3. `toolbox status|env` consumes store-owned live session resolution, +4. invalidation and invalidated-participant reads use the unified participant source walker, +5. `PLAN-05` tombstone invalidation and suppression semantics still pass after regrouping, +6. torn roots produced by normal runtime transitions degrade safely in reads, +7. canonical session-root layout is the write target by the end of the slice, +8. canonical-vs-flat precedence is resolved per object, not per root, +9. symlinked or non-regular canonical entries are ignored, +10. docs and fixtures stop describing flat handles or flat participant scans as the live authority. + +## Completion Summary + +- Step 0: scope accepted as-is after repo-state correction +- Architecture: store-owned session projection first, canonical pathing second +- Code shape: one grouping boundary, one participant walker, one precedence rule +- Tests: coverage diagrams produced, 33 concrete gaps or assertions identified +- Performance: no cache or index expansion justified +- NOT in scope: written +- What already exists: written +- Failure modes: 6 critical gaps remain until projection, precedence, and torn-write handling land +- Parallelization: 4 steps, 2 safe parallel implementation lanes +- Lake Score: complete option chosen for every in-slice decision + +## Decision Audit Trail + +| # | Phase | Decision | Classification | Principle | Rationale | Rejected | +| --- | --- | --- | --- | --- | --- | --- | +| 1 | Scope | Reframe the slice around session addressability, not directory shape | Mechanical | User outcome first | That is the real operator problem still left in the repo | Leading with layout as the problem statement | +| 2 | Architecture | Choose projection first, then layout migration | Mechanical | Pragmatic | Solves truth and ambiguity first, migrates disk locality second | Layout-first cutover | +| 3 | Compatibility | Merge by object, not by root | Mechanical | Explicit over clever | Parent precedence and participant precedence are different contracts | Root-level winner-take-all | +| 4 | Correctness | Move every enumerator and mutator onto one unified participant source walker | Mechanical | DRY | `PLAN-05` semantics cannot live on a different read path | Migrating only `status` and `toolbox` | +| 5 | Read contract | Model torn writes with warnings and incomplete-root handling | Mechanical | Systems over heroes | Parent and participant rewrites are already separate in the runtime | Pretending whole-session atomicity exists | +| 6 | Security | Reject symlinked or non-regular canonical entries | Mechanical | Boring by default | Nested directory traversal widens the local trust boundary | Trusting nested paths blindly | + +## GSTACK REVIEW REPORT + +| Review | Trigger | Why | Runs | Status | Findings | +| --- | --- | --- | --- | --- | --- | +| CEO Review | `/plan-ceo-review` | Scope and strategy | 1 | CLEAR | Reframed the slice around operator session addressability, added the hybrid projection-first alternative, and demoted layout migration from the problem statement to an enabling step | +| Codex Review | `/codex review` | Independent 2nd opinion | 0 | SKIPPED | No separate Codex review run | +| Eng Review | `/plan-eng-review` | Architecture and tests (required) | 1 | CLEAR | Added unified participant source walking, torn-root tolerance, object-level precedence, symlink hardening, and the broader regression matrix needed to keep `PLAN-05` semantics intact | +| Design Review | `/plan-design-review` | UI/UX gaps | 0 | SKIPPED | No UI scope | + +**CROSS-MODEL:** The review passes converged on the same correction. The plan only becomes sound when it is projection-first, consumer-first, and explicit about torn writes and compatibility precedence. + +**UNRESOLVED:** 0 blocking design decisions remain inside slice `06`. The main deliberate deferral is explicit selector UX for ambiguous multi-session operator workflows. + +**VERDICT:** CEO + ENG CLEARED. `PLAN-06` is ready to implement after `PLAN-05`. diff --git a/llm-last-mile/PLAN-07.md b/llm-last-mile/PLAN-07.md new file mode 100644 index 000000000..879b78cef --- /dev/null +++ b/llm-last-mile/PLAN-07.md @@ -0,0 +1,681 @@ +# PLAN-07: Shared-World Replacement Ordering, Rollback, and Atomic Metadata + +Source file: [07-world-replacement-ordering-rollback-atomic-metadata.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/07-world-replacement-ordering-rollback-atomic-metadata.md) +Branch: `feat/session-centric-state-store` +Plan type: Linux backend hardening and proof-preservation slice, no UI scope +Review posture: `/autoplan`-style scope tightening with `/plan-eng-review` structure and rigor +Status: execution-ready as a repo-truth and regression-proof contract + +## Objective + +This is not a greenfield design doc. The Linux backend already contains the core replacement +mechanics this slice needs. The job here is to turn that existing behavior into one explicit, +regression-proof contract that later slices can consume without reinterpreting it. + +`PLAN-07` freezes the backend contract that `PLAN-04`, `PLAN-05`, and `PLAN-06` depend on: + +- replacement ordering must never leave an orchestration session without one recoverable active + world, +- `session.json` writes must remain atomic for every shared-world transition, +- recovery must stay deterministic and fail closed on ambiguity, +- downstream proof consumers must continue to accept only `binding_state=Active`. + +This is a narrow slice. It is also a hard one. Everything after slice `03` assumes the backend can +answer one question honestly: "what is the one reusable active world for this orchestration session +right now?" If restart can lie about that, the rest of the stack becomes a more sophisticated way +to persist corruption. + +## Step 0: Scope Challenge + +### 0A. Repo truth and why this slice exists + +`PLAN-03` established explicit shared-world ownership. `PLAN-04`, `PLAN-05`, and `PLAN-06` all +consume that ownership. + +What still needs to be locked down is the restart seam inside [crates/world/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world/src/lib.rs) and [crates/world/src/session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world/src/session.rs). That seam decides whether a restart produces: + +- one durable active world, +- two ambiguous candidates, +- or zero valid worlds because the old one was moved aside before the new one was real. + +That is the whole game. + +### 0B. Existing code to reuse + +| Sub-problem | Existing code | Plan | +| --- | --- | --- | +| Shared-owner replacement entrypoint | [crates/world/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world/src/lib.rs) | Reuse | +| Pre-commit, rollback, and finalize state transitions | [SessionWorld::set_shared_binding_state(...)](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world/src/session.rs) | Reuse | +| Shared-world recovery from persisted metadata | [SessionWorld::recover_shared_active_from_root(...)](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world/src/session.rs) | Reuse | +| Atomic `session.json` write path | [SessionWorld::persist_metadata(...)](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world/src/session.rs) | Reuse | +| Downstream proof validation | [resolve_shared_world_binding(...)](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs) | Reuse | +| PTY and non-PTY shared-world transport | [crates/world-agent/src/service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs) and [crates/world-agent/src/pty.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/pty.rs) | Reuse | +| Existing regression tests for success, rollback, and recovery | [crates/world/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world/src/lib.rs) and [crates/world/src/session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world/src/session.rs) | Extend | + +### 0C. Minimum viable slice + +Do this, and only this: + +- preserve the current `Active -> Replacing -> Replaced` transaction shape, +- preserve rollback to `Active` on replacement-create failure, +- preserve atomic `session.json` writes for creation, replacement, rollback, and recovery repair, +- widen the regression net around finalize-warning, ambiguity, malformed proof, and temp-file + cleanup behavior, +- keep world-agent proof consumers fail closed. + +Do not: + +- redesign runtime-state projection, +- move ownership authority into shell state, +- introduce a new registry, index, or cache file, +- change selected-orchestrator UX, +- expand cross-platform shared-world parity in this slice. + +### 0D. Complexity check + +This slice is intentionally smaller than `PLAN-04` through `PLAN-06`. + +The honest production seam is four files: + +1. [crates/world/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world/src/lib.rs) +2. [crates/world/src/session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world/src/session.rs) +3. [crates/world-agent/src/service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs) +4. [crates/world-agent/src/pty.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/pty.rs) + +That is small enough to keep boring. Good. Boring is what you want in restart correctness. + +### 0E. Search and built-in check + +`[Layer 1]` wins here. + +The repo already has the right primitives: + +- explicit shared-owner proof types, +- explicit `Replacing` and `Replaced` states, +- one backend-local mutex for shared-owner replacement ordering, +- atomic rename-based metadata persistence, +- fail-closed proof validation at the agent boundary. + +The correct move is to preserve and tighten that contract, not invent a second authority or a more +abstract restart engine. + +### 0F. What already exists + +- the backend-local shared-owner mutex already serializes replacement entry, +- replacement already has a pre-commit `Replacing` state instead of pretending create/finalize is + one atomic in-memory action, +- recovery already distinguishes `Active`, `Replacing`, `Replaced`, and malformed metadata, +- proof validation already rejects mismatched ownership and non-Active state in + `resolve_shared_world_binding(...)`, +- repo tests already cover the happy-path replacement and rollback shape. + +That means the plan is not "invent the mechanism." It is "freeze the mechanism, tighten the +failure rules, and close the remaining proof gaps." + +### 0G. NOT in scope + +- runtime-state projection work already owned by [04-thread-world-binding-into-runtime-state.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/04-thread-world-binding-into-runtime-state.md) +- generation invalidation and member replacement semantics already owned by [05-restart-invalidation-semantics.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/05-restart-invalidation-semantics.md) +- session-centric registry reshaping already owned by [06-session-centric-state-store.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/06-session-centric-state-store.md) +- non-Linux shared-world replacement parity +- UI, CLI selector, or status-schema redesign + +## Architecture Contract + +### No-ambiguity rules + +1. Only `binding_state=Active` is reusable downstream. +2. `Replacing` is an internal pre-commit state, not a public steady state. +3. The old world may move from `Active` to `Replacing` before the new world exists, but only if + rollback to `Active` is still possible. +4. Replacement creation success must commit a new world with `world_generation = old + 1`. +5. Replacement creation failure must restore the old world to `Active` before returning failure. +6. Old-world finalization failure after replacement commit is a warning path, not a rollback path. +7. Recovery must prefer one committed `Active` replacement over an older `Replacing` predecessor. +8. Recovery must fail closed on multiple `Active` candidates or multiple unreconciled + `Replacing` candidates. +9. Malformed, partial, cross-owned, or ownerless shared metadata is never adopted for + shared-owner reuse. +10. Atomic metadata persistence applies to initial create, pre-commit, rollback, finalize, and + recovery-driven repair writes. + +### Replacement transaction diagram + +```text +shared owner replacement for orchestration_session_id = S + +current durable state + │ + └── old world = Active(g) + │ + ├── pre-commit metadata write + ▼ + old world = Replacing(g) + │ + ├── create replacement world root + metadata + │ │ + │ ├── success + │ ▼ + │ new world = Active(g+1) + │ │ + │ ├── finalize old world metadata + │ ▼ + │ old world = Replaced(g) + │ + └── failure + │ + ├── rollback old world metadata to Active(g) + ├── remove partial replacement root + ▼ + return failure with create/rollback/cleanup detail +``` + +Unsafe window allowed: + +- old world temporarily marked `Replacing`, but still recoverable + +Unsafe window forbidden: + +- old world no longer reusable, new world not durably committed, and recovery has no valid active + candidate + +### Atomic metadata write contract + +`SessionWorld::persist_metadata()` must keep this exact posture: + +```text +serialize metadata + │ + ├── write temp file inside world directory + ├── sync temp file + ├── rename temp -> session.json + ├── best-effort sync containing directory + └── remove temp file on failure paths +``` + +Why this matters: + +- restart state transitions are metadata-driven, +- recovery trusts `session.json`, +- a torn write in `session.json` is not "just a doc corruption issue," it is an authority + corruption issue. + +### Recovery precedence + +```text +recovery scan for one owner S + │ + ├── Active worlds + │ ├── 0 -> keep scanning + │ ├── 1 -> candidate winner unless a same/newer Replacing makes state ambiguous + │ └── >1 -> fail closed + │ + └── Replacing worlds + ├── 0 -> no recovery candidate + ├── 1 with no Active -> repair back to Active and return it + └── >1 -> fail closed +``` + +Hard rule: + +- `Replaced` and `Abandoned` are never reusable + +### Downstream proof boundary + +```text +crates/world + │ returns WorldHandle { id, shared_binding } + ▼ +crates/world-agent + │ resolve_shared_world_binding(...) + │ + ├── reject empty orchestration_session_id + ├── reject empty world_id + ├── reject world_id mismatch + ├── reject orchestration_session_id mismatch + └── reject binding_state != Active + ▼ +shell/runtime consumers + │ + └── trust only authoritative Active proof +``` + +This slice does not let later layers infer or "heal" backend proof mistakes. The backend contract +must already be correct when it crosses this boundary. + +## Concrete File Touch Plan + +### 1. Replacement orchestration in `crates/world/src/lib.rs` + +Primary seams: + +- [replace_shared_owner_session_from_root(...)](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world/src/lib.rs) +- [replace_shared_owner_session_from_root_with_creator(...)](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world/src/lib.rs) +- [cleanup_partial_shared_world_root(...)](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world/src/lib.rs) +- [ensure_shared_owner_session_from_root(...)](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world/src/lib.rs) + +Required behavior: + +- resolve one active shared-owner world, +- enforce `expected_generation`, +- pre-commit the old world to `Replacing`, +- create the replacement world with `generation + 1`, +- on create failure, roll back the old world and clean the partial root, +- on finalize-old-world failure, keep the replacement active and log a warning. + +Must not do: + +- bypass the shared-owner mutex, +- silently ignore generation mismatch, +- delete the old world on failure, +- fall back to generic compatible reuse for explicit shared-owner replacement. + +### 2. Metadata persistence and recovery in `crates/world/src/session.rs` + +Primary seams: + +- [persist_metadata(...)](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world/src/session.rs) +- [set_shared_binding_state(...)](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world/src/session.rs) +- [recover_shared_active_from_root(...)](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world/src/session.rs) + +Required behavior: + +- permit only `Active -> Replacing -> Active|Replaced`, +- persist every transition atomically, +- repair a lone `Replacing` world back to `Active`, +- ignore malformed metadata without deleting it, +- fail closed on multiple `Active` or multiple unreconciled `Replacing` candidates. + +Must not do: + +- add new binding states for this slice, +- silently auto-choose between two active candidates, +- treat ownerless legacy metadata as shared-owner compatible. + +### 3. Proof validation in `crates/world-agent` + +Primary seams: + +- [resolve_shared_world_binding(...)](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs) +- PTY startup path in [crates/world-agent/src/pty.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/pty.rs) + +Required behavior: + +- use the same helper for PTY and non-PTY shared-world proof validation, +- reject non-active or malformed proof snapshots, +- keep proof validation local and explicit instead of letting shell callers paper over a bad + binding. + +### 4. Tests and docs + +Primary test anchors: + +- [crates/world/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world/src/lib.rs) +- [crates/world/src/session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world/src/session.rs) +- [crates/world-agent/src/service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs) +- [crates/world-agent/tests/repl_persistent_session_bootstrap_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/tests/repl_persistent_session_bootstrap_v1.rs) + +This slice does not need a doc rewrite campaign. It does need the tests to prove the contract the +docs already claim. + +## Execution Plan + +### Ordered implementation sequence + +Implement in this order. Each phase creates one invariant the next phase can safely consume. + +1. **Freeze replacement ordering in `crates/world/src/lib.rs`** + - keep `expected_generation` enforcement hard, + - keep pre-commit `Active -> Replacing`, + - keep rollback-to-`Active` on create failure, + - preserve aggregated create/rollback/cleanup error reporting, + - preserve finalize-old-world failure as warning-only after replacement commit. + +2. **Freeze metadata authority in `crates/world/src/session.rs`** + - allow only `Active -> Replacing -> Active|Replaced`, + - keep `persist_metadata()` rename-based and temp-file-clean, + - keep recovery deterministic across `Active`, `Replacing`, `Replaced`, and malformed inputs, + - prove fail-closed ambiguity handling with targeted tests. + +3. **Freeze the proof boundary in `crates/world-agent`** + - keep `resolve_shared_world_binding(...)` as the single validation helper, + - ensure PTY and non-PTY paths both route through it, + - reject empty, mismatched, or non-Active proof snapshots before shell-visible success. + +4. **Widen regression coverage before calling the slice done** + - land the missing world/backend tests first, + - then land proof-boundary tests, + - then refresh the QA handoff artifact and plan closeout text. + +### Phase-by-phase acceptance gates + +| Phase | Acceptance gate | +| --- | --- | +| 1. Replacement ordering | no create-failure path can leave the owner without one recoverable `Active` world | +| 2. Metadata authority | every shared-world state write is atomic and recovery never silently chooses between ambiguous candidates | +| 3. Proof boundary | PTY and non-PTY consumers reject malformed or non-Active proof before reporting success | +| 4. Regression closeout | targeted tests cover success, rollback, ambiguity, malformed proof, and temp-file hygiene | + +## Architecture Review + +### Locked architecture decisions + +1. **Keep metadata authority in the backend.** + - `session.json` remains the recovery authority. Shell/runtime consumers read proof, they do not invent it. + +2. **Keep `Replacing` as the only publicized pre-commit checkpoint.** + - It is the durable rollback seam. Removing it would make failure recovery more magical, not simpler. + +3. **Keep replacement commit asymmetric.** + - Create failure rolls back. Old-world finalize failure after replacement commit does not. Once the new `Active` world exists durably, that becomes the truth. + +4. **Keep fail-closed recovery on ambiguity.** + - Two `Active` candidates or two unreconciled `Replacing` candidates are contract failures, not "pick the newer one" opportunities. + +5. **Keep proof validation local and explicit.** + - `resolve_shared_world_binding(...)` stays the one boundary. PTY and non-PTY codepaths must not fork their own interpretation logic. + +### Architecture acceptance gates + +1. **Restart gate** + - `ReplaceExpectedGeneration` either returns one new `Active` world or returns failure with the old world still recoverable. + +2. **Recovery gate** + - recovery returns one authoritative world, repairs one lone `Replacing` world, or fails closed. Nothing in between. + +3. **Boundary gate** + - world-agent success paths never expose `Replacing`, `Replaced`, empty ids, or mismatched owner proof as reusable. + +## Code Quality Review + +### Implementation guardrails + +1. one replacement transaction shape in `crates/world/src/lib.rs`, not separate success and failure semantics hidden across callsites, +2. one binding-state transition helper in `SessionWorld`, not ad hoc metadata mutation, +3. one proof-validation helper in `crates/world-agent`, not PTY-specific and service-specific drift, +4. no new binding states, cache files, registry files, or shell-side authority mirrors, +5. explicit aggregated errors on the bad restart path, because debugging restart corruption from partial logs is miserable. + +### Minimal-diff rules + +- keep the diff inside the four production files already identified unless a test helper extraction makes the test seams materially clearer, +- prefer extending existing tests in `crates/world` and `crates/world-agent` over creating a new harness, +- keep docs and plan updates bounded to contract wording, not a broader narrative rewrite. + +## Error & Rescue Registry + +| Failure point | What goes wrong | Required rescue | +| --- | --- | --- | +| replacement create fails after pre-commit | old world is stuck `Replacing` and no active world remains | roll old world back to `Active`, clean partial root, return one aggregated error | +| finalize-old-world write fails after replacement commit | operator assumes replacement failed and retries incorrectly | keep replacement authoritative, log warning, recover new `Active` world | +| multiple active shared worlds exist on disk | reuse becomes nondeterministic | fail closed with explicit ambiguity error | +| malformed or partial owner metadata is reused | a corrupted or cross-owned world becomes falsely authoritative | ignore malformed metadata and keep scanning | +| downstream proof accepts `Replacing` | shell/runtime consumes a pre-commit state as reusable | reject proof unless `binding_state=Active` | +| temp file survives or old bytes are partially replaced | restart state authority becomes vulnerable to torn metadata | keep rename-based atomicity and assert cleanup paths in tests | + +## Test Review + +One nice thing here: the repo already has real tests. This review is not inventing coverage from +scratch. It is checking the last few holes that would still hurt at 2am. + +### Code path coverage + +```text +CODE PATH COVERAGE +=========================== +[+] crates/world/src/lib.rs + │ + ├── [★★★ TESTED] replace success commits new Active and old Replaced + ├── [★★★ TESTED] create failure rolls back old world and cleans partial root + ├── [GAP] generation conflict returns fail-closed error + ├── [GAP] finalize-old-world warning path leaves replacement authoritative + └── [GAP] aggregated create+rollback+cleanup error text is preserved + +[+] crates/world/src/session.rs + │ + ├── [★★★ TESTED] allowed binding-state transitions persist + ├── [★★★ TESTED] lone Replacing world repairs back to Active + ├── [★★★ TESTED] newer Active outranks older Replacing + ├── [★★★ TESTED] ownerless, cross-owned, and partial metadata are rejected + ├── [★★★ TESTED] atomic persist failure preserves prior bytes + ├── [GAP] multiple Active candidates fail closed + └── [GAP] happy-path atomic persist leaves no stray temp files + +[+] crates/world-agent/src/service.rs + │ + ├── [★★★ TESTED] mismatched orchestration_session_id is rejected + ├── [GAP] non-Active binding_state is rejected explicitly + └── [GAP] empty world_id / empty orchestration_session_id are rejected explicitly + +───────────────────────────────── +COVERAGE: 8/15 paths tested (53%) +QUALITY: ★★★: 8 ★★: 0 ★: 0 +GAPS: 7 paths need regression tests +───────────────────────────────── +``` + +### User flow coverage + +```text +USER FLOW COVERAGE +=========================== +[+] ReplaceExpectedGeneration success + │ + ├── [★★★ TESTED] new generation becomes the only reusable Active world + └── [GAP] finalize warning path still returns success while old world finalize logs only + +[+] ReplaceExpectedGeneration create failure + │ + ├── [★★★ TESTED] old world returns to Active + └── [★★★ TESTED] partial replacement root is deleted + +[+] Crash or restart recovery + │ + ├── [★★★ TESTED] lone Replacing world repairs back to Active + ├── [★★★ TESTED] newer Active beats older Replacing + └── [GAP] multiple Active candidates produce hard failure, not silent winner selection + +[+] Agent boundary proof validation + │ + ├── [★★★ TESTED] owner mismatch is rejected + ├── [GAP] [→E2E] PTY and non-PTY both reject non-Active proof snapshots + └── [GAP] empty proof fields are rejected before shell consumers see them + +[+] Metadata durability + │ + ├── [★★★ TESTED] write failure preserves prior metadata bytes + └── [GAP] successful writes leave no stray temp files behind + +───────────────────────────────── +COVERAGE: 6/11 flows tested (55%) +GAPS: 5 flows need tests (1 should exercise the PTY/shared-world path) +───────────────────────────────── +``` + +### Required test additions by file + +#### `crates/world/src/lib.rs` + +Add regression coverage for: + +- generation-mismatch failure on `ReplaceExpectedGeneration`, +- finalize-old-world failure after replacement commit still recovering the replacement, +- create failure that also captures rollback and cleanup failure text in one returned error. + +#### `crates/world/src/session.rs` + +Add regression coverage for: + +- multiple `Active` shared-owner candidates fail closed, +- successful `persist_metadata()` leaves no `..*.tmp` files in the world directory. + +#### `crates/world-agent/src/service.rs` + +Add unit coverage for: + +- `binding_state=Replacing` and `binding_state=Replaced` rejection, +- empty `world_id`, +- empty `orchestration_session_id`. + +#### `crates/world-agent/tests/repl_persistent_session_bootstrap_v1.rs` + +Add one integration-style proof that the PTY startup path rejects a non-Active shared-world proof +before the ready state becomes visible to shell consumers. + +### Test commands + +Run at minimum: + +```bash +cargo test -p world replace_success_commits_new_active_and_finalizes_old_world -- --nocapture +cargo test -p world replace_failure_rolls_back_old_world_and_cleans_partial_root -- --nocapture +cargo test -p world -- --nocapture +cargo test -p world-agent resolve_shared_world_binding_rejects_mismatched_owner_proof -- --nocapture +cargo test -p world-agent -- --nocapture +``` + +Then run: + +```bash +cargo test -p world-api shared_world_contract_round_trips_with_canonical_shape -- --nocapture +cargo test -p agent-api-types execute_response_shared_world_round_trip -- --nocapture +``` + +### QA artifact + +Primary QA handoff artifact: +[spensermcconnell-feat-session-centric-state-store-eng-review-test-plan-20260501-124044.md](/Users/spensermcconnell/.gstack/projects/atomize-hq-substrate/spensermcconnell-feat-session-centric-state-store-eng-review-test-plan-20260501-124044.md) + +## Failure Modes Registry + +| New codepath | Real production failure | Test covers it? | Error handling exists? | User sees clear error? | Critical gap? | +| --- | --- | --- | --- | --- | --- | +| replacement finalize warning path | replacement commits, old-world finalize write fails, operator misreads warning as rollback need | no | yes | partial today | yes until tested | +| ambiguous multi-active recovery | two active worlds exist for one owner and reuse becomes nondeterministic | no | yes | yes | yes until tested | +| non-Active proof rejection at PTY boundary | shell consumes a `Replacing` proof as reusable | no | yes | yes | yes until tested | +| empty proof fields | malformed proof crosses world-agent boundary and poisons runtime state | no | yes | yes | yes until tested | +| happy-path temp cleanup | temp metadata files accumulate and confuse debugging or future recovery tooling | no | partial | no | no, but still worth locking down | +| create+rollback+cleanup failure aggregation | root cause detail is lost during a bad restart window | no | partial | yes | no, but debugging gets worse | + +Critical gap rule: + +If a non-Active proof can cross the world-agent boundary, or if ambiguity between two active +shared worlds is not fail closed, this slice is not done. + +## Performance Review + +This is a correctness slice, not a throughput slice. + +Still, four performance rules matter: + +1. the shared-owner mutex is acceptable because replacement is a restart-boundary operation, not a + per-command hot path, +2. atomic rename plus directory sync is the right cost to pay for authority writes, +3. recovery may scan world roots because correctness beats speculative indexing here, +4. adding a cache or index file would spend an innovation token on the wrong problem. + +The performance footgun would be "optimize" away the safety barriers because restart is rare. Rare +code is exactly where silent corruption likes to hide. + +## Worktree Parallelization Strategy + +### Dependency table + +| Step | Modules touched | Depends on | +| --- | --- | --- | +| A. Backend replacement and recovery hardening | `crates/world/src/` | — | +| B. Downstream proof validation expansion | `crates/world-agent/src/`, `crates/world-agent/tests/` | A | +| C. QA artifact and plan/docs closeout | `llm-last-mile/`, `~/.gstack/projects/` | A, B | + +### Parallel lanes + +- Lane A: step A +- Lane B: step B +- Lane C: step C + +### Execution order + +1. land Lane A first because it freezes the authority contract, +2. then run Lane B once the backend proof shape is stable, +3. finish with Lane C after tests and contract wording settle. + +### Conflict flags + +- `crates/world/src/lib.rs` and `crates/world/src/session.rs` are one coupled seam. Keep them in + the same lane. +- Lane B depends on the exact backend proof contract from Lane A. Do not start PTY proof tests + early. + +### Parallelization verdict + +This slice is mostly sequential. There is **one foundation lane**, **one late proof-validation +lane**, and **one closeout lane**. + +## Deferred Work + +There is no `TODOS.md` in the repo root, so explicit deferrals stay here: + +1. runtime-state projection of authoritative world binding + - owned by [PLAN-04](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-04.md) +2. participant invalidation and replacement-member semantics + - owned by [PLAN-05](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-05.md) +3. session-centric runtime-store regrouping + - owned by [PLAN-06](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-06.md) +4. non-Linux explicit shared-world replacement parity + - still intentionally fail closed outside Linux +5. richer operator-facing restart diagnostics + - useful later, not required for the backend authority contract itself + +## Definition of Done + +This slice is done only when all of the following are true: + +1. replacement create failure never leaves the owner without one recoverable active world, +2. old-world rollback to `Active` is durable and recoverable, +3. successful replacement commits a new `Active` world with `generation + 1`, +4. partial replacement roots are cleaned on failure, +5. atomic metadata writes protect every shared-world state transition, +6. recovery chooses one authoritative result or fails closed on ambiguity, +7. ownerless, cross-owned, and partial metadata are never adopted for shared-owner reuse, +8. world-agent proof consumers accept only `binding_state=Active`, +9. regression tests cover success, rollback, cleanup, ambiguity, malformed proof, and metadata + temp-file hygiene, +10. later slices can consume this backend contract without re-inventing authority in shell state. + +## Completion Summary + +- Step 0: scope accepted as-is after repo-truth correction +- Architecture Review: 2 issues found, both are proof gaps rather than structural redesign needs +- Code Quality Review: 1 issue found, happy-path temp-file hygiene lacks explicit regression proof +- Test Review: diagrams produced, 12 concrete gaps or assertions identified +- Performance Review: 0 issues found +- NOT in scope: written +- What already exists: written +- TODOS.md updates: 0, repo has no `TODOS.md`, deferrals captured here +- Failure modes: 4 critical gaps flagged +- Outside voice: skipped +- Parallelization: 3 lanes, 0 safe production-code parallel lanes before contract freeze +- Lake Score: complete option chosen for every in-slice decision + +## Decision Audit Trail + +| # | Phase | Decision | Classification | Principle | Rationale | Rejected | +| --- | --- | --- | --- | --- | --- | --- | +| 1 | Scope | Treat this as a repo-truth and regression-proof slice, not a greenfield redesign | Mechanical | Pragmatic | The core backend behavior already exists and should be tightened, not reimagined | Rewriting the restart architecture | +| 2 | Authority | Keep Linux `session.json` as the replacement authority seam | Mechanical | Explicit over clever | Recovery already trusts this file and later layers already depend on its truth | Moving authority into shell-local reconstruction | +| 3 | Safety | Preserve `Replacing` as the only allowed pre-commit state | Mechanical | Completeness | It gives rollback a durable checkpoint without advertising a reusable public state | Removing `Replacing` and hoping create/finalize is atomic enough | +| 4 | Proof boundary | Require world-agent to reject any non-Active proof | Mechanical | Systems over heroes | Later shell layers should not need to compensate for backend or transport mistakes | Letting shell consumers infer reusability from partial proof | +| 5 | Tests | Spend effort on finalize-warning, ambiguity, malformed proof, and temp-file hygiene | Mechanical | Boil the lake | These are the remaining holes most likely to bite during follow-on work | Stopping at the currently landed happy-path tests | + +## GSTACK REVIEW REPORT + +| Review | Trigger | Why | Runs | Status | Findings | +| --- | --- | --- | --- | --- | --- | +| CEO Review | `/plan-ceo-review` | Scope and strategy | 0 | SKIPPED | Backend correctness slice, no separate CEO pass run | +| Codex Review | `/codex review` | Independent 2nd opinion | 0 | SKIPPED | No separate Codex review run | +| Eng Review | `/plan-eng-review` | Architecture and tests (required) | 1 | CLEAR | Reframed this as a repo-truth contract, preserved the current replacement state machine, and identified the remaining regression-proof gaps around finalize warning, ambiguity, malformed proof, and temp-file hygiene | +| Design Review | `/plan-design-review` | UI/UX gaps | 0 | SKIPPED | No UI scope | + +**UNRESOLVED:** 0 blocking design decisions remain inside slice `07`. The remaining work is proof widening, not scope confusion. + +**VERDICT:** ENG CLEARED. `PLAN-07` is ready to execute as the backend correctness guardrail that `PLAN-04`, `PLAN-05`, and `PLAN-06` consume. diff --git a/llm-last-mile/PLAN-08.md b/llm-last-mile/PLAN-08.md new file mode 100644 index 000000000..c72f9380d --- /dev/null +++ b/llm-last-mile/PLAN-08.md @@ -0,0 +1,922 @@ + + +# PLAN-08: Explicit Orchestration Authority for Event Emission + +Source file: [08-explicit-orchestration-authority-event-emission.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/08-explicit-orchestration-authority-event-emission.md) +Branch: `feat/session-centric-state-store` +Plan type: shell/runtime authority cleanup slice, no UI scope +Review posture: `/autoplan`-style scope tightening with `/plan-eng-review` structure and rigor +Status: execution-ready after [PLAN-07](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-07.md), before [09-live-state-authority-and-compatibility-cutover.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/09-live-state-authority-and-compatibility-cutover.md) + +## Objective + +This slice is not about making more events. It is about making existing shell-owned event +emission honest. + +The repo already has the right authority boundary: + +- runtime-owned orchestration identity in + [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) + via `RuntimeOrchestrationContext` +- canonical session-root state in + [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) +- explicit trace contract in + [docs/TRACE.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/TRACE.md) + that says runtime-owned producers must emit a real `orchestration_session_id` or suppress + the row + +What is still wrong is the shell-owned emitter seam. Production code still does live PID-based +recovery on event emission paths in: + +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) +- [crates/shell/src/execution/routing/dispatch/exec.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/exec.rs) +- [crates/shell/src/execution/routing/dispatch/world_ops.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs) + +That is the wrong dependency direction. The shell already owns orchestration truth at command +launch time. It should pass that truth into emitters directly, not rediscover it later from +mutable runtime state and a shell PID. + +`PLAN-08` fixes that in one bounded slice: + +1. introduce one shell-local explicit event-emission context type, +2. thread that context from REPL/runtime-owned authority into host and world execution helpers, +3. require caller-owned `cmd_id` / `run_id` / `span_id` correlation for orchestration-scoped + shell rows, +4. suppress orchestration-scoped `agent_event` rows when context is absent instead of guessing, +5. keep stdout/stderr and ordinary trace spans working even when orchestration-scoped rows are + suppressed. + +This is the whole game. If a shell-owned emitter can still say "I will figure out who I belong to +later," the live-state and trace contracts built in `PLAN-04`, `PLAN-06`, and `PLAN-07` stay +soft. + +## Step 0: Scope Challenge + +### 0A. Repo truth and why this slice exists + +The SOW is directionally correct, and current repo truth backs it up. + +What the code already proves: + +1. `RuntimeOrchestrationContext` already exists in + [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) + and already holds authoritative orchestration session state. +2. `translate_wrapper_event(...)` and `build_runtime_message_event(...)` already emit fully + explicit event rows from manifest plus orchestration-session authority. +3. world-restart alert helpers already fail closed when orchestration context is absent. +4. `docs/TRACE.md` already says runtime-owned producers must emit a real + `orchestration_session_id` or suppress the row. + +What is still broken: + +1. `resolve_active_orchestration_session_id()` exists in both + [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) + and + [dispatch/exec.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/exec.rs), + and a third copy lives in + [dispatch/world_ops.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs). +2. those helpers call + `find_active_orchestration_session_for_pid(std::process::id())` at emit time. +3. `stream_non_pty_via_agent(...)` still falls back to `run_id = parent_cmd_id.unwrap_or("unknown")`. +4. `spawn_host_stream_thread(...)` still emits orchestration-scoped rows with `run_id = "unknown"`. + +That means this slice is not greenfield. It is contract repair. + +### 0B. Existing code to reuse + +| Sub-problem | Existing code | Plan | +| --- | --- | --- | +| Runtime-owned orchestration authority | [RuntimeOrchestrationContext in async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) | Reuse | +| Participant lineage and backend identity | [AgentRuntimeSessionManifest](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs) plus event builders in [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) | Reuse | +| Canonical session-root state authority | [AgentRuntimeStateStore](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) | Reuse, but stop consulting it on emit paths | +| Shell event channel and suppression posture | [publish_agent_event(...)](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_events.rs) and current `publish_command_completion(None, ...)` behavior | Reuse | +| Stream chunk terminal passthrough | [emit_stream_chunk(...)](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs) | Reuse stdout/stderr behavior, tighten event gating | +| Existing restart-alert explicit-context model | [build_world_restart_required_alert(...)](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) and [emit_world_restarted_alert(...)](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) | Reuse | +| Existing regression anchors | unit tests in [agent_events.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_events.rs), [world_ops.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs), and [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) | Extend | + +### 0C. Chosen approach + +| Approach | Summary | Effort | Risk | Decision | +| --- | --- | --- | --- | --- | +| A. Introduce one explicit shell event context and thread it through current helpers | Model shell-owned event authority directly, keep current execution flow mostly intact | Medium | Medium | **Accepted** | +| B. Keep `Option<&str>` plus add more loose optional params | Smaller diff up front, but keeps authority vague and drift-prone | Small | High | Rejected | +| C. Centralize PID lookup in one helper and keep using it | Looks tidy, still violates the trace contract | Small | Unacceptable | Rejected | +| D. Recover identity from trace/env/history when runtime context is absent | Clever fallback, wrong authority boundary | Small | Unacceptable | Rejected | + +The accepted path is the smallest correct slice. It does not redesign the event schema or the +runtime store. It simply stops letting shell emitters lie about how they know who they are. + +### 0D. Complexity check + +Naive scope here balloons fast. If you let the cleanup spill into runtime-store redesign, CLI +status semantics, or trace-family schema work, this becomes a 10-plus-file fog bank. + +The minimal correct production seam is six files: + +1. [crates/shell/src/execution/agent_events.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_events.rs) +2. [crates/shell/src/execution/routing/dispatch/exec.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/exec.rs) +3. [crates/shell/src/execution/routing/dispatch/world_ops.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs) +4. [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) +5. [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) +6. [docs/TRACE.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/TRACE.md) + +Clarification so this does not turn into scope theater: + +- the **core behavior seam** is really the first four files, +- `state_store.rs` is a mechanical cleanup/guard seam because production callers stop using the + PID helper even if the helper itself remains for tests or diagnostics, +- `docs/TRACE.md` is the contract lock, not new runtime logic, +- `crates/shell/src/execution/invocation/runtime.rs` and compile-fix test call sites are allowed + one-line fallout if `execute_command(...)` gains an explicit optional context parameter. + +Tests will expand the touch set. New production behavior should not. + +So the scope reduction is explicit: + +- no new services, +- no new registry or cache, +- no live-state authority redesign, +- no schema rewrite, +- no operator-surface selection changes. + +That keeps the diff engineered enough. Not under-engineered, not a new framework to solve one +honest plumbing problem. + +### 0E. Search and built-in check + +`[Layer 1]` wins. + +The repo already contains the right primitives: + +- authoritative orchestration-session state +- participant lineage fields +- explicit runtime event builders +- fail-closed suppression behavior +- `cmd_id` and `span_id` already threaded in major execution paths + +The correct move is to reuse those primitives and unify shell-owned emitters under them. + +No external concurrency or framework trick is needed here. In fact, adding one would be spending +an innovation token on the wrong thing. + +### 0F. What already exists + +- `translate_wrapper_event(...)` already stamps `role`, `backend_id`, `world_id`, + `world_generation`, and participant lineage onto explicit runtime-owned rows. +- `build_runtime_message_event(...)` already uses manifest plus orchestration-session authority + without PID lookup. +- restart alerts already suppress rows when orchestration context is absent. +- `publish_command_completion(None, ...)` already suppresses the orchestration-scoped row instead + of crashing. +- `emit_stream_chunk(...)` already preserves terminal stdout/stderr even when no + orchestration-scoped row is emitted. + +That means the plan is not "invent explicit authority." It is "make the weaker shell helpers obey +the same contract the stronger runtime helpers already obey." + +### 0G. NOT in scope + +- redesigning [substrate_common::agent_events::AgentEvent](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/common/src/agent_events.rs) +- changing `substrate agent status`, toolbox selection, or live-session ambiguity behavior +- moving or removing compatibility session-root readers introduced by + [PLAN-06](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-06.md) +- world restart/invalidation semantics already owned by + [PLAN-05](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-05.md) +- live-state authority cutover already owned by + [09-live-state-authority-and-compatibility-cutover.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/09-live-state-authority-and-compatibility-cutover.md) +- member runtime launch and lifecycle work already owned by + [10-member-runtime-launch-seam.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/10-member-runtime-launch-seam.md) +- UI work + +## Architecture Contract + +### Explicit shell event authority type + +Add one shell-local context model in +[crates/shell/src/execution/agent_events.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_events.rs) +or an immediately adjacent shell-local module. + +The exact Rust names can stay flexible. The contract cannot. + +Minimum required shape: + +```rust +struct ShellEventEmissionContext { + orchestration_session_id: String, + agent_id: String, + role: Option, + backend_id: Option, + participant_id: Option, + parent_participant_id: Option, + resumed_from_participant_id: Option, + world_id: Option, + world_generation: Option, +} + +struct ShellCommandEventContext { + emission: ShellEventEmissionContext, + cmd_id: String, + run_id: Option, + span_id: Option, +} +``` + +Rules: + +1. shell-owned orchestration-scoped rows are built from this context, not from loose optional + strings. +2. `run_id` stays optional at the type level so suppression can happen cleanly when the caller + does not have one. +3. once this context exists, production paths must not emit orchestration-scoped shell rows with + synthetic correlation like `"unknown"`. + +### Authority flow diagram + +```text +runtime-owned authority + │ + ├── RuntimeOrchestrationContext + ├── active manifest snapshot + └── command-owned correlation (cmd_id, run_id, span_id) + │ + ▼ + ShellEventEmissionContext / ShellCommandEventContext + │ + ├── publish_command_completion(...) + ├── execute_external(...) / spawn_host_stream_thread(...) + └── stream_non_pty_via_agent(...) / emit_stream_chunk(...) + │ + ├── context complete ----------> emit orchestration-scoped AgentEvent + └── context absent/incomplete -> suppress AgentEvent, keep stdout/stderr + spans +``` + +### No PID-based recovery on production emit paths + +After this slice, the following production pattern is forbidden in shell event-emission control +flow: + +```rust +AgentRuntimeStateStore::new()? + .find_active_orchestration_session_for_pid(std::process::id()) +``` + +That ban applies to: + +- REPL command-completion call sites +- host external command stream emission +- world non-PTY stream emission +- any future restart-alert or shell wrapper around those helpers + +The helper may remain in +[state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) +for tests or diagnostics. Production emitters must stop consulting it. + +### Suppress, do not guess + +If a caller cannot provide explicit orchestration authority: + +1. terminal stdout/stderr must still be written, +2. trace span emission must still proceed, +3. the orchestration-scoped `agent_event` row must be suppressed, +4. the shell must not scan runtime state, shell PID ownership, env vars, trace history, or any + other ambient source to reconstruct identity. + +This matches the posture already used by: + +- `publish_command_completion(None, ...)` +- `build_world_restart_required_alert(None, ...)` +- `emit_stream_chunk(..., None, ...)` + +### Caller-owned correlation contract + +For shell-owned orchestration-scoped rows: + +- command completion requires a real `cmd_id` +- stream chunk emission requires a real `run_id` +- `span_id` is attached when known and must not be synthesized + +Hard rule: + +- if the caller cannot supply a real `run_id` for an orchestration-scoped stream row, suppress the + row instead of emitting `"unknown"`. + +That matters because `docs/TRACE.md` already treats `run_id` as a required join key for structured +agent events. + +### Per-path authority and correlation matrix + +This is the part that has to be explicit or the implementation will drift. + +| Path | Authority source | `run_id` source | `span_id` source | If authority/correlation is missing | +| --- | --- | --- | --- | --- | +| REPL command completion | `RuntimeOrchestrationContext` plus active manifest/world binding snapshot | caller-owned `cmd_id` for that command-completion row | attach only if the caller already has one | suppress the orchestration-scoped row | +| Host external command stream | caller-owned explicit shell event context passed into `execute_command(...)` / `execute_external(...)` | stable caller-owned command/run id captured before stream threads start, never `"unknown"` | attach if command-span or downstream span is known | suppress the orchestration-scoped row, keep stdout/stderr | +| World non-PTY deny before agent start | caller-owned explicit shell event context from launch boundary | stable caller-owned command/run id from launch boundary | parent command span when known | suppress the orchestration-scoped row, still print deny text | +| World non-PTY started stream frames | same explicit caller-owned context captured before frame processing | same stable caller-owned run id as launch, do not swap it mid-stream | attach `ExecuteStreamFrame::Start { span_id }` when it arrives | suppress the orchestration-scoped row until both context and stable run correlation are real | +| Non-REPL host invocation callers | none, by design | none | none | always pass `None`; this slice does not fabricate orchestration context for wrap/pipe/runtime helpers | + +Two hard clarifications: + +1. this slice does **not** re-derive orchestration context for + [crates/shell/src/execution/invocation/runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/invocation/runtime.rs); + those call sites remain non-orchestrator callers and pass `None`, +2. once the launch boundary chooses the command/run correlation for a shell-owned row, downstream + frame processing may attach more metadata like `span_id`, but it may not silently replace the + chosen `run_id` with a later convenience value. + +### Existing explicit runtime translation remains the model + +The stronger paths are already correct in principle: + +- `translate_wrapper_event(...)` +- `build_runtime_message_event(...)` +- world-restart alert construction from `startup_context` + +This slice aligns weaker host/world shell emitters to that model. It does not invent a second +authority path. + +## Concrete File Touch Plan + +### 1. `crates/shell/src/execution/agent_events.rs` + +Primary seams: + +- [publish_agent_event(...)](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_events.rs) +- [publish_command_completion(...)](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_events.rs) + +Required behavior: + +- define the explicit shell event-emission context type, +- update `publish_command_completion(...)` to accept explicit context rather than + `Option<&str>`, +- stamp `role`, `backend_id`, lineage, `world_id`, and `world_generation` when the caller + supplies them, +- preserve suppression semantics when context is absent. + +Must not do: + +- keep the old helper shape as the preferred production path, +- let command completion remain only "session id plus message" when richer authority is already + available, +- move the context type into `substrate-common`; this is shell-local plumbing, not a wire-schema + expansion. + +### 2. `crates/shell/src/execution/routing/dispatch/exec.rs` + +Primary seams: + +- [execute_command(...)](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/exec.rs) +- [execute_external(...)](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/exec.rs) +- [spawn_host_stream_thread(...)](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/exec.rs) + +Required behavior: + +- remove `resolve_active_orchestration_session_id()`, +- thread optional explicit event context into `execute_command(...)` and `execute_external(...)`, +- capture caller-owned command/run correlation before any background stream thread starts, +- thread `None` explicitly from non-orchestrator callers such as + [crates/shell/src/execution/invocation/runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/invocation/runtime.rs), +- stop emitting orchestration-scoped host stream rows with `run_id = "unknown"`, +- keep host output behavior unchanged when no orchestration context exists. + +Must not do: + +- reintroduce the PID lookup behind a new helper name, +- require host-only mode to fabricate orchestration context, +- drop stdout/stderr just because the orchestration-scoped row is suppressed. + +### 3. `crates/shell/src/execution/routing/dispatch/world_ops.rs` + +Primary seams: + +- [stream_non_pty_via_agent(...)](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs) +- [process_agent_stream_body(...)](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs) +- [emit_stream_chunk(...)](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs) + +Required behavior: + +- remove `resolve_active_orchestration_session_id()`, +- accept explicit event context from the caller, +- emit stream rows only when both orchestration authority and real run correlation exist, +- continue to mirror stdout/stderr regardless, +- preserve current world-guard deny behavior while suppressing orchestration-scoped rows when + required context is missing. + +Must not do: + +- treat `active_span_id.unwrap_or("unknown")` as acceptable once explicit context plumbing exists, +- let background frame processing infer the orchestration session after the fact. + +### 4. `crates/shell/src/repl/async_repl.rs` + +Primary seams: + +- REPL command-completion call sites +- [RuntimeOrchestrationContext](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) +- [exec_host_line(...)](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) +- [exec_world_line(...)](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) +- [exec_world_pty(...)](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) + +Required behavior: + +- stop calling `resolve_active_orchestration_session_id()` for command completion, +- derive shell event context from `RuntimeOrchestrationContext` and the live manifest snapshot + when the runtime exists, +- pass that context into host/world execution helpers, +- keep the command/run correlation choice stable at the launch boundary instead of letting world + stream frame handlers invent or swap it later, +- keep host-only `--no-world` behavior suppression-only when there is no orchestrator runtime, +- preserve existing restart-alert explicit-context behavior. + +Must not do: + +- add new runtime-global state just for emitter lookup, +- turn `async_repl.rs` into a second event builder with duplicated field assignment logic. + +### 5. `crates/shell/src/execution/agent_runtime/state_store.rs` + +Required behavior: + +- leave `find_active_orchestration_session_for_pid(...)` available only if tests or diagnostics + still need it, +- ensure production event-emission paths no longer call it. + +This is intentionally a small file-touch item. The goal is not more state-store behavior. The goal +is to narrow who depends on the helper. + +### 6. `docs/TRACE.md` and test anchors + +Required behavior: + +- keep the documented contract aligned with the code, +- make shell-owned command completion and stream emitters explicit examples of the + "real id or suppress" rule, +- add bounded regression assertions so future diffs cannot quietly reintroduce PID lookup or + synthetic fallback correlation. + +## Execution Plan + +### Ordered implementation sequence + +Implement in this order. Each phase creates one invariant the next phase can safely consume. + +1. **Freeze the shell event context contract** + - add the explicit shell event-emission context type, + - update `publish_command_completion(...)`, + - keep suppression semantics intact. + +2. **Thread authority through the REPL boundary** + - derive optional explicit context from `RuntimeOrchestrationContext` plus live manifest + snapshots, + - pass it into host and world command entrypoints, + - remove REPL-local PID lookup. + +3. **Freeze host command stream semantics** + - update `execute_command(...)` and `execute_external(...)`, + - remove host-path PID lookup, + - require caller-owned correlation before host stream rows emit. + +4. **Freeze world non-PTY stream semantics** + - update `stream_non_pty_via_agent(...)`, `process_agent_stream_body(...)`, and + `emit_stream_chunk(...)`, + - remove world-path PID lookup, + - suppress orchestration-scoped rows when `run_id` is not authoritative. + +5. **Widen regression coverage and doc wording** + - lock command-completion suppression and explicit context behavior, + - lock stream suppression and no-`unknown` fallback behavior, + - lock the repo-wide no-PID-lookup production rule, + - clarify `docs/TRACE.md`. + +### Phase-by-phase acceptance gates + +| Phase | Acceptance gate | +| --- | --- | +| 1. Context contract | shell-owned event helpers accept explicit authority rather than loose optional strings | +| 2. REPL boundary | command-completion paths no longer resolve orchestration identity from shell PID | +| 3. Host streams | host stdout/stderr still work, but orchestration-scoped rows emit only with explicit authority plus real correlation | +| 4. World streams | non-PTY world stream rows emit only from caller-owned authority, never from ambient lookup or `"unknown"` correlation | +| 5. Regression closeout | tests and docs prove "real id or suppress" across shell-owned emitters | + +## Architecture Review + +### Locked architecture decisions + +1. **Keep event authority at the caller boundary.** + - The caller that launches the command already knows the runtime/session owner. The emitter + should consume that authority, not rediscover it. + +2. **Keep shell event context shell-local.** + - This is not a wire-schema change. It is an internal authority-plumbing type. + +3. **Keep suppression fail closed.** + - Missing context suppresses the orchestration-scoped row. It does not block terminal output + or spans, and it does not trigger a fallback scan. + +4. **Keep correlation explicit.** + - `run_id = "unknown"` is no longer acceptable for orchestration-scoped shell stream rows once + explicit context exists. + +5. **Keep runtime-owned event builders as the gold standard.** + - `translate_wrapper_event(...)` and `build_runtime_message_event(...)` already show the right + posture. Follow them. + +### Authority and correlation diagram + +```text +host/world command launch + │ + ├── cmd_id always known + ├── run_id known for orchestrator-owned runtime flows + ├── span_id known after stream start or command-span creation + └── orchestration authority known only if runtime/manifest says so + │ + ▼ + if orchestration authority && real run_id + emit orchestration-scoped AgentEvent + else + write stdout/stderr only + keep trace spans only + emit no orchestration-scoped AgentEvent +``` + +### Architecture acceptance gates + +1. **Authority gate** + - no production event-emission path in `async_repl`, `dispatch/exec`, or `dispatch/world_ops` + resolves orchestration identity from `shell_owner_pid`. + +2. **Correlation gate** + - orchestration-scoped stream rows never emit with synthetic `run_id = "unknown"`. + +3. **Suppression gate** + - missing context suppresses only the orchestration-scoped row, not terminal output or command + tracing. + +## Code Quality Review + +### Implementation guardrails + +1. one explicit context type for shell-owned event authority, not more raw `Option<&str>` sprawl, +2. one place that stamps lineage and backend identity for shell-owned rows, not field drift across + call sites, +3. one suppression posture across command completion and stream emission, +4. no new ambient lookup helper, +5. no new state-store contract or trace schema just to support this slice. + +### Minimal-diff rules + +- keep the production diff inside the six files already named, +- prefer extending current tests over inventing a new harness, +- if a helper extraction is needed, keep it local to the shell crate, +- do not broaden this slice into `PLAN-09` live-state authority work. + +## Error & Rescue Registry + +| Failure point | What goes wrong | Required rescue | +| --- | --- | --- | +| caller lacks orchestration authority | shell would be tempted to guess via PID lookup | suppress orchestration-scoped row, keep stdout/stderr and spans | +| caller lacks real `run_id` for stream row | event is emitted with fake correlation and poisons trace joins | suppress orchestration-scoped row instead of using `"unknown"` | +| REPL runtime snapshot is stale or absent | host/world completion path cannot build rich authority payload | emit no orchestration-scoped completion row | +| background stream thread outlives command setup assumptions | late chunks try to reconstruct session ownership ambiently | capture immutable explicit context before thread/frame loop starts | +| a future diff reintroduces PID lookup | trace contract silently regresses | add bounded repo/test guard proving production emitters no longer call the helper | +| docs drift from implementation | operators trust a contract the code no longer satisfies | tighten `docs/TRACE.md` in the same slice | + +## Test Review + +This slice already has some decent regression anchors. Good. The missing part is that the current +tests still prove "optional session id works" instead of "explicit authority is required." + +### Code path coverage + +```text +CODE PATH COVERAGE +=========================== +[+] crates/shell/src/execution/agent_events.rs + │ + ├── [★★★ TESTED] command completion emits when context/session id is present + ├── [★★★ TESTED] command completion suppresses when context/session id is absent + ├── [GAP] explicit shell event context stamps role/backend/lineage correctly + └── [GAP] success + failure completion both preserve suppression-only posture when context is absent + +[+] crates/shell/src/repl/async_repl.rs + │ + ├── [★★★ TESTED] restart alerts already require explicit orchestration context + ├── [GAP] host escape completion no longer reads PID-owned runtime state + ├── [GAP] world PTY completion no longer reads PID-owned runtime state + ├── [GAP] world line completion no longer reads PID-owned runtime state + └── [GAP] host-only completion suppresses event rows without affecting terminal behavior + +[+] crates/shell/src/execution/routing/dispatch/exec.rs + │ + ├── [GAP] execute_command accepts optional explicit event context + ├── [GAP] host stream threads emit orchestration-scoped rows only with real run_id + ├── [GAP] no host stream row emits with run_id="unknown" + └── [GAP] stdout/stderr still mirror normally when context is absent + +[+] crates/shell/src/execution/routing/dispatch/world_ops.rs + │ + ├── [★★★ TESTED] emit_stream_chunk emits with orchestration context + ├── [★★★ TESTED] emit_stream_chunk suppresses without orchestration context + ├── [GAP] stream_non_pty_via_agent no longer resolves orchestration session via PID lookup + ├── [GAP] world stream rows suppress when run_id would otherwise be synthetic + └── [GAP] world-guard deny path still prints error text while suppressing orchestration row if context is incomplete + +[+] repo-wide production guard + │ + └── [GAP] only tests/diagnostics may still reference find_active_orchestration_session_for_pid(...) + +───────────────────────────────── +COVERAGE: 5/16 paths tested (31%) +QUALITY: ★★★: 5 ★★: 0 ★: 0 +GAPS: 11 paths need regression tests +───────────────────────────────── +``` + +### User flow coverage + +```text +USER FLOW COVERAGE +=========================== +[+] REPL host escape inside active orchestrator runtime + │ + ├── [GAP] [→INTEGRATION] command completes and emits a completion row from explicit runtime context + └── [GAP] completion does not consult PID-owned runtime state + +[+] REPL host-only command with no active orchestrator runtime + │ + ├── [GAP] stdout/stderr and trace span still appear + └── [GAP] no orchestration-scoped completion row is emitted + +[+] World non-PTY command stream + │ + ├── [GAP] [→INTEGRATION] stream chunks emit with caller-owned orchestration authority and real run correlation + ├── [GAP] missing start/run correlation suppresses orchestration row only + └── [★★★ TESTED] chunk helper itself preserves terminal output + +[+] Host external command stream + │ + ├── [GAP] background stdout/stderr threads use captured explicit context, not late ambient lookup + └── [GAP] no stream row emits with synthetic run_id="unknown" + +[+] Restart alert behavior + │ + └── [★★★ TESTED] explicit-context-only posture already holds and must remain unchanged + +[+] Repo contract guard + │ + └── [GAP] production code no longer references find_active_orchestration_session_for_pid(...) + +───────────────────────────────── +COVERAGE: 2/11 flows tested (18%) +GAPS: 9 flows need tests (2 deserve integration-style crate coverage rather than unit-only assertions) +───────────────────────────────── +``` + +### Required test additions by file + +#### `crates/shell/src/execution/agent_events.rs` + +Add regression coverage for: + +- command completion built from explicit shell event context rather than a bare session id, +- failure and success completion rows preserving `cmd_id`, +- suppression when context is absent even if success events are enabled, +- lineage and backend identity stamping when context includes them. + +#### `crates/shell/src/repl/async_repl.rs` + +Add coverage for: + +- REPL host escape completion no longer consulting PID-owned runtime state, +- world PTY completion no longer consulting PID-owned runtime state, +- host-only completion suppressing the orchestration-scoped row without affecting terminal + behavior, +- restart-alert explicit-context behavior remaining unchanged. + +#### `crates/shell/src/execution/routing/dispatch/exec.rs` + +Add regression coverage for: + +- explicit event context threaded into `execute_command(...)` / `execute_external(...)`, +- non-REPL callers that pass `None` continuing to execute normally while suppressing + orchestration-scoped rows, +- host stream rows suppressing when real run correlation is absent, +- host stream rows keeping the launch-owned correlation stable instead of inventing a later one, +- no host stream row emitting with `run_id = "unknown"`, +- stdout/stderr still being mirrored when the orchestration-scoped row is suppressed. + +#### `crates/shell/src/execution/routing/dispatch/world_ops.rs` + +Add regression coverage for: + +- `stream_non_pty_via_agent(...)` no longer consulting PID lookup, +- `process_agent_stream_body(...)` suppressing orchestration rows if a real run id is not + authoritative yet, +- `ExecuteStreamFrame::Start { span_id }` enriching later chunk rows with `span_id` without + replacing the launch-owned run correlation, +- deny-path stderr printing surviving row suppression, +- repo-contract tests for no synthetic correlation. + +#### `docs/TRACE.md` or shell-level bounded repo guard + +Add one bounded guard that proves: + +- production shell emitters no longer reference + `find_active_orchestration_session_for_pid(...)`, +- any remaining references are tests or diagnostics only. + +### Test commands + +Run at minimum: + +```bash +cargo test -p shell publish_command_completion -- --nocapture +cargo test -p shell emit_stream_chunk -- --nocapture +cargo test -p shell build_world_restart_required_alert_only_builds_with_orchestration_context -- --nocapture +cargo test -p shell emit_world_restarted_alert_only_emits_with_orchestration_context -- --nocapture +cargo test -p shell start_host_orchestrator_runtime_persists_participant_snapshots_across_lifecycle_states -- --nocapture +cargo test -p shell -- --nocapture +``` + +Then run: + +```bash +cargo test -p world-agent -- --nocapture +cargo test -p world-api -- --nocapture +cargo test -p agent-api-types -- --nocapture +``` + +Only treat that second block as required if the implementation actually changes shared request, +stream-frame, or cross-crate event-shape assumptions. If this stays inside shell-local authority +plumbing, `cargo test -p shell -- --nocapture` is the non-negotiable gate and the cross-crate +runs are blast-radius confidence. + +### QA artifact + +Generate the standard eng-review handoff artifact at: + +```text +~/.gstack/projects/$SLUG/{user}-{branch}-eng-review-test-plan-{datetime}.md +``` + +Required contents for this slice: + +- affected shell execution paths: REPL completion, host external stream, world non-PTY stream, +- the suppression-only scenarios that must preserve stdout/stderr and trace spans, +- the negative assertions that no orchestration-scoped row is emitted when explicit context is + absent or correlation would be synthetic. + +## Failure Modes Registry + +| New codepath | Real production failure | Test covers it? | Error handling exists? | User sees clear error? | Critical gap? | +| --- | --- | --- | --- | --- | --- | +| REPL completion authority plumbing | completion row is attached to the wrong orchestration session after a stale PID-based lookup | no | partial today | no | yes until PID lookup is removed and tested | +| host stream emission | stdout/stderr rows get emitted with `run_id="unknown"` and poison trace joins | no | no | no | yes until suppression-only behavior is tested | +| world non-PTY stream emission | late stream chunks reconstruct orchestration ownership ambiently after command launch | no | partial today | no | yes until explicit context is captured and tested | +| host-only no-runtime command | shell suppresses event row but accidentally drops terminal output or trace span | no | yes | partial today | yes until regression-covered | +| repo-wide authority contract | a future diff reintroduces `find_active_orchestration_session_for_pid(...)` into production emitters | no | no | no | yes until bounded guard lands | +| restart alert helpers | explicit-context-only behavior regresses during refactor | yes | yes | yes | no, but keep the existing tests green | + +Critical gap rule: + +If production shell emitters can still reconstruct orchestration identity from shell PID, or if +they can still emit orchestration-scoped stream rows with synthetic run correlation, this slice is +not done. + +## Performance Review + +This is a correctness and authority slice, not a throughput slice. + +Still, four performance rules matter: + +1. capture immutable event context once per command, not once per stream chunk, +2. do not add runtime-store scans to hot chunk loops, +3. cloning a small explicit context object is cheap and dramatically cheaper than doing ambient + session scans, +4. do not add a cache or registry to "speed up" the wrong architecture. + +The performance footgun here would be spending more code to avoid cloning a few strings while +keeping the wrong authority path alive. That would be classic software: a 200-line optimization to +protect a bug. + +## Worktree Parallelization Strategy + +### Dependency table + +| Step | Modules touched | Depends on | +| --- | --- | --- | +| A. Freeze the context contract and launch-boundary mapping | `crates/shell/src/execution/`, `crates/shell/src/repl/` | — | +| B. Re-plumb host and world stream emitters onto the frozen contract | `crates/shell/src/execution/routing/dispatch/` | A | +| C. Land regression guards, test-plan artifact, and trace wording | `crates/shell/src/`, `docs/`, `~/.gstack/projects/` | A, B | + +### Parallel lanes + +- Lane A: step A +- Lane B: step B, only after Lane A lands +- Lane C: step C, can split from Lane B only after signatures stop moving + +### Execution order + +1. land Lane A first because it freezes both the context type and the per-path + authority/correlation mapping, +2. land Lane B next because `exec.rs` and `world_ops.rs` both depend on that exact contract, +3. once Lane B settles, Lane C can be handled as a short follow-up or parallel cleanup across + tests/docs only. + +### Conflict flags + +- `agent_events.rs`, `async_repl.rs`, `exec.rs`, and `world_ops.rs` are one coupled seam. +- `execute_command(...)` signature fallout into + [invocation/runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/invocation/runtime.rs) + is mechanically small, but it is still contract fallout and belongs with Lane A or B, not a + free-floating branch. +- Production-code parallelization before the context contract freezes is asking for merge-conflict + soup. + +### Parallelization verdict + +This slice is mostly sequential. + +- There are **three lanes** on paper. +- There are **zero safe parallel production-code lanes** before Lane A finishes. +- The only realistic post-freeze split is: one branch finishes execution plumbing, one branch + finishes regression/doc guardrails after the helper signatures stop moving. + +## Deferred Work + +There is no `TODOS.md` in the repo root, so explicit deferrals stay here: + +1. live-state authority cutover and compatibility-write removal + - owned by + [09-live-state-authority-and-compatibility-cutover.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/09-live-state-authority-and-compatibility-cutover.md) +2. production member runtime launch and world-scoped lifecycle wiring + - owned by + [10-member-runtime-launch-seam.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/10-member-runtime-launch-seam.md) +3. any broader agent-event schema redesign + - intentionally out of scope for this authority-plumbing slice +4. nicer operator-facing diagnostics for suppressed orchestration-scoped rows + - useful later, not required to make authority honest now + +## Definition of Done + +This slice is done only when all of the following are true: + +1. no production event-emission path in `async_repl`, `dispatch/exec`, or `dispatch/world_ops` + resolves orchestration identity from `shell_owner_pid`, +2. shell-owned command-completion rows consume explicit caller-provided authority, +3. shell-owned stream-chunk rows consume explicit caller-provided authority and real run + correlation, +4. missing orchestration context suppresses the orchestration-scoped `agent_event` row without + suppressing stdout/stderr or trace spans, +5. orchestration-scoped shell stream rows no longer emit with synthetic fallback correlation like + `"unknown"`, +6. runtime-owned event translation paths remain explicit and unchanged in principle, +7. docs and tests reflect the "real id or suppress" contract, +8. `find_active_orchestration_session_for_pid(...)`, if retained, is no longer part of production + emission control flow. + +## Completion Summary + +- Step 0: scope reduced to the six-file production seam, no schema or live-state redesign +- Architecture Review: 3 issues found, all authority/correlation contract gaps rather than a + need for new infrastructure +- Code Quality Review: 2 issues found, weak helper signatures and duplicated ambient lookup + posture +- Test Review: diagrams produced, 11 concrete regression gaps identified +- Performance Review: 0 issues found +- NOT in scope: written +- What already exists: written +- TODOS.md updates: 0, repo has no `TODOS.md`, deferrals captured here +- Failure modes: 5 critical gaps flagged +- Outside voice: skipped, `claude` CLI is installed but not authenticated on 2026-05-01 and no + separate subagent review was run +- Parallelization: 3 lanes, 0 safe production-code parallel lanes before context freeze +- Lake Score: complete option chosen for every in-slice decision + + +## Decision Audit Trail + +| # | Phase | Decision | Classification | Principle | Rationale | Rejected | +| --- | --- | --- | --- | --- | --- | --- | +| 1 | Scope | Treat this as authority-plumbing repair, not a trace-schema redesign | Mechanical | Pragmatic | The repo already has the right event schema and runtime authority sources | Redesigning `AgentEvent` | +| 2 | Authority | Introduce one shell-local explicit event context type | Mechanical | Explicit over clever | One context object is easier to audit than many raw optional strings | Extending the current `Option<&str>` pattern | +| 3 | Safety | Suppress orchestration-scoped rows when context or run correlation is absent | Mechanical | Completeness | Fail-closed behavior preserves correctness without breaking terminal output | PID lookup fallback or `run_id=\"unknown\"` | +| 4 | Reuse | Keep runtime-owned event builders as the model | Mechanical | DRY | The repo already has correct explicit event builders in `async_repl.rs` | Building a second shell-specific field-stamping path | +| 5 | Scope reduction | Keep production edits inside six files plus bounded docs/tests | Mechanical | Minimal diff | This slice is already sharp enough and should not swallow `PLAN-09` | Folding in live-state cutover or status-surface work | +| 6 | Tests | Add a bounded guard proving production emitters no longer call the PID helper | Mechanical | Systems over heroes | Without a guard, this regression will come back in a future refactor | Relying on reviewer memory | + +## GSTACK REVIEW REPORT + +| Review | Trigger | Why | Runs | Status | Findings | +| --- | --- | --- | --- | --- | --- | +| CEO Review | `/plan-ceo-review` | Scope and strategy | 0 | SKIPPED | This is a bounded shell/runtime authority cleanup slice, not a product-scope decision | +| Codex Review | `/codex review` | Independent 2nd opinion | 0 | SKIPPED | No separate Codex outside-voice review run | +| Eng Review | `/plan-eng-review` | Architecture and tests (required) | 1 | CLEAR | Locked the slice to explicit shell event authority, removed ambient PID lookup from the target design, and identified the remaining regression gaps around no-fallback correlation, suppression-only behavior, and bounded repo guards | +| Design Review | `/plan-design-review` | UI/UX gaps | 0 | SKIPPED | No UI scope | + +**UNRESOLVED:** 0 blocking architecture decisions remain inside slice `08`. The remaining work is +authority plumbing plus regression coverage. + +**VERDICT:** ENG CLEARED. `PLAN-08` is ready to execute as the shell event-authority cleanup slice +between the session-centric state-store work and the later live-state authority cutover. diff --git a/llm-last-mile/PLAN-09.md b/llm-last-mile/PLAN-09.md new file mode 100644 index 000000000..0bc1bf2be --- /dev/null +++ b/llm-last-mile/PLAN-09.md @@ -0,0 +1,955 @@ + + +# PLAN-09: Live-State Authority and Compatibility Cutover + +Source file: [09-live-state-authority-and-compatibility-cutover.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/09-live-state-authority-and-compatibility-cutover.md) +Branch: `feat/session-centric-state-store` +Plan type: shell/runtime authority-cutover contract, no UI scope, strong DX scope +Review posture: `/autoplan`-style scope tightening with `/plan-eng-review` structure and rigor +Status: execution-ready after [PLAN-08.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-08.md), with outside voice skipped on 2026-05-01 because `claude` CLI auth is missing + +## Objective + +This slice is not about inventing a new live registry. + +The repo already has the right building blocks: + +- canonical session-root records under + [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) +- fail-closed operator surfaces in + [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs) +- one runtime writer choke point in + [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) +- bounded contract tests in + [crates/shell/tests/agent_successor_contract_ahcsitc0.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_successor_contract_ahcsitc0.rs) +- operator wording in + [docs/TRACE.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/TRACE.md) + and + [docs/USAGE.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/USAGE.md) + +What is still too soft is the cutover contract. A future diff can still regress this area by: + +1. treating flat compatibility files as current truth again, +2. letting trace history authorize current toolbox state, +3. loosening torn-root or ambiguity behavior into "pick the newest one", +4. removing the compatibility bridge before fixtures, docs, and downstream readers are ready, +5. spreading write ownership outside the store. + +`PLAN-09` fixes that by freezing one boring rule set: + +1. canonical session-root parent and participant records are live-state authority, +2. flat compatibility files are bridge input/output only during cutover, +3. legacy `handles/*.json` stays read-only compatibility input of last resort, +4. trace is historical fallback only for status gaps, never current-session authority, +5. `status` and `toolbox` fail closed on ambiguity, corruption, or broken parent/child linkage, +6. flat bridge retirement gets explicit gates instead of wishful thinking. + +This matters because the user-facing experience is one of two things: + +- clean: `substrate agent status` and `substrate agent toolbox env` tell the truth about the current live runtime, or +- a mess: the shell shows a stale endpoint because some old trace row or flat file looked newer. + +This slice makes the first outcome durable. + +## Step 0: Scope Challenge + +### 0A. Repo truth and why this slice exists + +The SOW is directionally right, and the repo already proves most of it. + +What the code already does today: + +1. `load_authoritative_session(...)` in + [state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) + prefers canonical session-root parents and only falls back to flat parent files when the + canonical root is absent. +2. `list_sessions()` and `list_live_sessions()` already distinguish discoverable torn roots from + actually live sessions. +3. `resolve_single_live_session_for_agent(...)` already fails closed on multiple active parents, + multiple live host orchestrators, missing selected participants, and inactive selected + participants. +4. `build_status_report(...)` already checks live runtime state before projecting trace fallback. +5. `build_toolbox_status_report(...)` and `build_toolbox_env_report(...)` already resolve through + store-owned live session discovery and reject trace-only liveness. +6. `persist_runtime_snapshots(...)` in + [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) + already centralizes parent + participant persistence through the store. +7. existing tests already pin the most dangerous edges: + - canonical over flat precedence + - torn-root degradation + - trace not authorizing toolbox + - tombstone suppression beating stale trace + - multi-parent ambiguity failing closed + +What is still missing is not a new mechanism. It is one locked contract around those mechanisms. + +### 0B. Premise challenge + +Premise check, one by one: + +1. **Canonical session-root objects should be live-state authority.** + - Accepted. + - Verified by `load_authoritative_session(...)`, `list_live_sessions()`, and the canonical-over-flat tests. + +2. **Flat compatibility files still need to exist during the cutover.** + - Accepted. + - Verified by the explicit dual-write tests in + [state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) + and by docs/tests that still mention flat compatibility paths as bridge input. + +3. **Trace must stay historical, never current-session authority.** + - Accepted. + - Verified by + `agent_toolbox_env_trace_history_does_not_authorize_active_session` + and + `agent_status_prefers_live_manifest_over_trace_fallback_for_selected_orchestrator`. + +4. **`toolbox` should fail closed instead of picking the newest live candidate.** + - Accepted. + - That is the only sane 3am behavior. Choosing "latest" is a great way to aim an operator at the wrong socket. + +5. **This slice should remove the bridge now.** + - Rejected. + - The repo is not ready. Tests, docs, and compatibility readers still rely on flat bridge posture. Doing freeze plus removal in one slice is how you ship a migration footgun. + +Premise gate posture: + +- accepted as-is for this plan, +- with one explicit constraint: this slice freezes the bridge contract and removal gates, it does not remove the bridge. + +### 0C. Existing code to reuse + +| Sub-problem | Existing code | Plan | +| --- | --- | --- | +| Canonical parent precedence | `load_authoritative_session(...)` in [state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) | Reuse | +| Canonical + flat participant merge | `build_session_record(...)`, `list_sessions()`, canonical + flat participant readers in [state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) | Reuse, tighten contract language | +| Live-session discovery | `list_live_sessions()`, `resolve_single_live_session_for_agent(...)` in [state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) | Reuse, add any missing regression guards | +| Tombstone suppression | `list_invalidated_participants_across_sources()` plus fallback suppression helpers in [agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs) | Reuse | +| Operator projections | `build_status_report(...)`, `build_toolbox_status_report(...)`, `build_toolbox_env_report(...)` in [agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs) | Reuse | +| Centralized write ownership | `persist_runtime_snapshots(...)` in [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) | Reuse, keep as sole caller choke point | +| Contract-proof tests | [agent_successor_contract_ahcsitc0.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_successor_contract_ahcsitc0.rs) and store unit tests in [state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) | Extend | +| Operator wording | [docs/TRACE.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/TRACE.md), [docs/USAGE.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/USAGE.md) | Reuse, tighten if drift remains | + +### 0D. Dream state and 12-month ideal + +```text +CURRENT REPO + │ + ├── canonical session-root parent + participant records exist + ├── flat compatibility files still exist + ├── legacy handles are still readable + ├── status/toolbox mostly honor the right precedence + └── the final cutover contract is spread across code, docs, and tests + │ + ▼ +THIS PLAN + │ + ├── one explicit authority ladder + ├── one explicit operator-surface rule set + ├── one explicit torn-root posture + ├── one explicit dual-write ownership rule + └── one explicit bridge-removal gate list + │ + ▼ +12-MONTH IDEAL + │ + ├── canonical session-root records only + ├── no flat compatibility parent/participant/lease writes + ├── no legacy handle reads + ├── trace purely historical + └── docs + tests + runtime all say the same thing +``` + +What changes for the builder: + +- today, you still need to triangulate code, tests, and docs to know what is truth, +- after this slice, you read one contract and then implement against it, +- later, bridge retirement becomes a smaller, safer slice instead of a repo archeology expedition. + +### 0E. Implementation alternatives + +| Approach | Summary | Effort | Risk | Decision | +| --- | --- | --- | --- | --- | +| A. Freeze the contract now, keep the bridge, add explicit retirement gates | Smallest correct slice, matches current repo truth | Medium | Low | **Accepted** | +| B. Remove flat compatibility reads and writes immediately | Looks clean, breaks migration safety and fixtures | Medium | High | Rejected | +| C. Docs-only cleanup | Easy, but leaves runtime drift and caller misuse risk | Small | High | Rejected | +| D. Build a new transactional runtime registry | Fancy, unnecessary, spends an innovation token on the wrong thing | Large | Unacceptable | Rejected | + +### 0F. Complexity, search, completeness, and distribution checks + +`[Layer 1]` wins. + +The runtime already has the right primitives. The correct move is to reuse them and make them +hard to misread. Not to invent a new registry, cache, or selection heuristic. + +Minimal production seam: + +1. [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) +2. [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs) +3. [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) +4. [docs/TRACE.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/TRACE.md) +5. [docs/USAGE.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/USAGE.md) +6. [crates/shell/tests/agent_successor_contract_ahcsitc0.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_successor_contract_ahcsitc0.rs) +7. store unit tests in [state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) + +Secondary docs under +[docs/project_management/packs/draft/agent-hub-core-successor-identity-tuple-compatible/](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/project_management/packs/draft/agent-hub-core-successor-identity-tuple-compatible/) +should only be touched if they contradict the active runtime contract. + +Completeness check: + +- the complete version is to freeze the bridge contract, test it, and state removal gates now, +- the shortcut is to say "we know what we mean" and leave it half-encoded across code and tests, +- with AI-assisted implementation, the complete version is the obvious choice. + +Distribution check: + +- no new binary, package, image, or artifact type is introduced here, +- distribution work is not applicable. + +### 0G. What already exists + +1. canonical session-root precedence already exists, +2. dual-write to canonical + flat compatibility layouts already exists, +3. torn-root degradation already exists, +4. live runtime beats trace fallback already exists, +5. tombstones already suppress stale trace fallback, +6. toolbox already refuses trace-only liveness, +7. docs already say flat and handle files are compatibility input only, +8. contract tests already grep for some of that wording. + +This is good news. The plan is mostly about making that truth legible and harder to regress. + +### 0H. NOT in scope + +- removing flat compatibility reads in this slice +- removing flat dual-write in this slice +- removing legacy `handles/*.json` reads in this slice +- redesigning the parent session schema +- changing public JSON shape for `substrate agent status` or `substrate agent toolbox ...` +- changing trace schema or adding new trace families +- inventing a new live registry or transaction layer +- reopening [PLAN-08.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-08.md) event-emission authority cleanup +- reopening [PLAN-05.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-05.md) invalidation semantics +- member-runtime launch/lifecycle work from [10-member-runtime-launch-seam.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/10-member-runtime-launch-seam.md) +- UI work + +## Architecture Contract + +### Live-state authority order + +For live operator surfaces, authority is ordered exactly like this: + +1. canonical parent session record + `~/.substrate/run/agent-hub/sessions//session.json` +2. canonical participant record + `~/.substrate/run/agent-hub/sessions//participants/.json` +3. flat compatibility parent/participant files only to fill incomplete canonical roots on read +4. legacy `handles/*.json` only as last-resort compatibility input +5. trace fallback only for historical status projection gaps, never current liveness + +Hard rules: + +1. canonical objects outrank conflicting flat compatibility objects, +2. flat compatibility data can fill a gap, never overrule a present canonical object, +3. legacy handles are never live-state authority, +4. trace never authorizes toolbox or current control-plane liveness, +5. torn roots are discoverable but non-live, +6. ambiguity and corruption fail closed. + +### Authority flow diagram + +```text +operator surface + │ + ├── substrate agent status + └── substrate agent toolbox status|env + │ + ▼ + AgentRuntimeStateStore + │ + ├── canonical parent root + ├── canonical participant root + ├── flat compatibility gap-fill only + ├── legacy handle fallback only + └── trace fallback only for status tuples still uncovered + │ + ├── status: allowed after live-state + tombstone filtering + └── toolbox: never allowed +``` + +### Operator surface contract matrix + +| Surface | Current truth source | Allowed fallback | Forbidden fallback | Failure posture | +| --- | --- | --- | --- | --- | +| `substrate agent status` | `list_live_sessions()` projections | trace for tuples not covered by live state or tombstones | trace to override live state | keep concurrent sessions distinct, never collapse to "latest" | +| `substrate agent toolbox status` | `resolve_single_live_session_for_agent(...)` | flat participant when canonical root is parent-only | trace-only liveness | `dependency_unavailable` or fail closed | +| `substrate agent toolbox env` | `resolve_single_live_session_for_agent(...)` | flat participant when canonical root is parent-only | trace-only liveness | exit `3` or fail closed | + +### Torn-root contract + +```text +canonical parent present, canonical child present + -> complete, eligible for live discovery if parent active and owner alive + +canonical parent present, canonical child absent + -> incomplete torn root + -> flat participant fallback allowed during cutover + -> warnings preserved + -> never promoted to live without a valid selected participant + +canonical parent absent, canonical child present + -> discoverable torn root + -> warnings preserved + -> never promoted to live + +legacy handle only + -> compatibility input only + -> never treated as primary truth when canonical or flat data exists +``` + +### Dual-write contract + +Dual-write remains intentional during cutover, but only here: + +- `persist_orchestration_session(...)` +- `persist_participant(...)` +- `persist_parent_session_snapshot(...)` + +inside +[state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs). + +Rules: + +1. callers write canonical truth through the store, +2. the store may also write flat compatibility copies while the bridge exists, +3. callers do not write flat compatibility copies directly, +4. callers do not reintroduce `handles/*.json` write ownership, +5. flat dual-write removal happens only after explicit bridge-removal gates are green. + +## Concrete File Touch Plan + +### 1. `crates/shell/src/execution/agent_runtime/state_store.rs` + +Keep this file as the live-state owner. Do not split authority anywhere else. + +Required work: + +- preserve this read order in `load_authoritative_session(...)` and keep it obvious in code: + canonical parent -> flat parent only if canonical parent is absent -> canonical participant + -> flat participant only if canonical participant is absent -> legacy handle alias last, +- keep `list_live_sessions()` and `resolve_single_live_session_for_agent(...)` strict about + parent state, owner PID liveness, selected-participant presence, selected-participant activity, + and parent/child linkage, +- keep dual-write ownership local to `persist_orchestration_session(...)`, + `persist_participant(...)`, and `persist_parent_session_snapshot(...)`, +- add or retain direct store-level regressions for: + - canonical participant beating conflicting legacy-handle fallback, + - flat participant beating conflicting legacy-handle fallback when the canonical child is absent, + - selected participant present but inactive failing closed before any operator surface can project it, +- do not introduce a new cache, registry, or helper layer to "clarify" this logic. + +### 2. `crates/shell/src/execution/agents_cmd.rs` + +Keep operator surfaces strict and boring. + +Required work: + +- `build_status_report(...)` must keep live-session projections first and must only consult trace + after live-state projection plus tombstone suppression have already decided which tuples are still + unresolved, +- `build_toolbox_status_report(...)` must continue resolving through + `resolve_single_live_session_for_agent(...)` rather than adding local precedence or recovery + logic, +- `build_toolbox_env_report(...)` must continue to require a real resolved live endpoint and keep + exit `3` for `dependency_unavailable`, +- parent-only canonical roots may degrade through flat participant fallback during cutover, but + no path in this file may promote a torn root to live on its own, +- no heuristic "latest session", newest timestamp, or trace-only live-session recovery gets added + anywhere in this file. + +### 3. `crates/shell/src/repl/async_repl.rs` + +Keep `persist_runtime_snapshots(...)` as the caller choke point. + +Required work: + +- runtime code continues writing through `persist_runtime_snapshots(...)`, and that helper + continues delegating only to `store.persist_orchestration_session(...)` plus + `store.persist_participant(...)`, +- no direct writes to `sessions/.json`, `participants/*.json`, flat + lease files, or `handles/*.json` appear outside the store, +- if a helper extraction happens, it stays mechanical and local, with no new persistence owner and + no new call graph branch around the store. + +### 4. Active operator docs + +Primary targets: + +- [docs/TRACE.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/TRACE.md) +- [docs/USAGE.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/USAGE.md) + +Required work: + +- keep one clear authority statement: canonical session-root parent + participant records are the + live-state authority boundary, +- keep one clear toolbox fail-closed statement: `substrate agent toolbox env --json` emits + variables only for a current live host-scoped orchestrator session and otherwise exits `3`, +- keep one clear trace statement: trace is historical fallback for `status` gaps only, never + current-session toolbox authorization, +- keep one clear bridge-removal gate statement so a future cleanup diff cannot claim "the docs + looked ready" while runtime/tests still depend on the bridge, +- make sure docs do not imply that `handles/*.json` or trace history are current truth. + +### 5. Secondary docs audit + +Audit only if active wording still contradicts runtime truth: + +- [compatibility-spec.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/project_management/packs/draft/agent-hub-core-successor-identity-tuple-compatible/compatibility-spec.md) +- [manual_testing_playbook.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/project_management/packs/draft/agent-hub-core-successor-identity-tuple-compatible/manual_testing_playbook.md) + +If they already agree, leave them alone. Minimal diff matters. If they disagree, update only the +contradictory sentence or bullet, not the whole pack. + +### 6. Regression suites + +Primary anchors: + +- store unit tests in + [state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) +- contract tests in + [agent_successor_contract_ahcsitc0.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_successor_contract_ahcsitc0.rs) + +Required work: + +- keep existing coverage green, +- add only the missing contract edges listed in this plan, +- prefer test names that describe the actual authority rule, not implementation trivia. + +## Execution Plan + +### 1. Freeze store-owned authority first + +Make `state_store.rs` the irreversible source of truth before touching docs or operator wording. + +- canonical parent first, +- canonical participant first, +- flat compatibility only for gap fill, +- legacy handle last, +- torn roots discoverable but non-live. + +Required output: + +- store-level regressions proving both canonical-over-legacy and flat-over-legacy participant + precedence, +- a store-level regression proving a selected inactive participant fails closed, +- no production behavior change outside `state_store.rs` yet. + +Do not start the next step until these targeted tests pass. + +### 2. Keep operator surfaces strict + +Update `agents_cmd.rs` only after the store contract is frozen. This step is about projection and +error posture, not new authority logic. + +- no trace-based current liveness, +- no heuristic newest-session selection, +- no hidden live-state promotion for incomplete roots, +- no ambiguity collapse. + +Required output: + +- CLI-surface regression for an active parent whose selected participant exists but is inactive, +- `toolbox status|env` continuing to resolve through the store and fail closed rather than + inventing local recovery, +- `status` continuing to treat trace as additive historical fill only. + +### 3. Lock the write choke point + +Verify runtime persistence still routes through `persist_runtime_snapshots(...)` and store helpers +only. This is a drift guard, not a new behavior branch. + +Required output: + +- a bounded ownership proof, ideally a contract/grep-style regression in the shell contract suite, + that no production caller outside `state_store.rs` writes flat compatibility session, + participant, lease, or handle artifacts directly, +- if `async_repl.rs` needs edits, they stay mechanical. + +### 4. Finish docs while behavior is fresh + +Update active docs only where wording can still mislead an operator or future maintainer. + +This is not fluff. Docs are part of the contract because this slice is mostly about making truth +hard to misread. + +Required output: + +- `TRACE.md` and `USAGE.md` describing the same authority ladder as the runtime, +- bridge-removal gates written once in active docs or this plan, not contradicted elsewhere, +- secondary docs touched only if they disagree. + +### 5. Retire the bridge later, not now + +Write the bridge-removal gates into this plan, but do not execute them here. Bridge retirement is a +separate slice with its own green-light criteria. + +1. explicit migration-only tests are the only remaining flat readers, +2. active docs no longer describe flat files as anything but compatibility input, +3. no required off-repo consumer still depends on flat parent/participant files, +4. no caller-owned write path exists outside the store, +5. legacy handle reads have dedicated replacement coverage. + +### 6. Validate the slice end-to-end + +Do the boring proof before calling this done. + +Run: + +```bash +cargo test -p shell agent_runtime::state_store -- --nocapture +cargo test -p shell agent_successor_contract_ahcsitc0 -- --nocapture +cargo test -p shell -- --nocapture +``` + +Acceptance for this step: + +- all new authority-order and fail-closed regressions are green, +- no doc wording contradicts runtime behavior, +- no new files or abstractions were added without a direct contract reason. + +## Architecture Review + +### Locked architecture decisions + +1. **Keep the store as the only live-state owner.** + - `status` and `toolbox` consume store truth. They do not invent their own precedence logic. + +2. **Keep canonical session-root objects as authority.** + - Flat files exist only because migration is real. + +3. **Keep trace historical.** + - Trace helps explain what happened. It does not decide what is live now. + +4. **Keep fail-closed selection.** + - If there is ambiguity, broken linkage, or corruption, error out. Do not "helpfully" guess. + +5. **Keep dual-write localized.** + - One write choke point is maintainable. Many write owners turn this into a split-brain bug farm. + +6. **Keep bridge removal as a later slice.** + - One migration variable at a time. + +### Architecture acceptance gates + +1. no live operator surface reads current truth from trace alone, +2. no live operator surface treats `handles/*.json` as primary truth, +3. no caller writes flat compatibility copies outside the store, +4. no incomplete root becomes live, +5. no ambiguous multi-candidate state gets silently collapsed. + +### Architecture issues found + +1. The authority contract still lives in too many places at once. + - Fix: keep this plan anchored to store helpers, operator surfaces, and docs only. No new abstraction. + +2. Bridge-removal pressure is the main strategic risk. + - Fix: state explicit gates now, remove later. + +3. Legacy handle fallback is still easy to mentally over-credit. + - Fix: add one more direct regression and keep the docs blunt about last-resort posture. + +## Code Quality Review + +### Implementation guardrails + +1. one live-state owner: `AgentRuntimeStateStore`, +2. one persistence choke point: `persist_runtime_snapshots(...)`, +3. one operator-resolution path for toolbox: `resolve_single_live_session_for_agent(...)`, +4. one live-first then trace-fallback status flow, +5. no duplicated precedence logic in callers, +6. no hidden bridge writes outside the store. + +### Minimal-diff rules + +- prefer clarifying tests and doc wording over new helper layers, +- keep runtime changes small and local, +- if current code already matches the SOW, do not churn it just to look busy, +- if a comment or tiny assertion can prevent future regression, take it. + +### Code quality issues found + +1. The bridge contract is easy to regress because precedence lives in behavior, not one explicit local contract. + - Fix: add direct tests for the remaining edge cases instead of adding a new abstraction. + +2. Secondary docs can silently drift even when runtime is correct. + - Fix: audit only contradictory wording, keep active docs as the primary surface. + +## Error & Rescue Registry + +| Failure point | What goes wrong | Required rescue | +| --- | --- | --- | +| canonical parent exists, child missing | parent-only torn root is misread as live | keep warnings, allow flat child gap-fill only, never treat root as live without valid selected participant | +| multiple active parents for one orchestrator | operator gets pointed at the wrong session | fail closed with operator-readable error | +| active parent points at missing or inactive participant | toolbox endpoint looks plausible but is wrong | fail closed, no endpoint | +| trace has newer history than live store | stale trace resurrects dead session | keep live-first and tombstone suppression rules | +| caller writes flat compatibility file directly | bridge ownership splits across layers | route all writes through store only | +| docs imply handle files are live truth | human operator debugs the wrong artifact | keep active docs explicit and add grep-backed contract tests where helpful | + +## Test Review + +This slice is in much better shape than a typical migration seam. Good. + +The remaining work is not "add tests everywhere." It is "close the last holes that could let +someone weaken the authority boundary later." + +### Code path coverage + +```text +CODE PATH COVERAGE +=========================== +[+] crates/shell/src/execution/agent_runtime/state_store.rs + │ + ├── [★★★ TESTED] persist_writes_canonical_and_flat_compatibility_layouts + ├── [★★★ TESTED] list_sessions_discovers_participant_only_roots_and_excludes_them_from_live_results + ├── [★★★ TESTED] parent_only_torn_roots_degrade_with_warnings_instead_of_failing_discovery + ├── [★★★ TESTED] list_invalidated_participants_across_sources_includes_legacy_fallback_tombstones + ├── [GAP] explicit regression that legacy handle data never outranks canonical or flat participant data + └── [GAP] bounded guard that store persistence remains the only dual-write owner + +[+] crates/shell/src/execution/agents_cmd.rs + │ + ├── [★★★ TESTED] agent_toolbox_env_trace_history_does_not_authorize_active_session + ├── [★★★ TESTED] agent_toolbox_surfaces_prefer_canonical_session_roots_over_flat_compatibility_files + ├── [★★★ TESTED] agent_toolbox_surfaces_fall_back_to_flat_participant_when_canonical_root_is_incomplete + ├── [★★★ TESTED] operator_surfaces_fail_closed_when_multiple_active_parent_candidates_exist + ├── [★★★ TESTED] agent_status_prefers_live_manifest_over_trace_fallback_for_selected_orchestrator + ├── [★★★ TESTED] agent_status_tombstone_suppression_beats_stale_trace_fallback_for_world_member + └── [GAP] explicit operator-surface regression for active parent referencing an inactive selected participant + +[+] crates/shell/src/repl/async_repl.rs + │ + ├── [★★ TESTED] runtime startup/lifecycle paths already persist through store helpers + └── [GAP] narrow regression or grep-style proof that no caller-owned flat-file write path reappears outside the store + +───────────────────────────────── +COVERAGE: 10/14 paths tested (71%) +QUALITY: ★★★: 9 ★★: 1 ★: 0 +GAPS: 4 paths need direct regression coverage +───────────────────────────────── +``` + +### User flow coverage + +```text +USER FLOW COVERAGE +=========================== +[+] substrate agent toolbox env with trace-only history + │ + └── [★★★ TESTED] fails closed, exit 3, no live endpoint + +[+] substrate agent toolbox status/env with canonical parent + canonical participant + │ + └── [★★★ TESTED] canonical live session wins over conflicting flat compatibility files + +[+] substrate agent toolbox status/env with parent-only canonical root + │ + └── [★★★ TESTED] flat compatibility participant fallback remains readable during cutover + +[+] substrate agent status with live canonical orchestrator session + │ + └── [★★★ TESTED] live manifest beats trace fallback + +[+] substrate agent status --scope world with invalidated stale member + │ + └── [★★★ TESTED] tombstone suppression beats stale trace fallback + +[+] operator surfaces with broken parent linkage + │ + ├── [★★★ TESTED] multiple active parents fail closed + ├── [★★★ TESTED] missing active handle id fails closed + ├── [★★★ TESTED] missing selected participant fails closed + └── [GAP] inactive selected participant fail-closed case should be pinned explicitly at the CLI surface + +[+] bridge safety + │ + ├── [★★★ TESTED] canonical + flat dual-write still occurs during cutover + └── [GAP] direct proof that legacy handles never retake live authority + +───────────────────────────────── +COVERAGE: 8/10 flows tested (80%) +GAPS: 2 user-visible flows and 2 code-owner regressions need direct tests +───────────────────────────────── +``` + +### Required test additions by file + +#### `crates/shell/src/execution/agent_runtime/state_store.rs` + +Add direct coverage for exactly these authority rules: + +- canonical participant beats conflicting legacy-handle fallback for the same + `(orchestration_session_id, agent_id)` tuple, +- flat participant beats conflicting legacy-handle fallback when the canonical child is absent but + the canonical parent exists, +- selected participant present but inactive causes `resolve_single_live_session_for_agent(...)` to + fail closed with an operator-meaningful error instead of silently degrading to another candidate. + +#### `crates/shell/tests/agent_successor_contract_ahcsitc0.rs` + +Add direct coverage for exactly these operator-facing rules: + +- `substrate agent toolbox status --json` fails closed when an active parent points at an existing + but inactive selected participant, +- `substrate agent toolbox env --json` fails closed with exit `3` for the same scenario, +- contract/doc assertions only if the final wording still leaves room to misread bridge-removal + posture or live-state authority order. + +#### Write-ownership drift guard + +Add a narrow proof, preferably in the shell contract suite rather than a new REPL abstraction, +that production code outside `state_store.rs` does not write flat compatibility session, +participant, lease, or handle artifacts directly. The guard may be grep-backed if it is precise and +stable. + +### Test commands + +Run at minimum: + +```bash +cargo test -p shell agent_runtime::state_store -- --nocapture +cargo test -p shell agent_successor_contract_ahcsitc0 -- --nocapture +``` + +Then run: + +```bash +cargo test -p shell -- --nocapture +``` + +If docs under `docs/project_management/packs/draft/agent-hub-core-successor-identity-tuple-compatible/` +need edits, keep the existing pack-level contract assertions green as part of the same pass. + +### QA artifact + +The eng-review QA handoff artifact for this slice is: + +[spensermcconnell-feat-session-centric-state-store-eng-review-test-plan-20260501-155804.md](/Users/spensermcconnell/.gstack/projects/atomize-hq-substrate/spensermcconnell-feat-session-centric-state-store-eng-review-test-plan-20260501-155804.md) + +## Failure Modes Registry + +| New codepath | Real production failure | Test covers it? | Error handling exists? | User sees clear error? | Critical gap? | +| --- | --- | --- | --- | --- | --- | +| canonical + flat + legacy read merge | legacy handle fallback silently outranks canonical or flat data | no | partial | no | yes until direct precedence test lands | +| store-owned bridge writes | future caller writes flat compatibility files directly and creates split write ownership | no | no | no | yes until bounded ownership guard lands | +| operator surface linkage validation | active parent references inactive participant and toolbox resolves a stale endpoint | no | yes in store | partial | yes until CLI-surface regression lands | +| trace fallback overlay | stale trace row resurfaces invalidated member | yes | yes | yes | no | +| parent-only torn root | incomplete root is promoted to live | yes | yes | yes | no | +| trace-only toolbox authorization | `toolbox env` succeeds with no current live session | yes | yes | yes | no | + +Critical gap rule: + +If a future diff can let legacy handles outrank canonical/flat runtime state, let callers write +flat bridge files directly, or let toolbox resolve an inactive selected participant, the cutover +contract is not actually frozen. + +## Performance Review + +This is a correctness slice, not a throughput slice. + +Performance rules anyway: + +1. keep live resolution store-owned and bounded, +2. do not add new scans or caches just to make the contract "feel explicit", +3. keep trace fallback additive and late, not on the hot live path, +4. do not build migration infrastructure to solve a documentation problem. + +Performance issues found: + +- 0 material performance issues. + +The real footgun would be adding a new registry or cache to speed up a contract that the existing +store already resolves correctly. That would be classic software. Two hundred lines to avoid +reading the code you already have. + +## DX Review + +This slice has no UI scope. It absolutely has DX scope. + +The primary user here is the developer or operator who is trying to answer one question: +"What is the live orchestrator session right now, and can I trust the answer?" + +### Developer journey map + +| Stage | What the developer is doing | Current friction | Target after this slice | +| --- | --- | --- | --- | +| 1 | Launch runtime and expect live state to persist | low | keep low | +| 2 | Run `substrate agent status --json` | medium, contract is spread across code/tests/docs | one clear live-first contract | +| 3 | Run `substrate agent toolbox status --json` | medium | obvious fail-closed semantics | +| 4 | Run `substrate agent toolbox env --json` | medium | clear exit `3` when dependency unavailable | +| 5 | See conflicting flat files or old trace history | high | know instantly they are compatibility/history only | +| 6 | Debug torn-root or ambiguous parent linkage | medium | one failure model, no heuristics | +| 7 | Audit docs for truth | medium-high | active docs say the same thing as tests | +| 8 | Prepare later bridge removal | high today | explicit gate list | +| 9 | Onboard a new maintainer | high today | they can read one plan and one test file instead of spelunking | + +### Developer empathy narrative + +I ran `substrate agent toolbox env` because I needed the live socket now. + +If it fails, I should not have to wonder whether trace history is secretly good enough, or whether +some flat JSON file in `~/.substrate/run/agent-hub/` is more real than the canonical root. I +should get one clear answer: no live host-scoped orchestrator session, or ambiguity, or broken +parent linkage. + +Then I run `substrate agent status --json` and I should see the same truth model. Same session. +Same precedence. Same fail-closed posture. + +That is the DX win here. Less archaeology. More trust. + +### DX Scorecard + +| Dimension | Score | Notes | +| --- | --- | --- | +| Getting started | 6/10 | CLI names are good, authority contract still too spread out | +| Naming guessability | 8/10 | `status`, `toolbox status`, `toolbox env` are obvious | +| Error messages | 7/10 | fail-closed errors are mostly good, keep them operator-readable | +| Docs findability | 6/10 | active docs improved, still need one sharper contract story | +| Upgrade path safety | 8/10 | bridge stays until explicit retirement gates pass | +| Observability | 8/10 | trace is strong as history, once its boundary stays clear | +| Recovery guidance | 6/10 | torn-root and ambiguity rules need one clearer narrative | +| Escape hatches | 7/10 | compatibility bridge exists, but should stay obviously temporary | + +Overall DX score: **7/10** + +### DX Implementation Checklist + +- keep one canonical authority statement in `TRACE.md`, +- keep one operator-facing statement in `USAGE.md`, +- preserve explicit error text for missing live session, ambiguity, and broken linkage, +- add missing regression coverage for inactive selected participant and legacy-handle precedence, +- state bridge-removal gates where implementers will actually read them. + +### TTHW assessment + +Current TTHW for "understand what is live-state truth here" is about **10-12 minutes** for a new +maintainer. They have to read the SOW, the docs, and the tests. + +Target after this slice: **under 5 minutes**. + +That means: + +- read `PLAN-09.md`, +- read the active docs, +- run the two shell test suites. + +## Worktree Parallelization Strategy + +### Dependency table + +| Step | Modules touched | Depends on | +| --- | --- | --- | +| A. Freeze store-owned authority and add store-level regressions | `crates/shell/src/execution/agent_runtime/` | — | +| B. Tighten operator surfaces and CLI-surface regressions | `crates/shell/src/execution/`, `crates/shell/tests/` | A | +| C. Tighten active docs and secondary doc drift only if needed | `docs/`, `docs/project_management/` | A | +| D. Final validation and cleanup | `crates/shell/`, `docs/` | B, C | + +### Parallel lanes + +- Lane A: Step A +- Lane B: Step B, after Lane A +- Lane C: Step C, after Lane A, in parallel with Lane B once the authority wording from Lane A is + stable +- Lane D: Step D, after Lane B and Lane C merge + +### Execution order + +1. Launch Lane A first and let it own `state_store.rs`. +2. Once Lane A lands or is merged into the worktree base, launch Lane B and Lane C in parallel. +3. Lane B owns runtime/operator code plus `agent_successor_contract_ahcsitc0.rs`. +4. Lane C owns docs only. +5. Launch Lane D after B and C merge to run the full shell test pass and make any mechanical fixups. + +### Conflict flags + +- `state_store.rs` is Lane A only. Do not let Lane B touch it unless Lane A is already merged. +- `agent_successor_contract_ahcsitc0.rs` is Lane B only. Keep store-level tests in Lane A and + CLI-surface tests in Lane B to avoid test-file merge fights. +- `TRACE.md` and `USAGE.md` are Lane C only. +- If a secondary doc requires assertion changes in a contract suite, that work moves back to Lane D. + +### Parallelization verdict + +This slice is sequential at the authority seam, then safely parallel for docs versus operator +surface tightening. + +- **4 lanes total** +- **2 lanes can run in parallel after the authority freeze** +- **2 sequential checkpoints remain non-negotiable: store freeze first, full validation last** + +## Deferred Work + +There is no repo-root `TODOS.md`, so explicit deferrals stay here. + +1. remove flat compatibility reads after explicit migration-only coverage exists +2. remove flat dual-write after no required compatibility readers remain +3. remove legacy `handles/*.json` reads last +4. collapse active and secondary compatibility docs further only if later slices still need them +5. any broader runtime-registry redesign, intentionally out of scope here + +## Definition of Done + +This slice is done only when all of the following are true: + +1. canonical session-root parent and participant records are the documented and implemented + live-state authority, +2. flat parent/participant/lease files are explicitly bridge input/output only during cutover, +3. legacy `handles/*.json` is documented and tested as last-resort compatibility input only, +4. `status` keeps concurrent sessions distinct and uses trace only as bounded fallback, +5. `toolbox status|env` never authorize current liveness from trace, +6. torn roots remain discoverable with warnings but never become live, +7. store-owned persistence remains the only bridge write owner, +8. bridge-removal gates are written down before anyone starts deleting compatibility code. +9. targeted authority tests and the full `cargo test -p shell -- --nocapture` pass are green. + +## Completion Summary + +- Step 0: scope accepted as contract-freeze slice, bridge removal deferred intentionally +- Architecture Review: 3 issues found, all about authority-boundary hardening rather than new infrastructure +- Code Quality Review: 2 issues found, both drift-prevention problems +- Test Review: diagram produced, 4 direct regression gaps identified +- Performance Review: 0 issues found +- DX Review: 7/10 overall, TTHW 10-12 min to target under 5 min +- NOT in scope: written +- What already exists: written +- TODOS.md updates: 0, repo has no root `TODOS.md`, deferrals captured here +- Failure modes: 3 critical gaps flagged +- Outside voice: skipped, `claude` CLI is installed but unauthenticated on 2026-05-01 +- Parallelization: 4 lanes, with docs and operator-surface tightening parallel only after the authority freeze, plus a final validation lane +- Lake Score: complete option chosen for every in-slice decision + + +## Decision Audit Trail + +| # | Phase | Decision | Classification | Principle | Rationale | Rejected | +| --- | --- | --- | --- | --- | --- | --- | +| 1 | Scope | Treat `PLAN-09` as contract freeze plus removal gates, not bridge removal | Mechanical | Pragmatic | The repo still depends on compatibility bridge behavior in tests and docs | Immediate bridge deletion | +| 2 | Authority | Keep canonical session-root parent + participant records as live-state truth | Mechanical | Explicit over clever | The runtime already encodes this, and it is the only maintainable source of truth | Split authority across trace or flat files | +| 3 | Operator safety | Keep `status` and `toolbox` fail closed on ambiguity or broken linkage | Mechanical | Completeness | Guessing "latest" is worse than erroring for control-plane selection | Heuristic newest-session fallback | +| 4 | Persistence | Keep dual-write ownership inside `AgentRuntimeStateStore` only | Mechanical | DRY | One owner prevents write drift and split-brain migration bugs | Caller-owned flat writes | +| 5 | Trace boundary | Keep trace as historical fallback only | Mechanical | Systems over heroes | Historical telemetry should not be allowed to authorize a live endpoint | Trace-as-live-registry behavior | +| 6 | Docs | Treat active docs as part of the runtime contract | Mechanical | Bias toward action | Future regressions here are more likely to come from misread docs than missing code | Leaving the contract buried only in tests | + +## GSTACK REVIEW REPORT + +| Review | Trigger | Why | Runs | Status | Findings | +| --- | --- | --- | --- | --- | --- | +| CEO Review | `/plan-ceo-review` | Scope and strategy | 1 | CLEAR | Accepted the narrow contract-freeze direction, rejected premature bridge removal, and defined the 12-month end state explicitly | +| Codex Review | `/codex review` | Independent 2nd opinion | 0 | SKIPPED | No separate outside-model review run | +| Eng Review | `/plan-eng-review` | Architecture and tests (required) | 1 | CLEAR | Locked the authority ladder to store-owned canonical records, identified 4 direct regression gaps, and preserved minimal diff around existing runtime seams | +| Design Review | `/plan-design-review` | UI/UX gaps | 0 | SKIPPED | No UI scope | + +**UNRESOLVED:** 0 plan-level decision points remain. The remaining work is implementation and the four bounded regression additions already listed. + +**VERDICT:** CEO + ENG CLEARED. `PLAN-09` is ready to execute as the live-state authority +freeze and compatibility-cutover contract on top of the session-centric store work. diff --git a/llm-last-mile/PLAN-10.md b/llm-last-mile/PLAN-10.md new file mode 100644 index 000000000..5adf99769 --- /dev/null +++ b/llm-last-mile/PLAN-10.md @@ -0,0 +1,1112 @@ + + +# PLAN-10: Production World-Scoped Member Runtime Launch + +Source file: [10-member-runtime-launch-seam.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/10-member-runtime-launch-seam.md) +Branch: `feat/session-centric-state-store` +Plan type: shell/runtime launch seam, no UI scope, strong DX scope +Review posture: `/autoplan`-style scope tightening with `/plan-eng-review` structure and rigor +Status: execution-ready after [PLAN-09.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-09.md), with outside voice skipped on 2026-05-01 because `claude` CLI auth is missing + +## Objective + +This slice is not a scheduler. + +It is the first real production seam that opens one world-scoped member runtime through the same +UAA control boundary already used by the host orchestrator path, persists that member under the +existing session-centric runtime store, and keeps status, restart invalidation, and trace output +honest. + +The repo already has the right bones: + +- host orchestrator runtime ownership in + [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) +- participant constructors and live-state invariants in + [crates/shell/src/execution/agent_runtime/session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs) +- session-centric persistence and stale-generation invalidation in + [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) +- live operator projections in + [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs) + +What is missing is the actual producer. + +`PLAN-10` lands that producer in the smallest complete vertical slice: + +1. choose one world-scoped member backend from inventory, fail closed if the choice is ambiguous, +2. lazily launch it from the world-backed REPL command path under an already-live orchestrator + session, +3. persist the member in `allocating`, then only advertise it live after the same retained-control + checks already used by the orchestrator path pass, +4. replace it on world-generation rollover with fresh lineage, +5. prove the result through runtime, restart, status, and trace tests. + +That is enough to move the repo from "fixture-only member truth" to "real member launch exists in +production code" without quietly smuggling in a multi-member control plane. + +## Step 0: Scope Challenge + +### 0A. Repo truth and why this slice exists + +The SOW is directionally right, but parts of its implementation shape are more ambitious than the +codebase needs. + +What the repo already proves today: + +1. `prepare_host_orchestrator_runtime_startup(...)` and + `start_host_orchestrator_runtime_with_prepared(...)` in + [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) + already perform effective-config resolution, policy allowlisting, runtime realizability checks, + gateway construction, participant persistence, and retained-control lifecycle management. +2. `AsyncReplAgentRuntime` plus `RetainedRunControl` in + [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) + are already generic runtime-ownership carriers. They are not host-only in any structural sense. +3. `AgentRuntimeParticipantRecord::new_member_participant(...)` and + `AgentRuntimeParticipantRecord::new_replacement_participant(...)` in + [session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs) + already enforce: + - `role=member` + - `execution.scope=world` + - required `orchestrator_participant_id` + - required `world_id` + `world_generation` + - valid replacement lineage +4. `can_advertise_live()`, `has_valid_ownership()`, and `is_authoritative_live()` in + [session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs) + already encode the exact live-authority rule this slice needs: + no UAA session id, no retained control, no active event stream, or no completion observer means + the participant is not live. +5. `resolve_live_orchestrator_participant(...)`, + `list_live_participants_for_session(...)`, and + `invalidate_stale_world_members_for_session(...)` in + [state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) + already provide the parent lookup and invalidation primitives needed for launch and replacement. +6. `build_status_report(...)`, `build_toolbox_status_report(...)`, and the doctor posture in + [agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs) + already consume member rows correctly once a real runtime producer exists. + +What is still missing: + +1. no real member-selection rule exists, +2. no world-backed REPL command path ensures a member runtime exists before work needs it, +3. no production caller instantiates a member participant outside tests and fixtures, +4. no runtime replacement path creates a fresh member participant on the new world generation, +5. doctor and launch preflight can still drift because member selection truth lives in prose and + scattered helpers rather than one shared rule. + +### 0B. Premise challenge + +Premise check, one by one: + +1. **The first shipped slice should be one host orchestrator plus one in-world member, not a + generalized `/v1/agents` service.** + - Accepted. + - This matches the v1 runtime slice in + [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md). + +2. **The launch seam should reuse the same `run_control(...)` boundary and retained-control live + gating as the host orchestrator path.** + - Accepted. + - Anything weaker would be a fake live row with better spelling. + +3. **This slice needs a new global member-runtime registry or gateway cache.** + - Rejected. + - `build_gateway_for_descriptor(...)` in + [registry.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/registry.rs) + is already descriptor-generic, and `AsyncReplAgentRuntime` already carries the retained + runtime ownership that a member session needs. + - Add a small member-launch helper, not a hidden second control plane. + +4. **The shell should auto-launch arbitrary world members at REPL startup.** + - Rejected. + - The clean first caller is the world-backed command path in + [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) + immediately before `execute_command(...)`. Launch lazily on first need, not on session open. + +5. **The selection rule can stay vague for now.** + - Rejected. + - This is the biggest hidden gap. Without an explicit v1 selection rule, "select an allowed + member backend" just means future code gets to guess. + +6. **Replacement launch belongs in this slice, not later.** + - Accepted. + - Restart invalidation without a replacement producer leaves the live model half-real. + +### 0C. Existing code to reuse + +| Sub-problem | Existing code | Plan | +| --- | --- | --- | +| Unique host orchestrator discovery | `resolve_live_orchestrator_participant(...)` in [state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) | Reuse directly | +| Runtime descriptor shape | `RuntimeSelectionDescriptor` in [validator.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/validator.rs) | Reuse, keep generic | +| Runtime realizability | `validate_runtime_realizability(...)` in [validator.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/validator.rs) | Reuse after wording cleanup | +| Gateway construction | `build_gateway_for_descriptor(...)` in [registry.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/registry.rs) | Reuse, likely no logic change | +| Member record construction | `new_member_participant(...)` and `new_replacement_participant(...)` in [session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs) | Reuse, no parallel record shape | +| Live-state persistence | `persist_participant(...)` and `persist_orchestration_session(...)` in [state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) | Reuse | +| World binding authority | `persist_world_binding_authority(...)` in [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) | Reuse | +| Restart invalidation | `invalidate_stale_world_members_after_binding(...)` in [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) and `invalidate_stale_world_members_for_session(...)` in [state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) | Reuse, then add replacement launch on top | +| Lifecycle ownership gating | `can_advertise_live()` and `is_authoritative_live()` in [session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs) | Reuse exactly | +| Event and snapshot machinery | `translate_wrapper_event(...)`, `build_runtime_message_event(...)`, `persist_runtime_snapshots(...)` in [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) | Reuse, parameterize role-specific messages only | +| Operator projections | `build_status_report(...)` and `build_toolbox_status_report(...)` in [agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs) | Reuse, with required doctor-preflight alignment | + +### 0D. Dream state and 12-month ideal + +```text +CURRENT REPO + │ + ├── host orchestrator runtime launches for real + ├── member participant records exist mostly in fixtures/tests + ├── status/toolbox can project member truth if a producer exists + ├── restart invalidation can tombstone stale members + └── no production member launch or replacement producer + │ + ▼ +THIS PLAN + │ + ├── one explicit v1 member-selection rule + ├── one lazy launch caller in the REPL world-backed command path + ├── one real member runtime using the same retained-control contract + ├── one replacement flow after world-generation rollover + └── one test wall proving status, restart, and trace all consume the real producer + │ + ▼ +12-MONTH IDEAL + │ + ├── multiple world members can exist safely under the same session + ├── selection is explicit and productized + ├── routing/dispatch chooses among members intentionally + ├── restart replacement is routine, not special + └── live operator surfaces and trace stay boring because the authority model never forked +``` + +### 0E. Implementation alternatives + +| Approach | Summary | Effort | Risk | Decision | +| --- | --- | --- | --- | --- | +| A. Lazy, single-member v1 seam in the REPL world-backed command path | One explicit member-selection rule, one real launch path, one replacement path, minimal new abstractions | Medium | Low | **Accepted** | +| B. Auto-launch a member at REPL startup whenever world is enabled | Easier trigger, surprising behavior, launches work the user may never need | Medium | Medium | Rejected | +| C. General member manager with cache, scheduler, or `/v1/agents` surface | Future-looking, spends an innovation token too early | Large | High | Rejected | +| D. Keep fixtures/docs only and defer the real producer again | Avoids code now, keeps the hardest seam imaginary | Small | Unacceptable | Rejected | + +### 0F. Complexity, search, completeness, and distribution checks + +The full diff will still touch more than 8 files because the tests are non-negotiable. That is +fine. + +The production seam must stay bounded anyway: + +1. [crates/shell/src/execution/agent_runtime/validator.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/validator.rs) +2. [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) +3. required doctor-alignment work in + [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs) +4. test files: + - [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) + - [crates/shell/tests/agent_successor_contract_ahcsitc0.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_successor_contract_ahcsitc0.rs) + - [crates/shell/tests/repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs) + - [crates/shell/tests/agent_hub_trace_persistence.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_hub_trace_persistence.rs) + +Default no-touch posture unless the implementation proves otherwise: + +- [crates/shell/src/execution/agent_runtime/registry.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/registry.rs) +- [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) + +`[Layer 1]` wins: + +- reuse `RuntimeSelectionDescriptor` +- reuse `build_gateway_for_descriptor(...)` +- reuse `AsyncReplAgentRuntime` +- reuse `invalidate_stale_world_members_for_session(...)` +- reuse `build_status_report(...)` + +The complete version is still the obvious one: + +- define the v1 selection rule now, +- wire the real launch seam now, +- add the replacement producer now, +- add the regression wall now. + +Distribution check: + +- no new binary, package, container image, or artifact type is introduced here, +- distribution work is not applicable. + +### 0G. What already exists + +1. host orchestrator runtime ownership is already real, +2. member manifest shape and validation are already real, +3. authoritative live-state storage is already real, +4. stale-generation invalidation is already real, +5. status and toolbox projection logic are already real, +6. trace schema already expects pure-agent world rows to carry top-level `world_id` and + `world_generation`, +7. the remaining gap is one live production caller and its replacement path. + +### 0H. NOT in scope + +- a generalized multi-member scheduler +- a public `/v1/agents` service +- host-scoped member product work +- automatic startup-time launch of arbitrary members +- new world ownership or rollback semantics already frozen by + [PLAN-03.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-03.md) + and + [07-world-replacement-ordering-rollback-atomic-metadata.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/07-world-replacement-ordering-rollback-atomic-metadata.md) +- live-state authority-cutover changes already frozen by + [PLAN-09.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-09.md) +- reusing + [crates/shell/src/builtins/world_gateway.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/builtins/world_gateway.rs) + as member-session truth +- new UI + +## Architecture Contract + +### V1 member-selection rule + +This plan has to decide what "select an allowed member backend" means in code. + +V1 rule: + +1. read the effective inventory, +2. collect enabled pure-agent entries that resolve to: + - `execution.scope=world` + - `protocol=uaa.agent.session` + - CLI persistent mode that passes `validate_runtime_realizability(...)` +3. require **exactly one** matching member candidate, +4. if there are zero, skip launch and keep current host-only behavior, +5. if there are more than one, fail closed with an explicit "ambiguous world member selection" + error until a later slice adds a selector, +6. if the selected member backend is not allowlisted by `agents.allowed_backends`, fail closed + before any runtime is started. + +This does two useful things: + +- it gives the runtime a real production choice rule now, +- it prevents this slice from inventing a selector config surface by accident. + +### Launch preflight contract + +A member launch may begin only when all of the following are true: + +1. the REPL is in a world-backed command path, +2. `RuntimeOrchestrationContext` exists, +3. the runtime store resolves exactly one live orchestrator parent via + `resolve_live_orchestrator_participant(...)`, +4. the parent session has authoritative `world_id` plus `world_generation`, +5. the unique selected member passes inventory, capability, runtime-realizability, and policy + allowlist checks, +6. there is no already-live member runtime bound to the same generation for the same selected + member. + +Fail-closed rules: + +- missing parent session: hard error +- inactive/ambiguous parent: hard error +- missing authoritative binding: hard error +- ambiguous member candidate: hard error +- denied backend: hard error +- no synthetic fallback to host execution + +### First production caller + +The first production caller is the `world_session.is_some()` command branch in +[crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) +after `ensure_no_policy_drift(...)` succeeds and immediately before dispatch to either +`exec_world_pty(...)` or `exec_world_line(...)`. + +New internal helper: + +```text +ensure_member_runtime_ready(startup_context, world_session, selected_command_context) +``` + +Behavior: + +1. if no eligible world member exists, no-op, +2. if an eligible member exists and a matching live member runtime is already authoritative-live + on the current `world_generation`, reuse it, +3. otherwise launch a new member runtime lazily and persist it through the store, +4. after world restart, if the stored member runtime is from the stale generation, invalidate old + state first, then create a replacement participant and launch again on the new generation. + +This is the right owner because `async_repl.rs` already holds: + +- the live world session, +- the orchestration context, +- the host orchestrator runtime, +- the restart boundary, +- and the command loop that actually needs the member runtime. + +It also avoids a factual mistake in the earlier wording: `execute_command(...)` is the host-only +path. If this slice hooks there, world-backed commands still skip the member launch seam. + +### Runtime ownership and lifecycle + +Do not add a second member-only runtime holder unless reuse proves impossible. + +Plan: + +1. reuse `AsyncReplAgentRuntime` and `RetainedRunControl` for member sessions too, +2. add a tiny role-specific prepared-launch wrapper and role-specific helper functions, +3. share the existing event/completion/heartbeat/shutdown machinery instead of copying it. + +Minimum new runtime state in the REPL loop: + +```text +host_orchestrator_runtime: Option +member_runtime: Option +selected_member_agent_id: Option +``` + +That is enough for the v1 "one member per session" slice. + +### Lifecycle state machine + +```text +member selected + │ + ▼ +prepare_member_runtime_startup(...) + │ + ├── resolve live parent orchestrator + ├── read authoritative world binding + ├── allocate participant_id + lease_token + run_id + └── new_member_participant(... state=allocating) + │ + ▼ +persist allocating snapshot + │ + ▼ +gateway.run_control(...) + │ + ├── failure before retained ownership + │ └── state=failed, parent remains honest + │ + └── event + completion observers retained + │ + ├── UAA session id observed + ├── control retained + ├── event stream active + └── completion observer retained + │ + ▼ + can_advertise_live() == true + │ + ▼ + state=ready -> running + │ + ├── normal shutdown -> stopping -> stopped + ├── stream/completion loss -> invalidated or failed + └── world restart -> invalidated old, launch replacement +``` + +Critical invariant: + +`new_member_participant(...)` does **not** make the member live. Only the retained UAA ownership +boundary does. This must stay identical to the host orchestrator posture. + +### Dependency graph + +```text +effective inventory + policy + │ + ▼ +validate_member_selection(...) + │ + ▼ +validate_runtime_realizability(...) + │ + ▼ +build_gateway_for_descriptor(...) + │ + ▼ +async_repl ensure_member_runtime_ready(...) + │ + ├── resolve_live_orchestrator_participant(...) + ├── authoritative parent world binding + ├── new_member_participant(...) + ├── persist_participant(...) + └── gateway.run_control(...) + │ + ├── event stream -> translate_wrapper_event(...) + ├── completion observer + └── persist_runtime_snapshots(...) + │ + ▼ + state_store + canonical participant snapshot + │ + ├── agent status + ├── trace persistence + ├── tombstone suppression + └── restart invalidation + replacement +``` + +### Restart replacement flow + +```text +world restart committed + │ + ├── persist_world_binding_authority(...) + ├── invalidate_stale_world_members_after_binding(...) + └── if live member runtime existed for old generation: + │ + ├── allocate replacement participant_id + ├── new_replacement_participant(...) + ├── resumed_from_participant_id = old participant + ├── world_generation = new authoritative generation + └── relaunch through same retained-control path + │ + ├── success -> replacement becomes live + └── failure -> absence remains honest, stale member stays invalidated +``` + +### Error & Rescue Registry + +| Failure | Expected behavior | Persisted state | Operator-facing result | Rescue posture | +| --- | --- | --- | --- | --- | +| No unique world member candidate | fail before launch | no new live member | explicit selection error | fix inventory or add selector in later slice | +| Parent orchestrator missing or ambiguous | fail before launch | no new member | explicit live-parent error | repair parent session first | +| Parent world binding missing | fail before launch | no new member | explicit missing binding error | world must be restarted or rebound | +| Backend denied by policy | fail before launch | no new member | explicit allowlist error | update policy intentionally | +| `run_control(...)` bootstrap failure | mark member failed | `failed` participant with error bucket | clear launch failure | retry only after cause is fixed | +| Event stream ends after liveness | invalidate member | `invalidated` participant + terminal reason | status no longer shows it live | replacement on next use or after restart | +| Restart replacement fails | keep old member invalidated, do not resurrect | stale participant remains tombstoned | honest absence | retry replacement on next eligible command | + +## File Plan + +### Primary implementation surfaces + +#### 1. `validator.rs` + +Add two explicit helpers: + +1. `validate_member_selection(...)` +2. `validate_member_launch_preconditions(...)` or equivalent narrow helper set + +Required behavior: + +- require exactly one enabled world-scoped UAA member candidate, +- require the same conservative UAA capability floor as the first host caller for v1, +- reuse `validate_runtime_realizability(...)` after its user-facing wording is made neutral enough + to talk about "selected runtime" instead of always "selected orchestrator", +- return a normal `RuntimeSelectionDescriptor` so the rest of the runtime stays generic. + +#### 2. `async_repl.rs` + +This is the real seam. + +Add: + +1. lazy `ensure_member_runtime_ready(...)` in the world-backed command branch before + `exec_world_pty(...)` / `exec_world_line(...)`, +2. `prepare_member_runtime_startup(...)`, +3. `start_member_runtime_with_prepared(...)`, +4. `shutdown_member_runtime(...)`, +5. replacement launch after world restart when an old-generation member existed. + +Implementation rule: + +- factor common retained-control lifecycle logic out of the host path enough to avoid copying the + entire startup/completion/shutdown state machine, +- keep role-specific messages explicit, +- do not weaken the host path to make the member path "more generic." + +#### 3. `agents_cmd.rs` + +This file is required for this slice because the current doctor flow is still too loose. + +Current repo truth: + +- `build_doctor_report(...)` validates orchestrator selection and runtime realizability, +- `enabled_world_member_exists(...)` only answers "is there at least one world member," +- that means doctor currently cannot fail closed on ambiguous world-member selection even though the + runtime must. + +Required alignment: + +1. doctor must call the same member-selection helper or the same underlying rule set as runtime + launch preflight, +2. zero eligible world members must preserve host-only behavior, +3. exactly one eligible world member must continue through allowlist and world-boundary checks, +4. more than one eligible world member must fail closed with the same ambiguity posture the launch + path uses. + +`status` and `toolbox` still should not need redesign. They are consumers, not the seam owner. + +### Validation-first no-change gates + +#### `registry.rs` + +Default: no logic change. + +`build_gateway_for_descriptor(...)` is already descriptor-generic. Only touch if the member path +reveals a concrete backend-kind registration gap during real launch. If the selected descriptor +builds a gateway successfully in tests, this file stays untouched. + +#### `state_store.rs` + +Default: no new semantics. + +The store already has: + +- parent resolution, +- authoritative persistence, +- session-local live participant listing, +- stale-generation invalidation. + +Only add a helper if `async_repl.rs` would otherwise have to reimplement an existing store-owned +truth, for example "find the existing live member for this session/generation/agent." Do not move +launch ownership into the store. + +#### `world_gateway.rs` + +No changes in this slice. It is adjacent infrastructure, not the member runtime lifecycle owner. + +## Implementation Sequence + +This is the exact order to build the slice without reopening solved seams. + +### Step 1. Land the shared member-selection contract in `validator.rs` + +Deliverables: + +1. `validate_member_selection(...)` returns the unique eligible world-scoped member descriptor or a + fail-closed reason, +2. helper wording is neutralized so `validate_runtime_realizability(...)` can describe any selected + runtime, not only the orchestrator, +3. the low-level descriptor shape stays generic and reusable by both doctor and launch code. + +Acceptance gate: + +- unit coverage proves zero, one, and many eligible member cases, +- no REPL or doctor behavior changes yet. + +### Step 2. Align `agents_cmd.rs` doctor preflight to the same contract + +Deliverables: + +1. `build_doctor_report(...)` stops using a boolean "any world member exists" posture for launch + truth, +2. doctor reports host-only success when there are zero eligible members, +3. doctor fails closed on ambiguous selection before any world-member boundary claim is made, +4. allowlist and world-boundary checks continue to run only for the unique selected member path. + +Acceptance gate: + +- `substrate agent doctor --json` can distinguish zero vs one vs many world-member candidates, +- doctor and runtime now share one selection truth instead of two similar-but-different ones. + +### Step 3. Land the lazy member launch seam in `async_repl.rs` + +Deliverables: + +1. `ensure_member_runtime_ready(...)` runs in the world-backed command branch after + `ensure_no_policy_drift(...)` and before `exec_world_pty(...)` / `exec_world_line(...)`, +2. the helper reuses the shared selection contract and the existing authoritative parent-world + binding from `RuntimeOrchestrationContext`, +3. the REPL loop gains one `member_runtime: Option` slot and one + `selected_member_agent_id: Option` slot, +4. the member remains `allocating` until retained UAA control, event stream, and completion + observer are all live. + +Acceptance gate: + +- first world-backed command launches lazily, +- second command on the same generation reuses the live member, +- failed preflight exits without a half-live participant. + +### Step 4. Land restart replacement on the same retained-control seam + +Deliverables: + +1. once a new authoritative world binding is persisted and stale members are invalidated, the REPL + either launches a replacement participant for the new generation or leaves honest absence, +2. replacement uses `new_replacement_participant(...)` with fresh `participant_id` and + `resumed_from_participant_id=`, +3. stale member state never regains liveness through trace fallback. + +Acceptance gate: + +- restart with a live member yields either a live replacement on the new generation or a clear + failed replacement outcome, +- stale generation never appears live again. + +### Step 5. Close the test wall before any opportunistic cleanup + +Required test owners: + +1. `async_repl.rs` for launch-state progression and replacement lifecycle, +2. `agent_successor_contract_ahcsitc0.rs` for doctor/status/toolbox contract coverage, +3. `repl_world_first_routing_v1.rs` for world-backed command and restart integration, +4. `agent_hub_trace_persistence.rs` for trace lineage and terminal-row persistence. + +Acceptance gate: + +- all targeted `cargo test -p shell ...` commands below pass, +- no extra refactor or cleanup work starts before the runtime/test wall is green. + +### Step 6. Only then touch docs if the runtime wording actually changed + +This is not a docs-first slice. + +Touch `TRACE.md` or operator wording only if the landed runtime behavior changed user-visible +language or trace examples. If the code and tests align without doc drift, skip doc edits. + +## Code Quality Review + +### Issue 1. Do not duplicate the host retained-control state machine + +The biggest DRY trap is copying 300-400 lines of host lifecycle code and changing three strings. + +Recommendation: + +- extract shared internal runtime-startup and runtime-shutdown helpers, +- keep role-specific preflight separate, +- keep role-specific messages explicit. + +Why: + +- this repo already has the right live-gating logic in one place, +- if host and member lifecycle rules drift, the user gets two subtly different definitions of + "live" in the same product. + +### Issue 2. Do not overload "orchestrator selection" into vague dual-purpose helpers + +`validate_orchestrator_selection(...)` is specific and that is good. + +Recommendation: + +- add parallel member-specific selection helpers, +- keep the low-level descriptor and realizability helpers generic. + +Why: + +- explicit beats clever, +- future contributors should see "select orchestrator" and "select member" as separate product + decisions even if they share lower-level runtime plumbing. + +### Issue 3. Keep the store as the only persistence owner + +Do not let the launch seam write compatibility files, leases, or canonical session roots directly. + +Recommendation: + +- all parent/participant updates continue to flow through existing store helpers and existing + runtime snapshot choke points. + +Why: + +- `PLAN-09` already paid to freeze the authority ladder, +- this slice should consume that contract, not reopen it. + +### Issue 4. Minimize the production touch set + +The source SOW names `registry.rs`, `state_store.rs`, and `agents_cmd.rs` as primary surfaces. +That is too broad for the first implementation pass. + +Recommendation: + +- treat `validator.rs` and `async_repl.rs` as the only default production logic owners, +- touch other files only if tests or alignment require it. + +Why: + +- minimal diff, +- easier verification, +- less chance of spending a week rediscovering a problem the runtime did not actually have. + +## Test Review + +### Test framework detection + +- Runtime: Rust +- Framework: `cargo test` +- Primary package: `shell` +- No LLM prompt/eval suite expansion is required for this slice + +### Code path coverage + +```text +CODE PATH COVERAGE +=========================== +[+] crates/shell/src/execution/agent_runtime/validator.rs + │ + ├── validate_member_selection() + │ ├── [GAP] Zero world members -> no-op path documented and tested + │ ├── [GAP] Exactly one eligible world member -> success + │ ├── [GAP] Multiple eligible world members -> fail closed + │ ├── [GAP] Wrong protocol/scope -> fail closed + │ └── [GAP] Denied backend -> fail closed before launch + │ + └── validate_runtime_realizability() + ├── [★★ TESTED] Host orchestrator CLI persistent checks already covered + └── [GAP] Same checks exercised through member selection path + +[+] crates/shell/src/repl/async_repl.rs + │ + ├── ensure_member_runtime_ready() + │ ├── [GAP] [->E2E] First world-backed command launches member lazily + │ ├── [GAP] Existing live member on same generation is reused + │ └── [GAP] Missing live parent or binding fails before launch + │ + ├── prepare_member_runtime_startup() + │ ├── [GAP] Allocating snapshot persisted with parent linkage + │ └── [GAP] Replacement participant uses fresh lineage on restart + │ + ├── start_member_runtime_with_prepared() + │ ├── [GAP] UAA session id + retained control -> Ready + │ ├── [GAP] Ready -> Running after event flow + │ ├── [GAP] Bootstrap error -> Failed + │ ├── [GAP] Stream loss after liveness -> Invalidated + │ └── [GAP] Shutdown -> Stopped + │ + └── restart replacement hook + ├── [★★ TESTED] Old-generation invalidation behavior already covered with fixtures + ├── [GAP] [->E2E] Real replacement member launches on new generation + └── [GAP] [->E2E] Replacement failure leaves honest absence + +[+] crates/shell/tests/agent_successor_contract_ahcsitc0.rs + │ + ├── agent status projections + │ ├── [★★★ TESTED] Fixture/member rows keep world_id + world_generation + │ ├── [GAP] [->E2E] Real launched member appears from runtime state, not trace fallback + │ └── [★★★ TESTED] Invalidated tombstones suppress stale trace fallback + │ + └── agent doctor + ├── [★★★ TESTED] Allowlist and world-boundary fail-closed posture exists + └── [GAP] Ambiguous world-member selection fails closed + +[+] crates/shell/tests/agent_hub_trace_persistence.rs + │ + ├── host orchestrator runtime trace rows + │ └── [★★★ TESTED] Host path persists runtime-owned pure-agent rows + │ + └── member runtime trace rows + ├── [GAP] [->E2E] Registered/status/terminal rows persist for member runtime + └── [GAP] Replacement lineage persists in trace-compatible fields + +───────────────────────────────── +COVERAGE: 4/19 paths tested (~21%) + Code paths: 2/13 + Integration/user flows: 2/6 +QUALITY: ★★★: 3 ★★: 2 ★: 0 +GAPS: 15 paths need tests (4 need E2E/integration) +───────────────────────────────── +``` + +### User flow coverage + +```text +USER FLOW COVERAGE +=========================== +[+] First world-backed command in REPL + ├── [GAP] [->E2E] Launch member lazily on first need + ├── [GAP] Reuse existing live member on second command + └── [GAP] Missing binding prints clear fail-closed error + +[+] Shared-world restart with live member + ├── [★★ TESTED] Fixture invalidation of stale generation exists + ├── [GAP] [->E2E] Real replacement member becomes live on new generation + └── [GAP] [->E2E] Replacement failure leaves no fake liveness + +[+] Operator inspection + ├── [GAP] `substrate agent doctor --json` reports ambiguous member selection + ├── [GAP] `substrate agent status --json` shows live member from store + └── [GAP] `substrate agent toolbox status --json` stays orchestrator-anchored +``` + +### Missing tests to add to the plan + +1. **`crates/shell/src/repl/async_repl.rs`** + - Add unit/integration-style runtime tests for: + - first lazy member launch success under a live orchestrator + authoritative binding + - launch failure when binding is missing + - launch failure when parent orchestrator is missing/inactive + - ambiguous world-member selection failure + - replacement participant creation after restart with fresh `participant_id` and + `resumed_from_participant_id` + - clean shutdown and unexpected control-stream loss + +2. **`crates/shell/tests/agent_successor_contract_ahcsitc0.rs`** + - Add contract cases for: + - real launched member rows in `agent status --json` + - top-level `world_id` + `world_generation` coming from runtime state + - doctor failure on ambiguous world-member selection + - toolbox remaining anchored to the orchestrator session even when the member is live + +3. **`crates/shell/tests/repl_world_first_routing_v1.rs`** + - Add integration cases for: + - shared-world startup followed by first-command member launch + - shared-world restart causing member replacement on the new generation + - replacement failure leaving absence rather than resurrecting stale liveness + +4. **`crates/shell/tests/agent_hub_trace_persistence.rs`** + - Add trace persistence cases for: + - member `Registered` / `Status` / terminal events + - replacement lineage fields + - terminal member rows staying auditable without becoming live again + +### Test artifact + +The eng-review QA artifact for this plan is: + +[spensermcconnell-feat-session-centric-state-store-eng-review-test-plan-20260501-194949.md](/Users/spensermcconnell/.gstack/projects/atomize-hq-substrate/spensermcconnell-feat-session-centric-state-store-eng-review-test-plan-20260501-194949.md) + +## Failure Modes Registry + +| New codepath | Real production failure | Test covers it? | Error handling exists? | User sees clear error? | Critical gap? | +| --- | --- | --- | --- | --- | --- | +| unique world-member selection | two enabled world-scoped members exist and the runtime guesses | no | not yet | no | yes until ambiguous-selection test and error land | +| lazy member launch preflight | parent orchestrator or world binding is missing but a partial member row is still persisted | no | partial | partial | yes until preflight-fails-before-launch test lands | +| retained-control member liveness | constructor-created member is treated as live before UAA ownership is retained | no | yes in model helpers | no | yes until launch-state progression test lands | +| restart replacement | stale member is invalidated but replacement never starts and a trace row brings it back | partial | partial | no | yes until replacement + suppression integration lands | +| toolbox anchoring | active member causes toolbox to drift away from orchestrator session | no | yes in current design | partial | yes until real-launched-member toolbox regression lands | +| trace persistence | terminal member rows omit lineage or world metadata | no | partial | yes | yes until trace persistence coverage lands | + +Critical gap rule: + +If the implementation can guess among multiple members, persist a half-live member before retained +ownership exists, or let restart replacement drift into stale trace resurrection, this seam is not +actually done. + +## Performance Review + +This slice is still correctness-first, but there are performance footguns worth naming. + +Performance rules: + +1. do not add a global gateway cache just to avoid building one backend wrapper per member launch, +2. do not rescan the world or rerun launch preflight on every command once a matching live member + already exists for the current generation, +3. do not add extra live-state scans to `agent status` or `toolbox`, +4. do not let restart replacement loop over arbitrary historical members when the v1 slice only + manages one selected live member. + +Performance issues found: + +- 0 material throughput issues + +The real performance bug here would be architectural. Building a cache or manager to "speed up" a +single-member launch seam is exactly how a 50-line production need turns into a 500-line future +maintenance problem. + +## DX Review + +This slice has no UI scope. It absolutely has developer/operator scope. + +The user here is the engineer trying to answer: +"If a world member is supposed to be live right now, can I prove that without guessing?" + +### Developer journey map + +| Stage | What the developer is doing | Current friction | Target after this slice | +| --- | --- | --- | --- | +| 1 | Enable one world-scoped member in inventory | medium, selection rule is implied not encoded | explicit unique-member rule | +| 2 | Start a REPL with world enabled | low | keep low | +| 3 | Run the first world-backed command | high, no real member producer exists | lazy member launch on first need | +| 4 | Inspect `substrate agent doctor --json` | medium, preflight is looser than launch reality | doctor matches launch truth | +| 5 | Inspect `substrate agent status --json` | medium, member truth is mostly fixture-based today | live member row comes from runtime state | +| 6 | Restart the shared world | medium-high, invalidation exists but replacement does not | replacement member or honest absence | +| 7 | Inspect `substrate agent toolbox status --json` | medium | unchanged orchestrator anchoring | +| 8 | Debug terminal failure or control-stream loss | medium | persisted member terminal reason is explicit | +| 9 | Extend to future multi-member work | high today | clear boundary: this slice is single-member by design | + +### Developer empathy narrative + +I turned on one world-scoped member because I wanted a real agent running inside the selected +shared world, not another promising test fixture. + +When I run the first world-backed command, the system should either launch that member cleanly or +tell me exactly why it refused. Missing parent, missing world binding, ambiguous selection, denied +backend. Real reasons. No folklore. + +Then I run `substrate agent status --json` and I should see the member there because the runtime +store says it is live, not because trace history looked optimistic. If the world restarted and the +replacement failed, I should see honest absence. Painful maybe. Honest definitely. + +### DX Scorecard + +| Dimension | Score | Notes | +| --- | --- | --- | +| Getting started | 6/10 | inventory works, member launch rule is not yet explicit | +| Naming guessability | 8/10 | `doctor`, `status`, and runtime terms are good | +| Error messages | 7/10 | fail-closed posture exists, but member-specific causes still need first-class wording | +| Docs findability | 6/10 | planning docs exist, runtime truth still needs this seam to become real | +| Upgrade path safety | 8/10 | single-member v1 slice keeps the blast radius small | +| Observability | 8/10 | trace + store model are good once the producer exists | +| Recovery guidance | 7/10 | restart invalidation is already clear, replacement flow needs to match | +| Escape hatches | 6/10 | ambiguity deliberately fails closed until a real selector exists | + +Overall DX score: **7/10** + +### DX Implementation Checklist + +- make member selection explicit and deterministic, +- keep launch errors as problem + cause + fix style messages, +- keep `doctor` aligned with launch truth, +- keep `status` and `toolbox` consuming the existing live-state authority ladder, +- add one trace example or operator note only if the real member launch semantics drift from + current `TRACE.md` wording. + +### TTHW assessment + +Current TTHW for "understand whether a real world member can launch and stay live" is about +**12-15 minutes**. You have to read the seam doc, the host runtime code, the state-store rules, +and the test fixtures to reconstruct the missing piece. + +Target after this slice: **under 7 minutes**. + +That means a maintainer can: + +1. read `PLAN-10.md`, +2. run the targeted shell tests, +3. trust that `doctor`, `status`, and restart behavior all point at the same runtime truth. + +## Cross-Phase Themes + +These themes showed up across scope review, engineering review, and DX review: + +1. **Reuse the existing retained-control runtime seam.** + - The repo already paid for one honest definition of liveness. Use it. + +2. **Define the member-selection rule now.** + - Hidden selection logic is how runtime bugs become "config mysteries." + +3. **Make the first caller lazy and explicit.** + - Launch on the world-backed command path, not at startup and not in a hidden daemon. + +4. **Keep restart replacement honest.** + - Invalidating the stale generation without a replacement producer is only half a product. + +## Worktree Parallelization Strategy + +### Dependency table + +| Step | Modules touched | Depends on | +| --- | --- | --- | +| A. Shared member-selection contract | `crates/shell/src/execution/agent_runtime/` | — | +| B. Doctor alignment | `crates/shell/src/execution/` | A | +| C. Lazy launch + retained-control lifecycle reuse | `crates/shell/src/repl/` | A | +| D. Status/toolbox/trace contract tests | `crates/shell/tests/` | B, C | +| E. Restart replacement integration tests | `crates/shell/tests/`, `crates/shell/src/repl/` | C | +| F. Optional docs alignment + final validation | `docs/`, `crates/shell/` | D, E | + +### Parallel lanes + +- Lane A: shared selection helper work +- Lane B: doctor alignment after Lane A +- Lane C: runtime lifecycle work after Lane A +- Lane D: status/toolbox/trace tests after Lanes B and C +- Lane E: restart replacement integration tests after Lane C, in parallel with Lane D +- Lane F: final validation and optional docs after Lanes D and E + +### Execution order + +1. Launch Lane A first. +2. Merge Lane A into the working base. +3. Launch Lanes B and C in parallel worktrees because they share the selection contract but not the + same primary module. +4. Merge B and C. +5. Launch Lanes D and E in parallel worktrees. +6. Merge D and E, then run Lane F for cleanup and the targeted command wall. + +### Conflict flags + +- `validator.rs` belongs to Lane A. Do not let downstream lanes fork their own selection rules. +- `agents_cmd.rs` belongs to Lane B. +- `async_repl.rs` belongs to Lane C and Lane E. Lane E should only change it for restart-test + fallout, not for fresh architectural decisions. +- `agent_successor_contract_ahcsitc0.rs` and `agent_hub_trace_persistence.rs` belong to Lane D. +- `repl_world_first_routing_v1.rs` belongs to Lane E. + +### Parallelization verdict + +This slice has two clean parallel windows once the shared selection contract lands. + +- **6 lanes total** +- **Window 1:** Lanes B + C in parallel after Lane A +- **Window 2:** Lanes D + E in parallel after Lanes B + C +- **2 sequential checkpoints remain non-negotiable:** shared selection truth first, full + validation last + +## Deferred Work + +There is no repo-root `TODOS.md`, so explicit deferrals stay here. + +1. multi-member selection and scheduling +2. explicit member selector config surface if the product needs more than one world member +3. gateway pooling or caching if repeated member launch cost becomes measurable +4. host-scoped member product work +5. any public control-plane surface beyond the current REPL-owned seam + +## Definition of Done + +This slice is done only when all of the following are true: + +1. one unique eligible world-scoped member can be selected deterministically from inventory, +2. the first world-backed REPL command can lazily launch that member under a live orchestrator + session, +3. the launched member is persisted in `allocating` first and only becomes authoritative-live after + retained UAA ownership is proven, +4. `substrate agent status --json` can surface the live launched member from runtime state with + top-level `world_id` and `world_generation`, +5. `substrate agent toolbox status|env` remains orchestrator-scoped, +6. world restart invalidates the stale member and can launch a replacement participant on the new + generation, +7. failed replacement leaves honest absence, not stale liveness, +8. doctor selection/preflight matches the real runtime launch rules, +9. the targeted shell tests and the four recommended commands below pass. + +### Recommended verification commands + +```bash +cargo test -p shell async_repl -- --nocapture +cargo test -p shell agent_successor_contract_ahcsitc0 -- --nocapture +cargo test -p shell repl_world_first_routing_v1 -- --nocapture +cargo test -p shell agent_hub_trace_persistence -- --nocapture +``` + +## Completion Summary + +- Step 0: scope reduced to one lazy world-member launch seam, no scheduler +- Architecture Review: 4 issues found, all resolved in-plan by making ownership, selection, and the first caller explicit +- Code Quality Review: 4 issues found, all resolved in-plan with a smaller production touch set and shared lifecycle reuse +- Test Review: diagram produced, 15 direct coverage gaps identified +- Performance Review: 0 material performance issues found +- DX Review: 7/10 overall, TTHW 12-15 min to target under 7 min +- NOT in scope: written +- What already exists: written +- TODOS.md updates: 0, repo has no root `TODOS.md`, deferrals captured here +- Failure modes: 6 critical gaps flagged +- Outside voice: skipped, `claude` CLI is installed but unauthenticated on 2026-05-01 +- Parallelization: 6 lanes, with one parallel window after selection-contract work and a second at the test wall +- Lake Score: complete option chosen for every in-slice decision + + +## Decision Audit Trail + +| # | Phase | Decision | Classification | Principle | Rationale | Rejected | +| --- | --- | --- | --- | --- | --- | --- | +| 1 | Scope | Ship one lazy world-member seam, not a scheduler | Mechanical | Pragmatic | The repo needs one real producer before it needs a manager | Scheduler or `/v1/agents` service | +| 2 | Selection | Require exactly one eligible world member for v1 | Taste | Explicit over clever | No selector config exists yet, so guessing would be the bug | Hidden heuristic selection | +| 3 | Caller | Launch lazily from the world-backed REPL command path | Mechanical | Bias toward action | This is the narrowest real production owner with all needed authority already in hand | Startup-time auto launch | +| 4 | Runtime ownership | Reuse `AsyncReplAgentRuntime` and retained-control lifecycle gates | Mechanical | DRY | The host path already encodes the only honest live-state rule | New member-only runtime holder | +| 5 | Persistence | Keep store-owned canonical persistence as the only write path | Mechanical | Systems over heroes | `PLAN-09` already froze the authority ladder | Direct caller writes to compatibility files | +| 6 | Restart | Include replacement launch in the same slice | Mechanical | Completeness | Invalidation without replacement leaves the product half-real | Defer replacement producer | +| 7 | Validation | Keep orchestrator and member selection helpers distinct, while sharing low-level realizability plumbing | Mechanical | Explicit over clever | The product concepts differ even if the runtime descriptor is generic | One vague dual-purpose selector | +| 8 | Tests | Treat real-launch integration coverage as mandatory, not optional cleanup | Mechanical | Boil the lake | This seam is exactly where fixture-only truth stops being enough | Happy-path-only unit coverage | +| 9 | Caller location | Hook member launch into the world-backed REPL branch before `exec_world_pty(...)` / `exec_world_line(...)` | Mechanical | Explicit over clever | That is the real world-mode execution seam; `execute_command(...)` is host-only | Hooking the host-only path | +| 10 | Doctor contract | Make `agents_cmd.rs` a required production surface for this slice | Mechanical | Systems over heroes | Doctor and launch must share one fail-closed selection truth or operators will debug ghosts | Keeping doctor on a boolean world-member posture | +| 11 | Parallelization | Split doctor alignment from REPL lifecycle after the shared validator contract lands | Taste | Pragmatic | The modules are disjoint enough to parallelize safely once selection semantics are frozen | Forcing the whole slice through one serial lane | + +## GSTACK REVIEW REPORT + +| Review | Trigger | Why | Runs | Status | Findings | +| --- | --- | --- | --- | --- | --- | +| CEO Review | `/plan-ceo-review` | Scope and strategy | 1 | CLEAR | Reduced the slice to one lazy world-member launch seam, rejected startup auto-launch and scheduler drift, and made the v1 selection rule explicit | +| Codex Review | `/codex review` | Independent 2nd opinion | 0 | SKIPPED | No separate outside-model review run | +| Eng Review | `/plan-eng-review` | Architecture and tests (required) | 1 | CLEAR | Locked the first caller to the REPL world-backed command path, reused retained-control lifecycle machinery, and identified 15 direct coverage gaps that this plan closes | +| Design Review | `/plan-design-review` | UI/UX gaps | 0 | SKIPPED | No UI scope | + +**UNRESOLVED:** 0 plan-level decision points remain. The remaining work is implementation plus the +targeted runtime, restart, status, and trace tests already enumerated above. + +**VERDICT:** CEO + ENG CLEARED. `PLAN-10` is ready to execute as the first real production +world-scoped member runtime launch seam on top of the session-centric runtime authority already +frozen by [PLAN-09.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-09.md). diff --git a/llm-last-mile/PLAN-11.md b/llm-last-mile/PLAN-11.md new file mode 100644 index 000000000..5f1ac046c --- /dev/null +++ b/llm-last-mile/PLAN-11.md @@ -0,0 +1,777 @@ + + +# PLAN-11: Refreeze Member Dispatch Runtime Descriptor And Complete In-World Cutover + +Source file: [11-in-world-member-dispatch-over-existing-host-world-transport.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/11-in-world-member-dispatch-over-existing-host-world-transport.md) +Supersedes: the blocked earlier draft state of this file, as recorded in: +- [.runs/plan-11/run-state.json](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/plan-11/run-state.json) +- [.runs/plan-11/blocked.json](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/plan-11/blocked.json) +- [.runs/plan-11/session.log](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/plan-11/session.log) + +Branch: `feat/session-centric-state-store` +Plan type: Linux-first transport-backed placement cutover, no UI scope, strong runtime and DX scope +Review posture: `/autoplan`-style consolidation with `/plan-eng-review` rigor, rewritten as a single execution document +Status: execution-ready as one cohesive plan; outside voice remains unavailable on 2026-05-02 because `claude` CLI is installed but unauthenticated + +## Objective + +This slice is not a second agent hub. + +It is the honesty pass on member placement, now updated to incorporate what the blocked +execution proved instead of pretending the earlier transport freeze was already sufficient. + +The accepted freeze work already proved three important things: + +1. the shell can serialize typed member dispatch over the existing `/v1/execute/stream` seam, +2. the shell test harnesses can capture and script that traffic, +3. the old freeze was still missing the one fact `world-agent` actually needs to start the + member with the same UAA retained-control path the shell uses today. + +This consolidated `PLAN-11` does not restart from zero. + +It carries forward the accepted transport and harness work, reopens the unshipped internal +request contract exactly once, adds the resolved runtime descriptor that the blocked run proved +is mandatory, fixes the missing builder export seam, then completes the original runtime cutover +and regression wall. + +The required user outcome is unchanged: + +- when a world-scoped member says it is live on generation `N` of world `W`, +- it is actually running inside `world-agent` on generation `N` of world `W`, +- with real remote retained control, real remote cancel delivery, real replacement behavior, +- and producer-backed `status` plus trace rows that do not lie about placement. + +## Why The Earlier Draft Blocked + +The blocker is not hypothetical. It is captured in the run artifacts above. + +### Accepted carryover from the blocked run + +The blocked parent branch still contains accepted freeze work in: + +- [crates/agent-api-types/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/agent-api-types/src/lib.rs) +- [crates/shell/src/execution/routing/dispatch/world_ops.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs) +- [crates/shell/tests/support/socket.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/support/socket.rs) +- [crates/shell/tests/support/repl_world_agent.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/support/repl_world_agent.rs) + +Those surfaces already prove: + +1. deserialize-time `ExecuteRequest` boundary validation exists, +2. typed `member_dispatch` exists, +3. ready-event capture is pinned to the existing session-handle `AgentEvent` contract, +4. the transport seam is inspectable in shell tests. + +### The two real missing decisions + +The blocked run exposed two real gaps: + +1. **Parent-fixable shell seam** + - [crates/shell/src/execution/routing/dispatch/prelude.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/prelude.rs) + does not re-export the frozen member-dispatch builder, so the shell lane could not consume + the already-frozen transport helper from + [async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs). +2. **Actual hard stop** + - the frozen `member_dispatch` payload carries lineage, backend identity, protocol, run id, + and world identity, + - but it does **not** carry the resolved runtime descriptor from + [RuntimeSelectionDescriptor](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/validator.rs), + - and `world-agent` cannot honestly rebuild the same UAA startup path from `backend_id` plus + `protocol` alone without inventing a second resolver. + +That second gap is the whole game. + +## Step 0: Scope Challenge + +### 0A. What already exists and must be reused + +| Sub-problem | Existing code | Decision | +| --- | --- | --- | +| runtime selection and resolved binary path | [crates/shell/src/execution/agent_runtime/validator.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/validator.rs) | Reuse. This remains the only authoritative selector. | +| gateway-backed UAA startup in the shell | [crates/shell/src/execution/agent_runtime/registry.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/registry.rs), [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) | Reuse the startup model, not the location. | +| session authority, liveness, lineage, invalidation | [crates/shell/src/execution/agent_runtime/session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs), [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) | Reuse exactly. No new store semantics. | +| typed execute transport and capture harness | [crates/agent-api-types/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/agent-api-types/src/lib.rs), [crates/shell/src/execution/routing/dispatch/world_ops.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs), [crates/shell/tests/support/socket.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/support/socket.rs), [crates/shell/tests/support/repl_world_agent.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/support/repl_world_agent.rs) | Reuse, but refreeze with the resolved descriptor added. | +| Linux-first runtime-management rigor in world-agent | [crates/world-agent/src/gateway_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/gateway_runtime.rs) | Reuse as the lifecycle rigor reference. | +| ordinary execute-stream and execute-cancel routing | [crates/world-agent/src/service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs) | Reuse as the branch point only. | + +### 0B. Decision: choose contract refreeze, reject shared-crate extraction + +The blocked run left two plausible unblock paths: + +| Approach | Decision | Why | +| --- | --- | --- | +| Carry the resolved runtime descriptor over `member_dispatch` and let `world-agent` consume it directly | **Accepted** | Smallest honest diff. It removes the second resolver and gives `world-agent` the exact launch facts the shell already resolved. | +| Extract a new shared gateway/UAA startup crate first, then make both shell and `world-agent` consume it | **Rejected for this slice** | Overbuilt. It spends an innovation token, widens the blast radius, and still does not remove the need to carry the resolved launch facts over transport. | + +This is the key architecture decision in the consolidated `PLAN-11`. + +### 0C. Exact refreeze decision + +Keep the existing internal type name `MemberDispatchRequestV1`. + +Do **not** pay a fake `V2` tax for an unshipped internal seam. The earlier freeze was never +released outside this branch. Rewriting the same internal `V1` before the runtime cutover is the +minimal diff. + +Refreeze `member_dispatch` by adding a required nested resolved-runtime payload: + +```text +MemberDispatchRequestV1 + ├── schema_version + ├── orchestration_session_id + ├── participant_id + ├── orchestrator_participant_id + ├── parent_participant_id + ├── resumed_from_participant_id + ├── backend_id + ├── protocol + ├── run_id + ├── world_id + ├── world_generation + └── resolved_runtime: ResolvedMemberRuntimeDescriptorV1 + +ResolvedMemberRuntimeDescriptorV1 + ├── backend_kind + └── binary_path +``` + +Rules: + +1. `backend_kind` is explicit and boring. No backend-kind inference from `backend_id`. +2. `binary_path` is the already-resolved absolute path from the shell selector. +3. the top-level `agent_id` remains authoritative for traces, budgets, and diagnostics. +4. `protocol` remains part of the transport identity contract even though `world-agent` uses + `resolved_runtime.backend_kind` for UAA backend construction. +5. validation remains at the deserialize/parse boundary in + [crates/agent-api-types/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/agent-api-types/src/lib.rs). + +### 0D. Complexity check + +The plan still touches more than 8 files. That is acceptable and still the minimal complete +version. + +The tight production path is: + +1. [crates/agent-api-types/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/agent-api-types/src/lib.rs) +2. [crates/shell/src/execution/routing/dispatch/world_ops.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs) +3. [crates/shell/src/execution/routing/dispatch/prelude.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/prelude.rs) +4. [crates/world-agent/Cargo.toml](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/Cargo.toml) +5. [crates/world-agent/src/service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs) +6. new `crates/world-agent/src/member_runtime.rs` +7. [crates/world-agent/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/lib.rs) +8. [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) + +Everything else is regression wall and harness proof. + +### 0E. Completeness check + +The shortcut path would be: + +- keep the old contract, +- special-case local launch or backend-specific CLI startup in `world-agent`, +- and try to infer readiness from something weaker than the existing session-handle event + contract. + +That shortcut is bad software. + +It would save almost nothing with CC+gstack and it would preserve the exact lie this slice exists +to remove. + +### 0F. NOT in scope + +- a new public `/v1/member/*` or `/v1/agents/*` API family +- a new shared internal crate for gateway/UAA startup +- shell store/schema redesign +- status/doctor/toolbox redesign +- macOS or Windows member-dispatch parity +- auth-bundle redesign beyond continuing to use the existing execute request environment carrier +- broad backend expansion beyond the backends already supported by the shell-owned UAA path +- UI work + +## Architecture Contract + +### Hard invariants + +1. The shell remains the only canonical writer of orchestration session state and participant + state. +2. `world-agent` owns in-world member execution, remote cancel delivery, event streaming, and + completion observation only. +3. World-scoped member launch fails closed. No host fallback. +4. `/v1/execute/stream` and `/v1/execute/cancel` remain the only transport seam. +5. `ExecuteStreamFrame::{Start,Event,Exit,Error}` remain the only stream families. +6. Remote readiness still depends on the existing session-handle event contract unless the parent + explicitly refreezes it before worker lanes open. +7. The shell must represent retained control explicitly as local vs remote. +8. The shell must use an explicit remote-prepared launch shape for members. +9. Linux-first remains explicit. Non-Linux member dispatch fails closed. + +### New contract detail: resolved runtime descriptor + +`world-agent` must not perform runtime selection from agent inventory, effective config, or +host-side shell-only helpers. + +The shell already did that work in +[validate_runtime_realizability(...)](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/validator.rs). + +This plan makes that result transport-visible instead of forcing `world-agent` to guess. + +### New dependency decision + +Add direct `unified-agent-api` dependency to +[crates/world-agent/Cargo.toml](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/Cargo.toml). + +Do not extract a new shared crate in this slice. + +Rationale: + +1. `world-agent` must actually call the same UAA retained-control path the shell already trusts. +2. the only new cross-crate drift risk that matters here is the launch descriptor, and the + transport contract now pins that explicitly. +3. duplicating a tiny backend registration helper is acceptable in-slice; extracting a new crate + is deferred until a third backend or a second consumer makes the duplication real. + +### Builder export decision + +Re-export `build_agent_client_and_member_dispatch_request(...)` through: + +- [crates/shell/src/execution/routing/dispatch/prelude.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/prelude.rs) + +This is parent-owned pre-worker glue. Do not let the shell lane reopen transport modules ad hoc. + +### Data flow + +```text +shell selector + │ + ├── validate_runtime_realizability(...) + │ └── resolves backend_kind + absolute binary_path + │ + ├── build typed ExecuteRequest.member_dispatch + │ └── includes resolved_runtime + │ + ▼ +world-agent execute_stream(member_dispatch) + │ + ├── validate request boundary + ├── validate authoritative world binding + ├── build UAA gateway from resolved_runtime + ├── start run_control(...) inside world-agent + ├── retain remote cancel + events + completion + └── emit Start / Event / Exit / Error + │ + ▼ +shell retained-control consumer + │ + ├── records remote span_id + ├── waits for session-handle Event + ├── marks retained ownership + ├── persists Allocating -> Ready -> Running honestly + └── publishes status and trace from canonical shell state +``` + +### Retained-control model + +The original `PLAN-11` was already right about the shell-side abstraction. This consolidated +version keeps it. + +```text +RetainedRuntimeControl + ├── LocalUaa(RetainedRunControl) + └── RemoteMember(RemoteMemberControl) + +RemoteMemberControl + ├── span_id + ├── stream_task + ├── cancel_path + └── completion_observer +``` + +What changes is that the remote member path now has the data needed to exist for real. + +## File Plan + +### 1. `crates/agent-api-types/src/lib.rs` + +Deliver: + +- `ResolvedMemberRuntimeDescriptorV1` +- required `MemberDispatchRequestV1.resolved_runtime` +- parse-boundary validation for `backend_kind` and `binary_path` +- updated round-trip and invalid-shape tests + +Do not: + +- rename the transport family to `V2` +- add a generic execute target enum + +### 2. `crates/shell/src/execution/routing/dispatch/world_ops.rs` + +Deliver: + +- updated member-dispatch builder carrying `resolved_runtime` +- keep existing world env and trace plumbing + +Do not: + +- move lifecycle logic into this file + +### 3. `crates/shell/src/execution/routing/dispatch/prelude.rs` + +Deliver: + +- re-export of `build_agent_client_and_member_dispatch_request(...)` + +Do not: + +- widen the public routing surface any further than the one helper the shell lane actually needs + +### 4. `crates/shell/tests/support/socket.rs` + +Deliver: + +- updated request capture asserting nested `resolved_runtime` + +### 5. `crates/shell/tests/support/repl_world_agent.rs` + +Deliver: + +- updated typed request capture including nested `resolved_runtime` +- keep the accepted ready-event and cancel scripting contracts unchanged + +### 6. `crates/world-agent/Cargo.toml` + +Deliver: + +- direct `unified-agent-api` dependency needed for real UAA startup in-world + +### 7. `crates/world-agent/src/member_runtime.rs` + +New internal module. + +Deliver: + +- resolved-runtime validation +- small backend registration helper using `backend_kind` plus `binary_path` +- in-world `run_control(...)` startup +- remote cancel/event/completion retention +- terminal cleanup + +### 8. `crates/world-agent/src/service.rs` + +Deliver: + +- `execute_stream(...)` branch for typed member dispatch +- `execute_cancel(...)` delivery to member-dispatch spans + +Do not: + +- turn `service.rs` into the runtime implementation + +### 9. `crates/world-agent/src/lib.rs` + +Deliver: + +- module exposure only as needed for the new internal manager + +### 10. `crates/shell/src/repl/async_repl.rs` + +Deliver: + +- explicit `PreparedMemberDispatch` +- explicit `RetainedRuntimeControl` +- remote member startup through exported typed builder +- same-generation reuse +- honest cancel and replacement convergence + +Do not: + +- change host orchestrator startup behavior +- invent a second lifecycle model + +### 11. Late regression wall + +Primary regression files remain: + +- [crates/shell/tests/repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs) +- [crates/shell/tests/agent_successor_contract_ahcsitc0.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_successor_contract_ahcsitc0.rs) +- [crates/shell/tests/agent_hub_trace_persistence.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_hub_trace_persistence.rs) +- [crates/world-agent/tests/streamed_execute_cancel_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/tests/streamed_execute_cancel_v1.rs) + +## Implementation Sequence + +### Step 1. Import carryover state and explicitly invalidate the old freeze + +Deliverables: + +1. carry forward the accepted A1/A2/A3 work from the blocked run, +2. mark the old `member_dispatch` freeze as superseded inside the new run state, +3. keep the accepted ready-event scripting and harness capture work. + +Acceptance: + +- no worker lane opens before the new contract refreeze lands + +### Step 2. Refreeze the request contract and builder seam + +Deliverables: + +1. add `resolved_runtime` to `MemberDispatchRequestV1` +2. update builder and harness capture +3. re-export the builder through `dispatch/prelude.rs` +4. update compile and contract tests + +Acceptance: + +- the shell lane can consume the frozen builder through the normal routing seam +- the world-agent lane receives the full resolved launch facts without local re-resolution + +### Step 3. Implement the world-agent runtime lane + +Deliverables: + +1. add direct `unified-agent-api` dependency +2. build the UAA gateway from `resolved_runtime` +3. start run control inside `world-agent` +4. retain cancel/event/completion and emit the existing stream family + +Acceptance: + +- no backend-specific stdout parsing +- no local re-resolution from inventory +- no shell-private helper dependency + +### Step 4. Implement the shell runtime lane + +Deliverables: + +1. consume the exported builder +2. add explicit remote-prepared launch shape +3. add explicit remote retained-control carrier +4. wire cancel, startup, and replacement through the remote path + +Acceptance: + +- first world-backed command launches over typed execute-stream +- same-generation reuse still works +- failed preflight still stops before transport + +### Step 5. Land the regression wall + +Deliverables: + +1. cancel reaches member-dispatch spans +2. replacement launch carries fresh `participant_id` and correct `resumed_from_participant_id` +3. `status` and trace remain participant-correct for the real remote producer +4. stale liveness never revives + +Acceptance: + +- all targeted tests below are green in order + +## Test Review + +### Test framework detection + +- Runtime: Rust +- Framework: `cargo test` +- Packages: `agent-api-types`, `world-agent`, `shell` +- No separate LLM eval suite is required for this slice + +### Code path coverage + +```text +CODE PATH COVERAGE +=========================== +[+] crates/agent-api-types/src/lib.rs + │ + ├── member_dispatch request boundary + │ ├── [GAP] resolved_runtime.backend_kind validates + │ ├── [GAP] resolved_runtime.binary_path validates as non-empty absolute path + │ ├── [GAP] old invalid mixed cmd/member_dispatch shapes still fail + │ └── [GAP] ordinary process exec remains unchanged + │ + └── transport round trip + └── [GAP] nested resolved_runtime survives serde round trip + +[+] crates/shell/src/execution/routing/dispatch/world_ops.rs + prelude.rs + │ + ├── builder export seam + │ ├── [GAP] async_repl can import the builder through dispatch prelude + │ └── [GAP] builder carries backend_kind + binary_path correctly + │ + └── harness capture + └── [GAP] typed request capture asserts resolved_runtime + +[+] crates/world-agent/src/member_runtime.rs + service.rs + │ + ├── in-world UAA startup + │ ├── [GAP] resolved runtime builds the correct backend + │ ├── [GAP] unsupported backend_kind fails closed + │ ├── [GAP] missing binary path fails before startup + │ └── [GAP] session-bearing Event still surfaces readiness + │ + ├── cancel path + │ └── [GAP] execute-cancel reaches member span and converges terminally + │ + └── abnormal paths + ├── [GAP] bootstrap failure before readiness -> Failed + ├── [GAP] stream closes before readiness -> Failed + └── [GAP] stream closes after readiness -> Invalidated + +[+] crates/shell/src/repl/async_repl.rs + │ + ├── remote prepared launch + │ ├── [GAP] [->E2E] first world-backed command uses typed member dispatch with resolved runtime + │ ├── [GAP] Allocating persists before remote ownership + │ ├── [GAP] Ready waits for the real session-handle event + │ └── [GAP] Running requires retained ownership, not just any Event + │ + ├── remote retained control + │ ├── [GAP] shutdown uses execute-cancel for member spans + │ └── [GAP] cancel failure converges the member non-live + │ + └── replacement flow + ├── [GAP] [->E2E] world rollover replacement crosses the same transport seam + └── [GAP] [->E2E] replacement failure leaves honest absence + +[+] shell status and trace wall + │ + ├── [GAP] status shows a real remotely launched member + ├── [GAP] trace rows are emitted by the real remote producer + └── [GAP] stale generation never revives live appearance after failure + +───────────────────────────────── +COVERAGE: 0/21 new unblock paths tested yet +GAPS: 21 paths require coverage before the cutover is accepted +───────────────────────────────── +``` + +### User flow coverage + +```text +USER FLOW COVERAGE +=========================== +[+] First world-backed REPL command + ├── [GAP] [->E2E] Shell selects the world member, serializes resolved runtime, and launches remotely + ├── [GAP] Missing parent or world binding fails before transport + └── [GAP] Same-generation reuse does not relaunch + +[+] Live member cancel and shutdown + ├── [GAP] [->E2E] Shutdown issues execute-cancel against the remote span + ├── [GAP] Cancel delivery failure leaves non-live terminal state + └── [GAP] Clean cancel becomes Stopped, not Invalidated + +[+] Shared-world rollover with live member + ├── [GAP] [->E2E] Replacement launch includes resolved runtime and preserved lineage + ├── [GAP] Replacement bootstrap failure leaves no authoritative-live member + └── [GAP] Old generation never regains liveness + +[+] Operator inspection + ├── [GAP] `substrate agent status --json` reflects the real remote producer + └── [GAP] trace rows remain participant-correct after cancel and replacement +``` + +### Required tests to add + +1. `crates/agent-api-types/src/lib.rs` + - add request-boundary tests for nested `resolved_runtime` +2. `crates/shell/tests/support/socket.rs` + - assert captured requests expose `resolved_runtime` +3. `crates/shell/tests/support/repl_world_agent.rs` + - assert member-dispatch scripts receive `resolved_runtime` +4. `crates/world-agent/tests/streamed_execute_cancel_v1.rs` + - add success, unsupported backend, missing binary, cancel, and abnormal termination cases +5. `crates/shell/tests/repl_world_first_routing_v1.rs` + - add first launch, same-generation reuse, replacement launch, and replacement failure cases +6. `crates/shell/tests/agent_successor_contract_ahcsitc0.rs` + - add status correctness for real remote member launch +7. `crates/shell/tests/agent_hub_trace_persistence.rs` + - add producer-backed trace correctness for remote member lifecycle + +### Test artifact + +The eng-review QA artifact for this plan is: + +- [spensermcconnell-feat-session-centric-state-store-eng-review-test-plan-20260502-154419.md](/Users/spensermcconnell/.gstack/projects/atomize-hq-substrate/spensermcconnell-feat-session-centric-state-store-eng-review-test-plan-20260502-154419.md) + +## Failure Modes Registry + +| New codepath | Real production failure | Test covers it? | Error handling exists? | User sees clear error? | Critical gap? | +| --- | --- | --- | --- | --- | --- | +| request refreeze | shell emits typed member dispatch without `resolved_runtime` and world-agent guesses | no | no | no | yes | +| builder export seam | async_repl bypasses the frozen builder and hand-rolls request JSON | no | no | no | yes | +| resolved binary path | shell resolves a binary path that is missing or non-executable inside the Linux world | no | not yet | partial | yes | +| backend-kind startup | world-agent builds the wrong backend for the declared participant | no | not yet | no | yes | +| retained-control split | shell treats remote state like a local cancel handle and leaves live state behind | no | not yet | no | yes | +| remote readiness | readiness advances before the session-handle event arrives | no | partial | no | yes | +| cancel delivery | execute-cancel misses the member span and status keeps advertising live | no | partial | partial | yes | +| replacement wall | replacement launch fails and stale lineage still looks live in status or trace | no | partial | no | yes | + +Critical gap rule: + +If any path can still advertise a world member live without a real remote session handle, a real +remote cancel path, and real remote completion observation, the slice is not done. + +## Performance Review + +This is still correctness-first. + +Performance cautions: + +1. do not introduce a second transport client stack for member dispatch +2. do not re-run member startup if the same-generation remote member is already authoritative-live +3. keep execute-cancel bounded +4. do not add any new full-store scan on the steady-state happy path just to support the refreeze + +Performance verdict: + +- 0 throughput blockers +- 1 structural caution: remote cancel bookkeeping must stay bounded and span-indexed + +## DX Guardrails + +This slice is for developers and operators, not end-users. + +Required DX posture: + +1. the resolved runtime descriptor must be inspectable in tests, not hidden behind opaque helper + state +2. failure messages must name the problem and the fix +3. the plan must reduce "why did the first `PLAN-11` pass block?" from code archaeology to one + plan section and one blocked artifact link +4. proving real placement should remain under 10 minutes by reading plan plus tests + +## Worktree Parallelization Strategy + +### Dependency table + +| Step | Modules touched | Depends on | +| --- | --- | --- | +| A. Carryover import and contract refreeze | `crates/agent-api-types/`, `crates/shell/src/execution/routing/dispatch/`, `crates/shell/tests/support/` | — | +| B. World-agent in-world UAA runtime | `crates/world-agent/` | A | +| C. Shell remote member cutover | `crates/shell/src/repl/`, `crates/shell/tests/` | A | +| D. Regression wall | `crates/shell/tests/`, `crates/world-agent/tests/` | B + C | + +### Parallel lanes + +- Lane A: parent-only contract refreeze +- Lane B: world-agent runtime lane after A +- Lane C: shell remote-cutover lane after A +- Lane D: parent-only regression wall after B + C + +### Execution order + +1. Run Lane A first. +2. After A lands, launch B and C in parallel. +3. Merge B and C into the parent branch. +4. Run D last. + +### Conflict flags + +- B and C both depend on the refrozen request shape. If either lane requests another contract + change, stop and refreeze in the parent. +- C and D both touch shell tests. D stays last. +- The accepted ready-event contract remains shared risk. Do not let either worker rename it + independently. + +### Parallelization verdict + +Same worker cap as the blocked run: exactly `2`. + +That is still the honest cap. There are still only two real runtime seams after the parent-owned +refreeze. + +## Deferred Work + +1. Extract a shared gateway/UAA startup helper only if a third backend or a second consumer makes + the duplication real. +2. Revisit explicit wire version bump only if the internal member-dispatch seam becomes externally + consumed. +3. Revisit secret-safe auth-bundle transport only after this placement seam is boring in + production. +4. Add macOS and Windows parity later. + +## Definition of Done + +This slice is done only when all of the following are true: + +1. the member-dispatch request carries the resolved runtime descriptor +2. `async_repl` consumes the exported builder, not a hand-rolled request +3. `world-agent` starts the member with the real UAA retained-control path inside the world + boundary +4. the shell uses explicit remote retained control for members +5. cancel reaches live member spans through `/v1/execute/cancel` +6. failure before readiness becomes `Failed` +7. stream loss after readiness becomes `Invalidated` +8. replacement after world rollover crosses the same transport seam with correct lineage +9. `status` and trace remain producer-backed and participant-correct +10. no host fallback exists + +## Recommended verification commands + +```bash +cargo test -p agent-api-types -- --nocapture +cargo test -p shell --lib -- --nocapture +cargo test -p world-agent --test streamed_execute_cancel_v1 -- --nocapture +cargo test -p shell --test repl_world_first_routing_v1 -- --nocapture +cargo test -p shell --test agent_successor_contract_ahcsitc0 -- --nocapture +cargo test -p shell --test agent_hub_trace_persistence -- --nocapture +``` + +Compile-only gates must use Cargo's real form: + +```bash +cargo test -p world-agent --test streamed_execute_cancel_v1 --no-run +cargo test -p shell --test repl_world_first_routing_v1 --no-run +``` + +Not: + +```bash +cargo test -p shell --test repl_world_first_routing_v1 -- --no-run +``` + +## Completion Summary + +- Step 0: scope accepted as a continuation unblock, not a restart +- Architecture Review: 4 issues found and resolved in-plan +- Code Quality Review: 2 issues found and resolved in-plan +- Test Review: diagram produced, 21 unblock gaps identified +- Performance Review: 1 structural caution, 0 throughput blockers +- NOT in scope: written +- What already exists: written +- Failure modes: 8 critical gaps called out for the implementation wall +- Outside voice: unavailable because `claude` CLI auth is missing +- Parallelization: 4 execution phases, 1 real parallel window, worker cap stays `2` +- Lake Score: complete option chosen for the unblock instead of shortcut fallback or overbuilt + extraction + + +## Decision Audit Trail + +| # | Phase | Decision | Classification | Principle | Rationale | Rejected | +| --- | --- | --- | --- | --- | --- | --- | +| 1 | Scope | Treat accepted Gate A and Gate B work as carryover, not throwaway | Mechanical | Minimal diff | The blocked run already proved and landed useful freeze work | Restarting `PLAN-11` from zero | +| 2 | Contract | Reopen `MemberDispatchRequestV1` and add `resolved_runtime` | Taste | Completeness | The blocked run proved this fact is mandatory for honest remote startup | Keeping the old freeze and guessing remotely | +| 3 | Versioning | Keep the internal type name `V1` | Taste | Minimal diff | The old freeze was unshipped internal branch state, not a released public protocol | Renaming the whole seam to `V2` | +| 4 | Runtime startup | Add direct `unified-agent-api` dependency to `world-agent` | Mechanical | Explicit over clever | The world side must call the real retained-control path, not fake it | Backend-specific CLI stdout parsing | +| 5 | Reuse | Do not add a new shared startup crate in this slice | Mechanical | Engineered enough | A new crate widens scope and still does not remove transport facts that must be carried | Shared-crate extraction before proving the cutover | +| 6 | Builder seam | Re-export the frozen member-dispatch builder through `dispatch/prelude.rs` | Mechanical | DRY | The shell lane should consume the same builder the transport freeze already pinned | Hand-built request assembly in `async_repl.rs` | +| 7 | Backend semantics | Carry explicit `backend_kind` instead of inferring it from `backend_id` | Mechanical | Explicit over clever | Hidden inference is exactly the wrong move in a seam that already blocked once | Parsing `backend_id` for runtime meaning | +| 8 | State authority | Keep canonical store semantics unchanged | Mechanical | Systems over heroes | The state model is already correct; the problem is placement honesty | Remote canonical-state writes | +| 9 | Parallelism | Keep exact worker cap at `2` | Mechanical | Blast radius instinct | There are still only two honest runtime lanes after the parent-owned refreeze | Artificial third lane for status or docs | + +## GSTACK REVIEW REPORT + +| Review | Trigger | Why | Runs | Status | Findings | +| --- | --- | --- | --- | --- | --- | +| CEO Review | `/plan-ceo-review` | Scope and strategy | 5 | CLEAR | Kept the slice narrow, reused the accepted freeze carryover, and chose contract refreeze over shared-crate extraction | +| Codex Review | `/codex review` | Independent 2nd opinion | 0 | SKIPPED | No separate outside-model review run, and `claude` CLI auth is missing for outside voice | +| Eng Review | `/plan-eng-review` | Architecture and tests (required) | 5 | CLEAR | Locked the resolved-runtime refreeze, the builder export seam, the direct world-agent UAA startup path, and the full regression wall for cancel/replacement/status/trace truth | +| Design Review | `/plan-design-review` | UI/UX gaps | 0 | SKIPPED | No UI scope | + +**UNRESOLVED:** 0 plan-level decision points remain. The remaining work is implementation and +verification only. + +**VERDICT:** CEO + ENG CLEARED. `PLAN-11` is now the single honest execution plan for the +in-world member placement cutover, including the contract refreeze the blocked run proved was +required. diff --git a/llm-last-mile/PLAN-11_5.md b/llm-last-mile/PLAN-11_5.md new file mode 100644 index 000000000..abd67342c --- /dev/null +++ b/llm-last-mile/PLAN-11_5.md @@ -0,0 +1,772 @@ +# PLAN-11_5: Resume PLAN-11 From The Blocked Gate C State + +Source plan: [PLAN-11.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-11.md) +Source SOW: [11-in-world-member-dispatch-over-existing-host-world-transport.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/11-in-world-member-dispatch-over-existing-host-world-transport.md) +Blocked-run evidence: +- [.runs/plan-11/run-state.json](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/plan-11/run-state.json) +- [.runs/plan-11/blocked.json](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/plan-11/blocked.json) +- [.runs/plan-11/session.log](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/plan-11/session.log) +- [.runs/plan-11/quarantined-parent-b1.patch](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/plan-11/quarantined-parent-b1.patch) +- [.runs/plan-11/quarantined-parent-b2.patch](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/plan-11/quarantined-parent-b2.patch) + +Branch: `feat/session-centric-state-store` +Plan type: continuation and unblock plan, Linux-first, no UI scope, strong runtime and DX scope +Review posture: `/autoplan` scope discipline with `/plan-eng-review` depth, rewritten as one execution document +Status: execution-ready continuation pass on 2026-05-02 +Outside voice: unavailable on 2026-05-02 because `claude` CLI is installed but unauthenticated + +## Objective + +`PLAN-11` already did the hard honesty work. + +The request contract refreeze landed. The harness surfaces landed. The blocked run stopped +because the shell worker could not legally consume the frozen builder from +[async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) +through the actual crate-level routing surface, and because Gate C was opened before that +parent-owned visibility bridge was finished. + +`PLAN-11_5` does not reopen the transport contract again. + +It resumes from the accepted Gate A and Gate B carryover, lands the one missing parent-side +visibility hop in +[crates/shell/src/execution/routing.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing.rs), +then reopens exactly one honest two-lane runtime window to finish: + +1. the `world-agent` member runtime manager, +2. the shell remote-member cutover in `async_repl.rs`, +3. the status and trace regression wall that makes the placement claim true. + +The required user outcome is unchanged: + +- when a world-scoped member says it is live on generation `N` of world `W`, +- it is actually running inside `world-agent` on generation `N` of world `W`, +- with real remote cancel delivery, real replacement behavior, and real retained control, +- and `substrate agent status` plus trace rows do not lie about placement. + +## Imported Stop Point + +The current accepted parent state is: + +- Gate A: passed +- Gate B: passed +- Gate C: blocked +- Gate D: blocked + +The blocked run failed for one narrow reason: + +- [dispatch/prelude.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/prelude.rs) + already re-exports `build_agent_client_and_member_dispatch_request(...)`, +- but [routing.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing.rs) + still re-exports only the ordinary builders, +- so `async_repl.rs` could not reach the frozen member-dispatch builder through + `crate::execution::*` without violating the lane boundary. + +The accepted carryover files are frozen for this continuation unless a new regression proves +that the unblock assumptions are wrong: + +- [crates/agent-api-types/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/agent-api-types/src/lib.rs) +- [crates/shell/src/execution/routing/dispatch/world_ops.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs) +- [crates/shell/src/execution/routing/dispatch/prelude.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/prelude.rs) +- [crates/shell/tests/support/socket.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/support/socket.rs) +- [crates/shell/tests/support/repl_world_agent.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/support/repl_world_agent.rs) + +The quarantined worker patches are evidence and implementation hints only. + +They are not accepted branch truth. The parent selectively mines them after the new +`routing.rs` export lands, not before. + +## Frozen Execution Contract + +This section is the part that future implementers are not allowed to reinterpret. + +### Non-negotiable invariants + +1. The shell remains the only canonical writer of orchestration session state and participant + state. +2. `world-agent` owns in-world member execution, remote cancel delivery, event streaming, and + completion observation only. +3. World-scoped member launch fails closed. Host fallback is forbidden. +4. `POST /v1/execute/stream` and `POST /v1/execute/cancel` remain the only transport seam. +5. `ExecuteStreamFrame::{Start,Event,Exit,Error}` remain the only stream families. +6. Remote readiness still depends on the existing session-handle `AgentEvent` contract unless + the parent explicitly refreezes that contract before the worker lanes open. +7. The shell must represent retained control explicitly as local vs remote. +8. The shell must use an explicit remote prepared-launch shape for members. It must not reuse + the host-local orchestrator prepared-launch path as an implementation shortcut. +9. `world-agent` must validate the shell-supplied `world_id` and `world_generation` and reject + mismatches. +10. Linux-first remains explicit. Non-Linux member dispatch paths fail closed in this slice. + +### Frozen request and builder contract + +The `member_dispatch` contract is already correct for this continuation. The shell builder in +[world_ops.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs) +already serializes: + +```text +MemberDispatchRequestV1 + ├── schema_version + ├── orchestration_session_id + ├── participant_id + ├── orchestrator_participant_id + ├── parent_participant_id + ├── resumed_from_participant_id + ├── backend_id + ├── protocol + ├── run_id + ├── world_id + ├── world_generation + └── resolved_runtime + +resolved_runtime + ├── backend_kind + └── binary_path +``` + +Rules: + +1. `resolved_runtime.binary_path` remains absolute. +2. `resolved_runtime.backend_kind` remains explicit. No inference from `backend_id`. +3. Top-level `agent_id` remains authoritative for traces, budgets, and diagnostics. +4. `protocol` remains part of the transport identity contract even though `world-agent` + consumes `resolved_runtime.backend_kind` for backend construction. +5. No `V2` rename. No new request family. No second resolver inside `world-agent`. + +### File ownership and escalation boundary + +| Boundary | Files | Rule | +| --- | --- | --- | +| Parent-owned refreeze carryover | `crates/agent-api-types/src/lib.rs`, `crates/shell/src/execution/routing/dispatch/world_ops.rs`, `crates/shell/src/execution/routing/dispatch/prelude.rs`, `crates/shell/tests/support/socket.rs`, `crates/shell/tests/support/repl_world_agent.rs` | Frozen. Worker lanes do not edit these after Step 1 lands. | +| Parent-owned unblock seam | `crates/shell/src/execution/routing.rs` | Parent lands the missing crate-surface export before workers reopen. | +| Worker-safe world-agent lane | `crates/world-agent/Cargo.toml`, `crates/world-agent/src/service.rs`, `crates/world-agent/src/lib.rs`, `crates/world-agent/src/member_runtime.rs`, `crates/world-agent/tests/streamed_execute_cancel_v1.rs` | Lane B only. | +| Worker-safe shell lane | `crates/shell/src/repl/async_repl.rs`, `crates/shell/tests/repl_world_first_routing_v1.rs` | Lane C only. | +| Parent-owned late regression wall | `crates/shell/tests/agent_successor_contract_ahcsitc0.rs`, `crates/shell/tests/agent_hub_trace_persistence.rs` | Parent only after B and C are integrated. | +| Escalation-only surfaces | `crates/shell/src/execution/agent_runtime/session.rs`, `crates/shell/src/execution/agent_runtime/state_store.rs`, `crates/shell/src/execution/agents_cmd.rs` | Touch only if the regression wall proves the current assumptions false. | + +### Gate exit definitions + +Gate C is not "some tests passed." + +Gate C is passed only when: + +1. the shell can reach the frozen member-dispatch builder through `crate::execution::*`, +2. `world-agent` can launch and cancel member runtimes from the shell-resolved runtime + descriptor, +3. `async_repl.rs` no longer routes world-scoped member startup through + `start_host_orchestrator_runtime_with_prepared(...)`, +4. readiness, replacement, and shutdown all use the remote path honestly. + +Gate D is passed only when: + +1. status truth is green, +2. trace truth is green, +3. stale generation never revives after a failed replacement, +4. the targeted test matrix below passes in order. + +## Step 0: Scope Challenge + +### 0A. What already exists + +| Sub-problem | Existing code | Decision | +| --- | --- | --- | +| typed member-dispatch contract with nested resolved runtime | [crates/agent-api-types/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/agent-api-types/src/lib.rs) | Reuse exactly. Do not reopen `MemberDispatchRequestV1` again in this continuation. | +| world request builder and transport payload shaping | [crates/shell/src/execution/routing/dispatch/world_ops.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs) | Reuse exactly. The builder is already correct. | +| dispatch-module export seam | [crates/shell/src/execution/routing/dispatch/prelude.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/prelude.rs) | Reuse exactly. The missing hop is above this file now. | +| crate-level routing re-export pattern | [crates/shell/src/execution/routing.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing.rs) | Extend. This is the actual missing visibility bridge. | +| shell-owned member lifecycle semantics | [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) | Reuse startup, persistence, invalidation, and status semantics. Change placement only. | +| world-owned long-lived runtime management reference | [crates/world-agent/src/gateway_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/gateway_runtime.rs) | Reuse as the lifecycle pattern, not as the exact API surface. | +| canonical status and trace consumers | [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs), [crates/shell/tests/agent_successor_contract_ahcsitc0.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_successor_contract_ahcsitc0.rs), [crates/shell/tests/agent_hub_trace_persistence.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_hub_trace_persistence.rs) | Reuse. The regression wall only proves the producer is now remote. | + +### 0B. Minimum diff decision + +The smallest honest continuation is: + +1. add the missing member-dispatch builder re-export in `routing.rs`, +2. reopen the same two runtime lanes from `PLAN-11`, +3. keep Gate A and Gate B carryover files frozen, +4. reject any shell-side workaround that reaches into private dispatch modules or hand-rolls + request assembly again. + +This matters because the blocked shell worker already demonstrated the failure mode. The +contract is not the problem anymore. The visibility chain is. + +### 0C. Complexity check + +This continuation still touches more than 8 files, but that is the minimal complete version. + +The production path is: + +1. [crates/shell/src/execution/routing.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing.rs) +2. [crates/world-agent/Cargo.toml](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/Cargo.toml) +3. new `crates/world-agent/src/member_runtime.rs` +4. [crates/world-agent/src/service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs) +5. [crates/world-agent/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/lib.rs) +6. [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) +7. [crates/world-agent/tests/streamed_execute_cancel_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/tests/streamed_execute_cancel_v1.rs) +8. [crates/shell/tests/repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs) +9. [crates/shell/tests/agent_successor_contract_ahcsitc0.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_successor_contract_ahcsitc0.rs) +10. [crates/shell/tests/agent_hub_trace_persistence.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_hub_trace_persistence.rs) + +There is no credible smaller version that still proves placement honesty. + +### 0D. Search-before-building result + +- **[Layer 1]** Reuse the existing `routing.rs` re-export pattern. It already surfaces + `build_agent_client_and_request(...)` and `build_agent_client_and_pending_diff_request(...)`. +- **[Layer 1]** Reuse the existing `/v1/execute/stream` and `/v1/execute/cancel` transport seam. +- **[Layer 1]** Reuse the existing session-handle readiness event contract. +- **[Layer 3]** Do not let `world-agent` invent a second runtime selector. The shell already + resolved the authoritative runtime facts, and the request contract already carries them. + +### 0E. Completeness check + +The shortcut would be to let `async_repl.rs` reach into +`crate::execution::routing::dispatch::world_ops::MemberDispatchTransportRequest` directly or +manually rebuild the transport request again. + +Reject that shortcut. + +It is a code-smell version of the exact drift the Gate B refreeze was supposed to eliminate. + +### 0F. NOT in scope + +- another `MemberDispatchRequestV1` shape change +- a `V2` rename for an unshipped internal seam +- a new public `/v1/member/*` API family +- a new shared startup crate +- shell state-store redesign +- macOS or Windows member-dispatch parity +- status/doctor UX redesign +- docs cleanup before the regression wall is green + +## Architecture Review + +### Findings + +`[P1] (confidence: 10/10) crates/shell/src/execution/routing.rs:40-42 — the crate-level routing surface re-exports the ordinary world request builders, but not build_agent_client_and_member_dispatch_request(...), even though dispatch/prelude.rs:15-19 already exposes it one layer down, so async_repl cannot consume the frozen builder without breaking module boundaries.` + +Recommendation: + +- add `build_agent_client_and_member_dispatch_request` to the existing `pub(crate) use dispatch::{...}` block in `routing.rs` +- continue consuming it from `crate::execution::*`, the same way `async_repl.rs` already imports other routing helpers + +`[P1] (confidence: 10/10) crates/shell/src/repl/async_repl.rs:2706-2711 — member startup still delegates to start_host_orchestrator_runtime_with_prepared(...), so the participant can claim world placement while execution still happens in the shell process.` + +Recommendation: + +- split member launch from host orchestrator launch +- keep local startup for the orchestrator path +- move member startup onto a remote prepared-launch carrier plus remote retained-control carrier + +`[P1] (confidence: 9/10) crates/world-agent/src/service.rs:1415-1438 — execute_cancel only knows how to signal ordinary process-exec spans today, so world-agent still lacks a world-owned retained-control registry for member dispatch spans.` + +Recommendation: + +- add a Linux-only `member_runtime.rs` +- register active member runtimes by `span_id` +- route `member_dispatch` requests through that manager and teach `execute_cancel` to target both process exec spans and member spans + +### Ownership split + +This is the architecture contract in one table. + +| Concern | Owner | Why | +| --- | --- | --- | +| runtime selection, `backend_kind`, `binary_path` | shell | The shell already resolved it. Re-resolving inside `world-agent` would create drift. | +| typed request construction | shell routing layer | The transport contract is already frozen there. | +| in-world `run_control(...)` and cancel retention | `world-agent` | That is the actual placement move this slice exists to land. | +| participant persistence and liveness truth | shell | Status and trace truth must still be rooted in canonical state. | +| replacement decision and stale-generation invalidation | shell | Already landed and must stay authoritative. | +| remote stream forwarding and terminal observation | `world-agent` | Required to make remote placement observable rather than inferred. | + +### Architecture ASCII diagram + +```text +CURRENT BLOCKED STATE +===================== +async_repl.rs + │ + ├── prepare_member_runtime_startup_for_descriptor(...) + ├── start_member_runtime_with_prepared(...) + │ └── still delegates to local host startup + │ + └── cannot legally call the frozen transport builder + because routing.rs does not re-export it + +TARGET CONTINUATION STATE +========================= +crate::execution::routing.rs + └── re-export build_agent_client_and_member_dispatch_request(...) + │ + ▼ +async_repl.rs + ├── prepare remote member dispatch payload + ├── call frozen builder through crate::execution::* + ├── retain remote cancel + stream + completion ownership + └── persist live-state transitions only after remote evidence + │ + ▼ +world-agent member_runtime.rs + ├── validate authoritative world binding + ├── build gateway from resolved_runtime + ├── run_control(...) inside the world + ├── register span_id for cancel delivery + ├── forward Start/Event/Exit/Error + └── service execute_cancel() reaches member spans +``` + +### Liveness boundary diagram + +```text +Allocating + │ + ├── remote execute-stream not established + │ └── fail closed, terminal non-live + │ + ├── remote stream open, but no session-handle event yet + │ └── remain Allocating + │ + └── session-handle event + retained ownership + active stream + └── Ready / Running may be advertised + +Ready / Running + │ + ├── execute-cancel delivered + remote terminal observed + │ └── Stopped / terminal non-live + │ + └── replacement generation selected + └── old generation invalidated, new generation restarts the same proof chain +``` + +## Code Quality Review + +### Findings + +`[P2] (confidence: 9/10) crates/shell/src/execution/routing/dispatch/world_ops.rs:203-265 — MemberDispatchTransportRequest already centralizes the transport shape, so rebuilding those fields ad hoc in async_repl would reintroduce duplication across one of the most fragile seams in the slice.` + +Recommendation: + +- keep the mapping from shell lifecycle state to transport request in one helper path +- if `async_repl.rs` needs a convenience adapter, add it next to the exported builder surface, not as a local struct-construction fork + +`[P2] (confidence: 8/10) .runs/plan-11/quarantined-parent-b2.patch — the blocked shell patch reached into a private routing path because the public export hop was missing, which means blindly applying the patch now would preserve the wrong dependency direction even if it compiles after edits.` + +Recommendation: + +- treat both quarantined patches as reference implementations +- cherry-pick logic, not file hunks +- re-read every changed import and helper boundary after the `routing.rs` export lands + +### Allowed code shape + +This continuation stays boring on purpose. + +1. No new shared crate. +2. No second world transport client. +3. No new status reconstruction path. +4. No duplicate member-dispatch payload construction in `async_repl.rs`. +5. No silent reopening of Gate A/B files from worker lanes. + +If the implementation needs any of those to compile, stop and escalate. That means the plan +assumptions were wrong. + +## Test Review + +### Test framework detection + +- Runtime: Rust +- Framework: `cargo test` +- Packages: `world-agent`, `shell` +- No LLM eval suite is required for this slice + +### Code path coverage + +```text +CODE PATH COVERAGE +=========================== +[+] crates/shell/src/execution/routing.rs + │ + ├── [GAP] crate::execution::* exposes build_agent_client_and_member_dispatch_request(...) + └── [GAP] neighboring world-request exports remain unchanged + +[+] crates/world-agent/src/member_runtime.rs + service.rs + │ + ├── [GAP] success path launches from resolved_runtime inside the world + ├── [GAP] authoritative world mismatch fails before startup + ├── [GAP] unsupported backend or missing binary fails closed + ├── [GAP] execute-cancel reaches member spans and converges terminally + └── [GAP] pre-ready failure and post-ready terminal close produce distinct honest states + +[+] crates/shell/src/repl/async_repl.rs + │ + ├── [GAP] [->E2E] first world-backed command uses typed member dispatch through crate::execution::* + ├── [GAP] Allocating persists before remote ownership is retained + ├── [GAP] Ready/Running require session-handle evidence, not any Event + ├── [GAP] same-generation reuse does not relaunch + ├── [GAP] shutdown uses execute-cancel for remote member spans + └── [GAP] [->E2E] replacement launch preserves fresh participant_id + resumed_from lineage + +[+] shell status and trace wall + │ + ├── [GAP] status reports the real remote producer + ├── [GAP] trace rows remain producer-correct after cancel and replacement + └── [GAP] failed replacement never revives stale generation liveness + +───────────────────────────────── +COVERAGE: 0/16 remaining continuation paths proven +GAPS: 16 paths require coverage before Gate D can pass +───────────────────────────────── +``` + +### User flow coverage + +```text +USER FLOW COVERAGE +=========================== +[+] First world-backed REPL command + ├── [GAP] [->E2E] shell selects the member, serializes resolved runtime, and launches remotely + ├── [GAP] missing parent or world binding fails before transport + └── [GAP] same-generation reuse does not relaunch + +[+] Live member cancel and shutdown + ├── [GAP] [->E2E] shutdown issues execute-cancel against the remote span + ├── [GAP] cancel delivery failure leaves non-live terminal state + └── [GAP] clean cancel becomes Stopped, not Invalidated + +[+] Shared-world rollover with live member + ├── [GAP] [->E2E] replacement launch crosses the same transport seam with lineage preserved + ├── [GAP] replacement bootstrap failure leaves no authoritative-live member + └── [GAP] old generation never regains liveness + +[+] Operator inspection + ├── [GAP] substrate agent status --json reflects the remote producer + └── [GAP] trace rows remain participant-correct after replacement and cancel +``` + +### Required tests to add + +1. `crates/world-agent/tests/streamed_execute_cancel_v1.rs` + - add member-dispatch success, authoritative-world mismatch, missing binary, unsupported + backend, cancel, and abnormal terminal cases + - assert cancel delivery toggles `delivered=true` only after span registration + - assert pre-ready failures do not emit false live evidence +2. `crates/shell/tests/repl_world_first_routing_v1.rs` + - add first launch, same-generation reuse, remote readiness, replacement launch, and + replacement failure cases + - assert Ready/Running cannot appear before the session-handle event +3. `crates/shell/tests/agent_successor_contract_ahcsitc0.rs` + - add status truth checks for a real remotely launched member + - assert stale trace fallback does not beat live remote runtime state +4. `crates/shell/tests/agent_hub_trace_persistence.rs` + - add trace truth checks for remote member launch, cancel, and replacement + - assert lineage survives replacement while terminal predecessors stay auditable + +### Test artifact + +The eng-review QA artifact for this continuation is: + +- [spensermcconnell-feat-session-centric-state-store-eng-review-test-plan-20260502-184500.md](/Users/spensermcconnell/.gstack/projects/atomize-hq-substrate/spensermcconnell-feat-session-centric-state-store-eng-review-test-plan-20260502-184500.md) + +That artifact is still current for this pass because the user-visible flows did not change. +This rewrite only made the implementation contract less ambiguous. + +## Failure Modes Registry + +| New codepath | Real production failure | Test covers it? | Error handling exists? | User sees clear error? | Critical gap? | +| --- | --- | --- | --- | --- | --- | +| routing export bridge | `async_repl.rs` reaches into private routing modules again and drifts away from the frozen builder surface | no | no | no | yes | +| remote launch split | shell persists a world-scoped member as live while startup still runs locally | no | no | no | yes | +| authoritative world binding | world-agent launches against a stale or mismatched world binding | no | not yet | partial | yes | +| resolved runtime path | shell provides a binary path that is missing or not executable inside the Linux world | no | partial | partial | yes | +| remote readiness gate | readiness advances before the session-handle event arrives | no | partial | no | yes | +| cancel path | execute-cancel misses the member span and status keeps advertising live | no | partial | partial | yes | +| replacement wall | replacement fails and stale generation still appears live in status or trace | no | partial | no | yes | + +Critical gap rule: + +If any path can advertise a live world member without a real remote session handle, a real +remote cancel path, and a real remote terminal observer, Gate D fails. + +## Performance Review + +This is still correctness-first. + +Performance cautions: + +1. reuse the existing `AgentClient` transport stack. Do not add a second member-specific client. +2. keep remote stream handling incremental. Do not buffer whole member sessions in memory. +3. preserve same-generation reuse so the steady-state path does not relaunch on every + world-backed command. + +There is no new throughput blocker beyond that. The slice is spending complexity on honesty, not +on speculative optimization. + +## DX Guardrails + +This is a developer tool. Failure messages matter. + +Required error-message posture: + +1. parent-side export failures must name the missing surface, not just emit a generic visibility + error +2. remote launch failures must include `participant_id`, `world_id`, `world_generation`, and + backend kind +3. cancel failures must say whether delivery failed before span registration or after remote + startup +4. replacement failures must say whether the stale generation was already invalidated and whether + the successor ever reached remote readiness + +## Worktree Parallelization Strategy + +### Dependency table + +| Step | Modules touched | Depends on | +| --- | --- | --- | +| P0. Parent visibility bridge | `crates/shell/src/execution/` | — | +| L1. World-agent member runtime lane | `crates/world-agent/` | P0 | +| L2. Shell remote-member lane | `crates/shell/src/repl/`, `crates/shell/tests/` | P0 | +| P1. Parent regression wall | `crates/shell/tests/` | L1, L2 | + +### Worktree ownership + +| Lane | Task id | Worktree | Branch | Allowed files | +| --- | --- | --- | --- | --- | +| Lane A | parent | current checkout | `feat/session-centric-state-store` | `crates/shell/src/execution/routing.rs` | +| Lane B | `task/m11-b1-world-agent-member-manager` | `../substrate-m11-world-agent-member-manager` | `codex/feat-session-centric-state-store-m11-world-agent-member-manager` | `crates/world-agent/**` plus its targeted test | +| Lane C | `task/m11-b2-shell-remote-member-cutover` | `../substrate-m11-shell-remote-member-cutover` | `codex/feat-session-centric-state-store-m11-shell-remote-member-cutover` | `crates/shell/src/repl/async_repl.rs`, `crates/shell/tests/repl_world_first_routing_v1.rs` | +| Lane D | parent | current checkout | `feat/session-centric-state-store` | status + trace regression wall | + +### Parallel lanes + +- Lane A: `P0` only, sequential, parent-owned +- Lane B: `L1`, independent after `P0` +- Lane C: `L2`, independent after `P0` +- Lane D: `P1`, sequential after B + C merge + +### Execution order + +1. Land `P0` in the parent checkout. +2. Run `cargo test -p shell --lib -- --nocapture` to prove the new export surface is valid. +3. Reopen exactly one two-lane worker window: `L1` + `L2`. +4. Merge or manually integrate accepted outputs from both lanes. +5. Run the parent-owned regression wall in `P1`. + +### Merge protocol + +1. Parent lands Step 1 first and tags that commit as the reseed point. +2. Lane B and Lane C rebase or reseed from that exact parent commit before editing. +3. Each lane returns a file-scoped patch plus the exact test command it ran. +4. Parent integrates Lane B first, then Lane C, then reruns the regression wall. +5. If integration requires reopening any frozen Gate A/B file, parent stops the run and updates + the plan before accepting either lane. + +### Conflict flags + +- `L1` and `L2` are parallel only if neither reopens the frozen carryover files from Gate A/B. +- If either lane needs to edit `agent-api-types`, `dispatch/world_ops.rs`, + `dispatch/prelude.rs`, or `routing.rs` after `P0`, stop the run again. That means the unblock + assumptions were wrong. +- If the shell lane needs `session.rs`, `state_store.rs`, or `agents_cmd.rs` to make the remote + path honest, escalate before editing. That expands the blast radius beyond the planned lanes. + +### Parallelization verdict + +One real parallel window remains. Worker cap stays exactly `2`. + +## Deferred Work + +- shared startup-crate extraction, if a third consumer ever makes the duplication real +- non-Linux member-dispatch parity +- docs cleanup after the regression wall is green +- status/doctor UX polish after placement truth is proven + +No new `TODOS.md` entry is required in this continuation. Every deferred item above is an +explicit non-goal of the slice, not a forgotten follow-up. + +## Implementation Sequence + +### Step 1. Parent visibility bridge + +Files: + +- [crates/shell/src/execution/routing.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing.rs) + +Deliverables: + +1. add `build_agent_client_and_member_dispatch_request` to the `routing.rs` re-export block +2. keep Gate A/B carryover files unchanged +3. prove the shell crate can see the builder through `crate::execution::*` + +Acceptance: + +- `async_repl.rs` no longer needs to reference private routing internals +- `cargo test -p shell --lib -- --nocapture` passes before reopening workers + +Stop condition: + +- if Step 1 needs any file besides `routing.rs`, stop and reassess the unblock assumptions + +### Step 2. Reopen the world-agent lane + +Files: + +- [crates/world-agent/Cargo.toml](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/Cargo.toml) +- [crates/world-agent/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/lib.rs) +- [crates/world-agent/src/service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs) +- new `crates/world-agent/src/member_runtime.rs` +- [crates/world-agent/tests/streamed_execute_cancel_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/tests/streamed_execute_cancel_v1.rs) + +Deliverables: + +1. add `agent_api` to `crates/world-agent/Cargo.toml` +2. add Linux-only `member_runtime.rs` +3. route `member_dispatch` requests from `service.rs` +4. register member runtimes by `span_id` +5. extend `execute_cancel` to target member spans +6. fail closed on binding mismatch, missing runtime facts, unsupported backend, or missing binary + +Acceptance: + +- `world-agent` launches the selected backend from the shell-resolved runtime descriptor +- member dispatch fails closed on binding mismatch or missing runtime facts +- remote cancel reports truthfully against the member span registry + +Stop condition: + +- if this lane needs shell runtime-state files, stop and escalate + +### Step 3. Reopen the shell runtime lane + +Files: + +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) +- [crates/shell/tests/repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs) + +Deliverables: + +1. split host-orchestrator and remote-member prepared launch shapes +2. consume the exported builder through `crate::execution::*` +3. add explicit remote retained-control state +4. wire startup, readiness, cancel, and replacement through the remote path +5. keep same-generation reuse on the already-live remote member + +Acceptance: + +- first world-backed member launch crosses typed execute-stream +- Ready/Running require session-handle evidence, not just any event +- same-generation reuse still works +- replacement launches preserve lineage and fail closed honestly + +Stop condition: + +- if this lane needs to reopen `world_ops.rs`, `prelude.rs`, or `agent-api-types`, stop and + escalate + +### Step 4. Parent regression wall + +Files: + +- [crates/shell/tests/agent_successor_contract_ahcsitc0.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_successor_contract_ahcsitc0.rs) +- [crates/shell/tests/agent_hub_trace_persistence.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_hub_trace_persistence.rs) + +Deliverables: + +1. land status truth assertions +2. land trace truth assertions +3. verify stale generation never revives after failed replacement + +Acceptance: + +- all verification commands below are green in order +- the remote producer is observable through status and trace, not just inferred from state + +Stop condition: + +- if status or trace truth needs new production logic outside the planned blast radius, stop and + write the follow-up plan before continuing + +## Definition of Done + +1. `routing.rs` exports the frozen member-dispatch builder at the crate surface +2. `async_repl.rs` launches world-scoped members through the remote path only +3. `world-agent` owns member retained control inside the world +4. `execute_cancel` reaches both process-exec spans and member spans +5. status and trace show the real remote producer +6. same-generation reuse still avoids redundant relaunch +7. replacement preserves lineage and fails closed honestly +8. all targeted tests pass +9. `.runs/plan-11/blocked.json` remains historical evidence, not active truth for the continuation + +## Recommended verification commands + +```bash +cargo test -p shell --lib -- --nocapture +cargo test -p world-agent --test streamed_execute_cancel_v1 -- --nocapture +cargo test -p shell --test repl_world_first_routing_v1 -- --nocapture +cargo test -p shell --test agent_successor_contract_ahcsitc0 -- --nocapture +cargo test -p shell --test agent_hub_trace_persistence -- --nocapture +``` + +Compile-only gates still use Cargo's real form: + +```bash +cargo test -p world-agent --test streamed_execute_cancel_v1 --no-run +cargo test -p shell --test repl_world_first_routing_v1 --no-run +``` + +Recommended parent merge order: + +```bash +1. cargo test -p shell --lib -- --nocapture +2. cargo test -p world-agent --test streamed_execute_cancel_v1 --no-run +3. cargo test -p shell --test repl_world_first_routing_v1 --no-run +4. cargo test -p world-agent --test streamed_execute_cancel_v1 -- --nocapture +5. cargo test -p shell --test repl_world_first_routing_v1 -- --nocapture +6. cargo test -p shell --test agent_successor_contract_ahcsitc0 -- --nocapture +7. cargo test -p shell --test agent_hub_trace_persistence -- --nocapture +``` + +## Completion Summary + +- Step 0: scope accepted as a Gate C/Gate D continuation, not a restart +- Architecture Review: 3 issues found and resolved in-plan +- Code Quality Review: 2 issues found and resolved in-plan +- Test Review: diagram produced, 16 remaining continuation gaps identified +- Performance Review: 3 cautions, 0 throughput blockers +- NOT in scope: written +- What already exists: written +- TODOS.md updates: 0 durable TODOs proposed +- Failure modes: 7 critical gaps called out for the implementation wall +- Outside voice: unavailable because `claude` CLI auth is missing +- Parallelization: 4 execution phases, 1 real parallel window, worker cap stays `2` +- Lake Score: complete option chosen over a private-module shortcut or a new shared crate + + +## Decision Audit Trail + +| # | Phase | Decision | Classification | Principle | Rationale | Rejected | +| --- | --- | --- | --- | --- | --- | --- | +| 1 | Scope | Treat Gate A/B as accepted carryover and plan only the blocked remainder | Mechanical | Minimal diff | The transport contract and harness proof already landed | Restarting `PLAN-11` from zero | +| 2 | Export seam | Add the missing visibility bridge in `crates/shell/src/execution/routing.rs` | Mechanical | Explicit over clever | `async_repl.rs` consumes `crate::execution::*`, not private dispatch modules | Reaching into `routing::dispatch::*` directly | +| 3 | Contract | Keep `MemberDispatchRequestV1` frozen as-is in this continuation | Mechanical | DRY | Reopening the request shape again would create churn without addressing the actual blocker | Another contract refreeze | +| 4 | File boundaries | Freeze Gate A/B files after Step 1 and split the remaining work by lane | Mechanical | Blast radius instinct | The blocked run already proved loose boundaries create false progress | Letting both lanes touch shared transport files | +| 5 | Runtime startup | Keep direct `unified-agent-api` usage in `world-agent` | Mechanical | Engineered enough | The world side must call the real retained-control path now | Shared startup-crate extraction | +| 6 | Shell cutover | Split host-local orchestrator control from remote member control explicitly | Taste | Explicit over clever | The current coupling is exactly what preserves the placement lie | Reusing the host-local runtime path for members | +| 7 | Liveness | Require session-handle evidence before Ready/Running | Mechanical | Systems over heroes | Live-state truth cannot depend on optimistic transport success | Advertising live after any remote `Start` frame | +| 8 | Parallelism | Keep one worker window with cap `2` after the parent refreeze | Mechanical | Blast radius instinct | `world-agent` and `async_repl` are separable only after `routing.rs` lands | Opening more lanes or keeping everything parent-only | + +## GSTACK REVIEW REPORT + +| Review | Trigger | Why | Runs | Status | Findings | +| --- | --- | --- | --- | --- | --- | +| CEO Review | `/plan-ceo-review` | Scope and strategy | 1 | CLEAR | Kept the continuation narrow: fix the export hop, finish the runtime cutover, do not reopen the contract or widen scope | +| Codex Review | `/codex review` | Independent 2nd opinion | 0 | SKIPPED | No separate outside-model review run, and `claude` CLI auth is missing for outside voice | +| Eng Review | `/plan-eng-review` | Architecture and tests (required) | 1 | CLEAR | Locked the real blocker to `routing.rs`, froze file ownership, and defined the exact regression wall for remote launch, cancel, replacement, status, and trace truth | +| Design Review | `/plan-design-review` | UI/UX gaps | 0 | SKIPPED | No UI scope | + +**UNRESOLVED:** 0 plan-level decisions remain. The remaining work is implementation plus +verification only. + +**VERDICT:** CEO + ENG CLEARED. `PLAN-11_5` is the honest continuation plan for the blocked +`PLAN-11` run: land the missing crate-surface export, reopen one two-lane runtime window, and +finish the placement-truth regression wall. diff --git a/llm-last-mile/PLAN-12.md b/llm-last-mile/PLAN-12.md new file mode 100644 index 000000000..f61324963 --- /dev/null +++ b/llm-last-mile/PLAN-12.md @@ -0,0 +1,759 @@ +# PLAN-12: Thaw The Member-Dispatch Request Surface And Finish Remote Placement Truth + +Source plan: [PLAN-11_5.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-11_5.md) +Source SOW: [11-in-world-member-dispatch-over-existing-host-world-transport.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/11-in-world-member-dispatch-over-existing-host-world-transport.md) +Supersedes: the blocked continuation in `PLAN-11_5` for the remaining unshipped work only +Blocked-run evidence: +- [.runs/plan-11_5/run-state.json](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/plan-11_5/run-state.json) +- [.runs/plan-11_5/blocked.json](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/plan-11_5/blocked.json) +- [.runs/task-m11_5-l2-shell-remote-member-cutover/blocked.json](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/task-m11_5-l2-shell-remote-member-cutover/blocked.json) +- [.runs/task-m11_5-l2-shell-remote-member-cutover/worker-report.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/task-m11_5-l2-shell-remote-member-cutover/worker-report.md) +- [.runs/task-m11_5-l1-world-agent-member-runtime/worker-report.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/task-m11_5-l1-world-agent-member-runtime/worker-report.md) +- [.runs/task-m11_5-l1-world-agent-member-runtime/worker-output.patch](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/.runs/task-m11_5-l1-world-agent-member-runtime/worker-output.patch) + +Branch: `feat/session-centric-state-store` +Plan type: Linux-first continuation plan, backend-only, status/trace truth required +Review posture: `/autoplan` scope discipline with `/plan-eng-review` completeness, collapsed into one execution document +Status: execution-ready replacement for `PLAN-11_5` on 2026-05-02 +Outside voice: unavailable on 2026-05-02 because `claude` CLI is installed but unauthenticated + +## Objective + +This is the continuation plan. Not another design reset. + +`PLAN-11_5` already froze the payload contract, preserved the right worker artifacts, and proved +that the remaining blocker is not runtime design. The remaining blocker is that the shell lane +still cannot legally construct the already-frozen member-dispatch request through the sanctioned +`crate::execution::*` surface. + +This plan does exactly three things: + +1. thaw the crate-surface request bridge without reopening the payload contract, +2. finish the real remote member-runtime cutover in `world-agent` and `async_repl.rs`, +3. prove status and trace truth so world-scoped members stop lying about where they run. + +The user-visible outcome is unchanged: + +- when a world-scoped member says it is live on generation `N` of world `W`, +- it is actually running inside `world-agent` on generation `N` of world `W`, +- with real remote cancel delivery, real replacement behavior, and real retained control, +- and `substrate agent status` plus trace rows do not lie about placement. + +## Locked Starting State + +### Accepted carryover + +The accepted parent state entering `PLAN-12` is: + +- Gate A carryover: accepted +- Gate B carryover: accepted +- Gate C: blocked +- Gate D: not reached + +The following branch truth already exists and remains authoritative unless a later proof fails: + +- [crates/agent-api-types/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/agent-api-types/src/lib.rs) +- [crates/shell/src/execution/routing/dispatch/world_ops.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs) +- [crates/shell/src/execution/routing/dispatch/prelude.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/prelude.rs) +- [crates/shell/tests/support/socket.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/support/socket.rs) +- [crates/shell/tests/support/repl_world_agent.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/support/repl_world_agent.rs) +- [crates/shell/src/execution/routing.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing.rs), which already re-exports `build_agent_client_and_member_dispatch_request(...)` + +The preserved worker artifacts are evidence and logic references only. They are not accepted +branch truth until the parent reopens the correct lanes and integrates them. + +### Exact blocker + +The blocked state is concrete: + +1. [crates/shell/src/execution/routing.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing.rs) re-exports + `build_agent_client_and_member_dispatch_request(...)` at lines 40-43. +2. [crates/shell/src/execution/routing/dispatch/prelude.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/prelude.rs) + re-exports the same builder at lines 15-19. +3. [crates/shell/src/execution/routing/dispatch/world_ops.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs) + still keeps `MemberDispatchTransportRequest` crate-private at lines 203-216. +4. [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) + therefore cannot consume the builder through `crate::execution::*` without either reaching + into private modules or recreating the request shape locally. + +That is the block. Not runtime uncertainty. Not transport uncertainty. Not test ambiguity. + +## Frozen Execution Contract + +This section is the part implementers are not allowed to reinterpret. + +### Non-negotiable invariants + +1. The shell remains the only canonical writer of orchestration session state and participant state. +2. `world-agent` owns in-world member execution, remote cancel delivery, event streaming, and completion observation only. +3. World-scoped member launch fails closed. Host fallback is forbidden. +4. `POST /v1/execute/stream` and `POST /v1/execute/cancel` remain the only transport seam. +5. `ExecuteStreamFrame::{Start,Event,Exit,Error}` remain the only stream families. +6. Remote readiness still depends on the existing session-handle `AgentEvent` contract. +7. The shell must represent retained control explicitly as local vs remote. +8. The shell must use an explicit remote prepared-launch shape for members. It must not reuse the host-local orchestrator prepared-launch path as a shortcut. +9. `world-agent` must validate the shell-supplied `world_id` and `world_generation` and reject mismatches. +10. Linux-first remains explicit. Non-Linux member-dispatch paths fail closed in this slice. + +### Frozen payload, thawed crate-surface bridge + +The payload contract stays frozen, but the parent is now explicitly authorized to thaw the +crate-surface request bridge before reopening lanes. The plan chooses a preferred fix, direct +re-export of `MemberDispatchTransportRequest` through the allowed crate surface, and +pre-authorizes one fallback, a sanctioned adapter helper, so the run cannot block again on the +same boundary mistake. + +That means: + +- `MemberDispatchRequestV1` does not change. +- `resolved_runtime` does not change. +- `resolved_runtime.binary_path` stays absolute. +- `resolved_runtime.backend_kind` stays explicit. No inference from `backend_id`. +- top-level `agent_id` remains authoritative for traces, budgets, and diagnostics. +- `protocol` remains part of the transport identity contract. + +#### Preferred bridge, default + +Re-export `MemberDispatchTransportRequest` through the same crate-level surface that already +exports `build_agent_client_and_member_dispatch_request(...)`. + +Required shape: + +1. re-export the type from `dispatch/prelude.rs`, +2. re-export the type from `routing.rs`, +3. leave request serialization behavior in `world_ops.rs` unchanged. + +#### Authorized fallback bridge + +If direct type export proves to be the wrong seam after code review, the parent is pre-authorized +to add one crate-local adapter helper in `world_ops.rs`, then re-export that helper through +`dispatch/prelude.rs` and `routing.rs`. + +Hard rules: + +- the fallback may change visibility and helper shape only, +- it may not change serialized payload fields, +- it may not change `MemberDispatchRequestV1`, +- it may not add a second request-construction path inside `async_repl.rs`, +- it may not move runtime selection into `world-agent`. + +### File authority and escalation boundary + +| Boundary | Files | Rule | +| --- | --- | --- | +| Frozen payload contract | `crates/agent-api-types/src/lib.rs` | No request-shape changes. | +| Parent-owned bridge seam | `crates/shell/src/execution/routing.rs`, `crates/shell/src/execution/routing/dispatch/prelude.rs` | Parent may export the type or adapter. | +| Parent-owned surface-only fallback | `crates/shell/src/execution/routing/dispatch/world_ops.rs` | Parent may add visibility/helper glue only if the preferred bridge is rejected after code review. No payload-schema edits. | +| Worker-safe world-agent lane | `crates/world-agent/Cargo.toml`, `crates/world-agent/src/lib.rs`, `crates/world-agent/src/service.rs`, `crates/world-agent/src/member_runtime.rs`, `crates/world-agent/tests/streamed_execute_cancel_v1.rs` | Lane B only. | +| Worker-safe shell lane | `crates/shell/src/repl/async_repl.rs`, `crates/shell/tests/repl_world_first_routing_v1.rs` | Lane C only. | +| Parent-owned regression wall | `crates/shell/tests/agent_successor_contract_ahcsitc0.rs`, `crates/shell/tests/agent_hub_trace_persistence.rs` | Parent only after B and C integrate. | +| Escalation-only surfaces | `crates/shell/src/execution/agent_runtime/session.rs`, `crates/shell/src/execution/agent_runtime/state_store.rs`, `crates/shell/src/execution/agents_cmd.rs` | Touch only if the regression wall proves current assumptions false. | + +### Stop and escalate conditions + +Stop the run and write blocked state again only if one of these becomes true: + +1. the parent bridge step requires changing `crates/agent-api-types/src/lib.rs`, +2. the parent bridge step requires changing the serialized `member_dispatch` payload, +3. the shell lane still cannot construct the request through `crate::execution::*` after the parent bridge lands, +4. either worker lane needs to edit the other lane's files, +5. the world-agent lane needs a second runtime selector or backend inference from `backend_id`, +6. status or trace truth requires unplanned production logic outside the escalation-only surfaces, +7. a third independent worker lane becomes necessary. + +## Step 0: Scope Challenge + +### 0A. What already exists + +| Sub-problem | Existing code | Decision | +| --- | --- | --- | +| typed `member_dispatch` payload with resolved runtime | [crates/agent-api-types/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/agent-api-types/src/lib.rs), [crates/shell/src/execution/routing/dispatch/world_ops.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs) | Reuse exactly. Do not reopen payload shape. | +| builder function already visible at crate surface | [crates/shell/src/execution/routing.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing.rs) | Keep. The missing piece is request construction, not function visibility. | +| request-construction type lives one layer too low | [crates/shell/src/execution/routing/dispatch/world_ops.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs) | Bridge it. Do not duplicate it. | +| shell-owned member lifecycle semantics | [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) | Reuse persistence, invalidation, lineage, and status semantics. Change placement only. | +| world-owned retained-control lifecycle pattern | [crates/world-agent/src/gateway_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/gateway_runtime.rs) | Reuse as a lifecycle pattern, not as a second owner model. | +| status and trace truth consumers | [crates/shell/tests/agent_successor_contract_ahcsitc0.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_successor_contract_ahcsitc0.rs), [crates/shell/tests/agent_hub_trace_persistence.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_hub_trace_persistence.rs) | Reuse. The regression wall proves the producer is now actually remote. | + +### 0B. Minimum diff decision + +The smallest honest diff is: + +1. thaw the request-construction bridge only, +2. keep the payload contract frozen, +3. reopen the same two runtime lanes, +4. integrate the preserved `world-agent` lane work only after the shell lane is unblocked, +5. finish the status and trace regression wall. + +Anything smaller is fake progress. Anything larger is unnecessary scope. + +### 0C. Complexity, completeness, and search check + +This continuation still touches more than 8 files. That is justified and still minimal. + +The production path is: + +1. `crates/shell/src/execution/routing.rs` +2. `crates/shell/src/execution/routing/dispatch/prelude.rs` +3. optional surface-only bridge helper in `crates/shell/src/execution/routing/dispatch/world_ops.rs` +4. `crates/world-agent/Cargo.toml` +5. `crates/world-agent/src/lib.rs` +6. `crates/world-agent/src/service.rs` +7. `crates/world-agent/src/member_runtime.rs` +8. `crates/shell/src/repl/async_repl.rs` +9. `crates/world-agent/tests/streamed_execute_cancel_v1.rs` +10. `crates/shell/tests/repl_world_first_routing_v1.rs` +11. `crates/shell/tests/agent_successor_contract_ahcsitc0.rs` +12. `crates/shell/tests/agent_hub_trace_persistence.rs` + +Search-before-building result: + +- **[Layer 1]** Keep using the existing `crate::execution::*` export pattern instead of adding another import seam. +- **[Layer 1]** Keep using `/v1/execute/stream` and `/v1/execute/cancel` only. +- **[Layer 1]** Keep using the existing session-handle readiness event as the live gate. +- **[Layer 3]** Treat the bridge as crate-surface plumbing, not as contract refreeze. The payload already carries the authoritative runtime facts. + +Completeness decision: + +- private-module reach-in from `async_repl.rs`: rejected +- local reconstruction of `MemberDispatchTransportRequest`: rejected +- second ad hoc builder near the REPL: rejected + +Those shortcuts save minutes and permanently preserve drift in one of the most fragile seams in +the slice. + +### 0D. Distribution check + +No new distributable artifact type is introduced here. + +This slice changes runtime truth inside existing crates only. No new binary, package, container, +or publish pipeline is required. + +### 0E. NOT in scope + +- another `MemberDispatchRequestV1` shape change +- a `V2` rename for an unshipped internal seam +- a new public `/v1/member/*` or `/v1/agents/*` API family +- a new shared startup crate +- shell state-store redesign +- macOS or Windows member-dispatch parity +- status or doctor UX redesign +- docs cleanup before the regression wall is green + +## Architecture Review + +### Findings + +`[P1] (confidence: 10/10) crates/shell/src/execution/routing.rs:40-43 + crates/shell/src/execution/routing/dispatch/prelude.rs:15-19 + crates/shell/src/execution/routing/dispatch/world_ops.rs:203-216 — the crate-level surface exports the member-dispatch builder but not the request-construction surface needed to call it, so the shell cutover is blocked by module boundaries, not by runtime design.` + +Recommendation: + +- prefer direct type re-export through `prelude.rs` and `routing.rs` +- if that feels wrong after code review, add one sanctioned adapter in `world_ops.rs` +- do not reopen the payload contract + +`[P1] (confidence: 10/10) crates/shell/src/repl/async_repl.rs:2624-2819 — member startup still flows through preparation helpers that end at host-local runtime launch and shutdown helpers, so world-scoped members can still claim remote placement while control remains rooted in the shell process.` + +Recommendation: + +- split host-orchestrator startup from remote member startup explicitly +- keep local startup for the orchestrator path only +- move member startup, cancel retention, and completion observation onto the remote path + +`[P1] (confidence: 9/10) crates/world-agent/src/service.rs:1415-1438 — execute_cancel only knows the ordinary process-exec span registry today, so remote member spans still need a world-owned retained-control registry and cancel path.` + +Recommendation: + +- add Linux-only `member_runtime.rs` +- register member spans for cancel delivery +- route remote member shutdown through the same truthy terminal path used for process exec + +### Ownership split + +| Concern | Owner | Why | +| --- | --- | --- | +| runtime selection, backend kind, binary path | shell | The shell already resolved them. Re-resolving inside `world-agent` would create drift. | +| request payload construction | shell routing layer | The transport contract already lives there. | +| in-world `run_control(...)`, cancel delivery, completion observation | `world-agent` | That is the placement move this slice exists to land. | +| participant persistence, status truth, stale-generation invalidation | shell | Canonical state stays in the shell. | +| replacement decisions and lineage | shell | Already landed. Must remain authoritative. | +| remote event forwarding and terminal observation | `world-agent` | Required to make placement observable rather than inferred. | + +### Architecture ASCII diagram + +```text +CURRENT BLOCKED STATE +===================== +crate::execution::* + │ + ├── build_agent_client_and_member_dispatch_request(...) + │ + └── cannot expose the request-construction surface needed by async_repl.rs + │ + ▼ +async_repl.rs + ├── can prepare member runtime state + ├── cannot legally construct the transport request + └── still delegates member startup to host-local runtime launch + +TARGET PLAN-12 STATE +==================== +crate::execution::* + ├── build_agent_client_and_member_dispatch_request(...) + └── MemberDispatchTransportRequest OR one sanctioned adapter + │ + ▼ +async_repl.rs + ├── prepare remote member dispatch request + ├── launch over /v1/execute/stream + ├── retain remote cancel + event + completion ownership + └── persist live-state transitions only after remote evidence + │ + ▼ +world-agent member_runtime.rs + ├── validate authoritative world binding + ├── build gateway from resolved_runtime + ├── run_control(...) inside the world + ├── register span_id for execute-cancel + ├── forward Start / Event / Exit / Error + └── close terminally and honestly on cancel or failure +``` + +### Liveness boundary + +```text +Allocating + │ + ├── request not constructible at crate surface + │ └── stop before launch, non-live + │ + ├── remote stream established, no session-handle event yet + │ └── remain Allocating + │ + └── remote stream + retained control + session-handle event + └── Ready / Running may be advertised + +Ready / Running + │ + ├── execute-cancel delivered and remote terminal observed + │ └── Stopped / terminal non-live + │ + └── replacement generation selected + └── stale generation invalidated, successor restarts the proof chain +``` + +## Code Quality Review + +### Findings + +`[P2] (confidence: 9/10) crates/shell/src/execution/routing/dispatch/world_ops.rs:203-265 — MemberDispatchTransportRequest already centralizes the transport shape, so rebuilding those fields in async_repl.rs would create the exact duplication the carryover refreeze was supposed to remove.` + +Recommendation: + +- keep request assembly single-sourced near the builder surface +- let `async_repl.rs` consume the sanctioned surface only +- do not create a second local request-mapping path + +`[P2] (confidence: 8/10) .runs/task-m11_5-l1-world-agent-member-runtime/worker-output.patch — the preserved world-agent worker patch is useful, but blindly applying it before the shell lane is unblocked risks integrating code against a still-wrong request-construction boundary.` + +Recommendation: + +- mine logic, not hunks +- reseed both lanes from the post-bridge parent commit +- integrate only after the shell lane proves the crate-surface bridge is sufficient + +### Allowed code shape + +1. No new shared crate. +2. No second transport client. +3. No duplicate member-dispatch payload construction in `async_repl.rs`. +4. No new status reconstruction path. +5. No silent reopening of the payload schema. +6. No worker edits to the bridge files after Step 1 lands. + +If implementation needs any of those, stop and escalate. That means the plan assumptions were +wrong. + +## Test Review + +### Test framework detection + +- Runtime: Rust +- Framework: `cargo test` +- Packages: `shell`, `world-agent` +- No LLM eval suite is required for this slice + +### Code path coverage + +```text +CODE PATH COVERAGE +=========================== +[+] crate-surface bridge + │ + ├── [GAP] direct type export works through crate::execution::* + ├── [GAP] fallback adapter works through crate::execution::* if chosen + └── [GAP] neighboring exports remain unchanged + +[+] crates/world-agent/src/member_runtime.rs + service.rs + │ + ├── [GAP] success path launches from resolved_runtime inside the world + ├── [GAP] authoritative world mismatch fails before startup + ├── [GAP] unsupported backend or missing binary fails closed + ├── [GAP] execute-cancel reaches member spans and converges terminally + └── [GAP] pre-ready failure and post-ready terminal close produce distinct honest states + +[+] crates/shell/src/repl/async_repl.rs + │ + ├── [GAP] [->E2E] first world-backed command uses the sanctioned crate-surface bridge + ├── [GAP] Allocating persists before remote ownership is retained + ├── [GAP] Ready/Running require session-handle evidence, not any Event + ├── [GAP] same-generation reuse does not relaunch + ├── [GAP] shutdown uses execute-cancel for remote member spans + └── [GAP] [->E2E] replacement launch preserves fresh participant_id + resumed_from lineage + +[+] shell status and trace wall + │ + ├── [GAP] status reports the real remote producer + ├── [GAP] trace rows remain producer-correct after cancel and replacement + └── [GAP] failed replacement never revives stale generation liveness + +───────────────────────────────── +COVERAGE: 0/17 remaining PLAN-12 paths proven +GAPS: 17 paths require coverage before closeout +───────────────────────────────── +``` + +### User flow coverage + +```text +USER FLOW COVERAGE +=========================== +[+] First world-backed REPL command + ├── [GAP] [->E2E] shell selects the member and launches it remotely through the sanctioned bridge + ├── [GAP] missing parent or world binding fails before transport + └── [GAP] same-generation reuse does not relaunch + +[+] Live member cancel and shutdown + ├── [GAP] [->E2E] shutdown issues execute-cancel against the remote span + ├── [GAP] cancel delivery failure leaves non-live terminal state + └── [GAP] clean cancel becomes Stopped, not Invalidated + +[+] Shared-world rollover with live member + ├── [GAP] [->E2E] replacement launch crosses the same transport seam with lineage preserved + ├── [GAP] replacement bootstrap failure leaves no authoritative-live member + └── [GAP] old generation never regains liveness + +[+] Operator inspection + ├── [GAP] substrate agent status --json reflects the remote producer + └── [GAP] trace rows remain participant-correct after replacement and cancel +``` + +### Required tests to add + +1. `crates/world-agent/tests/streamed_execute_cancel_v1.rs` + - add member-dispatch success, authoritative-world mismatch, missing binary, + unsupported backend, cancel, and abnormal terminal cases + - assert cancel delivery toggles `delivered=true` only after member span registration +2. `crates/shell/tests/repl_world_first_routing_v1.rs` + - add first launch, same-generation reuse, remote readiness, replacement launch, and + replacement failure cases + - assert `Ready` and `Running` cannot appear before session-handle evidence +3. `crates/shell/tests/agent_successor_contract_ahcsitc0.rs` + - add status truth checks for a real remotely launched member + - assert stale trace fallback does not beat live remote runtime state +4. `crates/shell/tests/agent_hub_trace_persistence.rs` + - add trace truth checks for remote member launch, cancel, and replacement + - assert lineage survives replacement while terminal predecessors stay auditable + +### Test artifact + +The eng-review QA artifact for this continuation is: + +- [spensermcconnell-feat-session-centric-state-store-eng-review-test-plan-20260502-195956.md](/Users/spensermcconnell/.gstack/projects/atomize-hq-substrate/spensermcconnell-feat-session-centric-state-store-eng-review-test-plan-20260502-195956.md) + +## Failure Modes Registry + +| New codepath | Real production failure | Test covers it? | Error handling exists? | User sees clear error? | Critical gap? | +| --- | --- | --- | --- | --- | --- | +| crate-surface bridge | builder is exported but request still cannot be constructed from `crate::execution::*` | no | no | no | yes | +| fallback adapter | helper leaks a second request-construction contract and drifts from payload truth | no | no | no | yes | +| remote launch split | shell persists a world-scoped member as live while startup still runs locally | no | no | no | yes | +| authoritative world binding | world-agent launches against stale or mismatched `world_id` or `world_generation` | no | not yet | partial | yes | +| resolved runtime path | shell provides a binary path missing or unusable inside the Linux world | no | partial | partial | yes | +| remote readiness gate | readiness advances before the session-handle event arrives | no | partial | no | yes | +| cancel path | execute-cancel misses the remote member span and status keeps advertising live | no | partial | partial | yes | +| replacement wall | replacement fails and stale generation still appears live in status or trace | no | partial | no | yes | + +Critical gap rule: + +If any path can advertise a live world member without a real remote session handle, a real +remote cancel path, and a real remote terminal observer, the slice is not done. + +## Performance Review + +This is correctness-first. + +Performance cautions: + +1. keep using the existing `AgentClient` transport stack +2. keep remote stream handling incremental, not buffered +3. preserve same-generation reuse so steady state does not relaunch on every world-backed command + +There is no new throughput blocker beyond that. The complexity is being spent on truth, not on +speculative optimization. + +## DX Guardrails + +This is a developer tool. Failure messages matter. + +Required error-message posture: + +1. bridge failures must name the missing crate surface explicitly +2. remote launch failures must include `participant_id`, `world_id`, `world_generation`, and backend kind +3. cancel failures must say whether delivery failed before span registration or after remote startup +4. replacement failures must say whether stale generation was already invalidated and whether the successor ever reached remote readiness + +## Worktree Parallelization Strategy + +This slice has one real parallel window and no more. The bridge must land first. The regression +wall must land last. + +### Dependency table + +| Step | Modules touched | Depends on | +| --- | --- | --- | +| P0. Parent crate-surface bridge | `crates/shell/src/execution/` | — | +| L1. World-agent member runtime lane | `crates/world-agent/` | P0 | +| L2. Shell remote-member lane | `crates/shell/src/repl/`, `crates/shell/tests/` | P0 | +| P1. Parent regression wall | `crates/shell/tests/` | L1, L2 | + +### Parallel lanes + +- Lane A: `P0`, sequential, parent-owned +- Lane B: `L1`, independent after `P0` +- Lane C: `L2`, independent after `P0` +- Lane D: `P1`, sequential after B and C integrate + +### Suggested worktree ownership + +- Parent worktree: bridge + integration + regression wall +- `wt/plan12-world-agent`: Lane B only +- `wt/plan12-shell`: Lane C only + +No worker gets bridge-file ownership after Step 1. That is what keeps the merge cheap. + +### Execution order + +1. Parent lands the crate-surface bridge. +2. Parent reruns the shell library gate. +3. Reseed both worker lanes from that exact parent commit. +4. Reopen the `world-agent` lane and the shell lane in parallel. +5. Parent integrates accepted outputs. +6. Parent runs the regression wall. + +### Conflict flags + +- Lane B and Lane C stay parallel only if neither reopens the bridge files after `P0` +- if either lane needs `crates/agent-api-types/src/lib.rs`, stop +- if the shell lane needs `session.rs`, `state_store.rs`, or `agents_cmd.rs`, escalate before editing +- if the world-agent lane starts inferring runtime from `backend_id`, stop immediately + +### Parallelization verdict + +One real parallel window remains. Worker cap stays exactly `2`. + +## Implementation Sequence + +This is the execution contract. Steps are ordered. Do not skip a gate because later work seems +obvious. + +### Step 1. Parent crate-surface bridge + +Files: + +- [crates/shell/src/execution/routing.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing.rs) +- [crates/shell/src/execution/routing/dispatch/prelude.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/prelude.rs) +- optional surface-only fallback in [crates/shell/src/execution/routing/dispatch/world_ops.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs) + +Work: + +1. choose the preferred type export or the sanctioned adapter +2. keep serialized payload behavior unchanged +3. prove the shell lane can construct the request through `crate::execution::*` +4. record the bridge choice in run-state and closeout notes + +Validation gate: + +- `async_repl.rs` no longer needs private-module reach-in +- no second payload-construction path exists +- `cargo test -p shell --lib -- --nocapture` passes before workers reopen + +Escalate if: + +- Step 1 needs payload-schema edits +- neighboring exports regress +- the sanctioned surface still does not unblock the shell lane + +### Step 2. World-agent lane + +Files: + +- [crates/world-agent/Cargo.toml](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/Cargo.toml) +- [crates/world-agent/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/lib.rs) +- [crates/world-agent/src/service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs) +- new `crates/world-agent/src/member_runtime.rs` +- [crates/world-agent/tests/streamed_execute_cancel_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/tests/streamed_execute_cancel_v1.rs) + +Work: + +1. add direct `agent_api` dependency +2. add Linux-only `member_runtime.rs` +3. route `member_dispatch` through the member runtime manager +4. register remote member spans for cancel delivery +5. fail closed on binding mismatch, missing binary, or unsupported backend + +Validation gate: + +- world-agent launches from the shell-resolved runtime descriptor +- member dispatch fails closed on world mismatch or missing runtime facts +- remote cancel reports truthfully against the member span registry +- `cargo test -p world-agent --test streamed_execute_cancel_v1 -- --nocapture` passes + +Use the preserved worker patch as logic reference only. Do not blindly apply it. + +### Step 3. Shell remote-member lane + +Files: + +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) +- [crates/shell/tests/repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs) + +Work: + +1. split host-local orchestrator launch from remote member launch explicitly +2. consume only the sanctioned crate-surface bridge +3. wire startup, readiness, cancel, and replacement through the remote path +4. preserve same-generation reuse for already-live remote members + +Validation gate: + +- first world-backed member launch crosses typed execute-stream +- `Ready` and `Running` require session-handle evidence +- replacement preserves lineage and fails closed honestly +- `cargo test -p shell --test repl_world_first_routing_v1 -- --nocapture` passes + +Escalate if: + +- the shell lane needs to edit bridge files after Step 1 +- remote readiness cannot be proven without reopening the readiness contract +- status truth starts depending on local optimistic state again + +### Step 4. Parent regression wall + +Files: + +- [crates/shell/tests/agent_successor_contract_ahcsitc0.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_successor_contract_ahcsitc0.rs) +- [crates/shell/tests/agent_hub_trace_persistence.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_hub_trace_persistence.rs) + +Work: + +1. add status truth assertions for real remote producer state +2. add trace truth assertions for launch, cancel, and replacement +3. prove stale generation never revives after failed replacement + +Validation gate: + +- `cargo test -p shell --test agent_successor_contract_ahcsitc0 -- --nocapture` passes +- `cargo test -p shell --test agent_hub_trace_persistence -- --nocapture` passes +- remote producer truth is observable through status and trace, not inferred from optimistic state + +### Step 5. Closeout + +Work: + +1. update run-state with accepted worker outputs and final validation +2. leave quarantined but unused worker artifacts as evidence only +3. record whether the bridge choice was direct type export or adapter fallback +4. capture the final ordered verification transcript + +Closeout is not done until the bridge choice, lane outputs, and proof commands are all recorded. + +## Definition of Done + +1. the crate surface exposes one sanctioned request-construction bridge for member dispatch +2. `async_repl.rs` launches world-scoped members through the remote path only +3. `world-agent` owns member retained control inside the active world +4. `execute_cancel` reaches both process-exec spans and member spans +5. status and trace show the real remote producer +6. same-generation reuse still avoids redundant relaunch +7. replacement preserves lineage and fails closed honestly +8. all targeted tests pass in order +9. no worker lane reopened the payload contract +10. the run cannot block again on missing plan authority for the crate-surface bridge + +## Recommended Verification Commands + +Run in this order. Do not skip forward. + +```bash +cargo test -p shell --lib -- --nocapture +cargo test -p world-agent --test streamed_execute_cancel_v1 --no-run +cargo test -p shell --test repl_world_first_routing_v1 --no-run +cargo test -p world-agent --test streamed_execute_cancel_v1 -- --nocapture +cargo test -p shell --test repl_world_first_routing_v1 -- --nocapture +cargo test -p shell --test agent_successor_contract_ahcsitc0 -- --nocapture +cargo test -p shell --test agent_hub_trace_persistence -- --nocapture +``` + +## Deferred Work + +- shared startup-crate extraction, if a third consumer ever makes the duplication real +- non-Linux member-dispatch parity +- docs cleanup after the regression wall is green +- status and doctor UX polish after placement truth is proven + +No new `TODOS.md` entry is required here. These are explicit non-goals, not forgotten work. + +## Completion Summary + +- Step 0: scope accepted as a continuation, not a restart +- Architecture Review: 3 issues found and resolved in-plan +- Code Quality Review: 2 issues found and resolved in-plan +- Test Review: diagram produced, 17 remaining continuation gaps identified +- Performance Review: 3 cautions, 0 throughput blockers +- NOT in scope: written +- What already exists: written +- TODOS.md updates: 0 durable TODOs proposed +- Failure modes: 8 critical gaps called out for implementation +- Outside voice: unavailable because `claude` CLI auth is missing +- Parallelization: 4 execution phases, 1 real parallel window, worker cap stays `2` +- Lake Score: complete option chosen over private-module reach-in or another contract refreeze + + +## Decision Audit Trail + +| # | Phase | Decision | Classification | Principle | Rationale | Rejected | +| --- | --- | --- | --- | --- | --- | --- | +| 1 | Scope | Continue from `PLAN-11_5` blocked truth instead of restarting `PLAN-11` | Mechanical | Minimal diff | The payload contract and harness work are already landed | Restarting transport design from zero | +| 2 | Authority | Thaw the crate-surface bridge while keeping the payload contract frozen | Mechanical | Explicit over clever | The block is plan authority, not request schema | Treating bridge visibility as contract churn | +| 3 | Bridge shape | Prefer direct type re-export, authorize one adapter fallback | Taste | Engineered enough | Direct export is smaller, but one sanctioned adapter is allowed if the code shows a better seam | Private-module reach-in or second local builder | +| 4 | Boundaries | Expand parent-owned bridge files to `routing.rs` + `prelude.rs` and optionally surface-only `world_ops.rs` | Mechanical | Blast radius instinct | That is the smallest authority expansion that removes the false blocker | Freezing the bridge seam again | +| 5 | Runtime ownership | Keep runtime selection in the shell and retained control in `world-agent` | Mechanical | Systems over heroes | Two selectors would drift, two state writers would lie | Letting `world-agent` infer runtime from `backend_id` | +| 6 | Shell cutover | Split local orchestrator control from remote member control explicitly | Mechanical | Explicit over clever | The current coupling is the remaining placement lie | Reusing host-local runtime launch for members | +| 7 | Parallelization | Keep the worker cap at `2` with parent-owned bridge first and regression wall last | Mechanical | Minimal diff | That is the only merge-cheap parallel window in this slice | Opening a third lane or parallelizing the bridge | +| 8 | Regression wall | Keep status and trace proof parent-owned after both runtime lanes merge | Mechanical | Blast radius instinct | Producer truth crosses lane boundaries and needs one integrator | Letting a worker lane redefine status truth alone | + +## GSTACK REVIEW REPORT + +| Review | Trigger | Why | Runs | Status | Findings | +| --- | --- | --- | --- | --- | --- | +| CEO Review | `/plan-ceo-review` | Scope and strategy | 1 | CLEAR | Kept the continuation narrow: fix plan authority at the crate surface, then finish the original runtime cutover | +| Codex Review | `/codex review` | Independent 2nd opinion | 0 | SKIPPED | No separate outside-model review run, and `claude` CLI auth is missing for outside voice | +| Eng Review | `/plan-eng-review` | Architecture and tests (required) | 1 | CLEAR | Locked the real blocker to the request-construction bridge, froze the payload contract, and defined the regression wall for remote launch, cancel, replacement, status, and trace truth | +| Design Review | `/plan-design-review` | UI/UX gaps | 0 | SKIPPED | No UI scope | + +**UNRESOLVED:** 0 plan-level decisions remain. The remaining work is implementation plus +verification only. + +**VERDICT:** ENG CLEARED. `PLAN-12` replaces `PLAN-11_5` as the honest continuation plan: +thaw the request bridge, reopen exactly two runtime lanes, and finish the placement-truth +regression wall without reopening the payload contract. diff --git a/llm-last-mile/PLAN-13.md b/llm-last-mile/PLAN-13.md new file mode 100644 index 000000000..19a14c064 --- /dev/null +++ b/llm-last-mile/PLAN-13.md @@ -0,0 +1,813 @@ +# PLAN-13: Make World-Scoped Member Runtime Placement Real And Keep Runtime Contracts Aligned + +Source plan: [PLAN-12.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-12.md) +Source SOW: [13-member-runtime-world-placement-gap-sow.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/13-member-runtime-world-placement-gap-sow.md) +Supersedes: `PLAN-12` as the next execution document after the transport-owned remote member cutover landed +Branch: `feat/session-centric-state-store` +Base branch: `main` +Plan type: Linux-first runtime-correctness plan, backend-only, installer-sensitive, no UI scope +Review posture: `/autoplan` scope discipline with `/plan-eng-review` depth, rewritten as one cohesive execution plan +Status: execution-ready planning pass on 2026-05-03 +Outside voice: not used for this document generation + +## Objective + +`PLAN-12` closed the request-surface and remote-ownership gap. It did not close the placement gap. + +Today the shell selects a world-scoped member, persists the right world binding, and launches that +member through `world-agent` over the existing `/v1/execute/stream` seam. But the launched runtime +still comes from daemon-local `gateway.run_control(...)` inside +[crates/world-agent/src/member_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs:40), +so the process is correlated to the session world without actually entering the authoritative +session world boundary. + +This plan finishes the missing truth: + +1. keep shell authority, request shape, and transport seam exactly as they are now, +2. make `world-agent` launch the member runtime inside the active session world for real, +3. make the shell fail closed if that placement cannot be established, +4. align Linux, macOS Lima, and WSL provisioning posture with the final runtime contract, +5. prove placement via observable runtime facts instead of metadata-only claims. + +The user-visible outcome is simple: + +- when a world-scoped member says it is live in world `W` generation `N`, +- it is actually executing inside world `W` generation `N`, +- under the same filesystem, cgroup, and network posture contract as other world execution, +- and `substrate agent status`, cancellation, and trace surfaces do not lie. + +## Locked Starting State + +### What is already done + +The following work is already landed and must be treated as starting truth, not something to +reopen in this slice: + +- the shell still owns canonical orchestration session and participant persistence +- the shell still resolves backend kind and binary path before dispatch +- the shell already sends world-scoped members through typed `member_dispatch` over + `/v1/execute/stream` +- `world-agent` already validates the authoritative `world_id` and `world_generation` +- `execute_cancel` already has a remote member span path +- same-generation reuse and replacement semantics already exist at the shell lifecycle layer +- the crate-surface request bridge from `PLAN-12` is already open in + [crates/shell/src/execution/routing.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing.rs:35) + +### Carry-forward bridge authority from `PLAN-12` + +This placement plan does not reopen the transport problem, but it does carry forward the +anti-blocking authority from `PLAN-12` so the implementation cannot get stuck on the same seam +again. + +The payload contract stays frozen, but the parent is explicitly authorized to thaw the +crate-surface request bridge before reopening lanes if a rebase or merge reveals that the shell can +no longer legally consume the already-frozen request builder through `crate::execution::*`. + +The preferred fix remains: + +- direct re-export of `MemberDispatchTransportRequest` through the allowed crate surface + +The only pre-authorized fallback remains: + +- one sanctioned adapter helper on the existing shell routing surface + +Hard limits: + +- no `MemberDispatchRequestV1` shape change +- no serialized payload change +- no second request-construction path in `async_repl.rs` +- no runtime-selection move into `world-agent` + +### Exact gap + +The remaining correctness gap is narrow and concrete: + +1. `execute_stream(...)` ensures the authoritative world and validates the dispatch binding in + [crates/world-agent/src/service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs:1223). +2. The member lane then calls + `MemberRuntimeManager::launch(agent_id, cwd, env, span_id, dispatch, binding)` in + [crates/world-agent/src/service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs:1238). +3. `MemberRuntimeManager::launch(...)` validates the runtime binary, builds an `AgentWrapperGateway`, + then calls `gateway.run_control(...)` directly in + [crates/world-agent/src/member_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs:50). + +That means the member runtime is remote-owned, but not world-placed. + +### Why this is still unsafe + +The current implementation can truthfully say: + +- the member belongs to the current orchestration session +- the member belongs to the current `world_id` +- the member belongs to the current `world_generation` +- `world-agent` owns the retained control handle +- `/v1/execute/cancel` can reach it + +It still cannot truthfully say: + +- the member process is inside the authoritative session overlay/rootfs view +- the member process is attached to the session cgroup +- the member process sees the session network posture rather than the daemon host posture +- operator-visible world metadata matches the actual process boundary + +That is the gap this plan closes. + +## Frozen Execution Contract + +This section is not optional interpretation space. + +### Non-negotiable invariants + +1. The shell remains the only canonical writer of orchestration session and participant state. +2. `world-agent` remains the transport owner for member startup, streaming, cancel, and terminal observation. +3. `MemberDispatchRequestV1` stays frozen. No request-shape churn in this slice. +4. `/v1/execute/stream` and `/v1/execute/cancel` remain the only transport seam. +5. `ExecuteStreamFrame::{Start,Event,Exit,Error}` remain the stream families. +6. World-scoped member launch fails closed if true placement cannot be established. +7. Same-generation reuse and replacement semantics remain shell-owned. +8. Backend selection stays in the shell. `world-agent` must not infer runtime from `backend_id`. +9. Linux-first remains explicit. macOS and WSL must either align their service/runtime contract or fail closed with clear posture. +10. Installer and provisioning drift is part of the bug, not separate cleanup. + +### Chosen placement strategy + +This plan chooses one concrete strategy: + +- reuse the existing world-session discovery and cgroup-binding rules already used by the + gateway-runtime path in + [crates/world-agent/src/service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs:1560) + and + [crates/world-agent/src/gateway_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/gateway_runtime.rs:696) +- add a Linux-only placement-aware member launcher inside `world-agent` +- keep `agent_api` backend construction in `world-agent` +- move the actual child spawn under world-entry logic, rather than daemon-local direct spawn + +The plan does not require the exact API to match `backend.exec(&world, ...)`, but it does require +equivalent guarantees: + +- world already resolved and generation-validated +- child attached to the authoritative session cgroup +- child launched against the correct session filesystem view +- child subject to the active world network posture +- retained-control ownership, event forwarding, and cancel retention preserved + +### Execution policy for this slice + +This is a one-direction plan. The implementation is not allowed to drift into a hybrid state where +some members are merely metadata-bound and others are truly world-placed. + +Allowed outcomes: + +1. world-scoped member launch is observably world-placed and shell state reflects that truth +2. startup fails closed before the member is advertised as live + +Disallowed halfway states: + +- daemon-local launch with world metadata attached +- warning-and-continue startup for a selected world-scoped member +- installer parity asserted by docs only, without script or service proof +- "temporary" request-bridge churn that changes payload shape or duplicates request construction + +## Step 0: Scope Challenge + +### 0A. What already exists + +| Sub-problem | Existing code | Decision | +| --- | --- | --- | +| typed member-dispatch contract | [crates/agent-api-types/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/agent-api-types/src/lib.rs) | Reuse exactly. No payload changes. | +| shell-side request construction and remote startup seam | [crates/shell/src/execution/routing/dispatch/world_ops.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs), [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:2829) | Reuse. Tighten fail-closed behavior only. | +| authoritative world validation before member dispatch | [crates/world-agent/src/service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs:1223) | Reuse. Extend the post-validation carrier with placement context. | +| long-lived runtime cgroup attach pattern | [crates/world-agent/src/gateway_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/gateway_runtime.rs:696) | Reuse as the placement reference, not as a new ownership model. | +| cancel delivery by `span_id` | [crates/world-agent/src/service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs:1446), [crates/world-agent/src/member_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs:102) | Reuse. Preserve the current transport truth. | +| installer and provisioning authors | [scripts/substrate/dev-install-substrate.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/substrate/dev-install-substrate.sh:1482), [scripts/substrate/install-substrate.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/substrate/install-substrate.sh:1932), [scripts/linux/world-provision.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/linux/world-provision.sh:535), [scripts/mac/lima-warm.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/lima-warm.sh:659), [scripts/wsl/provision.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/wsl/provision.sh:31) | Reuse the existing surfaces, but align them in the same slice. | + +### 0B. Minimum diff decision + +The smallest honest diff is: + +1. add one internal placement-context carrier between `service.rs` and `member_runtime.rs` +2. change the member runtime launcher from daemon-local spawn to world-aware spawn +3. preserve request shape, stream framing, readiness contract, and shell authority +4. make the shell fail closed when placement cannot be established +5. update all service/provisioning writers that author the runtime contract +6. add proof tests for real placement, not just world metadata + +Anything smaller leaves the lie in place. Anything larger risks reopening already-landed transport work. + +### 0C. Complexity check + +This slice touches more than 8 files. That is justified and still minimal. + +The likely production path is: + +1. `crates/world-agent/src/service.rs` +2. `crates/world-agent/src/member_runtime.rs` +3. `crates/world-agent/src/gateway_runtime.rs` or a new internal helper extracted from it +4. `crates/world-agent/src/lib.rs` +5. `crates/world-agent/tests/streamed_execute_cancel_v1.rs` +6. one new Linux-only world-placement proof test in `crates/world-agent/tests/` +7. `crates/shell/src/repl/async_repl.rs` +8. `crates/shell/tests/repl_world_first_routing_v1.rs` +9. `crates/shell/tests/agent_successor_contract_ahcsitc0.rs` +10. `crates/shell/tests/agent_hub_trace_persistence.rs` +11. `scripts/substrate/dev-install-substrate.sh` +12. `scripts/substrate/install-substrate.sh` +13. `scripts/linux/world-provision.sh` +14. `scripts/mac/lima-warm.sh` +15. `scripts/wsl/provision.sh` +16. `scripts/windows/wsl-warm.ps1` +17. the relevant docs in `docs/WORLD.md`, `docs/INSTALLATION.md`, and cross-platform verification docs + +That is a smell only if the slice invents new abstraction layers. It does not need to. + +### 0D. Search and completeness check + +Search-before-building result: + +- **[Layer 1]** reuse the existing world-session and cgroup-attach logic instead of inventing a new isolation model +- **[Layer 1]** reuse the existing stream transport and cancel path +- **[Layer 1]** reuse the existing shell readiness gate keyed on the session-handle event +- **[Layer 3]** treat installer alignment as part of correctness because multiple scripts author the effective runtime contract + +Shortcut options that are explicitly rejected: + +- keep launching the member daemon-locally and document the mismatch +- infer placement from `world_id` metadata only +- update only Linux provisioning and leave dev-install or Lima stale +- keep the shell warning path that says "world-scoped member runtime unavailable" and then continues + +### 0E. Distribution and runtime contract check + +No new distributable artifact type is introduced, but the runtime contract changes anyway. + +That means the plan must cover: + +- service-unit capabilities +- service-unit `ReadWritePaths` +- `SUBSTRATE_HOME` or other required env propagation +- any cached Linux guest bundle used by macOS dev-install +- explicit WSL posture if it cannot be aligned in the same slice + +### 0F. NOT in scope + +- moving runtime selection into `world-agent` +- creating a second member-dispatch transport family +- redesigning shell-owned participant or orchestration-session persistence +- new public `substrate agent start|resume|fork|stop` productization work +- cross-platform feature parity beyond explicit runtime-contract posture +- unrelated status or doctor UX redesign +- extracting a shared installer template generator unless the duplication itself blocks correctness + +## Architecture Review + +### Findings + +`[P1] (confidence: 10/10) crates/world-agent/src/member_runtime.rs:50-61 - member launch still uses daemon-local gateway.run_control(...), so a runtime can be tagged with the active world binding without inheriting the world boundary itself.` + +Recommendation: + +- replace direct local spawn with a world-aware retained-control launcher +- keep stream framing and cancel registry intact +- do not treat metadata validation as a substitute for world entry + +`[P1] (confidence: 9/10) crates/world-agent/src/service.rs:1233-1247 - the member-dispatch lane passes only cwd, env, dispatch, and binding into the runtime manager, which is not enough to attach the child to the session cgroup or guarantee the session filesystem/network view.` + +Recommendation: + +- introduce one internal placement carrier resolved at the service layer +- include the world handle or equivalent resolved facts needed for cgroup, filesystem, and network placement +- keep that carrier internal to `world-agent` + +`[P1] (confidence: 9/10) crates/shell/src/repl/async_repl.rs:3406-3414 - the shell still prints a warning and continues when the world-scoped member runtime is unavailable, but after this slice that state becomes a fail-closed correctness failure, not a degradable warning.` + +Recommendation: + +- convert placement failure into a startup failure for selected world-scoped members +- keep the REPL alive, but do not advertise the member as available or silently fall back + +`[P1] (confidence: 9/10) scripts/substrate/install-substrate.sh:1932, scripts/linux/world-provision.sh:535, scripts/mac/lima-warm.sh:659, scripts/wsl/provision.sh:31 - multiple independent scripts author the world-agent runtime contract today, and they already drift on group/capability/home-path details, so a placement-sensitive change can easily ship half-applied.` + +Recommendation: + +- update all contract authors in the same slice +- add verification steps that compare the resulting service posture, not just whether the binary exists +- choose and document explicit WSL posture if parity is deferred + +### Ownership split + +| Concern | Owner | Why | +| --- | --- | --- | +| runtime selection, backend kind, binary path | shell | Already resolved there. Re-resolving creates drift. | +| authoritative world selection and generation validation | `world-agent` service layer | That is where the active world is proven. | +| actual child placement into the world | `world-agent` runtime layer | That is the missing truth this slice lands. | +| retained control, event streaming, cancel registry | `world-agent` runtime layer | Already remote-owned. Must remain there. | +| participant persistence, readiness transitions, replacement decisions | shell | Canonical state stays shell-owned. | +| service-unit/runtime-contract authoring | installers and provisioning scripts | Those scripts create the effective runtime environment. | + +### Architecture ASCII diagrams + +```text +CURRENT STATE +============= +shell + ├── selects member runtime + ├── persists world_id/world_generation + └── dispatches typed member request over /v1/execute/stream + │ + ▼ +world-agent service + ├── ensures authoritative session world + ├── validates dispatch world binding + └── calls MemberRuntimeManager::launch(...) + │ + ▼ +member_runtime.rs + ├── build AgentWrapperGateway + └── gateway.run_control(...) + │ + ▼ +daemon-local child process + +Result: + remote-owned, but not guaranteed world-placed +``` + +```text +TARGET STATE +============ +shell + ├── selects runtime and persists allocating state + ├── dispatches typed member request + └── waits for remote session-handle evidence + │ + ▼ +world-agent service + ├── ensures authoritative session world + ├── validates world_id/world_generation + ├── resolves placement context + │ ├── world identity + │ ├── cgroup path + │ ├── filesystem / overlay facts + │ └── network posture + └── calls placement-aware member launcher + │ + ▼ +member runtime launcher + ├── builds backend from resolved_runtime + ├── spawns inside session world + ├── attaches child to session cgroup + ├── registers span_id for cancel + ├── forwards Start / Event / Exit / Error + └── fails closed if placement cannot be established + │ + ▼ +shell + ├── marks Ready only after session-handle event + └── never advertises live state on placement failure +``` + +```text +RUNTIME CONTRACT AUTHORS +======================== +release install dev install direct provision macOS Lima WSL +install-substrate -> dev-install-substrate -> world-provision.sh -> lima-warm.sh -> wsl/provision.sh + \___________________________ all must agree on ___________________________/ + capabilities, RW paths, socket ownership, env, binary staging, posture +``` + +## Code Quality Review + +### Findings + +`[P2] (confidence: 9/10) crates/world-agent/src/service.rs + crates/world-agent/src/gateway_runtime.rs - the cgroup attach and world-binding preparation logic already exists in one long-lived runtime path, so duplicating that behavior independently for member runtime placement would create a second isolation truth.` + +Recommendation: + +- extract or mirror one internal helper from the gateway-runtime path +- keep the member-specific behavior in `member_runtime.rs` +- do not fork a second isolation-preparation implementation + +`[P2] (confidence: 8/10) scripts/substrate/install-substrate.sh + scripts/linux/world-provision.sh + scripts/mac/lima-warm.sh + scripts/wsl/provision.sh - service-unit duplication is already real, but introducing a brand-new shared generator in this slice would be over-engineering unless the current duplication blocks correctness.` + +Recommendation: + +- update all existing authors in lockstep now +- add parity assertions or smoke verification +- defer template extraction unless the same fields are still drifting after this slice + +`[P2] (confidence: 8/10) crates/shell/src/repl/async_repl.rs:3406-3414 - the shell currently conflates "member runtime unavailable" with a soft warning path, which is the wrong abstraction once placement becomes mandatory.` + +Recommendation: + +- split "transport setup failed" from "placement contract failed" +- make the latter a concrete persisted failure state with actionable error messaging + +### Allowed code shape + +1. One new internal placement-context carrier is allowed. +2. One internal world-aware launcher or helper extraction is allowed. +3. No new public request family. +4. No duplicate runtime selector. +5. No duplicate cgroup/world-setup implementation if existing helpers can be reused. +6. No silent installer drift. + +## Test Review + +### Test framework detection + +- Runtime: Rust +- Framework: `cargo test` +- Primary crates: `world-agent`, `shell` +- No prompt or LLM eval suite is required for this slice + +### Code path coverage + +```text +CODE PATH COVERAGE +=========================== +[+] world-agent service -> member runtime handoff + │ + ├── [GAP] binding validation passes and placement context resolves + ├── [GAP] missing cgroup/world facts fail before child spawn + └── [GAP] world mismatch still fails before startup + +[+] placement-aware member launch + │ + ├── [GAP] [->E2E] child observes session filesystem view, not daemon host view + ├── [GAP] child attaches to authoritative session cgroup + ├── [GAP] isolated-network session constrains member runtime + ├── [GAP] missing binary fails closed + ├── [GAP] unsupported backend kind fails closed + └── [GAP] cancel reaches the live placed member span + +[+] shell lifecycle truth + │ + ├── [GAP] selected world-scoped member fails closed on placement failure + ├── [GAP] Ready requires session-handle evidence from the placed runtime + ├── [GAP] same-generation reuse still does not relaunch + ├── [GAP] replacement launch still preserves lineage after placement work + └── [GAP] failed replacement never revives stale generation liveness + +[+] installer / provisioning contract + │ + ├── [GAP] Linux dev-install provisions a compatible world-agent contract + ├── [GAP] Linux release install provisions the same relevant contract + ├── [GAP] macOS Lima guest service contract matches Linux where required + └── [GAP] WSL is either aligned or explicitly fail-closed with docs + +[+] operator surfaces + │ + ├── [GAP] status reflects the real placed producer + └── [GAP] trace rows do not overclaim placement on failure paths + +───────────────────────────────── +COVERAGE: 0/16 placement-truth paths proven +GAPS: 16 paths require coverage before closeout +───────────────────────────────── +``` + +### User flow coverage + +```text +USER FLOW COVERAGE +=========================== +[+] First world-backed member launch + ├── [GAP] [->E2E] shell selects member -> remote launch -> session-handle -> Ready + ├── [GAP] placement failure produces failed startup, not warning-and-continue + └── [GAP] same-generation reuse remains no-op + +[+] Live member cancel + ├── [GAP] [->E2E] execute_cancel stops the placed runtime by span_id + ├── [GAP] cancel before span registration is reported clearly + └── [GAP] terminal state is honest after cancel + +[+] Shared-world generation rollover + ├── [GAP] [->E2E] replacement runtime launches in the replacement world + ├── [GAP] replacement placement failure leaves no authoritative-live successor + └── [GAP] stale generation never regains liveness + +[+] Operator verification + ├── [GAP] status shows remote producer truth + ├── [GAP] trace remains participant-correct after cancel + └── [GAP] trace does not claim world placement if launch failed before entry +``` + +### Required tests to add or extend + +1. `crates/world-agent/tests/streamed_execute_cancel_v1.rs` + - keep existing remote cancel proof + - extend with placed-member success, binding mismatch, missing binary, unsupported backend, and cancel cases + +2. New Linux-only proof test in `crates/world-agent/tests/` + - prove real placement using observable facts + - acceptable proof styles: + - session cgroup membership + - overlay/rootfs view rather than daemon-host view + - network posture restrictions when isolation is enabled + - do not infer placement from metadata alone + +3. `crates/shell/tests/repl_world_first_routing_v1.rs` + - assert fail-closed startup when placement cannot be established + - keep same-generation reuse and replacement semantics green + - assert Ready is still gated on the session-handle event + +4. `crates/shell/tests/agent_successor_contract_ahcsitc0.rs` + - assert status truth for a real placed member runtime + - ensure stale generation does not win over live runtime truth + +5. `crates/shell/tests/agent_hub_trace_persistence.rs` + - assert launch, cancel, failure, and replacement trace truth after placement changes + +### Failure-mode test rule + +Any path that can simultaneously satisfy all three conditions below is a critical gap and must get a test: + +1. no placement proof +2. no clear error handling +3. user-facing or operator-facing state still suggests the member is live + +## Failure Modes Registry + +| New codepath | Real production failure | Test covers it? | Error handling exists? | User sees clear error? | Critical gap? | +| --- | --- | --- | --- | --- | --- | +| placement context resolution | service validates world metadata but cannot resolve cgroup/world facts | no | partial | partial | yes | +| placed child spawn | runtime still starts on daemon host even though dispatch binding is valid | no | no | no | yes | +| cgroup attach | child spawns but never enters session cgroup | no | partial | no | yes | +| filesystem view | child sees daemon-host files instead of session overlay/rootfs | no | no | no | yes | +| isolated network posture | child escapes session network restrictions | no | no | no | yes | +| shell fail-closed path | placement failure is downgraded to warning and loop continues | no | partial | partial | yes | +| cancel path | cancel misses the placed runtime span and state remains live | no | partial | partial | yes | +| replacement path | replacement placement fails and stale generation remains authoritative-live | no | partial | no | yes | +| installer drift | one install path has the required capability/env/path contract and another does not | no | no | no | yes | +| WSL posture | WSL silently keeps an incompatible root/root socket contract | no | no | no | yes | + +## Performance Review + +This is a correctness-first slice, but a few performance rules still matter: + +1. keep using the existing request transport and streaming path +2. reuse the current session world rather than creating a fresh world per member launch +3. preserve same-generation reuse so steady-state commands do not relaunch the member +4. avoid re-deriving expensive world facts more than once per launch if the service layer can carry them down + +There is no new throughput feature here. The performance risk is accidental relaunch or accidental repeated world preparation. + +## DX Guardrails + +This is still a developer tool, so failures must explain themselves. + +Required error-message posture: + +1. placement failures must say whether the failure happened before spawn, during world entry, during cgroup attach, or after stream establishment +2. shell-visible errors must include `participant_id`, `world_id`, `world_generation`, and backend kind when available +3. installer failures must name the exact contract element that is missing, such as capability, env var, RW path, or socket/group posture +4. WSL or macOS non-parity must be explicit in docs and doctor output, not implicit behavior + +## Worktree Parallelization Strategy + +There is a real parallel window here, but only after the parent-owned preflight is frozen. +That preflight includes the placement contract and the carry-forward bridge safeguard above. + +### Dependency table + +| Step | Modules touched | Depends on | +| --- | --- | --- | +| P0. Freeze placement contract, carrier shape, and bridge safeguard | `crates/world-agent/`, `crates/shell/src/repl/`, `crates/shell/src/execution/`, planning docs | - | +| L1. World-agent placement core | `crates/world-agent/` | P0 | +| L2. Shell fail-closed and status/trace lane | `crates/shell/src/repl/`, `crates/shell/tests/` | P0 | +| L3. Installer and provisioning alignment | `scripts/`, `docs/` | P0 | +| P1. Parent integration and proof wall | `crates/world-agent/tests/`, `crates/shell/tests/`, `scripts/`, `docs/` | L1, L2, L3 | + +### Parallel lanes + +- Lane A: `P0`, sequential, parent-owned +- Lane B: `L1`, world-agent placement core +- Lane C: `L2`, shell lifecycle and regression wall preparation +- Lane D: `L3`, installer/provisioning and docs alignment +- Lane E: `P1`, sequential integration and final proof + +### Execution order + +1. Parent freezes the placement strategy, the internal carrier shape, and confirms the shell-side + request bridge is still legally reachable through the sanctioned crate surface. +2. Launch `L1`, `L2`, and `L3` in parallel worktrees. +3. Merge the world-agent lane first, because it defines the final runtime behavior. +4. Merge shell and installer lanes after they rebase on that reality. +5. Run the parent-owned proof wall and closeout. + +### Conflict flags + +- `L1` and `L2` must not change `MemberDispatchRequestV1` or request construction +- only the parent may reopen the sanctioned shell bridge safeguard if rebases reveal drift there +- `L2` must not invent a second readiness contract +- `L3` must not silently redefine capability or env requirements without feeding those back into `L1` +- if `L1` needs changes in `crates/agent-api-types/`, stop and escalate +- if `L3` discovers a new helper binary or artifact requirement, update the plan and rerun the dependency check before merging + +### Parallelization verdict + +Three independent workstreams exist after one parent-owned contract freeze. Worker cap can be `3`. + +## Implementation Sequence + +### Step 1. Freeze the placement contract and re-confirm the bridge safeguard + +Files: + +- [crates/world-agent/src/service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs) +- [crates/world-agent/src/member_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs) +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) +- [crates/shell/src/execution/routing.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing.rs) +- [crates/shell/src/execution/routing/dispatch/prelude.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/prelude.rs) + +Work: + +1. define one internal placement carrier resolved at the service layer +2. explicitly choose the placement helper reuse path from gateway-runtime logic +3. re-confirm that the shell can still reach the already-frozen request builder through the + sanctioned crate surface +4. if that bridge has drifted, the parent applies the pre-authorized fix before reopening worker + lanes: + - preferred: direct re-export of `MemberDispatchTransportRequest` + - fallback: one sanctioned adapter helper +5. lock the shell fail-closed posture for selected world-scoped members +6. record the WSL posture decision: aligned in-slice or explicitly unsupported/fail-closed + +Validation gate: + +- no payload changes +- no transport changes +- sanctioned request bridge still reachable, or parent-fixed without payload churn +- one agreed internal carrier shape +- one agreed fail-closed behavior + +### Step 2. World-agent placement core + +Files: + +- [crates/world-agent/src/service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs) +- [crates/world-agent/src/member_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs) +- [crates/world-agent/src/gateway_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/gateway_runtime.rs) +- [crates/world-agent/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/lib.rs) +- `crates/world-agent/tests/streamed_execute_cancel_v1.rs` +- one new Linux-only world-placement proof test + +Work: + +1. resolve authoritative placement facts after world validation +2. route member launch through a placement-aware launcher +3. reuse cgroup attach and runtime-preparation rules from the gateway-runtime path where possible +4. keep `span_id`, `Start`, `Event`, `Exit`, and `Error` behavior intact +5. preserve the existing remote ownership model, not a new shell-owned retained-control path +6. fail closed on missing binary, unsupported backend, world mismatch, or placement setup failure + +Validation gate: + +- live member runtime is observably placed in the session world +- cancel still reaches the placed runtime +- metadata and runtime truth cannot diverge silently + +### Step 3. Shell fail-closed and lifecycle truth + +Files: + +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) +- `crates/shell/tests/repl_world_first_routing_v1.rs` +- `crates/shell/tests/agent_successor_contract_ahcsitc0.rs` +- `crates/shell/tests/agent_hub_trace_persistence.rs` + +Work: + +1. replace the warning-and-continue path with explicit startup failure for selected world-scoped members +2. preserve Allocating -> Ready gating on session-handle evidence only +3. keep same-generation reuse behavior unchanged +4. keep replacement lineage and invalidation semantics unchanged in shape +5. update status and trace assertions to reflect real placed-runtime truth +6. forbid fallback to host-local retained control once a world-scoped member has been selected + +Validation gate: + +- placement failure never yields an authoritative-live member +- Ready and Running still require remote evidence +- replacement failure never revives stale generation + +### Step 4. Installer and provisioning alignment + +Files: + +- [scripts/substrate/dev-install-substrate.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/substrate/dev-install-substrate.sh) +- [scripts/substrate/install-substrate.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/substrate/install-substrate.sh) +- [scripts/linux/world-provision.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/linux/world-provision.sh) +- [scripts/mac/lima-warm.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/lima-warm.sh) +- [scripts/wsl/provision.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/wsl/provision.sh) +- [scripts/windows/wsl-warm.ps1](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/windows/wsl-warm.ps1) +- the matching docs in `docs/` + +Work: + +1. align service-unit capabilities, env, RW paths, group/socket posture, and binary staging with the final placement contract +2. ensure Linux dev-install and Linux release install land the same relevant runtime assumptions +3. ensure macOS Lima guest provisioning mirrors Linux where required +4. either align WSL or document and enforce fail-closed unsupported posture +5. update docs so operators know how to verify the contract + +Validation gate: + +- there is no silent drift between Linux dev-install, Linux release install, direct Linux provision, and macOS guest provisioning +- WSL posture is explicit + +### Step 5. Parent proof wall and closeout + +Work: + +1. run the targeted Rust tests in order +2. run one Linux dev-install verification pass +3. run one Linux release-install or `world-provision.sh` verification pass +4. run one macOS Lima verification pass if the guest service contract changed +5. record the WSL result or explicit deferral posture +6. update docs and closeout evidence + +Closeout is not done until the plan has runtime proof, installer proof, and operator-facing documentation proof. + +## Recommended Verification Commands + +Run in this order. Do not skip forward. The parent proof wall is not complete until the Rust test +wall is green and the affected install/provision paths have been exercised for the platforms they +claim to support. + +```bash +cargo test -p world-agent --test streamed_execute_cancel_v1 --no-run +cargo test -p shell --test repl_world_first_routing_v1 --no-run +cargo test -p shell --test agent_successor_contract_ahcsitc0 --no-run +cargo test -p shell --test agent_hub_trace_persistence --no-run +cargo test -p world-agent --test streamed_execute_cancel_v1 -- --nocapture +cargo test -p shell --test repl_world_first_routing_v1 -- --nocapture +cargo test -p shell --test agent_successor_contract_ahcsitc0 -- --nocapture +cargo test -p shell --test agent_hub_trace_persistence -- --nocapture +scripts/linux/world-provision.sh --profile release +scripts/mac/lima-warm.sh +pwsh -File scripts/windows/wsl-warm.ps1 -DistroName substrate-wsl -ProjectPath (Resolve-Path .) +``` + +If a dedicated Linux-only placement proof test is added, insert it between the world-agent stream +test and the shell test wall. + +Platform posture for this pass: + +- Linux verification is required +- macOS Lima verification is required if the guest service contract or staged runtime changes +- WSL verification is required if the slice claims WSL alignment; otherwise the proof is the + explicit fail-closed posture plus docs update + +## Definition of Done + +1. world-scoped member launch still uses `/v1/execute/stream` +2. the launched runtime is observably inside the authoritative session world +3. `/v1/execute/cancel` still cancels the live member by `span_id` +4. shell-owned persistence, readiness, invalidation, and replacement semantics remain authoritative +5. same-generation reuse still works +6. placement failure is fail-closed, not warning-and-continue +7. Linux dev-install, Linux release install, and Linux direct provision are aligned for the final runtime contract +8. macOS Lima is aligned where required, or explicitly documented if not +9. WSL is either aligned or explicitly fail-closed with documentation +10. status and trace surfaces do not overclaim placement + +The run is not done if the code is correct but the sanctioned request bridge regressed and was left +to "follow-up later." That is the same class of stall this plan is meant to prevent. + +## Deferred Work + +- broader public agent lifecycle productization +- cross-platform feature parity beyond explicit runtime-contract posture +- installer templating cleanup if duplication remains painful after this slice +- status and doctor UX polish unrelated to placement truth + +No new `TODOS.md` entry is required yet. These are explicit deferrals, not forgotten work. + +## Completion Summary + +- Step 0: scope accepted as a narrow post-`PLAN-12` correctness follow-on +- Architecture Review: 4 core issues found and resolved in-plan +- Code Quality Review: 3 structural cautions found and resolved in-plan +- Test Review: diagram produced, 16 placement-truth gaps identified +- Performance Review: 4 correctness-preserving cautions, 0 new throughput features +- NOT in scope: written +- What already exists: written +- TODOS.md updates: 0 durable TODOs proposed +- Failure modes: 10 critical gaps identified for implementation proof +- Outside voice: not used for this document generation +- Parallelization: 5 execution phases, 3 parallel lanes after one parent-owned freeze +- Lake Score: complete option chosen over metadata-only placement, partial installer updates, or warning-path fallback + + +## Decision Audit Trail + +| # | Phase | Decision | Classification | Principle | Rationale | Rejected | +| --- | --- | --- | --- | --- | --- | --- | +| 1 | Scope | Treat this as a post-`PLAN-12` placement-truth slice, not a transport redesign | Mechanical | Minimal diff | The request bridge and remote ownership are already landed | Reopening payload or transport work | +| 2 | Placement strategy | Reuse gateway-runtime world/cgroup preparation patterns for member launch | Taste | Explicit over clever | One existing runtime path already solves the hard placement problems | Inventing a separate world-entry system | +| 3 | Shell posture | Convert selected world-scoped member placement failure into fail-closed startup failure | Mechanical | Completeness | Warning-and-continue would preserve a correctness lie | Silent warning fallback | +| 4 | Installer scope | Update all runtime-contract authors in the same slice | Mechanical | Boil the lake | Partial installer updates would ship inconsistent truth | Linux-only script edits | +| 5 | WSL | Force an explicit aligned-or-fail-closed posture | Mechanical | Explicit over clever | Silent drift is worse than unsupported | Leaving WSL ambiguous | +| 6 | Tests | Require observable placement proof, not metadata-only assertions | Mechanical | Systems over heroes | Operator trust depends on runtime facts | Trusting `world_id` fields alone | +| 7 | Parallelization | Freeze the contract first, then run world-agent, shell, and installer lanes in parallel | Taste | Pragmatic | That is the widest parallel window with contained merge risk | Fully serial execution or uncontrolled overlap | diff --git a/llm-last-mile/PLAN-14.md b/llm-last-mile/PLAN-14.md new file mode 100644 index 000000000..900c2bf8b --- /dev/null +++ b/llm-last-mile/PLAN-14.md @@ -0,0 +1,824 @@ +# PLAN-14: Replace Gateway Secret Env Injection With Read-Once Auth-Bundle Delivery + +Source SOW: [14-secret-handoff-into-the-world-gateway.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/14-secret-handoff-into-the-world-gateway.md) +Gap matrix anchor: [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md) +Branch: `feat/session-centric-state-store` +Base branch: `main` +Plan type: Linux-first gateway secret-carrier hardening plan, backend-only, cross-crate (`world-agent` + `substrate-gateway`) with contract-doc impact, no UI scope +Review posture: `/autoplan` scope discipline with `/plan-eng-review` depth, rewritten as one cohesive execution plan +Status: execution-ready planning pass on 2026-05-04 +Outside voice: not used for this document generation + +## Objective + +The host-side auth sourcing, policy gating, and typed lifecycle request work is already landed. + +The remaining bug is narrower and more serious: the last hop still turns integrated auth into child-process environment variables, and the gateway still expects integrated auth from env-backed surfaces. That leaves the shell/request boundary honest and the world-runtime boundary dishonest. + +This plan closes that last-mile gap without reopening the earlier slices: + +1. keep shell-owned host auth sourcing and `GatewayLifecycleRequestV1.integrated_auth` exactly as they are, +2. replace secret-bearing child-env delivery with a read-once auth bundle written over an inherited FD/pipe, +3. expose only one non-secret pointer env var, `SUBSTRATE_LLM_AUTH_BUNDLE_FD`, +4. teach `substrate-gateway` integrated startup to consume that bundle once, in memory, before provider construction, +5. prove that the managed gateway process environment no longer contains `SUBSTRATE_LLM_BACKEND_AUTH_*`, `OPENAI_API_KEY`, or equivalent secret-bearing values by default, +6. preserve the current policy precedence, failure taxonomy, lifecycle semantics, and restart/rotation behavior. + +The user-visible result is simple: + +- `substrate world gateway sync` and `restart` still work, +- policy-denied, invalid-integration, unavailable, and transient-failure behavior stay distinct, +- but the in-world gateway no longer receives integrated auth secrets in its process environment by default. + +## Locked Starting State + +### What is already done + +The following work is already landed and is not reopened here: + +- host-side auth sourcing and policy enforcement in [crates/shell/src/builtins/world_gateway.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/builtins/world_gateway.rs) +- typed `integrated_auth` payload validation in [crates/agent-api-types/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/agent-api-types/src/lib.rs) +- backend selection and integrated gateway runtime binding in [crates/world-agent/src/gateway_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/gateway_runtime.rs) +- operator command family and exit-code contract in [docs/contracts/substrate-gateway-operator-contract.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/contracts/substrate-gateway-operator-contract.md) +- policy precedence and fail-closed auth sourcing rules in [docs/contracts/substrate-gateway-policy-evaluation.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/contracts/substrate-gateway-policy-evaluation.md) + +### Exact remaining gap + +The remaining gap is concrete: + +1. the shell builds `GatewayLifecycleRequestV1.integrated_auth` correctly in [crates/shell/src/builtins/world_gateway.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/builtins/world_gateway.rs:339) +2. `world-agent` validates the selected backend and request payload correctly in [crates/world-agent/src/service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs:1528) +3. `world-agent` still resolves that payload into `ResolvedGatewayAuthHandoff { env_vars }` in [crates/world-agent/src/gateway_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/gateway_runtime.rs:1097) +4. `start_runtime(...)` still injects those secret values into the child process environment in [crates/world-agent/src/gateway_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/gateway_runtime.rs:706) +5. `substrate-gateway` still expects integrated auth from env-backed seams: + - Codex integrated auth reads `SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_*` from env in [crates/gateway/src/auth/codex_auth_context.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/gateway/src/auth/codex_auth_context.rs:43) + - API-key providers still resolve `$OPENAI_API_KEY` style placeholders from process env during config load in [crates/gateway/src/cli/mod.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/gateway/src/cli/mod.rs:350) +6. runtime parity and fake gateways still encode the old posture in: + - [crates/world-agent/src/gateway_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/gateway_runtime.rs) + - [crates/world-agent/tests/gateway_runtime_parity.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/tests/gateway_runtime_parity.rs) + - [crates/gateway/tests/openai_shared_parity.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/gateway/tests/openai_shared_parity.rs) + +This means the trust boundary is still technically accurate at the shell and request layer, but false at the last mile. + +## Frozen Execution Contract + +This section is not interpretation space. + +### Non-negotiable invariants + +1. Host-side secret sourcing and policy precedence do not change. +2. `GatewayLifecycleRequestV1.integrated_auth` remains the only host-to-world auth seam. +3. The production integrated path stops placing secret-bearing values in the managed gateway process env by default. +4. The only new stable env surface for this slice is the non-secret pointer env var `SUBSTRATE_LLM_AUTH_BUNDLE_FD`. +5. The auth bundle is written once, read once, closed promptly, and never persisted to disk. +6. Canonical bundle field names stay in the `SUBSTRATE_LLM_BACKEND_AUTH_*` family even when values are no longer env vars. +7. Integrated runtime startup fails closed if the bundle is missing, malformed, unreadable, incomplete, or not consumed. +8. `sync` and `restart` always deliver a fresh bundle to the replacement process instance. +9. `host_only` gateway mode remains a bounded compatibility path and is not redefined by this slice. +10. No new public control plane, request family, or host credential source is introduced. + +### Shared schema ownership + +This plan freezes the bundle schema location so the implementation does not drift. + +- `crates/common/src/gateway_auth_bundle.rs` owns: + - `SUBSTRATE_LLM_AUTH_BUNDLE_FD` + - `GatewayAuthBundleV1` + - canonical field-name constants for `cli:codex` and `api:openai` +- `world-agent` serializes that schema +- `substrate-gateway` deserializes that schema +- `agent-api-types` does not own this bundle, because the bundle is not a new public transport contract + +This is the smallest DRY answer. Duplicating string keys across `world-agent`, `gateway`, and tests is a drift trap. Moving the schema into `agent-api-types` would incorrectly turn a private last-mile carrier into a public host/world API. + +### Chosen bundle contract + +The inherited FD/pipe carries one JSON document, written by `world-agent` and read to EOF by `substrate-gateway`: + +```json +{ + "schema_version": 1, + "backend_id": "cli:codex", + "fields": { + "SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCOUNT_ID": "acct_...", + "SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCESS_TOKEN": "header.payload.signature" + } +} +``` + +For `api:openai`, the canonical field key is: + +- `SUBSTRATE_LLM_BACKEND_AUTH_API_OPENAI_API_KEY` + +Required rules: + +1. `fields` keys are canonical auth field names, not raw host env names. +2. The host env name `OPENAI_API_KEY` remains only the shell-side source of truth, never the in-world delivery key. +3. The gateway reads the bundle exactly once during integrated startup, closes the FD, and retains only in-memory auth state. +4. Any missing required field for the selected backend is invalid integration at request-mapping time and a startup failure if a malformed bundle somehow reaches the child. +5. Re-reading after the startup read is not part of the contract. Restart gets a fresh bundle instead. + +### Chosen startup sequencing + +This plan also freezes the gateway startup order. The current order is wrong for integrated mode because config env interpolation runs before any integrated auth can be injected. + +The new required order is: + +1. resolve `GatewayLaunchContract` +2. parse config file without resolving provider secret placeholders yet +3. if `GatewayMode::InWorld`, read and validate `GatewayAuthBundleV1` +4. overlay integrated auth into in-memory gateway state: + - populate integrated Codex auth context + - patch API-key provider config in memory +5. resolve remaining non-secret env placeholders +6. construct `ProviderRegistry` +7. start serving traffic + +That means `crates/gateway/src/main.rs` and [crates/gateway/src/cli/mod.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/gateway/src/cli/mod.rs) must move from the current one-step `AppConfig::from_file(...) -> resolve_env_vars()` flow to a two-phase load for integrated mode. + +### Failure taxonomy freeze + +The plan does not leave error classification fuzzy. + +Request-layer failures remain what they already are: + +- policy-denied host secret reads stay policy denials +- selected-backend / payload mismatches stay invalid integration +- incomplete request-provided auth payloads stay invalid integration + +Bundle-transport failures are runtime failures, not auth-policy failures: + +- pipe creation failure +- bundle serialization failure +- partial write / write failure +- child spawn failure +- missing pointer env in integrated mode +- unreadable inherited FD +- malformed JSON in the bundle +- child startup failure after bundle read + +Those failures surface as transient integrated-startup failures. They are real bugs or runtime faults after the request already passed validation. They are not user policy mistakes. + +## Step 0: Scope Challenge + +### 0A. What already exists + +| Sub-problem | Existing code | Decision | +| --- | --- | --- | +| host-side auth precedence and policy gates | [crates/shell/src/builtins/world_gateway.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/builtins/world_gateway.rs), [crates/shell/tests/world_gateway.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/world_gateway.rs) | Reuse exactly. No precedence or policy behavior changes. | +| typed gateway lifecycle request contract | [crates/agent-api-types/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/agent-api-types/src/lib.rs) | Reuse exactly. No new request family. | +| managed gateway runtime root, config, logs, manifests, readiness loop | [crates/world-agent/src/gateway_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/gateway_runtime.rs) | Reuse. Swap only the secret carrier and startup contract. | +| gateway launch-mode and token-store contract | [crates/gateway/src/launch.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/gateway/src/launch.rs) | Reuse. Extend integrated mode with auth-bundle consumption. | +| Codex integrated auth resolution | [crates/gateway/src/auth/codex_auth_context.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/gateway/src/auth/codex_auth_context.rs) | Change consumer from env to startup-owned in-memory context for integrated mode only. | +| provider registry construction | [crates/gateway/src/providers/registry.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/gateway/src/providers/registry.rs), [crates/gateway/src/server/mod.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/gateway/src/server/mod.rs) | Reuse. Feed resolved integrated auth into this seam before providers are built. | +| canonical auth field-name family | [docs/project_management/packs/PHASE_8_CROSS_CUTTING_DECISION_REGISTRY.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/project_management/packs/PHASE_8_CROSS_CUTTING_DECISION_REGISTRY.md:250) | Reuse. Make code finally match the documented canonical names. | + +### 0B. Minimum honest diff + +The minimum honest implementation is: + +1. keep shell request shape and policy logic unchanged +2. replace `ResolvedGatewayAuthHandoff.env_vars` with a canonical auth-bundle model +3. pass the bundle to the child via inherited FD plus `SUBSTRATE_LLM_AUTH_BUNDLE_FD` +4. split gateway startup into parse -> integrated overlay -> remaining env resolution -> provider construction +5. remove test and doc assumptions that the gateway receives secret env vars + +Anything smaller is fake progress. + +In particular: + +- changing only the Codex auth helper is insufficient because `api:openai` still fails earlier in config env resolution +- changing only provider config interpolation is insufficient because integrated Codex auth still reads env +- writing bundle contents back into env inside the gateway just recreates the original bug one layer later + +### 0C. Complexity check + +This slice touches more than 8 files and multiple crates. That is justified and still minimal. + +Expected production surfaces: + +1. `crates/common/src/gateway_auth_bundle.rs` +2. `crates/common/Cargo.toml` +3. `crates/gateway/Cargo.toml` +4. `crates/world-agent/src/gateway_runtime.rs` +5. `crates/world-agent/tests/gateway_runtime_parity.rs` +6. `crates/gateway/src/main.rs` +7. `crates/gateway/src/cli/mod.rs` +8. `crates/gateway/src/auth/codex_auth_context.rs` +9. `crates/gateway/src/launch.rs` +10. `crates/gateway/src/server/mod.rs` +11. `crates/gateway/src/providers/registry.rs` +12. `crates/gateway/tests/openai_shared_parity.rs` +13. `crates/shell/tests/world_gateway.rs` +14. `docs/contracts/substrate-gateway-policy-evaluation.md` +15. `crates/gateway/docs/contracts/chatgpt-codex-auth-handoff-contract.md` +16. `AGENT_ORCHESTRATION_GAP_MATRIX.md` + +This is the minimal real blast radius because the change spans: + +- producer contract (`world-agent`) +- consumer contract (`substrate-gateway`) +- regression floor +- operator/documentation truth + +### 0D. Search and completeness check + +Search-before-building result: + +- **[Layer 1]** reuse the existing typed `integrated_auth` request, do not invent a second request family +- **[Layer 1]** reuse the documented canonical `SUBSTRATE_LLM_BACKEND_AUTH_*` field-name family instead of raw provider env names +- **[Layer 1]** reuse inherited FD/pipe delivery, not temp files, named pipes, or persisted bundle artifacts +- **[Layer 1]** reuse the gateway launch contract and provider registry seams, inject integrated auth there instead of scattering startup reads across providers +- **[EUREKA]** the real seam is not only `CodexIntegratedAuthHandoff::from_env()`. The earlier failure is config load in `AppConfig::from_file()`. If that order stays intact, `api:openai` still dies before the bundle can help + +Shortcut options explicitly rejected: + +- keep `OPENAI_API_KEY` in the child env and only remove Codex env vars +- read bundle contents into env vars inside the gateway before provider init +- persist auth bundles under the runtime dir and point the gateway at files +- keep env injection as an automatic fallback if bundle consumption fails + +### 0E. Distribution and runtime contract check + +No new distributable artifact type is introduced. This is runtime-contract work. + +That means this plan must cover: + +- `world-agent` runtime launch behavior +- `substrate-gateway` integrated startup behavior +- Linux parity tests and shell contract tests +- canonical docs and gap-matrix closeout + +No new installer or release-pipeline work is required beyond keeping existing binaries buildable and testable on Linux. + +### 0F. NOT in scope + +- redesigning `GatewayLifecycleRequestV1` +- changing host-side policy precedence or expanding auth sources +- generic token-store redesign in `substrate-gateway` +- adding support for new `api:*` backends beyond the current `api:openai` floor +- cross-platform parity beyond explicit Linux-first validation +- new operator commands or status schema families +- a compatibility env-fallback flag unless a real blocker appears during implementation + +## Architecture Review + +### Locked architecture decisions + +`PLAN-14` does not leave the core implementation path open-ended. + +1. `world-agent` owns bundle creation, serialization, FD inheritance, and child-env cleanup. +2. `substrate-gateway` owns one-time bundle consumption and in-memory auth hydration. +3. `substrate-common` owns the shared bundle schema and canonical field constants. +4. integrated-mode startup owns auth overlay before provider construction. +5. provider code consumes already-resolved auth context. Provider code does not become the owner of host/world secret delivery. + +This is the smallest honest architecture. The request seam is already good. The bug is the last-mile carrier and the gateway startup order. + +### Architecture findings resolved in-plan + +[P1] (confidence: 10/10) [crates/world-agent/src/gateway_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/gateway_runtime.rs:1097) still models the handoff as env assignments, which guarantees the carrier stays env-based no matter how correct the request contract is. + +Resolution in this plan: + +- replace `ResolvedGatewayAuthHandoff { env_vars }` with a bundle payload model +- make secret-bearing child env injection impossible on the default integrated path + +[P1] (confidence: 10/10) [crates/gateway/src/cli/mod.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/gateway/src/cli/mod.rs:350) resolves `$OPENAI_API_KEY` during config load, before integrated startup has any chance to supply API-key auth another way. + +Resolution in this plan: + +- split config load into parse-first and resolve-later phases +- overlay integrated API-key auth before generic env interpolation runs + +[P1] (confidence: 9/10) [crates/gateway/src/auth/codex_auth_context.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/gateway/src/auth/codex_auth_context.rs:43) still treats env as the source of truth for integrated Codex auth. + +Resolution in this plan: + +- change integrated Codex auth to resolve from a startup-owned in-memory context +- keep standalone local auth-file behavior for `host_only` +- preserve explicit-account-id-over-JWT precedence exactly as today + +[P1] (confidence: 9/10) [crates/world-agent/src/gateway_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/gateway_runtime.rs:706) is the lifecycle seam that must guarantee fresh delivery and prompt cleanup. If bundle creation is bolted on somewhere else, restart correctness will drift. + +Resolution in this plan: + +- make bundle creation part of `start_runtime(...)` +- make `sync` and `restart` pass through the same fresh-bundle creation path +- fail before readiness probing if bundle creation or write fails + +### Architecture ASCII diagrams + +```text +CURRENT STATE +============= +shell + ├── resolve_integrated_auth_payload(...) + └── GatewayLifecycleRequestV1 { integrated_auth } + │ + ▼ +world-agent + ├── resolve_integrated_auth_handoff(...) + │ └── Vec<(env_key, secret_value)> + └── start_runtime(...) + └── child.env(secret_key, secret_value) + │ + ▼ +substrate-gateway + ├── AppConfig::from_file(...) + │ └── resolve_env_vars() demands $OPENAI_API_KEY + └── CodexIntegratedAuthHandoff::from_env() + +Result: + integrated auth still lives in the gateway process environment +``` + +```text +TARGET STATE +============ +shell + ├── host policy-gated secret sourcing + └── GatewayLifecycleRequestV1 { integrated_auth } + │ + ▼ +world-agent + ├── validate selected backend + request payload + ├── map request payload -> canonical auth fields + ├── serialize GatewayAuthBundleV1 + ├── create one-time pipe + ├── write JSON bundle to write end + ├── pass read end to child + └── child env: + SUBSTRATE_LLM_AUTH_BUNDLE_FD= + │ + ▼ +substrate-gateway startup + ├── parse config without secret interpolation + ├── read bundle once from FD + ├── close FD + ├── hydrate in-memory integrated auth context + ├── patch integrated API-key config in memory + └── build ProviderRegistry from resolved in-memory state + +Result: + secret values live in memory only, not child env +``` + +```text +INTEGRATED STARTUP ORDER +======================== +main.rs + ├── GatewayLaunchContract::resolve(...) + ├── AppConfig::parse_file_without_secret_resolution(...) + ├── if InWorld: + │ ├── read GatewayAuthBundleV1 from SUBSTRATE_LLM_AUTH_BUNDLE_FD + │ ├── build IntegratedGatewayAuthContext + │ └── overlay provider/auth state in memory + ├── resolve remaining non-secret env placeholders + └── start_server(config, launch, integrated_auth_ctx) +``` + +### Required field mapping + +| Selected backend | Host-side source name(s) | Canonical bundle field(s) | Consumer | +| --- | --- | --- | --- | +| `cli:codex` | `SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCOUNT_ID`, `SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCESS_TOKEN`, or `~/.codex/auth.json` fallback allowed by policy | same canonical `SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_*` names | startup-owned `IntegratedGatewayAuthContext`, then [crates/gateway/src/auth/codex_auth_context.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/gateway/src/auth/codex_auth_context.rs) | +| `api:openai` | host `OPENAI_API_KEY` | `SUBSTRATE_LLM_BACKEND_AUTH_API_OPENAI_API_KEY` | integrated provider-config overlay before [ProviderRegistry::from_configs_with_models_and_mode(...)](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/gateway/src/providers/registry.rs) | + +## Code Quality Review + +### Findings resolved in-plan + +[P2] (confidence: 9/10) `world-agent` and `gateway` need the same bundle schema. Hand-rolled JSON keys in both crates plus tests would be a drift trap. + +Resolution: + +- put the serde schema and canonical field constants in `crates/common/src/gateway_auth_bundle.rs` +- do not duplicate free-form string literals across three modules + +[P2] (confidence: 9/10) integrated auth and generic config env interpolation are different concerns. Forcing one helper to do both keeps the secret path implicit and hard to audit. + +Resolution: + +- keep generic env interpolation for `host_only` and non-secret config +- add one explicit integrated-auth overlay step before provider construction +- make the integrated secret path obvious in startup code + +[P2] (confidence: 8/10) runtime parity fixtures currently encode the wrong security posture. Repeating slightly different shell fragments across tests will make the rewrite brittle. + +Resolution: + +- allow one test-only fake-gateway helper that reads `SUBSTRATE_LLM_AUTH_BUNDLE_FD` +- centralize assertions for pointer-env presence, secret-env absence, and read-once behavior + +### Allowed code shape + +1. one shared bundle-schema module in `substrate-common` +2. one startup-owned `IntegratedGatewayAuthContext` inside `substrate-gateway` +3. one explicit integrated provider-overlay step before provider construction +4. no public request-shape changes +5. no second integrated auth source inside the gateway +6. no temp-file bundle staging +7. no silent env fallback on the integrated path + +## Test Review + +### Test framework detection + +- Runtime: Rust +- Framework: `cargo test` +- Primary packages: `world-agent`, `substrate-gateway`, `shell` +- No LLM eval suite is required for this slice + +### Code path coverage + +```text +CODE PATH COVERAGE +=========================== +[+] world-agent request -> bundle mapping + │ + ├── [GAP] typed cli:codex payload -> canonical bundle fields + ├── [GAP] typed api:openai payload -> canonical bundle field + ├── [GAP] selected-backend mismatch fails invalid_integration + └── [GAP] empty/partial payload fails invalid_integration + +[+] world-agent spawn plumbing + │ + ├── [GAP] [->E2E] read end inherited by child, write end stays parent-only + ├── [GAP] secret env vars removed from child + ├── [GAP] bundle write failure fails before readiness loop + └── [GAP] restart delivers a fresh bundle and closes the old one + +[+] gateway integrated startup + │ + ├── [GAP] config parses before integrated auth overlay + ├── [GAP] pointer env present and readable + ├── [GAP] missing pointer env fails integrated startup + ├── [GAP] malformed JSON fails integrated startup + ├── [GAP] codex integrated auth resolves from bundle, not env + └── [GAP] api:openai provider config resolves from bundle, not env interpolation + +[+] contract and parity protection + │ + ├── [GAP] shell policy precedence unchanged + ├── [GAP] gateway env no longer contains secret-bearing auth fields + ├── [GAP] raw host env name OPENAI_API_KEY absent from child env + └── [GAP] operator-facing failure buckets stay distinct + +───────────────────────────────── +COVERAGE: 0/14 auth-carrier paths proven +GAPS: 14 paths need coverage before closeout +───────────────────────────────── +``` + +### User flow coverage + +```text +USER FLOW COVERAGE +=========================== +[+] cli:codex integrated sync + ├── [GAP] [->E2E] sync -> bundle read -> gateway ready + ├── [GAP] explicit account_id survives through bundle + └── [GAP] JWT fallback still works when explicit account_id is absent + +[+] api:openai integrated sync + ├── [GAP] [->E2E] sync -> config parse -> bundle overlay -> provider ready + ├── [GAP] no OPENAI_API_KEY in child env + └── [GAP] missing API-key field fails startup cleanly + +[+] restart / rotation + ├── [GAP] [->E2E] restart launches replacement process with fresh bundle + ├── [GAP] old bundle FD is closed and cannot be reused + └── [GAP] lifecycle status remains available after replacement + +[+] failure posture + ├── [GAP] malformed bundle -> transient startup failure + ├── [GAP] pointer missing or unreadable -> transient startup failure + └── [GAP] no silent fallback to secret env injection +``` + +### Required tests to add or extend + +1. `crates/world-agent/src/gateway_runtime.rs` + - replace env-oriented unit tests around `resolve_integrated_auth_handoff(...)` + - add canonical field-mapping tests for `cli:codex` and `api:openai` + - add bundle serialization tests against `GatewayAuthBundleV1` + +2. `crates/world-agent/tests/gateway_runtime_parity.rs` + - rewrite the fake gateway startup to read `SUBSTRATE_LLM_AUTH_BUNDLE_FD` + - prove pointer-env presence, secret-env absence, read-once behavior, restart rotation, and fail-closed startup + +3. `crates/gateway/src/main.rs` and `crates/gateway/src/cli/mod.rs` + - add tests proving integrated mode parses config before secret interpolation + - add tests proving host-only mode keeps the existing env-resolution behavior + +4. `crates/gateway/src/auth/codex_auth_context.rs` + - add integrated bundle-backed auth-resolution tests + - keep standalone local-path tests intact + - verify explicit account-id wins over JWT fallback exactly as before + +5. `crates/gateway/src/server/mod.rs` and `crates/gateway/src/providers/registry.rs` + - add startup tests that integrated mode fails without pointer env or with malformed bundle + - add integrated startup tests that patch API-key providers in memory + +6. `crates/gateway/tests/openai_shared_parity.rs` + - replace integrated env-assumption tests with bundle-backed parity coverage + - verify `ChatGPT-Account-ID` behavior still works without env-based secret delivery + +7. `crates/shell/tests/world_gateway.rs` + - keep existing host-side precedence, allowlist, partial-auth, and exit-code tests green + - add assertions only if response wording changes, not for carrier mechanics + +### Regression rule for this slice + +Any path that can satisfy all three conditions below is a critical gap and must get a regression test: + +1. request validation has already succeeded +2. integrated auth delivery can still be wrong or absent +3. operator-visible status could still suggest the runtime is healthy + +## Failure Modes Registry + +| New codepath | Real production failure | Test covers it? | Error handling exists? | User sees clear error? | Critical gap? | +| --- | --- | --- | --- | --- | --- | +| request -> canonical mapping | `api:openai` still maps to raw `OPENAI_API_KEY` instead of canonical bundle field | no | no | no | yes | +| bundle creation | pipe/FD creation fails before spawn | no | partial | partial | yes | +| bundle write | parent writes partial JSON then child starts | no | no | no | yes | +| child env cleanup | child still receives `SUBSTRATE_LLM_BACKEND_AUTH_*` or `OPENAI_API_KEY` | no | no | no | yes | +| config parse / overlay order | config still resolves `$OPENAI_API_KEY` before the bundle can be read | no | no | no | yes | +| gateway startup read | pointer env missing or unreadable | no | partial | partial | yes | +| gateway JSON parse | malformed bundle becomes generic startup crash | no | partial | partial | yes | +| codex integrated auth | gateway falls back to env in integrated mode | no | no | no | yes | +| api-key provider init | config interpolation still demands secret env in integrated mode | no | no | no | yes | +| restart rotation | replacement process reuses stale bundle or stale in-memory auth | no | no | no | yes | + +## Performance Review + +This is a correctness-first slice. The performance rules are still simple and important: + +1. read the auth bundle once at startup, not per request +2. keep the bundle small, one JSON document with the selected backend only +3. avoid any on-disk bundle persistence or retry loops that turn startup into file IO +4. keep provider request paths unchanged after startup hydration + +There is no throughput feature here. The risk is startup complexity, not steady-state QPS. + +## DX Guardrails + +This is a developer tool. Failure messages need to be precise. + +Required error-message posture: + +1. `world-agent` must say whether the failure was request validation, bundle creation, bundle write, child spawn, or readiness failure +2. integrated gateway startup must say whether the problem was missing pointer env, unreadable FD, malformed bundle, or missing required field for the selected backend +3. operator-visible diagnostics may name: + - selected backend id + - runtime log paths + - manifest path + - pointer env var name +4. operator-visible diagnostics must not print: + - bundle contents + - canonical secret field values + - raw host env names plus resolved secret values + +## Worktree Parallelization Strategy + +There is one safe parallel window here, but only after the parent freezes the bundle contract. Before that point, parallel work would just create schema drift. + +### Dependency table + +| Step | Modules touched | Depends on | +| --- | --- | --- | +| P0. Freeze bundle schema, canonical field mapping, pointer env name, startup order, and failure taxonomy | `llm-last-mile/`, `crates/common/`, `crates/world-agent/`, `crates/gateway/src/main.rs`, `crates/gateway/src/cli/`, `docs/contracts/` | - | +| L1. Producer lane, world-agent bundle generation and spawn plumbing | `crates/world-agent/` | P0 | +| L2. Consumer lane, gateway startup refactor and in-memory auth overlay | `crates/gateway/` | P0 | +| L3. Regression/docs lane, parity tests and contract-doc closeout | `crates/world-agent/tests/`, `crates/gateway/tests/`, `crates/shell/tests/`, `docs/contracts/`, `AGENT_ORCHESTRATION_GAP_MATRIX.md` | L1, L2 | +| P1. Parent proof wall and closeout | `crates/world-agent/`, `crates/gateway/`, `crates/shell/`, `docs/` | L3 | + +### Parallel lanes + +- Lane A: `P0`, sequential, parent-owned + - write scope: `PLAN-14`, `crates/common/src/gateway_auth_bundle.rs`, any shared constants needed by both code lanes +- Lane B: `L1`, world-agent producer lane + - write scope: `crates/world-agent/**` +- Lane C: `L2`, gateway consumer lane + - write scope: `crates/gateway/**` +- Lane D: `L3`, regression/docs lane after B and C merge + - write scope: tests + docs only +- Lane E: `P1`, sequential proof wall and closeout + +### Execution order + +1. Parent freezes the shared bundle schema, pointer env name, canonical field names, startup order, and error taxonomy. +2. Launch Lane B and Lane C in parallel worktrees. +3. Merge Lane B first only if it changes shared constants or serialization details Lane C consumes. Otherwise merge whichever lane stabilizes first. +4. Run Lane D only after both code lanes merge. The regression floor must lock the final contract, not an intermediate one. +5. Run the parent proof wall and closeout. + +### Conflict flags + +- Lane B and Lane C must not invent different `GatewayAuthBundleV1` schemas +- only one place may define `SUBSTRATE_LLM_AUTH_BUNDLE_FD` +- only one place may define `SUBSTRATE_LLM_BACKEND_AUTH_API_OPENAI_API_KEY` +- Lane C must not preserve secret-env fallback as an invisible compatibility path +- Lane D must not encode fake gateway fixtures against intermediate env behavior +- if the shared schema starts spreading into `agent-api-types`, stop and re-evaluate. That is over-scoping the slice + +### Parallelization verdict + +Two code lanes can run in parallel after one parent-owned contract freeze. Worker cap should stay `2`. + +## Implementation Sequence + +### Step 1. Freeze the shared auth-bundle contract + +Files: + +- [crates/common/src/gateway_auth_bundle.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/common/src/gateway_auth_bundle.rs) +- [crates/common/Cargo.toml](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/common/Cargo.toml) +- [crates/gateway/Cargo.toml](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/gateway/Cargo.toml) +- [crates/world-agent/src/gateway_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/gateway_runtime.rs) +- [crates/gateway/src/main.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/gateway/src/main.rs) +- [crates/gateway/src/cli/mod.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/gateway/src/cli/mod.rs) + +Work: + +1. add `GatewayAuthBundleV1` and the canonical field constants to `substrate-common` +2. freeze `SUBSTRATE_LLM_AUTH_BUNDLE_FD` there too +3. freeze the `cli:codex` and `api:openai` mapping rules +4. freeze the integrated startup order: parse -> read bundle -> overlay -> resolve remaining env -> build registry +5. freeze the transient-vs-invalid failure split for the carrier + +Validation gate: + +- one schema +- one pointer env name +- one canonical field mapping +- one startup order +- one failure taxonomy + +### Step 2. Replace world-agent env injection with bundle delivery + +Files: + +- [crates/world-agent/src/gateway_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/gateway_runtime.rs) +- [crates/world-agent/tests/gateway_runtime_parity.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/tests/gateway_runtime_parity.rs) + +Work: + +1. replace `ResolvedGatewayAuthHandoff { env_vars }` with a canonical bundle model +2. map `cli:codex` and `api:openai` request payloads into canonical bundle fields +3. create the one-time pipe/FD channel during `start_runtime(...)` +4. write the JSON bundle, pass the read end to the child, export only `SUBSTRATE_LLM_AUTH_BUNDLE_FD` +5. remove secret-bearing child env assignments from the default integrated path +6. ensure spawn failure, write failure, and restart replacement all clean up FDs correctly + +Validation gate: + +- child receives only the pointer env var +- bundle write and close discipline is correct +- restart always re-delivers a fresh bundle + +### Step 3. Refactor gateway startup to consume the bundle before provider construction + +Files: + +- [crates/gateway/src/main.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/gateway/src/main.rs) +- [crates/gateway/src/cli/mod.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/gateway/src/cli/mod.rs) +- [crates/gateway/src/launch.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/gateway/src/launch.rs) +- [crates/gateway/src/server/mod.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/gateway/src/server/mod.rs) +- [crates/gateway/src/providers/registry.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/gateway/src/providers/registry.rs) +- [crates/gateway/src/auth/codex_auth_context.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/gateway/src/auth/codex_auth_context.rs) + +Work: + +1. split config load so integrated mode can parse config without resolving secret env placeholders first +2. add a startup-owned `IntegratedGatewayAuthContext` built from `GatewayAuthBundleV1` +3. read the bundle exactly once during integrated startup and close the FD immediately on success or failure +4. patch API-key bundle fields into in-memory provider config before `ProviderRegistry::from_configs_with_models_and_mode(...)` +5. change integrated Codex auth resolution from env-based lookup to the injected in-memory context +6. preserve standalone local token-store and auth-file behavior for `host_only` +7. fail startup before serving traffic if the bundle is missing or malformed + +Validation gate: + +- `api:openai` no longer depends on process-env interpolation in integrated mode +- `cli:codex` no longer depends on process env in integrated mode +- host-only behavior remains unchanged + +### Step 4. Rewrite the regression floor and close the docs gap + +Files: + +- [crates/world-agent/tests/gateway_runtime_parity.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/tests/gateway_runtime_parity.rs) +- [crates/gateway/tests/openai_shared_parity.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/gateway/tests/openai_shared_parity.rs) +- [crates/shell/tests/world_gateway.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/world_gateway.rs) +- [docs/contracts/substrate-gateway-policy-evaluation.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/contracts/substrate-gateway-policy-evaluation.md) +- [crates/gateway/docs/contracts/chatgpt-codex-auth-handoff-contract.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/gateway/docs/contracts/chatgpt-codex-auth-handoff-contract.md) +- [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md) + +Work: + +1. invert env-oriented parity tests into bundle-oriented parity tests +2. prove pointer-env presence, secret-env absence, read-once behavior, and restart rotation +3. keep shell precedence and failure-taxonomy tests green +4. update docs so the old "preferred direction" becomes the landed default integrated path +5. close the gap-matrix row only after tests and docs match the code + +Validation gate: + +- regression floor protects the new posture +- docs and matrix no longer describe env injection as current behavior + +### Step 5. Parent proof wall and closeout + +Work: + +1. run the Rust unit and integration test wall +2. run one Linux manual proof that the gateway env no longer contains secret-bearing auth values +3. run `sync`, `status --json`, and `restart` against the integrated path +4. capture closeout evidence for the gap-matrix row + +Closeout is not done until the code, tests, and docs all agree that the default integrated path is bundle-based. + +## Recommended Verification Commands + +Run in this order. + +```bash +cargo test -p world-agent gateway_runtime -- --nocapture +cargo test -p world-agent --test gateway_runtime_parity -- --nocapture +cargo test -p substrate-gateway codex_auth_context -- --nocapture +cargo test -p substrate-gateway openai_shared_parity -- --nocapture +cargo test -p shell --test world_gateway -- --nocapture +``` + +Linux manual proof after the test wall: + +```bash +substrate world gateway sync +substrate world gateway status --json +substrate world gateway restart +``` + +Required manual assertions for the integrated runtime: + +1. status returns `available` +2. the managed gateway process environment contains `SUBSTRATE_LLM_AUTH_BUNDLE_FD` +3. the managed gateway process environment does not contain: + - `SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCOUNT_ID` + - `SUBSTRATE_LLM_BACKEND_AUTH_CLI_CODEX_ACCESS_TOKEN` + - `SUBSTRATE_LLM_BACKEND_AUTH_API_OPENAI_API_KEY` + - `OPENAI_API_KEY` +4. restart produces a fresh process instance and still reaches `available` + +## Definition of Done + +1. the shell still resolves and sends typed `integrated_auth` under existing policy gates +2. `world-agent` no longer models the default integrated handoff as child env assignments +3. `world-agent` launches the managed gateway with a read-once auth bundle and only the non-secret pointer env var +4. `substrate-gateway` integrated startup parses config before secret interpolation, reads the bundle once, and closes the FD promptly +5. `substrate-gateway` integrated `cli:codex` auth no longer depends on process env +6. `substrate-gateway` integrated `api:openai` startup no longer depends on process-env interpolation +7. the managed gateway process environment no longer contains secret-bearing auth values by default +8. `sync` and `restart` both re-deliver fresh auth without persistence +9. shell policy tests still prove precedence and failure taxonomy are unchanged +10. docs and gap matrix describe bundle delivery as the landed integrated default path + +## Deferred Work + +- additional `api:*` canonical field expansions beyond `api:openai` +- optional explicit compatibility env-fallback mode, only if a real blocker appears later +- cross-platform parity beyond Linux-first validation +- broader gateway auth-source refactors unrelated to this carrier swap + +No new `TODOS.md` entry is required yet. These are explicit deferrals, not forgotten work. + +## Completion Summary + +- Step 0: scope accepted as a narrow last-mile secret-carrier hardening slice +- Architecture Review: 4 core architecture seams frozen, no remaining contract ambiguity +- Code Quality Review: 3 structural drift risks identified and resolved in-plan +- Test Review: diagrams produced, 14 codepath gaps identified +- Performance Review: startup-only overhead accepted, 0 throughput redesign +- NOT in scope: written +- What already exists: written +- TODOS.md updates: 0 durable TODOs proposed +- Failure modes: 10 critical gaps identified for implementation proof +- Outside voice: not used for this document generation +- Parallelization: 5 execution phases, 1 real parallel window, worker cap stays `2` +- Lake Score: complete option chosen over partial carrier swaps, temp-file bundles, or env fallback + + +## Decision Audit Trail + +| # | Phase | Decision | Classification | Principle | Rationale | Rejected | +| --- | --- | --- | --- | --- | --- | --- | +| 1 | Scope | Keep shell request shape and host policy logic unchanged | Mechanical | Minimal diff | The request contract is already correct; the lie is in the last-mile carrier | New request family or shell-side redesign | +| 2 | Shared schema | Put `GatewayAuthBundleV1` and canonical field constants in `substrate-common` | Mechanical | DRY | Two runtime crates and the regression floor need one shared internal truth | Duplicated JSON builders, `agent-api-types`, or a new micro-crate | +| 3 | Bundle format | Use one read-once JSON bundle over inherited FD/pipe | Mechanical | Explicit over clever | JSON is auditable, small, and fits the closed auth field set | Temp files, sockets, or opaque binary blobs | +| 4 | Canonical names | Map `api:openai` to `SUBSTRATE_LLM_BACKEND_AUTH_API_OPENAI_API_KEY` | Mechanical | Completeness | Docs already define the canonical family, and raw host env names should stay host-only | Keeping raw `OPENAI_API_KEY` as the in-world delivery key | +| 5 | Gateway startup | Split startup into parse -> bundle overlay -> remaining env resolution -> provider construction | Mechanical | Pragmatic | This is the earliest honest place to break env dependence for both API-key and Codex integrated auth | Leaving `AppConfig::from_file()` env resolution first | +| 6 | Auth source | Make integrated Codex auth resolve from startup-owned in-memory context | Mechanical | Explicit over clever | Integrated auth should be injected once, not rediscovered from process env later | `from_env()` on the integrated path | +| 7 | Fallback posture | No silent fallback to env injection on the integrated path | Mechanical | Completeness | Silent fallback recreates the original security gap | Warning-only or automatic compatibility fallback | +| 8 | Parallelization | Run `world-agent` producer and gateway consumer lanes in parallel after the parent contract freeze | Taste | Pragmatic | That is the widest safe parallel window without schema drift | Fully serial execution or three overlapping code lanes | diff --git a/llm-last-mile/PLAN-15.md b/llm-last-mile/PLAN-15.md new file mode 100644 index 000000000..4f249e330 --- /dev/null +++ b/llm-last-mile/PLAN-15.md @@ -0,0 +1,911 @@ +# PLAN-15: Land Explicit Targeted REPL Agent Turns With Backend-Routed Session Resume + +Source SOW: [15-targeted-repl-agent-turns-linux-first.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/15-targeted-repl-agent-turns-linux-first.md) +Gap matrix anchor: [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md) +Branch: `feat/session-centric-state-store` +Base branch: `main` +Plan type: Linux-first targeted REPL agent-turn plan, developer-facing shell UX, cross-crate (`shell` + `agent-api-types` + `agent-api-client` + `world-agent`), no UI scope +Review posture: `/autoplan` scope discipline with `/plan-eng-review` depth, rewritten as one cohesive execution plan +Status: execution-ready planning pass on 2026-05-04 +Outside voice: not used for this document generation + +## Objective + +Land one narrow, production-honest caller surface where the interactive REPL can submit explicit follow-up turns to a named backend that is already part of the live orchestration session. + +The shipped behavior is: + +1. the REPL accepts exactly one targeted-turn grammar, `:: ` +2. the shell resolves that token to one exact configured backend such as `cli:codex` +3. routing is by explicit `backend_id`, never by "the one eligible member" +4. host-scoped turns resume the shell-owned UAA session locally +5. world-scoped turns resume the world-owned UAA session through a new typed `world-agent` route +6. streamed output comes back through the REPL with backend identity attached +7. plain REPL input still means shell execution +8. `substrate -c` stays wrap mode + +This is the whole point of the slice. Not new agent product surface. Not a generalized daemon. Not a broader CLI redesign. Just explicit targeted REPL turns that are true to the ownership model already in the repo. + +## Plan Summary + +This plan does not try to inject text into the original bootstrap process. That would be fake progress. + +It standardizes targeted turns as fresh submitted control runs against the exact surfaced UAA session id that bootstrap already established. Host turns stay shell-local. World turns go through one new typed `world-agent` stream route and reuse the existing host-to-world execution seam for placement, cancel, and authoritative world binding. + +The first landing is intentionally serialized for world members: one retained world member per REPL session, explicit stop-and-start when switching backends, no hidden multiplexing. + +## Locked Starting State + +### What already exists + +The repo already has the hard parts that this slice should reuse: + +- shell-owned orchestrator bootstrap through UAA in [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) +- authoritative runtime state persistence in [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) +- canonical backend identity derivation in [crates/shell/src/execution/agent_inventory.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_inventory.rs) +- runtime realizability and protocol gating in [crates/shell/src/execution/agent_runtime/validator.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/validator.rs) +- Linux world-member bootstrap over the existing host-to-world stream seam in [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) and [crates/world-agent/src/service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs) +- retained world-member control ownership inside [crates/world-agent/src/member_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs) +- surfaced UAA session ids already persisted into runtime manifests during bootstrap +- focused test seams in [crates/shell/tests/repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs), [crates/shell/tests/support/repl_world_agent.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/support/repl_world_agent.rs), and [crates/world-agent/tests/streamed_execute_cancel_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/tests/streamed_execute_cancel_v1.rs) + +### Exact remaining gap + +The remaining gap is smaller than the SOW had to assume: + +1. the REPL only treats `:host` and `:pty` as directives on single-line input, then falls through to shell execution +2. world-member routing still uses `validate_member_selection(...)`, which fails closed on ambiguity instead of selecting the exact backend the operator named +3. both host and world bootstrap paths are fixed bootstrap prompts, not arbitrary follow-up prompt submission +4. `world-agent` only exposes launch, stream, and cancel today, not "submit a turn to the already-live member session" +5. `MemberRuntimeManager` is keyed by launch `span_id`, which is fine for launch cancel and wrong for later participant-targeted resume +6. the shell still retains only one `member_runtime: Option`, so the first targeted-turn landing must embrace one-at-a-time world-member ownership +7. `substrate -c` and a broader public `substrate agent start|resume|fork|stop` surface remain intentionally out of scope + +## Frozen Execution Contract + +This section removes the wiggle room. If implementation wants to do something else, this plan is wrong and should be revised first. + +### Non-negotiable invariants + +1. Plain REPL input remains shell execution. +2. Only explicit `:: ` input enters the targeted-agent lane. +3. `:host` and `:pty` keep their current meaning. +4. `substrate -c` remains `ShellMode::Wrap`. +5. Backend selection for targeted turns is by exact `backend_id`, never by eligible-member heuristics. +6. Targeted turns reuse surfaced UAA session identity through resume semantics. They do not replay the bootstrap prompt. +7. Host-scoped targeted turns stay shell-local. +8. World-scoped targeted turns stay Linux-first and go through `world-agent`. +9. The first landing supports at most one retained world-member runtime per REPL session at a time. +10. Cancellation remains span-based. Bootstrap lifetime spans and submitted-turn spans are different things and must stay different. + +### Caller grammar + +Grammar is frozen to one spelling: + +- accepted: `:: ` +- accepted example: `::cli:codex summarize the last failure` +- rejected: `@cli:codex ...` +- rejected: `:: cli:codex ...` +- rejected: `::cli:codex` with no prompt +- rejected: multi-line targeted turns in the first landing + +Parser rules: + +1. directive parsing only runs when `has_embedded_newlines(...) == false` +2. targeted-turn parsing runs before shell fallback execution +3. malformed targeted-turn syntax returns a REPL error, never shell execution +4. non-targeted input never partially matches and never rewrites into agent execution + +### Backend-resolution contract + +Targeted turns use one new additive selector in [crates/shell/src/execution/agent_runtime/validator.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/validator.rs): + +`validate_targeted_backend_selection(requested_backend_id, effective_config, inventory, base_policy) -> RuntimeSelectionDescriptor` + +That helper must: + +1. validate `requested_backend_id` syntax +2. resolve one exact effective inventory entry +3. require `derived_backend_id == requested_backend_id` +4. require `protocol == PURE_AGENT_PROTOCOL` +5. reuse current runtime-realizability checks +6. reuse current backend allowlist checks +7. preserve the entry's configured `execution.scope` + +It must not call `validate_member_selection(...)` internally. + +### Session-resume contract + +Targeted turns are session-level follow-up runs, not stdin injection into the original process. + +That means: + +1. bootstrap still exists only to establish runtime ownership and surfaced UAA session identity +2. each targeted turn starts a new short-lived submitted control run against that same UAA session id +3. the submitted turn carries a fresh `run_id` and `span_id` +4. `participant_id`, `backend_id`, `orchestration_session_id`, and world binding stay stable +5. both host and world paths use the exact surfaced `internal.uaa_session_id`, not "resume last" + +### Host contract + +Host behavior is intentionally narrow: + +1. the only active host-side agent runtime in this slice is the orchestrator runtime already attached to the current REPL session +2. a host-targeted turn is valid only when `requested_backend_id == retained_orchestrator_manifest.backend_id` +3. if the backend is host-scoped but does not match the active orchestrator backend for this REPL session, fail closed with a REPL-visible error +4. the shell rebuilds a gateway via [crates/shell/src/execution/agent_runtime/registry.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/registry.rs) +5. it submits `AgentWrapperRunRequest { prompt, working_dir, extensions["agent_api.session.resume.v1"] = { selector: "id", id: } }` +6. shell-side output translation must tag the targeted `backend_id` and must end with a targeted-turn completion, not a shell-command completion + +### World contract + +World behavior is also narrow: + +1. the shell never recreates world placement locally +2. `world-agent` remains the retained owner of the live world-member runtime after bootstrap +3. world submission goes through one new typed stream route +4. the route resolves the active member by stable participant identity, not launch span +5. the route validates `backend_id`, `orchestration_session_id`, `world_id`, and `world_generation` against the retained runtime +6. the route rejects submission when no surfaced `uaa_session_id` is retained +7. the route rejects concurrent submitted turns for the same participant +8. switching between world backends is explicit stop-and-start because the shell still owns only one retained `member_runtime` pointer + +### Wire contract + +The world path gets one additive typed route. It does not overload `ExecuteRequest.member_dispatch`. + +New request type in [crates/agent-api-types/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/agent-api-types/src/lib.rs): + +`MemberTurnSubmitRequestV1` + +Frozen fields: + +| Field | Meaning | +| --- | --- | +| `schema_version` | must be `1` | +| `orchestration_session_id` | active REPL orchestration session | +| `participant_id` | stable world-member participant to resume | +| `orchestrator_participant_id` | owning host orchestrator participant | +| `backend_id` | exact targeted backend | +| `run_id` | new submitted-turn run id | +| `world_id` | authoritative world id | +| `world_generation` | authoritative generation that must still match | +| `prompt` | non-empty targeted user prompt | + +New route: + +- `POST /v1/member_turn/stream` + +New client method in [crates/agent-api-client/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/agent-api-client/src/lib.rs): + +- `submit_member_turn_stream(MemberTurnSubmitRequestV1) -> Response` + +Response contract: + +1. reuse `ExecuteStreamFrame` as the NDJSON envelope so the shell does not need a second stream-decoding model +2. emit a fresh `Start { span_id }` +3. emit translated events/output for the submitted turn +4. emit `Exit { span_id }` for the submitted turn +5. keep cancel span identity aligned with that submitted-turn `span_id` + +### Member-runtime registry contract + +`MemberRuntimeManager` must stop pretending launch span is stable identity. + +The refactor is frozen to two registries: + +1. `active_members_by_participant_id` + - key: `participant_id` + - value: retained bootstrap ownership and resume context +2. `active_turns_by_span_id` + - key: submitted-turn `span_id` + - value: cancel handle plus participant association + +`ActiveMemberRuntime` must retain at least: + +- `participant_id` +- `orchestration_session_id` +- `orchestrator_participant_id` +- `backend_id` +- `world_id` +- `world_generation` +- `backend_kind` +- `binary_path` +- `working_dir` +- effective env overrides needed for resumed runs +- surfaced `uaa_session_id` +- bootstrap cancel handle +- launcher cleanup state + +### Cancel contract + +Cancel semantics are frozen because this is an easy place to lie accidentally: + +1. the bootstrap lifetime span still cancels the retained member runtime itself +2. a submitted-turn span cancels only that submitted turn +3. `world-agent` must be able to resolve both span classes correctly +4. canceling a submitted turn must not silently tear down the retained bootstrap runtime +5. if a future cleanup step intentionally tears down the retained member, that is a separate action + +### Failure taxonomy + +Request and routing failures are not interchangeable: + +- malformed targeted syntax, unknown backend id, wrong protocol, missing prompt: REPL user error +- backend not allowlisted: policy-style deny with current exit posture +- host runtime missing surfaced `uaa_session_id`: fail-closed runtime error +- host backend mismatch against active orchestrator backend: fail-closed runtime error +- world targeted turn on non-Linux: explicit Linux-first error +- world binding mismatch or stale generation: fail-closed runtime error +- participant not present in `world-agent` active registry: runtime unavailable +- concurrent submitted turn for one participant: conflict-style runtime error +- submitted-turn resume failure after validation: runtime failure, not policy failure + +## Step 0: Scope Challenge + +### 0A. What already exists + +| Sub-problem | Existing code | Decision | +| --- | --- | --- | +| REPL one-line directive interception | [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) | Reuse the existing one-line directive gate. Insert targeted-turn parsing before shell fallback. | +| canonical `backend_id` derivation and inventory resolution | [crates/shell/src/execution/agent_inventory.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_inventory.rs) | Reuse. No second backend naming layer. | +| runtime realizability and allowlist checks | [crates/shell/src/execution/agent_runtime/validator.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/validator.rs) | Reuse realizability checks and add one explicit targeted selector. | +| shell-local backend construction | [crates/shell/src/execution/agent_runtime/registry.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/registry.rs) | Reuse for host-targeted resume runs. | +| authoritative live runtime identity | [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) | Reuse. No shadow session registry. | +| shell-to-world member launch transport | [crates/shell/src/execution/routing/dispatch/world_ops.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs) and [crates/world-agent/src/service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs) | Reuse for launch only. Add one second typed route for submitted turns. | +| retained world-member control ownership | [crates/world-agent/src/member_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs) | Extend. Do not move control ownership back into the shell. | +| UAA resume semantics | external `unified-agent-api` already consumed by `shell` and `world-agent` | Reuse `agent_api.session.resume.v1`. Do not invent a bespoke prompt-submission protocol. | +| integration harnesses | [crates/shell/tests/repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs) and [crates/world-agent/tests/streamed_execute_cancel_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/tests/streamed_execute_cancel_v1.rs) | Extend. Do not create a brand-new harness unless the current stub cannot express the route. | + +### 0B. Minimum honest diff + +The minimum honest implementation is: + +1. add a targeted-turn parser to the REPL loop +2. add exact named-backend selection in `validator.rs` +3. add a shell-local host submitted-turn path that uses UAA resume against the stored session id +4. add one typed world submit-turn route plus client +5. refactor `MemberRuntimeManager` to resolve active members by participant identity and active turns by span +6. extend integration tests and the gap matrix so repo truth matches code truth + +Anything smaller is fake progress. + +Rejected shortcuts: + +- overloading `substrate -c` +- injecting prompt text into the bootstrap process stdin +- overloading `ExecuteRequest.member_dispatch` for both launch and submit +- adding a generic new agent hub daemon +- pretending simultaneous retained world members already exist in the shell + +### 0C. Complexity check + +This slice touches more than 8 files. That is still the minimal blast radius because the seam crosses: + +- REPL grammar and routing +- backend validation and policy gating +- typed host-to-world transport +- retained runtime identity inside `world-agent` +- integration tests proving shell semantics did not drift + +Primary files expected to move: + +1. [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) +2. [crates/shell/src/execution/agent_runtime/validator.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/validator.rs) +3. [crates/shell/src/execution/agent_runtime/registry.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/registry.rs) +4. [crates/shell/tests/repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs) +5. [crates/shell/tests/support/repl_world_agent.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/support/repl_world_agent.rs) +6. [crates/shell/src/execution/invocation/tests.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/invocation/tests.rs) +7. [crates/agent-api-types/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/agent-api-types/src/lib.rs) +8. [crates/agent-api-client/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/agent-api-client/src/lib.rs) +9. [crates/world-agent/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/lib.rs) +10. [crates/world-agent/src/handlers.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/handlers.rs) +11. [crates/world-agent/src/service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs) +12. [crates/world-agent/src/member_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs) +13. [crates/world-agent/tests/streamed_execute_cancel_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/tests/streamed_execute_cancel_v1.rs) +14. [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md) + +### 0D. Search and completeness check + +Search-before-building result, in practical terms: + +- **[Layer 1]** reuse UAA `agent_api.session.resume.v1` +- **[Layer 1]** reuse `build_gateway_for_descriptor(...)` for host resume runs +- **[Layer 1]** reuse the existing span-based cancel surface +- **[Layer 1]** keep `ExecuteRequest.member_dispatch` launch-only and add a separate typed request for submitted turns +- **[EUREKA]** the real world-path blocker is not parser work. It is the registry keying model in `MemberRuntimeManager` +- **[EUREKA]** the shell's single `member_runtime: Option` is a product boundary, not an annoyance to paper over + +Completeness rule for this plan: + +- support the full explicit host path +- support the full explicit Linux world path +- include the regression floor for shell input, `:host`, `:pty`, and `-c` +- do not defer error handling for malformed syntax, backend deny, stale world generation, or submitted-turn cancel + +### 0E. Distribution and runtime contract check + +No new artifact type is introduced. This is not a packaging plan. + +The ship surface is behavioral: + +- REPL grammar and error posture +- typed `world-agent` transport contract +- retained runtime ownership correctness +- regression proof that shell semantics did not drift + +### 0F. NOT in scope + +- redesigning `substrate -c` +- adding non-interactive targeted-agent syntax +- public `substrate agent start|resume|fork|stop` +- implicit default-backend routing for ordinary REPL input +- simultaneous retained world members in one REPL session +- multi-line targeted prompts +- cross-platform world-target parity beyond explicit Linux-first behavior +- transcript persistence redesign +- toolbox or gateway work as a substitute for pure-agent targeted turns + +## Architecture Review + +### Locked architecture decisions + +1. Targeted turns are explicit-only and single-line-only in v1. +2. Backend routing is exact `backend_id` routing, not ambiguity resolution. +3. Host submitted turns reuse UAA resume extensions locally. +4. World submitted turns go through a new typed `world-agent` stream route. +5. `ExecuteRequest.member_dispatch` remains launch-only. +6. The first landing supports one retained world-member runtime at a time per REPL session. + +### Architecture findings resolved in-plan + +**Issue 1. Host follow-up turns cannot be "inject into the existing process."** + +The UAA boundary exposes new control runs, cancellation, surfaced session handles, and resume extensions. It does not expose "write another prompt into the already-running control handle." This plan standardizes follow-up turns as fresh resume runs keyed to the stored session id. + +**Issue 2. Host routing has to acknowledge the real product shape.** + +There is only one active host-side agent runtime in this slice: the orchestrator already booted by the REPL. This plan makes that explicit instead of pretending there is a general pool of host participants. A host-targeted turn either matches the active orchestrator backend or fails closed. + +**Issue 3. Overloading `ExecuteRequest.member_dispatch` would collapse two different meanings into one struct.** + +Launch needs runtime descriptor and empty `cmd`. Submit needs prompt plus active participant lookup. That is not one request with optional knobs. That is two different contracts. The plan keeps them separate. + +**Issue 4. World runtime identity is keyed incorrectly for later targeted turns.** + +Launch span is good enough for bootstrap cancellation and bad for later follow-up turns. The plan fixes the registry model before layering the new route on top. + +**Issue 5. Multi-backend world targeting is larger than the current shell ownership model.** + +The shell already keeps one retained world-member pointer. This plan freezes one-at-a-time world backend ownership and makes switching explicit instead of hiding that product boundary. + +### Architecture ASCII diagrams + +### REPL targeted-turn flow + +```text +operator input + | + v +"::cli:codex summarize failures" + | + +--> parse_targeted_agent_turn(...) + | + +--> malformed? ----------> REPL error, stop + | + v + validate_targeted_backend_selection(...) + | + +--> denied / unknown / unrealizable? --> REPL error, stop + | + v + execution.scope? + / \ + /host \world + v v +submit_host_ ensure_targeted_member_runtime_ready(...) +targeted_turn(...) | + | +--> same backend + same generation? reuse + | +--> different backend? stop old member, launch requested one + | +--> non-Linux / stale binding? fail closed + | v + | POST /v1/member_turn/stream + | | + | v + | world-agent MemberRuntimeManager + | | + | v + | UAA resume run_control(...) + | | + +-------- stream translated AgentEvents / text / completion --------+ + | + v + REPL printer + trace +``` + +### World-member ownership model + +```text +shell REPL session + | + +-- authoritative runtime state store + | orchestration_session_id + | participant_id + | backend_id + | world_id/world_generation + | + +-- retained local pointer + member_runtime: Option + | + +-- at most one active world member in this slice + | + +-- points to world-agent-owned runtime + | + +-- active_members_by_participant_id + | participant_id -> bootstrap ownership + uaa_session_id + | + +-- active_turns_by_span_id + submit span -> cancel handle +``` + +### Cancel model + +```text +bootstrap launch + | + +-- bootstrap span_id ----------------------------> cancel retained member lifetime + | +submitted targeted turn + | + +-- submitted-turn span_id -----------------------> cancel this turn only +``` + +## Code Quality Review + +### Findings resolved in-plan + +**Issue 1. Parser sprawl in `async_repl.rs`.** + +Keep the parser small and local: + +- one small `TargetedAgentTurn` value type +- one parser helper +- one routed handler per scope + +No generic directive framework. No new parser crate. Minimal diff wins here. + +**Issue 2. Selection logic drift risk.** + +Backend validation already lives in `validator.rs`. This plan keeps targeted-turn routing logic there too instead of re-implementing backend-id checks inside the REPL loop. + +**Issue 3. Event translation duplication risk.** + +Host and world paths must both stamp `backend_id`, `participant_id`, `run_id`, `span_id`, and scope consistently. Add one submitted-turn translation helper or equivalent shared path. Do not let host and world invent incompatible output shapes. + +**Issue 4. Registry explicitness is more important than abstraction purity.** + +The world registry changes are structural, not cosmetic. This plan prefers a blunt, obvious `ActiveMemberRuntime` over a clever generic state machine. + +### Allowed code shape + +1. Keep targeted-turn parsing in [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs). +2. Keep backend selection in [crates/shell/src/execution/agent_runtime/validator.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/validator.rs). +3. Add only one new typed host-to-world request for submitted turns. +4. Keep `MemberRuntimeManager` explicit. No trait hierarchy. No generalized runtime state machine. +5. Do not add new global shell state beyond what the REPL already owns. + +## Test Review + +### Test framework detection + +This repo is Rust-first. The test surface for this plan is the existing Rust unit and integration suite driven by `cargo test`. + +Relevant suites already exist in: + +- [crates/shell/tests/repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs) +- [crates/world-agent/tests/streamed_execute_cancel_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/tests/streamed_execute_cancel_v1.rs) +- [crates/world-agent/tests/member_runtime_world_placement_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/tests/member_runtime_world_placement_v1.rs) +- unit tests in [crates/shell/src/execution/agent_runtime/validator.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/validator.rs) +- wrap-mode tests in [crates/shell/src/execution/invocation/tests.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/invocation/tests.rs) +- config validation tests in [crates/shell/tests/agents_validate.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agents_validate.rs) + +### Code path coverage + +```text +CODE PATH COVERAGE +=========================== +[+] crates/shell/src/repl/async_repl.rs + | + ├── parse_targeted_agent_turn() + │ ├── [GAP] Accepts exact ":: " syntax + │ ├── [GAP] Rejects missing backend id + │ ├── [GAP] Rejects missing prompt + │ ├── [GAP] Rejects multi-line targeted input + │ └── [REGRESSION] Plain shell input still falls through untouched + | + ├── validate_targeted_backend_selection(...) + │ ├── [GAP] Unknown backend id + │ ├── [GAP] Wrong protocol + │ ├── [GAP] Policy deny + │ └── [GAP] Explicit world backend chosen among multiple configured members + | + ├── submit_host_targeted_turn(...) + │ ├── [GAP] Exact surfaced session id mapped into resume extension + │ ├── [GAP] Missing stored session id fails closed + │ ├── [GAP] Host backend mismatch fails closed + │ └── [GAP] Streamed completion is surfaced as targeted-turn completion + | + └── ensure_targeted_member_runtime_ready(...) + ├── [GAP] Launches requested backend when no world member is retained + ├── [GAP] Reuses current retained backend when same backend + generation + ├── [GAP] Switches retained world backend when target differs + └── [GAP] Non-Linux path fails closed with explicit error + +[+] crates/world-agent/src/member_runtime.rs + | + ├── register_active_member(...) + │ ├── [GAP] Stores stable participant identity + surfaced session id + │ └── [GAP] Cleanup removes participant registry entry and launcher dir + | + ├── submit_turn(...) + │ ├── [GAP] Resolves by participant/backend/world identity + │ ├── [GAP] Rejects concurrent active turn for the same participant + │ ├── [GAP] Uses exact session id resume, not "last" + │ └── [GAP] Submitted-turn cancel works by returned submit span id + | + └── cancel(...) + ├── [GAP] Cancels bootstrap span + └── [GAP] Cancels submitted-turn span + +[+] crates/agent-api-types/src/lib.rs + | + └── MemberTurnSubmitRequestV1::validate() + ├── [GAP] Empty prompt rejected + ├── [GAP] Empty participant/backend rejected + └── [GAP] Round-trip serde boundary + +───────────────────────────────── +COVERAGE: 0/19 targeted paths covered today +QUALITY TARGET: every new path reaches at least ★★, and all regressions reach ★★★ +GAPS: 19 new/changed paths need tests +───────────────────────────────── +``` + +### User flow coverage + +```text +USER FLOW COVERAGE +=========================== +[+] Operator targeted host turn + | + ├── [GAP] "::cli:codex hello" routes to host backend, not shell execution + ├── [GAP] streamed output is visible in REPL + ├── [GAP] completion/failure is visibly distinct from shell command completion + └── [GAP] host backend mismatch returns explicit guidance + +[+] Operator targeted world turn + | + ├── [GAP] "::cli:codex hello" on Linux launches or reuses world member then submits turn + ├── [GAP] world backend switch tears down previous retained backend explicitly + ├── [GAP] stale world generation fails closed + └── [GAP] cancel stops the submitted turn, not the bootstrap runtime + +[+] Operator error states + | + ├── [GAP] malformed syntax explains exact fix + ├── [GAP] unknown backend suggests `substrate agent list` + ├── [GAP] denied backend names the blocked backend id + └── [GAP] non-Linux world target returns explicit Linux-first error + +[+] Regression floor + | + ├── [REGRESSION] Plain "echo hi" still executes as shell input + ├── [REGRESSION] ":host ..." still works when enabled + ├── [REGRESSION] ":pty ..." still works + └── [REGRESSION] `substrate -c` still maps to wrap mode +``` + +### Required tests to add or extend + +1. Add parser unit tests in [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) for exact syntax, missing prompt, backend-id extraction, and multi-line rejection. +2. Add unit tests in [crates/shell/src/execution/agent_runtime/validator.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/validator.rs) for exact backend selection, deny, wrong protocol, unknown backend, and multi-member explicit routing. +3. Extend [crates/shell/tests/repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs) to prove: + - targeted syntax is not shell execution + - plain shell input is unchanged + - targeted host turn emits a resume request against the stored session id + - host backend mismatch fails closed + - targeted world turn uses the new world submit route + - targeted world turn chooses the named backend when multiple world members exist + - switching world backends is explicit and one-at-a-time +4. Extend [crates/shell/tests/support/repl_world_agent.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/support/repl_world_agent.rs) to capture and script `/v1/member_turn/stream`. +5. Add request boundary tests in [crates/agent-api-types/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/agent-api-types/src/lib.rs) for `MemberTurnSubmitRequestV1`. +6. Extend [crates/world-agent/tests/streamed_execute_cancel_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/tests/streamed_execute_cancel_v1.rs) to cover submitted-turn streaming and cancel. +7. Keep [crates/world-agent/tests/member_runtime_world_placement_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/tests/member_runtime_world_placement_v1.rs) green to prove the registry refactor did not break placement. +8. Keep [crates/shell/tests/agents_validate.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agents_validate.rs) green to prove config/inventory validation posture did not drift. +9. Add or extend a regression in [crates/shell/src/execution/invocation/tests.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/invocation/tests.rs) proving `-c` remains wrap mode after targeted-turn support lands. + +### Regression rule for this slice + +This slice changes existing REPL parsing behavior. That makes these regressions mandatory: + +1. plain shell input still executes as shell input +2. `:host` and `:pty` still behave exactly as before +3. `substrate -c` still remains wrap mode + +These are the highest-priority tests in the plan. If they fail, the feature is not shippable. + +## Failure Modes Registry + +| Failure mode | Surface | Test coverage required | Handling required | User-visible outcome | +| --- | --- | --- | --- | --- | +| malformed `::` syntax | shell parser | yes | reject before shell fallback | clear syntax error | +| backend id not in inventory | selector | yes | fail closed | "unknown backend ``" | +| backend denied by policy | selector | yes | fail closed | deny naming exact backend | +| host runtime missing `uaa_session_id` | host submit | yes | fail closed | runtime unavailable error | +| host backend mismatch | host submit | yes | fail closed | explicit mismatch guidance | +| world target on non-Linux | shell route | yes | fail closed | explicit Linux-first error | +| targeted world backend on stale generation | shell + world-agent | yes | fail closed | stale-generation error | +| participant not in `world-agent` registry | world submit | yes | fail closed | runtime unavailable error | +| concurrent submitted turn for one participant | world submit | yes | fail closed | "turn already in flight" | +| cancel delivered to wrong span class | world cancel | yes | handle both bootstrap and submitted-turn spans | correct target stops | +| plain shell input accidentally intercepted | shell parser | yes, regression | reject targeted parser path | shell behavior preserved | + +Critical gap rule for this plan: + +no failure mode is allowed to be both untested and silent. Every fail-closed path must produce either a typed transport error or a REPL-visible error message. + +## Performance Review + +This slice is latency-sensitive and human-paced, not throughput-sensitive. + +### Findings resolved in-plan + +1. Each targeted turn spawns a new short-lived resume process. That is acceptable because operator pacing and agent latency dominate process-spawn cost. +2. The plan avoids a new long-lived daemon or generic multiplexor. That keeps operational complexity down and spends zero extra innovation tokens on infrastructure. +3. The one-at-a-time world-member rule is also a performance guardrail. It prevents hidden concurrency and shared-state contention in the first landing. + +### Performance posture + +- no new N+1 style concern exists +- no caching layer is needed +- no new background polling loop is introduced +- the only persistent new memory is one stable active-member record per retained participant plus one active-turn record per submitted turn + +## DX Guardrails + +This is developer-facing product work even though it is backend code. + +Required operator experience: + +1. malformed syntax errors must show the exact accepted format: `:: ` +2. unknown backend errors should suggest `substrate agent list` +3. policy denies must name the exact blocked backend id +4. host backend mismatch must explain that the active REPL session is attached to a different host backend +5. world-target errors on non-Linux must say Linux-first explicitly +6. submitted-turn status and completion lines should include the targeted `backend_id` + +Small details. Real product impact. + +## Worktree Parallelization Strategy + +This plan has real parallelization opportunities after the wire contract is frozen. + +### Dependency table + +| Step | Modules touched | Depends on | +| --- | --- | --- | +| Freeze targeted-turn contract | `crates/agent-api-types/`, `crates/agent-api-client/`, `crates/world-agent/src/` | — | +| Shell parser + selector + host submit path | `crates/shell/src/repl/`, `crates/shell/src/execution/agent_runtime/` | Freeze targeted-turn contract | +| World submit route + member registry refactor | `crates/world-agent/src/`, `crates/agent-api-types/`, `crates/agent-api-client/` | Freeze targeted-turn contract | +| Integration tests + stubs + docs | `crates/shell/tests/`, `crates/world-agent/tests/`, repo docs | Shell parser + selector + host submit path, World submit route + member registry refactor | + +### Parallel lanes + +- Lane A: Shell parser + selector + host submit path + - sequential inside the lane because these steps share `crates/shell/src/repl/` and `crates/shell/src/execution/agent_runtime/` +- Lane B: World submit route + member registry refactor + - sequential inside the lane because these steps share `crates/world-agent/src/` and the new typed request/client contract +- Lane C: Integration tests + stubs + docs + - starts after A and B because the test stub must know the final route and request shape + +### Execution order + +1. Freeze the transport and resume contract. +2. Launch Lane A and Lane B in parallel worktrees. +3. Merge A and B. +4. Run Lane C on top for integration proof, cancel-path proof, and gap-matrix closeout. + +### Conflict flags + +- Lane A and Lane C both touch `crates/shell/tests/repl_world_first_routing_v1.rs`. Keep that file owned by Lane C after A lands. +- Lane B and Lane C both touch `crates/world-agent/tests/streamed_execute_cancel_v1.rs`. Same rule: B lands runtime changes, C lands final proof coverage. +- `crates/agent-api-types/src/lib.rs` is the contract hotspot. Only one lane edits it at a time. + +### Parallelization verdict + +Three workstreams, two parallel implementation lanes, one final integration lane. + +## Implementation Sequence + +### Step 1. Freeze the submit-turn wire contract + +Files: + +- [crates/agent-api-types/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/agent-api-types/src/lib.rs) +- [crates/agent-api-client/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/agent-api-client/src/lib.rs) +- [crates/world-agent/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/lib.rs) +- [crates/world-agent/src/handlers.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/handlers.rs) +- [crates/world-agent/src/service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs) + +Deliver: + +1. add `MemberTurnSubmitRequestV1` +2. validate non-empty prompt plus required identity fields +3. add `AgentClient::submit_member_turn_stream(...)` +4. add `POST /v1/member_turn/stream` +5. reuse `ExecuteStreamFrame` for stream output + +Done means the new transport contract compiles, validates at the boundary, and has request round-trip tests before any runtime logic is layered on top. + +### Step 2. Add exact targeted-turn parsing and selection in the shell + +Files: + +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) +- [crates/shell/src/execution/agent_runtime/validator.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/validator.rs) + +Deliver: + +1. add `TargetedAgentTurn { backend_id, prompt }` +2. add `parse_targeted_agent_turn(...)` +3. add `validate_targeted_backend_selection(...)` +4. route targeted turns before shell fallback and after the current single-line directive gate +5. keep plain shell input, `:host`, and `:pty` behavior unchanged + +Done means the shell can parse and route targeted turns to a scope-specific handler without depending on the world submit path yet. + +### Step 3. Implement the shell-local host submitted-turn path + +Files: + +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) +- [crates/shell/src/execution/agent_runtime/registry.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/registry.rs) + +Deliver: + +1. resolve the active host runtime from the retained orchestrator manifest +2. require `requested_backend_id == retained_orchestrator.backend_id` +3. fail closed if the retained manifest does not contain a surfaced `uaa_session_id` +4. build a new `AgentWrapperRunRequest` carrying: + - the operator prompt + - current working directory + - `agent_api.session.resume.v1 = { selector: "id", id: }` +5. translate wrapper events into targeted-turn REPL and trace output +6. surface completion and failures distinctly from shell-command completion + +Done means `::cli:codex hello` works for a host-scoped `cli:codex` REPL session and errors clearly when the requested host backend is not the active orchestrator backend. + +### Step 4. Refactor `world-agent` member ownership for submitted turns + +Files: + +- [crates/world-agent/src/member_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs) +- [crates/world-agent/src/service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs) + +Deliver: + +1. store stable active member state by `participant_id` +2. retain surfaced `uaa_session_id` from bootstrap registration events +3. keep submitted-turn cancel handles in a separate `span_id` map +4. add `submit_turn(...)` that: + - validates participant/backend/world identity + - rejects concurrent active turns + - uses UAA resume by exact session id + - streams `ExecuteStreamFrame` NDJSON back to the caller +5. extend cancel handling so both bootstrap spans and submitted-turn spans resolve correctly + +Done means `world-agent` can truthfully resume an already-live member session instead of only launching it. + +### Step 5. Wire the targeted Linux world path in the REPL + +Files: + +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) + +Deliver: + +1. add `ensure_targeted_member_runtime_ready(...)` +2. when no world member is retained, launch the requested backend through the existing member-dispatch seam +3. when the requested backend matches the retained backend and generation, reuse it +4. when the requested backend differs, stop the current retained member cleanly and launch the requested backend +5. submit the targeted prompt through `submit_member_turn_stream(...)` +6. reject non-Linux world targets explicitly +7. preserve current world drift and generation reconciliation behavior + +Done means explicit backend targeting works end to end for Linux world members, including backend switching, without pretending simultaneous retained members already exist. + +### Step 6. Close the regression floor and repo truth + +Files: + +- [crates/shell/tests/repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs) +- [crates/shell/tests/support/repl_world_agent.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/support/repl_world_agent.rs) +- [crates/world-agent/tests/streamed_execute_cancel_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/tests/streamed_execute_cancel_v1.rs) +- [crates/shell/tests/agents_validate.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agents_validate.rs) +- [crates/shell/src/execution/invocation/tests.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/invocation/tests.rs) +- [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md) + +Deliver: + +1. targeted-turn integration tests for host and world paths +2. submit-route capture and scripted responses in the REPL world-agent stub +3. cancel-path proof for submitted turns +4. explicit wrap-mode regression for `substrate -c` +5. gap-matrix update that marks explicit targeted turns and real user-turn submission as landed, while keeping broader CLI productization open + +Done means the docs tell the truth and the regression floor is real. + +## Recommended Verification Commands + +```bash +cargo fmt --all -- --check +cargo clippy --workspace --all-targets -- -D warnings +cargo test -p shell --lib -- --nocapture +cargo test -p shell --test repl_world_first_routing_v1 -- --nocapture +cargo test -p shell --test agents_validate -- --nocapture +cargo test -p world-agent --test streamed_execute_cancel_v1 -- --nocapture +cargo test -p world-agent --test member_runtime_world_placement_v1 -- --nocapture +``` + +## Definition of Done + +1. `:: ` is the only accepted targeted-turn grammar. +2. Plain REPL input still runs as shell input. +3. `:host` and `:pty` still behave as before. +4. Host-targeted turns resume the exact surfaced UAA session id. +5. Host-targeted turns fail closed when the requested host backend is not the active orchestrator backend for the current REPL session. +6. Linux world-targeted turns go through the new typed `world-agent` submit route. +7. Multiple configured world members no longer block targeted routing when `backend_id` is explicit. +8. The first landing enforces one retained world-member runtime at a time and makes backend switching explicit. +9. Submitted turns stream output visibly in the REPL and return a distinct completion signal. +10. Submitted-turn cancel works by returned span id without tearing down the retained runtime accidentally. +11. `substrate -c` still remains wrap mode. +12. Tests and gap-matrix docs are updated together. + +## Deferred Work + +- public `substrate agent start|resume|fork|stop` +- non-interactive targeted prompt surface +- multi-line targeted prompts +- simultaneous retained world members in one REPL session +- cross-platform world-target parity beyond Linux +- transcript persistence and richer session-history UX + +## Completion Summary + +- Step 0: Scope Challenge, scope accepted as-is with one explicit constraint added: one retained world member at a time +- Architecture Review: 5 issues found, all resolved in-plan +- Code Quality Review: 4 issues found, all resolved in-plan +- Test Review: diagram produced, 19 targeted gaps identified +- Performance Review: 3 issues found, all resolved in-plan +- NOT in scope: written +- What already exists: written +- TODOS.md updates: 0 items proposed, deferred work stays inside this plan +- Failure modes: 0 acceptable silent gaps, 0 unresolved critical gaps after planned coverage lands +- Outside voice: skipped for this document generation +- Parallelization: 3 lanes, 2 parallel / 1 sequential integration lane +- Lake Score: 9/9 recommendations chose the complete option + +## Decision Audit Trail + +| # | Phase | Decision | Classification | Principle | Rationale | Rejected | +| --- | --- | --- | --- | --- | --- | --- | +| 1 | Step 0 | Freeze grammar to `:: ` only | Mechanical | Explicit over clever | One spelling keeps parser, docs, and tests honest | `@backend`, implicit default-agent routing | +| 2 | Architecture | Use UAA resume semantics for submitted turns | Mechanical | DRY | The dependency already supports resume; do not invent a second backend protocol | stdin injection into bootstrap process | +| 3 | Architecture | Make host-targeted turns valid only for the active orchestrator backend of the current REPL session | Mechanical | Systems over heroes | There is one real host runtime in this slice, not a synthetic pool of host participants | fake host multi-runtime routing | +| 4 | Architecture | Add `MemberTurnSubmitRequestV1` instead of overloading `member_dispatch` | Mechanical | Explicit over clever | Launch and submit are different contracts and should stay different | dual-purpose `ExecuteRequest.member_dispatch` | +| 5 | Architecture | Refactor `MemberRuntimeManager` to key active members by participant id and active turns by span id | Mechanical | Systems over heroes | Stable participant identity is required for follow-up turns and correct cancel behavior | launch-span-only registry | +| 6 | Scope | Support one retained world member at a time in this landing | Taste, resolved | Pragmatic | Matches the current shell ownership model and keeps the diff bounded | pretending simultaneous retained members already exist | +| 7 | Test Review | Make plain-shell and `-c` behavior regressions mandatory | Mechanical | Completeness | Parser work is dangerous if the old shell contract is not explicitly proven | relying on manual validation | +| 8 | DX | Require explicit backend-specific errors for deny, mismatch, and Linux-only failures | Mechanical | Completeness | This feature is pure operator UX, so vague errors are product bugs | generic runtime failures | +| 9 | Parallelization | Freeze the wire contract first, then run shell and world lanes in parallel | Mechanical | Pragmatic | The request shape is the shared seam; parallel work is safe only after that stabilizes | parallel edits to the request contract | diff --git a/llm-last-mile/PLAN-16.md b/llm-last-mile/PLAN-16.md new file mode 100644 index 000000000..3387090f6 --- /dev/null +++ b/llm-last-mile/PLAN-16.md @@ -0,0 +1,752 @@ +# PLAN-16: Complete Backend-Id-Scoped Retained Runtime Selection And Multi-Member Coexistence + +Source SOW: [16-explicit-backend-selection-by-backend-id.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/16-explicit-backend-selection-by-backend-id.md) +Gap matrix anchor: [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md) +Adjacent slice: [17-turn-submit-reuse-for-selected-member.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/17-turn-submit-reuse-for-selected-member.md) +Branch: `feat/session-centric-state-store` +Base branch: `main` +Plan type: Linux-first runtime-identity and retained-member cardinality plan, developer-facing shell/runtime behavior, cross-crate (`shell` + `world-agent` + `agent-api-types`), no UI scope +Review posture: `/autoplan` scope discipline with `/plan-eng-review` depth, rewritten as one cohesive execution plan +Status: execution-ready planning pass on 2026-05-05 +Outside voice: not used for this document generation + +## Objective + +Finish the already-landed explicit backend-targeting model so the runtime behaves like the selector contract already claims it should. + +The shipped behavior is: + +1. every explicit backend-targeted path resolves by exact derived `backend_id` +2. retained world-member lookup and reuse are keyed by `backend_id`, not singleton state or `agent_id` heuristics +3. multiple distinct world-scoped backends such as `cli:codex` and `cli:claude-code` can coexist as simultaneously retained live runtimes within one orchestration session and one authoritative world generation +4. duplicate retained live members for the same backend key still fail closed +5. follow-up submit validation remains exact about `participant_id`, `orchestrator_participant_id`, `world_id`, `world_generation`, and `backend_id` +6. no grammar, CLI, or product-surface redesign is smuggled into this slice + +This is the whole point of the slice. Not new caller grammar. Not new submit transport. Not status-surface redesign. Just making backend identity the real ownership and reuse key all the way down. + +## Plan Summary + +The repo is already past the "can we target a named backend at all?" stage. The exact selector exists. The typed submit route exists. Backend identity already crosses the shell to world boundary. + +What is still wrong is the ownership model behind those surfaces. The shell REPL still keeps one retained member slot. `live_member_for_generation(...)` still searches by `agent_id`. `world-agent` still rejects coexistence by enforcing "only one participant may be retained at a time." That means the current exact-backend path is real at the route edge and still partly fake underneath. + +This plan fixes the identity model, not the product shape. It keeps the current targeted-turn REPL contract and typed world submit contract intact, then replaces both shell-side and world-side singleton assumptions with explicit backend-aware retained-member management. Distinct backend ids may coexist. Duplicate same-backend ownership still fails closed. + +## Locked Starting State + +### What already exists + +The repo already has the pieces this plan must reuse: + +- exact backend selection in [crates/shell/src/execution/agent_runtime/validator.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/validator.rs) +- REPL targeted-turn routing by exact backend id in [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) +- typed world follow-up submit via `MemberTurnSubmitRequestV1` in [crates/agent-api-types/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/agent-api-types/src/lib.rs) +- shell-to-world dispatch transport already carrying `backend_id` in [crates/shell/src/execution/routing/dispatch/world_ops.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs) +- retained-world submit validation already checking exact `backend_id` in [crates/world-agent/src/member_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs) +- Linux-first targeted-turn tests already proving grammar rejection, exact routing, typed submit, and same-generation same-backend reuse in [crates/shell/tests/repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs) + +### Exact remaining gap + +The remaining gap is narrower than the SOW had to assume, but it is still real: + +1. the shell still models retained world members as one `member_runtime: Option` in [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) +2. shell-side live-member reuse still searches by `agent_id` plus `world_generation` in `live_member_for_generation(...)` rather than by the requested `backend_id` +3. `ensure_member_runtime_ready_for_descriptor(...)` still treats a different targeted backend as a replacement of the one global slot instead of a sibling retained runtime +4. `world-agent` still rejects coexistence in `register_member(...)` by failing whenever any retained member already exists +5. the gap matrix still truthfully describes explicit backend targeting as only partially productized because the identity model is still mixed +6. the already-landed submit path is correct only as long as there is at most one retained eligible world member + +## Frozen Execution Contract + +This section removes the wiggle room. If implementation wants to do something else, this plan is wrong and should be revised first. + +### Non-negotiable invariants + +1. `backend_id` is the canonical selector for every explicit backend-targeted path. +2. Exact backend selection must fail closed. No fallback to another backend. No fallback to singleton-member heuristics. +3. `execution.scope` remains part of the selector contract. Host and world cannot silently cross-match. +4. Retained world-member reuse is by backend identity inside one orchestration session and one authoritative world generation. +5. Distinct backend ids may coexist as retained live world members. +6. Duplicate retained live members for the same backend key in the same session and generation are an error. +7. `MemberTurnSubmitRequestV1` stays the submit contract. This slice does not invent a second submit route. +8. The current REPL targeted-turn grammar stays unchanged. +9. `substrate -c` stays wrap mode. +10. Linux-first world-member posture stays unchanged. + +### Retained-member identity contract + +The canonical retained-member key for this slice is: + +```text +orchestration_session_id + world_generation + backend_id +``` + +That is the operator-visible and runtime-honest identity. `participant_id` is still the concrete retained instance identifier. It is not the reusable selection key. + +Consequences: + +1. shell-side readiness lookup must answer "do we already retain `cli:codex` for generation 7?" +2. world-side duplicate detection must answer "do we already retain one live runtime for `cli:codex` in this same session and generation?" +3. follow-up submit must still target the retained participant that owns that backend key + +### Exact-selection contract + +`validate_exact_backend_selection(...)` in [crates/shell/src/execution/agent_runtime/validator.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/validator.rs) stays authoritative for explicit backend-targeted paths. + +This plan does not replace it. It widens the rest of the runtime to match it. + +The selector contract must distinguish these failures: + +- backend missing from effective inventory +- backend present in inventory but wrong scope for the caller +- backend present but disallowed by policy +- backend present but not runtime-realizable +- backend duplicated across effective entries in the same scope + +### Cardinality contract + +This slice deliberately changes cardinality for world-scoped retained members. + +Required behavior: + +1. one orchestration session may retain more than one live world member at the same time +2. those retained live members may have different backend ids +3. the shell may reuse any retained member whose backend id exactly matches the requested backend and whose world generation is still authoritative +4. stale-generation members are still invalid and must not be silently reused +5. same-backend duplicates still fail closed + +### Transport contract + +The transport identity is already correct. Keep it that way. + +`backend_id` must remain explicit in: + +- member dispatch from shell to `world-agent` +- retained runtime state in `world-agent` +- world follow-up submit requests +- world follow-up submit validation +- emitted events and translated REPL output + +### Product-boundary contract + +This slice is not allowed to widen its surface area by convenience. + +Rejected expansions: + +- new caller grammar +- implicit default-backend routing +- new non-interactive agent caller syntax +- broader `substrate agent start|resume|fork|stop` productization +- toolbox or status redesign +- macOS parity + +## Step 0: Scope Challenge + +### 0A. What already exists + +| Sub-problem | Existing code | Decision | +| --- | --- | --- | +| exact backend selection | [crates/shell/src/execution/agent_runtime/validator.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/validator.rs) | Reuse. Do not invent a second explicit selector. | +| targeted-turn route already choosing host vs world by backend id | [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) | Reuse. This plan hardens what happens after route selection. | +| typed world follow-up submit contract | [crates/agent-api-types/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/agent-api-types/src/lib.rs) and [crates/world-agent/src/service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs) | Reuse. No second submit transport. | +| retained submit validation against exact backend/world identity | [crates/world-agent/src/member_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs) | Reuse. Keep strict validation. | +| same-generation same-backend reuse proof | [crates/shell/tests/repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs) | Extend. Add coexistence and duplicate-detection cases rather than replacing the harness. | +| shell and world invalidation on generation rollover | [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) | Reuse. Widen from one retained member to all retained members. | + +### 0B. Minimum honest diff + +The minimum honest implementation is: + +1. harden every explicit backend-targeted selection path around `validate_exact_backend_selection(...)` +2. replace shell-side singleton retained-member state with a backend-aware collection +3. replace shell-side `agent_id` reuse lookup with backend-aware lookup +4. widen `world-agent` retained-member ownership from global singleton to multi-member coexistence +5. preserve strict duplicate detection for the same backend key +6. extend tests and the gap matrix so repo truth matches code truth + +Anything smaller is fake progress. + +Rejected shortcuts: + +- keep one shell `Option` and silently stop/start whenever a second backend is targeted +- keep global singleton retention in `world-agent` and claim coexistence is "future work" +- keep `agent_id` as the reuse key because it is "close enough" +- add a backend aliasing layer +- reopen caller grammar or submit transport just because those files are already nearby + +### 0C. Complexity check + +This slice touches more than 8 files. That is still the minimal honest blast radius because the seam crosses selector hardening, shell retained-state ownership, world retained-state ownership, and integration proof. + +Primary files expected to move: + +1. [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) +2. [crates/shell/src/execution/agent_runtime/validator.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/validator.rs) +3. [crates/shell/tests/repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs) +4. [crates/shell/tests/support/repl_world_agent.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/support/repl_world_agent.rs) +5. [crates/world-agent/src/member_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs) +6. [crates/world-agent/src/service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs) +7. [crates/world-agent/tests/streamed_execute_cancel_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/tests/streamed_execute_cancel_v1.rs) +8. [crates/agent-api-types/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/agent-api-types/src/lib.rs) +9. [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md) + +### 0D. Search and completeness check + +Search-before-building result, in practical terms: + +- **[Layer 1]** reuse `validate_exact_backend_selection(...)` +- **[Layer 1]** reuse `MemberTurnSubmitRequestV1` and `/v1/member_turn/stream` +- **[Layer 1]** reuse current policy allowlist and runtime-realizability checks +- **[Layer 1]** reuse current world-generation invalidation rules +- **[EUREKA]** this is not a transport problem anymore. It is an identity-and-cardinality problem. +- **[EUREKA]** `participant_id` is the retained instance handle, not the correct reuse selector. The public contract is `backend_id`, so reuse must align to `backend_id`. + +Completeness rule for this plan: + +- support simultaneous retained live members for distinct backend ids +- preserve fail-closed behavior for duplicate same-backend retention +- preserve strict submit validation against exact backend identity +- keep prior grammar and submit regressions green + +### 0E. Distribution and runtime contract check + +No new artifact type is introduced. This is not packaging work. + +The ship surface is behavioral: + +- exact backend selection semantics +- retained-member coexistence and reuse +- fail-closed duplicate detection +- truthful tests and docs + +### 0F. NOT in scope + +- new REPL caller grammar +- `substrate -c` redesign +- non-interactive agent caller surface +- public `substrate agent start|resume|fork|stop` +- macOS/Lima parity +- transcript or status-surface redesign +- broader selected-member submit/reuse product polish outside this backend-identity slice + +## Architecture Review + +### Locked architecture decisions + +1. `validate_exact_backend_selection(...)` remains the canonical explicit selector. +2. Shell retained-member state becomes backend-aware, not participant-list guessing and not one global slot. +3. `world-agent` retained-member state allows coexistence of distinct backend ids. +4. Duplicate same-backend retention remains fail closed. +5. `MemberTurnSubmitRequestV1` remains the world submit contract. +6. World-generation invalidation still gates reuse. + +### Architecture findings resolved in-plan + +**Issue 1. The shell singleton is the main correctness bug now.** + +The exact selector can already name `cli:codex` vs `cli:claude-code`. The shell cannot honor that honestly while it keeps one `member_runtime` slot. This plan replaces the slot with backend-aware retained-member state inside the REPL session. + +**Issue 2. `agent_id` is the wrong reuse key for an operator-facing backend selector.** + +The operator targets a backend id. The transport carries a backend id. The submit validator checks a backend id. Reuse by `agent_id` in the middle is an identity leak. This plan removes that mismatch. + +**Issue 3. Shell and `world-agent` cardinality must widen together.** + +If the shell can retain two backends but `world-agent` still rejects the second retained member, the feature is fake. If `world-agent` allows two but the shell keeps overwriting one slot, the feature is still fake. Both sides move in the same slice. + +**Issue 4. Coexistence must not weaken duplicate detection.** + +Allowing `cli:codex` and `cli:claude-code` together is correct. Allowing two live `cli:codex` retained members for the same session and generation is not. The new cardinality must be explicit about that difference. + +**Issue 5. This slice does not need a new generic runtime registry abstraction.** + +The runtime counts are small and the semantics are specific. A blunt, explicit backend-aware collection plus tight helper functions is better than an abstraction project. + +### Architecture ASCII diagrams + +### Exact backend selection to retained reuse + +```text +operator targets backend_id + | + v +validate_exact_backend_selection(...) + | + +--> missing / wrong-scope / denied / unrealizable / duplicate + | | + | +--> fail closed with exact reason + | + v +descriptor { backend_id, scope, binary_path, ... } + | + v +scope == world ? + | + +--> no --> existing host path + | + v +retained lookup by: +orchestration_session_id + world_generation + backend_id + | + +--> retained exact match exists ----------> reuse that retained runtime + | + +--> none exists --------------------------> launch one retained runtime for that backend + | + +--> more than one exact match exists -----> fail closed +``` + +### Shell retained-member ownership model + +```text +REPL session + | + +-- authoritative world binding + | world_id + | world_generation + | + +-- retained world members by backend_id + "cli:codex" -> AsyncReplAgentRuntime + "cli:claude-code" -> AsyncReplAgentRuntime + ... + | + +-- world generation rollover + | + +--> invalidate every retained member for old generation + +--> persist invalidated manifests + +--> remove old backend entries from the active retained map +``` + +### `world-agent` retained-member ownership model + +```text +MemberRuntimeManager + | + +-- active_members_by_participant_id + | participant_id -> retained runtime ownership + | + +-- exact backend query helper + | (orchestration_session_id, world_generation, backend_id) + | | + | +--> 0 matches -> not retained + | +--> 1 match -> exact retained runtime + | +--> 2+ matches -> fail closed + | + +-- active_turns_by_span_id + submit span -> cancel handle + participant association +``` + +## Code Quality Review + +### Findings resolved in-plan + +**Issue 1. Keep the new shell state explicit.** + +Use an obvious backend-aware retained-member collection. No generic registry framework. No trait hierarchy. The runtime already has enough moving parts. + +**Issue 2. Centralize retained-member keying.** + +Add one small helper seam for: + +- exact retained lookup by backend id +- duplicate detection +- removal and invalidation on world-generation rollover + +Do not copy this logic into targeted-turn reuse, startup preparation, and teardown separately. + +**Issue 3. Keep selector logic in `validator.rs`, not in REPL routing code.** + +If backend-selection error taxonomy starts drifting between `validator.rs` and `async_repl.rs`, the product becomes untestable. The REPL should consume exact selection, not reinterpret it. + +**Issue 4. Update nearby diagrams or comments if they become stale.** + +This repo already benefits from textual diagrams in planning docs. If touched code comments or nearby documentation imply a singleton retained-member model after this slice lands, they must be updated in the same change. Stale architecture comments are worse than none. + +### Allowed code shape + +1. Keep explicit backend-selection rules in [crates/shell/src/execution/agent_runtime/validator.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/validator.rs). +2. Keep shell retained-member orchestration in [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs). +3. Keep `world-agent` retained-member ownership logic in [crates/world-agent/src/member_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs). +4. Add only the smallest helper seams needed for backend-aware lookup and duplicate detection. +5. Do not add a new persistence layer, background reconciler, or secondary daemon. + +## Test Review + +### Test framework detection + +This repo is Rust-first. The test surface for this plan is `cargo test` across shell unit tests, shell integration tests, and world-agent unit/integration tests. + +Relevant existing suites already exist in: + +- [crates/shell/src/execution/agent_runtime/validator.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/validator.rs) +- [crates/shell/tests/repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs) +- [crates/shell/tests/support/repl_world_agent.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/support/repl_world_agent.rs) +- [crates/world-agent/src/member_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs) +- [crates/world-agent/tests/streamed_execute_cancel_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/tests/streamed_execute_cancel_v1.rs) + +### Code path coverage + +```text +CODE PATH COVERAGE +=========================== +[+] crates/shell/src/execution/agent_runtime/validator.rs + | + ├── validate_exact_backend_selection(...) + │ ├── [★★★ TESTED] Exact backend match returns descriptor + │ ├── [★★★ TESTED] Duplicate exact backend entries fail closed + │ ├── [GAP] Wrong-scope error is distinct from "missing backend" + │ ├── [GAP] Policy deny stays exact when explicit backend is otherwise valid + │ └── [GAP] Runtime-unrealizable backend stays exact and does not fall back + | + └── validate_member_selection(...) + └── [GAP] Legacy/non-targeted callers are audited so explicit backend paths do not regress to singleton heuristics + +[+] crates/shell/src/repl/async_repl.rs + | + ├── retained world-member state + │ ├── [★★★ TESTED] Same-generation same-backend reuse keeps one retained member + │ ├── [GAP] Same-generation different backend can coexist without tearing down the first retained member + │ ├── [GAP] Exact backend lookup reuses the correct retained runtime when two backends coexist + │ ├── [GAP] Duplicate retained runtime for the same backend key fails closed + │ └── [GAP] World-generation rollover invalidates every retained backend, not just one slot + | + ├── live_member_for_generation / exact retained lookup helper + │ ├── [GAP] Backend-aware lookup returns exact match for cli:codex + │ ├── [GAP] Backend-aware lookup returns exact match for cli:claude-code + │ └── [GAP] Same-agent-family / different-backend coexistence does not collide through agent_id + | + └── ensure_member_runtime_ready_for_descriptor(...) + ├── [★★★ TESTED] Same-generation same-backend reuse does not relaunch + ├── [GAP] Different backend launches a second retained member instead of replacing the first + └── [GAP] Duplicate same-backend retained state is rejected before submit + +[+] crates/world-agent/src/member_runtime.rs + | + ├── register_member(...) + │ ├── [GAP] Distinct backend ids can coexist for one session and generation + │ ├── [GAP] Same backend id duplicate fails closed + │ └── [GAP] Cleanup removes only the exact retained participant that exited + | + ├── exact retained-member lookup + │ ├── [GAP] Resolve cli:codex among simultaneous retained members + │ ├── [GAP] Resolve cli:claude-code among simultaneous retained members + │ └── [GAP] Multiple exact matches fail closed + | + └── submit_turn(...) + ├── [★★★ TESTED] Backend mismatch is rejected + ├── [★★★ TESTED] Missing surfaced session id is rejected + ├── [★★★ TESTED] Concurrent submitted turn collision is rejected per participant + └── [GAP] Submit succeeds for one retained backend while another retained backend remains live and untouched + +───────────────────────────────── +COVERAGE: some exact-routing and submit-identity paths are already tested, but coexistence coverage is still missing +QUALITY TARGET: all new coexistence and duplicate-detection paths reach at least ★★, all regressions and fail-closed paths reach ★★★ +BIGGEST GAP: simultaneous retained distinct backends in one session/generation +───────────────────────────────── +``` + +### User flow coverage + +```text +USER FLOW COVERAGE +=========================== +[+] Operator targets one backend twice + | + ├── [★★★ TESTED] "::cli:codex second" reuses retained cli:codex for same generation + └── [GAP] Same proof for cli:claude-code + +[+] Operator targets two different backends in one live session + | + ├── [GAP] [→E2E] Launch cli:codex, then cli:claude-code, both remain retained + ├── [GAP] [→E2E] Later target cli:codex again, exact codex runtime is reused + └── [GAP] [→E2E] Later target cli:claude-code again, exact claude-code runtime is reused + +[+] Operator error states + | + ├── [GAP] Duplicate exact backend inventory entries fail with exact backend-specific error + ├── [GAP] Duplicate retained same-backend runtime fails closed instead of choosing one + ├── [GAP] Wrong-scope exact backend request remains exact and does not masquerade as missing + └── [★★★ TESTED] Backend mismatch on submit is rejected + +[+] Regression floor + | + ├── [★★★ TESTED] Exact targeted-turn grammar remains intact + ├── [★★★ TESTED] Typed world submit route remains intact + ├── [★★★ TESTED] Host non-active backend rejection remains intact + └── [GAP] Coexistence work must not regress the current same-generation single-backend reuse proof +``` + +### Required tests to add or extend + +1. Extend unit tests in [crates/shell/src/execution/agent_runtime/validator.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/validator.rs) to prove: + - explicit wrong-scope errors are distinguishable from "backend missing" + - policy-denied exact backend selection still reports the named backend + - runtime-unrealizable exact backend selection does not fall back to a sibling backend +2. Extend [crates/shell/tests/repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs) to prove: + - `cli:codex` and `cli:claude-code` can both be retained in one orchestration session and one generation + - targeting `cli:codex` after `cli:claude-code` reuses codex instead of replacing claude-code + - targeting `cli:claude-code` after `cli:codex` reuses claude-code instead of replacing codex + - duplicate same-backend retained state fails closed +3. Extend [crates/shell/tests/support/repl_world_agent.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/support/repl_world_agent.rs) so the stub can model two retained backend ids at once and record exact submit requests separately. +4. Extend unit tests in [crates/world-agent/src/member_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs) to prove: + - `register_member(...)` accepts distinct backend ids + - duplicate same-backend retention fails closed + - exact retained lookup fails closed when duplicates exist +5. Extend [crates/world-agent/tests/streamed_execute_cancel_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/tests/streamed_execute_cancel_v1.rs) to prove one retained backend can service a submitted turn while another retained backend remains live and unaffected. +6. Keep the existing SOW 15 regression floor green. This slice is not allowed to break: + - exact targeted-turn grammar + - typed world submit route + - host non-active backend rejection + - same-generation same-backend reuse + +### Regression rule for this slice + +This slice changes retained-member ownership semantics. That makes these regressions mandatory: + +1. same-generation same-backend reuse must keep passing +2. exact targeted-turn routing must keep passing +3. typed world submit validation must keep passing +4. host follow-up rejection for non-active backend must keep passing + +No AskUserQuestion needed here. These are hard requirements. + +## Failure Modes Registry + +| Failure mode | Surface | Test coverage required | Handling required | User-visible outcome | +| --- | --- | --- | --- | --- | +| exact backend missing from inventory | selector | yes | fail closed | exact missing-backend error | +| exact backend exists only in wrong scope | selector | yes | fail closed | exact wrong-scope error | +| exact backend denied by policy | selector | yes | fail closed | exact deny naming backend id | +| exact backend unrealizable | selector | yes | fail closed | exact runtime-unavailable error | +| duplicate exact backend entries in effective inventory | selector | yes | fail closed | exact duplicate-inventory error | +| shell retained lookup finds none for targeted backend | shell reuse | yes | launch exact backend, not sibling backend | normal backend-specific launch | +| shell retained lookup finds duplicate same-backend live members | shell reuse | yes | fail closed | duplicate-retained-runtime error | +| world-agent retained registry already has distinct other backend | world registry | yes | allow coexistence | no operator-visible error | +| world-agent retained registry gets second same-backend retained member | world registry | yes | fail closed | duplicate-retained-runtime error | +| submit request backend id mismatches retained runtime | world submit | yes | fail closed | backend mismatch error | +| world-generation rollover leaves stale retained backend live | shell/world cleanup | yes | invalidate and remove stale retained entries | stale-generation recovery path | +| coexistence change accidentally tears down the wrong backend | shell/world teardown | yes | exact backend-scoped cleanup | operator keeps other backend live | + +Critical gap rule for this plan: + +no same-backend duplicate path is allowed to be both untested and auto-resolved. Duplicate same-backend ownership must always fail closed. + +## Performance Review + +This slice increases retained-member cardinality. That matters, but it is still human-paced control-plane work, not high-throughput backend traffic. + +### Findings resolved in-plan + +1. Retaining two live world members instead of one increases steady-state memory and process count. That is acceptable because the number of eligible retained backends is tiny and explicitly configured. +2. Duplicate detection can stay simple. A small scan across retained members for one session/generation is cheaper and safer than inventing a complex cache or secondary index too early. +3. World-generation rollover must clean up all stale retained members. If cleanup misses one, the problem is correctness first and resource leakage second. + +### Performance posture + +- no new N+1 style data-access concern exists +- no caching layer is needed +- no background polling loop is needed +- the cardinality increase is bounded by configured world-scoped backends and one authoritative world generation +- correctness wins over micro-optimizing lookup structures here + +## DX Guardrails + +This is developer-facing runtime behavior even though it is backend code. + +Required operator experience: + +1. exact backend failures must still name the backend id the operator requested +2. coexistence must be invisible when it works and explicit when it fails +3. duplicate same-backend retained runtime failures must say that the state is ambiguous, not just "submit failed" +4. world-generation rollover failures must remain explicit about stale retained state +5. submit errors must preserve backend identity in translated REPL output + +Small details. Real product impact. + +## Worktree Parallelization Strategy + +This plan has real parallelization opportunities once the retained-member identity contract is frozen. + +### Dependency table + +| Step | Modules touched | Depends on | +| --- | --- | --- | +| Freeze retained-member identity contract and exact error taxonomy | `crates/shell/src/execution/agent_runtime/`, `crates/shell/src/repl/`, docs | — | +| Shell retained-member collection + backend-aware reuse | `crates/shell/src/repl/` | Freeze retained-member identity contract and exact error taxonomy | +| `world-agent` coexistence + duplicate detection | `crates/world-agent/src/` | Freeze retained-member identity contract and exact error taxonomy | +| Integration tests + gap-matrix closeout | `crates/shell/tests/`, `crates/world-agent/tests/`, repo docs | Shell retained-member collection + backend-aware reuse, `world-agent` coexistence + duplicate detection | + +### Parallel lanes + +- Lane A: Shell retained-member collection + backend-aware reuse + - sequential inside the lane because these steps share `crates/shell/src/repl/` +- Lane B: `world-agent` coexistence + duplicate detection + - sequential inside the lane because these steps share `crates/world-agent/src/` +- Lane C: Integration tests + gap-matrix closeout + - starts after A and B because the test stub and docs need the final retained-member contract + +### Execution order + +1. Freeze the retained-member identity contract and exact failure taxonomy. +2. Launch Lane A and Lane B in parallel worktrees. +3. Merge A and B. +4. Run Lane C for coexistence proof, duplicate-fail-closed proof, and gap-matrix truth updates. + +### Conflict flags + +- Lane A and Lane C both touch [crates/shell/tests/repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs). Keep final ownership with Lane C after shell behavior lands. +- Lane B and Lane C both touch [crates/world-agent/tests/streamed_execute_cancel_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/tests/streamed_execute_cancel_v1.rs). Same rule. +- [crates/world-agent/src/member_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs) is the cardinality hotspot. Only one lane edits it at a time. + +### Parallelization verdict + +Three workstreams, two parallel implementation lanes, one final integration lane. + +## Implementation Sequence + +### Step 1. Freeze the retained-member identity contract + +Files: + +- [crates/shell/src/execution/agent_runtime/validator.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/validator.rs) +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) +- [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md) + +Deliver: + +1. document the exact failure taxonomy for explicit backend-targeted selection +2. audit explicit backend-targeted callers so they all use `validate_exact_backend_selection(...)` +3. freeze the retained-member key as `orchestration_session_id + world_generation + backend_id` + +Done means the selector and identity rules are explicit before cardinality changes land. + +### Step 2. Replace shell-side singleton retained-member state + +Files: + +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) + +Deliver: + +1. replace `member_runtime: Option` with a backend-aware retained-member collection +2. replace `live_member_for_generation(...)` with exact retained lookup by backend id +3. update `ensure_member_runtime_ready_for_descriptor(...)` so: + - exact retained backend match is reused + - missing exact retained backend is launched + - duplicate same-backend retained entries fail closed +4. update world-generation rollover invalidation so every retained backend for the old generation is invalidated and removed + +Done means the shell can honestly retain more than one live world member at once. + +### Step 3. Widen `world-agent` retained-member ownership + +Files: + +- [crates/world-agent/src/member_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs) +- [crates/world-agent/src/service.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs) + +Deliver: + +1. remove the global "only one participant may be retained at a time" guard +2. add exact retained-member lookup by session/generation/backend id +3. allow distinct backend ids to coexist +4. reject duplicate same-backend retained members for the same session and generation +5. preserve current submit validation and submitted-turn collision behavior + +Done means `world-agent` can retain `cli:codex` and `cli:claude-code` at the same time without weakening exact submit validation. + +### Step 4. Tighten exact failure handling in adjacent explicit paths + +Files: + +- [crates/shell/src/execution/agent_runtime/validator.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/validator.rs) +- [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) + +Deliver: + +1. preserve exact backend-specific errors for missing, wrong-scope, denied, unrealizable, and duplicate cases +2. ensure coexistence changes do not accidentally reintroduce legacy ambiguity-based selection for explicit backend paths +3. keep targeted submit and reuse paths keyed to exact backend identity end to end + +Done means coexistence lands without softening the selector contract. + +### Step 5. Close the coexistence proof and repo truth + +Files: + +- [crates/shell/tests/repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs) +- [crates/shell/tests/support/repl_world_agent.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/support/repl_world_agent.rs) +- [crates/world-agent/tests/streamed_execute_cancel_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/tests/streamed_execute_cancel_v1.rs) +- [crates/world-agent/src/member_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs) +- [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md) + +Deliver: + +1. coexistence proof for `cli:codex` and `cli:claude-code` +2. exact reuse proof for each retained backend after both are live +3. duplicate same-backend fail-closed proof +4. unchanged current same-backend reuse proof +5. gap-matrix update from "REPL-first exact targeting with mixed identity model" to "backend-id-scoped retained coexistence landed on Linux-first runtime path" + +Done means the docs tell the truth and the coexistence behavior is proven, not implied. + +## Recommended Verification Commands + +```bash +cargo fmt --all -- --check +cargo clippy --workspace --all-targets -- -D warnings +cargo test -p shell validate_exact_backend_selection -- --nocapture +cargo test -p shell --test repl_world_first_routing_v1 -- --nocapture +cargo test -p world-agent member_runtime -- --nocapture +cargo test -p world-agent --test streamed_execute_cancel_v1 -- --nocapture +``` + +## Definition of Done + +1. Every explicit backend-targeted selector path still resolves by exact derived `backend_id`. +2. Wrong-scope, denied, unrealizable, duplicate, and missing backend failures remain exact and fail closed. +3. The shell can retain more than one live world-scoped member at once. +4. Retained shell reuse is by exact backend id, not by `agent_id`. +5. `world-agent` can retain distinct backend ids simultaneously in one session and generation. +6. `world-agent` still rejects duplicate same-backend retained members. +7. Exact targeted follow-up turns reuse the correct retained backend after coexistence is established. +8. World-generation rollover invalidates all stale retained backend entries. +9. Existing targeted-turn grammar and submit regressions remain green. +10. The gap matrix is updated to match shipped reality. + +## Deferred Work + +- non-REPL caller surface +- `substrate -c` redesign +- public `substrate agent start|resume|fork|stop` +- broader selected-member submit/reuse polish beyond this identity slice +- macOS/Lima parity +- status/toolbox work + +## Completion Summary + +- Step 0: Scope Challenge, scope accepted as-is +- Architecture Review: 5 issues found, all resolved in-plan +- Code Quality Review: 4 issues found, all resolved in-plan +- Test Review: diagram produced, coexistence and duplicate-detection gaps identified +- Performance Review: 3 issues found, all resolved in-plan +- NOT in scope: written +- What already exists: written +- TODOS.md updates: 0 items proposed, deferred work stays inside this plan +- Failure modes: 0 acceptable silent duplicate-resolution gaps, 0 unresolved critical gaps after planned coverage lands +- Outside voice: skipped for this document generation +- Parallelization: 3 lanes, 2 parallel / 1 sequential integration lane +- Lake Score: 8/8 recommendations chose the complete option + +## Decision Audit Trail + +| # | Phase | Decision | Classification | Principle | Rationale | Rejected | +| --- | --- | --- | --- | --- | --- | --- | +| 1 | Step 0 | Treat this as an identity-and-cardinality slice, not a transport slice | Mechanical | Pragmatic | The submit route and exact selector already exist; the bug is the retained ownership model | reopening submit transport | +| 2 | Architecture | Keep `validate_exact_backend_selection(...)` as the canonical explicit selector | Mechanical | DRY | One selector contract is safer than duplicating backend-id logic in the REPL | bespoke REPL-only exact selection | +| 3 | Architecture | Key retained world-member reuse by `orchestration_session_id + world_generation + backend_id` | Mechanical | Explicit over clever | This matches the public selector and the authoritative world-binding model | reuse by `agent_id`, participant-id-only selection | +| 4 | Architecture | Replace shell singleton retained-member state with a backend-aware collection | Mechanical | Completeness | The shell cannot truthfully support exact backend coexistence with one global slot | stop-and-replace hidden singleton behavior | +| 5 | Architecture | Allow distinct backend ids to coexist in `world-agent` while rejecting duplicate same-backend retention | Mechanical | Systems over heroes | Coexistence is required, ambiguity is not | global singleton retention, permissive duplicate selection | +| 6 | Code Quality | Keep new helper seams small and local instead of introducing a generalized registry abstraction | Taste, resolved | Minimal diff | The runtime counts are tiny and the semantics are narrow | generic runtime registry framework | +| 7 | Test Review | Make coexistence proof and duplicate fail-closed proof mandatory | Mechanical | Completeness | This slice is not real until both behaviors are exercised end to end | relying on current single-backend reuse tests | +| 8 | Parallelization | Freeze identity rules first, then run shell and world cardinality work in parallel | Mechanical | Pragmatic | The shared contract is small but central, and it reduces merge churn across the two main lanes | parallel edits before key semantics are frozen | diff --git a/llm-last-mile/PLAN-17.md b/llm-last-mile/PLAN-17.md new file mode 100644 index 000000000..832e606ad --- /dev/null +++ b/llm-last-mile/PLAN-17.md @@ -0,0 +1,701 @@ +# PLAN-17: Unify Selected-Member Follow-Up Submit And Reuse Semantics + +Source SOW: [17-turn-submit-reuse-for-selected-member.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/17-turn-submit-reuse-for-selected-member.md) +Gap matrix anchor: [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md) +Adjacent landed slices: [PLAN-15.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-15.md), [PLAN-16.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-16.md) +Branch: `feat/session-centric-state-store` +Base branch: `main` +Plan type: Linux-first REPL-targeted follow-up submit/reuse hardening, developer-facing shell/runtime behavior, no new caller surface +Review posture: `/autoplan` scope discipline with `/plan-eng-review` structure, rewritten as one cohesive execution plan +Status: execution-ready planning pass on 2026-05-05 +Outside voice: not used for this document generation + +## Objective + +Treat the current Linux-first targeted-turn path as shipped, then finish the part that still reads like implementation scatter instead of a deliberate product contract. + +This slice locks one explicit internal contract for selected-backend follow-up turns: + +1. exact `backend_id` targeting stays the only targeted REPL grammar +2. host follow-up turns resume only the active orchestrator backend +3. Linux world follow-up turns reuse the exact retained selected member when it is still valid +4. relaunch happens only when retained world runtime state is missing or stale for the current world generation +5. operator-visible failures stay fail-closed, precise, and documented +6. no new non-REPL caller surface is introduced in this slice + +## Plan Summary + +The repo is no longer missing targeted-turn submit or backend-aware retained-member reuse. That work already landed in [PLAN-15.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-15.md) and [PLAN-16.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-16.md). + +What is still missing is cohesion. The selected-member contract is spread across parser logic, route selection, readiness checks, submit functions, retained-member validation, and a few integration tests. The system works, but the contract still reads like code archaeology. + +This slice does not add a new runtime capability. It makes the already-landed capability explicit, centralized, and regression-proof: + +1. centralize targeted follow-up dispatch behind one internal helper seam in [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) +2. keep the host path shell-owned and session-resume based +3. keep the world path `world-agent`-owned and `MemberTurnSubmitRequestV1`-based +4. make reuse-vs-relaunch semantics deterministic and testable +5. tighten docs and operator-facing error truth so the repo says exactly what the runtime does + +## Locked Starting State + +### What already exists + +| Sub-problem | Existing code | Decision | +| --- | --- | --- | +| Exact backend selection by canonical `backend_id` | [crates/shell/src/execution/agent_runtime/validator.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/validator.rs) | Reuse as the only explicit selector. Do not duplicate selection logic in REPL routing. | +| Exact targeted-turn grammar and route-before-shell behavior | [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) | Keep grammar unchanged. Harden behavior after parse, not the parse itself. | +| Host follow-up submit via UAA session resume | [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) | Reuse. No new transport. | +| Linux world follow-up submit via typed member-turn route | [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs), [crates/world-agent/src/member_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs), [crates/agent-api-types/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/agent-api-types/src/lib.rs) | Reuse. Do not invent a second world submit path. | +| Backend-id-scoped retained-member coexistence and duplicate fail-closed behavior | [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs), [crates/world-agent/src/member_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs) | Treat as landed baseline from PLAN-16, not work to redo. | +| Integration floor for grammar rejection, world submit reuse, and host mismatch rejection | [crates/shell/tests/repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs) | Extend. Do not replace the harness. | +| Single-live-session status/toolbox preflight outside the REPL path | [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs), [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs) | Explicitly out of scope for this slice. Document the constraint, do not widen into status redesign. | + +### Exact remaining gap + +The remaining gap is narrower than the SOW had to assume: + +1. the targeted-turn contract is still split inline across parse, route, readiness, and submit branches in [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) +2. host and world targeted-turn paths are behaviorally aligned, but they do not yet share one explicit orchestration seam that prevents drift in error mapping and operator-facing semantics +3. the positive host follow-up submit path is under-proved compared to the world path and the host rejection path +4. reuse-vs-relaunch semantics are correct in code, but not yet documented as one contract with one decision table +5. repo docs still need one fresh statement of what is shipped, what is Linux-only, and what is intentionally deferred + +## Frozen Execution Contract + +If implementation wants to do something else, revise this plan first. Do not freestyle past these rules. + +### Non-negotiable invariants + +1. `:: ` remains the only targeted REPL follow-up grammar. +2. Plain REPL input remains shell execution, not implicit agent routing. +3. `validate_exact_backend_selection(...)` remains the canonical selector for explicit targeted-turn routing. +4. Host follow-up turns may target only the active orchestrator backend for the current REPL session. +5. Linux world follow-up turns must submit through `MemberTurnSubmitRequestV1` and `/v1/member_turn/stream`. +6. World-member reuse is by exact retained backend slot for the current orchestration session and current authoritative `world_generation`. +7. Relaunch is allowed only when the retained world runtime is missing, invalidated, or stale for the current `world_generation`. +8. Duplicate retained members for the same backend slot remain fail closed. +9. Non-Linux world follow-up turns remain explicit fail-closed behavior. +10. `substrate -c` remains shell wrap mode. No caller-surface widening in this slice. +11. `agent status` and toolbox ambiguity handling remain out of scope. + +### Submit/reuse decision table + +| Requested path | Preconditions | Required action | Explicitly disallowed fallback | +| --- | --- | --- | --- | +| Host targeted follow-up | exact host backend match, active orchestrator runtime retained, requested backend equals active orchestrator backend | call `submit_host_targeted_turn(...)` with session-resume extension | launching a second host runtime, retargeting to another host backend, shell fallback | +| Host targeted follow-up with wrong backend | exact host backend match, but requested backend is not the active orchestrator backend | fail closed with exact active-backend mismatch error | silent retarget, shell fallback, world-member launch | +| World targeted follow-up on Linux with valid retained member | exact world backend match, retained exact backend runtime authoritative-live for current generation | call `submit_world_targeted_turn(...)` against retained participant | relaunching a sibling member, switching backend slot, shell fallback | +| World targeted follow-up on Linux with missing or stale retained member | exact world backend match, but no authoritative-live retained runtime for current generation | call `ensure_member_runtime_ready_for_descriptor(...)`, then submit into the newly ready exact backend runtime | submit into stale participant, pick another retained member, shell fallback | +| World targeted follow-up on non-Linux | exact world backend match, but platform path does not support world-member reuse | fail closed with Linux-only error | pretending success, host submit, shell fallback | + +### Failure taxonomy that must be preserved + +| Failure class | Source of truth | Required behavior | +| --- | --- | --- | +| malformed targeted syntax | REPL parser in [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) | reject before shell fallback with exact accepted format | +| ambiguous exact backend entry | `validate_exact_backend_selection(...)` | reject with exact backend id and scope | +| wrong active host backend | host-targeted route gate in [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) | reject with expected and actual backend ids | +| no active orchestrator runtime | host-targeted dispatch seam | reject clearly, do not fabricate a new host runtime | +| world runtime unavailable after ready path | world-targeted dispatch seam | reject with named backend and platform/runtime context | +| retained identity drift | `validate_submit_turn_request(...)` in [crates/world-agent/src/member_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs) | reject with exact mismatched field | +| duplicate retained same-backend slot | retained-key validation in [crates/world-agent/src/member_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs) | reject fail closed, never choose one arbitrarily | +| concurrent submitted turn collision | submitted-turn slot reservation in [crates/world-agent/src/member_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs) | reject with active submitted-turn context, do not queue implicitly | + +## Step 0: Scope Challenge + +### 0A. What already exists + +| Sub-problem | Existing code | Decision | +| --- | --- | --- | +| Exact selector contract | [crates/shell/src/execution/agent_runtime/validator.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/validator.rs) | Reuse as-is. Do not move selector logic into the REPL helper. | +| Host-vs-world route classification | [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) | Centralize the branch behind one helper seam. Do not change the operator grammar. | +| World readiness and retained-member validation | [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs), [crates/world-agent/src/member_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs) | Reuse. Do not reopen retained-member identity or cardinality in this slice. | +| Host follow-up submit transport | [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) | Reuse session-resume submit. | +| World follow-up submit transport | [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs), [crates/agent-api-types/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/agent-api-types/src/lib.rs) | Reuse typed submit. | +| Existing integration harness | [crates/shell/tests/repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs), [crates/shell/tests/support/repl_world_agent.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/support/repl_world_agent.rs) | Extend. Keep one integration story. | + +### 0B. Minimum honest diff + +The minimum honest implementation is smaller than the original draft implied: + +1. add one explicit targeted-follow-up dispatch seam in [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) +2. preserve `submit_host_targeted_turn(...)` and `submit_world_targeted_turn(...)` as separate transport implementations +3. extend shell integration tests to prove the positive host path, stale-world relaunch path, and explicit fail-closed cases +4. add selector-level regression tests only where the new dispatch seam depends on behavior that is not already pinned +5. update repo-truth docs so the shipped contract is stated plainly + +Anything broader is scope creep. Anything smaller leaves the contract implicit. + +### 0C. Complexity check + +This plan does not trigger the overbuild smell if executed correctly. + +Expected touched files: + +1. [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) +2. [crates/shell/tests/repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs) +3. [crates/shell/tests/support/repl_world_agent.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/support/repl_world_agent.rs) if new stub scripting is required +4. [crates/shell/src/execution/agent_runtime/validator.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/validator.rs) for selector regression coverage only +5. [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md) +6. [llm-last-mile/README.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/README.md) if this packet index is used as current repo truth + +Rejected expansions: + +1. [crates/world-agent/src/member_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs) unless a testable operator-facing defect is discovered that cannot be pinned from the shell side +2. [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) +3. [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs) +4. non-REPL caller surfaces +5. macOS/Lima parity work +6. new public control-plane commands + +### 0D. Search and completeness check + +Search-before-building result, in practical terms: + +- **[Layer 1]** reuse `validate_exact_backend_selection(...)` +- **[Layer 1]** reuse `ensure_member_runtime_ready_for_descriptor(...)` +- **[Layer 1]** reuse `submit_host_targeted_turn(...)` +- **[Layer 1]** reuse `submit_world_targeted_turn(...)` +- **[Layer 1]** reuse `MemberTurnSubmitRequestV1` +- **[EUREKA]** this is no longer a transport problem or a retained-key problem. It is a contract-centralization and regression-floor problem. +- **[EUREKA]** the cheapest complete version is not more runtime capability. It is one dispatch seam plus the missing regression proofs. + +### 0E. Distribution and runtime contract check + +No new artifact type is introduced. + +Distribution is not the missing piece here. The missing deliverable is behavioral truth: + +1. one obvious shell seam for targeted follow-up dispatch +2. one explicit operator contract for reuse versus relaunch +3. one test floor that proves the contract instead of implying it +4. one doc truth that says Linux-first and REPL-first without hedging + +### 0F. NOT in scope + +- `substrate -c` redesign +- non-interactive prompt submission surface +- default-agent routing for plain REPL input +- `substrate agent start|resume|fork|stop` +- status/toolbox ambiguity redesign +- macOS/Lima parity +- new world-agent transport families +- redoing PLAN-16 retained-member cardinality work + +## Architecture Review + +### Locked architecture decisions + +1. Add one internal targeted-turn orchestration seam in [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs). The name is flexible. The behavior is not. +2. That seam owns exact backend selection, route classification, readiness, and operator-visible error mapping. +3. The seam does not replace `submit_host_targeted_turn(...)` or `submit_world_targeted_turn(...)`. It chooses between them. +4. Host follow-up submit remains shell-owned and session-resume based. +5. World follow-up submit remains `world-agent`-owned and typed-request based. +6. Reuse-vs-relaunch logic for world members remains inside the existing readiness path, not in the submit function. + +### Architecture findings resolved in-plan + +**Issue 1. The contract is correct but scattered.** + +Right now, the route branch, readiness branch, and submit branch each carry part of the targeted-turn story. That is fragile. The REPL should have one obvious path that says: parse exact backend, classify host vs world, ensure readiness if world, submit, surface exact failure. + +**Issue 2. Host and world flows must look unified to the operator without faking a shared transport.** + +One internal seam is correct. One generic transport abstraction is not. The operator should experience one selected-backend contract. The implementation should still preserve host-resume and world-submit as different mechanisms. + +**Issue 3. Relaunch must stay a readiness concern, not a submit concern.** + +If submit starts creating or replacing runtimes, the contract becomes impossible to reason about. World submit must target an exact retained participant. Readiness may create that participant only when the exact backend slot is missing or stale. + +**Issue 4. Status/toolbox ambiguity is adjacent, not part of this slice.** + +That constraint belongs in the plan because these files sit nearby and the temptation to widen scope will be high. This plan explicitly carries the boundary so implementation does not drift into control-plane redesign. + +### Architecture ASCII diagrams + +### End-to-end targeted follow-up dispatch + +```text +operator enters exact targeted turn + | + v +parse ":: " + | + +--> malformed ----------------------------> fail closed before shell fallback + | + v +validate_exact_backend_selection(scope, backend_id) + | + +--> no match / ambiguous / unrealizable --> fail closed with exact reason + | + v +descriptor { backend_id, execution_scope, binary_path, backend_kind, ... } + | + +--> host scope + | | + | +--> active orchestrator runtime exists? + | | | + | | +--> no --------------------> fail closed + | | | + | | +--> yes, backend matches --> submit_host_targeted_turn(...) + | | | + | | +--> yes, backend mismatches -> fail closed + | + +--> world scope + | + +--> non-Linux --------------------> fail closed + | + +--> Linux + | + +--> ensure_member_runtime_ready_for_descriptor(...) + | | + | +--> exact backend retained and current ----> reuse + | +--> missing/stale exact backend -----------> relaunch exact backend slot + | +--> duplicate/invalid ---------------------> fail closed + | + +--> submit_world_targeted_turn(...) +``` + +### World reuse versus relaunch + +```text +requested backend_id = "cli:codex" +current world_generation = 7 + | + v +member_runtimes["cli:codex"] ? + | + +--> no ----------------------------------> prepare + launch retained cli:codex for generation 7 + | + +--> yes + | + +--> authoritative-live, generation 7, exact participant matches live slot + | | + | +--> reuse retained participant + | + +--> stale generation / invalidated / mismatched live slot + | + +--> tear down old retained entry + +--> persist invalidation + +--> prepare + launch exact replacement +``` + +### Ownership split that must remain unchanged + +```text +host targeted follow-up + shell-owned runtime + -> build session resume extension + -> UAA run_control(...) + +world targeted follow-up + shell-owned exact backend selection + readiness + -> typed MemberTurnSubmitRequestV1 + -> world-agent retained participant validation + -> UAA run_control(...) inside world-agent +``` + +## Code Quality Review + +### Findings resolved in-plan + +**Issue 1. Inline branching is doing too much narrative work.** + +The code should tell the story in one place. Right now the reader has to follow parse logic, backend validation, runtime matching, readiness, and submit functions in separate jumps. The implementation should make that story obvious. + +**Issue 2. Error strings must not drift between route branches.** + +A targeted host mismatch, a missing active host runtime, and a Linux-only world error are different operator problems. The dispatch seam must centralize where those are chosen so tests can pin them. + +**Issue 3. This slice should avoid new abstractions that outlive the problem.** + +This is a small blast-radius problem. A direct helper and a small route enum are enough. Anything broader is engineering theater. + +**Issue 4. Docs are part of the quality bar here.** + +The runtime contract already exists. If the docs keep describing an earlier gap state, the repo becomes misleading. That is a quality defect, not a cosmetic issue. + +### Allowed code shape + +1. Prefer a small explicit helper or enum over a new dispatcher module. +2. Keep targeted-turn orchestration in [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) unless a tiny pure helper clearly improves readability in an already-related module. +3. Do not duplicate selector error taxonomy. Consume `validate_exact_backend_selection(...)` and preserve its exactness. +4. Do not duplicate retained-member identity validation. Leave `validate_submit_turn_request(...)` authoritative on the world side. +5. Do not introduce a generic runtime manager abstraction just to unify two call sites. + +## Test Review + +### Test framework detection + +This repo is Rust-first and the relevant review surface is `cargo test`. + +Primary suites for this slice: + +1. [crates/shell/tests/repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs) +2. [crates/shell/tests/support/repl_world_agent.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/support/repl_world_agent.rs) +3. selector tests in [crates/shell/src/execution/agent_runtime/validator.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/validator.rs) +4. existing retained-member validation tests in [crates/world-agent/src/member_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs), but only if a shell-level proof cannot pin the behavior adequately + +### Code path coverage + +```text +CODE PATH COVERAGE +=========================== +[+] crates/shell/src/repl/async_repl.rs + | + ├── exact targeted-turn parse + │ ├── [★★★ TESTED] malformed "::" rejects before shell fallback + │ └── [★★★ TESTED] exact single-line grammar enters targeted route + | + ├── host targeted route + │ ├── [★★★ TESTED] non-active host backend rejects fail-closed + │ ├── [GAP] active orchestrator backend follow-up turn resumes successfully + │ └── [GAP] missing active orchestrator runtime rejects without shell fallback + | + ├── world targeted route on Linux + │ ├── [★★★ TESTED] exact retained member submit uses typed route without relaunch + │ ├── [★★★ TESTED] same-generation world command reuses live member runtime + │ ├── [GAP] stale retained member relaunches exact backend slot before submit + │ └── [GAP] world-runtime-unavailable path names backend and fails closed + | + └── world targeted route on non-Linux + └── [GAP] Linux-only fail-closed contract is explicitly regression-tested + +[+] crates/shell/src/execution/agent_runtime/validator.rs + | + ├── validate_exact_backend_selection(...) + │ ├── [★★★ TESTED] exact backend match returns one descriptor + │ ├── [★★★ TESTED] duplicate exact backend entries fail closed + │ ├── [★★★ TESTED] exact world-vs-host scope mismatch is distinct behavior + │ └── [GAP] unrealizable exact backend stays exact and does not drift to another backend + +[+] crates/world-agent/src/member_runtime.rs + | + ├── retained-key validation + │ ├── [★★★ TESTED] identity drift rejects exact mismatched field + │ ├── [★★★ TESTED] duplicate retained same-backend slot rejects + │ └── [★★★ TESTED] concurrent submitted turn collision rejects + | + └── world submit path + └── [★★★ TESTED] typed submit validates participant/world/backend identity + +--------------------------------- +COVERAGE TARGET +- all targeted host path outcomes at ★★★ +- all selected-member relaunch and fail-closed outcomes at ★★★ +- no new targeted-turn branch ships without a direct regression test +--------------------------------- +``` + +### User flow coverage + +```text +USER FLOW COVERAGE +=========================== +[+] Operator targets active host backend + | + └── [GAP] [->E2E] ":: follow up" resumes the active orchestrator session + +[+] Operator targets retained world backend + | + ├── [★★★ TESTED] exact retained backend reuses member and typed submit route + └── [GAP] [->E2E] stale generation world member is replaced, then exact backend submit succeeds + +[+] Operator mistargets host backend + | + └── [★★★ TESTED] fail closed, no shell fallback, no world-member launch + +[+] Operator hits unsupported world follow-up path + | + └── [GAP] explicit Linux-only error remains pinned + +[+] Operator-visible error states + | + ├── [★★★ TESTED] malformed syntax + ├── [GAP] no active orchestrator runtime + ├── [GAP] world-runtime-unavailable after readiness + └── [★★★ TESTED] retained identity drift / concurrent submitted turn +``` + +### Required tests to add or extend + +1. Add a positive host targeted-follow-up integration test in [crates/shell/tests/repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs) that proves the active orchestrator backend can accept `:: ` follow-up turns via session resume. +2. Add a regression test for `no active orchestrator runtime is available for targeted follow-up turns` so host-targeted dispatch never silently recreates or reroutes a host runtime. +3. Add a stale-generation world-member test that proves the selected backend relaunches only after generation drift, then submits against the replacement exact backend slot. +4. Add a non-Linux fail-closed regression that pins `world-targeted follow-up turns are supported on Linux only`. +5. Add selector-level tests if needed so exact unrealizable-backend behavior remains explicit and exact. + +### QA-facing test artifact + +During implementation, write a QA-facing artifact to: + +```text +~/.gstack/projects//-feat-session-centric-state-store-eng-review-test-plan-.md +``` + +Required contents: + +1. targeted host follow-up flow +2. targeted world reuse flow +3. stale-generation world relaunch flow +4. malformed syntax and wrong-backend fail-closed flows +5. Linux-only fail-closed flow for non-Linux world follow-up paths + +This artifact is for `/qa` and `/qa-only`. Keep it user-journey oriented, not implementation-detail heavy. + +### Regression rule for this slice + +These tests are mandatory. No discussion: + +1. malformed targeted syntax still rejects before shell fallback +2. host non-active backend rejection still rejects before shell fallback +3. exact world submit still uses `MemberTurnSubmitRequestV1` +4. same-generation retained world-member reuse still stays green +5. duplicate same-backend retained-member rejection still stays green + +## Failure Modes Registry + +| Failure mode | Test required | Error handling exists | User sees clear failure | Critical gap before this slice lands | +| --- | --- | --- | --- | --- | +| malformed `::` syntax falls through to shell execution | yes | yes | yes | yes | +| host follow-up targets non-active backend | yes | yes | yes | no | +| active host runtime missing on targeted submit | yes | yes | yes | yes | +| exact world backend missing live retained runtime | yes | yes via ready path | yes | no | +| stale retained world member is reused instead of replaced | yes | partial until regression pinned | no unless tested | yes | +| ready path fails and shell silently falls back | yes | should fail closed | must be yes | yes | +| world follow-up on non-Linux appears supported | yes | yes | yes | yes | +| selector drifts from exact match to best-effort match | yes | partial via selector tests | yes if surfaced | yes | +| retained identity drift accepted by world submit | already yes | yes | yes | no | +| concurrent submitted turn collision gets queued implicitly | already yes | yes | yes | no | + +Critical gap rule for this plan: + +No failure mode is allowed to be both untested and silent. The stale-world relaunch path and the missing-active-host-runtime path are the main current hazards. + +## Performance Review + +Performance is not the main risk here, but there are still rules. + +### Findings resolved in-plan + +1. targeted follow-up dispatch must stay in-memory on the hot path. Do not add session-store scans or trace replays to decide host-vs-world routing. +2. exact backend dispatch must use existing descriptor and retained-runtime maps, not new filesystem lookup. +3. world relaunch must stay conditional. Same-generation reuse must not do unnecessary teardown or relaunch work. +4. test scaffolding may get more stateful, but runtime code must not add per-turn O(n) scans across unrelated backends if an exact keyed lookup already exists. + +### Performance posture + +- no new N+1 style data-access concern exists +- no caching layer is needed +- no background polling loop is needed +- targeted follow-up remains human-paced control-plane traffic +- correctness wins over micro-optimizing lookup structures here + +## DX Guardrails + +This is developer-facing runtime behavior even though it is backend code. + +Required operator experience: + +1. exact backend failures must still name the backend id the operator requested +2. host and world follow-up turns must feel like one contract even though they use different transports +3. Linux-only failure must be explicit, not implied by a timeout or absence of output +4. stale-world recovery must preserve exact backend identity in translated REPL output +5. docs must say REPL-first and Linux-first plainly so operators do not infer a broader public surface + +## Worktree Parallelization Strategy + +This plan has limited but real parallelization opportunities once the dispatch contract is frozen. + +### Dependency table + +| Step | Modules touched | Depends on | +| --- | --- | --- | +| Freeze targeted-follow-up contract and exact failure taxonomy | `crates/shell/src/repl/`, `crates/shell/src/execution/agent_runtime/`, repo docs | — | +| Dispatch seam extraction and shell-side error centralization | `crates/shell/src/repl/` | Freeze targeted-follow-up contract and exact failure taxonomy | +| Selector regression pinning | `crates/shell/src/execution/agent_runtime/` | Freeze targeted-follow-up contract and exact failure taxonomy | +| Integration tests, harness extensions, and repo-truth closeout | `crates/shell/tests/`, repo docs | Dispatch seam extraction and shell-side error centralization, Selector regression pinning | + +### Parallel lanes + +- Lane A: Dispatch seam extraction and shell-side error centralization + - sequential inside the lane because these steps share `crates/shell/src/repl/` +- Lane B: Selector regression pinning + - sequential inside the lane because these steps share `crates/shell/src/execution/agent_runtime/` +- Lane C: Integration tests, harness extensions, and repo-truth closeout + - starts after A and B because the assertions and doc truth depend on the final contract + +### Execution order + +1. Freeze the targeted-follow-up contract and exact failure taxonomy. +2. Launch Lane A and Lane B in parallel worktrees. +3. Merge A and B. +4. Run Lane C for positive host proof, stale-world relaunch proof, Linux-only fail-closed proof, and doc closeout. + +### Conflict flags + +- Lane A and Lane C both depend on [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) behavior. Lane C should not edit that file unless a test proves the seam is still missing a contract edge. +- Lane C owns [crates/shell/tests/repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs). Keep test-file ownership there to avoid merge churn. +- Repo docs should move last. Updating the gap matrix before the tests land is how documentation drifts from reality again. + +### Parallelization verdict + +Three workstreams, two parallel implementation lanes, one final integration lane. + +## Implementation Sequence + +### Step 1. Freeze the targeted-follow-up contract + +Files: + +1. [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) +2. [crates/shell/src/execution/agent_runtime/validator.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/validator.rs) +3. [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md) for language freeze only if needed during execution + +Deliver: + +1. freeze the exact failure taxonomy for targeted follow-up turns +2. document the route contract as host-active-only or world-exact-only, with no fallback +3. confirm the helper seam will own classification and error mapping, not selector semantics + +Done means the contract is explicit before helper extraction begins. + +### Step 2. Introduce one explicit targeted-follow-up dispatch seam + +Files: + +1. [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) + +Deliver: + +1. extract the inline targeted-turn route from the REPL loop into one helper that owns: + - exact backend selection + - host-vs-world classification + - host backend activity check + - Linux-only world gate + - world readiness call + - final call into host or world submit +2. keep `submit_host_targeted_turn(...)` and `submit_world_targeted_turn(...)` intact as transport implementations +3. keep reuse-vs-relaunch inside `ensure_member_runtime_ready_for_descriptor(...)` + +Done means one helper is the obvious place to read the selected-backend follow-up contract. + +### Step 3. Freeze host targeted-follow-up semantics with positive proof + +Files: + +1. [crates/shell/tests/repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs) +2. [crates/shell/tests/support/repl_world_agent.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/support/repl_world_agent.rs) only if harness support is needed + +Deliver: + +1. add a passing host targeted-follow-up test for the active orchestrator backend +2. add a missing-active-host-runtime fail-closed test if the runtime can be invalidated in the harness +3. assert no shell fallback and no world-member launch in both cases + +Done means the host path has both positive and negative proof. + +### Step 4. Freeze world selected-member relaunch semantics + +Files: + +1. [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) only if helper extraction still leaves relaunch ambiguity +2. [crates/shell/tests/repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs) +3. [crates/shell/tests/support/repl_world_agent.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/support/repl_world_agent.rs) if stale-generation scripting is needed + +Deliver: + +1. add a regression that simulates stale world generation and proves the exact backend slot is relaunched before submit +2. add a regression for explicit world-runtime-unavailable failure if the ready path cannot produce an authoritative-live selected member +3. keep the existing same-generation reuse proof unchanged + +Done means same-generation exact backend reuse remains the default and relaunch happens only after missing or stale retained state. + +### Step 5. Pin selector exactness where the dispatch seam depends on it + +Files: + +1. [crates/shell/src/execution/agent_runtime/validator.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/validator.rs) if tests are missing + +Deliver: + +1. add tests for any exact-backend behaviors the new seam relies on but are not yet pinned +2. especially protect exact unrealizable-backend behavior if it is not already covered + +Done means the dispatch seam depends on frozen selector behavior, not assumptions. + +### Step 6. Update repo-truth docs and planning index + +Files: + +1. [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md) +2. [llm-last-mile/README.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/README.md) if it is used as the current slice index + +Deliver: + +1. describe targeted follow-up submit and reuse as landed, Linux-first, and REPL-first +2. describe remaining open work as non-REPL caller surface, status/toolbox productization, and macOS parity +3. remove any stale language that still implies selected-member follow-up submit is broadly unimplemented + +Done means the repo says what the runtime actually does today. + +## Recommended Verification Commands + +```bash +cargo fmt --all -- --check +cargo clippy --workspace --all-targets -- -D warnings +cargo test -p shell validate_exact_backend_selection -- --nocapture +cargo test -p shell --test repl_world_first_routing_v1 -- --nocapture +cargo test -p world-agent member_runtime -- --nocapture +``` + +## Definition of Done + +1. `:: ` remains the only targeted follow-up grammar. +2. exact backend selection still resolves through `validate_exact_backend_selection(...)`. +3. host targeted follow-up turns resume only the active orchestrator backend. +4. world targeted follow-up turns on Linux still submit through `MemberTurnSubmitRequestV1`. +5. same-generation exact world-member reuse remains green. +6. stale or missing retained world state causes exact-backend relaunch before submit, not sibling fallback. +7. missing-active-host-runtime and non-Linux world-targeting failures remain explicit and fail closed. +8. no new non-REPL caller surface is introduced. +9. repo docs say REPL-first and Linux-first plainly. +10. the integration suite proves the contract instead of implying it. + +## Deferred Work + +- non-REPL caller surface +- `substrate -c` redesign +- public `substrate agent start|resume|fork|stop` +- status/toolbox ambiguity handling +- macOS/Lima parity +- broader operator-surface productization beyond the exact targeted REPL path + +## Completion Summary + +- Step 0: Scope Challenge, scope accepted as-is +- Architecture Review: 4 issues found, all resolved in-plan +- Code Quality Review: 4 issues found, all resolved in-plan +- Test Review: diagram produced, 5 concrete regression gaps identified +- Performance Review: 4 issues found, all resolved in-plan +- NOT in scope: written +- What already exists: written +- TODOS.md updates: 0 items proposed, deferred work stays inside this plan +- Failure modes: 4 critical gaps flagged until the new regression floor lands +- Outside voice: skipped for this document generation +- Parallelization: 3 lanes, 2 parallel / 1 sequential integration lane +- Lake Score: 8/8 recommendations chose the complete option + +## Decision Audit Trail + +| # | Phase | Decision | Classification | Principle | Rationale | Rejected | +| --- | --- | --- | --- | --- | --- | --- | +| 1 | Step 0 | Treat this as a contract-centralization and regression-floor slice, not a transport slice | Mechanical | Pragmatic | The submit transports and retained-member identity model already exist; the gap is cohesion and proof | reopening transport design | +| 2 | Architecture | Keep `validate_exact_backend_selection(...)` as the canonical explicit selector | Mechanical | DRY | One selector contract is safer than duplicating backend-id logic in the REPL | bespoke REPL-only exact selection | +| 3 | Architecture | Add one helper seam that chooses host vs world but does not abstract their transports together | Mechanical | Explicit over clever | Operators need one contract, not one fake implementation model | generic dispatcher framework | +| 4 | Architecture | Keep relaunch inside readiness, not submit | Mechanical | Systems over heroes | Submit should target retained identity, not mutate it | submit-triggered runtime creation | +| 5 | Code Quality | Keep changes inside existing shell and test modules | Mechanical | Minimal diff | The blast radius is small and already localized | new modules or services | +| 6 | Test Review | Make positive host-follow-up proof mandatory | Mechanical | Completeness | The host contract is not fully real until success is proven, not just rejection | inferring host behavior from negative tests | +| 7 | Test Review | Make stale-world relaunch proof mandatory | Mechanical | Completeness | Reuse-vs-relaunch is the main contract edge in this slice | relying on same-generation reuse tests only | +| 8 | Parallelization | Freeze contract first, then run shell seam work and selector pinning in parallel | Mechanical | Pragmatic | The shared contract is small but central, and it reduces merge churn | parallel edits before semantics are frozen | diff --git a/llm-last-mile/PLAN-18.md b/llm-last-mile/PLAN-18.md new file mode 100644 index 000000000..f067953e7 --- /dev/null +++ b/llm-last-mile/PLAN-18.md @@ -0,0 +1,798 @@ +# PLAN-18: Harden Agent Status Surfaces And Canonicalize Session Handle Semantics + +Source SOW: [18-status-surface-and-session-handle-hardening.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/18-status-surface-and-session-handle-hardening.md) +Gap matrix anchor: [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md) +Adjacent landed slices: [PLAN-15.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-15.md), [PLAN-16.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-16.md), [PLAN-17.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-17.md) +Branch: `feat/session-centric-state-store` +Base branch: `main` +Plan type: read-side status/control-surface hardening, developer-facing shell/runtime behavior, no new control command family +Review posture: `/autoplan` scope discipline with `/plan-eng-review` structure, rewritten as one cohesive execution plan +Status: execution-ready planning pass on 2026-05-05 +Outside voice: not used for this document generation + +## Objective + +Finish the narrow slice that must be true before public `start|resume|fork|stop` surfaces are safe to expose. + +This plan does three things and only three things: + +1. make `substrate agent status` render degraded but intelligible output instead of aborting on strict parent-session selection failures, +2. make trace-only fallback participant-aware so same-agent sibling participants inside one orchestration session do not collapse when trace evidence can distinguish them, +3. freeze one public naming contract now so later control surfaces do not inherit the current local handle ambiguity. + +This slice does not add new public control commands. It hardens the read-side and naming substrate those commands will rely on. + +## Plan Summary + +The repo is not missing agent runtime persistence, participant lineage, world-generation invalidation, or trace production. Those are already real: + +- strict active-session selection is implemented in [`resolve_single_live_session_for_agent(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:433), +- permissive session-record construction with warning accumulation already exists in [`build_session_record(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:858), +- live status projections already preserve `participant_id` in [`live_participant_status_projection(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:1140), +- producer-side trace rows already carry `participant_id`, `parent_participant_id`, and `resumed_from_participant_id` in [`AgentEvent`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/common/src/agent_events.rs:74) and are populated by [`agent_events.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_events.rs:117). + +What is still wrong is contract cohesion: + +1. `build_status_report(...)` still preflights the strict control-plane selector and therefore aborts status wholesale on ambiguity or stale linkage, even though the store can already preserve warnings and partial truth. +2. trace fallback suppression and parent-run correlation are still keyed too coarsely by `(orchestration_session_id, agent_id)` or `(orchestration_session_id, agent_id, execution.scope)`, which is not enough once sibling participants for the same agent can coexist inside one orchestration session. +3. persisted and local model names still overload "session handle" to mean a participant lineage id in some places and an upstream UAA session id in others. + +This slice does not invent new runtime behavior. It makes the already-landed behavior safe and legible for operators. + +## Locked Starting State + +### What already exists + +| Sub-problem | Existing code | Decision | +| --- | --- | --- | +| Strict active control target resolution | [`resolve_single_live_session_for_agent(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:433) | Keep strict. Do not weaken it for doctor, toolbox, or future mutating controls. | +| Permissive session record construction with warning accumulation | [`build_session_record(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:858) | Reuse as the status-read foundation instead of inventing a parallel warning model. | +| Status surface rendering and trace fallback merge | [`build_status_report(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:421) | Refactor this seam. Do not redesign unrelated agent commands. | +| Live participant status rows already expose `participant_id` | [`live_participant_status_projection(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:1140) | Preserve and extend with explicit provenance, not a new row model. | +| Producer-side trace rows already carry participant lineage | [`AgentEvent`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/common/src/agent_events.rs:74), [`agent_events.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_events.rs:117) | Reuse. Do not widen producer schema unless a regression proves a missing field. | +| Authoritative parent session record with overloaded `active_session_handle_id` | [`OrchestrationSessionRecord`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/orchestration_session.rs:24) | Keep storage shape for now, but stop treating the field name as canonical public semantics. | +| Participant model with local `session_handle_id` mirror and legacy aliases | [`AgentRuntimeParticipantHandle`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs:45), [`AgentRuntimeParticipantHandleWire`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs:528) | Preserve compatibility reads. Canonicalize docs and status/report naming instead of doing a storage flag day. | +| Regression anchors for strict fail-closed operator surfaces and status fallback rules | [`agent_successor_contract_ahcsitc0.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_successor_contract_ahcsitc0.rs) | Extend this suite. Do not replace it with a fresh harness. | + +### Exact remaining gap + +The remaining gap is narrower than the design doc had to assume: + +1. `agent status` still behaves like a control-surface selector gate because it calls the strict resolver up front in [`agents_cmd.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:446). +2. incomplete active parent linkage is already expressible as warnings in [`build_session_record(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:872), but that information is discarded before status rendering. +3. fallback suppression is still keyed by `(orchestration_session_id, agent_id, execution.scope)` in [`session_fallback_suppression_key(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:1185), so one live or invalidated participant can suppress a sibling that should remain visible. +4. pure-agent trace selection still keys by `(orchestration_session_id, agent_id)` in [`pure_session_key(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:1252), so trace-only sibling participants can collapse even though `participant_id` is present on the event. +5. nested parent-run correlation still groups selected parent runs by `(orchestration_session_id, agent_id)` in [`build_status_report(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:595), which is too coarse when multiple same-agent parents are plausible. +6. public/operator-facing naming still lacks one explicit rule separating `orchestration_session_id`, `participant_id`, and `internal.uaa_session_id`. + +## Frozen Execution Contract + +If implementation wants to do something else, revise this plan first. Do not freestyle past these rules. + +### Non-negotiable invariants + +1. `substrate agent status` becomes permissive in rendering, not permissive in authorization. +2. [`resolve_single_live_session_for_agent(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:433) stays fail-closed and remains the selector for doctor, toolbox, and future mutating control paths. +3. live authoritative participant/session state continues to beat trace fallback. +4. invalidated authoritative participant tombstones continue to suppress matching stale trace fallback rows. +5. participant-aware trace fallback uses `participant_id` when trace evidence supplies it. If trace evidence does not supply it, legacy coarse fallback behavior is allowed only for those legacy rows. +6. same-agent sibling participants in one orchestration session must remain distinct on the status surface when authoritative live state or trace lineage can distinguish them. +7. malformed selected tuples still fail closed. This slice does not relax world identity validation or nested tuple validation. +8. public parent-session selection uses `orchestration_session_id`. +9. `participant_id` remains a subordinate runtime/lineage identifier, not the canonical public session handle. +10. `internal.uaa_session_id` remains internal implementation detail and is never the default public selector. +11. compatibility reads for legacy `session_handle_id`, `parent_session_handle_id`, and `resumed_from_session_handle_id` remain supported. +12. new status/report surfaces must use canonical names even if storage still reads legacy aliases. + +### Public naming contract + +| Concept | Canonical name | Current source of truth | Rule after this slice | +| --- | --- | --- | --- | +| Parent orchestration selector | `orchestration_session_id` | [`OrchestrationSessionRecord`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/orchestration_session.rs:24) | This is the public control-plane session handle. | +| Runtime participant lineage id | `participant_id` | [`AgentRuntimeParticipantHandle`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs:45) | This stays visible in status, lineage, nesting, and debugging only. | +| Upstream backend-native handle | `internal.uaa_session_id` | [`AgentRuntimeSessionInternal`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs:75) | Internal only. Do not expose as the default operator target. | +| Legacy compatibility aliases | `session_handle_id`, `parent_session_handle_id`, `resumed_from_session_handle_id` | [`AgentRuntimeParticipantHandleWire`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs:528) | Read-compatible only. Never present these as canonical public terminology in new docs or status fields. | + +### Surface behavior split + +| Surface | Selection behavior | Trace behavior | Required result | +| --- | --- | --- | --- | +| `substrate agent status` | enumerate and degrade | fallback allowed, but never authoritative over live/tombstone state | render valid rows, attach warnings, never authorize control | +| `substrate agent doctor` | strict single-live-session selector | trace may inform diagnostics, not authorization | fail closed on ambiguity or stale parent linkage | +| `substrate agent toolbox status|env` | strict single-live-session selector | trace must not authorize endpoint selection | fail closed on ambiguity or stale parent linkage | +| future `substrate agent start|resume|fork|stop` | strict explicit selector | trace must not authorize control | fail closed on ambiguity or stale parent linkage | + +### Status report contract to freeze now + +These are exact contract choices, not examples: + +| Surface element | Exact contract | Why it is frozen now | +| --- | --- | --- | +| Status row provenance field | `StatusSessionJson.source_kind: "live_runtime" \| "trace_fallback"` | Operators need to see whether a row came from authoritative runtime state or from history. | +| Report warning field | `StatusReportJson.warnings: Vec` | Degraded status is allowed only if the surface says what degraded. | +| Warning ordering | sorted, deduplicated, human-readable strings | Output must stay stable in tests and readable in operator screenshots. | +| Record-level warnings | copied from `AgentRuntimeSessionRecord.warnings` | Reuse the store's existing warning truth instead of duplicating it. | +| Set-level warnings | emitted by `build_status_report(...)` for cross-record ambiguity such as multiple active parent candidates | `build_session_record(...)` only sees one record at a time. | + +### Participant-aware identity contract to freeze now + +Use one private key family everywhere status decides whether two rows represent the same logical participant: + +```text +StatusIdentityKey { + orchestration_session_id: String, + agent_id: String, + execution_scope: "host" | "world", + participant_id: Option, +} +``` + +Rules: + +1. if `participant_id` exists on the row, it participates in identity and equality +2. if `participant_id` is absent, fall back to the coarse legacy identity for that row only +3. the same key contract is used for pure-agent trace selection, live/tombstone fallback suppression, and selected-parent bucketing for nested correlation +4. this slice must not introduce a second "almost the same" tuple family + +## Step 0: Scope Challenge + +### 0A. What already exists + +| Sub-problem | Existing code | Decision | +| --- | --- | --- | +| Session enumeration across canonical + compatibility snapshots | [`list_sessions()`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:708) | Reuse. Add a status-specific filter, not a new storage scan path. | +| Active parent completeness checks and warnings | [`build_session_record(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:858) | Reuse as the degradation source of truth. | +| Status JSON rows and nested row shaping | [`StatusSessionJson`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:329), [`StatusReportJson`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:367) | Extend minimally with provenance and warnings. Do not create a second report format. | +| Trace event lineage fields | [`AgentEvent`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/common/src/agent_events.rs:74) | Reuse. Do not change producer schema unless a regression proves a missing correlation field. | +| Legacy handle alias support | [`legacy_handle_upgrade`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs:747) | Preserve. Canonicalize naming around it instead of deleting it. | +| Existing strict operator-surface failure tests | [`operator_surfaces_fail_closed_when_active_parent_omits_active_session_handle_id`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_successor_contract_ahcsitc0.rs:1514) and nearby tests | Keep green. This slice must not weaken them. | + +### 0B. Minimum honest diff + +The minimum honest implementation is: + +1. add `list_status_sessions_for_agent(&self, orchestrator_agent_id: &str) -> Result>` in [`state_store.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) so status can enumerate incomplete-but-readable records without authorizing control, +2. rework [`build_status_report(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:421) to consume that permissive path instead of strict preflight and to emit sorted, deduplicated report warnings, +3. add one participant-aware `StatusIdentityKey` family in [`agents_cmd.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs) and use it consistently for pure-agent trace selection, fallback suppression, and nested parent correlation, +4. extend status JSON/text output with the exact fields `source_kind` and `warnings`, +5. update tests and repo-truth docs so the public naming contract and degraded read-side contract are pinned. + +Anything broader is scope creep. Anything smaller leaves the core ambiguity un-fixed. + +### 0C. Complexity check + +This plan stays under the smell threshold if executed honestly. + +Expected production files: + +1. [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs) +2. [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) +3. [crates/shell/src/execution/agent_runtime/orchestration_session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/orchestration_session.rs) for comment/name clarification only if needed +4. [crates/shell/src/execution/agent_runtime/session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs) for naming comments and alias tests only + +Expected tests/docs: + +1. [crates/shell/tests/agent_successor_contract_ahcsitc0.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_successor_contract_ahcsitc0.rs) +2. [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md) +3. [llm-last-mile/README.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/README.md) if this index is used as current planning truth + +Rejected expansions: + +1. changing trace producer schema in [`crates/common/src/agent_events.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/common/src/agent_events.rs) unless a concrete missing-field regression appears +2. changing world-member launch, follow-up submit, or replacement semantics +3. adding the public `substrate agent start|resume|fork|stop` family +4. redesigning toolbox as a mutation surface +5. changing on-disk layout or dropping compatibility dual-read behavior +6. broad docs rewrites outside the gap matrix and active planning index + +### 0D. Search and completeness check + +Search-before-building result, in practical terms: + +- **[Layer 1]** reuse permissive warning construction in [`build_session_record(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:858) +- **[Layer 1]** reuse strict control-plane selection in [`resolve_single_live_session_for_agent(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:433) +- **[Layer 1]** reuse existing trace participant lineage fields instead of inventing new ones +- **[Layer 1]** reuse the existing status JSON surface and extend it minimally +- **[EUREKA]** the problem is not persistence or event emission anymore. The problem is that the read-side still throws away the permissive information the store already knows and then falls back with a key that is too coarse. +- **[EUREKA]** the cheapest complete version is not a new session model. It is one explicit split between strict control selection and permissive status rendering, plus participant-aware keys and regression proof. + +### 0E. Distribution check + +No new artifact type is introduced. No CI/CD or packaging work is required. + +The real distribution requirement here is operator truth: + +1. the CLI must say which rows are live and which are trace fallback, +2. it must not pretend a degraded surface is healthy when warnings exist, +3. it must not ask operators to learn the wrong handle names before public controls ship. + +### 0F. NOT in scope + +- adding `substrate agent start|resume|fork|stop`, because this slice is the read-side prerequisite for those controls, not the controls themselves +- redesigning `substrate -c`, because that is a caller-surface product decision unrelated to status correctness +- changing world-member follow-up submit or replacement behavior, because PLAN-17 already froze that runtime contract +- weakening doctor/toolbox authorization rules, because those surfaces must stay fail-closed even if status becomes permissive +- removing compatibility alias reads, because storage migration is not required to fix operator-facing naming +- changing trace producer schema unless a concrete missing-field regression appears, because the existing events already carry the lineage fields this slice needs +- replacing `active_session_handle_id` in storage with a new persisted field name, because that is a flag-day migration with no bearing on the immediate read-side bug +- macOS/Lima parity work, because this plan is about Linux-first status and naming cohesion, not cross-platform rollout + +## Architecture Review + +### Locked architecture decisions + +1. Add one status-specific enumeration seam named `list_status_sessions_for_agent(...)` in [`state_store.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs). +2. Keep [`resolve_single_live_session_for_agent(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:433) untouched for strict control-plane selection. +3. Extend the existing status report shape instead of creating a second status schema. +4. Use one participant-aware `StatusIdentityKey` family consistently for pure-agent trace projection, live/tombstone fallback suppression, and nested parent correlation. +5. Keep compatibility alias support at the serde boundary only. +6. Update docs and comments so public/operator-facing language says `orchestration_session_id` and `participant_id`, not `session_handle_id`. + +### Architecture findings resolved in-plan + +**Issue 1. Status is still paying the control-plane tax.** + +That is the main bug. A read-only surface should not die just because the authoritative parent is incomplete if it can still render truthful rows with warnings. + +**Issue 2. The store already knows how to degrade, but status ignores it.** + +This is the classic "we built the right primitive and then walked around it." [`build_session_record(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:858) already models incomplete active sessions and warnings. Status should consume that instead of re-imposing strictness. + +**Issue 3. Participant-aware trace fields exist, but the fallback reader still groups like the old world.** + +That is how sibling participants disappear. The trace producer is already richer than the trace consumer. + +**Issue 4. Naming drift will poison the next public surface if not frozen now.** + +If this slice leaves `session_handle_id` ambiguous, the later public control family will inherit a bad contract by default. That is the kind of bug that becomes documentation debt, then UX debt, then API debt. + +### Architecture ASCII diagrams + +### Status vs control selection split + +```text +STRICT CONTROL SURFACES +======================= +doctor / toolbox / future start|resume|fork|stop + | + v +resolve_single_live_session_for_agent(...) + | + +--> one authoritative active parent + one live host orchestrator --> authorize + | + +--> ambiguity / stale linkage / missing active handle -----------> fail closed + +PERMISSIVE STATUS SURFACE +========================= +agent status + | + v +list status candidate sessions + | + +--> complete authoritative records ------------------------------> project live rows + | + +--> incomplete authoritative records with live participants ----> project valid rows + warnings + | + +--> no live authoritative row ----------------------------------> allow trace fallback + | + \--> malformed selected tuple -----------------------------------> fail closed only for that malformed selected record class +``` + +### Status row precedence and fallback suppression + +```text +STATUS ROW PRECEDENCE +===================== +authoritative live participant row + beats +trace fallback row with same participant-aware identity + +invalidated authoritative tombstone + suppresses +trace fallback row with same participant-aware identity + +trace fallback row + only fills +gaps where no live row and no invalidated tombstone exists + +identity key +============ +(orchestration_session_id, agent_id, execution.scope, participant_id?) + +rules: +- if participant_id exists, use it +- if participant_id is absent, fall back to legacy coarse identity for that row only +- never let one participant suppress an unrelated sibling when participant_id differs +``` + +### Nested parent correlation with participant awareness + +```text +selected pure-agent parent rows + | + v +index by: +(orchestration_session_id, agent_id, participant_id?) + | + +--> if nested row has parent_participant_id + | require match inside that participant bucket + | + +--> else + | use legacy session+agent bucket + | + \--> parent_run_id must still match either: + - the selected winning parent run in that bucket, or + - a known historical pure-agent run in that bucket +``` + +### Public naming split that must remain unchanged + +```text +public/operator-facing parent selector + orchestration_session_id + +runtime lineage / nesting / debugging + participant_id + +backend-native upstream runtime handle + internal.uaa_session_id + +legacy alias names + serde-read compatibility only +``` + +## Code Quality Review + +### Findings resolved in-plan + +**Issue 1. The read-side contract should live in one obvious place.** + +`build_status_report(...)` currently mixes projection, fallback merge, validation, and strict preflight assumptions. This slice should leave one readable story instead of another fossil dig. + +**Issue 2. Minimal diff matters here.** + +The repo already has the right data. This is not the moment for a new status module, new storage format, or a generic "identity resolver" framework. + +**Issue 3. Storage comments and serde aliases need to stop teaching the wrong vocabulary.** + +We do not need a schema migration. We do need comments, tests, and report fields that say the right thing. + +**Issue 4. Warning handling must be deliberate, not incidental.** + +If degraded status output exists, operators need to see why it degraded. Silent partial truth is just a different kind of lie. + +### Allowed code shape + +1. Prefer one small status-read helper named `list_status_sessions_for_agent(...)` in [`state_store.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) over a new state-store abstraction layer. +2. Prefer one small private `StatusIdentityKey` type or helper family in [`agents_cmd.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs) over ad hoc tuple rewrites in three separate spots. +3. Extend [`StatusSessionJson`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:329) with the exact field `source_kind` and [`StatusReportJson`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:367) with the exact field `warnings`. Do not create a shadow JSON shape. +4. Keep legacy alias support in [`session.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs:528). Do not rename persisted fields in this slice. +5. Use comments to freeze the naming contract near [`OrchestrationSessionRecord`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/orchestration_session.rs:24) and [`AgentRuntimeParticipantHandle`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs:45), because those are where future control-surface authors will look first. + +## Test Review + +### Test framework detection + +This repo is Rust-first and the relevant review surface is `cargo test`. + +Primary suites for this slice: + +1. [crates/shell/tests/agent_successor_contract_ahcsitc0.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_successor_contract_ahcsitc0.rs) +2. unit tests in [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) +3. unit tests in [crates/shell/src/execution/agent_runtime/session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs) + +### Code path coverage + +```text +CODE PATH COVERAGE +=========================== +[+] crates/shell/src/execution/agent_runtime/state_store.rs + | + ├── resolve_single_live_session_for_agent(...) + │ ├── [★★★ TESTED] multiple active parent sessions fail closed + │ ├── [★★★ TESTED] missing active parent fails closed + │ ├── [★★★ TESTED] missing active_session_handle_id fails closed + │ └── [★★★ TESTED] inactive/mismatched orchestrator participant fails closed + | + └── new status-specific session enumeration helper + ├── [GAP] incomplete active parent with live readable participants degrades to warnings, not error + ├── [GAP] multiple active parent candidates remain visible to status as warnings, not abort + └── [GAP] missing active_session_handle_id remains visible to status as warning, not abort + +[+] crates/shell/src/execution/agents_cmd.rs + | + ├── build_status_report(...) + │ ├── [★★★ TESTED] live manifest beats trace fallback for selected orchestrator + │ ├── [★★★ TESTED] invalidated tombstone beats stale trace fallback + │ ├── [★★★ TESTED] same-agent concurrent live rows from authoritative session records remain visible + │ ├── [GAP] status no longer hard-preflights strict live-session selection + │ ├── [GAP] status emits warnings when parent linkage is incomplete + │ ├── [GAP] status marks row provenance as live_runtime vs trace_fallback + │ └── [GAP] malformed selected tuples still fail closed after the refactor + | + ├── participant-aware pure-agent trace projection + │ ├── [GAP] trace-only sibling participants with distinct participant_id stay distinct + │ └── [GAP] legacy trace rows without participant_id still collapse only within their own coarse bucket + | + ├── fallback suppression + │ ├── [★★★ TESTED] tombstone suppression works for coarse tuple today + │ ├── [GAP] live/tombstone suppression only suppresses the matching participant-aware row + │ └── [GAP] sibling participant fallback rows survive when only one sibling has a live or tombstoned match + | + └── nested parent correlation + ├── [★★★ TESTED] malformed nested tuples fail closed on selected rows + ├── [★★★ TESTED] malformed parent correlation fails closed today + └── [GAP] parent_participant_id-aware nested correlation chooses the right sibling parent bucket + +[+] crates/shell/src/execution/agent_runtime/session.rs + | + ├── legacy alias deserialization + │ └── [★★★ TESTED] legacy session_handle_id aliases deserialize into participant fields + | + └── canonical naming/documentation expectations + └── [GAP] tests/comments pin that canonical reporting uses participant/session terminology, not legacy alias names + +--------------------------------- +COVERAGE TARGET +- strict control-surface fail-closed tests stay green +- degraded status rendering paths gain direct regression proof +- participant-aware fallback and nested correlation get sibling-specific proof +- naming compatibility stays green while canonical terminology moves forward +--------------------------------- +``` + +### Operator flow coverage + +```text +OPERATOR FLOW COVERAGE +=========================== +[+] Operator runs `substrate agent status --json` during healthy live runtime + | + ├── [★★★ TESTED] live authoritative rows win over trace fallback + └── [GAP] each row exposes provenance = live_runtime + +[+] Operator runs `substrate agent status --json` with incomplete parent linkage + | + ├── [GAP] [->E2E] missing active_session_handle_id produces readable rows + warning + ├── [GAP] [->E2E] multiple active parent candidates produce warnings instead of whole-surface abort + └── [GAP] [->E2E] trace fallback fills only missing rows and is clearly labeled + +[+] Operator runs `substrate agent status --scope world --json` after same-agent sibling activity + | + ├── [★★★ TESTED] authoritative same-agent concurrent sessions stay visible + ├── [GAP] [->E2E] trace-only sibling participants stay visible when participant_id differs + └── [GAP] [->E2E] tombstone/live suppression only suppresses the matching sibling + +[+] Operator runs `substrate agent doctor --json` or `toolbox env --json` under ambiguous parent linkage + | + └── [★★★ TESTED] fail closed, no status-style degradation leaks into authorization paths +``` + +### Required tests to add or extend + +1. Add status regressions proving that `substrate agent status --json` no longer aborts when the active parent omits `active_session_handle_id`, and instead returns successful output with warnings plus whatever authoritative live rows remain readable. +2. Add status regressions proving that multiple active parent-session candidates for the same orchestrator agent are warnings on `agent status`, not fatal selection errors, while doctor/toolbox remain fail closed. +3. Add a trace-only sibling-participant regression showing that two status events with the same `(orchestration_session_id, agent_id)` but distinct `participant_id` values remain distinct on the status surface. +4. Add a sibling-specific suppression regression proving that a live or invalidated participant suppresses only the matching trace fallback row, not every sibling row sharing the same agent id and orchestration session. +5. Add a nested parent-correlation regression proving that `parent_participant_id` narrows the selected-parent bucket when present and preserves the current fail-closed behavior when the nested tuple is malformed. +6. Preserve existing strict fail-closed tests for `agent doctor`, `toolbox env`, and the strict selector unit tests. Those are not optional. +7. Keep the existing legacy alias deserialization test green and add one small naming assertion if status/report output gains canonical naming metadata. + +### QA-facing test artifact + +During implementation, write a QA-facing artifact to: + +```text +~/.gstack/projects//-feat-session-centric-state-store-eng-review-test-plan-.md +``` + +Required contents: + +1. healthy `agent status --json` with live authoritative rows +2. degraded `agent status --json` with missing `active_session_handle_id` +3. degraded `agent status --json` with multiple active parent candidates +4. same-agent sibling participant visibility under trace-only fallback +5. doctor/toolbox fail-closed behavior under the same degraded parent states + +This artifact is for `/qa` and `/qa-only`. Keep it operator-journey oriented. + +### Regression rule for this slice + +These tests are mandatory. No discussion: + +1. doctor/toolbox strict fail-closed behavior stays green +2. live authoritative rows still beat trace fallback +3. tombstones still beat stale trace fallback +4. same-agent concurrent live authoritative rows stay visible +5. legacy `session_handle_id` aliases still deserialize +6. degraded status paths are directly tested, not inferred from helper behavior + +## Failure Modes Registry + +| Failure mode | Test required | Error handling exists | Operator sees clear result | Critical gap before this slice lands | +| --- | --- | --- | --- | --- | +| status still aborts when active parent omits `active_session_handle_id` | yes | partial today, but only as hard error | no | yes | +| status still aborts on multiple active parent candidates | yes | partial today, but only as hard error | no | yes | +| trace-only sibling participants collapse into one row | yes | no | no | yes | +| one sibling tombstone suppresses a different sibling's fallback row | yes | no | no | yes | +| nested row attaches to the wrong sibling parent when `parent_participant_id` is present | yes | no | no | yes | +| degraded status output omits why it degraded | yes | no | no | yes | +| doctor/toolbox accidentally inherit permissive status behavior | yes | yes today via strict selector | yes | yes | +| malformed selected tuple becomes silently tolerated during refactor | yes | yes today | yes | yes | +| canonical naming changes break legacy snapshot reads | yes | yes today via serde aliases | yes | no | + +Critical gap rule for this plan: + +No degraded status path is allowed to be both partially truthful and silent. If the surface degrades, it must say why. + +## Performance Review + +Performance is not the main risk here, but the read-side can still get sloppier than it needs to. + +### Findings resolved in-plan + +1. Do not add extra filesystem passes when the store already materializes session records in one place. Status should consume a status-specific list helper, not re-scan storage three different ways. +2. Use one small participant-aware key type and `BTreeMap`/`BTreeSet` lookups. Do not introduce nested O(n²) sibling matching across the whole report. +3. Keep trace ingestion as the current single pass over `trace.jsonl`. The change is key shape and warning merge, not a second trace-processing pipeline. +4. Status remains human-paced operator traffic. Correctness and debuggability matter more than shaving a few map allocations. + +### Performance posture + +- no new N+1 data-access pattern is acceptable +- no background cache or index is needed +- no new persistence file is needed +- participant-aware keying must stay in-memory and linearithmic at worst +- live authoritative precedence must remain the first choice, not a fallback after extra work + +## DX Guardrails + +This is a developer/operator surface even though it is backend code. + +Required operator experience: + +1. `agent status` must still succeed in degraded states where doctor/toolbox would fail closed. +2. the output must say which rows are `live_runtime` and which are `trace_fallback`. +3. warning text must name the broken parent/session condition in plain terms, not bury it in an internal stack trace. +4. new docs and report shapes must say `orchestration_session_id` and `participant_id`. +5. no surface in this slice should ask an operator to type or target `uaa_session_id`. + +## Worktree Parallelization Strategy + +This plan has limited but real parallelization opportunities once the naming and degradation contract is frozen. + +### Dependency table + +| Step | Modules touched | Depends on | +| --- | --- | --- | +| Freeze degraded-status contract, naming contract, and participant-aware identity rules | `crates/shell/src/execution/`, `crates/shell/src/execution/agent_runtime/`, repo docs | — | +| Status-specific session enumeration and naming/comment cleanup | `crates/shell/src/execution/agent_runtime/` | Freeze degraded-status contract, naming contract, and participant-aware identity rules | +| Status rendering refactor, participant-aware fallback, and nested correlation tightening | `crates/shell/src/execution/` | Freeze degraded-status contract, naming contract, and participant-aware identity rules | +| Regression tests and repo-truth closeout | `crates/shell/tests/`, repo docs | Status-specific session enumeration and naming/comment cleanup, Status rendering refactor and participant-aware fallback | + +### Parallel lanes + +- Lane A: Status-specific session enumeration and naming/comment cleanup + - sequential inside the lane because these steps share `crates/shell/src/execution/agent_runtime/` +- Lane B: Status rendering refactor, participant-aware fallback, and nested correlation tightening + - sequential inside the lane because these steps share `crates/shell/src/execution/agents_cmd.rs` +- Lane C: Regression tests and repo-truth closeout + - starts after A and B because the tests and docs depend on the frozen helper behavior and key contract + +### Execution order + +1. Freeze the degraded-status contract and participant-aware identity rules. +2. Launch Lane A and Lane B in parallel worktrees. +3. Merge A and B. +4. Run Lane C for degraded-status regressions, sibling-participant regressions, strict-surface non-regressions, and docs closeout. + +### Conflict flags + +- Lane A and Lane B are intentionally split at the module boundary, but they still share the naming contract. Freeze the rule set first or you will get two "obvious" implementations that disagree. +- Lane B owns [`crates/shell/src/execution/agents_cmd.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs). Keep that ownership exclusive to avoid merge churn in the status refactor. +- Lane C owns [`crates/shell/tests/agent_successor_contract_ahcsitc0.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_successor_contract_ahcsitc0.rs). Do not let implementation lanes opportunistically add tests there mid-flight. +- Docs move last. Updating the gap matrix before the regressions land is how truth drifts again. + +### Parallelization verdict + +Three workstreams, two parallel implementation lanes, one final integration lane. + +## Implementation Sequence + +### Step 1. Freeze the degraded-status and naming contract + +Files: + +1. [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs) +2. [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) +3. [crates/shell/src/execution/agent_runtime/orchestration_session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/orchestration_session.rs) +4. [crates/shell/src/execution/agent_runtime/session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs) + +Deliver: + +1. decide and document that `agent status` is permissive and doctor/toolbox remain strict +2. freeze the exact JSON/report additions: + - `StatusSessionJson.source_kind` + - `StatusReportJson.warnings` + - warning strings sorted and deduplicated before render +3. freeze the participant-aware key contract: + - `(orchestration_session_id, agent_id, execution.scope, participant_id?)` + - participant-specific when present + - legacy coarse fallback only when participant identity is absent +4. freeze the naming contract: + - `orchestration_session_id` public selector + - `participant_id` runtime lineage id + - `internal.uaa_session_id` internal only +5. freeze warning ownership: + - record-local degradation comes from `AgentRuntimeSessionRecord.warnings` + - cross-record ambiguity warnings are emitted by `build_status_report(...)` + +Done means the contract is explicit before helper and projection work begins. + +### Step 2. Add one status-specific session enumeration seam + +Files: + +1. [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) + +Deliver: + +1. add `list_status_sessions_for_agent(...)` that enumerates status-visible session records for the selected orchestrator agent without calling the strict single-live-session selector +2. preserve `warnings`, `complete`, and `has_authoritative_parent` from [`build_session_record(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:858) +3. include every record from `list_sessions()` whose authoritative parent targets the selected orchestrator agent, plus parentless live-host-orchestrator cases for that agent that status must render as degraded warnings +4. keep owner-pid liveness and authoritative session/participant validation rules where they already belong +5. do not add a second record-construction path; this helper must still route through `build_session_record(...)` +6. do not weaken [`resolve_single_live_session_for_agent(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:433) + +Done means status can enumerate incomplete-but-readable active records without authorizing control. + +### Step 3. Rewire status rendering onto the permissive seam + +Files: + +1. [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs) + +Deliver: + +1. remove the unconditional strict preflight from [`build_status_report(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:446) +2. consume `list_status_sessions_for_agent(...)` +3. extend [`StatusSessionJson`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:329) with the exact field `source_kind` +4. extend [`StatusReportJson`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:367) with `warnings: Vec` +5. populate `warnings` as the sorted, deduplicated union of: + - selected record warnings + - cross-record ambiguity warnings such as multiple active parent candidates +6. render warnings in text mode before the `sessions` block and include them in JSON mode unchanged + +Done means degraded status succeeds, is explicit about degradation, and stays schema-minimal. + +### Step 4. Make pure-agent fallback and suppression participant-aware + +Files: + +1. [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs) + +Deliver: + +1. replace the current coarse pure-agent trace key in [`pure_session_key(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:1252) with `StatusIdentityKey` +2. replace the current suppression keys in [`session_fallback_suppression_key(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:1185) and [`participant_fallback_suppression_key(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:1193) with the same `StatusIdentityKey` +3. preserve legacy coarse grouping only for rows that truly lack participant identity +4. keep live authoritative rows and invalidated tombstones winning over trace fallback + +Done means sibling participants no longer disappear just because they share `agent_id`. + +### Step 5. Make nested parent correlation participant-aware without weakening validation + +Files: + +1. [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs) + +Deliver: + +1. key selected parent runs by `(orchestration_session_id, agent_id, participant_id?)` +2. when a nested row has `parent_participant_id`, require correlation within that participant bucket before checking `parent_run_id` +3. when a nested row lacks `parent_participant_id`, preserve current legacy session+agent fallback behavior +4. keep malformed tuple and malformed parent-run validation fail closed + +Done means the nested gateway surface attaches to the right parent sibling when lineage exists and still refuses malformed rows. + +### Step 6. Freeze the contract with regression tests + +Files: + +1. [crates/shell/tests/agent_successor_contract_ahcsitc0.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_successor_contract_ahcsitc0.rs) +2. [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) unit tests +3. [crates/shell/src/execution/agent_runtime/session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs) unit tests + +Deliver: + +1. add degraded-status success-path tests for missing active handle and multiple active parents +2. add participant-aware trace-only sibling visibility tests +3. add participant-aware suppression specificity tests +4. add parent-participant-aware nested correlation tests +5. preserve strict doctor/toolbox fail-closed tests +6. preserve legacy alias deserialization tests + +Done means the read-side and naming contract are proven, not implied. + +### Step 7. Update repo-truth docs + +Files: + +1. [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md) +2. [llm-last-mile/README.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/README.md) if this plan index is used + +Deliver: + +1. mark status ambiguity handling as landed once regressions are green +2. narrow the session-handle gap language to remaining public control-surface productization instead of current local ambiguity +3. describe the canonical naming contract plainly +4. describe trace-only fallback as participant-aware once the tests prove it + +Done means the repo says what the runtime actually does now. + +## Recommended Verification Commands + +```bash +cargo fmt --all -- --check +cargo clippy --workspace --all-targets -- -D warnings +cargo test -p shell --lib -- --nocapture +cargo test -p shell --test agent_successor_contract_ahcsitc0 -- --nocapture +``` + +Manual spot checks after tests are green: + +```bash +substrate agent status --json +substrate agent doctor --json +substrate agent toolbox env --json +``` + +## Definition of Done + +1. `agent status` no longer aborts on strict parent-selection ambiguity or stale linkage that can be rendered truthfully with warnings. +2. doctor and toolbox surfaces remain strict and fail closed. +3. status rows explicitly identify whether they come from `live_runtime` or `trace_fallback`. +4. same-agent sibling participants remain distinct on the status surface when lineage evidence exists. +5. live/tombstone suppression only suppresses matching participant-aware fallback rows. +6. nested parent correlation respects `parent_participant_id` when present and still rejects malformed tuples. +7. canonical naming in docs and status/report surfaces uses `orchestration_session_id` and `participant_id`. +8. legacy alias reads remain green. +9. no new public `start|resume|fork|stop` surface is introduced. +10. repo-truth docs reflect the landed behavior. + +## Deferred Work + +- public `substrate agent start|resume|fork|stop` +- explicit non-REPL caller surfaces +- storage-level rename away from `active_session_handle_id` +- broader toolbox mutation surface +- any producer-side trace schema expansion beyond currently shipped lineage fields +- macOS/Lima parity and broader operator-surface productization + +## Completion Summary + +- Step 0: Scope Challenge, scope accepted as-is +- Architecture Review: 4 issues found, all resolved in-plan +- Code Quality Review: 4 issues found, all resolved in-plan +- Test Review: diagram produced, 7 concrete regression gaps identified +- Performance Review: 4 issues found, all resolved in-plan +- NOT in scope: written +- What already exists: written +- TODOS.md updates: 0 items proposed, no `TODOS.md` exists in the repo today +- Failure modes: 6 critical gaps flagged until the new regression floor lands +- Outside voice: skipped for this document generation +- Parallelization: 3 lanes, 2 parallel / 1 sequential integration lane +- Lake Score: 9/9 recommendations chose the complete option + +## Decision Audit Trail + +| # | Phase | Decision | Classification | Principle | Rationale | Rejected | +| --- | --- | --- | --- | --- | --- | --- | +| 1 | Step 0 | Treat this as a read-side contract split, not a persistence or producer-schema slice | Mechanical | Pragmatic | The hard parts already landed; the bug is how the status surface consumes them | reopening storage or trace producer design | +| 2 | Architecture | Keep strict control selection in `resolve_single_live_session_for_agent(...)` | Mechanical | Explicit over clever | One helper should stay authoritative for control-plane safety | weakening the selector to make status easier | +| 3 | Architecture | Add a status-specific enumeration helper instead of teaching status to fake permissive selection itself | Mechanical | DRY | The store already knows how to build incomplete session records with warnings | duplicating permissive warning logic in `agents_cmd.rs` | +| 4 | Architecture | Extend the existing status report shape with provenance and warnings | Mechanical | Minimal diff | Operators need more truth, not a second schema | brand-new status JSON model | +| 5 | Architecture | Use one participant-aware identity key family everywhere fallback identity matters | Mechanical | Systems over heroes | One contract across selection, suppression, and nesting avoids sibling drift | three unrelated tuple rewrites | +| 6 | Code Quality | Keep compatibility aliases at serde-read boundaries only | Mechanical | Pragmatic | This avoids a storage flag day while still fixing public naming | renaming persisted fields in this slice | +| 7 | Test Review | Make degraded-status success cases mandatory regression tests | Mechanical | Completeness | The whole point of the slice is that status no longer dies on readable ambiguity | relying on unit helpers without CLI-surface proof | +| 8 | Test Review | Make sibling-specific suppression proof mandatory | Mechanical | Completeness | This is the main correctness edge once participant-aware fallback lands | assuming the key refactor is obviously right | +| 9 | Parallelization | Freeze naming and degradation rules first, then run state-store and status-surface lanes in parallel | Mechanical | Pragmatic | The module split is clean, but the contract is shared and easy to drift on | uncoordinated parallel edits before the rules are frozen | diff --git a/llm-last-mile/PLAN-19.md b/llm-last-mile/PLAN-19.md new file mode 100644 index 000000000..9c911574d --- /dev/null +++ b/llm-last-mile/PLAN-19.md @@ -0,0 +1,976 @@ +# PLAN-19: Publicize Agent Control Surfaces With Exact Session Selectors + +Source SOW: [19-public-agent-control-surfaces.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/19-public-agent-control-surfaces.md) +Gap matrix anchor: [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md) +Adjacent landed slices: [PLAN-15.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-15.md), [PLAN-16.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-16.md), [PLAN-17.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-17.md), [PLAN-18.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-18.md) +Branch: `feat/session-centric-state-store` +Base branch: `main` +Plan type: public CLI control-plane productization for orchestration sessions +Review posture: unified execution plan, tightened to `/plan-eng-review` structure and rigor +Status: execution-ready planning pass on 2026-05-05 + +## Objective + +Expose one narrow public control surface under `substrate agent` that turns orchestration session lifecycle into an explicit product contract instead of a REPL-only side effect. + +This slice does four things and only four things: + +1. adds public `start`, `resume`, `fork`, and `stop` verbs under `substrate agent`, +2. freezes `orchestration_session_id` as the only public selector for existing sessions, +3. keeps backend-native `internal.uaa_session_id` internal while still using it for exact resume and fork, +4. fails closed anywhere Substrate cannot prove backend exactness, ownership, linkage, platform posture, or policy allowability. + +This slice does not add a prompt-taking caller surface. It does not redesign `substrate -c`. It does not redesign the REPL `:: ` grammar. It does not turn the toolbox into a mutation plane. It does not introduce a general daemon. + +## Plan Summary + +The repo already has most of the raw mechanics this feature needs. The missing piece is not "how to launch an agent." The missing piece is honest retained ownership once a short-lived public CLI command exits. + +Repo truth today: + +- the public CLI still exposes inspection only in [`AgentAction`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/cli.rs:471), +- status already uses permissive read-side enumeration through [`list_status_sessions_for_agent(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:423) while strict control selectors stay fail closed through [`resolve_single_live_session_for_agent(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:444), +- host startup and shutdown already exist inside the REPL in [`start_host_orchestrator_runtime(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:1802) and [`shutdown_host_orchestrator_runtime(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:4090), +- targeted follow-up turns already shape exact resume extensions through [`build_session_resume_extension(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:3776), +- the state store already treats [`orchestration_session_id`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/orchestration_session.rs:24) as the public parent-session authority and keeps [`participant_id`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs:45) and [`internal.uaa_session_id`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs:75) separate, +- inventory and validator code already model `session_start`, `session_resume`, `session_fork`, and `session_stop` capability bits in [`agent_inventory.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_inventory.rs:89) and [`validator.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/validator.rs:246). + +The hard constraint is simple: + +Public `substrate agent start|resume|fork` commands cannot return success unless something keeps the cancel handle, event stream, completion observer, and ownership heartbeat alive after the calling CLI exits. Today that owner is still process-bound to the live REPL. + +This plan resolves that with one narrow owner plane: + +1. extract shared orchestration lifecycle logic out of `async_repl.rs`, +2. introduce one private per-session owner loop that can run either inside the REPL or inside a hidden helper subprocess, +3. expose one private per-session control transport with exactly one v1 mutation verb, `stop`, +4. make public `start`, `resume`, and `fork` launch or reconnect that owner loop, wait for authoritative readiness, emit the public handle, then exit, +5. keep `stop` routed through the live owner process so shutdown stays authoritative instead of pretending JSON mutation is control. + +That is the minimum honest diff. Anything smaller ships a surface that lies. + +## Locked Starting State + +### What already exists + +| Sub-problem | Existing code | Decision | +| --- | --- | --- | +| Public inspection CLI namespace | [`crates/shell/src/execution/cli.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/cli.rs) | Extend `AgentAction`. Do not create a second top-level namespace. | +| Status vs strict control split | [`build_status_report(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:131), [`list_status_sessions_for_agent(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:423), [`resolve_single_live_session_for_agent(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:444) | Reuse. Status may degrade, mutating controls may not. | +| Host orchestrator startup | [`prepare_host_orchestrator_runtime_startup(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:1813), [`start_host_orchestrator_runtime_with_prepared(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:1930) | Extract into shared control code. Do not duplicate in `agents_cmd.rs`. | +| Host targeted resume extension shaping | [`AGENT_API_SESSION_RESUME_V1`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:1572), [`build_session_resume_extension(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:3776) | Reuse the exact resume shape. | +| World-member readiness and exact-backend reuse | [`dispatch_targeted_follow_up_turn(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:2691), [`ensure_member_runtime_ready_for_descriptor(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:3619) | Reuse only for Linux world-sensitive reuse/stop posture. Do not broaden into public world-root start. | +| Authoritative parent session model | [`OrchestrationSessionRecord`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/orchestration_session.rs:24) | Keep `orchestration_session_id` as the public parent handle. | +| Participant lineage model and internal UAA-handle separation | [`AgentRuntimeParticipantHandle`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs:45), [`AgentRuntimeSessionInternal`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs:75) | Preserve. Public commands may read `internal.uaa_session_id`, never accept it as input. | +| Exact backend selection | [`validate_exact_backend_selection(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/validator.rs:246) | Reuse as the only public `start --backend` selector. | +| Capability gates | [`missing_required_orchestrator_capability(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/validator.rs:96) | Reuse for public start, resume, fork, and stop eligibility checks. | +| Unified Agent API selector grammar | local dependency `unified-agent-api 0.2.3` | Freeze `resume` and `fork` to the same `{selector,id}` object shape. Never send both together. | +| Internal shutdown path | [`shutdown_host_orchestrator_runtime(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:4090) | Reuse through the owner loop only. Never fake stop by rewriting `session.json`. | + +### Exact remaining gap + +The remaining gap is narrower than the SOW implied: + +1. the public CLI namespace still has no `start|resume|fork|stop` verbs, +2. the lifecycle code public verbs need is still trapped inside `async_repl.rs` private types and functions, +3. there is no durable owner loop outside the live REPL, so a short-lived public `start` would currently lose authoritative control on exit, +4. there is no strict public-action resolver that takes exact `orchestration_session_id`, rejects non-canonical handles, validates linkage, then selects action-specific behavior, +5. there is no public stop transport to reach the retained owner, +6. `fork` capability is advertised by inventory and UAA backend bindings, but the shell does not yet shape or route a fork extension on the orchestration path, +7. the docs still describe public control-plane productization as missing and do not explain the owner-model constraint that makes a naive immediate-return CLI invalid. + +## Frozen Execution Contract + +If implementation wants to do something else, revise this plan first. + +### Non-negotiable invariants + +1. Public existing-session targeting accepts only `--session `. +2. Public root-session creation accepts only `--backend `. +3. No public command accepts `participant_id`, `active_session_handle_id`, `session_handle_id`, or `internal.uaa_session_id` as input. +4. `start`, `resume`, and `fork` are short-lived launch commands. They return only after the owner loop has established authoritative state or failed explicitly. +5. `stop` never rewrites state-store rows directly to terminal. It must reach the live owner loop and let that owner perform authoritative shutdown. +6. Strict control-plane actions stay fail closed. No status-style warning degradation is allowed in `start`, `resume`, `fork`, or `stop`. +7. Public root session creation is host-orchestrator only in v1. A world-scoped backend is not a valid public root target. +8. World-sensitive stop or reuse logic remains Linux-first. If the selected live session depends on authoritative shared-world/member-runtime posture that the current platform cannot prove, return `unsupported_platform_or_posture`. +9. `resume` and `fork` use exact UAA selector objects and are mutually exclusive. +10. Public output may surface `participant_id` for debugging, but `orchestration_session_id` is always the public session handle. +11. This slice does not add prompt submission. A session can be created, resumed, forked, or stopped without widening caller semantics. +12. No new general daemon is introduced. The only new always-on ownership surface is a per-session owner loop. + +### Public verb contract + +```text +substrate agent start --backend [--json] +substrate agent resume --session [--json] +substrate agent fork --session [--json] +substrate agent stop --session [--json] +``` + +### Exact selector rules + +| Command | Required selector | Resolution rule | Explicitly rejected | +| --- | --- | --- | --- | +| `start` | `--backend ` | exact host-scoped backend match through `validate_exact_backend_selection(...)` | agent id, label, default backend, world-scoped root start | +| `resume` | `--session ` | exact parent session id, exact active participant lookup, exact internal `uaa_session_id` reuse | `participant_id`, `uaa_session_id`, fuzzy "latest" | +| `fork` | `--session ` | exact parent session id, exact active participant lookup, exact internal `uaa_session_id` fork source | `participant_id`, `uaa_session_id`, fuzzy "latest" | +| `stop` | `--session ` | exact parent session id, exact live owner transport, exact owner-mediated shutdown | direct store mutation, PID-only heuristic, fuzzy "latest" | + +### Success output contract + +| Field | `start` | `resume` | `fork` | `stop` | +| --- | --- | --- | --- | --- | +| `action` | `start` | `resume` | `fork` | `stop` | +| `orchestration_session_id` | new session id | existing session id | new session id | existing session id | +| `source_orchestration_session_id` | absent | absent | present | absent | +| `backend_id` | present | present | present | present | +| `scope` | `host` | `host` or `host_with_world_posture` | `host` or `host_with_world_posture` | same as resolved live session | +| `participant_id` | new active participant | new active participant | new active participant | final active participant before stop, optional | +| `state` | `active` | `active` | `active` | `stopped` or `invalidated` | +| `warnings` | empty | empty | empty | empty | + +Rules: + +1. no success output includes `internal.uaa_session_id`, +2. text mode prints the same information as a single deterministic summary line, +3. JSON mode is stable and exact so tests can pin it, +4. no success path reports readiness until authoritative state-store proof exists. + +### Failure taxonomy to freeze now + +Use stable operator-facing reason names. These are contract values, not implementation comments: + +- `unknown_backend` +- `ambiguous_backend` +- `unknown_session` +- `session_already_owned` +- `missing_active_parent` +- `stale_linkage` +- `missing_internal_session_id` +- `unsupported_platform_or_posture` +- `world_boundary_unavailable` +- `policy_disallow` +- `owner_unreachable` +- `fork_not_supported` + +No command in this slice should return a vague "not available" when one of the exact cases above is true. + +## Step 0: Scope Challenge + +### 0A. Minimum honest diff + +The minimum honest implementation is: + +1. add public `AgentAction::{Start, Resume, Fork, Stop}` plus exact arg structs in [`cli.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/cli.rs), +2. add one shared orchestration control module under `crates/shell/src/execution/agent_runtime/` that owns: + - shared host bootstrap and shutdown extraction from `async_repl.rs`, + - exact public-control session resolution, + - resume and fork extension shaping, + - owner-loop bootstrap, + - private per-session stop transport, + - public result rendering inputs, +3. teach the REPL owner path to register the same private owner transport so public `stop` can target REPL-owned sessions too, +4. add a hidden helper-owner entrypoint so public `start`, `resume`, and `fork` can launch a retained owner loop without keeping the invoking CLI process alive, +5. add strict tests for selectors, owner-loop launch, stop routing, resume, fork, and unsupported posture. + +Anything smaller leaves the public surface fake. + +### 0B. Complexity check + +This slice trips the file-count smell, but it does not justify splitting the feature. + +Expected production files: + +1. [crates/shell/src/execution/cli.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/cli.rs) +2. [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs) +3. [crates/shell/src/execution/agent_runtime/mod.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/mod.rs) +4. `crates/shell/src/execution/agent_runtime/control.rs` (new) +5. [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) +6. [crates/shell/src/execution/agent_runtime/orchestration_session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/orchestration_session.rs) +7. [crates/shell/src/execution/agent_runtime/session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs) +8. [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) + +Expected tests and docs: + +1. `crates/shell/tests/agent_public_control_surface_v1.rs` (new) +2. [crates/shell/tests/agent_successor_contract_ahcsitc0.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_successor_contract_ahcsitc0.rs) +3. [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md) +4. [llm-last-mile/README.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/README.md) + +Why the scope still stands: + +1. there is only one new real abstraction, the control module plus owner loop, +2. splitting `start|stop` and `resume|fork` into separate PRs duplicates the owner model and selector contract, +3. not solving ownership now means the public surface ships with a known lie. + +### 0C. Search and completeness check + +Search-before-building result, in practical terms: + +- **[Layer 1]** reuse exact backend selection and capability gates from [`validator.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/validator.rs), +- **[Layer 1]** reuse authoritative parent and participant persistence from [`state_store.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs), +- **[Layer 1]** reuse host runtime startup and shutdown machinery from [`async_repl.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs), +- **[Layer 1]** reuse UAA resume wire shaping exactly, +- **[Layer 1]** reuse the already-supported UAA fork selector grammar instead of inventing a Substrate-only fork shape, +- **[EUREKA]** the blocker is not "missing start/stop code." The blocker is that retained control ownership is still process-local, +- **[EUREKA]** the smallest complete version is not a daemon. It is one per-session owner loop with one private stop transport plus extracted shared lifecycle logic. + +### 0D. Distribution and runtime contract check + +No new downloadable artifact type is introduced. + +The distribution requirement here is behavioral truth: + +1. public verbs must exist in CLI help, +2. they must emit exact stable JSON, +3. they must not promise root world-session start, +4. they must not imply prompt submission exists, +5. they must keep live ownership real after the launching command exits. + +### 0E. What already exists + +Sub-problem reuse is locked: + +| Sub-problem | Existing code | Reuse rule | +| --- | --- | --- | +| exact backend targeting | [`validate_exact_backend_selection(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/validator.rs:246) | Reuse. No new selector logic. | +| parent session lookup by exact id | [`load_session(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:371) | Reuse as the read-side base, then layer a strict control resolver on top. | +| retained owner lifecycle | [`RuntimeOrchestrationContext`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:1601), [`PreparedAgentRuntime`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:1627), [`AsyncReplAgentRuntime`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:1654) | Extract, do not clone. | +| exact resume extension wire shape | [`build_session_resume_extension(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:3776) | Reuse for `resume`. | +| authoritative stop path | [`shutdown_host_orchestrator_runtime(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:4090) | Reuse through the owner loop only. | +| strict control-surface posture | [`run_doctor(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:138), toolbox commands in [`agents_cmd.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:74) | Preserve. Public control verbs join this strict family. | + +### 0F. NOT in scope + +- redesigning `substrate -c` +- adding any prompt-taking public caller surface +- default-agent routing +- member-level public selectors +- root world-session start for world-scoped backends +- using toolbox as a mutation plane +- a general daemonized control plane or long-lived agent-hub service +- macOS/Lima parity for world-sensitive control semantics +- storage flag-day renames away from `active_session_handle_id` +- broader session-history or list-sessions product work + +## Architecture Review + +### Locked architecture decisions + +1. Add one new shared control module, `crates/shell/src/execution/agent_runtime/control.rs`. +2. That module owns public lifecycle orchestration, exact session resolution, owner-loop bootstrap, private stop transport, and public result rendering inputs. +3. `async_repl.rs` stops being the sole owner of host bootstrap and shutdown logic and instead consumes the shared control module. +4. Public `start`, `resume`, and `fork` spawn a hidden owner helper or attach to the REPL owner path, then return only after authoritative readiness is visible in the state store. +5. Public `stop` connects to the private owner transport for the exact `orchestration_session_id` and asks the live owner to stop authoritatively. +6. Root public session creation is host-orchestrator only. World-sensitive posture matters only when existing live sessions already own authoritative world/member state. +7. `resume` stays in the existing parent orchestration session. +8. `fork` creates a fresh parent orchestration session and links provenance through participant lineage, not a brand-new lineage schema. + +### Architecture findings resolved in-plan + +**Issue 1. The current runtime owner dies with the calling process.** + +That is the blocker. A public lifecycle command that returns immediately without transferring control ownership is fake productization. The plan fixes that with a per-session owner loop. + +**Issue 2. Public `stop` cannot be implemented honestly by mutating JSON.** + +The retained cancel handle lives in memory. The public stop path must hit the process that owns it. That is why the plan introduces a private owner transport and routes `stop` through the owner. + +**Issue 3. `start` cannot mean "any backend".** + +The parent public handle is an orchestration session, and the current runtime architecture makes the orchestrator the parent authority. A world member is a subordinate participant, not a valid v1 root target. + +**Issue 4. `resume` and `fork` are close, but not the same thing.** + +`resume` rebinds the existing orchestration session to a new live participant in the same parent row. `fork` creates a new parent session, new live participant, and a lineage link back to the source active participant. If those semantics blur, operators lose track of which session id they should stop later. + +**Issue 5. Public control verbs belong in the strict family, not the status family.** + +This slice lands after `PLAN-18`. Read-side inspection may degrade. The control side still fails closed. Any implementation that lets `start|resume|fork|stop` "pick the only likely session" is wrong. + +### Hidden owner-helper contract + +V1 needs one private owner-helper entrypoint. That entrypoint is implementation-private but behaviorally frozen. + +Required contract: + +1. it is not documented in public help, +2. it can be reached only by internal command dispatch from `substrate agent start|resume|fork`, +3. it receives a fully resolved execution plan, not fuzzy CLI inputs, +4. it owns the live cancel handle, event stream task, completion observer, and readiness transition, +5. it writes authoritative state transitions into the existing session store, +6. it registers the same private stop transport that REPL-owned sessions register. + +Required modes: + +- `start` +- `resume` +- `fork` + +Required inputs after public resolution: + +- `orchestration_session_id` +- `backend_id` +- `workspace_root` +- `selector_mode` as `start|resume|fork` +- `resume_or_fork_source_uaa_session_id` when needed +- any validated world posture metadata when the action is allowed to proceed + +The helper must not perform fuzzy selection. By the time it starts, selection is done. + +### Private owner transport contract + +There is exactly one v1 owner mutation request: `stop`. + +Required transport rules: + +1. one private endpoint per live `orchestration_session_id`, +2. derived from Substrate-owned runtime state plus exact session id, +3. platform-native transport is allowed, but the addressing contract is session-exact, +4. no global listener or shared multi-session broker is introduced, +5. no toolbox reuse, +6. no PID-only signaling contract. + +Required request and response shape: + +```json +{"version":1,"action":"stop"} +``` + +Response outcomes: + +- `accepted` +- `already_terminal` +- `owner_unreachable` +- `protocol_error` + +Public `substrate agent stop` does not report success on `accepted` alone. It must wait until the parent orchestration session becomes `stopped` or `invalidated`. + +### Parent-session state machine + +Parent-session state transitions for this slice: + +```text +allocating -> active -> stopping -> stopped +allocating -> failed +active -> invalidated +stopping -> invalidated +``` + +Rules: + +1. public `start`, `resume`, and `fork` return success only after `active`, +2. public `stop` returns success only after `stopped` or `invalidated`, +3. if the owner process dies before readiness, the parent session becomes `failed` or `invalidated`, never silent, +4. an active parent must point at exactly one active participant for control operations, +5. `fork` creates a new parent session row. It never mutates the source parent into the fork target. + +### Architecture ASCII diagrams + +### Public command to owner flow + +```text +PUBLIC CLI +========== +substrate agent start|resume|fork|stop + | + +--> start --backend + | | + | +--> validate_exact_backend_selection(host, backend_id) + | +--> allocate orchestration_session_id + | +--> spawn hidden owner helper + | +--> wait for authoritative parent active + active participant + surfaced uaa_session_id + | +--> emit public result + | + +--> resume --session + | | + | +--> resolve exact parent session + | +--> reject if already owner-live + | +--> read active participant internal.uaa_session_id + | +--> spawn hidden owner helper in resume mode + | +--> wait for same parent session to rebind active participant + | +--> emit public result + | + +--> fork --session + | | + | +--> resolve exact source session + | +--> read active participant internal.uaa_session_id + | +--> allocate new orchestration_session_id + | +--> spawn hidden owner helper in fork mode + | +--> wait for new parent session active + | +--> emit public result with source_orchestration_session_id + | + +--> stop --session + | + +--> resolve exact live session + +--> derive exact owner transport from orchestration_session_id + +--> owner transport: stop + +--> wait for terminal parent state + +--> emit public result +``` + +### Owner-model split + +```text +LIVE ORCHESTRATION OWNERSHIP +============================ + +interactive REPL session + | + +--> shared control module + +--> retained cancel/event/completion handles + +--> private owner transport for stop + +public start/resume/fork helper + | + +--> shared control module + +--> retained cancel/event/completion handles + +--> private owner transport for stop + +public stop + | + +--> exact orchestration_session_id + +--> exact owner transport + +--> authoritative shutdown +``` + +### Resume vs fork lineage + +```text +RESUME +====== +existing orchestration_session_id = sess_A +existing active participant = ash_old +existing internal.uaa_session_id = uaa_123 + +resume(sess_A) + | + +--> launch new attached owner with session.resume(id = uaa_123) + +--> create successor participant ash_new + +--> ash_new.resumed_from_participant_id = ash_old + +--> parent session stays sess_A + +--> active participant moves to ash_new + +FORK +==== +source orchestration_session_id = sess_A +source active participant = ash_old +source internal.uaa_session_id = uaa_123 + +fork(sess_A) + | + +--> allocate new orchestration_session_id = sess_B + +--> launch new attached owner with session.fork(id = uaa_123) + +--> create new active participant ash_fork + +--> ash_fork.resumed_from_participant_id = ash_old + +--> parent session is new sess_B + +--> source sess_A remains unchanged +``` + +## Code Quality Review + +### Findings resolved in-plan + +1. Do not copy host bootstrap, state persistence, or shutdown logic out of `async_repl.rs` into `agents_cmd.rs`. That creates two lifecycle implementations that drift. +2. Do not implement separate ad hoc validators for `resume`, `fork`, and `stop`. Add one strict resolver in the control module and make each verb layer only its extra action-specific checks. +3. Do not let public command handlers know the shape of UAA extension payloads directly. Add one helper for `resume`, one helper for `fork`, and pin the mutual exclusion rule in tests. +4. Do not conflate "public selector" and "debugging field". `participant_id` may be surfaced. It must never become an accepted target. +5. Do not hard-code PID signaling as the control-plane contract. PID liveness remains a validation signal, not the stop transport. +6. Keep compatibility alias reads exactly where they are today, at serde and read boundaries. Public control naming moves forward without a storage flag day. + +### Required code comments and diagrams + +Add or update nearby ASCII comments in these places if code lands there: + +1. `control.rs`, owner loop and stop transport flow +2. `async_repl.rs`, if the REPL path starts registering the shared owner transport +3. `state_store.rs`, if a new public-control resolver introduces non-obvious linkage validation + +Stale diagrams are worse than no diagrams. If a nearby runtime-ownership diagram becomes wrong, update it in the same change. + +## Test Review + +### Test framework detection + +This repo is Rust-first and the relevant review surface is `cargo test`. + +Primary suites for this slice: + +1. `crates/shell/tests/agent_public_control_surface_v1.rs` (new integration suite) +2. [crates/shell/tests/agent_successor_contract_ahcsitc0.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_successor_contract_ahcsitc0.rs) +3. targeted unit tests in [`crates/shell/src/execution/agent_runtime/state_store.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) +4. targeted unit tests in [`crates/shell/src/repl/async_repl.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) when extraction leaves local invariants behind + +### Code path coverage + +```text +CODE PATH COVERAGE +=========================== +[+] crates/shell/src/execution/cli.rs + | + ├── AgentAction::Start / Resume / Fork / Stop parse + │ ├── [GAP] exact required selector flags + │ ├── [GAP] no prompt argument accepted + │ └── [GAP] hidden owner-helper entrypoint stays hidden + +[+] crates/shell/src/execution/agent_runtime/control.rs + | + ├── start path + │ ├── [GAP] exact host backend match launches owner helper + │ ├── [GAP] world-scoped backend root start rejects explicitly + │ ├── [GAP] policy-disallowed backend rejects explicitly + │ └── [GAP] readiness succeeds only after parent active + active participant + surfaced uaa_session_id + | + ├── resume path + │ ├── [GAP] exact session id resolves active participant + internal.uaa_session_id + │ ├── [GAP] already-owned live session rejects with session_already_owned + │ ├── [GAP] missing internal.uaa_session_id rejects explicitly + │ └── [GAP] resumed participant rebinds the same orchestration_session_id + | + ├── fork path + │ ├── [GAP] exact session id resolves active participant + internal.uaa_session_id + │ ├── [GAP] fork extension uses exact {selector,id} object + │ ├── [GAP] new orchestration_session_id is allocated + │ └── [GAP] new participant links back to source active participant + | + ├── stop path + │ ├── [GAP] exact session id resolves live owner transport + │ ├── [GAP] owner transport stop request triggers authoritative shutdown + │ ├── [GAP] owner unreachable rejects explicitly + │ └── [GAP] command waits for terminal state, not just request acceptance + | + └── selector rejection + ├── [GAP] participant_id is not accepted as session selector + ├── [GAP] internal.uaa_session_id is not accepted as session selector + └── [GAP] fuzzy or latest lookup is never attempted + +[+] crates/shell/src/repl/async_repl.rs + | + ├── REPL owner path + │ ├── [★★★ TESTED] host startup and shutdown already work + │ ├── [GAP] REPL-owned live session registers the shared owner transport + │ └── [GAP] stop over owner transport shuts down authoritatively without prompt-path regressions + | + ├── targeted follow-up submit + │ └── [★★★ TESTED] resume extension wiring already exists and must stay green + | + └── member runtime ownership + └── [★★★ TESTED] Linux retained-member reuse and shutdown remain the source of world-sensitive truth + +[+] crates/shell/src/execution/agent_runtime/state_store.rs + | + ├── exact public-control session resolution + │ ├── [GAP] missing active parent rejects + │ ├── [GAP] stale linkage rejects + │ ├── [GAP] inactive owner rejects when required + │ └── [GAP] exact session-id lookup does not consult trace history + | + └── canonical naming + └── [★★★ TESTED] orchestration_session_id remains the parent authority + +--------------------------------- +COVERAGE TARGET +- every public verb has a direct success-path test +- every public verb has exact selector rejection tests +- owner-model honesty is proven, not assumed +- existing REPL targeted-turn tests remain green +--------------------------------- +``` + +### Operator flow coverage + +```text +OPERATOR FLOW COVERAGE +=========================== +[+] Operator runs `substrate agent start --backend --json` + | + ├── [GAP] receives orchestration_session_id only after authoritative readiness + └── [GAP] receives explicit rejection for world-scoped root backend + +[+] Operator runs `substrate agent resume --session --json` + | + ├── [GAP] exact orphaned session resumes successfully + ├── [GAP] already-owned live session rejects with clear reason + └── [GAP] missing internal session id rejects with clear reason + +[+] Operator runs `substrate agent fork --session --json` + | + ├── [GAP] returns a new orchestration_session_id + ├── [GAP] preserves source session unchanged + └── [GAP] rejects if backend/runtime does not support session.fork.v1 + +[+] Operator runs `substrate agent stop --session --json` + | + ├── [GAP] owner transport accepts stop and session becomes terminal + ├── [GAP] unreachable owner rejects explicitly + └── [GAP] no silent state-store mutation occurs + +[+] Operator tries non-canonical handles + | + ├── [GAP] participant_id is rejected + └── [GAP] internal.uaa_session_id is rejected + +[+] Operator targets a world-sensitive live session on non-Linux + | + └── [GAP] returns unsupported_platform_or_posture, not a fake partial success +``` + +### Required tests to add or extend + +1. Add a public start integration test proving exact host backend startup returns only after authoritative session activation and surfaced session-handle persistence. +2. Add a public start rejection test proving a world-scoped backend is not a valid v1 root selector. +3. Add a public resume integration test proving an exact historical session with persisted `internal.uaa_session_id` rebinds the same `orchestration_session_id` to a new active participant. +4. Add a public resume rejection test for `session_already_owned`. +5. Add a public fork integration test proving a new orchestration session is created and source-session lineage is preserved through `resumed_from_participant_id`. +6. Add a public fork rejection test for missing fork capability or unsupported backend. +7. Add a public stop integration test proving the owner transport drives authoritative shutdown and the command waits for a terminal parent-session state. +8. Add selector rejection tests proving `participant_id`, `session_handle_id`, and `internal.uaa_session_id` are not accepted as public selectors. +9. Extend REPL tests to prove REPL-owned live sessions register the shared owner transport and remain stoppable through the same private owner-plane contract. +10. Keep all targeted-turn resume tests green. That wire shape is shared truth now. + +### QA-facing test artifact + +During implementation, write a QA-facing artifact to: + +```text +~/.gstack/projects//-feat-session-centric-state-store-eng-review-test-plan-.md +``` + +Required contents: + +1. public host start flow, +2. public resume flow for an orphaned session, +3. public fork flow producing a new session id, +4. public stop flow against a live owner, +5. selector rejection flows for wrong handle types, +6. non-Linux world-sensitive rejection flow. + +This artifact is for `/qa` and `/qa-only`. Keep it operator-journey oriented, not implementation-oriented. + +### Regression rule for this slice + +These tests are mandatory: + +1. existing REPL targeted-turn resume coverage stays green, +2. strict doctor and toolbox behavior stays green, +3. public `start` never reports success before authoritative readiness, +4. public `stop` never marks success from a dead owner transport, +5. `resume` and `fork` never accept both selector extensions at once, +6. no public command accepts `internal.uaa_session_id` as input. + +## Failure Modes Registry + +| Failure mode | Test required | Error handling exists | Operator sees clear result | Critical gap before this slice lands | +| --- | --- | --- | --- | --- | +| public `start` returns before ownership is established | yes | no | no | yes | +| public root start accepts a world-scoped backend and creates an unusable session | yes | no | no | yes | +| `resume` targets a still-owned live session and creates split brain | yes | no | no | yes | +| `resume` or `fork` silently uses `participant_id` or `uaa_session_id` as fallback selector | yes | no | no | yes | +| `stop` rewrites state store directly while retained control is still live | yes | no | no | yes | +| `stop` cannot reach the live owner and still reports success | yes | no | no | yes | +| `fork` reuses the source orchestration session id instead of creating a new one | yes | no | no | yes | +| world-sensitive live session appears stoppable or resumable on unsupported platform | yes | partial today via Linux-only runtime behavior | no | yes | +| REPL-owned live session is not reachable by the public stop path | yes | no | no | yes | +| canonical naming regresses and output surfaces `session_handle_id` as the public handle | yes | partial today in comments only | no | yes | + +Critical-gap rule for this plan: + +No public control action is allowed to be both state-changing and unowned. If the command cannot prove who owns the retained control plane, it must fail closed. + +## Performance Review + +Performance is not the main risk here, but the owner model can still get sloppy if the plan overbuilds. + +### Findings resolved in-plan + +1. Public control resolution must read exact session state from the authoritative store, not rescan trace history or broad inventory multiple times. +2. Readiness wait loops must poll bounded state transitions with backoff, not spin. +3. The owner transport is per-session and human-paced. It does not need a global listener registry or background reconciler. +4. Reusing the shared lifecycle module is cheaper and safer than cloning runtime launch paths into two executors. +5. World-sensitive rejection must happen before any expensive ready or rebind path when platform posture is unsupported. + +### Performance posture + +- no new trace scan is acceptable on public control paths, +- no new global daemon or watchdog is needed, +- no background polling loop beyond bounded readiness waits is needed, +- state-store lookups remain exact and small, +- correctness and explicit failure win over shaving a few milliseconds from human-paced commands. + +## DX Guardrails + +This is a developer and operator surface even though it is backend-heavy. + +Required operator experience: + +1. every command names either the exact `backend_id` or exact `orchestration_session_id` it acted on, +2. no command tells the operator to use `participant_id` or `uaa_session_id`, +3. stop failures distinguish `owner_unreachable` from `unknown_session`, +4. host-root-only start is explicit, not buried behind a confusing runtime-realizability error, +5. world-sensitive unsupported posture is explicit and named, +6. JSON output is stable and scriptable from day one. + +## Worktree Parallelization Strategy + +This plan has real parallelization opportunities once the owner model is frozen. + +### Dependency table + +| Step | Modules touched | Depends on | +| --- | --- | --- | +| Freeze public verb contract, owner model, output schema, and error taxonomy | `crates/shell/src/execution/`, `crates/shell/src/repl/`, repo docs | — | +| Shared control module extraction and REPL integration | `crates/shell/src/repl/`, `crates/shell/src/execution/agent_runtime/` | Freeze public verb contract, owner model, output schema, and error taxonomy | +| Public CLI and command-handler wiring | `crates/shell/src/execution/` | Freeze public verb contract, owner model, output schema, and error taxonomy | +| Exact session resolver and canonical naming guardrails | `crates/shell/src/execution/agent_runtime/` | Freeze public verb contract, owner model, output schema, and error taxonomy | +| Integration tests and repo-truth closeout | `crates/shell/tests/`, repo docs | Shared control module extraction and REPL integration, Public CLI and command-handler wiring, Exact session resolver and canonical naming guardrails | + +### Parallel lanes + +- Lane A: shared control module extraction and REPL integration + - sequential inside the lane because these steps share `crates/shell/src/repl/` and the new control module +- Lane B: public CLI and command-handler wiring + - sequential inside the lane because these steps share `crates/shell/src/execution/cli.rs` and `agents_cmd.rs` +- Lane C: exact session resolver and canonical naming guardrails + - sequential inside the lane because these steps share `crates/shell/src/execution/agent_runtime/` +- Lane D: integration tests and repo-truth closeout + - starts after A, B, and C merge + +### Execution order + +1. Freeze the public contract and owner model. +2. Launch Lane A, Lane B, and Lane C in parallel worktrees. +3. Merge A, B, and C. +4. Run Lane D for public control integration tests, REPL non-regression, and doc closeout. + +### Conflict flags + +- Lane A and Lane C both touch `agent_runtime/` semantics. Freeze the resolver contract before parallel work starts or there will be two equally plausible but incompatible owner models. +- Lane B must not invent public output fields independently. The output schema is frozen up front. +- Lane D owns the new `agent_public_control_surface_v1.rs` suite. Keep that ownership there to avoid test churn across implementation lanes. +- Docs move last. Updating the gap matrix before the owner model and tests land will drift repo truth again. + +### Parallelization verdict + +Four workstreams, three parallel implementation lanes, one final integration lane. + +## Implementation Sequence + +### Step 1. Freeze the public control contract + +Files: + +1. [crates/shell/src/execution/cli.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/cli.rs) +2. [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs) +3. [crates/shell/src/execution/agent_runtime/mod.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/mod.rs) +4. [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md) for language freeze only if needed during implementation + +Deliver: + +1. add public `Start`, `Resume`, `Fork`, and `Stop` actions plus exact args, +2. freeze root-start as host-only in v1, +3. freeze the public JSON result fields and exact error taxonomy, +4. freeze the owner-loop and owner-transport contract, +5. freeze the rule that `resume` stays in the same parent session and `fork` creates a new parent session. + +Done means the surface contract is explicit before lifecycle extraction begins. + +### Step 2. Extract the shared control module + +Files: + +1. `crates/shell/src/execution/agent_runtime/control.rs` +2. [crates/shell/src/execution/agent_runtime/mod.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/mod.rs) +3. [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) + +Deliver: + +1. extract shared host lifecycle pieces out of `async_repl.rs`, +2. move resume-extension shaping into the shared control module, +3. add fork-extension shaping with the exact UAA selector object, +4. add the private per-session owner loop and owner transport, +5. keep REPL startup and targeted-turn behavior functionally unchanged apart from reusing the shared control module. + +Done means there is one lifecycle implementation, not one REPL copy and one CLI copy. + +### Step 3. Add the strict public-control resolver + +Files: + +1. [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) +2. [crates/shell/src/execution/agent_runtime/orchestration_session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/orchestration_session.rs) +3. [crates/shell/src/execution/agent_runtime/session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs) +4. `crates/shell/src/execution/agent_runtime/control.rs` + +Deliver: + +1. add an exact public-control session resolver layered on authoritative state-store reads, +2. validate: + - parent session exists, + - active participant exists when the action requires it, + - owner linkage is alive when the action requires a live owner, + - `internal.uaa_session_id` exists for resume and fork, + - world-sensitive posture is Linux-supported when the resolved live session requires it, +3. reject all non-canonical handle forms explicitly, +4. keep status degradation untouched. + +Done means public actions can target exact session ids without consulting trace fallback or guessing. + +### Step 4. Wire public `start` + +Files: + +1. [crates/shell/src/execution/cli.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/cli.rs) +2. [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs) +3. `crates/shell/src/execution/agent_runtime/control.rs` +4. [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) if helper launch shares REPL owner-state machinery + +Deliver: + +1. validate exact host backend selection, +2. allocate a new `orchestration_session_id`, +3. launch the hidden owner helper, +4. wait for authoritative readiness, not process spawn alone, +5. emit the stable public result. + +Done means `substrate agent start --backend ` is real and does not lose ownership when the command exits. + +### Step 5. Wire public `resume` and `fork` + +Files: + +1. [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs) +2. `crates/shell/src/execution/agent_runtime/control.rs` +3. [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) + +Deliver: + +1. `resume` + - resolve exact session, + - reject if already owner-live, + - read exact `internal.uaa_session_id`, + - launch hidden owner helper with `agent_api.session.resume.v1`, + - rebind the same parent session to a new active participant, +2. `fork` + - resolve exact source session, + - read exact `internal.uaa_session_id`, + - allocate a new `orchestration_session_id`, + - launch hidden owner helper with `agent_api.session.fork.v1`, + - create a new parent session and participant lineage back to the source active participant, +3. enforce resume and fork mutual exclusion in request shaping and tests. + +Done means `resume` and `fork` both exist, mean different things, and remain exact. + +### Step 6. Wire public `stop` + +Files: + +1. [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs) +2. `crates/shell/src/execution/agent_runtime/control.rs` +3. [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) + +Deliver: + +1. derive the exact owner transport from `orchestration_session_id`, +2. teach REPL-owned sessions and helper-owned sessions to register that transport, +3. accept exactly one v1 owner request, `stop`, +4. route `stop` to the authoritative shutdown path, +5. wait for terminal state and emit stable result. + +Done means `stop` is real for any live session that advertises the owner plane and never fakes success through local JSON mutation. + +### Step 7. Freeze the contract with tests and repo-truth docs + +Files: + +1. `crates/shell/tests/agent_public_control_surface_v1.rs` +2. [crates/shell/tests/agent_successor_contract_ahcsitc0.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_successor_contract_ahcsitc0.rs) +3. [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md) +4. [llm-last-mile/README.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/README.md) + +Deliver: + +1. add success and rejection coverage for all four public verbs, +2. keep strict doctor and toolbox behavior green, +3. keep REPL targeted-turn resume behavior green, +4. update the gap matrix to say the public control family is landed for host-orchestrator sessions, with Linux-first world-sensitive stop and reuse posture, +5. update the planning index to include `PLAN-19`. + +Done means the repo says what the runtime actually does. + +## Recommended Verification Commands + +```bash +cargo fmt --all -- --check +cargo clippy --workspace --all-targets -- -D warnings +cargo test -p shell --test agent_public_control_surface_v1 -- --nocapture +cargo test -p shell --test agent_successor_contract_ahcsitc0 -- --nocapture +cargo test -p shell async_repl -- --nocapture +``` + +Manual spot checks after tests are green: + +```bash +substrate agent start --backend --json +substrate agent resume --session --json +substrate agent fork --session --json +substrate agent stop --session --json +substrate agent status --json +substrate agent doctor --json +substrate agent toolbox env --json +``` + +## Definition of Done + +1. `substrate agent` publicly exposes `start`, `resume`, `fork`, and `stop`. +2. `start` accepts only exact host-scoped `backend_id` selectors. +3. `resume`, `fork`, and `stop` accept only exact `orchestration_session_id` selectors. +4. no public command accepts or emits `internal.uaa_session_id` as a selector. +5. `start`, `resume`, and `fork` return only after authoritative readiness is visible in the store. +6. `stop` routes through the live owner and reaches a terminal parent-session state. +7. `resume` rebinds the same parent session. `fork` creates a new parent session. +8. REPL-owned live sessions expose the same private owner plane for public stop. +9. strict doctor and toolbox behavior remain fail closed. +10. root world-session start is rejected explicitly. +11. world-sensitive control remains Linux-first and fail closed elsewhere. +12. repo-truth docs reflect landed behavior. + +## Deferred Work + +- prompt-taking public caller surfaces +- `substrate -c` redesign +- member-level public control selectors +- root world-session start +- world-sensitive control parity on macOS/Lima and Windows/WSL +- any general daemonized agent-hub service +- toolbox mutation tools +- broader session-history and list-sessions product work + +## Completion Summary + +- Step 0: Scope Challenge, scope accepted as-is after correcting the owner-model gap +- Architecture Review: 5 issues found, all resolved in-plan +- Code Quality Review: 6 issues found, all resolved in-plan +- Test Review: diagrams produced, 10 concrete regression gaps identified +- Performance Review: 5 issues found, all resolved in-plan +- NOT in scope: written +- What already exists: written +- TODOS.md updates: 0 items proposed, no `TODOS.md` exists in this repo today +- Failure modes: 10 critical gaps flagged until the public owner plane and regression wall land +- Outside voice: skipped for this document pass +- Parallelization: 4 workstreams, 3 parallel implementation lanes, 1 final integration lane +- Lake Score: 10/10 recommendations chose the complete option + +## Decision Audit Trail + +| # | Phase | Decision | Classification | Principle | Rationale | Rejected | +| --- | --- | --- | --- | --- | --- | --- | +| 1 | Step 0 | Treat retained ownership as the primary blocker, not missing clap variants | Mechanical | Pragmatic | Public verbs are fake if ownership dies on CLI exit | wiring CLI actions first and hoping lifecycle details sort themselves out | +| 2 | Architecture | Add one shared control module instead of cloning `async_repl.rs` logic into `agents_cmd.rs` | Mechanical | DRY | One lifecycle truth is cheaper than two drifting ones | duplicated lifecycle code | +| 3 | Architecture | Add one private owner transport and keep toolbox introspection-only | Mechanical | Explicit over clever | Stop needs a real owner-plane transport, but toolbox must not become that plane | repurposing toolbox or direct JSON mutation | +| 4 | Architecture | Keep root public session creation host-only in v1 | Mechanical | Boring by default | The parent orchestration model is host-rooted today, and world members are subordinate participants | implicit world-root session creation | +| 5 | Architecture | Keep `resume` in the same parent session and `fork` in a new parent session | Mechanical | Explicit over clever | Operators need different handles and different semantics for those verbs | fuzzy "resume or fork, same end result" behavior | +| 6 | Code Quality | Keep `participant_id` visible but never public-selectable | Mechanical | Minimal diff | It is useful for debugging and lineage, not as an operator target | hiding participant lineage completely or accepting it as input | +| 7 | Code Quality | Use exact UAA selector extension objects for both resume and fork | Mechanical | Layer 1 | The dependency already supports the grammar, so Substrate should not invent a new one | Substrate-only extension shapes | +| 8 | Test Review | Require a new dedicated public-control integration suite | Mechanical | Systems over heroes | This surface deserves its own regression wall and subprocess coverage | burying all new cases inside existing successor tests | +| 9 | Test Review | Make owner-unreachable stop a mandatory failure-path test | Mechanical | Completeness | The stop path is the easiest place to fake success if the transport is wrong | assuming owner registration is obviously fine | +| 10 | Parallelization | Freeze the owner model before parallel implementation lanes | Mechanical | Pragmatic | The file split is clean, but the ownership contract is shared and central | parallel edits before the model is explicit | diff --git a/llm-last-mile/PLAN-20.md b/llm-last-mile/PLAN-20.md new file mode 100644 index 000000000..be67d2a92 --- /dev/null +++ b/llm-last-mile/PLAN-20.md @@ -0,0 +1,1044 @@ +# PLAN-20: Public Non-Interactive Agent Caller Surface With Exact Backend Targeting, Streamed Turns, And Session Posture Truth + +Source SOW: [20-public-non-interactive-agent-caller-surface.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/20-public-non-interactive-agent-caller-surface.md) +Gap matrix anchor: [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md) +Adjacent landed slices: [PLAN-18.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-18.md), [PLAN-19.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-19.md) +Branch: `feat/session-centric-state-store` +Base branch: `main` +Plan type: public non-interactive prompt-taking caller surface for exact orchestration sessions and exact retained backends +Review posture: unified execution plan, tightened to `/plan-eng-review` structure and explicit implementation sequencing +Status: execution-ready planning pass on 2026-05-08 +Outside voice: not used for this document generation + +## Objective + +Expose one honest non-interactive public prompt-taking surface under `substrate agent` without changing `substrate -c`, without fuzzy routing, and without pretending retained ownership is something it is not. + +This slice does five things and only five things: + +1. makes `substrate agent start` mean `new session + first real prompt`, +2. adds `substrate agent turn` as the exact follow-up prompt surface for an existing session/backend pair, +3. makes `substrate agent reattach` the canonical lifecycle-recovery verb for owner-loop recovery, +4. streams real caller-visible output for `start` and `turn` in both human mode and `--json` mode, +5. reports authoritative completion-time `session_posture` so callers can tell whether the session stayed live, became detached-but-reattachable, or is terminal. + +This slice does not redesign REPL grammar. It does not widen default-agent routing. It does not reinterpret plain stdin or `substrate -c` as agent prompting. It does not introduce a general daemon. + +## Plan Summary + +The runtime capability is mostly already here. The product contract is not. + +Today the repo already has: + +- public `substrate agent start|resume|fork|stop` wiring in [`AgentAction`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/cli.rs:493), +- exact session-only public control resolution in [`resolve_public_control_target(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:589), +- exact REPL targeted follow-up turns in [`dispatch_targeted_follow_up_turn(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:3180), +- host follow-up submission in [`submit_host_targeted_turn(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:4451), +- Linux world follow-up submission in [`submit_world_targeted_turn(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:4559), [`submit_member_turn_stream(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs:1472), and [`submit_turn(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs:249), +- hidden owner-helper launch machinery in [`launch_hidden_owner_helper(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:231). + +What is still wrong: + +1. public `start` does not take a real prompt source, +2. public `resume` is actually lifecycle reattachment, not conversational resume, +3. there is no public `turn` command, +4. the public resolver is session-centric only and cannot select the exact retained backend within a multi-backend orchestration session, +5. the retained-runtime bootstrap prompt still exists internally and must never leak into the public caller contract, +6. the public JSON contract is still a one-shot terminal object rather than a stream. + +The minimum honest fix is: + +1. add explicit prompt-source arguments, +2. add one exact `(orchestration_session_id, backend_id)` public turn resolver, +3. extract the existing targeted-turn execution seam so public CLI can reuse it, +4. bridge helper-owned output back to the invoking CLI in real time, +5. surface completion-time `session_posture` explicitly, +6. rename the canonical lifecycle verb to `reattach`. + +That is the whole game. + +## Step 0: Scope Challenge + +### 0A. What already exists + +| Sub-problem | Existing code | Decision | +| --- | --- | --- | +| Public agent namespace | [`crates/shell/src/execution/cli.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/cli.rs) | Reuse `substrate agent`. Do not create a second top-level caller family. | +| Public narrow control plane | [`run_start(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:291), [`run_resume(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:310), [`run_fork(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:328), [`run_stop(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:346) | Preserve the namespace, but change `start` semantics, add `turn`, and make `reattach` the canonical recovery term. | +| Hidden owner-helper launch seam | [`launch_hidden_owner_helper(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:231), [`HiddenOwnerHelperLaunchPlan`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/control.rs:139) | Reuse the helper. Do not replace it with a new daemon. Extend it to support foreground streaming for prompt-taking calls. | +| Exact public session selector guardrails | [`resolve_public_control_target(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:589) | Reuse for `reattach`, `fork`, and `stop`. Add a separate backend-aware resolver for `turn`. | +| Exact backend selection for root start | [`validate_exact_backend_selection(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/validator.rs:246) | Reuse as the only root `start --backend` selector. | +| Host targeted follow-up execution | [`submit_host_targeted_turn(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:4451) | Reuse the exact run-control and event-stream behavior. Do not fork a second host follow-up implementation. | +| Linux world targeted follow-up execution | [`submit_world_targeted_turn(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:4559), [`submit_member_turn_stream(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs:1472), [`submit_turn(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs:249) | Reuse the same exact Linux-first retained-member follow-up seam. | +| REPL startup prompt semantics | [`dispatch_targeted_follow_up_turn(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:3887), [`InitialExecPromptPlan::Replace`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:1697) | Preserve the rule that the first explicit prompt must ride the launch-time initial exec prompt instead of a hidden bootstrap prompt. | +| Public stop owner plane | [`private_stop_transport_path(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/control.rs), [`request_private_stop(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/control.rs) via [`run_stop(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:346) | Preserve. `stop` stays non-prompt-taking and exact. | +| Status degradation vs strict control split | [`PLAN-18.md`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-18.md) and current repo behavior | Preserve. Prompt-taking surfaces stay strict and fail closed. | + +### 0B. Minimum honest diff + +The minimum honest implementation is: + +1. extend CLI args so `start` accepts an exact prompt source and add a new `turn` action, +2. introduce one shared prompt-source parser/loader with exact pre-launch failures, +3. introduce one authoritative backend-aware public turn resolver for exact `(orchestration_session_id, backend_id)` lookup, +4. extract the existing targeted-turn submit and stream-translation seam out of REPL-only call sites so public CLI can reuse it, +5. extend the hidden owner-helper contract to support a foreground stream bridge for prompt-taking modes, +6. rename the canonical recovery verb to `reattach`, +7. classify and render `session_posture`, +8. update tests and repo-truth docs. + +Anything smaller leaves the prompt-taking surface fake. + +### 0C. Complexity check + +This slice trips the file-count smell, but it does not justify splitting the work. The surface crosses CLI parsing, exact resolution, retained ownership, streaming, and tests. If any one of those is deferred, the public contract lies. + +Expected production files: + +1. [crates/shell/src/execution/cli.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/cli.rs) +2. [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs) +3. [crates/shell/src/execution/agent_runtime/control.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/control.rs) +4. [crates/shell/src/execution/agent_runtime/mod.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/mod.rs) +5. [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) +6. [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) +7. possibly [`crates/shell/src/execution/agent_runtime/session.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs) for explicit posture helpers or metadata comments only + +Expected tests and docs: + +1. [crates/shell/tests/agent_public_control_surface_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_public_control_surface_v1.rs) +2. [crates/shell/tests/agent_successor_contract_ahcsitc0.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_successor_contract_ahcsitc0.rs) +3. [crates/shell/tests/repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs) +4. [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md) +5. [llm-last-mile/README.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/README.md) + +This is engineered enough. Not overbuilt, not underbuilt. + +### 0D. Search and completeness check + +Search-before-building result, in practical terms: + +- **[Layer 1]** reuse exact backend validation from [`validator.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/validator.rs), +- **[Layer 1]** reuse session-only strict control resolution from [`state_store.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs), +- **[Layer 1]** reuse exact host and world follow-up submission seams from [`async_repl.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs), +- **[Layer 1]** reuse the existing hidden owner-helper instead of inventing a daemon, +- **[EUREKA]** the missing thin slice is not runtime capability. It is public contract honesty: prompt source parsing, exact `(session, backend)` targeting, and foreground output streaming while ownership may outlive the caller, +- **[EUREKA]** the smallest complete version is one explicit bridge between the current public owner plane and the already-landed REPL targeted-turn seam. + +### 0E. Distribution check + +No new artifact type is introduced. + +The real distribution requirement is contract truth: + +1. CLI help must show the new prompt-taking grammar, +2. docs must say `reattach`, not conversational `resume`, +3. JSON streaming must be stable enough for scripts from day one, +4. operator surfaces must preserve the `-c` shell contract by construction. + +### 0F. NOT in scope + +- redesigning `substrate -c` or plain pipe mode +- default-agent routing +- fuzzy backend or latest-session selection +- public world-root `start` +- member-level public selectors outside `orchestration_session_id` +- toolbox mutation features +- a general multi-session daemon or hub service +- macOS/Lima parity for world-sensitive follow-up turns +- Windows/WSL parity for world-sensitive follow-up turns +- redesigning `fork` +- broader session-history or list-sessions product work + +## Frozen Execution Contract + +If implementation wants to do something else, revise this plan first. + +### Non-negotiable invariants + +1. `substrate -c`, `--command`, pipe mode, and plain stdin remain shell-wrap semantics. This slice must not reinterpret them as agent prompting. +2. Every prompt-taking call names an exact `backend_id`. +3. Every follow-up prompt-taking call names both exact `orchestration_session_id` and exact `backend_id`. +4. No public prompt-taking or lifecycle-recovery command accepts `participant_id`, `active_session_handle_id`, `session_handle_id`, or `internal.uaa_session_id` as input. +5. Prompt validation happens before any runtime launch, runtime recovery, or stream bridge setup. +6. Public `start` and `turn` stream caller-visible output while the turn is running. No full-output buffering is allowed. +7. Public `reattach`, `fork`, and `stop` remain non-prompt-taking operational surfaces. +8. Public prompt-taking must never inject `runtime_bootstrap_prompt` text as user-visible agent input. +9. Follow-up turn submission remains single-writer. If the exact session/backend slot is already live-owned, the turn routes through that owner or fails closed. No competing owner loops. +10. Host and world follow-up paths stay distinct. Linux world-sensitive prompt submission still goes through the exact retained-member seam. Non-Linux world-sensitive follow-up still fails closed. +11. Completion-time `session_posture` is explicit and authoritative only at command completion time, not a promise about the indefinite future. +12. `fork` remains the existing lifecycle surface. This slice does not redesign its meaning. + +### Public command contract + +```text +substrate agent start --backend (--prompt | --prompt-file | --prompt-file -) [--json] +substrate agent turn --session --backend (--prompt | --prompt-file | --prompt-file -) [--json] +substrate agent reattach --session [--json] +substrate agent fork --session [--json] +substrate agent stop --session [--json] +``` + +Compatibility rule: + +1. `reattach` is the canonical documented verb. +2. `resume` may remain as a hidden deprecated alias for one compatibility window only. +3. All docs, help text, human output, and JSON `action` values use `reattach`, never `resume`. +4. Future conversational resume work is still allowed because `resume` is not part of the canonical contract after this slice. + +### Exact selector rules + +| Command | Required selector | Resolution rule | Explicitly rejected | +| --- | --- | --- | --- | +| `start` | `--backend ` | exact host-scoped backend match via [`validate_exact_backend_selection(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/validator.rs:246) | agent id, default backend, world-only root start, fuzzy or prefix matching | +| `turn` | `--session ` + `--backend ` | exact authoritative session plus exact authoritative backend slot within that session | latest session, latest backend, agent id, `participant_id`, `internal.uaa_session_id`, fuzzy matching | +| `reattach` | `--session ` | exact existing session via strict control resolver | prompt args, backend-only resolution, non-canonical handles | +| `fork` | `--session ` | exact existing session via strict control resolver | prompt args, backend-only resolution, non-canonical handles | +| `stop` | `--session ` | exact existing session via strict control resolver plus exact owner transport | direct JSON mutation, PID-only heuristics, non-canonical handles | + +### Prompt-source contract + +Exactly one prompt source is required for `start` and `turn`: + +1. `--prompt ` +2. `--prompt-file ` +3. `--prompt-file -` meaning read prompt text from stdin + +Rules: + +1. `--prompt` and `--prompt-file` are mutually exclusive. +2. `--prompt-file -` consumes stdin once and only for prompt input. It does not change `-c` semantics. +3. Effective prompt text is trimmed for emptiness checks only. The original prompt bytes are preserved for actual agent input. +4. Empty or all-whitespace effective prompt is a hard pre-launch failure. +5. Missing file, unreadable file, invalid UTF-8, or stdin read failure is a hard pre-launch failure. +6. No missing or malformed prompt source may fall back to bootstrap text. + +### Session posture contract + +The completion-time `session_posture` enum is: + +- `active` +- `detached_reattachable` +- `terminal` + +Meanings: + +- `active` + - at command completion time, a live retained owner is still attached to the exact target session/backend + - later `turn`, `reattach`, `fork`, and `stop` remain subject to fresh reachability and posture checks +- `detached_reattachable` + - at command completion time, no live retained owner is attached, but authoritative recovery metadata still exists + - later `reattach`, `fork`, and possibly `turn` may succeed if exact metadata and posture remain valid +- `terminal` + - at command completion time, the relevant session is already stopped, invalidated, failed, or completed beyond follow-up + +Classification rule for `start`: + +1. if the backend supports no-turn retained startup and the owner remains attached after the inaugural prompt, report `active`, +2. if the inaugural prompt completed but no live owner remains and authoritative recovery metadata still exists, report `detached_reattachable`, +3. otherwise report `terminal`. + +### Streaming contract + +Human mode: + +1. stream user-visible agent output to stdout while the turn is running, +2. reserve stderr for pre-stream validation, setup, and bridge errors, +3. print one terminal summary line after completion that includes: + - `action` + - `orchestration_session_id` + - `backend_id` + - `participant_id` when available + - turn outcome + - `session_posture` + +JSON mode: + +1. stdout is line-delimited JSON only, +2. every record has `version: 1`, +3. every record has a stable string `kind`, +4. after the first stdout JSON record is emitted, all later runtime failures must be represented in-stream, not only on stderr. + +Minimum NDJSON shape: + +```json +{"version":1,"kind":"accepted","action":"start|turn","orchestration_session_id":"...","backend_id":"...","participant_id":"...","scope":"host|world"} +{"version":1,"kind":"event","event_kind":"message|status|stderr|tool","data":{...}} +{"version":1,"kind":"warning","message":"..."} +{"version":1,"kind":"completed","action":"start|turn","orchestration_session_id":"...","backend_id":"...","participant_id":"...","turn_outcome":"success|nonzero_exit|cancelled","session_posture":"active|detached_reattachable|terminal","state":"active|stopped|invalidated|failed","warnings":[]} +{"version":1,"kind":"failed","terminal":true,"stage":"setup|bridge|runtime","error_code":"...","message":"..."} +``` + +### Failure taxonomy to freeze now + +Use these stable operator-facing error codes: + +- `missing_backend` +- `unknown_backend` +- `ambiguous_backend` +- `missing_prompt_source` +- `malformed_prompt_source` +- `empty_prompt` +- `unknown_session` +- `noncanonical_session_selector` +- `backend_not_in_session` +- `ambiguous_backend_slot` +- `missing_active_parent` +- `stale_linkage` +- `missing_internal_session_id` +- `session_already_owned` +- `owner_unreachable` +- `unsupported_platform_or_posture` +- `policy_disallow` +- `stream_bridge_failed` +- `runtime_start_failed` + +No prompt-taking or recovery failure may fall back to shell execution, implicit REPL grammar, synthetic prompt text, or best-effort session guessing. + +## Architecture Review + +### Locked architecture decisions + +1. Keep all public caller and lifecycle verbs under `substrate agent`. +2. Make `reattach` the canonical recovery verb. `resume` is, at most, a hidden deprecated alias. +3. Add one shared prompt-source and stream-bridge implementation in [`control.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/control.rs). Do not invent a second lifecycle module. +4. Add one new authoritative backend-aware resolver for public `turn`. Keep session-only lifecycle resolution strict and separate. +5. Public `start` and `turn` are foreground one-shot surfaces. They may still rely on a helper-owned retained session internally, but the caller remains attached until the submitted turn reaches a terminal outcome. +6. Public `start` does not create an empty live session. A successful `start` always corresponds to one real user prompt submission. +7. Public `turn` never means root-start. Root prompt-taking belongs only to `start`. +8. `fork` stays the current lifecycle surface and is outside the prompt-taking redesign, except for keeping selector and status behavior coherent. +9. `stop` stays exact and non-prompt-taking. +10. `session_posture` is part of the public contract for `start` and `turn`. + +### Architecture findings resolved in-plan + +1. The current public `start` is still a detached lifecycle command, not a caller surface. This plan fixes that by turning `start` into a foreground prompt-taking surface with a stream bridge. +2. Session-only control resolution is not enough for public `turn`. A session can contain multiple retained backends. This plan fixes that with a backend-aware authoritative resolver. +3. Duplicating targeted-turn submission logic would be a mess. Host and world follow-up submission already exist and already know how to persist events and final state. This plan extracts and reuses them instead of rebuilding the same flow inside `agents_cmd.rs`. +4. Public streaming and retained ownership pull in opposite directions unless the helper contract is explicit. This plan freezes one bridge contract between them instead of pretending stdout inheritance solves it. +5. `reattach` and `turn` are not the same thing. `reattach` restores ownership. `turn` submits work. Keep them separate. + +### Hidden owner-helper contract + +The existing hidden helper is reused, not replaced. + +Required modes after this slice: + +- `start_prompted` +- `turn_prompted` +- `reattach` +- `fork` + +Required helper inputs after public CLI resolution: + +- exact action mode +- exact `orchestration_session_id` +- exact `backend_id` +- exact prompt source material for prompt-taking modes +- source `internal.uaa_session_id` for recovery or fork modes when required +- exact world posture metadata when the target is world-sensitive +- one one-shot bridge descriptor for prompt-taking modes + +The helper does not guess. By the time it starts, selection is done. + +### Stream bridge contract + +The helper needs one prompt-taking bridge for `start` and `turn`. + +Rules: + +1. the bridge is per-invocation, not global, +2. the bridge is used only for prompt-taking modes, +3. the bridge carries ordered `accepted|event|warning|completed|failed` frames, +4. the bridge closes when the one submitted turn reaches a terminal result, +5. `reattach`, `fork`, and `stop` do not use the prompt-taking bridge, +6. bridge setup failure after command acceptance emits an in-stream `kind="failed"` terminal record in JSON mode, +7. helper-local retained ownership may outlive the bridge. + +### Backend-aware public turn resolver + +Add one new resolver in [`state_store.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) for: + +```text +(orchestration_session_id, backend_id) -> exact authoritative target +``` + +Required behavior: + +1. load the exact parent orchestration session, +2. reject non-canonical session selectors using the same existing guardrails, +3. find the exact backend slot within that session, +4. reject if zero authoritative candidates exist, +5. reject if more than one authoritative candidate exists, +6. distinguish host-orchestrator target vs world-member target explicitly, +7. report whether the target is currently live-owned, detached-but-reattachable, or terminal, +8. preserve Linux-first failure posture for world-sensitive follow-up. + +### Session posture and execution split + +```text +START +===== +exact backend_id + | + +--> validate prompt source before runtime launch + +--> if backend supports no-turn retained startup: + | +--> establish retained owner first + | +--> submit inaugural real prompt through shared follow-up seam + | +--> classify final posture as active or detached_reattachable + | + +--> else: + +--> run inaugural prompt directly through helper-owned foreground path + +--> persist surfaced session metadata if any + +--> classify final posture as detached_reattachable or terminal + +TURN +==== +exact orchestration_session_id + exact backend_id + | + +--> resolve exact backend slot + +--> if live-owned: + | +--> submit through exact retained owner path + | + +--> else if detached but recoverable: + | +--> recover ownership first + | +--> submit through the same shared follow-up seam + | + +--> else: + +--> fail closed + +REATTACH +======== +exact orchestration_session_id + | + +--> recover retained ownership only + +--> no prompt submission +``` + +### Architecture ASCII diagrams + +#### Public prompt-taking flow + +```text +PUBLIC PROMPT CALLER +==================== +substrate agent start|turn + | + +--> parse exact prompt source + +--> validate prompt source before any runtime action + +--> resolve exact target + | | + | +--> start: exact host backend_id + | +--> turn: exact orchestration_session_id + exact backend_id + | + +--> spawn/helper or reuse owner plane + +--> open one-shot stream bridge + +--> emit accepted + +--> stream translated runtime events + +--> persist authoritative state updates + +--> classify completion-time session_posture + +--> emit completed or failed +``` + +#### Exact turn resolution + +```text +TURN TARGET RESOLUTION +====================== +input: orchestration_session_id + backend_id + | + +--> load authoritative parent session + +--> reject non-canonical selectors + +--> search exact backend candidates within session + | + +--> 0 candidates -> backend_not_in_session + +--> >1 candidates -> ambiguous_backend_slot + +--> 1 host candidate -> host follow-up target + +--> 1 world candidate -> world follow-up target + | + +--> check live owner / recovery metadata / posture + +--> return exact target enum or fail closed +``` + +#### Human and JSON bridge split + +```text +HELPER-OWNED TURN +================= +helper runtime events + | + +--> bridge frame + | + +--> human renderer -> stdout text stream + terminal summary + | + +--> json renderer -> NDJSON accepted/event/completed/failed + +stderr stays reserved for pre-stream validation and bridge setup failures. +``` + +## Code Quality Review + +### Findings resolved in-plan + +1. Do not duplicate prompt-source parsing between `start` and `turn`. Add one shared loader. +2. Do not duplicate targeted-turn submission logic between REPL and public CLI. Extract shared helpers from [`async_repl.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs). +3. Do not overload `resolve_public_control_target(...)` with backend-aware turn semantics. Keep lifecycle and prompt-taking resolution separate. +4. Do not let `agents_cmd.rs` hand-assemble UAA extension payloads. Keep those helpers in [`control.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/control.rs). +5. Do not let `resume` and `reattach` both appear as first-class documented verbs. Pick one public term. This plan picks `reattach`. +6. Do not leak `internal.uaa_session_id` into public success output while adding `session_posture`. +7. Do not invent a second event translation format for public CLI streaming. Reuse translated runtime event shapes wherever possible and wrap them in the frozen top-level NDJSON envelope. + +### Required code comments and diagrams + +Add or update nearby ASCII comments in these places if code lands there: + +1. [`control.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/control.rs), prompt-taking bridge and helper-owned turn flow +2. [`state_store.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs), exact `(session, backend)` public turn resolver +3. [`async_repl.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs), if shared submit helpers move and nearby ownership diagrams become stale + +Stale diagrams are worse than no diagrams. Update them in the same change. + +## Test Review + +### Test framework detection + +This repo is Rust-first and the relevant review surface is `cargo test`. + +Primary suites for this slice: + +1. [crates/shell/tests/agent_public_control_surface_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_public_control_surface_v1.rs) +2. [crates/shell/tests/agent_successor_contract_ahcsitc0.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_successor_contract_ahcsitc0.rs) +3. [crates/shell/tests/repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs) +4. targeted unit tests in [`crates/shell/src/execution/agent_runtime/state_store.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) +5. targeted unit tests in [`crates/shell/src/execution/agent_runtime/control.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/control.rs) + +### Code path coverage + +```text +CODE PATH COVERAGE +=========================== +[+] crates/shell/src/execution/cli.rs + | + ├── AgentStartArgs + │ ├── [GAP] exact one-of prompt source parsing + │ ├── [GAP] missing backend rejected at parse or validation boundary + │ └── [GAP] --json remains orthogonal to prompt source + | + ├── AgentTurnArgs (new) + │ ├── [GAP] requires exact --session and exact --backend + │ ├── [GAP] requires exact one-of prompt source + │ └── [GAP] root turn form is impossible + | + └── AgentReattachArgs / alias posture + ├── [GAP] canonical `reattach` parses + └── [GAP] deprecated `resume` alias behavior pinned if retained + +[+] crates/shell/src/execution/agent_runtime/control.rs + | + ├── prompt source loader + │ ├── [GAP] --prompt literal success + │ ├── [GAP] --prompt-file path success + │ ├── [GAP] --prompt-file - stdin success + │ ├── [GAP] missing source rejection + │ ├── [GAP] malformed source rejection + │ └── [GAP] empty effective prompt rejection + | + ├── start_prompted path + │ ├── [GAP] exact host backend root success with stream bridge + │ ├── [GAP] world-only backend root rejection + │ ├── [GAP] no synthetic bootstrap prompt used as public input + │ ├── [GAP] completion emits session_posture + │ └── [GAP] accepted/event/completed NDJSON contract + | + ├── turn_prompted path + │ ├── [GAP] exact live host follow-up success + │ ├── [GAP] exact live Linux world follow-up success + │ ├── [GAP] detached recoverable target can recover then submit + │ ├── [GAP] backend-not-in-session rejection + │ └── [GAP] ambiguous backend slot rejection + | + └── stream bridge + ├── [GAP] human-mode streaming surfaces output incrementally + ├── [GAP] JSON mode emits line-delimited records only + └── [GAP] bridge failure after stream start emits terminal kind="failed" + +[+] crates/shell/src/execution/agent_runtime/state_store.rs + | + ├── session-only public control resolver + │ └── [★★★ TESTED] strict non-canonical handle rejection already exists and must stay green + | + └── backend-aware public turn resolver (new) + ├── [GAP] exact backend host slot resolution + ├── [GAP] exact backend world slot resolution + ├── [GAP] missing backend slot rejection + ├── [GAP] ambiguous backend slot rejection + ├── [GAP] Linux-first posture rejection for world-sensitive follow-up + └── [GAP] detached-but-recoverable posture classification + +[+] crates/shell/src/repl/async_repl.rs + | + ├── dispatch_targeted_follow_up_turn(...) + │ ├── [★★★ TESTED] exact host/world targeted routing already exists + │ └── [GAP] extracted shared helper leaves REPL behavior unchanged + | + ├── submit_host_targeted_turn(...) + │ └── [★★★ TESTED] host follow-up streaming/persistence remains source truth + | + └── submit_world_targeted_turn(...) + └── [★★★ TESTED] Linux world retained-member submit remains source truth + +[+] crates/world-agent/src/service.rs + member_runtime.rs + | + └── world member submit seam + └── [★★★ TESTED] exact retained member turn stream remains unchanged functionally + +--------------------------------- +COVERAGE TARGET +- every new public caller path has success + rejection tests +- every streaming mode has at least one assertion on incremental behavior +- every session_posture outcome is explicitly covered +- REPL targeted-turn behavior stays green +--------------------------------- +``` + +### Operator flow coverage + +```text +OPERATOR FLOW COVERAGE +=========================== +[+] Operator runs `substrate agent start --backend cli:codex --prompt "hi" --json` + | + ├── [GAP] sees accepted early + ├── [GAP] sees streamed events during execution + └── [GAP] sees completed with session_posture + +[+] Operator runs `substrate agent start --backend cli:codex --prompt-file -` + | + ├── [GAP] stdin prompt is read once + └── [GAP] empty stdin rejects before runtime launch + +[+] Operator runs `substrate agent turn --session --backend cli:codex --prompt "next"` + | + ├── [GAP] exact live host follow-up succeeds + ├── [GAP] exact detached host follow-up can recover if authoritative metadata exists + └── [GAP] wrong backend in right session rejects explicitly + +[+] Operator runs `substrate agent turn --session --backend cli:claude_code --prompt "next"` + | + ├── [GAP] exact Linux world follow-up succeeds through retained member seam when posture is valid + └── [GAP] non-Linux world-sensitive follow-up rejects explicitly + +[+] Operator runs `substrate agent reattach --session --json` + | + ├── [GAP] canonical verb succeeds against recoverable session + └── [GAP] already-owned session rejects with session_already_owned + +[+] Operator targets non-canonical handles + | + ├── [GAP] participant_id rejected + ├── [GAP] internal.uaa_session_id rejected + └── [GAP] active_session_handle_id rejected + +[+] Operator runs `substrate -c "echo hi"` + | + └── [GAP] remains shell-wrap mode. No prompt-taking regression. +``` + +### Required tests to add or extend + +1. Add CLI parse coverage proving `start` requires exactly one prompt source. +2. Add CLI parse coverage proving `turn` requires `--session`, `--backend`, and exactly one prompt source. +3. Add prompt-source tests for `--prompt-file -`, unreadable files, invalid UTF-8, and empty effective prompt. +4. Extend the public control integration suite so `start` asserts accepted/event/completed streaming behavior rather than only one terminal JSON object. +5. Add a public `turn` integration test for an exact live host-owned backend. +6. Add a public `turn` integration test for an exact live Linux world-owned backend when authoritative posture is valid. +7. Add a public `turn` rejection test for `backend_not_in_session`. +8. Add a public `turn` rejection test for `ambiguous_backend_slot`. +9. Add a public `turn` rejection test for non-Linux world-sensitive follow-up. +10. Add a public `reattach` integration test for exact recoverable session rebind. +11. If the deprecated `resume` alias stays temporarily, add one compatibility test that the alias routes to `reattach` semantics while emitted `action` is still `reattach`. +12. Add `session_posture` classification tests covering `active`, `detached_reattachable`, and `terminal`. +13. Add a regression test proving no public prompt-taking path ever injects `runtime_bootstrap_prompt`. +14. Keep current `fork` and `stop` tests green. +15. Keep current REPL targeted-turn tests green. +16. Keep `substrate -c` shell-wrap behavior green. + +### QA-facing test artifact + +During implementation, write a QA-facing artifact to: + +```text +~/.gstack/projects//-feat-session-centric-state-store-eng-review-test-plan-.md +``` + +Required contents: + +1. root `start` happy path with `--prompt` +2. root `start` stdin path with `--prompt-file -` +3. exact host `turn` +4. exact Linux world `turn` +5. detached recoverable `turn` +6. `reattach` +7. wrong-handle rejection flows +8. `-c` non-regression spot check + +Keep it operator-journey oriented. + +## Failure Modes Registry + +| Failure mode | Test required | Error handling exists | Operator sees clear result | Critical gap before this slice lands | +| --- | --- | --- | --- | --- | +| public `start` launches ownership but never submits the real prompt | yes | no | no | yes | +| public `start` accepts missing or empty prompt input and falls back to bootstrap text | yes | no | no | yes | +| public `start` buffers output and only prints at the end | yes | no | no | yes | +| public `turn` routes by session alone and hits the wrong retained backend | yes | no | no | yes | +| public `turn` silently creates a competing retained owner for a live slot | yes | no | no | yes | +| public `turn` on non-Linux world-sensitive posture pretends to succeed | yes | partial today via fail-closed REPL posture | no | yes | +| `reattach` remains labeled as conversational resume in public output | yes | no | no | yes | +| bridge failure after stream start only appears on stderr in JSON mode | yes | no | no | yes | +| `session_posture` is guessed from helper spawn instead of command completion truth | yes | no | no | yes | +| `-c` begins accepting prompt-like input accidentally | yes | no | no | yes | +| public success output leaks `internal.uaa_session_id` | yes | partial today via omission in current control JSON | no | yes | +| prompt-taking path regresses current `fork` or `stop` behavior | yes | partial today via existing tests | no | yes | + +Critical-gap rule for this plan: + +No prompt-taking command is allowed to both mutate agent state and hide the actual turn outcome. If the caller cannot see the real submitted turn, the surface is incomplete. + +## Performance Review + +Performance is not the main risk, but streaming and helper bridging can still get sloppy. + +### Findings resolved in-plan + +1. Prompt-taking paths must stream incrementally. No full-output buffering into memory. +2. Bridge readers must use bounded frame processing. A single giant buffer of accumulated output is not acceptable. +3. The new backend-aware resolver must read exact authoritative session state, not scan trace history or broad inventory repeatedly. +4. Readiness and terminal waits must stay bounded with polling or backoff. No tight spin loops. +5. This slice does not justify a global bridge broker or daemon. One invocation-scoped bridge per prompt-taking call is enough. +6. World-sensitive rejection should happen before any expensive recovery attempt on unsupported platforms. + +### Performance posture + +- no new global listener +- no broad trace scans +- no full-response buffering +- bounded readiness waits only +- correctness and explicit failure beat a few milliseconds of latency + +## DX Guardrails + +This is a developer/operator surface even though the implementation is runtime-heavy. + +Required experience: + +1. the caller always knows whether they are starting, turning, reattaching, forking, or stopping, +2. the caller always sees the exact `backend_id` and exact `orchestration_session_id`, +3. JSON output is stable and line-delimited from day one, +4. errors distinguish `backend_not_in_session` from `unknown_session`, +5. errors distinguish `owner_unreachable` from `session_already_owned`, +6. help text and docs say `reattach`, not `resume`, +7. `-c` remains shell behavior by construction, not by best effort. + +## Worktree Parallelization Strategy + +This plan has real parallelization opportunities once the output contract is frozen. + +### Dependency table + +| Step | Modules touched | Depends on | +| --- | --- | --- | +| Freeze public caller grammar, prompt-source contract, `reattach` naming, NDJSON envelope, and failure taxonomy | `crates/shell/src/execution/`, repo docs | — | +| CLI and command-handler wiring for `start`, `turn`, and `reattach` | `crates/shell/src/execution/` | Freeze public caller grammar, prompt-source contract, `reattach` naming, NDJSON envelope, and failure taxonomy | +| Backend-aware public turn resolver and posture classification | `crates/shell/src/execution/agent_runtime/` | Freeze public caller grammar, prompt-source contract, `reattach` naming, NDJSON envelope, and failure taxonomy | +| Shared prompt-submit extraction and stream bridge | `crates/shell/src/repl/`, `crates/shell/src/execution/agent_runtime/` | Freeze public caller grammar, prompt-source contract, `reattach` naming, NDJSON envelope, and failure taxonomy | +| Integration tests and repo-truth closeout | `crates/shell/tests/`, repo docs | CLI and command-handler wiring, Backend-aware public turn resolver and posture classification, Shared prompt-submit extraction and stream bridge | + +### Parallel lanes + +- Lane A: CLI and command-handler wiring + - sequential inside the lane because these steps share `cli.rs` and `agents_cmd.rs` +- Lane B: backend-aware public turn resolver and posture classification + - sequential inside the lane because these steps share `state_store.rs` and nearby runtime metadata +- Lane C: shared prompt-submit extraction and stream bridge + - sequential inside the lane because these steps share `control.rs` and `async_repl.rs` +- Lane D: integration tests and repo-truth closeout + - starts only after A, B, and C merge + +### Execution order + +1. Freeze the caller grammar, naming, NDJSON contract, and failure taxonomy. +2. Launch Lanes A, B, and C in parallel worktrees. +3. Merge A, B, and C. +4. Run Lane D for integration tests, REPL non-regressions, and doc closeout. + +### Conflict flags + +- Lane A and Lane C both depend on the exact NDJSON envelope. Freeze it first or they will drift. +- Lane B and Lane C both influence `session_posture`. Lane B owns authoritative classification inputs. Lane C must consume, not reinvent, them. +- Lane D owns the prompt-taking integration assertions. Keep that ownership there to avoid test churn across implementation lanes. +- Docs move last. Updating the gap matrix before tests and stream behavior land will drift repo truth again. + +### Parallelization verdict + +Four workstreams, three parallel implementation lanes, one final integration lane. + +## Implementation Sequence + +### Step 1. Freeze the public caller contract + +Files: + +1. [crates/shell/src/execution/cli.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/cli.rs) +2. [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs) +3. [crates/shell/src/execution/agent_runtime/control.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/control.rs) +4. [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md) for wording freeze if needed during implementation + +Deliver: + +1. add explicit prompt-source arg shapes for `start`, +2. add `turn`, +3. make `reattach` canonical, +4. freeze NDJSON envelope and human summary contract, +5. freeze `session_posture` enum and meanings, +6. freeze failure taxonomy. + +Done means nobody is still guessing what the caller sees. + +### Step 2. Add shared prompt-source loading + +Files: + +1. [crates/shell/src/execution/agent_runtime/control.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/control.rs) +2. [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs) + +Deliver: + +1. one shared loader for `--prompt`, `--prompt-file `, and `--prompt-file -`, +2. exact pre-launch error mapping for missing, malformed, and empty prompt input, +3. no bootstrap-text fallback. + +Done means `start` and `turn` cannot disagree about prompt validation. + +### Step 3. Add the exact backend-aware public turn resolver + +Files: + +1. [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) +2. [crates/shell/src/execution/agent_runtime/session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/session.rs) only if posture helpers or metadata comments are needed +3. [crates/shell/src/execution/agent_runtime/orchestration_session.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/orchestration_session.rs) only if parent-state comments need clarification + +Deliver: + +1. exact `(orchestration_session_id, backend_id)` lookup, +2. host vs world target classification, +3. live vs detached vs terminal target posture classification, +4. exact failure codes for missing and ambiguous backend slots, +5. Linux-first fail-closed posture for world-sensitive follow-up. + +Done means public `turn` can target the right runtime without guessing. + +### Step 4. Extract shared prompt-submit helpers from REPL-only paths + +Files: + +1. [crates/shell/src/repl/async_repl.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) +2. [crates/shell/src/execution/agent_runtime/control.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/control.rs) + +Deliver: + +1. extract host follow-up submit behavior from [`submit_host_targeted_turn(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:4451), +2. extract Linux world follow-up submit behavior from [`submit_world_targeted_turn(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:4559), +3. preserve REPL behavior unchanged by calling the shared helpers from existing REPL code, +4. expose a bridge-friendly event sink for public CLI prompt-taking modes. + +Done means there is one prompt-submit truth, not a REPL truth and a public CLI truth. + +### Step 5. Extend the helper with a prompt-taking stream bridge + +Files: + +1. [crates/shell/src/execution/agent_runtime/control.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/control.rs) +2. [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs) + +Deliver: + +1. add prompt-taking helper modes, +2. add one invocation-scoped bridge descriptor to the helper launch plan, +3. translate runtime events into frozen bridge records, +4. close the bridge exactly once per submitted prompt, +5. keep helper ownership alive after caller exit when posture stays `active`. + +Done means the public caller can see the real turn without owning the retained session forever. + +### Step 6. Wire public `start` + +Files: + +1. [crates/shell/src/execution/cli.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/cli.rs) +2. [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs) +3. [crates/shell/src/execution/agent_runtime/control.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/control.rs) + +Deliver: + +1. validate exact host backend and exact prompt source, +2. allocate a new `orchestration_session_id`, +3. launch helper in prompted-start mode, +4. stream output to the caller, +5. classify and emit final `session_posture`, +6. preserve host-only root start rejection for world-only backends. + +Done means `start` is a real prompt-taking caller surface. + +### Step 7. Wire public `turn` + +Files: + +1. [crates/shell/src/execution/cli.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/cli.rs) +2. [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs) +3. [crates/shell/src/execution/agent_runtime/control.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/control.rs) +4. [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) + +Deliver: + +1. validate exact prompt source, +2. resolve exact session/backend target, +3. route through live host follow-up when exact host owner exists, +4. route through live Linux world retained-member follow-up when exact world owner exists, +5. recover ownership first when the target is detached but reattachable, +6. fail closed otherwise, +7. emit streamed records and final `session_posture`. + +Done means public `turn` is exact, streamed, and honest. + +### Step 8. Wire canonical `reattach` and keep `fork` and `stop` coherent + +Files: + +1. [crates/shell/src/execution/cli.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/cli.rs) +2. [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs) + +Deliver: + +1. expose canonical `reattach`, +2. optionally keep hidden deprecated `resume` alias for one compatibility window, +3. keep `fork` unchanged functionally, +4. keep `stop` unchanged functionally, +5. keep emitted JSON `action` strings canonical. + +Done means the public terminology matches the product truth. + +### Step 9. Freeze the contract with tests and repo-truth docs + +Files: + +1. [crates/shell/tests/agent_public_control_surface_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_public_control_surface_v1.rs) +2. [crates/shell/tests/agent_successor_contract_ahcsitc0.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_successor_contract_ahcsitc0.rs) +3. [crates/shell/tests/repl_world_first_routing_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs) +4. [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md) +5. [llm-last-mile/README.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/README.md) + +Deliver: + +1. success and rejection coverage for `start`, `turn`, `reattach`, `fork`, and `stop`, +2. stream-contract coverage, +3. `session_posture` coverage, +4. `-c` non-regression coverage, +5. update repo truth to say the public non-interactive caller surface is landed, +6. update packet index to include `PLAN-20`. + +Done means the repo says what the runtime actually does. + +## Recommended Verification Commands + +```bash +cargo fmt --all -- --check +cargo clippy --workspace --all-targets -- -D warnings +cargo test -p shell --test agent_public_control_surface_v1 -- --nocapture +cargo test -p shell --test agent_successor_contract_ahcsitc0 -- --nocapture +cargo test -p shell --test repl_world_first_routing_v1 -- --nocapture +``` + +If world-sensitive follow-up logic changes materially, also run: + +```bash +cargo test -p world-agent --test member_runtime_world_placement_v1 -- --nocapture +``` + +Required operator-surface checks after tests are green: + +```bash +substrate agent doctor --json +substrate shim doctor --json +substrate world doctor --json +substrate health --json +``` + +Manual spot checks: + +```bash +substrate agent start --backend --prompt "hello" --json +printf 'hello from stdin\n' | substrate agent start --backend --prompt-file - --json +substrate agent turn --session --backend --prompt "next" --json +substrate agent reattach --session --json +substrate agent fork --session --json +substrate agent stop --session --json +substrate agent status --json +substrate -c 'echo shell wrap still works' +``` + +## Definition of Done + +1. `substrate agent start` accepts exact prompt input and submits one real inaugural prompt. +2. `substrate agent turn` exists and requires exact `--session` plus exact `--backend`. +3. `substrate agent reattach` is the canonical recovery surface. +4. public prompt-taking commands stream output in human mode and JSON mode. +5. JSON mode is NDJSON with stable `version` and `kind`. +6. `session_posture` is emitted on successful `start` and `turn`. +7. no prompt-taking path uses synthetic bootstrap text as public input. +8. no public prompt-taking or lifecycle command accepts non-canonical session handles. +9. exact Linux world follow-up still goes through the retained-member seam. +10. non-Linux world-sensitive follow-up still fails closed. +11. `substrate -c` remains shell wrap mode. +12. repo-truth docs reflect the landed caller surface accurately. + +## Deferred Work + +- removing the temporary hidden `resume` alias entirely if one compatibility window is kept +- default-agent routing +- `substrate -c` redesign +- public world-root `start` +- member-level public selectors +- macOS/Lima world-follow-up parity +- Windows/WSL world-follow-up parity +- broader session-history and session-list product work +- any general daemonized agent-hub service + +## Completion Summary + +- Step 0: Scope Challenge, scope accepted as-is after tightening the public contract around exact prompt-taking rather than lifecycle launch alone +- Architecture Review: 5 issues found, all resolved in-plan +- Code Quality Review: 7 issues found, all resolved in-plan +- Test Review: diagrams produced, 16 concrete regression gaps identified +- Performance Review: 6 issues found, all resolved in-plan +- NOT in scope: written +- What already exists: written +- TODOS.md updates: 0 items proposed, no `TODOS.md` exists in this repo today +- Failure modes: 12 critical gaps flagged until streamed prompt-taking and exact backend-aware turn resolution land +- Outside voice: skipped for this document pass +- Parallelization: 4 workstreams, 3 parallel implementation lanes, 1 final integration lane +- Lake Score: 10/10 recommendations chose the complete option + +## Decision Audit Trail + +| # | Phase | Decision | Classification | Principle | Rationale | Rejected | +| --- | --- | --- | --- | --- | --- | --- | +| 1 | Step 0 | Treat public prompt-taking honesty, not just lifecycle launch, as the core blocker | Mechanical | Pragmatic | The runtime already exists. The missing thing is a truthful caller contract | only adding clap flags and leaving helper behavior detached | +| 2 | Architecture | Keep the public surface under `substrate agent` | Mechanical | Minimal diff | The namespace already exists and is the correct home | inventing a second top-level caller family | +| 3 | Architecture | Make `reattach` the canonical recovery verb | Mechanical | Explicit over clever | The current behavior is owner recovery, not conversational resume | keeping `resume` as the public term indefinitely | +| 4 | Architecture | Add a separate backend-aware public `turn` resolver | Mechanical | Explicit over clever | Session-only resolution is insufficient once one session can retain multiple backends | teaching the existing session-only resolver to guess the target | +| 5 | Architecture | Reuse hidden owner-helper with a prompt-taking stream bridge | Mechanical | Boring by default | The helper already owns retained control correctly. It just cannot yet stream prompt results | inventing a new daemon or broker | +| 6 | Code Quality | Extract shared prompt-submit helpers out of REPL-only code | Mechanical | DRY | Host and world follow-up seams already exist and should stay single-source | duplicating host and world submit logic in `agents_cmd.rs` | +| 7 | Code Quality | Keep `fork` out of the prompt-taking redesign | Mechanical | Minimal diff | `fork` already has a stable lifecycle meaning and does not need product-surface churn here | folding `fork` into `turn` or `reattach` | +| 8 | Streaming | Freeze NDJSON accepted/event/completed/failed envelope up front | Mechanical | Systems over heroes | CLI scripts need stable framing from day one | ad hoc JSON per code path | +| 9 | Tests | Make `session_posture` coverage mandatory | Mechanical | Completeness | Without posture assertions, the product still lies about retained state after completion | inferring posture from helper PID or assuming it is always active | +| 10 | Tests | Keep `-c` shell-wrap regression coverage in this slice | Mechanical | Blast radius instinct | This is the easiest accidental product regression while adding stdin prompt support | assuming prompt-source parsing cannot affect shell mode | diff --git a/llm-last-mile/PLAN-21.md b/llm-last-mile/PLAN-21.md new file mode 100644 index 000000000..c4598fd2d --- /dev/null +++ b/llm-last-mile/PLAN-21.md @@ -0,0 +1,653 @@ +# PLAN-21: macOS/Lima Shared-Owner and Member-Runtime Parity + +Source SOW: [21-macos-lima-shared-owner-and-member-runtime-parity.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/21-macos-lima-shared-owner-and-member-runtime-parity.md) +Gap matrix anchor: [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md:171) +Adjacent landed slices: [PLAN-15.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-15.md), [PLAN-17.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-17.md), [PLAN-19.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-19.md) +Branch: `feat/macos-lima-shared-owner-member-runtime-parity` +Base branch: `main` +Plan type: cross-platform orchestration parity, Linux contract extended across the Lima forwarded seam +Review posture: unified execution plan, tightened to `/autoplan` and `/plan-eng-review` rigor +Status: execution-ready planning pass on 2026-05-08 + +## Objective + +Bring macOS/Lima onto the same explicit shared-owner and retained member-runtime contract Linux already uses. + +This slice is complete only when a macOS host can: + +1. open or replace a shared REPL world through the forwarded Lima guest without host-side pre-rejection, +2. lazily launch and retain a world-scoped member runtime through the existing guest `member_dispatch` path, +3. send targeted follow-up turns through `/v1/member_turn/stream`, +4. cancel bootstrap or submitted turns through `/v1/execute/cancel`, +5. preserve authoritative shared-world binding and dispatch fields across backend-level `world-api` seams instead of silently zeroing them. + +This is parity work, not a new orchestration model. + +## Plan Summary + +The repo already has the right Linux contract and most of the host-to-guest plumbing. The gap is narrower than it looks and more structural than it looks. + +The real blockers are: + +1. the macOS host still rejects explicit shared-owner bootstrap before the forwarded guest can answer, +2. shell member-runtime orchestration is still compiled behind Linux-only gates and hard-errors on macOS, +3. the backend seam still drops orchestration-sensitive fields because [`ExecRequest`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-api/src/lib.rs:141) does not carry them and [`MacLimaBackend::convert_exec_request(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/lib.rs:275) hardcodes `shared_world: None` and `member_dispatch: None`. + +The minimum honest implementation is one cohesive slice with four ordered workstreams: + +1. remove the macOS shared-owner bootstrap rejection and prove forwarded `ready.shared_world`, +2. widen backend contracts so `world-api`, `world-agent`, and `world-mac-lima` can preserve shared-owner and member-dispatch semantics, +3. widen shell member-runtime orchestration off Linux-only cfg stubs onto the forwarded guest path, +4. add reproducible Lima orchestration validation and update docs to match reality. + +## Locked Starting State + +### What already exists + +| Sub-problem | Existing code | Decision | +| --- | --- | --- | +| Shared-owner request construction for REPL attach/create and replacement | [`async_repl.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:5596), [`async_repl.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:5816) | Reuse. Do not invent a second ownership shape for macOS. | +| Fail-closed validation of echoed owner proof | [`validate_shared_world_echo(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/repl_persistent_session.rs:308) | Reuse exactly. This remains the authoritative shell-side validator. | +| macOS forwarded persistent-session transport | [`build_ws_and_start_session_frame(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_persistent_session.rs:731) | Reuse the forwarded guest path. Remove only the non-Linux rejection. | +| macOS forwarded member-dispatch request builder | [`world_ops.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs:1369) | Reuse. Do not create a separate macOS request model. | +| Guest-side authoritative binding validation | [`validate_member_dispatch_binding(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs:2565) | Reuse exactly. The guest remains the world-sensitive authority. | +| Guest-side retained follow-up turn validation | [`validate_submit_turn_request(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs:869) | Reuse exactly. No loosening of identity checks. | +| Backend shared-binding surface | [`WorldHandle.shared_binding`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-api/src/lib.rs:131) | Reuse the field, but make Lima populate it when explicit shared-owner mode is active. | +| Existing session cache in Lima backend | [`MacLimaBackend::ensure_session(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/lib.rs:389) | Preserve. Do not regress session reuse. | + +### Exact remaining gap + +1. [`reject_non_linux_shared_owner_request(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/platform_world/mod.rs:61) still blocks the supported macOS path before the guest can prove anything. +2. [`build_ws_and_start_session_frame(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_persistent_session.rs:731) already forwards on macOS, but the guard means explicit shared-owner mode never reaches that transport. +3. [`ensure_member_runtime_ready_for_descriptor(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:4298) and [`submit_world_targeted_turn(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:4684) still hard-error on non-Linux. +4. [`ExecRequest`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-api/src/lib.rs:141) still lacks `shared_world` and `member_dispatch`, so backend-level parity is impossible without widening the contract. +5. [`MacLimaBackend::convert_exec_request(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/lib.rs:275) and [`MacLimaBackend::ensure_session(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/lib.rs:389) still erase exactly the fields Linux relies on. + +### Scope decision + +Proceed as one parity slice. + +Do not split this into a "bootstrap only" PR plus a later "member runtime maybe" PR. That would ship a partial contract and make the docs lie for another cycle. The shell either has real macOS/Lima parity for shared-owner plus retained members, or it does not. + +## Frozen Execution Contract + +If implementation wants to do something else, revise this plan first. + +### Non-negotiable invariants + +1. Linux remains the contract source of truth. +2. Shared-world ownership stays explicit via `SharedWorldOwnerSpec` and authoritative via `SharedWorldBindingSnapshot`. +3. macOS must stop rejecting the supported shared-owner path before guest bootstrap. +4. World-scoped member launch, follow-up, and cancel must fail closed if the forwarded guest contract cannot be established. +5. No host-local fallback is allowed for a world-scoped member runtime. +6. Exact identity checks remain unchanged: `orchestration_session_id`, `orchestrator_participant_id`, `backend_id`, `world_id`, and `world_generation` must still match. +7. Targeted follow-up turns keep using `/v1/member_turn/stream`. +8. Cancel stays guest-owned through `/v1/execute/cancel`. +9. Replacement remains shell-owned and fail-closed through `ReplaceExpectedGeneration`. +10. Backend-level execution and handles must stop silently dropping Linux orchestration fields. + +### Blast radius + +GitNexus marks both core seam types as `CRITICAL` blast radius changes: + +| Symbol | Risk | Why it matters | +| --- | --- | --- | +| [`ExecRequest`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-api/src/lib.rs:141) | `CRITICAL` | Direct callers span `world-mac-lima`, replay, examples, and tests. Any field addition must update every constructor and serializer path. | +| [`WorldHandle`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-api/src/lib.rs:131) | `CRITICAL` | Direct callers span Linux, macOS, Windows stubs, `world-agent`, and one indexed execution flow. Semantic changes to `shared_binding` affect cross-platform world ownership assumptions. | + +Implication: contract widening must land early, stay additive, and keep Linux plus Windows stubs compiling the whole time. + +## Step 0: Scope Challenge + +### 0A. Minimum honest diff + +The minimum honest implementation is: + +1. remove the host-side non-Linux shared-owner rejection for the Lima-backed persistent-session path, +2. make backend-level request and handle types capable of carrying shared-owner and member-dispatch state, +3. propagate those fields end to end through `world-agent` and `world-mac-lima`, +4. widen shell member-runtime orchestration so macOS uses the forwarded guest contract instead of Linux-only stubs, +5. add one reproducible macOS/Lima orchestration smoke harness and update docs. + +Anything smaller lies about parity. Anything larger is scope creep. + +### 0B. Complexity check + +This slice touches more than eight files and crosses shell, backend, and guest boundaries. That is a smell. It is still the right scope because the alternative is shipping a fake feature boundary. + +Expected primary modules: + +1. `crates/shell/src/execution/platform_world/` +2. `crates/shell/src/execution/routing/dispatch/` +3. `crates/shell/src/execution/repl_persistent_session.rs` +4. `crates/shell/src/repl/async_repl.rs` +5. `crates/world-api/src/lib.rs` +6. `crates/world-mac-lima/src/lib.rs` +7. `crates/world-agent/src/service.rs` +8. `scripts/mac/` +9. `docs/WORLD.md` +10. `AGENT_ORCHESTRATION_GAP_MATRIX.md` + +That is above the smell threshold, so the plan must stay boring: + +1. no new orchestration service, +2. no new request shape, +3. no new resume transport, +4. no platform-specific special case beyond enabling Lima to use the same guest contract Linux already uses. + +### 0C. Search and reuse check + +The repo already has the core mechanisms this slice needs: + +- **[Layer 1]** reuse shell-side proof validation in [`validate_shared_world_echo(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/repl_persistent_session.rs:308), +- **[Layer 1]** reuse forwarded guest request construction in [`world_ops.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs:1369), +- **[Layer 1]** reuse guest-side shared binding and member identity validation in [`service.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs:2565) and [`member_runtime.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs:869), +- **[EUREKA]** the missing piece is not transport invention. The missing piece is host-side honesty plus backend seam parity. + +### 0D. TODOS cross-reference + +There is no `TODOS.md` in the repo root today. This plan therefore needs to be explicit about deferrals inside its own `NOT in scope` section instead of assuming a separate backlog artifact will preserve intent. + +### 0E. Completeness check + +This cannot be a happy-path-only parity pass. + +The complete version includes: + +1. attach/create success, +2. replacement success, +3. retained member lazy launch, +4. retained member follow-up reuse, +5. cancel, +6. mismatch and stale-generation rejection, +7. backend contract preservation, +8. Lima-backed smoke validation, +9. docs and gap-matrix truth updates. + +Skipping any one of those saves little implementation time and creates a misleading product claim. Boil the lake. + +### 0F. Distribution check + +No new binary, package, or container artifact is introduced. + +Distribution still matters here: + +1. the repo must ship a reproducible `scripts/mac/orchestration-smoke.sh`, +2. docs must stop claiming non-Linux shared-owner rejection once this lands, +3. validation instructions must show how parity is proven from a real macOS host with Lima provisioned. + +## Architecture Review + +### Architecture thesis + +The shell remains the orchestrator owner. The Linux guest remains the world-sensitive authority. Lima remains a transport and backend adapter, not a second orchestration model. + +### Data flow + +```text +CURRENT +======= +macOS host shell + | + | StartSession { shared_world } + v +reject_non_linux_shared_owner_request() + | + `- fails before guest bootstrap + +world-member launch on macOS + | + v +Linux-only cfg stub + | + `- unsupported hard error + +backend exec path + | + v +ExecRequest / WorldHandle + | + `- silently erase shared_world, member_dispatch, shared_binding + + +TARGET +====== +macOS host shell + | + | StartSession { shared_world } + v +forwarded /v1/stream -> Lima guest world-agent + | + v +ready.shared_world + | + v +validate_shared_world_echo() + | + +--> authoritative world_id + world_generation persisted by shell + | + +--> member_dispatch launch over forwarded guest path + | + +--> /v1/member_turn/stream for targeted turns + | + `--> /v1/execute/cancel for bootstrap and submitted-turn cancel + +backend trait seam + | + v +ExecRequest carries shared_world + member_dispatch +WorldHandle carries authoritative shared_binding +``` + +### Implementation workstreams + +#### Workstream 1: persistent-session shared-owner parity + +Goal: allow macOS REPL shared-owner attach/create and replacement to reach the forwarded guest path, then prove the reply through the existing fail-closed validator. + +Files: + +- [`crates/shell/src/execution/platform_world/mod.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/platform_world/mod.rs:61) +- [`crates/shell/src/execution/routing/dispatch/world_persistent_session.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_persistent_session.rs:731) +- [`crates/shell/src/execution/repl_persistent_session.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/repl_persistent_session.rs:308) + +Required changes: + +1. Replace the blanket non-Linux shared-owner rejection with a capability check that allows the Lima-backed persistent-session path and still rejects unsupported platforms. +2. Keep [`validate_shared_world_echo(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/repl_persistent_session.rs:308) as the only shell-side proof validator. +3. Ensure replacement still uses `ReplaceExpectedGeneration` and still requires generation advancement. +4. Keep failure posture identical to Linux for missing, malformed, mismatched, or inactive proof. + +Exit criteria: + +1. macOS `StartSession { shared_world }` reaches the forwarded guest path. +2. `ready.shared_world` is required and validated on macOS exactly as on Linux. +3. unsupported non-Lima paths still reject explicitly. + +#### Workstream 2: backend contract parity + +Goal: stop losing shared-owner and member-dispatch semantics at the `world-api` boundary. + +Files: + +- [`crates/world-api/src/lib.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-api/src/lib.rs:131) +- [`crates/world-mac-lima/src/lib.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/lib.rs:275) +- [`crates/world-agent/src/service.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs:1230) +- direct callers identified by repo search: replay, macOS example, Windows tests and stubs + +Required changes: + +1. Extend [`ExecRequest`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-api/src/lib.rs:141) with additive optional fields for `shared_world` and `member_dispatch`. +2. Update all `ExecRequest` constructors to preserve existing behavior when those fields are absent. +3. Pass the new fields from `world-agent` into backend execution when the request is orchestration-sensitive. +4. Update [`MacLimaBackend::convert_exec_request(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/lib.rs:275) so the forwarded guest request preserves those fields instead of zeroing them. +5. Update [`MacLimaBackend::ensure_session(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/lib.rs:389) so `WorldHandle.shared_binding` is populated when explicit shared-owner mode is active. +6. Keep Linux and Windows implementations additive and compile-clean even if they do not use the new fields yet. + +Exit criteria: + +1. `ExecRequest` carries optional shared-owner and member-dispatch data. +2. `world-agent` can forward that data through backend exec when relevant. +3. Lima no longer zeros those fields. +4. owner-mode Lima sessions return an authoritative `shared_binding`. + +#### Workstream 3: shell member-runtime parity on macOS + +Goal: make macOS use the same guest-owned member runtime lifecycle Linux already uses. + +Files: + +- [`crates/shell/src/repl/async_repl.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:3274) +- [`crates/shell/src/execution/routing/dispatch/world_ops.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs:1369) + +Required changes: + +1. Widen Linux-only imports and helper access so the macOS build can construct forwarded member-dispatch requests. +2. Replace the current macOS hard-error stubs in [`ensure_member_runtime_ready_for_descriptor(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:4298) and [`submit_world_targeted_turn(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs:4684) with the forwarded guest path. +3. Preserve exact identity requirements for lazy launch, retained follow-up, replacement, and cancel. +4. Keep world-sensitive failures explicit and fail-closed. Unsupported posture must still error, not silently downgrade to host-local ownership. + +Exit criteria: + +1. macOS no longer hits Linux-only hard-error stubs for supported world-scoped orchestration. +2. lazy launch, targeted follow-up turns, and cancel use the guest-owned contract. +3. mismatch and stale-generation paths still fail closed. + +#### Workstream 4: validation and documentation + +Goal: prove the whole seam end to end and update repo truth. + +Files: + +- [`scripts/mac/lima-warm.sh`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/lima-warm.sh) +- `scripts/mac/orchestration-smoke.sh` (new) +- [`scripts/mac/smoke.sh`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/smoke.sh) +- [`docs/WORLD.md`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/WORLD.md:86) +- [`AGENT_ORCHESTRATION_GAP_MATRIX.md`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md:175) + +Required changes: + +1. Add a reproducible orchestration smoke harness for macOS/Lima. +2. Document the new validation workflow and remove stale Linux-only claims where this slice changes reality. +3. Record the parity outcome in the gap matrix. + +Exit criteria: + +1. there is a single commandable macOS/Lima orchestration smoke path, +2. docs no longer claim pre-bootstrap rejection on macOS for the supported path, +3. the gap matrix reflects reality, not intent. + +## Code Quality Review + +### Boring-by-default rules + +1. One request shape. The shell must keep using the existing Linux `member_dispatch` contract. +2. One proof validator. Do not duplicate `ready.shared_world` validation logic in a macOS-specific helper. +3. One backend seam. If `ExecRequest` is widened, every backend caller updates to the same additive type instead of creating a Lima-only side channel. +4. One fallback posture. World-scoped member orchestration either proves the guest contract or fails closed. + +### DRY and abstraction guardrails + +1. Prefer widening existing helpers over adding parallel macOS versions. +2. The platform helper in [`platform_world/mod.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/platform_world/mod.rs:61) should become capability-aware, not disappear into duplicated call-site conditionals. +3. Keep member dispatch request assembly in one place so Linux and macOS cannot drift on field population. +4. Do not introduce a new orchestration service, daemon, or abstract trait tower. This slice is already spending its complexity budget on cross-platform parity. + +### Diagram maintenance + +No nearby ASCII diagrams were identified in the cited seams during this planning pass. If implementation finds any local diagrams in touched files, updating them is part of the same commit, not optional cleanup. + +## Test Review + +Rust test infrastructure is already the source of truth here: targeted `cargo test -p ...` plus workspace coverage, then Lima-backed smoke validation. + +### Code path coverage plan + +```text +CODE PATH COVERAGE PLAN +======================= +[+] Shared-owner persistent-session bootstrap + | + |- attach/create on macOS forwarded path + | |- [GAP -> UNIT] request no longer rejected before guest bootstrap + | `- [GAP -> SMOKE] real Lima guest returns authoritative ready.shared_world + | + `- replacement on macOS forwarded path + |- [GAP -> UNIT] ReplaceExpectedGeneration reply must advance generation + `- [GAP -> SMOKE] stale generation fails closed after rollover + +[+] Backend contract parity + | + |- ExecRequest carries shared_world/member_dispatch + | |- [GAP -> UNIT] world-api round-trip and constructor coverage + | `- [GAP -> UNIT] world-agent passes new fields into backend exec + | + `- WorldHandle.shared_binding preserved on Lima path + |- [GAP -> UNIT] ensure_session populates shared_binding when owner mode active + `- [GAP -> UNIT] no regression for generic non-owner sessions + +[+] Member dispatch / retained runtime + | + |- lazy launch on macOS + | |- [GAP -> UNIT] shell no longer hard-errors on non-Linux cfg path + | `- [GAP -> SMOKE] guest receives typed member_dispatch launch + | + |- targeted follow-up turn + | |- [GAP -> UNIT] submit path routes through member_turn API on macOS + | `- [GAP -> SMOKE] :: prompt reuses retained guest member + | + `- cancel + |- [GAP -> UNIT] bootstrap cancel routes through execute/cancel + `- [GAP -> SMOKE] submitted-turn cancel reaches guest owner + +[+] Fail-closed mismatch handling + | + |- shared-world proof missing/malformed/inactive + | `- [GAP -> UNIT] validator rejects all invalid proof states + | + `- backend_id / world_id / world_generation mismatch + |- [GAP -> UNIT] member dispatch binding rejects mismatches + `- [GAP -> SMOKE] stale generation or wrong backend stays rejected on macOS +``` + +### Required test files and coverage + +| Area | File | Required assertions | +| --- | --- | --- | +| Persistent-session bootstrap | existing tests in [`world_persistent_session.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_persistent_session.rs) and [`repl_persistent_session.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/repl_persistent_session.rs) | forwarded macOS attach/create, replacement generation advancement, invalid proof rejection | +| Shell member-runtime parity | existing tests around [`async_repl.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/repl/async_repl.rs) | no non-Linux hard-error on supported Lima posture, follow-up routing, cancel routing | +| `world-api` contract | tests adjacent to [`crates/world-api/src/lib.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-api/src/lib.rs) | round-trip new fields, additive backward compatibility | +| Lima backend | tests adjacent to [`crates/world-mac-lima/src/lib.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/lib.rs) | `convert_exec_request(...)` preserves fields, `ensure_session(...)` returns shared binding in owner mode | +| Guest validation | existing tests in [`service.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs) and [`member_runtime.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs) | no identity relaxation, no binding relaxation | +| End-to-end macOS proof | `scripts/mac/orchestration-smoke.sh` (new) | attach/create, replacement, launch, targeted turn, cancel, mismatch failure | + +### Required commands + +```bash +cargo test -p world-agent member_runtime +cargo test -p shell +cargo test -p world-api +cargo test -p world-mac-lima +cargo test --workspace -- --nocapture +scripts/mac/lima-warm.sh +scripts/mac/orchestration-smoke.sh +``` + +### Test plan artifact + +Implementation should write the normal eng-review test artifact alongside validation work so QA-style follow-up has a durable input: + +`~/.gstack/projects//-feat-session-centric-state-store-eng-review-test-plan-.md` + +It should list: + +1. shared-owner attach/create on macOS, +2. replacement with generation advancement, +3. world-member lazy launch, +4. targeted follow-up reuse, +5. bootstrap cancel and submitted-turn cancel, +6. mismatch rejection paths, +7. authoritative shared-binding surfacing on the Lima path. + +## Performance Review + +No major algorithmic or database risk exists here. The real performance risks are accidental session churn and extra guest round trips. + +Guardrails: + +1. preserve Lima session caching in [`MacLimaBackend::ensure_session(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/lib.rs:389), +2. do not introduce duplicate guest bootstrap calls when a retained member already matches the authoritative binding, +3. keep added request fields additive and optional so non-orchestration backend exec paths do not pay new work beyond serialization, +4. make the smoke harness assert reuse behavior so silent re-spawn regressions are caught. + +## Failure Modes Registry + +| Codepath | Realistic production failure | Test required | Error handling required | User-visible outcome | +| --- | --- | --- | --- | --- | +| macOS shared-owner attach/create | guest reply lacks `ready.shared_world` | yes | existing fail-closed validator | explicit bootstrap failure | +| macOS replacement | reply reuses stale `world_generation` | yes | existing generation-advance check | explicit replacement failure | +| member dispatch launch | shell constructs request without authoritative `world_id` or `world_generation` | yes | shell-side bootstrap failure | explicit launch failure | +| retained follow-up turn | stale member survives after rollover | yes | guest identity mismatch rejection | explicit targeted-turn failure, no silent reuse | +| cancel | cancel routed to wrong span or not routed at all | yes | explicit cancel error plus retained task cleanup | explicit cancel failure, not silent hang | +| backend parity | Lima backend still zeros `shared_world` or `member_dispatch` | yes | unit regression coverage | explicit test failure before ship | +| shared binding parity | `WorldHandle.shared_binding` stays `None` in owner mode | yes | unit regression coverage | shared-owner flows fail closed instead of silently downgrading | + +Critical gap rule for this plan: + +Any path that both depends on authoritative world identity and would otherwise silently fall back to host-local behavior is a release blocker. This slice only ships with explicit proof or explicit failure. + +## Detailed Implementation Plan + +### Single-worktree execution order + +If one engineer is doing this in one worktree, use this order: + +1. shared-owner bootstrap parity, +2. backend contract parity, +3. shell member-runtime parity, +4. validation and docs. + +Why this order: + +1. Workstream 1 is the smallest visible user-facing parity win and proves the forwarded path is real. +2. Workstream 2 is the highest blast-radius change and should land before shell follow-up logic depends on it. +3. Workstream 3 depends on both 1 and 2. +4. Workstream 4 should validate final semantics, not an intermediate state. + +### Phase 1: enable forwarded shared-owner bootstrap on macOS + +Implementation steps: + +1. change the guard in [`platform_world/mod.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/platform_world/mod.rs:61) from "non-Linux always reject" to "reject only when the backend cannot prove the forwarded shared-owner path", +2. thread that capability into [`world_persistent_session.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_persistent_session.rs:731) without introducing a second request path, +3. keep [`validate_shared_world_echo(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/repl_persistent_session.rs:308) unchanged except for any additive test coverage needed for macOS call sites, +4. add or update unit tests for attach/create, replacement, and invalid proof on the forwarded macOS path. + +Phase-1 exit criteria: + +1. macOS explicit shared-owner requests reach the forwarded guest transport, +2. attach/create and replacement both validate authoritative reply data, +3. malformed or missing proof still fails closed. + +### Phase 2: widen backend request and handle contracts + +Implementation steps: + +1. add optional `shared_world` and `member_dispatch` fields to [`ExecRequest`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-api/src/lib.rs:141), +2. update every constructor and test that instantiates `ExecRequest` to remain explicit about absence or presence, +3. ensure `world-agent` can pass those fields into backend execution when the request is orchestration-sensitive, +4. update Lima conversion so those fields survive host-to-guest forwarding, +5. update Lima ensure-session handling so `WorldHandle.shared_binding` is populated in owner mode, +6. keep Linux and Windows backends compiling by making the new fields additive, not required. + +Phase-2 exit criteria: + +1. all `ExecRequest` call sites compile cleanly, +2. owner-sensitive requests preserve their fields across the backend seam, +3. owner-mode Lima handles surface authoritative binding data. + +### Phase 3: widen shell member-runtime orchestration for macOS + +Implementation steps: + +1. remove the macOS hard-error posture from member-runtime readiness for supported Lima-backed world orchestration, +2. route lazy launch through the existing forwarded request builder in [`world_ops.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs:1369), +3. route targeted turns through `/v1/member_turn/stream` on macOS exactly as Linux already does, +4. route bootstrap and submitted-turn cancel through `/v1/execute/cancel`, +5. add or update unit coverage for reuse, stale generation, wrong backend, and cancel paths. + +Phase-3 exit criteria: + +1. macOS can lazily launch a world-scoped member through the guest, +2. targeted follow-up turns reuse that retained member when the binding still matches, +3. stale or mismatched identity fails closed, +4. cancel remains guest-owned. + +### Phase 4: validate and document parity + +Implementation steps: + +1. add `scripts/mac/orchestration-smoke.sh`, +2. make the script cover attach/create, replacement, lazy launch, targeted follow-up, cancel, and mismatch rejection, +3. update [`docs/WORLD.md`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/WORLD.md) so its shared-owner section no longer claims macOS pre-rejection for the supported path, +4. update [`AGENT_ORCHESTRATION_GAP_MATRIX.md`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md) to describe the remaining gap accurately after this lands. + +Phase-4 exit criteria: + +1. a real macOS host with Lima can run the smoke path successfully, +2. documentation matches shipped behavior, +3. the gap matrix no longer treats macOS/Lima parity as open for this seam. + +## Validation Matrix + +| Promise | Proof | +| --- | --- | +| macOS shared-owner attach/create no longer pre-rejects | unit tests on persistent-session dispatch plus Lima smoke | +| replacement requires generation advancement | unit test plus smoke case with stale generation rejection | +| backend seam preserves shared-owner and member-dispatch data | `world-api`, `world-agent`, and `world-mac-lima` tests | +| owner-mode Lima handles surface shared binding | `world-mac-lima` unit test | +| macOS can launch and reuse retained members | shell unit tests plus smoke targeted-turn scenario | +| cancel stays guest-owned | shell unit tests plus smoke cancel scenario | +| failure posture remains explicit and fail-closed | validator tests plus smoke mismatch scenarios | +| docs and gap matrix tell the truth | manual diff review in the same PR | + +## NOT in scope + +- redesigning `substrate -c`, because this slice is parity for existing REPL-first orchestration semantics +- creating a new mac-native orchestration service, because Lima already forwards into the Linux guest authority +- adding Windows/WSL parity, because that is a separate backend rollout +- weakening `backend_id`, `world_id`, `world_generation`, or `orchestration_session_id` checks +- introducing host-local fallback for world-scoped members when guest proof is unavailable +- broadening public control-surface semantics beyond the already frozen session/runtime model + +## Worktree Parallelization Strategy + +This plan has real parallelization room, but only if ownership boundaries stay clean. + +### Dependency table + +| Step | Modules touched | Depends on | +| --- | --- | --- | +| Shared-owner bootstrap parity | `crates/shell/src/execution/platform_world/`, `crates/shell/src/execution/routing/dispatch/`, `crates/shell/src/execution/repl_persistent_session.rs` | — | +| Backend contract widening | `crates/world-api/`, `crates/world-mac-lima/`, `crates/world-agent/`, backend callers and tests | — | +| Shell member-runtime parity | `crates/shell/src/repl/`, `crates/shell/src/execution/routing/dispatch/world_ops.rs` | shared-owner bootstrap parity, backend contract widening | +| Smoke harness and docs | `scripts/mac/`, `docs/`, `AGENT_ORCHESTRATION_GAP_MATRIX.md` | shell member-runtime parity | + +### Parallel lanes + +Lane A: shared-owner bootstrap parity +Sequential within lane because the platform guard and persistent-session transport must agree. + +Lane B: backend contract widening +Sequential within lane because `world-api`, `world-agent`, and `world-mac-lima` all depend on the additive request shape. + +Lane C: shell member-runtime parity +Waits for A + B, then proceeds sequentially through request construction, launch, follow-up, and cancel. + +Lane D: smoke harness and docs +Can draft script structure and docs skeleton early, but final assertions and wording wait for C. + +### Execution order + +1. Launch Lane A and Lane B in parallel worktrees. +2. Merge A and B. +3. Launch Lane C on top of the merged result. +4. Finish with Lane D once C is stable enough to validate end to end. + +### Conflict flags + +1. Lane A and Lane C both depend on shell orchestration semantics, but they should avoid merge conflict if A stays inside `platform_world/` and `world_persistent_session.rs` while C owns `async_repl.rs` and `world_ops.rs`. +2. Lane B should not touch `async_repl.rs`. If it does, parallelization value drops fast. +3. Lane D will conflict with anyone still changing the documented contract. Keep docs last. + +## Completion Summary + +- Step 0: Scope Challenge, accepted as-is. This is the minimum honest parity slice. +- Architecture Review: resolved in-plan. Four workstreams, one contract, no second macOS model. +- Code Quality Review: one request shape, one validator, one backend seam, one fail-closed posture. +- Test Review: coverage diagram produced, all parity-critical gaps called out explicitly. +- Performance Review: low algorithmic risk, high churn-risk if session reuse regresses. +- NOT in scope: written. +- What already exists: written. +- Failure modes: critical-gap rule frozen. +- Parallelization: 4 lanes total, 2 immediately parallel, 2 sequentially dependent. +- Lake Score: the complete option won everywhere that mattered. + +## Completion Checklist + +- [ ] macOS forwarded persistent-session shared-owner attach/create works +- [ ] macOS replacement requires generation advancement +- [ ] macOS world-member lazy launch uses forwarded `member_dispatch` +- [ ] macOS targeted follow-up turns use `/v1/member_turn/stream` +- [ ] macOS bootstrap and submitted-turn cancel use `/v1/execute/cancel` +- [ ] `ExecRequest` preserves shared-owner and member-dispatch semantics end to end +- [ ] `WorldHandle.shared_binding` is authoritative on the Lima owner path +- [ ] no supported macOS path silently falls back to host-local member ownership +- [ ] new orchestration smoke harness passes on a real macOS host with Lima provisioned +- [ ] docs and gap matrix match the shipped behavior + +## Done Means + +This slice is done when macOS/Lima stops being a second-class exception and instead uses the same shared-owner proof, retained member launch, targeted turn, cancel, and fail-closed mismatch contract Linux already uses. + +Not "mostly there." Not "transport exists but cfg gates still lie." Real parity, proven by code-level tests and a reproducible Lima-backed validation path. diff --git a/llm-last-mile/PLAN-22.md b/llm-last-mile/PLAN-22.md new file mode 100644 index 000000000..31faac34b --- /dev/null +++ b/llm-last-mile/PLAN-22.md @@ -0,0 +1,785 @@ +# PLAN-22: Broaden Caller Surfaces From REPL-First To Public Session/Member Turns + +Source SOW: [22-broaden-caller-surfaces-from-repl-first-to-public-session-member-turns.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/22-broaden-caller-surfaces-from-repl-first-to-public-session-member-turns.md) +Gap matrix anchors: [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md:107), [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md:112) +Adjacent landed slices: [PLAN-15.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-15.md), [PLAN-19.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-19.md), [PLAN-20.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-20.md), [PLAN-21.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/PLAN-21.md) +Branch: `feat/broaden-caller-surfaces-from-repl` +Base branch: `main` +Plan type: public caller-surface hardening and Linux-first world-member follow-up validation +Review posture: unified execution plan, tightened to `/autoplan` and `/plan-eng-review` rigor +Status: execution-ready planning pass on 2026-05-09 + +## Objective + +Take the already-landed narrow public caller surface under `substrate agent` and make it an explicitly validated v1 contract for session-scoped and member-scoped prompt-taking. + +This slice is complete only when all of the following are true: + +1. `substrate agent start` remains the canonical public root prompt-taking surface for exact host backends. +2. `substrate agent turn` is proven end to end for exact `(orchestration_session_id, backend_id)` follow-up routing, not just lightly wired. +3. Linux world-sensitive public follow-up turns are explicitly proven from selector resolution through typed `MemberTurnSubmitRequestV1` submission into `world-agent`. +4. Detached host recovery and detached world fail-closed behavior are both explicit parts of the public contract. +5. The fail-closed taxonomy for public turns is covered with concrete tests instead of being partially implied by implementation. + +This is productization and hardening of a real surface. It is not a new orchestration model. + +## Plan Summary + +The repo is much further along than the draft SOW implies. + +Today the narrow public surface already exists: + +1. CLI verbs are already exposed in [`AgentAction`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/cli.rs:519). +2. Public `start` and `turn` already load prompt input and enter the shared prompt path in [`run_start(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:301) and [`run_turn(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:324). +3. Exact public turn routing already exists in [`resolve_public_turn_target(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:711). +4. The shared public prompt submission and rendering path already exists in [`run_public_prompt_command(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/control.rs:1104). +5. The Linux world-member submit boundary is already typed in [`MemberTurnSubmitRequestV1`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/agent-api-types/src/lib.rs:845), accepted in [`submit_member_turn_stream(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs:1485), and identity-validated in [`validate_submit_turn_request(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs:869). +6. Host-scoped public `start`, `turn`, `stop`, `reattach`, selector rejection, world-root start rejection, and `substrate -c` non-regression already have meaningful coverage in [`agent_public_control_surface_v1.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_public_control_surface_v1.rs:662). + +What is still missing is narrower and sharper: + +1. the public Linux world-member follow-up path is not yet explicitly proven in the public control suite, +2. several fail-closed public-turn cases exist in code but are not yet pinned as explicit acceptance gates, +3. detached-world follow-up rejection exists in code but still needs direct contract coverage, +4. retained-member identity drift is only lightly proven at the world-agent boundary and should become an explicit public-turn acceptance gate, +5. repo-truth docs still lag the shipped surface in a few places, especially [`llm-last-mile/README.md`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/README.md:64). + +The minimum honest implementation is one cohesive hardening slice with four ordered workstreams: + +1. tighten and prove the shell-side exact public turn contract, +2. add explicit public Linux world-member follow-up proof, +3. widen retained-member drift and detached-world fail-closed proof, +4. update operator and planning docs so repo truth matches reality. + +## Locked Starting State + +### What already exists + +| Sub-problem | Existing code | Decision | +| --- | --- | --- | +| Public caller verbs and canonical naming | [`AgentAction`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/cli.rs:519) | Reuse. Do not invent a second caller family outside `substrate agent`. | +| Public root prompt-taking wiring | [`run_start(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:301) | Reuse. This stays the canonical root caller surface. | +| Public follow-up prompt-taking wiring | [`run_turn(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:324) | Reuse and harden. Do not split host and world follow-up into separate public verbs. | +| Exact `(session, backend)` selector resolution | [`resolve_public_turn_target(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:711) | Reuse as the authoritative selector seam. Tighten coverage around it. | +| Shared prompt-source loading | [`load_public_prompt_source(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/control.rs:1067) | Reuse. This is already the one prompt-source contract. | +| Shared public prompt transport and rendering | [`run_public_prompt_command(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/control.rs:1104) | Reuse. Keep one public prompt path, not parallel host and world implementations. | +| Typed world-member submit request | [`MemberTurnSubmitRequestV1`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/agent-api-types/src/lib.rs:845) | Reuse exactly. Do not invent a second world-turn request surface. | +| Linux world-agent public-turn submit entrypoint | [`submit_member_turn_stream(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs:1485) | Reuse exactly. This remains the typed public world-member seam. | +| Retained-member tuple validation | [`validate_submit_turn_request(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs:869) | Reuse exactly. Tighten proof, do not loosen identity matching. | +| Host-scoped public control integration coverage | [`agent_public_control_surface_v1.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_public_control_surface_v1.rs:662) | Reuse as the primary shell integration suite and extend it. | +| `substrate -c` shell-wrap contract | [`agent_public_control_surface_v1.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_public_control_surface_v1.rs:1129) | Reuse exactly. This remains a hard non-regression gate. | + +### Exact remaining gap + +1. [`run_turn(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:324) already branches on `PublicSessionPosture`, but detached-world rejection is still only implementation truth until there is a direct test for it. +2. [`resolve_public_turn_target(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:711) already emits `missing_backend`, `missing_active_parent`, `stale_linkage`, `backend_not_in_session`, `ambiguous_backend_slot`, and `unsupported_platform_or_posture`, but the suite does not yet pin all of those as operator-facing acceptance cases. +3. [`run_public_prompt_command(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/control.rs:1104) already calls the exact turn resolver again before submission, but the suite still needs an explicit Linux world-member success path proving the public CLI reaches the retained world-member submit seam. +4. [`validate_submit_turn_request(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs:869) already rejects tuple drift, but only a narrow backend-drift unit case is currently pinned in [`member_runtime.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs:1112). That is not enough to make the hidden contract obvious. +5. [`llm-last-mile/README.md`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/README.md:64) still describes the older `start|resume|fork|stop` public surface and needs to reflect the landed `turn`/`reattach` contract. + +### Scope decision + +Proceed as one hardening slice. + +Do not split this into "shell fail-closed cleanup first" and "world-member proof later." That would keep the public surface half-proven and let docs drift another cycle. The public surface is either a deliberately validated v1 contract or it is not. + +### Blast radius + +GitNexus says the key caller-surface symbols are low individual blast-radius changes, but they sit on a cross-crate contract seam: + +| Symbol | GitNexus risk | Why it matters | +| --- | --- | --- | +| [`run_turn(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:324) | `LOW` | One direct caller, but it is the entire public `agent turn` path and participates in the `handle_agent_command` and `run_shell_with_cli` flows. | +| [`resolve_public_turn_target(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:711) | `LOW` | Two direct callers, [`run_turn(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:324) and [`run_public_prompt_command(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/control.rs:1104). | +| [`run_public_prompt_command(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/control.rs:1104) | `LOW` | It is the one prompt-submission bridge for both public `start` and public `turn`. | +| [`submit_turn(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs:249) | `LOW` | One direct caller, [`submit_member_turn_stream(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs:1485), but it is the retained world-member trust boundary. | + +Implication: keep changes boring and additive. The symbol graph is not huge, but the contract is sensitive because it crosses shell, state store, world-agent, and test fixtures. + +## Frozen Execution Contract + +If implementation wants to do something else, revise this plan first. + +### Non-negotiable invariants + +1. `substrate agent start` remains the canonical public root prompt-taking surface. +2. `substrate agent turn` remains the canonical public follow-up prompt-taking surface. +3. Public root start remains host-only in v1. World-only root start still fails closed. +4. Public follow-up turns require exact `--session ` and exact `--backend `. +5. Exact `(session, backend)` resolution stays authoritative. No fuzzy routing, no latest-session fallback, no selector widening. +6. Linux world-sensitive follow-up turns keep using the typed `MemberTurnSubmitRequestV1` to `/v1/member_turn/stream`. +7. Detached host follow-up may recover through reattach semantics. Detached world follow-up must fail closed until the operator runs `substrate agent reattach --session ...`. +8. No public prompt-taking path may fall back to REPL state, `substrate -c`, or synthetic bootstrap prompt text. +9. `substrate -c` remains shell wrap mode, full stop. +10. Failures stay explicit and classifier-stable. If the command cannot prove the exact contract, it fails closed. + +### Public start + +`substrate agent start` stays exactly what it is now: + +```text +substrate agent start --backend (--prompt | --prompt-file | --prompt-file -) [--json] +``` + +Rules: + +1. exact host backend only, +2. exact prompt-source requirement stays intact, +3. output remains streamed through the existing public prompt plane, +4. final `session_posture` remains authoritative only at command completion time. + +### Public turn + +`substrate agent turn` stays exactly what it is now: + +```text +substrate agent turn --session --backend (--prompt | --prompt-file | --prompt-file -) [--json] +``` + +Rules: + +1. session selector must be canonical orchestration-session identity only, +2. backend selector must be exact `:`, +3. exact selector resolution is necessary but not sufficient, the runtime path must also prove the target is currently valid, +4. host and world follow-up remain separate internal execution paths, hidden behind one public verb. + +### Hidden engineering contract for world-member follow-up + +For Linux world-sensitive public follow-up turns, the public selector pair `(orchestration_session_id, backend_id)` is only the front door. The retained identity tuple is the real trust boundary. + +That tuple remains: + +1. `participant_id` +2. `orchestrator_participant_id` +3. `backend_id` +4. `world_id` +5. `world_generation` + +The shell must submit those fields through [`MemberTurnSubmitRequestV1`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/agent-api-types/src/lib.rs:845), and [`validate_submit_turn_request(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs:869) must keep rejecting drift before turn submission resumes the retained member. + +### Fail-closed rules + +Public follow-up turns must fail closed for: + +1. `missing_backend` +2. `unknown_session` +3. all non-canonical session selectors +4. `missing_active_parent` +5. `backend_not_in_session` +6. `stale_linkage` +7. `ambiguous_backend_slot` +8. `unsupported_platform_or_posture` +9. `owner_unreachable` + +No fallback is allowed to a guessed session, guessed backend, REPL handle, or shell-wrap execution. + +## Step 0: Scope Challenge + +### 0A. Minimum honest diff + +The minimum honest implementation is: + +1. extend shell tests so the public `turn` path explicitly proves Linux world-member follow-up success, +2. extend shell tests so each public fail-closed classifier named above is explicitly pinned, +3. extend world-agent tests so retained-member identity drift is proven as a first-class contract, not a lightly covered side case, +4. update docs and planning artifacts that still describe the older narrower public caller surface. + +Anything smaller leaves the public surface partially implied instead of deliberately validated. + +### 0B. Complexity check + +This slice should stay below the "new infrastructure" threshold even if it touches more than eight files. The implementation should feel like hardening, not invention. + +Expected primary modules: + +1. `crates/shell/src/execution/agents_cmd.rs` +2. `crates/shell/src/execution/agent_runtime/state_store.rs` +3. `crates/shell/src/execution/agent_runtime/control.rs` +4. `crates/shell/tests/agent_public_control_surface_v1.rs` +5. `crates/shell/tests/support/` +6. `crates/world-agent/src/member_runtime.rs` +7. `docs/USAGE.md` +8. `AGENT_ORCHESTRATION_GAP_MATRIX.md` +9. `llm-last-mile/README.md` + +That is above the file-count smell, so the plan must stay boring: + +1. no new command surface, +2. no new transport, +3. no new public selector type, +4. no platform-general parity push beyond the Linux-first contract already frozen. + +### 0C. Search and reuse check + +Repo search says the right reuse story is already sitting in the code: + +- **[Layer 1]** reuse the exact public selector resolver in [`resolve_public_turn_target(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:711), +- **[Layer 1]** reuse the exact public prompt bridge in [`run_public_prompt_command(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/control.rs:1104), +- **[Layer 1]** reuse the typed world-member submit contract in [`MemberTurnSubmitRequestV1`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/agent-api-types/src/lib.rs:845), +- **[Layer 1]** reuse the existing public control integration harness in [`agent_public_control_surface_v1.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_public_control_surface_v1.rs:662), +- **[EUREKA]** the missing work is not "build a public caller surface." The public caller surface is already live. The missing work is proving the world-member path and proving the failure edges so the public contract is honest. + +### 0D. TODOS cross-reference + +There is no `TODOS.md` in the repo root today. That means deferrals must be captured explicitly inside this plan's `NOT in scope` section and, if implementation discovers adjacent but non-blocking work, in a follow-up backlog artifact rather than assumed root TODOs. + +### 0E. Completeness check + +This slice should boil the lake inside its blast radius. + +The complete version includes: + +1. host-scoped public start/turn regressions staying green, +2. Linux world-member public-turn success proof, +3. explicit classifier coverage for each fail-closed case named in the SOW, +4. detached host and detached world posture proof, +5. retained-member identity drift proof at the world-agent seam, +6. doc truth updates. + +Skipping any one of those would save little time and keep the public contract fuzzy. + +### 0F. Distribution and docs check + +No new artifact type is introduced. + +Distribution still matters in the boring sense: + +1. `docs/USAGE.md` must describe the public surface the code actually ships, +2. `AGENT_ORCHESTRATION_GAP_MATRIX.md` must stop talking about this surface as if it were still hypothetical, +3. `llm-last-mile/README.md` must stop lagging behind the landed `turn`/`reattach` surface, +4. `PLAN-22.md` itself becomes the implementable artifact for this slice. + +## Architecture Review + +### Architecture thesis + +Keep the public surface exactly where it is, `substrate agent`, and finish proving it. + +The shell remains the selector and owner-transport authority. The Linux world-agent remains the world-sensitive authority. The public `(session, backend)` pair remains the front door, not the full trust boundary. + +### Data flow + +```text +CURRENT +======= +substrate agent turn + | + +--> run_turn(...) + | | + | +--> load_public_prompt_source(...) + | +--> resolve_public_turn_target(...) + | +--> detached host can recover + | `--> detached world fails closed + | + `--> run_public_prompt_command(...) + | + +--> resolve_public_turn_target(...) again + +--> host: private owner transport + `--> world: typed retained-member submit on Linux + +What is weak today: + - host path is directly proven + - world-member path is mostly code truth + - several fail-closed branches are mostly code truth + + +TARGET +====== +substrate agent turn + | + +--> exact prompt load + +--> exact (session, backend) resolution + +--> exact posture branch + | | + | +--> active host -> private owner prompt stream + | +--> detached host -> reattach then private owner prompt stream + | +--> active Linux world -> MemberTurnSubmitRequestV1 -> /v1/member_turn/stream + | +--> detached world -> fail closed with reattach guidance + | `--> terminal/invalid -> fail closed with explicit classifier + | + `--> tests prove every branch above + +world-agent boundary + | + +--> MemberTurnSubmitRequestV1.validate() + +--> validate_submit_turn_request(...) + `--> submit_turn(...) + +Tuple drift or stale linkage never degrades into best effort. +``` + +### Workstream 1: shell-side public turn contract hardening + +Goal: make the exact public turn selector and posture contract fully explicit in shell-level tests and only minimally adjusted in code where gaps appear. + +Files: + +- [`crates/shell/src/execution/agents_cmd.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:324) +- [`crates/shell/src/execution/agent_runtime/state_store.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:711) +- [`crates/shell/src/execution/agent_runtime/control.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/control.rs:1104) +- [`crates/shell/tests/agent_public_control_surface_v1.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_public_control_surface_v1.rs:662) + +Required changes: + +1. Add explicit coverage for `missing_backend`, `unknown_session`, `missing_active_parent`, `backend_not_in_session`, `stale_linkage`, `ambiguous_backend_slot`, and `owner_unreachable`. +2. Add explicit coverage for detached-world follow-up rejection with the required `reattach` guidance. +3. Keep the current host-scoped public `start` and `turn` assertions green. +4. Keep `substrate -c` shell-wrap behavior green. + +Exit criteria: + +1. every operator-facing public-turn classifier named in the frozen contract has a direct assertion, +2. detached host and detached world posture behavior are both pinned, +3. no host-scoped regression is introduced while tightening the contract. + +### Workstream 2: public Linux world-member follow-up proof + +Goal: explicitly prove that a public `substrate agent turn --session ... --backend ` reaches the retained Linux world-member seam, not just a generic success path. + +Files: + +- [`crates/shell/tests/agent_public_control_surface_v1.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_public_control_surface_v1.rs:662) +- [`crates/shell/tests/support/`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/support/) +- [`crates/shell/tests/repl_world_first_routing_v1.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs) + +Required changes: + +1. Add an integration scenario that prepares an authoritative world-scoped retained slot for a public session/backend pair. +2. Submit a public `agent turn` against that exact pair and assert the world-sensitive path is used. +3. Assert that the shaped request carries the retained member tuple required by [`MemberTurnSubmitRequestV1`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/agent-api-types/src/lib.rs:845). +4. Assert that the path reaches the Linux `/v1/member_turn/stream` seam rather than a host-only shortcut. + +Exit criteria: + +1. the public suite proves Linux world-member follow-up success end to end, +2. the proof is explicit about retained-member identity, not just "turn succeeded", +3. REPL world-first routing tests still pass unchanged. + +### Workstream 3: retained-member identity drift and fail-closed proof + +Goal: make the hidden retained-member tuple contract impossible to miss during review or regression. + +Files: + +- [`crates/world-agent/src/member_runtime.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs:869) +- [`crates/world-agent/src/service.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/service.rs:1485) +- relevant `world-agent` tests + +Required changes: + +1. Keep the existing backend-drift rejection test. +2. Add at least one additional tuple-drift case at the retained submit boundary. `world_generation` or `world_id` is preferred because those are the highest-risk stale-slot cases. +3. Add or extend a boundary-level test proving the typed submit entrypoint keeps rejecting identity drift before turn submission starts. +4. Keep the failure text explicit enough that the operator and reviewer can tell which tuple field drifted. + +Exit criteria: + +1. retained-member identity drift is directly proven beyond one backend-only mismatch, +2. the world-agent boundary remains fail-closed before submitted-turn execution begins, +3. the hidden tuple contract becomes obvious from tests, not just from reading production code. + +### Workstream 4: operator and planning truth updates + +Goal: make docs tell the truth after the hardening work lands. + +Files: + +- [`docs/USAGE.md`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/USAGE.md:62) +- [`AGENT_ORCHESTRATION_GAP_MATRIX.md`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md:107) +- [`llm-last-mile/README.md`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/README.md:64) +- the source SOW and this plan if wording needs tightening for posterity + +Required changes: + +1. update operator docs only where behavior or wording is still stale, +2. explicitly describe `turn`/`reattach` where older docs still imply `resume` or omit follow-up prompt-taking, +3. keep the Linux-first posture and host-only root start caveat explicit, +4. keep docs aligned with the actual acceptance gates, not aspirational future parity. + +Exit criteria: + +1. no repo-truth doc still describes the pre-`turn` public surface, +2. docs match the shipped failure posture, +3. no doc claims broader platform parity than the code proves. + +## Code Quality Review + +### Boring-by-default rules + +1. One public caller family, `substrate agent`. +2. One exact public turn resolver, [`resolve_public_turn_target(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:711). +3. One public prompt bridge, [`run_public_prompt_command(...)`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/control.rs:1104). +4. One world-sensitive submit request, [`MemberTurnSubmitRequestV1`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/agent-api-types/src/lib.rs:845). +5. One fail-closed posture for detached world follow-up, explicit rejection with reattach guidance. + +### DRY and abstraction guardrails + +1. Do not add a second public turn resolver for tests. Tests should exercise the real one. +2. Do not add a second world-turn request just to make assertions easier. Assert against the existing typed request. +3. Do not fork host and world prompt rendering behavior inside docs or tests. The test should pin where the internal path differs, not create a new contract. +4. If implementation needs helper fixture changes, keep them in `crates/shell/tests/support/` rather than hand-building parallel fake state in each test. + +### Diagram maintenance + +If touched files already contain nearby ASCII diagrams or contract comments, update them in the same change. Stale diagrams are worse than no diagrams. + +Recommended inline diagram comment locations if code changes are needed: + +1. [`state_store.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:707), around the exact `(session, backend)` public-turn routing comment, +2. [`member_runtime.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs:848), around retained-member identity and submit validation, +3. [`agents_cmd.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:324), if posture branching gets any new nuance. + +## Test Review + +### Test framework detection + +This repo is Rust-first. The acceptance surface here is `cargo test`, with shell integration tests and world-agent unit or integration tests doing the real work. + +Primary suites for this slice: + +1. [`crates/shell/tests/agent_public_control_surface_v1.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_public_control_surface_v1.rs) +2. [`crates/shell/tests/repl_world_first_routing_v1.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs) +3. targeted tests around [`crates/shell/src/execution/agent_runtime/state_store.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) +4. targeted tests around [`crates/world-agent/src/member_runtime.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs) + +### Code path coverage + +```text +CODE PATH COVERAGE +=========================== +[+] crates/shell/tests/agent_public_control_surface_v1.rs + | + ├── public host start -> turn -> stop flow + │ └── [★★★ TESTED] start/turn streaming NDJSON + authoritative state + | + ├── public reattach + fork lineage flow + │ └── [★★★ TESTED] exact session + lineage preservation + | + ├── non-canonical selector rejection + │ └── [★★ TESTED] active handle / participant / internal session rejection + | + ├── public world-root start rejection + │ └── [★★★ TESTED] host-only v1 root start remains fail-closed + | + ├── shell-wrap `substrate -c` + │ └── [★★★ TESTED] remains shell mode, not agent prompting + | + ├── public Linux world-member turn success + │ └── [GAP -> INTEG] exact `(session, backend)` world follow-up reaches typed member-turn submit seam + | + ├── detached world follow-up rejection + │ └── [GAP -> INTEG] explicit `reattach` guidance assertion + | + ├── missing_backend + │ └── [GAP -> UNIT/INTEG] direct public-turn rejection + | + ├── unknown_session + │ └── [GAP -> UNIT/INTEG] direct public-turn rejection + | + ├── missing_active_parent + │ └── [GAP -> INTEG] missing parent metadata and inactive parent rejection + | + ├── backend_not_in_session + │ └── [GAP -> INTEG] explicit exact-backend miss + | + ├── stale_linkage + │ └── [GAP -> INTEG] backend mentioned but no authoritative retained target + | + ├── ambiguous_backend_slot + │ └── [GAP -> INTEG] multiple authoritative retained targets for one backend + | + └── owner_unreachable / terminal posture + └── [GAP -> INTEG] explicit terminal follow-up rejection + +[+] crates/world-agent/src/member_runtime.rs + | + ├── validate_submit_turn_request(...) + │ ├── [★★ TESTED] backend_id drift rejection + │ └── [GAP -> UNIT] world_id or world_generation drift rejection as explicit boundary proof + | + └── submit_turn(...) + └── [GAP -> UNIT/INTEG] boundary-level assertion that typed submit fails before execution on retained tuple drift + +--------------------------------- +COVERAGE TARGET +- keep existing host-scoped public caller coverage green +- add one explicit Linux world-member success path +- add explicit assertions for every named fail-closed classifier in this plan +- add at least one additional retained-tuple drift proof at the world-agent boundary +--------------------------------- +``` + +### Operator flow coverage + +```text +OPERATOR FLOW COVERAGE +=========================== +[+] Operator runs `substrate agent start --backend cli:codex --prompt "hello" --json` + | + └── [★★★ TESTED] accepted -> event stream -> completed with session_posture + +[+] Operator runs `substrate agent turn --session --backend cli:codex --prompt "next" --json` + | + └── [★★★ TESTED] exact host follow-up succeeds + +[+] Operator runs `substrate agent turn --session --backend cli:claude_code --prompt "next" --json` + | + ├── [GAP -> INTEG] active Linux world-member follow-up succeeds through `/v1/member_turn/stream` + └── [GAP -> INTEG] detached world follow-up rejects with reattach guidance + +[+] Operator targets wrong or stale session/backend state + | + ├── [GAP] unknown_session + ├── [GAP] missing_active_parent + ├── [GAP] backend_not_in_session + ├── [GAP] stale_linkage + ├── [GAP] ambiguous_backend_slot + └── [GAP] terminal owner_unreachable posture + +[+] Operator uses a non-canonical session selector + | + └── [★★ TESTED] reject active handle / participant / internal selector + +[+] Operator runs `substrate -c "printf shell-wrap"` + | + └── [★★★ TESTED] shell-wrap stays shell-wrap +``` + +### Required test files and assertions + +| Area | File | Required assertions | +| --- | --- | --- | +| Public host caller non-regression | [`crates/shell/tests/agent_public_control_surface_v1.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_public_control_surface_v1.rs) | current `start`/`turn`/`stop` and `reattach`/`fork` flows stay green | +| Public Linux world-member follow-up | [`crates/shell/tests/agent_public_control_surface_v1.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_public_control_surface_v1.rs) plus support fixtures | exact world-scoped public `turn` reaches typed world-member submit path | +| Exact fail-closed public-turn taxonomy | [`crates/shell/tests/agent_public_control_surface_v1.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_public_control_surface_v1.rs) | direct classifier assertions for missing backend, unknown session, missing parent, stale linkage, ambiguous slot, terminal posture, detached-world guidance | +| World-first routing non-regression | [`crates/shell/tests/repl_world_first_routing_v1.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/repl_world_first_routing_v1.rs) | REPL world routing stays unchanged while public proof is added | +| Retained identity drift | [`crates/world-agent/src/member_runtime.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs) tests | at least one non-backend tuple drift case plus boundary-level submit rejection | + +### Required commands + +```bash +cargo test -p shell --test agent_public_control_surface_v1 -- --nocapture +cargo test -p shell --test repl_world_first_routing_v1 -- --nocapture +cargo test -p shell agent_runtime::state_store -- --nocapture +cargo test -p world-agent member_runtime -- --nocapture +cargo test --workspace -- --nocapture +``` + +### Test plan artifact + +Implementation should write the normal eng-review test artifact alongside code changes: + +```text +~/.gstack/projects//-feat-broaden-caller-surfaces-from-repl-eng-review-test-plan-.md +``` + +Required contents: + +1. host public start with streaming acceptance and completion, +2. host public follow-up turn, +3. Linux public world-member follow-up turn, +4. detached host follow-up recovery, +5. detached world follow-up rejection with `reattach` guidance, +6. explicit fail-closed selector and posture cases, +7. retained-member tuple drift rejection, +8. `substrate -c` shell-wrap non-regression. + +## Performance Review + +This slice is not algorithmically heavy. The real performance risk is accidental retry or lookup churn while tightening proof. + +Guardrails: + +1. keep exact resolution store-backed, no broad trace scans, +2. do not add duplicate world-member submission attempts just to gather more proof, +3. keep fixture setup deterministic so tests do not depend on long polling loops, +4. prefer direct state-store fixtures over heavyweight full-runtime setup when proving fail-closed cases that do not need streaming. + +The performance principle here is simple: correctness first, but do not turn hardening into a slow flaky integration suite. + +## Failure Modes Registry + +| Codepath | Realistic production failure | Test required | Error handling required | User-visible outcome | +| --- | --- | --- | --- | --- | +| public host follow-up | session exists but backend slot does not | yes | explicit `backend_not_in_session` | clear follow-up rejection | +| public follow-up selector | session file exists but authoritative parent metadata is torn | yes | explicit `missing_active_parent` | clear follow-up rejection | +| public follow-up selector | backend mentioned in stale inventory but retained target is gone | yes | explicit `stale_linkage` | clear follow-up rejection | +| public follow-up selector | two authoritative slots claim the same backend | yes | explicit `ambiguous_backend_slot` | clear follow-up rejection | +| detached host follow-up | owner loop is gone but reattachable metadata remains | yes | reattach before submit | clear recovered success or explicit failure | +| detached world follow-up | world slot exists but no active host owner remains | yes | explicit detached-world fail-closed guidance | clear instruction to run `reattach` | +| active Linux world follow-up | selector resolves but request is mis-shaped before submit | yes | typed request validation plus exact tuple proof | explicit submission failure before execution | +| retained world-member follow-up | stale `world_generation` or wrong `world_id` reaches world-agent | yes | `validate_submit_turn_request(...)` rejection | clear tuple-drift rejection | +| public terminal follow-up | session is terminal but caller retries with a real backend | yes | explicit `owner_unreachable` | clear terminal rejection | +| shell-wrap mode | `-c` accidentally starts reading prompt semantics | yes | no prompt re-interpretation | normal shell output only | + +Critical-gap rule for this plan: + +Any path that can submit a public follow-up turn without proving the exact retained target identity is a release blocker. Any path that silently downgrades world-sensitive follow-up into host-local best effort is also a release blocker. + +## Implementation Sequence + +### Step 1. Freeze the exact public-turn acceptance matrix + +Files: + +1. [`crates/shell/tests/agent_public_control_surface_v1.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_public_control_surface_v1.rs) +2. small production adjustments only if a classifier or wording mismatch is discovered while pinning tests + +Deliver: + +1. enumerate every fail-closed case already named by production code, +2. add explicit assertions for each, +3. keep the current host flows green. + +Done means the operator-facing shell contract is no longer inferred from implementation. + +### Step 2. Add public Linux world-member follow-up proof + +Files: + +1. [`crates/shell/tests/agent_public_control_surface_v1.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_public_control_surface_v1.rs) +2. `crates/shell/tests/support/` +3. only the smallest production hooks needed to expose the right assertion points + +Deliver: + +1. prepare an authoritative world-scoped retained slot, +2. execute `substrate agent turn` against the exact pair, +3. prove the route reaches the typed retained-member seam, +4. prove detached-world posture rejects with reattach guidance. + +Done means the public world-member path is no longer "probably right." + +### Step 3. Tighten retained-identity drift proof at the world-agent boundary + +Files: + +1. [`crates/world-agent/src/member_runtime.rs`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/member_runtime.rs) +2. any directly adjacent tests + +Deliver: + +1. preserve existing backend-drift proof, +2. add at least one more tuple-drift proof, +3. prove the boundary rejects drift before submitted-turn execution begins. + +Done means the hidden tuple contract is obvious in tests and survives refactors. + +### Step 4. Update operator and planning docs + +Files: + +1. [`docs/USAGE.md`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/USAGE.md) +2. [`AGENT_ORCHESTRATION_GAP_MATRIX.md`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md) +3. [`llm-last-mile/README.md`](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/llm-last-mile/README.md) +4. the source SOW only if language is still materially misleading after code lands + +Deliver: + +1. doc wording matches the shipped public surface, +2. Linux-first and host-only caveats remain explicit, +3. no stale mention of the older pre-`turn` public contract remains. + +Done means repo truth matches code truth. + +## Validation Matrix + +| Promise | Proof | +| --- | --- | +| host-scoped public `start` and `turn` still work | existing `agent_public_control_surface_v1` assertions stay green | +| Linux public world-member follow-up now has direct proof | new shell integration scenario exercising exact `(session, backend)` world turn | +| detached-world follow-up remains fail-closed | explicit shell integration test with `reattach` guidance | +| each public-turn fail-closed classifier is stable | direct shell tests asserting classifier text | +| retained-member tuple drift remains rejected | world-agent tests around `validate_submit_turn_request(...)` and `submit_turn(...)` | +| `substrate -c` remains shell-wrap mode | existing shell integration assertion stays green | +| docs match shipped behavior | diff review in the same PR | + +## NOT in scope + +- redesigning `substrate -c`, because shell-wrap mode is already frozen and protected +- adding fuzzy or default backend routing, because exact `(session, backend)` is the whole public safety story +- public world-root start, because root start remains host-only in v1 +- broad status-surface redesign, because this slice is prompt-taking hardening, not status UX +- `fork` or `stop` product expansion, because they are non-prompt-taking lifecycle verbs and only need non-regression coverage here +- Windows/WSL world-sensitive follow-up parity, because this plan is Linux-first validation of the existing retained-member seam +- any new daemon, background broker, or alternate public transport, because the current surface already exists + +## Worktree Parallelization Strategy + +This plan has real parallelization room if write boundaries stay disciplined. + +### Dependency table + +| Step | Modules touched | Depends on | +| --- | --- | --- | +| Shell public-turn contract hardening | `crates/shell/src/execution/`, `crates/shell/tests/`, `crates/shell/tests/support/` | — | +| World-agent retained-identity drift proof | `crates/world-agent/`, `crates/agent-api-types/` | — | +| Public Linux world-member integration proof | `crates/shell/tests/`, `crates/shell/tests/support/`, possibly thin shell hooks | Shell public-turn contract hardening, World-agent retained-identity drift proof | +| Docs and planning truth updates | `docs/`, `AGENT_ORCHESTRATION_GAP_MATRIX.md`, `llm-last-mile/` | Public Linux world-member integration proof | + +### Parallel lanes + +Lane A: shell public-turn contract hardening +Sequential within lane because the shell tests and any tiny wording or classifier fixes share the same public caller modules. + +Lane B: world-agent retained-identity drift proof +Sequential within lane because `member_runtime.rs` and its tests own the retained tuple contract. + +Lane C: public Linux world-member integration proof +Waits for A + B, then proceeds through shell fixtures and integration assertions. + +Lane D: docs and planning truth updates +Waits for C so wording matches shipped proof, not intermediate intent. + +### Execution order + +1. Launch Lane A and Lane B in parallel worktrees. +2. Merge A and B. +3. Launch Lane C on top of the merged result. +4. Finish with Lane D once the final tests pass. + +### Conflict flags + +1. Lane A and Lane C both touch `crates/shell/tests/agent_public_control_surface_v1.rs`. Do not run them in parallel. +2. Lane B must stay out of shell test fixtures. If it starts changing shell support code, the lane split stops being useful. +3. Lane D goes last. Updating docs before the public world-member proof lands will drift repo truth again. + +### Parallelization verdict + +Four workstreams, two immediately parallel implementation lanes, one integration lane, one final doc lane. + +## Completion Summary + +- Step 0: Scope Challenge, accepted as-is. This is the minimum honest hardening slice. +- Architecture Review: resolved in-plan. Reuse the landed public surface, prove the missing world-member and fail-closed edges. +- Code Quality Review: one caller family, one turn resolver, one prompt bridge, one typed world submit request. +- Test Review: coverage diagram produced, explicit gaps identified for world-member proof and remaining fail-closed cases. +- Performance Review: low algorithmic risk, moderate test-fixture churn risk if proof is implemented sloppily. +- NOT in scope: written. +- What already exists: written. +- Failure modes: critical-gap rule frozen. +- Parallelization: 4 workstreams, 2 parallel implementation lanes, 1 integration lane, 1 doc lane. +- Lake Score: the complete option wins because partial proof would still leave the public contract fuzzy. + +## Completion Checklist + +- [ ] public host `start` and `turn` remain green +- [ ] public Linux world-member `turn` is proven end to end +- [ ] detached host recovery stays explicit and tested +- [ ] detached world follow-up rejection is explicit and tested +- [ ] `missing_backend` is explicitly tested +- [ ] `unknown_session` is explicitly tested +- [ ] `missing_active_parent` is explicitly tested +- [ ] `backend_not_in_session` is explicitly tested +- [ ] `stale_linkage` is explicitly tested +- [ ] `ambiguous_backend_slot` is explicitly tested +- [ ] `owner_unreachable` terminal follow-up is explicitly tested +- [ ] retained-member tuple drift is explicitly tested beyond backend-only mismatch +- [ ] `substrate -c` remains shell-wrap mode +- [ ] docs and llm-last-mile truth artifacts match the shipped public surface + +## Done Means + +This slice is done when `substrate agent start` and `substrate agent turn` stop being "the public surface that mostly exists in code" and become "the public surface that is deliberately proven, fail-closed, and documented." + +Not a new product surface. Not a transport rewrite. Just the last mile of honesty between the landed runtime and the contract we claim to support. diff --git a/llm-last-mile/PLAN-25.md b/llm-last-mile/PLAN-25.md new file mode 100644 index 000000000..b33939ead --- /dev/null +++ b/llm-last-mile/PLAN-25.md @@ -0,0 +1,729 @@ +# PLAN-25: Durable Host Session Closeout, Inbox Contract Honesty, And QA Hardening + +Source SOW: [25-host-durable-session-closeout-and-qa-hardening.md](/home/spenser/__Active_code/substrate/llm-last-mile/25-host-durable-session-closeout-and-qa-hardening.md) +Truth anchors: [HOST_ORCHESTRATOR_INTENDED_BEHAVIOR_TRUTH.md](/home/spenser/__Active_code/substrate/HOST_ORCHESTRATOR_INTENDED_BEHAVIOR_TRUTH.md:184), [AGENT_ORCHESTRATION_GAP_MATRIX.md](/home/spenser/__Active_code/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md:79), [docs/USAGE.md](/home/spenser/__Active_code/substrate/docs/USAGE.md:105), [llm-last-mile/README.md](/home/spenser/__Active_code/substrate/llm-last-mile/README.md:61) +Adjacent landed slices: [PLAN-22.md](/home/spenser/__Active_code/substrate/llm-last-mile/PLAN-22.md), [23-host-orchestrator-durable-session-and-parked-resumable-ownership.md](/home/spenser/__Active_code/substrate/llm-last-mile/23-host-orchestrator-durable-session-and-parked-resumable-ownership.md), [24-fix-host-bootstrap-readiness-and-clean-detach-parking.md](/home/spenser/__Active_code/substrate/llm-last-mile/24-fix-host-bootstrap-readiness-and-clean-detach-parking.md) +Post-landing follow-on: [26-async-persistent-session-bootstrap-readiness.md](/home/spenser/__Active_code/substrate/llm-last-mile/26-async-persistent-session-bootstrap-readiness.md), [PLAN.md](/home/spenser/__Active_code/substrate/PLAN.md), [ORCH_PLAN.md](/home/spenser/__Active_code/substrate/ORCH_PLAN.md) +Branch: `feat/host-orchestrator-durable-session` +Base branch: `main` +Plan type: closeout pass +Status: execution-ready planning pass on 2026-05-14; references refreshed against HEAD on 2026-05-16 + +Post-landing drift note: root planning moved to slice-26 async persistent-session bootstrap readiness after this plan landed. Keep `PLAN-25` scoped to durable host-session closeout and treat slice 26 as adjacent follow-on work, not part of this closeout pass. + +## Objective + +Close the durable host-session slice without reopening the product model. + +This plan is done only when all of the following are true: + +1. The repo states one unambiguous public recovery contract: + - `turn` is prompt-taking follow-up on the same durable session, + - `reattach` is attached-owner recovery only, + - `stop` is durable closeout for attached and parked host sessions, + - `status --json` reports durable session truth while the session is parked. +2. Parked and `awaiting_attention` host sessions stay visible on `substrate agent status --json` from authoritative session-root truth, not only from attached-live process truth. +3. One command-level lifecycle proof covers the same durable session through parked `status`, parked `turn`, `reattach`, and `stop`. +4. Docs stop overstating inbox reality. They must say what is shipped today and what is not shipped today. +5. Validation is complete enough that a future patch cannot silently regress the operator lifecycle or the contract wording. + +This is a closeout slice. It is not a new orchestration design and it is not inbox product expansion. + +## Why This Closeout Exists + +The branch has already landed most of the runtime behavior that the older SOW still treated as open. + +The remaining work is narrower and more important: + +1. The code, tests, usage docs, truth doc, gap matrix, and packet README still do not tell exactly the same story. +2. The runtime has real inbox persistence and posture math, but the repo still risks implying a finished operator-facing inbox workflow that the code does not prove. +3. Lifecycle QA is strong but fragmented. The branch still lacks one crisp closeout wall that proves the full operator path on one durable session. +4. `status` is already functionally landed, but it is a high-risk operator read surface. A regression there lies to operators even if the underlying session store is correct. + +The work here is boring on purpose. Freeze the contract, prove it end to end, and stop saying more than the code supports. + +## Plan Summary + +The runtime contract is already chosen on this branch: + +1. `substrate agent start --backend --prompt ... --json` is the public root prompt-taking surface. +2. `substrate agent turn --session --backend --prompt ... --json` is the public parked-session follow-up surface. +3. `substrate agent reattach --session --json` is the explicit attached-owner recovery surface. +4. `substrate agent stop --session --json` is the public closeout surface. +5. Parked host sessions are durable and routable. Detached-world follow-up remains fail closed until `reattach` restores an active host owner. + +What is not closed yet is the repo-wide explanation and the regression floor around that behavior. + +This plan therefore does four things, in order: + +1. freeze the already-chosen public contract in every truth surface, +2. add one command-level same-session lifecycle proof plus explicit parked-status assertions, +3. narrow inbox claims to the actual shipped surface, +4. publish one exact validation wall future patches must keep green. + +## Scope + +In scope: + +1. truth-doc convergence across usage docs, truth docs, gap matrix, and packet docs, +2. command-level lifecycle regression coverage for parked host `status`, `turn`, `reattach`, and `stop`, +3. command-level `status --json` assertions for `parked_resumable` and `awaiting_attention`, +4. code-comment tightening where the shipped inbox scope is currently easy to misread, +5. one exact manual validation path plus one exact automated validation wall. + +## NOT in scope + +This slice does not include: + +1. changing the public `turn` / `reattach` / `stop` split, +2. adding default-agent routing or fuzzy selectors, +3. adding a public inbox grammar such as `substrate agent inbox`, +4. adding a production runtime approval/completion/follow-up inbox workflow, +5. widening detached-world recovery beyond the current fail-closed contract, +6. redesigning the durable session model, daemon model, or state store, +7. Windows/WSL product-parity work. + +Anything in that list is a separate slice. This plan explicitly does not smuggle it in. + +## Locked Starting State + +### What already exists + +| Sub-problem | Existing code | Decision | +| --- | --- | --- | +| Public parked-session follow-up | [`run_turn(...)`](/home/spenser/__Active_code/substrate/crates/shell/src/execution/agents_cmd.rs:505) | Reuse. `turn` is already the public prompt-taking follow-up verb. | +| Public attached-owner recovery | [`run_reattach(...)`](/home/spenser/__Active_code/substrate/crates/shell/src/execution/agents_cmd.rs:565) | Reuse. `reattach` already exists and stays public in this slice. | +| Public closeout | [`run_stop(...)`](/home/spenser/__Active_code/substrate/crates/shell/src/execution/agents_cmd.rs:667) | Reuse. `stop` remains the canonical closeout path for attached and parked host sessions. | +| Authoritative parked-session read path | [`build_status_report(...)`](/home/spenser/__Active_code/substrate/crates/shell/src/execution/agents_cmd.rs:1333), [`list_status_sessions_for_agent(...)`](/home/spenser/__Active_code/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:630), [`status_visible_participants(...)`](/home/spenser/__Active_code/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:66) | Reuse and harden. Do not add a second status model. | +| Detached host continuity classification | [`classify_public_session_posture(...)`](/home/spenser/__Active_code/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:2191), [`valid_detached_host_continuity_posture(...)`](/home/spenser/__Active_code/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:2216) | Reuse exactly. This is the public continuity contract. | +| Durable inbox persistence and posture math | [`persist_inbox_item(...)`](/home/spenser/__Active_code/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:1039), [`apply_pending_inbox_count(...)`](/home/spenser/__Active_code/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:1726) | Reuse. Narrow the claims, not the implementation. | +| Dev-support runtime alert ingress | [`persist_runtime_alert_for_dev_support(...)`](/home/spenser/__Active_code/substrate/crates/shell/src/execution/agent_dev_support.rs:11) | Reuse and document honestly. Do not present this as a public product surface. | +| Existing lifecycle proof points | [`public_start_turn_and_stop_emit_streaming_ndjson_and_authoritative_state()`](/home/spenser/__Active_code/substrate/crates/shell/tests/agent_public_control_surface_v1.rs:1231), [`public_stop_cleanly_closes_same_durable_session_after_reattach()`](/home/spenser/__Active_code/substrate/crates/shell/tests/agent_public_control_surface_v1.rs:1631), [`public_start_persists_detached_session_when_hidden_owner_helper_exits()`](/home/spenser/__Active_code/substrate/crates/shell/tests/agent_public_control_surface_v1.rs:1880), [`detached_pending_inbox_normalizes_to_awaiting_attention()`](/home/spenser/__Active_code/substrate/crates/shell/tests/agent_public_control_surface_v1.rs:1980), [`public_turn_fail_closed_taxonomy_is_explicit_for_missing_backend_unknown_session_and_parent_slot_errors(...)`](/home/spenser/__Active_code/substrate/crates/shell/tests/agent_public_control_surface_v1.rs:2363), [`public_turn_fail_closed_taxonomy_is_explicit_for_world_linkage_ambiguity_and_detached_rejection()`](/home/spenser/__Active_code/substrate/crates/shell/tests/agent_public_control_surface_v1.rs:2492), [`public_turn_routes_linux_world_member_follow_up_through_typed_submit_path()`](/home/spenser/__Active_code/substrate/crates/shell/tests/agent_public_control_surface_v1.rs:2689) | Reuse. Add a cohesive closeout wall rather than inventing new semantics. | + +### Exact remaining gap + +The remaining gap is not a missing runtime model. The remaining gap is contract convergence and closeout rigor: + +1. Some truth docs still describe `reattach`, `stop`, and parked `status` as unfinished, even though the branch now ships them. +2. Inbox language is ahead of the proven runtime. Persistence exists. Posture math exists. A broad operator-facing inbox contract does not. +3. The branch has targeted lifecycle tests, but it still needs one same-session closeout proof that reads like the actual operator story. +4. The branch has store-level status visibility proof, but the command path still needs an explicit regression floor for parked and `awaiting_attention` rows. + +### Blast radius + +Even though the remaining work is narrow, the touched seams are high-consequence seams: + +1. `run_turn(...)`, `run_reattach(...)`, and `run_stop(...)` are top-level CLI contract surfaces. +2. `build_status_report(...)` is the operator truth surface for durable session visibility. +3. `persist_inbox_item(...)` is where documentation can drift far ahead of product reality because the persistence primitive looks more complete than the operator surface actually is. + +Implication: keep production code changes minimal, keep test additions explicit, and keep the docs exact. + +## Frozen Execution Contract + +If implementation wants a different contract, revise this plan first. Do not drift the behavior silently. + +### Public operator contract + +1. `substrate agent start --backend --prompt ... --json` remains the canonical public root prompt-taking surface and remains host-only in v1. +2. `substrate agent turn --session --backend --prompt ... --json` remains the canonical public follow-up surface. +3. `substrate agent reattach --session --json` remains attached-owner recovery only. It is not a prompt-taking alias. +4. `substrate agent stop --session --json` remains the durable closeout action for attached and parked host sessions. +5. `substrate agent status --json` remains the authoritative parked-session read surface for live-runtime rows, with `posture`, `attached_participant_id`, and `pending_inbox_count` sourced from durable session-root truth. +6. `parked_resumable` and `awaiting_attention` remain routable durable host postures. `terminal` remains the only non-routable posture family. +7. Detached-world follow-up remains fail closed until `reattach` restores an active host owner. +8. `substrate -c`, `--command`, and piped shell mode remain shell execution surfaces, not agent-prompt aliases. + +### Durable inbox contract + +This slice freezes the narrow, honest contract: + +1. Durable inbox persistence primitives are shipped. +2. `pending_inbox_count` plus posture normalization into `awaiting_attention` are shipped. +3. Internal ack/dismiss support is shipped. +4. Dev-support and test ingress are shipped. +5. A public operator-facing inbox command surface is not shipped. +6. A proven production runtime path for world-originated approval/completion/follow-up inbox items is not shipped as part of the supported public contract. +7. An automatic parked-session resume mechanism driven by inbox items is not shipped. + +### Contract diagram + +```text +parked host session + | + +--> agent status --json + | `--> read durable session-root truth + | + +--> agent turn --session --backend + | `--> submit a prompt against the same durable session/backend pair + | + +--> agent reattach --session + | `--> restore active_attached host ownership without submitting a prompt + | + `--> agent stop --session + `--> close the same durable session terminally + +detached world member + | + `--> agent turn ... while detached + `--> fail closed with reattach guidance +``` + +## Step 0: Scope Challenge + +### 0A. Minimum honest diff + +The minimum honest implementation is: + +1. update the truth docs so they stop claiming shipped behavior is unshipped, +2. add one same-session lifecycle regression that covers parked `status`, parked `turn`, `reattach`, and `stop`, +3. add explicit command-level parked-status assertions for both `parked_resumable` and `awaiting_attention`, +4. narrow inbox wording everywhere to the actually shipped surface, +5. publish one exact validation wall and one exact manual smoke path. + +Anything smaller leaves contradictions behind. Anything bigger expands scope into a new inbox feature. + +### 0B. Existing-code leverage + +This slice should build on the already-landed seams, not introduce parallel ones: + +1. extend [`agent_public_control_surface_v1.rs`](/home/spenser/__Active_code/substrate/crates/shell/tests/agent_public_control_surface_v1.rs), do not add a new integration harness, +2. keep the authoritative `status` path in `agents_cmd.rs` plus `state_store.rs`, +3. keep posture classification in `classify_public_session_posture(...)`, +4. keep inbox semantics where they are and document them correctly, +5. keep docs aligned to [`docs/USAGE.md`](/home/spenser/__Active_code/substrate/docs/USAGE.md:105) as the operator contract source. + +### 0C. Complexity check + +Expected touched areas: + +1. `crates/shell/tests/` +2. `crates/shell/src/execution/` +3. `docs/` +4. repo-root truth docs +5. `llm-last-mile/` + +That is a multi-file slice, but it is still the right size because: + +1. there are no new public verbs, +2. there is no new state model, +3. there is no new daemon or control plane, +4. almost all of the work is truth-sync and regression hardening, +5. any production code change should be tiny and test-driven. + +### 0D. Search and reuse conclusion + +This is a straight Layer 1 reuse exercise: + +1. reuse the existing command-level control suite, +2. reuse the existing live-status authority path, +3. reuse the existing detached continuity classifier, +4. reuse the existing docs as the operator contract source, +5. fix the contradiction, not the architecture. + +### 0E. TODO cross-reference + +There is no `TODOS.md` in the repo root today. + +That means all deferrals must stay explicit in the `NOT in scope` section of this plan. Do not assume a missing backlog file will preserve the reasoning. + +### 0F. Completeness check + +The complete version is still a lake: + +1. all truth surfaces agree, +2. the lifecycle is proven on one durable session, +3. parked and `awaiting_attention` rows are command-level regression protected, +4. inbox wording is exact, +5. validation is published. + +Trying to add a public inbox UX or broader runtime producer contract here would turn the lake into an ocean. This plan does not do that. + +## Architecture Review + +### Architecture thesis + +Do not redesign the durable host-session model. Freeze it and prove it. + +The architecture is already correct enough for v1: + +1. the durable session is the authority, +2. the attached host owner is an attachable execution client, not the identity of the session, +3. parked host state remains real state, +4. detached-world follow-up remains intentionally fail closed, +5. the operator contract is narrow and exact. + +### Current-to-target shape + +```text +CURRENT BRANCH SHAPE +==================== +runtime contract: mostly landed +docs contract: partially contradictory +qa contract: strong but fragmented +inbox wording: broader than the shipped operator surface + +TARGET CLOSEOUT SHAPE +===================== +runtime contract: unchanged +docs contract: one story everywhere +qa contract: one same-session lifecycle wall + explicit status assertions +inbox wording: exactly matches shipped reality +``` + +### Dependency graph + +```text +docs/USAGE.md + | + +--> repo-root truth docs + | +--> HOST_ORCHESTRATOR_INTENDED_BEHAVIOR_TRUTH.md + | `--> AGENT_ORCHESTRATION_GAP_MATRIX.md + | + `--> llm-last-mile docs + +--> README.md + +--> SOW 25 + `--> PLAN-25 + +crates/shell/src/execution/ + | + +--> agents_cmd.rs + | +--> run_turn(...) + | +--> run_reattach(...) + | +--> run_stop(...) + | `--> build_status_report(...) + | + +--> agent_runtime/state_store.rs + | +--> list_status_sessions_for_agent(...) + | +--> status_visible_participants(...) + | +--> persist_inbox_item(...) + | `--> classify_public_session_posture(...) + | + `--> agent_dev_support.rs + `--> persist_runtime_alert_for_dev_support(...) + +crates/shell/tests/ + | + `--> agent_public_control_surface_v1.rs + `--> command-level lifecycle and status contract wall +``` + +### Realistic production failure scenarios + +1. `status` regresses to trace-fallback null fields for a valid parked session. Operators think the session vanished or lost posture truth. +2. `reattach` returns success before durable attached truth is actually restored. Operators trust a fake success and the next command misbehaves. +3. `stop` still depends on the attached-live owner plane for a valid parked session. The session becomes impossible to close cleanly. +4. Docs advertise a richer inbox workflow than the code supports. Operators try a workflow that does not exist. + +This plan must close all four. + +## Code Quality Review + +### Boring-by-default rules + +1. One prompt-taking resume verb, `turn`. +2. One owner-recovery verb, `reattach`. +3. One closeout verb, `stop`. +4. One parked-session authority path, durable session-root truth. +5. One honest inbox story, no more and no less. + +### DRY guardrails + +1. Do not add a new status helper for tests. Test the real command path. +2. Do not add a second parked-session classifier for docs or assertions. Reuse the real posture logic. +3. Do not invent a fake production inbox producer just to make the docs sound nicer. +4. If comment tightening is needed, do it next to the real seams in `agent_dev_support.rs` or `state_store.rs`. + +### Engineered-enough boundary + +This slice should be explicit rather than clever: + +1. one new lifecycle regression is better than three new micro-tests that still leave the operator story fragmented, +2. one command-level parked-status assertion is better than a new helper or abstraction layer, +3. a doc narrowing diff is better than speculative runtime code that tries to make the docs true. + +### Diagram maintenance + +If any touched production file already has nearby contract comments or ASCII diagrams, update them in the same change. Stale diagrams are worse than no diagrams. + +Recommended comment targets if production code changes are needed: + +1. [`agents_cmd.rs`](/home/spenser/__Active_code/substrate/crates/shell/src/execution/agents_cmd.rs:505), around the `turn` / `reattach` / `stop` split, +2. [`state_store.rs`](/home/spenser/__Active_code/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:2191), around detached continuity and public posture classification, +3. [`agent_dev_support.rs`](/home/spenser/__Active_code/substrate/crates/shell/src/execution/agent_dev_support.rs:11), around dev-support-only inbox ingress. + +## Test Review + +### Test framework detection + +This is a Rust workspace. The relevant proof wall for this slice is `cargo test`, with shell integration tests carrying the operator contract and state-store tests carrying the durable posture math. + +Primary suites: + +1. [`crates/shell/tests/agent_public_control_surface_v1.rs`](/home/spenser/__Active_code/substrate/crates/shell/tests/agent_public_control_surface_v1.rs) +2. targeted `state_store.rs` tests for detached continuity and inbox posture math + +### Code path coverage + +```text +CODE PATH COVERAGE +=========================== +[+] Public durable host lifecycle + | + ├── [TESTED] start + streamed turn + authoritative stop + │ agent_public_control_surface_v1.rs:1231 + | + ├── [TESTED] detached parked start persists durable session + │ agent_public_control_surface_v1.rs:1880 + | + ├── [TESTED] pending inbox normalizes to awaiting_attention + │ agent_public_control_surface_v1.rs:1980 + | + ├── [TESTED] same-session stop after reattach + │ agent_public_control_surface_v1.rs:1631 + | + ├── [TESTED] exact world-member follow-up routing + │ agent_public_control_surface_v1.rs:2689 + | + ├── [TESTED] fail-closed taxonomy for unknown session, bad backend, detached world + │ agent_public_control_surface_v1.rs:2363, 2492 + | + ├── [GAP -> INTEG] one same-session lifecycle: + │ parked start -> status -> parked turn -> status -> reattach -> stop + | + └── [GAP -> INTEG] explicit command-level assertion that parked and + awaiting_attention rows expose live-runtime posture fields + +[+] Durable inbox persistence math + | + ├── [TESTED] pending count increments and posture becomes awaiting_attention + │ state_store.rs existing unit coverage + | + ├── [TESTED] ack/dismiss resolution updates pending count and posture + │ state_store.rs existing unit coverage + | + └── [GAP -> DOC HONESTY] no shipped public runtime producer beyond dev support +``` + +### Operator flow coverage + +```text +OPERATOR FLOW COVERAGE +=========================== +[+] start --backend --prompt ... + | + └── [TESTED] session can establish and park durably + +[+] status --json on that session + | + └── [GAP -> INTEG] parked and awaiting_attention command-level field assertions + +[+] turn --session --backend --prompt ... + | + └── [TESTED] exact follow-up resumes the durable session + +[+] reattach --session + | + └── [TESTED] attached-owner recovery works and remains same-session + +[+] stop --session + | + └── [TESTED] durable closeout works for valid parked or reattached host sessions + +[+] turn against detached world member + | + └── [TESTED] fail closed with reattach guidance +``` + +### Required test additions + +| Area | File | Required assertions | +| --- | --- | --- | +| Same-session lifecycle closeout | [`crates/shell/tests/agent_public_control_surface_v1.rs`](/home/spenser/__Active_code/substrate/crates/shell/tests/agent_public_control_surface_v1.rs) | parked `start` -> `status` -> parked `turn` -> `status` -> `reattach` -> `stop` on one orchestration-session id | +| Parked status authority | [`crates/shell/tests/agent_public_control_surface_v1.rs`](/home/spenser/__Active_code/substrate/crates/shell/tests/agent_public_control_surface_v1.rs) | parked rows show live-runtime `posture`, `attached_participant_id`, and `pending_inbox_count`, not trace-fallback nulls | +| Attention-needed status authority | [`crates/shell/tests/agent_public_control_surface_v1.rs`](/home/spenser/__Active_code/substrate/crates/shell/tests/agent_public_control_surface_v1.rs) | `awaiting_attention` rows show the same live-runtime fields | +| Detached-world non-regression | existing detached-world tests in `agent_public_control_surface_v1.rs` | detached-world follow-up still fails closed and still points operators to `reattach` | +| Inbox posture math | existing `state_store.rs` tests | keep pending-count and posture invariants green, add nothing that implies new product semantics | + +### Automated commands + +```bash +cargo test -p shell --test agent_public_control_surface_v1 -- --nocapture +cargo test -p shell agent_runtime::state_store -- --nocapture +cargo test --workspace -- --nocapture +``` + +### Test plan artifact + +Implementation should write the normal eng-review test artifact to: + +```text +~/.gstack/projects//-feat-host-orchestrator-durable-session-eng-review-test-plan-.md +``` + +Required contents: + +1. parked host `start` lifecycle, +2. parked `status` visibility, +3. parked host `turn` reuse of the same session id, +4. `reattach` recovery, +5. `stop` closeout from detached and reattached states, +6. detached-world fail-closed behavior, +7. explicit note that public inbox runtime producers are not part of the shipped operator contract yet. + +## Performance Review + +Runtime cost risk is low. Flake risk is real. + +Guardrails: + +1. extend existing fixture helpers, do not add new sleep-driven polling loops, +2. keep parked-status assertions store-backed and deterministic, +3. prefer authoritative runtime rows over broad trace scanning, +4. do not add a new smoke harness when the shell integration suite already models the lifecycle. + +The performance rule is simple: harden correctness without turning the closeout suite into an integration swamp. + +## Failure Modes Registry + +| Codepath | Realistic production failure | Test required | Error handling required | User-visible outcome | +| --- | --- | --- | --- | --- | +| parked `status` read | valid parked session regresses to trace-fallback null posture fields | yes | keep live-runtime rows authoritative | parked session stays visible and understandable | +| `awaiting_attention` status read | detached session with pending inbox work loses posture and count fields | yes | keep pending-count posture truth authoritative | operator sees pending attention state, not a ghost row | +| `reattach` success path | command reports success before durable attached truth is restored | yes | succeed only after durable attached proof | no fake reattach success | +| detached durable `stop` | stop still depends on attached-live control plane | yes | detached closeout remains authoritative | session stops cleanly | +| lifecycle truth docs | repo says `reattach`, `stop`, or parked `status` are unfinished | yes, via doc review | update truth docs | operator expectations stay aligned | +| inbox docs | repo implies a broader public inbox workflow than the code ships | yes, via doc review | narrow docs and comments | no false operator expectation | +| detached-world follow-up | later edits silently widen detached-world recovery | yes | preserve fail-closed routing and guidance | clear rejection, no unsafe fallback | + +Critical-gap rule for this plan: + +1. any regression that hides a valid parked or `awaiting_attention` session from `agent status --json` is a release blocker, +2. any fake `reattach` success is a release blocker, +3. any doc claim that advertises a public inbox workflow not backed by code is a release blocker. + +## Implementation Sequence + +### Step 1. Freeze repo truth first + +Files: + +1. [`HOST_ORCHESTRATOR_INTENDED_BEHAVIOR_TRUTH.md`](/home/spenser/__Active_code/substrate/HOST_ORCHESTRATOR_INTENDED_BEHAVIOR_TRUTH.md) +2. [`AGENT_ORCHESTRATION_GAP_MATRIX.md`](/home/spenser/__Active_code/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md) +3. [`docs/USAGE.md`](/home/spenser/__Active_code/substrate/docs/USAGE.md) +4. [`llm-last-mile/README.md`](/home/spenser/__Active_code/substrate/llm-last-mile/README.md) +5. [`llm-last-mile/25-host-durable-session-closeout-and-qa-hardening.md`](/home/spenser/__Active_code/substrate/llm-last-mile/25-host-durable-session-closeout-and-qa-hardening.md) + +Deliver: + +1. remove stale language that still calls parked `status`, `reattach`, or `stop` unfinished, +2. freeze the exact `turn` / `reattach` / `stop` contract everywhere, +3. narrow inbox claims to persisted scaffolding plus posture normalization plus dev-support/test ingress. + +Done means the repo has one story again before any new test work lands. + +### Step 2. Add one same-session lifecycle regression + +Files: + +1. [`crates/shell/tests/agent_public_control_surface_v1.rs`](/home/spenser/__Active_code/substrate/crates/shell/tests/agent_public_control_surface_v1.rs) + +Deliver: + +1. one lifecycle test covering parked `start` -> `status` -> parked `turn` -> `status` -> `reattach` -> `stop`, +2. stable orchestration-session id across every non-terminal step, +3. terminal closeout only after `stop`, +4. no change to the public command grammar. + +Done means the operator story is proven in one place instead of scattered across several tests. + +### Step 3. Add explicit parked-status assertions + +Files: + +1. [`crates/shell/tests/agent_public_control_surface_v1.rs`](/home/spenser/__Active_code/substrate/crates/shell/tests/agent_public_control_surface_v1.rs) +2. production files in `crates/shell/src/execution/` only if the new assertions expose a real mismatch + +Deliver: + +1. parked rows must expose live-runtime `posture`, `attached_participant_id`, and `pending_inbox_count`, +2. `awaiting_attention` rows must expose the same fields, +3. trace-fallback null behavior remains limited to true trace-only rows. + +Done means `status` is hardened as a real operator seam. + +### Step 4. Tighten inbox comments only if needed + +Files: + +1. [`crates/shell/src/execution/agent_dev_support.rs`](/home/spenser/__Active_code/substrate/crates/shell/src/execution/agent_dev_support.rs) +2. [`crates/shell/src/execution/agent_runtime/state_store.rs`](/home/spenser/__Active_code/substrate/crates/shell/src/execution/agent_runtime/state_store.rs) + +Deliver: + +1. make dev-support-only ingress obvious, +2. make persistence-vs-product-surface scope obvious, +3. add zero new runtime behavior. + +Done means code comments cannot be read as promising a public inbox feature that does not exist. + +### Step 5. Publish the validation wall + +Files: + +1. [`llm-last-mile/PLAN-25.md`](/home/spenser/__Active_code/substrate/llm-last-mile/PLAN-25.md) + +Deliver: + +1. exact automated command list, +2. exact manual host-lifecycle smoke path, +3. explicit note that `awaiting_attention` and detached-world fail-closed remain automated-only proofs because no public inbox producer surface exists and world-member reproduction is fixture-heavy. + +Done means future reviewers know exactly what must still work and what cannot honestly be claimed as a manual public flow. + +## Acceptance Criteria + +1. No repo truth surface says parked `status`, `reattach`, or `stop` are unfinished. +2. No repo truth surface implies a public inbox workflow beyond persisted scaffolding, posture normalization, and dev-support/test ingress. +3. One command-level lifecycle test proves parked `status`, parked `turn`, `reattach`, and `stop` on the same durable session id. +4. Command-level `status --json` assertions prove live-runtime fields for `parked_resumable` and `awaiting_attention`. +5. Detached-world follow-up remains fail closed with `reattach` guidance. +6. Existing lifecycle tests remain green. +7. Workspace tests stay green. +8. This plan contains the final validation wall and manual smoke rules. + +## Validation Matrix + +| Promise | Proof | +| --- | --- | +| parked durable host sessions stay visible on `status` | new command-level parked-status assertions plus existing store-level visibility tests | +| `awaiting_attention` remains authoritative | new command-level attention-needed assertions plus existing state-store posture math tests | +| same-session parked lifecycle remains stable through follow-up and closeout | new same-session lifecycle regression | +| `reattach` remains real attached-owner recovery | existing same-session stop-after-reattach coverage plus new lifecycle regression | +| detached parked `stop` remains canonical closeout | existing stop coverage plus new lifecycle regression | +| detached-world follow-up remains fail closed | existing detached-world fail-closed tests stay green | +| inbox posture math is real but public inbox workflow claims stay narrow | existing state-store tests plus doc/comment narrowing | +| repo truth matches code truth | doc diff review in the same PR | + +## Manual Validation + +### Exact manual smoke path + +The manual smoke path for this slice is host-only. It must be runnable without special internal inbox or world-member fixtures. + +```bash +substrate agent start --backend --prompt "hello" --json +substrate agent status --json +substrate agent turn --session --backend --prompt "next" --json +substrate agent status --json +substrate agent reattach --session --json +substrate agent stop --session --json +substrate agent status --json +``` + +Manual expectations: + +1. `start` creates one durable orchestration session and the prompt-driven host client can exit without invalidating it. +2. the first `status --json` call shows the session as `parked_resumable` if `pending_inbox_count == 0`. +3. `turn` reuses the same orchestration-session id and does not require `reattach` first. +4. the second `status --json` call still shows the same session with live-runtime posture fields populated. +5. `reattach` restores attached ownership without submitting a prompt. +6. `stop` closes the same durable session terminally. +7. the final `status --json` call no longer presents the session as a live non-terminal durable session. + +### Automated-only checks + +The following remain automated-only on purpose: + +1. `awaiting_attention` command-level visibility, because there is no public inbox producer surface to create that state manually without dev-support/test ingress. +2. detached-world fail-closed reproduction, because the retained world-member setup is fixture-heavy and the branch already treats the shell integration suite as the authoritative proof surface. + +Automated expectations: + +1. detached-world failure output still points operators to `substrate agent reattach --session `, +2. `awaiting_attention` rows still expose live-runtime fields at the command layer, +3. no test adds fake public inbox behavior just to make a manual demo path prettier. + +## Worktree Parallelization Strategy + +This slice has one real code lane and one real docs lane. It does not justify splitting the shell control suite across multiple workers. + +### Dependency table + +| Step | Modules touched | Depends on | +| --- | --- | --- | +| Truth-doc convergence | `docs/`, repo root docs, `llm-last-mile/` | frozen contract in this plan | +| Lifecycle QA hardening | `crates/shell/tests/`, maybe tiny `crates/shell/src/execution/` fixes | frozen contract in this plan | +| Inbox-scope comment tightening | `crates/shell/src/execution/`, maybe `docs/` | truth-doc convergence | +| Final validation wall | `llm-last-mile/` | truth-doc convergence, lifecycle QA hardening | + +### Parallel lanes + +Lane A: lifecycle QA hardening +Owns `crates/shell/tests/agent_public_control_surface_v1.rs` and any tiny supporting runtime fix it exposes. + +Lane B: truth-doc convergence +Owns `docs/`, repo-root truth docs, and packet docs. Do not edit Rust files in this lane unless Lane A is finished. + +Lane C: inbox-scope comments plus final validation wall +Runs after Lane A and Lane B converge. Small cleanup lane. + +### Execution order + +1. Freeze the contract in this plan. +2. Launch Lane A and Lane B in parallel worktrees. +3. Merge Lane A first if any runtime fix is needed. +4. Rebase Lane B if necessary and merge it next. +5. Run Lane C last for comment tightening and final validation publication. + +### Conflict flags + +1. Do not split `agent_public_control_surface_v1.rs` across multiple workers. That file is one hotspot and will just create merge noise. +2. If Lane B wants to edit comments in `state_store.rs` or `agent_dev_support.rs`, wait until Lane A settles. Both lanes may need those files. +3. Keep Lane B doc-only unless a doc contradiction cannot be resolved without a tiny code comment fix. + +### Parallelization verdict + +Three lanes total: + +1. one code lane, +2. one docs lane, +3. one short cleanup lane after both converge. + +If only one engineer is working the slice, execute sequentially in the same order and do not over-optimize for parallelism. + +## Completion Summary + +- Step 0: Scope Challenge, accepted as-is. The runtime contract is already chosen; the remaining work is closeout and truth convergence. +- Architecture Review: reuse the landed durable-session model and harden the contract seams around `status`, `reattach`, `stop`, and inbox wording. +- Code Quality Review: one public recovery model, one status authority path, one honest inbox story. +- Test Review: coverage diagram produced, two real test gaps identified, same-session lifecycle proof and command-level parked-status proof. +- Performance Review: low runtime-cost risk, moderate flake risk if new tests add custom polling or sleep-driven behavior. +- NOT in scope: written. +- What already exists: written. +- Failure modes: release blockers frozen around status visibility, fake reattach success, detached stop regression, and overstated inbox claims. +- Parallelization: 3 lanes, 2 can start independently, 1 waits for convergence. +- Lake Score: the complete option is to finish truth sync and closeout proof now, not ship another cycle of contradictory docs. + +## Completion Checklist + +- [ ] truth docs no longer say parked `status`, `reattach`, or `stop` are unfinished +- [ ] inbox docs say exactly what is shipped today and nothing more +- [ ] one same-session lifecycle regression proves parked `status`, parked `turn`, `reattach`, and `stop` +- [ ] command-level parked `status` rows show live-runtime posture fields +- [ ] command-level `awaiting_attention` rows show live-runtime posture fields +- [ ] detached-world fail-closed guidance remains green +- [ ] existing lifecycle tests stay green +- [ ] workspace test wall passes +- [ ] manual host lifecycle validation sequence is published in this plan +- [ ] automated-only checks are explicitly called out where no honest manual public flow exists + +## Done Means + +This slice is done when the durable host-session story stops depending on tribal knowledge. + +An operator should be able to read the docs, run `status`, `turn`, `reattach`, and `stop`, and get exactly the behavior the repo claims. + +A reviewer should be able to inspect one lifecycle regression, one validation matrix, and one manual smoke path and know the contract is real. + +And nobody should walk away thinking the inbox product is more complete than the code actually proves. diff --git a/llm-last-mile/README.md b/llm-last-mile/README.md index 57e2c4a6d..81afc9e12 100644 --- a/llm-last-mile/README.md +++ b/llm-last-mile/README.md @@ -1,12 +1,28 @@ # LLM Last Mile SOW Packet -This directory contains the reviewed statement-of-work packet for the prerequisites that should be completed before the planned live registry step: +This directory contains the reviewed statement-of-work packet for the prerequisites that led into the landed session-centric live registry step: -- planned registry target: - - `~/.substrate/run/agent-sessions/.json` - - registry wraps UAA handles with Substrate-owned metadata +- landed registry shape: + - `~/.substrate/run/agent-hub/sessions//session.json` + - `~/.substrate/run/agent-hub/sessions//participants/.json` + - registry wraps UAA handles with Substrate-owned orchestration metadata keyed by `orchestration_session_id` and `participant_id` - `trace.jsonl` remains audit history, not live state authority +## Current Status + +- `03-shared-world-ownership-linux-first` is the landed Linux-first interface/backend slice: + shared-world requests now use explicit request/response proof shapes, and Linux owner-bound reuse lives in `crates/world`. +- `04-thread-world-binding-into-runtime-state` is the runtime-state bridge: + authoritative projection of the active shared-world binding is the handoff that slice `05` invalidation semantics and slice `06` session-centric store work both consume. +- `05-restart-invalidation-semantics` is still pending: + invalidation/replacement registry semantics for generation changes and restart handling remain a follow-on slice. +- `07-world-replacement-ordering-rollback-atomic-metadata` is the landed backend-hardening reference: + shared-world replacement now has explicit ordering, rollback, cleanup, and atomic metadata expectations that later shell/runtime slices can rely on. +- `08` through `10` are the remaining follow-on SOWs after the replacement-ordering and replacement-member work: + they cover event-emission authority cleanup, live-state authority/cutover cleanup, and the missing production member-launch seam. +- `11` through `13` are addenda after the original packet: + they cover the host↔world member transport cutover, the gateway auth-bundle last-mile security slice, and the final true in-world member-placement hardening pass. + ## Reviewed Execution Order 1. [01-orchestration-session-identity.md](./01-orchestration-session-identity.md) @@ -17,10 +33,21 @@ This directory contains the reviewed statement-of-work packet for the prerequisi - Make shared-world ownership explicit on Linux and bind one active `world_id` to one `orchestration_session_id`. 4. [04-thread-world-binding-into-runtime-state.md](./04-thread-world-binding-into-runtime-state.md) - Persist authoritative `world_id` and `world_generation` in runtime state instead of relying on trace/event history. + - Bridge role: this is the runtime-state handoff that slice `05` restart invalidation and slice `06` session-centric state store changes build on. 5. [05-restart-invalidation-semantics.md](./05-restart-invalidation-semantics.md) - Define the live-state rule that a generation change invalidates all prior-generation world-scoped member sessions. + - Status: still pending after the slice `04` runtime-state bridge. 6. [06-session-centric-state-store.md](./06-session-centric-state-store.md) - Rework the runtime handle store APIs and layout so consumers resolve by orchestration session instead of by orchestrator agent heuristics. + - Landed runtime output now writes canonical session roots under `agent-hub/sessions//...` with per-session participant records keyed by `participant_id`. +7. [07-world-replacement-ordering-rollback-atomic-metadata.md](./07-world-replacement-ordering-rollback-atomic-metadata.md) + - Harden shared-world replacement ordering, rollback, cleanup, and atomic Linux world metadata writes so exactly one recoverable active world remains authoritative. +8. [08-explicit-orchestration-authority-event-emission.md](./08-explicit-orchestration-authority-event-emission.md) + - Remove ambient PID-based orchestration identity lookup from shell-owned event emission and require explicit runtime-owned context instead. +9. [09-live-state-authority-and-compatibility-cutover.md](./09-live-state-authority-and-compatibility-cutover.md) + - Freeze canonical session-root records as live-state authority, bound compatibility writes/reads during cutover, and keep torn-root control-plane selection fail-closed while read-only status can degrade to warnings. +10. [10-member-runtime-launch-seam.md](./10-member-runtime-launch-seam.md) + - Add the missing production world-scoped member runtime launch/lifecycle seam that consumes the existing authority and status machinery. ## Review Notes @@ -29,9 +56,35 @@ This directory contains the reviewed statement-of-work packet for the prerequisi - The packet assumes the existing shell-owned host orchestrator runtime remains the starting point. - The packet is ordered so that session identity and participant modeling land before world ownership, invalidation, and store reshaping rely on them. +## Follow-On Plans + +The original packet stops before public control-plane productization. The follow-on planning documents for that work are: + +- [PLAN-19.md](./PLAN-19.md) + - Publicize the narrow `substrate agent start|reattach|fork|stop` control plane with exact session selectors, `reattach` as exact attached-owner recovery for the named durable session, and `stop` as the canonical closeout path for attached or parked durable sessions. +- [ORCH_PLAN-19.md](./ORCH_PLAN-19.md) + - Parent-frozen execution controller for the `PLAN-19` rollout on `feat/session-centric-state-store`. +- [PLAN-20.md](./PLAN-20.md) + - Add the public prompt-taking caller surface with exact `start` / `turn` verbs, canonical `reattach`, backend-aware turn resolution, and helper-owned streaming. +- [ORCH_PLAN-20.md](./ORCH_PLAN-20.md) + - Parent-frozen execution controller for the `PLAN-20` rollout on `feat/session-centric-state-store`. +- [PLAN-21.md](./PLAN-21.md) + - Extend the shared-owner/member-runtime contract onto the supported macOS/Lima forwarded path without changing the Linux source-of-truth placement model. +- [ORCH_PLAN-21.md](./ORCH_PLAN-21.md) + - Parent-frozen execution controller for the `PLAN-21` rollout on the shared-owner/member-runtime parity branch. +- [PLAN-22.md](./PLAN-22.md) + - Harden the public `turn` surface around the already-landed `start|turn|reattach|fork|stop` contract, prove Linux world-member follow-up through the typed submit boundary, and pin explicit fail-closed coverage. +- [ORCH_PLAN-22.md](./ORCH_PLAN-22.md) + - Parent-frozen execution controller for the `PLAN-22` rollout on the authoritative shared-owner/member-runtime parity branch. +- [23-host-orchestrator-durable-session-and-parked-resumable-ownership.md](./23-host-orchestrator-durable-session-and-parked-resumable-ownership.md) + - Landed lifecycle correction: durable host orchestration sessions now survive clean attached-client exit, retain session-local inbox work under `sessions//inbox/`, let `status` surface `parked_resumable` and `awaiting_attention`, treat successful `reattach` as durable attached truth for that exact session, keep `stop` as the canonical closeout path for attached or parked durable sessions, and reserve non-routable meaning for `terminal` only. +- [24-fix-host-bootstrap-readiness-and-clean-detach-parking.md](./24-fix-host-bootstrap-readiness-and-clean-detach-parking.md) + - Hardening follow-on: host bootstrap and detach parking now preserve the exact durable-session fields that later `turn`, `reattach`, `stop`, and `status --json` rely on. +- [25-host-durable-session-closeout-and-qa-hardening.md](./25-host-durable-session-closeout-and-qa-hardening.md) + - Truth-convergence and QA-hardening follow-on: freeze `turn` as prompt-taking follow-up on the same durable session, `reattach` as attached-owner recovery only, `stop` as durable closeout for attached and parked host sessions, `status --json` as the authoritative parked-session read surface, detached-world follow-up as fail-closed until `reattach`, and inbox behavior as narrow retained state rather than a public inbox workflow. + ## Non-Goals in This Packet -- Full `agent start/resume/fork/stop` CLI productization -- Auth-bundle / `SUBSTRATE_LLM_AUTH_BUNDLE_FD` work +- Full agent caller-surface parity beyond the landed narrow `start|turn|reattach|fork|stop` contract - macOS and Windows shared-world parity - Renaming the local `agent-api-*` crates diff --git a/llm-last-mile/WORKTREE_ASSIGNMENT_WAVE_0.md b/llm-last-mile/WORKTREE_ASSIGNMENT_WAVE_0.md new file mode 100644 index 000000000..4f163150d --- /dev/null +++ b/llm-last-mile/WORKTREE_ASSIGNMENT_WAVE_0.md @@ -0,0 +1,64 @@ +# Worktree Assignment Plan — Wave 0 + +Status: planning document. This wave is decision-only and docs-only. It exists to reduce ambiguity before parallel implementation waves begin. + +## Objective + +Lock the naming and boundary decisions that affect later streams without introducing cross-cutting code churn during active implementation. + +This wave exists to answer: + +- whether `uaa.agent.session` remains the local protocol-family label, +- whether the local `agent-api-*` crate names are acceptable for now, +- what should be treated as docs-only clarification now, +- and what should be deferred to a later rename cleanup stream. + +## Stream + +### Stream 5a: Naming decision only + +- Worktree: `codex/sow-5-decision` +- Type: docs-only +- Goal: decide the naming posture for external `agent_api` versus local `agent-api-*`, and decide whether `uaa.agent.session` stays or is scheduled for later rename + +## File Ownership + +This stream owns only docs and decision artifacts: + +- [AGENT_ORCHESTRATION_GAP_MATRIX.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/AGENT_ORCHESTRATION_GAP_MATRIX.md:1) +- [docs/project_management/adrs/draft/ADR-0044-agent-hub-core-successor-identity-tuple-compatible.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/project_management/adrs/draft/ADR-0044-agent-hub-core-successor-identity-tuple-compatible.md:1) +- [docs/project_management/adrs/draft/ADR-0045-orchestration-toolbox-internal-mcp-identity-trace-contract.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/project_management/adrs/draft/ADR-0045-orchestration-toolbox-internal-mcp-identity-trace-contract.md:1) +- [docs/TRACE.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/TRACE.md:1) + +## Deliverables + +- one explicit decision on `uaa.agent.session` +- one explicit decision on whether local `agent-api-*` names are: + - acceptable for now, + - scheduled for later cleanup, + - or blocked on a rename before further rollout +- repo-facing wording that distinguishes: + - external `agent_api` / Unified Agent API + - local host/world transport crates +- a short follow-on note describing whether a future rename should be: + - docs-only, + - code + docs, + - or full crate/package migration + +## Non-Goals + +- no code symbol renames +- no crate renames +- no protocol-label migration in trace or runtime code +- no public-surface behavior change + +## Merge Policy + +- Merge this wave before or alongside Wave 1, but keep it docs-only. +- Do not let this wave expand into a broad refactor. + +## Exit Criteria + +- downstream wave owners can reference one settled naming decision +- no implementation stream needs to guess whether `uaa.agent.session` is canonical, transitional, or slated for removal + diff --git a/llm-last-mile/WORKTREE_ASSIGNMENT_WAVE_1.md b/llm-last-mile/WORKTREE_ASSIGNMENT_WAVE_1.md new file mode 100644 index 000000000..eaa768439 --- /dev/null +++ b/llm-last-mile/WORKTREE_ASSIGNMENT_WAVE_1.md @@ -0,0 +1,96 @@ +# Worktree Assignment Plan — Wave 1 + +Status: first implementation wave. These streams can run in parallel because they touch mostly different runtime boundaries. + +## Objective + +Land the next two highest-leverage slices in parallel: + +- operator recovery and session-remediation ergonomics on the shell/control plane +- inventory-backed integrated gateway backend realization on the gateway/world-gateway path + +## Streams + +### Stream 4: Session selector and remediation ergonomics + +- Worktree: `codex/sow-4-session-remediation` +- Goal: improve operator recovery without changing the public session handle contract +- Public handle remains `orchestration_session_id` + +#### Owns + +- [crates/shell/src/execution/agent_runtime/state_store.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/state_store.rs:1) +- [crates/shell/src/execution/agent_runtime/control.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agent_runtime/control.rs:1) +- [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:1) +- [crates/shell/src/execution/cli.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/cli.rs:1) +- [crates/shell/tests/agent_public_control_surface_v1.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_public_control_surface_v1.rs:1) +- [crates/shell/tests/agent_successor_contract_ahcsitc0.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/tests/agent_successor_contract_ahcsitc0.rs:1) +- [docs/USAGE.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/USAGE.md:1) + +#### Intended outcomes + +- better current-session discovery +- clearer stale-owner and torn-state diagnostics +- bounded repair/reap/invalidate flows where appropriate +- no expansion of public handles beyond `orchestration_session_id` + +#### Non-goals + +- no tuple policy work +- no gateway adapter realization work +- no toolbox server implementation + +### Stream 6: Inventory-backed integrated gateway backend realization + +- Worktree: `codex/sow-6-gateway-backend-realization` +- Goal: land the ADR-0046 runtime slice and remove the hardcoded integrated `cli:codex` realization path while keeping Codex as the regression floor + +#### Owns + +- [crates/world-agent/src/gateway_runtime.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/gateway_runtime.rs:1) +- [crates/shell/src/builtins/world_gateway.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/builtins/world_gateway.rs:1) +- [crates/gateway/docs/contracts/chatgpt-codex-auth-handoff-contract.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/gateway/docs/contracts/chatgpt-codex-auth-handoff-contract.md:1) +- [crates/gateway/docs/contracts/chatgpt-codex-conformance-and-drift-guard.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/gateway/docs/contracts/chatgpt-codex-conformance-and-drift-guard.md:1) +- [crates/gateway/docs/contracts/chatgpt-codex-route-contract.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/gateway/docs/contracts/chatgpt-codex-route-contract.md:1) +- [docs/project_management/adrs/draft/ADR-0046-gateway-backend-selection-runtime-integration.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/project_management/adrs/draft/ADR-0046-gateway-backend-selection-runtime-integration.md:1) +- integrated gateway/world-agent tests + +#### Intended outcomes + +- adapter binding resolved by inventory/backend id +- explicit capability gating per integrated backend +- adapter-driven runtime config rendering +- backend-aware auth handoff +- at least one path beyond hardcoded `cli:codex`, without regressing Codex + +#### Non-goals + +- no tuple-axis policy +- no toolbox server +- no shell session-remediation UX + +## Parallelism Rule + +These two streams may proceed in parallel. + +The intended boundary is: + +- Stream 4 owns shell/operator recovery behavior +- Stream 6 owns gateway/world-gateway/backend-realization behavior + +## Known Conflict Hotspots + +- documentation overlap in [docs/USAGE.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/USAGE.md:1) if Stream 6 needs operator docs +- possible shared references to backend-id wording in ADR/docs + +When in doubt: + +- Stream 4 owns user-facing session-control wording +- Stream 6 owns integrated gateway/backend wording + +## Exit Criteria + +- Stream 4 lands without changing backend realization semantics +- Stream 6 lands without inventing new public session-control semantics +- both merge cleanly and establish the dependency floor for Wave 2 + diff --git a/llm-last-mile/WORKTREE_ASSIGNMENT_WAVE_2.md b/llm-last-mile/WORKTREE_ASSIGNMENT_WAVE_2.md new file mode 100644 index 000000000..547f12248 --- /dev/null +++ b/llm-last-mile/WORKTREE_ASSIGNMENT_WAVE_2.md @@ -0,0 +1,111 @@ +# Worktree Assignment Plan — Wave 2 + +Status: second implementation wave. This wave starts after Wave 1, especially after backend/adaptor realization semantics from Stream 6 are stable. + +## Objective + +Build the next contract layer on top of the Wave 1 baseline: + +- tuple-axis policy and provenance +- tuple projection on status/trace surfaces +- the actual internal toolbox MCP server, still read-only + +## Start Condition + +Do not begin full Wave 2 implementation until Stream 6 is settled enough that backend ids, adapter realization behavior, and integrated auth/runtime expectations are no longer moving targets. + +## Streams + +### Stream 7a: Tuple-axis policy and broker/schema + +- Worktree: `codex/sow-7a-tuple-policy` +- Goal: add the additive tuple-axis policy layer under `llm.constraints.*` + +#### Owns + +- [docs/project_management/adrs/draft/ADR-0042-llm-and-agent-identity-tuple-and-deployment-posture.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/project_management/adrs/draft/ADR-0042-llm-and-agent-identity-tuple-and-deployment-posture.md:1) +- [docs/project_management/adrs/draft/ADR-0043-adr-0027-identity-tuple-policy-surface.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/project_management/adrs/draft/ADR-0043-adr-0027-identity-tuple-policy-surface.md:1) +- broker/config/policy model files under `crates/broker` +- relevant config/policy plumbing under `crates/shell/src/execution` +- policy/config tests + +#### Intended outcomes + +- `llm.constraints.routers` +- `llm.constraints.providers` +- `llm.constraints.protocols` +- `llm.constraints.auth_authorities` +- `--explain` provenance for the new policy surface + +#### Non-goals + +- no status rendering changes +- no toolbox server implementation + +### Stream 7b: Tuple projection on status and trace + +- Worktree: `codex/sow-7b-tuple-status-trace` +- Goal: project the accepted tuple vocabulary consistently into status and trace surfaces + +#### Owns + +- [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:1) +- [docs/TRACE.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/TRACE.md:1) +- [docs/USAGE.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/USAGE.md:1) +- trace/status tests in `crates/shell/tests` and `crates/common/tests` + +#### Depends on + +- field names and semantics from Stream 7a +- backend realization behavior from Stream 6 + +#### Non-goals + +- no new policy keys +- no toolbox server implementation + +### Stream 8: Internal toolbox MCP server, still read-only + +- Worktree: `codex/sow-8-toolbox-server` +- Goal: turn today’s `toolbox status|env` introspection surface into the actual host-scoped internal MCP service, while keeping v1 read-only + +#### Owns + +- [docs/project_management/adrs/draft/ADR-0045-orchestration-toolbox-internal-mcp-identity-trace-contract.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/project_management/adrs/draft/ADR-0045-orchestration-toolbox-internal-mcp-identity-trace-contract.md:1) +- toolbox implementation files under `crates/shell/src/execution` or a new toolbox module +- [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:1372) for CLI wiring if needed +- toolbox trace emission and `tool_call_id` joinability + +#### Shared contract dependency + +This stream must align with the tuple and trace vocabulary settled by Streams 7a and 7b. + +#### Non-goals + +- no mutation tools +- no router daemon behavior + +## Parallelism Rule + +The safest execution shape is: + +- Stream 7a begins first in Wave 2 +- Stream 7b and Stream 8 may proceed in parallel after the tuple field names and basic semantics are settled + +## Conflict Hotspots + +- [crates/shell/src/execution/agents_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/agents_cmd.rs:1) + - Stream 7b owns tuple/status projection + - Stream 8 owns toolbox CLI wiring only +- [docs/TRACE.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/TRACE.md:1) + - Stream 7b should be the final editor once field names settle +- [docs/USAGE.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/USAGE.md:1) + - batch final doc merge late in the wave + +## Exit Criteria + +- tuple policy keys are real and explainable +- tuple status/trace vocabulary is visible and coherent +- toolbox server exists as a read-only host-scoped MCP surface +- no stream in this wave invents a second execution plane + diff --git a/llm-last-mile/WORKTREE_ASSIGNMENT_WAVE_3.md b/llm-last-mile/WORKTREE_ASSIGNMENT_WAVE_3.md new file mode 100644 index 000000000..fc4387a20 --- /dev/null +++ b/llm-last-mile/WORKTREE_ASSIGNMENT_WAVE_3.md @@ -0,0 +1,63 @@ +# Worktree Assignment Plan — Wave 3 + +Status: third implementation wave. This wave is intentionally more serialized because it depends on the contracts stabilized in the earlier waves. + +## Objective + +Build the first real router-daemon implementation on top of stabilized gateway, tuple, trace, and toolbox contracts. + +## Stream + +### Stream 9: Router daemon v1 + +- Worktree: `codex/sow-9-router-daemon` +- Goal: implement the ADR-0029 trace-driven router with durable queueing, workspace registry, idempotent triggers, and policy re-evaluation + +## Owns + +- [docs/project_management/adrs/draft/ADR-0029-host-event-bus-and-router-daemon.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/project_management/adrs/draft/ADR-0029-host-event-bus-and-router-daemon.md:1) +- router daemon/runtime code +- durable inbox/work-queue or equivalent request-queue implementation +- cursor and dedupe state +- workspace registry and target-workspace resolution +- router-derived event tests and policy re-evaluation tests + +## Depends On + +- Stream 6 for stable backend realization semantics +- Stream 7a for tuple/policy vocabulary +- Stream 7b for trace/status field projection and correlation consistency +- Stream 8 for the toolbox/control-plane read contract if router interactions rely on it + +## Parallelism Rule + +This wave should be treated as mostly solo implementation. + +You may run small exploratory or scaffolding work in parallel, but the real router implementation should not start until the earlier contracts are merged and stable. + +## Non-Goals + +- no workflow-engine or forge composition +- no mutation expansion of the toolbox +- no reopening of accepted trace/config foundations +- no redefinition of backend-id semantics + +## Risks + +- contract drift if router event families are implemented before tuple/trace naming is stable +- hidden coupling to pre-stable toolbox behavior +- idempotency and cause-reference bugs if derived-event join keys are invented ad hoc + +## Merge Policy + +- Merge only after Wave 2 is materially complete. +- Keep this wave focused on router v1 itself, not on workflow composition. + +## Exit Criteria + +- router daemon has durable request semantics +- trigger evaluation is trace-driven and policy-rechecked +- workspace targeting is explicit +- derived events use the accepted correlation vocabulary +- the repo is ready for later workflow-engine composition work without reopening router foundations + diff --git a/macos-hardening/README.md b/macos-hardening/README.md index d8fc41fbf..86b06c935 100644 --- a/macos-hardening/README.md +++ b/macos-hardening/README.md @@ -4,38 +4,54 @@ Status: Draft Owner: Substrate world backend / platform security -Last updated: 2026-04-28 +Last updated: 2026-05-19 ## Purpose This directory is the planning root for the macOS hardening program described in -[thoughts/shared/research/2026-04-28-macos-lima-parity-lockdown.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/thoughts/shared/research/2026-04-28-macos-lima-parity-lockdown.md). +[the research note](./research/2026-04-28-macos-lima-parity-lockdown.md). It separates two product tracks that share some runtime foundations but have different security claims: - `macos-hardened-same-user-lima/`: improve behavioral parity with Linux and materially reduce bypass in the current same-user Lima model. - `macos-ownership-separated-world/`: redesign the macOS control plane so the - VM and its transport are no longer effectively owned by the invoking user. + VM, forwarding, and gateway lifecycle path are no longer effectively owned by + the invoking user. ## Why This Split Exists The current macOS backend can be improved substantially without changing the host ownership model, but that work does not satisfy the stronger requirement -that only Substrate can access the VM. The research note and current repo state -show that these are distinct outcomes: +that only Substrate controls the VM and its lifecycle path. The research note +and current repo state show that these are distinct outcomes: - Functional parity and hardening gaps live in runtime/backend surfaces such as - [crates/world-mac-lima/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/lib.rs:1), - [crates/world-mac-lima/src/forwarding.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/forwarding.rs:1), - [crates/world-mac-lima/src/transport.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/transport.rs:1), - and [crates/shell/src/execution/routing/dispatch/exec.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/exec.rs:699). -- Ownership-boundary gaps live in host lifecycle, installer, Lima ownership, - transport reachability, and operator workflow surfaces such as - [scripts/mac/lima-warm.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/lima-warm.sh:665), - [docs/WORLD.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/WORLD.md:136), - [docs/cross-platform/mac_world_setup.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/cross-platform/mac_world_setup.md:82), - and [scripts/substrate/install-substrate.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/substrate/install-substrate.sh:1854). + `crates/world-mac-lima/src/lib.rs`, + `crates/world-mac-lima/src/forwarding.rs`, + `crates/world-mac-lima/src/transport.rs`, and + `crates/shell/src/execution/routing/dispatch/exec.rs`. +- Same-user operator surfaces already exist for world and gateway lifecycle: + `scripts/substrate/world-enable.sh`, + `scripts/substrate/dev-install-substrate.sh`, + `scripts/substrate/install-substrate.sh`, + `scripts/substrate/uninstall-substrate.sh`, + `substrate host doctor`, + `substrate world doctor`, + `substrate health`, and + `substrate world gateway sync|status|restart`. +- Ownership-boundary gaps still live in host lifecycle, Lima ownership, + transport reachability, and support workflow surfaces such as + `scripts/mac/lima-warm.sh`, + `docs/WORLD.md`, + `docs/contracts/substrate-gateway-operator-contract.md`, + `docs/contracts/substrate-gateway-policy-evaluation.md`, and + `docs/contracts/substrate-gateway-status-schema.md`. +- Managed gateway runtime artifacts under + `/run/substrate/substrate-gateway-runtime/` already matter in the supported + operator story. Ownership separation must eventually cover how those + artifacts, lifecycle logs, and brokered status flows are exposed without + preserving same-user Lima control. ## Program Structure @@ -58,29 +74,42 @@ Each feature directory contains: The two tracks are related but not interchangeable. - The same-user track is the prerequisite for a credible hardened macOS story. - It resolves transport drift, policy drift, mount overexposure, unit drift, - and direct-guest operational shortcuts. + It resolves transport drift, policy drift, mount overexposure, gateway + runtime parity, unit drift, and direct-guest operational shortcuts. - The ownership-separated track depends on the same-user track for transport and - operational clarity, but then replaces the host ownership model with a daemon- - owned or service-account-owned control plane. + operational clarity, but then replaces the host ownership model with a + daemon-owned or service-account-owned control plane. - Work in the ownership-separated track should not reintroduce any direct guest admin path as part of the normal operator contract. +- `docs/WORLD.md` remains architecture evidence and descriptive context. It is + not the contract owner for gateway lifecycle/status or policy semantics. + Those durable contracts live under `docs/contracts/` and must stay the source + of truth when these plans discuss operator/status behavior. ## Required Evidence Sources All SOWs in this tree should stay anchored to the current repo and research inputs: -- [thoughts/shared/research/2026-04-28-macos-lima-parity-lockdown.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/thoughts/shared/research/2026-04-28-macos-lima-parity-lockdown.md) -- [docs/WORLD.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/WORLD.md) -- [docs/cross-platform/mac_world_setup.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/cross-platform/mac_world_setup.md) -- [crates/world-mac-lima/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/lib.rs) -- [crates/world-mac-lima/src/forwarding.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/forwarding.rs) -- [crates/world-mac-lima/src/transport.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/transport.rs) -- [scripts/mac/lima-warm.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/lima-warm.sh) -- [scripts/mac/lima/substrate.yaml](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/lima/substrate.yaml) -- [scripts/mac/smoke.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/smoke.sh) -- [scripts/mac/lima-doctor.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/lima-doctor.sh) +- [Research note](./research/2026-04-28-macos-lima-parity-lockdown.md) +- [`../docs/WORLD.md`](../docs/WORLD.md) +- [`../docs/contracts/substrate-gateway-operator-contract.md`](../docs/contracts/substrate-gateway-operator-contract.md) +- [`../docs/contracts/substrate-gateway-policy-evaluation.md`](../docs/contracts/substrate-gateway-policy-evaluation.md) +- [`../docs/contracts/substrate-gateway-status-schema.md`](../docs/contracts/substrate-gateway-status-schema.md) +- [`../docs/contracts/substrate-gateway-runtime-parity.md`](../docs/contracts/substrate-gateway-runtime-parity.md) +- `crates/world-mac-lima/src/lib.rs` +- `crates/world-mac-lima/src/forwarding.rs` +- `crates/world-mac-lima/src/transport.rs` +- `crates/shell/src/builtins/world_gateway.rs` +- `crates/world-agent/src/gateway_runtime.rs` +- `scripts/mac/lima-warm.sh` +- `scripts/mac/lima/substrate.yaml` +- `scripts/mac/smoke.sh` +- `scripts/mac/lima-doctor.sh` +- `scripts/substrate/world-enable.sh` +- `scripts/substrate/dev-install-substrate.sh` +- `scripts/substrate/install-substrate.sh` +- `scripts/substrate/uninstall-substrate.sh` ## Exit Condition @@ -90,4 +119,8 @@ This planning tree is complete when: - every milestone has a concrete SOW - dependencies between same-user hardening and ownership separation are explicit - acceptance criteria and validation evidence are defined per milestone +- the plans distinguish clearly between: + - current same-user world/gateway lifecycle surfaces that already exist + - unresolved ownership, mount, and direct-guest gaps that still block the + stronger macOS claim - cross-document consistency has been reviewed after the draft wave lands diff --git a/macos-hardening/macos-hardened-same-user-lima/README.md b/macos-hardening/macos-hardened-same-user-lima/README.md index d0c654554..fc85b4b8e 100644 --- a/macos-hardening/macos-hardened-same-user-lima/README.md +++ b/macos-hardening/macos-hardened-same-user-lima/README.md @@ -4,33 +4,42 @@ Status: Draft Owner: Substrate world backend / macOS hardening -Last updated: 2026-04-28 +Last updated: 2026-05-19 ## Purpose / outcome -Define the hardening program for the current macOS Lima-backed world backend without pretending it can match Linux's ownership boundary. The outcome of this feature is a documented target mode that preserves Linux-like execution semantics where possible, explicitly names the irreducible same-user limitations, and stages the repo changes required to stop treating direct guest administration and permissive transports as normal operation. +Define the hardening program for the current macOS Lima-backed world backend without pretending it can match Linux's ownership boundary. The target mode keeps the already-landed Substrate operator surfaces as the baseline, names the irreducible same-user limitations, and sequences the remaining repo work needed to make the supported path narrower and clearer. ## Why this feature exists -The current macOS path is functionally close to Linux, but its security and operator contract are still parity-by-assertion rather than parity-by-enforcement. +The current macOS path is much closer to Linux parity than these docs currently admit: -- `crates/world-mac-lima/src/lib.rs` injects a synthetic permissive `PolicySnapshotV3` and leaves `apply_policy` as a no-op. -- `crates/world-mac-lima/src/forwarding.rs` and `crates/world-mac-lima/src/transport.rs` disagree on TCP endpoints (`17788` vs `7788`) and still carry fallback logic that is not the hardened default. -- `scripts/mac/lima/substrate.yaml` mounts all of `$HOME` read-only and leaves guest service administration patterns closer to ad hoc VM management than to Substrate-owned lifecycle control. -- `docs/WORLD.md` and `docs/cross-platform/mac_world_setup.md` still normalize `limactl shell`, direct `systemctl`, and direct guest probes as routine operator paths. +- `substrate host doctor` and `substrate world doctor` already exist as canonical health surfaces. +- `substrate world gateway sync`, `substrate world gateway status`, and `substrate world gateway restart` already exist as canonical gateway lifecycle/status surfaces. +- Shared-world/orchestration support already exists on the Lima-backed path, including explicit shared-owner proof handling and macOS orchestration smoke coverage. -Phase 0 exists to freeze the security contract before implementation work starts, because the codebase currently mixes three different stories: +The remaining macOS problem is no longer "there is no Substrate-owned surface." It is that the hardened contract is still incomplete in several concrete places: -- Linux-like operator semantics. -- Same-user Lima convenience shortcuts. -- Breakglass guest administration that is documented as normal. +- `crates/world-mac-lima/src/lib.rs` still synthesizes a permissive `PolicySnapshotV3` in `MacLimaBackend::convert_exec_request` and leaves `apply_policy` as a no-op on the backend-mediated Lima path. +- The shell-side routed request builders already resolve and forward `policy_snapshot`, `world_network`, and `world_fs_mode` on direct routed paths, so the parity gap must be scoped to `MacLimaBackend` and other backend-mediated flows rather than described as a repo-wide absence of policy propagation. +- `scripts/mac/lima/substrate.yaml` still mounts broad host state, including a read-only `$HOME` mount that is wider than the hardened runtime contract should require. +- `scripts/mac/lima-warm.sh` still writes `SUBSTRATE_AGENT_TCP_PORT=61337`, leaving an extra guest listener enabled by default. +- Guest unit/socket definitions still drift between `scripts/mac/lima/substrate.yaml` and `scripts/mac/lima-warm.sh`. +- `docs/WORLD.md` and `docs/cross-platform/mac_world_setup.md` still normalize direct guest setup and troubleshooting more than the hardened same-user path should. +- `SUBSTRATE_WORLD_SOCKET` is still available as an advanced/test/breakglass bypass, but it is not the standard Lima-backed operator path and already rejects explicit shared-owner reuse on macOS. + +Phase 0 exists to lock that support contract before implementation work continues, because the repo still mixes three different stories: + +- a real Substrate-owned CLI surface that already exists +- same-user Lima implementation shortcuts that are still tolerated +- direct guest administration that is still too prominent in docs and troubleshooting ## In-scope - Define the supported target mode for hardened same-user Lima. - Document which Linux properties are expected to match and which cannot match under the current same-user VM model. -- Define the breakglass boundary for direct guest administration and manual transport debugging. -- Establish milestone sequencing for later code, docs, and validation work. +- Define the boundary between supported operation, degraded-but-supported operation, and breakglass operation. +- Establish milestone sequencing for backend, provisioning, and docs changes that remain open. ## Out-of-scope @@ -43,69 +52,79 @@ Phase 0 exists to freeze the security contract before implementation work starts The feature proceeds contract-first. -1. Phase 0 locks the target mode, support contract, version floor inputs, and breakglass rules. -2. Later phases will converge implementation to that contract: - - transport and policy semantics in `crates/world-mac-lima` - - provisioning and unit ownership in `scripts/mac/lima*` - - operator surface reduction in `docs/WORLD.md`, `docs/cross-platform/mac_world_setup.md`, and CLI-owned workflows -3. Validation shifts from "manual `limactl` can make it work" to "Substrate-owned entry points prove the supported mode, and anything else is breakglass." - -The key design constraint is explicit: same-user Lima can match Linux execution behavior and guest-local socket ACLs, but it cannot provide the same host-side ownership boundary while the host user can always drive `limactl` for that VM. +1. Phase 0 locks the target mode, support posture, version-floor assumptions, and breakglass rules. +2. Phase 1 converges backend-mediated runtime behavior with the already-landed shell/operator surfaces: + - transport behavior in `crates/world-mac-lima` + - backend policy application semantics for Lima-backed execution + - doctor/smoke evidence for the supported path +3. Phase 2 narrows the same-user Lima attack surface: + - remove default extra TCP exposure + - minimize broad host mounts + - unify guest unit/socket definitions +4. Phase 3 finishes the operator-surface cutover: + - teach CLI-owned doctor/gateway/lifecycle flows first + - reclassify direct guest setup and troubleshooting as breakglass + +The key design constraint is explicit: same-user Lima can match much of Linux's execution behavior, guest-local socket contract, gateway lifecycle surface, and shared-world/orchestration flow shape, but it still cannot provide Linux's host-side ownership boundary while the same host user can drive `limactl` for that VM. ## Dependencies / sequencing - Phase 0 is the gate for all later hardening work. -- Milestone 0.1 defines target mode and support posture. -- Milestone 0.2 defines Lima version assumptions and breakglass classification. -- Later implementation phases should not change transport defaults, guest mounts, policy application, or operator docs until phase-0 decisions are accepted. +- Later phases should treat `substrate host doctor`, `substrate world doctor`, and `substrate world gateway sync|status|restart` as baseline surfaces to harden around, not as future inventions. +- Later implementation phases should not broaden the operator contract beyond what the current CLI already owns; they should narrow Lima-specific bypasses and direct guest dependencies instead. ## Concrete repo surfaces and file pointers - Backend contract: `crates/world-mac-lima/src/lib.rs` - Forwarding and endpoint selection: `crates/world-mac-lima/src/forwarding.rs`, `crates/world-mac-lima/src/transport.rs` +- Shell-side routed request builders and policy/world input propagation: + `crates/shell/src/execution/routing/dispatch/world_ops.rs`, + `crates/shell/src/repl/async_repl.rs`, + `crates/shell/src/builtins/world_gateway.rs` - Lima profile and guest units: `scripts/mac/lima/substrate.yaml` -- Provisioning and readiness workflow: `scripts/mac/lima-warm.sh`, `scripts/mac/lima-doctor.sh`, `scripts/mac/smoke.sh` -- Operator-facing architecture and setup guidance: `docs/WORLD.md`, `docs/cross-platform/mac_world_setup.md` -- Phase-0 planning docs: `macos-hardening/macos-hardened-same-user-lima/phase-0-security-contract-and-scope/` +- Provisioning and readiness workflow: `scripts/mac/lima-warm.sh`, `scripts/mac/lima-doctor.sh`, `scripts/mac/smoke.sh`, `scripts/mac/orchestration-smoke.sh` +- Operator-facing architecture and setup guidance: `docs/WORLD.md`, `docs/cross-platform/mac_world_setup.md`, `docs/USAGE.md`, `docs/contracts/substrate-gateway-operator-contract.md` +- Phase overviews: `macos-hardening/macos-hardened-same-user-lima/phase-0-security-contract-and-scope/` ## Deliverables - This feature overview. -- Four phase overview documents that sequence the work from contract-setting - through operator-surface replacement. -- Ten milestone SOWs covering: - - target mode and support contract - - Lima version and breakglass contract - - transport contract unification - - policy application parity - - doctor and smoke parity - - agent surface hardening - - mount minimization and ingress definition - - guest service sandboxing and unit unification - - Substrate-managed diagnostics and lifecycle - - breakglass reclassification and docs cutover +- Four phase overview documents that sequence the work from contract-setting through operator-surface replacement. +- Milestone SOWs that focus later work on the remaining gaps: + - same-user Lima ownership boundary + - backend-mediated policy parity + - broad `$HOME` mount reduction + - default `SUBSTRATE_AGENT_TCP_PORT=61337` removal + - YAML versus warm-script unit convergence + - direct guest setup/troubleshooting demotion to breakglass ## Acceptance criteria - The feature docs state plainly that Linux ownership-boundary parity is not achievable in the current same-user Lima model. -- The docs distinguish supported operation, degraded operation, and breakglass operation. -- The docs identify the exact repo surfaces that later implementation phases must change. -- The docs sequence future work so transport, policy, provisioning, and operator docs converge on one contract rather than separate local fixes. +- The docs treat `substrate host doctor`, `substrate world doctor`, and `substrate world gateway sync|status|restart` as already-landed baseline surfaces. +- The docs scope policy-synthesis concerns to `MacLimaBackend` and backend-mediated Lima paths, while acknowledging that shell-side routed request builders already carry resolved policy/world inputs. +- The docs describe shared-world/orchestration support as current functionality, not speculative future work. +- The docs identify `SUBSTRATE_WORLD_SOCKET` as an advanced/test/breakglass bypass rather than the normal Lima-backed path. ## Validation / evidence plan - Cross-check every claimed gap against the cited repo surfaces before phase promotion. -- Use the phase-0 README and milestone SOWs as the review set for security, backend, and docs owners. +- Use the current CLI and smoke surfaces as the baseline evidence set: + - `substrate host doctor --json` + - `substrate world doctor --json` + - `substrate world gateway status --json` + - `scripts/mac/smoke.sh` + - `scripts/mac/orchestration-smoke.sh` - Treat this feature as ready for implementation only when reviewers can answer three questions from the docs alone: - What is the supported same-user mode? - What is explicitly not promised relative to Linux? - - Which current workflows become breakglass? + - Which current workflows are normal operation versus breakglass? ## Risks / open questions -- The repo currently documents and tests workflows that bypass the intended socket-activation boundary; unwinding those habits will require coordinated code and docs changes. -- A minimum supported Lima version is not yet frozen here; that decision is delegated to milestone 0.2 because it changes what transport and guest-management paths can be required. -- If the project later needs true host-side multi-user isolation on macOS, this feature will only be an intermediate hardening step, not the final security model. +- The repo already has real CLI-owned surfaces, so the main risk is overstating the amount of missing architecture and then widening scope unnecessarily. +- A minimum supported Lima version is not yet frozen here; that decision affects which forwarding and guest-management paths can be required. +- If the project later needs true host-side multi-user isolation on macOS, this feature remains an intermediate hardening step rather than the final security model. ## Phase Index diff --git a/macos-hardening/macos-hardened-same-user-lima/phase-0-security-contract-and-scope/README.md b/macos-hardening/macos-hardened-same-user-lima/phase-0-security-contract-and-scope/README.md index c0085f04c..0fc85dedc 100644 --- a/macos-hardening/macos-hardened-same-user-lima/phase-0-security-contract-and-scope/README.md +++ b/macos-hardening/macos-hardened-same-user-lima/phase-0-security-contract-and-scope/README.md @@ -1,35 +1,37 @@ # Phase 0: Security Contract and Scope Status: Draft -Last updated: 2026-04-28 +Last updated: 2026-05-19 ## Purpose / outcome -Freeze the security and support contract for hardened same-user Lima before any implementation phases change code or operator workflows. The outcome of phase 0 is one unambiguous planning baseline for what macOS should guarantee, what it cannot guarantee, which workflows are supported, and which existing guest-administration paths are reclassified as breakglass. +Freeze the security and support contract for hardened same-user Lima before later phases change backend behavior, provisioning, or operator docs. The outcome of phase 0 is one unambiguous planning baseline for what macOS should guarantee, what it cannot guarantee, which already-landed CLI surfaces are the supported baseline, and which existing direct guest workflows are breakglass only. ## Why this phase exists -The current macOS backend is already close enough to Linux behavior that implementation changes are tempting, but the repo still lacks a locked contract for the following disagreements: +The current macOS backend is already close enough to Linux behavior that implementation changes are tempting, but the repo still lacks a locked contract for several disagreements: -- The code claims Linux-like policy semantics while `MacLimaBackend` still synthesizes a permissive policy snapshot in `crates/world-mac-lima/src/lib.rs`. -- Transport selection is not fully coherent across `crates/world-mac-lima/src/forwarding.rs` and `crates/world-mac-lima/src/transport.rs`. +- The repo already has canonical operator surfaces in `substrate host doctor`, `substrate world doctor`, and `substrate world gateway sync|status|restart`, but the hardening docs still talk as if those surfaces are mostly future work. +- The code still claims Linux-like policy semantics while `MacLimaBackend` synthesizes a permissive policy snapshot in `crates/world-mac-lima/src/lib.rs`. +- The shell-side routed request builders already forward resolved `policy_snapshot`, `world_network`, and `world_fs_mode`, so the unresolved policy gap must be scoped to backend-mediated Lima behavior and bootstrap/ensure-session paths. - The Lima profile and setup docs still normalize broad host mounts and manual guest administration. -- Doctor and smoke workflows validate "it can be made to work" more than "it is operating in the supported hardened mode." +- The repo already supports shared-world/orchestration flows on the Lima-backed path, while `SUBSTRATE_WORLD_SOCKET` remains an advanced/test/breakglass bypass rather than the normal same-user operator path. -If phase 0 is skipped, later work risks hardening one layer while another layer keeps the old, more permissive support story. +If phase 0 is skipped, later work risks hardening one layer while another layer keeps the older, looser support story. ## In-scope - Define the target security posture for same-user Lima. - Define the support boundary versus Linux parity claims. -- Define breakglass operator rules. +- Define the baseline CLI/operator surfaces that are already supported. +- Define breakglass operator rules, including the status of `SUBSTRATE_WORLD_SOCKET` and direct `limactl shell` procedures. - Define the versioning and environment assumptions that later implementation can rely on. - Produce milestone SOWs that later code phases can execute against. ## Out-of-scope - Editing runtime code, provisioning scripts, or top-level docs outside this phase directory. -- Closing the current transport, policy, or mount gaps in this phase. +- Closing the current transport, policy, mount, or docs gaps in this phase. - Replacing Lima, redesigning the entire macOS backend, or solving multi-user host isolation. ## Architectural approach @@ -40,12 +42,14 @@ Phase 0 uses two planning milestones: - names the supported mode - names the non-goals - freezes the Linux-versus-macOS claims we will and will not make + - treats existing doctor/gateway/shared-world surfaces as baseline 2. `milestone-0-2-lima-version-and-breakglass-contract-sow.md` - defines required Lima/runtime assumptions - defines which direct guest workflows remain available only for breakglass - - defines which Substrate-owned workflows must become the normal operator path + - defines how `SUBSTRATE_WORLD_SOCKET` is classified + - defines which Substrate-owned workflows must remain the normal operator path -This phase is complete when later implementation can treat the phase docs as authoritative inputs instead of rediscovering policy, transport, and ops intent from scattered code comments and setup guides. +This phase is complete when later implementation can treat the phase docs as authoritative inputs instead of rediscovering support posture from scattered code comments and setup guides. ## Dependencies / sequencing @@ -53,23 +57,28 @@ This phase is complete when later implementation can treat the phase docs as aut - `crates/world-mac-lima/src/lib.rs` - `crates/world-mac-lima/src/forwarding.rs` - `crates/world-mac-lima/src/transport.rs` + - `crates/shell/src/execution/routing/dispatch/world_ops.rs` + - `crates/shell/src/repl/async_repl.rs` + - `crates/shell/src/builtins/world_gateway.rs` - `scripts/mac/lima-warm.sh` - `scripts/mac/lima/substrate.yaml` - `docs/WORLD.md` - `docs/cross-platform/mac_world_setup.md` - `scripts/mac/lima-doctor.sh` - `scripts/mac/smoke.sh` + - `scripts/mac/orchestration-smoke.sh` - Milestone 0.1 must land conceptually before milestone 0.2, because the version and breakglass contract depend on the chosen target mode. -- No later hardening phase should change default transport, guest mounts, or operator guidance until both milestone SOWs are accepted. +- No later hardening phase should describe doctor/gateway/shared-world support as future-only work after phase 0 is accepted. ## Concrete repo surfaces and file pointers - Policy snapshot injection and backend policy no-op: `crates/world-mac-lima/src/lib.rs` - Forwarding endpoint and transport fallback behavior: `crates/world-mac-lima/src/forwarding.rs`, `crates/world-mac-lima/src/transport.rs` -- Lima guest image, mount, and unit definitions: `scripts/mac/lima/substrate.yaml` +- Shell-side routed request/world input propagation: `crates/shell/src/execution/routing/dispatch/world_ops.rs`, `crates/shell/src/repl/async_repl.rs`, `crates/shell/src/builtins/world_gateway.rs` +- Lima guest image, mounts, and unit definitions: `scripts/mac/lima/substrate.yaml` - Provisioning and lifecycle commands: `scripts/mac/lima-warm.sh` -- Troubleshooting and smoke entry points: `scripts/mac/lima-doctor.sh`, `scripts/mac/smoke.sh` -- Operator-facing macOS setup and world architecture docs: `docs/cross-platform/mac_world_setup.md`, `docs/WORLD.md` +- Troubleshooting and smoke entry points: `scripts/mac/lima-doctor.sh`, `scripts/mac/smoke.sh`, `scripts/mac/orchestration-smoke.sh` +- Operator-facing macOS setup and world architecture docs: `docs/cross-platform/mac_world_setup.md`, `docs/WORLD.md`, `docs/USAGE.md` ## Deliverables @@ -81,18 +90,24 @@ This phase is complete when later implementation can treat the phase docs as aut ## Acceptance criteria - The phase documents define one supported same-user Lima posture and reject vague "Linux parity except where different" language. -- The phase documents identify the exact repo surfaces that encode the current mismatched behavior. -- The phase documents distinguish supported operator flows from breakglass flows. +- The phase documents treat `substrate host doctor`, `substrate world doctor`, and `substrate world gateway sync|status|restart` as already-landed operator surfaces. +- The phase documents distinguish supported flows from breakglass flows, including `SUBSTRATE_WORLD_SOCKET` and direct guest administration. +- The phase documents scope unresolved policy parity claims to `MacLimaBackend` and backend-mediated Lima paths. - The milestone ordering is clear enough that a future implementation owner can start with milestone 0.1 decisions and then execute 0.2 without reopening scope. ## Validation / evidence plan - Review the phase docs against the cited repo surfaces and confirm each claimed gap is observable today. - Require sign-off from backend, security, and docs owners before phase completion. -- Use later implementation-phase kickoffs to reference the phase-0 decisions by file path and section, not by verbal summary. +- Use current CLI/smoke/orchestration surfaces as the baseline evidence set: + - `substrate host doctor --json` + - `substrate world doctor --json` + - `substrate world gateway status --json` + - `scripts/mac/smoke.sh` + - `scripts/mac/orchestration-smoke.sh` ## Risks / open questions -- Some breakglass workflows are currently embedded in normal setup docs, so reclassification will create short-term friction for operators. -- The right Lima version floor may force changes in transport assumptions, especially around `vsock-proxy` availability and SSH config stability. +- Some breakglass workflows are still embedded in normal setup docs, so reclassification will create short-term friction for operators. +- The right Lima version floor may force changes in forwarding assumptions, especially around `vsock-proxy` availability and SSH config stability. - The project may need an ADR later if phase-0 decisions materially narrow the macOS support matrix. diff --git a/macos-hardening/macos-hardened-same-user-lima/phase-0-security-contract-and-scope/milestone-0-1-target-mode-and-support-contract-sow.md b/macos-hardening/macos-hardened-same-user-lima/phase-0-security-contract-and-scope/milestone-0-1-target-mode-and-support-contract-sow.md index 4182c06b9..9b22e9acc 100644 --- a/macos-hardening/macos-hardened-same-user-lima/phase-0-security-contract-and-scope/milestone-0-1-target-mode-and-support-contract-sow.md +++ b/macos-hardening/macos-hardened-same-user-lima/phase-0-security-contract-and-scope/milestone-0-1-target-mode-and-support-contract-sow.md @@ -1,7 +1,7 @@ # Milestone 0.1: Target Mode and Support Contract SOW Status: Draft -Last updated: 2026-04-28 +Last updated: 2026-05-19 ## Purpose / outcome @@ -9,11 +9,27 @@ Define the authoritative target mode for hardened same-user Lima and freeze the ## Why this milestone exists -Today the macOS backend mixes functional parity goals with security claims it cannot fully uphold. - -- `crates/world-mac-lima/src/lib.rs` presents Linux-like execution flow but still injects a permissive synthetic policy snapshot and does not apply backend policy after session setup. -- `docs/WORLD.md` describes socket ACLs and platform-hidden divergence, but does not sharply distinguish guest-local ACL parity from host-side ownership-boundary parity. -- `docs/cross-platform/mac_world_setup.md` still teaches direct `limactl shell` and direct guest `systemctl` as standard setup and validation steps. +Today the macOS backend mixes partially landed parity with security and support +claims that are still too broad. + +- `crates/world-mac-lima/src/lib.rs` already supports routed execution and + shared-world session reuse, but `MacLimaBackend` still synthesizes a + permissive backend-local `PolicySnapshotV3` and `apply_policy(...)` is still + effectively just `fs_mode` storage. +- Shell-side direct request builders already resolve and carry + broker-derived `policy_snapshot`, `world_network`, and world-fs enforcement + inputs in `crates/shell/src/execution/policy_snapshot.rs`, + `crates/shell/src/execution/routing/dispatch/world_ops.rs`, and + `crates/shell/src/builtins/world_gateway.rs`; the overstatement is specifically + in the backend-mediated macOS path, not in every shell path. +- Canonical operator surfaces already exist: + `substrate host doctor`, `substrate world doctor`, and + `substrate world gateway sync|status|restart`. + The remaining gap is to make those the supported contract first and classify + direct guest administration and host-side overrides accordingly. +- `docs/WORLD.md` and `docs/cross-platform/mac_world_setup.md` still do not + sharply separate guest-local Linux-like behavior from the unresolved same-user + host ownership limitation. Without a target-mode contract, later fixes will remain local patches instead of a coherent hardening program. @@ -40,14 +56,26 @@ This milestone should lock the contract around these rules: - One macOS host user owns the Substrate process, the Lima VM lifecycle, and the host-side forwarding artifacts. - The world-agent still runs inside Linux and keeps guest-local socket ACL semantics (`root:substrate 0660`) inside the guest. - Substrate-owned commands, not direct `limactl`, are the supported control plane. + - The supported control plane already includes `substrate host doctor`, + `substrate world doctor`, and + `substrate world gateway sync|status|restart`; later phases harden and cut + over around those surfaces rather than inventing new baseline commands. 2. Linux parity claims we keep - In-world execution remains the default. - Guest-local `world-agent` behavior, socket-activation shape, and policy evaluation intent must converge toward Linux semantics. - PTY and non-PTY routing should become transport-agnostic from the shell's point of view. + - Shared-world/orchestration support remains part of the supported runtime + story; hardening must preserve the existing `shared_world` request/binding + path instead of treating it as an unsupported corner case. + - Gateway lifecycle is a first-class operator and support surface, not an + optional add-on. 3. Linux claims we explicitly do not make - Host-side authorization is not equivalent to Linux multi-user socket ownership while the same host user can drive `limactl`. - Same-user Lima is not a privilege boundary against the owning host user. - Direct guest administration is not part of normal operation. + - Host-side `SUBSTRATE_WORLD_SOCKET` override use is not the default supported + Lima path; it is an advanced/test/breakglass escape hatch that bypasses the + normal forwarded socket discovery contract. ## Dependencies / sequencing @@ -62,10 +90,18 @@ This milestone should lock the contract around these rules: ## Concrete repo surfaces and file pointers -- Synthetic policy snapshot and backend no-op: +- Backend-local policy synthesis and weak policy application: - `crates/world-mac-lima/src/lib.rs` - - `convert_exec_request` - - `apply_policy` + - `crates/world-mac-lima/src/lib.rs:416` + - `crates/world-mac-lima/src/lib.rs:604` +- Shell-side already-landed policy and gateway request builders: + - `crates/shell/src/execution/policy_snapshot.rs` + - `crates/shell/src/execution/routing/dispatch/world_ops.rs` + - `crates/shell/src/builtins/world_gateway.rs` +- Existing operator/gateway contract surfaces: + - `crates/shell/src/execution/platform/macos.rs` + - `docs/contracts/substrate-gateway-operator-contract.md` + - `docs/contracts/substrate-gateway-status-schema.md` - Guest lifecycle and mount posture: - `scripts/mac/lima-warm.sh` - `scripts/mac/lima/substrate.yaml` @@ -82,8 +118,10 @@ This milestone should lock the contract around these rules: compatibility wrappers around them, not raw direct guest procedures - breakglass / unsupported direct guest mode - A gap list that future implementation must close, at minimum: - - remove permissive synthetic backend policy behavior + - remove backend-local policy synthesis from `MacLimaBackend` + - strengthen `apply_policy(...)` semantics in the backend-mediated path - stop documenting direct guest administration as normal + - classify host-side `SUBSTRATE_WORLD_SOCKET` override usage as advanced/test/breakglass - narrow host exposure and forwarding defaults to the supported mode ## Acceptance criteria @@ -93,18 +131,29 @@ This milestone should lock the contract around these rules: - The milestone names which Linux semantics remain implementation goals: - world-agent-first execution - guest-local socket activation and ACL discipline - - Linux-like backend policy semantics + - Linux-like backend policy semantics for the backend-mediated execution path - transport-agnostic PTY and non-PTY behavior + - supported shared-world/orchestration behavior + - gateway lifecycle/status as a normal support surface - The milestone names which semantics are explicitly non-goals: - host-side multi-user socket authorization parity - security isolation from the owning host user - routine reliance on direct `limactl shell` administration + - treating `SUBSTRATE_WORLD_SOCKET` override as the normal macOS Lima path - The milestone includes enough file pointers that an implementation owner can trace every target gap back to code or docs. ## Validation / evidence plan -- Read `crates/world-mac-lima/src/lib.rs` and confirm the current synthetic policy snapshot and `apply_policy` no-op are captured accurately in this SOW. -- Read `docs/WORLD.md` and `docs/cross-platform/mac_world_setup.md` and confirm the SOW correctly identifies where operator language currently over-normalizes direct guest management. +- Read `crates/world-mac-lima/src/lib.rs` and confirm the current synthetic + policy snapshot, `shared_world` passthrough, and weak `apply_policy(...)` + semantics are captured accurately in this SOW. +- Read `crates/shell/src/execution/policy_snapshot.rs`, + `crates/shell/src/execution/routing/dispatch/world_ops.rs`, and + `crates/shell/src/builtins/world_gateway.rs` and confirm the SOW correctly + narrows the policy-drift claim to the backend-mediated macOS path. +- Read `docs/WORLD.md` and `docs/cross-platform/mac_world_setup.md` and confirm + the SOW correctly identifies where operator language currently + over-normalizes direct guest management and host-side overrides. - Use milestone review to force a binary decision on this statement: - "macOS same-user Lima can match Linux behavior in the guest, but not Linux host authorization semantics." - Treat any attempt to keep ambiguous parity language as a phase-0 review defect. diff --git a/macos-hardening/macos-hardened-same-user-lima/phase-0-security-contract-and-scope/milestone-0-2-lima-version-and-breakglass-contract-sow.md b/macos-hardening/macos-hardened-same-user-lima/phase-0-security-contract-and-scope/milestone-0-2-lima-version-and-breakglass-contract-sow.md index 16c937b30..7d3e698fd 100644 --- a/macos-hardening/macos-hardened-same-user-lima/phase-0-security-contract-and-scope/milestone-0-2-lima-version-and-breakglass-contract-sow.md +++ b/macos-hardening/macos-hardened-same-user-lima/phase-0-security-contract-and-scope/milestone-0-2-lima-version-and-breakglass-contract-sow.md @@ -1,7 +1,7 @@ # Milestone 0.2: Lima Version and Breakglass Contract SOW Status: Draft -Last updated: 2026-04-28 +Last updated: 2026-05-19 ## Purpose / outcome @@ -9,12 +9,26 @@ Freeze the environment assumptions and operator escape hatches for hardened same ## Why this milestone exists -Current macOS behavior still depends on tooling and workflows that are only partly under Substrate's control. - -- `scripts/mac/lima/substrate.yaml` assumes `vmType: "vz"`, guest systemd, and a mount model that still exposes all of `$HOME` read-only. -- `scripts/mac/lima-warm.sh`, `scripts/mac/lima-doctor.sh`, and `scripts/mac/smoke.sh` still rely heavily on direct `limactl shell` execution. -- `crates/world-mac-lima/src/forwarding.rs` and `crates/world-mac-lima/src/transport.rs` encode transport assumptions that are not yet one hardened story. -- `docs/cross-platform/mac_world_setup.md` teaches direct guest build, install, service enablement, and raw guest probing as standard setup. +Current macOS behavior still depends on tooling and workflows that are only +partly under Substrate's control, but some of the intended control plane is now +already landed. + +- `scripts/mac/lima/substrate.yaml` assumes `vmType: "vz"`, guest systemd, and + a mount model that still exposes all of `$HOME` read-only. +- `scripts/mac/lima-warm.sh`, `scripts/mac/lima-doctor.sh`, and + `scripts/mac/smoke.sh` still rely heavily on direct `limactl shell` + execution even though canonical CLI doctors already exist. +- `crates/world-mac-lima/src/forwarding.rs` already routes to the guest UDS + endpoint `/run/substrate.sock` and intentionally skips SSH TCP fallback; + remaining transport drift is stale `7788` references plus doctor/probe + fallback use of host TCP `17788`. +- `crates/shell/src/builtins/world_gateway.rs`, + `docs/contracts/substrate-gateway-operator-contract.md`, and + `docs/contracts/substrate-gateway-status-schema.md` already define a gateway + lifecycle/status contract, including `substrate world gateway sync|status|restart` + and status JSON. +- `docs/cross-platform/mac_world_setup.md` still teaches direct guest build, + install, service enablement, and raw guest probing as standard setup. Hardening cannot succeed if the supported environment and breakglass boundary remain implicit. @@ -42,16 +56,33 @@ This milestone should produce four concrete decisions. - `vmType: "vz"` in `scripts/mac/lima/substrate.yaml` - guest systemd units and socket activation - stable `limactl` SSH config layout used by `crates/world-mac-lima/src/forwarding.rs` - - The contract must say whether `vsock-proxy` is required, optional, or unsupported for the hardened default. + - The contract must say whether `vsock-proxy` is required, optional, or + unsupported for the hardened default. + - The contract must explicitly include the already-landed managed gateway + runtime artifact location `/run/substrate/substrate-gateway-runtime/` as a + supported guest runtime surface. 2. Transport contract - Freeze one canonical forwarding story for supported mode. - - Eliminate ambiguity between `17788` and `7788` as an implementation requirement for later phases. - - State that guest raw TCP listeners such as `SUBSTRATE_AGENT_TCP_PORT=61337` are not part of the hardened default and require explicit breakglass treatment if kept at all. + - State that the canonical guest service endpoint is `/run/substrate.sock` + and that current runtime forwarding is UDS-backed. + - State that SSH TCP fallback is intentionally unsupported in the default + story. + - Eliminate stale `7788` references and classify host TCP `17788` use as a + compatibility/breakglass probe rather than the supported contract. + - State that guest raw TCP listeners such as + `SUBSTRATE_AGENT_TCP_PORT=61337` are not part of the hardened default and + require explicit breakglass treatment if kept at all. 3. Breakglass contract - - Reclassify direct `limactl shell`, direct guest `systemctl`, direct guest binary install, and direct guest socket curls as breakglass workflows. + - Reclassify direct `limactl shell`, direct guest `systemctl`, direct guest + binary install, direct guest socket curls, and host-side + `SUBSTRATE_WORLD_SOCKET` override use as breakglass or advanced/test-only + workflows. - Require those workflows to be documented as emergency recovery or deep debugging only. 4. Substrate-owned control-plane contract - - Normal operators should use Substrate entry points for warm, doctor, status, restart, and validation flows. + - Normal operators should use Substrate entry points for doctor, gateway + lifecycle/status, and validation flows now. + - Warm/provision repair remains transitional, but later phases must cut over + around the existing doctors and gateway commands instead of bypassing them. - Future docs and scripts must avoid telling users to repair normal operation by manually managing the guest first. ## Dependencies / sequencing @@ -79,6 +110,13 @@ This milestone should produce four concrete decisions. - Forwarding and endpoint mismatches: - `crates/world-mac-lima/src/forwarding.rs` - `crates/world-mac-lima/src/transport.rs` + - `crates/world-mac-lima/src/lib.rs` + - `crates/shell/src/execution/platform/macos.rs` + - `crates/shell/src/builtins/world_gateway.rs` +- Gateway lifecycle/status contract already landed: + - `docs/contracts/substrate-gateway-operator-contract.md` + - `docs/contracts/substrate-gateway-status-schema.md` + - `crates/world-agent/src/gateway_runtime.rs` - Operator guidance that must be reclassified: - `docs/WORLD.md` - `docs/cross-platform/mac_world_setup.md` @@ -91,26 +129,47 @@ This milestone should produce four concrete decisions. - why it is breakglass - supported replacement path - evidence that later phases must provide before demoting manual steps -- A transport baseline statement covering default forwarding, guest TCP posture, and endpoint consistency requirements. +- A transport baseline statement covering default forwarding, guest TCP posture, + endpoint consistency requirements, and the difference between the supported + UDS-backed path versus retained host TCP fallback probes. ## Acceptance criteria - The milestone defines the minimum supported environment in terms of required capabilities, not vague "recent Lima" wording. - The milestone decides whether `vsock-proxy` is optional fallback infrastructure or a prerequisite for hardened mode. -- The milestone states that the hardened default does not expose a default guest TCP listener and that any retained TCP path is explicit breakglass. +- The milestone states that the hardened default does not expose a default guest + TCP listener and that any retained TCP path is explicit breakglass or + compatibility-only, not the supported Lima contract. - The milestone identifies direct guest administration flows that must move out of normal setup and troubleshooting guidance. - The milestone defines a replacement expectation that future Substrate-owned commands must cover: - - warm/provision - doctor and status + - gateway lifecycle/status + - warm/provision - service restart - validation/smoke evidence -- The milestone names the transport consistency gap (`17788` vs `7788`) as an explicit later implementation requirement, not incidental cleanup. +- The milestone names the remaining transport consistency gaps + (`17788` compatibility probing and stale `7788` references) as explicit later + implementation requirements, not incidental cleanup. +- The milestone classifies host-side `SUBSTRATE_WORLD_SOCKET` override use as + advanced/test/breakglass rather than the default supported Lima path. ## Validation / evidence plan -- Review `scripts/mac/lima/substrate.yaml` and confirm the version contract accounts for every capability the profile already assumes. -- Review `crates/world-mac-lima/src/forwarding.rs` and `crates/world-mac-lima/src/transport.rs` and confirm the transport baseline captures the current port and fallback inconsistencies. -- Review `docs/cross-platform/mac_world_setup.md` and `docs/WORLD.md` and enumerate every direct guest command that should be labeled breakglass in later docs work. +- Review `scripts/mac/lima/substrate.yaml` and confirm the version contract + accounts for every capability the profile already assumes. +- Review `crates/world-mac-lima/src/forwarding.rs`, + `crates/world-mac-lima/src/transport.rs`, + `crates/world-mac-lima/src/lib.rs`, and + `crates/shell/src/execution/platform/macos.rs` and confirm the transport + baseline captures the current UDS-backed story, stale `7788` references, and + host TCP `17788` compatibility probing. +- Review `crates/shell/src/builtins/world_gateway.rs`, + `crates/world-agent/src/gateway_runtime.rs`, and + `scripts/mac/smoke.sh` and confirm the SOW acknowledges the already-landed + gateway lifecycle/status contract and smoke coverage. +- Review `docs/cross-platform/mac_world_setup.md` and `docs/WORLD.md` and + enumerate every direct guest command or host-side override that should be + labeled breakglass in later docs work. - Require future implementation-phase plans to cite this milestone when changing: - default transport selection - TCP listener posture diff --git a/macos-hardening/macos-hardened-same-user-lima/phase-1-runtime-parity-foundation/README.md b/macos-hardening/macos-hardened-same-user-lima/phase-1-runtime-parity-foundation/README.md index 1a73ebd5e..f5e27d041 100644 --- a/macos-hardening/macos-hardened-same-user-lima/phase-1-runtime-parity-foundation/README.md +++ b/macos-hardening/macos-hardened-same-user-lima/phase-1-runtime-parity-foundation/README.md @@ -4,174 +4,141 @@ Status: Draft Owner: Substrate world backend / macOS runtime parity -Last updated: 2026-04-28 - -## Purpose / Outcome - -Establish one credible macOS runtime contract for the current same-user Lima -model before deeper hardening work begins. The outcome of this phase is not -"macOS is secure enough"; it is "macOS and Linux describe the same runtime -behavior at the transport, policy-application, and readiness-evidence layers." - -Phase 1 is complete when the macOS backend no longer depends on transport drift, -synthetic policy injection, or direct-guest operational shortcuts to appear -healthy. - -## Why This Phase Exists - -The current macOS backend still diverges from Linux in three foundational ways: - -- transport behavior is split across - [crates/world-mac-lima/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/lib.rs:207), - [crates/world-mac-lima/src/forwarding.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/forwarding.rs:149), - and - [crates/world-mac-lima/src/transport.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/transport.rs:53), - including the current `17788` vs `7788` inconsistency. -- policy semantics are not actually applied by the backend because - [MacLimaBackend::convert_exec_request](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/lib.rs:275) - synthesizes a permissive snapshot and - [MacLimaBackend::apply_policy](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/lib.rs:463) - is a no-op. -- operational validation still normalizes direct `limactl shell` entry as part - of normal setup, diagnosis, and smoke flows in - [scripts/mac/lima-warm.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/lima-warm.sh:788), - [scripts/mac/lima-doctor.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/lima-doctor.sh:62), - [scripts/mac/smoke.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/smoke.sh:155), - [docs/WORLD.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/WORLD.md:148), - and - [docs/cross-platform/mac_world_setup.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/cross-platform/mac_world_setup.md:85). - -Until these three layers are aligned, later same-user hardening work would be -stacked on top of a runtime that still behaves differently from Linux and still -proves itself using bypass-heavy workflows. - -## In Scope - -- Define one canonical macOS host-to-guest transport contract and remove - internal constant drift. -- Make PTY, non-PTY, readiness probing, and doctor reporting consume the same - transport contract. -- Replace the synthetic permissive policy snapshot path with Linux-like policy - application semantics, even if the backend contract must be extended to carry - the normalized snapshot. -- Make doctor/smoke/setup evidence prove the routed Substrate path first, with - direct guest entry moved out of the normal operator path. - -## Out of Scope +Last updated: 2026-05-19 + +## Purpose / outcome + +Converge the current same-user Lima runtime with the already-landed Substrate shell and operator contract before deeper hardening work begins. The outcome of this phase is not "invent doctor and gateway commands for macOS"; it is "the backend-mediated Lima path matches the transport, policy, shared-world, and readiness story that the CLI already exposes." + +Phase 1 is complete when the macOS backend no longer depends on transport drift, backend-local policy synthesis, or direct-guest operational shortcuts to appear healthy. + +## Why this phase exists + +The current macOS backend still diverges from Linux in a few foundational places, but the gap is narrower than older docs suggest: + +- `substrate host doctor`, `substrate world doctor`, and `substrate world gateway sync|status|restart` already exist as canonical operator surfaces, and `scripts/mac/smoke.sh` already exercises gateway lifecycle on macOS. +- Shared-world/orchestration support already exists on the Lima-backed path, and `scripts/mac/orchestration-smoke.sh` already covers shared-owner bootstrap, ready-proof acceptance, replacement, lazy member launch, and mismatch rejection. +- The shell-side direct routed request builders already resolve and forward `policy_snapshot`, `world_network`, and `world_fs_mode` in: + - `crates/shell/src/execution/routing/dispatch/world_ops.rs` + - `crates/shell/src/repl/async_repl.rs` + - `crates/shell/src/builtins/world_gateway.rs` + +The remaining phase-1 gaps are therefore specific and concrete: + +- transport behavior is still split across `crates/world-mac-lima/src/lib.rs`, `crates/world-mac-lima/src/forwarding.rs`, and `crates/world-mac-lima/src/transport.rs`, including the current `17788` versus `7788` inconsistency. +- backend policy semantics are not actually applied on the backend-mediated Lima path because `MacLimaBackend::convert_exec_request` synthesizes a permissive snapshot and `MacLimaBackend::apply_policy` is a no-op. +- readiness and troubleshooting evidence still normalize direct guest entry in `scripts/mac/lima-warm.sh`, `scripts/mac/lima-doctor.sh`, `docs/WORLD.md`, and `docs/cross-platform/mac_world_setup.md` more than the supported Lima path should. +- `SUBSTRATE_WORLD_SOCKET` still bypasses Lima detection/startup and remains an advanced/test/breakglass override, not the standard macOS path. That matters because explicit shared-owner reuse already rejects under this bypass on macOS. + +Until those layers align, later same-user hardening work would sit on top of a backend that still proves itself using bypass-heavy workflows and still diverges from the shell-owned policy/runtime contract. + +## In scope + +- Define one canonical macOS host-to-guest transport contract and remove internal constant drift. +- Make PTY, non-PTY, readiness probing, and doctor reporting consume that same transport contract. +- Remove backend-local policy synthesis from `MacLimaBackend` and make backend-mediated Lima execution consume the same resolved policy/world inputs the shell already computes. +- Keep shared-world/orchestration parity explicit on the Lima-backed path, including the fail-closed behavior around `SUBSTRATE_WORLD_SOCKET`. +- Make doctor/smoke/setup evidence prove the routed Substrate path first, with direct guest entry moved out of the normal operator path. + +## Out of scope - Changing the host ownership boundary of Lima or `LIMA_HOME`. - Claiming Linux-equivalent ownership isolation on macOS. -- Removing all same-user hardening gaps such as broad host mounts, extra guest - listeners, or direct control-plane ownership. -- Designing the ownership-separated macOS architecture. +- Removing all same-user hardening gaps such as broad host mounts, extra guest listeners, or direct control-plane ownership. +- Designing an ownership-separated macOS architecture. - Rewriting provisioning to eliminate every internal `limactl` dependency. -## Architectural Approach +## Architectural approach -Phase 1 treats `/run/substrate.sock` inside the guest as the only canonical -service endpoint. VSock, SSH UDS, and any retained TCP surface are host-side -transport adapters, not separate behavioral modes. +Phase 1 treats `/run/substrate.sock` inside the guest as the only canonical service endpoint. VSock, SSH UDS, and any retained TCP surface are host-side transport adapters, not separate behavioral modes. -The backend must then stop inventing policy. The evaluated broker snapshot used -by Linux-facing world execution should become the source of truth for macOS as -well. If -[world_api::WorldSpec](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-api/src/lib.rs:14) -cannot currently carry the policy and routing state required for parity, this -phase extends that contract rather than preserving backend-local synthesis. - -Finally, readiness flows must prove the same path that users depend on: +The shell/operator baseline already exists: - `substrate host doctor` - `substrate world doctor` -- `scripts/mac/smoke.sh` +- `substrate world gateway sync` +- `substrate world gateway status` +- `substrate world gateway restart` + +Phase 1 hardens around those surfaces instead of introducing new ones. -Direct guest commands remain available only as breakglass diagnostics and must -no longer define the happy-path contract. +The policy-parity work is deliberately scoped. The shell already resolves and serializes the effective policy/world inputs for direct routed requests; the unresolved gap is that backend-mediated Lima execution still invents or drops those semantics inside `MacLimaBackend`. If `world_api::WorldSpec` or adjacent backend seams must be extended to carry the right inputs for parity, that is preferable to preserving backend-local synthesis. -## Dependencies / Sequencing +Shared-world parity is also already partially present. Phase 1 does not add shared-owner support from scratch; it ensures the Lima-backed path remains the authoritative macOS path for that support, while `SUBSTRATE_WORLD_SOCKET` remains classified as an advanced/test/breakglass bypass. -This phase is ordered and should be executed in the following sequence: +## Dependencies / sequencing -1. [Milestone 1.1: Transport Contract Unification](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/macos-hardening/macos-hardened-same-user-lima/phase-1-runtime-parity-foundation/milestone-1-1-transport-contract-unification-sow.md) -2. [Milestone 1.2: Policy Application Parity](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/macos-hardening/macos-hardened-same-user-lima/phase-1-runtime-parity-foundation/milestone-1-2-policy-application-parity-sow.md) -3. [Milestone 1.3: Doctor / Smoke Readiness Parity](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/macos-hardening/macos-hardened-same-user-lima/phase-1-runtime-parity-foundation/milestone-1-3-doctor-smoke-readiness-parity-sow.md) +This phase should be executed in the following sequence: -Milestone 1.3 can prepare doc rewrites in parallel, but final readiness -criteria should not be signed off until Milestones 1.1 and 1.2 land because the -doctor/smoke evidence needs to prove the new transport and policy behavior. +1. [Milestone 1.1: Transport Contract Unification](./milestone-1-1-transport-contract-unification-sow.md) +2. [Milestone 1.2: Policy Application Parity](./milestone-1-2-policy-application-parity-sow.md) +3. [Milestone 1.3: Doctor / Smoke Readiness Parity](./milestone-1-3-doctor-smoke-readiness-parity-sow.md) -## Concrete Repo Surfaces and File Pointers +Milestone 1.3 can prepare doc rewrites in parallel, but final readiness criteria should not be signed off until Milestones 1.1 and 1.2 land because the doctor/smoke evidence needs to prove the new transport and backend-policy behavior. + +## Concrete repo surfaces and file pointers Primary runtime/backend surfaces: -- [crates/world-mac-lima/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/lib.rs) -- [crates/world-mac-lima/src/forwarding.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/forwarding.rs) -- [crates/world-mac-lima/src/transport.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/transport.rs) -- [crates/world-api/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-api/src/lib.rs) -- [crates/shell/src/execution/policy_snapshot.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/policy_snapshot.rs) -- [crates/shell/src/execution/routing/world.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/world.rs) -- [crates/shell/src/execution/platform/macos.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/platform/macos.rs) -- [crates/shell/src/execution/routing/dispatch/exec.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/exec.rs) +- `crates/world-mac-lima/src/lib.rs` +- `crates/world-mac-lima/src/forwarding.rs` +- `crates/world-mac-lima/src/transport.rs` +- `crates/world-api/src/lib.rs` +- `crates/shell/src/execution/policy_snapshot.rs` +- `crates/shell/src/execution/routing/world.rs` +- `crates/shell/src/execution/routing/dispatch/world_ops.rs` +- `crates/shell/src/repl/async_repl.rs` +- `crates/shell/src/builtins/world_gateway.rs` Primary scripts and operator-doc surfaces: -- [scripts/mac/lima-warm.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/lima-warm.sh) -- [scripts/mac/lima-doctor.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/lima-doctor.sh) -- [scripts/mac/smoke.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/smoke.sh) -- [docs/WORLD.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/WORLD.md) -- [docs/cross-platform/mac_world_setup.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/cross-platform/mac_world_setup.md) +- `scripts/mac/lima-warm.sh` +- `scripts/mac/lima-doctor.sh` +- `scripts/mac/smoke.sh` +- `scripts/mac/orchestration-smoke.sh` +- `docs/WORLD.md` +- `docs/cross-platform/mac_world_setup.md` +- `docs/USAGE.md` Research input anchoring this phase: -- [thoughts/shared/research/2026-04-28-macos-lima-parity-lockdown.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/thoughts/shared/research/2026-04-28-macos-lima-parity-lockdown.md) +- `macos-hardening/research/2026-04-28-macos-lima-parity-lockdown.md` ## Deliverables - A phase-local milestone set with explicit sequencing and acceptance gates. -- A canonical macOS transport contract design covering backend, shell, doctor, - and smoke consumers. -- A backend policy-application plan that removes synthetic permissive snapshots. -- A readiness-evidence plan that makes CLI doctors and smoke flows authoritative - and relegates direct guest entry to breakglass. - -## Acceptance Criteria - -- There is one documented and implemented macOS transport contract for routed - world traffic, and no remaining `17788` vs `7788` disagreement. -- Mac PTY and non-PTY requests consume the same transport-selection rules and - endpoint semantics. -- `MacLimaBackend` no longer invents a permissive policy snapshot from - `WorldFsMode` alone. -- macOS doctor/smoke readiness proves routed Substrate behavior before any - direct guest fallback checks. -- The phase docs make explicit that same-user Lima still does not provide the - Linux ownership boundary. - -## Validation / Evidence Plan +- A canonical macOS transport contract covering backend, shell, doctor, and smoke consumers. +- A backend policy-application plan that removes synthetic permissive snapshots from `MacLimaBackend`. +- A readiness-evidence plan that keeps CLI doctors, gateway lifecycle, and smoke/orchestration flows authoritative while relegating direct guest entry to breakglass. + +## Acceptance criteria + +- There is one documented and implemented macOS transport contract for routed world traffic, and no remaining `17788` versus `7788` disagreement. +- Mac PTY and non-PTY requests consume the same transport-selection rules and endpoint semantics. +- `MacLimaBackend` no longer invents a permissive policy snapshot for backend-mediated execution. +- The phase docs explicitly acknowledge that shell-side direct routed request builders already carry resolved `policy_snapshot`, `world_network`, and `world_fs_mode`. +- macOS doctor/smoke readiness proves routed Substrate behavior before any direct guest fallback checks. +- The phase docs make explicit that same-user Lima still does not provide the Linux ownership boundary. + +## Validation / evidence plan Implementation work under this phase should attach evidence from: -- targeted Rust tests for `world-mac-lima`, `world-api`, and shell routing / - doctor surfaces +- targeted Rust tests for `world-mac-lima`, `world-api`, and shell routing/doctor surfaces - `cargo test -p world-mac-lima` -- relevant shell tests covering world routing and doctor output +- relevant shell tests covering world routing, shared-world proof handling, gateway status, and doctor output - `scripts/mac/lima-doctor.sh` - `scripts/mac/smoke.sh` +- `scripts/mac/orchestration-smoke.sh` - `substrate host doctor --json` - `substrate world doctor --json` +- `substrate world gateway status --json` + +Evidence must show selected transport, routed agent reachability, gateway lifecycle visibility, shared-world proof handling, and backend-policy parity through the CLI path, not only through direct in-guest `curl`. + +## Risks / open questions -Evidence must show the selected transport, routed agent reachability, and -policy-parity behavior through the CLI path, not only through direct in-guest -`curl`. - -## Risks / Open Questions - -- Extending `WorldSpec` or adjacent backend contracts may ripple into the - Windows backend and shared factory code. -- The repo currently mixes backend abstraction and direct agent API request - paths; parity work must avoid creating two competing policy sources of truth. -- Some direct guest checks may remain necessary during bootstrap failure, but - those must be classified as breakglass diagnostics rather than normal setup. -- Phase 1 improves parity, not ownership isolation; future hardening phases must - still address broad mounts, extra listeners, and same-user Lima control-plane - ownership. +- Extending `WorldSpec` or adjacent backend contracts may ripple into the Windows backend and shared factory code. +- The repo mixes backend abstraction and direct agent API request paths; parity work must avoid creating two competing policy sources of truth. +- Some direct guest checks may remain necessary during bootstrap failure, but those must be classified as breakglass diagnostics rather than normal setup. +- Phase 1 improves backend/runtime parity, not ownership isolation; future hardening phases must still address broad mounts, extra listeners, and same-user Lima control-plane ownership. diff --git a/macos-hardening/macos-hardened-same-user-lima/phase-1-runtime-parity-foundation/milestone-1-1-transport-contract-unification-sow.md b/macos-hardening/macos-hardened-same-user-lima/phase-1-runtime-parity-foundation/milestone-1-1-transport-contract-unification-sow.md index ff77eecfa..20f5d3381 100644 --- a/macos-hardening/macos-hardened-same-user-lima/phase-1-runtime-parity-foundation/milestone-1-1-transport-contract-unification-sow.md +++ b/macos-hardening/macos-hardened-same-user-lima/phase-1-runtime-parity-foundation/milestone-1-1-transport-contract-unification-sow.md @@ -4,28 +4,35 @@ Status: Draft Owner: Substrate world backend / macOS transport layer -Last updated: 2026-04-28 +Last updated: 2026-05-19 ## Purpose / Outcome Define and implement one canonical host-to-guest transport contract for the -macOS Lima backend so that execution, readiness probing, and diagnostics all -describe the same path. +macOS Lima backend so that execution, readiness probing, gateway lifecycle, and +diagnostics all describe the same path. -The concrete outcome is that VSock, SSH UDS, and any retained TCP fallback are -implementation details behind one declared contract rather than partially -independent behavior paths. +The concrete outcome is that the guest endpoint contract is unambiguously +`/run/substrate.sock`, with VSock and SSH UDS as ways to reach that socket from +the host. The remaining transport work is to remove stale `7788` references and +to reclassify or eliminate retained host TCP `17788` compatibility probes. ## Why This Milestone Exists -Transport behavior is currently fragmented: - -- `17788` is used in forwarding and doctor code, while `7788` is still hardcoded - elsewhere in the backend. -- the backend, forwarding layer, and doctor logic all carry separate endpoint - assumptions. -- PTY and non-PTY parity depends on those assumptions staying accidentally in - sync. +Transport behavior is partly unified already, but the repo still describes it +incoherently. + +- `crates/world-mac-lima/src/forwarding.rs` already forwards into the guest UDS + endpoint `/run/substrate.sock` and intentionally skips SSH TCP fallback. +- `crates/world-mac-lima/src/transport.rs` still advertises `127.0.0.1:7788` + for TCP and `MacLimaBackend::test_agent_connection(...)` still probes that + stale port. +- `crates/shell/src/execution/platform/macos.rs` and + `crates/shell/src/builtins/world_gateway.rs` prefer the host UDS when + available, then fall back to host TCP `17788` or in-VM probing. +- Host-side `SUBSTRATE_WORLD_SOCKET` override use bypasses the normal Lima path + and should be treated as advanced/test/breakglass rather than as the default + transport contract. This creates false confidence: macOS can appear "working" while different code paths are probing different endpoints. @@ -36,6 +43,8 @@ paths are probing different endpoints. by the macOS backend. - Make routed execution and readiness probes consume the same selected transport. +- Keep the supported story centered on UDS-backed routing rather than on guest + TCP reachability. - Normalize how transport selection is surfaced to shell doctor/reporting surfaces. - Remove stale TCP constant drift and stale fallback assumptions. @@ -53,23 +62,27 @@ paths are probing different endpoints. The transport contract for this milestone is: - canonical guest service endpoint: `/run/substrate.sock` -- canonical host transport abstraction: one selected forwarding mode exposed by - `world-mac-lima` -- canonical health/readiness contract: route the same capabilities / doctor - checks through the selected transport first +- canonical host default path: the managed host UDS under + `~/.substrate/sock/agent.sock` when present, otherwise one selected + forwarding mode exposed by `world-mac-lima` +- canonical forwarding modes: + - VSock proxy exposing host TCP `127.0.0.1:17788` into guest UDS + - SSH UDS forwarding into the same guest UDS +- unsupported default path: SSH TCP fallback +- canonical health/readiness contract: probe the selected routed transport + first, then classify any retained host TCP `17788` probing as + compatibility-only until removed Implementation should centralize the selected transport description in -[crates/world-mac-lima/src/transport.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/transport.rs) -or an adjacent shared type, then make -[crates/world-mac-lima/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/lib.rs), -[crates/world-mac-lima/src/forwarding.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/forwarding.rs), -and -[crates/shell/src/execution/platform/macos.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/platform/macos.rs) -consume that one source of truth. - -If TCP remains as a fallback, it must use one documented port constant and one -single reason for existence. If it is not truly supported, the contract should -say so and the dead fallback behavior should be removed. +`crates/world-mac-lima/src/transport.rs` or an adjacent shared type, then make +`crates/world-mac-lima/src/lib.rs`, +`crates/world-mac-lima/src/forwarding.rs`, +`crates/shell/src/execution/platform/macos.rs`, and +`crates/shell/src/builtins/world_gateway.rs` consume that one source of truth. + +If host TCP remains anywhere after centralization, it must use one documented +port constant, one explicit reason for existence, and must not be described as +the default supported Lima contract. ## Dependencies / Sequencing @@ -83,24 +96,27 @@ say so and the dead fallback behavior should be removed. Primary implementation surfaces: -- [crates/world-mac-lima/src/transport.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/transport.rs) -- [crates/world-mac-lima/src/forwarding.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/forwarding.rs) -- [crates/world-mac-lima/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/lib.rs) -- [crates/shell/src/execution/platform/macos.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/platform/macos.rs) -- [crates/shell/src/execution/routing/dispatch/exec.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/exec.rs) +- `crates/world-mac-lima/src/transport.rs` +- `crates/world-mac-lima/src/forwarding.rs` +- `crates/world-mac-lima/src/lib.rs` +- `crates/shell/src/execution/platform/macos.rs` +- `crates/shell/src/builtins/world_gateway.rs` +- `crates/shell/src/execution/routing/dispatch/exec.rs` Current drift evidence: -- [crates/world-mac-lima/src/forwarding.rs:149](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/forwarding.rs:149) -- [crates/world-mac-lima/src/transport.rs:53](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/transport.rs:53) -- [crates/world-mac-lima/src/lib.rs:207](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/lib.rs:207) -- [crates/shell/src/execution/platform/macos.rs:360](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/platform/macos.rs:360) +- `crates/world-mac-lima/src/forwarding.rs:83` +- `crates/world-mac-lima/src/forwarding.rs:189` +- `crates/world-mac-lima/src/transport.rs:20` +- `crates/world-mac-lima/src/lib.rs:206` +- `crates/shell/src/execution/platform/macos.rs:369` +- `crates/shell/src/builtins/world_gateway.rs:178` Secondary doc/script surfaces that must follow the contract: -- [scripts/mac/lima-doctor.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/lima-doctor.sh) -- [scripts/mac/smoke.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/smoke.sh) -- [docs/WORLD.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/WORLD.md) +- `scripts/mac/lima-doctor.sh` +- `scripts/mac/smoke.sh` +- `docs/WORLD.md` ## Deliverables @@ -113,15 +129,19 @@ Secondary doc/script surfaces that must follow the contract: ## Acceptance Criteria -- No remaining macOS backend code path disagrees on the loopback port used for a - TCP or VSock-routed agent path. +- No remaining macOS backend code path disagrees on the host compatibility port + used for VSock-proxy routing, and no stale `7788` references remain. - `MacLimaBackend` derives its agent endpoint from the same selected forwarding contract used by readiness probing. -- `substrate host doctor` and `substrate world doctor` do not probe a different - endpoint contract than routed command execution. +- `substrate host doctor`, `substrate world doctor`, and + `substrate world gateway status` do not define a different supported endpoint + contract than routed command execution. - PTY and non-PTY macOS execution paths continue to work after transport centralization. -- Documentation describes one transport ordering and one fallback story. +- Documentation describes one transport ordering, one default path, and one + explicitly-classified fallback story. +- Host-side `SUBSTRATE_WORLD_SOCKET` override use is documented as + advanced/test/breakglass, not as the default Lima path. ## Validation / Evidence Plan @@ -131,19 +151,23 @@ Required evidence for this milestone: - `cargo test -p world-mac-lima` - targeted shell tests for macOS doctor reporting if transport fields are surfaced there +- targeted shell tests for `substrate world gateway` macOS endpoint resolution - `scripts/mac/lima-doctor.sh` - `substrate host doctor --json` - `substrate world doctor --json` +- `substrate world gateway status --json` Evidence should explicitly show which transport was selected and that the same -selection is used for command execution and doctor probing. +selection is used for command execution, gateway lifecycle/status, and doctor +probing. ## Risks / Open Questions - The current shell doctor implementation has its own probing logic; deciding whether to reuse backend transport code directly or mirror it via a shared utility needs care to avoid cyclic dependencies. -- TCP fallback may be partly vestigial. The team needs to decide whether to make - it real, keep it only as an explicit breakglass path, or remove it. +- Host TCP `17788` use exists today as a VSock-proxy and compatibility probe + surface; the team needs to decide whether to keep that probe temporarily, + route everything through shared endpoint resolution, or remove it. - VSock availability varies with host setup; the contract must be stable even when VSock is absent. diff --git a/macos-hardening/macos-hardened-same-user-lima/phase-1-runtime-parity-foundation/milestone-1-2-policy-application-parity-sow.md b/macos-hardening/macos-hardened-same-user-lima/phase-1-runtime-parity-foundation/milestone-1-2-policy-application-parity-sow.md index 9f7da1b34..568cc5537 100644 --- a/macos-hardening/macos-hardened-same-user-lima/phase-1-runtime-parity-foundation/milestone-1-2-policy-application-parity-sow.md +++ b/macos-hardening/macos-hardened-same-user-lima/phase-1-runtime-parity-foundation/milestone-1-2-policy-application-parity-sow.md @@ -4,34 +4,43 @@ Status: Draft Owner: Substrate world backend / policy application path -Last updated: 2026-04-28 +Last updated: 2026-05-19 ## Purpose / Outcome -Make the macOS Lima backend execute with the same evaluated policy semantics -that Linux world execution already depends on, instead of synthesizing a -permissive backend-local snapshot. +Make the backend-mediated macOS Lima execution path use the same evaluated +policy semantics that Linux world execution already depends on, instead of +synthesizing a permissive backend-local snapshot. -The concrete outcome is that macOS world execution uses policy data derived from -the broker/shell policy-resolution pipeline, and backend policy updates become -observable and testable rather than effectively ignored. +The concrete outcome is that macOS world execution driven through +`MacLimaBackend` uses policy data derived from the broker/shell +policy-resolution pipeline, while preserving the shell-side direct request +builders that already carry resolved policy, world-network, and shared-world +inputs today. ## Why This Milestone Exists -Today the macOS backend only carries `WorldFsMode` state and invents the rest of -the policy request: - -- [convert_exec_request](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/lib.rs:275) - fabricates `PolicySnapshotV3` with host-visible, current-directory-only, - permissive settings. -- [apply_policy](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/lib.rs:463) - does not apply anything beyond storing `fs_mode`. -- [world_api::WorldSpec](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-api/src/lib.rs:14) - currently lacks the richer normalized policy payload that the Linux-facing - direct request path already uses. - -This means macOS parity can be overstated even when the backend is not honoring -the same policy contract the rest of Substrate resolves. +The repo already has two different truths, and only one of them is the actual +gap. + +- `crates/world-mac-lima/src/lib.rs:416` still fabricates `PolicySnapshotV3` + with host-visible, current-directory-only, permissive settings; it forwards + `shared_world` and `world_fs_mode`, but drops richer policy/network inputs. +- `crates/world-mac-lima/src/lib.rs:604` still does not apply anything beyond + storing `fs_mode`. +- Shell-side direct request builders already resolve and propagate + broker-derived policy snapshots, world-network routing, and world-fs + enforcement-plan inputs in: + - `crates/shell/src/execution/policy_snapshot.rs` + - `crates/shell/src/execution/routing/dispatch/world_ops.rs` + - `crates/shell/src/builtins/world_gateway.rs` +- Shared-world/orchestration support is already carried through request/response + types; the gap is not “macOS invents all policy,” it is that the + backend-mediated macOS path still invents too much of it locally. + +This means macOS parity is overstated specifically when execution flows through +`MacLimaBackend`, even though other shell request paths already carry the right +inputs. ## In Scope @@ -39,6 +48,8 @@ the same policy contract the rest of Substrate resolves. `world-mac-lima`. - Thread the evaluated policy snapshot and related world-routing inputs through the macOS backend path. +- Preserve the already-landed shell-side request builders and shared-world + semantics while aligning the backend-mediated path to them. - Implement meaningful `apply_policy` semantics for the macOS backend. - Align macOS world execution with Linux expectations for `net_allowed`, `world_fs.fail_closed`, and world-fs enforcement-plan inputs as far as the @@ -55,25 +66,29 @@ the same policy contract the rest of Substrate resolves. The shell / broker policy-resolution flow should remain the source of truth: -- policy resolution lives in - [crates/shell/src/execution/policy_snapshot.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/policy_snapshot.rs) +- policy resolution lives in `crates/shell/src/execution/policy_snapshot.rs` - direct world operations already construct agent requests with resolved `policy_snapshot`, `world_network`, and world-fs enforcement-plan inputs in - [crates/shell/src/execution/routing/dispatch/world_ops.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs) + `crates/shell/src/execution/routing/dispatch/world_ops.rs` +- gateway lifecycle requests already construct broker-derived + `policy_snapshot`, `world_network`, identity, placement, and integrated auth + inputs in `crates/shell/src/builtins/world_gateway.rs` The backend path used by `WorldBackend`-driven execution should be brought up to -that same contract. If -[crates/world-api/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-api/src/lib.rs) -cannot carry the necessary evaluated policy state today, this milestone extends -the world backend abstraction rather than preserving macOS-only synthesis. +that same contract. If `crates/world-api/src/lib.rs` cannot carry the necessary +evaluated policy state today, this milestone extends the world backend +abstraction rather than preserving macOS-only synthesis. The target behavior is: - the broker-resolved snapshot is canonical - the backend stores or receives the current snapshot for the world session +- the backend stores or receives current world-network routing inputs alongside + that snapshot - execution requests sent through `MacLimaBackend` reuse that snapshot - `apply_policy` becomes the point where drift between world session state and current policy is reconciled or rejected +- existing `shared_world` passthrough remains intact and test-covered ## Dependencies / Sequencing @@ -86,28 +101,33 @@ The target behavior is: Primary backend and contract surfaces: -- [crates/world-mac-lima/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/lib.rs) -- [crates/world-api/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-api/src/lib.rs) -- [crates/world-backend-factory/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-backend-factory/src/lib.rs) +- `crates/world-mac-lima/src/lib.rs` +- `crates/world-api/src/lib.rs` +- `crates/world-backend-factory/src/lib.rs` Primary shell policy-resolution surfaces: -- [crates/shell/src/execution/policy_snapshot.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/policy_snapshot.rs) -- [crates/shell/src/execution/routing/dispatch/world_ops.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs) -- [crates/shell/src/execution/routing/world.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/world.rs) +- `crates/shell/src/execution/policy_snapshot.rs` +- `crates/shell/src/execution/routing/dispatch/world_ops.rs` +- `crates/shell/src/execution/routing/world.rs` +- `crates/shell/src/builtins/world_gateway.rs` Current divergence evidence: -- [crates/world-mac-lima/src/lib.rs:275](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/lib.rs:275) -- [crates/world-mac-lima/src/lib.rs:463](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/lib.rs:463) -- [crates/shell/src/execution/policy_snapshot.rs:155](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/policy_snapshot.rs:155) -- [crates/shell/src/execution/routing/dispatch/world_ops.rs:1046](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/world_ops.rs:1046) +- `crates/world-mac-lima/src/lib.rs:416` +- `crates/world-mac-lima/src/lib.rs:451` +- `crates/world-mac-lima/src/lib.rs:604` +- `crates/shell/src/execution/policy_snapshot.rs:152` +- `crates/shell/src/execution/routing/dispatch/world_ops.rs:1146` +- `crates/shell/src/builtins/world_gateway.rs:312` ## Deliverables - A world-backend contract that can carry evaluated policy state needed for macOS parity. - Removal of synthetic permissive snapshot generation from `MacLimaBackend`. +- Propagation of world-network routing and preserved `shared_world` semantics in + the backend-mediated path. - Implemented `apply_policy` behavior for macOS world sessions. - Regression tests that prove macOS backend requests carry the expected policy values. @@ -115,13 +135,16 @@ Current divergence evidence: ## Acceptance Criteria -- No macOS backend request path fabricates a permissive `PolicySnapshotV3` from - `WorldFsMode` alone. +- No backend-mediated macOS request path fabricates a permissive + `PolicySnapshotV3` from `WorldFsMode` alone. - A policy change that affects world routing or world-fs semantics is visible to the macOS backend and can trigger session update, reapply, or fail-closed behavior by design. - macOS execution requests carry the same relevant policy fields that Linux world-agent requests already depend on. +- macOS backend requests also preserve the existing `shared_world` propagation + contract instead of regressing orchestration support while policy parity is + fixed. - `apply_policy` is no longer a semantic no-op. - Tests cover at least `net_allowed`, `world_fs.write.enabled`, and fail-closed/routing-relevant behavior in the macOS backend path. @@ -135,6 +158,8 @@ Required evidence for this milestone: extended - relevant shell tests that already assert policy snapshot contents for world requests +- targeted gateway lifecycle tests if shared request-building helpers or policy + carriers are reused there - `cargo test -p world-mac-lima` - targeted end-to-end proof via `substrate world doctor --json` and one or more routed world operations that would previously have succeeded under the diff --git a/macos-hardening/macos-hardened-same-user-lima/phase-1-runtime-parity-foundation/milestone-1-3-doctor-smoke-readiness-parity-sow.md b/macos-hardening/macos-hardened-same-user-lima/phase-1-runtime-parity-foundation/milestone-1-3-doctor-smoke-readiness-parity-sow.md index 9b304e5cb..890490ab4 100644 --- a/macos-hardening/macos-hardened-same-user-lima/phase-1-runtime-parity-foundation/milestone-1-3-doctor-smoke-readiness-parity-sow.md +++ b/macos-hardening/macos-hardened-same-user-lima/phase-1-runtime-parity-foundation/milestone-1-3-doctor-smoke-readiness-parity-sow.md @@ -4,7 +4,7 @@ Status: Draft Owner: Substrate operator UX / macOS validation surfaces -Last updated: 2026-04-28 +Last updated: 2026-05-19 ## Purpose / Outcome @@ -12,23 +12,31 @@ Make the macOS readiness story prove the same routed Substrate contract that Linux relies on, and stop treating direct guest administration as the normal path for setup, diagnosis, and smoke validation. -The concrete outcome is that CLI doctors and macOS smoke validation become the -authoritative readiness evidence, while direct `limactl shell` commands are -explicitly breakglass-only diagnostics. +The concrete outcome is not inventing doctors or gateway lifecycle from +scratch. Those surfaces already exist. The outcome is to make existing CLI +doctors, gateway lifecycle/status, and macOS smoke validation the authoritative +readiness evidence, while direct `limactl shell` commands become explicitly +breakglass-only diagnostics. ## Why This Milestone Exists -Current readiness and setup surfaces still normalize direct guest entry: - -- [scripts/mac/lima-doctor.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/lima-doctor.sh) - shells into the guest for core health checks. -- [scripts/mac/smoke.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/smoke.sh) - mixes routed Substrate checks with direct guest probes. -- [docs/WORLD.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/WORLD.md:148) - and - [docs/cross-platform/mac_world_setup.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/cross-platform/mac_world_setup.md:85) - still present `limactl shell`, in-guest `curl`, guest `systemctl`, and direct - guest logs as normal operator behavior. +Current readiness surfaces are partly landed, but the helpers and docs still +normalize direct guest entry. + +- `substrate host doctor` and `substrate world doctor` already exist and are + canonical CLI readiness surfaces. +- `substrate world gateway sync|status|restart` and + `substrate world gateway status --json` already exist as canonical gateway + lifecycle/status surfaces. +- `scripts/mac/smoke.sh` already includes gateway lifecycle smoke coverage. +- `scripts/mac/lima-doctor.sh` still shells into the guest for core health + checks, and `scripts/mac/smoke.sh` still mixes routed checks with direct + guest probes. +- `crates/shell/src/execution/platform/macos.rs` still falls back from host UDS + to host TCP `17788` or in-VM probing when collecting doctor evidence. +- `docs/WORLD.md` and `docs/cross-platform/mac_world_setup.md` still present + `limactl shell`, in-guest `curl`, guest `systemctl`, and direct guest logs as + normal operator behavior. That posture undermines the same-user hardening story and also weakens parity: it can prove that the guest is reachable, not that Substrate’s routed path is @@ -38,6 +46,8 @@ healthy. - Reframe `substrate host doctor` and `substrate world doctor` as the canonical readiness interfaces for macOS. +- Reframe `substrate world gateway sync|status|restart` and status JSON as + canonical readiness/support surfaces for managed gateway lifecycle. - Align `scripts/mac/lima-doctor.sh` and `scripts/mac/smoke.sh` to validate the routed Substrate path first. - Update readiness-oriented doc sections so direct guest commands are marked as @@ -60,6 +70,9 @@ This milestone should make the readiness stack flow in layers: 1. Routed CLI checks first: - `substrate host doctor` - `substrate world doctor` + - `substrate world gateway sync` + - `substrate world gateway status --json` + - `substrate world gateway restart` - routed `substrate --world` smoke operations 2. Script wrappers second: - `scripts/mac/lima-doctor.sh` @@ -72,6 +85,10 @@ the operator contract should prefer Substrate-owned commands and explain direct guest entry as a breakglass-only diagnostic path, not as the default workflow or a degraded-but-supported middle tier. +Host-side `SUBSTRATE_WORLD_SOCKET` override use is outside the normal Lima path +for this milestone. If docs mention it at all, it should be classified as +advanced/test/breakglass. + ## Dependencies / Sequencing - Depends on Milestone 1.1 so the doctor/smoke surfaces prove one transport @@ -84,21 +101,22 @@ or a degraded-but-supported middle tier. Primary readiness and docs surfaces: -- [crates/shell/src/execution/platform/macos.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/platform/macos.rs) -- [scripts/mac/lima-doctor.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/lima-doctor.sh) -- [scripts/mac/smoke.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/smoke.sh) -- [scripts/mac/lima-warm.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/lima-warm.sh) -- [docs/WORLD.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/WORLD.md) -- [docs/cross-platform/mac_world_setup.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/cross-platform/mac_world_setup.md) +- `crates/shell/src/execution/platform/macos.rs` +- `crates/shell/src/builtins/world_gateway.rs` +- `docs/contracts/substrate-gateway-operator-contract.md` +- `docs/contracts/substrate-gateway-status-schema.md` +- `scripts/mac/lima-doctor.sh` +- `scripts/mac/smoke.sh` +- `scripts/mac/lima-warm.sh` +- `docs/WORLD.md` +- `docs/cross-platform/mac_world_setup.md` Current normalization of direct guest administration: -- [docs/cross-platform/mac_world_setup.md:85](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/cross-platform/mac_world_setup.md:85) -- [docs/cross-platform/mac_world_setup.md:121](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/cross-platform/mac_world_setup.md:121) -- [docs/cross-platform/mac_world_setup.md:141](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/cross-platform/mac_world_setup.md:141) -- [scripts/mac/lima-doctor.sh:62](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/lima-doctor.sh:62) -- [scripts/mac/smoke.sh:155](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/smoke.sh:155) -- [scripts/mac/lima-warm.sh:788](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/lima-warm.sh:788) +- `docs/cross-platform/mac_world_setup.md` +- `scripts/mac/lima-doctor.sh` +- `scripts/mac/smoke.sh` +- `scripts/mac/lima-warm.sh` ## Deliverables @@ -106,6 +124,8 @@ Current normalization of direct guest administration: transport and policy state visible. - A macOS smoke path that asserts routed PTY and non-PTY behavior before any guest-direct checks. +- A readiness story that treats gateway lifecycle/status as a normal support + surface alongside doctor JSON. - Updated readiness-oriented excerpts in macOS setup and world docs that clearly separate routed readiness validation from breakglass diagnostics. - A validation matrix for operator evidence capture on macOS that mirrors Linux @@ -116,14 +136,19 @@ Current normalization of direct guest administration: - The happy-path macOS validation docs for an already provisioned backend use routed Substrate readiness commands first and do not require direct guest `curl` or `systemctl` for routine verification. +- The happy-path docs also lead with `substrate world gateway sync|status|restart` + for managed gateway lifecycle verification instead of raw guest probing. - `scripts/mac/lima-doctor.sh` fails when the routed Substrate path is unhealthy even if direct in-guest probing still succeeds. - `scripts/mac/smoke.sh` proves routed PTY and non-PTY behavior plus doctor - readiness against the canonical transport contract. + readiness against the canonical transport contract while preserving the + already-landed gateway lifecycle smoke proof. - Direct guest commands remain documented only as breakglass/unsupported procedures. - The docs explicitly state that same-user Lima still does not provide the Linux ownership boundary, even after readiness parity is achieved. +- Host-side `SUBSTRATE_WORLD_SOCKET` override use is not documented as the + default Lima path. ## Validation / Evidence Plan @@ -131,15 +156,18 @@ Required evidence for this milestone: - `substrate host doctor --json` - `substrate world doctor --json` +- `substrate world gateway sync` +- `substrate world gateway status --json` +- `substrate world gateway restart` - `scripts/mac/lima-doctor.sh` - `scripts/mac/smoke.sh` - updated doc excerpts in the implementation PR showing the happy-path command flow for readiness validation, not full lifecycle ownership Evidence should demonstrate that a user can validate and smoke-test an already -provisioned macOS backend through routed Substrate commands first, with direct -guest commands only used for deeper post-failure diagnosis. Full lifecycle -ownership and doc cutover remain Phase 3 scope. +provisioned macOS backend through routed Substrate doctor and gateway commands +first, with direct guest commands only used for deeper post-failure diagnosis. +Full lifecycle ownership and doc cutover remain Phase 3 scope. ## Risks / Open Questions diff --git a/macos-hardening/macos-hardened-same-user-lima/phase-2-same-user-hardening/README.md b/macos-hardening/macos-hardened-same-user-lima/phase-2-same-user-hardening/README.md index 53c8aa3d5..f06e36599 100644 --- a/macos-hardening/macos-hardened-same-user-lima/phase-2-same-user-hardening/README.md +++ b/macos-hardening/macos-hardened-same-user-lima/phase-2-same-user-hardening/README.md @@ -6,25 +6,28 @@ Draft ## Purpose / outcome -Tighten the current macOS Lima deployment so the same-user model exposes the smallest practical guest and host attack surface without pretending to provide the Linux ownership boundary. Phase 2 ends when the guest listener surface is reduced to the Unix socket contract, host-to-guest mounts are explicitly minimized, and the guest service definition is unified behind one hardened source of truth. +Tighten the current macOS Lima deployment so the same-user model exposes the smallest practical guest and host attack surface without pretending to provide the Linux ownership boundary. Phase 2 ends when the guest listener surface is reduced, host-to-guest mounts are explicitly minimized, and the guest service definition is unified behind one hardened source of truth. + +This phase does not invent new operator surfaces. It hardens the already-supported Lima-backed path that today is exercised through `substrate host doctor`, `substrate world doctor`, `substrate world gateway sync|status|restart`, `scripts/mac/smoke.sh`, and `scripts/mac/orchestration-smoke.sh`. ## Why this phase exists -As of 2026-04-28, macOS can match most Linux world behavior but not Linux’s `root:substrate` ownership boundary because Lima currently runs as the same host user that requested it. The remaining risk is not a single bug; it is a posture problem spread across multiple repo seams: +As of HEAD, macOS can match much of Linux's world behavior but not Linux's `root:substrate` ownership boundary because Lima still runs as the same host user that requested it. The remaining risk is not a single bug; it is a posture problem spread across a few concrete seams: - `scripts/mac/lima-warm.sh` still writes `Environment=SUBSTRATE_AGENT_TCP_PORT=61337`, leaving a default extra listener path enabled in the guest. -- `scripts/mac/lima/substrate.yaml` still mounts all of `$HOME` read-only, which is broader than the runtime contract actually needs. -- Guest service definitions are duplicated between the provisioning YAML and the warm script, which makes sandbox drift likely. -- Current docs normalize direct in-guest administration and transport-debug habits that are acceptable for breakglass but not for the hardened default. +- `scripts/mac/lima/substrate.yaml` still mounts broad host state, including a read-only `$HOME` mount that is wider than the hardened contract should require. +- Guest service definitions are still duplicated between the provisioning YAML and the warm script, which makes sandbox drift likely. +- Current docs still normalize direct in-guest setup and troubleshooting habits that are acceptable for breakglass but not for the hardened default. +- `SUBSTRATE_WORLD_SOCKET` still exists as an advanced/test/breakglass bypass and should stay outside the normal same-user Lima contract. -Phase 2 exists to reduce that spread before Phase 3 moves lifecycle and diagnostics behind Substrate-owned commands. +Phase 2 exists to reduce that spread before Phase 3 finishes the operator-surface cutover. ## In-scope - Remove default extra listener paths on macOS/Lima where the hardened contract only requires `/run/substrate.sock`. - Define and implement a reduced mount posture for the Lima guest, including explicit ingress requirements for source checkouts, auth material, and runtime artifacts. -- Unify macOS guest systemd unit generation so the service sandbox and socket-activation contract are defined once and consumed consistently. -- Tighten smoke, doctor, and documentation evidence so same-user hardening regressions are visible. +- Unify macOS guest systemd unit generation so service sandboxing and socket-activation settings are defined once and consumed consistently. +- Tighten smoke, doctor, gateway, and orchestration evidence so same-user hardening regressions are visible. ## Out-of-scope @@ -38,62 +41,64 @@ Phase 2 exists to reduce that spread before Phase 3 moves lifecycle and diagnost Phase 2 keeps the same-user Lima architecture but narrows it in three successive steps: 1. Remove listener sprawl so `world-agent` is consumed through the Unix socket and inherited socket-activation path by default. -2. Replace “mount broad host state and rely on convention” with an explicit ingress contract that names what the guest is allowed to see and why. +2. Replace "mount broad host state and rely on convention" with an explicit ingress contract that names what the guest is allowed to see and why. 3. Collapse duplicated guest service definitions into one generated contract so hardening changes apply identically whether the VM is created from the profile or repaired by the warm script. -The core rule for the phase is that macOS should behave like Linux wherever behavior parity is possible, and should explicitly document the remaining ownership-gap where parity is not possible. +The core rule for the phase is that macOS should behave like Linux wherever behavior parity is possible, and should explicitly document the remaining non-parity point as same-user Lima ownership, not as a diffuse set of extra listeners, broad mounts, or doc-endorsed bypasses. ## Dependencies / sequencing - This phase assumes the existing Lima backend remains the active macOS world backend. -- This phase depends on Phase 1 milestone 1.1 through milestone 1.3 being - complete first, so transport, policy, and readiness contracts are stable - before listener, mount, and unit hardening start encoding them. -- Milestone 2.1 must land before 2.3 because unit unification should encode the reduced listener surface, not preserve the old one. +- This phase depends on Phase 1 milestone 1.1 through milestone 1.3 being complete first, so transport, backend-policy, shared-world, and readiness contracts are stable before listener, mount, and unit hardening start encoding them. +- Milestone 2.1 should land before 2.3 because unit unification should encode the reduced listener surface, not preserve the old one. - Milestone 2.2 should land before 2.3 if the sandbox `ReadWritePaths`, environment, or guest-visible paths depend on the new ingress contract. -- Phase 3 depends on this phase because Substrate-owned lifecycle commands should not entrench today’s wider listener, mount, or admin surface. +- Phase 3 depends on this phase because CLI-owned lifecycle and diagnostics should not entrench today's wider listener, mount, or admin surface. ## Concrete repo surfaces and file pointers -- [scripts/mac/lima-warm.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/lima-warm.sh) +- `scripts/mac/lima-warm.sh` - writes guest service and socket units - currently injects `SUBSTRATE_AGENT_TCP_PORT=61337` - currently repairs the VM through direct `limactl shell` administration -- [scripts/mac/lima/substrate.yaml](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/lima/substrate.yaml) +- `scripts/mac/lima/substrate.yaml` - defines current guest mounts + - still includes broad host visibility - embeds a second copy of the `substrate-world-agent` unit -- [crates/world-agent/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/lib.rs) +- `crates/world-agent/src/lib.rs` - binds the Unix socket by default - conditionally enables loopback TCP via `SUBSTRATE_AGENT_TCP_PORT` - - already supports inherited socket activation listeners -- [scripts/mac/lima-doctor.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/lima-doctor.sh) - - currently validates guest health through direct `limactl shell` probes -- [scripts/mac/smoke.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/smoke.sh) - - currently exercises Lima-backed conformance and can become the primary regression harness for the hardening work -- [docs/WORLD.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/WORLD.md) - - currently documents transport selection and direct guest administration -- [docs/cross-platform/mac_world_setup.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/cross-platform/mac_world_setup.md) - - currently teaches direct `limactl shell` lifecycle, install, service, and troubleshooting flows + - already supports inherited socket-activation listeners +- `scripts/mac/lima-doctor.sh` + - currently validates guest health through direct `limactl shell` probes more than the hardened default should +- `scripts/mac/smoke.sh` + - exercises doctor/gateway/runtime parity and can remain the primary regression harness for the hardening work +- `scripts/mac/orchestration-smoke.sh` + - already validates shared-owner/orchestration flows and should keep doing so through hardening +- `docs/WORLD.md` + - documents transport selection and the current `SUBSTRATE_WORLD_SOCKET` bypass behavior +- `docs/cross-platform/mac_world_setup.md` + - still teaches direct `limactl shell` setup, service, and troubleshooting flows too prominently ## Deliverables - One phase packet that sequences the same-user hardening work into milestones 2.1 through 2.3. - A hardened listener contract for macOS that removes default TCP exposure. -- A mount and ingress contract for the Lima guest that is narrower than “all of `$HOME` plus `/src`”. +- A mount and ingress contract for the Lima guest that is narrower than "broad `$HOME` plus project convenience mounts." - One source of truth for guest `substrate-world-agent` unit and socket definitions. -- Updated evidence expectations for doctor, smoke, and doc review. +- Updated evidence expectations for doctor, gateway, smoke, orchestration, and doc review. ## Acceptance criteria - Every default macOS/Lima startup path uses the Unix socket contract without requiring `SUBSTRATE_AGENT_TCP_PORT`. -- The Lima profile and repair flow no longer mount all of `$HOME` by default unless a narrowly justified exception is explicitly documented and tested. +- The Lima profile and repair flow no longer mount broad host `$HOME` state by default unless a narrowly justified exception is explicitly documented and tested. - There is one authoritative guest service/unit definition path for macOS hardening-critical settings. -- A reviewer can identify the remaining non-parity point with Linux as “same-user Lima ownership,” not as a diffuse set of extra listeners, broad mounts, or doc-endorsed manual bypasses. +- `SUBSTRATE_WORLD_SOCKET` remains available only as an advanced/test/breakglass override, not as the documented standard path. +- A reviewer can identify the remaining non-parity point with Linux as same-user Lima ownership, not as a diffuse set of extra listeners, broad mounts, or doc-endorsed manual bypasses. ## Validation / evidence plan - Diff and review the rendered or generated guest unit contents to prove the TCP listener default is gone and the sandbox settings are unified. -- Capture `scripts/mac/lima-warm.sh --check-only`, `scripts/mac/lima-doctor.sh`, and `scripts/mac/smoke.sh` evidence before and after the changes. +- Capture `scripts/mac/lima-warm.sh --check-only`, `scripts/mac/lima-doctor.sh`, `scripts/mac/smoke.sh`, and `scripts/mac/orchestration-smoke.sh` evidence before and after the changes. - Probe `world-agent` listener mode through logs or explicit checks so the evidence distinguishes socket-activation-only operation from TCP-enabled operation. - Review `docs/WORLD.md` and `docs/cross-platform/mac_world_setup.md` for direct-admin guidance that still contradicts the hardened default. @@ -101,7 +106,7 @@ The core rule for the phase is that macOS should behave like Linux wherever beha - Some host-to-guest flows may currently rely on the broad `$HOME` mount in ways that are not yet explicit, especially auth or toolchain bootstrap paths. - Removing the default TCP listener may expose hidden forwarding or gateway assumptions on macOS. -- Unit unification can fail if the repo still needs a “bootstrap profile unit” and a “repair unit”; if that happens, the generated source of truth must still be singular. +- Unit unification can fail if the repo still needs a bootstrap profile unit and a repair unit; if that happens, the generated source of truth must still be singular. - The exact ingress contract for auth files, SSH material, and package-manager caches remains to be frozen in milestone 2.2. ## Milestones diff --git a/macos-hardening/macos-hardened-same-user-lima/phase-2-same-user-hardening/milestone-2-1-remove-extra-listeners-and-tighten-agent-surface-sow.md b/macos-hardening/macos-hardened-same-user-lima/phase-2-same-user-hardening/milestone-2-1-remove-extra-listeners-and-tighten-agent-surface-sow.md index bf1b9da9a..3b7f8322a 100644 --- a/macos-hardening/macos-hardened-same-user-lima/phase-2-same-user-hardening/milestone-2-1-remove-extra-listeners-and-tighten-agent-surface-sow.md +++ b/macos-hardening/macos-hardened-same-user-lima/phase-2-same-user-hardening/milestone-2-1-remove-extra-listeners-and-tighten-agent-surface-sow.md @@ -4,13 +4,28 @@ Draft +Last updated: 2026-05-19 + ## Purpose / outcome -Remove the default guest TCP listener posture from macOS/Lima so the hardened default is one world-agent transport contract: the Unix domain socket at `/run/substrate.sock`, preferably inherited through socket activation. +Remove the default guest TCP listener posture from macOS/Lima so the hardened +default is one world-agent transport contract: the Unix domain socket at +`/run/substrate.sock`, preferably inherited through socket activation. ## Why this milestone exists -The current macOS warm flow writes `Environment=SUBSTRATE_AGENT_TCP_PORT=61337` into the guest service in [scripts/mac/lima-warm.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/lima-warm.sh). In [crates/world-agent/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/lib.rs), that environment variable enables a loopback TCP listener whenever one was not inherited from socket activation. That widens the guest attack surface even though the documented contract centers on `/run/substrate.sock`. +The current macOS warm flow writes `Environment=SUBSTRATE_AGENT_TCP_PORT=61337` +into the guest service in `scripts/mac/lima-warm.sh`. In +`crates/world-agent/src/lib.rs`, that environment variable enables a loopback +TCP listener whenever one was not inherited from socket activation. That widens +the guest attack surface even though the actual transport story is already +UDS-backed and centered on `/run/substrate.sock`. + +This milestone is not about inventing a new transport baseline. The current +runtime already forwards into the guest UDS and intentionally skips SSH TCP +fallback. The remaining cleanup is to remove the extra guest listener and to +stop treating retained host TCP `17788` probing as evidence that guest TCP is a +supported contract. This milestone exists to make the listener surface match the intended contract before any lifecycle tooling is built on top of it. @@ -21,7 +36,8 @@ This milestone exists to make the listener surface match the intended contract b raw TCP path is breakglass/unsupported and not part of the supported or degraded-but-supported runtime contract.” - Update doctor, smoke, and docs so they validate or describe the hardened listener posture instead of silently tolerating the old one. -- Identify any transport code paths that still assume TCP fallback is always present. +- Identify any transport or doctor code paths that still assume guest TCP + fallback is always present. ## Out-of-scope @@ -34,6 +50,9 @@ This milestone exists to make the listener surface match the intended contract b - Keep `world-agent` support for optional TCP listeners in shared runtime code, but stop enabling it in the macOS hardened default. - Treat TCP on macOS as a breakglass-only exception, not as background service behavior or a supported compatibility mode. +- Treat host TCP `17788` probing as separate from guest TCP listener posture: + if it remains temporarily for VSock-proxy compatibility or doctor fallback, + it must not be described as a guest listener contract. - Align the warm script, doctor checks, smoke coverage, and operator docs around one default listener posture so later Phase 3 commands do not have to explain mixed semantics. ## Dependencies / sequencing @@ -44,19 +63,23 @@ This milestone exists to make the listener surface match the intended contract b ## Concrete repo surfaces and file pointers -- [scripts/mac/lima-warm.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/lima-warm.sh) +- `scripts/mac/lima-warm.sh` - remove the service-level `SUBSTRATE_AGENT_TCP_PORT=61337` default - update check-only output if it currently assumes that env exists -- [crates/world-agent/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/lib.rs) +- `crates/world-agent/src/lib.rs` - confirm the runtime behavior when the TCP env var is absent - preserve explicit opt-in semantics if shared platforms still need TCP -- [docs/WORLD.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/WORLD.md) +- `crates/world-mac-lima/src/forwarding.rs` + - preserve the UDS-backed forwarding contract and intentional lack of SSH TCP fallback +- `crates/shell/src/execution/platform/macos.rs` + - ensure doctor fallback behavior does not get mistaken for a guest TCP contract +- `docs/WORLD.md` - update macOS transport text so TCP is no longer presented as part of the default guest listener surface -- [docs/cross-platform/mac_world_setup.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/cross-platform/mac_world_setup.md) +- `docs/cross-platform/mac_world_setup.md` - remove or reframe guidance that implies the guest service normally exposes a TCP listener -- [scripts/mac/lima-doctor.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/lima-doctor.sh) +- `scripts/mac/lima-doctor.sh` - add or refine checks that show the UDS contract is healthy without relying on TCP -- [scripts/mac/smoke.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/smoke.sh) +- `scripts/mac/smoke.sh` - ensure the smoke path proves the macOS backend still works after the TCP default is removed ## Deliverables @@ -71,6 +94,8 @@ This milestone exists to make the listener surface match the intended contract b - The macOS warm/provision flow no longer injects `SUBSTRATE_AGENT_TCP_PORT=61337` by default. - A fresh or repaired Lima guest starts `world-agent` successfully with only `/run/substrate.sock` exposed by default. - macOS doctor and smoke evidence remains green without depending on the guest TCP listener. +- `substrate world gateway sync|status|restart` and gateway lifecycle smoke + coverage remain green without depending on the guest TCP listener. - Any remaining raw TCP path on macOS is explicitly labeled breakglass/unsupported rather than supported, degraded-but-supported, or compatibility mode. @@ -80,6 +105,8 @@ This milestone exists to make the listener surface match the intended contract b - Capture the rendered guest service unit before and after the change and prove the TCP env line is absent. - Run `scripts/mac/lima-doctor.sh` and confirm health is established through the socket path and systemd/socket-activation state, not a TCP port probe. - Run `scripts/mac/smoke.sh` and any transport-specific smoke needed to prove replay, PTY, and gateway flows still work. +- Run `substrate world gateway status --json` and confirm the managed gateway + path still resolves through the supported routed transport surface. - Inspect `world-agent` startup logs for `listener_kind = "tcp"` and `listener_mode` changes to prove the hardened default no longer enables direct-bind TCP. ## Risks / open questions diff --git a/macos-hardening/macos-hardened-same-user-lima/phase-2-same-user-hardening/milestone-2-2-mount-minimization-and-ingress-contract-sow.md b/macos-hardening/macos-hardened-same-user-lima/phase-2-same-user-hardening/milestone-2-2-mount-minimization-and-ingress-contract-sow.md index f41ce790f..200715e33 100644 --- a/macos-hardening/macos-hardened-same-user-lima/phase-2-same-user-hardening/milestone-2-2-mount-minimization-and-ingress-contract-sow.md +++ b/macos-hardening/macos-hardened-same-user-lima/phase-2-same-user-hardening/milestone-2-2-mount-minimization-and-ingress-contract-sow.md @@ -4,13 +4,27 @@ Draft +Last updated: 2026-05-19 + ## Purpose / outcome Replace the current broad Lima mount posture with a named ingress contract that states exactly which host inputs the guest may consume for normal operation, how they are mounted, and which flows must move to explicit Substrate-owned copying, syncing, or breakglass procedures. ## Why this milestone exists -The current Lima profile in [scripts/mac/lima/substrate.yaml](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/lima/substrate.yaml) mounts all of `$HOME` read-only and mounts the active project at `/src` read-write. That is convenient for bootstrapping, but it overexposes host content and leaves the real runtime contract underspecified. The same-user model already cannot replicate Linux ownership boundaries; broad host mounts make that limitation materially worse. +The current Lima profile in `scripts/mac/lima/substrate.yaml` mounts all of +`$HOME` read-only and mounts the active project at `/src` read-write. That is +convenient for bootstrapping, but it overexposes host content and leaves the +real runtime contract underspecified. The same-user model already cannot +replicate Linux ownership boundaries; broad host mounts make that limitation +materially worse. + +The repo has also outgrown one of the old justifications for broad home +visibility: managed gateway auth handoff now exists. Gateway lifecycle requests +already carry integrated auth payloads, and managed runtime artifacts already +live under `/run/substrate/substrate-gateway-runtime/`. Hardening should not +continue to rely on broad guest visibility into host home directories for normal +gateway operation. This milestone exists to shrink what the guest can see and to name the remaining required ingress paths instead of hiding them behind a full-home mount. @@ -19,6 +33,8 @@ This milestone exists to shrink what the guest can see and to name the remaining - Define the normal-operation ingress contract for the macOS Lima guest. - Decide which host inputs remain mounted, which become copied or synchronized by Substrate-owned flows, and which are only available via breakglass. - Narrow the default Lima profile mounts accordingly. +- Preserve already-landed shared-world/orchestration and gateway lifecycle + support while narrowing ingress. - Update docs and validation to reflect the new ingress model. ## Out-of-scope @@ -39,7 +55,19 @@ This milestone exists to shrink what the guest can see and to name the remaining - direct mount - Substrate-managed copy/sync - no default ingress -- Encode the result in the Lima profile and in the surrounding docs so the hardening stance is inspectable. +- Encode the result in the Lima profile and in the surrounding docs so the + hardening stance is inspectable. + +Specific ingress guidance for current repo truth: + +- Gateway auth should prefer request-provided integrated auth handoff over broad + host-home visibility. +- Managed gateway lifecycle/status and runtime artifacts under + `/run/substrate/substrate-gateway-runtime/` are supported surfaces that must + keep working after mount minimization. +- Host-side `SUBSTRATE_WORLD_SOCKET` override use is not a normal ingress model; + it is advanced/test/breakglass and should not drive the supported mount + contract. ## Dependencies / sequencing @@ -50,20 +78,24 @@ This milestone exists to shrink what the guest can see and to name the remaining ## Concrete repo surfaces and file pointers -- [scripts/mac/lima/substrate.yaml](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/lima/substrate.yaml) +- `scripts/mac/lima/substrate.yaml` - current home and project mount definitions - likely primary place where mount minimization is expressed -- [scripts/mac/lima-warm.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/lima-warm.sh) +- `scripts/mac/lima-warm.sh` - currently assumes `/src` exists and that the mounted repo can be built or copied from inside the guest - contains sentinel logic that detects whether the intended host checkout is mounted -- [scripts/mac/smoke.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/smoke.sh) +- `scripts/mac/smoke.sh` - depends on the mounted repo and may expose additional ingress assumptions -- [docs/WORLD.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/WORLD.md) +- `crates/shell/src/builtins/world_gateway.rs` + - already constructs integrated auth handoff and managed gateway lifecycle requests +- `crates/world-agent/src/gateway_runtime.rs` + - owns managed gateway runtime artifacts under `/run/substrate/substrate-gateway-runtime/` +- `docs/WORLD.md` - currently states that `/src` mirrors the active checkout - should be tightened to describe only the allowed ingress -- [docs/cross-platform/mac_world_setup.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/cross-platform/mac_world_setup.md) +- `docs/cross-platform/mac_world_setup.md` - currently documents the full-home mount as part of normal setup -- [crates/world-agent/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/lib.rs) +- `crates/world-agent/src/lib.rs` - not a mount definition point, but a useful anchor for separating transport hardening from guest-visible filesystem scope ## Deliverables @@ -77,7 +109,10 @@ This milestone exists to shrink what the guest can see and to name the remaining - The hardened macOS plan no longer depends on mounting all of `$HOME` by default. - Every remaining default host mount is justified by a named runtime need and file pointer. -- Flows that still need broader host visibility are explicitly categorized as future Substrate-owned sync work or breakglass operations. +- Flows that still need broader host visibility are explicitly categorized as + future Substrate-owned sync work or breakglass operations. +- Managed gateway auth and runtime do not depend on broad guest visibility into + host home directories for the supported path. - A reviewer can tell which guest-visible host paths remain after hardening without reading the Lima profile line-by-line. ## Validation / evidence plan @@ -85,6 +120,9 @@ This milestone exists to shrink what the guest can see and to name the remaining - Produce a before/after mount inventory from `scripts/mac/lima/substrate.yaml`. - Run `scripts/mac/lima-warm.sh --check-only` and a standard warm path to prove the narrowed mounts still support the intended happy path. - Run `scripts/mac/smoke.sh` to identify any hidden assumptions about `$HOME`, auth locations, or tool caches. +- Run `substrate world gateway sync` and `substrate world gateway status --json` + after mount minimization to confirm integrated auth handoff and managed + runtime artifacts still work without broad `$HOME` ingress. - Review docs for commands that still imply “the guest can see my host home directory” and either remove or reclassify them. ## Risks / open questions diff --git a/macos-hardening/macos-hardened-same-user-lima/phase-2-same-user-hardening/milestone-2-3-guest-service-sandbox-and-unit-unification-sow.md b/macos-hardening/macos-hardened-same-user-lima/phase-2-same-user-hardening/milestone-2-3-guest-service-sandbox-and-unit-unification-sow.md index 6b895ec2a..88a21750b 100644 --- a/macos-hardening/macos-hardened-same-user-lima/phase-2-same-user-hardening/milestone-2-3-guest-service-sandbox-and-unit-unification-sow.md +++ b/macos-hardening/macos-hardened-same-user-lima/phase-2-same-user-hardening/milestone-2-3-guest-service-sandbox-and-unit-unification-sow.md @@ -4,24 +4,42 @@ Draft +Last updated: 2026-05-19 + ## Purpose / outcome -Unify the macOS Lima guest `substrate-world-agent` service and socket definitions behind one authoritative source so the hardened sandbox, socket-activation contract, and environment are consistent across VM creation, repair, and diagnostics. +Unify the macOS Lima guest `substrate-world-agent` service and socket +definitions behind one authoritative source so the hardened sandbox, +socket-activation contract, and environment are consistent across VM creation, +repair, and diagnostics. ## Why this milestone exists Today the repo defines guest units in at least two places: -- [scripts/mac/lima/substrate.yaml](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/lima/substrate.yaml) -- [scripts/mac/lima-warm.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/lima-warm.sh) +- `scripts/mac/lima/substrate.yaml` +- `scripts/mac/lima-warm.sh` Those copies are already divergent in capability sets and environment lines. Hardening work cannot be trusted if the initial VM profile and the repair path can recreate different sandboxes. This milestone exists to make the guest unit definition a single hardening authority and to align it with the reduced listener and mount contract established in milestones 2.1 and 2.2. +Current repo truth also means the unit contract has to account for more than +the socket alone: + +- managed gateway runtime artifacts already live under + `/run/substrate/substrate-gateway-runtime/` +- gateway lifecycle smoke coverage already exists +- integrated auth handoff already exists + +So the unified service sandbox must preserve the supported gateway lifecycle +surface while tightening listener, mount, and writable-path posture. + ## In-scope - Define one authoritative source for the guest service and socket unit contents. - Eliminate hardening-critical drift across provisioning and repair paths. - Reconcile sandbox settings such as `ProtectHome`, `ReadWritePaths`, capabilities, runtime directories, and environment variables with the Phase 2 hardening goals. +- Reconcile those settings with the already-landed managed gateway runtime + surface under `/run/substrate/substrate-gateway-runtime/`. - Update doctor and documentation references so operators know which unit definition is authoritative. ## Out-of-scope @@ -39,7 +57,16 @@ Those copies are already divergent in capability sets and environment lines. Har - service group and runtime directory settings - capability bounding and ambient capabilities - writable guest paths -- Use the same generated unit content for create, repair, doctor expectations, and doc examples. +- Use the same generated unit content for create, repair, doctor expectations, + gateway lifecycle expectations, and doc examples. + +The unified contract must explicitly encode: + +- no default guest TCP listener injection for the hardened path +- writable/runtime paths required for `/run/substrate.sock` and + `/run/substrate/substrate-gateway-runtime/` +- only the minimum environment needed for supported world-agent and managed + gateway operation ## Dependencies / sequencing @@ -53,16 +80,20 @@ Those copies are already divergent in capability sets and environment lines. Har ## Concrete repo surfaces and file pointers -- [scripts/mac/lima/substrate.yaml](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/lima/substrate.yaml) +- `scripts/mac/lima/substrate.yaml` - bootstraps guest package install and unit files during VM creation -- [scripts/mac/lima-warm.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/lima-warm.sh) +- `scripts/mac/lima-warm.sh` - currently rewrites the service and socket units during warm/repair - currently injects hardening-relevant environment and capability settings -- [scripts/mac/lima-doctor.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/lima-doctor.sh) +- `crates/world-agent/src/gateway_runtime.rs` + - authoritative runtime-artifact expectations for `/run/substrate/substrate-gateway-runtime/` +- `scripts/mac/lima-doctor.sh` - should verify the unified unit contract rather than loosely checking partial state -- [docs/WORLD.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/WORLD.md) +- `scripts/mac/smoke.sh` + - should keep proving gateway lifecycle against the unified service contract +- `docs/WORLD.md` - references guest unit behavior and operator verification -- [docs/cross-platform/mac_world_setup.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/cross-platform/mac_world_setup.md) +- `docs/cross-platform/mac_world_setup.md` - currently includes manual service enable/start flows that must match the unified definition ## Deliverables @@ -76,14 +107,19 @@ Those copies are already divergent in capability sets and environment lines. Har - There is exactly one authoritative macOS guest unit definition path for `substrate-world-agent.service` and `substrate-world-agent.socket`. - Hardening-critical settings are identical whether the guest is freshly created or repaired. -- Doctor and troubleshooting guidance point to the unified unit contract instead of to whichever path happened to run last. +- Doctor, gateway lifecycle, and troubleshooting guidance point to the unified + unit contract instead of to whichever path happened to run last. - A reviewer can explain the guest service sandbox from one source file or generator rather than diffing two handwritten unit bodies. ## Validation / evidence plan - Produce a side-by-side diff of the current YAML-embedded unit and warm-script unit as the baseline evidence. - Define a rendered-unit verification step for macOS doctor or smoke coverage. -- Re-run `scripts/mac/lima-warm.sh`, `scripts/mac/lima-doctor.sh`, and `scripts/mac/smoke.sh` after unification to prove the created and repaired VM paths converge. +- Re-run `scripts/mac/lima-warm.sh`, `scripts/mac/lima-doctor.sh`, and + `scripts/mac/smoke.sh` after unification to prove the created and repaired VM + paths converge. +- Re-run `substrate world gateway sync|status|restart` and confirm the managed + runtime artifacts and support path stay healthy after unit unification. - Review doc examples to ensure any pasted unit snippets or systemctl guidance match the unified contract. ## Risks / open questions diff --git a/macos-hardening/macos-hardened-same-user-lima/phase-3-substrate-owned-operations/README.md b/macos-hardening/macos-hardened-same-user-lima/phase-3-substrate-owned-operations/README.md index 6dadba4c2..3d0203564 100644 --- a/macos-hardening/macos-hardened-same-user-lima/phase-3-substrate-owned-operations/README.md +++ b/macos-hardening/macos-hardened-same-user-lima/phase-3-substrate-owned-operations/README.md @@ -6,27 +6,36 @@ Draft ## Purpose / outcome -Move macOS Lima lifecycle, diagnostics, and operator guidance from direct guest administration toward explicit Substrate-owned commands and breakglass boundaries. Phase 3 ends when the normal operator path no longer depends on raw `limactl shell substrate ...`, direct guest systemctl manipulation, or ad hoc socket probing as the primary workflow. +Finish moving macOS Lima lifecycle, diagnostics, and operator guidance from direct guest administration toward explicit Substrate-owned commands and clearly bounded breakglass procedures. Phase 3 ends when the normal operator path no longer depends on raw `limactl shell substrate ...`, direct guest `systemctl` manipulation, or ad hoc socket probing as the primary workflow. + +This phase is an operational cutover, not an invention phase. The canonical CLI surfaces already exist in HEAD: + +- `substrate host doctor` +- `substrate world doctor` +- `substrate world gateway sync` +- `substrate world gateway status` +- `substrate world gateway restart` ## Why this phase exists -The current macOS backend works, but many repo surfaces still normalize direct guest operations: +The current macOS backend works, but many repo surfaces still normalize direct guest operations even though the CLI-owned path already exists: -- [docs/cross-platform/mac_world_setup.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/cross-platform/mac_world_setup.md) includes step-by-step `limactl shell substrate ...` build, install, service, and troubleshooting flows. -- [docs/WORLD.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/WORLD.md) still points operators to direct guest administration even while acknowledging CLI doctor surfaces. -- [scripts/mac/lima-doctor.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/lima-doctor.sh) is a guest-admin-oriented diagnostic script rather than a thin wrapper over a Substrate-owned contract. -- [scripts/mac/lima-warm.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/lima-warm.sh) relies heavily on direct `limactl shell` mutation, which is functional but not the posture we want to normalize. +- `docs/cross-platform/mac_world_setup.md` still includes step-by-step `limactl shell substrate ...` build, install, service, and troubleshooting flows. +- `docs/WORLD.md` already names CLI doctor surfaces, shared-world support, and the `SUBSTRATE_WORLD_SOCKET` bypass, but it still preserves too much direct guest guidance for the hardened default. +- `scripts/mac/lima-doctor.sh` remains a guest-admin-oriented diagnostic script rather than a thin wrapper around a fully Substrate-owned support story. +- `scripts/mac/lima-warm.sh` still relies heavily on direct `limactl shell` mutation, which is functional but not the posture that operator docs should normalize. +- `scripts/mac/smoke.sh` and `scripts/mac/orchestration-smoke.sh` already prove important parts of the supported path, but the top-level docs still present too much of the exception path as routine setup. -That matters because a hardened same-user Lima deployment should make “what Substrate owns” legible. Phase 2 narrows the technical surface. Phase 3 narrows the operational surface. +That matters because a hardened same-user Lima deployment should make "what Substrate owns" legible. Phase 2 narrows the technical surface. Phase 3 narrows the operational surface. ## In-scope -- Define the normal macOS lifecycle and diagnostics actions that should be owned by Substrate commands. -- Define the normal macOS ingress and sync actions that should be owned by - Substrate commands after Phase 2 mount minimization. +- Define the normal macOS lifecycle and diagnostics actions that should be taught through Substrate commands first. +- Define the normal macOS ingress and sync actions that should be taught through Substrate commands after Phase 2 mount minimization. - Reclassify direct guest administration as breakglass where it remains necessary. +- Reclassify `SUBSTRATE_WORLD_SOCKET` as an advanced/test/breakglass bypass rather than a standard Lima operator workflow. - Cut documentation over so the primary operator story runs through Substrate-owned commands first. -- Align doctor, smoke, and troubleshooting evidence around the new operational boundary. +- Align doctor, gateway, smoke, orchestration, and troubleshooting evidence around the new operational boundary. ## Out-of-scope @@ -37,70 +46,73 @@ That matters because a hardened same-user Lima deployment should make “what Su ## Architectural approach -Phase 3 treats the Lima guest as an internal substrate implementation detail for normal operations. The user-visible contract becomes: +Phase 3 treats the Lima guest as an internal implementation detail for normal operations. The user-visible contract becomes: -1. Substrate-owned commands decide when the VM must exist, start, sync, - diagnose, or repair. +1. Substrate-owned commands decide when the VM must exist, start, sync, diagnose, or repair. 2. Remaining fallback flows are classified using the phase-0 taxonomy: - supported, degraded-but-supported, and breakglass/unsupported. - Direct guest administration stays in the breakglass/unsupported bucket; - degraded-but-supported applies only to compatibility wrappers around - Substrate-owned flows, not to raw guest procedures. -3. Direct `limactl shell` access is preserved only for explicit breakglass or - deep debugging. -4. Docs, scripts, and evidence surfaces teach the Substrate-owned path first - and label manual guest administration as exception handling. + - supported + - degraded-but-supported + - breakglass/unsupported +3. Direct `limactl shell` access is preserved only for explicit breakglass or deep debugging. +4. `SUBSTRATE_WORLD_SOCKET` remains available for advanced testing and breakglass bypass use, but it is not documented as the standard Lima path and must stay outside the normal shared-owner/orchestration story. +5. Docs, scripts, and evidence surfaces teach the Substrate-owned path first and label manual guest administration as exception handling. This phase does not require hiding Lima from advanced users. It requires reclassifying Lima administration from the default happy path to a controlled fallback. ## Dependencies / sequencing -- Phase 2 must land first because Substrate-owned lifecycle commands should manage the hardened listener, mount, and unit contracts rather than today’s wider posture. +- Phase 2 must land first because CLI-owned lifecycle commands should manage the hardened listener, mount, and unit contracts rather than today's wider posture. - Milestone 3.1 comes before 3.2 because the docs cutover should follow a concrete owned command surface. +- The current doctor/gateway/shared-world surfaces are baseline inputs to this phase, not open design questions. ## Concrete repo surfaces and file pointers -- [scripts/mac/lima-warm.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/lima-warm.sh) +- `scripts/mac/lima-warm.sh` - current lifecycle/provisioning authority -- [scripts/mac/lima-doctor.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/lima-doctor.sh) +- `scripts/mac/lima-doctor.sh` - current diagnostic authority -- [scripts/mac/smoke.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/smoke.sh) - - should validate the owned-operations path after the cutover -- [docs/WORLD.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/WORLD.md) - - documents the runtime and operator model -- [docs/cross-platform/mac_world_setup.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/cross-platform/mac_world_setup.md) - - currently teaches direct Lima administration +- `scripts/mac/smoke.sh` + - already validates doctor/gateway/runtime behavior on the Lima-backed path +- `scripts/mac/orchestration-smoke.sh` + - already validates shared-owner/orchestration behavior on the Lima-backed path +- `docs/WORLD.md` + - documents the runtime model, shared-world behavior, and `SUBSTRATE_WORLD_SOCKET` bypass +- `docs/cross-platform/mac_world_setup.md` + - still teaches direct Lima administration too prominently +- `docs/USAGE.md` + - already treats `substrate world gateway sync|status|restart` as operator entrypoints +- `docs/contracts/substrate-gateway-operator-contract.md` + - already treats `substrate world gateway status --json` as the authoritative machine-readable wiring surface ## Deliverables - One phase packet sequencing the operational cutover into milestones 3.1 and 3.2. -- A defined Substrate-owned lifecycle and diagnostics surface for hardened same-user Lima. -- A defined Substrate-owned ingress and sync surface that replaces any - remaining dependence on broad convenience mounts for normal operation. -- A breakglass policy for remaining direct guest operations. +- A defined Substrate-owned lifecycle and diagnostics story for hardened same-user Lima. +- A defined Substrate-owned ingress and sync story that replaces any remaining dependence on broad convenience mounts for normal operation. +- A breakglass policy for remaining direct guest operations and bypasses. - Documentation and validation aligned to the new operational boundary. ## Acceptance criteria - The normal macOS world lifecycle and diagnostic story is driven by Substrate-owned commands, not by raw `limactl shell` recipes. -- The normal macOS workspace ingress and sync story is driven by - Substrate-owned commands, not by implicit broad mounts or ad hoc guest steps. +- The normal macOS workspace ingress and sync story is driven by Substrate-owned commands, not by implicit broad mounts or ad hoc guest steps. - Direct guest operations that remain are explicitly labeled breakglass and bounded. -- Doctor, smoke, and troubleshooting evidence can be captured through the owned path with manual guest access reserved for exceptions. +- `SUBSTRATE_WORLD_SOCKET` is documented as an advanced/test/breakglass bypass, not the standard Lima-backed path. +- Doctor, gateway, smoke, orchestration, and troubleshooting evidence can be captured through the owned path with manual guest access reserved for exceptions. - A reviewer can distinguish implementation detail from operator contract in the macOS docs. ## Validation / evidence plan - Inventory every current direct guest administration command in the macOS docs and scripts. - Define which ones become owned commands, which ones remain internal implementation details, and which ones are breakglass only. -- Re-run doctor and smoke coverage using the owned command path after the cutover plan is implemented. +- Re-run doctor, gateway, and smoke/orchestration coverage using the owned command path after the cutover plan is implemented. - Review doc text for any remaining places that still present direct Lima administration as the default happy path. ## Risks / open questions - Some diagnostics may still require direct guest access until Substrate exposes enough structured detail. -- The repo may need transitional wrappers before the final CLI contract is stable. -- “Breakglass” needs a precise meaning in docs: allowed for deep debugging, but not the default install or day-to-day maintenance path. +- The repo may need transitional wrappers before the final CLI contract is fully simplified. +- "Breakglass" needs a precise meaning in docs: allowed for deep debugging, but not the default install or day-to-day maintenance path. ## Milestones diff --git a/macos-hardening/macos-hardened-same-user-lima/phase-3-substrate-owned-operations/milestone-3-1-substrate-managed-diagnostics-and-lifecycle-sow.md b/macos-hardening/macos-hardened-same-user-lima/phase-3-substrate-owned-operations/milestone-3-1-substrate-managed-diagnostics-and-lifecycle-sow.md index 1e95016ee..8740d409c 100644 --- a/macos-hardening/macos-hardened-same-user-lima/phase-3-substrate-owned-operations/milestone-3-1-substrate-managed-diagnostics-and-lifecycle-sow.md +++ b/macos-hardening/macos-hardened-same-user-lima/phase-3-substrate-owned-operations/milestone-3-1-substrate-managed-diagnostics-and-lifecycle-sow.md @@ -4,17 +4,31 @@ Draft +Last updated: 2026-05-19 + ## Purpose / outcome -Define and implement the macOS/Lima lifecycle and diagnostic actions that Substrate must own directly so operators can create, warm, check, repair, and inspect the backend without relying on raw guest administration commands. +Define and implement the macOS/Lima lifecycle and diagnostic actions that +Substrate must own directly so operators can create, warm, check, repair, and +inspect the backend without relying on raw guest administration commands. ## Why this milestone exists -The current backend is operationally split between Substrate CLI surfaces and direct helper or guest commands: - -- `substrate host doctor` and `substrate world doctor` exist, but the repo still treats [scripts/mac/lima-doctor.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/lima-doctor.sh) and direct `limactl shell` probes as normal. -- [scripts/mac/lima-warm.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/lima-warm.sh) performs lifecycle repair through direct guest mutation rather than through a clearly productized Substrate-managed contract. -- [docs/cross-platform/mac_world_setup.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/cross-platform/mac_world_setup.md) still teaches manual build/install/service flows inside the guest. +The current backend is operationally split between already-landed Substrate +surfaces and still-normalized helper or guest commands. + +- `substrate host doctor` and `substrate world doctor` already exist as + canonical diagnostics. +- `substrate world gateway sync|status|restart` and status JSON already exist as + canonical gateway lifecycle/status surfaces. +- `scripts/mac/smoke.sh` already exercises gateway lifecycle smoke coverage. +- `scripts/mac/lima-doctor.sh` and direct `limactl shell` probes are still + treated too often as normal operator paths. +- `scripts/mac/lima-warm.sh` performs lifecycle repair through direct guest + mutation rather than through a clearly productized Substrate-managed + contract. +- `docs/cross-platform/mac_world_setup.md` still teaches manual + build/install/service flows inside the guest. This milestone exists to define what “Substrate-owned operations” means in concrete command terms before the docs are cut over. @@ -23,7 +37,11 @@ This milestone exists to define what “Substrate-owned operations” means in c - Identify the full macOS lifecycle/diagnostic action set that should be owned by Substrate. - Replace or wrap the current Lima-specific flows with Substrate-owned commands where feasible. - Make doctor output the primary structured evidence surface for macOS backend health. -- Ensure lifecycle ownership covers warm/create, service reachability, guest repair, and state reporting at a level operators can trust. +- Make gateway lifecycle/status a primary structured evidence and support + surface alongside doctor JSON. +- Ensure lifecycle ownership covers warm/create, service reachability, guest + repair, gateway runtime state, and state reporting at a level operators can + trust. - Define the Substrate-owned sync or copy path that replaces any remaining normal-path dependence on broad host mounts after Phase 2 ingress minimization. @@ -39,6 +57,7 @@ This milestone exists to define what “Substrate-owned operations” means in c - Treat the current helper scripts as implementation staging points, not as the final operator contract. - Promote Substrate commands as the entry points for: - backend readiness + - gateway lifecycle and status - workspace ingress or sync needed for normal backend operation - health diagnosis - service/socket verification @@ -46,11 +65,18 @@ This milestone exists to define what “Substrate-owned operations” means in c - Keep any required Lima shelling internal to those commands or clearly classified as breakglass. - Make structured doctor output the main evidence channel so operators do not need to infer health from scattered shell commands. +- Treat managed gateway runtime artifacts under + `/run/substrate/substrate-gateway-runtime/` as part of the supported + operator-facing lifecycle surface. - Preserve the phase-0 support taxonomy: - supported - degraded-but-supported - breakglass/unsupported +This milestone should also preserve already-supported shared-world/orchestration +behavior rather than hard-coding an owned-operations story that only works for +single-session happy paths. + ## Dependencies / sequencing - Depends on Phase 2 hardening decisions, especially the listener and unit contracts. @@ -58,32 +84,44 @@ This milestone exists to define what “Substrate-owned operations” means in c ## Concrete repo surfaces and file pointers -- [crates/shell/src/execution/platform/macos.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/platform/macos.rs) +- `crates/shell/src/execution/platform/macos.rs` - primary CLI/runtime surface for owned macOS doctor, lifecycle, and status commands -- [crates/shell/src/builtins/world_enable](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/builtins/world_enable) +- `crates/shell/src/builtins/world_gateway.rs` + - already-owned gateway lifecycle/status surface that Phase 3 should elevate, + not reinvent +- `docs/contracts/substrate-gateway-operator-contract.md` + - canonical operator contract for the already-landed gateway lifecycle family +- `docs/contracts/substrate-gateway-status-schema.md` + - authoritative machine-readable status contract +- `crates/shell/src/builtins/world_enable` - likely owner for provisioning and enablement flows that should stop delegating operator responsibility to raw Lima commands -- [crates/shell/src/execution/workspace_cmd.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/workspace_cmd.rs) +- `crates/shell/src/execution/workspace_cmd.rs` - likely owner for any Substrate-managed workspace ingress or sync command surface that replaces broad convenience mounts -- [crates/world-mac-lima/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/lib.rs) +- `crates/world-mac-lima/src/lib.rs` - backend surface that must honor the owned command contract once lifecycle and sync behavior are driven through the CLI -- [scripts/mac/lima-warm.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/lima-warm.sh) +- `crates/world-agent/src/gateway_runtime.rs` + - authoritative managed runtime state and artifact model under + `/run/substrate/substrate-gateway-runtime/` +- `scripts/mac/lima-warm.sh` - current owner of create/start/repair behavior -- [scripts/mac/lima-doctor.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/lima-doctor.sh) +- `scripts/mac/lima-doctor.sh` - current owner of host and guest health checks -- [scripts/mac/smoke.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/smoke.sh) +- `scripts/mac/smoke.sh` - can verify the managed lifecycle path after the contract is defined -- [docs/WORLD.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/WORLD.md) +- `docs/WORLD.md` - already names `substrate host doctor` and `substrate world doctor`, but still mixes them with direct guest commands -- [docs/cross-platform/mac_world_setup.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/cross-platform/mac_world_setup.md) +- `docs/cross-platform/mac_world_setup.md` - primary doc surface that will need cutover once the owned command set is frozen ## Deliverables - A concrete owned-command matrix for macOS lifecycle and diagnostics. +- A concrete owned-command matrix for gateway lifecycle/status, including + `substrate world gateway sync|status|restart`. - A concrete owned-command matrix for macOS workspace ingress or sync needed by the hardened same-user backend. - A migration plan from helper-script and raw guest workflows to Substrate-owned entry points. @@ -93,12 +131,16 @@ This milestone exists to define what “Substrate-owned operations” means in c ## Acceptance criteria - The macOS backend has a defined Substrate-owned path for readiness, health, and repair operations. +- The macOS backend has a defined Substrate-owned path for managed gateway + lifecycle and status operations. - The macOS backend has a defined Substrate-owned path for any required normal workspace ingress or sync after Phase 2 mount minimization. - Operators can gather the primary health evidence without running raw `limactl shell substrate ...` commands. - Any degraded-but-supported helper path is explicitly identified as such and is distinct from breakglass/unsupported flows. - A reviewer can list the normal macOS operational commands from one doc section without cross-referencing guest-admin recipes. +- Host-side `SUBSTRATE_WORLD_SOCKET` override use is not presented as the + default supported Lima path. ## Validation / evidence plan @@ -106,6 +148,8 @@ This milestone exists to define what “Substrate-owned operations” means in c - Build an ingress inventory mapping any remaining Phase 2 copy/sync needs to owned commands rather than implicit mounts. - Capture `substrate host doctor --json` and `substrate world doctor --json` as the primary health artifacts. +- Capture `substrate world gateway sync`, `substrate world gateway status --json`, + and `substrate world gateway restart` as primary lifecycle artifacts. - Run `scripts/mac/smoke.sh` or successor smoke coverage through the owned lifecycle path to prove the contract is usable end to end. - Record the set of residual manual actions that still lack owned coverage and carry them forward into milestone 3.2 as explicit breakglass exceptions. diff --git a/macos-hardening/macos-hardened-same-user-lima/phase-3-substrate-owned-operations/milestone-3-2-breakglass-reclassification-and-doc-cutover-sow.md b/macos-hardening/macos-hardened-same-user-lima/phase-3-substrate-owned-operations/milestone-3-2-breakglass-reclassification-and-doc-cutover-sow.md index 8656e5823..dda84e95e 100644 --- a/macos-hardening/macos-hardened-same-user-lima/phase-3-substrate-owned-operations/milestone-3-2-breakglass-reclassification-and-doc-cutover-sow.md +++ b/macos-hardening/macos-hardened-same-user-lima/phase-3-substrate-owned-operations/milestone-3-2-breakglass-reclassification-and-doc-cutover-sow.md @@ -4,19 +4,34 @@ Draft +Last updated: 2026-05-19 + ## Purpose / outcome -Reclassify remaining direct macOS/Lima guest administration as breakglass and cut the repo documentation over so the default operator path is the Substrate-owned lifecycle and diagnostics surface. +Reclassify remaining direct macOS/Lima guest administration as breakglass and +cut the repo documentation over so the default operator path is the +Substrate-owned lifecycle, diagnostics, and gateway lifecycle/status surface. ## Why this milestone exists -Even after owned commands exist, the repo will continue to communicate the wrong security posture if the docs still teach: +Even after owned commands exist, the repo will continue to communicate the +wrong security posture if the docs still teach: - building or installing the guest agent through raw `limactl shell substrate ...` - enabling or restarting guest services manually as the first-line flow - probing the guest socket directly as the normal health check - using direct guest logs and shell access without labeling them as exceptional +That mismatch is more visible now because the repo already has the surfaces the +docs should lead with: + +- `substrate host doctor` +- `substrate world doctor` +- `substrate world gateway sync|status|restart` +- `substrate world gateway status --json` +- managed runtime artifact paths under + `/run/substrate/substrate-gateway-runtime/` + This milestone exists because the hardening story is incomplete until the operator-facing documentation and troubleshooting posture match the new operational boundary. ## In-scope @@ -24,6 +39,8 @@ This milestone exists because the hardening story is incomplete until the operat - Reclassify direct guest commands in macOS docs and helper guidance as breakglass where they still exist. - Rewrite the macOS setup and operations narrative to lead with Substrate-owned commands. - Align troubleshooting and evidence sections with the new contract. +- Classify host-side `SUBSTRATE_WORLD_SOCKET` override use as advanced/test or + breakglass rather than the default supported Lima path. - Mark legacy scripts or workflows as supported, degraded-but-supported, or breakglass/unsupported as appropriate. @@ -42,6 +59,12 @@ This milestone exists because the hardening story is incomplete until the operat - breakglass/unsupported: allowed for deep debugging or recovery, but not part of normal setup or maintenance - Apply that classification consistently across `README`-style docs, setup docs, troubleshooting sections, and helper-script messaging. +- Treat `substrate world gateway sync|status|restart` and status JSON as part of + the supported operator contract, not as optional or experimental side paths. +- Treat runtime artifact references under + `/run/substrate/substrate-gateway-runtime/` as supported operator evidence + when surfaced by Substrate-owned commands, while keeping raw guest shelling as + breakglass. - Ensure the docs explicitly call out the unresolved same-user ownership gap so users do not mistake the cutover for Linux-equivalent authority separation. ## Dependencies / sequencing @@ -51,16 +74,20 @@ This milestone exists because the hardening story is incomplete until the operat ## Concrete repo surfaces and file pointers -- [docs/cross-platform/mac_world_setup.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/cross-platform/mac_world_setup.md) +- `docs/cross-platform/mac_world_setup.md` - primary setup and troubleshooting cutover target -- [docs/WORLD.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/WORLD.md) +- `docs/WORLD.md` - primary runtime and operator contract target -- [scripts/mac/lima-doctor.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/lima-doctor.sh) +- `docs/contracts/substrate-gateway-operator-contract.md` + - already-landed gateway lifecycle operator contract that docs should lead with +- `docs/contracts/substrate-gateway-status-schema.md` + - status JSON contract the docs should reference instead of redefining +- `scripts/mac/lima-doctor.sh` - messaging and role should match the reclassification -- [scripts/mac/lima-warm.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/lima-warm.sh) +- `scripts/mac/lima-warm.sh` - usage/help text should reflect whether it is supported, degraded-but-supported, or breakglass/unsupported -- [scripts/mac/smoke.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/smoke.sh) +- `scripts/mac/smoke.sh` - evidence instructions should match the owned operations narrative ## Deliverables @@ -68,23 +95,33 @@ This milestone exists because the hardening story is incomplete until the operat - A doc cutover plan for macOS hardened-Lima setup, lifecycle, troubleshooting, and validation. - A breakglass classification table for the remaining direct guest actions. - Updated script/help-text expectations aligned with the new classifications. -- A final operator narrative that distinguishes same-user hardening from Linux-equivalent ownership. +- A final operator narrative that distinguishes same-user hardening from + Linux-equivalent ownership and leads with already-landed doctor/gateway + commands. ## Acceptance criteria - The default macOS docs lead with Substrate-owned lifecycle and diagnostic commands. +- The default macOS docs also lead with + `substrate world gateway sync|status|restart` and status JSON for managed + gateway lifecycle. - Remaining `limactl shell` and direct guest systemctl/socket instructions are explicitly labeled breakglass/unsupported. - Any degraded-but-supported path that remains is a compatibility wrapper around a Substrate-owned command surface, not a raw direct guest procedure. - Troubleshooting sections make clear when a user should escalate from owned commands to breakglass commands. - A reviewer can read the macOS docs and understand both the hardened default and the remaining same-user limitation without external context. +- Host-side `SUBSTRATE_WORLD_SOCKET` override use is not documented as the + normal supported Lima path. ## Validation / evidence plan -- Perform a doc inventory of direct guest commands in `docs/WORLD.md` and `docs/cross-platform/mac_world_setup.md` and classify each one. +- Perform a doc inventory of direct guest commands and host-side overrides in + `docs/WORLD.md` and `docs/cross-platform/mac_world_setup.md` and classify + each one. - Review script help output and inline messaging to ensure it matches the new operator contract. -- Re-run the documented happy path using only Substrate-owned commands and verify that the required evidence can be gathered without breakglass access. +- Re-run the documented happy path using only Substrate-owned commands and + verify that the required evidence can be gathered without breakglass access. - Verify that any remaining breakglass guidance is concrete, bounded, and clearly separated from the normal path. ## Risks / open questions diff --git a/macos-hardening/macos-ownership-separated-world/README.md b/macos-hardening/macos-ownership-separated-world/README.md index 9d42ef4cc..23c42ce95 100644 --- a/macos-hardening/macos-ownership-separated-world/README.md +++ b/macos-hardening/macos-ownership-separated-world/README.md @@ -4,35 +4,55 @@ Status: Proposed Owner: Substrate world backend / macOS control plane -Last updated: 2026-04-28 +Last updated: 2026-05-19 ## Purpose / Outcome -Define and sequence the work required to make the macOS Lima-backed world meaningfully Substrate-controlled instead of developer-user-controlled. The target end state is a dedicated Substrate-owned host control plane that owns `LIMA_HOME`, VM state, credentials, forwarding, and the only routine host-reachable broker endpoint for the guest `world-agent`. +Define and sequence the work required to make the macOS Lima-backed world +meaningfully Substrate-controlled instead of developer-user-controlled. The +target end state is a dedicated Substrate-owned host control plane that owns +`LIMA_HOME`, VM state, credentials, forwarding, and the only routine +host-reachable broker endpoint for both world execution and gateway lifecycle. ## Why This Feature Exists -The current macOS backend gives Substrate a brokered execution path, but it does not give Substrate exclusive control of the VM. Today the same logged-in developer user can still: +The current macOS backend already gives Substrate a same-user brokered path for +execution, doctoring, and gateway lifecycle, but it still does not give +Substrate exclusive control of the VM. Today the same logged-in developer user +can still: - read and mutate Lima state under the default user-owned `LIMA_HOME` - invoke `limactl shell` - use direct SSH-based transport paths - reach the guest socket and services through host-local forwarding +- exercise `substrate world gateway sync|status|restart` over a path that still + depends on same-user Lima ownership and forwarding setup -That is sufficient for convenience and parity, but it is not sufficient for the security claim that only Substrate controls the macOS world. This planning set exists to convert the macOS backend from wrapper-hardening to host ownership separation. +That is sufficient for convenience and parity, but it is not sufficient for the +security claim that only Substrate controls the macOS world. This planning set +exists to convert the macOS backend from wrapper-hardening to host ownership +separation while preserving the already-landed operator surfaces. ## In-Scope -- A host ownership model where a dedicated daemon or service account owns Lima state and lifecycle. -- A control-plane split where developer-facing CLI code no longer owns or directly manipulates host-side VM artifacts. -- Replacement of direct host-local guest reachability with one Substrate-owned broker endpoint and authenticated broker-to-agent traffic. -- A breakglass path for maintenance that is explicit, auditable, and disabled for normal runtime use. +- A host ownership model where a dedicated daemon or service account owns Lima + state and lifecycle. +- A control-plane split where developer-facing CLI code no longer owns or + directly manipulates host-side VM artifacts. +- Replacement of direct host-local guest reachability with one + Substrate-owned broker endpoint and authenticated broker-to-agent traffic. +- Extension of ownership separation to the current gateway lifecycle path so + `substrate world gateway sync|status|restart` no longer depend on + developer-owned forwarding. +- A breakglass path for maintenance that is explicit, auditable, and disabled + for normal runtime use. - Repo planning for phase 0 architecture/contracts, phase 1 prototype implementation, and phase 2 productization and migration. ## Out-of-Scope -- Full productionization of macOS service packaging, notarization, or MDM deployment. +- Full productionization of macOS service packaging, notarization, or MDM + deployment. - Replacing the Linux guest or the in-guest `world-agent` API itself. - Redesigning Linux or Windows world ownership models in this workstream. - Broad product work unrelated to the macOS control-plane ownership boundary. @@ -41,86 +61,142 @@ That is sufficient for convenience and parity, but it is not sufficient for the The feature is delivered in three phases: -1. Phase 0 defines the contracts. It locks the host ownership model, the daemon responsibilities, the broker/auth boundary, and the breakglass rules so implementation work does not preserve same-user escape hatches by accident. -2. Phase 1 builds a working prototype. It moves `LIMA_HOME` and instance lifecycle under daemon ownership, changes the forwarding model to a private daemon-managed channel, exposes one developer-reachable broker endpoint, and adds a maintenance-only breakglass path. +1. Phase 0 defines the contracts. It locks the host ownership model, the daemon + responsibilities, the broker/auth boundary, the gateway lifecycle/status + owner lines, and the breakglass rules so implementation work does not + preserve same-user escape hatches by accident. +2. Phase 1 builds a working prototype. It moves `LIMA_HOME` and instance + lifecycle under daemon ownership, changes the forwarding model to a private + daemon-managed channel, exposes one developer-reachable broker endpoint, and + carries existing world and gateway lifecycle traffic over that path. 3. Phase 2 productizes the model. It covers installer, upgrade, migration, doctor, supportability, and GA validation so the stronger ownership boundary becomes an operable product contract instead of a prototype. -The intent is not to incrementally harden the existing `limactl` and SSH flows in place. The intent is to remove them from the standard runtime path. +The intent is not to incrementally harden the existing `limactl` and SSH flows +in place. The intent is to remove them from the standard runtime path. ## Dependencies / Sequencing -- This track depends on the same-user hardening track establishing the cleaned-up - transport contract, policy semantics, mount posture, and Substrate-owned - operational vocabulary that ownership separation will build on rather than - duplicate. +- This track depends on the same-user hardening track establishing the cleaned + up transport contract, policy semantics, mount posture, gateway runtime + parity, and Substrate-owned operational vocabulary that ownership separation + will build on rather than duplicate. - At minimum, ownership-separated phase 1 assumes the same-user track has completed phase 1 milestone 1.1 through milestone 1.3 plus phase 3 milestone - 3.2, so transport, policy, and operator-state terminology are already - normalized. + 3.2, so transport, policy, gateway lifecycle vocabulary, and operator-state + terminology are already normalized. - Ownership-separated phase 2 assumes the same-user track has also completed phase 2 milestone 2.2 and phase 3 milestone 3.1, so ingress/sync and Substrate-owned lifecycle surfaces are available for migration planning. - Phase 0 must complete before phase 1 implementation starts. -- Milestone 0.1 must land before milestone 0.2 because the broker/auth contract depends on the daemon ownership and lifecycle contract. -- Milestone 1.1 must land before milestone 1.2 because the private forwarding model depends on daemon-owned VM state and credentials. -- Milestone 1.3 depends on 1.1 and 1.2 because breakglass policy is defined after routine runtime paths are removed. -- Phase 2 depends on the phase-1 prototype proving that daemon-owned state, +- Milestone 0.1 must land before milestone 0.2 because the broker/auth + contract depends on the daemon ownership and lifecycle contract. +- Milestone 1.1 must land before milestone 1.2 because the private forwarding + model depends on daemon-owned VM state and credentials. +- Milestone 1.3 depends on 1.1 and 1.2 because breakglass policy is defined + after routine runtime paths are removed. +- Phase 2 depends on the phase 1 prototype proving that daemon-owned state, private forwarding, and the broker boundary can replace the current same-user - runtime path. + runtime path for both execution and gateway lifecycle. ## Concrete Repo Surfaces and File Pointers - `crates/world-mac-lima/src/lib.rs` - Current host-side lifecycle ownership, VM ensure/start logic, agent reachability checks, and endpoint selection. + Current host-side lifecycle ownership, VM ensure/start logic, agent + reachability checks, and endpoint selection. - `crates/world-mac-lima/src/forwarding.rs` - Current direct VSock/SSH forwarding selection and host-visible socket creation under `~/.substrate/sock/agent.sock`. + Current direct VSock/SSH forwarding selection and host-visible socket creation + under `~/.substrate/sock/agent.sock`. - `crates/world-mac-lima/src/transport.rs` - Current transport assumptions that treat VSock and SSH as directly host-usable options. + Current transport assumptions that treat VSock and SSH as directly + host-usable options. +- `crates/shell/src/builtins/world_gateway.rs` + Current macOS gateway lifecycle/status client that still rides the same + Lima-backed forwarded path. +- `crates/world-agent/src/gateway_runtime.rs` + Current managed gateway runtime root, lifecycle execution, and auth-bundle + handoff inside the guest. - `scripts/mac/lima-warm.sh` - Current same-user VM creation, guest provisioning, socket activation, and guest artifact ownership assumptions. + Current same-user VM creation, guest provisioning, socket activation, and + guest artifact ownership assumptions. - `scripts/mac/lima/substrate.yaml` Current Lima instance definition and guest-side service/socket configuration. +- `scripts/mac/smoke.sh` + Current macOS proof harness, including gateway lifecycle proof. +- `scripts/substrate/world-enable.sh` + Current post-install world provisioning entrypoint that assumes the CLI can + provision the macOS world directly. +- `scripts/substrate/dev-install-substrate.sh` + Current same-user dev lifecycle entrypoint that stages guest binaries and + writes world metadata. - `scripts/substrate/install-substrate.sh` - Current installer path that provisions the macOS world by calling `scripts/mac/lima-warm.sh`. + Current installer path that provisions the macOS world by calling + `scripts/mac/lima-warm.sh` and verifies `substrate-gateway` in the guest. - `scripts/substrate/uninstall-substrate.sh` Current teardown path that deletes the Lima VM and the host-forwarded socket. -- `scripts/substrate/world-enable.sh` - Current post-install world provisioning entrypoint that assumes the CLI can provision the macOS world directly. -- `docs/WORLD.md` - Current operator-facing description of macOS transport and direct forwarding behavior. -- `docs/cross-platform/mac_world_setup.md` - Current setup guide that instructs direct `limactl shell`, guest `systemctl`, and in-guest `curl`. +- `../../docs/WORLD.md` + Descriptive architecture evidence for current transport and runtime posture. +- `../../docs/contracts/substrate-gateway-operator-contract.md` + Durable owner of the gateway lifecycle command family. +- `../../docs/contracts/substrate-gateway-policy-evaluation.md` + Durable owner of the integrated auth and policy-evaluation boundary. +- `../../docs/contracts/substrate-gateway-status-schema.md` + Durable owner of `substrate world gateway status --json`. ## Deliverables -- Feature overview and phase breakdown for ownership-separated macOS world control. -- Phase 0 milestone SOWs covering the host owner model and broker/auth boundary. -- Phase 1 milestone SOWs covering daemon-owned Lima state, private forwarding, and breakglass maintenance. +- Feature overview and phase breakdown for ownership-separated macOS world + control. +- Phase 0 milestone SOWs covering the host owner model and broker/auth + boundary. +- Phase 1 milestone SOWs covering daemon-owned Lima state, private forwarding, + and breakglass maintenance. - Phase 2 milestone SOWs covering installer and migration, operator supportability, and GA validation. -- Clear acceptance and evidence expectations that future implementation PRs can execute against. +- Clear acceptance and evidence expectations that future implementation PRs can + execute against. ## Acceptance Criteria -- The planning set states an end state where the developer user no longer owns routine Lima control-plane artifacts. -- The planning set makes direct `limactl shell`, direct SSH, guest `systemctl`, and guest `curl` breakglass only. -- Each phase and milestone has concrete repo surfaces, deliverables, validation evidence, and dependency ordering. -- The planning set is precise enough to guide implementation without reopening core architecture questions. +- The planning set states an end state where the developer user no longer owns + routine Lima control-plane artifacts. +- The planning set makes direct `limactl shell`, direct SSH, guest + `systemctl`, and guest `curl` breakglass only. +- The planning set preserves the current `substrate host doctor`, + `substrate world doctor`, `substrate health`, and + `substrate world gateway sync|status|restart` operator surfaces, but moves + them behind a new ownership model. +- `docs/WORLD.md` is treated as descriptive evidence, not the owner of operator + or status semantics. +- Each phase and milestone has concrete repo surfaces, deliverables, validation + evidence, and dependency ordering. ## Validation / Evidence Plan -- Architecture reviews for phase 0 must produce explicit control-flow diagrams and contract deltas against current repo surfaces. -- Prototype work in phase 1 must demonstrate daemon-owned state paths, removal of developer-visible forwarded guest sockets from the default path, and authenticated broker-to-agent calls. -- Operator docs and installer flows must be mapped to the new architecture before implementation is considered complete. +- Architecture reviews for phase 0 must produce explicit control-flow diagrams + and contract deltas against current repo surfaces. +- Prototype work in phase 1 must demonstrate daemon-owned state paths, removal + of developer-visible forwarded guest sockets from the default path, and + broker-mediated execution plus gateway lifecycle calls. +- Operator docs and installer flows must be mapped to the new architecture + before implementation is considered complete. +- Validation evidence should include: + - `substrate host doctor --json` + - `substrate world doctor --json` + - `substrate health --json` + - `substrate world gateway status --json` ## Risks / Open Questions -- macOS service-account mechanics need a final decision between a root-owned `launchd` daemon and an app-managed service installed via `SMAppService`. -- Lima may still require some root or user-session integration details that change how service ownership is packaged. -- Broker-to-agent attestation needs a concrete trust anchor that works with Lima and survives VM recreation. -- Migration from existing user-owned `~/.lima/substrate` installs may require a destructive or semi-destructive conversion path. +- macOS service-account mechanics need a final decision between a root-owned + `launchd` daemon and an app-managed service installed via `SMAppService`. +- Lima may still require some root or user-session integration details that + change how service ownership is packaged. +- Broker-to-agent attestation needs a concrete trust anchor that works with + Lima and survives VM recreation. +- Migration from existing user-owned `~/.lima/substrate` installs may require a + destructive or semi-destructive conversion path. ## Phase Index diff --git a/macos-hardening/macos-ownership-separated-world/phase-0-architecture-and-host-ownership-model/README.md b/macos-hardening/macos-ownership-separated-world/phase-0-architecture-and-host-ownership-model/README.md index a33a4bc8d..5b7b1e081 100644 --- a/macos-hardening/macos-ownership-separated-world/phase-0-architecture-and-host-ownership-model/README.md +++ b/macos-hardening/macos-ownership-separated-world/phase-0-architecture-and-host-ownership-model/README.md @@ -2,87 +2,150 @@ Status: Proposed +Last updated: 2026-05-19 + ## Purpose / Outcome -Produce the architecture and contracts that make ownership separation concrete before implementation starts. Phase 0 ends when the repo has an agreed host ownership model, daemon responsibility boundary, broker/auth boundary, and breakglass policy for macOS Lima worlds. +Produce the architecture and contracts that make ownership separation concrete +before implementation starts. Phase 0 ends when the repo has an agreed host +ownership model, daemon responsibility boundary, broker/auth boundary, gateway +lifecycle/status owner line, and breakglass policy for macOS Lima worlds. ## Why This Phase Exists -The current backend is shaped around same-user Lima control. If implementation starts without first freezing the ownership and transport contracts, the project is likely to preserve the existing direct host-to-guest reachability and simply move logic between scripts and crates. Phase 0 prevents that failure mode. +The current backend is shaped around same-user Lima control. HEAD now also has +same-user world/gateway lifecycle surfaces that operators can actually use: +`substrate host doctor`, `substrate world doctor`, `substrate health`, and +`substrate world gateway sync|status|restart`. If implementation starts without +first freezing the ownership and transport contracts, the project is likely to +preserve those same-user assumptions and simply move logic between scripts and +crates. Phase 0 prevents that failure mode. ## In-Scope - Define the host-side owner for Lima state, sockets, keys, and lifecycle. - Define the contract between developer CLI code and the new daemon. -- Define the single broker endpoint and the authenticated daemon-to-agent boundary. +- Define the single broker endpoint and the authenticated daemon-to-agent + boundary. +- Define how the existing gateway lifecycle/status surface survives under the + new ownership model. - Define which current direct-access operations become breakglass only. -- Identify installer, uninstaller, backend, and docs surfaces that must change later. +- Identify installer, uninstaller, backend, contract-doc, and macOS docs + surfaces that must change later. ## Out-of-Scope - Shipping the daemon, service, or broker implementation. -- Changing guest-side `world-agent` behavior beyond what is needed to specify the control contract. +- Changing guest-side `world-agent` behavior beyond what is needed to specify + the control contract. - Finalizing distribution packaging or update delivery. ## Architectural Approach -Phase 0 treats the current macOS backend as evidence, not as the architecture to preserve. The phase documents a new control path: +Phase 0 treats the current macOS backend as evidence, not as the architecture +to preserve. The phase documents a new control path: - developer CLI talks to a Substrate-owned macOS daemon - daemon owns Lima host state and lifecycle - daemon exposes one broker endpoint to the CLI - daemon talks to the guest agent over a private, non-routine transport -- direct Lima, SSH, guest service control, and guest socket probing are removed from the normal runtime path +- gateway lifecycle/status commands continue to exist, but run through the same + daemon-owned path +- direct Lima, SSH, guest service control, and guest socket probing are removed + from the normal runtime path ## Dependencies / Sequencing -- This phase starts from the current behavior in `crates/world-mac-lima`, `scripts/mac/lima-warm.sh`, and existing macOS docs. +- This phase starts from the current behavior in `crates/world-mac-lima`, + `scripts/mac/lima-warm.sh`, `crates/shell/src/builtins/world_gateway.rs`, and + the existing macOS docs/contracts. - Milestone 0.1 defines the ownership and daemon contract first. -- Milestone 0.2 defines the broker/auth contract second because it depends on 0.1’s control-plane boundary. +- Milestone 0.2 defines the broker/auth contract second because it depends on + 0.1’s control-plane boundary and on the existing gateway contract owner docs. - Phase 1 prototype work cannot start until both milestone SOWs are accepted. ## Concrete Repo Surfaces and File Pointers - `crates/world-mac-lima/src/lib.rs` - Current VM ensure, agent readiness, forwarding setup, and endpoint selection that phase 0 must replace conceptually. + Current VM ensure, agent readiness, forwarding setup, and endpoint selection + that phase 0 must replace conceptually. - `crates/world-mac-lima/src/forwarding.rs` - Current direct host forwarding design that phase 0 must explicitly retire from normal operation. + Current direct host forwarding design that phase 0 must explicitly retire + from normal operation. +- `crates/shell/src/builtins/world_gateway.rs` + Current same-user gateway lifecycle/status client path that phase 0 must + preserve semantically while replacing operationally. +- `crates/world-agent/src/gateway_runtime.rs` + Current managed gateway runtime and auth-bundle launch model that future + ownership separation must route through a new host owner. - `scripts/mac/lima-warm.sh` - Current same-user bootstrap and guest ownership assumptions that phase 1 will rework under the phase 0 contract. + Current same-user bootstrap and guest ownership assumptions that phase 1 will + rework under the phase 0 contract. +- `scripts/substrate/world-enable.sh` + Current world provisioning helper that phase 0 must reframe around daemon + ownership or deprecate on macOS. +- `scripts/substrate/dev-install-substrate.sh` + Current dev lifecycle helper that already stages `substrate-gateway` and + writes world metadata in a same-user model. - `scripts/substrate/install-substrate.sh` - Current provisioning path that phase 0 must reframe around daemon installation and broker setup. + Current provisioning path that phase 0 must reframe around daemon + installation and broker setup. - `scripts/substrate/uninstall-substrate.sh` - Current teardown path that phase 0 must reframe around daemon-owned state cleanup. -- `docs/WORLD.md` - Current architecture description that will need a contract delta after implementation. -- `docs/cross-platform/mac_world_setup.md` - Current operator workflow with direct guest access that phase 0 must mark as incompatible with the new target state. + Current teardown path that phase 0 must reframe around daemon-owned state + cleanup. +- `../../../docs/WORLD.md` + Current descriptive architecture evidence that will need a contract delta + after implementation. +- `../../../docs/contracts/substrate-gateway-operator-contract.md` + Durable owner of the gateway lifecycle command family. +- `../../../docs/contracts/substrate-gateway-policy-evaluation.md` + Durable owner of the auth and policy boundary. +- `../../../docs/contracts/substrate-gateway-status-schema.md` + Durable owner of `status --json`. ## Deliverables - Phase overview README. - Milestone 0.1 SOW for host owner model and daemon contract. - Milestone 0.2 SOW for brokered access and authenticated boundary contract. -- A dependency map from current same-user surfaces to future daemon-owned surfaces. +- A dependency map from current same-user surfaces to future daemon-owned + surfaces. +- An owner-line map that distinguishes descriptive architecture docs from + durable operator/status contract docs. ## Acceptance Criteria -- The phase documents a single authoritative answer for who owns `LIMA_HOME`, SSH material, forwarding state, instance metadata, and host sockets. -- The phase documents a single authoritative answer for how the CLI reaches the macOS world in routine operation. -- The phase documents which current commands are removed from normal workflows and under what breakglass conditions they are still allowed. -- The milestone SOWs identify all repo surfaces future implementation must touch first. +- The phase documents a single authoritative answer for who owns `LIMA_HOME`, + SSH material, forwarding state, instance metadata, and host sockets. +- The phase documents a single authoritative answer for how the CLI reaches the + macOS world in routine operation. +- The phase documents how current doctor/health/gateway lifecycle commands keep + working without preserving same-user host ownership. +- The phase documents which current commands are removed from normal workflows + and under what breakglass conditions they are still allowed. +- The milestone SOWs identify all repo surfaces future implementation must + touch first. ## Validation / Evidence Plan -- Review the phase docs against the current code and confirm each existing same-user path has an explicit replacement or deprecation. -- Confirm that no milestone leaves the developer user owning routine Lima control-plane artifacts. -- Confirm that doc reviewers can derive a concrete phase 1 implementation order without reopening core boundary questions. +- Review the phase docs against the current code and confirm each existing + same-user path has an explicit replacement or deprecation. +- Confirm that no milestone leaves the developer user owning routine Lima + control-plane artifacts. +- Confirm that `docs/WORLD.md` is treated as descriptive evidence and that + gateway operator/status semantics are sourced from `docs/contracts/`. +- Confirm that doc reviewers can derive a concrete phase 1 implementation order + without reopening core boundary questions. ## Risks / Open Questions -- The daemon install model may affect UX, signing requirements, and whether the initial prototype uses a root daemon or a dedicated service account launched by `launchd`. -- Some Lima internals may still assume user-session conventions that need proof-of-concept work before the contract is final. -- The project needs a crisp definition of what “breakglass” means in code, docs, and operator evidence. +- The daemon install model may affect UX, signing requirements, and whether the + initial prototype uses a root daemon or a dedicated service account launched + by `launchd`. +- Some Lima internals may still assume user-session conventions that need + proof-of-concept work before the contract is final. +- The project needs a crisp definition of what "breakglass" means in code, + docs, and operator evidence. ## Milestones diff --git a/macos-hardening/macos-ownership-separated-world/phase-0-architecture-and-host-ownership-model/milestone-0-1-host-owner-model-and-daemon-contract-sow.md b/macos-hardening/macos-ownership-separated-world/phase-0-architecture-and-host-ownership-model/milestone-0-1-host-owner-model-and-daemon-contract-sow.md index 1796ce85e..d330a29f6 100644 --- a/macos-hardening/macos-ownership-separated-world/phase-0-architecture-and-host-ownership-model/milestone-0-1-host-owner-model-and-daemon-contract-sow.md +++ b/macos-hardening/macos-ownership-separated-world/phase-0-architecture-and-host-ownership-model/milestone-0-1-host-owner-model-and-daemon-contract-sow.md @@ -2,47 +2,77 @@ Status: Proposed +Last updated: 2026-05-19 + ## Purpose / Outcome -Specify the authoritative host ownership model for the macOS world and the contract between the Substrate CLI/backend code and a new host-side daemon. The outcome is a written implementation contract that removes ambiguity about who owns VM lifecycle, artifacts, credentials, and runtime authority. +Specify the authoritative host ownership model for the macOS world and the +contract between the Substrate CLI/backend code and a new host-side daemon. The +outcome is a written implementation contract that removes ambiguity about who +owns VM lifecycle, artifacts, credentials, runtime authority, and the host side +of the current gateway lifecycle path. ## Why This Milestone Exists -Current macOS behavior is anchored in the logged-in developer account. `MacLimaBackend` in `crates/world-mac-lima/src/lib.rs` starts the VM directly, `scripts/mac/lima-warm.sh` provisions the instance directly, and forwarding artifacts are created under the developer’s home directory. Without first changing the ownership contract, later transport hardening would still rest on a developer-owned substrate. +Current macOS behavior is anchored in the logged-in developer account. +`MacLimaBackend` starts the VM directly, `scripts/mac/lima-warm.sh` provisions +the instance directly, gateway/client forwarding artifacts are created under the +developer’s home directory, and same-user lifecycle helpers such as +`scripts/substrate/world-enable.sh` and `scripts/substrate/dev-install-substrate.sh` +assume the CLI can directly provision and verify the guest. Without first +changing the ownership contract, later transport hardening would still rest on a +developer-owned substrate. ## In-Scope -- Decide the host-side owner model for Lima instance state, including `LIMA_HOME`, instance metadata, SSH config material, and host-reachable sockets. -- Define the daemon responsibilities for VM create/start/stop, provisioning, health, and transport establishment. -- Define the CLI-to-daemon contract at the level needed for implementation planning. -- Define how installer and uninstaller flows change when the daemon, not the user, owns the control plane. +- Decide the host-side owner model for Lima instance state, including + `LIMA_HOME`, instance metadata, SSH config material, and host-reachable + sockets. +- Define the daemon responsibilities for VM create/start/stop, provisioning, + health, and transport establishment. +- Define the CLI-to-daemon contract at the level needed for implementation + planning. +- Define how installer, dev-install, world-enable, and uninstaller flows change + when the daemon, not the user, owns the control plane. - Define routine versus breakglass operations for lifecycle and maintenance. +- Define how the current same-user doctor/gateway surfaces remain available + after ownership changes. ## Out-of-Scope - Implementing the daemon or its IPC transport. - Implementing auth/attestation details for daemon-to-agent requests. -- Finalizing product UX text for daemon install prompts or privilege escalation. +- Finalizing product UX text for daemon install prompts or privilege + escalation. ## Architectural Approach This milestone should lock the following model: -- A Substrate-owned macOS daemon or service account becomes the only routine owner of Lima state. -- `LIMA_HOME` moves out of the developer-owned default location and into daemon-owned storage. -- SSH keys, SSH config, forwarded socket files, and instance lifecycle metadata move under daemon ownership as well. -- The developer-facing CLI no longer calls `limactl` for routine world lifecycle. It issues requests to the daemon instead. +- A Substrate-owned macOS daemon or service account becomes the only routine + owner of Lima state. +- `LIMA_HOME` moves out of the developer-owned default location and into + daemon-owned storage. +- SSH keys, SSH config, forwarded socket files, and instance lifecycle metadata + move under daemon ownership as well. +- The developer-facing CLI no longer calls `limactl` for routine world + lifecycle. It issues requests to the daemon instead. - Existing same-user scripts become one of: - daemon-internal implementation details - installer/bootstrap tools - breakglass-only maintenance tools +- Current world and gateway lifecycle operator commands remain visible to + operators, but become daemon-mediated rather than same-user Lima-mediated. The milestone must also state whether the prototype prefers: - a root-owned `launchd` daemon -- or a dedicated macOS service account whose service is launched and managed by `launchd` +- or a dedicated macOS service account whose service is launched and managed by + `launchd` -If `SMAppService` is considered for a later productized install path, that should be captured as a packaging option, not allowed to blur the runtime ownership boundary. +If `SMAppService` is considered for a later productized install path, that +should be captured as a packaging option, not allowed to blur the runtime +ownership boundary. ## Dependencies / Sequencing @@ -50,30 +80,47 @@ If `SMAppService` is considered for a later productized install path, that shoul - `crates/world-mac-lima/src/lib.rs` - `crates/world-mac-lima/src/forwarding.rs` - `scripts/mac/lima-warm.sh` + - `scripts/substrate/world-enable.sh` + - `scripts/substrate/dev-install-substrate.sh` - `scripts/substrate/install-substrate.sh` - `scripts/substrate/uninstall-substrate.sh` - - `docs/WORLD.md` - - `docs/cross-platform/mac_world_setup.md` -- Output of this milestone is required before milestone 0.2, because the broker/auth boundary depends on the daemon ownership boundary. + - `../../../docs/WORLD.md` + - `../../../docs/contracts/substrate-gateway-operator-contract.md` + - `../../../docs/contracts/substrate-gateway-policy-evaluation.md` +- Output of this milestone is required before milestone 0.2, because the + broker/auth boundary depends on the daemon ownership boundary. ## Concrete Repo Surfaces and File Pointers - `crates/world-mac-lima/src/lib.rs` - Replace direct `limactl`-driven `ensure_vm_running`, readiness checks, and direct forwarding establishment with a future daemon client boundary. + Replace direct `limactl`-driven `ensure_vm_running`, readiness checks, and + direct forwarding establishment with a future daemon client boundary. - `crates/world-mac-lima/src/transport.rs` - Reclassify current transport enumeration as daemon-internal or obsolete for routine CLI access. + Reclassify current transport enumeration as daemon-internal or obsolete for + routine CLI access. - `crates/world-mac-lima/src/forwarding.rs` - Stop treating `~/.substrate/sock/agent.sock` or host-local TCP as developer-owned routine endpoints. + Stop treating `~/.substrate/sock/agent.sock` or host-local TCP as + developer-owned routine endpoints. +- `crates/shell/src/builtins/world_gateway.rs` + Rebase current gateway lifecycle/status traffic on a daemon-owned host owner + and endpoint rather than the same-user Lima path. - `scripts/mac/lima-warm.sh` - Future split between daemon bootstrap/provisioning logic and breakglass repair logic. + Future split between daemon bootstrap/provisioning logic and breakglass + repair logic. - `scripts/mac/lima/substrate.yaml` - Future ownership-sensitive assumptions around mounts, sockets, and guest bootstrap. + Future ownership-sensitive assumptions around mounts, sockets, and guest + bootstrap. - `scripts/substrate/install-substrate.sh` - Replace “provision macOS world directly” with “install/register/start daemon and hand daemon the release root or packaged assets”. -- `scripts/substrate/uninstall-substrate.sh` - Replace same-user VM deletion and socket cleanup with daemon-owned teardown steps. + Replace "provision macOS world directly" with "install/register/start daemon + and hand daemon the release root or packaged assets." +- `scripts/substrate/dev-install-substrate.sh` + Decide whether dev-install stays same-user only, gets an ownership-separated + mode, or is explicitly excluded from the supported product contract. - `scripts/substrate/world-enable.sh` Reclassify this helper as daemon-triggering or deprecated on macOS. +- `scripts/substrate/uninstall-substrate.sh` + Replace same-user VM deletion and socket cleanup with daemon-owned teardown + steps. ## Deliverables @@ -83,6 +130,8 @@ If `SMAppService` is considered for a later productized install path, that shoul - Lima instance directory - SSH keys and known-hosts material - host broker socket or listener + - any host-side metadata needed to surface gateway lifecycle and runtime log + locations safely - any staging/build/cache directories needed for provisioning - A CLI-to-daemon responsibility table covering: - lifecycle @@ -90,27 +139,47 @@ If `SMAppService` is considered for a later productized install path, that shoul - provisioning - logging - diagnostics + - gateway lifecycle mediation - breakglass entry - A migration note for existing same-user installs. -- A contract delta list describing what becomes invalid in current macOS docs and scripts. +- A contract delta list describing what becomes invalid in current macOS docs + and scripts. ## Acceptance Criteria -- The milestone names one routine owner for all host-side Lima control-plane artifacts, and that owner is not the developer user. -- The milestone states that the developer CLI does not directly run `limactl`, direct SSH, or host socket forwarding in normal operation. -- The milestone defines a narrow daemon API surface sufficient for phase 1 prototype implementation. -- The milestone identifies how installation, upgrade, and uninstall flows will transfer or recreate ownership-separated state. -- The milestone explicitly marks direct `limactl shell`, guest `systemctl`, and guest `curl` as breakglass-only operations. +- The milestone names one routine owner for all host-side Lima control-plane + artifacts, and that owner is not the developer user. +- The milestone states that the developer CLI does not directly run `limactl`, + direct SSH, or host socket forwarding in normal operation. +- The milestone defines a narrow daemon API surface sufficient for phase 1 + prototype implementation. +- The milestone identifies how installation, upgrade, dev-install, + world-enable, and uninstall flows will transfer or recreate + ownership-separated state. +- The milestone explicitly marks direct `limactl shell`, guest `systemctl`, and + guest `curl` as breakglass-only operations. +- The milestone treats `docs/WORLD.md` as descriptive evidence and + `docs/contracts/` as the durable owner for gateway lifecycle/status and + policy semantics. ## Validation / Evidence Plan -- Build an artifact inventory from current repo surfaces and map each artifact to its future owner. -- Walk the current `MacLimaBackend` startup flow and show where daemon calls replace direct host actions. -- Walk the current installer and uninstaller logic and show how ownership moves from user scripts to daemon-managed operations. -- Review the milestone with the constraint “can the developer user still directly control the VM without going through Substrate?” The answer must be “not in routine operation.” +- Build an artifact inventory from current repo surfaces and map each artifact + to its future owner. +- Walk the current `MacLimaBackend` startup flow and show where daemon calls + replace direct host actions. +- Walk the current installer, dev-install, world-enable, and uninstaller logic + and show how ownership moves from user scripts to daemon-managed operations. +- Review the milestone with the constraint "can the developer user still + directly control the VM without going through Substrate?" The answer must be + "not in routine operation." ## Risks / Open Questions -- Root-owned and service-account-owned designs have different operational burdens on macOS; the milestone must choose one for the prototype, not defer the decision. -- Some Substrate assets currently assume `HOME`-relative paths and may need a new daemon state-root abstraction. -- A migration path from `~/.lima/substrate` to daemon-owned state may require instance re-creation rather than in-place adoption. +- Root-owned and service-account-owned designs have different operational + burdens on macOS; the milestone must choose one for the prototype, not defer + the decision. +- Some Substrate assets currently assume `HOME`-relative paths and may need a + new daemon state-root abstraction. +- A migration path from `~/.lima/substrate` to daemon-owned state may require + instance re-creation rather than in-place adoption. diff --git a/macos-hardening/macos-ownership-separated-world/phase-0-architecture-and-host-ownership-model/milestone-0-2-brokered-access-and-auth-boundary-contract-sow.md b/macos-hardening/macos-ownership-separated-world/phase-0-architecture-and-host-ownership-model/milestone-0-2-brokered-access-and-auth-boundary-contract-sow.md index 2080a4853..71515ccd6 100644 --- a/macos-hardening/macos-ownership-separated-world/phase-0-architecture-and-host-ownership-model/milestone-0-2-brokered-access-and-auth-boundary-contract-sow.md +++ b/macos-hardening/macos-ownership-separated-world/phase-0-architecture-and-host-ownership-model/milestone-0-2-brokered-access-and-auth-boundary-contract-sow.md @@ -2,64 +2,123 @@ Status: Proposed +Last updated: 2026-05-19 + ## Purpose / Outcome -Specify the runtime access boundary for the ownership-separated macOS world. The outcome is a contract for one developer-reachable Substrate broker endpoint, a private daemon-to-agent channel, and an authentication or attestation model that prevents the developer user from bypassing the broker in normal operation. +Specify the runtime access boundary for the ownership-separated macOS world. The +outcome is a contract for one developer-reachable Substrate broker endpoint, a +private daemon-to-agent channel, and an authentication or attestation model +that prevents the developer user from bypassing the broker in normal operation. ## Why This Milestone Exists -Moving Lima state under a daemon is necessary but insufficient. The current backend still assumes direct host-reachable transports: VSock proxy, SSH UDS forwarding into `~/.substrate/sock/agent.sock`, and direct guest socket probing. If those paths remain routine, the developer user can still talk to the guest agent outside the broker boundary. This milestone exists to remove that gap at the contract level before code is written. +Moving Lima state under a daemon is necessary but insufficient. The current +backend still assumes direct host-reachable transports: VSock proxy, SSH UDS +forwarding into `~/.substrate/sock/agent.sock`, and direct guest socket +probing. Those paths are now used for more than command execution: on macOS, +the current `substrate world gateway sync|status|restart` entrypoints also ride +the same forwarded world path. If those paths remain routine, the developer +user can still talk to the guest agent outside the broker boundary. This +milestone exists to remove that gap at the contract level before code is +written. ## In-Scope -- Define the single routine broker endpoint exposed to the CLI and any other host-side clients. -- Define how the daemon reaches the guest `world-agent` without exposing the guest agent directly to the developer user. -- Define authentication and attestation expectations between daemon/broker and `world-agent`. -- Define which current diagnostics and setup flows must be rewritten because they use direct guest reachability. +- Define the single routine broker endpoint exposed to the CLI and any other + host-side clients. +- Define how the daemon reaches the guest `world-agent` without exposing the + guest agent directly to the developer user. +- Define authentication and attestation expectations between daemon/broker and + `world-agent`. +- Define how the existing gateway lifecycle/status contract is preserved while + its transport is rehomed under the daemon. +- Define which current diagnostics and setup flows must be rewritten because + they use direct guest reachability. - Define logging and evidence expectations for broker-mediated access. ## Out-of-Scope - Implementing the broker transport or token machinery. - Rewriting the `world-agent` protocol in detail. -- Finalizing long-term cryptographic material handling if the prototype uses a simpler trust bootstrap. +- Finalizing long-term cryptographic material handling if the prototype uses a + simpler trust bootstrap. ## Architectural Approach This milestone should lock the following runtime boundary: - The CLI talks only to a Substrate-owned broker endpoint on the host. -- The host broker endpoint is owned by the daemon and permissioned so the intended local user can reach it, but cannot use it to escape the broker policy boundary. -- The daemon reaches the guest agent over a private transport that is not left exposed as a reusable developer-owned socket or direct shell path. -- The guest agent authenticates the daemon or broker, and the daemon verifies that it is talking to the expected guest instance. -- Requests are traced as broker-mediated operations, not as arbitrary local guest socket calls. +- The host broker endpoint is owned by the daemon and permissioned so the + intended local user can reach it, but cannot use it to escape the broker + policy boundary. +- The daemon reaches the guest agent over a private transport that is not left + exposed as a reusable developer-owned socket or direct shell path. +- The guest agent authenticates the daemon or broker, and the daemon verifies + that it is talking to the expected guest instance. +- Requests are traced as broker-mediated operations, not as arbitrary local + guest socket calls. +- The existing gateway lifecycle command family stays intact: + - `substrate world gateway sync` + - `substrate world gateway status` + - `substrate world gateway restart` + - `substrate world gateway status --json` +- The existing auth and status contracts stay intact: + - integrated auth remains the landed auth-bundle FD handoff described in + `docs/contracts/substrate-gateway-policy-evaluation.md` + - `status --json` remains governed by + `docs/contracts/substrate-gateway-status-schema.md` The milestone should also define how the existing transport candidates change: -- VSock may remain viable only if the daemon owns the host endpoint and the CLI never reaches the guest socket directly. -- SSH UDS forwarding in the current form should be considered incompatible with the target state because it places a host socket under the developer user’s home directory. +- VSock may remain viable only if the daemon owns the host endpoint and the CLI + never reaches the guest socket directly. +- SSH UDS forwarding in the current form should be considered incompatible with + the target state because it places a host socket under the developer user’s + home directory. - Direct SSH shell access is breakglass only. ## Dependencies / Sequencing - Depends on milestone 0.1’s daemon ownership contract. - Produces the boundary needed by phase 1 milestone 1.2. -- Must be reviewed against the current docs and diagnostics before phase 1 prototype work starts. +- Must be reviewed against the current docs and diagnostics before phase 1 + prototype work starts. ## Concrete Repo Surfaces and File Pointers - `crates/world-mac-lima/src/forwarding.rs` - Current direct VSock and SSH UDS logic to be conceptually replaced by a daemon-private forwarding model. + Current direct VSock and SSH UDS logic to be conceptually replaced by a + daemon-private forwarding model. - `crates/world-mac-lima/src/lib.rs` - Current endpoint creation and direct agent client setup to be replaced by broker client behavior. + Current endpoint creation and direct agent client setup to be replaced by + broker client behavior. - `crates/world-mac-lima/src/transport.rs` - Current host-visible transport selection assumptions that need a new classification: private daemon transport versus public broker endpoint. + Current host-visible transport selection assumptions that need a new + classification: private daemon transport versus public broker endpoint. +- `crates/shell/src/builtins/world_gateway.rs` + Current gateway lifecycle/status client that must keep its stable CLI surface + while changing its trust and transport boundary. +- `crates/world-agent/src/gateway_runtime.rs` + Current auth-bundle and managed-runtime path that must remain Substrate-owned + while moving behind the new macOS host owner. - `scripts/mac/lima-warm.sh` - Current guest socket activation and service bootstrap that may need broker identity bootstrap or daemon-to-guest trust material installation. -- `docs/WORLD.md` - Current description of host ⇄ guest transport using VSock and SSH forwarding that must be updated after implementation. -- `docs/cross-platform/mac_world_setup.md` - Current direct `curl --unix-socket /run/substrate.sock`, `limactl shell`, and socket troubleshooting flows that conflict with the new boundary. + Current guest socket activation and service bootstrap that may need broker + identity bootstrap or daemon-to-guest trust material installation. +- `../../../docs/WORLD.md` + Current description of host-to-guest transport using VSock and SSH + forwarding. It is evidence only and must be updated after implementation. +- `../../../docs/contracts/substrate-gateway-operator-contract.md` + Durable owner of the gateway lifecycle command family. +- `../../../docs/contracts/substrate-gateway-policy-evaluation.md` + Durable owner of the integrated auth and policy boundary. +- `../../../docs/contracts/substrate-gateway-status-schema.md` + Durable owner of `substrate world gateway status --json`. +- `../../../docs/contracts/substrate-gateway-runtime-parity.md` + Durable owner of the typed lifecycle/status runtime parity boundary. +- `../../../docs/cross-platform/mac_world_setup.md` + Current direct `curl --unix-socket /run/substrate.sock`, `limactl shell`, and + socket troubleshooting flows that conflict with the new boundary. ## Deliverables @@ -77,25 +136,45 @@ The milestone should also define how the existing transport candidates change: - daemon identity presented to the guest - guest identity verification by the daemon - key or token rotation expectations across VM recreation -- A diagnostics contract stating which checks remain routine and which become breakglass. +- A diagnostics contract stating which checks remain routine and which become + breakglass. ## Acceptance Criteria -- The milestone defines exactly one routine host entrypoint for macOS world execution. -- The milestone explicitly forbids developer-user routine access to a host-forwarded guest socket such as `~/.substrate/sock/agent.sock`. -- The milestone defines how broker-to-agent requests are authenticated or attested in the prototype. -- The milestone defines how `substrate host doctor` and `substrate world doctor` can continue to function without requiring direct guest socket reachability by the developer user. -- The milestone lists the current docs and scripts whose instructions become invalid under the new model. +- The milestone defines exactly one routine host entrypoint for macOS world + execution. +- The milestone explicitly forbids developer-user routine access to a + host-forwarded guest socket such as `~/.substrate/sock/agent.sock`. +- The milestone preserves the existing `substrate world gateway status --json` + authority line as the machine-readable gateway wiring surface. +- The milestone defines how broker-to-agent requests are authenticated or + attested in the prototype. +- The milestone defines how `substrate host doctor`, `substrate world doctor`, + `substrate health`, and `substrate world gateway status --json` can continue + to function without requiring direct guest socket reachability by the + developer user. +- The milestone lists the current docs and scripts whose instructions become + invalid under the new model. ## Validation / Evidence Plan -- Trace the current direct transport paths from `forwarding.rs` and demonstrate how each is either removed, privatized, or reclassified as breakglass. -- Produce a request-flow diagram for `CLI -> daemon broker -> guest agent`. -- Review the proposed boundary against the constraint “can a user with the normal CLI account talk to the guest agent without the broker?” The answer must be “no.” -- Confirm that planned doctor and health commands still have a routine evidence path through the broker. +- Trace the current direct transport paths from `forwarding.rs` and demonstrate + how each is either removed, privatized, or reclassified as breakglass. +- Produce a request-flow diagram for: + - `CLI -> daemon broker -> guest agent` + - `CLI -> daemon broker -> guest gateway lifecycle path` +- Review the proposed boundary against the constraint "can a user with the + normal CLI account talk to the guest agent without the broker?" The answer + must be "no." +- Confirm that planned doctor, health, and gateway commands still have a + routine evidence path through the broker. ## Risks / Open Questions -- The simplest prototype auth mechanism may not be the final production mechanism; the milestone must still define a trust anchor good enough to validate the boundary. -- Some debugging workflows will become slower once direct guest socket access is removed from normal use. -- If VSock support varies by environment, the private transport design needs a fallback that does not recreate a developer-owned socket escape hatch. +- The simplest prototype auth mechanism may not be the final production + mechanism; the milestone must still define a trust anchor good enough to + validate the boundary. +- Some debugging workflows will become slower once direct guest socket access + is removed from normal use. +- If VSock support varies by environment, the private transport design needs a + fallback that does not recreate a developer-owned socket escape hatch. diff --git a/macos-hardening/macos-ownership-separated-world/phase-1-control-plane-separation-prototype/README.md b/macos-hardening/macos-ownership-separated-world/phase-1-control-plane-separation-prototype/README.md index 0cd4070e7..adbbe4548 100644 --- a/macos-hardening/macos-ownership-separated-world/phase-1-control-plane-separation-prototype/README.md +++ b/macos-hardening/macos-ownership-separated-world/phase-1-control-plane-separation-prototype/README.md @@ -2,20 +2,33 @@ Status: Proposed +Last updated: 2026-05-19 + ## Purpose / Outcome -Build a working prototype of the ownership-separated macOS world control plane. Phase 1 ends when Substrate can boot and use a Lima-backed macOS world through a daemon-owned control plane, a single broker endpoint, and a breakglass channel that is no longer part of routine execution. +Build a working prototype of the ownership-separated macOS world control plane. +Phase 1 ends when Substrate can boot and use a Lima-backed macOS world through a +daemon-owned control plane, a single broker endpoint, and a breakglass channel +that is no longer part of routine execution. ## Why This Phase Exists -Phase 0 settles the architecture. Phase 1 proves the architecture can actually replace current same-user Lima behavior in this repo. The prototype should be strong enough to invalidate the current assumptions embedded in `MacLimaBackend`, `forwarding.rs`, `lima-warm.sh`, and the macOS operator docs. +Phase 0 settles the architecture. Phase 1 proves the architecture can actually +replace current same-user Lima behavior in this repo. The prototype should be +strong enough to invalidate the current assumptions embedded in +`MacLimaBackend`, `forwarding.rs`, `lima-warm.sh`, the gateway lifecycle client +path, and the macOS operator docs. ## In-Scope - Establish daemon-owned Lima state and instance lifecycle. -- Replace direct host-visible forwarding with a private forwarding model and one routine broker endpoint. +- Replace direct host-visible forwarding with a private forwarding model and + one routine broker endpoint. +- Carry current execution, doctor, and gateway lifecycle traffic over that new + path. - Add a breakglass maintenance channel that is explicit, gated, and auditable. -- Update the implementation and validation path enough to demonstrate end-to-end ownership separation. +- Update the implementation and validation path enough to demonstrate end-to-end + ownership separation. ## Out-of-Scope @@ -31,7 +44,14 @@ Phase 1 follows the contracts from phase 0 and lands in three milestones: 2. private forwarding plus a single broker endpoint 3. breakglass maintenance channel -The prototype should prefer minimal irreversible change in the Linux guest while making the host-side boundary unmistakable. +The prototype should prefer minimal irreversible change in the Linux guest while +making the host-side boundary unmistakable. It should preserve the current +operator command family and move it under the new owner: + +- `substrate host doctor` +- `substrate world doctor` +- `substrate health` +- `substrate world gateway sync|status|restart` ## Dependencies / Sequencing @@ -47,55 +67,86 @@ The prototype should prefer minimal irreversible change in the Linux guest while - Requires phase 0 milestone 0.1 and 0.2 acceptance. - Milestone 1.1 establishes ownership-separated host state first. - Milestone 1.2 depends on 1.1 and moves runtime traffic to the broker. -- Milestone 1.3 depends on 1.1 and 1.2 and reinstates only the maintenance access required for diagnosis and recovery. +- Milestone 1.3 depends on 1.1 and 1.2 and reinstates only the maintenance + access required for diagnosis and recovery. ## Concrete Repo Surfaces and File Pointers - `crates/world-mac-lima/src/lib.rs` - Primary host-side runtime path to redirect from direct VM and agent management to daemon IPC. + Primary host-side runtime path to redirect from direct VM and agent + management to daemon IPC. - `crates/world-mac-lima/src/forwarding.rs` Primary forwarding code to replace or quarantine behind daemon-private logic. - `crates/world-mac-lima/src/transport.rs` - Transport classification and endpoint selection surfaces to split into broker-public versus daemon-private paths. + Transport classification and endpoint selection surfaces to split into + broker-public versus daemon-private paths. +- `crates/shell/src/builtins/world_gateway.rs` + Current macOS gateway lifecycle/status client that phase 1 must rehome behind + the new broker boundary. +- `crates/world-agent/src/gateway_runtime.rs` + Current guest-owned managed runtime and auth-bundle seam that phase 1 must + reach through the new host owner. - `scripts/mac/lima-warm.sh` - Provisioning entrypoint that must either become daemon-internal or be replaced by daemon bootstrap hooks. + Provisioning entrypoint that must either become daemon-internal or be + replaced by daemon bootstrap hooks. - `scripts/mac/lima/substrate.yaml` - Guest definition and unit layout that may need trust-bootstrap or provisioning changes. + Guest definition and unit layout that may need trust-bootstrap or + provisioning changes. +- `scripts/mac/smoke.sh` + Prototype proof harness that already exercises gateway lifecycle and must be + updated to validate the new path. - `scripts/substrate/install-substrate.sh` - Install path that must provision the daemon-owned control plane, not same-user Lima. + Install path that must provision the daemon-owned control plane, not same-user + Lima. - `scripts/substrate/uninstall-substrate.sh` Teardown path that must remove daemon-owned state safely. -- `docs/WORLD.md` - Architecture doc that must eventually describe the new runtime path and breakglass classification. -- `docs/cross-platform/mac_world_setup.md` - Setup guide that must eventually stop teaching direct guest maintenance as routine. +- `../../../docs/WORLD.md` + Architecture evidence that must eventually describe the new runtime path and + breakglass classification. +- `../../../docs/contracts/substrate-gateway-operator-contract.md` + Durable gateway lifecycle command contract that phase 1 must preserve. ## Deliverables - Phase overview README. - Milestone 1.1, 1.2, and 1.3 SOWs. -- A prototype sequencing plan that ties implementation work back to the concrete repo surfaces above. -- A Phase 1 closeout artifact defining the stable macOS control-plane - registration mechanism that Phase 2 installer and migration work integrates +- A prototype sequencing plan that ties implementation work back to the + concrete repo surfaces above. +- A phase 1 closeout artifact defining the stable macOS control-plane + registration mechanism that phase 2 installer and migration work integrates into the shipped lifecycle. ## Acceptance Criteria -- The phase defines a prototype that is meaningfully different from same-user Lima and not just a wrapper around it. -- Each milestone has concrete deliverables, acceptance gates, and evidence requirements tied to current implementation surfaces. +- The phase defines a prototype that is meaningfully different from same-user + Lima and not just a wrapper around it. +- Each milestone has concrete deliverables, acceptance gates, and evidence + requirements tied to current implementation surfaces. - The breakglass path is explicitly narrower than current routine workflows. +- Prototype validation shows that gateway lifecycle traffic uses the same new + broker boundary rather than a leftover same-user forwarding seam. ## Validation / Evidence Plan -- Prototype validation must show developer-visible runtime access only through the broker endpoint. +- Prototype validation must show developer-visible runtime access only through + the broker endpoint. - Prototype validation must show daemon-owned state paths on the host. -- Prototype validation must show breakglass commands are disabled from normal runtime flow and require explicit operator action. +- Prototype validation must show breakglass commands are disabled from normal + runtime flow and require explicit operator action. +- Validation evidence should include: + - `substrate host doctor --json` + - `substrate world doctor --json` + - `substrate health --json` + - `substrate world gateway status --json` ## Risks / Open Questions -- The prototype may expose integration gaps in installer UX and daemon packaging that need a follow-on phase. -- The daemon-private transport may require extra guest bootstrap state not currently provisioned. -- Existing diagnostic scripts may need temporary compatibility shims during the prototype window. +- The prototype may expose integration gaps in installer UX and daemon + packaging that need a follow-on phase. +- The daemon-private transport may require extra guest bootstrap state not + currently provisioned. +- Existing diagnostic scripts may need temporary compatibility shims during the + prototype window. ## Milestones diff --git a/macos-hardening/macos-ownership-separated-world/phase-1-control-plane-separation-prototype/milestone-1-1-daemon-owned-lima-home-and-instance-lifecycle-sow.md b/macos-hardening/macos-ownership-separated-world/phase-1-control-plane-separation-prototype/milestone-1-1-daemon-owned-lima-home-and-instance-lifecycle-sow.md index 4d4bb4d28..dae2984f2 100644 --- a/macos-hardening/macos-ownership-separated-world/phase-1-control-plane-separation-prototype/milestone-1-1-daemon-owned-lima-home-and-instance-lifecycle-sow.md +++ b/macos-hardening/macos-ownership-separated-world/phase-1-control-plane-separation-prototype/milestone-1-1-daemon-owned-lima-home-and-instance-lifecycle-sow.md @@ -2,21 +2,35 @@ Status: Proposed +Last updated: 2026-05-19 + ## Purpose / Outcome -Prototype host-side ownership separation by moving Lima state and lifecycle under a Substrate-owned daemon or service account. The outcome is a running macOS world whose instance directory, credentials, and lifecycle commands are no longer routine developer-owned assets. +Prototype host-side ownership separation by moving Lima state and lifecycle +under a Substrate-owned daemon or service account. The outcome is a running +macOS world whose instance directory, credentials, guest-binary installation +path, and lifecycle commands are no longer routine developer-owned assets. ## Why This Milestone Exists -The current repo provisions and manages the macOS world as the developer user. `scripts/mac/lima-warm.sh` creates and configures the VM directly, and `MacLimaBackend` assumes the CLI can inspect and start the instance directly through `limactl`. Phase 1 must first break that ownership pattern before transport changes mean anything. +The current repo provisions and manages the macOS world as the developer user. +`scripts/mac/lima-warm.sh` creates and configures the VM directly, +`MacLimaBackend` assumes the CLI can inspect and start the instance directly +through `limactl`, and install/dev-install flows stage and verify +`substrate-gateway` inside the guest from a same-user lifecycle. Phase 1 must +first break that ownership pattern before transport changes mean anything. ## In-Scope - Create a daemon-owned state root for Lima control-plane artifacts. - Move or recreate `LIMA_HOME` under daemon ownership. - Move VM create/start/stop/provision operations behind daemon IPC. -- Update the macOS backend path so routine CLI execution no longer directly invokes `limactl`. -- Update install and uninstall planning so they manage daemon-owned state rather than same-user instance state. +- Update the macOS backend path so routine CLI execution no longer directly + invokes `limactl`. +- Update install, dev-install, world-enable, and uninstall planning so they + manage daemon-owned state rather than same-user instance state. +- Decide how guest-side `substrate-world-agent` and `substrate-gateway` + binaries are staged and verified under the new owner model. ## Out-of-Scope @@ -29,11 +43,13 @@ The current repo provisions and manages the macOS world as the developer user. ` Prototype the smallest credible ownership split: - the daemon owns the state root and launches `limactl` -- the daemon provisions or delegates provisioning of the guest image and guest binaries +- the daemon provisions or delegates provisioning of the guest image and guest + binaries - the CLI asks the daemon for lifecycle operations and health summaries - any remaining direct `limactl` usage is restricted to breakglass tooling -The prototype may choose to recreate the VM under daemon ownership instead of attempting in-place adoption of a user-owned `~/.lima/substrate` tree. +The prototype may choose to recreate the VM under daemon ownership instead of +attempting in-place adoption of a user-owned `~/.lima/substrate` tree. ## Dependencies / Sequencing @@ -43,33 +59,53 @@ The prototype may choose to recreate the VM under daemon ownership instead of at - phase 1 milestone 1.2 policy application parity - phase 1 milestone 1.3 doctor/smoke readiness parity - phase 3 milestone 3.2 breakglass reclassification and doc cutover -- Must land before milestone 1.2 because the broker/transport work depends on daemon-owned VM state and credentials. -- Should identify any state-root abstractions needed before installer changes are attempted. +- Must land before milestone 1.2 because the broker/transport work depends on + daemon-owned VM state and credentials. +- Should identify any state-root abstractions needed before installer changes + are attempted. ## Concrete Repo Surfaces and File Pointers - `crates/world-mac-lima/src/lib.rs` - Replace direct `ensure_vm_running`, `wait_for_agent`, and guest socket probing with daemon-mediated lifecycle requests. + Replace direct `ensure_vm_running`, `wait_for_agent`, and guest socket + probing with daemon-mediated lifecycle requests. - `crates/world-mac-lima/src/vm.rs` - Current `limactl` helpers to either move behind daemon internals or stop being called by routine CLI code. + Current `limactl` helpers to either move behind daemon internals or stop + being called by routine CLI code. - `crates/world-mac-lima/src/limactl.rs` - Current `limactl` discovery path that will no longer be a routine CLI requirement. + Current `limactl` discovery path that will no longer be a routine CLI + requirement. - `scripts/mac/lima-warm.sh` Extract or rehome provisioning logic so it can run under daemon ownership. - `scripts/mac/lima/substrate.yaml` - Evaluate any assumptions that the instance is created in a user-owned Lima home. + Evaluate any assumptions that the instance is created in a user-owned Lima + home. - `scripts/substrate/install-substrate.sh` - Add daemon installation/bootstrap flow instead of directly calling user-owned warm logic as the normal macOS path. + Add daemon installation/bootstrap flow instead of directly calling + user-owned warm logic as the normal macOS path. +- `scripts/substrate/dev-install-substrate.sh` + Decide whether dev-install remains intentionally same-user or gains an + ownership-separated mode. +- `scripts/substrate/world-enable.sh` + Replace direct same-user provisioning on macOS with a daemon-mediated hook or + retire the helper there. - `scripts/substrate/uninstall-substrate.sh` - Replace direct `limactl stop/delete substrate` assumptions with daemon-owned teardown. + Replace direct `limactl stop/delete substrate` assumptions with daemon-owned + teardown. +- `crates/world-agent/src/gateway_runtime.rs` + Confirm how the new owner model installs or verifies gateway runtime launch + prerequisites before gateway lifecycle calls are attempted. ## Deliverables -- Prototype design note for the daemon-owned state root and `LIMA_HOME` location. -- Implementation-ready lifecycle RPC list covering create/start/stop/status/provision. +- Prototype design note for the daemon-owned state root and `LIMA_HOME` + location. +- Implementation-ready lifecycle RPC list covering + create/start/stop/status/provision. - Prototype closeout note for the stable macOS control-plane registration - mechanism required by Phase 2 installer, upgrade, and migration work. -- A repo change map showing which current shell/backend code paths become daemon clients. + mechanism required by phase 2 installer, upgrade, and migration work. +- A repo change map showing which current shell/backend code paths become + daemon clients. - A migration strategy note for existing user-owned instances: - recreate - adopt with conversion @@ -77,20 +113,33 @@ The prototype may choose to recreate the VM under daemon ownership instead of at ## Acceptance Criteria -- The milestone defines a host state location owned by the daemon or dedicated service account, not the developer user. -- Routine CLI execution no longer requires direct `limactl` access in the intended prototype path. -- The milestone names the code and script seams where daemon lifecycle calls replace current direct lifecycle logic. -- The milestone states how install and uninstall flows treat daemon-owned instance state. +- The milestone defines a host state location owned by the daemon or dedicated + service account, not the developer user. +- Routine CLI execution no longer requires direct `limactl` access in the + intended prototype path. +- The milestone names the code and script seams where daemon lifecycle calls + replace current direct lifecycle logic. +- The milestone states how install, dev-install, world-enable, and uninstall + flows treat daemon-owned instance state. +- The milestone explains how guest `substrate-gateway` installation/verification + fits into the daemon-owned lifecycle. ## Validation / Evidence Plan -- Demonstrate that the prototype design has no routine dependency on `~/.lima/substrate` owned by the developer user. -- Walk the current `scripts/mac/lima-warm.sh` flow and mark which steps move behind daemon control. -- Walk the current `MacLimaBackend` startup path and show where direct lifecycle checks become daemon calls. -- Confirm that a normal developer session cannot create or start the routine VM without going through Substrate. +- Demonstrate that the prototype design has no routine dependency on + `~/.lima/substrate` owned by the developer user. +- Walk the current `scripts/mac/lima-warm.sh` flow and mark which steps move + behind daemon control. +- Walk the current `MacLimaBackend` startup path and show where direct lifecycle + checks become daemon calls. +- Confirm that a normal developer session cannot create or start the routine VM + without going through Substrate. ## Risks / Open Questions -- Lima may have undocumented assumptions about user-session state that complicate daemon ownership. -- Installer UX may need explicit privileged setup that is acceptable for the prototype but not for long-term product UX. -- Recreating instances may be simpler than migration, but it raises workspace continuity and debugging questions. +- Lima may have undocumented assumptions about user-session state that + complicate daemon ownership. +- Installer UX may need explicit privileged setup that is acceptable for the + prototype but not for long-term product UX. +- Recreating instances may be simpler than migration, but it raises workspace + continuity and debugging questions. diff --git a/macos-hardening/macos-ownership-separated-world/phase-1-control-plane-separation-prototype/milestone-1-2-private-forwarding-and-single-broker-endpoint-sow.md b/macos-hardening/macos-ownership-separated-world/phase-1-control-plane-separation-prototype/milestone-1-2-private-forwarding-and-single-broker-endpoint-sow.md index 6317e6218..499d92e3a 100644 --- a/macos-hardening/macos-ownership-separated-world/phase-1-control-plane-separation-prototype/milestone-1-2-private-forwarding-and-single-broker-endpoint-sow.md +++ b/macos-hardening/macos-ownership-separated-world/phase-1-control-plane-separation-prototype/milestone-1-2-private-forwarding-and-single-broker-endpoint-sow.md @@ -2,21 +2,37 @@ Status: Proposed +Last updated: 2026-05-19 + ## Purpose / Outcome -Prototype a runtime path where the developer-facing CLI reaches the macOS world only through one Substrate-owned broker endpoint, while daemon-to-guest transport stays private and authenticated. The outcome is removal of current developer-owned host-visible guest forwarding from routine execution. +Prototype a runtime path where the developer-facing CLI reaches the macOS world +only through one Substrate-owned broker endpoint, while daemon-to-guest +transport stays private and authenticated. The outcome is removal of current +developer-owned host-visible guest forwarding from routine execution. ## Why This Milestone Exists -Today `crates/world-mac-lima/src/forwarding.rs` can create a host socket in `~/.substrate/sock/agent.sock` or use a direct host port for VSock-backed access. That makes the guest agent reachable outside a Substrate-owned broker. Milestone 1.2 exists to close that hole in the prototype. +Today `crates/world-mac-lima/src/forwarding.rs` can create a host socket in +`~/.substrate/sock/agent.sock` or use a direct host port for VSock-backed +access. On macOS, the current `substrate world gateway sync|status|restart` +entrypoints also ride that same forwarded path through +`crates/shell/src/builtins/world_gateway.rs`. Milestone 1.2 exists to close +that hole in the prototype. ## In-Scope -- Replace routine direct host-visible guest forwarding with a daemon-private channel. +- Replace routine direct host-visible guest forwarding with a daemon-private + channel. - Introduce one developer-reachable broker endpoint owned by Substrate. -- Update the macOS backend runtime path so `AgentClient` traffic is broker-mediated rather than pointed at a guest-forwarded socket directly. -- Prototype broker-to-agent authentication or attestation as defined in phase 0. -- Update doctor and health planning so routine evidence flows through the broker. +- Update the macOS backend runtime path so `AgentClient` traffic is + broker-mediated rather than pointed at a guest-forwarded socket directly. +- Update the macOS gateway lifecycle client path so current + `sync|status|restart` traffic uses the same broker boundary. +- Prototype broker-to-agent authentication or attestation as defined in + phase 0. +- Update doctor and health planning so routine evidence flows through the + broker. ## Out-of-Scope @@ -29,9 +45,20 @@ Today `crates/world-mac-lima/src/forwarding.rs` can create a host socket in `~/. The prototype should separate public and private transports: - Public: one host-local broker endpoint with explicit ownership and ACLs. -- Private: daemon-to-guest transport using VSock, SSH, or another mechanism that is not exposed as a reusable developer-owned endpoint. +- Private: daemon-to-guest transport using VSock, SSH, or another mechanism + that is not exposed as a reusable developer-owned endpoint. + +The host CLI should stop thinking in terms of direct transport selection. +Instead, it should think in terms of contacting the daemon broker and letting +the daemon choose or maintain the private transport. + +The prototype must also preserve the durable gateway operator contract: -The host CLI should stop thinking in terms of direct transport selection. Instead, it should think in terms of contacting the daemon broker and letting the daemon choose or maintain the private transport. +- `substrate world gateway status --json` remains the authoritative + machine-readable wiring surface. +- Any runtime/log-path diagnostics surfaced through gateway lifecycle continue + to reference managed runtime artifacts under + `/run/substrate/substrate-gateway-runtime/`. ## Dependencies / Sequencing @@ -41,52 +68,88 @@ The host CLI should stop thinking in terms of direct transport selection. Instea - phase 1 milestone 1.2 policy application parity - phase 1 milestone 1.3 doctor/smoke readiness parity - phase 3 milestone 3.2 breakglass reclassification and doc cutover -- Requires milestone 1.1 because the daemon must own the VM state and transport artifacts. +- Requires milestone 1.1 because the daemon must own the VM state and + transport artifacts. - Implements the boundary specified by phase 0 milestone 0.2. -- Must land before milestone 1.3 so breakglass can be designed around the new routine path. +- Must land before milestone 1.3 so breakglass can be designed around the new + routine path. ## Concrete Repo Surfaces and File Pointers - `crates/world-mac-lima/src/forwarding.rs` - Replace current `auto_select`, `create_vsock_forwarding`, and `create_ssh_uds_forwarding` assumptions for routine CLI use. + Replace current `auto_select`, `create_vsock_forwarding`, and + `create_ssh_uds_forwarding` assumptions for routine CLI use. - `crates/world-mac-lima/src/lib.rs` - Replace direct agent endpoint derivation and local socket/TCP probing with broker endpoint usage. + Replace direct agent endpoint derivation and local socket/TCP probing with + broker endpoint usage. - `crates/world-mac-lima/src/transport.rs` - Rework current transport visibility assumptions so VSock or SSH are daemon-private implementation details, not CLI-facing choices. + Rework current transport visibility assumptions so VSock or SSH are + daemon-private implementation details, not CLI-facing choices. +- `crates/shell/src/builtins/world_gateway.rs` + Keep the current lifecycle command surface, but route it through the broker + rather than directly through Lima forwarding. +- `crates/world-agent/src/gateway_runtime.rs` + Keep current managed runtime semantics while changing who can reach the + runtime path. - `scripts/mac/lima-warm.sh` - Add any guest bootstrap required for daemon-to-agent trust or private transport support. -- `docs/WORLD.md` - Prepare to replace the current “host ⇄ guest transport” section that describes direct VSock and SSH forwarding. -- `docs/cross-platform/mac_world_setup.md` - Prepare to remove direct host socket and guest `curl` instructions from routine workflows. + Add any guest bootstrap required for daemon-to-agent trust or private + transport support. +- `scripts/mac/smoke.sh` + Update the existing gateway lifecycle proof to validate the brokered path. +- `../../../docs/WORLD.md` + Prepare to replace the current host-to-guest transport section that describes + direct VSock and SSH forwarding. +- `../../../docs/contracts/substrate-gateway-operator-contract.md` + Preserve the existing lifecycle command family and exit code contract. +- `../../../docs/contracts/substrate-gateway-status-schema.md` + Preserve the current `status --json` envelope and `client_wiring.*` owner + line. ## Deliverables - Prototype broker endpoint design and ownership model. -- Prototype private transport design with fallback rules that do not recreate a developer-owned bypass. +- Prototype private transport design with fallback rules that do not recreate a + developer-owned bypass. - A backend refactor plan showing how `MacLimaBackend` becomes a broker client. -- A diagnostics plan for `substrate host doctor` and `substrate world doctor` in the new path. +- A gateway-client refactor plan showing how the current lifecycle/status path + becomes broker-mediated. +- A diagnostics plan for `substrate host doctor`, `substrate world doctor`, + `substrate health`, and `substrate world gateway status --json` in the new + path. ## Acceptance Criteria -- Routine macOS execution uses one Substrate-owned broker endpoint, not a host-forwarded guest socket in the developer’s home directory. -- The milestone explicitly retires `~/.substrate/sock/agent.sock` from routine operation. -- The milestone defines how broker-to-agent requests are authenticated or attested in the prototype. -- The milestone preserves a routine validation path for world readiness and health without requiring direct guest socket access by the developer user. +- Routine macOS execution uses one Substrate-owned broker endpoint, not a + host-forwarded guest socket in the developer’s home directory. +- The milestone explicitly retires `~/.substrate/sock/agent.sock` from routine + operation. +- The milestone defines how broker-to-agent requests are authenticated or + attested in the prototype. +- The milestone preserves a routine validation path for world readiness, health, + and gateway lifecycle status without requiring direct guest socket access by + the developer user. ## Validation / Evidence Plan -- Show current `forwarding.rs` paths and identify which are deleted, privatized, or reserved for breakglass. +- Show current `forwarding.rs` paths and identify which are deleted, + privatized, or reserved for breakglass. - Produce a concrete runtime sequence for: - CLI command submission - broker request handling - daemon-to-agent execution + - gateway lifecycle request handling - result return -- Confirm there is no routine host artifact equivalent to the current developer-owned forwarded socket. -- Confirm diagnostic commands can produce evidence through broker-mediated calls. +- Confirm there is no routine host artifact equivalent to the current + developer-owned forwarded socket. +- Confirm diagnostic commands can produce evidence through broker-mediated + calls. ## Risks / Open Questions -- Some transport fallback designs are easier to implement but accidentally recreate host-visible escape hatches; those must be rejected even if convenient. -- Guest attestation may need additional state or identity material provisioned during VM creation. -- Existing tests and smoke scripts likely assume direct guest reachability and will need prototype-specific evidence paths. +- Some transport fallback designs are easier to implement but accidentally + recreate host-visible escape hatches; those must be rejected even if + convenient. +- Guest attestation may need additional state or identity material provisioned + during VM creation. +- Existing tests and smoke scripts likely assume direct guest reachability and + will need prototype-specific evidence paths. diff --git a/macos-hardening/macos-ownership-separated-world/phase-1-control-plane-separation-prototype/milestone-1-3-breakglass-maintenance-channel-sow.md b/macos-hardening/macos-ownership-separated-world/phase-1-control-plane-separation-prototype/milestone-1-3-breakglass-maintenance-channel-sow.md index 771ecb65a..ae7694868 100644 --- a/macos-hardening/macos-ownership-separated-world/phase-1-control-plane-separation-prototype/milestone-1-3-breakglass-maintenance-channel-sow.md +++ b/macos-hardening/macos-ownership-separated-world/phase-1-control-plane-separation-prototype/milestone-1-3-breakglass-maintenance-channel-sow.md @@ -2,20 +2,38 @@ Status: Proposed +Last updated: 2026-05-19 + ## Purpose / Outcome -Define and prototype the minimum maintenance path needed to recover, inspect, and repair the ownership-separated macOS world without making those operations routine runtime capabilities. The outcome is an explicit breakglass channel with gating, audit expectations, and repo-defined operator workflows. +Define and prototype the minimum maintenance path needed to recover, inspect, +and repair the ownership-separated macOS world without making those operations +routine runtime capabilities. The outcome is an explicit breakglass channel +with gating, audit expectations, and repo-defined operator workflows. ## Why This Milestone Exists -The current repo treats direct guest access as normal operations: `limactl shell`, guest `systemctl`, and direct `curl --unix-socket /run/substrate.sock` are written into docs and scripts. Once routine runtime access is brokered, the project still needs a way to diagnose failures and repair the environment. This milestone exists to make that path explicit and narrow instead of letting old same-user workflows linger informally. +The current repo treats direct guest access as normal operations: +`limactl shell`, guest `systemctl`, and direct +`curl --unix-socket /run/substrate.sock` are written into docs and scripts. +HEAD also has real same-user support surfaces for doctoring, health, and +gateway lifecycle. Once routine runtime access is brokered, the project still +needs a way to diagnose failures and repair the environment without letting old +same-user guest entry remain the de facto control plane. ## In-Scope -- Define the allowed breakglass operations for lifecycle repair, guest inspection, and agent recovery. -- Define how breakglass access is invoked, who can invoke it, and what evidence it must emit. -- Define how current direct maintenance instructions in docs and scripts are reclassified. -- Prototype an auditable breakglass path consistent with phase 0 contracts and phase 1 runtime changes. +- Define the allowed breakglass operations for lifecycle repair, guest + inspection, and agent recovery. +- Define how breakglass access is invoked, who can invoke it, and what evidence + it must emit. +- Define how current direct maintenance instructions in docs and scripts are + reclassified. +- Define how gateway-runtime diagnostics under + `/run/substrate/substrate-gateway-runtime/` are inspected in normal-path + support versus breakglass. +- Prototype an auditable breakglass path consistent with phase 0 contracts and + phase 1 runtime changes. ## Out-of-Scope @@ -25,15 +43,18 @@ The current repo treats direct guest access as normal operations: `limactl shell ## Architectural Approach -Breakglass should be treated as a maintenance mode, not as an alternate routine API. The prototype should: +Breakglass should be treated as a maintenance mode, not as an alternate routine +API. The prototype should: - require explicit operator action to enter breakglass mode - keep breakglass paths separate from normal CLI execution paths - emit traceable local evidence when breakglass is used - prefer daemon-mediated maintenance commands where possible -- reserve direct `limactl shell`, direct SSH, guest `systemctl`, and guest `curl` for the smallest necessary set of recovery operations +- reserve direct `limactl shell`, direct SSH, guest `systemctl`, and guest + `curl` for the smallest necessary set of recovery operations -The milestone should also define how breakglass exits and how operators return the system to normal broker-only operation. +The milestone should also define how breakglass exits and how operators return +the system to normal broker-only operation. ## Dependencies / Sequencing @@ -43,24 +64,36 @@ The milestone should also define how breakglass exits and how operators return t - phase 1 milestone 1.2 policy application parity - phase 1 milestone 1.3 doctor/smoke readiness parity - phase 3 milestone 3.2 breakglass reclassification and doc cutover -- Requires milestone 1.1 and 1.2 so breakglass is designed relative to the new normal path, not the old one. +- Requires milestone 1.1 and 1.2 so breakglass is designed relative to the new + normal path, not the old one. - Depends on phase 0 definitions of routine versus breakglass authority. - Should inform the follow-on doc rewrite for macOS setup and operations guides. ## Concrete Repo Surfaces and File Pointers -- `docs/WORLD.md` - Current operator narrative that still presents direct guest access as normal; future updates must mark these as breakglass only. -- `docs/cross-platform/mac_world_setup.md` - Current setup and troubleshooting instructions using direct `limactl shell`, guest `systemctl`, and direct socket `curl`. +- `../../../docs/WORLD.md` + Current operator narrative that still presents direct guest access as normal; + future updates must mark these as breakglass only. +- `../../../docs/cross-platform/mac_world_setup.md` + Current setup and troubleshooting instructions using direct `limactl shell`, + guest `systemctl`, and direct socket `curl`. +- `../../../docs/contracts/substrate-gateway-operator-contract.md` + Current gateway lifecycle owner line for normal-path operator commands. +- `crates/shell/src/builtins/world_gateway.rs` + Current gateway lifecycle/status command implementation that should remain + routine while direct guest entry becomes breakglass only. - `scripts/mac/lima-warm.sh` - Current repair and reprovision logic that may be retained only as a breakglass or daemon-internal tool. + Current repair and reprovision logic that may be retained only as a + breakglass or daemon-internal tool. - `scripts/mac/lima-doctor.sh` - Current direct diagnostic model that may need a breakglass classification or a broker-aware split. + Current direct diagnostic model that may need a breakglass classification or + a broker-aware split. - `scripts/substrate/install-substrate.sh` - Current install path that needs a documented recovery story when daemon-owned provisioning fails. + Current install path that needs a documented recovery story when + daemon-owned provisioning fails. - `scripts/substrate/uninstall-substrate.sh` - Current teardown path that needs a safe operator recovery path for partial or failed ownership-separated installs. + Current teardown path that needs a safe operator recovery path for partial or + failed ownership-separated installs. ## Deliverables @@ -68,31 +101,47 @@ The milestone should also define how breakglass exits and how operators return t - VM inspection - service status - agent socket probing + - gateway runtime log inspection - forced reprovision - full teardown and rebuild - Entry and exit rules for breakglass mode. - An evidence checklist for breakglass sessions. -- A mapping from current docs/scripts to future breakglass-only usage or retirement. +- A mapping from current docs/scripts to future breakglass-only usage or + retirement. ## Acceptance Criteria -- The milestone clearly separates routine runtime operations from maintenance-only operations. -- Direct `limactl shell`, direct SSH, guest `systemctl`, and guest `curl` are documented as breakglass only. -- The milestone defines how breakglass usage is made visible in logs, traces, or operator evidence. -- The milestone defines how to return a system from breakglass to normal broker-only operation. +- The milestone clearly separates routine runtime operations from + maintenance-only operations. +- Direct `limactl shell`, direct SSH, guest `systemctl`, and guest `curl` are + documented as breakglass only. +- The milestone defines how breakglass usage is made visible in logs, traces, or + operator evidence. +- The milestone defines how to return a system from breakglass to normal + broker-only operation. +- The milestone preserves normal-path use of `substrate host doctor`, + `substrate world doctor`, `substrate health`, and + `substrate world gateway status --json`. ## Validation / Evidence Plan -- Review every currently documented direct guest maintenance command and classify it as: +- Review every currently documented direct guest maintenance command and + classify it as: - retired - daemon-mediated - or breakglass-only -- Confirm the prototype still has a recoverability story when broker setup fails. -- Confirm breakglass is not required for normal doctor, health, or execution flows. -- Produce an operator evidence checklist that a future implementation PR can satisfy. +- Confirm the prototype still has a recoverability story when broker setup + fails. +- Confirm breakglass is not required for normal doctor, health, or gateway + lifecycle flows. +- Produce an operator evidence checklist that a future implementation PR can + satisfy. ## Risks / Open Questions -- If breakglass is too convenient, teams will keep using it as the real control plane; the prototype must avoid that. -- Some failure scenarios may still require more direct guest access than desired until daemon recovery flows mature. -- The repo may need temporary dual documentation during migration from same-user Lima to ownership-separated operation. +- If breakglass is too convenient, teams will keep using it as the real control + plane; the prototype must avoid that. +- Some failure scenarios may still require more direct guest access than + desired until daemon recovery flows mature. +- The repo may need temporary dual documentation during migration from + same-user Lima to ownership-separated operation. diff --git a/macos-hardening/macos-ownership-separated-world/phase-2-productization-and-migration/README.md b/macos-hardening/macos-ownership-separated-world/phase-2-productization-and-migration/README.md index 5c49861fb..3d9a88cb0 100644 --- a/macos-hardening/macos-ownership-separated-world/phase-2-productization-and-migration/README.md +++ b/macos-hardening/macos-ownership-separated-world/phase-2-productization-and-migration/README.md @@ -4,12 +4,12 @@ Status: Draft Owner: macOS world backend / installer / operator UX -Last updated: 2026-04-28 +Last updated: 2026-05-19 ## Purpose / Outcome Phase 2 turns the ownership-separated macOS world design into a shippable -operator workflow. The outcome is not “a secure architecture sketch,” but a +operator workflow. The outcome is not "a secure architecture sketch," but a macOS release path that can install, upgrade, migrate, diagnose, support, and validate the ownership-separated backend without telling normal operators to use direct `limactl shell`, direct guest `systemctl`, or direct guest `curl`. @@ -20,6 +20,7 @@ At the end of this phase, a supported macOS user should be able to: - upgrade from an older ownership-separated release - migrate from the current same-user Lima model - inspect health and status from canonical Substrate CLI surfaces +- use the existing gateway lifecycle/status surface through the new owner model - collect support evidence and perform bounded repair - remove Substrate cleanly - prove GA readiness through a published validation matrix @@ -37,6 +38,8 @@ The current repo still assumes same-user Lima operations in several places: - doctor and smoke scripts prove readiness by shelling into the guest directly - macOS status reporting is fragmented across shell code, helper scripts, and doc-only troubleshooting instructions +- the current gateway lifecycle surface is real and useful, but it still + depends on the same same-user Lima ownership model that phase 1 is replacing If those surfaces are not rewritten, the product still behaves like a developer tooling prototype even if the runtime boundary becomes stronger underneath. @@ -46,11 +49,12 @@ tooling prototype even if the runtime boundary becomes stronger underneath. - macOS install, upgrade, migration, rollback, and uninstall workflows for the ownership-separated backend - migration from same-user Lima state into the new control-plane owner model -- canonical doctor and status surfaces for host, world, and support workflows +- canonical doctor and status surfaces for host, world, gateway, and support + workflows - operator-facing supportability flows: log collection, repair guidance, state reporting, and breakglass boundaries -- a GA validation matrix covering install lifecycle, runtime lifecycle, and - failure-mode evidence +- a GA validation matrix covering install lifecycle, runtime lifecycle, gateway + lifecycle, and failure-mode evidence - documentation updates required to make the new path the default operator contract @@ -70,8 +74,10 @@ Phase 2 standardizes around one operator contract: 1. Host lifecycle is owned by Substrate-managed install and runtime surfaces, not by user-owned Lima state. 2. CLI status and doctor commands are the primary support interface. -3. Docs, smoke coverage, and uninstall behavior follow the same contract. -4. Direct guest access becomes breakglass-only, explicitly segregated from the +3. The current gateway lifecycle command family stays intact and is consumed as + a durable contract, not reinvented by local planning docs. +4. Docs, smoke coverage, and uninstall behavior follow the same contract. +5. Direct guest access becomes breakglass-only, explicitly segregated from the normal install and support path. The implementation threads through existing repo surfaces rather than inventing @@ -82,6 +88,8 @@ parallel tooling: contract followers rather than contract owners - CLI doctor/status behavior remains rooted in `crates/shell/src/execution/platform/macos.rs` +- gateway lifecycle/status behavior remains rooted in + `crates/shell/src/builtins/world_gateway.rs` plus the durable contract docs - operator docs remain rooted in `docs/INSTALLATION.md`, `docs/UNINSTALL.md`, `docs/WORLD.md`, and `docs/cross-platform/mac_world_setup.md` @@ -90,8 +98,8 @@ parallel tooling: Hard dependencies: - The same-user hardening track should already have converged the transport, - mount, and operator vocabulary so migration into ownership separation does not - inherit ambiguous same-user workflows. + mount, gateway runtime parity, and operator vocabulary so migration into + ownership separation does not inherit ambiguous same-user workflows. - More concretely, phase 2 assumes the same-user track has completed: - phase 2 milestone 2.2 mount minimization and ingress contract - phase 3 milestone 3.1 Substrate-managed diagnostics and lifecycle @@ -101,7 +109,7 @@ Hard dependencies: - The normal-path transport contract must already stop exposing same-user forwarded endpoints as the security boundary. -Execution order inside Phase 2: +Execution order inside phase 2: 1. Milestone 2.1 defines how the product gets onto the machine and how legacy same-user installs are migrated safely. @@ -115,17 +123,23 @@ Execution order inside Phase 2: Primary implementation and contract surfaces for this phase: +- `scripts/substrate/world-enable.sh` +- `scripts/substrate/dev-install-substrate.sh` - `scripts/substrate/install-substrate.sh` -- `scripts/substrate/uninstall-substrate.sh` - `scripts/substrate/dev-uninstall-substrate.sh` +- `scripts/substrate/uninstall-substrate.sh` - `scripts/mac/lima-warm.sh` - `scripts/mac/lima-doctor.sh` - `scripts/mac/smoke.sh` - `crates/shell/src/execution/platform/macos.rs` -- `docs/INSTALLATION.md` -- `docs/UNINSTALL.md` -- `docs/WORLD.md` -- `docs/cross-platform/mac_world_setup.md` +- `crates/shell/src/builtins/world_gateway.rs` +- `../../../docs/INSTALLATION.md` +- `../../../docs/UNINSTALL.md` +- `../../../docs/WORLD.md` +- `../../../docs/contracts/substrate-gateway-operator-contract.md` +- `../../../docs/contracts/substrate-gateway-status-schema.md` +- `../../../docs/contracts/substrate-gateway-policy-evaluation.md` +- `../../../docs/cross-platform/mac_world_setup.md` Secondary reference surfaces likely touched by downstream execution: @@ -136,6 +150,7 @@ Secondary reference surfaces likely touched by downstream execution: - `crates/world-mac-lima/src/lib.rs` - `crates/world-mac-lima/src/forwarding.rs` - `crates/world-mac-lima/src/transport.rs` +- `crates/world-agent/src/gateway_runtime.rs` ## Milestones @@ -165,29 +180,33 @@ validation matrix, evidence packaging, and release-blocking criteria. - one phase overview that fixes the productization scope and sequence - three milestone SOWs with concrete repo surfaces, deliverables, and gates - an explicit migration story from same-user Lima to ownership-separated macOS -- an explicit operator contract that replaces direct guest workflows +- an explicit operator contract that replaces direct guest workflows without + destabilizing current CLI surfaces - a GA evidence plan that can be executed and audited before release ## Acceptance Criteria - The phase docs describe one consistent normal-path operator contract across - install, health checks, troubleshooting, and uninstall. + install, health checks, troubleshooting, uninstall, and gateway lifecycle. - Same-user Lima migration is treated as a first-class lifecycle path, not a footnote. - Each milestone names concrete repo surfaces instead of generic workstreams. - Phase sequencing is explicit enough that execution can proceed milestone by milestone without reopening the scope question. +- `docs/WORLD.md` is treated as descriptive evidence and the durable gateway + contracts under `docs/contracts/` remain the owner lines for operator/status + semantics. - The phase does not rely on direct guest access as part of the normal support posture. ## Validation / Evidence Plan - Use the repo-backed evidence in - `thoughts/shared/research/2026-04-28-macos-lima-parity-lockdown.md` as the - baseline problem statement. + [the research note](../../research/2026-04-28-macos-lima-parity-lockdown.md) + as the baseline problem statement. - For milestone closeout, require install/uninstall transcripts, doctor/status - JSON, smoke artifacts, and doc diffs from the concrete repo surfaces named in - each SOW. + JSON, gateway lifecycle JSON, smoke artifacts, and doc diffs from the + concrete repo surfaces named in each SOW. - Treat migration from an existing same-user Lima installation as mandatory evidence, not optional beta validation. - Treat reboot, relogin, re-upgrade, and uninstall cleanup as mandatory diff --git a/macos-hardening/macos-ownership-separated-world/phase-2-productization-and-migration/milestone-2-1-installer-upgrade-and-migration-path-sow.md b/macos-hardening/macos-ownership-separated-world/phase-2-productization-and-migration/milestone-2-1-installer-upgrade-and-migration-path-sow.md index 6f01960b3..43aa947bc 100644 --- a/macos-hardening/macos-ownership-separated-world/phase-2-productization-and-migration/milestone-2-1-installer-upgrade-and-migration-path-sow.md +++ b/macos-hardening/macos-ownership-separated-world/phase-2-productization-and-migration/milestone-2-1-installer-upgrade-and-migration-path-sow.md @@ -4,7 +4,7 @@ Status: Draft Phase: 2 - Productization and Migration -Last updated: 2026-04-28 +Last updated: 2026-05-19 ## Purpose / Outcome @@ -19,6 +19,10 @@ machines. The current macOS lifecycle is same-user by construction: - `scripts/substrate/install-substrate.sh` provisions Lima as the invoking user +- `scripts/substrate/world-enable.sh` reuses the same-user installer + provisioning helpers +- `scripts/substrate/dev-install-substrate.sh` stages guest binaries and writes + same-user world metadata - `scripts/substrate/uninstall-substrate.sh` deletes the user-owned Lima VM - `scripts/substrate/dev-uninstall-substrate.sh` assumes user-owned helper state @@ -39,6 +43,7 @@ the host-side lifecycle around the new owner model. - installer/uninstaller metadata needed to distinguish legacy, migrated, and native ownership-separated states - operator documentation updates for install, upgrade, migration, and uninstall +- lifecycle integration for the already-landed gateway/runtime surfaces ## Out of Scope @@ -56,12 +61,14 @@ ownership: 1. Detect whether the machine is a clean install, an ownership-separated upgrade, or a same-user Lima migration candidate. 2. Materialize or verify the dedicated host owner and private control-plane - state required by Phase 1. + state required by phase 1. 3. Migrate only the artifacts that are safe and necessary to preserve. Do not carry forward same-user-owned control-plane material as trusted state. 4. Write explicit install metadata so upgrade and uninstall paths know whether the host is legacy, migrated, or natively ownership-separated. 5. Expose one documented rollback posture for interrupted or failed migration. +6. Ensure the normal lifecycle path still lands with working world and gateway + operator surfaces. The uninstall path must understand the new metadata and clean up the new owner model without guessing based on same-user Lima conventions. @@ -74,8 +81,8 @@ Depends on: - phase 2 milestone 2.2 mount minimization and ingress contract - phase 3 milestone 3.1 Substrate-managed diagnostics and lifecycle - phase 3 milestone 3.2 breakglass reclassification and doc cutover -- Phase 1 host ownership separation -- a stable macOS control-plane registration mechanism produced as a Phase 1 +- phase 1 host ownership separation +- a stable macOS control-plane registration mechanism produced as a phase 1 prototype closeout artifact, with this milestone owning its install, upgrade, and migration integration into the shipped macOS lifecycle - a stable normal-path transport endpoint that no longer treats user-owned @@ -83,37 +90,42 @@ Depends on: Sequences before: -- Milestone 2.2, because doctor/status/supportability must report the actual +- milestone 2.2, because doctor/status/supportability must report the actual lifecycle state this milestone defines -- Milestone 2.3, because the GA matrix depends on install, migration, and +- milestone 2.3, because the GA matrix depends on install, migration, and uninstall behaviors being fixed first ## Concrete Repo Surfaces and File Pointers Primary surfaces: +- `scripts/substrate/world-enable.sh` +- `scripts/substrate/dev-install-substrate.sh` - `scripts/substrate/install-substrate.sh` - `scripts/substrate/install.sh` +- `scripts/substrate/dev-uninstall-substrate.sh` - `scripts/substrate/uninstall-substrate.sh` - `scripts/substrate/uninstall.sh` -- `scripts/substrate/dev-uninstall-substrate.sh` - `scripts/mac/lima-warm.sh` -- `docs/INSTALLATION.md` -- `docs/UNINSTALL.md` -- `docs/WORLD.md` -- `docs/cross-platform/mac_world_setup.md` +- `../../../docs/INSTALLATION.md` +- `../../../docs/UNINSTALL.md` +- `../../../docs/WORLD.md` +- `../../../docs/cross-platform/mac_world_setup.md` State and runtime surfaces likely affected: - `scripts/mac/lima/substrate.yaml` - `scripts/mac/lima/substrate-dev.yaml` - `crates/shell/src/execution/platform/macos.rs` +- `crates/shell/src/builtins/world_gateway.rs` +- `crates/world-agent/src/gateway_runtime.rs` Legacy-state evidence to account for: - user-owned `~/.lima` - user-owned `~/.substrate` - same-user forwarded endpoints described in current docs and scripts +- same-user guest binary staging and verification for `substrate-gateway` ## Deliverables @@ -121,7 +133,7 @@ Legacy-state evidence to account for: - a macOS upgrade contract for already-migrated hosts - a same-user Lima migration contract with explicit preserve/recreate/discard rules for existing host and guest artifacts -- a shipped lifecycle integration plan for the Phase 1 control-plane +- a shipped lifecycle integration plan for the phase 1 control-plane registration mechanism, including clean install bootstrap, upgrade handling, and uninstall cleanup expectations - installer metadata schema updates needed to encode ownership model and @@ -143,6 +155,9 @@ Legacy-state evidence to account for: guest `systemctl`, or guest `curl`. - The install flow records enough metadata for downstream doctor/status surfaces to distinguish clean, upgraded, migrated, and partially failed states. +- The post-install normal path leaves `substrate host doctor`, + `substrate world doctor`, `substrate health`, and + `substrate world gateway status --json` working through the new owner model. ## Validation / Evidence Plan @@ -151,12 +166,16 @@ Legacy-state evidence to account for: - Capture migration transcripts from a machine preloaded with the current same-user Lima install path. - Capture rerun evidence showing idempotent install and upgrade behavior. -- Capture uninstall transcripts for both migrated and clean ownership-separated - hosts. +- Capture uninstall transcripts for both migrated and clean + ownership-separated hosts. - Capture post-install and post-migration readiness with: - `substrate host doctor --json` - `substrate world doctor --json` - `substrate health --json` + - `substrate world gateway status --json` +- Capture at least one gateway lifecycle round trip after install: + - `substrate world gateway sync` + - `substrate world gateway restart` - Update `docs/INSTALLATION.md`, `docs/UNINSTALL.md`, `docs/WORLD.md`, and `docs/cross-platform/mac_world_setup.md` so the documented contract matches the transcripts above. diff --git a/macos-hardening/macos-ownership-separated-world/phase-2-productization-and-migration/milestone-2-2-operator-doctor-status-and-supportability-sow.md b/macos-hardening/macos-ownership-separated-world/phase-2-productization-and-migration/milestone-2-2-operator-doctor-status-and-supportability-sow.md index fda5f94c3..3cc347b05 100644 --- a/macos-hardening/macos-ownership-separated-world/phase-2-productization-and-migration/milestone-2-2-operator-doctor-status-and-supportability-sow.md +++ b/macos-hardening/macos-ownership-separated-world/phase-2-productization-and-migration/milestone-2-2-operator-doctor-status-and-supportability-sow.md @@ -4,12 +4,12 @@ Status: Draft Phase: 2 - Productization and Migration -Last updated: 2026-04-28 +Last updated: 2026-05-19 ## Purpose / Outcome -Deliver the canonical macOS operator surfaces for the ownership-separated world: -health, lifecycle status, support evidence, repair guidance, and bounded +Deliver the canonical macOS operator surfaces for the ownership-separated +world: health, lifecycle status, support evidence, repair guidance, and bounded breakglass. After this milestone, normal macOS support work should be possible through Substrate-owned commands and docs rather than direct guest entry. @@ -18,6 +18,7 @@ through Substrate-owned commands and docs rather than direct guest entry. Today’s macOS support posture is split across: - `crates/shell/src/execution/platform/macos.rs` doctor logic +- `crates/shell/src/builtins/world_gateway.rs` gateway lifecycle/status logic - `scripts/mac/lima-doctor.sh` guest-shell-based checks - `scripts/mac/smoke.sh` as the main proof harness - `docs/WORLD.md` and `docs/cross-platform/mac_world_setup.md` troubleshooting @@ -25,9 +26,9 @@ Today’s macOS support posture is split across: and `limactl shell` That is acceptable for a same-user prototype, but not for a product whose core -claim is host-side ownership separation. Operators need one authoritative status -story that explains whether the new owner, transport, world agent, broker path, -migration state, and runtime health are correct. +claim is host-side ownership separation. Operators need one authoritative +status story that explains whether the new owner, transport, world agent, +gateway path, migration state, and runtime health are correct. ## In Scope @@ -48,8 +49,8 @@ migration state, and runtime health are correct. ## Architectural Approach -This milestone makes the CLI the contract owner and demotes helper scripts to -followers: +This milestone makes the CLI and durable contract docs the owner lines, and +demotes helper scripts to followers: 1. `substrate host doctor --json` reports host prerequisites, owner-model state, lifecycle state, and broker reachability. @@ -57,13 +58,14 @@ followers: readiness through the supported broker path. 3. `substrate health --json` summarizes operator actionability and must explain whether the problem is install state, owner state, transport state, world - state, or migration state. -4. A dedicated lifecycle/status surface is added for macOS ownership-separated - deployments. If implemented as `substrate world status --json`, it becomes - the machine-readable contract for support tooling. If the team chooses not to - add a new subcommand, the equivalent schema must still exist in a canonical - CLI JSON output. -5. `scripts/mac/lima-doctor.sh` becomes a support-engineering or breakglass + state, gateway state, or migration state. +4. `substrate world gateway status --json` remains the authoritative + machine-readable gateway wiring surface. This milestone must consume that + contract, not replace it. +5. If macOS-specific lifecycle state needs additive JSON beyond the existing + gateway status envelope, it should live in the appropriate doctor/health or + separate CLI surface without redefining the gateway contract. +6. `scripts/mac/lima-doctor.sh` becomes a support-engineering or breakglass helper, not the primary operator entrypoint. ## Dependencies / Sequencing @@ -74,12 +76,12 @@ Depends on: - phase 2 milestone 2.2 mount minimization and ingress contract - phase 3 milestone 3.1 Substrate-managed diagnostics and lifecycle - phase 3 milestone 3.2 breakglass reclassification and doc cutover -- Milestone 2.1 install and migration state definitions -- Phase 1 host ownership separation and broker architecture +- milestone 2.1 install and migration state definitions +- phase 1 host ownership separation and broker architecture Sequences before: -- Milestone 2.3, because the GA matrix needs stable status semantics and +- milestone 2.3, because the GA matrix needs stable status semantics and support evidence surfaces ## Concrete Repo Surfaces and File Pointers @@ -87,18 +89,23 @@ Sequences before: Primary surfaces: - `crates/shell/src/execution/platform/macos.rs` +- `crates/shell/src/builtins/world_gateway.rs` - `scripts/mac/lima-doctor.sh` - `scripts/mac/smoke.sh` -- `docs/WORLD.md` -- `docs/cross-platform/mac_world_setup.md` -- `docs/INSTALLATION.md` -- `docs/UNINSTALL.md` +- `../../../docs/WORLD.md` +- `../../../docs/cross-platform/mac_world_setup.md` +- `../../../docs/INSTALLATION.md` +- `../../../docs/UNINSTALL.md` +- `../../../docs/contracts/substrate-gateway-operator-contract.md` +- `../../../docs/contracts/substrate-gateway-status-schema.md` +- `../../../docs/contracts/substrate-gateway-policy-evaluation.md` Related implementation surfaces likely touched by downstream execution: - `crates/world-mac-lima/src/lib.rs` - `crates/world-mac-lima/src/forwarding.rs` - `crates/world-mac-lima/src/transport.rs` +- `crates/world-agent/src/gateway_runtime.rs` - `scripts/mac/lima-warm.sh` ## Deliverables @@ -120,15 +127,20 @@ Related implementation surfaces likely touched by downstream execution: ## Acceptance Criteria - A supported operator can determine install state, owner-model state, broker - state, and world readiness without using direct guest commands. + state, gateway state, and world readiness without using direct guest + commands. - CLI JSON output distinguishes at least these macOS states: clean ownership-separated install, migrated install, partial migration, - unavailable world backend, and breakglass-required failure. + unavailable world backend, unavailable gateway component, and + breakglass-required failure. - Human-readable doctor/health output explains next actions concretely instead of only reporting that a check failed. - `scripts/mac/lima-doctor.sh` no longer defines the normal support contract. - The published docs clearly separate normal-path support commands from breakglass-only procedures. +- Managed gateway runtime artifact paths under + `/run/substrate/substrate-gateway-runtime/` are surfaced as part of the + supported operator story when lifecycle failures reference them. ## Validation / Evidence Plan @@ -139,21 +151,23 @@ Related implementation surfaces likely touched by downstream execution: - broken broker path - broken world-agent path - incomplete migration state -- Expand `scripts/mac/smoke.sh` evidence to assert the new doctor/status schema - and human-readable remediation posture. -- If a standalone status surface is introduced, capture its JSON output in the - same scenarios above and document it in `docs/WORLD.md`. +- Capture `substrate world gateway status --json` in the same scenarios above, + plus after `substrate world gateway sync` and `restart`. +- Expand `scripts/mac/smoke.sh` evidence to assert the doctor/status schema and + human-readable remediation posture. - Compare the new CLI outputs against the legacy `scripts/mac/lima-doctor.sh` checks to ensure the script is no longer the only place where critical macOS health signals exist. +- Treat `docs/WORLD.md` as descriptive context only; validate gateway lifecycle + meaning against the durable contract docs under `docs/contracts/`. ## Risks / Open Questions - Where should ownership and migration state live in the CLI schema so it is stable enough for support automation but does not duplicate existing doctor payloads awkwardly? -- How much breakglass detail should be exposed in normal operator docs before it - starts undermining the product boundary? +- How much breakglass detail should be exposed in normal operator docs before + it starts undermining the product boundary? - If helper scripts remain for deep support, how do we keep them from drifting away from the CLI contract again? - What is the minimum viable support bundle for macOS incidents without diff --git a/macos-hardening/macos-ownership-separated-world/phase-2-productization-and-migration/milestone-2-3-ga-readiness-and-validation-matrix-sow.md b/macos-hardening/macos-ownership-separated-world/phase-2-productization-and-migration/milestone-2-3-ga-readiness-and-validation-matrix-sow.md index 0823d9905..addd48ed7 100644 --- a/macos-hardening/macos-ownership-separated-world/phase-2-productization-and-migration/milestone-2-3-ga-readiness-and-validation-matrix-sow.md +++ b/macos-hardening/macos-ownership-separated-world/phase-2-productization-and-migration/milestone-2-3-ga-readiness-and-validation-matrix-sow.md @@ -4,7 +4,7 @@ Status: Draft Phase: 2 - Productization and Migration -Last updated: 2026-04-28 +Last updated: 2026-05-19 ## Purpose / Outcome @@ -18,16 +18,19 @@ blockers. Without a formal validation matrix, the macOS ownership-separated path could ship with one successful demo path but miss the real release-critical cases: same-user migration, reinstall, reboot/relogin persistence, partial failure -recovery, uninstall cleanup, and operator supportability under broken states. +recovery, uninstall cleanup, gateway lifecycle parity, and operator +supportability under broken states. The current repo already has test and evidence entrypoints, but they do not yet express a GA gate for this new product claim: -- `scripts/mac/smoke.sh` proves selected runtime flows -- `scripts/mac/lima-doctor.sh` proves selected host/guest checks -- `docs/INSTALLATION.md`, `docs/UNINSTALL.md`, and - `docs/cross-platform/mac_world_setup.md` describe operator behavior but do not - define a release matrix +- `scripts/mac/smoke.sh` already proves selected runtime and gateway lifecycle + flows +- `scripts/mac/lima-doctor.sh` proves selected host/guest checks in the current + same-user model +- `docs/INSTALLATION.md`, `docs/UNINSTALL.md`, `docs/WORLD.md`, and + `docs/cross-platform/mac_world_setup.md` describe operator behavior but do + not define a release matrix This milestone closes that gap. @@ -35,10 +38,11 @@ This milestone closes that gap. - the macOS ownership-separated GA validation matrix - mandatory scenario coverage for install, upgrade, migration, reboot, repair, - uninstall, and support workflows + gateway lifecycle, uninstall, and support workflows - artifact expectations for each matrix row - release-blocking criteria -- final doc consistency review across install, world, and uninstall guidance +- final doc consistency review across install, world, gateway, and uninstall + guidance ## Out of Scope @@ -53,7 +57,7 @@ The GA gate is evidence-first and scenario-based: 1. Define the supported macOS versions, hardware assumptions, and install origins in a matrix that maps directly to repo verification surfaces. 2. For each scenario, name the expected artifacts: install transcript, doctor - JSON, health JSON, status JSON if present, smoke artifacts, and uninstall + JSON, health JSON, gateway status JSON, smoke artifacts, and uninstall evidence where applicable. 3. Treat migration from same-user Lima as a first-class GA scenario, not a beta follow-up. @@ -70,8 +74,8 @@ Depends on: - phase 2 milestone 2.2 mount minimization and ingress contract - phase 3 milestone 3.1 Substrate-managed diagnostics and lifecycle - phase 3 milestone 3.2 breakglass reclassification and doc cutover -- Milestone 2.1 for install/upgrade/migration/uninstall behavior -- Milestone 2.2 for doctor/status/supportability behavior +- milestone 2.1 for install/upgrade/migration/uninstall behavior +- milestone 2.2 for doctor/status/supportability behavior Final milestone in this phase: @@ -83,27 +87,34 @@ Primary evidence and contract surfaces: - `scripts/mac/smoke.sh` - `scripts/mac/lima-doctor.sh` +- `scripts/substrate/world-enable.sh` +- `scripts/substrate/dev-install-substrate.sh` - `scripts/substrate/install-substrate.sh` - `scripts/substrate/uninstall-substrate.sh` - `crates/shell/src/execution/platform/macos.rs` -- `docs/INSTALLATION.md` -- `docs/UNINSTALL.md` -- `docs/WORLD.md` -- `docs/cross-platform/mac_world_setup.md` +- `crates/shell/src/builtins/world_gateway.rs` +- `../../../docs/INSTALLATION.md` +- `../../../docs/UNINSTALL.md` +- `../../../docs/WORLD.md` +- `../../../docs/contracts/substrate-gateway-operator-contract.md` +- `../../../docs/contracts/substrate-gateway-status-schema.md` +- `../../../docs/cross-platform/mac_world_setup.md` Likely supporting surfaces: - `scripts/mac/lima-warm.sh` - `scripts/mac/lima/substrate.yaml` - `scripts/mac/lima/substrate-dev.yaml` +- `crates/world-agent/src/gateway_runtime.rs` ## Deliverables - a macOS ownership-separated validation matrix - a required-artifacts checklist per matrix scenario -- a release-blocker list for install, migration, status, supportability, and - uninstall regressions -- a final cross-doc review checklist for install/world/uninstall/mac setup docs +- a release-blocker list for install, migration, gateway/status, + supportability, and uninstall regressions +- a final cross-doc review checklist for install/world/gateway/uninstall/mac + setup docs - a phase-closeout evidence bundle definition ## Acceptance Criteria @@ -116,12 +127,13 @@ Likely supporting surfaces: - interrupted install or interrupted migration recovery - reboot/relogin persistence - healthy doctor/status/support outputs + - healthy gateway sync/status/restart outputs - representative broken-state doctor/status/support outputs - uninstall after clean install - uninstall after migration - Each matrix row names the exact required evidence artifacts. -- Release blockers are explicit enough that “security boundary claim not - supportable” is a shippability failure, not a soft concern. +- Release blockers are explicit enough that "security boundary claim not + supportable" is a shippability failure, not a soft concern. - The validated docs no longer present direct guest commands as the normal path for install or support. @@ -134,12 +146,17 @@ Likely supporting surfaces: - `substrate host doctor --json` - `substrate world doctor --json` - `substrate health --json` - - lifecycle/status JSON if introduced in milestone 2.2 + - `substrate world gateway status --json` + - `substrate world gateway sync` + - `substrate world gateway restart` - `scripts/mac/smoke.sh` artifacts for the relevant runtime scenario - Require one evidence set from a host that started on the legacy same-user Lima flow and was migrated successfully. - Require one evidence set showing operator guidance for a deliberately broken state without relying on direct guest commands. +- Require one evidence set that proves lifecycle failures pointing at + `/run/substrate/substrate-gateway-runtime/...` remain diagnosable through the + supported operator boundary. - Run a final doc review against `docs/INSTALLATION.md`, `docs/UNINSTALL.md`, `docs/WORLD.md`, and `docs/cross-platform/mac_world_setup.md` to ensure every validated scenario is documented consistently. @@ -152,5 +169,6 @@ Likely supporting surfaces: too costly to rerun for each release candidate? - If migration succeeds functionally but leaves behind legacy same-user state, is that a GA blocker or a documented cleanup defect? -- What is the exact threshold for “breakglass only” documentation so support can - still recover severely broken machines without normalizing direct guest entry? +- What is the exact threshold for "breakglass only" documentation so support + can still recover severely broken machines without normalizing direct guest + entry? diff --git a/macos-hardening/research/2026-04-28-macos-lima-parity-lockdown.md b/macos-hardening/research/2026-04-28-macos-lima-parity-lockdown.md index 0537f6eef..61832549f 100644 --- a/macos-hardening/research/2026-04-28-macos-lima-parity-lockdown.md +++ b/macos-hardening/research/2026-04-28-macos-lima-parity-lockdown.md @@ -1,5 +1,5 @@ --- -date: 2026-04-28T00:00:00-04:00 +date: 2026-05-19T00:00:00-04:00 type: external-research topic: "macOS Lima parity and lockdown" focus: general @@ -14,86 +14,213 @@ status: complete ## Summary -Substrate’s current macOS Lima backend is not “reachable only through Substrate.” The effective boundary is the owning macOS user plus any local process that can reach the forwarded endpoint. The strongest viable path to Linux-like ownership is not a wrapper around `limactl`; it is host-side ownership separation plus transport and workflow hardening. +Substrate’s current macOS Lima backend is still not "reachable only through +Substrate." The effective boundary remains the owning macOS user plus any local +process that can reach the forwarded endpoint. At the same time, HEAD now has +real same-user operator surfaces for gateway lifecycle, integrated auth handoff, +doctoring, and runtime metadata. Those landed surfaces are useful evidence and +good prerequisites, but they do not yet create true host-side ownership +separation. ## Repo-backed findings ### 1. Host ownership is same-user, not Substrate-owned -- `world-mac-lima` resolves the VM name from `SUBSTRATE_LIMA_VM_NAME` or `LIMA_VM_NAME`, and resolves the forwarded agent socket under the current user’s home as `~/.substrate/sock/agent.sock`: [crates/world-mac-lima/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/lib.rs:42) -- SSH forwarding resolves `LIMA_HOME` or defaults to `$HOME/.lima`, then reads `/ssh.config` from there: [crates/world-mac-lima/src/forwarding.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/forwarding.rs:376) -- The installer, uninstaller, and dev-uninstaller all operate Lima as the current user. None introduces a dedicated macOS host `substrate` account: [scripts/substrate/install-substrate.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/substrate/install-substrate.sh:1854), [scripts/substrate/uninstall-substrate.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/substrate/uninstall-substrate.sh:453), [scripts/substrate/dev-uninstall-substrate.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/substrate/dev-uninstall-substrate.sh:628) - -### 2. Direct guest entry is a supported normal path - -- Setup docs tell operators to build in the guest, install in the guest, restart guest `systemd`, probe the guest socket, and inspect guest logs with direct guest commands: [docs/cross-platform/mac_world_setup.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/cross-platform/mac_world_setup.md:82), [docs/cross-platform/mac_world_setup.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/cross-platform/mac_world_setup.md:105), [docs/cross-platform/mac_world_setup.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/cross-platform/mac_world_setup.md:119), [docs/cross-platform/mac_world_setup.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/cross-platform/mac_world_setup.md:146) -- Core docs continue that pattern for capability inspection, logs, and cleanup: [docs/WORLD.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/WORLD.md:139), [docs/WORLD.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/WORLD.md:148), [docs/WORLD.md](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/docs/WORLD.md:370) -- Doctor and smoke scripts validate by shelling into the guest directly: [scripts/mac/lima-doctor.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/lima-doctor.sh:62), [scripts/mac/smoke.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/smoke.sh:155) - -### 3. Current transport surfaces are broader than “Substrate only” - -- Host-visible surfaces: - - UDS: `~/.substrate/sock/agent.sock` via SSH local forwarding: [crates/world-mac-lima/src/forwarding.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/forwarding.rs:186) - - Loopback TCP: `127.0.0.1:17788` via `vsock-proxy`: [crates/world-mac-lima/src/forwarding.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/forwarding.rs:149) -- Guest-visible surfaces: - - UDS: `/run/substrate.sock`: [crates/world-agent/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/lib.rs:47) - - Optional TCP: `127.0.0.1:$SUBSTRATE_AGENT_TCP_PORT`: [crates/world-agent/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/lib.rs:151) -- The agent API itself is unauthenticated at the listener layer; access control is effectively “who can reach the socket/port”: [crates/world-agent/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-agent/src/lib.rs:252) - -### 4. The guest identity and mount model are far from locked down - -- The warm flow creates a guest `substrate` group and adds the login user to it: [scripts/mac/lima-warm.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/lima-warm.sh:263) -- The agent service runs as root with `Group=substrate`, not as a dedicated non-root service user: [scripts/mac/lima-warm.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/lima-warm.sh:665) -- Both base Lima profiles mount the entire host `$HOME` read-only and the active project read-write at `/src`: [scripts/mac/lima/substrate.yaml](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/lima/substrate.yaml:16), [scripts/mac/lima/substrate-dev.yaml](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/lima/substrate-dev.yaml:19) -- The warm rewrite also enables `SUBSTRATE_AGENT_TCP_PORT=61337` and broad ambient capabilities: [scripts/mac/lima-warm.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/lima-warm.sh:671), [scripts/mac/lima-warm.sh](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/scripts/mac/lima-warm.sh:687) - -### 5. There are parity gaps even before lockdown - -- `MacLimaBackend` still injects a synthetic permissive policy snapshot and leaves `apply_policy` as a no-op: [crates/world-mac-lima/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/lib.rs:274), [crates/world-mac-lima/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/lib.rs:463) -- PTY routing still effectively prefers the Unix-socket path and does not present one clean canonical transport contract across VSock, SSH UDS, and fallback TCP: [crates/shell/src/execution/routing/dispatch/exec.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/shell/src/execution/routing/dispatch/exec.rs:699) -- Transport constants are inconsistent (`17788` in forwarding code vs `7788` in some transport code): [crates/world-mac-lima/src/forwarding.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/forwarding.rs:149), [crates/world-mac-lima/src/transport.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/transport.rs:53), [crates/world-mac-lima/src/lib.rs](/Users/spensermcconnell/__Active_Code/atomize-hq/substrate/crates/world-mac-lima/src/lib.rs:205) +- `world-mac-lima` still resolves the VM name from `SUBSTRATE_LIMA_VM_NAME` or + `LIMA_VM_NAME`, and resolves the forwarded agent socket under the current + user’s home as `~/.substrate/sock/agent.sock`: + `crates/world-mac-lima/src/lib.rs` +- SSH forwarding still resolves `LIMA_HOME` or defaults to `$HOME/.lima`, then + reads `/ssh.config` from there: + `crates/world-mac-lima/src/forwarding.rs` +- The installer, uninstaller, and dev-uninstaller still operate Lima as the + current user. None introduces a dedicated macOS host `substrate` account: + `scripts/substrate/install-substrate.sh`, + `scripts/substrate/uninstall-substrate.sh`, and + `scripts/substrate/dev-uninstall-substrate.sh` +- `scripts/substrate/world-enable.sh` is also a same-user lifecycle helper + today. It reuses installer provisioning helpers rather than calling through a + daemon-owned macOS control plane. + +### 2. Direct guest entry is still a supported normal path + +- Setup docs still tell operators to build in the guest, install in the guest, + restart guest `systemd`, probe the guest socket, and inspect guest logs with + direct guest commands: `docs/cross-platform/mac_world_setup.md` +- `docs/WORLD.md` still records direct guest inspection and forwarding behavior + as descriptive architecture evidence. That is useful context, but it should + not be confused with the durable operator/status contract owner. +- `scripts/mac/lima-doctor.sh` and `scripts/mac/smoke.sh` still rely on direct + guest interactions as part of the current same-user verification model. + +### 3. Current transport surfaces are broader than "Substrate only" + +- Host-visible surfaces still include: + - UDS: `~/.substrate/sock/agent.sock` via SSH local forwarding + - loopback TCP via `vsock-proxy` +- Guest-visible surfaces still include: + - UDS: `/run/substrate.sock` + - optional TCP via `SUBSTRATE_AGENT_TCP_PORT` +- The agent API itself is still protected primarily by transport reachability: + whoever can reach the socket or port can talk to the listener: + `crates/world-agent/src/lib.rs` +- This same forwarded transport path now also carries the current gateway + lifecycle/status traffic on macOS. `substrate world gateway sync`, + `substrate world gateway status --json`, and `substrate world gateway restart` + already exist, but on macOS they still ride the same Lima-backed forwarded + path that ownership separation must eventually privatize: + `crates/shell/src/builtins/world_gateway.rs` + +### 4. Gateway lifecycle and runtime metadata are now real repo surfaces + +- `substrate-gateway` is now part of the installed and verified macOS story. + The macOS installer and dev-install paths verify or stage + `/usr/local/bin/substrate-gateway` in the Lima guest: + `scripts/substrate/install-substrate.sh` and + `scripts/substrate/dev-install-substrate.sh` +- `scripts/mac/smoke.sh` already exercises gateway lifecycle proof on macOS by + running: + - `substrate world gateway sync` + - `substrate world gateway status --json` + - `substrate world gateway restart` +- Gateway lifecycle/status meaning is no longer hypothetical. The durable + operator contract already lives in: + - `docs/contracts/substrate-gateway-operator-contract.md` + - `docs/contracts/substrate-gateway-status-schema.md` + - `docs/contracts/substrate-gateway-runtime-parity.md` +- Managed gateway runtime artifacts under + `/run/substrate/substrate-gateway-runtime/` already matter for diagnostics and + supportability: + `crates/world-agent/src/gateway_runtime.rs`, + `docs/WORLD.md`, and `docs/INSTALLATION.md` + +### 5. Integrated auth handoff is now landed, but not sufficient for ownership separation + +- The current integrated handoff no longer lives only in "future auth" design + language. The durable policy contract now states that the landed carrier is a + Substrate-owned auth bundle delivered over an inherited FD via + `SUBSTRATE_LLM_AUTH_BUNDLE_FD`: + `docs/contracts/substrate-gateway-policy-evaluation.md` +- `world-agent` now owns gateway runtime launch and hands the auth bundle to the + in-world gateway through the managed runtime path: + `crates/world-agent/src/gateway_runtime.rs` +- Gateway status now exposes non-secret lifecycle metadata through the existing + machine-readable surface, especially `status` and `client_wiring.*`: + `docs/contracts/substrate-gateway-status-schema.md` +- This is materially better than a purely speculative future design, but it + still does not solve the core macOS ownership problem. The same developer user + still owns the Lima control plane, forwarded transport setup, and direct guest + maintenance tools on the host side. + +### 6. The guest identity and mount model are still far from locked down + +- The warm flow still creates a guest `substrate` group and adds the login user + to it: `scripts/mac/lima-warm.sh` +- The agent service still runs as root with `Group=substrate`, not as a + dedicated non-root service user: `scripts/mac/lima-warm.sh` +- Both base Lima profiles still mount the entire host `$HOME` read-only and the + active project read-write at `/src`: + `scripts/mac/lima/substrate.yaml` and + `scripts/mac/lima/substrate-dev.yaml` +- The warm rewrite still enables `SUBSTRATE_AGENT_TCP_PORT=61337` and broad + ambient capabilities: `scripts/mac/lima-warm.sh` + +### 7. There are parity gaps even before lockdown + +- `MacLimaBackend` still injects a synthetic permissive policy snapshot and + leaves `apply_policy` as a no-op: + `crates/world-mac-lima/src/lib.rs` +- PTY routing still effectively prefers the Unix-socket path and does not + present one clean canonical transport contract across VSock, SSH UDS, and + fallback TCP: + `crates/shell/src/execution/routing/dispatch/exec.rs` +- Transport constants are still inconsistent in the current code: + `crates/world-mac-lima/src/forwarding.rs`, + `crates/world-mac-lima/src/transport.rs`, and + `crates/world-mac-lima/src/lib.rs` ## External findings ### Lima constraints -- `limactl shell` is SSH-backed, and direct SSH via per-instance config is an explicitly supported path: [Lima SSH docs](https://lima-vm.io/docs/usage/ssh/) -- `LIMA_HOME` defaults to `~/.lima`, and Lima control-plane material, SSH config, identity, and instance state live there: [Lima internals](https://lima-vm.io/docs/dev/internals/), [Lima env vars](https://lima-vm.io/docs/config/environment-variables/) -- Lima documents that default user-mode guest IP is not host-reachable by design, but host access remains available through SSH, forwarding, and control-plane artifacts: [Lima default user-mode network](https://lima-vm.io/docs/config/network/user/) -- `--mount-none` and `--plain` can remove default file-sharing surfaces, but `plain` is version-sensitive and should not be treated as a stable security guarantee by name alone: [Lima create reference](https://lima-vm.io/docs/reference/limactl_create/), [Lima breaking changes](https://lima-vm.io/docs/releases/breaking/), [Lima port forwarding](https://lima-vm.io/docs/config/port/) -- Current Lima docs remain partially inconsistent on `ssh.loadDotSSHPubKeys`; current release docs and templates indicate newer defaults than some internals text. Any hardening plan should pin the exact Lima version in scope. +- `limactl shell` is SSH-backed, and direct SSH via per-instance config is an + explicitly supported path: [Lima SSH docs](https://lima-vm.io/docs/usage/ssh/) +- `LIMA_HOME` defaults to `~/.lima`, and Lima control-plane material, SSH + config, identity, and instance state live there: + [Lima internals](https://lima-vm.io/docs/dev/internals/), + [Lima env vars](https://lima-vm.io/docs/config/environment-variables/) +- Lima documents that default user-mode guest IP is not host-reachable by + design, but host access still remains available through SSH, forwarding, and + control-plane artifacts: + [Lima default user-mode network](https://lima-vm.io/docs/config/network/user/) +- `--mount-none` and `--plain` can remove default file-sharing surfaces, but + `plain` is version-sensitive and should not be treated as a stable security + guarantee by name alone: + [Lima create reference](https://lima-vm.io/docs/reference/limactl_create/), + [Lima breaking changes](https://lima-vm.io/docs/releases/breaking/), + [Lima port forwarding](https://lima-vm.io/docs/config/port/) +- Current Lima docs remain partially inconsistent on + `ssh.loadDotSSHPubKeys`; any hardening plan should pin the exact Lima version + in scope. ### Apple/macOS constraints -- `launchd` supports daemon ownership separation and on-demand socket activation using daemon plist ownership fields and socket registration: [Apple launchd jobs](https://developer.apple.com/library/archive/documentation/MacOSX/Conceptual/BPSystemStartup/Chapters/CreatingLaunchdJobs.html), [Apple launch_activate_socket](https://developer.apple.com/documentation/xpc/launch_activate_socket) -- `SMAppService` is the modern Apple-supported registration path for LaunchDaemons on macOS 13+: [Apple SMAppService](https://developer.apple.com/documentation/servicemanagement/smappservice) -- Apple’s Virtualization.framework shared-directory model does not create a separate host principal; shared-directory access is still bound to host-side user permissions. This reinforces that same-user Lima ownership is not a meaningful same-user security boundary: [Apple VZVirtioFileSystemDevice](https://developer.apple.com/documentation/virtualization/vzvirtiofilesystemdevice) +- `launchd` supports daemon ownership separation and on-demand socket + activation: + [Apple launchd jobs](https://developer.apple.com/library/archive/documentation/MacOSX/Conceptual/BPSystemStartup/Chapters/CreatingLaunchdJobs.html), + [Apple launch_activate_socket](https://developer.apple.com/documentation/xpc/launch_activate_socket) +- `SMAppService` is the modern Apple-supported registration path for + LaunchDaemons on macOS 13+: + [Apple SMAppService](https://developer.apple.com/documentation/servicemanagement/smappservice) +- Apple’s Virtualization.framework shared-directory model still does not create + a separate host principal; shared-directory access remains bound to host-side + user permissions: + [Apple VZVirtioFileSystemDevice](https://developer.apple.com/documentation/virtualization/vzvirtiofilesystemdevice) ## Conclusion -If the target is merely “reduce accidental bypass,” Substrate can improve the current same-user Lima model by removing direct docs/scripts, tightening mounts, removing extra listeners, and authenticating the agent path. +If the target is merely "reduce accidental bypass," Substrate can continue to +improve the current same-user Lima model by removing direct docs/scripts, +tightening mounts, removing extra listeners, and normalizing gateway/runtime +parity. -If the target is “only Substrate can access the VM” against the same logged-in macOS user, wrapper-only hardening is insufficient. The repo findings and external Lima/macOS constraints both point to the same answer: the control-plane owner must not be the developer’s macOS account. +If the target is "only Substrate can access the VM" against the same logged-in +macOS user, wrapper-only hardening is still insufficient. The repo findings and +external Lima/macOS constraints continue to point to the same answer: the +control-plane owner must not be the developer’s macOS account. ## Recommended target architecture ### Required for true ownership separation -1. Run Lima under a dedicated macOS service account or daemon-owned context not controlled by the developer user. +1. Run Lima under a dedicated macOS service account or daemon-owned context not + controlled by the developer user. 2. Move `LIMA_HOME` into that owner’s private directory. -3. Remove direct user access to Lima SSH config, identities, hostagent state, and instance disks. -4. Replace direct host-local transport reachability with a single Substrate-owned broker endpoint. -5. Add request authentication/attestation on the broker-to-agent path; do not rely on socket reachability alone. -6. Treat all direct `limactl shell`, direct SSH, guest `systemctl`, and guest `curl` flows as breakglass only. - -### Required even in a same-user “hardened but not absolute” mode +3. Remove direct user access to Lima SSH config, identities, hostagent state, + and instance disks. +4. Replace direct host-local guest reachability with a single Substrate-owned + broker endpoint. +5. Extend ownership separation to the already-landed gateway lifecycle path so + `substrate world gateway sync|status|restart` no longer depend on same-user + forwarding. +6. Preserve the landed auth-bundle and status-metadata surfaces, but make them + flow through a daemon-owned control plane rather than a developer-owned Lima + path. +7. Treat all direct `limactl shell`, direct SSH, guest `systemctl`, and guest + `curl` flows as breakglass only. + +### Required even in a same-user "hardened but not absolute" mode 1. Remove `SUBSTRATE_AGENT_TCP_PORT=61337` by default. -2. Stop mounting all of `$HOME`; use `--mount-none` or an explicit narrow allowlist. Make `/src` the only normal writable ingress. -3. Normalize unit definitions into one source of truth and stop bypassing socket-activation ACLs. +2. Stop mounting all of `$HOME`; use `--mount-none` or an explicit narrow + allowlist. Make `/src` the only normal writable ingress. +3. Normalize unit definitions into one source of truth and stop bypassing + socket-activation ACLs. 4. Add a dedicated guest service user or at least trim capabilities materially. -5. Unify the macOS transport contract and eliminate the current `17788`/`7788` inconsistency and the permissive synthetic policy snapshot. -6. Replace operator-facing direct guest workflows with Substrate-owned commands for doctor, logs, repair, cleanup, and smoke validation. +5. Unify the macOS transport contract and eliminate the current transport drift + and permissive synthetic policy snapshot. +6. Replace operator-facing direct guest workflows with Substrate-owned commands + for doctor, logs, repair, cleanup, and gateway lifecycle validation. ## Practical sequencing @@ -112,12 +239,15 @@ If the answer is `B`, account separation is mandatory. - Fix unit/socket-activation drift. - Unify transport constants and readiness checks. - Stop normal-path direct guest docs and scripts. +- Keep the current gateway lifecycle/status contract stable while moving it away + from same-user forwarding assumptions. ### Phase 2: Reach functional parity - Apply real backend policy snapshots on macOS. - Bring PTY/non-PTY transport behavior into one canonical contract. -- Add authenticated brokered access instead of reachability-only listener access. +- Ensure doctor, health, and gateway lifecycle evidence all run through the + same supported path. ### Phase 3: Add true ownership separation @@ -128,9 +258,16 @@ If the answer is `B`, account separation is mandatory. ## Risks and open questions -- The exact Lima version in supported installs matters for `plain`, forwarding, SSH defaults, and any guestagent behavior. -- A dedicated host owner will require broad repo changes across runtime, scripts, tests, docs, and operator flows. -- If the product depends on current broad host file visibility inside the guest, mount minimization may surface feature gaps that need explicit ingress/sync design. +- The exact Lima version in supported installs still matters for `plain`, + forwarding, SSH defaults, and any guestagent behavior. +- A dedicated host owner will still require broad repo changes across runtime, + scripts, tests, docs, and operator flows. +- The repo already has same-user doctor/gateway surfaces. The risk is not + absence of operator UX; it is that those surfaces still sit on top of the + wrong host ownership model. +- If the product depends on current broad host file visibility inside the + guest, mount minimization may still surface feature gaps that need explicit + ingress and sync design. ## Sources diff --git a/scripts/linux/world-provision.sh b/scripts/linux/world-provision.sh index 1e6423701..7aa18d6e1 100755 --- a/scripts/linux/world-provision.sh +++ b/scripts/linux/world-provision.sh @@ -28,6 +28,10 @@ GATEWAY_PROOF_ELIGIBLE=0 GATEWAY_PROOF_AUTH_MODE="" declare -a GATEWAY_PROOF_SKIP_REASONS=() +is_wsl_host() { + grep -qi microsoft /proc/version 2>/dev/null +} + show_cmd() { printf '[dry-run]' for arg in "$@"; do @@ -468,6 +472,14 @@ if [[ ${EUID} -eq 0 ]]; then exit 1 fi +if is_wsl_host; then + cat >&2 <<'MSG' +ERROR: WSL world provisioning is intentionally fail-closed in this slice because the WSL helper path is not aligned with the Linux/macOS placement contract. +Re-run this helper on a Linux host-native install, or use --no-world / CLI-only flows inside WSL. +MSG + exit 4 +fi + INVOKING_USER="$(detect_invoking_user)" SCRIPT_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) @@ -526,6 +538,7 @@ fi if [[ -z "${INVOKING_HOME}" ]]; then INVOKING_HOME="$(dirname "${SUBSTRATE_HOME_FOR_AGENT}")" fi +SUBSTRATE_HOME_RW_PATH="${SUBSTRATE_HOME_FOR_AGENT}" NETFILTER_ENV_LINE="" if [[ "${ENABLE_WORLD_NETFILTER}" -eq 1 ]]; then @@ -560,7 +573,7 @@ StandardError=journal NoNewPrivileges=yes ProtectSystem=strict ProtectHome=read-only -ReadWritePaths=/var/lib/substrate /run /run/substrate /sys/fs/cgroup /tmp +ReadWritePaths=${SUBSTRATE_HOME_RW_PATH} /var/lib/substrate /run /run/substrate /sys/fs/cgroup /tmp CapabilityBoundingSet=CAP_NET_ADMIN CAP_NET_BIND_SERVICE CAP_SYS_ADMIN CAP_SYS_CHROOT CAP_DAC_OVERRIDE CAP_CHOWN CAP_SYS_PTRACE AmbientCapabilities=CAP_NET_ADMIN CAP_NET_BIND_SERVICE CAP_SYS_ADMIN CAP_SYS_CHROOT CAP_DAC_OVERRIDE CAP_CHOWN CAP_SYS_PTRACE @@ -594,6 +607,7 @@ sudo_cmd install -Dm0755 "${GATEWAY_BIN_PATH}" /usr/local/bin/substrate-gateway echo "==> Ensuring runtime directories exist" sudo_cmd install -d -m0750 -o root -g "${SUBSTRATE_GROUP}" /run/substrate sudo_cmd install -d -m0750 /var/lib/substrate +sudo_cmd install -d -m0755 "${SUBSTRATE_HOME_RW_PATH}" echo "==> Writing systemd units to ${SERVICE_PATH} and ${SOCKET_PATH}" install_unit "${SERVICE_PATH}" "${SERVICE_UNIT_CONTENT}" diff --git a/scripts/mac/lima-warm.sh b/scripts/mac/lima-warm.sh index b7617f383..69e47ce7f 100755 --- a/scripts/mac/lima-warm.sh +++ b/scripts/mac/lima-warm.sh @@ -4,6 +4,7 @@ set -euo pipefail VM_NAME="${LIMA_VM_NAME:-substrate}" PROFILE="${LIMA_PROFILE_PATH:-scripts/mac/lima/substrate.yaml}" PROJECT_PATH="" +PROJECT_PATH_EXPLICIT=0 CHECK_ONLY=0 BUILD_PROFILE="${LIMA_BUILD_PROFILE:-release}" LAYOUT_SENTINEL="/etc/substrate-lima-layout" @@ -50,6 +51,7 @@ while [[ $# -gt 0 ]]; do *) if [[ -z "${PROJECT_PATH}" ]]; then PROJECT_PATH="$1" + PROJECT_PATH_EXPLICIT=1 else fatal "Unexpected argument: $1" fi @@ -63,6 +65,30 @@ if [[ -z "${PROJECT_PATH}" ]]; then fi PROJECT_PATH="$(cd "${PROJECT_PATH}" && pwd)" +resolve_default_worktree_path() { + if [[ "${PROJECT_PATH_EXPLICIT}" -eq 1 ]]; then + return + fi + if ! command -v git >/dev/null 2>&1; then + return + fi + + local git_dir common_dir primary_worktree + git_dir="$(git -C "${PROJECT_PATH}" rev-parse --git-dir 2>/dev/null || true)" + common_dir="$(git -C "${PROJECT_PATH}" rev-parse --git-common-dir 2>/dev/null || true)" + if [[ -z "${git_dir}" || -z "${common_dir}" || "${git_dir}" == "${common_dir}" ]]; then + return + fi + + primary_worktree="$(git -C "${PROJECT_PATH}" worktree list --porcelain 2>/dev/null | awk '/^worktree / { print substr($0, 10); exit }')" + if [[ -n "${primary_worktree}" && -d "${primary_worktree}" && "${primary_worktree}" != "${PROJECT_PATH}" ]]; then + log "Detected linked worktree; defaulting Lima mount path to primary checkout ${primary_worktree}" + PROJECT_PATH="${primary_worktree}" + fi +} + +resolve_default_worktree_path + require_cmd() { local name="$1" if ! command -v "${name}" >/dev/null 2>&1; then @@ -637,6 +663,7 @@ EOF } write_systemd_units() { + local guest_substrate_home="$1" local enable_netfilter="${SUBSTRATE_WORLD_NETFILTER_ENABLE:-0}" case "${enable_netfilter}" in 1|true|yes|TRUE|YES) @@ -646,7 +673,7 @@ write_systemd_units() { log "Writing guest systemd unit without WORLD_NETFILTER_ENABLE=1" ;; esac - limactl shell "${VM_NAME}" env SUBSTRATE_WORLD_NETFILTER_ENABLE="${enable_netfilter}" bash <<'EOF' + limactl shell "${VM_NAME}" env SUBSTRATE_WORLD_NETFILTER_ENABLE="${enable_netfilter}" SUBSTRATE_GUEST_HOME="${guest_substrate_home}" bash <<'EOF' set -euo pipefail netfilter_env="" @@ -670,6 +697,7 @@ RestartSec=5 Environment=RUST_LOG=info Environment=SUBSTRATE_AGENT_TCP_PORT=61337 Environment=SUBSTRATE_WORLD_SOCKET=/run/substrate.sock +Environment=SUBSTRATE_HOME=${SUBSTRATE_GUEST_HOME} ${netfilter_env} Group=substrate UMask=0027 @@ -683,7 +711,7 @@ StandardError=journal NoNewPrivileges=yes ProtectSystem=strict ProtectHome=read-only -ReadWritePaths=/var/lib/substrate /run /run/substrate /sys/fs/cgroup /tmp +ReadWritePaths=${SUBSTRATE_GUEST_HOME} /var/lib/substrate /run /run/substrate /sys/fs/cgroup /tmp CapabilityBoundingSet=CAP_NET_ADMIN CAP_NET_BIND_SERVICE CAP_SYS_ADMIN CAP_SYS_CHROOT CAP_DAC_OVERRIDE CAP_CHOWN CAP_SYS_PTRACE AmbientCapabilities=CAP_NET_ADMIN CAP_NET_BIND_SERVICE CAP_SYS_ADMIN CAP_SYS_CHROOT CAP_DAC_OVERRIDE CAP_CHOWN CAP_SYS_PTRACE @@ -712,8 +740,10 @@ EOF } enable_socket_activation() { - limactl shell "${VM_NAME}" bash <<'EOF' + local guest_substrate_home="$1" + limactl shell "${VM_NAME}" env SUBSTRATE_GUEST_HOME="${guest_substrate_home}" bash <<'EOF' set -euo pipefail +sudo install -d -m0755 "${SUBSTRATE_GUEST_HOME}" sudo install -d -m0750 /var/lib/substrate sudo install -d -m0750 -o root -g substrate /run/substrate sudo systemctl daemon-reload @@ -762,15 +792,22 @@ linger_guidance() { configure_guest() { ensure_repo_mount local vm_user + local vm_home + local guest_substrate_home vm_user="$(limactl shell "${VM_NAME}" id -un | tr -d '\r')" if [[ -z "${vm_user}" ]]; then fatal "Unable to determine Lima guest user." fi + vm_home="$(limactl shell "${VM_NAME}" getent passwd "${vm_user}" | cut -d: -f6 | tr -d '\r')" + if [[ -z "${vm_home}" ]]; then + vm_home="/home/${vm_user}" + fi + guest_substrate_home="${vm_home}/.substrate" ensure_substrate_group "${vm_user}" install_guest_binaries verify_guest_binaries - write_systemd_units - enable_socket_activation + write_systemd_units "${guest_substrate_home}" + enable_socket_activation "${guest_substrate_home}" socket_summary write_layout_sentinel linger_guidance "${vm_user}" @@ -786,3 +823,4 @@ ensure_vm_ready configure_guest log "Lima world backend '${VM_NAME}' is ready. Verify with: limactl shell ${VM_NAME} sudo systemctl status substrate-world-agent.socket" +log "Optional orchestration parity proof: scripts/mac/orchestration-smoke.sh" diff --git a/scripts/mac/orchestration-smoke.sh b/scripts/mac/orchestration-smoke.sh new file mode 100755 index 000000000..a7419e4f6 --- /dev/null +++ b/scripts/mac/orchestration-smoke.sh @@ -0,0 +1,67 @@ +#!/usr/bin/env bash +set -euo pipefail + +if [[ "${EUID}" -eq 0 ]]; then + echo "Do not run this orchestration smoke script as root." >&2 + exit 1 +fi + +SCRIPTS_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO_ROOT="$(cd "${SCRIPTS_ROOT}/../.." && pwd)" + +log() { + printf '[mac-orchestration-smoke] %s\n' "$*" +} + +require_cmd() { + local cmd="$1" + local hint="${2:-}" + if ! command -v "${cmd}" >/dev/null 2>&1; then + if [[ -n "${hint}" ]]; then + echo "ERROR: ${cmd} not found on PATH. ${hint}" >&2 + else + echo "ERROR: ${cmd} not found on PATH." >&2 + fi + exit 1 + fi +} + +ensure_host_prereqs() { + if ! command -v limactl >/dev/null 2>&1; then + PATH="/opt/homebrew/opt/lima/bin:/opt/homebrew/bin:$PATH" + fi + + require_cmd limactl "Install Lima via Homebrew (brew install lima)." + require_cmd cargo "Install Rust via rustup." +} + +run_repo_cmd() { + log "$*" + ( + cd "${REPO_ROOT}" + "$@" + ) +} + +ensure_host_prereqs + +log "Warming the Lima-backed world backend" +"${SCRIPTS_ROOT}/lima-warm.sh" + +log "Running live Lima backend reachability smoke" +run_repo_cmd cargo run -p world-mac-lima --example mac_backend_smoke + +log "Running macOS shared-owner bootstrap routing proof" +run_repo_cmd cargo test -p shell shared_owner_macos_allows_lima_backed_bootstrap -- --nocapture + +log "Running shared-owner ready-proof acceptance and mismatch rejection checks" +run_repo_cmd cargo test -p shell --test persistent_session_client_v1 persistent_session_client_v1_accepts_shared_world_attach_create_ready_proof -- --nocapture +run_repo_cmd cargo test -p shell --test persistent_session_client_v1 persistent_session_client_v1_accepts_replacement_ready_when_generation_advances -- --nocapture +run_repo_cmd cargo test -p shell --test persistent_session_client_v1 persistent_session_client_v1_rejects_invalid_shared_world_ready_proof -- --nocapture + +log "Running retained world-member launch, reuse, and cancel checks" +run_repo_cmd cargo test -p shell --test repl_world_first_routing_v1 c3_first_targeted_world_turn_uses_initial_prompt_in_member_dispatch -- --nocapture +run_repo_cmd cargo test -p shell --test repl_world_first_routing_v1 c3_first_world_backed_command_lazily_launches_member_runtime -- --nocapture +run_repo_cmd cargo test -p shell --test repl_world_first_routing_v1 c3_targeted_world_turn_uses_typed_submit_route_without_relaunching_member -- --nocapture + +log "macOS/Lima orchestration smoke passed" diff --git a/scripts/mac/smoke.sh b/scripts/mac/smoke.sh index 6a316f571..15789c9ce 100755 --- a/scripts/mac/smoke.sh +++ b/scripts/mac/smoke.sh @@ -15,11 +15,13 @@ LOG_DIR="" usage() { cat <<'USAGE' -Usage: scripts/mac/smoke.sh [--netfilter-conformance | --bedpm-installer-conformance] [--log-dir DIR] +Usage: scripts/mac/smoke.sh [--orchestration-conformance | --netfilter-conformance | --bedpm-installer-conformance] [--log-dir DIR] Options: --world-disabled-diagnostics Run the world-disabled-diagnostics conformance smoke instead of the generic smoke + --orchestration-conformance + Run the macOS/Lima orchestration conformance smoke --netfilter-conformance Run the posture-aware Lima netfilter smoke instead of the generic smoke --bedpm-installer-conformance Run the BEDPM Linux installer smoke through the Lima-backed guest path @@ -38,6 +40,10 @@ while [[ $# -gt 0 ]]; do MODE="world-disabled-diagnostics" shift ;; + --orchestration-conformance) + MODE="orchestration-conformance" + shift + ;; --bedpm-installer-conformance) MODE="bedpm-installer-conformance" shift @@ -644,6 +650,11 @@ run_bedpm_installer_conformance() { "${SUBSTRATE_BIN}" -c "${smoke_cmd}" } +run_orchestration_conformance() { + log "Running macOS/Lima orchestration conformance smoke" + "${SCRIPTS_ROOT}/orchestration-smoke.sh" +} + run_netfilter_conformance() { local log_dir="$1" local fixture_home="${log_dir}/home" @@ -676,6 +687,9 @@ ensure_substrate_binary if [[ "${MODE}" == "world-disabled-diagnostics" ]]; then run_world_disabled_diagnostics +elif [[ "${MODE}" == "orchestration-conformance" ]]; then + ensure_host_prereqs + run_orchestration_conformance elif [[ "${MODE}" == "netfilter-conformance" ]]; then ensure_host_prereqs if [[ -z "${LOG_DIR}" ]]; then diff --git a/scripts/substrate/dev-install-substrate.sh b/scripts/substrate/dev-install-substrate.sh index 52e7226aa..abf240270 100755 --- a/scripts/substrate/dev-install-substrate.sh +++ b/scripts/substrate/dev-install-substrate.sh @@ -564,6 +564,152 @@ detect_platform_metadata() { return 1 } +resolve_package_for_runtime_library() { + local library="$1" + + case "${PKG_MANAGER}" in + apt-get) + case "${library}" in + libseccomp) echo "libseccomp2" ;; + *) echo "" ;; + esac + ;; + dnf|yum) + case "${library}" in + libseccomp) echo "libseccomp" ;; + *) echo "" ;; + esac + ;; + pacman) + case "${library}" in + libseccomp) echo "libseccomp" ;; + *) echo "" ;; + esac + ;; + zypper) + case "${library}" in + libseccomp) echo "libseccomp2" ;; + *) echo "" ;; + esac + ;; + *) + echo "" + ;; + esac +} + +seccomp_runtime_available() { + if command -v ldconfig >/dev/null 2>&1; then + if ldconfig -p 2>/dev/null | grep -Eq 'libseccomp\.so(\.2)?([[:space:]]|$)'; then + return 0 + fi + fi + + if compgen -G '/lib*/libseccomp.so*' >/dev/null; then + return 0 + fi + if compgen -G '/usr/lib*/libseccomp.so*' >/dev/null; then + return 0 + fi + + return 1 +} + +install_packages() { + local packages=("$@") + if [[ ${#packages[@]} -eq 0 ]]; then + return + fi + + log "Installing packages: ${packages[*]}" + case "${PKG_MANAGER}" in + apt-get) + run_privileged apt-get update + run_privileged apt-get install -y "${packages[@]}" + ;; + dnf) + run_privileged dnf install -y "${packages[@]}" + ;; + yum) + run_privileged yum install -y "${packages[@]}" + ;; + pacman) + run_privileged pacman -Sy --noconfirm --needed "${packages[@]}" + ;; + zypper) + run_privileged zypper --non-interactive install "${packages[@]}" + ;; + *) + fatal "Unsupported package manager '${PKG_MANAGER}'. Install required runtime libraries manually and re-run." + ;; + esac +} + +ensure_linux_runtime_libraries() { + local libraries=("$@") + local missing=() + local library="" + + if [[ "${IS_LINUX}" -ne 1 || "${WORLD_ENABLED}" -ne 1 ]]; then + return + fi + + for library in "${libraries[@]}"; do + case "${library}" in + libseccomp) + if ! seccomp_runtime_available; then + missing+=("${library}") + fi + ;; + *) + warn "No runtime library probe implemented for '${library}'; install it manually if required." + ;; + esac + done + + if [[ ${#missing[@]} -eq 0 ]]; then + return + fi + + if ! detect_platform_metadata; then + fatal "Unable to detect a supported package manager for runtime library installation. Install ${missing[*]} manually and re-run." + fi + + declare -A pkg_set=() + local pkg_list pkg + for library in "${missing[@]}"; do + pkg_list="$(resolve_package_for_runtime_library "${library}")" + if [[ -z "${pkg_list}" ]]; then + fatal "No package mapping for runtime library '${library}' under ${PKG_MANAGER}. Install it manually and re-run." + fi + for pkg in ${pkg_list}; do + pkg_set["${pkg}"]=1 + done + done + + local packages=() + for pkg in "${!pkg_set[@]}"; do + packages+=("${pkg}") + done + + install_packages "${packages[@]}" + + local remaining=() + for library in "${missing[@]}"; do + case "${library}" in + libseccomp) + if ! seccomp_runtime_available; then + remaining+=("${library}") + fi + ;; + esac + done + + if [[ ${#remaining[@]} -gt 0 ]]; then + fatal "Unable to install required runtime libraries: ${remaining[*]}. Install them manually and re-run." + fi +} + write_host_state_metadata() { if [[ "${IS_LINUX}" -ne 1 ]]; then return @@ -1096,16 +1242,16 @@ cache_linux_binary_from_lima() { } verify_prefix_linux_bundle() { - local missing=0 + local missing_status=0 local binary path for binary in substrate world-agent substrate-gateway; do path="${BIN_DIR}/linux/${binary}" if ! is_linux_elf "${path}"; then warn "Expected cached Linux ${binary} at ${path}, but it is missing or not a Linux ELF." - missing=1 + missing_status=1 fi done - return "${missing}" + return "${missing_status}" } stage_dev_world_runtime_bundle() { @@ -1254,6 +1400,7 @@ VERSION_LABEL="dev" ENABLE_WORLD_NETFILTER=0 IS_LINUX=0 IS_MAC=0 +IS_WSL=0 HOST_STATE_PATH="" HOST_STATE_GROUP_EXISTED="" HOST_STATE_GROUP_CREATED=0 @@ -1267,6 +1414,9 @@ PKG_MANAGER="" PKG_MANAGER_SOURCE="" if [[ "$(uname -s)" == "Linux" ]]; then IS_LINUX=1 + if grep -qi microsoft /proc/version 2>/dev/null; then + IS_WSL=1 + fi fi if [[ "$(uname -s)" == "Darwin" ]]; then IS_MAC=1 @@ -1335,6 +1485,11 @@ while [[ $# -gt 0 ]]; do esac done +if [[ "${IS_WSL}" -eq 1 && "${WORLD_ENABLED}" -eq 1 ]]; then + printf '[%s][ERROR] %s\n' "${SCRIPT_NAME}" "WSL world provisioning is intentionally fail-closed in this slice because the WSL helper path is not aligned with the Linux/macOS placement contract. Re-run with --no-world for a CLI-only dev install inside WSL." >&2 + exit 4 +fi + HOST_STATE_PATH="${PREFIX%/}/install_state.json" case "${PROFILE}" in @@ -1464,6 +1619,7 @@ fi stage_dev_world_runtime_bundle "${PREFIX}" "${REPO_ROOT}" "${TARGET_DIR}" if [[ "${WORLD_ENABLED}" -eq 1 && "${IS_LINUX}" -eq 1 ]]; then + ensure_linux_runtime_libraries libseccomp if [[ ${EUID} -ne 0 ]] && command -v sudo >/dev/null 2>&1; then log "Caching sudo credentials for world provisioning (you may be prompted)..." if ! sudo -v; then @@ -1473,7 +1629,7 @@ if [[ "${WORLD_ENABLED}" -eq 1 && "${IS_LINUX}" -eq 1 ]]; then write_env_sh_script "${WORLD_ENABLED}" write_manager_env_script "${WORLD_ENABLED}" warn "Unable to cache sudo credentials; world-agent service not provisioned." - warn "World has been disabled in ${INSTALL_CONFIG_PATH} to avoid confusing runtime failures. Re-run provisioning, then run `substrate world enable --home \"${PREFIX}\"` to flip it back on." + warn "World has been disabled in ${INSTALL_CONFIG_PATH} to avoid confusing runtime failures. Re-run provisioning, then run 'substrate world enable --home \"${PREFIX}\"' to flip it back on." fi fi if [[ "${WORLD_ENABLED}" -eq 0 ]]; then @@ -1494,7 +1650,7 @@ if [[ "${WORLD_ENABLED}" -eq 1 && "${IS_LINUX}" -eq 1 ]]; then write_env_sh_script "${WORLD_ENABLED}" write_manager_env_script "${WORLD_ENABLED}" warn "world-provision script reported an error; rerun ${PROVISION_SCRIPT} manually to enable the world-agent service." - warn "World has been disabled in ${INSTALL_CONFIG_PATH} to avoid confusing runtime failures. Re-run provisioning, then run `substrate world enable --home \"${PREFIX}\"` to flip it back on." + warn "World has been disabled in ${INSTALL_CONFIG_PATH} to avoid confusing runtime failures. Re-run provisioning, then run 'substrate world enable --home \"${PREFIX}\"' to flip it back on." fi else WORLD_PROVISION_FAILED=1 diff --git a/scripts/substrate/install-substrate.sh b/scripts/substrate/install-substrate.sh index ed046aebc..f9bf9fe10 100755 --- a/scripts/substrate/install-substrate.sh +++ b/scripts/substrate/install-substrate.sh @@ -65,6 +65,13 @@ fatal() { exit 1 } +fatal_with_code() { + local code="$1" + shift + printf '[%s][ERROR] %s\n' "${INSTALLER_NAME}" "$*" >&2 + exit "${code}" +} + print_usage() { cat <<'EOF' Substrate Installer @@ -1002,6 +1009,57 @@ resolve_package_for_command() { esac } +resolve_package_for_runtime_library() { + local library="$1" + + case "${PKG_MANAGER}" in + apt-get) + case "${library}" in + libseccomp) echo "libseccomp2" ;; + *) echo "" ;; + esac + ;; + dnf|yum) + case "${library}" in + libseccomp) echo "libseccomp" ;; + *) echo "" ;; + esac + ;; + pacman) + case "${library}" in + libseccomp) echo "libseccomp" ;; + *) echo "" ;; + esac + ;; + zypper) + case "${library}" in + libseccomp) echo "libseccomp2" ;; + *) echo "" ;; + esac + ;; + *) + echo "" + ;; + esac +} + +seccomp_runtime_available() { + if command -v ldconfig >/dev/null 2>&1; then + if ldconfig -p 2>/dev/null | grep -Eq 'libseccomp\.so(\.2)?([[:space:]]|$)'; then + return 0 + fi + fi + + if compgen -G '/lib*/libseccomp.so*' >/dev/null; then + return 0 + fi + if compgen -G '/usr/lib*/libseccomp.so*' >/dev/null; then + return 0 + fi + + return 1 +} + install_packages() { local packages=() packages=("$@") @@ -1105,6 +1163,78 @@ ensure_linux_packages_for_commands() { fi } +ensure_linux_packages_for_runtime_libraries() { + local libraries=("$@") + local missing_libraries=() + local library="" + + for library in "${libraries[@]}"; do + case "${library}" in + libseccomp) + if ! seccomp_runtime_available; then + missing_libraries+=("${library}") + fi + ;; + *) + warn "No runtime library probe implemented for '${library}'; please install it manually if required." + ;; + esac + done + + if [[ ${#missing_libraries[@]} -eq 0 ]]; then + return + fi + + if ! detect_package_manager; then + fail_no_supported_pkg_manager "${missing_libraries[@]}" + fi + + initialize_sudo + maybe_emit_package_manager_decision_line + + declare -A pkg_set=() + local pkg_list pkg + for library in "${missing_libraries[@]}"; do + pkg_list="$(resolve_package_for_runtime_library "${library}")" + if [[ -z "${pkg_list}" ]]; then + warn "No package mapping for runtime library '${library}' under ${PKG_MANAGER}; please install it manually." + continue + fi + for pkg in ${pkg_list}; do + pkg_set["${pkg}"]=1 + done + done + + if [[ ${#pkg_set[@]} -eq 0 ]]; then + return + fi + + local packages=() + for pkg in "${!pkg_set[@]}"; do + packages+=("${pkg}") + done + + install_packages "${packages[@]}" + + if [[ "${DRY_RUN}" -eq 1 ]]; then + return + fi + + local remaining=() + for library in "${missing_libraries[@]}"; do + case "${library}" in + libseccomp) + if ! seccomp_runtime_available; then + remaining+=("${library}") + fi + ;; + esac + done + if [[ ${#remaining[@]} -gt 0 ]]; then + fatal "Unable to install required runtime libraries: ${remaining[*]}. Install them manually and re-run." + fi +} + compute_file_sha256() { local file="$1" if command -v sha256sum >/dev/null 2>&1; then @@ -1163,6 +1293,14 @@ detect_platform() { esac } +ensure_supported_linux_world_posture() { + if [[ "${PLATFORM}" != "linux" || "${IS_WSL}" -ne 1 || "${NO_WORLD}" -eq 1 ]]; then + return + fi + + fatal_with_code 4 "WSL world provisioning is intentionally fail-closed in this slice because the WSL helper path is not aligned with the Linux/macOS placement contract. Re-run with --no-world for a CLI-only install inside WSL, or use a supported Linux host-native or macOS Lima world backend." +} + parse_args() { while [[ $# -gt 0 ]]; do case "$1" in @@ -1522,6 +1660,7 @@ ensure_linux_prereqs() { if [[ "${NO_WORLD}" -eq 0 ]]; then ensure_linux_packages_for_commands systemctl fuse-overlayfs nft ip + ensure_linux_packages_for_runtime_libraries libseccomp require_cmd systemctl require_cmd fuse-overlayfs require_cmd nft @@ -1989,8 +2128,8 @@ NoNewPrivileges=yes ProtectSystem=strict ProtectHome=read-only ReadWritePaths=${home_path} /var/lib/substrate /run /run/substrate /sys/fs/cgroup /tmp -CapabilityBoundingSet=CAP_NET_ADMIN CAP_NET_BIND_SERVICE CAP_SYS_ADMIN CAP_SYS_CHROOT CAP_DAC_OVERRIDE CAP_SYS_PTRACE -AmbientCapabilities=CAP_NET_ADMIN CAP_NET_BIND_SERVICE CAP_SYS_ADMIN CAP_SYS_CHROOT CAP_DAC_OVERRIDE CAP_SYS_PTRACE +CapabilityBoundingSet=CAP_NET_ADMIN CAP_NET_BIND_SERVICE CAP_SYS_ADMIN CAP_SYS_CHROOT CAP_DAC_OVERRIDE CAP_CHOWN CAP_SYS_PTRACE +AmbientCapabilities=CAP_NET_ADMIN CAP_NET_BIND_SERVICE CAP_SYS_ADMIN CAP_SYS_CHROOT CAP_DAC_OVERRIDE CAP_CHOWN CAP_SYS_PTRACE [Install] WantedBy=multi-user.target @@ -2373,6 +2512,7 @@ main() { normalize_prefix initialize_metadata_paths detect_platform + ensure_supported_linux_world_posture prepare_tmpdir case "${PLATFORM}" in diff --git a/scripts/windows/wsl-warm.ps1 b/scripts/windows/wsl-warm.ps1 index 20264dc73..21b8ed149 100644 --- a/scripts/windows/wsl-warm.ps1 +++ b/scripts/windows/wsl-warm.ps1 @@ -11,7 +11,7 @@ $ErrorActionPreference = 'Stop' function Write-Info($Message) { Write-Host "[INFO] $Message" -ForegroundColor Cyan } function Write-Warn($Message) { Write-Host "[WARN] $Message" -ForegroundColor Yellow } -function Write-ErrorAndExit($Message) { Write-Host "[FAIL] $Message" -ForegroundColor Red; exit 1 } +function Write-ErrorAndExit($Message, [int]$Code = 1) { Write-Host "[FAIL] $Message" -ForegroundColor Red; exit $Code } function Convert-ToWslPathFragment { param([string]$Path) @@ -90,6 +90,7 @@ Write-Info "Starting wsl-warm for distro '$DistroName'" $projectPath = Resolve-Path $ProjectPath | Select-Object -ExpandProperty Path Write-Info "Project path: $projectPath" +Write-ErrorAndExit "WSL world provisioning is intentionally fail-closed in this slice because the WSL helper path is not aligned with the Linux/macOS placement contract for SUBSTRATE_HOME placement, socket/group ownership, and runtime artifact access. Use Linux host-native provisioning, macOS Lima provisioning, or a CLI-only WSL install with --no-world instead." 4 $projectHasCargo = Test-Path (Join-Path $projectPath 'Cargo.toml') $packagedWorldAgent = Join-Path $projectPath 'bin\\linux\\world-agent' diff --git a/scripts/wsl/provision.sh b/scripts/wsl/provision.sh index 7aad506e9..6b0bc00e0 100644 --- a/scripts/wsl/provision.sh +++ b/scripts/wsl/provision.sh @@ -1,79 +1,15 @@ #!/usr/bin/env bash set -euo pipefail -export DEBIAN_FRONTEND=noninteractive -apt-get update -apt-get install -y \ - nftables \ - iproute2 \ - libseccomp-dev \ - curl \ - jq \ - git \ - python3 \ - python3-pip \ - build-essential \ - dnsmasq \ - openssh-server \ - unzip \ - ca-certificates - -if ! command -v cargo >/dev/null 2>&1; then - echo "[INFO] Installing Rust toolchain via rustup (required to build world-agent)" - curl -sSf https://sh.rustup.rs | sh -s -- -y --profile minimal --default-toolchain 1.89.0 -fi - -# Ensure cargo/rustup are on PATH for subsequent provisioning steps. -if [[ -f /root/.cargo/env ]]; then - # shellcheck disable=SC1091 - . /root/.cargo/env -fi - -install -d -m 0700 /run/substrate -install -d -m 0755 /etc/substrate -install -d -m 0755 /var/log/substrate -install -d -m 0755 /var/lib/substrate - -cat <<'UNIT' >/etc/systemd/system/substrate-world-agent.service -[Unit] -Description=Substrate World Agent -After=network.target - -[Service] -Type=simple -Environment="SUBSTRATE_AGENT_TCP_PORT=61337" -Environment="SUBSTRATE_WORLD_SOCKET=/run/substrate.sock" -ExecStart=/usr/local/bin/substrate-world-agent -Restart=always -User=root -Group=root -RuntimeDirectory=substrate -RuntimeDirectoryMode=0700 -StandardOutput=journal -StandardError=journal - -[Install] -WantedBy=multi-user.target -UNIT - -cat <<'UNIT' >/etc/systemd/system/substrate-world-agent.socket -[Unit] -Description=Substrate World Agent Socket -PartOf=substrate-world-agent.service - -[Socket] -ListenStream=/run/substrate.sock -SocketMode=0660 -SocketUser=root -SocketGroup=root -DirectoryMode=0750 -RemoveOnStop=yes -Service=substrate-world-agent.service - -[Install] -WantedBy=sockets.target -UNIT - -systemctl daemon-reload -systemctl enable substrate-world-agent.service -systemctl enable --now substrate-world-agent.socket +cat >&2 <<'EOF' +[substrate/wsl-provision][ERROR] WSL world provisioning is intentionally fail-closed in this slice. +The WSL helper path is not aligned with the Linux/macOS placement contract for +SUBSTRATE_HOME placement, socket/group ownership, and runtime artifact access. + +Do not use this script to mutate a WSL guest. Use one of these paths instead: + - Linux host-native provisioning via scripts/linux/world-provision.sh + - macOS Lima provisioning via scripts/mac/lima-warm.sh + - CLI-only flows inside WSL with --no-world until WSL alignment lands +EOF + +exit 4