Release Agents Remember 3.0.0rc4#101
Merged
Merged
Conversation
The boot-snapshot config is no longer launch authority: worktree_start
provider setup, provider_watchers start/restart/invalidate-indexes,
provider query one-shots, and the runtime_install watcher rebind re-read
the authority settings file and refuse fail-closed when providers are
disabled on disk (stop/status/cleanup stay always-legal). Benchmark
provider synthesis is filtered to the live authority set, persisted
workspace registrations are swept to it on every benchmark touch, and
the direct-script path is fail-closed behind
AR_BENCHMARK_ALLOW_UNFILTERED_PROVIDERS=1. Provider setup is serialized
host-wide (one prepare at a time, lock in the system tempdir out of
prune reach) so concurrent sessions cannot stack capped-but-summing
container storms. New containment metrics: the serving daemon samples
labeled provider containers every 30s into logs/observer/providers/ and
provider_status carries the snapshot even when providers are disabled.
Query funnels require their specific provider armed; docker stats is
fed only running names.
Root cause: 2026-07-07 WSL OOM — providers launched despite providers:{}
in every harness settings file (emptied 10:28) via the boot-snapshot and
benchmark-workspace vectors; one enabled worktree_start starts 7
containers including both workspace source backends.
Adversarial review: round 1 BLOCK (B1-B4) → all fixed → round 2
PASS-WITH-NOTES; full quality gate green (1656 tests).
…h-up state, never a teardown The cgc seed no longer refuses on HEAD divergence: relatable divergence seeds the near-perfect graph clone and hands the classified delta (--name-status; deletions and rename-sources are phantom residuals) to a post-watcher catch-up stage that waits for the watcher's post-subscribe log marker (inotify has no replay; no marker means no touch and honest staleness) and then utime-touches exactly the touchable files so the event-driven watcher re-indexes just the delta. caughtUp is claimed only for a clean delta delivered to a ready watcher; everything else reports staleIndex (served; a from-zero rebuild stays an explicit cgc refresh). Only unrelatable heads still refuse (foreign-graph protection). The implicit refresh-all fallback default is OFF (explicit opt-in flag; the hermetic benchmark path opts in for its synchronous bounded build); benign seed skips never fail a prepare. The grepai half: the worktree mtime sync leaves divergent-content files fresh so the watcher re-embeds exactly the delta instead of silently serving stale content. Index staleness is a reportable state: identity-carrying index-state rows land in the central metrics store and provider_status carries the newest ones. Root cause (developer-pinned): providers seeded fine on the first leaf only because the integration branch had no changes yet; after leaf 1 integrated, the next start hard-crashed into teardown + refresh-all. Adversarial review: round 1 BLOCK (B1 watcher-attach race, B2 over-reported catch-up, B3 scope gap, B4 benchmark half-built graph) -> all fixed -> round 2 PASS-WITH-NOTES; full quality gate green (1677 tests).
Delivery reports success only after a pane capture proves the paste landed. One origin baseline is held for the whole delivery over a history-inclusive capture window (capture-pane -p -S -200) and every retry re-captures before re-pasting, so a landed paste is never re-sent; verification runs a strongest-first probe ladder: payload-specific codex chip instance growth, generic chip-count growth across BOTH vocabularies ([Pasted text #N] claude, [Pasted Content N chars] codex — the codex form was unrecognized and every landed paste re-pasted blind), then payload-head echo. Payloads ride stdin into tmux load-buffer then paste-buffer -p -d (no argv/ARG_MAX seam). Failures are loud: the spawn payload ships deliveryCapture (explicit "(empty pane capture)" marker when empty), the inbox push records a bounded capture tail, and the paste endpoint attaches the capture on delivery or submit failure. Escape is refused at the paster — Enter is the only key this seam sends. Root cause (SF-1/F-V, sprint forensics): contextDelivered:true on a clean codex boot with no payload, and per-attempt re-baselining that stacked up to 7 duplicate pastes (empirically reproduced: 10 on the pre-fix seam). Adversarial review: round 1 PASS-WITH-NOTES -> N1 (viewport blindness) + N3 (parity) fixed -> round 2 PASS-WITH-NOTES; gate green (1695 tests).
Degradation detector over the L1/L2 metrics store (healthy/degraded/critical state machine, hysteresis-gated transitions, durable JSONL events surviving daemon restart) wired into the serving metrics sampling loop. Inbox notification on every state change (operator_inbox_post_payload-parity delivery: store append, compact, hosted-session paste) to the orchestrator and every active manager, carrying the metric snapshot and role-appropriate standing instruction. roles/system-specialist.md authored: investigate -first seat, fixes only on explicit orchestrator order, providers-only scope this iteration, Sentry-replaceable detection seam recorded. Manager doctrine: no new provider starts during a degradation-alert, no kill authority. Orchestrator doctrine: dispatch system-specialist, read the report, fix-in-session or stop providers before they take the host down. Settings loader gains providerDegradation config + system-specialist to KNOWN_ROLES (nine roles). Critical-threshold failsafe: MCP-initiated provider stop via the always-legal teardown path, durable even when the stopper itself raises, posted to the inbox as the automated last line.
…rewiring Activates the curator chain HFX-L6 ratified on paper. New dispatch-pack template templates/curator-brief.md feeds the curator the landed change set (leaf contract base-to-head range), the leaf task doc, and notes/ as inputs, with the mgmt-L4 routing rule (specific sidecar or governing overview; the L3 Operational-Notes target is last resort only; overview-dumping rejected) stated verbatim. roles/curator.md rewritten to describe the fed inputs, the seat-routing statement (builder = code + report only; curator = where c-05 runs), and the explicit binding of c-12's checks to this pass. roles/manager.md curator-spawn duty made concrete and enforced: names the brief template, the fed inputs, and a hard gate against running the closeout preview before the curator pass exists. templates/manager-brief.md updated to match. c-12-closeout reworded (Preconditions, External-Memory Order, Failure Conditions) so missing-onboarding/changed-sidecar-body checks are framed as verifying the curator's already-landed pass, never as the closing seat's cue to author onboarding inline; a still-failing check now names "rerun the curator's pass" as the fix, never "write it here". c-05-create-or-update-onboarding-files gains a seat-routing paragraph: the curator runs this skill's workflows during leaf work, not the builder; the strict 1-to-1 mapping, governing- overview-link, and metadata rules are unchanged. Doctrine-only change: no Python source touched.
…n-state Seat retirement (issue #12): session_retire MCP tool + POST /api/terminal/{session}/retire terminate the tmux session and mark the catalog row retired with provenance (retired_at/retired_by_session/ retired_reason/retired_edge) layered onto the existing terminated status, never a zombie row, never re-stamped on an already-retired seat. Server-side authority policy (serving/retire_policy.py): owner-never-self-retires checked first and unconditionally; a manager may retire only worker/reviewer seats of its own master; the orchestrator may retire any seat including a completed manager. Automation hooks at worktree_integrate (worker/reviewer, leaf edge) and lifecycle_finalize_task (manager/reviewer, master edge), config-gated (both default ON), best-effort and never able to fail their completion edge (the whole retire body is wrapped in a broad exception guard). Live identity (issue #4): session_rename MCP tool + POST /api/terminal/{session}/rename update the display label, freezing the original spawn-time label into spawned_label on first rename; identity text only, never touches spawn_role. Live turn-state: a marker-based classifier (working/turn-ended/awaiting-input/ stale) rides the existing L5 liveness-sweep cadence, no new hot loop; observer events fire only on an actual state transition. Doctrine additions to roles/manager.md, roles/orchestrator.md, templates/manager-brief.md.
…tem/decision-ruling relay Master-exit fix leaf. AgentRole gains architect and curator; InboxMessageKind gains decision-item and decision-ruling, making the HFX-L6-ratified minimal decision-item relay (orchestrator posts a decision-item to the architect; architect posts a decision-ruling back) representable and round-trippable through the operator inbox for the first time - the schema previously rejected the exact call the landed doctrine mandates. New round-trip test proves the doctrine-mandated call succeeds end to end through the tool-payload layer, and that a plain message to architect/curator succeeds. No other consumer of these Literals enumerates them exhaustively, so no downstream edits were needed. Also documents L8's retirement settings object in docs/reference/settings-json.md and examples/mcp/settings.example.json (parity with L7's providerDegradation), and registers designer and system-specialist in the dashboard's role-chip color registry.
…ranteed redelivery Ack semantics (consume=ack is the only terminal outcome; delivered never terminal; compaction never prunes pending/unacked rows), durable expectation rows written atomically at every dispatch surface (spawn/gate-open/signal-post) with configurable per-kind SLAs, redelivery backoff + rate limiting, hierarchical routing derivation from spawn provenance, and projection surfacing of pending rows/deadlines. 43 new tests, full code_quality.check green, sync-skills --check green.
Deterministic sweep loop (serving/supervisor.py, pane_signals.py, supervisor_heartbeat.py) hosted in the serving daemon lifespan alongside metrics_loop. Five predicate families (pane-state, expectation-deadline, turn-report staleness via missing_artifact()'s first caller, unacked-row redelivery, seat-liveness) read stores directly, never the projection. Four action executors (redeliver/auto-nudge/signal-emit/escalation-stub) with orchestration.supervisor.* observer events. Self-liveness heartbeat with MCP banner + dashboard header badge. 87+535 focused tests passed, full suite 1860 passed, code_quality.check exit 0, sync-skills --check green. Fix round 2 corrected a dashboard-store regression (unconditional write on idle re-snapshot) found at closeout; dashboard test/typecheck verified independently (27 passed, tsc clean). Includes rebuilt dashboard bundle sync (package_data/dashboard).
…anism
New deliver(row) -> {acked, landed-unacked, blocked(reason), failed(reason)}
contract (serving/injector.py) wrapping TerminalPaster's unchanged
capture-verify semantics, with harness-blocked-overrides-landed
classification. New HarnessAdapter interface (serving/harness_adapters.py)
composing existing pane_signals/turn_state classifiers for claude-code and
codex (boot-ready, composer states, mid-turn queueing, blocked-reason
detection incl. codex quota modal). Both existing delivery call sites
(inbox_delivery.deliver_inbox_entry, terminal.py spawn-brief path) now
route through the one path; standardized DeliveryRow envelope. Non-goals
(harness hooks/SDK/app-server) documented with a new-harness checklist.
1890 tests passed (0 regressions; L2's supervisor tests pass unmodified),
code_quality.check exit 0, sync-skills --check green.
New escalation_ladder.py (rung_due/next_step/seat_is_suspect) wired into serving/supervisor.py: rung 1 renudge, rung 2 skip-level via a new two-hop derive_skip_level_owner (kept separate from L1's one-hop derive_signal_owner), rung 3 developer attention (hard ceiling). Respawn-not-wait wired to HFX-L8's retire.retire_entry with pending-queue re-delivery; orphan_policy.py surfaces a dead manager's live workers (detection-only stub per spec). Dead-upstream detection signals the grandparent via the same two-hop walk. 1931 tests passed (0 regressions across L1-L3), code_quality.check exit 0, sync-skills --check green. Two scope gaps documented: no 260628 developer-notification seam exists yet (rung 3 uses the existing dashboard-visible inbox row); orphan re-parenting is detection-only.
Role contracts inverted from active vigilance to passive/supervised reliability across manager.md, orchestrator.md, worker.md, SKILL.md, and templates/turn-report.md: the L2 supervisor sweep detects missing/stale artifacts mechanically, the L4 escalation ladder handles inactivity — seats process-and-ack supervisor-injected signals rather than hand-rolling watchers (explicit watcher ban added). Notify-and-stop reframed as safe-by-design since silence is now supervised. New mcp/tests/test_liveness_simulations.py (11 tests) proves all 8 named P-15 incident classes end acked-or-escalated within SLA: 6/8 fully end-to-end through run_supervisor_sweep, 2/8 honestly documented as hybrid coverage (a real architectural gap - the sweep's pane capturer isn't injectable through SupervisorContext - flagged as a follow-up, not faked). Liveness report gates L6. 1942 tests passed (0 regressions across L1-L4), code_quality.check exit 0, sync-skills --check green across all 9 packages.
…sweep Close the two liveness gaps the 2026-07-08 dead-seat-storm incident exposed: - R1 ladder-resolved terminal state: a pending row at the terminal escalation rung whose target seat is provably dead (retired / no hosted session) transitions to a durable non-redeliverable state; redeliverable() excludes it via a state-keyed predicate (mid-climb / live-seat rows untouched); one orchestration.supervisor.ladder-resolved event per transition. - R2 bounded sweep: one in-sweep inbox snapshot threaded through redelivery, escalation, respawn, delivery recording, and ladder resolution; O(1) log reads per finding; heartbeat ticks unconditionally under backlog. - R3 compaction drops ladder-resolved terminal rows, preserves pending/unacked. - R4 per-sweep redeliverBudget (default 250, defaults-safe); terminal resolution is budget-independent; per-target rate-limit intact. - R5 dead-seat-storm liveness simulation (2000 rows) asserting termination, convergence, bounded log, and fresh heartbeat. - R6 backlog counts + last-sweep duration on /api/state + dashboard header; non-destructive recovery runbook; boot-safety skip of non-task JSON. Full manager/worker/adversarial-reviewer loop; reviewer verdict APPROVE-WITH-NITS, R1-R6 all PASS. Gates the L7 rc4 tail.
… reviewer criteria (CS-6, PR-6) Adds the D1/D2/D3 doctrine to system/coding-guidelines.md (memory), CS-6 to skills/l-01-agent-lifecycles/criteria/code-seam.md and PR-6 to criteria/plan-review.md (canonical, synced to all harness mirrors + mcp package data), and a Code Quality Instructions cross-reference in AGENTS.md. Encodes the 2026-07-08 dead-seat-storm retrospective (HFX2-L7) into standing coding rules and reviewer criteria so the correct-but-superlinear / unbounded-growth defect class is a standing review question. Leaf full-loop review: PASS-WITH-NOTES (notes/reports/260707-HFX2-L8-reviewer-report.md). No blocking findings.
…ling Enforces a 900s minimum effective redelivery/signal-cooldown floor across the operator-inbox and supervisor pane/seat-liveness paths (fail-loud below floor), adds a persisted per-(owner,leaf,kind,detail) signal cooldown store, and suppresses immediate escalation for mid-turn pane findings (still observable via time-based predicates). Fixes the 2026-07-09 live incident where the supervisor flooded the orchestrator with sub-minute redelivery/signal noise once manager/worker orchestration was active. Leaf full-loop review: PASS-WITH-NOTES (notes/reports/260707-HFX2-L9-reviewer-report.md). Known non-blocking finding: the new supervisor_signals.py cooldown store is itself an unbounded append-only log (CS-6-class defect) tracked for HFX2-L11; orchestration.supervisor.enabled stays false until L11 covers it.
…model/effort overrides Removes ordinary caller authority over spawn spend controls (harness, model, effort, launch/session controls, AR_SPAWN_MODEL/AR_SPAWN_EFFORT env, and maintained Claude/Anthropic + Codex/OpenAI harness-native spend/endpoint env keys) with a loud spend-override-unsupported refusal before any spawn side effect. Resolution is settings-only via resolved_role_knobs(role, level). Closes the live incident where an agent bypassed configured manager/worker harness settings. Leaf full-loop review across 3 rounds (hard cap): round 1 BLOCK (unfiltered env spend side-channel) -> round 2 ENFORCE fix + delta-verify PASS-WITH-NOTES (narrow residual NEW-1: 5 missing harness-native keys) -> round 3 closes NEW-1 + final delta-verify PASS-WITH-NOTES. No open blocking finding. notes/reports/260707-HFX2-L10-reviewer-report.md.
…leted seats)
Reverses the prior auto-retire-on-success completion edge: worktree_integrate
and lifecycle_finalize_task now land completed worker/reviewer/manager seats
(status:"landed", non-terminated, tmux kept alive, fully inspectable) instead
of retiring them. Manual session_retire and its authority policy are
unchanged for stuck/duplicate/abandoned/explicit-close cases. Adds a
dashboard landed-archive group with a group-cleanup button (backend-rechecked,
closes only still-landed rows, reports closed/skipped). Settings renamed to
autoLandOn{Integration,Finalize} (default on) with legacy autoRetireOn*
aliases honored for backward compatibility. Updates manager/orchestrator
doctrine and the manager-brief template to match.
Leaf full-loop review across 2 rounds: round 1 BLOCK (landed seats silently
enrolled into the liveness sweeper's per-cycle work -- unbounded O(N)
subprocess + O(N^2) catalog-read cost as landed seats accumulate by design,
the 3rd master catch of the CS-6 class after L7/L9) -> round 2 fixes the
sweep to skip landed rows entirely, verified by a flat-cost scaling
regression (5 vs 500 landed rows), plus 3 bundled low-severity cleanups
(orphaned retire_seats_for_leaf removed, stale landing.py doc corrected,
legacy exited rows routed to the existing ungrouped view instead of being
stuck in the landed archive) -> final delta-verify PASS-WITH-NOTES. No open
blocking finding. notes/reports/260707-HFX2-L11-reviewer-report.md.
…26-07-09) Ruled invariant: no event, message, or row outranks system health. - Escalation ladder mutates ONE row per root cause (re-anchor, re-address, redeliver, increment tries) -- never mints sibling rows. Retires the branching process behind the 2026-07-09 storm (67,925 lines / 227 MB / 20,553 pending rows, host down). - Supervisor posts coalesce: a re-firing condition renews its pending row (bumped ts, refreshed detail) instead of appending a duplicate. - Retention: pending rows age out (48 h TTL); hard health cap (500 folded rows, newest kept); inbox compacted every sweep. Supersedes HFX2-L1 R1 immortal-pending. - Ladder terminal is the ARCHITECT seat, not a "developer" mailbox: the human is an authority, never an address. Live architect seat when attached, role-addressed level-triggered row when not; architect seats are never suspect-respawned. Architect doctrine gains terminal-custody + catch-up-report duties. - CS-6 promoted with D4 quiescence probe: zero-input fixed-point simulation for any actor whose output is its own input class; executable counterpart test_unacked_backlog_reaches_a_fixed_point_with_absent_developer. Gate: 2006 passed / 3 skipped, ruff clean, pyright clean, sync-skills green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W1xRo5Dskk8Dc6SiFhfgjr
…propose-first strategist/solo Ruled 2026-07-09: - The developer's first chat is a LAUNCHER, never a role seat: research-only answers inline; any role-shaped work spawns the architect into its own chat with the settings-owned profile (no self-attach, no profile guessing, no ambiguity). - The chain is self-driving: free chat -> architect -> orchestrator -> managers (per plan + concurrency settings) -> workers. No seat waits for "spawn this, spawn that". - Strategist pass is proposed to the developer as a yes/no question, never auto-run (supersedes the 2026-07-06 mandatory pre-run); recommend skip when a ruled plan exists. Orchestrator dispatches only on a relayed yes. - Solo/short-root is the developer's call: propose it when the work is tiny (a line or two); more than ~2 leaves of work spools up full orchestration (work extends; single chats hit context limits). - Updated packaged + repo-root coordinator AGENTS.md routing accordingly. (Live deploy of ar-coordination/AGENTS.md and the stale .claude session-start hook applied out-of-repo alongside this commit.) Gate: 2006 passed / 3 skipped, sync-skills green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W1xRo5Dskk8Dc6SiFhfgjr
…running sprint) Ruled 2026-07-09: when the developer adds a master to a sprint, the architect attaches it to the sprint STRUCTURE (not chat context): master task doc first, slug appended to the sprint orchestration doc's top-level `orchestrates` list (the exact field the dashboard derives the Operations orchestration > master > leaf hierarchy from), decision-log entries on both docs, strategist fit-check PROPOSED as a yes/no question (recommend by in-flight interaction, never auto-run), and one inbox row to the sprint's orchestrator so it folds the master into its DAG/waves. Previously no instruction existed anywhere for this flow (`orchestrates` was mentioned once, in the orchestrator's adoption step). Deployed live alongside this commit: the changed l-01 files copied into ar-coordination/skills/l-01-agent-lifecycles (SKILL.md, roles/architect.md, roles/orchestrator.md, criteria/code-seam.md). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W1xRo5Dskk8Dc6SiFhfgjr
added 7 commits
July 10, 2026 13:18
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 79b2fd6c4d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| # per-entry probes' read-modify-writes and the terminated-row reclamation all hit the batch's | ||
| # in-memory buffer; the single atomic commit lands on ``batch()`` exit. Without this each of | ||
| # the n probes re-read and rewrote the full catalog file -- O(n^2) disk work per sweep. | ||
| with self._catalog.batch(): |
There was a problem hiding this comment.
When the dashboard liveness sweep runs while another process writes the same terminal catalog (for example the MCP process spawning or attaching a session), this batch holds an in-memory snapshot across the tmux probes and writes it back at the end. TerminalCatalog's lock is process-local, so those external writes never update _batch and the final batch commit can drop newly-created catalog rows, leaving live tmux sessions missing from the dashboard/leaf-uniqueness registry. The batch needs a cross-process merge/lock or a fresh read before commit.
Useful? React with 👍 / 👎.
| const attachActiveLeaf = async (leafKey: string) => { | ||
| if (!activeSession || !leafKey || activeSession.leafKey === leafKey) return; | ||
| const attachActiveLeaf = async (leafKey: string, seatRole: string) => { | ||
| if (!activeSession || !activeSessionIsRunning || !leafKey || activeSession.leafKey === leafKey) return; |
There was a problem hiding this comment.
Now that the attach picker also chooses seatRole, returning solely because the session is already on this leafKey prevents repairing or changing the role for an existing binding. In the common legacy/hand-opened case where a chat is already attached as chat and the user selects the same leaf with worker/manager, this exits before the backend attachSessionToLeaf(..., seatRole) call that would update the catalog, so routing/grouping/retire policy continue to see the stale role unless the user moves away and back.
Useful? React with 👍 / 👎.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Lands the completed Hotfix 1 and Hotfix 2 orchestration stack and prepares Agents Remember
3.0.0rc4.Highlights include durable supervisor delivery and acknowledgement semantics, bounded escalation and reclamation, settings-owned spawn dispatch, leaf-role seat bindings, landed chat archives, the sprint-local chat rail and adjustable Chats sidebar, strict quality-gate repairs, and the final consumed-inbox resurrection race correction.
Root cause addressed
The final live release gate exposed a race where a consumed inbox row could be resurrected by an in-flight supervisor delivery appending from a stale pending snapshot. Inbox current-state folding is now terminal-dominant and consume retains its terminal snapshot until normal compaction.
Validation