Release MCP 3.0.0rc6: projection tick scaling#104
Merged
Conversation
…keyed parse cache Each projection tick enumerated and re-parsed every leaf enclosure contract three times (read_enclosures, read_engine_process_facts, drift-snapshot pruning each ran their own iter_leaf_enclosure_contracts + load_contract pass). observer/contract_snapshot.py now builds ONE immutable ContractSnapshot per tick inside project_and_write, backed by a cross-tick parse cache keyed by stat identity (path + mtime_ns + size) and pruned to the live enumeration each build. The three readers take a keyword-only contracts=None parameter; a standalone call builds a local snapshot, so public signatures and behavior are unchanged. Parse failures are never cached (skip + retry next tick, the exact pre-existing containment). The landing refresher and supervisor sweep keep their own passes: the refresher walks on the asyncio loop thread while the cache is single-writer on the projection worker thread, so joining them is not trivially safe (R5). Tests: ContractSnapshotSharedPassTests proves N parses on tick 1, zero on an unchanged tick 2, exactly the changed file on tick 3 (R7); one enumeration per full project_and_write tick (R1); reader-output equality standalone vs shared vs warm-cache (R4); bounded retention across churn (R3); malformed-contract retry containment (R2). Full mcp suite 2096 passed / 3 skipped; ruff repo-wide clean; pyright clean on touched files. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BKStH52GkNm7UZ37eijbLh
load_contract no longer runs normalize_contract_leaf_id per read (the hidden resolve_leaf_ref tasks-tree walk that dominated the projection tick); write_contract keeps write-time normalization. Legacy stem-shaped leaf ids heal through the new idempotent heal_contract_leaf_ids sweep (per-root doc-id fast path, loud logs, error containment) invoked via the heal-leaf-ids CLI subcommand or directly at startup. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BKStH52GkNm7UZ37eijbLh
The projector re-projected the whole world unconditionally every --interval second (1.0s in production) even when nothing changed (py-spy: _tick_sync 11.1s of a 15s quiet-world sample). Waking is now change-driven: a debounced watchfiles (inotify) watcher over the projection's actual input surfaces feeds a ChangePacer -- settle 0.1s after the last write of a burst, max-delay = interval from the first (a busy world keeps the former cadence), never two projections closer than interval (the flag keeps meaning the fast-path cadence floor) -- and a slow configurable heartbeat (--heartbeat, default 15s) bounds /api/state staleness and time-derived field resolution for everything a watcher cannot see (memory-root TTL surfaces, landing state, git-derived worktree status, stale/overdue flips). - new serving/change_watcher.py: derived watch roots (tasks/, observer lifecycles/workspace/drift, providers status/setup, temp progress and tool-report dirs, worktrees/*/*/provider-runtime only -- never the checkouts), input-event filter (self-outputs, *.tmp, workspace river, supervisor heartbeat), 30s watch-set refresh, reconcile tick after re-establish, loud fixed-interval degrade on any failure (fail-open) - projector: pacer-or-legacy-sleep pacemaker; tick body, prime, SSE diff/broadcast, ETag revision, and cancellation semantics unchanged; sim replay and injected-now() tests keep the exact legacy pacing; projection_count/last_wake_reason instrumentation - create_app: heartbeat + watch_changes seams (watcher live-only); --heartbeat plumbed through the CLI, daemon spawn/ensure, and the --reload dev factory - watchfiles>=1.1,<2 added as an explicit dependency (decision logged; no watch library existed in the tree) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BKStH52GkNm7UZ37eijbLh
…dening) Adversarial review confirmed a permanent-staleness hole: chmod 000 on a contract changes neither mtime_ns nor size, so the (mtime_ns, size) cache kept serving the old good parse FOREVER, where the pre-cache readers degraded to skip-every-tick (PermissionError) -- and unlike the same-stat rewrite window, it never self-healed. Same trigger class: a rewrite with (mtime_ns, size) pinned via os.utime. ctime_ns changes on chmod/utime/rename/writes and is untouched for genuinely unchanged files, so adding it to _ParseCacheEntry and the _cached_contract comparison costs zero extra parses on the unchanged path. Tests: permission flip is skipped (not served stale) on the next build and self-heals when readability returns; a same-size utime-pinned rewrite is now detected and re-parsed. Both step past the kernel's coarse-clock granule before mutating (ctime cannot be set explicitly), and the chmod assertion is skipped when running as root. Focused suites 260 passed; full mcp suite 2098 passed / 3 skipped; ruff + pyright clean on touched files. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BKStH52GkNm7UZ37eijbLh
Two one-line hardenings adopted from the adversarial review (verdict INTEGRATE, no blocking findings): - F1: exclude workspace/operator-inbox.lock from the watch filter. The inbox store opens it "a+b" on every access (including each tick's read_agent_pickups), so its boot-time creation emitted exactly one spurious change-tick per daemon start. - F3: move the projection_input_roots derivation inside the watcher run loop's retry guard. A transient stat/glob failure now follows the same loud degrade-and-retry path as a watch failure instead of escaping run() and killing the task permanently (which was fail-open but lost the periodic self-heal). Tests: filter unit extended with the lock file; new recovery test proves a derivation that raises once degrades loudly and re-establishes on the next retry cycle. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BKStH52GkNm7UZ37eijbLh
The watch derivation globbed worktrees/*/*/provider-runtime and watched it recursively, descending into each task's container data (Postgres/grepai) — unreadable to the daemon user (permission-denied crashed the whole watch) and high-churn (every WAL write would re-project). Remove that root entirely: worktree provider-state.json changes are infrequent and heartbeat-covered, and central provider status is already watched via logs/providers. Add ignore_permission_denied=True to awatch as defence-in-depth.
Bump the three repository-owned version locations to 3.0.0rc6. This RC carries the 260712_projection-tick-scaling work: - walk-free load_contract + explicit heal-leaf-ids migration seam (PTS-L1) - one shared contract snapshot per tick with a stat-keyed parse cache (PTS-L2) - change-driven projection with idle heartbeat; no worktree/container watching (PTS-L3) Observer tick read cost ~750ms -> ~9ms; idle re-projections ~20/20s -> ~3/20s.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cuts MCP 3.0.0rc6. Carries the
260712_projection-tick-scalingwork that removes the observer daemon's per-tick O(contracts × tree × readers) filesystem storm.What changed
load_contract. Leaf-id normalization moved off the hot read path to write time plus an explicit idempotentheal-leaf-idsmigration sweep. A contract load no longer walks the whole tasks tree.read_enclosures,read_engine_process_facts, and drift pruning share a single enumeration, backed by a stat-identity (mtime_ns,size,ctime_ns) parse cache across ticks.Measured impact
Verification
Version bumped to 3.0.0rc6 in the three repository-owned locations (pyproject,
SERVER_VERSIONfallback, README).🤖 Generated with Claude Code