Migrate production spines to Arc batches for cross-thread arrangement sharing#37743
Draft
antiguru wants to merge 4 commits into
Draft
Migrate production spines to Arc batches for cross-thread arrangement sharing#37743antiguru wants to merge 4 commits into
antiguru wants to merge 4 commits into
Conversation
antiguru
force-pushed
the
claude/spines-differential-arc-j93mho
branch
2 times, most recently
from
July 20, 2026 08:30
dbf08aa to
798aa48
Compare
Batches shared across threads, for example behind an Arc to serve reads from outside the worker that maintains the trace, require their contents to be Send + Sync. This already holds for all row-spine batch types because the backing containers bottom out in Vecs, lgalloc regions, and CompactBytes. Add a compile-time assertion so it stays that way. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FwFQJa3HSy8oGJZ3X9uGZx
Switch Materialize's spine typedefs to Arc'd batches so arrangements can be read across thread boundaries, the prerequisite for sharing a trace between adjacent timely runtimes. - mz-row-spine: RowRowSpine, RowValSpine, RowSpine, ValRowSpine and their builders move from Rc/RcBuilder to Arc/ArcBuilder. ErrSpine and the generic spines are aliases of these and follow. - mz-compute typedefs: ColValSpine, ColKeySpine and builders likewise. - The arrangement-size logger held batches by Rc weak reference; it now uses Arc. The storage sink's stock OrdValSpine builder alias is renamed. The batch contents (lgalloc regions, CompactBytes, columnation stacks) were already Send + Sync, asserted in mz-row-spine, so only the wrapper changes. differential-dataflow is patched to the fork carrying arc_blanket_impls and ArcBuilder via a sibling-checkout path patch. This is prototype plumbing until an upstream release carries those, at which point the patch repoints at a git rev or drops. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FwFQJa3HSy8oGJZ3X9uGZx
The stock differential OrdValSpine is Rc-backed. The sink trace uses the Arc-backed ArcOrdValSpine variant so its batches match the Arc migration of the other production spines. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FwFQJa3HSy8oGJZ3X9uGZx
Repoint the [patch.crates-io] entry for differential-dataflow and differential-dogs3 from the local sibling checkout to the fork branch (antiguru/differential-dataflow, the Arc-batches PR). Prototype plumbing until that lands upstream and a released version carries arc_blanket_impls. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FwFQJa3HSy8oGJZ3X9uGZx
antiguru
force-pushed
the
claude/spines-differential-arc-j93mho
branch
from
July 20, 2026 14:06
0122ea5 to
d97c0a1
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
CPU-bound index/MV maintenance and latency-sensitive reads share the same worker threads today, so a peek can stall behind a large merge. Isolating reads from maintenance requires arrangement batches to be readable across thread boundaries, which means moving spine batches from
RctoArc.This PR is the
Rc→Arcmigration only. The architecture it serves is described in the design PR #37747 (two-runtime compute + shared arrangements). The differential-dataflow primitive it depends on is TimelyDataflow/differential-dataflow#807.This is a draft pending #807 landing upstream, and a benchmark pass.
Description
Send + Syncassertions inmz-row-spineconfirm the batch contents (lgalloc regions,CompactBytes, columnation stacks) are already thread-safe, so only the reference-count wrapper needs to change.Rc→Arc:RowRowSpine,RowValSpine,RowSpine,ValRowSpineand their builders inmz-row-spine, andColValSpine,ColKeySpineinmz-computetypedefs, now useArc/ArcBuilder.ErrSpineand the generic spines are aliases of these and follow.mz-computeheld batches byRcweak reference and now usesArc(the trace-box handle staysRc, as it should); the storage sink switched from the stockOrdValSpineto theArc-backedArcOrdValSpine.differential-dataflow/differential-dogs3are patched via[patch.crates-io]to the branch behind Arc-backed spines for cross-thread arrangement sharing TimelyDataflow/differential-dataflow#807, which carriesarc_blanket_impls/ArcBuilder. Prototype plumbing until that PR is released.The change is a minimal, mechanical
Rc→Arcswap. The batch layout, containers, and operator logic are untouched. No design docs live here — those moved to #37747.Verification
mz-row-spine,mz-compute, andmz-storagecargo checkclean against theArc-backed differential fork.mz-row-spinetests pass, including a new compile-timebatches_are_send_syncassertion over the productionRow/Timestamp/Difflayouts, which now covers the real spines rather than only their contents.Performance: the switch adds atomic reference counting, which occurs per batch clone/drop (a handful per merge/import), never per record, so the hot cursor/merge paths are unaffected. A benchmark comparison is worth running before this leaves draft.
🤖 Generated with Claude Code