doc: design for columnar dataflow edge migration#37744
Draft
antiguru wants to merge 17 commits into
Draft
Conversation
Plan to migrate compute dataflow edges from VecCollection to columnar Column batches, as a dependency DAG of fine-grained stacked PRs. Follows up the CollectionEdge scaffolding in MaterializeInc#36507. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Producer-flip invariant (producers flip only after all consumers are native), which removes the seam-heavy intermediate state and justifies no feature flag. Scope the compiler-enforced completeness claim to producers and add a whole-plan differential test as the consumer net. Add the import-boundary producers (P9), preserve FlatMap fueling (C2), expand LetRec scope and MVP spikes for the two unknowns (C7/C8), and note introspection golden churn. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
C7 (LetRec) and C8 (Union temporal-bucket) are sanctioned decode points for now; the blocker for native C7 is the ResultsIn/Negate/leave_dynamic feedback machinery, not branch_when. P7 (join output) confirmed native, builder swaps only. C7/C8 no longer depend on F1. Gate passed: nothing infeasible, T2 can complete. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fork PRs cannot stack across repos and gh-stack is strictly linear, so the DAG is flattened into a single topological chain (C1..T3) managed with gh-stack, each branch based on the previous. Replaces the multi-lane layout. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
An adversarial trace of delta_join.rs confirmed delta join reads inputs only via pre-built arrangements (arrangement(), and build_update_stream walks an Arranged trace), never an unarranged CollectionEdge. The edge decode for its inputs lives in arrange_collection, which C1 already migrated. Remove C5 from the chain; the delta-join output flip remains as P7. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The sink already consumes bundle.collection (the CollectionEdge) directly and leaf-decodes via into_vec at sinks.rs:71, with no pre-boundary Vec forcing. When a producer flips columnar the decode lands at that leaf, which is C6's goal, so no separate node exists. T2 must adapt the sinks.rs:71 call site when into_vec becomes a free function. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
LetRec already consumes bundle.collection (the CollectionEdge) and leaf-decodes via into_vec at render.rs:941/997, then runs consolidate + branch_when on the Vec. That is the desired sanctioned-decode end state, so no separate node. T2 must adapt those call sites when into_vec becomes a free function. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
C8 subsumed: Union already reads inputs as edges, temporal-bucket into_vec at render.rs:1358 is a gated sanctioned leaf, and concat_many->consolidate_named goes native via F1. Wave 1 is complete (C1/C3/C4/F1/C2 landed; C5-C8 subsumed). Also correct T1: vec_to_columnar is NOT dead. It survives as a leaf encode primitive for row-shaped producers (P9 imports) and the sanctioned-decode returns (C7 LetRec, C8 Union temporal-bucket), which decode to Vec, operate via branch_when / maybe_apply_temporal_bucketing, then re-encode. T1 now only retires the concat_many mixed-variant upgrade branch; the teardown collapses the enum, not the leaf conversions. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Standing rule for all producer flips: the pre-migration producers consolidated duplicates within a batch (ConsolidatingContainerBuilder). Plain ColumnBuilder drops that, a permanent regression on the broad Mfp-to-sink class. Columnar builders copy row bytes from a borrowed push either way, so ConsolidatingColumnBuilder keeps the borrowed push and restores within-batch consolidation. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Consolidation needs owned staging, so ConsolidatingColumnBuilder is owned-give only. Not a regression: a producer computes its Row fresh (mfp_plan.evaluate already owns it), so the give is a move into staging, exactly as pre-P1's ConsolidatingContainerBuilder. The borrowed-push/no-owned-Row property is a consumer optimization (C1/C3/C4), not applicable to producers. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Owned-give staging holds N live Rows per batch. A ref-accepting builder that stages bytes into a Column arena and consolidates via a sorted usize permutation would hold ~1 live Row, an allocator/peak-memory win on the hot Mfp-to-sink path. New primitive with its own correctness surface, so tracked as a focused follow-up PR rather than blocking the producer wave. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
C1 made arrange_collection return a variant-preserving passthrough and P1's raw-store rework dropped the last CollectionEdge::Vec force-wrap, so the ArrangeBy path is already columnar end to end. The Vec force-wraps P3 targeted do not exist in the tree. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…subsumed Reduce/Threshold/TopK-bucketed are arrangement producers (from_columns / from_expressions), not collection producers, so there is no per-operator from_collections to flip. Their arrangement is columnar-internal (C1); their output becomes Vec only at the shared as_specific_collection arrangement materialization. Repurpose P5 to flip that one shared path (covers Reduce + Threshold + TopK-bucketed), reduce P6 to TopK's from_collections monotonic/top1 paths, and drop P8 (subsumed by P5). Record the collection-vs-arrangement producer taxonomy. P5 lands before T2 so T2 stays a mechanical enum collapse. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The as_specific_collection Some(key) path reads an already-consolidated arrangement cursor (distinct rows per time), so it is a consumer read, not a producer computing new rows. Use plain ColumnBuilder + borrowed push (C1/C3/C4 pattern), not ConsolidatingColumnBuilder + owned give: consolidation would be wasted sort and would change the deliberate non-consolidating semantics. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
mz_join_core cannot emit Column (bound + CapacityContainerBuilder), and its stage output is intra-operator, so it stays Vec (columnar mz_join_core is a differential-side fast-follow). Only the final linear-join edge goes columnar: Some(closure) via flat_map_fallible + ConsolidatingColumnBuilder; None via vec_to_columnar of the raw stage (leaf-encode, one VecToColumnar op on the common identity join). Delta unchanged (half_join generic CB -> ColumnBuilder). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The spike's "flip half_join generic CB to ColumnBuilder" was wrong: half_join output carries Result<Row,_> (ColumnBuilder can't build it; ok_err demux runs after) and isn't the node output anyway (per-path map + finalization, then oks is their concat). Delta internals are all Vec-native/intra-operator, like linear's mz_join_core. Flip = vec_to_columnar(oks) at the node output. Both joins: Vec-internal operators, columnar edge produced by a leaf-encode at the node output boundary; columnar join algorithms are a differential-side fast-follow. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Whole-branch review: GO, no correctness defects. Deleting the Vec arm reveals legitimate internal Vec sites, so teardown expands to five ordered steps: Ta temporal-bucketing re-encode (unblocks Td), Tb linear-join accumulator off the edge (M1), Tc retire as_specific_collection flag-off (M2), Td retire concat_many mixed branch (needs Ta), Te collapse enum + de-match. The leaf conversions (into_vec/columnar_to_vec/vec_to_columnar) are never deleted. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Plan to migrate compute dataflow edges from
VecCollectionto columnarColumnbatches, following theCollectionEdgescaffolding in #36507.The doc decomposes the migration into a dependency DAG of fine-grained, independently landable changes, one per stacked PR. Key decisions: single code path with no feature flag, the
CollectionEdgeenum stays as the representation carrier until a final compiler-enforced teardown deletes theVecarm, andCollectionExtis left as-is rather than generalized.Design only, no code changes.
🤖 Generated with Claude Code