Skip to content

compute: columnar FlatMap input, fueling preserved (C2)#37758

Draft
antiguru wants to merge 1 commit into
columnar-f1-consolidatefrom
columnar-c2-flatmap-input
Draft

compute: columnar FlatMap input, fueling preserved (C2)#37758
antiguru wants to merge 1 commit into
columnar-f1-consolidatefrom
columnar-c2-flatmap-input

Conversation

@antiguru

@antiguru antiguru commented Jul 20, 2026

Copy link
Copy Markdown
Member

Rework render_flat_map so the bespoke FlatMapStage consumes the input CollectionEdge directly instead of decoding via as_specific_collection (which forces into_vec).
The unarranged path reads the edge, so the columnar arm never decodes rows at the input.
The keyed path still reads an existing arrangement, already columnar internally, presented as a Vec edge.

The fuel machinery is the point of this operator: it bounds how much work a single generate_series can do before yielding the worker.
To keep the Vec and columnar arms from drifting, the fuel queue, per-activation budget, and re-activation now live in one generic flat_map_stage.
The only arm-specific piece is reading records from a queued batch, factored into a small FlatMapBatch trait: the Vec arm drains owned rows, the columnar arm iterates the borrowed column via into_index_iter and never materializes an owned Row.
FlatMap is explicitly not routed through flat_map_datums, which drains a full batch with no fuel budget.

Producers still emit Vec, so the columnar arm is dead in production and this carries no runtime change.

Tests: flat_map_fuel_bounds_per_activation proves a small fuel budget spreads a multi-batch input across multiple activations while an unbounded budget drains it in one; flat_map_arms_agree proves the two arms produce identical output across multiple timestamps including a retraction.

Rework `render_flat_map` so the bespoke `FlatMapStage` consumes the input
`CollectionEdge` directly instead of decoding it via `as_specific_collection`
(which forces `into_vec`). The unarranged path reads the edge, so the columnar
arm never decodes rows at the input; the keyed path still reads an existing
arrangement, already columnar internally, presented as a `Vec` edge.

The fuel machinery is the point of this operator: it bounds how much work a
single `generate_series` can do before yielding the worker. To keep the `Vec`
and columnar arms from drifting, the fuel queue, per-activation budget, and
re-activation now live in one generic `flat_map_stage` function. The only
arm-specific piece is how records are read from a queued batch, factored into a
small `FlatMapBatch` trait: the `Vec` arm drains owned rows, the columnar arm
iterates the borrowed column via `into_index_iter` and never materializes an
owned `Row`. The per-record table-function expansion and mfp drain are shared
in `process_flat_map_row`. FlatMap is explicitly not routed through
`flat_map_datums`, which drains a full batch with no fuel budget.

Producers still emit `Vec`, so the columnar arm is dead in production and this
carries no runtime change.

Two unit tests: `flat_map_fuel_bounds_per_activation` proves a small fuel
budget spreads a multi-batch input across multiple activations while an
unbounded budget drains it in one, and `flat_map_arms_agree` proves the `Vec`
and columnar arms produce identical output across multiple timestamps including
a retraction.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant