Multi-partition tasks: partition_slice plumbing + SortShuffleWriter refactor#2038

Merged

avantgardnerio merged 15 commits into

apache:mainfrom

avantgardnerio:brent/multi-partition-tasks

Jul 19, 2026

Commits on Jul 17, 2026

Substrate: multi-partition tasks + K-drain ShuffleWriter
avantgardnerio
committed
fix: size stage.partitions from the shuffle writer's child output_partitioning

avantgardnerio
and
claude
committed
fix(task_builder): route each UnionExec parent partition to its child's local index

avantgardnerio
and
claude
committed
WIP: merge task_id / task_index into a single per-stage slot id

avantgardnerio
and
claude
committed
feat(scheduler): add ballista.scheduler.max_partitions_per_task cap

avantgardnerio
and
claude
committed
proto: reuse tag 3 for TaskId.global_output_partition_ids

avantgardnerio
and
claude
committed
fix: preserve per-partition metrics under multi-partition tasks

avantgardnerio
and
claude
committed
docs: explain multi-partition tasks, Spark comparison, and AQE composition

avantgardnerio
and
claude
committed

Commits on Jul 18, 2026

Commits on Jul 19, 2026