Skip to content

InterleaveExec::with_new_children panics when optimizer rewrites change children's partitioning #21826

@zhuqi-lucas

Description

@zhuqi-lucas

Describe the bug

InterleaveExec::with_new_children uses assert_or_internal_err!(can_interleave(children.iter())) which panics when children's partitioning diverges after optimizer rewrites. This can happen when multiple physical optimizer passes modify the plan between when InterleaveExec is created and when with_new_children is called.

To Reproduce

This occurs in practice when:

  1. EnforceDistribution creates InterleaveExec for a UnionExec whose children all have matching Hash partitioning
  2. Subsequent optimizer passes (e.g. JoinSelection, additional EnforceDistribution/EnforceSorting passes) modify children's plan structure, changing their output partitioning
  3. with_new_children is called on the InterleaveExec with the modified children → assertion panic

We hit this with a complex UNION ALL query (5 subqueries with JOINs, each reading from different table types — partitioned, single-file, MVs). The materialization task panics every time:

join_selection
caused by
Internal error: Assertion failed: can_interleave(children.iter()): Can not create InterleaveExec: new children can not be interleaved.

Expected behavior

InterleaveExec::with_new_children should gracefully fall back to UnionExec when children are no longer interleavable, instead of panicking. The existing comment in the code already acknowledges this possibility:

// New children are no longer interleavable, which might be a bug of optimization rewrite.

Correctness is preserved when falling back to UnionExecEnforceDistribution will add RepartitionExec as needed. Only the interleave optimization is lost.

Proposed fix

fn with_new_children(
    self: Arc<Self>,
    children: Vec<Arc<dyn ExecutionPlan>>,
) -> Result<Arc<dyn ExecutionPlan>> {
    if can_interleave(children.iter()) {
        Ok(Arc::new(InterleaveExec::try_new(children)?))
    } else {
        UnionExec::try_new(children)
    }
}

Will submit a PR with the fix and unit tests.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions