Skip to content

[multistage] Add tests for chaining colocated operations without data shuffle#18997

Open
yashmayya wants to merge 1 commit into
apache:masterfrom
yashmayya:colocated-chaining-tests
Open

[multistage] Add tests for chaining colocated operations without data shuffle#18997
yashmayya wants to merge 1 commit into
apache:masterfrom
yashmayya:colocated-chaining-tests

Conversation

@yashmayya

@yashmayya yashmayya commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Follow-up to #18804 (now merged). Adds test coverage verifying that colocated operations can be chained without a data shuffle: when a chain of joins / window functions / set operations is all keyed on the same partition column and the colocation hints are applied, every inter-stage hash exchange in the plan is pre-partitioned (direct, 1-to-1) and no full shuffle is inserted anywhere in the chain.

This exercises the interaction of the three colocation hints:

  • joinOptions(is_colocated_by_join_keys='true')
  • windowOptions(is_partitioned_by_window_keys='true')
  • setOpOptions(is_colocated_by_set_op_keys='true')

Tests added (test-only, no production changes):

  • QueryCompilationTest: a planner-level invariant check (testColocatedOperationsChainWithoutShuffle) that asserts every hash-distributed exchange in the plan is pre-partitioned, across four chains:

    • colocated join → window
    • set op of two colocated joins
    • set op of two colocated windows
    • the triple: colocated join → window → set op

    Plus two baselines proving the hints are load-bearing (so the positive assertions are not vacuous): testChainWithoutColocationHintsStillShuffles (the same join → window chain shuffles without the hints) and testSetOpChainWithoutSetOpHintShuffles (with is_colocated_by_set_op_keys='false' the set-op level of the chain shuffles again).

  • QueryHints.json: two H2-validated runtime cases (a colocated join → window, and a UNION ALL of two colocated joins) on tables physically partitioned by the key, confirming the chained colocation also produces correct results on both the V1 and V2 engines.

Hint placement note: because a set operation is an ancestor of its branch SELECTs, chaining through a set op uses the outer-SELECT-wrap form for the set-op hint with the join/window hints on each branch — this is what lets colocation propagate through every level of the chain.

@yashmayya yashmayya force-pushed the colocated-chaining-tests branch from 05bfe3e to 069f4c0 Compare July 15, 2026 23:30
@codecov-commenter

codecov-commenter commented Jul 16, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 65.25%. Comparing base (f054a90) to head (71dc519).

Additional details and impacted files
@@             Coverage Diff              @@
##             master   #18997      +/-   ##
============================================
+ Coverage     65.23%   65.25%   +0.01%     
  Complexity     1405     1405              
============================================
  Files          3419     3419              
  Lines        214702   214702              
  Branches      33931    33931              
============================================
+ Hits         140067   140094      +27     
+ Misses        63349    63324      -25     
+ Partials      11286    11284       -2     
Flag Coverage Δ
custom-integration1 100.00% <ø> (ø)
integration 100.00% <ø> (ø)
integration1 100.00% <ø> (ø)
integration2 0.00% <ø> (ø)
java-21 65.25% <ø> (+0.01%) ⬆️
temurin 65.25% <ø> (+0.01%) ⬆️
unittests 65.24% <ø> (+0.01%) ⬆️
unittests1 56.89% <ø> (+0.02%) ⬆️
unittests2 37.74% <ø> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@gortiz gortiz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@yashmayya yashmayya added testing Related to tests or test infrastructure multi-stage Related to the multi-stage query engine labels Jul 16, 2026
@yashmayya yashmayya force-pushed the colocated-chaining-tests branch from 069f4c0 to 71dc519 Compare July 16, 2026 20:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

multi-stage Related to the multi-stage query engine testing Related to tests or test infrastructure

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants