[multistage] Add tests for chaining colocated operations without data shuffle#18997
Open
yashmayya wants to merge 1 commit into
Open
[multistage] Add tests for chaining colocated operations without data shuffle#18997yashmayya wants to merge 1 commit into
yashmayya wants to merge 1 commit into
Conversation
05bfe3e to
069f4c0
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #18997 +/- ##
============================================
+ Coverage 65.23% 65.25% +0.01%
Complexity 1405 1405
============================================
Files 3419 3419
Lines 214702 214702
Branches 33931 33931
============================================
+ Hits 140067 140094 +27
+ Misses 63349 63324 -25
+ Partials 11286 11284 -2
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
…ations without shuffle
069f4c0 to
71dc519
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #18804 (now merged). Adds test coverage verifying that colocated operations can be chained without a data shuffle: when a chain of joins / window functions / set operations is all keyed on the same partition column and the colocation hints are applied, every inter-stage hash exchange in the plan is pre-partitioned (direct, 1-to-1) and no full shuffle is inserted anywhere in the chain.
This exercises the interaction of the three colocation hints:
joinOptions(is_colocated_by_join_keys='true')windowOptions(is_partitioned_by_window_keys='true')setOpOptions(is_colocated_by_set_op_keys='true')Tests added (test-only, no production changes):
QueryCompilationTest: a planner-level invariant check (testColocatedOperationsChainWithoutShuffle) that asserts every hash-distributed exchange in the plan is pre-partitioned, across four chains:Plus two baselines proving the hints are load-bearing (so the positive assertions are not vacuous):
testChainWithoutColocationHintsStillShuffles(the same join → window chain shuffles without the hints) andtestSetOpChainWithoutSetOpHintShuffles(withis_colocated_by_set_op_keys='false'the set-op level of the chain shuffles again).QueryHints.json: two H2-validated runtime cases (a colocated join → window, and aUNION ALLof two colocated joins) on tables physically partitioned by the key, confirming the chained colocation also produces correct results on both the V1 and V2 engines.Hint placement note: because a set operation is an ancestor of its branch
SELECTs, chaining through a set op uses the outer-SELECT-wrap form for the set-op hint with the join/window hints on each branch — this is what lets colocation propagate through every level of the chain.