Skip to content

#346 Disable AQE in broadcast notebook#347

Merged
jathavaan merged 1 commit into
mainfrom
fix/346-broadcast-notebook-disable-aqe
May 24, 2026
Merged

#346 Disable AQE in broadcast notebook#347
jathavaan merged 1 commit into
mainfrom
fix/346-broadcast-notebook-disable-aqe

Conversation

@jathavaan
Copy link
Copy Markdown
Collaborator

Summary

  • Disable Adaptive Query Execution (AQE) around the timed spatial join in the broadcast notebook
  • AQE was rewriting Sedona's BroadcastIndexJoinExec into Spark's native BroadcastNestedLoopJoinExec (brute-force cross product), causing spark.driver.maxResultSize overflow at national scale
  • Mirrors the same AQE guard already present in the partitioned notebook

Closes #346

Test plan

  • Run broadcast notebook on Databricks at national scale and verify join completes without maxResultSize error
  • Confirm Sedona selects BroadcastIndexJoin (check Spark UI physical plan)

Copilot AI review requested due to automatic review settings May 24, 2026 13:42
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

@jathavaan jathavaan self-assigned this May 24, 2026
@jathavaan jathavaan merged commit bf5e59e into main May 24, 2026
23 of 24 checks passed
@jathavaan jathavaan deleted the fix/346-broadcast-notebook-disable-aqe branch May 24, 2026 14:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Broadcast notebook missing AQE disable — falls back to BroadcastNestedLoopJoin

2 participants