Skip to content

Several runtime optimizations for CubicODSDelta#210

Merged
MaxAlex merged 4 commits into
mainfrom
delta_fact_job_runtime_optimizations
Jul 7, 2026
Merged

Several runtime optimizations for CubicODSDelta#210
MaxAlex merged 4 commits into
mainfrom
delta_fact_job_runtime_optimizations

Conversation

@MaxAlex

@MaxAlex MaxAlex commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Adds several performance optimizations to delta_ods.py:

  • Increases MAX_MERGE_RECORDS to 500k, approximating the limit on fact_ods.py
  • The 'more_pending' log flag is now based on whether we hit the merge records limit, rather than a separate scan of the fact tables
  • When reading in CDC changes, instead of a single query for full data that uses LIMIT to restrict batch size, do two stages: first, a query just against the header__change_seq to find the upper and lower bound values that match the limit, and then a second query using those bounds. The previous version, which used ORDER BY header__change_seq LIMIT {limit} would require reading and sorting all header__change_seq values across all tables, rather than sorting only values that land within the interval.
  • Restrict the merge predicate based on the odin month/year, so that the merge process is only touching files that need to be updated. This only works for updates which include the edw_inserted_dtm, but that looks like it's always the case (for tables that include that column.)

@ealexa05 ealexa05 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirming this as a safe change, will go ahead with dev performance test

Comment thread src/odin/generate/cubic/delta_ods.py
@MaxAlex MaxAlex merged commit d55d07f into main Jul 7, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants