feat(#2075): add courserun_upgrade_deadline to dim_course_run and enrollment_is_edx_enrolled to tfact_enrollment by quazi-h · Pull Request #2317 · mitodl/ol-data-platform

quazi-h · 2026-06-17T19:25:44Z

What are the relevant tickets?

Closes #2075 (partial — prerequisite for mart migration PR #2314)

Description (What does it do?)

Adds two missing fields to existing dimensional models. These are prerequisite changes that must be deployed (with --full-refresh) before the mart migration in #2314 is deployed.

dim_course_run.sql — add courserun_upgrade_deadline

MITxOnline: sourced directly from int__mitxonline__course_runs.courserun_upgrade_deadline (field already exists in that int model)
edX.org: sourced via a join to MicroMasters proctored exam course run records (courserun_edxorg_readable_id), which carry an upgrade deadline for the associated edxorg course run
All other platforms (MITxPro, Residential, Bootcamps): cast(null as varchar) — upgrade deadlines only exist on MITxOnline and edX.org (MicroMasters exam runs)
Propagated through: combined_courseruns_resolved, final SELECT, and records_to_expire SCD2 expiry block
courserun_upgrade_deadline added to the SCD2 change-detection predicate so deadline changes alone trigger a new SCD2 version

tfact_enrollment.sql — add enrollment_is_edx_enrolled

MITxOnline: courserunenrollment_is_edx_enrolled from int__mitxonline__courserunenrollments
MITxPro: courserunenrollment_is_edx_enrolled from int__mitxpro__courserunenrollments
edX.org: hardcoded true (all edxorg enrollments are on an edX-hosted platform)
Residential: hardcoded true (MIT Residential is OpenEdX-based)
Program enrollments: cast(null as boolean) (program-level concept, not course-run-level)
Bootcamps: cast(null as boolean) (Bootcamps is not edX-hosted)
Propagated through: final, final_deduped, and final SELECT

How can this be tested?

After deploying to QA with --full-refresh:

-- Q1: Verify courserun_upgrade_deadline is populated for MITxOnline runs
SELECT
    platform,
    count(*) AS total_rows,
    count(courserun_upgrade_deadline) AS non_null_upgrade_deadline,
    count(*) - count(courserun_upgrade_deadline) AS null_upgrade_deadline
FROM ol_warehouse_dimensional_qa.dim_course_run
WHERE is_current = true
GROUP BY 1
ORDER BY 1;
-- Expected: MITx Online has some non-null values; all other platforms are all null

-- Q2: Verify enrollment_is_edx_enrolled is populated per platform
SELECT
    platform,
    enrollment_is_edx_enrolled,
    count(*) AS row_count
FROM ol_warehouse_dimensional_qa.tfact_enrollment
WHERE enrollment_type = course
GROUP BY 1, 2
ORDER BY 1, 2;
-- Expected: edxorg and residential = all true; mitxonline/mitxpro = mix of true/false/null;
-- bootcamps = all null

Additional Context

Deployment note — --full-refresh required:

Both models are incremental (delete+insert). Adding a new column with on_schema_change='append_new_columns' will ALTER the table and add the column, but existing rows will have NULL for the new column until those rows are reprocessed by an incremental run.

Since most rows won't be reprocessed (course runs and enrollments don't change often), the columns will remain mostly NULL for historical data unless the models are run with --full-refresh.

Recommended deploy order:

Run int__bootcamps__course_runs --full-refresh (picks up recently-added course_readable_id column)
Run int__bootcamps__courserunenrollments --full-refresh (picks up recently-added courserunenrollment_updated_on column)
Run dim_course_run --full-refresh
Run tfact_enrollment --full-refresh
Confirm Q1/Q2 queries above show correct values
Then merge and deploy the mart migration PR (feat(#2075): migrate marts__combined_course_enrollment_detail to dimensional models #2314)

Steps 1–2 are needed because these bootcamps intermediate tables were materialized before their respective column additions were merged to main. Steps 3–4 depend on those columns being present.

This two-phase approach ensures the mart migration has fully-populated dimensional columns on deploy, with no regression window.

…ollment_is_edx_enrolled to tfact_enrollment - dim_course_run: add courserun_upgrade_deadline (MITxOnline from source; null for all other platforms) Propagated through combined_courseruns_resolved, final SELECT, and records_to_expire SCD2 block - tfact_enrollment: add enrollment_is_edx_enrolled (edxorg/residential=true, mitxonline/mitxpro from source field, program/bootcamps=null) Propagated through final, final_deduped, and final SELECT These fields are prerequisites for migrating marts__combined_course_enrollment_detail to the dimensional layer (tracked in a follow-up PR). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

These changes are now in a separate prerequisite PR: #2317 This branch now contains only the mart rewrite. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

This PR adds two missing attributes to dimensional-layer dbt models to support the upcoming migration of marts__combined_course_enrollment_detail to dimensional sources (issue #2075 / prerequisite for PR #2314).

Changes:

Add enrollment_is_edx_enrolled to tfact_enrollment, populated from MITxOnline/MITxPro sources and hardcoded per platform where appropriate.
Add courserun_upgrade_deadline to dim_course_run, populated for MITxOnline and null for other platforms, and propagated through the SCD2 flow.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
`src/ol_dbt/models/dimensional/tfact_enrollment.sql`	Adds `enrollment_is_edx_enrolled` across platform CTEs and propagates it to the final fact output.
`src/ol_dbt/models/dimensional/dim_course_run.sql`	Adds `courserun_upgrade_deadline` to the dimensional SCD2 model output and expiry block.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

… docs - Add courserun_upgrade_deadline to the incremental change-detection predicate (not exists block) so that a deadline change alone triggers a new SCD2 version. Previously, a change in upgrade_deadline with no other field changes would be silently ignored. - Add YAML column definition for courserun_upgrade_deadline in _dim_course_run.yml - Add YAML column definitions for enrollment_is_edx_enrolled and enrollment_updated_on in _fact_tables.yml Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

KatelynGit

LGTM

…ia micromasters join edxorg course run sources do not carry upgrade_deadline natively. The value is stored in the MicroMasters course run table keyed by courserun_edxorg_readable_id. Added micromasters_courseruns CTE and left-joined it to edxorg_courseruns so downstream marts (e.g. marts__combined_course_enrollment_detail) get parity with the existing combined enrollment detail output, which sources this field from int__edxorg__mitx_courserun_enrollments via the same micromasters join. Validated: 365 of 1,515 edxorg course runs now have upgrade_deadline populated. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

…te column docs Add WHERE courserun_platform = '{{ var("edxorg") }}' to the micromasters_courseruns CTE to prevent cross-platform false matches and 1:N fan-out if multiple MicroMasters rows share the same courserun_edxorg_readable_id. Update courserun_upgrade_deadline column description to reflect that edxorg runs are also populated (via MicroMasters join), not just MITxOnline. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

…org fan-out Add QUALIFY ROW_NUMBER() OVER (PARTITION BY courserun_edxorg_readable_id ORDER BY courserun_id DESC) = 1 to the micromasters_courseruns CTE. courserun_id is an auto-increment PK so DESC picks the most recently inserted row per readable ID, preventing 1:N fan-out if multiple edxorg MicroMasters rows share the same courserun_edxorg_readable_id. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

…no compatibility Trino does not support the QUALIFY clause. Replace the micromasters_courseruns dedup with the equivalent ROW_NUMBER() inner-subquery pattern, consistent with the existing pattern documented and used in dim_course.sql. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

Trino requires all subqueries in FROM clauses to have an alias. The derived table used for ROW_NUMBER dedup was missing 'as mcr_deduped', which would cause a parse error at runtime. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

…ginal formatting Reconstructs dim_course_run.sql to preserve the file's original leading-comma formatting instead of the full-file trailing-comma rewrite introduced earlier in this PR. Only the actual logical additions from this PR are retained: - new micromasters_courseruns CTE (deduped, edxorg-platform-filtered lookup for courserun_upgrade_deadline) - courserun_upgrade_deadline threaded through each per-platform CTE, the SCD2 change-detection predicate, and records_to_expire Verified equivalent to the prior version via token-level diff (only quote-style differences remain), dbt parse/compile against a local duckdb target, and sqlfluff-lint/-fix (no changes needed). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Resolves conflicts in dim_course_run.sql and _dim_course_run.yml between this PR's courserun_upgrade_deadline addition (#2075) and #2319's semester/passing_grade addition, which merged to main first. Both sets of changes are additive and independent (different CTEs, different platform sourcing), so the resolution keeps both: - micromasters_courseruns CTE (courserun_upgrade_deadline, edxorg-keyed) - micromasters_examruns CTE (semester/passing_grade, mitxonline-keyed) - Both columns threaded through mitxonline_courseruns, all other per-platform CTEs, the SCD2 change-detection predicate, final SELECT, and records_to_expire - Both column docs retained in _dim_course_run.yml Verified with dbt compile --select dim_course_run tfact_enrollment (clean) and sqlfluff-lint/-fix (no changes needed).

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Copilot AI review requested due to automatic review settings June 17, 2026 19:25

Copilot started reviewing on behalf of quazi-h June 17, 2026 19:26 View session

quazi-h mentioned this pull request Jun 17, 2026

feat(#2075): migrate marts__combined_course_enrollment_detail to dimensional models #2314

Open

Copilot AI reviewed Jun 17, 2026

View reviewed changes

Comment thread src/ol_dbt/models/dimensional/dim_course_run.sql

quazi-h requested a review from Copilot June 23, 2026 20:32

Copilot started reviewing on behalf of quazi-h June 23, 2026 20:32 View session

Copilot AI reviewed Jun 23, 2026

View reviewed changes

Comment thread src/ol_dbt/models/dimensional/dim_course_run.sql Outdated

KatelynGit self-assigned this Jun 24, 2026

KatelynGit approved these changes Jun 24, 2026

View reviewed changes

quazi-h requested a review from Copilot June 24, 2026 21:48

Copilot started reviewing on behalf of quazi-h June 24, 2026 21:49 View session

Copilot AI reviewed Jun 24, 2026

View reviewed changes

Comment thread src/ol_dbt/models/dimensional/dim_course_run.sql

Comment thread src/ol_dbt/models/dimensional/dim_course_run.sql

Comment thread src/ol_dbt/models/dimensional/_dim_course_run.yml Outdated

quazi-h requested a review from Copilot June 30, 2026 20:51

Copilot started reviewing on behalf of quazi-h June 30, 2026 20:51 View session

Copilot AI reviewed Jun 30, 2026

View reviewed changes

Comment thread src/ol_dbt/models/dimensional/dim_course_run.sql Outdated

quazi-h requested a review from Copilot July 1, 2026 15:26

Copilot started reviewing on behalf of quazi-h July 1, 2026 15:26 View session

Copilot AI reviewed Jul 1, 2026

View reviewed changes

Comment thread src/ol_dbt/models/dimensional/dim_course_run.sql Outdated

quazi-h requested a review from Copilot July 1, 2026 19:12

Copilot started reviewing on behalf of quazi-h July 1, 2026 19:13 View session

Copilot AI reviewed Jul 1, 2026

View reviewed changes

Comment thread src/ol_dbt/models/dimensional/dim_course_run.sql Outdated

quazi-h requested a review from Copilot July 1, 2026 21:19

Copilot started reviewing on behalf of quazi-h July 1, 2026 21:20 View session

Copilot AI reviewed Jul 1, 2026

View reviewed changes

quazi-h requested a review from Copilot July 7, 2026 21:00

Copilot started reviewing on behalf of quazi-h July 7, 2026 21:01 View session

Copilot AI reviewed Jul 7, 2026

View reviewed changes

Comment thread src/ol_dbt/models/dimensional/dim_course_run.sql

quazi-h requested a review from Copilot July 7, 2026 21:14

Copilot started reviewing on behalf of quazi-h July 7, 2026 21:25 View session

Copilot AI reviewed Jul 7, 2026

View reviewed changes

quazi-h requested a review from Copilot July 7, 2026 21:29

Copilot started reviewing on behalf of quazi-h July 7, 2026 21:29 View session

Copilot AI reviewed Jul 7, 2026

View reviewed changes

quazi-h merged commit d211669 into main Jul 7, 2026
6 checks passed

quazi-h deleted the copilot/2075-add-dimensional-fields branch July 7, 2026 21:33

Uh oh!

Conversation

quazi-h commented Jun 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What are the relevant tickets?

Description (What does it do?)

How can this be tested?

Additional Context

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

KatelynGit left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

quazi-h commented Jun 17, 2026 •

edited

Loading