feat(#2088): migrate marts__micromasters_dedp_exam_grades to dim_course_run by quazi-h · Pull Request #2403 · mitodl/ol-data-platform

quazi-h · 2026-07-07T19:57:03Z

What are the relevant tickets?

Partially addresses #2088 (follow-up to #2319 — dim_course_run semester/passing_grade)

⚠️ This PR must be merged and deployed AFTER #2319 (dim_course_run.semester/passing_grade columns). Deploying this mart before #2319 lands and dim_course_run is rebuilt will result in NULL semester/proctoredexamgrade_passing_grade for the MITxOnline branch.

Description (What does it do?)

Migrates the MITxOnline branch of marts__micromasters_dedp_exam_grades to source semester and passing_grade from dim_course_run (added in #2319) instead of re-deriving them via a duplicate join to the MicroMasters staging model inside int__mitxonline__proctored_exam_grades.

Not migrated in this PR (scope note, same reasoning documented on #2319 by @quazi-h):

The pure-MicroMasters branch (micromasters_exam_grades, sourced from int__micromasters__dedp_proctored_exam_grades) still reads semester/passing_grade directly. Those exam runs have no corresponding dim_course_run row (only MITxOnline course runs matched to a MicroMasters exam run are represented there), so this branch can't be migrated until MicroMasters grades are added to tfact_grade (epic [Epic] Migrate mart and reporting models to use the dimensional layer #2072).
User/grade percentage/created_on fields are unchanged — still sourced from the int__ models pending that same epic.

How can this be tested?

dbt build --select dim_course_run marts__micromasters_dedp_exam_grades (requires feat(dimensional): add semester + passing_grade to dim_course_run from MicroMasters exam runs #2319 merged first)
Confirm output columns/row counts match the pre-migration mart via a diff query, e.g.:

SELECT examrun_courserun_readable_id, semester, proctoredexamgrade_passing_grade, count(*)
FROM marts__micromasters_dedp_exam_grades
GROUP BY 1,2,3
ORDER BY 1;

Spot-check that MITxOnline rows' semester/proctoredexamgrade_passing_grade values are unchanged from before the migration (same underlying source join, now centralized in dim_course_run).

Additional Context

Deployment order:

Merge and deploy feat(dimensional): add semester + passing_grade to dim_course_run from MicroMasters exam runs #2319 first (adds semester/passing_grade to dim_course_run)
Merge and deploy this PR — marts__micromasters_dedp_exam_grades is a non-incremental view over int_/dim_course_run, so no full-refresh is needed for this mart itself.

Copilot

Pull request overview

This PR updates the marts__micromasters_dedp_exam_grades mart so the MITxOnline branch sources semester and passing_grade from the dimensional model dim_course_run (intended to centralize and de-duplicate that derivation).

Changes:

Add a dim_course_run-backed CTE for MITxOnline semester/passing_grade.
Replace MITxOnline output columns to read from dim_course_run instead of the intermediate proctored-exam grades model.
Keep the pure-MicroMasters branch unchanged (still sourcing those fields directly from the MicroMasters intermediate model).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 2 comments.

Copilot

Pull request overview

Copilot reviewed 1 out of 2 changed files in this pull request and generated no new comments.

KatelynGit · 2026-07-08T17:05:27Z

+    from (
+        select
+            courserun_readable_id
+            , semester


I'm getting the following error currently when running the build

15:21:32 Completed with 1 error, 0 partial successes, and 1 warning:
15:21:32
15:21:32 Failure in model marts__micromasters_dedp_exam_grades (models/marts/micromasters/marts__micromasters_dedp_exam_grades.sql)

15:21:32 Database Error in model marts__micromasters_dedp_exam_grades (models/marts/micromasters/marts__micromasters_dedp_exam_grades.sql)
TrinoUserError(type=USER_ERROR, name=COLUMN_NOT_FOUND, message="line 35:15: Column 'semester' cannot be resolved", query_id=20260708_152128_00035_4j8nd)
compiled code at target/run/open_learning/models/marts/micromasters/marts__micromasters_dedp_exam_grades.sql

This was hitting the production dim_course_run table before its full-refresh completed (needed after #2319 added semester/passing_grade — see the SCD2 self-referential-read caveat that also caused a similar Dagster failure on courserun_upgrade_deadline this week). The refresh finished successfully in production earlier today and semester/passing_grade are confirmed present. Please pull latest main and retry the build — let us know if it still fails.

I'm getting another error with the build but I think it might be taken care of the update you mentioned that went in before this:

00:55:24 Completed with 1 error, 0 partial successes, and 0 warnings:
00:55:24
00:55:24 Failure in test not_null_marts__micromasters_dedp_exam_grades_semester (models/marts/micromasters/_marts_micromasters__models.yml)
00:55:24 Got 8137 results, configured to fail if >10

…se_run Sources semester and passing_grade for the MITxOnline branch from dim_course_run (added in #2319) instead of re-deriving them via a duplicate join to the MicroMasters staging model in int__mitxonline__proctored_exam_grades. The pure-MicroMasters branch still reads these fields from int__micromasters__dedp_proctored_exam_grades, since those exam runs have no corresponding dim_course_run row until MicroMasters grades are added to tfact_grade (epic #2072). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Corrects a comment that incorrectly attributed the dim_course_run semester/passing_grade columns to #2088 instead of the PR that actually adds them, #2319. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- Add row_number()-based dedup (partition by courserun_readable_id, order by effective_date desc) before joining dim_course_run, guarding against the same SCD2 expiration-gap fan-out risk already handled in dim_product. - Rename mitxonline_courserun_semester_grade -> mitxonline_courserun_metadata since the CTE selects course-run metadata (semester, passing_grade), not grade facts. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

for more information, see https://pre-commit.ci

github-actions · 2026-07-16T14:07:24Z

🔎 ol-dbt impact — column-level blast radius

✅ No column-level downstream impact detected for the changed models.

_{Posted by ol-dbt impact (annotate-only — does not block merge).}

quazi-h requested a review from Copilot July 7, 2026 20:24

Copilot started reviewing on behalf of quazi-h July 7, 2026 20:24 View session

Copilot AI reviewed Jul 7, 2026

View reviewed changes

Comment thread src/ol_dbt/models/marts/micromasters/marts__micromasters_dedp_exam_grades.sql Outdated

Comment thread src/ol_dbt/models/marts/micromasters/marts__micromasters_dedp_exam_grades.sql Outdated

quazi-h requested a review from Copilot July 7, 2026 20:34

Copilot started reviewing on behalf of quazi-h July 7, 2026 20:34 View session

Copilot AI reviewed Jul 7, 2026

View reviewed changes

Comment thread src/ol_dbt/models/marts/micromasters/marts__micromasters_dedp_exam_grades.sql Outdated

Comment thread src/ol_dbt/models/marts/micromasters/marts__micromasters_dedp_exam_grades.sql Outdated

quazi-h requested a review from Copilot July 7, 2026 21:18

Copilot started reviewing on behalf of quazi-h July 7, 2026 21:18 View session

Copilot AI reviewed Jul 7, 2026

View reviewed changes

quazi-h marked this pull request as ready for review July 7, 2026 21:38

KatelynGit self-assigned this Jul 8, 2026

KatelynGit reviewed Jul 8, 2026

View reviewed changes

quazi-h and others added 4 commits July 16, 2026 09:58

[pre-commit.ci] auto fixes from pre-commit.com hooks

1ba8823

for more information, see https://pre-commit.ci

blarghmatey force-pushed the copilot/2088-migrate-dedp-exam-grades-mart branch from d64395e to 1ba8823 Compare July 16, 2026 14:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(#2088): migrate marts__micromasters_dedp_exam_grades to dim_course_run#2403

feat(#2088): migrate marts__micromasters_dedp_exam_grades to dim_course_run#2403
quazi-h wants to merge 4 commits into
mainfrom
copilot/2088-migrate-dedp-exam-grades-mart

quazi-h commented Jul 7, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

KatelynGit Jul 8, 2026 •

edited

Loading

Uh oh!

quazi-h Jul 8, 2026

Uh oh!

KatelynGit Jul 9, 2026

Uh oh!

github-actions Bot commented Jul 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

quazi-h commented Jul 7, 2026

What are the relevant tickets?

Description (What does it do?)

How can this be tested?

Additional Context

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

KatelynGit Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

quazi-h Jul 8, 2026

Choose a reason for hiding this comment

Uh oh!

KatelynGit Jul 9, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jul 16, 2026

🔎 ol-dbt impact — column-level blast radius

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

KatelynGit Jul 8, 2026 •

edited

Loading