feat(lakehouse): runtime data-trust signals on the dimensional layer (P3)

[blarghmatey]

What & why

Part of the dbt warehouse CI/QA hardening & data-trust effort (#2407). PR-time CI now runs (ol-dbt diff/validate, dimensional-layering lint, expanded pytest — #2425/#2428/#2430); this adds the runtime trust signals.

dagster-dbt already hoists every dbt test (including the heavy dbt_expectations suite) into Dagster asset checks automatically, so those are not reimplemented. This PR adds the signals dbt tests cannot express, focused on the declared contract boundary — the dimensional layer (models/dimensional).

Changes

• Freshness policy on the 47 dimensional dbt assets. A FreshnessPolicy (warn 30h / fail 48h) is attached via map_asset_specs. Evaluated automatically by the Dagster daemon — no sensor required (the modern replacement for the superseded build_last_update_freshness_checks + sensor).
• Column-schema-change checks on the same 47 assets. They ride along with each build and compare against the previously recorded TableSchema — viable for free because full_dbt_project already captures column metadata via .fetch_column_metadata().
• Scheduled dbt source freshness job publishing sources.json to S3 for OpenMetadata. Source freshness was configured but never actually run anywhere. Ships STOPPED per the repo convention (enable after a first verified run).
• Source-freshness config expanded 2 → 3 active platforms: adds a MIT Learn users_user canary on the native-timestamp updated_on field. (Intentionally not bootcamps/micromasters — those are dead/retiring and would only generate false alarms.)

Verification

• dagster definitions validate -m lakehouse.definitions → passes; 47/47 dimensional specs carry the freshness policy, all 655 keys intact, 47 schema-change check specs registered.
• dbt parse compiles the new freshness config into the manifest (mitxonline, mitxpro, mitlearn).
• pre-commit (ruff format + check, mypy, yamllint, detect-secrets) → green.

Follow-ups (need warehouse credentials to validate loaded_at_field per source): extend source freshness to more active platforms (mitxresidential, mitpe); push run_results.json into OpenMetadata as DBT test cases; enrich ol-dbt impact with OM downstream lineage.

🤖 Generated with Claude Code

[github-actions]

🔎 ol-dbt impact — column-level blast radius

✅ No column-level downstream impact detected for the changed models.

_{Posted by ol-dbt impact (annotate-only — does not block merge).}

[Copilot]

Pull request overview

Adds runtime data-trust signals for the lakehouse dbt dimensional layer so Dagster can surface operational “is this data still healthy?” indicators that dbt tests alone don’t cover. This complements the existing dbt-test-to-asset-check hoisting by dagster-dbt and the recent CI hardening work.

Changes:

• Attach a FreshnessPolicy (warn 30h / fail 48h) to dimensional dbt assets via map_asset_specs.
• Add column schema change asset checks for dimensional assets via build_column_schema_change_checks.
• Add a scheduled dbt source freshness job (STOPPED by default) that uploads sources.json to S3 for OpenMetadata, and expand MIT Learn source freshness config.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File	Description
src/ol_dbt/models/staging/mitlearn/_mitlearn__sources.yml	Adds freshness configuration + loaded_at_field canary for the MIT Learn users source table.
dg_projects/lakehouse/lakehouse/assets/lakehouse/dbt.py	Implements dimensional-layer freshness policy mapping, schema-change checks, and a dbt source freshness job that uploads sources.json.
dg_projects/lakehouse/lakehouse/definitions.py	Wires the new job/schedule into lakehouse Definitions and registers the dimensional schema-change checks.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.