Skip to content

feat: add NB_FMD_PURVIEW_LINEAGE_TABLE_COLUMN for standalone column-level Purview lineage#241

Draft
edkreuk with Copilot wants to merge 3 commits into
mainfrom
copilot/create-wiki-page-fabric-to-microsoft-purview
Draft

feat: add NB_FMD_PURVIEW_LINEAGE_TABLE_COLUMN for standalone column-level Purview lineage#241
edkreuk with Copilot wants to merge 3 commits into
mainfrom
copilot/create-wiki-page-fabric-to-microsoft-purview

Conversation

Copilot AI commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Adds a dedicated notebook for registering column-level lineage in Microsoft Purview, decoupled from the parallel orchestration notebook. The core addition is dual-mode schema resolution: columns can be read from either a registered Delta table in the attached lakehouse ("table") or any readable path — Delta, Parquet, or CSV — via an explicit ABFSS URI or lakehouse-relative path ("path").

New notebook: NB_FMD_PURVIEW_LINEAGE_TABLE_COLUMN

Schema resolution (source_type / target_type per side):

Mode Resolution
"table" spark.read.format("delta").load("Tables/{schema}/{table}")
"path" spark.read.format(fmt).load(path) — delta / parquet / csv

Purview Atlas entities registered per call:

  • Source & target azure_datalake_gen2_resource_set with onelake://{workspace_guid}/{lakehouse_name}/{schema}/{table} qualified names
  • Process entity carrying columnMapping (case-insensitive column-name match) enabling column-level lineage in the Purview catalog graph

Exit value — JSON summary (purview_status, mapped_columns, total_runtime) so the notebook is callable from pipelines or orchestration notebooks with inspectable results.

# Table mode — Bronze → Silver
source_type = "table"
source_lakehouse_name = "LH_BRONZE_LAYER"
source_schema = "sales"
source_table = "customer"

target_type = "table"
target_lakehouse_name = "LH_SILVER_LAYER"
target_schema = "sales"
target_table = "customer"

layer = "bronze_to_silver"
purview_account_name = "my-purview-account"

# Path mode — landing file → Bronze
source_type = "path"
source_path = "abfss://workspace@onelake.dfs.fabric.microsoft.com/lh_id/Files/raw/customer.parquet"
source_format = "parquet"

Copilot AI changed the title feat: add NB_FMD_PURVIEW_LINEAGE_TABLE_COLUMN notebook for column-level Purview lineage feat: add NB_FMD_PURVIEW_LINEAGE_TABLE_COLUMN for standalone column-level Purview lineage Jul 3, 2026
Copilot AI requested a review from edkreuk July 3, 2026 17:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants