[OSI][Databricks] Add Databricks Unity Catalog Metric View converter#171
Closed
Haoranli503 wants to merge 1 commit into
Closed
[OSI][Databricks] Add Databricks Unity Catalog Metric View converter#171Haoranli503 wants to merge 1 commit into
Haoranli503 wants to merge 1 commit into
Conversation
…icks) Bidirectional, offline converter between OSI semantic models and Databricks Unity Catalog Metric Views (YAML v1.1), filling the DATABRICKS spoke already listed in converters/index.md. Packaged like the dbt/gooddata spokes: pyproject.toml, an osi_databricks package under src/, and a tests/ suite (example-based + Hypothesis property-based round-trip; 73 tests). PyYAML is the only runtime dependency. Co-authored-by: jackstein21 <82542300+jackstein21@users.noreply.github.com>
Author
|
closed, will be track as #3 in the new ossie repo |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add Databricks Unity Catalog Metric View converter
Adds a bidirectional, offline converter between OSI semantic models and Databricks
Unity Catalog Metric Views (YAML
v1.1), filling the Databricks spoke in the hub-and-spoke architecture (
DATABRICKSisalready a listed vendor in
converters/index.md). Validated against the live DatabricksMetric View engine.
What's included
osi-databricks export): OSI semantic model -> Metric View(one fact
source+ a nestedjoinstree, flatteneddimensions,measures).osi-databricks import): Metric View -> OSI. Metric-View-only features(filter, window, format, rely, cardinality, parameters, materialization) are preserved
in
custom_extensions[DATABRICKS], soMV -> OSI -> MVis lossless.osi-databricksCLI and a string-in / string-out Python API.Design highlights
custom_extensions[DATABRICKS]stash -- the approachthe converter guide recommends.
Metric View's nested fact +
joinstree; a dataset reached by multiple paths (adiamond) is fanned out into one aliased join per path; cyclic graphs are rejected.
many_to_one/one_to_manyis derived from therelationship
from/todirection relative to a selectable grain (--source).Multiple facts are supported only narrowly -- when they share a conformed dimension
(named as the source); a general galaxy schema (facts each with their own dimensions)
or facts sharing no dimension can't map to one Metric View and are rejected with a
clear error, never silently mis-converted.
primary_key/unique_keyswhose columns cover a join key becomesrely: {at_most_one_match: true}, and vice versa.expressions; fact columns in measures are emitted bare (the DBR idiom).
(a non-equi or cross join), a cyclic or ambiguous-grain graph, and an unsupported
version are rejected with a clear
ConversionError; anything dropped (a foreign-vendorextension, an OSI-only annotation, a non-DATABRICKS/ANSI_SQL dialect) emits a warning
naming the object.
Conventions followed
converters/databricks/, packaged like the dbt/gooddata spokes:pyproject.toml(hatchling, Apache-2.0),src/osi_databricks/,tests/.DATABRICKS, falls back toANSI_SQL.Testing
Example-based unit tests plus Hypothesis property-based round-trip tests in both
directions, with a normalized TPC-DS fixture as a baseline.
Notes
custom_extensionshave no slot in the Metric View YAML; they aredropped on export with a warning.
Credits
Builds on the property-based round-trip testing approach from #152; @jackstein21 is
credited as a co-contributor.