Skip to content

[OSI][Databricks] Add Databricks Unity Catalog Metric View converter#171

Closed
Haoranli503 wants to merge 1 commit into
apache:mainfrom
Haoranli503:add-databricks-converter
Closed

[OSI][Databricks] Add Databricks Unity Catalog Metric View converter#171
Haoranli503 wants to merge 1 commit into
apache:mainfrom
Haoranli503:add-databricks-converter

Conversation

@Haoranli503

Copy link
Copy Markdown

Add Databricks Unity Catalog Metric View converter

Adds a bidirectional, offline converter between OSI semantic models and Databricks
Unity Catalog Metric Views (YAML
v1.1), filling the Databricks spoke in the hub-and-spoke architecture (DATABRICKS is
already a listed vendor in converters/index.md). Validated against the live Databricks
Metric View engine.

What's included

  • Export (osi-databricks export): OSI semantic model -> Metric View
    (one fact source + a nested joins tree, flattened dimensions, measures).
  • Import (osi-databricks import): Metric View -> OSI. Metric-View-only features
    (filter, window, format, rely, cardinality, parameters, materialization) are preserved
    in custom_extensions[DATABRICKS], so MV -> OSI -> MV is lossless.
  • A unified osi-databricks CLI and a string-in / string-out Python API.
  • A README with a mapping table and limitations.

Design highlights

  • Round-trip fidelity via the custom_extensions[DATABRICKS] stash -- the approach
    the converter guide recommends.
  • Join tree, not flat joins. The OSI relationship graph is reassembled into the
    Metric View's nested fact + joins tree; a dataset reached by multiple paths (a
    diamond) is fanned out into one aliased join per path; cyclic graphs are rejected.
  • Cardinality from direction. many_to_one / one_to_many is derived from the
    relationship from/to direction relative to a selectable grain (--source).
    Multiple facts are supported only narrowly -- when they share a conformed dimension
    (named as the source); a general galaxy schema (facts each with their own dimensions)
    or facts sharing no dimension can't map to one Metric View and are rejected with a
    clear error
    , never silently mis-converted.
  • Key bridge. A primary_key/unique_keys whose columns cover a join key becomes
    rely: {at_most_one_match: true}, and vice versa.
  • On export, joined-table columns are alias-qualified in dimension/measure
    expressions; fact columns in measures are emitted bare (the DBR idiom).
  • Fails loudly, never silently. A join condition an OSI relationship can't represent
    (a non-equi or cross join), a cyclic or ambiguous-grain graph, and an unsupported
    version are rejected with a clear ConversionError; anything dropped (a foreign-vendor
    extension, an OSI-only annotation, a non-DATABRICKS/ANSI_SQL dialect) emits a warning
    naming the object.

Conventions followed

  • Lives at converters/databricks/, packaged like the dbt/gooddata spokes:
    pyproject.toml (hatchling, Apache-2.0), src/osi_databricks/, tests/.
  • On export, dialect selection prefers DATABRICKS, falls back to ANSI_SQL.
  • The only runtime dependency is PyYAML; Python 3.9+.

Testing

cd converters/databricks
pip install -e . && pip install pytest hypothesis
python -m pytest tests/

Example-based unit tests plus Hypothesis property-based round-trip tests in both
directions, with a normalized TPC-DS fixture as a baseline.

Notes

  • Targets Databricks Unity Catalog Metric View YAML v1.1.
  • Foreign-vendor custom_extensions have no slot in the Metric View YAML; they are
    dropped on export with a warning.

Credits

Builds on the property-based round-trip testing approach from #152; @jackstein21 is
credited as a co-contributor.

…icks)

Bidirectional, offline converter between OSI semantic models and Databricks Unity
Catalog Metric Views (YAML v1.1), filling the DATABRICKS spoke already listed in
converters/index.md. Packaged like the dbt/gooddata spokes: pyproject.toml, an
osi_databricks package under src/, and a tests/ suite (example-based + Hypothesis
property-based round-trip; 73 tests). PyYAML is the only runtime dependency.

Co-authored-by: jackstein21 <82542300+jackstein21@users.noreply.github.com>
@Haoranli503 Haoranli503 changed the title Add Databricks Unity Catalog Metric View converter [OSI][Databricks] Add Databricks Unity Catalog Metric View converter Jun 29, 2026
@Haoranli503

Copy link
Copy Markdown
Author

closed, will be track as #3 in the new ossie repo

@Haoranli503 Haoranli503 closed this Jul 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant