Skip to content

[Ossie][Databricks] Add Databricks Unity Catalog Metric View converter#3

Closed
Haoranli503 wants to merge 1 commit into
apache:mainfrom
Haoranli503:add-databricks-converter
Closed

[Ossie][Databricks] Add Databricks Unity Catalog Metric View converter#3
Haoranli503 wants to merge 1 commit into
apache:mainfrom
Haoranli503:add-databricks-converter

Conversation

@Haoranli503

@Haoranli503 Haoranli503 commented Jul 7, 2026

Copy link
Copy Markdown

Add Databricks Unity Catalog Metric View converter

Adds a bidirectional, offline converter between Apache Ossie semantic models and Databricks Unity Catalog Metric Views (YAML v1.1), filling the Databricks spoke in the hub-and-spoke architecture (DATABRICKS is already a listed vendor in converters/index.md). Validated against the live Databricks Metric View engine.

What's included

  • Export (ossie-databricks export): Apache Ossie semantic model -> Metric View (one fact source + a nested joins tree, flattened dimensions, measures).
  • Import (ossie-databricks import): Metric View -> Apache Ossie. Metric-View-only features (filter, window, format, rely, cardinality, parameters, materialization) are preserved in custom_extensions[DATABRICKS], so MV -> Apache Ossie -> MV is lossless.
  • A unified ossie-databricks CLI and a string-in / string-out Python API.
  • A README with a mapping table and limitations.

Design highlights

  • Round-trip fidelity via the custom_extensions[DATABRICKS] stash -- the approach the converter guide recommends.
  • Join tree, not flat joins. The Apache Ossie relationship graph is reassembled into the Metric View's nested fact + joins tree; a dataset reached by multiple paths (a diamond) is fanned out into one aliased join per path; cyclic graphs are rejected.
  • Cardinality from direction. many_to_one / one_to_many is derived from the relationship from/to direction relative to a selectable grain (--source). Multiple facts are supported only narrowly -- when they share a conformed dimension (named as the source); a general galaxy schema (facts each with their own dimensions) or facts sharing no dimension can't map to one Metric View and are rejected with a clear error, never silently mis-converted.
  • Key bridge. A primary_key/unique_keys whose columns cover a join key becomes rely: {at_most_one_match: true}, and vice versa.
  • On export, joined-table columns are alias-qualified in dimension/measure expressions; fact columns in measures are emitted bare (the DBR idiom).
  • Fails loudly, never silently. A join condition an Apache Ossie relationship can't represent (a non-equi or cross join), a cyclic or ambiguous-grain graph, and an unsupported version are rejected with a clear ConversionError; anything dropped (a foreign-vendor extension, an Apache-Ossie-only annotation, a non-DATABRICKS/ANSI_SQL dialect) emits a warning naming the object.

Conventions followed

  • Lives at converters/databricks/, packaged like the dbt/gooddata spokes: pyproject.toml (hatchling, Apache-2.0), src/ossie_databricks/, tests/.
  • Every source file carries the ASF license header, matching the sibling spokes.
  • On export, dialect selection prefers DATABRICKS, falls back to ANSI_SQL.
  • The only runtime dependency is PyYAML; Python 3.11+.

Testing

cd converters/databricks
pip install -e . && pip install pytest hypothesis
python -m pytest tests/

Example-based unit tests plus Hypothesis property-based round-trip tests in both directions, with a normalized TPC-DS fixture as a baseline.

Notes

  • Targets Databricks Unity Catalog Metric View YAML v1.1.
  • Foreign-vendor custom_extensions have no slot in the Metric View YAML; they are dropped on export with a warning.

Credits

@jackstein21 is credited as a co-contributor (see the commit's Co-authored-by trailer), including the property-based round-trip testing approach.

…icks)

Bidirectional, offline converter between Apache Ossie semantic models and Databricks
Unity Catalog Metric Views (YAML v1.1), filling the DATABRICKS spoke already listed in
converters/index.md. Packaged like the dbt/gooddata spokes: pyproject.toml
(apache-ossie-databricks), an ossie_databricks package under src/, ASF license headers,
and a tests/ suite (example-based + Hypothesis property-based round-trip; 73 tests).
PyYAML is the only runtime dependency.
@jbonofre

Copy link
Copy Markdown
Member

Moved to apache/ossie#178

@jbonofre jbonofre closed this Jul 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants