Skip to content

[OSSIE][DATABRICKS] Add Databricks Unity Catalog Metric View converter#224

Open
Haoranli503 wants to merge 1 commit into
apache:mainfrom
Haoranli503:databricks-converter-v2
Open

[OSSIE][DATABRICKS] Add Databricks Unity Catalog Metric View converter#224
Haoranli503 wants to merge 1 commit into
apache:mainfrom
Haoranli503:databricks-converter-v2

Conversation

@Haoranli503

Copy link
Copy Markdown

Add Databricks Unity Catalog Metric View converter

Adds a bidirectional, offline converter between Apache Ossie semantic models and Databricks Unity Catalog Metric Views (YAML v1.1), filling the Databricks spoke in the hub-and-spoke architecture (DATABRICKS is already a listed vendor in converters/index.md). Validated against the live Databricks Metric View engine.

What's included

  • Export (ossie-databricks export): Apache Ossie semantic model -> Metric View (one fact source + a nested joins tree, flattened dimensions, measures).
  • Import (ossie-databricks import): Metric View -> Apache Ossie. Metric-View-only features (filter, window, format, rely, cardinality, parameters, materialization) are preserved in custom_extensions[DATABRICKS], so MV -> Apache Ossie -> MV is lossless.
  • A unified ossie-databricks CLI and a string-in / string-out Python API.
  • A README with a mapping table and limitations.

Design highlights

  • Round-trip fidelity via the custom_extensions[DATABRICKS] stash -- the approach the converter guide recommends.
  • Join tree, not flat joins. The Apache Ossie relationship graph is reassembled into the Metric View's nested fact + joins tree; a dataset reached by multiple paths (a diamond) is fanned out into one aliased join per path; cyclic graphs are rejected.
  • Cardinality from direction. many_to_one / one_to_many is derived from the relationship from/to direction relative to a selectable grain (--source). Multiple facts are supported only narrowly -- when they share a conformed dimension (named as the source); a general galaxy schema (facts each with their own dimensions) or facts sharing no dimension can't map to one Metric View and are rejected with a clear error, never silently mis-converted.
  • Key bridge. A primary_key/unique_keys whose columns cover a join key becomes rely: {at_most_one_match: true}, and vice versa.
  • On export, joined-table columns are alias-qualified in dimension/measure expressions; fact columns in measures are emitted bare (the DBR idiom).
  • Fails loudly, never silently. A join condition an Apache Ossie relationship can't represent (a non-equi or cross join), a cyclic or ambiguous-grain graph, and an unsupported version are rejected with a clear ConversionError; anything dropped (a foreign-vendor extension, an Apache-Ossie-only annotation, a non-DATABRICKS/ANSI_SQL dialect) emits a warning naming the object.

Conventions followed

  • Lives at converters/databricks/, packaged like the dbt/gooddata spokes: pyproject.toml (hatchling, Apache-2.0), src/ossie_databricks/, tests/.
  • Every source file carries the ASF license header, matching the sibling spokes.
  • On export, dialect selection prefers DATABRICKS, falls back to ANSI_SQL.
  • The only runtime dependency is PyYAML; Python 3.11+.

Testing

cd converters/databricks
pip install -e . && pip install pytest hypothesis
python -m pytest tests/

Example-based unit tests plus Hypothesis property-based round-trip tests in both directions, with a normalized TPC-DS fixture as a baseline.

Notes

  • Targets Databricks Unity Catalog Metric View YAML v1.1.
  • Foreign-vendor custom_extensions have no slot in the Metric View YAML; they are dropped on export with a warning.

Credits

@jackstein21 is credited as a co-contributor (see the commit's Co-authored-by trailer), including the property-based round-trip testing approach.

Related Issues

Will close #178, Ossie <-> Databricks converters will be tracked here.

Checklist

Specification

  • Spec changes are included in core-spec/ and follow the existing structure
  • Spec changes have been discussed on the mailing list or in a linked issue
  • Breaking changes to the spec are clearly called out in the summary

Ontology

  • Ontology changes in ontology/ are consistent with spec changes
  • New or modified terms are defined and documented

Converters

  • Converter logic in converters/ is updated to reflect spec or ontology changes
  • New converters include tests under the converter's test directory

Validation

  • Validation rules in validation/ are updated if the spec changed
  • New validation cases are covered by tests

Documentation

  • docs/ is updated to reflect any user-facing changes
  • New features or behaviors are documented with examples where appropriate
  • CONTRIBUTING.md is updated if the contribution process changed

Examples

  • examples/ are added or updated for any new spec constructs or converter support

Tests

  • All existing tests pass (pytest / CI green)
  • New functionality is covered by tests

Compliance

  • ASF license headers are present on all new source files
  • No third-party dependencies are added without PMC/IPMC approval

…icks)

Bidirectional, offline converter between Apache Ossie semantic models and Databricks
Unity Catalog Metric Views (YAML v1.1), filling the DATABRICKS spoke already listed in
converters/README.md. Packaged like the sibling spokes: pyproject.toml
(apache-ossie-databricks), an ossie_databricks package under src/, ASF license headers,
and a tests/ suite (example-based + Hypothesis property-based round-trip; 74 tests).
PyYAML is the only runtime dependency.

Co-authored-by: jackstein21 <82542300+jackstein21@users.noreply.github.com>
@Haoranli503

Copy link
Copy Markdown
Author

RE @jbonofre 's review on #178:
1, Renamed OSI to Ossie, done. Two "OSI"s in the ossie converter I left as-is on purpose:

$schema=.../core-spec/osi-schema.json — points at a real file in upstream apache/ossie (never renamed; all sibling spokes use this exact name). Renaming it breaks the schema link.

"License :: OSI Approved :: Apache Software License" — this OSI = Open Source Initiative, not Ossie. Fixed PyPI classifier, only valid spelling for Apache-2.0; editing it fails packaging.

Renaming either would be a bug, not a cleanup.

2, Duplicate dimension/measure name is now a ConversionError, done. Previously a name collision (a dimension/measure sharing a name, case-insensitively) only emitted a warning and still wrote the duplicate to the output. It now raises a ConversionError naming the offending field, consistent with the rest of the converter — the user is told exactly what to rename before handing a broken file to Databricks. Added tests for both the duplicate-dimension and measure-collides-with-dimension cases.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant