[OSSIE][DATABRICKS] Add Databricks Unity Catalog Metric View converter#224
[OSSIE][DATABRICKS] Add Databricks Unity Catalog Metric View converter#224Haoranli503 wants to merge 1 commit into
Conversation
…icks) Bidirectional, offline converter between Apache Ossie semantic models and Databricks Unity Catalog Metric Views (YAML v1.1), filling the DATABRICKS spoke already listed in converters/README.md. Packaged like the sibling spokes: pyproject.toml (apache-ossie-databricks), an ossie_databricks package under src/, ASF license headers, and a tests/ suite (example-based + Hypothesis property-based round-trip; 74 tests). PyYAML is the only runtime dependency. Co-authored-by: jackstein21 <82542300+jackstein21@users.noreply.github.com>
|
RE @jbonofre 's review on #178:
Renaming either would be a bug, not a cleanup. 2, Duplicate dimension/measure name is now a |
Add Databricks Unity Catalog Metric View converter
Adds a bidirectional, offline converter between Apache Ossie semantic models and Databricks Unity Catalog Metric Views (YAML v1.1), filling the Databricks spoke in the hub-and-spoke architecture (
DATABRICKSis already a listed vendor inconverters/index.md). Validated against the live Databricks Metric View engine.What's included
ossie-databricks export): Apache Ossie semantic model -> Metric View (one factsource+ a nestedjoinstree, flatteneddimensions,measures).ossie-databricks import): Metric View -> Apache Ossie. Metric-View-only features (filter, window, format, rely, cardinality, parameters, materialization) are preserved incustom_extensions[DATABRICKS], soMV -> Apache Ossie -> MVis lossless.ossie-databricksCLI and a string-in / string-out Python API.Design highlights
custom_extensions[DATABRICKS]stash -- the approach the converter guide recommends.joinstree; a dataset reached by multiple paths (a diamond) is fanned out into one aliased join per path; cyclic graphs are rejected.many_to_one/one_to_manyis derived from the relationshipfrom/todirection relative to a selectable grain (--source). Multiple facts are supported only narrowly -- when they share a conformed dimension (named as the source); a general galaxy schema (facts each with their own dimensions) or facts sharing no dimension can't map to one Metric View and are rejected with a clear error, never silently mis-converted.primary_key/unique_keyswhose columns cover a join key becomesrely: {at_most_one_match: true}, and vice versa.ConversionError; anything dropped (a foreign-vendor extension, an Apache-Ossie-only annotation, a non-DATABRICKS/ANSI_SQL dialect) emits a warning naming the object.Conventions followed
converters/databricks/, packaged like the dbt/gooddata spokes:pyproject.toml(hatchling, Apache-2.0),src/ossie_databricks/,tests/.DATABRICKS, falls back toANSI_SQL.Testing
Example-based unit tests plus Hypothesis property-based round-trip tests in both directions, with a normalized TPC-DS fixture as a baseline.
Notes
custom_extensionshave no slot in the Metric View YAML; they are dropped on export with a warning.Credits
@jackstein21 is credited as a co-contributor (see the commit's
Co-authored-bytrailer), including the property-based round-trip testing approach.Related Issues
Will close #178, Ossie <-> Databricks converters will be tracked here.
Checklist
Specification
core-spec/and follow the existing structureOntology
ontology/are consistent with spec changesConverters
converters/is updated to reflect spec or ontology changesValidation
validation/are updated if the spec changedDocumentation
docs/is updated to reflect any user-facing changesCONTRIBUTING.mdis updated if the contribution process changedExamples
examples/are added or updated for any new spec constructs or converter supportTests
pytest/ CI green)Compliance