issue #297 fixes — bundle template, compat, cli, demo, runtime

[park-peter]

Closes #297. Eight independent fixes; each section below is a separate commit.

What changed

1. DAB template — variables.yml validation warnings

Every variable in templates/dab/template/{{.bundle_name}}/resources/variables.yml.tmpl declared type: string or type: bool. The bundle CLI's variable schema only accepts type: complex, so every databricks bundle validate against a freshly-scaffolded bundle emitted 18 Warning: invalid value "string" for enum field. Valid values are [complex]. Dropped the type: line from all 18 declarations.

Also dropped the sync.exclude: block from databricks.yml.tmpl — the patterns .DS_Store, .vscode, .venv/ do not match any files in a freshly-scaffolded bundle, producing three additional "Pattern X does not match any files" warnings.

bundle-init --quickstart + bundle validate now returns Validation OK! with zero warnings.

2. compat/dlt_meta — runtime import deferral + pyspark/delta-spark bump

dataflow_pipeline.py and pipeline_writers.py do from pyspark import pipelines as dp, which is only available in pyspark >=4.1.0. setup.py DEV_REQUIREMENTS and the README pinned pyspark==3.5.5, which lacks pyspark.pipelines. The previous blanket try: ... except ImportError: pass in compat/dlt_meta/__init__.py then silently dropped every runtime re-export (DataflowPipeline, PipelineReaders, AppendFlowWriter, DLTSinkWriter, BronzeDataflowSpec, SilverDataflowSpec, OnboardDataflowspec), surfacing as cannot import name 'DataflowPipeline' from 'dlt_meta' with no hint at the pyspark version mismatch.

Bumped pyspark==3.5.5 → pyspark>=4.1.0 and delta-spark==3.0.0 → delta-spark>=4.0.0 (the old delta pin caps pyspark <3.6.0 and is incompatible with the bump). Split the compat re-exports: cli / install / config bind unconditionally; runtime symbols are wrapped in a narrow try/except ImportError that swaps in stubs raising a clear ImportError("requires pyspark>=4.1.0 ...") when the import fails. Expanded the sys.modules mock in tests/test_compat.py to cover pyspark, pyspark.sql.session, pyspark.sql.window, delta, and delta.tables so the file no longer requires a real install.

3. CLI — malformed onboarding_file_path job parameter

SDPMeta._get_onboarding_named_parameters built the named parameter as f"{cmd.uc_volume_path}/sdp_meta_conf/{cmd.onboarding_file_path}". By that point cmd.onboarding_file_path is the local absolute path (overwritten by update_ws_onboarding_paths to point at the rewritten onboarding.json), so the concatenation produced /Volumes/<cat>/<schema>/<vol>/sdp_meta_conf//Users/.../onboarding.json, which the onboarding job then failed to open. The launchers under demo/launch_*.py construct their own named_parameters dict and bypass this path, which is why no demo caught it. Use os.path.basename(cmd.onboarding_file_path) in both the UC and DBFS branches.

4. Demo — legacy import dlt in snapshot runner

demo/SDP_META_INTERACTIVE_DEMO.py builds the snapshot runner notebook as a string. The codebase migrated from import dlt to from pyspark import pipelines as dp (issue #274), but the inline snapshot_runner_content string still carried import dlt. The symbol was never referenced in the runner body — just stale. Dropped.

5. Demo (DAB template runner) — placeholder seed strip

demo/launch_dab_template_demo.py calls _strip_example_onboarding_entry to remove the template's seeded data_flow_id: "100" row after bundle-init. The strip was gated on scenario.name in _DELTA_SCENARIO_NAMES only. For the kafka and eventhub scenarios, the seed carries <your-kafka-host>:9092 / <your-eventhub-namespace> placeholders that the launcher's STAGE 5 sanity checks reject, so every --scenario kafka / --scenario eventhub run failed at STAGE 5 with flow data_flow_id='100' field source_details.kafka.bootstrap.servers is still the placeholder. Added _STRIP_EXAMPLE_SCENARIO_NAMES = _DELTA_SCENARIO_NAMES | {"kafka", "eventhub"} and gated the strip on the broader set. cloudfiles / cloudfiles_combined keep the seed — their placeholder-free source_path_dev validates fine.

6. Demo (DAB template runner) — duplicate sanity-check printing

stage_validate printed _sdp_meta_sanity_checks errors itself and then called bundle_validate, which prints the same list again under a different header. Compounded with #5 — every failing kafka/eventhub run dumped the same block twice. Dropped the launcher's local copy; bundle_validate owns the output.

7. Runtime — read_silver where-clause shadow

DataflowPipeline.read_silver had for where_clause in where_clause:, shadowing the outer list with the last clause string. Nothing downstream reads where_clause post-loop today, but the same logic was already implemented correctly in the private __apply_where_clause helper. Delegated read_silver's where-clause handling to __apply_where_clause.

8. Runtime — unused helper methods + tests

dataflow_pipeline.py defined _build_table_name, _get_source_table_info, _get_target_table_name, _create_dataframe_reader, _read_from_source, _apply_transformations — six methods whose only call sites were within the same orphan chain. read_silver / read_bronze / _get_target_table_info all kept their inline implementations. Deleted the six methods and the eight test_dataflow_pipeline.py tests that exercised them.

[park-peter]

@ravi-databricks
cli.py it's a bug fix. The onboarding_file_path job parameter was being built from the full local path instead of just the filename, so databricks labs sdp-meta onboard produced an unopenable /Volumes/.../sdp_meta_conf//Users/.../onboarding.json. The demos never caught it because the launchers build that parameter themselves. Switched to os.path.basename(...).

dataflow_pipeline.py two things here. First, the inline where-clause loop in read_silver now delegates to the existing __apply_where_clause helper instead of carrying its own copy (the inline copy reused its loop variable name, shadowing the outer list). Same behavior, one implementation. This is just a clean-up. Second part, I removed six private methods (_build_table_name, _get_source_table_info, _get_target_table_name, _create_dataframe_reader, _read_from_source, _apply_transformations) plus their tests. They had no caller in any production path, the only references were calls among themselves. The real read/write paths kept their own inline logic and never touched this cluster. No behavioral change. But let me know if this is an implementation you were meaning to continue to build, I can take the removal out of PR.

[CLAassistant]

All committers have signed the CLA.