bundle: don't set edition/channel defaults for ingestion pipelines#5418
Open
shreyas-goenka wants to merge 5 commits into
Open
bundle: don't set edition/channel defaults for ingestion pipelines#5418shreyas-goenka wants to merge 5 commits into
shreyas-goenka wants to merge 5 commits into
Conversation
Ingestion pipelines (those with `ingestion_definition`) use serverless compute by default. The Databricks API rejects `edition` and `channel` fields as "cluster settings" when a pipeline runs on serverless compute. Previously, DABs unconditionally set `edition: ADVANCED` and `channel: CURRENT` as defaults for all pipeline types. This caused creation of LakeFlow Connect ingestion pipelines to fail with: "cannot create pipeline: You cannot provide cluster settings when using serverless compute" The fix conditionally applies these defaults only to pipelines without `ingestion_definition`. The test `change-ingestion-definition` is restructured to use per-engine output files since the TF provider independently adds these defaults for all pipeline types. Co-authored-by: Shreyas Goenka <shreyas.goenka@databricks.com>
shreyas-goenka
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
shreyas-goenka
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
Collaborator
Integration test reportCommit: a8f3a47
26 interesting tests: 15 SKIP, 7 KNOWN, 2 FAIL, 2 flaky
Top 36 slowest tests (at least 2 minutes):
|
Replace the dyn.Value pattern walk with a direct loop over b.Config.Resources.Pipelines to conditionally apply edition/channel defaults. Also remove the now-unused setPipelineDefault helper. Add acceptance/bundle/resources/pipelines/ingestion-defaults, a cloud test (direct engine) that deploys an ingestion pipeline and asserts edition and channel are absent from the POST /api/2.0/pipelines request body. The deploy fails with "connection not found" — not "cluster settings" — confirming the fix. Co-authored-by: Shreyas Goenka <shreyas.goenka@databricks.com>
shreyas-goenka
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
shreyas-goenka
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
shreyas-goenka
changed the title
Extends the ingestion-defaults acceptance test to run both direct and terraform engines, showing the contrast: - Direct engine: edition/channel absent from POST (DABs fix working) - Terraform engine: edition/channel present (provider-level SetDefault, companion fix in terraform-provider-databricks#5783) Badness clause documents the known terraform engine gap. Co-authored-by: Shreyas Goenka <shreyas.goenka@databricks.com>
shreyas-goenka
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
shreyas-goenka
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
Replaces the error-capture test with a real end-to-end deployment on the azure cloud environment using a MySQL ingestion connection. The test: - Deploys an ingestion pipeline via the direct engine - Asserts deployment succeeds (no "cluster settings" error) - Verifies edition and channel are absent from the POST request body - Destroys the pipeline on exit Also confirms classic compute (serverless: false) works the same way. The Terraform-engine gap (provider-level SetDefault still injects edition/channel) is documented in Badness and fixed in terraform-provider-databricks#5783. Co-authored-by: Shreyas Goenka <shreyas.goenka@databricks.com>
shreyas-goenka
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
shreyas-goenka
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
…ogrammatically The change-ingestion-definition test now records create and recreate requests per engine, which documents the terraform provider injecting edition/channel for ingestion pipelines (fix: terraform-provider-databricks#5783). The destroy-phase DELETE is identical across engines, so it is recorded once. Also build the pipeline defaults pattern with dyn.NewPattern instead of string concatenation, which removes the pattern parse and its error paths. Co-authored-by: Isaac
shreyas-goenka
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
shreyas-goenka
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
Contributor
Approval status: pending
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ingestion pipelines (
ingestion_definition) use serverless compute. Settingeditionorchannelon them causes:This PR skips those defaults for ingestion pipelines. Regular DLT pipelines are unaffected. Gateway (CDC) pipelines run on classic compute where these fields are accepted, so they intentionally keep the defaults — mirroring the companion provider fix.
Note: The Terraform provider applies the same defaults unconditionally at the provider level (
resource_pipeline.go:253), so deploys through the terraform engine still fail until terraform-provider-databricks#5783 ships. The cloud acceptance test records both engines, with aBadnessentry tracking the terraform failure.Tests:
bundle/validate/pipeline_defaults(local): edition/channel defaulted for regular pipelines, absent for ingestion pipelines.bundle/resources/pipelines/recreate-keys/change-ingestion-definition(local): per-engine create/recreate request recordings document the provider-injected edition/channel.bundle/resources/pipelines/ingestion-defaults(cloud, azure): real deploy succeeds on the direct engine and fails on the terraform engine.