bundle: record deployment history in DMS after approval by shreyas-goenka · Pull Request #5386 · databricks/cli

shreyas-goenka · 2026-05-31T21:28:54Z

Records each approved deploy/destroy as a version with the Deployment Metadata Service (DMS), gated by experimental.record_deployment_history and the direct engine.

Stacked on #5667 (the lineage/state-layer foundation). This PR is the consumer.

Key behavior: the version is created only after the plan is approved — a cancelled or declined deploy/destroy records nothing, so there are no empty/abandoned versions for operations that never ran.

libs/dms: Recorder with CreateVersion / CompleteVersion. The deployment ID is the state lineage (GetOrInitLineage), so a bundle deployment maps one-to-one to a DMS deployment record. GetDeployment first, CreateDeployment only when missing, then the next version; heartbeat keeps the lease alive; CompleteVersion records success/failure and, for destroy, deletes the deployment on success. Independent of bundle/deploy/lock.
phases: newDeploymentRecorder builds it from the bundle (nil unless the flag is set and the engine is direct); deploy/destroy create the version inside the approved branch (after UpgradeToWrite, so the lineage is already in the WAL) and complete it in the deferred lock release.
libs/testserver: in-memory DMS handlers under /api/2.0/bundle/....
acceptance/bundle/dms: deploy/redeploy/destroy record versions while holding the file lock; redeploy after deleting .databricks recovers the lineage from remote state; enabling the flag after a plain deploy creates a new deployment; a declined destroy records no version and does not delete the deployment.

Follow-ups: inherit the state serial as the version id + previous_version_id staleness check (needs the SDK regen from universe #2061768).

eng-dev-ecosystem-bot · 2026-05-31T22:17:38Z

Integration test report

Commit: adfb856

Run: 27834001471

	Env	💚RECOVERED	🙈SKIP	✅pass	🙈skip	Time
💚	aws linux	7	13	265	1014	6:48
💚	aws windows	7	13	267	1012	7:53
💚	aws-ucws linux	7	13	361	928	7:08
💚	aws-ucws windows	7	13	363	926	9:37
💚	azure linux	1	15	268	1012	5:45
💚	azure windows	1	15	270	1010	8:12
💚	azure-ucws linux	1	15	366	924	8:09
💚	azure-ucws windows	1	15	368	922	9:02
💚	gcp linux	1	15	264	1015	7:04
💚	gcp windows	1	15	266	1013	9:09

20 interesting tests: 13 SKIP, 7 RECOVERED

	Test Name	aws linux	aws windows	aws-ucws linux	aws-ucws windows	azure linux	azure windows	azure-ucws linux	azure-ucws windows	gcp linux	gcp windows
💚	TestAccept	💚R	💚R	💚R	💚R	💚R	💚R	💚R	💚R	💚R	💚R
🙈	TestAccept/bundle/invariant/no_drift	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🙈	TestAccept/bundle/resources/permissions	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
💚	TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions	💚R	💚R	💚R	💚R	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
💚	TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=direct	💚R	💚R	💚R	💚R
💚	TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=terraform	💚R	💚R	💚R	💚R
💚	TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions	💚R	💚R	💚R	💚R	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
💚	TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=direct	💚R	💚R	💚R	💚R
💚	TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=terraform	💚R	💚R	💚R	💚R
🙈	TestAccept/bundle/resources/postgres_branches/basic	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🙈	TestAccept/bundle/resources/postgres_branches/recreate	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🙈	TestAccept/bundle/resources/postgres_branches/replace_existing	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🙈	TestAccept/bundle/resources/postgres_branches/update_protected	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🙈	TestAccept/bundle/resources/postgres_branches/without_branch_id	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🙈	TestAccept/bundle/resources/postgres_endpoints/basic	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🙈	TestAccept/bundle/resources/postgres_projects/update_display_name	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🙈	TestAccept/bundle/resources/synced_database_tables/basic	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🙈	TestAccept/bundle/resources/vector_search_endpoints/drift/recreated_same_name	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🙈	TestAccept/bundle/resources/vector_search_indexes/recreate/embedding_dimension	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🙈	TestAccept/ssh/connection	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S

Top 26 slowest tests (at least 2 minutes):

duration	env	testname
4:36	gcp windows	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
4:32	gcp windows	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
4:22	gcp linux	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
3:56	gcp linux	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
3:38	azure-ucws linux	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
3:33	azure windows	TestAccept
3:29	azure linux	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
3:29	azure-ucws linux	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
3:23	aws-ucws windows	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
3:22	azure-ucws windows	TestAccept
3:19	aws windows	TestAccept
3:15	gcp windows	TestAccept
3:14	aws-ucws windows	TestAccept
3:13	aws windows	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
3:07	azure-ucws windows	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:54	aws-ucws linux	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:50	azure windows	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:49	aws linux	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:46	aws windows	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:45	azure linux	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:44	aws linux	TestSecretsPutSecretStringValue
2:38	aws-ucws linux	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:31	aws linux	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:25	aws-ucws windows	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:25	azure windows	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:22	azure-ucws windows	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct

shreyas-goenka · 2026-06-01T17:32:44Z

+
+	// The server validates that versionID equals last_version_id + 1 and returns
+	// ABORTED otherwise (e.g. a concurrent deploy already created this version).
+	version, versionErr := svc.CreateVersion(ctx, sdkbundle.CreateVersionRequest{


this will not work well when the plan is serialized and potentially outdated (because we do not use serial here)

Will be fixed in a followup.

shreyas-goenka · 2026-06-15T11:40:24Z

We can remove the traditional file based lock in a followup. Not necessary for now / preview.

andrewnester · 2026-06-15T12:06:12Z

+		return fmt.Errorf("failed to parse version ID %q: %w", versionID, err)
+	}
+	r.versionNum = versionNum
+	r.stopHeartbeat = startHeartbeat(ctx, r.svc, r.deploymentID, versionID)


We call CreateVersion twice: in deploy and destroy and seem to start heatbeat twice, shall we have only 1 instance of heartbeat?

Can you clarify? Those are independent code paths and both need a heartbeat right?

Ah right, my bad, indeed a separate processes

andrewnester · 2026-06-16T12:35:45Z

 		bundle.ApplyContext(ctx, b, lock.Release(lock.GoalDeploy))
 	}()
+	if err := recorder.CreateVersion(ctx); err != nil {


When we remove file lock later on, how do we ensure that there's no race condition creating multiple versions? We don't seem to do any synchronisation / locking here now, is it part of follow up?

The CreateVersion implicitly has locking semantics. Only one client can have a "live" version that is in progress at a time.

The server-side version counter is the synchronization: CreateVersion only succeeds when the requested version is last_version_id + 1, otherwise it returns ABORTED (409). So even without the file lock, two concurrent deploys racing to create the next version would have one win and the other get ABORTED. Surfacing that as a clean user-facing error (retry/serial handling) is the follow-up I noted.

Got it, make sense. Could you add an acceptance test for it though to make sure this behaviour is recorded?

pietern · 2026-06-17T09:04:51Z

+	if db.Data.Lineage == "" {
+		db.Data.Lineage = uuid.New().String()
+	}
+	return db.Data.Lineage


This is initialized in Open() below.

For direct yes. In case of DMS this is initialized at pla n time here instead. I did not want to touch direct deployment code paths.

Records each approved deploy/destroy as a version with the Deployment Metadata Service (DMS), gated by experimental.record_deployment_history and the direct engine. The version is created only after the plan is approved — a cancelled or declined deploy/destroy records nothing, so there are no empty/abandoned versions for operations that never ran. - libs/dms: Recorder with CreateVersion / CompleteVersion. The deployment ID is the state lineage (from GetOrInitLineage), so a bundle deployment maps one-to-one to a DMS deployment record. GetDeployment first, CreateDeployment only when missing, then create the next version; heartbeat keeps the version's lease alive; CompleteVersion records success/failure and, for destroy, deletes the deployment record on success. Independent of bundle/lock. - phases: newDeploymentRecorder builds the recorder from the bundle (nil unless the flag is set and the engine is direct); deploy/destroy create the version inside the approved branch (after UpgradeToWrite, so the lineage is already in the WAL) and complete it in the deferred lock release. - libs/testserver: in-memory DMS handlers under /api/2.0/bundle/... - acceptance/bundle/dms: deploy/redeploy/destroy record versions and hold the file lock; redeploy after deleting .databricks recovers the lineage from remote state; enabling the flag after a plain deploy creates a new deployment; a declined destroy records no version and does not delete the deployment. Co-authored-by: Shreyas Goenka <shreyas.goenka@databricks.com>

shreyas-goenka temporarily deployed to test-trigger-is May 31, 2026 21:29 — with GitHub Actions Inactive

shreyas-goenka changed the title ~~bundle/deploy/lock: add DMS-backed DeploymentManager implementation~~ bundle/deploy/lock: add DMS-backed DeploymentManager implementation using SDK bundle client May 31, 2026

shreyas-goenka temporarily deployed to test-trigger-is May 31, 2026 21:41 — with GitHub Actions Inactive

shreyas-goenka force-pushed the shreyas-goenka/bundle-dms-implementation branch from 98dc0c7 to 4a4382f Compare June 1, 2026 12:33

shreyas-goenka temporarily deployed to test-trigger-is June 1, 2026 12:34 — with GitHub Actions Inactive

shreyas-goenka force-pushed the shreyas-goenka/bundle-dms-implementation branch from 4a4382f to de9adfd Compare June 1, 2026 12:35

shreyas-goenka mentioned this pull request Jun 1, 2026

bundle/deploy/lock: wire DMS into NewDeploymentLock via experimental.record_deployment_history #5395

Closed

5 tasks

shreyas-goenka temporarily deployed to test-trigger-is June 1, 2026 12:36 — with GitHub Actions Inactive

shreyas-goenka force-pushed the shreyas-goenka/bundle-dms-implementation branch from de9adfd to bb16fcd Compare June 1, 2026 15:11

shreyas-goenka temporarily deployed to test-trigger-is June 1, 2026 15:12 — with GitHub Actions Inactive

shreyas-goenka force-pushed the shreyas-goenka/bundle-dms-implementation branch from bb16fcd to bdc7ba2 Compare June 1, 2026 15:16

shreyas-goenka temporarily deployed to test-trigger-is June 1, 2026 15:17 — with GitHub Actions Inactive

shreyas-goenka force-pushed the shreyas-goenka/bundle-dms-implementation branch from bdc7ba2 to 1e8ba45 Compare June 1, 2026 15:23

shreyas-goenka changed the title ~~bundle/deploy/lock: add DMS-backed DeploymentManager implementation using SDK bundle client~~ bundle/deploy/lock: record deployment history in DMS behind experimental.record_deployment_history Jun 1, 2026

shreyas-goenka temporarily deployed to test-trigger-is June 1, 2026 15:23 — with GitHub Actions Inactive

shreyas-goenka mentioned this pull request Jun 1, 2026

Enable reading deployment state from DMS #5355

Open

shreyas-goenka force-pushed the shreyas-goenka/bundle-lock-abstraction branch from ff910cd to ee860e8 Compare June 1, 2026 15:31

shreyas-goenka force-pushed the shreyas-goenka/bundle-dms-implementation branch from 1e8ba45 to 76bf017 Compare June 1, 2026 15:31

shreyas-goenka temporarily deployed to test-trigger-is June 1, 2026 15:32 — with GitHub Actions Inactive

shreyas-goenka temporarily deployed to test-trigger-is June 1, 2026 15:38 — with GitHub Actions Inactive

shreyas-goenka temporarily deployed to test-trigger-is June 1, 2026 16:47 — with GitHub Actions Inactive

shreyas-goenka force-pushed the shreyas-goenka/bundle-dms-implementation branch from 063dad9 to 607bdd0 Compare June 1, 2026 16:54

shreyas-goenka temporarily deployed to test-trigger-is June 1, 2026 16:55 — with GitHub Actions Inactive

shreyas-goenka requested a review from denik June 1, 2026 17:02

shreyas-goenka commented Jun 1, 2026

View reviewed changes

shreyas-goenka requested a review from pietern June 15, 2026 11:39

shreyas-goenka assigned andrewnester Jun 15, 2026

shreyas-goenka requested a review from andrewnester June 15, 2026 11:39

shreyas-goenka unassigned andrewnester Jun 15, 2026

andrewnester reviewed Jun 15, 2026

View reviewed changes

shreyas-goenka force-pushed the shreyas-goenka/bundle-dms-implementation branch from 607bdd0 to 98f4444 Compare June 15, 2026 13:40

shreyas-goenka temporarily deployed to test-trigger-is June 15, 2026 13:41 — with GitHub Actions Inactive

shreyas-goenka requested a review from andrewnester June 16, 2026 11:42

andrewnester approved these changes Jun 16, 2026

View reviewed changes

andrewnester reviewed Jun 16, 2026

View reviewed changes

andrewnester self-requested a review June 16, 2026 12:36

shreyas-goenka force-pushed the shreyas-goenka/bundle-dms-implementation branch from 98f4444 to 87c99e8 Compare June 17, 2026 00:27

shreyas-goenka temporarily deployed to test-trigger-is June 17, 2026 00:28 — with GitHub Actions Inactive

shreyas-goenka force-pushed the shreyas-goenka/bundle-dms-implementation branch from 87c99e8 to 9a7feb1 Compare June 17, 2026 00:28

shreyas-goenka temporarily deployed to test-trigger-is June 17, 2026 00:29 — with GitHub Actions Inactive

andrewnester approved these changes Jun 17, 2026

View reviewed changes

pietern requested changes Jun 17, 2026

View reviewed changes

shreyas-goenka force-pushed the shreyas-goenka/bundle-dms-implementation branch from 9a7feb1 to 35f5a64 Compare June 19, 2026 07:43

shreyas-goenka mentioned this pull request Jun 19, 2026

bundle/direct/dstate: add GetOrInitLineage, the single lineage init point #5667

Draft

Conversation

shreyas-goenka commented May 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eng-dev-ecosystem-bot commented May 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Integration test report

Uh oh!

shreyas-goenka Jun 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shreyas-goenka commented Jun 15, 2026

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

shreyas-goenka commented May 31, 2026 •

edited

Loading

eng-dev-ecosystem-bot commented May 31, 2026 •

edited

Loading

shreyas-goenka Jun 1, 2026 •

edited

Loading