Immutable folder support in DABs by andrewnester · Pull Request #5254 · databricks/cli

andrewnester · 2026-05-15T09:56:18Z

Changes

Added support for deploying bundles to immutable folders in the workspace

Enabled by using

bundle:
  deployment: 
    immutable_folder: true

Why

Tests

Added an acceptance tests

eng-dev-ecosystem-bot · 2026-05-28T11:17:06Z

Integration test report

Commit: 9a6c898

Run: 27775865252

	Env	🪲BUG	🟨KNOWN	💚RECOVERED	🙈SKIP	✅pass	🙈skip	Time
🪲	aws linux	3	7		13	264	1013	7:44
🪲	aws windows	3	7		13	266	1011	13:29
🪲	aws-ucws linux	3	1	6	13	360	927	8:04
🪲	aws-ucws windows	3	1	6	13	362	925	12:33
🪲	azure linux	3	1		15	267	1011	7:04
🪲	azure windows	3	1		15	269	1009	11:06
🪲	azure-ucws linux	3	1		15	365	923	8:52
🪲	azure-ucws windows	3	1		15	367	921	12:04
🪲	gcp linux	3	1		15	263	1014	7:30
🪲	gcp windows	3	1		15	265	1012	11:00

23 interesting tests: 13 SKIP, 7 KNOWN, 3 BUG

	Test Name	aws linux	aws windows	aws-ucws linux	aws-ucws windows	azure linux	azure windows	azure-ucws linux	azure-ucws windows	gcp linux	gcp windows
🟨	TestAccept	🟨K	🟨K	🟨K	🟨K	🟨K	🟨K	🟨K	🟨K	🟨K	🟨K
🪲	TestAccept/bundle/deploy/immutable	🪲B	🪲B	🪲B	🪲B	🪲B	🪲B	🪲B	🪲B	🪲B	🪲B
🪲	TestAccept/bundle/deploy/immutable/DATABRICKS_BUNDLE_ENGINE=direct	🪲B	🪲B	🪲B	🪲B	🪲B	🪲B	🪲B	🪲B	🪲B	🪲B
🪲	TestAccept/bundle/deploy/immutable/DATABRICKS_BUNDLE_ENGINE=terraform	🪲B	🪲B	🪲B	🪲B	🪲B	🪲B	🪲B	🪲B	🪲B	🪲B
🙈	TestAccept/bundle/invariant/no_drift	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🙈	TestAccept/bundle/resources/permissions	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🟨	TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions	🟨K	🟨K	💚R	💚R	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🟨	TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=direct	🟨K	🟨K	💚R	💚R
🟨	TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=terraform	🟨K	🟨K	💚R	💚R
🟨	TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions	🟨K	🟨K	💚R	💚R	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🟨	TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=direct	🟨K	🟨K	💚R	💚R
🟨	TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=terraform	🟨K	🟨K	💚R	💚R
🙈	TestAccept/bundle/resources/postgres_branches/basic	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🙈	TestAccept/bundle/resources/postgres_branches/recreate	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🙈	TestAccept/bundle/resources/postgres_branches/replace_existing	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🙈	TestAccept/bundle/resources/postgres_branches/update_protected	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🙈	TestAccept/bundle/resources/postgres_branches/without_branch_id	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🙈	TestAccept/bundle/resources/postgres_endpoints/basic	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🙈	TestAccept/bundle/resources/postgres_projects/update_display_name	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🙈	TestAccept/bundle/resources/synced_database_tables/basic	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🙈	TestAccept/bundle/resources/vector_search_endpoints/drift/recreated_same_name	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🙈	TestAccept/bundle/resources/vector_search_indexes/recreate/embedding_dimension	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
🙈	TestAccept/ssh/connection	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S

Top 20 slowest tests (at least 2 minutes):

duration	env	testname
5:16	gcp windows	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
4:57	gcp linux	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
4:32	gcp linux	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
4:09	gcp windows	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
3:47	azure-ucws linux	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
3:38	azure-ucws windows	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
3:18	aws windows	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
3:16	aws-ucws windows	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
3:05	azure windows	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
3:01	aws-ucws windows	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:54	azure-ucws windows	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:54	azure linux	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:53	azure windows	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:48	aws-ucws linux	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:46	aws linux	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:39	aws linux	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:32	azure-ucws linux	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:31	azure linux	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:28	aws windows	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:15	aws-ucws linux	TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct

github-actions · 2026-06-01T11:35:34Z

Approval status: pending

`/acceptance/bundle/` - needs approval

23 files changed
Suggested: @denik
Also eligible: @pietern, @janniklasrose, @shreyas-goenka, @lennartkats-db, @anton-107

`/bundle/` - needs approval

19 files changed
Suggested: @denik
Also eligible: @pietern, @janniklasrose, @shreyas-goenka, @lennartkats-db, @anton-107

`/cmd/bundle/` - needs approval

Files: cmd/bundle/utils/process.go
Suggested: @denik
Also eligible: @pietern, @janniklasrose, @shreyas-goenka, @lennartkats-db, @anton-107

`/libs/sync/` - needs approval

Files: libs/sync/sync.go
Suggested: @simonfaltum
Also eligible: @tanmay-db, @Divyansh-db, @renaudhartert-db, @parthban-db, @hectorcast-db, @tejaskochar-db, @mihaimitrea-db, @chrisst, @rauchy

General files (require maintainer)

Files: libs/testserver/handlers.go
Based on git history:

@denik -- recent work in bundle/phases/, bundle/config/, cmd/bundle/utils/

_{Any maintainer (@anton-107, @denik, @pietern, @shreyas-goenka, @simonfaltum, @renaudhartert-db) can approve all areas.

See OWNERS for ownership rules.}

shreyas-goenka

Minor comments - other than the bit where we use metadata.json

shreyas-goenka · 2026-06-02T16:28:08Z

-	uploadLibraries(ctx, b, libs)
+	if b.Config.Bundle.Immutable {
+		// Upload all source files and built artifacts as a single immutable snapshot.
+		// The API assigns a content-addressed path, so workspace.snapshot_path (and


should we add a check explicitly disallowing workspace.snapshot_path references? We don't want users to accidentally rely on it and ensure that this is internal only.

shreyas-goenka · 2026-06-02T16:30:01Z

@@ -15,6 +15,9 @@ type Bundle struct {

 type Workspace struct {
 	FilePath string `json:"file_path"`
+	// SnapshotPath is the workspace path of the immutable snapshot uploaded
+	// during deployment. Only populated for bundles with bundle.immutable = true.
+	SnapshotPath string `json:"snapshot_path,omitempty"`


Will the UI use the immutable snapshot path? In that case we'll need to add it to DMS as well.

shreyas-goenka · 2026-06-02T16:30:57Z

+
+	cmdio.LogString(ctx, "Uploading immutable bundle snapshot...")
+
+	zipContent, err := BundleZip(ctx, b)


Should we have a test that for a set of files the bundle zip remains identical across operating system (simple unit test should do it). Windows vs linux often have different new line characters.

Yes, they likely will be different but what is the issue in this case?

It's a performace optimization to avoid unnecessary updates if the snapshot did not change. The backend stores the sha on the ZIP. Should be a rare edge case so we can skip it.

shreyas-goenka · 2026-06-02T16:40:24Z

+}
+
+func (m *load) Apply(ctx context.Context, b *bundle.Bundle) diag.Diagnostics {
+	f, err := filer.NewWorkspaceFilesClient(b.WorkspaceClient(ctx), b.Config.Workspace.StatePath)


metadata.json was only meant to be consumed by non DABs resources like jobs or the UI. We have deployment.json and resources.json. Maybe we should store the snapshot path there?

This is a new dep - before the CLI did not read metadata.json - and we are trying to get rid of this with DMS.

shreyas-goenka · 2026-06-02T16:44:42Z


 				// Perform resolution only if the path starts with one of the specified prefixes.
 				if slices.ContainsFunc(prefixes, path.HasPrefix) {
+					if slices.Contains(m.excludePaths, path.String()) {


very nitpicky: Make this more robust? This matches substrings - so "abc" would match "abc" and "abcd". Normally in the codebase - paths refer to exact paths, not patterns or substrings.

shreyas-goenka · 2026-06-02T16:49:32Z

+Deployment complete!
+
+>>> [CLI] jobs get [NUMID]
+"/Workspace/Users/[UUID]/.snapshots/test-bundle-immutable-[UNIQUE_NAME]/11f80ca6d8923bf75b57e475d4ca9ba4bb1d6d48c58aace8d3f2a1289b51c6e0/src/files/src/main.py"


I guess this test covers that the SHA remains identical for windows and linux?

Why would it matter though? Practically they might be different, especially wheels built

pietern

I see the existing tests pass against cloud, but recommend including a testserver implementation already. Makes it easier to iterate.

pietern · 2026-06-16T12:40:35Z

+	if b.Config.Bundle.Deployment.ImmutableFolder {
+		return nil
+	}
+


Is it possible we always delay this until the deploy phase? Keeps things simpler.

pietern · 2026-06-16T12:42:07Z

+		}
+		return 0
+	})
+	return files, nil


The logic that collects the list of files looks very similar to what we do in libs/sync. Any chance we can let that pkg take care of building the list and we refer to it here? This set of files is also configured as b.Files for telemetry. If we chase the path that sets it we might be able to reuse it?

pietern · 2026-06-16T12:46:05Z

+	}
+
+	return &SnapshotInfo{Path: resp.Snapshot.Path}, nil
+}


This doesn't seem related to the other filers or the filer interface.

It seems logical to colocate this client with the code that takes a couple of []file.FileSet and performs all the zipping and uploading.

shreyas-goenka · 2026-06-19T12:13:58Z

+		}
+
+		// The real API uses the workspace user UUID (not email) in the snapshot path,
+		// matching service-principal identities used in cloud acceptance tests.


optional: Should we also use UUIDs here for better fidelity? The benefits are minor if we have cloud coverage. The difference being users have CAN_MANAGE always on their home directory but that's likely not true for /Users/userId?

shreyas-goenka · 2026-06-19T12:27:21Z

 // Apply implements bundle.Mutator.
 func (*workspaceRootPermissions) Apply(ctx context.Context, b *bundle.Bundle) diag.Diagnostics {
+	// If the bundle is immutable, we don't need to apply any permissions to the workspace root.
+	if b.Config.Bundle.Deployment.ImmutableFolder {


is this safe to do? Given that state management and resource CRUD (workspace.state_path and workspace.resource_path ) both need to have ACLs applied for shared deployments to work.

shreyas-goenka · 2026-06-19T12:27:36Z

            Whether to fail on active runs. If this is set to true a deployment that is running can be interrupted.
+        "immutable_folder":
+          "description": |-
+            Whether to upload bundle files and artifacts as a single immutable snapshot. When true, all files are packaged into a zip and uploaded via the snapshot API, and workspace.file_path and workspace.artifact_path are set to the returned content-addressed path. The validate and plan commands make no mutative API calls when this is enabled.


Snapshots API is internal. We should not refer to this in the docs.

shreyas-goenka · 2026-06-19T12:30:36Z

+func (m *translateResourcePaths) Name() string { return "snapshot.TranslateResourcePaths" }
+
+func (m *translateResourcePaths) Apply(_ context.Context, b *bundle.Bundle) diag.Diagnostics {
+	localPrefix := b.SyncRootPath + "/"


Maybe strings.TrimSuffix("/") as well to account for trailing /? Or is that not possible?

shreyas-goenka · 2026-06-19T12:33:34Z

+func (s *loadState) Name() string { return "snapshot.LoadState" }
+
+func (s *saveState) Apply(ctx context.Context, b *bundle.Bundle) diag.Diagnostics {
+	if b.Config.Workspace.SnapshotPath == "" {


Do we not need to store this remotely? The code reads like it only does local storage but we need it in remote as well.

This sounds like something we can add to resources.json

shreyas-goenka · 2026-06-19T12:34:21Z

+
+// SaveState writes the snapshot path to the local deployment state directory
+// so it can be recovered during destroy without reading metadata.json.
+func SaveState() bundle.Mutator {


Consider integrating this with deployment WAL? If we can record the snapshot upload event we can avoid reuploading it on a subsequent deployment since it already exists.

We can't really, can we? The deployment WAL calculated before the execution of deployment as part of plan but we build and upload later in the phase

I assume during deployments we write to WAL so surely we should be able to do that during / after file upload? Otherwise do how do we capture if the plan was partially applied.

I could be misunderstanding the WAL - I'm not familiar with it (will look into it) - I just assumed it records actions as we do them.

shreyas-goenka · 2026-06-19T12:37:21Z

+>>> [CLI] jobs get [NUMID]
+"/Workspace/Users/[UUID]/.snapshots/test-bundle-immutable-no-artifacts-[UNIQUE_NAME]/[SNAPSHOT_HASH]/src/files/src/notebook"
+
+>>> [CLI] bundle destroy --auto-approve


can we assert that destroy deletes the snapshot? Even when .databricks is removed?

shreyas-goenka · 2026-06-19T12:40:23Z

 		// Updates (dynamic): resources.* (strings) (resolves variable references to their actual values)
 		// Resolves variable references in 'resources' using bundle, workspace, and variables prefixes
-		mutator.ResolveVariableReferencesOnlyResources(),
+		resourceResolver,


why this diff?

shreyas-goenka · 2026-06-19T12:42:55Z

@@ -0,0 +1,16 @@
+Local = false


we can run locally as well? As long as we fix the snapshot API impl in test server?

Immutable folder support in DABs

b685ec2

andrewnester requested a review from pietern May 15, 2026 09:56

andrewnester temporarily deployed to test-trigger-is May 15, 2026 09:56 — with GitHub Actions Inactive

andrewnester added 2 commits May 28, 2026 11:03

remove unused snapshot path method

a429b26

added an acceptance test

e7c1968

andrewnester temporarily deployed to test-trigger-is May 28, 2026 10:01 — with GitHub Actions Inactive

fix for notebook import

67914d0

andrewnester temporarily deployed to test-trigger-is May 28, 2026 10:19 — with GitHub Actions Inactive

removed unused function

549492a

andrewnester temporarily deployed to test-trigger-is May 28, 2026 10:41 — with GitHub Actions Inactive

andrewnester marked this pull request as ready for review June 1, 2026 11:34

fix schema + unit test

aedfdb0

andrewnester temporarily deployed to test-trigger-is June 1, 2026 11:39 — with GitHub Actions Inactive

andrewnester requested a review from denik June 2, 2026 10:01

shreyas-goenka reviewed Jun 2, 2026

View reviewed changes

andrewnester and others added 2 commits June 9, 2026 12:49

use immutable_folder config

eddec61

Merge branch 'main' into demo-immutable

27a6b02

andrewnester temporarily deployed to test-trigger-is June 9, 2026 10:51 — with GitHub Actions Inactive

remove merge conflict

4a9bcd9

andrewnester temporarily deployed to test-trigger-is June 9, 2026 10:53 — with GitHub Actions Inactive

fix empty artifact path + tests

ebd26ea

andrewnester temporarily deployed to test-trigger-is June 9, 2026 11:14 — with GitHub Actions Inactive

fixed test config

5efe1da

andrewnester temporarily deployed to test-trigger-is June 9, 2026 13:09 — with GitHub Actions Inactive

pietern reviewed Jun 16, 2026

View reviewed changes

andrewnester and others added 2 commits June 18, 2026 16:06

fixes

6215a49

Merge branch 'main' into demo-immutable

7b44126

andrewnester requested review from pietern and shreyas-goenka June 18, 2026 14:08

andrewnester temporarily deployed to test-trigger-is June 18, 2026 14:09 — with GitHub Actions Inactive

andrewnester added 2 commits June 18, 2026 18:49

fix fmt

be6dec3

fix annotations

0f1ed52

andrewnester temporarily deployed to test-trigger-is June 18, 2026 16:50 — with GitHub Actions Inactive

andrewnester added 2 commits June 18, 2026 18:56

fix lint

d978559

do not call set permissions on immutable ws root

9a6c898

andrewnester temporarily deployed to test-trigger-is June 18, 2026 17:00 — with GitHub Actions Inactive

shreyas-goenka reviewed Jun 19, 2026

View reviewed changes


		cmdio.LogString(ctx, "Uploading immutable bundle snapshot...")

		zipContent, err := BundleZip(ctx, b)

Conversation

andrewnester commented May 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Why

Tests

Uh oh!

eng-dev-ecosystem-bot commented May 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Integration test report

Uh oh!

github-actions Bot commented Jun 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Approval status: pending

/acceptance/bundle/ - needs approval

/bundle/ - needs approval

/cmd/bundle/ - needs approval

/libs/sync/ - needs approval

General files (require maintainer)

Uh oh!

shreyas-goenka left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pietern left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shreyas-goenka Jun 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

andrewnester commented May 15, 2026 •

edited

Loading

eng-dev-ecosystem-bot commented May 28, 2026 •

edited

Loading

github-actions Bot commented Jun 1, 2026 •

edited

Loading

`/acceptance/bundle/` - needs approval

`/bundle/` - needs approval

`/cmd/bundle/` - needs approval

`/libs/sync/` - needs approval

shreyas-goenka Jun 19, 2026 •

edited

Loading