Skip to content

Commit e8b9646

Browse files
shreyas-goenkashreyas-goenka
committed
acceptance: cover DMS operation recording for create/update/delete/recreate
Local acceptance tests (direct engine, RecordRequests) that deploy with experimental.record_deployment_history enabled and assert the recorded CreateOperation requests: - jobs/: create -> update -> delete (resource removed from config). Verifies action_type CREATE/UPDATE/DELETE, that the serialized state is carried for create/update and omitted for delete, and resource_key/status. - recreate/: changing a pipeline's immutable 'storage' field records RECREATE. Adds a testserver handler for POST /api/2.0/bundle/{path...}/operations so the recorder's CreateOperation calls succeed under the mock server. Co-authored-by: Shreyas Goenka <shreyas.goenka@databricks.com>
1 parent 465bc4a commit e8b9646

16 files changed

Lines changed: 258 additions & 17 deletions

File tree

‎acceptance/bundle/deploy/record-deployment-history/jobs/databricks.delete.yml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
bundle:
2+
name: test-rdh-jobs
3+
4+
experimental:
5+
record_deployment_history: true
6+
7+
resources:
8+
jobs: {}

‎acceptance/bundle/deploy/record-deployment-history/jobs/databricks.yml‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
bundle:
2+
name: test-rdh-jobs
3+
4+
experimental:
5+
record_deployment_history: true
6+
7+
resources:
8+
jobs:
9+
foo:
10+
name: foo-job

‎acceptance/bundle/deploy/record-deployment-history/jobs/out.test.toml‎

Lines changed: 3 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎acceptance/bundle/deploy/record-deployment-history/jobs/output.txt‎

Lines changed: 94 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
1+
2+
=== create: records OPERATION_ACTION_TYPE_CREATE
3+
>>> [CLI] bundle deploy
4+
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-rdh-jobs/default/files...
5+
Deploying resources...
6+
Updating deployment state...
7+
Deployment complete!
8+
9+
>>> print_requests.py //api/2.0/bundle
10+
{
11+
"method": "POST",
12+
"path": "/api/2.0/bundle/deployments/versions/0/operations",
13+
"q": {
14+
"resource_key": "resources.jobs.foo"
15+
},
16+
"body": {
17+
"action_type": "OPERATION_ACTION_TYPE_CREATE",
18+
"resource_id": "[NUMID]",
19+
"resource_key": "resources.jobs.foo",
20+
"state": {
21+
"deployment": {
22+
"kind": "BUNDLE",
23+
"metadata_file_path": "/Workspace/Users/[USERNAME]/.bundle/test-rdh-jobs/default/state/metadata.json"
24+
},
25+
"edit_mode": "UI_LOCKED",
26+
"format": "MULTI_TASK",
27+
"max_concurrent_runs": 1,
28+
"name": "foo-job",
29+
"queue": {
30+
"enabled": true
31+
}
32+
},
33+
"status": "OPERATION_STATUS_SUCCEEDED"
34+
}
35+
}
36+
37+
=== update: records OPERATION_ACTION_TYPE_UPDATE
38+
>>> update_file.py databricks.yml foo-job foo-job-renamed
39+
40+
>>> [CLI] bundle deploy
41+
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-rdh-jobs/default/files...
42+
Deploying resources...
43+
Updating deployment state...
44+
Deployment complete!
45+
46+
>>> print_requests.py //api/2.0/bundle
47+
{
48+
"method": "POST",
49+
"path": "/api/2.0/bundle/deployments/versions/0/operations",
50+
"q": {
51+
"resource_key": "resources.jobs.foo"
52+
},
53+
"body": {
54+
"action_type": "OPERATION_ACTION_TYPE_UPDATE",
55+
"resource_id": "[NUMID]",
56+
"resource_key": "resources.jobs.foo",
57+
"state": {
58+
"deployment": {
59+
"kind": "BUNDLE",
60+
"metadata_file_path": "/Workspace/Users/[USERNAME]/.bundle/test-rdh-jobs/default/state/metadata.json"
61+
},
62+
"edit_mode": "UI_LOCKED",
63+
"format": "MULTI_TASK",
64+
"max_concurrent_runs": 1,
65+
"name": "foo-job-renamed",
66+
"queue": {
67+
"enabled": true
68+
}
69+
},
70+
"status": "OPERATION_STATUS_SUCCEEDED"
71+
}
72+
}
73+
74+
=== delete: records OPERATION_ACTION_TYPE_DELETE (resource removed from config)
75+
>>> [CLI] bundle deploy
76+
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-rdh-jobs/default/files...
77+
Deploying resources...
78+
Updating deployment state...
79+
Deployment complete!
80+
81+
>>> print_requests.py //api/2.0/bundle
82+
{
83+
"method": "POST",
84+
"path": "/api/2.0/bundle/deployments/versions/0/operations",
85+
"q": {
86+
"resource_key": "resources.jobs.foo"
87+
},
88+
"body": {
89+
"action_type": "OPERATION_ACTION_TYPE_DELETE",
90+
"resource_id": "[NUMID]",
91+
"resource_key": "resources.jobs.foo",
92+
"status": "OPERATION_STATUS_SUCCEEDED"
93+
}
94+
}

‎acceptance/bundle/deploy/record-deployment-history/jobs/script‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
title "create: records OPERATION_ACTION_TYPE_CREATE"
2+
trace $CLI bundle deploy
3+
trace print_requests.py //api/2.0/bundle
4+
5+
title "update: records OPERATION_ACTION_TYPE_UPDATE"
6+
trace update_file.py databricks.yml foo-job foo-job-renamed
7+
trace $CLI bundle deploy
8+
trace print_requests.py //api/2.0/bundle
9+
10+
title "delete: records OPERATION_ACTION_TYPE_DELETE (resource removed from config)"
11+
cp databricks.delete.yml databricks.yml
12+
trace $CLI bundle deploy
13+
trace print_requests.py //api/2.0/bundle

‎acceptance/bundle/deploy/record-deployment-history/jobs/test.toml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
# databricks.yml is rewritten in-place by the script (update + delete steps).
2+
Ignore = [".databricks", "databricks.yml"]

‎acceptance/bundle/deploy/record-deployment-history/recreate/databricks.yml‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
bundle:
2+
name: test-rdh-recreate
3+
4+
experimental:
5+
record_deployment_history: true
6+
7+
resources:
8+
pipelines:
9+
bar:
10+
name: bar-pipeline
11+
storage: /tmp/storage-a

‎acceptance/bundle/deploy/record-deployment-history/recreate/out.test.toml‎

Lines changed: 3 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎acceptance/bundle/deploy/record-deployment-history/recreate/output.txt‎

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
2+
=== create the pipeline
3+
>>> [CLI] bundle deploy
4+
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-rdh-recreate/default/files...
5+
Deploying resources...
6+
Updating deployment state...
7+
Deployment complete!
8+
9+
>>> print_requests.py //api/2.0/bundle
10+
{
11+
"method": "POST",
12+
"path": "/api/2.0/bundle/deployments/versions/0/operations",
13+
"q": {
14+
"resource_key": "resources.pipelines.bar"
15+
},
16+
"body": {
17+
"action_type": "OPERATION_ACTION_TYPE_CREATE",
18+
"resource_id": "[UUID]",
19+
"resource_key": "resources.pipelines.bar",
20+
"state": {
21+
"channel": "CURRENT",
22+
"deployment": {
23+
"kind": "BUNDLE",
24+
"metadata_file_path": "/Workspace/Users/[USERNAME]/.bundle/test-rdh-recreate/default/state/metadata.json"
25+
},
26+
"edition": "ADVANCED",
27+
"name": "bar-pipeline",
28+
"storage": "/tmp/storage-a"
29+
},
30+
"status": "OPERATION_STATUS_SUCCEEDED"
31+
}
32+
}
33+
34+
=== recreate: changing immutable 'storage' records OPERATION_ACTION_TYPE_RECREATE
35+
>>> update_file.py databricks.yml /tmp/storage-a /tmp/storage-b
36+
37+
>>> [CLI] bundle deploy --auto-approve
38+
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-rdh-recreate/default/files...
39+
40+
This action will result in the deletion or recreation of the following Lakeflow Spark Declarative Pipelines along with the
41+
Streaming Tables (STs) and Materialized Views (MVs) managed by them. Recreating the pipelines will
42+
restore the defined STs and MVs through full refresh. Note that recreation is necessary when pipeline
43+
properties such as the 'catalog' or 'storage' are changed:
44+
recreate resources.pipelines.bar
45+
Deploying resources...
46+
Updating deployment state...
47+
Deployment complete!
48+
49+
>>> print_requests.py //api/2.0/bundle
50+
{
51+
"method": "POST",
52+
"path": "/api/2.0/bundle/deployments/versions/0/operations",
53+
"q": {
54+
"resource_key": "resources.pipelines.bar"
55+
},
56+
"body": {
57+
"action_type": "OPERATION_ACTION_TYPE_RECREATE",
58+
"resource_id": "[UUID]",
59+
"resource_key": "resources.pipelines.bar",
60+
"state": {
61+
"channel": "CURRENT",
62+
"deployment": {
63+
"kind": "BUNDLE",
64+
"metadata_file_path": "/Workspace/Users/[USERNAME]/.bundle/test-rdh-recreate/default/state/metadata.json"
65+
},
66+
"edition": "ADVANCED",
67+
"name": "bar-pipeline",
68+
"storage": "/tmp/storage-b"
69+
},
70+
"status": "OPERATION_STATUS_SUCCEEDED"
71+
}
72+
}

‎acceptance/bundle/deploy/record-deployment-history/recreate/script‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
title "create the pipeline"
2+
trace $CLI bundle deploy
3+
trace print_requests.py //api/2.0/bundle
4+
5+
title "recreate: changing immutable 'storage' records OPERATION_ACTION_TYPE_RECREATE"
6+
trace update_file.py databricks.yml /tmp/storage-a /tmp/storage-b
7+
trace $CLI bundle deploy --auto-approve
8+
trace print_requests.py //api/2.0/bundle

0 commit comments

Comments
 (0)