Describe the bug
In this doc https://learn.microsoft.com/en-us/cli/azure/synapse/spark/pool?view=azure-cli-latest#az-synapse-spark-pool-update I found a command to update a Synapse Spark Pool and its configuration.
The command snippet "az synapse spark pool update --name testpool --workspace-name testsynapseworkspace --resource-group rg \ --spark-config-file-path 'path/configfile.txt'" uses the parameter "--spark-config-file-path" that is the "Absolute path of Spark pool properties configuration file.": in Synapse the concept of path for Spark Configuration doesn't exist as they reside in the same virtual location "Apache Spark configurations" (see following screenshot):
Is there a way to update the Spark Configuration association to a Spark Pool as I can do in Azure Portal?
Related command
az synapse spark pool update --name testpool --workspace-name testsynapseworkspace --resource-group rg
--spark-config-file-path 'path/configfile.txt'
Errors
Spark config file path is invalid
Issue script & Debug output
cli.azure.cli.core.sdk.policies: Response status: 200
cli.azure.cli.core.sdk.policies: Response headers:
cli.azure.cli.core.sdk.policies: 'Cache-Control': 'no-cache'
cli.azure.cli.core.sdk.policies: 'Pragma': 'no-cache'
cli.azure.cli.core.sdk.policies: 'Transfer-Encoding': 'chunked'
cli.azure.cli.core.sdk.policies: 'Content-Type': 'application/json; charset=utf-8'
cli.azure.cli.core.sdk.policies: 'Content-Encoding': 'gzip'
cli.azure.cli.core.sdk.policies: 'Expires': '-1'
cli.azure.cli.core.sdk.policies: 'Vary': 'Accept-Encoding'
cli.azure.cli.core.sdk.policies: 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains'
cli.azure.cli.core.sdk.policies: 'x-ms-request-id': '28507422-9855-4cec-a51f-78861fecf923'
cli.azure.cli.core.sdk.policies: 'Server': 'Microsoft-HTTPAPI/2.0'
cli.azure.cli.core.sdk.policies: 'x-ms-ratelimit-remaining-subscription-reads': '11999'
cli.azure.cli.core.sdk.policies: 'x-ms-correlation-request-id': '74abc868-37a1-4cf1-91d7-a9144850db45'
cli.azure.cli.core.sdk.policies: 'x-ms-routing-request-id': 'WESTEUROPE:20230616T113511Z:74abc868-37a1-4cf1-91d7-a9144850db45'
cli.azure.cli.core.sdk.policies: 'X-Content-Type-Options': 'nosniff'
cli.azure.cli.core.sdk.policies: 'Date': 'Fri, 16 Jun 2023 11:35:11 GMT'
cli.azure.cli.core.sdk.policies: Response content:
cli.azure.cli.core.sdk.policies: {"properties":{"creationDate":"2023-06-07T20:23:31.6733333Z","sparkVersion":"3.2","nodeCount":10,"nodeSize":"Small","nodeSizeFamily":"MemoryOptimized","autoScale":{"enabled":true,"minNodeCount":3,"maxNodeCount":10},"autoPause":{"enabled":true,"delayInMinutes":15},"isComputeIsolationEnabled":false,"sessionLevelPackagesEnabled":true,"cacheSize":50,"dynamicExecutorAllocation":{"enabled":true,"minExecutors":1,"maxExecutors":4},"lastSucceededTimestamp":"2023-06-15T13:36:47.1133333Z","isAutotuneEnabled":false,"provisioningState":"Succeeded"},"id":"test/bigDataPools/sparktest","name":"sparktest","type":"Microsoft.Synapse/workspaces/bigDataPools","location":"westeurope","tags":{}}
cli.azure.cli.core.azclierror: Traceback (most recent call last):
File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/command_modules/synapse/manual/operations/sparkpool.py", line 116, in update_spark_pool
with open(spark_config_file_path, 'r') as stream:
FileNotFoundError: [Errno 2] No such file or directory: 'sparkConfiguration_test'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib64/az/lib/python3.9/site-packages/knack/cli.py", line 233, in invoke
cmd_result = self.invocation.execute(args)
File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/core/commands/init.py", line 663, in execute
raise ex
File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/core/commands/init.py", line 726, in _run_jobs_serially
results.append(self._run_job(expanded_arg, cmd_copy))
File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/core/commands/init.py", line 697, in _run_job
result = cmd_copy(params)
File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/core/commands/init.py", line 333, in call
return self.handler(*args, **kwargs)
File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/core/commands/command_operation.py", line 121, in handler
return op(**command_args)
File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/command_modules/synapse/manual/operations/sparkpool.py", line 121, in update_spark_pool
raise InvalidArgumentValueError(err_msg)
azure.cli.core.azclierror.InvalidArgumentValueError: Spark config file path is invalid
cli.azure.cli.core.azclierror: Spark config file path is invalid
az_command_data_logger: Spark config file path is invalid
cli.knack.cli: Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x7f72fc48f5e0>]
az_command_data_logger: exit code: 1
cli.main: Command ran in 0.728 seconds (init: 0.136, invoke: 0.592)
telemetry.main: Begin splitting cli events and extra events, total events: 1
telemetry.client: Accumulated 0 events. Flush the clients.
telemetry.main: Finish splitting cli events and extra events, cli events: 1
telemetry.save: Save telemetry record of length 3592 in cache
Expected behavior
I want to update the Spark Configuration association to a Spark Pool as I can do in Azure Portal:
Environment Summary
azure-cli 2.49.0
core 2.49.0
telemetry 1.0.8
Extensions:
ai-examples 0.2.5
ml 2.17.2
ssh 1.1.6
Dependencies:
msal 1.20.0
azure-mgmt-resource 22.0.0
Python location '/usr/bin/python3.9'
Extensions directory '/home/luca/.azure/cliextensions'
Extensions system directory '/usr/lib/python3.9/site-packages/azure-cli-extensions'
Python (Linux) 3.9.14 (main, Feb 8 2023, 03:39:52)
[GCC 11.2.0]
Legal docs and information: aka.ms/AzureCliLegal
Your CLI is up-to-date.
Additional context
No response
Describe the bug
In this doc https://learn.microsoft.com/en-us/cli/azure/synapse/spark/pool?view=azure-cli-latest#az-synapse-spark-pool-update I found a command to update a Synapse Spark Pool and its configuration.
The command snippet "az synapse spark pool update --name testpool --workspace-name testsynapseworkspace --resource-group rg \ --spark-config-file-path 'path/configfile.txt'" uses the parameter "--spark-config-file-path" that is the "Absolute path of Spark pool properties configuration file.": in Synapse the concept of path for Spark Configuration doesn't exist as they reside in the same virtual location "Apache Spark configurations" (see following screenshot):
Is there a way to update the Spark Configuration association to a Spark Pool as I can do in Azure Portal?
Related command
az synapse spark pool update --name testpool --workspace-name testsynapseworkspace --resource-group rg
--spark-config-file-path 'path/configfile.txt'
Errors
Spark config file path is invalid
Issue script & Debug output
cli.azure.cli.core.sdk.policies: Response status: 200
cli.azure.cli.core.sdk.policies: Response headers:
cli.azure.cli.core.sdk.policies: 'Cache-Control': 'no-cache'
cli.azure.cli.core.sdk.policies: 'Pragma': 'no-cache'
cli.azure.cli.core.sdk.policies: 'Transfer-Encoding': 'chunked'
cli.azure.cli.core.sdk.policies: 'Content-Type': 'application/json; charset=utf-8'
cli.azure.cli.core.sdk.policies: 'Content-Encoding': 'gzip'
cli.azure.cli.core.sdk.policies: 'Expires': '-1'
cli.azure.cli.core.sdk.policies: 'Vary': 'Accept-Encoding'
cli.azure.cli.core.sdk.policies: 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains'
cli.azure.cli.core.sdk.policies: 'x-ms-request-id': '28507422-9855-4cec-a51f-78861fecf923'
cli.azure.cli.core.sdk.policies: 'Server': 'Microsoft-HTTPAPI/2.0'
cli.azure.cli.core.sdk.policies: 'x-ms-ratelimit-remaining-subscription-reads': '11999'
cli.azure.cli.core.sdk.policies: 'x-ms-correlation-request-id': '74abc868-37a1-4cf1-91d7-a9144850db45'
cli.azure.cli.core.sdk.policies: 'x-ms-routing-request-id': 'WESTEUROPE:20230616T113511Z:74abc868-37a1-4cf1-91d7-a9144850db45'
cli.azure.cli.core.sdk.policies: 'X-Content-Type-Options': 'nosniff'
cli.azure.cli.core.sdk.policies: 'Date': 'Fri, 16 Jun 2023 11:35:11 GMT'
cli.azure.cli.core.sdk.policies: Response content:
cli.azure.cli.core.sdk.policies: {"properties":{"creationDate":"2023-06-07T20:23:31.6733333Z","sparkVersion":"3.2","nodeCount":10,"nodeSize":"Small","nodeSizeFamily":"MemoryOptimized","autoScale":{"enabled":true,"minNodeCount":3,"maxNodeCount":10},"autoPause":{"enabled":true,"delayInMinutes":15},"isComputeIsolationEnabled":false,"sessionLevelPackagesEnabled":true,"cacheSize":50,"dynamicExecutorAllocation":{"enabled":true,"minExecutors":1,"maxExecutors":4},"lastSucceededTimestamp":"2023-06-15T13:36:47.1133333Z","isAutotuneEnabled":false,"provisioningState":"Succeeded"},"id":"test/bigDataPools/sparktest","name":"sparktest","type":"Microsoft.Synapse/workspaces/bigDataPools","location":"westeurope","tags":{}}
cli.azure.cli.core.azclierror: Traceback (most recent call last):
File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/command_modules/synapse/manual/operations/sparkpool.py", line 116, in update_spark_pool
with open(spark_config_file_path, 'r') as stream:
FileNotFoundError: [Errno 2] No such file or directory: 'sparkConfiguration_test'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib64/az/lib/python3.9/site-packages/knack/cli.py", line 233, in invoke
cmd_result = self.invocation.execute(args)
File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/core/commands/init.py", line 663, in execute
raise ex
File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/core/commands/init.py", line 726, in _run_jobs_serially
results.append(self._run_job(expanded_arg, cmd_copy))
File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/core/commands/init.py", line 697, in _run_job
result = cmd_copy(params)
File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/core/commands/init.py", line 333, in call
return self.handler(*args, **kwargs)
File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/core/commands/command_operation.py", line 121, in handler
return op(**command_args)
File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/command_modules/synapse/manual/operations/sparkpool.py", line 121, in update_spark_pool
raise InvalidArgumentValueError(err_msg)
azure.cli.core.azclierror.InvalidArgumentValueError: Spark config file path is invalid
cli.azure.cli.core.azclierror: Spark config file path is invalid
az_command_data_logger: Spark config file path is invalid
cli.knack.cli: Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x7f72fc48f5e0>]
az_command_data_logger: exit code: 1
cli.main: Command ran in 0.728 seconds (init: 0.136, invoke: 0.592)
telemetry.main: Begin splitting cli events and extra events, total events: 1
telemetry.client: Accumulated 0 events. Flush the clients.
telemetry.main: Finish splitting cli events and extra events, cli events: 1
telemetry.save: Save telemetry record of length 3592 in cache
Expected behavior
I want to update the Spark Configuration association to a Spark Pool as I can do in Azure Portal:
Environment Summary
azure-cli 2.49.0
core 2.49.0
telemetry 1.0.8
Extensions:
ai-examples 0.2.5
ml 2.17.2
ssh 1.1.6
Dependencies:
msal 1.20.0
azure-mgmt-resource 22.0.0
Python location '/usr/bin/python3.9'
Extensions directory '/home/luca/.azure/cliextensions'
Extensions system directory '/usr/lib/python3.9/site-packages/azure-cli-extensions'
Python (Linux) 3.9.14 (main, Feb 8 2023, 03:39:52)
[GCC 11.2.0]
Legal docs and information: aka.ms/AzureCliLegal
Your CLI is up-to-date.
Additional context
No response