cli doesn't respect custom host within SAS URI

[ziyaddin]

Describe the bug

Azure CLI doesn't respect custom host within SAS URI.

Example: When a download command is run, the download request is sent to https://devstoreaccount1.blob.core.windows.net:443 while the (URI host + account) within SAS URI is https://127.0.0.1:10000/devstoreaccount1.

In contrast, the host is respected when passed within connection string.

Related command

Setting environment variables:

export AZURE_STORAGE_CONNECTION_STRING="DefaultEndpointsProtocol=https;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=https://127.0.0.1:10000/devstoreaccount1;QueueEndpoint=http://127.0.0.1:10001/devstoreaccount1;TableEndpoint=http://127.0.0.1:10002/devstoreaccount1;"

az storage blob generate-sas \
-c uploads \
-n manual-upload.jpg \
--connection-string $AZURE_STORAGE_CONNECTION_STRING \
--expiry 2023-10-20T18:59Z \
--permissions r \
--full-uri

export AZURE_STORAGE_SAS_TOKEN="https://127.0.0.1:10000/devstoreaccount1/uploads/manual-upload.jpg?se=2023-10-20T18%3A59Z&sp=r&sv=2022-11-02&sr=b&sig=7ob2HYJcmseqgiGaw20PDo6gjDx%2BCfL7kKAtNis3t5M%3D"

Failed command using SAS token:

az storage blob download \
-c uploads \
-n image.jpg \
-f downloads/image.jpg \
--account-name devstoreaccount1 \
--sas-token $AZURE_STORAGE_SAS_TOKEN \
--debug

Successful command using connection string:

az storage blob download \
-c uploads \
-n image.jpg \
-f downloads/image.jpg \
--account-name devstoreaccount1 \
--connection-string $AZURE_STORAGE_CONNECTION_STRING \
--debug

Errors

Authentication failure. This may be caused by either invalid account key, connection string or sas token value provided for your storage account.

Issue script & Debug output

Issue script:

az storage blob download \
-c uploads \
-n image.jpg \
-f downloads/image.jpg \
--account-name devstoreaccount1 \
--sas-token $AZURE_STORAGE_SAS_TOKEN \
--debug

Debug output:

cli.knack.cli: Command arguments: ['storage', 'blob', 'download', '-c', 'uploads', '-n', 'manual-upload.jpg', '--sas-token', 'https://127.0.0.1:10000/devstoreaccount1/uploads/manual-upload.jpg?se=2023-10-20T18%3A59Z&sp=r&sv=2022-11-02&sr=b&sig=7ob2HYJcmseqgiGaw20PDo6gjDx%2BCfL7kKAtNis3t5M%3D', '--account-name', 'devstoreaccount1', '-f', 'images/downloads/manual-upload.jpg', '--debug']
cli.knack.cli: __init__ debug log:
Enable color in terminal.
cli.knack.cli: Event: Cli.PreExecute []
cli.knack.cli: Event: CommandParser.OnGlobalArgumentsCreate [<function CLILogging.on_global_arguments at 0x10b0972e0>, <function OutputProducer.on_global_arguments at 0x10b174700>, <function CLIQuery.on_global_arguments at 0x10b195b40>]
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableCreate []
cli.azure.cli.core: Modules found from index for 'storage': ['azure.cli.command_modules.storage']
cli.azure.cli.core: Loading command modules:
cli.azure.cli.core: Name                  Load Time    Groups  Commands
cli.azure.cli.core: storage                   0.123        58       272
cli.azure.cli.core: Total (1)                 0.123        58       272
cli.azure.cli.core: Loaded 58 groups, 272 commands.
cli.azure.cli.core: Found a match in the command table.
cli.azure.cli.core: Raw command  : storage blob download
cli.azure.cli.core: Command table: storage blob download
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableTruncate [<function AzCliLogging.init_command_file_logging at 0x10be211b0>]
cli.azure.cli.core.azlogging: metadata file logging enabled - writing logs to '/Users/ziyaddin/.azure/commands/2023-10-20.13-25-40.storage_blob_download.20267.log'.
az_command_data_logger: command args: storage blob download -c {} -n {} --sas-token {} --account-name {} -f {} --debug
cli.knack.cli: Event: CommandInvoker.OnPreArgumentLoad [<function register_global_subscription_argument.<locals>.add_subscription_parameter at 0x10be3dc60>]
cli.azure.cli.core.profiles._shared: Traceback (most recent call last):
  File "/usr/local/Cellar/azure-cli/2.53.0/libexec/lib/python3.10/site-packages/azure/cli/core/profiles/_shared.py", line 655, in _get_attr
    op = getattr(op, part)
AttributeError: module 'azure.mgmt.storage.v2022_09_01.models' has no attribute 'ActiveDirectoryPropertiesAccountType'

cli.azure.cli.core.profiles._shared: Traceback (most recent call last):
  File "/usr/local/Cellar/azure-cli/2.53.0/libexec/lib/python3.10/site-packages/azure/cli/core/profiles/_shared.py", line 655, in _get_attr
    op = getattr(op, part)
AttributeError: module 'azure.mgmt.storage.v2022_09_01.models' has no attribute 'ListKeyExpand'

cli.azure.cli.core.profiles._shared: Traceback (most recent call last):
  File "/usr/local/Cellar/azure-cli/2.53.0/libexec/lib/python3.10/site-packages/azure/cli/core/profiles/_shared.py", line 655, in _get_attr
    op = getattr(op, part)
AttributeError: module 'azure.mgmt.storage.v2022_09_01.models' has no attribute 'CorsRuleAllowedMethodsItem'

cli.knack.cli: Event: CommandInvoker.OnPostArgumentLoad []
cli.knack.cli: Event: CommandInvoker.OnPostCommandTableCreate [<function register_ids_argument.<locals>.add_ids_arguments at 0x10be67910>, <function register_cache_arguments.<locals>.add_cache_arguments at 0x10be67a30>]
cli.knack.cli: Event: CommandInvoker.OnCommandTableLoaded []
cli.knack.cli: Event: CommandInvoker.OnPreParseArgs []
cli.knack.cli: Event: CommandInvoker.OnPostParseArgs [<function OutputProducer.handle_output_argument at 0x10b174790>, <function CLIQuery.handle_query_parameter at 0x10b195bd0>, <function register_ids_argument.<locals>.parse_ids_arguments at 0x10be679a0>]
cli.azure.cli.command_modules.storage._validators: Try to get storage auth_mode value from environment variables or config file.
cli.azure.cli.core._debug: Using CA bundle file at '/usr/local/Cellar/azure-cli/2.53.0/libexec/lib/python3.10/site-packages/certifi/cacert.pem'.
urllib3.connectionpool: Starting new HTTPS connection (1): devstoreaccount1.blob.core.windows.net:443
urllib3.connectionpool: https://devstoreaccount1.blob.core.windows.net:443 "GET /uploads/manual-upload.jpg HTTP/1.1" 403 760
azure.multiapi.storagev2.blob.v2022_11_02._generated._serialization: Ran into a deserialization error. Ignoring since this is failsafe deserialization
Traceback (most recent call last):
  File "/usr/local/Cellar/azure-cli/2.53.0/libexec/lib/python3.10/site-packages/azure/multiapi/storagev2/blob/v2022_11_02/_generated/_serialization.py", line 1506, in failsafe_deserialize
    return self(target_obj, data, content_type=content_type)
  File "/usr/local/Cellar/azure-cli/2.53.0/libexec/lib/python3.10/site-packages/azure/multiapi/storagev2/blob/v2022_11_02/_generated/_serialization.py", line 1377, in __call__
    data = self._unpack_content(response_data, content_type)
  File "/usr/local/Cellar/azure-cli/2.53.0/libexec/lib/python3.10/site-packages/azure/multiapi/storagev2/blob/v2022_11_02/_generated/_serialization.py", line 1535, in _unpack_content
    raise ValueError("This pipeline didn't have the RawDeserializer policy; can't deserialize")
ValueError: This pipeline didn't have the RawDeserializer policy; can't deserialize
cli.azure.cli.core.azclierror: Traceback (most recent call last):
  File "/usr/local/Cellar/azure-cli/2.53.0/libexec/lib/python3.10/site-packages/knack/cli.py", line 233, in invoke
    cmd_result = self.invocation.execute(args)
  File "/usr/local/Cellar/azure-cli/2.53.0/libexec/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 663, in execute
    raise ex
  File "/usr/local/Cellar/azure-cli/2.53.0/libexec/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 726, in _run_jobs_serially
    results.append(self._run_job(expanded_arg, cmd_copy))
  File "/usr/local/Cellar/azure-cli/2.53.0/libexec/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 718, in _run_job
    return cmd_copy.exception_handler(ex)
  File "/usr/local/Cellar/azure-cli/2.53.0/libexec/lib/python3.10/site-packages/azure/cli/command_modules/storage/__init__.py", line 419, in new_handler
    first(ex)
  File "/usr/local/Cellar/azure-cli/2.53.0/libexec/lib/python3.10/site-packages/azure/cli/command_modules/storage/__init__.py", line 419, in new_handler
    first(ex)
  File "/usr/local/Cellar/azure-cli/2.53.0/libexec/lib/python3.10/site-packages/azure/cli/command_modules/storage/_exception_handler.py", line 17, in file_related_exception_handler
    raise ex
  File "/usr/local/Cellar/azure-cli/2.53.0/libexec/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 697, in _run_job
    result = cmd_copy(params)
  File "/usr/local/Cellar/azure-cli/2.53.0/libexec/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 333, in __call__
    return self.handler(*args, **kwargs)
  File "/usr/local/Cellar/azure-cli/2.53.0/libexec/lib/python3.10/site-packages/azure/cli/core/commands/command_operation.py", line 121, in handler
    return op(**command_args)
  File "/usr/local/Cellar/azure-cli/2.53.0/libexec/lib/python3.10/site-packages/azure/cli/command_modules/storage/operations/blob.py", line 689, in download_blob
    download_stream = client.download_blob(offset=offset, length=length, **kwargs)
  File "/usr/local/Cellar/azure-cli/2.53.0/libexec/lib/python3.10/site-packages/azure/core/tracing/decorator.py", line 78, in wrapper_use_tracer
    return func(*args, **kwargs)
  File "/usr/local/Cellar/azure-cli/2.53.0/libexec/lib/python3.10/site-packages/azure/multiapi/storagev2/blob/v2022_11_02/_blob_client.py", line 914, in download_blob
    return StorageStreamDownloader(**options)
  File "/usr/local/Cellar/azure-cli/2.53.0/libexec/lib/python3.10/site-packages/azure/multiapi/storagev2/blob/v2022_11_02/_download.py", line 366, in __init__
    self._response = self._initial_request()
  File "/usr/local/Cellar/azure-cli/2.53.0/libexec/lib/python3.10/site-packages/azure/multiapi/storagev2/blob/v2022_11_02/_download.py", line 466, in _initial_request
    process_storage_error(error)
  File "/usr/local/Cellar/azure-cli/2.53.0/libexec/lib/python3.10/site-packages/azure/multiapi/storagev2/blob/v2022_11_02/_shared/response_handlers.py", line 189, in process_storage_error
    exec("raise error from None")   # pylint: disable=exec-used # nosec
  File "<string>", line 1, in <module>
azure.core.exceptions.ClientAuthenticationError:
Authentication failure. This may be caused by either invalid account key, connection string or sas token value provided for your storage account.

Content: <?xml version="1.0" encoding="utf-8"?><Error><Code>AuthenticationFailed</Code><Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
RequestId:aa1cbba8-201e-00da-1b37-0393fd000000
Time:2023-10-20T09:25:42.0917388Z</Message><AuthenticationErrorDetail>The MAC signature found in the HTTP request 'YK4MeUNyJCkQepG1wQJ4skGqgX29UOgYxiZ2OP3ZPtA=' is not the same as any computed signature. Server used following string to sign: 'GET











x-ms-client-request-id:a3502832-6f2a-11ee-8893-acbc327e0407
x-ms-date:Fri, 20 Oct 2023 09:25:41 GMT
x-ms-range:bytes=0-33554431
x-ms-version:2022-11-02
/devstoreaccount1/uploads/manual-upload.jpg'.</AuthenticationErrorDetail></Error>

cli.azure.cli.core.azclierror:
Authentication failure. This may be caused by either invalid account key, connection string or sas token value provided for your storage account.

az_command_data_logger:
Authentication failure. This may be caused by either invalid account key, connection string or sas token value provided for your storage account.

cli.knack.cli: Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x10be213f0>]
az_command_data_logger: exit code: 1
cli.__main__: Command ran in 1.391 seconds (init: 0.134, invoke: 1.257)
telemetry.main: Begin splitting cli events and extra events, total events: 1
telemetry.client: Accumulated 0 events. Flush the clients.
telemetry.main: Finish splitting cli events and extra events, cli events: 1
telemetry.save: Save telemetry record of length 3727 in cache
telemetry.main: Begin creating telemetry upload process.
telemetry.process: Creating upload process: "/usr/local/Cellar/azure-cli/2.53.0/libexec/bin/python /usr/local/Cellar/azure-cli/2.53.0/libexec/lib/python3.10/site-packages/azure/cli/telemetry/__init__.py /Users/ziyaddin/.azure"
telemetry.process: Return from creating process
telemetry.main: Finish creating telemetry upload process.

Expected behavior

The download request should be sent to https://127.0.0.1:10000/devstoreaccount1 instead of https://devstoreaccount1.blob.core.windows.net:443.

Environment Summary

azure-cli 2.53.0

core 2.53.0
telemetry 1.1.0

Dependencies:
msal 1.24.0b2
azure-mgmt-resource 23.1.0b2

Python location '/usr/local/Cellar/azure-cli/2.53.0/libexec/bin/python'
Extensions directory '/Users/ziyaddin/.azure/cliextensions'

Python (Darwin) 3.10.13 (main, Aug 24 2023, 22:46:18) [Clang 14.0.0 (clang-1400.0.29.202)]

Additional context

No response

[yonzhan]

Thank you for opening this issue, we will look into it.

[calvinhzy]

Hi @ziyaddin ,
May I ask if this is for local network access only? How was the blob storage created so I can try to reproduce this?

[ziyaddin]

Hi @calvinhzy ,

Yes, it's for local network access only. The blob storage was created via Azurite's Docker container.