az storage fs file exists utf-8' codec can't encode characters in position

[bb980701]

This is autogenerated. Please review and update as needed.

Describe the bug

Command Name
az storage fs file exists

Errors:

ERROR: The command failed with an unexpected error. Here is the traceback:
ERROR: 'utf-8' codec can't encode characters in position 35-36: surrogates not allowed
Traceback (most recent call last):
  File "/opt/az/lib/python3.6/site-packages/knack/cli.py", line 231, in invoke
    cmd_result = self.invocation.execute(args)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 658, in execute
    raise ex
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 721, in _run_jobs_serially
    results.append(self._run_job(expanded_arg, cmd_copy))
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 713, in _run_job
    return cmd_copy.exception_handler(ex)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/command_modules/storage/__init__.py", line 371, in new_handler
    first(ex)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/command_modules/storage/__init__.py", line 370, in new_handler
    raise ex
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 692, in _run_job
    result = cmd_copy(params)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 328, in __call__
    return self.handler(*args, **kwargs)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/command_operation.py", line 112, in handler
    client = self.client_factory(self.cli_ctx, command_args) if self.client_factory else None
  File "/opt/az/lib/python3.6/site-packages/azure/cli/command_modules/storage/_client_factory.py", line 287, in cf_adls_file
    file_path=kwargs.pop('path', None))
  File "/opt/az/lib/python3.6/site-packages/azure/multiapi/storagev2/filedatalake/v2020_02_10/_data_lake_service_client.py", line 494, in get_file_client
    key_resolver_function=self.key_resolver_function)
  File "/opt/az/lib/python3.6/site-packages/azure/multiapi/storagev2/filedatalake/v2020_02_10/_data_lake_file_client.py", line 75, in __init__
    credential=credential, **kwargs)
  File "/opt/az/lib/python3.6/site-packages/azure/multiapi/storagev2/filedatalake/v2020_02_10/_path_client.py", line 71, in __init__
    credential=credential, _hosts=blob_hosts, **kwargs)
  File "/opt/az/lib/python3.6/site-packages/azure/multiapi/storagev2/blob/v2020_02_10/_blob_client.py", line 167, in __init__
    self._client = AzureBlobStorage(self.url, pipeline=self._pipeline)
  File "/opt/az/lib/python3.6/site-packages/azure/multiapi/storagev2/blob/v2020_02_10/_shared/base_client.py", line 136, in url
    return self._format_url(self._hosts[self._location_mode])
  File "/opt/az/lib/python3.6/site-packages/azure/multiapi/storagev2/blob/v2020_02_10/_blob_client.py", line 178, in _format_url
    quote(self.blob_name, safe='~/'),
  File "/opt/az/lib/python3.6/urllib/parse.py", line 819, in quote
    string = string.encode(encoding, errors)
UnicodeEncodeError: 'utf-8' codec can't encode characters in position 35-36: surrogates not allowed

To Reproduce:

Steps to reproduce the behavior. Note that argument values have been redacted, as they may contain sensitive information.

• Set a files on a blob containing é è french character like 131318-(n) Hamilton Densité.JPG
• az storage fs file exists --path {} --account-name {} --account-key {} --file-system {}

Expected Behavior

should return true or false

under windows os , no problem

Environment Summary

Linux-5.4.72-microsoft-standard-WSL2-x86_64-with-debian-bullseye-sid, Ubuntu 20.04.3 LTS
Python 3.6.10
Installer: DEB

   "azure-cli": "2.33.1",
  "azure-cli-core": "2.33.1",
  "azure-cli-telemetry": "1.0.6",
  "extensions": {}

Additional Context

[yonzhan]

storage

[evelyn-ys]

Do you have any unusual characters in your path or file system name?

[bb980701]

Unusual for english speaker yes, usual in french :-) character like é è ç ° , all those character are seen as normal by the blob storage for file name

The strange stuff it is that under windows os with git terminal, no problem.
Same code under ubuntu docker , then problem arise

[evelyn-ys]

I guess the system need to have Unicode support for the language you use :)

[bb980701]

Linux-5.4.72-microsoft-standard-WSL2-x86_64-with-debian-bullseye-sid, Ubuntu 20.04.3 LTS is unicode and works normally with unicode file name , except when calling az command :-)

[evelyn-ys]

@jalauzon-msft Can you help take a look. It fails in azure-storage-blob:
https://github.com/Azure/azure-sdk-for-python/blob/94db701456044bdd00c1518f950ff8da93f4a8d8/sdk/storage/azure-storage-blob/azure/storage/blob/_blob_client.py#L190

[jalauzon-msft]

Hi @evelyn-ys Yishi, I did some testing on this and can confirm that the Datalake SDK does support these types of Unicode characters on Linux (tested on Ubuntu 18.04). I was able to create, update, check existence, and read a file with the name "131318-(n) Hamilton Densité.JPG". I tried both hardcoding the name in a string and accepting the file name from the terminal.

My guess is that this has something to do with how CLI reads in the file path string before sending to the SDK. I did a bit of research on this topic/error, and it seems that reading in file paths on Linux can add surrogates for non-supported characters when trying to convert to Unicode. The surrogates will then break any attempts to decode the string, like when the SDK is trying to URL encode the string. Here is a Stack Overflow that I found on the issue.

It seems like the CLI may need to do something to handle file names with non-default encodings. The Stack Overflow has a couple of suggestions.