warning downloading large files from cosmos

[hieuhoang]

ran this
az dls fs download ....(PM me for exact command, I'm MSFT)
I get a lot of warning eg
... errored, RuntimeError('Max number of ADL retries exceeded: exception AuthenticationError("Can't find token from MSAL cache.",)',)

However, download completed and created the correct file size but unsure if it's corrupted or not

[yonzhan]

@evelyn-ys for awareness

[navba-MSFT]

@hieuhoang Thanks for sharing the details of the issue on #21576. I am pasting the details here so that it can be looked further by our Service team.

Issue:
You are running the below Az CLI command from bash, Ubuntu 20.04 and it fails with the below error:

Command:

az dls fs download --account msr-XXXXXX-c08 --source-path /local/XXXXX/sidestream.en.1.gz --destination-path ./sidestream.en.1.gz

The command continues to run but eventually gives a corrupt file. The file is corrupt since the MD5checksum is incorrect on the downloaded file. The issue happens all the time with big files (>100GB). It works fine for the smaller files.

You are unable to reproduce the issue with --debug parameter.

Issue reproduced at : 11-Mar-2022 5:46 AM UTC time

Error:

WARNING: Command group 'dls' is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus
Alive[######################################################## ] 87.5589%
Alive[######################################################## ] 87.8041%
Alive[######################################################## ] 88.0494%
Alive[######################################################## ] 88.2947%ERROR: Download failed ./sidestream.en.1.gz; RuntimeError('Max number of ADL retries exceeded: exception AuthenticationError("Can't find token from MSAL cache.")')
ERROR: Download failed ./sidestream.en.1.gz; RuntimeError('Max number of ADL retries exceeded: exception AuthenticationError("Can't find token from MSAL cache.")')
ERROR: Download failed ./sidestream.en.1.gz; RuntimeError('Max number of ADL retries exceeded: exception AuthenticationError("Can't find token from MSAL cache.")')
ERROR: Download failed ./sidestream.en.1.gz; RuntimeError('Max number of ADL retries exceeded: exception AuthenticationError("Can't find token from MSAL cache.")')
ERROR: Download failed ./sidestream.en.1.gz; RuntimeError('Max number of ADL retries exceeded: exception AuthenticationError("Can't find token from MSAL cache.")')
ERROR: Download failed ./sidestream.en.1.gz; RuntimeError('Max number of ADL retries exceeded: exception AuthenticationError("Can't find token from MSAL cache.")')
ERROR: Download failed ./sidestream.en.1.gz; RuntimeError('Max number of ADL retries exceeded: exception AuthenticationError("Can't find token from MSAL cache.")')
ERROR: Download failed ./sidestream.en.1.gz; RuntimeError('Max number of ADL retries exceeded: exception AuthenticationError("Can't find token from MSAL cache.")')
<--Trimmed-->
Alive[########################################################## ] 91.9063%
Alive[########################################################## ] 92.1516%
Alive[########################################################### ] 92.3968%ERROR: Transfer failed: local/XXXXXX/sidestream.en.1.gz -> ./sidestream.en.1.gz
ERROR: local/XXXXXX/sidestream.en.1.gz -> ./sidestream.en.1.gz, chunk ./sidestream.en.1.gz 96636764160: errored, RuntimeError('Max number of ADL retries exceeded: exception AuthenticationError("Can't find token from MSAL cache.")')
ERROR: local/XXXXXX/sidestream.en.1.gz -> ./sidestream.en.1.gz, chunk ./sidestream.en.1.gz 96905199616: errored, RuntimeError('Max number of ADL retries exceeded: exception AuthenticationError("Can't find token from MSAL cache.")')
ERROR:local/XXXXXX/sidestream.en.1.gz -> ./sidestream.en.1.gz, chunk ./sidestream.en.1.gz 97173635072: errored, RuntimeError('Max number of ADL retries exceeded: exception AuthenticationError("Can't find token from MSAL cache.")')
ERROR: local/XXXXXX/sidestream.en.1.gz -> ./sidestream.en.1.gz, chunk ./sidestream.en.1.gz 106837311488: errored, RuntimeError('Max number of ADL retries exceeded: exception AuthenticationError("Can't find token from MSAL cache.")')
ERROR: The command failed with an unexpected error. Here is the traceback:
ERROR: One more more exceptions occured during transfer, resulting in an incomplete transfer.
local/XXXXXX/sidestream.en.1.gz -> ./sidestream.en.1.gz, chunk ./sidestream.en.1.gz 106837311488: errored, RuntimeError('Max number of ADL retries exceeded: exception AuthenticationError("Can't find token from MSAL cache.")')
Traceback (most recent call last):
File "/opt/az/lib/python3.8/site-packages/knack/cli.py", line 231, in invoke
cmd_result = self.invocation.execute(args)
File "/opt/az/lib/python3.8/site-packages/azure/cli/core/commands/init.py", line 658, in execute
raise ex
File "/opt/az/lib/python3.8/site-packages/azure/cli/core/commands/init.py", line 721, in _run_jobs_serially
results.append(self._run_job(expanded_arg, cmd_copy))
File "/opt/az/lib/python3.8/site-packages/azure/cli/core/commands/init.py", line 692, in _run_job
result = cmd_copy(params)
File "/opt/az/lib/python3.8/site-packages/azure/cli/core/commands/init.py", line 328, in call
return self.handler(*args, **kwargs)
File "/opt/az/lib/python3.8/site-packages/azure/cli/core/commands/command_operation.py", line 121, in handler
return op(**command_args)
File "/opt/az/lib/python3.8/site-packages/azure/cli/command_modules/dls/custom.py", line 200, in download_from_adls
ADLDownloader(
File "/opt/az/lib/python3.8/site-packages/azure/datalake/store/multithread.py", line 151, in init
self.run()
File "/opt/az/lib/python3.8/site-packages/azure/datalake/store/multithread.py", line 283, in run
self.client.run(nthreads, monitor, before_start=touch)
File "/opt/az/lib/python3.8/site-packages/azure/datalake/store/transfer.py", line 525, in run
raise DatalakeIncompleteTransferException('One more more exceptions occured during transfer, resulting in an incomplete transfer. \n\n List of exceptions and errors:\n {}'.format('\n'.join(error_list)))
azure.datalake.store.exceptions.DatalakeIncompleteTransferException: One more more exceptions occured during transfer, resulting in an incomplete transfer.
List of exceptions and errors:
local/XXXXXX/sidestream.en.1.gz /sidestream.en.1.gz -> ./sidestream.en.1.gz, chunk ./sidestream.en.1.gz 96636764160: errored, RuntimeError('Max number of ADL retries exceeded: exception AuthenticationError("Can't find token from MSAL cache.")')

You don't have the storage account details ( like subscription id / diagnostic logs etc) since you don't own this Storage account.

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @sumantmehtams.

Issue Details

---

ran this
az dls fs download ....(PM me for exact command, I'm MSFT)
I get a lot of warning eg
... errored, RuntimeError('Max number of ADL retries exceeded: exception AuthenticationError("Can't find token from MSAL cache.",)',)

However, download completed and created the correct file size but unsure if it's corrupted or not

Author:	hieuhoang
Assignees:	evelyn-ys
Labels:	Service Attention, Data Lake Store, customer-reported
Milestone:	Backlog

[YinTatMSR]

I am from Microsoft. I have the same issue. What is the correct way to download from cosmos beyond keep trying to repeat until it success? My alias is yintatlee.

[hieuhoang]

@navba-MSFT has this issue still not been resolved? It;s corrupting data. I can't use this service and can't recommend it to anyone until this is resolved