Support setting timeout for Azure CLI commands

[jiasli]

Extension developer verbatim:

Recently we encountered an issue where the post call made using send_raw_request function got stuck and it didn’t return any response and because of that onboarding was stuck in the same state for 6 days for a partner team. When they killed that job, another instance of the job came up and then onboarding was successful.

We want to know

1. what can be the reasons this function might get stuck
2. if there is any default timeout in this function
3. Can we pass some timeout parameter

Link to our code where we are using this for making the post call - https://github.com/Azure/azure-cli-extensions/blob/cbf3346a7ecb624ad14f086f92b280a5ac927e8b/src/connectedk8s/azext_connectedk8s/_utils.py#L416

[jiasli]

send_raw_request may get stuck at 2 operations:

1. Acquiring access token from MSAL, which acquires access token from AAD:

   azure-cli/src/azure-cli-core/azure/cli/core/util.py

   Line 984 in 5757778

   token_info, _, _ = profile.get_raw_token(resource, subscription=token_subscription)

2. Calling the specified url:

   azure-cli/src/azure-cli-core/azure/cli/core/util.py

   Line 1003 in 5757778

   r = s.send(prepped, **settings)

Currently, neither has a preconfigured timeout nor supports timeout parameter. Additionally, all CLI commands can get stuck when retrieving tokens from MSAL when connection to AAD is required, such as az account get-access-token --resource-type ms-graph.

Underlying libraries of Azure CLI support timeout parameter, but:

1. requests has no default timeout value: https://requests.readthedocs.io/en/latest/user/advanced/#timeouts
2. MSAL has no default timeout value: https://github.com/AzureAD/microsoft-authentication-library-for-python/blob/dev/msal/application.py#L171
3. Azure Python SDK has default timeout of 300 seconds: https://github.com/Azure/azure-sdk-for-python/blob/8176b0f2092f57ed6df9d46bc4e7ab57b4bf065a/sdk/core/azure-core/azure/core/configuration.py#L120-L121

They support timeout because they are individual developer libraries. However, if Azure CLI (as a user-facing tool) wants to support timeout, CLI needs to consider as a whole

1. Whether to support different timeout values for different services (ARM, AAD, MS Graph, Storage data-plane, etc)
2. Whether to support customizing timeout:
   • Same timeout for different services
   • Different timeout for different services
3. What the best default timeout for different services are

Workaround

The workaround for now is to set timeout for the Azure CLI command itself: https://stackoverflow.com/questions/687948/timeout-a-command-in-bash-without-unnecessary-delay

For example, in Bash:

$ timeout 1 az group list
$ echo $?
124

The timed out command will exit with 124.

[rayluo]

2. MSAL has no default timeout value

There is no one-size-fits-all timeout value, and MSAL itself does not have a preference, therefore, MSAL honors the default timeout behavior of your http stack (which is typically requests). And MSAL allows you to customize the timeout behavior.

[sirireddy12]

@jiasli any updates on this issue?

[rohan-dassani]

Business Justification - We use this send_raw_request to fetch the path for helm chart to install the agents we require for our connectedk8s extension. In the absence of a timeout , the post call that we make through this function , if for some reason is not able to get any response the request got stuck in between , it stays in the stuck state forever. This is exactly what happened with a partner team where we saw that their onboarding was stuck in the same state for 6 days. To get out of this situation they had to manually kill the job and another instance came up and the onboarding went through.
If a customer gets into the same situation they may have to interrupt the onboarding process after waiting for long and we wont even get the right telemetry as to why the onboarding was stopped in between. Having a timeout will help us understand that there was a problem in fetching the helm chart link and we will be able to show the user what happened.