2.61.0 `az webapp deploy` for zip deploy polls for status of async deployment and never finishes

[gburkecw]

Describe the bug

Yesterday, on azcli 2.60.0, deploying to a Linux web app with a zip deploy package worked fine and had the following output:

Today on azcli 2.61.0, the command polls forever and never exits, even though the Kudu portal shows that the deployment completed:

Notice the new "polling the status of async deployment". I tried heeding the deprecation warning and switching to use az webapp deploy and I set --async true, but I still get the message "polling the status of async deployment" which never finishes:

Related command

az webapp deployment source config-zip --name "" --resource-group "" --slot "" --src "";

also tried (due to deprecation warning), tried async=true to try to get rid of polling:

az webapp deploy --async true --name "" --resource-group "" --slot "" --src-path "" --type zip;

Errors

ERROR: Timeout reached while tracking deployment status, however, the deployment operation is still on-going.

Issue script & Debug output

WARNING: cli.azure.cli.command_modules.appservice.custom: Status: Build successful. Time: 78(s)
DEBUG: cli.azure.cli.core.util: Found subscription ID b731a80f-7771-4552-b828-0508b3e54da8 in the URL https://management.azure.com/subscriptions/b731a80f-7771-4552-b828-0508b3e54da8/resourceGroups/rg-webapp-api-pr-eus/providers/Microsoft.Web/sites/cwappapipreus/slots/Deploying/deploymentStatus/2b7fbb1c-7c22-4ff8-9df1-d4bb9ae34a75?api-version=2023-01-01
DEBUG: cli.azure.cli.core.util: Retrieving token for resource https://management.core.windows.net/, subscription b731a80f-7771-4552-b828-0508b3e54da8
DEBUG: urllib3.util.retry: Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=None, status=None)
INFO: msal.authority: Initializing with Entra authority: https://login.microsoftonline.com/***
DEBUG: msal.authority: openid_config("https://login.microsoftonline.com/***/v2.0/.well-known/openid-configuration") = {'token_endpoint': 'https://login.microsoftonline.com/***/oauth2/v2.0/token', 'token_endpoint_auth_methods_supported': ['client_secret_post', 'private_key_jwt', 'client_secret_basic'], 'jwks_uri': 'https://login.microsoftonline.com/***/discovery/v2.0/keys', 'response_modes_supported': ['query', 'fragment', 'form_post'], 'subject_types_supported': ['pairwise'], 'id_token_signing_alg_values_supported': ['RS256'], 'response_types_supported': ['code', 'id_token', 'code id_token', 'id_token token'], 'scopes_supported': ['openid', 'profile', 'email', 'offline_access'], 'issuer': 'https://login.microsoftonline.com/***/v2.0', 'request_uri_parameter_supported': False, 'userinfo_endpoint': 'https://graph.microsoft.com/oidc/userinfo', 'authorization_endpoint': 'https://login.microsoftonline.com/***/oauth2/v2.0/authorize', 'device_authorization_endpoint': 'https://login.microsoftonline.com/***/oauth2/v2.0
DEBUG: msal.application: Broker enabled? None
DEBUG: cli.azure.cli.core.auth.msal_authentication: ServicePrincipalCredential.get_token: scopes=('https://management.core.windows.net//.default',), kwargs={}
DEBUG: msal.application: Cache hit an AT
DEBUG: msal.telemetry: Generate or reuse correlation_id: 0e47d629-b568-48fa-a28c-a230e5aa6750
INFO: cli.azure.cli.core.util: Request URL: 'https://management.azure.com/subscriptions/b731a80f-7771-4552-b828-0508b3e54da8/resourceGroups/rg-webapp-api-pr-eus/providers/Microsoft.Web/sites/cwappapipreus/slots/Deploying/deploymentStatus/2b7fbb1c-7c22-4ff8-9df1-d4bb9ae34a75?api-version=2023-01-01'
INFO: cli.azure.cli.core.util: Request method: 'GET'
INFO: cli.azure.cli.core.util: Request headers:
INFO: cli.azure.cli.core.util: 'User-Agent': 'python/3.11.8 (Linux-5.10.102.2-microsoft-standard-x86_64-with-glibc2.35) AZURECLI/2.61.0 (DEB)'
INFO: cli.azure.cli.core.util: 'Accept-Encoding': 'gzip, deflate'
INFO: cli.azure.cli.core.util: 'Accept': '/'
INFO: cli.azure.cli.core.util: 'Connection': 'keep-alive'
INFO: cli.azure.cli.core.util: 'x-ms-client-request-id': 'dcabce40-3f4f-4739-8e31-d63076b08ac0'
INFO: cli.azure.cli.core.util: 'CommandName': 'webapp deploy'
INFO: cli.azure.cli.core.util: 'ParameterSetName': '--async --name --resource-group --slot --src-path --type --debug'
INFO: cli.azure.cli.core.util: 'Authorization': '***'
INFO: cli.azure.cli.core.util: Request body:
INFO: cli.azure.cli.core.util: None
DEBUG: urllib3.connectionpool: Starting new HTTPS connection (1): management.azure.com:443
DEBUG: urllib3.connectionpool: https://management.azure.com:443 "GET /subscriptions/b731a80f-7771-4552-b828-0508b3e54da8/resourceGroups/rg-webapp-api-pr-eus/providers/Microsoft.Web/sites/cwappapipreus/slots/Deploying/deploymentStatus/2b7fbb1c-7c22-4ff8-9df1-d4bb9ae34a75?api-version=2023-01-01 HTTP/1.1" 202 568
INFO: cli.azure.cli.core.util: Response status: 202
INFO: cli.azure.cli.core.util: Response headers:
INFO: cli.azure.cli.core.util: 'Cache-Control': 'no-cache'
INFO: cli.azure.cli.core.util: 'Pragma': 'no-cache'
INFO: cli.azure.cli.core.util: 'Content-Length': '568'
INFO: cli.azure.cli.core.util: 'Content-Type': 'application/json'
INFO: cli.azure.cli.core.util: 'Expires': '-1'
INFO: cli.azure.cli.core.util: 'ETag': '"1DAABB9DCD07780"'
INFO: cli.azure.cli.core.util: 'Location': 'https://management.azure.com/subscriptions/b731a80f-7771-4552-b828-0508b3e54da8/resourceGroups/rg-webapp-api-pr-eus/providers/Microsoft.Web/sites/cwappapipreus/slots/Deploying/deploymentStatus/2b7fbb1c-7c22-4ff8-9df1-d4bb9ae34a75?api-version=2023-01-01&t=638519186[759](https://github.com/checkwriters/Checkwriters.Api/actions/runs/9177549517/job/25245810420#step:10:759)709296&c=MIIHhzCCBm-gAwIBAgITfATUDnhanTLRPeyZxwAABNQOeDANBgkqhkiG9w0BAQsFADBEMRMwEQYKCZImiZPyLGQBGRYDR0JMMRMwEQYKCZImiZPyLGQBGRYDQU1FMRgwFgYDVQQDEw9BTUUgSW5mcmEgQ0EgMDUwHhcNMjQwNTA5MTgwOTM3WhcNMjUwNTA0MTgwOTM3WjBAMT4wPAYDVQQDEzVhc3luY29wZXJhdGlvbnNpZ25pbmdjZXJ0aWZpY2F0ZS5tYW5hZ2VtZW50LmF6dXJlLmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAL9dgMYYraJFlILLieCu28kujI4Eh_o_VNUMODlVyQkeY76IALT8f-X5D0AbgZujawag12jT_t6b5XuEUL5d-K_TBnvCQbYD9roAD-qWL6Aw7DznLoWvMoAX4tn8ngoKup3mxnKUPJ9c09dr_CTsE6QjpcT4DYzTZ0jQ_Nsi7kY20vZZLm1FsTuPzkG8N4XnapOt6O2JPSgdtRa68z-kHJia0Z0zfCVBpqN4T4a_depQa2MOOXTmsPmm6J4GRhj6UdTJZ1TyBGGI3MmjNY6ih6M6NilZJWZuQXuNG1LQ_PonDW8K_rfK2zzobonkASWw2_T1KI_w2cZ2zfPKmQ0gmgECAwEAAaOCBHQwggRwMCcGCSsG
INFO: cli.azure.cli.core.util: 'Retry-After': '15'
INFO: cli.azure.cli.core.util: 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains'
INFO: cli.azure.cli.core.util: 'X-AspNet-Version': '4.0.30319'
INFO: cli.azure.cli.core.util: 'X-Powered-By': 'ASP.NET'
INFO: cli.azure.cli.core.util: 'x-ms-ratelimit-remaining-subscription-reads': '11999'
INFO: cli.azure.cli.core.util: 'x-ms-request-id': '4fa9a185-bad3-45bf-adf4-f4e1146fcdf1'
INFO: cli.azure.cli.core.util: 'x-ms-correlation-request-id': '4fa9a185-bad3-45bf-adf4-f4e1146fcdf1'
INFO: cli.azure.cli.core.util: 'x-ms-routing-request-id': 'EASTUS:20240521T200435Z:4fa9a185-bad3-45bf-adf4-f4e1146fcdf1'
INFO: cli.azure.cli.core.util: 'X-Content-Type-Options': 'nosniff'
INFO: cli.azure.cli.core.util: 'X-Cache': 'CONFIG_NOCACHE'
INFO: cli.azure.cli.core.util: 'X-MSEdge-Ref': 'Ref A: 60B6B503A30549908BD08B3D1F8FE146 Ref B: BL2AA2030104003 Ref C: 2024-05-21T20:04:35Z'
INFO: cli.azure.cli.core.util: 'Date': 'Tue, 21 May 2024 20:04:35 GMT'
INFO: cli.azure.cli.core.util: Response content:
INFO: cli.azure.cli.core.util: {"id":"/subscriptions/b731a80f-7771-4552-b828-0508b3e54da8/resourceGroups/rg-webapp-api-pr-eus/providers/Microsoft.Web/sites/cwappapipreus/slots/Deploying/deploymentStatus/2b7fbb1c-7c22-4ff8-9df1-d4bb9ae34a75","name":"2b7fbb1c-7c22-4ff8-9df1-d4bb9ae34a75","type":"Microsoft.Web/sites/slots/deploymentStatus","location":"East US","tags":{},"properties":{"deploymentId":"2b7fbb1c-7c22-4ff8-9df1-d4bb9ae34a75","status":"BuildSuccessful","numberOfInstancesInProgress":0,"numberOfInstancesSuccessful":0,"numberOfInstancesFailed":0,"failedInstancesLogs":null,"errors":null}}
WARNING: cli.azure.cli.command_modules.appservice.custom: Status: Build successful. Time: 94(s)

Expected behavior

Either the deprecated az webapp deployment source config-zip command should continue to work as it did in 2.60.0 (returns successful deployment immediately) or the az webapp deploy command should exhibit the documented behavior of --async true and return right away.

Also, since the deployment is successful, I would think that --async false would also work fine in this case.

Environment Summary

azure-cli 2.61.0

core 2.61.0
telemetry 1.1.0

Dependencies:
msal 1.28.0
azure-mgmt-resource 23.1.1

Python location '/opt/az/bin/python3'
Extensions directory '/root/.azure/cliextensions'

Python (Linux) 3.11.8 (main, May 16 2024, 03:47:28) [GCC 11.4.0]

Additional context

No response

[yonzhan]

Thank you for opening this issue, we will look into it.

[gburkecw]

I think what is probably happening here is that 2.61.0 uses api-version=2023-01-01 for the deploymentStatus API, and it is returning 202 instead of 200. Perhaps the Azure CLI is looking for a 200 result specifically. I confirmed with an older version of Azure CLI (2.43) that it is using an older version of the deploymentStatus API which is returning 200 instead of 202.

[btardif]

@tulikac can you take a look, this might be related to deployment status?

[mderriey]

We're facing the same issue on two separate pipelines running on GitHub Actions.

The new GitHub Actions runner image for Ubuntu 22.04 was recently updated from 20240516.1 to 20240526.1 which bumped the Azure CLI from v2.60.0 to v2.61.0.

We'll investigate manually installing v2.60.0 at the start of the job to work around this.

[cjhannon84]

@mderriey We also faced this issue, I downgraded the cli version we are using and all's good while we await a fix in the latest version.

[mderriey]

Here's the step we use in our GitHub Actions deployment jobs running on ubuntu-latest until this issue is fixed:

- name: Install Azure CLI v2.60.0
  shell: bash
  run: |
    AZ_DIST=$(lsb_release -cs)
    AZ_VER=2.60.0
    sudo apt-get install -y --allow-downgrades azure-cli=${AZ_VER}-1~${AZ_DIST}