on MacOS, `az functionapp deployment source config-zip` causes `Cannot allocate memory` often

[s-nakagaki]

Describe the bug

Trying to deploy a windows function via azure-cli on VSCode zsh terminal of macOS.
Cannot allocate memory happens so many times. (NOT always)

According to --debug result, polling requests causes memory allocation error.

My Mac has 24 GB memory and there are 5GB free memory at least.
Of course, dotnet build is succeeded at local.

There is no error in the log of deploy center of azure function.

Copying file from "C:\home\.nuget\...
Copying file from "C:\home\.nuget\...
Copying file from "C:\home\.nuget\...
Copying file from "C:\home\.nuget\...
Copying file from "C:\home\.nuget\...
Creating directory "C:\local\Temp\zipdeploy\extracted\...
Copying file from "C:\home\.nuget\...
Creating directory "C:\local\Temp\zipdeploy\extracted\...
Copying file from "C:\home\.nuget\...
Thread was being aborted.
Copying file from "C:\home\.nuget\...
Thread was being aborted.

Related command

az functionapp deployment source config-zip
-g ${RESOURCE_GROUP}
-n ${FUNCTION_NAME}
--build-remote true
--src ./.zip/${FUNCTION_NAME}.zip
--timeout 1200

Errors

Getting scm site credentials for zip deployment
Starting zip deployment. This operation can take a while to complete ...
Deployment endpoint responded with status code 202
The command failed with an unexpected error. Here is the traceback:
HTTPSConnectionPool(host='func-hoge.scm.azurewebsites.net', port=443): Max retries exceeded with url: /api/deployments/latest (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x10839e6d0>: Failed to establish a new connection: [Errno 12] Cannot allocate memory'))
Traceback (most recent call last):
File "/opt/homebrew/Cellar/azure-cli/2.62.0/libexec/lib/python3.11/site-packages/urllib3/connection.py", line 174, in _new_conn
conn = connection.create_connection(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/azure-cli/2.62.0/libexec/lib/python3.11/site-packages/urllib3/util/connection.py", line 95, in create_connection
raise err
File "/opt/homebrew/Cellar/azure-cli/2.62.0/libexec/lib/python3.11/site-packages/urllib3/util/connection.py", line 85, in create_connection
sock.connect(sa)
OSError: [Errno 12] Cannot allocate memory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/homebrew/Cellar/azure-cli/2.62.0/libexec/lib/python3.11/site-packages/urllib3/connectionpool.py", line 715, in urlopen
httplib_response = self._make_request(
^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/azure-cli/2.62.0/libexec/lib/python3.11/site-packages/urllib3/connectionpool.py", line 404, in _make_request
self._validate_conn(conn)
File "/opt/homebrew/Cellar/azure-cli/2.62.0/libexec/lib/python3.11/site-packages/urllib3/connectionpool.py", line 1060, in _validate_conn
conn.connect()
File "/opt/homebrew/Cellar/azure-cli/2.62.0/libexec/lib/python3.11/site-packages/urllib3/connection.py", line 363, in connect
self.sock = conn = self._new_conn()
^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/azure-cli/2.62.0/libexec/lib/python3.11/site-packages/urllib3/connection.py", line 186, in _new_conn
raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x10839e6d0>: Failed to establish a new connection: [Errno 12] Cannot allocate memory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/homebrew/Cellar/azure-cli/2.62.0/libexec/lib/python3.11/site-packages/requests/adapters.py", line 564, in send
resp = conn.urlopen(
^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/azure-cli/2.62.0/libexec/lib/python3.11/site-packages/urllib3/connectionpool.py", line 801, in urlopen
retries = retries.increment(
^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/azure-cli/2.62.0/libexec/lib/python3.11/site-packages/urllib3/util/retry.py", line 594, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='func-hoge.scm.azurewebsites.net', port=443): Max retries exceeded with url: /api/deployments/latest (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x10839e6d0>: Failed to establish a new connection: [Errno 12] Cannot allocate memory'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/homebrew/Cellar/azure-cli/2.62.0/libexec/lib/python3.11/site-packages/knack/cli.py", line 233, in invoke
cmd_result = self.invocation.execute(args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/azure-cli/2.62.0/libexec/lib/python3.11/site-packages/azure/cli/core/commands/init.py", line 664, in execute
raise ex
File "/opt/homebrew/Cellar/azure-cli/2.62.0/libexec/lib/python3.11/site-packages/azure/cli/core/commands/init.py", line 731, in _run_jobs_serially
results.append(self._run_job(expanded_arg, cmd_copy))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/azure-cli/2.62.0/libexec/lib/python3.11/site-packages/azure/cli/core/commands/init.py", line 701, in _run_job
result = cmd_copy(params)
^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/azure-cli/2.62.0/libexec/lib/python3.11/site-packages/azure/cli/core/commands/init.py", line 334, in call
return self.handler(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/azure-cli/2.62.0/libexec/lib/python3.11/site-packages/azure/cli/core/commands/command_operation.py", line 121, in handler
return op(**command_args)
^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/azure-cli/2.62.0/libexec/lib/python3.11/site-packages/azure/cli/command_modules/appservice/custom.py", line 677, in enable_zip_deploy_functionapp
return enable_zip_deploy(cmd, resource_group_name, name, src, timeout, slot)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/azure-cli/2.62.0/libexec/lib/python3.11/site-packages/azure/cli/command_modules/appservice/custom.py", line 804, in enable_zip_deploy
response_body = _check_zip_deployment_status(cmd, resource_group_name, name, deployment_status_url,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/azure-cli/2.62.0/libexec/lib/python3.11/site-packages/azure/cli/command_modules/appservice/custom.py", line 5628, in _check_zip_deployment_status
response = requests.get(deployment_status_url, headers=headers,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/azure-cli/2.62.0/libexec/lib/python3.11/site-packages/requests/api.py", line 73, in get
return request("get", url, params=params, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/azure-cli/2.62.0/libexec/lib/python3.11/site-packages/requests/api.py", line 59, in request
return session.request(method=method, url=url, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/azure-cli/2.62.0/libexec/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/azure-cli/2.62.0/libexec/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/azure-cli/2.62.0/libexec/lib/python3.11/site-packages/requests/adapters.py", line 597, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='func-hoge.scm.azurewebsites.net', port=443): Max retries exceeded with url: /api/deployments/latest (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x10839e6d0>: Failed to establish a new connection: [Errno 12] Cannot allocate memory'))

Issue script & Debug output

urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443
urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585
cli.azure.cli.command_modules.appservice.custom: Running deployment command...
urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443
urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585
cli.azure.cli.command_modules.appservice.custom: Running deployment command...
urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443
urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585
cli.azure.cli.command_modules.appservice.custom: Running deployment command...
urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443
urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585
cli.azure.cli.command_modules.appservice.custom: Running deployment command...
urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443
urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585
cli.azure.cli.command_modules.appservice.custom: Running deployment command...
urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443
urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585
cli.azure.cli.command_modules.appservice.custom: Running deployment command...
urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443
urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585
cli.azure.cli.command_modules.appservice.custom: Running deployment command...
urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443
urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585
cli.azure.cli.command_modules.appservice.custom: Running deployment command...
urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443
urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585
cli.azure.cli.command_modules.appservice.custom: Running deployment command...
urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443
urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585
cli.azure.cli.command_modules.appservice.custom: Running deployment command...
urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443
urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585
cli.azure.cli.command_modules.appservice.custom: Running deployment command...
urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443
urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585
cli.azure.cli.command_modules.appservice.custom: Running deployment command...
urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443
urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585
cli.azure.cli.command_modules.appservice.custom: Running deployment command...
urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443
urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585
cli.azure.cli.command_modules.appservice.custom: Running deployment command...
urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443
urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585
cli.azure.cli.command_modules.appservice.custom: Running deployment command...
urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443
urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585
cli.azure.cli.command_modules.appservice.custom: Running deployment command...
urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443
urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585
cli.azure.cli.command_modules.appservice.custom: Running deployment command...
urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443
urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585
cli.azure.cli.command_modules.appservice.custom: Running deployment command...
urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443
urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585
cli.azure.cli.command_modules.appservice.custom: Running deployment command...
urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443
urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585
cli.azure.cli.command_modules.appservice.custom: Running deployment command...
urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443
urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585
cli.azure.cli.command_modules.appservice.custom: Running deployment command...
urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443
urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585
cli.azure.cli.command_modules.appservice.custom: Running deployment command...
urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443
urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585
cli.azure.cli.command_modules.appservice.custom: Running deployment command...
urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443
urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585
cli.azure.cli.command_modules.appservice.custom: Running deployment command...
urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443
urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585
cli.azure.cli.command_modules.appservice.custom: Running deployment command...
urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443
urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585
cli.azure.cli.command_modules.appservice.custom: Running deployment command...
urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443
urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585
cli.azure.cli.command_modules.appservice.custom: Running deployment command...
urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443
urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585
cli.azure.cli.command_modules.appservice.custom: Running deployment command...
urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443
urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585
cli.azure.cli.command_modules.appservice.custom: Running deployment command...
urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443
urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585
cli.azure.cli.command_modules.appservice.custom: Running deployment command...
urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443
urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585
cli.azure.cli.command_modules.appservice.custom: Running deployment command...
urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443
urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585
cli.azure.cli.command_modules.appservice.custom: Running deployment command...
urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443
urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585
cli.azure.cli.command_modules.appservice.custom: Running deployment command...
urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443
urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585
cli.azure.cli.command_modules.appservice.custom: Running deployment command...
urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443
urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585
cli.azure.cli.command_modules.appservice.custom: Running deployment command...
urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443
urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585
cli.azure.cli.command_modules.appservice.custom: Running deployment command...
urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443
urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585
cli.azure.cli.command_modules.appservice.custom: Running deployment command...
urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443
urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585
cli.azure.cli.command_modules.appservice.custom: Running deployment command...
urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443
urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585
cli.azure.cli.command_modules.appservice.custom: Running deployment command...
urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443
urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585
cli.azure.cli.command_modules.appservice.custom: Running deployment command...
urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443
urllib3.connectionpool: https://func-hoge.scm.azurewebsites.net:443 "GET /api/deployments/latest HTTP/1.1" 202 585
cli.azure.cli.command_modules.appservice.custom: Running deployment command...
urllib3.connectionpool: Starting new HTTPS connection (1): func-hoge.scm.azurewebsites.net:443
cli.azure.cli.core.azclierror: Traceback (most recent call last):
File "/opt/homebrew/Cellar/azure-cli/2.62.0/libexec/lib/python3.11/site-packages/urllib3/connection.py", line 174, in _new_conn
conn = connection.create_connection(

Expected behavior

no Cannot allocate memory

Environment Summary

azure-cli 2.62.0

core 2.62.0
telemetry 1.1.0

Extensions:
account 0.2.5

Dependencies:
msal 1.28.1
azure-mgmt-resource 23.1.1

Python location '/opt/homebrew/Cellar/azure-cli/2.62.0/libexec/bin/python'
Extensions directory '/Users/hoge/.azure/cliextensions'

Python (Darwin) 3.11.9 (main, Apr 2 2024, 08:25:04) [Clang 15.0.0 (clang-1500.3.9.4)]

Legal docs and information: aka.ms/AzureCliLegal

Your CLI is up-to-date.

Additional context

No response

[yonzhan]

Thank you for opening this issue, we will look into it.

[github-actions]

Here are some similar issues that might help you. Please check if they can solve your problem.

• az functionapp deployment source config-zip fails #23593

[microsoft-github-policy-service]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @kamperiadis, @khkh-ms, @shreyabatra4.

[s-nakagaki]

This might happen only on VSCode zsh terminal. I'm not sure.
This happens on macOS native terminal too.

[s-nakagaki]

When this happened, log view of deploy center of azure function in azure portal said Too many requests. Please try again later..

I guess, the retrying sequence of azure cli might not wait for response and might retry to connect very soon without cleaning-up old connection. So at client side, there are so many memory allocation for connection, at server side, there are so many request to check the deploy status.