Azure CLI connection timeout when running a long test using the ScenarioTest framework

[patrykkulik-microsoft]

Describe the bug

My team is in the process of creating a new Azure CLI extension. As part of creating this extension, we wanted to add integration tests which would test our new commands. According to this document: https://github.com/Azure/azure-cli/blob/dev/doc/authoring_tests.md in the azure-cli repository, I should be able to author integration tests by taking advantage of the ScenarioTest class.

When I run azdev test aosm the publish command in our extension manages to deploy a few resources but fails after a few minutes with the error attached below.

Interestingly, if I reduce the number of deployments that need to be done by this command, the command manages to reach completion. This (and the error) suggest to me that the testing framework implements some sort of timeout which I cannot modify and which does not allow my command to complete running and succeed. Is this correct? Is there anything I can do to remove this timeout condition?

Related command

azdev test aosm

Errors

self = <azure.core.pipeline.transport._requests_basic.RequestsTransport object at 0x7f60b2b5b580>
request = <HttpRequest [POST], url: 'REDACTED'>
kwargs = {'stream': False}, response = None
error = ServiceResponseError("('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))")
connection_timeout = 300, timeout = (300, 300), read_timeout = 300

    def send(self, request, **kwargs): # type: ignore
        """Send request object according to configuration.

        :param request: The request object to be sent.
        :type request: ~azure.core.pipeline.transport.HttpRequest
        :return: An HTTPResponse object.
        :rtype: ~azure.core.pipeline.transport.HttpResponse

        :keyword requests.Session session: will override the driver session and use yours.
         Should NOT be done unless really required. Anything else is sent straight to requests.
        :keyword dict proxies: will define the proxy to use. Proxy is a dict (protocol, url)
        """
        self.open()
        response = None
        error = None    # type: Optional[AzureErrorUnion]

        try:
            connection_timeout = kwargs.pop('connection_timeout', self.connection_config.timeout)

            if isinstance(connection_timeout, tuple):
                if 'read_timeout' in kwargs:
                    raise ValueError('Cannot set tuple connection_timeout and read_timeout together')
                _LOGGER.warning("Tuple timeout setting is deprecated")
                timeout = connection_timeout
            else:
                read_timeout = kwargs.pop('read_timeout', self.connection_config.read_timeout)
                timeout = (connection_timeout, read_timeout)
            response = self.session.request(  # type: ignore
                request.method,
                request.url,
                headers=request.headers,
                data=request.data,
                files=request.files,
                verify=kwargs.pop('connection_verify', self.connection_config.verify),
                timeout=timeout,
                cert=kwargs.pop('connection_cert', self.connection_config.cert),
                allow_redirects=False,
                **kwargs)
            response.raw.enforce_content_length = True

        except (urllib3.exceptions.NewConnectionError, urllib3.exceptions.ConnectTimeoutError) as err:
            error = ServiceRequestError(err, error=err)
        except requests.exceptions.ReadTimeout as err:
            error = ServiceResponseError(err, error=err)
        except requests.exceptions.ConnectionError as err:
            if err.args and isinstance(err.args[0], urllib3.exceptions.ProtocolError):
                error = ServiceResponseError(err, error=err)
            else:
                error = ServiceRequestError(err, error=err)
        except requests.exceptions.ChunkedEncodingError as err:
            msg = err.__str__()
            if 'IncompleteRead' in msg:
                _LOGGER.warning("Incomplete download: %s", err)
                error = IncompleteReadError(err, error=err)
            else:
                _LOGGER.warning("Unable to stream download: %s", err)
                error = HttpResponseError(err, error=err)
        except requests.RequestException as err:
            error = ServiceRequestError(err, error=err)

        if error:
>           raise error
E           azure.core.exceptions.ServiceResponseError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

../../../../.virtualenvs/az-cli-env/lib/python3.8/site-packages/azure/core/pipeline/transport/_requests_basic.py:361: ServiceResponseError

Issue script & Debug output

no different than the above error message

Expected behavior

I expect the test to pass.

Environment Summary

azure-cli 2.48.1 *

core 2.48.1 *
telemetry 1.0.8

Extensions:
connectedk8s 1.3.5
customlocation 0.1.3
k8s-extension 1.3.6
resource-graph 2.1.0
aosm 0.1.0 (dev) /home/developer/Projects/azure-cli-extensions/src/aosm

Dependencies:
msal 1.20.0
azure-mgmt-resource 22.0.0

Python location '/home/developer/.virtualenvs/az-cli-env/bin/python'
Extensions directory '/home/developer/.azure/cliextensions'
Development extension sources:
/home/developer/Projects/azure-cli-extensions

Python (Linux) 3.8.10 (default, Mar 13 2023, 10:26:41)
[GCC 9.4.0]

Legal docs and information: aka.ms/AzureCliLegal

Additional context

No response

[azure-client-tools-bot-prd]

Hi @patrykkulik-microsoft,

2.48.1 is not the latest Azure CLI(2.50.0).

Please upgrade to the latest Azure CLI version by following https://learn.microsoft.com/en-us/cli/azure/update-azure-cli.

[yonzhan]

Thank you for opening this issue, we will look into it.

[wangzelin007]

Hi @patrykkulik-microsoft,
I don't think we put a timeout limit on tests, like this one running for more than an hour.

How long will your test run?
Can your test be split into multiple smaller tests?
Is there a PR for me to debug?

[patrykkulik-microsoft]

@wangzelin007 thank you for your reply.

The limit that that I am hitting is not on the test scope but on the self.cmd scope. One command is running for about 10 minutes and then I hit the error which I pasted above.

The test itself runs for about 20min but as I said the test is not the problem and the command cannot be split into smaller commands.

We have a PR set to the azure-cli-extensions repo: Azure/azure-cli-extensions#6426

The problematic test is src/aosm/azext_aosm/tests/latest/test_aosm_vnf_publish_and_delete.py,
Note: in that PR, this test has been commented out and I have added a while loop to try the command twice which makes it pass so you will need to remove the while loop to see the error. I would be happy to connect with you to help you reproduce this error.

[evelyn-ys]

If one http request needs to take so long as >10 min, why don't you return a poller for continuously querying status? The timeout limit comes from azure-core which is adopted by all clients, not only cli

[patrykkulik-microsoft]

It is not a single http request that needs 10minutes. I am running a self.cmd() method of the ScenarioTest class. I am using this method to run a command from our CLI extension which publishes multiple resources in Azure. This commands runs multiple HTTP requests. None of these individually take longer than 10mins but all of them together do.

I am hitting an error during the runtime of this command with multiple HTTP requests, which seems to happen because the CLI command as a whole takes too long rather than any individual request. In fact, if I run the command twice, the second time in runs, some resources have already been published by it in the first run before the error happened so the second run takes less time and passes fine.