Skip to content

az extension add --name azure-devops reaches out to pypi.org and flagged in build pipeline #29109

@feiyushi

Description

@feiyushi

Describe the bug

Due to internal security push related to OSS network isolation, endpoint such as pypi.org is flagged during the build. The flagged task uses az extension add --name azure-devops which reaches out to https://pypi.org/simple/distro/ to search for versions of distro.

Related command

az extension add --name azure-devops

Errors

1 location(s) to search for versions of distro:
*https://pypi.org/simple/distro/
Fetching project page and analyzing links: https://pypi.org/simple/distro/
Getting page https://pypi.org/simple/distro/
Found index url https://pypi.org/simple/
Starting new HTTPS connection (1): pypi.org:443
https://pypi.org:443 "GET /simple/distro/ HTTP/1.1" 200 4086

Issue script & Debug output

cli.knack.cli: Command arguments: ['extension', 'add', '--name', 'azure-devops', '--debug']
cli.knack.cli: init debug log:
Enable color in terminal.
cli.knack.cli: Event: Cli.PreExecute []
cli.knack.cli: Event: CommandParser.OnGlobalArgumentsCreate [<function CLILogging.on_global_arguments at 0x7f636dd90040>, <function OutputProducer.on_global_arguments at 0x7f636dd36200>, <function CLIQuery.on_global_arguments at 0x7f636db4fce0>]
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableCreate []
cli.azure.cli.core: Modules found from index for 'extension': ['azure.cli.command_modules.extension']
cli.azure.cli.core: Loading command modules:
cli.azure.cli.core: Name Load Time Groups Commands
cli.azure.cli.core: extension 0.001 1 7
cli.azure.cli.core: Total (1) 0.001 1 7
cli.azure.cli.core: These extensions are not installed and will be skipped: ['azext_ai_examples', 'azext_next']
cli.azure.cli.core: Loading extensions:
cli.azure.cli.core: Name Load Time Groups Commands Directory
cli.azure.cli.core: Total (0) 0.000 0 0
cli.azure.cli.core: Loaded 1 groups, 7 commands.
cli.azure.cli.core: Found a match in the command table.
cli.azure.cli.core: Raw command : extension add
cli.azure.cli.core: Command table: extension add
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableTruncate [<function AzCliLogging.init_command_file_logging at 0x7f636cde4e00>]
cli.azure.cli.core.azlogging: metadata file logging enabled - writing logs to '/root/.azure/commands/2024-06-05.22-14-41.extension_add.2379.log'.
az_command_data_logger: command args: extension add --name {} --debug
cli.knack.cli: Event: CommandInvoker.OnPreArgumentLoad [<function register_global_subscription_argument..add_subscription_parameter at 0x7f636ce39ee0>]
cli.knack.cli: Event: CommandInvoker.OnPostArgumentLoad []
cli.knack.cli: Event: CommandInvoker.OnPostCommandTableCreate [<function register_ids_argument..add_ids_arguments at 0x7f636ce51120>, <function register_cache_arguments..add_cache_arguments at 0x7f636ce51260>]
cli.knack.cli: Event: CommandInvoker.OnCommandTableLoaded []
cli.knack.cli: Event: CommandInvoker.OnPreParseArgs []
cli.knack.cli: Event: CommandInvoker.OnPostParseArgs [<function OutputProducer.handle_output_argument at 0x7f636dd362a0>, <function CLIQuery.handle_query_parameter at 0x7f636db4fd80>, <function register_ids_argument..parse_ids_arguments at 0x7f636ce511c0>]
cli.azure.cli.core.extension.operations: Default enabled including preview versions for extension installation now. Disabled in future release. Use '--allow-preview true' to enable it specifically if needed. Use '--allow-preview false' to install stable version only.
urllib3.connectionpool: Starting new HTTPS connection (1): aka.ms:443
urllib3.connectionpool: https://aka.ms:443 "GET /azure-cli-extension-index-v1 HTTP/1.1" 301 0
urllib3.connectionpool: Starting new HTTPS connection (1): azcliextensionsync.blob.core.windows.net:443
urllib3.connectionpool: https://azcliextensionsync.blob.core.windows.net:443 "GET /index1/index.json HTTP/1.1" 200 4128036
cli.azure.cli.core.extension._resolve: Candidates ['azure_devops-0.12.0-py2.py3-none-any.whl', 'azure_devops-0.17.0-py2.py3-none-any.whl', 'azure_devops-0.21.0-py2.py3-none-any.whl', 'azure_devops-0.26.0-py2.py3-none-any.whl', 'azure_devops-1.0.0-py2.py3-none-any.whl', 'azure_devops-1.0.1-py2.py3-none-any.whl']
cli.azure.cli.core.extension._resolve: Candidates ['azure_devops-0.12.0-py2.py3-none-any.whl', 'azure_devops-0.17.0-py2.py3-none-any.whl', 'azure_devops-0.21.0-py2.py3-none-any.whl', 'azure_devops-0.26.0-py2.py3-none-any.whl', 'azure_devops-1.0.0-py2.py3-none-any.whl', 'azure_devops-1.0.1-py2.py3-none-any.whl']
cli.azure.cli.core.extension._resolve: Candidates ['azure_devops-0.12.0-py2.py3-none-any.whl', 'azure_devops-0.17.0-py2.py3-none-any.whl', 'azure_devops-0.21.0-py2.py3-none-any.whl', 'azure_devops-0.26.0-py2.py3-none-any.whl', 'azure_devops-1.0.0-py2.py3-none-any.whl', 'azure_devops-1.0.1-py2.py3-none-any.whl']
cli.azure.cli.core.extension.resolve: Candidates ['azure_devops-0.12.0-py2.py3-none-any.whl', 'azure_devops-0.17.0-py2.py3-none-any.whl', 'azure_devops-0.21.0-py2.py3-none-any.whl', 'azure_devops-0.26.0-py2.py3-none-any.whl', 'azure_devops-1.0.0-py2.py3-none-any.whl', 'azure_devops-1.0.1-py2.py3-none-any.whl']
cli.azure.cli.core.extension.resolve: Chosen {'downloadUrl': 'https://github.com/Azure/azure-devops-cli-extension/releases/download/20240514.1/azure_devops-1.0.1-py2.py3-none-any.whl', 'filename': 'azure_devops-1.0.1-py2.py3-none-any.whl', 'metadata': {'azext.minCliCoreVersion': '2.30.0', 'classifiers': ['Development Status :: 4 - Beta', 'Intended Audience :: Developers', 'Intended Audience :: System Administrators', 'Programming Language :: Python', 'Programming Language :: Python :: 3', 'Programming Language :: Python :: 3.4', 'Programming Language :: Python :: 3.5', 'Programming Language :: Python :: 3.6', 'License :: OSI Approved :: MIT License'], 'extensions': {'python.details': {'contacts': [{'email': 'VSTS_Social@microsoft.com', 'name': 'Microsoft', 'role': 'author'}], 'document_names': {'description': 'DESCRIPTION.rst'}, 'project_urls': {'Home': 'https://github.com/Microsoft/azure-devops-cli-extension'}}}, 'extras': [], 'generator': 'bdist_wheel (0.30.0)', 'license': 'MIT', 'metadata_version': '2.0', 'name': 'azure-devops', 'run_requires': [{'requires': ['distro (==1.3.0)', 'distro==1.3.0']}], 'summary': 'Tools for managing Azure DevOps.', 'version': '1.0.1'}, 'sha256Digest': 'f300d0288f017148514ebe6f5912aef10c7a6f29bdc0c916b922edf1d75bc7db'}
cli.azure.cli.core.extension.operations: Extension source is url? True
cli.azure.cli.core.extension.operations: Downloading https://github.com/Azure/azure-devops-cli-extension/releases/download/20240514.1/azure_devops-1.0.1-py2.py3-none-any.whl to /tmp/tmp0y0sbwmb/azure_devops-1.0.1-py2.py3-none-any.whl
urllib3.connectionpool: Starting new HTTPS connection (1): github.com:443
urllib3.connectionpool: https://github.com:443 "GET /Azure/azure-devops-cli-extension/releases/download/20240514.1/azure_devops-1.0.1-py2.py3-none-any.whl HTTP/1.1" 302 0
urllib3.connectionpool: Starting new HTTPS connection (1): objects.githubusercontent.com:443
urllib3.connectionpool: https://objects.githubusercontent.com:443 "GET /github-production-release-asset-2e65be/107708057/77ec1ba4-6f10-4d2b-bb30-9c1d420c4fbc?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20240605%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240605T221442Z&X-Amz-Expires=300&X-Amz-Signature=61bc135cef10bed4ce7fe1e02db2ce85027decb9161dfdff54da09594b3bdad0&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=107708057&response-content-disposition=attachment%3B%20filename%3Dazure_devops-1.0.1-py2.py3-none-any.whl&response-content-type=application%2Foctet-stream HTTP/1.1" 200 1195727
cli.azure.cli.core.extension.operations: Downloaded to /tmp/tmp0y0sbwmb/azure_devops-1.0.1-py2.py3-none-any.whl
cli.azure.cli.core.extension.operations: Validating the extension /tmp/tmp0y0sbwmb/azure_devops-1.0.1-py2.py3-none-any.whl
cli.azure.cli.core.extension.operations: Checksum of /tmp/tmp0y0sbwmb/azure_devops-1.0.1-py2.py3-none-any.whl is OK
cli.azure.cli.core.extension.operations: Validation successful on /tmp/tmp0y0sbwmb/azure_devops-1.0.1-py2.py3-none-any.whl
cli.azure.cli.core.extension.operations: Linux distro check: Reading from: /etc/apt/sources.list.d/azure-cli.list
cli.azure.cli.core.extension.operations: Linux distro check: An error occurred while checking linux distribution version source list consistency.
cli.azure.cli.core.extension.operations: [Errno 2] No such file or directory: '/etc/apt/sources.list.d/azure-cli.list'
cli.azure.cli.core.extension.operations: Executing pip with args: ['install', '--target', '/root/.azure/cliextensions/azure-devops', '/tmp/tmp0y0sbwmb/azure_devops-1.0.1-py2.py3-none-any.whl']
cli.azure.cli.core.extension.operations: Running: ['/opt/az/bin/python3', '-m', 'pip', 'install', '--target', '/root/.azure/cliextensions/azure-devops', '/tmp/tmp0y0sbwmb/azure_devops-1.0.1-py2.py3-none-any.whl', '-vv', '--disable-pip-version-check', '--no-cache-dir']
cli.azure.cli.core.extension.operations: Using pip 24.0 from /opt/az/lib/python3.11/site-packages/pip (python 3.11)
Non-user install due to --prefix or --target option
Created temporary directory: /tmp/pip-target-veukpuvo
Created temporary directory: /tmp/pip-build-tracker-60z43nj2
Initialized build tracking at /tmp/pip-build-tracker-60z43nj2
Created build tracker: /tmp/pip-build-tracker-60z43nj2
Entered build tracker: /tmp/pip-build-tracker-60z43nj2
Created temporary directory: /tmp/pip-install-xdcqadi6
Created temporary directory: /tmp/pip-ephem-wheel-cache-t6ychn8h
Processing /tmp/tmp0y0sbwmb/azure_devops-1.0.1-py2.py3-none-any.whl
1 location(s) to search for versions of distro:
https://pypi.org/simple/distro/
Fetching project page and analyzing links: https://pypi.org/simple/distro/
Getting page https://pypi.org/simple/distro/
Found index url https://pypi.org/simple/

Starting new HTTPS connection (1): pypi.org:443
https://pypi.org:443 "GET /simple/distro/ HTTP/1.1" 200 4086
Fetched page https://pypi.org/simple/distro/ as application/vnd.pypi.simple.v1+json
Found link https://files.pythonhosted.org/packages/b7/ff/876ab097c769295f880c9056d09c934f5c7c4c6054df1a83953b73f85f73/distro-0.5.0.tar.gz (from https://pypi.org/simple/distro/), version: 0.5.0
Found link https://files.pythonhosted.org/packages/01/9c/ea1b152ac247a5747598168b88ae82eb742461e2e556262ae741e69bd30f/distro-0.6.0.tar.gz (from https://pypi.org/simple/distro/), version: 0.6.0
Found link https://files.pythonhosted.org/packages/15/97/e2d5863d03cd01b250e51117be031dab2ec0916efc5915f5094f5fd7602c/distro-1.0.0.tar.gz (from https://pypi.org/simple/distro/), version: 1.0.0
Found link https://files.pythonhosted.org/packages/80/4e/22225a92917ebf7780ddb972e14add3d710c718ac00f152c7811ca07b4f3/distro-1.0.1-py2.py3-none-any.whl (from https://pypi.org/simple/distro/), version: 1.0.1
Found link https://files.pythonhosted.org/packages/0d/c5/de784640f0b434799d6eecb63baea9c099e7fe6c0908b4036c3a5200c281/distro-1.0.1.tar.gz (from https://pypi.org/simple/distro/), version: 1.0.1
Found link https://files.pythonhosted.org/packages/64/4d/2d450e0a6dcf394bc0c581e499a66e00d255938f74db6b15748373d36cc4/distro-1.0.2-py2.py3-none-any.whl (from https://pypi.org/simple/distro/), version: 1.0.2
Found link https://files.pythonhosted.org/packages/42/ac/89b295d2784d450ca71ac6f3665cb90f07afe0928e4436af627983faf2b1/distro-1.0.2.tar.gz (from https://pypi.org/simple/distro/), version: 1.0.2
Found link https://files.pythonhosted.org/packages/4a/c2/50dbfaac03c14b8e155e4329553ef1f73a51a5c3cece8fb62e849c0785ed/distro-1.0.3-py2.py3-none-any.whl (from https://pypi.org/simple/distro/), version: 1.0.3
Found link https://files.pythonhosted.org/packages/df/65/a8a94c1e069cf5f5bffab2a6b88d5a78089c60f2916e96d18b625731f191/distro-1.0.3.tar.gz (from https://pypi.org/simple/distro/), version: 1.0.3
Found link https://files.pythonhosted.org/packages/b5/82/363544dcfa3e7f1478e6839aa929a95ac6e1b0c3b56a277e6a6ef3ace2c9/distro-1.0.4-py2.py3-none-any.whl (from https://pypi.org/simple/distro/), version: 1.0.4
Found link https://files.pythonhosted.org/packages/aa/4e/2cf3e7f67abe101c053af838f8d9b3b5911fd9360b498a1ba66a23d1ed46/distro-1.0.4.tar.gz (from https://pypi.org/simple/distro/), version: 1.0.4
Found link https://files.pythonhosted.org/packages/b0/55/29bfd4d4d4149e860ed01aa446108eb17b240997b746c06a2d0c8ce04f69/distro-1.1.0-py2.py3-none-any.whl (from https://pypi.org/simple/distro/), version: 1.1.0
Found link https://files.pythonhosted.org/packages/21/7b/14198029b49abdf80c6b8aadd9862f863b683dc4d3c2418f01bc6fad9fa3/distro-1.1.0.tar.gz (from https://pypi.org/simple/distro/), version: 1.1.0
Found link https://files.pythonhosted.org/packages/c1/e4/933159b5f7f9f5b7ae463e76f58da84a30d0943ab3c162c366a9ad95f01e/distro-1.2.0-py2.py3-none-any.whl (from https://pypi.org/simple/distro/), version: 1.2.0
Found link https://files.pythonhosted.org/packages/b2/2e/e4b8b7f947465474e58bc9dbaa6ea8c4b4cc9e845711c0fc2f66601e464b/distro-1.2.0.tar.gz (from https://pypi.org/simple/distro/), version: 1.2.0
Found link https://files.pythonhosted.org/packages/f6/b1/ba5a96bccd3496241d8908164b9502a129156443cdd5acbdbf04a90b7a09/distro-1.3.0-py2.py3-none-any.whl (from https://pypi.org/simple/distro/), version: 1.3.0
Found link https://files.pythonhosted.org/packages/d2/42/3b059929a920cd9d4e91e7a5e35f0d2ed75211f8f4e877be9d1bde9fdf46/distro-1.3.0.tar.gz (from https://pypi.org/simple/distro/), version: 1.3.0
Found link https://files.pythonhosted.org/packages/ea/35/82f79b92fa4d937146c660a6482cee4f3dfa1f97ff3d2a6f3ecba33e712e/distro-1.4.0-py2.py3-none-any.whl (from https://pypi.org/simple/distro/), version: 1.4.0
Found link https://files.pythonhosted.org/packages/ca/e3/78443d739d7efeea86cbbe0216511d29b2f5ca8dbf51a6f2898432738987/distro-1.4.0.tar.gz (from https://pypi.org/simple/distro/), version: 1.4.0
Found link https://files.pythonhosted.org/packages/25/b7/b3c4270a11414cb22c6352ebc7a83aaa3712043be29daa05018fd5a5c956/distro-1.5.0-py2.py3-none-any.whl (from https://pypi.org/simple/distro/), version: 1.5.0
Found link https://files.pythonhosted.org/packages/a6/a4/75064c334d8ae433445a20816b788700db1651f21bdb0af33db2aab142fe/distro-1.5.0.tar.gz (from https://pypi.org/simple/distro/), version: 1.5.0
Found link https://files.pythonhosted.org/packages/b3/8d/a0a5c389d76f90c766e956515d34c3408a1e18f60fbaa08221d1f6b87490/distro-1.6.0-py2.py3-none-any.whl (from https://pypi.org/simple/distro/), version: 1.6.0
Found link https://files.pythonhosted.org/packages/a5/26/256fa167fe1bf8b97130b4609464be20331af8a3af190fb636a8a7efd7a2/distro-1.6.0.tar.gz (from https://pypi.org/simple/distro/), version: 1.6.0
Found link https://files.pythonhosted.org/packages/e1/54/d08d1ad53788515392bec14d2d6e8c410bffdc127780a9a4aa8e6854d502/distro-1.7.0-py3-none-any.whl (from https://pypi.org/simple/distro/) (requires-python:>=3.6), version: 1.7.0
Found link https://files.pythonhosted.org/packages/b5/7e/ddfbd640ac9a82e60718558a3de7d5988a7d4648385cf00318f60a8b073a/distro-1.7.0.tar.gz (from https://pypi.org/simple/distro/) (requires-python:>=3.6), version: 1.7.0
Found link https://files.pythonhosted.org/packages/f4/2c/c90a3adaf0ddb70afe193f5ebfb539612af57cffe677c3126be533df3098/distro-1.8.0-py3-none-any.whl (from https://pypi.org/simple/distro/) (requires-python:>=3.6), version: 1.8.0
Found link https://files.pythonhosted.org/packages/4b/89/eaa3a3587ebf8bed93e45aa79be8c2af77d50790d15b53f6dfc85b57f398/distro-1.8.0.tar.gz (from https://pypi.org/simple/distro/) (requires-python:>=3.6), version: 1.8.0
Found link https://files.pythonhosted.org/packages/12/b3/231ffd4ab1fc9d679809f356cebee130ac7daa00d6d6f3206dd4fd137e9e/distro-1.9.0-py3-none-any.whl (from https://pypi.org/simple/distro/) (requires-python:>=3.6), version: 1.9.0
Found link https://files.pythonhosted.org/packages/fc/f8/98eea607f65de6527f8a2e8885fc8015d3e6f5775df186e443e0964a11c3/distro-1.9.0.tar.gz (from https://pypi.org/simple/distro/) (requires-python:>=3.6), version: 1.9.0
Skipping link: not a file: https://pypi.org/simple/distro/
Given no hashes to check 2 links for project 'distro': discarding no candidates
Collecting distro==1.3.0 (from azure-devops==1.0.1)
Obtaining dependency information for distro==1.3.0 from https://files.pythonhosted.org/packages/f6/b1/ba5a96bccd3496241d8908164b9502a129156443cdd5acbdbf04a90b7a09/distro-1.3.0-py2.py3-none-any.whl.metadata
Created temporary directory: /tmp/pip-unpack-8al0l3s1
Starting new HTTPS connection (1): files.pythonhosted.org:443
https://files.pythonhosted.org:443 "GET /packages/f6/b1/ba5a96bccd3496241d8908164b9502a129156443cdd5acbdbf04a90b7a09/distro-1.3.0-py2.py3-none-any.whl.metadata HTTP/1.1" 200 1362
Downloading distro-1.3.0-py2.py3-none-any.whl.metadata (1.4 kB)
Created temporary directory: /tmp/pip-metadata-39x6pldp
Created temporary directory: /tmp/pip-unpack-jervwsa

https://files.pythonhosted.org:443 "GET /packages/f6/b1/ba5a96bccd3496241d8908164b9502a129156443cdd5acbdbf04a90b7a09/distro-1.3.0-py2.py3-none-any.whl HTTP/1.1" 200 16807
Downloading distro-1.3.0-py2.py3-none-any.whl (16 kB)
Downloading link https://files.pythonhosted.org/packages/f6/b1/ba5a96bccd3496241d8908164b9502a129156443cdd5acbdbf04a90b7a09/distro-1.3.0-py2.py3-none-any.whl (from https://pypi.org/simple/distro/) to /tmp/pip-unpack-jervwsa
/distro-1.3.0-py2.py3-none-any.whl
Installing collected packages: distro, azure-devops

Creating /tmp/pip-target-veukpuvo/bin
changing mode of /tmp/pip-target-veukpuvo/bin/distro to 755

Successfully installed azure-devops-1.0.1 distro-1.3.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Removed build tracker: '/tmp/pip-build-tracker-60z43nj2'

cli.azure.cli.core.extension.operations: Saved the whl to /root/.azure/cliextensions/azure-devops/azure_devops-1.0.1-py2.py3-none-any.whl
cli.azure.cli.core: Command index has been invalidated.
cli.knack.cli: Event: CommandInvoker.OnTransformResult [<function _resource_group_transform at 0x7f636ce3a340>, <function _x509_from_base64_to_hex_transform at 0x7f636ce3a3e0>]
cli.knack.cli: Event: CommandInvoker.OnFilterResult []
cli.knack.cli: Event: Cli.SuccessfulExecute []
cli.knack.cli: Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x7f636cde5080>]
az_command_data_logger: exit code: 0
cli.main: Command ran in 3.023 seconds (init: 0.099, invoke: 2.924)
telemetry.main: Begin splitting cli events and extra events, total events: 1
telemetry.client: Accumulated 0 events. Flush the clients.
telemetry.main: Finish splitting cli events and extra events, cli events: 1
telemetry.save: Save telemetry record of length 3548 in cache
telemetry.main: Begin creating telemetry upload process.
telemetry.process: Creating upload process: "/opt/az/bin/python3 /opt/az/lib/python3.11/site-packages/azure/cli/telemetry/init.py /root/.azure"
telemetry.process: Return from creating process
telemetry.main: Finish creating telemetry upload process.

Expected behavior

no internet connection to external endpoints.
OSS netiso doc: https://eng.ms/docs/cloud-ai-platform/azure-edge-platform-aep/aep-engineering-systems/productivity-and-experiences/network-isolation/oss/wave1_oss
looks like not all extension install reaches out to pypi.org. for example aks-preview extension add doesn't connect to pypi.org.

Environment Summary

azure-cli 2.61.0

core 2.61.0
telemetry 1.1.0

Extensions:
aks-preview 4.0.0b5
azure-devops 1.0.1

Dependencies:
msal 1.28.0
azure-mgmt-resource 23.1.1

Python location '/opt/az/bin/python3'
Extensions directory '/root/.azure/cliextensions'

Python (Linux) 3.11.8 (main, May 16 2024, 03:50:11) [GCC 10.2.1 20210110]

Legal docs and information: aka.ms/AzureCliLegal

Your CLI is up-to-date.

Additional context

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    DevOpsService AttentionThis issue is responsible by Azure service team.act-platform-engineering-squadbugThis issue requires a change to an existing behavior in the product in order to be resolved.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions