VMSS NotFound when retrieving nic list

[LiquidPT]

Describe the bug

We have an existing ADO pipeline that creates a VMSS (based on Linux images) and then later tries to retreive the list of NICs for the VMSS

This used to work just fine, but lately (since 01/18/2024) it receives a NotFound error. I can see the VMSS resource in Azure Portal though. This also repros from my local machine.

Related command

az vmss nic list

Errors

(NotFound) Resource /subscriptions/d71c7216-6409-45f8-be15-35cf57b8527c/resourceGroups/hpc-image-validation-5680c3-rg/providers/Microsoft.Compute/virtualMachineScaleSets/compute-1705972559 not found.
Code: NotFound
Message: Resource /subscriptions/d71c7216-6409-45f8-be15-35cf57b8527c/resourceGroups/hpc-image-validation-5680c3-rg/providers/Microsoft.Compute/virtualMachineScaleSets/compute-1705972559 not found.

Issue script & Debug output

az vmss nic list --resource-group hpc-image-validation-5680c3-rg --vmss-name compute-1705972559 --debug
cli.knack.cli: Command arguments: ['vmss', 'nic', 'list', '--resource-group', 'hpc-image-validation-5680c3-rg', '--vmss-name', 'compute-1705972559', '--debug']
cli.knack.cli: init debug log:
Enable color in terminal.
cli.knack.cli: Event: Cli.PreExecute []
cli.knack.cli: Event: CommandParser.OnGlobalArgumentsCreate [<function CLILogging.on_global_arguments at 0x0000027DC773B560>, <function OutputProducer.on_global_arguments at 0x0000027DC78C5C60>, <function CLIQuery.on_global_arguments at 0x0000027DC78F37E0>]
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableCreate []
cli.azure.cli.core: Modules found from index for 'vmss': ['azure.cli.command_modules.vm']
cli.azure.cli.core: Loading command modules:
cli.azure.cli.core: Name Load Time Groups Commands
cli.azure.cli.core: vm 0.708 59 274
cli.azure.cli.core: Total (1) 0.708 59 274
cli.azure.cli.core: Loaded 59 groups, 274 commands.
cli.azure.cli.core: Found a match in the command table.
cli.azure.cli.core: Raw command : vmss nic list
cli.azure.cli.core: Command table: vmss nic list
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableTruncate [<function AzCliLogging.init_command_file_logging at 0x0000027DCA81A160>]
cli.azure.cli.core.azlogging: metadata file logging enabled - writing logs to 'C:\Users<username>.azure\commands\2024-01-26.13-47-15.vmss_nic_list.1352.log'.
az_command_data_logger: command args: vmss nic list --resource-group {} --vmss-name {} --debug
cli.knack.cli: Event: CommandInvoker.OnPreArgumentLoad [<function register_global_subscription_argument..add_subscription_parameter at 0x0000027DCA852520>]
cli.knack.cli: Event: CommandInvoker.OnPostArgumentLoad []
cli.knack.cli: Event: CommandInvoker.OnPostCommandTableCreate [<function register_ids_argument..add_ids_arguments at 0x0000027DCA8BC5E0>, <function register_cache_arguments..add_cache_arguments at 0x0000027DCA8BC720>]
cli.knack.cli: Event: CommandInvoker.OnCommandTableLoaded []
cli.knack.cli: Event: CommandInvoker.OnPreParseArgs []
cli.knack.cli: Event: CommandInvoker.OnPostParseArgs [<function OutputProducer.handle_output_argument at 0x0000027DC78C5D00>, <function CLIQuery.handle_query_parameter at 0x0000027DC78F3880>, <function register_ids_argument..parse_ids_arguments at 0x0000027DCA8BC680>]
cli.azure.cli.core.auth.persistence: build_persistence: location='C:\Users\\.azure\msal_token_cache.bin', encrypt=True
cli.azure.cli.core.auth.binary_cache: load: C:\Users<username>.azure\msal_http_cache.bin
urllib3.util.retry: Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=None, status=None)
msal.authority: openid_config = {'token_endpoint': 'https://login.microsoftonline.com/72f988bf-86f1-41af-91ab-2d7cd011db47/oauth2/v2.0/token', 'token_endpoint_auth_methods_supported': ['client_secret_post', 'private_key_jwt', 'client_secret_basic'], 'jwks_uri': 'https://login.microsoftonline.com/72f988bf-86f1-41af-91ab-2d7cd011db47/discovery/v2.0/keys', 'response_modes_supported': ['query', 'fragment', 'form_post'], 'subject_types_supported': ['pairwise'], 'id_token_signing_alg_values_supported': ['RS256'], 'response_types_supported': ['code', 'id_token', 'code id_token', 'id_token token'], 'scopes_supported': ['openid', 'profile', 'email', 'offline_access'], 'issuer': 'https://login.microsoftonline.com/72f988bf-86f1-41af-91ab-2d7cd011db47/v2.0', 'request_uri_parameter_supported': False, 'userinfo_endpoint': 'https://graph.microsoft.com/oidc/userinfo', 'authorization_endpoint': 'https://login.microsoftonline.com/72f988bf-86f1-41af-91ab-2d7cd011db47/oauth2/v2.0/authorize', 'device_authorization_endpoint': 'https://login.microsoftonline.com/72f988bf-86f1-41af-91ab-2d7cd011db47/oauth2/v2.0/devicecode', 'http_logout_supported': True, 'frontchannel_logout_supported': True, 'end_session_endpoint': 'https://login.microsoftonline.com/72f988bf-86f1-41af-91ab-2d7cd011db47/oauth2/v2.0/logout', 'claims_supported': ['sub', 'iss', 'cloud_instance_name', 'cloud_instance_host_name', 'cloud_graph_host_name', 'msgraph_host', 'aud', 'exp', 'iat', 'auth_time', 'acr', 'nonce', 'preferred_username', 'name', 'tid', 'ver', 'at_hash', 'c_hash', 'email'], 'kerberos_endpoint': 'https://login.microsoftonline.com/72f988bf-86f1-41af-91ab-2d7cd011db47/kerberos', 'tenant_region_scope': 'WW', 'cloud_instance_name': 'microsoftonline.com', 'cloud_graph_host_name': 'graph.windows.net', 'msgraph_host': 'graph.microsoft.com', 'rbac_url': 'https://pas.windows.net'}
msal.application: Broker enabled? False
cli.azure.cli.core.auth.credential_adaptor: CredentialAdaptor.get_token: scopes=('https://management.core.windows.net//.default',), kwargs={}
cli.azure.cli.core.auth.msal_authentication: UserCredential.get_token: scopes=('https://management.core.windows.net//.default',), claims=None, kwargs={}
msal.application: Cache hit an AT
msal.telemetry: Generate or reuse correlation_id: eb33ef0b-cd9c-4281-b609-fb4eeb4f58e0
cli.azure.cli.core.sdk.policies: Request URL: 'https://management.azure.com/subscriptions/d71c7216-6409-45f8-be15-35cf57b8527c/resourceGroups/hpc-image-validation-5680c3-rg/providers/microsoft.Compute/virtualMachineScaleSets/compute-1705972559/networkInterfaces?api-version=2016-03-30'
cli.azure.cli.core.sdk.policies: Request method: 'GET'
cli.azure.cli.core.sdk.policies: Request headers:
cli.azure.cli.core.sdk.policies: 'Accept': 'application/json'
cli.azure.cli.core.sdk.policies: 'x-ms-client-request-id': '77a7ec47-bc94-11ee-a6b9-c8348e3bdd30'
cli.azure.cli.core.sdk.policies: 'CommandName': 'vmss nic list'
cli.azure.cli.core.sdk.policies: 'ParameterSetName': '--resource-group --vmss-name --debug'
cli.azure.cli.core.sdk.policies: 'User-Agent': 'AZURECLI/2.56.0 (MSI) (AAZ) azsdk-python-core/1.28.0 Python/3.11.5 (Windows-10-10.0.22621-SP0)'
cli.azure.cli.core.sdk.policies: 'Authorization': '*****'
cli.azure.cli.core.sdk.policies: Request body:
cli.azure.cli.core.sdk.policies: This request has no body
urllib3.connectionpool: Starting new HTTPS connection (1): management.azure.com:443
urllib3.connectionpool: https://management.azure.com:443 "GET /subscriptions/d71c7216-6409-45f8-be15-35cf57b8527c/resourceGroups/hpc-image-validation-5680c3-rg/providers/microsoft.Compute/virtualMachineScaleSets/compute-1705972559/networkInterfaces?api-version=2016-03-30 HTTP/1.1" 404 275
cli.azure.cli.core.sdk.policies: Response status: 404
cli.azure.cli.core.sdk.policies: Response headers:
cli.azure.cli.core.sdk.policies: 'Cache-Control': 'no-cache'
cli.azure.cli.core.sdk.policies: 'Pragma': 'no-cache'
cli.azure.cli.core.sdk.policies: 'Content-Length': '275'
cli.azure.cli.core.sdk.policies: 'Content-Type': 'application/json; charset=utf-8'
cli.azure.cli.core.sdk.policies: 'Expires': '-1'
cli.azure.cli.core.sdk.policies: 'x-ms-request-id': 'c22ba393-9ced-4585-a2aa-95f2fcf01045'
cli.azure.cli.core.sdk.policies: 'x-ms-correlation-request-id': '4c6543d1-df95-4b21-a281-2e83021437dc'
cli.azure.cli.core.sdk.policies: 'x-ms-arm-service-request-id': 'd6f98a2f-9a65-4517-85be-d5cc1b4a68df'
cli.azure.cli.core.sdk.policies: 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains'
cli.azure.cli.core.sdk.policies: 'x-ms-ratelimit-remaining-subscription-reads': '11999'
cli.azure.cli.core.sdk.policies: 'x-ms-routing-request-id': 'WESTUS2:20240126T214716Z:4c6543d1-df95-4b21-a281-2e83021437dc'
cli.azure.cli.core.sdk.policies: 'X-Content-Type-Options': 'nosniff'
cli.azure.cli.core.sdk.policies: 'X-Cache': 'CONFIG_NOCACHE'
cli.azure.cli.core.sdk.policies: 'X-MSEdge-Ref': 'Ref A: 97725676BE654B2295C44703B309F1D8 Ref B: CO6AA3150217009 Ref C: 2024-01-26T21:47:16Z'
cli.azure.cli.core.sdk.policies: 'Date': 'Fri, 26 Jan 2024 21:47:16 GMT'
cli.azure.cli.core.sdk.policies: Response content:
cli.azure.cli.core.sdk.policies: {
"error": {
"code": "NotFound",
"message": "Resource /subscriptions/d71c7216-6409-45f8-be15-35cf57b8527c/resourceGroups/hpc-image-validation-5680c3-rg/providers/Microsoft.Compute/virtualMachineScaleSets/compute-1705972559 not found.",
"details": []
}
}
cli.azure.cli.core.azclierror: Traceback (most recent call last):
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\knack/cli.py", line 233, in invoke
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 663, in execute
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 726, in _run_jobs_serially
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 710, in _run_job
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/aaz/_paging.py", line 87, in next
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/aaz/_paging.py", line 43, in next
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/aaz/_command.py", line 250, in executor_wrapper
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure\cli\command_modules\vm\aaz_compact\latest\vmss\nic__cmds.py", line 64, in _execute_operations
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure\cli\command_modules\vm\aaz_compact\latest\vmss\nic__cmds.py", line 89, in call
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/aaz/_operation.py", line 329, in on_error
azure.core.exceptions.ResourceNotFoundError: (NotFound) Resource /subscriptions/d71c7216-6409-45f8-be15-35cf57b8527c/resourceGroups/hpc-image-validation-5680c3-rg/providers/Microsoft.Compute/virtualMachineScaleSets/compute-1705972559 not found.
Code: NotFound
Message: Resource /subscriptions/d71c7216-6409-45f8-be15-35cf57b8527c/resourceGroups/hpc-image-validation-5680c3-rg/providers/Microsoft.Compute/virtualMachineScaleSets/compute-1705972559 not found.

cli.azure.cli.core.azclierror: (NotFound) Resource /subscriptions/d71c7216-6409-45f8-be15-35cf57b8527c/resourceGroups/hpc-image-validation-5680c3-rg/providers/Microsoft.Compute/virtualMachineScaleSets/compute-1705972559 not found.
Code: NotFound
Message: Resource /subscriptions/d71c7216-6409-45f8-be15-35cf57b8527c/resourceGroups/hpc-image-validation-5680c3-rg/providers/Microsoft.Compute/virtualMachineScaleSets/compute-1705972559 not found.
az_command_data_logger: (NotFound) Resource /subscriptions/d71c7216-6409-45f8-be15-35cf57b8527c/resourceGroups/hpc-image-validation-5680c3-rg/providers/Microsoft.Compute/virtualMachineScaleSets/compute-1705972559 not found.
Code: NotFound
Message: Resource /subscriptions/d71c7216-6409-45f8-be15-35cf57b8527c/resourceGroups/hpc-image-validation-5680c3-rg/providers/Microsoft.Compute/virtualMachineScaleSets/compute-1705972559 not found.
cli.knack.cli: Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x0000027DCA81A3E0>]
az_command_data_logger: exit code: 3
cli.main: Command ran in 3.113 seconds (init: 0.835, invoke: 2.278)
telemetry.main: Begin splitting cli events and extra events, total events: 1
telemetry.client: Accumulated 0 events. Flush the clients.
telemetry.main: Finish splitting cli events and extra events, cli events: 1
telemetry.save: Save telemetry record of length 3828 in cache
telemetry.main: Begin creating telemetry upload process.
telemetry.process: Creating upload process: "C:\Program Files\Microsoft SDKs\Azure\CLI2\python.exe C:\Program Files\Microsoft SDKs\Azure\CLI2\Lib\site-packages\azure\cli\telemetry_init_.pyc C:\Users<username>.azure"
telemetry.process: Return from creating process
telemetry.main: Finish creating telemetry upload process.

Expected behavior

Expected list of NICs attached to this VMSS

Environment Summary

This is the output from my local machine which also repros the issue. ADO build pool machine config unknown:

azure-cli 2.56.0

core 2.56.0
telemetry 1.1.0

Dependencies:
msal 1.24.0b2
azure-mgmt-resource 23.1.0b2

Python location 'C:\Program Files\Microsoft SDKs\Azure\CLI2\python.exe'
Extensions directory 'C:\Users<username>.azure\cliextensions'

Python (Windows) 3.11.5 (tags/v3.11.5:cce6ba9, Aug 24 2023, 14:38:34) [MSC v.1936 64 bit (AMD64)]

Additional context

No response

[yonzhan]

Thank you for opening this issue, we will look into it.

[LiquidPT]

Upon further investigation, with az cli 2.53.1, creating and listing the nics of a VMSS works. it no longer works in 2.55.0, so the breaking change was somewhere between these versions.

Furthermore, when a vmss is created in 2.53.1, and you try to list the nics in 2.56, the nic list is returned. for a vmss created in 2.55, the nic list returns NotFound in 2.53.1. So it appears the problem is in the creation of the VMSS.

This issue is a blocker for our pipeline and for supporting our customers

[yanzhudd]

Hi @LiquidPT
we are working on it, but still need some time to troubleshoot. Do you mind downgrading to 2.53.1 firstly to unblock your pipeline?

[LiquidPT]

Is there a way to control the AZ cli version used in a 1ES ADO pipeline? Thanks, Matt

…

________________________________ From: Yan Zhu ***@***.***> Sent: Wednesday, January 31, 2024 1:57:13 AM To: Azure/azure-cli ***@***.***> Cc: Mention ***@***.***>; Author ***@***.***>; Comment ***@***.***> Subject: Re: [Azure/azure-cli] VMSS NotFound when retrieving nic list (Issue #28258) Hi @LiquidPT<https://github.com/LiquidPT> we are working on it, but still need some time to troubleshoot. Do you mind downgrading to 2.53.1 firstly to unblock your pipeline? — Reply to this email directly, view it on GitHub<#28258 (comment)> or unsubscribe<https://github.com/notifications/unsubscribe-auth/ACQU4TPYWKBQ4KYCPY4STGLYRIIPVBFKMF2HI4TJMJ2XIZLTS6BKK5TBNR2WLJDUOJ2WLJDOMFWWLO3UNBZGKYLEL5YGC4TUNFRWS4DBNZ2F6YLDORUXM2LUPGBKK5TBNR2WLJDUOJ2WLJDOMFWWLLTXMF2GG2C7MFRXI2LWNF2HTAVFOZQWY5LFUVUXG43VMWSG4YLNMWVXI2DSMVQWIX3UPFYGLAVFOZQWY5LFVE2DCOBZGM3DQNZZURXGC3LFVFUGC427NRQWEZLMQKSXMYLMOVS2SNJZGM2DENBSHEYKI3TBNVS2S2DBONPWYYLCMVWIFJLWMFWHKZNKGM3TMMZUGM2DQNBUURXGC3LFVFUGC427NRQWEZLMQKSXMYLMOVS2UNBTHEZTSNRRGM4TTJDOMFWWLKLIMFZV63DBMJSWZLDTOVRGUZLDORPXI6LQMWWES43TOVSUG33NNVSW45FGORXXA2LDOOLIFJDUPFYGLKTSMVYG643JORXXE6NFOZQWY5LFVA2TCMBUGA4DQNUCUR2HS4DFUVUXG43VMWSXMYLMOVS2UMRRGAZDSNRVGIZDNAVEOR4XAZNFNRQWEZLMUV3GC3DVMWUTIMJYHEZTMOBXHGBKI5DZOBS2K3DBMJSWZJLWMFWHKZNJGU4TGNBSGQZDSMECUR2HS4DFUVWGCYTFNSSXMYLMOVS2UMZXGYZTIMZUHA2DJAVEOR4XAZNFNRQWEZLMUV3GC3DVMWVDIMZZGM4TMMJTHE42O5DSNFTWOZLSUZRXEZLBORSQ>. You are receiving this email because you were mentioned. Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[yanzhudd]

Hi @LiquidPT,
I reproduced this issue and now the az vmss nic list command can return correctly and CLI changed nothing, it seems a backend side issue.
You can have a try. It should be work now.

[LiquidPT]

This still repros for me using az cli 2.56.0. It appears that what cli version was used to create the vmss is what's important. If it's created with a pre-bug version, then newer versions will be able to use "az vmss nic list". But any vmss created with 2.56.0 isn't queryable. From: Yan Zhu ***@***.***> Sent: Thursday, February 1, 2024 2:26 AM To: Azure/azure-cli ***@***.***> Cc: Mention ***@***.***>; Author ***@***.***>; Comment ***@***.***> Subject: Re: [Azure/azure-cli] VMSS NotFound when retrieving nic list (Issue #28258) Hi @LiquidPT<https://github.com/LiquidPT>, I reproduced this issue and now the az vmss nic list command can return correctly and CLI changed nothing, it seems a backend side issue. You can have a try. It should be work now. - Reply to this email directly, view it on GitHub<#28258 (comment)> or unsubscribe<https://github.com/notifications/unsubscribe-auth/ACQU4TNTJNJCHPXO3UNPSXDYRNURXBFKMF2HI4TJMJ2XIZLTS6BKK5TBNR2WLJDUOJ2WLJDOMFWWLO3UNBZGKYLEL5YGC4TUNFRWS4DBNZ2F6YLDORUXM2LUPGBKK5TBNR2WLJDUOJ2WLJDOMFWWLLTXMF2GG2C7MFRXI2LWNF2HTAVFOZQWY5LFUVUXG43VMWSG4YLNMWVXI2DSMVQWIX3UPFYGLAVFOZQWY5LFVE2DCOBZGM3DQNZZURXGC3LFVFUGC427NRQWEZLMQKSXMYLMOVS2SNJZGM2DENBSHEYKI3TBNVS2S2DBONPWYYLCMVWIFJLWMFWHKZNKGM3TMMZUGM2DQNBUURXGC3LFVFUGC427NRQWEZLMQKSXMYLMOVS2UNBTHEZTSNRRGM4TTJDOMFWWLKLIMFZV63DBMJSWZLDTOVRGUZLDORPXI6LQMWWES43TOVSUG33NNVSW45FGORXXA2LDOOLIFJDUPFYGLKTSMVYG643JORXXE6NFOZQWY5LFVA2TCMBUGA4DQNUCUR2HS4DFUVUXG43VMWSXMYLMOVS2UMRRGAZDSNRVGIZDNAVEOR4XAZNFNRQWEZLMUV3GC3DVMWUTIMJYHEZTMOBXHGBKI5DZOBS2K3DBMJSWZJLWMFWHKZNJGU4TGNBSGQZDSMECUR2HS4DFUVWGCYTFNSSXMYLMOVS2UMZXGYZTIMZUHA2DJAVEOR4XAZNFNRQWEZLMUV3GC3DVMWVDIMZZGM4TMMJTHE42O5DSNFTWOZLSUZRXEZLBORSQ>. You are receiving this email because you were mentioned. Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.