CLI should use private FQDN for Keyvaults if available #31406
Description
Describe the bug
When configuring a Keyvault with private endpoint and no public access, I found that it never tries to use the private FQDN, instead always using the public FQDN.
Command: az keyvault secret list --vault-name my-vault -o table
Note, the vault name "my-vault" is not a real one I am using, simply changed the real one to avoid sharing the real hostname publicly.
Result:
(Forbidden) Connection is not an approved private link and caller was ignored because bypass is not set to 'AzureServices' and PublicNetworkAccess is set to 'Disabled'.
Vault: my-vault;location=eastus
Code: Forbidden
Message: Connection is not an approved private link and caller was ignored because bypass is not set to 'AzureServices' and PublicNetworkAccess is set to 'Disabled'.
Vault: my-vault;location=eastus
Inner error: {
"code": "ForbiddenByConnection"
}
In my hosts file, I added the privatelink FQDN my-vault.privatelink.vaultcore.azure.net pointing to the vault's private IP.
I expected for azure cli to try this FQDN, but instead I get the above error, indicating it is only trying the public FQDN my-vault.vault.azure.net
After adding a hosts file entry for my-vault.vault.azure.net pointing to the private IP, it works properly and I can list the secrets without issue.
This seems to be different from how the CLI interacts with our private Kubernetes cluster API server.
When I add either private or public FQDN to my /etc/hosts file specifically pointing at an unused private IP, or add both of them pointing to an unused private IP, then run az aks command invoke -g my-resource-group -n my-cluster -c ls I am able to get successful output from the command invocation which confirms that the cluster was reached despite me using an invalid API server IP address in my hosts file.
Related command
az keyvault secret list
az aks command invoke
Errors
(Forbidden) Connection is not an approved private link and caller was ignored because bypass is not set to 'AzureServices' and PublicNetworkAccess is set to 'Disabled'.
Vault: my-vault;location=eastus
Code: Forbidden
Message: Connection is not an approved private link and caller was ignored because bypass is not set to 'AzureServices' and PublicNetworkAccess is set to 'Disabled'.
Vault: my-vault;location=eastus
Inner error: {
"code": "ForbiddenByConnection"
}
Issue script & Debug output
(NOTE - Hostname has been redacted)
cli.knack.cli: Command arguments: ['keyvault', 'secret', 'list', '--vault-name', 'my-vault', '-o', 'table', '--debug']
cli.knack.cli: init debug log:
Enable color in terminal.
cli.knack.cli: Event: Cli.PreExecute []
cli.knack.cli: Event: CommandParser.OnGlobalArgumentsCreate [<function CLILogging.on_global_arguments at 0x1028c4550>, <function OutputProducer.on_global_arguments at 0x102a581f0>, <function CLIQuery.on_global_arguments at 0x102a753f0>]
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableCreate []
cli.azure.cli.core: Modules found from index for 'keyvault': ['azure.cli.command_modules.keyvault']
cli.azure.cli.core: Loading command modules:
cli.azure.cli.core: Name Load Time Groups Commands
cli.azure.cli.core: keyvault 0.004 20 113
cli.azure.cli.core: Total (1) 0.004 20 113
cli.azure.cli.core: These extensions are not installed and will be skipped: ['azext_ai_examples', 'azext_next']
cli.azure.cli.core: Loading extensions:
cli.azure.cli.core: Name Load Time Groups Commands Directory
cli.azure.cli.core: Total (0) 0.000 0 0
cli.azure.cli.core: Loaded 20 groups, 113 commands.
cli.azure.cli.core: Found a match in the command table.
cli.azure.cli.core: Raw command : keyvault secret list
cli.azure.cli.core: Command table: keyvault secret list
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableTruncate [<function AzCliLogging.init_command_file_logging at 0x103bd8670>]
cli.azure.cli.core.azlogging: metadata file logging enabled - writing logs to '/Users/tom.spear/.azure/commands/2025-05-05.12-24-24.keyvault_secret_list.88926.log'.
az_command_data_logger: command args: keyvault secret list --vault-name {} -o {} --debug
cli.knack.cli: Event: CommandInvoker.OnPreArgumentLoad [<function register_global_subscription_argument..add_subscription_parameter at 0x103c2add0>]
cli.knack.cli: Event: CommandInvoker.OnPostArgumentLoad []
cli.knack.cli: Event: CommandInvoker.OnPostCommandTableCreate [<function register_ids_argument..add_ids_arguments at 0x103c2ae60>, <function register_cache_arguments..add_cache_arguments at 0x103c2af80>, <function register_upcoming_breaking_change_info..update_breaking_change_info at 0x103c2b010>]
cli.knack.cli: Event: CommandInvoker.OnCommandTableLoaded []
cli.knack.cli: Event: CommandInvoker.OnPreParseArgs []
cli.knack.cli: Event: CommandInvoker.OnPostParseArgs [<function OutputProducer.handle_output_argument at 0x102a58280>, <function CLIQuery.handle_query_parameter at 0x102a75480>, <function register_ids_argument..parse_ids_arguments at 0x103c2aef0>]
cli.azure.cli.core.auth.persistence: build_persistence: location='/Users/tom.spear/.azure/service_principal_entries.json', encrypt=False
cli.azure.cli.core.auth.persistence: build_persistence: location='/Users/tom.spear/.azure/msal_token_cache.json', encrypt=False
cli.azure.cli.core.auth.binary_cache: load: /Users/tom.spear/.azure/msal_http_cache.bin
urllib3.util.retry: Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=None, status=None)
msal.authority: Initializing with Entra authority: https://login.microsoftonline.com/92867854-71a8-42f2-898a-6953be9825e8
msal.authority: openid_config("https://login.microsoftonline.com/92867854-71a8-42f2-898a-6953be9825e8/v2.0/.well-known/openid-configuration") = {'token_endpoint': 'https://login.microsoftonline.com/92867854-71a8-42f2-898a-6953be9825e8/oauth2/v2.0/token', 'token_endpoint_auth_methods_supported': ['client_secret_post', 'private_key_jwt', 'client_secret_basic'], 'jwks_uri': 'https://login.microsoftonline.com/92867854-71a8-42f2-898a-6953be9825e8/discovery/v2.0/keys', 'response_modes_supported': ['query', 'fragment', 'form_post'], 'subject_types_supported': ['pairwise'], 'id_token_signing_alg_values_supported': ['RS256'], 'response_types_supported': ['code', 'id_token', 'code id_token', 'id_token token'], 'scopes_supported': ['openid', 'profile', 'email', 'offline_access'], 'issuer': 'https://login.microsoftonline.com/92867854-71a8-42f2-898a-6953be9825e8/v2.0', 'request_uri_parameter_supported': False, 'userinfo_endpoint': 'https://graph.microsoft.com/oidc/userinfo', 'authorization_endpoint': 'https://login.microsoftonline.com/92867854-71a8-42f2-898a-6953be9825e8/oauth2/v2.0/authorize', 'device_authorization_endpoint': 'https://login.microsoftonline.com/92867854-71a8-42f2-898a-6953be9825e8/oauth2/v2.0/devicecode', 'http_logout_supported': True, 'frontchannel_logout_supported': True, 'end_session_endpoint': 'https://login.microsoftonline.com/92867854-71a8-42f2-898a-6953be9825e8/oauth2/v2.0/logout', 'claims_supported': ['sub', 'iss', 'cloud_instance_name', 'cloud_instance_host_name', 'cloud_graph_host_name', 'msgraph_host', 'aud', 'exp', 'iat', 'auth_time', 'acr', 'nonce', 'preferred_username', 'name', 'tid', 'ver', 'at_hash', 'c_hash', 'email'], 'kerberos_endpoint': 'https://login.microsoftonline.com/92867854-71a8-42f2-898a-6953be9825e8/kerberos', 'tenant_region_scope': 'NA', 'cloud_instance_name': 'microsoftonline.com', 'cloud_graph_host_name': 'graph.windows.net', 'msgraph_host': 'graph.microsoft.com', 'rbac_url': 'https://pas.windows.net'}
msal.application: Broker enabled? None
cli.azure.cli.core._debug: Using CA bundle file at '/opt/homebrew/etc/ca-certificates/cert.pem'.
cli.azure.cli.core.sdk.policies: Request URL: 'https://my-vault.vault.azure.net/secrets?api-version=7.4'
cli.azure.cli.core.sdk.policies: Request method: 'GET'
cli.azure.cli.core.sdk.policies: Request headers:
cli.azure.cli.core.sdk.policies: 'Accept': 'application/json'
cli.azure.cli.core.sdk.policies: 'x-ms-client-request-id': 'ca169868-29d5-11f0-b77c-1a6cfde437d3'
cli.azure.cli.core.sdk.policies: 'CommandName': 'keyvault secret list'
cli.azure.cli.core.sdk.policies: 'ParameterSetName': '--vault-name -o --debug'
cli.azure.cli.core.sdk.policies: 'User-Agent': 'AZURECLI/2.70.0 azsdk-python-core/1.32.0 Python/3.10.11 (macOS-15.4.1-arm64-arm-64bit)'
cli.azure.cli.core.sdk.policies: Request body:
cli.azure.cli.core.sdk.policies: This request has no body
urllib3.connectionpool: Starting new HTTPS connection (1): my-vault.vault.azure.net:443
urllib3.connectionpool: https://my-vault.vault.azure.net:443 "GET /secrets?api-version=7.4 HTTP/1.1" 401 97
cli.azure.cli.core.sdk.policies: Response status: 401
cli.azure.cli.core.sdk.policies: Response headers:
cli.azure.cli.core.sdk.policies: 'Cache-Control': 'no-cache'
cli.azure.cli.core.sdk.policies: 'Pragma': 'no-cache'
cli.azure.cli.core.sdk.policies: 'Content-Type': 'application/json; charset=utf-8'
cli.azure.cli.core.sdk.policies: 'Expires': '-1'
cli.azure.cli.core.sdk.policies: 'x-ms-keyvault-region': 'eastus'
cli.azure.cli.core.sdk.policies: 'x-ms-client-request-id': 'ca169868-29d5-11f0-b77c-1a6cfde437d3'
cli.azure.cli.core.sdk.policies: 'x-ms-request-id': '56cc2881-3be4-4e08-8826-9be4d32fc563'
cli.azure.cli.core.sdk.policies: 'x-ms-keyvault-service-version': '1.9.2336.1'
cli.azure.cli.core.sdk.policies: 'x-ms-keyvault-network-info': 'conn_type=Ipv4;addr=170.85.98.186;act_addr_fam=InterNetwork;'
cli.azure.cli.core.sdk.policies: 'X-Content-Type-Options': 'nosniff'
cli.azure.cli.core.sdk.policies: 'Strict-Transport-Security': 'max-age=31536000;includeSubDomains'
cli.azure.cli.core.sdk.policies: 'WWW-Authenticate': 'Bearer authorization="https://login.microsoftonline.com/92867854-71a8-42f2-898a-6953be9825e8", resource="https://vault.azure.net"'
cli.azure.cli.core.sdk.policies: 'Date': 'Mon, 05 May 2025 17:24:24 GMT'
cli.azure.cli.core.sdk.policies: 'Content-Length': '97'
cli.azure.cli.core.sdk.policies: Response content:
cli.azure.cli.core.sdk.policies: {"error":{"code":"Unauthorized","message":"AKV10000: Request is missing a Bearer or PoP token."}}
cli.azure.cli.core.auth.credential_adaptor: CredentialAdaptor.get_token: scopes=('https://vault.azure.net/.default',), kwargs={'tenant_id': '92867854-71a8-42f2-898a-6953be9825e8'}
cli.azure.cli.core.auth.msal_credentials: ServicePrincipalCredential.get_token: scopes=('https://vault.azure.net/.default',), kwargs={}
msal.application: Cache hit an AT
msal.telemetry: Generate or reuse correlation_id: 4483d1c7-7f65-42b8-b80c-b69e0ca6c50b
cli.azure.cli.core.sdk.policies: Request URL: 'https://my-vault.vault.azure.net/secrets?api-version=7.4'
cli.azure.cli.core.sdk.policies: Request method: 'GET'
cli.azure.cli.core.sdk.policies: Request headers:
cli.azure.cli.core.sdk.policies: 'Accept': 'application/json'
cli.azure.cli.core.sdk.policies: 'x-ms-client-request-id': 'ca169868-29d5-11f0-b77c-1a6cfde437d3'
cli.azure.cli.core.sdk.policies: 'CommandName': 'keyvault secret list'
cli.azure.cli.core.sdk.policies: 'ParameterSetName': '--vault-name -o --debug'
cli.azure.cli.core.sdk.policies: 'User-Agent': 'AZURECLI/2.70.0 azsdk-python-core/1.32.0 Python/3.10.11 (macOS-15.4.1-arm64-arm-64bit)'
cli.azure.cli.core.sdk.policies: 'Authorization': '*****'
cli.azure.cli.core.sdk.policies: Request body:
cli.azure.cli.core.sdk.policies: This request has no body
urllib3.connectionpool: https://my-vault.vault.azure.net:443 "GET /secrets?api-version=7.4 HTTP/1.1" 403 297
cli.azure.cli.core.sdk.policies: Response status: 403
cli.azure.cli.core.sdk.policies: Response headers:
cli.azure.cli.core.sdk.policies: 'Cache-Control': 'no-cache'
cli.azure.cli.core.sdk.policies: 'Pragma': 'no-cache'
cli.azure.cli.core.sdk.policies: 'Content-Type': 'application/json; charset=utf-8'
cli.azure.cli.core.sdk.policies: 'Expires': '-1'
cli.azure.cli.core.sdk.policies: 'x-ms-keyvault-region': 'eastus'
cli.azure.cli.core.sdk.policies: 'x-ms-client-request-id': 'ca169868-29d5-11f0-b77c-1a6cfde437d3'
cli.azure.cli.core.sdk.policies: 'x-ms-request-id': '03fe41c2-b56a-4bfa-981b-62d3b71f1b01'
cli.azure.cli.core.sdk.policies: 'x-ms-keyvault-service-version': '1.9.2336.1'
cli.azure.cli.core.sdk.policies: 'x-ms-keyvault-network-info': 'conn_type=Ipv4;addr=170.85.98.186;act_addr_fam=InterNetwork;'
cli.azure.cli.core.sdk.policies: 'X-Content-Type-Options': 'nosniff'
cli.azure.cli.core.sdk.policies: 'Strict-Transport-Security': 'max-age=31536000;includeSubDomains'
cli.azure.cli.core.sdk.policies: 'Date': 'Mon, 05 May 2025 17:24:24 GMT'
cli.azure.cli.core.sdk.policies: 'Content-Length': '297'
cli.azure.cli.core.sdk.policies: Response content:
cli.azure.cli.core.sdk.policies: {"error":{"code":"Forbidden","message":"Connection is not an approved private link and caller was ignored because bypass is not set to 'AzureServices' and PublicNetworkAccess is set to 'Disabled'. \r\nVault: my-vault;location=eastus","innererror":{"code":"ForbiddenByConnection"}}}
cli.azure.cli.core.azclierror: Traceback (most recent call last):
File "/Users/tom.spear/.asdf/installs/azure-cli/2.70.0/bin/venv/lib/python3.10/site-packages/azure/cli/command_modules/keyvault/_command_type.py", line 113, in keyvault_command_handler
return _encode_hex(transform_result(result, **{**command_args, 'kv_transform': True}))
File "/Users/tom.spear/.asdf/installs/azure-cli/2.70.0/bin/venv/lib/python3.10/site-packages/azure/cli/command_modules/keyvault/transformers.py", line 12, in multi_transformers
output = t(output, **command_args)
File "/Users/tom.spear/.asdf/installs/azure-cli/2.70.0/bin/venv/lib/python3.10/site-packages/azure/cli/command_modules/keyvault/transformers.py", line 29, in filter_out_managed_resources
return [ for _ in output if not getattr(, 'managed')] if output else output
File "/Users/tom.spear/.asdf/installs/azure-cli/2.70.0/bin/venv/lib/python3.10/site-packages/azure/cli/command_modules/keyvault/transformers.py", line 29, in
return [ for _ in output if not getattr(, 'managed')] if output else output
File "/Users/tom.spear/.asdf/installs/azure-cli/2.70.0/bin/venv/lib/python3.10/site-packages/azure/core/paging.py", line 123, in next
return next(self._page_iterator)
File "/Users/tom.spear/.asdf/installs/azure-cli/2.70.0/bin/venv/lib/python3.10/site-packages/azure/core/paging.py", line 75, in next
self._response = self._get_next(self.continuation_token)
File "/Users/tom.spear/.asdf/installs/azure-cli/2.70.0/bin/venv/lib/python3.10/site-packages/azure/keyvault/secrets/_generated/v7_4/operations/_key_vault_client_operations.py", line 869, in get_next
raise HttpResponseError(response=response, model=error)
azure.core.exceptions.HttpResponseError: (Forbidden) Connection is not an approved private link and caller was ignored because bypass is not set to 'AzureServices' and PublicNetworkAccess is set to 'Disabled'.
Vault: my-vault;location=eastus
Code: Forbidden
Message: Connection is not an approved private link and caller was ignored because bypass is not set to 'AzureServices' and PublicNetworkAccess is set to 'Disabled'.
Vault: my-vault;location=eastus
Inner error: {
"code": "ForbiddenByConnection"
}
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/tom.spear/.asdf/installs/azure-cli/2.70.0/bin/venv/lib/python3.10/site-packages/knack/cli.py", line 233, in invoke
cmd_result = self.invocation.execute(args)
File "/Users/tom.spear/.asdf/installs/azure-cli/2.70.0/bin/venv/lib/python3.10/site-packages/azure/cli/core/commands/init.py", line 666, in execute
raise ex
File "/Users/tom.spear/.asdf/installs/azure-cli/2.70.0/bin/venv/lib/python3.10/site-packages/azure/cli/core/commands/init.py", line 734, in _run_jobs_serially
results.append(self._run_job(expanded_arg, cmd_copy))
File "/Users/tom.spear/.asdf/installs/azure-cli/2.70.0/bin/venv/lib/python3.10/site-packages/azure/cli/core/commands/init.py", line 703, in _run_job
result = cmd_copy(params)
File "/Users/tom.spear/.asdf/installs/azure-cli/2.70.0/bin/venv/lib/python3.10/site-packages/azure/cli/core/commands/init.py", line 336, in call
return self.handler(*args, **kwargs)
File "/Users/tom.spear/.asdf/installs/azure-cli/2.70.0/bin/venv/lib/python3.10/site-packages/azure/cli/command_modules/keyvault/_command_type.py", line 135, in keyvault_command_handler
return keyvault_exception_handler(ex)
File "/Users/tom.spear/.asdf/installs/azure-cli/2.70.0/bin/venv/lib/python3.10/site-packages/azure/cli/command_modules/keyvault/_command_type.py", line 49, in keyvault_exception_handler
raise CLIError(ex)
knack.util.CLIError: (Forbidden) Connection is not an approved private link and caller was ignored because bypass is not set to 'AzureServices' and PublicNetworkAccess is set to 'Disabled'.
Vault: my-vault;location=eastus
Code: Forbidden
Message: Connection is not an approved private link and caller was ignored because bypass is not set to 'AzureServices' and PublicNetworkAccess is set to 'Disabled'.
Vault: my-vault;location=eastus
Inner error: {
"code": "ForbiddenByConnection"
}
cli.azure.cli.core.azclierror: (Forbidden) Connection is not an approved private link and caller was ignored because bypass is not set to 'AzureServices' and PublicNetworkAccess is set to 'Disabled'.
Vault: my-vault;location=eastus
Code: Forbidden
Message: Connection is not an approved private link and caller was ignored because bypass is not set to 'AzureServices' and PublicNetworkAccess is set to 'Disabled'.
Vault: my-vault;location=eastus
Inner error: {
"code": "ForbiddenByConnection"
}
az_command_data_logger: (Forbidden) Connection is not an approved private link and caller was ignored because bypass is not set to 'AzureServices' and PublicNetworkAccess is set to 'Disabled'.
Vault: my-vault;location=eastus
Code: Forbidden
Message: Connection is not an approved private link and caller was ignored because bypass is not set to 'AzureServices' and PublicNetworkAccess is set to 'Disabled'.
Vault: my-vault;location=eastus
Inner error: {
"code": "ForbiddenByConnection"
}
cli.knack.cli: Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x103bd88b0>]
az_command_data_logger: exit code: 1
cli.main: Command ran in 0.616 seconds (init: 0.080, invoke: 0.535)
telemetry.main: Begin splitting cli events and extra events, total events: 1
telemetry.client: Accumulated 0 events. Flush the clients.
telemetry.main: Finish splitting cli events and extra events, cli events: 1
telemetry.save: Save telemetry record of length 4412 in cache
telemetry.main: Begin creating telemetry upload process.
telemetry.process: Creating upload process: "/Users/tom.spear/.asdf/installs/azure-cli/2.70.0/bin/venv/bin/python3 /Users/tom.spear/.asdf/installs/azure-cli/2.70.0/bin/venv/lib/python3.10/site-packages/azure/cli/telemetry/init.py /Users/tom.spear/.azure"
telemetry.process: Return from creating process
telemetry.main: Finish creating telemetry upload process.
Expected behavior
CLI connects to keyvault using the private FQDN ending with privatelink.vaultcore.azure.net and shows the list of secrets.
Environment Summary
azure-cli 2.70.0 *
core 2.70.0 *
telemetry 1.1.0
Extensions:
aks-preview 13.0.0b2
Dependencies:
msal 1.31.2b1
azure-mgmt-resource 23.1.1
Python location '/Users/tom.spear/.asdf/installs/azure-cli/2.70.0/bin/venv/bin/python3'
Config directory '/Users/tom.spear/.azure'
Extensions directory '/Users/tom.spear/.azure/cliextensions'
Python (Darwin) 3.10.11 (main, Aug 13 2024, 22:14:06) [Clang 15.0.0 (clang-1500.3.9.4)]
Additional context
When running the az aks command invoke command with the --debug flag, I can see that the CLI connects to management.azure.com rather than making a direct request to the kubernetes API server; which leads to this command properly working even if I explicitly try to break the DNS by adding the public and private FQDN to my /etc/hosts file pointing to an invalid IP. This is the behavior I am expecting to also happen for the key vault requests.