Failed to add Load Balancer addresses to backend pools when syncMode of the Backend pool is set to 'Automatic'

[jorgeakanieves]

Describe the bug

When creating the backend pool setting the property 'sync-mode' to 'Automatic' and then adding an address to the pool is failing due of next error:

"Error: updating Backend Address Pool Address: (Address Name "<backend_pool_address_name>" / Backend Address Pool Name "<backend_pool_name>" / Load Balancer Name "<load_balancer_name>" / Resource Group "<resource_group_name>"): performing LoadBalancerBackendAddressPoolsCreateOrUpdate: unexpected status 400 (400 Bad Request) with error: ModificationOfBackendAddressesInPoolIsNotSupportedWhenSyncModeIsAutomatic: Modification of backend addresses in pools is not supported when sync mode is set as Automatic.

Related command

az network lb address-pool create--lb-name lbname -g rgname -n bpname--sync-mode "Automatic" --vnet "/subscriptions/<az_sub_id>/resourceGroups/rgname/providers/Microsoft.Network/virtualNetworks/vnetname"

az network lb address-pool address add --ip-address "10.0.1.10" --lb-name lbname -g rgname --name bpaddress_name --pool-name bpname --virtual-network "/subscriptions/<az_sub_id>/resourceGroups/rgname/providers/Microsoft.Network/virtualNetworks/vnetname"

Errors

Error: updating Backend Address Pool Address: (Address Name "<backend_pool_address_name>" / Backend Address Pool Name "<backend_pool_name>" / Load Balancer Name "<load_balancer_name>" / Resource Group "<resource_group_name>"): performing LoadBalancerBackendAddressPoolsCreateOrUpdate: unexpected status 400 (400 Bad Request) with error: ModificationOfBackendAddressesInPoolIsNotSupportedWhenSyncModeIsAutomatic: Modification of backend addresses in pools is not supported when sync mode is set as Automatic.

Issue script & Debug output

$> az network lb address-pool address add --ip-address "10.0.1.10" --lb-name lbname -g rsgname --name Testing_BackendPool --pool-name Testing_BackendPool --virtual-network "/subscriptions/<az_sub_id>/resourceGroups/rsgname/providers/Microsoft.Network/virtualNetworks/vnet_name" --debug

cli.knack.cli: Command arguments: ['network', 'lb', 'address-pool', 'address', 'add', '--ip-address', '10.0.1.10', '--lb-name', 'lbname', '-g', 'rsgname', '--name', 'Testing_BackendPool', '--pool-name', 'Testing_BackendPool', '--virtual-network', '/subscriptions/<az_sub_id>/resourceGroups/rsgname/providers/Microsoft.Network/virtualNetworks/vnet_name', '--debug']
cli.knack.cli: __init__ debug log:
Enable color in terminal.
cli.knack.cli: Event: Cli.PreExecute []
cli.knack.cli: Event: CommandParser.OnGlobalArgumentsCreate [<function CLILogging.on_global_arguments at 0x014E2A78>, <function OutputProducer.on_global_arguments at 0x0180AB68>, <function CLIQuery.on_global_arguments at 0x0182C938>]
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableCreate []
cli.azure.cli.core: Modules found from index for 'network': ['azure.cli.command_modules.network', 'azure.cli.command_modules.privatedns']
cli.azure.cli.core: Loading command modules:
cli.azure.cli.core: Name                  Load Time    Groups  Commands
cli.azure.cli.core: network                   0.892       124       488
cli.azure.cli.core: privatedns                0.051        14        60
cli.azure.cli.core: Total (2)                 0.943       138       548
cli.azure.cli.core: Loaded 137 groups, 548 commands.
cli.azure.cli.core: Found a match in the command table.
cli.azure.cli.core: Raw command  : network lb address-pool address add
cli.azure.cli.core: Command table: network lb address-pool address add
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableTruncate [<function AzCliLogging.init_command_file_logging at 0x03CEC258>]    
cli.azure.cli.core.azlogging: metadata file logging enabled - writing logs to 'C:\Users\path\.azure\commands\2024-08-14.12-54-21.network_lb_address-pool_address_add.9476.log'.
az_command_data_logger: command args: network lb address-pool address add --ip-address {} --lb-name {} -g {} --name {} --pool-name {} --virtual-network {} --debug
cli.knack.cli: Event: CommandInvoker.OnPreArgumentLoad [<function register_global_subscription_argument.<locals>.add_subscription_parameter at 0x03D1D3E8>]
cli.knack.cli: Event: CommandInvoker.OnPostArgumentLoad []
cli.knack.cli: Event: CommandInvoker.OnPostCommandTableCreate [<function register_ids_argument.<locals>.add_ids_arguments at 0x03D306B8>, <function register_cache_arguments.<locals>.add_cache_arguments at 0x03D30708>]
cli.knack.cli: Event: CommandInvoker.OnCommandTableLoaded []
cli.knack.cli: Event: CommandInvoker.OnPreParseArgs []
cli.knack.cli: Event: CommandInvoker.OnPostParseArgs [<function OutputProducer.handle_output_argument at 0x0180ABB8>, <function CLIQuery.handle_query_parameter at 0x0182C988>, <function register_ids_argument.<locals>.parse_ids_arguments at 0x03D30668>]
This command is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus
cli.azure.cli.core.auth.persistence: build_persistence: location='C:\\Users\\path\\.azure\\service_principal_entries.bin', encrypt=True
cli.azure.cli.core.auth.persistence: build_persistence: location='C:\\Users\\path\\.azure\\msal_token_cache.bin', encrypt=True
cli.azure.cli.core.auth.binary_cache: load: C:\Users\path\.azure\msal_http_cache.bin
urllib3.util.retry: Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=None, status=None)
msal.authority: Initializing with Entra authority: https://login.microsoftonline.com/<tenant_id>
msal.authority: openid_config("https://login.microsoftonline.com/<tenant_id>/v2.0/.well-known/openid-configuration") = {'token_endpoint': 'https://login.microsoftonline.com/<tenant_id>/oauth2/v2.0/token', 'token_endpoint_auth_methods_supported': ['client_secret_post', 'private_key_jwt', 'client_secret_basic'], 'jwks_uri': 'https://login.microsoftonline.com/<tenant_id>/discovery/v2.0/keys', 'response_modes_supported': ['query', 'fragment', 'form_post'], 'subject_types_supported': ['pairwise'], 'id_token_signing_alg_values_supported': ['RS256'], 'response_types_supported': ['code', 'id_token', 'code id_token', 'id_token token'], 'scopes_supported': ['openid', 'profile', 'email', 'offline_access'], 'issuer': 'https://login.microsoftonline.com/<tenant_id>/v2.0', 'request_uri_parameter_supported': False, 'userinfo_endpoint': 'https://graph.microsoft.com/oidc/userinfo', 'authorization_endpoint': 'https://login.microsoftonline.com/<tenant_id>/oauth2/v2.0/authorize', 'device_authorization_endpoint': 'https://login.microsoftonline.com/<tenant_id>/oauth2/v2.0/devicecode', 'http_logout_supported': True, 'frontchannel_logout_supported': True, 'end_session_endpoint': 'https://login.microsoftonline.com/<tenant_id>/oauth2/v2.0/logout', 'claims_supported': ['sub', 'iss', 'cloud_instance_name', 'cloud_instance_host_name', 'cloud_graph_host_name', 'msgraph_host', 'aud', 'exp', 'iat', 'auth_time', 'acr', 
'nonce', 'preferred_username', 'name', 'tid', 'ver', 'at_hash', 'c_hash', 'email'], 'kerberos_endpoint': 'https://login.microsoftonline.com/<tenant_id>/kerberos', 'tenant_region_scope': 'EU', 'cloud_instance_name': 'microsoftonline.com', 'cloud_graph_host_name': 'graph.windows.net', 'msgraph_host': 'graph.microsoft.com', 'rbac_url': 'https://pas.windows.net'}
msal.application: Broker enabled? None
cli.azure.cli.core.auth.credential_adaptor: CredentialAdaptor.get_token: scopes=('https://management.core.windows.net//.default',), 
kwargs={}
cli.azure.cli.core.auth.msal_authentication: ServicePrincipalCredential.get_token: scopes=('https://management.core.windows.net//.default',), kwargs={}
msal.telemetry: Generate or reuse correlation_id: cc265b56-7579-48fc-86b3-1db241262a37
urllib3.connectionpool: Starting new HTTPS connection (1): login.microsoftonline.com:443
urllib3.connectionpool: https://login.microsoftonline.com:443 "POST /<tenant_id>/oauth2/v2.0/token HTTP/1.1" 200 1499
msal.token_cache: event={
    "client_id": "<client_id>",
    "data": {
        "claims": "{\"access_token\": {\"xms_cc\": {\"values\": [\"CP1\"]}}}",
        "scope": [
            "https://management.core.windows.net//.default"
        ]
    },
    "environment": "login.microsoftonline.com",
    "grant_type": "client_credentials",
    "params": null,
    "response": {
        "access_token": "********",
        "expires_in": 3599,
        "ext_expires_in": 3599,
        "token_type": "Bearer"
    },
    "scope": [
        "https://management.core.windows.net//.default"
    ],
    "token_endpoint": "https://login.microsoftonline.com/<tenant_id>/oauth2/v2.0/token"
}
cli.azure.cli.core.sdk.policies: Request URL: 'https://management.azure.com/subscriptions/<az_sub_id>/resourceGroups/rsgname/providers/Microsoft.Network/loadBalancers/lbname/backendAddressPools/Testing_BackendPool?api-version=2023-04-01'
cli.azure.cli.core.sdk.policies: Request method: 'GET'
cli.azure.cli.core.sdk.policies: Request headers:
cli.azure.cli.core.sdk.policies:     'Accept': 'application/json'
cli.azure.cli.core.sdk.policies:     'x-ms-client-request-id': ''
cli.azure.cli.core.sdk.policies:     'CommandName': 'network lb address-pool address add'
cli.azure.cli.core.sdk.policies:     'ParameterSetName': '--ip-address --lb-name -g --name --pool-name --virtual-network --debug'   
cli.azure.cli.core.sdk.policies:     'User-Agent': 'AZURECLI/2.61.0 (MSI) azsdk-python-core/1.28.0 Python/3.11.8 (Windows-10-10.0.19045-SP0)'
cli.azure.cli.core.sdk.policies:     'Authorization': '*****'
cli.azure.cli.core.sdk.policies: Request body:
cli.azure.cli.core.sdk.policies: This request has no body
urllib3.connectionpool: Starting new HTTPS connection (1): management.azure.com:443
urllib3.connectionpool: https://management.azure.com:443 "GET /subscriptions/<az_sub_id>/resourceGroups/rsgname/providers/Microsoft.Network/loadBalancers/lbname/backendAddressPools/Testing_BackendPool?api-version=2023-04-01 HTTP/1.1" 200 935
cli.azure.cli.core.sdk.policies: Response status: 200
cli.azure.cli.core.sdk.policies: Response headers:
cli.azure.cli.core.sdk.policies:     'Cache-Control': 'no-cache'
cli.azure.cli.core.sdk.policies:     'Pragma': 'no-cache'
cli.azure.cli.core.sdk.policies:     'Content-Length': '935'
cli.azure.cli.core.sdk.policies:     'Content-Type': 'application/json; charset=utf-8'
cli.azure.cli.core.sdk.policies:     'Expires': '-1'
cli.azure.cli.core.sdk.policies:     'ETag': 'W/"2e7b4"'
cli.azure.cli.core.sdk.policies:     'x-ms-request-id': 'f0fe788f-b59e8d02'
cli.azure.cli.core.sdk.policies:     'x-ms-correlation-request-id': '69301b4ba2b6e'
cli.azure.cli.core.sdk.policies:     'x-ms-arm-service-request-id': '16d6194c3b7926'
cli.azure.cli.core.sdk.policies:     'Strict-Transport-Security': 'max-age=31536000; includeSubDomains'
cli.azure.cli.core.sdk.policies:     'x-ms-ratelimit-remaining-subscription-reads': '249'
cli.azure.cli.core.sdk.policies:     'x-ms-ratelimit-remaining-subscription-global-reads': '3749'
cli.azure.cli.core.sdk.policies:     'x-ms-routing-request-id': 'FRANCESOUTH:202a2b6e' 
cli.azure.cli.core.sdk.policies:     'X-Content-Type-Options': 'nosniff'
cli.azure.cli.core.sdk.policies:     'X-Cache': 'CONFIG_NOCACHE'
cli.azure.cli.core.sdk.policies:     'X-MSEdge-Ref': 'Ref A: E25E733 Ref C: 2024-08-14T10:54:23Z'
cli.azure.cli.core.sdk.policies:     'Date': 'Wed, 14 Aug 2024 10:54:23 GMT'
cli.azure.cli.core.sdk.policies: Response content:
cli.azure.cli.core.sdk.policies: {
  "name": "Testing_BackendPool",
  "id": "/subscriptions/<az_sub_id>/resourceGroups/rsgname/providers/Microsoft.Network/loadBalancers/lbname/backendAddressPools/Testing_BackendPool",
  "etag": "W/\"2b3dcb17b4\"",
  "properties": {
    "provisioningState": "Succeeded",
    "virtualNetwork": {
      "id": "/subscriptions/<az_sub_id>/resourceGroups/rsgname/providers/Microsoft.Network/virtualNetworks/vnet_name"
    },
    "loadBalancerBackendAddresses": [],
    "loadBalancingRules": [
      {
        "id": "/subscriptions/<az_sub_id>/resourceGroups/rsgname/providers/Microsoft.Network/loadBalancers/lbname/loadBalancingRules/Rule_3389"
      }
    ],
    "syncMode": "Automatic"
  },
  "type": "Microsoft.Network/loadBalancers/backendAddressPools"
}
cli.azure.cli.core.sdk.policies: Request URL: 'https://management.azure.com/subscriptions/<az_sub_id>/resourceGroups/rsgname/providers/Microsoft.Network/loadBalancers/lbname/backendAddressPools/Testing_BackendPool?api-version=2023-04-01'
cli.azure.cli.core.sdk.policies: Request method: 'PUT'
cli.azure.cli.core.sdk.policies: Request headers:
cli.azure.cli.core.sdk.policies:     'Content-Type': 'application/json'
cli.azure.cli.core.sdk.policies:     'Accept': 'application/json'
cli.azure.cli.core.sdk.policies:     'Content-Length': '730'
cli.azure.cli.core.sdk.policies:     'x-ms-client-request-id': '8e9'
cli.azure.cli.core.sdk.policies:     'CommandName': 'network lb address-pool address add'
cli.azure.cli.core.sdk.policies:     'ParameterSetName': '--ip-address --lb-name -g --name --pool-name --virtual-network --debug'   
cli.azure.cli.core.sdk.policies:     'User-Agent': 'AZURECLI/2.61.0 (MSI) azsdk-python-core/1.28.0 Python/3.11.8 (Windows-10-10.0.19045-SP0)'
cli.azure.cli.core.sdk.policies:     'Authorization': '*****'
cli.azure.cli.core.sdk.policies: Request body:
cli.azure.cli.core.sdk.policies: {"id": "/subscriptions/<az_sub_id>/resourceGroups/rsgname/providers/Microsoft.Network/loadBalancers/lbname/backendAddressPools/Testing_BackendPool", "name": "Testing_BackendPool", "properties": {"loadBalancerBackendAddresses": [{"name": "Testing_BackendPool", "properties": {"ipAddress": "10.0.1.10", "virtualNetwork": {"id": "/subscriptions/<az_sub_id>/resourceGroups/rsgname/providers/Microsoft.Network/virtualNetworks/vnet_name"}}}], "syncMode": "Automatic", "virtualNetwork": {"id": "/subscriptions/<az_sub_id>/resourceGroups/rsgname/providers/Microsoft.Network/virtualNetworks/vnet_name"}}}
urllib3.connectionpool: https://management.azure.com:443 "PUT /subscriptions/<az_sub_id>/resourceGroups/rsgname/providers/Microsoft.Network/loadBalancers/lbname/backendAddressPools/Testing_BackendPool?api-version=2023-04-01 HTTP/1.1" 400 247
cli.azure.cli.core.sdk.policies: Response status: 400
cli.azure.cli.core.sdk.policies: Response headers:
cli.azure.cli.core.sdk.policies:     'Cache-Control': 'no-cache'
cli.azure.cli.core.sdk.policies:     'Pragma': 'no-cache'
cli.azure.cli.core.sdk.policies:     'Content-Length': '247'
cli.azure.cli.core.sdk.policies:     'Content-Type': 'application/json; charset=utf-8'
cli.azure.cli.core.sdk.policies:     'Expires': '-1'
cli.azure.cli.core.sdk.policies:     'x-ms-request-id': '60c870c9514'
cli.azure.cli.core.sdk.policies:     'x-ms-correlation-request-id': '78ef36a'
cli.azure.cli.core.sdk.policies:     'x-ms-arm-service-request-id': '7fad'
cli.azure.cli.core.sdk.policies:     'Strict-Transport-Security': 'max-age=31536000; includeSubDomains'
cli.azure.cli.core.sdk.policies:     'x-ms-ratelimit-remaining-subscription-writes': '199'
cli.azure.cli.core.sdk.policies:     'x-ms-ratelimit-remaining-subscription-global-writes': '2999'
cli.azure.cli.core.sdk.policies:     'x-ms-routing-request-id': 'FRANCESOUTH:2021e9c6a' 
cli.azure.cli.core.sdk.policies:     'X-Content-Type-Options': 'nosniff'
cli.azure.cli.core.sdk.policies:     'X-Cache': 'CONFIG_NOCACHE'
cli.azure.cli.core.sdk.policies:     'X-MSEdge-Ref': 'Ref A: B03B6B8023 Ref C: 2024-08-14T10:54:24Z'
cli.azure.cli.core.sdk.policies:     'Date': 'Wed, 14 Aug 2024 10:54:23 GMT'
cli.azure.cli.core.sdk.policies: Response content:
cli.azure.cli.core.sdk.policies: {
  "error": {
    "code": "ModificationOfBackendAddressesInPoolIsNotSupportedWhenSyncModeIsAutomatic",
    "message": "Modification of backend addresses in pools is not supported when sync mode is set as Automatic.",
    "details": []
  }
}
cli.azure.cli.core.azclierror: Traceback (most recent call last):
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\knack/cli.py", line 233, in invoke
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/__init__.py", line 664, in execute
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/__init__.py", line 731, in _run_jobs_serially
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/__init__.py", line 712, in _run_job
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/__init__.py", line 1052, in __call__
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/__init__.py", line 1039, in __call__
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/aaz/_poller.py", line 108, in result    
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/core/tracing/decorator.py", line 76, in wrapper_use_tracer
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/aaz/_poller.py", line 130, in wait      
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/aaz/_poller.py", line 83, in _start     
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure\cli\command_modules\network\aaz_compact\latest\network\lb\address_pool\address\__cmds.py", line 118, in _execute_operations
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure\cli\command_modules\network\aaz_compact\latest\network\lb\address_pool\address\__cmds.py", line 278, in __call__
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/aaz/_operation.py", line 332, in on_error
azure.core.exceptions.HttpResponseError: (ModificationOfBackendAddressesInPoolIsNotSupportedWhenSyncModeIsAutomatic) Modification of backend addresses in pools is not supported when sync mode is set as Automatic.
Code: ModificationOfBackendAddressesInPoolIsNotSupportedWhenSyncModeIsAutomatic
Message: Modification of backend addresses in pools is not supported when sync mode is set as Automatic.

cli.azure.cli.core.azclierror: (ModificationOfBackendAddressesInPoolIsNotSupportedWhenSyncModeIsAutomatic) Modification of backend addresses in pools is not supported when sync mode is set as Automatic.
Code: ModificationOfBackendAddressesInPoolIsNotSupportedWhenSyncModeIsAutomatic
Message: Modification of backend addresses in pools is not supported when sync mode is set as Automatic.
az_command_data_logger: (ModificationOfBackendAddressesInPoolIsNotSupportedWhenSyncModeIsAutomatic) Modification of backend addresses in pools is not supported when sync mode is set as Automatic.
Code: ModificationOfBackendAddressesInPoolIsNotSupportedWhenSyncModeIsAutomatic
Message: Modification of backend addresses in pools is not supported when sync mode is set as Automatic.
cli.knack.cli: Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x03CEC398>]
az_command_data_logger: exit code: 1
cli.__main__: Command ran in 4.774 seconds (init: 0.712, invoke: 4.062)
telemetry.main: Begin splitting cli events and extra events, total events: 1
telemetry.client: Accumulated 0 events. Flush the clients.
telemetry.main: Finish splitting cli events and extra events, cli events: 1
telemetry.save: Save telemetry record of length 4242 in cache
telemetry.main: Begin creating telemetry upload process.
telemetry.process: Creating upload process: "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\python.exe C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\Lib\site-packages\azure\cli\telemetry\__init__.pyc C:\Users\path\.azure"
telemetry.process: Return from creating process
telemetry.main: Finish creating telemetry upload process.

Expected behavior

Adding addresses to the backend pool that it´s previously created should work as it works when the backend pool sync_mode property is set to 'Manual'. It appears that addresses can´t be added when sync mode of the backend pool is set to 'Automatic' but there´s nothing about it in the limitations doc https://learn.microsoft.com/en-us/azure/load-balancer/cross-subscription-overview#limitations

Environment Summary

azure-cli 2.61.0 *

core 2.61.0 *
telemetry 1.1.0

Dependencies:
msal 1.28.0
azure-mgmt-resource 23.1.1

Python (Windows) 3.11.8 (tags/v3.11.8:db85d51, Feb 6 2024, 21:52:07) [MSC v.1937 32 bit (Intel)]

Additional context

Also the alternative way creating the backend pool without sync_mode and adding the address to the pool so then changing the backend pool sync_mode property to 'Automatic' does not work:

(SyncModePropertyCannotBetSetWithLoadBalancerBackendAddressesPresent) Sync mode property of load balancer backend address pool cannot be set with backend addresses present in the pool. Code: SyncModePropertyCannotBetSetWithLoadBalancerBackendAddressesPresent Message: Sync mode property of load balancer backend address pool cannot be set with backend addresses present in the pool.

[azure-client-tools-bot-prd]

Hi @jorgeakanieves,

2.61.0 is not the latest Azure CLI(2.63.0).

If you haven't already attempted to do so, please upgrade to the latest Azure CLI version by following https://learn.microsoft.com/en-us/cli/azure/update-azure-cli.

[yonzhan]

Thank you for opening this issue, we will look into it.

[pattisanta]

Hey I am having the same issue. I can not add backend addresses to backend pool with syncMode is set automatic. I am using bicep for my deployments.

Thanks,
Patti

[microsoft-github-policy-service]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @aznetsuppgithub.