`az ad sp create-for-rbac` intermittently fails with `Directory_ConcurrencyViolation`

[jurjenoskam]

On Azure CLI version 2.37.0, the command az ad sp create-for-rbac command started to intermittently fail with the following error:

Error due to concurrent requests being made to the tenant. Please wait briefly and retry.

Looking at the debug log, the cause was the /addPassword POST call failing with an HTTP 503:

urllib3.connectionpool: Starting new HTTPS connection (1): graph.microsoft.com:443
urllib3.connectionpool: https://graph.microsoft.com:443 "POST /v1.0/applications/{redacted_for_github_issue}/addPassword HTTP/1.1" 503 None
cli.azure.cli.core.util: Response status: 503
cli.azure.cli.core.util: Response headers:
cli.azure.cli.core.util:     'Cache-Control': 'no-cache'
cli.azure.cli.core.util:     'Transfer-Encoding': 'chunked'
cli.azure.cli.core.util:     'Content-Type': 'application/json'
cli.azure.cli.core.util:     'Content-Encoding': 'gzip'
cli.azure.cli.core.util:     'Vary': 'Accept-Encoding'
cli.azure.cli.core.util:     'Strict-Transport-Security': 'max-age=31536000'
cli.azure.cli.core.util:     'request-id': 'dfef9c16-ab41-4e64-a05d-ed0501997cd1'
cli.azure.cli.core.util:     'client-request-id': 'dfef9c16-ab41-4e64-a05d-ed0501997cd1'
cli.azure.cli.core.util:     'x-ms-ags-diagnostic': '{"ServerInfo":{"DataCenter":"West Europe","Slice":"E","Ring":"5","ScaleUnit":"001","RoleInstance":"AM4PEPF00015127"}}'
cli.azure.cli.core.util:     'x-ms-resource-unit': '1'
cli.azure.cli.core.util:     'Date': 'Mon, 20 Jun 2022 12:50:27 GMT'
cli.azure.cli.core.util: Response content:
cli.azure.cli.core.util: {"error":{"code":"Directory_ConcurrencyViolation","message":"Error due to concurrent requests being made to the tenant. Please wait briefly and retry.","innerError":{"date":"2022-06-20T12:50:27","request-id":"dfef9c16-ab41-4e64-a05d-ed0501997cd1","client-request-id":"dfef9c16-ab41-4e64-a05d-ed0501997cd1"}}}

After opening a Microsoft Service Request (number 2206210050000221) against Graph to determine why this call was intermittently failing in our tenant, the response was that this can happen when there are multiple update requests to the same object within a few seconds, and that the request should be retried after 1-2 minutes when this error occurs.

Looking at the Azure CLI debug log, the /addPassword call was indeed immediately preceded by a /removePassword call for the same Application Registration.

It seems that Azure CLI issues the /addPassword call too soon after the /removePassword call and does not have a retry mechanism, causing the intermittent error we see.

Could Azure CLI be changed to wait a bit longer between update calls to Graph and/or implement a retry mechanism in case of concurrency violation errors?

[yonzhan]

@jiasli for awareness

[jiasli]

the request should be retried after 1-2 minutes when this error occurs.

I doubt this is the correct way to solve this issue. Waiting 1~2 min is definitely not a good practice for automation scenarios as this significantly slows down command/script execution. Consider az ad sp create-for-rbac now takes 1~2 min to finish when it should only take seconds.

I will contact Graph team internally for this issue and update you.

[jurjenoskam]

I think the suggestion was to only wait 1-2 minutes in case a Directory_ConcurrencyViolation is encountered. Waiting unconditionally for such a long time after each update would indeed be an enormous waste.

(Of course the best solution would be for Graph to only return success for synchronous API calls once it's ready to immediately receive another update for the same object, so I hope you can convince the Graph team of implementing that :) )

[jiasli]

@jurjenoskam, even waiting for Directory_ConcurrencyViolation is unacceptable from CLI perspective.

Of course the best solution would be for Graph to only return success for synchronous API calls once it's ready to immediately receive another update for the same object, so I hope you can convince the Graph team of implementing that

This is certainly good suggestion.

Since I don't have permission to view the Service Request 2206210050000221, do you mind sending me an email to {my_github_username}@microsoft.com and cc the support engineer who was working with you on that case? Thanks a lot.

[jurjenoskam]

@jiasli Thanks, I've just sent the mail.

[jiasli]

Thank you @jurjenoskam, I will internally discuss with Microsoft Graph product group and update you if there is any update.