az container app job create with managed identity for pulling images

[coin-op]

Describe the bug

When deploying a container app job in one subscription using a managed identity with acrpull rbac to an acr in another subscription, deployment fails with a 500 error

Managed id has acrpull access to a publicly accessible repository. No private endpoints.

When using acr password and username the below command works (taking out the mi stuff and using --registry-password and --registry-username

Related command

az containerapp job create `

--name "some-container-app-job" `

--resource-group "some-resource-group" `

--container-name "manual-jobs" `

--environment "some-environment" `

--trigger-type "Manual" `

--replica-timeout 1800 `

--replica-retry-limit 1 `

--replica-completion-count 1 `

--parallelism 1 `

--workload-profile-name "Consumption" `

--mi-user-assigned "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/some-rg/providers/Microsoft.ManagedIdentity/userAssignedIdentities/msi-with-acr-pull-access-in-another-sub" `

--registry-identity "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/some-rg/providers/Microsoft.ManagedIdentity/userAssignedIdentities/msi-with-acr-pull-access-in-another-sub" `

--registry-server "acrinothersub.azurecr.io" `

--image "acrinothersub.azurecr.io/image:latest" `

--cpu "0.25" `

--memory "0.5Gi" `

--debug

Errors

500 error code Internal server error occurred.

Issue script & Debug output

DEBUG: urllib3.connectionpool: https://management.azure.com:443 "PUT REDACTED HTTP/1.1" 500 203
INFO: cli.azure.cli.core.util: Response status: 500
INFO: cli.azure.cli.core.util: Response headers:
INFO: cli.azure.cli.core.util: 'Cache-Control': 'no-cache'
INFO: cli.azure.cli.core.util: 'Pragma': 'no-cache'
INFO: cli.azure.cli.core.util: 'Content-Length': '203'
INFO: cli.azure.cli.core.util: 'Content-Type': 'application/json; charset=utf-8'
INFO: cli.azure.cli.core.util: 'Expires': '-1'
INFO: cli.azure.cli.core.util: 'x-ms-ratelimit-remaining-subscription-resource-requests': '499'
INFO: cli.azure.cli.core.util: 'api-supported-versions': '2022-11-01-preview, 2023-04-01-preview, 2023-05-01, 2023-05-02-preview, 2023-08-01-preview, 2023-11-02-preview, 2024-02-02-preview'
INFO: cli.azure.cli.core.util: 'Server': 'Microsoft-IIS/10.0'
INFO: cli.azure.cli.core.util: 'X-Powered-By': 'ASP.NET'
INFO: cli.azure.cli.core.util: 'x-ms-failure-cause': 'service'
INFO: cli.azure.cli.core.util: 'x-ms-request-id': 'REDACTED'
INFO: cli.azure.cli.core.util: 'x-ms-correlation-request-id': 'REDACTED'
INFO: cli.azure.cli.core.util: 'x-ms-routing-request-id': 'REDACTED'
INFO: cli.azure.cli.core.util: 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains'
INFO: cli.azure.cli.core.util: 'X-Content-Type-Options': 'nosniff'
INFO: cli.azure.cli.core.util: 'Date': 'Mon, 12 Feb 2024 14:36:29 GMT'
INFO: cli.azure.cli.core.util: 'Connection': 'close'
INFO: cli.azure.cli.core.util: Response content:
INFO: cli.azure.cli.core.util: {"error":{"code":"InternalServerError","message":"Internal server error occurred. correlation ID: 45851f8b-ba37-4d72-9a81-81a758690348","details":null,"target":null,"additionalInfo":null,"traceId":null}}
DEBUG: cli.azure.cli.core.azclierror: Traceback (most recent call last):
File "/opt/az/lib/python3.11/site-packages/azure/cli/command_modules/containerapp/containerapp_job_decorator.py", line 201, in create
r = self.client.create_or_update(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/az/lib/python3.11/site-packages/azure/cli/command_modules/containerapp/_clients.py", line 848, in create_or_update
r = send_raw_request(cmd.cli_ctx, "PUT", request_url, body=json.dumps(containerapp_job_envelope))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/az/lib/python3.11/site-packages/azure/cli/core/util.py", line 1004, in send_raw_request
raise HTTPError(reason, r)
azure.cli.core.azclierror.HTTPError: Internal Server Error({"error":{"code":"InternalServerError","message":"Internal server error occurred. correlation ID: 45851f8b-ba37-4d72-9a81-81a758690348","details":null,"target":null,"additionalInfo":null,"traceId":null}})

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/az/lib/python3.11/site-packages/knack/cli.py", line 233, in invoke
cmd_result = self.invocation.execute(args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/az/lib/python3.11/site-packages/azure/cli/core/commands/init.py", line 663, in execute
raise ex
File "/opt/az/lib/python3.11/site-packages/azure/cli/core/commands/init.py", line 726, in _run_jobs_serially
results.append(self._run_job(expanded_arg, cmd_copy))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/az/lib/python3.11/site-packages/azure/cli/core/commands/init.py", line 718, in _run_job
return cmd_copy.exception_handler(ex)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/az/lib/python3.11/site-packages/azure/cli/command_modules/containerapp/_client_factory.py", line 28, in _polish_bad_errors
raise ex
File "/opt/az/lib/python3.11/site-packages/azure/cli/core/commands/init.py", line 697, in _run_job
result = cmd_copy(params)
^^^^^^^^^^^^^^^^
File "/opt/az/lib/python3.11/site-packages/azure/cli/core/commands/init.py", line 333, in call
return self.handler(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/az/lib/python3.11/site-packages/azure/cli/core/commands/command_operation.py", line 121, in handler
return op(**command_args)
^^^^^^^^^^^^^^^^^^
File "/opt/az/lib/python3.11/site-packages/azure/cli/command_modules/containerapp/custom.py", line 960, in create_containerappsjob
r = containerapp_job_create_decorator.create()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/az/lib/python3.11/site-packages/azure/cli/command_modules/containerapp/containerapp_job_decorator.py", line 206, in create
handle_raw_exception(e)
File "/opt/az/lib/python3.11/site-packages/azure/cli/command_modules/containerapp/_client_factory.py", line 53, in handle_raw_exception
raise CLIInternalError('({}) {}'.format(code, message))
azure.cli.core.azclierror.CLIInternalError: (InternalServerError) Internal server error occurred. correlation ID: 45851f8b-ba37-4d72-9a81-81a758690348

ERROR: cli.azure.cli.core.azclierror: (InternalServerError) Internal server error occurred. correlation ID: 45851f8b-ba37-4d72-9a81-81a758690348
ERROR: az_command_data_logger: (InternalServerError) Internal server error occurred. correlation ID: 45851f8b-ba37-4d72-9a81-81a758690348
DEBUG: cli.knack.cli: Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x7f9430171260>]

Expected behavior

Expect a new job to be created using the rbac permissions to pull from acr.

Environment Summary

azure-cli 2.56.0 *

core 2.56.0 *
telemetry 1.1.0

Extensions:
azure-devops 0.26.0

Dependencies:
msal 1.24.0b2
azure-mgmt-resource 23.1.0b2

(AzureCLI@2 pipeline task)

Additional context

No response

[azure-client-tools-bot-prd]

Hi @coin-op,

2.56.0 is not the latest Azure CLI(2.57.0).

If you haven't already attempted to do so, please upgrade to the latest Azure CLI version by following https://learn.microsoft.com/en-us/cli/azure/update-azure-cli.

[yonzhan]

Thank you for opening this issue, we will look into it.

[microsoft-github-policy-service]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @dkkapur.

[microsoft-github-policy-service]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @calvinsID.

[microsoft-github-policy-service]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @dkkapur.

[microsoft-github-policy-service]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @calvinsID.

[Greedygre]

Hi @anandanthony

Can you help to take a look or find someone relate it to handle? Thanks!