Azure ACR direct import from docker registry error

[ogbonnaec]

Describe the bug

Hello
I am currently having an error while trying to import the kubernetes-dashboard helm chart container images into our ACR.
I am using an existing template that works fine.
I have been able to import the helm chart directly but the containers referenced by the chart is the issue. Yesterday, I had 4 of them falling but today, one was successful. I have tried authenticating with my docker hub user and password. The log in was successful but I still get an error with the import command. All variables and parameters valid and not null or missing. I would appreciate any ideas thank you.

Related command

azure-pipelines-container.yaml

jobs:
- ${{ each image in parameters.image }} :
  - deployment: pullPushContainerImage${{ replace(replace(image.name, '-', ''), '/', '') }}
    displayName: "pull/push container image ${{ image.name }}"
    environment: ${{ parameters.environment }}
    strategy:
      runOnce:
        deploy:
          steps:
            - task: AzureCLI@2
              displayName: 'debug'
              inputs:
                inlineScript: |
                  echo "--image ${{ image.registry }}/${{ image.name }}:${{ image.tag }}"
                  echo "import --name ${{ parameters.azureContainerRegistry }} --source ${{ image.registry }}/${{ image.name }}:${{ image.tag }} --image ${{ image.registry }}/${{ image.name }}:${{ image.tag }} "
                scriptType: 'bash'
                scriptLocation: 'inlineScript'
                addSpnToEnvironment: true
                azureSubscription: ${{ parameters.azureSubscriptionEndpoint }}
            - task: AzureCLI@2
              displayName: 'pull/push ${{ image.name }}'
              inputs:
                inlineScript: |
                  set -x
                  az acr login -n ${{ parameters.azureContainerRegistry }} --expose-token
                  echo ${{ parameters.dockerPassword }} | docker login --username ${{ parameters.dockerUser }} --password-stdin 
                  if [[ ${{ image.registry }} =~ "azurecr.io" ]]; then
                    if ! az acr repository show -n ${{ parameters.azureContainerRegistry }} --image ${{ image.name }}:${{ image.tag }}  2>nul; then az acr import --name ${{ parameters.azureContainerRegistry }} --source ${{ image.registry }}/${{ image.name }}:${{ image.tag }} --image ${{ image.name }}:${{ image.tag }} 2>nul; else echo "${{ parameters.azureContainerRegistry }}/${{ image.name }}:${{ image.tag }} already present"; fi
                  else
                    if ! az acr repository show -n ${{ parameters.azureContainerRegistry }} --image ${{ image.registry }}/${{ image.name }}:${{ image.tag }}; then az acr import --name ${{ parameters.azureContainerRegistry }} --source ${{ image.registry }}/${{ image.name }}:${{ image.tag }} --image ${{ image.registry }}/${{ image.name }}:${{ image.tag }} 2>nul; else echo "${{ parameters.azureContainerRegistry }}/${{ image.name }}:${{ image.tag }} already present"; fi
                  fi
                scriptType: 'bash'
                scriptLocation: 'inlineScript'
                addSpnToEnvironment: true
                azureSubscription: ${{ parameters.azureSubscriptionEndpoint }}

azure-pipelines.yaml

stages:
- stage: Tags
  jobs:
  - job: GetTags
    steps:
      - checkout: self
        persistCredentials: true
        fetchDepth: 0
      - task: AzureCLI@2
        displayName: Get Kubernetes Dashboard image tags
        name: dashboardTags
        inputs:
          connectionType: 'Azure Resource Manager'
          azureSubscription: $(backendServiceConnectionName)
          addSpnToEnvironment: true
          scriptType: "bash"
          scriptLocation: 'inlineScript'
          inlineScript: |
            echo "Fetching the latest Kubernetes Dashboard Helm chart tag"
            #latestDashboardHelmChartTag=$(curl -s https://api.github.com/repos/kubernetes/dashboard/releases | jq -r '[.[] | select(.tag_name | startswith("kubernetes-dashboard-"))][0].tag_name | sub("kubernetes-dashboard-"; "")')
            latestDashboardHelmChartTag=7.11.1
            echo "Latest Dashboard Helm chart release is ${latestDashboardHelmChartTag}"

            echo "Fetching the corresponding container image tags from values.yaml"
            valuesYamlUrl="https://raw.githubusercontent.com/kubernetes/dashboard/kubernetes-dashboard-${latestDashboardHelmChartTag}/charts/kubernetes-dashboard/values.yaml"

            webImageTag=$(curl -s ${valuesYamlUrl} | yq -o=json -r '.web.image.tag')
            apiImageTag=$(curl -s ${valuesYamlUrl} | yq -o=json -r '.api.image.tag')
            authImageTag=$(curl -s ${valuesYamlUrl} | yq -o=json -r '.auth.image.tag')
            metricsScraperImageTag=$(curl -s ${valuesYamlUrl} | yq -o=json -r '.metricsScraper.image.tag')

            echo "dashboard-web:${webImageTag}"
            echo "dashboard-api:${apiImageTag}"
            echo "dashboard-auth:${authImageTag}"
            echo "metrics-scraper:${metricsScraperImageTag}"

            echo "##vso[task.setvariable variable=dashboardHelmChartTag;isOutput=true]${latestDashboardHelmChartTag}" 
            echo "##vso[task.setvariable variable=webImageTag;isOutput=true]${webImageTag}"
            echo "##vso[task.setvariable variable=apiImageTag;isOutput=true]${apiImageTag}"
            echo "##vso[task.setvariable variable=authImageTag;isOutput=true]${authImageTag}"
            echo "##vso[task.setvariable variable=metricsScraperImageTag;isOutput=true]${metricsScraperImageTag}" 
 
- stage: helmCharts
  displayName: "pull/push charts"
  condition: succeeded()
  dependsOn: Tags
  variables:
    dashboardChartTag: $[stageDependencies.Tags.GetTags.outputs['dashboardTags.dashboardHelmChartTag'] ] 
  jobs:
    - template: azure-pipelines-helm.yaml
      parameters:
        environment: $(environment)
        serviceConnectionName: $(azureSubscriptionEndpoint)
        azureContainerRegistry: $(azureContainerRegistry)
        azureSubscriptionEndpoint: $(azureSubscriptionEndpoint)
        chart:
          - name: kubernetes-dashboard
            url: 'https://github.com/kubernetes/dashboard/releases/download/kubernetes-dashboard-$(dashboardChartTag)/kubernetes-dashboard-$(dashboardChartTag).tgz'
            tag: $(dashboardChartTag)
 
- stage: containerImages
  displayName: "pull/push container images"
  condition: succeeded()
  dependsOn: Tags
  variables:
    webImageTag: $[stageDependencies.Tags.GetTags.outputs['dashboardTags.webImageTag'] ]
    apiImageTag: $[stageDependencies.Tags.GetTags.outputs['dashboardTags.apiImageTag'] ]
    authImageTag: $[stageDependencies.Tags.GetTags.outputs['dashboardTags.authImageTag'] ]
    metricsScraperImageTag: $[stageDependencies.Tags.GetTags.outputs['dashboardTags.metricsScraperImageTag'] ]
  jobs:
    - template: azure-pipelines-container.yaml
      parameters:
        environment: $(environment)
        serviceConnectionName: $(azureSubscriptionEndpoint)
        azureContainerRegistry: $(azureContainerRegistry)
        azureSubscriptionEndpoint: $(azureSubscriptionEndpoint)
        dockerUser: $(dockerUser)
        dockerPassword: $(dockerPassword)
        image:
          - name: 'kubernetesui/dashboard-auth'
            registry: 'docker.io'
            tag: $(authImageTag)
          - name: 'kubernetesui/dashboard-api'
            registry: 'docker.io'
            tag: $(apiImageTag)
          - name: 'kubernetesui/dashboard-web'
            registry: 'docker.io'
            tag: $(webImageTag)
          - name: 'kubernetesui/dashboard-metrics-scraper'
            registry: 'docker.io'
            tag: $(metricsScraperImageTag)

Errors

+ docker login --username ********** --password-stdin

Login Succeeded
+ [[ docker.io =~ azurecr\.io ]]
+ az acr repository show -n ourreg.azurecr.io --image docker.io/kubernetesui/dashboard-auth:1.2.4
WARNING: The login server endpoint suffix '.azurecr.io' is automatically omitted.
ERROR: 2025-05-28 12:13:49.913425 Error: the specified tag does not exist. Correlation ID: a7bd2cf8-8dc6-4a08-8056-7f04537a74e7.
+ az acr import --name ****.azurecr.io --source docker.io/kubernetesui/dashboard-auth:1.2.4 --image docker.io/kubernetesui/dashboard-auth:1.2.4 --debug

##[error]Script failed with exit code: 1

+ docker login --username ********* --password-stdin

Login Succeeded
+ [[ docker.io =~ azurecr\.io ]]
+ az acr repository show -n ourreg.azurecr.io --image docker.io/kubernetesui/dashboard-api:1.11.1
WARNING: The login server endpoint suffix '.azurecr.io' is automatically omitted.
ERROR: 2025-05-28 12:14:26.851169 Error: the specified tag does not exist. Correlation ID: 8f46f972-4d50-4596-b23d-2c90b4975ab3.
+ az acr import --name ourrepo.azurecr.io --source docker.io/kubernetesui/dashboard-api:1.11.1 --image docker.io/kubernetesui/dashboard-api:1.11.1 --debug

##[error]Script failed with exit code: 1

+ docker login --username ********** --password-stdin

Login Succeeded
+ [[ docker.io =~ azurecr\.io ]]
+ az acr repository show -n ourreg.azurecr.io --image docker.io/kubernetesui/dashboard-metrics-scraper:1.2.2
WARNING: The login server endpoint suffix '.azurecr.io' is automatically omitted.
ERROR: 2025-05-28 12:14:03.411593 Error: the specified tag does not exist. Correlation ID: 7e5c38f4-c2d7-4600-b469-7ca6dfd22c85.
+ az acr import --name ourrepo.azurecr.io --source docker.io/kubernetesui/dashboard-metrics-scraper:1.2.2 --image docker.io/kubernetesui/dashboard-metrics-scraper:1.2.2 --debug

##[error]Script failed with exit code: 1

Issue script & Debug output

The --debug and --verbose flag does not give additional information. Below are the output of my debug task

--image docker.io/kubernetesui/dashboard-metrics-scraper:1.2.2
import --name ourreg.azurecr.io --source docker.io/kubernetesui/dashboard-metrics-scraper:1.2.2 --image docker.io/kubernetesui/dashboard-metrics-scraper:1.2.2 

--image docker.io/kubernetesui/dashboard-api:1.11.1
import --name ourreg.azurecr.io --source docker.io/kubernetesui/dashboard-api:1.11.1 --image docker.io/kubernetesui/dashboard-api:1.11.1 

--image docker.io/kubernetesui/dashboard-auth:1.2.4
import --name ourreg.azurecr.io --source docker.io/kubernetesui/dashboard-auth:1.2.4 --image docker.io/kubernetesui/dashboard-auth:1.2.4 

Expected behavior

+ [[ docker.io =~ azurecr\.io ]]
+ az acr repository show -n ourreg.azurecr.io --image docker.io/kubernetesui/dashboard-web:1.6.2
WARNING: The login server endpoint suffix '.azurecr.io' is automatically omitted.
{
 .......
}
+ echo 'ourreg.azurecr.io/kubernetesui/dashboard-web:1.6.2 already present'
ourreg.azurecr.io/kubernetesui/dashboard-web:1.6.2 already present

Environment Summary

azure-cli 2.68.0 *

core 2.68.0 *
telemetry 1.1.0

Extensions:
azure-devops 1.0.1

Dependencies:
msal 1.31.1
azure-mgmt-resource 23.1.1

Python location '/opt/az/bin/python3'
Extensions directory '/opt/az/azcliextensions'

Python (Linux) 3.12.8 (main, Jan 8 2025, 03:38:17) [GCC 11.4.0]

Additional context

No response

[azure-client-tools-bot-prd]

Hi @ogbonnaec,

2.68.0 is not the latest Azure CLI(2.73.0).

If you haven't already attempted to do so, please upgrade to the latest Azure CLI version by following https://learn.microsoft.com/en-us/cli/azure/update-azure-cli.

[yonzhan]

Thank you for opening this issue, we will look into it.

[microsoft-github-policy-service]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @toddysm, @northtyphoon, @luisdlp, @terencet-dev, @shizhMSFT, @JXavierMSFT.

[github-actions]

Here are some similar issues that might help you. Please check if they can solve your problem.

• race condition in 'az acr import' can lead to 'manifest unknown' error in target registry #29974

---

Possible solution (Extracted from existing issue, might be incorrect; please verify carefully)

We worked around the issue by copying the images by digest, not by tag - but since that's just a workaround, it is very likely that the issue still exists.

Reference:

• race condition in 'az acr import' can lead to 'manifest unknown' error in target registry #29974 (comment)

Powered by issue-sentinel

[microsoft-github-policy-service]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @toddysm, @northtyphoon, @luisdlp, @terencet-dev, @shizhMSFT, @JXavierMSFT.

[ogbonnaec]

I have upgraded from 2.68.0 to latest Azure CLI(2.73.0) yet the error persists