Bug: Docker image-based Lambda failures

[jack-e-tabaska]

Description:

sam deploy intermittently fails while creating Docker image-based Lambdas.

Steps to reproduce:

sam deploy
--stack-name ${STACK_NAME}
--capabilities CAPABILITY_IAM
--no-fail-on-empty-changeset
--resolve-s3
--parameter-overrides REDACTED
--image-repositories bigDumperLambda=${bigDumperRepoUri}
--image-repositories bqLoaderLambda=${bqLoaderRepoUri}
--image-repositories littleCheckerLambda=${littleCheckerRepoUri}
--image-repositories littleDumperLambda=${littleDumperRepoUri}
--image-repositories publisherLambda=${publisherRepoUri}
--image-repositories jobCheckerLambda=${jobCheckerRepoUri}
--image-repositories tableMakerLambda=${tableMakerRepoUri}
--tags exd_version=${EXD_VERSION}

Observed result:

Error message from CloudFormation: Resource handler returned message: "Lambda does not have permission to access the ECR image. Check the ECR permissions. (Service: Lambda, Status Code: 403, Request ID: 3afd69aa-201d-4f73-a500-e739b9bee696) (SDK Attempt Count: 1)" (RequestToken: 35c52deb-ee89-080f-0a66-e94ef9fb4f8e, HandlerErrorCode: AccessDenied)

When I check the ECR repository in question, I find that any preexisting permissions document has been removed.

I can usually resolve this by rerunning the sam deploy command.

Expected result:

Lambdas deployed without errors, on the first try

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: Linux
2. sam --version: 1.142.1
3. AWS region: us-east-1

[dcabib]

@jack-e-tabaska This looks like a classic race condition between CloudFormation creating the Lambda functions and the ECR repository permissions being set up. The fact that you're seeing the ECR permissions document completely removed is particularly telling - this suggests that SAM might be overwriting or racing with the permission setup during deployment.

From what I can see, when deploying multiple Docker image-based Lambda functions simultaneously (you have 7 of them), there could be a timing issue where SAM or CloudFormation is trying to grant the Lambda service permissions to pull from ECR, but something in the deployment process is either removing those permissions prematurely or not waiting for them to propagate before attempting to create the functions. The intermittent nature of the failure - where rerunning fixes it - strongly points to a timing/race condition rather than a configuration issue.

I suspect this might be happening in the SAM CLI's image repository handling code, possibly in how it manages the ECR repository policies when multiple Lambda functions reference the same or different ECR repositories. It could be that when SAM sets up the repository policies for multiple functions, there's a race where one operation overwrites another, or the permissions aren't being applied atomically across all the image repositories you're specifying.

This appears to be a bug in SAM's deployment orchestration rather than something wrong with your configuration. The fact that your --image-repositories flags are correctly mapping each function to its repository suggests your setup is right. Could you share your template.yaml to see how these Lambda functions are defined? Specifically, I'm curious about whether they have explicit IAM roles defined or if they're using SAM-generated roles, as that might affect how the ECR permissions are being managed during the deployment process.

[jack-e-tabaska]

The template file is attached. All of the lambdas use SAM-generated roles.

lambdas.yaml

[abhishektang]

I've submitted a fix for this in PR #8945 as I have previously worked on the --image-repositories flag.

After testing and going through the comments here I found this Root cause: CloudFormation calls ecr:SetRepositoryPolicy concurrently — once per Lambda referencing an ECR image. Each call overwrites the existing policy rather than merging it, so whichever write lands last wins and earlier Lambdas lose pull access (403 on image pull).

This is the fix I came up with Fix: Before creating the changeset, SAM CLI now pre-sets a stable SAMCliLambdaECRAccess policy SID on every ECR repository referenced by the deployment (ImageRepository / ImageRepositories). Because the SID is deterministic, repeated calls are idempotent and concurrent CF calls no longer clobber each other.

Three private helpers added to deploy_context.py:

• _extract_ecr_repo_name — parse repo name from a full ECR URI
• _ensure_ecr_lambda_pull_policy — collect all unique repos and upsert the policy on each
• _upsert_ecr_lambda_policy — idempotently set or merge the statement; retries on concurrent SetRepositoryPolicy conflicts; skips gracefully on AccessDeniedException

PR: #8945. Please let me know if I have to make any other changes or any doubts to clarify regrading this PR, would be happy to do so!