`dd-metrics-forwarder` and `dd-logs-forwarder` error with `FunctionInvokeImageNotAvailable`

[florianmutter]

Summary

In datadog-oci-orm-v1.1.6 (and master), the regional-stacks submodule has two Terraform resources that both call manage_default_resource_id on the same VCN default security list:

1. module.vcn[0].oci_core_default_security_list.lockdown (inside oracle-terraform-modules/vcn/oci, enabled because the Datadog module passes lockdown_default_seclist = true) — writes no egress/ingress rules and has ignore_changes = [egress_security_rules, ingress_security_rules, defined_tags].
2. oci_core_default_security_list.dd_default (in modules/regional-stacks/main.tf) — writes egress-all + two ICMP ingress rules.

On terraform apply, both fire UpdateSecurityList PUTs back-to-back. In our deployment the lockdown resource won the race, leaving the default SL with empty egress_security_rules. The forwarder Functions then could not pull their images from OCIR, and every Service Connector Hub invocation returned:

{"code":"FunctionInvokeImageNotAvailable","message":"Failed to pull function image"}  (502)

This affects both dd-metrics-forwarder and dd-logs-forwarder.

How we found it

OCI Monitoring (oci_faas namespace) showed FunctionResponseCount with dimensions:

ErrorCode = 502                                           
ErrorMessage = FunctionInvokeImageNotAvailable

for both functions and only that error class.

The OCIR images themselves were healthy (anonymous token + manifest fetch returned 200 with the exact digests pinned on the Functions). The route table correctly sent all-fra-services-in-oracle-services-network to the service gateway. The only thing missing was the egress rule on the SL.

OCI audit log confirmed the race: two UpdateSecurityList calls within ~150 ms during apply, both by the Terraform service principal:

2026-04-24T09:51:44.525Z  UpdateSecurityList              
2026-04-24T09:51:44.683Z  UpdateSecurityList                                                                                                                                                                                                                                                                            

Reproduction

1. Apply datadog-terraform-onboarding v1.1.6 against any tenancy without supplying subnet_ocids (so the module creates its own VCN).
2. Wait for apply to finish.
3. oci network security-list get --security-list-id <vcn default SL> --query 'data."egress-security-rules"' → [].
4. Trigger a Service Connector Hub or invoke the Function directly → 502 FunctionInvokeImageNotAvailable.

Workaround

Adding the egress rule manually resolved the issue.

Suggested fix

The oci_core_default_security_list.dd_default resource is sufficient on its own — it already writes locked-down rules. Either:

1. Drop lockdown_default_seclist = true from the call to oracle-terraform-modules/vcn/oci, OR
2. Drop the dd_default resource and let the upstream VCN module's lockdown handle it (then add ICMP ingress separately if needed).

Option 1 keeps the locked-down rule set you intended in #99.

Context

• Tag: datadog-oci-orm-v1.1.6 (commit b495dc1)
• Introducing commit: d0f35cd (PR [CLOUDS-7599] Remove default seclists from VCNs to prevent opening port 22 unnecessarily #99)
• Region: eu-frankfurt-1
• Terraform: 1.12.2, OCI provider ~> 7