Auto-instrumentation hardcodes cloudwatch-agent.amazon-cloudwatch as a 2-label DNS name, forcing search-domain expansion and blocking ndots: 1
The default Instrumentation env vars injected by the operator's admission webhook (pkg/instrumentation/defaultinstrumentation.go) point at the CloudWatch agent service using a short, non-fully-qualified hostname:
cloudwatchAgentServiceEndpoint := "cloudwatch-agent.amazon-cloudwatch"
This is used to build OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, OTEL_AWS_APPLICATION_SIGNALS_EXPORTER_ENDPOINT, OTEL_TRACES_SAMPLER_ARG, OTEL_AWS_JMX_EXPORTER_METRICS_ENDPOINT, and related vars for every Java/Python/.NET/Node.js
pod that gets auto-instrumented. Since the value only has one dot, resolving it depends entirely on the pod's DNS search list (and ndots) to append ..svc.cluster.local / .svc.cluster.local before it succeeds.
This causes several concrete problems:
- Route53 Resolver query logs are polluted with NXDOMAIN. Every OTLP export from every instrumented pod triggers a chain of failed lookups before the correct search-domain suffix is found, and each failed attempt shows up as an NXDOMAIN entry in
Route53 Resolver query logging. On any cluster with meaningful traffic volume this drowns out real DNS issues and adds noise/cost to log-based DNS monitoring.
- ndots: 1 cannot be adopted. Lowering ndots from the Kubernetes default of 5 to 1 is a common, recommended tuning to cut down on exactly this kind of wasted search-domain resolution chatter. But because this hostname has only 1 dot, a resolver with
ndots: 1 would try it as an absolute name first, which fails outright since cloudwatch-agent.amazon-cloudwatch isn't a real absolute domain. So this hardcoded value blocks that DNS tuning cluster-wide for anyone using auto-instrumentation.
- Inconsistent with the Windows path. A few lines below, the Windows headless-service case already uses the fully-qualified form:
cloudwatchAgentServiceEndpoint = "cloudwatch-agent-windows-headless.amazon-cloudwatch.svc.cluster.local"
- There's no technical reason the Linux default couldn't do the same — the FQDN resolves identically and skips the search-list walk entirely.
- Pointless looping over the resolver search path. Every single request from every instrumented process re-triggers the same sequential, multi-query DNS walk (per-namespace suffix, then svc.cluster.local, then cluster domain, etc.) to resolve a name
that is fully known in advance. This adds needless latency and load to CoreDNS/Route53 Resolver for a destination the operator itself controls and could just as easily emit as a full FQDN from the start.
Auto-instrumentation hardcodes cloudwatch-agent.amazon-cloudwatch as a 2-label DNS name, forcing search-domain expansion and blocking ndots: 1
The default Instrumentation env vars injected by the operator's admission webhook (pkg/instrumentation/defaultinstrumentation.go) point at the CloudWatch agent service using a short, non-fully-qualified hostname:
cloudwatchAgentServiceEndpoint := "cloudwatch-agent.amazon-cloudwatch"
This is used to build OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, OTEL_AWS_APPLICATION_SIGNALS_EXPORTER_ENDPOINT, OTEL_TRACES_SAMPLER_ARG, OTEL_AWS_JMX_EXPORTER_METRICS_ENDPOINT, and related vars for every Java/Python/.NET/Node.js
pod that gets auto-instrumented. Since the value only has one dot, resolving it depends entirely on the pod's DNS search list (and ndots) to append ..svc.cluster.local / .svc.cluster.local before it succeeds.
This causes several concrete problems:
Route53 Resolver query logging. On any cluster with meaningful traffic volume this drowns out real DNS issues and adds noise/cost to log-based DNS monitoring.
ndots: 1 would try it as an absolute name first, which fails outright since cloudwatch-agent.amazon-cloudwatch isn't a real absolute domain. So this hardcoded value blocks that DNS tuning cluster-wide for anyone using auto-instrumentation.
cloudwatchAgentServiceEndpoint = "cloudwatch-agent-windows-headless.amazon-cloudwatch.svc.cluster.local"
that is fully known in advance. This adds needless latency and load to CoreDNS/Route53 Resolver for a destination the operator itself controls and could just as easily emit as a full FQDN from the start.