Skip to content

Auto-instrumentation hardcodes cloudwatch-agent.amazon-cloudwatch as a 2-label DNS name #392

Description

@pete911

Auto-instrumentation hardcodes cloudwatch-agent.amazon-cloudwatch as a 2-label DNS name, forcing search-domain expansion and blocking ndots: 1

The default Instrumentation env vars injected by the operator's admission webhook (pkg/instrumentation/defaultinstrumentation.go) point at the CloudWatch agent service using a short, non-fully-qualified hostname:

cloudwatchAgentServiceEndpoint := "cloudwatch-agent.amazon-cloudwatch"

This is used to build OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, OTEL_AWS_APPLICATION_SIGNALS_EXPORTER_ENDPOINT, OTEL_TRACES_SAMPLER_ARG, OTEL_AWS_JMX_EXPORTER_METRICS_ENDPOINT, and related vars for every Java/Python/.NET/Node.js
pod that gets auto-instrumented. Since the value only has one dot, resolving it depends entirely on the pod's DNS search list (and ndots) to append ..svc.cluster.local / .svc.cluster.local before it succeeds.

This causes several concrete problems:

  1. Route53 Resolver query logs are polluted with NXDOMAIN. Every OTLP export from every instrumented pod triggers a chain of failed lookups before the correct search-domain suffix is found, and each failed attempt shows up as an NXDOMAIN entry in
    Route53 Resolver query logging. On any cluster with meaningful traffic volume this drowns out real DNS issues and adds noise/cost to log-based DNS monitoring.
  2. ndots: 1 cannot be adopted. Lowering ndots from the Kubernetes default of 5 to 1 is a common, recommended tuning to cut down on exactly this kind of wasted search-domain resolution chatter. But because this hostname has only 1 dot, a resolver with
    ndots: 1 would try it as an absolute name first, which fails outright since cloudwatch-agent.amazon-cloudwatch isn't a real absolute domain. So this hardcoded value blocks that DNS tuning cluster-wide for anyone using auto-instrumentation.
  3. Inconsistent with the Windows path. A few lines below, the Windows headless-service case already uses the fully-qualified form:
    cloudwatchAgentServiceEndpoint = "cloudwatch-agent-windows-headless.amazon-cloudwatch.svc.cluster.local"
  4. There's no technical reason the Linux default couldn't do the same — the FQDN resolves identically and skips the search-list walk entirely.
  5. Pointless looping over the resolver search path. Every single request from every instrumented process re-triggers the same sequential, multi-query DNS walk (per-namespace suffix, then svc.cluster.local, then cluster domain, etc.) to resolve a name
    that is fully known in advance. This adds needless latency and load to CoreDNS/Route53 Resolver for a destination the operator itself controls and could just as easily emit as a full FQDN from the start.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions