Skip to content

[test/otel/pernode] Add per-node + zero-step CRD E2E suite and EKS harness#720

Open
wenegiemepraise wants to merge 4 commits into
aws:mainfrom
wenegiemepraise:pernode-e2e
Open

[test/otel/pernode] Add per-node + zero-step CRD E2E suite and EKS harness#720
wenegiemepraise wants to merge 4 commits into
aws:mainfrom
wenegiemepraise:pernode-e2e

Conversation

@wenegiemepraise

Copy link
Copy Markdown

Summary

Add the test/otel/pernode integration suite plus a Terraform EKS harness that validates
the ServiceMonitor/PodMonitor scraping path end to end on a live cluster: per-node target
locality, zero-step CRD bundling, and Target Allocator resilience to missing CRDs.

What's included

Suite (test/otel/pernode/, build tag integration):

  • per_node_test.go — every SM/PM series is node-local: target_node == @resource.k8s.node.name; targets span ≥2 nodes.
  • crd_bundling_test.go — SM + PM CRDs are served/established after a plain chart install (harness never installs them separately).
  • ta_resilience_test.go — TA Deployment Available, no CrashLoopBackOff, 0 restarts on a fresh cluster; and it actually discovers monitors.
  • setup_test.go, k8s_helpers_test.go, metrics_test.go — shared clientset/ground-truth, CRD discovery, otelmetrics client bootstrap.
  • resources/workload.yaml — sm-app/pm-app workloads + SM/PM with the target_node relabel + load generator.

Harness (terraform/eks/daemon/otel-pernode/): provisions EKS, installs the chart with the
per-node-capable operator/TA images, applies the workloads, and runs the suite. Because it
exercises unreleased code it takes operator_image_*, ta_image, helm_chart_repo/branch vars.

Verification method (per-node)

target_node             = scraped POD's node   (SM/PM relabel from __meta_kubernetes_pod_node_name)
@resource.k8s.node.name = scraping AGENT's node (from ${env:K8S_NODE_NAME})
per-node holds  ⇔  for every series, target_node == @resource.k8s.node.name

Dependencies / notes

  • Validates the feature PRs (operator per-node + CRD resilience, helm bundling) — most useful
    once those and their custom images are available; tests are skip-guarded so they don't break
    the broader suite where fixtures are absent.
  • go vet -tags integration ./test/otel/pernode/... is clean against current main.

Add an integration suite (test/otel/pernode) and Terraform harness
(terraform/eks/daemon/otel-pernode) that validate the Target Allocator
per-node allocation strategy end to end, plus the zero-step
ServiceMonitor/PodMonitor CRD bundling (G1) and TA resilience to missing
CRDs (G2).

Suite:
- per_node_test.go asserts every scraped series is collected by the agent on
  the scraped pod's own node (target_node == @resource.k8s.node.name) and that
  the workload spans >= 2 nodes.
- crd_bundling_test.go asserts the SM/PM CRDs are served via discovery after a
  plain chart install (no prometheus-operator prerequisite).
- ta_resilience_test.go asserts the TA Deployment is Available with zero
  container restarts and that it discovers the monitors once the CRDs exist.
- resources/workload.yaml: sm-app/pm-app behind a ServiceMonitor and PodMonitor
  with a target_node relabel, plus a load generator.

Harness installs a helm-charts checkout that bundles the CRDs (no separate CRD
install step, by design), deploys custom operator and Target Allocator images
carrying the per-node + CRD-watch code, forces the per-node strategy on the CR,
applies the workload, and runs the suite.
Validated the pernode suite against a live per-node cluster:
- Select Target Allocator pods by app.kubernetes.io/name=cloudwatch-agent-
  target-allocator (the operator labels component as
  amazon-cloudwatch-agent-target-allocator, so the previous selector matched
  nothing).
- Assert the TA is currently Ready/Running and not in CrashLoopBackOff instead
  of requiring a lifetime restart count of 0. The readiness/crashloop check
  still catches a TA that died on a missing CRD, but is portable across reruns
  on long-lived clusters; the lifetime restart count is now logged
  informationally (expected 0 only on a freshly provisioned harness cluster).
- Add local_chart_path var to install the chart from a local checkout
  (skipping the git clone) so unpushed working-tree changes can be exercised.
- Raise the helm_release timeout to 900s for the multi-deployment fresh
  install (operator, agent DaemonSet, Target Allocator, webhook).
@wenegiemepraise wenegiemepraise requested a review from a team as a code owner July 9, 2026 11:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant