[test/otel/pernode] Add per-node + zero-step CRD E2E suite and EKS harness#720
Open
wenegiemepraise wants to merge 4 commits into
Open
[test/otel/pernode] Add per-node + zero-step CRD E2E suite and EKS harness#720wenegiemepraise wants to merge 4 commits into
wenegiemepraise wants to merge 4 commits into
Conversation
Add an integration suite (test/otel/pernode) and Terraform harness (terraform/eks/daemon/otel-pernode) that validate the Target Allocator per-node allocation strategy end to end, plus the zero-step ServiceMonitor/PodMonitor CRD bundling (G1) and TA resilience to missing CRDs (G2). Suite: - per_node_test.go asserts every scraped series is collected by the agent on the scraped pod's own node (target_node == @resource.k8s.node.name) and that the workload spans >= 2 nodes. - crd_bundling_test.go asserts the SM/PM CRDs are served via discovery after a plain chart install (no prometheus-operator prerequisite). - ta_resilience_test.go asserts the TA Deployment is Available with zero container restarts and that it discovers the monitors once the CRDs exist. - resources/workload.yaml: sm-app/pm-app behind a ServiceMonitor and PodMonitor with a target_node relabel, plus a load generator. Harness installs a helm-charts checkout that bundles the CRDs (no separate CRD install step, by design), deploys custom operator and Target Allocator images carrying the per-node + CRD-watch code, forces the per-node strategy on the CR, applies the workload, and runs the suite.
Validated the pernode suite against a live per-node cluster: - Select Target Allocator pods by app.kubernetes.io/name=cloudwatch-agent- target-allocator (the operator labels component as amazon-cloudwatch-agent-target-allocator, so the previous selector matched nothing). - Assert the TA is currently Ready/Running and not in CrashLoopBackOff instead of requiring a lifetime restart count of 0. The readiness/crashloop check still catches a TA that died on a missing CRD, but is portable across reruns on long-lived clusters; the lifetime restart count is now logged informationally (expected 0 only on a freshly provisioned harness cluster).
- Add local_chart_path var to install the chart from a local checkout (skipping the git clone) so unpushed working-tree changes can be exercised. - Raise the helm_release timeout to 900s for the multi-deployment fresh install (operator, agent DaemonSet, Target Allocator, webhook).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add the
test/otel/pernodeintegration suite plus a Terraform EKS harness that validatesthe ServiceMonitor/PodMonitor scraping path end to end on a live cluster: per-node target
locality, zero-step CRD bundling, and Target Allocator resilience to missing CRDs.
What's included
Suite (
test/otel/pernode/, build tagintegration):per_node_test.go— every SM/PM series is node-local:target_node == @resource.k8s.node.name; targets span ≥2 nodes.crd_bundling_test.go— SM + PM CRDs are served/established after a plain chart install (harness never installs them separately).ta_resilience_test.go— TA Deployment Available, no CrashLoopBackOff, 0 restarts on a fresh cluster; and it actually discovers monitors.setup_test.go,k8s_helpers_test.go,metrics_test.go— shared clientset/ground-truth, CRD discovery, otelmetrics client bootstrap.resources/workload.yaml— sm-app/pm-app workloads + SM/PM with thetarget_noderelabel + load generator.Harness (
terraform/eks/daemon/otel-pernode/): provisions EKS, installs the chart with theper-node-capable operator/TA images, applies the workloads, and runs the suite. Because it
exercises unreleased code it takes
operator_image_*,ta_image,helm_chart_repo/branchvars.Verification method (per-node)
Dependencies / notes
once those and their custom images are available; tests are skip-guarded so they don't break
the broader suite where fixtures are absent.
go vet -tags integration ./test/otel/pernode/...is clean against currentmain.