Skip to content

Add root-level cluster_name and standardize collection_interval for OpenTelemetry features #2189

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Paamicky wants to merge 5 commits into
base: main
Choose a base branch
from
Open

Add root-level cluster_name and standardize collection_interval for OpenTelemetry features #2189

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
7e43242
Add root-level cluster_name and standardize collection_interval for …
Paamicky Jul 9, 2026
67501ae
Add root-level cluster_name and standardize collection_interval for …
Paamicky Jul 9, 2026
38aaca2
fix: make cluster_name for prometheus optional
Paamicky Jul 9, 2026
462d525
fix host metrics azure and aks config
Paamicky Jul 10, 2026
59cea26
fix resource statements for logs and metrics
Paamicky Jul 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion translator/config/sampleSchema/opentelemetry/invalidOpenTelemetryPrometheus.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,9 @@
"region": "us-west-2"
},
"opentelemetry": {
"cluster_name": "my-cluster",
"collect": {
"prometheus": {
"cluster_name": "my-cluster"
}
}
}
Expand Down
4 changes: 2 additions & 2 deletions translator/config/sampleSchema/opentelemetry/validCombinedV1V2Config.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,12 +34,12 @@
}
},
"opentelemetry": {
"cluster_name": "TestCluster",
"collect": {
"host_metrics": {
"metrics_collection_interval": 60
"collection_interval": 60
},
"container_insights": {
"cluster_name": "TestCluster",
"collection_interval": 30
},
"database_insights": {
Expand Down
15 changes: 5 additions & 10 deletions translator/config/schema.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1806,6 +1806,10 @@
"opentelemetryDefinition": {
"type": "object",
"properties": {
"cluster_name": {
"description": "The name of the EKS cluster. Applied to all components under opentelemetry.",
"type": "string"
},
"collect": {
"type": "object",
"properties": {
Expand Down Expand Up @@ -1840,7 +1844,7 @@
"host_metrics": {
"type": "object",
"properties": {
"metrics_collection_interval": {
"collection_interval": {
"description": "How often the metrics defined will be collected",
"$ref": "#/definitions/timeIntervalDefinition"
}
Expand All @@ -1850,10 +1854,6 @@
"container_insights": {
"type": "object",
"properties": {
"cluster_name": {
"description": "The name of the EKS cluster",
"type": "string"
},
"collection_interval": {
"description": "How often metrics are collected",
"$ref": "#/definitions/timeIntervalDefinition"
Expand All @@ -1874,7 +1874,6 @@
"additionalProperties": false
}
},
"required": ["cluster_name"],
"additionalProperties": false
},
"prometheus": {
Expand All @@ -1883,10 +1882,6 @@
"config_path": {
"description": "Path to the Prometheus scrape configuration file",
"type": "string"
},
"cluster_name": {
"description": "The name of the EKS cluster",
"type": "string"
}
},
"required": ["config_path"],
Expand Down
3 changes: 2 additions & 1 deletion translator/tocwconfig/sampleConfig/opentelemetry/combined_v1_v2_ec2_config.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -125,9 +125,10 @@
"endpoint_override": "https://x-ray-endpoint.us-west-2.amazonaws.com"
},
"opentelemetry": {
"cluster_name": "TestCluster",
"collect": {
"host_metrics": {
"metrics_collection_interval": 60
"collection_interval": 60
},
"database_insights": {
"postgresql": [
Expand Down
34 changes: 29 additions & 5 deletions translator/tocwconfig/sampleConfig/opentelemetry/combined_v1_v2_ec2_config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1017,6 +1017,26 @@ processors:
- set(scope.attributes["cloudwatch.source"], "cloudwatch-agent")
- set(scope.attributes["cloudwatch.solution"], "otel-prometheus")
trace_statements: []
transform/set_cluster_name:
error_mode: ignore
flatten_data: false
log_statements: []
metric_statements:
- context: resource
error_mode: ignore
statements:
- set(resource.attributes["k8s.cluster.name"], "TestCluster")
trace_statements: []
transform/set_cluster_name_logs:
error_mode: ignore
flatten_data: false
log_statements:
- context: resource
error_mode: ignore
statements:
- set(resource.attributes["k8s.cluster.name"], "TestCluster")
metric_statements: []
trace_statements: []
Comment on lines +1020 to +1039

@jefchien jefchien Jul 10, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't this have resource statements for each of the resource types? I see the same processor is being used in metric pipelines as well.

@Paamicky Paamicky Jul 10, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. otlp logs now uses its own processor (transform/set_cluster_name_logs) so it doesn't overwrite the metric_statements used by the metrics pipelines

receivers:
awsxray:
endpoint: 0.0.0.0:2000
Expand Down Expand Up @@ -1531,6 +1551,7 @@ service:
- forward/opentelemetry
processors:
- transform/otlp_scope
- transform/set_cluster_name_logs
- transform/otlp_log_source
receivers:
- otlp/grpc_127_0_0_1_4327
Expand All @@ -1553,18 +1574,19 @@ service:
- ec2tagger
- awsentity/resource
receivers:
- telegraf_cpu
- telegraf_processes
- telegraf_procstat/1917393364
- telegraf_swap
- telegraf_mem
- telegraf_cpu
- telegraf_netstat
- telegraf_disk
- telegraf_procstat/1917393364
- telegraf_swap
- telegraf_processes
metrics/host_metrics:
exporters:
- forward/opentelemetry
processors:
- transform/host_metrics_scope
- transform/set_cluster_name
receivers:
- hostmetrics/opentelemetry
metrics/hostCustomMetrics:
Expand All @@ -1584,8 +1606,8 @@ service:
- ec2tagger
- awsentity/resource
receivers:
- telegraf_net
- telegraf_diskio
- telegraf_net
metrics/opentelemetry:
exporters:
- otlphttp/metrics
Expand All @@ -1600,13 +1622,15 @@ service:
- forward/opentelemetry
processors:
- transform/prometheus_scope
- transform/set_cluster_name
receivers:
- prometheus/opentelemetry
metrics/otlp:
exporters:
- forward/opentelemetry
processors:
- transform/otlp_scope
- transform/set_cluster_name
receivers:
- otlp/grpc_127_0_0_1_4327
- otlp/http_127_0_0_1_4328
Expand Down
7 changes: 3 additions & 4 deletions translator/tocwconfig/sampleConfig/opentelemetry/combined_v1_v2_eks_config.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -125,12 +125,12 @@
"endpoint_override": "https://x-ray-endpoint.us-west-2.amazonaws.com"
},
"opentelemetry": {
"cluster_name": "TestCluster",
"collect": {
"host_metrics": {
"metrics_collection_interval": 60
"collection_interval": 60
},
"container_insights": {
"cluster_name": "TestCluster",
"collection_interval": 30
},
"database_insights": {
Expand All @@ -147,8 +147,7 @@
]
},
"prometheus": {
"config_path": "./sampleConfig/opentelemetry/prometheus_otel_scrape_config.yaml",
"cluster_name": "TestCluster"
"config_path": "./sampleConfig/opentelemetry/prometheus_otel_scrape_config.yaml"
},
"otlp": {
"grpc_endpoint": "127.0.0.1:4327",
Expand Down
25 changes: 19 additions & 6 deletions translator/tocwconfig/sampleConfig/opentelemetry/combined_v1_v2_eks_config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1557,6 +1557,16 @@ processors:
statements:
- set(resource.attributes["k8s.cluster.name"], "TestCluster")
trace_statements: []
transform/set_cluster_name_logs:
error_mode: ignore
flatten_data: false
log_statements:
- context: resource
error_mode: ignore
statements:
- set(resource.attributes["k8s.cluster.name"], "TestCluster")
metric_statements: []
trace_statements: []
receivers:
awsefareceiver/cw_k8s_ci_v0:
collection_interval: 30s
Expand Down Expand Up @@ -2228,6 +2238,7 @@ service:
- forward/opentelemetry
processors:
- transform/otlp_scope
- transform/set_cluster_name_logs
- transform/otlp_log_source
receivers:
- otlp/grpc_127_0_0_1_4327
Expand Down Expand Up @@ -2418,18 +2429,19 @@ service:
- ec2tagger
- awsentity/resource
receivers:
- telegraf_disk
- telegraf_netstat
- telegraf_cpu
- telegraf_procstat/1917393364
- telegraf_swap
- telegraf_mem
- telegraf_processes
- telegraf_netstat
- telegraf_disk
- telegraf_procstat/1917393364
- telegraf_cpu
metrics/host_metrics:
exporters:
- forward/opentelemetry
processors:
- transform/host_metrics_scope
- transform/set_cluster_name
receivers:
- hostmetrics/opentelemetry
metrics/hostCustomMetrics:
Expand All @@ -2439,8 +2451,8 @@ service:
- ec2tagger
- awsentity/service/telegraf
receivers:
- telegraf_statsd
- telegraf_socket_listener
- telegraf_statsd
metrics/hostDeltaMetrics:
exporters:
- awscloudwatch
Expand All @@ -2449,8 +2461,8 @@ service:
- ec2tagger
- awsentity/resource
receivers:
- telegraf_net
- telegraf_diskio
- telegraf_net
metrics/opentelemetry:
exporters:
- otlphttp/metrics
Expand All @@ -2474,6 +2486,7 @@ service:
- forward/opentelemetry
processors:
- transform/otlp_scope
- transform/set_cluster_name
receivers:
- otlp/grpc_127_0_0_1_4327
- otlp/http_127_0_0_1_4328
Expand Down
2 changes: 1 addition & 1 deletion translator/tocwconfig/sampleConfig/opentelemetry/container_insights_config.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,9 @@
"region": "us-east-1"
},
"opentelemetry": {
"cluster_name": "TestCluster",
"collect": {
"container_insights": {
"cluster_name": "TestCluster",
"collection_interval": 30
}
}
Expand Down
2 changes: 1 addition & 1 deletion translator/tocwconfig/sampleConfig/opentelemetry/host_metrics_aks_config.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
"opentelemetry": {
"collect": {
"host_metrics": {
"metrics_collection_interval": 10
"collection_interval": 10
}
}
}
Expand Down
2 changes: 1 addition & 1 deletion translator/tocwconfig/sampleConfig/opentelemetry/host_metrics_azurevm_config.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
"opentelemetry": {
"collect": {
"host_metrics": {
"metrics_collection_interval": 10
"collection_interval": 10
}
}
}
Expand Down
2 changes: 1 addition & 1 deletion ...slator/tocwconfig/sampleConfig/opentelemetry/host_metrics_azurevm_sharedcreds_config.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
"opentelemetry": {
"collect": {
"host_metrics": {
"metrics_collection_interval": 10
"collection_interval": 10
}
}
}
Expand Down
2 changes: 1 addition & 1 deletion translator/tocwconfig/sampleConfig/opentelemetry/host_metrics_config.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
"opentelemetry": {
"collect": {
"host_metrics": {
"metrics_collection_interval": 10
"collection_interval": 10
}
}
}
Expand Down
1 change: 1 addition & 0 deletions translator/tocwconfig/sampleConfig/opentelemetry/prometheus_otel_pipeline_config.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
"region": "us-east-1"
},
"opentelemetry": {
"cluster_name": "TestCluster",
"collect": {
"prometheus": {
"config_path": "./sampleConfig/opentelemetry/prometheus_otel_scrape_config.yaml"
Expand Down
11 changes: 11 additions & 0 deletions translator/tocwconfig/sampleConfig/opentelemetry/prometheus_otel_pipeline_config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -370,6 +370,16 @@ processors:
- set(scope.attributes["cloudwatch.source"], "cloudwatch-agent")
- set(scope.attributes["cloudwatch.solution"], "otel-prometheus")
trace_statements: []
transform/set_cluster_name:
error_mode: ignore
flatten_data: false
log_statements: []
metric_statements:
- context: resource
error_mode: ignore
statements:
- set(resource.attributes["k8s.cluster.name"], "TestCluster")
trace_statements: []
receivers:
prometheus/opentelemetry:
config:
Expand Down Expand Up @@ -425,6 +435,7 @@ service:
- forward/opentelemetry
processors:
- transform/prometheus_scope
- transform/set_cluster_name
receivers:
- prometheus/opentelemetry
telemetry:
Expand Down
1 change: 1 addition & 0 deletions translator/tocwconfig/sampleConfig/opentelemetry/prometheus_otel_pipeline_eks_config.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
"region": "us-east-1"
},
"opentelemetry": {
"cluster_name": "TestCluster",
"collect": {
"prometheus": {
"config_path": "./sampleConfig/opentelemetry/prometheus_otel_scrape_config.yaml"
Expand Down
11 changes: 11 additions & 0 deletions translator/tocwconfig/sampleConfig/opentelemetry/prometheus_otel_pipeline_eks_config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -532,6 +532,16 @@ processors:
- set(scope.attributes["cloudwatch.source"], "cloudwatch-agent")
- set(scope.attributes["cloudwatch.solution"], "otel-prometheus")
trace_statements: []
transform/set_cluster_name:
error_mode: ignore
flatten_data: false
log_statements: []
metric_statements:
- context: resource
error_mode: ignore
statements:
- set(resource.attributes["k8s.cluster.name"], "TestCluster")
trace_statements: []
receivers:
prometheus/opentelemetry:
config:
Expand Down Expand Up @@ -589,6 +599,7 @@ service:
- forward/opentelemetry
processors:
- transform/prometheus_scope
- transform/set_cluster_name
receivers:
- prometheus/opentelemetry
telemetry:
Expand Down
1 change: 1 addition & 0 deletions translator/tocwconfig/tocwconfig_unix_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -177,6 +177,7 @@ func TestCombinedV1V2EKSConfig(t *testing.T) {
yamlConfig, err := mapstructure.Marshal(cfg)
require.NoError(t, err)
yamlStr := toyamlconfig.ToYamlConfig(yamlConfig)
// assert.NoError(t, os.WriteFile("./sampleConfig/opentelemetry/combined_v1_v2_eks_config.yaml", []byte(yamlStr), 0644)) // useful for regenerating YAML
require.NoError(t, yaml.Unmarshal([]byte(yamlStr), &actual))

opt := cmpopts.SortSlices(func(x, y interface{}) bool {
Expand Down
Loading
Loading