Describe the bug
As part of our Amazon Linux 2 to Amazon Linux 2023 upgrade process, we're attempting to replace a python script we previously copied from an AWS blog post about monitoring GPUs to using the built-in Nvidia GPU monitoring in cloudwatch agent.
On our older AL2 instances, this worked as expected, and we started seeing nvidia_smi_ metrics under CWAgent. On our new AL2023 instances the nvidia metrics are not being reported, and the following was found in our cloudwatch agent log
W! [inputs.nvidia_smi] Unknown schema version "v13", using latest know schema for parsing.
Please report this as an issue to https://github.com/influxdata/telegraf together
with a sample output of 'nvidia_smi -q -x'!
Steps to reproduce
This is an abbreviated version, our pipeline creates a base image with some other config changes and then we create a gpu-specific base image on top of that
- Launch a g5 EC2 instance with a Amazon Linux 2023 Minimal image
- Install the CUDA driver following the instructions on https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/public-nvidia-driver.html
# enable nvidia driver repo
dnf install nvidia-release -y
# download nvidia driver and install
dnf install gcc kernel-devel kernel-headers -y
dnf clean all
# unclear if we need to do these all as separate steps or if we can do them all in one,
# the AWS instructions split them all out into separate calls
dnf install nvidia-driver-cuda -y
dnf install cuda-toolkit -y
dnf install nvidia-gds -y
dnf install nvidia-fabric-manager -y
systemctl enable --now nvidia-fabricmanager
systemctl enable --now nvidia-persistenced
- Enable cloudwatch agent with an nvidia section
What did you expect to see?
nvidia_smi_ metrics being added to cloudwatch for the instance
What did you see instead?
No metrics being sent, and an error message in the log
[inputs.nvidia_smi] Unknown schema version "v13", using latest know schema for parsing.
Please report this as an issue to https://github.com/influxdata/telegraf together
with a sample output of 'nvidia_smi -q -x'!
What version did you use?
amazon-cloudwatch-agent-ctl -a status
{
"status": "running",
"starttime": "2026-06-04T21:06:09+00:00",
"configstatus": "configured",
"version": "1.300064.2"
}
What config did you use?
{
"agent": {
"metrics_collection_interval": 300,
"run_as_user": "root"
},
"metrics": {
"aggregation_dimensions": [
["AutoScalingGroupName"],
["AutoScalingGroupName", "InstanceType"]
],
"append_dimensions": {
"AutoScalingGroupName": "${aws:AutoScalingGroupName}"
},
"metrics_collected": {
"nvidia_gpu":{
"measurement": [
{"name": "utilization_gpu", "unit": "Percent"},
{"name": "temperature_gpu"},
{"name": "power_draw"},
{"name": "utilization_memory", "unit": "Percent"}
],
"drop_original_metrics": [
"utilization_gpu", "temperature_gpu", "power_draw", "utilization_memory"
]
}
}
}
}
Environment
OS: Amazon Linux 2023
system-release-2023.11.20260514-0.amzn2023.noarch
Additional context
This seems to work fine on AL2, likely because the drivers were using an older version.
It appears that telegraf added support for this version of the nvidia_smi output a couple weeks ago with influxdata/telegraf#18690
Describe the bug
As part of our Amazon Linux 2 to Amazon Linux 2023 upgrade process, we're attempting to replace a python script we previously copied from an AWS blog post about monitoring GPUs to using the built-in Nvidia GPU monitoring in cloudwatch agent.
On our older AL2 instances, this worked as expected, and we started seeing
nvidia_smi_metrics underCWAgent. On our new AL2023 instances the nvidia metrics are not being reported, and the following was found in our cloudwatch agent logSteps to reproduce
This is an abbreviated version, our pipeline creates a base image with some other config changes and then we create a gpu-specific base image on top of that
What did you expect to see?
nvidia_smi_metrics being added to cloudwatch for the instanceWhat did you see instead?
No metrics being sent, and an error message in the log
What version did you use?
What config did you use?
Environment
OS: Amazon Linux 2023
system-release-2023.11.20260514-0.amzn2023.noarch
Additional context
This seems to work fine on AL2, likely because the drivers were using an older version.
It appears that telegraf added support for this version of the nvidia_smi output a couple weeks ago with influxdata/telegraf#18690