Skip to content

nvidia_smi input reports 'Unknown schema version "v13"' #2146

Description

@sleepdeprecation

Describe the bug
As part of our Amazon Linux 2 to Amazon Linux 2023 upgrade process, we're attempting to replace a python script we previously copied from an AWS blog post about monitoring GPUs to using the built-in Nvidia GPU monitoring in cloudwatch agent.

On our older AL2 instances, this worked as expected, and we started seeing nvidia_smi_ metrics under CWAgent. On our new AL2023 instances the nvidia metrics are not being reported, and the following was found in our cloudwatch agent log

W! [inputs.nvidia_smi] Unknown schema version "v13", using latest know schema for parsing.
                Please report this as an issue to https://github.com/influxdata/telegraf together
                with a sample output of 'nvidia_smi -q -x'!

Steps to reproduce
This is an abbreviated version, our pipeline creates a base image with some other config changes and then we create a gpu-specific base image on top of that

  • Launch a g5 EC2 instance with a Amazon Linux 2023 Minimal image
  • Install the CUDA driver following the instructions on https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/public-nvidia-driver.html
    # enable nvidia driver repo
    dnf install nvidia-release -y
    
    # download nvidia driver and install
    dnf install gcc kernel-devel kernel-headers -y
    dnf clean all
    
    # unclear if we need to do these all as separate steps or if we can do them all in one,
    # the AWS instructions split them all out into separate calls
    dnf install nvidia-driver-cuda -y
    dnf install cuda-toolkit -y
    dnf install nvidia-gds -y
    dnf install nvidia-fabric-manager -y
    
    systemctl enable --now nvidia-fabricmanager
    systemctl enable --now nvidia-persistenced
    
  • Enable cloudwatch agent with an nvidia section

What did you expect to see?
nvidia_smi_ metrics being added to cloudwatch for the instance

What did you see instead?
No metrics being sent, and an error message in the log

[inputs.nvidia_smi] Unknown schema version "v13", using latest know schema for parsing.
                Please report this as an issue to https://github.com/influxdata/telegraf together
                with a sample output of 'nvidia_smi -q -x'!

What version did you use?

amazon-cloudwatch-agent-ctl -a status
{
  "status": "running",
  "starttime": "2026-06-04T21:06:09+00:00",
  "configstatus": "configured",
  "version": "1.300064.2"
}

What config did you use?

{
  "agent": {
    "metrics_collection_interval": 300,
    "run_as_user": "root"
  },
  "metrics": {
    "aggregation_dimensions": [
      ["AutoScalingGroupName"],
      ["AutoScalingGroupName", "InstanceType"]
    ],
    "append_dimensions": {
      "AutoScalingGroupName": "${aws:AutoScalingGroupName}"
    },
    "metrics_collected": {
      "nvidia_gpu":{
        "measurement": [
          {"name": "utilization_gpu", "unit": "Percent"},
          {"name": "temperature_gpu"},
          {"name": "power_draw"},
          {"name": "utilization_memory", "unit": "Percent"}
        ],
        "drop_original_metrics": [
          "utilization_gpu", "temperature_gpu", "power_draw", "utilization_memory"
        ]
      }
    }
  }
}

Environment
OS: Amazon Linux 2023
system-release-2023.11.20260514-0.amzn2023.noarch

Additional context
This seems to work fine on AL2, likely because the drivers were using an older version.

It appears that telegraf added support for this version of the nvidia_smi output a couple weeks ago with influxdata/telegraf#18690

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions