Is this repo still available?

[jiekechoo]

Where can I get the k8s exporter and the grafana dashboard?

[eero-t]

Instructions and dockerfiles for building the exporter image: https://github.com/intel/xpumanager/tree/master/deployment

Dashboard [1]: https://github.com/intel/xpumanager/tree/master/deployment/kubernetes/monitoring

[1] Some notes on it: https://github.com/intel/xpumanager/blob/master/rest/Grafana_dashboard_setup_guide_for_XPUM.md

[jiekechoo]

@eero-t Thanks for reply.

I'm using Arc B580 dGPUs, I got the snapshot from Grafana Dashboard like below, there are a few metrics can't display:

The data from metrics endpoint, what should I do next?

# HELP xpum_frequency_mhz Avg (GPU) frequency (in MHz), per GPU tile
# TYPE xpum_frequency_mhz gauge
# HELP xpum_temperature_celsius Avg GPU temperature (in Celsius degree), per tile
# TYPE xpum_temperature_celsius gauge
# HELP xpum_max_temperature_celsius Max GPU temperature (in Celsius degree), per tile
# TYPE xpum_max_temperature_celsius gauge
# HELP xpum_memory_used_bytes Used GPU memory (in bytes), per GPU tile
# TYPE xpum_memory_used_bytes gauge
# HELP xpum_memory_ratio Used GPU memory / Total used GPU memory (in %), per GPU tile
# TYPE xpum_memory_ratio gauge
# HELP xpum_memory_bandwidth_ratio Avg memory throughput / max memory bandwidth (in %), per GPU tile
# TYPE xpum_memory_bandwidth_ratio gauge
# HELP xpum_memory_read_bytes_total Total memory read bytes (in bytes), per GPU tile
# TYPE xpum_memory_read_bytes_total counter
# HELP xpum_memory_read_bytes_created Total memory read bytes (in bytes), per GPU tile
# TYPE xpum_memory_read_bytes_created gauge
# HELP xpum_memory_write_bytes_total Total memory write bytes (in bytes), per GPU tile
# TYPE xpum_memory_write_bytes_total counter
# HELP xpum_memory_write_bytes_created Total memory write bytes (in bytes), per GPU tile
# TYPE xpum_memory_write_bytes_created gauge
# HELP xpum_eu_active_ratio GPU EU Array Active (in %), the normalized sum of all cycles on all EUs that were spent actively executing instructions. Per tile.
# TYPE xpum_eu_active_ratio gauge
# HELP xpum_eu_stall_ratio GPU EU Array Stall (in %), the normalized sum of all cycles on all EUs during which the EUs were stalled. Per tile. At least one thread is loaded, but the EU is stalled. Per tile.
# TYPE xpum_eu_stall_ratio gauge
# HELP xpum_eu_idle_ratio GPU EU Array Idle (in %), the normalized sum of all cycles on all cores when no threads were scheduled on a core. Per tile.
# TYPE xpum_eu_idle_ratio gauge
# HELP xpum_pcie_read_bytes_total Total PCIe read bytes (in bytes), per GPU
# TYPE xpum_pcie_read_bytes_total counter
# HELP xpum_pcie_read_bytes_created Total PCIe read bytes (in bytes), per GPU
# TYPE xpum_pcie_read_bytes_created gauge
# HELP xpum_pcie_write_bytes_total Total PCIe write bytes (in bytes), per GPU
# TYPE xpum_pcie_write_bytes_total counter
# HELP xpum_pcie_write_bytes_created Total PCIe write bytes (in bytes), per GPU
# TYPE xpum_pcie_write_bytes_created gauge
# HELP xpum_topology_link Connection type of two GPU tiles
# TYPE xpum_topology_link gauge
# HELP xpum_xelink_port_status The Xelink port status
# TYPE xpum_xelink_port_status gauge
# HELP xpum_engine_group_ratio Avg utilization of engine group (in %), per GPU tile
# TYPE xpum_engine_group_ratio gauge
# HELP xpum_per_engine_ratio Per-engine utilization (in %)
# TYPE xpum_per_engine_ratio gauge
xpum_engine_group_ratio{dev_file="card1",dev_name="Intel(R) Graphics",pci_bdf="0000:00:02.0",pci_dev="0x7d67",src="direct",type="compute",uuid="00000000-0000-0200-0000-00067d678086",vendor="Intel(R) Corporation"} 0.0
xpum_engine_group_ratio{dev_file="card1",dev_name="Intel(R) Graphics",pci_bdf="0000:00:02.0",pci_dev="0x7d67",src="direct",type="copy",uuid="00000000-0000-0200-0000-00067d678086",vendor="Intel(R) Corporation"} 0.0
xpum_engine_group_ratio{dev_file="card1",dev_name="Intel(R) Graphics",pci_bdf="0000:00:02.0",pci_dev="0x7d67",src="direct",type="render",uuid="00000000-0000-0200-0000-00067d678086",vendor="Intel(R) Corporation"} 0.0
xpum_eu_active_ratio{dev_file="card0",dev_name="Intel(R) Arc(TM) B580 Graphics",pci_bdf="0000:04:00.0",pci_dev="0xe20b",src="direct",uuid="00000000-0000-0004-0000-0000e20b8086",vendor="Intel(R) Corporation"} 0.0
xpum_eu_active_ratio{dev_file="card1",dev_name="Intel(R) Graphics",pci_bdf="0000:00:02.0",pci_dev="0x7d67",src="direct",uuid="00000000-0000-0200-0000-00067d678086",vendor="Intel(R) Corporation"} 0.0
xpum_eu_active_ratio{dev_file="card2",dev_name="Intel(R) Arc(TM) B580 Graphics",pci_bdf="0000:83:00.0",pci_dev="0xe20b",src="direct",uuid="00000000-0000-0083-0000-0000e20b8086",vendor="Intel(R) Corporation"} 0.0
xpum_eu_idle_ratio{dev_file="card0",dev_name="Intel(R) Arc(TM) B580 Graphics",pci_bdf="0000:04:00.0",pci_dev="0xe20b",src="direct",uuid="00000000-0000-0004-0000-0000e20b8086",vendor="Intel(R) Corporation"} 1.0
xpum_eu_idle_ratio{dev_file="card1",dev_name="Intel(R) Graphics",pci_bdf="0000:00:02.0",pci_dev="0x7d67",src="direct",uuid="00000000-0000-0200-0000-00067d678086",vendor="Intel(R) Corporation"} 1.0
xpum_eu_idle_ratio{dev_file="card2",dev_name="Intel(R) Arc(TM) B580 Graphics",pci_bdf="0000:83:00.0",pci_dev="0xe20b",src="direct",uuid="00000000-0000-0083-0000-0000e20b8086",vendor="Intel(R) Corporation"} 1.0
xpum_eu_stall_ratio{dev_file="card0",dev_name="Intel(R) Arc(TM) B580 Graphics",pci_bdf="0000:04:00.0",pci_dev="0xe20b",src="direct",uuid="00000000-0000-0004-0000-0000e20b8086",vendor="Intel(R) Corporation"} 0.0
xpum_eu_stall_ratio{dev_file="card1",dev_name="Intel(R) Graphics",pci_bdf="0000:00:02.0",pci_dev="0x7d67",src="direct",uuid="00000000-0000-0200-0000-00067d678086",vendor="Intel(R) Corporation"} 0.0
xpum_eu_stall_ratio{dev_file="card2",dev_name="Intel(R) Arc(TM) B580 Graphics",pci_bdf="0000:83:00.0",pci_dev="0xe20b",src="direct",uuid="00000000-0000-0083-0000-0000e20b8086",vendor="Intel(R) Corporation"} 0.0
xpum_frequency_mhz{dev_file="card0",dev_name="Intel(R) Arc(TM) B580 Graphics",location="gpu",pci_bdf="0000:04:00.0",pci_dev="0xe20b",src="direct",type="actual",uuid="00000000-0000-0004-0000-0000e20b8086",vendor="Intel(R) Corporation"} 754.0
xpum_frequency_mhz{dev_file="card1",dev_name="Intel(R) Graphics",location="gpu",pci_bdf="0000:00:02.0",pci_dev="0x7d67",src="direct",type="actual",uuid="00000000-0000-0200-0000-00067d678086",vendor="Intel(R) Corporation"} 550.0
xpum_frequency_mhz{dev_file="card2",dev_name="Intel(R) Arc(TM) B580 Graphics",location="gpu",pci_bdf="0000:83:00.0",pci_dev="0xe20b",src="direct",type="actual",uuid="00000000-0000-0083-0000-0000e20b8086",vendor="Intel(R) Corporation"} 1200.0
xpum_max_temperature_celsius{dev_file="card0",dev_name="Intel(R) Arc(TM) B580 Graphics",location="gpu",pci_bdf="0000:04:00.0",pci_dev="0xe20b",src="direct",uuid="00000000-0000-0004-0000-0000e20b8086",vendor="Intel(R) Corporation"} 55.0
xpum_max_temperature_celsius{dev_file="card0",dev_name="Intel(R) Arc(TM) B580 Graphics",location="mem",pci_bdf="0000:04:00.0",pci_dev="0xe20b",src="direct",uuid="00000000-0000-0004-0000-0000e20b8086",vendor="Intel(R) Corporation"} 58.0
xpum_max_temperature_celsius{dev_file="card2",dev_name="Intel(R) Arc(TM) B580 Graphics",location="gpu",pci_bdf="0000:83:00.0",pci_dev="0xe20b",src="direct",uuid="00000000-0000-0083-0000-0000e20b8086",vendor="Intel(R) Corporation"} 55.0
xpum_max_temperature_celsius{dev_file="card2",dev_name="Intel(R) Arc(TM) B580 Graphics",location="mem",pci_bdf="0000:83:00.0",pci_dev="0xe20b",src="direct",uuid="00000000-0000-0083-0000-0000e20b8086",vendor="Intel(R) Corporation"} 58.0
xpum_memory_bandwidth_ratio{dev_file="card0",dev_name="Intel(R) Arc(TM) B580 Graphics",pci_bdf="0000:04:00.0",pci_dev="0xe20b",src="direct",uuid="00000000-0000-0004-0000-0000e20b8086",vendor="Intel(R) Corporation"} 0.0
xpum_memory_ratio{dev_file="card0",dev_name="Intel(R) Arc(TM) B580 Graphics",pci_bdf="0000:04:00.0",pci_dev="0xe20b",src="direct",uuid="00000000-0000-0004-0000-0000e20b8086",vendor="Intel(R) Corporation"} 0.9984999999999999
xpum_memory_ratio{dev_file="card2",dev_name="Intel(R) Arc(TM) B580 Graphics",pci_bdf="0000:83:00.0",pci_dev="0xe20b",src="direct",uuid="00000000-0000-0083-0000-0000e20b8086",vendor="Intel(R) Corporation"} 0.9665
xpum_memory_read_bytes_created{dev_file="card0",dev_name="Intel(R) Arc(TM) B580 Graphics",pci_bdf="0000:04:00.0",pci_dev="0xe20b",src="direct",uuid="00000000-0000-0004-0000-0000e20b8086",vendor="Intel(R) Corporation"} 1.771084538173083e+09
xpum_memory_read_bytes_total{dev_file="card0",dev_name="Intel(R) Arc(TM) B580 Graphics",pci_bdf="0000:04:00.0",pci_dev="0xe20b",src="direct",uuid="00000000-0000-0004-0000-0000e20b8086",vendor="Intel(R) Corporation"} 4.85892817e+08
xpum_memory_used_bytes{dev_file="card0",dev_name="Intel(R) Arc(TM) B580 Graphics",pci_bdf="0000:04:00.0",pci_dev="0xe20b",src="direct",uuid="00000000-0000-0004-0000-0000e20b8086",vendor="Intel(R) Corporation"} 1.2790557352e+010
xpum_memory_used_bytes{dev_file="card2",dev_name="Intel(R) Arc(TM) B580 Graphics",pci_bdf="0000:83:00.0",pci_dev="0xe20b",src="direct",uuid="00000000-0000-0083-0000-0000e20b8086",vendor="Intel(R) Corporation"} 1.2381134848e+010
xpum_memory_write_bytes_created{dev_file="card0",dev_name="Intel(R) Arc(TM) B580 Graphics",pci_bdf="0000:04:00.0",pci_dev="0xe20b",src="direct",uuid="00000000-0000-0004-0000-0000e20b8086",vendor="Intel(R) Corporation"} 1.7710845381731172e+09
xpum_memory_write_bytes_total{dev_file="card0",dev_name="Intel(R) Arc(TM) B580 Graphics",pci_bdf="0000:04:00.0",pci_dev="0xe20b",src="direct",uuid="00000000-0000-0004-0000-0000e20b8086",vendor="Intel(R) Corporation"} 1.3715666e+07
xpum_pcie_read_bytes_created{dev_file="card0",dev_name="Intel(R) Arc(TM) B580 Graphics",pci_bdf="0000:04:00.0",pci_dev="0xe20b",src="direct",uuid="00000000-0000-0004-0000-0000e20b8086",vendor="Intel(R) Corporation"} 1.770452409961258e+09
xpum_pcie_read_bytes_created{dev_file="card1",dev_name="Intel(R) Graphics",pci_bdf="0000:00:02.0",pci_dev="0x7d67",src="direct",uuid="00000000-0000-0200-0000-00067d678086",vendor="Intel(R) Corporation"} 1.7704524099628284e+09
xpum_pcie_read_bytes_created{dev_file="card2",dev_name="Intel(R) Arc(TM) B580 Graphics",pci_bdf="0000:83:00.0",pci_dev="0xe20b",src="direct",uuid="00000000-0000-0083-0000-0000e20b8086",vendor="Intel(R) Corporation"} 1.7704524099621103e+09
xpum_pcie_read_bytes_total{dev_file="card0",dev_name="Intel(R) Arc(TM) B580 Graphics",pci_bdf="0000:04:00.0",pci_dev="0xe20b",src="direct",uuid="00000000-0000-0004-0000-0000e20b8086",vendor="Intel(R) Corporation"} 5.308777979844e+012
xpum_pcie_read_bytes_total{dev_file="card1",dev_name="Intel(R) Graphics",pci_bdf="0000:00:02.0",pci_dev="0x7d67",src="direct",uuid="00000000-0000-0200-0000-00067d678086",vendor="Intel(R) Corporation"} 0.0
xpum_pcie_read_bytes_total{dev_file="card2",dev_name="Intel(R) Arc(TM) B580 Graphics",pci_bdf="0000:83:00.0",pci_dev="0xe20b",src="direct",uuid="00000000-0000-0083-0000-0000e20b8086",vendor="Intel(R) Corporation"} 1.613591264364e+012
xpum_pcie_write_bytes_created{dev_file="card0",dev_name="Intel(R) Arc(TM) B580 Graphics",pci_bdf="0000:04:00.0",pci_dev="0xe20b",src="direct",uuid="00000000-0000-0004-0000-0000e20b8086",vendor="Intel(R) Corporation"} 1.7704524099612784e+09
xpum_pcie_write_bytes_created{dev_file="card1",dev_name="Intel(R) Graphics",pci_bdf="0000:00:02.0",pci_dev="0x7d67",src="direct",uuid="00000000-0000-0200-0000-00067d678086",vendor="Intel(R) Corporation"} 1.7704524099628532e+09
xpum_pcie_write_bytes_created{dev_file="card2",dev_name="Intel(R) Arc(TM) B580 Graphics",pci_bdf="0000:83:00.0",pci_dev="0xe20b",src="direct",uuid="00000000-0000-0083-0000-0000e20b8086",vendor="Intel(R) Corporation"} 1.7704524099621248e+09
xpum_pcie_write_bytes_total{dev_file="card0",dev_name="Intel(R) Arc(TM) B580 Graphics",pci_bdf="0000:04:00.0",pci_dev="0xe20b",src="direct",uuid="00000000-0000-0004-0000-0000e20b8086",vendor="Intel(R) Corporation"} 6.785614068984e+012
xpum_pcie_write_bytes_total{dev_file="card1",dev_name="Intel(R) Graphics",pci_bdf="0000:00:02.0",pci_dev="0x7d67",src="direct",uuid="00000000-0000-0200-0000-00067d678086",vendor="Intel(R) Corporation"} 0.0
xpum_pcie_write_bytes_total{dev_file="card2",dev_name="Intel(R) Arc(TM) B580 Graphics",pci_bdf="0000:83:00.0",pci_dev="0xe20b",src="direct",uuid="00000000-0000-0083-0000-0000e20b8086",vendor="Intel(R) Corporation"} 8.11861893816e+011
xpum_per_engine_ratio{dev_file="card1",dev_name="Intel(R) Graphics",engine_id="0",pci_bdf="0000:00:02.0",pci_dev="0x7d67",src="direct",type="compute",uuid="00000000-0000-0200-0000-00067d678086",vendor="Intel(R) Corporation"} 0.0
xpum_per_engine_ratio{dev_file="card1",dev_name="Intel(R) Graphics",engine_id="0",pci_bdf="0000:00:02.0",pci_dev="0x7d67",src="direct",type="copy",uuid="00000000-0000-0200-0000-00067d678086",vendor="Intel(R) Corporation"} 0.0
xpum_per_engine_ratio{dev_file="card1",dev_name="Intel(R) Graphics",engine_id="0",pci_bdf="0000:00:02.0",pci_dev="0x7d67",src="direct",type="decoder",uuid="00000000-0000-0200-0000-00067d678086",vendor="Intel(R) Corporation"} 0.0
xpum_per_engine_ratio{dev_file="card1",dev_name="Intel(R) Graphics",engine_id="0",pci_bdf="0000:00:02.0",pci_dev="0x7d67",src="direct",type="encoder",uuid="00000000-0000-0200-0000-00067d678086",vendor="Intel(R) Corporation"} 0.0
xpum_per_engine_ratio{dev_file="card1",dev_name="Intel(R) Graphics",engine_id="0",pci_bdf="0000:00:02.0",pci_dev="0x7d67",src="direct",type="media_enhancement",uuid="00000000-0000-0200-0000-00067d678086",vendor="Intel(R) Corporation"} 0.0
xpum_per_engine_ratio{dev_file="card1",dev_name="Intel(R) Graphics",engine_id="0",pci_bdf="0000:00:02.0",pci_dev="0x7d67",src="direct",type="render",uuid="00000000-0000-0200-0000-00067d678086",vendor="Intel(R) Corporation"} 0.0
xpum_per_engine_ratio{dev_file="card1",dev_name="Intel(R) Graphics",engine_id="1",pci_bdf="0000:00:02.0",pci_dev="0x7d67",src="direct",type="decoder",uuid="00000000-0000-0200-0000-00067d678086",vendor="Intel(R) Corporation"} 0.0
xpum_per_engine_ratio{dev_file="card1",dev_name="Intel(R) Graphics",engine_id="1",pci_bdf="0000:00:02.0",pci_dev="0x7d67",src="direct",type="encoder",uuid="00000000-0000-0200-0000-00067d678086",vendor="Intel(R) Corporation"} 0.0
xpum_temperature_celsius{dev_file="card0",dev_name="Intel(R) Arc(TM) B580 Graphics",location="gpu",pci_bdf="0000:04:00.0",pci_dev="0xe20b",src="direct",uuid="00000000-0000-0004-0000-0000e20b8086",vendor="Intel(R) Corporation"} 54.2
xpum_temperature_celsius{dev_file="card0",dev_name="Intel(R) Arc(TM) B580 Graphics",location="mem",pci_bdf="0000:04:00.0",pci_dev="0xe20b",src="direct",uuid="00000000-0000-0004-0000-0000e20b8086",vendor="Intel(R) Corporation"} 58.0
xpum_temperature_celsius{dev_file="card2",dev_name="Intel(R) Arc(TM) B580 Graphics",location="gpu",pci_bdf="0000:83:00.0",pci_dev="0xe20b",src="direct",uuid="00000000-0000-0083-0000-0000e20b8086",vendor="Intel(R) Corporation"} 54.53
xpum_temperature_celsius{dev_file="card2",dev_name="Intel(R) Arc(TM) B580 Graphics",location="mem",pci_bdf="0000:83:00.0",pci_dev="0xe20b",src="direct",uuid="00000000-0000-0083-0000-0000e20b8086",vendor="Intel(R) Corporation"} 58.0
xpum_topology_link{dev_file="card0",dev_name="Intel(R) Arc(TM) B580 Graphics",lane_count="0",local_cpu_affinity="0-23",local_device_id="0",local_numa_index="0",local_on_subdevice="true",local_subdevice_id="0",pci_bdf="0000:04:00.0",pci_dev="0xe20b",remote_device_id="0",remote_subdevice_id="0",src="direct",uuid="00000000-0000-0004-0000-0000e20b8086",vendor="Intel(R) Corporation"} 0.0
xpum_topology_link{dev_file="card0",dev_name="Intel(R) Arc(TM) B580 Graphics",lane_count="0",local_cpu_affinity="0-23",local_device_id="0",local_numa_index="0",local_on_subdevice="true",local_subdevice_id="0",pci_bdf="0000:04:00.0",pci_dev="0xe20b",remote_device_id="1",remote_subdevice_id="0",src="direct",uuid="00000000-0000-0004-0000-0000e20b8086",vendor="Intel(R) Corporation"} 4.0
xpum_topology_link{dev_file="card0",dev_name="Intel(R) Arc(TM) B580 Graphics",lane_count="0",local_cpu_affinity="0-23",local_device_id="0",local_numa_index="0",local_on_subdevice="true",local_subdevice_id="0",pci_bdf="0000:04:00.0",pci_dev="0xe20b",remote_device_id="2",remote_subdevice_id="0",src="direct",uuid="00000000-0000-0004-0000-0000e20b8086",vendor="Intel(R) Corporation"} 4.0
xpum_topology_link{dev_file="card1",dev_name="Intel(R) Graphics",lane_count="0",local_cpu_affinity="0-23",local_device_id="2",local_numa_index="0",local_on_subdevice="true",local_subdevice_id="0",pci_bdf="0000:00:02.0",pci_dev="0x7d67",remote_device_id="0",remote_subdevice_id="0",src="direct",uuid="00000000-0000-0200-0000-00067d678086",vendor="Intel(R) Corporation"} 4.0
xpum_topology_link{dev_file="card1",dev_name="Intel(R) Graphics",lane_count="0",local_cpu_affinity="0-23",local_device_id="2",local_numa_index="0",local_on_subdevice="true",local_subdevice_id="0",pci_bdf="0000:00:02.0",pci_dev="0x7d67",remote_device_id="1",remote_subdevice_id="0",src="direct",uuid="00000000-0000-0200-0000-00067d678086",vendor="Intel(R) Corporation"} 4.0
xpum_topology_link{dev_file="card1",dev_name="Intel(R) Graphics",lane_count="0",local_cpu_affinity="0-23",local_device_id="2",local_numa_index="0",local_on_subdevice="true",local_subdevice_id="0",pci_bdf="0000:00:02.0",pci_dev="0x7d67",remote_device_id="2",remote_subdevice_id="0",src="direct",uuid="00000000-0000-0200-0000-00067d678086",vendor="Intel(R) Corporation"} 0.0
xpum_topology_link{dev_file="card2",dev_name="Intel(R) Arc(TM) B580 Graphics",lane_count="0",local_cpu_affinity="0-23",local_device_id="1",local_numa_index="0",local_on_subdevice="true",local_subdevice_id="0",pci_bdf="0000:83:00.0",pci_dev="0xe20b",remote_device_id="0",remote_subdevice_id="0",src="direct",uuid="00000000-0000-0083-0000-0000e20b8086",vendor="Intel(R) Corporation"} 4.0
xpum_topology_link{dev_file="card2",dev_name="Intel(R) Arc(TM) B580 Graphics",lane_count="0",local_cpu_affinity="0-23",local_device_id="1",local_numa_index="0",local_on_subdevice="true",local_subdevice_id="0",pci_bdf="0000:83:00.0",pci_dev="0xe20b",remote_device_id="1",remote_subdevice_id="0",src="direct",uuid="00000000-0000-0083-0000-0000e20b8086",vendor="Intel(R) Corporation"} 0.0
xpum_topology_link{dev_file="card2",dev_name="Intel(R) Arc(TM) B580 Graphics",lane_count="0",local_cpu_affinity="0-23",local_device_id="1",local_numa_index="0",local_on_subdevice="true",local_subdevice_id="0",pci_bdf="0000:83:00.0",pci_dev="0xe20b",remote_device_id="2",remote_subdevice_id="0",src="direct",uuid="00000000-0000-0083-0000-0000e20b8086",vendor="Intel(R) Corporation"} 4.0
xpum_xelink_port_status{description="Device lacks Xe Link capability.",dev_file="card0",dev_name="Intel(R) Arc(TM) B580 Graphics",device_id="0",pci_bdf="0000:04:00.0",pci_dev="0xe20b",src="direct",uuid="00000000-0000-0004-0000-0000e20b8086",vendor="Intel(R) Corporation"} -1.0
xpum_xelink_port_status{description="Device lacks Xe Link capability.",dev_file="card1",dev_name="Intel(R) Graphics",device_id="2",pci_bdf="0000:00:02.0",pci_dev="0x7d67",src="direct",uuid="00000000-0000-0200-0000-00067d678086",vendor="Intel(R) Corporation"} -1.0
xpum_xelink_port_status{description="Device lacks Xe Link capability.",dev_file="card2",dev_name="Intel(R) Arc(TM) B580 Graphics",device_id="1",pci_bdf="0000:83:00.0",pci_dev="0xe20b",src="direct",uuid="00000000-0000-0083-0000-0000e20b8086",vendor="Intel(R) Corporation"} -1.0

[eero-t]

Which GPU FW, kernel, Xe KMD and oneAPI L0 driver versions you have, i.e. are they new enough to support missing BMG metrics?

Engine utilization comes through kernel perf interface, which can require PERFMON or SYS_ADMIN capability for the XPUM daemon (in addition to running it as user 0).

If power metrics (coming through kernel hwmon interface) is missing after adding those, does sudo dmesg show any relevant kernel messages?

PS. You could grep dmesg output for xe and hwmon. If it does not show anything, try output from journalctl -b 0 instead (if there have been lot of kernel messages since boot, kernel's dmesg buffer might not be enough to retain boot ones).

[jiekechoo]

This is my k8s daemonset yaml intel-gpu-exporter.yaml

not found xe or hwmon for dmesg output in the exporter pod.

The docker image built from https://github.com/intel/xpumanager/blob/master/BUILDING.md and https://github.com/intel/xpumanager/blob/master/deployment/docker/Dockerfile.ubuntu.22.04

Any other suggestion?

[eero-t]

This is my k8s daemonset yaml intel-gpu-exporter.yaml

As it seems to run in privileged mode, access rights is not an issue (privileged mode should not be needed, user 0 and SYS_ADMIN cap should be enough).

not found xe or hwmon for dmesg output in the exporter pod.

Check the kernel version (uname -a) and journalctl -b 0 | grep -e hwmon -e xe output from the host where the pod is running, and report those (as you'll need new enough kernel / Xe kernel module, GPU FW).

However, looking at this:

The docker image built from https://github.com/intel/xpumanager/blob/master/BUILDING.md and https://github.com/intel/xpumanager/blob/master/deployment/docker/Dockerfile.ubuntu.22.04

I noticed that it adds user-space driver repo for LTS packages: https://github.com/intel/xpumanager/blob/master/deployment/docker/Dockerfile.ubuntu.22.04#L61

Which differs a bit from the one recommended for BMG: https://dgpu-docs.intel.com/driver/client/overview.html#ubuntu-22.04

So user space driver inside the container might not be new enough for all BMG features.

Any other suggestion?

=> You could try updating the jammy repo used in Dockerfile to to one mentioned on the driver install page.

If that does not help, you could try updating container to Ubuntu 24.04 + Kobuk PPA. Just replace jammy-* (22.04) distro tags in FROM lines with 24.04, and the driver repo install with the kobuk PPA install: https://dgpu-docs.intel.com/driver/client/overview.html#ubuntu-latest

If even that does not help, please report what driver version dpkg -l libze-intel-gpu1 reports to be installed inside your container?

PS. I'm myself using 24.04 + Kobuk PPA with another program using Sysman API (not xpum), and metrics seems to work fine with that, at least for the BMG version (G21) that I had access to.

[jiekechoo]

OKOKOK

As an Intel dGPU user, if I want to use Intel GPU in k8s, should build all deps images and make own deployment scripts by myself. so...ha...ha...

Does your team still working on this project?

Even I MUST build this manually, I hope the scripts in the repo are available/useable for USERS.

The xpumanager docker images in hub.docker.com still on v1.2.27.

[eero-t]

As an Intel dGPU user, if I want to use Intel GPU in k8s, should build all deps images and make own deployment scripts by myself. so...ha...ha...

For using Intel GPUs in K8s, there are:

• GPU plugin: https://github.com/intel/intel-device-plugins-for-kubernetes/tree/main/cmd/gpu_plugin#readme
• DRA GPU driver: https://github.com/intel/intel-resource-drivers-for-kubernetes/tree/main/doc/gpu#readme

But I assume you mean monitoring GPUs... :-)

Does your team still working on this project?

Even I MUST build this manually, I hope the scripts in the repo are available/useable for USERS.

The xpumanager docker images in hub.docker.com still on v1.2.27.

Yes, sorry about that. XPUM team has changed, and the new team is completing long process of re-architecting & rewriting XPUM from scratch for much improved maintainability, while also maintaining the old version(s). Pre-release of the rewrite is expected in the next month(s). It drops some features (that have no known users), while adding others, including better K8s support (e.g. new images).

(XPUM 1.2.x targets Intel Data Center GPUs / K8s. XPUM 1.3.x targets mainly BMG client GPUs, and updates just the CLI tooling side. I.e. K8s support update unfortunately comes only with the XPUM rewrite version.)

[jiekechoo]

Hi @eero-t , I know you're busy, but I'd love to get a quick update when you have a chance.

[eero-t]

Pre-release of the XPUM daemon / exporter Golang rewrite, as an OpenTelemetry (OTel) collector, should come in next few weeks.

It drops all functionality allowing remote GPU state changes (REST API, FW updates etc)[1], and provides only GPU state information. That data can be configured to be sent to a remote OTel collector, or provided directly from a Prometheus endpoint.

Names of the metrics and their attributes are changed to follow OTel semantic conventions (xpum -> hw prefix etc): https://opentelemetry.io/docs/specs/semconv/hardware/

As that spec is very incomplete and not yet at alpha state for HW metrics, those names may still change. As long as the provided (updated) Grafana dashboard is used, that should not be user visible though.

Installation will be through a Helm chart instead of kustomize, most likely from ghcr.io, not DockerHub.

[1] xpu-smi 1.x, and its later released 2.x rewrite (for maintainability) can be used for the GPU state changes.

[jiekechoo]

Thanks for sharing this.

[eero-t]

First XPUM daemon/exporter pre-release image: https://github.com/intel/xpumanager/pkgs/container/xpumanager%2Fxpumd/796924308?tag=v2.0.0-rc.0

In case somebody wants to participate in pre-release testing, it can be used like this...

Running:

$ mkdir sockdir
$ chmod 777 sockdir
$ docker run -it --rm --security-opt label=disable \
  --env XDG_RUNTIME_DIR=/sockdir --volume $PWD/sockdir:/sockdir \
  --user 0 --cap-drop ALL --cap-add PERFMON --device /dev/dri \
  --publish 8080:8080 ghcr.io/intel/xpumanager/xpumd:v2.0.0-rc.0 \
  --config /etc/xpumd/config-example.yaml

(--security-opt is needed only if host uses SELinux. To get also GPU engine utilization, specify wider SYS_ADMIN capability instead of PERFMON one.)

Testing:
$ curl --no-progress-meter http://localhost:8080/metrics

Note: if one wants just Prometheus exporter, need for sockdir can be avoided by dropping intelxpuinfo exporter [1] from config exporters line: https://github.com/intel/xpumanager/blob/v2.x/xpumd/config-example.yaml#L96

With Kubernetes, one could use Helm chart for install instead: https://github.com/intel/xpumanager/tree/v2.x/xpumd/charts/xpumd#readme

---

[1]: intelxpuinfo exporter (in the container / default config) is (in future) intended to provide GPU health info for Kubernetes components. For now, one can build tester for its local gRPC endpoint by doing make build in: https://github.com/intel/xpumanager/blob/v2.x/xpumd/

And running the resulting binary like this:
$ ./dist/xpuinfo-cli -sock-dir $PWD/sockdir

PS. Doing make build here: https://github.com/intel/xpumanager/tree/v2.x/xpumd/receiver/internal/level-zero-go/examples/

builds command line tool that can dump GPU info (provided by Level-Zero driver) in JSON & YAML formats.

(That output can be used also as configuration for the mock driver under level-zero-go/ Level-Zero Sysman Go bindings.)