Skip to content

Add gpu metrics to system_metrics.csv #226

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
rcvalerio merged 1 commit into from
Nov 13, 2025
Merged

Add gpu metrics to system_metrics.csv #226

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 36 additions & 1 deletion task-runner/task_runner/system_monitor.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,23 +5,58 @@
from typing import List, Literal, Optional, Tuple
from uuid import UUID

import GPUtil

@bgregoinductiva bgregoinductiva Nov 13, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This project is not maintained: https://pypi.org/project/GPUtil

Do you know if nvidia-smi is stable?

@rcvalerio rcvalerio Nov 13, 2025

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From my testing GPUtil works great and we were already using it in the task-runner. Should we switch for another library?

I'm not sure, but wouldn't nvidia-smi only work on nvidia GPUs? We might need to support other types of GPU's (e.g. local machines)

@bgregoinductiva bgregoinductiva Nov 13, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please check this comment: https://github.com/inductiva/tasks/issues/1141#issuecomment-2969830897

Unfortunately, the current library and the others I mentioned in that comment are just wrappers of nvidia-smi. It might be enough for now, but if we want to run on different GPUs we may need to research an alternative.

import psutil

from task_runner import BaseEventLogger, events, utils


def _get_gpu_utilization_percent() -> Optional[float]:
Comment thread
HenriquePreto marked this conversation as resolved.
"""Average GPU utilization across GPUs, or None if no GPU."""
try:
gpus = GPUtil.getGPUs()
except Exception: # noqa: BLE001
return None

if not gpus:
return None

# GPUtil.gpu.load is in [0,1]
Comment thread
HenriquePreto marked this conversation as resolved.
avg_load = sum(gpu.load for gpu in gpus) / len(gpus)
return avg_load * 100.0


def _get_gpu_memory_utilization_percent() -> Optional[float]:
"""Average GPU memory utilization across GPUs, or None if no GPU."""
try:
gpus = GPUtil.getGPUs()
except Exception: # noqa: BLE001
return None

if not gpus:
return None

# GPUtil.gpu.memoryUtil is in [0,1]
avg_mem_util = sum(gpu.memoryUtil for gpu in gpus) / len(gpus)
return avg_mem_util * 100.0


class SystemMetrics(enum.Enum):
CPU_USAGE = "cpu-usage"
MEMORY_USAGE = "memory"
DISK_INPUT = "disk-input"
DISK_OUTPUT = "disk-output"
GPU_USAGE = "gpu-usage"
GPU_MEMORY = "gpu-memory"


SYSTEM_METRICS_TO_FUNC = {
SystemMetrics.CPU_USAGE: psutil.cpu_percent,
SystemMetrics.MEMORY_USAGE: lambda: psutil.virtual_memory().percent,
SystemMetrics.DISK_INPUT: lambda: psutil.disk_io_counters().read_bytes,
SystemMetrics.DISK_OUTPUT: lambda: psutil.disk_io_counters().write_bytes
SystemMetrics.DISK_OUTPUT: lambda: psutil.disk_io_counters().write_bytes,
SystemMetrics.GPU_USAGE: _get_gpu_utilization_percent,
SystemMetrics.GPU_MEMORY: _get_gpu_memory_utilization_percent,
}


Expand Down