Profile training pipeline for GPU utilization and address underutilization

[forklady42]

Overview

Profile the current training pipeline to identify when/if the GPU is inactive and address any significant underutilization.

Tasks

• Profile a training run using PyTorch profiler or similar tooling to capture GPU utilization over time
• Identify bottlenecks (data loading, preprocessing, CPU-GPU transfers, etc.)
• Determine root cause of any significant GPU idle time
• Implement fixes for identified bottlenecks (e.g., prefetching, num_workers tuning, pinned memory, async data loading)
• Measure and document improvement in GPU utilization after changes

Timebox

2 weeks

Acceptance Criteria

• GPU utilization profile captured and documented
• Any significant underutilization (>10–15% idle time) is addressed
• Before/after metrics recorded