Skip to content

Training hangs intermittently on single GPU (MetaX / NVIDIA) #106

Description

@zhaodq23

环境

  • 硬件:MetaX (沐曦) GPU + MUSA CUDA 兼容层 / NVIDIA A100
  • PyTorch:2.8.0+mars3.7.0.7 / torch 2.x
  • 数据集:DL3DV (1个场景,8个视角)

问题描述

训练过程中随机卡住(通常在第3-8步),进程仍在运行(CPU 200-400%),但没有崩溃,也没有GPU内存错误。卡住总是发生在 on_after_backward 之后(optimizer_step/Lightning步骤边界)。

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions