loss=0 and grad_norm=nan problem

[Wuxiaomann]

Hi, I was trying to reproduce M3D with pretraining mm_projector. However, when using bf16, I encountered an issue that at random stage of training, loss suddenly become 0 and grad_nrom=nan. I tried massive possible solutions from Internet and GPT, including overlap_comm=False and etc.. Thanks a lot if you have encountered this problem and share your solution!!

[Magic-Xin]

Hi, thank you for reaching out.

Unfortunately, we're not sure what’s causing this issue either. We're currently re-running training on our project this week, but so far we haven’t been able to reproduce the problem you mentioned.

We'll keep an eye on this during our experiments, and let you know if we find anything helpful.