Model parameters become NaN during training

[JamChaos]

I am encountering an issue where model parameters suddenly turn into NaN during training, leading to a complete crash.

I have noticed that the recorded grad explode significantly.

What makes this strange is that I have already implemented the following safeguards, yet the issue persists:

1. Gradient Clipping is active.
2. Loss Handling: Logic is in place to zero out NaN losses.
3. Loss Clipping: Loss values are capped within a specific range.

Despite these, the parameters still become corrupted. I noticed other users have reported similar issues in this repository, suggesting this might be a systemic numerical instability within the architecture or the specific layer implementation or specific scene.

[myc634]

We encountered the same issue during training, simply skip iter which the loss value of NaN will help