Exploding loss in training

[Mayamayb]

Hello,
Thanks for sharing your code.
I cloned the repo and began training on the Davis 480p dataset with the default configs from the repo.
After ~93K iterations, during the 3rd epoch, the loss suddenly began to go up very quickly and exploded, failing the training.
Has this happened to you? can you please share the training details?
Best regards.

[geekOctopus]

I meet the same problem today, maybe you can try to set the learning rate to 0.00001