[weather_data_assimilation] cannot replicate training results

[citronella3alain]

tl;dr: weather data assimilation inference works with provided weights, but training yields bad results

While running test_model.py with the provided weights yields expected results, I can't seem to get training (train_model.py) to work well be it atop the provided weights or from scratch; are there any parameters to update, e.g., learning rate, seed, etc., in order to get it to work?

This is the reconstruction gif that I'm getting:

timestep 0:

timestep 160:

Here's the loss plot I'm getting:

I think it's interesting that the loss plot looks "normal" and that the loss starts out so high for the existing model weights.

Let me know what other info I can provide / what else I should try. Thanks!

[danperazzo]

Hello!!

Do you have any updates on this? I was also trying to replicate the results of the paper and also had the same problem.

[glitch401]

@as2c has there been any resolution to this issue?