Numerical instability in UM with raised timestep?

[mo-willkay]

Hello,
I'm not quite sure if this is the best place to ask this but I had a question regarding a crash I had in the UM for a coupled climate model run.

I'm running GC5-central at N96O1 under historical forcings and with a timestep raised from 20 to 30 minutes. After 60 successful simulated years (and several other runs that had no issues) the model failed in the UM with a 'North/South halos too small for advection' message.

I believe this error is caused by excessive meridional winds breaking the advection scheme, and as suggested by the advice I reran the task printing max wind speeds out until it crashed again. In the final timestep before the crash the maximum horizontal winds jumped from 186 to 249 (or almost 900km/h if this is in m/s!) with this value being at level 81 (or about 60km). Unlike other failures seen with this setup, raising extended_halo_size_ns from 5 to 6 wasn't enough to prevent the crash.

So my actual question is whether a jump like this in one timestep is reasonable and should be tolerated or whether this might look more like numerical instability (possibly due to the increased timestep).

Thanks a lot,
William Kay

[MichaelWhitall]

Hi William,

I agree that sounds like a numerical instability. Whether this is caused by the increased timestep length or just a very rare pre-existing cause of instability is hard to say. Another thing you might want to check is where in the world the silly extreme wind-speed occurs. If its above a mountain-range, that points to one of various known instability mechanisms associated with resolved slopes. If its over the N or S pole, again that can be a useful clue.

Re what you can do about it, another key question is how many timesteps after the latest restart dump does it fail? If its a decent amount of time after a restart dump, you can use a nifty trick of changing one of the model parameters by only a tiny amount (such that its still scientifically the same configuration really) and resubmitting it. For these very rare model-failures, a little dose of butterfly-effect is usually enough to get past the misfortune ;) (a common target for getting past failures is to very-slightly change the namelist parameter called "puns" in the boundary-layer namelist; this is a numerics rather than physics constant so shouldn't affect the "science" of the run).

Cheers!
Mike

[mo-willkay]

Thanks Mike!
the silly winds are above 71.3W, 55S which is around the far southern tip of South America which does have some orography, but (looking at the gradients) is more on the level of Norway than a major mountain range, although it's not too far from the Andes.

The crash was 8 simulation months after the previous restart dump so restarting the model with reconfiguration from that dump did succeed in avoiding the crash a second time!

Thanks a lot for the help,
William