Number of CPUs for planar domain LFRic setups #648
Denis Sergeev (dennissergeev)
started this conversation in
LFRic
Replies: 2 comments 10 replies
-
|
Not sure that we have a guide. But you are right to consider the multigrid levels and so that it is the coarsest mesh that is partitioned. The next thing I would do is that if you have multiplied your total number of cells by 4, then I would also increase the cpus by 4 (so try 512 rather than 256). You may also need to increase the xios mpi/nodes. I'm not sure that we have a formula to work out the correct xios mpi? I would just go with the try multiplying that by 4 too. |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
Thanks cjohnson-pi but increasing to 512 didn't help... By the xios mpi, do you mean |
Beta Was this translation helpful? Give feedback.
-
|
256x256 should/does work with 256CPUs - in fact there's a rose-stem job which does exactly this: https://github.com/MetOffice/lfric_apps/blob/main/rose-stem/site/common/lfric_atm/tasks_lfric_atm.cylc#L723 - what is the problem you are seeing? |
Beta Was this translation helpful? Give feedback.
9 replies
-
|
No, the list of diagnostics is the same - so I guess it's just the size of the arrays... How do I switch the server on? |
Beta Was this translation helpful? Give feedback.
-
|
So you'd need to set xios_server_mode=true and specify the number of processors for the server (32 probably okay). You'd also need to increase the number of nodes in the PBS select command by 1 to account for the inclusion of the server. I think there's something else that needs setting up to pass to the launch-exe script which determines how the processors are ordered between model and xios... |
Beta Was this translation helpful? Give feedback.
-
|
Hi iboutle, thanks for your help with this! Sorry to butt in but could I quickly ask a naive question... what exactly is the limit when running without the xios server? Is there a quantitative threshold in terms of the size of the simulation beyond which the use of the xios server is required? |
Beta Was this translation helpful? Give feedback.
-
|
iboutle Unfortunately it's still falling over - now a bit further than the diagnostic output: PET filesI might have done something wrong in the command directives though. Is there an example standalone suite with the XIOS server enabled? |
Beta Was this translation helpful? Give feedback.
-
|
Alex Corbett (@CorboPy) - I've no idea, I'm just guessing here! Denis Sergeev (@dennissergeev) not a simple one - you could look at u-dz462 or the rose-stem suite and try to piece it together from there, unless anyone else can help. |
Beta Was this translation helpful? Give feedback.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
Is there a guide or a formula for how to choose the number of CPUs correctly for an LFRic setup on a planar mesh and the usual four multigrid levels?
For example, I don't understand why a 256x256 point domain doesn't run on 256 CPUs while a 128x128 point setup on 128 CPUs does. This is on Monsoon, so there are 128 cores per node.
Thank you.
Beta Was this translation helpful? Give feedback.
All reactions