Unexpected behaviour with `--jobs` and `--robot`

[nathanhaigh]

I have a Slurm cluster with nodes containing different CPU architectures (e.g. Westmere, Ivy Bridge and Haswell). Each type mounts a different device for the EasyBuild location so software/modules need to be built for each one separately.

I would like to use EasyBuild's --jobs to submit my software building/install to each of these architectures. I have multiple GC3Pie resource sections configured so I could do something like:

eb --robot --job --job-target-resource Haswell foss-2018b.eb

While this sucessfully submits jobs to Slurm, they actually fail due to unresolved dependencies. I traced the issue back to the fact that the dependencies are actually being calculated locally on my head node (which has already had most of the foss-2018b modules installed). I had thought that the dependency checks would also have been run on my compute nodes rather than locally on the head node. Is this intended behaviour? Am I doing something wrong? Is there an alternative?

[boegel]

@nathanhaigh The check is indeed being done locally before submitting the jobs, because using --robot and --job together implies that the installation will be done "in parallel", in the sense that a separate job will be submitted for each dependency. So, it's a feature, not a bug. ;-)

I'm not sure if there's a decent workaround fight now, but if not we can definitely look into one, for example by adding explicit support for not checking for existing modules locally (which would also imply not submitting separate jobs but passing the --robot down into the job).

[nathanhaigh]

@boegel Perhaps there is a way, if it sisn't already, to decouple the dependency graph creation from the checks as to whether each dependency is satisfied. This way an additioanl flag could be used to skip the latter such that a build job is created for all dependencies. Thus if the software is already built/available on the target system, the job would finish quickly stating nothing needs to be done.

Wouldn't this approach still allow the jobs to be dependant on each other and be executed in parallel?

[vanzod]

We have that exact same problem. Our approach has been to build for the head node's architecture last, but this is clearly sub-optimal and something better could be envisioned.
The approach @nathanhaigh suggested may not be the best since it can generate a lot of unnecessary jobs (and log files) especially when software with many dependencies are built.
I prefer the approach suggested by @boegel, i.e having the option of doing the full build (software + deps) within the same job.

[boegel]

Any suggestions for an option name that would make --robot to be passed down into the submitted job, and not checking locally which modules are already available?

[akesandgren]

Why not do what we will be doing once next EB is out...
Submit a job to the target node type that does eb --job=Slurm --robot
That way the dep resolution will be done on the correct target and stuff will be correct, if you have your slurm config correctly setup that is.

Otherwise --job-robot