Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 55 additions & 0 deletions docs/cwl/running.rst
Original file line number Diff line number Diff line change
Expand Up @@ -130,6 +130,61 @@ all CWL intermediate files on disk and share them between jobs using file
paths, instead of storing them in the file store and downloading them when jobs
need them.

Running CWL workflows with MPI
------------------------------

MPI support for CWL workflows in Toil is provided by ``cwltool`` through its
``cwltool:MPIRequirement`` extension. For details on using this extension, see
`the cwltool MPI documentation <cwltool_mpi_>`_.

The following CWL tool, saved as ``hostname.cwl``, requests four MPI
processes:

::

cwlVersion: v1.2
class: CommandLineTool
$namespaces:
cwltool: "http://commonwl.org/cwltool#"
requirements:
cwltool:MPIRequirement:
processes: 4

baseCommand: hostname

inputs: []
outputs: []

The MPI launcher is configured separately in ``mpi_config.yml``. For example,
to use Slurm's ``srun`` launcher:

::

runner: srun
nproc_flag: "-n"
extra_flags:
- "--partition=debug"

You can then run the workflow on a two-node Slurm allocation with:

::

$ toil-cwl-runner \
--batchSystem slurm \
--slurmArgs="--partition=debug \
--account=project_123 \
--nodes=2 \
--ntasks-per-node=2 \
Comment on lines +176 to +177

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This means that toil is not correctly configured for MPI. such flags should only be added to MPI jobs, not every single job.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I think I see what you mean. In this example it'd be fine as there is a single job and it is using MPI. But if we had other jobs that do not require MPI and must be executed elsewhere (cloud, locally, container) then we wouldn't want them to be under the same Slurm allocation.

--export=ALL" \
--mpi-config-file mpi_config.yml \
hostname.cwl

In this example, the workflow runs four MPI processes distributed across two
nodes. Adjust the Slurm and MPI configuration options as needed for your
cluster.

.. _cwltool_mpi: https://github.com/common-workflow-language/cwltool#running-mpi-based-tools-that-need-to-be-launched

Toil & CWL Tips
---------------

Expand Down