enable hwloc, cuFFTMp, and HeFFTe support in GROMACS easyblock by bedroge · Pull Request #3531 · easybuilders/easybuild-easyblocks

bedroge · 2024-12-13T13:12:40Z

In EESSI we noticed that GROMACS builds currently show the following with gmx -version:

Multi-GPU FFT:       none
Hwloc support:       disabled

Hwloc is part of the foss toolchain and can be easily enabled.

For Multi-GPU FFT support either cuFFTMp (https://manual.gromacs.org/documentation/current/install-guide/index.html#using-cufftmp) or HeFFTe (https://manual.gromacs.org/documentation/current/install-guide/index.html#using-heffte) is required. I was trying to add support for both, but cuFFTMp is part of NVHPC, and simply adding that as dependency will make GROMACS pick up other stuff from that installation (e.g. OpenMP libraries). Since cuFFTMp also imposes some additional requirements (see https://docs.nvidia.com/hpc-sdk/cufftmp/usage/requirements.html), I've only added HeFFTe support for now. I've also just opened an easyconfigs PR for HeFFTe with CUDA support: easybuilders/easybuild-easyconfigs#22024. Once that's merged, I'll make another to add this as a dependency to CUDA versions of GROMACS.

ocaisa

This looks good, matches what I can find in the docs. My only concern is that there is no version-checking with the new options. I think the hwloc one is there since 2016, but the HEFFTE one is more recent (I can't quite figure it out but I think it is 2023, see https://gitlab.com/gromacs/gromacs/-/issues/4090). For our own use case we could make the check be more recent than when it was first supported.

EDIT: Indeed, heffte seems to first appear in 2023.1: https://manual.gromacs.org/2023.1/install-guide/index.html

EDIT: The option for hwloc is first documented in 2016.4: https://manual.gromacs.org/2016.4/install-guide/index.html

bedroge · 2024-12-16T09:50:58Z

This looks good, matches what I can find in the docs. My only concern is that there is no version-checking with the new options. I think the hwloc one is there since 2016, but the HEFFTE one is more recent (I can't quite figure it out but I think it is 2023, see https://gitlab.com/gromacs/gromacs/-/issues/4090). For our own use case we could make the check be more recent than when it was first supported.

EDIT: Indeed, heffte seems to first appear in 2023.1: https://manual.gromacs.org/2023.1/install-guide/index.html

EDIT: The option for hwloc is first documented in 2016.4: https://manual.gromacs.org/2016.4/install-guide/index.html

Thanks! I added the version checks, hadn't seen your edits. But I also see hwloc being mentioned in the 2016.1 docs, and it's also in the code:
https://gitlab.com/gromacs/gromacs/-/blob/v2016.1/CMakeLists.txt?ref_type=tags#L506

HeFFTe is being mentioned in the 2023 docs (https://manual.gromacs.org/current/release-notes/2023/major/performance.html#pme-decomposition-support-with-cuda-and-sycl-backends), and also in the CMake file for 2023 (https://gitlab.com/gromacs/gromacs/-/blob/v2023/CMakeLists.txt?ref_type=tags#L741) and 2023.1 (https://gitlab.com/gromacs/gromacs/-/blob/v2023.1/CMakeLists.txt?ref_type=tags#L749).

bedroge · 2024-12-16T10:01:20Z

One thing I was a little bit worried about is that the HeFFTe installation requires a GPU (for the tests), hence simply installing GROMACS and its dependencies will also require a GPU if we enable this by default. Should we make it optional in some way (commenting out the HeFFTe depdendency or disabling the tests)? For EESSI it would currently already cause an issue, as we build on nodes without GPUs.

al42and · 2025-01-03T11:57:57Z

Hi!

Don't want to derail the discussion here, but, while I don't have any recent numbers, the situation has not changed much from what NVIDIA reports in their blog:

We find cuFFTMp to be up to 2x faster [than HeFFTe]

HeFFTe has the benefit of supporting AMD and Intel GPUs, but it's not the best choice for CUDA installation. cuFFTMp has its own share of issues, as @bedroge outlined in the PR description, but I think a performance difference is relevant for evaluating which effort is more worthwhile.

Regarding versioning, can confirm that heffte (and cufftmp) were added in 2023, and hwloc was added in 2016.

bedroge · 2025-01-03T12:14:23Z

Hi!

Don't want to derail the discussion here, but, while I don't have any recent numbers, the situation has not changed much from what NVIDIA reports in their blog:

We find cuFFTMp to be up to 2x faster [than HeFFTe]

HeFFTe has the benefit of supporting AMD and Intel GPUs, but it's not the best choice for CUDA installation. cuFFTMp has its own share of issues, as @bedroge outlined in the PR description, but I think a performance difference is relevant for evaluating which effort is more worthwhile.

Regarding versioning, can confirm that heffte (and cufftmp) were added in 2023, and hwloc was added in 2016.

Thanks for your input, it's definitely a fair point. I initially added only HeFFTe support in this PR, as it seemed like a more logical default option (e.g. no additional requirements on the hardware like with cuFFTMp), and a first attempt at adding cuFFTMp support failed miserably 😅 But I can have another look at it, ultimately it would be nice if the easyblock supports both, and people can choose between the two of them.

…s into gromacs_heffte

bedroge · 2026-04-28T14:50:30Z

Now that we have an easyconfig for cuFFTMp (https://github.com/easybuilders/easybuild-easyconfigs/blob/develop/easybuild/easyconfigs/c/cuFFTMp/cuFFTMp-11.4.0-gompi-2025b-CUDA-12.9.1.eb), it's trivial to add support for it. I've done that in 23eda47.

Initially it didn't compile because it was picking up the cufft.h from CUDA, while it should pick up the one from cuFFTMp, causing errors like:

/home/bob/eessi/versions/2025.06/software/linux/x86_64/intel/cascadelake/software/cuFFTMp/11.4.0-gompi-2025b-CUDA-12.9.1/include/cufftMp.h:74:4: error: #error cuFFT and cuFFTMp version mismatch. .../math_libs/X.Y/include/cufftmp/ should be included before .../math_libs/X.Y/include/
   74 |   #error cuFFT and cuFFTMp version mismatch. .../math_libs/X.Y/include/cufftmp/ should be included before .../math_libs/X.Y/include/
      |    ^~~~~

I couldn't really force it to use the header provided by cuFFTMp first (moving it up or down in the deps list didn't work), but adding -Xcompiler -v to the nvcc command showed that it first searches in the source dir. So, I solved things by adding a symlink to the correct cufft.h in the source dir, and that allowed me to complete the build. The tests are still failing for me, as I don't have a system that meets the requirements, see https://docs.nvidia.com/cuda/cufftmp/usage/requirements.html and https://docs.nvidia.com/nvshmem/release-notes-install-guide/install-guide/abstract.html#nvshmem-installation-guide.

bedroge · 2026-04-28T14:52:37Z

Assuming you cannot select the multi-GPU FFT library at runtime, we would need to select them at build time. We could do that by having an easyconfig parameter that controls this, or by having separate easyconfigs with a corresponding version suffix?

bedroge · 2026-04-28T15:01:10Z

@boegelbot please test @ jsc-zen3-a100
CORE_CNT=16
EB_ARGS="--installpath /tmp/$USER/pr3531 GROMACS-2025.3-foss-2025b-CUDA-12.9.1.eb"

boegelbot · 2026-04-28T15:13:08Z

@bedroge: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de

PR test command 'if [[ develop != 'develop' ]]; then EB_BRANCH=develop ./easybuild_develop.sh 2> /dev/null 1>&2; EB_PREFIX=/home/boegelbot/easybuild/develop source init_env_easybuild_develop.sh; fi; EB_PR=3531 EB_ARGS="--installpath /tmp/$USER/pr3531 GROMACS-2025.3-foss-2025b-CUDA-12.9.1.eb" EB_CONTAINER= EB_REPO=easybuild-easyblocks EB_BRANCH=develop /opt/software/slurm/bin/sbatch --job-name test_PR_3531 --ntasks="16" --partition=jsczen3g --gres=gpu:1 ~/boegelbot/eb_from_pr_upload_jsc-zen3.sh' executed!

exit code: 0
output:

Submitted batch job 10308

Test results coming soon (I hope)...

Details

- notification for comment with ID 4336494167 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

bedroge · 2026-04-28T15:14:07Z

@boegelbot please test @ jsc-zen3
CORE_CNT=16
EB_ARGS="--installpath /tmp/$USER/pr3531 GROMACS-2023.1-foss-2022a.eb GROMACS-2025.4-foss-2025b.eb"

boegelbot · 2026-04-28T15:33:08Z

@bedroge: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de

PR test command 'if [[ develop != 'develop' ]]; then EB_BRANCH=develop ./easybuild_develop.sh 2> /dev/null 1>&2; EB_PREFIX=/home/boegelbot/easybuild/develop source init_env_easybuild_develop.sh; fi; EB_PR=3531 EB_ARGS="--installpath /tmp/$USER/pr3531 GROMACS-2023.1-foss-2022a.eb GROMACS-2025.4-foss-2025b.eb" EB_CONTAINER= EB_REPO=easybuild-easyblocks EB_BRANCH=develop /opt/software/slurm/bin/sbatch --job-name test_PR_3531 --ntasks="16" ~/boegelbot/eb_from_pr_upload_jsc-zen3.sh' executed!

exit code: 0
output:

Submitted batch job 10310

Test results coming soon (I hope)...

Details

- notification for comment with ID 4336609134 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

boegelbot · 2026-04-28T15:42:43Z

Test report by @boegelbot

Overview of tested easyconfigs (in order)

SUCCESS GROMACS-2025.3-foss-2025b-CUDA-12.9.1.eb

Build succeeded for 1 out of 1 (total: 28 mins 39 secs) (1 easyconfigs in total)
jsczen3g1.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.7, x86_64, AMD EPYC-Milan Processor (zen3), 1 x NVIDIA NVIDIA A100 80GB PCIe, 590.48.01, Python 3.9.25
See https://gist.github.com/boegelbot/ab97d5bb6bfe0c6b8b22e861b87b1b52 for a full test report.

boegelbot · 2026-04-28T16:44:38Z

Test report by @boegelbot

Overview of tested easyconfigs (in order)

SUCCESS GROMACS-2023.1-foss-2022a.eb
SUCCESS GROMACS-2025.4-foss-2025b.eb

Build succeeded for 2 out of 2 (total: 1 hour 11 mins 15 secs) (2 easyconfigs in total)
jsczen3c2.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.7, x86_64, AMD EPYC-Milan Processor (zen3), Python 3.9.25
See https://gist.github.com/boegelbot/225f832fb2fec3d1ee57ebe047f9a51c for a full test report.

bedroge · 2026-04-28T21:15:02Z

So, I solved things by adding a symlink to the correct cufft.h in the source dir, and that allowed me to complete the build.

Hmm, forgot to actually include that fix, but I just pushed it (cfb6f93).

bedroge · 2026-04-28T21:15:19Z

@boegelbot please test @ jsc-zen3-a100
CORE_CNT=16
EB_ARGS="--installpath /tmp/$USER/pr3531 GROMACS-2025.3-foss-2025b-CUDA-12.9.1.eb"

ocaisa

Looks good to me

ocaisa · 2026-04-28T21:21:00Z

@boegelbot please test @ jsc-zen3-a100
CORE_CNT=16
EB_ARGS="--installpath /tmp/$USER/pr3531 GROMACS-2025.3-foss-2025b-CUDA-12.9.1.eb"

boegelbot · 2026-04-28T21:33:09Z

@ocaisa: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de

PR test command 'if [[ develop != 'develop' ]]; then EB_BRANCH=develop ./easybuild_develop.sh 2> /dev/null 1>&2; EB_PREFIX=/home/boegelbot/easybuild/develop source init_env_easybuild_develop.sh; fi; EB_PR=3531 EB_ARGS="--installpath /tmp/$USER/pr3531 GROMACS-2025.3-foss-2025b-CUDA-12.9.1.eb" EB_CONTAINER= EB_REPO=easybuild-easyblocks EB_BRANCH=develop /opt/software/slurm/bin/sbatch --job-name test_PR_3531 --ntasks="16" --partition=jsczen3g --gres=gpu:1 ~/boegelbot/eb_from_pr_upload_jsc-zen3.sh' executed!

exit code: 0
output:

Submitted batch job 10314

Test results coming soon (I hope)...

Details

- notification for comment with ID 4339145797 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

boegelbot · 2026-04-28T22:01:27Z

Test report by @boegelbot

Overview of tested easyconfigs (in order)

SUCCESS GROMACS-2025.3-foss-2025b-CUDA-12.9.1.eb

Build succeeded for 1 out of 1 (total: 28 mins 4 secs) (1 easyconfigs in total)
jsczen3g1.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.7, x86_64, AMD EPYC-Milan Processor (zen3), 1 x NVIDIA NVIDIA A100 80GB PCIe, 590.48.01, Python 3.9.25
See https://gist.github.com/boegelbot/9f6d82d29030b0597d7d12240a590bca for a full test report.

boegel · 2026-05-22T09:37:08Z


+        # Enable hwloc support (added in v2016) if it's listed as dependency
+        if gromacs_version >= '2016' and get_software_root('hwloc'):
+            self.cfg.update('configopts', '-DGMX_HWLOC=ON')


I don't mind this, but I do wonder what the impact is.

hwloc is essentially always there as a indirect dependency, via OpenMPI.

Maybe we should change this check to only enable this when hwloc is listed as a direct dependency of GROMACS, so checking via self.cfg.dependencies() ?

The documentation seems to suggest that it's useful to always enable this if hwloc is available:

Added CMake support to detect and build GROMACS with hwloc, which will improve GROMACS ability to recognize and take advantage of all the available hardware. If hwloc is unavailable, GROMACS will fall back on other detection routines.

and

Run-time detection of hardware capabilities can be improved by linking with hwloc. By default this is turned off since it might not be supported everywhere, but if you have hwloc installed it should work by just setting -DGMX_HWLOC=ON

But maybe @al42and can give some advice here: does that indeed make sense, or are there possible drawbacks?

There are no downsides to using hwloc, except for the issues you encountered on Grace Hopper. However, I would not expect these failed tests to translate into any actual issues for users, because the upsides of using hwloc in GROMACS are, for now, mostly aspirational. It can help somewhat with CPU affinity, particularly SMT, but NUMA information is not used at all except for logging; and, in the HPC setting, the scheduler should take care of CPU affinity.

So, dropping hwloc does not have any major drawbacks as far as I know.

So, dropping hwloc does not have any major drawbacks as far as I know.

@al42and Did you mean to state that "enabling hwloc support does not have any major drawbacks" ?
Just so we're clear 😅

With that in mind, I'm fine with merging this as is, if I understood it correctly.

Did you mean to state that "enabling hwloc support does not have any major drawbacks" ?

Neither enabling nor disabling has major drawbacks 🙃 But enabling has minor benefits.

Thanks @al42and, also for the other suggestions (which I can fix in a follow-up PR). @boegel could you merge this then, so we can ingest the GROMACS builds from EESSI/software-layer#1497?

al42and · 2026-05-22T14:59:42Z

        # The list of GMX_SIMD options can be found
        # http://manual.gromacs.org/documentation/2018/install-guide/index.html#simd-support
        if 'MIC-AVX512' in optarch and LooseVersion(self.version) >= LooseVersion('2016'):
            res = 'AVX_512_KNL'


Nit: AVX_512_KML removed in 2026: https://manual.gromacs.org/documentation/2026.0/release-notes/2026/major/removed-functionality.html

al42and · 2026-05-22T15:03:45Z

@@ -211,6 +211,23 @@
                    cuda_cc_semicolon_sep = self.cfg.get_cuda_cc_template_value(
                        "cuda_cc_semicolon_sep").replace('.', '')
                    self.cfg.update('configopts', '-DGMX_CUDA_TARGET_SM="%s"' % cuda_cc_semicolon_sep)


Nit: in 2026, we finally switched to traditional CMAKE_CUDA_ARCHITECTURES, which, if I understand correctly, is set automatically by the EB cmake module?

@al42and Any advantage of using -DCMAKE_CUDA_ARCHITECTURES over -DGMX_CUDA_TARGET_SM ?

We'll tackle that in a follow-up PR...

Should be identical in practice. The only advantage is that CMAKE_CUDA_ARCHITECTURES is standardized.

boegel

lgtm

boegel · 2026-06-07T10:22:29Z

Test report by @boegel

Overview of tested easyconfigs (in order)

SUCCESS GROMACS-2024.4-foss-2023b-CUDA-12.4.0.eb

Build succeeded for 1 out of 1 (total: 35 mins 46 secs) (1 easyconfigs in total)
node3300.joltik.os - Linux RHEL 9.6, x86_64, Intel(R) Xeon(R) Gold 6242 CPU @ 2.80GHz (cascadelake), 1 x NVIDIA Tesla V100-SXM2-32GB, 580.159.04, Python 3.9.21
See https://gist.github.com/boegel/eda70c9c75139768aaba845f9459da26 for a full test report.

boegel · 2026-06-07T10:50:21Z

Test report by @boegel

Overview of tested easyconfigs (in order)

SUCCESS GROMACS-2026.2-foss-2025b.eb

Build succeeded for 1 out of 1 (total: 1 hour 2 mins 52 secs) (1 easyconfigs in total)
node4235.shinx.os - Linux RHEL 9.6, x86_64, AMD EPYC 9654 96-Core Processor (zen4), Python 3.9.21
See https://gist.github.com/boegel/4bede08c4e977380549ea277bb61ffd0 for a full test report.

bedroge added 2 commits December 13, 2024 14:03

enable HeFFTe support

d1e5a64

enable hwloc support

53a9a23

bedroge added the enhancement label Dec 13, 2024

ocaisa requested changes Dec 16, 2024

View reviewed changes

bedroge added 4 commits December 16, 2024 10:41

only enable hwloc support for gromacs >= 2016

82c22bd

add version check for HeFFTe support

2042312

update comment about hwloc support

49ca92e

switch order of checks to keep things consistent

fa45a53

bedroge mentioned this pull request Dec 16, 2024

add dependency on HeFFTe (for multi-GPU FFT) to easyconfig for GROMACS with foss-2023b + CUDA-12.4.0 easybuilders/easybuild-easyconfigs#22032

Draft

boegel added this to the release after 5.1.1 milestone Jul 3, 2025

boegel modified the milestones: next release (5.1.2), release after 5.1.2 (5.2.0?) Sep 10, 2025

boegel modified the milestones: next release (5.2.0), release after 5.2.0 Dec 14, 2025

boegel modified the milestones: next release (5.2.1), release after 5.2.1 Feb 11, 2026

boegel modified the milestones: next release (5.3.0), release after 5.3.0 Apr 8, 2026

bedroge mentioned this pull request Apr 28, 2026

{bio}[foss/2025b] GROMACS v2025.4 w/ CUDA 12.9.1 (and apply new patch to 2025.3) easybuilders/easybuild-easyconfigs#25837

Merged

bedroge added 2 commits April 28, 2026 13:49

Merge branch 'develop' of github.com:easybuilders/easybuild-easyblock…

a82b4a2

…s into gromacs_heffte

also add support for cuFFTMp

23eda47

bedroge requested a review from ocaisa April 28, 2026 14:50

add fix for finding right cufft.h

cfb6f93

ocaisa reviewed Apr 28, 2026

View reviewed changes

Comment thread easybuild/easyblocks/g/gromacs.py Outdated

Apply suggestion from @ocaisa

1569e31

ocaisa approved these changes Apr 28, 2026

View reviewed changes

bedroge changed the title ~~enable hwloc and HeFFTe support in GROMACS easyblock~~ enable hwloc, cuFFTMp, and HeFFTe support in GROMACS easyblock Apr 28, 2026

boegelbot mentioned this pull request Apr 29, 2026

{bio}[foss/2025b] GROMACS v2026.2 easybuilders/easybuild-easyconfigs#25879

Merged

boegel reviewed May 22, 2026

View reviewed changes

al42and reviewed May 22, 2026

View reviewed changes

bedroge mentioned this pull request Jun 4, 2026

{2025.06}[2025b] GROMACS 2026.2 EESSI/software-layer#1497

Merged

boegel approved these changes Jun 7, 2026

View reviewed changes

boegel merged commit f4b4629 into easybuilders:develop Jun 7, 2026
22 checks passed

bedroge deleted the gromacs_heffte branch June 7, 2026 20:39

Conversation

bedroge commented Dec 13, 2024

Uh oh!

ocaisa left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bedroge commented Dec 16, 2024

Uh oh!

bedroge commented Dec 16, 2024

Uh oh!

al42and commented Jan 3, 2025

Uh oh!

bedroge commented Jan 3, 2025

Uh oh!

bedroge commented Apr 28, 2026

Uh oh!

bedroge commented Apr 28, 2026

Uh oh!

bedroge commented Apr 28, 2026

Uh oh!

boegelbot commented Apr 28, 2026

Uh oh!

bedroge commented Apr 28, 2026

Uh oh!

boegelbot commented Apr 28, 2026

Uh oh!

boegelbot commented Apr 28, 2026

Overview of tested easyconfigs (in order)

Uh oh!

boegelbot commented Apr 28, 2026

Overview of tested easyconfigs (in order)

Uh oh!

bedroge commented Apr 28, 2026

Uh oh!

bedroge commented Apr 28, 2026

Uh oh!

Uh oh!

ocaisa left a comment

Choose a reason for hiding this comment

Uh oh!

ocaisa commented Apr 28, 2026

Uh oh!

boegelbot commented Apr 28, 2026

Uh oh!

boegelbot commented Apr 28, 2026

Overview of tested easyconfigs (in order)

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

boegel left a comment

Choose a reason for hiding this comment

Uh oh!

boegel commented Jun 7, 2026

Overview of tested easyconfigs (in order)

Uh oh!

boegel commented Jun 7, 2026

Overview of tested easyconfigs (in order)

Uh oh!

Uh oh!

Reviewers

Assignees

ocaisa left a comment •

edited

Loading