An issue with var_type in modextrapaths on LUMI

[klust]

Consider the easyconfig

easyblock = 'Bundle'

name = 'singularity-slurm-bindings'
version = '3.0.0'

description = """Just a little testJust a little test."""

homepage = '()'

toolchain = SYSTEM

env_mod_extra_paths = {
    'SINGULARITY_BIND': {
        'paths': ['/pfs', '/scratch', '/appl'],
        'delimiter': ',',
        'prepend': False,
        #'var_type': 'STRING',
        #'var_type': 'PATH',
    }
}

env_mod_category = 'tools'

referring to 3 filesystems on LUMI.

Issue 1 When running this easyconfig as is, I get the module file

help([==[

Description
===========
Just a little testJust a little test.


More information
================
 - Homepage: ()
]==])

whatis([==[Description: Just a little testJust a little test.]==])
whatis([==[Homepage: ()]==])
whatis([==[URL: ()]==])

local root = "/users/kurtlust/LUMI-user/SW/container/singularity-slurm-bindings/3.0.0"

conflict("singularity-slurm-bindings")

append_path("SINGULARITY_BIND", "/pfs", ",")
append_path("SINGULARITY_BIND", "/appl", ",")
setenv("EBROOTSINGULARITYMINSLURMMINBINDINGS", root)
setenv("EBVERSIONSINGULARITYMINSLURMMINBINDINGS", "3.0.0")
setenv("EBDEVELSINGULARITYMINSLURMMINBINDINGS", pathJoin(root, "easybuild", "singularity-slurm-bindings-3.0.0-easybuild-devel"))

-- Built with EasyBuild version 5.2.1

so the line for /scratch is missing. The issue is not so much that it is missing, though I cannot see why it would be filtered out. Granted that on LUMI, those three directories are set up differently (/pfs is the base for Lustre, /appl is a symbolic link, but I'm not sure how /scratch is set up, probably some Lustre trick), but the main issue is that I get no warning in the screen output that something is wrong with /scratch and also cannot find anything in the log file about it being removed. The reason why /scratch is deleted may be LUMI-specific (though it could then happen on other systems with Lustre also), but the disturbing fact that there is no warning is likely not LUMI-specific.

It turns out that the issue is solved when using 'var_type': 'PATH', rather than the default value: The module is now what I would expect:

help([==[

Description
===========
Just a little testJust a little test.


More information
================
 - Homepage: ()
]==])

whatis([==[Description: Just a little testJust a little test.]==])
whatis([==[Homepage: ()]==])
whatis([==[URL: ()]==])

local root = "/users/kurtlust/LUMI-user/SW/container/singularity-slurm-bindings/3.0.0"

conflict("singularity-slurm-bindings")

append_path("SINGULARITY_BIND", "/pfs", ",")
append_path("SINGULARITY_BIND", "/scratch", ",")
append_path("SINGULARITY_BIND", "/appl", ",")
setenv("EBROOTSINGULARITYMINSLURMMINBINDINGS", root)
setenv("EBVERSIONSINGULARITYMINSLURMMINBINDINGS", "3.0.0")
setenv("EBDEVELSINGULARITYMINSLURMMINBINDINGS", pathJoin(root, "easybuild", "singularity-slurm-bindings-3.0.0-easybuild-devel"))

-- Built with EasyBuild version 5.2.1

But this brings me to the issue 2. My first idea was to set 'var_type': 'STRING' as I was hoping that the three directories would then just be treated as strings and no tests would be run. But this is not what happens: Now no append_path lines are generated at all:

help([==[

Description
===========
Just a little testJust a little test.


More information
================
 - Homepage: ()
]==])

whatis([==[Description: Just a little testJust a little test.]==])
whatis([==[Homepage: ()]==])
whatis([==[URL: ()]==])

local root = "/users/kurtlust/LUMI-user/SW/container/singularity-slurm-bindings/3.0.0"

conflict("singularity-slurm-bindings")

setenv("EBROOTSINGULARITYMINSLURMMINBINDINGS", root)
setenv("EBVERSIONSINGULARITYMINSLURMMINBINDINGS", "3.0.0")
setenv("EBDEVELSINGULARITYMINSLURMMINBINDINGS", pathJoin(root, "easybuild", "singularity-slurm-bindings-3.0.0-easybuild-devel"))

-- Built with EasyBuild version 5.2.1

[boegel]

(note to others, to avoid confusion: env_mod_extra_paths is the new name for modextrapaths, see also #4464)

@klust I didn't look deeply into this, but I think the reason that /scratch gets filtered out is that EasyBuild makes sure that the path actually exists before injecting the corresponding line in the module file. I would expect a warning/message in the EasyBuild log if the path doesn't exist, if that's not the case (or only done when --debug is used), we should fix that.

That said, I would also expect the same check to happen when 'var_type': 'PATH' is used.

And I can't explain why none of the lines appear when 'var_type': 'STRING' is used.

Maybe @lexming can clarify, since he was involved with refactoring the logic for this stuff in EasyBuild 5.0

[lexming]

The default setting for env_mod_extra_paths is PATH_WITH_FILES. In that case, paths must exist and contain at least one file in their top level directory.

Switching to PATHS allows those paths to be empty. So my guess is that /scratch is empty in the build environment where EB is running, so by default it will not be added as a search path because it is empty. In this case you should see a log message at debug level informing about the removal of this path. See

easybuild-framework/easybuild/tools/modules.py

Line 302 in 1f38202

self.log.debug(f"Discarded search path '{exp_path}' of type '{self.type}' to empty directory.")

Variable of STRING type are not yet supported. You should see a warning in the output of EB informing about this fact. See

easybuild-framework/easybuild/framework/easyblock.py

Lines 1747 to 1750 in 1f38202

	self.log.warning(
	f"Non-path variables found in module load environment: {non_path_envars}."
	"This is not yet supported by this version of EasyBuild."
	)

There is an open PR to add support for STRINGS: #5138

I would say that using PATHS is what you need if any of those directories is empty and that is what you want. So this is working as intended.

[klust]

I will try at the debug level and come back with more information. It is a bit annoying that that error would only be shown at the debug level though while so many other errors cause Easybuild to stop and show an error. And even more so because this error leads to an error in a modulefile that in this case would have stopped the software from working the way a user expects in a way that is not easy to test in the sanity checks. I'll check what happens when I used STRINGS as I don't think I went into the log with that one.

However, /scratch is not empty but contains 2728 entries at the moment. All are symbolic links though. So it looks like the test is happy if the item in the list of paths is itself a symbolic link, but not if the item is a directory containing only symbolic links (and in this case, to other directories).

[lexming]

@klust it is not shown as an error because it is not an error. EB is discarding empty paths because that is what you are asking it to do with PATH_WITH_FILES. It is expected behavior.

Regarding the symlinks, those are indeed not considered by PATH_WITH_FILES. It only checks for actual files. We can improve that check to resolve symlinks to files.

[klust]

Then I conclude that the current behaviour of EasyBuild for this parameter is very inconsistent with the way EB behaves for other parameters in the easyconfigs, not to say downright dangerous. For many parameters, EB warns even if there is nothing really wrong, but just could be wrong. The most extreme example is probably producing an error stopping EB when configure in autotools produces a warning that a parameter is not recognised. Which was a burden initially as several of our EB recipes broke at a time that we were in a rush to roll out a new software stack, but is a good thing in the end as it also helps catch typos or options that have changed. While here, a value is simply dropped without even a warning on the screen or in a regular non-debug log file (and it is not the place in the log file that one usually checks anyway). So, e.g., if the creator of the EB recipe makes a typo in the path, they wouldn't be warned either, or if the directory which is given there turns out to be empty because of something working in an unexpected way in the installation (or, e.g., something being moved between versions of software and the path not being adapted), the entry is simply silently dropped without any proper warning. Likely leading to unexpected behaviour of the application that only gets noticed when users start using the application as it may not have any effect on simple --version or -h calls in sanity check commands to see if the executable loads.

If PATH_WITH_FILES is supposed to only check for files, then it is probably also not working as advertised, as in the example I gave (which is a simplified version of one with a much longer list that I was using on LUMI):

• /pfs is a directory that only contains 5 subdirectories but no files itself (no hidden ones either), or does it try tracking the whole tree to find some files that are readable?
• /appl itself is a symbolic link, and the directory it links to also only contains directories. Similar situation: You would need to walk a deep tree to find files. The step goes quick enough that I doubt EB does that. But that one was also OK.
• /scratch is only different in that it contains symbolic links to directories elsewhere on the system. So it is surprising for a user that that one is not accepted while the two others are.

If all three would have been dropped, I would have been less surprised (as there are no regular files or symbolic links to regular files in either). Luckily I was checking the generated modules rather carefully as it was the first time I used that parameter and was wondering what the best value would be.

Besides that, the feature is underdocumented and I could only find possible values for var_type by analysing the code, which is how I also ended up trying STRING. Here too, usually when an easyconfig has a wrong value, it shows a warning and more often even halts execution, while here it is just a message in the log file.

That being said, in the log file with PATH_WITH_FILES I also found another strange line just before the one telling that scratch was discarded.

== 2026-03-25 17:30:48,783 modules.py:302 DEBUG Discarded search path '/pfs/lustrep1/projappl/project_462000008/kurtlust/LUMI-user-links/SW/container/singularity-slurm-bindings/3.0.0/' of type 'ModEnvVarType.PATH_WITH_FILES' to empty directory.

I wonder why it was even investigating that directory as I had no intent anywhere to add the installation directory to that path-style variable. Or is this a line from a test for another implicitly set path-style variable which probably uses the same function to check? I think the message could also be clearer, though maybe there is not enough information available at that point in the EB code? It is a bit strange to see only that one then and no lines for all other tentative module environment variables.

[lexming]

PATH_WITH_FILES checks for files at any level. There is also PATH_WITH_TOP_FILES which only checks for files at the top level, which we use for known search paths where there is no recursion like LD_LIBRARY_PATH. So, /pts and /appl are kept because they contain at least one file, even if very deep. Working as expected.

As I said in my previous message, the only issue here is not resolving symlinks to determine if a directory is populated, that I agree needs to be improved in PATH_WITH_FILES.

Not throwing an error is intended and working as expected. Having an empty directory in a search path is useless, so there is no difference in having it or not in the module file. And you actually don't want those search paths to avoid unnecessary lookups. If the lack of those search paths would break the installation, EB will already error out in the regular sanity checks.

The lack of detail about this feature in our documentation is a valid concern. When I have a bit of time, I'll try to throw that task to an LLM.