mlwrap (module-managed dispatcher) #605
Description
Hi,
In my HPC environment I have created mlwrap dispatcher, which loads requested module, resolves the executable from the resulting environment, and then execute it.
This help us with some challenges were you need an absolute path entry point to a module-managed tool when the caller cannot run module load, or need to have stable execution behavior (with backward compatibility) across OS/arch/toolchain (logic happen inside the modulefiles).
If you see it can be beneficial, maybe worth to consider providing similar approach natively.
This is how our mlwrap look like:
$ cat /usr/local/bin/mlwrap
#!/bin/bash
set -euo pipefail
usage() {
cat >&2 <<'EOF'
mlwrap - run a tool via environment-modules
Usage:
mlwrap <module[/version]> [--exec <binary>] [--] [args...]
mlwrap --help
mlwrap --list
Behavior:
- loads the requested module, then resolves the executable and run it.
- default <executable> is inferred from the module name
Options:
--exec <executable> override the executable after loading the module.
useful when module name != the executable name.
--help show this usage message.
--list list available modules (one per line).
-- optional: end of mlwrap options, pass remaining args to the executable only.
Examples:
mlwrap lcov/2.3 --version
mlwrap llvm/20.1.0 --exec clang --version
mlwrap python/3.11 -- --help
EOF
}
die() { echo "mlwrap: $*" >&2; exit 127; }
if [[ $# -eq 0 ]]; then
usage
exit 2
fi
# avoid sourcing the running user ~/.modulerc file
export MODULES_IGNORE_USER_RC=1
# initialize the 'module' function if not available
export MODULEPATH="/usr/local/share/modulefiles"
command -v module >/dev/null || source /usr/local/pkgs/environment-modules/stable/init/bash
case "$1" in
--help)
usage
exit 0
;;
--version)
module --version
exit 0
;;
--list)
module avail -t 2>/dev/null \
| sed -E '
s/[[:space:]]+$//;
s/[[:space:]]*\(default\)//g;
s/[[:space:]]*<\*[^>]*\*>//g;
s/[[:space:]]+$//;
/^[[:space:]]*$/d;
/:$/d
'
exit 0
;;
esac
mod="$1"; shift
exe="${mod%%[@/]*}"
while [[ $# -gt 0 ]]; do
case "$1" in
--exec) shift; [[ $# -ge 1 ]] || usage; exe="$1"; shift ;;
--) shift; break ;;
*) break ;;
esac
done
# ensure a clean baseline
module -s purge
# load the module
module -s load "$mod"
# run
exec -a "$exe" "$exe" "$@"
some examples:
$ mlwrap perl/5.40 --version
This is perl 5, version 40, subversion 2 (v5.40.2) built for x86_64-linux-thread-multi
$ mlwrap perl/5.40 -e 'print "hello\n";'
hello
$ mlwrap ripgrep/15.1.0 --exec rg --version
ripgrep 15.1.0 (rev af60c2de9d)
$ mlwrap yq --version
yq version 3.3.4
$ mlwrap llvm@20.1.0 --exec clang --version
clang version 20.1.0 (https://github.com/llvm/llvm-project 24a30daaa559829ad079f2ff7f73eb4e18095f88)
$ mlwrap uv@latest --version
uv 0.8.5
$ alias code="mlwrap vscode@latest --exec code"
$ code --version
1.104.3
385651c938df8a906869babee516bffd0ddb9829
x64
Thanks,
Zohman