PreCreate Injection Hook by fcruzcscs · Pull Request #7 · sarus-suite/performance-extensions

fcruzcscs · 2026-04-28T22:06:23Z

Precreate hook that plans library injection from the container rootfs and rewrites the OCI config
to add bind mounts to inject host libs.

Architecture Overview

This hook is architected as a small compiler for OCI specs.

Its lifecycle in main.rs is a five-stage pipeline:

Read the incoming OCI config JSON from stdin.
Load hook inputs from the config plus environment variables.
Discover what libraries the container already exposes.
Plan a set of safe config edits.
Apply those edits and emit a rewritten OCI config to stdout.

The core data model is:

HookInputs is the input contract
Library keeps the semantic unit of logic: path, parsed linker name, real name, and ABI version
ConfigEdits is the planned output: mounts, LD_LIBRARY_PATH additions, extra mounts, extra env, and warnings

For each input library, the planning layer makes one decision: overwrite an existing container library path, or inject through a directory and extend LD_LIBRARY_PATH
Always deciding replacement if ABI mayor is respected, otherwise it does directory placement.

Enabling this type of hook injection config:

{
  "version": "1.0.0",
  "hook": {
    "path": "/opt/hooks/pc_injection_hook",
    "env": [
      "LDCONFIG_PATH=/sbin/ldconfig",
      "INJECTION_PRIMARY_LIBS=/opt/cray/libfabric/1.22.0/lib64/libfabric.so.1",
      "INJECTION_DEPENDENCY_LIBS=/usr/lib64/libcxi.so.1:/usr/lib64/libcurl.so.4:/usr/lib64/libjson-c.so.3:/usr/lib64/libatomic.so.1:/usr/lib64/libnl-3.so.200:/usr/lib64/libnghttp2.so.14:/usr/lib64/libidn2.so.0:/usr/lib64/libssh.so.4:/usr/lib64/libpsl.so.5:/usr/lib64/libssl.so.1.1:/usr/lib64/libcrypto.so.1.1:/usr/lib64/libgssapi_krb5.so.2:/usr/lib64/libldap_r-2.4.so.2:/usr/lib64/liblber-2.4.so.2:/usr/lib64/libzstd.so.1:/usr/lib64/libbrotlidec.so.1:/usr/lib64/libz.so.1:/usr/lib64/libunistring.so.2:/usr/lib64/libjitterentropy.so.3:/usr/lib64/libkrb5.so.3:/usr/lib64/libk5crypto.so.3:/lib64/libcom_err.so.2:/usr/lib64/libkrb5support.so.0:/usr/lib64/libsasl2.so.3:/usr/lib64/libbrotlicommon.so.1:/usr/lib64/libkeyutils.so.1:/usr/lib64/libpcre.so.1:/lib64/libselinux.so.1:/usr/lib64/libxpmem.so.0",
      "INJECTION_EXTRA_MOUNTS=/var/spool/slurmd:/var/spool/slurmd:bind:bind,rw,nosuid,noexec,nodev,private;/var/lib/hugetlbfs:/var/lib/hugetlbfs:bind:rbind,rw,nosuid,nodev,private",
      "INJECTION_EXTRA_ENV=MPIR_CVAR_CH4_OFI_MULTI_NIC_STRIPING_THRESHOLD=100000000",
      "INJECTION_COMPATIBILITY=major"
    ]
  },
  "when": {
    "annotations": {
      "com.hooks.cxi.enabled": "^true$"
    }
  },
  "stages": ["precreate"]
}

fcruzcscs · 2026-04-28T22:08:16Z

This branch also adds a new test vagrant VM (setup for mac only at the moment) for local testing. This passes built tests and bats tests for mock cases.
Pending build and test on target envs.

fcruzcscs · 2026-04-28T23:42:06Z

What to do if the library to inject is not on the container already? (i.e. nothing to replace)

Works on target

{
  "version": "1.0.0",
  "hook": {
    "path": "/users/felipecr/.local/libexec/performance-extensions/pc_injection_hook_wrapper",
    "env": [
      "LDCONFIG_PATH=/sbin/ldconfig",
      "INJECTION_PRIMARY_LIBS=/opt/cray/libfabric/1.22.0/lib64/libfabric.so.1",
      "INJECTION_DEPENDENCY_LIBS=/usr/lib64/libcxi.so.1:/usr/lib64/libcurl.so.4:/usr/lib64/libjson-c.so.3:/usr/lib64/libatomic.so.1:/usr/lib64/libnl-3.so.200:/usr/lib64/libnghttp2.so.14:/usr/lib64/libidn2.so.0:/usr/lib64/libssh.so.4:/usr/lib64/libpsl.so.5:/usr/lib64/libssl.so.1.1:/usr/lib64/libcrypto.so.1.1:/usr/lib64/libgssapi_krb5.so.2:/usr/lib64/libldap_r-2.4.so.2:/usr/lib64/liblber-2.4.so.2:/usr/lib64/libzstd.so.1:/usr/lib64/libbrotlidec.so.1:/usr/lib64/libz.so.1:/usr/lib64/libunistring.so.2:/usr/lib64/libjitterentropy.so.3:/usr/lib64/libkrb5.so.3:/usr/lib64/libk5crypto.so.3:/lib64/libcom_err.so.2:/usr/lib64/libkrb5support.so.0:/usr/lib64/libsasl2.so.3:/usr/lib64/libbrotlicommon.so.1:/usr/lib64/libkeyutils.so.1:/usr/lib64/libpcre.so.1:/usr/lib64/libselinux.so.1:/usr/lib64/libxpmem.so.0",
      "INJECTION_EXTRA_MOUNTS=/var/spool/slurmd:/var/spool/slurmd:bind:bind,rw,nosuid,noexec,nodev,private;/var/lib/hugetlbfs:/var/lib/hugetlbfs:bind:rbind,rw,nosuid,nodev,private",
      "INJECTION_EXTRA_ENV=MPIR_CVAR_CH4_OFI_MULTI_NIC_STRIPING_THRESHOLD=100000000",
      "INJECTION_COMPATIBILITY=major"
    ]
  },
  "when": {
    "annotations": {
      "com.hooks.cxi.enabled": "^true$"
    }
  },
  "stages": ["precreate"]
}

Catch hook log with wrapper:

#!/bin/sh
exec /users/felipecr/.local/libexec/performance-extensions/pc_injection_hook \
  2>>/users/felipecr/.local/state/pc_injection_hook.log

ENV injection is here:

podman --hooks-dir="$HOME/.config/containers/oci/hooks.d" run --rm   --annotation com.hooks.cxi.enabled=true   quay.io/madeeks/osu-mb:6.2-mpich4.1-ubuntu22.04-arm64   env

PATH=/usr/local/libexec/osu-micro-benchmarks:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
container=podman
HOME=/root
HOSTNAME=781ec64501e8
LD_LIBRARY_PATH=/run/pc-injection/libcxi.so.1:/run/pc-injection/libcurl.so.4:/run/pc-injection/libjson-c.so.3:/run/pc-injection/libatomic.so.1:/run/pc-injection/libnl-3.so.200:/run/pc-injection/libnghttp2.so.14:/run/pc-injection/libidn2.so.0:/run/pc-injection/libssh.so.4:/run/pc-injection/libpsl.so.5:/run/pc-injection/libssl.so.1.1:/run/pc-injection/libcrypto.so.1.1:/run/pc-injection/libgssapi_krb5.so.2:/run/pc-injection/libldap_r-2.4.so.2:/run/pc-injection/liblber-2.4.so.2:/run/pc-injection/libzstd.so.1:/run/pc-injection/libbrotlidec.so.1:/run/pc-injection/libz.so.1:/run/pc-injection/libunistring.so.2:/run/pc-injection/libjitterentropy.so.3:/run/pc-injection/libkrb5.so.3:/run/pc-injection/libk5crypto.so.3:/run/pc-injection/libcom_err.so.2:/run/pc-injection/libkrb5support.so.0:/run/pc-injection/libsasl2.so.3:/run/pc-injection/libbrotlicommon.so.1:/run/pc-injection/libkeyutils.so.1:/run/pc-injection/libpcre.so.1:/run/pc-injection/libselinux.so.1:/run/pc-injection/libxpmem.so.0
MPIR_CVAR_CH4_OFI_MULTI_NIC_STRIPING_THRESHOLD=100000000

Mount dir logic is here:

podman --hooks-dir="$HOME/.config/containers/oci/hooks.d" run --rm   --annotation com.hooks.cxi.enabled=true   quay.io/madeeks/osu-mb:6.2-mpich4.1-ubuntu22.04-arm64     sh -lc 'echo "$LD_LIBRARY_PATH"; find /run/pc-injection -maxdepth 3 -type f 2>/dev/null'
/run/pc-injection/libcxi.so.1:/run/pc-injection/libcurl.so.4:/run/pc-injection/libjson-c.so.3:/run/pc-injection/libatomic.so.1:/run/pc-injection/libnl-3.so.200:/run/pc-injection/libnghttp2.so.14:/run/pc-injection/libidn2.so.0:/run/pc-injection/libssh.so.4:/run/pc-injection/libpsl.so.5:/run/pc-injection/libssl.so.1.1:/run/pc-injection/libcrypto.so.1.1:/run/pc-injection/libgssapi_krb5.so.2:/run/pc-injection/libldap_r-2.4.so.2:/run/pc-injection/liblber-2.4.so.2:/run/pc-injection/libzstd.so.1:/run/pc-injection/libbrotlidec.so.1:/run/pc-injection/libz.so.1:/run/pc-injection/libunistring.so.2:/run/pc-injection/libjitterentropy.so.3:/run/pc-injection/libkrb5.so.3:/run/pc-injection/libk5crypto.so.3:/run/pc-injection/libcom_err.so.2:/run/pc-injection/libkrb5support.so.0:/run/pc-injection/libsasl2.so.3:/run/pc-injection/libbrotlicommon.so.1:/run/pc-injection/libkeyutils.so.1:/run/pc-injection/libpcre.so.1:/run/pc-injection/libselinux.so.1:/run/pc-injection/libxpmem.so.0

fcruzcscs · 2026-04-28T23:46:50Z

Note: currently we pass new libs via LD_LIBRARY_PATH, it could be done via an ld cache refresh, but it would require to update the ldcache hook to enter the container namespace before start (instead of manipulating the bundle as it currently does), as otherwise, the new envs and mounts are not available to the hook (as it runs in host)

fcruzcscs · 2026-04-29T23:14:44Z

srun --mpi=pmix -A csstaff -N 2 \
  podman --hooks-dir "$HOOKS_DIR" --module hpc run --rm --pull=never -e PMIX_MCA_psec=^munge \
  --device /dev/cxi0 \
  --device /dev/cxi1 \
  --device /dev/cxi2 \
  --device /dev/cxi3 \
  -v /opt/cray:/opt/cray:ro \
 --env-host --device nvidia.com/gpu=all --annotation com.hooks.cxi.enabled=true "$IMAGE" ./pt2pt/osu_bw

# OSU MPI Bandwidth Test v7.5.2
# Datatype: MPI_CHAR.
# Size      Bandwidth (MB/s)
1                       0.32
2                       0.64
4                       1.30
8                       2.60
16                      5.20
32                     10.45
64                     20.95
128                    41.86
256                    82.63
512                   165.50
1024                  330.19
2048                  662.32
4096                 1323.76
8192                 2581.93
16384                5166.29
32768               10150.70
65536               20336.51
131072              22672.89
262144              23642.69
524288              23959.16
1048576             24121.08
2097152             24204.53
4194304             24239.91

CXI config works :)

fcruzcscs added 14 commits April 27, 2026 13:29

rustfmt alignment

df50f45

new vagrant test env for hooks

f2c3781

v1 single lib replace

cf5fac7

adding pc_injection to workspace

491ae42

tests for injection v1

e89c8da

primary replaces all hits, dependencies go over LD_LIB_PATH

22e8807

updated warning for mismatched libs

66547f5

simplification of mount handling

4a34da4

rust format fix

bddceef

added new options to match network injection capability

1065c65

test covers new options

26f1c1f

test helper handles deps as options, cleaner

503ba4e

minor detail tweak to readme

83aac09

tweak to readme

6874481

fcruzcscs requested a review from Madeeks April 29, 2026 08:33

fix primary lib contraint & real lib injection

e0f4181

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PreCreate Injection Hook#7

PreCreate Injection Hook#7
fcruzcscs wants to merge 15 commits intomainfrom
pc_injection_n_testing

fcruzcscs commented Apr 28, 2026 •

edited

Loading

Uh oh!

fcruzcscs commented Apr 28, 2026

Uh oh!

fcruzcscs commented Apr 28, 2026 •

edited

Loading

Uh oh!

fcruzcscs commented Apr 28, 2026

Uh oh!

fcruzcscs commented Apr 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

fcruzcscs commented Apr 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fcruzcscs commented Apr 28, 2026

Uh oh!

fcruzcscs commented Apr 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fcruzcscs commented Apr 28, 2026

Uh oh!

fcruzcscs commented Apr 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

fcruzcscs commented Apr 28, 2026 •

edited

Loading

fcruzcscs commented Apr 28, 2026 •

edited

Loading