Skip to content

fix(dind): translate sibling bind sources to runner snapshot upperdir#82

Merged
luthermonson merged 2 commits into
mainfrom
fix/dind-bind-mount-translate
May 28, 2026
Merged

fix(dind): translate sibling bind sources to runner snapshot upperdir#82
luthermonson merged 2 commits into
mainfrom
fix/dind-bind-mount-translate

Conversation

@luthermonson
Copy link
Copy Markdown
Contributor

Problem

GitHub Actions workflows that use the container: directive fail
immediately on ephemerd self-hosted runners:

##[command]/usr/bin/docker create --name ... \
  -v /home/runner/_work/_temp:/__w/_temp \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v /home/runner/_work:/__w \
  ...
##[command]/usr/bin/docker start <id>
sh: 0: cannot open /__w/_temp/<uuid>.sh: No such file
##[error]Process completed with exit code 2.

The dind shim accepted every -v in the API request but silently
dropped any bind whose source didn't os.Stat on the dind daemon's
filesystem. Because the sources arrive from inside the runner
container's mount namespace, the dind daemon (running outside that
namespace) saw none of them. Every bind was dropped. The sibling
container started but its _temp mountpoint was empty, so the step's
docker exec sh -e /__w/_temp/<uuid>.sh failed.

This broke every workflow using container: — ephpm, Anthropic-style
workflows, anything wanting a reproducible toolchain image.

Fix

A container-to-container bind translation layer. The runtime registers
the runner snapshot key plus the non-rootfs bind table (e.g.
/var/run/docker.sock → the per-job dind socket file) with the dind
server right after NewContainer succeeds. When a sibling-create
request arrives, each -v source resolves against:

  1. The runner's bind table (longest-prefix match).
  2. The runner snapshot's overlayfs upperdir — returned rw.
  3. The runner snapshot's lowerdirs — returned ro (image layers are
    shared across jobs; a rw mount on top would corrupt the cache).
  4. Otherwise: HTTP 400 with bind mount X -> Y rejected: source not visible to ephemerd dind. Loud failure replaces the previous silent
    drop.

Security envelope

Siblings can only see what the runner could already see. Bind table
entries are paths ephemerd itself installed into the runner; snapshot
upperdir/lowerdir entries are inside the runner's rootfs. There is no
code path that resolves attacker-supplied sources against the real host
filesystem — the silent-drop bug accidentally provided this property
and the loud-fail fix preserves it explicitly. See the arch doc.

Lifecycle

pkg/runtime.Destroy calls env.Dind.Stop() (which kills every
sibling and drops the dind namespace) before
container.Delete(WithSnapshotCleanup) removes the runner snapshot.
Siblings are gone before the upperdir disappears — no stale mounts in
normal teardown.

Scope

  • Linux-only for v1. The goruntime.GOOS != "windows" guard skips
    registration only on Windows-native runner code paths.
    Linux-on-Windows jobs run inside a Hyper-V Linux VM where the in-VM
    ephemerd process is Linux and takes the registration branch
    normally.
  • Windows-native container: is deferred — different snapshotter
    (windowsfilter), different bind semantics.
  • Snapshot lease extension was considered for sibling-outlives-runner
    scenarios; rejected because ephemerd's teardown order makes that
    impossible.

Tests

pkg/dind/bindtranslate_test.go:

  • 9 pure-function tests for translateBindSource.
  • 3 integration tests for buildBindMounts including the full 8-bind
    set from a real ephpm failure log — asserts docker.sock translation,
    _temp lands in upperdir rw, externals lands in lowerdir ro.

pkg/dind/bindtranslate_e2e_test.go:

  • Real overlayfs snapshotter via shared embedded containerd.
  • Plant _temp/marker.sh in the actual upperdir, register the snapshot
    with dind, translate, then os.ReadFile the marker through the
    translated source path. Proves what we hand containerd points at the
    right bytes on disk.
  • Foreign-source rejection guard for the loud-failure contract.

Test plan

  • CI green (lint, unit, e2e).
  • Build Windows binary, deploy to host ephemerd service.
  • Re-run a failing ephpm workflow that uses container: and
    observe actions/checkout + step scripts run inside the sibling
    container successfully.

Docs

  • docs/arch/dind-bind-translation.md — problem, two-container model,
    resolution policy, security envelope, lifecycle, wiring, Windows
    reasoning, deferred follow-ups.

The dind shim used to silently drop any `-v` source that didn't os.Stat on
the daemon's filesystem. GHA `container:` workflows ask for sources inside
the runner container's mount namespace (/home/runner/_work/_temp/<uuid>.sh),
which the dind daemon can't see — so every bind was dropped, the sibling
started without the script directory, and `docker exec sh -e
/__w/_temp/<uuid>.sh` failed with "cannot open".

Resolve each requested source against (1) the non-rootfs bind table
ephemerd installed into the runner (/var/run/docker.sock and friends), then
(2) the runner snapshot's overlayfs upperdir (rw — runner-written workspace
files), then (3) the lowerdirs (ro — image layers, shared across jobs so
writes would corrupt the cache). Anything that doesn't match returns 400
with a clear "bind mount X -> Y rejected" message instead of being
quietly dropped.

Linux-only for v1; Windows-native jobs use a different snapshotter and
mount model and are deferred. Architecture doc at
docs/arch/dind-bind-translation.md.

Closes the GHA `container:` regression for jobs running on ephemerd
self-hosted runners.
Prepare(ctx, key, "") with an empty parent returns a plain bind mount,
not an overlay — so the e2e never saw an upperdir= option to extract.
Stage an empty committed parent first, then Prepare on top, so the
active snapshot is a real overlayfs mount with upperdir/lowerdir layout
the translator can walk.

Also fix the lease-delete callback to set the runtime namespace before
calling the LeasesService, which was failing on cleanup with
"namespace is required".
@luthermonson luthermonson merged commit dab8a46 into main May 28, 2026
4 checks passed
@luthermonson luthermonson deleted the fix/dind-bind-mount-translate branch May 28, 2026 03:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant