feat(up): add --auto-forward for dynamic user-space port forwarding (daemon)

[pofallon]

Summary

Add a deacon up --auto-forward flag that enables dynamic, user-space port forwarding for the lifetime of a running devcontainer, modeled on how VS Code Dev Containers forwards ports. Unlike the existing static -p publishing (forwardPorts / --forward-port, fixed at container-create time), this forwards ports after the container is up, auto-detects ports as they start listening, reaches loopback-only services that -p cannot, and runs without the user having to keep a foreground terminal or background a process themselves.

It must work for multiple concurrent deacon up --auto-forward invocations (multiple devcontainers on the same host), allocating non-colliding host ports across all of them.

Motivation

How VS Code does it (the model we're emulating)

VS Code does not use Docker -p/--publish for its port forwarding. It runs a persistent agent inside the container that periodically scans the container's listening sockets (/proc/net/tcp{,6}), and for each port opens a host-side 127.0.0.1:N listener that relays bytes, in user space, over an existing connection into the container's network namespace. This is why it is dynamic (ports appear/disappear after launch), reaches 127.0.0.1-bound servers, requires no firewall / root / republish, and remaps to a free local port when the natural one is taken.

What deacon does today (and its limitation)

deacon already consumes forwardPorts / appPort / --forward-port and turns them into docker run -p publish args at container-create time (crates/core/src/docker.rs:470, crates/deacon/src/commands/up/ports.rs). This is static:

• Ports must be known before the container is created; you cannot add one later without recreating the container.
• It cannot reach a server bound to 127.0.0.1 inside the container (only 0.0.0.0).
• It does not auto-detect a server that a postStart hook, the entrypoint, a compose CMD, or an interactive exec session starts later.

--auto-forward closes that gap with the VS Code-style dynamic model.

Why a daemon, and why on up

Dynamic forwarding requires a host-side process alive for the lifetime of the forwards, plus on-demand byte pipes into the container netns. We explored anchoring this on exec (piggyback on the foreground session), but that only covers interactive workflows and cannot forward a server that starts at container boot with no shell attached. Modeling forwarding as a property of the running container — a detached daemon started by up --auto-forward and reaped on down — covers both workflows with a single switch:

• Headless: up --auto-forward, a server boots via entrypoint/postStart/compose, user hits localhost:3000 in a browser, never opens a shell.
• Interactive: plain deacon exec zsh (no new flag) starts npm run dev; the already-running daemon sees the new listening socket and forwards it. exec needs no changes — detection is socket-based and indifferent to who opened the port, and an exec'd process shares the container netns.

Goals

• deacon up --auto-forward starts dynamic user-space forwarding for the container and returns control to the shell (no occupied terminal, no manual &).
• Auto-detect listening ports inside the container and forward them to 127.0.0.1 on the host, including 127.0.0.1-bound container services.
• Honor declared intent: forwardPorts, --forward-port, appPort, and portsAttributes / otherPortsAttributes.
• Support multiple devcontainers concurrently on one host with collision-free host-port allocation and clear reporting of the actual localhost port per container.
• Reap the daemon on down (and on up --remove-existing-container), and self-exit if the container disappears out-of-band. No orphans.
• Loopback-only host binding; no new host-shell-from-workspace exec surface.

Non-Goals

• Replacing the existing static -p publishing when --auto-forward is absent (unchanged, backward compatible).
• Exposing forwarded ports on 0.0.0.0 / the LAN (loopback only for v1).
• A deacon forward standalone subcommand or an exec --auto-forward flag (explicitly out — the daemon on up covers both workflows; a post-hoc attach affordance is a possible future follow-up, not this issue).
• Remote-tunnel / Azure-style forwarding (local relay only).
• Feature-authoring or any non-consumer surface.

Proposed CLI surface

deacon up --auto-forward

• --auto-forward is a boolean flag on the up subcommand (Commands::Up, crates/deacon/src/cli.rs:225; threaded via UpArgs, crates/deacon/src/commands/up/args.rs:371).

Naming

The flag is --auto-forward, not --forward. --forward was rejected because it is too easily confused with the existing --forward-port (cli.rs:354) — they read like singular/plural of one another, yet they are different mechanisms (static -p publish vs dynamic daemon) that also interact (see below), so near-identical names would actively mislead. auto-forward instead names the distinguishing behavior (automatic/dynamic detection) and aligns with existing spec vocabulary (portsAttributes.onAutoForward), so the pairing reads correctly: "--forward-port declares a port; --auto-forward watches and forwards them dynamically."

Relationship to existing flags (must be specified precisely)

• The existing --forward-port <spec> (singular, repeatable, cli.rs:354) and config forwardPorts continue to mean "declared ports."
• When --auto-forward is set, declared ports are forwarded by the daemon (loopback relay) and are NOT also -p published — otherwise the daemon and -p would both try to bind the same host port. The daemon set = declared ports ∪ auto-detected ports.
• When --auto-forward is absent, behavior is unchanged: declared ports → docker run -p (static), no daemon.
• Optional future extension (note, not required for v1): a value form such as --auto-forward=declared (declared ports only, no auto-detection) vs the default --auto-forward (declared ∪ auto-detect).

Design

Components / suggested module layout

• crates/core/src/port_forward/ (new):
  • detect.rs — parse /proc/net/tcp{,6} LISTEN entries (pure logic, unit-testable).
  • registry.rs — host-global port allocation registry (pure logic + atomic file IO).
  • daemon.rs — the supervisor loop (detect → reconcile → relay), lifecycle, self-exit.
  • relay.rs — per-connection byte relay over docker exec.
• crates/deacon/src/commands/up/forward.rs (new) — wiring --auto-forward into the up flow, daemon spawn/detach, marker management.
• down.rs — reap hook.

1. Port detection (socket scan)

• Poll the container on an interval (default ~1s; configurable) via docker exec <id> cat /proc/net/tcp /proc/net/tcp6 using the existing ContainerRuntime::exec with ExecConfig { silent: true, .. } (piped stdout — crates/core/src/docker.rs:660 trait; silent=true path uses Stdio::piped() around ~1554).
• Parse st == 0A (LISTEN) rows; collect local ports bound to 0.0.0.0, 127.0.0.1, ::, ::1. Dedupe across v4/v6.
• This is process-agnostic by design: it catches entrypoint/postStart/compose servers and servers started inside an exec session (shared netns), which is what lets exec stay unmodified.
• Respect portsAttributes[port].onAutoForward (config.rs:567): at minimum honor ignore/silent/notify; openBrowser optional for v1. otherPortsAttributes (config.rs:573) supplies the default for undeclared auto-detected ports.

2. Relay (byte pipe into the container netns)

• For each forwarded (container_port → host_port), open a host listener on 127.0.0.1:host_port.
• On each accepted host connection, spawn a docker exec -i <id> <relay> that dials 127.0.0.1:container_port inside the container and pump bytes bidirectionally (per-connection relay for v1 — simplest; note overhead and a persistent-multiplexing-agent optimization as future work).
• Relay program decision (open question — see below): prefer a tiny embedded static relay binary docker cp'd into the container on first use (no container-side deps, works on alpine and distroless); fall back to socat / nc / bash /dev/tcp only if present. Do not silently assume socat exists (fail-fast per project principle if no relay strategy is available, with a clear error).
• Works for 127.0.0.1-bound servers precisely because docker exec shares the container network namespace — the selling point over -p.

3. Multi-container support — host-global port allocation (KEY REQUIREMENT)

Per-container daemons are naturally namespaced by container id (each up produces a distinct ContainerIdentity = workspace_hash + config_hash, crates/core/src/container.rs:33,111), so two up --auto-forward invocations spawn two independent daemons with no contention on the daemon itself.

The shared, host-global resource is host port numbers: two devcontainers each running a server on container port 3000 cannot both bind host 127.0.0.1:3000. Required design:

• Host-global registry file, e.g. {user_data_folder}/forwarded_ports.json (the same user_data_folder the trust store uses — NOT --container-data-folder, which may be per-workspace). Entries:

  { "host_port": 3001, "container_id": "abc123", "container_port": 3000,
    "workspace": "/path/...", "pid": 12345, "label": "web" }

• Allocation policy (VS Code-like): prefer the same number (container 3000 → host 3000) when free; if the registry or an actual bind shows it taken, pick the next free port and record the mapping. Always surface the actual localhost port, especially when remapped.
• Atomic + locked updates: serialize to a unique temp file then fs::rename (per the repo's durable-write pattern, cache/disk.rs::save_index), under a file lock around the allocate-and-bind critical section so two concurrent up --auto-forward cannot race onto the same host port.
• Stale reaping: on daemon start and on allocation, prune registry entries whose pid is dead or whose container no longer exists.
• Release: on daemon shutdown / down, remove that container's entries from the registry, freeing the host ports for reuse.

4. Daemon lifecycle contract

• Adopt / single-owner per container: marker {container_data_folder|user_data_folder}/forward_daemon_{container_id}.pid (mirrors the env-probe cache key pattern, container_env_probe.rs:155). If a live daemon for this container_id already exists, reuse it (a second up --auto-forward for the same container does not spawn a duplicate).
• Detach: the daemon must survive up returning to the shell — reparent via double-fork on Unix (platform equivalent on Windows; document the Windows path as a separate task). It is not left as a child of the foreground up.
• Logging: detached daemon has no terminal — write a per-container log to {container_data_folder|user_data_folder}/forward_daemon_{container_id}.log.
• Reap on down: down (crates/deacon/src/commands/down.rs) resolves the container(s) by workspace_hash / labels, reads the pid marker, SIGTERMs the daemon, waits, then removes marker + releases registry ports.
• Reap on replace: up --remove-existing-container kills the old daemon (markers are cleared on container replace per existing reset semantics) before creating the new container/daemon.
• Self-exit: the daemon polls container liveness (relay execs failing / docker inspect) and exits + cleans up when the container is gone out-of-band (e.g. docker rm -f).

5. exec transparency

No changes to exec. The daemon forwards ports opened inside an exec session automatically because they are just new listening sockets in the shared container netns. Document this explicitly so it is not "re-discovered."

6. Trust / security

• The daemon execs into the container (docker exec), which is standard consumer behavior (the exec subcommand already does this ungated). It does not introduce a new host-shell-from-workspace exec site, so the workspace-trust gate (crates/core/src/trust.rs) is not required for the exec itself.
• The genuinely new host surface is (a) a persistent host process and (b) binding host ports that expose container services. Mitigations: bind 127.0.0.1 only (never 0.0.0.0), and document the surface in SECURITY.md. Flag for security review whether any consent/opt-in beyond the explicit --auto-forward flag is warranted.

7. Output / events

• Integrate with the existing --ports-events flag (cli.rs:351) / PortForwardingManager events: emit forward/unforward events as ports come and go.
• Print human-readable mappings to stderr (per the Output Streams Contract), e.g. Forwarding container 3000 -> http://127.0.0.1:3001 (remapped; 3000 in use). The up result JSON on stdout is unaffected.

Open questions / design decisions to resolve in PR

1. Relay program: embedded static relay binary (docker cp) vs socat/nc//dev/tcp fallback. Recommendation: embedded binary primary, fallback secondary, fail-fast if neither.
2. Per-connection docker exec vs persistent in-container multiplexing agent. Recommendation: per-connection for v1; note the agent optimization as follow-up.
3. --auto-forward semantics vs static -p: confirm declared ports move to the daemon (no -p) when --auto-forward is set (recommended) vs supplement.
4. Poll interval / netlink: fixed poll for v1; consider inotify/netlink later.
5. Windows detach + signaling path (double-fork is Unix-only).
6. Does appPort join the daemon set? Recommendation: yes, treated like a declared port.

Task breakdown

• T1 detect.rs: parse /proc/net/tcp{,6} LISTEN entries → Vec<u16> (pure, unit-tested with fixtures).
• T2 registry.rs: host-global allocation registry with atomic temp+rename writes, file lock, stale reaping, release (pure logic + IO; unit-tested).
• T3 relay.rs: per-connection byte relay over docker exec -i; relay-program selection (embedded binary primary, socat/nc fallback, fail-fast).
• T4 daemon.rs: supervisor loop (detect → allocate → bind → relay → reconcile on changes), portsAttributes handling, structured logging to per-container log file.
• T5 Detach + marker/lock lifecycle (adopt-or-reuse, double-fork on Unix); Windows path tracked as sub-task.
• T6 up wiring: --auto-forward flag on Commands::Up + UpArgs; when set, route declared ports to daemon and suppress -p for them; spawn/adopt daemon after container is healthy.
• T7 down reap hook: SIGTERM daemon by pid marker, wait, remove marker, release registry ports. Also handle up --remove-existing-container.
• T8 Events/output: emit --ports-events, print loopback mappings to stderr.
• T9 SECURITY.md note + security review of the host-port/persistent-process surface.
• T10 Docs: docs/subcommand-specs/up/SPEC.md, README, an examples/ canary (exec.sh) demonstrating dynamic + multi-container forwarding.
• T11 Tests (below) + nextest group config in all three .config/nextest.toml spots per CLAUDE.md.

Testing plan

• Unit (hermetic): /proc/net/tcp parser (fixtures incl. v6 + 127.0.0.1-bound); registry allocation/collision/release/stale-reap; marker adopt-or-reuse logic.
• Integration (docker):
  • up --auto-forward against a container running a server on 127.0.0.1:PORT; assert the host 127.0.0.1:host_port is reachable (proves loopback reach -p can't do).
  • Dynamic: start a server after up --auto-forward (via exec); assert it becomes reachable without any exec flag.
  • Multi-container: two up --auto-forward on two workspaces, both with a server on container 3000; assert two distinct host ports allocated and both reachable; assert registry has both entries; down one and assert its host port is released and the other still works.
  • down reaps the daemon (pid gone, marker removed, port released); up --remove-existing-container kills the old daemon.
• Grouping: multi-container/daemon tests likely docker-exclusive (serial); single-container reachability may be docker-shared. Add the new test binary to all three nextest profile spots (default override filter, dev-fast default-filter exclusion, dev-fast override filter) per CLAUDE.md.

Acceptance criteria

1. deacon up --auto-forward returns control to the shell with forwarding active (no occupied terminal, no manual backgrounding).
2. A server bound to 127.0.0.1:P inside the container is reachable at 127.0.0.1:host_port on the host.
3. A port that starts listening after up (entrypoint/postStart/compose/exec) is auto-forwarded with no exec changes.
4. Two concurrent deacon up --auto-forward for different containers each running container 3000 get distinct, non-colliding host ports, both reachable, both reported.
5. down (and up --remove-existing-container) reaps the daemon and releases its host ports; no orphan process or leaked registry entries; daemon self-exits if the container vanishes out-of-band.
6. Host binds are 127.0.0.1-only; security surface documented in SECURITY.md.
7. With --auto-forward absent, behavior is unchanged (static -p publishing).

Relevant code references

• up args: crates/deacon/src/cli.rs:225 (Commands::Up, existing --forward-port at :354, --ports-events at :351); crates/deacon/src/commands/up/args.rs:371 (UpArgs).
• Existing static publishing: crates/core/src/docker.rs:470; crates/deacon/src/commands/up/ports.rs; crates/core/src/ports.rs (PortForwardingManager).
• Config port fields: crates/core/src/config.rs:554 (forward_ports: Vec<PortSpec>), :560 (app_port), :567 (ports_attributes), :573 (other_ports_attributes).
• Exec primitive: crates/core/src/docker.rs:660 (ContainerRuntime::exec trait), silent=true piped path ~:1554.
• Cache/marker pattern: crates/core/src/container_env_probe.rs:155; atomic-write pattern crates/core/src/cache/disk.rs::save_index.
• Identity/labels: crates/core/src/container.rs:33,100,111.
• Trust: crates/core/src/trust.rs (check_workspace_trust, resolve_policy, decision_to_result).
• Down/reap: crates/deacon/src/commands/down.rs.