lagerdata · danielrmerskine · Jul 13, 2026 · Jul 13, 2026 · Jul 13, 2026 · gemini-code-assist
@@ -2,6 +2,47 @@
 
 All notable changes to the Lager platform are documented here. For detailed release notes, see [docs.lagerdata.com](https://docs.lagerdata.com).
 
+## [0.31.7] - 2026-07-13
+
+`lager install` can no longer leave a box with a Docker daemon that will not start,
+and it stops rather than deploying into one that isn't running.
+
+### Fixed
+- **`lager install` no longer writes a DNS server Docker cannot parse into
+  `/etc/docker/daemon.json`.** The "Configuring Docker DNS" step reads the box's
+  upstream resolvers from systemd-resolved and merges them into `daemon.json`,
+  but excluded only the `127.0.0.53` stub — every other value was trusted to be a
+  usable IP. On a network that advertises DNS over IPv6 router advertisement,
+  systemd-resolved records a link-local resolver carrying a zone id
+  (`fe80::1%3`). Docker validates each `dns` entry as a bare IP address and
+  **refuses to start** when one does not parse — it does not skip the entry or fall
+  back to the valid resolvers beside it. Because `daemon.json` is persistent, the
+  daemon then stayed down across reboots, and re-running the installer rewrote the
+  file each time, undoing any manual repair. Resolvers are now validated before
+  they are written: link-local, loopback, unspecified, multicast and unparseable
+  values are dropped (a link-local address is unreachable from a container's
+  network namespace regardless), and anything dropped is named in the install log.
+- **A failed Docker DNS change is rolled back instead of left in place.** Pointing
+  Docker at the box's resolvers is an optimization; it must not be able to leave the
+  box worse off. `daemon.json` is now backed up before the change, and if Docker
+  will not come back with the new config, the previous file is restored and Docker
+  restarted on it. A box that cannot be improved is no longer a box that gets broken.
+- **The installer stops when the box's Docker daemon is down, instead of running six
+  more steps against it.** Every Docker command in the container step is
+  best-effort (`|| true`), so a dead daemon left no trace until `start_box.sh` failed
+  on `docker network create` with a bare "Cannot connect to the Docker daemon" —
+  far from whatever actually stopped it, and pointing at the wrong thing. The step
+  now checks the daemon first and fails with the commands needed to diagnose it.
+  Likewise, a DNS step that fails now reports that the box kept its previous Docker
+  configuration and that the install is continuing, rather than a bare warning.
+
+### Changed
+- The deployment script's Docker DNS logic moved out of an inline heredoc into
+  `cli/deployment/scripts/configure_docker_dns.{sh,py}`, shipped to the box like
+  `secure_box_firewall.sh` already is, and is now covered by unit tests.
+- The install step counter reports `[N/8]`; there were eight steps and a total of
+  seven, so the last one printed as `[8/7]`.
+
 ## [0.31.6] - 2026-07-13
 
 Help pages now share one usage grammar across every net-style command, and the box

@@ -7,4 +7,4 @@
 A Command Line Interface for Lager Data
 """
 
-__version__ = '0.31.6'
+__version__ = '0.31.7'
@@ -0,0 +1,125 @@
+#!/usr/bin/env python3
+# Copyright 2024-2026 Lager Data
+# SPDX-License-Identifier: Apache-2.0
+
+"""Choose usable upstream resolvers for Docker's container DNS.
+
+Runs on the box under the system python3, so stdlib only.
+
+Docker validates every entry of daemon.json's "dns" list as a bare IP address and
+refuses to start -- not warn, not skip the entry -- if one of them does not parse.
+A resolver learned over IPv6 router advertisement is a link-local address carrying
+a zone id (``fe80::1%3``), which does not parse. Writing one out therefore takes
+the daemon down, and it stays down across reboots, because daemon.json is
+persistent.
+
+Link-local resolvers are dropped rather than stripped of their zone: a container's
+network namespace cannot reach the host's link-local scope, so such an address is
+useless to the containers this list exists to serve.
+"""
+
+import argparse
+import ipaddress
+import json
+import os
+
+FALLBACKS = ("1.1.1.1", "8.8.8.8")
+MAX_UPSTREAM = 3
+DEFAULT_RESOLV_CONF = "/run/systemd/resolve/resolv.conf"
+DEFAULT_DAEMON_JSON = "/etc/docker/daemon.json"
+
+
+def rejection_reason(value):
+    """Why `value` is unusable as a Docker DNS server, or None if it is usable."""
+    address = value.split("%", 1)[0]
+    try:
+        ip = ipaddress.ip_address(address)
+    except ValueError:
+        return "not an IP address"
+    if ip.is_loopback:
+        return "loopback"
+    if ip.is_link_local:
+        return "link-local"
+    if ip.is_unspecified:
+        return "unspecified"
+    if ip.is_multicast:
+        return "multicast"
+    return None
+
+
+def _nameservers(text):
+    for line in text.splitlines():
+        fields = line.split()
+        if len(fields) >= 2 and fields[0] == "nameserver":
+            yield fields[1]
+
+
+def usable_resolvers(text, max_upstream=MAX_UPSTREAM):
+    """Usable resolvers from resolv.conf `text`, in order, deduplicated."""
+    servers = []
+    for value in _nameservers(text):
+        if rejection_reason(value) or value in servers:
+            continue
+        servers.append(value)
+        if len(servers) == max_upstream:
+            break
+    return servers
+
+
+def rejected_resolvers(text):
+    """(value, reason) pairs for every nameserver we refuse to hand to Docker."""
+    return [(v, rejection_reason(v)) for v in _nameservers(text) if rejection_reason(v)]
+
+
+def merge_dns(cfg, servers):
+    """Return `cfg` with "dns" set, preserving every operator-set key."""
+    dns = list(servers)
+    for fallback in FALLBACKS:
+        if fallback not in dns:
+            dns.append(fallback)
+    merged = dict(cfg)
+    merged["dns"] = dns
+    return merged
+
+
+def load_daemon_json(path):
+    """Existing daemon.json, or {} if it is absent, unreadable, or not an object."""
+    if not os.path.exists(path):
+        return {}
+    try:
+        with open(path) as fh:
+            cfg = json.load(fh)
+    except (OSError, ValueError):
+        # Docker cannot parse it either, so there is nothing worth preserving.
+        return {}
-def load_daemon_json(path):
-    """Existing daemon.json, or {} if it is absent or unreadable."""
-    if not os.path.exists(path):
-        return {}
-    try:
-        with open(path) as fh:
-            return json.load(fh) or {}
-    except (OSError, ValueError):
-        # Docker cannot parse it either, so there is nothing worth preserving.
-        return {}
+def load_daemon_json(path):
+    """Existing daemon.json, or {} if it is absent or unreadable."""
+    if not os.path.exists(path):
+        return {}
+    try:
+        with open(path) as fh:
+            data = json.load(fh)
+            return data if isinstance(data, dict) else {}
+    except (OSError, ValueError):
+        # Docker cannot parse it either, so there is nothing worth preserving.
+        return {}
-def load_daemon_json(path):
-    """Existing daemon.json, or {} if it is absent or unreadable."""
-    if not os.path.exists(path):
-        return {}
-    try:
-        with open(path) as fh:
-            return json.load(fh) or {}
-    except (OSError, ValueError):
-        # Docker cannot parse it either, so there is nothing worth preserving.
-        return {}
+def load_daemon_json(path):
+    """Existing daemon.json, or {} if it is absent or unreadable."""
+    if not os.path.exists(path):
+        return {}
+    try:
+        with open(path) as fh:
+            data = json.load(fh)
+            return data if isinstance(data, dict) else {}
+    except (OSError, ValueError):
+        # Docker cannot parse it either, so there is nothing worth preserving.
+        return {}
+    # Valid JSON that isn't an object (a list, say) is not a config Docker can use,
+    # and would break the merge. Same treatment as unparseable.
+    return cfg if isinstance(cfg, dict) else {}
+
+
+def main():
+    parser = argparse.ArgumentParser(description="Build Docker's daemon.json DNS config")
+    parser.add_argument("--resolv-conf", default=DEFAULT_RESOLV_CONF)
+    parser.add_argument("--daemon-json", default=DEFAULT_DAEMON_JSON)
+    parser.add_argument("--out", required=True, help="where to write the merged daemon.json")
+    args = parser.parse_args()
+
+    try:
+        with open(args.resolv_conf) as fh:
+            resolv = fh.read()
+    except OSError:
+        resolv = ""
+
+    cfg = merge_dns(load_daemon_json(args.daemon_json), usable_resolvers(resolv))
+
+    with open(args.out, "w") as fh:
+        json.dump(cfg, fh, indent=2)
+        fh.write("\n")
+
+    for value, reason in rejected_resolvers(resolv):
+        print("Ignoring unusable resolver {} ({})".format(value, reason))
+    print("Docker container DNS ->", ", ".join(cfg["dns"]))
+
+
+if __name__ == "__main__":
+    main()
@@ -0,0 +1,85 @@
+#!/bin/bash
+# Copyright 2024-2026 Lager Data
+# SPDX-License-Identifier: Apache-2.0
+#
+# Point Docker's container DNS at the box's real uplink resolvers. Runs on the box.
+#
+# On a systemd-resolved box, /etc/resolv.conf lists only the 127.0.0.53 stub, which
+# a container cannot use, so Docker falls back to 8.8.8.8; where that resolver is
+# unreachable, `docker build` cannot resolve github.com/pypi and the image build
+# dies partway through. systemd-resolved's own resolv.conf names the real upstream
+# servers, so we merge those into daemon.json. configure_docker_dns.py decides
+# which of them Docker can actually use.
+#
+# This is an optimization, so it must never leave the box worse off than it found
+# it: if Docker will not come back with the new config, the previous daemon.json is
+# restored and Docker is restarted on it before we exit non-zero.
+#
+# Every privileged action here is one the box's sudoers file already grants
+# NOPASSWD (see setup_and_deploy_box.sh): `install` from the fixed path
+# /tmp/lager_daemon.json, and `systemctl restart docker`. Staging anywhere else --
+# a mktemp path, say -- silently falls outside the grant and makes every run prompt
+# for a password. Snapshot and restore therefore route through that same path, and
+# the backup lives in /tmp, where no privileges are needed to write it.
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+DAEMON_JSON="${DAEMON_JSON:-/etc/docker/daemon.json}"
+RESOLV_CONF="${RESOLV_CONF:-/run/systemd/resolve/resolv.conf}"
+STAGED="${STAGED:-/tmp/lager_daemon.json}"
+BACKUP="${BACKUP:-/tmp/lager_daemon.json.bak}"
+
+rm -f "$STAGED" "$BACKUP" 2>/dev/null || true
+trap 'rm -f "$STAGED" "$BACKUP" 2>/dev/null || true' EXIT
+
+python3 "${SCRIPT_DIR}/configure_docker_dns.py" \
+    --resolv-conf "$RESOLV_CONF" --daemon-json "$DAEMON_JSON" --out "$STAGED"
+
+# Snapshot the current config so a failed restart can be undone. daemon.json is
+# world-readable, so this needs no sudo. Track whether there was one at all.
+if [ -f "$DAEMON_JSON" ]; then
+    HAD_CONFIG=1
+    cp -f "$DAEMON_JSON" "$BACKUP"
+else
+    HAD_CONFIG=0
+fi
+
+daemon_is_up() {
+    # `systemctl is-active` reads the unit's own state: unlike `docker info` it
+    # needs neither root nor docker-group membership, which a freshly added user
+    # may not have picked up in this SSH session yet.
+    local _
+    for _ in 1 2 3 4 5; do
+        if systemctl is-active --quiet docker; then
+            return 0
+        fi
+        sleep 1
+    done
+    return 1
+}
+
+restore_previous() {
+    # Where there was no daemon.json, an empty object restores Docker's defaults --
+    # which is what the absent file meant. We write that rather than removing the
+    # file because `rm` on /etc/docker is not in the sudoers grant, and a recovery
+    # path that stops to ask for a password is a recovery path that does not run.
+    if [ "$HAD_CONFIG" -eq 1 ]; then
+        cp -f "$BACKUP" "$STAGED"
+    else
+        echo '{}' > "$STAGED"
+    fi
+    sudo install -m 0644 "$STAGED" "$DAEMON_JSON" \
+        || echo "WARNING: could not restore ${DAEMON_JSON}" >&2
+    sudo systemctl restart docker || true
+}
-restore_previous() {
-    if [ "$HAD_CONFIG" -eq 1 ]; then
-        sudo install -m 0644 "$BACKUP" "$DAEMON_JSON"
-    else
-        sudo rm -f "$DAEMON_JSON"
-    fi
-    sudo systemctl restart docker || true
-}
+restore_previous() {
+    if [ "$HAD_CONFIG" -eq 1 ]; then
+        cp -f "$BACKUP" "$STAGED"
+        sudo install -m 0644 "$STAGED" "$DAEMON_JSON"
+    else
+        echo "{}" > "$STAGED"
+        sudo install -m 0644 "$STAGED" "$DAEMON_JSON"
+    fi
+    sudo systemctl restart docker || true
+}
-restore_previous() {
-    if [ "$HAD_CONFIG" -eq 1 ]; then
-        sudo install -m 0644 "$BACKUP" "$DAEMON_JSON"
-    else
-        sudo rm -f "$DAEMON_JSON"
-    fi
-    sudo systemctl restart docker || true
-}
+restore_previous() {
+    if [ "$HAD_CONFIG" -eq 1 ]; then
+        cp -f "$BACKUP" "$STAGED"
+        sudo install -m 0644 "$STAGED" "$DAEMON_JSON"
+    else
+        echo "{}" > "$STAGED"
+        sudo install -m 0644 "$STAGED" "$DAEMON_JSON"
+    fi
+    sudo systemctl restart docker || true
+}
+
+sudo install -m 0644 "$STAGED" "$DAEMON_JSON"
+
+if ! sudo systemctl restart docker || ! daemon_is_up; then
+    restore_previous
+    echo "ERROR: Docker would not start with the new DNS configuration." >&2
+    echo "       Restored the previous ${DAEMON_JSON} and restarted Docker." >&2
+    exit 1
+fi
+
+echo "Docker restarted with the new DNS configuration"
@@ -202,7 +202,7 @@ echo -e "${BLUE}Time:${NC}    $(date '+%Y-%m-%d %H:%M:%S')"
 echo ""
 
 # Step counter
-TOTAL_STEPS=7
+TOTAL_STEPS=8
 CURRENT_STEP=0
 
 print_step() {
@@ -722,54 +722,20 @@ fi
 # resolver is blocked or flaky, `docker build` can't resolve github.com/pypi
 # and the image build dies on its git/pip clone steps. Point Docker at the
 # box's real uplink resolvers (with public fallbacks) so builds resolve
-# reliably. Requires a docker restart to take effect.
+# reliably. Requires a docker restart to take effect; configure_docker_dns.sh
+# rolls the change back if Docker won't come up with it.
 print_step "Configuring Docker DNS"
 print_info "Pointing Docker's container DNS at the box's real uplink resolvers (you may be prompted for a password)..."
 echo ""
 
-TEMP_DNS_SCRIPT=$(mktemp)
-cat > "$TEMP_DNS_SCRIPT" << SCRIPT_EOF
-#!/bin/bash
-set -e
-# Real upstream resolvers systemd-resolved talks to (its own resolv.conf lists
-# the actual servers, not the 127.0.0.53 stub). Drop the loopback stub if it
-# appears, and cap at three.
-UPSTREAM=\$(awk '/^nameserver/ && \$2 != "127.0.0.53" { print \$2 }' /run/systemd/resolve/resolv.conf 2>/dev/null | head -3 | tr '\n' ' ')
-# Merge a "dns" key into any existing daemon.json (python3 is always present on
-# a lager box) rather than clobbering operator-set options.
-python3 - "\$UPSTREAM" <<'PYEOF'
-import json, os, sys
-servers = [s for s in sys.argv[1].split() if s]
-for fallback in ("1.1.1.1", "8.8.8.8"):
-    if fallback not in servers:
-        servers.append(fallback)
-path = "/etc/docker/daemon.json"
-cfg = {}
-if os.path.exists(path):
-    try:
-        with open(path) as fh:
-            cfg = json.load(fh) or {}
-    except Exception:
-        cfg = {}
-cfg["dns"] = servers
-with open("/tmp/lager_daemon.json", "w") as fh:
-    json.dump(cfg, fh, indent=2)
-    fh.write("\n")
-print("Docker container DNS ->", ", ".join(servers))
-PYEOF
-sudo install -m 0644 /tmp/lager_daemon.json /etc/docker/daemon.json
-rm -f /tmp/lager_daemon.json
-sudo systemctl restart docker
-SCRIPT_EOF
-
-scp $SCP_OPTS "$TEMP_DNS_SCRIPT" "${BOX_USER}@${BOX_IP}:/tmp/setup_docker_dns.sh" >/dev/null
-if ssh_t "${BOX_USER}@${BOX_IP}" "chmod +x /tmp/setup_docker_dns.sh && /tmp/setup_docker_dns.sh && rm /tmp/setup_docker_dns.sh"; then
+scp $SCP_OPTS "${SCRIPT_DIR}/configure_docker_dns.sh" "${SCRIPT_DIR}/configure_docker_dns.py" "${BOX_USER}@${BOX_IP}:/tmp/" >/dev/null
+if ssh_t "${BOX_USER}@${BOX_IP}" "chmod +x /tmp/configure_docker_dns.sh && /tmp/configure_docker_dns.sh; rc=\$?; rm -f /tmp/configure_docker_dns.sh /tmp/configure_docker_dns.py; exit \$rc"; then
     print_success "Docker DNS configured"
 else
-    print_warning "Could not configure Docker DNS automatically"
-    print_info "If image builds fail with 'Could not resolve host', set \"dns\" in /etc/docker/daemon.json on the box and restart docker."
+    print_warning "Could not configure Docker DNS - the box kept its previous Docker configuration"
+    print_info "Docker is still running and the install will continue. If image builds later fail"
+    print_info "with 'Could not resolve host', set \"dns\" in /etc/docker/daemon.json on the box."
 fi
-rm "$TEMP_DNS_SCRIPT"
 echo ""
 
 # Ensure /etc/lager directory exists (always check, even if sudo was already configured)
@@ -1402,6 +1368,25 @@ if [ -n "$VPN_INTERFACE" ]; then
     echo ""
 fi
 
+# Every docker command in this step is best-effort (`|| true`), so a daemon that is
+# down leaves no trace here and start_box.sh is the first thing to notice -- failing
+# on `docker network create` with a bare "Cannot connect to the Docker daemon", well
+# after whatever actually stopped it. Check explicitly, while the cause is still near.
+print_info "Verifying the Docker daemon is running..."
+if ! ssh $SSH_OPTS "${BOX_USER}@${BOX_IP}" "docker info >/dev/null 2>&1"; then
+    print_error "The Docker daemon is not running on the box"
+    echo ""
+    echo "  Nothing can be deployed until it starts. On the box:"
+    echo "    sudo systemctl status docker"
+    echo "    sudo journalctl -u docker -n 50 --no-pager"
+    echo ""
+    echo "  A daemon that refuses to start usually has a bad /etc/docker/daemon.json."
+    echo ""
+    exit 1
+fi
+print_success "Docker daemon is running"
+echo ""
+
 print_info "Building and starting containers..."
 echo ""
 ssh $SSH_OPTS "${BOX_USER}@${BOX_IP}" "cd ~/box && chmod +x start_box.sh && ./start_box.sh"

@@ -211,6 +211,7 @@
           {
             "group": "Version History",
             "pages": [
+              "source/release-notes/v0.31.7",
               "source/release-notes/v0.31.6",
               "source/release-notes/v0.31.5",
               "source/release-notes/v0.31.4",