v0.31.7

[danielrmerskine]

Summary

lager install could leave a box with a Docker daemon that would not start, then run six more steps against it and report a misleading error. This fixes the cause, makes the change reversible, and stops the installer from deploying into a dead daemon.

The "Configuring Docker DNS" step (added in 0.20.1) reads the box's upstream resolvers from systemd-resolved and merges them into /etc/docker/daemon.json so image builds resolve reliably. It excluded exactly one value:

awk '/^nameserver/ && $2 != "127.0.0.53" { print $2 }'

Everything else was trusted to be a usable IP. On a network that advertises DNS over IPv6 router advertisement, systemd-resolved records a link-local resolver carrying a zone id — fe80::1%3. Docker validates every dns entry as a bare IP address and refuses to start when one does not parse; it does not skip the entry or fall back to the valid resolvers written beside it. Because daemon.json is persistent, the daemon then stayed down across reboots, and every re-run of the installer rewrote the file, undoing any manual repair.

It has not surfaced before because it requires an IPv6-RA network; it is not specific to old boxes, and reproduces on a current release against a freshly installed one.

Changes

• Validate resolvers before writing them. New cli/deployment/scripts/configure_docker_dns.py drops link-local, loopback, unspecified, multicast and unparseable values, and names anything it dropped in the install log. Link-local addresses are dropped rather than stripped of their zone: a container's network namespace cannot reach the host's link-local scope, so they are useless here regardless.
• Roll back a DNS change that doesn't take. New cli/deployment/scripts/configure_docker_dns.sh backs up daemon.json, and if Docker will not come back with the new config, restores the previous file and restarts Docker on it before exiting non-zero. This step is an optimization and must not be able to leave a box worse off than it found it.
• Stop when the daemon is down. Every Docker command in the container step is best-effort (|| true), so a dead daemon left no trace until start_box.sh failed on docker network create with a bare "Cannot connect to the Docker daemon" — far from the actual cause. The step now checks the daemon first and prints the commands needed to diagnose it.
• The DNS logic moved out of an inline heredoc (which interpolated nothing) into the two files above, shipped to the box like secure_box_firewall.sh already is, and is now testable.
• TOTAL_STEPS 7 → 8; the last step printed as [8/7].

Tests

test/unit/cli/test_configure_docker_dns.py — 22 cases over the filter, including the exact resolv.conf shape that took a daemon down, both zone-id forms, and the invariant that every emitted server parses as an IP address (i.e. Docker will accept it).

test/unit/cli/test_configure_docker_dns_rollback.py — drives the shell script with sudo/systemctl/docker stubbed on PATH, asserting a failed restart restores the previous daemon.json byte-for-byte, and removes ours when there was no config to begin with.

711 unit tests pass. (test/unit/box/test_box_http_server_capabilities.py fails to collect in a full-suite run on main as well — a pre-existing lager package-name collision, unrelated.)

Hardware validation

On a box, against its real Debian python3 (3.12.3): fed a synthetic resolv.conf containing 192.168.100.1, fe80::1%3 and 127.0.0.53, the filter emits 192.168.100.1, 1.1.1.1, 8.8.8.8 and logs both rejections with reasons. This is the important one — the failing condition cannot be reproduced on an IPv4 network, so it is exercised directly.

Full lager install on a box: DNS step reports the box's real resolvers unchanged from what the previous release produced (no regression), no warning, no rollback, daemon healthy, box comes up, step counter reads [3/8].

The one-line resolver filter at the heart of this was also used, hot-patched, to recover the box that hit this in the field.

[gemini-code-assist]

Code Review

This pull request extracts the Docker DNS configuration logic from an inline heredoc in setup_and_deploy_box.sh into standalone scripts (configure_docker_dns.sh and configure_docker_dns.py), adds validation to filter out unusable resolvers (such as link-local IPv6 addresses), implements a rollback mechanism if Docker fails to restart, and introduces comprehensive unit tests. The reviewer feedback highlights critical permission issues with the rollback script on boxes without full passwordless sudo, as the current sudo cp and sudo rm commands are not whitelisted. To resolve this, the reviewer suggests copying the world-readable configuration without sudo, overwriting the configuration with an empty JSON object {} instead of deleting it during rollback, and updating the corresponding tests and type safety checks.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

The script uses sudo cp and sudo rm on /etc/docker/daemon.json.lager-bak, and a dynamic temporary file path via mktemp for $STAGED. However, the passwordless sudoers rules defined in setup_and_deploy_box.sh only permit /usr/bin/install for the specific path /tmp/lager_daemon.json, and do not permit cp or rm on /etc/docker/. This will cause the script to prompt for a password or fail on boxes without full passwordless sudo. Since /etc/docker/daemon.json is world-readable, we can copy it to /tmp/ without sudo, and use the whitelisted /tmp/lager_daemon.json path for $STAGED to keep the entire process passwordless.

DAEMON_JSON="${DAEMON_JSON:-/etc/docker/daemon.json}"
RESOLV_CONF="${RESOLV_CONF:-/run/systemd/resolve/resolv.conf}"
STAGED="${STAGED:-/tmp/lager_daemon.json}"
BACKUP="${BACKUP:-/tmp/lager_daemon.json.bak}"

# Clean up any stale files from previous runs
rm -f "$STAGED" "$BACKUP"
trap 'rm -f "$STAGED" "$BACKUP"' EXIT

python3 "${SCRIPT_DIR}/configure_docker_dns.py" \
    --resolv-conf "$RESOLV_CONF" --daemon-json "$DAEMON_JSON" --out "$STAGED"

# Snapshot the current config so a failed restart can be undone. Track whether
# there was one at all, so we restore the old file or write an empty config accordingly.
# Since /etc/docker/daemon.json is world-readable, we can copy it to /tmp without sudo.
if [ -f "$DAEMON_JSON" ]; then
    HAD_CONFIG=1
    cp -f "$DAEMON_JSON" "$BACKUP"
else
    HAD_CONFIG=0
fi

[gemini-code-assist]

To maintain passwordless execution, we should avoid running sudo rm on /etc/docker/daemon.json (which is not whitelisted in sudoers). Instead, if there was no previous config, we can overwrite it with an empty JSON object {} using the whitelisted /tmp/lager_daemon.json path.

Suggested change

	restore_previous() {
	if [ "$HAD_CONFIG" -eq 1 ]; then
	sudo install -m 0644 "$BACKUP" "$DAEMON_JSON"
	else
	sudo rm -f "$DAEMON_JSON"
	fi
	sudo systemctl restart docker || true
	}
	restore_previous() {
	if [ "$HAD_CONFIG" -eq 1 ]; then
	cp -f "$BACKUP" "$STAGED"
	sudo install -m 0644 "$STAGED" "$DAEMON_JSON"
	else
	echo "{}" > "$STAGED"
	sudo install -m 0644 "$STAGED" "$DAEMON_JSON"
	fi
	sudo systemctl restart docker || true
	}

[gemini-code-assist]

The load_daemon_json function currently returns json.load(fh) or {}. If the JSON file contains a valid JSON array or other non-object types, this can return a list or string, which will cause merge_dns to raise a TypeError or ValueError when calling dict(cfg). It is safer to explicitly verify that the loaded JSON is a dictionary.

Suggested change

	def load_daemon_json(path):
	"""Existing daemon.json, or {} if it is absent or unreadable."""
	if not os.path.exists(path):
	return {}
	try:
	with open(path) as fh:
	return json.load(fh) or {}
	except (OSError, ValueError):
	# Docker cannot parse it either, so there is nothing worth preserving.
	return {}
	def load_daemon_json(path):
	"""Existing daemon.json, or {} if it is absent or unreadable."""
	if not os.path.exists(path):
	return {}
	try:
	with open(path) as fh:
	data = json.load(fh)
	return data if isinstance(data, dict) else {}
	except (OSError, ValueError):
	# Docker cannot parse it either, so there is nothing worth preserving.
	return {}

[gemini-code-assist]

Update the test environment to pass whitelisted STAGED and BACKUP paths to the rollback script so it runs cleanly in the test's temporary directory.

    env = dict(os.environ)
    env.update(
        PATH="{}:{}".format(bin_dir, env["PATH"]),
        DAEMON_JSON=str(daemon_json),
        RESOLV_CONF=str(tmp_path / "resolv.conf"),
        STAGED=str(tmp_path / "lager_daemon.json"),
        BACKUP=str(tmp_path / "lager_daemon.json.bak"),
        RESTART_RC=str(restart_rc),
    )

[gemini-code-assist]

Since we now restore to an empty JSON object {} instead of deleting the file (to avoid non-whitelisted sudo rm commands), update the test assertion to verify that the config is correctly overwritten with {}.

Suggested change

	def test_failed_restart_removes_config_that_did_not_exist_before(box):
	"""Rolling back to "no daemon.json" means removing ours, not restoring an empty one."""
	result, daemon_json = _run(box, restart_rc=1, daemon_json_exists=False)
	assert result.returncode == 1
	assert not daemon_json.exists()
	def test_failed_restart_removes_config_that_did_not_exist_before(box):
	"""Rolling back to "no daemon.json" means restoring an empty config."""
	result, daemon_json = _run(box, restart_rc=1, daemon_json_exists=False)
	assert result.returncode == 1
	assert daemon_json.exists()
	assert json.loads(daemon_json.read_text()) == {}