Production droplet has drifted from ansible-managed state — reconcile or stop pretending

[espadonne]

Background

shithub-prod was hand-built. Several PRs since launch have updated ansible roles or scripts that should live on the droplet but were never actually deployed there because nobody runs the ansible play. We keep getting bitten by the same pattern: PR lands → repo is correct → droplet is stale → next operator (or scheduled task) hits a silent failure.

Concrete drift caught so far

• PR fix(rclone): single config at /etc/rclone-shithub.conf, postgres-readable #21 updated `backup-daily.sh` and `sync-cross-region.sh` to use `/etc/rclone-shithub.conf`. The repo got the fix; the droplet kept the old path `/root/.config/rclone/rclone.conf`. Daily Spaces upload silently failed for ~2 weeks. Local dumps and WAL archive masked it.
• PR feat(monitoring): replace promtail with Grafana Alloy → Cloud free tier #28 added the `monitoring-client` ansible role (Alloy + node_exporter). Role works but never ran against the droplet — had to install everything by hand over SSH.
• PR fix(monitoring): alloy enable_compression=false + manual install runbook fallback #32 committed `enable_compression = false` in the alloy template. Live droplet has the fix because we manually edited `config.alloy`, but the next `apt upgrade alloy` or systemd reload won't reapply ansible state.
• PR web: bypass Compress middleware on /metrics (closes #33) #34 moves `/metrics` out of the Compress middleware in shithubd. This deploys via the binary deploy pipeline (works), but related ansible role changes don't.

Why this keeps happening

• No `inventory/production` file exists anywhere — Mac, droplet, or repo (correctly gitignored).
• Ansible isn't installed locally.
• The droplet was built before the roles were written, so even one-time `ansible-playbook` runs would risk clobbering hand-tuned state (sshd config, postgres tuning, the operator git user).
• Every change ships via `ssh + scp + systemctl` instead.

Proposal

Pick one of the three honest paths. The current state is the worst of both worlds — roles exist as documentation but aren't authoritative.

Option A — make ansible authoritative

Build the inventory file, install ansible locally (or in CI), do a dry-run `--check --diff` against the droplet to surface the diff, reconcile by hand where needed, then commit to running the play after every merge. Long-term right answer.

Cost: significant up-front work to make the play idempotent against the hand-built state without breaking anything. Risk of clobbering during reconciliation.

Option B — drop ansible, document as bash + scp

Convert each role into a sequence of `ssh "..."` commands in a deploy script (`deploy/cutover/deploy-droplet.sh`). Operator runs it after merging anything that touches droplet state. Roles get archived for "this is what we'd do for a fresh droplet" reference.

Cost: less ceremony, no ansible dep. Loses some idempotence guarantees.

Option C (this issue) — short-term reconciliation pass

Independent of A/B, audit every file the roles claim to manage against what's on the droplet right now. Write a one-shot script that pushes the deltas. Mark it as a stopgap, not a permanent answer.

Then pick A or B as a follow-up.

Acceptance for this issue

• Audit script that lists every file the ansible roles install/template, diffs each against the live droplet, reports drift. Read-only.
• Apply fixes for current drift (manual scp/ssh is fine, document each).
• Decide A or B as the long-term direction; file follow-up issue.
• Add a CI check (or make target) that fails if a PR touches `deploy/ansible/` without a matching note in the PR body about how it was deployed.

Tackling now

Per the linked PR/conversation, this issue is being addressed in the immediately-following session pass rather than deferred.

[espadonne]

Pass 1 — short-term reconciliation (acceptance bullet 1 + 2)

Done in PR #40:

• ✅ Audit script committed at `deploy/audit/check-droplet-drift.sh` (read-only, single ssh round-trip).
• ✅ Initial drift fixed in this pass:
  • `shithub-backup-daily` — was using `/root/.config/rclone/rclone.conf` (PR fix(rclone): single config at /etc/rclone-shithub.conf, postgres-readable #21 fix never deployed). Daily Spaces upload had been silently failing for ~2 weeks. Now restored; verified by manual run, fresh dump landed in `spaces-prod:shithub-backups/daily/2026/05/10/`.
  • `shithub-spaces-sync` — same path issue, same fix.
  • `shithub-ssh-authkeys`, `git-shell-commands/shithubd` — comment-only drift (functionally identical), aligned to repo for cleanliness.
  • `/etc/aide/aide.conf.d/99_shithub_exclude` — was missing PR fix(aide): exclude postgres + sysstat logs from baseline #15's `/var/log/postgresql` + `/var/log/sysstat` exclusions, causing nightly false alarms. Updated. Operator needs to re-baseline aide via the runbook (`sudo aideinit && mv /var/lib/aide/aide.db.new.gz /var/lib/aide/aide.db.gz`) at their discretion.

Pass 2 — long-term decision (acceptance bullet 3)

Not done. Need to pick A (make ansible authoritative) or B (drop ansible, document as bash + scp). My recommendation: B for now, revisit in 6 months.

Rationale:

• We don't run ansible. We manually scp + ssh. The scripts are short and the operator (mfwolffe) is comfortable on the shell.
• The droplet was hand-built; reconciling it to a clean ansible state risks clobbering hand-tuned bits (sshd, Postgres tuning, the git user).
• Adding CI ssh access has its own security tradeoffs.
• The audit script in PR audit: read-only droplet-drift checker (issue #38) #40 catches the actual symptom (drift between repo and droplet) without requiring us to commit to either A or B.

If/when the operational footprint grows past one droplet, revisit and pick A.

Pass 3 — CI check (acceptance bullet 4)

Not done in this pass. Suggested approach when we do:

• `Makefile` target `make check-droplet-drift` that runs the audit script.
• A GitHub Action that runs it on PRs touching `deploy/ansible/`. Requires a deploy SSH key in CI secrets. Would also need to handle the case where the droplet's known_hosts isn't pre-populated.

Filing as a sub-issue is overkill at this scale; reopen this issue when we want to do it.