fix(observability): a single stuck coroot-db replica blocks every platform merge

[devantler]

🤖 Generated by the Daily AI Engineer

Problem

A single degraded replica in the coroot-db observability database silently blocked every
platform PR from merging for ~4 hours today, and degraded prod HA. The failure is systemic, not a
one-off.

The chain:

1. A CNPG switchover left the demoted old primary (coroot-db-7) unable to rejoin: pg_rewind found
   the servers diverged on timeline 8 and could not restore the WAL segment it needed from the archive.
2. CNPG retried pg_rewind forever — 4+ hours, zero restarts, no self-healing and no re-clone. The
   pod stayed Running but never Ready, so the Cluster sat in Waiting for the instances to become active (2 of 3 ready).
3. The infrastructure Flux Kustomization health-checks that Cluster, so it failed with
   health check failed after 20m: timeout waiting for: [Cluster/observability/coroot-db].
4. 🚀 Deploy to Prod therefore failed in every merge_group run, so every platform PR was
   evicted from the merge queue (observed on fix(backstage): give the startup probe a real budget so it survives a restart #2636).

The important part: the database was serving fine the whole time — primary healthy, one healthy
replica, ContinuousArchiving=True, LastBackupSucceeded=True. A non-critical, still-serving
observability database with one lagging replica took down the entire merge lane.

I unwedged prod by hand (deleted the diverged replica's PVC + pod so CNPG re-cloned it from the
primary; the cluster is back to 3/3 healthy and the merge queue is moving). That is a manual remedy,
not a fix.

Proposed direction

Two independent defects, either of which alone would have prevented this:

1. A degraded-but-serving observability DB must not gate the merge queue. Deploy health gating
   should not hang for 20 minutes on a replica that has no bearing on whether the deploy is good.
   Either scope the infrastructure health checks away from non-critical stateful replicas, or make
   this class alert instead of block.
2. A replica that cannot pg_rewind should be re-cloned, not retried forever. Retrying a rewind
   that can never succeed (the WAL segment is not in the archive and never will be) is an infinite
   loop. Options: ensure WAL needed for rewind is retained (HA replication slots / WAL retention), or
   detect the unrecoverable-rewind condition and re-bootstrap the instance.

Also worth noting for the runbook: this switchover was very likely triggered by the rolling Talos node
reboots performed during the GHCR-credential remediation earlier today. After a rolling node reboot,
stateful workloads must be verified re-converged — a reboot can leave a demoted primary permanently
stuck, and nothing alerts on it.

Rough size

M — (1) is a manifest/health-check scoping change; (2) is a CNPG configuration or operator-behaviour
question that may need an upstream conversation.

[coderabbitai]

🔗 Related PRs

#2330 - ci: heal prod when a merge-group deploy fails (restore main) [merged]
#2424 - docs(dr): add CNPG StorageClass migration runbook (longhorn → longhorn-wffc) [merged]
#2425 - fix(cnpg): move platform Postgres clusters to the longhorn-wffc StorageClass [merged]
#2578 - fix(policy-reporter): run all components at the HA replica floor [merged]
#2622 - fix(ci): heal cancelled merge-group deploys [merged]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!