Skip to content

Roll out R2 multipart fix to remaining CNPG clusters (PG 16 / 17 operands) #2

Description

@alleneubank

Problem

The uniform-multipart fix (#1 has review context) is only built as an overlay on the PG 18 CNPG operand (ghcr.io/0xsend/cnpg-postgresql:18-r2-*) and is deployed to just the two misfits clusters. Every other CNPG cluster backs up to the same Cloudflare R2 endpoint with stock barman and remains exposed to intermittent InvalidPart failures on CompleteMultipartUpload (EnterpriseDB#954).

Confirmed second victim: pqs-testnet/pqs-postgres (stock ghcr.io/cloudnative-pg/postgresql:17.2) failed its 2026-06-10 18:00 UTC scheduled backup with the exact signature:

ERROR: Upload error: An error occurred (InvalidPart) when calling the
CompleteMultipartUpload operation: One or more of the specified parts could not be found.

It self-recovered on the next run — the failures are racy and intermittent, which is why small clusters rarely trip it (few/no multipart parts) while large ones (misfits, pqs) do.

Exposure inventory (2026-06-11)

Cluster Namespace Image Status
misfits-pg18 misfits-postgres-{testnet,mainnet} fork 18-r2-* ✅ fixed
pqs-postgres pqs-testnet stock 17.2 ❌ confirmed hit 2026-06-10
bridge-pg18 bridge-{testnet,mainnet} stock 18 exposed
famo-pg18 featured-app-marker-orchestrator-{testnet,mainnet} stock 18 exposed
cos-db cos-{dev,prod} stock 17.2 exposed
metabase-postgres metabase-mainnet stock 17.2 exposed
analytics-postgres pqs-{testnet,mainnet} (image not pinned in status) exposed
scan-data-postgres pqs-testnet (image not pinned in status) exposed
pool-archive-postgres pool-archive-postgres-testnet stock 16 exposed

All 14 clusters share R2 endpoint 649b42...r2.cloudflarestorage.com.

Plan

  • Parameterize the overlay build (.github/workflows/cnpg-image.yml + Dockerfile) by base operand tag, producing 16-r2-*, 17-r2-* (and keeping 18-r2-*) variants from the same fork commit.
  • Roll out per app chart (each needs the ghcr-credentials imagePullSecret + namespace ghcr-enabled label, same pattern as misfits — canton-monorepo #5452 / #5461).
  • Prioritize by dataset size: pqs-postgres (confirmed) first, then analytics-postgres, then the small ones opportunistically.
  • Retire all of this once the fix is upstreamed (tracked in Follow-ups from code review of R2 uniform-multipart patch #1) or when migrating to the CNPG 1.30 barman-cloud plugin.

Interim safety

Backup failures are visible: CNPGBackupFailing (failure newer than last success) plus the freshness alerts CNPGBackupStale/CNPGBackupCritical (24h/72h) cover silent gaps, and backups retry on the next 6-hourly schedule.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions