CBG-5220: Freezable cluster compat version

[bbrks]

CBG-5220

Add ability to freeze current cluster compatibility version to "pin" a cluster to a given version and avoid rolling CCV forwards.
Allows for supported downgrades/rollbacks even if all nodes in the cluster have been upgraded but are pinned behind the frozen version.

REST API changes

• GET /_cluster_compat_version returns the cluster-wide version, per-node versions, and the frozen value if set.
• POST /_cluster_compat_version/freeze pins the version to the current value to preserve rollback capability across upgrades.
• POST /_cluster_compat_version/unfreeze clears the frozen version.

Implementation details

The trickier half of the change is making the existing ClusterCompatVersionHWM ratchet behave correctly in the presence of a freeze. Because HWM is monotonic and gates downgrades, advancing it off transient state would permanently
lock the cluster above the frozen version.

This implementation:

1. defers the HWM ratchet from synchronous startup registration until after StartOnlineProcesses
2. gates each per-bucket ratchet on at least one database in that bucket having reached DBOnline (computed once per Refresh to avoid re-acquiring _databasesLock)
3. clamps HWM to the freeze version inside RegisterNodeVersion so the single CAS-checked write is the only place HWM can move.

Both Freeze and Unfreeze are "success-on-all" (at the bucket level - since there can be multiple registries for a given cluster) - a 503 is returned in case some bucket ops fail (whilst some succeed) and it's expected that the operation is retried until successful.

Diagrams

Freeze and unfreeze admin flow

Both admin endpoints fan out across every tracked bucket, persist a CAS-checked mutation into each bucket's _sync:registry document.

  sequenceDiagram
      participant Admin
      participant Handler as handler_cluster_compat
      participant Mgr as clusterCompatManager
      participant Boot as bootstrapContext
      participant Bkts as Tracked buckets (registry docs)

      Admin->>Handler: POST /freeze
      Handler->>Mgr: Freeze
      Mgr->>Mgr: RLock and snapshot cachedVersion
      loop each tracked bucket
          Mgr->>Boot: SetRegistryFreeze(bucket, version)
          Boot->>Bkts: CAS write Frozen{version, FrozenAt}
          Boot-->>Mgr: freeze record
      end
      Mgr->>Mgr: mergeFreeze aggregate, update cachedFreeze and cachedVersion
      Mgr-->>Handler: aggregate
      Handler-->>Admin: 200 ClusterCompatVersionState

      Admin->>Handler: POST /unfreeze
      Handler->>Mgr: Unfreeze
      Mgr->>Mgr: snapshot previousFreeze
      loop each tracked bucket
          Mgr->>Boot: ClearRegistryFreeze(bucket)
          Boot->>Bkts: CAS write removing Frozen
      end
      Mgr->>Mgr: cachedFreeze = nil, recompute cachedVersion
      Mgr-->>Handler: previousFreeze
      Handler-->>Admin: 200

Loading

HWM ratchet under a freeze

ClusterCompatVersionHWM is the monotonic high-water mark each bucket has ever observed its live-node minimum reach, and it's the field the downgrade guardrail uses. Once HWM is advanced, the gate permanently refuses any node whose version is below it.
The freeze acts as a per-bucket cap on HWM incrementing, which is why the freeze operation has to be success-on-all. A bucket that didn't get the freeze record skips the clamp and keeps ratcheting HWM forward on the next Refresh tick, and HWM can't be pulled back.

  flowchart TD
      A[Each Refresh tick, per bucket] --> B{Bucket has a freeze?}
      B -- yes --> C[HWM capped at freeze version]
      B -- no --> D[HWM ratchets up to live-node minimum]
      C --> E[Per-bucket registry write]
      D --> E
      E -.later.-> F{Incoming node version below HWM?}
      F -- yes --> G[Rejected — rollback no longer possible]
      F -- no --> H[Accepted]

Loading

Dependencies (if applicable)

• CBG-5266: Prevent downgrades across cluster compatibility versions #8235

Integration Tests

• https://jenkins.sgwdev.com/job/SyncGatewayIntegration/646/

[github-actions]

Redocly previews

• Admin API
• Public API
• Metric API
• Diagnostic API

[Copilot]

Pull request overview

This PR introduces an admin-controlled “freeze” mechanism for the Sync Gateway cluster compatibility version (CCV), intended to keep CCV from advancing during upgrades so rollbacks/downgrades remain possible.

Changes:

• Adds new admin endpoints to read CCV state and to freeze/unfreeze it, including audit events.
• Persists a freeze record into each bucket’s registry document and updates CCV computation to account for the frozen value.
• Extends OpenAPI documentation and adds unit + audit REST tests for the new behavior.

Reviewed changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
rest/routing.go	Registers new admin routes for cluster compat version read/freeze/unfreeze.
rest/handler_cluster_compat.go	Implements REST handlers and response payload for CCV state + freeze/unfreeze actions.
rest/config_registry.go	Extends the persisted registry document schema with a freeze record pointer.
rest/config_manager.go	Adds CAS-retry helpers to set/clear the freeze record in a bucket registry.
rest/cluster_compat.go	Adds freeze-aware CCV computation and manager Freeze/Unfreeze operations.
rest/cluster_compat_test.go	Adds unit tests for freeze/unfreeze and pinning behavior.
rest/cluster_compat_audit_test.go	Adds end-to-end REST + audit emission coverage for the new endpoints.
docs/api/paths/admin/_cluster_compat_version.yaml	Documents the new GET endpoint.
docs/api/paths/admin/_cluster_compat_version-freeze.yaml	Documents the new freeze endpoint.
docs/api/paths/admin/_cluster_compat_version-unfreeze.yaml	Documents the new unfreeze endpoint.
docs/api/components/schemas.yaml	Adds new OpenAPI schemas and updates GatewayRegistry schema with freeze record.
docs/api/admin.yaml	Wires new paths into the admin OpenAPI spec.
base/version_cluster_compat.go	Adds the RegistryFreeze type used to persist the freeze record.
base/audit_events.go	Adds new audit IDs/events for CCV read/freeze/unfreeze.

[Copilot]

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated 3 comments.

[gregns1]

Looks good just one small question on a test assertion then should be good to go