Add transport URL secret rotation with consumer finalizer#716
Conversation
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: lmiccini The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
lmiccini
force-pushed
the
finalize-secret-rotation
branch
from
b6d036f to
b27f224
Compare
|
/test keystone-operator-build-deploy-kuttl |
lmiccini
force-pushed
the
finalize-secret-rotation
branch
from
b27f224 to
37e535d
Compare
|
Build failed (check pipeline). Post ❌ openstack-k8s-operators-content-provider FAILURE in 7m 36s |
lmiccini
force-pushed
the
finalize-secret-rotation
branch
from
37e535d to
b956f42
Compare
When infra-operator rotates a RabbitMQ transport URL (creating a new secret and user), consumer operators must hold a consumer finalizer on the old secret until all their pods have rolled out with the new credentials. Without this, infra-operator cleans up the old RabbitMQ user while pods are still connected with old credentials, causing message bus outages. Design: 1. Add consumer finalizer to the current transport URL secret early in reconcile. Set instance.Status.TransportURLSecret for first-time setup only (when empty or unchanged); during rotation the status is updated solely by FinalizeSecretRotation at end of reconcile. 2. Pass transportURL.Status.SecretName directly to sub-CR creation functions and config generation as a parameter — never read from instance.Status.TransportURLSecret for sub-CR specs. 3. Use statefulset.IsReady() / deployment.IsReady() from lib-common in all sub-CR controllers for accurate rollout status. 4. Use object.ManageRotationGracePeriod() to enforce a 60-second grace period before evaluating the rotation guard. This gives sub-CRs time to detect config changes, update their workloads, and roll pods — without relying on informer cache freshness. 5. Guard: CredentialRotationGuardReady(true, conditions) — evaluates AllSubConditionIsTrue after the grace period expires. Only when all sub-CR conditions are True does FinalizeSecretRotation remove the consumer finalizer from the old secret. The same pattern applies to notification transport URL secrets and application credential secrets where applicable. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
lmiccini
force-pushed
the
finalize-secret-rotation
branch
from
b956f42 to
78e0ae1
Compare
1 participant
When infra-operator rotates a RabbitMQ transport URL (creating a new secret and user), consumer operators must hold a consumer finalizer on the old secret until all their pods have rolled out with the new credentials. Without this, infra-operator cleans up the old RabbitMQ user while pods are still connected with old credentials, causing message bus outages.
Design:
Add consumer finalizer to the current transport URL secret early in reconcile. Set instance.Status.TransportURLSecret for first-time setup only (when empty or unchanged); during rotation the status is updated solely by FinalizeSecretRotation at end of reconcile.
Pass transportURL.Status.SecretName directly to sub-CR creation functions and config generation as a parameter — never read from instance.Status.TransportURLSecret for sub-CR specs.
Use statefulset.IsReady() / deployment.IsReady() from lib-common in all sub-CR controllers for accurate rollout status.
Use object.ManageRotationGracePeriod() to enforce a 60-second grace period before evaluating the rotation guard. This gives sub-CRs time to detect config changes, update their workloads, and roll pods — without relying on informer cache freshness.
Guard: CredentialRotationGuardReady(true, conditions) — evaluates AllSubConditionIsTrue after the grace period expires. Only when all sub-CR conditions are True does FinalizeSecretRotation remove the consumer finalizer from the old secret.
The same pattern applies to notification transport URL secrets and application credential secrets where applicable.