[18.0-fr6] Fix galera bootstrap deadlock when all pods are killed simultaneously#500
Merged
openshift-merge-bot[bot] merged 1 commit intoJul 7, 2026
Conversation
Three issues combined to delay galera cluster recovery by 12-54+ minutes when all pods were killed at once. 1. Status update race: injectGcommURI() stored gcomm state in Status.Attributes. The deferred PatchInstance (JSON merge patch) wrote this back, overwriting pod-pushed attributes (including ContainerIDs) with stale data from the start of the reconcile. Fix: stop writing gcomm state to Status.Attributes entirely. GaleraAttributes is now read-only from the operator's perspective. Bootstrap-in-progress state is tracked via a sync.Map on the reconciler, keyed by the Galera CR's NamespacedName, storing only the bootstrap pod name and its containerID at injection time. Joiner injection is not tracked at all — the filesystem check in isGaleraContainerStartedAndWaiting (test ! -f gcomm_uri && pgrep detect_gcomm_and_start.sh) already prevents double-injection, since the detect script exec's into mysqld after consuming the file. On operator restart, reconstructBootstrapState probes Running pods for a mysqld process (pgrep -aP1) to detect a bootstrap already in progress, avoiding spurious re-injection of gcomm:// files. 2. Unnecessary ContainerID check in findBestCandidate(): the function required all replicas to have attributes with ContainerIDs matching the currently running containers. But pods restart faster than the reconcile cycle, so CIDs never match. This check is unnecessary during bootstrap recovery because no pod has started mysqld (pods are blocked waiting for gcomm_uri), so the seqno on the persistent volume cannot change between container restarts. Fix: remove the CID comparison from findBestCandidate(). Only require that all replicas have pushed attributes with valid seqnos. Log CID mismatches for observability without blocking the decision. 3. Spurious joiner push: when all pods were killed, the StatefulSet's AvailableReplicas could remain > 0 briefly (stale due to the startup probe timeout, 240s by default), causing Bootstrapped to be true. The operator pushed joiner gcomm URIs to pods, making them start mysqld against dead peers and wasting a restart cycle. Fix: skip joiner gcomm push when Bootstrapped is set but no pods are actually Ready. Also add a periodic 10s requeue when not bootstrapped, ensuring the reconciler retries even when no external events trigger a reconcile. isBootstrapInProgress returns true only if the container that initiated the bootstrap is still present (matching CID). If the pod restarted or is gone, the entry is cleared so the operator can reprobe. injectGcommURI and getPodsWaitingForGcomm remain free functions with no shared state. Generated-By: claude-opus-4-6 Signed-off-by: Luca Miccini <lmiccini@redhat.com>
lmiccini
approved these changes
Jul 7, 2026
Contributor
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: lmiccini, openshift-cherrypick-robot The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-merge-bot
Bot
merged commit 9b81cf9
into
openstack-k8s-operators:18.0-fr6
6 checks passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is an automated cherry-pick of #499
/assign dciabrin