[backport release/v25.3.x] operator: defer rolling restart while a recently replaced pod is still coming up

[david-yu]

Summary

Backport of the HasRecentlyReplacedPods rolling-restart guard introduced on
main in #1446. The guard itself lives in operator/internal/lifecycle/pool.go
and is called from the rolling loop in operator/internal/controller/redpanda/redpanda_controller.go.

Why

During a rolling restart the operator deletes one pod at a time and waits for
the cluster to report healthy before proceeding. In practice the Redpanda
health API lags behind pod state: after deleting pod A, the new pod A can
appear in the cache with the correct revision before Redpanda detects broker
A's departure, so isHealthy remains true and the operator immediately
deletes pod B — causing two pods to be unavailable simultaneously.

HasRecentlyReplacedPods in pool.go closes this race. Before entering the
rolling loop (and only when there are pods to roll), it checks whether any
pod that already carries the latest StatefulSet revision is not yet
Running+Ready. If so, the reconciler defers with a requeue rather than
proceeding to the next deletion. This is narrower than "all pods ready" —
pods with the old revision that happen to be unhealthy for unrelated reasons
do not block the roll.

Why a fresh commit instead of a cherry-pick

#1446 landed as a large batch (acceptance test segmentation + multicluster
fixes + this guard). Cherry-picking it whole would drag in a much larger diff
and types (MulticlusterStatefulSet, multicluster client interfaces) that
don't apply cleanly to this release branch. This PR carries just the guard
chunk verbatim from main.

Test plan

• go vet ./operator/internal/lifecycle/... ./operator/internal/controller/redpanda/... — clean
• Manual rolling-restart soak on a 3-broker cluster under produce load —
  observe at most one pod Terminating + one pod NotReady at any moment
  (vs. two-down windows pre-guard)

Follow-ups

• ENG-222 (per-broker prestart-risk API) is the root-cause fix that replaces
  the cluster-IsHealthy heuristic with per-broker risks. Tracked separately:
  rpadmin client bindings in common-go#170,
  operator-side wiring after that lands.
• Operator-driven maintenance-mode + drain (decouple from preStop's 45 s budget)
  — separate PR.

🤖 Generated with Claude Code

[RafalKorepta]

Acceptance test suite is failing non deterministically. Maybe similar patch fix as in #1541 should be applied.