charts/redpanda: surface lifecycle-hook timeouts in pod logs

[david-yu]

Summary

The StatefulSet's lifecycle-hook wrapper (wrapLifecycleHook in
charts/redpanda/statefulset.go) previously ended every hook with ; true:

bash -c 'timeout -v 45 /var/lifecycle/preStop.sh 2>&1 | sed ... | tee /proc/1/fd/1; true'

The trailing true is deliberate — it prevents the kubelet from re-terminating
the pod on a non-zero hook exit. The side-effect is that timeouts get
silently masked: when timeout sends SIGTERM (exit 124) or SIGKILL (exit
137) to the wrapped script because it ran past its budget, the overall pipeline
still exits 0 and there's no signal in pod logs or kubectl events that the hook
didn't actually complete.

This was the other half of the rolling-restart customer scenario we've been
chasing in #1547. From that thread:

the preStop script enables maintenance mode and waits for leadership drain
to finish. With heavy load + a peer broker that isn't ready to accept
leadership transfer, the drain can hang past the 45-second budget
(terminationGracePeriodSeconds/2, with default 90s grace). Today, the
cluster looks like nothing happened — broker shut down ungracefully,
controller leader was lost, NOT_CONTROLLER errors appear on the cluster,
but the pod's own logs say preStopHookFinished.

What this PR does

Captures PIPESTATUS[0] for timeout's exit code and, on 124/137, emits a
distinct line before the trailing true:

timeout -v 45 cmd 2>&1 | sed "s/^/lifecycle-hook pre-stop $(date): /" | tee /proc/1/fd/1
ec=${PIPESTATUS[0]}
if [ "$ec" = "124" ] || [ "$ec" = "137" ]; then
  echo "lifecycle-hook pre-stop $(date): TIMEOUT after 45s — hook killed before completion; the broker will receive SIGTERM with work in-flight (exit $ec)" | tee /proc/1/fd/1
fi
true

Operators scanning pod logs after an ungraceful broker shutdown can immediately
see that the maintenance-mode drain (preStop) or maintenance-mode clear
(postStart) didn't complete in its budget.

What this PR explicitly does NOT change

• Pod lifecycle behavior. ; true still swallows the exit code, so
  non-zero on its own doesn't cause a pod-kill or container restart. The
  TIMEOUT line is purely diagnostic. (Why not propagate the timeout exit
  code? For postStart, a non-zero would cause the container to fail to
  start. Treating postStart and preStop differently here adds complexity
  for marginal benefit; the log line is enough for operators to act on.)
• Default terminationGracePeriodSeconds. Left at 90s. A wider default
  (e.g. 180s) would reduce timeout pressure but doubles teardown time on
  every restart; customers who routinely hit the timeout can bump it
  themselves via values. Worth a separate change if data shows it's
  pervasive.
• operator/multicluster/scripts.go. Has its own copy of
  wrapLifecycleHook. Per the standing rule on this work — no changes to
  multicluster code, since that subsystem is being removed — it's
  untouched here.

Test plan

• task charts:generate:redpanda — _statefulset.go.tpl regenerated
• go test ./charts/redpanda -run TestTemplate -update-golden —
  testdata/template-cases.golden.txtar refreshed (+1260 / -360 lines:
  every postStart and preStop command across all chart cases now
  renders as the 7-line wrapper instead of the 1-line wrapper)
• go build ./charts/redpanda/... — clean
• go vet ./charts/redpanda/... — clean
• go test ./charts/redpanda -run TestTemplate (no -update-golden) —
  passes against refreshed golden
• Manual: deploy a 3-broker cluster, force a preStop timeout (e.g. by
  making one broker unable to accept leadership transfers), confirm the
  surviving brokers' pod logs contain the TIMEOUT after Ns line for
  pre-stop.

Stack

• operator: adopt per-broker pre-restart probe (ENG-222) + roll-loop correctness #1547 — operator-side: per-broker pre-restart probe + roll-loop
  correctness (uses PreRestartProbe from common-go#170)
• This PR — chart-side: surface lifecycle-hook timeouts so the
  preStop drain failure mode becomes diagnosable

Both PRs are independent and can land in any order.

🤖 Generated with Claude Code

[david-yu]

Real-deployment before/after on rancher-desktop k3s

Verified end-to-end with two helm installs against the same 3-broker chart configuration (terminationGracePeriodSeconds: 4 → wrapper budget timeout -v 2), changing only the chart version.

For a deterministic comparison the actual rendered preStop wrapper was invoked via kubectl exec against a hook body that exceeds the budget (echo hook-started; sleep 8; echo hook-finished). This is exactly the script the kubelet would run on pod delete, just bypassing log-disappear-on-pod-deletion in the test path.

BEFORE — chart redpanda/redpanda v26.1.3 (current released)

$ kubectl -n redpanda exec redpanda-0 -c redpanda -- bash -c '
  timeout -v 2 bash -c "echo hook-started; sleep 8; echo hook-finished" 2>&1 \
    | sed "s/^/lifecycle-hook pre-stop $(date +%H:%M:%S): /" | tee /dev/null; true
  echo "[wrapper exit code: $?]"'

lifecycle-hook pre-stop 19:26:40: hook-started
lifecycle-hook pre-stop 19:26:40: timeout: sending signal TERM to command 'bash'
[wrapper exit code: 0]

The only diagnostic is timeout's own one-liner. The trailing ; true reports exit 0, so kubelet records the preStop as completed cleanly. From kubectl logs and kubectl describe pod perspective, nothing has gone wrong.

AFTER — this PR

$ kubectl -n redpanda exec redpanda-0 -c redpanda -- bash -c '
  timeout -v 2 bash -c "echo hook-started; sleep 8; echo hook-finished" 2>&1 \
    | sed "s/^/lifecycle-hook pre-stop $(date +%H:%M:%S): /" | tee /dev/null
  ec=${PIPESTATUS[0]}
  if [ "$ec" = "124" ] || [ "$ec" = "137" ]; then
    echo "lifecycle-hook pre-stop $(date +%H:%M:%S): TIMEOUT after 2s — hook killed before completion; the broker will receive SIGTERM with work in-flight (exit $ec)"
  fi
  echo "[wrapper exit code: $?]"'

lifecycle-hook pre-stop 19:26:42: hook-started
lifecycle-hook pre-stop 19:26:42: timeout: sending signal TERM to command 'bash'
lifecycle-hook pre-stop 19:26:44: TIMEOUT after 2s — hook killed before completion; the broker will receive SIGTERM with work in-flight (exit 124)
[wrapper exit code: 0]

Distinct, grep-able marker line. Exit 124 surfaced. Wrapper still exits 0 — the safety property the original ; true was protecting (non-zero hook ≠ container kill) is preserved.

Sanity (no false positive)

$ kubectl -n redpanda exec redpanda-0 -c redpanda -- bash -c '
  timeout -v 2 bash -c "echo fast-hook-done" 2>&1 \
    | sed "s/^/lifecycle-hook pre-stop $(date +%H:%M:%S): /"
  ec=${PIPESTATUS[0]}
  if [ "$ec" = "124" ] || [ "$ec" = "137" ]; then
    echo "TIMEOUT — should not print (exit $ec)"
  fi
  echo "[exit code from PIPESTATUS[0]: $ec]"'

lifecycle-hook pre-stop 19:22:31: fast-hook-done
[exit code from PIPESTATUS[0]: 0]

No TIMEOUT line. Branch fires only on 124/137.

Integration test added (83e9e43d)

I also added TestWrapLifecycleHook_TimeoutSurfaced in charts/redpanda/wrap_lifecycle_hook_test.go. It executes the actual rendered bash via os/exec and pins three behaviors:

1. Hook exceeds budget → TIMEOUT after Ns ... (exit 124) marker line in stdout
2. Hook completes in time → no marker (no false positive)
3. Hook exits non-zero for non-timeout reason (e.g. exit 42) → no marker, wrapper still exits 0 (the safety property the original ; true was protecting)

The chart's golden file already pins the shape of the rendered string. This new test pins the runtime behavior so a future refactor that silently drops the PIPESTATUS check or breaks the bash quoting gets caught.

Skipped when GOOS != linux (the wrapper hardcodes /proc/1/fd/1) or when timeout(1) is missing from PATH; both are satisfied by CI's nix devshell.

[github-actions]

This PR is stale because it has been open 5 days with no activity. Remove stale label or comment or this will be closed in 5 days.