Skip to content

[ENHANCE] Expose Deployment update strategy via Helm values #1147

Description

@shashank-altan

Problem

The Helm chart's deployments/kubernetes/chart/reloader/templates/deployment.yaml does not render a strategy: block at all, so the Reloader Deployment always runs with Kubernetes' default RollingUpdate strategy. There is no value in values.yaml to override it.

This is normally fine, but creates a real-world problem during migrations from older Reloader versions where a known bug exists in the old version's startup path. With RollingUpdate, Kubernetes can briefly surge the old ReplicaSet up by one to maintain availability while pulling the new image — booting a fresh old-version pod that then re-triggers the very bug being fixed by the upgrade.

Concrete use case (just hit this)

We were upgrading a fleet from chart v0.0.103 (Reloader appVersion v0.0.103) to chart 2.2.11 (appVersion v1.4.16). v0.0.103 has a well-known cross-controller initialization race where a single package-level controllerInitialized bool is shared between the ConfigMap and Secret controllers — whichever finishes WaitForCacheSync first flips the flag, and remaining Add events from the other controller are then treated as real creates and trigger doRollingUpgrade unconditionally. Cascade-restarts every annotated workload in the namespace.

The race is fixed cleanly in v1.4.16 (separate per-resource flags, Add reloads gated behind ReloadOnCreate). But during the rolling chart upgrade, Kubernetes scaled the old ReplicaSet up by one to maintain availability, booting a new v0.0.103 pod which immediately hit the race and cascaded restarts of 38+ workloads — exactly the bug the upgrade was meant to fix.

Events confirming the timeline:

11:34:23Z  ScalingReplicaSet  Scaled up replica set <old-rs> to 2 from 1
11:34:23Z  ScalingReplicaSet  Scaled up replica set <new-rs> to 1
11:34:25Z  Started container reloader  (old-image v0.0.103 pod)
11:34:25Z  <first cascade annotation observed on a watched workload>
11:34:33Z  Started container reloader  (new-image v1.4.16 pod)  — 8s late

For the chart upgrade specifically, we want strategy.type: Recreate so Kubernetes kills all existing pods before creating any new ones — no surge means no new old-version pod ever runs during the transition.

Workaround we shipped

We vendored the chart and patched deployment.yaml:

spec:
{{- with .Values.reloader.deployment.strategy }}
  strategy:
    {{- toYaml . | nindent 4 }}
{{- end }}

with reloader.deployment.strategy: { type: Recreate } in values. Worked end-to-end across our test cluster.

Proposed change

Add a strategy: value to values.yaml under reloader.deployment and render it in the deployment template only when set. Backwards-compatible — default behavior (RollingUpdate) is unchanged when the value is unset.

templates/deployment.yaml — add right after the spec: line:

spec:
{{- with .Values.reloader.deployment.strategy }}
  strategy:
    {{- toYaml . | nindent 4 }}
{{- end }}

values.yaml — add under reloader.deployment:

    # Deployment update strategy. If unset, Kubernetes default (RollingUpdate) is used.
    # Example:
    #   strategy:
    #     type: Recreate
    #
    #   strategy:
    #     type: RollingUpdate
    #     rollingUpdate:
    #       maxSurge: 0
    #       maxUnavailable: 1
    strategy: {}

Mirrors how the chart already exposes dnsConfig, topologySpreadConstraints, and similar — same pattern, same idiom.

Why this matters beyond the v0.0.103 migration

  • Operators recovering from any future startup-time regression have a knob to disable surge during the upgrade.
  • Air-gapped or resource-constrained clusters that don't want headroom for a surge pod (maxSurge: 0) can express that without a downstream fork.
  • Common Helm-chart hygiene — almost every major operator chart exposes update strategy as a value.

Happy to send a PR for this change if it'd help. Thanks!

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions