feat(cilium): stage homogeneous wildcard device selection for datapath activation

k8s/providers/hetzner/infrastructure/controllers/cilium/components/bandwidth-manager-bbr/kustomization.yaml

@@ -12,11 +12,14 @@
 # bbrHostNamespaceOnly stays true and limits BBR to hostNetwork workloads.
 # Do not remove that guard until #2609 has merged and completed its own soak.
 #
-# DEVICE GUARD: production currently pins two heterogeneous private-NIC names
-# (`enp7s0` on workers, `eth1` on control planes). Cilium requires every
-# manually listed bandwidth-manager device name on every managed node. Do not
-# activate this component until #2610 provides a homogeneous selection while
-# preserving the private-NIC WireGuard and mutual-auth path.
+# DEVICE GUARD: production pins two private-NIC names by UNION (static nodes
+# enp7s0 / autoscale nodes eth1 — see ../private-nic-devices/), so each node
+# matches only ONE listed name. Cilium requires every manually listed
+# bandwidth-manager device name on every managed node; wildcards are the
+# upstream-documented answer. Do not activate this component until #2610's
+# ../homogeneous-devices/ wildcard selection has been ACTIVATED and soaked
+# (it preserves the private-NIC WireGuard and mutual-auth path — see its
+# runbook).
 #
 # SAFETY: this component is staged, not activation-ready. Keep it unreferenced
 # until both #2609 and #2610 are complete, then use a separate activation/soak.

k8s/providers/hetzner/infrastructure/controllers/cilium/components/ebpf-host-routing/kustomization.yaml

@@ -28,18 +28,18 @@
 #     exactly what the activation soak must observe (Hubble drops,
 #     cilium status --verbose) before any production enable.
 #   - Private-NIC device pinning: NOT touched here — and it is a HARD
-#     ACTIVATION BLOCKER, not just a naming concern. The hetzner strict-mode
-#     patch pins `devices: "enp7s0 eth1"` (private NICs, required so SPIRE
-#     mutual-auth dials private primary addresses), while the default route —
-#     and therefore pod->internet egress — leaves via the PUBLIC NICs
-#     (workers enp1s0 / control planes eth0). BPF masquerading only processes
-#     interfaces carrying Cilium's eBPF programs, so with today's device list
-#     the public egress NICs would get NO masquerade program while iptables
-#     masquerade is disabled: pod traffic would leave with an unroutable
-#     pod-CIDR source and external egress would break cluster-wide. #2610
-#     must FIRST extend device selection so the masquerade program attaches
-#     to the public default-route NICs while preserving the private-NIC
-#     primary-address behaviour; only then may this component be referenced.
+#     ACTIVATION BLOCKER, not just a naming concern. The always-on
+#     ../private-nic-devices/ component pins the private NICs (required so
+#     the node's primary address stays private for the mutual-auth path),
+#     while the default route — and therefore pod->internet egress — leaves
+#     via the PUBLIC NICs (static nodes enp1s0 / autoscale nodes eth0). BPF
+#     masquerading only processes interfaces carrying Cilium's eBPF
+#     programs, so under the private-only pin the public egress NICs would
+#     get NO masquerade program while iptables masquerade is disabled: pod
+#     traffic would leave with an unroutable pod-CIDR source and external
+#     egress would break cluster-wide. #2610's ../homogeneous-devices/
+#     component (wildcard selection covering both NICs) must be ACTIVATED
+#     and soaked FIRST; only then may this component be referenced.
 #
 # SAFETY: this component is staged, not activation-ready. Keep it
 # unreferenced until its own low-traffic activation + soak; it changes the

k8s/providers/hetzner/infrastructure/controllers/cilium/components/homogeneous-devices/kustomization.yaml

@@ -0,0 +1,148 @@
+# Default-off staging component for platform#2029 / platform#2610.
+#
+# Widens Cilium's eBPF device selection from the private-NIC pin
+# (`devices: "enp7s0 eth1"`, enforce-wireguard-strict-mode.yaml) to BOTH NICs
+# on every node via wildcard patterns, so the BPF masquerade program that
+# ebpf-host-routing (#2609) stages can attach to the PUBLIC default-route
+# NICs — pod->internet egress leaves through them, and BPF masquerading only
+# processes interfaces carrying Cilium's eBPF programs. Without this, enabling
+# ebpf-host-routing would disable iptables masquerade while installing no BPF
+# masquerade on the egress NICs: pod traffic would leave with an unroutable
+# pod-CIDR source and external egress would break cluster-wide.
+#
+# WHY WILDCARDS (the homogeneous strategy, live-verified): upstream requires
+# manually listed bandwidth-manager device names to be consistent on every
+# managed node, and this fleet's NIC names are CLASS-heterogeneous — live
+# inventory 2026-07-18 (`talosctl get addresses` + `kubectl get nodes`, all
+# NINE nodes):
+#
+#   static nodes  (prod-control-plane-2/4/5, prod-worker-1/2/3):
+#     public enp1s0 (default route + public IP) / private enp7s0 (10.0.1.x)
+#   autoscale nodes (autoscale-cx33-*/cx43-* cattle):
+#     public eth0 (default route + public IP + IPv6) / private eth1 (10.0.1.x)
+#
+# Note this INVERTS the 2026-06-19 "workers enp7s0 / control-planes eth1"
+# premise recorded in enforce-wireguard-strict-mode.yaml: the control planes
+# have since been rebuilt onto enp-named machines (confirmed live: all three
+# CP agents report devices=[enp7s0]), and the split is now static-vs-autoscale.
+# Today's two-name pin still covers every node's PRIVATE NIC — but only by
+# UNION (static->enp7s0, autoscale->eth1), each node matching exactly one name.
+#
+# An explicit four-name list ("enp7s0 eth1 enp1s0 eth0") would leave every
+# node with two names that do not exist on it — exactly what upstream's
+# bandwidth-manager consistency guidance forbids. A wildcard pattern list is
+# IDENTICAL on every node and resolves per node to only the devices that
+# exist (upstream documents wildcards for exactly this heterogeneous-naming
+# case). "en+ eth+" matches all four current names, nothing else on these
+# hosts (bond0/dummy0 and Cilium's own cilium_*/lxc* interfaces don't match
+# or are auto-excluded), and also absorbs future autoscaler node-type naming
+# drift (ens*/eno* would still match en+) — retiring the standing WARNING in
+# the strict-mode patch about differently-named private NICs.
+#
+# PRIMARY-ADDRESS RISK — why this stays default-off and activation must
+# prove it per node: the private-only pin exists so each node's PRIMARY
+# address (VXLAN/WireGuard tunnel endpoint, cilium-health, routing — see
+# enforce-wireguard-strict-mode.yaml) stays the private 10.0.1.x that the
+# Talos ingress firewall admits on 4240/4250/8472. Re-admitting the public
+# NICs into the device set re-admits public addresses as primary candidates.
+# Upstream documents that multi-device direct-routing selection prefers the
+# device holding the Kubernetes InternalIP (private here, on every node), but
+# the node-address selection of the running 1.20.0-pre.4 is not documented
+# precisely enough to TRUST that the primary stays private — the 2026-06-19
+# incident showed device-set changes can flip it. So: activate one node at a
+# time and verify before proceeding (runbook below). Note the SPIRE
+# mutual-auth dial itself is intermittently public-IP-bound REGARDLESS of
+# this selection (upstream cilium/cilium#46767, random node-IP pick); that
+# pre-existing defect is tracked separately and is NOT widened by this
+# component as long as the primary/routing address stays private.
+#
+# ROLLOUT GATE (why this component ALSO sets updateStrategy OnDelete): the
+# base HelmRelease's `rollOutCiliumPods: true` would start an automatic
+# DaemonSet rollout the moment this component is referenced, proceeding to
+# the next node as soon as each updated agent reports Ready — no pause point
+# exists, so the per-node verification below would be unexecutable and a bad
+# primary-address selection could reach the fleet before detection. OnDelete
+# makes the roll operator-stepped: the new config only reaches a node when
+# that node's cilium pod is deleted. The override lives HERE so it applies
+# exactly while the widened device set is being introduced, and its removal
+# (after the fleet is verified) is a strategy-only change that rolls nothing.
+#
+# GATE LIMITS (what OnDelete does NOT stop): a pod that restarts for any
+# OTHER reason during the window (crash, eviction, node reboot) is recreated
+# from the NEW template immediately, and so is the agent on any NODE the
+# cluster autoscaler adds mid-window — OnDelete gates template-driven
+# replacement only. The runbook therefore suspends autoscaler node creation
+# for the window and treats any incidental restart as an involuntary canary
+# to verify at once.
+#
+# ACTIVATION RUNBOOK (separate PR referencing this component; low-traffic
+# window):
+#   0. Suspend autoscaler node creation for the window:
+#      `kubectl -n kube-system scale deploy
+#      cluster-autoscaler-hetzner-cluster-autoscaler --replicas=0`. The
+#      autoscaler is ksail-provisioned (ksail.prod.yaml `autoscaler.node`),
+#      not Flux-managed, so the scale-down holds until step 5 restores
+#      `--replicas=1`. Confirm the over-provisioning CapacityBuffer headroom
+#      is idle enough that no scale-up would have fired.
+#   1. Reference this component in ../../../kustomization.yaml `components:`.
+#      Under OnDelete, NO agent restarts yet — the new device set is staged
+#      in the agent ConfigMap only. Watch for INVOLUNTARY canaries from here
+#      on: any cilium pod that restarts for unrelated reasons now boots the
+#      widened set — run step 2's verification on it immediately.
+#   2. Delete ONE node's cilium pod (`kubectl -n kube-system delete pod
+#      <cilium-x>`) and verify its replacement before touching the next:
+#      agent Ready; its log shows the widened device set (e.g.
+#      devices=[enp1s0 enp7s0]) and NO public primary (the node's advertised
+#      router/tunnel addresses stay 10.0.1.x — check the agent's
+#      node-address/NodeAddress log lines and `kubectl get ciliumnodes`);
+#      cilium-health reachability intact; no new policy-verdict DROPs or
+#      auth-timeout storms in Hubble/agent logs (a `dial <public-ip>:4250`
+#      RATE INCREASE beyond the known #46767 background is a regression).
+#   3. Repeat node by node across all nine; soak.
+#   4. After the soak, a follow-up PR removes the `updateStrategy` override
+#      from this component — restoring RollingUpdate is a strategy-only
+#      change (the pod template is already current), so it rolls nothing.
+#   5. Restore the autoscaler (`--replicas=1`) and verify the FIRST
+#      autoscaler-created node after resume like a stepped node (its agent
+#      boots the widened set directly).
+#   6. Only then does ebpf-host-routing (#2609) become referenceable;
+#      bandwidth-manager-bbr activation follows its own soak (see those
+#      components' guards).
+#   ROLLBACK (any point): re-comment the component reference — the template
+#   reverts to the private-only pin and RollingUpdate resumes, so every
+#   already-stepped node rolls BACK automatically. Un-stepped nodes keep the
+#   old config UNLESS their pod restarted during the window (the involuntary
+#   canaries above) — those roll back with the stepped ones.
+#
+# MERGE-ORDER NOTE (load-bearing): parent `patches:` are applied AFTER all
+# component patches — a devices value carried by
+# patches/enforce-wireguard-strict-mode.yaml could NEVER be overridden by this
+# component (render-verified on #2610's delivery PR: with this component
+# enabled against a patch-carried pin, the patch value still won). That is why
+# the live pin lives in ../private-nic-devices/ (a component), and why THIS
+# component must be listed AFTER it in the parent's `components:` — components
+# apply in list order, later wins (also render-verified). Do not move either
+# device selection back into a parent patch.
+apiVersion: kustomize.config.k8s.io/v1alpha1
+kind: Component
+patches:
+  - target:
+      group: helm.toolkit.fluxcd.io
+      version: v2
+      kind: HelmRelease
+      name: cilium
+      namespace: kube-system
+    patch: |
+      apiVersion: helm.toolkit.fluxcd.io/v2
+      kind: HelmRelease
+      metadata:
+        name: cilium
+        namespace: kube-system
+      spec:
+        values:
+          devices: "en+ eth+"
+          # Operator-stepped rollout gate — see ROLLOUT GATE above. Removed
+          # by a follow-up PR once all nodes are verified (strategy-only
+          # change; rolls nothing).
+          updateStrategy:
+            type: OnDelete

k8s/providers/hetzner/infrastructure/controllers/cilium/components/private-nic-devices/kustomization.yaml

@@ -0,0 +1,66 @@
+# ALWAYS-ON component: pin the Cilium eBPF datapath to the PRIVATE-network
+# NICs. This is the live production device selection — it must stay referenced
+# in ../../../kustomization.yaml until homogeneous-devices/ (its staged
+# successor, see below) has been activated and soaked.
+#
+# WHY A COMPONENT AND NOT A PATCH: the staged homogeneous-devices/ component
+# must be able to OVERRIDE this value when referenced. Kustomize applies the
+# parent kustomization's `patches:` AFTER all component patches (verified by
+# render on #2610's delivery PR: a component could not override a `devices`
+# value carried by patches/enforce-wireguard-strict-mode.yaml), while
+# components themselves apply in list order — later wins. Keeping both device
+# selections as components, with the staged one listed after this one, is what
+# makes the flip possible as a one-line reference change. The devices key
+# previously lived in the strict-mode patch; it moved here UNCHANGED, so the
+# rendered HelmRelease is identical and moving it rolled no agents.
+#
+# WHY THE PIN (condensed from the original 2026-06-19 rationale, which lives
+# in git history of patches/enforce-wireguard-strict-mode.yaml): Hetzner nodes
+# are dual-homed. With devices unset, Cilium attaches to BOTH NICs and can
+# take the default-route (public) NIC's IP as the node's PRIMARY address; the
+# SPIRE mutual-auth control connection (agent->agent TCP 4250) then dials peer
+# PUBLIC IPs, which the Talos ingress firewall only admits from the private
+# node CIDR — dropped handshakes black-hole every auth-gated cross-node pod
+# flow (observed live 2026-06-19). Pinning the private NICs keeps each node's
+# primary address private (10.0.1.x), so 4250/4240/8472 ride the admitted
+# path. (A residual upstream defect remains regardless: cilium/cilium#46767 —
+# the auth dial picks a RANDOM advertised node IP, so intermittent public-IP
+# dials persist even with this pin. Tracked separately.)
+#
+# NIC-NAME INVENTORY (live-verified 2026-07-18, `talosctl get addresses`, all
+# NINE nodes — this UPDATES the stale 2026-06-19 "workers enp7s0 /
+# control-planes eth1" premise; the split is static-vs-autoscale now that the
+# control planes were rebuilt onto enp-named machines):
+#   static  (prod-control-plane-2/4/5, prod-worker-1/2/3):
+#     public enp1s0 (default route) / private enp7s0 (10.0.1.x)
+#   autoscale (autoscale-cx33-*/cx43-*): public eth0 / private eth1
+# So this two-name list covers every node's private NIC only by UNION — each
+# node matches exactly ONE of the two names, and the agent logs confirm e.g.
+# devices=[enp7s0] on all three control planes.
+#
+# WARNING (unchanged): hard-codes the private-NIC names. An autoscaler worker
+# on a server type whose private NIC is named differently (e.g. ens*) would
+# lose Cilium networking until its name is added here — keep in sync with the
+# node pool's machine type. The homogeneous-devices/ successor retires this
+# hazard via wildcard patterns.
+#
+# PROD/HETZNER ONLY: the docker/local provider is single-NIC, so device
+# autodetection is correct there.
+apiVersion: kustomize.config.k8s.io/v1alpha1
+kind: Component
+patches:
+  - target:
+      group: helm.toolkit.fluxcd.io
+      version: v2
+      kind: HelmRelease
+      name: cilium
+      namespace: kube-system
+    patch: |
+      apiVersion: helm.toolkit.fluxcd.io/v2
+      kind: HelmRelease
+      metadata:
+        name: cilium
+        namespace: kube-system
+      spec:
+        values:
+          devices: "enp7s0 eth1"

k8s/providers/hetzner/infrastructure/controllers/cilium/patches/enforce-wireguard-strict-mode.yaml

@@ -68,51 +68,11 @@ spec:
           # cilium-health host probes survive the IPCache propagation window
           # during node churn. Pod-to-pod across the CIDR above stays fail-closed.
           allowRemoteNodeIdentities: true
-    # Pin the eBPF datapath to the PRIVATE-network NICs so every node uses its
-    # private IP (10.0.1.x) as its primary node address for agent-to-agent
-    # control traffic — specifically the SPIRE mutual-auth handshake (TCP 4250).
-    #
-    # Hetzner nodes are dual-homed: a public NIC carrying the default route
-    # (workers enp1s0 / control-planes eth0) and the private-network NIC
-    # (workers enp7s0 / control-planes eth1). With devices unset, Cilium probes
-    # and attaches to BOTH, then takes the default-route (public) NIC's IP as the
-    # node's PRIMARY address. cilium-health (4240) and the VXLAN/WireGuard tunnel
-    # ride the private "Direct Routing" NIC and work — but the mutual-auth control
-    # connection (cilium-agent -> peer cilium-agent on 4250) dials the peer's
-    # PRIMARY (public) IP. The Talos ingress firewall only admits 4240/4250/8472
-    # from the private node CIDR (10.0.0.0/16, see
-    # talos/workers/allow-cilium-mutual-auth-ingress.yaml), so the public-sourced 4250 handshake
-    # is dropped (`dial <public-ip>:4250: i/o timeout`) and Cilium black-holes every
-    # cross-node pod flow the hetzner require-mutual-auth policy marks
-    # `authentication.mode: required` once its auth-cache entry lapses. Observed
-    # live (2026-06-19): the Crossview dashboard was unreachable — authenticated
-    # oauth2-proxy/auth-proxy -> crossview:3001 SYNs returned `policy-verdict:
-    # none ... Policy denied DROPPED` (the spire AUTH TYPE on that policy entry
-    # never completed its handshake) — while older dashboards limped on still-
-    # cached auth entries. Failures spanned all three workers, dialing each
-    # other's public IPs on 4250.
-    #
-    # Listing the private NICs makes the private IP each node's primary address,
-    # so the 4250 handshake uses 10.0.1.x and the firewall admits it.
-    #
-    # Safe on this topology (all verified live, 2026-06-19):
-    #   - Ingress: the Hetzner LB reaches nodes on their PRIVATE IPs
-    #     (load-balancer.hetzner.cloud/use-private-ip: "true", see
-    #     ../../../patches/attach-hcloud-load-balancer.yaml) and clients hit the LB's public IP, not
-    #     node public IPs. The private NIC stays a Cilium device, so the LB's
-    #     NodePort BPF is unaffected; only unused public-NIC NodePort BPF is shed.
-    #   - Egress: pod -> internet masquerade is the device-AGNOSTIC iptables rule
-    #     (-s podCIDR ! -d podCIDR ! -o cilium_+ -j MASQUERADE), not scoped to
-    #     this devices list, so pods still SNAT out the public NIC. (Masquerading
-    #     is iptables here; bpf.masquerade is off.)
-    #   - cilium-health / VXLAN / WireGuard already used the private NIC.
-    #
-    # WARNING: hard-codes the private-NIC names. Today every worker is enp7s0 and
-    # every control-plane is eth1 (verified on all 6 nodes). An autoscaler worker
-    # brought up on a server type whose private NIC is named differently would
-    # lose Cilium networking until its name is added here — keep this in sync with
-    # the node pool's machine type.
-    #
-    # PROD/HETZNER ONLY: the docker/local provider is single-NIC, so device
-    # autodetection is correct there. This pin lives in the hetzner overlay.
-    devices: "enp7s0 eth1"
+    # NB: the private-NIC `devices` pin that used to live here moved to
+    # ../components/private-nic-devices/ (same value, so the rendered
+    # HelmRelease is unchanged). It had to become a component so the staged
+    # homogeneous-devices/ successor can override it — a value carried by this
+    # parent patch is applied AFTER all component patches and can never be
+    # overridden by one. The pin's full rationale (2026-06-19 mutual-auth
+    # incident, LB/egress safety analysis) and the refreshed 2026-07-18
+    # NIC inventory live in that component's header.