diff --git a/k8s/providers/hetzner/infrastructure/controllers/cilium/components/bandwidth-manager-bbr/kustomization.yaml b/k8s/providers/hetzner/infrastructure/controllers/cilium/components/bandwidth-manager-bbr/kustomization.yaml index cccd20dad..ac2841450 100644 --- a/k8s/providers/hetzner/infrastructure/controllers/cilium/components/bandwidth-manager-bbr/kustomization.yaml +++ b/k8s/providers/hetzner/infrastructure/controllers/cilium/components/bandwidth-manager-bbr/kustomization.yaml @@ -12,11 +12,14 @@ # bbrHostNamespaceOnly stays true and limits BBR to hostNetwork workloads. # Do not remove that guard until #2609 has merged and completed its own soak. # -# DEVICE GUARD: production currently pins two heterogeneous private-NIC names -# (`enp7s0` on workers, `eth1` on control planes). Cilium requires every -# manually listed bandwidth-manager device name on every managed node. Do not -# activate this component until #2610 provides a homogeneous selection while -# preserving the private-NIC WireGuard and mutual-auth path. +# DEVICE GUARD: production pins two private-NIC names by UNION (static nodes +# enp7s0 / autoscale nodes eth1 — see ../private-nic-devices/), so each node +# matches only ONE listed name. Cilium requires every manually listed +# bandwidth-manager device name on every managed node; wildcards are the +# upstream-documented answer. Do not activate this component until #2610's +# ../homogeneous-devices/ wildcard selection has been ACTIVATED and soaked +# (it preserves the private-NIC WireGuard and mutual-auth path — see its +# runbook). # # SAFETY: this component is staged, not activation-ready. Keep it unreferenced # until both #2609 and #2610 are complete, then use a separate activation/soak. diff --git a/k8s/providers/hetzner/infrastructure/controllers/cilium/components/ebpf-host-routing/kustomization.yaml b/k8s/providers/hetzner/infrastructure/controllers/cilium/components/ebpf-host-routing/kustomization.yaml index 32eb2cbaa..6dc2d0cd2 100644 --- a/k8s/providers/hetzner/infrastructure/controllers/cilium/components/ebpf-host-routing/kustomization.yaml +++ b/k8s/providers/hetzner/infrastructure/controllers/cilium/components/ebpf-host-routing/kustomization.yaml @@ -28,18 +28,18 @@ # exactly what the activation soak must observe (Hubble drops, # cilium status --verbose) before any production enable. # - Private-NIC device pinning: NOT touched here — and it is a HARD -# ACTIVATION BLOCKER, not just a naming concern. The hetzner strict-mode -# patch pins `devices: "enp7s0 eth1"` (private NICs, required so SPIRE -# mutual-auth dials private primary addresses), while the default route — -# and therefore pod->internet egress — leaves via the PUBLIC NICs -# (workers enp1s0 / control planes eth0). BPF masquerading only processes -# interfaces carrying Cilium's eBPF programs, so with today's device list -# the public egress NICs would get NO masquerade program while iptables -# masquerade is disabled: pod traffic would leave with an unroutable -# pod-CIDR source and external egress would break cluster-wide. #2610 -# must FIRST extend device selection so the masquerade program attaches -# to the public default-route NICs while preserving the private-NIC -# primary-address behaviour; only then may this component be referenced. +# ACTIVATION BLOCKER, not just a naming concern. The always-on +# ../private-nic-devices/ component pins the private NICs (required so +# the node's primary address stays private for the mutual-auth path), +# while the default route — and therefore pod->internet egress — leaves +# via the PUBLIC NICs (static nodes enp1s0 / autoscale nodes eth0). BPF +# masquerading only processes interfaces carrying Cilium's eBPF +# programs, so under the private-only pin the public egress NICs would +# get NO masquerade program while iptables masquerade is disabled: pod +# traffic would leave with an unroutable pod-CIDR source and external +# egress would break cluster-wide. #2610's ../homogeneous-devices/ +# component (wildcard selection covering both NICs) must be ACTIVATED +# and soaked FIRST; only then may this component be referenced. # # SAFETY: this component is staged, not activation-ready. Keep it # unreferenced until its own low-traffic activation + soak; it changes the diff --git a/k8s/providers/hetzner/infrastructure/controllers/cilium/components/homogeneous-devices/kustomization.yaml b/k8s/providers/hetzner/infrastructure/controllers/cilium/components/homogeneous-devices/kustomization.yaml new file mode 100644 index 000000000..2302e1fc4 --- /dev/null +++ b/k8s/providers/hetzner/infrastructure/controllers/cilium/components/homogeneous-devices/kustomization.yaml @@ -0,0 +1,148 @@ +# Default-off staging component for platform#2029 / platform#2610. +# +# Widens Cilium's eBPF device selection from the private-NIC pin +# (`devices: "enp7s0 eth1"`, enforce-wireguard-strict-mode.yaml) to BOTH NICs +# on every node via wildcard patterns, so the BPF masquerade program that +# ebpf-host-routing (#2609) stages can attach to the PUBLIC default-route +# NICs — pod->internet egress leaves through them, and BPF masquerading only +# processes interfaces carrying Cilium's eBPF programs. Without this, enabling +# ebpf-host-routing would disable iptables masquerade while installing no BPF +# masquerade on the egress NICs: pod traffic would leave with an unroutable +# pod-CIDR source and external egress would break cluster-wide. +# +# WHY WILDCARDS (the homogeneous strategy, live-verified): upstream requires +# manually listed bandwidth-manager device names to be consistent on every +# managed node, and this fleet's NIC names are CLASS-heterogeneous — live +# inventory 2026-07-18 (`talosctl get addresses` + `kubectl get nodes`, all +# NINE nodes): +# +# static nodes (prod-control-plane-2/4/5, prod-worker-1/2/3): +# public enp1s0 (default route + public IP) / private enp7s0 (10.0.1.x) +# autoscale nodes (autoscale-cx33-*/cx43-* cattle): +# public eth0 (default route + public IP + IPv6) / private eth1 (10.0.1.x) +# +# Note this INVERTS the 2026-06-19 "workers enp7s0 / control-planes eth1" +# premise recorded in enforce-wireguard-strict-mode.yaml: the control planes +# have since been rebuilt onto enp-named machines (confirmed live: all three +# CP agents report devices=[enp7s0]), and the split is now static-vs-autoscale. +# Today's two-name pin still covers every node's PRIVATE NIC — but only by +# UNION (static->enp7s0, autoscale->eth1), each node matching exactly one name. +# +# An explicit four-name list ("enp7s0 eth1 enp1s0 eth0") would leave every +# node with two names that do not exist on it — exactly what upstream's +# bandwidth-manager consistency guidance forbids. A wildcard pattern list is +# IDENTICAL on every node and resolves per node to only the devices that +# exist (upstream documents wildcards for exactly this heterogeneous-naming +# case). "en+ eth+" matches all four current names, nothing else on these +# hosts (bond0/dummy0 and Cilium's own cilium_*/lxc* interfaces don't match +# or are auto-excluded), and also absorbs future autoscaler node-type naming +# drift (ens*/eno* would still match en+) — retiring the standing WARNING in +# the strict-mode patch about differently-named private NICs. +# +# PRIMARY-ADDRESS RISK — why this stays default-off and activation must +# prove it per node: the private-only pin exists so each node's PRIMARY +# address (VXLAN/WireGuard tunnel endpoint, cilium-health, routing — see +# enforce-wireguard-strict-mode.yaml) stays the private 10.0.1.x that the +# Talos ingress firewall admits on 4240/4250/8472. Re-admitting the public +# NICs into the device set re-admits public addresses as primary candidates. +# Upstream documents that multi-device direct-routing selection prefers the +# device holding the Kubernetes InternalIP (private here, on every node), but +# the node-address selection of the running 1.20.0-pre.4 is not documented +# precisely enough to TRUST that the primary stays private — the 2026-06-19 +# incident showed device-set changes can flip it. So: activate one node at a +# time and verify before proceeding (runbook below). Note the SPIRE +# mutual-auth dial itself is intermittently public-IP-bound REGARDLESS of +# this selection (upstream cilium/cilium#46767, random node-IP pick); that +# pre-existing defect is tracked separately and is NOT widened by this +# component as long as the primary/routing address stays private. +# +# ROLLOUT GATE (why this component ALSO sets updateStrategy OnDelete): the +# base HelmRelease's `rollOutCiliumPods: true` would start an automatic +# DaemonSet rollout the moment this component is referenced, proceeding to +# the next node as soon as each updated agent reports Ready — no pause point +# exists, so the per-node verification below would be unexecutable and a bad +# primary-address selection could reach the fleet before detection. OnDelete +# makes the roll operator-stepped: the new config only reaches a node when +# that node's cilium pod is deleted. The override lives HERE so it applies +# exactly while the widened device set is being introduced, and its removal +# (after the fleet is verified) is a strategy-only change that rolls nothing. +# +# GATE LIMITS (what OnDelete does NOT stop): a pod that restarts for any +# OTHER reason during the window (crash, eviction, node reboot) is recreated +# from the NEW template immediately, and so is the agent on any NODE the +# cluster autoscaler adds mid-window — OnDelete gates template-driven +# replacement only. The runbook therefore suspends autoscaler node creation +# for the window and treats any incidental restart as an involuntary canary +# to verify at once. +# +# ACTIVATION RUNBOOK (separate PR referencing this component; low-traffic +# window): +# 0. Suspend autoscaler node creation for the window: +# `kubectl -n kube-system scale deploy +# cluster-autoscaler-hetzner-cluster-autoscaler --replicas=0`. The +# autoscaler is ksail-provisioned (ksail.prod.yaml `autoscaler.node`), +# not Flux-managed, so the scale-down holds until step 5 restores +# `--replicas=1`. Confirm the over-provisioning CapacityBuffer headroom +# is idle enough that no scale-up would have fired. +# 1. Reference this component in ../../../kustomization.yaml `components:`. +# Under OnDelete, NO agent restarts yet — the new device set is staged +# in the agent ConfigMap only. Watch for INVOLUNTARY canaries from here +# on: any cilium pod that restarts for unrelated reasons now boots the +# widened set — run step 2's verification on it immediately. +# 2. Delete ONE node's cilium pod (`kubectl -n kube-system delete pod +# `) and verify its replacement before touching the next: +# agent Ready; its log shows the widened device set (e.g. +# devices=[enp1s0 enp7s0]) and NO public primary (the node's advertised +# router/tunnel addresses stay 10.0.1.x — check the agent's +# node-address/NodeAddress log lines and `kubectl get ciliumnodes`); +# cilium-health reachability intact; no new policy-verdict DROPs or +# auth-timeout storms in Hubble/agent logs (a `dial :4250` +# RATE INCREASE beyond the known #46767 background is a regression). +# 3. Repeat node by node across all nine; soak. +# 4. After the soak, a follow-up PR removes the `updateStrategy` override +# from this component — restoring RollingUpdate is a strategy-only +# change (the pod template is already current), so it rolls nothing. +# 5. Restore the autoscaler (`--replicas=1`) and verify the FIRST +# autoscaler-created node after resume like a stepped node (its agent +# boots the widened set directly). +# 6. Only then does ebpf-host-routing (#2609) become referenceable; +# bandwidth-manager-bbr activation follows its own soak (see those +# components' guards). +# ROLLBACK (any point): re-comment the component reference — the template +# reverts to the private-only pin and RollingUpdate resumes, so every +# already-stepped node rolls BACK automatically. Un-stepped nodes keep the +# old config UNLESS their pod restarted during the window (the involuntary +# canaries above) — those roll back with the stepped ones. +# +# MERGE-ORDER NOTE (load-bearing): parent `patches:` are applied AFTER all +# component patches — a devices value carried by +# patches/enforce-wireguard-strict-mode.yaml could NEVER be overridden by this +# component (render-verified on #2610's delivery PR: with this component +# enabled against a patch-carried pin, the patch value still won). That is why +# the live pin lives in ../private-nic-devices/ (a component), and why THIS +# component must be listed AFTER it in the parent's `components:` — components +# apply in list order, later wins (also render-verified). Do not move either +# device selection back into a parent patch. +apiVersion: kustomize.config.k8s.io/v1alpha1 +kind: Component +patches: + - target: + group: helm.toolkit.fluxcd.io + version: v2 + kind: HelmRelease + name: cilium + namespace: kube-system + patch: | + apiVersion: helm.toolkit.fluxcd.io/v2 + kind: HelmRelease + metadata: + name: cilium + namespace: kube-system + spec: + values: + devices: "en+ eth+" + # Operator-stepped rollout gate — see ROLLOUT GATE above. Removed + # by a follow-up PR once all nodes are verified (strategy-only + # change; rolls nothing). + updateStrategy: + type: OnDelete diff --git a/k8s/providers/hetzner/infrastructure/controllers/cilium/components/private-nic-devices/kustomization.yaml b/k8s/providers/hetzner/infrastructure/controllers/cilium/components/private-nic-devices/kustomization.yaml new file mode 100644 index 000000000..15568caf3 --- /dev/null +++ b/k8s/providers/hetzner/infrastructure/controllers/cilium/components/private-nic-devices/kustomization.yaml @@ -0,0 +1,66 @@ +# ALWAYS-ON component: pin the Cilium eBPF datapath to the PRIVATE-network +# NICs. This is the live production device selection — it must stay referenced +# in ../../../kustomization.yaml until homogeneous-devices/ (its staged +# successor, see below) has been activated and soaked. +# +# WHY A COMPONENT AND NOT A PATCH: the staged homogeneous-devices/ component +# must be able to OVERRIDE this value when referenced. Kustomize applies the +# parent kustomization's `patches:` AFTER all component patches (verified by +# render on #2610's delivery PR: a component could not override a `devices` +# value carried by patches/enforce-wireguard-strict-mode.yaml), while +# components themselves apply in list order — later wins. Keeping both device +# selections as components, with the staged one listed after this one, is what +# makes the flip possible as a one-line reference change. The devices key +# previously lived in the strict-mode patch; it moved here UNCHANGED, so the +# rendered HelmRelease is identical and moving it rolled no agents. +# +# WHY THE PIN (condensed from the original 2026-06-19 rationale, which lives +# in git history of patches/enforce-wireguard-strict-mode.yaml): Hetzner nodes +# are dual-homed. With devices unset, Cilium attaches to BOTH NICs and can +# take the default-route (public) NIC's IP as the node's PRIMARY address; the +# SPIRE mutual-auth control connection (agent->agent TCP 4250) then dials peer +# PUBLIC IPs, which the Talos ingress firewall only admits from the private +# node CIDR — dropped handshakes black-hole every auth-gated cross-node pod +# flow (observed live 2026-06-19). Pinning the private NICs keeps each node's +# primary address private (10.0.1.x), so 4250/4240/8472 ride the admitted +# path. (A residual upstream defect remains regardless: cilium/cilium#46767 — +# the auth dial picks a RANDOM advertised node IP, so intermittent public-IP +# dials persist even with this pin. Tracked separately.) +# +# NIC-NAME INVENTORY (live-verified 2026-07-18, `talosctl get addresses`, all +# NINE nodes — this UPDATES the stale 2026-06-19 "workers enp7s0 / +# control-planes eth1" premise; the split is static-vs-autoscale now that the +# control planes were rebuilt onto enp-named machines): +# static (prod-control-plane-2/4/5, prod-worker-1/2/3): +# public enp1s0 (default route) / private enp7s0 (10.0.1.x) +# autoscale (autoscale-cx33-*/cx43-*): public eth0 / private eth1 +# So this two-name list covers every node's private NIC only by UNION — each +# node matches exactly ONE of the two names, and the agent logs confirm e.g. +# devices=[enp7s0] on all three control planes. +# +# WARNING (unchanged): hard-codes the private-NIC names. An autoscaler worker +# on a server type whose private NIC is named differently (e.g. ens*) would +# lose Cilium networking until its name is added here — keep in sync with the +# node pool's machine type. The homogeneous-devices/ successor retires this +# hazard via wildcard patterns. +# +# PROD/HETZNER ONLY: the docker/local provider is single-NIC, so device +# autodetection is correct there. +apiVersion: kustomize.config.k8s.io/v1alpha1 +kind: Component +patches: + - target: + group: helm.toolkit.fluxcd.io + version: v2 + kind: HelmRelease + name: cilium + namespace: kube-system + patch: | + apiVersion: helm.toolkit.fluxcd.io/v2 + kind: HelmRelease + metadata: + name: cilium + namespace: kube-system + spec: + values: + devices: "enp7s0 eth1" diff --git a/k8s/providers/hetzner/infrastructure/controllers/cilium/patches/enforce-wireguard-strict-mode.yaml b/k8s/providers/hetzner/infrastructure/controllers/cilium/patches/enforce-wireguard-strict-mode.yaml index 45bbee164..79d778af6 100644 --- a/k8s/providers/hetzner/infrastructure/controllers/cilium/patches/enforce-wireguard-strict-mode.yaml +++ b/k8s/providers/hetzner/infrastructure/controllers/cilium/patches/enforce-wireguard-strict-mode.yaml @@ -68,51 +68,11 @@ spec: # cilium-health host probes survive the IPCache propagation window # during node churn. Pod-to-pod across the CIDR above stays fail-closed. allowRemoteNodeIdentities: true - # Pin the eBPF datapath to the PRIVATE-network NICs so every node uses its - # private IP (10.0.1.x) as its primary node address for agent-to-agent - # control traffic — specifically the SPIRE mutual-auth handshake (TCP 4250). - # - # Hetzner nodes are dual-homed: a public NIC carrying the default route - # (workers enp1s0 / control-planes eth0) and the private-network NIC - # (workers enp7s0 / control-planes eth1). With devices unset, Cilium probes - # and attaches to BOTH, then takes the default-route (public) NIC's IP as the - # node's PRIMARY address. cilium-health (4240) and the VXLAN/WireGuard tunnel - # ride the private "Direct Routing" NIC and work — but the mutual-auth control - # connection (cilium-agent -> peer cilium-agent on 4250) dials the peer's - # PRIMARY (public) IP. The Talos ingress firewall only admits 4240/4250/8472 - # from the private node CIDR (10.0.0.0/16, see - # talos/workers/allow-cilium-mutual-auth-ingress.yaml), so the public-sourced 4250 handshake - # is dropped (`dial :4250: i/o timeout`) and Cilium black-holes every - # cross-node pod flow the hetzner require-mutual-auth policy marks - # `authentication.mode: required` once its auth-cache entry lapses. Observed - # live (2026-06-19): the Crossview dashboard was unreachable — authenticated - # oauth2-proxy/auth-proxy -> crossview:3001 SYNs returned `policy-verdict: - # none ... Policy denied DROPPED` (the spire AUTH TYPE on that policy entry - # never completed its handshake) — while older dashboards limped on still- - # cached auth entries. Failures spanned all three workers, dialing each - # other's public IPs on 4250. - # - # Listing the private NICs makes the private IP each node's primary address, - # so the 4250 handshake uses 10.0.1.x and the firewall admits it. - # - # Safe on this topology (all verified live, 2026-06-19): - # - Ingress: the Hetzner LB reaches nodes on their PRIVATE IPs - # (load-balancer.hetzner.cloud/use-private-ip: "true", see - # ../../../patches/attach-hcloud-load-balancer.yaml) and clients hit the LB's public IP, not - # node public IPs. The private NIC stays a Cilium device, so the LB's - # NodePort BPF is unaffected; only unused public-NIC NodePort BPF is shed. - # - Egress: pod -> internet masquerade is the device-AGNOSTIC iptables rule - # (-s podCIDR ! -d podCIDR ! -o cilium_+ -j MASQUERADE), not scoped to - # this devices list, so pods still SNAT out the public NIC. (Masquerading - # is iptables here; bpf.masquerade is off.) - # - cilium-health / VXLAN / WireGuard already used the private NIC. - # - # WARNING: hard-codes the private-NIC names. Today every worker is enp7s0 and - # every control-plane is eth1 (verified on all 6 nodes). An autoscaler worker - # brought up on a server type whose private NIC is named differently would - # lose Cilium networking until its name is added here — keep this in sync with - # the node pool's machine type. - # - # PROD/HETZNER ONLY: the docker/local provider is single-NIC, so device - # autodetection is correct there. This pin lives in the hetzner overlay. - devices: "enp7s0 eth1" + # NB: the private-NIC `devices` pin that used to live here moved to + # ../components/private-nic-devices/ (same value, so the rendered + # HelmRelease is unchanged). It had to become a component so the staged + # homogeneous-devices/ successor can override it — a value carried by this + # parent patch is applied AFTER all component patches and can never be + # overridden by one. The pin's full rationale (2026-06-19 mutual-auth + # incident, LB/egress safety analysis) and the refreshed 2026-07-18 + # NIC inventory live in that component's header. diff --git a/k8s/providers/hetzner/infrastructure/controllers/kustomization.yaml b/k8s/providers/hetzner/infrastructure/controllers/kustomization.yaml index 347ed4e21..eb4e185ab 100644 --- a/k8s/providers/hetzner/infrastructure/controllers/kustomization.yaml +++ b/k8s/providers/hetzner/infrastructure/controllers/kustomization.yaml @@ -66,22 +66,38 @@ components: # silently-broken keda external-scaler and homepage Deployments. - ../../../../bases/components/helmrelease-drift-detection - ../../../../bases/components/helmrelease-flux-defaults + # ALWAYS-ON: the live private-NIC Cilium device pin (moved out of the + # strict-mode patch so the staged homogeneous-devices/ successor below can + # override it — parent `patches:` apply after component patches, so a value + # they carry is un-overridable by any component; see the component header). + # Keep referenced until homogeneous-devices/ is activated AND soaked. + - cilium/components/private-nic-devices/ + # Default-off homogeneous device selection (#2610): widens the device set to + # the public default-route NICs via wildcards ("en+ eth+") so + # ebpf-host-routing's BPF masquerade can attach where pod egress actually + # leaves. MUST stay listed AFTER private-nic-devices/ when enabled (later + # component wins). The component also sets updateStrategy OnDelete, so + # enabling it stages config without rolling any agent — the operator steps + # the roll pod-by-pod per the component's runbook (verify each node keeps a + # private primary address before the next). Rollback = re-comment (stepped + # nodes roll back automatically under the restored RollingUpdate). + # - cilium/components/homogeneous-devices/ # Default-off Cilium bandwidth-manager + host-only BBR staging component # (#2607). It is not activation-ready: #2609 must prove pod-BBR datapath - # prerequisites and #2610 must make device selection homogeneous first. + # prerequisites and #2610's homogeneous-devices/ must be activated first. # Enabling replaces the node qdisc and rolls every Cilium agent; after both # guards close, use a separate low-traffic activation. Rollback = re-comment. # - cilium/components/bandwidth-manager-bbr/ # Default-off eBPF host-routing/masquerade prerequisite staging for pod BBR # (#2609). Changes the NAT path for ALL pod egress and rolls every Cilium - # agent when enabled. HARD BLOCKER: do NOT enable until #2610 extends the - # pinned device list to the public default-route NICs — today's private-only - # `devices: "enp7s0 eth1"` would leave public egress without a masquerade - # program while iptables masquerade is off, breaking external egress - # cluster-wide (see the component's DEVICE section). Then activate on its - # own in a low-traffic window (before bandwidth-manager-bbr's host-only - # guard may be lifted), verify `cilium status --verbose` + Hubble for - # drops. Rollback = re-comment. + # agent when enabled. HARD BLOCKER: do NOT enable until #2610's + # homogeneous-devices/ component (above) has been ACTIVATED and soaked — + # under the private-only pin, public egress would get no masquerade program + # while iptables masquerade is off, breaking external egress cluster-wide + # (see the component's DEVICE section). Then activate on its own in a + # low-traffic window (before bandwidth-manager-bbr's host-only guard may be + # lifted), verify `cilium status --verbose` + Hubble for drops. + # Rollback = re-comment. # - cilium/components/ebpf-host-routing/ patches: - path: openbao/patches/store-data-on-hcloud.yaml