Incident Analysis: Cluster-wide Outage due to Zombie Worker Nodes and CNI/Storage Deadlocks

[cybersiddhu]

Incident Overview

On Friday, January 23, 2026, the Kubernetes cluster dictycr-dev-staging-dcr-experiments-k8s.local entered a degraded state where 50% of the worker nodes became unresponsive. This led to a cascading failure affecting networking (Cilium) and storage (Persistent Volumes).

Root Cause Analysis

1. Zombie Worker Nodes: Two worker nodes (nodes-us-central1-c-1mzf and nodes-us-central1-c-9psh) entered a state where the GCE instances were RUNNING, but the Kubelet service was unresponsive. This prevented the control plane from managing or draining them.
2. Orchestration Failure: kops rolling-update failed because it could not perform a graceful kubectl drain on the unresponsive nodes.
3. CNI Identity Synchronization: Forced node replacement resulted in a network partition. New nodes were unable to communicate with legacy nodes due to out-of-sync Cilium identities and routing tables.
4. Persistent Volume Deadlocks: Essential services like redis and logto failed to recover due to "Multi-Attach" errors. The Google Cloud API continued to report disks as attached to the deleted "zombie" instances.

Resolution Steps

1. Manual Node Purge: Manually deleted all 4 legacy worker instances in Google Cloud to force a clean cluster state.
2. Network Data-Plane Reset: Re-synchronized the CNI by performing a coordinated rollout of cilium-operator, the cilium DaemonSet, and coredns.
3. Manual Volume Unlock: Manually terminated lingering pod objects and scaled legacy ReplicaSets to 0 to release GCE Persistent Disks.
4. Service Restoration: Coordinated restart of backend microservices (annotation, content, stock, redis) and the graphql-api-server.

Key Insights & Recommendations

• Avoid Deadlock Drains: If a node is NotReady and SSH is unresponsive, bypass kops orchestration and delete the cloud instance immediately.
• Unified Data Plane: For CNI issues involving node rotation, a full worker pool refresh is often faster and more reliable than a piecemeal one.
• Monitoring Improvements: Implement alerting for sustained NotReady node conditions (>15 mins) to catch "zombie" states before they impact the entire cluster.
• Storage Management: Be prepared to manually scale down legacy ReplicaSets to 0 during rolling updates involving Persistent Volumes if "Multi-Attach" errors persist.

---

Analyzed and resolved with help from gemini-cli

[cybersiddhu]

Phase 3: Reliability Hardening & Observability

To prevent recurrence and improve cluster elasticity, the following hardening measures have been applied:

1. Node Problem Detector (NPD):

   • Action: Enabled nodeProblemDetector in the cluster spec.
   • Impact: The control plane now receives real-time signals from kernel logs (dmesg, journald) regarding hardware failures or kernel deadlocks. This ensures "zombie" nodes are detected and tainted faster.

2. Elastic Scalability:

   • Action: Increased Instance Group nodes-us-central1-c capacity:
     • minSize: 4 (Unchanged)
     • maxSize: 6 (Increased from 4)
   • Action: Enabled Cluster Autoscaler.
   • Impact: The cluster now has "surge" capacity. If a node fails or maintenance is required, the autoscaler can immediately provision replacement nodes without hitting a hard ceiling, preventing resource starvation during recovery.

3. Restored Observability:

   • Action: Enabled Metrics Server.
   • Impact: kubectl top nodes and Horizontal Pod Autoscaling (HPA) are now functional. Real-time resource utilization is visible for all nodes.

Current Status

• Cluster Health: Stable / Healthy
• Node Count: 4/4 (Ready)
• Autoscaler: Active (0 unneeded, 0 unschedulable)
• Workloads: All critical services (graphql, logto, redis, cilium) are Running and Healthy.

[cybersiddhu]

Phase 4: Monitoring & Alerting Setup Guide

To receive alerts for node issues (e.g., "Zombie Nodes", Kernel Deadlocks), you must bridge the Kubernetes API metrics to Google Cloud Monitoring.

Since this is a kops-managed cluster (not GKE), you must manually install the collection stack.

Architecture

Node Problem Detector (Events) -> kube-state-metrics (Metrics) -> Google Managed Prometheus (Collector) -> Google Cloud Monitoring (Alerts)

1. Install kube-state-metrics (KSM)

This component generates the metrics for node conditions.

Software: kube-state-metrics
Installation:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install kube-state-metrics prometheus-community/kube-state-metrics \
  --namespace monitoring --create-namespace

2. Install Google Managed Service for Prometheus (GMP)

This agent scrapes the metrics and pushes them to Google Cloud.

Software: Google Cloud Managed Service for Prometheus
Installation:

# 1. Enable the API
gcloud services enable monitoring.googleapis.com

# 2. Install the Managed Collection agent
kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/prometheus-engine/v0.8.2/manifests/setup.yaml
kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/prometheus-engine/v0.8.2/manifests/operator.yaml

3. Configure the Scraper

Tell the GMP agent to scrape kube-state-metrics.

Manifest (pod-monitoring.yaml):

apiVersion: monitoring.googleapis.com/v1
kind: PodMonitoring
metadata:
  name: kube-state-metrics-scraper
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: kube-state-metrics
  endpoints:
  - port: http
    interval: 30s

Apply:

kubectl apply -f pod-monitoring.yaml

4. Create the Alert Policy

In Google Cloud Console -> Monitoring -> Alerting:

Query (PromQL):

kube_node_status_condition{condition="KernelDeadlock", status="true"} > 0
OR
kube_node_status_condition{condition="Ready", status="unknown"} > 0

Trigger: Any time series > 0 for 5 minutes.
Notification: Email / Slack / PagerDuty.