From 4a21498b0d10d002f25df32e4a7bce428b394cc3 Mon Sep 17 00:00:00 2001 From: Matt2_Chang Date: Wed, 1 Jul 2026 17:48:58 +0800 Subject: [PATCH 01/13] docs: clarify PVC retention on StatefulSet scale-down AS IS: the scale-to-2 exercise showed the ordinal pattern appearing but never showed what happens to the PVC when scaling back down TO BE: adds a kubectl get pvc check and an explanation that pgdata-postgres-1 survives scale-down and gets reattached on scale-up, reinforcing the "PVCs outlive the Pod" lesson from the earlier delete/reattach test TASK # N/A --- docs/config-and-data/statefulset.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/config-and-data/statefulset.md b/docs/config-and-data/statefulset.md index ecea06b..dbce13a 100644 --- a/docs/config-and-data/statefulset.md +++ b/docs/config-and-data/statefulset.md @@ -100,10 +100,13 @@ Scale to two replicas and you will see the ordinal pattern repeat: `postgres-1` kubectl scale statefulset/postgres --replicas=2 kubectl get pods,pvc kubectl scale statefulset/postgres --replicas=1 +kubectl get pvc # pgdata-postgres-1 is still here ``` That second Pod is another independent PostgreSQL instance with its own volume. This lab does not configure PostgreSQL replication. +Scaling down only removes the Pod — `pgdata-postgres-1` is **not** deleted. If you scale back to 2, `postgres-1` reattaches to that same PVC instead of getting a fresh volume. This is the same "PVCs outlive the Pod" behavior as the delete/reattach test above, just triggered by scaling instead of a manual delete. + By default, StatefulSets use ordered lifecycle behavior: Kubernetes creates `postgres-0` before `postgres-1`, and deletes higher ordinals first when scaling down. The field behind that default is `podManagementPolicy: OrderedReady`. Updates are ordered too. The default `updateStrategy: RollingUpdate` replaces Pods from the highest ordinal down, so `postgres-1` updates before `postgres-0`. From 3ac196031d56652d66afe3d304cddb1d5093a773 Mon Sep 17 00:00:00 2001 From: Matt2_Chang Date: Wed, 1 Jul 2026 17:51:01 +0800 Subject: [PATCH 02/13] docs: ground ordered lifecycle claim in a concrete example MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit AS IS: the doc asserts ordered Pod creation "matters for clustered databases" without showing why TO BE: adds the pg_basebackup primary/replica pattern to show what breaks without OrderedReady — a replica bootstrapping before the primary's DNS is resolvable TASK # N/A --- docs/config-and-data/statefulset.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/config-and-data/statefulset.md b/docs/config-and-data/statefulset.md index dbce13a..469e66d 100644 --- a/docs/config-and-data/statefulset.md +++ b/docs/config-and-data/statefulset.md @@ -109,6 +109,8 @@ Scaling down only removes the Pod — `pgdata-postgres-1` is **not** deleted. If By default, StatefulSets use ordered lifecycle behavior: Kubernetes creates `postgres-0` before `postgres-1`, and deletes higher ordinals first when scaling down. The field behind that default is `podManagementPolicy: OrderedReady`. +This ordering isn't just cosmetic — it's what makes primary/replica database clustering possible. A classic pattern: `postgres-1`'s startup script runs `pg_basebackup` against the primary at `postgres-0.postgres.default.svc.cluster.local` to clone its data before joining as a replica. If `postgres-1` came up before `postgres-0` was Ready, that DNS name wouldn't resolve yet and the replica would fail to bootstrap. `OrderedReady` guarantees the primary exists and is healthy first, every time. + Updates are ordered too. The default `updateStrategy: RollingUpdate` replaces Pods from the highest ordinal down, so `postgres-1` updates before `postgres-0`. `PGDATA` points PostgreSQL at a subdirectory inside the mounted volume. That avoids a common lab failure where the image expects to initialize an empty data directory but the volume mount contains filesystem metadata. From ea642a9d42c986460457ed2ddb2b1ae540fd1647 Mon Sep 17 00:00:00 2001 From: Matt2_Chang Date: Wed, 1 Jul 2026 18:04:39 +0800 Subject: [PATCH 03/13] docs: prove PVC persistence with real data, not just Pod state AS IS: the delete/reattach exercise only checked that postgres-0 came back and pvc stayed Bound, which doesn't actually prove data survived TO BE: adds readiness/pg_isready waits, a DNS lookup check, and a real insert-then-verify test so the exercise proves data persistence instead of just Pod identity TASK # N/A --- docs/config-and-data/statefulset.md | 29 +++++++++++++++++++++++++++-- 1 file changed, 27 insertions(+), 2 deletions(-) diff --git a/docs/config-and-data/statefulset.md b/docs/config-and-data/statefulset.md index 469e66d..3a45a35 100644 --- a/docs/config-and-data/statefulset.md +++ b/docs/config-and-data/statefulset.md @@ -70,7 +70,10 @@ spec: ``` ```bash +kubectl apply -f manifests/config-and-data/app-secret.yaml kubectl apply -f manifests/config-and-data/postgres-statefulset.yaml +kubectl wait --for=condition=Ready pod/postgres-0 --timeout=120s +kubectl exec postgres-0 -- sh -c 'until pg_isready -U appuser; do sleep 2; done' kubectl get statefulset,pods -l app=postgres kubectl get pvc pgdata-postgres-0 kubectl exec postgres-0 -- hostname @@ -87,11 +90,33 @@ With the headless Service, that same Pod also gets a stable DNS name: postgres-0.postgres.default.svc.cluster.local ``` -Delete `postgres-0` and watch (in [k9s](../getting-started/k9s.md), `:sts` / `:pods`): it comes back with the **same name** and **reattaches the same PVC** — its data is intact. +You can verify that DNS name from another Pod: + +```bash +kubectl run dns-test --image=busybox:1.36 --restart=Never --command -- sleep 3600 +kubectl wait --for=condition=Ready pod/dns-test --timeout=60s +kubectl exec dns-test -- nslookup postgres-0.postgres.default.svc.cluster.local +kubectl delete pod dns-test +``` + +Now write a row into PostgreSQL so the persistence test proves more than "the Pod came back": + +```bash +kubectl exec postgres-0 -- psql -U appuser -d appuser -c \ + "CREATE TABLE IF NOT EXISTS statefulset_lab (id int primary key, note text);" +kubectl exec postgres-0 -- psql -U appuser -d appuser -c \ + "INSERT INTO statefulset_lab VALUES (1, 'survived a Pod delete') ON CONFLICT (id) DO UPDATE SET note = EXCLUDED.note;" +kubectl exec postgres-0 -- psql -U appuser -d appuser -c "SELECT * FROM statefulset_lab;" +``` + +Delete `postgres-0` and watch (in [k9s](../getting-started/k9s.md), `:sts` / `:pods`): it comes back with the **same name** and **reattaches the same PVC**. ```bash kubectl delete pod postgres-0 +kubectl wait --for=condition=Ready pod/postgres-0 --timeout=120s +kubectl exec postgres-0 -- sh -c 'until pg_isready -U appuser; do sleep 2; done' kubectl get pods,pvc +kubectl exec postgres-0 -- psql -U appuser -d appuser -c "SELECT * FROM statefulset_lab;" ``` Scale to two replicas and you will see the ordinal pattern repeat: `postgres-1` appears with its own `pgdata-postgres-1` PVC. @@ -109,7 +134,7 @@ Scaling down only removes the Pod — `pgdata-postgres-1` is **not** deleted. If By default, StatefulSets use ordered lifecycle behavior: Kubernetes creates `postgres-0` before `postgres-1`, and deletes higher ordinals first when scaling down. The field behind that default is `podManagementPolicy: OrderedReady`. -This ordering isn't just cosmetic — it's what makes primary/replica database clustering possible. A classic pattern: `postgres-1`'s startup script runs `pg_basebackup` against the primary at `postgres-0.postgres.default.svc.cluster.local` to clone its data before joining as a replica. If `postgres-1` came up before `postgres-0` was Ready, that DNS name wouldn't resolve yet and the replica would fail to bootstrap. `OrderedReady` guarantees the primary exists and is healthy first, every time. +This ordering isn't just cosmetic — it's what makes primary/replica database clustering possible. A classic pattern: `postgres-1`'s startup script runs `pg_basebackup` against the primary at `postgres-0.postgres.default.svc.cluster.local` to clone its data before joining as a replica. If `postgres-1` came up before `postgres-0` was Ready, the primary might not be discoverable or accepting connections yet, and the replica bootstrap could fail. `OrderedReady` makes Kubernetes wait for the lower ordinal Pod to become Ready before creating the next one. Updates are ordered too. The default `updateStrategy: RollingUpdate` replaces Pods from the highest ordinal down, so `postgres-1` updates before `postgres-0`. From ba63135a0eb310c14708ea345ca29deafebd6270 Mon Sep 17 00:00:00 2001 From: Matt2_Chang Date: Thu, 2 Jul 2026 09:51:21 +0800 Subject: [PATCH 04/13] docs: add readiness/liveness probes and resource limits to statefulset lab AS IS: the postgres StatefulSet had no probes or resource requests/limits, so kubectl wait reported Ready before postgres could accept connections, requiring a manual pg_isready polling loop as a workaround TO BE: add pg_isready-based readinessProbe/livenessProbe and cpu/memory requests/limits so the Pod's Ready condition reflects real DB availability, plus comments noting production should pin the image version and choose an explicit StorageClass TASK # N/A --- docs/config-and-data/statefulset.md | 26 ++++++++++++++++++- .../config-and-data/postgres-statefulset.yaml | 21 ++++++++++++++- 2 files changed, 45 insertions(+), 2 deletions(-) diff --git a/docs/config-and-data/statefulset.md b/docs/config-and-data/statefulset.md index 3a45a35..351fc4e 100644 --- a/docs/config-and-data/statefulset.md +++ b/docs/config-and-data/statefulset.md @@ -45,7 +45,10 @@ spec: spec: containers: - name: postgres - image: postgres:16 + image: postgres:16 # lab image tag; pin a patch version/digest in production + ports: + - containerPort: 5432 + name: postgres env: - name: POSTGRES_USER valueFrom: @@ -62,10 +65,28 @@ spec: volumeMounts: - name: pgdata mountPath: /var/lib/postgresql/data + readinessProbe: + exec: + command: ["pg_isready", "-U", "appuser"] + initialDelaySeconds: 5 + periodSeconds: 5 + livenessProbe: + exec: + command: ["pg_isready", "-U", "appuser"] + initialDelaySeconds: 30 + periodSeconds: 10 + resources: + requests: + cpu: 100m + memory: 256Mi + limits: + cpu: 500m + memory: 512Mi volumeClaimTemplates: # ← the key difference: a PVC PER replica - metadata: { name: pgdata } spec: accessModes: ["ReadWriteOnce"] + # storageClassName omitted: use the cluster's default StorageClass for this lab. resources: { requests: { storage: 1Gi } } ``` @@ -140,6 +161,8 @@ Updates are ordered too. The default `updateStrategy: RollingUpdate` replaces Po `PGDATA` points PostgreSQL at a subdirectory inside the mounted volume. That avoids a common lab failure where the image expects to initialize an empty data directory but the volume mount contains filesystem metadata. +The probes use `pg_isready` so Kubernetes does not mark the Pod Ready until PostgreSQL is accepting connections, and can restart the container if the process stops responding. The resource requests keep the scheduler honest for this lab; production sizing should come from load tests and monitoring. + ## When (not) to use one - ✅ Databases, message queues, anything where each replica owns data or needs a stable identity. @@ -150,6 +173,7 @@ Updates are ordered too. The default `updateStrategy: RollingUpdate` replaces Po - **Don't run production databases in-cluster casually** — managed DBs remove a lot of operational pain. Use a StatefulSet when you have a real reason to self-host. - **Always pair with a headless Service** (`clusterIP: None`) for stable per-Pod DNS. - **Size `volumeClaimTemplates` carefully** — they're created per replica and usually persist even after the StatefulSet is deleted (so data isn't lost by accident). +- **Pin production images and storage deliberately** — this lab uses `postgres:16` and the default StorageClass for portability, but production manifests should pin image versions/digests and choose an explicit storage class. ## Reset after Part 2 diff --git a/manifests/config-and-data/postgres-statefulset.yaml b/manifests/config-and-data/postgres-statefulset.yaml index 929161d..889a385 100644 --- a/manifests/config-and-data/postgres-statefulset.yaml +++ b/manifests/config-and-data/postgres-statefulset.yaml @@ -26,9 +26,10 @@ spec: spec: containers: - name: postgres - image: postgres:16 + image: postgres:16 # lab image tag; pin a patch version/digest in production ports: - containerPort: 5432 + name: postgres env: - name: POSTGRES_USER valueFrom: @@ -45,11 +46,29 @@ spec: volumeMounts: - name: pgdata mountPath: /var/lib/postgresql/data + readinessProbe: + exec: + command: ["pg_isready", "-U", "appuser"] + initialDelaySeconds: 5 + periodSeconds: 5 + livenessProbe: + exec: + command: ["pg_isready", "-U", "appuser"] + initialDelaySeconds: 30 + periodSeconds: 10 + resources: + requests: + cpu: 100m + memory: 256Mi + limits: + cpu: 500m + memory: 512Mi volumeClaimTemplates: # each replica gets its OWN PVC - metadata: name: pgdata spec: accessModes: ["ReadWriteOnce"] + # storageClassName omitted: use the cluster's default StorageClass for this lab. resources: requests: storage: 1Gi From 0b6de6a9e03ba4c1054a048e894403ddb24d326d Mon Sep 17 00:00:00 2001 From: Matt2_Chang Date: Thu, 2 Jul 2026 09:52:41 +0800 Subject: [PATCH 05/13] docs: harden statefulset lab with security context and startup probe AS IS: the postgres container ran as root with no startup grace period, and probe commands hardcoded "appuser" instead of reading POSTGRES_USER TO BE: run as the image's non-root user via securityContext/fsGroup, add a startupProbe so initialization isn't cut short by liveness checks, and have probes read $POSTGRES_USER so they stay in sync with the Secret TASK # N/A --- docs/config-and-data/statefulset.md | 23 +++++++++++++++---- .../config-and-data/postgres-statefulset.yaml | 19 ++++++++++++--- 2 files changed, 35 insertions(+), 7 deletions(-) diff --git a/docs/config-and-data/statefulset.md b/docs/config-and-data/statefulset.md index 351fc4e..31fa9a6 100644 --- a/docs/config-and-data/statefulset.md +++ b/docs/config-and-data/statefulset.md @@ -28,7 +28,9 @@ spec: selector: app: postgres ports: - - port: 5432 + - name: postgres + port: 5432 + targetPort: postgres --- apiVersion: apps/v1 kind: StatefulSet @@ -43,6 +45,12 @@ spec: metadata: labels: { app: postgres } spec: + terminationGracePeriodSeconds: 30 + securityContext: + runAsNonRoot: true + runAsUser: 999 + runAsGroup: 999 + fsGroup: 999 containers: - name: postgres image: postgres:16 # lab image tag; pin a patch version/digest in production @@ -65,14 +73,19 @@ spec: volumeMounts: - name: pgdata mountPath: /var/lib/postgresql/data + startupProbe: + exec: + command: ["sh", "-c", "pg_isready -U \"$POSTGRES_USER\""] + failureThreshold: 30 + periodSeconds: 5 readinessProbe: exec: - command: ["pg_isready", "-U", "appuser"] + command: ["sh", "-c", "pg_isready -U \"$POSTGRES_USER\""] initialDelaySeconds: 5 periodSeconds: 5 livenessProbe: exec: - command: ["pg_isready", "-U", "appuser"] + command: ["sh", "-c", "pg_isready -U \"$POSTGRES_USER\""] initialDelaySeconds: 30 periodSeconds: 10 resources: @@ -161,7 +174,9 @@ Updates are ordered too. The default `updateStrategy: RollingUpdate` replaces Po `PGDATA` points PostgreSQL at a subdirectory inside the mounted volume. That avoids a common lab failure where the image expects to initialize an empty data directory but the volume mount contains filesystem metadata. -The probes use `pg_isready` so Kubernetes does not mark the Pod Ready until PostgreSQL is accepting connections, and can restart the container if the process stops responding. The resource requests keep the scheduler honest for this lab; production sizing should come from load tests and monitoring. +The probes use `pg_isready` so Kubernetes does not mark the Pod Ready until PostgreSQL is accepting connections, and can restart the container if the process stops responding. The `startupProbe` gives first-time initialization more room before liveness checks begin. The resource requests keep the scheduler honest for this lab; production sizing should come from load tests and monitoring. + +The pod security context runs PostgreSQL as the image's non-root `postgres` user (`999`) and uses `fsGroup` so the mounted volume is writable. This is still intentionally small for a tutorial; hardened production databases usually add stricter security policy, backups, monitoring, and tested restore procedures. ## When (not) to use one diff --git a/manifests/config-and-data/postgres-statefulset.yaml b/manifests/config-and-data/postgres-statefulset.yaml index 889a385..4426728 100644 --- a/manifests/config-and-data/postgres-statefulset.yaml +++ b/manifests/config-and-data/postgres-statefulset.yaml @@ -7,7 +7,9 @@ spec: selector: app: postgres ports: - - port: 5432 + - name: postgres + port: 5432 + targetPort: postgres --- apiVersion: apps/v1 kind: StatefulSet @@ -24,6 +26,12 @@ spec: labels: app: postgres spec: + terminationGracePeriodSeconds: 30 + securityContext: + runAsNonRoot: true + runAsUser: 999 + runAsGroup: 999 + fsGroup: 999 containers: - name: postgres image: postgres:16 # lab image tag; pin a patch version/digest in production @@ -46,14 +54,19 @@ spec: volumeMounts: - name: pgdata mountPath: /var/lib/postgresql/data + startupProbe: + exec: + command: ["sh", "-c", "pg_isready -U \"$POSTGRES_USER\""] + failureThreshold: 30 + periodSeconds: 5 readinessProbe: exec: - command: ["pg_isready", "-U", "appuser"] + command: ["sh", "-c", "pg_isready -U \"$POSTGRES_USER\""] initialDelaySeconds: 5 periodSeconds: 5 livenessProbe: exec: - command: ["pg_isready", "-U", "appuser"] + command: ["sh", "-c", "pg_isready -U \"$POSTGRES_USER\""] initialDelaySeconds: 30 periodSeconds: 10 resources: From 96124b1f21baa7bf9f94625bf1c9d65b5b717fb2 Mon Sep 17 00:00:00 2001 From: Matt2_Chang Date: Thu, 2 Jul 2026 09:54:35 +0800 Subject: [PATCH 06/13] docs: extend postgres wait timeout to match startupProbe window AS IS: exercises used kubectl wait --timeout=120s, shorter than the startupProbe's worst-case 150s window (failureThreshold 30 * periodSeconds 5), so a slow first-time initdb could time out the wait before the probe itself gave up TO BE: bump the wait timeout to 180s so it comfortably covers the startupProbe's worst case TASK # N/A --- docs/config-and-data/statefulset.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/config-and-data/statefulset.md b/docs/config-and-data/statefulset.md index 31fa9a6..b8fec7f 100644 --- a/docs/config-and-data/statefulset.md +++ b/docs/config-and-data/statefulset.md @@ -106,7 +106,7 @@ spec: ```bash kubectl apply -f manifests/config-and-data/app-secret.yaml kubectl apply -f manifests/config-and-data/postgres-statefulset.yaml -kubectl wait --for=condition=Ready pod/postgres-0 --timeout=120s +kubectl wait --for=condition=Ready pod/postgres-0 --timeout=180s kubectl exec postgres-0 -- sh -c 'until pg_isready -U appuser; do sleep 2; done' kubectl get statefulset,pods -l app=postgres kubectl get pvc pgdata-postgres-0 @@ -147,7 +147,7 @@ Delete `postgres-0` and watch (in [k9s](../getting-started/k9s.md), `:sts` / `:p ```bash kubectl delete pod postgres-0 -kubectl wait --for=condition=Ready pod/postgres-0 --timeout=120s +kubectl wait --for=condition=Ready pod/postgres-0 --timeout=180s kubectl exec postgres-0 -- sh -c 'until pg_isready -U appuser; do sleep 2; done' kubectl get pods,pvc kubectl exec postgres-0 -- psql -U appuser -d appuser -c "SELECT * FROM statefulset_lab;" From 84071c73a70f66585955d97d89c0c0468d166dad Mon Sep 17 00:00:00 2001 From: Matt2_Chang Date: Thu, 2 Jul 2026 10:12:50 +0800 Subject: [PATCH 07/13] docs: explain headless Service concept before the StatefulSet example AS IS: the doc used "headless Service" and referenced per-Pod DNS names without ever defining what makes a headless Service different, leaving readers to infer it from the manifest alone TO BE: add a dedicated section defining clusterIP: None, a comparison table against a normal Service, and why StatefulSets need per-Pod DNS instead of load-balanced traffic TASK # N/A --- docs/config-and-data/statefulset.md | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/docs/config-and-data/statefulset.md b/docs/config-and-data/statefulset.md index b8fec7f..7172c04 100644 --- a/docs/config-and-data/statefulset.md +++ b/docs/config-and-data/statefulset.md @@ -16,6 +16,25 @@ It is also a **single-replica lab**, not high availability PostgreSQL. Scaling t - **Stable per-Pod storage** — a `volumeClaimTemplate` gives **each replica its own [PVC](volumes.md)** (`pgdata-postgres-0`, …). When `postgres-0` restarts, it reattaches to *its* volume, not a sibling's. - **Ordered, predictable lifecycle** — Pods are created/updated/deleted in order (`-0`, then `-1`, …), which matters for clustered databases. +### What a headless Service is + +A **headless Service** is a Service with `clusterIP: None` — it gets no virtual IP and does no load balancing. Instead of one DNS name that resolves to a single (load-balanced) IP, DNS resolves it to the IPs of **every matching Pod** individually. + +| | Normal Service | Headless Service | +|---|---|---| +| ClusterIP | Virtual IP assigned | `None` | +| DNS lookup returns | One A record (the virtual IP) | One A record per matching Pod | +| Traffic routing | kube-proxy load-balances to any backing Pod | Client resolves a specific Pod's IP/DNS name itself | + +Paired with a StatefulSet, this gives each Pod its own stable DNS name: + +``` +postgres-0.postgres.default.svc.cluster.local +postgres-1.postgres.default.svc.cluster.local +``` + +That matters because StatefulSet Pods are not interchangeable — you need to reach *a specific* replica (e.g. the primary at `postgres-0`), not "any Pod behind the Service," which is all a normal (non-headless) Service gives you. + ▶ **Runnable manifest:** [`manifests/config-and-data/postgres-statefulset.yaml`](../../manifests/config-and-data/postgres-statefulset.yaml) (a headless Service + StatefulSet; needs `app-secret` and a default StorageClass — see [Volumes](volumes.md)) ```yaml From c68bf185299ffb8b2a7b0bd385c5b79c24ba82d1 Mon Sep 17 00:00:00 2001 From: Matt2_Chang Date: Thu, 2 Jul 2026 10:31:09 +0800 Subject: [PATCH 08/13] docs: clarify default namespace in the StatefulSet DNS name AS IS: the DNS example postgres-0.postgres.default.svc.cluster.local was shown with no explanation, so "postgres" appearing twice and "default" could easily be misread as parts of the Service name TO BE: break down the .. format and note the manifests never set metadata.namespace, so default comes from the current kubectl context, not a hardcoded value TASK # N/A --- docs/config-and-data/statefulset.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/config-and-data/statefulset.md b/docs/config-and-data/statefulset.md index 7172c04..3acf719 100644 --- a/docs/config-and-data/statefulset.md +++ b/docs/config-and-data/statefulset.md @@ -143,6 +143,8 @@ With the headless Service, that same Pod also gets a stable DNS name: postgres-0.postgres.default.svc.cluster.local ``` +That name has the form `...svc.cluster.local`. `postgres` appears twice — once as the Pod's name, once as the Service's `metadata.name` — and `default` is the **namespace**, not part of the Service name. Neither manifest sets `metadata.namespace`, so `kubectl apply` used whatever namespace your current context defaults to; this tutorial has stayed on `default` throughout. Deploy into another namespace (e.g. `-n mydb`) and the DNS name changes to `postgres-0.postgres.mydb.svc.cluster.local` accordingly. + You can verify that DNS name from another Pod: ```bash From f4acb1e88fe671afb0cf932785b10bb8f26f0d3f Mon Sep 17 00:00:00 2001 From: Matt2_Chang Date: Thu, 2 Jul 2026 10:36:40 +0800 Subject: [PATCH 09/13] docs: split statefulset example into core shape and hardening notes AS IS: the single 84-line YAML mixed core StatefulSet mechanics with probes, securityContext, and resource limits, making it hard to see identity/PVC/ordering without wading through production-hardening noise TO BE: trim the walkthrough YAML to just the three core StatefulSet concepts, and move the hardening fields into a separate section with a field-by-field explanation table plus their own YAML snippet TASK # N/A --- docs/config-and-data/statefulset.md | 82 ++++++++++++++++++----------- 1 file changed, 52 insertions(+), 30 deletions(-) diff --git a/docs/config-and-data/statefulset.md b/docs/config-and-data/statefulset.md index 3acf719..af3f32f 100644 --- a/docs/config-and-data/statefulset.md +++ b/docs/config-and-data/statefulset.md @@ -37,6 +37,8 @@ That matters because StatefulSet Pods are not interchangeable — you need to re ▶ **Runnable manifest:** [`manifests/config-and-data/postgres-statefulset.yaml`](../../manifests/config-and-data/postgres-statefulset.yaml) (a headless Service + StatefulSet; needs `app-secret` and a default StorageClass — see [Volumes](volumes.md)) +Here's the core shape — just enough to see the three things a StatefulSet adds. (The runnable file also adds production-style hardening — probes, a security context, resource limits — explained separately in [Hardening this lab](#hardening-this-lab) below, after the core mechanics make sense.) + ```yaml apiVersion: v1 kind: Service @@ -64,12 +66,6 @@ spec: metadata: labels: { app: postgres } spec: - terminationGracePeriodSeconds: 30 - securityContext: - runAsNonRoot: true - runAsUser: 999 - runAsGroup: 999 - fsGroup: 999 containers: - name: postgres image: postgres:16 # lab image tag; pin a patch version/digest in production @@ -92,28 +88,6 @@ spec: volumeMounts: - name: pgdata mountPath: /var/lib/postgresql/data - startupProbe: - exec: - command: ["sh", "-c", "pg_isready -U \"$POSTGRES_USER\""] - failureThreshold: 30 - periodSeconds: 5 - readinessProbe: - exec: - command: ["sh", "-c", "pg_isready -U \"$POSTGRES_USER\""] - initialDelaySeconds: 5 - periodSeconds: 5 - livenessProbe: - exec: - command: ["sh", "-c", "pg_isready -U \"$POSTGRES_USER\""] - initialDelaySeconds: 30 - periodSeconds: 10 - resources: - requests: - cpu: 100m - memory: 256Mi - limits: - cpu: 500m - memory: 512Mi volumeClaimTemplates: # ← the key difference: a PVC PER replica - metadata: { name: pgdata } spec: @@ -195,9 +169,57 @@ Updates are ordered too. The default `updateStrategy: RollingUpdate` replaces Po `PGDATA` points PostgreSQL at a subdirectory inside the mounted volume. That avoids a common lab failure where the image expects to initialize an empty data directory but the volume mount contains filesystem metadata. -The probes use `pg_isready` so Kubernetes does not mark the Pod Ready until PostgreSQL is accepting connections, and can restart the container if the process stops responding. The `startupProbe` gives first-time initialization more room before liveness checks begin. The resource requests keep the scheduler honest for this lab; production sizing should come from load tests and monitoring. +## Hardening this lab + +The core YAML above is enough to see stable identity, per-Pod storage, and ordering. The runnable manifest goes further and adds a few fields you'll see in most real StatefulSets. Here's what each one does, in plain terms: + +| Field | What it does | Why it's here | +|---|---|---| +| `terminationGracePeriodSeconds: 30` | Gives PostgreSQL 30s to shut down cleanly before Kubernetes force-kills it | An abrupt `SIGKILL` mid-write risks a corrupted or unclean data directory | +| `securityContext` (`runAsUser/Group: 999`, `fsGroup`) | Runs the container as the image's built-in non-root `postgres` user instead of root, and makes the mounted volume writable by that user's group | Running databases as root is an unnecessary privilege; this is the standard hardening for the official postgres image | +| `startupProbe` | Gives the container up to `30 × 5s = 150s` to finish first-time initialization before liveness checks start judging it | Without this, a slow `initdb` on first boot could get killed by the liveness probe before it ever finishes starting | +| `readinessProbe` | Runs `pg_isready` every 5s; the Pod isn't marked `Ready` (and won't receive traffic) until it passes | Without it, `kubectl wait --for=condition=Ready` would report Ready as soon as the container process starts — before PostgreSQL can actually accept connections | +| `livenessProbe` | Runs `pg_isready` every 10s once the Pod is up; if it keeps failing, Kubernetes restarts the container | Recovers automatically if PostgreSQL wedges or stops responding without the process exiting | +| `resources` (requests/limits) | Reserves 100m CPU / 256Mi memory, caps at 500m CPU / 512Mi memory | Keeps the scheduler informed and stops one runaway Pod from starving others on the node | + +Here's just the additional fields, layered onto the container spec and Pod spec from the core example above: + +```yaml + # Pod-level, alongside `containers:` + terminationGracePeriodSeconds: 30 + securityContext: + runAsNonRoot: true + runAsUser: 999 + runAsGroup: 999 + fsGroup: 999 + containers: + - name: postgres + # ...same image/env/volumeMounts as above... + startupProbe: + exec: + command: ["sh", "-c", "pg_isready -U \"$POSTGRES_USER\""] + failureThreshold: 30 + periodSeconds: 5 + readinessProbe: + exec: + command: ["sh", "-c", "pg_isready -U \"$POSTGRES_USER\""] + initialDelaySeconds: 5 + periodSeconds: 5 + livenessProbe: + exec: + command: ["sh", "-c", "pg_isready -U \"$POSTGRES_USER\""] + initialDelaySeconds: 30 + periodSeconds: 10 + resources: + requests: + cpu: 100m + memory: 256Mi + limits: + cpu: 500m + memory: 512Mi +``` -The pod security context runs PostgreSQL as the image's non-root `postgres` user (`999`) and uses `fsGroup` so the mounted volume is writable. This is still intentionally small for a tutorial; hardened production databases usually add stricter security policy, backups, monitoring, and tested restore procedures. +This is still a tutorial-sized lab, not a hardened production posture — real production databases add network policy, pod security standards, backups, monitoring, and tested restore procedures on top of this. ## When (not) to use one From 2a69aefcec0aa1ca9e146462d2aa91c0ec05ad78 Mon Sep 17 00:00:00 2001 From: Matt2_Chang Date: Thu, 2 Jul 2026 10:49:58 +0800 Subject: [PATCH 10/13] docs: explain volumeClaimTemplates before the StatefulSet example AS IS: the doc mentioned volumeClaimTemplates gives each replica its own PVC but never explained how that differs from a normal Pod's volumes: reference, or where the pgdata-postgres-N naming comes from TO BE: add a dedicated section contrasting shared-PVC volumes: against per-replica volumeClaimTemplates, with the naming convention and a comparison table, linking to the scaling exercise and reclaim-policy docs TASK # N/A --- docs/config-and-data/statefulset.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/docs/config-and-data/statefulset.md b/docs/config-and-data/statefulset.md index af3f32f..4bd630f 100644 --- a/docs/config-and-data/statefulset.md +++ b/docs/config-and-data/statefulset.md @@ -35,6 +35,20 @@ postgres-1.postgres.default.svc.cluster.local That matters because StatefulSet Pods are not interchangeable — you need to reach *a specific* replica (e.g. the primary at `postgres-0`), not "any Pod behind the Service," which is all a normal (non-headless) Service gives you. +### What a volumeClaimTemplate is + +A regular Pod (or Deployment) references an *already-existing* PVC by name in `volumes:` — every replica that mounts it shares that one PVC, which is why [Deployment replicas sharing one PVC](../core-objects/deployment.md) is a footgun for anything that writes data. + +`volumeClaimTemplates` is different: instead of naming one PVC, it's a *template* the StatefulSet controller uses to create a **new PVC per replica**, automatically, the first time each ordinal comes up. Kubernetes names each one `--` — hence `pgdata-postgres-0`, `pgdata-postgres-1`, … + +| | Pod/Deployment `volumes:` | StatefulSet `volumeClaimTemplates:` | +|---|---|---| +| PVC | You create it yourself, once | Controller creates one automatically per replica | +| Sharing | All replicas mount the **same** PVC | Each replica gets its **own** PVC | +| Naming | Whatever you called it | `--` | + +This is what makes `postgres-1` safe to run alongside `postgres-0` without them corrupting each other's data — see the `pgdata-postgres-0` / `pgdata-postgres-1` PVCs in the scaling exercise below. It's also why the PVCs **outlive** the StatefulSet: they aren't owned by a single Pod, so scaling down or deleting the StatefulSet doesn't delete them (see [Best practices](#best-practices) and [Volumes](volumes.md) for the full reclaim-policy story). + ▶ **Runnable manifest:** [`manifests/config-and-data/postgres-statefulset.yaml`](../../manifests/config-and-data/postgres-statefulset.yaml) (a headless Service + StatefulSet; needs `app-secret` and a default StorageClass — see [Volumes](volumes.md)) Here's the core shape — just enough to see the three things a StatefulSet adds. (The runnable file also adds production-style hardening — probes, a security context, resource limits — explained separately in [Hardening this lab](#hardening-this-lab) below, after the core mechanics make sense.) From d53f0226c17bead229af85d0277ca3a89e5c98a4 Mon Sep 17 00:00:00 2001 From: Matt2_Chang Date: Thu, 2 Jul 2026 10:51:53 +0800 Subject: [PATCH 11/13] fix: correct broken cross-reference in volumeClaimTemplate section MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit AS IS: the link to "Deployment replicas sharing one PVC" pointed at deployment.md, which never mentions PVCs at all — the actual content lives in volumes.md TO BE: point the link at volumes.md so readers land on the section that actually explains the shared-PVC footgun TASK # N/A --- docs/config-and-data/statefulset.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/config-and-data/statefulset.md b/docs/config-and-data/statefulset.md index 4bd630f..7e1d3b8 100644 --- a/docs/config-and-data/statefulset.md +++ b/docs/config-and-data/statefulset.md @@ -37,7 +37,7 @@ That matters because StatefulSet Pods are not interchangeable — you need to re ### What a volumeClaimTemplate is -A regular Pod (or Deployment) references an *already-existing* PVC by name in `volumes:` — every replica that mounts it shares that one PVC, which is why [Deployment replicas sharing one PVC](../core-objects/deployment.md) is a footgun for anything that writes data. +A regular Pod (or Deployment) references an *already-existing* PVC by name in `volumes:` — every replica that mounts it shares that one PVC. As [Volumes](volumes.md) explains, that's fine for read-only data but a footgun once multiple replicas write to it. `volumeClaimTemplates` is different: instead of naming one PVC, it's a *template* the StatefulSet controller uses to create a **new PVC per replica**, automatically, the first time each ordinal comes up. Kubernetes names each one `--` — hence `pgdata-postgres-0`, `pgdata-postgres-1`, … From a94dc3e16e1d92dfdb34c8ca4b0a9426d6c6310a Mon Sep 17 00:00:00 2001 From: Matt2_Chang Date: Thu, 2 Jul 2026 11:48:25 +0800 Subject: [PATCH 12/13] docs: explain why the headless Service precedes the StatefulSet AS IS: the manifest listed Service before StatefulSet with no explanation, so readers could reasonably assume the order was arbitrary or even backwards (workload before its supporting Service) TO BE: note that serviceName references the Service by name, so listing it first mirrors creating a ConfigMap/Secret before the Pod that reads it, and avoids a DNS race if the order were reversed TASK # N/A --- docs/config-and-data/statefulset.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/config-and-data/statefulset.md b/docs/config-and-data/statefulset.md index 7e1d3b8..e0e4a2c 100644 --- a/docs/config-and-data/statefulset.md +++ b/docs/config-and-data/statefulset.md @@ -110,6 +110,8 @@ spec: resources: { requests: { storage: 1Gi } } ``` +Notice the Service comes **before** the StatefulSet in the file. That's deliberate, not arbitrary: `spec.serviceName: postgres` refers to that Service by name, and the Service is what gives each Pod its DNS entry — same "the thing being depended on goes first" rule as creating a ConfigMap/Secret before the Pod that reads it. `kubectl apply -f` on a multi-document file sends resources in the order they appear, so this ordering means the governing Service already exists by the time the StatefulSet starts creating Pods. Reversing the order wouldn't hard-fail — the StatefulSet controller still creates Pods even if `serviceName` doesn't resolve yet — but per-Pod DNS wouldn't work until the Service showed up, which is exactly the race this ordering avoids. + ```bash kubectl apply -f manifests/config-and-data/app-secret.yaml kubectl apply -f manifests/config-and-data/postgres-statefulset.yaml From d9cdaa0a96b1f7a0628e106e7d130658b0264e10 Mon Sep 17 00:00:00 2001 From: Matt2_Chang Date: Thu, 2 Jul 2026 11:53:03 +0800 Subject: [PATCH 13/13] docs: add k9s watch points to the StatefulSet lab AS IS: the lab only mentioned k9s once, for the delete/reattach test, leaving readers to poll kubectl get/wait for everything else even though the ordered Pod creation and PVC binding are exactly the kind of live reconciliation k9s is meant to make visible TO BE: add k9s tips at first pod startup and at the scale-to-2 exercise, pointing at :sts/:pods/:pvc so readers can watch ordering and per-replica PVC binding happen instead of just reading kubectl output TASK # N/A --- docs/config-and-data/statefulset.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/config-and-data/statefulset.md b/docs/config-and-data/statefulset.md index e0e4a2c..f2fbad8 100644 --- a/docs/config-and-data/statefulset.md +++ b/docs/config-and-data/statefulset.md @@ -127,6 +127,8 @@ pod/postgres-0 1/1 Running ← ordinal name, not a random suffix persistentvolumeclaim/pgdata-postgres-0 Bound ← its own dedicated volume ``` +In [k9s](../getting-started/k9s.md), `:sts` ⏎ shows the StatefulSet's `READY` column climb to `1/1`; `:pods` ⏎ lets you watch `postgres-0` move through `Pending` → `ContainerCreating` → `Running` (`0/1` while the `startupProbe`/`readinessProbe` are still warming up, then `1/1`). Press `l` on it to tail PostgreSQL's own boot log instead of polling `pg_isready` from your terminal. + With the headless Service, that same Pod also gets a stable DNS name: ```text @@ -173,6 +175,8 @@ kubectl scale statefulset/postgres --replicas=1 kubectl get pvc # pgdata-postgres-1 is still here ``` +This is worth watching live instead of just polling `kubectl get`. In k9s, split-screen it: `:sts` ⏎ to watch `READY` go `1/1` → `2/2`, and `:pvc` ⏎ in another view to watch `pgdata-postgres-1` appear as `Pending` then flip to `Bound` right as `postgres-1` starts. Scale back down and `:pvc` still lists `pgdata-postgres-1` — the PVC outlives the Pod that used it, same as pressing `Ctrl-D` to delete a Pod would. + That second Pod is another independent PostgreSQL instance with its own volume. This lab does not configure PostgreSQL replication. Scaling down only removes the Pod — `pgdata-postgres-1` is **not** deleted. If you scale back to 2, `postgres-1` reattaches to that same PVC instead of getting a fresh volume. This is the same "PVCs outlive the Pod" behavior as the delete/reattach test above, just triggered by scaling instead of a manual delete.