From b4819d938a7141f8a489cc4ffaafa435450547a8 Mon Sep 17 00:00:00 2001 From: Nikolai Emil Damm Date: Sat, 18 Jul 2026 09:07:12 +0200 Subject: [PATCH 1/3] docs: rewrite the README opening and jargon walls for a first-time reader State who the platform is for and that it is a working system rather than a product. Break the OpenBao prerequisite and the local-cluster paragraph into plain sentences and a scannable file list. Keep both diagrams in the templating section but defer file-level mechanics to docs/TEMPLATING.md, and drop the maintainer aside from the IMPORTANT callout. Component inventory, docs index, and cost table are unchanged. Fixes #2686 --- README.md | 69 ++++++++++++++++++++++++++++++++++++------------------- 1 file changed, 45 insertions(+), 24 deletions(-) diff --git a/README.md b/README.md index 577dc016b..1a2f6a57e 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,15 @@ image -This repo contains the deployment artifacts for the DevantlerTech Platform. The platform is a Kubernetes cluster that is highly automated with the use of Flux GitOps, CI/CD with Automated Testing, and much more. Feel free to look around. You might find some inspiration 🙌🏻 +My personal Kubernetes platform, in the open. Everything the cluster runs is described as files in +this repository, and changes go live by being merged here rather than by anyone running commands +against the cluster — that pattern is called *GitOps*, and [Flux](https://fluxcd.io) is what applies +it. + +This is a working system rather than a product: it is shaped around what I run, and it is not +packaged for reuse. Look around anyway — if you are building something similar, the repository +layout and the guides in [`docs/`](docs) are the useful parts, and +[`docs/TEMPLATING.md`](docs/TEMPLATING.md) lists exactly what a fork has to change. 🙌🏻 ## Prerequisites @@ -16,8 +24,13 @@ For the production cluster: - [Hetzner Cloud](https://www.hetzner.com/cloud/) — Infrastructure provider and managed Cloud Load Balancer for cluster ingress. KSail's native Hetzner provider handles Talos boot, CCM, CSI, and kubeconfig. - [Cloudflare](https://www.cloudflare.com) — DNS (A/AAAA records pointed at the Hetzner Cloud Load Balancer) and Origin CA. - [Flux GitOps](https://fluxcd.io) - For managing the kubernetes applications and infrastructure declaratively. -- [SOPS](https://getsops.io) and [Age](https://github.com/FiloSottile/age) - For encrypting the seed secrets that are committed to this repository (the `*.enc.yaml` files), allowing me to store them in git with confidence. -- [OpenBao](https://openbao.org) and the [External Secrets Operator](https://external-secrets.io) - The runtime secret store for most workloads. SOPS-decrypted seeds are pushed into OpenBao at bootstrap, and `ExternalSecret`s sync them into the namespaces that consume them. A few secrets (Dex and oauth2-proxy) are still read directly from the SOPS-encrypted bootstrap secrets via Flux `postBuild` substitution while the migration to OpenBao completes. See [`docs/secret-rotation.md`](docs/secret-rotation.md) for the full secrets architecture. +- [SOPS](https://getsops.io) and [Age](https://github.com/FiloSottile/age) — encrypt the starting + secrets that are committed here (the `*.enc.yaml` files), so they can live in Git safely. +- [OpenBao](https://openbao.org) and the [External Secrets Operator](https://external-secrets.io) — + where secrets live once the cluster is running. At startup the encrypted files above are loaded + into OpenBao, and the operator copies each secret into the namespace that needs it. Two of them + (Dex and oauth2-proxy) are still read straight from the encrypted files while that move finishes. + Full picture: [`docs/secret-rotation.md`](docs/secret-rotation.md). ## Usage @@ -38,7 +51,21 @@ ksail workload push ksail workload reconcile ``` -Ports 80 and 443 are automatically mapped to localhost via `extraPortMappings` in `ksail.yaml`. The local cluster is a **thin manual test-bed** — a small Talos cluster for trying a component out before promoting it to prod. By default it runs only the **core infrastructure** (CNI + Gateway, DNS, TLS, Flux, Kyverno + policies, VPA, OpenBao + External Secrets, the Dex SSO stack, and CloudNativePG). Heavier infrastructure (observability, autoscaling, backup, runtime security, the VM stack, …) and all apps are opt-in — uncomment the entries you want in the docker provider overlays (`k8s/providers/docker/infrastructure/controllers/kustomization.yaml`, `…/infrastructure/kustomization.yaml`) and the apps overlay (`k8s/providers/docker/apps/kustomization.yaml`), each of which carries a copy-paste template, then re-run `ksail workload push && ksail workload reconcile`. Reach the deployed services at their `*.platform.lan` hostnames (the `hosts` file maps these to `127.0.0.1`). +Ports 80 and 443 are mapped to localhost for you (via `extraPortMappings` in [`ksail.yaml`](ksail.yaml)), +and the [`hosts`](hosts) file points the `*.platform.lan` names at `127.0.0.1` — so deployed services +open in a browser. + +The local cluster is a **thin test-bed**: somewhere to try one component before promoting it to +production, not a copy of production. It starts with core infrastructure only — networking and +gateway, DNS, TLS, Flux, policy, autoscaling, secrets, single sign-on, and PostgreSQL. + +Everything heavier (observability, backup, runtime security, the VM stack, …) and all apps are +opt-in. Uncomment what you want in these files — each carries a copy-paste template — then re-run +`ksail workload push && ksail workload reconcile`: + +- `k8s/providers/docker/infrastructure/controllers/kustomization.yaml` +- `k8s/providers/docker/infrastructure/kustomization.yaml` +- `k8s/providers/docker/apps/kustomization.yaml` To tear down: @@ -122,16 +149,10 @@ The cluster configuration is stored in the `k8s/*` directories where the structu - [`apps`](k8s/bases/apps): Contains the different apps that are used for the different clusters and providers. - [`bootstrap`](k8s/bases/bootstrap): The foundational **bootstrap layer** (renamed from `variables/`). Holds the shared substitution variables (`variables-base` ConfigMap + SOPS-encrypted Secret) and cluster-scoped PriorityClasses (e.g. `platform-critical`), reconciled by the `bootstrap` Flux Kustomization before everything that `dependsOn` it. -### Kustomize and Flux Kustomization Flow - -> [!IMPORTANT] -> If you know of a different way to manage kustomize and flux kustomizations that results in less boilerplate code, please let me know. I am always looking for ways to improve the structure and make it more maintainable. +### How the layers fit together -To support hooking into the kustomize flow for adding or modifying resources for a specific cluster, a specific provider, or shared across all clusters, the following structure is used: - -#### Kustomize Overlay Flow - -Each cluster environment references a provider overlay, which in turn patches the shared base resources: +Two things stack up. First, each environment points at a provider, which patches the shared +resources — so a change can be made for one cluster, for one provider, or for everything at once: ```mermaid graph LR @@ -155,9 +176,7 @@ graph LR hetzner --> bases ``` -#### Flux Kustomization Dependency Chain - -Flux Kustomizations are reconciled sequentially. Each layer waits for the previous to become ready: +Second, Flux applies those layers in order, each waiting for the one before it to come up: ```mermaid graph TB @@ -171,16 +190,18 @@ graph TB apps -- "depends on" --> infra ``` -This means that for every Flux Kustomization applied to a cluster, there should be a corresponding resource folder in `providers//` or `bases/` that contains the manifests for that scope. For example, the `infrastructure` Flux Kustomization is backed by: - -- `k8s/providers//infrastructure/` -- `k8s/bases/infrastructure/` - -The Flux Kustomizations themselves live in `k8s/clusters/base/` (with sentinel `__CLUSTER__` / `__PROVIDER__` values in `spec.path`). Each `k8s/clusters//` overlay patches the `cluster-meta` ConfigMap with its `cluster_name` / `provider` and uses kustomize `replacements:` to rewrite those sentinels with the cluster's real values. Only the per-cluster `bootstrap/` directory holds cluster-specific manifests. +Each layer in that chain has a matching folder under `providers//` and `bases/`. The +`infrastructure` layer, for example, is backed by `k8s/providers//infrastructure/` and +`k8s/bases/infrastructure/`. -See [`docs/TEMPLATING.md`](docs/TEMPLATING.md) for the exact set of files a fork of this repo needs to edit to stand up its own instance. +The layer definitions themselves are written once in `k8s/clusters/base/` with placeholders where the +cluster and provider names go; each cluster's overlay fills those in. Only the per-cluster +`bootstrap/` directory holds manifests unique to one cluster. -See [`docs/TENANTS.md`](docs/TENANTS.md) for how to onboard a new GitOps **tenant** (an app that runs on the platform from its own repository). +- [`docs/TEMPLATING.md`](docs/TEMPLATING.md) — the exact files a fork must edit, including how those + placeholders get filled in. +- [`docs/TENANTS.md`](docs/TENANTS.md) — adding a **tenant**: an app that runs on the platform from + its own repository. ## Documentation From a85cad57f4cee7651dd0720f48fae5b1b422a2fc Mon Sep 17 00:00:00 2001 From: Nikolai Emil Damm Date: Sat, 18 Jul 2026 09:09:01 +0200 Subject: [PATCH 2/3] docs: drop the duplicated secrets explanation from the Usage callout Reading the rendered page showed the Usage callout restating the same SOPS/OpenBao/postBuild detail the Prerequisites section now explains in plain words. The callout is only about what a fork must swap out, so it says that and nothing else. Also finish the prerequisite bullets left in the old terse style. --- README.md | 21 ++++++++++++--------- 1 file changed, 12 insertions(+), 9 deletions(-) diff --git a/README.md b/README.md index 1a2f6a57e..285de4b08 100644 --- a/README.md +++ b/README.md @@ -16,14 +16,16 @@ layout and the guides in [`docs/`](docs) are the useful parts, and For local development: -- [Docker](https://docs.docker.com/get-docker/) - For running the cluster locally. -- [KSail](https://github.com/devantler-tech/ksail) - For developing the cluster locally, and for running the cluster in CI to ensure all changes are properly tested before being applied to the production cluster. +- [Docker](https://docs.docker.com/get-docker/) — runs the cluster on your machine. +- [KSail](https://github.com/devantler-tech/ksail) — creates and manages that cluster. CI uses the + same tool, so changes are exercised the same way before they reach production. For the production cluster: - [Hetzner Cloud](https://www.hetzner.com/cloud/) — Infrastructure provider and managed Cloud Load Balancer for cluster ingress. KSail's native Hetzner provider handles Talos boot, CCM, CSI, and kubeconfig. - [Cloudflare](https://www.cloudflare.com) — DNS (A/AAAA records pointed at the Hetzner Cloud Load Balancer) and Origin CA. -- [Flux GitOps](https://fluxcd.io) - For managing the kubernetes applications and infrastructure declaratively. +- [Flux GitOps](https://fluxcd.io) — watches this repository and applies whatever it finds here to + the cluster. - [SOPS](https://getsops.io) and [Age](https://github.com/FiloSottile/age) — encrypt the starting secrets that are committed here (the `*.enc.yaml` files), so they can live in Git safely. - [OpenBao](https://openbao.org) and the [External Secrets Operator](https://external-secrets.io) — @@ -35,13 +37,14 @@ For the production cluster: ## Usage > [!IMPORTANT] -> Secrets committed to this repo are encrypted at rest with SOPS + Age. At bootstrap they seed OpenBao, and the External Secrets Operator distributes most of them to workloads at runtime; a few (Dex, oauth2-proxy) are still injected directly via Flux `postBuild` substitution while the OpenBao migration completes. If you want to run the platform locally, or in your own Hetzner project, you will need to: +> The committed secrets are encrypted with my keys, so you cannot decrypt them. To run this locally +> or in your own Hetzner project, swap in your own: > -> 1. Fork this repo -> 2. Create your own Age keys -> 3. Update the `.sops.yaml` file in the root of the repository. -> 4. Update GitHub secrets with your Age key. -> 5. Replace all encrypted `*.enc.yaml` files in the `k8s/` folder with new ones that are encrypted with your own keys. +> 1. Fork this repo. +> 2. Create your own Age keys. +> 3. Update `.sops.yaml` in the repository root. +> 4. Put your Age key in your fork's GitHub secrets. +> 5. Re-encrypt every `*.enc.yaml` file under `k8s/` with your own keys. To run this cluster locally, simply run: From d642d9737144788e51bfa8aefc251d35367edb63 Mon Sep 17 00:00:00 2001 From: Nikolai Emil Damm Date: Sat, 18 Jul 2026 09:48:58 +0200 Subject: [PATCH 3/3] docs: correct five accuracy regressions from the README simplification Codex review found that plainer wording had made five claims wrong. Verified each against the manifests before fixing: - Flux does not watch this repo; the Kustomizations reference an OCIRepository, so say the manifests are published and pulled. - OpenBao is not the runtime path for everything committed here. Bootstrap-critical values plus Dex/oauth2-proxy are still substituted into manifests by Flux postBuild; restore the "most secrets" split. - CI does not boot a cluster. It runs workload validate and scan only, so "exercised the same way" overstated its coverage. - The local overlay enables the CloudNativePG operator, not PostgreSQL. The apps overlay is resources: [], so no database exists locally. - KEDA request-rate autoscaling is opt-in, not core. Only VPA is core. The lesson is that simplifying prose can quietly drop qualifiers that were load-bearing; each of these was a qualifier doing real work. --- README.md | 27 ++++++++++++++++----------- 1 file changed, 16 insertions(+), 11 deletions(-) diff --git a/README.md b/README.md index 285de4b08..a9cb90696 100644 --- a/README.md +++ b/README.md @@ -18,21 +18,24 @@ For local development: - [Docker](https://docs.docker.com/get-docker/) — runs the cluster on your machine. - [KSail](https://github.com/devantler-tech/ksail) — creates and manages that cluster. CI uses the - same tool, so changes are exercised the same way before they reach production. + same tool to validate and security-scan manifests, but does not boot a cluster; running one is a + local step. For the production cluster: - [Hetzner Cloud](https://www.hetzner.com/cloud/) — Infrastructure provider and managed Cloud Load Balancer for cluster ingress. KSail's native Hetzner provider handles Talos boot, CCM, CSI, and kubeconfig. - [Cloudflare](https://www.cloudflare.com) — DNS (A/AAAA records pointed at the Hetzner Cloud Load Balancer) and Origin CA. -- [Flux GitOps](https://fluxcd.io) — watches this repository and applies whatever it finds here to - the cluster. +- [Flux GitOps](https://fluxcd.io) — applies the manifests to the cluster. It does not read this + repository directly: the manifests are packaged and published as an OCI artifact, and Flux pulls + that. - [SOPS](https://getsops.io) and [Age](https://github.com/FiloSottile/age) — encrypt the starting secrets that are committed here (the `*.enc.yaml` files), so they can live in Git safely. - [OpenBao](https://openbao.org) and the [External Secrets Operator](https://external-secrets.io) — - where secrets live once the cluster is running. At startup the encrypted files above are loaded - into OpenBao, and the operator copies each secret into the namespace that needs it. Two of them - (Dex and oauth2-proxy) are still read straight from the encrypted files while that move finishes. - Full picture: [`docs/secret-rotation.md`](docs/secret-rotation.md). + where **most** secrets live once the cluster is running. Selected values from the encrypted files + above are seeded into OpenBao, and the operator copies each one into the namespace that needs it. + The rest — the bootstrap-critical values, plus Dex and oauth2-proxy — are still substituted + straight into manifests by Flux at apply time, so OpenBao is not the runtime path for everything + committed here. Full picture: [`docs/secret-rotation.md`](docs/secret-rotation.md). ## Usage @@ -60,11 +63,13 @@ open in a browser. The local cluster is a **thin test-bed**: somewhere to try one component before promoting it to production, not a copy of production. It starts with core infrastructure only — networking and -gateway, DNS, TLS, Flux, policy, autoscaling, secrets, single sign-on, and PostgreSQL. +gateway, DNS, TLS, Flux, policy, vertical pod autoscaling, secrets, single sign-on, and the +CloudNativePG operator. Note that the operator is installed but no database is created: apps are +opt-in, so a plain local bring-up has no PostgreSQL instance running. -Everything heavier (observability, backup, runtime security, the VM stack, …) and all apps are -opt-in. Uncomment what you want in these files — each carries a copy-paste template — then re-run -`ksail workload push && ksail workload reconcile`: +Everything heavier (observability, request-rate autoscaling, backup, runtime security, the VM stack, +…) and all apps are opt-in. Uncomment what you want in these files — each carries a copy-paste +template — then re-run `ksail workload push && ksail workload reconcile`: - `k8s/providers/docker/infrastructure/controllers/kustomization.yaml` - `k8s/providers/docker/infrastructure/kustomization.yaml`