From acd2d052bdac6bbad87a96bb6db18974835876e7 Mon Sep 17 00:00:00 2001 From: ixxeL2097 Date: Sun, 26 Apr 2026 09:30:07 +0200 Subject: [PATCH 1/4] docs: add Mermaid architecture diagrams across all documentation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Enable Mermaid support in mkdocs.yml via pymdownx.superfences custom_fences - index.md: global homelab architecture overview (two clusters, GitHub, key services) - argocd.md: GitOps reconciliation loop, App-of-Apps pattern, Helm value hierarchy; update directory structure to reflect current common/beelink/genmachine layout - talos.md: Talos cluster topology (3 control-plane VMs on Proxmox, etcd, Cilium) - certmanager.md: PKI chain from Vault Root CA to TLS Secret via cert-manager - externalsecrets.md: ESO → Vault Kubernetes auth flow; fix incomplete intro sentence - oidc.md: OIDC authorization code flow and group claim injection via scope mappings - proxy-auth.md: Traefik forward-auth flow with Authentik outpost and header passthrough - vault.md (new): Vault cross-cluster transit auto-unseal, seal migration, deadlock recovery, Kubernetes auth flow; added to mkdocs.yml nav under Secrets Co-Authored-By: Claude Sonnet 4.6 --- docs/argocd/argocd.md | 159 ++++++++++++++++---------- docs/authentication/oidc.md | 34 ++++++ docs/authentication/proxy-auth.md | 28 +++++ docs/certificates/certmanager.md | 37 +++++- docs/cluster/talos.md | 38 ++++++- docs/index.md | 71 +++++++++++- docs/secrets/externalsecrets.md | 33 +++++- docs/secrets/vault.md | 181 ++++++++++++++++++++++++++++++ mkdocs.yml | 7 +- 9 files changed, 516 insertions(+), 72 deletions(-) create mode 100644 docs/secrets/vault.md diff --git a/docs/argocd/argocd.md b/docs/argocd/argocd.md index f7a70019c..f34638350 100644 --- a/docs/argocd/argocd.md +++ b/docs/argocd/argocd.md @@ -1,4 +1,4 @@ -# GitOps-core +# GitOps with ArgoCD > [!CAUTION] > This structure is opinionated and results from multiple experiences using ArgoCD in enterprise-grade environments. @@ -9,78 +9,118 @@ This Git repository serves as the central ArgoCD repository, containing the defi The installation of ArgoCD follows the [App of Apps pattern](https://argo-cd.readthedocs.io/en/stable/operator-manual/cluster-bootstrapping/#app-of-apps-pattern), a recommended best practice for managing GitOps deployments at scale. -## Repository Structure +## GitOps Reconciliation Loop + +```mermaid +flowchart LR + dev["Developer\npush / PR"] + renovate["Renovate Bot\nautomated PRs"] + repo["GitHub\nrepository"] + ci["GitHub Actions\nhelm-rmp CI\n(diff preview)"] + argocd_b["ArgoCD\nbeelink"] + argocd_g["ArgoCD\ngenmachine"] + cluster_b["k0s cluster\nbeelink"] + cluster_g["Talos cluster\ngenmachine"] + + dev -->|git push| repo + renovate -->|auto-PR| repo + repo -->|webhook / poll| ci + repo -->|poll main| argocd_b + repo -->|poll main| argocd_g + argocd_b -->|apply| cluster_b + argocd_g -->|apply| cluster_g +``` + +## App of Apps Pattern + +```mermaid +graph TD + bootstrap["bootstrap/\nArgoCD install\n(Helm + kustomize)"] + + bootstrap -->|creates| root_b["Root App\nbeelink"] + bootstrap -->|creates| root_g["Root App\ngenmachine"] + + root_b -->|generates| appsets_b["ApplicationSets\nbeelink"] + root_g -->|generates| appsets_g["ApplicationSets\ngenmachine"] + + appsets_b -->|creates| app1["cert-manager\nbeelink"] + appsets_b -->|creates| app2["traefik\nbeelink"] + appsets_b -->|creates| app3["vault\nbeelink"] + appsets_b -->|creates| appN["..."] + + appsets_g -->|creates| app4["cilium\ngenmachine"] + appsets_g -->|creates| app5["prometheus\ngenmachine"] + appsets_g -->|creates| app6["vault\ngenmachine"] + appsets_g -->|creates| appM["..."] +``` -Below is the directory structure of the `gitops` repository: +## Repository Structure ```bash gitops ├── bootstrap -│ └── kustomization.yaml +│ ├── beelink +│ │ └── beelink-values.yaml # ArgoCD Helm values for beelink +│ └── genmachine +│ └── genmachine-values.yaml ├── core -│ ├── appProjects +│ ├── appProjects # RBAC project definitions │ ├── apps -│ └── repos -├── local-storage -│ ├── adguard-data -│ ├── headscale-data -│ └── vault-data +│ │ ├── beelink # ApplicationSets for k0s cluster +│ │ └── genmachine # ApplicationSets for Talos cluster +│ ├── clusters # Cluster secret references +│ └── repos # Repository credentials └── manifests - ├── adguard - ├── authentik - ├── crowdsec - ├── external-secrets - ├── headscale - ├── homarr - ├── local-path-provisioner - ├── metallb + ├── cert-manager + ├── cilium ├── traefik ├── vault - └── wireguard + └── ... # 30+ applications ``` -### Directory Breakdown - -- **`bootstrap/`**: Contains the ArgoCD installation manifests, which can be managed via `kustomization.yaml` or Helm charts. -- **`core/`**: Includes core ArgoCD resources such as `Application`, `ApplicationSet`, and `AppProject` definitions. -- **`local-storage/`** (optional): Used for applications requiring persistent storage, mapped via a local-path provisioner. -- **`manifests/`**: Stores Kubernetes manifests and Helm configurations for different cluster services and applications. - -## Multi-Environment Setup +## Multi-Environment Helm Structure -For a structured multi-environment approach, the `manifests` directory is organized as follows: +Each application directory follows a consistent layout that separates shared configuration from cluster-specific overrides: ```bash gitops/manifests/ -├── metallb -│ ├── k0s -│ │ ├── Chart.yaml -│ │ ├── k0s-values.yaml -│ │ └── templates -│ ├── talos +├── cert-manager +│ ├── common # shared across all clusters +│ │ └── common-values.yaml +│ ├── beelink # k0s cluster overrides │ │ ├── Chart.yaml -│ │ ├── talos-values.yaml +│ │ ├── beelink-values.yaml │ │ └── templates -│ └── values -│ └── common-values.yaml +│ └── genmachine # Talos cluster overrides +│ ├── Chart.yaml +│ ├── genmachine-values.yaml +│ └── templates └── traefik - ├── k0s - │ ├── Chart.yaml - │ ├── k0s-values.yaml - │ └── templates - ├── talos - │ ├── Chart.yaml - │ ├── talos-values.yaml - │ └── templates - └── values - └── common-values.yaml + ├── common + │ └── common-values.yaml + ├── beelink + │ └── ... + └── genmachine + └── ... ``` -Each application directory contains subdirectories for different environments (`k0s`, `talos`), allowing environment-specific Helm values while maintaining shared configurations in `common-values.yaml`. +```mermaid +graph LR + subgraph values["Helm value hierarchy (lowest → highest priority)"] + common["common/\ncommon-values.yaml\nshared defaults"] + env["beelink/ or genmachine/\nenv-values.yaml\ncluster overrides"] + end + + common -->|merged into| render["helm template\nfinal manifest"] + env -->|merged into| render + + appset["ApplicationSet\ngit directory generator"] -->|discovers| env + appset -->|excludes| common +``` ## ApplicationSet Usage -To efficiently deploy applications while supporting multiple environments, `ApplicationSet` is utilized: +`ApplicationSet` discovers cluster-specific directories automatically, then overlays the common values file on top. ```yaml --- @@ -90,7 +130,7 @@ metadata: name: cert-manager namespace: argocd annotations: - argocd.argoproj.io/manifest-generate-paths: .;../values + argocd.argoproj.io/manifest-generate-paths: .;../common spec: goTemplate: true generators: @@ -100,13 +140,13 @@ spec: directories: - path: "gitops/manifests/cert-manager/*" exclude: false - - path: "gitops/manifests/cert-manager/values/*" + - path: "gitops/manifests/cert-manager/common" exclude: true template: metadata: name: "cert-manager-{{ .path.basenameNormalized }}" annotations: - argocd.argoproj.io/manifest-generate-paths: .;../values + argocd.argoproj.io/manifest-generate-paths: .;../common spec: project: infra-security destination: @@ -118,8 +158,9 @@ spec: targetRevision: main helm: valueFiles: - - $values/gitops/manifests/cert-manager/values/common-values.yaml + - $values/gitops/manifests/cert-manager/common/common-values.yaml - $values/gitops/manifests/cert-manager/{{ .path.basenameNormalized }}/{{ .path.basenameNormalized }}-values.yaml + ignoreMissingValueFiles: true - repoURL: https://github.com/ixxeL-DevOps/fullstack.git targetRevision: main ref: values @@ -137,13 +178,11 @@ spec: - ServerSideApply=true ``` -The ApplicationSet is annotated following [ArgoCD optimization recommendations](https://argo-cd.readthedocs.io/en/stable/operator-manual/high_availability/#manifest-paths-annotation). +The `manifest-generate-paths` annotation (`.;../common`) ensures that ArgoCD refreshes the application when either the cluster-specific directory **or** the shared `common/` directory changes, following [ArgoCD optimization recommendations](https://argo-cd.readthedocs.io/en/stable/operator-manual/high_availability/#manifest-paths-annotation). ### Key Features -- **Multi-environment support**: Uses directory-based environment segregation. -- **Hierarchical Helm values**: Supports multiple value files (`common-values.yaml` and environment-specific values). -- **Automated synchronization**: Ensures ArgoCD keeps applications up-to-date and reconciled with Git. -- **Flexible exclusions**: Allows selective inclusion of manifests while ignoring specific files if necessary. - -By leveraging `ApplicationSet`, managing deployments across multiple clusters and environments becomes more scalable and maintainable. +- **Multi-cluster support**: The git directory generator discovers `beelink/` and `genmachine/` subdirectories and deploys to the matching cluster destination. +- **Hierarchical Helm values**: `common-values.yaml` provides shared defaults; cluster-specific files override them. +- **Automated synchronization**: Prune + self-heal keeps clusters reconciled with Git at all times. +- **Selective exclusions**: The `common/` directory is excluded from the generator so it is never deployed as a standalone application. diff --git a/docs/authentication/oidc.md b/docs/authentication/oidc.md index 7743a1b3b..dbfec90cf 100644 --- a/docs/authentication/oidc.md +++ b/docs/authentication/oidc.md @@ -1,5 +1,39 @@ # OIDC +## Overview + +[Authentik](https://goauthentik.io/) acts as the central Identity Provider (IdP) for the homelab. It provides OIDC/OAuth2 SSO for all applications that support it. + +```mermaid +sequenceDiagram + actor user as User + participant app as Application
(Vault / Homarr / WireGuard) + participant authentik as Authentik IdP + + user->>app: Access resource + app-->>user: Redirect to Authentik login + user->>authentik: Submit credentials + MFA + authentik-->>user: Authorization code (redirect back) + user->>app: Code callback + app->>authentik: Exchange code for tokens + authentik-->>app: ID token + Access token
(claims: email, groups, is_admin…) + app-->>user: Session granted +``` + +Group-based authorization is enforced via claims injected by Authentik scope mappings: + +```mermaid +graph LR + authentik["Authentik\nGroup membership"] + scope["Scope Mapping\nPython expression"] + claim["JWT claim\ngroups / is_admin"] + app["Application\nauthorization check"] + + authentik -->|evaluated by| scope + scope -->|produces| claim + claim -->|consumed by| app +``` + ## Vault Blueprint for Vault OIDC auth : diff --git a/docs/authentication/proxy-auth.md b/docs/authentication/proxy-auth.md index 7db9b2b56..da2e344db 100644 --- a/docs/authentication/proxy-auth.md +++ b/docs/authentication/proxy-auth.md @@ -1,5 +1,33 @@ # Forward Auth +## Overview + +Traefik uses the Authentik embedded outpost as a forward authentication provider. Every request to a protected application is checked against Authentik before being proxied upstream. + +```mermaid +sequenceDiagram + actor user as User + participant traefik as Traefik + participant outpost as Authentik Outpost
(ForwardAuth) + participant app as Protected App + + user->>traefik: HTTPS request + traefik->>outpost: GET /outpost.goauthentik.io/auth/traefik + alt No valid session + outpost-->>traefik: 302 → Authentik login + traefik-->>user: Redirect to login page + user->>outpost: Authenticate (credentials + MFA) + outpost-->>user: Session cookie set + user->>traefik: Retry original request + traefik->>outpost: Auth check (session cookie present) + end + outpost-->>traefik: 200 + X-authentik-* headers + traefik->>app: Proxied request + headers
(X-authentik-username, X-authentik-groups…) + app-->>user: Response +``` + +The `X-authentik-*` response headers expose identity information to the upstream application without it needing to implement OIDC itself. + ## Reverse-Proxy setup : Traefik ### In-cluster setup diff --git a/docs/certificates/certmanager.md b/docs/certificates/certmanager.md index 212a56443..d32a87fa9 100644 --- a/docs/certificates/certmanager.md +++ b/docs/certificates/certmanager.md @@ -1,8 +1,43 @@ # Cert-Manager +## Overview + +`cert-manager` automates the full lifecycle of TLS certificates in the cluster. It integrates with HashiCorp Vault as the PKI backend, using Kubernetes ServiceAccount authentication. + +```mermaid +graph TB + subgraph vault["HashiCorp Vault"] + root_pki["Root PKI\npki/"] + int_pki["Intermediate PKI\npki_int/"] + k8s_auth["Kubernetes Auth\n(cluster-k8s/)"] + end + + subgraph cluster["Kubernetes Cluster"] + cm["cert-manager"] + issuer["ClusterIssuer\nfredcorp-ca"] + sa["ServiceAccount\ncertmanager-auth"] + cert["Certificate\nresource"] + secret["TLS Secret\n(cert + key)"] + bundle["trust-manager\nBundle"] + cm_map["CA chain\nConfigMap / Secret\n(all namespaces)"] + end + + root_pki -->|signs| int_pki + int_pki -->|signs CSRs| issuer + k8s_auth -->|validates SA token| sa + sa -->|authenticates| k8s_auth + issuer -->|uses| cm + cert -->|requests via| cm + cm -->|CSR sign request\npki_int/sign/fredcorp.com| int_pki + cm -->|creates| secret + bundle -->|distributes CA chain| cm_map + + style vault fill:#1a2a3a,stroke:#FFB81C,color:#FFB81C +``` + ## Installation -Cert-manager can be used to handle certificates lifecycle in your cluster. `cert-manager` and `trust-manager` should be installed to get a complete lifycle management. +Cert-manager can be used to handle certificates lifecycle in your cluster. `cert-manager` and `trust-manager` should be installed to get a complete lifecycle management. ## Configuration diff --git a/docs/cluster/talos.md b/docs/cluster/talos.md index 1d487f6b7..c118e500f 100644 --- a/docs/cluster/talos.md +++ b/docs/cluster/talos.md @@ -10,7 +10,43 @@ The cluster is a **3 nodes Kubernetes cluster** running on a home server. It use - **Data Storage**: `etcd` - **CNI (Networking)**: `Cilium` -- **Environment Management**: Uses **Devbox** to handle binaries like `k0sctl`, `kubectl`, `task`, and other necessary CLI tools. +- **Environment Management**: Uses **Devbox** to handle binaries like `talosctl`, `kubectl`, `task`, and other necessary CLI tools. + +### Topology + +```mermaid +graph TB + subgraph proxmox["Proxmox Hypervisor — genmachine"] + subgraph talos["Talos Kubernetes Cluster"] + cp1["talos-1\n192.168.1.151\ncontrol-plane"] + cp2["talos-2\n192.168.1.152\ncontrol-plane"] + cp3["talos-3\n192.168.1.153\ncontrol-plane"] + + etcd["etcd\n(distributed, 3 members)"] + cilium["Cilium CNI\neBPF dataplane\nL2 announcements"] + argocd["ArgoCD\nGitOps controller"] + end + + storage["Proxmox storage\n(LVM volumes via CSI)"] + end + + subgraph network["Home Network"] + metallb["MetalLB / Cilium L2\nLoadBalancer IPs"] + traefik["Traefik\nIngress controller"] + end + + cp1 <-->|raft consensus| etcd + cp2 <-->|raft consensus| etcd + cp3 <-->|raft consensus| etcd + + cp1 & cp2 & cp3 --- cilium + cilium --- metallb + metallb --- traefik + cp1 & cp2 & cp3 --- storage + + style proxmox fill:#1a1a2a,stroke:#6666ff,color:#fff + style talos fill:#1a2a1a,stroke:#66ff66,color:#fff +``` ## 🛠️ Installation diff --git a/docs/index.md b/docs/index.md index 0fa32f604..32e02d2b6 100644 --- a/docs/index.md +++ b/docs/index.md @@ -2,15 +2,15 @@

-# 🚀 **My home-lab repository** +# **My home-lab repository** -**✨ Hosted with k0s & Talos** +**Hosted with k0s & Talos** -**✨ Managed by ArgoCD** +**Managed by ArgoCD** -**✨ Powered by Renovate and GitHub**' +**Powered by Renovate and GitHub** -**✨ Fueled by Cilium** +**Fueled by Cilium** --- @@ -21,7 +21,7 @@ **TOOLING** -![traefik](https://img.shields.io/badge/dynamic/yaml?url=https%3A%2F%2Fraw.githubusercontent.com%2FixxeL-DevOps%2Ffullstack%2Frefs%2Fheads%2Fmain%2Fgitops%2Fmanifests%2Ftraefik%2Fk0s%2Fk0s-values.yaml&query=%24.traefik.image.tag&style=for-the-badge&logo=traefikproxy&logoColor=%239D0FB0&label=traefik&color=%239D0FB0) +![traefik](https://img.shields.io/badge/dynamic/yaml?url=https%3A%2F%2Fraw.githubusercontent.com%2FixxeL-DevOps%2Ffullstack%2Frefs%2Fheads%2Fmain%2Fgitops%2Fmanifests%2Ftraefik%2Fbeelink%2Fbeelink-values.yaml&query=%24.traefik.image.tag&style=for-the-badge&logo=traefikproxy&logoColor=%239D0FB0&label=traefik&color=%239D0FB0) ![adguard](https://img.shields.io/badge/dynamic/yaml?url=https%3A%2F%2Fraw.githubusercontent.com%2FixxeL-DevOps%2Ffullstack%2Frefs%2Fheads%2Fmain%2Fgitops%2Fmanifests%2Fadguard%2Fbeelink%2Fbeelink-values.yaml&query=%24.adguard-home.image.tag&style=for-the-badge&logo=adguard&label=AdGuard&color=%2366B574) ![authentik](https://img.shields.io/badge/dynamic/yaml?url=https%3A%2F%2Fraw.githubusercontent.com%2FixxeL-DevOps%2Ffullstack%2Frefs%2Fheads%2Fmain%2Fgitops%2Fmanifests%2Fauthentik%2Fbeelink%2Fapp%2Fbeelink-values.yaml&query=%24.authentik.global.image.tag&style=for-the-badge&logo=authentik&label=Authentik&color=%23FD4B2D) ![vault](https://img.shields.io/badge/dynamic/yaml?url=https%3A%2F%2Fraw.githubusercontent.com%2FixxeL-DevOps%2Ffullstack%2Frefs%2Fheads%2Fmain%2Fgitops%2Fmanifests%2Fvault%2Fbeelink%2Fbeelink-values.yaml&query=%24.vault.server.image.tag&style=for-the-badge&logo=vault&label=Vault&color=%23FFB81C) @@ -56,3 +56,62 @@ --- This project utilizes Infrastructure as Code and GitOps to automate provisioning, operating, and updating self-hosted services in my homelab. + +## Architecture Overview + +```mermaid +graph TB + subgraph internet["Internet"] + user["User"] + end + + subgraph home["Home Network (192.168.1.x)"] + router["Router"] + adguard["AdGuard Home\nDNS resolver"] + end + + subgraph github["GitHub"] + repo["fullstack\nrepository"] + renovate["Renovate Bot\nautomated PRs"] + actions["GitHub Actions\nCI / helm-rmp"] + end + + subgraph beelink["Beelink — k0s cluster"] + traefik_b["Traefik\nIngress"] + argocd_b["ArgoCD"] + authentik["Authentik\nIdP / SSO"] + vault_b["Vault\nbeelink"] + wireguard["WireGuard\nVPN"] + end + + subgraph genmachine["Genmachine — Talos cluster"] + subgraph proxmox["Proxmox hypervisor"] + vm1["talos-1\n192.168.1.151"] + vm2["talos-2\n192.168.1.152"] + vm3["talos-3\n192.168.1.153"] + end + traefik_g["Traefik\nIngress"] + argocd_g["ArgoCD"] + vault_g["Vault\ngenmachine"] + prometheus["Prometheus\n+ Grafana"] + minio["MinIO\nobject storage"] + end + + user -->|HTTPS| router + router -->|ingress| traefik_b + router -->|ingress| traefik_g + router -->|DNS| adguard + + argocd_b -->|pull| repo + argocd_g -->|pull| repo + renovate -->|auto-PR| repo + actions -->|helm diff| repo + + vault_b <-->|transit auto-unseal| vault_g + + style beelink fill:#1a3a5c,stroke:#4d94ff,color:#fff + style genmachine fill:#1a3a1a,stroke:#4dff4d,color:#fff + style github fill:#2d1a3a,stroke:#9d4dff,color:#fff + style home fill:#3a2d1a,stroke:#ffb84d,color:#fff + style proxmox fill:#2a3a2a,stroke:#4dff4d,color:#fff +``` diff --git a/docs/secrets/externalsecrets.md b/docs/secrets/externalsecrets.md index a81339245..731dfe4e9 100644 --- a/docs/secrets/externalsecrets.md +++ b/docs/secrets/externalsecrets.md @@ -1,8 +1,35 @@ # ExternalSecrets -## Installation - -ExternalSecret make GitOps secured by using +## Overview + +ExternalSecrets Operator (ESO) bridges Kubernetes and HashiCorp Vault, allowing applications to consume secrets as native Kubernetes `Secret` objects while the source of truth stays in Vault. + +```mermaid +graph LR + subgraph vault["HashiCorp Vault"] + kv["KV Engine v2\nadmin/"] + k8s_auth["Kubernetes Auth\n(cluster-k8s/)"] + end + + subgraph cluster["Kubernetes Cluster"] + sa["ServiceAccount\neso-auth"] + css["ClusterSecretStore\nadmin"] + es["ExternalSecret"] + secret["K8s Secret"] + app["Application\n(env / volume)"] + end + + sa -->|JWT token| k8s_auth + k8s_auth -->|validates| sa + css -->|authenticates via| k8s_auth + es -->|references| css + css -->|reads from| kv + css -->|creates| secret + es -->|triggers sync| css + app -->|mounts| secret + + style vault fill:#1a2a3a,stroke:#FFB81C,color:#FFB81C +``` ## Configuration diff --git a/docs/secrets/vault.md b/docs/secrets/vault.md new file mode 100644 index 000000000..5e2ca7e57 --- /dev/null +++ b/docs/secrets/vault.md @@ -0,0 +1,181 @@ +# HashiCorp Vault + +## Overview + +Two Vault instances are deployed across the two clusters. Each instance serves as both a secrets engine and a PKI certificate authority for its cluster, and as an auto-unseal transit engine for the other cluster. + +```mermaid +graph TB + subgraph beelink["Beelink — k0s cluster"] + vault_b["Vault beelink\nvault.k0s-fullstack.fredcorp.com"] + pki_b["PKI Engine\nRoot + Intermediate CA\nfredcorp.com"] + transit_b["Transit Engine\nkey: unseal-genmachine"] + kv_b["KV Engine\nadmin/ secrets"] + end + + subgraph genmachine["Genmachine — Talos cluster"] + vault_g["Vault genmachine\nvault.talos-genmachine.fredcorp.com"] + pki_g["PKI Engine\nRoot + Intermediate CA\nfredcorp.com"] + transit_g["Transit Engine\nkey: unseal-beelink"] + kv_g["KV Engine\nadmin/ secrets"] + end + + vault_b -->|auto-unseal via transit| transit_g + vault_g -->|auto-unseal via transit| transit_b + + pki_b -->|signs certs for| cm_b["cert-manager\nbeelink"] + pki_g -->|signs certs for| cm_g["cert-manager\ngenmachine"] + kv_b -->|serves secrets| eso_b["ExternalSecrets\nbeelink"] + kv_g -->|serves secrets| eso_g["ExternalSecrets\ngenmachine"] + + style vault_b fill:#1a2a3a,stroke:#FFB81C,color:#FFB81C + style vault_g fill:#1a2a3a,stroke:#FFB81C,color:#FFB81C +``` + +## Transit Auto-Unseal + +Each Vault is configured to use the **other cluster's Transit engine** to automatically unseal on restart. This removes the need for manual Shamir key entry after a pod restart. + +```mermaid +sequenceDiagram + participant vb as Vault beelink + participant tg as Transit Engine\ngenmachine + participant vg as Vault genmachine + participant tb as Transit Engine\nbeelink + + Note over vb: Pod restarts (sealed) + vb->>tg: Decrypt unseal key\n(key: unseal-beelink) + tg-->>vb: Plaintext unseal key + Note over vb: Vault unsealed + + Note over vg: Pod restarts (sealed) + vg->>tb: Decrypt unseal key\n(key: unseal-genmachine) + tb-->>vg: Plaintext unseal key + Note over vg: Vault unsealed +``` + +### Helm configuration + +The transit seal block is injected into the Vault `standalone.config` HCL via a Kubernetes secret: + +```yaml +vault: + server: + extraSecretEnvironmentVars: + - envName: TRANSIT_UNSEAL_TOKEN + secretName: vault-transit-token + secretKey: token + standalone: + enabled: true + config: |- + ui = true + listener "tcp" { + tls_disable = 1 + address = "[::]:8200" + } + storage "file" { + path = "/vault/data" + } + seal "transit" { + address = "https://vault.talos-genmachine.fredcorp.com" + token = "$TRANSIT_UNSEAL_TOKEN" + key_name = "unseal-beelink" + mount_path = "transit/" + tls_skip_verify = "true" + } +``` + +The `vault-transit-token` secret must be created **before** ArgoCD syncs Vault. Use the Taskfile helpers: + +```bash +# Create the transit token secret on beelink (reads token from genmachine Vault) +task vault:unseal-secret:beelink + +# Create the transit token secret on genmachine (reads token from beelink Vault) +task vault:unseal-secret:genmachine +``` + +### Circular deadlock recovery + +> [!WARNING] +> If both Vaults restart simultaneously, they each try to contact the other's transit engine while both are sealed — causing a crash loop. + +```mermaid +flowchart TD + start["Both Vaults restart simultaneously"] + sealed_b["Vault beelink: sealed\ntrying to contact genmachine transit"] + sealed_g["Vault genmachine: sealed\ntrying to contact beelink transit"] + crash["Both in CrashLoopBackOff\n503 — no available server"] + + start --> sealed_b & sealed_g + sealed_b --> crash + sealed_g --> crash + + crash -->|recover one first| fix["1. Strip seal block from ConfigMap\n2. Delete pod → Shamir unseal\n3. Restore seal block\n4. vault operator unseal -migrate"] + fix -->|other unseals automatically| done["Both Vaults unsealed"] +``` + +Recovery is automated via Taskfile: + +```bash +# Recover beelink first (will allow genmachine to auto-unseal) +task vault:break-deadlock:beelink + +# Or recover genmachine first +task vault:break-deadlock:genmachine +``` + +## Seal Migration (Shamir → Transit) + +When enabling transit auto-unseal on an existing Vault initialized with Shamir keys, a seal migration is required. + +```mermaid +sequenceDiagram + participant ops as Operator + participant vault as Vault pod + participant cm as ConfigMap + + ops->>cm: Add seal "transit" block to config + ops->>vault: Delete pod (restart with new config) + Note over vault: Vault starts in migration mode + ops->>vault: vault operator unseal -migrate + Note over vault: Migration complete — sealed with transit + vault->>vault: Auto-unseal via transit on next restart +``` + +```bash +vault operator unseal -migrate -address=https://vault.k0s-fullstack.fredcorp.com +``` + +> [!NOTE] +> The `-migrate` flag is only accepted once — on the first unseal after the transit seal block is loaded. Subsequent restarts auto-unseal without any manual action. + +## Authentication Methods + +### Kubernetes Auth + +Used by cert-manager and ExternalSecrets to authenticate with Vault using their ServiceAccount tokens: + +```mermaid +sequenceDiagram + participant app as cert-manager / ESO + participant k8s as Kubernetes API + participant vault as Vault + + app->>vault: Login with ServiceAccount JWT\n(auth/-k8s/login) + vault->>k8s: TokenReview — validate JWT + k8s-->>vault: Token valid + bound SA info + vault-->>app: Vault token (scoped to policy) + app->>vault: Read secrets / sign certificates +``` + +Configure with: + +```bash +task vault:eso-auth-setup cluster=genmachine +task vault:certmanager-auth-setup cluster=genmachine +``` + +### OIDC Auth + +Human operators authenticate via Authentik SSO. See the [OIDC documentation](../authentication/oidc.md) for setup details. diff --git a/mkdocs.yml b/mkdocs.yml index 5c3cd06c5..33a228561 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -65,7 +65,11 @@ markdown_extensions: - pymdownx.blocks.caption - pymdownx.highlight: anchor_linenums: true - - pymdownx.superfences + - pymdownx.superfences: + custom_fences: + - name: mermaid + class: mermaid + format: !!python/name:pymdownx.superfences.fence_code_format - toc: permalink: true - pymdownx.tabbed: @@ -104,6 +108,7 @@ nav: - Proxmox-CSI: storage/csi-promox.md - Secrets: - ESO: secrets/externalsecrets.md + - Vault: secrets/vault.md - VPN: - Cluster config: VPN/wireguard.md - Encryption: From 2b063fa3775caa431ad43562f68aed586207495d Mon Sep 17 00:00:00 2001 From: ixxeL2097 Date: Sun, 26 Apr 2026 13:03:03 +0200 Subject: [PATCH 2/4] docs: rework index page with layered architecture diagrams and stack table MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Full page redesign: hero section, tabbed badges per cluster, stack table - Two Mermaid diagrams: infrastructure topology (layered TB graph with all components grouped by plane) and request flow (DNS → Traefik → ForwardAuth → app) - Badges reorganized into pymdownx.tabbed tabs (beelink vs genmachine) - Stack table listing every major component with its role Co-Authored-By: Claude Sonnet 4.6 --- docs/index.md | 215 +++++++++++++++++++++++++++++++------------------- 1 file changed, 136 insertions(+), 79 deletions(-) diff --git a/docs/index.md b/docs/index.md index 32e02d2b6..8872ec20a 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,117 +1,174 @@
-

+

+

ixxeL-DevOps HomeLab

+

Infrastructure as Code · GitOps · Self-hosted · Fully automated

-# **My home-lab repository** - -**Hosted with k0s & Talos** +--- -**Managed by ArgoCD** +[![pre-commit.ci status](https://results.pre-commit.ci/badge/github/ixxeL-DevOps/fullstack/main.svg)](https://results.pre-commit.ci/latest/github/ixxeL-DevOps/fullstack/main) +[![code style: prettier](https://img.shields.io/badge/code_style-prettier-ff69b4.svg?style=flat-square)](https://github.com/prettier/prettier) +![GitHub last commit (by committer)](https://img.shields.io/github/last-commit/ixxeL-DevOps/fullstack?style=flat-square) +![GitHub commit activity (branch)](https://img.shields.io/github/commit-activity/m/ixxeL-DevOps/fullstack?style=flat-square) +![Renovate](https://img.shields.io/badge/deps-renovate-ok?style=flat-square&logo=renovate&logoColor=%230099FF&logoSize=auto&color=%230099FF) -**Powered by Renovate and GitHub** +--- -**Fueled by Cilium** +## Overview ---- +This repository is the single source of truth for a fully automated homelab running on two Kubernetes clusters. Every component — from cluster bootstrap to application configuration — is declared as code, reconciled by ArgoCD, and kept up-to-date by Renovate. -**INFRASTRUCTURE K0S** +The two clusters are complementary: **Beelink** (k0s on bare metal) hosts security and access infrastructure, while **Genmachine** (Talos Linux on Proxmox VMs) hosts the production workloads and observability stack. Both share a unified GitOps structure and are managed from a single repository. -![K0sVersion](https://img.shields.io/badge/dynamic/yaml?url=https%3A%2F%2Fraw.githubusercontent.com%2FixxeL-DevOps%2Ffullstack%2Frefs%2Fheads%2Fmain%2Finfra%2Fk0s%2Ffullstack.yaml&query=%24.spec.k0s.version&style=for-the-badge&logo=kubernetes&logoColor=%23326CE5&label=k0s&color=%23326CE5) -![ArgoCD](https://img.shields.io/badge/argocd-v2.14.8-version?style=for-the-badge&logo=argo&logoColor=%23F76B39&color=%23F76B39) +=== "Beelink — k0s" -**TOOLING** + **Cluster** -![traefik](https://img.shields.io/badge/dynamic/yaml?url=https%3A%2F%2Fraw.githubusercontent.com%2FixxeL-DevOps%2Ffullstack%2Frefs%2Fheads%2Fmain%2Fgitops%2Fmanifests%2Ftraefik%2Fbeelink%2Fbeelink-values.yaml&query=%24.traefik.image.tag&style=for-the-badge&logo=traefikproxy&logoColor=%239D0FB0&label=traefik&color=%239D0FB0) -![adguard](https://img.shields.io/badge/dynamic/yaml?url=https%3A%2F%2Fraw.githubusercontent.com%2FixxeL-DevOps%2Ffullstack%2Frefs%2Fheads%2Fmain%2Fgitops%2Fmanifests%2Fadguard%2Fbeelink%2Fbeelink-values.yaml&query=%24.adguard-home.image.tag&style=for-the-badge&logo=adguard&label=AdGuard&color=%2366B574) -![authentik](https://img.shields.io/badge/dynamic/yaml?url=https%3A%2F%2Fraw.githubusercontent.com%2FixxeL-DevOps%2Ffullstack%2Frefs%2Fheads%2Fmain%2Fgitops%2Fmanifests%2Fauthentik%2Fbeelink%2Fapp%2Fbeelink-values.yaml&query=%24.authentik.global.image.tag&style=for-the-badge&logo=authentik&label=Authentik&color=%23FD4B2D) -![vault](https://img.shields.io/badge/dynamic/yaml?url=https%3A%2F%2Fraw.githubusercontent.com%2FixxeL-DevOps%2Ffullstack%2Frefs%2Fheads%2Fmain%2Fgitops%2Fmanifests%2Fvault%2Fbeelink%2Fbeelink-values.yaml&query=%24.vault.server.image.tag&style=for-the-badge&logo=vault&label=Vault&color=%23FFB81C) -![wireguard](https://img.shields.io/badge/dynamic/yaml?url=https%3A%2F%2Fraw.githubusercontent.com%2FixxeL-DevOps%2Ffullstack%2Frefs%2Fheads%2Fmain%2Fgitops%2Fmanifests%2Fwireguard%2Fbeelink%2Fbeelink-values.yaml&query=%24.wg-portal.image.tag&style=for-the-badge&logo=wireguard&logoColor=%23841618&label=wireguard&color=%23841618) -![homarr](https://img.shields.io/badge/dynamic/yaml?url=https%3A%2F%2Fraw.githubusercontent.com%2FixxeL-DevOps%2Ffullstack%2Frefs%2Fheads%2Fmain%2Fgitops%2Fmanifests%2Fhomarr%2Fbeelink%2Fbeelink-values.yaml&query=%24.homarr.image.tag&style=for-the-badge&logo=homarr&label=homarr&color=%23F44336) + ![K0sVersion](https://img.shields.io/badge/dynamic/yaml?url=https%3A%2F%2Fraw.githubusercontent.com%2FixxeL-DevOps%2Ffullstack%2Frefs%2Fheads%2Fmain%2Finfra%2Fk0s%2Ffullstack.yaml&query=%24.spec.k0s.version&style=for-the-badge&logo=kubernetes&logoColor=%23326CE5&label=k0s&color=%23326CE5) + ![ArgoCD](https://img.shields.io/badge/argocd-v2.14.8-version?style=for-the-badge&logo=argo&logoColor=%23F76B39&color=%23F76B39) ---- + **Applications** -**INFRASTRUCTURE TALOS** + ![traefik](https://img.shields.io/badge/dynamic/yaml?url=https%3A%2F%2Fraw.githubusercontent.com%2FixxeL-DevOps%2Ffullstack%2Frefs%2Fheads%2Fmain%2Fgitops%2Fmanifests%2Ftraefik%2Fbeelink%2Fbeelink-values.yaml&query=%24.traefik.image.tag&style=for-the-badge&logo=traefikproxy&logoColor=%239D0FB0&label=traefik&color=%239D0FB0) + ![adguard](https://img.shields.io/badge/dynamic/yaml?url=https%3A%2F%2Fraw.githubusercontent.com%2FixxeL-DevOps%2Ffullstack%2Frefs%2Fheads%2Fmain%2Fgitops%2Fmanifests%2Fadguard%2Fbeelink%2Fbeelink-values.yaml&query=%24.adguard-home.image.tag&style=for-the-badge&logo=adguard&label=AdGuard&color=%2366B574) + ![authentik](https://img.shields.io/badge/dynamic/yaml?url=https%3A%2F%2Fraw.githubusercontent.com%2FixxeL-DevOps%2Ffullstack%2Frefs%2Fheads%2Fmain%2Fgitops%2Fmanifests%2Fauthentik%2Fbeelink%2Fapp%2Fbeelink-values.yaml&query=%24.authentik.global.image.tag&style=for-the-badge&logo=authentik&label=Authentik&color=%23FD4B2D) + ![vault](https://img.shields.io/badge/dynamic/yaml?url=https%3A%2F%2Fraw.githubusercontent.com%2FixxeL-DevOps%2Ffullstack%2Frefs%2Fheads%2Fmain%2Fgitops%2Fmanifests%2Fvault%2Fbeelink%2Fbeelink-values.yaml&query=%24.vault.server.image.tag&style=for-the-badge&logo=vault&label=Vault&color=%23FFB81C) + ![wireguard](https://img.shields.io/badge/dynamic/yaml?url=https%3A%2F%2Fraw.githubusercontent.com%2FixxeL-DevOps%2Ffullstack%2Frefs%2Fheads%2Fmain%2Fgitops%2Fmanifests%2Fwireguard%2Fbeelink%2Fbeelink-values.yaml&query=%24.wg-portal.image.tag&style=for-the-badge&logo=wireguard&logoColor=%23841618&label=wireguard&color=%23841618) + ![homarr](https://img.shields.io/badge/dynamic/yaml?url=https%3A%2F%2Fraw.githubusercontent.com%2FixxeL-DevOps%2Ffullstack%2Frefs%2Fheads%2Fmain%2Fgitops%2Fmanifests%2Fhomarr%2Fbeelink%2Fbeelink-values.yaml&query=%24.homarr.image.tag&style=for-the-badge&logo=homarr&label=homarr&color=%23F44336) -![TalosVersion](https://img.shields.io/badge/dynamic/yaml?url=https%3A%2F%2Fraw.githubusercontent.com%2FixxeL-DevOps%2Ffullstack%2Frefs%2Fheads%2Fmain%2Finfra%2Ftalos%2Fgenmachine%2Fbootstrap%2Ftalconfig.yaml&query=%24.talosVersion&style=for-the-badge&logo=talos&label=Talos&color=%23FF4400) -![k8s](https://img.shields.io/badge/dynamic/yaml?url=https%3A%2F%2Fraw.githubusercontent.com%2FixxeL-DevOps%2Ffullstack%2Frefs%2Fheads%2Fmain%2Finfra%2Ftalos%2Fgenmachine%2Fbootstrap%2Ftalconfig.yaml&query=%24.kubernetesVersion&style=for-the-badge&logo=kubernetes&label=K8s&color=%23326CE5) -![Cilium](https://img.shields.io/badge/dynamic/yaml?url=https%3A%2F%2Fraw.githubusercontent.com%2FixxeL-DevOps%2Ffullstack%2Frefs%2Fheads%2Fmain%2Fgitops%2Fmanifests%2Fcilium%2Fgenmachine%2Fgenmachine-values.yaml&query=%24.cilium.image.tag&style=for-the-badge&logo=cilium&label=Cilium&color=%23E9B824) +=== "Genmachine — Talos" -**TOOLING** + **Cluster** -![Argocd](https://img.shields.io/badge/dynamic/yaml?url=https%3A%2F%2Fraw.githubusercontent.com%2FixxeL-DevOps%2Ffullstack%2Frefs%2Fheads%2Fmain%2Fgitops%2Fbootstrap%2Fgenmachine%2Fgenmachine-values.yaml&query=%24.argo-cd.global.image.tag&style=for-the-badge&logo=argo&label=Argocd&color=%23EF5A29) -![traefik](https://img.shields.io/badge/dynamic/yaml?url=https%3A%2F%2Fraw.githubusercontent.com%2FixxeL-DevOps%2Ffullstack%2Frefs%2Fheads%2Fmain%2Fgitops%2Fmanifests%2Ftraefik%2Fgenmachine%2Fgenmachine-values.yaml&query=%24.traefik.image.tag&style=for-the-badge&logo=traefikproxy&logoColor=%239D0FB0&label=traefik&color=%239D0FB0) -![prometheus](https://img.shields.io/badge/dynamic/yaml?url=https%3A%2F%2Fraw.githubusercontent.com%2FixxeL-DevOps%2Ffullstack%2Frefs%2Fheads%2Fmain%2Fgitops%2Fmanifests%2Fprometheus%2Fgenmachine%2Fgenmachine-values.yaml&query=%24.kube-prometheus-stack.prometheus.prometheusSpec.image.tag&style=for-the-badge&logo=prometheus&label=prometheus&color=%23E6522C) -![grafana](https://img.shields.io/badge/dynamic/yaml?url=https%3A%2F%2Fraw.githubusercontent.com%2FixxeL-DevOps%2Ffullstack%2Frefs%2Fheads%2Fmain%2Fgitops%2Fmanifests%2Fprometheus%2Fgenmachine%2Fgenmachine-values.yaml&query=%24.kube-prometheus-stack.grafana.image.tag&style=for-the-badge&logo=grafana&label=grafana&color=%23F46800) -![minio](https://img.shields.io/badge/dynamic/yaml?url=https%3A%2F%2Fraw.githubusercontent.com%2FixxeL-DevOps%2Ffullstack%2Frefs%2Fheads%2Fmain%2Fgitops%2Fmanifests%2Fminio%2Fgenmachine%2Fgenmachine-values.yaml&query=%24.minio.image.tag&style=for-the-badge&logo=minio&logoColor=e0e0e0&label=minio&color=e0e0e0) -![adguard](https://img.shields.io/badge/dynamic/yaml?url=https%3A%2F%2Fraw.githubusercontent.com%2FixxeL-DevOps%2Ffullstack%2Frefs%2Fheads%2Fmain%2Fgitops%2Fmanifests%2Fadguard%2Fgenmachine%2Fgenmachine-values.yaml&query=%24.adguard-home.image.tag&style=for-the-badge&logo=adguard&label=AdGuard&color=%2366B574) + ![TalosVersion](https://img.shields.io/badge/dynamic/yaml?url=https%3A%2F%2Fraw.githubusercontent.com%2FixxeL-DevOps%2Ffullstack%2Frefs%2Fheads%2Fmain%2Finfra%2Ftalos%2Fgenmachine%2Fbootstrap%2Ftalconfig.yaml&query=%24.talosVersion&style=for-the-badge&logo=talos&label=Talos&color=%23FF4400) + ![k8s](https://img.shields.io/badge/dynamic/yaml?url=https%3A%2F%2Fraw.githubusercontent.com%2FixxeL-DevOps%2Ffullstack%2Frefs%2Fheads%2Fmain%2Finfra%2Ftalos%2Fgenmachine%2Fbootstrap%2Ftalconfig.yaml&query=%24.kubernetesVersion&style=for-the-badge&logo=kubernetes&label=K8s&color=%23326CE5) + ![Cilium](https://img.shields.io/badge/dynamic/yaml?url=https%3A%2F%2Fraw.githubusercontent.com%2FixxeL-DevOps%2Ffullstack%2Frefs%2Fheads%2Fmain%2Fgitops%2Fmanifests%2Fcilium%2Fgenmachine%2Fgenmachine-values.yaml&query=%24.cilium.image.tag&style=for-the-badge&logo=cilium&label=Cilium&color=%23E9B824) ---- + **Applications** -[![pre-commit.ci status](https://results.pre-commit.ci/badge/github/ixxeL-DevOps/fullstack/main.svg)](https://results.pre-commit.ci/latest/github/ixxeL-DevOps/fullstack/main) -[![code style: prettier](https://img.shields.io/badge/code_style-prettier-ff69b4.svg?style=flat-square)](https://github.com/prettier/prettier) -![GitHub last commit (by committer)](https://img.shields.io/github/last-commit/ixxeL-DevOps/fullstack?style=flat-square) -![GitHub commit activity (branch)](https://img.shields.io/github/commit-activity/m/ixxeL-DevOps/fullstack?style=flat-square) -![Renovate](https://img.shields.io/badge/deps-renovate-ok?style=flat-square&logo=renovate&logoColor=%230099FF&logoSize=auto&color=%230099FF) + ![Argocd](https://img.shields.io/badge/dynamic/yaml?url=https%3A%2F%2Fraw.githubusercontent.com%2FixxeL-DevOps%2Ffullstack%2Frefs%2Fheads%2Fmain%2Fgitops%2Fbootstrap%2Fgenmachine%2Fgenmachine-values.yaml&query=%24.argo-cd.global.image.tag&style=for-the-badge&logo=argo&label=Argocd&color=%23EF5A29) + ![traefik](https://img.shields.io/badge/dynamic/yaml?url=https%3A%2F%2Fraw.githubusercontent.com%2FixxeL-DevOps%2Ffullstack%2Frefs%2Fheads%2Fmain%2Fgitops%2Fmanifests%2Ftraefik%2Fgenmachine%2Fgenmachine-values.yaml&query=%24.traefik.image.tag&style=for-the-badge&logo=traefikproxy&logoColor=%239D0FB0&label=traefik&color=%239D0FB0) + ![prometheus](https://img.shields.io/badge/dynamic/yaml?url=https%3A%2F%2Fraw.githubusercontent.com%2FixxeL-DevOps%2Ffullstack%2Frefs%2Fheads%2Fmain%2Fgitops%2Fmanifests%2Fprometheus%2Fgenmachine%2Fgenmachine-values.yaml&query=%24.kube-prometheus-stack.prometheus.prometheusSpec.image.tag&style=for-the-badge&logo=prometheus&label=prometheus&color=%23E6522C) + ![grafana](https://img.shields.io/badge/dynamic/yaml?url=https%3A%2F%2Fraw.githubusercontent.com%2FixxeL-DevOps%2Ffullstack%2Frefs%2Fheads%2Fmain%2Fgitops%2Fmanifests%2Fprometheus%2Fgenmachine%2Fgenmachine-values.yaml&query=%24.kube-prometheus-stack.grafana.image.tag&style=for-the-badge&logo=grafana&label=grafana&color=%23F46800) + ![minio](https://img.shields.io/badge/dynamic/yaml?url=https%3A%2F%2Fraw.githubusercontent.com%2FixxeL-DevOps%2Ffullstack%2Frefs%2Fheads%2Fmain%2Fgitops%2Fmanifests%2Fminio%2Fgenmachine%2Fgenmachine-values.yaml&query=%24.minio.image.tag&style=for-the-badge&logo=minio&logoColor=e0e0e0&label=minio&color=e0e0e0) + ![adguard](https://img.shields.io/badge/dynamic/yaml?url=https%3A%2F%2Fraw.githubusercontent.com%2FixxeL-DevOps%2Ffullstack%2Frefs%2Fheads%2Fmain%2Fgitops%2Fmanifests%2Fadguard%2Fgenmachine%2Fgenmachine-values.yaml&query=%24.adguard-home.image.tag&style=for-the-badge&logo=adguard&label=AdGuard&color=%2366B574) --- -This project utilizes Infrastructure as Code and GitOps to automate provisioning, operating, and updating self-hosted services in my homelab. +## Architecture -## Architecture Overview +### Infrastructure topology ```mermaid graph TB - subgraph internet["Internet"] - user["User"] + subgraph github[" GitHub "] + repo[("fullstack\nrepository")] + renovate["Renovate\nauto-update PRs"] + ci["GitHub Actions\nhelm diff · kubeconform"] end - subgraph home["Home Network (192.168.1.x)"] - router["Router"] - adguard["AdGuard Home\nDNS resolver"] + subgraph access[" Access layer "] + user(["User"]) + vpn["WireGuard\nVPN"] + adguard["AdGuard Home\nDNS · *.fredcorp.com"] end - subgraph github["GitHub"] - repo["fullstack\nrepository"] - renovate["Renovate Bot\nautomated PRs"] - actions["GitHub Actions\nCI / helm-rmp"] - end - - subgraph beelink["Beelink — k0s cluster"] - traefik_b["Traefik\nIngress"] + subgraph beelink[" Beelink — k0s (bare metal) "] + direction TB + traefik_b["Traefik\nIngress + TLS termination"] + subgraph security["Security plane"] + authentik["Authentik\nSSO / IdP"] + vault_b["Vault\nPKI · KV · Transit"] + certmgr_b["cert-manager\n+ trust-manager"] + eso_b["ExternalSecrets"] + end + subgraph apps_b["Applications"] + homarr["Homarr\ndashboard"] + adguard_b["AdGuard\nDNS"] + end argocd_b["ArgoCD"] - authentik["Authentik\nIdP / SSO"] - vault_b["Vault\nbeelink"] - wireguard["WireGuard\nVPN"] end - subgraph genmachine["Genmachine — Talos cluster"] - subgraph proxmox["Proxmox hypervisor"] - vm1["talos-1\n192.168.1.151"] - vm2["talos-2\n192.168.1.152"] - vm3["talos-3\n192.168.1.153"] + subgraph genmachine[" Genmachine — Talos (Proxmox VMs) "] + direction TB + subgraph vms["3× control-plane VMs · etcd · Cilium CNI"] + traefik_g["Traefik\nIngress + TLS termination"] + subgraph obs["Observability"] + prometheus["Prometheus\n+ Grafana"] + loki["Loki\nlog aggregation"] + end + subgraph storage_g["Storage"] + minio["MinIO\nobject store"] + csi["Proxmox CSI\nblock volumes"] + end + vault_g["Vault\nPKI · KV · Transit"] + certmgr_g["cert-manager"] + eso_g["ExternalSecrets"] + argocd_g["ArgoCD"] end - traefik_g["Traefik\nIngress"] - argocd_g["ArgoCD"] - vault_g["Vault\ngenmachine"] - prometheus["Prometheus\n+ Grafana"] - minio["MinIO\nobject storage"] end - user -->|HTTPS| router - router -->|ingress| traefik_b - router -->|ingress| traefik_g - router -->|DNS| adguard - - argocd_b -->|pull| repo - argocd_g -->|pull| repo - renovate -->|auto-PR| repo - actions -->|helm diff| repo + user -->|"HTTPS / DNS"| adguard + user -->|"WireGuard tunnel"| vpn + adguard -->|"routes *.fredcorp.com"| traefik_b & traefik_g + traefik_b -->|"ForwardAuth"| authentik + + renovate -->|"auto-PR"| repo + ci -->|"diff preview"| repo + argocd_b & argocd_g -->|"pull reconcile"| repo + + vault_b <-->|"transit auto-unseal"| vault_g + + style github fill:#21262d,stroke:#6e40c9,color:#e6edf3 + style access fill:#161b22,stroke:#3fb950,color:#e6edf3 + style beelink fill:#0d2137,stroke:#4d94ff,color:#e6edf3 + style security fill:#0d1a2e,stroke:#4d94ff,color:#e6edf3 + style apps_b fill:#0d1a2e,stroke:#4d94ff,color:#e6edf3 + style genmachine fill:#0d2010,stroke:#3fb950,color:#e6edf3 + style vms fill:#0d1a0d,stroke:#3fb950,color:#e6edf3 + style obs fill:#0d1a0d,stroke:#3fb950,color:#e6edf3 + style storage_g fill:#0d1a0d,stroke:#3fb950,color:#e6edf3 +``` - vault_b <-->|transit auto-unseal| vault_g +### Request flow — authenticated access - style beelink fill:#1a3a5c,stroke:#4d94ff,color:#fff - style genmachine fill:#1a3a1a,stroke:#4dff4d,color:#fff - style github fill:#2d1a3a,stroke:#9d4dff,color:#fff - style home fill:#3a2d1a,stroke:#ffb84d,color:#fff - style proxmox fill:#2a3a2a,stroke:#4dff4d,color:#fff +```mermaid +flowchart LR + user(["User"]) + dns["AdGuard\nDNS"] + traefik["Traefik"] + fwdauth["Authentik\noutpost"] + app["Application"] + vault["Vault\nPKI"] + certmgr["cert-manager"] + + user -->|"1 · DNS lookup\n*.fredcorp.com"| dns + dns -->|"2 · resolves to\nMetalLB IP"| traefik + traefik -->|"3 · ForwardAuth\nmiddleware"| fwdauth + fwdauth -->|"4 · 401 → login\nor 200 + headers"| traefik + traefik -->|"5 · proxy +\nX-authentik-* headers"| app + certmgr -->|"issues TLS cert\nfrom Vault PKI"| traefik + vault -->|"signs CSR"| certmgr ``` + +--- + +## Stack + +| Layer | Component | Role | +|---|---|---| +| **Cluster** | [Talos Linux](https://www.talos.dev/) | Immutable, API-driven OS for Genmachine nodes | +| **Cluster** | [k0s](https://k0sproject.io/) | Lightweight single-node Kubernetes for Beelink | +| **GitOps** | [ArgoCD](https://argo-cd.readthedocs.io/) | Continuous reconciliation of all cluster state | +| **GitOps** | [Renovate](https://docs.renovatebot.com/) | Automated dependency update PRs | +| **Networking** | [Cilium](https://cilium.io/) | eBPF CNI with L2 LoadBalancer announcements | +| **Ingress** | [Traefik](https://traefik.io/) | Reverse proxy with automatic TLS and ForwardAuth | +| **DNS** | [AdGuard Home](https://adguard.com/adguard-home/) | Local DNS resolver with ad-blocking | +| **PKI / Secrets** | [HashiCorp Vault](https://www.vaultproject.io/) | PKI CA, KV secrets, Transit auto-unseal | +| **Certificates** | [cert-manager](https://cert-manager.io/) | Automated certificate lifecycle from Vault PKI | +| **Secrets** | [ExternalSecrets](https://external-secrets.io/) | Vault → Kubernetes Secret synchronisation | +| **Auth** | [Authentik](https://goauthentik.io/) | SSO IdP — OIDC provider + ForwardAuth outpost | +| **VPN** | [WireGuard Portal](https://github.com/h44z/wg-portal) | Self-hosted VPN management UI | +| **Observability** | Prometheus · Grafana · Loki | Metrics, dashboards, and log aggregation | +| **Storage** | MinIO · Proxmox CSI | S3-compatible object store + block volumes | +| **Encryption** | [SOPS](https://github.com/getsops/sops) | Secrets encryption in Git via Vault Transit | From 70f4cf7356d5459550f46ba563962a987ec6b659 Mon Sep 17 00:00:00 2001 From: ixxeL2097 Date: Sun, 26 Apr 2026 13:13:07 +0200 Subject: [PATCH 3/4] chore: replace devbox with mise + uv for tooling management mise manages all CLI tools (kubectl, helm, vault, sops, talosctl, etc.) and Python/Node runtimes. uv manages Python packages declared in pyproject.toml; the .venv is auto-activated via mise's _.python.venv. First-time setup: mise install && task setup (or uv sync + helm/krew plugins) Co-Authored-By: Claude Sonnet 4.6 --- .gitignore | 5 ++++ mise.toml | 67 ++++++++++++++++++++++++++++++++++++++++++++++++++ pyproject.toml | 23 +++++++++++++++++ 3 files changed, 95 insertions(+) create mode 100644 mise.toml create mode 100644 pyproject.toml diff --git a/.gitignore b/.gitignore index cf9002d5c..0f49306bd 100644 --- a/.gitignore +++ b/.gitignore @@ -26,6 +26,11 @@ k3d-home/k3d.yaml node_modules .devbox +.mise-local.toml + +# uv / Python — .venv is excluded, uv.lock is committed intentionally +__pycache__/ +*.pyc # database *.sqlite diff --git a/mise.toml b/mise.toml new file mode 100644 index 000000000..96191ff56 --- /dev/null +++ b/mise.toml @@ -0,0 +1,67 @@ +# yaml-language-server: $schema=https://mise.jdx.dev/schema/mise.json +[tools] + +# ── Python (managed by uv, packages declared in pyproject.toml) ───────────── +python = "3.12" +uv = "latest" + +# ── Node (required for prettier) ───────────────────────────────────────────── +node = "lts" + +# ── Kubernetes ──────────────────────────────────────────────────────────────── +kubectl = "latest" +helm = "latest" +k9s = "latest" +kustomize = "latest" +talosctl = "latest" +argocd = "latest" +helmfile = "latest" +k0sctl = "latest" +kubectx = "latest" +krew = "latest" +popeye = "latest" + +# ── Secrets & encryption ────────────────────────────────────────────────────── +vault = "latest" +sops = "latest" +age = "latest" + +# ── Infrastructure ──────────────────────────────────────────────────────────── +terraform = "latest" + +# ── Dev tooling ─────────────────────────────────────────────────────────────── +task = "latest" +yq = "latest" +jq = "latest" +shellcheck = "latest" +yamlfmt = "latest" +gum = "latest" +restic = "latest" +"npm:prettier" = "latest" + +# ── Python venv ─────────────────────────────────────────────────────────────── +# Auto-activates the uv-managed venv when entering the repo directory. +# Run `uv sync` once after `mise install` to populate it. +[env] +_.python.venv = { path = ".venv", create = true } + +# ── Post-install tasks ──────────────────────────────────────────────────────── +[tasks.setup] +description = "Full first-time setup: install Python packages + Helm/kubectl plugins" +run = """ + uv sync + helm plugin install https://github.com/databus23/helm-diff 2>/dev/null || true + kubectl krew install neat 2>/dev/null || true +""" + +[tasks.update] +description = "Update all Python packages and regenerate uv.lock" +run = "uv sync --upgrade" + +[tasks.docs] +description = "Serve MkDocs documentation locally" +run = "mkdocs serve" + +[tasks.docs-build] +description = "Build static MkDocs site" +run = "mkdocs build --strict" diff --git a/pyproject.toml b/pyproject.toml new file mode 100644 index 000000000..8bb6828cc --- /dev/null +++ b/pyproject.toml @@ -0,0 +1,23 @@ +[project] +name = "fullstack" +version = "0.1.0" +description = "HomeLab GitOps tooling — Python dev dependencies" +requires-python = ">=3.12" +dependencies = [ + # ── MkDocs documentation stack ──────────────────────────────────────────── + "mkdocs-material>=9", + "mkdocs-git-revision-date-localized-plugin", + "mkdocs-git-committers-plugin-2", + "mkdocs-git-authors-plugin", + "mkdocs-awesome-pages-plugin", + "mkdocs-minify-plugin", + # ── Linting & commit tooling ────────────────────────────────────────────── + "yamllint", + "pre-commit", + "commitizen", + # ── GitGuardian (used in pre-commit hooks) ──────────────────────────────── + "pygitguardian", + "oauthlib", + "cffi", + "truststore", +] From 332759ea9691561359cf2de0a948a92201697379 Mon Sep 17 00:00:00 2001 From: ixxeL2097 Date: Sun, 26 Apr 2026 14:39:11 +0200 Subject: [PATCH 4/4] docs: remove vault transit auto-unseal, rework README with Mermaid diagrams - vault.md: remove Transit Auto-Unseal and Seal Migration sections; keep PKI/KV overview and Kubernetes auth flow - index.md: remove transit auto-unseal arrow from architecture diagram - README.md: replace Excalidraw SVG images with native GitHub Mermaid diagrams; fix traefik beelink badge URL (was pointing to old k0s path); reorganize badges by cluster; add GitOps flow diagram and stack table Co-Authored-By: Claude Sonnet 4.6 --- README.md | 162 +++++++++++++++++++++++++++--------------- docs/index.md | 2 - docs/secrets/vault.md | 134 ++-------------------------------- 3 files changed, 111 insertions(+), 187 deletions(-) diff --git a/README.md b/README.md index f42744a9b..83519ece5 100644 --- a/README.md +++ b/README.md @@ -1,52 +1,36 @@
-

+ - +# ixxeL-DevOps HomeLab -### My home-lab repository :rocket: - -✨*managed with k0s/Talos, ArgoCD, Renovate and GitHub*✨ +_Infrastructure as Code · GitOps · Self-hosted · Fully automated_
---- - -**INFRASTRUCTURE K0S** +**BEELINK — k0s** ![K0sVersion](https://img.shields.io/badge/dynamic/yaml?url=https%3A%2F%2Fraw.githubusercontent.com%2FixxeL-DevOps%2Ffullstack%2Frefs%2Fheads%2Fmain%2Finfra%2Fk0s%2Ffullstack.yaml&query=%24.spec.k0s.version&style=for-the-badge&logo=kubernetes&logoColor=%23326CE5&label=k0s&color=%23326CE5) ![ArgoCD](https://img.shields.io/badge/argocd-v2.14.8-version?style=for-the-badge&logo=argo&logoColor=%23F76B39&color=%23F76B39) - -**TOOLING** - -![traefik](https://img.shields.io/badge/dynamic/yaml?url=https%3A%2F%2Fraw.githubusercontent.com%2FixxeL-DevOps%2Ffullstack%2Frefs%2Fheads%2Fmain%2Fgitops%2Fmanifests%2Ftraefik%2Fk0s%2Fk0s-values.yaml&query=%24.traefik.image.tag&style=for-the-badge&logo=traefikproxy&logoColor=%239D0FB0&label=traefik&color=%239D0FB0) -![adguard](https://img.shields.io/badge/dynamic/yaml?url=https%3A%2F%2Fraw.githubusercontent.com%2FixxeL-DevOps%2Ffullstack%2Frefs%2Fheads%2Fmain%2Fgitops%2Fmanifests%2Fadguard%2Fbeelink%2Fbeelink-values.yaml&query=%24.adguard-home.image.tag&style=for-the-badge&logo=adguard&label=AdGuard&color=%2366B574) -![authentik](https://img.shields.io/badge/dynamic/yaml?url=https%3A%2F%2Fraw.githubusercontent.com%2FixxeL-DevOps%2Ffullstack%2Frefs%2Fheads%2Fmain%2Fgitops%2Fmanifests%2Fauthentik%2Fbeelink%2Fapp%2Fbeelink-values.yaml&query=%24.authentik.global.image.tag&style=for-the-badge&logo=authentik&label=Authentik&color=%23FD4B2D) +![traefik](https://img.shields.io/badge/dynamic/yaml?url=https%3A%2F%2Fraw.githubusercontent.com%2FixxeL-DevOps%2Ffullstack%2Frefs%2Fheads%2Fmain%2Fgitops%2Fmanifests%2Ftraefik%2Fbeelink%2Fbeelink-values.yaml&query=%24.traefik.image.tag&style=for-the-badge&logo=traefikproxy&logoColor=%239D0FB0&label=traefik&color=%239D0FB0) ![vault](https://img.shields.io/badge/dynamic/yaml?url=https%3A%2F%2Fraw.githubusercontent.com%2FixxeL-DevOps%2Ffullstack%2Frefs%2Fheads%2Fmain%2Fgitops%2Fmanifests%2Fvault%2Fbeelink%2Fbeelink-values.yaml&query=%24.vault.server.image.tag&style=for-the-badge&logo=vault&label=Vault&color=%23FFB81C) +![authentik](https://img.shields.io/badge/dynamic/yaml?url=https%3A%2F%2Fraw.githubusercontent.com%2FixxeL-DevOps%2Ffullstack%2Frefs%2Fheads%2Fmain%2Fgitops%2Fmanifests%2Fauthentik%2Fbeelink%2Fapp%2Fbeelink-values.yaml&query=%24.authentik.global.image.tag&style=for-the-badge&logo=authentik&label=Authentik&color=%23FD4B2D) ![wireguard](https://img.shields.io/badge/dynamic/yaml?url=https%3A%2F%2Fraw.githubusercontent.com%2FixxeL-DevOps%2Ffullstack%2Frefs%2Fheads%2Fmain%2Fgitops%2Fmanifests%2Fwireguard%2Fbeelink%2Fbeelink-values.yaml&query=%24.wg-portal.image.tag&style=for-the-badge&logo=wireguard&logoColor=%23841618&label=wireguard&color=%23841618) +![adguard](https://img.shields.io/badge/dynamic/yaml?url=https%3A%2F%2Fraw.githubusercontent.com%2FixxeL-DevOps%2Ffullstack%2Frefs%2Fheads%2Fmain%2Fgitops%2Fmanifests%2Fadguard%2Fbeelink%2Fbeelink-values.yaml&query=%24.adguard-home.image.tag&style=for-the-badge&logo=adguard&label=AdGuard&color=%2366B574) ![homarr](https://img.shields.io/badge/dynamic/yaml?url=https%3A%2F%2Fraw.githubusercontent.com%2FixxeL-DevOps%2Ffullstack%2Frefs%2Fheads%2Fmain%2Fgitops%2Fmanifests%2Fhomarr%2Fbeelink%2Fbeelink-values.yaml&query=%24.homarr.image.tag&style=for-the-badge&logo=homarr&label=homarr&color=%23F44336) -
- -
- ---- - -**INFRASTRUCTURE TALOS** +**GENMACHINE — Talos** ![TalosVersion](https://img.shields.io/badge/dynamic/yaml?url=https%3A%2F%2Fraw.githubusercontent.com%2FixxeL-DevOps%2Ffullstack%2Frefs%2Fheads%2Fmain%2Finfra%2Ftalos%2Fgenmachine%2Fbootstrap%2Ftalconfig.yaml&query=%24.talosVersion&style=for-the-badge&logo=talos&label=Talos&color=%23FF4400) ![k8s](https://img.shields.io/badge/dynamic/yaml?url=https%3A%2F%2Fraw.githubusercontent.com%2FixxeL-DevOps%2Ffullstack%2Frefs%2Fheads%2Fmain%2Finfra%2Ftalos%2Fgenmachine%2Fbootstrap%2Ftalconfig.yaml&query=%24.kubernetesVersion&style=for-the-badge&logo=kubernetes&label=K8s&color=%23326CE5) ![Cilium](https://img.shields.io/badge/dynamic/yaml?url=https%3A%2F%2Fraw.githubusercontent.com%2FixxeL-DevOps%2Ffullstack%2Frefs%2Fheads%2Fmain%2Fgitops%2Fmanifests%2Fcilium%2Fgenmachine%2Fgenmachine-values.yaml&query=%24.cilium.image.tag&style=for-the-badge&logo=cilium&label=Cilium&color=%23E9B824) - -**TOOLING** - ![Argocd](https://img.shields.io/badge/dynamic/yaml?url=https%3A%2F%2Fraw.githubusercontent.com%2FixxeL-DevOps%2Ffullstack%2Frefs%2Fheads%2Fmain%2Fgitops%2Fbootstrap%2Fgenmachine%2Fgenmachine-values.yaml&query=%24.argo-cd.global.image.tag&style=for-the-badge&logo=argo&label=Argocd&color=%23EF5A29) ![traefik](https://img.shields.io/badge/dynamic/yaml?url=https%3A%2F%2Fraw.githubusercontent.com%2FixxeL-DevOps%2Ffullstack%2Frefs%2Fheads%2Fmain%2Fgitops%2Fmanifests%2Ftraefik%2Fgenmachine%2Fgenmachine-values.yaml&query=%24.traefik.image.tag&style=for-the-badge&logo=traefikproxy&logoColor=%239D0FB0&label=traefik&color=%239D0FB0) +![vault](https://img.shields.io/badge/dynamic/yaml?url=https%3A%2F%2Fraw.githubusercontent.com%2FixxeL-DevOps%2Ffullstack%2Frefs%2Fheads%2Fmain%2Fgitops%2Fmanifests%2Fvault%2Fgenmachine%2Fgenmachine-values.yaml&query=%24.vault.server.image.tag&style=for-the-badge&logo=vault&label=Vault&color=%23FFB81C) ![prometheus](https://img.shields.io/badge/dynamic/yaml?url=https%3A%2F%2Fraw.githubusercontent.com%2FixxeL-DevOps%2Ffullstack%2Frefs%2Fheads%2Fmain%2Fgitops%2Fmanifests%2Fprometheus%2Fgenmachine%2Fgenmachine-values.yaml&query=%24.kube-prometheus-stack.prometheus.prometheusSpec.image.tag&style=for-the-badge&logo=prometheus&label=prometheus&color=%23E6522C) ![grafana](https://img.shields.io/badge/dynamic/yaml?url=https%3A%2F%2Fraw.githubusercontent.com%2FixxeL-DevOps%2Ffullstack%2Frefs%2Fheads%2Fmain%2Fgitops%2Fmanifests%2Fprometheus%2Fgenmachine%2Fgenmachine-values.yaml&query=%24.kube-prometheus-stack.grafana.image.tag&style=for-the-badge&logo=grafana&label=grafana&color=%23F46800) ![minio](https://img.shields.io/badge/dynamic/yaml?url=https%3A%2F%2Fraw.githubusercontent.com%2FixxeL-DevOps%2Ffullstack%2Frefs%2Fheads%2Fmain%2Fgitops%2Fmanifests%2Fminio%2Fgenmachine%2Fgenmachine-values.yaml&query=%24.minio.image.tag&style=for-the-badge&logo=minio&logoColor=e0e0e0&label=minio&color=e0e0e0) -![adguard](https://img.shields.io/badge/dynamic/yaml?url=https%3A%2F%2Fraw.githubusercontent.com%2FixxeL-DevOps%2Ffullstack%2Frefs%2Fheads%2Fmain%2Fgitops%2Fmanifests%2Fadguard%2Fgenmachine%2Fgenmachine-values.yaml&query=%24.adguard-home.image.tag&style=for-the-badge&logo=adguard&label=AdGuard&color=%2366B574)
@@ -60,36 +44,98 @@ --- -# Overview - -This is my mono repo for my home infrastructure. It's based loosely on the ideas from [szinn/k8s-homelab](https://github.com/szinn/k8s-homelab) as well as various templates and resources from GitHub and Reddit. - -It follows the concept of **Infrastructure as Code** and [**GitOps**](https://opengitops.dev/), leveraging tools such as ArgoCD, Renovate, and go-task to create an easily bootstrappable and manageable home lab environment, with a strong focus on automation for Day 1/Day 2 operations. - -The motivation behind setting up this home lab was to refactor my original environment, which was primarily based on a Raspberry Pi 4 running Docker Compose. While this setup worked, it lacked scalability, automation, and was not GitOps-friendly. To address these limitations, I decided to migrate to a fully Kubernetes-based infrastructure, leveraging its rich and advanced ecosystem. This transition allows for better workload orchestration, improved automation through GitOps practices, and seamless integration with cloud-native tools, making the entire environment more maintainable, resilient, and future-proof. - -The entire infrastructure is fully virtualized on **Proxmox**, where each server runs as a virtual machine within the Proxmox cluster. This setup provides flexibility, isolation, and ease of management while allowing efficient resource allocation. - -![Schéma Excalidraw](./docs/assets/extract.platform.excalidraw.svg) - -My homelab is built on a self-hosted Kubernetes platform, fully automated and managed using a suite of DevOps tools. GitHub serves as the central repository for code and configurations, with GitHub Actions ensuring reliable and reproducible deployments. Repetitive tasks are orchestrated using Taskfile, while Helm facilitates modular application deployment and management. Renovate automates dependency updates, keeping the infrastructure up to date with minimal manual intervention. This ecosystem enables me to efficiently manage my platform while experimenting with modern infrastructure-as-code and automation practices. - -![Schéma Excalidraw](./docs/assets/extract.capabilities.excalidraw.svg) - -# Kubernetes - -To experiment with different Kubernetes distributions, I use a mix of **k0s** and **Talos**. Each of these distributions offers unique advantages and match different requirements of my environment. - -- A cluster running k0s, intended for a lab environment running on the BeeLink hardware -- Another cluster running Talos, used for a production environment running on the GenMachine hardware. - -The choice of k0s for the lab cluster is due to its suitability for hardware with limited resources. In this case, k0s has been configured with a minimal setup and a low footprint. -On the other hand, Talos is used for the production cluster, allowing me to take advantage of advanced features and capabilities. - -# GitOps - -ArgoCD watches both clusters, leveraging `ApplicationSet` CRDs to centralize management in the main cluster. - -Renovate monitors my entire repository for dependency updates. When updates are found, a PR is automatically created and sometimes merged automatically. Once PRs are merged, ArgoCD applies the changes to my clusters. - -The security aspect of GitOps is managed using **HC Vault** as a secret manager and **External Secrets** for synchronization to prevent pushing sensitive information into Git. +## Overview + +This repository is the single source of truth for a homelab running on two Kubernetes clusters, following **GitOps** and **Infrastructure as Code** principles. Every component — cluster bootstrap, application configuration, secret management, and certificate lifecycle — is declared as code and reconciled automatically. + +The infrastructure evolved from a Raspberry Pi running Docker Compose into a production-grade Kubernetes platform. The two clusters are complementary: **Beelink** (k0s, bare metal) hosts the security and access infrastructure, while **Genmachine** (Talos Linux, Proxmox VMs) runs production workloads and the observability stack. + +## Architecture + +```mermaid +graph TB + subgraph GH["GitHub"] + repo[("fullstack\nrepository")] + renovate["🤖 Renovate\nauto-update PRs"] + ci["⚙️ GitHub Actions\nhelm diff · kubeconform"] + end + + subgraph NET["Home Network"] + user(["👤 User"]) + adguard["🛡️ AdGuard\nDNS · *.fredcorp.com"] + vpn["🔒 WireGuard VPN"] + end + + subgraph BK["🖥️ Beelink — k0s (bare metal)"] + traefik_b["Traefik\nIngress + TLS"] + authentik["Authentik\nSSO · IdP · ForwardAuth"] + vault_b["Vault\nPKI · KV"] + eso_b["ExternalSecrets"] + certmgr_b["cert-manager"] + argocd_b["ArgoCD"] + end + + subgraph GM["🖥️ Genmachine — Talos (3× VM on Proxmox)"] + traefik_g["Traefik\nIngress + TLS"] + vault_g["Vault\nPKI · KV"] + eso_g["ExternalSecrets"] + certmgr_g["cert-manager"] + argocd_g["ArgoCD"] + prometheus["Prometheus · Grafana\nLoki"] + minio["MinIO\nS3 object store"] + end + + user -- "DNS" --> adguard + user -- "VPN" --> vpn + adguard -- "ingress" --> traefik_b & traefik_g + traefik_b -- "ForwardAuth" --> authentik + + renovate -- "auto-PR" --> repo + ci -- "diff preview" --> repo + argocd_b & argocd_g -- "pull · reconcile" --> repo + + vault_b -- "PKI signs" --> certmgr_b + vault_g -- "PKI signs" --> certmgr_g + vault_b -- "KV secrets" --> eso_b + vault_g -- "KV secrets" --> eso_g +``` + +## GitOps Flow + +```mermaid +flowchart LR + dev(["💻 Developer\ngit push"]) + bot(["🤖 Renovate\nauto-PR"]) + repo(["📦 GitHub\nrepository"]) + ci["⚙️ CI\nhelm-rmp diff"] + argocd["🔄 ArgoCD\nbeelink + genmachine"] + clusters["☸️ Clusters\nbeelink · genmachine"] + + dev -->|push / PR| repo + bot -->|dependency PR| repo + repo -->|triggers| ci + repo -->|poll main| argocd + argocd -->|apply manifests| clusters + ci -->|posts diff comment| repo +``` + +## Stack + +| Layer | Tool | Role | +|---|---|---| +| Cluster | Talos Linux + k0s | Immutable OS (Genmachine) · lightweight K8s (Beelink) | +| GitOps | ArgoCD + Renovate | Continuous reconciliation · automated dependency updates | +| Networking | Cilium + Traefik | eBPF CNI · L2 LB announcements · TLS ingress | +| DNS | AdGuard Home | Local resolver · `*.fredcorp.com` split-horizon | +| PKI / Secrets | HashiCorp Vault | CA · KV secrets · SOPS transit encryption | +| Certificates | cert-manager + trust-manager | Automated TLS lifecycle from Vault PKI | +| Secret sync | ExternalSecrets | Vault → Kubernetes Secret synchronisation | +| Auth | Authentik | SSO IdP · OIDC provider · ForwardAuth outpost | +| VPN | WireGuard Portal | Self-hosted VPN management | +| Observability | Prometheus · Grafana · Loki | Metrics · dashboards · logs | +| Storage | MinIO · Proxmox CSI | S3 object store · block volumes | +| Encryption | SOPS | Secrets encrypted at rest in Git | + +## Documentation + +Full documentation is available at the [project docs site](https://ixxel-devops.github.io/fullstack). diff --git a/docs/index.md b/docs/index.md index 8872ec20a..d889fcf4a 100644 --- a/docs/index.md +++ b/docs/index.md @@ -117,8 +117,6 @@ graph TB ci -->|"diff preview"| repo argocd_b & argocd_g -->|"pull reconcile"| repo - vault_b <-->|"transit auto-unseal"| vault_g - style github fill:#21262d,stroke:#6e40c9,color:#e6edf3 style access fill:#161b22,stroke:#3fb950,color:#e6edf3 style beelink fill:#0d2137,stroke:#4d94ff,color:#e6edf3 diff --git a/docs/secrets/vault.md b/docs/secrets/vault.md index 5e2ca7e57..c23686242 100644 --- a/docs/secrets/vault.md +++ b/docs/secrets/vault.md @@ -2,154 +2,34 @@ ## Overview -Two Vault instances are deployed across the two clusters. Each instance serves as both a secrets engine and a PKI certificate authority for its cluster, and as an auto-unseal transit engine for the other cluster. +Two Vault instances are deployed across the two clusters. Each instance serves as both a secrets engine and a PKI certificate authority for its cluster. ```mermaid graph TB subgraph beelink["Beelink — k0s cluster"] vault_b["Vault beelink\nvault.k0s-fullstack.fredcorp.com"] pki_b["PKI Engine\nRoot + Intermediate CA\nfredcorp.com"] - transit_b["Transit Engine\nkey: unseal-genmachine"] + transit_b["Transit Engine\nSOPS encryption"] kv_b["KV Engine\nadmin/ secrets"] end subgraph genmachine["Genmachine — Talos cluster"] vault_g["Vault genmachine\nvault.talos-genmachine.fredcorp.com"] pki_g["PKI Engine\nRoot + Intermediate CA\nfredcorp.com"] - transit_g["Transit Engine\nkey: unseal-beelink"] + transit_g["Transit Engine\nSOPS encryption"] kv_g["KV Engine\nadmin/ secrets"] end - vault_b -->|auto-unseal via transit| transit_g - vault_g -->|auto-unseal via transit| transit_b - pki_b -->|signs certs for| cm_b["cert-manager\nbeelink"] pki_g -->|signs certs for| cm_g["cert-manager\ngenmachine"] - kv_b -->|serves secrets| eso_b["ExternalSecrets\nbeelink"] - kv_g -->|serves secrets| eso_g["ExternalSecrets\ngenmachine"] + kv_b -->|serves secrets to| eso_b["ExternalSecrets\nbeelink"] + kv_g -->|serves secrets to| eso_g["ExternalSecrets\ngenmachine"] + transit_b & transit_g -->|decrypt| sops["SOPS\nencrypted secrets in Git"] style vault_b fill:#1a2a3a,stroke:#FFB81C,color:#FFB81C style vault_g fill:#1a2a3a,stroke:#FFB81C,color:#FFB81C ``` -## Transit Auto-Unseal - -Each Vault is configured to use the **other cluster's Transit engine** to automatically unseal on restart. This removes the need for manual Shamir key entry after a pod restart. - -```mermaid -sequenceDiagram - participant vb as Vault beelink - participant tg as Transit Engine\ngenmachine - participant vg as Vault genmachine - participant tb as Transit Engine\nbeelink - - Note over vb: Pod restarts (sealed) - vb->>tg: Decrypt unseal key\n(key: unseal-beelink) - tg-->>vb: Plaintext unseal key - Note over vb: Vault unsealed - - Note over vg: Pod restarts (sealed) - vg->>tb: Decrypt unseal key\n(key: unseal-genmachine) - tb-->>vg: Plaintext unseal key - Note over vg: Vault unsealed -``` - -### Helm configuration - -The transit seal block is injected into the Vault `standalone.config` HCL via a Kubernetes secret: - -```yaml -vault: - server: - extraSecretEnvironmentVars: - - envName: TRANSIT_UNSEAL_TOKEN - secretName: vault-transit-token - secretKey: token - standalone: - enabled: true - config: |- - ui = true - listener "tcp" { - tls_disable = 1 - address = "[::]:8200" - } - storage "file" { - path = "/vault/data" - } - seal "transit" { - address = "https://vault.talos-genmachine.fredcorp.com" - token = "$TRANSIT_UNSEAL_TOKEN" - key_name = "unseal-beelink" - mount_path = "transit/" - tls_skip_verify = "true" - } -``` - -The `vault-transit-token` secret must be created **before** ArgoCD syncs Vault. Use the Taskfile helpers: - -```bash -# Create the transit token secret on beelink (reads token from genmachine Vault) -task vault:unseal-secret:beelink - -# Create the transit token secret on genmachine (reads token from beelink Vault) -task vault:unseal-secret:genmachine -``` - -### Circular deadlock recovery - -> [!WARNING] -> If both Vaults restart simultaneously, they each try to contact the other's transit engine while both are sealed — causing a crash loop. - -```mermaid -flowchart TD - start["Both Vaults restart simultaneously"] - sealed_b["Vault beelink: sealed\ntrying to contact genmachine transit"] - sealed_g["Vault genmachine: sealed\ntrying to contact beelink transit"] - crash["Both in CrashLoopBackOff\n503 — no available server"] - - start --> sealed_b & sealed_g - sealed_b --> crash - sealed_g --> crash - - crash -->|recover one first| fix["1. Strip seal block from ConfigMap\n2. Delete pod → Shamir unseal\n3. Restore seal block\n4. vault operator unseal -migrate"] - fix -->|other unseals automatically| done["Both Vaults unsealed"] -``` - -Recovery is automated via Taskfile: - -```bash -# Recover beelink first (will allow genmachine to auto-unseal) -task vault:break-deadlock:beelink - -# Or recover genmachine first -task vault:break-deadlock:genmachine -``` - -## Seal Migration (Shamir → Transit) - -When enabling transit auto-unseal on an existing Vault initialized with Shamir keys, a seal migration is required. - -```mermaid -sequenceDiagram - participant ops as Operator - participant vault as Vault pod - participant cm as ConfigMap - - ops->>cm: Add seal "transit" block to config - ops->>vault: Delete pod (restart with new config) - Note over vault: Vault starts in migration mode - ops->>vault: vault operator unseal -migrate - Note over vault: Migration complete — sealed with transit - vault->>vault: Auto-unseal via transit on next restart -``` - -```bash -vault operator unseal -migrate -address=https://vault.k0s-fullstack.fredcorp.com -``` - -> [!NOTE] -> The `-migrate` flag is only accepted once — on the first unseal after the transit seal block is loaded. Subsequent restarts auto-unseal without any manual action. - ## Authentication Methods ### Kubernetes Auth @@ -162,7 +42,7 @@ sequenceDiagram participant k8s as Kubernetes API participant vault as Vault - app->>vault: Login with ServiceAccount JWT\n(auth/-k8s/login) + app->>vault: Login with ServiceAccount JWT
(auth/<cluster>-k8s/login) vault->>k8s: TokenReview — validate JWT k8s-->>vault: Token valid + bound SA info vault-->>app: Vault token (scoped to policy)