You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
OpenBao is the platform's secrets backbone (k8s/bases/infrastructure/controllers/openbao/, consumed by every tenant via External Secrets), but it runs on the setup we bootstrapped rather than the capabilities OpenBao now ships:
Self-managed Shamir unseal is the dominant operational toil. The unseal key lives in a Secret (openbao-unseal) and a vault-config Job re-unseals every pod after a restart. The helm-release.yaml comments record the cost of this design directly: a 20–40 min cold-cluster bootstrap waiting on unseal, and a manual-intervention incident (2026-06-10) where rescheduled pods never auto-unsealed. This is the single most fragile part of the secrets layer.
Single replica (replicas: ${openbao_replicas:=1}); Raft HA is only documented (docs/dr/openbao-raft-ha-migration.md), not the running posture.
Adopting the 2.6.0 secrets capabilities removes the unseal toil, cuts cold-bootstrap time, and lowers the attack surface without regressing tenant secret delivery.
Unseal toil → 0 manual interventions across pod reschedules/reboots (baseline: the 2026-06-10 incident + every rolling node reboot today needing the re-unseal Job).
Cold-bootstrap ready time measurably lower than the current 20–40 min (proxy: time-to-Ready of the openbao StatefulSet on a fresh cluster).
Kubescape posture for the OpenBao workload holds/improves after the distroless move.
Zero tenant-secret regressions — every ExternalSecret/vault-dynamic-secret still resolves (verified live).
Proposed decomposition (ship oldest-first as children; #1 is the flagship, #6 is the enabling prerequisite)
Auto-unseal via the kms plugin — the highest-value slice; retires the Shamir key Secret + vault-config re-unseal loop. Stateful & reversible-with-care: keep the Shamir path as validated fallback, migrate behind a gate, and E2E-verify unseal-after-reschedule on the live cluster before and after (per the verify-it-works rule).
Move to openbao-distroless — image swap + Kubescape re-scan; small and independently shippable.
Adopt the Kubernetes JWT auth provider for workload → OpenBao auth.
Switch to authenticated root generation, retire the deprecated unauthenticated path.
Evaluate namespace sealing / first-class namespace API for per-tenant isolation — gated on whether the tenancy model wants distinct per-tenant seals (decision child; may close as "not now" with a reason).
Prerequisite: bump the OpenBao app/chart to 2.6.0+ and regression-test unseal + vault-backup + the DR runbooks. Startable now (S).
Guardrails
Secrets-layer changes are stateful and safety-critical: sequence them, keep a validated fallback, and verify behaviourally on the live cluster (not just schema/build), before and after each merge.
Keep sensitive operational detail (key material, exact reachability) out of public artifacts — this epic stays at the control-class/public-component level.
Size: L (epic). Relates to #2447 (Kubescape), #2043 (platform roadmap), and the OpenBao DR runbooks under docs/dr/.
Evidence (current state, from this repo)
OpenBao is the platform's secrets backbone (
k8s/bases/infrastructure/controllers/openbao/, consumed by every tenant via External Secrets), but it runs on the setup we bootstrapped rather than the capabilities OpenBao now ships:openbao-unseal) and avault-configJob re-unseals every pod after a restart. Thehelm-release.yamlcomments record the cost of this design directly: a 20–40 min cold-cluster bootstrap waiting on unseal, and a manual-intervention incident (2026-06-10) where rescheduled pods never auto-unsealed. This is the single most fragile part of the secrets layer.replicas: ${openbao_replicas:=1}); Raft HA is only documented (docs/dr/openbao-raft-ha-migration.md), not the running posture.What OpenBao 2.6.0 offers (released 2026-07-14)
Ref: https://openbao.org/community/release-notes/2-6-0/ — the features map onto the gaps above:
kmsplugin type, crash auto-recovery, prebuilt cloud KMS plugins)vault-configre-unseal Jobopenbao-distrolessimage (only OpenBao itself in the image)/sys/generate-root-tokenreplacing the deprecated unauthenticated path)/sys/namespacesAPIHypothesis & success signals
Adopting the 2.6.0 secrets capabilities removes the unseal toil, cuts cold-bootstrap time, and lowers the attack surface without regressing tenant secret delivery.
openbaoStatefulSet on a fresh cluster).ExternalSecret/vault-dynamic-secretstill resolves (verified live).Proposed decomposition (ship oldest-first as children; #1 is the flagship, #6 is the enabling prerequisite)
kmsplugin — the highest-value slice; retires the Shamir key Secret +vault-configre-unseal loop. Stateful & reversible-with-care: keep the Shamir path as validated fallback, migrate behind a gate, and E2E-verify unseal-after-reschedule on the live cluster before and after (per the verify-it-works rule).openbao-distroless— image swap + Kubescape re-scan; small and independently shippable.vault-backup+ the DR runbooks. Startable now (S).Guardrails
Size: L (epic). Relates to #2447 (Kubescape), #2043 (platform roadmap), and the OpenBao DR runbooks under
docs/dr/.