Skip to content

roadmap: adopt OpenBao 2.6.0 to de-toil and harden the platform's secrets layer #2653

Description

@devantler

🤖 Generated by the Daily AI Engineer

Evidence (current state, from this repo)

OpenBao is the platform's secrets backbone (k8s/bases/infrastructure/controllers/openbao/, consumed by every tenant via External Secrets), but it runs on the setup we bootstrapped rather than the capabilities OpenBao now ships:

  • Self-managed Shamir unseal is the dominant operational toil. The unseal key lives in a Secret (openbao-unseal) and a vault-config Job re-unseals every pod after a restart. The helm-release.yaml comments record the cost of this design directly: a 20–40 min cold-cluster bootstrap waiting on unseal, and a manual-intervention incident (2026-06-10) where rescheduled pods never auto-unsealed. This is the single most fragile part of the secrets layer.
  • Single replica (replicas: ${openbao_replicas:=1}); Raft HA is only documented (docs/dr/openbao-raft-ha-migration.md), not the running posture.
  • Security posture pressure. OpenBao is in scope for the Kubescape 100%-and-hold program (roadmap: Kubescape security stack → 100% and hold (posture · CVE · runtime) #2447); the container attack surface and the root-token generation path are both improvable.
  • Multi-tenant, but no per-tenant cryptographic isolation — every tenant shares one seal/namespace boundary.

What OpenBao 2.6.0 offers (released 2026-07-14)

Ref: https://openbao.org/community/release-notes/2-6-0/ — the features map onto the gaps above:

2.6.0 capability Gap it addresses
Auto Unseal plugins (new kms plugin type, crash auto-recovery, prebuilt cloud KMS plugins) Replaces the brittle self-managed Shamir unseal + vault-config re-unseal Job
openbao-distroless image (only OpenBao itself in the image) Attack-surface reduction → Kubescape posture (#2447)
Enhanced Kubernetes auth (K8s JWT provider using a pod's SA token) Hardens/streamlines how tenant workloads authenticate
Authenticated root generation (/sys/generate-root-token replacing the deprecated unauthenticated path) Security posture of break-glass operations
Namespace sealing + first-class /sys/namespaces API Optional per-tenant crypto isolation + independent revocation

Hypothesis & success signals

Adopting the 2.6.0 secrets capabilities removes the unseal toil, cuts cold-bootstrap time, and lowers the attack surface without regressing tenant secret delivery.

  • Unseal toil → 0 manual interventions across pod reschedules/reboots (baseline: the 2026-06-10 incident + every rolling node reboot today needing the re-unseal Job).
  • Cold-bootstrap ready time measurably lower than the current 20–40 min (proxy: time-to-Ready of the openbao StatefulSet on a fresh cluster).
  • Kubescape posture for the OpenBao workload holds/improves after the distroless move.
  • Zero tenant-secret regressions — every ExternalSecret/vault-dynamic-secret still resolves (verified live).

Proposed decomposition (ship oldest-first as children; #1 is the flagship, #6 is the enabling prerequisite)

  1. Auto-unseal via the kms plugin — the highest-value slice; retires the Shamir key Secret + vault-config re-unseal loop. Stateful & reversible-with-care: keep the Shamir path as validated fallback, migrate behind a gate, and E2E-verify unseal-after-reschedule on the live cluster before and after (per the verify-it-works rule).
  2. Move to openbao-distroless — image swap + Kubescape re-scan; small and independently shippable.
  3. Adopt the Kubernetes JWT auth provider for workload → OpenBao auth.
  4. Switch to authenticated root generation, retire the deprecated unauthenticated path.
  5. Evaluate namespace sealing / first-class namespace API for per-tenant isolation — gated on whether the tenancy model wants distinct per-tenant seals (decision child; may close as "not now" with a reason).
  6. Prerequisite: bump the OpenBao app/chart to 2.6.0+ and regression-test unseal + vault-backup + the DR runbooks. Startable now (S).

Guardrails

  • Secrets-layer changes are stateful and safety-critical: sequence them, keep a validated fallback, and verify behaviourally on the live cluster (not just schema/build), before and after each merge.
  • Keep sensitive operational detail (key material, exact reachability) out of public artifacts — this epic stays at the control-class/public-component level.

Size: L (epic). Relates to #2447 (Kubescape), #2043 (platform roadmap), and the OpenBao DR runbooks under docs/dr/.

Metadata

Metadata

Assignees

No one assigned

    Type

    Fields

    No fields configured for Epic.

    Projects

    Status
    📥 Backlog

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions