diff --git a/docs/architecture/concepts/external-key-management.md b/docs/architecture/concepts/external-key-management.md new file mode 100644 index 00000000..33ac8e2d --- /dev/null +++ b/docs/architecture/concepts/external-key-management.md @@ -0,0 +1,43 @@ +--- +title: "External Key Management" +description: "Data-at-rest encryption with external key management systems, enabling separation of duty, rotation, and audit." +weight: 30220 +--- + +Volume encryption protects data at rest by ciphering every block written to a logical volume. To encrypt data, the +cipher itself needs a key. However, the question of *where that key lives and who controls it* is the responsibility of +the key management layer. + +By default, simplyblock manages encryption keys internally. For environments with stricter security policies, such as +regulated environments or any deployment that separates storage and security duties, where the team operating the +storage cluster must not be in possession of the long-lived key material, the key-encryption keys can be offloaded to an +external Key Management Service (KMS). + +Simplyblock supports storing keys in external KMS solutions. Currently supported KMS backends are: +- [HashiCorp Vault](https://www.vaultproject.io/){:target="_blank" rel="noopener"} +- [OpenBao](https://openbao.org/){:target="_blank" rel="noopener"} + +## Two-Layer Key Model + +When an external KMS is configured, simplyblock applies a two-layer key model: + +- **Unseal Keys** are generated once and presented at the time of the KMS setup (for example, HashCorp Vault). + Typically, a certain number of all the unseal keys are required to unseal the KMS (e.g., 3 of 5 keys). These keys + should be stored in separate secure locations. +- **Data Encryption Keys (DEKs)** are generated per volume and used to encrypt the at-rest data blocks of that volume. + These keys are short-lived in cluster memory and never stored in plaintext at rest. The wrapped DEKs are stored inside + the external KMS. +- **Key Encryption Keys (KEKs)** live inside the KMS. The cluster asks the KMS to wrap each DEK on creation and to + unwrap it when the volume is brought online. The KEKs never leave the KMS. + +## Authentication and Trust + +The KMS authenticates simplyblock components using a client certificate issued by the +`simplyblock-certificate-authority-issuer` ClusterIssuer, which the operator creates as part of its mTLS setup. +Because the KMS depends on this CA, [mTLS](../../deployments/kubernetes/security.md#transport-layer-security-mutual-tls-mtls) +must be configured on the control plane before an external KMS can be wired up. + +Operationally, this means the KMS team and the storage team share only the CA bundle and an agreed-upon DNS-name for +the simplyblock client. No static passwords or long-lived tokens must be exchanged. + +For the setup steps, see [Securing the Control Plane: External KMS](../../deployments/kubernetes/security.md#external-key-management-kms). diff --git a/docs/deployments/kubernetes/k8s-control-plane.md b/docs/deployments/kubernetes/k8s-control-plane.md index 772f2eb9..42adb7ef 100644 --- a/docs/deployments/kubernetes/k8s-control-plane.md +++ b/docs/deployments/kubernetes/k8s-control-plane.md @@ -32,17 +32,14 @@ helm upgrade --install simplyblock -n simplyblock simplyblock/spdk-csi \ ``` !!! important "TLS Encryption" - {{experimental}} + {{ experimental }} - Simplyblock has just added support for TLS encryption for all internal communication. At the moment, it's an - experimental feature only available when installed into OpenShift clusters. + All internal control plane traffic can be encrypted with TLS. On OpenShift, the cluster's built-in certificate + manager is used out of the box. Mutual TLS (mTLS), where components additionally authenticate each other with + client certificates, is only works with Cert-Manager. That means that on OpenShift, the Cert-Manager must be + installed to enable mTLS. - It generally is a good idea to install the operator with TLS support enabled on OpenShift to ensure that all - internal communication is encrypted and secure. - - To enable TLS, add the `--set tls.enabled=true` flag to the `helm install` command. - - In the future, the support will be extended to further Kubernetes distributions by enabling Cert-Manager support. + See [Securing the Control Plane](security.md#transport-layer-security-mutual-tls-mtls) for configuration. After installation, verify the operator is running: diff --git a/docs/deployments/kubernetes/k8s-storage-plane.md b/docs/deployments/kubernetes/k8s-storage-plane.md index c76367fc..7ba9fff5 100644 --- a/docs/deployments/kubernetes/k8s-storage-plane.md +++ b/docs/deployments/kubernetes/k8s-storage-plane.md @@ -100,6 +100,12 @@ been registered, it has no storage nodes yet. Those are added in the next step. as NVMe-oF transport security, backup configuration, capacity thresholds, and more, are available at [Cluster Deployment Options](../cluster-deployment-options.md). +!!! tip "External KMS" + If volumes in this cluster should offload their encryption keys to an external KMS, set + `spec.hashicorpVaultSettings.base_url` on the `StorageCluster` now. The setting can also be added later, but + configuring it upfront means encrypted volumes use the external KMS from day one. See + [Securing the Control Plane: External KMS](security.md#external-key-management-kms). + ## Add Storage Nodes Now, Kubernetes worker nodes will be transformed into simplyblock storage nodes. To initiate the process, a diff --git a/docs/deployments/kubernetes/security.md b/docs/deployments/kubernetes/security.md new file mode 100644 index 00000000..a73897b0 --- /dev/null +++ b/docs/deployments/kubernetes/security.md @@ -0,0 +1,225 @@ +--- +title: "Securing the Control Plane" +description: "Configure mTLS for simplyblock control plane communication and offload at-rest encryption keys to an external KMS (HashiCorp Vault or OpenBao)." +weight: 30050 +--- + +This page covers two security features for simplyblock on Kubernetes: transport-layer encryption and mutual +authentication for the control plane (mTLS), and offloading volume encryption keys to an external Key Management +Service (KMS). + +mTLS must be configured before an external KMS can be wired up: the KMS authenticates simplyblock components using a +certificate issued by the operator-managed certificate authority, which is only provisioned when mTLS is active. + +## Transport Layer Security (Mutual TLS / mTLS) + +{{ experimental }} + +Internal control-plane traffic between the control plane, the operator, and the storage-node handlers can be encrypted +with TLS. Additionally, when mutual TLS is enabled, every component must present a valid client certificate, which +means components authenticate each other rather than relying on network position alone. + +!!! important "Mutual TLS on OpenShift" + **mTLS is only supported using the Cert-Manager certificate provider.** + + On OpenShift, the cluster's built-in certificate manager provides one-way TLS (server certificates) but does not + issue the client certificates required for mutual authentication. To enable mTLS on OpenShift, + [Cert-Manager](https://cert-manager.io/){:target="_blank" rel="noopener"} must be installed and the certificate + provider must be switched over. + +### Prerequisites + +- Cert-Manager must be installed in the cluster. +- A `ClusterIssuer` (or namespaced `Issuer`) for Cert-Manager to be able to mint certificates must exist. Most + installations point this at an internal corporate certificate authority (CA) or at the cluster-local self-signed + issuer. Any issuer that simplyblock components trust via the CA is acceptable. + +### Enabling mTLS + +Mutual TLS (mTLS) is configured at Helm install time by setting four values on the operator chart. Either with setting +the `tls` field directly in the values.yaml or via the `--set` flags on the Helm command line. + +```yaml title="Helm values for mTLS" +tls: + enabled: true + mutual_enabled: true + provider: cert-manager + cert-manager: + issuer: my-cluster-issuer +``` + +Apply the values during the operator installation (see [Install Simplyblock Operator](k8s-control-plane.md)): + +```bash title="Install the operator with mTLS" +helm upgrade --install simplyblock -n simplyblock simplyblock/spdk-csi \ + --create-namespace \ + --set operator.enabled=true \ + --set tls.enabled=true \ + --set tls.mutual_enabled=true \ + --set tls.provider=cert-manager \ + --set tls.cert-manager.issuer=my-cluster-issuer +``` + +Replace `my-cluster-issuer` with the name of the `ClusterIssuer` the operator should use to obtain its certificates. + +### What the Operator Provisions + +When mTLS is enabled, the operator creates a dedicated `ClusterIssuer` named +`simplyblock-certificate-authority-issuer` and issues all internal component certificates signed with the configured +certificate authority. The same issuer can be used to mint certificates for other workloads that need to talk to +simplyblock. These workloads specifically include external key management systems (KMS), as described in the next +section. + +!!! note "OpenShift" + On OpenShift, setting `tls.enabled=true` with the default `tls.provider=openshift` only activates one-way TLS using + OpenShift-managed certificates. + + Mutual TLS is **not** available with the OpenShift default provider. To use `tls.mutual_enabled=true` + requires `tls.provider=cert-manager` regardless of the underlying Kubernetes distribution. + +## External Key Management (KMS) + +{{ experimental }} + +By default, simplyblock manages volume encryption keys internally. For environments that require stricter key handling, +the cluster can be configured to keep the key-encryption material in an external KMS. This especially includes +environments with strict separation of duty between storage administrators and key custodians, regular rotation, or +audit trails. + +As of now, [HashiCorp Vault](https://www.vaultproject.io/){:target="_blank" rel="noopener"} and +[OpenBao](https://openbao.org/){:target="_blank" rel="noopener"} are supported. The configuration is identical for +either of them. + +### Prerequisites + +- [mTLS configured](#transport-layer-security-mutual-tls-mtls) is required, because the vault is authenticated to the cluster via + a certificate issued by the operator's `simplyblock-certificate-authority-issuer`. +- A Vault or OpenBao instance reachable from the simplyblock namespace. The instance must be initialized and unsealed + before configuring authentication. + +### Issue a TLS Certificate via Vault + +Create a Cert-Manager `Certificate` resource that uses the operator-managed issuer. The resulting secret holds the +TLS material that Vault serves to clients and is trusted by the simplyblock components because it chains to the same +CA. + +```yaml title="vault-tls.yaml" +apiVersion: cert-manager.io/v1 +kind: Certificate +metadata: + name: vault-tls + namespace: vault +spec: + secretName: vault-tls + issuerRef: + name: simplyblock-certificate-authority-issuer + kind: ClusterIssuer + commonName: vault + dnsNames: + - vault + - vault.vault + - vault.vault.svc + - vault.vault.svc.cluster.local +``` + +Mount the resulting `vault-tls` secret into the Vault deployment as its serving certificate. Mount the issuer's CA +bundle (typically `ca.crt`) at a path the Vault can read. The example below assumes `/vault/tls/ca.crt` for Vault +and `/bao/tls/ca.crt` for OpenBao. + +### Deploy the Vault + +Install Vault or OpenBao using their upstream Helm chart and expose it inside the cluster. For the rest of this guide +the in-cluster service is assumed to be `vault.vault:8200`. Adjust the URL to match the actual deployment. + +### Configure Auth, Policy, and Secret Engines + +Configure the vault with a policy that grants simplyblock access to the `transit` and `kv` backends, enable the +certificate authentication bound to the simplyblock CA, and enable the required secret engines. The script below +works for both Vault (`vault`) and OpenBao (`bao`). Assign the appropriate CLI to the `$CLI` variable. + +```bash title="Configure the vault for simplyblock" +CLI=vault # or: CLI=bao + +# Policy granting access to the transit and kv backends +$CLI policy write webappapi-policy - <` | + +For details, see [Securing the Control Plane](../../deployments/kubernetes/security.md). + ## Storage Node Parameters | Parameter | Description | Default | diff --git a/docs/reference/operator.md b/docs/reference/operator.md index 5ed9e2f1..b9451a6e 100644 --- a/docs/reference/operator.md +++ b/docs/reference/operator.md @@ -78,6 +78,7 @@ spec: | `warningThreshold.provisionedCapacity` | int | Provisioned capacity warning threshold (percent). | | `criticalThreshold.provisionedCapacity` | int | Provisioned capacity critical threshold (percent). | | `action` | string | Lifecycle action: `activate` or `expand`. | +| `hashicorpVaultSettings.base_url` | string | Base URL of an external Hashicorp Vault or Openbao instance used to manage volume encryption keys (e.g., `https://vault.vault:8200/`). See [Securing the Control Plane: External KMS](../deployments/kubernetes/security.md#external-key-management-kms). | | `backup.credentialsSecretRef.name` | string | Name of the Secret (in the same namespace) holding `access_key_id` and `secret_access_key`. **Required when `backup` is set**. | | `backup.localEndpoint` | string | S3-compatible endpoint URL for backup storage. | | `backup.snapshotBackups` | bool | Enable snapshot-based backups. | diff --git a/docs/release-notes/26-2-pre.md b/docs/release-notes/26-2-pre.md index 625d89ae..dd83764f 100644 --- a/docs/release-notes/26-2-pre.md +++ b/docs/release-notes/26-2-pre.md @@ -15,6 +15,8 @@ Simplyblock is happy to release the general availability release of Simplyblock - Storage Plane: Added support for FTT2 (failure to tolerate) which enables up to two failover paths. When enabled, simplyblock creates three communication paths between the initiators and the NVMe targets (primary, secondary, and tertiary) with identical subsystems able to process IO. This allows the loss or maintenance of any two nodes in the cluster, regardless of the combination, at a point in time. - Storage Plane: Added support for full multi-pathing within the storage cluster and between initiators and NVMe targets. This means that clients can have four (FTT=1) or six (FTT=2) connections via separate VLANs using separate networking paths. Each endpoint-combination uses two separate connections through the available network paths. It has advantages over simple bonding and multi-chassis link aggregation groups (MLAG). - Kubernetes: Added support for OpenShift-managed TLS certificates for communication within the control plane and cluster endpoints. +- Kubernetes: Added support for mutual TLS (mTLS) on the control plane, using cert-manager-issued certificates. +- Kubernetes: Added support for offloading volume encryption keys to an external KMS (Hashicorp Vault or OpenBao). - Kubernetes: Added support for namespaced volumes, which auto-create subsystems and namespaces within those subsystems in a pre-defined ratio (e.g., 8 or 16 namespaces per subsystem) in Kubernetes-based environments. The ratio can be configured via the operator. - Kubernetes: Added a new, experimental Kubernetes operator to simplify the deployment of Simplyblock control planes, storage planes, and the CSI driver in Kubernetes-based environments, such as OpenShift and Talos. - General: All container images are now based on RHEL 10 base images and use Python 3.12. This solves multiple open CVEs and additional vulnerabilities.