test/e2e: fix expired-cert rot (Mode=None self-generated certs) + diagnostics#41
Open
poissoncorp wants to merge 10 commits into
Open
test/e2e: fix expired-cert rot (Mode=None self-generated certs) + diagnostics#41poissoncorp wants to merge 10 commits into
poissoncorp wants to merge 10 commits into
Conversation
The e2e suite has been failing on main since the 2026-07-20 runner-image bump with only "condition never satisfied" and no pod-level detail, which makes the regression undiagnosable from CI logs alone. Replace the require.Eventually poll in WaitCondition with a manual poll that, on timeout, dumps best-effort cluster state while the kind cluster is still alive: the RavenDBCluster status conditions (which reconcile stage stalled), pods across namespaces, events, describe output, and operator logs. No behavior change on the happy path.
The first diagnostics pass showed the RavenDB node pods CrashLoopBackOff (exit 255) but not why. Add current + previous container logs for the node pods to the timeout dump so the crash reason is visible in CI.
…ecrets Root cause of the e2e rot: the e2e cert fixtures (ADMIN_PFX_B64, A..F_PFX_B64 secrets) were fixed, bounded-validity certificates. When they expired, every secured-cluster e2e broke on unchanged code — RavenDB refuses to start because it derives its admin client cert from the (now expired) server cert: "System.ArgumentException: The provided notBefore value is later than the notAfter value". This is time-based, not a code or runner regression. Replace the fixed secrets with hack/gen-e2e-certs.sh, which mints a fresh cluster CA, one server cert per node (A-F, correct SAN/EKU), and an admin client cert on every run. Nodes trust each other via the shared CA embedded in each PFX chain; the admin client cert is trusted at bootstrap by init-cluster.sh (rvn admin-channel trustClientCert), so no pre-registration is needed. Only RAVEN_LICENSE remains a required secret. Locally verified: RavenDB 6.2.11 boots secured on a generated node cert (previously crashed on the expired one). Kills the recurring expiry time-bomb.
The operator decodes the admin client PFX with golang.org/x/crypto/pkcs12 (pkg/upgrade/httpclient.go), which only supports legacy PKCS#12 (3DES + SHA-1 MAC) and rejects OpenSSL 3.x defaults (AES/SHA-256) with 'unknown digest algorithm: 2.16.840.1.101.3.4.2.1'. Force -keypbe/-certpbe PBE-SHA1-3DES and -macalg SHA1. Verified locally: x/crypto/pkcs12.ToPEM decodes the output, and RavenDB 6.2.11 still boots secured on it.
…er certs The generated certs are signed by a self-signed cluster CA. RavenDB trusts cluster peers signed by that shared CA (its setup-package model), but the bootstrapper's curl to /admin/cluster/node verifies the server via --cacert /ravendb/ca-cert/ca.crt, which only exists when caCertSecretRef is set. The previous fixed-secret certs were publicly trusted, so no CA was wired; the self-signed ones need it. - gen-e2e-certs.sh now also writes the CA to E2E_CA_CERT_PATH. - secrets-setup resolves that path; the suite seeds a ravendb-ca-cert secret. - BaseClusterLE/BaseClusterLEDEF set CACertSecretRef so the operator mounts the CA into the bootstrapper (curl --cacert) and the nodes. Also dump the bootstrapper Job logs in DumpClusterDiagnostics.
The runtime cert-generation approach cannot work for this suite: the e2e runs in LetsEncrypt mode, and the operator's admission validator rejects `spec.caCertSecretRef must not be set when mode is LetsEncrypt`. Self-signed certs therefore have no trust path (without a CA the bootstrapper's curl can't verify the server; with a CA the CR is rejected). The original fixtures were real, publicly-trusted Let's Encrypt certs for *.ravendb-operator-e2e.ravendb.run; the correct fix is to regenerate those and rotate the *_PFX_B64 secrets. Reverted: gen-e2e-certs.sh, secrets-setup cert generation, the ravendb-ca-cert seed, and the caCertSecretRef fixtures. Kept: the WaitCondition diagnostics dump (CR conditions, pods, events, node/bootstrap/operator logs), which is what identified the expired-cert root cause and stays useful for future failures.
These were pre-existing untracked files unrelated to the e2e diagnostics work; they should not be part of this branch.
The e2e suite depended on fixed Let's Encrypt cert fixtures (*_PFX_B64 secrets) for *.ravendb-operator-e2e.ravendb.run. Those expired and broke every secured test on unchanged code. LetsEncrypt mode forbids a custom CA (validator), so self-signed certs cannot be substituted there. Convert the suite to Mode=None (bring-your-own-cert), which the operator and cluster-chart already support and which permits a private CA. Certs are now generated fresh each run (hack/gen-e2e-certs.sh): a cluster CA, one cluster server cert with SANs for every node hostname, and an admin client cert, all legacy-PKCS#12 so the operator's golang.org/x/crypto/pkcs12 decoder accepts them. Nodes trust each other via the shared cluster cert; the admin cert is registered at bootstrap; the bootstrapper's curl trusts the server via the CA (caCertSecretRef). Only RAVEN_LICENSE remains a required secret. Trade-off (for review): this drops LetsEncrypt-mode coverage and the per-node cert topology in favour of an expiry-proof, self-contained suite. If you prefer to keep LetsEncrypt-mode coverage, revert this and rotate the real cert secrets instead. - hack/gen-e2e-certs.sh: generate CA + cluster cert + admin client (legacy PFX). - secrets-setup: require only the license; generate certs. - fixtures/cluster_chart/secrets: Mode=None, clusterCertSecretRef + caCertSecretRef. - cluster-chart provision path uses secrets.clusterCert/caCert; C1 asserts those. - Keep the WaitCondition diagnostics dump.
Two follow-ups after the Mode=None conversion (Kustomize e2e now passes): - The "Helm (cluster chart)" CI job templates the chart in LetsEncrypt mode via --set-file secrets.nodeCerts.*, which needs per-node cert files. Restore per-node cert generation (and their E2E_NODE_*_PFX_PATH env) so that job (and the E2E Helm jobs that depend on it) run again. The files are for templating only; the Mode=None clusters use the single cluster cert. - The upgrade tests curl RavenDB's API directly with the client cert but no --cacert; that worked with the old publicly-trusted certs but not the self-signed CA. Extract the CA from the server PFX (ExtractServerCertToTmp) and pass --cacert in CreateDatabaseRF3 and WaitDBDegradedOnNodes.
Two follow-ups after the Helm/cluster-chart e2e went green under Mode=None: - TestBootstrap_B1: the cluster Phase transitions to Running in a reconcile shortly after BootstrapCompleted flips True, so asserting Phase==Running instantly is racy (Phase was still "Deploying"). Poll for Running instead. - TestUpgrade pre_cluster_conn_fail sabotaged node B via the per-node ravendb-certs-b secret, which does not exist in Mode=None. Patch the shared ravendb-cert secret instead; only ravendb-b-0 is restarted afterward, so only B reloads the bogus cert (A and C keep the good cert in memory).
poissoncorp
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this fixes
The e2e suite broke on unchanged
mainbecause the fixed Let's Encrypt cert fixtures (*_PFX_B64secrets) for*.ravendb-operator-e2e.ravendb.runexpired. RavenDB then refuses to start (ArgumentException: notBefore later than notAfterwhen deriving the admin client cert from the expired server cert). LetsEncrypt mode forbids a custom CA, so self-signed certs can't be substituted there.Approach
Convert the e2e suite to Mode=None (bring-your-own-cert) and generate fresh certs on every run, so it no longer depends on external cert validity:
hack/gen-e2e-certs.sh: mint a cluster CA, one cluster server cert (SANs for every node), an admin client cert, and per-node certs (for the chart's LE templating job) — all legacy-PKCS#12 so the operator'sgolang.org/x/crypto/pkcs12decoder accepts them.secrets-setup: onlyRAVEN_LICENSEis a required secret now; certs are generated.clusterCertSecretRef+caCertSecretRef; secret names aligned to chart defaults.trustClientCert); nodes trust each other via the shared CA; upgrade-test curls use--cacert.WaitConditionnow dumps CR conditions, pods, events, and node/bootstrap/operator logs on timeout (this is what pinpointed the expired-cert root cause).Result
All CI green: unit, both Helm charts, E2E Helm (nginx C1–C6), E2E Helm (Traefik), E2E Kustomize, E2E RU (upgrade).
Trade-off (please review)
This drops LetsEncrypt-mode e2e coverage and the per-node cert topology in favour of an expiry-proof, self-contained suite. The alternative is to keep LetsEncrypt mode and instead rotate the real cert secrets (regenerate LE certs for the domain, update
*_PFX_B64). If you prefer that, revert this and rotate; the diagnostics commit is worth keeping either way.Known minor flake: C3 (Traefik) can hit a bootstrap-vs-Traefik-routing race (soft reachability check doesn't gate on ingress readiness); it passed on re-run.