Add opt-in ValidatingAdmissionPolicy for node-data-broker#410
Add opt-in ValidatingAdmissionPolicy for node-data-broker#410Shankar-v27 wants to merge 3 commits into
Conversation
Signed-off-by: Shankar V <shankarvelmurugan2018@gmail.com>
Greptile SummaryThis PR adds an opt-in
Confidence Score: 5/5Safe to merge; the feature is entirely opt-in and the CEL expression logic is correct. All changed code is behind an opt-in flag defaulting to false, so existing deployments are unaffected. The CEL expression correctly guards against the empty-list edge case, and the helm-unittest assertions use the right assertion type. The only open concerns are about hardcoded policy knobs that reduce operator flexibility but do not introduce a functional defect. charts/topograph/templates/nodeDataBroker/validatingadmissionpolicy.yaml — the hardcoded failurePolicy and validationActions are worth revisiting before the feature graduates from opt-in. Important Files Changed
Reviews (3): Last reviewed commit: "fix: address review feedback for node-da..." | Re-trigger Greptile |
Signed-off-by: Shankar V <shankarvelmurugan2018@gmail.com>
ArangoGutierrez
left a comment
There was a problem hiding this comment.
Nice defense-in-depth addition. A few issues keep it from working as intended as written — most importantly the CEL doesn't compile, and the version and feature-gate prerequisites are inaccurate. Details inline.
- This has() call takes an indexed map key, which CEL rejects at compile time (
invalid argument to has() macro— has() only accepts a field selection). The policy fails to compile, so the API server refuses it on apply and the restriction never takes effect. Use'authentication.kubernetes.io/node-name' in request.userInfo.extrafor the presence check. The helm-unittest suite only regex-matches the rendered string, so it doesn't catch this. (charts/topograph/templates/nodeDataBroker/validatingadmissionpolicy.yaml:20) - The template emits
admissionregistration.k8s.io/v1, which is only served at GA in 1.30 (v1alpha1 in 1.26, v1beta1 in 1.28-1.29). With Chart.yaml kubeVersion>=1.27.0-0, enabling this on a 1.27-1.29 cluster fails to apply. Worth stating the real floor (>= 1.30) here and in the docs prerequisites. (charts/topograph/values.yaml:316) - For the broker's default pod-bound token, the
authentication.kubernetes.io/node-nameextra is populated by the ServiceAccountTokenPodNodeInfo feature gate, not ServiceAccountTokenNodeBinding (which mints node-bound tokens the chart doesn't request). Enabling only the named gate can leave the claim absent, and since the policy is fail-closed that denies every broker node update. (docs/engines/k8s.md:319) - failurePolicy: Fail with a matchConstraints that selects every node UPDATE from any user means all node updates (including kubelet heartbeats) run through this CEL; a fault would deny them cluster-wide. Moving the SA check into spec.matchConditions (username == the broker SA) limits both the evaluation and the fail-closed blast radius to just the broker's requests. (charts/topograph/templates/nodeDataBroker/validatingadmissionpolicy.yaml:9)
- These substring matchRegex checks stay green even if the name-binding equality were dropped or inverted, and helm-unittest never compiles the CEL — which is how the invalid has() above passes CI. Consider asserting the full spec.validations[0].expression (the same contains->equal tightening as #385) and validating the policy on a live cluster as #392 did. (charts/topograph/tests/node-data-broker_validatingadmissionpolicy_test.yaml:90)
| - expression: >- | ||
| request.userInfo.username != 'system:serviceaccount:{{ .Release.Namespace }}:{{ include "nodeDataBroker.serviceAccountName" . }}' || | ||
| (has(request.userInfo.extra) && | ||
| has(request.userInfo.extra['authentication.kubernetes.io/node-name']) && |
There was a problem hiding this comment.
This has() call takes an indexed map key, which CEL rejects at compile time (invalid argument to has() macro — has() only accepts a field selection). The policy fails to compile, so the API server refuses it on apply and the restriction never takes effect. Use 'authentication.kubernetes.io/node-name' in request.userInfo.extra for the presence check. The helm-unittest suite only regex-matches the rendered string, so it doesn't catch this.
| create: true | ||
|
|
||
| # Opt-in ValidatingAdmissionPolicy to scope node writes. Requires | ||
| # Kubernetes >= 1.26 (GA in 1.30) and ServiceAccountTokenNodeBinding enabled. |
There was a problem hiding this comment.
The template emits admissionregistration.k8s.io/v1, which is only served at GA in 1.30 (v1alpha1 in 1.26, v1beta1 in 1.28-1.29). With Chart.yaml kubeVersion >=1.27.0-0, enabling this on a 1.27-1.29 cluster fails to apply. Worth stating the real floor (>= 1.30) here and in the docs prerequisites.
|
|
||
| **Prerequisites:** | ||
| 1. A Kubernetes cluster with the **`ValidatingAdmissionPolicy` API enabled** (`admissionregistration.k8s.io/v1`). | ||
| 2. ServiceAccount tokens carrying the node-binding identity claim (e.g., via `ServiceAccountTokenNodeBinding`). The policy denies updates from the broker ServiceAccount if this required node-name claim is missing from the request context. |
There was a problem hiding this comment.
For the broker's default pod-bound token, the authentication.kubernetes.io/node-name extra is populated by the ServiceAccountTokenPodNodeInfo feature gate, not ServiceAccountTokenNodeBinding (which mints node-bound tokens the chart doesn't request). Enabling only the named gate can leave the claim absent, and since the policy is fail-closed that denies every broker node update.
| labels: | ||
| {{- include "nodeDataBroker.labels" . | nindent 4 }} | ||
| spec: | ||
| failurePolicy: Fail |
There was a problem hiding this comment.
failurePolicy: Fail with a matchConstraints that selects every node UPDATE from any user means all node updates (including kubelet heartbeats) run through this CEL; a fault would deny them cluster-wide. Moving the SA check into spec.matchConditions (username == the broker SA) limits both the evaluation and the fail-closed blast radius to just the broker's requests.
| name: custom-node-data-broker-sa | ||
| documentIndex: 0 | ||
| asserts: | ||
| - matchRegex: |
There was a problem hiding this comment.
These substring matchRegex checks stay green even if the name-binding equality were dropped or inverted, and helm-unittest never compiles the CEL — which is how the invalid has() above passes CI. Consider asserting the full spec.validations[0].expression (the same contains->equal tightening as #385) and validating the policy on a live cluster as #392 did.
|
Thanks for the detailed review and for taking the time to go through the implementation so thoroughly. These are all valuable points. I'll go through each of them carefully:
I'll put together a follow-up commit addressing the applicable items after validating them against the Kubernetes API behavior and documentation. |
Signed-off-by: Shankar V <shankarvelmurugan2018@gmail.com>
e9d1959 to
733939a
Compare
Description
Closes #388.
This PR adds an opt-in
ValidatingAdmissionPolicyandValidatingAdmissionPolicyBindingto restrict thenode-data-brokerServiceAccount to updating only its own Node resource.Changes
nodeDataBroker.validatingAdmissionPolicy.enabled(default:false)ValidatingAdmissionPolicyandValidatingAdmissionPolicyBindingtemplatesTesting
Added Helm unit tests covering:
failurePolicyvalidationActions