RBAC: opt-in ValidatingAdmissionPolicy to scope node-data-broker node writes (audit F1)

[ArangoGutierrez]

Problem

The node-data-broker is a DaemonSet (charts/topograph/charts/node-data-broker/templates/daemonset.yaml:6) — one pod per node, each mounting the same ServiceAccount token. Its ClusterRole grants nodes: [get,list,update] cluster-wide whenever the subchart is enabled (charts/topograph/charts/node-data-broker/templates/rbac.yaml:9-11).

Kubernetes RBAC cannot scope this to "only my own node": resourceNames is a static list and cannot template per-pod, and the NodeRestriction admission plugin constrains only the system:nodes group (kubelet client certs), not ServiceAccount tokens. So any single compromised node — or any workload that obtains a broker pod's token — can update every Node object in the cluster. Because the schedulers that consume topograph's output (KAI, Kueue TAS, Slinky/Slurm) trust the topology data on nodes, a cluster-wide node-write is the largest blast radius in the chart.

A CEL ValidatingAdmissionPolicy (VAP) is the only mechanism that can constrain what the broker's SA is allowed to change on a Node.

Evidence (re-verified against the repo, not the audit report)

The broker's entire node-write surface is annotations under the topograph.nvidia.com/ prefix — nothing else:

• Write path: cmd/node-data-broker/main.go:176 Gets its own node (b.nodeName), :181 merges annotations via mergeNodeAnnotations, :183 Updates. mergeNodeAnnotations (cmd/node-data-broker/main.go:270-275) touches only node.Annotations (maps.Copy) — it never writes labels and never deletes keys.
• Keys, per provider dispatched from getAnnotations (cmd/node-data-broker/main.go:247-268), all defined in pkg/topology/topology.go:43-45:

Key	Constant	Value	Written by
instance	KeyNodeInstance	topograph.nvidia.com/instance	aws imds.go:89, gcp imds.go:89, oci imds.go:122, nebius imds.go:79, dra provider.go:130, lambdai k8s.go:55, infiniband k8s.go:129
region	KeyNodeRegion	topograph.nvidia.com/region	same seven providers (aws imds.go:90 … infiniband k8s.go:130)
cluster-id	KeyGpuClusterID	topograph.nvidia.com/cluster-id	infiniband k8s.go:140 only, and only when !useGpuCliqueLabel

• The --set/extraArgs "extras" are not written to the node. The only provider that receives them, infiniband, uses them purely for config lookup (pkg/providers/infiniband/k8s.go:84-94,146-148) — they never enter the returned annotations map. So a CEL policy restricting changes to the topograph.nvidia.com/* keyspace does not break any legitimate write, including infiniband with custom gpu-operator-namespace / device-plugin-daemonset overrides.

The audit report's F1 (localdev/rbac-audit-2026-07-09/report.md:82-92) reaches the same conclusion; the enumeration above is independently derived from source. Note: the report's shorthand "relabel/annotate" is imprecise for the broker specifically — the broker writes only annotations, so the tightest correct policy also forbids label changes.

Proposed approach

Ship an opt-in, default-OFF VAP + ValidatingAdmissionPolicyBinding in the node-data-broker subchart, gated on a new node-data-broker.admissionPolicy.enabled value and on the VAP API being served by the cluster. The CEL:

1. matchConditions restricts the policy to the broker's SA username only, so kubelet and all other identities are untouched (fail-safe: non-broker node updates never evaluate the constraints).
2. validations require: node labels unchanged; every added/modified annotation key is under topograph.nvidia.com/; no non-topograph annotation removed.
3. A best-effort per-node validation: when the cluster issues node-bound SA tokens (ServiceAccountTokenNodeBinding), the token carries authentication.kubernetes.io/node-name in request.userInfo.extra; the policy then requires the write to target that node. It is a no-op on clusters that don't populate the claim, so it never breaks a legitimate write.

Breaking-change / compatibility note

• Non-breaking, additive, default-OFF. Existing installs are unaffected until an operator sets admissionPolicy.enabled: true.
• Version floor: VAP is GA (admissionregistration.k8s.io/v1) in Kubernetes 1.30; the chart's kubeVersion floor is >=1.27.0-0 (charts/topograph/Chart.yaml:4). The template is guarded on .Capabilities.APIVersions.Has "admissionregistration.k8s.io/v1/ValidatingAdmissionPolicy", so on a 1.27–1.29 cluster the resources are silently skipped and helm install does not fail. (Consequence, mirroring the chart's existing httproute.yaml capability note: offline helm template won't render the VAP unless you pass --api-versions "admissionregistration.k8s.io/v1/ValidatingAdmissionPolicy", because Helm only discovers real API versions at install time.)

Residual risk (honest)

• No true per-node pinning without node-bound tokens. All broker pods share one SA identity; request.userInfo.username is identical across every node's pod. On clusters that do not issue node-bound SA tokens, a compromised broker token can still write topograph.nvidia.com/* annotations to any node — it just can no longer touch labels, other annotations, or (unaffected-identity) anything else. This shrinks the blast radius from "arbitrary write on every node" to "topograph-annotation write on every node," and adds true per-node pinning only where node-bound tokens are in effect.
• Topology annotations remain writable on any node. Because those annotations feed topology generation, a compromised token could still poison a node's instance/region/cluster-id. Containment is real but partial; it is defense-in-depth, not a complete fix.
• Node spec/taints are not constrained by this policy version (only labels + annotations). The broker never touches spec, so a spec-equality validation is a candidate hardening, but it is deferred to avoid any risk of rejecting a legitimate annotation-only update until validated live on a kind cluster.
• failurePolicy: Fail (fail-closed) means a future apiserver CEL-evaluation error on the broker's own updates would block annotation refresh; the matchCondition and validations are written to not error on well-formed requests, but this is the intended security trade-off.

Acceptance criteria

• New template charts/topograph/charts/node-data-broker/templates/validatingadmissionpolicy.yaml, default-OFF, gated on admissionPolicy.enabled and the VAP API capability.
• New node-data-broker.admissionPolicy value block with values.schema.json validation; docs/engines/k8s.md subsection; CHANGELOG.md [Unreleased] entry.
• helm-unittest suite proving: renders (policy+binding) only when enabled and the VAP API is present; skipped when disabled; skipped when the API is absent even if enabled; CEL contains the broker SA username, the topograph.nvidia.com/ keyspace restriction, the label-immutability rule, and validationActions: [Deny]; the SA username follows a custom serviceAccount.name.
• Default-values umbrella render snapshot is unchanged (the policy must not leak into a default render).
• make chart-test passes.

---

---

A staged implementation plan (helm-unittest cases, template edits, docs, CHANGELOG) exists and can follow this issue.

[Shankar-v27]

Hi! I'd like to work on this if it's available. My plan is to implement the opt-in ValidatingAdmissionPolicy following the existing chart patterns, along with the requested Helm tests, documentation, and CHANGELOG updates.