fix(kyverno): reports/admission leaders hot-loop ~25 no-op reconciles/s, inflating VPA and ratcheting autoscale nodes

[devantler]

🤖 Generated by the Daily AI Engineer

Evidence (measured live 2026-07-16 ~23:1x-23:4xZ)

• In both the kyverno admission and reports deployments, exactly the LEADER replica burns ~1 CPU steadily while its standby idles at 1-50m (admission 1079m vs 50m; reports 929m vs 3m).
• The reports leader's counters: aggregate-report-controller at 23.4 reconciles/s live (1.67M cumulative), background-scan-controller 4.7/s live (2.04M cumulative) — on a quiet cluster: 0 policyreport writes, 0 ephemeralreport events, and 0 kubescape profile events in 12s watch samples; apiserver-side webhook traffic ~0.3/s.
• So the reconciles are no-op recomputation, not real work. Restarting both leaders elected new leaders that resumed the same burn ⇒ structural, not stuck state.
• Consequence: the uncapped VPA sized both pairs for the pathology (1836m/1168m requests ⇒ ~6.2 cores), tipping the fleet past the static workers — the capacity half of the 2026-07-16 snapshot-slot deadlock. Mitigated by fix(vpa): cap kyverno controller CPU below the leader hot-loop level #2667 (CPU caps); this issue is the root-cause hunt.

Expected behaviour

On a quiet cluster the reports controller reconciles at near-zero; kyverno pairs fit comfortably in the static-worker budget without VPA caps doing load-bearing work.

Direction

Identify what feeds the aggregate-report/background-scan workqueues at ~25/s with no object churn (suspects: metadata-informer resync storms from a matched high-churn kind; a requeue-forever item; a kyverno bug worth an upstream issue — check kyverno releases/issues for known aggregate-controller loop fixes and consider a version bump first).

Acceptance criteria

• The feed source of the no-op reconciles is named with evidence.
• Fixed at root cause (config, policy scope, or upstream bump/contribution) — leader CPU at idle <100m sustained.
• The fix(vpa): cap kyverno controller CPU below the leader hot-loop level #2667 caps re-evaluated afterwards (keep as belt-and-braces or relax).

Rough size: M (investigation-heavy).

[coderabbitai]

🔗 Related PRs

#2313 - feat(vpa): enable VPA for kube-system and kyverno workloads (prod) [merged]
#2360 - fix(kyverno): grant reports controller read on KEDA ScaledObjects [merged]
#2370 - fix(autoscaling): break the DaemonSet-VPA node ratchet; keep Longhorn storage off autoscale nodes [merged]
#2378 - fix(policies): split restrict-storage-to-baseline-workers so Kyverno admits it (unfreezes infrastructure) [merged]
#2578 - fix(policy-reporter): run all components at the HA replica floor [merged]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!