feat: CEL-based mutating admission webhook for v1alpha2 Silences

[QuentinBisson]

Summary

• Introduces a mutating admission webhook that injects matchers, labels, and annotations into v1alpha2 Silence resources on CREATE/UPDATE, replacing the Kyverno policy in kyverno-policies-observability
• Rules are expressed in CEL: empty condition = always apply (Kyverno parity); non-empty condition gates the rule to matching objects (e.g. object.metadata.namespace == "prod")
• Webhook is disabled by default; enabled via webhook.enabled=true and webhook.mutating.celRules in Helm values

Kyverno parity config

webhook:
  enabled: true
  mutating:
    celRules:
    - name: exclude-heartbeat
      condition: ""
      matchers:
      - name: alertname
        value: Heartbeat
        matchType: "!="
    - name: exclude-all-pipelines
      condition: ""
      matchers:
      - name: all_pipelines
        value: "true"
        matchType: "!="

⚠️ Kyverno incompatibility

The mutating webhook and the Kyverno always-allow-heartbeats-and-all-pipelines-alerts ClusterPolicy cannot coexist on UPDATE operations.

Why: Kyverno's policy uses patchesJson6902 with op: add and path: /spec/matchers/-, which unconditionally appends matchers on every admission request — including UPDATEs where the matchers are already present. This creates duplicate matchers. The validating webhook (PR #661) then correctly rejects the object.

The webhook itself is idempotent (it checks before adding), but Kyverno bypasses that guarantee by appending after the mutating webhook has already run.

Required action before enabling the webhook: suspend or remove the Kyverno policy. In Kyverno v1.17+ this can be done via a never-true precondition:

kubectl patch clusterpolicy always-allow-heartbeats-and-all-pipelines-alerts --type=merge -p '{
  "spec": {
    "rules": [{
      "name": "always-allow-heartbeats-and-all-pipelines-alerts-v1alpha2",
      "preconditions": {"any": [{"key": "false", "operator": "Equals", "value": "true"}]}
    }]
  }
}'

[QuentinBisson]

This was tested with #661 on graveler

[hervenicol]

• Webhook is disabled by default; enabled via webhook.enabled=true and webhook.mutating.celRules in Helm values

The current code seems to use webhook.celRules, which is simpler but not similar to the kyverno example.

[hervenicol]

Required action before enabling the webhook

Could we automate it? Or test it?
If not, it should be clearly explained is the values, so we try to avoid somebody just enables the webhook without doing the kyverno cleanup.

[QuentinBisson]

You're right, it is done here #661 but I would rather we release the 2 together

[QuentinBisson]

I think it might be easier to just delete it via a script on all MCs manually as it is a one time migration

[hervenicol]

I think it might be easier to just delete it via a script on all MCs manually as it is a one time migration

Is there any chance someone out of giantswarm needs to do this migration? Or is it specific to our internal config?

[QuentinBisson]

We are the only one that I know that currently uses kyverno policy for silences but this is tightly coupled with our delivery

[hervenicol]

We are the only one that I know that currently uses kyverno policy for silences

Fine then, we're the only ones impacted. 👍

but this is tightly coupled with our delivery

You're making reference to this message saying we should run a script at release time?

I think it might be easier to just delete it via a script on all MCs manually as it is a one time migration

So, "a script" would be the kubectl patch command that is in the PR's description?

[QuentinBisson]

No this script disables the policy, the safest bet would be to remove the policy from the kyverno-observability-policies and then merge and release the operator, then update the config with the kyverno parity config

[hervenicol]

No this script disables the policy, the safest bet would be to remove the policy from the kyverno-observability-policies and then merge and release the operator, then update the config with the kyverno parity config

I'm not comfortable with having to do "the safest bet"; I'd prefer we don't bet on whatever during upgrades, and rather rely on tested procedures with a fallback/rollback plan as well 😅

Could you write and append a proper deployment/upgrade procedure to the PR description?

[QuentinBisson]

Tested PR #661 (mutating + validating webhooks) on graveler. Works as expected — config is already in place via konfigure, no additional changes needed to deploy.

Upgrade plan:

1. Disable the Kyverno policy before the chart upgrade (patch never-true preconditions on always-allow-heartbeats-and-all-pipelines-alerts, same as what was done for testing)
2. Merge feat: CEL-based mutating admission webhook for v1alpha2 Silences #660 then feat: validating admission webhook for v1alpha2 Silences #661, cut a release, Flux picks up the new chart automatically
3. Verify webhook is injecting matchers on new and updated Silences

Rollback plan:

1. Suspend the silence-operator HelmRelease
2. helm rollback silence-operator -n monitoring — reverts to 0.20.1, no webhook
3. Resume the HelmRelease
4. Re-enable Kyverno: flux reconcile helmrelease kyverno-policies-observability -n giantswarm