fix(addons): grafana-agent env() syntax + bitnamilegacy/kubectl image refs#16
Merged
Merged
Conversation
Two unrelated bugs surfaced when bringing up a fresh cluster against this repo: ─── grafana-agent: sys.env() is not a valid River function ────────── Three call sites in addons/observability/grafana-agent/values.yaml used `sys.env(...)` to read env-var-projected IRSA + AMP config (AMP_REMOTE_WRITE_URL, AWS_REGION, CLUSTER_NAME). Grafana Agent's River stdlib exposes `env(name)`, not `sys.env(name)` — the agent fails at config-load with `component "sys.env" does not exist or is out of scope`, and the DaemonSet pods CrashLoopBackOff before they can ship a single metric. ─── kyverno + velero: bitnami/kubectl hooks are unpullable ────────── Bitnami moved its public Docker Hub images to the `bitnamilegacy/` namespace in 2025; the `bitnami/kubectl` repo no longer publishes new tags, and `bitnami/kubectl:latest` 404s on pull. Both kyverno (cleanupJobs.admissionReports + clusterAdmissionReports) and velero (kubectl helm hook) were still pointing at the old repo, so their hook pods land in ImagePullBackOff and the cleanup never runs. Switched both to `bitnamilegacy/kubectl:1.33.4-debian-12-r0` — a specific tag verified to exist on the legacy registry. Pinning by version (vs `:latest`) is also safer for helm hooks, which run on upgrade and shouldn't pick up surprise image changes. Comments updated to explain WHY this repo exists (Bitnami's public namespace migration) so the next person who hits an unpullable `bitnami/*` image elsewhere knows where to look.
CI Results
All validations passed. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two unrelated bringup bugs that block fresh clusters from going green.
grafana-agent (
addons/observability/grafana-agent/values.yaml)River stdlib exposes
env(name), notsys.env(name). Three call sites (AMP_REMOTE_WRITE_URL,AWS_REGION,CLUSTER_NAME) fixed. Without this, the DaemonSet pods CrashLoopBackOff at config-load withcomponent "sys.env" does not exist or is out of scopeand ship no telemetry.kyverno + velero (
addons/security/kyverno/values.yaml,addons/operations/velero/values.yaml)Bitnami migrated its public Docker Hub images to
bitnamilegacy/in 2025; thebitnami/kubectlrepo no longer publishes new tags and:latest404s on pull. Helm-hook pods landed inImagePullBackOff. Pinned both tobitnamilegacy/kubectl:1.33.4-debian-12-r0— a specific verified tag on the legacy registry (a:latestonbitnamilegacy/kubectlisn't reliably maintained either, so we pin).Comments updated so the next person hitting an unpullable
bitnami/*image knows where to look.Test plan
kubectl -n monitoring get ds/grafana-agent— pods Ready 2/2 after Argo re-syncskubectl -n kyverno get pod kyverno-clean-reports-*— no longer ImagePullBackOff1.33.4-debian-12-r0is acceptable, or bump to a different pinned tag frombitnamilegacy/kubectl's tag list if you prefer a fresher one