pkg/operator/genevalogging: scrape OTel self-telemetry and alert on silence

[ventifus]

Fixes ARO-28161

What this does

Enables Prometheus scraping of OTel exporter self-telemetry and adds an SRE alert for pods that stop shipping logs.

Metrics endpoint

Adds service.telemetry.metrics to the OTel config, exposing self-telemetry on 0.0.0.0:8888 via a Prometheus pull reader. This uses the readers config introduced in OTel Collector v0.30+; the legacy address field is no longer valid.

A metrics container port (8888) is added to both DaemonSet specs.

Cluster Prometheus integration

• Labels openshift-azure-logging with openshift.io/cluster-monitoring: "true" so the cluster Prometheus evaluates rules from the namespace.
• Creates a Role and RoleBinding granting prometheus-k8s (in openshift-monitoring) read access to services, endpoints, and pods in openshift-azure-logging.
• Creates a PodMonitor targeting both OTel exporter DaemonSets on the metrics port. Relabels __meta_kubernetes_pod_node_name as node on each scraped series.

Alert

Adds PrometheusRule/otel-exporter-alerts with a single critical alert: OTelExporterNoLogsShippedSRE.

kube_pod_info{namespace="openshift-azure-logging",created_by_kind="DaemonSet",pod=~"otel-exporter-.*"}
unless on(pod) (
  (up{namespace="openshift-azure-logging"} == 1)
  * on(pod) group_left()
  (sum by (pod) (rate(otelcol_exporter_sent_log_records{namespace="openshift-azure-logging"}[1h])) > 0)
)

Fires for any OTel exporter pod that is NOT both scrapeable and actively exporting logs. The For: "10m" pending duration suppresses transient firing during pod restarts.

Effective timing by failure mode:

• Pod crashed or unreachable (up != 1): alert fires after 10 minutes.
• Pod running but silent (rate == 0): alert fires after the 1h rate window drains plus the 10-minute For — roughly 1h10m after exporting stops.

The namespace label in the selectors is not present in the raw OTel metric; Prometheus adds it at scrape time via Kubernetes service discovery.

Verified

Manually tested on a dev cluster. Confirmed OTelExporterNoLogsShippedSRE fires when a pod enters CrashLoopBackOff and resolves when the pod recovers.

[Copilot]

Pull request overview

This PR enhances the Geneva logging operator’s OTel exporter deployment so cluster Prometheus can scrape OTel Collector self-telemetry metrics and an SRE alert can detect DaemonSet pods that are down or have stopped exporting logs.

Changes:

• Expose OTel Collector self-telemetry metrics via a Prometheus pull reader on 0.0.0.0:8888 and add a metrics container port.
• Integrate with OpenShift cluster monitoring: label the namespace, add RBAC for prometheus-k8s, and create a PodMonitor to scrape both exporter DaemonSets.
• Add a PrometheusRule containing the OTelExporterNoLogsShippedSRE alert.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File	Description
pkg/operator/controllers/genevalogging/staticfiles/otel-config.yaml.tmpl	Adds OTel Collector self-telemetry metrics configuration with a Prometheus pull reader (host/port 8888).
pkg/operator/controllers/genevalogging/genevalogging.go	Adds namespace monitoring label, RBAC, PodMonitor, PrometheusRule, and exposes the metrics container port.
pkg/operator/controllers/genevalogging/genevalogging_test.go	Extends unit tests to validate the new monitoring resources and metrics port/config expectations.
pkg/operator/controllers/genevalogging/genevalogging_controller.go	Updates controller ownership to include the new monitoring and RBAC resources.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

pkg/operator/controllers/genevalogging/genevalogging.go:188

• The alert is configured with For: "10m", which delays firing in all cases (including up != 1). This contradicts the PR description claim that scrape/down failures fire immediately. If the intent is immediate alerting on down/crashloop while still waiting 1h for silence, remove the For clause (the 1h rate() window already gates the silence case).

								Alert: "OTelExporterNoLogsShippedSRE",
								For:   "10m",
								Expr: intstr.FromString(

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

[ventifus]

/azp run ci

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[jaitaiwan]

Ex-AROer... Was pinged by this repo; got bored so here's some nits 👍 LGTM otherwise

[ventifus]

/azp run ci

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

[github-actions]

Please rebase pull request.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.