fix: Scraping Grafana when ingressed

[MichaelThamm]

• backport this into track/2

Issue

Grafana's dashboard does not show data because Grafana has ingress (Traefik) and that ingress is signed by an external CA. When Prometheus (or Otelcol) scrapes Grafana, it needs to trust the CA which signed Traefik and these 2 PRs made it possible:

• fix: Prometheus should accept CA certs from root store when scraping prometheus-k8s-operator#752
• feat: receive-ca-cert relation endpoint prometheus-k8s-operator#750

Solution

Add the correct integrations for Prometheus (and Otelcol) to the external CA

Important

This PR is blocked by:

• certificates CA cert used incorrectly opentelemetry-collector-k8s-operator#147
• fix: Docs and Terraform usage after cutting track/2 #151

Checklist

• I have added or updated relevant documentation.
• PR title makes an appropriate release note and follows conventional commits syntax.
• Merge target is the correct branch, and relevant tandem backport PRs opened.

Context

Related:

• Missing Grafana self-mon dashboard #154

Testing Instructions

1. just integration cos
2. just integration cos_lite

To test for COS and COS Lite, deploy the TF modules, but update prometheus to rev278:

COS Lite

module "cos-lite" {
  source        = "git::https://github.com/canonical/observability-stack//terraform/cos-lite?ref=feat/external-ca-prom-rel"
  model_uuid                      = juju_model.cos-lite.uuid
  channel                         = "dev/edge"
  external_certificates_offer_url = "admin/external-ca.certificates"
  external_ca_cert_offer_url      = "admin/external-ca.send-ca-cert"
  prometheus                      = { revision = 278 }
}

COS

module "cos" {
  source = "git::https://github.com/canonical/observability-stack//terraform/cos?ref=feat/external-ca-prom-rel"
  model_uuid                      = juju_model.cos.uuid
  channel                         = "dev/edge"
  internal_tls                    = true
  external_certificates_offer_url = "admin/external-ca.certificates"
  external_ca_cert_offer_url      = "admin/external-ca.send-ca-cert"

  s3_endpoint   = "http://10.1.185.11:8333"  # Seaweed
  s3_secret_key = "placeholder"
  s3_access_key = "placeholder"

  loki_coordinator  = { units = 1 }
  loki_worker       = { backend_units = 1, read_units = 1, write_units = 1 }
  mimir_coordinator = { units = 1 }
  mimir_worker      = { backend_units = 1, read_units = 1, write_units = 1 }
  tempo_coordinator = { units = 1 }
  tempo_worker      = { compactor_units = 1, distributor_units = 1, ingester_units = 1, metrics_generator_units = 1, querier_units = 1, query_frontend_units = 1 }
  ssc               = { channel = "1/stable" }
  traefik           = { channel = "latest/edge" }
}

Important

COS does not have otelcol scraping Grafana. It also does not have a Grafana self-mon dashboard. To test, I added a relation grafana:metrics-endpoint otelcol:metrics-endpoint and then checked that the metrics make it to Grafana:

Upgrade Notes