A complete monitoring stack with Grafana

[Smana]

That's pretty simple, use and demonstrate the benefits of all Grafana components.

• Alloy
• Beyla
• Tempo
• Pyroscope
• Oncall
• Alert
• Loki

There are a few opened questions though:

• Using VictoriaMetrics instead of Prometheus/Mimir?
• How about defining alerts? the PrometheusRules with the prometheus operator are pretty cool. But we could also make use of vm-alerts? Or study the way it is recommended with the Grafana stack.

[Smana]

Which agent

As grafana alloy supports the same CRD's as Prometheus/VictoriaMetrics operators, it should be possible to only use Alloy. (reference: https://grafana.com/docs/alloy/latest/set-up/migrate/from-operator/). To be confirmed.

[Smana]

Metrics

I decided to use the VictoriaMetrics operator it brings many benefits

• ha and scalable architecture
• better resources consumption
• long time storage
• better webui
• metricql

Alloy or not Alloy?

• Will I be able to use Alloy anyway instead of VMAgent in order to send traces or should I even use it?
• Is it possible to push metrics from Alloy?
• Alloy allows to use a unique agent for logs, metrics, traces (multiple components).
  But what about having dedicated agents: Vector for logs, VMAgent for metrics, Otel for traces? I still need to decide ...

[Smana]

The first blog post is out and I chose VictoriaMetrics for metrics.
Next will be about alerting workflow and tools.

[Smana]

A second blog post on alerting has been published

[Smana]

Currently working on replacing Loki with VictoriaLogs