Seems like Vali and FluentBit aren't being scraped by Prometheus in the Garden cluster. This is most likely due to incorrect labels and naming of the ServiceMonitor and Service resources.
To check, access any garden runtime cluster and determine whether they are scraped.
Since we'll be replacing Vali with VictoriaLogs we don't need to worry about Vali, but we'll have to fix the FluentBit configs.
As a good example, check how the vpa-recommender component has setup its ServiceMonitor and Service resources. Key thing is that the labels and annotations are different based on whether it's in the Garden cluster. This is determined via the IsGardenCluster value param.
Seems like Vali and FluentBit aren't being scraped by Prometheus in the Garden cluster. This is most likely due to incorrect labels and naming of the ServiceMonitor and Service resources.
To check, access any garden runtime cluster and determine whether they are scraped.
Since we'll be replacing Vali with VictoriaLogs we don't need to worry about Vali, but we'll have to fix the FluentBit configs.
As a good example, check how the vpa-recommender component has setup its ServiceMonitor and Service resources. Key thing is that the labels and annotations are different based on whether it's in the Garden cluster. This is determined via the IsGardenCluster value param.