v1.8.1 Issues with metrics

[NBouchia]

Hi,

We installed version 1.8.1 and we found missing metrics after start cerstream-server-go.
We stopped the agent used by grafana before starting the server and we restart the agent after.
If I understand correctly, the Gauge metric certstreamservergo_certs_by_log_total is initialized only once in WritePrometheus function.

First question :
We run the command : curl http://localhost:8081/metrics and do not see all url.
How it's possible that WritePrometheus function is called with not all url ?

Second question :
The Gauge metrics certstreamservergo_certs_by_log_total are initialized only once (global boolean ctLogMetricsInitialized) and does not get updated when new URLs are added to Google’s log_list.json. Is that an accurate interpretation of the code?

internal/metrics/prometheus.go

// WritePrometheus provides an easy way to write metrics to a writer.
func WritePrometheus(w io.Writer, exposeProcessMetrics bool) {
	ctLogMetricsInitMutex.Lock()
	if !ctLogMetricsInitialized {
		initCtLogMetrics()
	}
	ctLogMetricsInitMutex.Unlock()

	getSkippedCertMetrics()

	metrics.WritePrometheus(w, exposeProcessMetrics)
}

// For having metrics regarding each individual CT log, we need to register them manually.
// initCtLogMetrics fetches all the CT Logs and registers one metric per log.
func initCtLogMetrics() {
	logs := certificatetransparency.GetLogOperators()

	for operator, urls := range logs {
		operator := operator // Copy variable to new scope

		for i := 0; i < len(urls); i++ {
			url := urls[i]
			name := fmt.Sprintf("certstreamservergo_certs_by_log_total{url=\"%s\",operator=\"%s\"}", url, operator)
			metrics.NewGauge(name, func() float64 {
				return float64(getCertCountForLog(operator, url))
			})
		}
	}

	if len(logs) > 0 {
		ctLogMetricsInitialized = true
	}
}

[d-Rickyy-b]

Hi there. Thanks for reporting. This is much rather an issue than a question.

How it's possible that WritePrometheus function is called with not all url ?

I am not certain if I understand your question correctly. But I assume that it can be explained with the answer to your second question.

The Gauge metrics certstreamservergo_certs_by_log_total are initialized only once (global boolean ctLogMetricsInitialized) and does not get updated when new URLs are added to Google’s log_list.json. Is that an accurate interpretation of the code?

You are right. This is a bug. I'll provide a fix for that asap. New logs must be added to the metrics as they are detected. This is probably the reason why you won't find all of the logs in the list.

[NBouchia]

Hi,
Thank you for your response. I didn't create an issue because I wanted to first get your feedback on my understanding.
On the first point, I'm not sure that it's related to the second because we observe it from the start.

What we do:
We start the server from the download at https://github.com/d-Rickyy-b/certstream-server-go/releases/download/v1.8.1/certstream-server-go_1.8.1_linux_amd64 with the config.yaml :

webserver:
  listen_addr: "127.0.0.1"
  listen_port: 8180
  full_url: "/full-stream"
  lite_url: "/"
  domains_only_url: "/domains-only"
  cert_path: ""
  cert_key_path: ""
  compression_enabled: false

prometheus:
  enabled: true
  listen_addr: "127.0.0.1"
  listen_port: 8181
  metrics_url: "/metrics"
  expose_system_metrics: false
  real_ip: false
  whitelist:
    - "127.0.0.1/8"

We can see a worker start for each URL in Google's log_list.json file. We don't add any additional URLs.
We have approximately 35 workers.

When we query /metrics after the server starts, we only see about ten metrics of type certstreamservergo_certs_by_log_total.

It's this behavior that I don't understand.

If we restart the server, we still have about ten metrics of type certstreamservergo_certs_by_log_total, but not necessarily the same ones as at the previous startup!

We do not observe this on version 1.9.0-beta2 and I do not see any changes related to metrics between these two versions!

[d-Rickyy-b]

The problem was, that v1.8.1 contained some bugs in the metrics code. When Additional logs were not specified, the former code just returned the Google loglist. But that early return caused metrics to stay unititialized.

certstream-server-go/internal/certificatetransparency/ct-watcher.go

Lines 395 to 398 in 8d71967

	// Add manually added logs from config to the allLogs list
	if config.AppConfig.General.AdditionalLogs == nil {
	return *allLogs, nil
	}

In the current v1.9.0-beta2, that code is commented out.

certstream-server-go/internal/certificatetransparency/ct-watcher.go

Lines 637 to 640 in f17d7fd

	// Add manually added logs from config to the allLogs list
	// if config.AppConfig.General.AdditionalLogs == nil {
	// return allLogs, nil
	// }

That resulted in the LogMetrics.metrics data structure being empty. It gets only filled when the LogMetrics methods "Set", "Get", or "Inc" get called. That issue was already fixed in 8359b42. The other issue you described - that newly added logs aren't showing in the metrics, is caused by a related issue. Previously, the list of metrics was pretty static. Upon initialization, no new log metrics could get added. This will get fixed in the next beta release.

[NBouchia]

Thank you for your answers.