Add opt-in Vec TTL via MetricVec and TTLRegistry by jpinsonneau · Pull Request #1989 · prometheus/client_golang

jpinsonneau · 2026-04-23T09:31:45Z

Summary

Opt-in Vec TTL via *VecOpts.TTL (Counter/Gauge/Histogram/Summary + MetricVecOpts)
Decorator wrappers refresh lastAccessed on cached child mutators
Unified metricMap (no TTLRegistry, no duplicated ttlMetricMap)
Registry.Gather runs CleanupExpired only for TTL-enabled collectors
Documented orphan-after-expiry (same as Delete); no background cleaner

Relates to #1983
Inspired by https://gist.github.com/bwplotka/5c4d8fc904398909d7c4dd10122230f1

Signed-off-by: Julien Pinsonneau jpinsonn@redhat.com
Made-with: Cursor

Introduce NewMetricVecWithTTL and per-child lastAccessed tracking so stale label sets can be omitted from Collect and removed with CleanupExpired. Store vec children as pointers to satisfy atomic copying rules. Keep default NewCounterVec/NewGaugeVec/NewHistogramVec on NewMetricVec without TTL for backwards compatibility. Opt-in TTLRegistry wraps a dedicated Registry, exposes New*Vec constructors with a fixed ttl, and runs CleanupExpired before each Gather. Relates to prometheus#1983 Signed-off-by: Julien Pinsonneau <jpinsonn@redhat.com> Made-with: Cursor

kakkoyun

Thanks for the PR. This is close to the finish line.

Please see my comments.

Move TTL logic to ttlMetricMap so metricMap stays unchanged for default Vecs. Wrap TTL tests with synctest instead of real sleeps. Signed-off-by: Julien Pinsonneau <jpinsonn@redhat.com> Made-with: Cursor

Copilot

Pull request overview

This PR introduces an opt-in TTL mechanism for *Vec metrics to automatically omit and optionally delete stale label sets, aimed at reducing unbounded growth for high-cardinality / ephemeral labels while preserving backwards compatibility for existing New*Vec constructors.

Changes:

Add NewMetricVecWithTTL plus CleanupExpired and per-child lastAccessed tracking to support TTL-based expiration.
Add TTLRegistry, a dedicated registry with New*Vec constructors that enable TTL and run CleanupExpired before each Gather.
Refactor CounterVec, GaugeVec, and HistogramVec constructors to share internal new*VecWithTTL helpers, and add test coverage for TTL behavior.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
prometheus/vec.go	Adds TTL-enabled `MetricVec` construction, routes core vec operations through TTL map when enabled, and exposes `CleanupExpired`.
prometheus/ttl_vec.go	Implements TTL-backed metric storage with per-child `lastAccessed`, TTL-aware `Collect`, and `cleanupExpired`.
prometheus/ttl_registry.go	Introduces `TTLRegistry` that auto-registers TTL vecs and runs cleanup before `Gather`.
prometheus/counter.go	Refactors vec creation into `newCounterVecWithTTL` and wires TTL registry support.
prometheus/gauge.go	Refactors vec creation into `newGaugeVecWithTTL` and wires TTL registry support.
prometheus/histogram.go	Refactors vec creation into `newHistogramVecWithTTL` and wires TTL registry support.
prometheus/vec_test.go	Adds TTL behavior tests for Counter/Gauge/Histogram vecs and helper for counting collected metrics.
prometheus/ttl_registry_test.go	Adds tests ensuring `NewTTLRegistry` validation and that `Gather` triggers cleanup.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Handle ttl<=0 in NewMetricVecWithTTL, nil slice tails in cleanupExpired for GC, and untrack Vecs on TTLRegistry.Unregister. Signed-off-by: Julien Pinsonneau <jpinsonn@redhat.com> Made-with: Cursor

jpinsonneau · 2026-07-15T12:58:20Z

@kakkoyun any update here ?

kakkoyun

LGTM.

I leave the decision to move forward with it to @bwplotka. He is the release shepherd for the next release.

bwplotka

Thanks for this!

I hate to block it but this creates tons of duplicated code that will be hard to maintain. Added comments inline. Especially registry API here is not ideal - registries in client_golang are for registering metrics not creating them.

I tried to play with LLM to play with alternative implementations e.g. https://gist.github.com/bwplotka/5c4d8fc904398909d7c4dd10122230f1 - this is not ideal but perhaps more acceptable.

Notably we could decorate/wrap gauge/counters etc with TTL logic to support a common "cached" metric refs flow. This also does not impact default path which is what @kakkoyun aimed for.

I am also worried about orphaned refs here, common with TTL. While this problem exists with Delete API it's more explicit to users when delete is invoked (user has to do it). Here TTL can kill the metrics any time. LLM gave some ideas to recover but it's a hot path (Inc).

There is also aspect of releasing memory. If someone has TTL to their metrics and expect it to be cleaned, but no-one scrapes then we have leak too. Have we considered time-based TTL cleanup?

bwplotka · 2026-07-16T10:56:25Z

+
+// newCounterVecWithTTL creates a CounterVec. ttl must be >= 0; ttl == 0 disables
+// TTL behavior (identical to NewMetricVec).
+func newCounterVecWithTTL(opts CounterVecOpts, ttl time.Duration) *CounterVec {


Are we sure this is what we want? How this will scale when we will have another option, newCounterVecWithTTLWithX?

If we have opts, why not using it with TTL for this purpose? Would it make sense to add ttl for those cases?

Agreed. Dropped the *WithTTL helpers.

TTL now lives on CounterVecOpts / GaugeVecOpts / HistogramVecOpts / SummaryVecOpts / MetricVecOpts, e.g.:

prometheus.V2.NewCounterVec(prometheus.CounterVecOpts{ CounterOpts: ..., VariableLabels: ..., TTL: time.Minute, })

See 8d33a81 (and SummaryVec in 649400b).

bwplotka · 2026-07-16T11:10:04Z

+// configured TTL. It returns the number of children removed. If TTL is not
+// configured (zero), this is a no-op and returns 0.
+func (m *MetricVec) CleanupExpired() int {
+	if m.ttlMap == nil {


This feels like an anti-pattern. We have a single type, but we workaround it by having a completely different data-structure inside to mask completely different, mostly duplicated code path (ttlMap vs map). 🤔

Agreed. Removed the parallel ttlMetricMap path.

TTL is a field on the existing metricMap; Collect skips expired ttlMetric children and CleanupExpired reclaims them. Default TTL == 0 keeps the previous Vec path (no wrapper, no expiration work).

See 49d10c3 (drop ttl_vec / TTLRegistry) and f366f3e (unified metricMap + MetricVecOpts).

bwplotka · 2026-07-16T11:32:08Z

+//
+// Default prometheus.NewRegistry, NewCounterVec, and Opts are unchanged; use
+// TTLRegistry only when you explicitly opt in to Vec TTL.
+type TTLRegistry struct {


Registry is for registering metrics and then gathering registered ones. This API breaks this by having a registry with tons of New* constructors. Are we sure this will not confuse users?

LLM review also captures this well in this point: https://gist.github.com/bwplotka/694d6c4a9be64e629d5c7b2f3541319b#a-coupling-constructors-to-registry

There's indeed also false impression that MustRegister would suddenly start TTL tracking: https://gist.github.com/bwplotka/694d6c4a9be64e629d5c7b2f3541319b#b-incomplete--error-prone-registration-surface

Agreed. Removed TTLRegistry entirely.

Construction stays on V2.New*Vec(...Opts{TTL: ...}); registration/gathering stay on a normal Registry.

See 49d10c3.

bwplotka · 2026-07-16T11:55:37Z

+			Help: "test",
+		}, []string{"status"})
+
+		vec.WithLabelValues("ok").Observe(0.5)


A common approach is to cache vec.WithLabelValues("xyz") and do cachedVec.Observe(xyz). In the current code we don't

If we want TTL to work correctly, it Observe and Add must update last access time, otherwise this feature won't work for typical cases.

See: https://gist.github.com/bwplotka/694d6c4a9be64e629d5c7b2f3541319b#2-correctness-issue-cached-child-metrics-cachedvecobservexyz

Even worse it creates bugs in this case: https://gist.github.com/bwplotka/694d6c4a9be64e629d5c7b2f3541319b#b-catastrophic-consequence

Although this is already the case DeleteSeries - it must be documented though.

Fixed via decorator wrappers (ttlCounter / ttlGauge / ttlHistogram / ttlSummary): mutating methods refresh lastAccessed.

Added TestTTLCachedChildKeepsAlive (and Summary coverage) for the cached-child hot path.

Active Observe/Add/Set on a cached child keeps the series alive, so TTL no longer deletes a hot cached handle out from under the caller.

Idle/orphaned handles after CleanupExpired still detach (same class of issue as Delete); documented and covered by TestTTLOrphanedCachedHandleAfterCleanup.

See 8d33a81 and 649400b.

bwplotka · 2026-07-16T12:02:57Z

@@ -0,0 +1,165 @@
+// Copyright 2014 The Prometheus Authors


Oops, bad copy paste. The new ttl.go uses the proper copyright header 👍

Registries in client_golang are for Register/Gather, not metric construction. Drop TTLRegistry and the parallel ttl_vec code path so a follow-up can reintroduce TTL via Opts without coupling constructors to a registry. Addresses review feedback on TTLRegistry API and code duplication. Signed-off-by: Julien Pinsonneau <jpinsonn@redhat.com> Made-with: Cursor

Replace NewMetricVecWithTTL / dual ttlMap branching with MetricVecOpts.TTL and a single metricMap path. Collect skips expired ttlMetric children; CleanupExpired reclaims them. Default TTL of 0 keeps the hot path free of expiration work. Addresses review feedback on new*WithTTL helpers and the ttlMap anti-pattern. Signed-off-by: Julien Pinsonneau <jpinsonn@redhat.com> Made-with: Cursor

Wrap Counter/Gauge/Histogram children when *VecOpts.TTL > 0 so Inc, Add, Set, Observe (and exemplar variants) update lastAccessed. Typical cachedObs := vec.WithLabelValues(...); cachedObs.Observe(x) flows keep series alive without re-hashing labels. Also plumb TTL through CounterVecOpts, GaugeVecOpts, and HistogramVecOpts instead of separate *WithTTL constructors. Addresses review feedback on cached child correctness and Opts-based configuration. Signed-off-by: Julien Pinsonneau <jpinsonn@redhat.com> Made-with: Cursor

Call CleanupExpired on collectors that implement ExpiredCleaner before Collect so scrapes reclaim memory. Document that there is no background TTL goroutine: without Gather (or an explicit CleanupExpired), expired children remain allocated. Document orphaned cached handles after TTL the same way Delete already does. Addresses review feedback on no-scrape leaks, Gather-time cleanup, and orphaned refs documentation. Copyright on new ttl.go is 2026. Signed-off-by: Julien Pinsonneau <jpinsonn@redhat.com> Made-with: Cursor

Only run CleanupExpired during Gather when ttlEnabled is set, panic if a custom MetricVec with TTL returns a non-TTL Metric, add SummaryVec TTL wrappers, and document wrapper/allocation trade-offs. Tests cover no-wrapper for TTL==0, orphaned cached handles, custom-metric panic, SummaryVec, and Gather skipping non-TTL collectors. Signed-off-by: Julien Pinsonneau <jpinsonn@redhat.com> Made-with: Cursor

jpinsonneau · 2026-07-21T09:54:36Z

Addressed feedback and refactored the PR. See PR summary. PTAL

Drop year from ttl.go copyright (repo policy for 2026+), use promoted metricMap selectors for staticcheck QF1008, and gofumpt vec_test helpers. Signed-off-by: Julien Pinsonneau <jpinsonn@redhat.com> Made-with: Cursor

jpinsonneau force-pushed the ttl-registry branch from 9ca0d4c to 530bbbe Compare April 23, 2026 09:47

jpinsonneau force-pushed the ttl-registry branch from 530bbbe to 384151c Compare April 23, 2026 09:53

This was referenced Apr 23, 2026

[DRAFT] NETOBSERV-2284 FLP metrics cache optimization (TTL registry) netobserv/flowlogs-pipeline#1247

Draft

Add optional TTL for *Vec children (MetricVec) #1986

Closed

jpinsonneau marked this pull request as ready for review April 23, 2026 10:49

jpinsonneau requested review from bwplotka, kakkoyun and vesari as code owners April 23, 2026 10:49

Merge branch 'main' into ttl-registry

d396d42

kakkoyun reviewed May 19, 2026

View reviewed changes

Comment thread prometheus/ttl_registry_test.go Outdated

Comment thread prometheus/vec_test.go

Comment thread prometheus/vec.go

Separate TTL Vec path and use synctest in TTL tests

cf4b9e9

Move TTL logic to ttlMetricMap so metricMap stays unchanged for default Vecs. Wrap TTL tests with synctest instead of real sleeps. Signed-off-by: Julien Pinsonneau <jpinsonn@redhat.com> Made-with: Cursor

kakkoyun requested review from Copilot and kakkoyun May 19, 2026 11:08

Copilot started reviewing on behalf of kakkoyun May 19, 2026 11:08 View session

Copilot AI reviewed May 19, 2026

View reviewed changes

Comment thread prometheus/vec.go Outdated

Comment thread prometheus/ttl_vec.go Outdated

Comment thread prometheus/ttl_registry.go Outdated

jpinsonneau and others added 3 commits May 19, 2026 14:52

Fix TTL edge cases from review feedback

e8d0bdf

Handle ttl<=0 in NewMetricVecWithTTL, nil slice tails in cleanupExpired for GC, and untrack Vecs on TTLRegistry.Unregister. Signed-off-by: Julien Pinsonneau <jpinsonn@redhat.com> Made-with: Cursor

Merge branch 'main' into ttl-registry

42f3524

Merge branch 'main' into ttl-registry

0ff0ef7

kakkoyun approved these changes Jul 16, 2026

View reviewed changes

bwplotka requested changes Jul 16, 2026

View reviewed changes

jpinsonneau added 5 commits July 21, 2026 11:13

jpinsonneau requested review from bwplotka and kakkoyun July 21, 2026 09:54

Fix lint and license headers for TTL changes

2b5c23b

Drop year from ttl.go copyright (repo policy for 2026+), use promoted metricMap selectors for staticcheck QF1008, and gofumpt vec_test helpers. Signed-off-by: Julien Pinsonneau <jpinsonn@redhat.com> Made-with: Cursor

Uh oh!

Conversation

jpinsonneau commented Apr 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Uh oh!

kakkoyun left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jpinsonneau commented Jul 15, 2026

Uh oh!

kakkoyun left a comment

Choose a reason for hiding this comment

Uh oh!

bwplotka left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jpinsonneau Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jpinsonneau commented Jul 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jpinsonneau commented Apr 23, 2026 •

edited

Loading

jpinsonneau Jul 21, 2026 •

edited

Loading