Skip to content

fix: [listener] make observedConnection.Close idempotent to prevent active connection metric negative drift#1019

Open
thinker0 wants to merge 1 commit into
trickstercache:mainfrom
thinker0:fix/listener-active-connection-drift
Open

fix: [listener] make observedConnection.Close idempotent to prevent active connection metric negative drift#1019
thinker0 wants to merge 1 commit into
trickstercache:mainfrom
thinker0:fix/listener-active-connection-drift

Conversation

@thinker0
Copy link
Copy Markdown

Description

This pull request solves the negative drift bug on the trickster_proxy_active_connections Prometheus gauge metric.

Root cause:
In pkg/proxy/listener/listener.go, the Close() method on observedConnection was not idempotent. Go's net/http server can invoke Close() multiple times in different execution paths (e.g., Serve goroutine defer, timeout handlers, Hijack). Each invocation decremented the active connections gauge (metrics.ProxyActiveConnections.Dec()) and incremented the closed total, leading to monotonic negative drift over time under normal load.

Solution:
Applied sync.Once inside observedConnection to guarantee that connection close metrics are updated exactly once per connection lifecycle, ensuring gauge consistency.

Also added high-stress unit testing (TestObservedConnectionIdempotentClose in listener_test.go) under go test -race to verify idempotence and prevent regressions.

Type of Change

  • Bug fix
  • New feature
  • Optimization
  • Test coverage
  • Documentation
  • Infrastructure

AI Disclosure

  • This contribution DOES include AI-generated changes, and I have reviewed the relevant contributing guidelines.

@thinker0 thinker0 requested a review from a team as a code owner May 29, 2026 06:24
…ctive connection metric negative drift

Signed-off-by: thinker0 <thinker0@gmail.com>
@thinker0 thinker0 force-pushed the fix/listener-active-connection-drift branch from d96e8e2 to 145c6a8 Compare June 2, 2026 00:30
Comment on lines +75 to +78
o.once.Do(func() {
metrics.ProxyActiveConnections.Dec()
metrics.ProxyConnectionClosed.Inc()
})
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a second close should return an err

Suggested change
o.once.Do(func() {
metrics.ProxyActiveConnections.Dec()
metrics.ProxyConnectionClosed.Inc()
})
if err == nil {
metrics.ProxyActiveConnections.Dec()
metrics.ProxyConnectionClosed.Inc()
}

I think it would be simpler to skip sync.Once here if we add an err check


// Subsequent Closes should also succeed but NOT decrement metric further
for i := 0; i < 5; i++ {
_ = oconn.Close()
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you're not checking for success here, unlike what the comment says

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants