feat: add built-in Prometheus metrics for HTTP and LLM observability#3439
feat: add built-in Prometheus metrics for HTTP and LLM observability#3439RichardoMrMu wants to merge 3 commits intoQuantumNous:mainfrom
Conversation
- Register PrometheusMiddleware in main.go for HTTP golden metrics - Add /metrics endpoint in router for Prometheus scraping - Call RecordAIMetrics in PostTextConsumeQuota (text_quota.go) and PostWssConsumeQuota (quota.go) for LLM metrics collection HTTP metrics (middleware/metrics.go - already present): - newapi_http_requests_total (Counter) - newapi_http_request_duration_seconds (Histogram) - newapi_http_requests_in_flight (Gauge) - newapi_http_response_size_bytes (Histogram) LLM metrics (metrics/ai_metrics.go - already present): - newapi_llm_input_token_total (Counter) - newapi_llm_output_token_total (Counter) - newapi_llm_request_total (Counter) - newapi_llm_service_duration_seconds (Histogram) - newapi_llm_first_token_duration_seconds (Histogram/TTFT) Closes QuantumNous#2402
WalkthroughAdds Prometheus monitoring: a Prometheus middleware is injected into the Gin middleware chain, a Changes
Sequence DiagramsequenceDiagram
autonumber
actor Client
participant Gin as "Gin\nServer"
participant PromM as "Prometheus\nMiddleware"
participant Handler as "Request\nHandler"
participant Quota as "Quota\nService"
participant Billing as "Task\nBilling"
participant Metrics as "Metrics\nRecorder"
participant Scrape as "/metrics\nEndpoint"
Client->>Gin: HTTP request
Gin->>PromM: pass through middleware
PromM->>PromM: record HTTP metrics
PromM->>Handler: next()
Handler->>Quota: consume / record consume log
Quota->>Metrics: RecordAIMetrics(usage)
Handler->>Billing: log task consumption
Billing->>Metrics: RecordAIMetrics(nil)
Handler-->>Client: response
Client->>Scrape: GET /metrics (scrape)
Scrape->>Metrics: promhttp handler serves metrics
Metrics-->>Scrape: metrics payload
Scrape-->>Client: Prometheus format
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
service/quota.go (1)
266-348:⚠️ Potential issue | 🟡 Minor
PostAudioConsumeQuotais missing metrics recording.
PostTextConsumeQuotaandPostWssConsumeQuotanow record AI metrics after quota consumption, butPostAudioConsumeQuotadoes not. This creates inconsistent observability coverage for audio-based LLM requests.Consider adding the same
metrics.RecordAIMetrics(relayInfo, usage)call aftermodel.RecordConsumeLog(around line 347) for completeness.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@service/quota.go` around lines 266 - 348, PostAudioConsumeQuota does not record AI metrics like the text and WSS paths do; after the call to model.RecordConsumeLog in PostAudioConsumeQuota add a call to metrics.RecordAIMetrics(relayInfo, usage) (or the same metrics function used by PostTextConsumeQuota/PostWssConsumeQuota) to capture metrics for audio requests, and ensure the metrics package is imported and any required context or parameters match the other implementations so compilation and telemetry behavior are consistent.
🧹 Nitpick comments (2)
service/quota.go (1)
238-242: Inconsistentdto.Usageconstruction compared toPostTextConsumeQuota.
PostTextConsumeQuotapasses the full upstreamusageobject tometrics.RecordAIMetrics, while this function constructs a minimaldto.Usagewith only three fields. IfRecordAIMetricsuses other fields likeInputTokens,OutputTokens, or token detail fields, the WebSocket path will record incomplete/zeroed metrics.Consider using
usage.InputTokensandusage.OutputTokensdirectly for consistency:♻️ Proposed fix for consistent usage construction
metrics.RecordAIMetrics(relayInfo, &dto.Usage{ - PromptTokens: usage.InputTokens, - CompletionTokens: usage.OutputTokens, - TotalTokens: usage.TotalTokens, + PromptTokens: usage.InputTokens, + CompletionTokens: usage.OutputTokens, + TotalTokens: usage.TotalTokens, + InputTokens: usage.InputTokens, + OutputTokens: usage.OutputTokens, })🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@service/quota.go` around lines 238 - 242, The WebSocket path constructs a minimal dto.Usage (setting only PromptTokens, CompletionTokens, TotalTokens) before calling metrics.RecordAIMetrics, causing missing fields compared to the PostTextConsumeQuota path; instead pass the full upstream usage object (the existing usage variable) to metrics.RecordAIMetrics (use the same pattern as PostTextConsumeQuota) or populate all fields from usage (e.g., InputTokens, OutputTokens, TokenDetails) so RecordAIMetrics receives consistent, complete data; update the call that currently uses dto.Usage {...} to use usage directly or fully mirror usage's fields.router/main.go (1)
17-17: Consider access control for the metrics endpoint.The
/metricsendpoint is publicly accessible without authentication. While this is common practice for Prometheus scraping (and scrape configs typically use network-level controls), exposing metrics can reveal operational details (request volumes, model usage patterns, channel IDs) that may be sensitive in some deployments.Consider adding an optional configuration flag to:
- Disable the endpoint entirely in production if not needed
- Require basic auth or a bearer token for scraping
- Bind to a separate internal port
This is not blocking if your deployment model assumes network-level protection.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@router/main.go` at line 17, The /metrics route is currently registered unconditionally via router.GET("/metrics", middleware.MetricsHandler()), so add configuration-driven access control: introduce a config flag (e.g., EnableMetrics or MetricsAuthToken) and only register the route when enabled, or wrap MetricsHandler() with an auth middleware that validates Basic auth or a bearer token (or bind metrics to a separate internal listener if your app supports that); update the router setup to check the new config and either skip registering router.GET("/metrics", ...) or replace the handler with an authenticated version that checks the configured token before calling middleware.MetricsHandler().
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@service/text_quota.go`:
- Around line 428-429: LogTaskConsumption in service/task_billing.go is missing
a call to metrics.RecordAIMetrics, causing task-based requests to skip token
metrics; update LogTaskConsumption to call metrics.RecordAIMetrics(relayInfo,
usage) after computing the usage object (the same way
PostTextConsumeQuota/PostWssConsumeQuota do), ensuring you pass the same
relayInfo and the usage struct calculated in LogTaskConsumption so
task/image/audio/video requests are recorded for AI token metrics.
- Around line 428-429: The call to metrics.RecordAIMetrics in
service/text_quota.go references a non-existent function and package; either
implement the function in the metrics package with the appropriate signature
(e.g., RecordAIMetrics(relayInfo, usage) exported) and restore the
"github.com/QuantumNous/new-api/metrics" import, or remove the call and the
unused metrics import from service/text_quota.go so the code compiles; update
any callers or tests that expect AI metrics accordingly.
---
Outside diff comments:
In `@service/quota.go`:
- Around line 266-348: PostAudioConsumeQuota does not record AI metrics like the
text and WSS paths do; after the call to model.RecordConsumeLog in
PostAudioConsumeQuota add a call to metrics.RecordAIMetrics(relayInfo, usage)
(or the same metrics function used by PostTextConsumeQuota/PostWssConsumeQuota)
to capture metrics for audio requests, and ensure the metrics package is
imported and any required context or parameters match the other implementations
so compilation and telemetry behavior are consistent.
---
Nitpick comments:
In `@router/main.go`:
- Line 17: The /metrics route is currently registered unconditionally via
router.GET("/metrics", middleware.MetricsHandler()), so add configuration-driven
access control: introduce a config flag (e.g., EnableMetrics or
MetricsAuthToken) and only register the route when enabled, or wrap
MetricsHandler() with an auth middleware that validates Basic auth or a bearer
token (or bind metrics to a separate internal listener if your app supports
that); update the router setup to check the new config and either skip
registering router.GET("/metrics", ...) or replace the handler with an
authenticated version that checks the configured token before calling
middleware.MetricsHandler().
In `@service/quota.go`:
- Around line 238-242: The WebSocket path constructs a minimal dto.Usage
(setting only PromptTokens, CompletionTokens, TotalTokens) before calling
metrics.RecordAIMetrics, causing missing fields compared to the
PostTextConsumeQuota path; instead pass the full upstream usage object (the
existing usage variable) to metrics.RecordAIMetrics (use the same pattern as
PostTextConsumeQuota) or populate all fields from usage (e.g., InputTokens,
OutputTokens, TokenDetails) so RecordAIMetrics receives consistent, complete
data; update the call that currently uses dto.Usage {...} to use usage directly
or fully mirror usage's fields.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 1859a035-a3a5-467e-9ab9-8596b692f610
📒 Files selected for processing (4)
main.gorouter/main.goservice/quota.goservice/text_quota.go
|
|
||
| metrics.RecordAIMetrics(relayInfo, usage) |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Check all quota consumption functions to verify metrics coverage
rg -n 'RecordConsumeLog' --type go -A 3 | head -80
# Check if LogTaskConsumption has metrics call
rg -n -B 5 -A 30 'func LogTaskConsumption' --type goRepository: QuantumNous/new-api
Length of output: 5172
🏁 Script executed:
# Search for all RecordAIMetrics calls
rg -n 'RecordAIMetrics' --type go -B 2 -A 2
# Check PostTextConsumeQuota and PostWssConsumeQuota
rg -n -B 5 -A 35 'func PostTextConsumeQuota|func PostWssConsumeQuota' --type goRepository: QuantumNous/new-api
Length of output: 6729
🏁 Script executed:
# Get the complete PostWssConsumeQuota function (from line 158 onwards)
sed -n '158,250p' service/quota.go
# Get complete context around RecordAIMetrics at quota.go:238
sed -n '220,250p' service/quota.go
# Get full LogTaskConsumption function
sed -n '19,68p' service/task_billing.goRepository: QuantumNous/new-api
Length of output: 6705
Add metrics recording to LogTaskConsumption for observability parity.
PostTextConsumeQuota and PostWssConsumeQuota both call metrics.RecordAIMetrics, but LogTaskConsumption in service/task_billing.go (lines 19-68) does not. Task-based requests (image generation, audio processing, video tasks) will lack token metrics recording, creating an observability gap for monitoring LLM usage.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@service/text_quota.go` around lines 428 - 429, LogTaskConsumption in
service/task_billing.go is missing a call to metrics.RecordAIMetrics, causing
task-based requests to skip token metrics; update LogTaskConsumption to call
metrics.RecordAIMetrics(relayInfo, usage) after computing the usage object (the
same way PostTextConsumeQuota/PostWssConsumeQuota do), ensuring you pass the
same relayInfo and the usage struct calculated in LogTaskConsumption so
task/image/audio/video requests are recorded for AI token metrics.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Check if metrics.RecordAIMetrics handles nil usage
ast-grep --pattern $'func RecordAIMetrics($_, $_) {
$$$
}'
# Also search for nil checks in the function
rg -n -A 20 'func RecordAIMetrics' --type goRepository: QuantumNous/new-api
Length of output: 45
🏁 Script executed:
#!/bin/bash
# Search for RecordAIMetrics in all Go files
rg -l 'RecordAIMetrics' --type go
# Also search broadly for where metrics package defines functions
rg 'func.*RecordAIMetrics' --type go -A 5
# Find the metrics package location
fd -type f -name 'metrics*.go' | head -20Repository: QuantumNous/new-api
Length of output: 289
🏁 Script executed:
#!/bin/bash
# Read the relevant section of service/text_quota.go
wc -l service/text_quota.goRepository: QuantumNous/new-api
Length of output: 88
🏁 Script executed:
#!/bin/bash
# Read the relevant section of service/text_quota.go around line 428-429
sed -n '410,430p' service/text_quota.go
# Also look for imports and earlier context
sed -n '1,50p' service/text_quota.goRepository: QuantumNous/new-api
Length of output: 2240
🏁 Script executed:
#!/bin/bash
# Search for RecordAIMetrics definition across all files
rg 'RecordAIMetrics' -A 15 --max-count=5Repository: QuantumNous/new-api
Length of output: 986
🏁 Script executed:
#!/bin/bash
# Find the metrics package
fd -type f -path '*/metrics/*' -name '*.go' | head -20Repository: QuantumNous/new-api
Length of output: 233
🏁 Script executed:
#!/bin/bash
# Search for RecordAIMetrics definition in metrics package
find . -path '*/metrics*' -name '*.go' -exec grep -l 'RecordAIMetrics' {} \;Repository: QuantumNous/new-api
Length of output: 45
🏁 Script executed:
#!/bin/bash
# Read context before line 428 to understand usage variable
sed -n '280,310p' service/text_quota.goRepository: QuantumNous/new-api
Length of output: 1708
🏁 Script executed:
#!/bin/bash
# Find metrics package files
find . -type d -name 'metrics' | head -10Repository: QuantumNous/new-api
Length of output: 45
🏁 Script executed:
#!/bin/bash
# Locate all files and search for RecordAIMetrics
find . -type f -name '*.go' | xargs grep -l 'func RecordAIMetrics' 2>/dev/nullRepository: QuantumNous/new-api
Length of output: 47
🏁 Script executed:
#!/bin/bash
# Search more broadly for RecordAIMetrics
rg 'RecordAIMetrics' --max-count=20Repository: QuantumNous/new-api
Length of output: 192
🏁 Script executed:
#!/bin/bash
# Check if there's a metrics directory or file
ls -la | grep -i metric
find . -type d -iname '*metric*' 2>/dev/null
find . -type f -iname '*metric*' -name '*.go' 2>/dev/null | head -20Repository: QuantumNous/new-api
Length of output: 45
🏁 Script executed:
#!/bin/bash
# Check what's being imported from metrics in the files
grep -r 'from.*metrics' service/*.go 2>/dev/null || grep 'metrics' service/text_quota.go | head -5
# Look for all files that import metrics
rg 'github.com/QuantumNous/new-api/metrics' --max-count=20Repository: QuantumNous/new-api
Length of output: 270
🏁 Script executed:
#!/bin/bash
# Look for any file that might contain metrics functions
find . -type f -name '*.go' -exec grep -l 'func.*Record' {} \; | head -20Repository: QuantumNous/new-api
Length of output: 270
🏁 Script executed:
#!/bin/bash
# Check all imports in service/text_quota.go
sed -n '1,30p' service/text_quota.go | grep -E '(import|metrics)'Repository: QuantumNous/new-api
Length of output: 113
🏁 Script executed:
#!/bin/bash
# List all Go files to get a sense of the repository structure
git ls-files | grep '\.go$' | grep -i metricRepository: QuantumNous/new-api
Length of output: 45
🏁 Script executed:
#!/bin/bash
# Check entire go file list
git ls-files | grep '\.go$' | wc -l
# Look for any metrics-related files
git ls-files | grep -E '(metric|telemetry|observ)'Repository: QuantumNous/new-api
Length of output: 66
🏁 Script executed:
#!/bin/bash
# Search for the actual definition of RecordAIMetrics with broader search
git grep 'RecordAIMetrics' 2>/dev/null || rg 'RecordAIMetrics' --type-listRepository: QuantumNous/new-api
Length of output: 192
🏁 Script executed:
#!/bin/bash
# Check if metrics package directory exists anywhere
find . -type d -name 'metrics' 2>/dev/nullRepository: QuantumNous/new-api
Length of output: 45
🏁 Script executed:
#!/bin/bash
# Search for files containing 'metrics' in their path
git ls-files 2>/dev/null | grep metrics || echo "No metrics files found"Repository: QuantumNous/new-api
Length of output: 85
🏁 Script executed:
#!/bin/bash
# Look at the go.mod to see if metrics is external
head -50 go.mod 2>/dev/null || echo "No go.mod found"Repository: QuantumNous/new-api
Length of output: 1873
🏁 Script executed:
#!/bin/bash
# Try broader pattern - look for any file that has the word "RecordAIMetrics" or similar patterns
rg 'Record.*Metrics|RecordMetrics' --max-count=20Repository: QuantumNous/new-api
Length of output: 192
🏁 Script executed:
#!/bin/bash
# Search for any definition of metrics package or RecordAIMetrics
rg 'package metrics' --max-count=10
rg 'var.*Metrics' --max-count=10Repository: QuantumNous/new-api
Length of output: 45
🏁 Script executed:
#!/bin/bash
# Check if there's a metrics folder at repo root level
ls -la | head -20
tree -L 2 -d 2>/dev/null | head -30 || find . -maxdepth 2 -type d | grep -v '^\.$' | sort | head -20Repository: QuantumNous/new-api
Length of output: 1631
🏁 Script executed:
#!/bin/bash
# Search for where usage variable comes from and what type it is
sed -n '290,430p' service/text_quota.go | head -60Repository: QuantumNous/new-api
Length of output: 3821
🏁 Script executed:
#!/bin/bash
# Final confirmation: check if metrics package could be created in this PR
# Search for 'metrics' in any files to see if it's being added
rg 'metrics' --type-list 2>/dev/null | head -5
# Look at PR changes more carefully - is metrics being created?
git ls-files | grep metrics || echo "No metrics files in git"
# Check if maybe the metrics package exists but is empty
find . -name "*.go" -path "*/metrics/*" 2>/dev/null || echo "No metrics go files found"Repository: QuantumNous/new-api
Length of output: 86
Remove call to undefined metrics.RecordAIMetrics function.
The metrics.RecordAIMetrics function does not exist in the repository. The import "github.com/QuantumNous/new-api/metrics" references a non-existent package, and the call at line 428 will fail to compile. Either the metrics package needs to be implemented with this function, or this call should be removed.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@service/text_quota.go` around lines 428 - 429, The call to
metrics.RecordAIMetrics in service/text_quota.go references a non-existent
function and package; either implement the function in the metrics package with
the appropriate signature (e.g., RecordAIMetrics(relayInfo, usage) exported) and
restore the "github.com/QuantumNous/new-api/metrics" import, or remove the call
and the unused metrics import from service/text_quota.go so the code compiles;
update any callers or tests that expect AI metrics accordingly.
Address CodeRabbit review feedback: LogTaskConsumption (image generation, audio processing, video tasks) was missing metrics.RecordAIMetrics call, creating an observability gap for task-based requests. The call passes nil usage since task-based billing is per-invocation rather than per-token, but still records request count and duration.
|
Thanks for the review @coderabbitai! Both issues have been addressed: 🟡 LogTaskConsumption missing metrics — Fixed in cc0a5ba. Added 🔴 "Undefined metrics.RecordAIMetrics" — This is a false positive. The function is defined at |
|
Tip For best results, initiate chat on the files or code changes. 🧩 Analysis chain🏁 Script executed: Length of output: 404 🏁 Script executed: Length of output: 68
However, I need to flag a real build-breaking issue regarding the second point. After searching the entire PR branch:
This means the codebase will fail to compile as-is. The PR description mentions Could you double-check with |
These files were present locally but never committed to the branch, causing compilation failures. This commit adds: - metrics/ai_metrics.go: LLM metrics definitions and RecordAIMetrics() - middleware/metrics.go: HTTP Prometheus middleware and MetricsHandler() Both files are required by the integration points in main.go, router/main.go, service/quota.go, service/text_quota.go, and service/task_billing.go.
|
You are absolutely right — apologies for the confusion! The This has now been fixed in commit The branch should now compile correctly with all metric definitions in place. Thank you for catching this critical issue! 🙏 |
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
middleware/metrics.go (1)
94-99: Consider protecting/metricsin production deployments.Serving raw Prometheus metrics directly is fine for internal networks, but internet-exposed deployments should gate this endpoint (allowlist/auth/reverse-proxy ACL).
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@middleware/metrics.go` around lines 94 - 99, The MetricsHandler currently exposes promhttp.Handler() unconditionally; update MetricsHandler to gate access in production by checking a configured policy before calling handler.ServeHTTP — e.g., read an allowlist or auth flag from configuration (env var or app config), verify client IP via c.ClientIP() against an allowlist or enforce an auth header/basic token via c.GetHeader()/c.Request, and return c.AbortWithStatus(403) when unauthorized; modify the function that constructs the handler (MetricsHandler) to perform this check and only call handler.ServeHTTP(c.Writer, c.Request) when the request passes the configured allowlist/auth check.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@metrics/ai_metrics.go`:
- Around line 45-51: The code is exposing raw token keys via the "token_name"
metric label (relayInfo.TokenKey) causing secret leakage and unbounded
cardinality; remove "token_name" from llmTokenLabelNames and any counters/gauges
that use it, and replace the mapped value with a safe, low-cardinality
alternative (e.g., a token_type, provider identifier, a boolean like
"has_token", or a fixed hash/salt-and-hash function) wherever relayInfo.TokenKey
is currently assigned; update the metric registration (variables referencing
llmTokenLabelNames and any counters using that label) and the code path that
sets token label (referencing relayInfo.TokenKey and "token_name") to emit the
new safe label instead.
- Around line 185-193: The code computes durations from relayInfo.StartTime
without checking it; guard against a zero StartTime before computing
serviceDuration and ttft by verifying relayInfo.StartTime.IsZero() and skipping
observation when it is zero. Specifically, in the block that calculates
serviceDuration (variable serviceDuration and metric llmServiceDuration) and the
TTFT logic that uses relayInfo.FirstResponseTime and llmFirstTokenDuration, add
a check for relayInfo.StartTime.IsZero() and only call time.Since(...) /
FirstResponseTime.Sub(...) and observe metrics when StartTime is non-zero (and
ttft > 0 as already done).
---
Nitpick comments:
In `@middleware/metrics.go`:
- Around line 94-99: The MetricsHandler currently exposes promhttp.Handler()
unconditionally; update MetricsHandler to gate access in production by checking
a configured policy before calling handler.ServeHTTP — e.g., read an allowlist
or auth flag from configuration (env var or app config), verify client IP via
c.ClientIP() against an allowlist or enforce an auth header/basic token via
c.GetHeader()/c.Request, and return c.AbortWithStatus(403) when unauthorized;
modify the function that constructs the handler (MetricsHandler) to perform this
check and only call handler.ServeHTTP(c.Writer, c.Request) when the request
passes the configured allowlist/auth check.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 262aa856-4130-4fec-834f-0961ea61005f
📒 Files selected for processing (2)
metrics/ai_metrics.gomiddleware/metrics.go
| "model", "channel", "upstream_model", "status", "error_type", | ||
| "region", "is_stream", "token_name", | ||
| } | ||
|
|
||
| var llmTokenLabelNames = []string{ | ||
| "model", "channel", "upstream_model", "region", "token_name", | ||
| } |
There was a problem hiding this comment.
Remove raw token keys from metric labels (secret leakage + cardinality blow-up).
Line 164-Line 166 maps relayInfo.TokenKey into token_name, and that label is used in counters on Line 170-Line 179. This can expose credentials in /metrics and create unbounded label cardinality.
Suggested fix
var llmRequestLabelNames = []string{
"model", "channel", "upstream_model", "status", "error_type",
- "region", "is_stream", "token_name",
+ "region", "is_stream",
}
var llmTokenLabelNames = []string{
- "model", "channel", "upstream_model", "region", "token_name",
+ "model", "channel", "upstream_model", "region",
}
@@
- tokenName := ""
- if relayInfo.TokenKey != "" {
- tokenName = relayInfo.TokenKey
- }
isStream := strconv.FormatBool(relayInfo.IsStream)
@@
llmRequestTotal.WithLabelValues(
model, channel, upstreamModel, status, errorType,
- region, isStream, tokenName,
+ region, isStream,
).Inc()
@@
- tokenLabels := []string{model, channel, upstreamModel, region, tokenName}
+ tokenLabels := []string{model, channel, upstreamModel, region}
llmInputTokenTotal.WithLabelValues(tokenLabels...).Add(float64(usage.PromptTokens))
llmOutputTokenTotal.WithLabelValues(tokenLabels...).Add(float64(usage.CompletionTokens))
}Also applies to: 163-166, 170-179
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@metrics/ai_metrics.go` around lines 45 - 51, The code is exposing raw token
keys via the "token_name" metric label (relayInfo.TokenKey) causing secret
leakage and unbounded cardinality; remove "token_name" from llmTokenLabelNames
and any counters/gauges that use it, and replace the mapped value with a safe,
low-cardinality alternative (e.g., a token_type, provider identifier, a boolean
like "has_token", or a fixed hash/salt-and-hash function) wherever
relayInfo.TokenKey is currently assigned; update the metric registration
(variables referencing llmTokenLabelNames and any counters using that label) and
the code path that sets token label (referencing relayInfo.TokenKey and
"token_name") to emit the new safe label instead.
| serviceDuration := time.Since(relayInfo.StartTime).Seconds() | ||
| llmServiceDuration.WithLabelValues(latencyLabels...).Observe(serviceDuration) | ||
|
|
||
| // Time-to-first-token (only meaningful when FirstResponseTime was recorded) | ||
| if !relayInfo.FirstResponseTime.IsZero() { | ||
| ttft := relayInfo.FirstResponseTime.Sub(relayInfo.StartTime).Seconds() | ||
| if ttft > 0 { | ||
| llmFirstTokenDuration.WithLabelValues(latencyLabels...).Observe(ttft) | ||
| } |
There was a problem hiding this comment.
Guard zero StartTime before duration calculations.
Line 185 computes time.Since(relayInfo.StartTime) unconditionally. If StartTime is zero, duration/TTFT values become invalid and heavily skew histograms.
Suggested fix
latencyLabels := []string{model, channel, region}
+ if relayInfo.StartTime.IsZero() {
+ return
+ }
+
// Service duration (total time from request start to now)
serviceDuration := time.Since(relayInfo.StartTime).Seconds()
llmServiceDuration.WithLabelValues(latencyLabels...).Observe(serviceDuration)
// Time-to-first-token (only meaningful when FirstResponseTime was recorded)
- if !relayInfo.FirstResponseTime.IsZero() {
+ if !relayInfo.FirstResponseTime.IsZero() && relayInfo.FirstResponseTime.After(relayInfo.StartTime) {
ttft := relayInfo.FirstResponseTime.Sub(relayInfo.StartTime).Seconds()🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@metrics/ai_metrics.go` around lines 185 - 193, The code computes durations
from relayInfo.StartTime without checking it; guard against a zero StartTime
before computing serviceDuration and ttft by verifying
relayInfo.StartTime.IsZero() and skipping observation when it is zero.
Specifically, in the block that calculates serviceDuration (variable
serviceDuration and metric llmServiceDuration) and the TTFT logic that uses
relayInfo.FirstResponseTime and llmFirstTokenDuration, add a check for
relayInfo.StartTime.IsZero() and only call time.Since(...) /
FirstResponseTime.Sub(...) and observe metrics when StartTime is non-zero (and
ttft > 0 as already done).
1 participant
Summary
Add native Prometheus metrics support to New API, covering both HTTP gateway metrics and LLM-specific metrics. This enables production-grade observability without external instrumentation.
Closes #2402
Motivation
New API currently lacks built-in observability. Operators must rely on external reverse proxies or custom scripts to monitor request rates, latencies, and token usage. This PR adds first-class Prometheus metrics that can be scraped by any Prometheus-compatible system (Prometheus, Grafana Agent, VictoriaMetrics, etc.).
Changes
New Files (already in the repo)
middleware/metrics.go— GIN middleware that collects HTTP golden signalsmetrics/ai_metrics.go— LLM metrics recorder with token counts, latency, and TTFTModified Files (this PR)
main.go— RegisterPrometheusMiddleware()in the GIN middleware chainrouter/main.go— AddGET /metricsendpoint for Prometheus scrapingservice/text_quota.go— Callmetrics.RecordAIMetrics()after text quota consumptionservice/quota.go— Callmetrics.RecordAIMetrics()after WSS quota consumptionMetrics Exposed
HTTP Metrics (4)
newapi_http_requests_totalnewapi_http_request_duration_secondsnewapi_http_requests_in_flightnewapi_http_response_size_bytesLLM Metrics (5)
newapi_llm_input_token_totalnewapi_llm_output_token_totalnewapi_llm_request_totalnewapi_llm_service_duration_secondsnewapi_llm_first_token_duration_secondsDesign Decisions
prometheus/client_golangis already an indirect dependencyFullPath()to avoid high-cardinality label explosion/metricsis scrapedUsage
Testing
/metricsendpoint returns valid Prometheus exposition formatSummary by CodeRabbit