fix(go/observability): control-plane telemetry pipeline fixes#271
Merged
Conversation
Zhipu/GLM's Anthropic-compatible streaming reports the real usage (input+output) only in the final message_delta and leaves message_start's usage all-zero. The decoder only read output_tokens from message_delta, so input_tokens was always 0 in metrics/logs. Adopt input/cache counts from message_delta when present; real Anthropic omits them there (input stays from message_start), so its behavior is unchanged. Golden files updated: they had codified the input_tokens=0 bug (the anthropic stream cassette carries input_tokens in message_delta).
gateway_error -> GATEWAY_ERROR (proxy handlers and the admin WebUI NotFound) and auth_error -> AUTH_ERROR (admin unauthorized). The AiErrorKind protocol values (authentication_error, server_error, ...) stay snake_case: they are the OpenAI/Anthropic wire error.type contract, not framework errors.
In --config-server mode the gateway built its OTel provider once from the still-empty config cache (resolving to the fixed stdout default) and never rebuilt it, so the admin-seeded otlp settings that arrive with the first config-sync snapshot never took effect: logs went to the gateway's stdout and nothing reached the control plane. Introduce SwappableProvider behind the once-registered phase hooks, add a ConfigCache OnSwap callback, and an obsManager that re-resolves the obs config on each snapshot and atomically rebuilds the provider (and reconciles the prometheus scrape server) when it changes.
…erval The admin-seeded OTLP endpoint is a base URL (scheme://host:port) shared across the three signals; otlploghttp/otlpmetrichttp/otlptracehttp's WithEndpointURL uses the URL path verbatim and does NOT append the default /v1/<signal> when it is empty, so exports hit "/" and the receiver 404'd. Append the per-signal path (otlpSignalURL), leaving an explicit path untouched. Also thread the configured "interval" into the logs BatchProcessor (WithExportInterval); it was dropped, so the WebUI Export Interval had no effect for logs (fixed 1s SDK default) while metrics and traces honored it.
The parquet sinks flushed only on their size trigger (maxRows) or at shutdown, so a low-traffic admin never persisted rows to a file and /logs and /stats stayed empty until restart. Add the missing time trigger: the receiver runs one flusher per signal, cadence from the admin-local obs_<signal>_flush_interval settings (siblings of the retention settings, default 5s), editable in the WebUI's Local Telemetry card. Size-or-time, whichever fires first.
Left over from removing the startMetricsServer tests; `make go-check` (golangci-lint fmt) flagged it.
Two fixes to what the WebUI logs table shows, both from the same report:
- Protocol column was blank. Logs stored the versioned endpoint id
("anthropic-messages/2023-06-01"), but the UI resolves display names
against the base protocol ids. Store the base Protocol
("anthropic-messages") instead — the version is an internal routing
detail, not part of the protocol's identity — so the column resolves.
- Key column showed the key preview, not a name. The consumer key's Name
was never propagated to the data plane (config-sync only carried
preview+hash). Add ConsumerKeyRef.name to the config-sync contract, thread
it through the snapshot/standalone paths, and log it as nyro.api_key_name
(falling back to the preview for unnamed keys). Also emit
nyro.api_key_preview so the UI can show the name with the preview beneath.
…sked
The request-log table showed the raw key_preview (12 leading + 6 trailing
chars, no mask), inconsistent with the API-keys page which renders it masked
via formatKeyPreview. Move formatKeyPreview into lib/format and reuse it in
both the API-keys page and the logs table, so a preview reads the same
everywhere ("sk-abcdef****…****klmnop") instead of looking like a full key.
The request-log key column showed the masked preview on a second line under the consumer key name. Move it into a hover tooltip on the name instead — the same affordance as the nodes list' connection-time tooltip — so the row stays compact. Unnamed keys (whose name falls back to the preview) render the masked preview directly, and rows with no key still show a dash.
golangci-lint errcheck flagged the deferred prov.Shutdown; discard its return explicitly.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes a chain of issues found while bringing up the default data-plane + control-plane telemetry flow (OpenAI Responses client → Nyro → Anthropic Messages/Zhipu). Each is a separate commit.
What was broken
--config-servermode). The OTel provider was built once at startup from the still-empty config cache → resolved to the stdout default and was never rebuilt when the first config-sync snapshot arrived. Result: request logs went to the gateway's stdout; nothing reached the control plane.http://host:port) shared by all three signals;WithEndpointURLdoes not append the default/v1/<signal>path when it is empty, so exports POSTed to/→ adminNotFound.maxRows=50000) or at shutdown — no time trigger — so low-traffic data never became queryable until restart.message_delta; the Anthropic decoder readinput_tokensonly frommessage_start(which GLM sends as 0).typecasing. Framework error envelope types lowercased.Changes (per commit)
fix(go/anthropic): adopt input/cache token counts frommessage_deltawhen present (real Anthropic unchanged); goldens updated (they had codified the bug).refactor(go):gateway_error→GATEWAY_ERROR,auth_error→AUTH_ERROR.AiErrorKindprotocol values stay snake_case (OpenAI/Anthropic wire contract).fix(go/observability): hot-reload —SwappableProviderbehind the once-registered hooks +ConfigCacheOnSwap callback +obsManagerthat rebuilds the provider (and prometheus server) on obs-config change.fix(go/observability): append per-signal/v1/<signal>path to base OTLP endpoints; honor the logs export interval.feat(go/observability): per-signal time-triggered parquet flush viaobs_<signal>_flush_interval(admin-local settings, sibling of retention, default 5s, editable in the WebUI Local Telemetry card).Verification
go build ./... && go vet ./... && go test ./...green (protocoltest goldens regenerated).npm run lint+npm run build+ settings-layout tests green./logsand/statspopulate and showinput_tokens=11, output_tokens=81.Notes for reviewers
obs_<signal>_flush_intervaland retention are admin-local (not pushed to the data plane) and applied at admin boot — the WebUI card notes "Restart Admin to apply".