feat(observability): frontend Faro RUM + worker→Loki log shipping (ships dark)#86
feat(observability): frontend Faro RUM + worker→Loki log shipping (ships dark)#86xBalbinus wants to merge 6 commits into
Conversation
RUM + product events for the client (Carey's frontend-observability spec, slice 5): web vitals, uncaught errors/unhandled rejections, React error-boundary catches, TanStack route transitions, sanitized API failures, WS lifecycle, and a small set of product events (prompt_submitted, session_created, workflow_run, approval_resolved). Hard no-op unless VITE_FARO_URL is set at build time — mirrors the worker's OTLP-endpoint gate, and the SDK is dynamically imported so dark builds contain zero Faro code (verified against the emitted chunks). Privacy: a beforeSend hook strips query strings from every URL in every beacon and redacts token-shaped values (glc_/gh*_/Bearer/api-token); identity is the opaque user id only (set on login, reset on logout); event attributes are ids/enums/counts, never free text; console capture is off. Verified end-to-end by a real-SDK beacon test (jsdom + local HTTP collector). traceparent/tracestate are propagated only to the API origin (derived from VITE_API_URL) and now allowed by the worker's CORS preflight so frontend spans correlate with worker traces in Tempo.
Structured log lines from lib/log.ts were console-only — visible in
wrangler tail but absent from Grafana, so trace<->log pivoting on
trace_id never worked in Loki. Buffer each emitted line (bounded at 500,
oldest dropped and reported as dropped_total on the next flush) and push
batches to {LOKI_PUSH_URL}/loki/api/v1/push labeled {service_name,
level}, keeping console output byte-identical.
Workers have no cross-request timers, so flushing rides existing
lifecycle hooks via waitUntil: a Hono middleware on the fetch path, the
end of the scheduled tick, and the DOs' flushTraces drivers
(alarm/close/hibernate) — no new flush sites. Unset LOKI_PUSH_URL costs
one boolean per line and zero network; push failures are swallowed
(single console.warn, batch counted dropped) like the trace exporter.
…nting Review fixes on the Loki log-shipping branch: - log.test.ts cast '[]' -> tuple via unknown (TS2352 failed 'tsc --build', the CI typecheck gate, even though vitest passed) - flushLogs now ships a pending drop report even when the buffer is empty, so drops from a failed final push are visible in Loki, not just console - a failed push no longer re-counts the synthetic report line in dropped_total, and drops whose report line was lost are re-reported - docs: drop the verbatim Tempo push instance ID; the warning reads the same without committing an internal account identifier
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
|
Warning Review the following alerts detected in dependencies. According to your organization's Security Policy, it is recommended to resolve "Warn" alerts. Learn more about Socket for GitHub.
|
|
Preview deployment: https://pr-86.dev-valet-turnkey-client.pages.dev |
xBalbinus
changed the title
# Conflicts: # packages/client/package.json # pnpm-lock.yaml
…in ids - route_change now carries the from pathname (pathname only) so route funnels get edges, not just nodes - ws_disconnected carries the close code and wasClean flag; ws_connected carries the reconnect attempt count that got the connection up - approval_resolved carries valet.session.id and valet.invocation.id so approval behavior joins to server-side action_invocations All attributes stay ids/enums/numbers — no free text.
1 participant
Observability rails: frontend Faro RUM + worker→Loki logs — both shipped dark
Two halves of the same story — closing the frontend fog-of-war and making backend log lines Grafana-visible — combined per review-flow preference. Each half is a hard no-op until its env var is set, so this merges safely with zero behavior change.
Part 1 — Grafana Faro frontend observability (
packages/client)RUM + product events, shipped dark (hard no-op unless
VITE_FARO_URLis set; dark builds contain zero Faro code via dynamic import, verified by grepping emitted chunks). Implements slice 5 of Carey's observability roadmap (#24): the "Grafana Cloud equivalent of Sentry/PostHog".Captured when enabled
onResolved, pathname only)trackEvent:prompt_submitted(counts/ids only),session_created,workflow_run,approval_resolved,api_error(status/code/query-stripped path),ws_connected/disconnected/errorTracingInstrumentationpropagatestraceparentonly to the API origin (anchored-regex match; worker CORS now allowstraceparent/tracestate)Privacy:
beforeSenddeep-scrubs every beacon — query strings stripped from all URLs (incl. stack frames + fetch spans),glc_/gh*_/Bearer/api-token shapes redacted, identity is the opaque user id only (synced from the auth store, cleared on logout). No prompts, no message content, no emails.Part 2 — Worker/DO log shipping to Loki (
packages/worker)Today
log.warn/log.infolines are visible only inwrangler tail(verified 2026-07-06: no valet service in Loki, so #70/#82'sintegration auth/refresh failedwarns are invisible in Grafana). Dark by default.lib/log.tsemit()additionally buffers each line (bounded 500; oldest dropped + drop-count reported as a synthetic warn on next flush) whenLOKI_PUSH_URLis configured; unconfigured cost = one boolean check, console output byte-identical (tail unaffected)flushLogs()POSTs Loki push payloads (service_name+levellabels, ns timestamps, line = the exact JSON incl.trace_id/span_id→ trace↔log correlation)waitUntilafter response), the DOs' existing trace-flush drivers, and the scheduled handlerEvidence
?code=oauth-secretstripped andglc_token[REDACTED]); dark build has zero Faro symbols indist; enabled build emits Faro chunks cleanlygrafana/otel-lgtm— real worker underwrangler dev→/healthtraffic → Lokiquery_rangereturns the lines withtrace_idpresentTo turn it on (independent switches)
FARO_URLin the deploy env (plumbed throughbuild_clientasVITE_FARO_URL)logs:writetoken + the stack's Loki instance ID from Paul (not the traces token — 410780 is the Tempo push instance) → setLOKI_PUSH_URL+LOKI_BASIC_AUTHas worker secretsUntil then every deploy stays dark. Docs:
docs/observability.md§§ Frontend observability / Log shipping to Loki.