feat(worker): ship trace-stamped log lines to Loki, dark by default#87
Closed
xBalbinus wants to merge 2 commits into
Closed
feat(worker): ship trace-stamped log lines to Loki, dark by default#87xBalbinus wants to merge 2 commits into
xBalbinus wants to merge 2 commits into
Conversation
Structured log lines from lib/log.ts were console-only — visible in
wrangler tail but absent from Grafana, so trace<->log pivoting on
trace_id never worked in Loki. Buffer each emitted line (bounded at 500,
oldest dropped and reported as dropped_total on the next flush) and push
batches to {LOKI_PUSH_URL}/loki/api/v1/push labeled {service_name,
level}, keeping console output byte-identical.
Workers have no cross-request timers, so flushing rides existing
lifecycle hooks via waitUntil: a Hono middleware on the fetch path, the
end of the scheduled tick, and the DOs' flushTraces drivers
(alarm/close/hibernate) — no new flush sites. Unset LOKI_PUSH_URL costs
one boolean per line and zero network; push failures are swallowed
(single console.warn, batch counted dropped) like the trace exporter.
…nting Review fixes on the Loki log-shipping branch: - log.test.ts cast '[]' -> tuple via unknown (TS2352 failed 'tsc --build', the CI typecheck gate, even though vitest passed) - flushLogs now ships a pending drop report even when the buffer is empty, so drops from a failed final push are visible in Loki, not just console - a failed push no longer re-counts the synthetic report line in dropped_total, and drops whose report line was lost are re-reported - docs: drop the verbatim Tempo push instance ID; the warning reads the same without committing an internal account identifier
|
Preview deployment: https://pr-87.dev-valet-turnkey-client.pages.dev |
Author
|
Consolidated into #86 (observability rails) — the branch was merged there with full history (signed merge commit); all tests re-run green on the combined branch. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Ship the worker's (and DOs') trace-stamped JSON log lines to Loki, dark by default — today
log.warn/log.infolines are visible only inwrangler tail; verified 2026-07-06 that no valet service exists in Loki, so e.g. #70/#82'sintegration auth/refresh failedwarns are invisible in Grafana.Design
lib/log.tsemit()additionally buffers each line (bounded 500; oldest dropped + drop-count reported as a synthetic warn on next flush) whenLOKI_PUSH_URLis configured; unconfigured cost = one boolean check, console output byte-identical (tail unaffected)flushLogs()POSTs Loki push payloads (service_name+levellabels, ns timestamps, line = the exact JSON incl.trace_id/span_id→ trace↔log correlation in Grafana)waitUntilafter response), the DOs' existing trace-flush drivers, and the scheduled handler (cron lines would otherwise sit buffered)Evidence
grafana/otel-lgtm: real worker underwrangler dev→/healthtraffic → Lokiquery_rangereturns the lines withtrace_idpresentTo turn it on
Needs a
logs:writetoken + the stack's Loki instance ID from Paul — this is not the existing traces token (410780 is the Tempo push instance). SetLOKI_PUSH_URL+LOKI_BASIC_AUTHas worker secrets. Docs:docs/observability.md§ Log shipping to Loki (includes the local otel-lgtm test recipe).