logger: write to stdout asynchronously#39
Closed
firecow wants to merge 1 commit into
Closed
Conversation
Wraps os.Stdout in a shared AsyncWriter that drains on a single background goroutine through a buffered channel, so callers don't stall when stdout's consumer (filebeat, the container runtime, kubectl logs) falls behind. Drops + counts when the buffer is full. The change is transparent: logger.New() and logger.NewWithLevel() keep their signatures, all existing callers become async on upgrade. Added logger.Flush() for graceful shutdown and logger.Dropped() for observability. Also fixes a non-deterministic bug in periodic.Run: when ctx.Done() and ticker.C are both ready, select can pick the tick first and run fn after cancellation. Re-checks ctx.Err() after consuming a tick.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
os.Stdoutin a sharedAsyncWriter(buffered channel + single drain goroutine) so callers don't stall when stdout's consumer (filebeat, the container runtime,kubectl logs) falls behind. Dropped lines are counted.logger.New()/logger.NewWithLevel()signatures unchanged, all existing callers become async on upgrade.logger.Flush()for graceful shutdown andlogger.Dropped()for observability.periodic.Run: whenctx.Done()andticker.Care both ready,selectcan pick the tick first and runfnafter cancellation. Re-checkctx.Err()after consuming a tick. Found while running this PR's tests under-race.Motivation
Two SSO outages on 2026-05-19 traced to synchronous JSON-to-stdout writes from request handlers: when stdout backpressured, handlers stalled long enough that kubelet's 1s liveness probes timed out and pods got killed. Anything depending on this package has the same fragility.
Test plan
go test -race -count=10 ./logger/...go test -race -count=20 ./periodic/...(verifying the fix)go test -race -count=3 ./...golangci-lint run ./...