Fix shutdown ordering and WaitGroup race between discovery and search close - #326
Open
damilolaedwards wants to merge 1 commit into
Open
Conversation
… close Server cleanup closed the search runtime before stopping the app, and the proxy client's Stop only signaled its background refresh goroutine to exit rather than waiting for it. A discovery tick already in flight could still call the search runtime's OnDiscover, adding to its WaitGroup at the same time Close was waiting on it, which the WaitGroup docs call misuse and the Go runtime turns into a panic. Even without the panic, Close could block for however long that stray activation took, with no bound. Three changes close this from both ends: cleanup now stops the app before closing the search runtime, the proxy client's Stop joins its background refresh goroutine instead of just signaling it, and the search runtime tracks whether it has started closing so OnDiscover becomes a no-op instead of adding to the WaitGroup once it has. Any of the three would narrow the window; together they close it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Server cleanup closed the search runtime before stopping the app, and the
proxy client's Stop only signaled its background refresh goroutine to exit
rather than waiting for it. A discovery tick already in flight could still
call the search runtime's OnDiscover, adding to its WaitGroup at the same
time Close was waiting on it, which the WaitGroup docs call misuse and the
Go runtime turns into a panic. Even without the panic, Close could block
for however long that stray activation took, with no bound.
Three changes close this from both ends: cleanup now stops the app before
closing the search runtime, the proxy client's Stop joins its background
refresh goroutine instead of just signaling it, and the search runtime
tracks whether it has started closing so OnDiscover becomes a no-op
instead of adding to the WaitGroup once it has. Any of the three would
narrow the window; together they close it.
Test plan
go test -race ./pkg/searchruntime/... ./pkg/proxy/... ./pkg/server/... ./pkg/app/...and confirm Stop waits for an in-flight background refresh tick;
confirmed both catch the old bugs (a real data race and panic on one,
a premature return on the other)
go build ./...,go vet ./...