Skip to content

Fix shutdown ordering and WaitGroup race between discovery and search close - #326

Open
damilolaedwards wants to merge 1 commit into
ethpandaops:masterfrom
damilolaedwards:fix/discovery-shutdown-waitgroup-race
Open

Fix shutdown ordering and WaitGroup race between discovery and search close#326
damilolaedwards wants to merge 1 commit into
ethpandaops:masterfrom
damilolaedwards:fix/discovery-shutdown-waitgroup-race

Conversation

@damilolaedwards

Copy link
Copy Markdown

Summary

Server cleanup closed the search runtime before stopping the app, and the
proxy client's Stop only signaled its background refresh goroutine to exit
rather than waiting for it. A discovery tick already in flight could still
call the search runtime's OnDiscover, adding to its WaitGroup at the same
time Close was waiting on it, which the WaitGroup docs call misuse and the
Go runtime turns into a panic. Even without the panic, Close could block
for however long that stray activation took, with no bound.

Three changes close this from both ends: cleanup now stops the app before
closing the search runtime, the proxy client's Stop joins its background
refresh goroutine instead of just signaling it, and the search runtime
tracks whether it has started closing so OnDiscover becomes a no-op
instead of adding to the WaitGroup once it has. Any of the three would
narrow the window; together they close it.

Test plan

  • go test -race ./pkg/searchruntime/... ./pkg/proxy/... ./pkg/server/... ./pkg/app/...
  • New tests hammer concurrent OnDiscover/Close calls across many trials
    and confirm Stop waits for an in-flight background refresh tick;
    confirmed both catch the old bugs (a real data race and panic on one,
    a premature return on the other)
  • go build ./..., go vet ./...

… close

Server cleanup closed the search runtime before stopping the app, and
the proxy client's Stop only signaled its background refresh goroutine
to exit rather than waiting for it. A discovery tick already in flight
could still call the search runtime's OnDiscover, adding to its
WaitGroup at the same time Close was waiting on it, which the
WaitGroup docs call misuse and the Go runtime turns into a panic. Even
without the panic, Close could block for however long that stray
activation took, with no bound.

Three changes close this from both ends: cleanup now stops the app
before closing the search runtime, the proxy client's Stop joins its
background refresh goroutine instead of just signaling it, and the
search runtime tracks whether it has started closing so OnDiscover
becomes a no-op instead of adding to the WaitGroup once it has. Any of
the three would narrow the window; together they close it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant