Skip to content

Stop holding the write lock across proxy shutdown drain - #324

Open
damilolaedwards wants to merge 1 commit into
ethpandaops:masterfrom
damilolaedwards:fix/proxy-shutdown-lock-across-drain
Open

Stop holding the write lock across proxy shutdown drain#324
damilolaedwards wants to merge 1 commit into
ethpandaops:masterfrom
damilolaedwards:fix/proxy-shutdown-lock-across-drain

Conversation

@damilolaedwards

Copy link
Copy Markdown

Summary

Stop held the server's write lock for its entire duration, including the
call to httpSrv.Shutdown, which blocks until in-flight requests finish. The
/ready handler took the same lock to read the readiness flag, so a /ready
request landing while Stop held the lock would park on it, keeping its
connection open. Shutdown then waited on that same connection to go idle,
which could never happen while the handler waited on a lock only Stop's own
completion would release. A rolling restart with active readiness probing
could deadlock for the entire shutdown budget.

Makes the readiness flag an atomic so /ready and Ready() never contend for
the lock at all, and restructures Stop to hold the lock only for the
bounded teardown steps, releasing it before the blocking shutdown and drain
wait. Stop is also now idempotent via an atomic compare-and-swap instead of
a started check under the lock.

Test plan

  • go test -race ./pkg/proxy/...
  • New tests drive a real running server under concurrent /ready traffic
    and confirm Stop returns promptly, and confirm Stop is safe under
    concurrent callers; confirmed the traffic test fails (full timeout,
    deadline exceeded) on the old code
  • go build ./..., go vet ./...

Stop held the server's write lock for its entire duration, including
the call to httpSrv.Shutdown, which blocks until in-flight requests
finish. The /ready handler took the same lock to read the readiness
flag, so a /ready request landing while Stop held the lock would park
on it, keeping its connection open. Shutdown then waited on that same
connection to go idle, which could never happen while the handler
waited on a lock only Stop's own completion would release. A rolling
restart with active readiness probing could deadlock for the entire
shutdown budget.

Make the readiness flag an atomic so /ready and Ready() never contend
for the lock at all, and restructure Stop to hold the lock only for
the bounded teardown steps, releasing it before the blocking shutdown
and drain wait. Stop is also now idempotent via an atomic compare-and-
swap instead of a started check under the lock.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant