fix(spanner): stop unclosed multiplexed clients leaking, and fix close() races (P1-7, P1-8, P1-9)#10
Open
fornwall wants to merge 2 commits into
Open
fix(spanner): stop unclosed multiplexed clients leaking, and fix close() races (P1-7, P1-8, P1-9)#10fornwall wants to merge 2 commits into
fornwall wants to merge 2 commits into
Conversation
…e() races Three related lifecycle defects in MultiplexedSessionDatabaseClient: 1. Static roots pinned every un-close()d client. The static CHANNEL_USAGE map was keyed strongly on SpannerImpl, and the maintainer task scheduled on the static MAINTAINER_SERVICE held a strong reference to the maintainer (and so to the whole client graph). An application that recreates Spanner instances without closing them leaked channels, session clients and per-database caches permanently, plus a live 10-minute task each. CHANNEL_USAGE is now a WeakHashMap, and the scheduled task holds only a WeakReference to the maintainer and cancels itself once that reference is cleared. 2. close() racing the initial CreateSession leaked a permanent maintainer task. If close() ran while the initial CreateSession RPC was in flight, maintainer.stop() found scheduledFuture == null and no-oped, and the later onSessionReady() called maintainer.start(), scheduling a fixed-rate task that nothing ever cancelled. The maintainer now records that it was stopped and start() returns early in that case; both methods are synchronized on the maintainer, so the ordering holds regardless of which side wins the race. 3. isClosed was written under synchronized(this) in close() but read without any synchronization in createMultiplexedSessionTransaction(), so a thread creating a transaction could keep operating on a closed client instead of getting an IllegalStateException. It is now volatile. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XjifXkL1rXXuiWzHmpXJGA
…ns twice Only the last scheduled future is retained by the maintainer, so a second start() would leak the first task. Both callers of start() are one-shot today, but the guard makes the leak impossible rather than incidental. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XjifXkL1rXXuiWzHmpXJGA
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes three related lifecycle defects in
MultiplexedSessionDatabaseClient, from the review file entries P1-7, P1-8 and P1-9. They are in one PR because they share a failure surface (client shutdown / client lifetime) and one of them cannot be fixed meaningfully without the other.P1-7 — static roots pin every un-
close()d clientTwo independent static roots kept the whole client graph alive forever:
CHANNEL_USAGEwas aHashMapkeyed strongly onSpannerImpl. Chain:CHANNEL_USAGE→SpannerImpl→dbClients→DatabaseClientImpl→MultiplexedSessionDatabaseClient→ channels, session clients, per-database caches.MAINTAINER_SERVICEwas athis::maintainmethod reference, so the executor's queue held the maintainer → the client → the same graph, and kept running every 10 minutes.Both had to be fixed: bounding one while the other still pins is a no-op.
CHANNEL_USAGEis now aWeakHashMap. Entries for aSpannerImplstill in use stay alive because everyMultiplexedSessionDatabaseClientholds a strongspannerfield, so live clients are unaffected.SharedChannelUsageholds only aBitSetand anint, so there is no value→key reference to defeat the weak key.MaintainerTaskholding aWeakReferenceto the maintainer. When the reference has been cleared it cancels its own future, so an abandoned client's task stops instead of running for the lifetime of the JVM.P1-8 —
close()racing initial session creation leaked a permanent maintainer taskIf
close()ran while the initialCreateSessionRPC was in flight,maintainer.stop()foundscheduledFuture == nulland no-oped.onSessionReadythen calledmaintainer.start(), scheduling a fixed-rate task on the static executor that nothing ever cancelled — after the 7-day expiry it kept callingasyncCreateMultiplexedSessionagainst a closed client.The maintainer now records that it was stopped, and
start()returns early in that case.start()andstop()are bothsynchronizedon the maintainer, so the race resolves correctly either way round:start()wins the monitor → it schedules;close()then blocks instop()untilstart()returns, and cancels the future it finds.stop()wins → it setsstopped;start()then sees it and skips scheduling.Also added a guard against
start()scheduling a second task if it is ever called twice — only the last future is retained, so a second call would leak the first. Both call sites are one-shot today; the guard makes it structural rather than incidental.P1-9 —
isClosedread without the lock that writes itisClosedwas written undersynchronized(this)inclose()but read unsynchronized increateMultiplexedSessionTransaction(). A thread creating a transaction was not guaranteed to observe the close and could keep operating on a closed client instead of gettingIllegalStateException. It is nowvolatile.Tests
Added to
MultiplexedSessionDatabaseClientTest:testChannelUsageDoesNotPinSpannerThatIsNotClosedSpannerImplbecomes collectable and the map entry is expungedtestMaintainerTaskCancelsItselfWhenMaintainerIsCollectedWeakReferencedirectly (no GC dependence) and asserts the task cancels its own futuretestCloseBeforeSessionReadyDoesNotScheduleMaintainerCreateSessionis in flight, asserts no task is scheduledtestSessionReadySchedulesMaintainerclose()testMaintainerIsOnlyScheduledOncestart()guardtestIsClosedIsVolatile,testUseAfterCloseThrowsA new
DeferredMultiplexedSessionClienttest helper withholdsonSessionReadyuntil the test triggers it, which is what makes the P1-8 race deterministic.Verified the tests actually catch the bugs: with the three source fixes reverted,
testIsClosedIsVolatile,testChannelUsageDoesNotPinSpannerThatIsNotClosedandtestCloseBeforeSessionReadyDoesNotScheduleMaintainerall fail; with the fixes they pass.Full
google-cloud-spannermodule suite: 9781 tests, 0 failures, 0 errors.fmt:checkclean.🤖 Generated with Claude Code
https://claude.ai/code/session_01XjifXkL1rXXuiWzHmpXJGA