Skip to content

fix(spanner): stop unclosed multiplexed clients leaking, and fix close() races (P1-7, P1-8, P1-9)#10

Open
fornwall wants to merge 2 commits into
mainfrom
fix/p1-7-9-multiplexed-client-lifecycle
Open

fix(spanner): stop unclosed multiplexed clients leaking, and fix close() races (P1-7, P1-8, P1-9)#10
fornwall wants to merge 2 commits into
mainfrom
fix/p1-7-9-multiplexed-client-lifecycle

Conversation

@fornwall

Copy link
Copy Markdown
Owner

Fixes three related lifecycle defects in MultiplexedSessionDatabaseClient, from the review file entries P1-7, P1-8 and P1-9. They are in one PR because they share a failure surface (client shutdown / client lifetime) and one of them cannot be fixed meaningfully without the other.

P1-7 — static roots pin every un-close()d client

Two independent static roots kept the whole client graph alive forever:

  • CHANNEL_USAGE was a HashMap keyed strongly on SpannerImpl. Chain: CHANNEL_USAGESpannerImpldbClientsDatabaseClientImplMultiplexedSessionDatabaseClient → channels, session clients, per-database caches.
  • The maintenance task scheduled on the static MAINTAINER_SERVICE was a this::maintain method reference, so the executor's queue held the maintainer → the client → the same graph, and kept running every 10 minutes.

Both had to be fixed: bounding one while the other still pins is a no-op.

  • CHANNEL_USAGE is now a WeakHashMap. Entries for a SpannerImpl still in use stay alive because every MultiplexedSessionDatabaseClient holds a strong spanner field, so live clients are unaffected. SharedChannelUsage holds only a BitSet and an int, so there is no value→key reference to defeat the weak key.
  • The scheduled task is now a MaintainerTask holding a WeakReference to the maintainer. When the reference has been cleared it cancels its own future, so an abandoned client's task stops instead of running for the lifetime of the JVM.

P1-8 — close() racing initial session creation leaked a permanent maintainer task

If close() ran while the initial CreateSession RPC was in flight, maintainer.stop() found scheduledFuture == null and no-oped. onSessionReady then called maintainer.start(), scheduling a fixed-rate task on the static executor that nothing ever cancelled — after the 7-day expiry it kept calling asyncCreateMultiplexedSession against a closed client.

The maintainer now records that it was stopped, and start() returns early in that case. start() and stop() are both synchronized on the maintainer, so the race resolves correctly either way round:

  • start() wins the monitor → it schedules; close() then blocks in stop() until start() returns, and cancels the future it finds.
  • stop() wins → it sets stopped; start() then sees it and skips scheduling.

Also added a guard against start() scheduling a second task if it is ever called twice — only the last future is retained, so a second call would leak the first. Both call sites are one-shot today; the guard makes it structural rather than incidental.

P1-9 — isClosed read without the lock that writes it

isClosed was written under synchronized(this) in close() but read unsynchronized in createMultiplexedSessionTransaction(). A thread creating a transaction was not guaranteed to observe the close and could keep operating on a closed client instead of getting IllegalStateException. It is now volatile.

Tests

Added to MultiplexedSessionDatabaseClientTest:

Test Covers
testChannelUsageDoesNotPinSpannerThatIsNotClosed P1-7a — drops all references without closing, asserts the SpannerImpl becomes collectable and the map entry is expunged
testMaintainerTaskCancelsItselfWhenMaintainerIsCollected P1-7b — clears the WeakReference directly (no GC dependence) and asserts the task cancels its own future
testCloseBeforeSessionReadyDoesNotScheduleMaintainer P1-8 — closes while CreateSession is in flight, asserts no task is scheduled
testSessionReadySchedulesMaintainer positive control — task is scheduled, and cancelled by close()
testMaintainerIsOnlyScheduledOnce double-start() guard
testIsClosedIsVolatile, testUseAfterCloseThrows P1-9

A new DeferredMultiplexedSessionClient test helper withholds onSessionReady until the test triggers it, which is what makes the P1-8 race deterministic.

Verified the tests actually catch the bugs: with the three source fixes reverted, testIsClosedIsVolatile, testChannelUsageDoesNotPinSpannerThatIsNotClosed and testCloseBeforeSessionReadyDoesNotScheduleMaintainer all fail; with the fixes they pass.

Full google-cloud-spanner module suite: 9781 tests, 0 failures, 0 errors. fmt:check clean.

🤖 Generated with Claude Code

https://claude.ai/code/session_01XjifXkL1rXXuiWzHmpXJGA

fornwall and others added 2 commits July 20, 2026 02:00
…e() races

Three related lifecycle defects in MultiplexedSessionDatabaseClient:

1. Static roots pinned every un-close()d client. The static CHANNEL_USAGE map
   was keyed strongly on SpannerImpl, and the maintainer task scheduled on the
   static MAINTAINER_SERVICE held a strong reference to the maintainer (and so
   to the whole client graph). An application that recreates Spanner instances
   without closing them leaked channels, session clients and per-database
   caches permanently, plus a live 10-minute task each. CHANNEL_USAGE is now a
   WeakHashMap, and the scheduled task holds only a WeakReference to the
   maintainer and cancels itself once that reference is cleared.

2. close() racing the initial CreateSession leaked a permanent maintainer task.
   If close() ran while the initial CreateSession RPC was in flight,
   maintainer.stop() found scheduledFuture == null and no-oped, and the later
   onSessionReady() called maintainer.start(), scheduling a fixed-rate task that
   nothing ever cancelled. The maintainer now records that it was stopped and
   start() returns early in that case; both methods are synchronized on the
   maintainer, so the ordering holds regardless of which side wins the race.

3. isClosed was written under synchronized(this) in close() but read without
   any synchronization in createMultiplexedSessionTransaction(), so a thread
   creating a transaction could keep operating on a closed client instead of
   getting an IllegalStateException. It is now volatile.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XjifXkL1rXXuiWzHmpXJGA
…ns twice

Only the last scheduled future is retained by the maintainer, so a second
start() would leak the first task. Both callers of start() are one-shot today,
but the guard makes the leak impossible rather than incidental.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XjifXkL1rXXuiWzHmpXJGA
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant