Fix OA-1097: evict ServerRoutingStatsManager entries when servers are decommissioned#177
Open
siddharthteotia wants to merge 1 commit into
Open
Conversation
… decommissioned ServerRoutingStatsManager._serverQueryStatsMap accumulated entries forever. Each ServerRoutingStatsEntry holds two ExponentialMovingAverage objects, each scheduling a periodic auto-decay task in _periodicTaskExecutor. On long-running brokers, decommissioned servers left orphaned entries and live ScheduledFutures that were never cleaned up. Fix: - ExponentialMovingAverage: store the ScheduledFuture from scheduleAtFixedRate and expose cancel() to stop the periodic decay task. - ServerRoutingStatsEntry: add cancel() that delegates to both EMA instances. - ServerRoutingStatsManager: add removeServerStats(instanceId) that atomically removes the entry from the map and calls cancel() on it, releasing both the map reference and the scheduled tasks so the entry can be GC'd. - BrokerRoutingManager.processInstanceConfigsChange(): call removeServerStats() for each newly-disabled server immediately after removing it from _enabledServerInstanceMap, which is the authoritative point where the broker learns a server has left the cluster. Test: testRemoveServerStats verifies that entries are evicted correctly, that removing an absent server is a safe no-op, and that the method is a no-op when the manager is disabled. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes a slow memory leak in
ServerRoutingStatsManagerwhere stats entries for decommissioned servers accumulated indefinitely on long-running brokers.Root cause
ServerRoutingStatsManager._serverQueryStatsMapusedcomputeIfAbsent()to create entries on first use but never removed them. The original comment at line 99 acknowledged this: "Entries in this map are never deleted unless the broker process restarts."The leak was two-layered:
Layer 1 — Map entries never evicted
Every server that ever received a query got a
ServerRoutingStatsEntryin the map. Servers decommissioned and replaced over months left orphaned entries forever.Layer 2 — ScheduledFutures never cancelled
Each
ServerRoutingStatsEntryholds twoExponentialMovingAverageobjects. Each EMA schedules a periodic auto-decay task viaScheduledExecutorService.scheduleAtFixedRate(). The returnedScheduledFuturewas discarded — not stored, not cancellable. Removing the map entry was not enough to release these tasks.Fix
The eviction is hooked into
BrokerRoutingManager.processInstanceConfigsChange()— the authoritative point where the broker learns a server has left the cluster (Helix disables it).removeServerStats()is safe to call concurrently with in-flight recording: any task that already obtained the entry before removal completes normally; the entry is GC-eligible once all such references drop.Testing Done
testRemoveServerStats— verifies entries are removed from the map afterremoveServerStats(); removing an absent/non-existent server is a no-op; method is a no-op when stats collection is disabledServerRoutingStatsManagerTesttests pass unchangedFixes: OA-1097