Skip to content

Fix OA-1097: evict ServerRoutingStatsManager entries when servers are decommissioned#177

Open
siddharthteotia wants to merge 1 commit into
masterfrom
steotia/oa-1097-server-routing-stats-memory-leak
Open

Fix OA-1097: evict ServerRoutingStatsManager entries when servers are decommissioned#177
siddharthteotia wants to merge 1 commit into
masterfrom
steotia/oa-1097-server-routing-stats-memory-leak

Conversation

@siddharthteotia

Copy link
Copy Markdown
Collaborator

Summary

Fixes a slow memory leak in ServerRoutingStatsManager where stats entries for decommissioned servers accumulated indefinitely on long-running brokers.

Root cause

ServerRoutingStatsManager._serverQueryStatsMap used computeIfAbsent() to create entries on first use but never removed them. The original comment at line 99 acknowledged this: "Entries in this map are never deleted unless the broker process restarts."

The leak was two-layered:

Layer 1 — Map entries never evicted
Every server that ever received a query got a ServerRoutingStatsEntry in the map. Servers decommissioned and replaced over months left orphaned entries forever.

Layer 2 — ScheduledFutures never cancelled
Each ServerRoutingStatsEntry holds two ExponentialMovingAverage objects. Each EMA schedules a periodic auto-decay task via ScheduledExecutorService.scheduleAtFixedRate(). The returned ScheduledFuture was discarded — not stored, not cancellable. Removing the map entry was not enough to release these tasks.

_serverQueryStatsMap
  └─ "Server_host1_8080" → ServerRoutingStatsEntry   ← never removed
        ├─ _inFlightRequestsEMA
        │     └─ ScheduledFuture (decay task, runs every N ms forever)  ← never cancelled
        └─ _latencyMsEMA
              └─ ScheduledFuture (decay task, runs every N ms forever)  ← never cancelled

Fix

ExponentialMovingAverage
  + _decayTaskFuture: ScheduledFuture<?>   (store the future on creation)
  + cancel()                               (cancel the decay task)

ServerRoutingStatsEntry
  + cancel()                               (delegates to both EMA instances)

ServerRoutingStatsManager
  + removeServerStats(instanceId)          (remove from map + cancel entry)

BrokerRoutingManager.processInstanceConfigsChange()
  for each newDisabledServer:
    _enabledServerInstanceMap.remove(...)  (existing)
    _serverRoutingStatsManager             (new)
        .removeServerStats(...)

The eviction is hooked into BrokerRoutingManager.processInstanceConfigsChange() — the authoritative point where the broker learns a server has left the cluster (Helix disables it). removeServerStats() is safe to call concurrently with in-flight recording: any task that already obtained the entry before removal completes normally; the entry is GC-eligible once all such references drop.

Testing Done

  • testRemoveServerStats — verifies entries are removed from the map after removeServerStats(); removing an absent/non-existent server is a no-op; method is a no-op when stats collection is disabled
  • All existing ServerRoutingStatsManagerTest tests pass unchanged

Fixes: OA-1097

… decommissioned

ServerRoutingStatsManager._serverQueryStatsMap accumulated entries forever.
Each ServerRoutingStatsEntry holds two ExponentialMovingAverage objects, each
scheduling a periodic auto-decay task in _periodicTaskExecutor. On long-running
brokers, decommissioned servers left orphaned entries and live ScheduledFutures
that were never cleaned up.

Fix:
- ExponentialMovingAverage: store the ScheduledFuture from scheduleAtFixedRate
  and expose cancel() to stop the periodic decay task.
- ServerRoutingStatsEntry: add cancel() that delegates to both EMA instances.
- ServerRoutingStatsManager: add removeServerStats(instanceId) that atomically
  removes the entry from the map and calls cancel() on it, releasing both the
  map reference and the scheduled tasks so the entry can be GC'd.
- BrokerRoutingManager.processInstanceConfigsChange(): call removeServerStats()
  for each newly-disabled server immediately after removing it from
  _enabledServerInstanceMap, which is the authoritative point where the broker
  learns a server has left the cluster.

Test: testRemoveServerStats verifies that entries are evicted correctly, that
removing an absent server is a safe no-op, and that the method is a no-op when
the manager is disabled.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant