RedisCluster: Per-node health tracking and non-blocking initialize()

[ngabhanenetskope]

Summary

Adds two mechanisms to improve RedisCluster resilience during transient node failures under high concurrency:

1. Per-node health tracking (NodeHealthManager) — After N consecutive TimeoutError/ConnectionError on a node (default: 5), subsequent requests to that node fail immediately instead of blocking for socket_timeout. After recovery_time (default: 10s), one probe request is allowed through (half-open pattern). On success, the node is marked available again.

2. Non-blocking initialize() — Changed from blocking lock acquisition to try-lock. If another thread is already refreshing topology, additional threads return immediately using stale (but functional for healthy nodes) topology instead of queuing on _initialization_lock.

Motivation

Under high concurrency (e.g., gevent with 250 greenlets, max_connections_per_node=50, socket_timeout=5s), a single node failure causes:

1. 50 greenlets block on the dead node for 5s each (pool saturates instantly)
2. All 50 timeout simultaneously and call initialize() (thundering herd)
3. All queue on _initialization_lock, serializing 50 redundant CLUSTER SLOTS calls
4. Health check can't respond → readiness probe fails → pod dies

With this fix:

• Health tracking: greenlet RFE: Sharding #6 fails instantly (0ms) instead of blocking 5s → pool never saturates beyond 5 connections
• Non-blocking initialize: only the first thread refreshes topology; others continue serving requests to healthy nodes

See #4074 for full analysis.

Configuration

rc = RedisCluster(
    startup_nodes=[...],
    node_health_failure_threshold=5,   # failures before marking unavailable
    node_health_recovery_time=10.0,    # seconds before allowing probe
)

Behavior

Scenario	Before	After
6th request to dead node	Blocks for socket_timeout (5s)	Instant ConnectionError (0ms)
50 threads calling initialize()	All queue, 50 serial CLUSTER SLOTS calls	1 refreshes, 49 return immediately
Node recovers after failover	Must wait for blocked threads to timeout	Probe succeeds after recovery_time → instant recovery

Tests

• TestNodeHealthManager — 7 tests covering threshold, recovery, per-node isolation, and cluster integration

Related

• Fixes RedisCluster: Transient node failure causes thundering herd on initialize(), leading to pool exhaustion under high concurrency #4074
• Complements PR Randomize cluster startup node order during topology refresh #4060 (randomize nodes in initialize)
• Follow-up to PR Fix ConnectionPool to raise MaxConnectionsError instead of Connection… #3698 (MaxConnectionsError distinction)

---

Note

Medium Risk
Changes cluster command routing and topology refresh concurrency under failure; mis-tuned thresholds could mark healthy nodes unavailable or delay slot updates, but behavior is configurable and scoped to connection failures.

Overview
Adds per-node health tracking to RedisCluster so repeated TimeoutError/ConnectionError on one node (configurable threshold, default 5) marks it unavailable and later commands raise NodeUnavailableError without opening a connection—avoiding pool saturation on a dead peer. Recovery uses a half-open probe after node_health_recovery_time (default 10s); success clears state, failed probe resets the timer.

NodesManager.initialize() now uses a non-blocking _initialization_lock: concurrent refresh callers skip redundant CLUSTER SLOTS work and keep using the current topology until the in-flight refresh finishes.

New public pieces: NodeUnavailableError, NodeHealthManager, and constructor kwargs node_health_failure_threshold / node_health_recovery_time. Tests cover the health manager and fail-fast behavior in _execute_command.

^{Reviewed by Cursor Bugbot for commit b998891. Bugbot is set up for automated code reviews on this repo. Configure here.}

[jit-ci]

Hi, I’m Jit, a friendly security platform designed to help developers build secure applications from day zero with an MVS (Minimal viable security) mindset.

In case there are security findings, they will be communicated to you as a comment inside the PR.

Hope you’ll enjoy using Jit.

Questions? Comments? Want to learn more? Get in touch with us.

[petyaslavova]

Hi @ngabhanenetskope, thank you for your contribution! I'll take a look at it soon.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{Reviewed by Cursor Bugbot for commit 143ef26. Configure here.}

[ngabhanenetskope]

Hi @ngabhanenetskope, thank you for your contribution! I'll take a look at it soon.

@petyaslavova
Did you get a chance to take a look!