diff --git a/README.md b/README.md index 133ebc5..f971aa9 100644 --- a/README.md +++ b/README.md @@ -721,7 +721,9 @@ the client automatically falls back to the next scope in the chain: For datacenter or rack scoped routing in a multi-datacenter cluster, include a seed host from the target datacenter. The client queries configured seeds with the scope filter and only falls back -after no seed returns nodes for that scope. +after no seed returns nodes for that scope. Configured seed hosts remain discovery candidates, but +they are not added to the routing node list unless `/localnodes` returns them for the selected +scope. ```java // Rack -> Datacenter -> Cluster fallback chain @@ -886,6 +888,61 @@ The default configuration works well for most use cases. Consider adjusting sett - **Long-running connections**: Default settings are optimal; session resumption primarily benefits reconnection scenarios +### Node Health + +The client tracks active, quarantined, and down nodes while routing requests: + +- transport failures count as health failures +- consecutive health failures mark a node down +- down nodes are probed in the background with `GET /localnodes` +- enough consecutive HTTP 200 probe responses move a down node into quarantine +- quarantined nodes are sampled into routing once every configured number of attempts +- enough consecutive successful contacts from a quarantined node promote it back to active + +For DynamoDB API requests, most HTTP responses mean the node was reachable for health purposes, +including application-level errors. Authentication failures are detected from the parsed DynamoDB +error code, such as `InvalidSignatureException`, `MissingAuthenticationTokenException`, or +`UnrecognizedClientException`, because Alternator usually reports these as HTTP 400 responses. +Authentication HTTP statuses (`401` or `403`) and transport failures also count as health failures. +For `/localnodes`, any non-200 HTTP status or transport failure counts as a health failure. +Traffic and `/localnodes` probe failure streaks are tracked separately, so a successful probe does +not clear DynamoDB traffic/auth failures, and a successful DynamoDB request does not clear +`/localnodes` probe failures. + +Key route affinity hashes and votes over all known discovered nodes so candidate order is stable +while health changes. The final request-routing guard normally skips down nodes and admits at most +one quarantined node when the quarantine sampling policy allows verification traffic. If there are +no active nodes, routing fails open across all known nodes without changing their stored health +state. + +The defaults are: 10 consecutive health failures mark a node down, 3 consecutive successful +down-node probes move it into quarantine, 10 consecutive successful quarantined contacts promote it +to active, down nodes are probed every 30 seconds, and quarantined nodes are sampled once every 10 +routing attempts. + +```java +import com.scylladb.alternator.NodeHealthConfig; + +DynamoDbClient client = AlternatorDynamoDbClient.builder() + .endpointOverride(URI.create("https://127.0.0.1:8043")) + .credentialsProvider(myCredentials) + .withNodeHealthConfig(NodeHealthConfig.builder() + .withConsecutiveFailureThreshold(5) + .withDownNodeRecoverySuccessThreshold(3) + .withDownNodeProbePeriodMs(10_000) + .withQuarantineTrafficInterval(20) + .withQuarantineSuccessThreshold(10) + .build()) + .build(); +``` + +For custom integrations, `AlternatorLiveNodes` exposes `reportNodeResult(...)`, +`getActiveNodes()`, `getQuarantinedNodes()`, and `getDownNodes()`. Report routed DynamoDB request +outcomes with `TRAFFIC_SUCCESS` or `TRAFFIC_FAILURE`, and local node-health probe outcomes with +`PROBE_SUCCESS` or `PROBE_FAILURE`. Probe success moves a `DOWN` node to quarantine; sampled traffic +success moves a quarantined node back to active. Node health can be disabled with +`AlternatorConfig.builder().withNodeHealthDisabled()`. + ### Key Route Affinity (LWT Optimization) Key route affinity is an optimization for Lightweight Transactions (LWT) that use Paxos @@ -894,7 +951,7 @@ it reduces Paxos round-trips and improves latency for conditional writes. **Note:** Synchronous clients automatically discover missing partition-key names via `DescribeTable`. Async clients can use key route affinity with pre-configured partition-key -names; requests for tables without pre-configured metadata fall back to round-robin routing. +names; requests for tables without pre-configured metadata fall back to random query-plan routing. #### Quick start @@ -915,7 +972,7 @@ DynamoDbClient client = AlternatorDynamoDbClient.builder() | Mode | Description | |------|-------------| -| `KeyRouteAffinity.NONE` | Default — standard round-robin load balancing | +| `KeyRouteAffinity.NONE` | Default — standard random query-plan load balancing | | `KeyRouteAffinity.RMW` | Optimize read-before-write operations (conditional updates/puts/deletes with `ConditionExpression`, `Expected`, or non-NONE `ReturnValues`) | | `KeyRouteAffinity.ANY_WRITE` | Optimize all write operations (`PutItem`, `UpdateItem`, `DeleteItem`, `BatchWriteItem`) | @@ -949,7 +1006,7 @@ DynamoDbClient client = AlternatorDynamoDbClient.builder() 4. For `BatchWriteItem`, each usable write votes for its preferred node; voted nodes are tried by vote count, then by node URL 5. Non-qualifying operations and requests without usable partition keys continue to use - round-robin load balancing + random query-plan load balancing #### When to use key route affinity diff --git a/src/integration-test/java/com/scylladb/alternator/AlternatorDynamoDbAsyncClientIT.java b/src/integration-test/java/com/scylladb/alternator/AlternatorDynamoDbAsyncClientIT.java index 2e4f68b..13d67a0 100644 --- a/src/integration-test/java/com/scylladb/alternator/AlternatorDynamoDbAsyncClientIT.java +++ b/src/integration-test/java/com/scylladb/alternator/AlternatorDynamoDbAsyncClientIT.java @@ -4,6 +4,7 @@ import static org.junit.Assume.*; import com.scylladb.alternator.internal.AlternatorLiveNodes; +import com.scylladb.alternator.internal.LazyQueryPlan; import com.scylladb.alternator.routing.ClusterScope; import com.scylladb.alternator.routing.DatacenterScope; import com.scylladb.alternator.routing.RackScope; @@ -175,99 +176,123 @@ public void testUpdateLiveNodes() throws Exception { } @Test - public void testNodeDiscoveryWithRoundRobin() throws Exception { + public void testNodeDiscoveryWithRandomQueryPlan() throws Exception { AlternatorDynamoDbAsyncClientWrapper client = buildClient(IntegrationTestConfig.DATACENTER, ""); // Wait for node discovery Thread.sleep(1000); - Set meetNodes = new HashSet<>(); List allNodes = client.getLiveNodes(); + assertFalse("Should have at least one live node", allNodes.isEmpty()); - // Call nextAsURI more times than there are nodes to verify round-robin - for (int i = 0; i < allNodes.size() * 2; i++) { - meetNodes.add(client.nextAsURI()); + Set metNodes = new HashSet<>(); + LazyQueryPlan queryPlan = new LazyQueryPlan(client.getAlternatorLiveNodes()); + while (queryPlan.hasNext()) { + metNodes.add(queryPlan.next()); } - assertEquals("Should visit all nodes via round-robin", allNodes.size(), meetNodes.size()); + assertEquals("Should visit all live nodes via random query plan", new HashSet<>(allNodes), metNodes); client.close(); } @Test public void testDynamoDBOperations() throws Exception { - String tableName = "java_async_client_test_table"; + String tableName = "java_async_client_test_table_" + seedUri.getScheme(); AlternatorDynamoDbAsyncClientWrapper wrapper = buildClient(); DynamoDbAsyncClient client = wrapper.getClient(); - // Clean up if table exists try { + // Clean up if table exists + try { + client.deleteTable(DeleteTableRequest.builder().tableName(tableName).build()).get(); + Thread.sleep(500); + } catch (Exception e) { + // Table doesn't exist, that's fine + } + + // Create table + client + .createTable( + CreateTableRequest.builder() + .tableName(tableName) + .keySchema( + KeySchemaElement.builder().attributeName("ID").keyType(KeyType.HASH).build()) + .attributeDefinitions( + AttributeDefinition.builder() + .attributeName("ID") + .attributeType(ScalarAttributeType.S) + .build()) + .provisionedThroughput( + ProvisionedThroughput.builder() + .readCapacityUnits(1L) + .writeCapacityUnits(1L) + .build()) + .build()) + .get(); + + // Put item + client + .putItem( + PutItemRequest.builder() + .tableName(tableName) + .item( + java.util.Map.of( + "ID", AttributeValue.builder().s("123").build(), + "Name", AttributeValue.builder().s("test-value").build())) + .build()) + .get(); + + GetItemResponse getResult = getItemEventually(client, tableName, "123"); + java.util.Map item = getResult.item(); + + assertNotNull("Should get item back", item); + assertTrue("Should get ID attribute back", item.containsKey("ID")); + assertTrue("Should get Name attribute back", item.containsKey("Name")); + assertEquals("123", item.get("ID").s()); + assertEquals("test-value", item.get("Name").s()); + + // Delete item + client + .deleteItem( + DeleteItemRequest.builder() + .tableName(tableName) + .key(java.util.Map.of("ID", AttributeValue.builder().s("123").build())) + .build()) + .get(); + + // Clean up client.deleteTable(DeleteTableRequest.builder().tableName(tableName).build()).get(); - Thread.sleep(500); - } catch (Exception e) { - // Table doesn't exist, that's fine - } + } finally { + try { + client.deleteTable(DeleteTableRequest.builder().tableName(tableName).build()).get(); + } catch (Exception e) { + // Best effort cleanup + } - // Create table - client - .createTable( - CreateTableRequest.builder() - .tableName(tableName) - .keySchema( - KeySchemaElement.builder().attributeName("ID").keyType(KeyType.HASH).build()) - .attributeDefinitions( - AttributeDefinition.builder() - .attributeName("ID") - .attributeType(ScalarAttributeType.S) - .build()) - .provisionedThroughput( - ProvisionedThroughput.builder() - .readCapacityUnits(1L) - .writeCapacityUnits(1L) - .build()) - .build()) - .get(); - - // Put item - client - .putItem( - PutItemRequest.builder() - .tableName(tableName) - .item( - java.util.Map.of( - "ID", AttributeValue.builder().s("123").build(), - "Name", AttributeValue.builder().s("test-value").build())) - .build()) - .get(); - - // Get item - GetItemResponse getResult = - client - .getItem( - GetItemRequest.builder() - .tableName(tableName) - .key(java.util.Map.of("ID", AttributeValue.builder().s("123").build())) - .build()) - .get(); - - assertNotNull("Should get item back", getResult.item()); - assertEquals("123", getResult.item().get("ID").s()); - assertEquals("test-value", getResult.item().get("Name").s()); - - // Delete item - client - .deleteItem( - DeleteItemRequest.builder() - .tableName(tableName) - .key(java.util.Map.of("ID", AttributeValue.builder().s("123").build())) - .build()) - .get(); - - // Clean up - client.deleteTable(DeleteTableRequest.builder().tableName(tableName).build()).get(); + wrapper.close(); + } + } - wrapper.close(); + private GetItemResponse getItemEventually( + DynamoDbAsyncClient client, String tableName, String id) throws Exception { + GetItemRequest request = + GetItemRequest.builder() + .tableName(tableName) + .key(java.util.Map.of("ID", AttributeValue.builder().s(id).build())) + .consistentRead(true) + .build(); + GetItemResponse response = null; + for (int attempt = 0; attempt < 10; attempt++) { + response = client.getItem(request).get(); + java.util.Map item = response.item(); + if (item != null && item.containsKey("ID") && item.containsKey("Name")) { + return response; + } + Thread.sleep(200); + } + return response; } @Test diff --git a/src/integration-test/java/com/scylladb/alternator/AlternatorDynamoDbClientIT.java b/src/integration-test/java/com/scylladb/alternator/AlternatorDynamoDbClientIT.java index 688e5d2..2db91f7 100644 --- a/src/integration-test/java/com/scylladb/alternator/AlternatorDynamoDbClientIT.java +++ b/src/integration-test/java/com/scylladb/alternator/AlternatorDynamoDbClientIT.java @@ -4,6 +4,7 @@ import static org.junit.Assume.*; import com.scylladb.alternator.internal.AlternatorLiveNodes; +import com.scylladb.alternator.internal.LazyQueryPlan; import com.scylladb.alternator.routing.ClusterScope; import com.scylladb.alternator.routing.DatacenterScope; import com.scylladb.alternator.routing.RackScope; @@ -175,21 +176,22 @@ public void testUpdateLiveNodes() throws Exception { } @Test - public void testNodeDiscoveryWithRoundRobin() throws Exception { + public void testNodeDiscoveryWithRandomQueryPlan() throws Exception { AlternatorDynamoDbClientWrapper client = buildClient(IntegrationTestConfig.DATACENTER, ""); // Wait for node discovery Thread.sleep(1000); - Set meetNodes = new HashSet<>(); List allNodes = client.getLiveNodes(); + assertFalse("Should have at least one live node", allNodes.isEmpty()); - // Call nextAsURI more times than there are nodes to verify round-robin - for (int i = 0; i < allNodes.size() * 2; i++) { - meetNodes.add(client.nextAsURI()); + Set metNodes = new HashSet<>(); + LazyQueryPlan queryPlan = new LazyQueryPlan(client.getAlternatorLiveNodes()); + while (queryPlan.hasNext()) { + metNodes.add(queryPlan.next()); } - assertEquals("Should visit all nodes via round-robin", allNodes.size(), meetNodes.size()); + assertEquals("Should visit all live nodes via random query plan", new HashSet<>(allNodes), metNodes); client.close(); } diff --git a/src/integration-test/java/com/scylladb/alternator/DynamoDbOperationsIT.java b/src/integration-test/java/com/scylladb/alternator/DynamoDbOperationsIT.java index 2898fde..5a2e1cf 100644 --- a/src/integration-test/java/com/scylladb/alternator/DynamoDbOperationsIT.java +++ b/src/integration-test/java/com/scylladb/alternator/DynamoDbOperationsIT.java @@ -262,7 +262,7 @@ public void testOperationsAcrossMultipleNodes() throws Exception { // Create table client.createTable(createTableRequest(tableName)); - // Write many items - requests will be distributed across nodes via round-robin + // Write many items - requests will be distributed across nodes via random query-plan routing int itemCount = 30; for (int i = 0; i < itemCount; i++) { Map item = diff --git a/src/integration-test/java/com/scylladb/alternator/KeyRouteAffinityAutodiscoveryIT.java b/src/integration-test/java/com/scylladb/alternator/KeyRouteAffinityAutodiscoveryIT.java index 5a297e4..5a0082d 100644 --- a/src/integration-test/java/com/scylladb/alternator/KeyRouteAffinityAutodiscoveryIT.java +++ b/src/integration-test/java/com/scylladb/alternator/KeyRouteAffinityAutodiscoveryIT.java @@ -125,7 +125,7 @@ public void testAutodiscoveryFindsPartitionKey() throws Exception { Thread.sleep(2000); // Second write should now benefit from cached PK info. - // If autodiscovery failed, this would still succeed (falls back to round-robin), + // If autodiscovery failed, this would still succeed (falls back to random query-plan routing), // but we verify the data is correct regardless. client.putItem( PutItemRequest.builder() diff --git a/src/integration-test/java/com/scylladb/alternator/internal/ConnectionPoolIT.java b/src/integration-test/java/com/scylladb/alternator/internal/ConnectionPoolIT.java index 7a327cb..31270fc 100644 --- a/src/integration-test/java/com/scylladb/alternator/internal/ConnectionPoolIT.java +++ b/src/integration-test/java/com/scylladb/alternator/internal/ConnectionPoolIT.java @@ -70,21 +70,25 @@ private AlternatorLiveNodes createLiveNodes() { } /** - * Verifies that connections are pooled and reused. After many {@code updateLiveNodes()} calls, + * Verifies that connections are pooled and reused. After many {@code refreshDiscoveredNodes()} calls, * all requests should succeed without hanging (which would indicate connection leaks). */ @Test(timeout = 10_000) public void testConnectionsArePooledAndReused() throws Exception { AlternatorLiveNodes liveNodes = createLiveNodes(); - // Make many requests — enough to round-robin through all nodes multiple times. - // If connections are leaked, this will hang due to pool exhaustion. - for (int i = 0; i < 30; i++) { - liveNodes.updateLiveNodes(); - } + try { + // Make many requests — enough to use all nodes multiple times. + // If connections are leaked, this will hang due to pool exhaustion. + for (int i = 0; i < 30; i++) { + liveNodes.refreshDiscoveredNodes(); + } - int nodeCount = liveNodes.getLiveNodes().size(); - assertTrue("Should have discovered at least one node", nodeCount > 0); + int nodeCount = liveNodes.getDiscoveredNodes().size(); + assertTrue("Should have discovered at least one node", nodeCount > 0); + } finally { + liveNodes.shutdownAndWait(); + } } /** @@ -102,26 +106,30 @@ public void testConnectionsArePooledAndReused() throws Exception { public void testConnectionSurvivesIdlePeriod() throws Exception { AlternatorLiveNodes liveNodes = createLiveNodes(); - // Establish pooled connections by round-robining through all nodes - for (int i = 0; i < 30; i++) { - liveNodes.updateLiveNodes(); - } - int nodesBefore = liveNodes.getLiveNodes().size(); - assertTrue("Should have discovered at least one node", nodesBefore > 0); - - // Let connections sit idle for 10 seconds — enough to verify survival - // without hitting the default 60-second idle reaper - Thread.sleep(10_000); - - // Use the connections again — they should be reused from the pool. - // If connections were dropped, these requests would still succeed - // but would need new connections. If connections are leaked, - // this would hang due to pool exhaustion. - for (int i = 0; i < 30; i++) { - liveNodes.updateLiveNodes(); + try { + // Establish pooled connections by using all nodes + for (int i = 0; i < 30; i++) { + liveNodes.refreshDiscoveredNodes(); + } + int nodesBefore = liveNodes.getDiscoveredNodes().size(); + assertTrue("Should have discovered at least one node", nodesBefore > 0); + + // Let connections sit idle for 10 seconds — enough to verify survival + // without hitting the default 60-second idle reaper + Thread.sleep(10_000); + + // Use the connections again — they should be reused from the pool. + // If connections were dropped, these requests would still succeed + // but would need new connections. If connections are leaked, + // this would hang due to pool exhaustion. + for (int i = 0; i < 30; i++) { + liveNodes.refreshDiscoveredNodes(); + } + int nodesAfter = liveNodes.getDiscoveredNodes().size(); + assertEquals("Node count should remain consistent after idle period", nodesBefore, nodesAfter); + } finally { + liveNodes.shutdownAndWait(); } - int nodesAfter = liveNodes.getLiveNodes().size(); - assertEquals("Node count should remain consistent after idle period", nodesBefore, nodesAfter); } /** @@ -573,56 +581,60 @@ public void testPollingConnectionsStableUnderSs() throws Exception { AlternatorLiveNodes liveNodes = createLiveNodes(); int port = seedUri.getPort(); - // Warm up — establish pooled connections to all discovered nodes - for (int i = 0; i < 10; i++) { - liveNodes.updateLiveNodes(); - } - assertTrue( - "Should have discovered at least one node", liveNodes.getLiveNodes().size() > 0); - // Wait for stale connections from previous tests (reuseForks=true) to drain - // and for the polling client's connection pool to stabilize. - long baseline = countEstablishedConnections(port); - for (int attempt = 0; attempt < 10; attempt++) { - Thread.sleep(1000); - long current = countEstablishedConnections(port); - if (current >= baseline) break; - baseline = current; - } - assertTrue("Should have at least 1 established connection after warmup", baseline >= 1); + try { + // Warm up — establish pooled connections to all discovered nodes + for (int i = 0; i < 10; i++) { + liveNodes.refreshDiscoveredNodes(); + } + assertTrue( + "Should have discovered at least one node", liveNodes.getDiscoveredNodes().size() > 0); + // Wait for stale connections from previous tests (reuseForks=true) to drain + // and for the polling client's connection pool to stabilize. + long baseline = countEstablishedConnections(port); + for (int attempt = 0; attempt < 10; attempt++) { + Thread.sleep(1000); + long current = countEstablishedConnections(port); + if (current >= baseline) break; + baseline = current; + } + assertTrue("Should have at least 1 established connection after warmup", baseline >= 1); - // Perform many more polling cycles — connection count should stay bounded - for (int i = 0; i < 50; i++) { - liveNodes.updateLiveNodes(); - } + // Perform many more polling cycles — connection count should stay bounded + for (int i = 0; i < 50; i++) { + liveNodes.refreshDiscoveredNodes(); + } - long afterBulk = countEstablishedConnections(port); - assertTrue( - "Polling connections should not grow significantly during 50 polling cycles" - + " (baseline=" - + baseline - + ", after=" - + afterBulk - + ")", - afterBulk <= baseline * 1.5); - - // Verify connections survive a short idle period - Thread.sleep(3_000); - - for (int i = 0; i < 10; i++) { - liveNodes.updateLiveNodes(); - } + long afterBulk = countEstablishedConnections(port); + assertTrue( + "Polling connections should not grow significantly during 50 polling cycles" + + " (baseline=" + + baseline + + ", after=" + + afterBulk + + ")", + afterBulk <= baseline * 1.5); + + // Verify connections survive a short idle period + Thread.sleep(3_000); - long afterIdle = countEstablishedConnections(port); - long minAcceptable = Math.max(1, baseline / 3); - assertTrue( - "Polling connections should survive 3s idle period" - + " (baseline=" - + baseline - + ", after=" - + afterIdle - + ", minAcceptable=" - + minAcceptable - + ")", - afterIdle >= minAcceptable); + for (int i = 0; i < 10; i++) { + liveNodes.refreshDiscoveredNodes(); + } + + long afterIdle = countEstablishedConnections(port); + long minAcceptable = Math.max(1, baseline / 3); + assertTrue( + "Polling connections should survive 3s idle period" + + " (baseline=" + + baseline + + ", after=" + + afterIdle + + ", minAcceptable=" + + minAcceptable + + ")", + afterIdle >= minAcceptable); + } finally { + liveNodes.shutdownAndWait(); + } } } diff --git a/src/main/java/com/scylladb/alternator/AlternatorConfig.java b/src/main/java/com/scylladb/alternator/AlternatorConfig.java index 96ec073..3178cac 100644 --- a/src/main/java/com/scylladb/alternator/AlternatorConfig.java +++ b/src/main/java/com/scylladb/alternator/AlternatorConfig.java @@ -208,9 +208,10 @@ public class AlternatorConfig { private final long connectionTimeToLiveMs; private final long connectionAcquisitionTimeoutMs; private final long connectionTimeoutMs; + private final NodeHealthConfig nodeHealthConfig; /** - * Package-private constructor. Use {@link AlternatorConfig#builder()} to create instances. + * Protected constructor kept for source and binary compatibility with 2.0.x subclasses. * * @param seedHosts the list of seed host addresses (IP or hostname) for cluster discovery * @param scheme the URI scheme (http or https) @@ -259,6 +260,83 @@ protected AlternatorConfig( long connectionTimeToLiveMs, long connectionAcquisitionTimeoutMs, long connectionTimeoutMs) { + this( + seedHosts, + scheme, + port, + routingScope, + compressionAlgorithm, + minCompressionSizeBytes, + responseCompressionAlgorithms, + optimizeHeaders, + headersWhitelist, + userAgentEnabled, + authenticationEnabled, + tlsSessionCacheConfig, + tlsConfig, + keyRouteAffinityConfig, + activeRefreshIntervalMs, + idleRefreshIntervalMs, + maxConnections, + connectionMaxIdleTimeMs, + connectionTimeToLiveMs, + connectionAcquisitionTimeoutMs, + connectionTimeoutMs, + NodeHealthConfig.getDefault()); + } + + /** + * Private constructor. Use {@link AlternatorConfig#builder()} to create instances. + * + * @param seedHosts the list of seed host addresses (IP or hostname) for cluster discovery + * @param scheme the URI scheme (http or https) + * @param port the port number + * @param routingScope the routing scope for node targeting + * @param compressionAlgorithm the compression algorithm to use + * @param minCompressionSizeBytes minimum request size in bytes to trigger compression + * @param responseCompressionAlgorithms response compression algorithms to advertise and decode + * @param optimizeHeaders whether to enable HTTP header optimization + * @param headersWhitelist the set of headers to preserve when optimization is enabled + * @param userAgentEnabled whether user-agent reporting is enabled + * @param authenticationEnabled whether authentication is enabled + * @param tlsSessionCacheConfig the TLS session cache configuration for quick TLS renegotiation + * @param tlsConfig the TLS configuration including CA certificates and verification settings + * @param keyRouteAffinityConfig the key route affinity configuration + * @param activeRefreshIntervalMs refresh interval when there are active requests + * @param idleRefreshIntervalMs refresh interval when the client is idle + * @param maxConnections maximum number of connections in the HTTP client pool + * @param connectionMaxIdleTimeMs maximum idle time for pooled connections in milliseconds + * @param connectionTimeToLiveMs maximum lifetime for pooled connections in milliseconds (0 = + * unlimited) + * @param connectionAcquisitionTimeoutMs maximum time to wait for a connection from the pool in + * milliseconds + * @param connectionTimeoutMs maximum time to wait for a connection to be established in + * milliseconds + * @param nodeHealthConfig configuration for node health tracking + */ + private AlternatorConfig( + List seedHosts, + String scheme, + int port, + RoutingScope routingScope, + RequestCompressionAlgorithm compressionAlgorithm, + int minCompressionSizeBytes, + List responseCompressionAlgorithms, + boolean optimizeHeaders, + Set headersWhitelist, + boolean userAgentEnabled, + boolean authenticationEnabled, + TlsSessionCacheConfig tlsSessionCacheConfig, + TlsConfig tlsConfig, + KeyRouteAffinityConfig keyRouteAffinityConfig, + long activeRefreshIntervalMs, + long idleRefreshIntervalMs, + int maxConnections, + long connectionMaxIdleTimeMs, + long connectionTimeToLiveMs, + long connectionAcquisitionTimeoutMs, + long connectionTimeoutMs, + NodeHealthConfig nodeHealthConfig) { this.seedHosts = seedHosts != null ? Collections.unmodifiableList(new ArrayList<>(seedHosts)) @@ -300,6 +378,8 @@ protected AlternatorConfig( this.connectionTimeToLiveMs = connectionTimeToLiveMs; this.connectionAcquisitionTimeoutMs = connectionAcquisitionTimeoutMs; this.connectionTimeoutMs = connectionTimeoutMs; + this.nodeHealthConfig = + nodeHealthConfig != null ? nodeHealthConfig : NodeHealthConfig.getDefault(); } /** @@ -594,6 +674,16 @@ public long getConnectionTimeoutMs() { return connectionTimeoutMs; } + /** + * Gets the node health tracking configuration. + * + * @return the node health configuration, never null + * @since 2.0.6 + */ + public NodeHealthConfig getNodeHealthConfig() { + return nodeHealthConfig; + } + /** * Returns the set of HTTP headers required for this configuration. * @@ -654,6 +744,7 @@ public static class Builder { private long connectionTimeToLiveMs = DEFAULT_CONNECTION_TIME_TO_LIVE_MS; private long connectionAcquisitionTimeoutMs = DEFAULT_CONNECTION_ACQUISITION_TIMEOUT_MS; private long connectionTimeoutMs = DEFAULT_CONNECTION_TIMEOUT_MS; + private NodeHealthConfig nodeHealthConfig = NodeHealthConfig.getDefault(); /** Package-private constructor. Use {@link AlternatorConfig#builder()} to create instances. */ Builder() {} @@ -1268,6 +1359,30 @@ public Builder withConnectionTimeoutMs(long connectionTimeoutMs) { return this; } + /** + * Sets node health tracking configuration. + * + * @param nodeHealthConfig node health configuration, or null to use defaults + * @return this builder instance + * @since 2.0.6 + */ + public Builder withNodeHealthConfig(NodeHealthConfig nodeHealthConfig) { + this.nodeHealthConfig = + nodeHealthConfig != null ? nodeHealthConfig : NodeHealthConfig.getDefault(); + return this; + } + + /** + * Disables node health tracking. + * + * @return this builder instance + * @since 2.0.6 + */ + public Builder withNodeHealthDisabled() { + this.nodeHealthConfig = NodeHealthConfig.disabled(); + return this; + } + /** * Builds and returns an {@link AlternatorConfig} instance with the configured settings. * @@ -1381,7 +1496,8 @@ public AlternatorConfig build() { connectionMaxIdleTimeMs, connectionTimeToLiveMs, connectionAcquisitionTimeoutMs, - connectionTimeoutMs); + connectionTimeoutMs, + nodeHealthConfig); } } } diff --git a/src/main/java/com/scylladb/alternator/AlternatorDynamoDbAsyncClient.java b/src/main/java/com/scylladb/alternator/AlternatorDynamoDbAsyncClient.java index 15df7f7..deeda52 100644 --- a/src/main/java/com/scylladb/alternator/AlternatorDynamoDbAsyncClient.java +++ b/src/main/java/com/scylladb/alternator/AlternatorDynamoDbAsyncClient.java @@ -58,7 +58,6 @@ * * // Access Alternator-specific functionality * List nodes = client.getLiveNodes(); - * URI nextNode = client.nextAsURI(); * } * * @author dmitry.kropachev @@ -371,6 +370,30 @@ public AlternatorDynamoDbAsyncClientBuilder withIdleRefreshIntervalMs(long inter return this; } + /** + * Sets the node health tracking configuration. + * + * @param nodeHealthConfig the node health configuration, or null to use defaults + * @return this builder instance + * @since 2.0.6 + */ + public AlternatorDynamoDbAsyncClientBuilder withNodeHealthConfig( + NodeHealthConfig nodeHealthConfig) { + configBuilder.withNodeHealthConfig(nodeHealthConfig); + return this; + } + + /** + * Disables node health tracking. + * + * @return this builder instance + * @since 2.0.6 + */ + public AlternatorDynamoDbAsyncClientBuilder withNodeHealthDisabled() { + configBuilder.withNodeHealthDisabled(); + return this; + } + /** * Sets the maximum number of connections in the HTTP client connection pool. * @@ -470,6 +493,7 @@ public AlternatorDynamoDbAsyncClientBuilder withAlternatorConfig(AlternatorConfi configBuilder.withKeyRouteAffinity(config.getKeyRouteAffinityConfig()); configBuilder.withActiveRefreshIntervalMs(config.getActiveRefreshIntervalMs()); configBuilder.withIdleRefreshIntervalMs(config.getIdleRefreshIntervalMs()); + configBuilder.withNodeHealthConfig(config.getNodeHealthConfig()); configBuilder.withMaxConnections(config.getMaxConnections()); configBuilder.withConnectionMaxIdleTimeMs(config.getConnectionMaxIdleTimeMs()); configBuilder.withConnectionTimeToLiveMs(config.getConnectionTimeToLiveMs()); diff --git a/src/main/java/com/scylladb/alternator/AlternatorDynamoDbAsyncClientWrapper.java b/src/main/java/com/scylladb/alternator/AlternatorDynamoDbAsyncClientWrapper.java index a3752dd..0af501e 100644 --- a/src/main/java/com/scylladb/alternator/AlternatorDynamoDbAsyncClientWrapper.java +++ b/src/main/java/com/scylladb/alternator/AlternatorDynamoDbAsyncClientWrapper.java @@ -28,7 +28,6 @@ * * // Access Alternator-specific functionality * List nodes = wrapper.getLiveNodes(); - * URI nextNode = wrapper.nextAsURI(); * } * * @author dmitry.kropachev @@ -116,19 +115,35 @@ public DynamoDbAsyncClient getClient() { } /** - * Returns a snapshot of the current live nodes list. + * Returns a snapshot of the current discovered nodes list. * - * @return an unmodifiable list of the current live node URIs + * @return an unmodifiable list of the current discovered node URIs + * @since 2.1.0 + */ + public List getDiscoveredNodes() { + return liveNodes.getDiscoveredNodes(); + } + + /** + * Returns a snapshot of discovered nodes currently active for normal routing. + * + * @return an unmodifiable list of active discovered node URIs */ public List getLiveNodes() { return liveNodes.getLiveNodes(); } /** - * Returns the next node URI using round-robin selection. + * Returns the next node URI using the current query-plan selection. + * + *

This method is retained for source and binary compatibility with callers compiled against + * versions that exposed direct node selection. * - * @return the next {@link URI} in the round-robin sequence + * @return a selected node URI + * @deprecated Use normal DynamoDB client operations for routing, or {@link + * #getAlternatorLiveNodes()} with {@code LazyQueryPlan} for custom routing. */ + @Deprecated public URI nextAsURI() { return liveNodes.nextAsURI(); } diff --git a/src/main/java/com/scylladb/alternator/AlternatorDynamoDbClient.java b/src/main/java/com/scylladb/alternator/AlternatorDynamoDbClient.java index 632ebe1..246baa9 100644 --- a/src/main/java/com/scylladb/alternator/AlternatorDynamoDbClient.java +++ b/src/main/java/com/scylladb/alternator/AlternatorDynamoDbClient.java @@ -55,7 +55,6 @@ * * // Access Alternator-specific functionality * List nodes = client.getLiveNodes(); - * URI nextNode = client.nextAsURI(); * } * * @author dmitry.kropachev @@ -408,6 +407,29 @@ public AlternatorDynamoDbClientBuilder withIdleRefreshIntervalMs(long intervalMs return this; } + /** + * Sets the node health tracking configuration. + * + * @param nodeHealthConfig the node health configuration, or null to use defaults + * @return this builder instance + * @since 2.0.6 + */ + public AlternatorDynamoDbClientBuilder withNodeHealthConfig(NodeHealthConfig nodeHealthConfig) { + configBuilder.withNodeHealthConfig(nodeHealthConfig); + return this; + } + + /** + * Disables node health tracking. + * + * @return this builder instance + * @since 2.0.6 + */ + public AlternatorDynamoDbClientBuilder withNodeHealthDisabled() { + configBuilder.withNodeHealthDisabled(); + return this; + } + /** * Sets the maximum number of connections in the HTTP client connection pool. * @@ -507,6 +529,7 @@ public AlternatorDynamoDbClientBuilder withAlternatorConfig(AlternatorConfig con configBuilder.withKeyRouteAffinity(config.getKeyRouteAffinityConfig()); configBuilder.withActiveRefreshIntervalMs(config.getActiveRefreshIntervalMs()); configBuilder.withIdleRefreshIntervalMs(config.getIdleRefreshIntervalMs()); + configBuilder.withNodeHealthConfig(config.getNodeHealthConfig()); configBuilder.withMaxConnections(config.getMaxConnections()); configBuilder.withConnectionMaxIdleTimeMs(config.getConnectionMaxIdleTimeMs()); configBuilder.withConnectionTimeToLiveMs(config.getConnectionTimeToLiveMs()); diff --git a/src/main/java/com/scylladb/alternator/AlternatorDynamoDbClientWrapper.java b/src/main/java/com/scylladb/alternator/AlternatorDynamoDbClientWrapper.java index ad2e852..27abe9c 100644 --- a/src/main/java/com/scylladb/alternator/AlternatorDynamoDbClientWrapper.java +++ b/src/main/java/com/scylladb/alternator/AlternatorDynamoDbClientWrapper.java @@ -28,7 +28,6 @@ * * // Access Alternator-specific functionality * List nodes = wrapper.getLiveNodes(); - * URI nextNode = wrapper.nextAsURI(); * } * * @author dmitry.kropachev @@ -119,19 +118,35 @@ public DynamoDbClient getClient() { } /** - * Returns a snapshot of the current live nodes list. + * Returns a snapshot of the current discovered nodes list. * - * @return an unmodifiable list of the current live node URIs + * @return an unmodifiable list of the current discovered node URIs + * @since 2.1.0 + */ + public List getDiscoveredNodes() { + return liveNodes.getDiscoveredNodes(); + } + + /** + * Returns a snapshot of discovered nodes currently active for normal routing. + * + * @return an unmodifiable list of active discovered node URIs */ public List getLiveNodes() { return liveNodes.getLiveNodes(); } /** - * Returns the next node URI using round-robin selection. + * Returns the next node URI using the current query-plan selection. + * + *

This method is retained for source and binary compatibility with callers compiled against + * versions that exposed direct node selection. * - * @return the next {@link URI} in the round-robin sequence + * @return a selected node URI + * @deprecated Use normal DynamoDB client operations for routing, or {@link + * #getAlternatorLiveNodes()} with {@code LazyQueryPlan} for custom routing. */ + @Deprecated public URI nextAsURI() { return liveNodes.nextAsURI(); } diff --git a/src/main/java/com/scylladb/alternator/NodeHealthConfig.java b/src/main/java/com/scylladb/alternator/NodeHealthConfig.java new file mode 100644 index 0000000..7380d53 --- /dev/null +++ b/src/main/java/com/scylladb/alternator/NodeHealthConfig.java @@ -0,0 +1,217 @@ +package com.scylladb.alternator; + +/** Configuration for Alternator node health tracking. */ +public final class NodeHealthConfig { + /** Default consecutive health failure threshold before a node is marked down. */ + public static final int DEFAULT_CONSECUTIVE_FAILURE_THRESHOLD = 10; + + /** Default consecutive successful down-node probes before a node enters quarantine. */ + public static final int DEFAULT_DOWN_NODE_RECOVERY_SUCCESS_THRESHOLD = 3; + + /** Default consecutive successful-contact threshold before a quarantined node is promoted. */ + public static final int DEFAULT_QUARANTINE_SUCCESS_THRESHOLD = 10; + + /** Default background period for probing down nodes. */ + public static final long DEFAULT_DOWN_NODE_PROBE_PERIOD_MS = 30_000; + + /** Default routing-attempt interval for sampling quarantined nodes. */ + public static final int DEFAULT_QUARANTINE_TRAFFIC_INTERVAL = 10; + + private final int consecutiveFailureThreshold; + private final int downNodeRecoverySuccessThreshold; + private final int quarantineSuccessThreshold; + private final long downNodeProbePeriodMs; + private final int quarantineTrafficInterval; + private final boolean disabled; + + private NodeHealthConfig( + int consecutiveFailureThreshold, + int downNodeRecoverySuccessThreshold, + int quarantineSuccessThreshold, + long downNodeProbePeriodMs, + int quarantineTrafficInterval, + boolean disabled) { + this.consecutiveFailureThreshold = Math.max(1, consecutiveFailureThreshold); + this.downNodeRecoverySuccessThreshold = Math.max(1, downNodeRecoverySuccessThreshold); + this.quarantineSuccessThreshold = Math.max(1, quarantineSuccessThreshold); + this.downNodeProbePeriodMs = downNodeProbePeriodMs; + this.quarantineTrafficInterval = Math.max(1, quarantineTrafficInterval); + this.disabled = disabled; + } + + /** + * Returns the default node health configuration. + * + * @return default configuration + */ + public static NodeHealthConfig getDefault() { + return builder().build(); + } + + /** + * Returns a configuration with health tracking disabled. + * + * @return disabled configuration + */ + public static NodeHealthConfig disabled() { + return builder().withDisabled(true).build(); + } + + /** + * Creates a new builder. + * + * @return a builder + */ + public static Builder builder() { + return new Builder(); + } + + /** + * Returns the consecutive health failure threshold. + * + * @return threshold + */ + public int getConsecutiveFailureThreshold() { + return consecutiveFailureThreshold; + } + + /** + * Returns the consecutive successful down-node probe threshold for entering quarantine. + * + * @return threshold + */ + public int getDownNodeRecoverySuccessThreshold() { + return downNodeRecoverySuccessThreshold; + } + + /** + * Returns the consecutive successful-contact promotion threshold. + * + * @return threshold + */ + public int getQuarantineSuccessThreshold() { + return quarantineSuccessThreshold; + } + + /** + * Returns the background period for probing down nodes. + * + * @return probe period in milliseconds; non-positive disables background probing + */ + public long getDownNodeProbePeriodMs() { + return downNodeProbePeriodMs; + } + + /** + * Returns the quarantine sampling interval. + * + * @return routing-attempt interval + */ + public int getQuarantineTrafficInterval() { + return quarantineTrafficInterval; + } + + /** + * Returns whether node health tracking is disabled. + * + * @return true when disabled + */ + public boolean isDisabled() { + return disabled; + } + + /** Builder for {@link NodeHealthConfig}. */ + public static final class Builder { + private int consecutiveFailureThreshold = DEFAULT_CONSECUTIVE_FAILURE_THRESHOLD; + private int downNodeRecoverySuccessThreshold = DEFAULT_DOWN_NODE_RECOVERY_SUCCESS_THRESHOLD; + private int quarantineSuccessThreshold = DEFAULT_QUARANTINE_SUCCESS_THRESHOLD; + private long downNodeProbePeriodMs = DEFAULT_DOWN_NODE_PROBE_PERIOD_MS; + private int quarantineTrafficInterval = DEFAULT_QUARANTINE_TRAFFIC_INTERVAL; + private boolean disabled = false; + + private Builder() {} + + /** + * Sets the consecutive health failure threshold. Values less than one are normalized to one. + * + * @param threshold threshold value + * @return this builder + */ + public Builder withConsecutiveFailureThreshold(int threshold) { + this.consecutiveFailureThreshold = threshold; + return this; + } + + /** + * Sets the consecutive successful down-node probe threshold for entering quarantine. Values + * less than one are normalized to one. + * + * @param threshold threshold value + * @return this builder + */ + public Builder withDownNodeRecoverySuccessThreshold(int threshold) { + this.downNodeRecoverySuccessThreshold = threshold; + return this; + } + + /** + * Sets the consecutive successful-contact promotion threshold. Values less than one are + * normalized to one. + * + * @param threshold threshold value + * @return this builder + */ + public Builder withQuarantineSuccessThreshold(int threshold) { + this.quarantineSuccessThreshold = threshold; + return this; + } + + /** + * Sets the background period for probing down nodes. + * + * @param periodMs period in milliseconds; non-positive disables background probing + * @return this builder + */ + public Builder withDownNodeProbePeriodMs(long periodMs) { + this.downNodeProbePeriodMs = periodMs; + return this; + } + + /** + * Sets the quarantine sampling interval. Values less than one are normalized to one. + * + * @param interval routing-attempt interval + * @return this builder + */ + public Builder withQuarantineTrafficInterval(int interval) { + this.quarantineTrafficInterval = interval; + return this; + } + + /** + * Enables or disables node health tracking. + * + * @param disabled true to disable health tracking + * @return this builder + */ + public Builder withDisabled(boolean disabled) { + this.disabled = disabled; + return this; + } + + /** + * Builds a node health configuration. + * + * @return node health configuration + */ + public NodeHealthConfig build() { + return new NodeHealthConfig( + consecutiveFailureThreshold, + downNodeRecoverySuccessThreshold, + quarantineSuccessThreshold, + downNodeProbePeriodMs, + quarantineTrafficInterval, + disabled); + } + } +} diff --git a/src/main/java/com/scylladb/alternator/NodeHealthObservation.java b/src/main/java/com/scylladb/alternator/NodeHealthObservation.java new file mode 100644 index 0000000..ac36206 --- /dev/null +++ b/src/main/java/com/scylladb/alternator/NodeHealthObservation.java @@ -0,0 +1,16 @@ +package com.scylladb.alternator; + +/** Result observed for an Alternator node health decision. */ +public enum NodeHealthObservation { + /** A routed DynamoDB request completed without a node-health failure. */ + TRAFFIC_SUCCESS, + + /** A routed DynamoDB request hit a transport or authentication health failure. */ + TRAFFIC_FAILURE, + + /** A local node-health probe, such as {@code GET /localnodes}, completed successfully. */ + PROBE_SUCCESS, + + /** A local node-health probe failed or returned an unhealthy response. */ + PROBE_FAILURE +} diff --git a/src/main/java/com/scylladb/alternator/NodeHealthState.java b/src/main/java/com/scylladb/alternator/NodeHealthState.java new file mode 100644 index 0000000..bd7dfb4 --- /dev/null +++ b/src/main/java/com/scylladb/alternator/NodeHealthState.java @@ -0,0 +1,13 @@ +package com.scylladb.alternator; + +/** Current routing state for an Alternator node. */ +public enum NodeHealthState { + /** Node receives normal traffic. */ + ACTIVE, + + /** Node recovered from down state and receives sampled verification traffic. */ + QUARANTINED, + + /** Node is excluded from normal routing and is probed in the background. */ + DOWN +} diff --git a/src/main/java/com/scylladb/alternator/NodeHealthStatus.java b/src/main/java/com/scylladb/alternator/NodeHealthStatus.java new file mode 100644 index 0000000..31878b8 --- /dev/null +++ b/src/main/java/com/scylladb/alternator/NodeHealthStatus.java @@ -0,0 +1,64 @@ +package com.scylladb.alternator; + +/** Snapshot of one node's health state. */ +public final class NodeHealthStatus { + private final NodeHealthState state; + private final int consecutiveFailures; + private final int consecutiveSuccesses; + private final long updatedAtNanos; + + /** + * Creates a node health status snapshot. + * + * @param state current node state + * @param consecutiveFailures consecutive health failure observations + * @param consecutiveSuccesses consecutive successful observations in the current recovery state + * @param updatedAtNanos monotonic timestamp from {@link System#nanoTime()} + */ + public NodeHealthStatus( + NodeHealthState state, + int consecutiveFailures, + int consecutiveSuccesses, + long updatedAtNanos) { + this.state = state; + this.consecutiveFailures = consecutiveFailures; + this.consecutiveSuccesses = consecutiveSuccesses; + this.updatedAtNanos = updatedAtNanos; + } + + /** + * Returns the node state. + * + * @return the node state + */ + public NodeHealthState getState() { + return state; + } + + /** + * Returns consecutive health failure observations. + * + * @return consecutive health failure observations + */ + public int getConsecutiveFailures() { + return consecutiveFailures; + } + + /** + * Returns consecutive successful observations in the current recovery state. + * + * @return consecutive successful observations in the current recovery state + */ + public int getConsecutiveSuccesses() { + return consecutiveSuccesses; + } + + /** + * Returns the monotonic update timestamp. + * + * @return timestamp from {@link System#nanoTime()} + */ + public long getUpdatedAtNanos() { + return updatedAtNanos; + } +} diff --git a/src/main/java/com/scylladb/alternator/internal/AlternatorLiveNodes.java b/src/main/java/com/scylladb/alternator/internal/AlternatorLiveNodes.java index 99b1865..efea2c5 100644 --- a/src/main/java/com/scylladb/alternator/internal/AlternatorLiveNodes.java +++ b/src/main/java/com/scylladb/alternator/internal/AlternatorLiveNodes.java @@ -1,6 +1,9 @@ package com.scylladb.alternator.internal; import com.scylladb.alternator.AlternatorConfig; +import com.scylladb.alternator.NodeHealthObservation; +import com.scylladb.alternator.NodeHealthState; +import com.scylladb.alternator.NodeHealthStatus; import com.scylladb.alternator.routing.ClusterScope; import com.scylladb.alternator.routing.DatacenterScope; import com.scylladb.alternator.routing.RackScope; @@ -13,7 +16,6 @@ import java.net.URISyntaxException; import java.util.*; import java.util.concurrent.atomic.AtomicBoolean; -import java.util.concurrent.atomic.AtomicInteger; import java.util.concurrent.atomic.AtomicLong; import java.util.concurrent.atomic.AtomicReference; import java.util.logging.Level; @@ -28,11 +30,9 @@ import software.amazon.awssdk.http.SdkHttpRequest; /** - * Maintains and automatically updates a list of known live Alternator nodes. Live Alternator nodes - * should answer alternatorScheme (http or https) requests on port alternatorPort. One of these - * livenodes will be used, at round-robin order, for every connection. The list of live nodes starts - * with one or more known nodes, but then a thread periodically replaces this list by an up-to-date - * list retrieved from making a "/localnodes" requests to one of these nodes. + * Maintains and automatically updates a list of discovered Alternator nodes in the configured + * routing scope. Node health is tracked separately from discovery; query plans start from + * discovered nodes and apply health rules only when choosing nodes for routing. * * @author dmitry.kropachev */ @@ -41,15 +41,29 @@ public class AlternatorLiveNodes extends Thread { private final String alternatorScheme; private final int alternatorPort; - private final AtomicReference> liveNodes; + + /** + * Nodes discovered for the configured routing scope. + * + *

This is not a health-filtered live-node list. Health state lives in {@link #healthStore}; + * down and quarantined nodes can remain here so key-affinity hashing uses a stable scoped node + * ring. Initial seed nodes are kept separately and used as discovery fallbacks without being + * published into this routing ring unless they are returned by discovery for the configured + * scope. Query plans later skip down nodes and sample quarantined nodes according to the + * configured quarantine interval. + */ + private final AtomicReference> discoveredNodes; + private final List initialNodes; - private final AtomicInteger nextLiveNodeIndex; private final AlternatorConfig config; private final AtomicBoolean running = new AtomicBoolean(false); private final AtomicBoolean shutdownRequested = new AtomicBoolean(false); + private final AtomicBoolean pollingClientClosed = new AtomicBoolean(false); private final SdkHttpClient pollingHttpClient; private final boolean ownsPollingClient; private final AtomicLong lastActivityTime = new AtomicLong(0); + private final NodeHealthStore healthStore; + private final AtomicLong quarantineSamplingCounter = new AtomicLong(0); private static Logger logger = Logger.getLogger(AlternatorLiveNodes.class.getName()); @@ -59,24 +73,49 @@ public void run() { logger.log(Level.INFO, "AlternatorLiveNodes thread started"); running.set(true); try { + long nextRefreshAt = 0; + long probePeriodMs = config.getNodeHealthConfig().getDownNodeProbePeriodMs(); + long nextDownProbeAt = + probePeriodMs > 0 ? System.currentTimeMillis() + probePeriodMs : Long.MAX_VALUE; while (!shutdownRequested.get()) { - try { - updateLiveNodes(); - } catch (IOException e) { - if (shutdownRequested.get()) { - logger.log(Level.FINE, "AlternatorLiveNodes polling stopped during shutdown", e); - return; + long now = System.currentTimeMillis(); + if (now >= nextRefreshAt) { + try { + refreshDiscoveredNodes(); + } catch (IOException e) { + if (shutdownRequested.get()) { + logger.log(Level.FINE, "AlternatorLiveNodes polling stopped during shutdown", e); + return; + } + logger.log(Level.SEVERE, "AlternatorLiveNodes failed to sync nodes list", e); + } catch (RuntimeException e) { + if (shutdownRequested.get()) { + logger.log(Level.FINE, "AlternatorLiveNodes polling stopped during shutdown", e); + return; + } + logger.log(Level.SEVERE, "AlternatorLiveNodes polling failed unexpectedly", e); + } finally { + nextRefreshAt = System.currentTimeMillis() + getRefreshInterval(); } - logger.log(Level.SEVERE, "AlternatorLiveNodes failed to sync nodes list", e); - } catch (RuntimeException e) { - if (shutdownRequested.get()) { - logger.log(Level.FINE, "AlternatorLiveNodes polling stopped during shutdown", e); - return; + } + if (probePeriodMs > 0 && now >= nextDownProbeAt) { + try { + runDownNodeProbes(); + } catch (RuntimeException e) { + if (shutdownRequested.get()) { + logger.log( + Level.FINE, "AlternatorLiveNodes down-node probing stopped during shutdown", e); + return; + } + logger.log(Level.SEVERE, "AlternatorLiveNodes down-node probing failed", e); + } finally { + nextDownProbeAt = System.currentTimeMillis() + probePeriodMs; } - logger.log(Level.SEVERE, "AlternatorLiveNodes polling failed unexpectedly", e); } try { - Thread.sleep(getRefreshInterval()); + long wakeAt = Math.min(nextRefreshAt, nextDownProbeAt); + long sleepMs = Math.max(1, wakeAt - System.currentTimeMillis()); + Thread.sleep(sleepMs); } catch (InterruptedException e) { if (shutdownRequested.get()) { logger.log(Level.INFO, "AlternatorLiveNodes thread interrupted and stopping"); @@ -84,6 +123,7 @@ public void run() { return; } logger.log(Level.FINE, "AlternatorLiveNodes thread interrupted without shutdown request"); + nextRefreshAt = 0; } } } finally { @@ -95,7 +135,9 @@ public void run() { /** Closes the polling HTTP client if this instance owns it. */ private void closePollingClient() { - if (ownsPollingClient && pollingHttpClient != null) { + if (ownsPollingClient + && pollingHttpClient != null + && pollingClientClosed.compareAndSet(false, true)) { pollingHttpClient.close(); } } @@ -136,11 +178,19 @@ public boolean shutdownAndWait(long timeoutMs) { return false; } if (timeoutMs <= 0) { - return !isAlive(); + boolean stopped = !isAlive(); + if (stopped) { + closePollingClient(); + } + return stopped; } try { join(timeoutMs); - return !isAlive(); + boolean stopped = !isAlive(); + if (stopped) { + closePollingClient(); + } + return stopped; } catch (InterruptedException e) { Thread.currentThread().interrupt(); return false; @@ -236,7 +286,8 @@ private static AlternatorConfig configWithSeedUri(URI seedUri, AlternatorConfig .withCompressionAlgorithm(config.getCompressionAlgorithm()) .withMinCompressionSizeBytes(config.getMinCompressionSizeBytes()) .withOptimizeHeaders(config.isOptimizeHeaders()) - .withHeadersWhitelist(config.getHeadersWhitelist()); + .withHeadersWhitelist(config.getHeadersWhitelist()) + .withNodeHealthConfig(config.getNodeHealthConfig()); if (config.isResponseCompressionEnabled()) { builder.withResponseCompression(config.getResponseCompressionAlgorithms()); } else { @@ -361,18 +412,18 @@ private AlternatorLiveNodes( } this.alternatorScheme = config.getScheme(); this.alternatorPort = config.getPort(); - this.initialNodes = hostsToUris(seedHosts); - this.liveNodes = new AtomicReference<>(); - this.nextLiveNodeIndex = new AtomicInteger(0); + this.initialNodes = dedupePreservingOrder(hostsToUris(seedHosts)); + this.discoveredNodes = new AtomicReference<>(); this.config = config; this.pollingHttpClient = pollingHttpClient; this.ownsPollingClient = ownsPollingClient; + this.healthStore = new NodeHealthStore(config.getNodeHealthConfig(), initialNodes); try { this.validate(); } catch (ValidationError e) { throw new RuntimeException(e); } - this.liveNodes.set(initialNodes); + this.discoveredNodes.set(initialNodes); } /** @@ -477,33 +528,41 @@ private List hostsToUris(List hosts) { } /** - * nextAsURI. + * Returns the next node URI using the current query-plan selection. + * + *

This method is retained for source and binary compatibility with callers compiled against + * versions that exposed direct node selection. New routing code should use {@link LazyQueryPlan} + * so request retries get a per-request plan. * - * @return a {@link java.net.URI} object + * @return a selected node URI + * @deprecated Use {@link LazyQueryPlan} for request routing. */ + @Deprecated public URI nextAsURI() { markActivity(); - List nodes = liveNodes.get(); - if (nodes.isEmpty()) { - throw new IllegalStateException("No live nodes available"); + LazyQueryPlan queryPlan = new LazyQueryPlan(this); + QueryPlanNodeFilter filter = newQueryPlanNodeFilter(); + URI node = filter.nextRouteCandidate(queryPlan, new ArrayDeque<>()); + if (node != null) { + return node; } - return nodes.get(Math.abs(nextLiveNodeIndex.getAndIncrement() % nodes.size())); + throw new IllegalStateException("No live nodes available"); } /** - * nextAsURI. + * Returns the next node URI with the provided path and query. * - * @param path a {@link java.lang.String} object - * @param query a {@link java.lang.String} object - * @return a {@link java.net.URI} object - * @since 1.0.1 + * @param path URI path + * @param query URI query + * @return a selected node URI with the provided path and query + * @deprecated Use {@link LazyQueryPlan} for request routing and compose request paths through the + * SDK. */ + @Deprecated public URI nextAsURI(String path, String query) { try { - URI uri = this.nextAsURI(); - return withPathAndQuery(uri, path, query); + return withPathAndQuery(nextAsURI(), path, query); } catch (URISyntaxException e) { - // Should never happen, nextAsURI content is already validated throw new RuntimeException(e); } } @@ -524,16 +583,28 @@ private static String streamToString(InputStream stream) throws IOException { return result; } - void updateLiveNodes() throws IOException { + /** + * Refreshes discovered nodes from {@code /localnodes} for the configured routing scope. + * + *

This updates the discovered-node candidate set only. Per-node health state is maintained + * separately and synchronized by {@link #setDiscoveredNodes(List)}. + * + * @throws IOException reserved for discovery implementations that surface polling failures + */ + void refreshDiscoveredNodes() throws IOException { RoutingScope scope = this.config.getRoutingScope(); IOException lastException = null; while (scope != null) { try { List nodes = getNodesForScope(scope); if (!nodes.isEmpty()) { - liveNodes.set(nodes); + setDiscoveredNodes(nodes); logger.log( - Level.FINE, "Updated hosts to " + liveNodes + " using " + scope.getDescription()); + Level.FINE, + "Updated discovered nodes to " + + discoveredNodes.get() + + " using " + + scope.getDescription()); return; } } catch (IOException e) { @@ -551,17 +622,38 @@ void updateLiveNodes() throws IOException { } scope = fallback; } - // No nodes found in any scope - keep the current list. Initial seed nodes are retained - // separately and remain discovery candidates without being injected into the routing list. + // No nodes found in any scope - keep the current routing list. Initial seed nodes remain + // available as discovery fallback candidates, but are not injected into the routing ring. if (lastException != null) { logger.log( Level.WARNING, - "All nodes unreachable in every routing scope, keeping existing node list"); + "All nodes unreachable in every routing scope, keeping existing discovered node list"); } else { logger.log(Level.WARNING, "No nodes found in any routing scope, keeping existing node list"); } } + void updateLiveNodes() throws IOException { + refreshDiscoveredNodes(); + } + + /** + * Publishes the discovered node set and synchronizes node-health bookkeeping. + * + *

The stored list is sorted and deduplicated on every discovery update. Newly discovered nodes + * are added to the health store as active. Nodes that disappear from one discovery response keep + * their health state so a later rediscovery cannot silently resurrect a down node as active. + * + * @param nodes discovered node candidates for the configured routing scope + */ + private void setDiscoveredNodes(List nodes) { + List deduped = sortAndDedupeNodes(nodes); + for (URI node : deduped) { + healthStore.addNode(node); + } + discoveredNodes.set(deduped); + } + private List getNodesForScope(RoutingScope scope) throws IOException { String query = scope.getLocalNodesQuery(); String requestQuery = query.isEmpty() ? null : query; @@ -596,6 +688,7 @@ private DiscoveryAttempt discoverNodes( try { List discoveredNodes = getNodes(withPathAndRawQuery(candidate, "/localnodes", requestQuery)); + reportNodeResult(candidate, NodeHealthObservation.PROBE_SUCCESS, false); if (!discoveredNodes.isEmpty()) { if (!(scope instanceof ClusterScope)) { return new DiscoveryAttempt(candidates, discoveredNodes, lastException); @@ -603,16 +696,9 @@ private DiscoveryAttempt discoverNodes( nodes.addAll(discoveredNodes); } } catch (IOException e) { - logger.log( - Level.WARNING, - "Failed to contact " - + candidateDescription - + " " - + candidate - + " for " - + scope.getDescription(), - e); - lastException = e; + lastException = recordDiscoveryFailure(scope, candidate, candidateDescription, e); + } catch (RuntimeException e) { + lastException = recordDiscoveryFailure(scope, candidate, candidateDescription, e); } catch (URISyntaxException e) { throw new RuntimeException(e); } @@ -621,14 +707,33 @@ private DiscoveryAttempt discoverNodes( return new DiscoveryAttempt(candidates, new ArrayList<>(nodes), lastException); } + private IOException recordDiscoveryFailure( + RoutingScope scope, URI candidate, String candidateDescription, Exception failure) { + reportNodeResult(candidate, NodeHealthObservation.PROBE_FAILURE, false); + logger.log( + Level.WARNING, + "Failed to contact " + + candidateDescription + + " " + + candidate + + " for " + + scope.getDescription(), + failure); + if (failure instanceof IOException) { + return (IOException) failure; + } + return new IOException( + "runtime failure contacting " + candidateDescription + " " + candidate, failure); + } + private List liveDiscoveryCandidates() { - return new ArrayList<>(new LinkedHashSet<>(liveNodes.get())); + return filterDownDiscoveryNodes(dedupePreservingOrder(getDiscoveredNodesInternal())); } private List initialDiscoveryCandidates(List alreadyTried) { Set candidates = new LinkedHashSet<>(initialNodes); candidates.removeAll(alreadyTried); - return new ArrayList<>(candidates); + return filterDownDiscoveryNodes(new ArrayList<>(candidates)); } private static class DiscoveryAttempt { @@ -657,23 +762,13 @@ private URI withPathAndRawQuery(URI uri, String path, String rawQuery) throws UR } private List getNodes(URI uri) throws IOException { - SdkHttpRequest sdkRequest = - SdkHttpRequest.builder() - .uri(uri) - .method(SdkHttpMethod.GET) - .putHeader("Host", uri.getHost() + ":" + uri.getPort()) - .putHeader("Connection", "keep-alive") - .build(); - HttpExecuteRequest executeRequest = HttpExecuteRequest.builder().request(sdkRequest).build(); - ExecutableHttpRequest preparedRequest = pollingHttpClient.prepareRequest(executeRequest); - HttpExecuteResponse response = preparedRequest.call(); + HttpExecuteResponse response = executeGet(uri); try { int statusCode = response.httpResponse().statusCode(); if (statusCode != HttpURLConnection.HTTP_OK) { - // Consume and close the response body to release the connection response.responseBody().ifPresent(this::consumeAndClose); - return Collections.emptyList(); + throw new HttpStatusException(uri, statusCode); } Optional bodyOpt = response.responseBody(); @@ -687,19 +782,43 @@ private List getNodes(URI uri) throws IOException { } return parseLocalNodesResponse(responseStr); + } catch (HttpStatusException e) { + throw e; } catch (IOException e) { - // Ensure the response body is consumed on error response.responseBody().ifPresent(this::consumeAndClose); throw e; } } - private List parseLocalNodesResponse(String responseStr) { - // response looks like: ["127.0.0.2","127.0.0.3","127.0.0.1"] - responseStr = responseStr.trim(); + private int getHttpStatus(URI uri) throws IOException { + HttpExecuteResponse response = executeGet(uri); + try { + int statusCode = response.httpResponse().statusCode(); + response.responseBody().ifPresent(this::consumeAndClose); + return statusCode; + } catch (RuntimeException e) { + response.responseBody().ifPresent(this::consumeAndClose); + throw e; + } + } + + private HttpExecuteResponse executeGet(URI uri) throws IOException { + SdkHttpRequest sdkRequest = + SdkHttpRequest.builder() + .uri(uri) + .method(SdkHttpMethod.GET) + .putHeader("Host", uri.getHost() + ":" + uri.getPort()) + .putHeader("Connection", "keep-alive") + .build(); + HttpExecuteRequest executeRequest = HttpExecuteRequest.builder().request(sdkRequest).build(); + ExecutableHttpRequest preparedRequest = pollingHttpClient.prepareRequest(executeRequest); + return preparedRequest.call(); + } + + private List parseLocalNodesResponse(String responseStr) throws IOException { + responseStr = responseStr != null ? responseStr.trim() : ""; if (!responseStr.startsWith("[") || !responseStr.endsWith("]")) { - logger.log(Level.WARNING, "Malformed /localnodes response: " + responseStr); - return Collections.emptyList(); + throw new InvalidLocalNodesResponseException("invalid /localnodes JSON response"); } String responseBody = responseStr.substring(1, responseStr.length() - 1).trim(); @@ -711,12 +830,8 @@ private List parseLocalNodesResponse(String responseStr) { List newHosts = new ArrayList<>(); for (String host : list) { host = host.trim(); - if (host.isEmpty()) { - continue; - } if (host.length() < 2 || !host.startsWith("\"") || !host.endsWith("\"")) { - logger.log(Level.WARNING, "Malformed host entry in /localnodes response: " + host); - continue; + throw new InvalidLocalNodesResponseException("invalid host in /localnodes JSON response"); } host = host.substring(1, host.length() - 1); try { @@ -728,6 +843,21 @@ private List parseLocalNodesResponse(String responseStr) { return newHosts; } + private static class HttpStatusException extends IOException { + private final int statusCode; + + private HttpStatusException(URI uri, int statusCode) { + super("non-200 response from " + uri + ": " + statusCode); + this.statusCode = statusCode; + } + } + + private static class InvalidLocalNodesResponseException extends IOException { + private InvalidLocalNodesResponseException(String message) { + super(message); + } + } + /** * Consumes and closes an AbortableInputStream to release the underlying connection back to the * pool. @@ -749,15 +879,6 @@ private void consumeAndClose(AbortableInputStream stream) { } } - /** - * Returns the polling HTTP client. Intended for testing only. - * - * @return the polling SdkHttpClient - */ - SdkHttpClient getPollingHttpClient() { - return pollingHttpClient; - } - /** Exception thrown when a check operation cannot be completed. */ public static class FailedToCheck extends Exception { /** @@ -819,7 +940,38 @@ public void checkIfRackAndDatacenterSetCorrectly() throws FailedToCheck, Validat * @since 1.0.1 */ public Boolean checkIfRackDatacenterFeatureIsSupported() throws FailedToCheck { - URI uri = nextAsURI("/localnodes", null); + markActivity(); + LazyQueryPlan queryPlan = new LazyQueryPlan(this); + QueryPlanNodeFilter filter = newQueryPlanNodeFilter(); + FailedToCheck lastFailure = null; + while (queryPlan.hasNext()) { + URI selected = queryPlan.next(); + if (selected == null) { + lastFailure = new FailedToCheck("query plan returned null node"); + continue; + } + if (!filter.shouldRouteTo(selected)) { + continue; + } + + URI uri; + try { + uri = withPathAndQuery(selected, "/localnodes", null); + } catch (URISyntaxException e) { + lastFailure = new FailedToCheck("Invalid URI selected by query plan: " + selected, e); + continue; + } + + try { + return checkIfRackDatacenterFeatureIsSupported(selected, uri); + } catch (FailedToCheck e) { + lastFailure = e; + } + } + throw lastFailure != null ? lastFailure : new FailedToCheck("No live nodes available"); + } + + private Boolean checkIfRackDatacenterFeatureIsSupported(URI node, URI uri) throws FailedToCheck { URI fakeRackUrl; try { fakeRackUrl = @@ -844,13 +996,96 @@ public Boolean checkIfRackDatacenterFeatureIsSupported() throws FailedToCheck { // filtering or not. throw new FailedToCheck(String.format("host %s returned empty list", uri)); } + reportNodeResult(node, NodeHealthObservation.PROBE_SUCCESS); // When rack filtering is not supported server returns same nodes. return hostsWithFakeRack.size() != hostsWithoutRack.size(); - } catch (IOException e) { + } catch (IOException | RuntimeException e) { + reportNodeResult(node, NodeHealthObservation.PROBE_FAILURE); throw new FailedToCheck("failed to read list of nodes from the node", e); } } + /** + * Returns active nodes eligible for normal routing. + * + * @return active node URIs + * @since 2.0.6 + */ + public List getActiveNodes() { + return Collections.unmodifiableList(new ArrayList<>(getActiveNodesInternal())); + } + + /** + * Returns quarantined nodes eligible for sampled verification traffic. + * + * @return quarantined node URIs + * @since 2.0.6 + */ + public List getQuarantinedNodes() { + return Collections.unmodifiableList(new ArrayList<>(getQuarantinedNodesInternal())); + } + + /** + * Returns down nodes excluded from normal routing. + * + * @return down node URIs + * @since 2.0.6 + */ + public List getDownNodes() { + return Collections.unmodifiableList(new ArrayList<>(getDownNodesInternal())); + } + + /** + * Returns a node health status snapshot. + * + * @param node node URI + * @return status snapshot, or null when the node is unknown + * @since 2.0.6 + */ + public NodeHealthStatus getNodeHealthStatus(URI node) { + return healthStore.getNodeStatus(node); + } + + /** + * Reports a node request outcome to the health tracker. + * + * @param node node URI + * @param observation observed request result + * @since 2.0.6 + */ + public void reportNodeResult(URI node, NodeHealthObservation observation) { + reportNodeResult(node, observation, true); + } + + private void reportNodeResult(URI node, NodeHealthObservation observation, boolean markActivity) { + if (markActivity) { + markActivity(); + } + healthStore.reportNodeResult(node, observation); + } + + /** Runs one background probe cycle for down nodes with {@code GET /localnodes}. */ + List runDownNodeProbes() { + return healthStore.probeDownNodes( + getDownNodeProbeCandidates(), + (node, status) -> { + try { + int statusCode = getHttpStatus(withPathAndQuery(node, "/localnodes", null)); + return statusCode == HttpURLConnection.HTTP_OK + ? NodeHealthObservation.PROBE_SUCCESS + : NodeHealthObservation.PROBE_FAILURE; + } catch (IOException | RuntimeException | URISyntaxException e) { + return NodeHealthObservation.PROBE_FAILURE; + } + }); + } + + private List getDownNodeProbeCandidates() { + List candidates = dedupePreservingOrder(getDiscoveredNodesInternal()); + appendUniqueNodes(candidates, initialNodes); + return candidates; + } + /** * Returns the routing scope configured for this instance. * @@ -862,27 +1097,382 @@ public RoutingScope getRoutingScope() { } /** - * Returns the internal live nodes list directly. This is intended for use by {@link + * Returns nodes for a normal request query plan. + * + *

The returned list contains all known discovered candidates. Health filtering is + * intentionally applied at the last routing moment by {@link QueryPlanNodeFilter} so plan + * ordering remains stable while node health changes. + * + * @return query-plan node URIs + * @since 2.0.6 + */ + public List getQueryPlanNodes() { + return getDiscoveredNodesForAffinityQueryPlan(); + } + + /** + * Returns nodes for a partition-key hash query plan. + * + *

The candidate order is the seeded affinity order over all known discovered nodes. Health + * filtering is intentionally applied at the last routing moment by {@link QueryPlanNodeFilter}. + * + * @param hash partition-key hash + * @return query-plan node URIs + * @since 2.0.6 + */ + public List getQueryPlanNodesForHash(long hash) { + return drainSeeded(getDiscoveredNodesForAffinityQueryPlan(), hash); + } + + List getQueryPlanNodesWithPreferredNodes(List preferredNodes) { + if (preferredNodes == null) { + throw new IllegalArgumentException("preferredNodes cannot be null"); + } + return orderPreferredNodesFirst(getDiscoveredNodesForAffinityQueryPlan(), preferredNodes); + } + + /** + * Returns the first known node for a partition-key hash without consuming quarantine sampling. + * + *

This is used by batch-write key-affinity vote aggregation, where each item contributes its + * preferred coordinator but the batch as a whole gets one query plan. Health filtering is applied + * later by {@link QueryPlanNodeFilter}. + * + * @param hash partition-key hash + * @return the preferred node, or null when no candidates exist + */ + public URI getPreferredQueryPlanNodeForHash(long hash) { + List candidates = drainSeeded(getDiscoveredNodesForAffinityQueryPlan(), hash); + return candidates.isEmpty() ? null : candidates.get(0); + } + + /** + * Returns discovered scoped nodes in the canonical order used to seed key-affinity plans. + * + * @return sorted discovered nodes for affinity hashing + */ + List getDiscoveredNodesForAffinityQueryPlan() { + return sortAndDedupeNodes(getDiscoveredNodesInternal()); + } + + private NodeHealthState getQueryPlanNodeState(URI node) { + NodeHealthStatus status = healthStore.getNodeStatus(node); + return status != null ? status.getState() : NodeHealthState.ACTIVE; + } + + private boolean shouldSampleQuarantinedNodeForRouting() { + return shouldSampleQuarantinedNode(getActiveNodesInternal().isEmpty()); + } + + /** Creates a per-request node-health filter for query-plan candidates. */ + public QueryPlanNodeFilter newQueryPlanNodeFilter() { + return new QueryPlanNodeFilter(); + } + + /** + * Per-request final health gate for query-plan candidates. + * + *

{@link LazyQueryPlan} keeps all known nodes in candidate order. This filter decides, at the + * last routing moment, whether a candidate can receive the request attempt. If no active nodes + * remain, the filter fails open and allows all known candidates without mutating their stored + * health state. + */ + public final class QueryPlanNodeFilter { + private Boolean sampleQuarantinedNode; + private boolean usedQuarantinedNode; + + private QueryPlanNodeFilter() {} + + /** + * Returns true when a query attempt may be routed to the candidate. + * + *

Active nodes are always allowed. When no active nodes exist, all known nodes are allowed + * as an emergency fallback. Otherwise, down nodes are skipped, and at most one quarantined node + * is allowed when the quarantine sampling policy admits it for this request. + * + * @param node candidate node + * @return true if the request may be routed to this node + */ + public boolean shouldRouteTo(URI node) { + if (node == null) { + return false; + } + NodeHealthState state = getQueryPlanNodeState(node); + if (activeNodesEmpty()) { + return true; + } + if (state == NodeHealthState.ACTIVE) { + return true; + } + if (state != NodeHealthState.QUARANTINED || usedQuarantinedNode) { + return false; + } + if (!shouldPreferQuarantinedNode()) { + return false; + } + usedQuarantinedNode = true; + return true; + } + + /** + * Returns the next node this request should route to, preserving skipped active candidates for + * later attempts. + * + * @param candidates query-plan candidates + * @param skippedCandidates active candidates delayed while checking quarantine sampling + * @return the next routable node, or null when none is available + */ + public URI nextRouteCandidate(Iterator candidates, Deque skippedCandidates) { + Deque delayedCandidates = + skippedCandidates != null ? skippedCandidates : new ArrayDeque<>(); + URI skippedCandidate = pollSkippedRouteCandidate(delayedCandidates); + if (skippedCandidate != null || candidates == null) { + return skippedCandidate; + } + + boolean hasQuarantinedCandidate = hasQuarantinedCandidate(); + while (candidates.hasNext()) { + URI candidate = candidates.next(); + if (!shouldRouteTo(candidate)) { + continue; + } + if (!hasQuarantinedCandidate || isQuarantinedNode(candidate)) { + return candidate; + } + delayedCandidates.addLast(candidate); + } + return pollSkippedRouteCandidate(delayedCandidates); + } + + private URI pollSkippedRouteCandidate(Deque skippedCandidates) { + while (skippedCandidates != null && !skippedCandidates.isEmpty()) { + URI candidate = skippedCandidates.removeFirst(); + if (shouldRouteTo(candidate)) { + return candidate; + } + } + return null; + } + + private boolean shouldPreferQuarantinedNode() { + if (activeNodesEmpty()) { + return false; + } + if (sampleQuarantinedNode == null) { + sampleQuarantinedNode = shouldSampleQuarantinedNodeForRouting(); + } + return sampleQuarantinedNode; + } + + private boolean hasQuarantinedCandidate() { + return !activeNodesEmpty() && !getQuarantinedNodesInternal().isEmpty(); + } + + private boolean isQuarantinedNode(URI node) { + return node != null && getQueryPlanNodeState(node) == NodeHealthState.QUARANTINED; + } + + private boolean activeNodesEmpty() { + return getActiveNodesInternal().isEmpty(); + } + } + + private static List orderPreferredNodesFirst(List sortedNodes, List preferred) { + List ordered = new ArrayList<>(sortedNodes.size()); + Map availableNodes = new HashMap<>(); + for (URI node : sortedNodes) { + availableNodes.putIfAbsent(NodeHealthStore.canonicalNodeKey(node), node); + } + Set orderedNodes = new HashSet<>(); + for (URI preferredNode : preferred) { + URI key = NodeHealthStore.canonicalNodeKey(preferredNode); + URI node = availableNodes.get(key); + if (node != null && orderedNodes.add(key)) { + ordered.add(node); + } + } + for (URI node : sortedNodes) { + if (orderedNodes.add(NodeHealthStore.canonicalNodeKey(node))) { + ordered.add(node); + } + } + return ordered; + } + + /** + * Returns the internal discovered nodes list directly. This is intended for use by {@link * LazyQueryPlan} to avoid copying the list on every access. * *

Note: The returned list should not be modified. It may be replaced atomically at any time by - * the background refresh thread. + * the background refresh thread. Discovery updates publish sorted lists, while the initial seed + * list preserves configured seed order until the first successful update. * *

This method is protected to allow test mocks to override it. * - * @return the current live nodes list (not a copy) + *

The default implementation intentionally delegates to {@link #getLiveNodesInternal()} so + * existing subclasses that override the deprecated hook continue to feed query planning. + * + * @return the current discovered nodes list (not a copy) + */ + protected List getDiscoveredNodesInternal() { + return getLiveNodesInternal(); + } + + /** + * Returns the internal discovered nodes list directly. + * + *

This method is retained for source and binary compatibility with subclasses compiled against + * versions where the raw discovered-node hook used this name. + * + * @return the current discovered nodes list (not a copy) + * @deprecated Use {@link #getDiscoveredNodesInternal()} instead. */ + @Deprecated protected List getLiveNodesInternal() { - return liveNodes.get(); + return discoveredNodes.get(); + } + + protected List getActiveNodesInternal() { + List activeNodes = new ArrayList<>(); + for (URI node : getDiscoveredNodesInternal()) { + NodeHealthStatus status = healthStore.getNodeStatus(node); + if (status == null || status.getState() == NodeHealthState.ACTIVE) { + activeNodes.add(node); + } + } + return sortAndDedupeNodes(activeNodes); + } + + private List getQuarantinedNodesInternal() { + return getDiscoveredNodesByState(NodeHealthState.QUARANTINED); + } + + private List getDownNodesInternal() { + return getDiscoveredNodesByState(NodeHealthState.DOWN); + } + + private List getDiscoveredNodesByState(NodeHealthState state) { + List nodes = new ArrayList<>(); + for (URI node : getDiscoveredNodesInternal()) { + NodeHealthStatus status = healthStore.getNodeStatus(node); + if (status != null && status.getState() == state) { + nodes.add(node); + } + } + return sortAndDedupeNodes(nodes); + } + + private List filterDownDiscoveryNodes(List nodes) { + List filtered = new ArrayList<>(); + for (URI node : nodes) { + if (getQueryPlanNodeState(node) != NodeHealthState.DOWN) { + filtered.add(node); + } + } + return filtered; + } + + private boolean shouldSampleQuarantinedNode(boolean activeNodesEmpty) { + List quarantinedNodes = getQuarantinedNodesInternal(); + if (quarantinedNodes.isEmpty()) { + return false; + } + if (activeNodesEmpty) { + return true; + } + + return consumeQuarantineSamplingSlot(); + } + + private boolean consumeQuarantineSamplingSlot() { + long attempt = quarantineSamplingCounter.getAndIncrement() + 1; + return attempt % config.getNodeHealthConfig().getQuarantineTrafficInterval() == 0; + } + + static URI firstNodeWithSeed(List nodes, long seed) { + List candidates = sortAndDedupeNodes(nodes); + if (candidates.isEmpty()) { + return null; + } + return candidates.get(new GoRand(seed).intn(candidates.size())); + } + + static List drainSeeded(List nodes, long seed) { + List remainingNodes = sortAndDedupeNodes(nodes); + List out = new ArrayList<>(); + GoRand rand = new GoRand(seed); + while (!remainingNodes.isEmpty()) { + int idx = rand.intn(remainingNodes.size()); + URI node = remainingNodes.get(idx); + int last = remainingNodes.size() - 1; + remainingNodes.set(idx, remainingNodes.get(last)); + remainingNodes.remove(last); + out.add(node); + } + return out; + } + + static List sortAndDedupeNodes(List nodes) { + List sorted = dedupePreservingOrder(nodes); + sorted.sort(Comparator.comparing(URI::toString)); + return sorted; + } + + static List dedupePreservingOrder(List nodes) { + List deduped = new ArrayList<>(); + appendUniqueNodes(deduped, nodes); + return deduped; + } + + static void appendUniqueNodes(List out, List nodes) { + Set seen = nodeKeys(out); + for (URI node : nodes) { + URI key = NodeHealthStore.canonicalNodeKey(node); + if (node != null && seen.add(key)) { + out.add(node); + } + } + } + + private static Set nodeKeys(List nodes) { + Set keys = new HashSet<>(); + for (URI node : nodes) { + keys.add(NodeHealthStore.canonicalNodeKey(node)); + } + return keys; } /** - * Returns a snapshot of the current live nodes list. + * Returns a snapshot of the current discovered nodes list. * - * @return an unmodifiable list of the current live node URIs + *

The list is the raw discovered candidate set, not a health-filtered routing list. Discovery + * updates publish sorted nodes, while the initial seed list preserves configured seed order until + * the first successful update. + * + * @return an unmodifiable list of the current discovered node URIs + * @since 2.1.0 + */ + public List getDiscoveredNodes() { + return Collections.unmodifiableList(new ArrayList<>(discoveredNodes.get())); + } + + /** + * Returns a snapshot of discovered nodes currently active for normal routing. + * + *

Nodes that are quarantined or down remain visible through {@link #getDiscoveredNodes()} but + * are excluded from this live-node view. + * + * @return an unmodifiable list of active discovered node URIs in stored discovered-node order * @since 2.0.0 */ public List getLiveNodes() { - return Collections.unmodifiableList(new ArrayList<>(liveNodes.get())); + List live = new ArrayList<>(); + for (URI node : getDiscoveredNodesInternal()) { + NodeHealthStatus status = healthStore.getNodeStatus(node); + if (status == null || status.getState() == NodeHealthState.ACTIVE) { + live.add(node); + } + } + return Collections.unmodifiableList(dedupePreservingOrder(live)); } } diff --git a/src/main/java/com/scylladb/alternator/internal/LazyQueryPlan.java b/src/main/java/com/scylladb/alternator/internal/LazyQueryPlan.java index b1f37c8..4c0bb84 100644 --- a/src/main/java/com/scylladb/alternator/internal/LazyQueryPlan.java +++ b/src/main/java/com/scylladb/alternator/internal/LazyQueryPlan.java @@ -2,43 +2,45 @@ import java.net.URI; import java.util.ArrayList; -import java.util.Comparator; +import java.util.Collections; import java.util.Iterator; import java.util.List; -import java.util.NoSuchElementException; import java.util.concurrent.ThreadLocalRandom; /** * A lazy iterator over URI nodes that pulls nodes from {@link AlternatorLiveNodes} one at a time * only when needed. * + *

Query plans build their candidate order from all known nodes supplied by {@link + * AlternatorLiveNodes}. Final node-health eligibility and request outcome reporting are handled + * outside this iterator by the query-plan interceptor. + * *

When created with a seed, this implementation uses a Go-compatible Lagged Fibonacci Generator - * ({@link GoRand}) and pick-and-remove selection to produce identical node sequences across all - * Alternator client implementations (Go, Java, etc.) for the same seed. This cross-language - * compatibility is critical for key route affinity, where requests with the same partition key hash - * must be routed to the same coordinator node regardless of client language. + * ({@link GoRand}) and pick-and-remove selection to produce identical discovered-node candidate + * sequences across all Alternator client implementations (Go, Java, etc.) for the same seed. This + * cross-language compatibility is critical for key route affinity, where requests with the same + * partition key hash must start from the same coordinator candidate regardless of client language. * *

When created without a seed, uses {@link ThreadLocalRandom} for non-deterministic load * balancing. * - *

The pick-and-remove algorithm works by maintaining a mutable copy of the node list. On each - * {@link #next()} call, a random index is chosen, the node at that index is returned, and the node - * is replaced with the last element in the list (which is then truncated). This ensures each node - * is returned exactly once with O(1) per selection. + *

The pick-and-remove algorithm works by maintaining a mutable copy of the candidate list. For + * random plans it picks candidates while advancing the iterator; for seeded plans it builds the + * deterministic candidate order up front. Each candidate is returned at most once. * *

Example usage: * * 

{@code
  * AlternatorLiveNodes liveNodes = ...;
  *
- * // Create with random seed for load balancing
+ * // Create with non-deterministic order for load balancing
  * LazyQueryPlan plan = new LazyQueryPlan(liveNodes);
  *
  * // Or create with deterministic seed for key affinity
  * long partitionKeyHash = AttributeValueHasher.hash(pkValue);
  * LazyQueryPlan affinityPlan = new LazyQueryPlan(liveNodes, partitionKeyHash);
  *
- * // Get the first available node
+ * // Get the first currently eligible node
  * if (plan.hasNext()) {
  *     URI node = plan.next();
  *     // Use node...
@@ -54,27 +56,18 @@
  */
 public class LazyQueryPlan implements Iterator, Iterable {
   private final AlternatorLiveNodes liveNodes;
-  private final GoRand goRand;
+  private final long seed;
   private final List preferredNodes;
+  private final boolean seeded;
 
-  /**
-   * Mutable list of remaining nodes for pick-and-remove. Initialized lazily on first access when
-   * using a seeded GoRand. Null when using ThreadLocalRandom (non-seeded mode).
-   */
-  private List remaining;
+  /** Iterator over a lazily captured candidate snapshot. */
+  private Iterator candidates;
 
-  /** Whether the remaining list has been initialized (only used in seeded mode). */
   private boolean initialized;
 
-  /** Cached next node for hasNext()/next() contract in non-seeded mode. */
-  private URI nextNode;
-
-  /** Tracks nodes already returned in non-seeded mode to avoid duplicates. */
-  private java.util.Set usedNodes;
-
   /**
-   * Creates a LazyQueryPlan with a random seed. The iteration order will be non-deterministic,
-   * suitable for general load balancing.
+   * Creates a LazyQueryPlan with non-deterministic candidate order, suitable for general load
+   * balancing over discovered nodes.
    *
    * @param liveNodes the {@link AlternatorLiveNodes} instance to pull nodes from
    */
@@ -83,15 +76,15 @@ public LazyQueryPlan(AlternatorLiveNodes liveNodes) {
       throw new IllegalArgumentException("liveNodes cannot be null");
     }
     this.liveNodes = liveNodes;
-    this.goRand = null;
+    this.seed = 0;
     this.preferredNodes = null;
-    this.usedNodes = new java.util.HashSet<>();
+    this.seeded = false;
   }
 
   /**
    * Creates a LazyQueryPlan with a specified seed for deterministic iteration. Uses Go-compatible
-   * PRNG and pick-and-remove selection to produce identical sequences across all Alternator client
-   * implementations.
+   * PRNG and pick-and-remove selection over discovered nodes to produce identical candidate
+   * sequences across all Alternator client implementations.
    *
    * @param liveNodes the {@link AlternatorLiveNodes} instance to pull nodes from
    * @param seed a long value used to initialize the Go-compatible PRNG
@@ -101,13 +94,14 @@ public LazyQueryPlan(AlternatorLiveNodes liveNodes, long seed) {
       throw new IllegalArgumentException("liveNodes cannot be null");
     }
     this.liveNodes = liveNodes;
-    this.goRand = new GoRand(seed);
+    this.seed = seed;
     this.preferredNodes = null;
+    this.seeded = true;
   }
 
   /**
-   * Creates a LazyQueryPlan that tries the given preferred nodes first, then the remaining live
-   * nodes in canonical affinity order.
+   * Creates a LazyQueryPlan that tries the given preferred nodes first, then the remaining
+   * discovered nodes in canonical affinity order.
    *
    * @param liveNodes the {@link AlternatorLiveNodes} instance to pull nodes from
    * @param preferredNodes nodes to try first, in preference order
@@ -120,37 +114,48 @@ public LazyQueryPlan(AlternatorLiveNodes liveNodes, List preferredNodes) {
       throw new IllegalArgumentException("preferredNodes cannot be null");
     }
     this.liveNodes = liveNodes;
-    this.goRand = null;
+    this.seed = 0;
     this.preferredNodes = new ArrayList<>(preferredNodes);
+    this.seeded = false;
   }
 
   /**
-   * Initializes the remaining list from current live nodes (seeded mode only). Called lazily on
-   * first access so the snapshot is as fresh as possible.
+   * Initializes the candidate list from current discovered nodes. Called lazily on first access so
+   * the candidate snapshot is as fresh as possible.
    */
   private void ensureInitialized() {
     if (!initialized) {
-      remaining = sortedAffinityNodes(liveNodes);
       if (preferredNodes != null) {
-        remaining = orderPreferredNodesFirst(remaining, preferredNodes);
+        candidates = liveNodes.getQueryPlanNodesWithPreferredNodes(preferredNodes).iterator();
+      } else if (seeded) {
+        candidates = liveNodes.getQueryPlanNodesForHash(seed).iterator();
+      } else {
+        candidates = randomPlanCandidates().iterator();
       }
       initialized = true;
     }
   }
 
+  private List randomPlanCandidates() {
+    List queryPlanNodes = liveNodes.getQueryPlanNodes();
+    List candidates = new ArrayList<>(queryPlanNodes);
+    Collections.shuffle(candidates, ThreadLocalRandom.current());
+    return candidates;
+  }
+
   /**
-   * Returns the current live nodes in the canonical order used by affinity routing.
+   * Returns the current discovered nodes in the canonical order used by affinity routing.
    *
    * @param liveNodes the live nodes manager
-   * @return a sorted copy of current live nodes
+   * @return sorted discovered nodes for affinity hashing
+   * @deprecated Use {@link AlternatorLiveNodes#getQueryPlanNodes()} instead.
    */
+  @Deprecated
   public static List sortedAffinityNodes(AlternatorLiveNodes liveNodes) {
     if (liveNodes == null) {
       throw new IllegalArgumentException("liveNodes cannot be null");
     }
-    List nodes = new ArrayList<>(liveNodes.getLiveNodesInternal());
-    nodes.sort(Comparator.comparing(URI::toString));
-    return nodes;
+    return liveNodes.getQueryPlanNodes();
   }
 
   /**
@@ -159,109 +164,27 @@ public static List sortedAffinityNodes(AlternatorLiveNodes liveNodes) {
    *
    * @param liveNodes the live nodes manager
    * @param seed deterministic seed
-   * @return the preferred node, or null when no live nodes exist
+   * @return the preferred node, or null when no candidates exist
+   * @deprecated Use {@link AlternatorLiveNodes#getPreferredQueryPlanNodeForHash(long)} instead.
    */
+  @Deprecated
   public static URI preferredNodeForHash(AlternatorLiveNodes liveNodes, long seed) {
-    List nodes = sortedAffinityNodes(liveNodes);
-    if (nodes.isEmpty()) {
-      return null;
-    }
-    return nodes.get(new GoRand(seed).intn(nodes.size()));
-  }
-
-  private static List orderPreferredNodesFirst(List sortedNodes, List preferred) {
-    List ordered = new ArrayList<>(sortedNodes.size());
-    List remainingNodes = new ArrayList<>(sortedNodes);
-    for (URI preferredNode : preferred) {
-      int index = remainingNodes.indexOf(preferredNode);
-      if (index >= 0) {
-        ordered.add(remainingNodes.remove(index));
-      }
-    }
-    ordered.addAll(remainingNodes);
-    return ordered;
-  }
-
-  /**
-   * Picks and removes a random node from the remaining list using Go's pick-and-remove algorithm.
-   *
-   * @return the selected node, or null if no nodes remain
-   */
-  private URI pickAndRemove() {
-    ensureInitialized();
-    if (remaining.isEmpty()) {
-      return null;
-    }
-    int idx = goRand.intn(remaining.size());
-    URI node = remaining.get(idx);
-    int last = remaining.size() - 1;
-    remaining.set(idx, remaining.get(last));
-    remaining.remove(last);
-    return node;
-  }
-
-  /**
-   * Computes next node for non-seeded mode (ThreadLocalRandom).
-   *
-   * @return the next node, or null if no more available
-   */
-  private URI computeNextNonSeeded() {
-    if (nextNode != null) {
-      return nextNode;
-    }
-
-    List currentNodes = liveNodes.getLiveNodesInternal();
-
-    List availableNodes = new ArrayList<>();
-    for (URI node : currentNodes) {
-      if (!usedNodes.contains(node)) {
-        availableNodes.add(node);
-      }
-    }
-
-    if (availableNodes.isEmpty()) {
-      return null;
+    if (liveNodes == null) {
+      throw new IllegalArgumentException("liveNodes cannot be null");
     }
-
-    int index = ThreadLocalRandom.current().nextInt(availableNodes.size());
-    nextNode = availableNodes.get(index);
-    return nextNode;
+    return liveNodes.getPreferredQueryPlanNodeForHash(seed);
   }
 
   @Override
   public boolean hasNext() {
-    if (goRand != null || preferredNodes != null) {
-      ensureInitialized();
-      return !remaining.isEmpty();
-    }
-    return computeNextNonSeeded() != null;
+    ensureInitialized();
+    return candidates.hasNext();
   }
 
   @Override
   public URI next() {
-    if (goRand != null) {
-      URI node = pickAndRemove();
-      if (node == null) {
-        throw new NoSuchElementException("No more nodes available in query plan");
-      }
-      return node;
-    }
-
-    if (preferredNodes != null) {
-      ensureInitialized();
-      if (remaining.isEmpty()) {
-        throw new NoSuchElementException("No more nodes available in query plan");
-      }
-      return remaining.remove(0);
-    }
-
-    URI node = computeNextNonSeeded();
-    if (node == null) {
-      throw new NoSuchElementException("No more nodes available in query plan");
-    }
-    usedNodes.add(node);
-    nextNode = null;
-    return node;
+    ensureInitialized();
+    return candidates.next();
   }
 
   @Override
diff --git a/src/main/java/com/scylladb/alternator/internal/NodeHealthStore.java b/src/main/java/com/scylladb/alternator/internal/NodeHealthStore.java
new file mode 100644
index 0000000..a8b6527
--- /dev/null
+++ b/src/main/java/com/scylladb/alternator/internal/NodeHealthStore.java
@@ -0,0 +1,275 @@
+package com.scylladb.alternator.internal;
+
+import com.scylladb.alternator.NodeHealthConfig;
+import com.scylladb.alternator.NodeHealthObservation;
+import com.scylladb.alternator.NodeHealthState;
+import com.scylladb.alternator.NodeHealthStatus;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.function.BiFunction;
+
+final class NodeHealthStore {
+  private final NodeHealthConfig config;
+  private final Map statuses = new TreeMap<>();
+
+  NodeHealthStore(NodeHealthConfig config, List initialNodes) {
+    this.config = config != null ? config : NodeHealthConfig.getDefault();
+    for (URI node : initialNodes) {
+      addNode(node);
+    }
+  }
+
+  synchronized List getActiveNodes() {
+    if (config.isDisabled()) {
+      return getStoredNodes();
+    }
+    return getNodesByState(NodeHealthState.ACTIVE);
+  }
+
+  synchronized List getQuarantinedNodes() {
+    if (config.isDisabled()) {
+      return Collections.emptyList();
+    }
+    return getNodesByState(NodeHealthState.QUARANTINED);
+  }
+
+  synchronized List getDownNodes() {
+    if (config.isDisabled()) {
+      return Collections.emptyList();
+    }
+    return getNodesByState(NodeHealthState.DOWN);
+  }
+
+  synchronized NodeHealthStatus getNodeStatus(URI node) {
+    MutableStatus status = getMutableStatus(node);
+    return status != null ? status.snapshot() : null;
+  }
+
+  synchronized void addNode(URI node) {
+    URI key = canonicalNodeKey(node);
+    if (key == null || statuses.containsKey(key)) {
+      return;
+    }
+    MutableStatus status = new MutableStatus(node);
+    status.state = NodeHealthState.ACTIVE;
+    status.consecutiveTrafficFailures = 0;
+    status.consecutiveProbeFailures = 0;
+    status.consecutiveSuccesses = config.getQuarantineSuccessThreshold();
+    status.updatedAtNanos = System.nanoTime();
+    statuses.put(key, status);
+  }
+
+  void reportNodeResult(URI node, NodeHealthObservation observation) {
+    if (config.isDisabled()) {
+      return;
+    }
+    synchronized (this) {
+      MutableStatus status = getMutableStatus(node);
+      if (status == null || observation == null) {
+        return;
+      }
+
+      switch (observation) {
+        case TRAFFIC_SUCCESS:
+          reportTrafficSuccess(status);
+          break;
+        case PROBE_SUCCESS:
+          reportProbeSuccess(status);
+          break;
+        case TRAFFIC_FAILURE:
+        case PROBE_FAILURE:
+          reportHealthFailure(status, observation);
+          break;
+        default:
+          break;
+      }
+      status.updatedAtNanos = System.nanoTime();
+    }
+  }
+
+  private void reportTrafficSuccess(MutableStatus status) {
+    status.consecutiveTrafficFailures = 0;
+    if (status.state == NodeHealthState.QUARANTINED) {
+      reportQuarantineTrafficSuccess(status);
+    } else if (status.state == NodeHealthState.ACTIVE) {
+      reportActiveSuccess(status);
+    } else {
+      status.consecutiveSuccesses = 0;
+    }
+  }
+
+  private void reportProbeSuccess(MutableStatus status) {
+    status.consecutiveProbeFailures = 0;
+    if (status.state == NodeHealthState.DOWN) {
+      status.consecutiveTrafficFailures = 0;
+      status.consecutiveSuccesses++;
+      if (status.consecutiveSuccesses >= config.getDownNodeRecoverySuccessThreshold()) {
+        status.state = NodeHealthState.QUARANTINED;
+        status.consecutiveSuccesses = 0;
+      }
+    } else if (status.state == NodeHealthState.ACTIVE) {
+      reportActiveSuccess(status);
+    }
+  }
+
+  private void reportActiveSuccess(MutableStatus status) {
+    status.consecutiveSuccesses = config.getQuarantineSuccessThreshold();
+  }
+
+  private void reportQuarantineTrafficSuccess(MutableStatus status) {
+    status.consecutiveSuccesses++;
+    if (status.consecutiveSuccesses >= config.getQuarantineSuccessThreshold()) {
+      status.state = NodeHealthState.ACTIVE;
+      status.consecutiveSuccesses = config.getQuarantineSuccessThreshold();
+    }
+  }
+
+  private void reportHealthFailure(MutableStatus status, NodeHealthObservation observation) {
+    int consecutiveFailures = incrementFailure(status, observation);
+    if (status.state == NodeHealthState.QUARANTINED) {
+      markDown(status, observation);
+      return;
+    }
+    status.consecutiveSuccesses = 0;
+    if (consecutiveFailures >= config.getConsecutiveFailureThreshold()) {
+      markDown(status, observation);
+    }
+  }
+
+  private int incrementFailure(MutableStatus status, NodeHealthObservation observation) {
+    if (observation == NodeHealthObservation.PROBE_FAILURE) {
+      return ++status.consecutiveProbeFailures;
+    }
+    return ++status.consecutiveTrafficFailures;
+  }
+
+  private void markDown(MutableStatus status, NodeHealthObservation observation) {
+    status.state = NodeHealthState.DOWN;
+    if (observation == NodeHealthObservation.PROBE_FAILURE) {
+      status.consecutiveProbeFailures =
+          Math.max(status.consecutiveProbeFailures, config.getConsecutiveFailureThreshold());
+    } else {
+      status.consecutiveTrafficFailures =
+          Math.max(status.consecutiveTrafficFailures, config.getConsecutiveFailureThreshold());
+    }
+    status.consecutiveSuccesses = 0;
+  }
+
+  List probeDownNodes(BiFunction probe) {
+    return probeDownNodes(getDownNodes(), probe);
+  }
+
+  List probeDownNodes(
+      List candidateNodes, BiFunction probe) {
+    if (config.isDisabled() || probe == null) {
+      return Collections.emptyList();
+    }
+
+    List candidates =
+        candidateNodes != null ? new ArrayList<>(candidateNodes) : Collections.emptyList();
+    List recovered = new ArrayList<>();
+    Set probedNodes = new HashSet<>();
+    for (URI node : candidates) {
+      URI key = canonicalNodeKey(node);
+      if (key == null || !probedNodes.add(key)) {
+        continue;
+      }
+      NodeHealthStatus status = getNodeStatus(node);
+      if (status == null || status.getState() != NodeHealthState.DOWN) {
+        continue;
+      }
+      NodeHealthObservation observation = probe.apply(node, status);
+      reportNodeResult(node, observation);
+      NodeHealthStatus updatedStatus = getNodeStatus(node);
+      if (observation == NodeHealthObservation.PROBE_SUCCESS
+          && updatedStatus != null
+          && updatedStatus.getState() == NodeHealthState.QUARANTINED) {
+        recovered.add(node);
+      }
+    }
+    return recovered;
+  }
+
+  private List getNodesByState(NodeHealthState state) {
+    List nodes = new ArrayList<>();
+    for (MutableStatus status : statuses.values()) {
+      if (status.state == state) {
+        nodes.add(status.node);
+      }
+    }
+    Collections.sort(nodes);
+    return nodes;
+  }
+
+  private List getStoredNodes() {
+    List nodes = new ArrayList<>();
+    for (MutableStatus status : statuses.values()) {
+      nodes.add(status.node);
+    }
+    Collections.sort(nodes);
+    return nodes;
+  }
+
+  private MutableStatus getMutableStatus(URI node) {
+    URI key = canonicalNodeKey(node);
+    return key != null ? statuses.get(key) : null;
+  }
+
+  static URI canonicalNodeKey(URI node) {
+    int defaultPort = node != null ? defaultPort(node.getScheme()) : -1;
+    if (node == null || defaultPort < 0 || node.getPort() != defaultPort) {
+      return node;
+    }
+    try {
+      return new URI(
+          node.getScheme(),
+          node.getUserInfo(),
+          node.getHost(),
+          -1,
+          node.getPath(),
+          node.getQuery(),
+          node.getFragment());
+    } catch (IllegalArgumentException | URISyntaxException e) {
+      return node;
+    }
+  }
+
+  private static int defaultPort(String scheme) {
+    if ("http".equalsIgnoreCase(scheme)) {
+      return 80;
+    }
+    if ("https".equalsIgnoreCase(scheme)) {
+      return 443;
+    }
+    return -1;
+  }
+
+  private static final class MutableStatus {
+    private final URI node;
+    private NodeHealthState state = NodeHealthState.ACTIVE;
+    private int consecutiveTrafficFailures = 0;
+    private int consecutiveProbeFailures = 0;
+    private int consecutiveSuccesses = 0;
+    private long updatedAtNanos = 0;
+
+    private MutableStatus(URI node) {
+      this.node = node;
+    }
+
+    private NodeHealthStatus snapshot() {
+      return new NodeHealthStatus(
+          state, consecutiveFailures(), consecutiveSuccesses, updatedAtNanos);
+    }
+
+    private int consecutiveFailures() {
+      return Math.max(consecutiveTrafficFailures, consecutiveProbeFailures);
+    }
+  }
+}
diff --git a/src/main/java/com/scylladb/alternator/queryplan/AffinityQueryPlanInterceptor.java b/src/main/java/com/scylladb/alternator/queryplan/AffinityQueryPlanInterceptor.java
index becb303..bbd3233 100644
--- a/src/main/java/com/scylladb/alternator/queryplan/AffinityQueryPlanInterceptor.java
+++ b/src/main/java/com/scylladb/alternator/queryplan/AffinityQueryPlanInterceptor.java
@@ -23,10 +23,11 @@
 /**
  * Execution interceptor that implements key-based route affinity.
  *
- * 
This interceptor extends {@link BasicQueryPlanInterceptor} to provide deterministic routing
- * based on partition key values. When key affinity conditions are met, it creates a {@link
- * LazyQueryPlan} with a seed derived from the partition key hash, ensuring that requests for the
- * same partition key are routed to the same node.
+ * 
This interceptor extends {@link BasicQueryPlanInterceptor} to provide deterministic
+ * discovered-node candidate order based on partition key values. When key affinity conditions are
+ * met, it creates a {@link LazyQueryPlan} with a seed derived from the partition key hash. Final
+ * node-health eligibility is applied by {@link BasicQueryPlanInterceptor#modifyHttpRequest} just
+ * before each request attempt is routed.
  *
  * 
When key affinity conditions are not met (e.g., request type doesn't qualify, partition key
  * not found), the interceptor falls back to the random plan created by the base class.
@@ -129,7 +130,7 @@ private LazyQueryPlan getQueryPlan(SdkRequest request) {
       return null;
     }
 
-    // Hash the partition key and create a deterministic query plan
+    // Hash the partition key and create a deterministic known-node candidate order.
     try {
       long hash = AttributeValueHasher.hash(pkValue);
       return new LazyQueryPlan(liveNodes, hash);
@@ -159,15 +160,15 @@ private LazyQueryPlan getBatchWriteQueryPlan(BatchWriteItemRequest request) {
 
       try {
         long hash = AttributeValueHasher.hash(pkValue);
-        URI preferredNode = LazyQueryPlan.preferredNodeForHash(liveNodes, hash);
-        if (preferredNode == null) {
-          return null;
+        URI preferredNode = liveNodes.getPreferredQueryPlanNodeForHash(hash);
+        if (preferredNode != null) {
+          votes.merge(preferredNode, 1, Integer::sum);
         }
-        votes.merge(preferredNode, 1, Integer::sum);
       } catch (IllegalArgumentException e) {
         // Unsupported partition-key shapes cannot be routed by key affinity.
       }
     }
+
     List preferredNodes = votePreferenceOrder(votes);
     if (preferredNodes.isEmpty()) {
       return null;
diff --git a/src/main/java/com/scylladb/alternator/queryplan/BasicQueryPlanInterceptor.java b/src/main/java/com/scylladb/alternator/queryplan/BasicQueryPlanInterceptor.java
index 61893cb..b0b990e 100644
--- a/src/main/java/com/scylladb/alternator/queryplan/BasicQueryPlanInterceptor.java
+++ b/src/main/java/com/scylladb/alternator/queryplan/BasicQueryPlanInterceptor.java
@@ -1,21 +1,31 @@
 package com.scylladb.alternator.queryplan;
 
+import com.scylladb.alternator.NodeHealthObservation;
 import com.scylladb.alternator.internal.AlternatorLiveNodes;
 import com.scylladb.alternator.internal.LazyQueryPlan;
 import java.net.URI;
+import java.net.URISyntaxException;
+import java.util.ArrayDeque;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Deque;
+import java.util.HashSet;
+import java.util.Set;
+import software.amazon.awssdk.awscore.exception.AwsErrorDetails;
+import software.amazon.awssdk.awscore.exception.AwsServiceException;
 import software.amazon.awssdk.core.interceptor.Context;
 import software.amazon.awssdk.core.interceptor.ExecutionAttribute;
 import software.amazon.awssdk.core.interceptor.ExecutionAttributes;
 import software.amazon.awssdk.core.interceptor.ExecutionInterceptor;
 import software.amazon.awssdk.http.SdkHttpRequest;
+import software.amazon.awssdk.http.SdkHttpResponse;
 
 /**
  * Execution interceptor that creates a {@link LazyQueryPlan} for each request.
  *
- * 
This interceptor creates a query plan with a random seed during {@link #beforeExecution},
- * which provides pseudo-random load balancing across available nodes. The plan is stored in {@link
- * ExecutionAttributes} and applied during {@link #modifyHttpRequest} to route the request to the
- * selected node.
+ * 
This interceptor creates a non-deterministic query plan during {@link #beforeExecution}, which
+ * provides load balancing over query-plan nodes supplied by {@link AlternatorLiveNodes}. The plan
+ * is stored in {@link ExecutionAttributes} and applied during {@link #modifyHttpRequest}.
  *
  * 
This approach correctly handles both synchronous and asynchronous clients, as {@link
  * ExecutionAttributes} travel with the request throughout its lifecycle.
@@ -29,6 +39,35 @@ public class BasicQueryPlanInterceptor implements ExecutionInterceptor {
   protected static final ExecutionAttribute QUERY_PLAN =
       new ExecutionAttribute<>("QueryPlanInterceptor.queryPlan");
 
+  private static final ExecutionAttribute IN_FLIGHT_NODE =
+      new ExecutionAttribute<>("QueryPlanInterceptor.inFlightNode");
+
+  private static final ExecutionAttribute PENDING_HTTP_RESPONSE_NODE =
+      new ExecutionAttribute<>("QueryPlanInterceptor.pendingHttpResponseNode");
+
+  private static final ExecutionAttribute
+      ROUTING_NODE_FILTER = new ExecutionAttribute<>("QueryPlanInterceptor.routingNodeFilter");
+
+  private static final ExecutionAttribute> SKIPPED_ROUTE_CANDIDATES =
+      new ExecutionAttribute<>("QueryPlanInterceptor.skippedRouteCandidates");
+
+  private static final int HTTP_UNAUTHORIZED = 401;
+  private static final int HTTP_FORBIDDEN = 403;
+  private static final Set AUTHENTICATION_ERROR_CODES =
+      Collections.unmodifiableSet(
+          new HashSet<>(
+              Arrays.asList(
+                  "InvalidSignatureException",
+                  "MissingAuthenticationTokenException",
+                  "UnrecognizedClientException",
+                  "SignatureDoesNotMatch",
+                  "IncompleteSignatureException",
+                  "InvalidClientTokenId",
+                  "InvalidAccessKeyId",
+                  "ExpiredTokenException",
+                  "RequestExpired",
+                  "TokenRefreshRequired")));
+
   protected final AlternatorLiveNodes liveNodes;
 
   /**
@@ -43,21 +82,37 @@ public BasicQueryPlanInterceptor(AlternatorLiveNodes liveNodes) {
   @Override
   public void beforeExecution(
       Context.BeforeExecution context, ExecutionAttributes executionAttributes) {
-    // Create a query plan with random seed for pseudo-random load balancing
+    // Create a non-deterministic query plan for load balancing over known nodes.
     executionAttributes.putAttribute(QUERY_PLAN, new LazyQueryPlan(liveNodes));
+    executionAttributes.putAttribute(ROUTING_NODE_FILTER, liveNodes.newQueryPlanNodeFilter());
+    executionAttributes.putAttribute(SKIPPED_ROUTE_CANDIDATES, new ArrayDeque<>());
   }
 
   @Override
   public SdkHttpRequest modifyHttpRequest(
       Context.ModifyHttpRequest context, ExecutionAttributes executionAttributes) {
     LazyQueryPlan plan = executionAttributes.getAttribute(QUERY_PLAN);
-    if (plan == null || !plan.hasNext()) {
-      // No plan available, return original request
-      return context.httpRequest();
+    SdkHttpRequest originalRequest = context.httpRequest();
+    if (plan == null) {
+      return originalRequest;
     }
 
-    URI targetUri = plan.next();
-    SdkHttpRequest originalRequest = context.httpRequest();
+    AlternatorLiveNodes.QueryPlanNodeFilter routingFilter =
+        executionAttributes.getAttribute(ROUTING_NODE_FILTER);
+    if (routingFilter == null) {
+      routingFilter = liveNodes.newQueryPlanNodeFilter();
+      executionAttributes.putAttribute(ROUTING_NODE_FILTER, routingFilter);
+    }
+
+    URI targetUri =
+        routingFilter.nextRouteCandidate(plan, skippedRouteCandidates(executionAttributes));
+    if (targetUri == null) {
+      URI originalEndpoint = requestEndpoint(originalRequest);
+      if (routingFilter.shouldRouteTo(originalEndpoint)) {
+        return originalRequest;
+      }
+      throw new IllegalStateException("No live nodes available");
+    }
 
     // Build new request with the target node's host and port
     return originalRequest.toBuilder()
@@ -68,6 +123,156 @@ public SdkHttpRequest modifyHttpRequest(
         .build();
   }
 
+  private Deque skippedRouteCandidates(ExecutionAttributes executionAttributes) {
+    Deque skippedCandidates = executionAttributes.getAttribute(SKIPPED_ROUTE_CANDIDATES);
+    if (skippedCandidates == null) {
+      skippedCandidates = new ArrayDeque<>();
+      executionAttributes.putAttribute(SKIPPED_ROUTE_CANDIDATES, skippedCandidates);
+    }
+    return skippedCandidates;
+  }
+
+  private URI requestEndpoint(SdkHttpRequest request) {
+    try {
+      return new URI(request.protocol(), null, request.host(), request.port(), null, null, null);
+    } catch (URISyntaxException e) {
+      throw new IllegalArgumentException("Invalid request endpoint: " + request.getUri(), e);
+    }
+  }
+
+  private void reportInFlightTransportFailure(ExecutionAttributes executionAttributes) {
+    reportPendingHttpResponseResult(executionAttributes, NodeHealthObservation.TRAFFIC_SUCCESS);
+    reportInFlightNodeResult(executionAttributes, NodeHealthObservation.TRAFFIC_FAILURE);
+  }
+
+  private void reportInFlightNodeResult(
+      ExecutionAttributes executionAttributes, NodeHealthObservation observation) {
+    URI node = executionAttributes.getAttribute(IN_FLIGHT_NODE);
+    if (node == null) {
+      return;
+    }
+    executionAttributes.putAttribute(IN_FLIGHT_NODE, null);
+    liveNodes.reportNodeResult(node, observation);
+  }
+
+  private void deferInFlightHttpResponse(ExecutionAttributes executionAttributes) {
+    URI node = executionAttributes.getAttribute(IN_FLIGHT_NODE);
+    if (node == null) {
+      return;
+    }
+    executionAttributes.putAttribute(IN_FLIGHT_NODE, null);
+    executionAttributes.putAttribute(PENDING_HTTP_RESPONSE_NODE, node);
+  }
+
+  private void reportPendingHttpResponseResult(
+      ExecutionAttributes executionAttributes, NodeHealthObservation observation) {
+    URI node = executionAttributes.getAttribute(PENDING_HTTP_RESPONSE_NODE);
+    if (node == null) {
+      return;
+    }
+    executionAttributes.putAttribute(PENDING_HTTP_RESPONSE_NODE, null);
+    liveNodes.reportNodeResult(node, observation);
+  }
+
+  private void reportHttpResponseResult(
+      ExecutionAttributes executionAttributes, NodeHealthObservation observation) {
+    URI pendingNode = executionAttributes.getAttribute(PENDING_HTTP_RESPONSE_NODE);
+    if (pendingNode != null) {
+      reportPendingHttpResponseResult(executionAttributes, observation);
+      return;
+    }
+    reportInFlightNodeResult(executionAttributes, observation);
+  }
+
+  private boolean isAuthenticationStatus(SdkHttpResponse response) {
+    int statusCode = response.statusCode();
+    return statusCode == HTTP_UNAUTHORIZED || statusCode == HTTP_FORBIDDEN;
+  }
+
+  private boolean shouldDeferHttpResponseObservation(SdkHttpResponse response) {
+    return response.statusCode() >= 400 && !isAuthenticationStatus(response);
+  }
+
+  private NodeHealthObservation httpResponseObservation(SdkHttpResponse response) {
+    return isAuthenticationStatus(response)
+        ? NodeHealthObservation.TRAFFIC_FAILURE
+        : NodeHealthObservation.TRAFFIC_SUCCESS;
+  }
+
+  private NodeHealthObservation failedExecutionObservation(Context.FailedExecution context) {
+    if (context.httpResponse().isPresent()
+        && isAuthenticationStatus(context.httpResponse().get())) {
+      return NodeHealthObservation.TRAFFIC_FAILURE;
+    }
+    return isAuthenticationException(context.exception())
+        ? NodeHealthObservation.TRAFFIC_FAILURE
+        : NodeHealthObservation.TRAFFIC_SUCCESS;
+  }
+
+  private boolean isAuthenticationException(Throwable exception) {
+    Throwable current = exception;
+    while (current != null) {
+      if (current instanceof AwsServiceException
+          && isAuthenticationErrorCode(((AwsServiceException) current).awsErrorDetails())) {
+        return true;
+      }
+
+      Throwable cause = current.getCause();
+      if (cause == current) {
+        return false;
+      }
+      current = cause;
+    }
+    return false;
+  }
+
+  private boolean isAuthenticationErrorCode(AwsErrorDetails details) {
+    if (details == null || details.errorCode() == null) {
+      return false;
+    }
+
+    return AUTHENTICATION_ERROR_CODES.contains(errorCodeName(details.errorCode()));
+  }
+
+  private String errorCodeName(String errorCode) {
+    int separator = Math.max(errorCode.lastIndexOf('#'), errorCode.lastIndexOf(':'));
+    if (separator >= 0 && separator + 1 < errorCode.length()) {
+      return errorCode.substring(separator + 1);
+    }
+    return errorCode;
+  }
+
+  @Override
+  public void beforeTransmission(
+      Context.BeforeTransmission context, ExecutionAttributes executionAttributes) {
+    reportInFlightTransportFailure(executionAttributes);
+    executionAttributes.putAttribute(IN_FLIGHT_NODE, requestEndpoint(context.httpRequest()));
+  }
+
+  @Override
+  public void afterTransmission(
+      Context.AfterTransmission context, ExecutionAttributes executionAttributes) {
+    if (shouldDeferHttpResponseObservation(context.httpResponse())) {
+      deferInFlightHttpResponse(executionAttributes);
+      return;
+    }
+    reportHttpResponseResult(executionAttributes, httpResponseObservation(context.httpResponse()));
+  }
+
+  @Override
+  public void onExecutionFailure(
+      Context.FailedExecution context, ExecutionAttributes executionAttributes) {
+    if (context.httpResponse().isPresent()) {
+      reportHttpResponseResult(executionAttributes, failedExecutionObservation(context));
+      return;
+    }
+    if (isAuthenticationException(context.exception())) {
+      reportHttpResponseResult(executionAttributes, NodeHealthObservation.TRAFFIC_FAILURE);
+      return;
+    }
+    reportInFlightTransportFailure(executionAttributes);
+  }
+
   /**
    * Returns the live nodes manager used by this interceptor.
    *
diff --git a/src/test/java/com/scylladb/alternator/AffinityQueryPlanInterceptorTest.java b/src/test/java/com/scylladb/alternator/AffinityQueryPlanInterceptorTest.java
index a0842f2..8dfb1e6 100644
--- a/src/test/java/com/scylladb/alternator/AffinityQueryPlanInterceptorTest.java
+++ b/src/test/java/com/scylladb/alternator/AffinityQueryPlanInterceptorTest.java
@@ -54,7 +54,7 @@
  * 30+ operation variants to match Go's WithKeyRouteAffinity test coverage.
  *
  * 
For each operation variant, the test checks whether affinity routing (same key always goes to
- * the same node) or round-robin routing (requests spread across nodes) is used, depending on the
+ * the same node) or random routing (requests spread across nodes) is used, depending on the
  * affinity mode.
  *
  * @author dmitry.kropachev
@@ -86,6 +86,11 @@ public void tearDown() {
 
   private DynamoDbClient createClient(KeyRouteAffinityConfig keyAffinity) {
     AlternatorLiveNodes liveNodes = new MockAlternatorLiveNodes(testNodeUris);
+    return createClient(keyAffinity, liveNodes);
+  }
+
+  private DynamoDbClient createClient(
+      KeyRouteAffinityConfig keyAffinity, AlternatorLiveNodes liveNodes) {
     ClientOverrideConfiguration.Builder overrideBuilder = ClientOverrideConfiguration.builder();
     if (keyAffinity != null && keyAffinity.isEnabled()) {
       overrideBuilder.addExecutionInterceptor(
@@ -122,8 +127,10 @@ private void assertAffinityRouting(Runnable operation) {
     }
   }
 
-  /** Verify that an operation routes to different nodes (round-robin, no affinity). */
-  private void assertRoundRobinRouting(Runnable operation) {
+  /**
+   * Verify that an operation routes to different nodes (random query-plan routing, no affinity).
+   */
+  private void assertRandomRouting(Runnable operation) {
     Set nodesUsed = new HashSet<>();
     for (int i = 0; i < 20; i++) {
       mockHttpClient.clearCapturedRequests();
@@ -170,6 +177,31 @@ private String findPkValueWithDifferentRoute(String prefix, String otherPkValue)
     return null;
   }
 
+  private String findPkValueWithPreferredRoute(String prefix, List candidates, URI expected) {
+    for (int i = 0; i < 1000; i++) {
+      String candidate = prefix + i;
+      long hash = AttributeValueHasher.hash(AttributeValue.builder().s(candidate).build());
+      if (expected.equals(firstNodeForSeed(candidates, hash))) {
+        return candidate;
+      }
+    }
+    fail("Could not find test partition key with expected route " + expected);
+    return null;
+  }
+
+  private URI firstNodeForSeed(List candidates, long seed) {
+    LazyQueryPlan plan = new LazyQueryPlan(new MockAlternatorLiveNodes(candidates), seed);
+    return plan.hasNext() ? plan.next() : null;
+  }
+
+  private List collectNodes(LazyQueryPlan plan) {
+    List collected = new ArrayList<>();
+    while (plan.hasNext()) {
+      collected.add(plan.next());
+    }
+    return collected;
+  }
+
   private void assertSameRoute(String message, URI expected, URI actual) {
     assertNotNull(message + ": actual route should not be null", actual);
     assertEquals(message + ": scheme", expected.getScheme(), actual.getScheme());
@@ -195,10 +227,10 @@ public void testPutItemSimple_AnyWrite_Affinity() {
   }
 
   @Test
-  public void testPutItemSimple_Rmw_RoundRobin() {
+  public void testPutItemSimple_Rmw_RandomRouting() {
     DynamoDbClient client = createClient(buildConfig(KeyRouteAffinity.RMW));
     try {
-      assertRoundRobinRouting(
+      assertRandomRouting(
           () ->
               client.putItem(
                   PutItemRequest.builder().tableName(TABLE_NAME).item(makeItem()).build()));
@@ -208,10 +240,10 @@ public void testPutItemSimple_Rmw_RoundRobin() {
   }
 
   @Test
-  public void testPutItemSimple_None_RoundRobin() {
+  public void testPutItemSimple_None_RandomRouting() {
     DynamoDbClient client = createClient(buildConfig(KeyRouteAffinity.NONE));
     try {
-      assertRoundRobinRouting(
+      assertRandomRouting(
           () ->
               client.putItem(
                   PutItemRequest.builder().tableName(TABLE_NAME).item(makeItem()).build()));
@@ -257,10 +289,10 @@ public void testPutItemWithConditionExpression_Rmw_Affinity() {
   }
 
   @Test
-  public void testPutItemWithConditionExpression_None_RoundRobin() {
+  public void testPutItemWithConditionExpression_None_RandomRouting() {
     DynamoDbClient client = createClient(buildConfig(KeyRouteAffinity.NONE));
     try {
-      assertRoundRobinRouting(
+      assertRandomRouting(
           () ->
               client.putItem(
                   PutItemRequest.builder()
@@ -322,7 +354,7 @@ public void testPutItemWithExpected_Rmw_Affinity() {
   }
 
   @Test
-  public void testPutItemWithExpected_None_RoundRobin() {
+  public void testPutItemWithExpected_None_RandomRouting() {
     DynamoDbClient client = createClient(buildConfig(KeyRouteAffinity.NONE));
     try {
       Map expected = new HashMap<>();
@@ -331,7 +363,7 @@ public void testPutItemWithExpected_None_RoundRobin() {
           ExpectedAttributeValue.builder()
               .value(AttributeValue.builder().s("old").build())
               .build());
-      assertRoundRobinRouting(
+      assertRandomRouting(
           () ->
               client.putItem(
                   PutItemRequest.builder()
@@ -381,10 +413,10 @@ public void testPutItemReturnValuesAllOld_Rmw_Affinity() {
   }
 
   @Test
-  public void testPutItemReturnValuesAllOld_None_RoundRobin() {
+  public void testPutItemReturnValuesAllOld_None_RandomRouting() {
     DynamoDbClient client = createClient(buildConfig(KeyRouteAffinity.NONE));
     try {
-      assertRoundRobinRouting(
+      assertRandomRouting(
           () ->
               client.putItem(
                   PutItemRequest.builder()
@@ -417,10 +449,10 @@ public void testPutItemReturnValuesNone_AnyWrite_Affinity() {
   }
 
   @Test
-  public void testPutItemReturnValuesNone_Rmw_RoundRobin() {
+  public void testPutItemReturnValuesNone_Rmw_RandomRouting() {
     DynamoDbClient client = createClient(buildConfig(KeyRouteAffinity.RMW));
     try {
-      assertRoundRobinRouting(
+      assertRandomRouting(
           () ->
               client.putItem(
                   PutItemRequest.builder()
@@ -434,10 +466,10 @@ public void testPutItemReturnValuesNone_Rmw_RoundRobin() {
   }
 
   @Test
-  public void testPutItemReturnValuesNone_None_RoundRobin() {
+  public void testPutItemReturnValuesNone_None_RandomRouting() {
     DynamoDbClient client = createClient(buildConfig(KeyRouteAffinity.NONE));
     try {
-      assertRoundRobinRouting(
+      assertRandomRouting(
           () ->
               client.putItem(
                   PutItemRequest.builder()
@@ -468,10 +500,10 @@ public void testUpdateItemSimple_AnyWrite_Affinity() {
   }
 
   @Test
-  public void testUpdateItemSimple_Rmw_RoundRobin() {
+  public void testUpdateItemSimple_Rmw_RandomRouting() {
     DynamoDbClient client = createClient(buildConfig(KeyRouteAffinity.RMW));
     try {
-      assertRoundRobinRouting(
+      assertRandomRouting(
           () ->
               client.updateItem(
                   UpdateItemRequest.builder().tableName(TABLE_NAME).key(makeKey()).build()));
@@ -481,10 +513,10 @@ public void testUpdateItemSimple_Rmw_RoundRobin() {
   }
 
   @Test
-  public void testUpdateItemSimple_None_RoundRobin() {
+  public void testUpdateItemSimple_None_RandomRouting() {
     DynamoDbClient client = createClient(buildConfig(KeyRouteAffinity.NONE));
     try {
-      assertRoundRobinRouting(
+      assertRandomRouting(
           () ->
               client.updateItem(
                   UpdateItemRequest.builder().tableName(TABLE_NAME).key(makeKey()).build()));
@@ -536,10 +568,10 @@ public void testUpdateItemWithUpdateExpression_Rmw_Affinity() {
   }
 
   @Test
-  public void testUpdateItemWithUpdateExpression_None_RoundRobin() {
+  public void testUpdateItemWithUpdateExpression_None_RandomRouting() {
     DynamoDbClient client = createClient(buildConfig(KeyRouteAffinity.NONE));
     try {
-      assertRoundRobinRouting(
+      assertRandomRouting(
           () ->
               client.updateItem(
                   UpdateItemRequest.builder()
@@ -592,10 +624,10 @@ public void testUpdateItemWithConditionExpression_Rmw_Affinity() {
   }
 
   @Test
-  public void testUpdateItemWithConditionExpression_None_RoundRobin() {
+  public void testUpdateItemWithConditionExpression_None_RandomRouting() {
     DynamoDbClient client = createClient(buildConfig(KeyRouteAffinity.NONE));
     try {
-      assertRoundRobinRouting(
+      assertRandomRouting(
           () ->
               client.updateItem(
                   UpdateItemRequest.builder()
@@ -657,7 +689,7 @@ public void testUpdateItemWithExpected_Rmw_Affinity() {
   }
 
   @Test
-  public void testUpdateItemWithExpected_None_RoundRobin() {
+  public void testUpdateItemWithExpected_None_RandomRouting() {
     DynamoDbClient client = createClient(buildConfig(KeyRouteAffinity.NONE));
     try {
       Map expected = new HashMap<>();
@@ -666,7 +698,7 @@ public void testUpdateItemWithExpected_None_RoundRobin() {
           ExpectedAttributeValue.builder()
               .value(AttributeValue.builder().s("old").build())
               .build());
-      assertRoundRobinRouting(
+      assertRandomRouting(
           () ->
               client.updateItem(
                   UpdateItemRequest.builder()
@@ -716,10 +748,10 @@ public void testUpdateItemReturnValuesAllOld_Rmw_Affinity() {
   }
 
   @Test
-  public void testUpdateItemReturnValuesAllOld_None_RoundRobin() {
+  public void testUpdateItemReturnValuesAllOld_None_RandomRouting() {
     DynamoDbClient client = createClient(buildConfig(KeyRouteAffinity.NONE));
     try {
-      assertRoundRobinRouting(
+      assertRandomRouting(
           () ->
               client.updateItem(
                   UpdateItemRequest.builder()
@@ -769,10 +801,10 @@ public void testUpdateItemReturnValuesUpdatedOld_Rmw_Affinity() {
   }
 
   @Test
-  public void testUpdateItemReturnValuesUpdatedOld_None_RoundRobin() {
+  public void testUpdateItemReturnValuesUpdatedOld_None_RandomRouting() {
     DynamoDbClient client = createClient(buildConfig(KeyRouteAffinity.NONE));
     try {
-      assertRoundRobinRouting(
+      assertRandomRouting(
           () ->
               client.updateItem(
                   UpdateItemRequest.builder()
@@ -822,10 +854,10 @@ public void testUpdateItemReturnValuesAllNew_Rmw_Affinity() {
   }
 
   @Test
-  public void testUpdateItemReturnValuesAllNew_None_RoundRobin() {
+  public void testUpdateItemReturnValuesAllNew_None_RandomRouting() {
     DynamoDbClient client = createClient(buildConfig(KeyRouteAffinity.NONE));
     try {
-      assertRoundRobinRouting(
+      assertRandomRouting(
           () ->
               client.updateItem(
                   UpdateItemRequest.builder()
@@ -858,11 +890,11 @@ public void testUpdateItemReturnValuesUpdatedNew_AnyWrite_Affinity() {
   }
 
   @Test
-  public void testUpdateItemReturnValuesUpdatedNew_Rmw_RoundRobin() {
+  public void testUpdateItemReturnValuesUpdatedNew_Rmw_RandomRouting() {
     // UPDATED_NEW does NOT trigger RMW - can be computed from update alone
     DynamoDbClient client = createClient(buildConfig(KeyRouteAffinity.RMW));
     try {
-      assertRoundRobinRouting(
+      assertRandomRouting(
           () ->
               client.updateItem(
                   UpdateItemRequest.builder()
@@ -876,10 +908,10 @@ public void testUpdateItemReturnValuesUpdatedNew_Rmw_RoundRobin() {
   }
 
   @Test
-  public void testUpdateItemReturnValuesUpdatedNew_None_RoundRobin() {
+  public void testUpdateItemReturnValuesUpdatedNew_None_RandomRouting() {
     DynamoDbClient client = createClient(buildConfig(KeyRouteAffinity.NONE));
     try {
-      assertRoundRobinRouting(
+      assertRandomRouting(
           () ->
               client.updateItem(
                   UpdateItemRequest.builder()
@@ -944,7 +976,7 @@ public void testUpdateItemAddAction_Rmw_Affinity() {
   }
 
   @Test
-  public void testUpdateItemAddAction_None_RoundRobin() {
+  public void testUpdateItemAddAction_None_RandomRouting() {
     DynamoDbClient client = createClient(buildConfig(KeyRouteAffinity.NONE));
     try {
       Map updates = new HashMap<>();
@@ -954,7 +986,7 @@ public void testUpdateItemAddAction_None_RoundRobin() {
               .action(AttributeAction.ADD)
               .value(AttributeValue.builder().n("1").build())
               .build());
-      assertRoundRobinRouting(
+      assertRandomRouting(
           () ->
               client.updateItem(
                   UpdateItemRequest.builder()
@@ -1019,7 +1051,7 @@ public void testUpdateItemDeleteActionWithValue_Rmw_Affinity() {
   }
 
   @Test
-  public void testUpdateItemDeleteActionWithValue_None_RoundRobin() {
+  public void testUpdateItemDeleteActionWithValue_None_RandomRouting() {
     DynamoDbClient client = createClient(buildConfig(KeyRouteAffinity.NONE));
     try {
       Map updates = new HashMap<>();
@@ -1029,7 +1061,7 @@ public void testUpdateItemDeleteActionWithValue_None_RoundRobin() {
               .action(AttributeAction.DELETE)
               .value(AttributeValue.builder().ss("old-tag").build())
               .build());
-      assertRoundRobinRouting(
+      assertRandomRouting(
           () ->
               client.updateItem(
                   UpdateItemRequest.builder()
@@ -1065,14 +1097,14 @@ public void testUpdateItemDeleteActionWithoutValue_AnyWrite_Affinity() {
   }
 
   @Test
-  public void testUpdateItemDeleteActionWithoutValue_Rmw_RoundRobin() {
+  public void testUpdateItemDeleteActionWithoutValue_Rmw_RandomRouting() {
     // DELETE without value just removes the attribute - no read needed
     DynamoDbClient client = createClient(buildConfig(KeyRouteAffinity.RMW));
     try {
       Map updates = new HashMap<>();
       updates.put(
           "obsolete_field", AttributeValueUpdate.builder().action(AttributeAction.DELETE).build());
-      assertRoundRobinRouting(
+      assertRandomRouting(
           () ->
               client.updateItem(
                   UpdateItemRequest.builder()
@@ -1086,13 +1118,13 @@ public void testUpdateItemDeleteActionWithoutValue_Rmw_RoundRobin() {
   }
 
   @Test
-  public void testUpdateItemDeleteActionWithoutValue_None_RoundRobin() {
+  public void testUpdateItemDeleteActionWithoutValue_None_RandomRouting() {
     DynamoDbClient client = createClient(buildConfig(KeyRouteAffinity.NONE));
     try {
       Map updates = new HashMap<>();
       updates.put(
           "obsolete_field", AttributeValueUpdate.builder().action(AttributeAction.DELETE).build());
-      assertRoundRobinRouting(
+      assertRandomRouting(
           () ->
               client.updateItem(
                   UpdateItemRequest.builder()
@@ -1132,7 +1164,7 @@ public void testUpdateItemPutAction_AnyWrite_Affinity() {
   }
 
   @Test
-  public void testUpdateItemPutAction_Rmw_RoundRobin() {
+  public void testUpdateItemPutAction_Rmw_RandomRouting() {
     // PUT action is a simple overwrite - no read needed
     DynamoDbClient client = createClient(buildConfig(KeyRouteAffinity.RMW));
     try {
@@ -1143,7 +1175,7 @@ public void testUpdateItemPutAction_Rmw_RoundRobin() {
               .action(AttributeAction.PUT)
               .value(AttributeValue.builder().s("new-value").build())
               .build());
-      assertRoundRobinRouting(
+      assertRandomRouting(
           () ->
               client.updateItem(
                   UpdateItemRequest.builder()
@@ -1157,7 +1189,7 @@ public void testUpdateItemPutAction_Rmw_RoundRobin() {
   }
 
   @Test
-  public void testUpdateItemPutAction_None_RoundRobin() {
+  public void testUpdateItemPutAction_None_RandomRouting() {
     DynamoDbClient client = createClient(buildConfig(KeyRouteAffinity.NONE));
     try {
       Map updates = new HashMap<>();
@@ -1167,7 +1199,7 @@ public void testUpdateItemPutAction_None_RoundRobin() {
               .action(AttributeAction.PUT)
               .value(AttributeValue.builder().s("new-value").build())
               .build());
-      assertRoundRobinRouting(
+      assertRandomRouting(
           () ->
               client.updateItem(
                   UpdateItemRequest.builder()
@@ -1198,10 +1230,10 @@ public void testDeleteItemSimple_AnyWrite_Affinity() {
   }
 
   @Test
-  public void testDeleteItemSimple_Rmw_RoundRobin() {
+  public void testDeleteItemSimple_Rmw_RandomRouting() {
     DynamoDbClient client = createClient(buildConfig(KeyRouteAffinity.RMW));
     try {
-      assertRoundRobinRouting(
+      assertRandomRouting(
           () ->
               client.deleteItem(
                   DeleteItemRequest.builder().tableName(TABLE_NAME).key(makeKey()).build()));
@@ -1211,10 +1243,10 @@ public void testDeleteItemSimple_Rmw_RoundRobin() {
   }
 
   @Test
-  public void testDeleteItemSimple_None_RoundRobin() {
+  public void testDeleteItemSimple_None_RandomRouting() {
     DynamoDbClient client = createClient(buildConfig(KeyRouteAffinity.NONE));
     try {
-      assertRoundRobinRouting(
+      assertRandomRouting(
           () ->
               client.deleteItem(
                   DeleteItemRequest.builder().tableName(TABLE_NAME).key(makeKey()).build()));
@@ -1260,10 +1292,10 @@ public void testDeleteItemWithConditionExpression_Rmw_Affinity() {
   }
 
   @Test
-  public void testDeleteItemWithConditionExpression_None_RoundRobin() {
+  public void testDeleteItemWithConditionExpression_None_RandomRouting() {
     DynamoDbClient client = createClient(buildConfig(KeyRouteAffinity.NONE));
     try {
-      assertRoundRobinRouting(
+      assertRandomRouting(
           () ->
               client.deleteItem(
                   DeleteItemRequest.builder()
@@ -1325,7 +1357,7 @@ public void testDeleteItemWithExpected_Rmw_Affinity() {
   }
 
   @Test
-  public void testDeleteItemWithExpected_None_RoundRobin() {
+  public void testDeleteItemWithExpected_None_RandomRouting() {
     DynamoDbClient client = createClient(buildConfig(KeyRouteAffinity.NONE));
     try {
       Map expected = new HashMap<>();
@@ -1334,7 +1366,7 @@ public void testDeleteItemWithExpected_None_RoundRobin() {
           ExpectedAttributeValue.builder()
               .value(AttributeValue.builder().s("old").build())
               .build());
-      assertRoundRobinRouting(
+      assertRandomRouting(
           () ->
               client.deleteItem(
                   DeleteItemRequest.builder()
@@ -1384,10 +1416,10 @@ public void testDeleteItemReturnValuesAllOld_Rmw_Affinity() {
   }
 
   @Test
-  public void testDeleteItemReturnValuesAllOld_None_RoundRobin() {
+  public void testDeleteItemReturnValuesAllOld_None_RandomRouting() {
     DynamoDbClient client = createClient(buildConfig(KeyRouteAffinity.NONE));
     try {
-      assertRoundRobinRouting(
+      assertRandomRouting(
           () ->
               client.deleteItem(
                   DeleteItemRequest.builder()
@@ -1405,10 +1437,10 @@ public void testDeleteItemReturnValuesAllOld_None_RoundRobin() {
   // --- GetItem ---
 
   @Test
-  public void testGetItem_AnyWrite_RoundRobin() {
+  public void testGetItem_AnyWrite_RandomRouting() {
     DynamoDbClient client = createClient(buildConfig(KeyRouteAffinity.ANY_WRITE));
     try {
-      assertRoundRobinRouting(
+      assertRandomRouting(
           () ->
               client.getItem(
                   GetItemRequest.builder().tableName(TABLE_NAME).key(makeKey()).build()));
@@ -1418,10 +1450,10 @@ public void testGetItem_AnyWrite_RoundRobin() {
   }
 
   @Test
-  public void testGetItem_Rmw_RoundRobin() {
+  public void testGetItem_Rmw_RandomRouting() {
     DynamoDbClient client = createClient(buildConfig(KeyRouteAffinity.RMW));
     try {
-      assertRoundRobinRouting(
+      assertRandomRouting(
           () ->
               client.getItem(
                   GetItemRequest.builder().tableName(TABLE_NAME).key(makeKey()).build()));
@@ -1431,10 +1463,10 @@ public void testGetItem_Rmw_RoundRobin() {
   }
 
   @Test
-  public void testGetItem_None_RoundRobin() {
+  public void testGetItem_None_RandomRouting() {
     DynamoDbClient client = createClient(buildConfig(KeyRouteAffinity.NONE));
     try {
-      assertRoundRobinRouting(
+      assertRandomRouting(
           () ->
               client.getItem(
                   GetItemRequest.builder().tableName(TABLE_NAME).key(makeKey()).build()));
@@ -1446,10 +1478,10 @@ public void testGetItem_None_RoundRobin() {
   // --- Query ---
 
   @Test
-  public void testQuery_AnyWrite_RoundRobin() {
+  public void testQuery_AnyWrite_RandomRouting() {
     DynamoDbClient client = createClient(buildConfig(KeyRouteAffinity.ANY_WRITE));
     try {
-      assertRoundRobinRouting(
+      assertRandomRouting(
           () ->
               client.query(
                   QueryRequest.builder()
@@ -1465,10 +1497,10 @@ public void testQuery_AnyWrite_RoundRobin() {
   }
 
   @Test
-  public void testQuery_Rmw_RoundRobin() {
+  public void testQuery_Rmw_RandomRouting() {
     DynamoDbClient client = createClient(buildConfig(KeyRouteAffinity.RMW));
     try {
-      assertRoundRobinRouting(
+      assertRandomRouting(
           () ->
               client.query(
                   QueryRequest.builder()
@@ -1484,10 +1516,10 @@ public void testQuery_Rmw_RoundRobin() {
   }
 
   @Test
-  public void testQuery_None_RoundRobin() {
+  public void testQuery_None_RandomRouting() {
     DynamoDbClient client = createClient(buildConfig(KeyRouteAffinity.NONE));
     try {
-      assertRoundRobinRouting(
+      assertRandomRouting(
           () ->
               client.query(
                   QueryRequest.builder()
@@ -1505,33 +1537,30 @@ public void testQuery_None_RoundRobin() {
   // --- Scan ---
 
   @Test
-  public void testScan_AnyWrite_RoundRobin() {
+  public void testScan_AnyWrite_RandomRouting() {
     DynamoDbClient client = createClient(buildConfig(KeyRouteAffinity.ANY_WRITE));
     try {
-      assertRoundRobinRouting(
-          () -> client.scan(ScanRequest.builder().tableName(TABLE_NAME).build()));
+      assertRandomRouting(() -> client.scan(ScanRequest.builder().tableName(TABLE_NAME).build()));
     } finally {
       client.close();
     }
   }
 
   @Test
-  public void testScan_Rmw_RoundRobin() {
+  public void testScan_Rmw_RandomRouting() {
     DynamoDbClient client = createClient(buildConfig(KeyRouteAffinity.RMW));
     try {
-      assertRoundRobinRouting(
-          () -> client.scan(ScanRequest.builder().tableName(TABLE_NAME).build()));
+      assertRandomRouting(() -> client.scan(ScanRequest.builder().tableName(TABLE_NAME).build()));
     } finally {
       client.close();
     }
   }
 
   @Test
-  public void testScan_None_RoundRobin() {
+  public void testScan_None_RandomRouting() {
     DynamoDbClient client = createClient(buildConfig(KeyRouteAffinity.NONE));
     try {
-      assertRoundRobinRouting(
-          () -> client.scan(ScanRequest.builder().tableName(TABLE_NAME).build()));
+      assertRandomRouting(() -> client.scan(ScanRequest.builder().tableName(TABLE_NAME).build()));
     } finally {
       client.close();
     }
@@ -1559,7 +1588,7 @@ public void testBatchWriteItem_AnyWrite_Affinity() {
   }
 
   @Test
-  public void testBatchWriteItem_Rmw_RoundRobin() {
+  public void testBatchWriteItem_Rmw_RandomRouting() {
     DynamoDbClient client = createClient(buildConfig(KeyRouteAffinity.RMW));
     try {
       Map> requestItems = new HashMap<>();
@@ -1568,7 +1597,7 @@ public void testBatchWriteItem_Rmw_RoundRobin() {
           WriteRequest.builder().putRequest(PutRequest.builder().item(makeItem()).build()).build());
       requestItems.put(TABLE_NAME, writes);
 
-      assertRoundRobinRouting(
+      assertRandomRouting(
           () ->
               client.batchWriteItem(
                   BatchWriteItemRequest.builder().requestItems(requestItems).build()));
@@ -1578,7 +1607,7 @@ public void testBatchWriteItem_Rmw_RoundRobin() {
   }
 
   @Test
-  public void testBatchWriteItem_None_RoundRobin() {
+  public void testBatchWriteItem_None_RandomRouting() {
     DynamoDbClient client = createClient(buildConfig(KeyRouteAffinity.NONE));
     try {
       Map> requestItems = new HashMap<>();
@@ -1587,7 +1616,7 @@ public void testBatchWriteItem_None_RoundRobin() {
           WriteRequest.builder().putRequest(PutRequest.builder().item(makeItem()).build()).build());
       requestItems.put(TABLE_NAME, writes);
 
-      assertRoundRobinRouting(
+      assertRandomRouting(
           () ->
               client.batchWriteItem(
                   BatchWriteItemRequest.builder().requestItems(requestItems).build()));
@@ -1821,7 +1850,7 @@ public void testBatchWriteItemTieUsesLexicographicNodeOrder() {
   }
 
   @Test
-  public void testBatchWriteItemNoUsableCandidatesFallsBackToRoundRobin() {
+  public void testBatchWriteItemNoUsableCandidatesFallsBackToRandomRouting() {
     DynamoDbClient client = createClient(buildConfig(KeyRouteAffinity.ANY_WRITE));
     try {
       Map unsupportedPk = new HashMap<>();
@@ -1834,7 +1863,7 @@ public void testBatchWriteItemNoUsableCandidatesFallsBackToRoundRobin() {
                   .putRequest(PutRequest.builder().item(unsupportedPk).build())
                   .build()));
 
-      assertRoundRobinRouting(
+      assertRandomRouting(
           () ->
               client.batchWriteItem(
                   BatchWriteItemRequest.builder().requestItems(requestItems).build()));
@@ -1844,13 +1873,49 @@ public void testBatchWriteItemNoUsableCandidatesFallsBackToRoundRobin() {
   }
 
   @Test
-  public void testPutItemUnsupportedPartitionKeyTypeFallsBackToRoundRobin() {
+  public void testBatchWriteItemSkipsDownPreferredNodeUsingStableKnownOrder() {
+    URI downNode = testNodeUris.get(1);
+    MockAlternatorLiveNodes liveNodes =
+        new MockAlternatorLiveNodes(
+            testNodeUris, NodeHealthConfig.builder().withConsecutiveFailureThreshold(1).build());
+    String pkValue = findPkValueWithPreferredRoute("down-route-", testNodeUris, downNode);
+    long hash = AttributeValueHasher.hash(AttributeValue.builder().s(pkValue).build());
+    List allHealthyOrder =
+        collectNodes(new LazyQueryPlan(new MockAlternatorLiveNodes(testNodeUris), hash));
+    assertEquals(downNode, allHealthyOrder.get(0));
+    URI expectedAfterSkip = allHealthyOrder.get(1);
+
+    liveNodes.reportNodeResult(downNode, NodeHealthObservation.TRAFFIC_FAILURE);
+
+    DynamoDbClient client = createClient(buildConfig(KeyRouteAffinity.ANY_WRITE), liveNodes);
+    try {
+      Map> requestItems = new LinkedHashMap<>();
+      requestItems.put(
+          TABLE_NAME,
+          Collections.singletonList(
+              WriteRequest.builder()
+                  .putRequest(PutRequest.builder().item(makeItem(pkValue)).build())
+                  .build()));
+
+      mockHttpClient.clearCapturedRequests();
+      client.batchWriteItem(BatchWriteItemRequest.builder().requestItems(requestItems).build());
+      URI route = mockHttpClient.getLastRequestUri();
+
+      assertNotEquals("Down preferred node should be skipped", downNode.getHost(), route.getHost());
+      assertSameRoute("Batch write should use next healthy node", expectedAfterSkip, route);
+    } finally {
+      client.close();
+    }
+  }
+
+  @Test
+  public void testPutItemUnsupportedPartitionKeyTypeFallsBackToRandomRouting() {
     DynamoDbClient client = createClient(buildConfig(KeyRouteAffinity.ANY_WRITE));
     try {
       Map item = new HashMap<>();
       item.put(PK_NAME, AttributeValue.builder().bool(true).build());
 
-      assertRoundRobinRouting(
+      assertRandomRouting(
           () -> client.putItem(PutItemRequest.builder().tableName(TABLE_NAME).item(item).build()));
     } finally {
       client.close();
@@ -1860,7 +1925,7 @@ public void testPutItemUnsupportedPartitionKeyTypeFallsBackToRoundRobin() {
   // --- BatchGetItem ---
 
   @Test
-  public void testBatchGetItem_AnyWrite_RoundRobin() {
+  public void testBatchGetItem_AnyWrite_RandomRouting() {
     DynamoDbClient client = createClient(buildConfig(KeyRouteAffinity.ANY_WRITE));
     try {
       Map requestItems = new HashMap<>();
@@ -1868,7 +1933,7 @@ public void testBatchGetItem_AnyWrite_RoundRobin() {
           TABLE_NAME,
           KeysAndAttributes.builder().keys(Collections.singletonList(makeKey())).build());
 
-      assertRoundRobinRouting(
+      assertRandomRouting(
           () ->
               client.batchGetItem(
                   BatchGetItemRequest.builder().requestItems(requestItems).build()));
@@ -1878,7 +1943,7 @@ public void testBatchGetItem_AnyWrite_RoundRobin() {
   }
 
   @Test
-  public void testBatchGetItem_Rmw_RoundRobin() {
+  public void testBatchGetItem_Rmw_RandomRouting() {
     DynamoDbClient client = createClient(buildConfig(KeyRouteAffinity.RMW));
     try {
       Map requestItems = new HashMap<>();
@@ -1886,7 +1951,7 @@ public void testBatchGetItem_Rmw_RoundRobin() {
           TABLE_NAME,
           KeysAndAttributes.builder().keys(Collections.singletonList(makeKey())).build());
 
-      assertRoundRobinRouting(
+      assertRandomRouting(
           () ->
               client.batchGetItem(
                   BatchGetItemRequest.builder().requestItems(requestItems).build()));
@@ -1896,7 +1961,7 @@ public void testBatchGetItem_Rmw_RoundRobin() {
   }
 
   @Test
-  public void testBatchGetItem_None_RoundRobin() {
+  public void testBatchGetItem_None_RandomRouting() {
     DynamoDbClient client = createClient(buildConfig(KeyRouteAffinity.NONE));
     try {
       Map requestItems = new HashMap<>();
@@ -1904,7 +1969,7 @@ public void testBatchGetItem_None_RoundRobin() {
           TABLE_NAME,
           KeysAndAttributes.builder().keys(Collections.singletonList(makeKey())).build());
 
-      assertRoundRobinRouting(
+      assertRandomRouting(
           () ->
               client.batchGetItem(
                   BatchGetItemRequest.builder().requestItems(requestItems).build()));
@@ -1990,11 +2055,11 @@ public void testSameKeyAcrossOperationTypes_Rmw_SameNode() {
   // ========== Null/disabled config ==========
 
   @Test
-  public void testNullConfig_RoundRobin() {
-    // Passing null config should use basic round-robin
+  public void testNullConfig_RandomRouting() {
+    // Passing null config should use basic random query-plan routing
     DynamoDbClient client = createClient(null);
     try {
-      assertRoundRobinRouting(
+      assertRandomRouting(
           () ->
               client.putItem(
                   PutItemRequest.builder().tableName(TABLE_NAME).item(makeItem()).build()));
@@ -2014,41 +2079,41 @@ public void testNullConfig_RoundRobin() {
    *   
  • Does not start any background threads
    *   
  • Provides a fixed list of test nodes
    *   
  • Supports LazyQueryPlan creation for key affinity (via base class)
-   *   
  • Implements round-robin across all test nodes
    * 
    */
   private static class MockAlternatorLiveNodes extends AlternatorLiveNodes {
     private final List nodes;
-    private final java.util.concurrent.atomic.AtomicInteger counter =
-        new java.util.concurrent.atomic.AtomicInteger(0);
 
     MockAlternatorLiveNodes(List nodes) {
+      this(nodes, NodeHealthConfig.getDefault());
+    }
+
+    MockAlternatorLiveNodes(List nodes, NodeHealthConfig nodeHealthConfig) {
       super(
           AlternatorConfig.builder()
-              .withSeedHosts(Collections.singletonList(nodes.get(0).getHost()))
+              .withSeedHosts(nodeHosts(nodes))
               .withScheme(nodes.get(0).getScheme())
               .withPort(nodes.get(0).getPort())
+              .withNodeHealthConfig(nodeHealthConfig)
               .build());
       this.nodes = new ArrayList<>(nodes);
     }
 
     @Override
-    protected List getLiveNodesInternal() {
+    protected List getDiscoveredNodesInternal() {
       return nodes;
     }
 
-    @Override
-    public URI nextAsURI() {
-      return nodes.get(Math.abs(counter.getAndIncrement() % nodes.size()));
-    }
-
     @Override
     public void start() {}
+  }
 
-    @Override
-    public List getLiveNodes() {
-      return Collections.unmodifiableList(new ArrayList<>(nodes));
+  private static List nodeHosts(List nodes) {
+    List hosts = new ArrayList<>();
+    for (URI node : nodes) {
+      hosts.add(node.getHost());
     }
+    return hosts;
   }
 
   /**
diff --git a/src/test/java/com/scylladb/alternator/AlternatorConfigCompatibilityTest.java b/src/test/java/com/scylladb/alternator/AlternatorConfigCompatibilityTest.java
new file mode 100644
index 0000000..6bed4af
--- /dev/null
+++ b/src/test/java/com/scylladb/alternator/AlternatorConfigCompatibilityTest.java
@@ -0,0 +1,60 @@
+package com.scylladb.alternator;
+
+import static org.junit.Assert.*;
+
+import com.scylladb.alternator.routing.ClusterScope;
+import java.net.URI;
+import java.util.Collections;
+import org.junit.Test;
+
+public class AlternatorConfigCompatibilityTest {
+  @Test
+  public void builderWithoutNodeHealthConfigUsesDefaultNodeHealthConfig() {
+    AlternatorConfig config =
+        AlternatorConfig.builder().withSeedNode(URI.create("http://localhost:8000")).build();
+
+    assertNotNull(config.getNodeHealthConfig());
+    assertEquals(
+        NodeHealthConfig.DEFAULT_DOWN_NODE_PROBE_PERIOD_MS,
+        config.getNodeHealthConfig().getDownNodeProbePeriodMs());
+    assertFalse(config.getNodeHealthConfig().isDisabled());
+  }
+
+  @Test
+  public void legacyProtectedConstructorUsesDefaultNodeHealthConfig() {
+    AlternatorConfig config = new LegacyAlternatorConfig();
+
+    assertNotNull(config.getNodeHealthConfig());
+    assertEquals(
+        NodeHealthConfig.DEFAULT_DOWN_NODE_PROBE_PERIOD_MS,
+        config.getNodeHealthConfig().getDownNodeProbePeriodMs());
+    assertFalse(config.getNodeHealthConfig().isDisabled());
+  }
+
+  private static final class LegacyAlternatorConfig extends AlternatorConfig {
+    private LegacyAlternatorConfig() {
+      super(
+          Collections.singletonList("localhost"),
+          "http",
+          8000,
+          ClusterScope.create(),
+          RequestCompressionAlgorithm.NONE,
+          AlternatorConfig.DEFAULT_MIN_COMPRESSION_SIZE_BYTES,
+          Collections.emptyList(),
+          false,
+          null,
+          true,
+          true,
+          null,
+          null,
+          null,
+          AlternatorConfig.DEFAULT_ACTIVE_REFRESH_INTERVAL_MS,
+          AlternatorConfig.DEFAULT_IDLE_REFRESH_INTERVAL_MS,
+          AlternatorConfig.DEFAULT_MAX_CONNECTIONS,
+          AlternatorConfig.DEFAULT_CONNECTION_MAX_IDLE_TIME_MS,
+          AlternatorConfig.DEFAULT_CONNECTION_TIME_TO_LIVE_MS,
+          AlternatorConfig.DEFAULT_CONNECTION_ACQUISITION_TIMEOUT_MS,
+          AlternatorConfig.DEFAULT_CONNECTION_TIMEOUT_MS);
+    }
+  }
+}
diff --git a/src/test/java/com/scylladb/alternator/AlternatorDynamoDbAsyncClientCustomizerTest.java b/src/test/java/com/scylladb/alternator/AlternatorDynamoDbAsyncClientCustomizerTest.java
index 16da972..9a7849d 100644
--- a/src/test/java/com/scylladb/alternator/AlternatorDynamoDbAsyncClientCustomizerTest.java
+++ b/src/test/java/com/scylladb/alternator/AlternatorDynamoDbAsyncClientCustomizerTest.java
@@ -466,6 +466,25 @@ public void testPollingClientAppliesCustomUserAgent() throws Exception {
         delegate.capturedRequest.firstMatchingHeader("User-Agent").get());
   }
 
+  @Test
+  @SuppressWarnings("deprecation")
+  public void testWithAlternatorConfigCopiesNodeHealthConfig() throws Exception {
+    NodeHealthConfig nodeHealthConfig =
+        NodeHealthConfig.builder().withDownNodeProbePeriodMs(1234).build();
+    AlternatorConfig config =
+        AlternatorConfig.builder().withNodeHealthConfig(nodeHealthConfig).build();
+    var builder = AlternatorDynamoDbAsyncClient.builder().withAlternatorConfig(config);
+
+    assertSame(nodeHealthConfig, alternatorConfig(builder).getNodeHealthConfig());
+  }
+
+  @Test
+  public void testWithNodeHealthDisabledConfiguresBuilder() throws Exception {
+    var builder = AlternatorDynamoDbAsyncClient.builder().withNodeHealthDisabled();
+
+    assertTrue(alternatorConfig(builder).getNodeHealthConfig().isDisabled());
+  }
+
   @Test(expected = IllegalStateException.class)
   public void testHttpClientTypeConflictsWithHttpClientBuilder() {
     AlternatorDynamoDbAsyncClient.builder()
@@ -541,4 +560,11 @@ public String clientName() {
       return "capturing";
     }
   }
+
+  private AlternatorConfig alternatorConfig(
+      AlternatorDynamoDbAsyncClient.AlternatorDynamoDbAsyncClientBuilder builder) throws Exception {
+    Field field = builder.getClass().getDeclaredField("configBuilder");
+    field.setAccessible(true);
+    return ((AlternatorConfig.Builder) field.get(builder)).build();
+  }
 }
diff --git a/src/test/java/com/scylladb/alternator/AlternatorDynamoDbClientCustomizerTest.java b/src/test/java/com/scylladb/alternator/AlternatorDynamoDbClientCustomizerTest.java
index b4253e8..5438932 100644
--- a/src/test/java/com/scylladb/alternator/AlternatorDynamoDbClientCustomizerTest.java
+++ b/src/test/java/com/scylladb/alternator/AlternatorDynamoDbClientCustomizerTest.java
@@ -465,6 +465,25 @@ public void testPollingClientAppliesCustomUserAgent() throws Exception {
         delegate.capturedRequest.firstMatchingHeader("User-Agent").get());
   }
 
+  @Test
+  @SuppressWarnings("deprecation")
+  public void testWithAlternatorConfigCopiesNodeHealthConfig() throws Exception {
+    NodeHealthConfig nodeHealthConfig =
+        NodeHealthConfig.builder().withDownNodeProbePeriodMs(1234).build();
+    AlternatorConfig config =
+        AlternatorConfig.builder().withNodeHealthConfig(nodeHealthConfig).build();
+    var builder = AlternatorDynamoDbClient.builder().withAlternatorConfig(config);
+
+    assertSame(nodeHealthConfig, alternatorConfig(builder).getNodeHealthConfig());
+  }
+
+  @Test
+  public void testWithNodeHealthDisabledConfiguresBuilder() throws Exception {
+    var builder = AlternatorDynamoDbClient.builder().withNodeHealthDisabled();
+
+    assertTrue(alternatorConfig(builder).getNodeHealthConfig().isDisabled());
+  }
+
   @Test(expected = IllegalStateException.class)
   public void testHttpClientTypeConflictsWithHttpClientBuilder() {
     AlternatorDynamoDbClient.builder()
@@ -540,4 +559,11 @@ public String clientName() {
       return "capturing";
     }
   }
+
+  private AlternatorConfig alternatorConfig(
+      AlternatorDynamoDbClient.AlternatorDynamoDbClientBuilder builder) throws Exception {
+    Field field = builder.getClass().getDeclaredField("configBuilder");
+    field.setAccessible(true);
+    return ((AlternatorConfig.Builder) field.get(builder)).build();
+  }
 }
diff --git a/src/test/java/com/scylladb/alternator/AlternatorDynamoDbClientWrapperShutdownTest.java b/src/test/java/com/scylladb/alternator/AlternatorDynamoDbClientWrapperShutdownTest.java
index 9953ad7..8473cd4 100644
--- a/src/test/java/com/scylladb/alternator/AlternatorDynamoDbClientWrapperShutdownTest.java
+++ b/src/test/java/com/scylladb/alternator/AlternatorDynamoDbClientWrapperShutdownTest.java
@@ -98,6 +98,32 @@ public void testAsyncWrapperShutsDownAffinityResolverBeforeClosingClients() {
     assertEquals(Arrays.asList("resolver", "live-nodes", "polling-client", "client"), events);
   }
 
+  @Test
+  @SuppressWarnings("deprecation")
+  public void testSyncWrapperDeprecatedNextAsURIDelegatesToLiveNodes() {
+    URI node = URI.create("http://127.0.0.2:8000");
+    AlternatorLiveNodes liveNodes =
+        new AlternatorLiveNodes(
+            AlternatorConfig.builder().withSeedNode(node).build(), new NoopPollingClient());
+    AlternatorDynamoDbClientWrapper wrapper =
+        new AlternatorDynamoDbClientWrapper(mock(DynamoDbClient.class), liveNodes);
+
+    assertEquals(node, wrapper.nextAsURI());
+  }
+
+  @Test
+  @SuppressWarnings("deprecation")
+  public void testAsyncWrapperDeprecatedNextAsURIDelegatesToLiveNodes() {
+    URI node = URI.create("http://127.0.0.2:8000");
+    AlternatorLiveNodes liveNodes =
+        new AlternatorLiveNodes(
+            AlternatorConfig.builder().withSeedNode(node).build(), new NoopPollingClient());
+    AlternatorDynamoDbAsyncClientWrapper wrapper =
+        new AlternatorDynamoDbAsyncClientWrapper(mock(DynamoDbAsyncClient.class), liveNodes);
+
+    assertEquals(node, wrapper.nextAsURI());
+  }
+
   private static final class TrackingLiveNodes extends AlternatorLiveNodes {
     private final List events;
 
diff --git a/src/test/java/com/scylladb/alternator/AlternatorLiveNodesTest.java b/src/test/java/com/scylladb/alternator/AlternatorLiveNodesTest.java
index 739f3ff..b6f643a 100644
--- a/src/test/java/com/scylladb/alternator/AlternatorLiveNodesTest.java
+++ b/src/test/java/com/scylladb/alternator/AlternatorLiveNodesTest.java
@@ -44,36 +44,6 @@ public void testConstructorWithEmptyNodes() {
     new AlternatorLiveNodes(Collections.emptyList(), "http", 8000, "", "");
   }
 
-  @Test
-  public void testNextAsURIRoundRobin() throws URISyntaxException {
-    List nodes =
-        Arrays.asList(
-            new URI("http://node1.example.com:8000"),
-            new URI("http://node2.example.com:8000"),
-            new URI("http://node3.example.com:8000"));
-    AlternatorLiveNodes liveNodes = new AlternatorLiveNodes(nodes, "http", 8000, "", "");
-
-    Set returnedNodes = new HashSet<>();
-    for (int i = 0; i < 6; i++) {
-      returnedNodes.add(liveNodes.nextAsURI());
-    }
-
-    assertEquals("Round-robin should return all nodes", 3, returnedNodes.size());
-  }
-
-  @Test
-  public void testNextAsURIWithPathAndQuery() throws URISyntaxException {
-    URI node = new URI("http://localhost:8000");
-    AlternatorLiveNodes liveNodes = new AlternatorLiveNodes(node, "", "");
-
-    URI result = liveNodes.nextAsURI("/test/path", "key=value");
-
-    assertEquals("/test/path", result.getPath());
-    assertEquals("key=value", result.getQuery());
-    assertEquals("localhost", result.getHost());
-    assertEquals(8000, result.getPort());
-  }
-
   @Test
   public void testGetLiveNodes() throws URISyntaxException {
     List nodes =
@@ -102,6 +72,59 @@ public void testGetLiveNodesReturnsUnmodifiableList() throws URISyntaxException
     }
   }
 
+  @Test
+  public void testGetLiveNodesExcludesDownNodes() throws URISyntaxException {
+    URI active = new URI("http://node1.example.com:8000");
+    URI down = new URI("http://node2.example.com:8000");
+    AlternatorLiveNodes liveNodes =
+        new AlternatorLiveNodes(Arrays.asList(active, down), "http", 8000, "", "");
+
+    markNodeDown(liveNodes, down);
+
+    assertEquals(Arrays.asList(active, down), liveNodes.getDiscoveredNodes());
+    assertEquals(Collections.singletonList(active), liveNodes.getLiveNodes());
+  }
+
+  @Test
+  @SuppressWarnings("deprecation")
+  public void testDeprecatedNextAsURIUsesQueryPlan() throws URISyntaxException {
+    URI active = new URI("http://node1.example.com:8000");
+    URI down = new URI("http://node2.example.com:8000");
+    AlternatorLiveNodes liveNodes =
+        new AlternatorLiveNodes(Arrays.asList(active, down), "http", 8000, "", "");
+
+    markNodeDown(liveNodes, down);
+
+    assertEquals(active, liveNodes.nextAsURI());
+  }
+
+  @Test
+  @SuppressWarnings("deprecation")
+  public void testDeprecatedNextAsURIWithPathAndQuery() throws URISyntaxException {
+    URI node = new URI("http://node1.example.com:8000");
+    AlternatorLiveNodes liveNodes = new AlternatorLiveNodes(node, "", "");
+
+    URI result = liveNodes.nextAsURI("/localnodes", "dc=dc1");
+
+    assertEquals("http", result.getScheme());
+    assertEquals("node1.example.com", result.getHost());
+    assertEquals(8000, result.getPort());
+    assertEquals("/localnodes", result.getPath());
+    assertEquals("dc=dc1", result.getQuery());
+  }
+
+  @Test
+  public void testDeprecatedLiveNodesInternalOverrideFeedsQueryPlan() throws URISyntaxException {
+    URI seed = new URI("http://seed.example.com:8000");
+    URI overrideNode = new URI("http://override.example.com:8000");
+    LegacyLiveNodesOverride liveNodes =
+        new LegacyLiveNodesOverride(seed, Collections.singletonList(overrideNode));
+
+    assertEquals(
+        Collections.singletonList(overrideNode), collectNodes(new LazyQueryPlan(liveNodes)));
+    assertEquals(Collections.singletonList(seed), liveNodes.baseLiveNodesInternal());
+  }
+
   @Test
   public void testNewQueryPlan() throws URISyntaxException {
     List nodes =
@@ -178,9 +201,42 @@ public void testHttpsScheme() throws URISyntaxException {
     URI node = new URI("https://localhost:8043");
     AlternatorLiveNodes liveNodes = new AlternatorLiveNodes(node, "", "");
 
-    URI result = liveNodes.nextAsURI();
+    URI result = new LazyQueryPlan(liveNodes).next();
 
     assertEquals("https", result.getScheme());
     assertEquals(8043, result.getPort());
   }
+
+  private static List collectNodes(LazyQueryPlan plan) {
+    List collected = new ArrayList<>();
+    while (plan.hasNext()) {
+      collected.add(plan.next());
+    }
+    return collected;
+  }
+
+  private static void markNodeDown(AlternatorLiveNodes liveNodes, URI node) {
+    for (int i = 0; i < NodeHealthConfig.DEFAULT_CONSECUTIVE_FAILURE_THRESHOLD; i++) {
+      liveNodes.reportNodeResult(node, NodeHealthObservation.TRAFFIC_FAILURE);
+    }
+  }
+
+  private static final class LegacyLiveNodesOverride extends AlternatorLiveNodes {
+    private final List nodes;
+
+    private LegacyLiveNodesOverride(URI seed, List nodes) {
+      super(seed, "", "");
+      this.nodes = new ArrayList<>(nodes);
+    }
+
+    @Override
+    @Deprecated
+    protected List getLiveNodesInternal() {
+      return nodes;
+    }
+
+    private List baseLiveNodesInternal() {
+      return super.getLiveNodesInternal();
+    }
+  }
 }
diff --git a/src/test/java/com/scylladb/alternator/DynamoDbEnhancedClientTest.java b/src/test/java/com/scylladb/alternator/DynamoDbEnhancedClientTest.java
index de7aaf6..e97b62e 100644
--- a/src/test/java/com/scylladb/alternator/DynamoDbEnhancedClientTest.java
+++ b/src/test/java/com/scylladb/alternator/DynamoDbEnhancedClientTest.java
@@ -97,13 +97,25 @@ private static List createLocalNodes() throws IOException, URISyntaxExcepti
         new URI("http://127.0.0.3:" + port));
   }
 
+  private static AlternatorLiveNodes liveNodesWithHealthDisabled(List nodes) {
+    AlternatorConfig config =
+        AlternatorConfig.builder()
+            .withSeedHosts(
+                Arrays.asList(
+                    nodes.get(0).getHost(), nodes.get(1).getHost(), nodes.get(2).getHost()))
+            .withScheme(nodes.get(0).getScheme())
+            .withPort(nodes.get(0).getPort())
+            .withNodeHealthConfig(NodeHealthConfig.disabled())
+            .build();
+    return new AlternatorLiveNodes(config);
+  }
+
   @Test
   public void testEnhancedClientCanBeBuiltFromAlternatorClient()
       throws URISyntaxException, IOException {
     List nodes = createLocalNodes();
 
-    AlternatorLiveNodes liveNodes =
-        new AlternatorLiveNodes(nodes, "http", nodes.get(0).getPort(), "", "");
+    AlternatorLiveNodes liveNodes = liveNodesWithHealthDisabled(nodes);
     BasicQueryPlanInterceptor interceptor = new BasicQueryPlanInterceptor(liveNodes);
 
     ClientOverrideConfiguration overrideConfig =
@@ -136,8 +148,7 @@ public void testEnhancedAsyncClientCanBeBuiltFromAlternatorAsyncClient()
       throws URISyntaxException, IOException {
     List nodes = createLocalNodes();
 
-    AlternatorLiveNodes liveNodes =
-        new AlternatorLiveNodes(nodes, "http", nodes.get(0).getPort(), "", "");
+    AlternatorLiveNodes liveNodes = liveNodesWithHealthDisabled(nodes);
     BasicQueryPlanInterceptor interceptor = new BasicQueryPlanInterceptor(liveNodes);
 
     ClientOverrideConfiguration overrideConfig =
@@ -164,8 +175,7 @@ public void testEnhancedAsyncClientCanBeBuiltFromAlternatorAsyncClient()
   public void testLoadBalancingWorksWithEnhancedClient() throws URISyntaxException, IOException {
     List nodes = createLocalNodes();
 
-    AlternatorLiveNodes liveNodes =
-        new AlternatorLiveNodes(nodes, "http", nodes.get(0).getPort(), "", "");
+    AlternatorLiveNodes liveNodes = liveNodesWithHealthDisabled(nodes);
     BasicQueryPlanInterceptor interceptor = new BasicQueryPlanInterceptor(liveNodes);
 
     // Track which hosts are being targeted
diff --git a/src/test/java/com/scylladb/alternator/KeyRouteAffinityClientTest.java b/src/test/java/com/scylladb/alternator/KeyRouteAffinityClientTest.java
index 416fa33..0f7a515 100644
--- a/src/test/java/com/scylladb/alternator/KeyRouteAffinityClientTest.java
+++ b/src/test/java/com/scylladb/alternator/KeyRouteAffinityClientTest.java
@@ -307,7 +307,7 @@ public void testRmwModeOnlyAppliesToConditionalWrites() {
     try {
       String partitionKeyValue = "user-123";
 
-      // Simple PutItem without conditions - uses round-robin, may hit different nodes
+      // Simple PutItem without conditions - uses random query-plan routing, may hit different nodes
       Set nodesForSimplePut = new HashSet<>();
       for (int i = 0; i < 10; i++) {
         Map item = new HashMap<>();
@@ -352,9 +352,9 @@ public void testRmwModeOnlyAppliesToConditionalWrites() {
       // Verify conditional writes go to one consistent node
       assertNotNull("Conditional writes should have a target node", expectedNodeForConditional);
 
-      // Simple puts should use round-robin (multiple nodes) since RMW doesn't apply
+      // Simple puts should use random query-plan routing (multiple nodes) since RMW doesn't apply
       assertTrue(
-          "Simple puts in RMW mode should use round-robin (got "
+          "Simple puts in RMW mode should use random query-plan routing (got "
               + nodesForSimplePut.size()
               + " nodes)",
           nodesForSimplePut.size() > 1);
@@ -555,7 +555,7 @@ public void testHashDistributionAcrossNodes() {
   // ========== Tests for NONE mode ==========
 
   @Test
-  public void testNoneModeUsesRoundRobin() {
+  public void testNoneModeUsesRandomRouting() {
     KeyRouteAffinityConfig keyAffinity =
         KeyRouteAffinityConfig.builder()
             .withType(KeyRouteAffinity.NONE)
@@ -565,7 +565,7 @@ public void testNoneModeUsesRoundRobin() {
     DynamoDbClient client = createClientWithKeyAffinity(keyAffinity);
 
     try {
-      // In NONE mode, same key should use round-robin across nodes
+      // In NONE mode, same key should use random query-plan routing across nodes
       Set nodesUsed = new HashSet<>();
       String partitionKey = "user-same-key";
 
@@ -581,9 +581,11 @@ public void testNoneModeUsesRoundRobin() {
         nodesUsed.add(mockHttpClient.getLastRequestUri());
       }
 
-      // With 20 requests in round-robin mode across 5 nodes, we should see multiple nodes
+      // With 20 requests in random query-plan mode across 5 nodes, we should see multiple nodes
       assertTrue(
-          "NONE mode should use round-robin across multiple nodes (got " + nodesUsed.size() + ")",
+          "NONE mode should use random query-plan routing across multiple nodes (got "
+              + nodesUsed.size()
+              + ")",
           nodesUsed.size() > 1);
     } finally {
       client.close();
@@ -855,8 +857,8 @@ public void testRmwModeWithConditionExpressionOnDelete() {
   }
 
   @Test
-  public void testRmwModeSimpleDeleteUsesRoundRobin() {
-    // Simple DeleteItem without conditions should use round-robin in RMW mode
+  public void testRmwModeSimpleDeleteUsesRandomRouting() {
+    // Simple DeleteItem without conditions should use random query-plan routing in RMW mode
     KeyRouteAffinityConfig keyAffinity =
         KeyRouteAffinityConfig.builder()
             .withType(KeyRouteAffinity.RMW)
@@ -882,9 +884,11 @@ public void testRmwModeSimpleDeleteUsesRoundRobin() {
         nodesUsed.add(mockHttpClient.getLastRequestUri());
       }
 
-      // Simple deletes in RMW mode should use round-robin
+      // Simple deletes in RMW mode should use random query-plan routing
       assertTrue(
-          "Simple delete in RMW mode should use round-robin (got " + nodesUsed.size() + " nodes)",
+          "Simple delete in RMW mode should use random query-plan routing (got "
+              + nodesUsed.size()
+              + " nodes)",
           nodesUsed.size() > 1);
     } finally {
       client.close();
@@ -923,9 +927,11 @@ public void testRmwModeWithReturnValuesUpdatedNew_DoesNotTrigger() {
         nodesUsed.add(mockHttpClient.getLastRequestUri());
       }
 
-      // UPDATED_NEW alone doesn't trigger RMW, should use round-robin
+      // UPDATED_NEW alone doesn't trigger RMW, should use random query-plan routing
       assertTrue(
-          "UPDATED_NEW alone in RMW mode should use round-robin (got " + nodesUsed.size() + ")",
+          "UPDATED_NEW alone in RMW mode should use random query-plan routing (got "
+              + nodesUsed.size()
+              + ")",
           nodesUsed.size() > 1);
     } finally {
       client.close();
@@ -988,13 +994,10 @@ public void testAsyncClientAcceptsNoneMode() {
    *   
  • Does not start any background threads
    *   
  • Provides a fixed list of test nodes
    *   
  • Supports LazyQueryPlan creation for key affinity (via base class)
-   *   
  • Implements round-robin across all test nodes
    * 
    */
   private static class MockAlternatorLiveNodes extends AlternatorLiveNodes {
     private final List nodes;
-    private final java.util.concurrent.atomic.AtomicInteger counter =
-        new java.util.concurrent.atomic.AtomicInteger(0);
 
     MockAlternatorLiveNodes(List nodes) {
       super(
@@ -1007,25 +1010,14 @@ private static class MockAlternatorLiveNodes extends AlternatorLiveNodes {
     }
 
     @Override
-    protected List getLiveNodesInternal() {
+    protected List getDiscoveredNodesInternal() {
       return nodes;
     }
 
-    @Override
-    public URI nextAsURI() {
-      // Round-robin through all test nodes (not just seed node)
-      return nodes.get(Math.abs(counter.getAndIncrement() % nodes.size()));
-    }
-
     @Override
     public void start() {
       // Don't start background thread in tests
     }
-
-    @Override
-    public List getLiveNodes() {
-      return Collections.unmodifiableList(new ArrayList<>(nodes));
-    }
   }
 
   /**
diff --git a/src/test/java/com/scylladb/alternator/LazyQueryPlanCrossLanguageTest.java b/src/test/java/com/scylladb/alternator/LazyQueryPlanCrossLanguageTest.java
index f5767e1..8ab6b41 100644
--- a/src/test/java/com/scylladb/alternator/LazyQueryPlanCrossLanguageTest.java
+++ b/src/test/java/com/scylladb/alternator/LazyQueryPlanCrossLanguageTest.java
@@ -41,7 +41,7 @@ private static List createNodes(String prefix, int count) throws URISyntaxE
 
   private static AlternatorLiveNodes createLiveNodes(int activeCount, int quarantinedCount)
       throws URISyntaxException {
-    // Only active nodes are in the live nodes list; quarantined nodes are excluded
+    // Cross-language vectors use only the canonical known-node list.
     List active = createNodes("node", activeCount);
     return new AlternatorLiveNodes(active, SCHEME, PORT, "", "");
   }
diff --git a/src/test/java/com/scylladb/alternator/LazyQueryPlanTest.java b/src/test/java/com/scylladb/alternator/LazyQueryPlanTest.java
index 10168b1..8f1c015 100644
--- a/src/test/java/com/scylladb/alternator/LazyQueryPlanTest.java
+++ b/src/test/java/com/scylladb/alternator/LazyQueryPlanTest.java
@@ -224,6 +224,22 @@ public void testPreferredNodesAreReturnedBeforeSortedRemaining() {
         actual);
   }
 
+  @Test
+  @SuppressWarnings("deprecation")
+  public void testDeprecatedSortedAffinityNodesForwardsToLiveNodes() {
+    assertEquals(liveNodes.getQueryPlanNodes(), LazyQueryPlan.sortedAffinityNodes(liveNodes));
+  }
+
+  @Test
+  @SuppressWarnings("deprecation")
+  public void testDeprecatedPreferredNodeForHashForwardsToLiveNodes() {
+    long seed = 42L;
+
+    assertEquals(
+        liveNodes.getPreferredQueryPlanNodeForHash(seed),
+        LazyQueryPlan.preferredNodeForHash(liveNodes, seed));
+  }
+
   @Test
   public void testSeededAffinityUsesSortedNodeOrder() throws URISyntaxException {
     List unsortedNodes =
@@ -255,4 +271,62 @@ public void testSeededAffinityUsesSortedNodeOrder() throws URISyntaxException {
 
     assertEquals(sortedSequence, unsortedSequence);
   }
+
+  @Test
+  public void testSeededAffinityKeepsDownPreferredNodeInStableKnownOrder() {
+    URI downNode = nodes.get(2);
+    long seed = seedWhereFirstNodeIs(nodes, downNode);
+    List allHealthyOrder = collectNodes(new LazyQueryPlan(liveNodes, seed));
+
+    markNodeDown(liveNodes, downNode);
+
+    List degradedOrder = collectNodes(new LazyQueryPlan(liveNodes, seed));
+    assertEquals("Health must not change seeded candidate order", allHealthyOrder, degradedOrder);
+    assertEquals("Down preferred node stays in the plan", downNode, degradedOrder.get(0));
+  }
+
+  @Test
+  public void testSeededAffinityDoesNotRemapUnaffectedHealthyPreferredNode() {
+    URI preferredNode = nodes.get(0);
+    URI unrelatedDownNode = nodes.get(1);
+    long seed = seedWhereFirstNodeIs(nodes, preferredNode);
+
+    markNodeDown(liveNodes, unrelatedDownNode);
+
+    LazyQueryPlan degradedPlan = new LazyQueryPlan(liveNodes, seed);
+    assertEquals(
+        "Healthy preferred node should remain first when another node is down",
+        preferredNode,
+        degradedPlan.next());
+  }
+
+  private long seedWhereFirstNodeIs(List candidates, URI expected) {
+    for (long seed = -1000; seed < 1000; seed++) {
+      if (expected.equals(firstNodeForSeed(candidates, seed))) {
+        return seed;
+      }
+    }
+    fail("Could not find seed for " + expected);
+    return 0;
+  }
+
+  private URI firstNodeForSeed(List candidates, long seed) {
+    LazyQueryPlan plan =
+        new LazyQueryPlan(new AlternatorLiveNodes(candidates, "http", 8000, "", ""), seed);
+    return plan.hasNext() ? plan.next() : null;
+  }
+
+  private List collectNodes(LazyQueryPlan plan) {
+    List collected = new ArrayList<>();
+    while (plan.hasNext()) {
+      collected.add(plan.next());
+    }
+    return collected;
+  }
+
+  private void markNodeDown(AlternatorLiveNodes liveNodes, URI node) {
+    for (int i = 0; i < NodeHealthConfig.DEFAULT_CONSECUTIVE_FAILURE_THRESHOLD; i++) {
+      liveNodes.reportNodeResult(node, NodeHealthObservation.TRAFFIC_FAILURE);
+    }
+  }
 }
diff --git a/src/test/java/com/scylladb/alternator/NodeHealthConfigTest.java b/src/test/java/com/scylladb/alternator/NodeHealthConfigTest.java
new file mode 100644
index 0000000..f692ca7
--- /dev/null
+++ b/src/test/java/com/scylladb/alternator/NodeHealthConfigTest.java
@@ -0,0 +1,14 @@
+package com.scylladb.alternator;
+
+import static org.junit.Assert.*;
+
+import org.junit.Test;
+
+public class NodeHealthConfigTest {
+  @Test
+  public void defaultConsecutiveFailureThresholdIsTen() {
+    assertEquals(10, NodeHealthConfig.DEFAULT_CONSECUTIVE_FAILURE_THRESHOLD);
+    assertEquals(10, NodeHealthConfig.getDefault().getConsecutiveFailureThreshold());
+    assertEquals(10, NodeHealthConfig.builder().build().getConsecutiveFailureThreshold());
+  }
+}
diff --git a/src/test/java/com/scylladb/alternator/RetryDistributionTest.java b/src/test/java/com/scylladb/alternator/RetryDistributionTest.java
index 42196d0..c06c730 100644
--- a/src/test/java/com/scylladb/alternator/RetryDistributionTest.java
+++ b/src/test/java/com/scylladb/alternator/RetryDistributionTest.java
@@ -3,13 +3,20 @@
 import static org.junit.Assert.*;
 
 import com.scylladb.alternator.internal.AlternatorLiveNodes;
+import com.scylladb.alternator.internal.LazyQueryPlan;
 import com.scylladb.alternator.queryplan.BasicQueryPlanInterceptor;
 import java.net.URI;
 import java.util.*;
 import java.util.stream.Collectors;
 import org.junit.Test;
+import software.amazon.awssdk.awscore.exception.AwsErrorDetails;
+import software.amazon.awssdk.core.SdkRequest;
+import software.amazon.awssdk.core.SdkResponse;
 import software.amazon.awssdk.core.interceptor.ExecutionAttributes;
+import software.amazon.awssdk.http.SdkHttpFullResponse;
 import software.amazon.awssdk.http.SdkHttpRequest;
+import software.amazon.awssdk.http.SdkHttpResponse;
+import software.amazon.awssdk.services.dynamodb.model.DynamoDbException;
 
 /**
  * Tests that verify retry distribution across nodes via {@link
@@ -167,7 +174,7 @@ public void testThreeNodesTwoRetries() throws Exception {
   /**
    * With only one node and two retries, all attempts must go to the same node since it is the only
    * one available. After the LazyQueryPlan is exhausted, modifyHttpRequest returns the original
-   * request (which still targets the first node via endpointOverride).
+   * request only when that endpoint is still eligible for routing.
    */
   @Test
   public void testSingleNodeMultipleRetries() throws Exception {
@@ -247,8 +254,8 @@ public void testRetryDistributionAcrossMultipleRequests() throws Exception {
 
   /**
    * Verifies that with more retry attempts than available nodes, the plan is exhausted and
-   * subsequent attempts return the original request. This tests the boundary condition where
-   * retries exceed the node count.
+   * subsequent attempts return the original request when that endpoint is still eligible. This
+   * tests the boundary condition where retries exceed the node count.
    */
   @Test
   public void testRetriesExceedNodeCount() throws Exception {
@@ -281,8 +288,356 @@ public void testRetriesExceedNodeCount() throws Exception {
     assertEquals("After plan exhaustion, returns original request port", 9999, third.port());
   }
 
+  @Test
+  public void testRetryReportsPreviousAttemptTransportFailure() throws Exception {
+    List nodes = createNodes(2);
+    MockAlternatorLiveNodes liveNodes = new MockAlternatorLiveNodes(nodes);
+    BasicQueryPlanInterceptor interceptor = new BasicQueryPlanInterceptor(liveNodes);
+
+    ExecutionAttributes attrs = ExecutionAttributes.builder().build();
+    interceptor.beforeExecution(null, attrs);
+    MockModifyHttpRequestContext context = new MockModifyHttpRequestContext(baseRequest());
+
+    SdkHttpRequest first = interceptor.modifyHttpRequest(context, attrs);
+    interceptor.beforeTransmission(new MockBeforeTransmissionContext(first), attrs);
+    assertTrue(liveNodes.reports.isEmpty());
+
+    interceptor.beforeTransmission(new MockBeforeTransmissionContext(first), attrs);
+
+    URI firstNode = endpoint(first);
+    assertEquals(1, liveNodes.reports.size());
+    assertEquals(
+        new NodeReport(firstNode, NodeHealthObservation.TRAFFIC_FAILURE), liveNodes.reports.get(0));
+  }
+
+  @Test
+  public void testDynamoDbHttpErrorResponseClearsPendingTransportFailure() throws Exception {
+    List nodes = createNodes(2);
+    MockAlternatorLiveNodes liveNodes = new MockAlternatorLiveNodes(nodes);
+    BasicQueryPlanInterceptor interceptor = new BasicQueryPlanInterceptor(liveNodes);
+
+    ExecutionAttributes attrs = ExecutionAttributes.builder().build();
+    interceptor.beforeExecution(null, attrs);
+    MockModifyHttpRequestContext context = new MockModifyHttpRequestContext(baseRequest());
+
+    SdkHttpRequest first = interceptor.modifyHttpRequest(context, attrs);
+    interceptor.beforeTransmission(new MockBeforeTransmissionContext(first), attrs);
+    interceptor.afterTransmission(new MockAfterTransmissionContext(first, 500), attrs);
+    assertTrue(liveNodes.reports.isEmpty());
+    interceptor.onExecutionFailure(
+        new MockFailedExecutionContext(first, 500, dynamoDbException(500, "InternalServerError")),
+        attrs);
+
+    SdkHttpRequest second = interceptor.modifyHttpRequest(context, attrs);
+    interceptor.beforeTransmission(new MockBeforeTransmissionContext(second), attrs);
+
+    URI firstNode = endpoint(first);
+    URI secondNode = endpoint(second);
+    assertNotEquals(firstNode, secondNode);
+    assertEquals(1, liveNodes.reports.size());
+    assertEquals(
+        new NodeReport(firstNode, NodeHealthObservation.TRAFFIC_SUCCESS), liveNodes.reports.get(0));
+  }
+
+  @Test
+  public void testRetryableHttpResponseClearsBeforeNextRetryWithoutFailedExecution()
+      throws Exception {
+    List nodes = createNodes(2);
+    MockAlternatorLiveNodes liveNodes = new MockAlternatorLiveNodes(nodes);
+    BasicQueryPlanInterceptor interceptor = new BasicQueryPlanInterceptor(liveNodes);
+
+    ExecutionAttributes attrs = ExecutionAttributes.builder().build();
+    interceptor.beforeExecution(null, attrs);
+    MockModifyHttpRequestContext context = new MockModifyHttpRequestContext(baseRequest());
+
+    SdkHttpRequest first = interceptor.modifyHttpRequest(context, attrs);
+    interceptor.beforeTransmission(new MockBeforeTransmissionContext(first), attrs);
+    interceptor.afterTransmission(new MockAfterTransmissionContext(first, 500), attrs);
+    assertTrue(liveNodes.reports.isEmpty());
+
+    SdkHttpRequest second = interceptor.modifyHttpRequest(context, attrs);
+    interceptor.beforeTransmission(new MockBeforeTransmissionContext(second), attrs);
+
+    URI firstNode = endpoint(first);
+    URI secondNode = endpoint(second);
+    assertNotEquals(firstNode, secondNode);
+    assertEquals(1, liveNodes.reports.size());
+    assertEquals(
+        new NodeReport(firstNode, NodeHealthObservation.TRAFFIC_SUCCESS), liveNodes.reports.get(0));
+  }
+
+  @Test
+  public void testDynamoDbAuthenticationErrorCodeCountsAsFailure() throws Exception {
+    List nodes = createNodes(2);
+    MockAlternatorLiveNodes liveNodes = new MockAlternatorLiveNodes(nodes);
+    BasicQueryPlanInterceptor interceptor = new BasicQueryPlanInterceptor(liveNodes);
+
+    ExecutionAttributes attrs = ExecutionAttributes.builder().build();
+    interceptor.beforeExecution(null, attrs);
+    MockModifyHttpRequestContext context = new MockModifyHttpRequestContext(baseRequest());
+
+    SdkHttpRequest first = interceptor.modifyHttpRequest(context, attrs);
+    interceptor.beforeTransmission(new MockBeforeTransmissionContext(first), attrs);
+    interceptor.afterTransmission(new MockAfterTransmissionContext(first, 400), attrs);
+    assertTrue(liveNodes.reports.isEmpty());
+    interceptor.onExecutionFailure(
+        new MockFailedExecutionContext(
+            first,
+            400,
+            dynamoDbException(
+                400, "com.amazonaws.dynamodb.v20120810#MissingAuthenticationTokenException")),
+        attrs);
+
+    URI firstNode = endpoint(first);
+    assertEquals(1, liveNodes.reports.size());
+    assertEquals(
+        new NodeReport(firstNode, NodeHealthObservation.TRAFFIC_FAILURE), liveNodes.reports.get(0));
+  }
+
+  @Test
+  public void testDynamoDbAuthenticationHttpStatusCountsAsFailure() throws Exception {
+    List nodes = createNodes(2);
+    MockAlternatorLiveNodes liveNodes = new MockAlternatorLiveNodes(nodes);
+    BasicQueryPlanInterceptor interceptor = new BasicQueryPlanInterceptor(liveNodes);
+
+    ExecutionAttributes attrs = ExecutionAttributes.builder().build();
+    interceptor.beforeExecution(null, attrs);
+    MockModifyHttpRequestContext context = new MockModifyHttpRequestContext(baseRequest());
+
+    SdkHttpRequest first = interceptor.modifyHttpRequest(context, attrs);
+    interceptor.beforeTransmission(new MockBeforeTransmissionContext(first), attrs);
+    interceptor.afterTransmission(new MockAfterTransmissionContext(first, 403), attrs);
+
+    URI firstNode = endpoint(first);
+    assertEquals(1, liveNodes.reports.size());
+    assertEquals(
+        new NodeReport(firstNode, NodeHealthObservation.TRAFFIC_FAILURE), liveNodes.reports.get(0));
+  }
+
+  @Test
+  public void testModifyHttpRequestSkipsDownCandidateAtFinalGate() throws Exception {
+    List nodes = createNodes(2);
+    URI down = nodes.get(0);
+    URI active = nodes.get(1);
+    MockAlternatorLiveNodes liveNodes =
+        new MockAlternatorLiveNodes(
+            nodes, NodeHealthConfig.builder().withConsecutiveFailureThreshold(1).build());
+    liveNodes.reportNodeResult(down, NodeHealthObservation.TRAFFIC_FAILURE);
+    liveNodes.reports.clear();
+    TestableBasicQueryPlanInterceptor interceptor =
+        new TestableBasicQueryPlanInterceptor(liveNodes);
+
+    ExecutionAttributes attrs = ExecutionAttributes.builder().build();
+    interceptor.setQueryPlan(
+        attrs, new LazyQueryPlan(liveNodes, seedWhereFirstNodeIs(nodes, down)));
+
+    SdkHttpRequest routed =
+        interceptor.modifyHttpRequest(new MockModifyHttpRequestContext(baseRequest()), attrs);
+
+    assertEquals(active, endpoint(routed));
+  }
+
+  @Test
+  public void testModifyHttpRequestRoutesToKnownDownNodeWhenNoActiveNodes() throws Exception {
+    List nodes = createNodes(1);
+    URI down = nodes.get(0);
+    MockAlternatorLiveNodes liveNodes =
+        new MockAlternatorLiveNodes(
+            nodes, NodeHealthConfig.builder().withConsecutiveFailureThreshold(1).build());
+    liveNodes.reportNodeResult(down, NodeHealthObservation.TRAFFIC_FAILURE);
+    TestableBasicQueryPlanInterceptor interceptor =
+        new TestableBasicQueryPlanInterceptor(liveNodes);
+
+    ExecutionAttributes attrs = ExecutionAttributes.builder().build();
+    interceptor.setQueryPlan(attrs, new LazyQueryPlan(liveNodes, 0));
+
+    SdkHttpRequest originalRequest =
+        SdkHttpRequest.builder()
+            .protocol(down.getScheme())
+            .host(down.getHost())
+            .port(down.getPort())
+            .method(software.amazon.awssdk.http.SdkHttpMethod.POST)
+            .encodedPath("/")
+            .build();
+
+    SdkHttpRequest routed =
+        interceptor.modifyHttpRequest(new MockModifyHttpRequestContext(originalRequest), attrs);
+
+    assertEquals(down, endpoint(routed));
+    assertEquals(NodeHealthState.DOWN, liveNodes.getNodeHealthStatus(down).getState());
+  }
+
+  @Test
+  public void testModifyHttpRequestRetriesAcrossAllKnownDownNodesWhenNoActiveNodes()
+      throws Exception {
+    List nodes = createNodes(2);
+    MockAlternatorLiveNodes liveNodes =
+        new MockAlternatorLiveNodes(
+            nodes, NodeHealthConfig.builder().withConsecutiveFailureThreshold(1).build());
+    liveNodes.reportNodeResult(nodes.get(0), NodeHealthObservation.TRAFFIC_FAILURE);
+    liveNodes.reportNodeResult(nodes.get(1), NodeHealthObservation.TRAFFIC_FAILURE);
+    TestableBasicQueryPlanInterceptor interceptor =
+        new TestableBasicQueryPlanInterceptor(liveNodes);
+
+    ExecutionAttributes attrs = ExecutionAttributes.builder().build();
+    interceptor.setQueryPlan(attrs, new LazyQueryPlan(liveNodes, 0));
+    MockModifyHttpRequestContext context = new MockModifyHttpRequestContext(baseRequest());
+
+    SdkHttpRequest first = interceptor.modifyHttpRequest(context, attrs);
+    SdkHttpRequest second = interceptor.modifyHttpRequest(context, attrs);
+
+    assertEquals(
+        new HashSet<>(nodes), new HashSet<>(Arrays.asList(endpoint(first), endpoint(second))));
+    assertEquals(NodeHealthState.DOWN, liveNodes.getNodeHealthStatus(nodes.get(0)).getState());
+    assertEquals(NodeHealthState.DOWN, liveNodes.getNodeHealthStatus(nodes.get(1)).getState());
+  }
+
+  @Test
+  public void testModifyHttpRequestSamplesQuarantinedCandidateAtFinalGate() throws Exception {
+    List nodes = createNodes(2);
+    URI recovering = nodes.get(0);
+    URI active = nodes.get(1);
+    MockAlternatorLiveNodes liveNodes =
+        new MockAlternatorLiveNodes(
+            nodes,
+            NodeHealthConfig.builder()
+                .withConsecutiveFailureThreshold(1)
+                .withDownNodeRecoverySuccessThreshold(1)
+                .withQuarantineTrafficInterval(2)
+                .build());
+    liveNodes.reportNodeResult(recovering, NodeHealthObservation.TRAFFIC_FAILURE);
+    liveNodes.reportNodeResult(recovering, NodeHealthObservation.PROBE_SUCCESS);
+    liveNodes.reports.clear();
+    long seed = seedWhereFirstNodeIs(nodes, recovering);
+    TestableBasicQueryPlanInterceptor interceptor =
+        new TestableBasicQueryPlanInterceptor(liveNodes);
+
+    ExecutionAttributes firstAttrs = ExecutionAttributes.builder().build();
+    interceptor.setQueryPlan(firstAttrs, new LazyQueryPlan(liveNodes, seed));
+    SdkHttpRequest first =
+        interceptor.modifyHttpRequest(new MockModifyHttpRequestContext(baseRequest()), firstAttrs);
+
+    ExecutionAttributes secondAttrs = ExecutionAttributes.builder().build();
+    interceptor.setQueryPlan(secondAttrs, new LazyQueryPlan(liveNodes, seed));
+    SdkHttpRequest second =
+        interceptor.modifyHttpRequest(new MockModifyHttpRequestContext(baseRequest()), secondAttrs);
+
+    assertEquals(active, endpoint(first));
+    assertEquals(recovering, endpoint(second));
+  }
+
+  @Test
+  public void testModifyHttpRequestSamplesQuarantinedCandidateAfterActiveCandidate()
+      throws Exception {
+    List nodes = createNodes(2);
+    URI recovering = nodes.get(0);
+    URI active = nodes.get(1);
+    MockAlternatorLiveNodes liveNodes =
+        new MockAlternatorLiveNodes(
+            nodes,
+            NodeHealthConfig.builder()
+                .withConsecutiveFailureThreshold(1)
+                .withDownNodeRecoverySuccessThreshold(1)
+                .withQuarantineTrafficInterval(2)
+                .build());
+    liveNodes.reportNodeResult(recovering, NodeHealthObservation.TRAFFIC_FAILURE);
+    liveNodes.reportNodeResult(recovering, NodeHealthObservation.PROBE_SUCCESS);
+    liveNodes.reports.clear();
+    long seed = seedWhereFirstNodeIs(nodes, active);
+    TestableBasicQueryPlanInterceptor interceptor =
+        new TestableBasicQueryPlanInterceptor(liveNodes);
+
+    ExecutionAttributes firstAttrs = ExecutionAttributes.builder().build();
+    interceptor.setQueryPlan(firstAttrs, new LazyQueryPlan(liveNodes, seed));
+    SdkHttpRequest first =
+        interceptor.modifyHttpRequest(new MockModifyHttpRequestContext(baseRequest()), firstAttrs);
+
+    ExecutionAttributes secondAttrs = ExecutionAttributes.builder().build();
+    interceptor.setQueryPlan(secondAttrs, new LazyQueryPlan(liveNodes, seed));
+    SdkHttpRequest second =
+        interceptor.modifyHttpRequest(new MockModifyHttpRequestContext(baseRequest()), secondAttrs);
+
+    assertEquals(active, endpoint(first));
+    assertEquals(recovering, endpoint(second));
+  }
+
+  @Test
+  public void testModifyHttpRequestPreservesSkippedActiveCandidateAfterQuarantineSample()
+      throws Exception {
+    List nodes = createNodes(2);
+    URI active = nodes.get(0);
+    URI recovering = nodes.get(1);
+    MockAlternatorLiveNodes liveNodes =
+        new MockAlternatorLiveNodes(
+            nodes,
+            NodeHealthConfig.builder()
+                .withConsecutiveFailureThreshold(1)
+                .withDownNodeRecoverySuccessThreshold(1)
+                .withQuarantineTrafficInterval(1)
+                .build());
+    liveNodes.reportNodeResult(recovering, NodeHealthObservation.TRAFFIC_FAILURE);
+    liveNodes.reportNodeResult(recovering, NodeHealthObservation.PROBE_SUCCESS);
+    liveNodes.reports.clear();
+    long seed = seedWhereFirstNodeIs(nodes, active);
+    TestableBasicQueryPlanInterceptor interceptor =
+        new TestableBasicQueryPlanInterceptor(liveNodes);
+
+    ExecutionAttributes attrs = ExecutionAttributes.builder().build();
+    interceptor.setQueryPlan(attrs, new LazyQueryPlan(liveNodes, seed));
+    MockModifyHttpRequestContext context = new MockModifyHttpRequestContext(baseRequest());
+
+    SdkHttpRequest first = interceptor.modifyHttpRequest(context, attrs);
+    liveNodes.reportNodeResult(recovering, NodeHealthObservation.TRAFFIC_FAILURE);
+    SdkHttpRequest second = interceptor.modifyHttpRequest(context, attrs);
+
+    assertEquals(recovering, endpoint(first));
+    assertEquals(active, endpoint(second));
+  }
+
   // ========== Mock implementations ==========
 
+  private SdkHttpRequest baseRequest() {
+    return SdkHttpRequest.builder()
+        .protocol("http")
+        .host("placeholder")
+        .port(8000)
+        .method(software.amazon.awssdk.http.SdkHttpMethod.POST)
+        .encodedPath("/")
+        .build();
+  }
+
+  private URI endpoint(SdkHttpRequest request) throws Exception {
+    return new URI(request.protocol(), null, request.host(), request.port(), null, null, null);
+  }
+
+  private long seedWhereFirstNodeIs(List candidates, URI expected) throws Exception {
+    for (long seed = -1000; seed < 1000; seed++) {
+      if (expected.equals(firstNodeForSeed(candidates, seed))) {
+        return seed;
+      }
+    }
+    fail("Could not find seed for " + expected);
+    return 0;
+  }
+
+  private URI firstNodeForSeed(List candidates, long seed) {
+    LazyQueryPlan plan = new LazyQueryPlan(new MockAlternatorLiveNodes(candidates), seed);
+    return plan.hasNext() ? plan.next() : null;
+  }
+
+  private Throwable dynamoDbException(int statusCode, String errorCode) {
+    return DynamoDbException.builder()
+        .message(errorCode)
+        .statusCode(statusCode)
+        .awsErrorDetails(
+            AwsErrorDetails.builder()
+                .errorCode(errorCode)
+                .errorMessage(errorCode)
+                .sdkHttpResponse(SdkHttpFullResponse.builder().statusCode(statusCode).build())
+                .build())
+        .build();
+  }
+
   /**
    * Mock AlternatorLiveNodes that provides a fixed list of nodes without network calls.
    *
@@ -292,40 +647,73 @@ public void testRetriesExceedNodeCount() throws Exception {
    *   
  • Does not start any background threads
    *   
  • Provides a fixed list of test nodes
    *   
  • Supports LazyQueryPlan creation (via base class)
-   *   
  • Implements round-robin across all test nodes
    * 
    */
   private static class MockAlternatorLiveNodes extends AlternatorLiveNodes {
     private final List nodes;
-    private final java.util.concurrent.atomic.AtomicInteger counter =
-        new java.util.concurrent.atomic.AtomicInteger(0);
+    private final List reports = new ArrayList<>();
 
     MockAlternatorLiveNodes(List nodes) {
+      this(nodes, NodeHealthConfig.getDefault());
+    }
+
+    MockAlternatorLiveNodes(List nodes, NodeHealthConfig nodeHealthConfig) {
       super(
           AlternatorConfig.builder()
-              .withSeedHosts(Collections.singletonList(nodes.get(0).getHost()))
+              .withSeedHosts(nodes.stream().map(URI::getHost).collect(Collectors.toList()))
               .withScheme(nodes.get(0).getScheme())
               .withPort(nodes.get(0).getPort())
+              .withNodeHealthConfig(nodeHealthConfig)
               .build());
       this.nodes = new ArrayList<>(nodes);
     }
 
     @Override
-    protected List getLiveNodesInternal() {
+    protected List getDiscoveredNodesInternal() {
       return nodes;
     }
 
     @Override
-    public URI nextAsURI() {
-      return nodes.get(Math.abs(counter.getAndIncrement() % nodes.size()));
+    public void start() {}
+
+    @Override
+    public void reportNodeResult(URI node, NodeHealthObservation observation) {
+      super.reportNodeResult(node, observation);
+      reports.add(new NodeReport(node, observation));
+    }
+  }
+
+  private static class TestableBasicQueryPlanInterceptor extends BasicQueryPlanInterceptor {
+    TestableBasicQueryPlanInterceptor(AlternatorLiveNodes liveNodes) {
+      super(liveNodes);
+    }
+
+    void setQueryPlan(ExecutionAttributes attrs, LazyQueryPlan plan) {
+      attrs.putAttribute(QUERY_PLAN, plan);
+    }
+  }
+
+  private static class NodeReport {
+    private final URI node;
+    private final NodeHealthObservation observation;
+
+    NodeReport(URI node, NodeHealthObservation observation) {
+      this.node = node;
+      this.observation = observation;
     }
 
     @Override
-    public void start() {}
+    public boolean equals(Object other) {
+      if (!(other instanceof NodeReport)) {
+        return false;
+      }
+      NodeReport that = (NodeReport) other;
+      return Objects.equals(node, that.node) && observation == that.observation;
+    }
 
     @Override
-    public List getLiveNodes() {
-      return Collections.unmodifiableList(new ArrayList<>(nodes));
+    public int hashCode() {
+      return Objects.hash(node, observation);
     }
   }
 
@@ -367,4 +755,77 @@ public Optional asyncRequest
       return Optional.empty();
     }
   }
+
+  private static class MockBeforeTransmissionContext extends MockModifyHttpRequestContext
+      implements software.amazon.awssdk.core.interceptor.Context.BeforeTransmission {
+
+    MockBeforeTransmissionContext(SdkHttpRequest httpRequest) {
+      super(httpRequest);
+    }
+  }
+
+  private static class MockAfterTransmissionContext extends MockModifyHttpRequestContext
+      implements software.amazon.awssdk.core.interceptor.Context.AfterTransmission {
+
+    private final SdkHttpResponse response;
+
+    MockAfterTransmissionContext(SdkHttpRequest httpRequest, int statusCode) {
+      super(httpRequest);
+      this.response = SdkHttpFullResponse.builder().statusCode(statusCode).build();
+    }
+
+    @Override
+    public SdkHttpResponse httpResponse() {
+      return response;
+    }
+
+    @Override
+    public Optional> responsePublisher() {
+      return Optional.empty();
+    }
+
+    @Override
+    public Optional responseBody() {
+      return Optional.empty();
+    }
+  }
+
+  private static class MockFailedExecutionContext
+      implements software.amazon.awssdk.core.interceptor.Context.FailedExecution {
+
+    private final SdkHttpRequest httpRequest;
+    private final SdkHttpResponse httpResponse;
+    private final Throwable exception;
+
+    MockFailedExecutionContext(SdkHttpRequest httpRequest, int statusCode, Throwable exception) {
+      this.httpRequest = httpRequest;
+      this.httpResponse = SdkHttpFullResponse.builder().statusCode(statusCode).build();
+      this.exception = exception;
+    }
+
+    @Override
+    public Throwable exception() {
+      return exception;
+    }
+
+    @Override
+    public SdkRequest request() {
+      return null;
+    }
+
+    @Override
+    public Optional httpRequest() {
+      return Optional.of(httpRequest);
+    }
+
+    @Override
+    public Optional httpResponse() {
+      return Optional.of(httpResponse);
+    }
+
+    @Override
+    public Optional response() {
+      return Optional.empty();
+    }
+  }
 }
diff --git a/src/test/java/com/scylladb/alternator/internal/AlternatorLiveNodesClusterDiscoveryTest.java b/src/test/java/com/scylladb/alternator/internal/AlternatorLiveNodesClusterDiscoveryTest.java
index aacc0ba..04d09df 100644
--- a/src/test/java/com/scylladb/alternator/internal/AlternatorLiveNodesClusterDiscoveryTest.java
+++ b/src/test/java/com/scylladb/alternator/internal/AlternatorLiveNodesClusterDiscoveryTest.java
@@ -9,6 +9,7 @@
 import com.scylladb.alternator.routing.RoutingScope;
 import java.io.ByteArrayInputStream;
 import java.io.IOException;
+import java.io.InputStream;
 import java.net.URI;
 import java.nio.charset.StandardCharsets;
 import java.util.ArrayList;
@@ -99,7 +100,7 @@ public void testClusterScopeMergesConfiguredSeedNodes() throws Exception {
 
     AlternatorLiveNodes liveNodes = new AlternatorLiveNodes(config, httpClient);
 
-    liveNodes.updateLiveNodes();
+    liveNodes.refreshDiscoveredNodes();
 
     assertEquals(
         new LinkedHashSet<>(
@@ -108,7 +109,7 @@ public void testClusterScopeMergesConfiguredSeedNodes() throws Exception {
                 "dc1-node2.example.com",
                 "dc2-node1.example.com",
                 "dc2-node2.example.com")),
-        hostSet(liveNodes.getLiveNodes()));
+        hostSet(liveNodes.getDiscoveredNodes()));
     assertEquals(
         new HashSet<>(Arrays.asList("dc1-node1.example.com", "dc2-node1.example.com")),
         capturedHostSet(httpClient.capturedRequests));
@@ -135,11 +136,11 @@ public void testClusterScopeKeepsSuccessfulDiscoveryWhenAnotherSeedFails() throw
 
     AlternatorLiveNodes liveNodes = new AlternatorLiveNodes(config, httpClient);
 
-    liveNodes.updateLiveNodes();
+    liveNodes.refreshDiscoveredNodes();
 
     assertEquals(
         new LinkedHashSet<>(Arrays.asList("dc1-node1.example.com")),
-        hostSet(liveNodes.getLiveNodes()));
+        hostSet(liveNodes.getDiscoveredNodes()));
     assertEquals(
         new HashSet<>(Arrays.asList("dc1-node1.example.com", "dc2-node1.example.com")),
         capturedHostSet(httpClient.capturedRequests));
@@ -234,11 +235,11 @@ public void testRackScopeTriesNextSeedWhenFirstSeedIsInWrongDatacenter() throws
 
     AlternatorLiveNodes liveNodes = new AlternatorLiveNodes(config, httpClient);
 
-    liveNodes.updateLiveNodes();
+    liveNodes.refreshDiscoveredNodes();
 
     assertEquals(
         new LinkedHashSet<>(Arrays.asList("dc1-rack1-node.example.com")),
-        hostSet(liveNodes.getLiveNodes()));
+        hostSet(liveNodes.getDiscoveredNodes()));
     assertEquals(
         Arrays.asList("dc2-node1.example.com", "dc1-node1.example.com"),
         capturedHosts(httpClient.capturedRequests));
@@ -248,6 +249,44 @@ public void testRackScopeTriesNextSeedWhenFirstSeedIsInWrongDatacenter() throws
     }
   }
 
+  @Test
+  public void testScopedDiscoveryUsesSeedsWithoutPublishingThem() throws Exception {
+    Map responses = new HashMap<>();
+    responses.put("dc2-node1.example.com", "[]");
+    responses.put("dc1-node1.example.com", "[\"dc1-rack1-node.example.com\"]");
+    DiscoveryHttpClient httpClient = new DiscoveryHttpClient(responses);
+
+    AlternatorConfig config =
+        AlternatorConfig.builder()
+            .withSeedHosts(Arrays.asList("dc2-node1.example.com", "dc1-node1.example.com"))
+            .withScheme("http")
+            .withPort(8000)
+            .withRoutingScope(
+                RackScope.of("dc1", "rack1", DatacenterScope.of("dc1", ClusterScope.create())))
+            .build();
+
+    AlternatorLiveNodes liveNodes = new AlternatorLiveNodes(config, httpClient);
+    liveNodes.refreshDiscoveredNodes();
+
+    assertEquals(
+        new LinkedHashSet<>(Arrays.asList("dc1-rack1-node.example.com")),
+        hostSet(liveNodes.getDiscoveredNodes()));
+
+    responses.put("dc1-rack1-node.example.com", "[]");
+    responses.put("dc1-node1.example.com", "[\"dc1-rack1-node2.example.com\"]");
+    httpClient.capturedRequests.clear();
+
+    liveNodes.refreshDiscoveredNodes();
+
+    assertEquals(
+        Arrays.asList(
+            "dc1-rack1-node.example.com", "dc2-node1.example.com", "dc1-node1.example.com"),
+        capturedHosts(httpClient.capturedRequests));
+    assertEquals(
+        new LinkedHashSet<>(Arrays.asList("dc1-rack1-node2.example.com")),
+        hostSet(liveNodes.getDiscoveredNodes()));
+  }
+
   @Test
   public void testScopedLocalNodesQueryValuesAreEncodedOnce() throws Exception {
     Map responses = new HashMap<>();
@@ -323,6 +362,106 @@ public void testRackScopeValidationTriesNextSeedWhenFirstSeedIsInWrongDatacenter
     }
   }
 
+  @Test
+  public void testNon200DiscoveryResponseBodyIsConsumedOnce() throws Exception {
+    CloseTrackingInputStream body = new CloseTrackingInputStream("Service Unavailable");
+    SdkHttpClient httpClient =
+        new SdkHttpClient() {
+          @Override
+          public ExecutableHttpRequest prepareRequest(HttpExecuteRequest request) {
+            return new ExecutableHttpRequest() {
+              @Override
+              public HttpExecuteResponse call() {
+                return HttpExecuteResponse.builder()
+                    .response(SdkHttpFullResponse.builder().statusCode(503).build())
+                    .responseBody(AbortableInputStream.create(body, body::abort))
+                    .build();
+              }
+
+              @Override
+              public void abort() {}
+            };
+          }
+
+          @Override
+          public void close() {}
+
+          @Override
+          public String clientName() {
+            return "non-200";
+          }
+        };
+
+    AlternatorConfig config =
+        AlternatorConfig.builder()
+            .withSeedHost("seed.example.com")
+            .withScheme("http")
+            .withPort(8000)
+            .withRoutingScope(ClusterScope.create())
+            .build();
+
+    AlternatorLiveNodes liveNodes = new AlternatorLiveNodes(config, httpClient);
+    liveNodes.refreshDiscoveredNodes();
+
+    assertEquals(1, body.closeCount());
+    assertEquals(0, body.readAfterCloseCount());
+    assertEquals(0, body.abortCount());
+  }
+
+  private static final class CloseTrackingInputStream extends InputStream {
+    private final ByteArrayInputStream delegate;
+    private int closeCount;
+    private int readAfterCloseCount;
+    private int abortCount;
+    private boolean closed;
+
+    private CloseTrackingInputStream(String body) {
+      this.delegate = new ByteArrayInputStream(body.getBytes(StandardCharsets.UTF_8));
+    }
+
+    @Override
+    public int read() throws IOException {
+      ensureOpen();
+      return delegate.read();
+    }
+
+    @Override
+    public int read(byte[] bytes, int offset, int length) throws IOException {
+      ensureOpen();
+      return delegate.read(bytes, offset, length);
+    }
+
+    @Override
+    public void close() throws IOException {
+      closeCount++;
+      closed = true;
+      delegate.close();
+    }
+
+    private void abort() {
+      abortCount++;
+    }
+
+    private int closeCount() {
+      return closeCount;
+    }
+
+    private int readAfterCloseCount() {
+      return readAfterCloseCount;
+    }
+
+    private int abortCount() {
+      return abortCount;
+    }
+
+    private void ensureOpen() throws IOException {
+      if (closed) {
+        readAfterCloseCount++;
+        throw new IOException("read after close");
+      }
+    }
+  }
+
   private static Set hostSet(List uris) {
     Set hosts = new LinkedHashSet<>();
     for (URI uri : uris) {
diff --git a/src/test/java/com/scylladb/alternator/internal/AlternatorLiveNodesKeepAliveTest.java b/src/test/java/com/scylladb/alternator/internal/AlternatorLiveNodesKeepAliveTest.java
index 6270ca5..3fe2014 100644
--- a/src/test/java/com/scylladb/alternator/internal/AlternatorLiveNodesKeepAliveTest.java
+++ b/src/test/java/com/scylladb/alternator/internal/AlternatorLiveNodesKeepAliveTest.java
@@ -25,7 +25,7 @@ public class AlternatorLiveNodesKeepAliveTest {
 
   /**
    * Mock SdkHttpClient that captures outgoing requests and returns a valid /localnodes JSON
-   * response so that updateLiveNodes() succeeds.
+   * response so that refreshDiscoveredNodes() succeeds.
    */
   private static class CapturingHttpClient implements SdkHttpClient {
     final List capturedRequests = new CopyOnWriteArrayList<>();
@@ -65,7 +65,7 @@ public void testPollingRequestIncludesConnectionKeepAliveHeader() throws Excepti
     AlternatorLiveNodes liveNodes = new AlternatorLiveNodes(config, capturingClient);
 
     // Trigger a polling cycle
-    liveNodes.updateLiveNodes();
+    liveNodes.refreshDiscoveredNodes();
 
     assertFalse(
         "Should have captured at least one request", capturingClient.capturedRequests.isEmpty());
@@ -90,7 +90,7 @@ public void testMultiplePollingRequestsAllIncludeKeepAlive() throws Exception {
 
     // Multiple polling cycles
     for (int i = 0; i < 5; i++) {
-      liveNodes.updateLiveNodes();
+      liveNodes.refreshDiscoveredNodes();
     }
 
     assertTrue(
@@ -114,7 +114,7 @@ public void testPollingRequestIncludesConnectionKeepAliveHeaderForHttps() throws
     AlternatorLiveNodes liveNodes = new AlternatorLiveNodes(config, capturingClient);
 
     // Trigger a polling cycle
-    liveNodes.updateLiveNodes();
+    liveNodes.refreshDiscoveredNodes();
 
     assertFalse(
         "Should have captured at least one request", capturingClient.capturedRequests.isEmpty());
@@ -140,7 +140,7 @@ public void testMultipleHttpsPollingRequestsAllIncludeKeepAlive() throws Excepti
 
     // Multiple polling cycles
     for (int i = 0; i < 5; i++) {
-      liveNodes.updateLiveNodes();
+      liveNodes.refreshDiscoveredNodes();
     }
 
     assertTrue(
diff --git a/src/test/java/com/scylladb/alternator/internal/AlternatorLiveNodesNodeHealthTest.java b/src/test/java/com/scylladb/alternator/internal/AlternatorLiveNodesNodeHealthTest.java
new file mode 100644
index 0000000..6de6280
--- /dev/null
+++ b/src/test/java/com/scylladb/alternator/internal/AlternatorLiveNodesNodeHealthTest.java
@@ -0,0 +1,1053 @@
+package com.scylladb.alternator.internal;
+
+import static org.junit.Assert.*;
+
+import com.scylladb.alternator.AlternatorConfig;
+import com.scylladb.alternator.NodeHealthConfig;
+import com.scylladb.alternator.NodeHealthObservation;
+import com.scylladb.alternator.NodeHealthState;
+import com.scylladb.alternator.routing.DatacenterScope;
+import com.scylladb.alternator.routing.RoutingScope;
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.lang.reflect.Field;
+import java.net.URI;
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.List;
+import java.util.concurrent.CopyOnWriteArrayList;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+import org.junit.Test;
+import software.amazon.awssdk.http.AbortableInputStream;
+import software.amazon.awssdk.http.ExecutableHttpRequest;
+import software.amazon.awssdk.http.HttpExecuteRequest;
+import software.amazon.awssdk.http.HttpExecuteResponse;
+import software.amazon.awssdk.http.SdkHttpClient;
+import software.amazon.awssdk.http.SdkHttpFullResponse;
+import software.amazon.awssdk.http.SdkHttpRequest;
+
+public class AlternatorLiveNodesNodeHealthTest {
+  @Test
+  public void runDownNodeProbesMovesRecoveredNodeToQuarantine() {
+    AlternatorLiveNodes liveNodes =
+        liveNodes(
+            NodeHealthConfig.builder().withDownNodeRecoverySuccessThreshold(1).build(),
+            new LocalNodesHttpClient());
+    URI recovering = node("recovering.local");
+
+    markNodeDown(liveNodes, recovering);
+    List recovered = liveNodes.runDownNodeProbes();
+
+    assertEquals(Arrays.asList(recovering), recovered);
+    assertEquals(Arrays.asList(recovering), liveNodes.getQuarantinedNodes());
+    assertTrue(liveNodes.getDownNodes().isEmpty());
+  }
+
+  @Test
+  public void runDownNodeProbesRequiresConfiguredRecoverySuccessesBeforeQuarantine() {
+    AlternatorLiveNodes liveNodes =
+        liveNodes(
+            NodeHealthConfig.builder().withDownNodeRecoverySuccessThreshold(2).build(),
+            new LocalNodesHttpClient());
+    URI recovering = node("recovering.local");
+
+    markNodeDown(liveNodes, recovering);
+
+    assertTrue(liveNodes.runDownNodeProbes().isEmpty());
+    assertEquals(Arrays.asList(recovering), liveNodes.getDownNodes());
+    assertEquals(1, liveNodes.getNodeHealthStatus(recovering).getConsecutiveSuccesses());
+
+    assertEquals(Arrays.asList(recovering), liveNodes.runDownNodeProbes());
+    assertEquals(Arrays.asList(recovering), liveNodes.getQuarantinedNodes());
+    assertEquals(0, liveNodes.getNodeHealthStatus(recovering).getConsecutiveSuccesses());
+  }
+
+  @Test
+  public void queryPlanFilterSamplesQuarantineByConfiguredInterval() {
+    AlternatorLiveNodes liveNodes =
+        liveNodes(
+            NodeHealthConfig.builder()
+                .withDownNodeRecoverySuccessThreshold(1)
+                .withQuarantineSuccessThreshold(3)
+                .withQuarantineTrafficInterval(2)
+                .build(),
+            new LocalNodesHttpClient());
+    URI active = node("active.local");
+    URI recovering = node("recovering.local");
+
+    markNodeDown(liveNodes, recovering);
+    liveNodes.runDownNodeProbes();
+
+    assertEquals(Arrays.asList(active, recovering), liveNodes.getQueryPlanNodes());
+
+    AlternatorLiveNodes.QueryPlanNodeFilter firstFilter = liveNodes.newQueryPlanNodeFilter();
+    assertTrue(firstFilter.shouldRouteTo(active));
+    assertFalse(firstFilter.shouldRouteTo(recovering));
+
+    AlternatorLiveNodes.QueryPlanNodeFilter secondFilter = liveNodes.newQueryPlanNodeFilter();
+    assertTrue(secondFilter.shouldRouteTo(active));
+    assertTrue(secondFilter.shouldRouteTo(recovering));
+
+    AlternatorLiveNodes.QueryPlanNodeFilter thirdFilter = liveNodes.newQueryPlanNodeFilter();
+    assertTrue(thirdFilter.shouldRouteTo(active));
+    assertFalse(thirdFilter.shouldRouteTo(recovering));
+  }
+
+  @Test
+  public void queryPlanFilterAllowsAllDownNodesWhenNoActiveNodes() {
+    AlternatorLiveNodes liveNodes =
+        liveNodes(NodeHealthConfig.getDefault(), new LocalNodesHttpClient());
+    URI first = node("active.local");
+    URI second = node("recovering.local");
+
+    markNodeDown(liveNodes, first);
+    markNodeDown(liveNodes, second);
+
+    AlternatorLiveNodes.QueryPlanNodeFilter filter = liveNodes.newQueryPlanNodeFilter();
+
+    assertTrue(liveNodes.getActiveNodes().isEmpty());
+    assertTrue(filter.shouldRouteTo(first));
+    assertTrue(filter.shouldRouteTo(second));
+    assertEquals(NodeHealthState.DOWN, liveNodes.getNodeHealthStatus(first).getState());
+    assertEquals(NodeHealthState.DOWN, liveNodes.getNodeHealthStatus(second).getState());
+  }
+
+  @Test
+  public void queryPlanFilterFailsOpenAfterLastActiveNodeGoesDown() {
+    AlternatorLiveNodes liveNodes =
+        liveNodes(NodeHealthConfig.getDefault(), new LocalNodesHttpClient());
+    URI active = node("active.local");
+    URI down = node("recovering.local");
+
+    markNodeDown(liveNodes, down);
+    AlternatorLiveNodes.QueryPlanNodeFilter filter = liveNodes.newQueryPlanNodeFilter();
+
+    assertFalse(filter.shouldRouteTo(down));
+
+    markNodeDown(liveNodes, active);
+
+    assertTrue(liveNodes.getActiveNodes().isEmpty());
+    assertTrue(filter.shouldRouteTo(down));
+    assertTrue(filter.shouldRouteTo(active));
+  }
+
+  @Test
+  public void queryPlanFilterAllowsAllQuarantinedNodesWhenNoActiveNodes() {
+    AlternatorLiveNodes liveNodes =
+        liveNodes(
+            NodeHealthConfig.builder().withDownNodeRecoverySuccessThreshold(1).build(),
+            new LocalNodesHttpClient());
+    URI first = node("active.local");
+    URI second = node("recovering.local");
+
+    markNodeDown(liveNodes, first);
+    markNodeDown(liveNodes, second);
+    liveNodes.runDownNodeProbes();
+
+    AlternatorLiveNodes.QueryPlanNodeFilter filter = liveNodes.newQueryPlanNodeFilter();
+
+    assertTrue(liveNodes.getActiveNodes().isEmpty());
+    assertTrue(filter.shouldRouteTo(first));
+    assertTrue(filter.shouldRouteTo(second));
+    assertEquals(NodeHealthState.QUARANTINED, liveNodes.getNodeHealthStatus(first).getState());
+    assertEquals(NodeHealthState.QUARANTINED, liveNodes.getNodeHealthStatus(second).getState());
+  }
+
+  @Test
+  public void featureCheckUsesRandomQueryPlanNode() throws Exception {
+    LocalNodesHttpClient httpClient = new LocalNodesHttpClient("[\"active.local\"]");
+    AlternatorLiveNodes liveNodes = liveNodes(NodeHealthConfig.getDefault(), httpClient);
+    URI recovering = node("recovering.local");
+
+    markNodeDown(liveNodes, recovering);
+
+    assertFalse(liveNodes.checkIfRackDatacenterFeatureIsSupported());
+
+    assertEquals(2, httpClient.requests.size());
+    assertEquals("active.local", httpClient.requests.get(0).host());
+    assertEquals("active.local", httpClient.requests.get(1).host());
+    assertEquals("/localnodes", httpClient.requests.get(0).encodedPath());
+    assertEquals("/localnodes", httpClient.requests.get(1).encodedPath());
+    assertTrue(httpClient.requests.get(0).rawQueryParameters().containsKey("rack"));
+    assertTrue(httpClient.requests.get(1).rawQueryParameters().isEmpty());
+  }
+
+  @Test
+  public void featureCheckTriesNextQueryPlanNodeWhenOneNodeFails() throws Exception {
+    LocalNodesHttpClient httpClient = new LocalNodesHttpClient("[\"active.local\"]", 1);
+    AlternatorConfig config =
+        AlternatorConfig.builder()
+            .withSeedHosts(Arrays.asList("node-a.local", "node-b.local"))
+            .withScheme("http")
+            .withPort(8080)
+            .withActiveRefreshIntervalMs(60_000)
+            .withIdleRefreshIntervalMs(60_000)
+            .build();
+    AlternatorLiveNodes liveNodes =
+        new AlternatorLiveNodes(config, httpClient) {
+          @Override
+          public List getQueryPlanNodes() {
+            return Arrays.asList(node("node-a.local"), node("node-b.local"));
+          }
+        };
+
+    assertFalse(liveNodes.checkIfRackDatacenterFeatureIsSupported());
+
+    assertEquals(3, httpClient.requests.size());
+    assertTrue(httpClient.requests.get(0).rawQueryParameters().containsKey("rack"));
+    assertNotEquals(httpClient.requests.get(0).host(), httpClient.requests.get(1).host());
+    assertEquals(httpClient.requests.get(1).host(), httpClient.requests.get(2).host());
+    assertTrue(httpClient.requests.get(1).rawQueryParameters().containsKey("rack"));
+    assertTrue(httpClient.requests.get(2).rawQueryParameters().isEmpty());
+  }
+
+  @Test
+  public void featureCheckSuccessClearsPreviousProbeFailures() throws Exception {
+    LocalNodesHttpClient httpClient = new LocalNodesHttpClient("[\"active.local\"]", 1);
+    AlternatorLiveNodes liveNodes =
+        liveNodes(
+            NodeHealthConfig.builder().withConsecutiveFailureThreshold(2).build(),
+            httpClient,
+            null,
+            Arrays.asList("active.local"));
+    URI active = node("active.local");
+
+    try {
+      liveNodes.checkIfRackDatacenterFeatureIsSupported();
+      fail("expected feature check failure");
+    } catch (AlternatorLiveNodes.FailedToCheck expected) {
+      // expected
+    }
+    assertEquals(1, liveNodes.getNodeHealthStatus(active).getConsecutiveFailures());
+
+    assertFalse(liveNodes.checkIfRackDatacenterFeatureIsSupported());
+
+    assertEquals(0, liveNodes.getNodeHealthStatus(active).getConsecutiveFailures());
+    assertTrue(liveNodes.getDownNodes().isEmpty());
+  }
+
+  @Test
+  public void featureCheckRejectsNullQueryPlanNodeWithoutNpe() throws Exception {
+    LocalNodesHttpClient httpClient = new LocalNodesHttpClient();
+    AlternatorConfig config =
+        AlternatorConfig.builder()
+            .withSeedHosts(Arrays.asList("active.local"))
+            .withScheme("http")
+            .withPort(8080)
+            .withActiveRefreshIntervalMs(60_000)
+            .withIdleRefreshIntervalMs(60_000)
+            .build();
+    AlternatorLiveNodes liveNodes =
+        new AlternatorLiveNodes(config, httpClient) {
+          @Override
+          public List getQueryPlanNodes() {
+            return Arrays.asList((URI) null);
+          }
+        };
+
+    try {
+      liveNodes.checkIfRackDatacenterFeatureIsSupported();
+      fail("Expected FailedToCheck");
+    } catch (AlternatorLiveNodes.FailedToCheck e) {
+      assertEquals("query plan returned null node", e.getMessage());
+    }
+    assertTrue(httpClient.requests.isEmpty());
+  }
+
+  @Test
+  public void normalQueryPlanContainsQuarantinedNodeAndFilterAdmitsSample() {
+    AlternatorLiveNodes liveNodes =
+        liveNodes(
+            NodeHealthConfig.builder()
+                .withDownNodeRecoverySuccessThreshold(1)
+                .withQuarantineSuccessThreshold(3)
+                .withQuarantineTrafficInterval(1)
+                .build(),
+            new LocalNodesHttpClient());
+    URI active = node("active.local");
+    URI recovering = node("recovering.local");
+
+    markNodeDown(liveNodes, recovering);
+    liveNodes.runDownNodeProbes();
+
+    LazyQueryPlan plan = new LazyQueryPlan(liveNodes);
+    List selected = Arrays.asList(plan.next(), plan.next());
+    AlternatorLiveNodes.QueryPlanNodeFilter filter = liveNodes.newQueryPlanNodeFilter();
+
+    assertTrue(selected.contains(recovering));
+    assertTrue(selected.contains(active));
+    assertFalse(plan.hasNext());
+    assertTrue(filter.shouldRouteTo(recovering));
+  }
+
+  @Test
+  public void randomQueryPlanKeepsQuarantinedCandidateInKnownNodeSet() {
+    AlternatorLiveNodes liveNodes =
+        liveNodes(
+            NodeHealthConfig.builder()
+                .withDownNodeRecoverySuccessThreshold(1)
+                .withQuarantineSuccessThreshold(3)
+                .withQuarantineTrafficInterval(1)
+                .build(),
+            new LocalNodesHttpClient(),
+            null,
+            Arrays.asList(
+                "active01.local",
+                "active02.local",
+                "active03.local",
+                "active04.local",
+                "active05.local",
+                "active06.local",
+                "active07.local",
+                "active08.local",
+                "active09.local",
+                "active10.local",
+                "active11.local",
+                "active12.local",
+                "recovering.local"));
+    URI recovering = node("recovering.local");
+
+    markNodeDown(liveNodes, recovering);
+    liveNodes.runDownNodeProbes();
+
+    assertTrue(collectNodes(new LazyQueryPlan(liveNodes)).contains(recovering));
+  }
+
+  @Test
+  public void quarantineHashSamplingUsesSeededPlanOrderUntilVerified() {
+    AlternatorLiveNodes liveNodes =
+        liveNodes(
+            NodeHealthConfig.builder()
+                .withDownNodeRecoverySuccessThreshold(1)
+                .withQuarantineSuccessThreshold(2)
+                .withQuarantineTrafficInterval(2)
+                .build(),
+            new LocalNodesHttpClient());
+    URI active = node("active.local");
+    URI recovering = node("recovering.local");
+    long activeHash = hashWhereFirstNodeIs(Arrays.asList(active, recovering), active);
+    long recoveringHash = hashWhereFirstNodeIs(Arrays.asList(active, recovering), recovering);
+
+    markNodeDown(liveNodes, recovering);
+    liveNodes.runDownNodeProbes();
+
+    assertEquals(
+        Arrays.asList(recovering, active), liveNodes.getQueryPlanNodesForHash(recoveringHash));
+    AlternatorLiveNodes.QueryPlanNodeFilter firstFilter = liveNodes.newQueryPlanNodeFilter();
+    assertFalse(firstFilter.shouldRouteTo(recovering));
+    assertTrue(firstFilter.shouldRouteTo(active));
+
+    AlternatorLiveNodes.QueryPlanNodeFilter secondFilter = liveNodes.newQueryPlanNodeFilter();
+    assertTrue(secondFilter.shouldRouteTo(recovering));
+    liveNodes.reportNodeResult(recovering, NodeHealthObservation.TRAFFIC_SUCCESS);
+
+    assertEquals(Arrays.asList(active, recovering), liveNodes.getQueryPlanNodesForHash(activeHash));
+    liveNodes.reportNodeResult(recovering, NodeHealthObservation.TRAFFIC_SUCCESS);
+
+    assertTrue(liveNodes.getQuarantinedNodes().isEmpty());
+    assertEquals(NodeHealthState.ACTIVE, liveNodes.getNodeHealthStatus(recovering).getState());
+    assertEquals(Arrays.asList(active, recovering), liveNodes.getActiveNodes());
+    assertEquals(
+        Arrays.asList(recovering, active), liveNodes.getQueryPlanNodesForHash(recoveringHash));
+  }
+
+  @Test
+  public void queryPlanFilterCountsQuarantineSamplingWhenQuarantinedNodeReached() throws Exception {
+    AlternatorLiveNodes liveNodes =
+        liveNodes(
+            NodeHealthConfig.builder()
+                .withDownNodeRecoverySuccessThreshold(1)
+                .withQuarantineSuccessThreshold(3)
+                .withQuarantineTrafficInterval(2)
+                .build(),
+            new LocalNodesHttpClient());
+    URI active = node("active.local");
+    URI recovering = node("recovering.local");
+    long activeHash = hashWhereFirstNodeIs(Arrays.asList(active, recovering), active);
+
+    markNodeDown(liveNodes, recovering);
+    liveNodes.runDownNodeProbes();
+
+    assertEquals(Arrays.asList(active, recovering), liveNodes.getQueryPlanNodesForHash(activeHash));
+    AlternatorLiveNodes.QueryPlanNodeFilter firstFilter = liveNodes.newQueryPlanNodeFilter();
+    assertTrue(firstFilter.shouldRouteTo(active));
+    assertEquals(0, quarantineSamplingCounter(liveNodes).get());
+    assertFalse(firstFilter.shouldRouteTo(recovering));
+    assertEquals(1, quarantineSamplingCounter(liveNodes).get());
+
+    assertEquals(Arrays.asList(active, recovering), liveNodes.getQueryPlanNodesForHash(activeHash));
+    AlternatorLiveNodes.QueryPlanNodeFilter secondFilter = liveNodes.newQueryPlanNodeFilter();
+    assertTrue(secondFilter.shouldRouteTo(active));
+    assertEquals(1, quarantineSamplingCounter(liveNodes).get());
+    assertTrue(secondFilter.shouldRouteTo(recovering));
+    assertEquals(2, quarantineSamplingCounter(liveNodes).get());
+  }
+
+  @Test
+  @SuppressWarnings("deprecation")
+  public void deprecatedNextAsUriSamplesQuarantinedNodeByInterval() throws Exception {
+    AlternatorLiveNodes liveNodes =
+        liveNodes(
+            NodeHealthConfig.builder()
+                .withDownNodeRecoverySuccessThreshold(1)
+                .withQuarantineSuccessThreshold(3)
+                .withQuarantineTrafficInterval(2)
+                .build(),
+            new LocalNodesHttpClient());
+    URI active = node("active.local");
+    URI recovering = node("recovering.local");
+
+    markNodeDown(liveNodes, recovering);
+    liveNodes.runDownNodeProbes();
+
+    assertEquals(active, liveNodes.nextAsURI());
+    assertEquals(1, quarantineSamplingCounter(liveNodes).get());
+    assertEquals(recovering, liveNodes.nextAsURI());
+    assertEquals(2, quarantineSamplingCounter(liveNodes).get());
+  }
+
+  @Test
+  public void preferredQueryPlanNodeForHashKeepsQuarantinedPreferredNode() {
+    AlternatorLiveNodes liveNodes =
+        liveNodes(
+            NodeHealthConfig.builder()
+                .withDownNodeRecoverySuccessThreshold(1)
+                .withQuarantineSuccessThreshold(3)
+                .withQuarantineTrafficInterval(1)
+                .build(),
+            new LocalNodesHttpClient());
+    URI active = node("active.local");
+    URI recovering = node("recovering.local");
+    long recoveringHash = hashWhereFirstNodeIs(Arrays.asList(active, recovering), recovering);
+
+    markNodeDown(liveNodes, recovering);
+    liveNodes.runDownNodeProbes();
+
+    URI preferred = liveNodes.getPreferredQueryPlanNodeForHash(recoveringHash);
+    LazyQueryPlan plan = new LazyQueryPlan(liveNodes, recoveringHash);
+
+    assertEquals(recovering, preferred);
+    assertEquals(preferred, plan.next());
+  }
+
+  @Test
+  public void quarantineHashAssignmentKeepsDownNodeButFilterSkipsIt() {
+    AlternatorLiveNodes liveNodes =
+        liveNodes(
+            NodeHealthConfig.builder()
+                .withDownNodeRecoverySuccessThreshold(1)
+                .withQuarantineSuccessThreshold(3)
+                .withQuarantineTrafficInterval(1)
+                .build(),
+            new LocalNodesHttpClient());
+    URI active = node("active.local");
+    URI recovering = node("recovering.local");
+    long recoveringHash = hashWhereFirstNodeIs(Arrays.asList(active, recovering), recovering);
+
+    markNodeDown(liveNodes, recovering);
+    liveNodes.runDownNodeProbes();
+    assertEquals(
+        Arrays.asList(recovering, active), liveNodes.getQueryPlanNodesForHash(recoveringHash));
+
+    markNodeDown(liveNodes, recovering);
+
+    assertEquals(Arrays.asList(recovering), liveNodes.getDownNodes());
+    assertEquals(
+        Arrays.asList(recovering, active), liveNodes.getQueryPlanNodesForHash(recoveringHash));
+    AlternatorLiveNodes.QueryPlanNodeFilter filter = liveNodes.newQueryPlanNodeFilter();
+    assertFalse(filter.shouldRouteTo(recovering));
+    assertTrue(filter.shouldRouteTo(active));
+  }
+
+  @Test
+  public void quarantinePromotesAfterSuccessfulTraffic() {
+    AlternatorLiveNodes liveNodes =
+        liveNodes(
+            NodeHealthConfig.builder()
+                .withDownNodeRecoverySuccessThreshold(1)
+                .withQuarantineSuccessThreshold(2)
+                .build(),
+            new LocalNodesHttpClient());
+    URI recovering = node("recovering.local");
+
+    markNodeDown(liveNodes, recovering);
+    liveNodes.runDownNodeProbes();
+    assertEquals(Arrays.asList(recovering), liveNodes.getQuarantinedNodes());
+
+    liveNodes.reportNodeResult(recovering, NodeHealthObservation.TRAFFIC_SUCCESS);
+
+    assertEquals(Arrays.asList(recovering), liveNodes.getQuarantinedNodes());
+
+    liveNodes.reportNodeResult(recovering, NodeHealthObservation.TRAFFIC_SUCCESS);
+
+    assertTrue(liveNodes.getQuarantinedNodes().isEmpty());
+    assertTrue(liveNodes.getDownNodes().isEmpty());
+    assertEquals(Arrays.asList(node("active.local"), recovering), liveNodes.getActiveNodes());
+  }
+
+  @Test
+  public void backgroundMovesDownNodeToQuarantine() throws Exception {
+    LocalNodesHttpClient httpClient = new LocalNodesHttpClient();
+    AlternatorLiveNodes liveNodes =
+        liveNodes(
+            NodeHealthConfig.builder()
+                .withDownNodeProbePeriodMs(10)
+                .withQuarantineSuccessThreshold(2)
+                .build(),
+            httpClient);
+    URI recovering = node("recovering.local");
+
+    markNodeDown(liveNodes, recovering);
+    liveNodes.start();
+    try {
+      long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(5);
+      while (liveNodes.getQuarantinedNodes().isEmpty() && System.nanoTime() < deadline) {
+        Thread.sleep(10);
+      }
+    } finally {
+      liveNodes.shutdownAndWait();
+    }
+
+    assertFalse(httpClient.requests.isEmpty());
+    assertEquals(Arrays.asList(recovering), liveNodes.getQuarantinedNodes());
+  }
+
+  @Test
+  public void refreshPublishesDiscoveredNodesInSortedOrder() throws Exception {
+    LocalNodesHttpClient httpClient = new LocalNodesHttpClient("[\"z.local\",\"a.local\"]");
+    AlternatorLiveNodes liveNodes = liveNodes(NodeHealthConfig.getDefault(), httpClient);
+
+    liveNodes.refreshDiscoveredNodes();
+
+    assertEquals(Arrays.asList(node("a.local"), node("z.local")), liveNodes.getDiscoveredNodes());
+  }
+
+  @Test
+  public void localNodesNonOkMarksSeedDownWhenThresholdReached() throws Exception {
+    LocalNodesHttpClient httpClient = new LocalNodesHttpClient(400);
+    AlternatorLiveNodes liveNodes =
+        liveNodes(
+            NodeHealthConfig.builder().withConsecutiveFailureThreshold(1).build(),
+            httpClient,
+            null,
+            Arrays.asList("active.local"));
+
+    liveNodes.refreshDiscoveredNodes();
+
+    assertEquals(Arrays.asList(node("active.local")), liveNodes.getDownNodes());
+  }
+
+  @Test
+  public void duplicateSeedFailureCountsOncePerRefresh() throws Exception {
+    LocalNodesHttpClient httpClient = new LocalNodesHttpClient(400);
+    AlternatorLiveNodes liveNodes =
+        liveNodes(
+            NodeHealthConfig.builder().withConsecutiveFailureThreshold(2).build(),
+            httpClient,
+            null,
+            Arrays.asList("active.local", "active.local"));
+    URI active = node("active.local");
+
+    liveNodes.refreshDiscoveredNodes();
+
+    assertTrue(liveNodes.getDownNodes().isEmpty());
+    assertEquals(1, liveNodes.getNodeHealthStatus(active).getConsecutiveFailures());
+    assertEquals(1, httpClient.requests.size());
+
+    liveNodes.refreshDiscoveredNodes();
+
+    assertEquals(Arrays.asList(active), liveNodes.getDownNodes());
+    assertEquals(2, liveNodes.getNodeHealthStatus(active).getConsecutiveFailures());
+  }
+
+  @Test
+  public void localNodesRuntimeFailureMarksSeedDownWhenThresholdReached() throws Exception {
+    RuntimeFailureHttpClient httpClient = new RuntimeFailureHttpClient();
+    AlternatorLiveNodes liveNodes =
+        liveNodes(
+            NodeHealthConfig.builder().withConsecutiveFailureThreshold(1).build(),
+            httpClient,
+            null,
+            Arrays.asList("active.local"));
+
+    liveNodes.refreshDiscoveredNodes();
+
+    assertEquals(Arrays.asList(node("active.local")), liveNodes.getDownNodes());
+  }
+
+  @Test
+  public void localNodesInvalidBodyMarksSeedDownWhenThresholdReached() throws Exception {
+    LocalNodesHttpClient httpClient = new LocalNodesHttpClient("not-json");
+    AlternatorLiveNodes liveNodes =
+        liveNodes(
+            NodeHealthConfig.builder().withConsecutiveFailureThreshold(1).build(),
+            httpClient,
+            null,
+            Arrays.asList("active.local"));
+
+    liveNodes.refreshDiscoveredNodes();
+
+    assertEquals(Arrays.asList(node("active.local")), liveNodes.getDownNodes());
+  }
+
+  @Test
+  public void featureCheckRuntimeFailureMarksSelectedNodeDown() throws Exception {
+    RuntimeFailureHttpClient httpClient = new RuntimeFailureHttpClient();
+    AlternatorLiveNodes liveNodes =
+        liveNodes(
+            NodeHealthConfig.builder().withConsecutiveFailureThreshold(1).build(),
+            httpClient,
+            null,
+            Arrays.asList("active.local"));
+
+    try {
+      liveNodes.checkIfRackDatacenterFeatureIsSupported();
+      fail("expected feature check failure");
+    } catch (AlternatorLiveNodes.FailedToCheck expected) {
+      // expected
+    }
+
+    assertEquals(Arrays.asList(node("active.local")), liveNodes.getDownNodes());
+  }
+
+  @Test
+  public void featureCheckInvalidBodyMarksSelectedNodeDown() throws Exception {
+    LocalNodesHttpClient httpClient = new LocalNodesHttpClient("not-json");
+    AlternatorLiveNodes liveNodes =
+        liveNodes(
+            NodeHealthConfig.builder().withConsecutiveFailureThreshold(1).build(),
+            httpClient,
+            null,
+            Arrays.asList("active.local"));
+
+    try {
+      liveNodes.checkIfRackDatacenterFeatureIsSupported();
+      fail("expected feature check failure");
+    } catch (AlternatorLiveNodes.FailedToCheck expected) {
+      // expected
+    }
+
+    assertEquals(Arrays.asList(node("active.local")), liveNodes.getDownNodes());
+  }
+
+  @Test
+  public void rediscoveredNodeKeepsPreviousHealthState() throws Exception {
+    LocalNodesHttpClient httpClient = new LocalNodesHttpClient("[\"transient.local\"]");
+    AlternatorLiveNodes liveNodes =
+        liveNodes(
+            NodeHealthConfig.builder().withConsecutiveFailureThreshold(1).build(),
+            httpClient,
+            null,
+            Arrays.asList("active.local"));
+    URI transientNode = node("transient.local");
+
+    liveNodes.refreshDiscoveredNodes();
+    markNodeDown(liveNodes, transientNode);
+    assertEquals(NodeHealthState.DOWN, liveNodes.getNodeHealthStatus(transientNode).getState());
+
+    httpClient.setResponseBody("[\"other.local\"]");
+    liveNodes.refreshDiscoveredNodes();
+    assertFalse(liveNodes.getDiscoveredNodes().contains(transientNode));
+    assertEquals(NodeHealthState.DOWN, liveNodes.getNodeHealthStatus(transientNode).getState());
+
+    httpClient.setResponseBody("[\"transient.local\"]");
+    liveNodes.refreshDiscoveredNodes();
+
+    assertTrue(liveNodes.getDiscoveredNodes().contains(transientNode));
+    assertEquals(NodeHealthState.DOWN, liveNodes.getNodeHealthStatus(transientNode).getState());
+    assertFalse(liveNodes.getLiveNodes().contains(transientNode));
+  }
+
+  @Test
+  public void runDownNodeProbesSkipsDownNodeMissingFromDiscoveredRing() throws Exception {
+    LocalNodesHttpClient httpClient = new LocalNodesHttpClient("[\"transient.local\"]");
+    AlternatorLiveNodes liveNodes =
+        liveNodes(
+            NodeHealthConfig.builder()
+                .withConsecutiveFailureThreshold(1)
+                .withDownNodeRecoverySuccessThreshold(1)
+                .build(),
+            httpClient,
+            null,
+            Arrays.asList("active.local"));
+    URI transientNode = node("transient.local");
+
+    liveNodes.refreshDiscoveredNodes();
+    markNodeDown(liveNodes, transientNode);
+    assertEquals(NodeHealthState.DOWN, liveNodes.getNodeHealthStatus(transientNode).getState());
+
+    httpClient.setResponseBody("[\"other.local\"]");
+    liveNodes.refreshDiscoveredNodes();
+    httpClient.requests.clear();
+
+    assertTrue(liveNodes.runDownNodeProbes().isEmpty());
+    assertTrue(httpClient.requests.isEmpty());
+    assertEquals(NodeHealthState.DOWN, liveNodes.getNodeHealthStatus(transientNode).getState());
+  }
+
+  @Test
+  public void runDownNodeProbesProbesDownSeedMissingFromDiscoveredRing() throws Exception {
+    LocalNodesHttpClient httpClient = new LocalNodesHttpClient("[\"active.local\"]", 1);
+    AlternatorLiveNodes liveNodes =
+        liveNodes(
+            NodeHealthConfig.builder()
+                .withConsecutiveFailureThreshold(1)
+                .withDownNodeRecoverySuccessThreshold(1)
+                .build(),
+            httpClient,
+            null,
+            Arrays.asList("seed.local", "active.local"));
+    URI seed = node("seed.local");
+
+    liveNodes.refreshDiscoveredNodes();
+    assertFalse(liveNodes.getDiscoveredNodes().contains(seed));
+    assertEquals(NodeHealthState.DOWN, liveNodes.getNodeHealthStatus(seed).getState());
+    httpClient.requests.clear();
+
+    assertEquals(Arrays.asList(seed), liveNodes.runDownNodeProbes());
+
+    assertEquals(1, httpClient.requests.size());
+    assertEquals("seed.local", httpClient.requests.get(0).host());
+    assertEquals(NodeHealthState.QUARANTINED, liveNodes.getNodeHealthStatus(seed).getState());
+  }
+
+  @Test
+  public void duplicateDiscoveredDownNodeProbeCountsOncePerCycle() throws Exception {
+    LocalNodesHttpClient httpClient =
+        new LocalNodesHttpClient("[\"recovering.local\",\"recovering.local\"]");
+    AlternatorLiveNodes liveNodes =
+        liveNodes(
+            NodeHealthConfig.builder()
+                .withConsecutiveFailureThreshold(1)
+                .withDownNodeRecoverySuccessThreshold(2)
+                .build(),
+            httpClient,
+            null,
+            Arrays.asList("active.local"));
+    URI recovering = node("recovering.local");
+
+    liveNodes.refreshDiscoveredNodes();
+    markNodeDown(liveNodes, recovering);
+    httpClient.requests.clear();
+
+    assertTrue(liveNodes.runDownNodeProbes().isEmpty());
+
+    assertEquals(1, liveNodes.getNodeHealthStatus(recovering).getConsecutiveSuccesses());
+    assertEquals(1, httpClient.requests.size());
+
+    assertEquals(Arrays.asList(recovering), liveNodes.runDownNodeProbes());
+  }
+
+  @Test
+  public void omittedActiveNodeDoesNotBlockNoActiveFallback() throws Exception {
+    LocalNodesHttpClient httpClient = new LocalNodesHttpClient("[\"stale-active.local\"]");
+    AlternatorLiveNodes liveNodes =
+        liveNodes(
+            NodeHealthConfig.builder().withConsecutiveFailureThreshold(1).build(),
+            httpClient,
+            null,
+            Arrays.asList("seed.local"));
+    URI seed = node("seed.local");
+    URI current = node("current.local");
+
+    liveNodes.refreshDiscoveredNodes();
+    assertTrue(liveNodes.getActiveNodes().contains(node("stale-active.local")));
+
+    httpClient.setResponseBody("[\"current.local\"]");
+    liveNodes.refreshDiscoveredNodes();
+    markNodeDown(liveNodes, seed);
+    markNodeDown(liveNodes, current);
+
+    assertTrue(liveNodes.getActiveNodes().isEmpty());
+    AlternatorLiveNodes.QueryPlanNodeFilter filter = liveNodes.newQueryPlanNodeFilter();
+    assertTrue(filter.shouldRouteTo(current));
+  }
+
+  @Test
+  public void downNodeProbeRequiresHttpOk() {
+    LocalNodesHttpClient httpClient = new LocalNodesHttpClient(403);
+    AlternatorLiveNodes liveNodes =
+        liveNodes(
+            NodeHealthConfig.builder().withConsecutiveFailureThreshold(1).build(), httpClient);
+    URI recovering = node("recovering.local");
+
+    markNodeDown(liveNodes, recovering);
+
+    assertTrue(liveNodes.runDownNodeProbes().isEmpty());
+    assertEquals(Arrays.asList(recovering), liveNodes.getDownNodes());
+  }
+
+  @Test
+  public void refreshDoesNotRunDownNodeProbesWhenBackgroundProbesDisabled() throws Exception {
+    LocalNodesHttpClient httpClient =
+        new LocalNodesHttpClient("[\"active.local\",\"recovering.local\"]");
+    AlternatorLiveNodes liveNodes =
+        liveNodes(
+            NodeHealthConfig.builder().withDownNodeProbePeriodMs(0).build(),
+            httpClient,
+            DatacenterScope.of("dc1", null));
+    URI recovering = node("recovering.local");
+
+    markNodeDown(liveNodes, recovering);
+    httpClient.requests.clear();
+    liveNodes.refreshDiscoveredNodes();
+
+    assertEquals(1, httpClient.requests.size());
+    assertEquals("active.local", httpClient.requests.get(0).host());
+    assertEquals(Arrays.asList(recovering), liveNodes.getDownNodes());
+  }
+
+  @Test
+  public void clusterRefreshDoesNotProbeDownSeedWhenBackgroundProbesDisabled() throws Exception {
+    LocalNodesHttpClient httpClient =
+        new LocalNodesHttpClient("[\"active.local\",\"recovering.local\"]");
+    AlternatorLiveNodes liveNodes =
+        liveNodes(NodeHealthConfig.builder().withDownNodeProbePeriodMs(0).build(), httpClient);
+    URI recovering = node("recovering.local");
+
+    markNodeDown(liveNodes, recovering);
+    httpClient.requests.clear();
+    liveNodes.refreshDiscoveredNodes();
+
+    assertEquals(1, httpClient.requests.size());
+    assertEquals("active.local", httpClient.requests.get(0).host());
+    assertEquals(Arrays.asList(recovering), liveNodes.getDownNodes());
+  }
+
+  @Test
+  public void clusterRefreshDoesNotProbeDownSeedAfterProbePeriod() throws Exception {
+    LocalNodesHttpClient httpClient =
+        new LocalNodesHttpClient("[\"active.local\",\"recovering.local\"]");
+    AlternatorLiveNodes liveNodes =
+        liveNodes(NodeHealthConfig.builder().withDownNodeProbePeriodMs(1).build(), httpClient);
+    URI recovering = node("recovering.local");
+
+    markNodeDown(liveNodes, recovering);
+    Thread.sleep(5);
+    httpClient.requests.clear();
+    liveNodes.refreshDiscoveredNodes();
+
+    assertEquals(1, httpClient.requests.size());
+    assertEquals("active.local", httpClient.requests.get(0).host());
+    assertEquals(Arrays.asList(recovering), liveNodes.getDownNodes());
+  }
+
+  @Test
+  public void backgroundDiscoveryDoesNotMarkClientActivity() throws Exception {
+    LocalNodesHttpClient httpClient =
+        new LocalNodesHttpClient("[\"active.local\",\"recovering.local\"]");
+    AlternatorLiveNodes liveNodes =
+        liveNodes(NodeHealthConfig.getDefault(), httpClient, DatacenterScope.of("dc1", null));
+
+    assertEquals(0, lastActivityTime(liveNodes).get());
+    liveNodes.refreshDiscoveredNodes();
+
+    assertEquals(0, lastActivityTime(liveNodes).get());
+  }
+
+  @Test
+  public void scopedRefreshDoesNotConsumeQuarantineTrafficSample() throws Exception {
+    LocalNodesHttpClient httpClient =
+        new LocalNodesHttpClient("[\"active.local\",\"recovering.local\"]");
+    AlternatorLiveNodes liveNodes =
+        liveNodes(
+            NodeHealthConfig.builder()
+                .withDownNodeRecoverySuccessThreshold(1)
+                .withQuarantineSuccessThreshold(3)
+                .withQuarantineTrafficInterval(2)
+                .build(),
+            httpClient,
+            DatacenterScope.of("dc1", null));
+    URI active = node("active.local");
+    URI recovering = node("recovering.local");
+
+    markNodeDown(liveNodes, recovering);
+    liveNodes.runDownNodeProbes();
+    assertEquals(Arrays.asList(recovering), liveNodes.getQuarantinedNodes());
+    assertEquals(0, quarantineSamplingCounter(liveNodes).get());
+
+    liveNodes.refreshDiscoveredNodes();
+
+    assertEquals(0, quarantineSamplingCounter(liveNodes).get());
+    assertEquals(Arrays.asList(active, recovering), liveNodes.getQueryPlanNodes());
+    AlternatorLiveNodes.QueryPlanNodeFilter firstFilter = liveNodes.newQueryPlanNodeFilter();
+    assertTrue(firstFilter.shouldRouteTo(active));
+    assertFalse(firstFilter.shouldRouteTo(recovering));
+    AlternatorLiveNodes.QueryPlanNodeFilter secondFilter = liveNodes.newQueryPlanNodeFilter();
+    assertTrue(secondFilter.shouldRouteTo(active));
+    assertTrue(secondFilter.shouldRouteTo(recovering));
+  }
+
+  private static AlternatorLiveNodes liveNodes(
+      NodeHealthConfig nodeHealthConfig, SdkHttpClient httpClient) {
+    return liveNodes(nodeHealthConfig, httpClient, null);
+  }
+
+  private static AlternatorLiveNodes liveNodes(
+      NodeHealthConfig nodeHealthConfig, SdkHttpClient httpClient, RoutingScope routingScope) {
+    return liveNodes(
+        nodeHealthConfig,
+        httpClient,
+        routingScope,
+        Arrays.asList("active.local", "recovering.local"));
+  }
+
+  private static AlternatorLiveNodes liveNodes(
+      NodeHealthConfig nodeHealthConfig,
+      SdkHttpClient httpClient,
+      RoutingScope routingScope,
+      List seedHosts) {
+    AlternatorConfig config =
+        AlternatorConfig.builder()
+            .withSeedHosts(seedHosts)
+            .withScheme("http")
+            .withPort(8080)
+            .withActiveRefreshIntervalMs(60_000)
+            .withIdleRefreshIntervalMs(60_000)
+            .withRoutingScope(routingScope)
+            .withNodeHealthConfig(nodeHealthConfig)
+            .build();
+    return new AlternatorLiveNodes(config, httpClient);
+  }
+
+  private static long hashWhereFirstNodeIs(List nodes, URI expected) {
+    for (long hash = -1000; hash < 1000; hash++) {
+      if (expected.equals(AlternatorLiveNodes.firstNodeWithSeed(nodes, hash))) {
+        return hash;
+      }
+    }
+    throw new AssertionError("Could not find hash for " + expected);
+  }
+
+  private static URI node(String host) {
+    return URI.create("http://" + host + ":8080");
+  }
+
+  private static void markNodeDown(AlternatorLiveNodes liveNodes, URI node) {
+    for (int i = 0; i < NodeHealthConfig.DEFAULT_CONSECUTIVE_FAILURE_THRESHOLD; i++) {
+      liveNodes.reportNodeResult(node, NodeHealthObservation.TRAFFIC_FAILURE);
+    }
+  }
+
+  private static List collectNodes(LazyQueryPlan plan) {
+    List nodes = new java.util.ArrayList<>();
+    while (plan.hasNext()) {
+      nodes.add(plan.next());
+    }
+    return nodes;
+  }
+
+  private static AtomicLong lastActivityTime(AlternatorLiveNodes liveNodes) throws Exception {
+    Field field = AlternatorLiveNodes.class.getDeclaredField("lastActivityTime");
+    field.setAccessible(true);
+    return (AtomicLong) field.get(liveNodes);
+  }
+
+  private static AtomicLong quarantineSamplingCounter(AlternatorLiveNodes liveNodes)
+      throws Exception {
+    Field field = AlternatorLiveNodes.class.getDeclaredField("quarantineSamplingCounter");
+    field.setAccessible(true);
+    return (AtomicLong) field.get(liveNodes);
+  }
+
+  private static final class LocalNodesHttpClient implements SdkHttpClient {
+    private final List requests = new CopyOnWriteArrayList<>();
+    private volatile String responseBody;
+    private final String failingHost;
+    private final int statusCode;
+    private final AtomicLong requestsToFail;
+
+    private LocalNodesHttpClient() {
+      this("[]");
+    }
+
+    private LocalNodesHttpClient(int statusCode) {
+      this("[]", null, statusCode);
+    }
+
+    private LocalNodesHttpClient(String responseBody) {
+      this(responseBody, null);
+    }
+
+    private LocalNodesHttpClient(String responseBody, String failingHost) {
+      this(responseBody, failingHost, 200);
+    }
+
+    private LocalNodesHttpClient(String responseBody, int requestsToFail) {
+      this(responseBody, null, 200, requestsToFail);
+    }
+
+    private LocalNodesHttpClient(String responseBody, String failingHost, int statusCode) {
+      this(responseBody, failingHost, statusCode, 0);
+    }
+
+    private LocalNodesHttpClient(
+        String responseBody, String failingHost, int statusCode, long requestsToFail) {
+      this.responseBody = responseBody;
+      this.failingHost = failingHost;
+      this.statusCode = statusCode;
+      this.requestsToFail = new AtomicLong(requestsToFail);
+    }
+
+    private void setResponseBody(String responseBody) {
+      this.responseBody = responseBody;
+    }
+
+    @Override
+    public ExecutableHttpRequest prepareRequest(HttpExecuteRequest request) {
+      SdkHttpRequest httpRequest = request.httpRequest();
+      requests.add(httpRequest);
+      return new ExecutableHttpRequest() {
+        @Override
+        public HttpExecuteResponse call() throws IOException {
+          if (requestsToFail.getAndUpdate(value -> value > 0 ? value - 1 : value) > 0) {
+            throw new IOException("simulated connection failure");
+          }
+          if (httpRequest.host().equals(failingHost)) {
+            throw new IOException("simulated connection failure for " + failingHost);
+          }
+          byte[] body = responseBody.getBytes(StandardCharsets.UTF_8);
+          return HttpExecuteResponse.builder()
+              .response(SdkHttpFullResponse.builder().statusCode(statusCode).build())
+              .responseBody(AbortableInputStream.create(new ByteArrayInputStream(body)))
+              .build();
+        }
+
+        @Override
+        public void abort() {}
+      };
+    }
+
+    @Override
+    public void close() {}
+
+    @Override
+    public String clientName() {
+      return "localnodes";
+    }
+  }
+
+  private static final class RuntimeFailureHttpClient implements SdkHttpClient {
+    @Override
+    public ExecutableHttpRequest prepareRequest(HttpExecuteRequest request) {
+      return new ExecutableHttpRequest() {
+        @Override
+        public HttpExecuteResponse call() {
+          throw new IllegalStateException("simulated runtime transport failure");
+        }
+
+        @Override
+        public void abort() {}
+      };
+    }
+
+    @Override
+    public void close() {}
+
+    @Override
+    public String clientName() {
+      return "runtime-failure";
+    }
+  }
+}
diff --git a/src/test/java/com/scylladb/alternator/internal/AlternatorLiveNodesScopeFallbackTest.java b/src/test/java/com/scylladb/alternator/internal/AlternatorLiveNodesScopeFallbackTest.java
index 3610acd..e35c2d2 100644
--- a/src/test/java/com/scylladb/alternator/internal/AlternatorLiveNodesScopeFallbackTest.java
+++ b/src/test/java/com/scylladb/alternator/internal/AlternatorLiveNodesScopeFallbackTest.java
@@ -28,9 +28,8 @@
  * Tests for GitHub issue #83: Fallback behavior when all nodes in a scope become unreachable.
  *
  * 
    When all nodes in the client's current scope become unreachable (IOException), the scope
- * fallback chain should still be traversed. Currently, the IOException propagates out of
- * updateLiveNodes() and the fallback never happens. These tests assert the correct/desired behavior
- * and are expected to FAIL until the fix is implemented.
+ * fallback chain should still be traversed. These tests assert that fallback behavior and that
+ * configured seed nodes remain discovery candidates without being leaked into scoped routing.
  *
  * @see Issue #83
  */
@@ -108,8 +107,8 @@ public String clientName() {
   }
 
   /**
-   * Verifies that when all nodes are unreachable, updateLiveNodes() should traverse the entire
-   * scope fallback chain (Rack -> DC -> Cluster) before giving up.
+   * Verifies that when all nodes are unreachable, refreshDiscoveredNodes() should traverse the
+   * entire scope fallback chain (Rack -> DC -> Cluster) before giving up.
    *
    * 
    With a 3-level fallback chain, all 3 scope levels should be attempted even when nodes throw
    * IOException. The method should not throw IOException until all scopes have been exhausted.
@@ -130,11 +129,11 @@ public void testScopeFallbackTraversedOnIOException() throws Exception {
 
     AlternatorLiveNodes liveNodes = new AlternatorLiveNodes(config, unreachableClient);
 
-    // updateLiveNodes() should attempt all 3 scope levels before giving up.
+    // refreshDiscoveredNodes() should attempt all 3 scope levels before giving up.
     // It may throw IOException after exhausting all scopes, but the key assertion is
     // that all 3 scopes were tried.
     try {
-      liveNodes.updateLiveNodes();
+      liveNodes.refreshDiscoveredNodes();
     } catch (IOException e) {
       // Acceptable to throw after exhausting all scopes
     }
@@ -149,17 +148,17 @@ public void testScopeFallbackTraversedOnIOException() throws Exception {
   }
 
   /**
-   * Verifies that when all discovered nodes become unreachable, the client should fall back to the
-   * original seed nodes for re-discovery.
+   * Verifies that seed nodes remain available as discovery candidates without being published into
+   * the discovered routing ring.
    *
-   * 
    After a successful discovery replaces liveNodes with new nodes, if those live nodes cannot
-   * return a usable /localnodes response, the client should fall back to the original seed URLs so
-   * it can recover through the configured entry points.
+   * 
    After a successful discovery replaces discovered nodes with new nodes, the seed URL should
+   * stay out of the routable discovered-node list unless it was returned by discovery. Later
+   * discovery cycles should still use the original seed entry point.
    *
    * @throws Exception if an unexpected error occurs
    */
   @Test
-  public void testSeedNodesReusedWhenDiscoveredNodesUnreachable() throws Exception {
+  public void testSeedNodesUsedForDiscoveryButNotPublishedAfterRefresh() throws Exception {
     // Phase 1: Successful discovery returns nodes B,C (not the seed A)
     ReachableHttpClient reachableClient = new ReachableHttpClient("[\"10.0.0.2\",\"10.0.0.3\"]");
 
@@ -168,24 +167,30 @@ public void testSeedNodesReusedWhenDiscoveredNodesUnreachable() throws Exception
 
     AlternatorLiveNodes liveNodes = new AlternatorLiveNodes(config, reachableClient);
 
-    // Initial liveNodes should contain seed (10.0.0.1)
-    List initialList = liveNodes.getLiveNodes();
+    // Initial discovered nodes should contain seed (10.0.0.1)
+    List initialList = liveNodes.getDiscoveredNodes();
     assertEquals(1, initialList.size());
     assertEquals("10.0.0.1", initialList.get(0).getHost());
 
-    // After a successful update, liveNodes should contain discovered nodes without injecting seeds
-    liveNodes.updateLiveNodes();
-    List updatedList = liveNodes.getLiveNodes();
+    // After a successful update, discovered nodes should contain only discovered routing nodes.
+    liveNodes.refreshDiscoveredNodes();
+    List updatedList = liveNodes.getDiscoveredNodes();
     List hosts = new ArrayList<>();
     for (URI uri : updatedList) {
       hosts.add(uri.getHost());
     }
     assertTrue("Should contain discovered node 10.0.0.2", hosts.contains("10.0.0.2"));
     assertTrue("Should contain discovered node 10.0.0.3", hosts.contains("10.0.0.3"));
-
     assertFalse(
-        "Seed node 10.0.0.1 should remain a discovery candidate, not a routing target",
+        "Seed node 10.0.0.1 should not be published as a discovered routing node",
         hosts.contains("10.0.0.1"));
+
+    reachableClient.capturedRequests.clear();
+    liveNodes.refreshDiscoveredNodes();
+
+    assertEquals(2, reachableClient.capturedRequests.size());
+    assertEquals("10.0.0.2", reachableClient.capturedRequests.get(0).host());
+    assertEquals("10.0.0.3", reachableClient.capturedRequests.get(1).host());
   }
 
   /**
@@ -251,7 +256,7 @@ public String clientName() {
     AlternatorLiveNodes liveNodes = new AlternatorLiveNodes(config, scopeAwareClient);
 
     // This should NOT throw - the fallback chain should traverse Rack -> DC -> Cluster
-    liveNodes.updateLiveNodes();
+    liveNodes.refreshDiscoveredNodes();
 
     // All 3 scopes should have been queried (rack returned empty, dc returned empty,
     // cluster returned a node)
@@ -260,21 +265,21 @@ public String clientName() {
         3,
         requestCount.get());
 
-    // liveNodes should now contain only the cluster-scope node
-    List nodes = liveNodes.getLiveNodes();
+    // Discovered nodes should now contain the cluster-scope routing node. The seed remains a
+    // discovery fallback, but is not published unless returned by discovery.
+    List nodes = liveNodes.getDiscoveredNodes();
     List nodeHosts = new ArrayList<>();
     for (URI uri : nodes) {
       nodeHosts.add(uri.getHost());
     }
     assertTrue("Should contain cluster-scope node 10.0.0.5", nodeHosts.contains("10.0.0.5"));
     assertFalse(
-        "Seed node 10.0.0.1 should not be added to the routing list",
-        nodeHosts.contains("10.0.0.1"));
+        "Should not publish seed node 10.0.0.1 as routing node", nodeHosts.contains("10.0.0.1"));
   }
 
   /**
-   * Verifies that repeated updateLiveNodes() calls with all-unreachable discovered nodes should
-   * eventually try the seed nodes, allowing recovery if the seeds are still alive.
+   * Verifies that repeated refreshDiscoveredNodes() calls with all-unreachable discovered nodes
+   * should eventually try the seed nodes, allowing recovery if the seeds are still alive.
    *
    * @throws Exception if an unexpected error occurs
    */
@@ -296,7 +301,7 @@ public void testRepeatedUpdatesWithUnreachableNodesEventuallyTrySeedNodes() thro
     // Simulate multiple update cycles
     for (int i = 0; i < 9; i++) {
       try {
-        liveNodes.updateLiveNodes();
+        liveNodes.refreshDiscoveredNodes();
       } catch (IOException e) {
         // May throw, that's fine for this test
       }
@@ -304,11 +309,11 @@ public void testRepeatedUpdatesWithUnreachableNodesEventuallyTrySeedNodes() thro
 
     // The liveNodes list should remain unchanged (the unreachable nodes should not be removed
     // without replacement)
-    List currentNodes = liveNodes.getLiveNodes();
+    List currentNodes = liveNodes.getDiscoveredNodes();
     assertEquals(
         "liveNodes list should remain unchanged after failed updates", 3, currentNodes.size());
 
-    // Verify the requests cycled through all 3 nodes via round-robin
+    // Verify the requests contacted all 3 seed nodes
     List contactedHosts = new ArrayList<>();
     for (SdkHttpRequest req : unreachableClient.capturedRequests) {
       contactedHosts.add(req.host());
@@ -371,7 +376,7 @@ public String clientName() {
     AlternatorLiveNodes liveNodesEmptyCase =
         new AlternatorLiveNodes(configBuilder.build(), emptyListClient);
 
-    liveNodesEmptyCase.updateLiveNodes();
+    liveNodesEmptyCase.refreshDiscoveredNodes();
     int emptyListScopeAttempts = emptyListRequestCount.get();
 
     // --- Case 2: IOException responses ---
@@ -381,7 +386,7 @@ public String clientName() {
         new AlternatorLiveNodes(configBuilder.build(), ioExceptionClient);
 
     try {
-      liveNodesIOExceptionCase.updateLiveNodes();
+      liveNodesIOExceptionCase.refreshDiscoveredNodes();
     } catch (IOException e) {
       // May throw after exhausting all scopes
     }
diff --git a/src/test/java/com/scylladb/alternator/internal/AlternatorLiveNodesShutdownTest.java b/src/test/java/com/scylladb/alternator/internal/AlternatorLiveNodesShutdownTest.java
index 7174cbf..248fb55 100644
--- a/src/test/java/com/scylladb/alternator/internal/AlternatorLiveNodesShutdownTest.java
+++ b/src/test/java/com/scylladb/alternator/internal/AlternatorLiveNodesShutdownTest.java
@@ -1,12 +1,19 @@
 package com.scylladb.alternator.internal;
 
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
 import static org.junit.Assert.assertNull;
 import static org.junit.Assert.assertTrue;
 
 import com.scylladb.alternator.AlternatorConfig;
+import com.scylladb.alternator.NodeHealthConfig;
 import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.lang.reflect.Constructor;
 import java.net.URI;
 import java.nio.charset.StandardCharsets;
+import java.util.Collections;
+import java.util.List;
 import java.util.concurrent.CountDownLatch;
 import java.util.concurrent.TimeUnit;
 import java.util.concurrent.atomic.AtomicInteger;
@@ -21,6 +28,37 @@
 
 public class AlternatorLiveNodesShutdownTest {
 
+  @Test
+  public void testPollingFailureRespectsRefreshInterval() throws Exception {
+    FailingRefreshLiveNodes liveNodes = new FailingRefreshLiveNodes();
+
+    try {
+      liveNodes.start();
+      assertTrue(
+          "initial polling request should start", liveNodes.firstCall.await(5, TimeUnit.SECONDS));
+
+      assertFalse(
+          "failed polling should wait for the refresh interval before retrying",
+          liveNodes.secondCall.await(500, TimeUnit.MILLISECONDS));
+    } finally {
+      liveNodes.shutdownAndWait(5_000);
+    }
+  }
+
+  @Test
+  public void testDownNodeProbeRunsWhenRefreshFails() throws Exception {
+    FailingRefreshWithProbeLiveNodes liveNodes = new FailingRefreshWithProbeLiveNodes();
+
+    try {
+      liveNodes.start();
+      assertTrue(
+          "down-node probe should run even when refresh keeps failing",
+          liveNodes.probeCall.await(5, TimeUnit.SECONDS));
+    } finally {
+      liveNodes.shutdownAndWait(5_000);
+    }
+  }
+
   @Test
   public void testShutdownPathPollingFailureDoesNotEscapeThread() throws Exception {
     BlockingShutdownHttpClient client = new BlockingShutdownHttpClient();
@@ -61,6 +99,127 @@ public void testRuntimePollingFailureDoesNotStopThread() throws Exception {
     assertTrue("live-node thread should stop", liveNodes.shutdownAndWait(5_000));
   }
 
+  @Test
+  public void testShutdownAndWaitClosesOwnedPollingClientWhenThreadNeverStarted() throws Exception {
+    CloseCountingHttpClient client = new CloseCountingHttpClient();
+    AlternatorLiveNodes liveNodes = newOwnedLiveNodes(client);
+
+    assertTrue("never-started live-node thread should be stopped", liveNodes.shutdownAndWait(0));
+    assertTrue(
+        "second shutdown should remain stopped and close should be idempotent",
+        liveNodes.shutdownAndWait(0));
+
+    assertEquals("owned polling client should close exactly once", 1, client.closeCount.get());
+  }
+
+  private static AlternatorLiveNodes newOwnedLiveNodes(SdkHttpClient client) throws Exception {
+    Constructor constructor =
+        AlternatorLiveNodes.class.getDeclaredConstructor(
+            AlternatorConfig.class, SdkHttpClient.class, boolean.class);
+    constructor.setAccessible(true);
+    return constructor.newInstance(
+        AlternatorConfig.builder().withSeedNode(URI.create("http://127.0.0.1:8000")).build(),
+        client,
+        true);
+  }
+
+  private static final class FailingRefreshLiveNodes extends AlternatorLiveNodes {
+    private static final long REFRESH_INTERVAL_MS = 10_000;
+
+    private final AtomicInteger callCount = new AtomicInteger(0);
+    private final CountDownLatch firstCall = new CountDownLatch(1);
+    private final CountDownLatch secondCall = new CountDownLatch(1);
+
+    private FailingRefreshLiveNodes() {
+      super(failingRefreshConfig(), new UnusedHttpClient());
+    }
+
+    @Override
+    void refreshDiscoveredNodes() throws IOException {
+      int attempt = callCount.incrementAndGet();
+      if (attempt == 1) {
+        firstCall.countDown();
+      } else if (attempt == 2) {
+        secondCall.countDown();
+      }
+      throw new IOException("simulated polling failure");
+    }
+
+    private static AlternatorConfig failingRefreshConfig() {
+      return AlternatorConfig.builder()
+          .withSeedNode(URI.create("http://127.0.0.1:8000"))
+          .withActiveRefreshIntervalMs(REFRESH_INTERVAL_MS)
+          .withIdleRefreshIntervalMs(REFRESH_INTERVAL_MS)
+          .withNodeHealthConfig(NodeHealthConfig.builder().withDownNodeProbePeriodMs(0).build())
+          .build();
+    }
+  }
+
+  private static final class FailingRefreshWithProbeLiveNodes extends AlternatorLiveNodes {
+    private static final long PERIOD_MS = 10;
+
+    private final CountDownLatch probeCall = new CountDownLatch(1);
+
+    private FailingRefreshWithProbeLiveNodes() {
+      super(config(), new UnusedHttpClient());
+    }
+
+    @Override
+    void refreshDiscoveredNodes() throws IOException {
+      throw new IOException("simulated polling failure");
+    }
+
+    @Override
+    List runDownNodeProbes() {
+      probeCall.countDown();
+      return Collections.emptyList();
+    }
+
+    private static AlternatorConfig config() {
+      return AlternatorConfig.builder()
+          .withSeedNode(URI.create("http://127.0.0.1:8000"))
+          .withActiveRefreshIntervalMs(PERIOD_MS)
+          .withIdleRefreshIntervalMs(PERIOD_MS)
+          .withNodeHealthConfig(
+              NodeHealthConfig.builder().withDownNodeProbePeriodMs(PERIOD_MS).build())
+          .build();
+    }
+  }
+
+  private static final class UnusedHttpClient implements SdkHttpClient {
+    @Override
+    public ExecutableHttpRequest prepareRequest(HttpExecuteRequest request) {
+      throw new AssertionError("polling client should not be used by this test");
+    }
+
+    @Override
+    public void close() {}
+
+    @Override
+    public String clientName() {
+      return "unused";
+    }
+  }
+
+  private static final class CloseCountingHttpClient implements SdkHttpClient {
+    private final AtomicInteger closeCount = new AtomicInteger();
+
+    @Override
+    public ExecutableHttpRequest prepareRequest(HttpExecuteRequest request) {
+      throw new AssertionError("polling client should not be used by this test");
+    }
+
+    @Override
+    public void close() {
+      closeCount.incrementAndGet();
+    }
+
+    @Override
+    public String clientName() {
+      return "close-counting";
+    }
+  }
+
   private static final class BlockingShutdownHttpClient implements SdkHttpClient {
     private final CountDownLatch callStarted = new CountDownLatch(1);
     private final CountDownLatch releaseCall = new CountDownLatch(1);
diff --git a/src/test/java/com/scylladb/alternator/internal/ConnectionLeakTest.java b/src/test/java/com/scylladb/alternator/internal/ConnectionLeakTest.java
index ccc79cc..b382ad8 100644
--- a/src/test/java/com/scylladb/alternator/internal/ConnectionLeakTest.java
+++ b/src/test/java/com/scylladb/alternator/internal/ConnectionLeakTest.java
@@ -16,9 +16,9 @@
 /**
  * Tests that HTTP connections are properly managed and not leaked in AlternatorLiveNodes.
  *
- * 
    These tests call {@link AlternatorLiveNodes#updateLiveNodes()} synchronously (no background
- * thread) and verify that connections are properly released by checking that subsequent requests
- * succeed (they would hang/fail if connections were leaked from a small pool).
+ * 
    These tests call {@link AlternatorLiveNodes#refreshDiscoveredNodes()} synchronously (no
+ * background thread) and verify that connections are properly released by checking that subsequent
+ * requests succeed (they would hang/fail if connections were leaked from a small pool).
  */
 public class ConnectionLeakTest {
 
@@ -74,7 +74,7 @@ public void testNoConnectionLeakOnNon200Responses() throws Exception {
 
     // If connections are leaked, this will eventually hang due to pool exhaustion
     for (int i = 0; i < 20; i++) {
-      liveNodes.updateLiveNodes();
+      liveNodes.refreshDiscoveredNodes();
     }
     // If we get here without timeout, connections are properly released
   }
@@ -87,10 +87,10 @@ public void testNoConnectionLeakOnSuccessfulResponses() throws Exception {
     AlternatorLiveNodes liveNodes = createLiveNodes();
 
     for (int i = 0; i < 30; i++) {
-      liveNodes.updateLiveNodes();
+      liveNodes.refreshDiscoveredNodes();
     }
 
-    assertEquals(2, liveNodes.getLiveNodes().size());
+    assertEquals(2, liveNodes.getDiscoveredNodes().size());
   }
 
   /** Verifies malformed successful /localnodes responses do not terminate discovery. */
@@ -102,11 +102,11 @@ public void testMalformedSuccessfulResponsesDoNotThrowOrLeakConnections() throws
 
     for (int i = 0; i < 30; i++) {
       responseBody = malformedBodies[i % malformedBodies.length];
-      liveNodes.updateLiveNodes();
+      liveNodes.refreshDiscoveredNodes();
     }
 
-    assertEquals(1, liveNodes.getLiveNodes().size());
-    assertEquals("127.0.0.1", liveNodes.getLiveNodes().get(0).getHost());
+    assertEquals(1, liveNodes.getDiscoveredNodes().size());
+    assertEquals("127.0.0.1", liveNodes.getDiscoveredNodes().get(0).getHost());
   }
 
   /** Verifies no leaks when alternating between error and success responses. */
@@ -136,7 +136,7 @@ public void testNoConnectionLeakOnAlternatingResponses() throws Exception {
     AlternatorLiveNodes liveNodes = createLiveNodes();
 
     for (int i = 0; i < 40; i++) {
-      liveNodes.updateLiveNodes();
+      liveNodes.refreshDiscoveredNodes();
     }
     // If we get here without timeout, connections are properly released
   }
@@ -156,10 +156,10 @@ public void testConnectionsAreReused() throws Exception {
 
     // Make several requests so the connection pool has a chance to stabilize
     for (int i = 0; i < 20; i++) {
-      liveNodes.updateLiveNodes();
+      liveNodes.refreshDiscoveredNodes();
     }
 
     // All requests should succeed, proving connections are properly managed
-    assertEquals(1, liveNodes.getLiveNodes().size());
+    assertEquals(1, liveNodes.getDiscoveredNodes().size());
   }
 }
diff --git a/src/test/java/com/scylladb/alternator/internal/NodeHealthStoreTest.java b/src/test/java/com/scylladb/alternator/internal/NodeHealthStoreTest.java
new file mode 100644
index 0000000..634a10b
--- /dev/null
+++ b/src/test/java/com/scylladb/alternator/internal/NodeHealthStoreTest.java
@@ -0,0 +1,371 @@
+package com.scylladb.alternator.internal;
+
+import static org.junit.Assert.*;
+
+import com.scylladb.alternator.NodeHealthConfig;
+import com.scylladb.alternator.NodeHealthObservation;
+import com.scylladb.alternator.NodeHealthState;
+import com.scylladb.alternator.NodeHealthStatus;
+import java.net.URI;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.concurrent.atomic.AtomicInteger;
+import org.junit.Test;
+
+public class NodeHealthStoreTest {
+  @Test
+  public void failuresMarkNodeDownAfterConfiguredThreshold() {
+    URI node = node("node1.local");
+    NodeHealthConfig config = NodeHealthConfig.builder().withConsecutiveFailureThreshold(2).build();
+    NodeHealthStore store = new NodeHealthStore(config, Arrays.asList(node));
+
+    assertEquals(Arrays.asList(node), store.getActiveNodes());
+
+    store.reportNodeResult(node, NodeHealthObservation.TRAFFIC_FAILURE);
+
+    assertEquals(Arrays.asList(node), store.getActiveNodes());
+    assertTrue(store.getDownNodes().isEmpty());
+    assertEquals(1, store.getNodeStatus(node).getConsecutiveFailures());
+
+    store.reportNodeResult(node, NodeHealthObservation.TRAFFIC_FAILURE);
+
+    assertTrue(store.getActiveNodes().isEmpty());
+    assertTrue(store.getQuarantinedNodes().isEmpty());
+    assertEquals(Arrays.asList(node), store.getDownNodes());
+    assertEquals(NodeHealthState.DOWN, store.getNodeStatus(node).getState());
+    assertEquals(2, store.getNodeStatus(node).getConsecutiveFailures());
+  }
+
+  @Test
+  public void httpDefaultPortReportMatchesNodeWithoutExplicitPort() {
+    URI configured = URI.create("http://node1.local");
+    URI reported = URI.create("http://node1.local:80");
+    NodeHealthConfig config = NodeHealthConfig.builder().withConsecutiveFailureThreshold(1).build();
+    NodeHealthStore store = new NodeHealthStore(config, Arrays.asList(configured));
+
+    store.reportNodeResult(reported, NodeHealthObservation.TRAFFIC_FAILURE);
+
+    assertTrue(store.getActiveNodes().isEmpty());
+    assertEquals(Arrays.asList(configured), store.getDownNodes());
+    assertEquals(NodeHealthState.DOWN, store.getNodeStatus(configured).getState());
+    assertEquals(NodeHealthState.DOWN, store.getNodeStatus(reported).getState());
+  }
+
+  @Test
+  public void httpsDefaultPortReportMatchesNodeWithoutExplicitPort() {
+    URI configured = URI.create("https://node1.local");
+    URI reported = URI.create("https://node1.local:443");
+    NodeHealthConfig config = NodeHealthConfig.builder().withConsecutiveFailureThreshold(1).build();
+    NodeHealthStore store = new NodeHealthStore(config, Arrays.asList(configured));
+
+    store.reportNodeResult(reported, NodeHealthObservation.TRAFFIC_FAILURE);
+
+    assertTrue(store.getActiveNodes().isEmpty());
+    assertEquals(Arrays.asList(configured), store.getDownNodes());
+    assertEquals(NodeHealthState.DOWN, store.getNodeStatus(configured).getState());
+    assertEquals(NodeHealthState.DOWN, store.getNodeStatus(reported).getState());
+  }
+
+  @Test
+  public void consecutiveFailuresMarkNodeDown() {
+    URI node = node("node1.local");
+    NodeHealthConfig config = NodeHealthConfig.builder().withConsecutiveFailureThreshold(2).build();
+    NodeHealthStore store = new NodeHealthStore(config, Arrays.asList(node));
+
+    store.reportNodeResult(node, NodeHealthObservation.TRAFFIC_FAILURE);
+    assertEquals(Arrays.asList(node), store.getActiveNodes());
+    assertEquals(1, store.getNodeStatus(node).getConsecutiveFailures());
+
+    store.reportNodeResult(node, NodeHealthObservation.TRAFFIC_FAILURE);
+    assertTrue(store.getActiveNodes().isEmpty());
+    assertEquals(Arrays.asList(node), store.getDownNodes());
+    assertEquals(NodeHealthState.DOWN, store.getNodeStatus(node).getState());
+    assertEquals(2, store.getNodeStatus(node).getConsecutiveFailures());
+  }
+
+  @Test
+  public void downNodeMovesToQuarantineAfterConfiguredRecoverySuccesses() {
+    URI node = node("node1.local");
+    NodeHealthConfig config =
+        NodeHealthConfig.builder()
+            .withConsecutiveFailureThreshold(1)
+            .withDownNodeRecoverySuccessThreshold(2)
+            .withQuarantineSuccessThreshold(1)
+            .build();
+    NodeHealthStore store = new NodeHealthStore(config, Arrays.asList(node));
+
+    reportFailures(store, node, config.getConsecutiveFailureThreshold());
+    assertEquals(Arrays.asList(node), store.getDownNodes());
+
+    assertEquals(
+        Collections.emptyList(),
+        store.probeDownNodes(
+            (uri, status) ->
+                status.getState() == NodeHealthState.DOWN
+                    ? NodeHealthObservation.PROBE_SUCCESS
+                    : NodeHealthObservation.PROBE_FAILURE));
+
+    assertEquals(Arrays.asList(node), store.getDownNodes());
+    NodeHealthStatus firstSuccessStatus = store.getNodeStatus(node);
+    assertEquals(NodeHealthState.DOWN, firstSuccessStatus.getState());
+    assertEquals(1, firstSuccessStatus.getConsecutiveSuccesses());
+
+    assertEquals(
+        Arrays.asList(node),
+        store.probeDownNodes(
+            (uri, status) ->
+                status.getState() == NodeHealthState.DOWN
+                    ? NodeHealthObservation.PROBE_SUCCESS
+                    : NodeHealthObservation.PROBE_FAILURE));
+
+    assertTrue(store.getActiveNodes().isEmpty());
+    assertEquals(Arrays.asList(node), store.getQuarantinedNodes());
+    assertTrue(store.getDownNodes().isEmpty());
+    NodeHealthStatus status = store.getNodeStatus(node);
+    assertEquals(NodeHealthState.QUARANTINED, status.getState());
+    assertEquals(0, status.getConsecutiveSuccesses());
+  }
+
+  @Test
+  public void downNodeProbeCountsDuplicateCandidatesOncePerCycle() {
+    URI node = URI.create("http://node1.local");
+    URI nodeWithDefaultPort = URI.create("http://node1.local:80");
+    NodeHealthConfig config =
+        NodeHealthConfig.builder()
+            .withConsecutiveFailureThreshold(1)
+            .withDownNodeRecoverySuccessThreshold(2)
+            .build();
+    NodeHealthStore store = new NodeHealthStore(config, Arrays.asList(node));
+    AtomicInteger probes = new AtomicInteger();
+
+    store.reportNodeResult(node, NodeHealthObservation.TRAFFIC_FAILURE);
+
+    assertTrue(
+        store
+            .probeDownNodes(
+                Arrays.asList(node, nodeWithDefaultPort, node),
+                (uri, status) -> {
+                  probes.incrementAndGet();
+                  return NodeHealthObservation.PROBE_SUCCESS;
+                })
+            .isEmpty());
+
+    assertEquals(1, probes.get());
+    assertEquals(NodeHealthState.DOWN, store.getNodeStatus(node).getState());
+    assertEquals(1, store.getNodeStatus(node).getConsecutiveSuccesses());
+  }
+
+  @Test
+  public void probeSuccessDoesNotClearActiveTrafficFailures() {
+    URI node = node("node1.local");
+    NodeHealthConfig config = NodeHealthConfig.builder().withConsecutiveFailureThreshold(2).build();
+    NodeHealthStore store = new NodeHealthStore(config, Arrays.asList(node));
+
+    store.reportNodeResult(node, NodeHealthObservation.TRAFFIC_FAILURE);
+    store.reportNodeResult(node, NodeHealthObservation.PROBE_SUCCESS);
+
+    assertEquals(1, store.getNodeStatus(node).getConsecutiveFailures());
+    assertEquals(NodeHealthState.ACTIVE, store.getNodeStatus(node).getState());
+
+    store.reportNodeResult(node, NodeHealthObservation.TRAFFIC_FAILURE);
+
+    assertEquals(Arrays.asList(node), store.getDownNodes());
+    assertEquals(2, store.getNodeStatus(node).getConsecutiveFailures());
+  }
+
+  @Test
+  public void trafficSuccessDoesNotClearActiveProbeFailures() {
+    URI node = node("node1.local");
+    NodeHealthConfig config = NodeHealthConfig.builder().withConsecutiveFailureThreshold(2).build();
+    NodeHealthStore store = new NodeHealthStore(config, Arrays.asList(node));
+
+    store.reportNodeResult(node, NodeHealthObservation.PROBE_FAILURE);
+    store.reportNodeResult(node, NodeHealthObservation.TRAFFIC_SUCCESS);
+
+    assertEquals(1, store.getNodeStatus(node).getConsecutiveFailures());
+    assertEquals(NodeHealthState.ACTIVE, store.getNodeStatus(node).getState());
+
+    store.reportNodeResult(node, NodeHealthObservation.PROBE_FAILURE);
+
+    assertEquals(Arrays.asList(node), store.getDownNodes());
+    assertEquals(2, store.getNodeStatus(node).getConsecutiveFailures());
+  }
+
+  @Test
+  public void probeSuccessClearsActiveProbeFailures() {
+    URI node = node("node1.local");
+    NodeHealthConfig config = NodeHealthConfig.builder().withConsecutiveFailureThreshold(2).build();
+    NodeHealthStore store = new NodeHealthStore(config, Arrays.asList(node));
+
+    store.reportNodeResult(node, NodeHealthObservation.PROBE_FAILURE);
+    store.reportNodeResult(node, NodeHealthObservation.PROBE_SUCCESS);
+    store.reportNodeResult(node, NodeHealthObservation.PROBE_FAILURE);
+
+    assertEquals(NodeHealthState.ACTIVE, store.getNodeStatus(node).getState());
+    assertEquals(1, store.getNodeStatus(node).getConsecutiveFailures());
+  }
+
+  @Test
+  public void downTrafficSuccessDoesNotMoveNodeToQuarantine() {
+    URI node = node("node1.local");
+    NodeHealthConfig config =
+        NodeHealthConfig.builder()
+            .withConsecutiveFailureThreshold(1)
+            .withDownNodeRecoverySuccessThreshold(2)
+            .withQuarantineSuccessThreshold(1)
+            .build();
+    NodeHealthStore store = new NodeHealthStore(config, Arrays.asList(node));
+
+    reportFailures(store, node, config.getConsecutiveFailureThreshold());
+
+    store.reportNodeResult(node, NodeHealthObservation.TRAFFIC_SUCCESS);
+    store.reportNodeResult(node, NodeHealthObservation.TRAFFIC_SUCCESS);
+
+    NodeHealthStatus status = store.getNodeStatus(node);
+    assertEquals(NodeHealthState.DOWN, status.getState());
+    assertEquals(0, status.getConsecutiveFailures());
+    assertEquals(0, status.getConsecutiveSuccesses());
+    assertEquals(Arrays.asList(node), store.getDownNodes());
+    assertTrue(store.getQuarantinedNodes().isEmpty());
+  }
+
+  @Test
+  public void quarantinePromotesToActiveAfterConfiguredTrafficSuccesses() {
+    URI node = node("node1.local");
+    NodeHealthConfig config =
+        NodeHealthConfig.builder()
+            .withConsecutiveFailureThreshold(1)
+            .withDownNodeRecoverySuccessThreshold(1)
+            .withQuarantineSuccessThreshold(2)
+            .build();
+    NodeHealthStore store = new NodeHealthStore(config, Arrays.asList(node));
+
+    reportFailures(store, node, config.getConsecutiveFailureThreshold());
+    store.reportNodeResult(node, NodeHealthObservation.PROBE_SUCCESS);
+    assertEquals(Arrays.asList(node), store.getQuarantinedNodes());
+
+    store.reportNodeResult(node, NodeHealthObservation.TRAFFIC_SUCCESS);
+    assertEquals(Arrays.asList(node), store.getQuarantinedNodes());
+    assertEquals(1, store.getNodeStatus(node).getConsecutiveSuccesses());
+
+    store.reportNodeResult(node, NodeHealthObservation.TRAFFIC_SUCCESS);
+
+    assertEquals(Arrays.asList(node), store.getActiveNodes());
+    assertTrue(store.getQuarantinedNodes().isEmpty());
+    assertTrue(store.getDownNodes().isEmpty());
+    assertEquals(NodeHealthState.ACTIVE, store.getNodeStatus(node).getState());
+    assertEquals(2, store.getNodeStatus(node).getConsecutiveSuccesses());
+  }
+
+  @Test
+  public void downNodeDoesNotPromoteImmediatelyWhenQuarantineThresholdIsOne() {
+    URI node = node("node1.local");
+    NodeHealthConfig config =
+        NodeHealthConfig.builder()
+            .withConsecutiveFailureThreshold(1)
+            .withDownNodeRecoverySuccessThreshold(1)
+            .withQuarantineSuccessThreshold(1)
+            .build();
+    NodeHealthStore store = new NodeHealthStore(config, Arrays.asList(node));
+
+    store.reportNodeResult(node, NodeHealthObservation.TRAFFIC_FAILURE);
+    store.reportNodeResult(node, NodeHealthObservation.PROBE_SUCCESS);
+
+    assertEquals(Arrays.asList(node), store.getQuarantinedNodes());
+    assertTrue(store.getActiveNodes().isEmpty());
+    assertEquals(NodeHealthState.QUARANTINED, store.getNodeStatus(node).getState());
+    assertEquals(0, store.getNodeStatus(node).getConsecutiveSuccesses());
+
+    store.reportNodeResult(node, NodeHealthObservation.TRAFFIC_SUCCESS);
+
+    assertEquals(Arrays.asList(node), store.getActiveNodes());
+    assertTrue(store.getQuarantinedNodes().isEmpty());
+    assertTrue(store.getDownNodes().isEmpty());
+    assertEquals(NodeHealthState.ACTIVE, store.getNodeStatus(node).getState());
+    assertEquals(1, store.getNodeStatus(node).getConsecutiveSuccesses());
+  }
+
+  @Test
+  public void probeSuccessDoesNotPromoteQuarantinedNode() {
+    URI node = node("node1.local");
+    NodeHealthConfig config =
+        NodeHealthConfig.builder()
+            .withConsecutiveFailureThreshold(3)
+            .withDownNodeRecoverySuccessThreshold(1)
+            .withQuarantineSuccessThreshold(2)
+            .build();
+    NodeHealthStore store = new NodeHealthStore(config, Arrays.asList(node));
+
+    reportFailures(store, node, config.getConsecutiveFailureThreshold());
+    store.reportNodeResult(node, NodeHealthObservation.PROBE_SUCCESS);
+    assertEquals(Arrays.asList(node), store.getQuarantinedNodes());
+
+    store.reportNodeResult(node, NodeHealthObservation.TRAFFIC_SUCCESS);
+
+    assertTrue(store.getActiveNodes().isEmpty());
+    assertEquals(Arrays.asList(node), store.getQuarantinedNodes());
+    assertEquals(NodeHealthState.QUARANTINED, store.getNodeStatus(node).getState());
+    assertEquals(1, store.getNodeStatus(node).getConsecutiveSuccesses());
+
+    store.reportNodeResult(node, NodeHealthObservation.PROBE_SUCCESS);
+
+    assertTrue(store.getActiveNodes().isEmpty());
+    assertEquals(Arrays.asList(node), store.getQuarantinedNodes());
+    assertEquals(NodeHealthState.QUARANTINED, store.getNodeStatus(node).getState());
+    assertEquals(1, store.getNodeStatus(node).getConsecutiveSuccesses());
+
+    store.reportNodeResult(node, NodeHealthObservation.TRAFFIC_SUCCESS);
+
+    assertEquals(Arrays.asList(node), store.getActiveNodes());
+    assertTrue(store.getQuarantinedNodes().isEmpty());
+    assertEquals(NodeHealthState.ACTIVE, store.getNodeStatus(node).getState());
+  }
+
+  @Test
+  public void quarantineFailureReturnsNodeToDown() {
+    URI node = node("node1.local");
+    NodeHealthConfig config =
+        NodeHealthConfig.builder()
+            .withConsecutiveFailureThreshold(3)
+            .withDownNodeRecoverySuccessThreshold(1)
+            .withQuarantineSuccessThreshold(2)
+            .build();
+    NodeHealthStore store = new NodeHealthStore(config, Arrays.asList(node));
+
+    reportFailures(store, node, config.getConsecutiveFailureThreshold());
+    store.reportNodeResult(node, NodeHealthObservation.PROBE_SUCCESS);
+    assertEquals(Arrays.asList(node), store.getQuarantinedNodes());
+
+    store.reportNodeResult(node, NodeHealthObservation.TRAFFIC_FAILURE);
+
+    assertTrue(store.getQuarantinedNodes().isEmpty());
+    assertEquals(Arrays.asList(node), store.getDownNodes());
+    assertEquals(NodeHealthState.DOWN, store.getNodeStatus(node).getState());
+    assertEquals(3, store.getNodeStatus(node).getConsecutiveFailures());
+  }
+
+  @Test
+  public void disabledKeepsAllNodesActive() {
+    URI node = node("node1.local");
+    NodeHealthStore store = new NodeHealthStore(NodeHealthConfig.disabled(), Arrays.asList(node));
+
+    store.reportNodeResult(node, NodeHealthObservation.TRAFFIC_FAILURE);
+
+    assertEquals(Arrays.asList(node), store.getActiveNodes());
+    assertTrue(store.getQuarantinedNodes().isEmpty());
+    assertTrue(store.getDownNodes().isEmpty());
+    assertEquals(
+        Collections.emptyList(),
+        store.probeDownNodes((uri, status) -> NodeHealthObservation.PROBE_SUCCESS));
+  }
+
+  private static URI node(String host) {
+    return URI.create("http://" + host + ":8080");
+  }
+
+  private static void reportFailures(NodeHealthStore store, URI node, int count) {
+    for (int i = 0; i < count; i++) {
+      store.reportNodeResult(node, NodeHealthObservation.TRAFFIC_FAILURE);
+    }
+  }
+}
diff --git a/src/test/java/com/scylladb/alternator/keyrouting/BatchWriteItemKeyRouteAffinityCrossLanguageTest.java b/src/test/java/com/scylladb/alternator/keyrouting/BatchWriteItemKeyRouteAffinityCrossLanguageTest.java
index eb2ae94..ed9a7db 100644
--- a/src/test/java/com/scylladb/alternator/keyrouting/BatchWriteItemKeyRouteAffinityCrossLanguageTest.java
+++ b/src/test/java/com/scylladb/alternator/keyrouting/BatchWriteItemKeyRouteAffinityCrossLanguageTest.java
@@ -4,7 +4,6 @@
 import static org.junit.Assert.assertTrue;
 
 import com.scylladb.alternator.internal.AlternatorLiveNodes;
-import com.scylladb.alternator.internal.LazyQueryPlan;
 import com.scylladb.alternator.keyrouting.KeyAffinityRequestClassifier.BatchWriteRoutingTarget;
 import java.net.URI;
 import java.net.URISyntaxException;
@@ -99,7 +98,7 @@ private static List votePreferenceOrder(
       }
       try {
         long hash = AttributeValueHasher.hash(pkValue);
-        URI preferredNode = LazyQueryPlan.preferredNodeForHash(liveNodes, hash);
+        URI preferredNode = liveNodes.getPreferredQueryPlanNodeForHash(hash);
         if (preferredNode != null) {
           votes.merge(preferredNode, 1, Integer::sum);
         }