Configurable Retry Status Codes and Max Retry Count until Node Failover - RestClient

[sebastian-mue]

Description

Affected versions: 8.x
Related issue: #25578 — RestClient: allow to customize retry logic

Problem Description

The low-level RestClient decides whether a response is retryable and therefore whether to rotate to another node via the method RestClient#isRetryStatus(int), which is hardcoded to only treat 502, 503, and 504 as retryable.
A consequence is that other HTTP codes, for instance HTTP 429 (Too Many Requests), are not treated as retryable, even though there are scenarios in which a failover to a different node is definitely required.

Impact of the Current Behavior

A single overloaded or throttling node effectively poisons every request routed to it, even when the cluster as a whole has ample healthy capacity:

• One ES node returning 4xxs or 5xx unequal to 502, 503 and 504 can lead to a huge amount of unanswered Elasticsearch requests, because the client keeps routing to it and never tries a different ES node for the same request. End users see errors despite the cluster being nominally healthy.
• The FailureListener and dead-node tracking are only triggered for the three hardcoded statuses, so a node consistently returning 429 is never marked dead and never taken out of rotation.
• The Elasticsearch Sniffer does not help. The issue is not which nodes exist, but which status codes justify rotating between them.
• There is no extension point. Applications must either wrap every call in their own retry logic (defeating the purpose of the client's built-in node-rotation machinery), use a HttpResponseInterceptor to fake HTTP response codes or fork the client.

Real-world example we experienced: In a production system with 32 Elasticsearch worker nodes, we encountered an incident where the node the ElasticsearchClient was routing requests to consistently responded with HTTP 429. Because 429 is not treated as retryable, no failover to any of the healthy nodes took place, and data in Elasticsearch could not be retrieved for hours despite 31 other nodes being available to serve the requests.

Proposed Solution

Make the retry policy configurable.

1. Configurable retry status codes. A builder method to supply a set of retryable HTTP status codes, defaulting to the current {502, 503, 504}. Users who need 429 (or 408, etc.) treated as retryable can add them explicitly.
2. Configurable maximum retry count per node. A builder method to configure how many failing requests (HTTP codes unequal to 1xx, 2xx and 3xx) against a node are allowed until the ElasticsearchClient failovers to a different ES node.

[l-trotta]

Hello! A configurable retry mechanism on the client is something we've been thinking about for a long time, there's a PR I've worked on for some time, which has been reviewed, but I recently didn't have the time to complete it ^^".

If you'd like to contribute to this, I'd suggest starting from the existing PR and with the reviews, otherwise I'll get to it hopefully soon. The key requirements for this feature are the following:

1. Must reuse the BackoffPolicy we already have from the BulkIngester
2. Must not be specific to any client implementation (must work regardless of RestClient or Rest5Client)

I explained all the details here.
As for the features you require, Configurable maximum retry count per node. is already a part of BackoffPolicy, while Configurable retry status codes I'd say we can add to the requirement list.

[sebastian-mue]

Thanks a lot for the quick and detailed response, and great to hear this is still on the radar! 🙌
I'll take a look at the existing PR and the requirements you linked to get a better picture of the scope. Once I've had a chance to go through it, I'll come back here to let you know whether I can pick it up from where it stands.

[sebastian-mue]

@l-trotta I'm currently having a look on your PR. Did I get it right that the goal is to execute retries for failed requests but no failover (taken care of on a lower level) to a different node if all retries failed?

[l-trotta]

@sebastian-mue different node retry is something that the underlying RestClient (or Rest5Client) already does:

• RestClient
• Rest5Client

[sebastian-mue]

@l-trotta Thanks for the clarification. I asked because the different-node retry in RestClient/Rest5Client doesn't cover all failure scenarios that warrant a node failover from the client's perspective.

Here's the issue with status codes like HTTP 429:

• httpResponse = client.execute(…); succeeds — the node responds, just with HTTP 429.
• Since no exception is thrown, the catch block that triggers node rotation is never entered.
• In convertResponse(), isRetryStatus() only returns true for 502, 503, and 504 — so 429 is not considered retryable against a different node.
• The request fails, no other node is tried, and the client keeps routing to the same overloaded node.

In our production incident, this meant a single node returning 429 blocked all traffic for hours, despite 31 healthy nodes being available. This may well be a platform-level issue, but from a client perspective, when a node signals it cannot handle a request, the expectation is that the client tries a different node.

Is there any plan to either make isRetryStatus() configurable in RestClient/Rest5Client, or provide another mechanism at the transport level that ensures failover to a different node when a configurable set of status codes is encountered, or when a maximum number of retries against the same node has been exhausted?

[l-trotta]

@sebastian-mue there's no plan to update the RestClient as far as I know, but the Rest5Client can be updated to handle the 429 case better.

[sebastian-mue]

@l-trotta does this imply that a PR also updating the legacy RestClient would probably not get accepted?

[l-trotta]

@sebastian-mue the legacy RestClient is part of the elasticsearch repo and changes are accepted very rarely, mostly for security fixes or bugs