Fix dead-seed eviction in /localnodes refresh + hot-path optimizations

[CodeLieutenant]

Summary

Fixes the bug behind scylladb/alternator-client-java#88: when the seed Alternator node went down during a rolling upgrade, the load balancer kept round-robining caller traffic onto it, producing tens of thousands of connection refused errors even though the rest of the cluster was healthy.

Reproducer (in YCSB, scylladb/YCSB PR linked separately) confirmed the root cause is in this library, not in YCSB — see the PR description below and the new AlternatorLiveNodesDeadSeedTest.

The branch contains two commits, intentionally split so the fix can be reviewed in isolation from the unrelated micro-optimizations.

Commit 1 — fix(live-nodes): evict dead seeds from refresh rotation

Two root causes in AlternatorLiveNodes.updateLiveNodes():

1. One-attempt-per-scope. Each refresh attempted /localnodes against a single node per scope and gave up on the scope if that one call failed. A refresh that happened to pick the dead seed could not see the rest of the cluster.
2. Seeds unconditionally re-injected. mergeWithInitialNodes() re-added every initial seed to the live list after each refresh, so the dead seed was repeatedly put back into the round-robin rotation.

Fix:

• Iterate every known live node within each scope before falling back to the next; a node that fails IOException in one scope is still retried in the next (preserves the existing scope-fallback / empty-list symmetry).
• Replace mergeWithInitialNodes() with mergePostRefresh(): the responding peer's view of /localnodes is treated as authoritative. The responder is retained (it just proved itself alive), nodes that failed earlier in this cycle are excluded, and dead seeds are no longer silently re-injected.
• When every known node is unreachable, restore the original seed list as a last-resort recovery candidate (preserves testRepeatedUpdatesWithUnreachableNodesEventuallyTrySeedNodes).
• Mark refresh activity explicitly so the poller stays at the active interval; the old code did this implicitly via nextAsURI(), which the new selection path no longer calls.

Adds AlternatorLiveNodesDeadSeedTest covering: dead-seed eviction once a refresh discovers healthy peers; first-pick failure still contacts a peer; seed list restored when every node is unreachable.

Commit 2 — perf+correctness: hot-path allocations and refresh efficiency

Bundle of small, individually-trivial improvements. None changes observable behavior beyond the IndexOutOfBoundsException fix.

LazyQueryPlan (per-request hot path):

• Three-tier computeNextNonSeeded(): first-call direct pick, second-call linear skip, third+ call HashSet filter. Eliminates the per-request ArrayList allocation on the happy path.
• usedNodes HashSet lazily allocated only on the third consumption; seeded (key-affinity) mode never touches it. Single-field state for the common case.

AlternatorLiveNodes:

• nextAsURI() uses Math.floorMod instead of Math.abs(... % size). Math.abs(Integer.MIN_VALUE) is still negative, so the previous code would throw IndexOutOfBoundsException once every ~2³¹ calls when the AtomicInteger wrapped.
• getLiveNodes() drops the redundant ArrayList copy: the list stored in the AtomicReference is always built internally and never mutated.
• /localnodes JSON parsing replaced with a small streaming parser — tolerates whitespace and escaped chars, no regex compile per refresh, skips malformed entries.
• hostToURI() skips toURL() validation after validateConfig() has proven the scheme/port pair valid once.
• Constructor builds initial node URIs via hostToURI() (deduplicated).
• consumeAndClose() uses a ThreadLocal<byte[1024]> drain buffer instead of allocating per response cleanup.

BasicQueryPlanInterceptor:

• modifyHttpRequest() only sets Connection: keep-alive when the inbound request doesn't already have a Connection header (both Apache and CRT SDK clients add it by default).

Test plan

• mvn -DskipITs test — 656 / 656 unit tests pass (3 new in AlternatorLiveNodesDeadSeedTest).
• YCSB reproducer (AlternatorSeedFailoverFunctionalTest, opened as a separate PR against scylladb/YCSB) — pre-fix: 94/200 post-kill updates fail with Connection reset; post-fix: 200/200 OK, zero requests dispatched to the dead seed.

Related

• Fixes Failed to recover when node sending /localnodes goes down #88
• Production failure: https://scylladb.atlassian.net/browse/SCT-370

[soyacz]

@nikagra this is hurting our testing. Can you take a look and merge?

[CodeLieutenant]

@dkropachev Can you review this PR, if it's the right direction, or something is off, it's pretty urgent as loadbalacing feature inside YCBS does not work reliably without it.

If the problems are the optimizations I did, they are in a separate commit, I can extract them and send a separate PR

[nikagra]

Generally looks good, I would only ask more coverage for parseLocalNodes method

[CodeLieutenant]

@nikagra Can you review again please

[nikagra]

LGTM 👍

[CodeLieutenant]

@nikagra PR ready, conflicts resolved. Can we merge it and have a new release?

[dkropachev]

@CodeLieutenant , thanks for the PR, unfortunately it is not what we do in other clients, it is better to switchover to #136