Redis consumer crash: RESP protocol desync ("ERR unknown command '$11'") from reused pooled connections

[pditommaso]

Summary

Consumers backed by the Redis stream/work-queue libraries can crash with a RESP
protocol-desync error such as:

redis.clients.jedis.exceptions.JedisDataException: ERR unknown command '$11', with args beginning with:
    at redis.clients.jedis.Jedis.xautoclaim(Jedis.java:9658)
    at io.seqera.data.stream.impl.RedisMessageStream.claimMessage(RedisMessageStream.groovy:191)
    at io.seqera.data.stream.impl.RedisMessageStream.consume(RedisMessageStream.groovy:142)
    at io.seqera.data.stream.AbstractMessageStream.processMessages(AbstractMessageStream.groovy:225)

xautoclaim here is the victim, not the cause: it is simply the first command
issued on a pooled Jedis connection whose read buffer was already out of sync
from a previous use.

Root cause

A pooled RESP connection relies on one invariant every time it is returned to the pool:

every command written has had its full reply read back; the socket read buffer is drained.

Jedis does not correlate replies to requests — it reads the next bytes off the socket
as the reply to whatever it last sent. If a connection is ever returned to the pool
with one reply left unread, it is permanently offset by one: the next borrower reads
the leftover reply as its own answer, and so on. Eventually the server parses leftover
reply bytes as a command — it hits a bulk-string length header ($11\r\n…, an 11-byte
value from the prior reply) where RESP command framing expects an array (*<argc>), and
reports ERR unknown command '$11'. The 11 is just the byte-length of some leftover value.

Two conditions must both hold for a corrupt connection to be reused rather than discarded:

1. the buffer is unbalanced (a reply went unread), and
2. Jedis did not flag the connection broken.

This is the trap: Jedis only sets broken=true on a socket-level failure
(IOException/JedisConnectionException). A server error reply (JedisDataException
— -ERR, -NOGROUP, -WRONGTYPE, …) is a valid protocol event, so the connection is
considered healthy and goes back into the pool. A fault that leaves the buffer offset
without tripping the socket-broken flag therefore yields a connection that is both corrupt
and eligible for reuse — and because a RESP offset is permanent for the life of the
connection, it then fails on every subsequent borrow. That's why this surfaces as a
recurring error rather than a one-off.

Why the old code opened the window

The pre-rewrite RedisMessageStream.consume() (Groovy) held one connection across a
compound, non-atomic sequence:

getResource()
  → xautoclaim
  → xreadgroup
  → consumer.accept(…)   ← arbitrary user code, connection still held open
  → MULTI / xack / xdel / EXEC
close()  → back to pool

Any failure in that chain that surfaces as something other than a socket break (an aborted
MULTI whose +QUEUED/EXEC replies weren't drained, a data-type error mid-transaction,
an exception thrown while the connection sits mid-sequence) returns the connection offset-by-one
but not flagged broken. Holding it across the user callback widens that window enormously.

Why this still matters in the new version (lib-data-workqueue-redis)

lib-data-stream-redis is deprecated and superseded by lib-data-workqueue(-redis), so the fix
belongs there — that is the code everyone migrates onto.

The good news: RedisWorkQueue already refactored to per-command borrows — receive() /
ack() / renewLease() each open their own short try (Jedis jedis = pool.getResource())
scope, and no connection is held across consumer.accept(...). That closes the main
structural window.

But the underlying hazard is not fully removed, for two reasons:

1. ack() still uses MULTI/EXEC (RedisWorkQueue.ack()), and any transaction that fails
   to drain cleanly is exactly the kind of fault that leaves a buffer offset without marking the
   connection broken.
2. The shared pool has no connection validation. JedisPoolFactory configures only
   minIdle/maxIdle/maxTotal — there is no testOnBorrow, testOnReturn, or testWhileIdle.
   So if any operation ever leaves a connection desynced-but-not-broken, the pool will keep
   handing that poisoned connection back out indefinitely. Per-command scoping makes the bug much
   less likely to be triggered, but nothing makes the pool self-heal once it happens.

Recommendation

Primary (pool-wide, defends the whole class of bug regardless of which command desyncs):
enable connection validation in JedisPoolFactory, e.g.

config.setTestOnBorrow(true);   // or testWhileIdle=true + a time-between-eviction-runs

A desynced connection fails its validation PING and is evicted instead of served, so a single
transient fault can no longer poison the pool for the life of the connection.

Secondary (defense in depth in RedisWorkQueue): ensure the ack() transaction always drains
(explicitly discard()/close the Transaction on error), and treat a JedisDataException on any
stream op as a reason to discard the connection rather than return it.

References

• New impl: lib-data-workqueue-redis/src/main/java/io/seqera/data/workqueue/redis/RedisWorkQueue.java (ack(), consume())
• Pool: lib-jedis-pool/src/main/java/io/seqera/jedis/JedisPoolFactory.java
• Old impl that produced the trace: RedisMessageStream.groovy (pre-rewrite lib-data-stream-redis)

Relationship to Redis restart / failover

Worth correlating in triage: the disruption most likely to seed a poisoned connection is
Redis restarting, failing over, or an idle TCP connection being dropped by a proxy/ELB —
not the application shutting down.

The discriminator is whether the disruption surfaces as a socket IOException:

• A clean TCP teardown (reset/EOF) raises JedisConnectionException, flags the connection
  broken, and the pool discards it on return — self-correcting, cannot cause the $11 reuse.
• Only a fault that leaves the read buffer offset without breaking the socket recycles a
  corrupt connection. A Redis bounce / half-open connection / proxy-buffered stale bytes is the
  realistic source of that, and since JedisPoolFactory has no testOnBorrow/testWhileIdle,
  those stale connections are handed back out unvalidated.

This is the same root the recommended fix addresses: testWhileIdle/testOnBorrow would evict a
stale post-restart connection before it is ever used — the classic mitigation for "errors
clustered right after a Redis failover/restart".

For whoever triages: check whether the $11 occurrences cluster shortly after a Redis
restart, failover, or network blip on the Redis path.