Rework work queue around invocation-scoped leases

[pditommaso]

Summary

• make lib-data-workqueue concurrency invocation-scoped: a permit and thread are held only while one handler call executes
• leave non-terminal Redis entries in the Stream PEL and reclaim them after visibility timeout, with weighted fairness between new intake and retries
• heartbeat only active handler invocations, including handlers longer than visibility/max-processing time, without timeout-driven overlap
• keep lib-data-workqueue-redis at-least-once and backed only by Redis Stream data structures
• move near-exactly-once coordination into lib-cmd-queue-redis with renewable command-attempt guards, atomic stale-writer fencing, terminal-state checks, and stable command IDs
• add SUBMITTING so interrupted external submission is retried with the same idempotency key
• make command submission idempotent by ID while allowing retries to repair the state-write/queue-offer gap

Fixes #96.

Design and compatibility

This intentionally changes the WorkQueue SPI by separating new delivery (receiveNew) from expired PEL reclaim (reclaim). concurrency() now limits active handler invocations rather than admitted/live task lifecycles.

lib-data-workqueue guarantees at-least-once delivery. It does not claim exactly-once side effects. lib-cmd-queue-redis adds the state/ownership layer needed for near-exactly-once processing under normal operation. Command handlers must use command.id() as the idempotency key for external side effects because no local protocol can make an uncooperative remote API exactly once across process failure.

SUBMITTING is a new serialized command status and therefore needs consideration during mixed-version rollout.

Adversarial review

The review specifically exercised:

• more non-terminal tasks than the concurrency limit, verifying intake does not freeze
• multi-threaded handler execution while preserving the configured invocation bound
• handler runtime longer than Stream visibility timeout and max-processing-time, verifying no overlapping call
• two application contexts sharing real Redis, duplicate command submission, and a handler longer than claim timeout
• dead-owner lease expiry and takeover
• stale owner renewal and state writes
• state-created/queue-offer repair through idempotent resubmission
• executor rejection, close while active, ack failure, cancellation, and duplicate delivery paths

Two races found during review were corrected:

1. Redis lease renewal now checks the current Stream PEL owner before touching the entry, so a stale worker cannot claim it back from a peer.
2. Command ownership verification and state persistence are now one atomic state-provider operation, so a lease expiring between a read and write cannot let a stale attempt overwrite the new owner.

Validation

./gradlew :lib-data-store-state-redis:check :lib-data-workqueue:check :lib-data-workqueue-redis:check :lib-cmd-queue-redis:check
BUILD SUCCESSFUL

The suite includes unit tests and Testcontainers-backed Redis end-to-end validation.