Work-queue concurrency permit is held for the whole command lifetime — caps concurrently-tracked commands and silently freezes new intake

[pditommaso]

Summary

Since #84, the concurrency() semaphore in the work-queue dispatch layer (now lib-data-workqueue after #86) does not bound concurrent handler invocations — it bounds concurrent live commands. A permit is acquired at message pickup and held across re-polls, including the idle pollInterval waits, until the handler returns a terminal result. For long-lived commands (e.g. sched's TaskSubmitCommand, which stays non-terminal for the whole task runtime), this makes concurrency × replicas a hard ceiling on concurrently-tracked tasks — and once reached, new intake freezes completely and silently: queued commands never execute, no log line, no metric, no back-signal.

Observed (sched-dev, 2026-07-22)

A 1200-task fan-out of long-running tasks (sched command-queue.concurrency: 100, 2 replicas):

• Exactly 200 tasks launched, then intake froze: the remaining ~1000 TaskSubmitCommands never produced a single Submitting task log — they were never dispatched, not deferred.
• No quota/capacity errors AWS-side; no signal sched-side.
• When long tasks started completing, launches resumed strictly one-in-one-out (26 completions → 25 launches per 5-min window) — the signature of a held semaphore recycling.

Code anchors (lib-data-workqueue/.../AbstractWorkQueue.java)

• slots declaration (~:163): "a permit is acquired when a lease is picked up and held for the whole lease lifetime (across re-polls), released on terminal ack / eviction / release" — the behavior is documented as intent.
• run() (~:483): terminal → acknowledge() → releaseLease() → slots.release(); non-terminal → scheduleRepoll() — no release.
• scheduleRepoll() (~:542): resubmits the invocation via the scheduler without touching the semaphore — re-polls have reserved capacity by construction.
• The stated rationale (dispatcher comment ~:396): re-polls of admitted commands must never be starved by new intake.

Design history — what #84 traded away

Pre-#84 (lib-data-stream-redis 1.5.0): one listener thread, handler inline and serial, non-terminal messages left in the PEL and re-delivered cross-pod via XAUTOCLAIM after claim-timeout (~60s). That model had three real defects that #84 correctly fixed (head-of-line blocking → sched#600; duplicate execution on reclaim; ~60s status-detection lag → sched#303). But it also had two implicit invariants that did not survive, and neither was an explicit design decision:

1. Unbounded admission — every queued command eventually ran; an admission freeze was impossible. Post-Async, non-blocking consumer processing with heartbeat lease (lib-data-stream-redis + lib-cmd-queue-redis) #84 admission is bounded by live-command count.
2. Self-limited API throughput — the serial inline handler was an accidental rate limiter (~1 in-flight cloud call per pod). Post-Async, non-blocking consumer processing with heartbeat lease (lib-data-stream-redis + lib-cmd-queue-redis) #84, in-flight/pollInterval scales with concurrency (e.g. 200 slots / 5s = 40 calls/s), which is what made the ECS control-plane storms possible: the timeline Async, non-blocking consumer processing with heartbeat lease (lib-data-stream-redis + lib-cmd-queue-redis) #84 (Jul 13) → sched#756/#758 storms (Jul ~17) → sched#767 livelock (Jul 21) is a direct consequence chain. sched#758/#769/#771 have been progressively re-building, deliberately, the rate-limiting the old architecture had by accident.

Consistency check from the sched#767 prod incident: the livelock's flat error rate (~1.8 timeouts/s) equals concurrency-slots ÷ apiCallTimeout (200 ÷ 90s) — the slot pool spinning at full occupancy with every call burning its whole budget. The queue ceiling also means the 556-task prod run never had more than 200 tasks in flight; the 370s task failures were the only slot recycler.

Options

	Admission	Re-poll cadence guarantee	API rate bound
A. Status quo, raise the knob (safe once sched#771's batched polling decouples API cost from slot count)	bounded (higher)	✓ pollInterval	✓ explicit
B. Permit per invocation (release before scheduleRepoll, re-acquire on re-poll)	unbounded	✗ re-polls compete with intake under backlog — the starvation the current design guards against	✗ lost again
C. Two lanes: an intake budget for new commands + a separate (or uncapped) re-poll budget	unbounded, paced	✓ bounded per lane	✓ explicit per lane

Recommendation: C — admission can never freeze, re-polls can never starve, and both rates are independently tunable; A as the pragmatic interim for sched once sched#771 merges. B just re-trades the same invariants back.

Whatever the choice, the freeze must stop being silent: at minimum a gauge for held permits + queued-but-never-started commands, and a WARN when intake is starved for more than N poll cycles.

Related

• Async, non-blocking consumer processing with heartbeat lease (lib-data-stream-redis + lib-cmd-queue-redis) #84 (introduced the model), Add lib-data-workqueue (+ -redis) superseding lib-data-stream-redis (deprecated); migrate cmd-queue #86 (workqueue migration), [release] Retry command on handler exception instead of terminal-failing it #87/[release] Track command processing errors on CommandState (errorsCount, error, modifiedAt) #89 (error handling follow-ups)
• seqeralabs/sched#767 (incident + dev replication evidence), seqeralabs/sched#758 (concurrency 500→100), seqeralabs/sched#771 (batched DescribeTasks — prerequisite for option A)