Skip to content

Per-key execution control: concurrency limits and fairness within a queue #340

Description

@hardbyte

Problem

Awa controls execution at queue granularity — max_workers, min_workers/weight (ADR-011), and rate_limit (ADR-010) — and ordering_key (ADR-025) pins related jobs to one enqueue shard for FIFO ordering. But there is no execution-time control keyed on job data:

  1. Per-key concurrency limits — "at most N in-flight jobs per customer/tenant/resource". Today the only approximations are one-queue-per-key (unbounded queue cardinality) or unique_key with running-state uniqueness (caps at 1 and conflates dedup with concurrency).
  2. Fairness across keys within a queue — one hot tenant enqueuing 100k jobs starves every other tenant's jobs in the same queue until aging kicks in. Priority aging (ADR-005) bounds total starvation time but is not per-key fair scheduling.

This is the main multi-tenant gap in the dispatch surface; the comparable art is Oban Pro's partitioned concurrency/rate limits and BullMQ groups.

Scope boundary

This is a claim/dispatch-plane feature, not workflow semantics — no dependencies or fan-in (ADR-021 explicitly leaves those to coordinator jobs or external engines). The key is data the producer stamps on the job; the engine only gates and interleaves claims.

Design questions

  • Where does the key live? ordering_key already exists on the insert path and is the natural candidate; per-key concurrency on top of per-shard FIFO needs claim-time awareness of in-flight counts per key, which the lane-cursor claim path (Move queue storage cursors off hot rows #321) does not currently track.
  • Limit enforcement point: claim-time (skip keys at limit — interacts with cursor monotonicity and tombstones) vs. post-claim gating in the dispatcher (simpler, but burns claim throughput re-queueing gated jobs).
  • Fairness: round-robin across keys with ready work is a different mechanism than a concurrency cap; decide whether both are in scope or fairness ships first as interleaved claim ordering.
  • Cardinality: per-key state must not become a new hot mutable table under pinned MVCC horizons (Validate MVCC / dead-tuple behaviour under sustained load #169) — any per-key counter family needs the same delta-ledger discipline as ADR-026.

Refs: ADR-005, ADR-010, ADR-011, ADR-025, #169.

Metadata

Metadata

Assignees

No one assigned

    Labels

    featureNew functionality

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions