Commit 4fa7e74
feat(tools): queue hosted-key tool calls instead of failing with 429 (#4416)
* Add queueing for hosted keys
* feat(rate-limiter): FIFO queue for hosted-key per-workspace fairness
Replace the per-call distributed lock with a Redis-backed FIFO queue so
callers within a workspace get strict ordering instead of racing the
bucket. Adds heartbeat-based crash recovery and dead-head reaping in a
single Lua script. Bumps Exa search hosted RPM from 5 to 60.
* fix(rate-limiter): bound hosted-key queue wait to execution budget; fix heartbeat + telemetry
Tie the per-workspace hosted-key queue wait to the surrounding execution
budget instead of a flat 5-minute cap. acquireKey now accepts the execution
AbortSignal (threaded from ExecutionContext): when present, the wait is
bounded by the run's actual plan timeout / cancellation, with the enterprise
async ceiling as a backstop; when absent it falls back to MAX_QUEUE_WAIT_MS.
This lets long-running async (Trigger.dev) runs use their full budget while
no longer letting a single queued call burn a short sync run's entire budget.
Also addresses Greptile review:
- P1: share one lastHeartbeatAt across all wait phases and cap every sleep to
HEARTBEAT_REFRESH_INTERVAL_MS so a long low-RPM retryAfterMs can no longer
let the head's heartbeat lapse mid-wait and break FIFO ordering.
- P2: derive hostedKeyQueueWaited telemetry reason from the actual bottleneck
(queue_position / dimension / actor_requests) instead of hardcoding it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(rate-limiter): make hosted-key queue waits abort-interruptible
Replace the plain capped sleeps in the queue-head and bucket-capacity wait
loops with an interruptibleSleep that resolves early when the execution
AbortSignal fires (timeout or cancellation), cleaning up its own timer and
listener. Previously a cancelled/timed-out run could overshoot by up to the
heartbeat cap (~10s) before the loop re-checked its budget; now it wakes
within a tick. The cap remains for heartbeat renewal.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>1 parent 7218185 commit 4fa7e74
6 files changed
Lines changed: 1293 additions & 48 deletions
File tree
- apps/sim
- lib/core
- rate-limiter/hosted-key
- tools
0 commit comments