Skip to content

Commit 1e42819

Browse files
fix(tables): jitter rate-limit retry backoff to avoid thundering herd
Passing the bucket's shared resetAt as retryAfterMs made backoffWithJitter return a fixed clamped value (no jitter, attempt ignored), so all concurrent cells retried in lockstep. Pass null to get jittered exponential backoff.
1 parent b00dbfa commit 1e42819

1 file changed

Lines changed: 6 additions & 2 deletions

File tree

apps/sim/background/workflow-column-execution.ts

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -487,8 +487,12 @@ async function runWorkflowAndWriteTerminal(
487487
})
488488
return 'error'
489489
}
490-
const retryAfterMs = rl.retryAfterMs ?? Math.max(0, rl.resetAt.getTime() - Date.now())
491-
const waitMs = backoffWithJitter(attempt, retryAfterMs)
490+
// Exponential backoff WITH jitter — pass null, not the bucket's
491+
// resetAt. That reset time is shared across all waiters, and
492+
// backoffWithJitter clamps a non-null hint to a fixed value with no
493+
// jitter, so honoring it would wake all ~20 concurrent cells in
494+
// lockstep and stampede the bucket. Jittered backoff spreads retries.
495+
const waitMs = backoffWithJitter(attempt, null)
492496
logger.info(
493497
`Rate limited — waiting ${Math.round(waitMs)}ms before retry ${attempt + 1} (table=${tableId} row=${rowId} group=${groupId})`
494498
)

0 commit comments

Comments
 (0)