Skip to content

Non-atomic rate-limit guard in ReflectorService causes thundering-herd: concurrent requests bypass MIN_REQUEST_INTERVAL and exhaust CoinGecko quota #28

Description

@Uchechukwu-Ekezie

Summary

ReflectorService.getFreshPrices() performs a read-check-write on this.lastRequestTime without any mutex or atomic operation. Under concurrent price requests (e.g. multiple portfolios triggering rebalance checks simultaneously via BullMQ workers), all callers can read a stale lastRequestTime, pass the guard, and simultaneously fire HTTP requests to CoinGecko:

// backend/src/services/reflector.ts
const now = Date.now()
if (now - this.lastRequestTime < this.MIN_REQUEST_INTERVAL) {  // ← all concurrent callers read old value
    return {}
}
this.lastRequestTime = now  // ← all callers overwrite with ~same timestamp, each fires a request

Impact

  • CoinGecko 429 rate-limit hammering: N concurrent BullMQ workers each bypass the interval guard and fire simultaneous requests, exhausting the free-tier quota (10–30 req/min) in seconds.
  • Cascading fallback to stale/mock prices: After 429s, the service falls back to getFallbackPrices() which adds ±1% random noise to hardcoded base prices — rebalance thresholds may trigger or suppress incorrectly.
  • Silent price divergence: getCurrentPrices() returns merged results from concurrent workers pulling CoinGecko at the same moment with different response ordering, potentially returning different prices for the same asset within the same rebalance cycle.

Steps to Reproduce

  1. Configure Redis + BullMQ with ≥ 4 concurrent workers.
  2. Create 10 portfolios and trigger forceCheck().
  3. Watch logs — multiple [DEBUG] Full URL: ...coingecko...] lines appear within the same 90-second window.
  4. CoinGecko returns 429; service falls back to fallback prices silently.

Root Cause

JavaScript single-threaded event loop does NOT protect against this: await fetch() yields execution, allowing multiple coroutines to pass the guard before any of them updates lastRequestTime.

Suggested Fix

Use an in-flight promise gate (singleton request deduplication):

private inflightPriceRequest: Promise<PricesMap> | null = null

private async getFreshPrices(assets: string[]): Promise<PricesMap> {
    if (this.inflightPriceRequest) return this.inflightPriceRequest
    this.inflightPriceRequest = this._doFetch(assets).finally(() => {
        this.inflightPriceRequest = null
    })
    return this.inflightPriceRequest
}

This collapses all concurrent callers onto one in-flight HTTP request, honouring the rate limit regardless of concurrency level.

References

  • backend/src/services/reflector.tsgetFreshPrices(), getCurrentPrices()
  • backend/src/services/autoRebalancer.ts — BullMQ worker concurrency

Severity: High — CoinGecko quota exhaustion causes all portfolios to rebalance on fallback/stale prices

Metadata

Metadata

Assignees

Labels

GrantFox OSSIssue tracked in GrantFox OSSMaybe RewardedIssue may be eligible for a GrantFox rewardOfficial CampaignCampaign: Official CampaignbugSomething isn't working

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions