Skip to content

BullMQ portfolio-check jobs lack deduplication: forceCheck() + scheduled repeatable job create overlapping concurrent rebalances that violate the on-chain cooldown contract #31

Description

@Uchechukwu-Ekezie

Summary

AutoRebalancerService adds jobs to the portfolio-check queue without a deduplication key. A manual forceCheck() and the 30-minute repeatable scheduler can coexist in the queue simultaneously, causing concurrent workers to pick up two jobs for the same portfolio and attempt overlapping rebalances:

// backend/src/services/autoRebalancer.ts — forceCheck()
await queue.add('force-portfolio-check', { triggeredBy: 'manual' }, { priority: 1 })

// startup in start()
await queue.add('startup-portfolio-check', { triggeredBy: 'startup' }, { priority: 1 })

Neither job carries a BullMQ jobId (deduplication key). BullMQ's default behaviour is to enqueue all jobs unconditionally, so N calls to forceCheck() produce N independent jobs.

Impact

1. Concurrent Stellar transaction submission race

Two workers processing the same portfolio simultaneously both call StellarService to build and submit a rebalance transaction. Stellar's sequence number mechanism means one will succeed and one will return tx_bad_seq — but both workers may already have partially updated off-chain state (rebalance history, WebSocket push), leaving the backend and contract in an inconsistent state.

2. On-chain cooldown bypass through job timing

Worker A processes portfolio 42 at T=0, succeeds, sets last_rebalance = T. Worker B, delayed in the queue, processes the same portfolio at T=60 seconds. The contract's execute_rebalance checks current_time < last_rebalance + 3600 — this rejects Worker B correctly on-chain, but Worker B has already consumed circuit-breaker budget, written a failed rebalance history entry, and potentially sent a webhook notification for a rebalance that never happened.

3. MAX_AUTO_REBALANCES_PER_DAY counter drift

The service declares MAX_AUTO_REBALANCES_PER_DAY = 3 but enforces it nowhere in the queue layer. Duplicate jobs can trigger more than 3 rebalances per portfolio per day, violating the documented safety limit.

Steps to Reproduce

  1. Start with Redis + ≥ 2 BullMQ workers.
  2. Call POST /api/rebalancer/force-check three times in rapid succession.
  3. Observe: three force-portfolio-check jobs in the queue, all processed concurrently for the same portfolio set.
  4. Check rebalance history — multiple entries for the same portfolio within seconds of each other.

Root Cause

BullMQ supports job-level deduplication via a stable jobId. Without it, every queue.add() call creates a unique job regardless of payload similarity.

Suggested Fix

Job-level deduplication

// forceCheck()
await queue.add(
  'force-portfolio-check',
  { triggeredBy: 'manual' },
  {
    priority: 1,
    jobId: 'force-check-singleton',  // BullMQ deduplicates by jobId
  }
)

Per-portfolio locks in the worker

// In the worker processor
const lockKey = `rebalance-lock:${portfolioId}`
const acquired = await redis.set(lockKey, '1', 'NX', 'EX', 60)
if (!acquired) {
  logger.warn(`[WORKER] Skipping ${portfolioId} — rebalance already in progress`)
  return
}
// ... process rebalance
await redis.del(lockKey)

Enforce MAX_AUTO_REBALANCES_PER_DAY in the worker

Check rebalance history count before processing; skip and log if limit is reached.

References

  • backend/src/services/autoRebalancer.tsstart(), forceCheck(), MAX_AUTO_REBALANCES_PER_DAY
  • contracts/src/lib.rsexecute_rebalance cooldown check (current_time < last_rebalance + 3600)
  • backend/src/api/routes.tsPOST /rebalancer/force-check endpoint

Severity: High — concurrent workers submit duplicate on-chain transactions and corrupt off-chain rebalance history

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions