Skip to content

feat(queue): per-partition isolation and fix subscriber shutdown race conditions#108

Merged
behinddwalls merged 1 commit into
mainfrom
preetam/queue
Mar 3, 2026
Merged

feat(queue): per-partition isolation and fix subscriber shutdown race conditions#108
behinddwalls merged 1 commit into
mainfrom
preetam/queue

Conversation

@behinddwalls
Copy link
Copy Markdown
Collaborator

Summary

  • Per-partition isolation in consumer: deliveries are dispatched to per-partition goroutines so a slow message on one partition does not block others. Per-partition ordering is preserved.
  • Per-partition workers in MySQL subscriber: each leased partition gets its own polling goroutine, managed by a supervisor that reconciles workers based on current leases.
  • Fix shutdown race conditions: separate workerWg tracks partition workers independently from the supervisor. workerWg.Wait() blocks before closing deliveryCh to prevent sends on a closed channel. Workers are removed from the map immediately on stop so reconciliation can start replacements without zombie entries.

Test plan

  • TestConsumer_PerPartitionProcessing — verifies a slow partition A does not block partition B
  • TestConsumer_PartitionOrdering — verifies messages within a single partition are processed in order
  • TestConsumer_PartitionWorkerCleanup — verifies all partition goroutines exit cleanly on Stop()
  • TestSubscriber_ReconcilePartitionWorkers — verifies workers are started/stopped based on lease changes
  • TestSubscriber_PartitionWorkerPollAndDeliver — verifies a partition worker delivers messages
  • TestSubscriber_StopAllWorkers — verifies all workers stop gracefully
  • All existing tests pass via bazel test

@behinddwalls behinddwalls requested review from a team and sbalabanov as code owners March 2, 2026 16:35
@behinddwalls behinddwalls enabled auto-merge March 2, 2026 16:39
Comment thread core/consumer/consumer.go Outdated
Comment thread core/consumer/consumer_test.go Outdated
Comment thread core/consumer/consumer_test.go
Comment thread core/consumer/README.md Outdated
Comment thread extension/queue/mysql/subscriber_test.go Outdated
Comment thread extension/queue/mysql/subscriber_test.go Outdated
Comment thread extension/queue/README.md Outdated
@behinddwalls behinddwalls added this pull request to the merge queue Mar 3, 2026
Merged via the queue into main with commit 348a416 Mar 3, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants