Skip to content

refactor(queue/mysql): immutable log, delivery state, heartbeat store, fair leasing#126

Merged
behinddwalls merged 0 commit into
preetam/docsfrom
preetam/queue
Mar 9, 2026
Merged

refactor(queue/mysql): immutable log, delivery state, heartbeat store, fair leasing#126
behinddwalls merged 0 commit into
preetam/docsfrom
preetam/queue

Conversation

@behinddwalls
Copy link
Copy Markdown
Collaborator

@behinddwalls behinddwalls commented Mar 7, 2026

Summary

  • Add delivery state store for tracking per-consumer message state separately from the immutable log
  • Add subscriber heartbeat store for liveness detection and fair partition leasing
  • Refactor message store to append-only immutable log with contiguous watermark tracking
  • Refactor partition lease store with heartbeat-based fair leasing
  • Add batch query support for offset and delivery state operations
  • Add GC throttling and dead code removal
  • Fix make fmt to skip read-only pkg/mod directory

Test Plan

Issues

@behinddwalls behinddwalls requested review from a team and sbalabanov as code owners March 7, 2026 09:00
@behinddwalls behinddwalls force-pushed the preetam/queue branch 2 times, most recently from 6260b28 to 8cd279d Compare March 7, 2026 09:43
@sbalabanov
Copy link
Copy Markdown
Contributor

I think overall error processing and expectations on reliability of the heartbeats need to be better articulated. Both contracting the interface with comments and readme.

Comment thread extension/queue/mysql/partition_lease_store.go Outdated
Comment thread extension/queue/mysql/subscriber.go Outdated
Comment thread extension/queue/mysql/subscriber.go Outdated
Comment thread extension/queue/mysql/subscriber.go Outdated
Comment thread extension/queue/mysql/subscriber.go Outdated
@behinddwalls behinddwalls force-pushed the preetam/queue branch 2 times, most recently from 3d99a7c to c7cd0c1 Compare March 8, 2026 04:19
@behinddwalls
Copy link
Copy Markdown
Collaborator Author

I think overall error processing and expectations on reliability of the heartbeats need to be better articulated. Both contracting the interface with comments and readme.

I need to update those, I will do them in a separate PR.

@behinddwalls behinddwalls requested a review from sbalabanov March 8, 2026 04:24
@behinddwalls behinddwalls force-pushed the preetam/queue branch 2 times, most recently from ca7dd1a to d0c68c7 Compare March 8, 2026 23:50
@behinddwalls behinddwalls changed the title feat(queue/mysql): add fair partition leasing with subscriber heartbeat table refactor(queue/mysql): immutable log, delivery state, heartbeat store, fair leasing Mar 8, 2026
@behinddwalls behinddwalls force-pushed the preetam/queue branch 2 times, most recently from 1ee89f9 to 2debf5d Compare March 9, 2026 02:39
@behinddwalls behinddwalls changed the base branch from main to preetam/docs March 9, 2026 02:39
@behinddwalls behinddwalls merged commit 931ad78 into preetam/docs Mar 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants