Exploring Delayed Message Support in Iggy: Use Cases & Implementation Ideas

[l-7-l]

Iggy is absolutely fantastic, delayed messages is sometimes quite an useful feature. In our e-commerce system, we currently rely on delayed messages for three critical workflows:

1. Order timeout closure (30-minute threshold)

2. Payment settlement initiation (N days post-fulfillment)

3. Scheduled product listing management (launches/takedowns)

While Iggy's performance is excellent, native delayed message support would be a valuable feature. This capability would eliminate the need for additional components, reducing technical debt.

As noted by @spetz, delayed messages serve crucial roles in certain scenarios. They proposed a potential implementation approach:

"Create dedicated topics with background jobs that move messages to destination topics upon schedule expiration. This could be treated as a special topic type – either allowing direct message pulls regardless of timing, or adding a behavioral flag."

Open questions from the proposal:

1. How might this affect consumer access patterns for standard vs. delayed topics?

We welcome architecture insights and alternative implementation approaches from the community.

[l-7-l]

I found an article about introducing qmq delayed messages:
QMQ offers delayed/scheduled messaging for any time within a configurable period (up to two years by default). You can specify the delivery time of a message at any point within this future timeframe.

I placed QMQ last because I consider its design for delayed messages to be the most rational among open-source message queues (MQ). The core design, simply put, is multi-level timing wheel + lazy loading + dedicated disk storage for delayed messages.

QMQ implements delayed/scheduled messages using a two-layer hash wheel:

Disk Layer: Uses hourly intervals as the unit (default; adjustable via configuration). Each interval generates a log file (schedule log). Since QMQ supports delayed messages up to two years (configurable), this results in a maximum of 2 * 366 * 24 = 17,568 files (fewer files are needed if the maximum delay time is shorter).

Memory Layer: When a message's delivery time approaches, the indices for messages within that hour (containing the offset and size within the schedule log) are loaded from disk into a hash wheel in memory. This in-memory hash wheel uses 500ms intervals.

Summary of Key Design Highlights:

1. The Timing Wheel algorithm is ideal for delayed/scheduled messaging. It eliminates the need for sorting delayed messages, offering O(1) time complexity for both insertion and deletion operations.

2. The multi-level timing wheel design enables support for delayed messages with extremely large time spans.

3. Lazy loading ensures only messages nearing their delivery time reside in memory. Messages scheduled further in the future are stored on disk, making the design memory efficient.

4. Storing delayed messages separately in schedule log files prevents them from interfering with the space reclamation of normal messages.

[spetz]

Thank you for the sharing the very interesting design for this feature. As discussed already on the Discord, while having the delayed messages is quite a common thing for the message queues, it's not that "popular" for the message streaming engines. I think that the initial idea, where there could be a dedicated "system" topic storing the delayed messages and then appending them (when the time comes), to the desired topic e.g. based on the message metadata/headers could work. Currently, there's a huge refactoring as we move to the io_uring runtime with thread-per-core & shared nothing design, but it's certainly the feature worth considering in the future :)

[hubcio]

I did some research on this topic. from my understanding it's a send now, deliver later mechanism. e.g. "cancel this order in 30 min unless paid".

it's tricky, because topic is a numbered line read in order. we can't just hide message 5 until later, message 6 would either get stuck behind it or leave a gap in the numbering.

this pairs naturally with #2854 (deferred response), which lets a consumer sleep until a message arrives: since the worker copying into the real topic is just a normal write, a sleeping consumer wakes up on its own. it's not a hard requirement though, without it the consumer just polls the dest topic as usual. the real prerequisite is server-ng having a working poll path, which right now is still a stub.

my idea: we will create a special waiting room topic (assigned 1:1 to existing topic, it can be created lazily), so the flow would look something like:

• producer tags the message with a delivery time
• it lands in a hidden "waiting room" topic, not the real one
• a small background worker watches the clock
• when the time comes, it copies the message into the real topic

now it's just a normal message. consumer reads it like any other, never knew it waited. no gaps, no blocking.

to keep the worker cheap: a few fixed delay buckets (1s, 30s, 1m, 1h...), really just partitions inside that one waiting room, one per level. same delay in a bucket means they come due in order, so the worker just takes them off the front. a proper timing wheel for arbitrary times can come later if anyone needs it.

two things not to forget: the move is at-least-once, a crash between copying to the real topic and marking the waiting-room copy as done can deliver it twice, so consumers should dedup on the message id. note this leans on producers setting a unique id per message, the server doesn't assign one (id defaults to 0), and the same uniqueness is what cancel-by-id below needs. and the waiting room must not expire its own messages by time, a 5-day-delayed message can't get garbage-collected at hour 24, so normal retention is off there. cleanup is by promotion instead: a waiting-room segment fully behind the worker's cursor is all promoted-or-cancelled, so it gets reclaimed, and the log stays bounded by the not-yet-due backlog rather than growing forever.

the worker's progress is nothing fancy: a committed consumer offset on each waiting-room partition (reusing the offset machinery we already have, persisted and replicated). promote a batch, commit the offset, resume from there on restart, and that same offset is what the cleanup above keys off. cancels can't ride that cursor though, a cancel's offset has nothing to do with its target's, so they go in a small side cancel-log the worker reads to rebuild the cancelled-set, dropped once the target is promoted or skipped.

cancelling a waiting message is actually fine here, and that's the nice part of moving instead of hiding. the waiting room is pre-real-topic, the message never entered the consumer's log yet, so dropping it has no offset-hole or ordering problem, the real topic just never sees it. the worker promotes "due AND not cancelled". i can't flip a bit on the stored message (segments are immutable), so cancel is its own little record in a side cancel-log, and the worker keeps a cancelled-set rebuilt from that log on restart. flow becomes: peel the due ones off the front, skip the cancelled ids, append the rest.

what needs to be done for cancel / abort of delayed message:

• a cancel command keyed by message id (so producers must set their own ids), or by waiting-room offset if we make delayed-send return it (today send returns nothing)
• store each cancel in a small side cancel-log (not the message log, a cancel's offset is unrelated to its target's)
• worker keeps a cancelled-set, rebuilt from that cancel-log on restart
• at promote: skip cancelled ids, append the rest
• prune the set once a message is past due (only pending cancels stay)
• accept it's best-effort: a cancel after the fire tick is too late

OR maybe skip all of it and let the consumer re-check on arrival, "still unpaid? if paid, do nothing"?

other append-only-log brokers do delayed delivery too. most use this same move/promote trick (park, then re-append to the real log when due); pulsar is the exception worth knowing about, it withholds in place instead, which it can only do for shared subscriptions that give up strict ordering.

• RocketMQ - internal scheduled topic + timer re-commit; 18 fixed levels (4.x), or arbitrary-time timer wheel (5.x, RIP-43). closest match. [4.x](https://rocketmq.apache.org/docs/4.x/producer/04message3/), [5.x](https://rocketmq.apache.org/docs/featureBehavior/02delaymessage/), [RIP-43]([RIP-43]Support Timing Messages with Arbitrary Time Delay rocketmq#4557)
• QMQ - the design l-7-l pointed at in [discussion 2031](Exploring Delayed Message Support in Iggy: Use Cases & Implementation Ideas #2031): hourly schedule log + in-memory hashed wheel. [docs](https://github.com/qunarcorp/qmq/blob/master/docs/cn/delay.md)
• Pulsar - the contrast: withholds in place via an in-memory DelayedDeliveryTracker, shared/key-shared subs only (no offset ordering). [docs](https://pulsar.apache.org/docs/concepts-messaging/)
• Kafka - no native; KIP-1277 proposal or manual tiered topics. KIP-1277
• NATS JetStream - 2.12 scheduler re-publishes when due, and can cancel a pending schedule. [ADR-51](https://github.com/nats-io/nats-architecture-and-design/blob/main/adr/ADR-51.md)
• Azure Service Bus (not a log) - cleanest cancel-before-fire ref: cancel by sequence number. [docs](https://learn.microsoft.com/en-us/azure/service-bus-messaging/message-sequencing)

@spetz @numinnex what do you think?
also tagging @arunsingh because he's working on deferred replies in #2854