The Dual Write Problem

Modern event-driven systems often need to:

1. Insert an event row into an SQL database table
2. Push an event handler job onto a Redis queue

These two steps usually happen one after the other—but critically, they cannot be wrapped in a single transaction as they involve different systems.

---

A Common Failure Mode

Example

# 1. Write event row into the database
event = Event.create!(...)

# 💀 Process crashes here

# 2. Queue a handler for out of band event processing
EventCreatedJob.perform_async(event.id)

---

What Can Go Wrong

If the process crashes or the network blips between the two steps:

• The event is committed to the database
• The job is never enqueued
• Downstream systems never hear about the change

Your application now believes something happened—but the rest of the system disagrees.

---

Real-World Consequences

• Emails aren’t sent
• Webhooks don’t fire
• Integrations fall out of sync
• Data becomes stale or incorrect
• Bugs appear that are hard to reproduce and harder to debug

This class of bug is known as the dual write problem.

---

The Solution: A Transactional Outbox

Instead of publishing messages directly to a queue, you record the intent to publish inside the same database transaction as the event row being inserted.

This is the foundation of how Outboxer works.

Atomic Write

ActiveRecord::Base.transaction do
  invoice = Invoice.find_by!(id: id)
  event = InvoiceUpdatedEvent.create!(eventable: invoice)

  Outboxer::Message.queue(messageable: event)
end

Both records succeed or fail together—no partial state.

---

How Outboxer Works

1. Begin a database transaction
2. Persist your domain change (e.g. Event)
3. Persist an outbox message referencing that change
4. Commit the transaction
5. A separate publisher process polls the outbox table
6. Messages are published to Redis / Kafka / SQS / etc.
7. Successfully published messages are marked published or cleaned up

If the app crashes at any point, nothing is lost.

---

Why This Matters

Outboxer gives you:

• Atomicity – no more partial writes
• Durability – messages survive crashes and deploys
• Reliability – guaranteed eventual delivery
• Observability – inspect, retry, or replay messages
• Scalability – works across services and infrastructures

This pattern is used in high-scale systems because it eliminates an entire class of production bugs.

---

Further Reading

• Confluent – The Dual Write Problem
  https://www.confluent.io/blog/dual-write-problem/

• AWS Prescriptive Guidance – Transactional Outbox Pattern
  https://docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/transactional-outbox.html

---

Outboxer exists to make this pattern simple, explicit, and production-ready.