docs(queue/sql): add comprehensive RFC for SQL queue design

## Why?
Document the design decisions, alternatives, and rationale behind the SQL queue implementation.

## What?
- RFC covering requirements, alternatives (including Watermill), and trade-offs
- Explanation of partition leasing and visibility timeout mechanisms
- Performance characteristics and observability
- Concise format without verbose SQL queries or implementation details

## Test Plan
- RFC reviewed for technical accuracy
- All described mechanisms match implementation

Author: behinddwalls

docs/designs/sql-queue-rfc.md

@@ -0,0 +1,246 @@
+# RFC: SQL-Based Distributed Queue
+
+**Status:** Implemented
+**Created:** 2026-02-16
+
+## Summary
+
+MySQL-based distributed message queue with partition leasing, visibility timeout, and at-least-once delivery. Workers coordinate via database-native primitives without external systems.
+
+## Background
+
+### Motivation
+
+SubmitQueue needs a reliable message queue for coordinating asynchronous workflows:
+- **Orchestrator** publishes merge jobs to workers
+- **Speculator** publishes speculative build requests
+- **Workers** need distributed coordination without duplicate processing
+- **Crash recovery** must preserve exactly where processing stopped
+
+### Existing Solutions
+
+We evaluated several approaches:
+
+1. **External Message Brokers** (Kafka, RabbitMQ)
+   - ❌ Additional operational overhead and infrastructure
+   - ❌ Network hops increase latency
+   - ✅ Battle-tested and highly scalable
+
+2. **Watermill Library** (github.com/ThreeDotsLabs/watermill)
+   - ✅ Database-backed queue with mature abstractions
+   - ✅ Built-in middleware (retry, poison queue, metrics)
+   - ❌ Generic interface hides database-specific optimizations
+   - ❌ Additional dependency and learning curve
+   - ❌ Less control over exact SQL queries and behavior
+
+3. **Database-Backed Queue** (Custom implementation)
+   - ✅ Reuses existing MySQL infrastructure
+   - ✅ Full control over queries and behavior
+   - ✅ No additional services or dependencies
+   - ❌ More code to maintain
+
+### Decision
+
+We chose **custom database-backed queue** because:
+- Full control over SQL queries for optimal performance
+- No additional libraries - direct use of database/sql
+- Simpler to understand and debug (no abstraction layers)
+- Can optimize for our specific use case (partition ordering, visibility timeout)
+- Watermill adds valuable abstractions but we need fine-grained control
+
+## Requirements
+
+### Functional Requirements
+
+1. **Publish/Subscribe** - Standard pub/sub with topics
+2. **Partitioning** - Messages with same key processed in order by single worker
+3. **At-Least-Once Delivery** - Guaranteed delivery, duplicates possible
+4. **Crash Recovery** - Workers resume from last committed offset
+5. **Distributed Workers** - Multiple workers coordinate without duplicate processing
+6. **Dead Letter Queue** - Failed messages isolated after max retries
+7. **Visibility Timeout** - Messages invisible during processing, visible if worker crashes
+
+### Non-Functional Requirements
+
+1. **Operational Simplicity** - No additional infrastructure
+2. **Observability** - Metrics and logging for debugging
+3. **Testability** - In-memory testing without external MySQL
+4. **Performance** - Sub-second latency for typical workloads
+5. **Scalability** - Handle hundreds of workers, thousands of partitions
+
+### Non-Goals
+
+1. **Exactly-Once Delivery** - Application must handle duplicates
+2. **Kafka-Scale Throughput** - Not optimizing for millions of messages/sec
+3. **Cross-Datacenter Replication** - Single MySQL instance only
+4. **Message Ordering Across Partitions** - Only within partition
+5. **Real-Time Streaming** - Polling introduces configurable latency
+
+## Design Overview
+
+### Core Concepts
+
+**Partition Leasing:** Workers coordinate using database-native leases. Each partition leased by exactly one worker. Stale leases automatically stolen on crash.
+
+**Visibility Timeout:** Messages invisible during processing. Auto-retry on crash when timeout expires.
+
+**Persistent Retry Tracking:** `retry_count` incremented atomically on fetch, survives crashes, triggers DLQ.
+
+**Offset Tracking:** Per-partition offsets enable crash recovery from last acked message.
+
+## Database Schema
+
+### Tables
+
+**queue_messages** - All messages across topics
+- Composite PK: `(topic, partition_key, offset)`
+- `offset` AUTO_INCREMENT ensures ordering within partition
+- `invisible_until` for visibility timeout
+- `retry_count` for persistent retry tracking
+
+**queue_partition_leases** - Worker coordination
+- PK: `(consumer_group, topic, partition_key)`
+- `leased_by` identifies owner
+- `lease_renewed_at` enables stale lease detection
+
+**queue_offsets** - Consumption progress
+- PK: `(consumer_group, topic, partition_key)`
+- `offset_acked` tracks last processed message
+
+**queue_dlq** - Failed messages
+- Same structure as messages table
+- Stores messages exceeding max retries
+
+See full schema: `schema/queue/mysql/schema.sql`
+
+## Message Flow
+
+**1. Publish** - Insert messages with AUTO_INCREMENT offset
+
+**2. Lease Acquisition** - `INSERT ... ON DUPLICATE KEY UPDATE` with stale lease detection
+
+**3. Fetch** - Atomic UPDATE sets `invisible_until` and increments `retry_count`
+
+**4. Ack** - Transaction: DELETE message + UPDATE offset_acked
+
+**5. Nack** - UPDATE `invisible_until` for retry after delay
+
+**6. DLQ** - If `retry_count >= MaxAttempts`: DELETE from messages + INSERT into dlq
+
+## Crash Recovery
+
+**Scenario:** Worker crashes while processing message
+
+**What happens:**
+1. Message has `invisible_until = crash_time + VisibilityTimeout`
+2. After timeout expires, message becomes visible
+3. Another worker detects stale lease and steals partition
+4. Message redelivered (at-least-once guarantee)
+5. `retry_count` incremented prevents infinite retries
+
+**Key properties:** Automatic failover, no data loss, configurable retry delay
+
+## Distributed Processing
+
+**Same Consumer Group:** Workers distribute partitions via leasing. Each partition processed by one worker.
+
+**Different Consumer Groups:** Independent consumption with separate offsets. Same messages delivered to all groups.
+
+## Alternatives Considered
+
+### Watermill Library
+
+**Evaluation:** We prototyped a full implementation using `github.com/ThreeDotsLabs/watermill-sql`
+
+**Pros:**
+- Mature abstractions for pub/sub
+- Built-in middleware (poison queue, retry, metrics)
+- Multi-backend support (MySQL, PostgreSQL, Kafka)
+- Well-tested and documented
+
+**Cons:**
+- Generic interface hides database-specific optimizations
+- Less control over exact SQL queries (e.g., can't optimize visibility timeout logic)
+- Middleware adds complexity for simple use case
+- Additional dependency to maintain and version
+- Learning curve for team (new library semantics)
+
+**Decision:** Custom implementation gives us full control. Watermill is valuable for complex multi-backend scenarios but overkill for our focused MySQL use case.
+
+### PostgreSQL SKIP LOCKED
+
+**Pros:** `SELECT ... FOR UPDATE SKIP LOCKED` provides truly atomic fetch
+
+**Cons:** SubmitQueue uses MySQL, not PostgreSQL. Migration not justified.
+
+### Redis Streams
+
+**Pros:** Lower latency, built-in consumer groups
+
+**Cons:** Additional infrastructure. No transactional consistency with MySQL data.
+
+### Single-Table Per Topic
+
+**Pros:** Better isolation, easier to drop topics
+
+**Cons:** Schema migration per topic. Not friendly for dynamic topic creation.
+
+**Decision:** Single-table design for operational simplicity.
+
+## Trade-offs
+
+**Polling vs Push**
+- ✅ Simpler (no connection management), natural backpressure
+- ❌ Higher latency (configurable via PollInterval)
+- Mitigation: Tune PollInterval (default 100ms, tests 20ms)
+
+**Visibility Timeout vs Heartbeat**
+- ✅ No heartbeat protocol, automatic retry
+- ❌ Full timeout delay even on immediate crash
+- Mitigation: ExtendVisibilityTimeout() for long tasks
+
+**Database Leasing vs External Coordinator**
+- ✅ No ZooKeeper/etcd, transactional consistency
+- ❌ Lease renewal overhead
+- Mitigation: Tunable renewal interval (default 10s)
+
+**At-Least-Once vs Exactly-Once**
+- ✅ Simpler, better performance
+- ❌ Applications must handle duplicates
+- Mitigation: Idempotency keys (e.g., merge request ID)
+
+
+## Observability
+
+**Metrics (via tally):**
+- Publisher: `messages_published`, `publish_errors`
+- Subscriber: `messages_acked`, `messages_nacked`, `messages_moved_to_dlq`, `message_age`, `leases_acquired`
+- Stores: `insert.latency`, `fetch.latency`, `ack_message.latency`, `renew_lease.latency`
+
+**Logging (via zap):**
+- Debug: Message fetch, lease operations
+- Info: Publish success, DLQ moves, partition acquisition
+- Error: Database errors, unrecoverable failures
+- Structured fields: `topic`, `partition_key`, `message_id`, `offset`, `retry_count`
+
+## Performance
+
+**Throughput:** ~1k-5k msg/sec publish, ~500-2k msg/sec consume (single MySQL)
+**Latency:** Best case = PollInterval (100ms), Retry after crash = VisibilityTimeout (60s)
+**Bottlenecks:** MySQL write throughput, lease renewal overhead, polling overhead
+
+## Configuration
+
+```go
+type Config struct {
+    ConsumerGroup         string        // e.g., "orchestrator"
+    WorkerID              string        // e.g., "worker-1"
+    PollInterval          time.Duration // Default: 100ms
+    BatchSize             int           // Default: 10
+    VisibilityTimeout     time.Duration // Default: 60s
+    LeaseDuration         time.Duration // Default: 30s
+    LeaseRenewalInterval  time.Duration // Default: 10s
+    Retry.MaxAttempts     int           // Default: 3
+    DLQ.Enabled           bool          // Default: true
+}
+```