test(queue/sql): add comprehensive e2e integration tests#35
Merged
Conversation
This was referenced Feb 20, 2026
behinddwalls
force-pushed
the
preetam/sql-queue-e2e
branch
from
abebb38 to
422e3e9
Compare
behinddwalls
added a commit
that referenced
this pull request
Feb 20, 2026
## Summary ## Why? Need to integrate publisher and subscriber with Queue factory interface and provide comprehensive documentation and end-to-end testing. ## What? - Factory creates publisher and subscriber singletons with lifecycle management - Comprehensive README with usage examples and architecture docs ## Test Plan ## Issues ## Stack 1. @ #24 1. #34 1. #35
sbalabanov
approved these changes
Feb 20, 2026
Contributor
sbalabanov
left a comment
sbalabanov
left a comment
There was a problem hiding this comment.
We need to discuss how many e2e test suites we want to maintain. All these containers are notoriously slow to setup and start
| ) | ||
|
|
||
| // TestLogger is a simple test-aware logger that records elapsed time between logs. | ||
| type TestLogger struct { |
Contributor
There was a problem hiding this comment.
mysql.go is probably not the best filename for this now
Collaborator
Author
There was a problem hiding this comment.
Yes..we could probably just do one container for mysql for the whole suite and use it for all cases...
behinddwalls
force-pushed
the
preetam/sql-queue-refactor
branch
from
4cfb4c5 to
fca4ddc
Compare
behinddwalls
force-pushed
the
preetam/sql-queue-e2e
branch
from
422e3e9 to
265677d
Compare
behinddwalls
added a commit
that referenced
this pull request
Feb 21, 2026
## Summary Why? Previously, DLQ messages were stored in a separate queue_dlq table, requiring separate schema management and limiting the ability to consume from DLQ topics. By merging DLQ into the messages table with a topic suffix pattern (e.g., "orders" → "orders_dlq"), we can: - Reuse existing subscriber/publisher logic for DLQ consumption - Simplify schema management (single table) - Track DLQ-specific metadata (failure_count, last_error, etc.) - Enable DLQ reprocessing workflows What? - Add TopicSuffix to DLQConfig (default "_dlq") - Add DLQ columns to queue_messages schema (failed_at, failure_count, last_error, original_topic) as nullable fields - Update MoveToDLQ to insert into queue_messages with DLQ topic name - Expose DLQ metadata through delivery metadata for testing - Remove separate queue_dlq.sql schema - Update FetchByOffset to select and populate DLQ fields - Fix unit test mocks to include all 11 columns - Document retry_count vs failure_count distinction in schema ## Stack 1. @ #34 1. #35
behinddwalls
force-pushed
the
preetam/sql-queue-e2e
branch
from
265677d to
dee854d
Compare
behinddwalls
changed the base branch from
preetam/sql-queue-refactor
to
main
Why? The SQL queue implementation needs thorough integration testing against a real MySQL database to validate end-to-end functionality including: - Message publishing and consumption workflows - Partition management and isolation - Visibility timeout and retry mechanisms - DLQ (Dead Letter Queue) handling - Concurrency and crash recovery scenarios - Consumer group coordination and load balancing Without these tests, we cannot verify that the queue behaves correctly in production-like scenarios with a real database. What? - Add 15 comprehensive integration tests using testcontainers-go: * Basic operations: publish/subscribe, metadata preservation * Partition management: multiple partitions, isolation * Reliability: visibility timeout, retry, nack with delay * Correctness: idempotent publish, message ordering * Concurrency: concurrent publishers, concurrent subscribers * Crash recovery: lease expiration and message recovery * Consumer groups: multiple groups, multiple workers, load balancing * DLQ: max retries, DLQ metadata, failure tracking * Edge cases: late subscriber, empty topic, graceful shutdown - Add integration_tests/queue/README.md with: * Test coverage documentation * Prerequisites (Docker) * Running instructions (Bazel, go test, verbose output) * Test descriptions and what each validates - Refactor integration_tests/testutil for reusability: * Extract MySQL container utilities to testutil/mysql.go * Share MySQL setup across integration test suites * Add schema application helpers - Test infrastructure improvements: * Use testify/suite for setup/teardown lifecycle * Tag tests with "integration" for selective execution * Support for verbose test output (--test_arg=-test.v) * Proper container cleanup in all scenarios All 15 tests passing in ~2 minutes runtime. Co-Authored-By: Preetam Dwivedi <preetam@uber.com>
behinddwalls
force-pushed
the
preetam/sql-queue-e2e
branch
from
dee854d to
223a809
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Why?
The SQL queue implementation needs comprehensive integration testing against real MySQL to verify production-like behavior including message workflows, partition management, retries, DLQ
handling, and consumer group coordination.
What?
testutil/mysql.gofor reusability across test suitesintegration_tests/queue/README.mddocumenting test coverage and running instructionsTest Plan
bazel test //integration_tests/queue:queue_testIssues