refactor(test): consolidate testing infrastructure with Docker Compose by behinddwalls · Pull Request #41 · uber/submitqueue

behinddwalls · 2026-02-23T02:50:27Z

Summary

Migrate from manual server setup to Docker Compose-based hermetic testing.
Consolidate tests under test/ directory with consistent patterns.

Key changes:

New test structure: test/e2e/, test/integration/{gateway,orchestrator,extensions}/
Container naming: sq-test-{context}-{unique-id} for parallel execution
Shared utilities in test/testutil/ (compose, docker, mysql, schema helpers)
Removed old {service}/integration_test/ directories and speculator service
Documentation: TESTING.md → doc/howto/, STRUCTURE.md → PROJECT_STRUCTURE.md
Makefile: alphabetical targets, new integration test targets, local-start/stop naming
Consolidate CI and create separate jobs for faster run and add bazel cache

Test Plan

Unit Tested
Integration Tested
CI Validation

Issues

sbalabanov

The only blocking comment I have is ditching Testcontainers-Go. Justification is not clear, in fact I do see more manual steps to do it with docker-compose (which testcontainers managed before)

## Summary Wire up the first stage of the queue pipeline: Gateway publishes land requests to the queue, Orchestrator consumes and processes them. **Consumer Infrastructure:** - Add Consumer interface (Register/Start/Stop) for orchestrating multiple controllers - Add consumer.Delivery interface to enforce separation of concerns (type-safe) - Controllers receive consumer.Delivery (no Ack/Nack), Consumer handles ack/nack - Implement subscription lifecycle, automatic ack/nack, metrics, graceful shutdown **Gateway:** - Land controller publishes requests to land_request queue after storage - Queue infrastructure optional (controlled by QUEUE_MYSQL_DSN env var) **Orchestrator:** - Request controller consumes from land_request queue - Wire up consumer with graceful shutdown in main.go **CLAUDE.md:** - Document RPC vs Queue Message controller patterns - Add code style guidelines: use SugaredLogger, use interfaces for contracts All unit and integration tests pass. Backward compatible with existing tests. ## Test Plan ## Issues ## Stack 1. @ #39 1. #41

behinddwalls · 2026-02-24T00:58:40Z

So i made changes with TC-go and here is the side by side comparision,

  Current (Testcontainers-Go) vs PR #41's Docker Compose Approach                                                                                                                            
                                                                                                                                                                                             
  What PR #41 delivered for Docker Compose                                                                                                                                                   
                                                                                                                                                                                             
  PR #41 added the YAML infrastructure (docker-compose.yml, Dockerfiles, CI actions) and documented the compose.go API in CLAUDE.md, but never actually implemented compose.go. The Go test  
  wrapper (NewComposeStack(), ConnectMySQLService(), etc.) was aspirational — it was documented but not written.                                                                             
                                                                                                                                                                                             
  Line counts

  ┌─────────────────────────────┬───────────────────────────────────────┬─────────────────────────────────────┐
  │          Approach           │            Infrastructure             │                Notes                │
  ├─────────────────────────────┼───────────────────────────────────────┼─────────────────────────────────────┤
  │ Testcontainers-Go (current) │ 474 lines across 6 Go files + BUILD   │ Working, tested                     │
  ├─────────────────────────────┼───────────────────────────────────────┼─────────────────────────────────────┤
  │ Docker Compose (PR #41)     │ ~100 lines YAML + ~0 lines Go wrapper │ Incomplete — compose.go never built │
  └─────────────────────────────┴───────────────────────────────────────┴─────────────────────────────────────┘

  Feature comparison

  ┌───────────────────┬────────────────────────────────────────────────────────────┬────────────────────────────────────────────────┐
  │      Feature      │                      TC-Go (current)                       │            Docker Compose (PR #41)             │
  ├───────────────────┼────────────────────────────────────────────────────────────┼────────────────────────────────────────────────┤
  │ Log streaming     │ Real-time via LogConsumer → t.Log                          │ Would need docker logs shell-out               │
  ├───────────────────┼────────────────────────────────────────────────────────────┼────────────────────────────────────────────────┤
  │ Readiness         │ wait.ForLog(...), MySQL module built-in                    │ healthcheck in YAML + --wait flag              │
  ├───────────────────┼────────────────────────────────────────────────────────────┼────────────────────────────────────────────────┤
  │ Port discovery    │ ctr.MappedPort() — Go API                                  │ docker port — CLI subprocess                   │
  ├───────────────────┼────────────────────────────────────────────────────────────┼────────────────────────────────────────────────┤
  │ Cleanup           │ t.Cleanup + SKIP_CLEANUP + t.Failed()                      │ docker-compose down                            │
  ├───────────────────┼────────────────────────────────────────────────────────────┼────────────────────────────────────────────────┤
  │ Bazel sandbox     │ Works — init.go handles HOME/Ryuk                          │ Docker Compose CLI inside sandbox is awkward   │
  ├───────────────────┼────────────────────────────────────────────────────────────┼────────────────────────────────────────────────┤
  │ Extension tests   │ SetupMySQL(... nil ...) — no network needed                │ Would need separate compose file or workaround │
  ├───────────────────┼────────────────────────────────────────────────────────────┼────────────────────────────────────────────────┤
  │ Topology          │ Imperative Go code                                         │ Declarative YAML                               │
  ├───────────────────┼────────────────────────────────────────────────────────────┼────────────────────────────────────────────────┤
  │ Local dev sharing │ Separate — example/server/docker-compose.yml for local dev │ Same file for tests + local dev                │
  └───────────────────┴────────────────────────────────────────────────────────────┴────────────────────────────────────────────────┘

I am leaning towards keeping docker compose model as it works for both, CI and local development vs needing 2 separate set-ups for both.

This commit has TC-Go changes for ref: 5ce015e

Replace docker-compose based test setup with Testcontainers-Go for programmatic container lifecycle management in Go tests. - Add testutil helpers: SetupMySQL, StartServer, SetupNetwork - Real-time log streaming via LogConsumer -> t.Log - Readiness via wait.ForLog / MySQL module built-in health - SKIP_CLEANUP + t.Failed() container preservation for debugging - Container naming: sq-test-{context}-{shortid}-{service} - Delete compose.go (316 lines) and 5 docker-compose.yml test files - Fix PR #41 review comments: CI workflow, counter tests, SIGTERM Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Migrate from manual server setup to Docker Compose-based hermetic testing. Consolidate tests under test/ directory with consistent patterns. Key changes: - New test structure: test/e2e/, test/integration/{gateway,orchestrator,extensions}/ - Container naming: sq-test-{context}-{unique-id} for parallel execution - Shared utilities in test/testutil/ (compose, docker, mysql, schema helpers) - Removed old {service}/integration_test/ directories and speculator service - Documentation: TESTING.md → doc/howto/, STRUCTURE.md → PROJECT_STRUCTURE.md - Makefile: alphabetical targets, new integration test targets, local-start/stop naming - Consolidate CI and create separate jobs for faster run and add bazel cache

sbalabanov · 2026-02-24T01:51:49Z

Hm, I do not know how it computed "infrastructure" code. I do see fewer lines in TC-based implementation.
TC should also work for both test and cli.
Not blocking, feel free to proceed as you see fit.

behinddwalls mentioned this pull request Feb 23, 2026

feat(queue): complete Gateway → Orchestrator queue integration #39

Merged

behinddwalls force-pushed the preetam/test-refactor-consistent-schema branch 4 times, most recently from c6f1d37 to b22f5b8 Compare February 23, 2026 03:29

behinddwalls marked this pull request as ready for review February 23, 2026 03:44

behinddwalls requested review from a team and sbalabanov as code owners February 23, 2026 03:44

sbalabanov requested changes Feb 23, 2026

View reviewed changes

behinddwalls force-pushed the preetam/queue-integration branch 3 times, most recently from d9faf53 to 1cb8a36 Compare February 23, 2026 22:50

behinddwalls force-pushed the preetam/test-refactor-consistent-schema branch from b22f5b8 to 892f043 Compare February 24, 2026 01:19

behinddwalls changed the base branch from preetam/queue-integration to main February 24, 2026 01:19

behinddwalls requested a review from sbalabanov February 24, 2026 01:19

behinddwalls force-pushed the preetam/test-refactor-consistent-schema branch from 892f043 to 52e87c4 Compare February 24, 2026 01:32

kevinlnew approved these changes Feb 24, 2026

View reviewed changes

behinddwalls merged commit 7fd5211 into main Feb 24, 2026
8 checks passed

behinddwalls deleted the preetam/test-refactor-consistent-schema branch June 2, 2026 18:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor(test): consolidate testing infrastructure with Docker Compose#41

refactor(test): consolidate testing infrastructure with Docker Compose#41
behinddwalls merged 1 commit into
mainfrom
preetam/test-refactor-consistent-schema

behinddwalls commented Feb 23, 2026 •

edited

Loading

Uh oh!

sbalabanov left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

behinddwalls commented Feb 24, 2026 •

edited

Loading

Uh oh!

Uh oh!

sbalabanov commented Feb 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

behinddwalls commented Feb 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test Plan

Issues

Uh oh!

sbalabanov left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

behinddwalls commented Feb 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

sbalabanov commented Feb 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

behinddwalls commented Feb 23, 2026 •

edited

Loading

behinddwalls commented Feb 24, 2026 •

edited

Loading