Skip to content

Commit 35f59cb

Browse files
committed
refactor: relocate counter and rename queue messaging packages
Structural moves only — no behavior change. - Relocate submitqueue/extension/counter -> extension/counter (generic infrastructure belongs at the shared top level alongside the other cross-domain extensions). - Rename the messaging-transport packages entity/queue -> entity/messagequeue and extension/queue -> extension/messagequeue (package `messagequeue`), including the mysql impl, mock, ctl, and schema. "queue" was overloaded with the submit queue (Request.Queue, QueueConfig); messagequeue disambiguates. Repoints all import paths, converges call sites onto the entityqueue / extqueue / queuemock / queueMySQL aliases (no bare `queue.` package ref remains), moves the integration suites, and updates Makefile, CI targets, and docs. Test compose contexts renamed to ext-counter-mysql / ext-messagequeue-sql. ## Test Plan ✅ make build, make test ✅ make gazelle / tidy / mocks clean
1 parent 524b020 commit 35f59cb

137 files changed

Lines changed: 449 additions & 449 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.github/workflows/ci.yml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -117,7 +117,7 @@ jobs:
117117
- uses: ./.github/actions/setup
118118
- uses: ./.github/actions/run-bazel-test
119119
with:
120-
target: //test/integration/submitqueue/extension/counter/...
120+
target: //test/integration/extension/counter/...
121121

122122
queue-integration-test:
123123
name: Queue Extension Test
@@ -128,7 +128,7 @@ jobs:
128128
- uses: ./.github/actions/setup
129129
- uses: ./.github/actions/run-bazel-test
130130
with:
131-
target: //test/integration/extension/queue/...
131+
target: //test/integration/extension/messagequeue/...
132132

133133
storage-integration-test:
134134
name: Storage Extension Test

CLAUDE.md

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -140,8 +140,8 @@ Paths follow the directory layout: shared code is top-level, domain code nests u
140140
- Domain entities: `github.com/uber/submitqueue/{domain}/entity` (e.g. `.../submitqueue/entity`)
141141
- Domain extensions: `github.com/uber/submitqueue/{domain}/extension/{ext}[/{impl}]` (e.g. `.../submitqueue/extension/storage/mysql`)
142142
- Domain-internal infra: `github.com/uber/submitqueue/{domain}/core/{pkg}` (e.g. `.../submitqueue/core/consumer`, `.../submitqueue/core/request`)
143-
- Shared entities: `github.com/uber/submitqueue/entity/{name}` (e.g. `.../entity/queue`)
144-
- Shared extensions: `github.com/uber/submitqueue/extension/{name}` (e.g. `.../extension/queue`)
143+
- Shared entities: `github.com/uber/submitqueue/entity/{name}` (e.g. `.../entity/messagequeue`)
144+
- Shared extensions: `github.com/uber/submitqueue/extension/{name}` (e.g. `.../extension/messagequeue`)
145145
- Cross-domain infra: `github.com/uber/submitqueue/core/{pkg}` (e.g. `.../core/errs`, `.../core/metrics`)
146146
147147
## Development
@@ -167,7 +167,7 @@ Generated proto files are committed. When modifying `.proto` files:
167167
- **Directories**: singular (`mock/`, `entity/`, not `mocks/`, `entities/`)
168168
- **Files**: `{method}.go`, `{entity}.go`, `{file}_test.go`, `BUILD.bazel`
169169
- **Proto files**: `{service}.proto`
170-
- **Test compose contexts**: the `testContext` passed to `NewComposeStack` (and thus the `sq-test-{context}-…` Docker project/container names) must be **domain-qualified** — `{category}-{domain}-{name}` where `{category}` is `svc`/`ext`/`core`/`e2e` and `{domain}` is `submitqueue`/`stovepipe`/… (omit the domain only for shared/cross-domain suites, e.g. `ext-queue-sql`). This keeps containers unambiguous and lets suites run in parallel. See [doc/howto/TESTING.md](doc/howto/TESTING.md#container-naming).
170+
- **Test compose contexts**: the `testContext` passed to `NewComposeStack` (and thus the `sq-test-{context}-…` Docker project/container names) must be **domain-qualified** — `{category}-{domain}-{name}` where `{category}` is `svc`/`ext`/`core`/`e2e` and `{domain}` is `submitqueue`/`stovepipe`/… (omit the domain only for shared/cross-domain suites, e.g. `ext-messagequeue-sql`). This keeps containers unambiguous and lets suites run in parallel. See [doc/howto/TESTING.md](doc/howto/TESTING.md#container-naming).
171171
- **README files**: Do not duplicate interface or type definitions as code blocks in READMEs. Describe behavior in prose and let readers navigate to the source. Only include code samples when explicitly instructed.
172172
- **Markdown prose width**: Do not hard-wrap prose in Markdown docs (RFCs under `doc/`, READMEs). Write one line per paragraph and one line per list item, and let the editor soft-wrap — hard wrapping at a fixed column renders as a narrow fixed-width column regardless of window size. Code blocks, tables, and ASCII diagrams keep their own line breaks.
173173
@@ -241,7 +241,7 @@ To create a mock package for a new extension (e.g., `submitqueue/extension/newex
241241
3. Run `make mocks` to generate mock files into the new directory.
242242
4. Run `make gazelle` to create the `BUILD.bazel` file automatically.
243243

244-
For inline mocks (mock in the same package, e.g., `extension/queue/mysql/mock_stores.go`):
244+
For inline mocks (mock in the same package, e.g., `extension/messagequeue/mysql/mock_stores.go`):
245245

246246
1. Add a `//go:generate` directive with `-package=mypkg` and `-destination=mock_file.go`.
247247
2. Run `make mocks` and `make gazelle`.

Makefile

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -194,20 +194,20 @@ local-init-submitqueue-schemas: ## Manually apply all database schemas
194194
docker exec -i $(SUBMITQUEUE_LOCAL_PROJECT)-mysql-app-1 mysql -uroot -proot submitqueue < $$file 2>&1 | grep -v "Using a password" || true; \
195195
done
196196
@echo "Applying counter schema to mysql-app..."
197-
@for file in submitqueue/extension/counter/mysql/schema/*.sql; do \
197+
@for file in extension/counter/mysql/schema/*.sql; do \
198198
echo " - Applying $$(basename $$file)..."; \
199199
docker exec -i $(SUBMITQUEUE_LOCAL_PROJECT)-mysql-app-1 mysql -uroot -proot submitqueue < $$file 2>&1 | grep -v "Using a password" || true; \
200200
done
201201
@echo "Applying queue schema to mysql-queue..."
202-
@for file in extension/queue/mysql/schema/*.sql; do \
202+
@for file in extension/messagequeue/mysql/schema/*.sql; do \
203203
echo " - Applying $$(basename $$file)..."; \
204204
docker exec -i $(SUBMITQUEUE_LOCAL_PROJECT)-mysql-queue-1 mysql -uroot -proot submitqueue < $$file 2>&1 | grep -v "Using a password" || true; \
205205
done
206206
@echo "✅ All schemas applied successfully"
207207

208208
local-init-stovepipe-queue-schema: ## Apply queue schema only (mysql-queue) for Stovepipe compose stacks
209209
@echo "Applying queue schema to mysql-queue (Stovepipe; no app storage/counter schema yet)..."
210-
@for file in extension/queue/mysql/schema/*.sql; do \
210+
@for file in extension/messagequeue/mysql/schema/*.sql; do \
211211
echo " - Applying $$(basename $$file)..."; \
212212
docker exec -i $(STOVEPIPE_LOCAL_PROJECT)-mysql-queue-1 mysql -uroot -proot submitqueue < $$file 2>&1 | grep -v "Using a password" || true; \
213213
done
@@ -334,7 +334,7 @@ local-stovepipe-gateway-start: build-stovepipe-gateway-linux ## Start Stovepipe
334334

335335
mocks: ## Generate mock files using mockgen
336336
@echo "Generating mocks..."
337-
@$(BAZEL) run @rules_go//go -- generate ./submitqueue/extension/storage/... ./submitqueue/extension/buildrunner/... ./submitqueue/extension/changestore/... ./submitqueue/extension/counter/... ./extension/queue/... ./submitqueue/extension/queueconfig/... ./submitqueue/extension/mergechecker/... ./submitqueue/extension/pusher/... ./submitqueue/extension/scorer/... ./submitqueue/extension/conflict/... ./submitqueue/core/consumer/...
337+
@$(BAZEL) run @rules_go//go -- generate ./submitqueue/extension/storage/... ./submitqueue/extension/buildrunner/... ./submitqueue/extension/changestore/... ./extension/counter/... ./extension/messagequeue/... ./submitqueue/extension/queueconfig/... ./submitqueue/extension/mergechecker/... ./submitqueue/extension/pusher/... ./submitqueue/extension/scorer/... ./submitqueue/extension/conflict/... ./submitqueue/core/consumer/...
338338
@echo "Mocks generated successfully!"
339339

340340
proto: ## Generate protobuf files from .proto definitions
@@ -381,7 +381,7 @@ run-client-stovepipe-orchestrator:
381381
@$(BAZEL) run //example/stovepipe/orchestrator/client:orchestrator -- -addr $(or $(SERVER_ADDR),localhost:8084) -message "$(or $(MESSAGE),ping)"
382382

383383
run-queue-admin: ## Run queue-admin CLI (use ARGS to pass arguments, e.g. make run-queue-admin ARGS="list-topics")
384-
@$(BAZEL) run //extension/queue/mysql/ctl -- $(ARGS)
384+
@$(BAZEL) run //extension/messagequeue/mysql/ctl -- $(ARGS)
385385

386386
test: ## Run unit tests
387387
@echo "Running unit tests..."

doc/howto/TESTING.md

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -23,13 +23,13 @@ SubmitQueue uses **two separate databases** to demonstrate proper architectural
2323

2424
### 1. Application Database
2525
- **Purpose**: Business data (requests, counters, batches)
26-
- **Schema**: `submitqueue/extension/storage/mysql/schema`, `submitqueue/extension/counter/mysql/schema`
26+
- **Schema**: `submitqueue/extension/storage/mysql/schema`, `extension/counter/mysql/schema`
2727
- **Used by**: Gateway (stores requests), Orchestrator (reads/updates request state)
2828
- **Connection**: `MYSQL_DSN`
2929

3030
### 2. Queue Database
3131
- **Purpose**: Messaging infrastructure (queue messages, offsets, partition leases)
32-
- **Schema**: `extension/queue/mysql/schema`
32+
- **Schema**: `extension/messagequeue/mysql/schema`
3333
- **Used by**: Gateway (publishes), Orchestrator (consumes)
3434
- **Connection**: `QUEUE_MYSQL_DSN`
3535

@@ -128,7 +128,7 @@ The `{context}` passed to `NewComposeStack` is **domain-qualified** so that the
128128
- `{domain}``submitqueue`, `stovepipe`, … — **omit for shared/cross-domain code**
129129
- `{name}` — the specific service/extension (e.g. `gateway`, `storage-mysql`)
130130

131-
Shared (cross-domain) suites carry no domain segment — e.g. the shared queue extension uses `ext-queue-sql`.
131+
Shared (cross-domain) suites carry no domain segment — e.g. the shared queue extension uses `ext-messagequeue-sql`.
132132

133133
### Context reference
134134

@@ -138,9 +138,9 @@ Shared (cross-domain) suites carry no domain segment — e.g. the shared queue e
138138
| SubmitQueue orchestrator | `svc-submitqueue-orchestrator` | `sq-test-svc-submitqueue-orchestrator-xyz789-orchestrator-service-1` |
139139
| Stovepipe gateway | `svc-stovepipe-gateway` | `sq-test-svc-stovepipe-gateway-abc123-gateway-service-1` |
140140
| SubmitQueue storage extension | `ext-submitqueue-storage-mysql` | `sq-test-ext-submitqueue-storage-mysql-2ce1d0-mysql-1` |
141-
| SubmitQueue counter extension | `ext-submitqueue-counter-mysql` | `sq-test-ext-submitqueue-counter-mysql-…-mysql-1` |
141+
| Counter extension (shared) | `ext-counter-mysql` | `sq-test-ext-counter-mysql-…-mysql-1` |
142142
| SubmitQueue changestore extension | `ext-submitqueue-changestore-mysql` | `sq-test-ext-submitqueue-changestore-mysql-…-mysql-1` |
143-
| Shared queue extension | `ext-queue-sql` | `sq-test-ext-queue-sql-a1b2c3-mysql-1` |
143+
| Shared queue extension | `ext-messagequeue-sql` | `sq-test-ext-messagequeue-sql-a1b2c3-mysql-1` |
144144
| SubmitQueue consumer (core) | `core-submitqueue-consumer` | `sq-test-core-submitqueue-consumer-…-mysql-1` |
145145
| SubmitQueue e2e (full stack) | `e2e-submitqueue` | `sq-test-e2e-submitqueue-def456-gateway-service-1` |
146146

doc/rfc/sql-queue-rfc.md

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -172,7 +172,7 @@ Messages are append-only. No per-message mutation during delivery.
172172
- `(topic, partition_key, offset)`: Core fetch query — poll messages in partition ordered by offset
173173
- `(topic, partition_key, id)`: Unique constraint and idempotent publish
174174

175-
See `extension/queue/mysql/schema/queue_messages.sql` for full schema.
175+
See `extension/messagequeue/mysql/schema/queue_messages.sql` for full schema.
176176

177177
### Delivery State Table
178178

@@ -184,7 +184,7 @@ Per-consumer-group delivery tracking with explicit ack state.
184184
- `invisible_until`: Visibility timeout in epoch milliseconds (only meaningful when `acked = FALSE`)
185185
- `retry_count`: Number of times message has been redelivered to this consumer group
186186

187-
See `extension/queue/mysql/schema/queue_delivery_state.sql` for full schema.
187+
See `extension/messagequeue/mysql/schema/queue_delivery_state.sql` for full schema.
188188

189189
### Partition Leases Table
190190

@@ -197,7 +197,7 @@ See `extension/queue/mysql/schema/queue_delivery_state.sql` for full schema.
197197
- `(leased_by)`: Find all partitions owned by a worker
198198
- `(lease_renewed_at)`: Detect stale leases across workers
199199

200-
See `extension/queue/mysql/schema/queue_partition_leases.sql` for full schema.
200+
See `extension/messagequeue/mysql/schema/queue_partition_leases.sql` for full schema.
201201

202202
### Consumer Offsets Table
203203

@@ -210,7 +210,7 @@ See `extension/queue/mysql/schema/queue_partition_leases.sql` for full schema.
210210
- `(consumer_group)`: Monitor all offsets for a consumer group
211211
- `(topic)`: Find all consumers for a topic
212212

213-
See `extension/queue/mysql/schema/queue_offsets.sql` for full schema.
213+
See `extension/messagequeue/mysql/schema/queue_offsets.sql` for full schema.
214214

215215
### Subscriber Heartbeats Table
216216

@@ -219,7 +219,7 @@ See `extension/queue/mysql/schema/queue_offsets.sql` for full schema.
219219
- `heartbeat_at`: Unix timestamp in milliseconds of last heartbeat
220220
- `deregistered_at`: Soft-delete timestamp (0 = active, >0 = deregistered during graceful shutdown)
221221

222-
See `extension/queue/mysql/schema/queue_subscriber_heartbeats.sql` for full schema.
222+
See `extension/messagequeue/mysql/schema/queue_subscriber_heartbeats.sql` for full schema.
223223

224224
### Dead Letter Queue
225225

Original file line numberDiff line numberDiff line change
@@ -1,15 +1,15 @@
11
load("@rules_go//go:def.bzl", "go_library", "go_test")
22

33
go_library(
4-
name = "queue",
4+
name = "messagequeue",
55
srcs = ["message.go"],
6-
importpath = "github.com/uber/submitqueue/entity/queue",
6+
importpath = "github.com/uber/submitqueue/entity/messagequeue",
77
visibility = ["//visibility:public"],
88
)
99

1010
go_test(
11-
name = "queue_test",
11+
name = "messagequeue_test",
1212
srcs = ["message_test.go"],
13-
embed = [":queue"],
13+
embed = [":messagequeue"],
1414
deps = ["@com_github_stretchr_testify//assert"],
1515
)
Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@
1212
// See the License for the specific language governing permissions and
1313
// limitations under the License.
1414

15-
package queue
15+
package messagequeue
1616

1717
import (
1818
"maps"
Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@
1212
// See the License for the specific language governing permissions and
1313
// limitations under the License.
1414

15-
package queue
15+
package messagequeue
1616

1717
import (
1818
"testing"

example/submitqueue/gateway/server/BUILD.bazel

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -11,9 +11,9 @@ go_library(
1111
importpath = "github.com/uber/submitqueue/example/submitqueue/gateway/server",
1212
visibility = ["//visibility:private"],
1313
deps = [
14-
"//extension/queue/mysql",
14+
"//extension/counter/mysql",
15+
"//extension/messagequeue/mysql",
1516
"//submitqueue/core/consumer",
16-
"//submitqueue/extension/counter/mysql",
1717
"//submitqueue/extension/queueconfig/yaml",
1818
"//submitqueue/extension/storage/mysql",
1919
"//submitqueue/gateway/controller",

example/submitqueue/gateway/server/main.go

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -28,9 +28,9 @@ import (
2828

2929
_ "github.com/go-sql-driver/mysql"
3030
"github.com/uber-go/tally/v4"
31-
queueMySQL "github.com/uber/submitqueue/extension/queue/mysql"
31+
mysqlcounter "github.com/uber/submitqueue/extension/counter/mysql"
32+
queueMySQL "github.com/uber/submitqueue/extension/messagequeue/mysql"
3233
"github.com/uber/submitqueue/submitqueue/core/consumer"
33-
mysqlcounter "github.com/uber/submitqueue/submitqueue/extension/counter/mysql"
3434
yamlqueueconfig "github.com/uber/submitqueue/submitqueue/extension/queueconfig/yaml"
3535
mysqlstorage "github.com/uber/submitqueue/submitqueue/extension/storage/mysql"
3636
"github.com/uber/submitqueue/submitqueue/gateway/controller"

0 commit comments

Comments
 (0)