You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
refactor: nest domain code under submitqueue/, extract shared layers
## Summary
### Why?
SubmitQueue's domain code lived at the repo root (gateway/, orchestrator/,
entity/, extension/) while a second service family, Stovepipe, already lived
under stovepipe/. The layout was asymmetric and the line between
domain-specific and genuinely shared code was implicit. This restructure
gives every domain its own top-level folder and pulls shared code into
clearly labeled shared layers, so "what's shared vs domain" is obvious from
the directory tree.
### What?
New layout — shared building blocks at the top level, one folder per domain:
```
core/ # SHARED cross-domain infra: errs, httpclient, metrics (no domain deps)
entity/ # SHARED entities: queue/
extension/ # SHARED extensions: queue/
submitqueue/ # SubmitQueue domain
├── gateway/
├── orchestrator/
├── entity/
├── extension/
└── core/ # consumer, request (shared between SubmitQueue's own services)
stovepipe/ # Stovepipe domain (gateway/ orchestrator/ entity/ extension/ core/)
example/
├── submitqueue/ # servers + clients + docker-compose
└── stovepipe/
test/
├── integration/{core,submitqueue,stovepipe}/
└── e2e/submitqueue/
doc/rfc/ # sql-queue-rfc.md (shared) + submitqueue/{workflow,build-runner}.md
```
- Domain code nests per domain: submitqueue/{gateway,orchestrator,entity,extension,core}, mirroring the existing stovepipe/ layout.
- Shared layers at the top level: core/ (cross-domain infra — errs, httpclient, metrics, now with zero domain deps), entity/ (shared entities: queue), extension/ (shared extensions: queue).
- submitqueue/core/ holds infra shared only between SubmitQueue's own services (consumer, request), moved out of the top-level core/ so it is domain-free. stovepipe/core/ added as a placeholder for symmetry.
- example/ and test/ mirror the split: example/submitqueue/..., test/integration/{core,submitqueue,stovepipe}/..., test/e2e/submitqueue/.
- RFCs: SubmitQueue-specific RFCs moved under doc/rfc/submitqueue/; the shared sql-queue RFC stays at doc/rfc/.
- Make targets made domain-explicit (build-submitqueue-*, integration-test-submitqueue-*, local-submitqueue-*), mirroring the existing *-stovepipe-* targets; CI updated to match.
- Tests: domain-qualified docker-compose project names; integration/e2e run in parallel (--test_output=errors) and always execute a real pass (tags=["external"]) since they are non-hermetic.
- Import paths rewritten throughout; protos regenerated; CLAUDE.md, READMEs, and doc/howto/TESTING.md updated to the new structure and naming convention.
No behavior changes — this is a structural move.
## Test Plan
- ✅ `make build` (bazel build //...) and `make test` (40/40 unit tests)
- ✅ `make integration-test` — 8/8 pass; verified running in parallel (67s vs 275s serial) and always-run (uncached re-runs)
- ✅ `make e2e-test` — passes; always-run
- ✅ `make lint` (incl. license headers); gazelle / mocks / tidy / fmt all stable (CI gates clean)
Copy file name to clipboardExpand all lines: CLAUDE.md
+54-39Lines changed: 54 additions & 39 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -9,7 +9,7 @@ SubmitQueue is a distributed system for managing code submission workflows. It f
9
9
1.**Immutable entities** — once created, don't modify in place. Create new versions with updated fields.
10
10
2.**Eventual consistency** — handle stale reads, idempotent operations, and convergence over time.
11
11
3.**Event sourcing** — store events (what happened) rather than just current state for critical changes.
12
-
4.**Optimistic locking** — use version numbers instead of pessimistic locks. Avoid transactions; prefer optimistic concurrency and retries. **Version arithmetic lives in the controller, not the storage layer.** Update methods take both `oldVersion` (the where-clause guard) and `newVersion` (the value to write); the store performs a pure conditional write. Controllers compute `newVersion = oldVersion + 1`, call the store, and only assign `entity.Version = newVersion` after the call succeeds. Pre-incrementing in memory before the call is a bug pattern — on error the in-memory version drifts ahead of the database. See [extension/storage/README.md](extension/storage/README.md).
12
+
4.**Optimistic locking** — use version numbers instead of pessimistic locks. Avoid transactions; prefer optimistic concurrency and retries. **Version arithmetic lives in the controller, not the storage layer.** Update methods take both `oldVersion` (the where-clause guard) and `newVersion` (the value to write); the store performs a pure conditional write. Controllers compute `newVersion = oldVersion + 1`, call the store, and only assign `entity.Version = newVersion` after the call succeeds. Pre-incrementing in memory before the call is a bug pattern — on error the in-memory version drifts ahead of the database. See [submitqueue/extension/storage/README.md](submitqueue/extension/storage/README.md).
13
13
5.**Idempotency keys** — include unique request IDs, check for duplicates before executing.
14
14
15
15
```go
@@ -35,27 +35,37 @@ request.Version = newVersion
35
35
### Project Layout
36
36
37
37
```
38
-
submitqueue/
39
-
├── gateway/ # Gateway service (port 8081) - entry point
40
-
├── orchestrator/ # Orchestrator service (port 8082) - coordinates jobs
│ └── testutil/ # Test utilities (ComposeStack, MySQL helpers)
56
64
└── doc/ # Documentation
57
65
```
58
66
67
+
The repo hosts shared building blocks at the top level — cross-domain infrastructure in `core/`, shared entities in `entity/`, shared extensions in `extension/` — followed by one folder per **domain** (`submitqueue/`, `stovepipe/`). Each domain owns the same internal layout (`gateway/`, `orchestrator/`, `entity/`, `extension/`, `core/`); a domain's own `core/` (e.g. `submitqueue/core/`) holds infra shared only between that domain's services.
68
+
59
69
### Services
60
70
61
71
Each service follows the same layout:
@@ -122,13 +132,17 @@ When in doubt, ask: *"If the next implementation were DynamoDB / Kafka / Bigtabl
- **Test compose contexts**: the `testContext` passed to `NewComposeStack` (and thus the `sq-test-{context}-…` Docker project/container names) must be **domain-qualified** — `{category}-{domain}-{name}` where `{category}` is `svc`/`ext`/`core`/`e2e` and `{domain}` is `submitqueue`/`stovepipe`/… (omit the domain only for shared/cross-domain suites, e.g. `ext-queue-sql`). This keeps containers unambiguous and lets suites run in parallel. See [doc/howto/TESTING.md](doc/howto/TESTING.md#container-naming).
156
171
- **README files**: Do not duplicate interface or type definitions as code blocks in READMEs. Describe behavior in prose and let readers navigate to the source. Only include code samples when explicitly instructed.
157
172
- **Markdown prose width**: Do not hard-wrap prose in Markdown docs (RFCs under `doc/`, READMEs). Write one line per paragraph and one line per list item, and let the editor soft-wrap — hard wrapping at a fixed column renders as a narrow fixed-width column regardless of window size. Code blocks, tables, and ASCII diagrams keep their own line breaks.
158
173
@@ -180,36 +195,36 @@ make mocks # Generate mock files using mockgen
180
195
make integration-test # Run all integration tests (Docker-based)
181
196
make e2e-test # Run end-to-end tests
182
197
make proto # Regenerate proto files
183
-
make local-start # Start full stack with Docker Compose
184
-
make local-ps # Show running containers and ports
185
-
make local-logs # View logs from all services
198
+
make local-submitqueue-start # Start full stack with Docker Compose
199
+
make local-submitqueue-ps # Show running containers and ports
200
+
make local-submitqueue-logs # View logs from all services
2. Wire up in `example/{domain}/{service}/server/main.go`
200
215
201
216
**Add new extension:**
202
-
1. Create `extension/{ext}/{impl}/` with factory and interfaces
217
+
1. Create the extension under `{domain}/extension/{ext}/{impl}/` (domain-specific, e.g. `submitqueue/extension/...`) or top-level `extension/{ext}/{impl}/` (shared across domains) with factory and interfaces
203
218
2. Add `BUILD.bazel`, tests, and README.md
204
219
205
220
**Add new entity:**
206
-
1. Create `entity/{domain}/{entity}.go` with test file and `BUILD.bazel`
221
+
1. Create `{domain}/entity/{entity}.go` (domain-specific) or top-level `entity/{name}/{entity}.go` (shared) with test file and `BUILD.bazel`
207
222
208
223
**Add gomock for an extension interface:**
209
224
210
-
Mocks are checked-in files generated by [mockgen](https://github.com/uber-go/mock). Run `make mocks` to regenerate, then `make gazelle` to update BUILD files. See `extension/storage/mock/` for the canonical example.
225
+
Mocks are checked-in files generated by [mockgen](https://github.com/uber-go/mock). Run `make mocks` to regenerate, then `make gazelle` to update BUILD files. See `submitqueue/extension/storage/mock/` for the canonical example.
211
226
212
-
To add a mock for a new interface file in an existing mock package (e.g., `extension/storage/new_store.go`):
227
+
To add a mock for a new interface file in an existing mock package (e.g., `submitqueue/extension/storage/new_store.go`):
213
228
214
229
1. Add a `//go:generate` directive to the interface file:
215
230
```go
@@ -219,10 +234,10 @@ To add a mock for a new interface file in an existing mock package (e.g., `exten
219
234
3. Run `make gazelle` to update `BUILD.bazel` files.
220
235
4. Commit the generated mock file.
221
236
222
-
To create a mock package for a new extension (e.g., `extension/newext/mock/`):
237
+
To create a mock package for a new extension (e.g., `submitqueue/extension/newext/mock/`):
223
238
224
239
1. Add `//go:generate` directives to each interface file (same pattern as above).
225
-
2. Create the `mock/` directory: `mkdir extension/newext/mock/`.
240
+
2. Create the `mock/` directory: `mkdir submitqueue/extension/newext/mock/`.
226
241
3. Run `make mocks` to generate mock files into the new directory.
227
242
4. Run `make gazelle` to create the `BUILD.bazel` file automatically.
228
243
@@ -233,7 +248,7 @@ For inline mocks (mock in the same package, e.g., `extension/queue/mysql/mock_st
0 commit comments