Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .envrc
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
PATH_add tools
PATH_add tool

REPO_ROOT=$(pwd)
export REPO_ROOT
Expand Down
42 changes: 21 additions & 21 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,19 +49,19 @@ Three services, each following the same layout:
├── controller/ # Business logic (pure, transport-agnostic)
├── proto/ # Proto definitions (.proto files)
├── protopb/ # Generated proto code (committed to repo)
└── integration_tests/
└── integration_test/
```

### Controllers

Controllers contain pure business logic, independent of the transport layer (gRPC/YARPC). They live in `{service}/controller/` and are wired up in `examples/server/{service}/main.go`.
Controllers contain pure business logic, independent of the transport layer (gRPC/YARPC). They live in `{service}/controller/` and are wired up in `example/server/{service}/main.go`.

### Entities

Domain objects in `entities/`, organized by domain. Top-level entities live directly in `entities/`; domain-specific ones go in subdirectories.
Domain objects in `entity/`, organized by domain. Top-level entities live directly in `entity/`; domain-specific ones go in subdirectories.

```
entities/
entity/
├── request.go # Request, Change, enums (RequestState, RequestLandStrategy)
└── queue/
└── message.go # Message entity
Expand All @@ -80,7 +80,7 @@ entities/
Extensions are **vendor-agnostic, pluggable interfaces** for backend implementations. Each defines interfaces at the top level with implementations in subdirectories.

```
extensions/
extension/
├── counter/ # Atomic sequential number generation
│ ├── counter.go # Counter interface
│ └── mysql/ # MySQL implementation
Expand All @@ -91,24 +91,24 @@ extensions/
│ ├── delivery.go # Delivery interface
│ └── sql/ # SQL (MySQL) implementation
└── storage/ # Storage abstraction
├── storage.go # StoreFactory interface + sentinel errors
├── storage.go # Storage (factory) interface + sentinel errors
├── request_store.go # RequestStore interface
└── mysql/ # MySQL implementation
```

**Extension pattern:**
1. Define vendor-agnostic interfaces at `extensions/{extension}/`
2. Implementations go in `extensions/{extension}/{impl}/`
1. Define vendor-agnostic interfaces at `extension/{ext}/`
2. Implementations go in `extension/{ext}/{impl}/`
3. Most extensions use a Factory interface for dependency injection and lifecycle management
4. Include a README.md documenting interfaces and usage

### Import Paths

- Controllers: `github.com/uber/submitqueue/{service}/controller`
- Proto (generated): `github.com/uber/submitqueue/{service}/protopb`
- Extensions: `github.com/uber/submitqueue/extensions/{extension}`
- Extension impl: `github.com/uber/submitqueue/extensions/{extension}/{impl}`
- Entities: `github.com/uber/submitqueue/entities/{domain}`
- Extensions: `github.com/uber/submitqueue/extension/{extension}`
- Extension impl: `github.com/uber/submitqueue/extension/{extension}/{impl}`
- Entities: `github.com/uber/submitqueue/entity/{domain}`

## Development

Expand All @@ -122,17 +122,17 @@ submitqueue/
├── Makefile # Build automation
├── .bazelversion # Pinned Bazel version
├── .envrc # direnv configuration
├── tools/bazel # Bazelisk wrapper
├── tool/bazel # Bazelisk wrapper
├── gateway/ # Gateway service
├── orchestrator/ # Orchestrator service
├── speculator/ # Speculator service
├── extensions/ # Pluggable backend implementations
├── entities/ # Domain entities
├── examples/ # Server and client examples
├── extension/ # Pluggable backend implementations
├── entity/ # Domain entities
├── example/ # Server and client examples
│ ├── server/{service}/
│ └── client/{service}/
├── integration_tests/ # Cross-service hermetic tests (Testcontainers)
├── docs/ # Documentation
├── e2e_test/ # Cross-service hermetic tests (Testcontainers)
├── doc/ # Documentation
└── bin/ # Compiled binaries (gitignored)
```

Expand All @@ -143,7 +143,7 @@ This repository uses **Bazel with Bzlmod** (NOT WORKSPACE) for dependency manage
- **Version pinning**: `.bazelversion` pins the Bazel version
- **Dependencies**: Managed in `MODULE.bazel` (NOT a WORKSPACE file)
- **Go version**: Defined in `go.mod`, read by `MODULE.bazel` via `go_sdk.from_file()`
- **Bazel wrapper**: `./tools/bazel` (Bazelisk wrapper). With direnv (`.envrc`), use `bazel` directly.
- **Bazel wrapper**: `./tool/bazel` (Bazelisk wrapper). With direnv (`.envrc`), use `bazel` directly.
- **External dependencies**: Must be added to both `go.mod` AND `MODULE.bazel`
- **BUILD files**: Every Go package must have a `BUILD.bazel` file

Expand Down Expand Up @@ -187,16 +187,16 @@ make clean-proto # Remove generated proto files
1. Edit `{service}/proto/*.proto`
2. `make proto`
3. Add controller in `{service}/controller/`
4. Wire up in `examples/server/{service}/main.go`
4. Wire up in `example/server/{service}/main.go`

**Add new extension implementation:**
1. Create `extensions/{extension}/{impl}/` directory
1. Create `extension/{extension}/{impl}/` directory
2. Implement factory and core interfaces
3. Add `BUILD.bazel`
4. Add tests and document in README.md

**Add new entity:**
1. Create `entities/{domain}/{entity}.go` with test file
1. Create `entity/{domain}/{entity}.go` with test file
2. Add `BUILD.bazel` with `go_library` and `go_test` targets

### Testing Guidelines
Expand Down
50 changes: 25 additions & 25 deletions Makefile
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
.PHONY: proto build test integration-test integration-test-gateway integration-test-orchestrator integration-test-speculator e2e-test gazelle clean run-all start-servers stop-servers run-gateway run-orchestrator run-speculator run-client-gateway run-client-orchestrator run-client-speculator

# Bazel wrapper
BAZEL = ./tools/bazel
BAZEL = ./tool/bazel

# Generate protobuf files for all services using protoc
proto:
Expand All @@ -26,18 +26,18 @@ build:
@$(BAZEL) build //...
@echo "Copying binaries to ./bin/..."
@mkdir -p bin
@cp -f bazel-bin/examples/server/gateway/gateway_/gateway bin/gateway_server 2>/dev/null || \
cp -f bazel-bin/examples/server/gateway/gateway bin/gateway_server 2>/dev/null || true
@cp -f bazel-bin/examples/server/orchestrator/orchestrator_/orchestrator bin/orchestrator_server 2>/dev/null || \
cp -f bazel-bin/examples/server/orchestrator/orchestrator bin/orchestrator_server 2>/dev/null || true
@cp -f bazel-bin/examples/server/speculator/speculator_/speculator bin/speculator_server 2>/dev/null || \
cp -f bazel-bin/examples/server/speculator/speculator bin/speculator_server 2>/dev/null || true
@cp -f bazel-bin/examples/client/gateway/gateway_/gateway bin/gateway_client 2>/dev/null || \
cp -f bazel-bin/examples/client/gateway/gateway bin/gateway_client 2>/dev/null || true
@cp -f bazel-bin/examples/client/orchestrator/orchestrator_/orchestrator bin/orchestrator_client 2>/dev/null || \
cp -f bazel-bin/examples/client/orchestrator/orchestrator bin/orchestrator_client 2>/dev/null || true
@cp -f bazel-bin/examples/client/speculator/speculator_/speculator bin/speculator_client 2>/dev/null || \
cp -f bazel-bin/examples/client/speculator/speculator bin/speculator_client 2>/dev/null || true
@cp -f bazel-bin/example/server/gateway/gateway_/gateway bin/gateway_server 2>/dev/null || \
cp -f bazel-bin/example/server/gateway/gateway bin/gateway_server 2>/dev/null || true
@cp -f bazel-bin/example/server/orchestrator/orchestrator_/orchestrator bin/orchestrator_server 2>/dev/null || \
cp -f bazel-bin/example/server/orchestrator/orchestrator bin/orchestrator_server 2>/dev/null || true
@cp -f bazel-bin/example/server/speculator/speculator_/speculator bin/speculator_server 2>/dev/null || \
cp -f bazel-bin/example/server/speculator/speculator bin/speculator_server 2>/dev/null || true
@cp -f bazel-bin/example/client/gateway/gateway_/gateway bin/gateway_client 2>/dev/null || \
cp -f bazel-bin/example/client/gateway/gateway bin/gateway_client 2>/dev/null || true
@cp -f bazel-bin/example/client/orchestrator/orchestrator_/orchestrator bin/orchestrator_client 2>/dev/null || \
cp -f bazel-bin/example/client/orchestrator/orchestrator bin/orchestrator_client 2>/dev/null || true
@cp -f bazel-bin/example/client/speculator/speculator_/speculator bin/speculator_client 2>/dev/null || \
cp -f bazel-bin/example/client/speculator/speculator bin/speculator_client 2>/dev/null || true
@echo "Build complete! Binaries are in ./bin/"

# Run unit tests using Bazel (excludes integration tests which require running servers)
Expand All @@ -53,25 +53,25 @@ gazelle:
# Run integration tests for a specific service (requires that service to be running)
integration-test-gateway:
@echo "Running Gateway integration tests..."
@$(BAZEL) test //gateway/integration_tests:integration_tests_test --test_output=all
@$(BAZEL) test //gateway/integration_test:integration_test_test --test_output=all

integration-test-orchestrator:
@echo "Running Orchestrator integration tests..."
@$(BAZEL) test //orchestrator/integration_tests:integration_tests_test --test_output=all
@$(BAZEL) test //orchestrator/integration_test:integration_test_test --test_output=all

integration-test-speculator:
@echo "Running Speculator integration tests..."
@$(BAZEL) test //speculator/integration_tests:integration_tests_test --test_output=all
@$(BAZEL) test //speculator/integration_test:integration_test_test --test_output=all

# Run all service integration tests (requires all services to be running)
integration-test:
@echo "Running all service integration tests..."
@$(BAZEL) test //gateway/integration_tests:integration_tests_test //orchestrator/integration_tests:integration_tests_test //speculator/integration_tests:integration_tests_test --test_output=all
@$(BAZEL) test //gateway/integration_test:integration_test_test //orchestrator/integration_test:integration_test_test //speculator/integration_test:integration_test_test --test_output=all

# Run end-to-end integration tests (hermetic, no manual server setup needed)
e2e-test:
@echo "Running integration tests..."
@$(BAZEL) test //integration_tests:integration_test --test_output=all
@$(BAZEL) test //e2e_test:e2e_test --test_output=all

# Clean generated files and binaries
clean:
Expand Down Expand Up @@ -124,29 +124,29 @@ run-all: start-servers
# Run gateway server using Bazel
run-gateway:
@echo "Starting gateway server on port 8081..."
@$(BAZEL) run //examples/server/gateway:gateway
@$(BAZEL) run //example/server/gateway:gateway

# Run orchestrator server using Bazel
run-orchestrator:
@echo "Starting orchestrator server on port 8082..."
@$(BAZEL) run //examples/server/orchestrator:orchestrator
@$(BAZEL) run //example/server/orchestrator:orchestrator

# Run speculator server using Bazel
run-speculator:
@echo "Starting speculator server on port 8083..."
@$(BAZEL) run //examples/server/speculator:speculator
@$(BAZEL) run //example/server/speculator:speculator

# Run gateway client using Bazel
run-client-gateway:
@$(BAZEL) run //examples/client/gateway:gateway -- -addr $(or $(SERVER_ADDR),localhost:8081) -message "$(or $(MESSAGE),ping)"
@$(BAZEL) run //example/client/gateway:gateway -- -addr $(or $(SERVER_ADDR),localhost:8081) -message "$(or $(MESSAGE),ping)"

# Run orchestrator client using Bazel
run-client-orchestrator:
@$(BAZEL) run //examples/client/orchestrator:orchestrator -- -addr $(or $(SERVER_ADDR),localhost:8082) -message "$(or $(MESSAGE),ping)"
@$(BAZEL) run //example/client/orchestrator:orchestrator -- -addr $(or $(SERVER_ADDR),localhost:8082) -message "$(or $(MESSAGE),ping)"

# Run speculator client using Bazel
run-client-speculator:
@$(BAZEL) run //examples/client/speculator:speculator -- -addr $(or $(SERVER_ADDR),localhost:8083) -message "$(or $(MESSAGE),ping)"
@$(BAZEL) run //example/client/speculator:speculator -- -addr $(or $(SERVER_ADDR),localhost:8083) -message "$(or $(MESSAGE),ping)"

# Install dependencies (for go mod users)
deps:
Expand All @@ -160,7 +160,7 @@ query-targets:
@$(BAZEL) query //...

query-deps:
@$(BAZEL) query 'deps(//examples/server/gateway:gateway)'
@$(BAZEL) query 'deps(//example/server/gateway:gateway)'

# Help
help:
Expand Down
54 changes: 27 additions & 27 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,13 +23,13 @@ Build and run a service:
make run-gateway

# Using Go directly
go run examples/server/gateway/main.go
go run example/server/gateway/main.go

# Using Bazel (with direnv)
bazel run //examples/server/gateway:gateway
bazel run //example/server/gateway:gateway

# Or without direnv
./tools/bazel run //examples/server/gateway:gateway
./tool/bazel run //example/server/gateway:gateway
```

Test the service:
Expand All @@ -38,17 +38,17 @@ Test the service:
make run-client-gateway MESSAGE="hello"

# Or using Go directly
go run examples/client/gateway/main.go -message "hello"
go run example/client/gateway/main.go -message "hello"

# Or using grpcurl
grpcurl -plaintext -d '{"message": "hello"}' localhost:8081 uber.devexp.submitqueue.gateway.SubmitQueueGateway/Ping
```

For detailed instructions, see [examples/README.md](examples/README.md).
For detailed instructions, see [example/README.md](example/README.md).

## Project Structure

See [docs/architecture/STRUCTURE.md](docs/architecture/STRUCTURE.md) for a detailed breakdown of the project structure.
See [doc/architecture/STRUCTURE.md](doc/architecture/STRUCTURE.md) for a detailed breakdown of the project structure.

## Architecture

Expand All @@ -58,7 +58,7 @@ The project follows clean architecture principles with clear separation of conce
- Only depend on logger, metrics, and protobuf types
- Example: `PingController` handles ping business logic

- **Server Adapters** (`examples/server/`): gRPC transport layer
- **Server Adapters** (`example/server/`): gRPC transport layer
- Wrap controllers and implement gRPC service interfaces
- Handle protocol-specific concerns (e.g., `UnimplementedServiceServer`)

Expand All @@ -76,7 +76,7 @@ The project follows clean architecture principles with clear separation of conce
- **grpcurl** (optional, for manual testing)
- **direnv** (recommended, to automatically load `.envrc`)

**Note**: The project includes `./tools/bazel` (bazelisk wrapper) and `.bazelversion`, so you don't need to install Bazel or Bazelisk separately.
**Note**: The project includes `./tool/bazel` (bazelisk wrapper) and `.bazelversion`, so you don't need to install Bazel or Bazelisk separately.

#### Using direnv (Recommended)

Expand All @@ -92,7 +92,7 @@ eval "$(direnv hook zsh)" # or bash, fish, etc.
direnv allow
```

With direnv enabled, you can use `bazel` directly instead of `./tools/bazel`.
With direnv enabled, you can use `bazel` directly instead of `./tool/bazel`.

Install optional tools:
```bash
Expand Down Expand Up @@ -145,14 +145,14 @@ make proto
go build ./...

# Build example servers
go build -o bin/gateway_server ./examples/server/gateway/
go build -o bin/orchestrator_server ./examples/server/orchestrator/
go build -o bin/speculator_server ./examples/server/speculator/
go build -o bin/gateway_server ./example/server/gateway/
go build -o bin/orchestrator_server ./example/server/orchestrator/
go build -o bin/speculator_server ./example/server/speculator/

# Build clients
go build -o bin/gateway_client ./examples/client/gateway/
go build -o bin/orchestrator_client ./examples/client/orchestrator/
go build -o bin/speculator_client ./examples/client/speculator/
go build -o bin/gateway_client ./example/client/gateway/
go build -o bin/orchestrator_client ./example/client/orchestrator/
go build -o bin/speculator_client ./example/client/speculator/

# Run a server
./bin/gateway_server
Expand All @@ -165,25 +165,25 @@ go build -o bin/speculator_client ./examples/client/speculator/

The project uses **Bzlmod** (not WORKSPACE) for dependency management. Bazel version is pinned at 8.4.1 in `.bazelversion`.

The project includes `./tools/bazel` which automatically downloads the correct Bazel version. If you're using `direnv`, you can simply use `bazel` instead of `./tools/bazel`.
The project includes `./tool/bazel` which automatically downloads the correct Bazel version. If you're using `direnv`, you can simply use `bazel` instead of `./tool/bazel`.

```bash
# Build everything (with direnv)
bazel build //...

# Or without direnv
./tools/bazel build //...
./tool/bazel build //...

# Build specific components
bazel build //gateway/protopb
bazel build //examples/server/gateway:gateway
bazel build //examples/client/gateway:gateway
bazel build //example/server/gateway:gateway
bazel build //example/client/gateway:gateway

# Run a server
bazel run //examples/server/gateway:gateway
bazel run //example/server/gateway:gateway

# Run a client
bazel run //examples/client/gateway:gateway -- -message "hello"
bazel run //example/client/gateway:gateway -- -message "hello"

# Or use the Makefile (recommended)
make build
Expand All @@ -193,11 +193,11 @@ make run-gateway
**Note**:
- The repository uses Bzlmod for modern dependency management
- All generated proto files are committed to the repository
- With `direnv` enabled, use `bazel` directly; otherwise use `./tools/bazel`
- With `direnv` enabled, use `bazel` directly; otherwise use `./tool/bazel`

### Running Services

See the [examples directory](examples/) for examples of running each service.
See the [examples directory](example/) for examples of running each service.

## Development Workflow

Expand Down Expand Up @@ -226,7 +226,7 @@ When you make changes to `.proto` files, you need to regenerate the protobuf cod

4. Update client examples to display new fields:
```bash
# Edit examples/client/gateway/main.go to show the new field
# Edit example/client/gateway/main.go to show the new field
```

5. Rebuild and test:
Expand Down Expand Up @@ -358,7 +358,7 @@ make test

4. **Create server wrapper in example:**
```go
// In examples/server/gateway/main.go
// In example/server/gateway/main.go
// Add method delegation to GatewayServer struct:
func (s *GatewayServer) NewMethod(ctx context.Context, req *pb.NewRequest) (*pb.NewResponse, error) {
return s.newController.NewMethod(ctx, req)
Expand Down Expand Up @@ -403,6 +403,6 @@ bazel clean

**Bazel build issues:**
- Bazel version is pinned to 8.4.1 in `.bazelversion`
- With `direnv`, you can use `bazel` directly; otherwise use `./tools/bazel`
- Try `bazel shutdown` (or `./tools/bazel shutdown`) and rebuild
- With `direnv`, you can use `bazel` directly; otherwise use `./tool/bazel`
- Try `bazel shutdown` (or `./tool/bazel shutdown`) and rebuild
- The wrapper automatically downloads the correct Bazel version
Loading