Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion BUILD.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,6 @@ load("@gazelle//:def.bzl", "gazelle")
# Resolve protobuf import ambiguities - use the actual protopb packages, not the proto aliases
# gazelle:resolve go github.com/uber/submitqueue/gateway/protopb //gateway/protopb
# gazelle:resolve go github.com/uber/submitqueue/orchestrator/protopb //orchestrator/protopb
# gazelle:resolve go github.com/uber/submitqueue/speculator/protopb //speculator/protopb

# Export marker files for test data dependencies (used by FindRepoRoot in tests)
exports_files(
Expand Down
11 changes: 1 addition & 10 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ LOCAL_PROJECT = submitqueue
# Set REPO_ROOT for docker-compose
export REPO_ROOT := $(shell pwd)

.PHONY: build build-all-linux build-gateway-linux build-orchestrator-linux clean clean-proto deps e2e-test gazelle integration-test integration-test-extensions integration-test-gateway integration-test-orchestrator local-clean local-gateway-start local-gateway-stop local-init-schemas local-logs local-orchestrator-start local-orchestrator-stop local-ps local-restart local-start local-stop proto query-deps query-targets run-client-gateway run-client-orchestrator run-client-speculator run-queue-admin test test-no-cache help
.PHONY: build build-all-linux build-gateway-linux build-orchestrator-linux clean clean-proto deps e2e-test gazelle integration-test integration-test-extensions integration-test-gateway integration-test-orchestrator local-clean local-gateway-start local-gateway-stop local-init-schemas local-logs local-orchestrator-start local-orchestrator-stop local-ps local-restart local-start local-stop proto query-deps query-targets run-client-gateway run-client-orchestrator run-queue-admin test test-no-cache help


build: ## Build all services and examples
Expand Down Expand Up @@ -51,7 +51,6 @@ clean-proto: ## Clean generated proto files
@echo "Cleaning generated proto files..."
@rm -rf gateway/protopb/*.pb.go
@rm -rf orchestrator/protopb/*.pb.go
@rm -rf speculator/protopb/*.pb.go
@echo "Proto clean complete!"

deps: ## Install Go dependencies
Expand Down Expand Up @@ -202,10 +201,6 @@ proto: ## Generate protobuf files from .proto definitions
--go-grpc_out=orchestrator/protopb --go-grpc_opt=paths=source_relative \
--yarpc-go_out=orchestrator/protopb --yarpc-go_opt=paths=source_relative \
--proto_path=orchestrator/proto orchestrator/proto/orchestrator.proto
@protoc --go_out=speculator/protopb --go_opt=paths=source_relative \
--go-grpc_out=speculator/protopb --go-grpc_opt=paths=source_relative \
--yarpc-go_out=speculator/protopb --yarpc-go_opt=paths=source_relative \
--proto_path=speculator/proto speculator/proto/speculator.proto
@echo "Protobuf files generated successfully!"

# Bazel query helpers
Expand All @@ -223,10 +218,6 @@ run-client-gateway:
run-client-orchestrator:
@$(BAZEL) run //example/client/orchestrator:orchestrator -- -addr $(or $(SERVER_ADDR),localhost:8082) -message "$(or $(MESSAGE),ping)"

# Run speculator client (connects to any running speculator service)
run-client-speculator:
@$(BAZEL) run //example/client/speculator:speculator -- -addr $(or $(SERVER_ADDR),localhost:8083) -message "$(or $(MESSAGE),ping)"

run-queue-admin: ## Run queue-admin CLI (use ARGS to pass arguments, e.g. make run-queue-admin ARGS="list-topics")
@$(BAZEL) run //extension/queue/mysql/ctl -- $(ARGS)

Expand Down
16 changes: 3 additions & 13 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,18 +2,16 @@

## Services

Submit Queue consists of three main services:
Submit Queue consists of two main services:

- **Gateway**: Entry point for external requests (port 8081)
- **Orchestrator**: Coordinates job execution (port 8082)
- **Speculator**: Performs speculative builds (port 8083)

## gRPC API

Each service has its own proto definitions and exposes its own gRPC API:
- **SubmitQueueGateway**: Gateway API with Ping method (port 8081)
- **SubmitQueueGateway**: Gateway API with Ping and Land methods (port 8081)
- **SubmitQueueOrchestrator**: Orchestrator API with Ping method (port 8082)
- **SubmitQueueSpeculator**: Speculator API with Ping method (port 8083)

### Quick Start

Expand Down Expand Up @@ -185,12 +183,10 @@ make proto
# Run servers
make run-gateway # Gateway on port 8081
make run-orchestrator # Orchestrator on port 8082
make run-speculator # Speculator on port 8083

# Run clients
make run-client-gateway MESSAGE="hello"
make run-client-orchestrator MESSAGE="hello"
make run-client-speculator MESSAGE="hello"

# Clean binaries
make clean
Expand All @@ -211,12 +207,10 @@ go build ./...
# Build example servers
go build -o bin/gateway_server ./example/server/gateway/
go build -o bin/orchestrator_server ./example/server/orchestrator/
go build -o bin/speculator_server ./example/server/speculator/

# Build clients
go build -o bin/gateway_client ./example/client/gateway/
go build -o bin/orchestrator_client ./example/client/orchestrator/
go build -o bin/speculator_client ./example/client/speculator/

# Run a server
./bin/gateway_server
Expand Down Expand Up @@ -354,13 +348,9 @@ make run-gateway
# Terminal 2: Start orchestrator
make run-orchestrator

# Terminal 3: Start speculator
make run-speculator

# Terminal 4: Test each service
# Terminal 3: Test each service
make run-client-gateway MESSAGE="test gateway"
make run-client-orchestrator MESSAGE="test orchestrator"
make run-client-speculator MESSAGE="test speculator"
```

#### Using Bazel for Testing
Expand Down
3 changes: 1 addition & 2 deletions doc/rfc/sql-queue-rfc.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,7 @@ MySQL-based distributed message queue with partition leasing, visibility timeout
### Motivation

SubmitQueue needs a reliable message queue for coordinating asynchronous workflows:
- **Orchestrator** publishes merge jobs to workers
- **Speculator** publishes speculative build requests
- **Orchestrator** publishes merge jobs and speculative build requests to workers
- **Workers** need distributed coordination without duplicate processing
- **Crash recovery** must preserve exactly where processing stopped

Expand Down
Loading