From 68c2f8272d270641ef46c05f88f7c2fe71982c0c Mon Sep 17 00:00:00 2001 From: Preetam Dwivedi Date: Thu, 5 Mar 2026 19:01:58 -0800 Subject: [PATCH 1/5] Add issue templates, contributing guide, code of conduct, and streamline README Add community contribution infrastructure: bug report and feature request issue templates, CONTRIBUTING.md, and CODE_OF_CONDUCT.md (Contributor Covenant v1.4). Streamline README from 463 lines to a concise entry point that links to CLAUDE.md, CONTRIBUTING.md, and TESTING.md, fixing broken links and inaccuracies. --- .github/ISSUE_TEMPLATE/bug_report.md | 38 ++ .github/ISSUE_TEMPLATE/config.yml | 1 + .github/ISSUE_TEMPLATE/feature_request.md | 22 ++ CODE_OF_CONDUCT.md | 75 ++++ CONTRIBUTING.md | 170 ++++++++ README.md | 460 +--------------------- 6 files changed, 325 insertions(+), 441 deletions(-) create mode 100644 .github/ISSUE_TEMPLATE/bug_report.md create mode 100644 .github/ISSUE_TEMPLATE/config.yml create mode 100644 .github/ISSUE_TEMPLATE/feature_request.md create mode 100644 CODE_OF_CONDUCT.md create mode 100644 CONTRIBUTING.md diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md new file mode 100644 index 00000000..8dcb9cb5 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/bug_report.md @@ -0,0 +1,38 @@ +--- +name: Bug Report +about: Report a bug to help us improve SubmitQueue +title: "[Bug] " +labels: bug +--- + +## Description + +A clear and concise description of the bug. + +## Steps to Reproduce + +1. +2. +3. + +## Expected Behavior + +What you expected to happen. + +## Actual Behavior + +What actually happened. + +## Environment + +- **Go version**: +- **OS**: +- **Bazel version**: + +## Logs / Screenshots + +If applicable, add logs or screenshots to help explain the problem. + +## Additional Context + +Any other context about the problem. diff --git a/.github/ISSUE_TEMPLATE/config.yml b/.github/ISSUE_TEMPLATE/config.yml new file mode 100644 index 00000000..3ba13e0c --- /dev/null +++ b/.github/ISSUE_TEMPLATE/config.yml @@ -0,0 +1 @@ +blank_issues_enabled: false diff --git a/.github/ISSUE_TEMPLATE/feature_request.md b/.github/ISSUE_TEMPLATE/feature_request.md new file mode 100644 index 00000000..22922c84 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/feature_request.md @@ -0,0 +1,22 @@ +--- +name: Feature Request +about: Suggest a new feature or improvement for SubmitQueue +title: "[Feature] " +labels: enhancement +--- + +## Problem Statement + +A clear and concise description of the problem this feature would solve. + +## Proposed Solution + +Describe the solution you'd like. + +## Alternatives Considered + +Describe any alternative solutions or features you've considered. + +## Additional Context + +Any other context, mockups, or examples about the feature request. diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md new file mode 100644 index 00000000..bc988b72 --- /dev/null +++ b/CODE_OF_CONDUCT.md @@ -0,0 +1,75 @@ +# Contributor Covenant Code of Conduct + +## Our Pledge + +In the interest of fostering an open and welcoming environment, we as +contributors and maintainers pledge to making participation in our project and +our community a harassment-free experience for everyone, regardless of age, +body size, disability, ethnicity, gender identity and expression, level of +experience, nationality, personal appearance, race, religion, or sexual +identity and orientation. + +## Our Standards + +Examples of behavior that contributes to creating a positive environment +include: + +* Using welcoming and inclusive language +* Being respectful of differing viewpoints and experiences +* Gracefully accepting constructive criticism +* Focusing on what is best for the community +* Showing empathy towards other community members + +Examples of unacceptable behavior by participants include: + +* The use of sexualized language or imagery and unwelcome sexual attention or + advances +* Trolling, insulting/derogatory comments, and personal or political attacks +* Public or private harassment +* Publishing others' private information, such as a physical or electronic + address, without explicit permission +* Other conduct which could reasonably be considered inappropriate in a + professional setting + +## Our Responsibilities + +Project maintainers are responsible for clarifying the standards of acceptable +behavior and are expected to take appropriate and fair corrective action in +response to any instances of unacceptable behavior. + +Project maintainers have the right and responsibility to remove, edit, or +reject comments, commits, code, wiki edits, issues, and other contributions +that are not aligned to this Code of Conduct, or to ban temporarily or +permanently any contributor for other behaviors that they deem inappropriate, +threatening, offensive, or harmful. + +## Scope + +This Code of Conduct applies both within project spaces and in public spaces +when an individual is representing the project or its community. Examples of +representing a project or community include using an official project e-mail +address, posting via an official social media account, or acting as an +appointed representative at an online or offline event. Representation of a +project may be further defined and clarified by project maintainers. + +## Enforcement + +Instances of abusive, harassing, or otherwise unacceptable behavior may be +reported by contacting the project team at oss-conduct@uber.com. The project +team will review and investigate all complaints, and will respond in a way +that it deems appropriate to the circumstances. The project team is obligated +to maintain confidentiality with regard to the reporter of an incident. +Further details of specific enforcement policies may be posted separately. + +Project maintainers who do not follow or enforce the Code of Conduct in good +faith may face temporary or permanent repercussions as determined by other +members of the project's leadership. + +## Attribution + +This Code of Conduct is adapted from the [Contributor Covenant][homepage], +version 1.4, available at +[http://contributor-covenant.org/version/1/4][version]. + +[homepage]: https://contributor-covenant.org +[version]: https://contributor-covenant.org/version/1/4/ diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 00000000..0369be3b --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,170 @@ +# Contributing to SubmitQueue + +Thank you for your interest in contributing to SubmitQueue! This document provides guidelines and instructions for contributing. + +## Getting Started + +### Prerequisites + +- **Go 1.24 or later** (optional — Bazel manages its own Go toolchain) +- **Docker** and **Docker Compose** (for integration and e2e tests) +- **direnv** (recommended — automatically loads `.envrc` so you can use `bazel` directly) + +The project includes `./tool/bazel` (Bazelisk wrapper) and `.bazelversion`, so you don't need to install Bazel separately. + +### Clone and Build + +```bash +git clone https://github.com/uber/submitqueue.git +cd submitqueue + +# Optional: allow direnv +direnv allow + +# Build everything +make build + +# Run unit tests +make test +``` + +### Optional Tools + +```bash +# macOS +brew install protobuf grpcurl direnv + +# Go protoc plugins (only if modifying .proto files) +go install google.golang.org/protobuf/cmd/protoc-gen-go@latest +go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest +go install go.uber.org/yarpc/encoding/protobuf/protoc-gen-yarpc-go@latest +``` + +## Development Workflow + +1. Fork the repository and create a feature branch from `main`. +2. Make your changes, following the code style and conventions below. +3. Add or update tests as appropriate. +4. Run `make gazelle` if you added or removed Go files. +5. Run `make test` to ensure unit tests pass. +6. Commit with a clear, descriptive message. +7. Open a pull request against `main`. + +## Code Style + +The project follows specific conventions for code style, error handling, logging, and entity design. See [CLAUDE.md](CLAUDE.md) for the full set of conventions, including: + +- Structured logging with `zap.SugaredLogger` +- Interfaces for behavior, structs for data +- Value types over pointers +- Error classification with `core/errs` +- Immutable entities and optimistic locking + +## Testing + +See [doc/howto/TESTING.md](doc/howto/TESTING.md) for the full testing guide, including integration and end-to-end tests. + +Key commands: + +```bash +make test # Unit tests +make integration-test # Integration tests (Docker-based, auto-builds binaries) +make e2e-test # End-to-end tests +``` + +All tests use table-driven style with `t.Run` subtests. Use `assert`/`require` from testify. + +## Proto Changes + +When modifying `.proto` files: + +1. Edit the proto file in `{service}/proto/`. +2. Run `make proto` to regenerate `*.pb.go`, `*_grpc.pb.go`, and `*.pb.yarpc.go`. +3. Update controllers and clients as needed. +4. Commit all generated files. + +See [CLAUDE.md](CLAUDE.md) for detailed workflows (adding RPC methods, queue controllers, extensions, entities). + +## Adding Extensions + +Extensions follow a vendor-agnostic interface pattern: + +1. Define the interface at `extension/{ext}/`. +2. Add implementations at `extension/{ext}/{impl}/`. +3. Include a factory interface for dependency injection. +4. Add `BUILD.bazel`, tests, and README. + +See [CLAUDE.md](CLAUDE.md) for the full extension guide and mock setup instructions. + +## Reporting Issues + +Use the [issue templates](.github/ISSUE_TEMPLATE/) when filing bugs or requesting features. Blank issues are disabled — please choose the appropriate template. + +## Code Review + +- All submissions require review before merging. +- Maintainers may request changes or suggest improvements. +- Keep PRs focused — one logical change per PR. + +## Shell Configuration (Optional) + +### Using direnv (Recommended) + +```bash +brew install direnv + +# Add to ~/.zshrc or ~/.bashrc +eval "$(direnv hook zsh)" # or bash, fish, etc. + +# In the project directory +direnv allow +``` + +### Make Target Auto-Completion (zsh) + +Add to `~/.zshrc` for tab-completion of Makefile targets with descriptions: + +```bash +autoload -Uz compinit +compinit + +function _make_targets() { + local -a targets + local makefile_cache=".make_targets_cache" + + if [[ -f Makefile ]]; then + if [[ ! -f $makefile_cache ]] || [[ Makefile -nt $makefile_cache ]]; then + awk -F':.*?## ' '/^[a-zA-Z0-9_-]+:.*?## / {printf "%s:%s\n", $1, $2}' Makefile > $makefile_cache + fi + targets=(${(f)"$(<$makefile_cache)"}) + if [[ -s $makefile_cache ]] && grep -q ':' $makefile_cache 2>/dev/null; then + _describe 'make targets' targets + else + awk -F: '/^[a-zA-Z0-9_-]+:/ {print $1}' Makefile > $makefile_cache + targets=(${(f)"$(<$makefile_cache)"}) + _describe 'make targets' targets + fi + fi +} + +compdef _make_targets make +``` + +The completion cache (`.make_targets_cache`) is gitignored and automatically regenerates when the Makefile changes. + +## Troubleshooting + +**Proto generation fails:** +- Ensure all three protoc plugins are installed (see Optional Tools above) +- Check that `protoc` is in your PATH: `which protoc` + +**Build fails after proto changes:** +- Run `make proto` to regenerate proto files +- Ensure you updated all service implementations for new/changed fields + +**Server won't start:** +- Check if port is already in use: `lsof -i :8081` + +**Bazel build issues:** +- Version is pinned in `.bazelversion`; use `./tool/bazel` or `bazel` with direnv +- Try `bazel shutdown` and rebuild diff --git a/README.md b/README.md index 9e60aebc..b0cb3d70 100644 --- a/README.md +++ b/README.md @@ -1,462 +1,40 @@ # SubmitQueue -## Services +A distributed system for managing code submission workflows. SubmitQueue coordinates the lifecycle of code changes — from submission through validation to landing — using a clean, extensible architecture with pluggable backends. -Submit Queue consists of two main services: - -- **Gateway**: Entry point for external requests (port 8081) -- **Orchestrator**: Coordinates job execution (port 8082) - -## gRPC API - -Each service has its own proto definitions and exposes its own gRPC API: -- **SubmitQueueGateway**: Gateway API with Ping and Land methods (port 8081) -- **SubmitQueueOrchestrator**: Orchestrator API with Ping method (port 8082) - -### Quick Start - -Build and run a service: -```bash -# Using Make (recommended) -make run-gateway - -# Using Go directly -go run example/server/gateway/main.go - -# Using Bazel (with direnv) -bazel run //example/server/gateway:gateway - -# Or without direnv -./tool/bazel run //example/server/gateway:gateway -``` - -Test the service: -```bash -# Using the provided client (in another terminal) -make run-client-gateway MESSAGE="hello" - -# Or using Go directly -go run example/client/gateway/main.go -message "hello" - -# Or using grpcurl -grpcurl -plaintext -d '{"message": "hello"}' localhost:8081 uber.submitqueue.gateway.SubmitQueueGateway/Ping -``` - -For detailed instructions, see [example/README.md](example/README.md). - -## Project Structure - -See [doc/architecture/STRUCTURE.md](doc/architecture/STRUCTURE.md) for a detailed breakdown of the project structure. - -## Architecture - -The project follows clean architecture principles with clear separation of concerns: - -- **Controllers** (`core/controller/`): Pure business logic, independent of transport layer - - Only depend on logger, metrics, and protobuf types - - Example: `PingController` handles ping business logic - -- **Server Adapters** (`example/server/`): gRPC transport layer - - Wrap controllers and implement gRPC service interfaces - - Handle protocol-specific concerns (e.g., `UnimplementedServiceServer`) - -- **Observability**: Built-in logging and metrics - - Structured logging with [Zap](https://github.com/uber-go/zap) - - Metrics collection with [Tally](https://github.com/uber-go/tally) - - Development servers use human-readable console logging - -## Development - -### Prerequisites - -- **Go 1.24 or later** (optional, Bazel manages its own Go toolchain) -- **protoc** and Go plugins (optional, only needed if modifying proto files) -- **grpcurl** (optional, for manual testing) -- **direnv** (recommended, to automatically load `.envrc`) - -**Note**: The project includes `./tool/bazel` (bazelisk wrapper) and `.bazelversion`, so you don't need to install Bazel or Bazelisk separately. - -#### Using direnv (Recommended) - -Install direnv and allow the `.envrc` file: -```bash -# macOS -brew install direnv - -# Add to your shell profile (~/.zshrc or ~/.bashrc) -eval "$(direnv hook zsh)" # or bash, fish, etc. - -# In the project directory -direnv allow -``` - -With direnv enabled, you can use `bazel` directly instead of `./tool/bazel`. - -### Shell Configuration (Optional) - -#### Make Target Auto-Completion (zsh) - -Enable tab-completion for Makefile targets with descriptions: - -1. **Add to `~/.zshrc`:** - ```bash - # Initialize completion system - autoload -Uz compinit - compinit - - # Makefile target completion with caching and help text - function _make_targets() { - local -a targets - local makefile_cache=".make_targets_cache" - - if [[ -f Makefile ]]; then - # Regenerate cache if Makefile is newer - if [[ ! -f $makefile_cache ]] || [[ Makefile -nt $makefile_cache ]]; then - awk -F':.*?## ' '/^[a-zA-Z0-9_-]+:.*?## / {printf "%s:%s\n", $1, $2}' Makefile > $makefile_cache - fi - - # Read cache into targets array - targets=(${(f)"$(<$makefile_cache)"}) - - # If cache has descriptions, use them; otherwise fallback to simple list - if [[ -s $makefile_cache ]] && grep -q ':' $makefile_cache 2>/dev/null; then - _describe 'make targets' targets - else - # Fallback: just target names - awk -F: '/^[a-zA-Z0-9_-]+:/ {print $1}' Makefile > $makefile_cache - targets=(${(f)"$(<$makefile_cache)"}) - _describe 'make targets' targets - fi - fi - } - - compdef _make_targets make - ``` - -2. **Reload your shell:** - ```bash - source ~/.zshrc - ``` - -3. **Try it out:** - ```bash - make # Shows all targets with descriptions - make local- # Shows all local-* targets - make integration # Shows integration test options - ``` - - You'll see: - ``` - build -- Build all services and examples - test -- Run unit tests - local-start -- Start full stack (Gateway + Orchestrator + MySQL) - integration-test -- Run all integration tests (auto-builds binaries) - # ... and more! - ``` - -**Note**: The completion cache (`.make_targets_cache`) is gitignored and automatically regenerates when the Makefile changes. - -Install optional tools: -```bash -# macOS -brew install protobuf grpcurl - -# Install Go plugins (only if you need to regenerate proto files) -go install google.golang.org/protobuf/cmd/protoc-gen-go@latest -go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest -go install go.uber.org/yarpc/encoding/protobuf/protoc-gen-yarpc-go@latest -``` - -**Note**: Proto files are committed to the repository, so you don't need protoc unless you're modifying `.proto` files. - -### Building - -#### Using Make (Recommended) +## Quick Start ```bash # Build everything make build -# Generate proto files (proto files are committed, so this is optional) -make proto - -# Run servers -make run-gateway # Gateway on port 8081 -make run-orchestrator # Orchestrator on port 8082 - -# Run clients -make run-client-gateway MESSAGE="hello" -make run-client-orchestrator MESSAGE="hello" - -# Clean binaries -make clean - -# Clean proto files (not normally needed) -make clean-proto -``` - -#### Using Go directly - -```bash -# Generate proto files (if needed) -make proto - -# Build everything -go build ./... - -# Build example servers -go build -o bin/gateway_server ./example/server/gateway/ -go build -o bin/orchestrator_server ./example/server/orchestrator/ - -# Build clients -go build -o bin/gateway_client ./example/client/gateway/ -go build -o bin/orchestrator_client ./example/client/orchestrator/ - -# Run a server -./bin/gateway_server - -# Run a client (in another terminal) -./bin/gateway_client -message "hello" -``` - -#### Using Bazel - -The project uses **Bzlmod** (not WORKSPACE) for dependency management. Bazel version is pinned at 8.4.1 in `.bazelversion`. - -The project includes `./tool/bazel` which automatically downloads the correct Bazel version. If you're using `direnv`, you can simply use `bazel` instead of `./tool/bazel`. - -```bash -# Build everything (with direnv) -bazel build //... - -# Or without direnv -./tool/bazel build //... - -# Build specific components -bazel build //gateway/protopb -bazel build //example/server/gateway:gateway -bazel build //example/client/gateway:gateway - -# Run a server -bazel run //example/server/gateway:gateway - -# Run a client -bazel run //example/client/gateway:gateway -- -message "hello" - -# Or use the Makefile (recommended) -make build -make run-gateway -``` - -**Note**: -- The repository uses Bzlmod for modern dependency management -- All generated proto files are committed to the repository -- With `direnv` enabled, use `bazel` directly; otherwise use `./tool/bazel` - -### Running Services - -See the [examples directory](example/) for examples of running each service. - -## Development Workflow - -### Modifying Proto Files - -When you make changes to `.proto` files, you need to regenerate the protobuf code. The project generates three types of files for each proto: - -1. `*.pb.go` - Standard protobuf code (protoc-gen-go) -2. `*_grpc.pb.go` - gRPC service code (protoc-gen-go-grpc) -3. `*.pb.yarpc.go` - YARPC service code (protoc-gen-yarpc-go) for Uber's RPC framework - -**Step-by-step process:** - -1. Edit the proto file (e.g., `gateway/proto/gateway.proto`) - -2. Regenerate the protobuf code: - ```bash - make proto - ``` - -3. Update service implementations if you added/changed fields: - ```bash - # For example, if you added a field to PingResponse: - # Edit gateway/core/controller/ping.go to populate the new field - ``` - -4. Update client examples to display new fields: - ```bash - # Edit example/client/gateway/main.go to show the new field - ``` - -5. Rebuild and test: - ```bash - make build - ``` - -**Example: Adding a new field to PingResponse** - -```protobuf -// In gateway/proto/gateway.proto -message PingResponse { - string message = 1; - string service_name = 2; - int64 timestamp = 3; - string hostname = 4; - string new_field = 5; // New field added -} -``` - -After editing the proto: -```bash -# Regenerate proto files -make proto - -# The following files are updated automatically: -# - gateway/protopb/gateway.pb.go -# - gateway/protopb/gateway_grpc.pb.go -# - gateway/protopb/gateway.pb.yarpc.go - -# Now update the controller implementation -# Edit gateway/core/controller/ping.go to populate the new field in the PingResponse -``` - -### Testing - -#### Manual Testing - -1. **Start a server:** - ```bash - make run-gateway - ``` - -2. **Test with the client (in another terminal):** - ```bash - make run-client-gateway MESSAGE="test message" - ``` - -3. **Or use grpcurl:** - ```bash - grpcurl -plaintext -d '{"message": "hello"}' \ - localhost:8081 uber.submitqueue.gateway.SubmitQueueGateway/Ping - ``` - -#### Testing All Services - -```bash -# Terminal 1: Start gateway -make run-gateway - -# Terminal 2: Start orchestrator -make run-orchestrator - -# Terminal 3: Test each service -make run-client-gateway MESSAGE="test gateway" -make run-client-orchestrator MESSAGE="test orchestrator" -``` - -#### Using Bazel for Testing - -```bash -# Run tests (when tests are added) -bazel test //... - -# Or with make +# Run unit tests make test -``` - -### Common Development Tasks - -#### Adding a New RPC Method - -1. **Update the proto file:** - ```protobuf - // In gateway/proto/gateway.proto - service SubmitQueueGateway { - rpc Ping(PingRequest) returns (PingResponse) {} - rpc NewMethod(NewRequest) returns (NewResponse) {} // Add new method - } - - message NewRequest { ... } - message NewResponse { ... } - ``` - -2. **Regenerate proto files:** - ```bash - make proto - ``` -3. **Implement the method in the controller:** - ```go - // In gateway/core/controller/ (create a new controller file) - type NewController struct { - logger *zap.Logger - metricsScope tally.Scope - } +# Start full stack locally (Gateway + Orchestrator + MySQL via Docker Compose) +make local-start - func NewNewController(logger *zap.Logger, scope tally.Scope) *NewController { - if logger == nil { - logger = zap.NewNop() - } - if scope == nil { - scope = tally.NoopScope - } - return &NewController{logger: logger, metricsScope: scope} - } - - func (c *NewController) NewMethod(ctx context.Context, req *pb.NewRequest) (*pb.NewResponse, error) { - // Business logic here - c.logger.Info("new method called") - c.metricsScope.Counter("new_method_total").Inc(1) - // ... - } - ``` - -4. **Create server wrapper in example:** - ```go - // In example/server/gateway/main.go - // Add method delegation to GatewayServer struct: - func (s *GatewayServer) NewMethod(ctx context.Context, req *pb.NewRequest) (*pb.NewResponse, error) { - return s.newController.NewMethod(ctx, req) - } - ``` - -5. **Update clients to call the new method** +# Test with grpcurl +grpcurl -plaintext -d '{"message": "hello"}' localhost:8081 uber.submitqueue.gateway.SubmitQueueGateway/Ping -6. **Rebuild and test:** - ```bash - make build - ``` +# Stop services +make local-stop +``` -#### Cleaning Up +See [example/README.md](example/README.md) for more examples including running individual services and clients. -```bash -# Remove built binaries -make clean +## Architecture -# Remove generated proto files (regenerate with 'make proto') -make clean-proto +See [CLAUDE.md](CLAUDE.md) for the full architecture guide, including project layout, controller patterns, entity conventions, extension system, and development workflows. -# Clean Bazel cache -bazel clean -``` +## Contributing -### Troubleshooting +See [CONTRIBUTING.md](CONTRIBUTING.md) for how to get started, development workflow, code style, and testing. -**Proto generation fails:** -- Ensure all three protoc plugins are installed (see Prerequisites) -- Check that `protoc` is in your PATH: `which protoc` -- Check plugin versions: `protoc-gen-go --version` +## Code of Conduct -**Build fails after proto changes:** -- Run `make proto` to regenerate proto files -- Ensure you updated all service implementations for new/changed fields -- Check import paths in generated files match your module path +This project is governed by the [Contributor Covenant Code of Conduct](CODE_OF_CONDUCT.md). By participating, you are expected to uphold this code. -**Server won't start:** -- Check if port is already in use: `lsof -i :8081` -- Kill existing process: `lsof -ti tcp:8081 | xargs kill -9` +## License -**Bazel build issues:** -- Bazel version is pinned to 8.4.1 in `.bazelversion` -- With `direnv`, you can use `bazel` directly; otherwise use `./tool/bazel` -- Try `bazel shutdown` (or `./tool/bazel shutdown`) and rebuild -- The wrapper automatically downloads the correct Bazel version +[Apache 2.0](LICENSE) From d0b6866b2fa9ab6efb0b77ee35cb5331b87433a4 Mon Sep 17 00:00:00 2001 From: Preetam Dwivedi Date: Thu, 5 Mar 2026 19:17:21 -0800 Subject: [PATCH 2/5] Clean up CONTRIBUTING.md and README.md Remove redundant Proto Changes and Adding Extensions sections from CONTRIBUTING.md (covered by CLAUDE.md). Move shell config to doc/howto/SHELL.md. Reorder sections so developer setup comes before Code of Conduct/License. Remove Code of Conduct and License from README since they live in CONTRIBUTING.md. --- CONTRIBUTING.md | 97 ++++++++++------------------------------------ README.md | 8 ---- doc/howto/SHELL.md | 45 +++++++++++++++++++++ 3 files changed, 65 insertions(+), 85 deletions(-) create mode 100644 doc/howto/SHELL.md diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 0369be3b..19283972 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -6,7 +6,6 @@ Thank you for your interest in contributing to SubmitQueue! This document provid ### Prerequisites -- **Go 1.24 or later** (optional — Bazel manages its own Go toolchain) - **Docker** and **Docker Compose** (for integration and e2e tests) - **direnv** (recommended — automatically loads `.envrc` so you can use `bazel` directly) @@ -43,7 +42,7 @@ go install go.uber.org/yarpc/encoding/protobuf/protoc-gen-yarpc-go@latest ## Development Workflow 1. Fork the repository and create a feature branch from `main`. -2. Make your changes, following the code style and conventions below. +2. Make your changes, following the project conventions. 3. Add or update tests as appropriate. 4. Run `make gazelle` if you added or removed Go files. 5. Run `make test` to ensure unit tests pass. @@ -52,13 +51,7 @@ go install go.uber.org/yarpc/encoding/protobuf/protoc-gen-yarpc-go@latest ## Code Style -The project follows specific conventions for code style, error handling, logging, and entity design. See [CLAUDE.md](CLAUDE.md) for the full set of conventions, including: - -- Structured logging with `zap.SugaredLogger` -- Interfaces for behavior, structs for data -- Value types over pointers -- Error classification with `core/errs` -- Immutable entities and optimistic locking +See [CLAUDE.md](CLAUDE.md) for code style conventions. ## Testing @@ -74,83 +67,25 @@ make e2e-test # End-to-end tests All tests use table-driven style with `t.Run` subtests. Use `assert`/`require` from testify. -## Proto Changes - -When modifying `.proto` files: - -1. Edit the proto file in `{service}/proto/`. -2. Run `make proto` to regenerate `*.pb.go`, `*_grpc.pb.go`, and `*.pb.yarpc.go`. -3. Update controllers and clients as needed. -4. Commit all generated files. - -See [CLAUDE.md](CLAUDE.md) for detailed workflows (adding RPC methods, queue controllers, extensions, entities). +## Pull Request Guidelines -## Adding Extensions +- Keep PRs focused on a single change. +- Include tests for new functionality. +- Ensure all existing tests pass (`bazel test //...`). +- Follow the existing code style and patterns (see [CLAUDE.md](CLAUDE.md) for detailed conventions). +- Fill out the PR template with a description, motivation, and test plan. -Extensions follow a vendor-agnostic interface pattern: - -1. Define the interface at `extension/{ext}/`. -2. Add implementations at `extension/{ext}/{impl}/`. -3. Include a factory interface for dependency injection. -4. Add `BUILD.bazel`, tests, and README. +## Code Review -See [CLAUDE.md](CLAUDE.md) for the full extension guide and mock setup instructions. +All submissions require review before merging. We use GitHub pull requests for this purpose. A maintainer will review your PR and may request changes. ## Reporting Issues -Use the [issue templates](.github/ISSUE_TEMPLATE/) when filing bugs or requesting features. Blank issues are disabled — please choose the appropriate template. - -## Code Review - -- All submissions require review before merging. -- Maintainers may request changes or suggest improvements. -- Keep PRs focused — one logical change per PR. +Use GitHub Issues to report bugs or request features. Please check existing issues before creating a new one. ## Shell Configuration (Optional) -### Using direnv (Recommended) - -```bash -brew install direnv - -# Add to ~/.zshrc or ~/.bashrc -eval "$(direnv hook zsh)" # or bash, fish, etc. - -# In the project directory -direnv allow -``` - -### Make Target Auto-Completion (zsh) - -Add to `~/.zshrc` for tab-completion of Makefile targets with descriptions: - -```bash -autoload -Uz compinit -compinit - -function _make_targets() { - local -a targets - local makefile_cache=".make_targets_cache" - - if [[ -f Makefile ]]; then - if [[ ! -f $makefile_cache ]] || [[ Makefile -nt $makefile_cache ]]; then - awk -F':.*?## ' '/^[a-zA-Z0-9_-]+:.*?## / {printf "%s:%s\n", $1, $2}' Makefile > $makefile_cache - fi - targets=(${(f)"$(<$makefile_cache)"}) - if [[ -s $makefile_cache ]] && grep -q ':' $makefile_cache 2>/dev/null; then - _describe 'make targets' targets - else - awk -F: '/^[a-zA-Z0-9_-]+:/ {print $1}' Makefile > $makefile_cache - targets=(${(f)"$(<$makefile_cache)"}) - _describe 'make targets' targets - fi - fi -} - -compdef _make_targets make -``` - -The completion cache (`.make_targets_cache`) is gitignored and automatically regenerates when the Makefile changes. +See [doc/howto/SHELL.md](doc/howto/SHELL.md) for direnv setup and Make target auto-completion. ## Troubleshooting @@ -168,3 +103,11 @@ The completion cache (`.make_targets_cache`) is gitignored and automatically reg **Bazel build issues:** - Version is pinned in `.bazelversion`; use `./tool/bazel` or `bazel` with direnv - Try `bazel shutdown` and rebuild + +## Code of Conduct + +This project adheres to the [Contributor Covenant Code of Conduct](CODE_OF_CONDUCT.md). By participating, you are expected to uphold this code. + +## License + +By contributing to SubmitQueue, you agree that your contributions will be licensed under the [Apache License 2.0](LICENSE). diff --git a/README.md b/README.md index b0cb3d70..861ed6ac 100644 --- a/README.md +++ b/README.md @@ -30,11 +30,3 @@ See [CLAUDE.md](CLAUDE.md) for the full architecture guide, including project la ## Contributing See [CONTRIBUTING.md](CONTRIBUTING.md) for how to get started, development workflow, code style, and testing. - -## Code of Conduct - -This project is governed by the [Contributor Covenant Code of Conduct](CODE_OF_CONDUCT.md). By participating, you are expected to uphold this code. - -## License - -[Apache 2.0](LICENSE) diff --git a/doc/howto/SHELL.md b/doc/howto/SHELL.md new file mode 100644 index 00000000..98aa0d64 --- /dev/null +++ b/doc/howto/SHELL.md @@ -0,0 +1,45 @@ +# Shell Configuration + +## Using direnv (Recommended) + +```bash +brew install direnv + +# Add to ~/.zshrc or ~/.bashrc +eval "$(direnv hook zsh)" # or bash, fish, etc. + +# In the project directory +direnv allow +``` + +## Make Target Auto-Completion (zsh) + +Add to `~/.zshrc` for tab-completion of Makefile targets with descriptions: + +```bash +autoload -Uz compinit +compinit + +function _make_targets() { + local -a targets + local makefile_cache=".make_targets_cache" + + if [[ -f Makefile ]]; then + if [[ ! -f $makefile_cache ]] || [[ Makefile -nt $makefile_cache ]]; then + awk -F':.*?## ' '/^[a-zA-Z0-9_-]+:.*?## / {printf "%s:%s\n", $1, $2}' Makefile > $makefile_cache + fi + targets=(${(f)"$(<$makefile_cache)"}) + if [[ -s $makefile_cache ]] && grep -q ':' $makefile_cache 2>/dev/null; then + _describe 'make targets' targets + else + awk -F: '/^[a-zA-Z0-9_-]+:/ {print $1}' Makefile > $makefile_cache + targets=(${(f)"$(<$makefile_cache)"}) + _describe 'make targets' targets + fi + fi +} + +compdef _make_targets make +``` + +The completion cache (`.make_targets_cache`) is gitignored and automatically regenerates when the Makefile changes. From c3e28a90637b379447a625307dcb105180f85b25 Mon Sep 17 00:00:00 2001 From: Preetam Dwivedi Date: Thu, 5 Mar 2026 19:23:41 -0800 Subject: [PATCH 3/5] Add project overview from EuroSys paper and restructure docs Add high-level SubmitQueue overview to README synthesized from the EuroSys '19 paper. Move development setup and troubleshooting from CONTRIBUTING.md to doc/howto/DEVELOPMENT.md. Slim down CONTRIBUTING.md to focus on contribution guidelines only. --- CONTRIBUTING.md | 58 +++------------------------------------- README.md | 6 ++++- doc/howto/DEVELOPMENT.md | 53 ++++++++++++++++++++++++++++++++++++ 3 files changed, 61 insertions(+), 56 deletions(-) create mode 100644 doc/howto/DEVELOPMENT.md diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 19283972..e58aedf9 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,43 +1,10 @@ # Contributing to SubmitQueue -Thank you for your interest in contributing to SubmitQueue! This document provides guidelines and instructions for contributing. +Thank you for your interest in contributing to SubmitQueue! ## Getting Started -### Prerequisites - -- **Docker** and **Docker Compose** (for integration and e2e tests) -- **direnv** (recommended — automatically loads `.envrc` so you can use `bazel` directly) - -The project includes `./tool/bazel` (Bazelisk wrapper) and `.bazelversion`, so you don't need to install Bazel separately. - -### Clone and Build - -```bash -git clone https://github.com/uber/submitqueue.git -cd submitqueue - -# Optional: allow direnv -direnv allow - -# Build everything -make build - -# Run unit tests -make test -``` - -### Optional Tools - -```bash -# macOS -brew install protobuf grpcurl direnv - -# Go protoc plugins (only if modifying .proto files) -go install google.golang.org/protobuf/cmd/protoc-gen-go@latest -go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest -go install go.uber.org/yarpc/encoding/protobuf/protoc-gen-yarpc-go@latest -``` +See [doc/howto/DEVELOPMENT.md](doc/howto/DEVELOPMENT.md) for prerequisites, setup, and troubleshooting. ## Development Workflow @@ -55,7 +22,7 @@ See [CLAUDE.md](CLAUDE.md) for code style conventions. ## Testing -See [doc/howto/TESTING.md](doc/howto/TESTING.md) for the full testing guide, including integration and end-to-end tests. +See [doc/howto/TESTING.md](doc/howto/TESTING.md) for the full testing guide. Key commands: @@ -65,8 +32,6 @@ make integration-test # Integration tests (Docker-based, auto-builds binaries make e2e-test # End-to-end tests ``` -All tests use table-driven style with `t.Run` subtests. Use `assert`/`require` from testify. - ## Pull Request Guidelines - Keep PRs focused on a single change. @@ -87,23 +52,6 @@ Use GitHub Issues to report bugs or request features. Please check existing issu See [doc/howto/SHELL.md](doc/howto/SHELL.md) for direnv setup and Make target auto-completion. -## Troubleshooting - -**Proto generation fails:** -- Ensure all three protoc plugins are installed (see Optional Tools above) -- Check that `protoc` is in your PATH: `which protoc` - -**Build fails after proto changes:** -- Run `make proto` to regenerate proto files -- Ensure you updated all service implementations for new/changed fields - -**Server won't start:** -- Check if port is already in use: `lsof -i :8081` - -**Bazel build issues:** -- Version is pinned in `.bazelversion`; use `./tool/bazel` or `bazel` with direnv -- Try `bazel shutdown` and rebuild - ## Code of Conduct This project adheres to the [Contributor Covenant Code of Conduct](CODE_OF_CONDUCT.md). By participating, you are expected to uphold this code. diff --git a/README.md b/README.md index 861ed6ac..3f38775b 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,10 @@ # SubmitQueue -A distributed system for managing code submission workflows. SubmitQueue coordinates the lifecycle of code changes — from submission through validation to landing — using a clean, extensible architecture with pluggable backends. +SubmitQueue is a change management system that guarantees an always-green master branch at scale. It addresses a critical challenge in large monorepo environments: as commit volume grows, concurrent changes can introduce conflicts and broken builds that are difficult to detect through traditional CI alone. + +SubmitQueue solves this by serializing and validating changes before they land on master, ensuring that every commit point passes all build steps — compilation, unit tests, and integration tests. The system is designed for high throughput and low turnaround time, handling thousands of daily commits without becoming a bottleneck. + +For more details, see the EuroSys '19 paper: [Keeping Master Green at Scale](https://dl.acm.org/doi/10.1145/3302424.3303970). ## Quick Start diff --git a/doc/howto/DEVELOPMENT.md b/doc/howto/DEVELOPMENT.md new file mode 100644 index 00000000..8d37037a --- /dev/null +++ b/doc/howto/DEVELOPMENT.md @@ -0,0 +1,53 @@ +# Development + +## Prerequisites + +- **Docker** and **Docker Compose** (for integration and e2e tests) +- **direnv** (recommended — automatically loads `.envrc` so you can use `bazel` directly) + +The project includes `./tool/bazel` (Bazelisk wrapper) and `.bazelversion`, so you don't need to install Bazel or Go separately. + +## Clone and Build + +```bash +git clone https://github.com/uber/submitqueue.git +cd submitqueue + +# Optional: allow direnv +direnv allow + +# Build everything +make build + +# Run unit tests +make test +``` + +## Optional Tools + +```bash +# macOS +brew install protobuf grpcurl direnv + +# Go protoc plugins (only if modifying .proto files) +go install google.golang.org/protobuf/cmd/protoc-gen-go@latest +go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest +go install go.uber.org/yarpc/encoding/protobuf/protoc-gen-yarpc-go@latest +``` + +## Troubleshooting + +**Proto generation fails:** +- Ensure all three protoc plugins are installed (see Optional Tools above) +- Check that `protoc` is in your PATH: `which protoc` + +**Build fails after proto changes:** +- Run `make proto` to regenerate proto files +- Ensure you updated all service implementations for new/changed fields + +**Server won't start:** +- Check if port is already in use: `lsof -i :8081` + +**Bazel build issues:** +- Version is pinned in `.bazelversion`; use `./tool/bazel` or `bazel` with direnv +- Try `bazel shutdown` and rebuild From 0145c066d103ce4741008adbce7dbaf9330a5de4 Mon Sep 17 00:00:00 2001 From: Preetam Dwivedi Date: Thu, 5 Mar 2026 19:32:43 -0800 Subject: [PATCH 4/5] Refine README overview and clean up doc structure Simplify README overview to a single paragraph, use 'speculative merge queue' instead of 'change management system', remove paper reference and specific build step assumptions. Rename Architecture to Developer Guide. Move shell config from CONTRIBUTING.md to DEVELOPMENT.md, add links to TESTING.md and SHELL.md from DEVELOPMENT.md, fix issues link in CONTRIBUTING.md. --- CONTRIBUTING.md | 6 +----- README.md | 10 +++------- doc/howto/DEVELOPMENT.md | 8 ++++++++ 3 files changed, 12 insertions(+), 12 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index e58aedf9..27b2541d 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -46,11 +46,7 @@ All submissions require review before merging. We use GitHub pull requests for t ## Reporting Issues -Use GitHub Issues to report bugs or request features. Please check existing issues before creating a new one. - -## Shell Configuration (Optional) - -See [doc/howto/SHELL.md](doc/howto/SHELL.md) for direnv setup and Make target auto-completion. +Use [GitHub Issues](https://github.com/uber/submitqueue/issues) to report bugs or request features. Please use the provided [issue templates](.github/ISSUE_TEMPLATE/) and check existing issues before creating a new one. ## Code of Conduct diff --git a/README.md b/README.md index 3f38775b..fcb0764c 100644 --- a/README.md +++ b/README.md @@ -1,10 +1,6 @@ # SubmitQueue -SubmitQueue is a change management system that guarantees an always-green master branch at scale. It addresses a critical challenge in large monorepo environments: as commit volume grows, concurrent changes can introduce conflicts and broken builds that are difficult to detect through traditional CI alone. - -SubmitQueue solves this by serializing and validating changes before they land on master, ensuring that every commit point passes all build steps — compilation, unit tests, and integration tests. The system is designed for high throughput and low turnaround time, handling thousands of daily commits without becoming a bottleneck. - -For more details, see the EuroSys '19 paper: [Keeping Master Green at Scale](https://dl.acm.org/doi/10.1145/3302424.3303970). +SubmitQueue is a speculative merge queue that keeps the main branch green at scale. In large monorepo environments, concurrent changes can introduce conflicts and broken builds. SubmitQueue solves this by serializing and validating changes before they land on main, ensuring that every commit point passes defined validations. ## Quick Start @@ -27,9 +23,9 @@ make local-stop See [example/README.md](example/README.md) for more examples including running individual services and clients. -## Architecture +## Developer Guide -See [CLAUDE.md](CLAUDE.md) for the full architecture guide, including project layout, controller patterns, entity conventions, extension system, and development workflows. +See [CLAUDE.md](CLAUDE.md) for the full developer guide, including project layout, controller patterns, entity conventions, extension system, and development workflows. ## Contributing diff --git a/doc/howto/DEVELOPMENT.md b/doc/howto/DEVELOPMENT.md index 8d37037a..b3617083 100644 --- a/doc/howto/DEVELOPMENT.md +++ b/doc/howto/DEVELOPMENT.md @@ -51,3 +51,11 @@ go install go.uber.org/yarpc/encoding/protobuf/protoc-gen-yarpc-go@latest **Bazel build issues:** - Version is pinned in `.bazelversion`; use `./tool/bazel` or `bazel` with direnv - Try `bazel shutdown` and rebuild + +## Testing + +See [TESTING.md](TESTING.md) for the full testing guide, including integration and end-to-end tests. + +## Shell Configuration (Optional) + +See [SHELL.md](SHELL.md) for direnv setup and Make target auto-completion. From b10e5f11201253b9c29ffee9260045cfe737cf75 Mon Sep 17 00:00:00 2001 From: Preetam Dwivedi Date: Thu, 5 Mar 2026 19:44:10 -0800 Subject: [PATCH 5/5] Remove Code Style duplication, merge SHELL.md into DEVELOPMENT.md --- CONTRIBUTING.md | 56 +++++++------- README.md | 31 +++++++- doc/howto/DEVELOPMENT.md | 159 +++++++++++++++++++++++++++++++++++++-- doc/howto/SHELL.md | 45 ----------- 4 files changed, 207 insertions(+), 84 deletions(-) delete mode 100644 doc/howto/SHELL.md diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 27b2541d..94352628 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,43 +1,40 @@ # Contributing to SubmitQueue -Thank you for your interest in contributing to SubmitQueue! +Thank you for your interest in contributing to SubmitQueue! Whether you are reporting a bug, suggesting a feature, improving documentation, or writing code, your contribution is welcome. ## Getting Started -See [doc/howto/DEVELOPMENT.md](doc/howto/DEVELOPMENT.md) for prerequisites, setup, and troubleshooting. +1. Read the [Development Setup](doc/howto/DEVELOPMENT.md) guide for prerequisites, building, and running tests. +2. Review the [Architecture Guide](CLAUDE.md) to understand project layout, conventions, and code style. +3. Check the [Testing Guide](doc/howto/TESTING.md) for testing patterns and requirements. ## Development Workflow -1. Fork the repository and create a feature branch from `main`. -2. Make your changes, following the project conventions. -3. Add or update tests as appropriate. -4. Run `make gazelle` if you added or removed Go files. -5. Run `make test` to ensure unit tests pass. -6. Commit with a clear, descriptive message. -7. Open a pull request against `main`. - -## Code Style - -See [CLAUDE.md](CLAUDE.md) for code style conventions. - -## Testing - -See [doc/howto/TESTING.md](doc/howto/TESTING.md) for the full testing guide. - -Key commands: - -```bash -make test # Unit tests -make integration-test # Integration tests (Docker-based, auto-builds binaries) -make e2e-test # End-to-end tests -``` +1. Fork the repository and clone your fork. +2. Create a feature branch from `main`: + ```bash + git checkout -b yourname/short-description + ``` + Use branch naming like `yourname/short-description` or `fix/issue-123`. +3. Make your changes, following the project conventions. +4. Add or update tests as appropriate. +5. Run `make gazelle` if you added or removed Go files. +6. Run `make test` to ensure unit tests pass. +7. Commit with a clear, descriptive message. +8. Push to your fork: + ```bash + git push origin yourname/short-description + ``` +9. Open a pull request against `main`. ## Pull Request Guidelines - Keep PRs focused on a single change. +- Reference related issues: use `Closes #123` for fixes or `Part of #123` for incremental work. - Include tests for new functionality. -- Ensure all existing tests pass (`bazel test //...`). -- Follow the existing code style and patterns (see [CLAUDE.md](CLAUDE.md) for detailed conventions). +- Ensure all existing tests pass (`make test`). +- Ensure CI passes before requesting review. +- Follow the existing code style and patterns described in the [Architecture Guide](CLAUDE.md). - Fill out the PR template with a description, motivation, and test plan. ## Code Review @@ -48,6 +45,11 @@ All submissions require review before merging. We use GitHub pull requests for t Use [GitHub Issues](https://github.com/uber/submitqueue/issues) to report bugs or request features. Please use the provided [issue templates](.github/ISSUE_TEMPLATE/) and check existing issues before creating a new one. +## Communication + +- **Bug reports and feature requests** — [GitHub Issues](https://github.com/uber/submitqueue/issues) +- **Questions and discussions** — open an issue with the `question` label + ## Code of Conduct This project adheres to the [Contributor Covenant Code of Conduct](CODE_OF_CONDUCT.md). By participating, you are expected to uphold this code. diff --git a/README.md b/README.md index fcb0764c..5e128dd7 100644 --- a/README.md +++ b/README.md @@ -1,9 +1,17 @@ # SubmitQueue -SubmitQueue is a speculative merge queue that keeps the main branch green at scale. In large monorepo environments, concurrent changes can introduce conflicts and broken builds. SubmitQueue solves this by serializing and validating changes before they land on main, ensuring that every commit point passes defined validations. +[![CI](https://github.com/uber/submitqueue/actions/workflows/ci.yml/badge.svg)](https://github.com/uber/submitqueue/actions/workflows/ci.yml) +[![Go Version](https://img.shields.io/github/go-mod/go-version/uber/submitqueue)](go.mod) +[![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](LICENSE) + +SubmitQueue is a high-performance speculative merge queue that keeps your trunk consistently green at scale. Rather than validating changes one at a time, SubmitQueue speculatively rebases and validates multiple changes in parallel against predicted future states of HEAD. When validations pass, changes land automatically. When they fail, SubmitQueue isolates the offending change and retries the rest — all without human intervention. + +Designed for large monorepos and fast-moving teams where concurrent changes can introduce subtle conflicts and destabilize builds. ## Quick Start +Requires Docker and Docker Compose. See [Development Setup](doc/howto/DEVELOPMENT.md) for full prerequisites. + ```bash # Build everything make build @@ -23,10 +31,25 @@ make local-stop See [example/README.md](example/README.md) for more examples including running individual services and clients. -## Developer Guide +## Documentation + +| Document | Description | +|----------|-------------| +| [Development Setup](doc/howto/DEVELOPMENT.md) | Prerequisites, build, environment, IDE setup | +| [Contributing](CONTRIBUTING.md) | How to contribute, workflow, guidelines | +| [Testing Guide](doc/howto/TESTING.md) | Unit, integration, and E2E testing patterns | +| [Architecture Guide](CLAUDE.md) | Project layout, patterns, conventions | +| [Examples](example/README.md) | Running services, clients, API reference | +| [RFCs](doc/rfc/index.md) | Design documents and proposals | -See [CLAUDE.md](CLAUDE.md) for the full developer guide, including project layout, controller patterns, entity conventions, extension system, and development workflows. +## Project Status + +SubmitQueue is under active development. We welcome contributions and feedback. ## Contributing -See [CONTRIBUTING.md](CONTRIBUTING.md) for how to get started, development workflow, code style, and testing. +See [CONTRIBUTING.md](CONTRIBUTING.md) for how to get started. + +## License + +Licensed under the [Apache License 2.0](LICENSE). diff --git a/doc/howto/DEVELOPMENT.md b/doc/howto/DEVELOPMENT.md index b3617083..76451154 100644 --- a/doc/howto/DEVELOPMENT.md +++ b/doc/howto/DEVELOPMENT.md @@ -2,10 +2,36 @@ ## Prerequisites -- **Docker** and **Docker Compose** (for integration and e2e tests) -- **direnv** (recommended — automatically loads `.envrc` so you can use `bazel` directly) +- **Go 1.24+** — needed for `gopls`, `go mod`, and installing protoc plugins. Download from [go.dev/dl](https://go.dev/dl/). Note: Bazel manages its own Go toolchain for builds, but a local Go installation is required for editor tooling and dependency management. +- **Docker** and **Docker Compose** — for integration and e2e tests, and for running services locally. +- **direnv** (recommended) — automatically loads `.envrc` so you can use `bazel` directly instead of `./tool/bazel`. -The project includes `./tool/bazel` (Bazelisk wrapper) and `.bazelversion`, so you don't need to install Bazel or Go separately. +The project includes `./tool/bazel` (Bazelisk wrapper) and `.bazelversion`, so you don't need to install Bazel separately. Bazel manages its own Go toolchain for building and testing. + +### Setting up direnv + +```bash +brew install direnv +``` + +Add the hook for your shell: + +```bash +# zsh — add to ~/.zshrc +eval "$(direnv hook zsh)" + +# bash — add to ~/.bashrc +eval "$(direnv hook bash)" + +# fish — add to ~/.config/fish/config.fish +direnv hook fish | source +``` + +Then allow it in the project directory: + +```bash +direnv allow +``` ## Clone and Build @@ -23,11 +49,44 @@ make build make test ``` +## Try It Locally + +After building, start the full stack to confirm everything works end to end: + +```bash +# 1. Confirm Docker is running +docker ps + +# 2. Start the full stack +make local-start + +# 3. Check services are up (Gateway on :8081, Orchestrator on :8082) +make local-ps + +# 4. Test Gateway with grpcurl +grpcurl -plaintext -d '{"message": "hello"}' localhost:8081 uber.submitqueue.gateway.SubmitQueueGateway/Ping + +# 5. Stop services +make local-stop +``` + +If any step fails, see [Troubleshooting](#troubleshooting) below. + +## IDE Setup + +### VS Code + +Install the [Go extension](https://marketplace.visualstudio.com/items?itemName=golang.Go), which uses `gopls` for code intelligence. It works with the project's `go.mod` out of the box. + +### GoLand / IntelliJ + +GoLand works with Go modules automatically. Open the project root and GoLand will detect `go.mod`. + ## Optional Tools ```bash # macOS -brew install protobuf grpcurl direnv +brew install protobuf grpcurl # Go protoc plugins (only if modifying .proto files) go install google.golang.org/protobuf/cmd/protoc-gen-go@latest @@ -35,6 +94,47 @@ go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest go install go.uber.org/yarpc/encoding/protobuf/protoc-gen-yarpc-go@latest ``` +## Common Make Targets + +| Target | Description | +|--------|-------------| +| `make build` | Build all services | +| `make test` | Run unit tests | +| `make integration-test` | Run all integration tests (Docker-based) | +| `make e2e-test` | Run end-to-end tests | +| `make proto` | Regenerate protobuf files | +| `make gazelle` | Update BUILD.bazel files | +| `make local-start` | Start full stack (Gateway + Orchestrator + MySQL) | +| `make local-ps` | Show running containers and ports | +| `make local-logs` | View logs from all services | +| `make local-stop` | Stop all services | +| `make clean` | Clean generated files and binaries | +| `make help` | Show all available targets with descriptions | + +## Running Specific Tests + +```bash +# Run tests for a single package +bazel test //gateway/controller:controller_test + +# Run a single test function +bazel test //gateway/controller:controller_test --test_filter=TestLand + +# Run Gateway integration tests only +make integration-test-gateway + +# Run Orchestrator integration tests only +make integration-test-orchestrator + +# Run extension integration tests only +make integration-test-extensions + +# Run unit tests without cache +make test-no-cache +``` + +See [TESTING.md](TESTING.md) for the full testing guide, including integration and end-to-end test patterns. + ## Troubleshooting **Proto generation fails:** @@ -52,10 +152,53 @@ go install go.uber.org/yarpc/encoding/protobuf/protoc-gen-yarpc-go@latest - Version is pinned in `.bazelversion`; use `./tool/bazel` or `bazel` with direnv - Try `bazel shutdown` and rebuild -## Testing +**`gopls` or `go mod tidy` errors:** +- Run `go mod download` to fetch all dependencies +- Check that your Go version matches what's in `go.mod` (currently Go 1.24) +- If using VS Code, restart the Go language server: `Ctrl+Shift+P` > "Go: Restart Language Server" + +## Shell Auto-Completion + +### zsh + +Add to `~/.zshrc` for tab-completion of Makefile targets with descriptions: + +```bash +autoload -Uz compinit +compinit + +function _make_targets() { + local -a targets + local makefile_cache=".make_targets_cache" + + if [[ -f Makefile ]]; then + if [[ ! -f $makefile_cache ]] || [[ Makefile -nt $makefile_cache ]]; then + awk -F':.*?## ' '/^[a-zA-Z0-9_-]+:.*?## / {printf "%s:%s\n", $1, $2}' Makefile > $makefile_cache + fi + targets=(${(f)"$(<$makefile_cache)"}) + if [[ -s $makefile_cache ]] && grep -q ':' $makefile_cache 2>/dev/null; then + _describe 'make targets' targets + else + awk -F: '/^[a-zA-Z0-9_-]+:/ {print $1}' Makefile > $makefile_cache + targets=(${(f)"$(<$makefile_cache)"}) + _describe 'make targets' targets + fi + fi +} + +compdef _make_targets make +``` + +The completion cache (`.make_targets_cache`) is gitignored and automatically regenerates when the Makefile changes. + +### bash -See [TESTING.md](TESTING.md) for the full testing guide, including integration and end-to-end tests. +If you use `bash-completion`, Makefile target completion typically works out of the box. Otherwise, add to `~/.bashrc`: + +```bash +complete -W "\$(grep -oE '^[a-zA-Z0-9_-]+:' Makefile | sed 's/://')" make +``` -## Shell Configuration (Optional) +### Universal alternative -See [SHELL.md](SHELL.md) for direnv setup and Make target auto-completion. +Run `make help` to see all available targets and their descriptions at any time. diff --git a/doc/howto/SHELL.md b/doc/howto/SHELL.md deleted file mode 100644 index 98aa0d64..00000000 --- a/doc/howto/SHELL.md +++ /dev/null @@ -1,45 +0,0 @@ -# Shell Configuration - -## Using direnv (Recommended) - -```bash -brew install direnv - -# Add to ~/.zshrc or ~/.bashrc -eval "$(direnv hook zsh)" # or bash, fish, etc. - -# In the project directory -direnv allow -``` - -## Make Target Auto-Completion (zsh) - -Add to `~/.zshrc` for tab-completion of Makefile targets with descriptions: - -```bash -autoload -Uz compinit -compinit - -function _make_targets() { - local -a targets - local makefile_cache=".make_targets_cache" - - if [[ -f Makefile ]]; then - if [[ ! -f $makefile_cache ]] || [[ Makefile -nt $makefile_cache ]]; then - awk -F':.*?## ' '/^[a-zA-Z0-9_-]+:.*?## / {printf "%s:%s\n", $1, $2}' Makefile > $makefile_cache - fi - targets=(${(f)"$(<$makefile_cache)"}) - if [[ -s $makefile_cache ]] && grep -q ':' $makefile_cache 2>/dev/null; then - _describe 'make targets' targets - else - awk -F: '/^[a-zA-Z0-9_-]+:/ {print $1}' Makefile > $makefile_cache - targets=(${(f)"$(<$makefile_cache)"}) - _describe 'make targets' targets - fi - fi -} - -compdef _make_targets make -``` - -The completion cache (`.make_targets_cache`) is gitignored and automatically regenerates when the Makefile changes.