Skip to content

Commit 80a4041

Browse files
behinddwallsbehinddwalls
committed
chore(claude): add CLAUDE instructions for repo
1 parent 2fe0918 commit 80a4041

1 file changed

Lines changed: 393 additions & 0 deletions

File tree

‎CLAUDE.md‎

Lines changed: 393 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,393 @@
1+
# SubmitQueue Repository Guide for Claude
2+
3+
## Overview
4+
5+
SubmitQueue is a distributed system for managing code submission workflows. The project follows clean architecture principles with three main services:
6+
7+
- **Gateway** (port 8081): Entry point for external requests
8+
- **Orchestrator** (port 8082): Coordinates job execution
9+
- **Speculator** (port 8083): Performs speculative builds
10+
11+
## Build System
12+
13+
### Bazel with Bzlmod
14+
15+
This repository uses **Bazel 8.4.1** with **Bzlmod** (NOT WORKSPACE) for dependency management.
16+
17+
- **Version pinning**: `.bazelversion` pins Bazel to 8.4.1
18+
- **Dependencies**: Managed in `MODULE.bazel` (NOT in a WORKSPACE file)
19+
- **Go version**: 1.24.5 (defined in MODULE.bazel)
20+
- **Bazel wrapper**: `./tools/bazel` (Python-based Bazelisk wrapper)
21+
- **direnv**: When enabled via `.envrc`, use `bazel` directly; otherwise use `./tools/bazel`
22+
23+
### Key Bazel Commands
24+
25+
```bash
26+
# Build everything
27+
bazel build //...
28+
29+
# Build specific service
30+
bazel build //gateway/protopb
31+
bazel build //examples/server/gateway:gateway
32+
33+
# Run a service
34+
bazel run //examples/server/gateway:gateway
35+
36+
# Run tests
37+
bazel test //...
38+
```
39+
40+
### Go Module
41+
42+
The project has both `go.mod` (for Go dependencies) and `MODULE.bazel` (for Bazel). Dependencies are:
43+
- **YARPC**: Uber's RPC framework
44+
- **gRPC**: Standard gRPC
45+
- **Zap**: Structured logging
46+
- **Tally**: Metrics collection
47+
- **Protobuf**: Protocol buffers
48+
49+
## Architecture
50+
51+
### Service Structure
52+
53+
Each service follows the same layout:
54+
55+
```
56+
<service>/
57+
├── controller/ # Business logic (pure, transport-agnostic)
58+
│ ├── BUILD.bazel
59+
│ └── *.go # Controller implementations
60+
├── proto/ # Proto definitions (.proto files)
61+
│ ├── BUILD.bazel
62+
│ └── *.proto
63+
├── protopb/ # Generated proto code (committed to repo)
64+
│ ├── BUILD.bazel
65+
│ ├── *.pb.go # Standard protobuf
66+
│ ├── *_grpc.pb.go # gRPC service code
67+
│ └── *.pb.yarpc.go # YARPC service code
68+
└── integration_tests/ # Integration tests
69+
```
70+
71+
**Key principle**: Controllers contain pure business logic and are independent of transport layer (gRPC/YARPC).
72+
73+
### Proto File Generation
74+
75+
All generated proto files are **committed to the repository**. When modifying `.proto` files:
76+
77+
1. Edit the `.proto` file
78+
2. Run `make proto` to regenerate all three file types:
79+
- `*.pb.go` (protobuf code)
80+
- `*_grpc.pb.go` (gRPC service code)
81+
- `*.pb.yarpc.go` (YARPC service code)
82+
3. Update controller implementations if needed
83+
4. Commit all generated files
84+
85+
## Extension System
86+
87+
Extensions are **vendor-agnostic, pluggable interfaces** for different backend implementations. This is a core architectural pattern in the repository.
88+
89+
### Current Extensions
90+
91+
```
92+
extensions/
93+
├── queue/ # Messaging queue abstraction
94+
│ ├── queue.go # Factory interface
95+
│ ├── publisher.go # Publisher interface
96+
│ ├── subscriber.go # Subscriber interface
97+
│ └── README.md # Documentation
98+
└── storage/ # Storage abstraction
99+
├── factory.go # Factory interface
100+
├── request_store.go # RequestStore interface
101+
└── mysql/ # MySQL implementation
102+
├── factory.go
103+
└── request_store.go
104+
```
105+
106+
### Extension Interface Pattern
107+
108+
Each extension defines:
109+
1. **Factory** interface for creating instances
110+
2. **Core interfaces** for the functionality (e.g., Publisher, Subscriber, RequestStore)
111+
3. **Implementation directories** under `extensions/{extension}/{impl}/`
112+
113+
### Adding New Extension Implementations
114+
115+
When implementing a new backend for an existing extension:
116+
117+
**Structure:**
118+
```
119+
extensions/{extension}/{impl}/
120+
├── BUILD.bazel
121+
├── factory.go # Implements Factory interface
122+
└── {interface}.go # Implements core interfaces
123+
```
124+
125+
**Examples:**
126+
- Queue implementations: `extensions/queue/sql/`, `extensions/queue/kafka/`
127+
- Storage implementations: `extensions/storage/postgres/`, `extensions/storage/cassandra/`
128+
129+
**Steps:**
130+
1. Create `extensions/{extension}/{impl}/` directory
131+
2. Implement the Factory interface from `extensions/{extension}/`
132+
3. Implement all required interfaces (Publisher/Subscriber for queue, RequestStore for storage)
133+
4. Add BUILD.bazel with appropriate go_library target
134+
5. Map domain entities to/from backend format
135+
6. Wire up lifecycle methods (Close, Ack/Nack, etc.)
136+
137+
### Adding New Extension Types
138+
139+
When adding a completely new extension category (e.g., cache, auth, etc.):
140+
141+
**Structure:**
142+
```
143+
extensions/{new_extension}/
144+
├── BUILD.bazel
145+
├── README.md # Document the interfaces and usage
146+
├── factory.go # Factory interface
147+
├── {interface}.go # Core interfaces
148+
└── {first_impl}/ # First implementation
149+
├── BUILD.bazel
150+
├── factory.go
151+
└── {interface}.go
152+
```
153+
154+
**Pattern to follow:**
155+
1. Define vendor-agnostic interfaces at `extensions/{new_extension}/`
156+
2. Document interfaces and usage patterns in README.md
157+
3. Create first implementation under `extensions/{new_extension}/{impl}/`
158+
4. Follow the same Factory pattern as queue and storage extensions
159+
160+
## Entities
161+
162+
Entities are domain objects used across the project. They live in the `entities/` directory, organized by domain:
163+
164+
```
165+
entities/
166+
├── queue/ # Queue domain entities
167+
│ ├── BUILD.bazel
168+
│ ├── message.go # Message entity
169+
│ ├── message_test.go
170+
│ ├── delivery.go # Delivery entity
171+
│ └── delivery_test.go
172+
└── storage/ # Storage domain entities
173+
├── BUILD.bazel
174+
└── land_request.go # LandRequest entity
175+
```
176+
177+
### Adding New Entities
178+
179+
When adding new domain entities:
180+
181+
**Structure:**
182+
```
183+
entities/{domain}/
184+
├── BUILD.bazel
185+
├── {entity}.go
186+
└── {entity}_test.go
187+
```
188+
189+
**Guidelines:**
190+
1. Group entities by domain (queue, storage, workflow, etc.)
191+
2. Keep entities pure and framework-agnostic
192+
3. Add corresponding tests
193+
4. Update BUILD.bazel with go_library and go_test targets
194+
5. Import entities using: `github.com/uber/submitqueue/entities/{domain}`
195+
196+
**Examples:**
197+
- Queue entities: `entities/queue/message.go`, `entities/queue/delivery.go`
198+
- Storage entities: `entities/storage/land_request.go`
199+
- New workflow entity: `entities/workflow/job.go`
200+
201+
## Directory Structure
202+
203+
```
204+
submitqueue/
205+
├── MODULE.bazel # Bzlmod dependencies
206+
├── go.mod # Go module dependencies
207+
├── BUILD.bazel # Root build configuration
208+
├── Makefile # Build automation
209+
├── .bazelversion # Bazel version (8.4.1)
210+
├── .envrc # direnv configuration
211+
│
212+
├── tools/ # Bazel tooling
213+
│ └── bazel # Bazelisk wrapper
214+
│
215+
├── gateway/ # Gateway service
216+
│ ├── controller/ # Business logic
217+
│ ├── proto/ # Proto definitions
218+
│ ├── protopb/ # Generated proto code
219+
│ └── integration_tests/
220+
│
221+
├── orchestrator/ # Orchestrator service
222+
│ ├── controller/
223+
│ ├── proto/
224+
│ ├── protopb/
225+
│ └── integration_tests/
226+
│
227+
├── speculator/ # Speculator service
228+
│ ├── controller/
229+
│ ├── proto/
230+
│ ├── protopb/
231+
│ └── integration_tests/
232+
│
233+
├── extensions/ # Pluggable backend implementations
234+
│ ├── queue/ # Queue abstraction
235+
│ │ └── {impl}/ # Implementation (sql, kafka, etc.)
236+
│ └── storage/ # Storage abstraction
237+
│ └── {impl}/ # Implementation (mysql, postgres, etc.)
238+
│
239+
├── entities/ # Domain entities
240+
│ ├── queue/ # Queue entities
241+
│ └── storage/ # Storage entities
242+
│
243+
├── examples/ # Example implementations
244+
│ ├── server/ # Server examples
245+
│ │ ├── gateway/
246+
│ │ ├── orchestrator/
247+
│ │ └── speculator/
248+
│ └── client/ # Client examples
249+
│ ├── gateway/
250+
│ ├── orchestrator/
251+
│ └── speculator/
252+
│
253+
├── integration_tests/ # Cross-service integration tests
254+
├── docs/ # Documentation
255+
│ ├── architecture/ # Architecture docs
256+
│ └── designs/ # Design documents
257+
└── bin/ # Compiled binaries (gitignored)
258+
```
259+
260+
## Development Workflow
261+
262+
### Making Changes
263+
264+
**1. Modifying Proto Files:**
265+
```bash
266+
# Edit proto file
267+
vim gateway/proto/gateway.proto
268+
269+
# Regenerate proto code
270+
make proto
271+
272+
# Update controller implementation
273+
vim gateway/controller/*.go
274+
275+
# Rebuild
276+
make build
277+
```
278+
279+
**2. Adding New RPC Method:**
280+
1. Update proto file with new method
281+
2. Run `make proto`
282+
3. Create controller in `{service}/controller/`
283+
4. Wire up controller in `examples/server/{service}/main.go`
284+
5. Test with client
285+
286+
**3. Adding New Extension Implementation:**
287+
1. Create `extensions/{extension}/{impl}/` directory
288+
2. Implement Factory and core interfaces
289+
3. Add BUILD.bazel
290+
4. Add tests
291+
5. Document in extension's README.md
292+
293+
**4. Adding New Entity:**
294+
1. Create `entities/{domain}/{entity}.go`
295+
2. Add corresponding test file
296+
3. Update BUILD.bazel
297+
4. Use entity in extensions/controllers as needed
298+
299+
### Testing
300+
301+
```bash
302+
# Build everything
303+
make build
304+
305+
# Run a service
306+
make run-gateway
307+
308+
# Test with client (in another terminal)
309+
make run-client-gateway MESSAGE="hello"
310+
311+
# Use grpcurl for manual testing
312+
grpcurl -plaintext -d '{"message": "hello"}' \
313+
localhost:8081 uber.devexp.submitqueue.gateway.SubmitQueueGateway/Ping
314+
```
315+
316+
### Common Make Targets
317+
318+
```bash
319+
make build # Build all services
320+
make proto # Regenerate proto files
321+
make run-gateway # Run gateway service
322+
make run-orchestrator # Run orchestrator service
323+
make run-speculator # Run speculator service
324+
make clean # Remove binaries
325+
make clean-proto # Remove generated proto files
326+
make test # Run tests
327+
```
328+
329+
## Key Conventions
330+
331+
### Import Paths
332+
333+
- Controllers: `github.com/uber/submitqueue/{service}/controller`
334+
- Proto (generated): `github.com/uber/submitqueue/{service}/protopb`
335+
- Extensions: `github.com/uber/submitqueue/extensions/{extension}`
336+
- Extension impl: `github.com/uber/submitqueue/extensions/{extension}/{impl}`
337+
- Entities: `github.com/uber/submitqueue/entities/{domain}`
338+
339+
### Code Organization
340+
341+
1. **Separation of Concerns**: Controllers are pure business logic, independent of transport
342+
2. **Interface-Driven**: Extensions define interfaces, implementations live in subdirectories
343+
3. **Generated Code Committed**: All proto-generated files are committed to repo
344+
4. **Build Files**: Every Go package has a BUILD.bazel file
345+
5. **Testing**: Each package should have corresponding tests
346+
347+
### Dependencies
348+
349+
- **External dependencies**: Add to both `go.mod` AND `MODULE.bazel`
350+
- **Internal dependencies**: Reference via import paths and Bazel deps
351+
- **Proto dependencies**: Defined in BUILD.bazel files
352+
353+
### File Naming
354+
355+
- Proto files: `{service}.proto`
356+
- Controllers: `{method}.go` or `{feature}.go`
357+
- Entities: `{entity}.go`
358+
- Tests: `{file}_test.go`
359+
- BUILD files: Always `BUILD.bazel`
360+
361+
## Important Notes
362+
363+
1. **Never use WORKSPACE**: This repo uses Bzlmod exclusively
364+
2. **Commit generated files**: All `*pb.go` files are committed
365+
3. **Use interfaces for extensions**: Keep implementations swappable
366+
4. **Follow the Factory pattern**: All extensions use Factory interface
367+
5. **Keep entities pure**: No framework dependencies in entity types
368+
6. **Test coverage**: Add tests for new functionality
369+
7. **Update BUILD.bazel**: When adding new Go files, update BUILD.bazel
370+
371+
## Quick Reference
372+
373+
**Add new service method:**
374+
1. Edit `{service}/proto/*.proto`
375+
2. Run `make proto`
376+
3. Add controller in `{service}/controller/`
377+
4. Wire up in `examples/server/{service}/main.go`
378+
379+
**Add new extension implementation:**
380+
1. Create `extensions/{extension}/{impl}/`
381+
2. Implement Factory and interfaces
382+
3. Add BUILD.bazel
383+
4. Document usage
384+
385+
**Add new entity:**
386+
1. Create `entities/{domain}/{entity}.go`
387+
2. Add test file
388+
3. Update BUILD.bazel
389+
390+
**Run/test locally:**
391+
1. `make run-{service}` to start service
392+
2. `make run-client-{service}` to test
393+
3. Or use `grpcurl` for ad-hoc testing

0 commit comments

Comments
 (0)