feat: add orchestrator pipeline, TopicRegistry, and CI improvements (#59)

## Summary

- **TopicRegistry**: Centralizes queue topic configuration, decoupling
controllers from subscription details. Controllers declare their topic
and consumer group; the registry provides the queue and subscription
config.
- **Orchestrator pipeline**: Adds request controller with publisher for
forwarding messages to the next pipeline stage (request → batch →
speculation → build → merge).
- **Consumer refactor**: Accepts `TopicRegistry` instead of raw
`queue.Queue` + subscriber name. Removes `SubscriptionConfig()` from the
`Controller` interface.
- **Server startup simplification**: Removes retry loops from gateway
and orchestrator servers — relies on Docker Compose healthchecks
(`--wait`) to ensure dependencies are ready before services start.
- **CI caching fix**: Caches `bazel-disk-cache` and `bazel-repo-cache`
(matching `.bazelrc` paths) with per-job keys to avoid concurrent save
races. Previous setup cached the wrong directory and all jobs raced on a
single key.
- **Auto-tail container logs**: Streams Docker container logs to stderr
in real-time during integration/e2e tests via `ComposeStack.tailLogs()`.
Removes the now-redundant `.github/actions/logs` action.
- **Test output streaming**: All test targets use
`--test_output=streamed` for real-time output in both local and CI runs.
- **Documentation consolidation**: Merges `PROJECT_STRUCTURE.md` into
`CLAUDE.md`, updates `TESTING.md` for new ComposeStack behavior.

## Test plan

- [x] `make build` passes
- [x] `make test` passes
- [x] `make gazelle` produces no changes
- [x] CI integration and e2e tests pass
- [x] Verify container logs stream to stderr during test runs
- [x] Verify CI Bazel cache saves per-job and restores on subsequent
runs
- First Run:
https://github.com/uber/submitqueue/actions/runs/22336834292/job/64632139444?pr=59
(~10m)
- Second Run:
https://github.com/uber/submitqueue/actions/runs/22337340518/job/64633633575?pr=59
(1m24s)

Author: behinddwalls

.bazelrc

@@ -1 +1,2 @@
 common --disk_cache=~/.cache/bazel-disk-cache
+common --repository_cache=~/.cache/bazel-repo-cache

.github/actions/logs/action.yml

@@ -1,5 +1,5 @@
 name: Run Bazel Test
-description: Run Bazel test and show logs on failure
+description: Run a Bazel test target
 
 inputs:
   target:
@@ -11,7 +11,4 @@ runs:
   steps:
     - name: Run Bazel test
       shell: bash
-      run: ./tool/bazel test ${{ inputs.target }} --test_output=errors
-
-    - uses: ./.github/actions/logs
-      if: always()
+      run: ./tool/bazel test ${{ inputs.target }} --test_output=streamed

.github/actions/run-bazel-test/action.yml

@@ -1,5 +1,17 @@
 name: Setup
-description: Setup Bazel cache for CI jobs
+description: Setup Bazel caches for CI jobs
+
+# Caches three directories that match the paths configured in .bazelrc:
+#   - disk-cache:  compiled build outputs (biggest speedup — avoids recompilation)
+#   - repo-cache:  downloaded external dependencies (avoids re-downloading Go modules, protobuf, etc.)
+#   - bazelisk:    the Bazel binary itself
+#
+# Cache key includes github.job so each CI job saves its own cache entry
+# (otherwise concurrent jobs race on the same key and only one wins).
+# github.run_id ensures the key never matches exactly, forcing actions/cache
+# to always save — so the disk cache accumulates entries across runs.
+# The restore-keys fallback restores from the most recent prior run of the
+# same job (preferred), then same deps, then any.
 
 runs:
   using: composite
@@ -8,8 +20,11 @@ runs:
       uses: actions/cache@v4
       with:
         path: |
-          ~/.cache/bazel
+          ~/.cache/bazel-disk-cache
+          ~/.cache/bazel-repo-cache
           ~/.cache/bazelisk
-        key: bazel-${{ runner.os }}-${{ hashFiles('MODULE.bazel', 'go.mod', '.bazelversion') }}
+        key: bazel-${{ runner.os }}-${{ github.job }}-${{ hashFiles('MODULE.bazel', 'go.mod', 'go.sum', '.bazelversion', '.bazelrc') }}-${{ github.run_id }}
         restore-keys: |
+          bazel-${{ runner.os }}-${{ github.job }}-${{ hashFiles('MODULE.bazel', 'go.mod', 'go.sum', '.bazelversion', '.bazelrc') }}-
+          bazel-${{ runner.os }}-${{ github.job }}-
           bazel-${{ runner.os }}-

.github/actions/setup/action.yml

@@ -59,8 +59,6 @@ jobs:
       - name: Run E2E tests
         run: make e2e-test
 
-      - uses: ./.github/actions/logs
-
   gateway-integration-test:
     name: Gateway Integration Test
     runs-on: ubuntu-latest
@@ -71,8 +69,6 @@ jobs:
       - name: Run Gateway integration tests
         run: make integration-test-gateway
 
-      - uses: ./.github/actions/logs
-
   orchestrator-integration-test:
     name: Orchestrator Integration Test
     runs-on: ubuntu-latest
@@ -83,8 +79,6 @@ jobs:
       - name: Run Orchestrator integration tests
         run: make integration-test-orchestrator
 
-      - uses: ./.github/actions/logs
-
   # ---------------------------------------------------------------------------
   # EXTENSION TESTS
   # ---------------------------------------------------------------------------