feat(dogfood): Athens in-cluster Go module cache (Q244)#466
Merged
Conversation
Stand up an Athens Go module proxy in the gag-dogfood namespace so vendor-check and tidy-check can run on GAG runners without external egress to proxy.golang.org. GKE Dataplane V2's managed Cilium lacks the CiliumNetworkPolicy CRD, so CiliumFQDN egress mode is unusable; a CIDR allowlist for Google- fronted hosts like proxy.golang.org is a footgun. Athens sidesteps both constraints: it runs in-cluster with free egress (not covered by the workload NetworkPolicy) and serves cached modules to workers over plain HTTP port 3000. deploy/athens/ adds: - Deployment (gomods/athens:v0.15.1, disk storage, system node pool) - ClusterIP Service on port 3000 - 20 Gi PVC (standard-rwo) for the module cache - Additive NetworkPolicy allowing workload pods → Athens pods on 3000 dogfood-setup.sh gains apply_athens() (runs before apply_cr) and the RunnerTemplate gets GOPROXY + GONOSUMDB env vars so vendor-check/ tidy-check pick up Athens automatically. Closes Q244. Unblocks Q224 (run dogfood-start.sh + validate green CI).
…ntral1-b Two fixes in one commit: 1. Athens Service renamed from "athens" to "go-module-proxy". A Service named "athens" causes Kubernetes to inject ATHENS_PORT=tcp://clusterip:3000 into pods in the namespace; Athens reads ATHENS_PORT as its listen address and fails with "too many colons in address". The rename avoids the collision entirely. GOPROXY in dogfood-setup.sh and docs updated to match. 2. Dogfood zone moved from us-central1-a to us-central1-b. us-central1-a returned ZONE_RESOURCE_POOL_EXHAUSTED for e2-standard-2 nodes; us-central1-b confirmed available. Updated in gke-dogfood.md and all dogfood scripts. Validated on the recreated us-central1-b cluster: - Athens starts cleanly (tcpPort=:3000, healthz 200) - go-module-proxy service reachable from pods in gag-dogfood namespace
Athens was returning 404 for github.com/onsi/ginkgo/v2@v2.32.0 because proxy.golang.org serves bytes that hash differently from sum.golang.org's recorded value — a known upstream divergence for this version. Athens correctly detected the mismatch and rejected the download. Set ATHENS_GONOSUMDB=*, ATHENS_GONOSUMCHECK=*, GONOSUMDB=*, and GONOSUMCHECK=* so Athens skips the remote checksum database entirely. Integrity is enforced by go.sum on the worker side (GONOSUMDB=* means Go uses the local go.sum, not the remote checksum database). The ginkgo/v2@v2.32.0 zip was also seeded directly into the Athens PVC from the local module cache (which has the correct bytes matching go.sum) to work around the proxy.golang.org divergence.
Reverts the checksum-skip approach from the previous commit, which was the wrong fix. Investigation showed ginkgo/v2@v2.32.0 is not tampered: proxy.golang.org, sum.golang.org, GitHub direct-VCS (with a modern Go), and our committed go.sum all agree on h1:Hw7s2pVrQo..., and the tag was never moved (still commit 9ff1646). The only outlier was Athens' own locally-rebuilt hash. Root cause: Athens defaults GoBinaryEnvVars to ["GOPROXY=direct"], forcing its internal go subprocess to clone each module from origin and rebuild the zip with the image's bundled (older) Go. That toolchain computes a different module hash for modules declaring a newer go directive (ginkgo declares go 1.25), which Athens' own checksum check then rejects against sum.golang.org — surfacing as a 404 to workers. Fix: set ATHENS_GO_BINARY_ENV_VARS=GOPROXY=https://proxy.golang.org so Athens fetches the immutable, notarized zip from the public proxy instead of rebuilding it. Its hash matches sum.golang.org and our go.sum, so the checksum check passes — verification stays ON (ATHENS_GONOSUMCHECK=""). The proxy is also faster (pre-built zips over a CDN) and more available (one upstream vs. every module's origin host). A container-level GOPROXY env var does not work; Athens overrides it for the subprocess via config. Validated on the dogfood cluster: cold-cache fetch of ginkgo/v2@v2.32.0 returns 200 for both .mod and .zip, no SECURITY ERROR in Athens logs, and the served zip is byte-identical (sha256 2d1a42b7...) to proxy.golang.org's notarized artifact.
Two changes surfaced while validating Athens end-to-end: - Upgrade image v0.15.1 -> v0.18.0. v0.15.1 bundled Go 1.20.14, which is end-of-life and receives no security patches; Athens shells out to this toolchain to process modules. v0.18.0 ships Go 1.26.2. (With the GOPROXY=proxy.golang.org pin, Athens no longer rebuilds zips, so the Go version no longer affects correctness — but keeping a maintained toolchain in the image is the right default.) No config-breaking changes land between 0.15 and 0.18 for our disk-storage setup. - Lower the CPU request 250m -> 100m. The single system node sits near its CPU-request ceiling during CI worker bursts, and at 250m Athens could not co-schedule with the AGC and got stuck Pending (NotTriggerScaleUp: workers node pool is tainted). Athens is I/O-bound serving cached zips from the PVC; the generous limits still allow bursting for cold `go mod download` fetches. Validated on the dogfood cluster: v0.18.0 reports Go 1.26.2 and a cold-cache fetch of ginkgo/v2@v2.32.0 through Athens returns 200 with a byte-identical notarized zip (sha256 2d1a42b7...).
The previous commit dropped the ",direct" fallback on the mistaken belief that the ATHENS_GO_BINARY_ENV_VARS form splits on commas. It splits on ";" (EnvList.Decode), so GOPROXY=https://proxy.golang.org,direct is parsed as a single assignment with the comma intact. Restoring the fallback lets Athens resolve any module proxy.golang.org doesn't mirror (e.g. a future private dep), matching Go's own default GOPROXY. Validated: pod stays healthy (config Validate passes) and a cold ginkgo fetch still returns 200.
Commit 0f0f473 added a stray trailing newline to go.work.sum purely to trip the unit-test "modules" path filter and force vendor-check/tidy-check to run on the GAG dogfood runners for Athens validation. That validation is done: the GAG runner fetched 267 modules through Athens cleanly (no 404s, no checksum errors). Revert the hack so #466's module files match main and the tidy/vendor gates aren't triggered by an unrelated change. GAG_RUNNER is reset to ubuntu-latest separately (repo variable), returning dogfood CI to on-demand.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Deploys an Athens in-cluster Go module proxy (
deploy/athens/) in thegag-dogfoodnamespace so GAG self-hosted runners can fetch modules forvendor-check/tidy-checkwithout each worker needing egress toproxy.golang.org.dogfood-setup.shapplies Athens before provisioning tenant objects; the RunnerTemplate points workers at Athens. Documented as Part B6b in the GKE dogfood runbook (Q244).Why this shape
GKE Dataplane V2's managed Cilium lacks the
CiliumNetworkPolicyCRD (dropped since GKE 1.21.5-gke.1300), soCiliumFQDNegress is unusable, and a CIDR allowlist forproxy.golang.org/sum.golang.orgis a footgun (Google-fronted ⇒ opens all of Google's frontend). Athens sidesteps both: only the Athens pod reaches upstream (once per module, then cached to a PVC); workers only talk to Athens in-cluster.app=athens) is not labelledactions-gateway/component=workload, so it's outside the workloadNetworkPolicyand has free egress. An additiveNetworkPolicyopens port 3000 from workload pods to Athens.GOPROXY=http://go-module-proxy.gag-dogfood.svc.cluster.local:3000,off+GONOSUMDB=*(no egress tosum.golang.org; integrity comes from the committedgo.sum).Key implementation details (worth a look in review)
go-module-proxy, notathens— a Service namedathensmakes Kubernetes injectATHENS_PORT=tcp://…:3000into every pod in the namespace, which Athens misreads as its own listen address and crashes.proxy.golang.org, not direct-from-VCS — viaATHENS_GO_BINARY_ENV_VARS=GOPROXY=https://proxy.golang.org,direct. Athens' built-in default (GOPROXY=direct) makes it clone each module and rebuild the zip locally with its bundled Go; an older bundled Go produces a different module hash than the notarized one for modules declaring a newergodirective (e.g.ginkgo/v2@v2.32.0,go 1.25), which Athens then rejects as a checksum "SECURITY ERROR" → 404. Routing through the public proxy serves the immutable notarized zip, so its own checksum verification (left on,ATHENS_GONOSUMCHECK="") passes. (Upstream fix proposed: fix(config): default GoBinaryEnvVars to proxy.golang.org, not direct-only gomods/athens#2143.)gomods/athens:v0.18.0(Go 1.26.2). Older tags shipped EOL Go (v0.15.1 → 1.20.14).go.sumregardless. Rationale documented indeploy/athens/networkpolicy.yamland the runbook.Validation (on the GKE dogfood cluster)
kubectl apply -k deploy/athensapplies cleanly; Athens pod reaches Ready with no startup errors.ginkgo/v2@v2.32.0through Athens returns 200 for both.modand.zip, with no checksum error in Athens logs; the served zip is byte-identical (sha2562d1a42b7…) toproxy.golang.org's notarized artifact.Notes
GAG_RUNNERwas reset toubuntu-latestafter validation, returning dogfood CI to on-demand. Flipping production CI onto the dogfood runners is tracked under Q224 (still blocked by separate GAG-runner issues: worker→GitHub-CDN egress and AGC session recovery).Closes Q244. Unblocks Q224.