[FEATURE] Foreman coder gate: cluster-backed clean-room Job tier for envtest/e2e

[Defilan]

Feature Description

Add a heavyweight, cluster-backed verification tier to the Foreman coder gate: a clean-room Kubernetes Job that runs envtest + e2e (kind) on a node that has a cluster, executed after the fast in-workspace gate passes, with failures fed back to the coder loop the same way the fast gate's failures are.

Problem Statement

As a maintainer running Foreman coders, I want the gate to catch real-cluster integration regressions before a PR is opened, so coder solutions don't pass the gate and then fail CI on envtest/e2e.

The current coder gate runs entirely in the coder's workspace: gofmt, go vet, go build, golangci-lint, and (since #762/#763) a fast unit-test tier. It has no Kubernetes cluster, so it structurally cannot run envtest (KUBEBUILDER_ASSETS) or e2e (kind). Any coder change whose behavior only manifests against a real cluster — CRD reconciliation, operator logic, CLI-against-cluster — can pass the full in-workspace gate (including unit tests, which mock the cluster and give false confidence) and only fail in CI after the PR is open.

This is systematic, not a one-off. It has now bitten the #731 cache-list work twice and the earlier envtest trap once:

• [FEATURE] Make llmkube cache list per-InferenceService cache aware (post #729) #731: the coder produced a correct cache list per-InferenceService feature (verified working against a live 31-model cluster), but it regressed a pre-existing e2e: cache list inspects every labeled cache PVC, and a Pending (unbound) per-service PVC — created by an InferenceService whose fake e2e model never produces a running serving pod — sent the inspector into a waitForPodRunning block until the suite's global timeout. The unit tests passed (mocked client); only the kind e2e caught it.
• The earlier envtest trap: the coder ran envtest in its workspace where KUBEBUILDER_ASSETS is unset and hung.

No amount of better unit testing closes this: mocked tests cannot model real PVC layout, real pod/volume lifecycle, or real reconciliation timing. It is integration territory by nature.

Proposed Solution

A post-fast-gate tier that runs the integration suite on a cluster-backed Job:

• After the in-workspace fast gate (gofmt/vet/build/lint/unit) passes a coder's GO, dispatch a clean-room Job on a node that has a cluster (Shadowstack; ties to [FEATURE] Example: spot-capacity GPU NodePool for Foreman gate Jobs #659 spot GPU NodePool for gate Jobs).
• The Job checks out the coder's branch and runs make test (envtest) and/or make test-e2e (kind), bounded by a timeout.
• On failure, feed the output back to the coder loop as gate feedback (same mechanism as the fast gate), so the coder fixes and resubmits; on pass, the GO stands.
• Keep the fast in-workspace gate as the first, cheap line of defense; the cluster Job is the heavyweight second tier (only run when the fast gate is green).

Alternatives Considered

• Run e2e/envtest in the coder workspace — rejected: the workspace has no cluster, and provisioning one per coder is heavy and slow; the envtest trap shows the coder should not run these directly.
• Rely on better unit tests — insufficient: mocked tests gave false confidence on exactly the [FEATURE] Make llmkube cache list per-InferenceService cache aware (post #729) #731 regression.
• Accept CI as the only integration gate — status quo; it pushes the failure past the GO and into review, which is what this issue exists to fix.

Additional Context

• Related: [FEATURE] Example: spot-capacity GPU NodePool for Foreman gate Jobs #659 (spot GPU NodePool for Foreman gate Jobs), [BUG] Coder gate runs no tests; failing/panicking unit tests reach a GO and are only caught by CI #762/feat(foreman): add unit-test tier to coder gate #763 (fast unit-test tier), [FEATURE] Make llmkube cache list per-InferenceService cache aware (post #729) #731 (the motivating regression).
• The fast gate's design (deterministic checks + feedback loop, feat(foreman): deterministic coder verification gate with feedback loop #749) is the pattern to extend: same feedback mechanism, heavier tier, cluster-side.

Priority

• High - Would significantly improve my workflow

Willingness to Contribute

• Yes, I can submit a PR
• Yes, I can help test

[Defilan]

Update (2026-06-21): the cluster gate tier already exists; remaining work is auto-wiring + the e2e half

Discovered while dogfooding this weekend: the heavyweight tier this issue
asks for is already built and deployed, just not wired into the bare
issue-fix loop:

• run_gate_job tool (pkg/foreman/agent/tools/run_gate_job.go) submits a
  K8s Job that clones the branch and runs make fmt vet lint test manifests chart-crds foreman-chart-crds + a dirty-tree check.
• verify task kind + the gate verifier Agent (in-cluster foreman-agent
  runs --roles=worker,verifier), foreman-gate-cache PVC.

Validated end-to-end this weekend (manually):

• [BUG] InferenceService reconcileService is create-only: endpoint.type/port changes ignored on existing Service #720: fast gate GO'd a controller fix whose envtest test failed; the
  cluster gate caught it (GATE-FAIL), and feeding the raw failure back drove
  a 3-cycle self-correction to GATE-PASS (PR fix(controller): reconcileService updates existing Service on endpoint changes (#720) #773).
• [BUG] checkAcceleratorAvailability ignores GPU.resourceClaims; AcceleratorReady inaccurate for DRA-only Models #754: fast gate GO; cluster gate caught stale CRDs ("tree dirty") and,
  with review, a hollow availability check + a namespace regression.

So the remaining work for this issue is narrower than originally framed:

1. Auto-wire the verify gate after a coder GO (opt-in per Agent/Workload),
   and on GATE-FAIL auto-re-dispatch the coder with the raw gate output
   injected (the payload.prompt path), up to N cycles, else INCOMPLETE.
   This automates exactly the loop run by hand for [BUG] InferenceService reconcileService is create-only: endpoint.type/port changes ignored on existing Service #720/[BUG] checkAcceleratorAvailability ignores GPU.resourceClaims; AcceleratorReady inaccurate for DRA-only Models #754, reusing the
   feat(foreman): deterministic coder verification gate with feedback loop #749 feedback envelope. This is the envtest half and is doable now.
2. e2e: the existing gate runs make test (envtest) but not
   make test-e2e; the e2e half still needs kind-in-the-Job (dind/privileged)
   or an ephemeral-namespace run on a real cluster. This is the genuine
   remaining build.

Note: the cheap in-workspace half of the codegen-drift case is split out to
the fast gate in #775.