Wire Firecracker E2E tests into GitHub Actions with ephemeral GCP spot runner

[mathaix]

Goal

Wire maintainer-gated Firecracker/KVM E2E tests into GitHub Actions using an ephemeral GCP spot N2 runner.

The workflow should:

1. Start a GCP spot N2 VM with nested virtualization/KVM enabled.
2. Register or expose it as a GitHub self-hosted runner for this repo.
3. Run the trusted Firecracker E2E/integration test lane.
4. Capture test output, reports, and useful diagnostics as GitHub Actions artifacts.
5. Always shut the VM down/delete it, even when tests fail.

Context

The public repo already keeps normal PR checks on free GitHub-hosted runners and keeps KVM tests maintainer-gated. Existing docs/workflows to review first:

• docs/ci-release.md
• docs/kvm-integration-ci.md
• .github/workflows/kvm-integration.yml
• ci/start-kvm-runner.sh
• ci/stop-kvm-runner.sh
• ci/wait-for-runner-online.sh
• Makefile targets: make integration-kvm, make test-integration, make test-integration-run TEST=...

This issue should make the runner lifecycle concrete for GCP instead of leaving it as generic start/stop hooks.

Requirements

• Use a GCP spot VM, preferably N2 family, with nested virtualization available for Firecracker.
• Keep public PR safety boundaries intact:
  • Do not run KVM tests automatically on arbitrary public PR code.
  • Do not use pull_request_target to execute PR code.
  • Keep PR KVM runs behind maintainer-triggered workflow_dispatch with exact PR head SHA validation.
• Run on pushes to main and manual dispatch for reviewed PR SHAs.
• Ensure the VM has:
  • Linux x86_64
  • /dev/kvm
  • Firecracker
  • Go/Node/npm/make/git
  • rootfs/kernel/runtime assets or the ability to fetch/stage them
  • network tools required by the integration suite
• Prefer an ephemeral self-hosted runner registered only for this run.
• Use least-privilege GCP credentials and document required IAM roles.
• Do not print GCP service account keys, GitHub runner tokens, Cloudflare tokens, or other secrets.
• Always run cleanup:
  • sudo bash scripts/test-cleanup.sh where appropriate
  • unregister GitHub runner
  • delete/stop the GCP spot VM

Report/artifact capture

Capture enough information to debug failures without SSHing into the runner:

• Raw go test -v -tags integration output.
• JUnit or structured test report if practical.
• Firecracker/OCM integration state summaries when failures occur.
• df -h, free -h, lsblk, /dev/kvm presence, Firecracker version.
• Cleanup logs.
• GitHub Actions artifact upload for reports/logs.

Acceptance criteria

• A maintainer can trigger the workflow for a PR by entering PR number and exact SHA.
• The workflow validates the SHA matches the PR head before running code.
• A GCP spot N2 KVM runner is created for the run and becomes online with the expected labels.
• make integration-kvm runs on that runner.
• Test logs/reports are uploaded as artifacts on both success and failure.
• The runner is unregistered and the GCP VM is deleted/stopped on success, failure, and cancellation.
• The workflow remains safe for public PRs and does not expose privileged secrets to untrusted jobs.
• Documentation is updated with required GitHub secrets, GCP IAM, machine type/zone settings, and manual dispatch instructions.

Open questions

• Which GCP project/region/zone should be used for the public repo runner?
• Should the VM image be built ahead of time with Firecracker/rootfs assets, or should the workflow bootstrap from a base image each run?
• Should reports be plain artifacts only, or should the workflow also publish a commit status/check summary with the failed test names?

[mathaix]

Verification: a different approach was built than this issue specifies — needs a decision before closing

While verifying the CI lanes, I found that what landed on main (the Spot E2E lane) diverges from this issue's literal acceptance criteria. Both readings are defensible, so flagging for a maintainer call.

What this issue asked for

• A GCP spot N2 VM registered as a GitHub self-hosted runner (kvm label), running make integration-kvm (the Go -tags integration suite), with concrete GCP lifecycle "instead of leaving it as generic start/stop hooks," plus diagnostic artifacts (go test -v output, JUnit, df -h/free -h/lsblk//dev/kvm, cleanup logs).

What's actually on main

Two separate lanes, neither of which is the above:

1. spot-e2e.yml + spot-e2e-run.yml + ci/spot-e2e.sh — runs on ubuntu-latest, provisions a real GCE spot host via the OCM control plane's own POST /api/admin/hosts provisioner, then runs a Playwright smoke (frontend/e2e/spot-smoke.spec.ts) that drives create → Firecracker boot through the product UI. Admin-gated, SHA-validated, robust teardown, Playwright artifacts uploaded if: always(). ✅ for those criteria — but it is not the KVM integration suite and registers no self-hosted runner.
2. kvm-integration.yml — does run make integration-kvm on [self-hosted, linux, x64, kvm], but its lifecycle is still the generic OCM_KVM_RUNNER_START_COMMAND/STOP_COMMAND hooks — exactly what this issue wanted to make concrete for GCP — and it uploads no diagnostic artifacts (only a commit status).

Criteria status

Criterion	Status
Maintainer triggers via PR# + SHA	✅
Validates SHA = PR head before running code	✅
GCP spot N2 runner created & online with kvm labels	❌ (provisions a host, not a GH runner)
make integration-kvm runs on that runner	❌ (runs Playwright smoke instead)
Artifacts on success + failure	✅ Playwright lane / ❌ KVM lane
VM deleted on success/failure/cancel	✅
Docs: secrets/IAM/machine-type/dispatch	✅ (setup-spot-e2e.sh, docs/spot-e2e-ci.md)

Decision needed

• (A) If the Spot E2E Playwright lane satisfies the intent (Firecracker-on-GCP-spot, admin-gated, with teardown + artifacts) → close this issue with a note that the design changed, and spin off the leftovers (concrete GCP lifecycle for kvm-integration.yml + Go-suite diagnostics) only if still wanted.
• (B) If the Go KVM integration suite on concrete GCP spot infra with full diagnostics is specifically required → this is only partially done; that work remains.

Which interpretation should we take?

🤖 Generated with Claude Code