feat: v0.3 core CI (M8) — .gitflare/ci.yml jobs on Cloudflare Sandboxes

PLAN.md

@@ -478,7 +478,7 @@ That's it for v0.1 — three permissions. *Workers Routes* is a Zone-level permi
 
 **Three scopes for v0.1. Every one named. No "Edit all of Cloudflare" hand-grenade tokens.** Each later version adds scopes; never replaces:
 - v0.2 adds: `Workers KV Storage: Edit` (for cached deploy state)
-- v0.3 adds: Sandbox-related scopes when those stabilize; `R2: Edit` (specifically *Workers R2 Storage*, for build cache); `D1: Edit` (for CI run history)
+- v0.3 adds: the account-level **Containers** permission (needed the moment `gitflare ci enable` puts a `[[containers]]` block in the Worker config — exact permission name to be confirmed on the first live run; `ci enable` warns about this up front). Later in v0.3: `R2: Edit` (specifically *Workers R2 Storage*, for build cache); `D1: Edit` (for CI run history)
 - v0.4 adds: `Cloudflare Access: Apps and Policies: Edit`
 - v0.5 adds: `Cloudflare One Connector: Edit`, `Cloudflare Tunnel: Edit`
 - Custom domains (any version): `Zone → Workers Routes: Edit`, `Zone → DNS: Edit`
@@ -553,7 +553,8 @@ The roadmap in §4 is what we're shipping. This section is *where we are right n
 | M5 | Privacy via Cloudflare Access | 🧪 implemented, not yet live-validated | Worker: `accessGuard` middleware verifies the `Cf-Access-Jwt-Assertion` JWT (RS256 via WebCrypto + JWKS cache, no `jose`) and gates `/` + `/r/*` + `/api/*`; enforces only when `ACCESS_AUD` var is set, so public mirrors stay open. `/webhooks/github` + `/health` left unauth. 8 unit tests. CLI: opt-in `gitflare access enable/disable` creates/deletes a self-hosted Access app + allow-list policy and redeploys with `ACCESS_AUD`/`ACCESS_TEAM_DOMAIN` vars. **Caveat:** this gates the dashboard/API only — `git clone` hits Artifacts directly (`*.artifacts.cloudflare.net`), so it is NOT Access-gated; private-clone is a later (v0.4+) item per §6/§11. **TODO before ✅:** verify the live Access apps/policies API shapes + `aud` field, and that `*.workers.dev` hosts are accepted, against a real account. |
 | M5.5 | v0.1 polish | 🧪 implemented | Syntax highlighting in the blob viewer (highlight.js core + ~20 curated grammars, server-side; 512 KB cap with `<pre>` fallback; bundle 681→901 KB). README image proxy: new `GET /r/:name/raw/*` serves blob bytes from the Artifacts mirror so images render for private repos and survive GitHub outages — README rewriting now points there instead of `raw.githubusercontent.com`. Styled empty/error states (`ui/states.tsx`): home shows the `npx gitflare init` command, browse 404/500 render through the layout with a way back. |
 | M6 | v0.2 CD (MVP slice) | 🧪 implemented | **Self-deploy model (user chose Worker-Secret path).** On push, after sync, the webhook fires `DeployDO` (`waitUntil`), which clones the repo, parses `.gitflare/deploy.yml`, and uploads the pre-built `cloudflare/deploy { kind: worker }` entry via the Workers Scripts multipart API. History in `DeployDO`; Deployments UI; CLI `gitflare deploy enable/disable` stores `CF_DEPLOY_TOKEN` + `CD_ENABLED`. Superseded by M7. |
-| M7 | v0.2 CD — complete (per §4) | 🧪 implemented, not yet live-validated | Everything in §4 v0.2: **(1)** real YAML-subset parser (`deploy/yaml.ts`, no heavyweight dep) → **worker bindings** (vars/KV/R2/D1/DO/services) in the Scripts metadata; **(2)** **Pages** deploys via Direct Upload (upload-token → check-missing → upload → create-deployment), with per-branch **previews** (`production_branch`); **(3)** **D1 migrations** applied in order via the D1 query API, opt-in (`apply: true`), idempotent (applied set tracked per-DB in the DO); **(4)** **live deploy-log streaming** over a hibernatable WebSocket (`/r/:name/deployments/stream`) with the Deployments page subscribing; **(5)** **manual / GitHub-down trigger** `gitflare deploy run` + `/control/deploy/run` (auth'd by a `CONTROL_SECRET`, outside Access); **(6)** `gitflare deploy list` + `gitflare deploy rollback [--to <id>]` (rollback redeploys a previous successful commit via a full clone; migrations are forward-only and skipped). 56 unit tests across yaml/workflow/cf-deploy/highlight/access. **TODO before ✅:** one live run against a real Cloudflare account to confirm the Scripts multipart, Pages Direct Upload, and D1 query wire shapes. |
+| M7 | v0.2 CD — complete (per §4) | 🧪 implemented, not yet live-validated | Everything in §4 v0.2: **(1)** real YAML-subset parser (`deploy/yaml.ts`, no heavyweight dep) → **worker bindings** (vars/KV/R2/D1/DO/services) in the Scripts metadata; **(2)** **Pages** deploys via Direct Upload (upload-token → check-missing → upload → create-deployment), with per-branch **previews** (`production_branch`); **(3)** **D1 migrations** applied in order via the D1 query API, opt-in (`apply: true`), idempotent (applied set tracked per-DB in the DO); **(4)** **live deploy-log streaming** over a hibernatable WebSocket (`/r/:name/deployments/stream`) with the Deployments page subscribing; **(5)** **manual / GitHub-down trigger** `gitflare deploy run` + `/control/deploy/run` (auth'd by a `CONTROL_SECRET`, outside Access); **(6)** `gitflare deploy list` + `gitflare deploy rollback [--to <id>]` (rollback redeploys a previous successful commit via a full clone; migrations are forward-only and skipped). 56 unit tests across yaml/workflow/cf-deploy/highlight/access. **TODO before ✅:** one live run against a real Cloudflare account to confirm the Scripts multipart, Pages Direct Upload, and D1 query wire shapes. Known quirk (kept in M8): the v0.2 push path reads workflow/content at the clone's HEAD while recording the webhook sha — same-branch seconds-apart race only; the v0.3 CI path reads strictly at the pushed sha by construction. |
+| M8 | v0.3 core CI — sandbox jobs, needs-gated deploys | ✅ core live-validated (2026-07-17, `sinameraji/kimiflare`) | **Format** (`.gitflare/ci.yml`, parsed by `src/ci/workflow.ts` reusing the yaml.ts subset + the shared `cloudflare/deploy` step validator): `on`/`branches`, `jobs:` with `needs:` (parse-time unknown/cycle validation, Kahn topo order), `run:` steps (multi-line via new `\|`/`\|-` block-scalar support in yaml.ts), per-job `env:` + `timeout_minutes` (default 15, max 60); a job is either a run job or a deploy job (mixing = parse error). **Execution**: new `CiDO` (binding `CI`, migration v3) — POST `/run` answers 202 and the pipeline runs as detached DO work; ONE sandbox per run (`@cloudflare/sandbox@0.12.3`, image pinned to match) cloned once from the Artifacts mirror via env-scoped `http.extraHeader` auth (token never in argv/.git/config/stderr; every log line scrubbed for the secret + its base64), then jobs execute in topo order with streamed line-buffered logs (hibernatable WS at `/r/:name/ci/stream`, cap 1000 lines/job). **Deploy jobs delegate to DeployDO** (`mode: "ci"`, strict at-sha ref-aware clone via new `cloneRepoAtRef`, feature branches included) so deploy history/rollback stay in one place; `DeployRecord` gains `workflow`/`job` for cross-format rollback. **Artifact handover**: worker entries built by run jobs are read out of the sandbox (5 MB cap) and shipped via `entryOverrides` — deploys ship what CI built, not the stale committed file. **Safety**: alarm watchdog fails interrupted/over-budget runs + destroys orphaned sandboxes; cancel (CLI `gitflare ci cancel`, `/control/ci/cancel`, runs-page button); push-storm supersede; fail-closed — committed ci.yml + CI disabled means deploy.yml does NOT run ungated; delegated deploys count as green ONLY on `status: "success"` (a "CD not enabled" skip fails the job loudly). **Status postback** via the legacy Commit Status API (context `gitflare/ci`; classic PAT can't use Checks), soft-fail. **CLI**: `gitflare ci enable/disable/run/list/cancel`; enable provisions `[[containers]]` (public `docker.io/cloudflare/sandbox` image — no local Docker; `--instance-type`, default `standard-1`); migrations v3+v4 are emitted UNconditionally so cross-machine redeploys never miss applied tags; CONTROL_SECRET reuse is symmetric with `deploy enable`; disable drops only `CI_ENABLED` (idle containers cost $0). 51 new unit tests (97 total worker-side). Worker bundle 901 KB → 1557 KB (sandbox SDK +433 KB, capnweb +104 KB, containers +75 KB); ~400 KB gzipped, fine for Workers limits. **Deliberate deviations from §4 v0.3**: `runtime: worker` rejected with a clear error (Dynamic Workers can't run npm — no Linux userland); per-job `image:` rejected (containers pin the image at Worker-deploy time); ArtifactFS mount not used (sandbox `git clone`; cold container start pulls the image from Docker Hub — minutes, not the §4 \<500ms); jobs in one run share a sandbox/workspace (GHA users expect isolation — M9); Pages artifact handover, R2 build cache, Browser Run, Actions importer, per-job status contexts, `gitflare sandbox` verb → M9. **Post-review hardening** (a 4-lens adversarial review of the diff, 18 findings, all verifier-confirmed): FIXED — watchdog/pipeline verdict races (a `finalizedRunIds` fence stops a lagging coroutine resurrecting a run the watchdog failed, and the watchdog no longer erases a concurrent cancel); a fallback alarm armed in `begin()` so eviction before the deadline alarm can't leave a zombie "running" record; `ARTIFACTS.get` moved inside the try (+ a last-resort commit status) so a push can't vanish traceless; tag/non-branch pushes skipped at the webhook (were minting failed runs + red statuses); artifact handover gated on the whole `needs` closure of run jobs succeeding (was shipping a failed build's output); the dead "fall back to deploy.yml" path on missing SANDBOX replaced with an honest fail-closed error; delegated-deploy response mapping extracted to a tested pure fn (`ci/delegation.ts`, all four shapes incl. skipped-is-not-green); the remote-URL allowlist tightened to reject shell metacharacters; duplicate webhook deliveries de-duped by branch+sha; dashboard cancel gated behind Access (was open on public mirrors — CLI `gitflare ci cancel` uses CONTROL_SECRET); CONTROL_SECRET now always (re)written on enable and config persisted the moment the Worker goes live (was strandable across machines / on a partial enable). RESIDUAL (documented, not yet fixed — acceptable for a not-live-validated milestone): a run still queued in-memory when the DO is evicted (e.g. by a redeploy) is dropped with no record; webhook delivery order isn't guaranteed by GitHub, so the supersede optimization is best-effort (out-of-order delivery can run the older sha last); cancel is a no-op during a run's brief pre-clone window and cannot un-ship a deploy job that already returned success (the run message says so). **Live-validated end-to-end** (2026-07-17, `sinameraji/kimiflare` on a real Workers Paid account): `gitflare ci enable` provisions the `[[containers]]` app from wrangler TOML; a push → webhook → sync → CI run #1 succeeded in 12s — the Sandbox booted (`node v22`), cloned the mirror at the pushed sha with a short-lived read token (URL logged WITHOUT credentials — env-scoped auth + scrubbing confirmed), and ran all 4 `run:` steps with streamed line-buffered logs, resolved env vars, and captured exit codes. Findings fixed along the way (all on this branch): (1) container deploy needs BOTH **Account → Cloudchamber:Edit** and **Account → Containers:Edit** token permissions (Containers alone → `Forbidden`); (2) the container app name can't contain consecutive dashes, so we emit an explicit sanitized `name` (the `owner--repo` worker name has `--`); (3) a latent M1 sync bug — `artifactsRepo.remote` on the current Artifacts beta is a lazy RPC proxy that stringifies to `[object JsRpcProperty]`; now threads the REPO_MAP remote string (would have broken any live v0.2 deploy too); (4) the push webhook awaited the full sync and blew past GitHub's 10s delivery timeout on a real repo (504 + redeliveries) — now responds 202 immediately, sync + dispatch in `waitUntil`; (5) commit-status postback soft-fails observably (the run logs *why* — here, an expired GITHUB_TOKEN). Config also cross-checked via `wrangler deploy --dry-run` (both CI-provisioned and default TOML resolve all four DO exports incl. unbound-Sandbox migration v4). **Not yet exercised live** (deferred soak): the deploy-job → DeployDO delegation, the alarm watchdog firing, cancel mid-run, hour-scale runs / `sleepAfter` container-activity behavior, and commit statuses actually rendering (needs a fresh GITHUB_TOKEN). |
 
 ### What's in the repo right now (as of M0)
 

README.md

@@ -22,7 +22,7 @@ GitFlare ships in versions. Each one stands alone — if the next one never gets
 |---|---|---|
 | **v0.1** | ✅ **shipping — you are here** | **Read replica.** One command mirrors a GitHub repo into your Cloudflare account: Artifacts for git storage, a Worker that takes GitHub webhooks + serves a dashboard, file browsing with syntax highlighting, README rendering (images proxied through your Worker), sync status. Optional Cloudflare Access gates the dashboard for private repos. If GitHub is down, reads + clones still work. |
 | v0.2 | 🧪 code-complete (pending a live run) | **CD that doesn't depend on GitHub.** Push → your Worker deploys to your own account: Workers + Pages (with preview deploys), bindings (vars/KV/R2/D1/DO/services), opt-in D1 migrations, live deploy logs over WebSocket, plus `deploy run` (the GitHub-down escape hatch), `deploy list`, and `deploy rollback`. Deploys **pre-built** artifacts via `.gitflare/deploy.yml`; arbitrary build steps arrive with v0.3 CI. |
-| v0.3 | 📋 planned | **Generic CI.** A small declarative workflow format that runs tests on Cloudflare Sandboxes (full Linux) or Dynamic Workers (fast JS path). Build cache in R2. Browser Run for E2E. |
+| v0.3 | 🚧 in progress (core CI live-validated) | **Generic CI.** `.gitflare/ci.yml` with jobs / `needs:` / `run:` steps, executed on Cloudflare Sandboxes (full Linux containers on your account) — validated end-to-end: push → sandbox boots → clones → runs your steps with live logs. Deploy jobs gate on test jobs, and a Worker deploy ships **what CI just built** (not the stale committed file). Cancel, run history, GitHub commit statuses. Still to come in v0.3: R2 build cache, Browser Run for E2E, a GitHub Actions importer, Pages build artifacts. |
 | v0.4 | 📋 planned | **Multi-user teams.** PRs, reviews, comments — native to GitFlare, bidirectionally mirrored to GitHub. Stacked diffs. "Open PR in sandbox" one-click ephemeral env. |
 | v0.5 | 📋 planned | **Cross-tenant collaboration via Cloudflare Mesh.** Alice and Bob on separate Cloudflare accounts; private repos served Mesh-only with per-identity policies instead of SSH keys. |
 | v0.6 | 📋 planned | **Public repos + discovery.** A real code browser for the public web, search, forks across accounts. |
@@ -83,12 +83,35 @@ GitFlare never sees your code, your token, or your traffic. It's an MIT-licensed
 
   Deploys and their **live logs** show up at `<dashboard-url>/r/<repo>/deployments`.
 
+- `gitflare ci enable` — turn on generic CI (v0.3, requires the Workers Paid plan for Containers). Commit a `.gitflare/ci.yml` and every push runs your jobs in a **Cloudflare Sandbox on your own account** — full Linux, Node 20 + Python 3.11 — no GitHub Actions involved:
+
+  ```yaml
+  on: push
+  branches: [main]
+  jobs:
+    test:
+      steps:
+        - run: npm ci
+        - run: npm test
+    deploy:
+      needs: [test]          # deploys only if tests pass
+      steps:
+        - cloudflare/deploy:
+            project: my-worker
+            kind: worker
+            entry: dist/worker.js
+  ```
+
+  If a job builds `entry` (e.g. `npm run build`), the deploy ships the freshly built file from the CI workspace, not the committed copy. When `ci.yml` exists, it owns the pipeline — `deploy.yml` no longer runs ungated. Deploy jobs also need `gitflare deploy enable` (that's where the deploy token lives).
+
+- `gitflare ci run` / `list` / `cancel` — trigger the pipeline for the current Artifacts HEAD (GitHub-down escape hatch), review runs, or stop a runaway one. Runs + **live logs** stream at `<dashboard-url>/r/<repo>/ci`, and results post back to GitHub as commit statuses (`gitflare/ci`) when GitHub is reachable.
+
 ## Contributing
 
-Pre-alpha, built in the open, and there's a lot of obvious next work. Cloudflare Access (M5), syntax highlighting, the image proxy, and the full v0.2 CD feature set have landed — see [PLAN.md §12](./PLAN.md#12-milestones-and-development-log) for current status. PRs and issues are welcome — particularly on:
+Pre-alpha, built in the open, and there's a lot of obvious next work. Cloudflare Access (M5), the full v0.2 CD feature set, and the v0.3 core CI (M8: sandbox jobs, needs-gated deploys, artifact handover) have landed — see [PLAN.md §12](./PLAN.md#12-milestones-and-development-log) for current status. PRs and issues are welcome — particularly on:
 
-- **Live-validating M5 + v0.2** against a real Cloudflare account. The Access apps/policies API, the Workers Scripts upload, Pages Direct Upload, and the D1 query path are coded to spec but need an end-to-end run.
-- **v0.2 build steps.** CD currently deploys pre-built artifacts; running `npm run build` needs a Linux runtime (Sandboxes) — that's [v0.3](./PLAN.md#v03--generic-ci-tests-lint-build).
+- **Live-validating M5 + v0.2 + M8** against a real Cloudflare account. The Access apps/policies API, the Workers Scripts upload, Pages Direct Upload, the D1 query path, and now the Containers provisioning + Sandbox exec paths are coded to spec but need an end-to-end run.
+- **v0.3 remainder.** R2 build cache keyed on lockfile hash, Browser Run for E2E, the GitHub Actions importer, and Pages build-artifact handover — see [PLAN.md §12 M8 notes](./PLAN.md#12-milestones-and-development-log).
 - **Private `git clone`.** Access gates the dashboard, but clone still hits Artifacts directly. Closing that needs an Access service token / Mesh path (v0.4+).
 - **Custom domains** in front of the Worker, and **better empty states / error messages** anywhere in the CLI or dashboard.
 - **Anything in [PLAN.md §8 Open Questions](./PLAN.md#8-open-questions-to-resolve-before-v01-starts)** you have a strong opinion on.