Add runner update channels for canary field testing on Bencher Cloud

[epompeii]

What

Adds runner update channels so runner changes can be field-tested on Bencher Cloud before a versioned release, mirroring how the API server is dogfooded via cloud branch deploys.

• New runner up --update-channel <stable|canary> flag (env BENCHER_UPDATE_CHANNEL, default stable, conflicts with --no-auto-update)
• The stable channel keeps today's behavior: exact version match against the server, updates from versioned GitHub Releases
• The canary channel converges by content: the runner reports the SHA-256 of its own binary in Ready metadata, and the server compares it against the published checksum of a rolling canary prerelease (TTL-cached per architecture), sending the existing Update directive on mismatch

How

• Protocol: JsonRunnerMetadata gains optional channel and checksum fields; absent means stable, so old runners and old servers interoperate unchanged and the stable wire format is byte-identical
• Server: new RunnerUpdate context (URL construction + canary checksum cache with injectable Clock), channel-aware handle_ready with an injectable checksum fetcher for unit testing, and a new runners.update_base_url config knob. Checksum fetch failures now log, count (runner.self_update.check.failed; updates count as runner.self_update.sent), and skip the update offer instead of erroring the channel into a reconnect loop (this also fixes the stable path when GitHub is unreachable)
• CI: the runner build now runs on cloud pushes (path-filtered, build-only skips the KVM scenario tests already run on devel) and is versioned canary; a new publish_runner_canary job clobbers the assets on a rolling canary prerelease, binaries before checksums so a partial upload never advertises a missing binary. The tag is canary rather than cloud because a tag named like the branch is ambiguous in git ref resolution and would break git checkout cloud in rebase_cloud.yml
• Ops: cargo ops deploy|start --update-channel and an optional update_channel field in runners.json write BENCHER_UPDATE_CHANNEL into the systemd drop-in
• Docs: runner up reference, Runner Protocol reference, and the Self-Hosted Runners explanation updated in all 9 languages; changelog entries added under Pending v0.6.9

Self-update remains idle-only on both sides (server sends Update only from wait_for_ready; the runner handles it only in AwaitingJob), so a canary rollout never interrupts a running benchmark.

Field testing flow

1. Merge a runner change to devel, reset cloud to devel, and push
2. Cloud CI publishes the new canary assets while the API deploys as usual
3. At their next idle poll, canary channel runners detect the checksum mismatch, self-update, and reconnect
4. Rollback is re-uploading previous assets with --clobber or pushing a revert to cloud

[github-actions]

🤖 Claude Code Review

PR: #927
Base: devel
Head: claude/runner-canary-channel
Commit: 7c7c98e360c9807a4a0436b3c67ab16263e8af92

---

I now have a thorough understanding of the PR. Let me verify one edge in the canary caching logic around the documented "downgrade" window before finalizing.

The code and tests are consistent. Here is my review.

---

Code Review: Runner Update Channels (canary field testing)

Summary

This PR adds a canary update channel alongside the existing stable channel, letting selected runners track the rolling cloud-branch build (converging by binary checksum) instead of waiting for versioned releases (converging by version). The work is well-structured: a new UpdateChannel valid type, a RunnerUpdate context holding a per-arch TTL checksum cache, a refactored handle_ready that branches per channel, runner-side checksum reporting, CI publishing to a canary prerelease, and runner_ops wiring.

Overall this is high-quality, well-tested work that closely follows the project conventions. Findings below are minor.

Strengths

• Excellent test coverage at every layer: the valid type, RunnerUpdate cache (hit/miss/expiry/per-arch), handle_ready channel matrix (via an injected FakeFetcher — nice dependency-injection refactor for testability), runner-side metadata construction, CLI parsing, systemd conf generation, and end-to-end WebSocket integration tests. Deterministic clocks used throughout (Clock::Custom, DateTime::TEST) per CLAUDE.md.
• Backward compatibility is carefully preserved and tested: stable runners stay checksum-free on the wire (skip_serializing_if), legacy runners with no channel field default to Stable (ready_legacy_metadata_deserializes, ready_stable_metadata_wire_format_is_legacy, absent_channel_defaults_to_stable).
• Graceful degradation: a checksum-fetch failure now skips the update (warn + otel counter) rather than erroring the channel, a genuine improvement over the prior ?-propagation. Covered by integration tests using an unroutable base URL.
• Thoughtful comments documenting non-obvious invariants (trailing-slash Url::join, TTL as rollout propagation bound + self-healing checksum mismatches, CI upload ordering of binaries-before-checksums).
• Conventions followed: strong types (UpdateChannel, Sha256, Architecture), thiserror with wrapped source errors (SelfChecksumError), #[expect] over #[allow], otel adverse-event counters with dimensional update.channel attribute, camino/Utf8Path boundaries, no emdashes in added text.

Minor observations

1. ValidError::UpdateChannel(String) wraps String, not an error type. CLAUDE.md says variants should wrap the original error type. Here there is no underlying error (the payload is the rejected input value), and this exactly matches the established sibling pattern in ValidError (Sandbox(String), ProjectKey(String), …). Consistent and acceptable; flagging only for completeness.

2. Canary republish "downgrade" window. When a new canary is published, the server's cached checksum is stale for up to CANARY_CHECKSUM_TTL_SECS (120s). A runner already reporting the new checksum during that window will be told to "update" to the stale cached checksum, download the new binary, fail verification, and retry. This is correctly documented and self-healing (runner_update.rs comment), but it does mean transient checksum mismatch runner logs and a wasted download right after each canary deploy. Fine for a small canary fleet; worth keeping in mind if the fleet grows.

3. update_base_url trust model. The new operator-configurable base URL means the checksum and binary come from the same origin (TOFU), so a compromised/mis-set mirror could serve a malicious binary plus a matching checksum. This is unchanged from the pre-existing stable-channel behavior (GitHub Releases) and is operator-controlled config, so not a regression, but it's the one place the update mechanism's integrity rests entirely on the download origin.

4. CI publish_runner_canary skip semantics are correct: when runner code is unchanged on a cloud push the runner job is skipped, needs.runner.result == 'success' is false, publish is skipped, and ci-success accepts skipped. Good. Confirm the runner.yml build actually emits the .sha256 companion files and names artifacts runner-canary-linux-* (the publish job's pattern/upload depend on it) — the VERSION="canary" rename handles this, but it's the one link I couldn't fully see in the diff.

Correctness spot-checks (all pass)

• runner.channel.unwrap_or_default() → Stable for legacy runners ✓
• Stable path emits one checksum fetch per update cycle (runner transitions to UpdateSent and restarts), not per-poll — so the "not cached" stable path is not a fetch-storm ✓
• ensure_trailing_slash handles empty///non-slash paths ✓
• Non-plus build unaffected: pub mod runner is #[cfg(feature = "plus")], so dropping the #[cfg(feature = "plus")] from the Sha256 import in websocket.rs is safe ✓
• Self-update verifies the downloaded binary against the server-sent checksum; version: "canary" is display-only ✓

Recommendation

Approve. Clean, conservative, thoroughly tested. Consider (optionally) a single-flight guard or shorter propagation note if the canary fleet ever scales, but nothing here is blocking.

Note: I was unable to run cargo check/cargo nextest in this environment (sandbox blocked cargo), so the compile/test verification above is by static reading of the diff and the included tests, not an actual build.

---

_{Model: claude-opus-4-8}

[github-actions]

Bencher Report

Branch	claude/runner-canary-channel
Testbed	intel-v1

Click to view all benchmark results

Benchmark	Latency	Benchmark Result microseconds (µs) (Result Δ%)	Upper Boundary microseconds (µs) (Limit %)
Adapter::Json	📈 view plot 🚷 view threshold	4.76 µs (+2.41%) Baseline: 4.64 µs	4.90 µs (97.03%)
Adapter::Magic (JSON)	📈 view plot 🚷 view threshold	4.55 µs (+0.83%) Baseline: 4.52 µs	4.72 µs (96.40%)
Adapter::Magic (Rust)	📈 view plot 🚷 view threshold	25.92 µs (+1.18%) Baseline: 25.62 µs	26.76 µs (96.88%)
Adapter::Rust	📈 view plot 🚷 view threshold	3.51 µs (+0.47%) Baseline: 3.49 µs	3.60 µs (97.39%)
Adapter::RustBench	📈 view plot 🚷 view threshold	3.50 µs (+0.24%) Baseline: 3.49 µs	3.60 µs (97.21%)

🐰 View full continuous benchmarking report in Bencher

[epompeii]

Review follow-up on the five observations:

1. Confirmed correct: the changes filter has no explicit base, so cloud pushes diff against the default branch (main), which CI resets to cloud after each successful deploy. That yields "changed since the last deploy" semantics. Now documented in a comment on the runner job.
2. Intentional asymmetry; added a note to the stable_update doc comment.
3. No action needed, as noted.
4. Keeping the direct UpdateChannelKind constructions: inside the per-channel functions they are clearer than routing through the shared helper.
5. Keeping the tuple in runner_ops: it is an internal helper in an admin task crate and the types are distinct.

[epompeii]

Follow-up on the latest review round:

1. Fixed: RunnerSelfUpdateSent now increments after a successful tx.send, so a serialization or send failure no longer counts as a sent update.
2. Acknowledged as designed: a canary runner without a self-checksum stays put, with warnings on both sides.
3. As documented: the stable path intentionally skips caching.
4. Keeping the wasm export for consistency with sibling validators.

Local gates confirmed: cargo clippy --all-features -- -Dwarnings and cargo check --no-default-features are both clean.

[epompeii]

On observation 4: confirmed locally. In runner.yml, the build renames the binary to runner-${RUNNER_VERSION}-linux-{x86-64,arm-64} (with VERSION=canary on cloud pushes), writes the sha256sum companion next to it, and uploads both files under an artifact of that same name, which matches the runner-canary-linux-* download pattern in publish_runner_canary. Observations 1 through 3 need no change, as noted in the review itself.