docs(api): reconcile snapshot-run with existing ato-api run infra (#896)#911
Closed
Koh0920 wants to merge 875 commits into
Closed
docs(api): reconcile snapshot-run with existing ato-api run infra (#896)#911Koh0920 wants to merge 875 commits into
Koh0920 wants to merge 875 commits into
Conversation
…ed helpers (#685) Two infrastructure primitives had drifted into security/correctness-relevant divergence across crates (#658). Each now resolves through one helper in capsule-core foundation, with the safe semantics pinned by tests. Recursive tree copy (capsule_core::common::fs::copy_dir_recursive with an explicit SymlinkPolicy): the local-source staging path used to copy symlink *target contents* (is_file() || is_symlink() -> fs::copy), so a source tree containing `cfg -> ~/.ssh/config` embedded the foreign file into the staged capsule. Symlinks are now never followed; callers choose Skip (fail closed, used by local staging and the github cache) or Preserve (recreate the literal link, Unix only). The executable bit still survives via fs::copy. Readiness probe (capsule_core::common::readiness::http_status_indicates_ready): the same manifest readiness_probe evaluated differently per execution path (strict 200/201/204 vs. any 200..500 where 404 counted as ready). All paths now share Kubernetes httpGet semantics: ready iff 200 <= status < 400. Redirects count as ready; a missing probe path (404) is misconfigured, not ready. nacelle mirrors the predicate inline (it does not depend on capsule-core) and keeps a comment pointing at the source of truth. Tests: capsule-core unit tests pin both edge cases (skip drops symlinks, preserve recreates the literal target, redirect-ready, 404-not-ready); an ato-cli regression test asserts local staging never materializes a symlink. Other duplicate clusters listed in #658 (atomic-write, tilde expansion, version compare, streaming sha256, nacelle home dir, MCP host-cleanup) are maintainability-only and left for follow-ups to keep this diff scoped to the two behavioral drifts the issue prioritizes.
Remove dead code surfaced by the code review (no production callers, grepped across the workspace including tests): - application/ports.rs: InteractionPort trait + SharedInteractionPort alias (zero implementors/references). - application/pipeline/phases/install.rs: run_publish_install_phase, a dead futures::executor::block_on wrapper with a hardcoded None arg. - application/execution_receipts.rs: write_receipt_atomic / read_receipt default-root wrappers (tests use the *_at variants; mark read_receipt_at allow(dead_code) to match its test-only sibling). - routing/router.rs (capsule-core): resolve_target_field + resolve_command_string TOML command-form parsers (zero callers), and the now-unused module-local 'use shell_words'. - application/preflight.rs: preflight_oci_image_resolution + OciImageResolutionReport and their in-file-only tests; trim the now-unused imports. - adapters/runtime/executors/oci_compose_runner.rs: resolve_images_for_compose, a copy of the non-reuse branch of resolve_images_with_lock_replay; route the three in-file tests through the production fn with existing_lock: None. - application/source_inference/mod.rs: use_global_run_state field (compile-time-asserted constant-true), WORKSPACE_RUN_SOURCE_INFERENCE_DIR, the USE_HOME_RUN_STATE const + assertion, and the dead cwd-based else branch in materialize_run_result. Run-attempt state is now unconditionally under ~/.ato/runs/source-inference/. Adds a regression test asserting the global root is used independent of workspace_root. Behavior is unchanged: the SessionInfo status/readiness_confirmed pair (issue item 8, lower priority) is intentionally left as-is to avoid altering the serialized 'ato run' JSON contract.
…-wire type (#692) The ExecutionPlan consent-required payload was hand-mirrored in three places with duplicated validation over two channels: - consent_store::ConsentRequiredLine (stdout CONSENT-REQUIRED: producer) - runner_agent::ConsentRequest + is_valid (stdout consumer) - ato-desktop cli_envelope::ConsentRequiredDetailsDto + its empty-field check (stderr details consumer) A schema bump or field rename compiled on all three sides while silently making the validators reject, so no consent prompt would appear. Move the shared identity 5-tuple + summary, the schema/reason constants, and the single validation into capsule-wire::consent (the DAG-root crate whose purpose is to make exactly this drift impossible). Both channel envelopes flatten the shared ConsentIdentity, so the wire bytes are unchanged; capsule-core re-exports CONSENT_REF_SCHEMA from the wire constant so the consent_ref hash input cannot diverge from the validator. Behavior preserved: stdout line and stderr details parse and validate exactly as before (unrelated E302 still falls through to the fatal-toast path; empty identity fields still yield no modal). Scoped to the consent payload unification (issue item 2); the sentinel/error-substring and consent-pinning items are left to follow-ups as the issue staged.
) The source/python provision command quoted its setuptools upper bound with POSIX single quotes (`'setuptools<72'`). cmd.exe does not treat `'` as a quote character, so the `<` was exposed and parsed as input redirection, breaking `ato run` on windows/x86_64 with 'Access is denied.' / 'The system cannot find the file specified.' (issue #629). Quote the constraint with double quotes, honoured by both POSIX sh and cmd.exe, and route the Windows lifecycle command through `cmd /D /S /C` via `raw_arg` so the inner quotes reach cmd verbatim instead of being backslash-escaped by Rust's argument quoter. `/S` makes cmd strip only the outer quote pair, leaving "setuptools<72" intact as one redirection-safe argument.
…runs (#695) * fix: auto-provision headless state for runner --sandbox runs A Connected Runner spawns `ato run <source> --sandbox` with no `--state` binding, so any recipe declaring a `[state.*]` block hard-errored on the unbound persistent state ("requires an explicit persistent binding") or failed container creation on the un-creatable `/var/lib/ato/state` ephemeral base. The desktop/session path already auto-provisions a per-source state directory; the headless run path did not. Provide headless state auto-provisioning equivalent to the desktop path: for sandboxed runs, every declared state still unbound after explicit `--state` bindings is bound to a stable `~/.ato/state/run/<name>/<state>` directory. Applied in both the authoritative-input and non-authoritative branches of run_prepare_phase. Fails closed on any state kind other than filesystem. Non-sandbox `ato run` keeps its existing fail-closed behavior. * test: pin #687 fix wiring via resolve_explicit_or_auto_state_source_overrides The prior tests exercised auto_provision_headless_state_overrides in isolation (a symbol new to this PR), so they could not regress if the sandbox dispatch were removed. Add two tests that drive the actual fix wiring: sandbox mode tolerates a declared-but-unbound persistent state (deferring it to the auto-provisioner), while non-sandbox still fails closed with the explicit-binding error.
) On macOS the native seatbelt sandbox redirects the workload child's stdout/stderr to a per-workload log file and registers it as a sync child. nacelle's start_log_forwarding only drains live pipes (dev-mode async children), so for production sandbox runs it forwarded nothing: when the child crashed, the real failure (e.g. a Python traceback) lived only in an opaque temp log under $TMPDIR/nacelle-runs/<id>/logs/ that nacelle never read back and the runner log tail never saw. The exit code was reported with no diagnostics on any runner. nacelle: on a non-zero child exit (both the readiness Exited branch and the final wait), read the tail of that log file and echo it to nacelle's stderr (the stream ato-cli forwards into the runner log), framed and [child]- prefixed. Capped at 16 KiB from the end, starting on a line boundary. No-ops on success or when no log file exists (dev-mode pipes). With the child output now observable, the underlying macOS early-exit is a read-only-filesystem error: ato-cli mounts the writable per-run session-data dir at the guest path /runs/ato/session and points ATO_DATA_DIR / DATABASE_PATH there. Linux bwrap remaps that to the host dir, but macOS seatbelt has no mount namespace -- the child sees host paths -- so a stateful capsule tried to mkdir /runs on the read-only host root and uvicorn exited before readiness (OSError [Errno 30] Read-only file system: '/runs'). On macOS, point the data-path env at the writable host session dir instead. Verified locally on macos/aarch64: hello-capsule now reaches readiness and writes app.db under the host session dir; a forced crash surfaces the full traceback in the runner output.
* fix: wire readiness emission into foreground NodeCompat runs Foreground NodeCompat runs (Connected Runner dispatch and `ato run … --sandbox -y`) went through `node_compat::execute`, which blocked on the child with inherited stdio and emitted no readiness signal: no TCP probe, no LifecycleEvent pump, no `LIFECYCLE: ready port=N` line, and no V2 receipt readiness re-stamp. The runner agent therefore never saw the canonical ready line, and a healthy node capsule serving its declared port sat in "Assigned to runner…" until the 600s ready timeout. The background path (`spawn_background`) already wired this via `spawn_host_lifecycle_events`; the foreground path did not. Add `node_compat::spawn_foreground`, mirroring `spawn_background`'s readiness wiring (a `CapsuleProcess` whose `event_rx` is fed by `spawn_host_lifecycle_events`) but inheriting stdio so guest output still reaches the terminal. Route the foreground NodeCompat branch through it and `complete_foreground_source_process`, exactly like the Native and host fallback branches: TCP-probe the declared port, print the machine `LIFECYCLE: ready` line, and re-stamp the V2 receipt from the observed event. Capsules with no declared port keep honest StartedWithoutReadiness (no fake ready). The shared command assembly is extracted into `build_orchestration_command` so both spawn paths stay in lockstep. * test: drive readiness-port derivation the foreground node pump consumes The two #623 regression tests asserted derive_launch_spec(&plan).port, which this PR never modifies — they passed against the old buggy code and exercised none of the new path. Extract the readiness-port derivation build_orchestration_command performs (override_port(launch_spec.port), the value spawn_host_lifecycle_events actually probes) into resolve_readiness_port and assert against that instead. Add a session-override-precedence case (the #565 remap) that the raw launch_spec.port assertion structurally cannot express.
…694) * fix(cli): tear down bwrap-sandboxed keep-alive server on Linux stop On Linux a non-network `ato run` launches its workload through `bwrap --unshare-all`, which leads its own process group and hides the sandboxed command line and cwd behind a PID namespace. The import keep-alive stop path only signaled process groups it could verify via the ATO_IMPORT_SESSION_ID marker or a shadow_dir reference, neither of which is visible for a namespaced child, so the recorded bwrap group was skipped and the preview server kept listening after `ato stop` returned success. Verify a recorded process group as Ato-owned when any of its members descends, via the ppid chain, from the confirmed-owned `ato run` pid. This is gated on ato_run_owned (root pid + start time matched), so it stays fail closed and can never kill a group rooted under a recycled or unrelated pid. Capture ppid in the ps snapshot to support the walk. Re-enables the two import keep-alive teardown tests that were skipped on Linux in #636 so they guard against regression. * test(cli): clarify namespaced grandchild is the unverifiable member The accept-sandboxed-descendant test comment implied the bwrap monitor itself carries no shadow_dir reference. In production the bwrap argv does carry the host shadow_dir via --ro-bind <shadow_dir> /app and may match on that; the genuinely unverifiable member is the namespaced grandchild whose host-visible argv uses guest paths and whose cwd resolves inside the namespace mount. Note this in the fixture so future readers do not conclude bwrap itself was unverifiable. * fix(cli): kill bwrap-sandboxed keep-alive subtree by host pid on stop The Linux import keep-alive server survived `ato stop` because the host teardown only signaled process groups via kill(-pgid). bwrap launches the workload with --unshare-all --new-session: --new-session makes the sandboxed server setsid() into a process group whose pgid is not the recorded bwrap pgid, so kill(-pgid) never reaches it. --die-with-parent is the only other teardown and does not propagate to namespaced grandchildren on a force kill. stop_import_preview_session_record now enumerates every host pid descending from the recorded ato run pid via the ppid chain (host ps sees namespaced pids) and SIGKILLs them directly by host pid. The descendant set is captured before the root is killed, since killing the root reparents the subtree to pid 1 and severs the chain that identifies it. A bounded post-kill reap loop re-SIGKILLs any captured pid still alive until the subtree is gone, which is what closes the keep-alive preview port. * fix(cli): record durable workload pids and SIGKILL them on stop The Linux import keep-alive server kept listening after `ato stop` because the stop path had no durable handle on the sandboxed workload. It relied on process-group signaling (kill(-pgid)) — which misses a server that bwrap --new-session puts in its own setsid group — and on a ppid walk from the recorded ato_run_pid, which can come up empty: the keep-alive supervisor is the import's detached child, so once import returns "running" the supervisor is reparented and the stop-time ppid chain no longer reaches the subtree. Capture the workload subtree's host pids at spawn time instead. The import readiness loop now walks the supervisor's descendants (the bwrap wrapper plus its namespaced children) via probe_workload_pids and stores them, each paired with its OS start time, in the new ImportPreviewSession.workload_pids field. stop_import_preview_session_record now SIGTERMs then SIGKILLs each recorded workload pid directly by host pid — guarded by the recorded start time to defeat pid reuse — in addition to the existing process-group and ppid-walk teardown, and reaps them in the bounded post-kill loop. Killing the bwrap wrapper pid directly collapses its PID namespace, so the sandboxed server dies and the port closes even when the supervisor is already gone.
The Connected Runner now reports its agent_version on register + heartbeat, and
acts on a server self-update directive: when the heartbeat response carries
{ update: { target_version } } and the runner is idle, it self-updates
(axoupdater, async) and re-execs into the new binary (startup reconcile settles
any orphaned leases). A given target is attempted once so an unsatisfiable
target (newer than the latest release) is not retried every heartbeat.
- agent_version() = CARGO_PKG_VERSION, sent on register + heartbeat
- HeartbeatResponse.update parsed → HeartbeatOutcome::Ok.update_target
- run_self_update_async() in update.rs (async-safe; needs install receipt)
- reexec_serve() (unix exec / non-unix spawn+exit)
Pairs with ato-api (runner self-update directive) + ato-pwa (Update button).
3 new tests; 53 runner_agent tests green.
Addresses review: - Directive field target_version → minimum_version: the runner self-updates to LATEST (axoupdater), so the server value is a floor it must reach, not an exact version. The runner no longer implies it installs an exact target. - run_self_update_async returns a classified SelfUpdateOutcome (Updated / AlreadyLatest / NoReceipt). The serve loop no longer gives up forever on every failure: NoReceipt is terminal; a transient failure retries after a 5m cooldown; already-latest-but-below-minimum retries after 1h. Reset when the requested minimum changes. 53 runner_agent tests green; rustfmt clean.
…sted runner failure reports (#707) PR #697 added surface_child_log_tail in nacelle, which echoes a sandboxed child's raw stdout/stderr tail to nacelle's stderr ([child] …). ato-cli forwards that stream into the runner log and persisted run-failure reports (R2/D1). The existing scrub_secrets only redacted ato_rnr_ runner tokens, so a traceback carrying an API key, a GitHub token, an OpenAI/Anthropic key, or a .env-style KEY=value secret was persisted in the clear. Strengthen scrub_secrets — the single common boundary every runner sink already routes through before persistence — to also redact GitHub tokens (ghp_/gho_/ghu_/ghs_/ghr_/github_pat_), OpenAI & Anthropic keys (sk-…, sk-ant-…), npm tokens, AWS access-key ids (AKIA…), Bearer/Authorization headers, URL userinfo credentials, and .env-style KEY=value / KEY: value secret assignments. The surrounding traceback shape is preserved; only the secret value is replaced with [REDACTED]. The dedicated ato_rnr_[REDACTED] runner-token form is preserved (the regex passes skip already-redacted spans). All four sinks are covered because they funnel through scrub_secrets: - saved log_tail: BoundedLog::line scrubs every persisted run-log line - failure report / run error: scrub_secrets(&format!("{err:#}")) reports - runner lease error: LeaseReport::Failed message scrubbed in report_lease_status Add tests asserting an API key, ghp_…, sk-…, and a .env-style secret are redacted in the persisted form (including via the BoundedLog write path). Fixes #702
… honor durability (#706) * fix(cli): key headless state auto-provisioning on source identity and honor durability Headless state auto-provisioning (#695) keyed the state root on manifest.name alone (~/.ato/state/run/<name>/<state>), so two different sources sharing a name SHARE state (isolation hazard), and it bound durability="ephemeral" state to the stable persistent root, so ephemeral state persisted across runs. Derive a stable, source-scoped headless_state_instance_id via the existing canonical_hash helper (JCS + blake3) over (normalized_source_ref, owner scope, selected target, profile, runner namespace). Per-execution facts (execution id, ports, session/process/container ids, dynamic env) are deliberately excluded so the same source reuses its state across runs and revisions; the materialized-source tree hash is used only as a fallback when the source ref is unstable. Split the path scheme by durability: persistent state lands under ~/.ato/state/run/<instance-id>/<state>, ephemeral state under a per-run ~/.ato/runs/<token>/state/<state> registered with the run cleanup scope. Explicit --state bindings still take precedence; kind != filesystem stays fail-closed. Fixes #700 * fix(cli): short-circuit headless state no-op path and isolate sample-recipe tests CI regression on macos+windows: session_start_allows_no_persistent_state_ without_attach_state failed reading a sample-recipe capsule.toml from a tmp dir that had been torn down mid-test. Two compounding causes: 1. auto_provision_headless_state_overrides derived the source key (and could hash_tree the source tree) even when there was nothing to provision. Add a pre-scan that collects unbound filesystem states first and returns a pure no-op when none remain; only compute headless_state_instance_id when an unbound PERSISTENT filesystem state actually needs it. kind != filesystem stays fail-closed in the pre-scan. The no-state and ephemeral-only paths now touch no source ref / source_tree_hash / ato_state_dir. 2. The two pgweb session tests materialized their recipe under the ambient ATO_HOME with no guard, so a parallel #[serial] ATO_HOME test could delete the tempdir between the recipe write and the manifest read (wider window on macos/windows). Make both pgweb tests hermetic: take env_lock, pin ATO_HOME to a private tempdir via TestEnvGuard, and mark them #[serial] — matching the other sample-recipe tests. Add run.rs regression tests proving the no-state, fully-bound, and ephemeral-only paths perform no source read (unstable ref + non-existent workspace would error if the key were derived; the no-op returns Ok).
feat(runner): self-update on a server-pushed target version
* feat(runner): attach scrubbed log tail to run-failure reports On a run failure the Connected Runner now reads the tail of the child's (already secret-scrubbed) run log and attaches it as error.log_tail on the existing POST /v1/runner-leases/:id/status callback. ato-api derives a diagnostic report from it (ato-api#38) so a failed run's logs can be viewed remotely in the PWA (ato-pwa#52). Best-effort: a missing/empty log just omits the field; the tail is capped at the API's 16 KiB limit, drops a partial leading line when truncated, and is scrubbed once more before send. read_log_tail split into a path-taking core for unit tests. 3 new tests; 54 runner_agent tests green. * review(#698): precise log-tail truncation doc + no-newline test Clarify read_log_tail_from: a truncated tail is marked, and the partial leading line is dropped only when an interior newline exists; a single over-long line (no newline) is kept verbatim under the marker rather than blanked. Add a test locking that behavior. * style(#698): rustfmt the new log_tail tests
) Add one end-to-end regression test that a NodeCompat capsule dispatched through the runner's real readiness pump reaches ready, the path that hung until the 600s runner deadline before #693 added node_compat::spawn_foreground. The test drives the real run_lease_child with a real `ato run <fixture> --sandbox -y` child (the exact spawn_run_child shape) against the installed-relaunch-node fixture (declared port 18880) and asserts LeaseReport::Ready { port: Some(18880) } - the assertion that previously timed out. #[ignore]d because it spawns the real ato binary plus a managed node toolchain (needs node + nacelle), matching the repo convention for toolchain-dependent E2E tests; run with --ignored. Fixes #703
…pids (#711) merge_workload_pids fixed a recorded keep-alive workload pid on first observation. If the first probe captured start_time_unix_ms=None and a later probe obtained Some, the value was never filled in, weakening the pid-reuse guard (process_start_time_matches) used at stop time. Backfill None -> Some on a re-observed pid; an already-recorded Some is still never overwritten.
feat(#699): headless runner enrollment login (ato runner login --enrollment-token)
#712) Squash-merged. The one red required check (cargo test `ato_managed_installer_extracts_verified_binary` on ubuntu/windows) is a pre-existing flaky podman-installer test unrelated to this change (macOS full pass; same test flakes on dev). #712 fix verified end-to-end (gitea PASS on Connected Runner).
Replace the single-slot `busy` boolean with an indexed SlotPool so a Connected Runner serves multiple concurrent apps per device. Per-slot proxy ports (base+index, range-validated), honest per-slot public URLs via --public-url-template ({port}/{slot}, placeholder-required) else slot-0-only legacy mapping, heartbeat advertises max_slots/active_slots. Default max_slots=1 preserves single-slot behavior. Partially addresses #632 (runner slice); follow-ups ato-api#59 (capacity dispatch) + ato-pwa#57 (multi-tile UI). Admin override authorized by repo owner: the two red CI jobs are pre-existing #708 debt (podman_install flaky + pkgs.txt CRLF), unrelated; macOS test + check + clippy + lints green.
`verify_binary_runs` execs a freshly-extracted podman binary with
`--version`. On Linux under parallel load that exec races the write and
fails with ETXTBSY ("Text file busy", os error 26) — a transient
condition that clears once the writer handle settles. The single-shot
exec turned that race into a hard `PodmanInstallError::Extract`, which
surfaced as a flaky `cargo test` failure on ubuntu/windows (different
podman_install test failing each run; macOS unaffected) and is also a
real robustness gap for actual installs on loaded Linux hosts.
Wrap the exec in `exec_retrying_busy`: a bounded retry (5 attempts,
20ms·attempt backoff) that retries ONLY the transient busy code
(ETXTBSY 26 on Unix, sharing-violation 32 on Windows) and surfaces every
other error immediately, so genuine exec failures are never masked.
Tests: classification (busy vs ENOENT/other), recovery after N transient
busies, no-retry on a real error, and give-up at MAX_ATTEMPTS. Focused
`cargo test -p ato-cli --lib podman_install` → 49 pass; clippy + fmt clean.
Refs #708 (podman_install flaky portion).
Add Bun as a managed auxiliary runtime tool declarable via `runtime_tools` (parse/fold/lock-pin/cache-invalidation), put the managed Bun shim on the source/node RUN PATH in node-compat (execute/spawn/foreground/background), exact-release pins for v0, plus a networked e2e and a shadow-test ATO_HOME isolation fix. Partially addresses #723 (Managed Cloud bun --version smoke is a tracked follow-up).
…orted_lease_kinds (#731 slice 1) (#733) * feat(runner): managed-state-root auto-bind + run_capsule lease parse + supported_lease_kinds First minimal slice of ato#731 (run OCI-image capsules on a Connected Runner). - ato run --managed-state-root <DIR>: auto-bind unbound persistent attach="explicit" state to a stable, path-safe dir <DIR>/<blake3(name\nversion\ntarget)>/<sanitized state_key>. Never lease-scoped; owner isolation is the caller's job (encoded into <DIR>), never derived from capsule input. --state always wins. Reuses the existing ensure_registered_state_binding path (no parallel logic). - runner: parse_run_capsule_command validates a stable capsule identity (capsule_id + owner_id required; version/revision/target/run_id optional; capsule_slug is display-only, never used for resolution or state). - runner heartbeat advertises supported_lease_kinds = ["run_source_sandbox"] so the control plane can capability-gate dispatch. run_capsule is parsed but NOT yet advertised (execution wiring is a follow-up). Tests: 9 new unit tests (path derivation stability/isolation, sanitize, managed auto-bind + explicit-wins, run_capsule parse, heartbeat advertising). Existing state/lease/heartbeat tests unaffected. * fix(runner): address review — namespace contract, collision-free segments, capsule_id immutability - managed_state_dir appends ONLY <target>/<state_key>. Per the namespace contract, <root> MUST already be scoped by the server-confirmed owner AND a stable immutable capsule identity. Dropped name/version hashing: those are capsule-controlled and not globally unique, so two distinct capsules sharing name/version/target would have collided. Docs + CLI help aligned. - path_segment is now collision-free + path-safe: <sanitized>-<blake3-16>, so inputs that sanitize alike (a/b vs a_b, . vs _) get distinct segments. - RunCapsuleCommand.capsule_id documented as a revision-resolved IMMUTABLE identity (else send revision and treat the pair as the identity). Tests updated: path_segment collision/path-safety; managed_state_dir appends only target+state_key + collision-free. All green.
…te lock path (#731 slice 2) (#734) * feat(runner): execute run_capsule leases + emit OCI readiness + managed-state lock path #731 follow-up: make run_capsule actually runnable on a Connected Runner. - resolve_lease_execution dispatches by lease kind: run_source_sandbox (source repo, no managed state) vs run_capsule (capsule ref + a runner-managed state root <base>/state/<owner>/<immutable-identity>; owner from the server-confirmed owner_id, identity = revision | capsule_id). The capsule ref is validated against flag/whitespace smuggling. spawn_run_child gains --managed-state-root. - Advertise run_capsule in SUPPORTED_LEASE_KINDS now that execution is wired, so the control-plane capability gate may dispatch it. - OCI executor now emits the machine-readable `LIFECYCLE: ready port=N` line so the runner monitor detects OCI readiness and reports a ready_url. Previously only the human "OCI service available" line was emitted, so OCI runs stayed provisioning forever on a runner. Runtime-verified. - Fix --managed-state-root on the lock-route path: the lock branch reused the materialized overrides and skipped the managed auto-bind, so a persistent OCI capsule failed E999 on a runner. Now augments the lock overrides with managed binds. Runtime-verified: dir created at <root>/<target>/<state_key>. - path_segment is now pub(crate), shared by the runner for the owner/identity segments so the managed namespace is consistent end to end. Tests: 14 unit tests (lease dispatch both kinds, ref validation, state-root identity fallback, heartbeat advertises run_capsule, path derivation). G3 and --managed-state-root additionally verified at runtime with a real OCI capsule. * fix(runner): immutable run_capsule run_ref + hard-fail managed-state lock load (review #734) - RunCapsuleCommand collapses capsule_id/version/revision/target into a single required `run_ref`: the runner executes it AND keys persistent state on it, so the executed artifact and the state namespace can never diverge (the prior split keyed state on `revision` but executed `capsule_id`, which could point at a different, mutable resolution). Contract: the control plane resolves any mutable slug to an immutable, point-in-time ref before dispatch. Tests use revision-pinned refs; no mutable success example. - Lock-route `--managed-state-root`: a manifest that cannot be read is now a hard error (`with_context(...)?`), not a silent skip that would resurface later as a confusing unbound-state failure. `--managed-state-root` is an explicit contract for non-interactive runner execution.
Add Ubuntu + NVIDIA GPU host provisioning to the Connected Runner
surface so users can turn a bare Ubuntu server with RTX 3060s into a
GPU-capable Ato runner in one command.
New capsule-core modules:
- foundation/host_gpu.rs: HostGpuProfile detection (OS, Secure Boot,
GPU devices, driver, CUDA, Docker, nvidia-container-toolkit).
Read-only probes — never mutates host state.
- foundation/provision_receipt.rs: ProvisionReceipt and
ProvisionMarker types for post-install state recording and
--resume after reboot.
New ato-cli commands (visible, was hidden):
- ato runner doctor: read-only GPU host diagnostics with OK/WARN/FAIL
checks and recommended next steps. JSON output via --json.
- ato runner provision: installs nvidia-driver-575, Docker Engine,
and nvidia-container-toolkit on Ubuntu 22.04/24.04. Idempotent
(--force to reinstall), --resume after reboot, --dry-run for
preview (no state mutation), --enroll to chain into runner login.
Writes a provision-receipt.json with full hardware state.
Exits non-zero on GPU smoke test failure.
- Secure Boot: detected and warned about (MOK enrollment
instructions printed), never auto-enrolled.
Safety:
- Root required for all non-dry-run paths (including --resume).
- --dry-run never writes receipt or marker.
- apt-get update runs before every apt install.
- systemctl and nvidia-ctk exit statuses are checked; bail on failure.
- ca-certificates, curl, gnupg installed before toolkit repo setup.
- GPG key temp file uses /tmp, not CWD-relative path.
CLI surface change:
- ato runner is now visible (hide removed from root command).
login/serve remain hidden; doctor/provision are user-facing.
feat(runner): GPU host provisioning — doctor, provision
3 no-binding apps (tiny-http/darkhttpd, light-python/FastAPI, light-node/express) on n2-standard-4 / fc v1.16.0 / 512MiB mem, 5 iterations/mode. Median restore ms: app file-cold file-warm uffd-local uffd-hotset uffd-remote tiny-http 803 118 180 137 165 light-python 903 218 433 260 428 light-node 873 193 443 233 465 Conclusions: (1) UFFD beats File-COLD 3.3-6x (hotset) — its product value is cold cache / large images. (2) File-WARM (118-218ms) is fastest but needs 512MiB on disk per capsule; uffd-hotset is competitive (~40ms behind on 1GB apps, 0 bytes materialized). (3) Hotset is NEEDED: cuts demand faults ~3000->2-4, closes the gap to warm File. (4) Remote read-through fetched 0 chunks — CAS dedup already puts 52/55 mem chunks locally; remote buys little unless the image has unique content (keep off until P4). Feeds U16. Artifacts under benchmarks/ready-state/uffd-productization/2026-07-01/ (summary.md + raw/*.json).
snapshot: U9 File-vs-UFFD benchmark harness + real-app results (#876)
Opt-in ATO_READY_STATE_UFFD_DIAGNOSTICS=1: `ato run` computes which mem_backend a selector WOULD choose (via the pure U14 snapshot::mem_backend_selector) and records it (mem-backend-diagnostics.json + tracing + reporter notify), then restores via File EXACTLY as before. Pure observation, no behavior change — the P0 step that connects the placement contract (#816) without touching the restore path. Inputs gathered honestly: host_supports_uffd (backend.probe), capsule_no_bindings (requires_runtime_bindings guard), local_cas_has_memory (memory layer + first chunk present), validation_mode (ready_state_enabled). Not-yet-wired inputs are the safe value: hotset=false (until U12), remote=false (until P4). Off by default → default ato run unchanged. Flag test + diagnostics serialization test pass; cli builds; no new clippy warnings (fixed an un-awaited notify along the way — it now actually sends).
…stics cli: U10 Ready-State mem_backend selection diagnostics (#877)
Opt-in ATO_READY_STATE_UFFD_PREVIEW=1: `ato run` restores a no-binding capsule via the UFFD local-CAS demand path (instead of eager File rehydrate) on a supported host, and FAILS CLOSED on an unsupported host. Off by default → File path unchanged. - snapshot: RestoreReadyStateInput gains `uffd_preview: bool`; FirecrackerBackend restore uses Cas (local) mode when set, else the test-only env gate (uffd_mode). The product path is now INPUT-driven, not env — the env gate stays for KVM smokes. - cli: uffd_preview_enabled() flag; run.rs checks eligibility before restore (host supports uffd via probe → else bail; memory in local CAS → else bail; no-binding already guarded) and threads uffd_preview through restore_and_expose. - KVM smoke fc_kvm_uffd_preview_input_reaches_health: uffd_preview=true (env unset) reaches /health, demand-only, receipt mem_backend="uffd", clean teardown. Flag test (off by default); snapshot lib 95 pass; clippy clean. Local CAS only; remote/hotset not engaged (U12/P4).
snapshot+cli: U11 local UFFD preview command (#878)
HotsetProfileStore persists hotset profiles keyed by HotsetKey (capsule_manifest_hash + runner_class_id + memory_image_hash + backend_id + page_size + memory_size). load() returns a profile ONLY for an exact identity match — a different memory image / runner / backend / page-size / mem-size is a MISS, so a stale profile is never applied to the wrong image (the safety invariant; unit-tested). save() is atomic (tmp+rename). Engine wiring (ATO_FC_UFFD_HOTSET_STORE=<dir>): load-before-serve (keyed store first, then the explicit ATO_FC_UFFD_HOTSET file for smokes) + save-after-health (persist this run's pre-health hotset for the next restore). Off unless the store env is set → no behavior change to the default or U11 preview paths. Tests: hotset_profile_store_keys_by_identity_and_never_mismatches (unit); KVM smoke fc_kvm_uffd_hotset_persistence_round_trip (demand run saves → 2nd restore loads + prefetches → faults cut >2x). snapshot lib green; clippy clean.
…-persistence snapshot: U12 hotset profile persistence (keyed store) (#879)
Promotes the U5 fail-closed invariant to the path `ato run --experimental-uffd` actually takes, and hardens the corrupt-profile case: - KVM smoke fc_kvm_uffd_preview_corrupt_cas_fails_closed: uffd_preview=true (no env gate) + corrupt CAS → restore Err, no orphan firecracker/tap (product-path fail-closed, mirroring U5). - Unit hotset_profile_store_ignores_corrupt_file: a corrupt profile file → load() None, never a bad prefetch list reaches the page-server. Existing guarantees U13 relies on: FcProcess::Drop kills the FC child on any restore Err (no orphan VM); the page-server prefetch skips offsets that don't map (host_page_for_offset), so a wrong/corrupt profile can never UFFDIO_COPY bad bytes into guest memory. snapshot lib green; clippy clean.
…e-hardening snapshot: U13 failure hardening for the product preview path (#880)
ATO_READY_STATE_UFFD_AUTO_PREVIEW=1: the pure U14 selector (decide_mem_backend) CHOOSES File vs UFFD from the real facts (host UFFD support via probe, no-binding, local CAS has memory; remote off; hotset auto-loaded by the engine per U12), instead of the U11 forced-on preview. Graceful File fallback on an unsupported host (never bails — that's the explicit U11 preview's behavior). Takes precedence over U11's flag. Records the decision via tracing + reporter notify. Off by default → File path unchanged. Composes already-validated pieces: the selector is U14 unit-tested (7 branches); the engaged engine path is U11 KVM-validated (uffd_preview); the real-app win is U9 (uffd-hotset). Flag test (off by default); cli builds; no new clippy warnings.
…elect cli: U15 opt-in auto-selection preview (#882)
…883) The go/no-go for taking UFFD nightly->dev->main. Synthesizes U0-U6 + U7-U15 (all hardware-validated): what shipped, the U9 real-app benchmark evidence, the recommended default policy (default stays File; enable UFFD+hotset for cold/large/no-binding), the release gate (default-path invariant, frozen receipt schema, one-per-invocation KVM smokes NOT the flaky combined suite, product-path fail-closed, no-binding-only), known limitations, and the hard Phase 8 dependency. Verdict: product PREVIEW is GO (opt-in, no-binding, local CAS, fail-closed); DEFAULT enablement is NO-GO until the placement contract (P1) + Phase 8 land.
docs(ready-state): U16 UFFD product readiness report + release gate (#883)
PR 1 of the Phase 8a vertical slice — types only, no guest-agent, no `ato run`
behavior change. Cites the merged contract docs/ready-state/binding-lease.md.
New module protocol::binding_lease (serde-only, host + future guest-agent shareable):
- BindingName with validation (safe single path component: [a-z0-9_.-], not ./..).
- BindingLeaseId (opaque, host-generated).
- SecretValue — Debug/Display REDACTED; raw reachable only via expose(); serializes to
the wire for delivery but never leaks into a log.
- BindingLease { schema_version, id, name, value, issued_at_ms, expires_at_ms,
revoked_at_ms } with issue()/revoke()/status()/is_active()/receipt(); TTL model on
caller-supplied unix-millis (deterministic + testable). Debug redacts the value.
- BindingLeaseStatus { Active, Expired, Revoked } (revoked dominates expiry).
- BindingLeaseReceipt — the loggable/recordable form, carries NO secret value.
- BINDING_LEASE_SCHEMA_VERSION = 1.
Tests (5): name validation; SecretValue + whole-lease Debug never leak the secret;
active/expired/revoked transitions + idempotent revoke; receipt has no secret and
round-trips; wire lease round-trips WITH the value. No secret in the snapshot: nothing
here is written to CAS/manifest/rootfs/memory/vmstate. Default run behavior unchanged
(no consumer wires these in yet).
…review) Addresses the #895 request-changes. Debug/Display redaction alone did not stop an accidental serde_json::to_string(&lease) from emitting the raw secret. Fix at the type level: - SecretValue: no longer derives Serialize/Deserialize (internal host-side only). - BindingLease: no longer derives Serialize/Deserialize — cannot be JSON-dumped by accident. Wire form is the explicit BindingLeaseDelivery; loggable form is the value-free BindingLeaseReceipt. - New WireSecret (transparent Serialize, redacted Debug) + BindingLeaseDelivery: the ONLY type whose serialization carries the value, built via BindingLease::to_delivery(). Tests: only the explicit delivery payload serializes the secret (and its Debug still redacts); a static guard asserts only the value-free/wire types are Serialize (BindingLease/SecretValue are intentionally not). 6 tests; clippy clean.
protocol: Phase 8a BindingLease core model + receipt (#863)
The coordination artifact for "Store Capsule Ready-State Run E2E": the API contract every workstream (Snapshot Registry, Build Worker, Run Control, PWA, Desktop) implements against. No endpoints yet — contract only. Covers: the 3-server split (Build / Registry / Run Control, no per-app server), no-binding eligibility rules, Snapshot Registry + Run Control endpoints, job/run state machines, artifact metadata, minimal DB schema, error codes, the E2E lifecycle, and flags/gating (nightly + staging/admin only; File default; UFFD preview-only; binding-required capsules rejected — Phase 8 firewall intact).
PR 2 of Phase 8a — the host<->guest-agent control messages + the pure session state
machine, NO real secret delivery yet. No ato run behavior change.
- protocol::binding_control: HostToAgent { Deliver(BindingLeaseDelivery), Revoke{id},
QueryBoundReady, Stop } and AgentToHost { Ack, BoundReady{ready,pending}, Scrubbed,
Error }. Deliver is the ONLY value-bearing message (carries the explicit wire
payload); every response is value-free. BINDING_CONTROL_SCHEMA_VERSION = 1.
- new crate crates/guest-agent: BindingSession — a pure state machine that consumes the
control messages and computes the bound-ready gate. Skeleton: it tracks binding
PRESENCE (name -> id + expiry) for the gate but NEVER stores or writes the secret
value (the Deliver value is intentionally dropped until PR 3 tmpfs delivery). No
vsock transport, no /run/ato/bindings I/O yet.
Tests: Deliver is the only value-bearing message + responses are value-free (protocol);
bound-ready only when all required delivered, revoke/expiry/stop-scrub drop readiness,
no-binding session trivially ready (guest-agent). clippy clean.
…leton guest-agent: Phase 8a control-plane skeleton over vsock (#863)
Real secret delivery inside the guest. On Deliver the value is written to a 0600 file at /run/ato/bindings/<name>; on Revoke/Stop/expiry the file is scrubbed. The value is ONLY ever on the sink (tmpfs) — never retained in the agent's memory (the session drops its in-memory copy after handing it to the sink). No ato run behavior change. - BindingSink trait: deliver(name, value) / scrub(name). - TmpfsBindingSink: atomic 0600 write (tmp created 0600 then rename — never briefly world-readable), scrub = best-effort wipe + unlink, idempotent. Binding names are validated single path components (no traversal). Unix-only (guest is Linux); non-unix fails closed. - BindingSession now delivers through the sink; a sink error FAILS CLOSED (Error, not Ack; the binding is not marked present); QueryBoundReady scrubs expired bindings so tmpfs never outlives the TTL; Stop scrubs all. Tests (6): 0600 file created with the value + scrub removes + idempotent; atomic replace on renew; value reaches the sink; expiry/revoke/stop scrub the sink + drop bound-ready; sink failure fails closed. clippy clean.
guest-agent: PR 3 tmpfs binding-file delivery (#863)
… handoff, capacity states (#904) Address the 5 control-plane invariants before the contract is frozen for parallel implementation: 1. Idempotency (§2.6): Idempotency-Key / client_request_id on POST snapshot-jobs and POST runs; same key+scope returns the existing job/run (no duplicate builder/runner). DB idempotency_key columns. 2. Auth/ownership (§2.5): enqueue is admin/staging; runs are user-scoped; only the run owner or admin may GET/STOP (RUN_FORBIDDEN); public capsule != public run control. 3. Artifact integrity (§5.1): runners MUST verify artifact_manifest_hash / CAS chunk hashes / snapshot_backend / runner_class_id / execution_id before restore; artifact_location is not trusted; mismatch fails closed. 4. Desktop handoff (§3.1): provider=desktop returns handoff_url + desktop_session_id or unsupported_reason (not app_url); Desktop reuses the Registry metadata, not a re-implementation. 5. Run states (§4): add queued/dispatching before restoring, composing with managed Cloud Slots / runner capacity.
…ntract docs(api): snapshot run-control contract (PR 1, #896)
Record that snapshot-run EXTENDS the existing ato-api runs table / /v1/runs / runner-lease / Cloud Slots (managed-cloud), rather than forking a parallel run system. Only capsule_snapshot_jobs + capsule_snapshots are new (Ready-State sealed artifacts, distinct from source_snapshots). Additive columns on runs; additive read-model fields on the Store capsules listing.
Contributor
Author
|
Superseded: the reconciliation was implemented (Tracks A–D) per merged contract #904; this doc no longer reflects open work. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Records the decision (approved on #896): snapshot-run extends the existing ato-api
runstable //v1/runs/ runner-lease / Cloud Slots (managed-cloud), rather than forking a parallel run system. Onlycapsule_snapshot_jobs+capsule_snapshotsare new (Ready-State sealed artifacts — distinct fromsource_snapshots). Additive columns onruns; additive read-model on the Storecapsuleslisting.Follows the merged contract #904. Prevents an orphan-sweep-style duplicate run model before Track A/C implementation.