Skip to content
This repository was archived by the owner on Jul 15, 2026. It is now read-only.

feat(agent-os): run Firstmate as a portable Kubernetes agent#1

Merged
robinbraemer merged 56 commits into
mainfrom
feat/orbstack-demo
Jul 15, 2026
Merged

feat(agent-os): run Firstmate as a portable Kubernetes agent#1
robinbraemer merged 56 commits into
mainfrom
feat/orbstack-demo

Conversation

@robinbraemer

@robinbraemer robinbraemer commented Jul 12, 2026

Copy link
Copy Markdown
Member

What Changed

  • Package Firstmate as a portable multi-architecture Agent OS image with its toolchain, verified source bundle, and persistent home-scoped tooling.
  • Add guarded Kubernetes install, upgrade, rollback, uninstall, RBAC, and runtime crewmate lifecycle flows, including an OrbStack profile and optional Akua authorization bootstrap.
  • Add immutable candidate/release image automation, provenance and compliance records, shell/Bun coverage, and the initial Agent OS address CLI.

Risk Assessment

✅ Low: Captain, the three targeted fixes are well-bounded and preserve fail-closed candidate recovery, Treehouse ownership binding, and lock contention behavior.

Testing

Completed 1 recorded test check.

  • Outcome: ⚠️ 1 error across 3 runs (1h1m57s)

Pipeline

Updates from git push no-mistakes

⏭️ **intent** - skipped

✅ No issues found.

🔧 **Rebase** - 1 issue found → auto-fixed ✅
  • ⚠️ docs/kubernetes.md - merge conflict rebasing onto origin/feat/orbstack-demo

🔧 Fix applied.
✅ Re-checked - no issues remain.

🔧 **Review** - 12 issues found → auto-fixed (17) ✅
  • 🚨 bin/agent-os-crewmate.sh:281 - The post-delete wait is capped at the 5-second request ceiling instead of the 180-second deletion deadline. Pods taking longer to terminate make stop and restart fail incomplete; use the remaining operation budget while reserving time for reconciliation.
  • 🚨 bin/agent-os-kubernetes.sh:766 - JSON merge-patching the credential-free StatefulSet replaces its container and volume arrays, removing the official Akua authorization env, mount, and volume on every upgrade. Decide whether upgrades preserve external overlays or reapply them transactionally.
  • 🚨 bin/agent-os-kubernetes.sh:1092 - Upgrade does not detect an active rollback checkpoint. The merge patch preserves its annotations while changing the template, so a later rollback follows the stale checkpoint rather than the revision preceding the upgrade. Define whether upgrade refuses, completes, or supersedes an active rollback.
  • 🚨 bin/agent-os-kubernetes.sh:1023 - Namespace deletion inventories resources and then deletes without coordinating with per-crewmate lifecycle operations. A concurrent create can add a Pod/PVC after inventory, and namespace deletion will destroy that persistent home. Add a fleet-wide creation barrier or shared lifecycle lock.
  • 🚨 bin/agent-os-kubernetes.sh:433 - Cluster-scoped preflight only examines resources in the current render. After a cluster-admin uninstall or partial install leaves the owned ClusterRoleBinding, reinstalling with namespace or none silently recreates the same ServiceAccount and reactivates cluster-admin. Always inspect the deterministic binding before install or upgrade.
  • 🚨 .dockerignore:3 - These ignore rules remove all tracked harness integrations from the image, including Pi watcher/turn-end extensions and Codex, Claude, Grok, and OpenCode hooks. Primary sessions therefore lack required supervision controls, and Pi recovery references nonexistent paths. Exclude credential files selectively while retaining tracked integrations.
  • 🚨 .dockerignore:1 - Excluding .git leaves /opt/agent-os as a non-repository, with no runtime initialization. This breaks firstmate self-update, firstmate-on-itself work, and secondmate provisioning paths that use Treehouse or clone FM_ROOT. Decide whether the image supports those advertised capabilities or explicitly disables them.
  • ⚠️ .agents/skills/akua-intelligence-bootstrap/SKILL.md:59 - The authorization grant patches by name without verifying installation ownership, UID/resourceVersion, or the lifecycle Lease. A mistargeted or concurrent operation can mount the token into a foreign same-name workload or race an upgrade; add exact ownership/CAS checks and serialization.
  • ⚠️ bin/agent-os-init.sh:18 - Overlaying /usr/local without deleting removed image files means upgrades cannot retire vulnerable bundled binaries and can leave stale files inside upgraded packages. Separate image-owned and user-owned paths, or reconcile image-owned entries from a manifest.
  • ⚠️ bin/agent-os-kubernetes-lease.sh:214 - Lease acquisition can treat another invocation as its own when both share a caller-controlled operation ID and second-resolution timestamps. Concurrent retries can therefore enter the critical section together; add a per-invocation nonce to the Lease identity.
  • ⚠️ .github/workflows/agent-os-image.yml:43 - The image publication job runs for every main push and v* tag, while validation is PR-only and the separate CI workflow does not run on tags. A tag targeting an unvalidated commit can publish a packages-write image; gate publication on the behavior checks or verified main ancestry.
  • ⚠️ bin/agent-os-crewmate.sh:23 - Crewmate IDs have no derived-name length check. IDs longer than 34 characters can pass validation but produce a lifecycle Lease name exceeding Kubernetes' 63-character limit. Validate the strictest generated resource name before any cluster call.

🔧 Fix: Captain, harden Kubernetes lifecycle and image provenance
11 issues (6 errors, 5 warnings) still open:

  • 🚨 Dockerfile:172 - The Dockerfile now requires a generated source bundle and commit/tree build arguments, but bin/agent-os-local.sh build still invokes plain docker build. A clean OrbStack checkout therefore cannot build the advertised demo image; generate the exact bundle and pass all required arguments in the local build path.
  • 🚨 bin/agent-os-container-entrypoint.sh:28 - The canonical primary repository is explicitly checked out detached, while fm-update.sh refuses detached primary roots. This leaves /updatefirstmate and default-branch-dependent provisioning unavailable; seed and check out the canonical default branch while retaining exact commit/tree verification.
  • 🚨 .dockerignore:4 - Re-including entire harness directories causes every nested untracked file under .pi/extensions, .grok/hooks, and .opencode/plugins to enter the Docker context, not only the listed controls. Re-ignore each directory's contents before allowing the exact tracked files.
  • 🚨 bin/agent-os-akua-auth.sh:128 - The Secret UID/resourceVersion/key snapshot is captured before waiting for the lifecycle Lease. A changed or recreated Secret can therefore be mounted and rolled out before the stale evidence is rejected afterward; acquire the Lease before capturing and validating the Secret record.
  • 🚨 bin/agent-os-kubernetes.sh:213 - Cluster-RBAC cleanup uses a kube-system lock only while the namespace is absent, whereas install creates the namespace before taking its namespace-local lock. Concurrent cleanup and install can therefore mutate the same deterministic ClusterRoleBinding under different locks; use one stable installation-wide lock domain across namespace creation/deletion transitions.
  • 🚨 bin/agent-os-kubernetes.sh:1231 - The first upgrade from the legacy ServiceAccount deletes it after creating a new StatefulSet revision, but the retained pre-upgrade ControllerRevision still references that legacy account. A subsequent rollback cannot create its Pod; preserve or transactionally reconcile rollback dependencies without reactivating stale cluster authority.
  • ⚠️ bin/agent-os-akua-auth.sh:153 - Grant/revoke mutates the StatefulSet template without rejecting an active rollback checkpoint. Retrying that rollback can later undo a successfully reported authorization change; apply the checkpoint guard to this template mutation path.
  • ⚠️ bin/agent-os-container-entrypoint.sh:60 - After switching runtime execution to the persistent canonical repository, agent-os-crewmate.sh still defaults its template to the immutable /opt/agent-os copy. Self-updated source therefore continues creating mates from a stale template; resolve the default relative to the executing canonical source.
  • ⚠️ Dockerfile:171 - The generated full-history bundle is included by COPY ., copied again explicitly, and retained alongside the cloned Git object database. This makes image size grow by multiple copies of repository history; isolate bundle processing in a build stage and keep only the required bootstrap repository in the final image.
  • ⚠️ bin/agent-os-kubernetes.sh:1229 - Upgrade checks the preserved Secret UID/resourceVersion only after the complete rollout. If it changes concurrently, the command fails with the new workload already applied and cleanup unfinished; revalidate immediately before the StatefulSet CAS and provide deterministic reconciliation for a failed final postcondition.
  • ⚠️ bin/agent-os-container-entrypoint.sh:38 - Any clean local commit descended from the image commit is accepted as canonical runtime source without proving the current HEAD/tree is reachable from a trusted remote ref. Confirm whether local-ahead primary state is intentional; otherwise require trusted HEAD provenance rather than ancestry and an origin URL string alone.

🔧 Fix: Captain, harden image provenance and Kubernetes lifecycle safety
10 issues (8 errors, 2 warnings) still open:

  • 🚨 bin/agent-os-source-bundle.sh:16 - Any clean local HEAD is labeled as trusted main/official-origin source, and the Dockerfile then synthesizes origin/main from it. Require equality with a freshly verified allowlisted remote ref or release before bundling.
  • 🚨 bin/agent-os-container-entrypoint.sh:89 - Runtime provenance compares HEAD only with the locally mutable origin/main ref while accepting local-ahead commits. Updating both refs bypasses the check and persists untrusted code across restarts; verify against a freshly fetched allowlisted ref or immutable attestation.
  • ⚠️ bin/agent-os-source-bundle.sh:21 - git bundle create ... HEAD retains all reachable history, including deleted .no-mistakes evidence, so checking only the current tree does not produce the promised minimal sanitized repository. Seed a shallow exact-commit repository without ancestor objects.
  • 🚨 .dockerignore:26 - The cleanliness check ignores Git-ignored files, but this denylist omits local state such as .lavish/ and .fm-secondmate-home; COPY . can embed that data in the image. Generate the context from tracked files or use a global default-deny allowlist.
  • 🚨 bin/agent-os-akua-auth.sh:152 - Authorization mutations take only the control lock, while crewmate operations take only the namespace fleet lock. They can overlap, and the authorization rollout can restart the Firstmate executing an in-flight crewmate mutation; acquire both locks in the shared order.
  • 🚨 bin/agent-os-kubernetes.sh:412 - An initially absent overlay is represented by an empty record, which disables the expected-state comparison. An overlay added before the StatefulSet CAS is silently accepted; encode and enforce an explicit absent snapshot.
  • 🚨 bin/agent-os-akua-auth.sh:132 - If the Secret changes after grant's final pre-CAS read, the postcondition reports failure but leaves the changed Secret mounted, and rerunning blesses its new identity. Reconcile grant failures by CAS-removing the overlay and verifying the fail-closed rollout.
  • ⚠️ bin/agent-os-kubernetes.sh:1274 - Only randomized current ServiceAccounts are preflighted. Upgrade can retain an absent or foreign legacy agent-os-firstmate dependency, making the saved rollback revision unusable; verify exact ownership before mutation.
  • 🚨 bin/agent-os-kubernetes.sh:355 - Rollback verifies that the historical ServiceAccount exists, but upgrade has rebound the RoleBinding or ClusterRoleBinding to the randomized account. Rolling back to the legacy account can report success while the Pod lacks required permissions; define whether rollback preserves the current account or transactionally moves authority.
  • 🚨 bin/agent-os-kubernetes.sh:1539 - Uninstall deletes only the current and freshly rendered ServiceAccounts, not the legacy account retained for ControllerRevision rollback. This leaves an owned orphan and makes --delete-namespace fail its emptiness check; delete it after the StatefulSet history is gone.

🔧 Fix: Captain, harden source provenance and lifecycle safety
12 issues (8 errors, 4 warnings) still open:

  • 🚨 bin/agent-os-crewmate.sh:192 - Control-lock YAML records $INSTALLATION_ID, but acquisition expects $CONTROL_LOCK_INSTALLATION_ID; exact verification therefore fails and breaks every mutating crewmate command. Render the scope-specific lock identity.
  • 🚨 bin/agent-os-akua-auth.sh:270 - With set -e, an ambiguously accepted patch or failed rollout exits before reconcile_failed_grant, potentially leaving the Secret overlay mounted. Route every post-mutation failure through fail-closed CAS reconciliation.
  • 🚨 bin/agent-os-kubernetes.sh:1494 - A changed Secret postcondition only reports failure after upgrade; the overlay remains mounted and a retry snapshots the new identity. Deterministically remove or roll back the overlay and verify the fail-closed rollout.
  • 🚨 Dockerfile:29 - The source archive checksum is trusted only through the adjacent mutable attestation, with no proof that extracted /opt/agent-os matches the verified bootstrap tree. Materialize source from the verified commit or compare its computed Git tree identity before use.
  • 🚨 bin/agent-os-container-entrypoint.sh:96 - Trusted fetch and merge operations honor persistent Git configuration before hooks are checked. url.*.insteadOf can redirect the allowlisted origin, while core.hooksPath can execute external hooks; run provenance operations with isolated, validated Git configuration.
  • 🚨 bin/agent-os-kubernetes.sh:1703 - After authority transfer and revision activation, a StatefulSet identity mismatch exits without compensation, leaving authority assigned to the rollback target. Restore source authority before reporting the foreign-workload conflict.
  • 🚨 bin/agent-os-kubernetes.sh:1829 - Uninstall captures only the current and deterministic legacy ServiceAccounts. Randomized accounts referenced by older ControllerRevisions survive rollback and become undeletable orphans after history removal; inventory exact-owned revision dependencies before deletion and remove them afterward.
  • 🚨 bin/agent-os-kubernetes.sh:1502 - Changing RBAC from namespace to none removes namespaced RBAC but retains the kube-system control Role and RoleBinding, so the supposedly authority-free ServiceAccount can still update the lifecycle Lease. Revoke exact-owned control access during the transition.
  • ⚠️ bin/agent-os-kubernetes.sh:392 - Authority transfer returns without verification when ServiceAccount names match or RBAC mode is none. Rollback can therefore succeed with missing, drifted, or unexpectedly retained bindings; skip mutation but still enforce exact postconditions.
  • ⚠️ bin/agent-os-kubernetes.sh:517 - Rollback compensation verifies only the ServiceAccount after rollout, not that current/update revision content matches the checkpointed source digest. Verify both ControllerRevisions before claiming restoration and clearing recovery state.
  • ⚠️ .github/workflows/agent-os-image.yml:67 - Pull-request image validation packages a fresh main clone rather than the PR source, so runtime-script and integration regressions introduced by the PR are not exercised before merge. Add a non-publishing, PR-specific trusted-source validation path.
  • ⚠️ bin/agent-os-source-bundle.sh:7 - Source selection is hardcoded to refs/heads/main; an older tagged release cannot be reproduced once main advances. Add an explicitly allowlisted, immutable release-ref mode if release reproduction remains required.

🔧 Fix: Harden provenance and lifecycle compensation
10 issues (9 errors, 1 warning) still open:

  • 🚨 bin/agent-os-source-bundle.sh:23 - env tries to execute command as an external binary, but Debian/Ubuntu provide it only as a shell builtin. Source preparation and the identical entrypoint wrapper therefore fail before invoking Git; execute git directly after env.
  • 🚨 bin/agent-os-crewmate.sh:192 - The fallback still renders the workload installation ID because LOCK_INSTALLATION_ID is never assigned; control-lock acquisition therefore fails identity verification for every mutating crewmate command. Render CONTROL_LOCK_INSTALLATION_ID for this scope.
  • 🚨 bin/agent-os-akua-auth.sh:248 - Secret existence and its authorization key are required for both grant and revoke. If an authorized or rejected Secret is deleted or changed, revoke cannot remove the overlay or rejected marker; apply this validation only to grant.
  • 🚨 bin/agent-os-kubernetes.sh:1596 - apply_rendered runs directly under set -e; an ambiguous patch or failed rollout exits before Secret postcondition reconciliation. Route every post-CAS failure through bounded fail-closed overlay reconciliation.
  • 🚨 bin/agent-os-container-entrypoint.sh:36 - The repository-config blacklist still permits executable and transport-affecting settings such as filter.*, core.fsmonitor, and http.sslVerify=false, which subsequent Git commands honor. Use an allowlisted isolated configuration rather than filtering known-dangerous keys.
  • 🚨 .github/workflows/agent-os-image.yml:183 - The release record's image, SBOM, and provenance hashes are only syntax-checked; publication merely prints the resulting image digest. Compare every actual published artifact and attestation against the immutable record before accepting publication.
  • 🚨 .github/workflows/agent-os-image.yml:53 - Hash equality proves only that the fetched ruleset matches the record, not that it is active, targets tags, covers this tag, or has an acceptable bypass policy. Validate those semantics before authorizing the packages-write job.
  • 🚨 bin/agent-os-kubernetes.sh:1932 - Historical ServiceAccounts are inventoried only while the StatefulSet exists. If uninstall deletes it and then fails before account cleanup, a retry cannot recover the randomized account list and leaves exact-owned orphans; persist or independently inventory residual dependencies.
  • 🚨 bin/agent-os-kubernetes.sh:430 - Rollback verifies RoleBinding subjects and roleRefs but never verifies the referenced runtime or control Role rules. Drifted or widened Roles can therefore pass exact-authority verification; validate both rule sets before and after transfer.
  • ⚠️ bin/agent-os-kubernetes.sh:1964 - Uninstall schedules control-RBAC removal only when the current or desired mode is namespace. An installation that previously transitioned from namespace to cluster-admin retains its exact-owned kube-system control Role and RoleBinding indefinitely; remove control access unconditionally during uninstall.

🔧 Fix: Harden provenance and lifecycle safety
8 errors still open:

  • 🚨 bin/agent-os-source-bundle.sh:35 - The preflight requires an origin ending in .git, but the pinned actions/checkout configures the HTTPS origin without that suffix. Every workflow invocation therefore stops before source bundling; normalize the URL or set the checkout remote explicitly.
  • 🚨 bin/agent-os-akua-auth.sh:301 - Revoke leaves SECRET_RECORD empty but invokes the default Secret verification. If the Secret still exists, revoke removes the overlay and then incorrectly reports failure; disable Secret verification for this absent-overlay check.
  • 🚨 bin/agent-os-container-entrypoint.sh:25 - Trusted Git operations still honor ambient variables such as GIT_DIR, GIT_WORK_TREE, object alternates, and GIT_EXEC_PATH. These can redirect provenance checks or substitute helpers; run both trusted wrappers with a strict environment allowlist.
  • 🚨 bin/agent-os-source-bundle.sh:39 - Source preparation still reads unchecked repository-local Git configuration, allowing executable settings such as core.fsmonitor or process filters during trusted operations. Validate an allowlisted local configuration or operate from synthesized config-free metadata.
  • 🚨 bin/agent-os-kubernetes.sh:739 - Fail-closed reconciliation accepts any same-name StatefulSet with matching installation labels instead of requiring the pre-CAS UID. An ambiguous patch plus delete/recreate race can therefore mutate a replacement workload; retain and verify the captured UID before compensation.
  • 🚨 bin/agent-os-kubernetes.sh:63 - Cleanup releases lifecycle locks before deleting control RBAC, allowing another operation to acquire the lock and then lose its authority mid-operation. Delete and verify control RBAC under both locks before namespace deletion, then release the locks.
  • 🚨 .github/workflows/agent-os-image.yml:185 - The tag job rebuilds with maximum BuildKit provenance and expects its OCI and attestation digests to match a pre-existing immutable record, although invocation and timestamp metadata make those outputs non-reproducible. Promote the exact recorded artifact or define a reproducible attestation scheme.
  • 🚨 .github/workflows/agent-os-image.yml:80 - Python fnmatch allows * to cross /, unlike GitHub ruleset pathname matching. The workflow can therefore claim a ruleset covers a tag when GitHub does not enforce it; reproduce GitHub's exact semantics or query evaluated coverage.

🔧 Fix: Secure candidate promotion and lifecycle compensation
7 errors still open:

  • 🚨 bin/agent-os-source-bundle.sh:64 - When extensions.worktreeConfig is disabled, Git defines --worktree as equivalent to --local; this therefore repeats the normal repository configuration and always fails the non-empty check in standard clones such as Actions checkout. Inspect a real config.worktree only when the extension is enabled.
  • 🚨 bin/agent-os-container-entrypoint.sh:29 - GIT_BIN is resolved before env -i, while the image PATH places persistent $HOME/.local/bin first. A persisted fake git is consequently executed as the trusted binary; pin the system executable or resolve it using the fixed sanitized PATH. The source-bundle wrapper has the same issue.
  • 🚨 .github/workflows/agent-os-image.yml:326 - When an image exists without a candidate record, the workflow adopts it and trusts BuildKit provenance stored inside that same unauthenticated image. Another packages writer can seed arbitrary image bytes and matching caller-supplied build arguments; require an independent trusted-workflow attestation or treat this state as a coordinate conflict.
  • 🚨 .github/workflows/agent-os-image.yml:297 - Each attestation subject only needs to match any platform manifest, so both can cover amd64 while arm64 remains unattested; SBOM statements are never subject-checked. Require a one-to-one mapping between attestations and both platform digests, validating each SBOM and provenance statement.
  • 🚨 .github/workflows/agent-os-image.yml:376 - Existing candidate records accept bootstrap_archive_sha256 based only on syntax, and release preparation never compares regenerated or retained bootstrap bytes with it. Verify the exact bootstrap digest on retries and release promotion.
  • 🚨 .github/workflows/agent-os-image.yml:357 - Candidate-record creation checks for absence and then performs an unconditional tagged push. A conflicting manifest written between those operations is overwritten, while readback merely confirms the overwrite; use a conditional create or another genuinely immutable coordinate.
  • 🚨 bin/agent-os-container-entrypoint.sh:19 - Promoted release bytes remain in candidate mode and map their trusted ref to moving main. /updatefirstmate can therefore fast-forward persistent FM_ROOT beyond SOURCE_COMMIT, after which the next Pod restart rejects that home; define an immutable release trust ref or explicitly disable self-update for promoted candidates.

🔧 Fix: Secure immutable candidate promotion
5 issues (3 errors, 2 warnings) still open:

  • 🚨 bin/agent-os-container-entrypoint.sh:162 - Promoted images retain candidate mode and require persistent FM_ROOT HEAD to equal their commit. After upgrading to release B, rolling back to release A reuses the PVC with HEAD at B, so A exits before readiness; failed-upgrade compensation can break similarly. Define a provenance-safe source rollback strategy.
  • 🚨 .github/workflows/agent-os-image.yml:515 - Release promotion checks that the GHCR tag is absent and then runs an unconditional oras tag. A concurrent packages writer can create a conflicting tag between those operations, which this job overwrites instead of rejecting. Use a genuinely conditional create or registry-enforced immutable tags. ORAS tag reference
  • 🚨 bin/agent-os-source-bundle.sh:205 - Release preparation regenerates and byte-compares a tarball containing Git pack/index storage. Those bytes depend on packing, compression, and object reuse rather than solely on commit/tree identity, so a valid later release can fail. Retain the exact candidate bootstrap artifact or use canonical storage. Git pack-objects
  • ⚠️ .github/workflows/agent-os-image.yml:229 - Every protected-main rerun unconditionally publishes another untagged candidate before creating its content-addressed record. If record creation fails, no commit-indexed coordinate identifies the orphan on retry, defeating build-once/conflict detection and allowing multiple candidate records for one commit. Add a trusted conditional commit-to-record mapping.
  • ⚠️ bin/fm-update.sh:50 - Immutable update policy exists only as an environment variable exported by PID 1. kubectl exec, cron, or another sanitized launcher invokes fm-update.sh without it and defaults to fast-forward, advancing release source toward main and causing the next restart to fail. Derive the policy from immutable image provenance when present.

🔧 Fix: Captain, secure immutable release rollback and promotion
6 issues (5 errors, 1 warning) still open:

  • 🚨 bin/agent-os-runtime-source.sh:42 - Existing persistent slots run git status before the entrypoint allowlists .git/config. A tampered core.fsmonitor can therefore execute persisted code before provenance rejection; validate local configuration without loading it before any -C operation.
  • 🚨 .github/workflows/agent-os-image.yml:265 - A persisted reservation with zero referrers always exits. Any transient failure between reservation and record publication permanently blocks every retry for that commit; add an owner-bound recovery path that safely resumes the incomplete reservation.
  • 🚨 .github/workflows/agent-os-image.yml:244 - Reservation ownership relies on a fetch-before-push check of deterministic content-addressed bytes. Two packages writers can both observe absence, push identical reservations, retain existed=false, and build concurrently; use an atomic, owner-bound coordinator or conditional reservation.
  • 🚨 bin/agent-os-container-entrypoint.sh:84 - A→B selects an independent Git object store, while secondmate synchronization assumes the new primary commit already exists in each linked worktree or clone. Existing homes therefore remain on A, and rollback cannot be represented by the fast-forward-only path; add provenance-safe bidirectional source selection for secondmates.
  • 🚨 bin/fm-update.sh:51 - Immutable policy is detected only at $FM_ROOT/.git/agent-os-runtime-source. Linked secondmates use a .git pointer file and standalone clones receive no marker, so their own /updatefirstmate does not derive the release policy and standalone homes can advance toward moving main; persist and validate exact policy per home.
  • ⚠️ bin/agent-os-runtime-source.sh:52 - An invocation that sees the newly renamed target can remove the materializer's still-owned empty lock. The owner then fails its mandatory rmdir under set -e despite completing a valid source; make lock cleanup owner-specific or tolerate a verified committed handoff.

🔧 Fix: Captain, secure immutable source convergence and candidate reservations
4 errors still open:

  • 🚨 .github/workflows/agent-os-image.yml:358 - The candidate referrer is published before its trusted Actions artifact is uploaded. Cancellation or upload failure leaves one referrer but zero artifacts, so every retry permanently exits here. Stage durable evidence first or support owner-bound exact regeneration.
  • 🚨 bin/agent-os-runtime-secondmates.sh:122 - Runtime convergence trusts home paths from state/registry and checks only that a marker exists, omitting marker-ID equality and canonical path/overlap invariants. A stale or planted record can redirect checkout into another clean repository. Preserve IDs and apply the existing secondmate-home validation before mutation.
  • 🚨 bin/fm-update.sh:93 - Missing immutable provenance fails closed only beneath $FM_HOME/runtime-sources. Linked and standalone secondmates live elsewhere, so losing their Git-dir marker silently restores fast-forward updates toward main. Persist an explicit per-home policy and reject missing or mismatched immutable provenance.
  • 🚨 bin/agent-os-runtime-source.sh:50 - Configuration validation invokes sed through the ambient PATH, which places persistent user-owned .local/bin first. A shadow binary executes before source verification and can falsify parsing of executable Git configuration. Use a fixed sanitized PATH or absolute system utilities throughout provenance checks.

🔧 Fix: Captain, harden candidate recovery and secondmate provenance
5 errors still open:

  • 🚨 .github/workflows/agent-os-image.yml:452 - The candidate image is pushed before its exact digest is durably recorded at line 612. Cancellation during record preparation leaves no referrer or artifact, so retry selects build=true and rebuilds the reserved commit. Persist exact build evidence immediately or recover the pushed digest without rebuilding.
  • 🚨 bin/agent-os-runtime-secondmates.sh:164 - Home validation does not bind an absolute .git pointer to that home. A correctly marked clean home can point at another repository’s Git metadata, which subsequent fetch, checkout, and marker writes corrupt. Verify standalone Git-dir containment or the linked-worktree backlink and owned common directory.
  • 🚨 bin/fm-update.sh:80 - The incomplete-policy guard runs only when at least one marker exists. Removing both markers from an immutable linked or standalone secondmate restores fast-forward updates; retain deletion-evident immutable state and fail closed when its policies are missing.
  • 🚨 bin/agent-os-runtime-secondmates.sh:224 - The home and Git-dir policy copies are replaced sequentially, while startup rejects mismatches. Interruption between writes leaves different A/B markers that both restart and rollback compensation refuse before repair. Journal or safely reconcile verified partial transitions.
  • 🚨 .github/workflows/agent-os-image.yml:408 - Existing-record recovery trusts an artifact using only its name and head SHA, without proving the exact workflow, event, repository, or reservation-owner lineage. Bind the independent artifact to the authorized claim chain and publication workflow before accepting it.

🔧 Fix: Captain, harden candidate evidence and secondmate recovery
3 errors still open:

  • 🚨 .github/workflows/agent-os-image.yml:549 - The image digest is pushed at line 521 but persisted only by this later artifact step. Cancellation or upload failure between them leaves no recoverable evidence, so retry rebuilds the reserved commit. Publish evidence atomically or make the pushed digest owner-bound and discoverable.
  • 🚨 bin/agent-os-runtime-secondmates.sh:88 - Linked-worktree validation proves pointer/backlink shape but not ownership of the common Git directory. A correctly marked worktree belonging to another repository passes, after which checkout and remote mutation affect that foreign repository. Bind the common directory to persisted installation or Treehouse ownership.
  • 🚨 bin/agent-os-runtime-secondmates.sh:217 - Cleanliness is required before the pending transition journal is inspected. If checkout at line 286 is interrupted after modifying tracked files, restart exits here and never performs journal recovery, blocking startup and rollback compensation. Validate the journal first and recover a verified partial checkout.

🔧 Fix: Captain, harden candidate evidence and secondmate recovery
3 errors still open:

  • 🚨 .github/workflows/agent-os-image.yml:555 - Buildx finishes while the exact OCI archive remains runner-local until this separate artifact upload. Cancellation or upload failure in between leaves no recoverable evidence, causing the reserved commit to be rebuilt; make the build/export itself produce a durable owner-bound coordinate before completing.
  • 🚨 bin/agent-os-runtime-secondmates.sh:89 - The common-directory check proves membership in this installation's runtime-source store, but not that this specific worktree is Treehouse-leased to the registered secondmate. A pooled same-common worktree with a stale or planted marker still passes and can be mutated; verify its persisted lease holder or equivalent exact assignment.
  • 🚨 bin/agent-os-runtime-secondmates.sh:380 - Git ownership is validated in the first loop, but subsequent trusted_git -C "$home" mutations re-read the live .git pointer without a spanning lock. An atomic pointer replacement can redirect fetch, checkout, and remote mutation into another repository; hold the ownership lock and use or revalidate the captured exact Git directory for every mutation.

🔧 Fix: Captain, prevent ambiguous candidate rebuilds
2 issues (1 error, 1 warning) still open:

  • 🚨 .github/workflows/agent-os-image.yml:488 - load_build_attempt is evaluated as an if condition, disabling Bash errexit within the function. After finding the attempt ref, a failed commit read or metadata validation is treated like absence, allowing build=true and a second build. Return a distinct status only for an explicit 404 and fail closed on every other error.
  • ⚠️ tests/agent-os-container.test.sh:292 - The new coverage only checks textual ordering; it never exercises cancellation/upload failure or unreadable attempt metadata. Add a deterministic state-transition test proving existing or unverifiable attempt state cannot authorize another build.

🔧 Fix: Captain, harden candidate and secondmate release integrity
7 issues (4 errors, 3 warnings) still open:

  • 🚨 .github/workflows/agent-os-image.yml:414 - Attempt/evidence lookup covers only the latest claim owner. Recovery then advances ownership without copying that evidence, so cancellation after the handoff lets the next run see absent/absent/exact and authorize a second Buildx despite an ancestor build. Carry evidence across the handoff or inspect the full validated claim chain.
  • 🚨 bin/agent-os-runtime-bound.sh:33 - agent_os_bound_git ignores the captured common-directory handle. Linked-worktree Git commands re-read the mutable commondir file, so a swap can redirect fetch, object access, and remote set-url into a foreign repository. Pass the bound common directory explicitly to every Git invocation.
  • 🚨 bin/agent-os-runtime-secondmates.sh:417 - Git/common directories are bound only after their configuration and provenance are validated. A rename replacement in that gap becomes the trusted binding without repeating those checks, allowing subsequent Git commands to load unvalidated configuration. Bind first, then validate exclusively through the bound handles.
  • 🚨 bin/agent-os-runtime-secondmates.sh:370 - Every registered home now requires Treehouse state and an active lease, but the supported explicit-home provisioning path creates standalone clones without leases. Existing explicit-home secondmates will block immutable-image startup; decide whether to migrate/refuse them or support equivalent authoritative ownership evidence.
  • ⚠️ tests/agent-os-candidate-state.test.sh:34 - The executable test supplies abstract states directly and never exercises workflow classification, ownership handoff, or cancellation/upload boundaries. It therefore misses the ancestor-evidence rebuild path; drive retries through the real classifier and count Buildx authorization.
  • ⚠️ tests/agent-os-runtime-secondmates.test.sh:314 - The pointer-swap test invokes the bound-path helper directly with a fake non-locking flock, bypassing the production synchronization loop. It cannot prove the lock spans every mutation or catch mutable commondir redirection; inject swaps during the actual runtime command and verify foreign repositories remain unchanged.
  • ⚠️ tests/agent-os-runtime-secondmates.test.sh:269 - Only the missing-lease case verifies pre-mutation failure. Stale, other-holder, duplicate, planted, and unreadable cases accept any nonzero exit, including failure after mutation; assert unchanged HEAD, policies, and foreign homes for every rejected state.

🔧 Fix: Harden candidate ancestry and secondmate ownership
4 errors still open:

  • 🚨 bin/agent-os-runtime-secondmates.sh:33 - Linux bindings use /proc/self/fd/<fd>, which is a symlink, but this check rejects symlink common-directory handles. Every bound Git operation for registered secondmates therefore fails and aborts container startup; accept the previously verified proc-fd handle.
  • 🚨 bin/agent-os-runtime-secondmates.sh:243 - The authoritative Treehouse lease is not bound to the captured Git metadata identity: any unique directory under common/worktrees can pass. A planted same-common worktree can therefore receive HEAD/index mutations; require the exact leased worktree metadata directory under the held lock.
  • 🚨 .github/workflows/agent-os-image.yml:441 - A normal interrupted publication can leave both exact build and exact record artifacts for one attempted owner. This branch labels that state ambiguous, permanently blocking recovery; validate both and prefer the more complete exact record.
  • 🚨 bin/agent-os-runtime-secondmates.sh:663 - The TOCTOU barrier is disabled whenever real /proc bindings are available, including Ubuntu CI, so the new test never reaches its ready signal. Enable test barriers independently of fallback binding and exercise each mutation phase, not only fetch.

🔧 Fix: Harden candidate recovery and secondmate ownership bindings
3 errors still open:

  • 🚨 bin/agent-os-candidate-state.sh:52 - Recovery evidence can legitimately span owners: run A owns the attempted build/build artifact, while recovering run B uploads the record artifact before cancellation. The classifier rejects B’s record because B has no build attempt, permanently blocking publication. Correlate and digest-match evidence across the validated claim chain; the test currently places both artifacts under A and misses this path.
  • 🚨 bin/agent-os-runtime-secondmates.sh:525 - This equates Treehouse state .name with Git’s worktree-metadata basename. Pinned Treehouse v2 stores the numeric pool slot as name but creates <pool>/<slot>/<repoName>, so real leased secondmates fail startup. Bind the leased state path to its captured .git pointer/backlink instead. Pinned Treehouse source
  • 🚨 tests/agent-os-runtime-secondmates.test.sh:407 - The fetch-phase check synchronously starts a contender while the barrier-paused runtime holds the ownership lock. Ubuntu selects blocking /usr/bin/flock, so the contender never returns and the release file is never touched, timing out Behavior CI. Probe asynchronously with a bound or explicitly use a nonblocking test lock.

🔧 Fix: Harden candidate recovery and Treehouse lease locking
✅ Re-checked - no issues remain.

⚠️ **Test** - 1 error
  • 🚨 tests failed with exit code 1
  • command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; for t in tests/*.test.sh; do echo "== $t =="; bash "$t" || rc=1; done; exit "$rc"

🔧 Fix: Captain: isolate canonical source build tests
1 error still open:

  • 🚨 tests failed with exit code 1
  • command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; for t in tests/*.test.sh; do echo "== $t =="; bash "$t" || rc=1; done; exit "$rc"

🔧 Fix: Captain: stabilize Zellij cwd smoke polling
1 error still open:

  • 🚨 tests failed with exit code 1
  • command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; for t in tests/*.test.sh; do echo "== $t =="; bash "$t" || rc=1; done; exit "$rc"
✅ **Document** - passed

✅ No issues found.

🔧 **Lint** - 1 issue found → auto-fixed ✅
  • ⚠️ linter found issues (exit code 1)

🔧 Fix: Resolve ShellCheck lint findings
✅ Re-checked - no issues remain.

✅ **Push** - passed

✅ No issues found.

robinbraemer and others added 30 commits July 14, 2026 13:13
@robinbraemer robinbraemer changed the title feat: run Firstmate as a Kubernetes agent OS feat(agent-os): run Firstmate as a portable Kubernetes agent Jul 14, 2026
@robinbraemer
robinbraemer merged commit dc4e23c into main Jul 15, 2026
4 checks passed
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant