Skip to content
This repository was archived by the owner on Jul 15, 2026. It is now read-only.
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
56 commits
Select commit Hold shift + click to select a range
a949456
feat: run firstmate as a Kubernetes agent OS
robinbraemer Jul 12, 2026
ec8e5e3
docs: design persistent root agent toolchains
robinbraemer Jul 12, 2026
3a7d3f4
docs: plan persistent root agent toolchains
robinbraemer Jul 12, 2026
33f5660
feat: bundle the Firstmate toolchain
robinbraemer Jul 12, 2026
2482889
feat: persist agent-installed tools
robinbraemer Jul 12, 2026
a9786f6
feat: isolate root agents with user namespaces
robinbraemer Jul 12, 2026
4c0284f
fix: run root agents on local OrbStack
robinbraemer Jul 12, 2026
5cb9f21
docs: explain persistent root agents
robinbraemer Jul 12, 2026
b4a1274
feat: scaffold Agent OS operating tool
robinbraemer Jul 12, 2026
c9423d4
feat: add Akua mate packaging
robinbraemer Jul 12, 2026
c542f0c
feat: bundle k9s in agent image
robinbraemer Jul 12, 2026
282d05e
refactor: keep mate workflows tool-native
robinbraemer Jul 12, 2026
e1e3b10
feat: add Akua Firstmate bootstrap package
robinbraemer Jul 12, 2026
b40df4b
fix: probe model capacity and portable evidence
robinbraemer Jul 12, 2026
3ffbec9
Prove persistent Kubernetes mate recovery
robinbraemer Jul 13, 2026
4580220
Pin Terra in remote mate demo
robinbraemer Jul 13, 2026
6e63629
fix: select rebuilt local demo image
Jul 13, 2026
7765088
Record GitHub issue to PR fleet proof
robinbraemer Jul 13, 2026
0da294a
Record fleet task reconciliation
robinbraemer Jul 13, 2026
0b36ea7
docs: keep Agent OS work tracking private
robinbraemer Jul 13, 2026
25f6232
docs: separate Kubernetes core from Akua integration
robinbraemer Jul 13, 2026
bdbcbce
feat: make Agent OS package portable
robinbraemer Jul 13, 2026
25873de
docs: add Herdr source compliance bundle
robinbraemer Jul 13, 2026
c94408e
no-mistakes(review): Captain, harden Kubernetes credentials and RBAC …
robinbraemer Jul 13, 2026
82419a3
no-mistakes(review): Captain, harden Kubernetes lifecycle and authori…
robinbraemer Jul 13, 2026
69b1d2b
no-mistakes(review): Captain, harden Kubernetes lifecycle and image s…
robinbraemer Jul 13, 2026
a8e56ee
no-mistakes(review): Captain, harden Kubernetes lifecycle and image p…
robinbraemer Jul 13, 2026
f8bb5ce
no-mistakes(review): Captain, harden Kubernetes recovery and reproduc…
robinbraemer Jul 13, 2026
d9fa651
no-mistakes(review): Captain, harden Kubernetes lifecycle and depende…
robinbraemer Jul 13, 2026
d38fb6d
no-mistakes(review): Captain, harden Kubernetes lifecycle locking and…
robinbraemer Jul 13, 2026
79f594b
no-mistakes(review): Captain, harden Kubernetes lifecycle CAS and rol…
robinbraemer Jul 14, 2026
e70e921
no-mistakes(review): Captain, harden Kubernetes lifecycle deadlines a…
robinbraemer Jul 14, 2026
eed7d54
no-mistakes(review): Captain, harden Kubernetes rollback and lifecycl…
robinbraemer Jul 14, 2026
3e85721
no-mistakes(review): Captain, harden Kubernetes lifecycle and image p…
robinbraemer Jul 14, 2026
881cfc8
no-mistakes(review): Captain, harden image provenance and Kubernetes …
robinbraemer Jul 14, 2026
77c47e7
no-mistakes(review): Captain, harden source provenance and lifecycle …
robinbraemer Jul 14, 2026
b3fe07a
no-mistakes(review): Harden provenance and lifecycle compensation
robinbraemer Jul 14, 2026
972ce9e
no-mistakes(review): Harden provenance and lifecycle safety
robinbraemer Jul 14, 2026
cc5c997
no-mistakes(review): Secure candidate promotion and lifecycle compens…
robinbraemer Jul 14, 2026
4d5c833
no-mistakes(review): Secure immutable candidate promotion
robinbraemer Jul 14, 2026
89199de
no-mistakes(review): Captain, secure immutable release rollback and p…
robinbraemer Jul 14, 2026
e77a7d9
no-mistakes(review): Captain, secure immutable source convergence and…
robinbraemer Jul 14, 2026
a6cb41e
no-mistakes(review): Captain, harden candidate recovery and secondmat…
robinbraemer Jul 14, 2026
3f360a4
no-mistakes(review): Captain, harden candidate evidence and secondmat…
robinbraemer Jul 14, 2026
801ee5b
no-mistakes(review): Captain, harden candidate evidence and secondmat…
robinbraemer Jul 14, 2026
3ac84d9
no-mistakes(review): Captain, prevent ambiguous candidate rebuilds
robinbraemer Jul 14, 2026
74dfa75
no-mistakes(review): Captain, harden candidate and secondmate release…
robinbraemer Jul 14, 2026
65949c1
no-mistakes(review): Harden candidate ancestry and secondmate ownership
robinbraemer Jul 14, 2026
0bdc782
no-mistakes(review): Harden candidate recovery and secondmate ownersh…
robinbraemer Jul 14, 2026
df55062
no-mistakes(review): Harden candidate recovery and Treehouse lease lo…
robinbraemer Jul 14, 2026
11d0b56
no-mistakes(test): Captain: isolate canonical source build tests
robinbraemer Jul 14, 2026
aa0eb7a
no-mistakes(test): Captain: stabilize Zellij cwd smoke polling
robinbraemer Jul 14, 2026
26caf34
no-mistakes(document): Document immutable image update policy
robinbraemer Jul 14, 2026
a0961da
no-mistakes(lint): Resolve ShellCheck lint findings
robinbraemer Jul 14, 2026
860f02a
fix: restore portable deploy and image workflow
robinbraemer Jul 15, 2026
214b4b2
test: preserve fm-send baseline helper
robinbraemer Jul 15, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
113 changes: 113 additions & 0 deletions .agents/skills/akua-intelligence-bootstrap/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,113 @@
---
name: akua-intelligence-bootstrap
description: "Bootstrap or recover a dedicated Agent OS intelligence cluster through Akua's native API, MCP, CLI, Akua packages, and Kubernetes without adding an Agent OS provisioning service."
user-invocable: false
metadata:
internal: true
---

# Akua intelligence bootstrap

Load this skill before provisioning, recovering, or handing off a Firstmate in an Akua-managed intelligence cluster.

## Boundary

- The Akua workspace and Hetzner project must be dedicated to Agent OS infrastructure and contain no product, production, or customer resources.
- Provisioning machines, creating or revoking credentials, retrieving kubeconfig, and granting cluster-admin require the captain's explicit approval for the exact run.
- An available credential proves capability, not approval.
- Use Akua's native Platform MCP, REST API, CLI, and packages directly.
- Do not build or invoke an Agent OS provisioning wrapper, controller, Install, or GitOps workflow.
- Use a fresh idempotency key for each intended create and reuse that key only when retrying the same intended operation.
- Never put an Akua token, Hetzner token, kubeconfig, or authorization header in Git, a prompt, a status file, command arguments, or retained evidence.

## Native API surface

The current public API base is `https://api.akua.dev/v1`.
When using REST, keep a complete HTTP authorization header in a mode-`0400` file and pass it with curl's native `-H @file` support.
Write secret-bearing JSON through protected files or stdin, never shell arguments.

Use the Platform MCP `search` tool again before a mutation so current schemas outrank this routing table.

| Outcome | Native endpoint |
| --- | --- |
| Store the approved Hetzner token | `POST /secrets`, kind `cloud_provider/hcloud` |
| Validate the provider token | `POST /secrets/validate_token` |
| Create managed KaaS | `POST /clusters` |
| Wait for cluster creation | `POST /operations/{id}:wait?timeout=60` |
| Create Hetzner compute configuration | `POST /compute/configs` |
| Inspect available capacity and price | `GET /compute/instance_types?config={config}` |
| Create a worker | `POST /compute/machines` |
| Inspect worker state | `GET /compute/machines/{providerId}` |
| Retrieve the approved owner kubeconfig | `GET /clusters/{id}/kubeconfig` |
| Create the clustered-Firstmate token | `POST /api_tokens` |
| Inventory tokens | `GET /api_tokens` |
| Revoke the local bootstrap token | `DELETE /api_tokens/{id}` |

## Authorization overlay contract

The public Firstmate package never accepts an Akua credential input or renders an Akua Secret reference.
Akua authorization belongs to the separate namespace-local integration overlay under `deploy/akua/`.
The grant patch mounts only the selected Secret at `/var/run/secrets/agent-os/akua` and sets `AKUA_AUTH_HEADER_FILE=/var/run/secrets/agent-os/akua/authorization`.
The Secret must contain one `authorization` key with the complete header and must live in the same namespace as the Firstmate StatefulSet.
Never put the Secret value into the overlay, package inputs, or command arguments.

Set `context` and `namespace` from the separately approved target and grant the overlay through the serialized integration helper:

```sh
AGENT_OS_CONTEXT="$context" AGENT_OS_NAMESPACE="$namespace" bin/agent-os-akua-auth.sh grant "$secret_name"
```

The helper holds the stable control Lease and namespace fleet Lease in that order, validates exact StatefulSet UID and resourceVersion, verifies the named Secret reference without reading Secret bytes, applies one CAS strategic patch, and verifies both the retained StatefulSet overlay and its exact-owned Pod.
If grant verification fails, the helper removes the overlay by CAS, verifies the fail-closed rollout, records only bounded Secret metadata evidence, and requires an explicit revoke before a different Secret identity can be granted.

Revocation is owned by the same integration boundary.
Revoke the Akua API token and prove it fails first, then revoke the overlay through the same serialized helper:

```sh
AGENT_OS_CONTEXT="$context" AGENT_OS_NAMESPACE="$namespace" bin/agent-os-akua-auth.sh revoke "$secret_name"
```

Delete only the named namespace-local Secret after explicit cleanup approval.
The revoke patch removes the environment entry, volume mount, and volume without changing the public package or any other Firstmate setting.

## Bootstrap procedure

1. Record the approved workspace ID, intended region, machine constraints, expiration or cleanup condition, and evidence directory without recording secrets.
2. Read the live workspace, Secret, cluster, compute configuration, machine, and token inventories before creating anything.
3. Stop if the workspace or Hetzner project contains production resources or if an existing matching resource makes intent ambiguous.
4. Validate the provider token without storing it, then create or select the `cloud_provider/hcloud` Secret and require server validation state `valid` before creating compute.
5. Create the managed cluster with an idempotency key, preserve its operation ID, and wait for terminal `SUCCEEDED` state.
6. Create the compute configuration from the validated Secret, inspect available instance types, and select the smallest type satisfying the approved CPU, RAM, disk, architecture, price, and region constraints.
7. Create the worker, then verify both Akua machine state and Kubernetes Node readiness.
8. Retrieve kubeconfig only into a protected temporary file and verify the cluster identity before applying anything.
9. Build or select a published Agent OS image by immutable digest.
10. Render `tools/agent-os/packages/firstmate/package.k` with `akua render`, inspect the ordinary credential-free YAML, and apply it with `kubectl`.
11. Create a distinct clustered-Firstmate API token, create the namespace-local authorization Secret from a protected header file, and apply the grant overlay exactly as defined above.
12. Verify the StatefulSet is ready, Herdr responds, the persistent home is writable, the `in-cluster` context is cluster-admin only in this intelligence cluster, and the Pod can list its Akua workspace through `curl -H @"$AKUA_AUTH_HEADER_FILE"`.
13. Probe the approved primary model route with a bounded request and record only provider, model, result, timing, and cost.
If there is no independently approved fallback provider, record the single-provider availability risk instead of presenting the fleet as quota-resilient.
14. From the clustered Firstmate, inventory all workspace token IDs and prove the new token can perform the required Akua and Kubernetes reads.
15. Only after that handoff succeeds, revoke the known local bootstrap token and prove it now receives an authentication failure.
16. Destroy every protected temporary file and record only non-secret resource IDs, operation IDs, timestamps, states, image digest, test results, costs, and interventions.

## Recovery and cleanup

- A Pod restart reuses the Firstmate PVC and the authorization mount declared by the separate overlay.
- A worker replacement must preserve or reattach every PVC holding unique work before deleting the old worker.
- Do not claim cluster-loss recovery until an external encrypted backup has been restored in a clean cluster.
- Machine, cluster, authorization overlay, Secret, token, and workspace deletion are separate destructive actions.
- Execute only the approved cleanup scope and verify retained resources afterward.

## Completion evidence

The bootstrap is complete only when current external state proves all of these:

- Akua operation IDs reached `SUCCEEDED`;
- the worker is ready in Akua and Kubernetes;
- the Firstmate StatefulSet and Herdr server are ready;
- at least one approved model route completes a bounded request;
- the home and installed-tool paths survive a Pod replacement;
- the clustered token works from the Pod;
- the bootstrap token is revoked and fails authentication;
- no product or production resource exists in the intelligence workspace; and
- the evidence record contains no secret values.
2 changes: 2 additions & 0 deletions .agents/skills/harness-adapters/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,8 @@ The supported launch-profile flags below were verified locally on 2026-06-30 wit
| pi | `--model <model>` | `--thinking <low\|medium\|high\|xhigh>` | Verified on pi 0.80.2. `max` prints an invalid-thinking warning, so firstmate omits Pi effort when the requested effort is `max`. |
| opencode | `--model <provider/model>` | none for firstmate's interactive launch | Verified on opencode 1.17.6. `opencode run` has `--variant`, but firstmate launches the interactive `opencode --prompt` path, which has no verified effort flag. |

When a Pi provider route is known, pass a provider-qualified model id such as `openai-codex/gpt-5.6-terra`; unqualified names may resolve to an unauthenticated provider.

When a requested effort value is outside the harness-specific accepted set, `fm-spawn` records the requested `effort=` in meta but emits no effort flag for that harness.
This preserves launch success instead of passing a known-bad value.

Expand Down
51 changes: 51 additions & 0 deletions .agents/skills/kubernetes-fleet/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
---
name: kubernetes-fleet
description: "Operate Agent OS crewmates as Kubernetes Pods with persistent homes and explicit authority boundaries."
user-invocable: false
metadata:
internal: true
---

# Kubernetes fleet

Load this skill only when the current firstmate is running in Kubernetes or is explicitly managing Kubernetes-backed crewmates.

## Operating contract

- Keep every crewmate general-purpose; its brief, tools, and authority specialize it for the current task.
- A crewmate normally communicates only with its parent through its terminal, status files, reports, and delivered Git state.
- The image includes Akua's CLI and the canonical runtime template lives at `tools/agent-os/packages/firstmate/crewmate.yaml` inside the one public Firstmate package.
- The optional persistent controller package lives at `tools/agent-os/packages/firstmate/`; load `akua-intelligence-bootstrap` before using the separate Akua authorization overlay against Akua-managed infrastructure.
- Use the bundled K9s terminal UI when it makes live Kubernetes inspection faster than individual `kubectl` reads.
- Render the package with `akua render`, inspect or edit its ordinary YAML when useful, and apply it with `kubectl`.
- Treat AI credentials as explicit per-mate grants, never as ambient inheritance merely because Firstmate can read them.
- Create or select a namespace-local Kubernetes Secret only after its AI authority is explicitly authorized.
- The Secret must contain the selected provider's `auth.json` key and must be provisioned independently instead of cloning or sharing the primary credential.
- Pass only its name through `AGENT_OS_AI_SECRET` when invoking `bin/agent-os-crewmate.sh create <id>`.
- The helper never reads or discovers the Secret value, and the default Role intentionally has no Secret-read permission.
- Kubernetes projects only the selected `auth.json` key into a dedicated read-only runtime directory, and the entrypoint links that file into the writable PVC-backed Pi state without copying credential bytes.
- A missing Secret or key keeps the Pod unready, makes create fail, removes the non-running Pod, and retains the PVC for an authorized retry.
- Probe the selected model route before launching work; when quota is unavailable, use another explicitly granted provider or report the capacity blocker instead of repeatedly spawning agents.
- Give every launched Herdr agent a task-unique name, close only a confirmed dead restored pane before reuse, and never replace a live agent.
- Grade completion by the promised artifact or delivered Git state; Herdr `idle` alone is not a completion signal.
- Use `bin/agent-os-crewmate.sh create <id>` to create a separate Pod and persistent home.
- Every mutating crewmate operation acquires the stable control Lease, namespace fleet Lease, and per-crewmate Lease in that order and releases them in reverse order.
- Use `status` to inspect it, `stop` to remove only the Pod, and `restart` to replace only the Pod on its retained PVC.
- The ambiguous `delete` command is rejected.
- `purge <id> --yes` is the only operation that destroys a persistent home.
- Stopping the owned Pod invalidates earlier checkpoint evidence and records a non-secret `agent-os.dev/quiesced-operation` generation only after proving Pod absence.
- Before purge, independently checkpoint or deliver unique work from the stopped home, then annotate the owned PVC with `agent-os.dev/checkpoint-state=clean`, a non-secret RFC3339 `agent-os.dev/checkpoint-at` value, and `agent-os.dev/checkpoint-operation` equal to the current `agent-os.dev/quiesced-operation` value.
- Purge verifies exact installation and crewmate ownership, displays the target, requires its own confirmation, and records requested and completed phases in `AGENT_OS_PURGE_EVIDENCE_FILE` or `$FM_HOME/data/crewmate-purge-evidence.log`.
- Purge evidence contains only time, namespace, crewmate ID, resource names, phase, and checkpoint time.
- Never mount the primary home into a child Pod.
- The demo child receives no Kubernetes ServiceAccount token by default.
- A Pod with an authorized ServiceAccount gets a token-file-backed `in-cluster` kubeconfig automatically, so Firstmate does not need to copy a bearer token into a temporary kubeconfig.
- The OrbStack primary's cluster-admin binding is a local-demo trust decision, not a production-safe default.
- Pin an explicit host context for host-side `kubectl`; inside an authorized Pod use the generated `in-cluster` context.

For normal credential rotation, update the explicitly authorized namespace-local Secret, restart the owned Pod with the same PVC, prove a bounded request uses the replacement credential, and revoke the old credential only after that proof succeeds.
For urgent revocation, stop the owned Pod first, revoke the old credential, update or select the approved replacement Secret, and restart only when that replacement is ready.
Never print, copy, persist, or place credential values in command arguments or evidence during either flow.

Use ordinary `kubectl exec`, Herdr, files, and Git to supervise the child.
Do not add a custom inter-agent chat protocol or Task/Run service.
2 changes: 1 addition & 1 deletion .agents/skills/secondmate-provisioning/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ This is secondmate-only: crewmate/scout model resolution is untouched by this fi
This section is the single owner of the secondmate sync and inheritable-config propagation contract; `AGENTS.md` sections 3 and 4 point here.
Before launch, `fm-spawn.sh --secondmate` locally fast-forwards the home to the primary firstmate checkout's current default-branch commit when it is safe; dirty, diverged, or in-flight homes launch unchanged with a warning.
The locked session-start bootstrap sweep runs the same guarded fast-forward for every live secondmate home, discovered from `state/<id>.meta` records with `kind=secondmate` (`data/secondmates.md` only backfills `home=` for older records).
That no-fetch path is a purely local fast-forward of tracked files, never an origin fetch, and it never touches the gitignored operational dirs, so a secondmate's backlog, projects, and in-flight work are never disturbed; a linked worktree advances immediately, while a standalone clone that lacks the target receives firstmate updates through `/updatefirstmate`'s origin refresh.
That no-fetch path is a purely local fast-forward of tracked files, never an origin fetch, and it never touches the gitignored operational dirs, so a secondmate's backlog, projects, and in-flight work are never disturbed; a linked worktree advances immediately, while a mutable-source standalone clone that lacks the target receives firstmate updates through `/updatefirstmate`'s origin refresh.
The same launch and the same locked bootstrap sweep also propagate the primary's declared inheritable local config, currently `config/crew-dispatch.json`, `config/crew-harness`, and `config/backlog-backend`, into the secondmate home's `config/`.
Because `config/` is gitignored, that propagation is a separate, primary-authoritative copy independent of the tracked-files fast-forward: it re-converges every live home whether or not its tracked files advanced, and it touches only the declared items.
Inheritance copies the literal `config/crew-harness` file, so a secondmate's own crewmates use the primary's crewmate harness only when it names a concrete adapter such as `codex`; an unset or `default` value has nothing concrete to inherit, and the secondmate's own crewmates fall back to the secondmate's own or detected harness instead.
Expand Down
Loading