Implement sourceos-agent local runtime CLI and doctor commands

[mdheller]

Context

The Mac node-commander incident exposed a platform gap: SourceOS needs a reusable local-agent control surface for Nix-governed services, launchd/systemd persistence, Podman runtime checks, auth isolation, observability, and quarantine.

Canonical spec: SourceOS-Linux/sourceos-spec specs/local-agent-runtime.md.

Problem

We had a service land as a root-owned system-wide LaunchAgent, invoking a Nix store wrapper that called Podman, depending on noninteractive registry auth, writing logs to /tmp, and using unbounded launch behavior. The failure path created thousands of runs and looked like hostile persistence.

Deliverables

Implement a sourceos-agent CLI with:

• sourceos-agent preflight <name>
• sourceos-agent doctor <name>
• sourceos-agent status <name>
• sourceos-agent logs <name>
• sourceos-agent install <name>
• sourceos-agent stage <name>
• sourceos-agent start <name>
• sourceos-agent stop <name>
• sourceos-agent restart <name>
• sourceos-agent quarantine <name>
• sourceos-agent uninstall <name>

Also add sourceos doctor local-runtime.

Required checks

The CLI must report:

• launchd/systemd backend state
• disabled override or stale service labels
• plist/unit lint and permissions
• Nix generation/source when available
• Podman binary/version
• Podman machine existence/running/socket state
• local image presence
• container state including Stopping
• auth mode and host credential-helper risk
• log paths and last exit reason
• suspicious run/restart counts

Acceptance criteria

• A stopped/refusing Podman machine produces one clear preflight failure and does not install active persistence.
• A host Docker config with Google credential helpers does not poison local runtime when empty-authfile is declared.
• A local image can run with --authfile sourceos-empty-auth.json.
• KeepAlive=true, /tmp logs, direct registry runtime images, and raw podman run persistence are flagged.
• sourceos-agent quarantine node-commander captures service definition, logs, podman state, image metadata, and redacted auth config.

[mdheller]

Initial scaffold landed on main.

Committed artifacts:

• sourceosctl/commands/local_agent.py: local-agent declaration, preflight/doctor/status/logs surfaces, Podman/launchd/auth checks, guarded mutation scaffolding.
• bin/sourceos-agent: direct CLI entrypoint.
• bin/sourceosctl: routes sourceosctl local-agent ... and sourceosctl doctor local-runtime into the new module.
• tests/test_local_agent.py: unit coverage for parser, guardrails, empty authfile detection, local runtime image policy, and mutation gating.
• docs/local-agent-runtime.md: developer/operator documentation.

Current behavior:

• Read-only: list, preflight, doctor, status, logs.
• Mutating verbs are guarded and currently plan/refuse partial mutation unless the full implementation is added.
• node-commander is the first declared agent, using local image localhost/socioprophet/node-commander:dev-boot, source image provenance, explicit empty authfile, launchd checks, and legacy system-wide plist detection.

Remaining work:

1. Implement real guarded quarantine with evidence capture.
2. Implement staged install/start/stop/restart/uninstall.
3. Add JSON output mode for machine consumption.
4. Add Linux systemd/Quadlet backend detection.
5. Add CI policy scanner for unsafe persistence patterns.
6. Move the prototype declaration into a Nix-generated local-agent registry once the module lands.

[mdheller]

Second implementation slice landed: guarded quarantine.

New commits:

• b373985: implements sourceos-agent quarantine <agent> --execute --policy-ok.
• 7d96db4: updates tests for executed quarantine and auth redaction.
• 8a6806b: documents quarantine evidence and behavior.

Quarantine now:

• captures doctor/preflight checks;
• captures launchd state/disabled overrides/bootout/disable results on macOS;
• captures Podman connections, machines, info, ps, image inspect, and container inspect;
• captures redacted auth files;
• copies app and related logs;
• moves writable user plist into the quarantine directory as disabled;
• attempts to move legacy system plist and records warnings when permissions prevent it;
• writes manifest.json and remediation.md.

Remaining implementation work:

1. Add staged install/start/stop/restart/uninstall.
2. Add JSON output mode for all read-only commands.
3. Add systemd/Quadlet backend.
4. Add CI policy scanner for unsafe persistence patterns.
5. Move declaration source into a Nix-generated local-agent registry.

[mdheller]

Third implementation slice landed: guarded lifecycle commands.

New commits:

• 8f58272: adds generated launcher/plist/systemd-unit rendering plus guarded stage, install, start, stop, restart, and uninstall implementations.
• 49422f2: expands tests for executed staging, install behavior with mocked runtime checks, start preflight refusal, stop execution with mocked backends, uninstall planning, and launcher policy assertions.

Implemented behavior:

• stage writes generated launcher, launchd plist, systemd user unit, and agent declaration into a timestamped stage directory.
• install writes the empty authfile, generated launcher, log marker, user LaunchAgent on macOS or systemd user unit on Linux, and install manifest.
• install is runtime-preflight-aware and refuses blocking failures unless --force is passed.
• start refuses runtime-blocking failures and then starts via launchd or systemd user backend.
• stop stops launchd/systemd and best-effort stops the Podman container.
• restart runs guarded stop then guarded start.
• uninstall defaults to quarantine-before-uninstall, with --no-quarantine for direct movement of generated user-owned files.

Remaining work:

1. Add JSON output mode.
2. Improve backend separation and Linux systemd/Quadlet maturity.
3. Add CI scanner for unsafe persistence patterns.
4. Move hardcoded declarations into a Nix-generated local-agent registry.
5. Add integration tests against fixture plists/units and mocked subprocess transcripts.

[mdheller]

Fourth implementation slice landed: unsafe local-persistence scanner.

New commits:

• fb86690: adds scripts/scan_local_persistence.py.
• d5ddac3: wires scanner into make validate as report-only via scan-local-persistence.
• 77859a7: adds unit tests for scanner detection, Markdown inclusion/exclusion, and fail thresholds.

Scanner rules currently detect:

• system-wide /Library/LaunchAgents usage;
• KeepAlive=true without bounded restart review;
• primary product logs under /tmp;
• ambient gcloud/Docker credential-helper risk;
• direct Google Artifact Registry runtime image references;
• raw podman run persistence;
• raw /bin/sh -c / /bin/bash -c persistence;
• opaque /nix/store/... service targets.

Current validation posture is report-only: python3 scripts/scan_local_persistence.py . --fail-on none. This lets us surface findings before turning on fail-on-high enforcement after repo review/allowlisting.

Updated remaining work estimate: 4–5 focused turns.

Remaining work:

1. JSON output mode for local-agent read-only and action commands.
2. Better backend separation and Linux systemd/Quadlet maturity.
3. Move hardcoded declarations into a Nix-generated local-agent registry.
4. Add integration fixtures/transcript tests for launchd/systemd/Podman paths.
5. Switch scanner from report-only to fail-on-high after intentional examples are allowlisted or moved to docs.

[mdheller]

Fifth implementation slice landed: local-agent registry declaration and validation.

New commits:

• 1ab63c9: adds sourceosctl/local_agents/node-commander.json as the governed registry declaration for the first local agent.
• a91902d: adds scripts/validate_local_agents.py with policy validation for local-agent registry files.
• 2fe9e1d: wires registry validation into make validate.
• 766293e: adds tests for registry validation and policy rejection cases.

What this accomplishes:

• Extracts the node-commander declaration into a repo-owned registry artifact.
• Enforces local runtime image tags (localhost/...) in registry validation.
• Rejects ambient Docker config and interactive credential helper allowance by default.
• Rejects KeepAlive=true and /tmp product logs.
• Rejects user LaunchAgents under /Library/LaunchAgents.
• Makes the registry contract part of normal validation.

Remaining work estimate: 3–4 focused turns to reach a strong stopping point.

Remaining work:

1. Wire sourceos-agent to load declarations from the registry instead of the hardcoded Python declaration.
2. Add JSON output mode for read-only and action commands.
3. Improve backend separation / Linux systemd and Quadlet maturity.
4. Turn unsafe persistence scanner from report-only to fail-on-high after reviewing/allowlisting intentional examples.

[mdheller]

Sixth implementation slice landed: Linux backend templates and validation.

New commits:

• e5679bd: adds sourceosctl/local_agents/templates/systemd-user.service.tmpl.
• f2cd65f: adds sourceosctl/local_agents/templates/quadlet.container.tmpl.
• 813368c: adds scripts/validate_local_agent_templates.py.
• 679b99f: wires template validation into make validate.
• 5ea8550: adds tests for backend template validation.

What this accomplishes:

• Defines a governed systemd user service template for local agents.
• Defines a governed Quadlet container template for Podman-backed local agents.
• Ensures templates carry bounded restart controls and explicit authfile handling.
• Rejects dangerous drift such as Restart=always, Pull=always, registry auto-update, and /tmp paths.
• Makes backend template checks part of normal validation.

Remaining work estimate: 2–3 focused turns to reach a strong stopping point.

Remaining work:

1. Wire sourceos-agent runtime to load declarations from sourceosctl/local_agents/*.json instead of hardcoded Python.
2. Add JSON output mode, if we can safely rewrite the large CLI module or split output helpers into a smaller module.
3. Turn unsafe persistence scanner from report-only to fail-on-high after reviewing intentional findings / allowlisting examples.

[mdheller]

Seventh implementation slice landed: local-agent registry loader.

New commits:

• 617c06f: adds sourceosctl/local_agents/__init__.py.
• a36ad6d: adds sourceosctl/local_agents/registry.py for loading and normalizing registry declarations.
• 8bd3c7f: adds tests for repo registry loading, duplicate rejection, missing field rejection, and unknown agent behavior.

What this accomplishes:

• Establishes a typed loader for sourceosctl/local_agents/*.json declarations.
• Normalizes registry JSON into the same operational fields expected by the CLI.
• Gives us a safe path to remove the hardcoded Python declaration in the next pass.

Remaining work estimate: 2 focused turns to reach a strong stopping point.

Remaining work:

1. Wire sourceos-agent itself to use registry.load_records() instead of the hardcoded DEFAULT_AGENTS declaration.
2. Add JSON output mode, probably by splitting output helpers into a smaller module to avoid risky full-file rewrites.
3. Optionally switch unsafe persistence scanner to fail-on-high after checking current repo findings.

[mdheller]

Eighth implementation slice landed: registry-backed CLI routing and JSON read-only output.

New commits:

• 84dc090: adds sourceosctl/commands/local_agent_registry_cli.py, a registry-backed adapter over the existing local-agent CLI.
• 526ad91: routes bin/sourceos-agent through the registry-backed adapter.
• 25a4b65: routes sourceosctl local-agent ... and sourceosctl doctor local-runtime through the same registry-backed adapter.
• e2b6316: adds tests for registry patching, JSON list/status output, unknown-agent JSON errors, and non-JSON delegation.

What this accomplishes:

• The direct sourceos-agent entrypoint now loads sourceosctl/local_agents/*.json declarations before dispatch.
• sourceosctl local-agent ... uses the same registry-backed route.
• sourceosctl doctor local-runtime uses the registry-backed route.
• Read-only JSON output is now available for list, preflight, doctor, and status through --json.
• The large legacy local_agent module remains in place, but runtime declarations are no longer locked to its hardcoded defaults when using the entrypoints.

Remaining work estimate: 1 focused turn to reach a strong stopping point.

Remaining work:

1. Run/verify validation if possible and fix any surfaced test issues.
2. Optionally switch unsafe persistence scanner from report-only to fail-on-high after checking current findings.
3. Leave a final roll-up comment across related repos/issues with current state and next tranche.

[mdheller]

Tranche 1 roll-up: local-agent runtime hardening has reached a strong stopping point.

Committed implementation summary:

1. Canonical policy/spec exists in SourceOS-Linux/sourceos-spec/specs/local-agent-runtime.md.
2. sourceos-agent direct entrypoint exists.
3. sourceosctl local-agent ... and sourceosctl doctor local-runtime route through the local-agent surface.
4. node-commander has a governed registry declaration at sourceosctl/local_agents/node-commander.json.
5. Registry loader exists in sourceosctl/local_agents/registry.py.
6. Registry-backed CLI adapter exists in sourceosctl/commands/local_agent_registry_cli.py.
7. Read-only JSON output exists for list, preflight, doctor, and status via --json.
8. Guarded quarantine is implemented with evidence capture, redacted auth capture, launchd/Podman capture, logs, manifest, and remediation.
9. Guarded lifecycle commands exist for stage, install, start, stop, restart, and uninstall.
10. Unsafe persistence scanner exists at scripts/scan_local_persistence.py and is wired into make validate in report-only mode.
11. Registry validator exists at scripts/validate_local_agents.py and is wired into make validate.
12. Systemd and Quadlet backend templates exist under sourceosctl/local_agents/templates/.
13. Backend template validator exists at scripts/validate_local_agent_templates.py and is wired into make validate.
14. Unit coverage exists for CLI scaffold, quarantine, lifecycle, registry validation, registry loading, registry-backed CLI routing, and template scanning.

Current validation posture:

• make validate now runs unit tests, unsafe-persistence scan in report-only mode, registry validation, backend template validation, and scaffold validation.
• I could not execute make validate inside the remote GitHub connector from this environment, so the next operator/dev action is to run it locally or in CI and patch surfaced failures.

Known limitation / technical debt:

• The large legacy sourceosctl/commands/local_agent.py still contains the original default declaration. Runtime entrypoints now patch declarations from the registry before dispatch, but a later cleanup should remove duplication by moving the core implementation fully behind the registry model.
• JSON output currently lives in the registry-backed adapter for read-only commands, not every mutation path.
• Scanner is report-only (--fail-on none) until current intentional findings are reviewed or allowlisted.

Recommended next tranche:

1. Run make validate locally / in CI and fix any failures.
2. Remove hardcoded declaration duplication from local_agent.py.
3. Add JSON output for action/mutation results.
4. Promote scanner to --fail-on high once intentional examples are handled.
5. Move SourceOS boot activation and agent-machine implementations to consume the devtools contract.

Status: Tranche 1 is complete enough to hand off to CI/local validation and downstream integration.