docs(dark-factory): Autonomous Agent Coding Pattern design#32
Open
elamaran11 wants to merge 68 commits into
Open
docs(dark-factory): Autonomous Agent Coding Pattern design#32elamaran11 wants to merge 68 commits into
elamaran11 wants to merge 68 commits into
Conversation
… diagrams Design doc for the Dark Factory pattern (doc-only, no runtime code): - docs/dark-factory/README.md — full design: two flows (Agent Sandbox capability + Dark Factory pipeline), pluggable coder (Claude Code / Kiro), holdout gate, AWS frontier agents, live PR status, lifecycle, security model, industry alignment + anti-patterns (lethal trifecta, reward hacking, LLM-judge bias), phased delivery. - docs/dark-factory/diagrams/flow-a-sandbox-capability.md — capability architecture + warm-pool state machine (mermaid). - docs/dark-factory/diagrams/flow-b-dark-factory.md — end-to-end pipeline, detailed sequence, live-status mock (mermaid). Grounds every reuse in eks-platform-openclaw (Sandbox CRD, session-router, LiteLLM), appmod-blueprints (PlatformCluster), and agent-platform-amazon-eks (addon ApplicationSets). Validated against Copilot/Codex/Devin/StrongDM.
Unify all 5 mermaid diagrams on one palette + font (was clumsy: mixed light/dark backgrounds, too many colors, inconsistent fonts): - Squid Ink navy canvas (#1B2733) with layered navy bands/nodes, light text - single AWS Orange (#FF9900) accent reserved for the active/happy path - Amazon Ember font, uniform rounded nodes + stroke weights - fixed linkStyle edge indices (chained edges were mis-counted) Flow A (capability + warm-pool state machine) and Flow B (pipeline + sequence + live-status) now read as one consistent visual system.
The agent platform implements Bifrost as the LLM proxy (bifrost.bifrost.svc:8080, Bedrock via Pod Identity), not LiteLLM. Fixed the Flow A diagram node and all README references (coder contract env, security/egress model, profile notes, intro). Clarified that eks-platform-openclaw uses LiteLLM but this platform uses Bifrost.
…us-agent-coding-pattern
Flow A — Agent Sandbox capability. New GitOps addon chart that packages the Kata micro-VM sandbox substrate for the Open Agent Platform.
Tunable surface: operator image, kata runtime classes + default (kata-clh), warm-pool sizing (targetIdle=3, scale-to-zero TTL, reap TTL), pool-manager config, and the coder SandboxTemplate defaults (Bifrost LLM gateway URL).
Namespace enforces PSS 'restricted' since kata sandboxes run untrusted LLM-generated code. Verified: helm template renders cleanly.
Renders one RuntimeClass per values.kata.runtimeClasses entry with the nodeSelector + toleration that steers pods onto the kata Karpenter pool. Verified: helm template renders kata-clh + kata-qemu with correct handlers.
SandboxTemplate the warm pool clones: kata-clh runtime, no SA token, runAsNonRoot + restricted seccomp + drop ALL caps, readOnlyRootFilesystem, ephemeral workspace PVC, Bifrost LLM gateway URL. Image/secrets patched per-claim by the consumer. Verified: helm template renders correctly.
Locks coder-sandbox egress to DNS + Bifrost:8080 + HTTPS (git/gh/registries), blocks IMDS (169.254.169.254). Breaks the lethal trifecta so untrusted issue text can't reach arbitrary internal services or exfiltrate. Verified: renders + chart lints clean.
Narrowly-scoped Role: sandboxes (+scale subresource), sandboxtemplates/claims read, PVC/Service delete — capability namespace only, no cluster-wide access. Gated on warmPool.enabled. Verified: renders; disabling warmPool emits 0 resources.
Every-minute CronJob that reconciles the warm pool: REFILL idle sandboxes to targetIdle via SandboxClaims, plus scale-to-zero + reap hooks. Runs non-root, readOnlyRootFilesystem, drop ALL. Verified: helm renders + YAML parses. NOTE: scale-to-zero/reap steps are stubbed — jq logic refined next commit.
Replace stubbed steps with real jq logic: idle sandboxes past idleScaleToZeroSeconds → kubectl scale --replicas=0 (PVC kept); claimed sandboxes past reapAfterSeconds → delete (abandoned-run safety net), keyed on the agent-sandbox.io/last-seen annotation. Verified: helm renders, YAML parses, embedded script passes sh -n.
…tefulSet/Service) Controller runtime for agents.x-k8s.io (upstream image v0.1.0) that materializes a Kata-VM pod+service per Sandbox CR. CRDs (~4k lines) come from the upstream bundle at an earlier sync-wave, not vendored. Gated on operator.installCRDs. Verified: full chart renders 14 manifests, all parse, helm lint clean.
Add agent-sandbox to gitops/addons/bootstrap/default/addons.yaml: sync-wave 2, selector enable_agent_sandbox, Bifrost URL from cluster annotation. The ApplicationSet cluster-generator fans it onto any cluster carrying the label. Verified: addons.yaml parses.
Turn on agent_sandbox in the dev environment overlay — the warm-pool capability comes up on spoke-dev where the Dark Factory pipeline runs. Verified: YAML parses.
Install the sandbox capability on prod as a permanent platform feature, but the Dark Factory never runs here — prod warm pool stays dormant, unreviewed code is only built/tested on spoke-dev (production-safety). Verified: YAML parses.
…un Kata) Comment out agent_sandbox on dev. Spoke clusters run EKS Auto Mode + Bottlerocket, which cannot host Kata micro-VMs (no nested-virt/kvm control). Chart is kept; enabling needs a kata-capable nodepool strategy for Auto Mode first. Verified: YAML parses.
…run Kata) Same as dev: comment out agent_sandbox until EKS Auto Mode + Kata integration is designed. Chart retained. Verified: YAML parses.
…options Record why agent-sandbox is disabled on spokes: EKS Auto Mode + Bottlerocket cannot host Kata (no nested-virt/kvm control, no kata-deploy). List 3 paths to evaluate: self-managed kata nodepool, Bedrock AgentCore/Fargate microVM, or gVisor. Top item to resolve before enabling Flow A live.
Record the spike outcome: self-managed nested-virt MNG coexists with EKS Auto Mode and /dev/kvm works via CpuOptions.NestedVirtualization (aws-cli >=2.35). Decision: MNG (not a 2nd Karpenter); AgentCore/gVisor rejected with reasons. Two implementation lessons: AL2023 nodeadm MIME userData (don't clobber bootstrap) + delete MNG-first teardown ordering. Replaces the old blocker note.
Depend on the upstream kata-deploy chart (v3.32.0) to install the Kata runtime + containerd handlers on kata MNG nodes. Gated by kataDeploy.enabled so the chart stays inert on clusters without a kata-capable nodepool. Verified: YAML parses.
Mirror openclaw's kata-deploy config: target the static kata-enabled label (not katacontainers.io/kata-runtime, which deadlocks), tolerate kata + runtime-not-ready taints, install qemu+clh shims, disable kata-deploy's own RuntimeClasses (our chart owns them). Default kataDeploy.enabled=false so it's inert without a kata nodepool. Verified: YAML parses.
The correct AL2023 MIME userData for the nested-virt kata MNG: a boot script that modprobes kvm_intel + persists it (so /dev/kvm exists before Ready), then a nodeadm NodeConfig that joins the cluster with the kata-enabled label + kata=true taint. Encodes spike lesson #1 (don't clobber nodeadm bootstrap).
Declarative nested-virt (c8i/m8i) Managed Node Group for spoke-dev alongside Auto Mode: minSize 0 (scale-to-zero), kata-enabled label + kata=true taint, AL2023, scoped node IAM. Coexistence validated by the spike. Verified: YAML parses.
…late Terraform variant for teams on the appmod/openclaw IaC path: launch template with cpu_options.nested_virtualization=enabled (the flag eksctl can't set) + MIME nodeadm userData (modprobe kvm_intel + join), MNG scale-to-zero, kata=true taint, kata labels. Verified: braces balanced.
…pass) Expand one-line variable blocks to valid multi-line HCL; terraform fmt applied. Verified with real terraform: 'terraform validate → Success! The configuration is valid.'
…ate ArgoCD app) An unbuilt OCI subchart dependency blocks the whole chart from rendering (helm won't template with a missing dep) and is fragile under ArgoCD. Remove the dependency; kata-deploy will be a separate gated ArgoCD Application pulling the upstream OCI chart directly. Verified: chart renders standalone again + lint clean.
…-to-end Live test success: pod under kata-clh ran with guest kernel 6.18.35 vs host 6.12.90 = real VM isolation. The real crux (NOT containerd restart / nydus — those were red herrings): self-managed MNG nodes on Auto Mode need BOTH vpc-cni AND kube-proxy addons, else kata-deploy can't reach kubernetes.default.svc and crashloops. Plus the runtime-not-ready startup-taint gate (openclaw PR #10) and an IAM access-entry-recreate gotcha. Full working recipe documented.
Real kata VM ran (guest kernel 6.18.35 vs host 6.12.90). kata-deploy 1/1 zero restarts after kube-proxy addon. Flow A Kata isolation fully proven on Auto Mode.
… schema)
Live apply caught it: SandboxTemplate.spec supports ONLY podTemplate — the
volumeClaimTemplates field belongs on the Sandbox CR, not the template
('unknown field spec.volumeClaimTemplates'). Move workspace to an emptyDir
(sizeLimit) in the podTemplate. Verified: SandboxTemplate applies cleanly.
…roven) Uncomment agent_sandbox + agent_sandbox_kata on DEV now that Kata-on-Auto-Mode is validated end-to-end. ArgoCD manages the in-cluster pieces; the kata MNG + vpc-cni/kube-proxy node layer is provisioned out-of-band (documented recipe). Prod stays disabled. Verified: YAML parses.
…espace) Live GitOps deploy caught it: chartNamespace made the appset build chart=kata-containers/kata-deploy → wrong OCI url (ghcr.io/.../kata-deploy-charts/kata-containers/kata-deploy). The namespace is already in chartRepository; chart must be just 'kata-deploy'. Verified: YAML parses.
… has neither) Live GitOps deploy caught it: the pool-manager CronJob used aws-cli image which lacks kubectl/jq, so the script exited immediately (BackoffLimitExceeded). Switch to alpine/k8s:1.31.0 (bundles kubectl + jq + helm). Verified: YAML + render.
…ifests Add the upstream v0.5.1 manifest.yaml (operator Deployment + CRDs + RBAC + webhooks) and extensions.yaml (SandboxClaim/SandboxTemplate/SandboxWarmPool CRDs). v0.5.0+ is required for the native SandboxWarmPool + SandboxClaim reconcilers — live testing proved v0.1.0 (openclaw bundle) lacks them (operator only ran the Sandbox controller). Source: kubernetes-sigs/agent-sandbox.
…5.1 upstream) The v0.1.0 operator we hand-wrote only reconciled Sandbox CRs (no warm pool). Replaced by the vendored upstream v0.5.1 manifest which includes the full operator + webhooks + SandboxClaim/WarmPool reconcilers.
…WarmPool) v0.5.0+ ships a native SandboxWarmPool CR (spec: replicas + sandboxTemplateRef) reconciled by the operator. Our bash CronJob pool-manager was redundant AND buggy (aws-cli image lacked kubectl/jq; SandboxClaim adoption needs the v0.5 reconciler). Replace with a SandboxWarmPool CR (next commit).
…ager) Declares a SandboxWarmPool (replicas=targetIdle, sandboxTemplateRef) the v0.5.1 operator reconciles to keep the warm buffer. Gated by warmPool.enabled. Verified: renders.
…int) Ported from openclaw PR #10 — the missing piece our chart lacked. Waits for kata-deploy Ready on each kata node, then removes the runtime-not-ready startup taint so kata sandboxes can schedule. Without it, Sandbox pods stay Pending (live-test finding). Verified: renders + chart lints clean.
…he operator) Live test: PSS restricted on the namespace blocked the upstream v0.5.1 operator pod (not restricted-compliant) — Deployment stuck 0/1. The namespace hosts the operator + kata-readiness too, not just coder pods. Coder isolation comes from the Kata VM boundary + the SandboxTemplate's restricted securityContext, not namespace PSS. Set enforce=baseline (audit/warn stay restricted).
Operator v0.5.1 deprecates v1alpha1 for these CRDs. Bump to v1beta1 (schema compatible: podTemplate / replicas+sandboxTemplateRef). Verified live: warm pool reconciles 3/3 kata VMs, SandboxClaim(warmPoolRef) binds instantly + pool refills. Guest kernel 6.18.35 vs host 6.12.90 = real VM isolation.
Add agent-sandbox-operator app (vendored v0.5.1 upstream, directory source, sync-wave 0) so the operator + CRDs deploy via GitOps — previously applied manually. Correct ordering: operator (wave 0) → kata-deploy runtime (wave 1) → agent-sandbox chart CRs (wave 2). This is the order Flow B builds on. Verified: YAML parses.
…eeds it) The platform ApplicationSet renders path-based addons as Helm charts, so the operator app failed reading upstream/Chart.yaml (raw manifests, no chart). Add a thin Chart.yaml + move the v0.5.1 manifests under templates/ (they have no Helm templating → applied verbatim). Drop the ignored 'directory' config. Verified: helm renders 16 manifests (operator + 4 CRDs + RBAC + webhooks).
The agent-sandbox, agent-sandbox-operator, and kata-deploy addons were generated on every cluster carrying enable_agent_platform (hub, dev, prod) because their per-addon selector was dropped when the global useSelectors path isn't taken — producing unwanted agent-sandbox-hub and agent-sandbox-spoke-prod ArgoCD apps. Switch each entry from selector (enable_agent_sandbox*) to alwaysSelector (environment In [dev]), mirroring the langfuse hub-only precedent. alwaysSelector is honored regardless of useSelectors, so the generated ApplicationSets match ONLY the environment=dev cluster; hub and prod apps are pruned. Kata needs a nested-virt MNG that exists only on spoke-dev, so dev-only is also the correct capability scope.
The vendored v0.5.1 bundle defined the agent-sandbox-controller Deployment
twice — once in manifest.yaml (core API + webhook port 9443, no --extensions)
and once in extensions.yaml (--extensions + config-volume, no 9443). Upstream
ships them as two separate 'kubectl apply' bundles where the second overrides
the first, but vendoring both into one Helm chart made ArgoCD render the
Deployment twice (RepeatedResourceWarning) with last-apply-wins semantics:
- if manifest won → controller lacked --extensions → the SandboxTemplate
v1alpha1→v1beta1 conversion webhook failed with 'no kind "SandboxTemplate"'
(this is what wedged the hub/prod app deletes on a cascade).
- if extensions won → controller lacked port 9443 → webhook unreachable.
Fix: keep a single controller Deployment in manifest.yaml carrying the union
(--leader-elect --extensions, ports 8080/8081/9443, config-volume) and remove
the duplicate from extensions.yaml. Now one controller serves both the core
API and the extensions conversion webhook. Renders to exactly one Deployment;
helm template + YAML parse verified.
…SandboxClaim
Adapts the openclaw session-router to Flow B: keyed on GitHub issue-id, binds
a pre-warmed Kata VM from the Flow A SandboxWarmPool via a SandboxClaim
(warmPoolRef=coder-warmpool), waits for status.sandbox{name,podIPs}+Ready, and
releases on teardown. CRD contract verified against the live v1beta1 schema.
One canonical status comment edited in place (found by a hidden marker so re-runs update the same comment), ticking each pipeline stage. P2/P3 stages render as pending placeholders so the surface is stable across phases.
…sandbox Posts the run spec (SPEC.md + repo + branch df/issue-N + profile) to the coder agent's in-VM control endpoint and awaits result.json. No secrets cross here — they reach the sandbox via projected tmpfs, out of band.
Ties the pipeline together: POST /run claims a warm sandbox, drives the coder, opens a PR (no auto-merge), and maintains the live sticky status; POST /teardown releases the sandbox. Trusted component, runs outside the Kata VM.
…ace-scoped SA + Role + RoleBinding scoped to sandboxclaims (CRUD) + read sandboxes in agent-sandbox-system. No pod/exec, no secrets, no cluster scope — mirrors the session-router trust invariants. Validated server-side on spoke-dev.
Runs on general Auto Mode nodes (trusted, no micro-VM needed), non-root, readOnlyRootFilesystem, drop ALL caps, healthz probes. Validated server-side.
Gates on the label, acknowledges on the issue, and POSTs the run request (issue/repo/title/body/base + short-TTL github.token) to the orchestrator. Net-new — OAP had no GitHub Actions. Least-privilege permissions block.
Untrusted side: reads Bifrost key + short-TTL gh-token from tmpfs (0400, never in env), writes SPEC.md, checks out df/issue-N, runs Claude Code headless via Bifrost, builds+tests until green (bounded, RETRY.md on failure), returns result.json. Kiro is a one-branch swap in runProfile().
…act, trust boundary
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Design document (doc-only, no runtime code) for a Dark Factory — Autonomous Agent Coding Pattern on the Open Agent Platform.
A human files a GitHub issue (a spec); AI agents do the rest — implement → build → test → security/devops review → PR → (human approves results) → merge → teardown. Autonomy Level 3.
Contents
docs/dark-factory/README.md— full designdocs/dark-factory/diagrams/flow-a-sandbox-capability.md— Flow A: the permanent Agent Sandbox capability (Kata/CLH micro-VMs +SandboxCRD + warm pool), capability architecture + warm-pool state machinedocs/dark-factory/diagrams/flow-b-dark-factory.md— Flow B: the Dark Factory pipeline, end-to-end flow + detailed sequence + live-status mockDesign highlights
eks-platform-openclaw(Sandbox CRD, session-router lifecycle, LiteLLM→Bedrock),appmod-blueprints(PlatformClusterephemeral EKS),agent-platform-amazon-eks(addon ApplicationSets).Validated against industry practice
Aligned with GitHub Copilot coding agent, OpenAI Codex, Devin, and StrongDM's Software Factory. Explicitly designs against documented anti-patterns: lethal trifecta / prompt injection, reward hacking / test-gaming, LLM-judge self-preference bias, multi-agent over-orchestration, non-converging comment loops, warm-pool idle burn, and rubber-stamp reviews. Sources cited in the README.
Note
Doc-first by design — this PR ships the pattern/design. Implementation follows the phased roadmap (P1–P5) in the README. Mermaid diagrams render natively on GitHub.