API: Add taskTemplates (plural) to TaskSpawner for per-item multi-step pipeline spawning

[kelos-bot]

🤖 Kelos Strategist Agent @gjkim42

Area: New CRDs & API Extensions

Summary

TaskSpawner currently creates exactly one Task per discovered work item. Many real-world workflows need a coordinated sequence of specialized agent steps per work item — plan, then implement, then test, then open PR. Today this requires either cramming everything into one monolithic prompt (losing specialization) or setting up multiple independent TaskSpawners with fragile label-based gating conventions. This proposal adds taskTemplates (plural) to TaskSpawner so that each discovered work item spawns a pipeline of Tasks with automatic dependency wiring and result passing between steps.

Problem

1. TaskSpawner is limited to one Task per work item

The spawner binary (cmd/kelos-spawner/main.go:309-337) creates a single Task per discovered item:

taskName := fmt.Sprintf("%s-%s", ts.Name, item.ID)
task := &kelosv1alpha1.Task{
    Spec: kelosv1alpha1.TaskSpec{
        Type:        ts.Spec.TaskTemplate.Type,
        Prompt:      prompt,
        Credentials: ts.Spec.TaskTemplate.Credentials,
        // ... single template fields
    },
}

The taskTemplate field is singular. There is no way to define multiple steps that should be created together for each work item.

2. The pipeline pattern already exists — but only as manual Task creation

Example 07 (examples/07-task-pipeline/) demonstrates multi-step workflows using dependsOn and result passing via {{.Deps}} templates. This works well for one-off pipelines, but cannot be triggered automatically by a TaskSpawner. There is a fundamental gap between "I can define a pipeline of Tasks" and "I can have that pipeline triggered automatically per work item."

3. Kelos's own self-development proves the need — and the workaround is complex

Kelos uses 7+ separate TaskSpawners (self-development/kelos-workers.yaml, kelos-reviewer.yaml, kelos-pr-responder.yaml, etc.) with label-based gating (/kelos pick-up, excludeComments) to achieve multi-step workflows. This works but requires:

• Understanding undocumented label conventions
• Setting up separate TaskSpawners that must agree on naming/gating patterns
• No automatic result passing between stages (each stage re-discovers context from the GitHub issue)
• No aggregate visibility into the full workflow for a single work item

4. Single-prompt approach loses agent specialization

The alternative — putting everything in one prompt — means the agent must be expert at planning, implementing, testing, AND reviewing. In practice, specialized prompts with different models (e.g., Opus for planning, Sonnet for implementation) produce better results. The pipeline example already demonstrates this principle, but TaskSpawner can't leverage it.

Proposed API Extension

Option A: taskTemplates on TaskSpawner (recommended — incremental, no new CRD)

Add a taskTemplates field to TaskSpawnerSpec as an alternative to the existing taskTemplate:

apiVersion: kelos.dev/v1alpha1
kind: TaskSpawner
metadata:
  name: issue-pipeline
spec:
  when:
    githubIssues:
      labels: ["agent-pipeline"]
  maxConcurrency: 2
  taskTemplates:
    - name: plan
      type: claude-code
      model: opus
      credentials:
        type: oauth
        secretRef:
          name: claude-credentials
      workspaceRef:
        name: my-workspace
      promptTemplate: |
        You are a technical planner. Analyze this issue and create
        a detailed implementation plan.

        Issue #{{.Number}}: {{.Title}}
        {{.Body}}

        Output a structured plan. Do NOT write code.

    - name: implement
      type: claude-code
      model: sonnet
      dependsOn: [plan]
      credentials:
        type: oauth
        secretRef:
          name: claude-credentials
      workspaceRef:
        name: my-workspace
      branch: "kelos-{{.Number}}"
      promptTemplate: |
        Implement the following plan from the planning agent:

        {{index .Deps "plan" "Outputs"}}

        Issue #{{.Number}}: {{.Title}}
        Create a branch, write code, commit, and push.

    - name: test
      type: claude-code
      model: sonnet
      dependsOn: [implement]
      credentials:
        type: oauth
        secretRef:
          name: claude-credentials
      workspaceRef:
        name: my-workspace
      branch: "kelos-{{.Number}}"
      promptTemplate: |
        Write tests for the implementation on branch
        {{index .Deps "implement" "Results" "branch"}}.

        Run all tests and fix any failures. Commit and push.

    - name: open-pr
      type: claude-code
      dependsOn: [test]
      credentials:
        type: oauth
        secretRef:
          name: claude-credentials
      workspaceRef:
        name: my-workspace
      branch: "kelos-{{.Number}}"
      promptTemplate: |
        Open a pull request for branch
        {{index .Deps "test" "Results" "branch"}}.
        Reference issue #{{.Number}}.

Spawner behavior change

For each discovered work item, the spawner creates N tasks instead of 1:

Task name pattern: {spawner}-{itemID}-{stepName}
Example:          issue-pipeline-42-plan
                  issue-pipeline-42-implement
                  issue-pipeline-42-test
                  issue-pipeline-42-open-pr

The dependsOn field in each template step references sibling step names within the same item's task set. The spawner translates relative step names to fully-qualified task names:

dependsOn: [plan]  →  dependsOn: [issue-pipeline-42-plan]

Option B: New TaskPipeline CRD (alternative — more reusable, larger scope)

Define a reusable pipeline template as a separate CRD, referenced by TaskSpawner:

apiVersion: kelos.dev/v1alpha1
kind: TaskPipeline
metadata:
  name: standard-issue-workflow
spec:
  steps:
    - name: plan
      taskTemplate: { ... }
    - name: implement
      dependsOn: [plan]
      taskTemplate: { ... }
---
apiVersion: kelos.dev/v1alpha1
kind: TaskSpawner
metadata:
  name: issue-handler
spec:
  when:
    githubIssues:
      labels: ["agent"]
  taskPipelineRef:
    name: standard-issue-workflow

This approach is more reusable (multiple TaskSpawners can share a pipeline definition) but requires a new CRD and more implementation work.

Recommendation

Start with Option A (taskTemplates on TaskSpawner). It is incremental, requires no new CRD, and can be extended to Option B later if reuse demand emerges. Option A is also backward-compatible — existing TaskSpawners with taskTemplate (singular) continue to work unchanged.

Implementation sketch

1. API change (api/v1alpha1/taskspawner_types.go): Add TaskTemplates []NamedTaskTemplate field to TaskSpawnerSpec. The NamedTaskTemplate struct extends TaskTemplate with a Name field and makes DependsOn reference sibling step names.

2. Validation: At most one of taskTemplate or taskTemplates may be set. Step names must be unique. dependsOn references must form a DAG (existing cycle detection in the Task controller already handles this).

3. Spawner change (cmd/kelos-spawner/main.go): When taskTemplates is set, the createTasksForItem loop creates N tasks per item, translating relative dependsOn names to fully-qualified task names ({spawner}-{itemID}-{stepName}).

4. Concurrency accounting: maxConcurrency should count pipeline instances (not individual tasks within a pipeline), since one work item = one pipeline. Add a pipelineConcurrency or redefine the semantics clearly.

5. Status tracking: Add totalPipelinesCreated to TaskSpawnerStatus alongside totalTasksCreated for clear accounting.

Use cases enabled

Use case	Pipeline steps
Standard issue workflow	plan → implement → test → open PR
PR quality gate	lint-check → security-scan → review → approve
Dependency update	detect-changes → update-code → run-tests → open PR
Incident remediation	diagnose → implement-fix → validate → deploy
Documentation	analyze-code → generate-docs → review-docs → commit

Compatibility

• taskTemplate (singular) remains the default for simple one-step spawners
• taskTemplates (plural) is opt-in and additive
• No breaking changes to existing resources
• Natural fit for the v1alpha2 cleanup (API: Plan TaskSpawner v1alpha2 as a breaking cleanup and field removal #704) — taskTemplate could be deprecated in favor of always using taskTemplates (with single-step being a pipeline of length 1)

Design questions

1. Should maxConcurrency count individual tasks or pipeline instances?
2. Should taskTemplates support conditional execution (e.g., "run step C only if step B's result contains 'needs-review'")?
3. How should TTL work — per-task or per-pipeline (delete all tasks when the last step finishes + TTL)?
4. Should the spawner skip creating a new pipeline if any task from a previous pipeline for the same item is still running?

[kelos-bot]

🤖 Kelos Task Status

Task kelos-triage-710 has succeeded. ✅

[kelos-bot]

🤖 Kelos Agent @gjkim42

Kelos Triage Report

Kind: kind/api
Priority: priority/important-longterm — Valuable API extension but requires significant design decisions; not blocking current workflows
Actor: needs human — Requires architectural decisions on API shape (Option A vs B), CRD design, concurrency semantics, and backward compatibility strategy

Status: STILL VALID

Evidence:

• TaskSpawnerSpec has only a singular taskTemplate field (api/v1alpha1/taskspawner_types.go:370) — no taskTemplates (plural) exists
• The spawner creates exactly one task per work item (cmd/kelos-spawner/main.go:309-337), confirming the limitation described in the issue
• No merged PRs or commits reference this issue or taskTemplates
• The codebase has no implementation of multi-step pipeline spawning

Duplicates found: #427 (near-duplicate — proposes adding steps to TaskTemplate for spawner-driven multi-step pipelines, solving the same core problem with a slightly different API shape: linear steps chain vs DAG-style taskTemplates with dependsOn). #283 is related but takes a different approach (chaining separate TaskSpawners via a taskCompletion trigger rather than multi-task-per-item).

Recommendation: KEEP OPEN — Consider consolidating with #427 since both address the same gap. The maintainer should decide on the preferred API shape (this issue's taskTemplates with DAG dependencies vs #427's steps with linear chaining) before implementation begins.

[gjkim42]

/kelos plan
I prefer this issue to #427.

[kelos-bot]

🤖 Kelos Task Status

Task kelos-planner-710 has succeeded. ✅