Skip to content

Design & implement native LLM provider support (§21) #128

@AlexChesser

Description

@AlexChesser

Summary

Design and implement built-in runners that call LLM APIs directly (Anthropic, OpenAI, etc.) without requiring an external CLI tool.

Parent issue: #105 — Full Inventory, §21 Planned

Why

Currently the only production runner is ClaudeCliRunner, which shells out to the Claude CLI. Native API support enables:

  • Lower latency — no CLI process overhead
  • Richer control — direct access to sampling parameters, logprobs, streaming
  • Multi-provider — prerequisite for model cascading and ensemble patterns (see multi-provider routing)
  • Confidence signals — logprobs and token probabilities only available via API

Design Decisions Needed

  • Which providers to support first — Anthropic API? OpenAI? Both?
  • Authentication — API keys in env vars? Pipeline-level config? Credential store?
  • Runner trait changes — does the trait need new methods for streaming, token counting, logprobs?
  • Provider string format (§15) — how to specify provider/model (e.g. anthropic/claude-sonnet-4-20250514)
  • Feature parity — what subset of CLI runner features must native runners support?
  • Whether native runners replace or complement the CLI runner

Spec Reference

  • Referenced in spec/core/s21*.md as a planned feature
  • Provider string format in spec/core/s15*.md

Acceptance Criteria

  • Spec section authored
  • At least one native LLM provider runner implemented
  • Provider selection via step-level or pipeline-level config
  • Token usage reported per step
  • Streaming support

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions