OVO integration assessment: paths forward for ail ↔ ovoment/ovo-local-llm

[AlexChesser]

Summary

Assessment of how OVO — a local, macOS/Apple-Silicon desktop LLM runtime — could be consumed from ail pipelines. Three integration paths are possible; one works today with zero code.

This issue is a scoping document, not an implementation plan. It exists so we can decide which path to invest in before writing code.

What OVO is

OVO is a desktop application (Tauri + FastAPI sidecar, MLX inference). It is not a CLI, not a stdin/stdout subprocess, and not a library — it is a long-running service that exposes three HTTP APIs on localhost:

Port	Flavor	Key endpoints
11435	Ollama-compatible	POST /api/chat, POST /api/generate, POST /api/pull, GET /api/tags
11436	OpenAI-compatible	POST /v1/chat/completions (SSE via stream=true), POST /v1/completions, GET /v1/models
11437	OVO-native	GET /ovo/models, POST /ovo/models/download, GET /ovo/settings, GET /ovo/claude/context, GET /ovo/audit

Platform: macOS 13+ on Apple Silicon only (no Intel, no Linux, no Windows). No documented auth scheme (localhost-only). Tool/function-calling support: not documented; needs empirical verification.

What ail already provides

• Runner trait (ail-core/src/runner/mod.rs:280) with invoke() and invoke_streaming().
• Built-in http / ollama runner (ail-core/src/runner/http.rs) that already speaks OpenAI-compatible /chat/completions. Configured via AIL_HTTP_BASE_URL, AIL_HTTP_TOKEN, AIL_HTTP_MODEL, AIL_HTTP_THINK. Does not stream, has no tool support, keeps session history in-process with full replay each turn (O(N²) tokens).
• Runtime plugin protocol (spec/runner/r10-plugin-protocol.md, spec/runner/r11-plugin-discovery.md) — JSON-RPC 2.0 / NDJSON over stdin-stdout, manifests at ~/.ail/runners/*.yaml, handshake + invoke + streaming notifications (stream/delta, stream/thinking, stream/tool_use, stream/tool_result, stream/cost_update, stream/permission_request). Fully implemented in ail-core/src/runner/plugin/.
• Built-in name-collision guard reserves claude, http, ollama, stub. The name ovo is free.

Path A — Zero-code: point the existing http runner at OVO

What it looks like:

export AIL_HTTP_BASE_URL="http://localhost:11436/v1"
export AIL_HTTP_MODEL="<ovo-model-name>"
ail --once "hello" --pipeline demo.ail.yaml

# demo.ail.yaml
pipeline:
  - id: think
    prompt: "{{ step.invocation.prompt }}"
    runner: ollama   # alias for http

Pros: works today, no code, no release.
Cons: no SSE streaming (buffered full response), no tool-calling path, full-history replay on each resume, no access to OVO's native 11437 endpoints (model list, audit, settings).
Effort: docs + a demo/ovo.ail.yaml fixture only.

Path B — Native Rust runner (ovo built-in inside ail-core)

What it looks like: new module ail-core/src/runner/ovo.rs, register in ail-core/src/runner/factory.rs, reserve the name alongside claude/http/ollama/stub.

Pros: direct, in-process, can call the native 11437 endpoints, can add SSE streaming and tool bridging, no subprocess overhead.
Cons: couples ail-core to a specific vendor's product, duplicates most of HttpRunner, burns a built-in name, adds an HTTP client dependency path we'd maintain forever. Contradicts the "Runner trait is the seam" principle in ARCHITECTURE.md — built-ins should be generic (HTTP, Claude CLI, Codex CLI), not per-vendor.

Recommendation: don't do this. Not appropriate for a third-party desktop app.

Path C — External plugin (~/.ail/runners/ovo.yaml)

What it looks like:

# ~/.ail/runners/ovo.yaml
name: ovo
version: "0.1.0"
executable: /usr/local/bin/ovo-ail-plugin
protocol_version: "1"
env:
  OVO_BASE_URL: http://localhost:11436

# pipeline
pipeline:
  - id: think
    runner: ovo
    prompt: "{{ step.invocation.prompt }}"

The plugin is a standalone executable (Python, Go, Rust — any language) that:

1. Reads JSON-RPC 2.0 on stdin, writes on stdout (NDJSON, \n-terminated).
2. On initialize, declares { name: "ovo", version, protocol_version: "1", capabilities: { streaming: true, session_resume: true, tool_events: <tbd>, permission_requests: false } }.
3. On invoke, calls OVO's POST /v1/chat/completions with stream=true, translates SSE deltas to stream/delta notifications, closes with a final invoke response populated with response, session_id, input_tokens, output_tokens, model.
4. Holds conversation history in-plugin keyed by session_id so resume_session_id is cheap (no replay from ail).
5. Optionally calls GET /ovo/models on initialize to advertise available models, and reports /ovo/audit events through stream/cost_update or log lines.

Pros: zero changes to ail-core, isolates OVO's API churn to one executable, unlocks SSE streaming + cheap session resume + future tool-event bridging, ships and versions independently, written in whatever language is most convenient.
Cons: subprocess spawn per invocation, plugin author owns retry/timeout/error-mapping, need to distribute a binary (or ask users to pip install a Python entry point).

Effort: ~1–2 days for a minimum-viable plugin + manifest + demo/ovo.ail.yaml + a short README. Spec is already published at spec/runner/r10-plugin-protocol.md.

Gaps / missing features to flag

Not blockers, but worth tracking:

• ail's built-in HttpRunner does not consume SSE. Even when the backend streams, ail buffers. A plugin can translate SSE to stream/delta notifications; a future enhancement to HttpRunner could do the same generically (#TBD — worth a separate issue).
• ail's HttpRunner has no tool/function-calling path. If OVO's /v1/chat/completions honors OpenAI tools, ail cannot currently route tool_call / tool_result pairs through the generic runner. A plugin can emit stream/tool_use / stream/tool_result notifications, so Path C is the unlock here.
• Session resume in HttpRunner is full-replay. Plugin can hold server-side state.
• OVO's tool-calling capability is undocumented. Needs verification (run a probe against /v1/chat/completions with tools in the payload). Outcome determines whether a plugin should bother with the tool-event surface.
• OVO's auth model is undocumented. Assumed localhost-only. Any non-local deployment story (SSH tunnel, remote Mac) needs separate design.
• OVO is macOS-only. Linux / Windows ail users cannot run OVO locally; documentation should be explicit about this.
• Plugin protocol capabilities.permission_requests: true is not useful for OVO (no tools with side effects). Fine — plugin can report false.

Recommendation

Do both A and C, skip B.

1. Immediately: document Path A in README.md and add demo/ovo.ail.yaml so users can use OVO via the existing http/ollama runner today. Effort: hours.
2. When there is demand (or as a reference plugin): build Path C as a separate repository (e.g. ail-runner-ovo or a folder under demo/plugins/). This doubles as the first real-world validation of the plugin protocol beyond the in-tree tests, and is the right home for OVO-specific behavior. Effort: 1–2 days.
3. Explicitly reject Path B. A vendor-specific built-in is the wrong layer; the plugin seam exists precisely to avoid this.

A formal implementation plan for Path C can follow once this direction is agreed.

Questions before committing to a plan

• Is OVO's /v1/chat/completions faithful enough to OpenAI's spec that a plugin can assume tools, tool_choice, and stream all work? (Needs empirical probe.)
• Is there appetite to enhance HttpRunner with SSE streaming generically, independent of OVO? That would let Path A cover more ground and reduce the need for a plugin.
• Should the OVO plugin live in this repo under demo/plugins/ovo/ (as a reference) or in a separate repo?

[AlexChesser]

Follow-up: refined path forward — generic tool-calling in HttpRunner

After further discussion, the real leverage point is not an OVO-specific runner. The insight: if ail's existing http / ollama runner supported OpenAI-style tool calling, OVO would work today as a base_url config change — and so would every other OpenAI-compatible backend (vLLM, LM Studio, LiteLLM, Groq, Together, Fireworks, etc.). One enhancement, N backends.

What "native runner with tool calling" means here

Not a new runner — extend the existing HttpRunner (ail-core/src/runner/http.rs) to bridge OpenAI tool calls into ail's RunnerEvent stream. Concretely:

1. Request side. Accept tools / tool_choice (and the ail-side tool_policy) and include them in the POST /v1/chat/completions body when the backend supports them.
2. Response side. Parse choices[0].message.tool_calls[] out of the response and emit them as RunnerEvent::ToolUse { tool_name, tool_use_id, input }. Append corresponding ToolEvent entries to RunResult.tool_events so the turn log captures them.
3. Streaming. Consume SSE (stream: true) from /v1/chat/completions, accumulate delta.content into StreamDelta events and delta.tool_calls[] fragments into ToolUse events. Today HttpRunner posts with stream: false and buffers; this needs to change either way for OVO's SSE to be useful.
4. Tool results. When the pipeline feeds tool output back (e.g. next invoke in a do_while loop), translate to role: "tool" messages with matching tool_call_id so the conversation state on the server stays consistent.
5. Capability negotiation. Many OpenAI-compatible servers don't support tools. HttpRunnerConfig should gain an explicit supports_tools: bool (env var AIL_HTTP_TOOLS) so ail fails cleanly on backends that reject the tools field rather than silently dropping it.

Why this is the native path (and why not Path B as originally scoped)

The original Path B proposed a vendor-specific ovo built-in runner. That's still inappropriate. But the user's observation reframes the same work as a generic enhancement: OVO becomes one consumer of a capability the core crate should have anyway. This respects the "Runner trait is the seam" principle — the built-ins stay generic (claude, http/ollama, codex, stub), and vendor-specific flourishes (OVO's :11437/ovo/* model management, audit endpoints) stay out of core.

Claude CLI against OVO — explicitly out of scope

For the record: the ClaudeCliRunner cannot be pointed at OVO via ANTHROPIC_BASE_URL. The Claude CLI speaks the Anthropic Messages API (POST /v1/messages with typed content blocks, tool_use/tool_result shapes, Anthropic-flavored SSE event types). OVO exposes only Ollama-shaped and OpenAI-shaped APIs. For Claude-CLI-against-OVO to work, OVO would have to ship a fourth API surface (/v1/messages with full Anthropic fidelity). That's OVO's decision, not ail's, and it's a substantial lift. Not on the ail roadmap.

Revised recommendation

1. File a separate issue: "Add OpenAI tool-calling + SSE streaming to HttpRunner." This is the highest-leverage change. Benefits every OpenAI-compatible backend, including OVO, without a line of vendor-specific code.
2. Keep Path A (docs + demo/ovo.ail.yaml) as the user-facing story for basic prompt/response. Once add pipeline switching to the TUI #1 lands, Path A covers tool-calling pipelines too.
3. Path C (OVO plugin) becomes optional — only worth building if someone wants ail to drive OVO's native :11437 endpoints (model download, audit, extended-thinking hints). For pure chat + tools, redundant with an enhanced HttpRunner.
4. Path B stays rejected as originally scoped (vendor-specific built-in). The reframing above is not Path B — it's a generic capability upgrade to an existing built-in.

What this does not solve

• Session resume stays full-replay in HttpRunner (O(N²) tokens). A plugin can still do cheaper server-side session storage. Not a blocker for most pipelines; worth a separate tracking issue.
• Anthropic-flavored features (extended thinking as Anthropic renders it, MCP-driven permission prompts, server-side --resume semantics) remain Claude-CLI-only. Expected.
• Per-backend tool-calling quirks (OpenAI vs Ollama's functions vs Mistral vs Groq naming deltas) will need a compatibility note — the SPEC for HttpRunner should document which shape ail emits and require backends to accept it.

Next step: open the HttpRunner tool-calling + SSE issue and link it here as the implementation vehicle.

---

Generated by Claude Code

[ovoment]

Hey @AlexChesser — just read this. Incredibly thorough assessment, thank you for mapping the surface area so carefully.

Path 1 confirmation

HttpRunner → OVO OpenAI API works today. Quick sanity check from my side:

curl -s http://localhost:11436/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"mlx-community/gemma-3-4b-it-qat-4bit","messages":[{"role":"user","content":"Hello"}],"max_tokens":10,"stream":false}'

Returns standard OpenAI response with usage fields. Streaming (stream=true) emits SSE frames with data: {...} + final data: [DONE].

Tool/function calling gap

You're right that this is the blocker for Path 2. Currently OVO's local models generate tool-use as XML blocks internally (parsed by the frontend), but we don't expose OpenAI-style function_call / tools schema in the API response format.

I can prioritize adding OpenAI function-call format to the /v1/chat/completions response if that unblocks your plugin protocol work. Would be a clean addition to the existing streaming layer.

Testing

I run a 128GB M4 Max — happy to test any ail pipeline config or runner YAML you throw at me. If you need specific OVO endpoint behavior documented or tweaked, just ask here or open an issue on our side.

Let's ship this 🦉

[ovoment]

Update: OVO-side tracking + answers to your questions

Filed ovoment/ovo-local-llm#9 to track OpenAI function calling format support on OVO's side.

Answering your three questions:

1. Is /v1/chat/completions faithful enough — do tools, tool_choice, and stream all work?

stream=true ✅ works today. tools / tool_choice ❌ not yet — local models generate tool-use as XML internally (parsed by the frontend), but we don't expose OpenAI tool_calls format in the API response. Now tracked in #9.

2. Appetite to enhance HttpRunner with SSE streaming generically?

Your call, but +1 from OVO's side — generic SSE benefits every backend, not just us. "One enhancement, N backends" is the right frame.

3. Should the OVO plugin live in ail repo or separate?

I'd vote demo/plugins/ovo/ inside ail — keeps it discoverable and doubles as real-world validation of your plugin spec.

Summary of what OVO committed to:

• Confirmed Path A works today (OpenAI-compat at :11436/v1)
• Filed #9 for OpenAI function calling format
• Available to test any ail pipeline config on M4 Max 128GB
• Anthropic /v1/messages — agreed, out of scope for now

Let's keep iterating 🦉

[ovoment]

Update: OpenAI function calling shipped in v0.0.5

Just pushed v0.0.5 with OpenAI function calling support.

POST /v1/chat/completions now accepts:

• tools array (OpenAI format)
• tool_choice parameter
• role: "tool" messages with tool_call_id

Response includes tool_calls[] in choices[0].message when the model invokes tools, with finish_reason: "tool_calls".

This should unblock your HttpRunner tool-calling enhancement — once ail sends tools in the request, OVO will return structured tool_calls in the response. 🦉