Support Ollama as a native LLM provider

[cl0ckt0wer]

Summary

Ollama is the most popular self-hosted LLM runtime (30K+ GitHub stars, Windows/Linux/macOS). Users with powerful local GPUs (e.g. RX 7900 XTX, RTX 4090) want to run everything locally — embeddings AND the LLM for compression/summarization. Currently there's no path to use Ollama as a compression/summarization provider without running a separate translation proxy (LiteLLM).

Current state

• Embeddings already work with Ollama via EMBEDDING_PROVIDER=openai + OPENAI_BASE_URL — Ollama speaks the OpenAI embeddings API natively
• LLM compression/summarization requires Anthropic protocol — Ollama does not speak this
• ANTHROPIC_BASE_URL exists and supports proxies, but adding LiteLLM as another container adds ~200MB and operational complexity
• No Ollama issues have been filed — this is a gap

Proposed solution (simplest path)

Add an Ollama provider directly to src/config.ts that uses raw fetch() (no SDK dependency) to call POST http://host:11434/api/chat with an Ollama-compatible payload. This is exactly what MiniMax already does — it uses raw fetch to avoid SDK stainless headers.

OLLAMA_BASE_URL=http://host.docker.internal:11434   # default: http://localhost:11434
OLLAMA_MODEL=qwen3:14b                                # default: llama3.2

Provider detection in detectProvider():

• Check OLLAMA_BASE_URL or presence of Ollama on localhost
• Return provider: "ollama" with raw-fetch-based chat completion

Why this is the right approach

1. Same pattern as MiniMax — MiniMax already uses provider: "minimax" with raw fetch for Anthropic-compatible APIs. Ollama would do the same.
2. Zero new dependencies — just fetch() calls to the Ollama /api/chat endpoint
3. Completes the self-hosted story — users can already self-host with EMBEDDING_PROVIDER=openai → Ollama. Adding the LLM provider closes the loop.
4. No proxy needed — eliminates the LiteLLM requirement entirely

Environment

• OS: Windows 11
• Hardware: Ryzen 9 5950X, RX 7900 XTX (24GB)
• agentmemory: v0.9.4 (Docker, iii-engine 0.11.6)
• Ollama: v0.23.1 with nomic-embed-text (embeddings working)

[rohitg00]

Thanks @cl0ckt0wer — the self-hosted GPU story is exactly where we want agentmemory to land for the privacy / cost-conscious slice of users.

Quick correction first: Ollama actually already speaks the OpenAI chat-completions protocol (/v1/chat/completions), not just the embeddings API. So the right path here isn't a separate provider: "ollama" with raw fetch — it's the OpenAI-compatible provider that's just been added in #240.

Once #240 lands, your config becomes:

OPENAI_API_KEY=ollama                                          # Ollama ignores the key but a non-empty value is required
OPENAI_BASE_URL=http://host.docker.internal:11434              # or http://localhost:11434 outside Docker
OPENAI_MODEL=qwen3:14b                                         # whatever you've ollama-pulled

This works because:

• Ollama's /v1/chat/completions is a thin OpenAI-compatible shim over /api/chat. Same request body, same response shape.
• agentmemory's new OpenAI provider uses raw fetch (no SDK stainless headers), which avoids the auth-header rejection path that bites SDKs.
• One provider implementation covers OpenAI / DeepSeek / SiliconFlow / vLLM / LM Studio / Ollama. Adding a second ollama: provider would duplicate ~80% of the same code.

There's one real reason to add a separate native Ollama provider: if Ollama's /api/chat exposes options that /v1/chat/completions strips out (custom system prompts, mirostat tuning, etc.). I'm open to that as a follow-up if a concrete option is missing. But for compress / summarize use cases (which is all agentmemory uses the LLM for), the OpenAI-compatible path covers it.

Plan:

• Land feat: add OpenAI-compatible LLM provider #240. Add an "Ollama" row to the README LLM-provider table pointing at the OpenAI-compat config above (and ditto for the OPENAI_BASE_URL value host.docker.internal callout from your Docs: setup gaps found during end-to-end OpenCode + Docker + Ollama install #233 gap analysis).
• If you find a concrete option that only /api/chat exposes, reopen this and we'll spec a thin native provider on top.

Holding open until #240 lands + the README row goes in. Will close as addressed at that point.

[flamerged]

I tested current Ollama Cloud/local behavior while evaluating this. The OpenAI-compatible provider path in #240 can cover Ollama, including direct https://ollama.com/v1, but thinking cloud models need a reasoning-control passthrough (reasoning_effort=none or equivalent) or they can return reasoning with empty message.content. Added concrete test details on #240: #240 (comment)

[rohitg00]

Tracking — this will be closed once #307 (OpenAI-compatible LLM provider) lands.

Ollama exposes an OpenAI-compatible chat-completions endpoint on /v1 when started with ollama serve --enable-openai. After #307 merges, the path becomes:

# ~/.agentmemory/.env
OPENAI_API_KEY=ollama                         # arbitrary; Ollama ignores it
OPENAI_BASE_URL=http://localhost:11434/v1
OPENAI_MODEL=llama3.1:8b                      # or any model you have pulled

agentmemory will then route compress / summarize calls through Ollama with no Ollama-specific code in the codebase. Same OPENAI_BASE_URL shape unlocks DeepSeek, SiliconFlow, vLLM, LM Studio, and Ollama Cloud's thinking models (with OPENAI_REASONING_EFFORT=none).

Will close this issue with a link to the #307 merge commit once it lands.

[rohitg00]

Closing — superseded by #307 (merged 2026-05-16). The OpenAI-compatible provider lands the universal adapter shape; Ollama works via:

ollama serve --enable-openai
OPENAI_API_KEY=ollama
OPENAI_BASE_URL=http://localhost:11434/v1
OPENAI_MODEL=llama3.1:8b

Same surface unlocks DeepSeek, SiliconFlow, vLLM, LM Studio, Azure (auto-detected from hostname). See README's LLM provider section for the full env-var matrix.