Shipping a desktop AI app to non-technical users? Ollama-based embeddings break the moment the daemon isn't running. Run the same model in-process instead.
If you build embeddings on top of Ollama, every embedding call is a request to a background service on the user's machine. That's fine on your dev box, where Ollama is always up. It is not fine on a stranger's laptop, where the daemon may never have started. This repo shows the alternative: load the embedding model into your own process and skip the daemon entirely.
We're building a desktop app with retrieval-augmented answers. Embeddings went through Ollama. On our dev machines everything worked. Then we ran the flow a real user would hit — fresh boot, Ollama never started — and this happened:
/askcame back with empty context — the model answered from nothing./searchreturned no results for queries that obviously had matches.- Ingestion accepted documents happily, but they never got indexed, because embedding them failed quietly.
Nothing crashed. No error dialog. The app just quietly did less than it promised.
Here's the honest version of what went wrong. Ollama itself fails loudly — if the daemon is down, the HTTP call gets connection refused, plain as day. The swallowed failures were our fault: our error handling caught the exception and moved on with an empty vector list. So yes, we had a bug in our error handling, and we fixed it.
But fixing the error handling only turns silent breakage into loud breakage. The deeper issue is architectural: we let a separate daemon's lifecycle become our app's problem. On a machine we don't control, "is Ollama running right now?" is a question we can't answer and shouldn't have to. The fix isn't better error handling around the daemon call. It's not making the call.
Before — a request to a daemon that may or may not exist:
import requests
resp = requests.post(
"http://localhost:11434/api/embed",
json={"model": "bge-m3", "input": texts},
)
vectors = resp.json()["embeddings"] # only if the daemon is upAfter — the same model, in this process:
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("BAAI/bge-m3")
vectors = model.encode(texts, normalize_embeddings=True)The real implementation adds a thread-safe lazy singleton (load once, on first use), an async wrapper so the encode doesn't block the event loop, and a deterministic status so /health never has to guess. See embeddings.py.
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
uvicorn main:appThen, in another terminal:
# Deterministic — reports embedder status without loading anything.
curl http://localhost:8000/health
# First call downloads the model, then returns a 1024-d vector.
curl -X POST http://localhost:8000/embed \
-H 'Content-Type: application/json' \
-d '{"text": "semantic search over local documents"}'
# Top-3 matches from the built-in demo corpus.
curl -X POST http://localhost:8000/search \
-H 'Content-Type: application/json' \
-d '{"query": "the network service was not running"}'Note: the first /embed (or /search) call downloads roughly 2.2 GB of model weights from Hugging Face. It's cached after that, so subsequent runs are fast, but the first one takes a while and holds real RAM. /health stays deterministic throughout — it reports NOT_LOADED, LOADING, READY, or FAILED based on what actually happened, and never triggers the download itself.
This pattern is a real improvement for the shipped-app case, but it is not free. Be honest with yourself about the costs before adopting it.
- You're trading a daemon dependency for a heavyweight Python dependency.
sentence-transformerspulls in PyTorch, which is gigabytes of install on its own. This only makes sense if your app already ships a Python backend or sidecar. If it doesn't, adding one just to embed in-process may cost more than it saves. - ~2.2 GB first-run download. bge-m3's weights come from Hugging Face on first use. For non-technical users, don't make them wait on that at random — bundle the model with your app or host it on your own CDN. Anonymous Hugging Face downloads also get throttled, so a room full of first-time users hitting HF at once is a bad time.
- RAM. Running the model at fp32 in your process uses more memory than Ollama's quantized GGUF did, and now that memory sits inside your app, competing with your LLM and everything else for the same budget. Size for it.
The tempting design is: try Ollama first, fall back to in-process if the daemon is down. For a chat completion, sure. For embeddings, it's a bug.
The two backends produce different vector spaces. Ollama serves a quantized GGUF; sentence-transformers runs fp32. An index written by one backend is quietly wrong when queried by the other — the vectors don't line up, so nearest-neighbor search returns plausible-looking nonsense. And a fallback means you can't even predict which backend embedded any given document; it depends on whether the daemon happened to be up that day. You'd be mixing vector spaces at random inside one index.
Pick one backend per index and stick with it. That's why this repo is a replacement, not a fallback.
If you're moving off Ollama-served embeddings, you have to re-embed everything once. There's no shortcut. See reindex_embeddings.py for a runnable example — it reads your stored texts and writes fresh fp32 vectors.
To be clear: the new vectors are not "the same modulo floating-point noise." Quantization changes the numbers themselves, not just the last few bits. Old and new vectors are genuinely different, which is exactly why you can't mix them and exactly why re-embedding is mandatory rather than optional.
If your users are developers running their own stack — bring-your-own-model, their own daemon already up and configured — then leaning on Ollama is the right call. The daemon is right there, they manage it, and you get model flexibility for free.
This pattern is for the other case: an app shipped to people who will never open a terminal, on machines you don't control. There, "just run the daemon" isn't an instruction you can give.
MIT — see LICENSE.