perf: embedding-bearing verbs cost ~2s per op inside the warm daemon (recall/search); non-embedding verbs are at the 20-50ms floor

[ohdearquant]

Measurements (idle machine, warm kkernel mcp --daemon, production-scale DB)

Op	Latency
stats()	0.04-0.05s
get(id=<full-uuid>)	0.02s
memory.recall(query=..., limit=3)	2.0-3.5s, flat (same or novel query)
search(kind="entity", query=..., limit=3)	2.9s
batch [recall, recall] (one request, distinct queries)	3.0s
first recall after daemon cold start	~37s
KHIVE_NO_DAEMON=1 recall (in-process)	~37s every time

Decomposition

• The warm daemon IS serving (KHIVE_NO_DAEMON=1 forces the in-process path and pays the ~37s cold ANN + model load every call; the daemon path does not).
• The ~2s cost is shared by every embedding-bearing verb (memory.recall and entity search alike) while by-ID/aggregate verbs sit at the 20-50ms floor — so it is not ANN search, not FTS, not SQLite, and not per-request dispatch overhead.
• Batch of 2 recalls in one request = 3.0s vs 2.1s single: per-op embedding dominates, with weak parallel scaling (consistent with contention on a shared embedder).
• Repeating an identical query does not get faster: no query-embedding cache.

Hypothesis

The per-query embedding (embed_one, model all-minilm-l6-v2 via lattice-embed) costs ~2s per call inside the daemon. KhiveRuntime::embed documents lazy-load-then-reuse of model weights, so either (a) the cached-weights path still costs ~2s of pure-Rust inference for a single short query, or (b) the daemon is not actually reusing the loaded embedder across requests. A MiniLM-class forward pass for one short sentence should be tens of milliseconds, not seconds.

Suggested next steps

1. Instrument the daemon serve path with per-stage timing (embed / vector search / FTS / fusion / hydration) behind a debug flag, and confirm where the 2s lands.
2. If (a): profile lattice-embed's forward pass for the single-query case (SIMD/quantized path, batch-of-1 overhead).
3. If (b): fix embedder reuse across daemon requests.
4. Consider a query-embedding LRU as a cheap orthogonal win for repeated queries.

Why it matters: every recall-shaped caller (hooks, agents, interactive sessions) pays this on every call. Dropping recall from ~2.1s to ~200ms improves every consumer of the substrate, not just any single integration.

[ohdearquant]

Follow-up measurement (2026-07-01 ~23:14 EDT), daemon pid 52681 (kkernel mcp --daemon) alive throughout:

• Back-to-back recalls from the same shell: 31.9s then 2.0s. The slow one is in the cold in-process class (17-37s), the fast one is the warm daemon-forward class (~2s).
• lambda:leo independently observed recall=17s with stats()=0.04s and concluded the daemon was down. It was not — his probe used pgrep khived (process is named kkernel) and ~/.khive/run/khived.sock (socket is at ~/.khive/khived.sock).

Two additional findings for this issue:

1. Intermittent silent in-process fallback with the daemon alive. kkernel exec falls through to the in-process runtime when the daemon forward is rejected (config/namespace fingerprint mismatch) or unreachable/timed out (exec.rs steps 2-3). Nothing is logged, so a fallback is indistinguishable from a slow daemon — both callers above misattributed it.
2. UX gap: the route taken should be observable. Instrumentation (already step 1 in this issue) should stamp each response or log line with served_by: daemon|in-process and, on fallback, the reason (mismatch vs unreachable vs timeout). A caller whose config fingerprint drifts from the daemon's silently pays 17-37s per embedding-bearing call forever.

Reproduction guidance: time two identical recalls back-to-back; a >10x spread between them is the fallback signature, not embedder variance.

[ohdearquant]

Timeline overlay across two independent callers refines the fallback hypothesis toward contention:

time (EDT)	caller	recall latency
~21:35	lambda:leo	2.17s (quiet window)
~23:10	lambda:leo	17.0s
~23:14	lambda:khive	31.9s, then 2.0s immediately after
~23:19	lambda:leo	32.0s

All slow calls fall in the 23:10-23:19 window when both callers were probing concurrently; daemon pid alive throughout. The config-divergence candidate is refuted: ~/.khive/config.toml mtime 21:11 predates the daemon start 21:16, so the running daemon matches on-disk config.

Refined hypothesis: the daemon serializes (or near-serializes) request handling; a caller that loses the race hits the client-side socket timeout and silently falls back to the cold in-process path (17-37s: model load + ANN build). Repro: fire two identical recalls from two shells simultaneously — expect one ~2s and one ~17-37s. The served_by: daemon|in-process + fallback-reason stamping proposed above would confirm directly.

[ohdearquant]

Discriminating experiment run by a second caller (lambda:leo): two simultaneous recalls from two shells → BOTH slow, 46.0s and 46.6s; single control immediately after = 2.31s warm.

This refutes the one-winner-one-loser variant. Refined model: the daemon serves only uncontended embedding traffic — any concurrent arrival sends every caller to the cold in-process path, and two coincident cold paths compound (46s exceeds the 17-37s solo-cold band, consistent with double model-load contention).

Consequences for this issue:

1. Instrumentation must log concurrent arrivals at the daemon, not just per-call route — the discriminator is arrival overlap, not caller identity or config drift.
2. Any synchronous consumer of embedding-bearing verbs cannot survive a contended window (46s); asynchronous consumers are robust to this regardless of the fix.
3. This also explains earlier fleet-level anecdotes: multiple agents' hooks firing on coincident turns are themselves a contention source, so the slow path amplifies its own trigger.