feat(embed): add a replaceable local embedding provider

[Marsu6996]

Draft / stacked PR

Do not merge before #67 through #71. This is the final feature PR in the stack, after the baseline, graph-integrity, and bounded-topology fixes. The GitHub diff includes dependency commits until those PRs land; it will then be rebased onto main so the review diff contains only the configurable-provider and migration work.

Problem

IAE constructs the built-in English BGE model inside each process. Operators
cannot select a multilingual or domain-specific model without replacing the
native implementation, and changing model or dimension invalidates every
stored vector and ANN index.

Why this matters: French retrieval evidence

This work was motivated by a separate French-memory benchmark, not by an
assumption that IAE users already run Solon. Solon is a public, MIT-licensed
French embedding model family published by OrdalieTech; its model card requires
different query and passage handling, which is one reason the provider contract
distinguishes query from document inputs.

The benchmark used 31 synthetic facts, 30 independent memory threads and 100
French questions, with no customer or personal data. Each model ranked the same
fact corpus by dense cosine similarity in the same warm Apple Silicon
environment:

Model	Dimension	Recall@1	Recall@5	MRR	Query p50	Query p95
BAAI/bge-small-en-v1.5	384	0.43	0.67	0.5571	6.6 ms	19.9 ms
OrdalieTech/Solon-embeddings-base-0.1	768	0.70	0.91	0.7852	8.0 ms	50.4 ms
OrdalieTech/Solon-embeddings-large-0.1	1,024	0.74	0.92	0.8169	17.2 ms	48.0 ms

On this corpus, Solon Large improved first-result recall by 31 percentage
points and Recall@5 by 25 points over IAE's built-in English BGE model. A
shared loopback service measured 24.0 ms p50 / 32.2 ms p95 for individual
queries, about 7 ms of median local HTTP overhead, 198.9 items/s for a batch of
100, and roughly 493 MiB of warm RSS. The external model is documented at
https://huggingface.co/OrdalieTech/Solon-embeddings-large-0.1.

These numbers are corpus-specific and do not argue for changing IAE's
default model to Solon. They demonstrate the practical value of letting users
choose a multilingual or domain-specific embedder while preserving native BGE
as the zero-configuration default. Solon is the validation model for this PR,
not a new dependency or a required deployment.

Solution

• keep the native Rust backend as the zero-configuration default;
• add an optional loopback-only HTTP provider with explicit model ID,
  dimension, timeout, and strict response validation;
• distinguish retrieval queries from stored documents for asymmetric models;
• avoid importing or constructing native BGE in HTTP mode;
• add a forced, batched, resumable re-embedding migration that stages every
  record and swaps only after an exact zero-failure count;
• make both table renames and _hippo_meta.embed_dim one SQLite transaction;
• preserve every non-embedding SQL field, including encrypted ciphertext,
  byte-for-byte;
• reject a stale on-disk HNSW index at boot by comparing sampled vectors with
  SQLite, covering dimension changes and same-dimension model changes;
• retain the previous records table and require a daemon restart.

Security and compatibility

Only unauthenticated http URLs on 127.0.0.1, localhost, or ::1 are
accepted. Credentials, query strings, fragments, and alternate paths are
rejected. Responses are capped at 32 MiB and must match the configured model,
dimension, vector count, numeric type, and finiteness. Existing users remain on
the native 384-dimensional backend unless they opt in.

Deliberate exclusions

The PR defines a generic local protocol; it does not vendor Solon,
sentence-transformers, model weights, or a model server into IAE. Operators own
the loopback service and must stop the daemon and back up the store before
migration.

Validation

• provider, URL, response-cap, query/document, and real HTTP batch-path tests;
• migration, resume, encryption, atomic swap/rollback, and non-embedding field
  preservation tests;
• stale-HNSW, standby-map, ANN serialization, and native fail-loud tests;
• separate French benchmark: BGE 0.43/0.67 versus Solon Large 0.74/0.92 at
  Recall@1/Recall@5 on 100 synthetic questions;
• complete 384-to-1,024-dimensional rehearsal against the real loopback Solon
  service, followed by HNSW rebuild and successful vector recall;
• broad focused stack: 485 passed, 18 skipped;
• wrapper suite: 48/48, plus TypeScript typecheck;
• exhaustive default suite: 5,159 passed, 212 skipped, 4 xfailed, 0 failed;
• final adversarial Claude verdict: APPROVE, with no blocking code change;
  activation notes are captured in the local deployment runbook.

Designed by Marsu — Refined by Codex.

Public CI result

The final cumulative macOS correctness gate is fully green on c901f41: 5,152 passed, 113 skipped, 107 deselected, and 4 xfailed across 5,375 collected tests in 1:15:51.
The production universal2 wheel builds successfully, and the secret scan passes.

[CodeAbra]

Thanks for the whole stack, #67 through #71 are all merged and shipped in v2.2.2. For #72: I rebased it locally and it comes out clean (the stacked deps drop away, only your provider commit lands), but GitHub still has the old branch state so it shows a conflict here. Could you rebase pr/configurable-embedder-provider onto the current main and push? Then I'll merge it right away, landing with your authorship. Nice design on the provider, the loopback-only URL check, strict response validation, and resumable zero-failure migration are exactly the right shape.