feat: implement per-scope Markov chain preflight prediction

config.example.toml

@@ -0,0 +1,91 @@
+# MoFA Engine example configuration.
+#
+# Copy to `config.toml` (current directory) or
+# `~/.config/mofa-engine/config.toml`. Any section may be omitted; defaults
+# shown below are applied. API keys support `${ENV_VAR}` substitution.
+
+[listen]
+host = "127.0.0.1"   # loopback by default; remote access is opt-in
+port = 8420
+
+[memory]
+# Omit budget_mb to auto-detect (70% of system RAM).
+budget_mb = 12288
+idle_timeout_secs = 120   # unload a resident model after this many idle seconds
+
+[timeouts]
+request_secs = 240     # overall budget for one /v1/invoke, across all candidates
+queue_secs = 30        # max wait for a concurrency permit
+load_secs = 60         # max time to load/warm one model
+inference_secs = 180   # max time for one provider inference call
+discovery_secs = 8     # per-provider discovery probe
+health_secs = 5        # per-provider health probe
+
+[preflight]
+enabled = true              # master switch for predictive warming
+history_learning = true     # learn capability transition history
+speculative_warming = true  # warm models from hints/history/subscriptions
+confidence_threshold = 0.6  # min confidence (0.0-1.0) before a history prediction warms
+min_samples = 3             # min observed transitions before history is trusted
+subscription_ttl_secs = 3600
+
+[artifacts]
+# dir = "/tmp"            # where generated audio/artifacts are scanned (default: system temp)
+retention_secs = 3600     # delete engine-generated artifacts older than this; 0 disables
+
+[security]
+# Allowlist of directory roots that request `input_file` paths must fall under.
+# Empty (default) allows any local path — appropriate only on a trusted machine.
+# input_roots = ["/Users/me/mofa-workspace"]
+input_roots = []
+
+# Local Ollama (auto-discovered models). Priority is lower = preferred.
+[[providers]]
+name = "ollama"
+kind = "ollama"
+base_url = "http://127.0.0.1:11434"
+priority = 1
+cost_tier = "free"
+
+# Cloud OpenAI-compatible backend. Models are declared explicitly.
+[[providers]]
+name = "openai"
+kind = "openai_compatible"
+base_url = "https://api.openai.com/v1"
+api_key = "${OPENAI_API_KEY}"
+priority = 10
+cost_tier = "high"
+
+  [[providers.models]]
+  name = "gpt-4o"
+  capability = "chat"
+  context_window = 128000
+
+  [[providers.models]]
+  name = "tts-1"
+  capability = "tts"
+
+  [[providers.models]]
+  name = "whisper-1"
+  capability = "asr"
+
+# Local TTS process-adapter backend. Runs a local command (an MLX/Kokoro or
+# Piper-style CLI) once per synthesis and returns the produced audio file.
+# Enable and point `command`/`args` at your installed runtime. Placeholders:
+#   {text}       input text passed as an argument
+#   {text_file}  path to a temp file holding the input text
+#   {output}     path the command must write audio to
+[[providers]]
+name = "local-tts"
+kind = "local_tts"
+base_url = ""            # unused for local_tts, but the field is required
+command = "kokoro-tts"
+args = ["--text-file", "{text_file}", "--output", "{output}"]
+output_format = "wav"
+priority = 2
+enabled = false          # opt-in: set true once the command is installed
+
+  [[providers.models]]
+  name = "kokoro"
+  capability = "tts"
+  memory_mb = 1024

docs/acceptance.md

@@ -14,15 +14,61 @@ This file converts the RFC acceptance checklist into implementation evidence.
 - [x] Ambiguous short model names are rejected.
 - [x] Unsupported provider operations return typed errors.
 
-## MVP Acceptance Still Pending
-
-- [ ] Auto-discover local MLX models.
-- [ ] Local Qwen via Ollama handles type-level LLM calls.
-- [ ] Named GPT request routes to cloud exactly.
-- [ ] Local TTS returns a managed audio artifact.
-- [ ] Hint Preflight starts TTS loading before LLM completion.
-- [ ] Idle models unload and release accounted memory.
-- [ ] Local TTS failure follows documented fallback policy.
-- [ ] HTTP, Rust, and Python interfaces complete equivalent flows.
-- [ ] mofa-fm completes English article to Chinese podcast end to end.
+## Stage 3 — Routing, Concurrency, and Failure Handling
+
+- [x] Routing applies hard constraints, then ranks candidates and returns a
+      reason (`router::route_ranked`).
+- [x] Static memory feasibility filter drops models larger than the budget.
+- [x] Primary selection and failover walk the *same* ranked candidate plan.
+- [x] Capability routing prefers local models under the default policy.
+- [x] Named routing resolves exactly; ambiguous short names are rejected.
+- [x] A retryable local failure falls back to the next valid candidate.
+- [x] Invalid input and unsupported operations do **not** trigger fallback.
+- [x] Named routing is strict by default; `allow_named` opts into fallback.
+- [x] Per-model concurrency admission via semaphores with a queue timeout.
+- [x] Ten concurrent requests respect the model's concurrency limit (no over-admission).
+- [x] Circuit breaker admits exactly one half-open probe.
+- [x] Overall, queue, load, inference, discovery, and health timeouts are configurable.
+
+## Stage 4 — Memory Manager and Model Lifecycle
+
+- [x] Atomic reservation before loading (no time-of-check/time-of-use overcommit).
+- [x] Reserved vs observed memory tracked separately; reconciled after load.
+- [x] Active leases prevent eviction of in-flight models.
+- [x] LRU eviction with incoming/subscription protection.
+- [x] Reservation rolled back on load failure or timeout.
+- [x] Supervised idle-timeout task unloads stale models and releases memory.
+- [x] Lifecycle history and current allocations exposed via `/v1/lifecycle`, `/v1/memory`.
+- [x] Memory pressure returns a structured error when nothing can be freed.
+
+## Stage 5 — Preflight v1
+
+- [x] Explicit hints warm the next model concurrently with the current request.
+- [x] Warm tasks are deduplicated and cancellable; speculative loads go through
+      normal memory admission.
+- [x] Capability subscriptions (app/session-owned, TTL or explicit removal) warm
+      and protect their models.
+- [x] History is keyed by app/session with a global fallback; one app's history
+      does not pollute another's predictions.
+- [x] Predictions are gated by minimum samples and a confidence threshold.
+- [x] Memory-unsafe predictions do not load a model.
+- [x] Preflight effectiveness is visible via events and `/v1/preflight`.
+- [x] History learning and speculative warming can be disabled by config.
+
+## MVP Acceptance Still Pending (needs live backends / later stages)
+
+- [ ] Auto-discover local MLX models. *(Stage 6.)*
+- [x] Engine routes type-level LLM calls to a local model. *(Logic + tests done;
+      live demo needs a running Ollama with Qwen.)*
+- [x] Named request routes to the cloud model exactly. *(Logic + tests done;
+      live demo needs a configured cloud key.)*
+- [ ] Local TTS returns a managed audio artifact. *(Cloud TTS works today; local
+      MLX/Kokoro is Stage 6.)*
+- [x] Hint Preflight starts the next model loading before the current completes.
+- [x] Idle models unload and release accounted memory.
+- [x] A retryable failure follows the documented fallback policy (engine logic);
+      local-TTS-to-cloud demo needs the Stage 6 local backend.
+- [ ] Python UniFFI interface completes equivalent flows. *(Stage 7; HTTP done.)*
+- [ ] mofa-fm completes English article to Chinese podcast end to end. *(Needs
+      live Ollama + TTS; engine-side flow is in place.)*
 

docs/decisions/0002-stage-3-5-routing-memory-preflight.md

@@ -0,0 +1,80 @@
+# ADR 0002: Stage 3-5 — Routing, Memory Lifecycle, and Preflight
+
+## Status
+
+Accepted for the Stage 3-5 implementation.
+
+## Context
+
+ADR 0001 froze the kernel contracts and registry. Stages 3-5 turn that baseline
+into a system that selects models explainably, manages constrained memory
+safely, and reduces cold-start latency without destabilising scheduling. The
+prototype had a single selection path, no real concurrency control, no memory
+reservation, no idle eviction, and a global (cross-application) preflight chain.
+
+## Decisions
+
+### Routing & failure handling (Stage 3)
+
+- The router produces one **ranked candidate plan** (`route_ranked`). The engine
+  walks it in order; the first entry is the primary, each subsequent one a
+  failover. Primary and failover therefore never diverge.
+- A **static memory-feasibility** filter removes local models whose estimate
+  exceeds the whole budget; remote (cloud) models are exempt.
+- Fallback advances **only on retryable errors**. `InvalidRequest` and
+  `UnsupportedOperation` fail immediately. This satisfies "invalid input does
+  not trigger fallback."
+- **Named routing is strict by default** (only the named model is attempted).
+  `fallback_policy = allow_named` appends capability candidates after the named
+  model. Capability requests fail over across all candidates unless
+  `fallback_policy = disabled`.
+- **Concurrency admission** is a per-model `tokio::Semaphore` sized by the
+  model's `max_concurrency`, acquired with a configurable queue timeout. Busy
+  models are no longer hard-filtered from routing; they queue. A queue-wait
+  timeout is retryable, so it can fail over to another candidate.
+- The circuit breaker admits **exactly one half-open probe**; concurrent callers
+  fail fast until the probe's outcome is recorded.
+- All phase timeouts (overall request, queue, load, inference, discovery,
+  health) are configurable under `[timeouts]`.
+
+### Memory & lifecycle (Stage 4)
+
+- Memory is **reserved atomically before loading**. The evict-then-reserve
+  critical section runs under a `load_gate`; the slow backend load runs outside
+  it so independent models can load in parallel without overcommitting.
+- The manager tracks **reserved** (accounting) and **observed** (reported by the
+  backend) bytes separately, reconciling reserved upward to observed.
+- **Leases** protect in-flight models from eviction; eviction and the idle sweep
+  both skip leased and subscription-protected models.
+- A **supervised idle-timeout task** (holding a `Weak<Engine>`, aborted on drop)
+  unloads models idle past `memory.idle_timeout_secs` using monotonic time.
+- Failed/timed-out loads **roll back** the reservation. When nothing can be
+  freed, admission returns a structured `MemoryPressure` error.
+- Lifecycle history (bounded ring buffer) and current allocations are exposed via
+  `/v1/lifecycle` and `/v1/memory`.
+
+### Preflight (Stage 5)
+
+- Signal priority follows the RFC feedback: **hint → subscription → history**.
+- History is a **per-scope Markov chain** keyed by `session_id`, else `app_id`,
+  else a shared global scope. A scope with too little data falls back to global;
+  one application's ordering never leaks into another's predictions.
+- Predictions are gated by `min_samples` and `confidence_threshold`.
+- Warm tasks are **deduplicated per model** and **cancellable**, and route
+  speculative loads through the same reservation/eviction admission as real
+  requests, so a memory-unsafe prediction simply fails to load.
+- Subscriptions are app/session-owned with a TTL or explicit removal; subscribed
+  resident models are eviction-protected.
+- Effectiveness counters (warms, predictions, hits, misses) are exposed via
+  `/v1/preflight`.
+
+## Consequences
+
+- Subscription protection can, by design, make memory pressure unsatisfiable
+  rather than break a keep-alive policy; the engine returns `MemoryPressure` in
+  that case. Revisit if subscription/pressure conflict needs a softer policy
+  (RFC open question 8).
+- History and lifecycle state are in-memory only; persistence across restarts is
+  deferred (RFC open question 11) behind clean seams.
+- The local TTS backend (Stage 6) and Python UniFFI bindings (Stage 7) remain
+  out of scope; cloud TTS and the HTTP API cover the demo path meanwhile.