LM layer: per-module LM, config merging, retry/backoff

[darinkishore]

Context

DSPy Python has a 3-level LM resolution cascade:

1. Call-time parameter (predictor(q="...", lm=other_lm))
2. Per-module (predict.lm = my_lm)
3. Global (dspy.configure(lm=...))

Plus lm.copy(**overrides) for creating variant LMs, and dspy.context(lm=...) for thread-local temporary switching.

dspy-rs currently has global-only LM via GLOBAL_SETTINGS.lm. No per-module assignment, no config merging, no context-style overrides, and no retry/backoff on transient failures.

What's needed

Per-module LM

• Predict<S> and Module should support an optional .lm field
• Resolution order: call param > self.lm > GLOBAL_SETTINGS.lm

Config merging / copy

• LM::copy() or similar to create variant LMs with overridden params
• Equivalent of DSPy's Predict.config + call-time kwargs merge

Retry / backoff

• Configurable num_retries on LM
• Exponential backoff for transient errors (network, rate limit, 5xx)
• Non-transient errors (4xx except 429) fail immediately
• Currently one-shot: any transient failure is terminal

Context-local LM

• Something like dspy::context(lm) for temporary overrides (task-local or scope-based)

Related

• LM tool loop treats max_tool_iterations=1 as immediate failure #61 (tool loop off-by-one)
• proper error handling #67 (error handling)

[darinkishore]

Codebase audit — current LM resolution

settings.rs — global only:

pub static GLOBAL_SETTINGS: LazyLock<RwLock<Option<Settings>>> = ...;
pub fn get_lm() -> Arc<LM> { ... }
pub fn configure(lm: LM, adapter: impl Adapter + 'static) { ... }

Settings stores Arc<LM> (concrete type, not trait object). No per-module, no call-time, no context-local.

Predict struct — no LM field:

pub struct Predict<S: Signature> { ... }
// forward() calls get_lm() directly

Dependency chain for this issue:

1. DummyLM should share a trait with LM #77 first — extract LanguageModel trait from LM/DummyLM. Without a trait, you can't store Arc<dyn LanguageModel> in Settings or Predict.

2. proper error handling #67 partially — retry/backoff needs typed LmCallError to distinguish transient (429, 5xx, network) from permanent (401, 400) errors.

3. Then this issue becomes:

   • Add lm: Option<Arc<dyn LanguageModel>> to Predict<S>
   • Resolution: self.lm.unwrap_or_else(|| get_lm())
   • LM::copy(**overrides) or a builder .with_temperature(0.0) pattern
   • Retry: wrap the client.completion() call with exponential backoff on transient errors
   • Context-local: tokio::task_local! or a scoped guard that temporarily overrides GLOBAL_SETTINGS.lm

DSPy Python reference (for parity):

• dspy.context(lm=...) → thread-local temporary override
• predictor.lm = my_lm → per-module
• predictor(question="...", lm=other_lm) → call-time
• lm.copy(temperature=0.0) → variant creation
• lm.num_retries with exponential backoff, 429/5xx are transient