Integration test coverage gaps (non-blocking)

[rlippmann]

Description

There are known gaps in integration-level testing that are intentionally not covered by current automated tests.

Current test coverage focuses on deterministic logic contracts (engine, checkpoint, precompiler), but does not fully cover real runtime environments.

---

Known gaps

• Live provider end-to-end calls:

  • OpenAI
  • Ollama
  • LiteLLM
  • OpenWebUI

• True OpenWebUI runtime loading:

  • pipe import behavior
  • frontmatter install behavior inside a running OpenWebUI instance

• LiteLLM proxy lifecycle:

  • callback behavior under real proxy server
  • concurrency / multi-request handling

• Streaming / chunked responses:

  • upstream streaming through integrations

• Cross-process / multi-worker behavior:

  • persistence across processes
  • non in-memory deployment scenarios

---

Current state

The following are already well covered:

• import safety and dependency handling
• configuration and error normalization
• checkpoint persistence and restore behavior
• precompiler fallback validation and reject-first behavior
• pending-clarification bypass logic

---

Why this is deferred

• requires external systems and networked environments
• introduces flakiness in CI
• not required for validating core correctness or contract behavior

---

Suggested future work

• optional local smoke test scripts (not CI)
• environment-gated integration tests
• selective end-to-end validation for demos or production readiness

---

Notes

This issue tracks test debt only.

It does not block any current release and should be addressed only if/when integration realism becomes a priority.