feat: continue interrupted generation streams

dwctl/src/sync/onwards_config/mod.rs

@@ -938,6 +938,7 @@ fn convert_composite_to_target_spec(
                 .and_then(|n| usize::try_from(n).ok().filter(|&v| v >= 1)),
             backoff,
             max_total_backoff_ms,
+            stream_continuation: None,
         })
     } else {
         None
@@ -1202,6 +1203,7 @@ fn convert_to_config_file(
                         .and_then(|n| usize::try_from(n).ok().filter(|&v| v >= 1)),
                     backoff,
                     max_total_backoff_ms,
+                    stream_continuation: None,
                 })
             } else {
                 None

onwards/CHANGELOG.md

@@ -23,6 +23,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Added
+
+- add best-effort prefix continuation for eligible Completions, Chat Completions, and Responses streams
+
+### Changed
+
+- **Rust API migration (next minor):** `FallbackConfig` now includes `stream_continuation`; callers using exhaustive struct literals must initialize it, usually with `None`
+
 ## [0.35.6](https://github.com/doublewordai/onwards/compare/v0.35.5...v0.35.6) - 2026-07-21
 
 ### Fixed

onwards/Cargo.toml

@@ -67,6 +67,7 @@ bon = "3.6.5"
 subtle = "2.6.1"
 axum-prometheus = "0.10.0"
 metrics = "0.24"
+mime = "0.3"
 governor = "0.10.1"
 rand = "0.9"
 uuid = { version = "1", features = ["v4"] }

onwards/README.md

@@ -51,12 +51,91 @@ curl -X POST http://localhost:3000/v1/chat/completions \
 - Rate limiting and concurrency limiting (per-target and per-key)
 - Load balancing with weighted random and priority strategies
 - Automatic failover across multiple providers
+- Best-effort continuation of interrupted text-generation streams
 - Strict mode for request validation and error standardization
 - Response sanitization for OpenAI schema compliance
 - Prometheus metrics
 - Custom response headers
 - Optional multi-step Open Responses orchestration loop (`multi-step` feature)
 
+## Stream Continuation
+
+Stream continuation is an opt-in fallback mode for eligible streaming requests
+to `/v1/completions`, `/v1/chat/completions`, and `/v1/responses`. If an
+upstream stops before a terminal event, Onwards sends the text already emitted
+to another provider selected by the pool, which may be the same provider.
+
+Configure it inside the pool's `fallback` settings:
+
+```json
+{
+  "targets": {
+    "gpt-4": {
+      "fallback": {
+        "enabled": true,
+        "on_status": [429, 5],
+        "stream_continuation": {
+          "enabled": true,
+          "endpoints": [
+            "/v1/completions",
+            "/v1/chat/completions",
+            "/v1/responses"
+          ],
+          "max_attempts": 1,
+          "max_buffered_bytes": 1048576,
+          "idle_timeout_ms": 30000
+        }
+      },
+      "providers": [
+        { "url": "https://primary.example.com", "onwards_key": "sk-primary" },
+        { "url": "https://backup.example.com", "onwards_key": "sk-backup" }
+      ]
+    }
+  }
+}
+```
+
+| Option | Default | Description |
+|--------|---------|-------------|
+| `enabled` | `false` | Enables continuation, subject to `fallback.enabled` |
+| `endpoints` | `[]` | Exact supported request paths eligible for continuation |
+| `max_attempts` | `1` | Fresh continuation-attempt budget after the response starts |
+| `max_buffered_bytes` | `1048576` | Maximum emitted-text bytes retained for a continuation request |
+| `idle_timeout_ms` | `null` | Maximum idle time between upstream events; `null` disables the timeout |
+
+`fallback.max_attempts` controls retries before response headers are committed.
+`stream_continuation.max_attempts` is a separate budget after streaming starts.
+Continuation reuses the pool's provider selection, request headers, status
+matching, local rate-limit behavior, and backoff configuration. Its cumulative
+backoff budget is also fresh.
+
+Only narrow, single-output text requests are eligible. Completions require a
+string prompt, one choice, no echo, and `stream: true`. Chat Completions require
+ordinary message content, one choice, no logprobs, and `stream: true`. Responses
+require string or message-only input, plain-text foreground output, no storage,
+and `stream: true`. Tool calls, reasoning, structured output, multi-choice or
+multi-item output, and provider-specific guided generation are excluded.
+Unsupported requests pass through normally. If an eligible stream later emits
+an unknown event shape, continuation is disabled rather than guessing how to
+splice it.
+
+Completions append emitted text to the original prompt. Chat Completions append
+one assistant message and suppress a repeated assistant-role chunk. Responses
+append an assistant `output_text` message while preserving response and item
+identities, sequence numbers, and the cumulative terminal text snapshot.
+
+Eligible upstream responses must be successful, identity-encoded SSE using the
+`text/event-stream` media type. Onwards requests `Accept-Encoding: identity` and
+rejects an encoded or non-SSE continuation response instead of splicing it.
+
+This is prompt-prefix continuation, not native token-offset resume. A provider
+may repeat text, add a preamble, or diverge. The emitted prefix is buffered only
+in process memory and is lost on restart. Exceeding the buffer cap disables
+continuation for that request. Exhausted retries can leave the client with a
+partial stream and no synthetic terminal event. Usage comes from the provider
+that supplies the terminal stream and is not aggregate usage or billing across
+attempts.
+
 ## Multi-Step Open Responses
 
 The `multi-step` Cargo feature adds an orchestration loop that drives

onwards/docs/src/load-balancing.md

@@ -37,6 +37,11 @@ Controls automatic retry on other providers when requests fail:
 | `enabled` | bool | `false` | Master switch for fallback |
 | `on_status` | int[] | -- | Status codes that trigger fallback (supports wildcards) |
 | `on_rate_limit` | bool | `false` | Fallback when hitting local rate limits |
+| `with_replacement` | bool | `false` | For `weighted_random`, allow a provider to be selected again |
+| `max_attempts` | int? | provider count | Maximum pre-response failover attempts |
+| `backoff` | object? | `null` | Delay between attempts; no delay when omitted |
+| `max_total_backoff_ms` | int? | `null` | Maximum cumulative time spent sleeping between attempts |
+| `stream_continuation` | object? | `null` | Best-effort continuation for eligible interrupted text-generation streams |
 
 Status code wildcards:
 
@@ -46,6 +51,105 @@ Status code wildcards:
 
 When fallback triggers, the next provider is selected based on strategy (weighted random resamples from remaining pool; priority uses definition order).
 
+### Stream continuation
+
+Stream continuation is opt-in for eligible `POST /v1/completions`,
+`POST /v1/chat/completions`, and `POST /v1/responses` streams. When an upstream
+stops before a terminal event, Onwards sends the text already emitted to a newly
+selected provider, which may be the same provider. The prefix is kept in memory
+for the lifetime of the request; it is not persisted and is lost if the process
+restarts. An interruption or exhausted continuation budget can leave the client
+with a partial stream, and the proxy does not synthesize a terminal event.
+
+Configure it inside `fallback`:
+
+```json
+{
+  "targets": {
+    "gpt-4": {
+      "fallback": {
+        "enabled": true,
+        "on_status": [429, 5],
+        "on_rate_limit": true,
+        "stream_continuation": {
+          "enabled": true,
+          "endpoints": [
+            "/v1/completions",
+            "/v1/chat/completions",
+            "/v1/responses"
+          ],
+          "max_attempts": 1,
+          "max_buffered_bytes": 1048576,
+          "idle_timeout_ms": 30000
+        }
+      },
+      "providers": [
+        { "url": "https://primary.example.com", "onwards_key": "sk-primary" },
+        { "url": "https://backup.example.com", "onwards_key": "sk-backup" }
+      ]
+    }
+  }
+}
+```
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `enabled` | bool | `false` | Enables continuation, subject to the parent `fallback.enabled` switch |
+| `endpoints` | string[] | `[]` | Exact request paths eligible for continuation; supported values are `/v1/completions`, `/v1/chat/completions`, and `/v1/responses` |
+| `max_attempts` | int | `1` | Fresh post-commit continuation-attempt budget |
+| `max_buffered_bytes` | int | `1048576` | Maximum generated-text bytes retained for a continuation request |
+| `idle_timeout_ms` | int? | `null` | Maximum idle time between upstream stream events; `null` disables this timeout |
+
+`fallback.max_attempts` controls failover before the response starts and its
+headers are committed. `stream_continuation.max_attempts` is a separate, fresh
+budget after a successful stream has committed its response. Continuation
+reuses the parent fallback status matching, local rate-limit behavior, provider
+selection, request/header handling, and backoff settings, including
+`max_total_backoff_ms`. Stream continuation also starts a fresh cumulative
+backoff budget; the parent `max_total_backoff_ms` limit applies independently
+to the post-commit continuation attempts.
+
+Only narrow, single-output text shapes are supported. Completions require a
+string `prompt`, `stream: true`, `n` omitted or `1`, and `echo` omitted or
+`false`. Chat Completions require ordinary message content, `stream: true`, one
+choice, and no logprobs. Responses require string or message-only input,
+`stream: true`, plain-text output, and foreground, non-stored execution. Tool
+calls, reasoning, structured output, multi-choice or multi-item output, and
+provider-specific guided generation controls are excluded. Unsupported requests
+pass through normally without continuation. If an initially eligible stream
+later emits an unrecognized shape, continuation is disabled rather than guessing
+how to splice it.
+
+Continuation is protocol-specific. For Completions, Onwards appends the emitted
+text to the original prompt. For Chat Completions, it appends one assistant
+message containing the emitted text and suppresses a repeated assistant-role
+chunk. For Responses, it appends an assistant `output_text` message, suppresses
+repeated lifecycle setup events, and keeps response IDs, item IDs, and sequence
+numbers continuous. Terminal Responses snapshots are rewritten with the full
+cumulative text.
+
+The upstream response must be successful, use the exact `text/event-stream`
+media type (case-insensitive; parameters are allowed), and have no content
+encoding or `Content-Encoding: identity`. Onwards sends
+`Accept-Encoding: identity` for eligible requests and continues only unencoded
+responses. An encoded eligible stream is left untouched; an encoded or non-SSE
+continuation response is rejected rather than spliced into the client stream.
+In strict mode, body sanitization cannot be guaranteed when an eligible encoded
+response ignores the identity request because preserving its encoded
+representation requires forwarding it unwrapped.
+
+This is prefix-based continuation, not native token-offset resume. Chat
+Completions and Responses do not expose a continuation cursor: the emitted text
+is supplied as prior assistant output in a new request. The next model call may
+add a preamble, repeat text, or diverge from the interrupted generation.
+
+The buffer cap is measured in UTF-8 bytes. Once appending a recognized text
+chunk would exceed the cap, continuation is disabled for that request and the
+overflow text is not retained. `idle_timeout_ms` applies between upstream
+events and treats an idle stream as interrupted. Final usage is forwarded from
+the provider that supplies the terminal stream; it is not aggregate usage or
+aggregate billing across the initial and continuation providers.
+
 ## Pool-level options
 
 Settings that apply to the entire alias: