Skip to content

fix(model-router): bound SSE rewriter holdback buffer#1933

Open
Lightheartdevs wants to merge 1 commit into
mainfrom
fix/sse-rewriter-bounded-holdback
Open

fix(model-router): bound SSE rewriter holdback buffer#1933
Lightheartdevs wants to merge 1 commit into
mainfrom
fix/sse-rewriter-bounded-holdback

Conversation

@Lightheartdevs

Copy link
Copy Markdown
Collaborator

Fix-forward carry-over from #1914/#1931 review

_SSERewriter.buffer appended every streamed chunk but only trimmed it when an SSE delimiter matched. An upstream that streams without a delimiter grew the buffer without bound — MAX_BODY_BYTES only guards the inbound request body, not the rewriter's held-back partial event. (Flagged in the #1914 Claude-lane review; #1920 had bounded its holdback at 1MB.)

Change

  • New MAX_SSE_HOLDBACK (default 1MB, ODS_ROUTER_MAX_SSE_HOLDBACK_BYTES).
  • _SSERewriter.feed() flushes the partial buffer via _rewrite_sse_event (same best-effort rewrite finish() does) once it exceeds the cap, then resets. Memory bounded; framing continues correctly afterward.
  • Read-path only; no observe/legacy parity impact.

Test

  • test_sse_holdback_is_bounded_without_delimiter: a delimiter-less oversized chunk is flushed (not held), buffer stays ≤ cap, subsequent well-framed event still frames.
  • Full router suite: 39 passed, ruff clean.

🤖 Generated with Claude Code

_SSERewriter.buffer accumulated every chunk but only trimmed when an SSE
delimiter matched, so an upstream that streamed without a delimiter grew
it without limit (MAX_BODY_BYTES only guards the request body). Bound the
held-back partial event at MAX_SSE_HOLDBACK (1MB, env-overridable): once
exceeded, flush it via _rewrite_sse_event (same best-effort rewrite as
finish()) and reset. Fix-forward carry-over from the #1914/#1931 review
(mirrors the 1MB holdback bound noted in #1920).

Adds test_sse_holdback_is_bounded_without_delimiter; full router suite
(39 tests) green, ruff clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@Lightheartdevs
Lightheartdevs force-pushed the fix/sse-rewriter-bounded-holdback branch from a74d232 to 7f9a499 Compare July 21, 2026 08:28
@Lightheartdevs

Copy link
Copy Markdown
Collaborator Author

Thanks for taking on the model-router streaming edge case; bounding the SSE rewriter holdback is a useful hardening step for malformed or delimiter-less upstream streams.

Advisory status: looks merge-ready

What looks good:

  • ods/extensions/services/model-router/app/main.py now caps _SSERewriter.buffer with ODS_ROUTER_MAX_SSE_HOLDBACK_BYTES, so an upstream that never sends an SSE delimiter cannot grow memory without bound.
  • The overflow path uses the same _rewrite_sse_event() best-effort rewrite behavior as finish(), then resets the buffer, which keeps the existing alias-redaction path coherent instead of bypassing it.
  • ods/extensions/services/model-router/tests/test_router.py adds focused regression coverage for the delimiter-less stream and verifies framing still works after a forced flush.

What I checked:

  • Reviewed the PR title, description, changed files, and patch for fix(model-router): bound SSE rewriter holdback buffer #1933.
  • Ran python -m py_compile extensions/services/model-router/app/main.py locally on this head.
  • Ran python -m pytest extensions/services/model-router/tests/test_router.py -q locally: 42 passed, with only existing FastAPI on_event deprecation warnings.
  • Ran git diff --check origin/main...HEAD locally.
  • Confirmed GitHub checks are green, including Python lint, type check, dashboard API/frontend, shell/PowerShell lint, integration smoke, matrix smoke, macOS smoke, secret scan, and env validation.

Thanks again for pushing this forward.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants