Skip to content

Mirror reasoning summaries when requested#43

Open
Ali-Aleph-Alpha wants to merge 4 commits intomainfrom
fix/reasoning-summary-raw
Open

Mirror reasoning summaries when requested#43
Ali-Aleph-Alpha wants to merge 4 commits intomainfrom
fix/reasoning-summary-raw

Conversation

@Ali-Aleph-Alpha
Copy link
Copy Markdown

@Ali-Aleph-Alpha Ali-Aleph-Alpha commented Apr 29, 2026

Summary

  • Adds opt-in reasoning summaries for Responses API reasoning items via reasoning.summary.
  • When callers request reasoning.summary: "auto" or "raw", raw reasoning text is mirrored into output[].summary[] as summary_text while preserving the existing raw content[] reasoning payload.
  • Streams OpenAI-compatible reasoning summary events (response.reasoning_summary_part.* and response.reasoning_summary_text.*) so streaming clients can consume summaries as they arrive.

Why

PydanticAI intentionally treats raw chain-of-thought differently from reasoning summaries. Raw reasoning in content[] is stored in provider_details["raw_content"], so the default AG-UI adapter does not show it and PydanticAI's OTEL serializer does not include it in Langfuse traces.

By emitting summaries in the same shape OpenAI Responses uses, PydanticAI can populate ThinkingPart.content without app-specific workarounds. This should let clients like tiger-chat remove the custom RawCoTAdapter path and avoid separate Langfuse/OTEL hacks once they opt in with openai_reasoning_summary="auto".

Compatibility

  • Default behavior is unchanged: without reasoning.summary, summary[] remains empty and existing raw reasoning events continue to stream.
  • The new behavior is opt-in through reasoning.summary, so existing callers should not see reasoning exposed differently unless they request it.
  • "none" is accepted as an explicit opt-out; unsupported summary modes still fail fast.

Test plan

  • pnpm --dir /Users/mohamed.ali/repos/aleph-alpha/responses.js run test -- src/routes/responses/handleOneTurn.test.ts src/routes/responses/closeOutputItem.test.ts src/routes/responses/innerStream.test.ts src/schemas.test.ts
  • pnpm --dir /Users/mohamed.ali/repos/aleph-alpha/responses.js run check
  • pnpm --dir /Users/mohamed.ali/repos/aleph-alpha/responses.js exec prettier --check src/openai_patch.ts src/schemas.ts src/schemas.test.ts src/routes/responses/innerStream.ts src/routes/responses/innerStream.test.ts src/routes/responses/handleOneTurn.ts src/routes/responses/handleOneTurn.test.ts src/routes/responses/closeOutputItem.ts src/routes/responses/closeOutputItem.test.ts

ref: https://aleph-alpha.atlassian.net/wiki/spaces/CHAT/pages/2507604161/CHAT-RFC-011+Reasoning+visibility+with+Stateful+Responses+API

Populate reasoning summary text for opt-in requests so downstream OpenAI-compatible clients can surface reasoning without client-side adapter patches.
@github-actions
Copy link
Copy Markdown

github-actions Bot commented Apr 29, 2026

⚠️ Deprecation Warning: The deny-licenses option is deprecated for possible removal in the next major release. For more information, see issue 997.

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Scanned Files

None

@Ali-Aleph-Alpha Ali-Aleph-Alpha marked this pull request as draft April 29, 2026 14:56
Emit OpenAI-compatible reasoning summary stream events for opt-in requests so PydanticAI can populate ThinkingPart content during streaming.
@Ali-Aleph-Alpha Ali-Aleph-Alpha marked this pull request as ready for review April 29, 2026 19:32
@frac frac requested a review from Copilot April 30, 2026 06:11
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds opt-in support for exposing reasoning summaries in an OpenAI Responses-compatible shape, primarily by mirroring raw reasoning into output[].summary[] and emitting corresponding streaming events so downstream clients can consume “thinking” summaries without custom adapters.

Changes:

  • Extends request schema to accept additional reasoning.summary modes (raw, none).
  • Streams new reasoning-summary events and mirrors reasoning text into output[].summary[] when requested.
  • Closes reasoning-summary stream lifecycle with corresponding *.done events and adds/updates tests.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
src/schemas.ts Extends reasoning.summary enum with raw/none (but still includes concise/detailed).
src/schemas.test.ts Updates schema tests to cover summary: "raw" and opt-out via "none".
src/routes/responses/innerStream.ts Validates supported summary modes and forwards the mode into handleOneTurnStream.
src/routes/responses/innerStream.test.ts Updates tests for new validation error text and verifies mode forwarding.
src/routes/responses/handleOneTurn.ts Mirrors reasoning text into summary[] and emits response.reasoning_summary_* streaming events when requested.
src/routes/responses/handleOneTurn.test.ts Adds coverage for mirroring behavior and default empty summaries.
src/routes/responses/closeOutputItem.ts Emits reasoning_summary_* .done events when closing a reasoning item with summaries.
src/routes/responses/closeOutputItem.test.ts Adds test ensuring summary done events are emitted.
src/openai_patch.ts Adds patched event/type definitions for reasoning summary stream events.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/schemas.ts Outdated
Comment thread src/schemas.test.ts
Comment thread src/routes/responses/closeOutputItem.test.ts
Comment thread src/routes/responses/handleOneTurn.ts Outdated
Comment thread src/routes/responses/handleOneTurn.ts
Copy link
Copy Markdown

@phvalguima phvalguima left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have the impression the sequence_number will not be followed correctly with this change, can we confirm this is or not the case?

Comment thread src/routes/responses/closeOutputItem.ts Outdated
output_index: responseObject.output.length - 1,
summary_index: currentReasoningItem.summary.length - 1,
delta: reasoningText as string,
sequence_number: SEQUENCE_NUMBER_PLACEHOLDER,
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is response.reasoning_summary_text.done? It feels this if (mirrorRawReasoningToSummary) { should also be done on the closeOutputItem.ts.

Refactor reasoning summary handling to support detailed summaries and adjust related tests. The reasoning summary schema now excludes "raw" and "none" options, and tests have been updated to reflect these changes, ensuring correct behavior for omitted summaries and detailed reasoning. Additionally, new tests verify the default behavior of reasoning summary events.
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/routes/responses/handleOneTurn.ts
Comment thread src/routes/responses/handleOneTurn.ts
Copy link
Copy Markdown

@phvalguima phvalguima left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think these issues still apply: #43 (comment)

Comment thread src/routes/responses/closeOutputItem.ts Outdated
output_index: responseObject.output.length - 1,
summary_index: summaryIndex,
part,
sequence_number: SEQUENCE_NUMBER_PLACEHOLDER,
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happened with this check?

The SEQUENCE_NUMBER_PLACEHOLDER is not getting updated. So, this problem still applies:

response.reasoning_text.done
REASONING_TEXT
sequence_number: 10

response.reasoning_summary_text.done
REASONING_TEXT
sequence_number: 10

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did look into it, and it seems to work fine and ran the tests fine as well.

so from my limited understanding of the code base and with the help of cursor, the inner stream uses SEQUENCE_NUMBER_PLACEHOLDER only as a local sentinel; runCreateResponseStream rewrites sequence_number for every emitted event with a monotonic counter. I added a regression test (rewrites inner placeholder sequence numbers so reasoning events never collide) that simulates two inner events both set to -1 and verifies the streamed events have unique, strictly increasing sequence numbers, so those reasoning events can’t collide on the wire.

@Ali-Aleph-Alpha Ali-Aleph-Alpha requested a review from phvalguima May 4, 2026 16:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants