Fix gemini OpenAI chat tool calling 2 #264

zikajk · 2026-01-13T12:15:36Z

I added a entry in changelog under unreleased section.

Handle OpenAI-compatible providers that send finish_reason=stop alongside tool_calls by deferring :finish, return executed tools from the accumulator, and emit a finish only when no valid tools can run to prevent hung prompts.

Emit string :content for text-only messages (including reason/tag fallback) to improve Gemini/OpenAI-compat payload shape, and update tests to expect string content.

zikajk · 2026-01-13T13:24:17Z

@ericdallo Do you think the failing build can be related to my changes?

ericdallo · 2026-01-13T13:48:54Z

no, I did a mistake in master that I'm taking a look, I let you know when fixed

ericdallo · 2026-01-13T14:08:54Z

src/eca/llm_providers/openai_chat.clj

    (string? content)
-    [{:type "text"
-      :text (string/trim content)}]
+    (string/trim content)


I used maps to keep consistency in the data structure, but I'm ok if that fix problems, I just hope to not cause any other

It is supported, but often not recommended so I decided to take the safe route here :-).
And I suppose it will save some tokens as well.

I don't think tokens are calculated based on the structure, but on the chars content, but I'm ok with that

zikajk · 2026-01-13T14:30:17Z

@ericdallo The Gemini is still buggy though. It looks like its OPENAI api isn't perfect.
E.g. it sends us though> with missing < which is fed back to the model and it gets very confused.

charmbracelet/crush#1698

zikajk · 2026-01-13T16:00:04Z

@ericdallo Regarding the rare problem with broken gemini thinking tag. It is not something I will fix in this PR.

ericdallo · 2026-01-13T19:47:42Z

@zikajk does that means it's a bug only with gemini or other providers? is it possible to repro with other providers so I can try?

zikajk · 2026-01-14T10:14:08Z

@zikajk does that means it's a bug only with gemini or other providers? is it possible to repro with other providers so I can try?

Only with Gemini. I have an experimental branch where reasoning is purged from history (user can opt-out via provider config).
And what I have found so far, it is quite possible that Gemini (and maybe other models as well?) doesn't even like to have reasoning returned in the same turn (nothing in official docs though...).
I want to see if the problem is gone when i won't send ANY reasoning back. If yes, then I would send "turn reasoning" only for models doing delta-reasoning (Deepseek) - again with a opt-out via provider config.

ericdallo · 2026-01-14T12:50:57Z

it is quite possible that Gemini (and maybe other models as well?) doesn't even like to have reasoning returned in the same turn (nothing in official docs though...).

I see, but for other models I'd say this is unlikely, as at least major models like openai and anthropic this is required, but let's test, but I'd say it should affect model quality

zikajk · 2026-01-14T13:40:32Z

it is quite possible that Gemini (and maybe other models as well?) doesn't even like to have reasoning returned in the same turn (nothing in official docs though...).

I see, but for other models I'd say this is unlikely, as at least major models like openai and anthropic this is required, but let's test, but I'd say it should affect model quality

Can you point me to the documentation? I know they use different API but I am curious.

ericdallo · 2026-01-14T14:41:57Z

Can you point me to the documentation? I know they use different API but I am curious.

https://platform.openai.com/docs/api-reference/responses/create

Includes an encrypted version of reasoning tokens in reasoning item outputs. This enables reasoning items to be used in multi-turn conversations when using the Responses API statelessly (like when the store parameter is set to false, or when an organization is enrolled in the zero data retention program).

Anthropic has the same here, mentioning to pass back thinking blocks

zikajk · 2026-01-14T15:03:53Z

@ericdallo

Can you point me to the documentation? I know they use different API but I am curious.

https://platform.openai.com/docs/api-reference/responses/create

Includes an encrypted version of reasoning tokens in reasoning item outputs. This enables reasoning items to be used in multi-turn conversations when using the Responses API statelessly (like when the store parameter is set to false, or when an organization is enrolled in the zero data retention program).

Anthropic has the same here, mentioning to pass back thinking blocks

The Gemini also requires to keep the encrypted version (thought_signatures).
And the Anthropic behavior is the one I consider to be the best default. Only this part is something I wonder about (if other providers throws past thinking tokens away):

While you can omit thinking blocks from prior assistant role turns, we suggest always passing back all thinking blocks to the API for any multi-turn conversation. The API will:

Automatically filter the provided thinking blocks
Use the relevant thinking blocks necessary to preserve the model's reasoning
Only bill for the input tokens for the blocks shown to Claude

But yeah, I am keeping the mentioned branch experimental and I don't plan it to make it PR until I am absolutely sure that it helps.
Not related to this branch though :-)

ericdallo

Is this done? can you add changelog entry please?

zikajk · 2026-01-14T17:50:29Z

@ericdallo Done.

zikajk added 2 commits January 13, 2026 11:29

Defer stream finish until tool loop completes

1332446

Handle OpenAI-compatible providers that send finish_reason=stop alongside tool_calls by deferring :finish, return executed tools from the accumulator, and emit a finish only when no valid tools can run to prevent hung prompts.

Normalize text-only content to strings for Openai chat

2de7889

Emit string :content for text-only messages (including reason/tag fallback) to improve Gemini/OpenAI-compat payload shape, and update tests to expect string content.

zikajk requested a review from ericdallo January 13, 2026 12:15

Merge branch 'master' into fix-gemini-openai-chat-tool-calling-2

755109a

Merge branch 'master' into fix-gemini-openai-chat-tool-calling-2

8f63e37

ericdallo reviewed Jan 13, 2026

View reviewed changes

Merge branch 'master' into fix-gemini-openai-chat-tool-calling-2

5622a2e

ericdallo approved these changes Jan 14, 2026

View reviewed changes

zikajk and others added 2 commits January 14, 2026 18:38

Merge branch 'master' into fix-gemini-openai-chat-tool-calling-2

4b08749

update changelog

f5cc5de

ericdallo merged commit dbdf726 into master Jan 14, 2026
9 checks passed

Uh oh!

Fix gemini OpenAI chat tool calling 2 #264

Fix gemini OpenAI chat tool calling 2 #264

Conversation

zikajk commented Jan 13, 2026

Uh oh!

zikajk commented Jan 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ericdallo commented Jan 13, 2026

Uh oh!

ericdallo Jan 13, 2026

Choose a reason for hiding this comment

Uh oh!

zikajk Jan 13, 2026

Choose a reason for hiding this comment

Uh oh!

ericdallo Jan 13, 2026

Choose a reason for hiding this comment

Uh oh!

zikajk commented Jan 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zikajk commented Jan 13, 2026

Uh oh!

ericdallo commented Jan 13, 2026

Uh oh!

zikajk commented Jan 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ericdallo commented Jan 14, 2026

Uh oh!

zikajk commented Jan 14, 2026

Uh oh!

ericdallo commented Jan 14, 2026

Uh oh!

zikajk commented Jan 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ericdallo left a comment

Choose a reason for hiding this comment

Uh oh!

zikajk commented Jan 14, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

zikajk commented Jan 13, 2026 •

edited

Loading

zikajk commented Jan 13, 2026 •

edited

Loading

zikajk commented Jan 14, 2026 •

edited

Loading

zikajk commented Jan 14, 2026 •

edited

Loading