fix(api): support Claude Code Anthropic tool flows with Qwen chat templates by ace-xc · Pull Request #5051 · xorbitsai/inference

ace-xc · 2026-06-18T09:31:40Z

Summary

This PR improves Anthropic-compatible message handling for Claude Code style requests and fixes Qwen/Qwen3 tool-call prompt rendering failures.

Claude Code sends Anthropic Messages API payloads with content blocks such as system, tool_use, and tool_result. These need to be converted before reaching Xinference model backends and chat templates. Without normalization, Qwen/Qwen3 templates can receive OpenAI-style tool call arguments as JSON strings and fail while rendering Jinja templates with:

TypeError: Can only get item pairs from a mapping.

Changes

Convert Anthropic-style messages into OpenAI-style internal messages:
- fold top-level and in-array system content into a leading system message
- flatten text content blocks
- convert tool_use blocks into assistant tool_calls
- convert tool_result blocks into tool role messages
- convert Anthropic tools / tool_choice into OpenAI-compatible shapes
Normalize tool-call arguments before chat template rendering:
- detect templates that iterate over tool_call.arguments with |items / .items()
- parse JSON-string tool arguments into mappings only for those templates
- keep protocol-facing OpenAI responses unchanged, where function.arguments remains a JSON string
Harden tool schema handling for templates that iterate over tool.parameters.properties.
Make Anthropic response conversion accept both string and dict function.arguments.
Add compatibility fallback for BuildGradioEmbeddingInterfaceRequest to avoid startup failures from mixed/stale installs.

gemini-code-assist

Code Review

This pull request introduces compatibility and conversion layers to support Anthropic-style Messages API requests (such as those from Claude Code) by mapping them into OpenAI-style messages and tools expected by Xinference backends. It also adds normalization helpers to coerce tool arguments and parameters into dictionaries when required by chat templates. The reviewer feedback suggests loosening the string-matching logic in _chat_template_needs_mapping_tool_arguments and _chat_template_needs_mapping_tool_parameters to ensure compatibility with templates using different variable names.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

gemini-code-assist · 2026-06-18T09:33:31Z

+    def _chat_template_needs_mapping_tool_arguments(chat_template: str) -> bool:
+        compact_template = "".join(chat_template.split())
+        return (
+            "tool_call.arguments|items" in compact_template
+            or "tool_call.arguments.items()" in compact_template
+            or (
+                (
+                    "_args=tool_call.arguments" in compact_template
+                    or "_args=tc.arguments" in compact_template
+                )
+                and "_args.items()" in compact_template
+            )
+        )


这里的匹配逻辑过于严格。如果 chat template 中直接使用 tool_call.function.arguments|items 或 call.arguments|items（而不是 tool_call.arguments|items），当前的检测逻辑将会漏掉，导致模板渲染失败。\n\n由于 _normalize_messages_for_chat_template 只会针对消息中的 tool_calls 进行参数规范化，因此将匹配范围放宽到 "arguments|items" 和 "arguments.items()" 是完全安全且更具鲁棒性的。

@staticmethod\n def _chat_template_needs_mapping_tool_arguments(chat_template: str) -> bool:\n compact_template = "".join(chat_template.split())\n return (\n "arguments|items" in compact_template\n or "arguments.items()" in compact_template\n or (\n (\n "_args=tool_call.arguments" in compact_template\n or "_args=tc.arguments" in compact_template\n )\n and "_args.items()" in compact_template\n )\n )

gemini-code-assist · 2026-06-18T09:33:31Z

+    def _chat_template_needs_mapping_tool_parameters(chat_template: str) -> bool:
+        compact_template = "".join(chat_template.split())
+        return (
+            "tool.parameters.properties|items" in compact_template
+            or "tool.parameters.properties.items()" in compact_template
+        )


与工具参数的匹配类似，如果模板中使用的是 tool.function.parameters.properties|items 或者是其他变量名（例如 parameter.properties|items），当前的硬编码检测也会失效。\n\n建议将匹配条件放宽为 "parameters.properties|items" 和 "parameters.properties.items()"，以提高对不同模型模板的兼容性。

@staticmethod\n def _chat_template_needs_mapping_tool_parameters(chat_template: str) -> bool:\n compact_template = "".join(chat_template.split())\n return (\n "parameters.properties|items" in compact_template\n or "parameters.properties.items()" in compact_template\n )

qinxuye · 2026-06-18T09:34:53Z

Thanks, could you use English for PR title and content?

ace-xc · 2026-06-18T09:40:28Z

Thanks, could you use English for PR title and content?

ok, I have edited it.

qinxuye · 2026-06-18T10:02:58Z

Lint(pre-commit) failed, could you fix it?

ace-xc · 2026-06-18T10:06:43Z

OK, I'll deal with it later.

…

---- Replied Message ---- | From | Chris ***@***.***> | | Date | 06/18/2026 18:03 | | To | xorbitsai/inference ***@***.***> | | Cc | xc ***@***.***>, Author ***@***.***> | | Subject | Re: [xorbitsai/inference] fix(api): support Claude Code Anthropic tool flows with Qwen chat templates (PR #5051) | qinxuye left a comment (xorbitsai/inference#5051) Lint(pre-commit) failed, could you fix it? — Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS and Android. Download it today! You are receiving this because you authored the thread.Message ID: ***@***.***>

qinxuye · 2026-06-18T10:08:10Z

Also, gemini gives some comments, could you please resolve them?

qinxuye · 2026-06-18T12:47:35Z

#5049 seemed have solved this issue, can you confirm that?

Address review feedback on xorbitsai#5051 and make the lint CI pass. - Broaden chat-template detection to `arguments|items` and `parameters.properties|items` so aliased variable names (e.g. tool_call.function.arguments) are also normalized - Add tests for tool_call / tool parameter template aliases - Drop the duplicate BuildGradioEmbeddingInterfaceRequest import fallback that triggered mypy no-redef - Annotate template_for_normalization as Optional[str] (mypy) - Apply black formatting Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

ace-xc · 2026-06-18T12:57:39Z

#5049 seemed have solved this issue, can you confirm that?

ok, I'll continue checking.

Address review feedback on xorbitsai#5051 and make the lint CI pass. - Broaden chat-template detection to `arguments|items` and `parameters.properties|items` so aliased variable names (e.g. tool_call.function.arguments) are also normalized - Add tests for tool_call / tool parameter template aliases - Drop the duplicate BuildGradioEmbeddingInterfaceRequest import fallback that triggered mypy no-redef - Annotate template_for_normalization as Optional[str] (mypy) - Apply black formatting

qinxuye · 2026-06-19T04:38:34Z

#5049 seemed have solved this issue, can you confirm that?

ok, I'll continue checking.

Have you had any conclusion?

…bitsai#5037) Builds on xorbitsai#5049, which folds the Anthropic system prompt. Claude Code >= 2.1.154 also sends tool history as Anthropic content blocks and tool definitions carrying `input_schema`. These reach the Qwen/Qwen3 chat template unconverted and crash it with `TypeError: Can only get item pairs from a mapping` when a tool call's `arguments` is a JSON string but the template iterates `tool_call.arguments | items`. - Convert Anthropic content blocks into OpenAI-style messages: tool_use -> assistant tool_calls, tool_result -> tool role, text/image flattened (system folding stays in _normalize_anthropic_messages from xorbitsai#5049) - Convert Anthropic tools / tool_choice into OpenAI-compatible shapes - Normalize JSON-string tool-call arguments and tool parameters into mappings only for templates that iterate them with |items / .items(), leaving protocol-facing OpenAI responses unchanged - Accept "tool" as a valid final message role - Add tests for the conversion helpers and the template normalization

ace-xc · 2026-06-25T04:33:18Z

Builds on xorbitsai#5049, which folds the Anthropic system prompt. Claude Code

= 2.1.154 also sends tool history as Anthropic content blocks and tool
definitions carrying input_schema. These reach the Qwen/Qwen3 chat
template unconverted and crash it with
TypeError: Can only get item pairs from a mapping when a tool call's
arguments is a JSON string but the template iterates
tool_call.arguments | items.

Convert Anthropic content blocks into OpenAI-style messages: tool_use
-> assistant tool_calls, tool_result -> tool role, text/image flattened
(system folding stays in _normalize_anthropic_messages from xorbitsai#5049)

Convert Anthropic tools / tool_choice into OpenAI-compatible shapes

Normalize JSON-string tool-call arguments and tool parameters into
mappings only for templates that iterate them with |items / .items(),
leaving protocol-facing OpenAI responses unchanged

Accept "tool" as a valid final message role

Add tests for the conversion helpers and the template normalization

Yes. I rechecked this against v2.11.0 and the latest main.
My conclusion is that v2.11.0 already includes the Anthropic system prompt handling fix, but it does not cover Claude Code's tool_use/tool_result flow, Anthropic input_schema/tool_choice conversion, or the Qwen chat-template mapping issue. I have rebased this PR onto the latest main and made it compatible with the newer tool-call argument normalization changes from #5060.
The PR has been updated. I also added/kept tests for the Claude Code Anthropic tool-flow conversion and Qwen chat-template normalization.

qinxuye · 2026-06-26T16:40:02Z

Builds on xorbitsai#5049, which folds the Anthropic system prompt. Claude Code

= 2.1.154 also sends tool history as Anthropic content blocks and tool
definitions carrying input_schema. These reach the Qwen/Qwen3 chat
template unconverted and crash it with
TypeError: Can only get item pairs from a mapping when a tool call's
arguments is a JSON string but the template iterates
tool_call.arguments | items.

Convert Anthropic content blocks into OpenAI-style messages: tool_use
-> assistant tool_calls, tool_result -> tool role, text/image flattened
(system folding stays in _normalize_anthropic_messages from xorbitsai#5049)

Convert Anthropic tools / tool_choice into OpenAI-compatible shapes

Normalize JSON-string tool-call arguments and tool parameters into
mappings only for templates that iterate them with |items / .items(),
leaving protocol-facing OpenAI responses unchanged

Accept "tool" as a valid final message role

Add tests for the conversion helpers and the template normalization

Yes. I rechecked this against v2.11.0 and the latest main. My conclusion is that v2.11.0 already includes the Anthropic system prompt handling fix, but it does not cover Claude Code's tool_use/tool_result flow, Anthropic input_schema/tool_choice conversion, or the Qwen chat-template mapping issue. I have rebased this PR onto the latest main and made it compatible with the newer tool-call argument normalization changes from #5060. The PR has been updated. I also added/kept tests for the Claude Code Anthropic tool-flow conversion and Qwen chat-template normalization.

okay, I will review this PR asap.

qinxuye

I think one response-conversion edge case is still missing. In xinference/api/restful_api.py, _convert_openai_to_anthropic still does json.loads(arguments) unconditionally and only catches JSONDecodeError. If a backend/tool parser returns function.arguments as a dict, this raises TypeError and the Anthropic /v1/messages non-streaming response becomes 500. I reproduced it locally with arguments={"city":"Beijing"}.

Please handle mapping values directly before calling json.loads, and catch TypeError for non-string invalid values, e.g. keep dicts as input_data = arguments, otherwise parse strings and fall back to {} on invalid input.

XprobeBot added the bug Something isn't working label Jun 18, 2026

XprobeBot added this to the v2.x milestone Jun 18, 2026

gemini-code-assist Bot reviewed Jun 18, 2026

View reviewed changes

ace-xc changed the title ~~fix(api): 支持 Claude Code Anthropic 工具调用与 Qwen 模板渲染~~ fix(api): support Claude Code Anthropic tool flows with Qwen chat templates Jun 18, 2026

ace-xc force-pushed the fix/anthropic-qwen-tool-template branch 3 times, most recently from c2e5110 to 5a4694e Compare June 18, 2026 13:57

ace-xc force-pushed the fix/anthropic-qwen-tool-template branch from 5a4694e to 464e872 Compare June 25, 2026 04:30

qinxuye reviewed Jun 26, 2026

View reviewed changes

Uh oh!

Conversation

ace-xc commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot Jun 18, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Jun 18, 2026

Choose a reason for hiding this comment

Uh oh!

qinxuye commented Jun 18, 2026

Uh oh!

ace-xc commented Jun 18, 2026

Uh oh!

qinxuye commented Jun 18, 2026

Uh oh!

ace-xc commented Jun 18, 2026 via email

Uh oh!

qinxuye commented Jun 18, 2026

Uh oh!

qinxuye commented Jun 18, 2026

Uh oh!

ace-xc commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

qinxuye commented Jun 19, 2026

Uh oh!

ace-xc commented Jun 25, 2026

Uh oh!

qinxuye commented Jun 26, 2026

Uh oh!

qinxuye left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ace-xc commented Jun 18, 2026 •

edited

Loading

ace-xc commented Jun 18, 2026 •

edited

Loading