Skip to content

fix: Read 工具支持非 UTF-8 编码文件(GBK/GB18030 等)#50

Draft
lloydzhou wants to merge 1 commit into
mainfrom
fix/read-non-utf8-encoding
Draft

fix: Read 工具支持非 UTF-8 编码文件(GBK/GB18030 等)#50
lloydzhou wants to merge 1 commit into
mainfrom
fix/read-non-utf8-encoding

Conversation

@lloydzhou

Copy link
Copy Markdown
Owner

读取 GBK 等非 UTF-8 编码文件时,原始字节直接进入 conversation 并提交
LLM API,导致请求报错。四个实现版本的 Read 工具均缺失编码处理。

修复方案(四版本统一):

  • 检测文件编码(file -bi),非 UTF-8 时用 iconv 转码(GB18030 兼容 GBK)
  • 转码失败 fallback 到 sanitize_utf8(保证 JSON 序列化不破坏)
  • bash: 新增 util_iconv_if_needed 函数,tool_read 调用之
  • go: toolRead 增加 utf8.Valid 检测 + iconvToUTF8 辅助函数
  • rust: read() 改用 fs::read 读字节 + iconv_to_utf8 方法
  • c: 新增 util_iconv_if_needed,tool_read 读取后调用

新增测试: test_read_gbk_encoding 验证 GBK 文件转码为 UTF-8

读取 GBK 等非 UTF-8 编码文件时,原始字节直接进入 conversation 并提交
LLM API,导致请求报错。四个实现版本的 Read 工具均缺失编码处理。

修复方案(四版本统一):
- 检测文件编码(file -bi),非 UTF-8 时用 iconv 转码(GB18030 兼容 GBK)
- 转码失败 fallback 到 sanitize_utf8(保证 JSON 序列化不破坏)
- bash: 新增 util_iconv_if_needed 函数,tool_read 调用之
- go: toolRead 增加 utf8.Valid 检测 + iconvToUTF8 辅助函数
- rust: read() 改用 fs::read 读字节 + iconv_to_utf8 方法
- c: 新增 util_iconv_if_needed,tool_read 读取后调用

新增测试: test_read_gbk_encoding 验证 GBK 文件转码为 UTF-8
@lloydzhou
lloydzhou marked this pull request as draft June 17, 2026 12:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant