Skip to content

fix: emit runtime LLM request metadata#23

Merged
1aal merged 1 commit into
apecloud-basefrom
fix/runtime-provider-request-debug
Jun 11, 2026
Merged

fix: emit runtime LLM request metadata#23
1aal merged 1 commit into
apecloud-basefrom
fix/runtime-provider-request-debug

Conversation

@1aal

@1aal 1aal commented Jun 11, 2026

Copy link
Copy Markdown
Collaborator

背景

线上诊断会话中出现了模型未调用工具、直接以文本完成的情况。现有 Runtime 事件和日志只能看到最终 finish_reason=stoptool_calls=nulltool_turns=0,但无法确认当时发给 provider 的请求是否包含工具 schema、tool_choice 是什么,也无法直接看到 provider 返回的工具调用数量。

范围

本 PR 只在 Runtime worker 外层增加安全取证事件,不改变 Hermes Agent 的工具调用策略、Agent loop、Prompt/Skill 逻辑或 provider 行为。

主要改动

  • 在 Runtime worker 生命周期内注册临时 pre_api_request / post_api_request hook,run 结束后移除。
  • 新增 llm.request.metadata 事件,记录非敏感请求摘要:
    • skills_count / skill_names / skill_prompt_presence
    • system_prompt_chars / system_prompt_sha256
    • enabled_toolsets / disabled_toolsets
    • tools_count / tool_names / tool_choice
    • provider/model/base URL host、message/token 计数等
  • 新增 llm.response.metadata 事件,记录非敏感响应摘要:
    • finish_reason
    • assistant_tool_calls_count / assistant_tool_names
    • response model、usage、duration 等
  • 事件不包含 API key、完整 system prompt、完整 tool schema、用户输出、工具输出或 kubeconfig 内容。

验证

  • uv run pytest tests/runtime_manager/test_registry.py -k "llm_request_metadata or llm_response_metadata"
  • uv run pytest tests/runtime_manager/test_registry.py
  • uv run pytest tests/runtime_manager
  • uv run ruff check runtime_manager/worker_main.py tests/runtime_manager/test_registry.py
  • python3 -m py_compile runtime_manager/worker_main.py
  • git diff --check && git diff --check origin/apecloud-base...HEAD

风险与边界

  • 这是取证 PR,不是业务修复 PR。
  • 事件数量会随每次 LLM API 调用增加两条 metadata 事件,但每条只包含小型结构化字段。
  • 后续需要基于线上复现 run 的 metadata 判断:是 Runtime 没有把 tools 发给 provider,还是 tools 已发但 provider 在自动工具选择下返回了普通文本完成。

@1aal 1aal marked this pull request as ready for review June 11, 2026 08:57
@1aal 1aal merged commit d622029 into apecloud-base Jun 11, 2026
7 checks passed
@1aal 1aal deleted the fix/runtime-provider-request-debug branch June 11, 2026 08:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant