fix issue#931: multimodal and peerMentions#945
Conversation
|
Bug 1 data flow (two-layer fix): openclaw.json models.providers[].models[].input: ["text","image"] ✅ Controller Related: Issue #931 (opened before fix, same root cause analysis) |
|
Full report in zh-cn |
Bug 1 (multimodal): Two-layer fix - Controller generator.go: infer supports_multimodal/supports_image from model spec Input field and write to agents.defaults in openclaw.json. - copaw_worker bridge.py: propagate input capability flags (supports_image/supports_video/supports_multimodal) when constructing provider models, so copaw react_agent can read them. Bug 2 (peerMentions): Five-layer data chain - types.go → generator.go → team_controller.go → member_reconcile.go → deployer.go. Injects peerMentions into channels.matrix of openclaw.json, enabling cross-room Matrix @mention routing for team workers. Note: The original Bug 3 (ResolveRuntime fallback) was a misdiagnosis. leaderWorkerSpec() already hardcodes Runtime="copaw" for team leaders. The global default of RuntimeOpenClaw is correct — reverting. Tests: all passing, go vet clean. Files: 8 (7 Go + 1 Python), +244/-2 lines.
LUOSENGWA
force-pushed
the
fix/multimodal-peermentions-runtime
branch
from
fc76b4e to
e14afd9
Compare
LUOSENGWA
changed the title
PR 附属说明 — 多模态 & peerMentions 修复
Bug 1：多模态静默（supports_multimodal 缺失）🔴问题：所有通道（QQ/Element/Matrix）的图片被框架静默删除。 根因：Config generation 有两个裂口：
修复：
测试：2 个新测试（vision model → true，text-only → 无此字段）。图片 E2E：Node1 Chrome 图标识别正确 ✅ Bug 2：peerMentions 不生成 🟡问题：Team 内 Leader↔Worker @mention 失效（3 次生产事故）。 根因：Config generation 五层数据链断裂——Controller 内部有 修复：五层数据链补齐 PR Side Notes — Multimodal & peerMentions fixes
Bug 1: Multimodal silence (supports_multimodal missing) 🔴Issue: Images of all channels (QQ/Element/Matrix) are silently deleted by the frame. Root cause: There are two gaps in Config generation:
Fix:
Tests: 2 new tests (vision model → true, text-only → no field). Image E2E: Node1 Chrome icon recognized correctly ✅ Bug 2: peerMentions not generated 🟡Problem: Leader↔Worker @mention in Team failed (3 production accidents). Root cause: The five-layer data link of Config generation is broken - there is Fix: Complete the five-layer data link |
The multimodal inference in GenerateOpenClawConfig() was applying supports_multimodal to ALL configs including Manager's openclaw.json. When Manager uses openclaw runtime, the gateway fails to start (unrecognized field), causing CI integration tests to 300s timeout. Guard the inference behind req.WorkerName != "manager" — Manager does not receive or process images, and the openclaw gateway does not understand agents.defaults.supports_multimodal.
📊 CI Metrics ReportSummary
By Role
Per-Test Breakdown
Trends✅ 3 test(s) improved (fewer LLM calls) Generated by HiClaw CI on 2026-06-18 16:10:51 UTC |
…kerName v1 guard (WorkerName != "manager") left openclaw Workers polluted — test-15 failed with 180s no-reply because supports_multimodal silently breaks openclaw message processing. v2 uses AgentRuntime == "copaw" so multimodal fields only reach agents that actually run copaw. - types.go: add AgentRuntime string to WorkerConfigRequest - deployer.go: pass req.Spec.Runtime for Worker and Manager paths - generator.go: guard on req.AgentRuntime == "copaw" - test_multimodal_test.go: add AgentRuntime to vision and text tests
LUOSENGWA
changed the title
LUOSENGWA
changed the title
LUOSENGWA
changed the title
1 participant
Summary
Fix four bugs in the Controller's openclaw.json generation pipeline and one in copaw_worker's bridge.py. These bugs collectively prevented image handling and cross-worker @mention routing in HiClaw agent teams.
Changes (10 files, ~100 lines)
Bug 1: Multimodal — two-layer fix
Controller generator.go: Infer supports_multimodal / supports_image from ModelSpec.Input and write to agents.defaults
copaw_worker bridge.py: Propagate input capability flags (supports_image, supports_video, supports_multimodal) when constructing provider models. Previously bridge.py discarded input entirely → supports_multimodal: null → bool(null)=False → all images stripped. Aligns with analysis in Issue #931.
Bug 2: peerMentions — five-layer data chain
types.go → generator.go → team_controller.go → member_reconcile.go → deployer.go
Injects channels.matrix.peerMentions into openclaw.json, enabling cross-room Matrix @mention delivery.
Peer calculation: Leader peers = all team workers; Worker peers = leader + all other workers.
Bug 3: Runtime fallback
ResolveRuntime() default corrected from RuntimeOpenClaw to RuntimeCopaw (leader path only). Worker CRD spec stored separately and unaffected. The RuntimeOpenClaw constant is preserved for explicit runtime: openclaw users.
Bug 4
Leader model propagation chain verified correct — no code change needed.
Testing
go test -v ./internal/agentconfig/ → 22/22 PASS
go test -v ./internal/backend/ -run TestDockerCreateRuntime → 6/6 PASS
go vet ./... → zero warnings
E2E verified on Node1 production with 20/20 Workers:
Chrome icon recognition (image → text) via Element Web DM ✅
Cross-worker @mention routing ✅