Skip to content

fix: switch copaw-worker runtime entrypoint to qwenpaw#950

Open
LUOSENGWA wants to merge 13 commits into
agentscope-ai:mainfrom
LUOSENGWA:fix/copaw-to-qwenpaw-migration
Open

fix: switch copaw-worker runtime entrypoint to qwenpaw#950
LUOSENGWA wants to merge 13 commits into
agentscope-ai:mainfrom
LUOSENGWA:fix/copaw-to-qwenpaw-migration

Conversation

@LUOSENGWA

Copy link
Copy Markdown

Summary

Switch copaw-worker from copaw.app to qwenpaw.app as the runtime
entrypoint. qwenpaw is already present in the image as a dependency
of copaw; this change makes Workers use it directly, enabling
spawn_subagent and the full qwenpaw multi-agent feature set.

Changes (6 files, +31/-12)

  • worker.py: import path + uvicorn target → qwenpaw.app
  • bridge.py: inject agents.profiles into config.json (qwenpaw requirement)
  • channel.py: channel base import → qwenpaw.app
  • Dockerfile: overlay matrix channel → qwenpaw path; +pip install qwenpaw
  • health.py: docstring sync
  • test_worker_health.py: mock path sync

Validation

  • 18/18 Workers migrated in production, all Up, zero errors
  • spawn_subagent E2E: subagent creation → task execution → result returned
  • Matrix messaging (Element) + QQ channel both verified
  • All Workers running >24h with no regression

Relation to TeamHarness

This PR is a prerequisite for TeamHarness Phase 4 (QwenPaw Worker
integration). The adapter code in plugins/teamharness/adapters/qwenpaw/
already imports from qwenpaw.* modules and calls qwenpaw plugin install,
but those can't work until Workers actually start qwenpaw.app.

The agents.profiles injection in bridge.py feeds directly into
TeamHarness's _iter_qwenpaw_agents().

What this does NOT change

  • Runtime name "copaw" in CRDs (stable API, unchanged)
  • Controller image selection logic
  • HICLAW_COPAW_WORKER_IMAGE default value
  • No Go controller code touched

@github-actions

github-actions Bot commented Jun 21, 2026

Copy link
Copy Markdown
Contributor

📊 CI Metrics Report

Summary

Metric Current Baseline Change
LLM Calls 162 81 +81 ↑ +100.0%
Input Tokens 5706472 2803871 +2902601 ↑ +103.5%
Output Tokens 40410 16791 +23619 ↑ +140.7%
Total Tokens 5746882 2820662 +2926220 ↑ +103.7%

By Role

Role Metric Current Baseline Change
🧠 Manager LLM Calls 118 68 +50 ↑ +73.5%
Input Tokens 4596272 2502214 +2094058 ↑ +83.7%
Output Tokens 25147 13725 +11422 ↑ +83.2%
Total Tokens 4621419 2515939 +2105480 ↑ +83.7%
🔧 Workers LLM Calls 44 13 +31 ↑ +238.5%
Input Tokens 1110200 301657 +808543 ↑ +268.0%
Output Tokens 15263 3066 +12197 ↑ +397.8%
Total Tokens 1125463 304723 +820740 ↑ +269.3%

Per-Test Breakdown

Test Mgr Calls Wkr Calls Δ Calls Mgr In Wkr In Mgr Out Wkr Out Δ Tokens Trend
02-create-worker 4 0 -8 ↓ -66.7% 103190 0 465 0 -254967 ↓ -71.1% ✅ improved
03-assign-task 17 6 +8 ↑ +53.3% 478745 138616 2961 1186 +147852 ↑ +31.2% ⚠️ regressed
04-human-intervene 16 0 +3 ↑ +23.1% 435834 0 2535 0 +5371 ↑ +1.2% ⚠️ regressed
05-heartbeat 10 0 +3 ↑ +42.9% 328921 0 2622 0 +56291 ↑ +20.5% ⚠️ regressed
06-multi-worker 71 38 +75 ↑ +220.6% 3249582 971584 16564 14077 +2971673 ↑ +232.1% ⚠️ regressed

Trends

1 test(s) improved (fewer LLM calls)
⚠️ 4 test(s) regressed (more LLM calls)


Generated by HiClaw CI on 2026-07-04 05:09:12 UTC


📦 Download debug logs & test artifacts

@github-actions

github-actions Bot commented Jun 21, 2026

Copy link
Copy Markdown
Contributor

❌ Integration Tests Failed (controller-cr-2 / mgr=copaw / wk=copaw)

Commit: 06ea624
Workflow run: #1430

Test Results
No test output captured.
Debug Log (tail)
No debug logs available.

📦 Download full debug logs & test artifacts

@github-actions

github-actions Bot commented Jun 21, 2026

Copy link
Copy Markdown
Contributor

❌ Integration Tests Failed (llm-interaction-2 / mgr=copaw / wk=copaw)

Commit: 865708a
Workflow run: #1424

Test Results
No test output captured.
Debug Log (tail)
No debug logs available.

📦 Download full debug logs & test artifacts

@github-actions

github-actions Bot commented Jun 21, 2026

Copy link
Copy Markdown
Contributor

❌ Integration Tests Failed (llm-interaction-2 / mgr=copaw / wk=hermes)

Commit: bc5731e
Workflow run: #1423

Test Results
No test output captured.
Debug Log (tail)
No debug logs available.

📦 Download full debug logs & test artifacts

@github-actions

github-actions Bot commented Jun 21, 2026

Copy link
Copy Markdown
Contributor

❌ Integration Tests Failed (llm-interaction / mgr=copaw / wk=copaw)

Commit: 06ea624
Workflow run: #1430

Test Results
No test output captured.
Debug Log (tail)
No debug logs available.

📦 Download full debug logs & test artifacts

@github-actions

github-actions Bot commented Jun 21, 2026

Copy link
Copy Markdown
Contributor

❌ Integration Tests Failed (llm-interaction / mgr=copaw / wk=hermes)

Commit: bc5731e
Workflow run: #1423

Test Results
No test output captured.
Debug Log (tail)
No debug logs available.

📦 Download full debug logs & test artifacts

@flystar32 flystar32 requested a review from maplefeng-a June 22, 2026 02:17
@flystar32

flystar32 commented Jun 22, 2026

Copy link
Copy Markdown
Collaborator

这个迁移方向是合理的,但当前实现还有几个合并前需要补齐的运行时接线问题。

  1. Worker 运行目录目前只接到了 copaw,没有明确接到 qwenpaw

copaw/scripts/copaw-worker-entrypoint.sh 只导出了 COPAW_WORKING_DIRcopaw/src/copaw_worker/bridge.py 也只 patch 了 copaw.constant / copaw.providers.store / copaw.envs.store。切到 qwenpaw.app._app:app 后,如果 qwenpaw 使用自己的 working dir 环境变量或 qwenpaw.* 模块状态,就可能读写默认目录,而不是 Worker 的 .copaw runtime dir。

建议二选一:

  • 同步导出 qwenpaw 对应的 working-dir 环境变量,并 patch qwenpaw.* 路径模块;
  • 或补充测试/说明,证明 qwenpaw.app 完全复用现有 copaw.* 路径状态。
  1. Matrix config overlay 可能还在写旧包路径

Dockerfile 已经把 Matrix channel 覆盖到 $SITE/qwenpaw/app/channels/,但 config.py 仍然复制到 $SITE/copaw/config/config.py。如果 qwenpaw.app 加载的是 qwenpaw.config.config,channel 和 config schema 会在运行时不一致。需要确认 import 路径,必要时也把 config overlay 写到 qwenpaw 包下。

  1. qwenpaw CLI 没有暴露到 PATH

PR 描述里说 TeamHarness adapter 会调用 qwenpaw plugin install,但镜像里目前只把 /opt/venv/copaw/bin/copaw symlink 到 /usr/local/bin/copaw。如果 qwenpaw console script 安装在 venv 里,运行时仍然可能找不到命令。建议暴露 venv bin 目录,或补一个 qwenpaw symlink。

  1. agents.profiles 注入逻辑太浅

现在用的是 cfg.setdefault("agents", {...}),只在 agents 整个不存在时生效。如果已有 config.json 里存在 agents 但缺少 profiles,qwenpaw 仍然拿不到必要的 profile 引用。建议改成嵌套 merge,在保留用户字段的同时确保这些字段存在:

  • agents.active_agent
  • agents.profiles.default.id
  • agents.profiles.default.workspace_dir
  1. 需要补 changelog

这个 PR 修改了 copaw/ 下会进入镜像的内容,按仓库规则需要在 changelog/current.md 记录一条变更。

另外,CI 评论里有旧提交上的集成测试失败,以及 06-multi-worker 的 token/call 明显上涨。建议在合并前重新跑最新 head,并明确给出 mgr=copaw / wk=copaw 路径的结果,因为这是本 PR 直接影响的路径。

最后,@maplefeng-a 也在做相关的工作,可以看看是否存在逻辑上不一致的地方。


This migration direction is reasonable, but the current implementation still has several runtime wiring issues that need to be patched before merging.

  1. The Worker running directory currently only receives copaw, but does not explicitly receive qwenpaw

copaw/scripts/copaw-worker-entrypoint.sh only exports COPAW_WORKING_DIR, and copaw/src/copaw_worker/bridge.py only patches copaw.constant / copaw.providers.store / copaw.envs.store. After switching to qwenpaw.app._app:app, if qwenpaw uses its own working dir environment variable or qwenpaw.* module state, it may read and write the default directory instead of the Worker's .copaw runtime dir.

It is recommended to choose one of the two:

  • Synchronously export the working-dir environment variable corresponding to qwenpaw, and patch the qwenpaw.* path module;
  • Or additional tests/explanations to demonstrate that qwenpaw.app fully reuses existing copaw.* path state.
  1. Matrix config overlay may still be writing the old package path

The Dockerfile has overwritten the Matrix channel to $SITE/qwenpaw/app/channels/, but config.py is still copied to $SITE/copaw/config/config.py. If qwenpaw.app loads qwenpaw.config.config, the channel and config schema will be inconsistent at runtime. You need to confirm the import path and write the config overlay to the qwenpaw package if necessary.

  1. qwenpaw CLI is not exposed to PATH

The PR description says that the TeamHarness adapter will call qwenpaw plugin install, but the image currently only symlinks /opt/venv/copaw/bin/copaw to /usr/local/bin/copaw. If the qwenpaw console script is installed in the venv, the command may still not be found when running. It is recommended to expose the venv bin directory or add a qwenpaw symlink.

  1. The injection logic of agents.profiles is too shallow

Currently, cfg.setdefault("agents", {...}) is used, which only takes effect when agents does not exist. If agents already exists in config.json but profiles is missing, qwenpaw still cannot get the necessary profile reference. It is recommended to change to a nested merge to ensure that these fields exist while retaining user fields:

  • agents.active_agent
  • agents.profiles.default.id
  • agents.profiles.default.workspace_dir
  1. Need to add changelog

This PR modifies the content that will enter the image under copaw/. According to the warehouse rules, a change needs to be recorded in changelog/current.md.

In addition, there are integration test failures on old commits in the CI comments, and the token/call of 06-multi-worker has increased significantly. It is recommended to re-run the latest head before merging, and clearly give the results of the mgr=copaw / wk=copaw path, because this is the path directly affected by this PR.

Finally, @mapplefeng-a is also doing related work to see if there are any logical inconsistencies.

LUOSENGWA pushed a commit to LUOSENGWA/HiClaw that referenced this pull request Jun 22, 2026
1. Working dir: export QWENPAW_WORKING_DIR alongside COPAW_WORKING_DIR
   so qwenpaw.constant resolves the runtime dir without relying on
   legacy COPAW_ fallback alone.

2. Dockerfile: add qwenpaw CLI symlink to PATH for TeamHarness
   adapter compatibility.

3. bridge.py: change agents.profiles injection from shallow
   setdefault() to nested merge. Previously, if config.json already
   had an 'agents' key, profiles/active_agent/workspace_dir were
   not injected. Now each sub-key is setdefault'd individually.

4. chanhelog: add PR agentscope-ai#950 entries for copaw/ image-affecting changes.

Note on Matrix config overlay: the maintainer asked about potential
inconsistency between the config.py overlay (copied to copaw/config/)
and qwenpaw.config.config. Analysis confirms no inconsistency:
- Matrix channel.py (overlaid to qwenpaw/app/channels/matrix/) does
  not import from config.py at all — it only depends on BaseChannel
  and agentscope_runtime, both available via the dual fallback.
- The HiClaw config overlay (copaw/src/matrix/config.py) adds
  HiClaw-specific channel types to copaw.config.config and is only
  needed by copaw.app.  qwenpaw.app loads qwenpaw.config.config
  (its own module), and the channel layer is config-agnostic.

@maplefeng-a maplefeng-a left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the follow-up. I checked the latest head, and this PR still has conflicts with main (copaw/Dockerfile, copaw/scripts/copaw-worker-entrypoint.sh, copaw/src/copaw_worker/bridge.py, and copaw/src/copaw_worker/worker.py).

Please rebase or merge main and resolve those conflicts first. Once the branch is clean again, I can continue the code review on the updated diff.


我看了最新 head,目前这个 PR 仍然和 main 有冲突,涉及 copaw/Dockerfilecopaw/scripts/copaw-worker-entrypoint.shcopaw/src/copaw_worker/bridge.pycopaw/src/copaw_worker/worker.py

请先 rebase 或 merge main 并解决冲突;分支干净后我再继续基于最新 diff 做代码 review。

AgentTeamGuide added 7 commits July 3, 2026 12:53
- worker.py: change import and uvicorn target from copaw.app to qwenpaw.app
- bridge.py: inject agents.profiles section into config.json (required by qwenpaw)
- matrix/channel.py: update channel base import path to qwenpaw
- Dockerfile: overlay matrix channel to qwenpaw path; add pip install qwenpaw
- health.py: update docstring to reflect qwenpaw entrypoint
- test_worker_health.py: update mock paths for qwenpaw namespace

This enables spawn_subagent and other qwenpaw multi-agent features
for all Workers while keeping the 'copaw' runtime name in HiClaw CRDs.
The qwenpaw package is installed alongside copaw in the same venv;
only the Worker entrypoint switches to qwenpaw directly.

This also unblocks TeamHarness Phase 4 (QwenPaw Worker integration) —
the adapter code already targets qwenpaw imports and CLI, but workers
currently start copaw.app.
The import from qwenpaw.app.channels.base is correct for Workers (which
have qwenpaw installed), but the Manager copaw image (manager/Dockerfile.copaw)
also overlays this channel.py and only has copaw installed. Without qwenpaw,
the ImportError handler fell back to BaseChannel=object, making MatrixChannel
invisible to the copaw framework's channel loader (issubclass check fails).

Add a nested try/except that imports from copaw.app.channels.base as a
second fallback before degrading to object. This keeps the Manager copaw
image working without requiring qwenpaw to be installed there.
The Manager copaw image overlays copaw/src/matrix/channel.py which now
imports from qwenpaw.app.channels.base. While channel.py has a copaw
fallback, installing qwenpaw in the Manager image ensures the primary
import path works and aligns the Manager's Python environment with the
Worker image.
The old comment said 'only executed inside a qwenpaw environment' which
is no longer accurate since manager/Dockerfile.copaw also overlays this
file but may only have copaw installed.
The dual fallback in channel.py correctly uses qwenpaw's BaseChannel
when qwenpaw is available (Worker) and falls back to copaw's BaseChannel
when it's not (Manager). Installing qwenpaw in the Manager causes the
fallback to pick qwenpaw's BaseChannel, which then fails the copaw
framework's issubclass check during channel registration — since
copaw.app.channels.base.BaseChannel and qwenpaw.app.channels.base.BaseChannel
are different classes in different packages.

The Manager only needs copaw; the channel.py dual fallback handles the
rest.
… replies

When the copaw manager (without qwenpaw) uses the fallback path,
ContentType/MessageType/RunStatus were set to None even though
agentscope_runtime is available. This caused all Manager replies
to be empty (null), failing 5 CI tests.

Move the agentscope_runtime import into the copaw fallback try block,
only falling through to None when the import truly fails.
1. Working dir: export QWENPAW_WORKING_DIR alongside COPAW_WORKING_DIR
   so qwenpaw.constant resolves the runtime dir without relying on
   legacy COPAW_ fallback alone.

2. Dockerfile: add qwenpaw CLI symlink to PATH for TeamHarness
   adapter compatibility.

3. bridge.py: change agents.profiles injection from shallow
   setdefault() to nested merge. Previously, if config.json already
   had an 'agents' key, profiles/active_agent/workspace_dir were
   not injected. Now each sub-key is setdefault'd individually.

4. chanhelog: add PR agentscope-ai#950 entries for copaw/ image-affecting changes.

Note on Matrix config overlay: the maintainer asked about potential
inconsistency between the config.py overlay (copied to copaw/config/)
and qwenpaw.config.config. Analysis confirms no inconsistency:
- Matrix channel.py (overlaid to qwenpaw/app/channels/matrix/) does
  not import from config.py at all — it only depends on BaseChannel
  and agentscope_runtime, both available via the dual fallback.
- The HiClaw config overlay (copaw/src/matrix/config.py) adds
  HiClaw-specific channel types to copaw.config.config and is only
  needed by copaw.app.  qwenpaw.app loads qwenpaw.config.config
  (its own module), and the channel layer is config-agnostic.
@LUOSENGWA LUOSENGWA force-pushed the fix/copaw-to-qwenpaw-migration branch from 3014704 to 9911d12 Compare July 3, 2026 13:09
@github-actions

github-actions Bot commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

❌ Integration Tests Failed (controller-cr / mgr=copaw / wk=hermes)

Commit: 59782d5
Workflow run: #1444

Test Results
No test output captured.
Debug Log (tail)
No debug logs available.

📦 Download full debug logs & test artifacts

@github-actions

github-actions Bot commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

❌ Integration Tests Failed (controller-cr / mgr=copaw / wk=copaw)

Commit: 06ea624
Workflow run: #1430

Test Results
No test output captured.
Debug Log (tail)
No debug logs available.

📦 Download full debug logs & test artifacts

@github-actions

github-actions Bot commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

❌ Integration Tests Failed (controller-cr / mgr=openclaw / wk=openclaw)

Commit: 06ea624
Workflow run: #1430

Test Results
No test output captured.
Debug Log (tail)
No debug logs available.

📦 Download full debug logs & test artifacts

AgentTeamGuide and others added 6 commits July 3, 2026 14:32
qwenpaw depends on agentscope==1.0.20 which conflicts with copaw's
agentscope version when resolved in the same pip install. Installing
them in separate pip commands (copaw-worker first, then qwenpaw)
mirrors the production deployment approach that works on Node1.
The copaw worker adapter's build_worker_readiness() waited for a marker
file (/tmp/hiclaw-copaw-{name}-matrix-ready) that was written by copaw's
own Matrix channel after initial sync. After switching to qwenpaw, the
Matrix channel is qwenpaw/app/channels/matrix/channel.py which has no
knowledge of HICLAW_MATRIX_CHANNEL_READY_FILE — the marker was never
created, causing /worker/readyz to permanently return not_ready.

Fix: use qwenpaw's standard GET /config/channels/matrix/health API
instead of the legacy inter-process marker file side-channel. This
API is available for every channel in qwenpaw and is the canonical
way to check channel readiness.

Root cause of CI test failures: test-02, test-15, test-21.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants