Skip to content

fix(tokenless): fix rtk pytest 'No tests collected' regression#881

Open
shiloong wants to merge 1 commit into
alibaba:mainfrom
shiloong:rtk
Open

fix(tokenless): fix rtk pytest 'No tests collected' regression#881
shiloong wants to merge 1 commit into
alibaba:mainfrom
shiloong:rtk

Conversation

@shiloong

Copy link
Copy Markdown
Collaborator

Description

Root cause: pytest in CI/Docker outputs ANSI escape codes (\x1b[1m...\x1b[0m) wrapping === headers. strip_ansi was not called before starts_with("===") checks, causing ALL state transitions to fail.

Changes:

  • pytest_cmd.rs: call strip_ansi() before trim() in the filter loop
  • pytest_cmd.rs: add errors field + ERRORS section + error keyword parsing
  • runner.rs: stderr fallback replaces 'No tests collected' with stderr content
  • runner.rs: separate display/tracking to preserve stats accuracy

Related Issue

fixes # bugs found in benchmarking

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Refactoring (no functional change)
  • Performance improvement
  • CI/CD or build changes

Scope

  • cosh (copilot-shell)
  • sec-core (agent-sec-core)
  • skill (os-skills)
  • sight (agentsight)
  • tokenless (tokenless)
  • ckpt (ws-ckpt)
  • memory (agent-memory)
  • anolisa (anolisa-cli)
  • Multiple / Project-wide

Checklist

  • I have read the Contributing Guide
  • My code follows the project's code style
  • I have added tests that prove my fix is effective or that my feature works
  • I have updated the documentation accordingly
  • For cosh: Lint passes, type check passes, and tests pass
  • For sec-core (Rust): cargo clippy -- -D warnings and cargo fmt --check pass
  • For sec-core (Python): Ruff format and pytest pass
  • For skill: Skill directory structure is valid and shell scripts pass syntax check
  • For sight: cargo clippy -- -D warnings and cargo fmt --check pass
  • For tokenless: cargo clippy -- -D warnings and cargo fmt --check pass
  • For memory (Linux only): cargo clippy --all-targets -- -D warnings, cargo fmt --check, and cargo test pass
  • For anolisa: cargo clippy --all-targets --locked -- -D warnings, cargo fmt --all --check, and cargo test --locked pass
  • Lock files are up to date (package-lock.json / Cargo.lock)

Testing

Additional Notes

Root cause: pytest in CI/Docker outputs ANSI escape codes
(\x1b[1m...\x1b[0m) wrapping === headers. strip_ansi was not called
before starts_with("===") checks, causing ALL state transitions to fail.

Changes:
- pytest_cmd.rs: call strip_ansi() before trim() in the filter loop
- pytest_cmd.rs: add errors field + ERRORS section + error keyword parsing
- runner.rs: stderr fallback replaces 'No tests collected' with stderr content
- runner.rs: separate display/tracking to preserve stats accuracy

Signed-off-by: Shile Zhang <shile.zhang@linux.alibaba.com>
@shiloong shiloong requested a review from Forrest-ly as a code owner June 12, 2026 10:17
@github-actions github-actions Bot added the component:tokenless src/tokenless/ label Jun 12, 2026
@shiloong shiloong requested a review from samchu-zsl June 12, 2026 10:18

@Forrest-ly Forrest-ly left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review — fix(tokenless): fix rtk pytest 'No tests collected' regression

Summary

此 PR 修复了 rtk pytest 输出解析器在 CI/Docker 环境中的回归问题。核心根因是 pytest 在带色彩的终端中输出 ANSI 转义码包裹 === 头部,导致状态机所有转换失败,最终误报 "No tests collected"。

变更涉及两部分:

  1. pytest_cmd.rs — 在匹配前调用 strip_ansi();新增 errors 字段和 ERRORS section 解析
  2. runner.rs — stderr 回退逻辑重构,分离 display/tracking 以保持统计准确性

正面评价

  • 根因定位准确:ANSI 转义码问题是典型的 CI 环境陷阱,strip_ansitrim() 前调用是正确且最小化的修复
  • 测试覆盖充分:8 个新增测试用例覆盖了 errors section header、mixed errors/failures、empty stdout、raw traceback 等边界场景
  • display/tracking 分离设计合理:避免 stderr 内容污染 token savings 统计

问题与建议

1. [Medium] filtered.clone() 无条件执行(runner.rs)

let display = if opts.filter_stdout_only && ... {
    // stderr fallback
    preview
} else {
    filtered.clone()  // ← 所有命令都付出 clone 代价
};

filtered 后续仅在 timer.track(filtered, ...) 中被消费(moved)。对于非 pytest 命令(filter_stdout_only=false),这里产生了一次不必要的 String 克隆。建议改为:

let (display, tracked) = if opts.filter_stdout_only && ... {
    (preview, filtered.clone())
} else {
    (filtered.clone(), filtered)
};

或者使用 Cow<str> 避免 else 分支的分配。考虑到实际命令输出可能较大(几十 KB),值得优化。

2. [Low] pytest_cmd.rs 与 runner.rs 之间的隐式耦合

pytest_cmd.rs 在 stdout 为空时返回:

"Pytest: No tests collected (empty stdout, see stderr below)"

runner.rs 通过 filtered.contains("No tests collected") 来触发 stderr 回退。这个 contains 检查恰好能匹配带后缀的版本,但两个文件之间靠子串匹配隐式耦合。如果未来 pytest_cmd.rs 修改措辞(比如改为 "No tests found"),runner.rs 会静默失效。

建议:在 filter_pytest_output 返回值中增加一个结构化标记(如 enum 或 bool 字段),而非依赖字符串子串判断。

3. [Low] parse_summary_line 中 error 关键词匹配

} else if *word == "error" || *word == "errors" {
    counts.errors = n;
}

这里用精确匹配 == 是正确且安全的(与其他字段用 .contains() 不同),因为 pytest summary 中 error 可能与 "collection error" 等组合出现。当前实现能正确处理 "1 error" 和 "2 errors",但无法处理 "1 collection error" 这种 pytest 偶尔输出的格式。如果有此类 case,建议回退到 .contains("error")

4. [Nit] patch 文件的 index hash

旧 patch:index 2febb2e..2816158
新 patch:index 2febb2e..773cb55

目标 hash 变了说明 patch 应用后的结果不同,这是预期的。但建议在 PR 描述中说明这个 patch 是基于 rtk 的哪个 commit 生成的,以便后续维护者验证 patch 是否仍然适用。


总结

这是一个针对真实 CI 环境问题的有效修复,根因分析清晰,测试覆盖到位。主要关注点是 runner.rs 中的无条件 clone 和跨文件隐式耦合。建议处理 #1(性能)和 #2(可维护性)后合入。

Verdict: Approve with minor suggestions — 不阻塞合入,但建议 follow-up 优化。

@Forrest-ly

Copy link
Copy Markdown
Collaborator

Code Review — PR #881

结论:✅ Approve

优点

  • 根因定位精准:ANSI 转义码导致 starts_with("===") 状态机转移全部失败
  • 修复层次清晰:① strip_ansi() 从源头消除 ② stderr 回退作为防御网 ③ display/tracking 分离确保统计不被污染
  • 测试覆盖充分(~200 行新测试)
  • max_lines=40 限制合理

建议(非阻塞)

  1. 确认 strip_ansi 在 rtk 的 core::utils 模块中已导出 — 若不存在则补丁编译失败
  2. 建议在 commit message 中补充 errors 计数增强的说明(新增 ERRORS section 解析),当前描述仅提及 ANSI 和 stderr 回退

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

component:tokenless src/tokenless/

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants