Skip to content

fix(skill_eval): add trigger-only mode#29

Open
mxl wants to merge 1 commit into
antongulin:mainfrom
mxl:fix/skill-eval-trigger-only
Open

fix(skill_eval): add trigger-only mode#29
mxl wants to merge 1 commit into
antongulin:mainfrom
mxl:fix/skill-eval-trigger-only

Conversation

@mxl

@mxl mxl commented Jun 23, 2026

Copy link
Copy Markdown

Problem

skill_eval runs every query through opencode run even after the synthetic skill has already triggered. This wastes time on downstream workflow output that is irrelevant to trigger detection, and can cause false negatives when the model later hits rate limits, timeouts, or unrelated errors.

Changes

  1. plugin/lib/process.ts: Allow onStdoutChunk callback to return boolean. When true, kill the child process early (with graceful + SIGKILL fallback).

  2. plugin/lib/run-eval.ts:

    • Add triggerOnly option (default true) to RunEvalOptions and runSingleQuery.
    • When triggerOnly=true and the synthetic skill name appears in stdout, immediately return true without waiting for the full process to finish.
  3. plugin/lib/run-loop.ts: Thread triggerOnly through RunLoopOptions into runEval.

  4. plugin/skill-creator.ts: Expose triggerOnly boolean schema arg on both skill_eval and skill_optimize_loop tools.

  5. Tests: Add early-stop test in process.test.ts; add triggerOnly: true to run-loop.test.ts fixture.

Testing

  • npm run build passes
  • npm run test:ts: 44 pass, 0 fail
  • Live eval on video-summary with GPT-5.4: 3/3 passed, triggerOnly stopped runs immediately after trigger

@mxl mxl force-pushed the fix/skill-eval-trigger-only branch from 2935dd7 to 9e9d948 Compare June 23, 2026 08:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant