feat: SkillOpt — benchmark-driven prompt optimization via Harbor + YAML surface

[osilkin98]

Summary

Implements a SkillOpt-style training loop that optimizes agent prompt instructions via workflow YAML annotations, with benchmark execution through Harbor containers using factory workflow run. Includes SearchQA as the reference benchmark, validated against Microsoft's upstream SkillOpt results.

Architecture

The system has two loops:

Inner loop (benchmark execution via Harbor):

• factory workflow run searchqa executes inside a Harbor container
• The workflow reads the optimized skill from SEARCHQA_SKILL_B64 env var + appends fixed task instructions
• Harbor's verifier scores the answer against gold via EM, writes reward.json
• Results collected back to the trainer with per-item predicted/gold answers

Outer loop (SkillOpt optimizer):

• Reflects on rollout traces (minibatch failure + success analysis via LLM)
• Hierarchically merges edit proposals with failure priority
• Clips edits to learning rate budget (operation count)
• Applies edits to YAML prompt slots, re-renders SKILL.md via deterministic workflow engine
• Gates on full 200-item validation set (strict improvement required)
• Epoch-level slow update compares skill versions longitudinally

Key components

Workflow YAML surface (factory/skillopt/yaml_surface.py):

• Only task_prompt_* slots are mutable; graph structure is frozen
• yaml_to_workflow() converts edited YAML → valid Pydantic Workflow for factory workflow run
• workflow_to_yaml() for the reverse path
• 7 end-to-end round-trip tests verify exact value preservation

Harbor integration (factory/skillopt/adapters/searchqa.py):

• Calls run-harbor.sh searchqa --all with SEARCHQA_SKILL_B64 env var
• SEARCHQA_SPLIT env var switches between train/val Harbor task directories
• Verifier outputs (predicted/gold answers) enriched into RolloutResult for reflect phase
• FACTORY_GIT_REF ensures containers install from current branch

SearchQA benchmark (benchmarks/searchqa-harbor/):

• 400 train + 200 val Harbor tasks from lucadiliello/searchqa (Jeopardy QA)
• Each task: task.toml + instruction.md + Dockerfile + test.sh verifier
• Verifier extracts <answer> tags, compares to gold with SQuAD-style EM normalization, writes /logs/verifier/reward.json
• Single-node workflow: builder agent reads task, answers, writes /workspace/answer.txt

Slow update (factory/skillopt/slow_update.py):

• Epoch-level longitudinal comparison (previous vs current skill on same samples)
• Categorizes items as improved/regressed/persistent_fail/stable_success
• Generates strategic guidance in a protected region that step-level edits cannot touch

Mapping to InnerLoop/CycleAnalyzer proposal (PR #1051)

Our system maps to PR #1051's abstractions:

PR #1051 concept	Our implementation
InnerLoop.step()	adapter.rollout() → run-harbor.sh → factory workflow run in container
CycleRecord.score_end	Aggregate EM from Harbor result JSON
Evaluator.parse()	Read Harbor reward.json + verifier test-stdout.txt
directives	SEARCHQA_SKILL_B64 env var (edited skill → container)
CycleAnalyzer	_collect_results() + _extract_verifier_outputs() from Harbor jobs dir

Three extensions needed in PR #1051 for this use case:

1. Configurable execution backend — InnerLoop needs to support run-harbor.sh alongside factory ceo
2. Per-item results in CycleRecord — outer optimizer needs individual question results for reflection, not just aggregate scores
3. Env var directives — outer loop passes edited skill via env var, not file-based messages

Training results

SearchQA (Haiku student, Opus optimizer, direct claude -p baseline):

	SkillOpt Upstream	Factory SkillOpt
Baseline (no skill)	70.7%	76.6%
Best test score (1400 items)	78.9%	84.4%
Improvement	+8.2 pp	+7.9 pp
Steps to best	45 / 50	2 / 40

Harbor-based training (1 epoch, in progress): validates the full loop through factory workflow run inside containers.

Module structure

File	Role
types.py	Edit, Patch, RolloutResult, RawPatch, GateResult, SlowUpdateResult
yaml_surface.py	YAML loading, slot extraction, structural validation, round-trip conversion
adapter.py	Abstract EnvAdapter base class
adapters/swebench.py	Harbor SWE-bench adapter
adapters/searchqa.py	Harbor SearchQA adapter (train/val split, verifier output enrichment)
reflect.py	Minibatch reflection with parallel failure + success analysts
aggregate.py	Hierarchical tree-structured patch merge
clip.py	Index-based LLM edit ranking
skill.py	Edit application with protected regions
gate.py	Three-outcome validation gate
slow_update.py	Epoch-level longitudinal comparison + strategic guidance
trainer.py	DL-style training loop with epochs, LR, rejected-edit buffer, slow update
prompts/	7 prompt templates (analyst error/success, merge x3, ranking, slow update)

SearchQA Harbor benchmark

File	Role
benchmarks/searchqa-harbor/{train,val}/	400 + 200 Harbor task directories
benchmarks/searchqa/generate_harbor_tasks.py	Generates tasks from JSONL dataset
benchmarks/searchqa/evaluator.py	SQuAD-style EM/F1/sub_em scoring
benchmarks/factory_harbor_agent.py	SearchQAFactoryCeo agent class
factory/workflow/contributed/searchqa/workflow.py	Single-node workflow with env var skill injection

Test plan

• 150 unit tests (types, skill ops, gate, clip, adapter, trainer, yaml surface, slow update, round-trip)
• ruff check clean
• mypy --strict clean
• SearchQA single task resolves through Harbor with optimized skill
• SearchQA single task fails with blank seed, resolves with trained skill (U2 example)
• YAML round-trip: 7 end-to-end tests including real swebench workflow
• yaml_to_workflow() produces valid Workflow accepted by WorkflowExecutor
• Full training loop (direct baseline): 0.725 → 0.83 val EM, 0.766 → 0.844 test EM
• Full training loop through Harbor (1 epoch, in progress)

🤖 Generated with Claude Code

[github-actions]

Sentrux Quality Report

Absolute

Scanning ....
[scan] git ls-files: 433 total, 421 kept, 12 dropped (ext:12, meta:0, big:0)
[build_project_map] 421 files, 73 unique dirs, 67 cache misses, 3.4ms
[resolve] 671 resolved, 1013 unresolved (of 1684 total specs)
[resolve_imports] project_map 3.5ms, suffix_idx 0.8ms, suffix_resolve 12.6ms, total 16.8ms
[build_graphs] 421 files | maps 1.5ms, imports 17.0ms, calls+inherit 3.3ms, total 21.8ms | 670 import, 5023 call, 8 inherit edges
sentrux check — 3 rules checked

Quality: 4564

✗ [Error] max_cc: 5 function(s) exceed max cyclomatic complexity of 30
    factory/cli/ceo.py:cmd_ceo (cc=99)
    factory/study.py:study_project_local (cc=42)
    factory/cli/_wizard.py:_welcome_wizard (cc=39)
    factory/cli/ceo.py:cmd_run (cc=37)
    factory/workflow/validation.py:validate_workflow (cc=31)

✗ 1 violation(s) found

Diff (vs base branch)

Scanning ....
[scan] git ls-files: 433 total, 421 kept, 12 dropped (ext:12, meta:0, big:0)
[build_project_map] 421 files, 73 unique dirs, 67 cache misses, 3.6ms
[resolve] 671 resolved, 1013 unresolved (of 1684 total specs)
[resolve_imports] project_map 3.7ms, suffix_idx 0.8ms, suffix_resolve 13.0ms, total 17.5ms
[build_graphs] 421 files | maps 1.6ms, imports 17.6ms, calls+inherit 3.5ms, total 22.6ms | 670 import, 5023 call, 8 inherit edges
sentrux gate — structural regression check

Quality:      4552 -> 4564
Coupling:     0.80 → 0.79
Cycles:       5 → 5
God files:    2 → 3

Distance from Main Sequence: 0.39

✗ DEGRADED
  ✗ God files increased: 2 → 3
  ✗ Complex functions increased: 32 → 35

[codecov]

Codecov Report

❌ Patch coverage is 66.22296% with 406 lines in your changes missing coverage. Please review.
✅ Project coverage is 87.36%. Comparing base (2aa5770) to head (b9f5f00).
⚠️ Report is 17 commits behind head on main.

Files with missing lines	Patch %	Lines
factory/skillopt/adapters/featurebench.py	22.61%	130 Missing ⚠️
factory/skillopt/trainer.py	57.26%	100 Missing ⚠️
factory/skillopt/yaml_surface.py	50.35%	69 Missing ⚠️
factory/skillopt/adapters/swebench.py	73.80%	44 Missing ⚠️
factory/skillopt/reflect.py	80.98%	27 Missing ⚠️
factory/skillopt/aggregate.py	83.52%	14 Missing ⚠️
factory/skillopt/clip.py	75.00%	12 Missing ⚠️
factory/skillopt/__main__.py	78.04%	9 Missing ⚠️
factory/skillopt/skill.py	98.48%	1 Missing ⚠️

❌ Your patch check has failed because the patch coverage (66.22%) is below the target coverage (80.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1013      +/-   ##
==========================================
- Coverage   88.98%   87.36%   -1.62%     
==========================================
  Files         124      139      +15     
  Lines       14575    15840    +1265     
==========================================
+ Hits        12969    13839     +870     
- Misses       1606     2001     +395     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.