feat: add pre_eval and post_eval nodes to evolve workflow for InnerLoop integration - #1052
Draft
abhi1092 wants to merge 16 commits into
Draft
feat: add pre_eval and post_eval nodes to evolve workflow for InnerLoop integration#1052abhi1092 wants to merge 16 commits into
abhi1092 wants to merge 16 commits into
Conversation
…e effectiveness Adds W₁₂ optimize mode — a meta-mode that analyzes how well an existing factory mode performs, identifies weaknesses (high redirect rates, agent timeouts, low keep rates), generates a targeted change specification, and delegates implementation to create mode. Interactive-only with user approval gate before modifying workflows. Changes: - Add optimize_workflow() to factory/workflow/definitions.py (8 nodes, 9 edges) - Register in register_all() and __all__ - Add WORKFLOW_META entry in skill_export.py - Wire --mode optimize in CLI (_helpers.py, _main.py, ceo.py) - Add optimize to CycleState.mode Literal in models.py - Add 12 tests in tests/test_optimize_workflow.py Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
feat: add optimize workflow mode for factory mode effectiveness tuning
Signed-off-by: Abhishek Bhandwaldar <abhi1092@gmail.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…evaluation Implements W₁₄: Evolve Mode — a tight evaluate-mutate-select loop that optimizes a single scalar metric by evolving code within EVOLVE-BLOCK boundaries, evaluated externally via MCP server tools (get_benchmark_info, evaluate_solution). Key design decisions: - No deep-QA pipeline (single health_checker with MCP eval replaces the 3-specialist QA pipeline — tighter loops produce better results) - Elitist keep/revert: score must strictly improve - FEEC stuck detection: 3 consecutive reverts → re-research - Convergence: target score, max cycles, or diminishing returns Changes: - factory/workflow/definitions.py: evolve_workflow() + register_all() - factory/workflow/skill_export.py: WORKFLOW_META entry - factory/cli/_helpers.py: CEO_MODES list - factory/cli/ceo.py: validation, ceo_mode routing, _build_ceo_task - tests/test_evolve_workflow.py: 20 tests (5 classes) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The evolve workflow was added in 7a14aee but its generated SKILL.md and annotations were not committed, so the CEO cannot load the evolve mode playbook at runtime. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
feat: add evolve workflow mode and research queue baseline
Reads .factory/ artifacts (events.jsonl, results.tsv, experiment dirs) and produces structured CycleRecord objects that any outer-loop optimizer can consume. Mode-agnostic — works with evolve, improve, research, refine. Tiered data assembly: - Tier 1: events.jsonl for agent steps, costs, durations - Tier 2: results.tsv for experiment scores, verdicts, hypotheses - Tier 3: experiment dirs for eval artifact discovery - Tier 4: workflow definition for DAG node mapping Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
feat: add CycleAnalyzer for outer-loop optimizer observability
…ation InnerLoop wraps a factory mode + evaluator into a model.forward()-like interface that any outer-loop optimizer can call. CycleAnalyzer reads .factory/ artifacts and produces structured CycleRecord objects. Includes CirclePackingEvaluator as the first concrete evaluator implementation, tested against skydiscover's circle packing benchmark. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Evaluator now reads output artifacts that the inner loop already produced,
rather than calling the evaluator binary directly. CirclePackingEvaluator
parses {sum_radii, validity, eval_time, combined_score} JSON files.
CycleAnalyzer: execution tracing (agents, costs, verdicts)
Evaluator: score interpretation (evaluator-specific artifact parsing)
InnerLoop: composes both
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
feat: add InnerLoop and CycleAnalyzer for outer-loop optimizer integration
…op integration Add two FnNode stages to the evolve workflow DAG so that CycleAnalyzer can discover per-experiment artifacts and build score trajectories: - pre_eval: copies current_score.json → experiments/$EXP_ID/eval_before.json between begin and builder (closes Gap 4) - post_eval: emits eval.completed event to events.jsonl between health_checker and gate_eval (closes Gap 2) Also updates baseline node writes to include experiments/000/eval_before.json (Gap 3) and health_checker writes/prompt to produce structured eval_after.json in the experiment directory (Gap 1). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…prompts Replace domain-specific prompt hints with domain-agnostic instructions that direct agents to discover the benchmark domain at runtime from eval.json, making the evolve workflow work correctly across all benchmark types. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The SKILL.md was stale after the prompt_template changes in definitions.py (commit 2cabd28). Regenerated via `factory workflow export-skills` to reflect the domain-agnostic researcher and health_checker prompts. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Allow evolve mode to accept --focus to specify a benchmark name. The focus is injected as a ## Benchmark Target section in the CEO task, and the Baseline FnNode notes reference it for get_benchmark_info() calls. Changes: - cli/ceo.py: add 'evolve' to allowed focus modes in both cmd_ceo and cmd_run gates - cli/ceo.py: add evolve-specific Benchmark Target task section before the generic Focus Directive block - workflow/definitions.py: update Baseline FnNode notes to read benchmark name from the CEO task - Regenerate workflow-evolve/SKILL.md Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Collaborator
Author
Update: benchmark-agnostic evolve promptsCherry-picked 3 commits from PR #1051:
Combined with the pre_eval/post_eval nodes from this PR, the evolve mode is now:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
.factory/evolve/current_score.json→.factory/experiments/$EXP_ID/eval_before.jsonbetweenbeginandbuilder, giving CycleAnalyzer a before-snapshot for every experimenteval.completedevent toevents.jsonlbetweenhealth_checkerandgate_eval, enabling CycleAnalyzer._extract_scores() to build score_trajectorywritesset to include.factory/experiments/000/eval_before.jsonand added steps 7-8 to noteswritesset to include.factory/experiments/$EXP_ID/eval_after.jsonand updated prompt step 7 with exact JSON schemabegin→builderwithbegin→pre_eval→builderandhealth_checker→gate_evalwithhealth_checker→post_eval→gate_evalAll 27 evolve workflow tests pass. Lint clean on modified files.