feat: make evolve workflow domain-agnostic and improve stuck detection - #1070
Closed
abhi1092 wants to merge 25 commits into
Closed
feat: make evolve workflow domain-agnostic and improve stuck detection#1070abhi1092 wants to merge 25 commits into
abhi1092 wants to merge 25 commits into
Conversation
…e effectiveness Adds W₁₂ optimize mode — a meta-mode that analyzes how well an existing factory mode performs, identifies weaknesses (high redirect rates, agent timeouts, low keep rates), generates a targeted change specification, and delegates implementation to create mode. Interactive-only with user approval gate before modifying workflows. Changes: - Add optimize_workflow() to factory/workflow/definitions.py (8 nodes, 9 edges) - Register in register_all() and __all__ - Add WORKFLOW_META entry in skill_export.py - Wire --mode optimize in CLI (_helpers.py, _main.py, ceo.py) - Add optimize to CycleState.mode Literal in models.py - Add 12 tests in tests/test_optimize_workflow.py Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
feat: add optimize workflow mode for factory mode effectiveness tuning
Signed-off-by: Abhishek Bhandwaldar <abhi1092@gmail.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…evaluation Implements W₁₄: Evolve Mode — a tight evaluate-mutate-select loop that optimizes a single scalar metric by evolving code within EVOLVE-BLOCK boundaries, evaluated externally via MCP server tools (get_benchmark_info, evaluate_solution). Key design decisions: - No deep-QA pipeline (single health_checker with MCP eval replaces the 3-specialist QA pipeline — tighter loops produce better results) - Elitist keep/revert: score must strictly improve - FEEC stuck detection: 3 consecutive reverts → re-research - Convergence: target score, max cycles, or diminishing returns Changes: - factory/workflow/definitions.py: evolve_workflow() + register_all() - factory/workflow/skill_export.py: WORKFLOW_META entry - factory/cli/_helpers.py: CEO_MODES list - factory/cli/ceo.py: validation, ceo_mode routing, _build_ceo_task - tests/test_evolve_workflow.py: 20 tests (5 classes) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The evolve workflow was added in 7a14aee but its generated SKILL.md and annotations were not committed, so the CEO cannot load the evolve mode playbook at runtime. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
feat: add evolve workflow mode and research queue baseline
Reads .factory/ artifacts (events.jsonl, results.tsv, experiment dirs) and produces structured CycleRecord objects that any outer-loop optimizer can consume. Mode-agnostic — works with evolve, improve, research, refine. Tiered data assembly: - Tier 1: events.jsonl for agent steps, costs, durations - Tier 2: results.tsv for experiment scores, verdicts, hypotheses - Tier 3: experiment dirs for eval artifact discovery - Tier 4: workflow definition for DAG node mapping Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
feat: add CycleAnalyzer for outer-loop optimizer observability
…ation InnerLoop wraps a factory mode + evaluator into a model.forward()-like interface that any outer-loop optimizer can call. CycleAnalyzer reads .factory/ artifacts and produces structured CycleRecord objects. Includes CirclePackingEvaluator as the first concrete evaluator implementation, tested against skydiscover's circle packing benchmark. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Evaluator now reads output artifacts that the inner loop already produced,
rather than calling the evaluator binary directly. CirclePackingEvaluator
parses {sum_radii, validity, eval_time, combined_score} JSON files.
CycleAnalyzer: execution tracing (agents, costs, verdicts)
Evaluator: score interpretation (evaluator-specific artifact parsing)
InnerLoop: composes both
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
feat: add InnerLoop and CycleAnalyzer for outer-loop optimizer integration
…prompts Replace domain-specific prompt hints with domain-agnostic instructions that direct agents to discover the benchmark domain at runtime from eval.json, making the evolve workflow work correctly across all benchmark types. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The SKILL.md was stale after the prompt_template changes in definitions.py (commit 2cabd28). Regenerated via `factory workflow export-skills` to reflect the domain-agnostic researcher and health_checker prompts. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
fix: make evolve workflow prompts domain-agnostic
Allow evolve mode to accept --focus to specify a benchmark name. The focus is injected as a ## Benchmark Target section in the CEO task, and the Baseline FnNode notes reference it for get_benchmark_info() calls. Changes: - cli/ceo.py: add 'evolve' to allowed focus modes in both cmd_ceo and cmd_run gates - cli/ceo.py: add evolve-specific Benchmark Target task section before the generic Focus Directive block - workflow/definitions.py: update Baseline FnNode notes to read benchmark name from the CEO task - Regenerate workflow-evolve/SKILL.md Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
feat: wire up --focus support for evolve mode
…op integration Add two FnNode stages to the evolve workflow DAG so that CycleAnalyzer can discover per-experiment artifacts and build score trajectories: - pre_eval: copies current_score.json → experiments/$EXP_ID/eval_before.json between begin and builder (closes Gap 4) - post_eval: emits eval.completed event to events.jsonl between health_checker and gate_eval (closes Gap 2) Also updates baseline node writes to include experiments/000/eval_before.json (Gap 3) and health_checker writes/prompt to produce structured eval_after.json in the experiment directory (Gap 1). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add 4 new nodes (run_target, collect_cycle, reflect_researcher, gate_convergence) and 5 new edges to create a convergence loop between gate_strategy and delegate_create. The loop runs the target mode, collects structured CycleRecord data via InnerLoop.collect(), has a Researcher analyze bottlenecks, and a CEO-evaluated convergence gate decides whether to loop again or proceed to delegation. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
feat: add run-collect-reflect inner loop to optimize workflow
Parse --focus "mode:/path/to/project" syntax so optimize mode can run the inner loop on a different project than the factory itself. The run_target and collect_cycle FnNodes use $OPTIMIZE_PROJECT for the target project path. Example: factory ceo . --mode optimize --focus "evolve:/path/to/skydiscover" Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Format: --focus "mode:project_path:benchmark_name" Example: --focus "evolve:/path/to/skydiscover:math/signal_processing" All three segments are optional after the mode name. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Three targeted changes to evolve_workflow(): 1. Health checker: replace hardcoded sum_radii/target_ratio eval schema with generic instructions to include ALL MCP-returned fields verbatim. 2. Stuck detection: lower threshold from 3→2 consecutive reverts and require same FEEC family (Category field) match before triggering fresh research, in both strategist and gate_strategy nodes. 3. Benchmark context: add .factory/baseline/benchmark_info.json to baseline writes, researcher reads, and strategist reads. Update prompts to reference structured domain context from this file instead of extracting domain from eval.json benchmark name. No graph topology changes — same 16 nodes, 19 edges. All 20-point verification checklist items pass. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ions.py The previous commit updated definitions.py correctly but the generated artifacts (SKILL.md and SKILL.annotations.yaml) were stale. Regenerated via Python to reflect all three changes: - health_checker uses generic eval schema (no hardcoded sum_radii/target_ratio) - stuck detection lowered from 3 to 2 same-FEEC-family reverts - benchmark_info.json added to baseline/researcher/strategist reads/writes Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Collaborator
Author
|
Closing — reopening on origin (abhi1092/remote-factory) instead. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #14
Changes
Domain-agnostic health checker eval schema: Replaced hardcoded
sum_radii/target_ratiofields in the health_checker prompt with generic instructions to include ALL fields returned byevaluate_solution()verbatim. The schema is now determined by the MCP evaluator's response, not the template.Smarter stuck detection (3→2 same-family): Lowered the stuck detection threshold from 3 consecutive reverts to 2, and added a same-FEEC-family requirement (same Category field). Applied to both the strategist prompt and gate_strategy gate prompt.
Benchmark context via benchmark_info.json: Added
.factory/baseline/benchmark_info.jsonto baseline writes, researcher reads, and strategist reads. Updated prompts to reference structured domain context (evaluation_criteria, constraints, optimization_target) instead of extracting domain from eval.json benchmark name.No graph topology changes — same 16 nodes, 19 edges. All 20-point verification checklist items pass. 216 tests pass, lint clean.