feat: add constraint validation to evolve workflow - #1067
Open
abhi1092 wants to merge 18 commits into
Open
Conversation
…e effectiveness Adds W₁₂ optimize mode — a meta-mode that analyzes how well an existing factory mode performs, identifies weaknesses (high redirect rates, agent timeouts, low keep rates), generates a targeted change specification, and delegates implementation to create mode. Interactive-only with user approval gate before modifying workflows. Changes: - Add optimize_workflow() to factory/workflow/definitions.py (8 nodes, 9 edges) - Register in register_all() and __all__ - Add WORKFLOW_META entry in skill_export.py - Wire --mode optimize in CLI (_helpers.py, _main.py, ceo.py) - Add optimize to CycleState.mode Literal in models.py - Add 12 tests in tests/test_optimize_workflow.py Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
feat: add optimize workflow mode for factory mode effectiveness tuning
Signed-off-by: Abhishek Bhandwaldar <abhi1092@gmail.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…evaluation Implements W₁₄: Evolve Mode — a tight evaluate-mutate-select loop that optimizes a single scalar metric by evolving code within EVOLVE-BLOCK boundaries, evaluated externally via MCP server tools (get_benchmark_info, evaluate_solution). Key design decisions: - No deep-QA pipeline (single health_checker with MCP eval replaces the 3-specialist QA pipeline — tighter loops produce better results) - Elitist keep/revert: score must strictly improve - FEEC stuck detection: 3 consecutive reverts → re-research - Convergence: target score, max cycles, or diminishing returns Changes: - factory/workflow/definitions.py: evolve_workflow() + register_all() - factory/workflow/skill_export.py: WORKFLOW_META entry - factory/cli/_helpers.py: CEO_MODES list - factory/cli/ceo.py: validation, ceo_mode routing, _build_ceo_task - tests/test_evolve_workflow.py: 20 tests (5 classes) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The evolve workflow was added in 7a14aee but its generated SKILL.md and annotations were not committed, so the CEO cannot load the evolve mode playbook at runtime. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
feat: add evolve workflow mode and research queue baseline
Reads .factory/ artifacts (events.jsonl, results.tsv, experiment dirs) and produces structured CycleRecord objects that any outer-loop optimizer can consume. Mode-agnostic — works with evolve, improve, research, refine. Tiered data assembly: - Tier 1: events.jsonl for agent steps, costs, durations - Tier 2: results.tsv for experiment scores, verdicts, hypotheses - Tier 3: experiment dirs for eval artifact discovery - Tier 4: workflow definition for DAG node mapping Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
feat: add CycleAnalyzer for outer-loop optimizer observability
…ation InnerLoop wraps a factory mode + evaluator into a model.forward()-like interface that any outer-loop optimizer can call. CycleAnalyzer reads .factory/ artifacts and produces structured CycleRecord objects. Includes CirclePackingEvaluator as the first concrete evaluator implementation, tested against skydiscover's circle packing benchmark. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Evaluator now reads output artifacts that the inner loop already produced,
rather than calling the evaluator binary directly. CirclePackingEvaluator
parses {sum_radii, validity, eval_time, combined_score} JSON files.
CycleAnalyzer: execution tracing (agents, costs, verdicts)
Evaluator: score interpretation (evaluator-specific artifact parsing)
InnerLoop: composes both
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
feat: add InnerLoop and CycleAnalyzer for outer-loop optimizer integration
…prompts Replace domain-specific prompt hints with domain-agnostic instructions that direct agents to discover the benchmark domain at runtime from eval.json, making the evolve workflow work correctly across all benchmark types. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The SKILL.md was stale after the prompt_template changes in definitions.py (commit 2cabd28). Regenerated via `factory workflow export-skills` to reflect the domain-agnostic researcher and health_checker prompts. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
fix: make evolve workflow prompts domain-agnostic
Allow evolve mode to accept --focus to specify a benchmark name. The focus is injected as a ## Benchmark Target section in the CEO task, and the Baseline FnNode notes reference it for get_benchmark_info() calls. Changes: - cli/ceo.py: add 'evolve' to allowed focus modes in both cmd_ceo and cmd_run gates - cli/ceo.py: add evolve-specific Benchmark Target task section before the generic Focus Directive block - workflow/definitions.py: update Baseline FnNode notes to read benchmark name from the CEO task - Regenerate workflow-evolve/SKILL.md Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
feat: wire up --focus support for evolve mode
…e workflow Update 4 nodes in the evolve workflow for constraint awareness: - researcher: reads constraints from eval.json, filters techniques, requires Constraint Compliance section in research output - gate_research: verifies researcher documented constraints, REDIRECTs if constraints exist but were ignored - strategist: reads eval.json constraints, validates hypothesis compliance, adds Constraint Compliance field to output - gate_strategy: MANDATORY constraint check as first gate item, NON-NEGOTIABLE stuck detection (3 consecutive reverts → forced reloop to researcher) No topology changes — same 14 nodes, 17 edges. Only prompt_template, gate_prompt, and reads sets modified. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
eval.json, filters optimization techniques, requires Constraint Compliance section in outputChanges
4 node updates in
factory/workflow/definitions.py(evolve_workflow function):researcher— prompt_template adds constraint reading/filteringgate_research— gate_prompt adds CONSTRAINT COVERAGE verificationstrategist— prompt_template + reads set add eval.json constraint validationgate_strategy— gate_prompt reordered: constraints first, then mandatory stuck detection, then quality checks; reads set adds eval.jsonNo topology changes (14 nodes, 17 edges unchanged). SKILL.md and annotations regenerated.
Test plan
factory workflow validate evolvepasses (14 nodes, 17 edges)factory workflow export-skills --verify— all 25 skills validruff check factory/workflow/definitions.py— cleanpytest tests/test_evolve_workflow.py -v— 20/20 pass🤖 Generated with Claude Code