refactor: deepen objective-evaluation choreography (PR 2) - #41
Merged
Conversation
Extract lib/core/objective_evaluation.dart as the single deep module owning ordered evaluator execution, blocked-result insertion, previous-result propagation, and aggregate/primitive derivation: - runObjectiveEvaluators: runs evaluators in order, inserting a blocked placeholder once a hard blocker is observed, and giving each evaluator every prior result via previousResults. - blockEvaluatorsForHardFailure: records a hard failure (prepare failure, patch capture failure, workspace/isolation failure) and blocks all remaining evaluators, without duplicating an already recorded hard blocker. - finalizeObjectiveEvaluation: derives aggregate score, primaryPass, and failureTag together. Migrates the three real duplicated callers onto this choreography: CodegenTaskExecutor's success and prepare-failure paths, AgenticRunOrchestrator's clean-replay grading loop and its three failure-result builders, and TaskQaRunner's evaluator loop (used for baseline, reference, and negative-case runs). Codegen generation and agentic clean replay/preparation remain untouched in their adapters; sandbox, process, patch replay, harness, and task-policy preparation are not moved. Refs #32
- Add test/runner/objective_evaluation_test.dart covering evaluator order, immutable prior-result snapshots (including blocked placeholders), before/after-evaluator cancellation checkpoints, hard-failure dedup with placeholders referencing the earlier blocker, and finalizeObjectiveEvaluation outputs for known inputs. - Move lib/core/objective_evaluation.dart to lib/runner/, since it depends on analytics and all three real callers are runners; avoids a core<->analytics package-layer cycle. - Simplify blockEvaluatorsForHardFailure's failure parameter to non-nullable, since every real caller supplies one, and fix the doc wording; hard-blocker dedup semantics are unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR 2 of #32: deepen objective-evaluation choreography without changing codegen generation, agentic clean replay/preparation, or Task QA's role as an independent equivalence oracle.
Extracts
lib/core/objective_evaluation.dartas the single deep module owning ordered evaluator execution, blocked-result insertion, previous-result propagation, aggregate calculation, and result primitive (primary pass/failure tag) derivation:runObjectiveEvaluators— runs evaluators in order against a mutableevaluationslist, inserting a blocked placeholder (via existingblockedEvaluationFor) once a hard blocker has been observed instead of invoking the evaluator, and giving each evaluator every prior result aspreviousResults.blockEvaluatorsForHardFailure— records a hard failure (prepare failure, patch capture failure, workspace/isolation failure) and blocks all remaining evaluators, without duplicating an already-recorded hard blocker.finalizeObjectiveEvaluation— derives the aggregate score together withprimaryPass/failureTagfrom a completed evaluation list.Migrates the three real duplicated callers onto this choreography:
CodegenTaskExecutor's success path and prepare-failure path.AgenticRunOrchestrator's clean-replay grading loop and its three failure-result builders (missing harness, environment failure, grading prepare/patch-apply failure).TaskQaRunner's evaluator loop, used identically for baseline, reference, and negative-case runs — Task QA remains an independent equivalence oracle, now routed through the same deterministic choreography for the same objective evaluation it always performed.Codegen generation, agentic clean replay/preparation, sandbox, process, patch replay, harness, and task-policy preparation are untouched and stay in their adapters. No generic single-adapter seam was introduced; all three real callers were migrated.
Validation:
cd app && dart test --concurrency=1 test/runner/codegen_task_executor_blocking_test.dart test/runner/agentic_run_orchestrator_test.dart test/core/evaluator_blocking_test.dart test/analytics/result_primitives_test.dart test/runner/task_qa_runner_test.dart(46/46 pass),dart analyze(no issues), full serialdart test -j 1(872/872 pass).Refs #32