fix(diagnosis): pin temperature=0 + repair via tool_result turn (#63)#79
Merged
Merged
Conversation
Two correctness/honesty fixes for the single-shot diagnosis path: - The Anthropic call set no temperature, so it ran at the API default (1.0) despite the README calling the path "deterministic". Pin temperature=0 for run-to-run stability (the API exposes no seed; this is as close as a single-shot call gets) and soften the README claim to "run at temperature 0". - On a schema ValidationError the retry concatenated the error onto the user string and re-sent a fresh single-user-turn request. Replace that with a proper multi-turn repair exchange: replay the model's prior tool_use turn and answer it with a tool_result (is_error=True) carrying the validation error. Adds a RepairTurn dataclass + repair param threaded through the client, and captures the tool_use block id on LLMResult so the turn can be replayed. Cassette keys are (prompt_version, model_id, case_id, shot_index) — independent of the request body — so no cassettes need re-recording and the eval gate is unaffected. Tests assert temperature=0, the tool_use_id capture, the repair wire shape, and that the agent passes a RepairTurn on retry. Closes #63 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #63. Two correctness/honesty fixes to the single-shot diagnosis path.
1.
temperature=0The Anthropic call set no
temperature/top_p/seed, so it ran at the API default (1.0) — yet the README called the path "deterministic". Pintemperature=0for run-to-run stability (the API exposes no seed; this is as close as a single-shot call gets) and soften the README claim to "run at temperature 0".2. Repair as a real
tool_resultturnOn a schema
ValidationError, the retry previously concatenated the error onto theuserstring and re-sent a fresh single-user-turn request. Now it replays a proper multi-turn exchange:RepairTurndataclass +repairparam threaded throughdiagnose_call/_call_with_retry/_stream_once.LLMResultgainstool_use_id(captured from the returned block) so the assistant turn can be replayed by id.userprompt.Eval-gate safety
Cassette keys are
(prompt_version, model_id, case_id, shot_index)— independent of the request body — so addingtemperature/ changingmessagesinvalidates no cassettes and the replay-based eval gate is unaffected. No re-recording needed.Tests
test_stream_call_sets_temperature_zerotest_llm_result_carries_tool_use_idtest_repair_builds_tool_result_turn(asserts the full wire shape: roles, tool_use id/name/input, tool_result tool_use_id + is_error)test_retry_passes_repair_turn_with_prior_tool_use_id(agent wiring)492 unit tests pass; ruff + mypy --strict clean. Reviewed: the repair exchange is API-valid (matched tool_use/tool_result ids, ends on a user turn under forced tool_choice), and
temperature=0is valid for the pinnedclaude-sonnet-4-5.