test: add Claude coverage for the model-pin drift guard in test_health.py#483
Merged
Merged
Conversation
…alth.py TestCheckAll's three model-pin drift guard tests (configured-model-present, retired-model-pin-unhealthy, unparseable-body-stays-healthy) exercised utils/health.py's expect_model check only for Grok, even though check_all() runs the identical _ping(..., expect_model=...) call for Claude via the symmetric hybrid+claude.enabled block. The retired/renamed-model detection that protects against a stale Grok pin (the exact class of bug the grok-4->grok-4.3 retirement already hit) had zero coverage for the same code path when it's Claude's model id that drifts. Mirrors all three cases. GROK_API_KEY=dummy pytest tests/test_health.py -q --tb=short -> 17 passed (14 existing + 3 new).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds 3 new tests to
tests/test_health.py::TestCheckAll, mirroring the 3 existing Grok model-pin drift guard tests for the Claude provider:test_hybrid_claude_configured_model_present_is_healthytest_hybrid_retired_claude_model_pin_reports_unhealthytest_hybrid_unparseable_models_body_stays_healthy_claudeWhy / benefit
utils/health.py::_ping'sexpect_modeldrift check runs identically forgrok_apiandclaude_api(same_ping(..., expect_model=...)call shape, symmetrichybrid+<provider>.enabledgate incheck_all()). ButTestCheckAll's three drift-guard tests only ever passedgrok_enabled=True/grok_model=...and asserted ongrok_api. The retired/renamed-model detection that protects against a stale pin — the exact class of bug thegrok-4→grok-4.3retirement already hit — had zero coverage for the same code path when it's Claude's model id (config.yaml'smodels.claude.model) that drifts instead.Risk to monitor
None — test-only change, no production code touched.
_write_cfgalready supportedclaude_enabled/claude_modelparams (used by two pre-existing tests), so no fixture changes were needed.Verify
GROK_API_KEY=dummy pytest tests/test_health.py -q --tb=short→ 17 passed (14 existing + 3 new). Ran on the sandbox's available Python (3.11); CI validates against the pinned 3.12 perci.yml's matrix.Scan context
CyClaw-Optimize (ponytail + Karpathy + fable-protocol loaded). Finding #8 of an 8-finding scan; deduped against open PRs #473, #477, #415.
Generated by Claude Code