Fail fast on invalid OpenRouter model names

[ScuttleBot]

Problem

Running a benchmark with a typo'd model name (e.g. anthropic/claude-sonet-4 instead of claude-sonnet-4) wastes a full benchmark run that just scores 0% on every task.

Solution

Validate the model ID against OpenRouter's /api/v1/models endpoint before running any tasks.

Features

• Queries OpenRouter API to check if model exists
• Suggests close matches if model name looks like a typo
• Lists available models from the same provider as hints
• Gracefully skips validation if OPENROUTER_API_KEY not set or API errors
• Exits immediately with code 1 if model is invalid

Example Output

❌ Model 'anthropic/claude-sonet-4' not found on OpenRouter.
   Did you mean: anthropic/claude-sonnet-4?

Or for completely wrong names:

❌ Model 'anthropic/gpt-5' not found on OpenRouter.
   Available anthropic models: anthropic/claude-3-opus, anthropic/claude-3-sonnet, anthropic/claude-sonnet-4, ...

[kilo-code-bot]

Code Review Summary

Status: No Issues Found | Recommendation: Merge

The PR adds a validate_openrouter_model() function that checks whether a given model ID exists on OpenRouter before running the benchmark. This is a well-structured fail-fast mechanism that:

• Tries the specific model endpoint first (fast path), falling back to the full catalog only on 404
• Gracefully handles network/API errors by skipping validation rather than blocking
• Provides helpful "did you mean" suggestions for typos
• Is properly integrated into benchmark.py with clean error handling

The error handling is thorough, covering HTTPError, URLError, and JSONDecodeError. The API key is transmitted securely over HTTPS. The code follows existing patterns in the codebase.

Files Reviewed (2 files)

• scripts/lib_agent.py - New ModelValidationError class and validate_openrouter_model() function
• scripts/benchmark.py - Integration of model validation before benchmark execution