Skip to content

Expose reasoning effort for custom Claude models#4194

Open
IsraelAraujo70 wants to merge 5 commits into
pingdotgg:mainfrom
IsraelAraujo70:feat/custom-claude-reasoning
Open

Expose reasoning effort for custom Claude models#4194
IsraelAraujo70 wants to merge 5 commits into
pingdotgg:mainfrom
IsraelAraujo70:feat/custom-claude-reasoning

Conversation

@IsraelAraujo70

@IsraelAraujo70 IsraelAraujo70 commented Jul 20, 2026

Copy link
Copy Markdown

Summary

  • discover reasoning capabilities dynamically for each custom Claude model ID through the Claude Agent SDK
  • build the T3 reasoning selector only from the supportedEffortLevels announced for that selected model
  • avoid assigning high or any other effort by default, and hide the control when no levels are advertised
  • preserve explicit selections such as xhigh end-to-end for Claude-compatible custom models

Why

Claude-compatible gateways can expose Codex/OpenAI models through Claude Code using custom model IDs. Claude Code only announces supportedEffortLevels for a custom model when that model is selected during initialization, so static capabilities would either hide valid reasoning levels or advertise options the model does not support. T3 should respect the model's dynamically discovered reasoning capabilities instead of assuming a fixed capability set.

The custom model IDs are therefore initialized individually through the Claude Agent SDK. The resulting initializationResult().models metadata drives the provider capabilities while preserving the existing generic provider-option UI. The adapter and CLI text generation now reuse the provider’s cached capabilities for runtime validation, so they stay aligned with the levels actually announced by each model. Custom-model probes run sequentially to preserve failure isolation without unbounded subprocess fan-out.

Closes #4192.

Before / After

Before After
Custom Claude models did not expose their model-specific reasoning capabilities. Each custom Claude model exposes only the reasoning levels announced by Claude Code; models with none do not show a reasoning control, and no effort is selected by default.
Before: no reasoning control After: reasoning effort menu

Testing

  • vp test run apps/server/src/provider/Layers/ProviderRegistry.test.ts apps/server/src/provider/Layers/ClaudeAdapter.test.ts apps/server/src/textGeneration/ClaudeTextGeneration.test.ts (111 tests)
  • vp run --filter t3 typecheck
  • targeted vp lint on the four changed files
  • integrated web verification with custom gpt-5.6-sol Claude model: the Claude SDK advertised Low, Medium, High, Extra High, and Max, the selector initially had no selection, and an explicit Extra High selection worked end-to-end

Note

Medium Risk
Increases probe subprocess work and changes effort handling for custom models; behavior is localized to Claude provider paths and covered by new tests, but mis-probes could hide or mislabel reasoning options.

Overview
Custom Claude model IDs now get reasoning effort from Claude Code’s per-model SDK metadata instead of static or built-in assumptions.

The capabilities probe runs a lightweight init for each configured custom model (concurrency 2, 25s timeout per probe), merges supportedEffortLevels into provider model capabilities, and feeds that through a cached resolveModelCapabilities hook shared by the adapter, CLI text generation, and provider status. Custom models show a reasoning selector only when levels are advertised; nothing is defaulted when they are not. Explicit selections like xhigh pass through when advertised, and built-in-only mappings (e.g. coercing xhigh to max) no longer apply to custom slugs.

Reviewed by Cursor Bugbot for commit ec95dc6. Bugbot is set up for automated code reviews on this repo. Configure here.

Note

Expose reasoning effort options for custom Claude models based on SDK capability probes

  • probeClaudeCapabilities in ClaudeProvider.ts now concurrently initializes the SDK for each custom model, collects advertised supportedEffortLevels, and returns them. Individual probes time out without discarding completed results.
  • getClaudeModelCapabilities and normalizeClaudeCliEffort are extended to use SDK-advertised effort levels for custom models, and xhigh is no longer coerced to max for custom models.
  • makeClaudeAdapter and makeClaudeTextGeneration accept an async resolveModelCapabilities resolver so capability lookups use the per-instance probe cache at runtime.
  • Provider status now surfaces custom models with effort-level selectors in the model selection UI when the SDK advertises them.

Macroscope summarized ec95dc6.

@coderabbitai

coderabbitai Bot commented Jul 20, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: a9dd7cfb-3beb-4c06-9bfc-1a0d377b94ca

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added vouch:unvouched PR author is not yet trusted in the VOUCHED list. size:S 10-29 changed lines (additions + deletions). labels Jul 20, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 89d24938e7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread apps/server/src/provider/Layers/ClaudeProvider.ts Outdated
@macroscopeapp

macroscopeapp Bot commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Needs human review

This PR introduces new feature behavior - enabling reasoning effort configuration for custom Claude models by probing each model individually and propagating capabilities through multiple layers. The concurrent probing logic and new capability resolution flow represent significant runtime changes that warrant human review.

You can customize Macroscope's approvability policy. Learn more.

@github-actions github-actions Bot added size:L 100-499 changed lines (additions + deletions). and removed size:S 10-29 changed lines (additions + deletions). labels Jul 20, 2026
Comment thread apps/server/src/provider/Layers/ClaudeProvider.ts Outdated
@IsraelAraujo70

Copy link
Copy Markdown
Author

Resolved the Codex feedback: removed the default high, discover capabilities from initializationResult().models / supportedEffortLevels per custom model, and added regression coverage proving no --effort is sent when nothing is selected.

Comment thread apps/server/src/provider/Layers/ClaudeProvider.ts Outdated
Comment thread apps/server/src/provider/Layers/ClaudeProvider.ts Outdated
Comment thread apps/server/src/provider/Layers/ClaudeProvider.ts Outdated

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit b7cff35. Configure here.

Comment thread apps/server/src/provider/Layers/ClaudeProvider.ts Outdated
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:L 100-499 changed lines (additions + deletions). vouch:unvouched PR author is not yet trusted in the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature]: Expose reasoning effort for custom Claude models

1 participant