[Feat] feat(aliases): allow setting reasoning level#912
[Feat] feat(aliases): allow setting reasoning level#912Killusions wants to merge 1 commit intovllm-project:mainfrom
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces an enhanced alias configuration system that allows for parameter overrides, specifically adding support for 'reasoning_effort'. It implements a new AliasConfig dataclass designed for backward compatibility with existing string-based aliases while enabling more complex configurations via CLI, JSON, or YAML. The routing logic has been updated to inject these configured parameters into backend requests if they are not already provided by the client. Review feedback focuses on improving the robustness of the alias string parsing to handle empty entries and ensuring that null values in YAML configurations are correctly handled to avoid validation errors.
2f21394 to
d00e070
Compare
|
@nejch Could you maybe take a look at this? |
0783bff to
7b0ff46
Compare
Signed-off-by: Linus Schlumberger <linus.schlumberger@siemens.com>
7b0ff46 to
0b6c5e0
Compare
|
@ruizhang0101 Thank you for merging my bugfix, could you maybe also have a look here, I'm also open to alternate approaches, but we are currently using this patch internally and we'd really appreciate having some way to configure reasoning and especially disable it |
Adds optional reasoning_effort support to static model aliases. The main use case is replacing non-reasoning models with reasoning ones without changing client behaviour: map the old alias to the new model with reasoning_effort=none and clients see no difference. It also makes it easy to expose multiple capability tiers (fast, thorough, etc.) from a single engine.
Aliases can be configured as alias:model|reasoning_effort=high (CLI/JSON) or as a dict with model and reasoning_effort keys (YAML). The router injects the configured value into forwarded requests unless the client already set one.
FIX #911
-swhen doinggit commit[Bugfix],[Feat], and[CI].Detailed Checklist (Click to Expand)
Thank you for your contribution to production-stack! Before submitting the pull request, please ensure the PR meets the following criteria. This helps us maintain the code quality and improve the efficiency of the review process.
PR Title and Classification
Please try to classify PRs for easy understanding of the type of changes. The PR title is prefixed appropriately to indicate the type of change. Please use one of the following:
[Bugfix]for bug fixes.[CI/Build]for build or continuous integration improvements.[Doc]for documentation fixes and improvements.[Feat]for new features in the cluster (e.g., autoscaling, disaggregated prefill, etc.).[Router]for changes to thevllm_router(e.g., routing algorithm, router observability, etc.).[Misc]for PRs that do not fit the above categories. Please use this sparingly.Note: If the PR spans more than one category, please include all relevant prefixes.
Code Quality
The PR need to meet the following code quality standards:
pre-committo format your code. SeeREADME.mdfor installation.DCO and Signed-off-by
When contributing changes to this project, you must agree to the DCO. Commits must include a
Signed-off-by:header which certifies agreement with the terms of the DCO.Using
-swithgit commitwill automatically add this header.What to Expect for the Reviews
We aim to address all PRs in a timely manner. If no one reviews your PR within 5 days, please @-mention one of YuhanLiu11
, Shaoting-Feng or ApostaC.