fix: honor virtualenv exemption in engine matching for multimodal LLM, embedding and rerank#5179
Conversation
There was a problem hiding this comment.
Code Review
This pull request updates the LMDeploy, SGLang, and vLLM engines to tolerate missing libraries, GPUs, or OS requirements when virtualenv mode is enabled, allowing these engines to be listed and installed on demand. It also adds a regression test for multimodal engines under virtualenv. The review feedback suggests simplifying the implementation by importing the XINFERENCE_ENABLE_VIRTUAL_ENV constant directly at the top level of the modules instead of defining redundant helper functions with try-except blocks.
…hing Multimodal (vision) models such as qwen3.5 could not select vLLM or SGLang in the WebUI even when virtualenv on-demand installation was enabled, while chat-only models like qwen3 could. The vision/multimodal match_json branches gated on runtime-only conditions (library installed, CUDA GPU present, Linux) before or without the virtualenv exemption that the chat branch already had, so a declared engine was hidden at listing time. Make the exemption consistent across LLM engines: when virtualenv is enabled, skip environment-only checks (library present, GPU, OS) so an architecture/format-compatible engine is still offered and installed on demand at launch. - vLLM: exempt GPU/OS/library checks in VLLMModel and VLLMMultiModel. - SGLang: add _virtual_env_allows_missing_sglang and exempt GPU/OS/library checks in the generate, chat and vision match_json branches. - LMDeploy: add _virtual_env_allows_missing_lmdeploy and exempt the library check in the chat branch. MLX (Apple-silicon-only) and Transformers/llama.cpp (environment-agnostic) already behave correctly and are unchanged. Add a regression test driving the real engine-listing path to assert qwen3.5 offers vLLM/SGLang/Transformers/llama.cpp under virtualenv.
…hing Extend the multimodal virtualenv exemption to embedding and rerank engines. The vLLM match_json for Qwen3-VL-Embedding and Qwen3-VL-Reranker gated on a local vLLM install (and a specific version) inside match_json, so the engine was hidden at listing time even though virtualenv can install it on demand. - Add a shared virtual_env_allows_missing_engine() helper in model/utils.py. - embedding vLLM: exempt the "vLLM is not installed" and version checks for Qwen3-VL-Embedding when virtualenv is enabled. - rerank vLLM: replace the unconditional "requires vLLM>=0.14.0" rejection for Qwen3-VL-Reranker with a real import + version check that is exempt under virtualenv, so the engine is offered (and installed on demand) instead of being permanently disabled. Other embedding/rerank engines (sentence_transformers, flag, llama.cpp) and image (stable_diffusion) already keep library checks in check_lib, so their listing honors the exemption and needs no change. Strengthen the embedding engine-params test and add a rerank engine-params test asserting vLLM is available under virtualenv; both fail on the previous code and pass with this change. When virtualenv is disabled the strict library/version checks are preserved.
def4708 to
67c6ab9
Compare
|
Thanks for the review. Addressed the feedback: removed the redundant |
qinxuye
left a comment
There was a problem hiding this comment.
Two correctness issues remain before this can be approved.
| # environment-only checks (GPU, OS, library) must not hide an | ||
| # otherwise-compatible engine at listing time. | ||
| allow_missing_env = _virtual_env_allows_missing_vllm() | ||
| if not allow_missing_env: |
There was a problem hiding this comment.
Virtualenv can install Python packages, but it cannot add accelerator hardware or turn a non-Linux worker into Linux. Since XINFERENCE_ENABLE_VIRTUAL_ENV defaults to true, this makes vLLM/SGLang match on unsupported workers; on a macOS host with no CUDA, VLLMMultiModel.match_json() now returns True. It also ignores an explicit request-level enable_virtual_env=False because match_json reads the process-global constant. Please keep the hardware/OS checks unconditional and use the effective request flag only to exempt missing runtime libraries. The same issue applies to the SGLang branches changed in this PR.
| if version.parse(vllm.__version__) < version.parse("0.14.0"): | ||
| if not allow_missing_env and version.parse( | ||
| vllm.__version__ | ||
| ) < version.parse("0.14.0"): |
There was a problem hiding this comment.
This assumes the virtualenv will install a compatible vLLM, but these Qwen3-VL model specs only declare #vllm_dependencies#, which currently expands to vllm>=0.11.2. An existing environment with 0.11.2 already satisfies that requirement, while model initialization later requires vLLM>=0.14.0 and fails. Please encode vllm>=0.14.0 in the Qwen3-VL embedding and reranker virtualenv requirements (and test the expanded requirement) before bypassing this version check.
Problem
On an environment without vLLM/SGLang installed (or without a local GPU), a
chat-only model like
qwen3can be launched with vLLM/SGLang, but theirmultimodal counterparts cannot — the engines don't appear as selectable in the
WebUI. The same asymmetry affects
Qwen3-VL-EmbeddingandQwen3-VL-Reranker.Engine availability is computed by
get_engine_params_by_name_with_virtual_env,which calls each engine's
match_json. Some engines put runtime-only checks— library installed, specific version, CUDA GPU, Linux — inside
match_json,before (or instead of) the virtualenv exemption. Chat LLM engines already had
the exemption; the multimodal / embedding / rerank vLLM paths did not, so a
declared engine was hidden at listing time even though virtualenv installs it on
demand at launch (
check_engine_by_spec_parameters_with_virtual_envalreadybypasses this).
Fix
Make the virtualenv exemption consistent: when virtualenv is enabled,
environment-only checks (library present, version, GPU, OS) are skipped so an
architecture/format-compatible engine is still offered and installed on demand.
When virtualenv is disabled, all strict checks are preserved.
LLM
VLLMModelandVLLMMultiModel._virtual_env_allows_missing_sglang; exempt GPU/OS/library checksin the generate, chat and vision branches.
_virtual_env_allows_missing_lmdeploy; exempt the library checkin the chat branch.
Embedding / Rerank
virtual_env_allows_missing_engine()helper inmodel/utils.py.Qwen3-VL-Embeddingunder virtualenv.requires vLLM>=0.14.0rejection forQwen3-VL-Rerankerwith a real import + version check that is exempt undervirtualenv, so the engine is offered instead of being permanently disabled.
Unchanged (already correct): MLX (Apple-silicon-only, marker auto-dropped on
non-macOS), Transformers / llama.cpp, embedding/rerank
sentence_transformers/flag/llama.cpp, and image (stable_diffusion) — all keep
library checks in
check_lib, so listing already honors the exemption. Audio /video do not use the engine-selection mechanism.
Tests
qwen3.5offers vLLM/SGLang/Transformers/llama.cpp under virtualenv while MLX/LMDeploy (not declared) are not listed.
Qwen3-VL-Embedding-2Boffers vLLM.Qwen3-VL-Reranker-2Boffers vLLM.All new/updated assertions fail on the previous code and pass with this change.
Verified manually for a no-GPU host and a simulated CUDA+Linux host with the
libraries absent.