Skip to content

fix: honor virtualenv exemption in engine matching for multimodal LLM, embedding and rerank#5179

Open
OliverBryant wants to merge 2 commits into
xorbitsai:mainfrom
OliverBryant:fix/multimodal-engine-virtualenv-exemption
Open

fix: honor virtualenv exemption in engine matching for multimodal LLM, embedding and rerank#5179
OliverBryant wants to merge 2 commits into
xorbitsai:mainfrom
OliverBryant:fix/multimodal-engine-virtualenv-exemption

Conversation

@OliverBryant

@OliverBryant OliverBryant commented Jul 15, 2026

Copy link
Copy Markdown
Collaborator

Problem

On an environment without vLLM/SGLang installed (or without a local GPU), a
chat-only model like qwen3 can be launched with vLLM/SGLang, but their
multimodal counterparts cannot — the engines don't appear as selectable in the
WebUI. The same asymmetry affects Qwen3-VL-Embedding and Qwen3-VL-Reranker.

Engine availability is computed by get_engine_params_by_name_with_virtual_env,
which calls each engine's match_json. Some engines put runtime-only checks
— library installed, specific version, CUDA GPU, Linux — inside match_json,
before (or instead of) the virtualenv exemption. Chat LLM engines already had
the exemption; the multimodal / embedding / rerank vLLM paths did not, so a
declared engine was hidden at listing time even though virtualenv installs it on
demand at launch (check_engine_by_spec_parameters_with_virtual_env already
bypasses this).

Fix

Make the virtualenv exemption consistent: when virtualenv is enabled,
environment-only checks (library present, version, GPU, OS) are skipped so an
architecture/format-compatible engine is still offered and installed on demand.
When virtualenv is disabled, all strict checks are preserved.

LLM

  • vLLM: exempt GPU/OS/library checks in VLLMModel and VLLMMultiModel.
  • SGLang: add _virtual_env_allows_missing_sglang; exempt GPU/OS/library checks
    in the generate, chat and vision branches.
  • LMDeploy: add _virtual_env_allows_missing_lmdeploy; exempt the library check
    in the chat branch.

Embedding / Rerank

  • Add a shared virtual_env_allows_missing_engine() helper in model/utils.py.
  • embedding vLLM: exempt the "vLLM is not installed" and version checks for
    Qwen3-VL-Embedding under virtualenv.
  • rerank vLLM: replace the unconditional requires vLLM>=0.14.0 rejection for
    Qwen3-VL-Reranker with a real import + version check that is exempt under
    virtualenv, so the engine is offered instead of being permanently disabled.

Unchanged (already correct): MLX (Apple-silicon-only, marker auto-dropped on
non-macOS), Transformers / llama.cpp, embedding/rerank
sentence_transformers/flag/llama.cpp, and image (stable_diffusion) — all keep
library checks in check_lib, so listing already honors the exemption. Audio /
video do not use the engine-selection mechanism.

Tests

  • LLM: regression test asserting qwen3.5 offers vLLM/SGLang/Transformers/
    llama.cpp under virtualenv while MLX/LMDeploy (not declared) are not listed.
  • Embedding: strengthened to assert Qwen3-VL-Embedding-2B offers vLLM.
  • Rerank: new engine-params test asserting Qwen3-VL-Reranker-2B offers vLLM.

All new/updated assertions fail on the previous code and pass with this change.
Verified manually for a no-GPU host and a simulated CUDA+Linux host with the
libraries absent.

@XprobeBot XprobeBot added bug Something isn't working gpu labels Jul 15, 2026
@XprobeBot XprobeBot added this to the v2.x milestone Jul 15, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the LMDeploy, SGLang, and vLLM engines to tolerate missing libraries, GPUs, or OS requirements when virtualenv mode is enabled, allowing these engines to be listed and installed on demand. It also adds a regression test for multimodal engines under virtualenv. The review feedback suggests simplifying the implementation by importing the XINFERENCE_ENABLE_VIRTUAL_ENV constant directly at the top level of the modules instead of defining redundant helper functions with try-except blocks.

Comment thread xinference/model/llm/lmdeploy/core.py Outdated
Comment thread xinference/model/llm/lmdeploy/core.py Outdated
Comment thread xinference/model/llm/sglang/core.py Outdated
Comment thread xinference/model/llm/sglang/core.py Outdated
Comment thread xinference/model/llm/sglang/core.py Outdated
Comment thread xinference/model/llm/sglang/core.py Outdated
@OliverBryant OliverBryant changed the title fix(llm): honor virtualenv exemption in multimodal engine matching fix: honor virtualenv exemption in engine matching for multimodal LLM, embedding and rerank Jul 15, 2026
…hing

Multimodal (vision) models such as qwen3.5 could not select vLLM or SGLang
in the WebUI even when virtualenv on-demand installation was enabled, while
chat-only models like qwen3 could. The vision/multimodal match_json branches
gated on runtime-only conditions (library installed, CUDA GPU present, Linux)
before or without the virtualenv exemption that the chat branch already had,
so a declared engine was hidden at listing time.

Make the exemption consistent across LLM engines: when virtualenv is enabled,
skip environment-only checks (library present, GPU, OS) so an
architecture/format-compatible engine is still offered and installed on demand
at launch.

- vLLM: exempt GPU/OS/library checks in VLLMModel and VLLMMultiModel.
- SGLang: add _virtual_env_allows_missing_sglang and exempt GPU/OS/library
  checks in the generate, chat and vision match_json branches.
- LMDeploy: add _virtual_env_allows_missing_lmdeploy and exempt the library
  check in the chat branch.

MLX (Apple-silicon-only) and Transformers/llama.cpp (environment-agnostic)
already behave correctly and are unchanged.

Add a regression test driving the real engine-listing path to assert qwen3.5
offers vLLM/SGLang/Transformers/llama.cpp under virtualenv.
…hing

Extend the multimodal virtualenv exemption to embedding and rerank engines.
The vLLM match_json for Qwen3-VL-Embedding and Qwen3-VL-Reranker gated on a
local vLLM install (and a specific version) inside match_json, so the engine
was hidden at listing time even though virtualenv can install it on demand.

- Add a shared virtual_env_allows_missing_engine() helper in model/utils.py.
- embedding vLLM: exempt the "vLLM is not installed" and version checks for
  Qwen3-VL-Embedding when virtualenv is enabled.
- rerank vLLM: replace the unconditional "requires vLLM>=0.14.0" rejection for
  Qwen3-VL-Reranker with a real import + version check that is exempt under
  virtualenv, so the engine is offered (and installed on demand) instead of
  being permanently disabled.

Other embedding/rerank engines (sentence_transformers, flag, llama.cpp) and
image (stable_diffusion) already keep library checks in check_lib, so their
listing honors the exemption and needs no change.

Strengthen the embedding engine-params test and add a rerank engine-params
test asserting vLLM is available under virtualenv; both fail on the previous
code and pass with this change. When virtualenv is disabled the strict
library/version checks are preserved.
@OliverBryant OliverBryant force-pushed the fix/multimodal-engine-virtualenv-exemption branch from def4708 to 67c6ab9 Compare July 15, 2026 04:22
@OliverBryant

Copy link
Copy Markdown
Collaborator Author

Thanks for the review. Addressed the feedback: removed the redundant _virtual_env_allows_missing_* helper functions in sglang/core.py and lmdeploy/core.py and now import XINFERENCE_ENABLE_VIRTUAL_ENV at the module top level, using it directly (confirmed no circular-import risk since constants only depends on os/pathlib). The shared virtual_env_allows_missing_engine() helper in model/utils.py (used by the embedding/rerank vLLM paths) was likewise simplified to reference the top-level constant. Left the pre-existing _virtual_env_allows_missing_vllm in vllm/core.py unchanged to keep the diff focused. Also rebased onto the latest main to drop unrelated commits.

@qinxuye qinxuye left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two correctness issues remain before this can be approved.

# environment-only checks (GPU, OS, library) must not hide an
# otherwise-compatible engine at listing time.
allow_missing_env = _virtual_env_allows_missing_vllm()
if not allow_missing_env:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Virtualenv can install Python packages, but it cannot add accelerator hardware or turn a non-Linux worker into Linux. Since XINFERENCE_ENABLE_VIRTUAL_ENV defaults to true, this makes vLLM/SGLang match on unsupported workers; on a macOS host with no CUDA, VLLMMultiModel.match_json() now returns True. It also ignores an explicit request-level enable_virtual_env=False because match_json reads the process-global constant. Please keep the hardware/OS checks unconditional and use the effective request flag only to exempt missing runtime libraries. The same issue applies to the SGLang branches changed in this PR.

if version.parse(vllm.__version__) < version.parse("0.14.0"):
if not allow_missing_env and version.parse(
vllm.__version__
) < version.parse("0.14.0"):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This assumes the virtualenv will install a compatible vLLM, but these Qwen3-VL model specs only declare #vllm_dependencies#, which currently expands to vllm>=0.11.2. An existing environment with 0.11.2 already satisfies that requirement, while model initialization later requires vLLM>=0.14.0 and fails. Please encode vllm>=0.14.0 in the Qwen3-VL embedding and reranker virtualenv requirements (and test the expanded requirement) before bypassing this version check.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working gpu

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants