Skip to content

[LLM] Validate accelerator_type for CPU VLLM engine config#64235

Open
LeThienTrong wants to merge 2 commits into
ray-project:masterfrom
LeThienTrong:fix-llm-accelerator-cpu-validation
Open

[LLM] Validate accelerator_type for CPU VLLM engine config#64235
LeThienTrong wants to merge 2 commits into
ray-project:masterfrom
LeThienTrong:fix-llm-accelerator-cpu-validation

Conversation

@LeThienTrong

Copy link
Copy Markdown

Why are these changes needed?

This PR addresses #62138.

LLMConfig already rejects accelerator_type when the resolved hardware configuration is CPU-only. However, VLLMEngineConfig can still be instantiated directly with both accelerator_type and CPUConfig, which can make the accelerator hint inconsistent with a CPU-only configuration.

What changes were proposed in this pull request?

  • Added validation in VLLMEngineConfig to reject accelerator_type when accelerator_config is CPUConfig.
  • Added regression coverage for the invalid CPU config case.
  • Added a positive test case to ensure accelerator_type remains valid with a GPU config.

How was this patch tested?

  • Added unit tests in python/ray/llm/tests/serve/cpu/configs/test_models.py.

I could not run the full test locally because the Ray compiled extension _raylet is not built in my local Windows source checkout.

@LeThienTrong LeThienTrong requested a review from a team as a code owner June 20, 2026 17:28

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a validation check in VLLMEngineConfig to prevent the use of an accelerator_type with CPU-only configurations, raising an error if both are provided. It also adds corresponding unit tests to verify this behavior. The feedback points out that because VLLMEngineConfig is a Pydantic model, instantiating it with invalid values will raise a pydantic.ValidationError rather than a raw ValueError, and suggests updating the test assertion accordingly.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread python/ray/llm/tests/serve/cpu/configs/test_models.py
Signed-off-by: LeThienTrong <lethientrong655@gmail.com>
Signed-off-by: LeThienTrong <lethientrong655@gmail.com>
@LeThienTrong LeThienTrong force-pushed the fix-llm-accelerator-cpu-validation branch from 8e2ee46 to eb3d361 Compare June 20, 2026 17:40
@ray-gardener ray-gardener Bot added serve Ray Serve Related Issue llm community-contribution Contributed by the community labels Jun 20, 2026
@kouroshHakha

Copy link
Copy Markdown
Contributor

@LeThienTrong could u run the lint hook?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-contribution Contributed by the community llm serve Ray Serve Related Issue

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants