feat: add prefix-cache-affinity dp rank routing.

[shifengmin]

Description

Related Issues

Change Type

• Bug fix
• New feature
• Performance improvement
• Refactor
• Documentation
• Test
• Build or CI

Pull Request Checklist

Thank you for contributing to xLLM. Before requesting review, please make sure the following items are complete.

PR Title and Commit Messages

• The PR title and each commit message follow the xLLM commit format: <type>: <subject>.

Allowed types: feat, bugfix, docs, test, refactor, chore, style, revert, perf, model, build, release.
The subject should use clear English, start with a verb, include at least 4 words, and end with ..

Pre-commit Checks

• I have installed pre-commit by running pip install pre-commit or an equivalent command.
• I have installed the hooks with pre-commit install.
• I have run pre-commit run --all-files and fixed any reported issues.

If you are unsure how to set up pre-commit, see the pre-commit documentation.

Self Review

• [] I have self-reviewed the code according to .agents/skills/code-review/references/custom-code-style.md, especially code written or assisted by AI.
• [] I have rebased this PR onto the latest main branch.

Build and Test Coverage

• Tests have been added or updated as needed.
• CUDA: python setup.py build test has passed on a CUDA machine.
• NPU: python setup.py build test has passed on an NPU machine.
• MLU: python setup.py build test has passed on an MLU machine.

Reviewer Notes

[gemini-code-assist]

Code Review

This pull request introduces prefix-cache-aware data-parallel (DP) routing to optimize KV cache reuse across DP ranks. It adds configuration options (enable_prefix_cache_aware_dp_routing, prefix_cache_aware_dp_match_threshold, and prefix_cache_aware_dp_imbalance_threshold) and implements a read-only prefix_cache_match_length probe to determine prefix-cache hit lengths without mutating state. The routing logic in BlockManagerPool uses these metrics to route sequences to the rank with the longest prefix match, while guarding against cold-start herding and popular-prefix overloading. Feedback suggests adding a defensive null check for the sequence pointer in get_cache_aware_dp_rank to prevent potential segmentation faults.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[XuZhang99]

also add to xllm-docs

[shifengmin]

xllm-docs PR xLLM-AI/xllm-docs#35

[XuZhang99]

and then remove docs in this pr

[shifengmin]

removed