FEAT: prebuilt pypiserver mirror image as the offline compose package source

[qinrui777]

What this PR does

Ships xprobe/xinference-pypiserver: a prebuilt PyPI mirror image carrying every wheel the
xinference runtime may install into per-model virtual environments, and makes it the default
package source of the offline Docker Compose profile introduced in #5174. Air-gapped users
get working vllm / sglang / transformers / … model launches with zero wheel preparation.

This is a rework of the original approach in this PR. Testing showed that pointing
PIP_INDEX_URL / PIP_TRUSTED_HOST at the mirror does not reach the model-venv install
chain: per-model venv installs go through uv (which ignores PIP_* variables), and
xinference's pip-config inheritance (get_pip_config_args) only reads global.* keys from
pip config files, where PIP_INDEX_URL surfaces as :env:.index-url and is never inherited.
In a measured launch, uv installed packages while the mirror's TX byte counter stayed flat —
all traffic went to public PyPI. The compose/env-var half of this PR is therefore replaced by
the pip.conf + UV_* wiring that #5174 already merged (verified to reach uv:
settings(index_url=http://…:8080/simple/, …) in the worker log, mirror TX counter grows).

Changes

• Runtime: resolve direct wheel URLs from the private index (rewrite_direct_url_packages_for_index).
  Direct-URL requirements (the sgl_kernel GitHub wheels) bypass any index, breaking sglang
  in air-gapped setups even when the wheel is mirrored. When settings.index_url is explicitly
  configured, such requirements are rewritten to name==version (keeping local segments, e.g.
  sgl_kernel==0.3.21+cu130) and resolved from the mirror. Online deployments are unchanged.
• Runtime: drop support for CUDA below 13.0 in the virtualenv engine dependencies
  (PYTORCH_CUDA_WHEEL_URLS now cu130-only; the sgl_kernel ; cuda_version < "13.0" fallbacks
  are removed; the GLM-4.6/4.7 sgl_kernel URLs are aligned to v0.3.21 to match the code).
• Image build pipeline (xinference/deploy/docker/pypiserver/):
  • generate_package_lists.py loads ENGINE_VIRTUALENV_PACKAGES and the marker-filter logic
    straight from the xinference sources (no install), so the mirror cannot drift from what the
    runtime installs — this replaces the hand-maintained expansions in list-packages.sh.
  • download_packages.py locks the torch family to the runtime image's version
    (TORCH_VERSION=2.11.*), locks each engine set with uv pip compile, and fetches the
    fully-pinned locks with pip download --no-deps — one coherent resolution per engine
    instead of a per-line pip download loop that baked up to 12 versions of the same package.
    vllm/sglang pin their own torch stacks and are locked unconstrained (their venvs install a
    self-consistent stack). Per-model pins keep their exact versions. Git sources and
    sdist-only downloads are built into wheels so the runtime never compiles.
  • A selfcheck build stage re-resolves every engine set, model pin and direct wheel against
    the baked mirror alone (--disable-fallback); any gap fails the image build instead of
    failing an air-gapped user at model-launch time.
• CI (build-pypiserver-image.yaml): native amd64/arm64 builds merged into a multi-arch
  manifest. Pushes (including :latest) only happen on release tags of xorbitsai/inference,
  gated by check-release-permission; workflow_dispatch builds run the full download +
  selfcheck without publishing.
• Compose/docs: the offline profile defaults to the baked image (--disable-fallback);
  the previous stock-pypiserver + ./wheels setup moves to a docker-compose.byo-wheels.yml
  override; docs updated accordingly.

Verification

• pytest xinference/core/tests/test_virtualenv_package_rewrite.py — unit tests for the
  direct-URL rewrite (PEP 508 forms, markers, local versions, non-wheel passthrough).
• Generator run for both platforms: sglang set contains sglang>=0.5.6 + numpy>=2.4.1 with
  the arch-matching sgl_kernel wheel split into the direct-URL list; httpx==0.24.0
  correctly classified as a pin, not a URL.
• Full CI build (workflow_dispatch semantics, no publishing) on both architectures:
  amd64 562 files / 11.37 GiB, arm64 561 files / 11.32 GiB, and the offline selfcheck
  passed on both — all 6 engine sets, 150 model pins and 3 direct wheels resolve from the
  mirror alone ("mirror is self-sufficient"). ~20 legacy model pins took the recorded
  unconstrained-fallback path, as designed (old pins conflict with the modern locks;
  per-model venv isolation makes the extra versions legitimate).
  Locked engine stacks: vllm 0.25.1 (torch 2.11.0+cu130, matching the runtime image),
  sglang 0.5.9 + sgl-kernel 0.3.21 (torch 2.9.1+cu130, sglang's own pin),
  transformers/sentence_transformers aligned to torch 2.11.0+cu130.
• End-to-end on a 2×RTX 3090 Ti host (CUDA 13, Docker Compose v5.3), with the image built
  from this branch's pipeline and the runtime patched with this branch's code:
  • docker compose --profile offline up -d: worker log shows
    settings(index_url=http://xinference-pypiserver:8080/simple, extra_index_url=..., trusted_host=...)
    — pip.conf inheritance reaches the per-model uv installer, and the vLLM engine's
    hardcoded public extra indexes are overridden by the private one.
  • transformers engine (qwen3 0.6b): venv install traffic measured on the mirror
    container's NIC (+92 KiB for the forced extra wheels), inference OK.
  • sglang engine: the requirement list shows the GitHub direct wheel rewritten to
    sgl_kernel==0.3.21+cu130, resolved from the mirror; mirror NIC counter grew by
    ~390 MB during the venv install; sgl-kernel 0.3.21 + torch 2.11.0+cu130 landed in the
    venv. (Model load afterwards crashes on nightly-main because that image ships an
    incompatible sglang 0.5.6 / transformers 5.5.0 pair in its parent env — reproducible
    online, unrelated to this PR; will be reported separately.)
  • Air-gap override (docker-compose.airgap.yml, internal network, no external routing;
    curl https://pypi.org from the container fails): model launch + venv install +
    inference all succeed fully offline through the mirror.

Not run: the readthedocs render of the updated compose docs page.

[gemini-code-assist]

Code Review

This pull request introduces a Docker Compose setup for both online and offline modes, including a local PyPI server (pypiserver) to host offline wheels. Key feedback includes fixing the pypiserver command in docker-compose.yaml to ensure wheels are served, using --chown in the Dockerfile to avoid permission errors, correcting invalid CUDA index URLs, and resolving script robustness issues in list-packages.sh (such as potential hangs, pipeline failures, and incorrect relative paths).

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[qinxuye]

One issue remains around the offline package mirror build.

[rogercloud]

Requesting changes for the release gating and offline package integrity issues below.

[rogercloud]

Three design-level concerns about whether the offline mirror is actually usable in a true air-gapped deployment. These are independent of the silent-failure / release-gating points already raised.

[rogercloud]

Summary

This PR hardens air-gapped deployments against silent egress to public PyPI/GitHub during per-model venv installs. It rewrites direct wheel-URL requirements (e.g. sgl_kernel GitHub-release wheels) into name==version specs so they resolve from a configured index, ships a prebuilt xprobe/xinference-pypiserver mirror image whose package lists are generated directly from xinference source (a solid anti-drift choice), and adds a build-time selfcheck gate that fails the image build if the mirror cannot self-sufficiently resolve everything. It also trims the CUDA wheel matrix (drops cu128/cu129 torch index entries and the sgl_kernel ; cuda_version < "13.0" fallback) to bound the mirror's size.

Approach verdict: acceptable-with-reservations

The mirror-image + build-time selfcheck gate is a sound architecture, and generating the package lists straight from xinference source rather than a hand-maintained copy is genuinely good design. However, three changes that are motivated as offline-only concerns are actually wired into the unconditional / already-online install path, and the most important one (Finding A below) is a real regression for a large, common class of online users — recommend fixing that before merge.

Findings B and C are also MAJOR; D and E are MINOR test-coverage gaps. See inline comments for details, and the Simplification note at the bottom.

Verification performed

• Ran xinference/core/tests/test_virtualenv_package_rewrite.py: 7/7 pass.
• Full editable install with dev deps succeeded in a review worktree; no other test files exercise the changed code paths directly.
• Did not run the Docker image build itself (180-min CI-only step requiring registry access) — relying on the author's reported CI results for that portion.
• Coverage gaps: the three new pypiserver build scripts (selfcheck.py, download_packages.py, generate_package_lists.py) have no unit tests (Finding E); the CUDA-marker behavior change from dropping CUDA<13.0 has no regression test (Finding D).

Simplification opportunity

xinference/deploy/docker/pypiserver/download_packages.py:402 — the hand-rolled wheel_re regex (used only for the multi_version_packages report stat) can be replaced with packaging.utils.parse_wheel_filename. This file already imports and uses packaging.utils.canonicalize_name on the same names for the same purpose, so canonicalization is already the expected behavior here (no behavior mismatch), and the risk is confined to a reporting statistic. Needs a .whl-suffix check plus try/except InvalidWheelFilename to preserve the current silent-skip-on-non-wheel behavior (e.g. leftover .tar.gz sdist artifacts in the same directory).

net: -3 to -5 lines possible

(Two similar suggestions — applying the same swap to rewrite_direct_url_packages_for_index in utils.py:427 and to wheel_url_to_spec in selfcheck.py:50 — were considered and dropped: parse_wheel_filename PEP-503-canonicalizes names (sgl_kernel → sgl-kernel), which would break 5+ existing tests asserting underscore-preserving output, and raises on unparsable filenames rather than returning None.)

Out of scope (verified but not this PR's issue)

CI publishes :latest on any tag push to main, not only version-format tags (check-release-permission only checks who/where, never the tag's format). Confirmed real, but this is a pre-existing repo-wide convention also present in .github/workflows/docker-cd.yaml — not introduced by this PR.

[rogercloud]

[MAJOR] Finding A — Rewrite gate breaks online installs for any user with a pip mirror configured

This gate is a bare if settings.index_url:, which cannot distinguish "an offline mirror was explicitly configured for this feature" from "the user merely has a normal pip mirror in pip.conf" (Tsinghua/Aliyun/USTC, or a corporate Nexus/Artifactory proxy).

inherit_pip_config defaults to True (xinference/model/core.py:136) and unconditionally copies pip config list's global.index-url into settings.index_url for every model install (worker.py:2548-2553 via get_pip_config_args() in xinference/utils.py:42-81) — online or not.

Concrete failure: a fully-online user with a standard China mirror (or any corporate pip proxy) launches sglang. Pre-PR, the sgl_kernel GitHub release wheel (xinference/core/virtual_env_manager.py:36-37, carrying a +cu130 local-version segment) is fetched directly and works. Post-PR, rewrite_direct_url_packages_for_index (xinference/core/utils.py:427) rewrites it to sgl_kernel==0.3.21+cu130, resolved against the user's PyPI-proxy mirror — but PyPI disallows local-version-segment uploads, so no PyPI-compatible mirror can ever serve this. Per the comment at worker.py:2604-2611, uv treats an unreachable/unsatisfiable index entry as fatal, not a soft fallback to the original URL. Net: an install that worked before this PR now hard-fails for these users.

Suggested fix: gate the rewrite on an explicit offline/mirror-only signal (e.g. a distinct flag set by the docker offline profile) rather than the mere presence of settings.index_url.

[rogercloud]

[MAJOR] Finding B — Git-sourced package dependencies are excluded from the selfcheck gate

selfcheck.py's main() only re-resolves engines/*.in (line 124), pins.json (line 133), and urls.txt wheel URLs via wheel_url_to_spec (line 136) — it never reads or checks any git-sourced requirements list, even though download_packages.py:304-365 does build git-sourced deps into wheels and bakes them into the mirror (reading git.txt).

rewrite_direct_url_packages_for_index in xinference/core/utils.py intentionally leaves git+/name @ git+ specs unchanged (documented in its own docstring). At runtime, worker.py:2651-2676 passes the raw git spec straight to uv/pip; an explicit git/URL requirement is always fetched from that URL directly, regardless of a configured index. These specs are live in shipped model specs, not just test fixtures: xinference/model/image/model_spec.json:1234,1679,1714 and xinference/model/llm/llm_family.json:1854,2012.

So an air-gapped launch of such a model passes this build-time selfcheck, then fails at actual model-launch time trying to git clone from GitHub with no network — exactly the failure this selfcheck mechanism exists to catch before it reaches a user, silently missed for this one category.

Suggested fix: add a git-source check here (realistically: verify the wheel download_packages.py built for each git dep exists and installs standalone, since a live git-ref reachability check is impossible offline), or explicitly document git-sourced-dependency models as unsupported by the offline profile.

[rogercloud]

[MAJOR] Finding C — CUDA <13.0 drop applies unconditionally to the online path, with silent degradation

Removing the sgl_kernel ; cuda_version < "13.0" fallback here (and dropping cu128/cu129 from PYTORCH_CUDA_WHEEL_URLS below at line 84) is applied unconditionally in _prepare_virtual_env (xinference/core/worker.py:2574-2650) — not gated to offline/mirror mode.

Two distinct effects:

1. Torch-index part: a CUDA 12.8/12.9 online user now gets PYTORCH_CUDA_WHEEL_URLS.get(cuda_version) → None → no auto extra-index-url — same as any previously-unmapped CUDA version (cu121 was already unmapped). This is a reduction in supported versions, not a new failure mode.
2. sgl_kernel part (sharper regression): with the < "13.0" fallback gone, a CUDA 12.x online sglang install now resolves zero sgl_kernel entries (both remaining entries require cuda_version == "13.0"), silently, with no install-time error — the failure only surfaces later as a runtime import/missing-kernel error inside sglang.

The commit that made this change frames it as an offline-mirror wheel-matrix concern, but the code affects every deployment including fully online ones, with no breaking-change surfacing beyond a docker-compose docs note.

Suggested fix: document this as a product-level breaking change (release notes / CLI warning), and consider raising an explicit, actionable error when sgl_kernel is unavailable for CUDA<13.0 instead of silently installing sglang without its kernel.

[rogercloud]

[MINOR] Finding D — No test coverage for the new CUDA-version-marker filtering behavior

This is the only marker-filtering test (test_filter_then_rewrite_sglang_cu130), and it hardcodes cuda_version="13.0". No test anywhere in the repo exercises filter_virtualenv_packages_by_markers with a CUDA 12.x or 13.1+ value, so the behavior change from Finding C (sgl_kernel silently dropped for CUDA<13.0) has zero regression coverage.

Suggested fix: add a test asserting that for cuda_version="12.4" (or similar) the sglang package list contains no sgl_kernel entry — documents the intentional change and guards the marker-evaluation logic against future regressions.

[rogercloud]

[MINOR] Finding E — New image-build pipeline scripts have no unit test coverage

selfcheck.py (155 lines), download_packages.py (427 lines), and generate_package_lists.py (312 lines) are all new with non-trivial branching/parsing logic (lock/resolve/fallback handling, platform.machine monkeypatching for cross-arch resolution). No test file anywhere imports or exercises any of these three scripts.

wheel_url_to_spec here is independently written from the tested rewrite_direct_url_packages_for_index in xinference/core/utils.py, so the existing unit tests give only indirect, not-guaranteed confidence in this parser. build-pypiserver-image.yaml:32 sets a 180-minute timeout on the only job that exercises these scripts, and it triggers only on tag-push/workflow_dispatch, not on ordinary PRs.

Suggested fix (non-blocking follow-up): add fast unit tests for generate_package_lists.py's classification logic and this file's wheel_url_to_spec, independent of a full image build.

[rogercloud]

shrink: hand-rolled wheel_re regex reinvents wheel-filename parsing (used only for the multi_version_packages report stat). This file already imports and uses packaging.utils.canonicalize_name on the same names for the same purpose, so canonicalization is already the expected behavior here — replace with packaging.utils.parse_wheel_filename, adding a .whl-suffix check plus try/except InvalidWheelFilename to preserve the current silent-skip-on-non-wheel behavior (e.g. leftover .tar.gz sdist artifacts in the same directory). net: -3 to -5 lines possible.