FEAT: production-ready Docker Compose deployment with offline private PyPI profile#5174
FEAT: production-ready Docker Compose deployment with offline private PyPI profile#5174OliverBryant wants to merge 4 commits into
Conversation
… PyPI profile Rework the standalone docker-compose.yml into a first-class deployment: - parameterize image/port/model source/shm/log level/cache dirs via .env - add /status healthcheck and unless-stopped restart policy - persist caches in named volumes by default, overridable to host paths - drop the deprecated 'version' field Add an 'offline' compose profile that starts a private PyPI server (pypiserver backed by ./wheels) for air-gapped hosts. The offline config routes all runtime virtual-env installs to the private index through two complementary channels: a mounted pip.conf consumed by the pip-config inheritance (which also blocks the hardcoded vLLM/SGLang public extra indexes), and UV_* env vars covering uv invocations without index flags. Add docker-compose.cpu.yml override for CPU-only hosts, offline.env and .env templates, official documentation page (with zh_CN translation) and toctree entry.
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a robust Docker Compose deployment setup for Xinference, supporting both online and offline/air-gapped environments. It includes a private PyPI server configuration to serve local Python wheels, a CPU-only override file, environment templates, and comprehensive documentation in English and Chinese. The review feedback highlights two issues in the offline configuration: the pypiserver healthcheck endpoint in docker-compose.yml needs to be corrected to ~health instead of /health to prevent 404 errors, and the UV_INSECURE_HOST variable in offline.env.example should specify only the hostname without the port to ensure proper insecure connection handling by uv.
…erver on Linux The pypiserver image entrypoint refuses to start unless its internal user (UID 9898) has read/write/execute permission bits on /data/packages, even though the compose file mounts the wheels directory read-only. Docker Desktop on macOS masks this, but on Linux hosts the offline profile crash-loops without a chmod. Found during verification on a real GPU host.
GPU verification results (real hardware)Completed end-to-end verification on a Linux x86_64 host with 2× RTX 3090 Ti (driver 580, Docker 29.6.1, Compose v5.3.1), using the official Online GPU deployment
Offline profile on GPU
Issue found & fixed (commit 4601926): on Linux hosts the pypiserver container crash-loops unless the mounted wheels directory grants rwx permission bits to its internal user (UID 9898) — its entrypoint enforces this even though the volume is mounted read-only. Docker Desktop on macOS masks the problem. Documented Unrelated observation for a potential follow-up issue: when a vLLM engine fails during load (e.g. insufficient free VRAM), |
The offline profile redirects all downloads to local sources but does not by itself prevent containers from reaching the Internet when the host has connectivity. docker-compose.airgap.yml moves xinference and the private PyPI server onto an internal Docker network with no external routing, and adds a minimal socat TCP gateway to keep the API port published on the host (Docker does not publish ports of internal-only networks). The gateway only relays inbound traffic and cannot serve as an egress path. Verified on a Linux GPU host: external requests (pypi.org, pytorch.org, raw TCP to 8.8.8.8) are blocked from inside the container while the private index stays reachable, and vLLM inference through the gateway works end to end.
Added: enforced network isolation (air-gap override)The offline profile alone redirects downloads to local sources but does not prevent egress when the host happens to have connectivity. Commit 176dc3a adds
docker compose --profile offline -f docker-compose.yml -f docker-compose.airgap.yml up -dVerified on the Linux GPU host, from inside the xinference container:
Docs updated with an "Enforced network isolation (optional)" subsection (incl. zh_CN translation) covering usage, the extra |
qinxuye
left a comment
qinxuye
left a comment
There was a problem hiding this comment.
One blocking issue remains in the true offline installation path.
…nfigured install paths When the package list carries a #system_torch#-style marker and the installed torch has a CUDA suffix, _prepare_virtual_env force-prepended https://download.pytorch.org/whl/cu* to extra_index_url even when an extra index had been configured explicitly via the model spec or inherited pip config. uv treats an unreachable extra index as fatal rather than falling back, so in air-gapped deployments model launches failed even with all wheels present on the private index. Only inject the auto-detected CUDA wheel URL when the extra index is unset or was filled in by engine defaults; an explicitly configured index now stays authoritative. Default online behavior is unchanged. Add regression tests covering both the suppression and the default injection paths for #system_torch#.
3 participants
What do these changes do?
The existing
docker-compose.yml/docker-compose-distributed.ymlwere added as bare examples and are not referenced anywhere in the documentation. This PR turns the standalone compose file into a first-class, documented deployment option, with built-in support for offline / air-gapped environments.Reworked
docker-compose.yml${VAR:-default}and an.env.exampletemplate (image tag, port, model source, shm size, log level, cache locations)./statushealthcheck (viapython3+urllib, since not every image variant shipscurl) andrestart: unless-stopped.versionfield and the stale commented-out blocks.New:
offlineprofile with a private PyPI serverdocker compose --profile offline up -dadditionally starts a pypiserver service backed by a local./wheelsdirectory. The offline configuration (offline.env+ mountedpip.conf) routes every runtime package install of the model virtual-env machinery to the private index through two complementary channels:pip.conffeeds theinherit_pip_configmechanism (get_pip_config_args()only readsglobal.*keys frompip config list, so env vars alone are not picked up). A non-emptyextra-index-urlalso prevents the hardcoded public vLLM/SGLang extra indexes from being injected (apply_engine_virtualenv_settingsonly fills them when unset).UV_DEFAULT_INDEX/UV_INDEX_URL/UV_INSECURE_HOSTcoveruvinvocations that do not carry index flags, e.g. the dependency-resolution dry-run used byXINFERENCE_VIRTUAL_ENV_SKIP_INSTALLED.Model weights are handled by
HF_HUB_OFFLINE=1/TRANSFORMERS_OFFLINE=1plus the mounted cache volumes. Users who prefer fully pre-provisioned images can setXINFERENCE_ENABLE_VIRTUAL_ENV=0instead.Other additions
docker-compose.cpu.yml: 4-line override for CPU-only hosts (-cpuimage +deploy: !reset null).getting_started/using_docker_compose.rst(added to the toctree) covering quick start, configuration, CPU-only, and a step-by-step offline guide, with zh_CN translation..gitignore: keep thewheels/mount point (.gitkeep), ignore user-localoffline.env.Online users are unaffected:
docker compose up -dworks with zero configuration, and the pypiserver service only starts when theofflineprofile is enabled.Verification
docker compose configvalidated for all three views (default /--profile offline/ CPU override).pip config listexposes theglobal.*keys; requesting a version absent from the local index fails without falling back to pypi.org, provinguvresolves against the private index only.qwen30.6B (transformers engine) through the compose stack on a slim image without torch/transformers — the runtime virtual-env installed 34 packages and the model came up successfully.make htmlandmake html_zh_cnbuild without new warnings.Notes
env_file: required: false,!reset); stated in the docs and file comments.