docs(prod): warn about $(nproc) on Kubernetes; add Downward API guidance#28
Open
Anai-Guo wants to merge 1 commit intoBerriAI:mainfrom
Open
docs(prod): warn about $(nproc) on Kubernetes; add Downward API guidance#28Anai-Guo wants to merge 1 commit intoBerriAI:mainfrom
Anai-Guo wants to merge 1 commit intoBerriAI:mainfrom
Conversation
The $(nproc) recommendation in section 3 is incorrect on Kubernetes
because it returns the host node's CPU count, not the pod's cgroup
CPU request/limit. On a 16-vCPU node with a pod that has
requests.cpu=4 and limits.cpu=8, $(nproc) returns 16, spawning
2-4x more Uvicorn workers than the pod can actually run. The result
is CPU oversubscription, worse latency, and the opposite of the
stated goal.
Replace the K8s nproc examples with:
- A warning explaining the cgroup mismatch with a concrete example.
- A Downward API recipe (recommended) that exposes requests.cpu
as $CPU_REQUEST in the container env, then uses
--num_workers ${CPU_REQUEST:-4} in the CMD.
- A hardcoded --num_workers fallback for users who prefer it.
- A bare-metal/VM section that keeps the original nproc CMD,
scoped to environments where the process actually has access
to all host CPUs (no cgroup limit).
The two follow-up shell blocks (--max_requests_before_restart
and --run_gunicorn variants) are updated to use the same
${CPU_REQUEST:-4} pattern so they stay consistent with the
recommended K8s setup.
Fixes BerriAI/litellm#26620
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes BerriAI/litellm#26620.
The "Match Uvicorn Workers to CPU Count" section in the production docs recommends
--num_workers $(nproc)for Kubernetes deployments.nprocis incorrect on Kubernetes — it reports the host node's CPU count, not the pod's cgroup CPU request or limit.Concrete failure mode (from #26620):
e2-standard-16(16 vCPU)requests.cpu4limits.cpu8$(nproc)returnsThis causes CPU oversubscription, context-switching overhead, and worse latency — the opposite of what the section is trying to achieve.
Fix
Restructure section 3 of
docs/proxy/prod.md:requests.cpuas$CPU_REQUESTin the container env, then uses--num_workers ${CPU_REQUEST:-4}in theCMD.--num_workers 4for users who prefer not to wire up the Downward API.$(nproc)CMD, scoped to environments where the process actually has access to all host CPUs.The two follow-up shell blocks (
--max_requests_before_restartand--run_gunicornvariants) are updated to use the same${CPU_REQUEST:-4}pattern so they stay consistent with the recommended K8s setup.Files changed
docs/proxy/prod.md— section 3 onlyWhy this is correct
The Downward API approach is the standard Kubernetes idiom for surfacing pod-level resource requests/limits to the container at runtime. It's documented at https://kubernetes.io/docs/tasks/inject-data-application/environment-variable-expose-pod-information/.
The
${CPU_REQUEST:-4}default keeps the container bootable in non-K8s environments (local Docker, etc.) where the env var won't be injected.🤖 Generated with Claude Code