Skip to content

GPU pools can't scale to zero on clouds using vanilla cluster-autoscaler #345

Description

@negz

What problem are you facing?

#306 fixed GPU pools cold-starting from zero on GKE. That fix works because GKE's autoscaler has custom support for DRA: it recognises the cloud.google.com/gke-nvidia-gpu-dra-driver=true node label and models the ResourceSlices a not-yet-existing node would publish, so it can tell a scaled-to-zero pool would satisfy a GPU ResourceClaim. Nebius's autoscaler appears to have similar custom support.

The upstream Kubernetes cluster-autoscaler doesn't have this in general. Scale-from-zero relies on each cloud provider's TemplateNodeInfo() synthesising the node a pool would create, and only the cluster-api provider models DRA ResourceSlices there (via the capacity.cluster-autoscaler.kubernetes.io/dra-driver annotation added in kubernetes/autoscaler#7804). The AWS provider passes nil slices, so an EKS GPU pool at zero presents no DRA capacity to the autoscaler and never wakes up. Scaling 1→N works because the autoscaler templates off a real node's live slices; 0→1 doesn't. This is tracked upstream, broadly, in kubernetes/autoscaler#7799.

We can't sidestep this on EKS with Karpenter or EKS Auto Mode, because neither supports DRA yet, which is why we run the vanilla cluster-autoscaler. So any Modelplane cluster on a provider without custom autoscaler DRA support — EKS today, and any self-managed cluster running the stock cluster-autoscaler — can't scale GPU pools to zero.

The immediate user-facing problem: the InferenceCluster API accepts minNodeCount: 0 on GPU pools regardless of provider, but on these providers a pool set to zero silently stays pending with cannot allocate all claims instead of cold-starting. Nothing tells the user it won't work.

How could Modelplane help solve your problem?

Mostly this issue is to track the limitation.

In the meantime, we could stop the API silently accepting a configuration that won't work, e.g. reject or warn on minNodeCount: 0 for GPU pools on providers we know don't support it, and document per-provider support.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    Status
    Backlog

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions