Summary
Design a same-node warm tier for nvsnap: a tier-0 hot layer above the existing
cross-node tiers (peer cascade / shared-volume L2 / blobstore) that keeps a
model resident on the node so a same-node restart or scale-up is near-instant.
This is explicitly same-node only; it complements the cross-node fan-out path,
it does not replace it.
Motivation
Cross-node restore is served by the shared-volume and cascade tiers. The
remaining floor on those paths is the per-process CUDA context re-creation
(module/kernel reload, the cuInit cost), which storage speedups cannot remove.
For workloads whose recovery pattern is same-node (crash recovery, rolling
restart, same-node autoscale burst), a resident warm donor removes that floor
entirely.
External corroboration: SGLang is adding a persistent weight-cache daemon
(sgl-project/sglang#27139) and NVIDIA GMS holds weights resident via the CUDA
VMM API; both target the same same-node fast-restart problem.
Two flavors
A. Frozen donor (full checkpoint resident)
Keep a checkpointed process live and frozen on the node: CUDA context, compiled
kernels, captured graphs, and weights all resident. A same-node restore thaws or
clones the donor instead of cold-restoring. Skips the entire restore path,
including the CUDA context re-creation floor. This is the differentiated option
because nvsnap already has the frozen-process machinery.
- Pro: fastest possible restart; nothing re-materialized.
- Con: pins a full GPU context + the model's VRAM per held donor.
B. Shared weight cache
A persistent process holds only the weights resident in VRAM and maps them into
restarting engines via CUDA IPC. Cheaper than a full donor, and it has a second
payoff: multiple same-node replicas can share one weight copy instead of each
holding its own, so it is also a VRAM-density win, not only a latency win.
- Pro: lower VRAM cost; enables replica packing per node.
- Con: skips weight load only; context/graph re-init still runs. Overlaps
SGLang/GMS work, so integrate rather than reinvent where possible.
Open design decisions
- Which flavor first (A is the nvsnap differentiator; B is cheaper and denser).
- Single-consumer thaw vs multi-consumer clone of a donor.
- VRAM budget: how many hot models may be held resident per node.
- Eviction policy when GPU memory is contended.
- Interaction with the existing tiers: warm tier is tried first on same-node
restore, falling through to L1/L2 when no donor is resident.
Non-goals
- Cross-node restore (served by the existing shared-volume and cascade tiers;
CUDA IPC and a resident donor are same-node by nature).
- Replacing the cross-node fan-out path.
References
Summary
Design a same-node warm tier for nvsnap: a tier-0 hot layer above the existing
cross-node tiers (peer cascade / shared-volume L2 / blobstore) that keeps a
model resident on the node so a same-node restart or scale-up is near-instant.
This is explicitly same-node only; it complements the cross-node fan-out path,
it does not replace it.
Motivation
Cross-node restore is served by the shared-volume and cascade tiers. The
remaining floor on those paths is the per-process CUDA context re-creation
(module/kernel reload, the cuInit cost), which storage speedups cannot remove.
For workloads whose recovery pattern is same-node (crash recovery, rolling
restart, same-node autoscale burst), a resident warm donor removes that floor
entirely.
External corroboration: SGLang is adding a persistent weight-cache daemon
(sgl-project/sglang#27139) and NVIDIA GMS holds weights resident via the CUDA
VMM API; both target the same same-node fast-restart problem.
Two flavors
A. Frozen donor (full checkpoint resident)
Keep a checkpointed process live and frozen on the node: CUDA context, compiled
kernels, captured graphs, and weights all resident. A same-node restore thaws or
clones the donor instead of cold-restoring. Skips the entire restore path,
including the CUDA context re-creation floor. This is the differentiated option
because nvsnap already has the frozen-process machinery.
B. Shared weight cache
A persistent process holds only the weights resident in VRAM and maps them into
restarting engines via CUDA IPC. Cheaper than a full donor, and it has a second
payoff: multiple same-node replicas can share one weight copy instead of each
holding its own, so it is also a VRAM-density win, not only a latency win.
SGLang/GMS work, so integrate rather than reinvent where possible.
Open design decisions
restore, falling through to L1/L2 when no donor is resident.
Non-goals
CUDA IPC and a resident donor are same-node by nature).
References
CUDA IPC)