NVIDIA Snapshot Cuts GPU Workload Startup to Seconds

  • PublicPublic
  • 11 Nov, 2026
NVIDIA Snapshot Cuts GPU Workload Startup to Seconds

NVIDIA has introduced Snapshot, an open-source component of the NVIDIA DSX AI Factory Platform that makes initialized GPU workloads reusable across inference replicas. The problem it targets is familiar to anyone running LLM serving: when demand spikes or a replica fails, available GPUs do not immediately become serving capacity, because initializing an LLM workload - a cold start - can take minutes. Snapshot is a checkpoint and restore system for GPU workloads on Kubernetes. It captures a fully initialized workload, including CPU and GPU memory, and restores that state on compatible infrastructure, bringing inference capacity online in seconds rather than minutes. Infrastructure provisioning and container image loading remain outside the measured restore path.

The tool exposes two operations for GPU Pods, both driven by Kubernetes custom resources. To checkpoint a running Pod, you create a PodSnapshot that names it; to start a Pod from a template and checkpoint it as soon as it is ready, you create a SnapshotJob instead. Restoring means adding the annotation nvidia.com/restore-from to a new Pod and naming the PodSnapshot to restore from. The workload itself only needs a small handshake to pause before the checkpoint and resume after the restore, and the repository ships ready-made examples for vLLM, SGLang, and NVIDIA TensorRT LLM. Snapshot can checkpoint and restore compatible workloads spanning multiple GPUs on a single node as a single artifact across ranks, which supports fast startup for large mixture-of-experts models using tensor parallelism, data and expert parallelism, and decode context parallelism, provided the workload meets the checkpoint/restore and runtime compatibility requirements.

Figure 1. Cold start sequence: machine provisioning, container image loading, inference engine startup, loading model weights into GPU memory, torch.compile compilation, and NVIDIA CUDA graph capture
Figure 1. Cold start sequence: machine provisioning, container image loading, inference engine startup, loading model weights into GPU memory, torch.compile compilation, and NVIDIA CUDA graph capture

Benchmarks cover models from 600 million parameters up to MoE models with 1 TB checkpoints on 8 GPUs, tested on the same hardware and software stack under two strategies: cold start versus Snapshot restore. Across all tested configurations, Snapshot brings a replica back in under 45 seconds, 8x to 36x faster than a cold start. Per-engine restore ranges are as follows: vLLM at 4.7 to 39.6 seconds, SGLang at 3.6 to 42.7 seconds, and TensorRT LLM at 7.79 to 33.55 seconds. Both measurements exclude pulling the container image and setting up the container runtime, and the KV cache release is triggered before every checkpoint, so it is not part of the artifact. Checkpoint time and artifact size are reported as a one-time cost per model and configuration.

Figure 2. vLLM: cold start versus Snapshot for each model and each tested parallelism configuration. Snapshot brings up each replica in 4.7-39.6 seconds
Figure 2. vLLM: cold start versus Snapshot for each model and each tested parallelism configuration. Snapshot brings up each replica in 4.7-39.6 seconds

The restore path itself breaks down into agent setup, CRIU restore, CUDA restore, and wake/remap, and the cold-start breakdown shows which initialization phases a restore skips. Snapshot was originally developed for NVIDIA Dynamo inference workers, but it is designed for integration by other inference engines, schedulers, controllers, and platforms. Benchmarking tools and reproduction instructions live in the repository, alongside installation instructions and usage guides, and the Dynamo user guide covers restoring workers. Design partners have also evaluated Snapshot on their own workloads.

Figure 3. SGLang: cold start versus Snapshot for each model and each tested parallelism configuration. Snapshot brings each replica online in 3.6-42.7 seconds
Figure 3. SGLang: cold start versus Snapshot for each model and each tested parallelism configuration. Snapshot brings each replica online in 3.6-42.7 seconds
Figure 4. TensorRT LLM (TRT-LLM): cold start versus Snapshot for each model and each tested parallelism configuration. Snapshot brings each replica online in 7.79-33.55 seconds
Figure 4. TensorRT LLM (TRT-LLM): cold start versus Snapshot for each model and each tested parallelism configuration. Snapshot brings each replica online in 7.79-33.55 seconds
Figure 5. Snapshot restore time broken down by phase on vLLM, shown as an example. The number above each bar is the total restore time in seconds
Figure 5. Snapshot restore time broken down by phase on vLLM, shown as an example. The number above each bar is the total restore time in seconds