GPU Memory Limits per Workload in Kubernetes with CUDA 13.4 and KAI Scheduler
Public- 28 Oct, 2026

AI workloads often require only a fraction of a GPU, yet assigning entire devices leads to idle capacity and inefficient infrastructure use. NVIDIA GPUs support various sharing mechanisms, including Multi-instance GPU (MIG) for hardware isolation, time-slicing for sequential process execution, and Multi-Process Service (MPS) for concurrent execution in separate address spaces. While MPS allows compute sharing, it previously lacked strict memory boundaries. CUDA 13.4 addresses this gap by introducing a new time-sharing mode that enforces memory limits transparently to the application, ensuring that each workload operates within defined memory constraints without requiring code changes.

Implementing these capabilities in a Kubernetes cluster requires node-level coordination independent of the scheduler. The open-source gpu-fractioning operator, hosted under the KAI Scheduler organization on GitHub, runs alongside the NVIDIA GPU Operator to handle this configuration. It performs three critical functions: reading pod annotations to determine memory and compute requirements, configuring the assigned GPUs to enforce these limits, and managing the transition between different sharing modes. This operator ensures that the hardware respects the logical boundaries defined by the cluster scheduler, bridging the gap between high-level scheduling decisions and low-level GPU execution.
The workflow begins with developers specifying resource requirements and modes via pod YAML annotations. The scheduler then selects appropriate GPUs, records the memory allocation and compute-sharing mode in per-container pod annotations, and assigns the workload. The gpu-fractioning operator reads these annotations and configures the GPUs accordingly. Any Kubernetes scheduler capable of writing these annotations can enforce GPU fractions, provided it adheres to two rules: selecting GPUs with sufficient free memory and avoiding the mixing of compute modes on a single device. KAI Scheduler is the first to integrate this feature, introducing a GPU-sharing mode called NvFractions starting with version v0.18.0.

KAI Scheduler's NvFractions mode allows administrators to define fractional workloads using four key annotations that address memory requests, growth limits, compute sharing modes, and device count. Memory requests can be specified as absolute sizes using standard Kubernetes units or as portions of a GPU using the gpu-fraction annotation. A new capability allows workloads to declare a guaranteed minimum request and a maximum burst limit, which is optional and defaults to the request value if omitted. Portion limits must exceed the gpu-fraction and remain below 1. Compute sharing is now explicit and set per container, allowing choices between sm-sharing and time-slicing.

A shared GPU operates in only one compute mode at a time, so KAI Scheduler prevents placing workloads with different modes on the same GPU, though a single node can host both modes across different devices. For multi-GPU setups, the gpu-fraction-num-devices annotation specifies how many devices the workload spans, with the memory request applying to each device individually. For example, a 40Gi request with four devices allocates 40Gi on each GPU. This setup is ideal for inference servers that need guaranteed memory per GPU and can burst when neighbors are idle, utilizing sm-sharing for trusted workloads that run concurrently in a single error domain.

Conversely, time-slicing is suitable for development or batch jobs such as notebooks and small fine-tuning runs, where isolation is preferred. In this mode, each workload gets its own error domain, meaning that a crash in one job does not affect the others. A workload can request half of two GPUs with a limit to grow to 80% when memory is available. Running nvidia-smi inside the container will display the fractional memory share for each assigned GPU. Both KAI Scheduler and the gpu-fractioning operator are available on GitHub under the KAI Scheduler organization, with detailed installation and configuration guides for all sharing mechanisms.
