NVIDIA Green Contexts: Explicit GPU Resource Partitioning for Concurrent Workloads

  • PublicPublic
  • 07 Oct, 2026
NVIDIA Green Contexts: Explicit GPU Resource Partitioning for Concurrent Workloads

Modern GPU applications frequently run multiple independent components within a single process. For example, latency-sensitive operators run alongside throughput-oriented background kernels. These components often share a single GPU, leading to unpredictable interference and resource contention. Traditional CUDA contexts were designed for an era when GPUs handled a single dominant workload. As a result, they are heavyweight and ill-suited for fine-grained resource partitioning. They incur significant hardware context-switch overhead and lack the mechanisms to explicitly define how execution resources are divided among concurrent tasks.

Green contexts solve this by allowing applications to explicitly select a subset of GPU execution resources and target work directly to those resources. Available in the Driver API since CUDA 12.4, they are now accessible through the Runtime API starting with CUDA 13.1. This feature enables developers to define where work runs and how resources are divided within a process. The primary use case is SM partitioning, where specific Streaming Multiprocessors are assigned to a green context. This allows multiple workloads to run concurrently without competing for the same compute units.

Beyond SM allocation, green contexts can provision workqueue resources. In the traditional model, independent stream-ordered workloads might map to the same underlying workqueues, causing unintended serialization even when sufficient execution resources are available. By explicitly provisioning workqueues, green contexts allow applications to express expected concurrency and eliminate false dependencies. These contexts are lightweight to create and destroy, and their lifecycle does not implicitly synchronize unrelated GPU work. This provides a more explicit programming model than relying on implicit device state.

The programming model shifts from implicit to explicit targeting. Historically, applications used cudaSetDevice() and created streams whose execution target was inferred from the current thread-local device state. With green contexts, represented by the cudaExecutionContext_t type, developers create a context for a chosen resource set and then create streams from it using cudaExecutionCtxStreamCreate(). This ensures that work submitted to those streams is strictly associated with the green context's resources. For applications targeting the full device, the traditional model remains available via cudaDeviceGetExecutionCtx(), ensuring backward compatibility and incremental adoption.

A canonical use case is overlapping communication with GEMM kernels in distributed training or handling latency-sensitive operators in AI sensor platforms. Stream priority alone is insufficient when bulk kernels occupy all SMs, as the scheduler cannot preempt executing blocks. Green contexts dedicate specific SMs to critical work, bypassing the wait for bulk blocks to drain. On an NVIDIA Blackwell GPU with 148 SMs, testing showed that while stream priority offers a 27x improvement over equal-priority streams, green contexts provide an additional 20x latency reduction by ensuring dedicated resources for the critical kernel.

The implementation involves querying device resources, splitting SMs into critical and remaining groups, and packing each with specific workqueue configurations. Developers use cudaDevSmResourceSplit to carve out resources and cudaDevResourceGenerateDesc to create descriptors. Green contexts are opt-in and additive, allowing existing applications to continue using the full device while adopting finer control where needed. This approach is particularly useful for achieving desired concurrency or latency targets in complex workflows, offering a robust mechanism for managing GPU resource sharing in modern, multi-component applications.