Optimizing AI Inference Through Full-Stack Memory Codesign
Public- 06 Nov, 2026

The rapid expansion of large language models, extended context windows, and agentic applications is intensifying pressure on GPU memory resources. As supply tightens and prices rise, the industry faces a critical challenge: how to process more data without simply adding expensive hardware. NVIDIA is tackling this by moving beyond the traditional approach of packing more High Bandwidth Memory (HBM) onto chips. Instead, the company employs a strategy of full-stack codesign, where GPUs, CPUs, networking, storage, and software are engineered in tandem. This holistic approach aims to extract maximum useful work from every byte of memory across the entire AI factory infrastructure, rather than relying solely on raw capacity increases.
The core of this strategy lies in managing the memory and storage hierarchy effectively. In an AI factory, HBM serves as the fastest but most expensive and contested tier, holding critical data such as model weights and the Key-Value (KV) cache. Below this tier sit CPU memory, local SSDs, and shared storage, which offer greater capacity at lower cost but with reduced speed. The efficiency of the system depends on how well data is placed and moved between these tiers. By shrinking the footprint of model weights, offloading inactive KV cache contexts to cheaper storage, and compacting Retrieval-Augmented Generation (RAG) embeddings, NVIDIA ensures that the limited HBM space is reserved for active, high-priority operations. This prevents the "memory wall" phenomenon, where compute units sit idle waiting for data because bandwidth is saturated, thereby maintaining low latency for concurrent users.

The impact of this codesign approach is evident in recent platform performance metrics. From the Blackwell Ultra generation to the upcoming Vera Rubin platform, HBM capacity per GPU has increased by only about 3%. However, the efficiency gains are substantial. On the SemiAnalysis AgentX benchmark, agentic throughput per dollar has risen by up to 30x. Specifically, the NVIDIA GB300 NVL72 delivers up to 10x higher throughput per dollar compared to the H200 platform at 100 tokens per second per user, despite a doubling of HBM capacity. The Vera Rubin NVL72 is projected to deliver up to 30x higher throughput per dollar than the GB300 NVL72 at 125 tokens per second per user, with HBM capacity increasing by just 3%. This demonstrates that optimizing data flow and storage placement yields far greater returns than simply expanding memory capacity.
For operators of AI factories, the primary metric is the number of useful tokens delivered within a specific power budget while meeting strict response-time requirements. Increasing tokens per second per watt translates directly into higher revenue from the same power infrastructure and a lower cost per token. By relieving pressure on HBM from model weights, idle context, and RAG data, full-stack codesign ensures that memory resources are used for active inference rather than passive storage. This method allows systems to handle trillions of parameters and complex agentic trajectories, which can involve processing up to 3 million tokens, without being bottlenecked by memory bandwidth. The result is a more scalable and cost-effective infrastructure capable of supporting the next generation of AI applications.

