Why Nvidia’s Next AI Battle Is About Tokens per Watt 

As hyperscalers move from building AI infrastructure to monetizing it, tokens per watt helps to reflect if revenue is scaling and if profitability is improving. Tokens are the revenue side of the inference equation, while power consumption (watts) represents the associated costs. The more tokens a hyperscaler can generate from each unit of power, the more revenue it can drive without a proportional increase in infrastructure costs. 

Nvidia’s offload-engine approach is designed to address the problem of accelerators sitting idle, waiting for KV cache, memory or data movement. In other words, offload engines can increase tokens per watt by improving utilization and by increasing the amount of inference revenue generated from the same power footprint. The result is improved margins as revenue generation scales faster than costs. 

It’s become quite clear that power is no longer an abundant resource. As new capacity becomes harder to secure, inference growth will be constrained by how efficiently hyperscalers use the power and accelerators they already have. 

When it comes to improving tokens per watt, memory is the biggest constraint, rather than the raw power offered by GPUs or XPUs. If Nvidia’s GPUs or Big Tech’s XPUs are not supplied with sufficient memory to constantly perform inference tasks, tokens per watt falls as these accelerators are underutilized. 

To solve this issue, offload engines are being actively designed into the rack-scale architectures of AI infrastructure leaders like Nvidia. Meanwhile, other players are building open-standard offload engine solutions through Compute Express Link (CXL). 

In this analysis, we break down why XPU underutilization matters, how offload engines improve tokens per watt, and which architectures may create the strongest opportunity for investors. 

Nvidia’s Offload Engines and the KV Cache: Solving AI’s Inference Memory Bottleneck  

While Nvidia has already been embedding ‘offload engines’ such as its SmartNICs and DPUs to offload data processing from host CPUs, the latest offload engines target GPU memory to accelerate inference. These solutions look to address one of the main causes of XPU underutilization: the growing size of model KV caches, which consume the majority of their working memory during inference. 

In an ideal world, the KV cache would be fully stored in the memory tier that XPUs can access most quickly, which is high-bandwidth memory (HBM). This minimizes latency, optimizes XPU utilization, and maximizes tokens per watt. 

mid

However, as models and workflows increase in complexity, such as for multi-step reasoning or agentic deployments, KV cache can exceed HBM memory capacity. In this case, the KV cache is either offloaded to slower tiers of the memory stack, introducing latency, or recomputed at each request, wasting resources on work that has already been performed – both key factors in reducing XPU utilization and reducing tokens per watt. 

Generally speaking, offload engines aim to keep XPUs fed with KV cache more efficiently, improving XPU utilization and tokens per watt. 

The Memory Wall: AI Compute Gains Are Outpacing Memory 

A key reason why offload engines can be important solutions is due to the ‘memory wall’. Over the 20 years ending in 2023, accelerator processing power (Peak FLOPs) increased by 3X every two years. However, memory bandwidth has increased by only 1.6X every two years, and memory capacity has increased by only 2X every two years. 

With this, the theoretical speed at which XPUs could produce tokens is growing faster than the rate they can in practice, because memory is not feeding them data fast enough. Thus, memory-driven XPU underutilization is increasing. 

PChart showing the increase in compute peak FLOPs versus memory bandwidth over time. From 2003 to 2023, peak FLOPs scales dramatically faster than memory bandwidth, widening the gap between the two and increasing the size of the memory wall. Source: Astera Labs 

We can continue to see this play out with Rubin. Nvidia notes that Rubin’s inference processing power is 5X higher than that of Blackwell. However, as Nvidia moves from HBM3E in Blackwell to HBM4 in Rubin, overall HBM bandwidth only increases by 2.8X while capacity rises just 1.5X. Although HBM capacity and bandwidth are increasing, the memory wall is firmly intact.  

Amid this, one of the core issues in addressing the KV cache bottleneck is that HBM cannot be added independently of accelerators, as they are packaged together. While adding more XPUs increases HBM capacity and allows operators to serve more concurrent requests, doing so also means that more XPUs go underutilized as the ratio of processing power to memory does not change. 

Additionally, due to the intense memory shortage, HBM has become an increasingly large chunk of AI accelerator bill of materials (BOM). In the B200, HBM represents approximately 52% of total BOM. In Rubin, this is expected to increase to 62%. As this percentage rises, hyperscalers are paying more for memory, even though the pace of computing power growth continues to exceed memory capacity and bandwidth growth. In other words, hyperscalers are forced to spend a higher percentage of their capex on memory even as memory-driven XPU underutilization further deteriorates. 

In turn, hyperscalers need to find a way to use memory more efficiently and increase memory capacity independent of HBM. This helps alleviate KV cache bottlenecks, sidestep the increasing costs of scaling memory through XPU purchases, and expand revenue and margins amid power constraints. This is the exact value that offload engine solutions like Nvidia’s CMX intend to provide. 

I/O Fund Premium members recently received an 18-page deep dive on the investable shifts in the networking stack and what companies are positioned to benefit. Sign up here.  

Tokens Per Watt: Scaling Revenue and Margins 

For hyperscalers, improving tokens per watt is important from three key perspectives; generating higher inference revenue, increasing inference margins, and better allocating capex spend. 

Why Offload Engines Can Increase Revenue Amid Power Constraints 

While power is an inference cost, it is also a required input for token and revenue generation. However, power availability is a key AI bottleneck. Notably, ERCOT is tracking more than 438 GW of large load interconnection requests, with nearly 89%, or approximately 390 GW coming from data centers alone. Meanwhile, only 23 GW of capacity were added between 2024-2025, demonstrating the massive gap between data center energy demand and the ability of the grid to meet that demand quickly. 

Additionally, GPU power consumption is increasing with each new generation of chips. H100 power consumption was approximately 700W per chip. In Blackwell and Blackwell Ultra, this approximately doubles to 1,200W-1,400W per chip. Rubin will come with another substantial increase to around 2,300W per chip. As each newer chip consumes more power and has a higher upfront cost, the penalty of underutilization rises. 

With such a massive backlog of power capacity demand and rising power consumption per chip, hyperscalers that generate more tokens within the fixed power envelope they have already secured pull one of the most immediate levers to increasing inference revenue. This creates a strong incentive for hyperscalers to increase token throughput using offload engine solutions. 

Nvidia’s data on the Groq 3 LPX rack provides a clear example of how offload engine solutions can increase inference revenue generation by improving tokens per watt. Compared to a GB200 NVL72, Nvidia says that a Vera Rubin NVL72 rack combined with a Groq 3 LPX rack can drive up to 35X higher token throughput per MW. 

Chart showing the improvements in tokens per second per MW across GPU generations. Near 450 tokens per second per user, the Vera Rubin NVL72 + the Groq 3 LPX rack can deliver a 35X increase in tokens per second per MW versus the GB NVL72. Source: Nvidia 

Tying back directly to revenue, Nvidia estimates that Vera Rubin + LPX deployments give AI factories an up to 10X revenue generation opportunity versus Blackwell for trillion-parameter models. 

As hyperscalers that adopt offload engine solutions can gain an upper hand in increasing tokens per watt, those that do not forfeit this benefit, creating competitive pressure to incorporate these technologies. 

Offload Engines Can Boost Hyperscaler Margins 

Margin improvement is another key aspect of increasing tokens per watt. As tokens represent the revenue side of the profitability equation, producing more with the same power envelope directly increases inference margins. With offload engines demonstrating the ability to increase tokens per watt, hyperscalers can benefit from higher inference profitability. 

Additionally, other methods of increasing data center energy efficiency are being exhausted. Power usage effectiveness (PUE) is a metric that helps highlight this. PUE divides the total energy a data center consumes by the energy computing resources consume. It is a proxy for measuring the energy efficiency of non-computing resources, like cooling and power distribution systems. 

PUEs that are closest to 1 show that the majority of power is ultimately delivered to computing resources and can contribute to token generation. Across the four leading U.S. hyperscalers, PUEs have dropped very close to 1. Overall, Google, Meta, Microsoft, and AWS have recently reported PUEs ranging from just 1.09 to 1.17. 

This means the available runway to improve AI data center power efficiency through better cooling and distribution systems is becoming somewhat limited. This limits the ability to improve margins by simply reducing energy costs at the non-compute level, shifting the emphasis to driving higher token output at a similar level of power consumption. 

Optimizing Capex Allocation Through Offload Engine Memory Scaling 

Offload engines can help hyperscalers better allocate their capex spending and achieve an improved overall return on AI investment. Rather than adding costly XPUs to increase memory capacity that may otherwise be unnecessary, hyperscalers can directly address XPU underutilization by allocating that spend toward lower-cost offload solutions. 

For example, Marvell recently noted that server DDR5 pricing has risen as high as $40/GB or $40,000 for 1TB. Meanwhile, recent estimates place the price of a B200 chip at $40,000. With 192 GB of HBM each, the total cost to acquire 1 TB of HBM would be more than $200,000, or more than 5X higher than equivalent capacity of DDR5. While DRAM-based CXL solutions would come with additional costs and a vendor markup, it is fair to say the cost of these solutions would still be far lower than purchasing GPUs. This becomes increasingly relevant as HBM consumes a higher percentage of XPU BOM, yet doing so does not ameliorate XPU underutilization. 

Conclusion: 

Tokens per watt is becoming the metric that will help distinguish inference winners from those that fall behind on monetizing the extraordinary, high costs of the AI buildout. It's clear that memory and power will be constrained for years to come. In response, hyperscalers will turn their focus toward higher utilization, by keeping GPUs and XPUs fed with KV cache, and by turning stranded memory into tokens. The goal is to scale revenue faster, and more profitably, using the same power envelope. 

What we will explore in our follow-up article coming out on Sunday, is which architectures are poised to capture the opportunity. While Nvidia pursues a proprietary path with CMX, there is an open ecosystem forming around CXL, and the two are prepared to capture AI capex as it sets to reach $1 trillion by 2027. 

Sign up for our newsletter to receive Part 2: Nvidia, CXL, and the Battle to Improve AI Inference Economics. 

The I/O Fund has crushed the AI networking trade with over 50% of our portfolio allocation to this booming, yet little understood trend. Our cumulative return of 326% would place us as #1 if we were a hedge fund and #3 if we were a tech ETF over a 5-year period. Notably, this does not include our YTD 2026 outperformance.  

➡️ Sign up now to see the portfolio, allocations, and real-time trade alerts of a top-performing team in AI stocks.
Sign up now

Please note: The I/O Fund conducts research and draws conclusions for the company’s portfolio. We then share that information with our readers and offer real-time trade notifications. This is not a guarantee of a stock’s performance and it is not financial advice. Please consult your personal financial advisor before buying any stock in the companies mentioned in this analysis. Beth Kindig and the I/O Fund own shares in NVDA at the time of writing and may own stocks pictured in the charts.    

Leo Miller, AI and Semiconductor Investment Writer at I/O Fund, contributed to this analysis. Leo Miller owns shares of NVDA. 

👉🏻 Share with a Fellow Investor
Help someone else benefit from this insight.

• Micron Is Up 900%. Here’s Why the AI Memory Trade May Still Have Room to Run
• Nvidia, CoreWeave, and Nebius: Inside the Circular Financing of the GPU Boom
• AMD, Nvidia, Arm, Intel: Inside the $120 Billion CPU Gold Rush
• Google TPU v8 vs Nvidia: How Inference Is Rewriting the AI Market