Nvidia, CXL, and the Battle to Improve AI Inference Economics

This is Part 2 of our two-part series on AI inference economics. In Part 1 — Why Nvidia's Next AI Battle Is About Tokens per Watt, we laid out why tokens per watt has become the defining metric for inference profitability: as memory and power grow scarce, the hyperscalers that generate the most tokens from a fixed power footprint scale revenue faster than cost. We also discussed how the "memory wall,” defined as compute outpacing memory bandwidth and capacity, leaves expensive accelerators underutilized. One solution is offload engines, like those offered by Nvidia, that lift tokens per watt by keeping XPUs fed with KV cache instead of sitting idle or recomputing. 

In the article below, we turn from the why to the how and the who. Two architectural paths are competing to solve the KV cache bottleneck: Nvidia's proprietary CMX platform and the open, vendor-agnostic CXL standard. Although they tackle the same problem, each approach points to different sets of beneficiaries. 

With CMX, Nvidia is designing a tightly co-designed platform leveraging its BlueField DPUs and software optimizations to control KV cache allocation and movement, aiming to increase KV cache capacity and sharing across the pod.  

CXL offers a vendor-agnostic route to share memory resources coherently across the pod, including for KV cache tasks, enabling similar gains in inference throughput. 

In the analysis below, we explain how both CMX and CXL work, touch on the broader CXL ecosystem, and provide our take on which of the two architectures we believe creates the strongest opportunity for investors. 

Nvidia CMX: Adding Shared Memory for KV Cache Offloading 

To solve for KV cache bottlenecks, Nvidia has developed its CMX Context Memory Storage Platform. CMX uses an SSD-based enclosure that is connected to Rubin GPUs and Vera CPUs over Spectrum-X Ethernet.  

The hardware offload engine in CMX is the BlueField-4 DPUs. CMX incorporates 64 BlueField-4 DPUs, which connect to approximately 9,600 TB of SSD storage capacity per rack. Through this, CMX extends effective GPU KV cache capacity to a massive degree with relatively inexpensive SSDs. For reference, one GB200 NVL72 rack comes with 13.4 TB of HBM capacity.  

BlueField-4 is dedicated to controlling how the KV cache moves between the high-capacity SSD tier and GPU memory. It is enabled by a mixture of software to manage KV caches, including DOCA Memos, Dynamo, and Nvidia Inference Transfer Library (NIXL). Together, they determine the proper memory tier that certain parts of the KV cache should reside on at a given time, as well as when the KV cache should be sent to GPUs to improve utilization. 

By employing offload engines in the form of BlueField and specialized software, GPU KV cache capacity greatly increases, and GPUs can retrieve it quickly, avoiding idle and recompute time and improving utilization. Additionally, CMX allows for pod-wide KV cache sharing. Nvidia notes that this improves XPU utilization by reducing KV cache duplication and stranded memory capacity between nodes. 

mid

Nvidia says that CMX allows for up to 5X higher token throughput and up to 5X better power efficiency for KV cache operations compared to traditional storage methods—directly targeting the tokens per watt improvements that are critical to hyperscaler inference economics. 

Nvidia also created the STX reference architecture to provide a standardized blueprint for how storage vendors should build CMX systems to connect to Vera Rubin racks. This allows an ecosystem of storage providers to proliferate and provide CMX products that can be seamlessly integrated with computing resources, accelerating adoption.  

STX is modular, which is key to CMX being a new revenue driver for Nvidia. Because data center operators can add CMX independently of computing resources, capex dollars can be incrementally allocated to directly improving GPU utilization in KV-cache intensive inference and agentic workflows. 

We also covered Nvidia’s other offload solution, its new Groq 3 LPX racks, that aim to significantly accelerate inference throughput in our March analysis, Nvidia Stock to See New Growth Catalyst; 35X Faster AI with Groq 3 LPX. 

CXL Memory: The Open Standard Alternative to CMX 

Compute Express Link (CXL) is another pathway for building offload engine solutions. Unlike Nvidia’s proprietary CMX, it is vendor-agnostic, though it shares the same ideas as CMX: adding memory where the KV cache can be stored and allowing multiple devices to access it.  

One of the key benefits of CXL is that it is built on Peripheral Component Interconnect Express (PCIe). PCIe is a universally adopted interconnect standard, making CXL-based systems relatively easy to implement and able to benefit from PCIe’s 16X increase in data rates since 2019, rising from 32 GB/s to 512 GB/s.  

As a vendor agnostic solution, CXL is advantaged in the fact that it can allow data center operators to scale memory independently without being further locked into the Nvidia ecosystem. Notably, the Vera CPU supports CXL 3.1, meaning that data center operators using Nvidia hardware have an option to scale memory capacity through this open standard rather than only through its proprietary offerings. 

Yole Research estimated that in Q1 2025, two-thirds of servers were CXL-capable, and projects this figure will rise to more than 90% by the end of 2026. Meanwhile, Yole estimates that near 0% of servers are CXL enabled, and expects that percentage to rise to 13% by 2030.  

Networking Stocks Show Large Improvements in Inference Throughput With CXL 

CXL 2.0 allows for memory pooling and switching, and is the standard currently being deployed or soon to be deployed in data centers. Through this, CPUs can dynamically allocate memory from the pool between processors. 

According to data from Marvell, by adding a 16TB DRAM memory pool through its Structera S CXL switches, CXL pooling enabled a 4.8X improvement in inference throughput, greatly increasing GPU utilization. Additionally, this enabled an 82.7% drop in time to first token.  

CXL 3.0 will extend beyond memory pooling into memory sharing—where any device, including CPUs, GPUs, and others, can access the same parts of the memory pool simultaneously. Marvell expects to begin sampling its Structera S CXL 3.0 switch in calendar Q3 2026. 

Marvell also makes CXL memory controllers, such as the Structera X, which Marvell says consumes < 30 watts, compared to 150W to 700W for an additional CPU or GPU to increase memory. This means that adding memory capacity through CXL can help minimize power consumption and help drive token throughput gains, allowing for greater tokens per watt improvements. 

These products are key to Marvell’s ambitions of generating significant CXL-based revenue. Marvell expects its CXL business to exceed $1 billion by calendar 2028, though analysts are a bit more optimistic with UBS seeing Marvell’s CXL revenue hitting $1 billion in calendar 2027 and doubling to $2 billion in calendar 2028. 

The I/O Fund’s portfolio is loaded with networking stocks, including one key beneficiary of this trend up more than 200% since our latest entry earlier this year, but Marvell isn’t one of them. ➡️ Sign up today to see where the I/O Fund is currently positioning for growth in the networking stack.

Google Validating the Importance of Shared Memory with its New TPUs 

We recently covered how Google’s inference-optimized TPU 8i is bringing the importance of shared, coherent memory to the forefront, with Google positioning the new chip around this as a key anchor in improving inference efficiency. For context, Google’s new 8i pods scale to 1,152 TPUs, with pod-level HBM capacity rising to 331.8TB shared across all 1,152 chips. 

Here is what we stated in our recent analysis: 

This is arguably the most critical point to understand surrounding Google’s architectural advantage with the 8i, that this 331.8 TB of memory is shared across the entire pod over Google’s inter-chip interconnect (ICI). ICI is similar to Nvidia’s NVLink—with both allowing for the fastest chip-to-chip memory access within a pod. Compare this to Nvidia’s NVL72, where true memory coherency only extends at rack-scale across 72 GPUs and just 20.7TB of HBM.  

Scaling out to 1,152 of Nvidia’s GPUs would span 16 racks, yet memory does not become a unified pool shared across the entire cluster. By keeping the maximum amount of memory in a shared domain with the TPU 8i, large frontier models with long context windows can run with minimal latency. 

Google’s approach with 8i validates the core objective of CXL, enabling coherent memory sharing at the pod-level to increase inference throughput and improve GPU utilization. These are both key factors in enabling complex reasoning models and autonomous agentic workflows to proliferate. 

Conclusion: 

CMX and CXL both tackle improving utilization to improve tokens per watt, but they have very different investment outcomes. Nvidia’s CMX could gain significant traction, but is unlikely to be a stock catalyst at such a high market cap. Meanwhile, CXL can make an impact on much smaller players as it’s open and vendor-agnostic.  

Analyzing the networking ecosystem and narrowing down which stocks have the best positioning is far from easy, but it is exactly what the I/O Fund specializes in. Behind the paywall, we provide deeper coverage on the key players in CXL and the stocks we see at top long-term beneficiaries. This research builds off our demonstrated success with a 326% cumulative over a 5-year period, which would rank us #1 if we were a hedge fund and #3 if we were a tech ETF.  

As Q2 wraps up, the I/O Fund is preparing to identify the next wave of AI winners in our upcoming Top 15 AI Stocks for Q3 2026 report. Previous reports identified Micron as a major beneficiary, Lumentum as a networking leader, along with names such as Bloom Energy, up over 1800% since our April 2025 entry. We publish more than 100 paywalled articles each year on AI stocks, hold weekly 1-hour webinars, and offer an actively managed portfolio with real-time trade alerts.
Sign Up Now

Please note: The I/O Fund conducts research and draws conclusions for the company’s portfolio. We then share that information with our readers and offer real-time trade notifications. This is not a guarantee of a stock’s performance and it is not financial advice. Please consult your personal financial advisor before buying any stock in the companies mentioned in this analysis. Beth Kindig and the I/O Fund own shares in NVDA and GOOGL at the time of writing and may own stocks pictured in the charts.    

Leo Miller, AI and Semiconductor Investment Writer at I/O Fund, contributed to this analysis. Leo Miller owns shares in NVDA and GOOGL. 

👉🏻 Share with a Fellow Investor
Help someone else benefit from this insight.

Recommended Reading:

• Micron Is Up 900%. Here’s Why the AI Memory Trade May Still Have Room to Run
• Nvidia, CoreWeave, and Nebius: Inside the Circular Financing of the GPU Boom
• AMD, Nvidia, Arm, Intel: Inside the $120 Billion CPU Gold Rush
• Google TPU v8 vs Nvidia: How Inference Is Rewriting the AI Market