At GTC this week, Jensen Huang stated the revenue opportunity for Nvidia’s artificial intelligence chips may reach at least $1 trillion through 2027, up from a previous target of $500 billion. While that grabbed most of the headlines, there was another jaw-dropping statistic that will set the stage in the coming years – which was the ability to drive up to 35X higher throughput per megawatt with its new Groq 3 LPX racks.
The 256-chip LPX rack introduces Groq’s unique SRAM‑based architecture that allows Nvidia to offload decode‑phase workloads and massively increase token throughput. This primarily targets trillion‑parameter LLMs, million-token context, and multi‑agent systems, which are bottlenecked less by compute and more by how efficiently a system can move data and generate tokens. Paired with the new Vera Rubin GPUs, Nvidia claims this architecture can deliver up to 35X higher throughput per megawatt, with seamless integration into Vera Rubin deployments.
In some ways, this acquisition draws parallels to Nvidia’s $6.9 billion acquisition of Mellanox, which my firm covered for premium research members in 2020. Mellanox was a strategic purchase to clear the bottleneck at the time on GPU performance, which was scale-out networking. By combining Nvidia’s GPUs with the strength of Mellanox’s InfiniBand, smart NICs and switching, Nvidia was able to turn accelerators into clusters by removing the limiter at that time (scale-out networking).
The Groq acquisition is aimed to solve a different limiter, which is inference throughput per watt, where memory bandwidth can become the gating factor to token output and cost. Nvidia is preparing to position its GPUs to be among the best inference options available, utilizing Groq’s unique SRAM-based architecture to significantly turbocharge token throughput and accelerate inference performance.
Nvidia expects Groq will help drive up to a 15X increase in tokens per second, directly translating into higher tokens per megawatt, which is already scaling by a factor of 10X between Blackwell and Rubin. If these claims hold true, then cheaper inference will unlock more usage, and more usage should lead to higher revenue and higher profits as the AI monetization wave plays out.
Below, we cover how Nvidia, the de facto leader in training, is now shifting its focus to inference architecture as the next catalyst.
Why Nvidia is Rethinking AI Inference Architecture
Year after year (and generation after generation), Nvidia has proven that it can consistently deliver massive efficiency gains on inference throughput and token processing speed. For example, Nvidia’s GB200 NVL72 boosts per-GPU throughput by up to 30X versus the HGX H100 platform, while the GB300 NVL72 boasts up to a 50x increase in AI factory output via a 10X increase in tokens per second per user and a 5X increase in throughput per MW.
Versus the Blackwell NVL72 systems, Nvidia says Vera Rubin can deliver up to 10X more throughput per megawatt, rapidly compounding performance gains from its Hopper generation in just three years.
Source: Nvidia
However, the more important piece of the puzzle is not just the rapid pace of these throughput gains, but how Nvidia can continue to deliver exponentially more throughput gains and how Nvidia will accelerate inference workloads further. The key answer to this is Groq, and ‘inference disaggregation’ at the rack level.
Inference disaggregation refers to splitting up the two-step process of token generation, prefill and decode, instead of running both steps together. The prefill phase processes the entire input token sequence in parallel and stores information in the KV cache, relying heavily on GPU compute and not as much on memory (yet). The decode phase generates the output tokens one by one in a sequential manner, relying on the KV cache and previous tokens, making it extremely reliant on memory bandwidth and capacity to rapidly access cached tokens. When discussing how AI workloads are memory constrained, it comes from the decode phase.
When both prefill and decode shared the same hardware (the GPUs), the two would interfere with each other and lead to delays, as a new prefill request would either force the system to pause decodes and prioritize the prefill, or run both again at the same time, elongating response times.
With inference disaggregation, prefill and decode can be scaled and scheduled on different optimized hardware via Nvidia’s Dynamo; in this case the Rubin GPUs handle prefill and Groq LPUs handle decode. With disaggregation and the LPU’s massive memory bandwidth, Nvidia CEO Jensen Huang says the two combined can deliver up to 35X higher throughput per MW on trillion-parameter LLMs:
“What if we disaggregated inference altogether with a piece of software called Dynamo? What if we rearchitected the way that inference is done in the pipeline, so that we could put the work that makes perfect sense on Vera Rubin and then offload the decode generation, the low latency, the bandwidth limited challenged part of the workload for Groq. And so we united, unified processors of extreme differences, one for high throughput, one for low latency.
It still doesn’t change the fact that we need a lot of memory. And so Groq, we’re just going to add a whole bunch of Groq chips, which expands the amount of memory it has. And so if you could just imagine, out of 1 trillion parameter model, we have to store all of that in Groq chips. However, it sits next to NVIDIA Vera Rubin, where we could hold the massive amounts of KV cache that’s necessary in processing all of these agentic AI systems. It’s based upon this idea of disaggregated inference. We do the prefill, that’s the easy part, but we also tightly integrate the decode.
So the attention part of decode is done on NVIDIA’s Vera Rubin, which needs a lot of math and the feed forward network part of it, the decode part is done — the token generation part is done — on the Groq chip. The 2 of them working tightly coupled together over today, Ethernet with a special mode to reduce its latency by about half.
And so that capability allows us to integrate these 2 systems. We run Dynamo, this incredible operating system for AI factories on top of it, and you get 35x increase, not to mention additional new tiers of inference performance for token generation the world has never seen.”
Inference disaggregation is not an entirely new concept, but rather it is the way Nvidia is approaching disaggregation that makes this move noteworthy. Instead of seeing disaggregation as a separate, service-layer optimization, such as what AWS is eyeing with its recent partnership with Cerebras, Nvidia is pushing to directly embed disaggregation into the rack to maximize throughput.
Inside Groq’s SRAM Architecture and Its Massive Bandwidth Advantage
Groq’s chips feature a completely different memory-based architecture than Nvidia’s GPUs, utilizing SRAM instead of HBM. This unique architecture gives Groq’s language-processing units (LPUs) a significant advantage in the decode phase and in low-latency, high-query inference workloads from extremely higher bandwidth.
SRAM offers a major trade-off versus DRAM and HBM when it comes to memory storage capabilities within AI accelerators. Unlike typical DRAM, SRAM does not require capacitors and stores data without the need for periodic refreshing, as long as power is available. Because of this design, SRAM can offer the fastest memory access speeds with minimal latency, though at the cost of having a mere fraction of the capacity of HBM chips – the LPUs have just 500MB of capacity versus 288GB of HBM in its Rubin GPUs.
Despite having just 500MB of capacity, each LPU delivers 150 TB/s of SRAM bandwidth — this is nearly 7X the 22 TB/s HBM bandwidth per Rubin GPU. In the rack-scale configuration, the Groq 3 LPX delivers an astounding ~2.5X increase in total scale-up bandwidth and a 25X increase in SRAM bandwidth versus HBM bandwidth.
The Groq 3 LPX combines 256 individual LPUs for a total of 128GB of SRAM capacity, yet it offers 40 PB/s of SRAM bandwidth versus 1.6 PB/s of HBM bandwidth in the Vera Rubin NVL72. Total scale-up bandwidth reaches 640 TB/s versus 260 TB/s in the NVL72. This also dwarfs the upcoming NVL576 rack which offers just 4.6 PB/s of HBM bandwidth.
This 25X increase in bandwidth is precisely the reason why Nvidia is aiming to offload decode and low-latency workloads to the LPX racks, as more bandwidth means more weight data can be processed per second, which, at its core, means more tokens can be generated per second.
Nvidia Positioning Groq 3 LPX as a ‘Token Accelerator’
Nvidia is positioning its new Groq 3 LPX racks as a ‘token accelerator’ functioning in tandem with Vera Rubin GPUs to significantly boost token throughput and address the upcoming multi-agent future. The Groq LPUs are not meant to replace GPUs in inference workloads, but rather compliment them by optimizing for memory-intensive decode.
Off the bat, Nvidia expects that combining Rubin GPUs and Groq racks will drive substantial increase in token throughput, with Nvidia VP Ian Buck claiming the combination “moves us from a world where 100 tokens per second is a reasonable throughput to one of 1500 TPS or more for AI agent intercommunication.”
