This article was originally published on Forbes on Jun 20, 2024,04:13pm EDTForbesForbes on Jun 20, 2024,04:13pm EDT
Big Tech is spending tens of billions quarterly on AI accelerators, which has led to an exponential increase in power consumption. Over the past few months, multiple forecasts and data points reveal soaring data center electricity demand, and surging power consumption. The rise of generative AI and surging GPU shipments is causing data centers to scale from tens of thousands to 100,000-plus accelerators, shifting the emphasis to power as a mission-critical problem to solve.
Increasing Power Consumption Per Chip
As Nvidia, AMD, and soon Intel begin to roll out their next generation of AI accelerators, the focus is now shifting towards power consumption per chip, whereas the focus has been primarily on compute and memory. As each new generation boosts computing performance, it also consumes more power than its predecessor, meaning that as shipment volumes rise, so does total power demand.
Nvidia’s A100 max power consumption is 250W with PCIe and 400W with SXM (Server PCIe Express Module), and the H100’s power consumption is up to 75% higher versus the A100. With PCIe, the H100 consumes 300-350W, and with SXM, up to 700W. The 75% increase in GPU power consumption happened rapidly, within two brief years, across one generation of GPUs.
When we look at other GPUs on the market today, AMD’s MI250 accelerators draw 500W of power, up to 560W at peak, while the MI300x consumes 750W at peak, up to a 50% increase. Intel’s Gaudi 2 accelerator consumes 600W, and its successor, the Gaudi 3, consumes 900W, again another 50% increase over the previous generation. Intel’s upcoming hybrid AI processor, codenamed Falcon Shores, is expected to consume a whopping 1,500W of power per chip, the highest on the market.
Nvidia’s upcoming Blackwell generation boosts power consumption even further, with the B200 consuming up to 1,200W, and the GB200 (which combines two B200 GPUs and one Grace CPU) expected to consume 2,700W. This represents up to a 300% increase in power consumption across one generation of GPUs with AI systems increasing power consumption at a higher rate. SXM allows the GPUs to operate beyond the PCIe bus restrictions, offer higher memory bandwidth, high data throughput and higher speeds for maximal HPC and AI performance, thus drawing more power.
It’s important to note that each subsequent generation is likely to be more power-efficient than the last generation, such as the H100 reportedly boasting 3x better performance-per-watt than the A100, meaning it can deliver more TFLOPS per watt and complete more work for the same power consumption. However, GPUs are becoming more powerful in order to support trillion-plus large language models. The result is that AI requires more power consumption with each future generation of AI acceleration.
Sign up for I/O Fund's free newsletter with gains of up to 2600% because of Nvidia's epic run - Click hereClick hereClick here
Big Tech’s AI Ambitions Lead to Surging GPU Shipments
From Big Tech’s perspective, we’re still in the early stages of this AI capex cycle. Most recently, we covered how Big Tech is boosting capex by more than 35% YoY in 2024, likely upwards of $200 billion to $210 billion, predominantly for AI infrastructure. The majority is flowing to GPU purchases and custom silicon, to power AI training, model development, and to meet elevated demand in the cloud.
2023 was a breakout year for Nvidia’s data center GPUs, with reports placing annual shipments at 3.76 million, for an increase of more than 1.1 million units YoY. A report stated that at peak of 700W and ~61% annual utilization, each GPU would draw 3.74 MWh; this means that Nvidia’s 3.76 million GPU shipments could consume as much 14,384 GWh (14.38 TWh). A separate report estimated that with 3.5 million H100 shipments through 2023 and 2024, that H100 alone could see total power consumption of 13.1 TWh annually.
The 14.4 TWh is equivalent to the annual power needs of more than 1.3 million households in the US. This also does not include AMD, Intel, or any of Big Tech’s custom silicon, nor does it take into account existing GPUs deployed or upcoming Blackwell shipments in 2024 and 2025. As such, the total energy consumption is likely to be far higher by the end of the year as Nvidia’s Blackwell generation comes online in larger quantities.
To read more about Nvidia’s upcoming Blackwell architecture, reference our previous analysis: Nvidia Q1 Earnings Preview: Blackwell and the $200B Data Center.Nvidia Q1 Earnings Preview: Blackwell and the $200B Data Center. If you own AI stocks, or are looking to own AI stocks and want to learn more, we encourage you to attend our upcoming weekly webinar, held this Thursday at 4:30 pm EST. Learn more here.here.
A Path to Million GPU Scale
Nvidia and other industry executives have laid out a path for GPU clusters in data centers to scale from the tens of thousands of GPUs per cluster to the hundred-thousand-plus range, even up to the millions of GPUs by 2027 and beyond. We’re already seeing signs of strong demand for Nvidia’s Blackwell platform, but overall, the million-plus GPU data center target is still years away.
Oracle’s Chairman Larry Ellison sees this creating secular tailwinds for data center construction, due to both rising GPU demand and increased power requirements driving a shift to liquid cooling:
“This AI race is going to go on for a long time. It's not a matter of getting ahead, just simply getting ahead in AI, but you also have to keep your model current. And that's going to take larger and larger data centers. … The data centers we are building include the power plants and the transmission of the power directly into the data center and liquid cooling. And because these modern data centers are moving from air cooled to liquid cooled, and you have to engineer them from scratch. And that's what we've been doing for some time. And that's what we'll continue to do.”
As the industry progresses towards that million-GPU scale, this puts more emphasis on future generations of AI accelerators to focus on power consumption and efficiency while delivering increasing levels of compute. Data centers are expected to adopt liquid cooling technologies to meet the cooling requirements to house these increasingly large GPU clusters.
For more information on investing in AI, check out our 1-hour interview “AI is the Best Opportunity of our Lifetime.”AI is the Best Opportunity of our Lifetime.”
AI Electricity Demand Forecast to Surge
As a result of booming demand for generative AI and for GPUs, AI’s electricity demand is forecast to surge, especially in the data center. We have a handful of different viewpoints and analyst projections that, while differing slightly in the timelines, all point to that same conclusion.
For example, Morgan Stanley is estimating global data center power use will triple this year, from ~15 TWh in 2023 to ~46 TWh in 2024. This coincides with the ramp of Nvidia’s Blackwell chip later in the year as well as utilization of the entirety of its deployed Hopper GPUs, and increased shipments from AMD and custom silicon ramps from Big Tech.
Morgan Stanley also projects generative AI power demand may exceed 2022’s data center power usage by 2027 if GPU utilization rates are high, at ~90% on average; however, their base case still calls for a nearly 5x increase in power demand over the next three years.
