Learning Objectives
- Explain the energy, GPU, memory, and cooling constraints that limit AI data center scaling
- Describe the geopolitical dimensions of AI chip supply and export controls
- Summarize near-term and long-term infrastructure trajectories, including nuclear and space-based solutions
Why Infrastructure Matters for Developers
Most developers call an API and think nothing of what happens on the other side. But the physical infrastructure that makes those API calls possible is now the primary constraint on the rate of AI progress — more than algorithmic breakthroughs or research investment.
Understanding the infrastructure landscape helps explain: why frontier model inference costs what it does, why GPU availability affects development timelines, why AI companies are signing nuclear power deals, and where the bottlenecks are likely to ease (or tighten) in the coming years.
The Energy Problem
A single large-scale AI training run for a frontier model consumes roughly the same electricity as a small city uses in a year. A hyperscaler's AI data center cluster — the kind needed to train GPT-5 or Gemini 3 — consumes 1-2 gigawatts of power continuously.
Context: 1 GW is approximately the output of a large nuclear power plant. Global data center power consumption is reaching 96 GW by 2026 — nearly doubled from 2023. US data centers now consume approximately 6% of total US electricity. AI drove the majority of that growth.
Power purchase agreements (PPAs): The hyperscalers — Google, Microsoft, Amazon — have all signed nuclear power purchase agreements in the last two years. Microsoft signed a deal to restart the Three Mile Island nuclear plant in Pennsylvania specifically for AI compute needs. Google has agreements with multiple nuclear operators. Amazon has partnered with Dominion Energy for nuclear capacity.
These aren't ideological choices — they're engineering ones. AI data centers need 24/7 reliable power that doesn't depend on weather. Solar and wind require storage. Nuclear runs continuously.
The Stargate Project: The largest AI infrastructure initiative to date, a joint venture between OpenAI, SoftBank, Oracle, and others targeting $400 billion+ in total investment. The flagship Abilene, Texas facility is operational (~1 GW by mid-2026), with 5 additional sites announced (7 GW total capacity). However, the Texas expansion was scrapped due to financing disputes — illustrating the execution challenges of mega-scale data center projects. OpenAI is also developing a custom "Titan" chip with Broadcom (TSMC 3nm, expected H2 2026) to reduce NVIDIA dependency.
Small Modular Reactors (SMRs): SMRs are factory-manufactured nuclear reactors at 1/10 the scale of traditional plants — designed to be sited adjacent to data centers. Progress has accelerated: NuScale signed a 6 GW deal with the Tennessee Valley Authority (TVA). Oklo broke ground at Idaho National Laboratory and signed a 12 GW deal with Switch. X-energy is under an 18-month NRC review for design certification. First commercial SMR-powered data centers are expected 2027-2030. Microsoft, Google, and Amazon all have SMR agreements signed.
The GPU Supply Chain
NVIDIA controls approximately 80% of AI training compute capacity, primarily through the H100, H200, and GB200 Blackwell GPU families. This concentration creates supply chain risks that have already affected company timelines.
Lead times: Large GPU orders (10,000+ GPUs) have required 6-12+ months of lead time at peak demand. Major AI companies have paid significant premiums for priority allocation.
CUDA moat: NVIDIA's competitive advantage is not only hardware — it's the CUDA programming ecosystem built over 20 years. The vast majority of AI frameworks, optimization tools, and model architectures are CUDA-optimized. Switching to AMD hardware requires re-optimization work that many teams have not done.
AMD MI300X / MI350: AMD has made the most credible challenge to NVIDIA dominance. The MI300X has 192GB of HBM3 memory — significantly more than the H100's 80GB — making it advantageous for very large model inference. The MI350 competes directly with the H200. AMD's ROCm software stack has improved dramatically, though it still trails CUDA in ecosystem depth.
Custom ASICs: The hyperscalers are reducing NVIDIA dependency for their own workloads. Google's TPUs handle Gemini training and inference. Amazon's Trainium3 runs a growing percentage of AWS customer inference. Microsoft's Maia 200 handles Copilot workloads. This internal substitution limits hyperscaler demand for NVIDIA GPUs, but doesn't affect the broader AI ecosystem.
NVIDIA's answer, as of August 2026, is to host the substitutes rather than resist them. It invested $3.5 billion in MediaTek and opened its NVLink Fusion interconnect, so custom accelerators MediaTek designs for other companies can run inside NVIDIA-based racks on NVIDIA's fabric. That softens the substitution story above: a customer's own silicon increasingly displaces some NVIDIA GPUs while still buying NVIDIA's interconnect, networking and rack platform.
The Memory Bottleneck
Even with sufficient GPUs, a supply chain constraint on High Bandwidth Memory (HBM) limits GPU production:
The HBM situation: HBM is specialized DRAM stacked on the GPU package. Only three companies produce it: SK Hynix (~50% market share), Samsung (~30%), and Micron (~20%). The manufacturing process is extremely complex and yields are low.
Why it matters: NVIDIA's H100 requires 5 HBM3 stacks per GPU. At peak demand, HBM supply has constrained NVIDIA's ability to produce H100s even when the GPU die was available. The HBM supply chain is a chokepoint for the entire AI hardware industry.
HBM4 in production: HBM3e remains the standard in the installed base (H200, MI300X), but all three suppliers completed the HBM4 jump within a single quarter — SK Hynix began production in February 2026, Micron reached high volume in March, and Samsung ramped in the second quarter. All three now feed NVIDIA's Vera Rubin architecture. Each generation roughly doubles bandwidth and capacity — HBM4 enables the next generation of GPU architectures.
Cooling Systems
Modern AI chips run extremely hot. The H100 SXM module dissipates approximately 700W per GPU; a rack of 8 H100s dissipates 5,600W — more than a typical home's total electrical consumption.
Air cooling limits: Traditional data center air cooling can handle roughly 20-30kW per rack. Modern AI racks push 50-100kW per rack, requiring fundamentally different cooling approaches.
Direct Liquid Cooling (DLC): Pipes coolant directly to chip heat spreaders. Removes heat 3-5-times more efficiently than air cooling. Now standard for dense AI deployments.
Immersion cooling: Entire server boards submerged in dielectric (non-conductive) fluid. Removes heat directly from all components. Enables extremely high density — 300+ kW per rack is achievable. Being deployed in the most cutting-edge AI data centers.
The retrofit problem: Existing data centers built for air-cooled IT equipment cannot simply add liquid cooling without significant infrastructure changes. The need to retrofit or replace existing data center inventory is a significant capital cost for hyperscalers.
Geopolitics: Export Controls
AI chip capability has become a geopolitical resource, and governments are acting accordingly.
US Bureau of Industry and Security (BIS) export controls: The US government has restricted the export of advanced AI chips (NVIDIA H100 and above, AMD MI300X and above) to China, Russia, and a growing list of designated countries. The restrictions apply not just to direct sales but to cloud access from those countries.
This has several effects:
- Chinese AI companies cannot directly purchase the most advanced NVIDIA training hardware
- It has accelerated Chinese domestic chip development (Biren, Cambricon, Huawei Ascend)
- It has created a gray market for smuggled chips in some regions — and that market is now being prosecuted. In August 2026 Taiwanese prosecutors charged nine people over servers carrying NVIDIA B300 GPUs that reached China through Indonesia, Japan and Hong Kong, in what is understood to be the island's first criminal case of its kind. The accused included a distribution manager at NVIDIA's own Taiwan office and two sales managers at Supermicro's, which is the part developers should note: the diversion ran through the vendors' authorized sales channel using forged paperwork, not around it
- It has bifurcated the global AI ecosystem into US-aligned and China-domestic capability tracks
For developers: this affects which models are accessible from which countries, and which data center regions certain AI services can be hosted in.
The Future of AI Infrastructure (2026-2035)
Near-term (2026-2028):
- Nuclear SMR deals accelerate; first commercial SMR deployments adjacent to data centers
- HBM4 production ramps, though the memory constraint has so far tightened rather than eased — it has shifted from a lead-time problem to a price problem, and is now flowing through to AI system prices
- Liquid cooling becomes standard for new AI data center construction
- NVIDIA maintains dominance but AMD gains share in inference
Medium-term (2028-2035):
- Space-based solar power: the Japanese Aerospace Exploration Agency (JAXA) and European Space Agency (ESA) have active programs. Theoretical advantage: continuous 24/7 solar power from orbit, beamed to Earth as microwave. Demonstration projects targeting 2030-2035; commercial viability in the 2035-2040 range.
- Quantum computing: theoretical advantages for certain ML workloads (optimization, simulation of quantum systems) remain genuinely uncertain. Current quantum computers are not useful for ML training. A 2030 timeline for practically useful quantum ML is aggressive but held by some researchers.
Cost trajectory — two curves moving in opposite directions. These are easy to conflate, and conflating them will mislead your planning. Cost per unit of capability keeps falling fast: inference costs have dropped roughly 10-times per year for equivalent capability, and frontier-class AI that costs dollars per query today will likely cost fractions of a cent within 3-5 years. But the purchase price of the hardware is currently rising. Server builders supplying the largest cloud operators have been told to expect increases above 15 percent on many AI systems shipping in early 2027 — driven by memory, not by processors, as HBM and conventional DRAM suppliers fail to keep pace with demand. A team can therefore see its cost per token fall and its capital budget rise in the same year.
Key Takeaways
- Energy is the hard physical constraint on AI progress — data centers consuming gigawatts have driven hyperscalers to nuclear power agreements and SMR investments
- NVIDIA's GPU supply dominance (with the CUDA ecosystem as the software moat) and the HBM memory bottleneck are the two primary hardware supply chain constraints
- US export controls on advanced AI chips have bifurcated the global AI hardware ecosystem and accelerated Chinese domestic chip investment
- Inference cost per unit of capability follows a steep decline curve — the most reliable long-term prediction is that frontier AI inference costs will be 100-times lower within 5 years, enabling applications currently too expensive to be practical — but do not read that as hardware getting cheaper: memory shortages are pushing 2027 AI server prices up by more than 15 percent, so cost per token and capital cost can move in opposite directions at the same time
