AI Hardware & Chips

Learning Objectives

• Identify the major AI chip families and their primary use cases
• Explain NVIDIA's software ecosystem advantage beyond raw hardware performance
• Compare custom ASICs (TPUs, Trainium, Cerebras) with general-purpose GPUs for specific workloads

Why Hardware Matters for Developers

For developers calling APIs, the hardware is abstracted — you never see the GPU your request runs on. But hardware literacy matters for several reasons:

• Local model inference: Running Llama or DeepSeek locally requires choosing compatible hardware
• Cost reasoning: Understanding why inference costs what it does helps make informed build-vs-buy decisions
• Cloud instance selection: Choosing AWS GPU instances (P4d, P5) vs. Trainium vs. Inferentia requires understanding the tradeoffs
• Future planning: Hardware trajectories determine what AI capabilities will be affordable in 12-24 months

NVIDIA — The Dominant Ecosystem

NVIDIA holds approximately 80% of AI training and inference compute market share. Understanding why reveals as much about software as hardware.

The Hardware: H100 to Blackwell Ultra

H100 SXM (Hopper architecture, 2022-2024): The workhorse GPU of the current AI era.

• 80GB HBM3 memory; 3.35 petaflops FP16 performance
• NVLink interconnect for multi-GPU communication
• Market price: $25,000-$35,000 per GPU; cloud instance price: $2-3/hour

H200 SXM (2024-2025): Evolutionary improvement over H100.

• 141GB HBM3e — 1.75-times the H100's memory, with faster bandwidth
• Same Hopper GPU die as H100; the upgrade is primarily the memory
• Drop-in compatibility with H100 infrastructure

GB200 Blackwell NVL72 (2025): NVIDIA's rack-scale flagship.

• 72 Grace CPUs + 72 Blackwell GPUs interconnected as a single logical unit
• 720GB of total HBM3e memory shared across the system
• Designed for inference of very large models (405 billion+ parameter models) and next-generation training runs
• Not a single server — a rack-scale unit with liquid cooling

GB300 Blackwell Ultra (announced GTC 2025): Enhanced Blackwell variant.

• Pairs two upgraded Blackwell Ultra GPUs with one Grace CPU via NVLink-C2C
• 576GB HBM3e per superchip (up from 384GB on GB200)
• Enhanced FP4 and FP6 performance for both training and inference
• Bridge product between Blackwell and the next-generation Vera Rubin architecture

The Roadmap: Vera Rubin (Shipping H2 2026)

Vera Rubin NVL72 is now in production and shipping H2 2026 — NVIDIA's next-generation architecture following Blackwell:

• HBM4 memory — 2.8-times faster memory bandwidth than Blackwell
• 10-times inference cost reduction vs. Blackwell for equivalent workloads
• Pairs with next-gen Grace CPU (Vera CPU)
• Part of NVIDIA's annual architecture cadence: Hopper (2022) → Blackwell (2024) → Vera Rubin (2026)
• Rubin Ultra variant on the roadmap for 2027

This annual cadence is a key competitive advantage — each generation delivers roughly 2-times the performance of the prior, keeping NVIDIA ahead of competitors who update on longer cycles.

Consumer GPUs: RTX 5090

The RTX 5090 (released January 2025, $1,999 MSRP) is NVIDIA's flagship consumer GPU:

• 32GB GDDR7 memory — up from the RTX 4090's 24GB GDDR6X
• Significant performance improvement for local model inference
• Best consumer option for running quantized 30 billion-70 billion parameter models locally

NVIDIA DGX Spark is now shipping at $4,699 (raised from the original $3,999 announcement price) — a desktop AI workstation with 128GB unified memory and 1 PFLOPS of AI performance.

The Software Moat: CUDA and NIM

CUDA is NVIDIA's parallel computing platform — a programming model and compiler toolchain for GPU programming. Every major AI framework (PyTorch, TensorFlow, JAX) is deeply optimized for CUDA. Every major AI model is trained, profiled, and optimized on CUDA hardware.

This ecosystem has been built over 20 years. The gap isn't just current performance — it's documentation, community answers, optimized kernels, and production code that all assumes CUDA. Switching to AMD or other hardware requires adapting or rewriting significant portions of this stack.

NVIDIA NIM (Neural Inference Microservices) simplifies deployment: pre-optimized Docker containers that package LLMs with TensorRT-LLM optimization. Developers can deploy optimized Llama, Mistral, and other models with a single docker pull. Free for development; enterprise license for production.

TensorRT-LLM is the open-source engine behind NIM's performance — automatic batching, quantization (FP8, INT4), KV cache optimization, and multi-GPU scheduling for production inference.

The CUDA moat is why NVIDIA's pricing power persists even as the hardware becomes more expensive. Customers are buying into the full ecosystem — hardware, software, and developer tools. See the full NVIDIA company profile for more on their product lineup.

AMD — The Memory-Rich Challenger

AMD's MI300X has found real traction specifically for large model inference:

192GB of HBM3 per GPU — 2.4-times the memory of the H100's 80GB. For running very large models (Llama 70 billion+, Mixtral 8x7 billion), being able to fit more of the model in GPU memory reduces costly memory bandwidth bottlenecks.

Large cloud providers (Azure, Meta's own infrastructure) have deployed MI300X at scale for inference workloads where the memory advantage justifies the ROCm software migration cost.

ROCm and HIP: AMD's answer to CUDA. HIP (Heterogeneous-computing Interface for Portability) allows porting CUDA code to ROCm with minimal changes for common operations. Major frameworks support ROCm, but the ecosystem depth still trails CUDA.

MI350 (shipped Q3 2025): 288GB HBM3e — AMD's next-generation, competing directly with the H200. Performance parity on many benchmarks; ROCm has improved to v7.0-7.2 but still trails CUDA in ecosystem depth. The MI400 generation launched in July 2026 as AMD's answer to NVIDIA's Vera Rubin, deployed in the new Helios rack-scale system — 72 MI455X GPUs per rack — with Microsoft, Anthropic, OpenAI, and Meta among the launch customers.

Apple Silicon — Local AI on Mac

Apple's M-series chips reached the M6 generation on August 25, 2026, announced alongside the M5 Ultra and a rebuilt Mac Studio and Mac mini. The M-series families (Pro, Max, Ultra variants) have made MacBook Pro, Mac Studio, and Mac Pro compelling platforms for local AI inference:

Unified memory architecture: CPU and GPU share the same memory pool, so there is no memory copying between them — the bottleneck on discrete GPU systems. For models that fit in unified memory, this is highly efficient. The ceiling matters more than the throughput here, and it just doubled: the M5 Ultra reaches 512 gigabytes at 1.2 terabytes per second, against 256 gigabytes at 819 gigabytes per second on the M3 Ultra it replaces. The M5 Max tops out at 128 gigabytes. Note that there was never an M4 Ultra — Apple skipped that generation at the Ultra tier, which is why the comparison is against the M3.

MLX: Apple's open-source machine learning framework optimized for M-series. Model inference on MLX on an M-series Max chip is often faster than equivalent CPU inference and competitive with consumer NVIDIA GPUs. The mlx-community on Hugging Face hosts hundreds of pre-converted model weights optimized for MLX.

Practical local models at 128 gigabytes (M-series Max):

• Llama 4 Scout: fits comfortably; full-quality inference
• DeepSeek R1 7 billion/14 billion: excellent performance
• Mistral 7 billion: fast, practical for daily use
• Gemma 4 31 billion (dense): fits in 128 gigabytes; high quality

At the 512-gigabyte M5 Ultra tier the constraint stops being which model fits and becomes how fast it runs. Apple says that configuration holds models with hundreds of billions of parameters entirely on device, which is a class of model no consumer discrete-GPU workstation can load at all — but throughput per dollar still favors dedicated accelerators for models small enough to fit on them.

Apple Intelligence is Apple's privacy-first AI system built into iOS 18+, iPadOS, and macOS — featuring Writing Tools, Image Playground, Genmoji, and enhanced Siri. Most processing runs on-device via the Neural Engine; complex tasks use Private Cloud Compute, Apple's confidential cloud infrastructure running on Apple Silicon servers.

For developers who need a powerful portable workstation for local AI development, a MacBook Pro with an M-series Max chip and 128 gigabytes remains the practical pick — the Ultra chips are desktop-only, so the 512-gigabyte ceiling means a Mac Studio. See the full Apple company profile for more on their AI tools.

Custom ASICs: The Hyperscaler Strategy

The largest cloud providers are building their own AI chips to reduce NVIDIA dependence for internal workloads:

Google TPUs (Ironwood v7)

TPUs are optimized for Google's JAX framework and the model architectures Google uses internally. The latest Ironwood v7 generation (GA 2026) achieves 42.5 Exaflops at 9,216 chips — a massive leap from Trillium v6. Anthropic committed to over 1 million Ironwood chips for Claude training and inference.

Available on Google Cloud; best for teams using JAX-based training or deploying Google's models on GCP.

Cerebras WSE-3 — The Unusual Architecture

Cerebras WSE-3 is architecturally distinct: instead of multiple chips connected via NVLink, WSE-3 is a single wafer-scale chip — the entire silicon wafer is one chip.

• 900,000 AI cores on a single die
• 44GB of on-chip SRAM — memory directly on the chip, not HBM stacked alongside it
• Result: extremely high memory bandwidth (on-chip SRAM is much faster than HBM)
• 2,100 tokens per second inference (8-times faster than H200) — this is what powers GPT-5.3-Codex-Spark's real-time performance

Cerebras is available as a cloud service and is targeting an IPO in Q2 2026 at an estimated ~$22 billion valuation. The tradeoff: limited software compatibility (not general-purpose CUDA), but for applications where inference speed is the primary requirement, WSE-3 is in a different performance class.

AWS Trainium3 and Inferentia2

AWS's custom chips are optimized for the AWS ecosystem:

• Trainium3 (EC2 Trn2): training chips; ~40% cost reduction vs. equivalent NVIDIA for supported models (Llama, BERT variants)
• Inferentia2 (EC2 Inf2): inference chips; ~60% cost reduction for supported models; up to 2.3 petaflops

Neither Trainium nor Inferentia supports the full CUDA ecosystem — they use AWS's own Neuron compiler. For teams on AWS running high-volume inference on common models, the cost savings justify the adaptation. For custom architectures or research workloads, NVIDIA remains simpler.

Microsoft Maia 200

Microsoft's Maia 200 is used internally for Azure's AI inference workloads — specifically Copilot and Azure OpenAI Service responses. It reduces Microsoft's NVIDIA dependence for predictable, high-volume inference. Not a cloud product customers can purchase directly.

NVIDIA's Counter-Move: Renting the Rack Instead of Losing the Customer

Everything above describes custom silicon as substitution — a hyperscaler builds its own chip and buys fewer NVIDIA GPUs. As of August 2026 that framing is incomplete, because NVIDIA started selling to the substitution rather than fighting it.

On August 31, 2026, NVIDIA invested $3.5 billion in the Taiwanese chip designer MediaTek and opened its NVLink Fusion interconnect to it. The point is not the money; it is what MediaTek can now build. MediaTek designs custom accelerators for other companies, and NVLink Fusion lets those accelerators speak NVIDIA's rack-scale interconnect — so a customer's own ASIC can sit in an NVIDIA-based rack, next to NVIDIA GPUs, on the same fabric and the same platform. MediaTek expects roughly $2 billion of custom data-center chip revenue across 2026.

Read strategically, this converts NVIDIA's biggest structural risk into a smaller one. If a hyperscaler's alternative to buying NVIDIA GPUs is building a whole rack architecture, that is a hard, multi-year project and NVIDIA can lose the account entirely. If the alternative is designing one accelerator that plugs into NVIDIA's rack, NVIDIA still sells the interconnect, the networking, the systems software and usually some GPUs alongside it. The company stops being the chip that gets replaced and becomes the platform the replacement plugs into.

Memory Technology Deep Dive

The memory attached to AI chips is as important as the compute:

HBM (High Bandwidth Memory): The standard for AI chips. Multiple DRAM dies stacked vertically and bonded directly to the chip package. Current standard: HBM3e (H200, MI300X, MI350). HBM4 entered production in February 2026 (SK Hynix first, Micron at high volume in March, Samsung ramping in Q2), enabling the Vera Rubin architecture. Only three suppliers: SK Hynix, Samsung, Micron — and all three now ship HBM4.

GDDR7: Consumer GPU memory (gaming cards). Lower bandwidth than HBM, dramatically cheaper. Used in consumer GPUs that also run local AI models (RTX 4090: 24GB GDDR6X; RTX 5090: GDDR7).

On-chip SRAM: The fastest memory, directly on the chip die. Limited by chip area. Used for KV cache and active computation — the bottleneck for inference speed in many models.

CXL (Compute Express Link): Emerging PCIe-based standard for memory pooling and disaggregation — enables connecting additional memory to servers beyond what's physically on the GPU. Important for next-generation AI systems that need to exceed single-server memory limits.

Moving Compute to the Memory

As models grew, the binding constraint on inference shifted from arithmetic to memory — both how much you can attach and how fast you can reach it. Two designs presented at Hot Chips 2026 attack that from opposite ends, and together they show where the industry thinks the problem goes next.

Near-memory compute (Xcena and Samsung, MX1): A CXL expansion card that carries up to 2 terabytes of DDR5 and 3,072 small RISC-V cores on the card itself, so operations run beside the memory instead of dragging every byte back across the link. The arithmetic behind the idea is the point: a link topping out near 128 gigabytes per second would throttle a hypothetical accelerator rated at 8.8 teraflops down to roughly 64 gigaflops if it had to fetch one byte per operation — while doing the work on the card clears 200 gigaflops. Capacity, not speed, is what this buys.

Processing-in-memory (Samsung, LPDDR5X-PIM): Compute units placed inside the DRAM banks of ordinary mobile memory. Inside the chip the sixteen banks reach about 614 gigabytes per second in aggregate, against roughly 76.8 gigabytes per second available through the normal external interface — an eight-fold gap that exists only because the data never leaves the package. Each chip contributes about 1.2 trillion operations per second at 8-bit precision.

Key Takeaways

• NVIDIA dominates AI compute (~80% market share) with hardware performance and the CUDA software ecosystem — 20 years of ecosystem depth is harder to compete with than chip specs
• AMD MI300X is the strongest NVIDIA alternative, particularly for memory-intensive large model inference (192GB vs. H100's 80GB); ROCm ecosystem is improving but still trails CUDA
• Apple reached the M6 generation in August 2026 alongside the M5 Ultra, which holds 512 gigabytes of unified memory at 1.2 terabytes per second — double the M3 Ultra ceiling it replaces, and enough to keep models with hundreds of billions of parameters entirely on device. The M5 Max tops out at 128 gigabytes and remains the practical portable pick, since Ultra chips are desktop-only. There was never an M4 Ultra
• Memory, not arithmetic, is the live constraint on inference — which is why Hot Chips 2026 produced two different attacks on it: near-memory compute on a CXL card (Xcena and Samsung's MX1) and processing-in-memory inside mobile DRAM (Samsung's LPDDR5X-PIM)
• Custom ASICs are maturing fast: Google TPU Ironwood v7 (42.5 Exaflops), Cerebras WSE-3 (2,100 tok/s, IPO Q2 2026), AWS Trainium3 GA (3nm) — all optimize for specific workloads at lower cost
• Custom silicon is no longer purely a substitute for NVIDIA. With the August 2026 MediaTek investment and NVLink Fusion licensing, NVIDIA lets third-party accelerators plug into its rack architecture — so the realistic question for a buyer is less "NVIDIA or a custom chip" and more "whose rack does the custom chip live in"
• HBM4 entered production (February 2026), enabling the next generation of GPU architectures (NVIDIA Vera Rubin, shipping H2 2026)