Edge AI

Learning Objectives

• Explain the five primary motivations for edge AI deployment and when each is the deciding factor
• Identify the leading models and frameworks for local inference
• Evaluate which hardware options support local model inference at the consumer and professional levels

The Cloud vs. Edge Tradeoff

Every AI application faces a fundamental architectural choice: where does inference happen?

Cloud inference: Send a request to a remote API, receive a response. Simpler to build, access to frontier models, scales automatically, no local hardware required. Costs per query, requires internet connectivity, data leaves the device.

Edge inference: The model runs locally — on a laptop, a server in your building, an on-device chip in a phone. More complex to set up, limited to models that fit available hardware, but: private by design, no latency beyond local compute, zero marginal cost after hardware purchase, works offline.

The right architecture depends on your requirements. Cloud inference is the right default. Edge becomes the right choice when specific constraints make cloud inference unsuitable.

Five Reasons to Go to the Edge

1. Privacy

When data never leaves the device, there is no privacy concern. The model processes your input locally and produces output locally — nothing is transmitted to a third party.

This matters for:

• Healthcare applications processing patient information
• Legal work with privileged documents
• Financial analysis with confidential data
• Personal assistants that should never send your private conversations to external servers
• Enterprise deployments in industries with strict data handling regulations

2. Latency

Cloud round-trips add latency: 50-500ms per request depending on geographic distance and server load. For real-time applications, this is prohibitive:

• Voice assistants where conversational delay is jarring
• Autonomous vehicles making split-second decisions
• AR/VR applications where AI response must be imperceptible
• Industrial control systems with hard real-time requirements

Edge inference can process in under 10ms on capable hardware — 10-50x lower latency than cloud.

3. Cost at Scale

Cloud inference costs accumulate: at $0.003 per thousand tokens (Haiku 4.5 pricing), a million daily queries cost thousands per month. For applications with sufficient query volume, local hardware can pay for itself in months.

Consumer hardware example: an RTX 4090 (~$2,000) running Llama 4 Scout locally eliminates ongoing API costs for that workload. Break-even calculation: if you'd otherwise spend $500/month on inference, the hardware pays back in 4 months.

4. Reliability

Edge AI works offline. Cloud AI requires internet connectivity. For:

• Aircraft and ships in transit
• Remote industrial equipment
• Medical devices in facilities with limited connectivity
• Disaster response scenarios
• Any application where connectivity is intermittent

Edge inference provides the reliability guarantee that cloud cannot.

5. Data Sovereignty

Some countries require data to remain within their geographic boundaries. Some industries have regulations specifying where data can be processed. On-premise deployment satisfies requirements that cloud APIs — which may route through data centers in other jurisdictions — cannot guarantee.

Edge LLMs (Local Inference Models)

Not every model is designed for edge deployment. The best edge models are small, efficient, and specifically optimized for local inference:

Phi-4 Family (Microsoft, MIT License)

Microsoft's Phi-4 series has expanded significantly — now a full family of edge-optimized models:

• Phi-4 Mini (3.8 billion): Optimized for mobile devices; fits in 8GB of GPU memory
• Phi-4 Multimodal (5.6 billion): Handles text, images, and audio on-device
• Phi-4 Reasoning and Phi-4 Reasoning Plus: Specialized for chain-of-thought reasoning at small scale
• Phi-4 Reasoning Vision 15 billion (March 2026): Vision + reasoning capabilities in a single model
• All MIT license: free for commercial deployment
• Works with ONNX Runtime and DirectML for Windows NPU acceleration

Apple MLX

MLX is Apple's machine learning framework for M-series chips. It's not a model — it's a framework that enables efficient inference of popular models on Apple Silicon. Recent versions add Neural Accelerator support for the M5 generation, leveraging its roughly 4 times AI performance improvement over M4. Models converted to MLX format often run faster on recent Apple Silicon than in llama.cpp or other frameworks on the same hardware.

Llama, Gemma, Mistral, and Phi models are all available in MLX-converted form on Hugging Face.

Llama 4 Scout (Meta, Open License)

Llama 4 Scout (17 billion active parameters / 16 experts MoE, 10 million token context) is Meta's edge-friendly model — designed to be deployable on a single high-end GPU. The Scout variant (vs. Maverick with 128 experts) prioritizes deployability over maximum capability. Both are natively multimodal (text + images).

Strong performance on most knowledge and reasoning tasks; the active open-source community has produced optimized inference configurations for multiple hardware targets.

Gemma Family (Google, Apache 2.0)

Google's open small-model line now spans two live generations, and for edge work you will reach for both:

• Gemma 4 E2B / E4B: the current edge tier — the "E" is effective parameters, roughly 2 billion and 4.5 billion, built for phones and embedded targets
• Gemma 4 12B: laptop-class and natively multimodal, taking text, image and audio in one model at around 16 gigabytes
• Gemma 3 270 million: still the ultra-compact option — Gemma 4 does not go this small, so the older generation remains the right pick for the most constrained hardware
• FunctionGemma (270 million): specialized for agent tool-calling — enables on-device AI agents
• 128K context on the small models, rising to 256K on the mid-sized ones
• Strong multilingual support
• Available in quantized versions for memory-constrained deployments

The combination of multilingual depth, long context, and a size ladder running from a 270 million-parameter microcontroller target up to a 31 billion dense model makes Gemma the most versatile edge family from any provider.

Ollama — The Easiest Path to Local Models

Ollama is not a model — it's a tool that makes running local models as simple as a single terminal command:

# Install Ollama (macOS/Linux)
curl -fsSL https://ollama.com/install.sh | sh

# Download and run Llama 4 Scout
ollama run llama4:scout

# Run DeepSeek R1
ollama run deepseek-r1:7b

# Run Gemma 4
ollama run gemma4:26b

Ollama (v0.18.2) handles model downloading, quantization selection, CPU/GPU configuration, and presents a simple REST API. Recent additions include web search (agents can search the web), cloud model support (connect to remote APIs), and Windows ARM64 native builds:

import requests
response = requests.post("http://localhost:11434/api/generate",
    json={"model": "llama4:scout", "prompt": "Explain gradient descent"})

For any developer experimenting with local models, Ollama is the right starting point. It works on Mac (M-series), Linux, and Windows.

Edge AI Hardware

NVIDIA DGX Spark GB10

NVIDIA's DGX Spark is a desktop-form-factor AI workstation, now shipping at $4,699 (raised from the $3,999 announcement price):

• 128GB of unified memory
• 1 PFLOPS of AI performance
• Runs frontier-class models (Llama 4 Maverick, DeepSeek V3) locally

Targeted at AI researchers and professionals who need powerful local inference for privacy-sensitive or cost-sensitive workloads without building a full server infrastructure.

Apple Mac Studio (M5 Max and M5 Ultra)

Announced August 25, 2026, the Mac Studio is the most capable consumer-priced hardware for local inference, and the reason is memory capacity rather than raw compute:

• M5 Ultra: up to 512 gigabytes of unified memory at 1.2 terabytes per second, from $5,499
• M5 Max: up to 128 gigabytes at 614 gigabytes per second, from $2,499
• Apple claims 4.5 times the peak graphics compute for AI against the M3 Ultra

The capacity number is the one that changes what is possible. The previous Ultra ceiling was 256 gigabytes on the M3 Ultra, so the addressable model size on a single desktop machine has doubled — Apple says the top configuration runs models with hundreds of billions of parameters entirely on device. Note there was never an M4 Ultra; Apple skipped that generation and went from the M3 Ultra straight to the M5 Ultra.

The trade-off against a discrete-GPU workstation is unchanged: unified memory buys you capacity cheaply, while raw throughput per dollar still favors dedicated accelerators. A 512-gigabyte Mac Studio runs models an NVIDIA workstation at the same price cannot hold at all, and runs them more slowly than that workstation runs the models it can hold.

For a portable machine, the MacBook Pro with an M-series Max chip and 128 gigabytes remains the practical choice — the Ultra chips are desktop-only.

Windows AI PCs (Copilot+ PCs)

Intel and AMD have introduced Copilot+ PCs with dedicated Neural Processing Units (NPUs). Microsoft is now shifting priority from NPU to GPU/CPU for AI workloads, as discrete GPUs deliver more capable inference:

• NPUs still handle small model inference (Phi-4 Mini, Whisper) efficiently
• Recall (controversial screen recording feature) is now fully opt-in
• Intel Panther Lake processors target 50 TOPS NPU performance
• More limited than discrete GPU setups for large models — the NPU is supplemental, not primary

The NPU category continues to mature — practical for always-on features (transcription, local assistant) but not for frontier model inference.

Consumer NVIDIA GPUs

The RTX 4090 (24GB GDDR6X) and RTX 5090 (32GB GDDR7) are the best consumer NVIDIA options for local inference:

• More VRAM than most workstation GPUs at the price
• Fast enough for real-time inference on most 7 billion-13 billion models
• Can run quantized versions of larger models (30 billion-70 billion with 4-bit quantization)

For developers who already have a gaming-capable GPU, local model inference is often possible without additional hardware investment.

Key Takeaways

• Edge AI is the right choice when privacy, offline reliability, latency, cost at scale, or data sovereignty requirements make cloud-only inference unsuitable — not as a default
• The leading edge LLMs have expanded: Phi-4 family (6 variants from 3.8 billion mobile to 15 billion reasoning-vision); Llama 4 Scout (17 billion MoE, 10 million context); the Gemma 4 family (E2B and E4B for edge, a 12 billion multimodal mid-size), with Gemma 3's 270 million compact model and FunctionGemma still the smallest options since Gemma 4 does not go that small; DeepSeek R1 for open-source reasoning
• Ollama is the fastest path to experimenting with local models: one installation command, then ollama run [model-name] to download and run any supported model
• Hardware options range from M4 Max MacBook Pro (128GB unified, excellent consumer option) to NVIDIA DGX Spark (purpose-built desktop AI workstation) to consumer RTX cards for development use