Nemotron

Learning Objectives

• Understand what the Nemotron model family is and how it fits into NVIDIA's AI strategy beyond hardware
• Compare the Nemotron models and identify their intended use cases
• Evaluate when Nemotron models are the right choice versus other open-weight alternatives

What Is Nemotron?

Nemotron is NVIDIA's family of open-weight large language models, developed by NVIDIA Research. While NVIDIA is known for GPU hardware, Nemotron represents the company's push into the model layer — competing directly with Meta's Llama, Google's Gemma, and Microsoft's Phi in the open-weight model ecosystem.

The Nemotron family serves a specific strategic purpose: demonstrating and optimizing the full NVIDIA AI stack. Models are trained on NVIDIA hardware, optimized for NVIDIA inference infrastructure (TensorRT-LLM, NIM), and designed to showcase what's possible when hardware and software are tightly integrated.

The Nemotron Model Family

Nemotron 3.5 Lightning

NVIDIA's newest release and its clearest statement about where it thinks open models earn their keep: a 30 billion parameter mixture-of-experts (MoE) model built not to top a leaderboard but to be the fast worker inside a multi-agent system. NVIDIA reports roughly four-times faster output speed than other models in its class, translating to about 30 percent faster completion on agentic tasks — the distinction matters, because raw token throughput and end-to-end task time are not the same number.

Lightning is explicitly built to be post-trained on an organization's own data rather than used as shipped, and it runs across an unusually wide hardware span for a single model — RTX PCs, DGX Spark, DGX Station, Jetson devices, RTX PRO workstations, data centers and cloud. NVIDIA also published the Nemotron-RL-Agentic-Terminal-Pivot dataset used for agentic reinforcement-learning training, which is the more useful half of the release for anyone building their own agent stack. Weights are on Hugging Face, ModelScope and OpenRouter, and it is served as an NVIDIA NIM microservice through build.nvidia.com.

Nemotron 3 (Nano, Super, Ultra)

NVIDIA's newest generation and current flagship line, built on a hybrid Mamba-Transformer architecture. The larger Super and Ultra sizes add Latent MoE — a hardware-aware mixture-of-experts (MoE) expert design — and the whole family is trained in NVIDIA's 4-bit NVFP4 format on Blackwell GPUs, with a 1-million-token context window across all three sizes:

• Nano — roughly 31 billion total parameters with about 3 billion active per token. Tuned for efficient, high-throughput inference and the first size to ship.
• Super — roughly 100 billion total parameters with about 10 billion active per token.
• Ultra — the flagship, roughly 550 billion total parameters with about 50 billion active per token. NVIDIA CEO Jensen Huang unveiled Ultra at his Computex keynote in Taipei, positioning it as the smartest US open-weights model: NVIDIA reports it leads US open-weights rankings on the Artificial Analysis Intelligence Index, generates more than 300 tokens per second, and runs roughly 30 percent cheaper than leading alternatives while delivering up to five-times faster inference.

Weights and training recipes ship free under the NVIDIA Open Model License, optimized for deployment through NIM containers and TensorRT-LLM. Nemotron 3 is aimed squarely at agentic coding, search, and long-context reasoning workloads.

Koa — the first major third-party model built on Nemotron

On September 15, 2026 Salesforce announced Koa, its first CRM reasoning model, built by post-training Nemotron 3 Super with supervised fine-tuning and reinforcement learning using Group Relative Policy Optimization, on NVIDIA's NeMo RL, NeMo Gym and NeMo AutoModel stack.

The reason Salesforce gave is the more interesting part, because it is the open-weights argument made by a buyer rather than by NVIDIA. Jayesh Govindarajan, who runs Salesforce AI, said the blocker had never been ambition but the absence of a sovereign American pre-trained model that was state of the art and had clear data provenance — adding, of the leading Chinese open-weight alternative, "we have no idea what Qwen trains on." Salesforce holds Koa's weights and runs both post-training and inference inside its own trust boundary, which is precisely the deployment pattern an open-weight license exists to permit and a hosted frontier API cannot offer.

Nemotron-4 340 Billion

NVIDIA's earlier flagship open model, trained on 9 trillion tokens. Available in three variants:

• Base — Pre-trained foundation model for further fine-tuning
• Instruct — Chat-optimized for direct use in conversational applications
• Reward — Specialized for scoring and filtering synthetic training data; widely used in RLHF pipelines

The Reward model is particularly notable — it has become a standard tool for teams building synthetic data pipelines, where it scores AI-generated training examples to filter out low-quality outputs before they contaminate training sets.

Llama-3.1-Nemotron-70 Billion-Instruct

A fine-tuned version of Meta's Llama 3.1 70 billion parameter model, enhanced using NVIDIA's Nemotron reward model and RLHF techniques. At release, it outperformed GPT-4o on several benchmarks — demonstrating that fine-tuning expertise can be as important as raw model scale.

This model follows a practical pattern: take the best available open-weight base (Llama), apply superior fine-tuning techniques, and produce a derivative model that exceeds the original. It runs on a single high-end GPU (H100 or A100 with 80GB memory).

Minitron (8 Billion and 4 Billion)

Smaller models derived from Nemotron-4 15 billion through pruning and knowledge distillation — NVIDIA's research into making large models smaller without proportional quality loss. These target edge deployment and cost-sensitive inference.

Nemotron-Mini-4B-Instruct

Designed specifically for on-device and edge deployment. Pairs naturally with NVIDIA's Jetson hardware platform for embedded AI applications where cloud connectivity is unavailable or undesirable.

Nemotron Elastic 30 Billion (May 2026)

The newest and most architecturally interesting addition. Nemotron Elastic is a single 30 billion-parameter reasoning-model checkpoint that contains 30 billion, 23 billion, and 12 billion-parameter nested submodels — extractable at inference time via zero-shot slicing with no further fine-tuning required. One file ships, three model sizes deploy.

The recipe was published by NVIDIA Research with three concrete wins:

• 360-times token reduction over training the three sizes from scratch — the elastic post-training run consumed roughly 160 billion tokens versus the multi-trillion-token equivalent for three independent pretraining runs
• 18.7 gigabytes for the 30 billion checkpoint under NVFP4 quantization (NVIDIA's 4-bit floating point format) — small enough to fit on a single consumer-class GPU
• Up to 16 percent higher accuracy and 1.9-times lower latency in the 23-to-30 billion configuration compared to Nemotron Nano v3's default budget control. The accuracy-and-latency Pareto frontier moves outward across all three sizes.

The intuition behind elastic slicing: reasoning tokens are high-volume but tolerant of some capacity reduction, while the final answer requires higher precision. Nested QAD (quantization-aware distillation) preserves the slicing property even after dropping to FP8 or NVFP4, so a single quantized checkpoint serves all three sizes at all three precisions (BF16, FP8, NVFP4) — nine deployment configurations from one training run.

Practical impact: deployment teams that want to A-B test a smaller model against a larger one no longer need two separate fine-tuning runs and two model artifacts. Pull the same checkpoint, slice differently per request based on task complexity. The model card and quantized variants are on Hugging Face under nvidia/NVIDIA-Nemotron-Labs-3-Elastic-30B-A3B.

Nemotron Diffusion 14 Billion (May 2026)

The newest variant, posted to Hugging Face in late May 2026 alongside an NVIDIA Labs technical report. Nemotron Diffusion is a 14 billion-parameter language model that switches between three decoding modes — autoregressive, parallel diffusion, and a "self-speculation" mode that drafts with diffusion and verifies with autoregression — all without changing model weights or attention patterns.

The throughput numbers from NVIDIA's published benchmarks:

• 2.2-times faster than the comparable Qwen 3 8 billion-parameter baseline at matched accuracy in self-speculation mode
• 850 tokens per second on a GB200 system — a 3.3-times lift over the baseline
• 5.9-times tokens per forward pass versus standard Qwen 3 8 billion with matching accuracy, by reusing model weights to compute multiple tokens per step

Base, instruct, and vision-language variants all ship open-weight. The architecture is positioned as a path from memory-bound to compute-bound inference as GPUs continue to outrun memory bandwidth — a structural concern that has been growing as HBM capacity per accelerator climbs slower than raw compute. The model card lives at nvidia/Nemotron-Labs-Diffusion-14B.

NeMo Switchyard — Routing Between Models

Shipped alongside Nemotron 3.5 Lightning, NeMo Switchyard is an open-source routing library that sends each request to whichever model best fits it on quality, latency or cost — the software layer that makes a small fast model like Lightning useful next to an expensive frontier one rather than a replacement for it. NVIDIA's internal benchmarks claim it holds frontier-level accuracy while cutting task-completion cost to roughly one-third of running Claude Opus 4.8 alone.

That is a vendor benchmark against a specific competitor model, so treat the ratio as directional. The architectural point stands on its own though, and it is the reason routing is having a moment: most production requests do not need a frontier model, and the expensive part of an agent workload is usually the calls that could have been cheaper. Switchyard is on GitHub, with partner-platform integrations still to come.

Access

Strengths

• Frontier open-weights flagship — Nemotron 3 Ultra (~550 billion total parameters, ~50 billion active) leads US open-weights rankings while running roughly 30 percent cheaper than leading alternatives, with a 1-million-token context window across the whole Nemotron 3 line
• Tightly optimized for NVIDIA hardware — models ship with TensorRT-LLM optimizations and NIM containers out of the box
• Reward model for synthetic data — Nemotron-4 Reward is widely used beyond NVIDIA's own models for scoring synthetic training data
• Full-stack demonstration — showcases NVIDIA's training, fine-tuning, and inference capabilities end to end
• Range of sizes — from 4 billion (edge) to 340 billion (datacenter), covering diverse deployment scenarios
• Strong fine-tuning results — Llama-Nemotron-70 billion demonstrates benchmark-leading performance through fine-tuning alone
• Elastic post-training (May 2026) — Nemotron Elastic 30 billion packs three nested model sizes in one checkpoint with zero-shot slicing, collapsing what used to be three separate training runs into one
• Tri-mode decoding (late May 2026) — Nemotron Diffusion 14 billion ships a single architecture that switches between autoregressive, diffusion, and self-speculation modes for a 2.2-times throughput lift over the comparable Qwen 3 baseline at matched accuracy

Limitations & Considerations

• Ecosystem is smaller than Llama/Gemma — fewer community fine-tunes, fewer third-party tutorials, less Stack Overflow coverage
• NVIDIA hardware advantage — models are most optimized for NVIDIA GPUs; running on AMD or other hardware loses the performance edge
• Licensing varies by model — Nemotron-4 uses NVIDIA's own license (check commercial terms); Llama-Nemotron inherits Meta's license
• Large model sizes — the 340 billion parameter flagship requires significant GPU infrastructure to run
• Less brand recognition — NVIDIA is known for hardware; developers may overlook Nemotron when choosing open models

Key Takeaways

• Nemotron 3.5 Lightning is a 30 billion parameter mixture-of-experts (MoE) model built to be the fast worker inside a multi-agent system rather than a frontier competitor — NVIDIA reports roughly four-times faster output and about 30 percent faster agentic task completion, and ships it alongside the agentic reinforcement-learning dataset used to train it
• NeMo Switchyard is the companion open-source routing library that picks a model per request on quality, latency or cost; NVIDIA's internal benchmark puts task cost at roughly one-third of running Claude Opus 4.8 alone, which is a vendor figure but points at a real pattern — most production requests do not need a frontier model
• The Nemotron 3 generation (Nano, Super, Ultra) is NVIDIA's open-weight flagship family — a hybrid Mamba-Transformer architecture with Latent MoE and a 1-million-token context; the 550 billion parameter Ultra flagship launched at Computex as the top-ranked US open-weights model, running roughly 30 percent cheaper than leading alternatives
• Nemotron is NVIDIA's open-weight model family spanning edge-scale models to datacenter-class flagships — from 4 billion parameters up to the 550 billion parameter Nemotron 3 Ultra
• The Nemotron-4 Reward model has found broad adoption for scoring synthetic training data in RLHF pipelines, extending its impact beyond direct model use
• Llama-3.1-Nemotron-70 billion demonstrates that fine-tuning expertise can produce models that outperform larger competitors on key benchmarks
• Nemotron Elastic 30 billion (May 2026) ships three nested model sizes (30, 23, 12 billion) in a single checkpoint with zero-shot slicing — a 360-times token reduction over training the sizes independently, with NVFP4 quantization shrinking the 30 billion checkpoint to under 19 gigabytes
• Nemotron Diffusion 14 billion (late May 2026) is the newest variant — a tri-mode architecture (autoregressive, diffusion, self-speculation) that delivers a 2.2-times throughput lift over the comparable Qwen 3 baseline at matched accuracy, scaling to 850 tokens per second on a GB200
• Nemotron models are most compelling when deployed on NVIDIA infrastructure (NIM + TensorRT-LLM), where tight hardware-software integration delivers maximum performance