Learning Objectives
- Compare the leading AI coding models on benchmark performance and practical strengths
- Explain what SWE-bench Verified measures and why it's the relevant coding benchmark
- Apply a selection framework to choose the right model for different coding task types
The Coding Model Landscape
Coding is the AI capability domain with the most rigorous, public benchmarking. The primary benchmark — SWE-bench Verified — presents models with real GitHub issues from popular open-source repositories. Models are scored on how many they can resolve autonomously, without human guidance.
⚠️Warning
Benchmark contamination warning. As of early 2026, OpenAI stopped reporting SWE-bench Verified scores due to concerns about training data contamination across all frontier models. OpenAI now recommends SWE-bench Pro as a cleaner measure. The broader industry still reports Verified scores, but treat all leaderboard positions with appropriate skepticism — real-world evaluation on your own tasks matters more than ever.
💡Key Concept
Where the leaderboard stands in September 2026: Anthropic's Claude Fable 5.1 is the public flagship and the Claude line still leads SWE-bench Pro at 80.3 percent, with Claude Opus 5 (July 24, 2026) close behind at half the price. On the OpenAI side, GPT-6 Astra replaced the GPT-5.6 family as the flagship on September 3, 2026 and leads the agentic command-line and computer-use benchmarks, but has published no SWE-bench Pro score and trails Fable 5.1 on three other published indices. The models this lesson ranked in April — Claude Opus 4.7 and GPT-5.5 — are both still available, but both are now prior generations.
Beyond benchmarks, context window size matters enormously for coding. A model that can hold an entire large codebase in context can reason about cross-file dependencies, understand architectural patterns, and make changes that fit coherently into the existing system. Models with 1 million token context windows can now hold entire large repositories, while those limited to 128-200K tokens require more strategic file selection.
The Leading Models
1. Claude — Best for Complex Agentic Coding
Anthropic holds the strongest documented position on real software-engineering work. Claude Fable 5 is the public flagship and leads SWE-bench Pro at 80.3 percent, the cleanest widely-cited measure of end-to-end issue resolution; it runs $10 in / $50 out per million tokens and falls back to Claude Opus 5 on dual-use cybersecurity and biology requests. Claude Opus 5 (July 24, 2026) is the value pick — Anthropic puts it close to Fable 5's frontier intelligence at half the price ($5 in / $25 out), and its practical edge is token efficiency: comparable results in fewer tokens and fewer turns.
The figures below describe Claude Opus 4.7, the April 2026 model this lesson originally ranked first. It remains available and is a useful reference point for how fast the ceiling moved, but it is two generations back — start from Fable 5 or Opus 5 for new work.
- SWE-bench Verified: 87.6% — the highest score among generally available models at the time, a 6.8-point jump from Opus 4.6
- SWE-bench Pro: 64.3% — leading on the cleaner benchmark too, up from 53.4%
- CursorBench: 70% — up from 58%, measuring real-world IDE coding performance
- 1 million token context window — can hold an entire large repository in context simultaneously; particularly valuable for understanding cross-file dependencies and architectural patterns across hundreds of files
- 3.75 megapixel vision — 3.3x higher resolution than Opus 4.6, enabling better reading of screenshots, diagrams, and documentation images during coding tasks
- xhigh effort level — new default in Claude Code, sitting between high and max for finer control over reasoning depth vs. latency
- Task budgets (public beta) — cap token spend on autonomous agents for predictable cost control
- Agentic strength: Cleaner code output than Opus 4.6 — fewer unnecessary wrapper functions and fallback scaffolding; fixes its own mistakes as it works
- Powers Claude Code, Anthropic's terminal-based coding agent (with Voice Mode, Computer Use, and sub-agent capabilities)
Claude Opus 4.7
Anthropic
Strengths
Best for complex agentic coding; SWE-bench 87.6% / Pro 64.3% / CursorBench 70%; 3.75MP vision; xhigh effort; task budgets
Context Window
1 million tokens
Pricing
$5/$25 per million input/output tokens (API); included in Claude Pro
When to choose: Complex tasks requiring understanding of large codebases, multi-file implementations, debugging across system boundaries, architectural analysis, vision-heavy code review (screenshots, diagrams).
2. GPT-6 Astra — OpenAI's Flagship
GPT-6 Astra was released on September 3, 2026 across ChatGPT paid tiers, the API, Microsoft Azure and AWS Bedrock, replacing the GPT-5.6 family as OpenAI's flagship. Below it, GPT-5.6 still splits into Sol, Terra and Luna tiers, which is where the cost savings now live.
- Terminal-Bench 4.0: 57.9% against GPT-5.6 Sol's 37.3 percent, as reported by OpenAI
- Computer use — 72.6 percent on the OSWorld 2.0 offline set in roughly 47 percent less time per task than Sol
- Cross-context-window notes in Codex — long sessions keep searchable detail instead of compacting it into a summary
- Long context — 96.3 percent on OpenAI's MRCR v2 eight-needle test in the 512K to one-million-token band
- Astra runs $10 in / $50 out per million tokens — double Sol's input rate, so GPT-5.6 Terra and Luna remain the cost-efficient tiers
- Powers the OpenAI Codex platform (desktop app + web at chatgpt.com/codex)
⚠️Warning
Astra does not lead everywhere, and there is still no SWE-bench Pro score. On OpenAI's own comparison table Astra trails Claude Fable 5.1 on the Artificial Analysis Intelligence Index (61.2 against 65.7), on Humanity's Last Exam with tools (57.2 percent against 65.0 percent) and on the Artificial Analysis Coding Agent Index (67.0 against Claude Fable 5's 68.1). Neither Astra nor GPT-5.6 has a published SWE-bench Pro number — the benchmark many engineers weight most for real production work, where Claude Fable 5 leads at 80.3 percent. Every figure above is vendor-reported. For end-to-end resolution of real GitHub issues, Claude remains the better-documented choice.
GPT-6 Astra
OpenAI
Strengths
Computer use (OSWorld 2.0 72.6%); Terminal-Bench 4.0 57.9%; FrontierMath Tier 4 97.6%; cross-context-window notes in Codex
Context Window
Tested to 1 million tokens
Pricing
$10 in / $50 out per million tokens; Fast mode at double
When to choose: OpenAI ecosystem preference; computer-use and browser automation; long agentic runs where losing context to compaction has been the failure mode. Verify on your own workload before moving high-stakes coding onto it, and price-check against the cheaper GPT-5.6 tiers.
3. Gemini 3.1 Pro — Frontier Performance Across All Coding Benchmarks
Gemini 3.1 Pro achieves 80.6% on SWE-bench Verified — effectively tied with Claude Opus for the top position — while excelling across all major coding benchmarks.
- SWE-bench Pro: 54.2% — Terminal-Bench 2.0: 68.5% — LiveCodeBench Pro Elo: 2887
- 1 million token context window — matching Claude Opus 4.7 for full-repository reasoning
- Can analyze and reason over enormous codebases that would require chunking on smaller-context models
- Exceptional for: code review across large PRs, understanding complex legacy systems, cross-repository analysis
- 100+ simultaneous tool calls — parallelized execution for agentic workflows
- Powers Gemini CLI, Google Antigravity IDE, and Google AI Studio
When to choose: Need top-tier coding performance with massive context; cost-sensitive at scale (Flash variants available for lighter tasks); Google Cloud ecosystem preference.
4. GPT-5.3-Codex-Spark — Real-Time Coding Experience
Codex-Spark is a specialized variant optimized for one property: speed.
Running on Cerebras hardware (wafer-scale chip architecture, not NVIDIA GPUs), it delivers:
- 1,000+ tokens per second — approximately 10x faster than standard hosted inference
- Sub-second response for most coding queries
- 128K context window
When to choose: IDE inline completion where latency is felt by the developer; high-volume applications where cost-per-query matters; tight iteration loops where waiting for responses breaks flow.
5. Grok-Code-Fast-1 — xAI's Coding Specialist
Grok-Code-Fast-1 is xAI's agentic coding specialist, integrated natively in Cursor, GitHub Copilot, and Windsurf.
Designed for fast, multi-step coding tasks with deep integration into the developer tooling ecosystem. The integration breadth — appearing in three major AI IDEs — reflects substantial traction among the developer audience. The Cursor integration is especially strategic now that SpaceX has closed its $60 billion purchase of Cursor (August 14, 2026); xAI and Cursor sit in the same SpaceXAI division, pointing to tighter Grok-in-Cursor integration over time.
When to choose: Already using Cursor, Windsurf, or GitHub Copilot and want a fast agentic option without switching tools.
6. Devstral 2 — Mistral's Coding Specialist (Replacing Codestral)
Mistral's Devstral 2 (123 billion dense, 256K context) replaces the original Codestral as Mistral's flagship coding model. It achieves 72.2% on SWE-bench Verified — a major leap from Codestral's fill-in-the-middle focus to full agentic coding capability.
- Modified MIT license (commercial use free under $20 million/month revenue)
- Devstral Small 2 (24 billion, 68.0% SWE-bench) — Apache 2.0, runs locally on consumer hardware
- Powers the new Mistral Vibe CLI (open-source terminal coding agent)
- 80+ programming languages with strong fill-in-the-middle capabilities
When to choose: Teams wanting a strong open-weight coding model with a permissive license; Devstral Small 2 for local/edge deployment; Mistral ecosystem preference.
7. Kimi K2.5 — Best Open-Source Coding Model
Kimi K2.5 from Moonshot AI (released January 2026) is the leading open-source coding model, surpassing DeepSeek V3.2.
- SWE-bench Verified: 76.8% — the highest score among open-source/open-weight models
- 1 trillion MoE architecture (32 billion active parameters) — efficient inference despite massive total size
- Agent Swarm: Can orchestrate up to 100 sub-agents for complex multi-file tasks
- Modified MIT license: free to download and deploy commercially
- DeepSeek V3.2 remains a strong alternative (MIT license, 40%+ improvement over V3)
When to choose: Privacy requirements (data can't leave your infrastructure), cost sensitivity at scale, need for the best open-source coding performance, multi-agent orchestration workflows.
Choosing the Right Model
The model selection decision distills to a few key factors:
| Scenario | Recommended Model |
|---|---|
| Complex multi-file agentic task | Claude Fable 5 (or Claude Opus 5 at half the price) |
| OpenAI ecosystem; reasoning + coding | GPT-5.6 |
| Very large codebase (1 million tokens) | Claude Fable 5, Gemini 3.1 Pro, or GPT-5.6 (all 1 million context) |
| IDE autocomplete; minimal latency | GPT-5.3-Codex-Spark or Devstral Small 2 |
| Open-source / self-hosted requirement | Kimi K2.5 (76.8%) or DeepSeek V3.2 |
| Open-weight; local deployment | Devstral Small 2 (24 billion, Apache 2.0) |
| Already in Cursor / GitHub Copilot | Grok-Code-Fast-1 (via integration) |
📝Note
Benchmarks are necessary but not sufficient. SWE-bench Verified is the most-cited public benchmark, but training data contamination is a growing concern (OpenAI no longer reports Verified scores). SWE-bench Pro is emerging as a cleaner alternative. Regardless of which benchmark you reference, your real-world task distribution will differ. Run your own evaluation on representative tasks from your actual workflow before committing to a primary model.
Key Takeaways
- SWE-bench Verified remains the most-cited coding benchmark, but contamination concerns mean SWE-bench Pro is becoming the cleaner measure — always validate on your own tasks
- Claude Fable 5 leads SWE-bench Pro at 80.3 percent, with Claude Opus 5 close behind at half the price; GPT-5.6 leads the agentic command-line benchmarks but has published no SWE-bench Pro score, and Gemini 3.1 Pro remains a strong competitor — all offer roughly 1 million token context
- Codex-Spark and Devstral Small 2 serve the real-time autocomplete use case where latency is the primary requirement
- Kimi K2.5 (76.8% SWE-bench, 1 trillion MoE) has overtaken DeepSeek V3.2 as the leading open-source coding model; Devstral Small 2 (24 billion, Apache 2.0) is the best option for local deployment