Learning Objectives
- Compare the leading AI coding models on benchmark performance and practical strengths
- Explain what SWE-bench Verified measures and why it's the relevant coding benchmark
- Apply a selection framework to choose the right model for different coding task types
The Coding Model Landscape
Coding is the AI capability domain with the most rigorous, public benchmarking. The primary benchmark — SWE-bench Verified — presents models with real GitHub issues from popular open-source repositories. Models are scored on how many they can resolve autonomously, without human guidance.
⚠️Warning
Benchmark contamination warning. As of early 2026, OpenAI stopped reporting SWE-bench Verified scores due to concerns about training data contamination across all frontier models. OpenAI now recommends SWE-bench Pro as a cleaner measure. The broader industry still reports Verified scores, but treat all leaderboard positions with appropriate skepticism — real-world evaluation on your own tasks matters more than ever.
💡Key Concept
Where the leaderboard stands in September 2026: Anthropic's Claude Fable 5.1 is the public flagship and the Claude line still leads SWE-bench Pro at 80.3 percent, with Claude Opus 5 (July 24, 2026) close behind at half the price. On the OpenAI side, GPT-6 Astra replaced the GPT-5.6 family as the flagship on September 3, 2026 and leads the agentic command-line and computer-use benchmarks, but has published no SWE-bench Pro score and trails Fable 5.1 on three other published indices. The models this lesson ranked in April — Claude Opus 4.7 and GPT-5.5 — are both still available, but both are now prior generations.
Beyond benchmarks, context window size matters enormously for coding. A model that can hold an entire large codebase in context can reason about cross-file dependencies, understand architectural patterns, and make changes that fit coherently into the existing system. Models with 1 million token context windows can now hold entire large repositories, while those limited to 128-200K tokens require more strategic file selection.
The Leading Models
1. Claude — Best for Complex Agentic Coding
Anthropic holds the strongest documented position on real software-engineering work. Claude Fable 5 is the public flagship and leads SWE-bench Pro at 80.3 percent, the cleanest widely-cited measure of end-to-end issue resolution; it runs $10 in / $50 out per million tokens and falls back to Claude Opus 5 on dual-use cybersecurity and biology requests. Claude Opus 5 (July 24, 2026) is the value pick — Anthropic puts it close to Fable 5's frontier intelligence at half the price ($5 in / $25 out), and its practical edge is token efficiency: comparable results in fewer tokens and fewer turns.
The figures below describe Claude Opus 4.7, the April 2026 model this lesson originally ranked first. It remains available and is a useful reference point for how fast the ceiling moved, but it is two generations back — start from Fable 5 or Opus 5 for new work.
- SWE-bench Verified: 87.6% — the highest score among generally available models at the time, a 6.8-point jump from Opus 4.6
- SWE-bench Pro: 64.3% — leading on the cleaner benchmark too, up from 53.4%
- CursorBench: 70% — up from 58%, measuring real-world IDE coding performance
- 1 million token context window — can hold an entire large repository in context simultaneously; particularly valuable for understanding cross-file dependencies and architectural patterns across hundreds of files
- 3.75 megapixel vision — 3.3x higher resolution than Opus 4.6, enabling better reading of screenshots, diagrams, and documentation images during coding tasks
- xhigh effort level — new default in Claude Code, sitting between high and max for finer control over reasoning depth vs. latency
- Task budgets (public beta) — cap token spend on autonomous agents for predictable cost control
- Agentic strength: Cleaner code output than Opus 4.6 — fewer unnecessary wrapper functions and fallback scaffolding; fixes its own mistakes as it works
- Powers Claude Code, Anthropic's terminal-based coding agent (with Voice Mode, Computer Use, and sub-agent capabilities)
Claude Opus 4.7
Anthropic
Strengths
Best for complex agentic coding; SWE-bench 87.6% / Pro 64.3% / CursorBench 70%; 3.75MP vision; xhigh effort; task budgets
Context Window
1 million tokens
Pricing
$5/$25 per million input/output tokens (API); included in Claude Pro
When to choose: Complex tasks requiring understanding of large codebases, multi-file implementations, debugging across system boundaries, architectural analysis, vision-heavy code review (screenshots, diagrams).
2. GPT-6 Astra — OpenAI's Flagship
GPT-6 Astra was released on September 3, 2026 across ChatGPT paid tiers, the API, Microsoft Azure and AWS Bedrock, replacing the GPT-5.6 family as OpenAI's flagship. Below it sit cheaper GPT-6-generation Sol, Terra and Luna tiers, which is where the cost savings now live.
- Terminal-Bench 4.0: 57.9% against GPT-5.6 Sol's 37.3 percent, as reported by OpenAI
- Computer use — 72.6 percent on the OSWorld 2.0 offline set in roughly 47 percent less time per task than Sol
- Cross-context-window notes in Codex — long sessions keep searchable detail instead of compacting it into a summary
- Long context — 96.3 percent on OpenAI's MRCR v2 eight-needle test in the 512K to one-million-token band
- Astra runs 10 dollars in and 50 dollars out per million tokens, so the lower tiers are the cost-efficient choice — GPT-6.1 Sol at two dollars in, GPT-6.1 Terra (October 26, 2026) at one dollar in and five dollars out, and GPT-6 Luna below that
- Powers the OpenAI Codex platform (the desktop app and the web version inside ChatGPT)
⚠️Warning
Astra does not lead everywhere, and there is still no SWE-bench Pro score. On OpenAI's own comparison table Astra trails Claude Fable 5.1 on the Artificial Analysis Intelligence Index (61.2 against 65.7), on Humanity's Last Exam with tools (57.2 percent against 65.0 percent) and on the Artificial Analysis Coding Agent Index (67.0 against Claude Fable 5's 68.1). Neither Astra nor GPT-5.6 has a published SWE-bench Pro number — the benchmark many engineers weight most for real production work, where Claude Fable 5 leads at 80.3 percent. Every figure above is vendor-reported. For end-to-end resolution of real GitHub issues, Claude remains the better-documented choice.
GPT-6 Astra
OpenAI
Strengths
Computer use (OSWorld 2.0 72.6%); Terminal-Bench 4.0 57.9%; FrontierMath Tier 4 97.6%; cross-context-window notes in Codex
Context Window
Tested to 1 million tokens
Pricing
$10 in / $50 out per million tokens; Fast mode at double
When to choose: OpenAI ecosystem preference; computer-use and browser automation; long agentic runs where losing context to compaction has been the failure mode. Verify on your own workload before moving high-stakes coding onto it, and price-check against the cheaper GPT-5.6 tiers.
3. Gemini 4 Argon — Google's Long-Horizon Coding Model
Gemini 4 Argon, generally available since October 13, 2026, replaced Gemini 3.1 Pro as Google's flagship. Google reports 77.9 percent on DeepSWE, a benchmark of long, real-world software engineering tasks, and ranks it first on that benchmark ahead of GPT-6 Astra and the current Claude flagships. Those results are Google's own and have not been independently reproduced, and Google has published no SWE-bench Pro score for it.
- Output limit of 1 million tokens, up from 64,000, so a single run can carry a long migration or refactor without being restarted
- Google says that inside the company, Argon in Antigravity raised agentic code submissions by more than 35 percent while cutting rollbacks by nearly a third
- Trained to find and patch security vulnerabilities, tying for first on CWE-bench by Google's count
- API at an introductory 2 dollars per million input tokens and 10 dollars per million output through November 30, 2026, then 4 dollars and 20 dollars
- Agentic coding in Google Antigravity requires the Google AI Ultra plan; Pro subscribers can code with it only in the Gemini app's Canvas mode
When to choose: Long, multi-step engineering work such as large refactors and code migrations, or teams already on Google Cloud. Gemini 3.1 Pro remains available, and the Flash line is the cheaper choice for lighter tasks.
4. GPT-5.3-Codex-Spark — Real-Time Coding Experience
Codex-Spark is a specialized variant optimized for one property: speed.
Running on Cerebras hardware (wafer-scale chip architecture, not NVIDIA GPUs), it delivers:
- 1,000+ tokens per second — approximately 10x faster than standard hosted inference
- Sub-second response for most coding queries
- 128K context window
When to choose: IDE inline completion where latency is felt by the developer; high-volume applications where cost-per-query matters; tight iteration loops where waiting for responses breaks flow.
5. Grok-Code-Fast-1 — xAI's Coding Specialist
Grok-Code-Fast-1 is xAI's agentic coding specialist, integrated natively in Cursor, GitHub Copilot, and Windsurf.
Designed for fast, multi-step coding tasks with deep integration into the developer tooling ecosystem. The integration breadth — appearing in three major AI IDEs — reflects substantial traction among the developer audience. The Cursor integration is especially strategic now that SpaceX has closed its $60 billion purchase of Cursor (August 14, 2026); xAI and Cursor sit in the same SpaceXAI division, pointing to tighter Grok-in-Cursor integration over time.
When to choose: Already using Cursor, Windsurf, or GitHub Copilot and want a fast agentic option without switching tools.
6. Devstral 2 — Mistral's Coding Specialist (Replacing Codestral)
Mistral's Devstral 2 (123 billion dense, 256K context) replaces the original Codestral as Mistral's flagship coding model. It achieves 72.2% on SWE-bench Verified — a major leap from Codestral's fill-in-the-middle focus to full agentic coding capability.
- Modified MIT license (commercial use free under $20 million a month in revenue)
- Devstral Small 2 (24 billion, 68.0% SWE-bench) — Apache 2.0, runs locally on consumer hardware
- Powers the new Mistral Vibe CLI (open-source terminal coding agent)
- 80+ programming languages with strong fill-in-the-middle capabilities
When to choose: Teams wanting a strong open-weight coding model with a permissive license; Devstral Small 2 for local and edge deployment; Mistral ecosystem preference.
7. Kimi K2.5 — Best Open-Source Coding Model
Kimi K2.5 from Moonshot AI (released January 2026) is the leading open-source coding model, surpassing DeepSeek V3.2.
- SWE-bench Verified: 76.8% — the highest score among open-weight models
- 1 trillion MoE architecture (32 billion active parameters) — efficient inference despite massive total size
- Agent Swarm: Can orchestrate up to 100 sub-agents for complex multi-file tasks
- Modified MIT license: free to download and deploy commercially
- DeepSeek V3.2 remains a strong alternative (MIT license, 40%+ improvement over V3)
When to choose: Privacy requirements (data can't leave your infrastructure), cost sensitivity at scale, need for the best open-source coding performance, multi-agent orchestration workflows.
Choosing the Right Model
The model selection decision distills to a few key factors:
| Scenario | Recommended Model |
|---|---|
| Complex multi-file agentic task | Claude Fable 5 (or Claude Opus 5 at half the price) |
| OpenAI ecosystem; reasoning + coding | GPT-5.6 |
| Very large codebase (1 million tokens) | Claude Fable 5, Gemini 4 Argon, or GPT-5.6 |
| Long migration or refactor in one run | Gemini 4 Argon (1 million token output limit) |
| IDE autocomplete; minimal latency | GPT-5.3-Codex-Spark or Devstral Small 2 |
| Open-source / self-hosted requirement | Kimi K2.5 (76.8%) or DeepSeek V3.2 |
| Open-weight; local deployment | Devstral Small 2 (24 billion, Apache 2.0) |
| Already in Cursor / GitHub Copilot | Grok-Code-Fast-1 (via integration) |
📝Note
Benchmarks are necessary but not sufficient. SWE-bench Verified is the most-cited public benchmark, but training data contamination is a growing concern (OpenAI no longer reports Verified scores). SWE-bench Pro is emerging as a cleaner alternative. Regardless of which benchmark you reference, your real-world task distribution will differ. Run your own evaluation on representative tasks from your actual workflow before committing to a primary model.
Key Takeaways
- SWE-bench Verified remains the most-cited coding benchmark, but contamination concerns mean SWE-bench Pro is becoming the cleaner measure — always validate on your own tasks
- Claude Fable 5 leads SWE-bench Pro at 80.3 percent, with Claude Opus 5 close behind at half the price; GPT-5.6 leads the agentic command-line benchmarks but has published no SWE-bench Pro score, and Gemini 4 Argon, Google's flagship since October 2026, leads Google's own long-horizon coding results with a 1 million token output limit
- Codex-Spark and Devstral Small 2 serve the real-time autocomplete use case where latency is the primary requirement
- Kimi K2.5 (76.8% SWE-bench, 1 trillion MoE) has overtaken DeepSeek V3.2 as the leading open-source coding model; Devstral Small 2 (24 billion, Apache 2.0) is the best option for local deployment