What it means
Benchmarks are fixed test sets with scoreable answers, spanning graduate-level questions, competitive programming, mathematics, and agentic software tasks. They are how vendors substantiate launch claims and how the field tracks progress.
They have well-documented weaknesses. Contamination is the central one: benchmarks are public, so their contents leak into training data and a model may have effectively seen the answers. Benchmarks also saturate — once scores cluster near the ceiling, the test stops discriminating. And because leaderboard position drives sales, there is direct pressure to optimize for the test rather than the underlying ability.
Human-preference arenas, where people vote blind between two models' answers, address some of this but introduce their own bias toward answers that read well.
Why it matters
Benchmark scores are marketing artifacts as much as measurements, and the gap between "scores well" and "works for us" is where most disappointing AI deployments live. Two models a point apart on a public leaderboard can differ substantially on your actual task, in either direction.
In practice
Use benchmarks to build a shortlist, never to make the decision. Then run your own eval on your own data — twenty representative examples from your workload tell you more than any leaderboard.
Where this shows up
Tools and models in our catalog.
