Eval

An eval is a set of representative inputs plus a way of scoring the outputs, run against your own system. Where a benchmark compares models in general, an eval measures whether *your* application does *your* job.

It exists because AI systems break the normal software testing model. Outputs are non-deterministic and often have no single correct answer, so conventional assertions do not apply. Evals substitute graded quality: exact matching where possible, rule-based checks where not, and model-graded scoring for subjective qualities, usually with human spot-checks to keep the grader honest.

Without evals, changing a prompt is guesswork — you improve one case and silently break three others, with no way to notice. With them, you can compare models, prompts and retrieval strategies on evidence.

This is the single practice that most separates AI projects that make it to production from ones that stall in demo. It's also how you catch regressions when a vendor silently updates a model beneath your product.

Start absurdly small. Twenty real examples with expected outcomes, run before every prompt change, already beats intuition. Grow the set every time something fails in production — a failure that becomes a test case cannot silently return.