What it means
Red teaming is adversarial testing: people whose objective is to break the system attempt jailbreaks, prompt injection, extraction of the system prompt or training data, and induction of harmful or off-brand output.
It is distinct from evals, and the distinction is the point. Evals measure whether the system does its job on representative input; red teaming asks what a motivated adversary can make it do on hostile input. A system can pass its evals comprehensively and fall over immediately under attack.
Major labs run red-team programs before frontier releases, and third-party red teaming is increasingly expected — and in some regulatory frameworks required — for high-risk deployments.
Why it matters
The failure modes that damage a brand are almost never the ones normal testing surfaces. Discovering that your customer-facing assistant can be talked into disparaging your product is much cheaper before launch than after a screenshot circulates.
In practice
Have someone who did not build the system try to break it, with an explicit brief to succeed. Feed every success into your eval set so it becomes a permanent regression test rather than a one-time finding.