What it means
A language model generates plausible continuations, and a fabricated citation is every bit as plausible as a real one. So hallucination is not a bug in the ordinary sense — it is the same mechanism that produces correct answers, operating where the model lacks grounding. That is why it cannot simply be patched out.
The term is contested; some researchers prefer "confabulation", on the grounds that hallucination implies faulty perception when the system is really producing confident narrative to fill a gap. Either way, the defining property is the absence of a confidence signal: the model gives no outward indication that it has moved from recall to invention.
Why it matters
This is the single biggest obstacle to deploying AI in high-stakes work, and the risk is inverted from normal software. Conventional systems fail loudly; a hallucinating model fails silently and persuasively, so errors reach production precisely when the reviewer is least prompted to check. Cases of fabricated legal citations reaching filed court documents are the canonical example.
In practice
Grounding is the mitigation that works: retrieval with citations, so answers trace to a source a human can check. The organizational counterpart matters as much — reviewers need to know that fluency carries no information about accuracy, because the intuition everyone brings from human writing points the wrong way.