Hallucination

A language model generates plausible continuations, and a fabricated citation is every bit as plausible as a real one. So hallucination is not a bug in the ordinary sense — it is the same mechanism that produces correct answers, operating where the model lacks grounding. That is why it cannot simply be patched out.

The term is contested; some researchers prefer "confabulation", on the grounds that hallucination implies faulty perception when the system is really producing confident narrative to fill a gap. Either way, the defining property is the absence of a confidence signal: the model gives no outward indication that it has moved from recall to invention.

This is the single biggest obstacle to deploying AI in high-stakes work, and the risk is inverted from normal software. Conventional systems fail loudly; a hallucinating model fails silently and persuasively, so errors reach production precisely when the reviewer is least prompted to check. Cases of fabricated legal citations reaching filed court documents are the canonical example.

That it means the model is lying, or malfunctioning. Neither. Lying requires knowing the truth and choosing otherwise; a model has no separate store of facts to consult and contradict. And nothing is broken — the same next-token machinery that produces correct answers produces fabricated ones, operating where it has no grounding. That is why hallucination cannot be patched out the way a bug can.

That a confident tone means a reliable answer. This is the costly one, because the intuition is well-earned everywhere else: in human writing, hedging usually signals uncertainty and fluency usually signals command of the subject. In a language model the two are unlinked. Fluency is what the system optimizes for; accuracy is not something it can measure about itself, so it has no way to sound less sure when it should be.

That lowering the temperature fixes it. Temperature controls randomness, not truth. Set it to zero and a model that does not know something will give you the same wrong answer every time — arguably worse, because consistency reads as reliability.

Grounding is the mitigation that works: retrieval with citations, so answers trace to a source a human can check. The organizational counterpart matters as much — reviewers need to know that fluency carries no information about accuracy, because the intuition everyone brings from human writing points the wrong way.

The practical tell is specificity without a source: exact figures, named citations, precise dates and quoted passages are where fabrication concentrates, because those are the details a plausible continuation supplies most readily. Verify the specifics, not the shape of the answer.