Hallucination & grounding.
A model that invents a plausible answer sounds exactly like a model that knows one — that is the whole problem, and no amount of "be accurate" in the prompt fixes it. This entry explains why fabrication is a structural property of how these systems generate text, not a bug awaiting a patch, and what grounding actually buys you.
Fluency and truth are separate axes.
A large language model is trained to produce the most likely next token, not the most likely true next token. Nothing in that objective distinguishes a correct citation from a well-formed fake one — both are fluent sequences, and fluency is what the model optimizes.
Two consequences follow, and they explain most of what people find surprising:
- There is no built-in "I don't know." Asked something outside its knowledge, the model does not hit a missing row and return null — it continues, because continuing is all it does. The output is the most plausible continuation, and plausible is a very low bar for a system that has read the shape of a million correct answers.
- Confidence is a style, not a signal. The assured tone of a fabricated answer is borrowed from the assured tone of the correct answers in training data. You cannot read reliability off the prose, which is why "it sounded confident" is worthless as evidence.
"Hallucination" is the conventional name for this. It is a slightly misleading one — it suggests a malfunction, when the mechanism producing the false answer is the same mechanism producing the true ones.
Three kinds you will actually meet.
Lumping them together hides the fact that they have different fixes:
- Fabrication from memory. The model answers from what it absorbed in training and gets it wrong — an invented statistic, a library function that does not exist, a paper with a real author and a fake title. Most common on specifics: numbers, names, dates, versions, URLs.
- Unfaithfulness to provided context. You hand the model a document and it summarizes something the document does not say — it blends in prior knowledge, over-generalizes, or resolves an ambiguity by inventing. This one matters most because it is the failure mode of RAG: retrieval worked, generation still lied.
- Confabulated structure. The model produces a well-shaped object whose contents are fiction — an API call with plausible-looking parameters that the real API does not accept, a JSON record with an invented ID. Schema-constrained output guarantees the shape, never the contents; a validator that passes tells you nothing about whether the values are real.
Rate rises predictably with distance from the training distribution, with specificity of the ask, and with how long the answer is. Long answers are compound bets: every additional claim is another chance to be wrong.
Grounding: make the answer depend on something you control.
Grounding is the practice of putting the evidence in front of the model and requiring the answer to come from it, rather than from the weights. It is the single highest-leverage intervention, and it has three parts that people routinely do only one of:
- Supply the source. Retrieve the relevant documents, call the tool, query the database — put the ground truth in the context window rather than hoping it is in the weights. This is what RAG and tool calling are for.
- Constrain the answer to it. Instruct explicitly: answer only from the provided material, and if the answer is not there, say so. Giving the model an approved exit — an explicit "not found in the sources" — matters, because a model with no permitted way to fail will invent a way to succeed.
- Demand attribution. Require each claim to cite the passage it came from. Citations are not decoration: they convert an unverifiable paragraph into a set of claims a human or a script can spot-check, and the requirement itself measurably suppresses invention.
Grounding reduces fabrication sharply; it does not eliminate it. The model can still misread a passage, merge two sources, or cite a real document for a claim the document does not make. Treat a grounded answer as checkable, not as checked — the value is that verification became cheap, not that it became unnecessary.
Detection, and why agents raise the stakes.
Since you cannot prevent it outright, budget for catching it:
- Verify against the source, not against the model's confidence. Where a claim is machine-checkable — a number that must appear in the document, a function that must exist in the library, an ID that must resolve — check it in code. A cheap deterministic check beats an expensive probabilistic one.
- Cross-check by sampling. Ask the same question several times; claims that change between samples are the ones the model is least sure of. This catches fabrication precisely because inventions are unstable while recalled facts are not.
- Score faithfulness explicitly. In an eval set, "is this supported by the retrieved context?" is a different question from "is this the right answer," and a system can pass one while failing the other. Measure both.
In an agent, a hallucination is not a wrong sentence — it is a wrong action. An invented file path gets written to; a fabricated record ID gets updated; a misremembered parameter gets executed against a live system. Worse, the fabricated result re-enters the context as an apparent observation and the loop builds on it. This is the argument for validating tool arguments against the real environment before executing them, rather than trusting that a schema-valid call is a meaningful one.
The practical stance: assume some fraction of any answer is invented, ground everything you can, make the rest cheap to verify, and reserve human attention for the claims where being wrong actually costs something. The evaluating RAG deep-dive covers faithfulness and groundedness metrics in production detail, and the LLM-as-judge operations chapter covers automating the checks at scale.