Fluent, Confident, and Wrong: Why LLMs Hallucinate and How Grounding Fixes It

In June 2026, we ran a training session called “Grounding LLMs” six times across our development centers in Vietnam. It is the second session in our internal AI Roadmap, and it exists because of one recurring problem. Engineers who use LLMs every day still get caught off guard when the model states something false with the same fluent confidence it uses for everything true. Understanding why LLMs hallucinate is where the session starts. 

The demo we open with is a legal one. Ask a model for a court case to back up an argument, and it hands you Varghese v. China Southern Airlines Co., Ltd., 925 F.3d 1339 (11th Cir. 2019). The airline is real. The court is real. The citation format is valid. The case never existed. A real lawyer did exactly this in 2023, filed six invented cases in federal court, and was fined $5,000 together with his firm (Mata v. Avianca). He had even asked ChatGPT whether the cases were real, and it said yes.

This blog covers what we teach in that session. What hallucination actually is, why it matters more for professional work than most people assume, and what grounding can and cannot do about it. 

Hallucination is the machinery, not a bug

A hallucination is not a glitch that slipped past quality control. It is the normal machinery doing exactly what it always does. 

Here is what happens to your prompt. The text is broken into tokens. The model predicts the next token from a probability distribution, picks one by a weighted roll, appends it, and repeats until the answer is done. Behind every one of those predictions is a score that measures one thing only, namely how typical the next chunk is in the text the model learned from. Nowhere in that loop is a step that asks whether the claim is true. There is no lookup, no verification, no source in front of the model. 

Truth and plausibility usually overlap, which is why the model is right so often. When they diverge, the model has no way to notice. It optimizes for what sounds like a likely continuation, and sometimes the most likely-sounding words happen to be false. The model cannot tell the difference, because nothing in its architecture ever asks the question. 

This is why “why does it lie to me” is the wrong framing. The model is not deciding to deceive you. It has no step that could separate truth from fiction in the first place. The better question is why we expected it to know the difference.

Why it never says “I don’t know” 

The second half of the puzzle is the confidence. A model that guessed timidly would be less dangerous. Instead it states fabrications in the same assured voice as facts, and the reason sits in how models are scored during training. 

Think of a multiple-choice exam where a blank scores zero and a wrong answer costs nothing. The winning strategy is to never leave a blank. Language models were effectively graded this way. Saying “I don’t know” scored zero points. A confident guess could score whenever it happened to be right. Run the numbers on a 100-question exam where the model is sure of 60 answers and unsure of 40, with four options each. The honest model scores 60. The model that always guesses scores about 70. Bluffing wins, and over millions of training examples the model learned the test-taker’s lesson. 

Recent research (Kalai et al., 2025) makes an important point about this. The bias toward confident guessing lives in the scoring, not in the model’s nature. Change how models are rewarded, and they can be pushed toward admitting uncertainty. That is encouraging for the future, but the default behavior you meet today is a model that would rather make something up than say it does not know. 

One more consequence follows from the training process. A model’s knowledge is frozen at its cut-off date. It cannot know anything that happened afterward, yet it will answer questions about recent events with the very same confidence, a reliable source of hallucinations. 

Why this matters more in prose than in code 

For a software company, there is a distinction worth teaching explicitly. When code hallucinates, it usually breaks loudly. A fabricated API will not compile; a wrong call crashes, and you know within minutes. Prose gives you no such alarm. A fabricated fact in a document, a contract summary, or a compliance report looks exactly like a real one. 

The training deck collects real cases. In 2023, Google’s first big Bard demo made three claims about the James Webb telescope, and the third one, that Webb took the first image of an exoplanet, was false. That image was taken in 2004 by other telescopes. The error in a public demo helped wipe roughly $100 billion off Alphabet’s value in a day. In 2024, Air Canada’s support chatbot invented a refund policy that did not exist, and a tribunal held the airline to the made-up promise. In 2025, the Chicago Sun-Times published an AI-generated summer reading list in which 10 of 15 books were fictional, each attributed to a real author. The same year, Deloitte Australia had to partially refund a government report that cited fabricated references and a made-up court quote. By 2026, more than 1,200 court filings had been caught containing AI-fabricated legal citations. 

The pattern is the same in every case: the output is fluent, and nothing in it warns you which sentence is the invented one.

Grounding changes the input, not the model 

The fix we teach does not involve a bigger or smarter model. It involves changing what goes into the prompt. Instead of asking the model to answer from its own fuzzy memory, you take a real, trusted source, a policy page, a handbook, a set of contracts, and inject it into the prompt alongside the question. The model runs the exact same prediction loop as always, but now the most likely next tokens are drawn from the source sitting in front of it. The answer comes from the page, not from memory. That practice is called grounding.

Figure 1: The prediction loop without and with grounding. No step in the loop checks truth — grounding changes what goes into the prompt, not the model.
Figure 1: The prediction loop without and with grounding. No step in the loop checks truth — grounding changes what goes into the prompt, not the model. 

Run the fake-court-case demo again with grounding and the difference is immediate. Connect the model to the court’s official records, ask the same question, and the invented case simply disappears. The model reports that no matching case appears in the records. Same model, same question. Only the input changed. 

When this is automated, it goes by a name you will hear constantly, RAG, short for retrieval-augmented generation. Three steps. Retrieve the most relevant source for the question, searching by meaning rather than exact keywords, so a question about time off can find a document titled “leave policy.” Augment the prompt by adding that source into it. Generate the answer from it. The source can be your own company documents, an employee handbook, contracts, or a live web search for fresh information. This is how most company AI assistants and document-chat tools work under the hood. 

Does it work? The numbers say yes, with a catch 

The session includes hard data from a 2025 study of a cancer-information chatbot, published in JMIR Cancer, measuring how often it made things up. Answering from its own knowledge with no source, it hallucinated about 40 percent of the time. Grounded on a plain, general web search, about 35 percent, barely an improvement. Grounded on the right source, the specific authoritative medical document, the rate dropped to 6 percent.

Figure 2: Hallucination rates from the 2025 JMIR Cancer chatbot study. A general web search barely helped — the right document cut the rate from 40 to 6 percent.
Figure 2: Hallucination rates from the 2025 JMIR Cancer chatbot study. A general web search barely helped — the right document cut the rate from 40 to 6 percent. 

Two lessons come out of that table. Grounding is powerful: 40 percent is unusable; 6 percent is trustworthy. And it only works with the right source. Throwing any old web page at the model barely helped, which is why the retrieve step is the hard part of building these systems. 

Grounding also has two failure modes worth knowing. The source itself can be wrong. Google’s AI famously told people to put glue on pizza to stop the cheese sliding, because it had retrieved a years-old Reddit joke and repeated it faithfully as fact. Grounding makes the model loyal to the source, which is bad when the source is junk. And even with a good source, the model can misread it. Given real, correct case law, specialized legal AI tools were still wrong 17 to 33 percent of the time (Magesh et al., 2025), misquoting or misapplying documents that were right in front of them. 

The line we want every engineer to remember is that a citation tells you where an answer supposedly came from, not whether it is correct. Grounding lowers the risk dramatically. It does not remove it.

Where grounding stops

One example in the training marks the outer limit of the technique. The viral car-wash test of 2026 asks a model a simple question. I want to wash my car, and the car wash is 100 meters away. Should I drive there or walk? Most models confidently answer “walk,” with tidy reasons about short distances, cold starts, and health. The car, of course, has to be at the car wash to be washed. Across 53 models tested by Opper.ai, only 5 reliably said drive. 

Where grounding stops: the car-wash test

The reason this closes the grounding chapter is that no missing document explains the failure. There is no source you could retrieve that would fix it. The model pattern-matched to the common drive-or-walk question, where walking is usually the wise answer, and missed the one detail that flips it. Grounding fixes the “it doesn’t know the fact” kind of wrong. It does not fix the “it reasoned straight past the obvious” kind. When our trainers last checked, Gemini still said walk while OpenAI’s newest model had started saying drive, so individual models do learn individual tricks, but the underlying limit stands.

What we ask our engineers to practice 

The live sessions, run by our AI coaches in June 2026, ended with concrete homework rather than theory. The discussion in those sessions kept circling the same practical questions. Does stuffing an entire document into the prompt count as grounding? It barely helps, as the 35 percent number shows. Why does a chat tool seem to remember earlier turns when the model itself has no memory? Because the application re-sends the history, which is itself a form of grounding. 

The assignments that came out of those discussions translate directly to project work. When building RAG, invest in retrieval quality instead of dumping whole documents into the prompt, because attention gets diluted and the middle of a long context is poorly attended. Require the model to cite a source for any quantitative or interpretive claim, then check the claim against that source. Design an explicit verification step into any pipeline that produces customer-facing text, because code review has a compiler backing it up and prose review does not. 

The session closes on one habit: give the model a real source to answer from, then verify both the source and the answer against it. No tooling does that last check for you. It buys most of the model’s power at far less risk.

Bring this session to your team

Grounding, retrieval quality, and verification are not features you add at the end. They are habits engineers build through practice. The same AI coaches who ran this session deliver hands-on training and enablement programs tailored to your engineering organization, so your developers learn when to trust an LLM’s output, how to ground it in the right sources, and how to verify the answer before it reaches a customer.

Sources

  • Air Canada chatbot ruling (CBS News, 2024), Chicago Sun-Times AI reading list (NPR, 2025), Deloitte Australia report refund (Fortune, 2025), and the Opper.ai car-wash test (2026)

Frequently Asked Questions

What exactly is an LLM hallucination? 

A hallucination is the model’s normal machinery at work, not a glitch that slipped past quality control. The model predicts the next token by how typical it is in the text it learned from, and no step in that loop checks whether a claim is true. Truth and plausibility usually overlap, which is why the model is right so often; when they diverge, the model has no way to notice.

Why do LLMs sound confident even when they are wrong? 

Because of how models were scored during training. As in a multiple-choice exam where a blank scores zero and a wrong answer costs nothing, a confident guess could score points while “I don’t know” never did, so models learned to bluff. Research from 2025 (Kalai et al.) shows this bias lives in the scoring rather than in the model’s nature, so changed rewards can push models toward admitting uncertainty.

What is grounding, and how does RAG relate to it? 

Grounding means injecting a real, trusted source into the prompt alongside the question, so the model’s most likely next tokens come from the page in front of it instead of its memory. RAG, short for retrieval-augmented generation, is the automated version: retrieve the most relevant source, augment the prompt with it, and generate the answer from it. Most company AI assistants and document-chat tools work this way under the hood. 

How much does grounding reduce hallucinations? 

In a 2025 study of a cancer-information chatbot published in JMIR Cancer, the model hallucinated about 40 percent of the time without a source, about 35 percent when grounded in a general web search, and just 6 percent when grounded in the relevant medical document. This highlights the importance of retrieving the right source, which is often one of the most challenging parts of building reliable RAG systems.

Does grounding remove hallucinations completely? 

No. The retrieved source can itself be wrong, and grounding makes the model loyal to it. Even with correct sources in front of them, specialized legal AI tools were still wrong 17 to 33 percent of the time. Reasoning failures like the car-wash test have no missing document to retrieve at all. Grounding lowers the risk dramatically, and a human still has to verify both the source and the answer.