Why do AI models make things up? Because they are built to produce plausible text, and trained to guess.
Short answer. Because a language model predicts plausible next words rather than looking facts up, and its training and testing reward a confident guess over “I don’t know”. A 2025 research paper, “Why Language Models Hallucinate”, argues that hallucinations arise as ordinary statistical errors when a model cannot tell a true statement from a plausible false one, and that they persist because most benchmarks give no credit for admitting uncertainty. NIST calls the result confabulation: confident, false content. It is most likely in open-ended long answers and specialist topics. Grounding answers in your own documents and demanding sources reduces it but does not remove it: retrieval-based legal tools still hallucinated 17% to 33% of the time in a Stanford study.
01
What actually happens
A language model writes one token at a time, choosing what is statistically likely to come next given the text so far. NIST puts it this way: generative models produce outputs that approximate the statistical distribution of their training data. That can yield factually accurate and consistent output, but it can also yield output that is factually inaccurate or internally inconsistent.
NIST adds that this is particularly relevant for open-ended prompts that ask for long-form responses, and for domains that need highly contextual or specialist expertise.
02
Why guessing is rewarded
In “Why Language Models Hallucinate” (Kalai, Nachum, Vempala and Zhang, September 2025) the authors compare models with students facing hard exam questions: when unsure, they sometimes guess. They argue that hallucinations originate as errors in binary classification: if incorrect statements cannot be distinguished from facts, hallucinations in pretrained models arise through natural statistical pressures. They then argue that hallucinations persist because most evaluations are graded in a way that rewards guessing over acknowledging uncertainty, and they suggest changing how existing benchmarks score uncertain answers.
03
Why citations and retrieval do not cure it
Evidence
Finding
Magesh et al., Stanford, 2024
Legal research tools that retrieve sources still hallucinated between 17% and 33% of the time
Liu, Zhang and Liang, 2023
In four generative search engines, only 51.5% of generated sentences were fully supported by their citations
NIST AI 600-1, 2024
Outputs can include confabulated logic or citations that purport to justify an answer, which may mislead people into trusting it
Gekhman et al., 2024
Fine-tuning a model on new facts raises its tendency to hallucinate as those facts are learned
04
What reduces it
Ground answers in approved documents and show the source passage beside the answer.
Tell the model to say when the answer is not in the documents, then test that it does, using questions that have no answer in your files.
Keep the task narrow: summarise this document, rather than “tell me everything about this client”.
Do not fine-tune to teach facts. Keep facts in documents that can be retrieved and corrected.
Check the specifics every time. The page on checking AI answers sets out a routine.
05
Where ANVE fits
ANVE’s document question answering shows the source passage, every meeting action item must quote the transcript line it came from, and in the agent loop a claim made without a successful tool call is challenged. These make errors easier to see; they do not remove them. ANVE is pre-release.
06
Questions people ask
Is hallucination a bug that will be fixed?
Not entirely. The 2025 paper argues that it follows from how models are trained and scored, and that part of the remedy is changing how mainstream benchmarks score uncertain answers. Expect fewer errors, not none.
Are local models worse at it than ChatGPT?
Smaller models usually score lower on hard tests, but any model can invent details. What matters is whether the system grounds answers in sources and whether people check them.
What is the difference between hallucination and confabulation?
NIST uses “confabulation” and notes that the phenomenon is colloquially called hallucination or fabrication. Both mean confident, false output.
Can I just ask the model whether it is sure?
Not reliably. NIST notes that models sometimes give logical steps for how they reached an answer even when the answer is wrong, so a confident explanation is not evidence. Check the source instead.
07
What is not real yet
No sealed ANVE unit exists; every image of the appliance on this site is a labelled render. The software runs today on a development machine. No customers yet, no certifications, no delivery date, and the final price is not announced.
Reservations are ₹999 in India or US$12 elsewhere, refundable in full until you confirm an order in writing (terms). Offices that cannot use cloud AI can apply to be one of ten design partners.
