Hallucination When a model sounds sure and is wrong.
Short answer. A hallucination is a statement from an AI model that is fluent and confident but false or unsupported, such as an invented figure, a made-up source or an action item given to the wrong person. It happens because a language model produces likely text rather than checking facts, and because training and testing reward guessing over saying “I do not know”. Retrieval and citations reduce it but do not remove it: a 2024 study found commercial legal research tools built on retrieval hallucinated between 17% and 33% of the time. For offices the answer is design, not trust: ground answers in sources, show the source, and have a person confirm anything that matters.
01
Why it happens
A language model is trained to predict the next token. Nothing in that goal checks whether a sentence is true. A 2025 paper argues that hallucinations persist because training and evaluation procedures reward guessing over acknowledging uncertainty: on tests that mark answers only right or wrong, a model that guesses can outscore one that admits doubt. The same paper explains that when false statements cannot be told apart from true ones in the training data, errors arise from ordinary statistical effects.
Missing context is the other cause. If the answer is not in the model’s training and not in the documents it is given, it can still produce fluent text. This is why a scope instruction such as “answer only from the documents and say so when the answer is absent” matters.
02
Kinds seen in office work
Kind
Example
How to catch it
Invented figure or date
A fee, rate or deadline that is not in the source
Require a quoted passage and check the quote
Invented source
A case citation, circular or standard that does not exist
Verify it in the original database before use
Wrong attribution
An action item given to the person who asked, not the person who will do it
Require the transcript line, and have a person confirm the owner
Unfaithful summary
A summary that drops a condition or adds one
Compare against the source passage
Answering when the source is silent
A policy answer for a case the policy does not cover
Test with questions the documents cannot answer
03
What the evidence says
Source
Finding
Study of retrieval-based legal research tools (Magesh et al., 2024)
Each tool hallucinated between 17% and 33% of the time. That is fewer than a general-purpose chatbot, but short of the elimination some providers had claimed
ALCE citation benchmark, 2023
On the ELI5 dataset, even the best models lacked complete citation support 50% of the time
ANVE meeting-minutes harness, gemma3:4b
34 of 35 facts recalled, 19 of 20 action-item owners correct with none misassigned, 0 invented numbers, 5 of 5 questions that the document could not answer declined
Same harness, a 3B model
32 of 35 facts and one misassigned owner, which is why ANVE uses nothing smaller than 4B-class for minutes
04
How to reduce it
Ground answers in the office’s own documents and show the passage used.
Make “not in the documents” an acceptable answer, and test that the system gives it.
Use a low temperature for records such as minutes and extracted amounts.
Use a model large enough for the task, and test it on your own material.
Have a person confirm anything that becomes a record or leaves the building.
Keep a test set of real questions, including ones with no answer, and re-run it after every model change.
05
Why it matters for an office
Responsibility for work product stays with the professional who signs it, whatever tool helped draft it. In ANVE’s meeting minutes, every action item must quote the transcript line it came from and a person confirms the owner; in document question answering, the source passage is shown beside the answer. These make errors visible quickly, but they do not make a model correct. The ANVE harness is five synthetic meetings with one run each, and its note says so. This is general information, not legal advice.
06
Questions people ask
Can hallucination be eliminated?
Not at present. The 2024 legal-tools study found rates of 17% to 33% even with retrieval, and a 2025 analysis argues that guessing is rewarded by how models are trained and scored. Plan for it with sources and review.
Does running a model locally change how often it hallucinates?
Where a model runs does not change how it hallucinates. The model, its settings and whether it is given sources do. Test the exact setup you will use.
Is a hallucination a lie?
No. The model has no intention to deceive. It generates plausible text, and sometimes that text is false.
How do I test a system for hallucination?
Build a small test set from your own documents with known answers, add questions the documents cannot answer, and count invented details and wrong answers. Repeat the test after any model or setting change.
07
What is not real yet
No sealed ANVE unit exists; every image of the appliance on this site is a labelled render. The software runs today on a development machine. No customers yet, no certifications, no delivery date, and the final price is not announced.
Reservations are ₹999 in India or US$12 elsewhere, refundable in full until you confirm an order in writing (terms). Offices that cannot use cloud AI can apply to be one of ten design partners.
