Which open models summarise documents well? What one public benchmark shows for open models, and how to check your own.
Short answer. No open model can be trusted without checking, but some add fewer unsupported facts than others. In Vectara’s summarisation leaderboard, updated 22 September 2026, Llama 3.3 70B Instruct had a hallucination rate of 4.1 per cent, Qwen3-8B 4.8, Qwen3-32B 5.9, Gemma 3 27B 7.4 and gpt-oss-120b 14.2. Lower is better, and bigger or newer is not always lower. The benchmark measures whether a summary sticks to its source, not usefulness, language skill or tone, so test candidates on your own documents.
01
What the benchmark measures
Vectara’s Hallucination Leaderboard asks each model to summarise a document using only the facts in it, then uses Vectara’s HHEM-2.3 model to judge whether the summary stays consistent with the source. The private dataset has more than 7,700 articles, from 50 words to 24,000 words, drawn from news, technology, science, medicine, law, sports, business and education. The hallucination rate is the share of summaries judged to contain unsupported content. The answer rate is the share of documents where the model produced a summary at all.
02
Open-model results, 22 September 2026
The leaderboard runs hosted copies through providers’ APIs; its notes say gpt-oss-120b was accessed through Together AI and the Qwen3 models through Alibaba’s DashScope. A quantized copy on your own machine can score differently. Qwen3.8-27B and gpt-oss-20b did not appear in the table on that date.
Model (as listed)
Hallucination rate (lower is better)
Answer rate
Average summary length
Llama-3.3-70B-Instruct-Turbo
4.1 per cent
99.5 per cent
64.6 words
Qwen3-8B
4.8 per cent
99.9 per cent
83.6 words
Qwen3-14B
5.4 per cent
99.9 per cent
111.1 words
Qwen3-32B
5.9 per cent
99.9 per cent
115.8 words
Gemma-3-27B-it
7.4 per cent
98.8 per cent
96.4 words
Qwen3.5-27B
12.1 per cent
99.8 per cent
94.4 words
gpt-oss-120b
14.2 per cent
99.9 per cent
135.2 words
03
What the table does and does not say
Size does not decide it. Qwen3-8B (4.8 per cent) sits ahead of Qwen3-32B (5.9) and far ahead of the newer Qwen3.5-27B (12.1).
Style probably matters. Longer or more interpretive summaries give a model more chances to add something the source does not say, even though Vectara’s prompt told every model not to infer.
It measures faithfulness, not usefulness. A short, safe summary can score well and still miss what you needed.
It does not test your languages. Llama 3.3’s card lists Hindi among its supported languages and Qwen3 claims support for 100 or more languages, but neither claim is tested here.
04
A test on your own documents
Pick 20 real documents of the kinds you handle, including one long one and one in Hindi or Hinglish if you use them.
Ask each candidate model the same instruction, such as “summarise using only this text and quote the sentence behind each fact”.
Have two people mark every statement the source does not support, then compare counts.
Keep the checking step after you choose. ANVE’s document answers show the source passage next to the answer, and its meeting minutes require each action item to quote the transcript line it came from, so a reader can verify a claim.
05
Questions people ask
Which open model summarises most faithfully?
No single model wins on every measure. On this benchmark Llama 3.3 70B Instruct had the lowest rate among the open models listed here, but the benchmark is narrow and these were hosted copies. Test two or three on your own documents.
Is a lower hallucination rate always better?
Not by itself. Check the answer rate, because a model that declines many documents looks good on the rate it does answer, and check summary length, because a very short summary says little.
Does a bigger model summarise more faithfully?
Not in this table. Qwen3-8B had a lower rate than Qwen3-32B, and a 70B model had the lowest rate among the open models listed while a 120B model sat near the other end. Faithfulness depends on training and style as well as size.
06
What is not real yet
No sealed ANVE unit exists; every image of the appliance on this site is a labelled render. The software runs today on a development machine. No customers yet, no certifications, no delivery date, and the final price is not announced.
Reservations are ₹999 in India or US$12 elsewhere, refundable in full until you confirm an order in writing (terms). Offices that cannot use cloud AI can apply to be one of ten design partners.
