Does quantization hurt answer quality? What was measured at 8, 6, 4 and 2 bits, and what it means for office work.
Short answer. Slightly at 8-bit and 4-bit, clearly at 2-bit. In llama.cpp’s own test on Llama 3 8B, 8-bit (Q8_0) changed perplexity by under 0.1 per cent and chose the same next word as the original model 97.7 per cent of the time. 4-bit (Q4_K_M) raised perplexity by 2.8 per cent and matched the original 91.9 per cent of the time; 2-bit (Q2_K) raised it by 56 per cent. A larger study of Llama 3.1 found 8-bit floating-point formats effectively lossless and 4-bit weight-only formats competitive. The effect varies by model and task, so test on your own documents.
01
A measured example: Llama 3 8B in llama.cpp
These are the project’s own figures for the Llama 3 8B base model, measured on the Wikitext-2 test set with the CUDA backend on an RTX 4090. Perplexity measures how surprised the model is by real text, so lower is better. The last column is the share of positions where the quantized and the original model give their highest probability to the same token. The test is English text, not your documents.
Format
Perplexity compared with 16-bit
Chose the same next word as 16-bit
Q8_0 (8-bit)
+0.04 per cent (ratio 1.0004)
97.7 per cent
Q6_K (6-bit)
+0.35 per cent (ratio 1.0035)
96.0 per cent
Q4_K_M (4-bit)
+2.8 per cent (ratio 1.0282)
91.9 per cent
Q2_K (2-bit)
+56 per cent (ratio 1.5648)
71.1 per cent
02
What larger studies found
Kurtic and colleagues ran more than 500,000 evaluations across the whole Llama 3.1 family. They report that 8-bit floating point (FP8) is effectively lossless, that well-tuned 8-bit integer formats lose 1 to 3 per cent, and that 4-bit weight-only formats are more competitive than expected, rivaling 8-bit. These are the formats used by serving engines such as vLLM, not llama.cpp’s GGUF files, so treat them as the same direction of result rather than the same numbers.
Google trains some Gemma 3 models with quantization-aware training: a short fine-tune that adapts the model to its 4-bit form. Its report lists the 27B at 54.0 GB in BF16 and 14.1 GB in 4-bit, a quarter of the memory.
03
What these numbers do not tell you
Perplexity averages over ordinary text. Office work leans on exact names, amounts, dates, quoted clauses and, for many readers, Hindi or Hinglish. A model that gets the gist right can still swap a digit. The studies above cannot tell you how your documents behave.
04
A test you can run in an afternoon
Take 20 to 50 real documents or transcripts. Ask the same questions of the 4-bit and the 8-bit file.
Check every number, name and date against the source, not just the tone of the answer.
Include a long document, a Hindi or Hinglish sample and a question whose answer is not in the text, to see whether the model says so.
If 4-bit passes, keep it: it leaves memory for more conversations. If it fails on a task, try 6-bit or 8-bit for that task only.
05
Where ANVE fits
ANVE’s meeting minutes are built so that a slip is caught by a person: every action item must quote the transcript line it came from, and its owner must be confirmed by a person before it is approved, with each confirmation logged. That does not remove the need to test a model, but it puts a check where errors would otherwise hide. ANVE is pre-release; no sealed unit exists yet.
06
Questions people ask
Is Q8_0 lossless?
Not strictly, but in llama.cpp’s Llama 3 8B test it was very close: perplexity within 0.04 per cent and the same next word 97.7 per cent of the time. For most office work you would not see the difference.
Is Q4_K_M the same as 4-bit GPTQ or AWQ?
No. They are different schemes. Q4_K_M is a llama.cpp format that mixes precisions across tensors; GPTQ and AWQ are used by other engines. Results for one do not carry over exactly to the other.
Does a bigger model lose less from quantization?
This page cannot say for every model. The vLLM-format study reports FP8 as effectively lossless across all sizes of the Llama 3.1 family, but llama.cpp’s tables here cover only 7B and 8B models, so test the size you intend to run.
07
What is not real yet
No sealed ANVE unit exists; every image of the appliance on this site is a labelled render. The software runs today on a development machine. No customers yet, no certifications, no delivery date, and the final price is not announced.
Reservations are ₹999 in India or US$12 elsewhere, refundable in full until you confirm an order in writing (terms). Offices that cannot use cloud AI can apply to be one of ten design partners.
