Q4 or Q8: which should an office use? A decision in terms of memory, conversations and risk.
Short answer. Start with Q4_K_M and move a task to Q8_0 only if your own test shows a difference that matters. Q4_K_M stores weights in about 4.8 bits and Q8_0 in about 8.5, so a 27B model takes about 16 GB against 29 GB. In llama.cpp’s Llama 3 8B test, Q8_0 was almost identical to the 16-bit original (perplexity +0.04 per cent) and Q4_K_M was 2.8 per cent higher. The 12.5 GB that Q4_K_M saves on a 27B model holds about 5 more conversations of 32,768 tokens on Qwen3.8-27B, and it is the difference between fitting a 24 GB card (16 GB) and not (29 GB).
01
The trade in numbers
Weights (GB) ≈ parameters in billions × bits per weight ÷ 8, with Q4_K_M at about 4.8 bits and Q8_0 at about 8.5. The last column divides the saved memory by one conversation’s KV cache at 32,768 tokens (NVIDIA’s per-token formula; for Qwen3.8-27B it includes about 0.15 GB of fixed state, my estimate). All figures are estimates.
Model size
Q4_K_M weights
Q8_0 weights
Memory Q4_K_M saves
Extra 32,768-token conversations that memory holds
8B (Llama 3.1)
4.8 GB
8.5 GB
3.7 GB
Under one
27B (Qwen3.8-27B)
16.2 GB
28.7 GB
12.5 GB
About 5
32B (Qwen3-32B)
19.7 GB
34.8 GB
15.2 GB
About 1
70B (Llama 3.3)
42.0 GB
74.4 GB
32.4 GB
About 3
02
What the quality difference looks like
In llama.cpp’s own test on the Llama 3 8B base model, Q8_0 raised perplexity by 0.04 per cent over the 16-bit original and chose the same next word 97.7 per cent of the time. Q6_K raised it by 0.35 per cent (96.0 per cent same word). Q4_K_M raised it by 2.8 per cent (91.9 per cent same word). A study of Llama 3.1 in vLLM’s formats found 4-bit weight-only models rivaling 8-bit ones. The question on quantization quality has the details.
Speed also differs. Q8_0 moves about 1.8 times as many bytes for every word (8.5 ÷ 4.8), so it is usually slower on the same machine, by an amount that depends on the hardware and software.
03
A simple rule
Situation
Pick
Why
Summaries, drafting, questions over documents
Q4_K_M
The loss is small and the memory goes to more or longer conversations
Extraction of amounts, dates and clause wording
Test both; use Q8_0 if it is clearly better
Exact characters matter more than fluent prose
A 128 GB machine with one or two users
Q8_0
There is memory to spare, so spend it on precision
A 24 GB card or many simultaneous users
Q4_K_M
Every gigabyte saved is cache for another conversation
Scanned documents read by a vision model
Keep the vision part at 16-bit or 8-bit
llama.cpp advises keeping multimodal encoders at bf16 or q8 because their quality feeds the whole answer
Anything below 4-bit (Q3, Q2)
Avoid for work that must be right
Q2_K raised perplexity by 56 per cent in the Llama 3 8B test
04
What the saved memory buys
On a 27B model the 12.5 GB saved by Q4_K_M equals about 5 extra long conversations on Qwen3.8-27B, whose cache is small, but less than one on a design that caches every layer, such as Gemma 3 27B counted that way (16.6 GB each). The value of the saved memory depends on the model’s cache, so check that too.
ANVE One’s 64 GB of unified memory leaves room for either choice with a 27B-class model; on a 24 GB graphics card, as in ANVE Office, check the 8-bit size against the card first. The right setting for a task is found by testing it, not by habit. ANVE is pre-release: no sealed unit exists yet, and the specifications are for launch.
05
Questions people ask
Is a middle option such as Q5 or Q6 worth using?
Sometimes. In llama.cpp’s Llama 3 8B test Q6_K added 0.35 per cent perplexity and matched the original’s next word 96.0 per cent of the time, between Q8_0 and Q4_K_M. It costs about 6.6 bits per weight against 4.9 for Q4_K_M in the project’s Llama 3.1 8B table.
Does 4-bit hurt Hindi or Hinglish more than English?
No published figure was found for this. The llama.cpp numbers are English text. Test a Hindi and Hinglish sample of your own at both precisions before choosing.
Can I use Q8_0 for one task and Q4_K_M for another?
Yes, if both models fit in memory together. Ollama keeps up to three models loaded by default if they fit, and otherwise queues requests until one is unloaded.
06
What is not real yet
No sealed ANVE unit exists; every image of the appliance on this site is a labelled render. The software runs today on a development machine. No customers yet, no certifications, no delivery date, and the final price is not announced.
Reservations are ₹999 in India or US$12 elsewhere, refundable in full until you confirm an order in writing (terms). Offices that cannot use cloud AI can apply to be one of ten design partners.
