INT8 and INT4 Whole-number formats that shrink a model to a half or a quarter of its 16-bit size.
Short answer. INT8 and INT4 are whole-number formats that store each value of an AI model in 8 or 4 bits. A model held in 16-bit form is converted by mapping its values onto a small set of integers, 256 levels for INT8 and 16 for INT4, with a scale factor to convert back. That halves or quarters the memory: a 27-billion-parameter model needing about 54 GB at 16 bits needs about 27 GB at 8 bits and about 13.5 GB at 4 bits, before overhead. Published methods report little loss at 8 bits and small loss at 4 bits, but results vary by model and task.
01
How integers stand in for decimals
Quantisation maps a range of decimal values onto whole numbers. The Google researchers who set out the common scheme write it as r = S(q − Z): the real value r is a scale S times the stored integer q, shifted by a zero point Z. The coarser the grid, the larger the rounding error. A worked example for a weight of 0.5 in a range of −1 to +1:
Format
Levels available
Step size
0.5 is stored as
Read back as
Error
INT8 (−127 to 127)
256
0.0079
64
0.504
0.004
INT4 (−7 to 7)
16
0.143
4
0.571
0.071
02
Memory saved
The arithmetic below counts weights only. Real files are larger. AMD lists Qwen3.5 9B at the 4-bit Q4_K_M setting as a 6.2 GB file against 4.5 GB from the bare arithmetic, the 35-billion-parameter model as 20.5 GB against 17.5 GB, and the 122-billion-parameter model as 76 GB against 61 GB.
Model size
16-bit
8-bit (INT8)
4-bit (INT4)
7 billion parameters
14 GB
7 GB
3.5 GB
27 billion parameters
54 GB
27 GB
13.5 GB
70 billion parameters
140 GB
70 GB
35 GB
03
Weights only, or weights and activations
NVIDIA explains that weights are easy to quantise because they are fixed after training, while activations contain outliers that make low precision harder. The LLM.int8() method handles this by multiplying more than 99.9% of values in 8-bit and isolating the outlier dimensions in 16-bit.
NVIDIA also notes that GPUs have no dedicated hardware for multiplying INT8 with FP16 numbers, so weight-only quantisation converts the weights back to higher precision for the arithmetic. It saves memory and bandwidth rather than calculation. NPUs are different: Microsoft says many support only low-bit integer maths such as INT8, so models must be quantised to run on them.
04
How much quality is lost
The LLM.int8() paper reports halving inference memory while keeping full-precision performance on models up to 175 billion parameters. The GPTQ paper reports reaching 3 or 4 bits per weight with negligible accuracy loss on the models it tested, and reasonable accuracy even at 2 bits. Both used benchmark tasks and 2022-era models.
Hindi minutes, legal PDFs and spreadsheets are not those benchmarks. Treat the published results as a reason to try 8-bit and 4-bit, not as proof they suit your work.
05
What to check
Test the quantised model on a sample of your own documents and recordings before you commit.
Ask which precision a vendor’s speed and quality figures used, and compare like with like.
Remember that 4-bit files carry scale factors, so they are larger than the bit count suggests.
Check that your runtime and chip support the format: AMD’s ROCm guide for Ryzen said only FP16 had been officially validated, though other types may work.
Newer GPUs add native 4-bit floating point (FP4), which is a different format from INT4.
06
Where ANVE stands
ANVE One is specified to pair a 27B-class model with a fast 4B-class model in 64 GB of unified memory. Even at 8 bits the arithmetic above puts those at roughly 27 GB and 4 GB of weights, well inside 64 GB. ANVE has published no speed figures, and no sealed unit exists yet.
07
Questions people ask
Is INT4 much worse than INT8?
It can be, because the grid is 16 levels instead of 256. Methods such as GPTQ report small losses at 4 bits on the models they tested, but test on your own tasks.
Do I need special hardware to run INT8 or INT4 models?
Not usually. Runtimes convert the stored integers back as they compute. NVIDIA notes that GPUs lack dedicated INT8 by FP16 multiplication, so the saving is mainly memory and bandwidth.
Why do NPU specifications mention INT8?
Microsoft says many NPUs support only low-bit integer maths such as INT8, so AI models must be quantised to run on them. When you see an NPU’s TOPS rating, check which precision it refers to.
Is quantisation the same as zipping a file?
No. Zipping is lossless. Quantisation is lossy: it changes the numbers the model computes with, which is why quality must be tested.
08
What is not real yet
No sealed ANVE unit exists; every image of the appliance on this site is a labelled render. The software runs today on a development machine. No customers yet, no certifications, no delivery date, and the final price is not announced.
Reservations are ₹999 in India or US$12 elsewhere, refundable in full until you confirm an order in writing (terms). Offices that cannot use cloud AI can apply to be one of ten design partners.
