GGUF The one-file format most local AI tools use to load a model.
Short answer. GGUF is a single-file binary format that stores a model’s weights together with its metadata, including the tokenizer and often the chat template, so a tool can load it with nothing else. It was created for llama.cpp, and tools such as LM Studio and Ollama can load it. GGUF files are usually quantised, and the name says how: Q4_K_M is a 4-bit mix that averages 4.89 bits per weight, so Llama 3.1 8B shrinks from 14.96 GiB at 16-bit to 4.58 GiB. Choose a larger code, such as Q6_K or Q8_0, when accuracy matters more than memory.
01
What is inside a GGUF file
The GGUF specification lists its aims: single-file deployment, extensibility, compatibility with memory mapping for fast loading, and carrying all the information needed to load the model. It succeeds the older GGML, GGMF and GGJT formats. A file holds key-value metadata, such as the architecture, the list of tokens and the chat template, followed by the tensors, each stored in a declared type such as 16-bit floats or a quantised format. Models are little-endian by default.
Because the tokenizer and chat template travel with the weights, llama.cpp’s server reads the template from the model file by default. Large models are sometimes split into shards named like 00003-of-00009, and a vision model can come with a separate projector file whose name starts with mmproj. Hugging Face’s Hub has a viewer that shows a GGUF file’s metadata and tensor details.
02
Reading the name
In a name such as Q4_K_M, Q means quantised and the number is the nominal bits per weight. K marks the k-quant method, which stores weights in blocks, each with its own scale. A trailing S, M or L marks a mix of tensor types; the M mix keeps more precision than S, since llama.cpp lists Q4_K_S at 4.67 bits per weight and Q4_K_M at 4.89. IQ names are quantisations that use an importance matrix. The figures below are llama.cpp’s measured sizes for Llama 3.1 8B.
Code
Average bits per weight
Llama 3.1 8B size
A 27B model, approx.
F16
16.0
14.96 GiB
54 GB
Q8_0
8.50
7.95 GiB
28.7 GB
Q6_K
6.56
6.14 GiB
22.2 GB
Q5_K_M
5.70
5.33 GiB
19.3 GB
Q4_K_M
4.89
4.58 GiB
16.5 GB
Q3_K_M
4.00
3.74 GiB
13.5 GB
Q2_K
3.16
2.95 GiB
10.7 GB
03
Which one to pick
Q4_K_M is the usual starting point: llama.cpp makes it the default when you fetch a model by repository name.
Move to Q6_K or Q8_0 when the task is sensitive to small errors, such as reading amounts from bills, and memory allows.
Treat 3-bit and 2-bit files as experiments. llama.cpp warns that quantisation may introduce accuracy loss, usually measured as perplexity or KL divergence, so test on your own work.
Smaller files are also faster for one user, since less is read per token: in llama.cpp’s table, generation rose from 29.2 tokens per second at 16-bit to 71.9 at Q4_K_M on its test machine, which it does not name.
Check the file against the checksum the publisher gives, and prefer files from the model’s maker or an established converter.
04
GGUF compared with other formats
Format
Holds
Loaded by
Note
GGUF
Tensors, metadata, tokenizer, chat template
llama.cpp, LM Studio, Ollama
One file; vLLM support is described as highly experimental
Safetensors
Tensors only, with config and tokenizer files beside them
Transformers, vLLM and many others
Designed so a downloaded file cannot run code on load
PyTorch pickle (.bin, .pt)
Tensors, as Python objects
PyTorch
Can run code when loaded; avoid files from unknown sources
05
Questions people ask
Is GGUF a model?
No. It is a container format. The model inside is the same as the one distributed in other formats, usually with its numbers quantised to save memory.
What does Q4_K_M mean?
A 4-bit k-quant of medium mix. It averages 4.89 bits per weight in llama.cpp’s figures, which makes a Llama 3.1 8B file about 4.6 GiB instead of 15 GiB.
Can I convert a model to GGUF myself?
Yes. llama.cpp includes tools to convert a model to GGUF and to quantise it, and Hugging Face hosts a space called gguf-my-repo that does it online.
Does vLLM run GGUF files?
Its documentation says GGUF support is highly experimental and under-optimised, and now lives in a separate plugin. For vLLM, safetensors models are the normal route.
06
What is not real yet
No sealed ANVE unit exists; every image of the appliance on this site is a labelled render. The software runs today on a development machine. No customers yet, no certifications, no delivery date, and the final price is not announced.
Reservations are ₹999 in India or US$12 elsewhere, refundable in full until you confirm an order in writing (terms). Offices that cannot use cloud AI can apply to be one of ten design partners.
