Short answer. A dense model is a language model in which every parameter takes part in producing every token, as opposed to a mixture-of-experts model that uses only a selected few. Llama 3 and Qwen3-32B are dense. The effect is simple: the whole file is read from memory for each token, so a 32.8-billion-parameter dense model at 4-bit moves about 20 GB per token, roughly ten times what Qwen3-30B-A3B moves. Dense models are heavier to run than sparse ones of similar size, but simpler to train and tune. Meta chose a dense design for Llama 3 to keep its training stable.
01
How it works
In a dense Transformer, every layer applies the same large feed-forward block to every token, and every attention head is used. Nothing is skipped. If a model has 70 billion parameters, all 70 billion take part in each step. Meta’s Llama 3 paper says the team chose a standard dense Transformer rather than a mixture-of-experts model to maximise training stability.
Most families offer several dense sizes. Qwen3 has six dense models, from 0.6B to 32B, alongside two mixture-of-experts models. Small sizes such as 4B and 8B are what laptops and small appliances run.
02
Dense and mixture of experts side by side
Both models below are from the Qwen3 family and share a licence, Apache 2.0, so the comparison isolates the design. Ceilings divide memory bandwidth by the bytes read per token and are upper bounds for one user.
Measure
Qwen3-32B (dense)
Qwen3-30B-A3B (mixture of experts)
Total parameters
32.8 billion
30.5 billion
Parameters used per token
32.8 billion
3.3 billion
Weights at 4-bit
about 20 GB
about 19 GB
Bytes read per token at 4-bit
about 20 GB
about 2 GB
Speed ceiling at 200 GB/s
about 10 tokens per second
about 100 tokens per second
KV cache per token, 16-bit
262 KB
98 KB
03
When a dense model is the better fit
Predictable speed: the bytes read per token are the same on every request, so the ceiling is easy to estimate from memory bandwidth and file size.
Simpler tuning: Hugging Face notes that mixture-of-experts models face challenges in fine-tuning, while dense models follow the standard recipes, including LoRA.
Small sizes: a dense 4B or 8B model is the usual choice when memory is tight, since a mixture-of-experts design does not reduce the memory needed.
Consider a mixture-of-experts model instead when answer speed for one or a few users matters more than the memory it needs.
04
Why it matters for an office
A dense 70B model is the heavy end of what an office machine can serve: about 43 GB for the weights at 4-bit, and every token reads all of it, so interactive use needs memory bandwidth in the hundreds of gigabytes per second. A dense 8B model is cheap to run but less capable. The 27B to 35B class, where ANVE’s default model sits, is the middle ground. When a vendor quotes a speed, ask whether the model is dense or sparse, because the same file size gives very different speeds.
05
Questions people ask
Is a dense model better than a mixture-of-experts model?
Neither wins everywhere. Dense models have steady, predictable speed and simpler tuning. Mixture-of-experts models give faster answers to one user for the same memory class. Compare them on your own tasks.
Are most open models dense?
Many are. Llama 3 and six of the eight models in the original Qwen3 release are dense. Mixture-of-experts designs are common at larger sizes, such as gpt-oss, Mixtral and Qwen3-30B-A3B.
What is the opposite of a dense model?
A sparse model. A mixture-of-experts model is sparse because only some of its parameters are used for each token.
Does dense here mean the same as a dense vector in search?
No. A dense model uses all its parameters on every token. A dense vector, in search, is a list of numbers that represents meaning; see embeddings.
06
What is not real yet
No sealed ANVE unit exists; every image of the appliance on this site is a labelled render. The software runs today on a development machine. No customers yet, no certifications, no delivery date, and the final price is not announced.
Reservations are ₹999 in India or US$12 elsewhere, refundable in full until you confirm an order in writing (terms). Offices that cannot use cloud AI can apply to be one of ten design partners.
