Does an MoE model need less memory?

No. Every expert must be stored, so the weights file is about the size of a dense model with the same total parameters: 61.1 GB against 65.5 GB at BF16 for Qwen3-30B-A3B and Qwen3-32B.

Why was the 4-bit MoE slower in Qwen’s table?

Qwen’s benchmark shows 31.29 tokens per second for the 4-bit MoE against 155.55 at FP8 and notes that the 4-bit path in SGLang needed improvement. It shows that quantization format and kernel support can outweigh the expert design.

Are MoE models as accurate as dense ones?

Qwen reports that its MoE base models match older dense models while using about 10 per cent of the active parameters, and OpenAI reports strong results for gpt-oss. These are vendor claims, so check them on your own documents.

ANVE: the private AI office, on the record. See what that means

ANVE

Private AI, answered · updated 2026-10-16

Are mixture-of-experts models faster on local hardware?
What “3 billion active” means, what it saves and what it does not.

Short answer. Often yes for one user, but they are not smaller. A mixture-of-experts (MoE) model holds all its parameters in memory yet reads only a few billion of them for each word, so it can write faster than a dense model of similar total size. In Qwen’s own benchmark on one NVIDIA H20 96GB, Qwen3-30B-A3B generated 137 tokens per second at BF16 against 21 for the dense Qwen3-32B, with a similar weights file (61.1 GB against 65.5 GB). The gain depends on the runtime and format and is not guaranteed on every setup, so test on your own hardware.

Total and active parameters

A router in each layer picks a few experts for every word. Mixtral’s paper describes the idea: each word has access to all the parameters, but uses only the selected experts, and the selection can differ at each step. The result is that the whole model must be stored, while only a fraction is read to write one word.

Model	Design	Total parameters	Active for each word	Weights file
Qwen3-32B	Dense	32.8 billion	32.8 billion	19.8 GB at Q4_K_M; 65.5 GB at BF16
Qwen3-30B-A3B	MoE: 128 experts, 8 used	30.5 billion	3.3 billion	18.6 GB at Q4_K_M; 61.1 GB at BF16
gpt-oss-20b	MoE	21 billion	3.6 billion	12.1 GB (MXFP4)
gpt-oss-120b	MoE	117 billion	5.1 billion	63.4 GB (MXFP4)

A measured comparison

These are Qwen’s own figures from its speed benchmark: one NVIDIA H20 with 96 GB, SGLang 0.4.6, batch size 1. Qwen counts prompt and generated tokens together, but with an input of one token the figure is essentially generation speed. At 16-bit and 8-bit the MoE model is 3 to 7 times quicker. The 4-bit row goes the other way, and Qwen’s notes say the 4-bit path in SGLang needed improvement, so the format and the runtime matter. This is a datacentre GPU, not office hardware, so read the ratios, not the absolute figures.

Setting (input length 1, 2,048 tokens generated)	Qwen3-32B, dense	Qwen3-30B-A3B, MoE
BF16	20.72 tokens/s	137.18 tokens/s
FP8	46.17 tokens/s	155.55 tokens/s
4-bit (AWQ for the dense model, GPTQ for the MoE)	47.67 tokens/s	31.29 tokens/s

What it saves and what it does not

Writing each word is memory-bound, as NVIDIA explains, so reading about a tenth of the weights is much of the gain. Storage is not saved: AMD notes that Llama 4 Scout, a mixture-of-experts model, activates 17 billion of its 109 billion parameters, yet all 109 billion must be held in memory, so its footprint equals a dense model of that size. The cache for long conversations is also not reduced by the expert design.

With many people using the server at once, different words are routed to different experts, so more of the weights are read on each step and the advantage should shrink. That is my reasoning from how routing works; the sources checked here did not benchmark it.

Choosing for an office

For one person or a few, a small-active MoE such as Qwen3-30B-A3B or gpt-oss-20b gives quick answers from modest memory (12 to 19 GB at 4-bit).
For heavy shared use, test a dense model against an MoE under load. The winner depends on the number of simultaneous users and the runtime.
Compare quality too. Qwen reports that Qwen3-30B-A3B outcompetes its earlier 32B reasoning model with a tenth of the active parameters, a vendor claim to verify on your own tasks.
ANVE’s launch default, Qwen 3.8 27B, is described on its model card as a dense model. ANVE has not published speed figures for its launch hardware.

Questions people ask

Does an MoE model need less memory?: No. Every expert must be stored, so the weights file is about the size of a dense model with the same total parameters: 61.1 GB against 65.5 GB at BF16 for Qwen3-30B-A3B and Qwen3-32B.
Why was the 4-bit MoE slower in Qwen’s table?: Qwen’s benchmark shows 31.29 tokens per second for the 4-bit MoE against 155.55 at FP8 and notes that the 4-bit path in SGLang needed improvement. It shows that quantization format and kernel support can outweigh the expert design.
Are MoE models as accurate as dense ones?: Qwen reports that its MoE base models match older dense models while using about 10 per cent of the active parameters, and OpenAI reports strong results for gpt-oss. These are vendor claims, so check them on your own documents.

What is not real yet

No sealed ANVE unit exists; every image of the appliance on this site is a labelled render. The software runs today on a development machine. No customers yet, no certifications, no delivery date, and the final price is not announced.

Reservations are ₹999 in India or US$12 elsewhere, refundable in full until you confirm an order in writing (terms). Offices that cannot use cloud AI can apply to be one of ten design partners.

Are mixture-of-experts models faster on local hardware?What “3 billion active” means, what it saves and what it does not.