How much memory does a 70B model need?
Weights, KV cache and overhead, worked out for an office budget.

Three parts add up. Weights: parameters in billions × bits per weight ÷ 8, with BF16 at 16 bits, Q8_0 at about 8.5 bits and Q4_K_M at about 4.8 bits. KV cache: the memory a model keeps for every token of the conversation so far, which NVIDIA gives as 2 × layers × key/value heads × head size × bytes per value, per token. Overhead: 1 to 2 GB for the runtime’s working buffers, counted here as 2 GB. Everything on this page is an estimate.

For the Llama 3 70B family, Meta’s paper lists 80 layers and 8 key/value heads (grouped-query attention) in a model 8,192 values wide with 64 attention heads, so each head is 128 values wide. That gives 2 × 80 × 8 × 128 × 2 bytes = 327,680 bytes, or 320 KiB, for every token.

A check against a measurement: Qwen’s published speed benchmark on an NVIDIA H20 shows Qwen3-30B-A3B using about 2,880 MB more GPU memory at a 30,720-token input than at a 1-token input. That is 96 KiB per token, reading the table’s megabytes as the usual GPU-memory mebibytes, and it is what the formula gives from the model’s configuration (2 × 48 layers × 4 key/value heads × 128 values × 2 bytes). For the dense Qwen3-32B the same table grows within 4 per cent of the formula at 30,720 tokens.

The table sets each estimate beside a real file. Real files run 1 to 3 per cent above the estimate because a few tensors are kept at higher precision; llama.cpp’s own table lists 4.89 bits per weight for Q4_K_M on an 8B model, against the 4.8 used here.

Each extra conversation in progress needs its own cache. Ollama can store the cache at 8 bits instead of 16, which its documentation says uses about half the memory with a very small loss in precision.

Speed: at 4-bit a 70B model reads about 42 GB of weights for every word it writes. A 27B model at the same precision reads about 16 GB. On the same machine the 70B answers more slowly, because NVIDIA’s notes on inference explain that generating text is limited by memory bandwidth, not by raw compute.

Quality: a bigger model is not automatically better for office tasks. Qwen reports that its 32B base model performs as well as its previous 72B base model, so test a recent 27B to 35B-class model on your own documents before paying for 70B-class memory.

ANVE One and ANVE Office are specified around 27B-class models, not 70B-class ones. Those specifications are for launch and the prices indicative; no sealed unit exists yet.

No sealed ANVE unit exists; every image of the appliance on this site is a labelled render. The software runs today on a development machine. No customers yet, no certifications, no delivery date, and the final price is not announced.

Reservations are ₹999 in India or US$12 elsewhere, refundable in full until you confirm an order in writing (terms). Offices that cannot use cloud AI can apply to be one of ten design partners.