KV cache Why a model needs more memory as the conversation gets longer.
Short answer. The KV cache is the working memory a language model builds up as a conversation grows: for every token so far it stores two sets of numbers, called keys and values, so that it does not recompute them for each new word. It grows in direct proportion to the context length and to the number of people using the model at once. For Llama 3.1 8B in 16-bit precision it costs about 131 KB per token, which is about 1.1 GB for 8,192 tokens and about 17 GB at the full 128K window. For Qwen3-32B it is about 262 KB per token, so one 32,768-token chat needs about 8.6 GB on top of the model’s own weights.
01
How it works
To write each new token, a model compares it with every earlier token. That comparison uses two vectors of numbers for each earlier token in each layer: a key and a value. Without a cache the model would recompute them for the whole conversation at every step. With a cache it computes them once, stores them, and adds only the new token’s pair. The price is memory, which grows with each token.
NVIDIA gives the formula 2 × layers × (heads × head size) × bytes per number; with grouped-query attention, the head count to use is the number of KV heads. Grouped-query attention shares each key and value pair across several query heads, and most modern models use it. Llama 3 has 8 KV heads against 32 to 128 query heads. An older design with a KV head for every query head, such as Llama 2 7B, costs about 524 KB per token, four times Llama 3.1 8B.
02
How much memory it takes
Sizes below are for a 16-bit cache, computed from the layer and head counts published for each model. They are per conversation, and they come on top of the weights.
Model
Per token
8,192 tokens
32,768 tokens
131,072 tokens
Qwen3-30B-A3B (48 layers, 4 KV heads)
98 KB
0.8 GB
3.2 GB
12.9 GB
Llama 3.1 8B (32 layers, 8 KV heads)
131 KB
1.1 GB
4.3 GB
17.2 GB
Qwen3-8B (36 layers, 8 KV heads)
147 KB
1.2 GB
4.8 GB
19.3 GB
Qwen3-32B (64 layers, 8 KV heads)
262 KB
2.1 GB
8.6 GB
34.4 GB
Llama 3 70B (80 layers, 8 KV heads)
328 KB
2.7 GB
10.7 GB
42.9 GB
03
Ways to shrink it
Choose a model with grouped-query attention. It is part of the model’s design, so you pick it when you pick the model.
Quantise the cache. Ollama can store it in 8-bit, which uses about half the memory of 16-bit, or 4-bit, about a quarter, at some cost in accuracy.
Shorten the context: retrieve the relevant passages instead of pasting whole files.
Share the start of prompts. Servers such as vLLM can reuse the cache of a shared prompt prefix across requests.
Prefer designs that limit long-range attention. gpt-oss alternates layers that see the whole context with layers that look back only 128 tokens, so half its layers need a cache that does not grow with context, provided the runtime makes use of it.
Avoid waste. In earlier serving systems only 20.4% to 38.2% of cache memory held actual token states, according to the vLLM paper. Its PagedAttention manages cache memory in pages and reports near-zero waste, with 2 to 4 times the throughput of earlier systems.
04
Why it matters for an office
Each active conversation holds its own cache. Ten people chatting at once, each with 16,384 tokens of context on a Qwen3-32B-class model, need about 43 GB of cache alone, more than the 4-bit weights of the model itself. A machine can fit the model and still run out of room for the users. When comparing products, ask how many users at what context length a memory figure assumes.
05
Questions people ask
Is the KV cache stored on the GPU or in system memory?
It sits next to the weights, in GPU memory on a machine with a discrete GPU or in the shared pool on a unified-memory machine. If it does not fit, the runtime has to use slower memory or refuse the request.
Why does the model slow down in a long conversation?
Each new token must read the whole cache as well as the weights, and a long prompt takes longer to process before the first word appears. Both effects grow with context length.
Is the KV cache the same as the context window?
They are related but different. The context window is the limit on how many tokens can be in view. The KV cache is the memory those tokens occupy while the model is working.
Can the cache be reused between questions?
Yes, when the start of the prompt is identical. Prefix caching keeps the cache for a shared document or system prompt so only the new question is processed.
06
What is not real yet
No sealed ANVE unit exists; every image of the appliance on this site is a labelled render. The software runs today on a development machine. No customers yet, no certifications, no delivery date, and the final price is not announced.
Reservations are ₹999 in India or US$12 elsewhere, refundable in full until you confirm an order in writing (terms). Offices that cannot use cloud AI can apply to be one of ten design partners.
