Tokens per second The speed number for AI answers, and what it hides.
Short answer. Tokens per second (tok/s) is how many tokens a model generates each second once it starts answering, the usual measure of response speed. An adult reads about 238 English words a minute, around 5 tokens a second, so 10 to 15 tokens a second already outpaces most readers. Speed is limited mainly by memory bandwidth, because every new token needs the model’s weights read from memory: a 3.3 GB model on a 150 GB/s machine cannot exceed about 45 tokens per second, and ANVE’s own test on an Apple M3 Pro laptop measured 40.9 for a 4B model. Quote any speed together with the model, precision, context length and number of users.
01
Three different speeds
Generation speed (decode): tokens written per second after the answer starts. This is what people usually mean by tokens per second. Databricks calls the time for each output token TPOT.
Prompt-processing speed (prefill): tokens read per second before the answer starts. It decides time to first token. In llama.cpp’s own table for Llama 3.1 8B at 4-bit, it was about 820 tokens per second against about 72 for generation.
Throughput: tokens per second across all users together. A server can raise throughput by serving many people at once, while each person’s own speed falls.
02
A ceiling from memory bandwidth
For one user, each new token needs the whole model read from memory once, so speed cannot exceed memory bandwidth divided by the bytes read per token. For a dense model, the bytes read are about the file size. Example: Ollama lists Gemma 3 4B at 3.3 GB, and Apple specifies 150 GB/s for the M3 Pro. 150 divided by 3.3 gives a ceiling of about 45 tokens per second. ANVE’s engineering note measured 40.9 for gemma3:4b on that laptop through Ollama, about 90% of the ceiling.
The table shows the same ceiling for other cases, in tokens per second. It is an upper bound for one user and ignores the cache and software overhead, so real speeds are lower. A mixture-of-experts model reads only its active experts: Qwen3-30B-A3B activates 3.3 billion of its 30.5 billion parameters, about 2 GB at 4-bit.
Bytes read per token
100 GB/s
200 GB/s
400 GB/s
800 GB/s
2.5 GB (4B dense, 4-bit)
40
80
160
320
5 GB (8B dense, 4-bit)
20
40
80
160
about 2 GB (MoE, 3.3B active, 4-bit)
50
100
200
400
17 GB (27B dense, 4-bit)
5.9
11.8
23.5
47
43 GB (70B dense, 4-bit)
2.3
4.7
9.3
19
03
One model at four precisions
llama.cpp’s documentation lists measured speeds for Llama 3.1 8B in several file formats. It does not name the test machine, so read the pattern, not the values. Shrinking the file from 14.96 GiB to 4.58 GiB raised generation from 29.2 to 71.9 tokens per second, while prompt processing barely moved. Writing is limited by reading the weights; reading the prompt is limited by arithmetic.
Format
File size
Generation (tokens/s)
Prompt processing (tokens/s)
16-bit (F16)
14.96 GiB
29.2
923
8-bit (Q8_0)
7.95 GiB
50.9
865
4-bit (Q4_K_M)
4.58 GiB
71.9
822
2-bit class (Q2_K)
2.95 GiB
79.9
784
04
How to read a speed claim
Which model and which precision? An 8-bit file reads twice the bytes of a 4-bit one.
What context length? Speed drops as the cache grows.
One user or many? Aggregate throughput and per-user speed are different numbers.
Does it include prompt processing? A figure for generation alone hides the wait before the first word.
Measured how? Ollama’s API returns eval_count and eval_duration in nanoseconds, and tokens per second is their ratio times a billion. llama.cpp ships a benchmark tool, llama-bench.
05
Why it matters for an office
Time is simple arithmetic. A 2,000-token summary takes 100 seconds at 20 tokens per second and 400 seconds at 5. Speed matters more when several people share one machine, or when an agent makes several model calls to finish one task, because the waits add up. For a single reader, a very high number is wasted: the answer would scroll faster than anyone reads.
ANVE has published no speed figures for its own hardware, so be wary of anyone quoting one before units exist. The speeds it has published, such as 40.9 tokens per second for a 4B model, come from an Apple laptop in an engineering note and say nothing about the launch unit.
06
Questions people ask
What is a good tokens-per-second for a local LLM?
For one reader, about 10 to 15 tokens per second is faster than reading speed: Databricks puts 10 tokens per second at roughly 450 words a minute, against about 238 for silent reading. For long outputs, agents and several users, more is better. Judge the number together with time to first token.
Why is the first token slow and the rest fast?
The model must read the whole prompt first, which is time to first token. After that, it writes at its steady generation speed.
Does a bigger model run slower?
Yes, for dense models. Each token reads the whole file, so doubling the file size roughly halves the ceiling. Mixture-of-experts models read only their active experts.
How do I measure tokens per second on my machine?
With Ollama, divide eval_count by eval_duration and multiply by a billion, using the fields in the API response. With llama.cpp, use llama-bench.
07
What is not real yet
No sealed ANVE unit exists; every image of the appliance on this site is a labelled render. The software runs today on a development machine. No customers yet, no certifications, no delivery date, and the final price is not announced.
Reservations are ₹999 in India or US$12 elsewhere, refundable in full until you confirm an order in writing (terms). Offices that cannot use cloud AI can apply to be one of ten design partners.
