What makes a local AI model fast or slow? The levers that move speed, and how to read a vendor’s number.
Short answer. Memory bandwidth, model size, precision, context length and the software that serves the model decide most of the speed. Writing each word means reading the model’s active weights from memory, so a smaller, more heavily quantized or mixture-of-experts model reads less and answers sooner. Reading a long document before the first word (prefill) is limited by compute instead. Software matters as much as hardware: on one graphics card, one public benchmark transcribes 13 minutes of audio in 2 minutes 23 seconds with one program and 17 seconds with another.
01
Two phases with different limits
NVIDIA’s inference notes split a request in two. Prefill takes in your whole prompt in parallel, a matrix-by-matrix job that keeps the GPU’s arithmetic units busy, so it is limited by compute. Decode then writes one word at a time, and each word needs the weights and the cache to be read again from memory, so decode is memory-bound. The first phase decides how long you wait for the first word (time to first token); the second decides how fast the answer scrolls (tokens per second).
02
The levers
AMD’s own FAQ ties bandwidth to speed directly: it says that doubling bandwidth to 256 GB/s on Strix Halo roughly doubles tokens per second against its lower tier. NVIDIA adds that quantization lets more parameters cross the same bandwidth, which speeds up bandwidth-limited models.
Lever
What changes
What you can do
Memory bandwidth
Sets the pace of one conversation
Compare vendor figures: 256 GB/s for Ryzen AI Max+ 395, 273 GB/s for DGX Spark, 460 to 614 GB/s for Mac Studio M5 Max, 672 GB/s for RTX PRO 4000 Blackwell
Model size
Bytes read per word grow with the number of parameters
Use the smallest model that passes your own test
Precision
Q8_0 reads about 1.8 times the bytes of Q4_K_M (8.5 against 4.8 bits)
Start at 4-bit; move up only if a test shows a loss that matters
Active parameters
A mixture-of-experts model reads only some of its weights for each word
See the question on mixture-of-experts models
Context length
Longer prompts add prefill time, and the cache grows with every token
Retrieve the relevant passages instead of pasting whole documents
Simultaneous users
Batching raises total output but each person’s share falls
Size for the busiest hour, not the average
Software
The same card can differ several times over
Use a maintained runtime and check its settings
Heat
A sustained load can make a chip slow itself down (thermal throttling)
Check cooling and where the machine sits
03
Same card, different software
This is the faster-whisper project’s benchmark on an NVIDIA RTX 3070 Ti with 8 GB (CUDA 12.4). Same card, same model, same precision: 143 seconds against 17 seconds, about 8 times apart, from the software and batching alone. It is speech recognition, not a chat model, but the lesson carries over.
Program (large-v2 model, 16-bit, beam size 5)
Time for 13 minutes of audio
GPU memory
openai/whisper
2 min 23 s
4,708 MB
faster-whisper
1 min 03 s
4,525 MB
faster-whisper, batch size 8
17 s
6,090 MB
04
How to read a speed claim
Ask which model, which precision and which context length. A figure for a small 4-bit model says little about a large 8-bit one.
Ask what is counted. Qwen’s own speed benchmark counts prompt and generated tokens together in one tokens-per-second figure, so large numbers at long inputs mostly reflect prompt processing.
Ask how many conversations ran at once. One user alone and twenty users together give different numbers from the same machine.
ANVE has not published speed figures for its launch hardware. When it does, expect them to name the model, precision, context and concurrency.
05
Questions people ask
Does more TOPS mean a faster model?
Not by itself. TOPS is a peak figure for low-precision arithmetic. AMD lists the Ryzen AI Max+ 395 at up to 126 TOPS overall and 50 TOPS for the NPU, but a language model’s pace on one conversation is mostly set by memory bandwidth.
Why is the first word slow after I paste a long document?
The model has to read the whole prompt before it can write, and that prefill step grows with the length of the prompt. Shorter prompts or retrieving only the relevant passages reduce the wait.
Is a quantized model faster?
Usually, yes, because fewer bytes are read for each word. The cost is a small loss of accuracy that varies by model and task, covered in the question on quantization quality.
06
What is not real yet
No sealed ANVE unit exists; every image of the appliance on this site is a labelled render. The software runs today on a development machine. No customers yet, no certifications, no delivery date, and the final price is not announced.
Reservations are ₹999 in India or US$12 elsewhere, refundable in full until you confirm an order in writing (terms). Offices that cannot use cloud AI can apply to be one of ten design partners.
