Memory bandwidth The number that limits how fast a local model can write its answer.
Short answer. Memory bandwidth is the amount of data, in gigabytes per second, that a chip can read from its memory. It matters for AI because writing each word of an answer means reading the model’s weights from memory, so bandwidth sets a ceiling on speed: a 16 GB model on memory delivering 256 GB/s cannot exceed about 16 tokens (word pieces) a second, however fast the processor is. Bandwidth equals bus width times data rate. AMD’s Ryzen AI Max+ 395 pairs a 256-bit bus with LPDDR5x-8000, which works out at 256 GB/s, while NVIDIA lists 672 GB/s for the RTX PRO 4000 Blackwell.
01
How to work it out
Bandwidth in gigabytes per second equals the bus width in bits, divided by 8 to give bytes, multiplied by the data rate in billions of transfers per second. A 256-bit bus moves 32 bytes per transfer; at 8,000 million transfers a second that is 256 GB/s.
Standards bodies publish the ceilings. JEDEC’s LPDDR5X update raised the top speed to 8,533 Mbps from 6,400, and its GDDR7 standard reaches up to 192 GB/s per memory device.
Chip
Memory
Bandwidth
How we know
AMD Ryzen AI Max+ 395
256-bit LPDDR5x, up to LPDDR5x-8000
256 GB/s
Calculated from AMD’s published bus width and speed
NVIDIA RTX PRO 4000 Blackwell
24 GB GDDR7
672 GB/s
Listed by NVIDIA
02
Why it caps the speed of an answer
NVIDIA’s engineers describe the decode phase, where the model writes one token at a time, as memory-bound: the speed at which weights, keys and values move from memory to the processor dominates the delay, not how fast the arithmetic runs. Each new token needs the weights again, so a rough ceiling is the bandwidth divided by the bytes read per token.
Smaller weights read faster. NVIDIA notes that quantisation lets you move more parameters over the same bandwidth, which helps models that are bandwidth-limited.
03
Ceilings for a few model sizes
The table divides each bandwidth by the weights read per token, in tokens per second. These are upper bounds from arithmetic, not predictions.
Weights read per token
At 256 GB/s
At 672 GB/s
4 GB (about an 8-billion-parameter model at 4-bit)
64 tokens/s
168 tokens/s
8 GB
32 tokens/s
84 tokens/s
16 GB
16 tokens/s
42 tokens/s
32 GB
8 tokens/s
21 tokens/s
04
Ceiling against measurement
AMD measured a 6.2 GB model (Qwen3.5 9B at 4-bit) at 29.84 tokens a second on a Ryzen AI Max+ 395 system with 128 GB of unified memory. The same sum gives a ceiling of about 41, so the measurement reached roughly 70% of it. A mixture-of-experts model reads only its active experts for each token, so AMD’s 35-billion-parameter model with 3 billion active ran faster, at 42.04 tokens a second, despite a 20.5 GB file.
Bandwidth is not the only limit. NVIDIA says reading a prompt (prefill) keeps the GPU busy with arithmetic, and it advises doing as much as possible with the weights each time they are loaded, for example by batching requests, so several users can share one read of the weights.
05
What to check
Ask for bandwidth in GB/s, not only the memory type. Compute it from bus width and data rate if the vendor gives those.
Treat “up to” memory speeds as maximums: AMD’s page lists the 395’s maximum memory speed, not what every system ships with.
Compare chips at the same quantisation, since file size sets the bytes read per token.
Do not use TOPS or TFLOPS as a proxy: they say nothing about bandwidth.
Ask for measured tokens per second for your model, precision and context length.
06
Where ANVE stands
ANVE has published no speed figures for its units, and nothing on this page is one. The tables are arithmetic from chip specifications, plus tests AMD ran on its own system.
07
Questions people ask
Is memory bandwidth more important than TOPS for local AI?
For writing answers, usually yes: NVIDIA calls that phase memory-bound. TOPS ratings say nothing about bandwidth, so ask for both.
What is a good memory bandwidth for local AI?
There is no single threshold. Divide the bandwidth by the size of the model you want to run to get a ceiling, then decide whether that many tokens a second is enough for your users.
Does adding more memory raise bandwidth?
Not by itself. Capacity decides what fits. Bandwidth depends on the bus width and the data rate.
Why do quantised models run faster?
They have fewer bytes to read for each token. NVIDIA notes that quantisation lets you transfer more parameters over the same amount of bandwidth.
08
What is not real yet
No sealed ANVE unit exists; every image of the appliance on this site is a labelled render. The software runs today on a development machine. No customers yet, no certifications, no delivery date, and the final price is not announced.
Reservations are ₹999 in India or US$12 elsewhere, refundable in full until you confirm an order in writing (terms). Offices that cannot use cloud AI can apply to be one of ten design partners.
