Do I need a GPU to run AI? No, but memory speed decides how fast, and memory size decides how big.
Short answer. No. llama.cpp, an open-source inference engine, runs on CPUs and can split a model between CPU and GPU when it is larger than the GPU’s memory. What changes is speed. Generating text is limited mainly by how fast the model’s weights can be read from memory, so a rough ceiling is memory bandwidth divided by model size: a laptop with dual-channel DDR5 (89.6 GB/s) reading a 17 GB model tops out near 5 tokens a second, while NVIDIA’s DGX Spark, rated at 273 GB/s, tops out near 16. A GPU, or a chip with fast unified memory, is what makes 27B-class models comfortable and lets several people share them. For one person asking short questions of a 4B to 8B model, a CPU can be enough.
01
The rule of thumb
Databricks’ engineering guide to LLM inference explains why: when a model generates tokens one at a time, speed depends on how quickly the parameters can be loaded from memory rather than on how quickly the chip can compute. Its worked example is a 7B model in 16-bit precision, which moves 14 GB of parameters per token; at 14 milliseconds per token that is 1 TB/s of memory bandwidth in use.
So the ceiling on tokens per second is roughly the memory bandwidth divided by the size of the model in memory. Real speeds are lower. The figures below are arithmetic, not benchmarks.
Hardware (example)
Memory bandwidth
Ceiling for a 5.2 GB model (Qwen3 8B)
Ceiling for a 17 GB model (Gemma 3 27B)
Laptop CPU with dual-channel DDR5-5600 (Intel Core i7-13850HX)
A workstation GPU has its own fast memory. NVIDIA lists the 24 GB RTX PRO 4000 Blackwell at 672 GB/s, more than twice the DGX Spark’s bandwidth. The catch is capacity: a model must fit in the GPU’s memory to get that speed. llama.cpp can run CPU and GPU together for models larger than the GPU’s memory, but the slower memory then sets the pace. Unified memory, where the CPU and GPU share one large pool, trades some bandwidth for much more capacity.
03
What a CPU can and cannot do
It can run a model that fits in system memory: llama.cpp supports plain CPUs, Apple silicon through Metal, and GPU backends including CUDA, HIP, SYCL and Vulkan.
It will be slow for large models, as the ceilings above show.
Ollama tells you where a model loaded: its Processor column shows 100% GPU for a model loaded entirely into the GPU and 100% CPU for one loaded entirely into system memory.
Speech to text needs little: Whisper small needs about 2 GB and large about 10 GB of VRAM, according to its README.
04
Sharing a model between people
Ollama’s default is one parallel request per model, so several people asking at once queue unless you raise OLLAMA_NUM_PARALLEL and have the memory for it. Alibaba’s Qwen3.8 card recommends dedicated serving engines such as SGLang, vLLM or TokenSpeed for production and high-throughput workloads. A team needs enough memory for the model plus several conversations at once, and that is where GPUs and large unified-memory machines earn their price.
05
A simple way to choose
Use
Hardware that is usually enough
One person, short questions, a 4B to 8B model
A recent laptop or mini PC; LM Studio recommends 16 GB of RAM or more
One person, a 27B-class model
Memory for the 17 GB weights plus the conversation and the operating system, with high bandwidth if you want speed
A team sharing a 27B-class model
Fast unified memory or a workstation GPU, and a serving engine rather than a single-user app
Speech to text only
Whisper small needs about 2 GB; large about 10 GB
06
Where ANVE fits
At launch, ANVE One uses AMD Strix Halo class hardware with 64 GB of unified memory, ANVE Office pairs an NVIDIA RTX PRO 4000 Blackwell SFF Edition (24 GB) with 64 GB of system memory, and ANVE Office Pro pairs an NVIDIA RTX PRO 4000 Blackwell (24 GB) with 128 GB of system memory. ANVE has not published speed figures for its hardware; concurrent-user capacity will be published after benchmarks on the shipping unit. ANVE is pre-release.
07
Questions people ask
What does “100% CPU” mean in Ollama?
Ollama’s Processor column shows which memory a model was loaded into: 100% GPU means entirely into the GPU, and 100% CPU means entirely into system memory.
Is more VRAM or more bandwidth better?
Both matter. Memory size decides whether a model fits, and bandwidth decides how fast it answers. If a model spills out of GPU memory, the slower memory sets the pace.
Do I need an NVIDIA card?
Not necessarily. llama.cpp lists backends for NVIDIA (CUDA), AMD (HIP), Apple silicon (Metal), Intel (SYCL) and Vulkan, among others.
Can several people share one laptop-class machine?
For light use, perhaps, but with Ollama’s default of one parallel request per model they queue. Test with your real users before you buy.
08
What is not real yet
No sealed ANVE unit exists; every image of the appliance on this site is a labelled render. The software runs today on a development machine. No customers yet, no certifications, no delivery date, and the final price is not announced.
Reservations are ₹999 in India or US$12 elsewhere, refundable in full until you confirm an order in writing (terms). Offices that cannot use cloud AI can apply to be one of ten design partners.
