What is PagedAttention?

A way of storing the KV cache in small non-contiguous pages, like an operating system’s virtual memory. It avoids wasted space and lets requests share common parts.

Can vLLM run on a laptop?

Its quickstart lists Linux as the platform, with NVIDIA, AMD, Intel, TPU or Ascend hardware, and notes that it also works on macOS through vLLM-Metal for Apple silicon. It is aimed at servers; for a single laptop user, Ollama or llama.cpp are simpler.

Does vLLM run GGUF files?

Support exists but the documentation calls it highly experimental and under-optimised, and it now lives in a separate plugin. The normal route is a Hugging Face model in safetensors form.

Is vLLM faster than Ollama?

For many simultaneous users on GPUs, it is designed to be, because of its batching and memory management. For one user the difference can be small. Measure on your own hardware.

ANVE: the private AI office, on the record. See what that means

ANVE

Private AI glossary · updated 2026-10-16

vLLM
A serving engine built for many users on one set of GPUs.

Short answer. vLLM is an open-source library and server for running language models efficiently when many people use them at once. Its core idea, PagedAttention, manages the KV cache in small pages so that almost no GPU memory is wasted, and in its 2023 paper it improved throughput by 2 to 4 times over the best systems then available at the same latency. It adds continuous batching, prefix caching, speculative decoding, many quantisation formats and multi-LoRA serving, runs on NVIDIA and AMD GPUs and some CPUs, and offers an OpenAI-compatible server. Ollama is simpler for one user; vLLM is designed for throughput.

How it works

The vLLM authors observed that the KV cache of each request is huge and grows and shrinks as the answer is written, so serving systems wasted memory through fragmentation and duplication, which limited how many requests could share a GPU. PagedAttention borrows the idea of virtual memory: the cache is held in fixed-size pages that need not be contiguous, and pages can be shared between requests. The paper measured that in earlier systems only 20.4% to 38.2% of KV cache memory held actual token states. It reports near-zero waste with vLLM and 2 to 4 times the throughput of FasterTransformer and Orca at the same latency.

Anyscale’s June 2023 test of OPT-13B on one 40 GB A100 reported up to 23 times the throughput of naive static batching for vLLM, which pairs continuous batching with its memory optimisations. Continuous batching alone gave about 8 times. Naive static batching had fallen to 81 tokens per second at the most varied output lengths, so read the 23 times as an upper bound.

What it offers, as of October 2026

Feature	What it means for a shared machine
PagedAttention	Less wasted memory, so more users fit on the same GPU
Continuous batching and chunked prefill	New requests join running batches instead of waiting
Prefix caching	A long shared start, such as a policy document, is processed once
Speculative decoding	Lower delay per token for light loads
Quantisation	FP8, MXFP4, INT8, INT4, GPTQ, AWQ and GGUF among others
Multi-LoRA	Several adapters on one base model
Parallelism	Tensor, pipeline, data, expert and context parallelism across GPUs
OpenAI-compatible server	Completions, chat completions, embeddings, audio transcription and translation

vLLM and Ollama side by side

Aspect	Ollama	vLLM
Operating systems	macOS, Windows, Linux and Docker	Linux; macOS on Apple silicon through vLLM-Metal
Hardware	Runs through llama.cpp, whose back ends include Metal, CUDA, HIP, Vulkan and CPUs	NVIDIA and AMD GPUs, CPUs, and plugins for TPUs, Ascend, Gaudi and Apple silicon
Concurrency default	1 parallel request per loaded model, queue of 512	Continuous batching of incoming requests
Model source	Ollama library, or import of GGUF and Safetensors	Hugging Face models; GGUF is experimental
Default API address	localhost:11434	localhost:8000 in the quickstart example

Why it matters for an office

For one person, the engine hardly matters. For 25 or 100 named users sharing a machine, how well the engine packs simultaneous requests decides how many people can work at once and how long each waits. vLLM is aimed at that case, with the trade-off that it is a server product that expects Linux and a supported accelerator. Accounts, roles and an audit trail are not among its listed features, so an office still needs those layered on top.

ANVE’s models are reachable today through the runtime’s own OpenAI-compatible endpoint, which is Ollama or vLLM. The governed ANVE API, with a key per user or agent and every request in the audit trail, is not on ANVE yet.

Questions people ask

What is PagedAttention?: A way of storing the KV cache in small non-contiguous pages, like an operating system’s virtual memory. It avoids wasted space and lets requests share common parts.
Can vLLM run on a laptop?: Its quickstart lists Linux as the platform, with NVIDIA, AMD, Intel, TPU or Ascend hardware, and notes that it also works on macOS through vLLM-Metal for Apple silicon. It is aimed at servers; for a single laptop user, Ollama or llama.cpp are simpler.
Does vLLM run GGUF files?: Support exists but the documentation calls it highly experimental and under-optimised, and it now lives in a separate plugin. The normal route is a Hugging Face model in safetensors form.
Is vLLM faster than Ollama?: For many simultaneous users on GPUs, it is designed to be, because of its batching and memory management. For one user the difference can be small. Measure on your own hardware.

What is not real yet

No sealed ANVE unit exists; every image of the appliance on this site is a labelled render. The software runs today on a development machine. No customers yet, no certifications, no delivery date, and the final price is not announced.

Reservations are ₹999 in India or US$12 elsewhere, refundable in full until you confirm an order in writing (terms). Offices that cannot use cloud AI can apply to be one of ten design partners.

vLLMA serving engine built for many users on one set of GPUs.