Can one machine run several AI models at once? A memory budget for a realistic office stack, and what the runtimes do.
Short answer. Yes, as long as the models fit in memory together. Ollama can load several models at once, three per GPU by default, provided each fits entirely in GPU memory; otherwise requests queue until another model is unloaded. A realistic office stack of a 27B chat model, a small 4B model, speech recognition, an embedding model and an image model comes to about 39 GB with caches, which fits in 48 GB of graphics memory (the share AMD allows on a 64 GB machine) but not in a 24 GB card. vLLM serves one model per server process, so several models means several processes sharing the GPU. Figures are estimates.
01
A memory budget for an office stack
The stack is the kind of mix an office needs: a main chat model, a small fast model, speech recognition, an embedding model for search and an image model. All figures are estimates using weights = parameters in billions × bits per weight ÷ 8, or the published file size, plus NVIDIA’s per-token cache formula. In practice you would not keep the image model loaded all day.
Component
Memory
Basis
27B model, Q4_K_M (Qwen3.8-27B)
16.2 GB
Parameters × 4.8 ÷ 8
4B model, Q4_K_M (Qwen3-4B class)
2.4 GB
Parameters × 4.8 ÷ 8
Whisper large-v3, 16-bit
3.1 GB
Published file
Embedding model BGE-M3, about 16-bit
1.1 GB
Half the 2.27 GB published file, estimate
Image model SDXL, 16-bit
7.1 GB
Published files
Caches: two 27B conversations at 32,768 tokens
4.6 GB
NVIDIA’s formula, plus 0.15 GB state each
Caches: two 4B conversations at 8,192 tokens
2.4 GB
NVIDIA’s formula
Runtime overhead
2.0 GB
Planning allowance
Total
39.0 GB
Sum of the rows above
02
Where it fits
Memory budget
Fits the 39 GB stack?
24 GB graphics card
No. The main model plus caches takes about 23 GB before anything else
48 GB of a 64 GB Strix Halo machine (AMD’s stated limit)
Yes, with about 9 GB spare
96 GB of a 128 GB Strix Halo machine
Yes, with about 57 GB spare for more conversations or a second large model
03
How the runtimes handle it
Ollama: supports two levels of concurrency. Several models can be loaded at once if memory allows, and each model can serve parallel requests. OLLAMA_MAX_LOADED_MODELS defaults to 3 times the number of GPUs (3 for CPU-only). On a GPU, a new model must fit entirely in VRAM to be loaded alongside others; otherwise new requests wait until a loaded model becomes idle and is unloaded.
Ollama keeps a model in memory for 5 minutes after its last request by default. The keep_alive setting changes that, and a negative value keeps it loaded.
vLLM: the vllm serve command takes one model, and its documentation says it pre-allocates a share of GPU memory (set by gpu_memory_utilization) for the cache. To serve several models you run several server processes and divide the GPU between them.
Reloading costs time: a model swapped out and back in must be read from disk again, which is why the storage question matters.
04
What this means for ANVE’s plan
ANVE One is specified with a 27B-class default model (Qwen 3.8 27B) plus a fast 4B model on 64 GB of unified memory, and in its development build the software also runs image generation through ComfyUI and vision models alongside the language models. These are launch specifications with indicative prices; no sealed unit exists yet, and ANVE has not published concurrency or speed figures.
05
Questions people ask
How many models can Ollama keep loaded?
By default three per GPU (three for CPU-only), if each fits in memory. It unloads an idle model to make room for a new request and queues requests until it can.
Can two models share one GPU in vLLM?
Yes, by running two server processes and giving each a share of the GPU’s memory. vLLM’s documentation describes pre-allocating a share of memory for the cache, so the shares must not exceed what the card has.
Does running several models slow each one down?
They compete for memory bandwidth and compute whenever they work at the same moment. When they take turns, the cost is mostly the time to load a model that was swapped out.
06
What is not real yet
No sealed ANVE unit exists; every image of the appliance on this site is a labelled render. The software runs today on a development machine. No customers yet, no certifications, no delivery date, and the final price is not announced.
Reservations are ₹999 in India or US$12 elsewhere, refundable in full until you confirm an order in writing (terms). Offices that cannot use cloud AI can apply to be one of ten design partners.
