Can a 24 GB card run a 70B model?

Only partly. The 4-bit file is 42.5 GB, so about half of it would run from system memory through llama.cpp’s CPU and GPU split, and the answers would be much slower than a model that fits.

Can two 24 GB cards be combined?

Ollama spreads a model across several GPUs when it does not fit on one. That adds capacity, but its documentation notes that fitting on a single GPU usually performs best because less data crosses the PCI bus.

Is 24 GB enough for image generation?

For SDXL, yes: its files total about 7.1 GB. For larger image models such as FLUX.1 schnell, whose transformer file is 23.8 GB, you need offloading or a smaller format.

Is 24 GB enough for a small team?

Yes for a 4-bit 27B model if conversations stay short: about 8 at 8,192 tokens or 2 at 32,768 tokens fit in the leftover memory. Long documents from many people at once need more memory.

ANVE: the private AI office, on the record. See what that means

ANVE

Private AI, answered · updated 2026-10-16

Is 24 GB of VRAM enough for local AI?
What fits on a 24 GB card, and where it stops.

Short answer. Yes for one or two people using models up to about 32B parameters at 4-bit, and no for a 70B model or for many people reading long documents at once. A 24 GB card holds a 4-bit 27B model (a 16.5 GB file) with a few gigabytes left for conversation memory, a 4-bit 32B model (19.8 GB) at short context, and gpt-oss-20b (12.1 GB) with room to spare. A 70B model is a 42.5 GB file at 4-bit and would spill into slower system memory. Totals below are estimates built from published file sizes.

What fits in 24 GB

Total = published file size + KV cache at 16 bits + 2 GB of runtime overhead, all estimates. Qwen3.8-27B also adds 0.15 GB of fixed state per conversation, my estimate from its configuration. For gpt-oss-20b the cache counts every layer, an upper bound, because half its layers use a 128-token sliding window. The KV cache uses NVIDIA’s formula: 2 × layers × key/value heads × head size × bytes, per token.

Model	Weights file	Total at 8,192 tokens	Total at 32,768 tokens	Verdict
gpt-oss-20b (MXFP4)	12.1 GB	14.5 GB	15.7 GB	Yes, with room to spare
Qwen3.8-27B (UD-Q4_K_M, Unsloth build)	16.5 GB	19.1 GB	20.8 GB	Yes
Qwen3-30B-A3B (Q4_K_M)	18.6 GB	21.4 GB	23.8 GB	Yes at 8,192 tokens; very tight at 32,768
Qwen3-32B (Q4_K_M)	19.8 GB	23.9 GB	30.3 GB	Barely at 8,192 tokens; no at 32,768
Llama 3.3 70B (Q4_K_M)	42.5 GB	47.2 GB	55.3 GB	No

How many conversations can share what is left

This is a memory limit only. How quickly the card can serve those conversations at once depends on its bandwidth and the serving software, which this page does not measure.

Model	Left after weights and 2 GB overhead	Conversations at 8,192 tokens	Conversations at 32,768 tokens
Qwen3.8-27B (UD-Q4_K_M, Unsloth build)	5.5 GB	About 8	2
Qwen3-30B-A3B (Q4_K_M)	3.4 GB	About 4	1
Qwen3-32B (Q4_K_M)	2.2 GB	1	0

What happens when a model does not fit

llama.cpp can split a model between the card and system memory, but its README describes this as partly accelerating a model larger than the card, so only the part on the card runs at full speed. Ollama’s documentation adds that a second model can only be loaded alongside the first if it fits entirely in VRAM; otherwise requests queue until the first is unloaded. In practice a 24 GB card serves one large model at a time.

NVIDIA lists the RTX PRO 4000 Blackwell with 24 GB of GDDR7 with ECC, 672 GB/s of memory bandwidth, a 145 W maximum and a single-slot form. ANVE Office Pro is specified around this card with 128 GB of system memory, for up to 100 named users, and ANVE Office around its SFF Edition with 64 GB of system memory, for up to 25. That is a launch specification, and named users are not simultaneous requests: the arithmetic above is why.

Other work that shares the card

Speech recognition: the Whisper large-v3 file is about 3.1 GB at 16-bit.
Image generation: the SDXL base files total about 7.1 GB at 16-bit, while the FLUX.1 [schnell] transformer alone is 23.8 GB, so it needs offloading or a smaller format on a 24 GB card. AMD’s own FAQ says FLUX.1 schnell can need 32 GB in partial-offload mode.
Embeddings and a second small model: usually 1 to 3 GB each, and they compete with conversation memory.

Questions people ask

Can a 24 GB card run a 70B model?: Only partly. The 4-bit file is 42.5 GB, so about half of it would run from system memory through llama.cpp’s CPU and GPU split, and the answers would be much slower than a model that fits.
Can two 24 GB cards be combined?: Ollama spreads a model across several GPUs when it does not fit on one. That adds capacity, but its documentation notes that fitting on a single GPU usually performs best because less data crosses the PCI bus.
Is 24 GB enough for image generation?: For SDXL, yes: its files total about 7.1 GB. For larger image models such as FLUX.1 schnell, whose transformer file is 23.8 GB, you need offloading or a smaller format.
Is 24 GB enough for a small team?: Yes for a 4-bit 27B model if conversations stay short: about 8 at 8,192 tokens or 2 at 32,768 tokens fit in the leftover memory. Long documents from many people at once need more memory.

What is not real yet

No sealed ANVE unit exists; every image of the appliance on this site is a labelled render. The software runs today on a development machine. No customers yet, no certifications, no delivery date, and the final price is not announced.

Reservations are ₹999 in India or US$12 elsewhere, refundable in full until you confirm an order in writing (terms). Offices that cannot use cloud AI can apply to be one of ten design partners.

Is 24 GB of VRAM enough for local AI?What fits on a 24 GB card, and where it stops.