ANVE: the private AI office, on the record. See what that means

ANVE

Private AI glossary · updated 2026-10-16

Continuous batching
How one machine serves many people without making them queue.

Short answer. Continuous batching is a scheduling method in which a model server adds new requests to the running batch, and removes finished ones, at every generation step, instead of waiting for a whole batch to complete. It matters because each decoding step reads all the model’s weights from memory, so serving several requests per read multiplies useful output. The Orca paper reported 36.9 times the throughput of FasterTransformer at the same latency, and Anyscale measured about 8 times from continuous batching alone and up to 23 times with vLLM’s memory optimisations, both against naive static batching. vLLM and llama.cpp’s server both use it. Each user’s own speed falls as the batch grows, so capacity planning needs both numbers.

01

How it works

In static batching, a group of requests starts together and the batch stays fixed until every answer is finished. Short answers wait for long ones, and the hardware idles. Continuous batching, also called iteration-level or dynamic batching, checks after every step: when a sequence ends, a waiting request takes its place. Orca introduced the idea of scheduling at the granularity of an iteration instead of a request, together with selective batching of operations.

Hugging Face adds two supporting ideas. Ragged batching joins sequences end to end and uses attention masks so they do not interact, avoiding padding. Chunked prefill splits long prompts so that prompt processing and decoding can share a batch.

A small example: four requests need 20, 40, 80 and 160 tokens. A static batch runs 160 steps, and the slots that finish early sit idle, so only 300 of 640 slot-steps do useful work, or 47%. With continuous batching, each freed slot is refilled at once.

02

Measured gains

SourceSetupResult
Orca, OSDI 2022GPT-3 175B against NVIDIA FasterTransformer36.9 times the throughput at the same latency
Anyscale, June 2023OPT-13B on one 40 GB A100; output lengths up to 1,536 tokens; baseline is naive static batching, which fell to 81 tokens per secondAbout 8 times with continuous batching alone (Ray Serve, text-generation-inference); up to 23 times with vLLM; about 4 times with an optimised static system, FasterTransformer
vLLM paper, 2023Popular models against FasterTransformer and Orca2 to 4 times the throughput at the same latency, from better memory management
03

The trade-offs

  • Per-user speed falls as the batch grows. Throughput counts all users together; each person sees less.
  • Memory limits the batch. Every active request holds its own KV cache, so memory can cap the number of simultaneous conversations before compute does.
  • Waiting grows under load. Anyscale found that continuous batching improved latency at every percentile in its tests at 1 and 4 queries per second, but noted that as a server fills up there are fewer chances to slot in new requests.
  • Read the baselines carefully. The Orca and Anyscale comparisons are against static batching, which current servers such as vLLM and llama.cpp’s server have replaced with continuous batching.
04

What to look for in practice

EngineSettingDefault in the documentation
llama.cpp server-np, --parallel: server slots-1, meaning automatic
llama.cpp server-cb, --cont-batchingEnabled
OllamaOLLAMA_NUM_PARALLEL; OLLAMA_MAX_QUEUE1; 512
vLLMContinuous batching is a core featureBuilt in; concurrency depends on memory and configuration
05

Sizing an office

Count simultaneous requests, not named users. Not everyone with an account is waiting on the model at the same moment, so measure how many requests overlap at your busiest time. Multiply that by the cache each conversation needs: with Qwen3-32B at 32,768 tokens, about 8.6 GB each. Then ask what speed each person gets at that load, and aim above reading speed of about 5 tokens a second. ANVE’s launch tiers are specified in named users, which is a different number from concurrent requests, so ask any vendor for both.

06

Questions people ask

Is continuous batching the same as batching requests?
No. Ordinary batching waits for a full group and runs it to completion. Continuous batching changes the group at every step, so finished requests leave and new ones join immediately.
Does it make each answer faster?
No. It raises the total output of the machine. A single user alone sees the same speed, and each user’s speed can fall as more people join the batch.
Which software does it?
vLLM and the llama.cpp server both list continuous batching, and llama.cpp enables it by default. Ollama handles simultaneous requests through its parallel and queue settings.
What is chunked prefill?
Splitting a long prompt into pieces so that prompt processing for one request can be mixed with decoding for others in the same batch.
07

What is not real yet

No sealed ANVE unit exists; every image of the appliance on this site is a labelled render. The software runs today on a development machine. No customers yet, no certifications, no delivery date, and the final price is not announced.

Reservations are ₹999 in India or US$12 elsewhere, refundable in full until you confirm an order in writing (terms). Offices that cannot use cloud AI can apply to be one of ten design partners.

08

Sources