Time to first token The wait before the first word of an answer appears.
Short answer. Time to first token (TTFT) is the delay between sending a request and receiving the first token of the reply. It covers any queueing, loading the model if it is not already in memory, and prefill, the step in which the model reads the whole prompt. TTFT therefore grows with prompt length: if a machine reads 500 prompt tokens a second, an example figure, a 10,000-token document takes about 20 seconds before the answer begins, however fast it writes afterwards. Ollama unloads an idle model after 5 minutes by default, so the first question after a quiet spell also pays a model-load delay.
01
What TTFT includes
Queueing: waiting for other requests to finish when the machine is busy.
Model loading: reading the weights from disk into memory, if the model was unloaded. By default Ollama keeps a model in memory for 5 minutes after its last request.
Prefill: the model processes every prompt token in one parallel pass and builds the KV cache. NVIDIA describes this phase as compute-bound.
The first decode step: producing the first output token.
Reasoning models add a further delay before the first visible word, because they generate thinking tokens first.
02
Prompt length drives the wait
Prefill time is roughly the number of prompt tokens divided by the prompt-processing speed. The table uses three example speeds, not measurements of any product. Real values depend on the model, the hardware and the engine.
Prompt size
At 200 tokens/s
At 500 tokens/s
At 2,000 tokens/s
500 tokens (short question)
2.5 s
1 s
0.25 s
2,000 tokens (about 3 pages)
10 s
4 s
1 s
10,000 tokens (about 15 pages)
50 s
20 s
5 s
50,000 tokens (about 75 pages)
250 s
100 s
25 s
03
Ways to cut it
Send less: retrieve the relevant passages instead of pasting the whole file.
Keep the model loaded. Ollama’s keep_alive setting controls how long it stays in memory, and a value of -1 keeps it loaded indefinitely.
Reuse work. vLLM’s automatic prefix caching reuses the stored cache of a shared start of the prompt, such as a long document asked about repeatedly, so only the new question is processed. It speeds prefill, not generation.
Use a smaller model, or a faster prompt-processing engine, for interactive tasks.
Stream the reply so the first words show as soon as they exist, and show a progress message while the prompt is read.
04
Why it matters for an office
People judge speed by the pause, not by tokens per second. A two-second wait feels normal. A forty-second silence after pasting a long contract feels broken, even if the answer then arrives quickly. Design for it: decide which tasks are interactive, such as a quick question, and which can run in the background, such as summarising a thick file. When several people share one machine, queueing adds to everyone’s TTFT, so measure it under load, not on an idle system.
05
Questions people ask
What is a good time to first token?
There is no single standard. A short chat question should start within a second or two. For a long document, several seconds is normal. Set a target for each task and test it under realistic load.
Is TTFT the same as latency?
No. Total latency is TTFT plus the time per output token multiplied by the number of tokens generated, as Databricks defines it.
Why is the first request of the day slow?
The model may have been unloaded from memory and must be read from disk again. Keeping it loaded removes that delay.
Does streaming reduce TTFT?
No. Streaming shows tokens as they are produced, so the answer seems faster, but TTFT is still the time until the first one.
06
What is not real yet
No sealed ANVE unit exists; every image of the appliance on this site is a labelled render. The software runs today on a development machine. No customers yet, no certifications, no delivery date, and the final price is not announced.
Reservations are ₹999 in India or US$12 elsewhere, refundable in full until you confirm an order in writing (terms). Offices that cannot use cloud AI can apply to be one of ten design partners.
