Speculative decoding A draft-and-check trick that speeds up answers without changing them.
Short answer. Speculative decoding speeds up text generation by letting a fast draft, such as a small model, guess several next tokens at once and having the large model check all of them in a single pass. Correct guesses are kept, the first wrong one is replaced, and the output has the same distribution the large model would have produced alone. The original papers reported 2 to 3 times faster decoding for T5-XXL and 2 to 2.5 times for Chinchilla 70B. It helps most when one or a few users are waiting and the machine is limited by memory bandwidth; at high load the gain shrinks.
01
How it works
Decoding is slow because each token needs one full read of the model’s weights from memory. Checking several tokens costs little more than producing one, because the weights are read once either way. So a cheap draft proposes, say, four tokens. The large model scores all four positions in one pass. It accepts the guesses that fit its own distribution, up to the first one it rejects, replaces that one with a token of its own, and continues. Every pass yields at least one token, and up to the number of guesses plus one.
Two papers set this out in late 2022 and early 2023. They show the method gives the same output distribution as the large model alone, so quality is unchanged, and that no retraining is needed. The first paper found that a draft about two orders of magnitude smaller than the target usually worked best.
02
How much it can help
The gain depends on how often the draft’s guesses are accepted. The first paper’s table assumes the draft is free, so real gains are somewhat lower. The pair of numbers is the acceptance rate α and the number of guesses per pass γ.
Acceptance rate α
Guesses per pass γ
Speed against normal decoding
0.6
2
1.96 times
0.7
3
2.53 times
0.8
5
3.69 times
0.9
10
6.86 times
03
Ways to draft
Method
What it needs
Where it is offered
Small draft model
A second, much smaller model of the same family
vLLM; llama.cpp server; Ollama has a draft_num_predict setting for draft models
N-gram or suffix lookup
Nothing extra: it copies likely continuations from text already seen
vLLM; llama.cpp server. The first paper notes that copying from context can work where long sequences repeat, as in summarising or editing
EAGLE or multi-token prediction heads
Extra prediction layers trained with or for the model
vLLM; llama.cpp server
04
When it helps and when it does not
Helps: one or a few users, long answers, memory-bound machines, and predictable text such as code, structured output or quotations from the prompt.
Helps less: many simultaneous users. vLLM’s documentation frames speculative decoding as reducing delay under medium-to-low request rates, and it rates the gain for throughput-focused use as medium.
Costs: memory for the draft model and its own cache, and more complex settings.
Quality: vLLM describes the sampling as theoretically lossless up to the precision limits of hardware numerics; tiny numeric differences can still occur.
Measure with your own prompts. The gain on a quoted-source summary can differ a lot from the gain on free writing.
05
Why it matters for an office
It is a way to make a shared machine feel quicker to the person typing, without a bigger GPU and without a change in wording. It does not change how many people the machine can serve at once, and it adds a moving part to maintain. When a vendor quotes a speed, ask whether speculative decoding was switched on, and whether the figure is for one user or many.
06
Questions people ask
Does speculative decoding change the answer?
In theory no: both papers show the large model’s output distribution is preserved. In practice vLLM notes that tiny differences from numerics and batch size can occur.
Do I need two models?
Not always. The classic method uses a small draft model. N-gram lookup needs no extra model, and EAGLE or multi-token prediction use extra prediction layers added to the main model.
Does Ollama support speculative decoding?
Its Modelfile reference lists a draft_num_predict parameter for the number of draft tokens when a draft model is available. Check the current documentation for supported models.
Will it help many users at once?
Less so. A busy server is already making good use of each memory read, so there is less to gain. vLLM presents it as a latency tool for lighter loads.
07
What is not real yet
No sealed ANVE unit exists; every image of the appliance on this site is a labelled render. The software runs today on a development machine. No customers yet, no certifications, no delivery date, and the final price is not announced.
Reservations are ₹999 in India or US$12 elsewhere, refundable in full until you confirm an order in writing (terms). Offices that cannot use cloud AI can apply to be one of ten design partners.
