Distillation How a small model is taught by a big one, and what the rules say.
Short answer. Distillation is a way to make a small AI model by training it to imitate a larger one. The small model, the student, learns from the large model’s outputs instead of only from raw data, which moves much of the larger model’s skill into a file that is cheaper to run. DistilBERT was 40% smaller and 60% faster than BERT while keeping 97% of its language understanding. DeepSeek released distilled versions from 1.5B to 70B parameters, trained on about 800,000 samples written by its R1 model. Distillation is not quantisation: it changes what the model knows, while quantisation stores the same numbers more coarsely.
01
How it works
In the classic method from Hinton, Vinyals and Dean (2015), the teacher’s class probabilities are used as “soft targets” for training the student, with the teacher’s softmax run at a raised temperature to make them softer. Those probabilities carry more than the right label does: they also say which wrong answers are nearly right. The student, trained to match them, learns more per example than it would from labels alone.
For language models there are two common routes. In the first, the student is trained to match the teacher’s token probabilities. Google’s Gemma 3 report says the Gemma 3 models are trained with distillation. In the second, the teacher writes a large set of answers and the student is fine-tuned on them. DeepSeek did this with R1, fine-tuning Qwen and Llama models on about 800,000 reasoning samples it had written.
02
Examples in numbers
Model
Teacher and student
What the source reports
DistilBERT
BERT taught a model 40% smaller
97% of language understanding kept, 60% faster
DeepSeek-R1-Distill-Qwen-32B
DeepSeek-R1 taught Qwen2.5-32B
AIME 2024 pass@1 of 72.6 and MATH-500 of 94.3 on its model card
Gemma 3 (1B to 27B)
A larger teacher, not named in the abstract
The technical report states the models are trained with distillation
03
What it keeps and what it loses
It keeps what the teacher demonstrates in the training data: formats, styles and, for reasoning data, step-by-step habits.
It may not keep the teacher’s breadth. A small student has fewer parameters to hold facts, so test it on your own knowledge questions.
It can inherit the teacher’s mistakes and biases along with its skills.
A distilled model is a normal model to run: it needs the same memory as any model of its size and precision.
04
The rules around the teacher
Whether you may distil depends on the teacher’s licence or terms. The DeepSeek distilled models carry an MIT licence, and the Qwen-based ones also keep the Apache 2.0 licence of their base. DeepSeek’s model card says its R1 series allows modifications and derivative works, including distillation for training other models. Some hosted services restrict using their outputs this way. Google’s Gemini API terms (effective 23 March 2026) say you may not use the services to develop models that compete with them, and Anthropic’s commercial terms (effective 17 June 2025) bar using the services to train competing AI models. Read the terms of any hosted model before using its answers as training data. This is general information, not legal advice.
05
Why it matters for an office
Most small models you can run in an office were built with some distillation, so a model’s card should say how. Distillation is why a 4B to 8B model can handle summaries, extraction and question answering well enough for daily work. It is also a route an office could use for itself, training a small model on the answers of a larger one for a narrow, repeated task. Check the teacher’s terms first, and keep the examples free of personal data.
06
Questions people ask
Is a distilled model the same as a quantised model?
No. Distillation trains a new, smaller model to imitate a bigger one. Quantisation takes an existing model and stores its numbers with fewer bits. The two can be combined.
Can an office distil its own model?
Technically yes: collect good answers from a larger model for a narrow task and fine-tune a small model on them, for example with LoRA. The teacher’s licence or terms decide whether that is allowed.
Why are distilled reasoning models so good at maths?
Because they were trained on many worked solutions written by a stronger reasoning model. DeepSeek reports that its distilled models surpass their original instruction-tuned counterparts.
Does distillation reduce hallucination?
Not by itself. A student can repeat its teacher’s errors, and a smaller student has less room for facts. Grounding answers in your documents matters more.
07
What is not real yet
No sealed ANVE unit exists; every image of the appliance on this site is a labelled render. The software runs today on a development machine. No customers yet, no certifications, no delivery date, and the final price is not announced.
Reservations are ₹999 in India or US$12 elsewhere, refundable in full until you confirm an order in writing (terms). Offices that cannot use cloud AI can apply to be one of ten design partners.
