LLM distillation is the process of training a smaller student model to reproduce the answers or probability distributions of a larger teacher model. As a result, the compact model solves the chosen task almost as well as the larger one, but uses less GPU memory and responds faster. The method was described by Geoffrey Hinton and coauthors in 2015, and today it has been used to train, for example, Gemma 2 with 2 and 9 billion parameters and six DeepSeek-R1-Distill models.
Prepared by AIrassvet October 23, 2026 for executives and CIOs who are deciding how to run a language model on their own servers more cheaply and faster. We rely on research papers and model cards, with links provided at the end. This article includes no original distillation experiments: all results belong to the researchers and model developers.
Contents
- Quick Summary
- What This Article Covers and Does Not Cover
- What Distillation Is in Simple Terms
- How Distillation Works: Three Approaches
- Examples of Distilled Models
- Distillation, Quantization, Fine-Tuning, and RAG: What’s the Difference?
- Where Distillation Is Used in Business
- How a Distillation Project Works
- Limitations and Risks
- Glossary
- Frequently Asked Questions
- Primary Sources and Method
Quick Summary
- The core idea: a large teacher model generates answers or probabilities, and a smaller student model learns to imitate them.
- Why it matters: the student is much smaller, so it is cheaper to run on your own servers and returns answers faster.
- The tradeoff: the student usually falls short of the teacher and works well only in the domain it was trained on.
- How it differs from quantization: quantization compresses the same model, while distillation trains a new, smaller one.
- Legal question: the teacher’s license or terms may restrict training other models on its outputs.
What This Article Covers and Does Not Cover
We explain distillation for text-based language models: how it works, where it is used, and what its limitations are. We do not cover quantization, which has its own separate article “LLM Quantization: What It Is”, or pruning and step-by-step training setup. General fine-tuning questions are covered in FAQ on Fine-Tuning LLMs on Company Data.
What Distillation Is in Simple Terms
Knowledge distillation is a way to transfer the skills of a large model to a smaller one. The large model is called the teacher, and the smaller one is called the student. The student learns not only from the correct answers in the data, but also from how the teacher responds.
The idea was introduced by Geoffrey Hinton, Oriol Vinyals, and Jeff Dean in the 2015 paper “Distilling the Knowledge in a Neural Network.” They showed that the knowledge of an ensemble of many models can be compressed into a single model that is easier to deploy for a large number of users. The key technique is to train the student on the teacher’s “soft” outputs: not on a single correct option, but on the probabilities of all options.
Analogy: a student chef learns not from a single note saying “the dish is done,” but by watching how the master chooses between several techniques and how confident they are in each one. In that way, the student absorbs not only the result, but also the decision-making process.
How Distillation Works: Three Approaches
1. Probability-based distillation. For each step, the teacher produces a probability distribution for the next token, and the student learns to match that distribution. Hinton and coauthors proposed “softening” the distribution with temperature: at a higher temperature, you can see which incorrect options the teacher considers close to the right one, and that information helps the student. This method requires full access to the teacher’s outputs, which means an open model.
2. Response-based distillation. The teacher generates ready-made answers for a large set of prompts, and the student is usually fine-tuned on prompt-answer pairs from the teacher. Yun Kim and Alexander Rush described this approach for machine translation in 2016: their best student ran 10 times faster than the teacher with only a small loss in quality. This method also works for a closed model available only through an API, if its terms allow it.
3. Distillation on the student’s own outputs. The student generates answers, the teacher evaluates them, and the student is corrected where it went wrong. This approach is described, for example, in the 2023 MiniLLM paper: the authors replaced the standard training objective with reverse Kullback-Leibler divergence so that the student would not overestimate low-probability teacher outputs.
In practice, these methods are often combined: first the student is fine-tuned on the teacher’s answers, then it is refined using probabilities or its own outputs.
Examples of Distilled Models
| Model | Teacher and Method | What the Authors Report |
|---|---|---|
| DistilBERT, 2019 | BERT, distillation during pretraining | A model 40% smaller and 60% faster than BERT, retaining 97% of its language understanding capabilities |
| Gemma 2 2B and 9B, 2024 | Knowledge distillation based on Hinton instead of next-token prediction training | Best results for their size, competing with models 2–3 times larger |
| DeepSeek-R1-Distill, 2025 | DeepSeek-R1, fine-tuned on 800,000 teacher-generated examples | Six models from 1.5 billion to 70 billion parameters based on Qwen2.5 and Llama 3 |
The DeepSeek example is a telling one. The teacher, DeepSeek-R1, is a 671-billion-parameter model, of which 37 billion are active during generation. According to the model card, the student DeepSeek-R1-Distill-Qwen-32B solves 72.6% of AIME 2024 problems on the first try, compared with 79.8% for DeepSeek-R1 itself and 63.6% for OpenAI o1-mini. These are the developer’s results on a math benchmark, not a guarantee of quality for your own tasks.
Distillation, Quantization, Fine-Tuning, and RAG: What’s the Difference?
| Method | What Changes | What You Get | When to Choose It |
|---|---|---|---|
| Distillation | A new, smaller model is trained on the teacher’s outputs | A compact model that behaves like the teacher on the chosen task | The task repeats often, and the large model is too expensive or too slow |
| Quantization | The same weights are stored at lower precision | The same model with lower memory requirements | The model meets your quality needs but does not fit your hardware |
| Fine-tuning | The model’s weights are changed using your examples | A model that follows your format and style better | You need a consistent response format or specialized terminology |
| RAG | The model does not change; it is given retrieved documents | Answers with links to current documents | Knowledge changes often, and sources are needed |
The methods are compatible: a distilled model can be quantized and connected to RAG. We covered how to choose between RAG, fine-tuning, and prompts in a separate comparison.
Where distillation is used in business
Distillation makes sense where the same narrow task is performed thousands of times and the large model’s quality is already good enough. Typical scenarios:
- Classification and routing of inquiries, emails, and requests into a fixed list of categories.
- Data extraction from documents into predefined fields: company details, amounts, dates, item names.
- Template-based responses in support, where the answer is built from a limited set of facts.
- Operation in a closed environment: the large model is used once as a teacher, and a smaller model runs in production on your own servers.
- Fast responses on modest hardware: on workstations, in branch offices, or on equipment without powerful GPUs.
What tasks small models can handle at all, we covered in our article on small language models.
How a distillation project works
- Choose the task and the quality metric. For example, the share of correctly identified categories on a validation set.
- Choose the teacher and verify its license. The teacher should perform the task well enough: the student usually does not outperform the teacher.
- Collect input data. Real queries, emails, or documents on which the teacher will generate answers.
- Get and filter the teacher’s answers. Incorrect answers are removed, otherwise the student will learn them.
- Train the student. Usually, an existing small open model is fine-tuned rather than trained from scratch.
- Compare the student with the teacher on a holdout set that was not used in training.
- Launch and monitor. Track quality on real data and repeat distillation if the task changes.
Limitations and risks
- The student inherits the teacher’s mistakes. If the teacher makes systematic errors, the student will repeat them without warning.
- Narrow specialization. The student performs well on the task it was trained on and may do noticeably worse on others.
- You need many examples. DeepSeek used 800,000 examples for its students. For a narrow task, fewer are needed, but the volume is still determined by testing.
- Licenses and terms. The Gemma Terms of Use, which govern Gemma 1–3, classify anything trained by transferring Gemma behavior, including distillation, as derivative models. Commercial API terms may also restrict training models on responses. Details are in our FAQ on open LLM licenses.
- Data is sent to the teacher. If the teacher is a cloud service, all input data for distillation is sent to the provider.
- No test, no result. The numbers in articles and model cards come from their benchmarks, not from your data.
Glossary
- Teacher — the large model whose answers or probabilities serve as the reference.
- Student — the smaller model trained to imitate the teacher.
- Soft targets — the probabilities of all answer options, not just the correct one.
- Temperature — a parameter that smooths the teacher’s probability distribution.
- Logits — the model’s raw scores before they are converted into probabilities.
- Kullback-Leibler divergence — a measure of the difference between two probability distributions that is minimized when training the student.
- SFT (supervised fine-tuning) — fine-tuning a model on pairs of “prompt — correct answer.”
Frequently asked questions
How is distillation different from fine-tuning?
Fine-tuning changes the model on your examples, while distillation transfers the behavior of a large model into a smaller one. Technically, response-based distillation is also fine-tuning, but the examples are created by the teacher model, not by people. That is why distillation is chosen when the large model already solves the task well and needs to be replaced with a cheaper one.
Can you distill a closed model available through an API?
Technically, yes, you can distill from its responses, but first check the service terms: they may restrict training other models on those responses. Also, all input data will go to the provider. Open models are easier to work with, but they also have terms: for example, the Gemma Terms of Use for Gemma 1–3 consider a model trained via Gemma distillation to be derivative.
How much data is needed for distillation?
There is no fixed number: the amount depends on the task and the size of the student. DeepSeek used 800,000 examples to transfer general reasoning ability. For a narrow task, such as inquiry classification, you usually start with a smaller dataset and increase it until the student’s quality on a holdout test stops improving.
Do you need distillation if you can quantize the model?
Not always. If the original model fits your hardware after quantization and responds fast enough, quantization is sufficient. Distillation is needed when even the quantized large model is still too expensive for your query volume, or when the task is narrow and a small specialized model can handle it just as well.
Primary sources and method
We checked the arXiv article abstracts, the DeepSeek-R1 card on Hugging Face, and the Gemma terms on October 23, 2026:
- Distilling the Knowledge in a Neural Network — Hinton, Vinyals, Dean, 2015.
- Sequence-Level Knowledge Distillation — Kim, Rush, 2016.
- DistilBERT — Sanh et al., 2019.
- MiniLLM: On-Policy Distillation of Large Language Models — Gu et al., 2023.
- A Survey on Knowledge Distillation of Large Language Models — Xu et al., 2024.
- Gemma 2: Improving Open Language Models at a Practical Size — Google DeepMind, 2024.
- DeepSeek-R1 and model card with student results — DeepSeek, 2025.
- Gemma Terms of Use — Google.
Limitations: percentages and benchmarks are cited from the authors’ data; we did not reproduce them. Distillation methods are evolving rapidly, so before starting a project, it’s worth checking the latest research.
How AIrassvet helps you get a compact model for your own servers
Distillation pays off when the task is well defined and you have data for validation. On the AIrassvet website you can find the service areas that address this need:
- LLM implementation and fine-tuning, including deployment on the client’s servers: selecting a teacher and a student model for the task and available hardware;
- ML solutions for repetitive classification and forecasting tasks, where a smaller model replaces a more expensive one;
- process audit, data preparation, and testing: a validation set, comparing the student model with the teacher, and determining whether distillation is needed at all.
The first step is to choose one process in which the model handles repetitive requests, and document the current result, data sources, data-processing constraints, and the acceptance criteria.
Bottom line: distillation is a way to make a good model affordable
Distillation does not make the model smarter than the teacher. It transfers what the larger model already does well in a narrow task into a smaller model that is easier and cheaper to run.
Start by checking whether the larger model performs your task well enough, whether its license allows distillation, and whether quantization is sufficient. If the answers are yes, yes, and no, distillation is the next step.
