Transcription
Large language models today don't just learn from raw text, but also from each other. For instance, Google's Gemini 3 was trained by Gemini, and Meta's smaller Llama 4 models, Maverick and Scout, were trained by the larger Behemoth and Deepseek. We'll get to that. So, this process is called knowledge distillation. Distillation. Distilling. Distilled. Distillation. Distilled. Distillation. Distill. Distilled. Distillation. Distillation, distill, distilling, distill, distill.
Despite the buzz, this concept gets quite often misunderstood. In this video, we'll break down what knowledge distillation really is, so that you don't have to settle for half-truths.
First of all, why does everyone care about knowledge distillation? Well, it's all about scale. This landmark paper from 2020 confirmed what we already suspected: To make models better, we have to go bigger. There are three main directions of scale: compute, data set size, and parameter count. Knowledge distillation helps us scale along the last two, data, and parameters. For data, we hit a wall. We basically scrape the entirety of the internet. But distillation helps overcome this limitation by enabling powerful LLMs to generate more data. And for parameters, one big bottleneck is inference latency because bigger models are slower at generating outputs. Distillation decouples training from inference. We can use a giant model during training, but end up with a smaller and faster model for deployment.
Given these benefits, it's no wonder that every LLM provider is embracing distillation. Google, for example, uses knowledge distillation to create miniature open-source versions of their proprietary Gemini model, like Gemma 2 and Gemma 3. Similarly, Meta's Llama 4 family uses knowledge distillation to shrink their largest model, Behemoth, with its two trillion parameters, into the smaller versions, Scout and Maverick. Deepseek R1, I'm intentionally saving it for last, even though it came out before Gemini 3 and Llama 4. On paper, Deepseek uses knowledge distillation to compress their large R1 model, which is a mixture of experts, into smaller, dense models. But personally, I think calling it distillation is a stretch.
To explain why, we need to go back, way back. It's the year 2006. Twitter just launched, setting the stage for the micro-blogging era. Google acquires YouTube for $1.6 billion, and the first Borat movie hits the theaters. Great success. Meanwhile, researchers at Cornell publish a paper that quietly introduces a powerful idea: model compression. In 2006, models weren't huge. Maybe 500 megabytes to 1 GB. So why did they even need model compression? Well, because they wanted models to fit on PDAs, or personal digital assistants, which had very limited storage. And if you're too young to remember, PDAs looked something like this.
Here's another twist. Back then, the best performing models weren't single models. They were actually ensembles, or collections of thousands of models whose predictions were averaged into a final answer. So let's say we were trying to classify an email as spam or not spam. To get a final verdict, we train 1,000 separate classifiers, then average their output probabilities.
So why were ensembles so popular back then? It's because training used to be noisy. Models were very sensitive to random initialization and data shuffling. So some of the training runs would lead to very good models, and others would just flop. So ensembles were a way of playing this lottery game basically thousands of times. And if you're thinking, "Wait a second, I've heard this before," it's called mixture of experts. They're not quite the same thing. In ensembles, models are trained completely separately and they become replicas or copies of each other. But in mixture of experts, the experts are trained jointly with the gating network, and they learn to become complimentary to each other and develop specialized skills.
So obviously schlepping 1,000 models onto a PDA wasn't practical. That's when the 2006 paper proposed a clever idea: Train one compressed model that predicts this average output directly. So training became a two-step process. Step one: train 1,000 independent classifiers to match the true label from the training data. In this case, if it's a Nigerian prince email, the true label would be 100% spam. After training the 1,000 classifiers, we freeze them, meaning we stop updating their weights. Then we move on to step two: training a compressed classifier. But instead of teaching it to match the true label from the data, we interrogate the 1,000 classifiers, average their outputs, and use that average as a soft label. We call it soft because it's not just a hard yes or no anymore. It's a full probability distribution. Once the compressed classifier is trained, we can throw away the 1,000 originals and ship just the small model onto the device. This way, we've preserved the wisdom of a thousand models, but kept the inference cost tiny.
By now, you might be thinking, "Okay, cool story about ensembles, but I clicked on a video about knowledge distillation." Well, this is knowledge distillation. It's just that the catchy name wasn't coined until 9 years later. Jeffrey Hinton and Jeff Dean introduced it in a 2015 paper while at Google. Fun fact: Hinton went on to win the Turing Award in 2019, and Jeff Dean became Google's chief scientist in 2023 after a long tenure there. So they revisited the 2006 idea, but this time in the context of digit classification. Given a grayscale 28 by 28 pixel image of a handwritten digit, the task was to classify it as one of the 10 digits, 0 through 9. Another fun fact: the data set they used, called MNIST, was published by Yann LeCun, who today is Meta's chief scientist. The same recipe from 2006 applies here, but this time the output distribution is over the 10 digits, not just the two yes and no labels. And building on top of this, the newer paper from Google makes a very interesting discovery: They realize distillation is still useful even without an ensemble. So if we have a single classifier that we want to compress into a smaller model, we are better off training the compressed model on the soft labels rather than directly on the one-hot true label. In this paper, they also coined the teacher-student terminology, which reflects so well the relationship between these two models.
So, what is so special about these soft labels? Well, the key is soft labels carry more information than hard labels. If you squint at this messy five, you might argue it looks a little bit like a three. The hard labels don't reflect this information at all, but the soft labels do. They assign the second highest probability to the digit three. The paper poetically calls this extra information dark knowledge: knowledge about what the input is not. So the teacher does the hard work of surfacing it and hands it to the student explicitly. And as a result, the student has less work to do. It can model the training set more easily with fewer parameters. Kind of like how I spend hours reading research papers and then compress it all into a 15-minute video for you.
So knowledge distillation lets us scale up training by building bigger and better teachers without scaling up inference. Even if we keep the student model small to avoid latency issues, it still gets smarter because it learns from a more powerful teacher. Hinton and his co-authors use a beautiful analogy to describe this split between training and inference: The life stages of a butterfly. The larvae and adult forms are completely different because the butterfly's needs change over time. Similarly, a model needs a large parameter space during training, but at inference time, it needs to be fast, light, and efficient.
This paper introduced a lot of iconic terminology and beautiful metaphors, but there's one term in particular that you've probably been using without knowing that it comes from distillation. And that term is temperature. Today, we use it as a creativity knob. High temperature means more surprising LLM outputs. But originally, temperature was introduced for distillation. The goal was to amplify dark knowledge by artificially increasing the probabilities of the wrong labels. When those probabilities are tiny, like 10^-8, the student can't learn much from them. The gradients are basically zero. But if we boost them even to something like 10^-3, suddenly the student can learn faster.
So how does temperature work mathematically? Well, when making a prediction, the teacher first outputs logits: those raw scores that are eventually turned into probabilities using a function like softmax. Temperature simply means divide the logits by a constant before applying softmax. So a higher temperature smooths the output, and a lower one makes it more peaky.
Now let's fast forward to today. As we saw at the start of the video, a lot of LLM papers mention knowledge distillation, but not all distillation claims are created equal. In the open-source ecosystem, there are three prominent LLM providers that embrace distillation: Google's proprietary Gemini was distilled into the open-source models Gemma 2 and 3; Meta's Llama 4 Behemoth version into Llama 4 Scout and Maverick; and DeepSeek, which is a mixture of experts, was distilled into dense students with a different architecture from their teacher, including Llama 3.1 and Qwen 2.5.
When distilling LLMs, the first question is: At what stage of the training pipeline should we apply distillation? Option one is pre-training. That's when we train the models on a huge data set scraped from the internet using next-word prediction. That's what Llama 4 seems to do based on their blog post. Option two is post-training. That's when we fine-tune the LLM on question-answer pairs to align it with human preferences and to tweak its reasoning abilities. This is where DeepSeek operates. And finally, option three is to apply distillation during both these stages, like Gemini 3 does.
The second question when distilling LLMs is: How exactly does the information transfer between the teacher and the student? When looking into these implementation details, my takeaway is that Google and Meta are doing proper distillation. But for DeepSeek, a more accurate name for their model compression technique is behavioral cloning.
Here's the difference between these two approaches. First, we'll look at proper distillation. Let's assume the teacher is already trained, and we're currently training the student. At each training step, we pick a document from our corpus and have the student predict it one word at a time. Say we're at time step six trying to predict "Matt". Under proper distillation, we feed the incomplete document to both the teacher and the student. And both models output a full probability distribution over the entire vocabulary. Then we train the student using the teacher's output as a soft label. The student weights are then adjusted to minimize the gap between its prediction and the teacher's distribution.
Now let's take a look at what DeepSeek R1 calls distillation. Instead of starting from a pre-collected data set, they ask the teacher to generate it. Then they train the student using regular next-word prediction. But here's the main difference: The labels aren't soft anymore. They're just one-hot. So "Matt" gets 100% of the probability. There's no dark knowledge being passed. The student is simply learning to imitate the teacher's final outputs, not the reasoning behind them. That's why I would rather call it behavioral cloning.
So if proper distillation offers more information to the student and leads to better training, then why doesn't everybody do it? Well, there are two very good reasons. One is that the teacher needs to be a white box. You need to own the teacher in order to access its probability distributions. So if someone, we won't name names, wanted to distill a GPT model outside of OpenAI, then they would be stuck with behavioral cloning. And the second reason is that proper distillation is computationally expensive. For every single word in the training data, we need a full distribution over the entire vocabulary from the teacher to use it as soft labels. Let's do some quick back-of-the-napkin math. Say your training set is 8 trillion tokens, and the vocabulary has 128,000 tokens. That's roughly 10^18 soft label values. If we stored all of them in precision floating-point 32, we would need as much storage as the entire contents of YouTube. Well, at least according to ChatGPT, that's way too much for a single distillation run, even for Google.
So, in practice, models like Gemini don't use the full distribution. Instead, Gemini sampled 256 tokens from the teacher's probability distribution, and any soft labels other than the 256 were zeroed out. That still leaves us with around 10^15 values, which is more manageable, but still massive. So, it's likely these soft labels are computed on the fly and then immediately discarded. Either way, running 8 trillion teacher inferences is not for the faint of heart.
In fact, Llama hints at this issue in their blog post. They use the term "code distillation" to help advertise the cost of all those expensive forward passes through the teacher. In regular distillation, you need two forward passes per example: one during teacher training and one during student training. In step one, when the teacher is being trained, we need a forward pass to produce an output which is compared against the ground truth label. In step two, when the teacher is frozen and the student is being trained, we need another forward pass through the teacher to generate the soft labels for the student. In contrast, code distillation, the trick used by Meta, merges the two steps into one. The teacher and the student are trained at the same time. The teacher produces its probability distribution and updates itself based on the hard labels. Meanwhile, the student produces its own output. The teacher output is readily available, so we can immediately use it as a soft label. But the teacher isn't fully trained yet. It's still learning, which means its soft labels might be somewhat wrong. So early on, code distillation can feel a little bit like the blind leading the blind, and to compensate, we train the student on a combination of soft and hard labels.
I hope this video showed you that distillation is a lot more than generating synthetic data with a powerful LLM. It's a technique with deep roots, nearly 20 years old. And even though it might seem straightforward on the surface, the implementation details can really make it or break it. If you want to dive deeper, I've shared all the companion resources to this video, including my slide deck and my reading list on Patreon for free. So, thanks for watching, and I'll see you next time.