# What Is Model Distillation? How Smaller AI Models Learn

## Executive summary

Model distillation is a technique used to transfer knowledge from a large, highly capable AI model (the Teacher) to a much smaller, more efficient model (the Student). This process allows developers to maintain high AI performance while significantly reducing the model's memory footprint, computational requirements, and deployment cost, making it viable for edge devices like phones and laptops.

## Key takeaways

- Model Distillation Defined: Model distillation is the process of transferring knowledge from a larger model (Teacher) to a smaller model (Student). This is crucial because large models, while capable, require significant memory and computing power, increasing operational costs.
- The Role of Parameters: Model size is typically measured by the number of parameters (internal numbers learned during training). Fewer parameters mean less memory required, lower latency, and higher speed, enabling deployment on resource-constrained hardware.
- Soft Labels and Dark Knowledge: Instead of training the Student on 'hard labels' (the single correct answer), distillation trains it on the Teacher's 'soft labels.' Soft labels provide a probability distribution across all possible answers, revealing 'dark knowledge'—the relationships between plausible and implausible alternatives.
- The Goal of Distillation: The primary goal is to shrink the performance gap between the large Teacher and the small Student, achieving most of the Teacher's capability in a fraction of the size and cost.

## Technical details

- LLM Prediction Mechanism: Large Language Models (LLMs) predict text one token at a time. At each step, the model assigns a likelihood (probability) to every possible next token in its vocabulary, not just the final chosen word.
- Training the Teacher (Hard Labels): The Teacher model is trained conventionally using hard labels (ground truth data), where the model is compared against the known, single correct answer (e.g., 100% probability for 'mat', 0% for all others).
- Training the Student (Soft Labels): The Student model is trained by matching its output distribution to the Teacher's soft labels. The Student adjusts its parameters to replicate the Teacher's full probability distribution (e.g., 60% for 'mat', 20% for 'floor', 10% for 'chair').
- Model Efficiency Trade-offs: While the Student model is smaller, faster, and cheaper to run, it is generally not as capable as the Teacher. Distillation aims to maximize performance retention while minimizing size.

## Practical implications

- Enables the deployment of highly capable AI models on resource-constrained edge devices (phones, laptops).
- Reduces operational costs and latency by minimizing the required computational power and memory.
- Allows engineers to balance high AI capability with practical deployment efficiency.

## Topics

Model Distillation, Large Language Models (LLMs), Machine Learning, Soft Labels, Hard Labels, AI Optimization, Knowledge Distillation Guide, IBM Technology YouTube

Source: https://www.youtube.com/watch?v=4uvq8Cz3A7Y
