# What Is Quantization? How AI Models Get Smaller & Faster

## Executive summary

Quantization is a critical technique for compressing large AI models (LLMs) by storing their parameters at lower precision (e.g., INT8 or INT4). This process significantly reduces memory footprint and increases inference speed, making massive models deployable on constrained hardware. The core trade-off involves balancing memory savings and computational efficiency against potential loss of model accuracy.

## Key takeaways

- Purpose of Quantization: Quantization is the process of storing numerical information at a lower precision to save space or computation, particularly vital for managing the massive parameter counts of LLMs (e.g., Llama 3 with 70 billion parameters).
- Memory Scaling: A 70B parameter model stored in FP16 requires approximately 140 GB of memory. Quantizing to INT8 reduces this to about 70 GB, and further to INT4 reduces it to 35 GB, making the model much more manageable.
- Quantization Methods: Quantization can be applied via Post-Training Quantization (PTQ), which compresses an already trained model (fast and common), or Quantization-Aware Training (QAT), which simulates lower precision during training, leading to better accuracy retention at aggressive levels like INT4.

## Technical details

- Precision Formats: Common precision formats include FP16 (16 bits/2 bytes), FP8 and INT8 (8 bits/1 byte), and INT4 (4 bits/half a byte). Model names often reflect the precision, such as 'quantized to FP8' or 'weights are at 8 bit precision while activations are still at 16 bit.'
- Quantization Process: Quantization happens layer by layer. The process involves mapping the full range of values in a layer down to a smaller set of values that fit in fewer bits, minimizing information loss through careful scaling and rounding.
- Accuracy vs. Precision: Reducing precision reduces the memory footprint but also the level of detail. While INT8 generally maintains accuracy, INT4 carries a higher risk of accuracy loss, though careful mapping can keep accuracy above 90% of the original model's performance.

## Practical implications

- Build engineers must manage the trade-off between model size/speed (favored by INT4/INT8) and model accuracy (favored by FP16/BF16) when deploying LLMs.
- The choice between PTQ (simpler, faster) and QAT (more compute time, better accuracy retention) dictates the model optimization pipeline.
- The memory requirements for large models scale dramatically with precision; optimizing quantization is key to fitting models onto limited GPU memory.

## Topics

Quantization, LLM Compression, AI Inference, Memory Optimization, Machine Learning Model Deployment, IBM Technology

Source: https://www.youtube.com/watch?v=mmns33BsBcY