What Is Quantization? How AI Models Get Smaller & Faster
Quantization is the process of storing numerical information at a lower precision to save space or computation. It is a key technique for compressing LLMs, reducing a 70B parameter model's memory requirement from 140 GB (FP16) to 35 GB (INT4).
Read article