# Lecture 122:Disaggregated Quantization

## Executive summary

This lecture introduces Disaggregated Quantization, a novel approach to optimizing Large Language Model (LLM) inference by recognizing and specializing the computational pathways for the two distinct phases: prefill (input processing) and decode (token generation). The core finding is that prefill and decode are bottlenecked by different hardware principles—prefill is compute-bound, while decode is memory-bound. Disaggregated Quantization proposes using separate, specialized quantization formats and weights for each phase, leading to significant accuracy and speed improvements, especially when combined with advanced techniques like weight-only quantization and offloading.

## Key takeaways

- Prefill vs. Decode Bottlenecks: Modern LLM inference involves two phases: prefill (processing the entire input sequence) and decode (generating tokens one by one). Hardware analysis shows that prefill is compute-bound (scaling with matrix multiplications), while decode is memory-bound (limited by weight movement). [0:04:30]
- Disaggregated Quantization Principle: Instead of using a single quantization format for both phases, Disaggregated Quantization specializes the formats and weights for prefill and decode. This approach leverages the inherent separation of input (red) and output (blue) data found in post-training data. [0:08:10]
- Full Disaggregation Benefits: Full Disaggregation uses entirely separate weight sets for prefill and decode. This allows for higher-bit-width weights (e.g., 4-bit) for the prefill model while maintaining the speed and low-bit quantization (e.g., 2-bit) for the decode model, resulting in significantly higher accuracy on both prefill-heavy and decode-heavy tasks. [0:14:20]
- Offloading Prefill Model: For long context sequences (e.g., >8,000 tokens), the prefill model can be offloaded to SSD and loaded layer-by-layer. Since the computation cost grows faster than the constant loading cost, the loading overhead can be fully overlapped with computation, maintaining high efficiency. [0:23:00]

## Technical details

- Quantization Formats: The lecture discusses specialized formats like NVFP4 (Nvidia proprietary 4-bit float) and general 3-bit/4-bit weight-only formats. For compute-bound prefill, both weights and activations must be quantized. For memory-bound decode, only weight quantization is necessary, making weight-only formats preferable. [0:06:10]
- Quantization Aware Distillation: The proposed optimization method uses Quantization Aware Distillation (QAD) to train the model. It minimizes the KL divergence between the quantized, phase-aware model and the unquantized teacher model, ensuring the model learns to mimic the teacher while minimizing quantization error. [0:08:40]
- Architecture Modification: The separation is implemented at the linear layer level by using binary masks derived from the prefill/decode separation, allowing the model to select the correct computational pathway for each phase. [0:09:30]
- Offloading Strategy: To implement offloaded prefill, the model weights are loaded from SSD one block/layer at a time. This is efficient for long sequences because the computation cost grows faster than the constant loading cost, allowing the loading overhead to be fully hidden behind computation. [0:23:00]

## Practical implications

- Engineers can significantly boost LLM accuracy and speed by treating prefill and decode phases as separate, specialized computational units.
- The use of weight-only quantization is a key optimization for memory-bound decode phases, reducing quantization error and complexity.
- For long-context applications, implementing SSD offloading for the prefill model can maintain performance by overlapping loading costs with computation.
- The technique can be applied to augment existing, frozen (black box) quantized decoders by training a dedicated prefiller model.

## Topics

Large Language Models (LLMs), Quantization, Model Optimization, Hardware Acceleration, Transformer Architecture, Inference Optimization, Paper: Disaggregated Quantization, GPU MODE

Source: https://www.youtube.com/watch?v=RF6lox0YfSU
