Disaggregated Quantization: Specializing LLM Prefill and Decode
Summary
This talk introduces Disaggregated Quantization, a method designed to improve the efficiency and accuracy of Large Language Model (LLM) inference by treating the two distinct computational phases—Prefill (input processing) and Decode (token generation)—separately. The core finding is that Prefill and Decode are bottlenecked by different hardware principles: Prefill is compute-bound, while Decode is memory-bound. By specializing quantization formats and potentially using separate weights for each phase, significant accuracy gains and speed improvements can be achieved, even enabling the offloading of the Prefill model to SSD for long context sequences.
Key takeaways
-
Prefill vs. Decode Bottlenecks
7:55
Prefill processes the entire input sequence (e.g., 10,000 tokens) in one forward pass, making it compute-bound. Decode generates tokens one by one, making it memory-bound due to the high cost of moving weights. This difference necessitates specialized quantization strategies.
-
Full Disaggregation for Accuracy
20:50
Full Disaggregation uses separate, native-precision weights for Prefill and Decode. This approach significantly boosts accuracy across both Prefill-heavy and Decode-heavy tasks compared to non-disaggregated methods, while maintaining speed.
-
Prefill Augmentation for Frozen Decoders
30:50
A novel technique allows improving a pre-quantized, frozen Decoder model by training a separate Prefill model (a 'prefiller') to generate better initial KV caches. This boosts quality even when the Decoder weights cannot be updated.
-
Offloading Prefill to SSD
36:40
For very long context sequences (e.g., >8,000 tokens), the Prefill model can be offloaded to SSD. Since the computation cost grows faster than the constant loading cost, the loading process can be fully overlapped with computation, maintaining high performance.
More digests on Large Language Models (LLMs)
Technical details
-
Disaggregated Quantization
720s
This approach specializes quantization formats and weights for the Prefill and Decode phases of LLM inference. It leverages the inherent separation of input (user request) and output (desired generation) present in modern LLMs.
-
Quantization Formats and Binding
950s
Prefill (compute-bound) benefits from formats that accelerate matrix multiplication (e.g., NVFP4, 3-bit storage). Decode (memory-bound) benefits from weight-only formats that minimize memory movement, such as those using vector quantization or complex on-the-fly decoding.
-
Format Disaggregation
1150s
This scheme uses fast computation for Prefill (e.g., 3-bit storage/NVFP4 requantization) while leaving Decode as a weight-only quantization pathway that avoids unnecessary activation quantization, leading to lower quantization error and higher accuracy.
-
Full Disaggregation
1250s
This method uses separate, native-precision weight checkpoints for Prefill and Decode. For example, storing a full NVFP4 checkpoint for Prefill while using a separate 3-bit lookup table for Decode. This improves accuracy by allowing the Prefill weights to be of higher bit width than the Decode weights.
Mentioned resources
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.