GPU MODE

Disaggregated Quantization: Specializing LLM Prefill and Decode

Published 2026-11-09 · Duration 48:31

Summary

This talk introduces Disaggregated Quantization, a method designed to improve the efficiency and accuracy of Large Language Model (LLM) inference by treating the two distinct computational phases—Prefill (input processing) and Decode (token generation)—separately. The core finding is that Prefill and Decode are bottlenecked by different hardware principles: Prefill is compute-bound, while Decode is memory-bound. By specializing quantization formats and potentially using separate weights for each phase, significant accuracy gains and speed improvements can be achieved, even enabling the offloading of the Prefill model to SSD for long context sequences.

Download summary

Key takeaways

  1. Prefill vs. Decode Bottlenecks 7:55

    Prefill processes the entire input sequence (e.g., 10,000 tokens) in one forward pass, making it compute-bound. Decode generates tokens one by one, making it memory-bound due to the high cost of moving weights. This difference necessitates specialized quantization strategies.

  2. Full Disaggregation for Accuracy 20:50

    Full Disaggregation uses separate, native-precision weights for Prefill and Decode. This approach significantly boosts accuracy across both Prefill-heavy and Decode-heavy tasks compared to non-disaggregated methods, while maintaining speed.

  3. Prefill Augmentation for Frozen Decoders 30:50

    A novel technique allows improving a pre-quantized, frozen Decoder model by training a separate Prefill model (a 'prefiller') to generate better initial KV caches. This boosts quality even when the Decoder weights cannot be updated.

  4. Offloading Prefill to SSD 36:40

    For very long context sequences (e.g., >8,000 tokens), the Prefill model can be offloaded to SSD. Since the computation cost grows faster than the constant loading cost, the loading process can be fully overlapped with computation, maintaining high performance.

More digests on Large Language Models (LLMs)

Technical details

  • Disaggregated Quantization 720s

    This approach specializes quantization formats and weights for the Prefill and Decode phases of LLM inference. It leverages the inherent separation of input (user request) and output (desired generation) present in modern LLMs.

  • Quantization Formats and Binding 950s

    Prefill (compute-bound) benefits from formats that accelerate matrix multiplication (e.g., NVFP4, 3-bit storage). Decode (memory-bound) benefits from weight-only formats that minimize memory movement, such as those using vector quantization or complex on-the-fly decoding.

  • Format Disaggregation 1150s

    This scheme uses fast computation for Prefill (e.g., 3-bit storage/NVFP4 requantization) while leaving Decode as a weight-only quantization pathway that avoids unnecessary activation quantization, leading to lower quantization error and higher accuracy.

  • Full Disaggregation 1250s

    This method uses separate, native-precision weight checkpoints for Prefill and Decode. For example, storing a full NVFP4 checkpoint for Prefill while using a separate 3-bit lookup table for Decode. This improves accuracy by allowing the Prefill weights to be of higher bit width than the Decode weights.

Mentioned resources

Channel & topics

Watch on YouTube · Back to latest

This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.