# Disaggregated Quantization: Specializing LLM Prefill and Decode

## Executive summary

This talk introduces Disaggregated Quantization, a method designed to improve the efficiency and accuracy of Large Language Model (LLM) inference by treating the two distinct computational phases—Prefill (input processing) and Decode (token generation)—separately. The core finding is that Prefill and Decode are bottlenecked by different hardware principles: Prefill is compute-bound, while Decode is memory-bound. By specializing quantization formats and potentially using separate weights for each phase, significant accuracy gains and speed improvements can be achieved, even enabling the offloading of the Prefill model to SSD for long context sequences.

## Key takeaways

- Prefill vs. Decode Bottlenecks: Prefill processes the entire input sequence (e.g., 10,000 tokens) in one forward pass, making it compute-bound. Decode generates tokens one by one, making it memory-bound due to the high cost of moving weights. This difference necessitates specialized quantization strategies.
- Full Disaggregation for Accuracy: Full Disaggregation uses separate, native-precision weights for Prefill and Decode. This approach significantly boosts accuracy across both Prefill-heavy and Decode-heavy tasks compared to non-disaggregated methods, while maintaining speed.
- Prefill Augmentation for Frozen Decoders: A novel technique allows improving a pre-quantized, frozen Decoder model by training a separate Prefill model (a 'prefiller') to generate better initial KV caches. This boosts quality even when the Decoder weights cannot be updated.
- Offloading Prefill to SSD: For very long context sequences (e.g., >8,000 tokens), the Prefill model can be offloaded to SSD. Since the computation cost grows faster than the constant loading cost, the loading process can be fully overlapped with computation, maintaining high performance.

## Technical details

- Disaggregated Quantization: This approach specializes quantization formats and weights for the Prefill and Decode phases of LLM inference. It leverages the inherent separation of input (user request) and output (desired generation) present in modern LLMs.
- Quantization Formats and Binding: Prefill (compute-bound) benefits from formats that accelerate matrix multiplication (e.g., NVFP4, 3-bit storage). Decode (memory-bound) benefits from weight-only formats that minimize memory movement, such as those using vector quantization or complex on-the-fly decoding.
- Format Disaggregation: This scheme uses fast computation for Prefill (e.g., 3-bit storage/NVFP4 requantization) while leaving Decode as a weight-only quantization pathway that avoids unnecessary activation quantization, leading to lower quantization error and higher accuracy.
- Full Disaggregation: This method uses separate, native-precision weight checkpoints for Prefill and Decode. For example, storing a full NVFP4 checkpoint for Prefill while using a separate 3-bit lookup table for Decode. This improves accuracy by allowing the Prefill weights to be of higher bit width than the Decode weights.

## Practical implications

- Achieving higher LLM accuracy at low bit-width quantization levels by optimizing the Prefill and Decode phases independently.
- Enabling the use of specialized, non-trainable Prefill models to augment the performance of existing, frozen Decoder models.
- Improving the scalability of LLMs by allowing the Prefill model to be offloaded to slower storage (SSD) for extremely long context sequences, provided the sequence length exceeds the cost intersection point (e.g., >8,000 tokens).
- The techniques are applicable to models up to 2.88 trillion parameters.

## Topics

Large Language Models (LLMs), Quantization, Hardware Acceleration, Transformer Architecture, Memory Bandwidth, Compute-Bound vs. Memory-Bound Systems, Disaggregated Quantization Paper, GPU MODE

Source: https://www.youtube.com/watch?v=UpOR8a6Pqbc
