Stanford Online

Stanford CME295 Transformers & LLMs | Autumn 2026 | Lecture 5 - LLM Systems

Published 2026-11-05 · Duration 1:47:01

Summary

This lecture provides a deep dive into the architectural challenges of training and serving Large Language Models (LLMs) at scale. The core limitations discussed are GPU memory capacity and computational speed. Solutions involve advanced techniques like FlashAttention (optimizing self-attention to reduce HBM I/O), and various parallelism strategies (Data, Tensor, Pipeline, Context, Expert) to distribute model components across multiple GPUs. For inference, optimizations include KV caching, Paged Attention, Grouped-Query Attention (GQA), and Speculative Decoding to minimize latency and memory usage.

Download summary

Key takeaways

  1. Memory and Compute Bottlenecks 1:24:08

    Training LLMs requires storing not only billions of parameters but also activations, gradients, and optimizer states. The sheer size of these components often exceeds the limited High Bandwidth Memory (HBM) capacity of a single GPU, necessitating advanced parallelism and memory management techniques.

  2. FlashAttention: Optimizing Self-Attention 15:12

    FlashAttention is an exact method that improves self-attention efficiency by avoiding the materialization and storage of the large $N imes N$ attention matrix. It achieves this by iteratively computing the softmax numerator and denominator, significantly reducing reads and writes to the slow HBM and leveraging the fast SRAM.

  3. Advanced Parallelism for Scaling 1:32:10

    To fit massive models, techniques like Zero Redundancy Optimizer (ZeRO) shard parameters, gradients, and optimizer states. Model parallelism methods include Tensor Parallelism (splitting matrix multiplications), Pipeline Parallelism (splitting computation across layers), and Expert Parallelism (routing tokens to specific experts).

  4. Inference Optimizations

    For fast inference, techniques include KV caching (reusing past key/value computations), Paged Attention (managing KV cache memory fragmentation using block-based allocation), Grouped-Query Attention (GQA) (reducing memory by grouping key/value projections), and Speculative Decoding (generating multiple tokens using a fast draft model for verification).

More digests on Large Language Models (LLMs)

Technical details

  • Hardware Fundamentals 120s

    GPUs are excellent for matrix multiplications due to their parallel cores. Key metrics include FLOPs (compute budget) and FLOPs/s (compute speed). The memory hierarchy involves HBM (large, slow, e.g., 141 GB) and SRAM (small, fast, e.g., 100 MB).

  • FlashAttention Mechanism 912s

    The method computes the softmax of $Q K^T$ times $V$ by reformulating the weighted sum into an iterative process for the numerator and denominator, thereby minimizing HBM I/O and enabling the operation to be performed faster.

  • Parallelism Techniques

    Model parallelism techniques include: Data Parallelism (dividing data across devices), Tensor Parallelism (splitting matrix computations across devices), Pipeline Parallelism (splitting computation across layers), Context Parallelism (dividing operations across sequence length), and Expert Parallelism (routing tokens to specific experts). These methods can be combined.

  • Inference Optimization: Paged Attention

    To combat memory fragmentation when caching Key/Value pairs, Paged Attention uses a logical view of memory, storing K/V pairs in fixed blocks (e.g., size B=16) and maintaining a lookup map.

  • Inference Optimization: Speculative Decoding

    This technique accelerates token generation by using a smaller 'draft model' to predict multiple tokens ($K$) and then verifying them using the larger target model. This leverages the fact that verification is computationally cheaper than sequential generation.

Mentioned resources

Channel & topics

Watch on YouTube · Back to latest

This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.