Stanford CME295 Transformers & LLMs | Autumn 2026 | Lecture 5 - LLM Systems
Summary
This lecture provides a deep dive into the architectural challenges of training and serving Large Language Models (LLMs) at scale. The core limitations discussed are GPU memory capacity and computational speed. Solutions involve advanced techniques like FlashAttention (optimizing self-attention to reduce HBM I/O), and various parallelism strategies (Data, Tensor, Pipeline, Context, Expert) to distribute model components across multiple GPUs. For inference, optimizations include KV caching, Paged Attention, Grouped-Query Attention (GQA), and Speculative Decoding to minimize latency and memory usage.
Key takeaways
-
Memory and Compute Bottlenecks
1:24:08
Training LLMs requires storing not only billions of parameters but also activations, gradients, and optimizer states. The sheer size of these components often exceeds the limited High Bandwidth Memory (HBM) capacity of a single GPU, necessitating advanced parallelism and memory management techniques.
-
FlashAttention: Optimizing Self-Attention
15:12
FlashAttention is an exact method that improves self-attention efficiency by avoiding the materialization and storage of the large $N imes N$ attention matrix. It achieves this by iteratively computing the softmax numerator and denominator, significantly reducing reads and writes to the slow HBM and leveraging the fast SRAM.
-
Advanced Parallelism for Scaling
1:32:10
To fit massive models, techniques like Zero Redundancy Optimizer (ZeRO) shard parameters, gradients, and optimizer states. Model parallelism methods include Tensor Parallelism (splitting matrix multiplications), Pipeline Parallelism (splitting computation across layers), and Expert Parallelism (routing tokens to specific experts).
-
Inference Optimizations
For fast inference, techniques include KV caching (reusing past key/value computations), Paged Attention (managing KV cache memory fragmentation using block-based allocation), Grouped-Query Attention (GQA) (reducing memory by grouping key/value projections), and Speculative Decoding (generating multiple tokens using a fast draft model for verification).
More digests on Large Language Models (LLMs)
Technical details
-
Hardware Fundamentals
120s
GPUs are excellent for matrix multiplications due to their parallel cores. Key metrics include FLOPs (compute budget) and FLOPs/s (compute speed). The memory hierarchy involves HBM (large, slow, e.g., 141 GB) and SRAM (small, fast, e.g., 100 MB).
-
FlashAttention Mechanism
912s
The method computes the softmax of $Q K^T$ times $V$ by reformulating the weighted sum into an iterative process for the numerator and denominator, thereby minimizing HBM I/O and enabling the operation to be performed faster.
-
Parallelism Techniques
Model parallelism techniques include: Data Parallelism (dividing data across devices), Tensor Parallelism (splitting matrix computations across devices), Pipeline Parallelism (splitting computation across layers), Context Parallelism (dividing operations across sequence length), and Expert Parallelism (routing tokens to specific experts). These methods can be combined.
-
Inference Optimization: Paged Attention
To combat memory fragmentation when caching Key/Value pairs, Paged Attention uses a logical view of memory, storing K/V pairs in fixed blocks (e.g., size B=16) and maintaining a lookup map.
-
Inference Optimization: Speculative Decoding
This technique accelerates token generation by using a smaller 'draft model' to predict multiple tokens ($K$) and then verifying them using the larger target model. This leverages the fact that verification is computationally cheaper than sequential generation.
Mentioned resources
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.