# Stanford CME295 Transformers & LLMs | Autumn 2026 | Lecture 5 - LLM Systems

## Executive summary

This lecture provides a deep dive into the architectural challenges of training and serving Large Language Models (LLMs) at scale. The core limitations discussed are GPU memory capacity and computational speed. Solutions involve advanced techniques like FlashAttention (optimizing self-attention to reduce HBM I/O), and various parallelism strategies (Data, Tensor, Pipeline, Context, Expert) to distribute model components across multiple GPUs. For inference, optimizations include KV caching, Paged Attention, Grouped-Query Attention (GQA), and Speculative Decoding to minimize latency and memory usage.

## Key takeaways

- Memory and Compute Bottlenecks: Training LLMs requires storing not only billions of parameters but also activations, gradients, and optimizer states. The sheer size of these components often exceeds the limited High Bandwidth Memory (HBM) capacity of a single GPU, necessitating advanced parallelism and memory management techniques.
- FlashAttention: Optimizing Self-Attention: FlashAttention is an exact method that improves self-attention efficiency by avoiding the materialization and storage of the large $N imes N$ attention matrix. It achieves this by iteratively computing the softmax numerator and denominator, significantly reducing reads and writes to the slow HBM and leveraging the fast SRAM.
- Advanced Parallelism for Scaling: To fit massive models, techniques like Zero Redundancy Optimizer (ZeRO) shard parameters, gradients, and optimizer states. Model parallelism methods include Tensor Parallelism (splitting matrix multiplications), Pipeline Parallelism (splitting computation across layers), and Expert Parallelism (routing tokens to specific experts).
- Inference Optimizations: For fast inference, techniques include KV caching (reusing past key/value computations), Paged Attention (managing KV cache memory fragmentation using block-based allocation), Grouped-Query Attention (GQA) (reducing memory by grouping key/value projections), and Speculative Decoding (generating multiple tokens using a fast draft model for verification).

## Technical details

- Hardware Fundamentals: GPUs are excellent for matrix multiplications due to their parallel cores. Key metrics include FLOPs (compute budget) and FLOPs/s (compute speed). The memory hierarchy involves HBM (large, slow, e.g., 141 GB) and SRAM (small, fast, e.g., 100 MB).
- FlashAttention Mechanism: The method computes the softmax of $Q K^T$ times $V$ by reformulating the weighted sum into an iterative process for the numerator and denominator, thereby minimizing HBM I/O and enabling the operation to be performed faster.
- Parallelism Techniques: Model parallelism techniques include: Data Parallelism (dividing data across devices), Tensor Parallelism (splitting matrix computations across devices), Pipeline Parallelism (splitting computation across layers), Context Parallelism (dividing operations across sequence length), and Expert Parallelism (routing tokens to specific experts). These methods can be combined.
- Inference Optimization: Paged Attention: To combat memory fragmentation when caching Key/Value pairs, Paged Attention uses a logical view of memory, storing K/V pairs in fixed blocks (e.g., size B=16) and maintaining a lookup map.
- Inference Optimization: Speculative Decoding: This technique accelerates token generation by using a smaller 'draft model' to predict multiple tokens ($K$) and then verifying them using the larger target model. This leverages the fact that verification is computationally cheaper than sequential generation.

## Practical implications

- System architects must balance memory constraints (HBM capacity) against computational throughput (FLOPs/s) when designing LLM serving infrastructure.
- Optimizing for inference latency requires implementing techniques like KV caching and Paged Attention to manage memory efficiently and minimize time-to-first-token (TTFT) and time-per-output-token (TPOT).
- The choice of parallelism (e.g., ZeRO, TP, PP) dictates the required inter-device communication bandwidth and overall system cost.
- The 'Roofline Model' helps predict whether a system will be memory-bound or compute-bound, guiding hardware selection and optimization efforts.

## Topics

Large Language Models (LLMs), Transformer Architecture, GPU Optimization, Distributed Computing, Parallelism, Inference Optimization, Stanford CME295 Transformers & LLMs

Source: https://www.youtube.com/watch?v=25KIyaeI_Fg
