LLMs for Agent Builders: Context Engineering | Stanford CS329Z Lecture 2 with Michael Ryan
Summary
This lecture provides a deep dive into the entire Large Language Model (LLM) stack, covering everything from foundational transformer architecture to advanced agentic deployment strategies. Key focus areas include optimizing inference efficiency (Prefill vs. Decode, KV cache management, Speculative Decoding), mitigating context length limitations (Context Rot, Compaction, Recursive Models), and ensuring reliable system integration through Structured I/O and Constrained Decoding. For build engineers, understanding these underlying design decisions is crucial for selecting the right model and optimizing the agent's operational pipeline.
Key takeaways
-
LLM Stack Hierarchy
The LLM stack is structured into three layers: LLM Engineering (training, architecture, scaling laws), LLM Adaptation (fine-tuning, distillation, LoRA), and Applications (agents, RAG, chatbots). Building agents requires operating at the highest level of this stack.
-
Inference Efficiency: Prefill vs. Decode
The process of generating text involves two stages: Prefill (processing the initial prompt, which is compute-bound) and Decode (generating tokens one by one, which is memory-bound). The KV cache stores Key and Value activations to avoid recomputing these values for every token.
-
Context Management and Limitations
LLMs suffer from 'Context Rot' and 'Compaction' when context length is excessive. To mitigate this, engineers should use Retrieval-Augmented Generation (RAG) or implement Recursive Language Models, which treat the prompt itself as a variable to be manipulated by the LLM.
-
Reliable Agent Output (Structured I/O)
To ensure LLMs output predictable data structures (e.g., JSON), use Constrained Decoding or libraries like DSPy. This forces the model to adhere to a defined output format, significantly improving reliability for tool calling and system integration.
-
Advanced Training Paradigms
1:00:01
Model improvement follows a pipeline: Pre-training (massive data scraping/filtering, e.g., Common Crawl) $ ightarrow$ Mid-training (domain-specific data mixing) $ ightarrow$ Post-training (Supervised Fine-Tuning (SFT), Reinforcement Learning from Human Feedback (RLHF), or RL with verifiable rewards).
More digests on LLM Architecture
Technical details
-
Transformer Attention Mechanism
1603s
The core mechanism involves calculating attention scores using Query (Q), Key (K), and Value (V) vectors. The standard attention layer is $O(N^2)$ in sequence length ($N$), which is computationally expensive for long contexts. Multi-Head Attention is an extension that runs multiple attention mechanisms in parallel.
-
Attention Complexity Reduction
2827s
Linear Attention mechanisms (e.g., used in RWKV and Mamba) approximate the softmax function to reduce complexity from $O(N^2)$ to $O(N)$ or $O(D^2)$ (where $D$ is the hidden dimension), improving memory footprint and scaling for long contexts, though they may struggle with very long-range dependencies.
-
Decoding Strategies
746s
Sampling methods include Greedy (always picking the most likely token), Top-K (sampling from the K most likely tokens), and Top-P (sampling from the cumulative probability mass until P is reached). Temperature ($T$) is a hyperparameter used to scale logits before normalization, controlling the diversity of the output (high $T$ = diverse, low $T$ = predictable).
-
Speculative Decoding
An inference optimization technique where a small, fast 'draft model' generates multiple tokens, which are then checked and verified by the main, larger model. This allows the main model to process tokens in batches, improving throughput.
-
Loss Function
3900s
LLMs are trained using Cross-Entropy Loss, which calculates the negative log probability of the actual ground truth next token given the model's predicted probability distribution over the entire vocabulary.
Mentioned resources
- Stanford CS329Z: Engineering AI Agents
- Common Crawl / The Pile
- DSPy
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.