# LLMs for Agent Builders: Context Engineering | Stanford CS329Z Lecture 2 with Michael Ryan

## Executive summary

This lecture provides a deep dive into the entire Large Language Model (LLM) stack, covering everything from foundational transformer architecture to advanced agentic deployment strategies. Key focus areas include optimizing inference efficiency (Prefill vs. Decode, KV cache management, Speculative Decoding), mitigating context length limitations (Context Rot, Compaction, Recursive Models), and ensuring reliable system integration through Structured I/O and Constrained Decoding. For build engineers, understanding these underlying design decisions is crucial for selecting the right model and optimizing the agent's operational pipeline.

## Key takeaways

- LLM Stack Hierarchy: The LLM stack is structured into three layers: LLM Engineering (training, architecture, scaling laws), LLM Adaptation (fine-tuning, distillation, LoRA), and Applications (agents, RAG, chatbots). Building agents requires operating at the highest level of this stack.
- Inference Efficiency: Prefill vs. Decode: The process of generating text involves two stages: Prefill (processing the initial prompt, which is compute-bound) and Decode (generating tokens one by one, which is memory-bound). The KV cache stores Key and Value activations to avoid recomputing these values for every token.
- Context Management and Limitations: LLMs suffer from 'Context Rot' and 'Compaction' when context length is excessive. To mitigate this, engineers should use Retrieval-Augmented Generation (RAG) or implement Recursive Language Models, which treat the prompt itself as a variable to be manipulated by the LLM.
- Reliable Agent Output (Structured I/O): To ensure LLMs output predictable data structures (e.g., JSON), use Constrained Decoding or libraries like DSPy. This forces the model to adhere to a defined output format, significantly improving reliability for tool calling and system integration.
- Advanced Training Paradigms: Model improvement follows a pipeline: Pre-training (massive data scraping/filtering, e.g., Common Crawl) $ ightarrow$ Mid-training (domain-specific data mixing) $ ightarrow$ Post-training (Supervised Fine-Tuning (SFT), Reinforcement Learning from Human Feedback (RLHF), or RL with verifiable rewards).

## Technical details

- Transformer Attention Mechanism: The core mechanism involves calculating attention scores using Query (Q), Key (K), and Value (V) vectors. The standard attention layer is $O(N^2)$ in sequence length ($N$), which is computationally expensive for long contexts. Multi-Head Attention is an extension that runs multiple attention mechanisms in parallel.
- Attention Complexity Reduction: Linear Attention mechanisms (e.g., used in RWKV and Mamba) approximate the softmax function to reduce complexity from $O(N^2)$ to $O(N)$ or $O(D^2)$ (where $D$ is the hidden dimension), improving memory footprint and scaling for long contexts, though they may struggle with very long-range dependencies.
- Decoding Strategies: Sampling methods include Greedy (always picking the most likely token), Top-K (sampling from the K most likely tokens), and Top-P (sampling from the cumulative probability mass until P is reached). Temperature ($T$) is a hyperparameter used to scale logits before normalization, controlling the diversity of the output (high $T$ = diverse, low $T$ = predictable).
- Speculative Decoding: An inference optimization technique where a small, fast 'draft model' generates multiple tokens, which are then checked and verified by the main, larger model. This allows the main model to process tokens in batches, improving throughput.
- Loss Function: LLMs are trained using Cross-Entropy Loss, which calculates the negative log probability of the actual ground truth next token given the model's predicted probability distribution over the entire vocabulary.

## Practical implications

- When building agentic systems, prioritize memory-efficient architectures (e.g., linear attention) if dealing with extremely long context inputs to manage computational load.
- Implement structured output constraints (e.g., JSON schema validation) using techniques like Constrained Decoding to ensure reliable tool calling and data parsing.
- For long-horizon tasks, do not rely solely on the context window; use external memory retrieval (RAG) and monitor for context compaction.
- Optimize inference speed by understanding the bottleneck: use speculative decoding when the system is compute-bound (Prefill) or when memory bandwidth is limited (Decode).

## Topics

LLM Architecture, Transformer Attention, LLM Training Pipelines (SFT, RLHF), Inference Optimization, Context Engineering, Agentic Systems and Tool Calling, Stanford CS329Z: Engineering AI Agents, Common Crawl / The Pile, DSPy

Source: https://www.youtube.com/watch?v=Vr1NzH8bT44
