Stanford CME295 Transformers & LLMs | Autumn 2026 | Lecture 6 - AI Agents
Summary
This lecture details the evolution of Large Language Models (LLMs) from simple text predictors to sophisticated AI agents. The discussion addresses core LLM limitations—namely, knowledge cut-off dates and the inability to perform real-world actions. Solutions presented include Retrieval-Augmented Generation (RAG) for grounding knowledge, and Tool Calling for enabling actions. The concept of 'Harness Engineering' is introduced, defining the entire system surrounding the LLM (including memory, tool selection, and reasoning protocols like ReAct). Advanced topics cover context management (compaction, memory), standardized protocols (MCP), and multi-agent systems (agent swarms).
Key takeaways
-
LLM Limitations and the Need for Augmentation
2:00
LLMs are constrained by their training data knowledge cut-off date and cannot perform actions in the real world. This necessitates external systems like RAG and Tool Calling to overcome these limitations.
-
RAG Architecture for Knowledge Grounding
5:50
RAG (Retrieval-Augmented Generation) mitigates knowledge cut-off by retrieving relevant documents from a knowledge base. This process involves chunking documents, creating embedding vectors, and using a two-step retrieval process: candidate retrieval (bi-encoder, maximizing recall) and re-ranking (using a shared encoder to capture nuance).
-
Tool Calling and Agentic Systems
13:38
Tools enable LLMs to perform actions (e.g., querying GPS, running code). This is a three-stage process: the LLM predicts the tool call, the backend executes the action, and the result is fed back to the LLM for a final response.
-
Harness Engineering and ReAct
20:20
The 'Harness' refers to the entire system surrounding the LLM (tools, memory, prompting). The ReAct (Reason + Act) paradigm formalizes the agent loop: Observe $ ightarrow$ Plan $ ightarrow$ Act $ ightarrow$ Observe, allowing the agent to iteratively reach a goal.
Technical details
-
RAG Implementation Details
283s
The process requires: 1) Chunking documents and representing chunks with embedding vectors. 2) Candidate retrieval (using a bi-encoder setup for efficiency) to find potential relevant chunks. 3) Re-ranking (using a single shared encoder) to capture nuanced relationships between the query and the document, improving accuracy.
-
Evaluation Metrics for Retrieval
443s
Evaluation of retrieval systems uses metrics like Normalized Discounted Cumulative Gain (NDCG) at K, Reciprocal Rank, Precision, and Recall. These metrics quantify how well the top K retrieved documents match the actual relevant documents.
-
Tool Calling Mechanics
1081s
Tools are implemented via functions (often in Python) with clear APIs, documentation, and structured input/output types (e.g., enums). The input and output must be interpretable for the LLM to process the result.
-
Agent Protocols and Standardization
1400s
The Model Context Protocol (MCP) is a standardization effort to simplify tool implementation across different LLM providers. It standardizes tools, prompts, and resources into defined buckets, allowing for interoperability.
-
Context Management
1550s
To handle context window limits, compaction strategies include: 1) Deleting unnecessary information. 2) Time-based methods (e.g., sliding window). 3) Summarization, structuring the conversation into predefined buckets (e.g., current progress, remaining work).
Mentioned resources
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.