Stanford Online

Stanford CME295 Transformers & LLMs | Autumn 2026 | Lecture 6 - AI Agents

Published 2026-11-13 · Duration 1:40:44

Summary

This lecture details the evolution of Large Language Models (LLMs) from simple text predictors to sophisticated AI agents. The discussion addresses core LLM limitations—namely, knowledge cut-off dates and the inability to perform real-world actions. Solutions presented include Retrieval-Augmented Generation (RAG) for grounding knowledge, and Tool Calling for enabling actions. The concept of 'Harness Engineering' is introduced, defining the entire system surrounding the LLM (including memory, tool selection, and reasoning protocols like ReAct). Advanced topics cover context management (compaction, memory), standardized protocols (MCP), and multi-agent systems (agent swarms).

Download summary

Key takeaways

  1. LLM Limitations and the Need for Augmentation 2:00

    LLMs are constrained by their training data knowledge cut-off date and cannot perform actions in the real world. This necessitates external systems like RAG and Tool Calling to overcome these limitations.

  2. RAG Architecture for Knowledge Grounding 5:50

    RAG (Retrieval-Augmented Generation) mitigates knowledge cut-off by retrieving relevant documents from a knowledge base. This process involves chunking documents, creating embedding vectors, and using a two-step retrieval process: candidate retrieval (bi-encoder, maximizing recall) and re-ranking (using a shared encoder to capture nuance).

  3. Tool Calling and Agentic Systems 13:38

    Tools enable LLMs to perform actions (e.g., querying GPS, running code). This is a three-stage process: the LLM predicts the tool call, the backend executes the action, and the result is fed back to the LLM for a final response.

  4. Harness Engineering and ReAct 20:20

    The 'Harness' refers to the entire system surrounding the LLM (tools, memory, prompting). The ReAct (Reason + Act) paradigm formalizes the agent loop: Observe $ ightarrow$ Plan $ ightarrow$ Act $ ightarrow$ Observe, allowing the agent to iteratively reach a goal.

More digests on AI Agents

Technical details

  • RAG Implementation Details 283s

    The process requires: 1) Chunking documents and representing chunks with embedding vectors. 2) Candidate retrieval (using a bi-encoder setup for efficiency) to find potential relevant chunks. 3) Re-ranking (using a single shared encoder) to capture nuanced relationships between the query and the document, improving accuracy.

  • Evaluation Metrics for Retrieval 443s

    Evaluation of retrieval systems uses metrics like Normalized Discounted Cumulative Gain (NDCG) at K, Reciprocal Rank, Precision, and Recall. These metrics quantify how well the top K retrieved documents match the actual relevant documents.

  • Tool Calling Mechanics 1081s

    Tools are implemented via functions (often in Python) with clear APIs, documentation, and structured input/output types (e.g., enums). The input and output must be interpretable for the LLM to process the result.

  • Agent Protocols and Standardization 1400s

    The Model Context Protocol (MCP) is a standardization effort to simplify tool implementation across different LLM providers. It standardizes tools, prompts, and resources into defined buckets, allowing for interoperability.

  • Context Management 1550s

    To handle context window limits, compaction strategies include: 1) Deleting unnecessary information. 2) Time-based methods (e.g., sliding window). 3) Summarization, structuring the conversation into predefined buckets (e.g., current progress, remaining work).

Mentioned resources

Channel & topics

Watch on YouTube · Back to latest

This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.