How AI Agents Use Tools: Function Calling & MCP | Stanford CS329Z Lecture 4 with Diyi Yang
Summary
This lecture provides a deep dive into the architecture of AI agents, detailing why Large Language Models (LLMs) require external tools and standardized protocols to function reliably. Key topics include addressing the limitations of Retrieval-Augmented Generation (RAG), exploring advanced tool-use frameworks (like Toolformer and ToolLLM), and comparing three generations of tool interfaces: REST APIs, structured Function Calling, and the Model Context Protocol (MCP). The discussion emphasizes the shift from training models to use tools to standardizing the tool interface itself for scalability and robustness.
Key takeaways
-
RAG Limitations and Failure Modes
RAG systems suffer from 'Lost in the Middle' (position bias), where models perform poorly if the correct information is not at the beginning or end of a long context window. Furthermore, citation generation can hallucinate, with studies showing that citation precision can be significantly lower than perceived utility.
-
The Necessity of Tools for LLMs
1:00
LLMs struggle with tasks requiring external state, complex multi-step logic, or precise computation (e.g., complex multiplication). Tools allow agents to delegate these tasks to external, specialized systems (e.g., calculators, web search, databases).
-
Standardization: From Function Calling to MCP
43:20
The evolution of tool interfaces moves from human-readable REST APIs to structured JSON schemas (Function Calling), culminating in the Model Context Protocol (MCP). MCP is presented as a standardized, protocol-level abstraction layer designed for the entire AI ecosystem, allowing multiple clients to interact with a single server without the host needing to understand the underlying data format.
-
Advanced Tool Learning and Evaluation
20:00
Modern frameworks like ToolLLM and Gorilla use synthetic data generation (e.g., crawling APIs and using ChatGPT to simulate user queries) to train models on thousands of tools. Evaluation must move beyond simple accuracy to include metrics like Abstract Syntax Tree (AST) matching to quantify hallucination and using Pass@K for reliable multi-turn interactions.
Technical details
-
Lost in the Middle Phenomenon
300s
When provided with many documents, LLMs tend to prioritize information at the beginning or end of the context window, degrading performance if the correct answer is located in the middle of the sequence.
-
Toolformer Architecture
1500s
An early framework that trains LLMs to decide *when* and *where* to insert an API call within the token stream. Training involves optimizing a loss function (weighted decay cross-entropy) that compares the loss of generating text versus the loss of generating a tool call and its execution output.
-
Model Context Protocol (MCP)
2900s
A proposed standardized protocol designed to sit between AI applications (clients) and various data sources (servers). It provides a one-layer abstraction, allowing clients to discover available tools via a `tools/list` call and execute them via `tools/call`, regardless of the underlying data format (e.g., file system, database).
-
ToolBench and ToolLLM
2000s
These frameworks address the scalability issue of limited tools. They use synthetic data construction, leveraging large, capable models (like ChatGPT) to generate complex, multi-step reasoning traces involving thousands of APIs, which are then used to fine-tune smaller, more efficient models.
Mentioned resources
- Toolformer
- ToolLLM
- Gorilla
- Model Context Protocol (MCP)
- Berkeley Function Call Leaderboard
- ToolBench
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.