# RAG Explained: From BM25 to Agentic RAG | Stanford CS329Z Lecture 3 with Diyi Yang

## Executive summary

This lecture provides a comprehensive, deep dive into Retrieval-Augmented Generation (RAG), detailing its evolution from classic document retrieval to advanced, agentic systems. The discussion covers the entire RAG pipeline, including indexing strategies (e.g., chunking, contextual retrieval), retrieval methods (BM25, DPR, ColBERT), and advanced search architectures (HNSW, GraphRAG). A key focus is placed on moving beyond static RAG by incorporating agentic components like Self-RAG, CRAG, and Search-R1, which allow the system to dynamically decide whether to retrieve, what query to issue, and when to stop, thereby mitigating compound errors.

## Key takeaways

- RAG's Core Principle: RAG's design principle is to prevent LLMs from relying solely on memorized knowledge by providing relevant, verifiable information 'just in time' via a retrieval mechanism. This makes the process dynamic and interpretable, allowing for citations.
- Hybrid and Advanced Retrieval: Optimal retrieval often requires combining methods. Hybrid search (e.g., combining BM25's exact term matching with DPR's semantic understanding) and contextual retrieval (prepending surrounding context to a chunk) significantly improve performance and robustness.
- Vector Search Efficiency: Due to the high complexity of exact search (O(ND)), Approximate Nearest Neighbor (ANN) algorithms like HNSW (Hierarchical Navigable Small World) are essential for scalable, fast retrieval.
- Agentic RAG for Decision Making: Modern RAG moves toward agentic systems. Instead of a fixed pipeline, agents (e.g., Self-RAG, CRAG, Search-R1) incorporate decision-making capabilities, allowing the model to decide whether to retrieve, critique the evidence, or refine the query, which is crucial for complex, multi-step tasks.

## Technical details

- Document Indexing & Chunking: Chunking strategies range from simple fixed-size segmentation to advanced methods like semantic segmentation or AI-driven dynamic chunking, which use LLMs to detect natural breakpoints. Contextual retrieval enhances units by prepending surrounding context to the segment, improving both embedding and BM25 performance.
- Retrieval Components: Retrievers include: 1) **BM25**: A classic, non-neural, term-frequency approach (no training required). 2) **DPR (Dense Passage Retrieval)**: A neural approach using a query encoder and a passage encoder, trained with contrastive loss to maximize the inner product similarity between relevant pairs. 3) **ColBERT**: A method that improves scoring by keeping a separate vector for each token, calculating a max similarity score across all query tokens and document tokens.
- Advanced Indexing Structures: For complex knowledge, indexing can be hierarchical (PeaKtor, building a tree of summaries from segments) or graph-based (GraphRAG, extracting entities and relationships to build a knowledge graph, allowing retrieval across multiple documents and modalities).
- Encoder Architectures: Retrieval can use **Bi-encoders** (query and passage encoded separately, e.g., DPR), **Cross-encoders** (query and passage processed together for full attention), or **Late Interaction** (encoding independently but calculating interaction/score at the final stage, e.g., ColBERT).

## Practical implications

- When designing a system, evaluate the trade-off between speed (e.g., BM25) and semantic accuracy (e.g., DPR).
- For large-scale systems, use ANN techniques like HNSW to manage the computational cost of vector search.
- To build robust agents, incorporate self-critique or an evaluator component (like in CRAG) to validate retrieved evidence before generation.
- Always consider the availability of data, agentic frameworks, computation, and evaluation metrics when scoping a project.

## Topics

Retrieval-Augmented Generation (RAG), Vector Search (ANN, HNSW), Indexing Strategies (Chunking, Contextual Retrieval), Information Retrieval (BM25, DPR), Graph Neural Networks (GraphRAG), AI Agents (Self-RAG, CRAG, Search-R1), Stanford CS329Z: Engineering AI Agents, Full CS329Z playlist

Source: https://www.youtube.com/watch?v=psLc-46GqY0
