How to Evaluate Agent Retrieval
Summary
Glean's search quality team details their rigorous, multi-stage process for evaluating AI retrieval systems, moving from offline evaluation to live A/B testing. They emphasize that due to the scale (billions of documents) and privacy requirements of enterprise data, open-source benchmarks are insufficient. The methodology involves building custom eval sets from historical usage and synthetic data, utilizing advanced metrics like Recall@K and Comparison Ranking (CR@K), and employing 'agentic judges' for complex chat evaluations.
Key takeaways
-
Custom Eval Set Generation
5:05
Eval sets are built from two sources: 1) Historical usage data (mining search queries and citations from real user interactions) and 2) Synthetic data (generating question-answer pairs from a controlled document corpus).
-
Multi-Stage Evaluation Pipeline
32:17
Evaluation progresses from offline testing (Search Evals $\rightarrow$ Context Selection Evals $\rightarrow$ End-to-End Chat Evals) to live A/B testing. Offline evals are preferred because they are deterministic and cheaper, while A/B tests provide real-time user satisfaction signals.
-
Advanced Metrics for LLM Output
20:32
Beyond standard search metrics (Recall@K, Mean Average Precision), they use specialized metrics for LLM-driven context selection (Snippet Recall) and chat quality, including Pairwise Completeness Judge and agentic judges for correctness.
-
Infrastructure and Ownership
The process relies on dedicated infrastructure (eval orchestrator, logging sinks) and a culture where every engineer launching a feature is responsible for creating or updating the necessary 'golden' eval sets for regression testing.
More digests on AI Evaluation (Evals)
Technical details
-
Eval Set Types
305s
Search eval sets map a query to a canonical document; Context selection eval sets map a query to a canonical snippet; Chat eval sets are input-only, based on user queries.
-
Search Metrics
1553s
Key metrics include Recall@K (ensuring canonical documents are in the top K results) and Comparison Ranking (CR@K), which measures the overall change in ranking across the top K positions.
-
Context Selection Metrics
1232s
Focuses on Snippet Recall: ensuring all canonical snippets are retrieved while minimizing the total token count to manage cost and context window size.
-
Chat Evaluation
1553s
Uses LLM judges for correctness and completeness. The advanced 'agentic judges' simulate tool loops (e.g., using the search tool) to validate responses, providing high-signal evaluation.
-
Performance Monitoring
1937s
Critical non-quality metrics include tracking token count, latency (time to first token, end-to-end), and loop count to ensure efficiency and manage operational costs.
Mentioned resources
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.