Hamel Husain

How to Evaluate Agent Retrieval

Published 2026-11-12 · Duration 50:15

Summary

Glean's search quality team details their rigorous, multi-stage process for evaluating AI retrieval systems, moving from offline evaluation to live A/B testing. They emphasize that due to the scale (billions of documents) and privacy requirements of enterprise data, open-source benchmarks are insufficient. The methodology involves building custom eval sets from historical usage and synthetic data, utilizing advanced metrics like Recall@K and Comparison Ranking (CR@K), and employing 'agentic judges' for complex chat evaluations.

Download summary

Key takeaways

  1. Custom Eval Set Generation 5:05

    Eval sets are built from two sources: 1) Historical usage data (mining search queries and citations from real user interactions) and 2) Synthetic data (generating question-answer pairs from a controlled document corpus).

  2. Multi-Stage Evaluation Pipeline 32:17

    Evaluation progresses from offline testing (Search Evals $\rightarrow$ Context Selection Evals $\rightarrow$ End-to-End Chat Evals) to live A/B testing. Offline evals are preferred because they are deterministic and cheaper, while A/B tests provide real-time user satisfaction signals.

  3. Advanced Metrics for LLM Output 20:32

    Beyond standard search metrics (Recall@K, Mean Average Precision), they use specialized metrics for LLM-driven context selection (Snippet Recall) and chat quality, including Pairwise Completeness Judge and agentic judges for correctness.

  4. Infrastructure and Ownership

    The process relies on dedicated infrastructure (eval orchestrator, logging sinks) and a culture where every engineer launching a feature is responsible for creating or updating the necessary 'golden' eval sets for regression testing.

More digests on AI Evaluation (Evals)

Technical details

  • Eval Set Types 305s

    Search eval sets map a query to a canonical document; Context selection eval sets map a query to a canonical snippet; Chat eval sets are input-only, based on user queries.

  • Search Metrics 1553s

    Key metrics include Recall@K (ensuring canonical documents are in the top K results) and Comparison Ranking (CR@K), which measures the overall change in ranking across the top K positions.

  • Context Selection Metrics 1232s

    Focuses on Snippet Recall: ensuring all canonical snippets are retrieved while minimizing the total token count to manage cost and context window size.

  • Chat Evaluation 1553s

    Uses LLM judges for correctness and completeness. The advanced 'agentic judges' simulate tool loops (e.g., using the search tool) to validate responses, providing high-signal evaluation.

  • Performance Monitoring 1937s

    Critical non-quality metrics include tracking token count, latency (time to first token, end-to-end), and loop count to ensure efficiency and manage operational costs.

Mentioned resources

Channel & topics

Watch on YouTube · Back to latest

This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.