# How to Evaluate Agent Retrieval

## Executive summary

Glean's search quality team details their rigorous, multi-stage process for evaluating AI retrieval systems, moving from offline evaluation to live A/B testing. They emphasize that due to the scale (billions of documents) and privacy requirements of enterprise data, open-source benchmarks are insufficient. The methodology involves building custom eval sets from historical usage and synthetic data, utilizing advanced metrics like Recall@K and Comparison Ranking (CR@K), and employing 'agentic judges' for complex chat evaluations.

## Key takeaways

- Custom Eval Set Generation: Eval sets are built from two sources: 1) Historical usage data (mining search queries and citations from real user interactions) and 2) Synthetic data (generating question-answer pairs from a controlled document corpus).
- Multi-Stage Evaluation Pipeline: Evaluation progresses from offline testing (Search Evals $\rightarrow$ Context Selection Evals $\rightarrow$ End-to-End Chat Evals) to live A/B testing. Offline evals are preferred because they are deterministic and cheaper, while A/B tests provide real-time user satisfaction signals.
- Advanced Metrics for LLM Output: Beyond standard search metrics (Recall@K, Mean Average Precision), they use specialized metrics for LLM-driven context selection (Snippet Recall) and chat quality, including Pairwise Completeness Judge and agentic judges for correctness.
- Infrastructure and Ownership: The process relies on dedicated infrastructure (eval orchestrator, logging sinks) and a culture where every engineer launching a feature is responsible for creating or updating the necessary 'golden' eval sets for regression testing.

## Technical details

- Eval Set Types: Search eval sets map a query to a canonical document; Context selection eval sets map a query to a canonical snippet; Chat eval sets are input-only, based on user queries.
- Search Metrics: Key metrics include Recall@K (ensuring canonical documents are in the top K results) and Comparison Ranking (CR@K), which measures the overall change in ranking across the top K positions.
- Context Selection Metrics: Focuses on Snippet Recall: ensuring all canonical snippets are retrieved while minimizing the total token count to manage cost and context window size.
- Chat Evaluation: Uses LLM judges for correctness and completeness. The advanced 'agentic judges' simulate tool loops (e.g., using the search tool) to validate responses, providing high-signal evaluation.
- Performance Monitoring: Critical non-quality metrics include tracking token count, latency (time to first token, end-to-end), and loop count to ensure efficiency and manage operational costs.

## Practical implications

- When building AI features, always establish a 'golden' eval set for regression testing, even if the feature is complex.
- For LLM-powered systems, focus on optimizing for Recall@K (e.g., Recall@16) rather than just the top-ranked result, as LLMs read through more context.
- Monitor cost and latency metrics (token count, time to first token) alongside quality metrics, as efficiency is crucial for production readiness.
- The most robust evaluation strategy combines offline, deterministic evals with live, real-user A/B testing.

## Topics

AI Evaluation (Evals), Retrieval Augmented Generation (RAG), Enterprise Search, LLM Judging, A/B Testing, Data Signal Mining, AI Evals Cohort, AI Evals Guide

Source: https://www.youtube.com/watch?v=hzWv-apa4k4
