# Stanford CME295 Transformers & LLMs | Autumn 2026 | Lecture 7 - LLM Evaluation

## Executive summary

This lecture provides a comprehensive guide to evaluating Large Language Models (LLMs) and AI agents, focusing on the transition from human-centric evaluation to scalable, automated methods. Key topics include quantifying output quality using metrics like Pass@k and LLM-as-a-Judge (LLAJ), analyzing the failure modes of the Observe-Act-Plan agent loop, and understanding the complexities of modern benchmarking (e.g., Benchmaxxing and Pareto frontiers). The central theme is the necessity of an 'eval-driven' mindset in AI development.

## Key takeaways

- LLM Evaluation Challenges: LLM performance is difficult to quantify because tasks are vast (code, summarization, reasoning) and outputs are often non-deterministic, meaning there is no single correct reference answer.
- LLM-as-a-Judge (LLAJ) Mechanics: LLAJ leverages the LLM's understanding capabilities to act as a judge, requiring the input of the prompt, the target criteria, and the candidate response. This process can be optimized via 'prompt hillclimbing' to achieve a target metric (e.g., recall).
- Agent Failure Modes: Agent failures can occur at three stages: 1) Tool prediction failure (e.g., hallucinating a tool), 2) Backend computation failure (e.g., returning an error instead of a value), and 3) Final response generation failure (e.g., failing to surface relevant JSON data).
- Benchmarking Risks: Benchmarks, while useful, are prone to 'Benchmaxxing'—where models optimize for proxy metrics (like line count) rather than actual user capability, leading to a mismatch between score and perceived quality. This is governed by 'Goodhart's Law.'

## Technical details

- Evaluation Metrics: Metrics include Cohen's kappa (which compares observed agreement to expected random agreement) for subjective criteria, Pass@k (the probability that at least one of k attempts succeeds), and traditional NLP metrics like BLEU, METEOR, and ROUGE.
- LLAJ Biases: LLAJ models exhibit biases, including position bias (preferring a certain position) and verbosity bias (preferring more verbose answers), which must be mitigated by careful prompt design.
- Agent Tooling and Context: Tool prediction failures can stem from poor tool routing, hallucinating non-existent APIs, or the API description lacking sufficient detail. Solutions include refining API names and using structured outputs.
- Optimization and Guardrails: The Pareto frontier defines the optimal boundary between quality and cost. To ensure benchmark integrity, techniques like BenchGuard are used to check for expected input formatting and problem solvability.

## Practical implications

- Adopt an 'eval-driven' methodology: build the evaluation system (the 'eval') around the agent, rather than just building the agent itself.
- Use structured outputs (e.g., JSON) and constrained decoding to guarantee the LLM adheres to required formats, mitigating non-determinism.
- When debugging agents, systematically check all three stages (Tool Prediction, Backend Computation, Final Response Generation) to pinpoint the failure mode.
- Calibrate LLAJ models using human ratings to ensure the model's judgment aligns with human expectations, especially for subjective criteria.

## Topics

LLM Evaluation, AI Agents, Benchmarking, Prompt Engineering, Machine Learning Operations (MLOps), Stanford CME295 Transformers & LLMs, CME295 Syllabus

Source: https://www.youtube.com/watch?v=uoSynlTDyMs
