# Annotation Queues | Introduction to LangSmith (8/12)

## Executive summary

This session details the critical role of human review in LLM-as-judge evaluation, using LangSmith's annotation queue to correct sentiment misclassifications. It also demonstrates how tracing tool calls can uncover deep state corruption bugs, specifically identifying that the `confirm_order` tool was incorrectly summing only sides, ignoring the pizza subtotal.

## Key takeaways

- The Need for Human Annotation Queues: An annotation queue serves as a human review inbox to validate machine scores, preventing a 'judge that's never been checked against human labels' from being a trustworthy metric (0:00).
- Improving Evaluator Rubrics: LLM judges can fail if the prompt only describes 'frustration' and not the 'outcome.' The solution is to add an outcome section to the rubric, instructing the evaluator to score zero when the problem is unresolved (3:07).
- Debugging Tool Call Failures: A bug was found where the `confirm_order` tool failed to include the pizza subtotal in the delivery minimum check, only summing the sides. This was identified by tracing the tool calls, which showed a subtotal of zero despite the conversation showing $43 (4:29).
- Leveraging LangSmith Chat: LangSmith Chat can read the thread and tool calls to describe the mechanism of a bug, such as 'order state corruption,' providing a high-level diagnosis (5:12).

## Technical details

- Annotation Queue Setup: To annotate, instructions (written standards) and a separate feedback key (e.g., `sentiment human label`) must be added to prevent overwriting the LLM judge's scores (0:50).
- Tool Call Discrepancy: The bug involved the `confirm_order` tool showing an order subtotal of zero, even when the conversation trace showed a subtotal of $43. The `add_side` tool call showed a Caesar salad was $8 (4:29).
- LangSmith Chat Functionality: LangSmith Chat can analyze a thread, including tool calls, and describe the underlying mechanism, such as detecting 'order state corruption' (5:12).

## Practical implications

- Build engineers can use annotation queues to validate LLM-driven quality metrics, ensuring the AI judge aligns with human expectations.
- Tracing tool calls is a powerful debugging method that can expose state corruption or logic errors (e.g., incorrect subtotal calculation) that are not apparent in the conversational text alone.
- LangSmith Chat provides an automated mechanism to summarize complex failure modes, accelerating the debugging cycle.

## Topics

LLM Evaluation, Agent Engineering, Tool Debugging, LangSmith, Sentiment Triage, LangSmith Essentials course, Course repo

Source: https://www.youtube.com/watch?v=LfdGwdUhGW4
