LangChain

Annotation Queues | Introduction to LangSmith (8/12)

Published 2026-11-17 · Duration 6:29

Summary

This session details the critical role of human review in LLM-as-judge evaluation, using LangSmith's annotation queue to correct sentiment misclassifications. It also demonstrates how tracing tool calls can uncover deep state corruption bugs, specifically identifying that the `confirm_order` tool was incorrectly summing only sides, ignoring the pizza subtotal.

Download summary

Key takeaways

  1. The Need for Human Annotation Queues

    An annotation queue serves as a human review inbox to validate machine scores, preventing a 'judge that's never been checked against human labels' from being a trustworthy metric (0:00).

  2. Improving Evaluator Rubrics 5:07

    LLM judges can fail if the prompt only describes 'frustration' and not the 'outcome.' The solution is to add an outcome section to the rubric, instructing the evaluator to score zero when the problem is unresolved (3:07).

  3. Debugging Tool Call Failures

    A bug was found where the `confirm_order` tool failed to include the pizza subtotal in the delivery minimum check, only summing the sides. This was identified by tracing the tool calls, which showed a subtotal of zero despite the conversation showing $43 (4:29).

  4. Leveraging LangSmith Chat

    LangSmith Chat can read the thread and tool calls to describe the mechanism of a bug, such as 'order state corruption,' providing a high-level diagnosis (5:12).

More digests on LLM Evaluation

Technical details

  • Annotation Queue Setup 50s

    To annotate, instructions (written standards) and a separate feedback key (e.g., `sentiment human label`) must be added to prevent overwriting the LLM judge's scores (0:50).

  • Tool Call Discrepancy

    The bug involved the `confirm_order` tool showing an order subtotal of zero, even when the conversation trace showed a subtotal of $43. The `add_side` tool call showed a Caesar salad was $8 (4:29).

  • LangSmith Chat Functionality

    LangSmith Chat can analyze a thread, including tool calls, and describe the underlying mechanism, such as detecting 'order state corruption' (5:12).

Mentioned resources

Channel & topics

Watch on YouTube · Back to latest

This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.