Annotation Queues | Introduction to LangSmith (8/12)
Summary
This session details the critical role of human review in LLM-as-judge evaluation, using LangSmith's annotation queue to correct sentiment misclassifications. It also demonstrates how tracing tool calls can uncover deep state corruption bugs, specifically identifying that the `confirm_order` tool was incorrectly summing only sides, ignoring the pizza subtotal.
Key takeaways
-
The Need for Human Annotation Queues
An annotation queue serves as a human review inbox to validate machine scores, preventing a 'judge that's never been checked against human labels' from being a trustworthy metric (0:00).
-
Improving Evaluator Rubrics
5:07
LLM judges can fail if the prompt only describes 'frustration' and not the 'outcome.' The solution is to add an outcome section to the rubric, instructing the evaluator to score zero when the problem is unresolved (3:07).
-
Debugging Tool Call Failures
A bug was found where the `confirm_order` tool failed to include the pizza subtotal in the delivery minimum check, only summing the sides. This was identified by tracing the tool calls, which showed a subtotal of zero despite the conversation showing $43 (4:29).
-
Leveraging LangSmith Chat
LangSmith Chat can read the thread and tool calls to describe the mechanism of a bug, such as 'order state corruption,' providing a high-level diagnosis (5:12).
More digests on LLM Evaluation
Technical details
-
Annotation Queue Setup
50s
To annotate, instructions (written standards) and a separate feedback key (e.g., `sentiment human label`) must be added to prevent overwriting the LLM judge's scores (0:50).
-
Tool Call Discrepancy
The bug involved the `confirm_order` tool showing an order subtotal of zero, even when the conversation trace showed a subtotal of $43. The `add_side` tool call showed a Caesar salad was $8 (4:29).
-
LangSmith Chat Functionality
LangSmith Chat can analyze a thread, including tool calls, and describe the underlying mechanism, such as detecting 'order state corruption' (5:12).
Mentioned resources
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.