Insights | Introduction to LangSmith (9/12)
Summary
This lesson introduces LangSmith Insights, a feature used to expand the search for unknown issues in agent traffic. Unlike score-based evaluators, Insights groups conversations into emergent categories, allowing teams to discover new areas of customer engagement, such as unsupported requests or ordering patterns, which can then be converted into datasets for further analysis.
Key takeaways
-
Insights vs. Evaluators
Evaluators answer questions that can be quantified (e.g., 'Is this negative sentiment?'), while Insights asks about unknown categories (e.g., 'What are people talking about when they're upset?'). Both pull signal from production traffic.
-
Insights Workflow
0:01
The Insights report process involves defining the scope (e.g., 'how are customers engaging with the agent?') and the agent's function (e.g., 'customer ordering agent for the LangSlice pizzeria').
-
Category Lifecycle
0:03
Any identified category or subcategory can be sent directly to a dataset or an annotation queue, enabling a continuous improvement loop: Insights -> Category -> Dataset -> Evaluator -> Production.
Technical details
-
Insights Report Structure
2s
The resulting report includes an executive summary, patterns (with percentage coverage), and clickable references to the underlying traces.
-
Feature Requirements
1s
Insights requires a minimum of a LangSmith Plus plan to run.
-
Auto Flow Process
1s
The auto flow requires input on what the team wants to learn, what the agent does, and optionally, how the traces are structured.
Mentioned resources
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.