I Tracked Every Mistake My AI Race Engineer Made (Agent Lens) | Part 2
Summary
This session details the process of using Agent Lens and ARIA within CoreWeave Forge to analyze the performance and failure patterns of a locally run AI race engineer (based on a Qwen model). The speaker captured granular data—including conversations, turns, and failures like tool misuse and context loss—to generate a dataset for subsequent supervised fine-tuning, demonstrating a robust methodology for iterative AI agent improvement.
Key takeaways
-
Granular Failure Analysis via Agent Lens
2:08
Agent Lens captures specific failure patterns, including tool misuse, misrequirements, incorrect results, and context loss, allowing for targeted agent improvement.
-
Defining Conversational Units
0:45
A 'conversation' is defined as a single game session (not time trials), ensuring cohesive data for comparison. A 'turn' is defined as a single radio call (agent or user).
-
Role Separation in AI Agents
3:05
The system utilizes two agents: the Race Coach (deterministic code/LLM judge) and the Race Engineer (the primary suggestion engine), separating calculation from data presentation.
-
Automated Tagging with ARIA
4:30
The research agent ARIA reads traces and insights to automatically create tags (e.g., 'frustrated driver'), enabling quick categorization of user intent and emotional state.
Technical details
-
Agent Architecture
185s
The AI system uses a local Qwen model. The Race Coach is designed as deterministic code with LLM judges to ensure data accuracy, while the Race Engineer handles complex calculations and suggestions.
-
Data Capture Mechanisms
95s
Agent Lens captures the full span of interaction: speech-to-text, tools called, and text-to-speech responses, allowing for detailed review of agent actions.
-
Model Refinement Pipeline
300s
The captured insights and conversations are intended to be added to a dataset, which can then be used for supervised fine-tuning of the Qwen model.
Mentioned resources
- Agent Lens
- ARIA
- CoreWeave Forge
- Qwen model
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.