# I Tracked Every Mistake My AI Race Engineer Made (Agent Lens) | Part 2

## Executive summary

This session details the process of using Agent Lens and ARIA within CoreWeave Forge to analyze the performance and failure patterns of a locally run AI race engineer (based on a Qwen model). The speaker captured granular data—including conversations, turns, and failures like tool misuse and context loss—to generate a dataset for subsequent supervised fine-tuning, demonstrating a robust methodology for iterative AI agent improvement.

## Key takeaways

- Granular Failure Analysis via Agent Lens: Agent Lens captures specific failure patterns, including tool misuse, misrequirements, incorrect results, and context loss, allowing for targeted agent improvement.
- Defining Conversational Units: A 'conversation' is defined as a single game session (not time trials), ensuring cohesive data for comparison. A 'turn' is defined as a single radio call (agent or user).
- Role Separation in AI Agents: The system utilizes two agents: the Race Coach (deterministic code/LLM judge) and the Race Engineer (the primary suggestion engine), separating calculation from data presentation.
- Automated Tagging with ARIA: The research agent ARIA reads traces and insights to automatically create tags (e.g., 'frustrated driver'), enabling quick categorization of user intent and emotional state.

## Technical details

- Agent Architecture: The AI system uses a local Qwen model. The Race Coach is designed as deterministic code with LLM judges to ensure data accuracy, while the Race Engineer handles complex calculations and suggestions.
- Data Capture Mechanisms: Agent Lens captures the full span of interaction: speech-to-text, tools called, and text-to-speech responses, allowing for detailed review of agent actions.
- Model Refinement Pipeline: The captured insights and conversations are intended to be added to a dataset, which can then be used for supervised fine-tuning of the Qwen model.

## Practical implications

- The methodology provides a complete feedback loop for AI development: execution -> logging (Agent Lens) -> analysis (ARIA) -> dataset creation -> fine-tuning.
- Build engineers can leverage granular failure data (e.g., tool misuse, context loss) to debug and improve complex, multi-agent systems.
- The use of deterministic code (Race Coach) alongside LLMs helps isolate and verify data integrity in AI applications.

## Topics

AI Agents, LLM Fine-Tuning, Observability, Data Engineering, Agentic Systems, Agent Lens, ARIA, CoreWeave Forge, Qwen model

Source: https://www.youtube.com/watch?v=2QcPx7rZ4ug
