How We Took an AI Agent From 41% to 97% Accurate - Milind Sunil Shah, Brainbase Labs
Summary
Milind Sunil Shah outlines a six-step framework for significantly improving back-office AI agents, using a case study involving tracking competitor pizza prices for Domino's. The process demonstrates that by focusing on structured instructions, skills, and playbooks—rather than solely on model retraining—an agent's accuracy improved from 41% (Version 1) to 97% (Version 11). Key results included halving the cost per source (from $2 to $1) and reducing run time from 129 to 71 minutes. The framework emphasizes defining the win, establishing comprehensive ontologies, rigorous benchmarking, and iterative improvement.
Key takeaways
-
The Six-Step Agent Improvement Framework
0:02
The methodology involves: 1. Kickoff (agreeing on the desired outcome, cost, and accuracy); 2. Ontology (defining every term and source of truth to prevent assumptions); 3. Benchmarks and Evals (establishing offline and online testing criteria); 4. Shipping a narrow first agent (focusing on a small, measurable scope); 5. Iteration (scoring failures, fixing instructions, and expanding scope); and 6. Handing off the whole loop (empowering the customer to manage the agent's continuous improvement).
-
Measurable Performance Gains (Domino's Case)
0:01
The agent's performance improved dramatically: Accuracy increased from 41% to 97%; Cost per source decreased from $2 to $1; and Run time fell from 129 to 71 minutes. These gains were achieved primarily through refining instructions, skills, and playbooks.
-
Importance of Ontology
0:03
To prevent the agent from guessing, every term must be defined. This includes defining what an item is, where the answer comes from, and specific business rules (e.g., marking down the price paid at checkout, not the strikethrough price).
Technical details
-
Data Structure and Memory
5s
The framework utilizes a memory primitive (like a Postgres database) to store structured data, such as 'price observations,' linking observed prices, full prices, and the exact source for verification.
-
Benchmarking and Evaluation
5s
Evaluation methods include: Code checks (e.g., verifying if every ID cell was replaced with a price); LLM as a judge (checking if the right item/promo was applied); and Agent as a judge (having the agent take multiple tool calls to confirm the correctness of an output artifact).
-
Advanced Tool Use and Fixes
9s
Improvements often require deep analysis of tool call traces. Examples include adding rules to look beyond the initially linked section of a webpage to find deals below it, or instructing the agent to prioritize analyzing the HTML DOM (like image titles) before relying solely on image recognition to reduce cost and improve accuracy.
-
Handing Off the Loop
12s
The goal is to hand off an entire operational loop, not just the agent. This requires defining who runs the agent, who reviews exceptions, and how those human corrections are automatically converted back into benchmark cases or rules for the next version.
Mentioned resources
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.