# How We Took an AI Agent From 41% to 97% Accurate - Milind Sunil Shah, Brainbase Labs

## Executive summary

Milind Sunil Shah outlines a six-step framework for significantly improving back-office AI agents, using a case study involving tracking competitor pizza prices for Domino's. The process demonstrates that by focusing on structured instructions, skills, and playbooks—rather than solely on model retraining—an agent's accuracy improved from 41% (Version 1) to 97% (Version 11). Key results included halving the cost per source (from $2 to $1) and reducing run time from 129 to 71 minutes. The framework emphasizes defining the win, establishing comprehensive ontologies, rigorous benchmarking, and iterative improvement.

## Key takeaways

- The Six-Step Agent Improvement Framework: The methodology involves: 1. Kickoff (agreeing on the desired outcome, cost, and accuracy); 2. Ontology (defining every term and source of truth to prevent assumptions); 3. Benchmarks and Evals (establishing offline and online testing criteria); 4. Shipping a narrow first agent (focusing on a small, measurable scope); 5. Iteration (scoring failures, fixing instructions, and expanding scope); and 6. Handing off the whole loop (empowering the customer to manage the agent's continuous improvement).
- Measurable Performance Gains (Domino's Case): The agent's performance improved dramatically: Accuracy increased from 41% to 97%; Cost per source decreased from $2 to $1; and Run time fell from 129 to 71 minutes. These gains were achieved primarily through refining instructions, skills, and playbooks.
- Importance of Ontology: To prevent the agent from guessing, every term must be defined. This includes defining what an item is, where the answer comes from, and specific business rules (e.g., marking down the price paid at checkout, not the strikethrough price).

## Technical details

- Data Structure and Memory: The framework utilizes a memory primitive (like a Postgres database) to store structured data, such as 'price observations,' linking observed prices, full prices, and the exact source for verification.
- Benchmarking and Evaluation: Evaluation methods include: Code checks (e.g., verifying if every ID cell was replaced with a price); LLM as a judge (checking if the right item/promo was applied); and Agent as a judge (having the agent take multiple tool calls to confirm the correctness of an output artifact).
- Advanced Tool Use and Fixes: Improvements often require deep analysis of tool call traces. Examples include adding rules to look beyond the initially linked section of a webpage to find deals below it, or instructing the agent to prioritize analyzing the HTML DOM (like image titles) before relying solely on image recognition to reduce cost and improve accuracy.
- Handing Off the Loop: The goal is to hand off an entire operational loop, not just the agent. This requires defining who runs the agent, who reviews exceptions, and how those human corrections are automatically converted back into benchmark cases or rules for the next version.

## Practical implications

- Focus on refining instructions, skills, and playbooks (the 'how') rather than assuming the solution requires a new, larger model (the 'what').
- Start with a narrow scope (Version 1) that is measurable, even if the initial accuracy is low, to establish a baseline score.
- Treat failure analysis as a core part of the development cycle: every failure should be used to create a new benchmark case or rule.
- For build engineers, the concept of 'Ontology' is critical: ensure every data term has a documented definition and source of truth to prevent agent hallucination.

## Topics

AI Agents, Process Automation, Natural Language Processing (NLP), Build Engineering, Data Scraping, Machine Learning Operations (MLOps), AIE CODE summit

Source: https://www.youtube.com/watch?v=4VmugsE-iP0
