# How Ramp Tests Its AI Bookkeeping Agent Before Customers Do - Ryan Stevens, Ramp

## Executive summary

Ryan Stevens details Ramp's process for building robust end-to-end evaluations (evals) for Stacks, an AI agent designed for bookkeepers. The methodology involves breaking down complex jobs (like reconciliation) into discrete tasks, building synthetic business environments, and running extensive back-tests. Key findings highlight that model failure often stems from reasoning gaps rather than data gathering, and successful optimization requires techniques like skill ablation, testing memories across time periods, and implementing scheduled regression testing.

## Key takeaways

- Deconstruct the Job: The first step in building an agent benchmark is breaking down the target process (e.g., bookkeeping) into discrete, measurable tasks (e.g., reconciliation, data entry, variance analysis).
- Prioritize Synthetic Data: When insufficient user data is available, building synthetic businesses and comprehensive data sets (ERP, vendor lists, etc.) is crucial for effective offline evaluation.
- Focus on Pass@k for High-Stakes Tasks: For critical areas like accounting, where the cost of error is high, optimizing for 'Pass@k' (an all-or-nothing metric) is often more valuable than simple accuracy.
- Implement Regression Testing: Instead of relying on CI gating, establishing an internal platform for scheduled, recurring background evals is vital to detect performance regressions over time.

## Technical details

- Agent Evaluation Metrics: The speaker distinguishes between 'benchmarks,' 'evals,' and 'verifiability,' noting they are often used interchangeably. Performance is measured using 'Accuracy' (partial credit counts) and 'Pass@k' (an all-or-nothing metric, where k is the attempt number).
- Model Failure Analysis: Analysis of model failures in accounting suggests that the primary issue is 'reasoning failure' (incorrect reasoning through data) rather than failure to gather necessary data.
- Optimization Techniques: Effective optimization includes 'Skill Ablation' (systematically removing skills to identify redundancy and improve efficiency) and 'Trajectory Distilled Memories,' which tests memory retention by running the same task over two time periods.
- Synthetic Environment Setup: A synthetic business environment must include all necessary data components: ERP, customer list, vendor list, and accounts payable data, to allow for rigorous back-testing.

## Practical implications

- When building agents for complex, non-optimized processes, allocate significant time (4-6 months) to building the evaluation framework.
- Focus on improving the agent's reasoning capabilities, as this is the primary bottleneck in specialized domains like accounting.
- Systematically test agent components (skills/tools) using ablation techniques to ensure efficiency and prevent bloat.
- Establish automated, scheduled regression testing pipelines to monitor agent performance drift over time.

## Topics

AI Agents, LLM Evaluation, Bookkeeping, Synthetic Data, Machine Learning Engineering, Ramp, AIE CODE summit

Source: https://www.youtube.com/watch?v=WJTQBENv0KI
