How Ramp Tests Its AI Bookkeeping Agent Before Customers Do - Ryan Stevens, Ramp
Summary
Ryan Stevens details Ramp's process for building robust end-to-end evaluations (evals) for Stacks, an AI agent designed for bookkeepers. The methodology involves breaking down complex jobs (like reconciliation) into discrete tasks, building synthetic business environments, and running extensive back-tests. Key findings highlight that model failure often stems from reasoning gaps rather than data gathering, and successful optimization requires techniques like skill ablation, testing memories across time periods, and implementing scheduled regression testing.
Key takeaways
-
Deconstruct the Job
1:56
The first step in building an agent benchmark is breaking down the target process (e.g., bookkeeping) into discrete, measurable tasks (e.g., reconciliation, data entry, variance analysis).
-
Prioritize Synthetic Data
7:28
When insufficient user data is available, building synthetic businesses and comprehensive data sets (ERP, vendor lists, etc.) is crucial for effective offline evaluation.
-
Focus on Pass@k for High-Stakes Tasks
15:14
For critical areas like accounting, where the cost of error is high, optimizing for 'Pass@k' (an all-or-nothing metric) is often more valuable than simple accuracy.
-
Implement Regression Testing
Instead of relying on CI gating, establishing an internal platform for scheduled, recurring background evals is vital to detect performance regressions over time.
Technical details
-
Agent Evaluation Metrics
914s
The speaker distinguishes between 'benchmarks,' 'evals,' and 'verifiability,' noting they are often used interchangeably. Performance is measured using 'Accuracy' (partial credit counts) and 'Pass@k' (an all-or-nothing metric, where k is the attempt number).
-
Model Failure Analysis
1118s
Analysis of model failures in accounting suggests that the primary issue is 'reasoning failure' (incorrect reasoning through data) rather than failure to gather necessary data.
-
Optimization Techniques
Effective optimization includes 'Skill Ablation' (systematically removing skills to identify redundancy and improve efficiency) and 'Trajectory Distilled Memories,' which tests memory retention by running the same task over two time periods.
-
Synthetic Environment Setup
448s
A synthetic business environment must include all necessary data components: ERP, customer list, vendor list, and accounts payable data, to allow for rigorous back-testing.
Mentioned resources
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.