AI Engineer

How Ramp Tests Its AI Bookkeeping Agent Before Customers Do - Ryan Stevens, Ramp

Published 2026-11-08 · Duration 18:39

Summary

Ryan Stevens details Ramp's process for building robust end-to-end evaluations (evals) for Stacks, an AI agent designed for bookkeepers. The methodology involves breaking down complex jobs (like reconciliation) into discrete tasks, building synthetic business environments, and running extensive back-tests. Key findings highlight that model failure often stems from reasoning gaps rather than data gathering, and successful optimization requires techniques like skill ablation, testing memories across time periods, and implementing scheduled regression testing.

Download summary

Key takeaways

  1. Deconstruct the Job 1:56

    The first step in building an agent benchmark is breaking down the target process (e.g., bookkeeping) into discrete, measurable tasks (e.g., reconciliation, data entry, variance analysis).

  2. Prioritize Synthetic Data 7:28

    When insufficient user data is available, building synthetic businesses and comprehensive data sets (ERP, vendor lists, etc.) is crucial for effective offline evaluation.

  3. Focus on Pass@k for High-Stakes Tasks 15:14

    For critical areas like accounting, where the cost of error is high, optimizing for 'Pass@k' (an all-or-nothing metric) is often more valuable than simple accuracy.

  4. Implement Regression Testing

    Instead of relying on CI gating, establishing an internal platform for scheduled, recurring background evals is vital to detect performance regressions over time.

More digests on AI Agents

Technical details

  • Agent Evaluation Metrics 914s

    The speaker distinguishes between 'benchmarks,' 'evals,' and 'verifiability,' noting they are often used interchangeably. Performance is measured using 'Accuracy' (partial credit counts) and 'Pass@k' (an all-or-nothing metric, where k is the attempt number).

  • Model Failure Analysis 1118s

    Analysis of model failures in accounting suggests that the primary issue is 'reasoning failure' (incorrect reasoning through data) rather than failure to gather necessary data.

  • Optimization Techniques

    Effective optimization includes 'Skill Ablation' (systematically removing skills to identify redundancy and improve efficiency) and 'Trajectory Distilled Memories,' which tests memory retention by running the same task over two time periods.

  • Synthetic Environment Setup 448s

    A synthetic business environment must include all necessary data components: ERP, customer list, vendor list, and accounts payable data, to allow for rigorous back-testing.

Mentioned resources

Channel & topics

Watch on YouTube · Back to latest

This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.