# Voice AI Where One Wrong Question Breaks the Rules - Shiv Indap, Upstart & Brooke Hopkins, Coval

## Executive summary

This talk details the critical infrastructure required to safely deploy and continuously improve voice AI agents in highly regulated industries, such as lending. The core challenge is ensuring compliance, as a single wrong question can lead to regulatory issues. The solution involves establishing a robust, human-in-the-loop evaluation (Evals) process. This process moves beyond simple manual review (like Google Sheets) by defining structured metrics (e.g., policy alignment, accuracy) and utilizing LLMs as judges. Crucially, the system implements a continuous improvement loop: monitoring live calls, flagging failures (known and unknown unknowns), routing them for human review, re-simulating fixes in a controlled environment, and finally, deploying changes using feature flags, all while maintaining human sign-off.

## Key takeaways

- Regulatory Risk in Voice AI: In lending, voice AI calls must be compliant. The system must be designed to handle specific conversational constraints (e.g., if a user mentions medical purposes, the agent cannot ask for the specific amount, or it must handle ambiguous fund usage).
- Scaling Evaluation Infrastructure: Manual review processes quickly become unmanageable (e.g., moving from a Google Sheet to 15 people adding random columns). A structured process is needed to gather data for both agent performance and evaluation improvement.
- Improving LLM Judge Agreement: Initial compliance agreement between human reviewers and the LLM judge was only 33%. By systematically incorporating expert feedback and encoding 'tribal knowledge,' the agreement rate was significantly improved to 92%.
- The Continuous Improvement Loop: The full cycle involves: 1) Defining metrics (Product Definition); 2) Monitoring live calls for failures; 3) Routing failures for human review; 4) Re-simulating fixes (voice-to-voice evaluation) to verify improvements; and 5) Deploying changes using feature flags, ensuring humans remain in the loop.

## Technical details

- Metrics and Evaluation Types: Metrics include Accuracy, Clarity, Completeness, Policy Alignment, and CSAT score. Evaluation types include LLM as a judge, heuristics, and audio processing. The process requires defining metrics based on human review criteria, rather than starting with technical metrics like Word Error Rate.
- Failure Detection and Routing: The system monitors live calls to detect both known failure patterns (e.g., failure to verify correct loan pieces) and unknown unknowns using clustering analysis and anomaly detection. Failures are then routed to specific teams (e.g., compliance violations go to the compliance team, off-script answers go to the agent team).
- Safe Deployment and Re-simulation: To 'shift left' the failure detection, fixes must be re-simulated using voice-to-voice evaluation. This tests the fix against various conditions (background noise, accents, interruptions) before impacting live users. All changes are gated by human review and feature flags.
- Self-Healing Agents: The final stage involves enabling self-healing loops, where agents can identify fixes using RL pipelines or fine-tuned LLMs, but these automated fixes must still be monitored and signed off by humans (e.g., engineers or compliance officers).

## Practical implications

- Build robust evaluation infrastructure (Evals) as a core product feature, not an afterthought.
- Implement a structured feedback loop that captures human expert judgment to improve LLM judges and metrics.
- Design failure detection to identify both known failure modes and unknown unknowns (anomaly detection).
- Prioritize 'shifting left' by using re-simulation and controlled testing environments before deploying fixes to production.

## Topics

Voice AI, Compliance Engineering, LLM Evaluation, Continuous Integration (CI), DevOps, Fintech, Shiv Indap (Upstart), Brooke Hopkins (Coval), AIE CODE summit

Source: https://www.youtube.com/watch?v=3osCbSufv8g