Voice AI Where One Wrong Question Breaks the Rules - Shiv Indap, Upstart & Brooke Hopkins, Coval
Summary
This talk details the critical infrastructure required to safely deploy and continuously improve voice AI agents in highly regulated industries, such as lending. The core challenge is ensuring compliance, as a single wrong question can lead to regulatory issues. The solution involves establishing a robust, human-in-the-loop evaluation (Evals) process. This process moves beyond simple manual review (like Google Sheets) by defining structured metrics (e.g., policy alignment, accuracy) and utilizing LLMs as judges. Crucially, the system implements a continuous improvement loop: monitoring live calls, flagging failures (known and unknown unknowns), routing them for human review, re-simulating fixes in a controlled environment, and finally, deploying changes using feature flags, all while maintaining human sign-off.
Key takeaways
-
Regulatory Risk in Voice AI
3:50
In lending, voice AI calls must be compliant. The system must be designed to handle specific conversational constraints (e.g., if a user mentions medical purposes, the agent cannot ask for the specific amount, or it must handle ambiguous fund usage).
-
Scaling Evaluation Infrastructure
7:40
Manual review processes quickly become unmanageable (e.g., moving from a Google Sheet to 15 people adding random columns). A structured process is needed to gather data for both agent performance and evaluation improvement.
-
Improving LLM Judge Agreement
11:43
Initial compliance agreement between human reviewers and the LLM judge was only 33%. By systematically incorporating expert feedback and encoding 'tribal knowledge,' the agreement rate was significantly improved to 92%.
-
The Continuous Improvement Loop
The full cycle involves: 1) Defining metrics (Product Definition); 2) Monitoring live calls for failures; 3) Routing failures for human review; 4) Re-simulating fixes (voice-to-voice evaluation) to verify improvements; and 5) Deploying changes using feature flags, ensuring humans remain in the loop.
Technical details
-
Metrics and Evaluation Types
528s
Metrics include Accuracy, Clarity, Completeness, Policy Alignment, and CSAT score. Evaluation types include LLM as a judge, heuristics, and audio processing. The process requires defining metrics based on human review criteria, rather than starting with technical metrics like Word Error Rate.
-
Failure Detection and Routing
852s
The system monitors live calls to detect both known failure patterns (e.g., failure to verify correct loan pieces) and unknown unknowns using clustering analysis and anomaly detection. Failures are then routed to specific teams (e.g., compliance violations go to the compliance team, off-script answers go to the agent team).
-
Safe Deployment and Re-simulation
To 'shift left' the failure detection, fixes must be re-simulated using voice-to-voice evaluation. This tests the fix against various conditions (background noise, accents, interruptions) before impacting live users. All changes are gated by human review and feature flags.
-
Self-Healing Agents
The final stage involves enabling self-healing loops, where agents can identify fixes using RL pipelines or fine-tuned LLMs, but these automated fixes must still be monitored and signed off by humans (e.g., engineers or compliance officers).
Mentioned resources
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.