Building the AI Agent Is Easy. Proving It's Safe Isn't - Jonathan Osborne, NatWest
Summary
Jonathan Osborne details the process of moving a major retail bank's digital assistant (Cora) from scripted responses to advanced AI agents. The core challenge is not the agent's creation, but proving its safety and reliability in a regulated banking environment with 19 million customers. The solution involves implementing a multi-layered 'routing funnel' to limit model exposure, moving from single accuracy scores to multi-dimensional quality vectors (Safety, Task, Experience), and establishing robust MLOps practices like shadow deployments, hard regression gates in CI/CD, and continuous monitoring for model and data drift.
Key takeaways
-
Safety is the primary concern (Blast Radius)
3:13
With 19 million customers, any failure in an AI agent is not merely a bug ticket but a significant operational and reputational risk. This necessitates rigorous pre-deployment validation to gain regulatory approval.
-
AI Agents introduce four new failure modes
6:18
Unlike scripted bots, agents can be confidently wrong (hallucination), susceptible to prompt injection, provide unhelpful circular answers, or be nondeterministic (same input, different output).
-
The Routing Funnel limits model exposure
7:32
The system uses a multi-stage funnel: Contextual filtering (removes common events), deterministic pre-classification rules (no ML/AI involved), followed by relevancy/rephrasing, an intent classifier (neural network over embeddings), and final policy rules. This minimizes the attack surface.
-
Quality must be measured as a vector, not a score
10:34
The bank moved away from single accuracy scores, treating quality as a vector across three independent categories: Safety (could it harm the customer?), Task (did it do its job?), and Experience (how did it feel to deal with?).
-
Continuous evaluation is part of production infrastructure
Monitoring uses tools like Comet, capturing traces and scores from both Generative AI (OpenTelemetry/Open Inference) and traditional ML. This includes monitoring for input feature drift and establishing hard regression gates in CI/CD.
Technical details
-
Agent Failure Modes
378s
The four failure modes are: 1) Confidently wrong (hallucination); 2) Prompt injection (adversarial input); 3) Unhelpful (circular answers); and 4) Nondeterministic (inconsistent output).
-
Routing Funnel Components
452s
The funnel includes: Contextual filter (removes common events, ~20%); Deterministic pre-classification rules; Relevancy and rephrasing step; Intent classifier (neural network over embeddings); and Policy rules (for access control, auditability, and governance).
-
Evaluation Metrics
743s
Quality is scored across Safety, Task, and Experience. Human validation showed that while the overall pass rate was high (83%), the score masked significant gaps (e.g., 54-point gap in edge cases).
-
LLM as a Judge Caveats
916s
LLM judges can be overly positive (generous). Correlation between the judge and human validators must be checked, and the judge model itself must be treated as part of the configuration, as changes can introduce breaking changes.
-
Deployment Strategy
Deployment starts with 'shadow mode,' where the model classifies utterances but the existing scripted solution responds, allowing risk-free observation of production data. Rollout proceeds via feature flagging and gradual, percentage-based deployment.
Mentioned resources
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.