# Don't Approve the AI Model. Approve the App - Doneyli De Jesus, Langfuse

## Executive summary

This talk details a framework for deploying AI agents in highly regulated environments, such as top-five banks. The core principle established is that institutions certify the *application*, not the underlying model. The methodology involves establishing strict deployment gates—requiring on-premises, gated, and air-gapped setups—and proving the application's readiness using a 'golden data set.' Evaluation is multi-dimensional, scoring traces across metrics like groundness, numerical accuracy, and regulatory compliance, allowing teams to improve performance (e.g., raising numerical accuracy from 60% to 95%) by integrating deterministic tools and human expert feedback, thereby reducing evaluation time from weeks to minutes.

## Key takeaways

- Certify Applications, Not Models: In regulated industries, the focus shifts from general model benchmarks (like Opus 5 leading benchmarks) to proving that the specific application built for internal use cases is reliable and safe for production deployment. (3:01)
- Mandatory Deployment Constraints: Financial institutions require solutions to be deployed on-premises, gated (preventing data leakage/training on external providers), and air-gapped to maintain data control and compliance. (3:18)
- The Evaluation Framework: The process requires building a 'golden data set' (test cases) and operationalizing testing to generate traces. Scores are calculated across multiple dimensions (e.g., accuracy, regulatory compliance) and must pass domain-expert-defined thresholds to clear the production gate. (6:00, 13:48)
- Improving Performance with Tools: A model can be highly grounded (e.g., 97% grounded) but fail on specific tasks like math. By providing deterministic tools (e.g., a Python code tool), the application's overall accuracy can be significantly improved (e.g., from 60% to 95%). (7:20, 11:32)

## Technical details

- Deployment Architecture: Solutions must adhere to three main requirements: staying on-premises (repurposing data centers for GPU infrastructure), being gated (preventing data exposure to model providers), and requiring air-gapped deployment. (3:18)
- Evaluation Methodology: The evaluation framework uses a 'golden data set' (key-value pairs with inputs and expected outputs) to generate traces. Scoring is multi-pronged, combining: 1) Deterministic scores (code-based, non-LLM), 2) Path evaluation (checking the agent's logical route), and 3) LLM judge scores (for qualitative aspects). (11:32)
- Gate Definition and Iteration: Gate thresholds are set by domain experts (e.g., risk personnel) and must be defined per use case, as a single average score is insufficient. Failures (failure modes) are collected in an 'annotation queue' for human review and feedback, which then feeds back into the golden data set for iterative improvement. (13:48, 15:53)

## Practical implications

- For build engineers working on AI applications, the focus must shift from optimizing model performance to building robust, auditable, and gate-controlled application workflows.
- Implementing a rigorous testing pipeline that incorporates deterministic, path-based, and LLM-judged metrics is critical for compliance and reliability.
- The architecture must support on-premises, air-gapped deployment to meet the data sensitivity requirements of regulated industries.

## Topics

AI Agents, Generative AI, Compliance, MLOps, Financial Services, Testing Frameworks, Langfuse, AIE CODE summit

Source: https://www.youtube.com/watch?v=-uBABM2AuFg