# If Your AI Scores 100%, Your Eval Is Wrong - Ramakrishnan Lokanathan, eslams.com

## Executive summary

The speaker argues that current AI evaluations (evals) are often insufficient or 'broken' because they focus too heavily on single scores or final answers. For high-stakes AI applications, evals must be treated as comprehensive examinations that test the entire system workflow, not just the model's output. A robust evaluation framework requires defining the most ambitious success state, integrating automated checks into the CI pipeline, incorporating continuous human review, and utilizing adversarial testing and complex games (like Chess or Go) to measure deep reasoning and agentic action.

## Key takeaways

- Define the Most Ambitious Success: When designing an eval, aim for the best possible outcome (e.g., AGI level) rather than optimizing for a high score. The goal is to define what *must* happen and what *must never* happen, ensuring the test pushes the model to its limits.
- Test the Entire Workflow, Not Just the Output: Evaluation must cover every step of the process. It is insufficient to only check the start point and the end point; the objective must be evaluated at every transition within the workflow.
- Build a Holistic Evaluation System: Relying on a single score is inadequate for modern agentic systems. Evaluation must combine automated checks within the CI pipeline, continuous human review, and adversarial testing to ensure reliability.
- Use Games for Agentic Reasoning: For measuring long-chain reasoning and agentic action, complex board games (like Chess, Go, and Poker) provide a superior evaluation paradigm compared to traditional Q&A tasks.
- Prioritize Regression and Real-World Failures: It is critical to continuously test for regression cases (what used to work) and incorporate real user failures into the evaluation pipeline. The user is the most important evaluator.

## Technical details

- eSlams Framework: The speaker introduces eSlams (electronic slams), an open-source Python PyPi package designed to test AI models by having them play games (e.g., Chess, Go, Poker). It supports approximately 50 games and 90 different providers, allowing evaluation of planning, hidden information decision-making, and cooperation.
- Evaluation Workflow Best Practices: For robust testing, the evaluation must be designed for specific components of the workflow (e.g., tool calling accuracy) rather than treating the entire process as a single unit. The evaluation must measure the success of the element itself.
- CI/CD Integration: Automated checks and evaluation questions must be integrated into the CI pipeline and run every time the agent goes through a production deployment process.

## Practical implications

- Shift the focus of evaluation from achieving a high score (e.g., 100%) to defining the most ambitious, ideal outcome.
- Implement comprehensive testing that tracks inputs, outputs, and expected outcomes at every single step of the operational workflow.
- Establish a feedback loop that systematically captures and integrates real-world user failures and edge cases into the formal test suite.
- Utilize complex, multi-turn games as a specialized evaluation paradigm to test deep reasoning and agentic decision-making.

## Topics

AI Safety, Evaluation Metrics, Agentic Systems, CI/CD, LLMs, Regression Testing, eSlams, GPT-3 API, Gemini

Source: https://www.youtube.com/watch?v=7WUzHIF32vA