How To Decide Who Owns AI Evals
AI Evals should not be a separate role; they must be an inner loop process. Domain expertise is key, and establishing a 'benevolent dictator' to define 'good' is crucial for efficient evaluation.
Read articleTopic
AI Evaluation (Evals)
All digests tagged AI Evaluation (Evals)
Suggested route
Topic journey
Move from earlier context through the available material to the newest signals. This route is a reading aid, not a difficulty ranking.
Start here
Why The Prompt Matters Less Than The Context The speaker argues that for LLMs, context matters more than the specific wording of the prompt. Overly elaborate 'mannered' prompts are often imprecise, suggesting that simple, literal context is the most powerful guide.Build context
Evals in AI: A Deep Dive — Tejas Kumar, IBM Evals are 'fuzzy unit tests' for non-deterministic AI systems. They provide ahead-of-time reliability, complementing runtime harnesses. Learn how to build an Eval suite, mitigate judge biases, and integrate it into a CI gate for production-grade safety. I Ranked Every AI Eval Concept From Worst to Best Stop using generic metrics and off-the-shelf LLM judges. The key to AI evaluation is always looking at your data first, using custom annotation tools, and validating judges with human labels. Is Opus 5.5 Better At Writing? An in-depth comparison of Opus 5 vs. Opus 5.5 reveals that while the newer model improves writing flow, both still struggle with 'slop' and require heavy human editing for technical content.Recent signals
How To Decide Who Owns AI Evals AI Evals should not be a separate role; they must be an inner loop process. Domain expertise is key, and establishing a 'benevolent dictator' to define 'good' is crucial for efficient evaluation. Can Claude Build Your Evals? A deep dive into Anthropic's Claude skill for automating AI eval design. While promising, the speakers argue the automated process is flawed, advocating for a 'data-first' approach over automated eval creation. 7 Most Common AI Eval Mistakes (From Teaching 5K+ Engineers & PMs) Learn the 7 most common mistakes in AI evaluation (Evals). From using generic metrics to failing to validate LLM judges, this guide shows how to build Evals you can actually trust.
Can Claude Build Your Evals?
A deep dive into Anthropic's Claude skill for automating AI eval design. While promising, the speakers argue the automated process is flawed, advocating for a 'data-first' approach over automated eval creation.
Read article
7 Most Common AI Eval Mistakes (From Teaching 5K+ Engineers & PMs)
Learn the 7 most common mistakes in AI evaluation (Evals). From using generic metrics to failing to validate LLM judges, this guide shows how to build Evals you can actually trust.
Read article
Is Opus 5.5 Better At Writing?
An in-depth comparison of Opus 5 vs. Opus 5.5 reveals that while the newer model improves writing flow, both still struggle with 'slop' and require heavy human editing for technical content.
Read article
I Ranked Every AI Eval Concept From Worst to Best
Stop using generic metrics and off-the-shelf LLM judges. The key to AI evaluation is always looking at your data first, using custom annotation tools, and validating judges with human labels.
Read article
Evals in AI: A Deep Dive — Tejas Kumar, IBM
Evals are 'fuzzy unit tests' for non-deterministic AI systems. They provide ahead-of-time reliability, complementing runtime harnesses. Learn how to build an Eval suite, mitigate judge biases, and integrate it into a CI gate for production-grade safety.
Read article
Why The Prompt Matters Less Than The Context
The speaker argues that for LLMs, context matters more than the specific wording of the prompt. Overly elaborate 'mannered' prompts are often imprecise, suggesting that simple, literal context is the most powerful guide.
Read article