How to Measure OCR Quality
Evaluating OCR quality is challenging because benchmarks are often biased or measure formatting, not true quality. The solution is to build custom, unit-test style evals tailored to your specific use case.
Read articleChannel
Hamel Husain
Digests from Hamel Husain — Hamel Husain videos on AI engineering, evaluation, and practical machine learning workflows.
How to Evaluate Coding Agents
Building rigorous evaluation systems for AI coding agents is complex. The talk details why public benchmarks fail and how to balance measurable vs. representative tasks, emphasizing observability and trace analysis.
Read article
How To Get Your Team To Invest In Evals
To convince your team to invest in AI Evals, don't hard sell. Instead, perform structured error analysis on user traces, quantify failure modes, and consistently demonstrate value by showing how you find and fix issues.
Read article
How To Decide Who Owns AI Evals
AI Evals should not be a separate role; they must be an inner loop process. Domain expertise is key, and establishing a 'benevolent dictator' to define 'good' is crucial for efficient evaluation.
Read article
Can One Model Run Your App And Evals?
Can you use the same LLM for both your agent and your judge? The answer is usually yes. Focus on achieving strong alignment with human labels, as this is the most critical factor for successful AI product evaluation.
Read article
How to Ace AI Eval Interview Questions (for Engineers & PMs)
Mastering AI eval interviews requires a structured approach: focus on data skepticism, resourcefulness, and analyzing trade-offs (cost, latency, performance).
Read article
Can Claude Build Your Evals?
A deep dive into Anthropic's Claude skill for automating AI eval design. While promising, the speakers argue the automated process is flawed, advocating for a 'data-first' approach over automated eval creation.
Read article
7 Most Common AI Eval Mistakes (From Teaching 5K+ Engineers & PMs)
Learn the 7 most common mistakes in AI evaluation (Evals). From using generic metrics to failing to validate LLM judges, this guide shows how to build Evals you can actually trust.
Read article
How To Test AI Agents With Simulations
For complex AI agents, traditional input-output testing fails. The solution is 'simulation': replaying real historical production traces against changes to detect side effects and ensure the agent's behavior remains consistent.
Read article
Is Opus 5.5 Better At Writing?
An in-depth comparison of Opus 5 vs. Opus 5.5 reveals that while the newer model improves writing flow, both still struggle with 'slop' and require heavy human editing for technical content.
Read article
I Ranked Every AI Eval Concept From Worst to Best
Stop using generic metrics and off-the-shelf LLM judges. The key to AI evaluation is always looking at your data first, using custom annotation tools, and validating judges with human labels.
Read article
How to Evaluate OCR for AI Agents
Deep dive into evaluating OCR and AI agents on complex documents. Learn how to perform root cause analysis when an agent fails, tracing the error back to upstream data issues like OCR inaccuracies or missing watermarks.
Read article
How To Make PDFs Your AI Can Actually Read
Standard PDFs often fail to provide structural metadata needed for screen readers or LLMs. Learn how Data Lab's API solves this by transforming complex documents into highly structured, machine-readable formats.
Read article
Giving AI a Credit Card Is Still Painful
Watch how an AI agent attempts to purchase a course using a Mercury agent card. The demo highlights the security benefits of scoped permissions but reveals significant friction points in the payment process.
Read article
8 Claude Code skills I use for building better AI evals
Learn how to build better AI evaluations using eight Claude Code skills. Focus on Error Discovery and Eval Audit to move beyond writing evals before analyzing real product failures.
Read article
Why The Prompt Matters Less Than The Context
The speaker argues that for LLMs, context matters more than the specific wording of the prompt. Overly elaborate 'mannered' prompts are often imprecise, suggesting that simple, literal context is the most powerful guide.
Read article
How to Apply Data Science Skills to AI Engineering
In the age of AI, data science skills—like handling noisy, non-deterministic signals—are more valuable than ever. Human judgment is critical for measuring AI output and designing robust RAG systems.
Read article
How to Design a Data Agent People Can Verify
Building trustworthy AI agents requires more than just an answer. The product must be designed to show its work, providing full data provenance, SQL queries, and intermediate calculations for expert verification.
Read article
Putting Claudes "AI Slop" Solution to the Test
The speaker critiques AI 'slop' and argues that accurate model evaluation requires building small, realistic datasets and testing against specific failure modes, rather than relying on leaderboards or vendor demos.
Read article
Trying the new Claude Eval tool
A deep dive into Claude's new `plugin eval` tool. While promising for automated skill testing, the speakers highlight that the process is currently bottlenecked by manual data collection and lacks an 'in-the-loop' annotation interface.
Read article