How to Measure OCR Quality
Evaluating OCR quality is challenging because benchmarks are often biased or measure formatting, not true quality. The solution is to build custom, unit-test style evals tailored to your specific use case.
Read articleTopic
AI Evals October 2026 cohort
All digests tagged AI Evals October 2026 cohort
Suggested route
Topic journey
Move from earlier context through the available material to the newest signals. This route is a reading aid, not a difficulty ranking.
Start here
How to Design a Data Agent People Can Verify Building trustworthy AI agents requires more than just an answer. The product must be designed to show its work, providing full data provenance, SQL queries, and intermediate calculations for expert verification.Build context
How to Apply Data Science Skills to AI Engineering In the age of AI, data science skills—like handling noisy, non-deterministic signals—are more valuable than ever. Human judgment is critical for measuring AI output and designing robust RAG systems. Why The Prompt Matters Less Than The Context The speaker argues that for LLMs, context matters more than the specific wording of the prompt. Overly elaborate 'mannered' prompts are often imprecise, suggesting that simple, literal context is the most powerful guide. Giving AI a Credit Card Is Still Painful Watch how an AI agent attempts to purchase a course using a Mercury agent card. The demo highlights the security benefits of scoped permissions but reveals significant friction points in the payment process.Recent signals
How to Measure OCR Quality Evaluating OCR quality is challenging because benchmarks are often biased or measure formatting, not true quality. The solution is to build custom, unit-test style evals tailored to your specific use case. How To Decide Who Owns AI Evals AI Evals should not be a separate role; they must be an inner loop process. Domain expertise is key, and establishing a 'benevolent dictator' to define 'good' is crucial for efficient evaluation. Can Claude Build Your Evals? A deep dive into Anthropic's Claude skill for automating AI eval design. While promising, the speakers argue the automated process is flawed, advocating for a 'data-first' approach over automated eval creation.
How To Decide Who Owns AI Evals
AI Evals should not be a separate role; they must be an inner loop process. Domain expertise is key, and establishing a 'benevolent dictator' to define 'good' is crucial for efficient evaluation.
Read article
Can Claude Build Your Evals?
A deep dive into Anthropic's Claude skill for automating AI eval design. While promising, the speakers argue the automated process is flawed, advocating for a 'data-first' approach over automated eval creation.
Read article
How to Evaluate OCR for AI Agents
Deep dive into evaluating OCR and AI agents on complex documents. Learn how to perform root cause analysis when an agent fails, tracing the error back to upstream data issues like OCR inaccuracies or missing watermarks.
Read article
Giving AI a Credit Card Is Still Painful
Watch how an AI agent attempts to purchase a course using a Mercury agent card. The demo highlights the security benefits of scoped permissions but reveals significant friction points in the payment process.
Read article
Why The Prompt Matters Less Than The Context
The speaker argues that for LLMs, context matters more than the specific wording of the prompt. Overly elaborate 'mannered' prompts are often imprecise, suggesting that simple, literal context is the most powerful guide.
Read article
How to Apply Data Science Skills to AI Engineering
In the age of AI, data science skills—like handling noisy, non-deterministic signals—are more valuable than ever. Human judgment is critical for measuring AI output and designing robust RAG systems.
Read article
How to Design a Data Agent People Can Verify
Building trustworthy AI agents requires more than just an answer. The product must be designed to show its work, providing full data provenance, SQL queries, and intermediate calculations for expert verification.
Read article