Topic
All digests tagged AI Evaluation
Suggested route
Move from earlier context through the available material to the newest signals. This route is a reading aid, not a difficulty ranking.
Learn how to move from subjective 'vibe checks' to systematic AI evaluations (evals). This guide covers defining concrete pass/fail criteria, analyzing agent traces, and building robust eval templates using LLM judges and Decision Models.
Read article
Building rigorous evaluation systems for AI coding agents is complex. The talk details why public benchmarks fail and how to balance measurable vs. representative tasks, emphasizing observability and trace analysis.
Read article
Comparing agentic search vs. vector search for coding agents, this talk reveals that while both methods achieved equal accuracy, vector search cost four times more due to lack of connective context.
Read article
Evaluating high-stakes AI requires moving beyond static rubrics. The solution is a continuous loop: Discover failure modes from real outputs, capture expert judgment, and calibrate every output against accumulated context.
Read article
Abridge addresses the administrative burden in healthcare by building clinical intelligence tools that decompose complex tasks into smaller, specialized models to manage cost and latency at scale.
Read article