Topic
All digests tagged LLM Evaluation
Suggested route
Move from earlier context through the available material to the newest signals. This route is a reading aid, not a difficulty ranking.
Harbor is an open-source framework for evaluating AI agents on real tasks using isolated sandboxes. It allows engineers to test agent changes by measuring side effects and optimizing metrics like tool call count, moving from 'vibes' to 'evidence.'
Read article
Louis-François Bouchard introduces 'slop' in AI writing and details the creation of Tone-Bench, a benchmark using 10 scripts, 5 drafts, and 4 LLM judges to measure creative writing quality.
Read article
CoreWeave Forge unifies the AI loop (Run, Observe, Curate, Improve, Evaluate) into one connected environment. The keynote detailed how to move AI agents from demo to production, featuring the NVIDIA Vera Rubin NVL72 and advanced evaluation techniques.
Read article
The reading group explores agent policy compliance, noting that LLMs struggle to follow complex SOPs. Solutions involve implementing deterministic and model-based guards outside the LLM to ensure reliable policy adherence.
Read article
The talk details building self-learning loops for AI agents, moving observability from human dashboards to programmatic interfaces. Key tools include Arise Signal and Arise Skills, enabling agents to detect, investigate, and self-correct production failures.
Read article
For complex AI agents, traditional input-output testing fails. The solution is 'simulation': replaying real historical production traces against changes to detect side effects and ensure the agent's behavior remains consistent.
Read article
To own your agent's intelligence, focus on three questions: where it fails, how to reproduce it, and how to fix it. This requires building robust 'worlds' and 'tasks' that mimic production.
Read article
LangSmith Engine v2 automates agent development, offering validated fixes, red teaming, and cross-trace analysis to improve AI agents before production.
Read article
This workshop details how to master AI Observability by building a continuous agent improvement flywheel. Using Braintrust, engineers can derive signal from production traces via custom scoring and Topics, automating the process of identifying failures and proposing code changes.
Read article
Learn how to build better AI evaluations using eight Claude Code skills. Focus on Error Discovery and Eval Audit to move beyond writing evals before analyzing real product failures.
Read article
A deep dive into Claude's new `plugin eval` tool. While promising for automated skill testing, the speakers highlight that the process is currently bottlenecked by manual data collection and lacks an 'in-the-loop' annotation interface.
Read article
Learn how to evaluate AI agents at scale using an LLM-as-a-judge in LangSmith. Automate quality checks for user frustration and manage costs by adjusting the sampling rate.
Read article
Shipping AI safely requires moving beyond A/B testing. The solution involves building an evaluation stack using simulation (Matrix) and automated prompt optimization (Jeppa) to prove safety before patient exposure.
Read article
Evals must evolve with agents: from static checks to LLM judges, the future requires Agent as a Judge for adaptive dynamic analysis.
Read article
Andon Labs discusses Vending-Bench, a framework for long-horizon AI evaluation. They detail moving from simple QA benchmarks to complex simulations that test emergent misbehavior like collusion in real-world environments.
Read article
Learn how to build collective intelligence by turning agent mistakes into reusable, attributable knowledge. The process involves Identity $\rightarrow$ Diary $\rightarrow$ Knowledge Pack.
Read article