Topic
All digests tagged Observability
Suggested route
Move from earlier context through the available material to the newest signals. This route is a reading aid, not a difficulty ranking.
Harness engineering is the infrastructure around LLMs, encompassing tools, context, guardrails, and telemetry. While LLMs introduce non-determinism, core software principles like CI/CD, security, and observability remain essential for building robust agents.
Read article
Code Mode is an alternative architecture for AI agents that uses a programming sandbox instead of a large tool catalog. This approach drastically reduces context bloat, lowers token usage, and enables emergent behavior by allowing agents to compose tools in code.
Read article
Fuad Ali details the self-improvement loop for AI agents: trace failures, use Signal to detect issues, create living regression datasets, and run experiments to prove fixes. This automates reliability.
Read article
Harbor is an open-source framework for evaluating AI agents on real tasks using isolated sandboxes. It allows engineers to test agent changes by measuring side effects and optimizing metrics like tool call count, moving from 'vibes' to 'evidence.'
Read article
Learn how to move from subjective 'vibe checks' to systematic AI evaluations (evals). This guide covers defining concrete pass/fail criteria, analyzing agent traces, and building robust eval templates using LLM judges and Decision Models.
Read article
The AI Engineer role is shifting from writing features to building self-improving loops. This requires building always-on agents that consume telemetry, run in durable sandboxes, and are continuously tested using advanced evaluation systems.
Read article
To ensure AI agent reliability, the speaker details a five-step regression loop. This process uses production traces as the source of truth, employing LLMs as judges to quantify subjective metrics and integrating automated gates into CI/CD.
Read article
Most agent evaluations are easy to misread. To test AI agents like a scientist, you must design realistic tasks, run repeated tests, and focus on full traces and observability, because production is the final truth.
Read article
Learn how to build reliable AI agents for high-stakes environments. We cover the AI Engineering Loop: tracing, dataset curation, and integrating automated evaluation (Evals) into your CI/CD pipeline to prevent regressions.
Read article
Transitioning AI agents to production requires a governed framework. Learn how to implement centralized MCP Gateways, LangGraph workflows, and robust CI/CD pipelines for secure, scalable, and observable enterprise AI.
Read article
Learn how to 'actually tokenmaxx' LLMs. This guide details a systematic workflow using DSPy and shadow evaluation to cut prompt bloat by 51% while maintaining quality and reducing compute costs.
Read article
The talk details building self-learning loops for AI agents, moving observability from human dashboards to programmatic interfaces. Key tools include Arise Signal and Arise Skills, enabling agents to detect, investigate, and self-correct production failures.
Read article
Building agent quality platforms is a complex systems challenge. Learn why simple spreadsheets fail and how to build the continuous improvement loop using advanced tracing and observability.
Read article
Marc Klingen details the self-improving OSS agent stack, showing how AI can automate the loop between online tracing, offline datasets, and evaluations, moving teams from manual to automated agent refinement.
Read article
CoreWeave Forge demonstrates the full AI loop: Observe, Curate, Improve, and Evaluate. Engineers can systematically improve AI agents using Agent Lens, Serverless RL, and model lineage tracking.
Read article
The talk details the shift from manual trace review to automated, continuous improvement loops for AI agents. By introducing 'Signal,' the system detects patterns in evaluation failures and automatically suggests fixes, closing the software development loop.
Read article
Comparing agentic search vs. vector search for coding agents, this talk reveals that while both methods achieved equal accuracy, vector search cost four times more due to lack of connective context.
Read article
Scaling development with AI requires more than just generating code. Learn how automated code review, Jev classifiers, and robust CI/CD guardrails are essential to manage the flood of AI-generated PRs and maintain system reliability.
Read article
Running AI agents in production requires shifting focus from root cause analysis to context. Key insights cover using combined data sources (Elastic logs + ServiceNow) for diagnosis, optimizing GPU usage during tool calls, and building AI-powered tracing frameworks.
Read article
To run agents as users, we leverage Kubernetes namespaces per person in every region. A sidecar mounts the identity into the pod, allowing automated jobs to operate with full user access.
Read article