# He Used 100 Billion Tokens in a Month - Waymo Talks

## Executive summary

This discussion explores the evolution of AI coding agents, focusing on the challenges of achieving convergence and quality control. Speakers detail custom harnesses like Evener and the use of tools like RoboRev to manage the 'endless code review problem.' Key insights cover the shift from per-token billing to concurrent request models, the rise of local open-weights models, and the implementation of AI colleagues (PMs, developers) that manage complex, long-horizon tasks and conduct postmortems.

## Key takeaways

- The Convergence Problem in AI Code Review: As LLMs become better at finding bugs (especially edge cases), the process often fails to converge, leading to endless refinement loops. Solutions involve decomposition (breaking down large tasks) or flagging low-severity issues for later review [35:57].
- The Rise of AI Colleagues and Systemic Roles: The future involves moving beyond single-agent tasks to multi-agent systems where AI colleagues (PM, Dev, Ops) collaborate, manage tasks, and run blameless postmortems, treating work as 'jobs' and 'roles' rather than just 'tasks' [37:05].
- Local Models Challenge Cloud Costs: Open-weights models (e.g., Qwen 38B) running on consumer hardware (e.g., 64GB MacBook) can achieve intelligence levels comparable to older frontier models, challenging the per-token cost structure and suggesting a shift toward concurrent request billing [20:13].
- Structured Agent Communication (Evener): Custom harnesses like Evener are designed to be provider-agnostic, dispatching tasks to subagents using structured task lists that include effort levels and automated task progression, rather than relying solely on prompts [26:06].

## Technical details

- Agent Tooling and Process Control: RoboRev is an agent code review tool that utilizes post-commit git hooks and includes skills for 'review' or 'refine.' It is integrated into a harness alongside issue trackers like Katada and Message Vault [31:07].
- AI Verification and Quality Assurance: A pattern for verification involves 'proving it works with a movie,' where the agent is tasked with generating a screen recording of the successful execution cycle. This process is used to validate functionality beyond simple unit tests [34:10].
- Harness Design Philosophy: Evener is described as a 'light harness' that is highly opinionated, contrasting with the 'plugin-style' approach. It supports multiple models (e.g., Claude, Codex) and allows dispatching to subagents across different providers [39:09].
- Context Management and Compaction: To manage context window size, agents are given a 'compaction tool' that prompts the agent to summarize its history and 'write a letter of all the important stuff you want to remember' when the context is over 60% full [31:07].

## Practical implications

- Build engineers must evolve their quality gates beyond simple unit testing to include process-level checks (e.g., code review, postmortems) to handle the complexity of AI-generated code.
- The concept of 'AI colleagues' suggests a shift in team structure, where roles (PM, Dev, Ops) are assigned to agents, requiring robust, multi-agent orchestration.
- Harness design should prioritize provider agnosticism and structured task management over proprietary plugin ecosystems to ensure resilience and scalability.

## Topics

AI Agents, LLM Engineering, DevOps, Software Quality, Harness Design, Evener, Superpowers, RoboRev, Katada, Deepseek 41 Flash

Source: https://www.youtube.com/watch?v=NTLBGHsdD7c
