He Used 100 Billion Tokens in a Month - Waymo Talks
Summary
This discussion explores the evolution of AI coding agents, focusing on the challenges of achieving convergence and quality control. Speakers detail custom harnesses like Evener and the use of tools like RoboRev to manage the 'endless code review problem.' Key insights cover the shift from per-token billing to concurrent request models, the rise of local open-weights models, and the implementation of AI colleagues (PMs, developers) that manage complex, long-horizon tasks and conduct postmortems.
Key takeaways
-
The Convergence Problem in AI Code Review
As LLMs become better at finding bugs (especially edge cases), the process often fails to converge, leading to endless refinement loops. Solutions involve decomposition (breaking down large tasks) or flagging low-severity issues for later review [35:57].
-
The Rise of AI Colleagues and Systemic Roles
The future involves moving beyond single-agent tasks to multi-agent systems where AI colleagues (PM, Dev, Ops) collaborate, manage tasks, and run blameless postmortems, treating work as 'jobs' and 'roles' rather than just 'tasks' [37:05].
-
Local Models Challenge Cloud Costs
33:33
Open-weights models (e.g., Qwen 38B) running on consumer hardware (e.g., 64GB MacBook) can achieve intelligence levels comparable to older frontier models, challenging the per-token cost structure and suggesting a shift toward concurrent request billing [20:13].
-
Structured Agent Communication (Evener)
Custom harnesses like Evener are designed to be provider-agnostic, dispatching tasks to subagents using structured task lists that include effort levels and automated task progression, rather than relying solely on prompts [26:06].
Technical details
-
Agent Tooling and Process Control
RoboRev is an agent code review tool that utilizes post-commit git hooks and includes skills for 'review' or 'refine.' It is integrated into a harness alongside issue trackers like Katada and Message Vault [31:07].
-
AI Verification and Quality Assurance
A pattern for verification involves 'proving it works with a movie,' where the agent is tasked with generating a screen recording of the successful execution cycle. This process is used to validate functionality beyond simple unit tests [34:10].
-
Harness Design Philosophy
Evener is described as a 'light harness' that is highly opinionated, contrasting with the 'plugin-style' approach. It supports multiple models (e.g., Claude, Codex) and allows dispatching to subagents across different providers [39:09].
-
Context Management and Compaction
To manage context window size, agents are given a 'compaction tool' that prompts the agent to summarize its history and 'write a letter of all the important stuff you want to remember' when the context is over 60% full [31:07].
Mentioned resources
- Evener
- Superpowers
- RoboRev
- Katada
- Deepseek 41 Flash
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.