I Let Agents Refactor My Codebase for 3 Weeks. Then I Read the Code. — Keiji Kanazawa, Microsoft
Summary
The speaker details a three-week experience letting coding agents refactor a personal research assistant system. Initial attempts revealed that the 'agent' was merely a deterministic workflow, with architectural concepts like 'skills' and 'actions' being aliases for the same underlying mechanism. After realizing the limitations of trusting the agents for a full refactor, the speaker rebuilt the system using a fresh set of specifications and an explicit agent runtime. The resulting architecture emphasizes clear goal rubrics and iterative evaluation loops, culminating in a video deep dive agent that extracts keyframes and blackboard equations from lectures, demonstrating a robust, generalized workflow.
Key takeaways
-
The L/Z Debate: Ownership of Goals
The speaker, who identifies as a Product Manager, emphasizes that while they delegate implementation to agents, they must 'own the goals and evaluation loop.' This places them firmly on the 'L' (read the code) side of the spectrum, asserting that product managers must maintain oversight of the system's objectives.
-
Agentic Systems vs. Deterministic Workflows
6:40
Initial agent implementations were found to be deterministic workflows. The speaker discovered that system components labeled 'skills' and 'actions' were functionally identical, lacking true agentic independence or proper SDK usage.
-
The Importance of Evaluation Rubrics
The system's success relies on defining explicit goal rubrics. The speaker demonstrated that agents can 'overfit' to a single training video, failing when tested on other content unless the rubric is iteratively refined to ensure generalization.
-
Adversarial Review Loops
Beyond standard evaluation, the speaker implemented adversarial code reviews, having agents check each other's work over several rounds, creating a robust, multi-layered refinement loop.
More digests on Agentic Workflows
Technical details
-
Agent Runtime Architecture
633s
The speaker designed a new agent runtime featuring a registry of different agents, ensuring that every component is treated as an independent, manageable entity, moving beyond simple sequential workflows.
-
Video Deep Dive Agent
A working example involves a video deep dive agent that extracts approximately 440 keyframes from a lecture, uses an AI eval rubric to pull out blackboard equations, and organizes them into an explorable canvas alongside explanations.
-
System Generalization
The video deep dive agent's methodology generalizes beyond blackboard equations to handle other content types, including slide decks, coding demos, and terminal recordings.
-
Agent Framework Comparison
921s
The speaker compared four different agent frameworks (including Codeex, Claude, and Pi), ultimately settling on a combination of Claude and Codeex for personal projects, while noting that Pi is a specialized framework designed for hardcore engineers.
Mentioned resources
- Codeex
- Claude
- Pi
- Copilot CLI
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.