AI Engineer

I Let Agents Refactor My Codebase for 3 Weeks. Then I Read the Code. — Keiji Kanazawa, Microsoft

Published 2026-11-09 · Duration 19:51

Summary

The speaker details a three-week experience letting coding agents refactor a personal research assistant system. Initial attempts revealed that the 'agent' was merely a deterministic workflow, with architectural concepts like 'skills' and 'actions' being aliases for the same underlying mechanism. After realizing the limitations of trusting the agents for a full refactor, the speaker rebuilt the system using a fresh set of specifications and an explicit agent runtime. The resulting architecture emphasizes clear goal rubrics and iterative evaluation loops, culminating in a video deep dive agent that extracts keyframes and blackboard equations from lectures, demonstrating a robust, generalized workflow.

Download summary

Key takeaways

  1. The L/Z Debate: Ownership of Goals

    The speaker, who identifies as a Product Manager, emphasizes that while they delegate implementation to agents, they must 'own the goals and evaluation loop.' This places them firmly on the 'L' (read the code) side of the spectrum, asserting that product managers must maintain oversight of the system's objectives.

  2. Agentic Systems vs. Deterministic Workflows 6:40

    Initial agent implementations were found to be deterministic workflows. The speaker discovered that system components labeled 'skills' and 'actions' were functionally identical, lacking true agentic independence or proper SDK usage.

  3. The Importance of Evaluation Rubrics

    The system's success relies on defining explicit goal rubrics. The speaker demonstrated that agents can 'overfit' to a single training video, failing when tested on other content unless the rubric is iteratively refined to ensure generalization.

  4. Adversarial Review Loops

    Beyond standard evaluation, the speaker implemented adversarial code reviews, having agents check each other's work over several rounds, creating a robust, multi-layered refinement loop.

More digests on Agentic Workflows

Technical details

  • Agent Runtime Architecture 633s

    The speaker designed a new agent runtime featuring a registry of different agents, ensuring that every component is treated as an independent, manageable entity, moving beyond simple sequential workflows.

  • Video Deep Dive Agent

    A working example involves a video deep dive agent that extracts approximately 440 keyframes from a lecture, uses an AI eval rubric to pull out blackboard equations, and organizes them into an explorable canvas alongside explanations.

  • System Generalization

    The video deep dive agent's methodology generalizes beyond blackboard equations to handle other content types, including slide decks, coding demos, and terminal recordings.

  • Agent Framework Comparison 921s

    The speaker compared four different agent frameworks (including Codeex, Claude, and Pi), ultimately settling on a combination of Claude and Codeex for personal projects, while noting that Pi is a specialized framework designed for hardcore engineers.

Mentioned resources

  • Codeex (Coding Agent/LLM)
  • Claude (Coding Agent/LLM)
  • Pi (Agent Framework)
  • Copilot CLI (Development Tool)

Channel & topics

Watch on YouTube · Back to latest

This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.