AI Engineer

Engineer Your Coding Agents Like Any Other System — Zach Lloyd, Warp

Published 2026-11-20 · Duration 18:31

Summary

The shift from running coding agents locally to implementing a 'Software Factory' is presented as a necessary evolution for scalable, governed AI development. A Software Factory is defined as a code-defined, closed-loop system designed to continuously measure and improve cloud coding agents. This system moves beyond simple automation by integrating declarative configuration, comprehensive data recording, sandboxed compute, and automated self-improvement loops driven by metrics and benchmarks.

Download summary

Key takeaways

  1. The Limitations of Local Agents 0:01

    Relying on local, interactive prompting limits throughput and creates significant challenges in cost management, security, and governance across platform teams (0:57, 1:42).

  2. Definition of a Software Factory 0:03

    A Software Factory is a code-defined closed-loop system for continuously measuring and improving cloud coding agents. It is not merely a collection of AI teammates or personal agent swarms (3:57).

  3. The Closed Loop Principle 0:04

    The system must be a continuous closed loop, where agents process work (from triage to verification) and the resulting data is used to drive efficiency improvements (4:42).

  4. New Engineering Primitives 0:12

    The discipline of 'Factory Engineering' requires mastering tools like online evals, writing benchmarks, and conducting A/B tests on the factory itself, rather than just the product (7:42, 12:57).

More digests on AI Engineering

Technical details

  • Factory Infrastructure Stack

    The stack is layered: 1. Factories as Code (declarative definition, similar to Terraform); 2. Data/Context (recording all agent conversations, artifacts, and videos); 3. Compute (sandboxing for security and repeatability); 4. Inference (supporting swappable models like Claude Code, Codex, and custom harnesses); 5. Orchestration (moving work and integrating with tools like Jira and GitHub) (7:42).

  • Continuous Improvement Mechanisms

    Improvement is achieved through three mechanisms: 1. Measurement (DORA metrics, human interactions per PR, cost tracking); 2. Online Evals (having an agent score another agent's output, e.g., detecting redundant tests); 3. Self-Improvement (agents diagnosing failures and suggesting fixes as PRs against the factory definition) (12:57, 14:27).

  • Benchmarking

    To validate model or skill choices, benchmarks must be run against proprietary, real-world data, allowing teams to visualize trade-offs between cost and quality (15:57).

Mentioned resources

  • Warp (Platform/Product)
  • Warp Agentic Terminal (Open Source Tool)

Channel & topics

Watch on YouTube · Back to latest

This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.