AI Native Dev

Harness Engineering: The Loops That Make Agents Trustworthy | Dru Knox, Tessl

Published 2026-11-13 · Duration 30:24

Summary

The talk outlines the concept of a 'Software Factory'—a development discipline defined as the automated workflow from ticket creation to Pull Request (PR). It argues that success relies less on the capabilities of AI agents and more on improving the underlying process through automation loops. Key focus areas include monitoring diagnostic metrics (like Cost per PR and PR volume), addressing five common bottlenecks (Agent IT, Code Quality, Repetition, Test Quality, and CI time budgets), and scaling up to orchestrate large, multi-milestone work streams.

Download summary

Key takeaways

  1. Software Factory Definition 0:11

    A software factory is fundamentally a process loop: moving work from an issue tracker (ticket) to a PR. It is a discipline enabled by technology, not a single feature. The core workflow involves delegating a ticket to a cloud sandbox, generating a PR, automated code review, and a 'PR shepherd' loop that addresses feedback until convergence.

  2. Process Correction over Code Correction 1:13

    The most effective shift is moving away from directly correcting code via PR comments. Instead, teams must correct the process that produces the code by improving the environment, adding more tests, or enhancing linting rules to achieve a 'one-shot' PR.

  3. Critical Diagnostic Metrics 3:05

    To gauge factory health, track metrics like PR volume, Cost per PR (a key leading indicator that should decrease), Review rounds, and Time to merge. Divergence from baseline in these metrics signals a pipeline failure.

  4. The Five Bottlenecks 3:53

    The first five problems encountered are: 1) Agent IT (access to all necessary systems); 2) Code Quality (architectural issues like 'god modules'); 3) Repetition (identifying and automating recurring review comments); 4) Test Quality (using mutation testing and managing bloat); and 5) CI (setting time budgets for different test types).

More digests on Software Factory

Technical details

  • Software Factory Workflow 113s

    The process involves three main loops: 1) Delegating a ticket to a cloud sandbox for PR creation. 2) Automated code review (using CI/GitHub signals). 3) The PR shepherd loop, which monitors CI results and agent/human comments, iteratively pushing changes until the PR converges.

  • Code Quality Metrics 261s

    To improve architectural quality, monitor metrics like file line count (flagging files >1000 lines) and method cohesion. Low cohesion suggests methods are working with disparate objects (e.g., 'user' vs. 'home'), indicating a need for module splitting.

  • Test Quality Improvement 299s

    Beyond simple coverage, implement mutation testing (e.g., using Stryker) to fuzz the codebase and ensure tests catch actual breakage. Also, manage test bloat by setting time budgets for different test types (quick, loose, unit, integration, nightly).

  • Orchestrating Large Work 350s

    For complex features, the process should move from a high-level 'spec' to breaking it down into vertical milestones, then into implementation issues. This is followed by a PR merge shepherd that updates the issue tracker state based on the merged PR's impact (e.g., API contract changes).

Mentioned resources

  • Tessl (Company/Product)
  • Linear (Issue Tracker)
  • Stryker (Tool/Library)

Channel & topics

Watch on YouTube · Back to latest

This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.