# Harness Engineering: The Loops That Make Agents Trustworthy | Dru Knox, Tessl

## Executive summary

The talk outlines the concept of a 'Software Factory'—a development discipline defined as the automated workflow from ticket creation to Pull Request (PR). It argues that success relies less on the capabilities of AI agents and more on improving the underlying process through automation loops. Key focus areas include monitoring diagnostic metrics (like Cost per PR and PR volume), addressing five common bottlenecks (Agent IT, Code Quality, Repetition, Test Quality, and CI time budgets), and scaling up to orchestrate large, multi-milestone work streams.

## Key takeaways

- Software Factory Definition: A software factory is fundamentally a process loop: moving work from an issue tracker (ticket) to a PR. It is a discipline enabled by technology, not a single feature. The core workflow involves delegating a ticket to a cloud sandbox, generating a PR, automated code review, and a 'PR shepherd' loop that addresses feedback until convergence.
- Process Correction over Code Correction: The most effective shift is moving away from directly correcting code via PR comments. Instead, teams must correct the process that produces the code by improving the environment, adding more tests, or enhancing linting rules to achieve a 'one-shot' PR.
- Critical Diagnostic Metrics: To gauge factory health, track metrics like PR volume, Cost per PR (a key leading indicator that should decrease), Review rounds, and Time to merge. Divergence from baseline in these metrics signals a pipeline failure.
- The Five Bottlenecks: The first five problems encountered are: 1) Agent IT (access to all necessary systems); 2) Code Quality (architectural issues like 'god modules'); 3) Repetition (identifying and automating recurring review comments); 4) Test Quality (using mutation testing and managing bloat); and 5) CI (setting time budgets for different test types).

## Technical details

- Software Factory Workflow: The process involves three main loops: 1) Delegating a ticket to a cloud sandbox for PR creation. 2) Automated code review (using CI/GitHub signals). 3) The PR shepherd loop, which monitors CI results and agent/human comments, iteratively pushing changes until the PR converges.
- Code Quality Metrics: To improve architectural quality, monitor metrics like file line count (flagging files >1000 lines) and method cohesion. Low cohesion suggests methods are working with disparate objects (e.g., 'user' vs. 'home'), indicating a need for module splitting.
- Test Quality Improvement: Beyond simple coverage, implement mutation testing (e.g., using Stryker) to fuzz the codebase and ensure tests catch actual breakage. Also, manage test bloat by setting time budgets for different test types (quick, loose, unit, integration, nightly).
- Orchestrating Large Work: For complex features, the process should move from a high-level 'spec' to breaking it down into vertical milestones, then into implementation issues. This is followed by a PR merge shepherd that updates the issue tracker state based on the merged PR's impact (e.g., API contract changes).

## Practical implications

- Implement automated loops that monitor agent logs and CI/CD signals to proactively identify system access failures (Agent IT).
- Treat 'Cost per PR' as the primary leading indicator of factory health, aiming for a downward trend.
- Establish time budgets for different test suites (e.g., quick vs. nightly) to prevent CI pipelines from becoming bottlenecks.
- Shift automation efforts to capture and resolve recurring issues found in PR comments, preventing manual repetition.

## Topics

Software Factory, AI Agents, DevOps Metrics, CI/CD, Process Automation, Code Quality, Tessl, Linear, Stryker

Source: https://www.youtube.com/watch?v=RQerOLmFiiE
