# 225,000 Lines of Tests, Written and Merged by Agents - Kush Agrawal, Duolingo

## Executive summary

Duolingo implemented an autonomous pipeline using AI agents to significantly boost unit test coverage across its large, legacy iOS codebase. The system, which operates on a continuous loop (Identify, Write, Verify, Resolve, Iterate), uses tools like Temporal, Cloud Code, and S3 to automate test generation and merging. This effort raised overall unit test coverage from 11% to 24% (for the MVVM slice, from 9% to 59%), merging over 225,000 lines of tests and 700+ PRs since March.

## Key takeaways

- Autonomous Testing Pipeline Architecture: The pipeline uses Temporal for workflow orchestration, Cloud Code as the coding agent, and S3 for shared state management. It coordinates five steps: identifying files needing tests, writing tests, verifying quality, resolving the PR, and iterating on feedback.
- Coverage Improvement Metrics: Overall unit test coverage increased from 11% to 24%. Specifically, the MVVM slice coverage jumped from 9% to 59%, with repositories increasing from 11% to 72%.
- Handling PR Lifecycle: The PR lifecycle workflow acts as a state machine, checking for merge conflicts or age (3 weeks). If CI is green and no reviewer is assigned, it finds the top contributor via `git blame` and assigns them, automating the merge process.
- Addressing CI/CD Flakiness: To improve reliability, the team implemented a GitHub action to auto-fix lint errors in CI, and eventually moved agents to MacRunners to allow local code building and linting before opening a PR.

## Technical details

- Codebase Architecture: The iOS codebase is in transition, utilizing MVVM (Model-View-ViewModel) architecture. The project focused on the MVVM layers (Repositories, View Models, Data Sources) because they are the easiest to test.
- Workflow Components: The system components include: Temporal (durable workflow execution engine), Cloud Code (coding agent), S3 (shared state), and GitHub (actions, labels, PRs).
- Test Generation Strategy: The backfill workflow identifies files by running an Xcode coverage report (uploaded to S3). A selector script scores files, giving a 2.5x boost to View Models, Repositories, and Data Sources, while penalizing larger files.
- Agent Guardrails and Review: To ensure PR quality, the system enforces deterministic rules (e.g., closing PRs that touch non-test files). An LLM-as-a-judge agent reviews PR diffs, rejecting production edits, changes/deletions of existing tests, and long waits.

## Practical implications

- The 'Identify, Write, Verify, Resolve, Iterate' loop pattern is generalizable beyond unit testing, allowing the company to apply the same infrastructure to modernize screens (e.g., migrating to SwiftUI) and generate snapshot tests.
- The process shifts the engineering bottleneck from 'writing code' to 'verifying code,' making unit tests critical guardrails for AI agents.
- The system demonstrates how to use AI agents to automate complex, multi-step CI/CD processes, including failure analysis and remediation attempts (e.g., fixing test failures in CI).

## Topics

AI Agents, Unit Testing, CI/CD, Software Architecture, Workflow Automation, Kush Agrawal, Duolingo, Dueling engineering blog

Source: https://www.youtube.com/watch?v=D_4Rn6uLdQM
