225,000 Lines of Tests, Written and Merged by Agents - Kush Agrawal, Duolingo
Summary
Duolingo implemented an autonomous pipeline using AI agents to significantly boost unit test coverage across its large, legacy iOS codebase. The system, which operates on a continuous loop (Identify, Write, Verify, Resolve, Iterate), uses tools like Temporal, Cloud Code, and S3 to automate test generation and merging. This effort raised overall unit test coverage from 11% to 24% (for the MVVM slice, from 9% to 59%), merging over 225,000 lines of tests and 700+ PRs since March.
Key takeaways
-
Autonomous Testing Pipeline Architecture
5:20
The pipeline uses Temporal for workflow orchestration, Cloud Code as the coding agent, and S3 for shared state management. It coordinates five steps: identifying files needing tests, writing tests, verifying quality, resolving the PR, and iterating on feedback.
-
Coverage Improvement Metrics
2:20
Overall unit test coverage increased from 11% to 24%. Specifically, the MVVM slice coverage jumped from 9% to 59%, with repositories increasing from 11% to 72%.
-
Handling PR Lifecycle
7:20
The PR lifecycle workflow acts as a state machine, checking for merge conflicts or age (3 weeks). If CI is green and no reviewer is assigned, it finds the top contributor via `git blame` and assigns them, automating the merge process.
-
Addressing CI/CD Flakiness
9:40
To improve reliability, the team implemented a GitHub action to auto-fix lint errors in CI, and eventually moved agents to MacRunners to allow local code building and linting before opening a PR.
Technical details
-
Codebase Architecture
190s
The iOS codebase is in transition, utilizing MVVM (Model-View-ViewModel) architecture. The project focused on the MVVM layers (Repositories, View Models, Data Sources) because they are the easiest to test.
-
Workflow Components
320s
The system components include: Temporal (durable workflow execution engine), Cloud Code (coding agent), S3 (shared state), and GitHub (actions, labels, PRs).
-
Test Generation Strategy
380s
The backfill workflow identifies files by running an Xcode coverage report (uploaded to S3). A selector script scores files, giving a 2.5x boost to View Models, Repositories, and Data Sources, while penalizing larger files.
-
Agent Guardrails and Review
670s
To ensure PR quality, the system enforces deterministic rules (e.g., closing PRs that touch non-test files). An LLM-as-a-judge agent reviews PR diffs, rejecting production edits, changes/deletions of existing tests, and long waits.
Mentioned resources
- Kush Agrawal, Duolingo
- Dueling engineering blog
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.