AI Engineer

225,000 Lines of Tests, Written and Merged by Agents - Kush Agrawal, Duolingo

Published 2026-11-11 · Duration 12:38

Summary

Duolingo implemented an autonomous pipeline using AI agents to significantly boost unit test coverage across its large, legacy iOS codebase. The system, which operates on a continuous loop (Identify, Write, Verify, Resolve, Iterate), uses tools like Temporal, Cloud Code, and S3 to automate test generation and merging. This effort raised overall unit test coverage from 11% to 24% (for the MVVM slice, from 9% to 59%), merging over 225,000 lines of tests and 700+ PRs since March.

Download summary

Key takeaways

  1. Autonomous Testing Pipeline Architecture 5:20

    The pipeline uses Temporal for workflow orchestration, Cloud Code as the coding agent, and S3 for shared state management. It coordinates five steps: identifying files needing tests, writing tests, verifying quality, resolving the PR, and iterating on feedback.

  2. Coverage Improvement Metrics 2:20

    Overall unit test coverage increased from 11% to 24%. Specifically, the MVVM slice coverage jumped from 9% to 59%, with repositories increasing from 11% to 72%.

  3. Handling PR Lifecycle 7:20

    The PR lifecycle workflow acts as a state machine, checking for merge conflicts or age (3 weeks). If CI is green and no reviewer is assigned, it finds the top contributor via `git blame` and assigns them, automating the merge process.

  4. Addressing CI/CD Flakiness 9:40

    To improve reliability, the team implemented a GitHub action to auto-fix lint errors in CI, and eventually moved agents to MacRunners to allow local code building and linting before opening a PR.

More digests on AI Agents

Technical details

  • Codebase Architecture 190s

    The iOS codebase is in transition, utilizing MVVM (Model-View-ViewModel) architecture. The project focused on the MVVM layers (Repositories, View Models, Data Sources) because they are the easiest to test.

  • Workflow Components 320s

    The system components include: Temporal (durable workflow execution engine), Cloud Code (coding agent), S3 (shared state), and GitHub (actions, labels, PRs).

  • Test Generation Strategy 380s

    The backfill workflow identifies files by running an Xcode coverage report (uploaded to S3). A selector script scores files, giving a 2.5x boost to View Models, Repositories, and Data Sources, while penalizing larger files.

  • Agent Guardrails and Review 670s

    To ensure PR quality, the system enforces deterministic rules (e.g., closing PRs that touch non-test files). An LLM-as-a-judge agent reviews PR diffs, rejecting production edits, changes/deletions of existing tests, and long waits.

Mentioned resources

Channel & topics

Watch on YouTube · Back to latest

This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.