AI Engineer

60% of Our Data PRs Now Merge With No Human Review - Chetas Joshi, Robinhood

Published 2026-11-10 · Duration 28:00

Summary

Robinhood implemented a hybrid PR reviewer system to manage the influx of data pipeline changes arriving at machine speed, which human review could not match. The system combines three components: deterministic scripts (for verifiable facts), an AI agent (for judgment and finding missed context), and human reviewers (for protected calls). This approach allows approximately 60% of data pipeline PRs to merge without human review, saving an estimated 132 engineering hours monthly, while maintaining safety by escalating high-risk changes.

Download summary

Key takeaways

  1. Hybrid Review Model (Three Duties, Three Owners) 0:54

    The system uses a hybrid approach: Scripts prove what can be proven (deterministic checks against source of truth), the Agent judges what needs context (analyzing code/repo for missed risks), and People own the calls (mandatory for sensitive grants, intent changes, or low agent confidence).

  2. Layer 1 Checks (Scripts) 2:30

    Scripts perform deterministic checks on live systems, including Data Contracts (schema comparison against a registry), required tags (owner/PII labeling), Lineage Blast Radius (checking registered downstream consumers), and Performance testing (running changes in staging to ensure SLA adherence).

  3. Agent's Role (Layer 2) 3:50

    The AI agent reasons over the code and repository, looking for relationships missed by registered lineage (e.g., finding an unregistered job reading a widened column). It scores the risk and either approves or escalates to a human with detailed reasoning.

  4. Safety and Learning 10:00

    The system learns from human feedback; manual corrections to the agent's verdict update the review rules. The goal is to measure and reduce 'misses' (bad PRs approved) and 'false escalations' (fine PRs unnecessarily flagged).

More digests on Data Engineering

Technical details

  • Data Pipeline PR Scope 100s

    PRs cover schema changes, pipeline code, configuration changes, and access grants. The challenge is that PRs arrive at machine speed, but safe review runs at human speed.

  • Layer 1 Checks (Scripts) 150s

    Checks include: 1) Data Contracts (schema registry comparison); 2) Required Tags (catalog checks for owner/PII); 3) Lineage Blast Radius (checking registered downstream jobs); 4) Performance (staging run to validate freshness SLA).

  • Agent Capabilities 230s

    The agent uses repository tools (read, search, list files) to find relationships beyond the diff, such as finding an unregistered job reading a modified column, but it cannot query live catalog lineage or production state.

  • Merge Requirements 120s

    A PR merge requires passing Layer 1 checks (scripts) AND an approving review (agent or person).

  • Future Architecture Proposal 1100s

    The proposed system would use a classifier to view the whole PR, allowing the agent to treat Layer 1 checks as callable tools, enabling judgment based on live facts, rather than just repository code.

Mentioned resources

Channel & topics

Watch on YouTube · Back to latest

This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.