# 60% of Our Data PRs Now Merge With No Human Review - Chetas Joshi, Robinhood

## Executive summary

Robinhood implemented a hybrid PR reviewer system to manage the influx of data pipeline changes arriving at machine speed, which human review could not match. The system combines three components: deterministic scripts (for verifiable facts), an AI agent (for judgment and finding missed context), and human reviewers (for protected calls). This approach allows approximately 60% of data pipeline PRs to merge without human review, saving an estimated 132 engineering hours monthly, while maintaining safety by escalating high-risk changes.

## Key takeaways

- Hybrid Review Model (Three Duties, Three Owners): The system uses a hybrid approach: Scripts prove what can be proven (deterministic checks against source of truth), the Agent judges what needs context (analyzing code/repo for missed risks), and People own the calls (mandatory for sensitive grants, intent changes, or low agent confidence).
- Layer 1 Checks (Scripts): Scripts perform deterministic checks on live systems, including Data Contracts (schema comparison against a registry), required tags (owner/PII labeling), Lineage Blast Radius (checking registered downstream consumers), and Performance testing (running changes in staging to ensure SLA adherence).
- Agent's Role (Layer 2): The AI agent reasons over the code and repository, looking for relationships missed by registered lineage (e.g., finding an unregistered job reading a widened column). It scores the risk and either approves or escalates to a human with detailed reasoning.
- Safety and Learning: The system learns from human feedback; manual corrections to the agent's verdict update the review rules. The goal is to measure and reduce 'misses' (bad PRs approved) and 'false escalations' (fine PRs unnecessarily flagged).

## Technical details

- Data Pipeline PR Scope: PRs cover schema changes, pipeline code, configuration changes, and access grants. The challenge is that PRs arrive at machine speed, but safe review runs at human speed.
- Layer 1 Checks (Scripts): Checks include: 1) Data Contracts (schema registry comparison); 2) Required Tags (catalog checks for owner/PII); 3) Lineage Blast Radius (checking registered downstream jobs); 4) Performance (staging run to validate freshness SLA).
- Agent Capabilities: The agent uses repository tools (read, search, list files) to find relationships beyond the diff, such as finding an unregistered job reading a modified column, but it cannot query live catalog lineage or production state.
- Merge Requirements: A PR merge requires passing Layer 1 checks (scripts) AND an approving review (agent or person).
- Future Architecture Proposal: The proposed system would use a classifier to view the whole PR, allowing the agent to treat Layer 1 checks as callable tools, enabling judgment based on live facts, rather than just repository code.

## Practical implications

- The system significantly reduces manual review effort (estimated 75% less), freeing up engineering time.
- The process maintains accountability by ensuring protected decisions (e.g., sensitive data grants, intent changes) always require a human sign-off.
- The architecture is designed to be continuously improved by treating human corrections and feedback as new rules, rather than just votes.

## Topics

Data Engineering, DevOps, AI/ML, CI/CD, Data Governance, Chetas Joshi, Robinhood

Source: https://www.youtube.com/watch?v=MQiojLnK-E0
