An Agent That Fixes Its Own Mistakes - Saloni Gandhi & Ziwei Guo, LinkedIn
Summary
This presentation details the implementation of Recursive Self-Improvement (RSI) for AI agents, exemplified by a 'user understanding agent' at LinkedIn. RSI is a closed-loop system designed to enable AI agents to self-improve by continuously evaluating their own outputs, identifying failure patterns, diagnosing root causes, and proposing targeted fixes. The process involves seven key steps, ranging from initial agent execution and evaluation (L1 Judge) to deep root cause analysis (L2 Judge) and automated prompt updates. A major focus was placed on improving the reliability of the evaluation process itself, leading to targeted rubric and evaluation system enhancements.
Key takeaways
-
The Recursive Self-Improvement (RSI) Loop
2:00
The RSI process is a seven-step cycle: 1) Run the agent to generate traces; 2) Use an evaluation engine (Judge) to score outputs; 3) Perform agent reflection to identify root causes (e.g., prompt issue, agent issue); 4) Generate edits proposals (using JPA); 5) Validate these edits; 6) Update the prompt/baseline; and 7) Re-evaluate on a hold-out set to confirm improvement. This loop optimizes the agent's 'harness' (orchestrator), not the base LLM itself.
-
Evaluation Quality is Paramount
2:40
The reliability of the improvement system depends on the quality of the evaluator. Analysis of human annotations revealed that the evaluation judge was better at recognizing strong outputs than distinguishing between weaker ones. Recurring weaknesses included the judge prioritizing plausibility over factual completeness, and ambiguities in rubrics (e.g., distinguishing between domain, function, and skill).
-
Scaling RSI for Enterprise Use
3:30
To scale RSI across an organization, the focus must shift to standardizing infrastructure. Key prerequisites include developing reusable evaluation frameworks, automating feedback loops (collecting execution traces and detecting recurring failures), and implementing robust validation mechanisms (like hold-out evaluations) to ensure automated changes are genuinely beneficial.
More digests on Recursive Self-Improvement (RSI)
Technical details
-
RSI Architecture Components
100s
The AI system's 'harness' can be optimized across several components: Planning and Logic (orchestration), Context and Memory (session state), Tool Interface (external/internal APIs), and Artifact Storage. The optimization ladder progresses from simple prompt updates to automating the entire workflow and eventually optimizing the optimizer code itself.
-
Root Cause Diagnosis (L2 Judge)
140s
The reflection process involves clustering failing traces and running an L2 Judge to assign a root cause and tag. Root causes can be attributed to the agent, the model, the prompt, the harness, the judge, or the rubric. A key challenge identified was that the original reflection taxonomy lacked a mechanism to explicitly state that the L1 Judge was incorrect.
-
Optimization and Validation
155s
Edits proposals are generated based on identified root causes. These proposals are validated using the same evaluation engine on a mini-batch validation set before the prompt is updated and integrated as the new baseline.
Mentioned resources
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.