Your PR Says the Tests Passed. Prove It. - Stephen Chin & Cassandra Chin
Summary
The talk addresses the critical issue of PRs making claims of passing tests without providing sufficient evidence or context. Cassandra Chin introduces Tracetoon, a framework designed to improve PR quality by recovering context across agent sessions and linking changes to their rationale and executed commands. The solution is enhanced by using Relate, an embedded graph database written in Rust, which builds an 'evidence graph' (or 'connected brief'). This graph stores relationships and provenance locally, significantly reducing reliance on expensive LLM tokens while providing reviewers and agents with a comprehensive, verifiable source of truth for every claim.
Key takeaways
-
Improving PR Quality and Provenance
1:42
Traditional PR generation often relies only on diffs, which can lead to inaccurate or incomplete information. Tracetoon improves this by reading through different sessions to reconstruct the full context of what was built, going beyond simple line-by-line diffs.
-
The Role of Graph Databases (Relate)
12:15
Relate, an embedded graph database in Rust, is used to build an 'evidence graph' that connects all submitted changes, code, and checks. This process is crucial because it provides a connected set of facts with their sources, allowing agents to draft PRs with higher accuracy and fewer passes.
-
Token Efficiency and Cost Reduction
18:30
By pre-computing the 'evidence graph' using Relate, the system significantly reduces the need for expensive LLM calls during the writing and reviewing phases. The speakers report that the graph solution reduces token usage by roughly half, improving both quality and cost-efficiency.
-
The Review Process Pattern
20:30
The optimal pattern is to use a high-quality model for the final writing (the 'writer') but use cheaper, less expensive models for the mechanical review and cross-checking steps. This balances quality with cost.
More digests on AI Engineering
Technical details
-
Tracetoon Framework
320s
Tracetoon is a framework designed to improve PRs by pulling in agent context and information about the code. It generates better quality PRs and optionally includes a comic for explanation. It processes all traces for a specific branch to research and review the description, looping until sufficient evidence is gathered.
-
Relate Database
800s
Relate is an open-source, embedded graph database written in Rust. It is designed for agent-focused use cases, allowing for accurate, multihop queries and explainable traces without the overhead of heavyweight graph approaches. It is fully in-memory and highly optimized.
-
PRBench
1260s
PRBench is a pull request benchmark and testing framework used to evaluate the quality of PRs. It utilizes audit traces, code, and LLM judges to measure improvements in PR quality on a 1 to 100 scale.
-
Agent Context and Provenance
540s
The core technical problem addressed is that AI agents' context windows compress and become inaccurate during long sessions. Tracetoon solves this by reading through audit trails/different sessions to capture the full history of work, preserving provenance outside the model's conversation.
Mentioned resources
- Tracetoon
- Relate
- PRBench
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.