# Your PR Says the Tests Passed. Prove It. - Stephen Chin & Cassandra Chin

## Executive summary

The talk addresses the critical issue of PRs making claims of passing tests without providing sufficient evidence or context. Cassandra Chin introduces Tracetoon, a framework designed to improve PR quality by recovering context across agent sessions and linking changes to their rationale and executed commands. The solution is enhanced by using Relate, an embedded graph database written in Rust, which builds an 'evidence graph' (or 'connected brief'). This graph stores relationships and provenance locally, significantly reducing reliance on expensive LLM tokens while providing reviewers and agents with a comprehensive, verifiable source of truth for every claim.

## Key takeaways

- Improving PR Quality and Provenance: Traditional PR generation often relies only on diffs, which can lead to inaccurate or incomplete information. Tracetoon improves this by reading through different sessions to reconstruct the full context of what was built, going beyond simple line-by-line diffs.
- The Role of Graph Databases (Relate): Relate, an embedded graph database in Rust, is used to build an 'evidence graph' that connects all submitted changes, code, and checks. This process is crucial because it provides a connected set of facts with their sources, allowing agents to draft PRs with higher accuracy and fewer passes.
- Token Efficiency and Cost Reduction: By pre-computing the 'evidence graph' using Relate, the system significantly reduces the need for expensive LLM calls during the writing and reviewing phases. The speakers report that the graph solution reduces token usage by roughly half, improving both quality and cost-efficiency.
- The Review Process Pattern: The optimal pattern is to use a high-quality model for the final writing (the 'writer') but use cheaper, less expensive models for the mechanical review and cross-checking steps. This balances quality with cost.

## Technical details

- Tracetoon Framework: Tracetoon is a framework designed to improve PRs by pulling in agent context and information about the code. It generates better quality PRs and optionally includes a comic for explanation. It processes all traces for a specific branch to research and review the description, looping until sufficient evidence is gathered.
- Relate Database: Relate is an open-source, embedded graph database written in Rust. It is designed for agent-focused use cases, allowing for accurate, multihop queries and explainable traces without the overhead of heavyweight graph approaches. It is fully in-memory and highly optimized.
- PRBench: PRBench is a pull request benchmark and testing framework used to evaluate the quality of PRs. It utilizes audit traces, code, and LLM judges to measure improvements in PR quality on a 1 to 100 scale.
- Agent Context and Provenance: The core technical problem addressed is that AI agents' context windows compress and become inaccurate during long sessions. Tracetoon solves this by reading through audit trails/different sessions to capture the full history of work, preserving provenance outside the model's conversation.

## Practical implications

- Implement graph databases (like Relate) to store and manage the provenance and relationships of code changes and test evidence, rather than relying solely on LLM context windows.
- Structure the PR review process to use high-quality LLMs for creative writing/drafting, but use cheaper, specialized models for mechanical cross-checking and evidence validation.
- Use the 'connected brief' pattern to provide reviewers with a single, verifiable source of truth that links claims directly to the executed code and logs.

## Topics

AI Engineering, Pull Requests (PR), Graph Databases, Context Management, LLM Efficiency, Build Automation, Tracetoon, Relate, PRBench

Source: https://www.youtube.com/watch?v=gbeuC0ud1p4
