Stop Chatting With Spreadsheets — Francesco Galletta & Felippe Felisola Caso, Monzo
Summary
The presentation argues that for high-stakes domains like financial analysis, the primary bottleneck when using AI agents is not speed, but trust. To build reliable AI systems, speakers introduce five core design principles—Shared Domain, Consistency, Evidence, Accountability, and Verification. These principles guide the construction of a 'harness' that enforces structured, deterministic, and traceable processes, ensuring that the system's outputs are reliable and auditable, especially when dealing with money.
Key takeaways
-
Trust is the Bottleneck, Not Speed
7:02
For financial applications, the challenge is not generating fast answers, but generating answers that are trustworthy. The focus must shift from maximizing tokens to building robust, reliable systems.
-
The Five Principles of Trust
14:05
Trust must be engineered into the system using five principles: Shared Domain (ensuring model and user speak the same language), Consistency (using code for calculations), Evidence (showing receipts/data lineage), Accountability (maintaining human oversight), and Verification (using external reference points).
-
Code for Determinism
1:00:48
Calculations and any process that must yield a consistent result should be implemented in code (e.g., Python scripts) rather than relying on the language model's internal arithmetic capabilities, which are non-deterministic.
-
The Role of the Harness
20:13
A 'harness' is a structured framework designed to guide the agent through a problem-solving process, forcing the user to address critical steps and assumptions, thereby building trust and preventing the system from becoming a 'black box.'
Technical details
-
Shared Domain
2040s
The system must ensure the user's domain knowledge aligns with the model's understanding. The harness guides the user to challenge assumptions and clarify ambiguous terms, preventing the model from making incorrect inferences.
-
Consistency and Determinism
3648s
To ensure reliable results, deterministic calculations must be implemented via code (e.g., Python scripts) and unit tested. Relying on the LLM's internal reasoning for arithmetic leads to non-deterministic and often incorrect outputs.
-
Evidence and Traceability
5550s
The system must provide a clear audit trail, showing the source of every piece of information (the 'receipts'). This includes labeling inputs as facts (e.g., bank statements), assumptions (agent reasoning), or interpretations. Traceability must extend to code changes.
-
Accountability
The human user remains ultimately accountable for the agent's output. The harness must incorporate intentional 'gates' (e.g., reviewing tasks instead of individual changes) to ensure the user reviews critical decisions and assumptions, preventing the system from becoming a black box.
-
Verification
Verification requires a separate, external point of reference (a counterfactual) to check the agent's conclusions against. This prevents the system from being biased by its own internal data or assumptions.
Mentioned resources
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.