Your Financial Agent Needs to Show Its Work - Anirudh Veeraragavan, Fieldguide
Summary
In financial auditing, the defensibility of an AI agent's decision-making process is paramount. The speaker argues that merely achieving a correct answer is insufficient; the system must provide a complete, auditable trail of how the decision was reached. This requires implementing four core controls—Authorization, Execution Control, Evidence Provenance, and Evaluation Integrity—to ensure that the agent's output can be fully reconstructed and verified by a human auditor.
Key takeaways
-
The Need for Defensibility in AI Agents
In finance, the opinion (the product) is only as good as the basis behind it. The focus must shift from whether the agent is correct or wrong to demonstrating the integrity of the decision-making process. This pattern of 'decision assurance' applies beyond audit to domains like credit and Anti-Money-Laundering (AML).
-
Authorization Control
7:07
Authorization must be defined not just by tool call permission, but by the object's state at the time of the call. An agent must not overwrite work that a human has already reviewed and signed off on, even if the underlying evidence changes.
-
Execution Control
8:57
To ensure reproducibility, the system must maintain a detailed record of every step (inputs, outputs) taken during the agent's run. This allows auditors to compare runs across different points in time and identify the exact step where a discrepancy occurred.
-
Evidence Provenance
12:13
A citation must be an 'identity,' meaning it must include a specific ID, version, and a hash of the document's contents, rather than just a document name. This prevents ambiguity and allows for quality constraints on the source of evidence.
-
Evaluation Integrity
The gold data or evaluation set used to train or test the agent must itself be audited. A high overall score (e.g., 97%) can mask significant underlying failures or methodological changes in the evaluation data.
Technical details
-
Financial Audit Use Case
100s
The example used is a bank reconciliation, where the agent compares a client's book cash balance to the bank's cash balance, reconciling differences using items like deposits in transit and outstanding checks.
-
Decision Assurance Pattern
The agent must be programmed to recognize its limitations and stop, escalating the decision to a human when certain exceptional cases arise (e.g., when a new run disagrees with a previous run, or when evidence fails to conform to established rules).
Mentioned resources
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.