Claude Code Looked Right but Missed $800K - Natalia Connolly & Kevin Madura, AlixPartners
Summary
This talk details the challenges of applying Large Language Models (LLMs), specifically Claude Code, to complex, messy real-world data like post-merger procurement contracts (40,000+ documents). The speakers highlight that raw contract data often contains structural issues (e.g., orphaned amendments, rate tables spanning multiple pages, or multiple amendments restating values) that cause LLMs to generate incorrect totals or miss critical deadlines. The solution presented is a highly robust, five-stage data processing pipeline that structures the data first, followed by custom tools and embedded business rules, ensuring that every LLM output is traceable to its source.
Key takeaways
-
Data Structure is Paramount
LLMs struggle with unstructured contract data. To ensure accuracy, a multi-stage pipeline is necessary: 1) Preprocessing (OCR/Document Intelligence), 2) Classification (identifying MSA, Amendments, etc.), 3) Extraction (handling multi-page tables), 4) Mapping, and finally, 5) LLM processing. This upfront structuring prevents the LLM from 'spinning its wheels' [17:37].
-
Embed Business Logic in Tools
13:46
Instead of relying solely on prompt context, embed business rules and domain knowledge into custom tools (e.g., `get rate card page`). This allows the LLM to perform specific, governed actions, such as querying for a specific vendor's spend or enforcing the most recent amendment's value, thereby mitigating common data gotchas [8:26].
-
Handle Data Governance and Errors Gracefully
When integrating data from multiple sources (Entity A vs. Entity B), the system must anticipate governance failures (e.g., access denied, malformed data). The LLM must be given feedback on the failure type to fail gracefully or query an alternative authoritative database, rather than assuming a zero value [13:25].
-
Context Must Be Business-Specific
Beyond technical context, the agent requires deep business context (e.g., the fact that the analysis is part of an active M&A deal). Failing to provide this context can lead to irrelevant or invalid recommendations, even if the data extraction is perfect [16:04].
More digests on Large Language Models (LLMs)
Technical details
-
Data Pipeline Architecture
546s
A five-stage pipeline was implemented: 1) Preprocessing (OCR, Document Intelligence); 2) Classification (identifying contract type: MSA, Amendment, NDA); 3) Terms, Rates, and Spend Extraction (handling multi-page tables); 4) Mapping (services to spend categories); 5) LLM Processing (Claude Code). This structure ensures data is highly structured before the LLM interacts with it [5:46].
-
Multimodal Data Processing
708s
To process charts and graphs, the method involved treating contract pages as images and feeding them through DSPI (Document Structure Processing Intelligence) to generate a highly structured output, converting unstructured data into a database of relevant elements [7:08].
-
LLM Tooling and Customization
826s
Custom tools are built to enforce business logic (e.g., `get rate card page`). This approach is preferred over relying solely on prompt instructions, as it allows for programmatic control over data querying and retrieval, making the analysis repeatable and auditable [8:26].
Mentioned resources
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.