# Claude Code Looked Right but Missed $800K - Natalia Connolly & Kevin Madura, AlixPartners

## Executive summary

This talk details the challenges of applying Large Language Models (LLMs), specifically Claude Code, to complex, messy real-world data like post-merger procurement contracts (40,000+ documents). The speakers highlight that raw contract data often contains structural issues (e.g., orphaned amendments, rate tables spanning multiple pages, or multiple amendments restating values) that cause LLMs to generate incorrect totals or miss critical deadlines. The solution presented is a highly robust, five-stage data processing pipeline that structures the data first, followed by custom tools and embedded business rules, ensuring that every LLM output is traceable to its source.

## Key takeaways

- Data Structure is Paramount: LLMs struggle with unstructured contract data. To ensure accuracy, a multi-stage pipeline is necessary: 1) Preprocessing (OCR/Document Intelligence), 2) Classification (identifying MSA, Amendments, etc.), 3) Extraction (handling multi-page tables), 4) Mapping, and finally, 5) LLM processing. This upfront structuring prevents the LLM from 'spinning its wheels' [17:37].
- Embed Business Logic in Tools: Instead of relying solely on prompt context, embed business rules and domain knowledge into custom tools (e.g., `get rate card page`). This allows the LLM to perform specific, governed actions, such as querying for a specific vendor's spend or enforcing the most recent amendment's value, thereby mitigating common data gotchas [8:26].
- Handle Data Governance and Errors Gracefully: When integrating data from multiple sources (Entity A vs. Entity B), the system must anticipate governance failures (e.g., access denied, malformed data). The LLM must be given feedback on the failure type to fail gracefully or query an alternative authoritative database, rather than assuming a zero value [13:25].
- Context Must Be Business-Specific: Beyond technical context, the agent requires deep business context (e.g., the fact that the analysis is part of an active M&A deal). Failing to provide this context can lead to irrelevant or invalid recommendations, even if the data extraction is perfect [16:04].

## Technical details

- Data Pipeline Architecture: A five-stage pipeline was implemented: 1) Preprocessing (OCR, Document Intelligence); 2) Classification (identifying contract type: MSA, Amendment, NDA); 3) Terms, Rates, and Spend Extraction (handling multi-page tables); 4) Mapping (services to spend categories); 5) LLM Processing (Claude Code). This structure ensures data is highly structured before the LLM interacts with it [5:46].
- Multimodal Data Processing: To process charts and graphs, the method involved treating contract pages as images and feeding them through DSPI (Document Structure Processing Intelligence) to generate a highly structured output, converting unstructured data into a database of relevant elements [7:08].
- LLM Tooling and Customization: Custom tools are built to enforce business logic (e.g., `get rate card page`). This approach is preferred over relying solely on prompt instructions, as it allows for programmatic control over data querying and retrieval, making the analysis repeatable and auditable [8:26].

## Practical implications

- When building LLM-powered systems for structured data (like contracts or invoices), prioritize building a robust, multi-stage data ingestion pipeline over relying solely on the LLM's raw extraction capabilities.
- Design custom tools and views within the database to embed critical business logic and domain knowledge, ensuring the LLM's output is constrained and verifiable.
- Implement comprehensive error handling and governance checks (e.g., handling 'access denied' or 'malformed' data) so the agent can recover or flag issues rather than proceeding with incorrect assumptions.
- Always provide the LLM with the current operational context (e.g., M&A status, recent management actions) to prevent generating irrelevant or invalid recommendations.

## Topics

Large Language Models (LLMs), Natural Language Processing (NLP), Data Pipeline Engineering, Contract Lifecycle Management (CLM), Business Process Automation, AlixPartners, AIE CODE summit

Source: https://www.youtube.com/watch?v=u7fsyrXsPBE
