Why Waiting for the Next AI Model Can Pay Off - John Stecher, Blackstone & Scott Worland, Norm AI
Summary
The discussion outlines a strategic approach to integrating advanced AI agents into enterprise systems, particularly within highly regulated sectors like finance. Key advice emphasizes that while AI agents are powerful, successful deployment hinges on establishing a cohesive data strategy, prioritizing straight-through processing (STP), and maintaining architectural flexibility. Engineers are advised to view internal applications as potentially disposable, focusing instead on rapid Proof of Concept (POC) development and building adaptable platforms that can accommodate future, more sophisticated frontier models.
Key takeaways
-
Data Strategy is Paramount for AI Success
3:10
For AI agents to succeed, companies must establish a cohesive data strategy that moves proprietary data out of transactional systems into an easily managed and recallable format. Without this, AI initiatives are likely to fail (John Stecher).
-
STP Remains Critical, Even with Agents
4:30
Straight-through processing (STP) is not rendered obsolete by AI. The best approach involves using agents to craft workflows and then optimizing the underlying code to ensure STP, using agents only to inject intelligence where human decisions were previously required (John Stecher).
-
The Importance of Evaluation Sets (Evals)
7:50
In regulated fields, rigorous testing is crucial. Building large, hand-labeled evaluation sets (tens of thousands of examples) is necessary to benchmark agent performance and ensure consistency as model versions or underlying harnesses change (Scott Worland).
-
Architectural Flexibility Over Maintenance
Engineers should assume that internal applications can be 'thrown away' and rebuilt, as they will become obsolete faster than ever before. The focus should be on building adaptable platforms with 'conduits' for future frontier intelligence, rather than creating rigid systems (John Stecher, Scott Worland).
Technical details
-
Agentic Development Workflow
360s
Norm AI's platform was built by first adding significant structure to the system, treating the process like a 'large map reduce problem' to ensure high quality and consistency in regulatory outputs. This allowed them to leverage a benchmark set for agentic testing (Scott Worland).
-
Proof of Concept (POC) Methodology
1025s
To accelerate development, engineers should prioritize getting to a workable POC quickly. This involves leveraging tools to move from concept mapping to writing code, focusing on producing value rather than worrying about full cybersecurity or production readiness initially (John Stecher).
-
System Resilience and Oversight
For highly sensitive data, core primitives and code ownership must retain mandatory human oversight. This is necessary to manage data security and ensure that the resulting software is traceable and monitorable within the SRE platform (Scott Worland).
Mentioned resources
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.