Our Smaller Model Beat Opus on Finance Tasks - Alexander Shevchenko, Ramp
Summary
Alexander Shevchenko details why applied research remains critical for application companies, even with advanced frontier LLMs. He presents three production-ready projects: developing a specialized, faster model (Qwen 3.5) for financial QA that outperforms Opus 4.6 on latency; implementing KV cache cartridges to compress context for complex tasks like month-end close; and utilizing agentic swarms of 10,000 parallel agents for comprehensive security pen-testing. The overarching theme is achieving significant gains in time and cost efficiency through deep model specialization and advanced orchestration.
Key takeaways
-
Specialist Models Beat Frontier Models on Latency
8:36
By post-training a Qwen 3.5 model (35B parameters) for financial question answering, the team achieved accuracy above Opus 4.6 while significantly improving latency, making the model viable for high-volume production use cases. (5:16)
-
KV Cache Cartridges for Context Compression
9:58
This technique compresses context up to 20 to 30 times more efficiently into the KV cache, allowing agents to be pre-loaded with necessary context (e.g., for month-end close) before the user starts work. (5:58)
-
Agentic Swarms for Security and Efficiency
12:30
Using 10,000 parallel agents (via Ramp Inspect) for pen-testing found seven high-severity vulnerabilities. Furthermore, open-weight models can be used in swarms with detection rates comparable to frontier models, at a fraction of the cost. (7:50)
Technical details
-
Model Specialization and Post-Training
306s
The team trained a retrieval/financial QA model by partnering with Prime Intellect to post-train Qwen 3.5. This approach allows the creation of highly specialized, faster models that complement, rather than replace, powerful frontier orchestrators. (3:06)
-
Context Management (KV Cache Cartridges)
598s
Cartridges are a technique to compress large amounts of context into the KV cache offline, preparing the agent for complex, multi-step tasks like financial reconciliation. This is described as a lower-lift post-training method compared to full fine-tuning. (5:58)
-
Distributed Agent Orchestration
750s
The use of 10,000 parallel agents to perform 'agentic map reduce' for vulnerability scanning. The process involves agents finding vulnerabilities, deduplicating tickets using LLMs, and adversarially reproducing findings. (7:50)
-
Optimization and Routing
Ramp Router and Token Spend Management tools help optimize LLM usage by providing deep visibility into token spend (by category/taxonomy) and improving agent latency/cost per token through strategies like Nvidia Switchyard and smart flex offloading. (10:36)
Mentioned resources
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.