I REALLY Want Open Models to Win, But They Just Don't Cut It (Yet)
Summary
While open-weight models (e.g., Kimi K3.1 Pro) show impressive benchmarks and offer significant cost savings for simple tasks, they currently fail to maintain reliability and context during long-running, complex agentic coding workflows. The speaker demonstrates that for robust build engineering processes, a hybrid approach is necessary: using powerful frontier models (like Claude Opus 5.5) for critical steps such as planning and reviewing, and open models as the cost-effective 'workhorse' for implementation.
Key takeaways
-
Open Models Excel at Short Tasks, Fail at Long Runs
5:52
Open models perform comparably to frontier models on short, isolated tasks, making them ideal for routine business automation steps. However, they struggle with maintaining context and following complex rules over extended periods, leading to 'drift' in large codebases. (3:52)
-
Cost Efficiency is Misleading
Although open models may have a lower API cost per token, their tendency to stall, fail, and require extensive iteration makes them significantly more expensive per completed task than proprietary models in complex workflows. (13:20)
-
The Solution is Model Mixing
The most effective current strategy is to use the most powerful model (e.g., Opus 5.5) for high-level planning and rigorous review, and then utilize a capable open model (e.g., Kimi) for the bulk implementation (the 'workhorse') to achieve both reliability and cost savings. (17:48)
More digests on Agentic Workflow Design
Technical details
-
Benchmark Performance
112s
Open models like Kimi K3.1 Pro show strong performance on benchmarks (e.g., Terminal Bench 4.0), sometimes matching or exceeding proprietary models like GPT-6 Astra and Claude Opus 5.5 on short tasks. (1:52)
-
Agentic Workflow Failure
343s
In the 'AI Software Factory' test, open models struggled to adhere to strict global rules (e.g., 'don't touch the PRD'), leading to constant failures and required manual intervention. (5:43)
-
Comparative Test Results (Northstar Game)
The test showed Claude (Opus 5.5/Sonnet 5.5) completing 12/12 tickets in approximately 14 hours ($255 cost), while Kimi 3.1 Pro only completed 5/12 tickets in 44 hours, costing $180 but resulting in a higher cost per ticket ($36 vs. $21). (11:22)
-
Process Guidance
The speaker's workflow involves a structured process: planning, implementing, and then having separate agents review and playtest the output before merging. This process is crucial for managing complexity. (9:11)
Mentioned resources
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.