Cole Medin

I REALLY Want Open Models to Win, But They Just Don't Cut It (Yet)

Published 2026-11-05 · Duration 18:34

Summary

While open-weight models (e.g., Kimi K3.1 Pro) show impressive benchmarks and offer significant cost savings for simple tasks, they currently fail to maintain reliability and context during long-running, complex agentic coding workflows. The speaker demonstrates that for robust build engineering processes, a hybrid approach is necessary: using powerful frontier models (like Claude Opus 5.5) for critical steps such as planning and reviewing, and open models as the cost-effective 'workhorse' for implementation.

Download summary

Key takeaways

  1. Open Models Excel at Short Tasks, Fail at Long Runs 5:52

    Open models perform comparably to frontier models on short, isolated tasks, making them ideal for routine business automation steps. However, they struggle with maintaining context and following complex rules over extended periods, leading to 'drift' in large codebases. (3:52)

  2. Cost Efficiency is Misleading

    Although open models may have a lower API cost per token, their tendency to stall, fail, and require extensive iteration makes them significantly more expensive per completed task than proprietary models in complex workflows. (13:20)

  3. The Solution is Model Mixing

    The most effective current strategy is to use the most powerful model (e.g., Opus 5.5) for high-level planning and rigorous review, and then utilize a capable open model (e.g., Kimi) for the bulk implementation (the 'workhorse') to achieve both reliability and cost savings. (17:48)

More digests on Agentic Workflow Design

Technical details

  • Benchmark Performance 112s

    Open models like Kimi K3.1 Pro show strong performance on benchmarks (e.g., Terminal Bench 4.0), sometimes matching or exceeding proprietary models like GPT-6 Astra and Claude Opus 5.5 on short tasks. (1:52)

  • Agentic Workflow Failure 343s

    In the 'AI Software Factory' test, open models struggled to adhere to strict global rules (e.g., 'don't touch the PRD'), leading to constant failures and required manual intervention. (5:43)

  • Comparative Test Results (Northstar Game)

    The test showed Claude (Opus 5.5/Sonnet 5.5) completing 12/12 tickets in approximately 14 hours ($255 cost), while Kimi 3.1 Pro only completed 5/12 tickets in 44 hours, costing $180 but resulting in a higher cost per ticket ($36 vs. $21). (11:22)

  • Process Guidance

    The speaker's workflow involves a structured process: planning, implementing, and then having separate agents review and playtest the output before merging. This process is crucial for managing complexity. (9:11)

Mentioned resources

Channel & topics

Watch on YouTube · Back to latest

This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.