# I REALLY Want Open Models to Win, But They Just Don't Cut It (Yet)

## Executive summary

While open-weight models (e.g., Kimi K3.1 Pro) show impressive benchmarks and offer significant cost savings for simple tasks, they currently fail to maintain reliability and context during long-running, complex agentic coding workflows. The speaker demonstrates that for robust build engineering processes, a hybrid approach is necessary: using powerful frontier models (like Claude Opus 5.5) for critical steps such as planning and reviewing, and open models as the cost-effective 'workhorse' for implementation.

## Key takeaways

- Open Models Excel at Short Tasks, Fail at Long Runs: Open models perform comparably to frontier models on short, isolated tasks, making them ideal for routine business automation steps. However, they struggle with maintaining context and following complex rules over extended periods, leading to 'drift' in large codebases. (3:52)
- Cost Efficiency is Misleading: Although open models may have a lower API cost per token, their tendency to stall, fail, and require extensive iteration makes them significantly more expensive per completed task than proprietary models in complex workflows. (13:20)
- The Solution is Model Mixing: The most effective current strategy is to use the most powerful model (e.g., Opus 5.5) for high-level planning and rigorous review, and then utilize a capable open model (e.g., Kimi) for the bulk implementation (the 'workhorse') to achieve both reliability and cost savings. (17:48)

## Technical details

- Benchmark Performance: Open models like Kimi K3.1 Pro show strong performance on benchmarks (e.g., Terminal Bench 4.0), sometimes matching or exceeding proprietary models like GPT-6 Astra and Claude Opus 5.5 on short tasks. (1:52)
- Agentic Workflow Failure: In the 'AI Software Factory' test, open models struggled to adhere to strict global rules (e.g., 'don't touch the PRD'), leading to constant failures and required manual intervention. (5:43)
- Comparative Test Results (Northstar Game): The test showed Claude (Opus 5.5/Sonnet 5.5) completing 12/12 tickets in approximately 14 hours ($255 cost), while Kimi 3.1 Pro only completed 5/12 tickets in 44 hours, costing $180 but resulting in a higher cost per ticket ($36 vs. $21). (11:22)
- Process Guidance: The speaker's workflow involves a structured process: planning, implementing, and then having separate agents review and playtest the output before merging. This process is crucial for managing complexity. (9:11)

## Practical implications

- When designing agentic workflows, do not rely solely on cost metrics; evaluate reliability and context retention, especially for multi-step tasks.
- Implement a tiered model strategy: use the most powerful model for defining the scope and validating the plan, and use a cheaper, capable model for the bulk code generation.
- Focus on process engineering (rules, reviews, playtesting) to mitigate the inherent weaknesses of any single LLM, regardless of its open or proprietary status.

## Topics

Agentic Workflow Design, LLM Architecture, Build Engineering, Cost Optimization, Open-Source AI, AI Software Factory, Dynamous Agentic Coding Course 2.0

Source: https://www.youtube.com/watch?v=RYIV5E2Xq60
