# How to Get Frontier Coding Results at 70% Lower Cost - Nouha Dziri, Cohere

## Executive summary

This presentation details methods for optimizing multi-agent coding teams to achieve 'frontier coding' results while drastically reducing computational costs. The core finding is that while teams of agents significantly outperform solo agents, the cost increase is substantial (4 to 7 times more). Optimization requires treating team design as a compute allocation problem, identifying the task's bottleneck stage (e.g., planning in ProgramBench), and strategically allocating more agents to that specific stage. Furthermore, using a strong lead model (e.g., Sol) combined with cheaper worker models (e.g., Luna) can achieve near-frontier performance at a fraction of the cost.

## Key takeaways

- Strategic Resource Allocation is Key: The performance gain is highly dependent on where agents are deployed. The extra agent pays off most in the task's bottleneck stage. For ProgramBench, the bottleneck was planning, meaning allocating more agents to planning provided significant performance gains at half the cost compared to the solo agent.
- Lead Strength Sets the Ceiling, Workers Set the Price: The lead model dictates the upper bound of performance, while the worker models determine the cost. Using a strong lead (e.g., Sol) with cheaper workers (e.g., Luna) can achieve near-frontier results at a much lower cost.
- Hierarchy Over Peer Communication: Agents tend to operate in a hierarchical manner, reporting primarily to the lead (84% of messages went from worker to lead). Direct peer-to-peer communication, while possible, was found to be largely ineffective or distracting, often acting as overhead.

## Technical details

- Agent Team Architecture: The multi-agent harness is designed as a top-down hierarchy with a lead agent that dynamically decides the team size and worker roles. Two lead types were tested: the 'blocking lead' (waits for all workers to report back) and the 'asynchronous lead' (continues working while workers report). The blocking lead was found to be more reliable for task completion.
- Benchmarking and Metrics: Experiments were conducted on ProgramBench (a reverse engineering task starting from a binary executable) and Commit0. The key metric is 'efficiency,' defined as the reward gained per added dollar of inference compute, allowing researchers to pinpoint the most valuable stage (planning, implementation, or verification).
- Scaling and Optimization: The team design was framed as a compute allocation problem. The study demonstrated that scaling agents in the planning stage provided the highest reward-to-cost ratio, suggesting that discovery and understanding the binary's behavior is the primary bottleneck.

## Practical implications

- For build engineering teams, this research suggests that optimizing CI/CD pipelines involving complex, multi-stage tasks (like reverse engineering or complex feature implementation) should focus resources on the initial discovery/planning phase rather than simply adding more workers to implementation or verification.
- When budgeting for AI-driven development tools, prioritizing the quality and capability of the 'lead' or orchestrating agent is more critical than maximizing the number of workers.
- The concept of 'efficiency' (reward per added dollar) provides a quantifiable metric for determining where compute resources yield the highest return in complex software tasks.

## Topics

Multi-Agent Systems, AI Agents, Compute Optimization, Software Engineering, ProgramBench, LLM Orchestration, Commit0, Cohere

Source: https://www.youtube.com/watch?v=maitMe9HCvw
