# Why AI Makes You 100x Faster but Not Your Company - Mitchell Troyanovsky, Basis

## Executive summary

While individual engineers are experiencing massive productivity gains using AI tools (e.g., Co-pilot, ChatGPT), enterprise-level productivity has not matched this increase. Mitchell Troyanovsky argues that for AI agents to scale reliably in production environments, they must be simple and highly opinionated. He details that traditional outcome-based evaluations are insufficient for guaranteeing generalization. Instead, he proposes using 'behavior specs' and advanced diagnostic tools like 'forking' and 'sub-agent simulations' to ensure agents follow consistent, reliable processes, thereby building organizational trust.

## Key takeaways

- The Productivity Gap: Individual productivity gains from AI are high (e.g., 100x), but average enterprise productivity has not scaled accordingly, suggesting a systemic issue in agent deployment and reliability.
- The Need for Process: Organizations scale using processes, not relying on the deterministic nature of people. Agents must replicate this reliability by adhering to defined processes to ensure consistent outcomes.
- The Problem with Current Agents: Current 'coding-style agent products' suffer from a 'competency gap' (outsourcing complexity to the user) and a 'process gap' (allowing the user to customize agent behavior, which conflicts with organizational scaling).
- The Solution: Opinionated Agents: To achieve simplicity and reliability, agents must be 'opinionated'—meaning they follow a consistent, defined process (e.g., always reading the primary source document first) rather than having infinite, unpredictable possibilities.
- Advanced Evaluation Techniques: To validate agent behavior, focus on 'behavior specs' rather than just outcome-based evals. Techniques like 'forking' and 'sub-agent simulations' provide crucial diagnostic signal to understand *why* a failure occurred.

## Technical details

- Behavior Specs: A 'behavior spec' acts as a product specification for the agent era, defining the core intent and most important opinions. It allows developers to supervise the agent's behavior (the process) rather than just testing the final outcome, providing a scalable signal for production systems.
- Outcome vs. Behavior Evals: Outcome-based evals (which assume a single 'right answer') cannot generalize to the hundreds of thousands of situations encountered in production. Behavior specs allow for supervising the process itself, which is more scalable than ground-truth outcome testing.
- Forking for Diagnosis: Forking is a method used to diagnose why an agent failed a behavior evaluation. By forking the agent's state immediately before the failure point, developers can ask the agent 'why did you do that?' or update the context to improve the behavior.
- Sub-agent Simulations: This technique involves asking a coding agent to create a 'sub-agent' that mimics the production agent's behavior. This is a cheaper and faster alternative to forking an entire environment, allowing for easy iteration and testing of context tweaks.

## Practical implications

- When building production agents, focus on defining explicit, opinionated behaviors (e.g., 'always read the primary source first') rather than relying solely on system prompts or outcome metrics.
- Implement diagnostic tooling like forking and sub-agent simulations into the development lifecycle to understand the root cause of agent failures.
- Treat the agent's runtime context as 'runtime training data,' meaning every token must be understood to govern behavior, not just the initial prompt.
- For enterprise adoption, prioritize reliability and consistency over achieving perfect, genius-level performance.

## Topics

AI Agents, LLM Reliability, Behavior Specification, Process Engineering, Productivity Scaling, Software Testing, AIE CODE summit

Source: https://www.youtube.com/watch?v=JGtkTjn4120