# Cut One AI Cost and Two More Grow Back

## Executive summary

The total cost of ownership (TCO) for AI systems, especially agent-based applications, quickly exceeds initial estimates when scaling to thousands of users. The discussion emphasizes that relying solely on external cloud APIs (like GCP or OpenAI) is unsustainable. To achieve enterprise scale, organizations must shift focus to controlling the context, implementing basic model routers, and building robust internal evaluation suites (Evals). Furthermore, fine-tuning smaller models for repetitive tasks (like data analytics) offers significant cost and latency improvements, while building internal stack capabilities mitigates vendor lock-in and builds valuable engineering muscle.

## Key takeaways

- Scaling AI requires controlling the stack, not just calling APIs.: Relying purely on external cloud APIs does not scale to 5,000–10,000 person companies because the consumption model is highly expensive. The moment an agent harness is implemented, the cost of tokens becomes significant, especially when connecting multiple data sources (Slack, Google Doc, Notion, etc.).
- Evals are critical for professionalizing AI engineering.: Building good Evals allows teams to benchmark all models against different tasks, enabling the selection of the optimal model for a specific use case, rather than defaulting to the most expensive model.
- Fine-tuning small models drastically improves efficiency.: For repetitive tasks like data analytics, fine-tuning smaller models can lead to performance improvements (e.g., 20-30x better performance) at a fraction of the cost and latency, making the solution smaller, faster, and cheaper.
- The value of optionality requires building internal capabilities.: To avoid vendor lock-in (e.g., committing entirely to OpenAI or Claude), organizations must build internal capabilities. This 'optionality' is a key advantage of owning a large part of the stack, allowing for quick pivots.

## Technical details

- Agent Architecture & Cost Management: Agent harnesses and sandboxes, while powerful for integrating multiple services (Slack, email, Notion), are extremely token-greedy. Cost control is achieved by implementing a basic model router and minimizing context size, rather than relying on the most advanced model.
- Model Optimization and Training: For specific, repetitive workflows (e.g., data analytics), fine-tuning smaller models is recommended. This approach yields smaller, faster, and cheaper models compared to general-purpose APIs.
- System Design and Hiring: AI engineering roles require more than just knowing how to use frameworks (like LangChain) or titles. Candidates must demonstrate strong software engineering design thinking (whiteboard design systems), an understanding of infra/platform (SRE experience), and the ability to recognize trade-offs between manual development and agent delegation.

## Practical implications

- Prioritize building robust Evals suites to objectively benchmark models and select the right tool for the job.
- Focus on optimizing generalizable, critical business processes (e.g., data analytics, customer support) rather than optimizing single, isolated use cases.
- Design systems to be modular to maintain optionality and avoid deep vendor lock-in to any single cloud provider or LLM vendor.
- When hiring, assess candidates' ability to design scalable solutions and understand the trade-offs between manual engineering and agent-driven workflows.

## Topics

AI Total Cost of Ownership (TCO), Agentic Workflows, MLOps, Model Evaluation (Evals), Vendor Lock-in Mitigation, Software Architecture, Demetrios Brinkmann, Ioannis Zempekakis

Source: https://www.youtube.com/watch?v=_ctB0shWyL4
