MLOps Community

Cut One AI Cost and Two More Grow Back

Published 2026-11-05 · Duration 41:44

Summary

The total cost of ownership (TCO) for AI systems, especially agent-based applications, quickly exceeds initial estimates when scaling to thousands of users. The discussion emphasizes that relying solely on external cloud APIs (like GCP or OpenAI) is unsustainable. To achieve enterprise scale, organizations must shift focus to controlling the context, implementing basic model routers, and building robust internal evaluation suites (Evals). Furthermore, fine-tuning smaller models for repetitive tasks (like data analytics) offers significant cost and latency improvements, while building internal stack capabilities mitigates vendor lock-in and builds valuable engineering muscle.

Download summary

Key takeaways

  1. Scaling AI requires controlling the stack, not just calling APIs. 2:00

    Relying purely on external cloud APIs does not scale to 5,000–10,000 person companies because the consumption model is highly expensive. The moment an agent harness is implemented, the cost of tokens becomes significant, especially when connecting multiple data sources (Slack, Google Doc, Notion, etc.).

  2. Evals are critical for professionalizing AI engineering. 17:26

    Building good Evals allows teams to benchmark all models against different tasks, enabling the selection of the optimal model for a specific use case, rather than defaulting to the most expensive model.

  3. Fine-tuning small models drastically improves efficiency. 27:40

    For repetitive tasks like data analytics, fine-tuning smaller models can lead to performance improvements (e.g., 20-30x better performance) at a fraction of the cost and latency, making the solution smaller, faster, and cheaper.

  4. The value of optionality requires building internal capabilities. 35:00

    To avoid vendor lock-in (e.g., committing entirely to OpenAI or Claude), organizations must build internal capabilities. This 'optionality' is a key advantage of owning a large part of the stack, allowing for quick pivots.

More digests on AI Total Cost of Ownership (TCO)

Technical details

  • Agent Architecture & Cost Management 120s

    Agent harnesses and sandboxes, while powerful for integrating multiple services (Slack, email, Notion), are extremely token-greedy. Cost control is achieved by implementing a basic model router and minimizing context size, rather than relying on the most advanced model.

  • Model Optimization and Training 1660s

    For specific, repetitive workflows (e.g., data analytics), fine-tuning smaller models is recommended. This approach yields smaller, faster, and cheaper models compared to general-purpose APIs.

  • System Design and Hiring 2300s

    AI engineering roles require more than just knowing how to use frameworks (like LangChain) or titles. Candidates must demonstrate strong software engineering design thinking (whiteboard design systems), an understanding of infra/platform (SRE experience), and the ability to recognize trade-offs between manual development and agent delegation.

Mentioned resources

Channel & topics

Watch on YouTube · Back to latest

This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.