Stanford Online

AI Costs: Tokens, Energy, Model Choice & API Keys | Stanford CS193T Lecture 7 with Cynthia Bailey

Published 2026-11-19 · Duration 49:00

Summary

This lecture provides a deep dive into the operational costs and resource consumption of Large Language Models (LLMs). It analyzes token usage (input vs. output), the efficiency gains from caching, and the critical role of model selection (e.g., Claude Opus 5.5 vs. Claude Haiku) in multi-step agentic workflows. The discussion also covers the technical mechanics of billing via API keys and the security risks associated with key leakage in web applications.

Download summary

Key takeaways

  1. Caching significantly reduces input token costs. 17:13

    Because the attention mechanism calculation is expensive, precomputing and reusing results via cache reads can provide a discount of up to 90% compared to raw input tokens.

  2. Model choice must be balanced against latency. 23:07

    Bigger models are not always better; latency (the time to process results) compounds significantly in multi-agent workflows, making thoughtful model assignment crucial.

  3. Unused skills still incur token overhead. 14:09

    Every enabled skill places its text into the context window on each prompt, meaning unused skills still contribute to input token costs, necessitating agent decluttering.

  4. API keys are essential for distributed cost sharing. 29:51

    API keys allow a computer program to act as a user, enabling systems to charge individual users for their usage, which is critical for sustainable tool deployment.

More digests on AI Economics

Technical details

  • Token Usage Metrics 380s

    Input tokens include the system prompt, settings, and enabled tool connectors. Output tokens include the LLM's response and any internal reasoning/chain-of-thought generation.

  • Duration Metrics 465s

    API duration measures the time spent waiting for the model (including network transit and data center load), while wall-clock time is the total elapsed time since the session began.

  • Agent Architecture and Guardrails 1150s

    Complex systems, like the Wayfinder AI tutor, require multiple agents (Input Guardrail, Response Generation, Output Guardrail) to prevent hallucination and ensure compliance, which increases latency.

  • API Key Security Pitfalls 1650s

    A common security vulnerability is encoding an API key into public-facing website code (e.g., HTML), making it visible to anyone who views the page source.

Mentioned resources

  • Claude Code (LLM/Tool)
  • Claude Opus 5.5 (LLM)
  • Claude Haiku (LLM)
  • Hopscotch (Resource Management Tool)
  • Open Router (Resource Management Tool)

Channel & topics

Watch on YouTube · Back to latest

This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.