AI Engineer

Your Old Prompts Are Making Claude Worse - Lance Martin, Anthropic

Published 2026-11-06 · Duration 1:04:03

Summary

This talk provides a comprehensive guide for optimizing the use of frontier Large Language Models (LLMs) like Claude, focusing on improving performance and significantly reducing operational costs. Key strategies include auditing prompts to remove outdated anti-patterns (e.g., 'think step-by-step'), maximizing the use of the prompt cache in multi-turn applications, and designing robust, 'climbable' evaluation sets. For complex, long-running systems, the speaker emphasizes the architectural necessity of 'Managed Agents' to ensure reliability and security by decoupling the agent's 'brain' from its 'hands'.

Download summary

Key takeaways

  1. Audit Prompts for Anti-Patterns 5:49

    Frontier models do not require outdated instructions like 'think step-by-step' or 'double-check your work.' These anti-patterns increase cost and can sometimes degrade performance. Use the `checkup prompt audit` or `Claude API prompt audit` commands to automatically remove these unnecessary instructions.

  2. Maximize Prompt Cache Usage 8:50

    In multi-turn applications, the prompt cache is crucial for cost efficiency and speed. Since it is built at a fraction of the base token rate, engineers must structure prompts to maintain a high cache hit rate. The `Claude code` environment offers diagnostics to help optimize prompt structure for caching.

  3. Design Climbable Evaluations 14:05

    An evaluation must be 'climbable' for autonomous improvement. Good design requires low run-run variance, expected performance scaling with model/thinking, and sufficient headroom at the frontier. Use `Claude API build eval` to assist in this process.

  4. Implement Managed Agents for Reliability 33:41

    For long-duration agents (days/weeks of work), reliability and security are paramount. The recommended architecture is 'decoupling the brain from the hands,' where the agent harness (brain) and the sandbox (hands) are separate, recoverable components. Secrets must be managed via vaults, never given direct access to the agent.

More digests on Prompt Engineering

Technical details

  • Prompt Auditing 349s

    The `checkup prompt audit` skill (available in Claude code) automatically identifies and removes anti-patterns (e.g., verification rituals, boosters) that cause frontier models to expend excessive tokens, leading to cost savings and performance improvements.

  • Prompt Cache Mechanics 530s

    The prompt cache is a cost-saving mechanism for multi-turn applications. It operates on a strict prefix match, meaning any byte change in the prefix will invalidate the cache. Tracking the prompt cache hit rate is vital for application economics.

  • Hill Climbing 1147s

    Frontier models are highly effective at autonomously improving performance on a well-structured evaluation. The `Claude API hill climb` skill helps automate this process but requires careful auditing to prevent overfitting to the evaluation distribution, which would render the improvements useless in production.

  • Cost Optimization via Hill Climbing 1630s

    The hill climber can be directed to jointly optimize for cost and performance. By searching through model/effort combinations, it can find configurations (e.g., Opus 5.5 low effort) that achieve parity or better performance while drastically reducing token usage (e.g., achieving a 90% cost reduction).

  • Managed Agents Architecture 2021s

    This architecture separates the agent's core logic (harness/brain) from its execution environment (sandbox/hands). The session log is maintained as an independent, append-only, recoverable object, ensuring state persistence even if the harness or sandbox fails.

Mentioned resources

  • Anthropic Blog (on Managed Agents) (Blog Post)
  • Claude code / Claude API (Development Environment/Skill)

Channel & topics

Watch on YouTube · Back to latest

This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.