AI Engineer

Your Agent Hates Your CLI - Ayush Bhardwaj, Allos AI

Published 2026-11-10 · Duration 12:20

Summary

The talk analyzes how AI agents interact with CLIs, revealing that agents often work around tool friction (e.g., output truncation, incorrect exit codes) without filing bug reports. The core finding is that agent transcripts are invaluable product analytics, revealing patterns in tool usage and failure modes. The speaker advocates for designing CLIs and skills that 'fail loudly,' provide compact results, and allow agents to specify the exact data slice needed, rather than relying on generic piping.

Download summary

Key takeaways

  1. Agent Transcripts are Product Analytics 1:42

    Instead of waiting for bug reports, analyze agent's own historical calls (e.g., 20,000 calls) to find patterns, common workarounds, and areas where tools are ambiguous or fail silently. The agent's subsequent commands reveal what information was missing or misinterpreted.

  2. Design for Failure and Clarity 5:30

    Tools must 'fail loudly' when an error occurs, ensuring the error message reaches the model and is not cut off by piping. The output should include a short summary and a pointer to the full log, even if the primary output is truncated.

  3. Improve Tool Interfaces with Skills 11:20

    The skill file, help text, and error messages are the agent's primary interface. Stale documentation or flags that do nothing (a 'no-op') are considered bugs because agents will follow them literally.

  4. Measure Useful Outcomes, Not Just Tokens 11:50

    When evaluating tool improvements, focus on measuring reduced turns and improved answer accuracy, not just token count or runtime. Cost and runtime improvements may not correlate with better workflow efficiency.

More digests on AI Agents

Technical details

  • Output Truncation and Piping 180s

    Agents frequently use filters like `head`, `tail`, `grep`, and `head` to manage large outputs. However, piping can hide critical information or allow a command to report success (green dot) even if the underlying test failed (e.g., piping `cargo test` into `head`).

  • Compact and Targeted Output 432s

    Improvements include allowing agents to request the message *around* a search match (instead of the whole page), offering compact search results (e.g., one line per hit), and implementing selection flags to tell the tool exactly which part of the output is needed.

  • Batch and Multi-Session Context 580s

    Instead of manually looping over multiple sessions, agents should be able to ask several questions about several sessions in a single call (e.g., using a dedicated `inspect` command).

  • CLI Tooling Examples 120s

    The speaker's custom CLI allows agents to search their own past sessions across tools like Cloud Code, Codex, and Cursor, providing a mechanism for context reuse.

Mentioned resources

Channel & topics

Watch on YouTube · Back to latest

This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.