Your Agent Hates Your CLI - Ayush Bhardwaj, Allos AI
Summary
The talk analyzes how AI agents interact with CLIs, revealing that agents often work around tool friction (e.g., output truncation, incorrect exit codes) without filing bug reports. The core finding is that agent transcripts are invaluable product analytics, revealing patterns in tool usage and failure modes. The speaker advocates for designing CLIs and skills that 'fail loudly,' provide compact results, and allow agents to specify the exact data slice needed, rather than relying on generic piping.
Key takeaways
-
Agent Transcripts are Product Analytics
1:42
Instead of waiting for bug reports, analyze agent's own historical calls (e.g., 20,000 calls) to find patterns, common workarounds, and areas where tools are ambiguous or fail silently. The agent's subsequent commands reveal what information was missing or misinterpreted.
-
Design for Failure and Clarity
5:30
Tools must 'fail loudly' when an error occurs, ensuring the error message reaches the model and is not cut off by piping. The output should include a short summary and a pointer to the full log, even if the primary output is truncated.
-
Improve Tool Interfaces with Skills
11:20
The skill file, help text, and error messages are the agent's primary interface. Stale documentation or flags that do nothing (a 'no-op') are considered bugs because agents will follow them literally.
-
Measure Useful Outcomes, Not Just Tokens
11:50
When evaluating tool improvements, focus on measuring reduced turns and improved answer accuracy, not just token count or runtime. Cost and runtime improvements may not correlate with better workflow efficiency.
Technical details
-
Output Truncation and Piping
180s
Agents frequently use filters like `head`, `tail`, `grep`, and `head` to manage large outputs. However, piping can hide critical information or allow a command to report success (green dot) even if the underlying test failed (e.g., piping `cargo test` into `head`).
-
Compact and Targeted Output
432s
Improvements include allowing agents to request the message *around* a search match (instead of the whole page), offering compact search results (e.g., one line per hit), and implementing selection flags to tell the tool exactly which part of the output is needed.
-
Batch and Multi-Session Context
580s
Instead of manually looping over multiple sessions, agents should be able to ask several questions about several sessions in a single call (e.g., using a dedicated `inspect` command).
-
CLI Tooling Examples
120s
The speaker's custom CLI allows agents to search their own past sessions across tools like Cloud Code, Codex, and Cursor, providing a mechanism for context reuse.
Mentioned resources
- chat-history CLI
- GitHub CLI
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.