# Your Agent Hates Your CLI - Ayush Bhardwaj, Allos AI

## Executive summary

The talk analyzes how AI agents interact with CLIs, revealing that agents often work around tool friction (e.g., output truncation, incorrect exit codes) without filing bug reports. The core finding is that agent transcripts are invaluable product analytics, revealing patterns in tool usage and failure modes. The speaker advocates for designing CLIs and skills that 'fail loudly,' provide compact results, and allow agents to specify the exact data slice needed, rather than relying on generic piping.

## Key takeaways

- Agent Transcripts are Product Analytics: Instead of waiting for bug reports, analyze agent's own historical calls (e.g., 20,000 calls) to find patterns, common workarounds, and areas where tools are ambiguous or fail silently. The agent's subsequent commands reveal what information was missing or misinterpreted.
- Design for Failure and Clarity: Tools must 'fail loudly' when an error occurs, ensuring the error message reaches the model and is not cut off by piping. The output should include a short summary and a pointer to the full log, even if the primary output is truncated.
- Improve Tool Interfaces with Skills: The skill file, help text, and error messages are the agent's primary interface. Stale documentation or flags that do nothing (a 'no-op') are considered bugs because agents will follow them literally.
- Measure Useful Outcomes, Not Just Tokens: When evaluating tool improvements, focus on measuring reduced turns and improved answer accuracy, not just token count or runtime. Cost and runtime improvements may not correlate with better workflow efficiency.

## Technical details

- Output Truncation and Piping: Agents frequently use filters like `head`, `tail`, `grep`, and `head` to manage large outputs. However, piping can hide critical information or allow a command to report success (green dot) even if the underlying test failed (e.g., piping `cargo test` into `head`).
- Compact and Targeted Output: Improvements include allowing agents to request the message *around* a search match (instead of the whole page), offering compact search results (e.g., one line per hit), and implementing selection flags to tell the tool exactly which part of the output is needed.
- Batch and Multi-Session Context: Instead of manually looping over multiple sessions, agents should be able to ask several questions about several sessions in a single call (e.g., using a dedicated `inspect` command).
- CLI Tooling Examples: The speaker's custom CLI allows agents to search their own past sessions across tools like Cloud Code, Codex, and Cursor, providing a mechanism for context reuse.

## Practical implications

- When building tools for agents, treat the agent's interaction logs (transcripts) as the primary source of product analytics, not just the intended use cases.
- Implement explicit flags and skills that guide the agent to provide compact, relevant output slices rather than dumping entire JSON reports or log files.
- For CI/CD integration, ensure that failure states are always visible to the agent, even if the output is piped or filtered.
- Use agent logs to generate replay tests: run the recorded commands through both the old and new versions of the tool to check for regressions or missing information.

## Topics

AI Agents, CLI Design, Product Analytics, Build Engineering, LLM Tooling, CI/CD, chat-history CLI, GitHub CLI

Source: https://www.youtube.com/watch?v=rS36lYuvXQ8
