# Everyone Uses AI Agents. So Where Are the Agent Builders?

## Executive summary

The conversation critiques the current hype surrounding 'agents in production,' arguing that much of the perceived progress involves using vendor-provided agents (like Claude Code or Codex) rather than building proprietary, homegrown platforms. The primary technical challenge identified is achieving continuous, cross-application context, which is necessary for agents to operate autonomously and reliably. Furthermore, the discussion highlights the shift in AI strategy, noting that the push for agentic AI often originates from corporate boardrooms (strategic mandate) rather than specific, solvable engineering problems. Key architectural concepts discussed include Agent-to-Agent (A2A) protocols and the necessity of robust data integration (RAG) to handle personal, siloed data.

## Key takeaways

- The Agent Push is Strategic, Not Problem-Driven: The push for agentic AI is often driven by corporate leadership and 'AI or agent strategy' (boardrooms) rather than by a clear, immediate operational problem that needs solving. This contrasts with historical tech shifts like microservices, which were driven by scaling needs.
- Vendor Agents vs. Homegrown Agents: Many companies are currently leveraging vendor-provided agents (e.g., Claude Code, Codex) in production workflows. The true 'agent builders'—those deploying proprietary, scaled agents—are harder to find, suggesting a reliance on integration rather than pure internal development.
- The Criticality of Context Management: The biggest technical hurdle is maintaining continuous context across disparate systems (Slack, email, WhatsApp, etc.). Current personal agents struggle to synthesize information from multiple, non-linear sources, leading to context fragmentation.
- Gartner's Warning on Vendor Reliance: Gartner predicts that 70% of enterprises will abandon agentic AI built by vendor forward deployed engineering by 2028. This suggests a move toward greater internal ownership and control over agentic solutions.

## Technical details

- Data Retrieval and Context: Advanced personal agents can be built using open-source 'agent librarian' systems that download embedding models to chunk and process massive amounts of personal data (e.g., 7,000+ Slack DMs, LinkedIn data). This creates a high-quality Retrieval-Augmented Generation (RAG) system that goes beyond traditional RAG by cross-referencing multiple data types to confirm facts.
- Agent Protocols and Architecture: The discussion differentiates between a 'tool' (which should be deterministic) and an 'agent' (which should not be). The need for protocols like MCP (Messaging/Context Protocol) and A2A (Agent-to-Agent) is highlighted to enable agents to interact with remote, third-party systems and services.
- Delegation and Autonomy: The concept of agent autonomy is framed by managerial delegation levels: Task (fix this bug), Problem (solve this bug), Outcome (make the software fault-tolerant), and Responsibility (ensure the software meets client needs). The highest level, Responsibility, is the most difficult for current agents to achieve.
- Software Development Risk Management: In AI coding, the process of reaching a 'good enough' state is difficult. The discussion emphasizes that software development, whether for AI or human teams, requires taking informed risks and considering system-level trade-offs (e.g., stability vs. security) rather than just optimizing the code itself.

## Practical implications

- Focus on building integration layers (harnesses) that connect existing vendor agents and internal data sources, rather than attempting to build monolithic, proprietary agents from scratch.
- Prioritize developing robust context management systems that can ingest and synthesize data from multiple, non-linear sources (Slack, email, CRM, etc.) to achieve true operational continuity.
- When evaluating AI solutions, assess the level of ownership and exit strategy required, recognizing that relying solely on third-party 'forward deployed engineering' may not lead to long-term operational control.
- For build engineers, the focus should shift from building the agent itself to building the secure, governed pipelines that allow agents to interact with and act upon proprietary, contextual data.

## Topics

AI Agents, Build Engineering, MLOps, Context Management, Software Architecture, Generative AI, Claude Code / Codex, Netlify, Ramp, MCP (Messaging/Context Protocol), A2A (Agent-to-Agent)

Source: https://www.youtube.com/watch?v=qVI74TCqf0g
