# Your Sub-Agents Are a Distributed System. Treat Them Like One. - Girish Konda, Microsoft

## Executive summary

The talk argues that when an orchestrator delegates tasks to multiple sub-agents running concurrently, the system functions as a classic distributed system. Failures encountered (e.g., partial success reports, dirty reads, lost replies) are not unique to AI agents. Girish Konda proposes the 'SAFE handoff' consistency model to manage these failures, ensuring that every fact, decision, and side effect is traceable, bounded, and explicitly stated, thereby treating every agent handoff as a reliable network call.

## Key takeaways

- Treat every handoff as a network call: Sub-agents must be treated like components in a distributed system, acknowledging that calls can fail halfway, be slow, or return duplicate results. The orchestrator must read a status, not just text, to determine the true state.
- Adopt the SAFE Consistency Model: The proposed SAFE model enforces strict rules: Source Feeds (S), At Most Once effects (A), Fanout Budgets (F), Explicit Outcomes (E), and Unknown timeouts (U). This model must be enforced in code, not just relied upon in text output.
- Enforce Source Fact Provenance: Every fact must carry its source, version, and scope (provenance) to separate an observation (fact with source/read time) from an inference (unsupported claim).

## Technical details

- Distributed System Failures in Agents: Classic failures include partial failure (returning success text when work failed halfway), dirty reads (reading shared context guesses), unbounded fanout (rate limits the orchestrator never sees), and retries (running side effects twice without idempotency).
- Dirty Reads and Quorums: A dirty read occurs when a sub-agent reads a value before the writer commits. While SQL isolation levels exist, the shared context in agents is inherently unsafe. The traditional fix of using read/write quorums (Read Quorum + Write Quorum > Total) does not apply because agents read the same guess multiple times, not independent copies.
- SAFE Hand-off Model Components: S (Source Feeds): Facts must be treated as records with provenance (source, version, scope, read time). A (At Most Once): Requires idempotency and identity for every write/side effect. F (Fanout Budgets): Implements flow control and bounded weights for downstream calls. E (Explicit Outcomes): Every handoff must end in a named state (e.g., 'stale', 'unknown'), not just text.
- Consistency Model Comparison: The model rejects strong consistency guarantees (Linearizable, Sequential, Causal) because they cannot maintain availability during network splits (CAP theorem). The SAFE model prioritizes consistency over availability for writes, refusing rather than guessing.

## Practical implications

- Implement explicit state management (e.g., 'stale', 'unknown') instead of relying on text output to signal failure or uncertainty.
- Use fencing tokens (like a version number) for side effects to ensure that a resumed task does not overwrite a newer state.
- Enforce rate limiting and flow control (Fanout Budgets) at the orchestrator level to prevent cascading failures from concurrent sub-agents.

## Topics

Distributed Systems, AI Agents, Consistency Models, Concurrency, Reliability Engineering, MCP Pacemaker, FLP paper (1985), Chandra and Toueg (1996)

Source: https://www.youtube.com/watch?v=DxP5aW4VknQ
