# Your Agent Doesn't Need an LLM for Every Decision - Raymond Weitekamp, OpenProse

## Executive summary

The talk introduces Recursive Decision Models (RDMs) as a paradigm shift for building efficient AI agents, arguing that complex reasoning tasks can be decomposed into many small, fast decisions rather than relying solely on large, generative language models (LLMs). RDMs, which are inspired by the concept of System One (fast, intuitive judgments), allow a system to recursively decide whether to call a sub-decision or an LLM. The ultimate goal is 'Operating System One' (oSo), a computer-used agent designed to control devices and perform tasks without requiring physical interaction (keyboard/mouse), starting with voice input.

## Key takeaways

- Recursive Decision Models (RDMs): RDMs are a concept where a decision model can decide to make another decision, allowing for a recursive structure. This contrasts with generative LLMs, which are prone to divergence, by focusing on convergence and structured choices.
- The Yin and Yang of AI: The core strength of decision models (like Jev) is that they can only select among options, making them a counterbalancing, or 'yin,' force to the open-ended, free-wheeling text generation of LLMs ('yang').
- Operating System One (oSo): oSo is the vision for a computer-used agent built on RDMs, aiming to decouple human physicality from the computer. The initial focus is on radical human-computer interfacing, starting with voice as the primary input.

## Technical details

- RDM Architecture: The RDM architecture is defined by three rules: 1) The context must stay outside the model (externalized context); 2) The agent can choose to call another decision (recursion); and 3) It must function without any generative model at all. This allows it to compose with LLMs and agents.
- Composition of AI Primitives: RDMs excel at composing multiple distinct systems. Examples include using two separate, coupled decision agents (e.g., one for first-person shooting, one for map exploration) to create a single, complex agent (e.g., a Doom-playing agent).
- The Arbiter Primitive: The Arbiter acts as a gatekeeper when running multiple decision models in parallel. It manages potential state collisions by determining which action to allow through, which is crucial for speculative tool calling and reliable parallel execution.
- Shared Interface (The Map): For an LLM and an RDM to communicate, a structured shared interface, called 'the map,' is required. This map provides pointers (e.g., 'TV') with simple descriptions and defined functions (e.g., `lights_on: lambda function`), allowing the LLM to learn and modify the system's capabilities.

## Practical implications

- Building agents that perform complex, multi-step tasks (like navigating a browser or playing a video game) by decomposing the problem into small, verifiable decisions rather than relying on single, massive reasoning calls.
- Creating highly efficient, low-latency interfaces for computer control (oSo) that can operate through voice and other non-physical inputs.
- Developing robust, parallel systems using the Arbiter primitive to manage state changes and prevent collisions when multiple decision agents run concurrently.
- Enabling LLMs to interact with structured systems by defining a shared, programmatic interface ('the map'), thereby grounding generative output.

## Topics

AI Agents, Decision Theory, Human-Computer Interaction, Machine Learning Architecture, Recursive Systems, Raymond Weitekamp, OpenProse, Recursive Decision Models, rdm GitHub Repo, oSo Website

Source: https://www.youtube.com/watch?v=hFNbPmzzcEs
