AI Engineer

Your Agent Doesn't Need an LLM for Every Decision - Raymond Weitekamp, OpenProse

Published 2026-11-10 · Duration 51:48

Summary

The talk introduces Recursive Decision Models (RDMs) as a paradigm shift for building efficient AI agents, arguing that complex reasoning tasks can be decomposed into many small, fast decisions rather than relying solely on large, generative language models (LLMs). RDMs, which are inspired by the concept of System One (fast, intuitive judgments), allow a system to recursively decide whether to call a sub-decision or an LLM. The ultimate goal is 'Operating System One' (oSo), a computer-used agent designed to control devices and perform tasks without requiring physical interaction (keyboard/mouse), starting with voice input.

Download summary

Key takeaways

  1. Recursive Decision Models (RDMs) 1:58

    RDMs are a concept where a decision model can decide to make another decision, allowing for a recursive structure. This contrasts with generative LLMs, which are prone to divergence, by focusing on convergence and structured choices.

  2. The Yin and Yang of AI 3:08

    The core strength of decision models (like Jev) is that they can only select among options, making them a counterbalancing, or 'yin,' force to the open-ended, free-wheeling text generation of LLMs ('yang').

  3. Operating System One (oSo) 17:25

    oSo is the vision for a computer-used agent built on RDMs, aiming to decouple human physicality from the computer. The initial focus is on radical human-computer interfacing, starting with voice as the primary input.

More digests on AI Agents

Technical details

  • RDM Architecture 340s

    The RDM architecture is defined by three rules: 1) The context must stay outside the model (externalized context); 2) The agent can choose to call another decision (recursion); and 3) It must function without any generative model at all. This allows it to compose with LLMs and agents.

  • Composition of AI Primitives 430s

    RDMs excel at composing multiple distinct systems. Examples include using two separate, coupled decision agents (e.g., one for first-person shooting, one for map exploration) to create a single, complex agent (e.g., a Doom-playing agent).

  • The Arbiter Primitive 840s

    The Arbiter acts as a gatekeeper when running multiple decision models in parallel. It manages potential state collisions by determining which action to allow through, which is crucial for speculative tool calling and reliable parallel execution.

  • Shared Interface (The Map) 950s

    For an LLM and an RDM to communicate, a structured shared interface, called 'the map,' is required. This map provides pointers (e.g., 'TV') with simple descriptions and defined functions (e.g., `lights_on: lambda function`), allowing the LLM to learn and modify the system's capabilities.

Mentioned resources

Channel & topics

Watch on YouTube · Back to latest

This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.