Robots Will Be Trained on Video Games - Pim de Witte, General Intuition
Summary
General Intuition proposes a paradigm shift for robotics AI, arguing that models designed for continuous action in space and time are fundamentally different from turn-based Large Language Models (LLMs). By leveraging Medal's massive video game recording dataset (estimated at over 1.5 trillion action tokens per month), the company trains general foundation models that convert pixels directly into actions. This approach allows a single model to control multiple types of robots (10+ embodiments) using the same action primitives (like game controllers), making it highly scalable for complex, real-world robotics tasks.
Key takeaways
-
Action Space Equivalence
3:47
The action space of video games, often governed by game controllers, is functionally equivalent to the action space required for teleoperating robots in the real world. This allows the use of game data to train general-purpose robotic controllers.
-
Data Scale and Density
10:21
Medal records tens of thousands of video games, generating massive amounts of action data. The sheer volume (over 1.5 trillion action tokens per month) is compared to the data used by other robotics model trainers, highlighting the scale advantage.
-
Continuous vs. Turn-Based Prediction
5:28
Unlike LLMs, which are turn-based (predicting the next token), the proposed models continuously receive frames and predict actions, mimicking the constant observation inherent in human behavior (e.g., Olaf rolling down a mountain).
-
Generalization to Embodiments
15:52
The system is designed to be highly general, allowing one model to control over 10 different types of robots using the same core action primitives (pixels in, actions out).
Technical details
-
Model Architecture (Pixels-to-Actions)
521s
The model takes frames (pixels) as input and outputs actions, which are semantically labeled. This approach bypasses the need for complex motor math or predicting exact motor torque values, instead generalizing human ability to control devices via standard controllers.
-
LLM vs. Action Model Limitations
328s
LLMs are optimized for predicting the next text token in a turn-based manner, making them difficult to generalize to environments involving continuous space and time. Action models are designed for continuous input (frames) and continuous output (actions).
-
World Models and Learning
714s
The approach focuses on world models (similar to those pioneered by Mira), allowing models to learn in the native modality of the human (pixels). This avoids the computational overhead and complexity of creating digital twins or passing data through an LLM for simulation.
-
Action Density and APM
855s
Video games are inherently 'action-dense,' allowing the model to predict high Actions Per Minute (APM). This is crucial for complex robotics tasks, such as operating a drone, where high-frequency adjustments are required.
Mentioned resources
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.