# You Can't Tell What's Real Anymore. Here's What Comes Next - Cristóbal Valenzuela, Runway

## Executive summary

The talk details the evolution of AI video generation, moving beyond simple next-frame pixel prediction to creating complex, real-time, interactive 'world models.' By scaling video models and incorporating concepts like object permanence and physics, the technology aims to simulate entire environments and processes, suggesting a future where screens are interactive portals to simulated realities, exemplified by the 'Continuum' concept.

## Key takeaways

- Evolution of AI Video Generation: Runway's early AI film (2019) was limited to 128x128 pixels. Modern state-of-the-art video generation is significantly advanced, demonstrating capabilities that are indistinguishable from real footage.
- Core Mechanism: Next-Frame Prediction: Video models function by predicting the next frame based on previous frames. Scaling this process allows the model to implicitly learn complex physical laws, including object permanence, gravity, and occlusion.
- Shift to World Models and Simulation: The next frontier involves making next-frame prediction real-time and interactive. This leads to 'world interface models' that allow users to manipulate elements within a simulated image or video, even without a pre-existing 3D representation (e.g., turning a car around in a static image).
- The Continuum Vision: The ultimate goal is 'Continuum,' an operating system concept where the screen itself is a real-time, fully interactive video model. This allows users to select any content (e.g., a website section) and turn it into a navigable, interactive portal.
- Visual Thinking for Agents: The technology can be connected to coding agents, enabling 'visual thinking.' This allows agents to process and visualize problems (e.g., creating a 3D model) through an interactive, visual simulation rather than solely via text-based code.

## Technical details

- Video Model Scaling: The core principle relies on scaling laws applied to video, increasing compute power to improve emergent properties and realism in next-frame prediction.
- World Interface Model: A model trained to understand actions and clicks within an interface, allowing real-time interaction with images or videos without requiring a physical 3D engine (like Unity or Unreal).
- Continuum: A proposed operating system concept where the screen content is a real-time, interactive video model. It allows users to create 'portals' from any digital content, making it fully navigable and reactive.
- Visual Thinking: A new paradigm where AI agents can use a visual simulation (Continuum) to think through and solve problems, guiding the process visually rather than just through text prompts.

## Practical implications

- The development of world interface models could revolutionize how digital content is consumed, turning static images or videos into fully interactive, navigable applications.
- The integration of visual thinking into coding agents suggests a future where complex problem-solving and design can be guided through visual simulation rather than purely textual commands.
- The 'Continuum' concept implies a fundamental shift in operating systems, where the screen itself becomes a dynamic, generative, and interactive simulation engine.

## Topics

AI Video Generation, World Models, Next-Frame Prediction, Simulation, Generative AI, Operating Systems, RunwayML, AIE CODE summit

Source: https://www.youtube.com/watch?v=q4ITwvlFplQ