Latent Space

Why FLUX 3 Trains on Images, Video and Sound — Andreas Blattmann & Dustin Podell, Black Forest Labs

Published 2026-11-16 · Duration 1:38:24

Summary

This technical deep dive explores the evolution of generative AI, focusing on Black Forest Labs' (BFL) FLUX 3 model. The discussion highlights the shift from image-only generation to comprehensive multimodal world modeling, integrating images, video, audio, and action prediction. Key technical advancements include FLUX 3's ability to learn general world physics, the architectural improvements of SDXL (multi-aspect training, refiners), and the comparison between autoregressive and diffusion models. The central thesis is that true general intelligence requires modeling the physical world, not just pixels, leading to the development of 'world action models' for robotics and real-world applications.

Download summary

Key takeaways

  1. FLUX 3 is a Multimodal World Model 28:24

    FLUX 3 is designed as a single, general model trained on images, video, and audio. This allows it to build an inherent understanding of the world's physics, enabling tasks like predicting actions and understanding 3D space from 2D video data, which is crucial for robotics.

  2. The Shift to World Action Models 20:05

    BFL's core thesis is that video modeling is the central means to understanding how the world works. The goal is to move beyond simple image generation to 'world action models' that predict physical states and actions, enabling applications like autonomous robotics.

  3. Architectural Advancements in Diffusion 18:51

    The evolution of models like SDXL involved pioneering techniques such as large-scale multi-aspect training and the use of separate 'refiner' models to improve high-frequency detail, demonstrating that complex capabilities can be achieved through specialized pipelines.

  4. Autoregressive vs. Diffusion Models 24:10

    While autoregressive models are noted as being more data-efficient during training (as they can utilize all tokens in parallel), diffusion models offer advantages in parallelization and speed, making the debate over the 'best' approach unsettled and highly active.

More digests on Generative AI

Technical details

  • Latent Diffusion & Flow Matching 1050s

    The latent diffusion algorithm was foundational (co-invented by Andy Blattmann). A major algorithmic leap was the adoption of Flow Matching (Rectified Flow), which simplifies the mathematical complexity of translating between two probability distributions, making the process mathematically easier and improving scalability compared to classic diffusion.

  • FLUX Kontext (Image Editing) 1320s

    This model focuses on general image editing by leveraging the deep spatial relationships learned during training. It was developed by training on a relatively small, high-quality dataset of simple labeled relationships, demonstrating that understanding pre-built spatial context is key to advanced editing.

  • World Modeling and Action Prediction 1450s

    The FLUX Action model is trained by pre-training on multimodal data (images, video, audio) and then adding action prediction data. It predicts the next chunk of actions by conditioning on the previous video and an instruction, enabling the model to reason about physical tasks.

  • Training Data and Scaling Laws 1300s

    The speakers emphasized that data is the most critical lever. There is a continuous trade-off between data quantity and data quality. Furthermore, the ability to derive multiple specialized models (e.g., upscalers, video editors) from a single, general multimodal model demonstrates the compounding power of general pre-training.

Mentioned resources

Channel & topics

Watch on YouTube · Back to latest

This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.