# Why FLUX 3 Trains on Images, Video and Sound — Andreas Blattmann & Dustin Podell, Black Forest Labs

## Executive summary

This technical deep dive explores the evolution of generative AI, focusing on Black Forest Labs' (BFL) FLUX 3 model. The discussion highlights the shift from image-only generation to comprehensive multimodal world modeling, integrating images, video, audio, and action prediction. Key technical advancements include FLUX 3's ability to learn general world physics, the architectural improvements of SDXL (multi-aspect training, refiners), and the comparison between autoregressive and diffusion models. The central thesis is that true general intelligence requires modeling the physical world, not just pixels, leading to the development of 'world action models' for robotics and real-world applications.

## Key takeaways

- FLUX 3 is a Multimodal World Model: FLUX 3 is designed as a single, general model trained on images, video, and audio. This allows it to build an inherent understanding of the world's physics, enabling tasks like predicting actions and understanding 3D space from 2D video data, which is crucial for robotics.
- The Shift to World Action Models: BFL's core thesis is that video modeling is the central means to understanding how the world works. The goal is to move beyond simple image generation to 'world action models' that predict physical states and actions, enabling applications like autonomous robotics.
- Architectural Advancements in Diffusion: The evolution of models like SDXL involved pioneering techniques such as large-scale multi-aspect training and the use of separate 'refiner' models to improve high-frequency detail, demonstrating that complex capabilities can be achieved through specialized pipelines.
- Autoregressive vs. Diffusion Models: While autoregressive models are noted as being more data-efficient during training (as they can utilize all tokens in parallel), diffusion models offer advantages in parallelization and speed, making the debate over the 'best' approach unsettled and highly active.

## Technical details

- Latent Diffusion & Flow Matching: The latent diffusion algorithm was foundational (co-invented by Andy Blattmann). A major algorithmic leap was the adoption of Flow Matching (Rectified Flow), which simplifies the mathematical complexity of translating between two probability distributions, making the process mathematically easier and improving scalability compared to classic diffusion.
- FLUX Kontext (Image Editing): This model focuses on general image editing by leveraging the deep spatial relationships learned during training. It was developed by training on a relatively small, high-quality dataset of simple labeled relationships, demonstrating that understanding pre-built spatial context is key to advanced editing.
- World Modeling and Action Prediction: The FLUX Action model is trained by pre-training on multimodal data (images, video, audio) and then adding action prediction data. It predicts the next chunk of actions by conditioning on the previous video and an instruction, enabling the model to reason about physical tasks.
- Training Data and Scaling Laws: The speakers emphasized that data is the most critical lever. There is a continuous trade-off between data quantity and data quality. Furthermore, the ability to derive multiple specialized models (e.g., upscalers, video editors) from a single, general multimodal model demonstrates the compounding power of general pre-training.

## Practical implications

- The industry trend is moving toward 'omni-models' that integrate multiple modalities (text, image, video, audio, action) into a single backbone, simplifying complex workflows.
- The development of 'world action models' suggests a future where AI can move beyond simulation and directly assist in real-world physical tasks, such as robotics.
- For developers, the focus should shift from optimizing individual model components to designing robust, general-purpose pipelines that can be easily fine-tuned for specific, high-value industry applications (e.g., manufacturing, medicine).

## Topics

Generative AI, Multimodal AI, World Modeling, Diffusion Models, Robotics, Latent Diffusion, Model Scaling, BFL’s account of why it trained FLUX 3 across images, video and audio, FLUX 3 video generation with synchronized sound, FLUX Kontext (Night-to-day image edit), Latent Space (Podcast)

Source: https://www.youtube.com/watch?v=8cGfJOU3AlQ
