# Train, Simulate, and Serve World Models with NVIDIA Cosmos Framework and NIM | Cosmos Labs

## Executive summary

This technical deep dive introduces the NVIDIA Cosmos Framework and Cosmos NIM, providing a comprehensive pipeline for developing, simulating, and deploying world models. The framework enables developers to train and customize foundation models (like Cosmos 3) using advanced techniques such as distillation and reinforcement learning. For deployment, Cosmos NIM offers highly optimized, containerized microservices, featuring advanced capabilities like speculative decoding and live video inference, significantly improving real-world performance and reducing latency.

## Key takeaways

- Cosmos Framework for Model Customization: The open-source Cosmos Framework provides recipes for post-training, including Domain Adaptation, Teacher Forging, Consistency Distillation, and Self-Forcing. This four-stage process transforms the initial bidirectional Cosmos 3 model into a causal, fast-sampling model suitable for interactive simulation.
- Simulation Capabilities (Auto & Manual): The platform supports specialized simulation checkpoints for autonomous driving (Transfer Auto, Sim Auto) and robotics (Sim By Manual). These models generate multi-view, multi-modal video streams, enabling long-term, closed-loop simulation for policy training.
- Optimized Deployment via Cosmos NIM: Cosmos NIM is an optimized, containerized inference microservice that simplifies deployment by handling drivers and library mismatches. New features include speculative decoding (up to 2-3x faster token generation) and live video inference (sub-second alerting), crucial for real-time applications.
- Agentic Workflow Integration: The Cosmos Framework is made agentic, allowing developers to point coding agents to the repository. The agent can then autonomously execute complex tasks, such as running the full training pipeline (setup, download, pre-flight test, training, scoring) with a single prompt.

## Technical details

- World Model Architecture: The Cosmos family includes three versions: Cosmos 3 Super (64B parameters, for data center GPUs), Cosmos 3 Nano (16M parameters, good balance of performance/size), and Cosmos 3 Edge (4B parameters, for smaller edge devices).
- Simulation Inputs: The Transfer Auto model generates realistic driving videos using inputs including World Scenario Maps (WSM), which provide bounding boxes, traffic lanes, text prompts (weather, time of day), and spatial controls (depth, segmentation, edges, blur).
- NIM Deployment Flags: To enable advanced inference features when deploying Cosmos NIM, specific flags must be used: `--use_dflash` for speculative decoding and `--enable_streaming` for live video inference.

## Practical implications

- Accelerates the entire AI lifecycle from data preparation (data factory) to model training (Cosmos Framework) and production deployment (Cosmos NIM).
- Reduces the complexity of deploying large models by containerizing the environment and optimizing inference at the microservice level.
- Enables complex, long-term closed-loop simulations for robotics and autonomous vehicles by providing interactive, causal model generation.
- The agentic capabilities allow build engineers to automate and standardize the execution of complex, multi-step training pipelines.

## Topics

World Models, AI, Robotics, Simulation, Machine Learning, Edge Computing, Containerization, NVIDIA Cosmos Framework, NVIDIA NGC Platform

Source: https://www.youtube.com/watch?v=8lNoGg_j8S8
