# From Human Demonstrations to Robot Training Data With RLWRLD and Reactor | Cosmos Labs

## Executive summary

This session details a comprehensive pipeline for converting human video demonstrations into high-quality, robot-ready training data, leveraging NVIDIA Cosmos 3 and RLWRLD. The process moves from synchronized human video capture to robot action generation, utilizing advanced hand tracking (Reand) and real-time world model inference via Reactor. Key innovations include the ability to synthesize observations for complex, deformable objects (like garments) without explicit 3D object scanning, and the demonstration of low-latency policy evaluation in a cloud environment.

## Key takeaways

- Human Data Scales Faster than Teleoperation: Leveraging human demonstrations allows for significantly faster data collection compared to teleoperation, which requires a robot and control interface. A single human demonstration can be converted into episodes for various robot embodiments (1:25:00).
- Cosmos 3 Enables Deformable Object Synthesis: For tasks like garment packing, Cosmos 3 can generate the movement of soft objects (clothes, tissue paper) following the robot hands, bypassing the need for difficult 3D object scanning or mesh modeling (1:40:00).
- Real-Time World Model Inference via Reactor: Reactor provides a global, low-latency infrastructure (50 milliseconds worldwide) to serve and deploy models like Cosmos 3 Nano Sim Bimanual, enabling real-time policy evaluation and synthetic data generation (0:35:00).

## Technical details

- RLWRLD Hand Tracking (Reand): Reand is a state-of-the-art hand tracker built on the Cosmos 3 encoder that processes egocentric video to predict not only 3D hand poses but also technical information, including contact and force estimation, without requiring specialized hardware (0:30:00).
- The Human-to-Robot Pipeline: The process involves four steps: 1) Human video capture (calibrated multi-view/egocentric cameras); 2) 3D dynamics reconstruction (using Reand to extract pose and tactile info); 3) Robot action generation (retargeting the reconstructed human motion); and 4) Observation synthesis (using Isaac Sim for rigid objects and Cosmos 3 for complex objects) (2:00:00).
- Cosmos 3 Architecture and Usage: Cosmos 3 is an omni-model that accepts text, image, video, action, and sound as input. It uses a reasoning tower to reason through spatial and physical accuracy, generating a reasoned text outcome for a diffusion transformer to predict the next action or video (0:10:00).

## Practical implications

- Lowers the barrier to entry for robotics development by enabling the use of human data for training, reducing reliance on complex teleoperation setups.
- Allows developers to rapidly prototype and evaluate robot policies using real-time simulation and synthetic data generation (e.g., running Cosmos 3 Nano Sim Bimanual on Reactor).
- Provides a method to scale robot data generation across different robot embodiments using a single Cosmos 3 checkpoint.

## Topics

World Models, Robotics, Foundation Models, Synthetic Data Generation, Hand Tracking, Real-Time AI, RLWRLD Technical Blog, Cosmos 3 on Hugging Face

Source: https://www.youtube.com/watch?v=09igP-kQXtw
