NVIDIA Developer

From Human Demonstrations to Robot Training Data With RLWRLD and Reactor | Cosmos Labs

Published 2026-11-06 · Duration 1:07:41

Summary

This session details a comprehensive pipeline for converting human video demonstrations into high-quality, robot-ready training data, leveraging NVIDIA Cosmos 3 and RLWRLD. The process moves from synchronized human video capture to robot action generation, utilizing advanced hand tracking (Reand) and real-time world model inference via Reactor. Key innovations include the ability to synthesize observations for complex, deformable objects (like garments) without explicit 3D object scanning, and the demonstration of low-latency policy evaluation in a cloud environment.

Download summary

Key takeaways

  1. Human Data Scales Faster than Teleoperation 14:10

    Leveraging human demonstrations allows for significantly faster data collection compared to teleoperation, which requires a robot and control interface. A single human demonstration can be converted into episodes for various robot embodiments (1:25:00).

  2. Cosmos 3 Enables Deformable Object Synthesis 11:00

    For tasks like garment packing, Cosmos 3 can generate the movement of soft objects (clothes, tissue paper) following the robot hands, bypassing the need for difficult 3D object scanning or mesh modeling (1:40:00).

  3. Real-Time World Model Inference via Reactor 20:50

    Reactor provides a global, low-latency infrastructure (50 milliseconds worldwide) to serve and deploy models like Cosmos 3 Nano Sim Bimanual, enabling real-time policy evaluation and synthetic data generation (0:35:00).

More digests on World Models

Technical details

  • RLWRLD Hand Tracking (Reand) 1900s

    Reand is a state-of-the-art hand tracker built on the Cosmos 3 encoder that processes egocentric video to predict not only 3D hand poses but also technical information, including contact and force estimation, without requiring specialized hardware (0:30:00).

  • The Human-to-Robot Pipeline 2100s

    The process involves four steps: 1) Human video capture (calibrated multi-view/egocentric cameras); 2) 3D dynamics reconstruction (using Reand to extract pose and tactile info); 3) Robot action generation (retargeting the reconstructed human motion); and 4) Observation synthesis (using Isaac Sim for rigid objects and Cosmos 3 for complex objects) (2:00:00).

  • Cosmos 3 Architecture and Usage 100s

    Cosmos 3 is an omni-model that accepts text, image, video, action, and sound as input. It uses a reasoning tower to reason through spatial and physical accuracy, generating a reasoned text outcome for a diffusion transformer to predict the next action or video (0:10:00).

Mentioned resources

Channel & topics

Watch on YouTube · Back to latest

This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.