NVIDIA Developer

Ask the Experts: Dory Looped Transformer Model | Nemotron Labs

Published 2026-11-07 · Duration 49:46

Summary

This session introduces Dory, a looped transformer model from Nemotron Labs, which addresses the trade-off between compute cost and model accuracy at inference time. The core innovation is looping, where a shared block of layers is repeatedly processed, allowing the model to perform multiple passes of computation without increasing the model's parameter count. This method significantly improves performance on reasoning and agentic tasks by condensing the thinking process and is highly memory efficient due to the use of last-loop KV caching, which keeps memory usage constant regardless of the loop count.

Download summary

Key takeaways

  1. Looping trades compute for accuracy 0:03

    By repeating the processing through a shared set of layers (the loop), the model can increase its reasoning depth and accuracy. For example, Dory improved its SWE-bench Verified score from 50.3 at one loop to 61.8 at four loops, demonstrating a monotonic increase in performance with more loops. (0:00, 0:03)

  2. Memory efficiency via last-loop KV caching 0:58

    A major advantage of Dory is its memory management. Instead of requiring a separate KV cache for every loop, the model uses a single set of KV cache that is overwritten during looping. This keeps the memory overhead constant, solving a major memory problem in long-context workloads. (0:25, 0:58)

  3. Looping vs. Chain of Thought (CoT) 2:10

    Unlike traditional Chain of Thought (CoT) prompting, which forces the model to generate thinking steps as tokens, looping performs thinking in the model's latent space. This process condenses the thinking, making the final response more succinct and high-quality while maintaining visibility into the reasoning process. (1:30, 2:10)

  4. Current limitations and future research 0:45

    Currently, the user must manually set the number of loops (Dory supports 1 to 4 loops, recommending 2 for the best trade-off). Adaptive systems—where the model automatically determines the optimal number of loops per query or even per token—are an active area of research. (0:45, 2:20)

More digests on Looped Transformers

Technical details

  • Looped Transformer Architecture 15s

    The model utilizes a loop structure where a specific, shared block of layers is repeatedly applied to the input token. This allows for increased computation depth without increasing the model's parameter count. The model is based on familiar dense patterns but implements this looping mechanism. (0:15, 2:30)

  • Memory Optimization (KV Cache) 58s

    The implementation uses a 'last-loop KV cache' method. This technique ensures that the memory required for the KV cache remains constant during inference, even when increasing the number of loops, thereby solving a dominant memory cost in long-context tasks. (0:58)

  • Training and Deployment 2s

    The model supports running on DGX Spark and Hopper and Blackwell GPUs. Training and checkpoint formats include native NVFP4 pre-training, BF16, MXFP8, and NVFP4 checkpoints. (0:02)

  • Scaling and Compute Trade-off 35s

    The architecture allows for 'test time scaling' (or inference time compute scaling), where the user can dynamically adjust the compute effort by setting the loop count. This provides a direct trade-off between computational cost and resulting accuracy. (0:35, 2:30)

Mentioned resources

  • Dory (Hugging Face) (Model Repository)
  • NVIDIA Neumotron GitHub repo (Documentation)

Channel & topics

Watch on YouTube · Back to latest

This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.