# Ask the Experts: Dory Looped Transformer Model | Nemotron Labs

## Executive summary

This session introduces Dory, a looped transformer model from Nemotron Labs, which addresses the trade-off between compute cost and model accuracy at inference time. The core innovation is looping, where a shared block of layers is repeatedly processed, allowing the model to perform multiple passes of computation without increasing the model's parameter count. This method significantly improves performance on reasoning and agentic tasks by condensing the thinking process and is highly memory efficient due to the use of last-loop KV caching, which keeps memory usage constant regardless of the loop count.

## Key takeaways

- Looping trades compute for accuracy: By repeating the processing through a shared set of layers (the loop), the model can increase its reasoning depth and accuracy. For example, Dory improved its SWE-bench Verified score from 50.3 at one loop to 61.8 at four loops, demonstrating a monotonic increase in performance with more loops. (0:00, 0:03)
- Memory efficiency via last-loop KV caching: A major advantage of Dory is its memory management. Instead of requiring a separate KV cache for every loop, the model uses a single set of KV cache that is overwritten during looping. This keeps the memory overhead constant, solving a major memory problem in long-context workloads. (0:25, 0:58)
- Looping vs. Chain of Thought (CoT): Unlike traditional Chain of Thought (CoT) prompting, which forces the model to generate thinking steps as tokens, looping performs thinking in the model's latent space. This process condenses the thinking, making the final response more succinct and high-quality while maintaining visibility into the reasoning process. (1:30, 2:10)
- Current limitations and future research: Currently, the user must manually set the number of loops (Dory supports 1 to 4 loops, recommending 2 for the best trade-off). Adaptive systems—where the model automatically determines the optimal number of loops per query or even per token—are an active area of research. (0:45, 2:20)

## Technical details

- Looped Transformer Architecture: The model utilizes a loop structure where a specific, shared block of layers is repeatedly applied to the input token. This allows for increased computation depth without increasing the model's parameter count. The model is based on familiar dense patterns but implements this looping mechanism. (0:15, 2:30)
- Memory Optimization (KV Cache): The implementation uses a 'last-loop KV cache' method. This technique ensures that the memory required for the KV cache remains constant during inference, even when increasing the number of loops, thereby solving a dominant memory cost in long-context tasks. (0:58)
- Training and Deployment: The model supports running on DGX Spark and Hopper and Blackwell GPUs. Training and checkpoint formats include native NVFP4 pre-training, BF16, MXFP8, and NVFP4 checkpoints. (0:02)
- Scaling and Compute Trade-off: The architecture allows for 'test time scaling' (or inference time compute scaling), where the user can dynamically adjust the compute effort by setting the loop count. This provides a direct trade-off between computational cost and resulting accuracy. (0:35, 2:30)

## Practical implications

- Build engineers can design inference pipelines that dynamically adjust the number of loops based on the required task difficulty, optimizing the cost/accuracy balance.
- The constant memory footprint achieved through last-loop KV caching is critical for deploying long-context models on resource-constrained or high-throughput systems.
- The architecture provides a novel method for improving model performance (reasoning) without requiring additional parameters or massive increases in model size.

## Topics

Looped Transformers, LLM Architecture, Inference Optimization, KV Caching, Compute Scaling, Nemotron Labs, NVIDIA, Dory (Hugging Face), NVIDIA Neumotron GitHub repo

Source: https://www.youtube.com/watch?v=_MEqhFwnp8o
