GPU MODE

Lecture 123: Larger Fusions and Consistent RL: Recent Work in FlashInfer on Blackwell and Rubin

Published 2026-11-20 · Duration 1:06:43

Summary

This talk details recent advancements in FlashInfer, focusing on two critical areas: optimizing MegaMoE for inference efficiency on NVIDIA Blackwell and Rubin GPUs, and ensuring numerical consistency for Reinforcement Learning (RL) workloads. MegaMoE optimization involves advanced kernel techniques like dynamic tile sizing and data movement strategies (pull vs. push) to reduce latency. Numerical consistency requires defining a 'numerical contract' to guarantee bitwise agreement across different execution phases (prefill, decode) and hardware, mitigating the challenges posed by non-associative floating-point arithmetic.

Download summary

Key takeaways

  1. MegaMoE Latency Reduction 16:58

    Optimizations like dynamic N value and A/B swapping allow the kernel to adapt to sparse token counts during decoding, reducing padding waste and improving latency for latency-sensitive workloads.

  2. Numerical Consistency is Paramount for RL 27:28

    RL frameworks require strict numerical guarantees (Determinism, Bitwise Batch Invariance, Phase Consistency) because the same model is used in different execution settings (training vs. rollout).

  3. Constraining the Optimization Search Space 38:46

    To achieve batch invariance, the team defined 'numerical classes' that fix the computation (e.g., split K factor, rounding points), allowing optimization to occur only within that constrained class.

  4. Performance Trade-offs are Real 58:18

    While the batch-invariant kernels are correct, they incur a performance cost (up to 30% loss in some cases) compared to non-batch-invariant, highly optimized alternatives, highlighting the cost of numerical rigor.

More digests on AI

Technical details

  • MegaMoE Kernel Implementation 443s

    The main kernel uses a persistent grid of clusters (CTA) to overlap communication and compute. Synchronization is managed at three levels: GPU (system scope fence), CTA (counter), and warp (M barrier).

  • Data Movement Strategies (Pull vs. Push) 1227s

    Push mode sends tokens to all required GPUs simultaneously, which can save bandwidth if multiple experts need the same token. Pull mode allows the receiving GPU to organize routes and pull tokens, which is better when tokens are scattered.

  • Batch Invariance Mechanisms 1777s

    Batch invariance is maintained by fixing the order of floating-point arithmetic (e.g., softmax recurrence, reduction order) and ensuring that the output bits match across permutations and subbatches.

  • Attention Phase Consistency 3190s

    Attention requires the same logical query to maintain its output bits across three distinct execution paths: ragged prefill, paged prefill, and decode, which use vastly different algorithms.

Mentioned resources

Channel & topics

Watch on YouTube · Back to latest

This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.