GPU MODE

Larger Fusions and Consistent RL: Recent Work in FlashInfer on Blackwell and Rubin

Published 2026-11-09 · Duration 1:06:39

Summary

This talk details recent advancements in FlashInfer, focusing on two critical areas: MegaMo optimization for inference efficiency on NVIDIA Blackwell and Rubin GPUs, and ensuring numerical consistency for Reinforcement Learning (RL) workloads. MegaMo fuses layers to overlap communication and compute, while the numerical consistency work introduces strict requirements (determinism, batch invariance, phase consistency) to prevent numerical drift in complex RL pipelines. The speakers demonstrate that while these optimizations yield significant speedups, maintaining numerical correctness requires constraining the kernel design space.

Download summary

Key takeaways

  1. MegaMo Overlap Mechanism 3:04

    MegaMo fuses the entire layer into one pipeline, allowing data movement and compute to run concurrently. This reduces latency by enabling the GPU to start computing on a tile as soon as sufficient input is available, rather than waiting for all input transfers to complete [03:24].

  2. The Cost of Numerical Consistency 23:50

    Achieving Batch Invariance and Phase Consistency in complex operators like attention requires fixing the rounding point, tile boundaries, and reduction order. This constraint limits the degrees of freedom available for optimization, resulting in a performance cost (up to 30% loss in some cases) compared to non-batch-invariant alternatives [14:30].

  3. Token Push vs. Token Pull 11:15

    To reduce memory traffic, the system utilizes a 'Token Push' mechanism where the source GPU sends a token to all required experts simultaneously, rather than requiring each receiving expert to pull the token separately. This is facilitated by TMA gather mode [11:15].

More digests on AI

Technical details

  • MegaMo Implementation Details 270s

    The core kernel uses a persistent grid of clusters (CTA) to keep the GPU busy. The two CTA MMA approach on Rubin uses two CTAs to handle a full B tile, avoiding redundant B data loading. Synchronization is managed at three levels: GPU (system scope fence), CTA (counter for intermediate parts), and warp (M barrier) [04:30].

  • Batch Invariance Requirements 1200s

    Batch Invariance requires that permuting the rows of the activation input (A) does not change the output bits. Subbatch invariance requires that partitioning the input and running GEMs independently results in the same output as the unbatched input [12:00].

  • Attention Phase Consistency 1330s

    Attention kernels must maintain bitwise agreement across three distinct execution paths: ragged/unpaged prefill (contiguous KV), paged prefill (jumbled blocks), and decode (single query). The online softmax recurrence, which carries state (max, denominator), is a key challenge that must be fixed to ensure consistency [13:30].

Mentioned resources

Channel & topics

Watch on YouTube · Back to latest

This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.