# Larger Fusions and Consistent RL: Recent Work in FlashInfer on Blackwell and Rubin

## Executive summary

This talk details recent advancements in FlashInfer, focusing on two critical areas: MegaMo optimization for inference efficiency on NVIDIA Blackwell and Rubin GPUs, and ensuring numerical consistency for Reinforcement Learning (RL) workloads. MegaMo fuses layers to overlap communication and compute, while the numerical consistency work introduces strict requirements (determinism, batch invariance, phase consistency) to prevent numerical drift in complex RL pipelines. The speakers demonstrate that while these optimizations yield significant speedups, maintaining numerical correctness requires constraining the kernel design space.

## Key takeaways

- MegaMo Overlap Mechanism: MegaMo fuses the entire layer into one pipeline, allowing data movement and compute to run concurrently. This reduces latency by enabling the GPU to start computing on a tile as soon as sufficient input is available, rather than waiting for all input transfers to complete [03:24].
- The Cost of Numerical Consistency: Achieving Batch Invariance and Phase Consistency in complex operators like attention requires fixing the rounding point, tile boundaries, and reduction order. This constraint limits the degrees of freedom available for optimization, resulting in a performance cost (up to 30% loss in some cases) compared to non-batch-invariant alternatives [14:30].
- Token Push vs. Token Pull: To reduce memory traffic, the system utilizes a 'Token Push' mechanism where the source GPU sends a token to all required experts simultaneously, rather than requiring each receiving expert to pull the token separately. This is facilitated by TMA gather mode [11:15].

## Technical details

- MegaMo Implementation Details: The core kernel uses a persistent grid of clusters (CTA) to keep the GPU busy. The two CTA MMA approach on Rubin uses two CTAs to handle a full B tile, avoiding redundant B data loading. Synchronization is managed at three levels: GPU (system scope fence), CTA (counter for intermediate parts), and warp (M barrier) [04:30].
- Batch Invariance Requirements: Batch Invariance requires that permuting the rows of the activation input (A) does not change the output bits. Subbatch invariance requires that partitioning the input and running GEMs independently results in the same output as the unbatched input [12:00].
- Attention Phase Consistency: Attention kernels must maintain bitwise agreement across three distinct execution paths: ragged/unpaged prefill (contiguous KV), paged prefill (jumbled blocks), and decode (single query). The online softmax recurrence, which carries state (max, denominator), is a key challenge that must be fixed to ensure consistency [13:30].

## Practical implications

- LLM post-training via RL requires kernels to maintain numerical agreement across different execution settings (prefill, decode, training microbatch).
- The open-source nature of FlashInfer allows the community to inspect and contribute to the underlying batch-invariant kernels.
- The design process involves co-designing kernels and proofs using symbolic traces and Lean models to guarantee numerical correctness before performance tuning.

## Topics

AI, GPU Computing, Reinforcement Learning, Kernel Optimization, Numerical Stability, gpumode.com

Source: https://www.youtube.com/watch?v=jhBF3UaPD2E
