# Lecture 123: Larger Fusions and Consistent RL: Recent Work in FlashInfer on Blackwell and Rubin

## Executive summary

This talk details recent advancements in FlashInfer, focusing on two critical areas: optimizing MegaMoE for inference efficiency on NVIDIA Blackwell and Rubin GPUs, and ensuring numerical consistency for Reinforcement Learning (RL) workloads. MegaMoE optimization involves advanced kernel techniques like dynamic tile sizing and data movement strategies (pull vs. push) to reduce latency. Numerical consistency requires defining a 'numerical contract' to guarantee bitwise agreement across different execution phases (prefill, decode) and hardware, mitigating the challenges posed by non-associative floating-point arithmetic.

## Key takeaways

- MegaMoE Latency Reduction: Optimizations like dynamic N value and A/B swapping allow the kernel to adapt to sparse token counts during decoding, reducing padding waste and improving latency for latency-sensitive workloads.
- Numerical Consistency is Paramount for RL: RL frameworks require strict numerical guarantees (Determinism, Bitwise Batch Invariance, Phase Consistency) because the same model is used in different execution settings (training vs. rollout).
- Constraining the Optimization Search Space: To achieve batch invariance, the team defined 'numerical classes' that fix the computation (e.g., split K factor, rounding points), allowing optimization to occur only within that constrained class.
- Performance Trade-offs are Real: While the batch-invariant kernels are correct, they incur a performance cost (up to 30% loss in some cases) compared to non-batch-invariant, highly optimized alternatives, highlighting the cost of numerical rigor.

## Technical details

- MegaMoE Kernel Implementation: The main kernel uses a persistent grid of clusters (CTA) to overlap communication and compute. Synchronization is managed at three levels: GPU (system scope fence), CTA (counter), and warp (M barrier).
- Data Movement Strategies (Pull vs. Push): Push mode sends tokens to all required GPUs simultaneously, which can save bandwidth if multiple experts need the same token. Pull mode allows the receiving GPU to organize routes and pull tokens, which is better when tokens are scattered.
- Batch Invariance Mechanisms: Batch invariance is maintained by fixing the order of floating-point arithmetic (e.g., softmax recurrence, reduction order) and ensuring that the output bits match across permutations and subbatches.
- Attention Phase Consistency: Attention requires the same logical query to maintain its output bits across three distinct execution paths: ragged prefill, paged prefill, and decode, which use vastly different algorithms.

## Practical implications

- The development of numerically consistent kernels is essential for reliable LLM post-training via RL, ensuring that training and rollout processes yield identical results.
- The concept of 'numerical classes' provides a structured way to constrain the vast search space of kernel optimizations, allowing for high-confidence, verifiable kernel design.
- The work demonstrates that achieving strict numerical guarantees requires a trade-off in performance, necessitating careful evaluation of the cost of invariance.

## Topics

AI, Build Engineering, GPU Architecture, LLM Optimization, Reinforcement Learning, gpumode.com

Source: https://www.youtube.com/watch?v=YoaSiClZtuQ
