Lecture 123: Larger Fusions and Consistent RL: Recent Work in FlashInfer on Blackwell and Rubin
Summary
This talk details recent advancements in FlashInfer, focusing on two critical areas: optimizing MegaMoE for inference efficiency on NVIDIA Blackwell and Rubin GPUs, and ensuring numerical consistency for Reinforcement Learning (RL) workloads. MegaMoE optimization involves advanced kernel techniques like dynamic tile sizing and data movement strategies (pull vs. push) to reduce latency. Numerical consistency requires defining a 'numerical contract' to guarantee bitwise agreement across different execution phases (prefill, decode) and hardware, mitigating the challenges posed by non-associative floating-point arithmetic.
Key takeaways
-
MegaMoE Latency Reduction
16:58
Optimizations like dynamic N value and A/B swapping allow the kernel to adapt to sparse token counts during decoding, reducing padding waste and improving latency for latency-sensitive workloads.
-
Numerical Consistency is Paramount for RL
27:28
RL frameworks require strict numerical guarantees (Determinism, Bitwise Batch Invariance, Phase Consistency) because the same model is used in different execution settings (training vs. rollout).
-
Constraining the Optimization Search Space
38:46
To achieve batch invariance, the team defined 'numerical classes' that fix the computation (e.g., split K factor, rounding points), allowing optimization to occur only within that constrained class.
-
Performance Trade-offs are Real
58:18
While the batch-invariant kernels are correct, they incur a performance cost (up to 30% loss in some cases) compared to non-batch-invariant, highly optimized alternatives, highlighting the cost of numerical rigor.
Technical details
-
MegaMoE Kernel Implementation
443s
The main kernel uses a persistent grid of clusters (CTA) to overlap communication and compute. Synchronization is managed at three levels: GPU (system scope fence), CTA (counter), and warp (M barrier).
-
Data Movement Strategies (Pull vs. Push)
1227s
Push mode sends tokens to all required GPUs simultaneously, which can save bandwidth if multiple experts need the same token. Pull mode allows the receiving GPU to organize routes and pull tokens, which is better when tokens are scattered.
-
Batch Invariance Mechanisms
1777s
Batch invariance is maintained by fixing the order of floating-point arithmetic (e.g., softmax recurrence, reduction order) and ensuring that the output bits match across permutations and subbatches.
-
Attention Phase Consistency
3190s
Attention requires the same logical query to maintain its output bits across three distinct execution paths: ragged prefill, paged prefill, and decode, which use vastly different algorithms.
Mentioned resources
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.