GPU MODE

Lecture 120: Mercury Multi GPU operator optimization

Published 2026-11-06 · Duration 53:03

Summary

This lecture introduces advanced compiler techniques for optimizing multi-GPU programs, particularly for large AI workloads like attention mechanisms in LLMs. The speaker details the development of 'Mercury,' an autotuning compiler that automates the discovery of optimal asynchronous overlap schedules. Furthermore, he presents 'Syncopate,' a compiler designed to extend this optimization to the kernel level by fusing communication and computation into single, fine-grained kernels, addressing the overhead of traditional device synchronization.

Download summary

Key takeaways

  1. Asynchronous Overlap for Multi-GPU Programs 2:00

    Traditional collective communication (e.g., `reduce C`) becomes a bottleneck when scaling across multiple GPUs. The solution involves asynchronous overlap by partitioning the workload and using peer-to-peer communications, eliminating the need for synchronous reduction and allowing intermediate results to be exchanged during local computation.

  2. Mercury Compiler and CommIR 3:00

    Mercury is an autotuning compiler that separates the algorithm (input expression) from the optimization transformations (the schedule). It introduces a specialized Intermediate Representation (CommIR) composed of compute and communication primitives (e.g., `paralyze`, `shift`, `join`) to express complex asynchronous communication patterns.

  3. Kernel-Level Optimization with Syncopate 4:20

    To overcome the overhead of device synchronization between separate communication and computation kernels, Syncopate fuses these into a single, large kernel. It achieves fine-grain overlap by splitting the computation into 'chunks' and scheduling communications between chunks before they are fully executed.

  4. Performance Gains 3:40

    The proposed methods achieve significant speedups: Mercury can reach up to 4.7x speedup (compared to manual fuse overlap), and Syncopate shows an overall 1.3x speedup compared to state-of-the-art kernel-level baselines.

More digests on Compiler Optimization

Technical details

  • Autotuning Compiler Design Space 150s

    Autotuning compilers operate by parsing an input expression into an Intermediate Representation (IR) and searching a design space of transformation primitives (schedules) to find the optimal configuration (e.g., selecting Schedule X for best latency).

  • CommIR Primitives 200s

    The CommIR uses transformation primitives like `paralyze` (partitioning a loop across a network hierarchy), `shift` (shifting a temporal loop based on a spatial loop), and `join` (merging two axes) to model complex communication patterns.

  • Compiler Pipeline 240s

    The full pipeline involves a user-defined DSL (in Python) for defining operators, which is parsed into the CommIR. The autotuner then generates the final parallel CommIR, which is passed to low-level compilers and communication libraries (like NCCL/TMA) for execution.

  • Dynamic Workloads 300s

    For dynamic shapes (e.g., speculative decoding), the DCode compiler extends tensor programs to allow autotuning across different shapes, reducing both search time and runtime overhead by dispatching results at runtime.

Mentioned resources

  • Mercury (Compiler/Tool)
  • Syncopate (Compiler/Tool)
  • CommIR (Intermediate Representation)
  • Triton (Low-Level Compiler)

Channel & topics

Watch on YouTube · Back to latest

This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.