Mercury: Multi GPU operator optimization
Summary
This talk introduces Mercury, an auto-tuning compiler designed to optimize multi-GPU programs, particularly for demanding AI workloads like those found in large language models (LLMs). The core problem addressed is that existing manual optimizations often fail to achieve optimal performance due to communication bottlenecks. Mercury solves this by creating a new Intermediate Representation (ComIR) that explicitly models asynchronous communication overlap, allowing the compiler to search a vast design space for optimal schedules, leading to significant speedups (up to 4x at the operator level). The speaker also details SinglePass, an extension that fuses communication and computation into single, fine-grained kernels for further optimization.
Key takeaways
-
Asynchronous Overlap for Multi-GPU Programs
2:00
Traditional collective communication bottlenecks can be mitigated by implementing asynchronous overlap, which avoids requiring all input data to be prepared synchronously. This is achieved by partitioning the workload and allowing devices to process tiles independently, enabling peer-to-peer communication during local computation.
-
Mercury's ComIR Design Space
4:10
Mercury introduces a Communication IR (ComIR) that extends standard auto-tuning compiler primitives (like `tile`, `reorder`, `join`) with communication primitives (`parallelize`, `shift`, `shard`). This allows the compiler to express complex communication patterns, such as two-level asynchronous communication, in a unified, searchable design space.
-
SinglePass for Fused Kernels
5:10
To improve upon kernel-level overlap, SinglePass fuses communication and computation into single, large kernels. This new representation splits intra-kernel computation into 'chunks' and schedules communications based on these chunks, enabling better fine-grained overlap and reducing synchronization overhead.
More digests on Compiler Optimization
Technical details
-
Compiler Architecture
180s
Auto-tuning compilers operate by parsing an input expression into an Intermediate Representation (IR), applying transformation primitives (a 'schedule'), and benchmarking the resulting code to find the optimal configuration. The process separates the algorithm (input expression) from the optimization transformations (schedule).
-
ComIR Primitives
250s
The ComIR includes compute primitives (e.g., `tile`, `join`, `reorder`) and communication primitives (e.g., `parallelize`, `shift`, `shard`). The `shift` primitive is key for achieving asynchronous overlap by allowing different devices to start computation from different, offset input data tiles.
-
Optimization Pipeline
300s
The overall pipeline involves a Python-defined DSL for the algorithm, the ComIR for the schedule, and an auto-tuner that uses an analytic model to search the design space. The output is a distributed operator that can be compiled by low-level compilers or communication libraries like NCCL.
Mentioned resources
- Mercury
- SinglePass
- Triton
- SOSP '25
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.