# Mercury: Multi GPU operator optimization

## Executive summary

This talk introduces Mercury, an auto-tuning compiler designed to optimize multi-GPU programs, particularly for demanding AI workloads like those found in large language models (LLMs). The core problem addressed is that existing manual optimizations often fail to achieve optimal performance due to communication bottlenecks. Mercury solves this by creating a new Intermediate Representation (ComIR) that explicitly models asynchronous communication overlap, allowing the compiler to search a vast design space for optimal schedules, leading to significant speedups (up to 4x at the operator level). The speaker also details SinglePass, an extension that fuses communication and computation into single, fine-grained kernels for further optimization.

## Key takeaways

- Asynchronous Overlap for Multi-GPU Programs: Traditional collective communication bottlenecks can be mitigated by implementing asynchronous overlap, which avoids requiring all input data to be prepared synchronously. This is achieved by partitioning the workload and allowing devices to process tiles independently, enabling peer-to-peer communication during local computation.
- Mercury's ComIR Design Space: Mercury introduces a Communication IR (ComIR) that extends standard auto-tuning compiler primitives (like `tile`, `reorder`, `join`) with communication primitives (`parallelize`, `shift`, `shard`). This allows the compiler to express complex communication patterns, such as two-level asynchronous communication, in a unified, searchable design space.
- SinglePass for Fused Kernels: To improve upon kernel-level overlap, SinglePass fuses communication and computation into single, large kernels. This new representation splits intra-kernel computation into 'chunks' and schedules communications based on these chunks, enabling better fine-grained overlap and reducing synchronization overhead.

## Technical details

- Compiler Architecture: Auto-tuning compilers operate by parsing an input expression into an Intermediate Representation (IR), applying transformation primitives (a 'schedule'), and benchmarking the resulting code to find the optimal configuration. The process separates the algorithm (input expression) from the optimization transformations (schedule).
- ComIR Primitives: The ComIR includes compute primitives (e.g., `tile`, `join`, `reorder`) and communication primitives (e.g., `parallelize`, `shift`, `shard`). The `shift` primitive is key for achieving asynchronous overlap by allowing different devices to start computation from different, offset input data tiles.
- Optimization Pipeline: The overall pipeline involves a Python-defined DSL for the algorithm, the ComIR for the schedule, and an auto-tuner that uses an analytic model to search the design space. The output is a distributed operator that can be compiled by low-level compilers or communication libraries like NCCL.

## Practical implications

- Achieves significant performance improvements (up to 4x on attention operators) for LLM workloads by optimizing communication and computation overlap.
- The compiler is designed to be highly hardware-agnostic in its IR and language design, making it adaptable to various accelerator platforms (e.g., CUDA core, TMA, Copy Engine).
- By searching the entire design space, the compiler can find optimal schedules that outperform manually designed, hardware-specific baselines, especially in complex network settings.

## Topics

Compiler Optimization, Multi-GPU Computing, AI/ML Accelerators, Intermediate Representation (IR), Distributed Systems, Mercury, SinglePass, Triton, SOSP '25

Source: https://www.youtube.com/watch?v=UuLKzcd8edQ
