# Lecture 120: Mercury Multi GPU operator optimization

## Executive summary

This lecture introduces advanced compiler techniques for optimizing multi-GPU programs, particularly for large AI workloads like attention mechanisms in LLMs. The speaker details the development of 'Mercury,' an autotuning compiler that automates the discovery of optimal asynchronous overlap schedules. Furthermore, he presents 'Syncopate,' a compiler designed to extend this optimization to the kernel level by fusing communication and computation into single, fine-grained kernels, addressing the overhead of traditional device synchronization.

## Key takeaways

- Asynchronous Overlap for Multi-GPU Programs: Traditional collective communication (e.g., `reduce C`) becomes a bottleneck when scaling across multiple GPUs. The solution involves asynchronous overlap by partitioning the workload and using peer-to-peer communications, eliminating the need for synchronous reduction and allowing intermediate results to be exchanged during local computation.
- Mercury Compiler and CommIR: Mercury is an autotuning compiler that separates the algorithm (input expression) from the optimization transformations (the schedule). It introduces a specialized Intermediate Representation (CommIR) composed of compute and communication primitives (e.g., `paralyze`, `shift`, `join`) to express complex asynchronous communication patterns.
- Kernel-Level Optimization with Syncopate: To overcome the overhead of device synchronization between separate communication and computation kernels, Syncopate fuses these into a single, large kernel. It achieves fine-grain overlap by splitting the computation into 'chunks' and scheduling communications between chunks before they are fully executed.
- Performance Gains: The proposed methods achieve significant speedups: Mercury can reach up to 4.7x speedup (compared to manual fuse overlap), and Syncopate shows an overall 1.3x speedup compared to state-of-the-art kernel-level baselines.

## Technical details

- Autotuning Compiler Design Space: Autotuning compilers operate by parsing an input expression into an Intermediate Representation (IR) and searching a design space of transformation primitives (schedules) to find the optimal configuration (e.g., selecting Schedule X for best latency).
- CommIR Primitives: The CommIR uses transformation primitives like `paralyze` (partitioning a loop across a network hierarchy), `shift` (shifting a temporal loop based on a spatial loop), and `join` (merging two axes) to model complex communication patterns.
- Compiler Pipeline: The full pipeline involves a user-defined DSL (in Python) for defining operators, which is parsed into the CommIR. The autotuner then generates the final parallel CommIR, which is passed to low-level compilers and communication libraries (like NCCL/TMA) for execution.
- Dynamic Workloads: For dynamic shapes (e.g., speculative decoding), the DCode compiler extends tensor programs to allow autotuning across different shapes, reducing both search time and runtime overhead by dispatching results at runtime.

## Practical implications

- Enables efficient scaling of AI models (like LLMs) across massive multi-GPU data centers by optimizing communication bottlenecks.
- Provides a systematic, automated way to discover optimal hardware scheduling and communication patterns, surpassing manual optimization efforts.
- The Syncopate approach minimizes synchronization overhead by fusing communication and computation into single kernels, crucial for high-performance computing.

## Topics

Compiler Optimization, Multi-GPU Programming, AI Workloads (LLMs), Autotuning, Intermediate Representation (IR), Distributed Computing, Mercury, Syncopate, CommIR, Triton

Source: https://www.youtube.com/watch?v=mYlyLsXu9Fg
