ParallelKernelBench (PKB) is an evaluation framework designed to measure the efficacy of LLMs in generating multi-GPU CUDA kernels that outperform standard PyTorch/NCCL implementations. The authors demonstrate that while LLMs show promise in single-GPU scenarios, they struggle with the complex communication constraints, topological awareness, and data movement strategies required for multi-GPU performance.
Key points
Multi-GPU kernel optimization shifts the primary bottleneck from local compute/memory bandwidth to interconnect communication.
Effective multi-GPU kernel design requires selecting specific data movement primitives like TMA, SM load/store, or NVLS.
Current state-of-the-art LLMs struggle to correctly implement distributed kernels, with fewer than 25% of successful implementations beating naive NCCL baselines.
PKB establishes a methodology for benchmarking kernels by requiring LLMs to optimize communication topology alongside compute.