FlashAttention-4 addresses performance bottlenecks in attention kernels on Blackwell GPUs by optimizing for asymmetric hardware scaling. The approach implements new software pipelining to overlap tensor core operations with memory movement and uses polynomial approximations on FMA units to bypass SFU constraints. These kernel design strategies achieve significant throughput improvements over existing standard libraries by reducing shared memory traffic and load imbalance.
Key points
Performance in modern GPUs is increasingly bottlenecked by non-GEMM resources like SFUs and memory bandwidth rather than just raw tensor core throughput.
Polynomial approximation on FMA units can successfully mitigate SFU bottlenecks during softmax computations in forward passes.
Utilizing Tensor Memory (TMEM) and 2-CTA MMA modes significantly reduces shared memory traffic during backward passes.
Efficient tile scheduling is critical for addressing load imbalance caused by causal masking and variable sequence lengths in LLM training.