The article introduces Higher-Order Pruning of Experts (HOPE), a method for compressing Mixture-of-Experts (MoE) models by identifying and removing redundant experts based on pairwise interactions. Unlike traditional first-order methods that score experts in isolation, HOPE accounts for co-activation patterns, leading to significant performance improvements at aggressive pruning rates. The authors demonstrate that considering these expert dependencies is critical for maintaining model accuracy when reducing the memory footprint of large-scale MoE architectures.
Key points
First-order pruning methods fail to account for cooperative expert clusters in MoE models, leading to performance degradation at high pruning rates.
HOPE utilizes a second-order approach to model pairwise expert interactions, improving output quality when removing up to 50% of model experts.
Expert pruning is an effective technique for reducing inference-time memory usage without requiring additional training or complex dynamic loading mechanisms.
The effectiveness of pairwise interaction modeling becomes significantly more pronounced as the overall pruning fraction increases.