Distribution-aware speculative decoding (DAS) addresses the rollout bottleneck in reinforcement learning by using an adaptive suffix tree drafter and a length-aware scheduler. By exploiting historical prompt trajectories, the framework mitigates GPU idle time caused by long-tail generation sequences without impacting model training outcomes.
Key points
RL training rollouts suffer from a long-tail distribution where straggler sequences force synchronous GPU idle time.
DAS uses an adaptive suffix tree drafter that evolves based on historical rollout data without needing extra gradient updates.
The framework preserves the target distribution, ensuring identical model outputs and training curves compared to standard decoding.
Length-aware scheduling and historical trajectory exploitation significantly improve hardware utilization during RL post-training.