This article discusses the implementation of push-based Key-Value (KV) cache transfer in vLLM to optimize disaggregated inference. By moving transfer coordination and memory allocation off the critical path of the Time-to-First-Token (TTFT), the push-based connector reduces latency compared to the traditional pull-based approach. The authors demonstrate that this optimization utilizes RDMA via the NIXL connector to achieve significant performance gains on AWS infrastructure.
Key points
Disaggregated inference separates prefill and decode tasks, requiring efficient KV cache transfers between instances.
Pull-based KV transfer models introduce serial bottlenecks in the TTFT path due to coordination and allocation delays.
Push-based transfers move setup operations out of the critical path to improve overall system responsiveness.
NIXL with RDMA allows direct GPU-to-GPU memory transfer, bypassing host staging for reduced overhead.
Choosing between push and pull connectors requires balancing load-aware routing flexibility against TTFT requirements.