This article explores bi-directional KV cache transfer as a method to optimize multi-turn agentic workloads in disaggregated serving architectures. By enabling prefill nodes to retrieve previously computed KV blocks from decode nodes, the approach minimizes redundant computations and improves Time to First Token (TTFT) as conversation history grows. The implementation utilizes RDMA via AWS EFA to facilitate efficient data movement between accelerator memory across nodes.
Key points
Standard disaggregated inference performs one-way KV cache transfer, causing redundant prefill computation during multi-turn conversations.
Bi-directional KV cache transfer allows prefill nodes to reuse previously generated context held by decode nodes.
Utilizing GPUDirect RDMA or Neuron Direct RDMA via AWS EFA enables efficient KV block movement across accelerator memory.
Configuring block-retention and recompute thresholds allows for granular control over when cache reuse occurs.