This article explores a cache-aware disaggregation (CPD) architecture designed to optimize inference for long-context LLMs. By isolating compute resources for 'warm' requests that reuse existing KV caches from 'cold' requests requiring full prefill, the system prevents long prompts from blocking latency-sensitive tasks. Evaluations demonstrate a 35–40% improvement in sustainable queries-per-second (QPS) and more stable tail latencies under high-load, mixed workloads.
Key points
Long-context inference suffers from 'head-of-line blocking' where expensive cold prefills stall warm requests that could benefit from KV cache reuse.
Decoupling prefill resources for warm versus cold requests significantly reduces TTFT variability in mixed-load environments.
System-level scheduling based on context reusability is as critical as model compute efficiency for scaling long-context AI applications.
Cache-aware disaggregation maintains throughput benefits even when scaling decode capacity, confirming the strategy's effectiveness in end-to-end pipelines.