DistServe: what the paper actually says
Reading Zhong et al. (Peking University, UC San Diego, OSDI 2024) while staring at a TTFT p99 graph that looked like a seismogram during an earthquake.
We were running LLaMA-70B on a cluster of A100 nodes for a coding assistant. Average prompts were ~1,500 tokens. Average completions were ~400 tokens. TPOT — time per output token — was fine: ~50ms per token, well inside our 200ms SLA. TTFT — time to first token — was the problem: p50 was 800ms, p99 was 4.2 seconds. Users would submit a request and stare at a blank screen for four seconds before anything appeared. And this happened regardless of total server load; even at 60% GPU utilization, p99 TTFT was unacceptable.
The diagnosis took two weeks. The fix was conceptually simple and operationally brutal: prefill and decode are not the same job, you shouldn't run them on the same GPU at the same time, and building a serving system that respects this distinction requires rethinking the entire request lifecycle.
The paper that formalized this is "DistServe: Disaggregating Prefill and Decoding for Goodput-Optimized Large Language Model Serving", by Zhong, Liu, Chen, Hu, Zhu, Liu, Jin, and Zhang, published at OSDI 2024.
The problem: two workloads, one GPU
LLM inference has two distinct phases with different computational character.
Prefill: the model processes the entire input prompt in a single forward pass. For a 1,500-token prompt, all 1,500 tokens are run through all 80 layers of the transformer simultaneously. This is a large, dense matrix multiplication — the input is a (1500 × 8192) token matrix, and every layer transforms it. This is compute-bound. The GPU's arithmetic units (CUDA cores and Tensor Cores) are fully utilized; memory bandwidth is not the bottleneck because the same weight matrices are read once and applied to 1,500 tokens in parallel. Prefill for a 1,500-token LLaMA-70B prompt on an A100 takes roughly 700ms in isolation.
Decode: after prefill, the model generates one token at a time. Each decode step is a single forward pass for one token (batch size 1 in the attention step, though batches of multiple requests can be packed). The input is a (1 × 8192) vector. This is memory-bandwidth-bound. The GPU must load all 140 GB of LLaMA-70B weights plus the KV cache for every in-flight sequence. Arithmetic utilization is low — most of the time is spent reading weights and KV cache from HBM, not computing. Each decode step takes ~20ms in isolation.
These are not variations on the same operation. They have different optimal batch sizes, different parallelism strategies, and different hardware utilization profiles. Prefill wants fewer, larger GPUs with high compute density. Decode wants more GPUs with high memory bandwidth and large aggregate KV cache capacity.
When you run both on the same GPU in the same serving process, you get interference:
Prefill preempts decode. A new request arrives. The serving system must process its 1,500-token prompt before producing the first output token. While prefill runs — 700ms — every in-flight decode request is paused. Those decode requests are mid-generation; their users are watching a streaming output that just stopped. After 700ms, decode resumes, everyone gets their next token, and the TTFT clock for the new request stops. 700ms is your minimum TTFT floor even at zero server load. Under real load with multiple concurrent prefill requests, the floor compounds.
Decode degrades prefill throughput. KV cache from active decode sequences occupies HBM. The prefill forward pass loads weight matrices; if HBM is also busy serving KV cache reads for decode, effective memory bandwidth to weights drops. The interference is asymmetric — decode is more sensitive to this than prefill — but it's real.
The paper quantifies this: on LLaMA-13B with a chatbot workload (128-token prompts, 128-token completions), vLLM collocated achieves 6 requests/second while meeting both a 500ms TTFT SLA and a 50ms TPOT SLA. DistServe disaggregated achieves 27 requests/second under the same SLAs — 4.48× higher goodput. The gain is larger for longer prompts (code completion, document QA), reaching 10× for workloads where prompts are 2,048 tokens.
The disaggregation mechanism
DistServe's architecture splits the serving cluster into two pools: prefill instances and decode instances.
A new request arrives at the router and is dispatched to a prefill instance. The prefill instance runs the full forward pass for the prompt, generating the KV cache for all prompt tokens. It generates the first output token as a side effect (the prefill pass produces an output logit, from which the first token is sampled). Then — and this is the critical step — it transfers the KV cache across the network to a decode instance. The decode instance resumes generation from that KV cache, producing subsequent tokens one by one, sending streaming output back to the client. The prefill instance is immediately available for the next request.
The invariant: no prefill computation ever runs on a decode instance's GPU. No long-running decode sequence ever competes for GPU time with a prefill operation.
KV cache transfer
The KV cache transfer is load-bearing and expensive. For LLaMA-70B:
KV cache per token = 2 (key + value) × 80 layers × 8 KV heads × 128 head_dim × 2 bytes (FP16)
= 327,680 bytes ≈ 320 KB per token
A 1,500-token prompt generates ~480 MB of KV cache. At 100 GB/s (PCIe 4.0 bandwidth between two servers), that transfer takes ~4.8ms. At 400 GB/s (InfiniBand HDR), it's ~1.2ms. At 900 GB/s (NVLink 4.0 within a node), ~0.5ms.
For short prompts (64 tokens), the KV cache is ~20 MB — negligible. For long prompts (4,096 tokens), it's ~1.3 GB, and the transfer time at PCIe bandwidth becomes 13ms — enough to noticeably inflate TTFT. The paper's measurements assume InfiniBand interconnect (100 GB/s+) in the datacenter; the absolute TTFT improvements are hardware-dependent.
The transfer is implemented via RDMA (Remote Direct Memory Access) where available: the prefill GPU writes directly to the decode GPU's HBM without involving the CPU or main memory. This bypasses OS overhead and achieves near-hardware-limit bandwidth. Without RDMA, you're going through the CPU, through main memory, across the network — expect 2-4× higher latency.
Routing and placement policy
The router needs to decide which prefill instance and which decode instance to assign each request. The paper describes two strategies:
Load balancing across prefill instances: new requests are dispatched to the prefill instance with the shortest estimated queue time. Prefill queue time is roughly proportional to the sum of prompt lengths of queued requests, since prefill throughput is roughly linear in token count.
Decode instance affinity: the prefill instance transfers KV cache to the decode instance with the most available KV cache capacity. This is a capacity-first policy; it minimizes the probability of decode-side KV cache eviction.
The paper also addresses a subtlety: when a decode instance is overloaded, it must preempt or migrate requests. In vLLM, preemption means recomputing the prompt KV cache on the same machine. In DistServe, preemption would mean re-transferring the KV cache from... where? The prefill instance no longer has it; it was freed after the transfer. The paper's approach is to fall back to recomputing from the original prompt, which the decode instance retained. The cost is a full prefill pass on the decode GPU — which is more expensive than on a prefill-optimized GPU, but avoids re-routing. In practice, the paper's evaluation shows this happens rarely under good placement policies.
Independent scaling
The key operational property: prefill instances and decode instances can be scaled independently, and can run on different hardware.
If your workload shifts to longer prompts (higher prefill compute load), add prefill instances. If you're hitting KV cache memory limits during long multi-turn conversations (decode bottleneck), add decode instances. With collocated serving, these two dimensions are coupled; you add whole GPU nodes and get both capacity increases together, which is rarely what you need.
The paper shows hardware heterogeneity is also possible: prefill instances can be A100 SXM (higher compute density, better for large matrix ops) while decode instances are A100 PCIe (lower cost, sufficient compute for per-token ops). This is a hardware provisioning optimization that only makes sense once disaggregation is in place.
Goodput: the metric that makes this paper coherent
The paper introduces goodput as its primary metric, which is more useful than throughput for production SLO analysis.
Goodput is the fraction of requests that meet both TTFT and TPOT latency SLAs simultaneously, multiplied by request rate. A system processing 100 requests/second where 80% meet both SLAs has goodput of 80 req/s. A system processing 200 requests/second where 30% meet both SLAs has goodput of 60 req/s — lower goodput despite higher raw throughput.
This matters because collocated systems often look good on aggregate throughput while failing silently on SLO compliance. When prefill stalls decode, TPOT for affected requests may exceed the SLA even if average TPOT is within bounds. The request completed, it counted in throughput, but it violated the SLA the user actually cares about. Goodput captures this; throughput doesn't.
In the paper's experiments, comparing goodput at matched SLAs (500ms TTFT, 100ms TPOT) shows disaggregated DistServe achieving 4-10× higher goodput than vLLM depending on workload. For operators tracking SLO compliance as a first-class business metric (rather than average latency), this reframing is immediately legible.
Production tradeoffs that aren't in the paper
Chunked prefill achieves some of the same benefit at lower operational cost. Instead of disaggregating prefill onto separate instances, chunked prefill splits the prompt into smaller chunks (256–512 tokens each) and interleaves prefill chunks with decode steps. This prevents prefill from monopolizing the GPU for 700ms; each chunk takes ~80ms, decode gets GPU time between chunks, and TTFT p99 smooths out. The bandwidth cost of moving KV cache across the network doesn't exist. vLLM supports chunked prefill; it was added after the original PagedAttention paper. For many workloads — particularly those with moderate prompt lengths (under 1,000 tokens) and modest concurrency — chunked prefill solves the TTFT/TPOT interference problem without disaggregation overhead. DistServe's gains are largest where chunked prefill struggles: very long prompts (2,048+ tokens) and high concurrency.
KV transfer cost is not always negligible. The paper's experiments use InfiniBand HDR (200 Gb/s effective ~24 GB/s). Many production clusters use 100GbE or 25GbE networking. At 25GbE (~3 GB/s usable), a 500 MB KV cache transfer takes 167ms — comparable to the TTFT you're trying to reduce. Disaggregation is only beneficial if your interconnect can transfer the KV cache in under ~20ms, which requires either short prompts, high-bandwidth networking, or co-located servers (preferably same rack, NVLink).
Stateful decode sessions complicate routing. For multi-turn chat applications, each turn in a conversation arrives as a new request but needs the KV cache from previous turns. With collocated serving, KV cache from prior turns is already on the same GPU. With disaggregation, prior-turn KV cache is on the decode instance, but the new turn's prefill must happen on a prefill instance. Options: ship the prior-turn KV cache from decode to prefill (expensive), run prefill on the decode instance (defeats disaggregation), or cache prior-turn KV on a shared storage layer (adds latency). The paper doesn't deeply address this — it models independent requests. Multi-turn chat is a real engineering problem for any DistServe implementation.
Request lifecycle is more complex to instrument. With collocated serving, each request has a single owner GPU from arrival to completion. Traces, metrics, and logs are all local to one process. With disaggregation, each request involves at least two GPU instances, a network transfer, and a router. Tracing across this boundary requires distributed trace propagation (Jaeger/OTLP span linking). If you don't build this from the start, debugging TTFT regressions means correlating logs from three separate services — I've done this, it's painful.
GPU utilization metrics become misleading. With collocated serving, GPU utilization is a reasonable proxy for load. With disaggregation, you have separate utilization numbers for prefill and decode pools. It's easy to be at 95% prefill utilization (TTFT degrading) and 40% decode utilization simultaneously, or vice versa. Your autoscaler and alerting need to understand these as separate capacity dimensions. A naive "average GPU utilization across cluster" alert will miss the asymmetry.
Failure modes in practice
KV cache network saturation under bursty prefill load. If 20 requests arrive simultaneously and all complete prefill at the same time, 20 × 480 MB = 9.6 GB of KV cache hits the network simultaneously. At 25 GB/s InfiniBand, that's ~400ms of network saturation. Subsequent requests queue behind this burst. The paper assumes smooth Poisson arrivals; production has bursty arrivals correlated with user behavior. Building backpressure into the KV transfer path — rate limiting, queuing at the prefill side before initiating transfer — is necessary.
Decode instance KV cache exhaustion causing cascading re-prefill. If decode instances fill their KV cache (all in-flight sequences occupy all KV blocks), new KV cache transfers can't land. The transfer blocks; the prefill instance can't free the just-completed prefill GPU memory because the transfer hasn't completed; that prefill instance's queue backs up. The fix is pre-reservation: before prefill begins, the router reserves space on a decode instance for the expected KV cache. If no reservation is available, the router queues the request rather than starting prefill. This requires coordination state between the router and decode instances, which is non-trivial distributed state management.
Partial transfer failures. If a network timeout or CRC error interrupts a KV cache transfer mid-flight, the decode instance has a partial KV cache that can't be used. The recovery options are: retry the full transfer (adds latency, re-uses network bandwidth), recompute the KV cache from the prompt on the decode instance (adds CPU/GPU time), or drop the request and return an error. The paper doesn't discuss retry semantics. Production implementations need a defined error handling policy here, because partial KV transfers happen at nontrivial rates on busy InfiniBand fabrics.
When NOT to use disaggregated prefill-decode
When your prompts are short. If median prompt length is under 200 tokens, the KV cache per request is ~64 MB, the prefill duration is ~90ms, and chunked prefill handles the interference problem adequately without the operational complexity of disaggregation. Disaggregation's benefit grows quadratically with prompt length (longer prompt → more KV to transfer but also much more benefit from not stalling decode). Below ~500-token prompts, chunked prefill is typically sufficient.
When you have slow inter-GPU networking. If prefill and decode instances communicate over standard 10GbE or 25GbE Ethernet without RDMA, KV cache transfers add hundreds of milliseconds of latency that erase the TTFT benefit. DistServe requires NVLink (for same-node disaggregation), InfiniBand, or RDMA over Ethernet at 100 Gb/s+ to achieve the transfer times the paper assumes.
When you need single-instance simplicity. Disaggregation adds a router, a transfer protocol, reservation state, and distributed tracing to your serving stack. For a startup with two GPU servers, the operational overhead — monitoring two separate pools, managing KV transfer quotas, handling partial failures — likely exceeds the TTFT benefit. Start with vLLM + chunked prefill; add disaggregation when you have the engineering capacity to run it and the scale where it pays off.
When multi-turn chat is your primary workload. As discussed above, disaggregation's stateless-request model doesn't cleanly handle sessions with shared KV cache across turns. If 90% of your traffic is multi-turn conversations where every turn reuses prior context, you'll spend most of your network bandwidth re-transferring KV cache that was already on the decode instance. A hybrid approach — colocate for multi-turn, disaggregate for single-turn — is possible but complex.
When your TPOT SLA is loose and TTFT is acceptable. If users are tolerant of 3-second TTFT but very sensitive to output streaming smoothness (TPOT < 30ms), the relevant interference to fix is decode-decode, not prefill-decode. Disaggregation fixes the prefill-on-decode interference; it doesn't improve per-token decode latency when decode instances are fully loaded. Know which dimension of your SLA is violated before choosing the architecture.
What the paper actually gives you
DistServe is a proof that heterogeneous serving pools are the right architecture for LLM inference at scale, even when that feels overengineered compared to vLLM's elegant single-process design.
The insight is not novel in distributed systems — separating read-heavy and write-heavy workloads onto different infrastructure is standard OLTP/OLAP practice. Separating compute-heavy preprocessing (ETL) from latency-sensitive serving is standard ML infrastructure. The LLM community has been slower to apply this because vLLM's collocated design works well enough at moderate scale, and the systems cost of disaggregation is real.
But the operational math changes at large scale. A 4× goodput improvement at matched SLAs means your inference cluster can be 4× smaller for the same served load. At $2/hour per A100 GPU, a 100-GPU cluster running DistServe replaces a 400-GPU collocated cluster. The engineering cost of building and operating disaggregated serving — estimated at 3-6 engineer-months in my experience — amortizes quickly against hardware savings.
For your specific situation: if your serving cluster is processing more than ~50 requests/second of long-prompt workloads and your TTFT p99 consistently exceeds 2× your p50, the prefill-decode interference problem is real and disaggregation is worth seriously evaluating. Start by enabling chunked prefill in your existing vLLM deployment; if that doesn't fix it, measure your inter-GPU bandwidth and model the KV cache transfer cost for your typical prompt lengths. If the math works out, DistServe's design is the right direction.
The four-second TTFT on our coding assistant? We implemented chunked prefill first — TTFT p99 dropped to 1.8 seconds. Then we disaggregated prefill and decode onto separate A100 pools connected via InfiniBand — p99 dropped to 350ms. Same model, same hardware budget, different topology. The GPU was always capable of it; we were just asking it to do two incompatible jobs simultaneously.
DistServe: Disaggregating Prefill and Decoding for Goodput-Optimized Large Language Model Serving — Zhong, Liu, Chen, Hu, Zhu, Liu, Jin, Zhang. OSDI 2024.