Prefill-Decode Aggregation or Disaggregation? Unifying Both for Goodput-Optimized LLM Serving
Chao Wang, Pengfei Zuo, Zhangyu Chen, Yunkai Liang, Zhou Yu, Ming-Chang Yang · 2025
Read Prefill-Decode Aggregation or Disaggregation? Unifying Both for Goodput-Optimized LLM Serving by Chao Wang, Pengfei Zuo, Zhangyu Chen, Yunkai Liang, Zhou Yu, Ming-Chang Yang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-01989.txt.
Introduction
Chao Wang and co-authors settle a serving debate for large language models, in a 2025 paper. Serving splits into a prefill stage, which reads the prompt, and a decode stage, which writes tokens. Keeping both on one machine favors fast first tokens, while splitting them across machines favors steady token flow. Each wins under unlike service targets.
Under balanced targets, though, neither wins. Their system, TaiChi, mixes both layouts using two machine kinds, one strong at prefill and one calm at decode. Three sliders tune the mix to the targets at hand. A trick called latency shifting moves spare power from requests already on target to those at risk. The result is top useful throughput across target mixes.
Catalog record
| CREATOR | Chao Wang, Pengfei Zuo, Zhangyu Chen, Yunkai Liang, Zhou Yu, Ming-Chang Yang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 73 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-01989
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Prefill-Decode Aggregation or Disaggregation? Unifying Both for Goodput-Optimized LLM Serving Chao Wang, Pengfei Zuo, Zhangyu Chen, Yunkai Liang, Zhou Yu, Ming-Chang Yang Abstract . There is an ongoing debate on whether prefill-decode (PD) aggregation or disaggregation is the superior approach for serving large language models (LLMs). This debate has driven optimizations on both sides, each showcasing distinct advantages. This paper presents a comprehensive comparison between PD aggregation and disaggregation, showing that each excels under different service-level objectives (SLOs): PD aggregation is optimal under tight time-to-first-token (TTFT) and relaxed time-per-output-token (TPOT), while PD disaggregation excels under strict TPOT and relaxed TTFT. However, under balanced TTFT and TPOT SLOs, neither approach can deliver optimal goodput. Based on these insights, this paper proposes TaiChi, an LLM serving system that unifies PD disaggregation and aggregation to achieve optimal goodput under any combination of TTFT and TPOT SLOs. TaiChi leverages a unified disaggregation-aggregation architecture composed of differentiated-capability GPU instances: prefill-heavy instances (fast prefill but high-interference decode) and decode-heavy instances (low-interference decode but slow prefill). It exposes three configurable sliders to control the ratio between prefill-heavy and decode-heavy instances, and the chunk sizes for each. TaiChi adapts to various SLO regimes by adjusting these sliders. When TTFT constraints are tight, TaiChi can be tuned to resemble a PD aggregation configuration; when TPOT dominates, it adapts toward PD disaggregation. Crucially, under balanced SLOs, TaiChi enables a hybrid mode that achieves superior goodput. The key innovation behind this hybrid mode is latency shifting: by selectively reallocating GPU resources from requests that meet TTFT or TPOT SLOs to those at risk of violation, TaiChi maximizes the number of SLO-satisfied requests. This fine-grained, request-level latency shifting is orchestrated through two targeted scheduling mechanisms: flowing decode scheduling to control TPOTs and length-aware prefill scheduling to manage TTFTs, jointly optimizing request assignment. Our extensive experimental results demonstrate that TaiChi improves goodput by up to 77% compared to state-of-the-art systems under balanced TTFT and TPOT SLOs. Keywords: Large Language Models, LLM Serving, Prefill-Decode, Service Level Objectives, Goodput, Latency Shifting 1. Introduction Large language models (LLMs) have demonstrated unprecedented performance across various applications, such as personal assistants ( AI 2023 ) , translation ( Rozière et al. 2024a ; Kleidermacher and Zou 2025 ) , document analysis ( Lin et al. 2024 ) , chatbots ( OpenAI 2022 ; OpenAI 2023 ) , and code generators ( Chen et al. 2021 ; Rozière et al. 2024b ) . Serving these LLM applications requires substantial and costly computational resources, particularly GPUs. Consequently, optimizing the LLM serving cost has received significant attention in system research ( Kwon et al. 2023 ; Yu et al. 2022 ; Agrawal et al. 2024b ; Zhong et al. 2024 ; Strati et al. 2024 ; Patel et al. 2024 ; Gao et al. 2024 ) . Meeting service level objective (SLO) constraints is critical for LLM service providers to ensure application-level performance ( Zhong et al. 2024 ; Agrawal et al. 2024b ) . In LLM serving, a user request is processed in two distinct phases, each with its own SLO constraint ( Zhong et al. 2024 ) . The first phase, known as prefill , involves preprocessing the user request and generating the first token with intensive computation. The latency of prefill is constrained by time-to-first-token (TTFT), reflecting system responsiveness ( Wang et al. 2024b ; Zhong et al. 2024 ) . The second phase, known as decode , outputs the subsequent tokens autoregressively (i.e., one token per iteration). The average time taken to output a token (except for the first token) is constrained by time-per-output-token (TPOT), which indicates the service speed users can experience ( Wang et al. 2024b ; Zhong et al. 2024 ) . The throughput of requests while meeting both SLO constraints is referred to as goodput ( Wang et al. 2024b ; Zhong et al. 2024 ) . Enhancing goodput with the same hardware resources lowers the cost per LLM request ( Zhong et al. 2024 ) . However, application-level SLOs vary widely ( Agrawal et al. 2024b ; Zhong et al. 2024 ; Hong et al. 2025 ; Zhang et al. 2025 ) : some prioritize low TTFT with relaxed TPOT, others require tight TPOT with relaxed TTFT, while many demand a balanced trade-off between the two. None Figure 1 . Distribution of requests’ TTFT and TPOT under different scheduling approaches, using the same number of compute nodes and QPS. (PD aggregation performs best when TTFT constraints are tight and TPOT is relaxed, while PD disaggregation excels under tight TPOT and relaxed TTFT. However, under balanced SLO constraints, PD aggregation results in TPOT violations due to high-interference decode, while PD disaggregation leads to TTFT violations due to low prefill processing capacity. By proposing a hybrid-mode inference, we mitigate both issues and achieve better SLO attainment across different combinations of TTFT and TPOT.) None Currently, an active debate is ongoing regarding whether PD aggregation ( Agrawal et al. 2024b ) or disaggregation ( Zhong et al. 2024 ) is the superior approach for serving LLMs. PD aggregation (like Orca ( Yu et al. 2022 ) and Sarathi-Serve ( Agrawal et al. 2024b ) ) co-locates the prefill and decode phases of a request on the same hardware instance to achieve high resource utilization. In contrast, PD disaggregation (like Splitwise ( Patel et al. 2024 ) and DistServe ( Zhong et al. 2024 ) ) physically separates prefill and decode onto different hardware instances. This approach eliminates interference between the two phases and allows for independent scaling of resources. The debate drives optimizations on both sides, each showcasing the distinct advantages ( Agrawal et al. 2024b ; Zhong et al. 2024 ; Patel et al. 2024 ) . In this paper, we present a comprehensive comparison between PD aggregation and disaggregation, showing that each approach achieves optimal request goodput under different SLOs: PD aggregation is optimal under tight TTFT and relaxed TPOT, while PD disaggregation excels under strict TPOT and relaxed TTFT, as illustrated in Figure 1 . PD aggregation achieves low TTFT by having all instances participate in the prefill phase, but suffers in TPOT due to interference between prefill and decode. In contrast, PD disaggregation improves TPOT by isolating prefill and decode on separate resources, but incurs higher TTFT since fewer instances handle prefill. However, under balanced TTFT and TPOT SLOs, neither approach can deliver optimal goodput. This is because PD aggregation tends to violate TPOT due to decode interference, while PD disaggregation often fails to meet TTFT, as only a subset of instances handle the prefill phase. To end this debate about these two branches, we propose TaiChi, an LLM serving system that unifies PD disaggregation and aggregation to achieve optimal goodput under any combination of TTFT and TPOT SLOs. TaiChi leverages a unified disaggregation-aggregation architecture composed of differentiated-capability GPU instances: P-heavy instances (fast prefill but high-interference decode) and D-heavy instances (low-interference decode but slow prefill). It exposes three configurable sliders to control the ratio between P-heavy and D-heavy instances, and the chunk sizes for each. TaiChi adapts to various SLO regimes by adjusting these sliders. When TTFT constraints are tight, TaiChi can be tuned to resemble a PD aggregation configuration; when