GreenLLM: Disaggregating Large Language Model Serving on Heterogeneous GPUs for Lower Carbon Emissions

Tianyao Shi, Yanran Wu, Sihang Liu, Yi Ding · 2024

Read GreenLLM: Disaggregating Large Language Model Serving on Heterogeneous GPUs for Lower Carbon Emissions by Tianyao Shi, Yanran Wu, Sihang Liu, Yi Ding (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-20322.txt.

Description

Abstract LLMs have been widely adopted across many real-world applications. However, their widespread use comes with significant environmental costs due to their high computational intensity and resource demands. Specifically, this has driven the development of new generations of high-performing GPUs, exacerbating the problem of electronic waste and accelerating the premature disposal of devices.

Catalog record

CREATORTianyao Shi, Yanran Wu, Sihang Liu, Yi Ding
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-20322

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

GreenLLM: Disaggregating Large Language Model Serving on Heterogeneous GPUs for Lower Carbon Emissions Tianyao Shi, Yanran Wu, Sihang Liu, Yi Ding Abstract LLMs have been widely adopted across many real-world applications. However, their widespread use comes with significant environmental costs due to their high computational intensity and resource demands. Specifically, this has driven the development of new generations of high-performing GPUs, exacerbating the problem of electronic waste and accelerating the premature disposal of devices. To address this problem, this paper focuses on reducing the carbon emissions of LLM serving by reusing older, low-performing GPUs. We present GreenLLM, an SLO-aware LLM serving framework designed to minimize carbon emissions by reusing older GPUs. GreenLLM builds on two identified use cases that disaggregate specific computations onto older GPUs, reducing carbon emissions while meeting performance goals. To deepen our understanding of the potential carbon savings from disaggregation, we also provide a theoretical analysis of its relationship with carbon intensity and GPU lifetime. Our evaluations show that GreenLLM reduces carbon emissions by up to 40.6% compared to running standard LLM serving on new GPU only, meeting latency SLOs for over 90% of requests across various applications, latency requirements, carbon intensities, and GPU lifetimes. * * footnotetext: These authors contributed equally. 1 Introduction Large language models (LLMs) have been widely adopted in various real-world applications for their ability to perform complex natural language processing tasks [ 44 , 39 , 28 ] . Despite their widespread use, serving LLMs causes significant environmental impact in terms of carbon emissions 1 1 1 Detailed definitions of various carbon emissions are provided in section 2.3 . . For example, processing a single ChatGPT prompt generates over 4 grams of CO 2 eq [ 47 ] , which is more than 20 times the operational carbon emission of a web search query [ 15 ] . Serving LLMs relies on high-performing but power-hungry GPUs. The rising demand for computation has driven the development of new generations of high-performing GPUs, whose manufacturing processes contribute significantly to environmental impact in terms of embodied carbon emissions [ 48 , 19 ] . Meanwhile, older GPUs become increasingly overlooked and inadequate for running the latest LLM serving systems, further exacerbating the growing problem of electronic waste (e-waste) [ 11 , 7 ] . Recent studies estimate that e-waste from generative artificial intelligence (AI) could reach 1.2 to 5.0 million tons between 2020 and 2030 [ 46 ] , with rapid advances in AI that accelerate obsolescence and premature disposal of devices [ 1 , 42 ] . In this paper, we tackle the challenge of reducing the total carbon emissions of LLM serving by reusing old, low-performing GPUs. Since embodied carbon emissions are amortized over a device’s lifetime, extending hardware lifetimes by reusing older GPUs will effectively reduce these emissions [ 18 ] . To this end, we explore GPU heterogeneity to enable the reuse of older GPUs for serving LLMs, while meeting performance goals. Specifically, we investigate strategies for disaggregating a portion of the computation of LLM serving to older GPUs to reduce total carbon emissions. We first explore allocating the prefill and decoding phases of LLM serving onto different types of GPUs. LLM serving operates in two phases—prefill and decoding (as detailed in section 2.1 ). The prefill phase processes the prompt, stores the context in the key-value (KV) cache, and is compute-bound, while the decoding phase generates tokens autoregressively and is memory-bound. Prior work has either separated the prefill and decoding phases on homogeneous GPUs to eliminate interference and improve performance [ 50 , 22 ] , or heterogeneous GPUs to reduce power and costs [ 34 , 16 ] . In contrast, we minimize carbon emissions by using heterogeneous GPUs, especially old, low-performing GPUs. We allocate the prefill phase to newer, more advanced GPU for faster prompt processing, while assigning the decoding phase to older, less advanced GPU to reduce embodied carbon emissions. We call this approach Disg-Pref-Decode , and our experiments show that it can effectively reduce total carbon emissions while meeting latency service level objectives (SLOs). While Disg-Pref-Decode demonstrates a feasible use case to reduce carbon emissions for LLM serving, it has a key limitation: the high network bandwidth requirements due to the large KV cache transfer between GPUs. For instance, disaggregating the prefill and decoding phases of a Llama-7B model on A100 and T4 GPUs requires bandwidth over 10Gbps even when the request rate of 1 prompt/s (see Figure 4 ). In practice, the carbon reduction benefits of Disg-Pref-Decode can diminish as the KV cache size increases, particularly because the network devices connecting heterogeneous GPUs can have slower data transfer speeds. This overhead can offset the gains achieved by Disg-Pref-Decode when high network bandwidth is not available. To address this limitation, we explore a second disaggregation use case based on speculative decoding for systems without high-bandwidth interconnect, while still reducing total carbon emissions. Speculative decoding accelerates LLM serving by running two LLM models concurrently [ 24 , 30 ] : a larger, slower model and a smaller, faster model. Instead of generating tokens sequentially as in standard LLMs, the smaller model generates a set of tokens, and the larger model verifies them in parallel. This approach reduces the computational load on the larger model by offloading initial predictions to the smaller model, allowing it to focus primarily on validation. Building on this concept, we introduce a disaggregated version of speculative decoding ( Disg-Spec-Decode ) using heterogeneous GPUs. In Disg-Spec-Decode, the larger model is assigned to the newer GPU, while the smaller model runs on the older GPU. Unlike Disg-Pref-Decode, Disg-Spec-Decode has a low network bandwidth requirement since it only transfers speculative tokens and the corresponding probability distribution between GPUs. Building on the two use cases we identified, we design and implement GreenLLM, an SLO-aware LLM serving system that minimizes total carbon emissions by leveraging heterogeneous GPUs. GreenLLM includes three components: a disaggregated system supporting various LLM serving configurations such as Disg-Pref-Decode and Disg-Spec-Decode; a profiler measuring performance and energy consumption across various request sizes and rates; and an SLO-aware scheduler that searches for the optimal configuration for minimal carbon emissions while meeting latency SLOs. To further understand disaggregation’s carbon savings, we theoretically compare standalone (new GPU only) and disaggregated (new and old GPUs) setups. We address three questions: 1) When does disaggregation yield greater carbon savings? 2) How does carbon intensity influence carbon savings? 3) How does GPU lifetime influence carbon savings? We evaluate GreenLLM on three real-world LLM applications: chatbots, programming assistants, and document summarization. Compared to running standard LLM serving on new GPU only, GreenLLM reduces carbon emissions by up to 40.6% while meeting latency SLOs. Additionally, GreenLLM achieves up to 27.9% carbon savings across regions and GPU lifetimes, even in low-carbon intensity settings (17 gCO 2 /kWh). We summarize the contributions of this paper as follows: • We leverage GPU heterogeneity to explore the reusing of older GPUs for LLM serving to reduce carbon emissions. • We identify two use cases, Disg-Pref-Decode and Disg-Spec-Decode, to enable the disaggregation of LLM serving on heterogeneous GPUs. • We address high bandwidth constraints by introducing a disaggregated speculative decoding approach with low network bandwidth requirements. • We

Read the complete text