DiSCo : Device-Server Collaborative LLM-Based Text Streaming Services

Ting Sun, Penghan Wang, Fan Lai · 2025

Read DiSCo : Device-Server Collaborative LLM-Based Text Streaming Services by Ting Sun, Penghan Wang, Fan Lai (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-11417.txt.

Introduction

Ting Sun, Penghan Wang, and Fan Lai build DiSCo, a tool that splits text work for large language models. Part runs on the user's own device. Part runs on a far-off server. The work came out in 2025. Text apps serve huge crowds each day. They need fast first words and a smooth flow of text.

A server alone costs a lot and can lag on the last link to the user. A device alone is fast to start but has tight limits. DiSCo sends each job to the side that fits best. It can also move a reply mid-stream from one side to the other. It weighs cost, the wait for the first token, and the pace of later tokens. The aim is a good feel for users at a cost that stays in bounds.

Catalog record

CREATORTing Sun, Penghan Wang, Fan Lai
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-11417

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DiSCo : Device-Server Collaborative LLM-Based Text Streaming Services Ting Sun, Penghan Wang, Fan Lai Abstract The rapid rise of large language models (LLMs) in text streaming services has introduced significant cost and Quality of Experience (QoE) challenges in serving millions of daily requests, especially in meeting Time-To-First-Token (TTFT) and Time-Between-Token (TBT) requirements for real-time interactions. Our real-world measurements show that both server-based and on-device deployments struggle to meet diverse QoE demands: server deployments face high costs and last-hop issues (e.g., Internet latency and dynamics), while on-device LLM inference is constrained by resources. We introduce DiSCo , a device-server cooperative scheduler designed to optimize users’ QoE by adaptively routing requests and migrating response generation between endpoints while maintaining cost constraints. DiSCo employs cost-aware scheduling, leveraging the predictable speed of on-device LLM inference with the flexible capacity of server-based inference to dispatch requests on the fly, while introducing a token-level migration mechanism to ensure consistent token delivery during migration. Evaluations on real-world workloads—including commercial services like OpenAI GPT and DeepSeek, and open-source deployments such as LLaMA3—show that DiSCo can improve users’ QoE by reducing tail TTFT (11-52%) and mean TTFT (6-78%) across different model-device configurations, while dramatically reducing serving costs by up to 84% through its migration mechanism while maintaining comparable QoE levels. 1 Introduction Large language models (LLMs) have revolutionized various applications, with over 60% focusing on conversational interactions such as chatbots ( Grand View Research, 2023 ) . Meeting high serving demands requires scaling deployments across on-premise servers in the cloud and on-device inference, as seen in Apple Intelligence ( Gunter et al., 2024 ) and Google’s Gemini Nano ( Google, 2024 ) . The Quality of Experience (QoE) for interactive applications is primarily evaluated by two critical metrics: Time-To-First-Token (TTFT) in the prefill stage, which quantifies the initial response latency, and Time-Between-Token (TBT) during the decode stage, which measures the consistency of token delivery speed ( Databricks, 2023 ; Liu et al., 2024a ; Liu et al., 2024c ) . On-server deployments lower serving costs by sharing infrastructure among many requests but often introduce unpredictable high latency due to request queuing delays ( Agrawal et al., 2024 ) and the internet speed fluctuations. While on-device deployment is able to serve increasingly capable LLMs with sufficient accuracy, it suffers from slow processing speeds for long prompts and high energy consumption. For example, a fully-charged iPhone can only operate for less than two hours running an LLM with 7B parameters ( Liu et al., 2024d ) . This paper introduces a novel paradigm for cost-constrained device-server cooperative inference. We incorporate both server usage (e.g., monetary costs) and device energy costs via a dynamic exchange rate, which can be adjusted by endpoint users to balance response generation between the cloud and devices. As such, we can strategically distribute inference requests between endpoints and dynamically migrate ongoing token generation to maximize QoE. However, realizing this vision presents several fundamental challenges: • Unified Cost Management: Total serving costs include resource expenditures from both endpoints—monetary costs from server API usage and energy costs from device computation. The relative value of energy costs varies dynamically based on device context (e.g., battery level, charging status) and user preferences for server spending, making it challenging to establish a unified optimization strategy. • Runtime Uncertainty: The dynamic nature of networks (e.g., latency jitters) and serving loads make it challenging to accurately predict TTFT for in-flight request migration. Moreover, any scheduling mechanism must be lightweight to avoid introducing large overhead to the already latency-sensitive services. • Migration Impact on Token Delivery: While dynamic migration between endpoints can reduce overall operating costs, it risks disrupting TBT. The challenge lies in determining when and how to migrate while minimizing user experience degradation and cost increase. Figure 1: DiSCo acts as a middleware to optimize QoE by adaptively dispatching and migrating response generation between device and server endpoints under cost constraints. As shown in Figure 1 , we introduce DiSCo , a D ev i ce- S erver Co operative scheduler that addresses these challenges via two key innovations: • Cost-Aware Dispatching Policies: We introduce two dispatching mechanisms targeting different cost constraints. For server cost constraints, we employ a length-threshold based dispatching mechanism that routes requests shorter than a dynamically computed threshold to devices. For device energy constraints, we implement a delay-based dispatching mechanism where devices wait for a computed interval before starting local inference. Both mechanisms adapt their thresholds based on unified cost measures that combine server monetary costs and device energy consumption. • Token-Level Migration Framework: We enable seamless generation handoff between endpoints through a novel migration protocol that preserves the consistency of token delivery. Our framework employs delayed migration timing to minimize interruption, while a token buffer ensures smooth delivery during transitions. This design maintains user experience while saving resource costs across endpoints. Through extensive evaluation using real-world traces from commercial LLM streaming API services, including GPT and DeepSeek, and on-device deployments, we demonstrate that DiSCo improves mean and tail TTFT by up to 50% without TBT violation, significantly reducing costs. Overall, we make the following contributions: • We characterize QoE challenges in device-server cooperative LLM inference through extensive real-world measurements. • We design novel scheduling policies that optimize QoE under cost constraints. • We develop a token-level migration framework to enable generation handoff between endpoints, preserving token delivery consistency. • We demonstrate DiSCo ’s effectiveness in commercial services and open-source benchmarks. 2 Background and Motivation 2.1 LLM Token Mixture and Routing Device-server collaborative approaches have evolved in two directions. First, systems like EdgeShard Zhang et al. (2024) and WDMoE ( Xue et al., 2024a ) partition LLMs across multiple endpoints when a single device cannot host the entire model. LLMCad Xu et al. (2023) uses on-device models to reduce server costs, while PerLLM ( Yang et al., 2024 ) optimizes energy consumption across devices and servers under constraints. Second, routing-based approaches ( Ong et al., 2024 ; Ding et al., 2024 ) balance cost and accuracy by directing simple requests to small models and complex queries to advanced ones. However, these approaches do not optimize token delivery metrics (TTFT and TBT) under cost constraints. 2.2 LLM-Based Text Streaming Applications Over 60% of LLM-backed applications focus on streaming conversational interactions, such as chatbots, virtual assistants, and language translation. QoE in these text streaming services is often quantified by two critical metrics: time-to-first-token (TTFT) for initial responsiveness and time-between-tokens (TBT) for delivery smoothness throughout the entire interaction timeline. Current LLM systems struggle to meet user expectations for these metrics, with TTFTs ranging from hundreds of milliseconds to over ten seconds—far exceeding the ideal latencies of tens of milliseconds for interactive applications Mäki-Patola and Hämäläinen (2004) ; Žádník et al. (2022) . Token consumption patterns vary by output modality: In visual text scenarios, reading speeds differ across demographic groups, with the majority (52%) aged 25-44 reading 4-5 tokens per second, while older groups generally read more slowly Liu et al. (2024a) ; Brysbaert (2019) ; Petrov et al. (2024) . Audio output consumption shows more consistency, averaging 3-4 tokens per second across languages Liu et al. (2024a) ; Parachuk (2022) ; Barnard, Dom (2022) . Notably, conventional evaluation metrics like token generation

Read the complete text