EdgeLoRA: An Efficient Multi-Tenant LLM Serving System on Edge Devices Conference: The 23rd Annual International Conference on Mobile Systems, Applications and Services; June 23–27, 2025; Anaheim, CA, USA The 23rd Annual International Conference on Mobile Systems, Applications and Services (MobiSys
Zheyu Shen, Yexiao He, Ziyao Wang, Yuning Zhang, Guoheng Sun, Wanghao Ye, Ang Li · 2025
Read EdgeLoRA: An Efficient Multi-Tenant LLM Serving System on Edge Devices Conference: The 23rd Annual International Conference on Mobile Systems, Applications and Services; June 23–27, 2025; Anaheim, CA, USA The 23rd Annual International Conference on Mobile Systems, Applications and Services (MobiSys by Zheyu Shen, Yexiao He, Ziyao Wang, Yuning Zhang, Guoheng Sun, Wanghao Ye, Ang Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-01438.txt.
Introduction
Running a fine-tuned language model on a phone or small edge box is hard enough; running many tenants' custom versions at once is harder. Zheyu Shen and co-authors present EdgeLoRA in 2025, a serving system for edge devices where each user brings a small LoRA adapter that tailors the shared base model.
Three ideas carry the design. An adaptive mechanism picks the right adapter for each request. Memory management caches and pools adapters to cut swapping costs. And batch LoRA inference processes requests together instead of one by one. On a Llama 3.1 model with eight billion parameters, the system reached up to four times the throughput of the standard local engine, and served orders of magnitude more adapters at once. The authors aim it at private, personal, low-latency AI on modest hardware.
Catalog record
| CREATOR | Zheyu Shen, Yexiao He, Ziyao Wang, Yuning Zhang, Guoheng Sun, Wanghao Ye, Ang Li |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 68 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-01438
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
EdgeLoRA: An Efficient Multi-Tenant LLM Serving System on Edge Devices Conference: The 23rd Annual International Conference on Mobile Systems, Applications and Services; June 23–27, 2025; Anaheim, CA, USA The 23rd Annual International Conference on Mobile Systems, Applications and Services (MobiSys ’25), June 23–27, 2025, Anaheim, CA, USA DOI: 10.1145/3711875.3729141 ISBN: 979-8-4007-1453-5/2025/06 CCS: Computing methodologies Natural language processing CCS: Human-centered computing Ubiquitous and mobile computing systems and tools Zheyu Shen, Yexiao He, Ziyao Wang, Yuning Zhang, Guoheng Sun, Wanghao Ye, Ang Li Abstract . Large Language Models (LLMs) have gained significant attention due to their versatility across a wide array of applications. Fine-tuning LLMs with parameter-efficient adapters, such as Low-Rank Adaptation (LoRA), enables these models to efficiently adapt to downstream tasks without extensive retraining. Deploying fine-tuned LLMs on multi-tenant edge devices offers substantial benefits, such as reduced latency, enhanced privacy, and personalized responses. However, serving LLMs efficiently on resource-constrained edge devices presents critical challenges, including the complexity of adapter selection for different tasks, memory overhead from frequent adapter swapping. Moreover, given the multiple requests in the multi-tenant settings, processing requests sequentially will result in underutilization of computational resources and significant latency. This paper introduces EdgeLoRA , an efficient system for serving LLMs on edge devices in multi-tenant environments. EdgeLoRA incorporates three key innovations: (1) an adaptive adapter selection mechanism to streamline the adapter configuration process; (2) heterogeneous memory management, leveraging intelligent adapter caching and pooling to mitigate memory operation overhead; and (3) batch LoRA inference, which enables efficient batch processing to significantly reduce computational latency. Comprehensive evaluations using the Llama3.1-8B model demonstrates that EdgeLoRA significantly outperforms the status quo ( i.e. , llama.cpp ) in terms of both latency and throughput. The results demonstrates EdgeLoRA could achieve up to 4 × \times boost in throughput with less energy consumption. Even more impressively, it manages to serve several orders of magnitude more adapters simultaneously without sacrificing inference performance. These results highlight EdgeLoRA ’s potential to transform edge deployment of LLMs in multi-tenant scenarios, offering a scalable and efficient solution for resource-constrained environments. † † cc-license: by 1. Introduction Large language models (LLMs) have emerged as transformative tools in natural language processing (NLP), excelling in diverse applications such as chatbots ( Adiwardana et al. 2020 ; Luo et al. 2022 ) , summarization ( Nallapati et al. 2016 ; El-Kassas et al. 2021 ; Widyassari et al. 2022 ) , code generation ( Roziere et al. 2023 ; Ouyang et al. 2024 ) , and captioning ( Yang et al. 2016 ) . Innovations from Anthropic ( Anthropic 2023 ) , Google ( Team 2024 ) , Meta ( Touvron et al. 2023 ; Grattafiori et al. 2024 ) , Microsoft ( Abdin et al. 2024 ) , and OpenAI ( Brown et al. 2020 ; OpenAI 2024 ) demonstrate their ability to reframe tasks like translation ( Brown et al. 1990 ) , classification ( Schick and Schütze 2020 ) , and question-answering ( Kwiatkowski et al. 2019 ) as language generation problems, achieving remarkable improvements. Beyond text, LLMs extend to image ( Dosovitskiy et al. 2021 ; Wang et al. 2022 ) , video ( Lian et al. 2023 ) , speech ( Shu et al. 2023 ) , and multimodal ( Zhang et al. 2024 ; Yin et al. 2024 ) generation. Their success is driven by the Transformer architecture ( Vaswani 2017 ) and its variants ( Liu et al. 2021b ; Chowdhery et al. 2023 ) , which effectively model complex relationships in sequences, outperforming traditional methods ( Graves and Graves 2012 ) . These advancements position LLMs as foundational to modern AI, transforming research and applications across domains. Although pretrained LLMs already demonstrate superior performance on general tasks, they can be further enhanced for domain-specific applications through fine-tuning. This adaptation process refines a pretrained LLM for optimal performance across diverse, specialized tasks. This pretrain-then-finetune paradigm has led to the proliferation of numerous fine-tuned variants of a single base LLM, each tailored to a specific task ( e.g. , dialogue ( Gupta et al. 2022 ) , writing ( Zhang et al. 2023b ) , and code generation ( Roziere et al. 2023 ; Ouyang et al. 2024 ) ) or domain ( e.g. , medical ( Li et al. 2023a ) , mathematics ( Tang et al. 2024 ; Liu and Low 2023 ) , and legal ( Yue et al. 2023 ) ). To address the computational overhead of fine-tuning, parameter-efficient methods like Low-rank adaptation (LoRA) have been developed ( Hu et al. 2021 ; Dettmers et al. 2024 ) . LoRA reduces computational demands by updating only a small portion of model parameters, which exploits the low dimensionality of parameter updates in fine-tuning and representing them with pairs of low-rank matrices, i.e. , LoRA adapters. Compared to fully fine-tuning, LoRA can reduce the number of training parameters by 10,000\times while maintaining comparable performance ( Hu et al. 2021 ) . Figure 1 . Multi-tenant LLM Serving on Edge Devices. The use of multi-tenant LLM applications on edge devices has grown significantly, spanning diverse domains such as personalized virtual assistants ( Li et al. 2024b ) , real-time translation tools ( Stahlberg 2020 ) , context-aware chatbots ( Adiwardana et al. 2020 ; Luo et al. 2022 ) , and on-device content moderation ( Zhang et al. 2023b ; Mysore et al. 2023 ) . These applications highlight the demand for flexible, efficient, and scalable serving strategies tailored for edge environments. To enable personalization and task-specific optimization, one promising solution involves serving a set of LoRA models fine-tuned on a shared base LLM on edge devices, as illustrated in Figure 1 . As presented in LoRA ( Hu et al. 2021 ) , merging LoRA adapters into the pretrained weights eliminates additional latency overhead during inference. However, serving multiple LoRA models concurrently introduces complexity, i.e. , adapters can be swapped by adding or subtracting LoRA matrices from the base model, but this approach incurs significant latency overhead and drastically reduces throughput. Alternatively, inference can proceed without merging by processing the base model and LoRA adapters in parallel, though this method also presents inefficiencies. Despite these approaches, serving multiple LoRA adapters with a shared base model on edge devices remains challenging: (1) users must manually select the appropriate adapter from a large pool, a process that is cumbersome and prone to errors; (2) efficient memory management is essential, particularly for swapping adapters between memory and disk to minimize resource overhead while increasing the number of adapters hosted by a single device; and (3) unmerged LoRA inference incurs substantial latency, as the parallel computation capabilities of edge devices are often underutilized when managing dynamic workloads. These challenges are particularly pronounced in the context of multi-tenant edge devices, which must efficiently manage multiple fine-tuned models concurrently. Status Quo and Its Limitations. Current LLM serving systems, such as vLLM ( Kwon et al. 2023 ) and SGLang ( Zheng et al. 2024 ) , are primarily designed to serve pretrained models efficiently. Some approaches, like SLoRA ( Sheng et al. 2023 ) , mitigate memory constraints by dynamically swapping LoRA adapters between memory and disk, avoiding the need to load