EdgePrompt: A Distributed Key-Value Inference Framework for LLMs in 6G Networks

Jiahong Ning, Pengyan Zhu, Ce Zheng, Gary Lee, Sumei Sun, Tingting Yang · 2025

Read EdgePrompt: A Distributed Key-Value Inference Framework for LLMs in 6G Networks by Jiahong Ning, Pengyan Zhu, Ce Zheng, Gary Lee, Sumei Sun, Tingting Yang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-11729.txt.

Introduction

Jiahong Ning and co-authors published a 2025 paper on running large language models inside future 6G mobile networks. Network operators hope these models will help manage the network and add smart services. Standard model designs, though, are slow and can leak data, and long inputs make both problems worse. Sixth-generation networks demand quick, safe responses.

Their framework, EdgePrompt, splits work between the cloud and edge devices near the user. It stores and reuses parts of the model's attention work in a distributed way, tuned to network conditions. That speeds up answers and raises throughput. A privacy step keeps sensitive information apart during processing, cutting the risk of leaks. Tests on a public dataset showed better speed and lower delay than the standard setup. The authors present the design as a workable base for language models in 6G settings. It targets the strict timing and security needs of live networks.

Catalog record

CREATORJiahong Ning, Pengyan Zhu, Ce Zheng, Gary Lee, Sumei Sun, Tingting Yang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-11729

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

EdgePrompt: A Distributed Key-Value Inference Framework for LLMs in 6G Networks Thanks: Jiahong Ning, Tingting Yang and Pengyan Zhu are with Dalian Marine University, School of navigation, China (Email: [email protected], [email protected]). Ce ZHENG is with Pengcheng Lab, Shenzhen, China ([email protected]). Sumei Sun, and Gary Lee are with the Institute for Infocomm Research (I2R), A*STAR, Singapore (e-mail: [email protected], [email protected]), Jiahong Ning is also with the Institute for Infocomm Research (I2R), A*STAR, Singapore ([email protected]), and Pengcheng Lab, Shenzhen, China ([email protected]). This work has been accepted by the IEEE INFOCOM 2025 Workshop on PerAI-6G. The final publication will be available via IEEE Xplore Jiahong Ning, Pengyan Zhu, Ce Zheng, Gary Lee, Sumei Sun, Tingting Yang Abstract As sixth-generation (6G) networks advance, large language models (LLMs) are increasingly integrated into 6G infrastructure to enhance network management and intelligence. However, traditional LLMs architecture struggle to meet the stringent latency and security requirements of 6G, especially as the increasing in sequence length leads to greater task complexity. This paper proposes EdgePrompt, a cloud-edge collaborative framework based on a hierarchical attention splicing mechanism. EdgePrompt employs distributed key-value (KV) pair optimization techniques to accelerate inference and adapt to network conditions. Additionally, to reduce the risk of data leakage, EdgePrompt incorporates a privacy preserving strategy by isolating sensitive information during processing. Experiments on public dataset show that EdgePrompt effectively improves the inference throughput and reduces the latency, which provides a reliable solution for LLMs deployment in 6G environments. K eywords Large Language Model (LLMs), Key-value pair (KV pair), The sixth-generation (6G), EdgePrompt 1 Introduction The advent of Large Language Models (LLMs), such as GPT-4 and LLaMA, has revolutionized the field of natural language processing, excelling in tasks such as text generation, question answering, and conversational systems [ 1 ] . These models have been widely adopted across various industries, including education, healthcare, and finance. As we approach the era of sixth-generation (6G) communication networks, the integration of LLMs into 6G infrastructure presents a transformative opportunity. Integrating LLMs within 6G architecture could enable intelligent network management, data analytics, and personalized experiences. However, deploying LLMs in the 6G infrastructure also introduces several challenges. First, LLMs inference incurs significant latency due to computational demands introduced by sequential processing [ 2 , 3 ] . This latency poses a critical challenge for real-time 6G applications such as autonomous driving and virtual reality. Second, transmitting user data to cloud-based LLMs for inference raises severe privacy issues, because sensitive information may be exposed during transmission and processing [ 4 , 5 ] . Recently, researchers focus on model compression technology and the deployment on the edge, in order to reduce network latency and enhance data security. According to [ 6 ] , LLMs inference is often hindered by high memory access demands or insufficient compute resources, referred to as being memory-bound or compute-bound, respectively. To tackle the memory-bound issue, [ 7 ] proposed the quantization aware training (QAT), which embeds low-precision representations directly into the training pipeline, providing faster device inference with minimal accuracy loss. The paper [ 8 ] leverages weight-only quantization to optimize matrix multiplications in LLMs, achieving reduced latency and enhanced computational efficiency. However, in the transformer architecture, both the quadratic complexity of self-attention with sequence length and the increased computational load of large batches shift the bottleneck from memory bound to compute bound. Even aggressive quantization fails to alleviate system limitations. Quantization reduces model size and memory usage to accelerate inference in memory-bound scenarios [ 9 ] , but offers limited benefits in compute-bound cases. As a result, researchers resort to parallel decoding approaches. One of the viable solutions is speculative sampling [ 10 ] , where a sequence of candidate tokens is generated serially by the draft model and then verified in parallel by the target model. This significantly accelerate inference without any change to the model architectures, training regimes and output distributions. Accordingly, a new paradigm of cloud-edge collaboration was proposed: a fast small model with low computational cost resides on device, while a large language model is deployed in the cloud, which has sufficient memory and computational resources [ 11 , 12 , 13 ] . However, the inference acceleration achieved through speculative decoding should offset the latency introduced by data exchange between cloud and edge. What is more, cloud-edge interactions increase the risk of sensitive data exposure. It introduces additional latency and synchronization challenges due to frequent data exchanges and alignment requirements. And the cloud-edge interactions increase the risk of sensitive data exposed, which limited the envision of 6G network. To address this problem, we propose EdgePrompt , a novel cloud-edge collaborative inference architecture for communication networks. To balance inference efficiency and privacy protection, we propose EdgePrompt , a novel cloud-edge collaborative inference architecture for communication network. Unlike traditional task offloading, EdgePrompt separates sensitive data from cloud prompts, ensuring that sensitive information stays on edge devices, while leveraging powerful cloud resources for large-scale computation. EdgePrompt not only enhances privacy protection but also improve bandwidth and compute bottlenecks. By the high throughput and low latency of 6G networks, EdgePrompt dynamically adapts to varying traffic and workload patterns. Moreover, as a complementary solution, EdgePrompt complements existing compression and parallelization strategies. As shown in Fig. 1a , the EdgePrompt architecture ensures that sensitive data remain on local devices, significantly mitigating privacy risks by localized data processing. By partitioning the computationally intensive processing of cloud prompts to the cloud, EdgePrompt balances the computational load on edge devices. Additionally, the exchange of KV pair minimizes communication overhead, resulting in decreased inference latency and improved throughput compared to traditional scheme. This strategic utilization facilitates the efficient handling of complex LLM tasks in 6G networks, thereby meeting the demands of applications. Our primary contributions are as follows: • Proposed a novel architecture for inference efficiency and privacy protection: We introduce EdgePrompt, a cloud-edge collaborative inference framework designed for LLMs foundation model into communication networks. By separating cloud and edge prompt and managing KV Cache locally, this design enhances inference efficiency while ensuring privacy. • Theoretical and empirical analysis: We conduct a detailed theoretical and performance analysis of EdgePrompt, including the establishment of optimization models and rigorous modeling specific to communication network environments. Our analysis demonstrates the practicality and effectiveness of EdgePrompt in optimizing LLMs inference under cloud-edge collaboration. • Performance improvements: We design experiments and analysis to demonstrate that EdgePrompt achieves significant improvements in system throughput and user data privacy within 6G networks, providing a robust solution for deploying LLMs in privacy sensitive and latency critical applications. 2 Methodology In this section, we elaborate on the models and methods we have established. Section 2.1 provides a brief analysis of the inference process. Section 2.2 delves into the mathematical expressions to analyze the problems and presents our solutions. Section 2.3 details the design and analysis of the EdgePrompt system modules with time slot. (a) The workflow of EdgePrompt (b) Attention structure of EdgePrompt Figure 1: The EdgePrompt architecture 2.1 The processing of inference The inference process of LLMs follows an autoregressive generation paradigm, where each token is generated sequentially based on preceding tokens in the context. This process comprises two stages: Prompting stage , where input sequences are processed in parallel to generate query (Q)

Read the complete text