Prompt Inversion Attack against Collaborative Inference of Large Language Models
Wenjie Qu, Yuguang Zhou, Yongji Wu, Tingsong Xiao, Binhang Yuan, Yiming Li, Jiaheng Zhang · 2025
Read Prompt Inversion Attack against Collaborative Inference of Large Language Models by Wenjie Qu, Yuguang Zhou, Yongji Wu, Tingsong Xiao, Binhang Yuan, Yiming Li, Jiaheng Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-09022.txt.
Introduction
Wenjie Qu and co-authors present a 2025 paper on a privacy risk in shared language model use. Big models cost a lot to run, so users may split the layers across many machines. Each host passes hidden values to the next. A bad host could try to rebuild the prompt a user typed from those values.
The team built such an attack and tested it on real data. Their method got far more words right than prior attacks did. On one data set with a large model, it reached high word accuracy where the best rival scored far lower. The result shows a real threat to shared systems. Hosts in such chains cannot be assumed safe by default.
Catalog record
| CREATOR | Wenjie Qu, Yuguang Zhou, Yongji Wu, Tingsong Xiao, Binhang Yuan, Yiming Li, Jiaheng Zhang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 80 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-09022
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Prompt Inversion Attack against Collaborative Inference of Large Language Models Wenjie Qu, Yuguang Zhou, Yongji Wu, Tingsong Xiao, Binhang Yuan, Yiming Li, Jiaheng Zhang Abstract Large language models (LLMs) have been widely applied for their remarkable capability of content generation. However, the practical use of open-source LLMs is hindered by high resource requirements, making deployment expensive and limiting widespread development. The collaborative inference is a promising solution for this problem, in which users collaborate by each hosting a subset of layers and transmitting intermediate activation. Many companies are building collaborative inference platforms to reduce LLM serving costs, leveraging users’ underutilized GPUs. Despite widespread interest in collaborative inference within academia and industry, the privacy risks associated with LLM collaborative inference have not been well studied. This is largely because of the challenge posed by inverting LLM activation due to its strong non-linearity. In this paper, to validate the severity of privacy threats in LLM collaborative inference, we introduce the concept of prompt inversion attack (PIA), where a malicious participant intends to recover the input prompt through the activation transmitted by its previous participant. Specifically, we design a two-stage method to execute this attack. In the first stage, we optimize the input embedding with a constraint term derived from the LLM’s embedding matrix to enforce the optimized embedding to be close to the ground truth. In the second stage, we accurately recover discrete tokens by incorporating activation calibration and semantic speculation. Extensive experiments show that our PIA method substantially outperforms existing baselines. For example, our method achieves an 88.4% token accuracy on the Skytrax dataset with the Llama-65B model when inverting the maximum number of transformer layers, while the best baseline method only achieves 22.8% accuracy. The results verify the effectiveness of our PIA attack and highlights its practical threat to LLM collaborative inference systems. † † ∗ The first three authors contributed equally. I Introduction (a) Fig. 1: Outline of our attack setting. In collaborative inference, the LLM’s transformer blocks are distributed across collaborating participants, each holding a portion of the model. In this figure, ’ACT’ represents the intermediate activation transmitted between participants. User 3 is a malicious participant who records the intermediate activation received from User 2, completing the computation as usual. After inference, User 3 then attempts to reconstruct the input prompt using the recorded activation. Large language models (LLMs) ( e.g. , GPT-4 [ 1 ] ), have demonstrated exceptional abilities in various domains. The model scale of LLMs has expanded extensively in recent years. Many LLMs with hundreds of billions of parameters, such as Llama2-70B [ 2 ] and OPT-175B [ 3 ] , have been released. The practical usage of LLMs is hindered by high memory and computational requirements. For instance, OPT-175B demands over 350 GB of GPU memory for inference. Deploying LLMs necessitates many expensive GPUs, making them unaffordable for most developers. This requirement heavy burden significantly hinders the widespread deployment of LLMs in practice. Recent studies have been conducted to democratize LLM’s access. They can be mainly divided into two categories: offloading and collaborative inference. Offloading-based systems [ 4 , 5 , 6 ] reduce demand on GPU memory by leveraging CPU memory to store model weights that exceed GPU’s capacity. However, they suffer from significant serving latency due to frequent transmission of large model weights ( e.g. , gigabytes in size) between CPU and GPU. Collaborative inference [ 7 , 8 , 9 ] is an alternative to offloading, which addresses this challenge. In collaborative inference, participants connect their underutilized GPUs to a pool. Each participant hosts a subset of consecutive LLM layers, receives the preceding layers’ activation, computes the hosted layers, and sends the calculated activation to the subsequent participant for follow-up computations. Similar to BitTorrent [ 10 ] , LLM users benefit from other peoples’ GPUs while they can also contribute their own GPUs to the pool. The data transmission volume of collaborative inference is typically orders of magnitude smaller than that of offloading, leading to much lower overhead. Several reports [ 11 , 12 ] have pointed out that collaborative inference can output about five tokens per second, similar to human typing speed, which is acceptable in the real world. Collaborative inference has recently attracted much attention from industry and academia [ 7 , 13 , 9 , 14 ] . Multiple companies are building collaborative inference platforms to meet the demand for AI computing power. For example, Huggingface [ 15 ] and Yandex research [ 16 ] built PETALS [ 7 , 8 ] , enabling users to share their underutilized GPUs through the network to offer people access to cheap computing power. An influential open source project LocalAI [ 17 ] also added support to collaborative inference to reduce cost and enhance efficiency [ 18 ] . Meanwhile, the startup Nesa [ 19 ] has combined collaborative inference with blockchain technology to improve censorship resistance. In academia, extensive research [ 9 , 13 , 14 , 20 ] has focused on enhancing the efficiency and scalability of collaborative inference. Additionally, several studies [ 21 , 22 , 23 ] have explored the specific case of LLM collaborative inference in edge cloud environments. However, due to the complex nature of the internet, malicious users are likely to exist. For example, a recent work [ 24 ] (ICML’24) studied the risk of poisoning under collaborative fine-tuning. In particular, there are two potential threats collaborative inference systems may encounter: (1) utility threats that a malicious client might intentionally delay processing to increase inference latency or alter the activations sent to subsequent participants, thereby manipulating the generated text and (2) privacy threats that infer sensitive information from the activation received from the previous participant. This paper mainly focuses on the privacy threats in LLM collaborative inference since input prompts often contain sensitive information ( e.g. , personal names and contact details) and high-quality LLM prompts themselves hold significant commercial value [ 25 ] . To the best of our knowledge, there is no research regarding the privacy threats in collaborative LLM inference. Nevertheless, we notice that some techniques, including attribute inference attack [ 26 , 27 , 28 ] and embedding inversion attack [ 29 , 30 , 31 ] , are closely related to it. These attacks have similar threat models with collaborative inference where the attacker is given embedding/activation. Specifically, attribute inference attacks can accurately infer various sensitive categorized attributes ( e.g. , location and gender) from language embedding with high accuracy. However, they are unlikely to practically threaten LLM collaborative inference , since LLM input prompts are typically long and often contain multiple different values for a single attribute, making the attack results confusing. This naturally mitigates the threat of attribute inference. Embedding inversion attack directly recovers input text from embeddings and has been shown to recover a certain proportion of input from BERT [ 32 ] embeddings. However, we reveal that embedding inversion attacks achieve low accuracy in recovering prompts from LLM activations. The fundamental challenge comes from strong non-linearity of LLMs due to deep depth and large size. We discover in our experiments that the non-linearity of LLM strengthens with the increase of model depth. The strong