What Does the Server See? Understanding Privacy Leakage from Large Language Models in Split Inference DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Security and privacy CCS: Computing methodologies Machine learning

Mingyuan Fan, Yu Liu, Fuyi Wang, Cen Chen · 2026

Read What Does the Server See? Understanding Privacy Leakage from Large Language Models in Split Inference DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Security and privacy CCS: Computing methodologies Machine learning by Mingyuan Fan, Yu Liu, Fuyi Wang, Cen Chen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-23158.txt.

Introduction

Fan and three coauthors examine privacy leakage from large language models in split inference. There, models split between client and server, so only activations travel. They introduce ActInv, which matches activations to rebuild client inputs. It rebuilds well even against Gaussian noise and activation thinning. They develop the Perturbation Amplification Factor, a metric for a layer's defense to rebuilds. Weakness varies strongly by layer. Tuning noise angles to raise rebuild error improves defense. Their PriPert design is tested on privacy, utility, and compute cost.

Catalog record

CREATORMingyuan Fan, Yu Liu, Fuyi Wang, Cen Chen
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE79387
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.23158

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

What Does the Server See? Understanding Privacy Leakage from Large Language Models in Split Inference DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Security and privacy CCS: Computing methodologies Machine learning Mingyuan Fan, Yu Liu, Fuyi Wang, Cen Chen Abstract . The deployment of large language models (LLMs) on resource-constrained devices remains challenging, spurring interest in split inference, where models are partitioned between client and server to reduce computational burden and enhance privacy by transmitting only intermediate activations. However, the privacy-preserving capabilities of split inference, particularly in the context of LLMs, have not been exhaustively investigated. To fill this gap, we introduce ActInv , which solves an intermediate activation matching problem to reconstruct the client’s input. Extensive evaluations demonstrate that ActInv achieves high-fidelity reconstructions, even in the presence of common perturbation-based defenses such as Gaussian noise injection and activation sparsification. To systematically understand this vulnerability, we develop Perturbation Amplification Factor (PAF), a metric for quantifying a layer’s inherent resistance to reconstruction. Our analysis reveals that privacy vulnerability is not uniform across layers, with some layers being highly susceptible to leakage while others offer natural resistance. Furthermore, we demonstrate that defense effectiveness can be significantly improved by calibrating perturbation directions to maximize reconstruction error during backpropagation. Building on these insights, we design PriPert and conduct comprehensive evaluations, covering privacy, utility, and computational overhead, to demonstrate its effectiveness. 1. Introduction The rapid proliferation of large language models (LLMs) has revolutionized numerous applications ( 33 ; 22 ; 32 ) , yet their deployment on resource-constrained devices, such as smartphones and embedded systems, remains a significant challenge ( 34 ; 27 ; 35 ) . Modern LLMs contain billions of parameters and demand substantial computational and memory resources, rendering full on-device inference largely impractical ( 35 ) . While recent advancements in model compression ( 15 ; 18 ; 13 ) have enabled the construction of smaller LLMs, these approaches often incur undesirable accuracy–efficiency trade-offs. At the same time, the proprietary nature of modern LLMs, coupled with the competitive advantage they confer, disincentivizes vendors from releasing fully deployable models ( 22 ) , further limiting client-side deployment. A natural alternative is to offload the entire inference process to the cloud ( 28 ; 26 ) , where the client sends raw inputs to a server hosting the LLM and receives the outputs in return. However, this indeed introduces severe privacy risks, particularly in privacy-sensitive domains like healthcare ( 27 ; 8 ; 24 ) . For instance, if a user’s medical queries or personal diary entries are sent directly to a cloud server for LLM processing, it would expose highly confidential information, raising serious privacy concerns. Split inference has emerged as a promising middle ground ( 9 ; 27 ; 6 ; 11 ; 40 ) . This paradigm partitions an LLM into two non-overlapping components: a small initial segment residing on the client device, and a large, computationally intensive segment hosted on a remote server. The client generates intermediate activations that are subsequently transmitted to the server. The server then completes the remainder of the inference pipeline, with the final inference result sent back to the client. Split inference offers three key advantages: (1) It significantly alleviates the computational and storage burden on the client. (2) It enhances client privacy by transmitting only intermediate activations, rather than raw inputs, to the server. (3) It allows model owners to protect their intellectual property, as the bulk of the model parameters and architecture remain on the server. Despite these benefits, to the best of our knowledge, there remains a blank in the comprehensive investigation of the privacy landscape of LLMs within the split inference. The assumption that intermediate activations inherently preserve privacy requires rigorous scrutiny, as these activations may still encode information about the original sensitive inputs. In this paper, we study the privacy implications of LLMs under split inference by answering three key research questions (RQs). RQ 1: How can the privacy leakage of LLMs in split inference be efficiently quantified and evaluated? We propose ActInv , an attack method that reconstructs the client’s raw input from intermediate activations transmitted to the server. ActInv casts reconstruction as an optimization problem over the input embedding space: it searches for embeddings whose forward pass through the client-side submodel reproduces the intercepted activations at the cut layer, and then projects the optimized embeddings back to discrete tokens. Unlike common attack methods that sequentially recover tokens via complex enumeration or gradient search, ActInv optimizes the full sequence simultaneously, achieving comparable reconstruction accuracy with considerably less computation. We evaluate ActInv across diverse datasets and LLMs in which ActInv can reconstruct high-fidelity reconstructions, even when the intermediate activations are significantly perturbed. RQ 2: What factors govern the severity of privacy leakage? To disentangle the root causes of privacy leakage, we introduce Perturbation Amplification Factor (PAF) to measures the invertibility hardness of a layer. A high PAF indicates that small activation perturbations translate into large uncertainty in the inferred input, making accurate reconstruction difficult. Using PAF, we uncover strong non-uniformity in privacy risk across layers: some layers naturally exhibit high PAF and thus resist reconstruction, whereas others present low PAF and are highly susceptible to leakage. In particular, we find that many commonly used activation layers introduce low-PAF regions, making them easier to invert despite their benefits for model expressivity. Moreover, we show that carefully aligned noise in activation space can induce much larger reconstruction errors than isotropic noise of the same magnitude, implying that naive perturbation defenses are suboptimal. RQ 3: How can we effectively mitigate privacy leakage with a small utility loss? Building on these insights, we propose PriPert , which injects adversarially calibrated perturbations into intermediate activations to obscure sensitive information while preserving model utility. PriPert formulates an optimization problem over perturbations with two competing objectives: maximizing reconstruction error for any plausible inversion (privacy) and minimizing the degradation of the server-side output (utility). Through extensive experiments across multiple LLMs and datasets, we show that PriPert achieves considerable improvements over baseline defenses in reducing ActInv ’s reconstruction quality, with small degradation in downstream task performance. 2. Related Work Since the release of the GPT series by OpenAI ( 22 ) , LLMs have dominated the NLP landscape. The architectures of modern LLMs have largely converged: they are predominantly based on autoregressive prediction paradigms and exhibit highly similar layer structures (Figure 3 ) ( 4 ) . The primary distinctions among them therefore arise from differences in training data and optimization strategies. Research on privacy risks in LLMs can be divided into two stages: training and inference. At the inference stage, most studies focus on risks of training data exposure, i.e., constructing adversarial prompts to elicit memorized training samples from LLMs. Early research ( 3 ; 2 ; 30 ) demonstrated that LLMs tend to memorize portions of their pre-training corpus, raising the possibility of extracting training data. Follow-up studies refined these findings by targeting specific categories of private data ( 12 ; 39 ) , analyzing how training and decoding choices affect leakage ( 37 ; 16 ) , and evaluating the

Read the complete text