CipherPrune: Efficient and Scalable Private Transformer Inference

Yancheng Zhang, Jiaqi Xue, Mengxin Zheng, Mimi Xie, Mingzhe Zhang, Lei Jiang, Qian Lou · 2025

Read CipherPrune: Efficient and Scalable Private Transformer Inference by Yancheng Zhang, Jiaqi Xue, Mengxin Zheng, Mimi Xie, Mingzhe Zhang, Lei Jiang, Qian Lou (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-16782.txt.

Introduction

Yancheng Zhang and co-authors speed in a 2025 study private transformer inference, where a model answers on encrypted text it cannot read. Cost grows with the square of input length, and many tokens are filler. Their CipherPrune framework trims the input while it stays encrypted.

Unimportant tokens are pruned layer by layer, and the survivors' heavy math is approximated with cheaper polynomials where the tokens matter least. A search step tunes the pruning to hold accuracy. The payoff was about a sixfold cut in runtime on hundred-twenty-eight-token inputs. Private chatbots and document tools need exactly this kind of speedup.

Catalog record

CREATORYancheng Zhang, Jiaqi Xue, Mengxin Zheng, Mimi Xie, Mingzhe Zhang, Lei Jiang, Qian Lou
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-16782

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CipherPrune: Efficient and Scalable Private Transformer Inference Yancheng Zhang, Jiaqi Xue, Mengxin Zheng, Mimi Xie, Mingzhe Zhang, Lei Jiang, Qian Lou Abstract Private Transformer inference using cryptographic protocols offers promising solutions for privacy-preserving machine learning; however, it still faces significant runtime overhead (efficiency issues) and challenges in handling long-token inputs (scalability issues). We observe that the Transformer’s operational complexity scales quadratically with the number of input tokens, making it essential to reduce the input token length. Notably, each token varies in importance, and many inputs contain redundant tokens. Additionally, prior private inference methods that rely on high-degree polynomial approximations for non-linear activations are computationally expensive. Therefore, reducing the polynomial degree for less important tokens can significantly accelerate private inference. Building on these observations, we propose CipherPrune , an efficient and scalable private inference framework that includes a secure encrypted token pruning protocol, a polynomial reduction protocol, and corresponding Transformer network optimizations. At the protocol level, encrypted token pruning adaptively removes unimportant tokens from encrypted inputs in a progressive, layer-wise manner. Additionally, encrypted polynomial reduction assigns lower-degree polynomials to less important tokens after pruning, enhancing efficiency without decryption. At the network level, we introduce protocol-aware network optimization via a gradient-based search to maximize pruning thresholds and polynomial reduction conditions while maintaining the desired accuracy. Our experiments demonstrate that CipherPrune reduces the execution overhead of private Transformer inference by approximately 6.1 × 6.1\times for 128-token inputs and 10.6 × 10.6\times for 512-token inputs, compared to previous methods, with only a marginal drop in accuracy. The code is publicly available at https://github.com/UCF-Lou-Lab-PET/cipher-prune-inference . $*$ $*$ footnotetext: Corresponding Author. Email: [email protected]. 1 Introduction Transformers ( Vaswani et al., 2017 ) have become the predominant approach for tackling a wide range of machine learning tasks, spanning Natural Language Processing (NLP) ( Xue et al., 2024 ) and Computer Vision (CV) ( Zheng et al., 2022 ) domains. Notably, Transformer-as-a-Service (TaaS) ( Radford et al., 2018 ) has emerged as an effective means for average users to harness the capabilities of sophisticated and accurate Transformers deployed on cloud servers. Privacy has become a major concern, driving a growing demand for privacy-preserving TaaS solutions ( Zheng et al., 2023 ; Hao et al., 2022 ; Zeng et al., 2023 ; Santriaji et al., 2025 ; Lou et al., 2021 ) . Homomorphic Encryption (HE) ( Gentry, 2009 ) is a promising secure computing technology that protects data privacy by enabling computations on encrypted data without decryption. However, applying HE continuously for deep computation tasks often results in prohibitively high latency ( Lou et al., 2019 ; Lou & Jiang, 2019 ; Zheng et al., 2024 ; Zhang et al., 2023a ; Jiang et al., 2022 ) . To address this, hybrid HE/Multi-party Computation (MPC)-based techniques ( Chen et al., 2022 ; Zheng et al., 2023 ; Hao et al., 2022 ; Zeng et al., 2023 ; Zhang et al., 2023b ; Lu et al., 2025 ; Pang et al., 2024 ; Xu et al., 2024b ) have been widely adopted for private Transformer inference, as illustrated in Figure 1 (a). This hybrid approach achieves state-of-the-art performance by using HE for linear operations and MPC for non-linear operations. Unfortunately, prior private Transformer inferences ( Lu et al., 2025 ; Pang et al., 2024 ; Zheng et al., 2023 ) still suffer from significant latency and poor scalability over long-token inputs. As Figure 1 (b) shows, the prior private inference process for a GPT2 Transformer ( Pang et al., 2024 ) with 128 input tokens extends to \sim 10 minutes. It necessitates the exchange of over 60 gigabytes of data between the server and the client. Furthermore, as the token length increases, the runtime overhead grows super-linearly, indicating poor scalability. This is primarily because the operational complexity of Transformers ( Vaswani et al., 2017 ; Kim et al., 2022 ) scales quadratically with the number of input tokens. Reducing the number of input tokens without compromising accuracy is essential. We observe that most inputs contain redundant words/tokens, with varying levels of redundancy across different inputs. As illustrated in Figure 1 (c), in a sentiment analysis task, an input that retains only the tokens movie and great while removing almost all others still maintains inference confidence and accuracy. We refer to such tokens that can be removed without significantly impacting accuracy as redundancy . Meanwhile, the different inputs have various levels of redundancy ( Wang et al., 2021 ; Kim et al., 2022 ; Yudha et al., 2024 ) . Figure 1 (d) illustrates that some inputs exhibit greater redundancy, while others have less, with this variation being particularly evident across different tasks. Classification tasks typically have more redundancy compared to sequence-to-sequence tasks ( Fu et al., 2024 ) . To effectively prune more tokens from longer inputs and potentially reduce the Transformer’s quadratic complexity to linear, pruning should be done progressively—that is, tokens should be pruned layer by layer over multiple stages, as illustrated in Figure 1 (e), rather than performing a one-time pruning at the first layer ( Wang et al., 2021 ; Kim et al., 2022 ; Xu et al., 2024a ) . Another key observation is that previous private Transformer inference methods, whether relying on precise non-linear activations ( Hao et al., 2022 ; Yudha et al., 2024 ) or using large-degree polynomial approximations ( Lu et al., 2025 ; Pang et al., 2024 ) for these activations, continue to suffer from significant execution overhead for non-linear operations. Therefore, replacing non-linear activations or high-degree polynomials with lower-degree polynomials can be beneficial. As shown in Figure 2 , a degree- d polynomial activation for tokens (Figure 2 (a)) can be reduced to a degree- d_{i} polynomial (Figure 2 (b)), where d_{i}\leq d . Adopting existing plaintext-level pruning techniques ( PlainPrune ) ( Wang et al., 2021 ; Kim et al., 2022 ) to accelerate private Transformer inferences presents a formidable challenge. The primary reason is that the tokens are encrypted, and we need to calculate token importance scores layer by layer for specific encrypted inputs. This requires redesigning a new encrypted token pruning protocol. Meanwhile, the polynomial reduction in the encrypted domain poses a similar challenge, and we need to design an encrypted polynomial reduction protocol for efficient private activation. Also, at the network level, we need to learn the pruning and polynomial reduction thresholds while maximizing efficiency , ensuring privacy , and maintaining the desired level of accuracy . Figure 1: (a) An illustration of HE/MPC-based private inference. (b) The high-latency and scalable challenge of private Transformer models over lengthy inputs. (c) An example of redundant input in sentiment analysis tasks. (d) Demonstration of varying levels of redundancy across different inputs. (e) An example showcasing progressive redundancy pruning. To address these challenges, we introduce CipherPrune, a scalable and efficient framework for private inference that incorporates a secure encrypted token pruning protocol, a polynomial reduction protocol, and tailored Transformer network optimizations. At the protocol level, CipherPrune adaptively prunes unimportant tokens from encrypted inputs in a progressive, layer-by-layer manner. It also applies encrypted polynomial reduction by assigning lower-degree polynomials to less important tokens post-pruning, thereby improving efficiency without Figure 2: Polynomial Reduction. requiring decryption. At the network level,

Read the complete text