Privacy-Preserving Inference for Quantized BERT Models
Tianpei Lu, Bingsheng Zhang, Lekun Peng, Bowen Zheng, Lichun Li, Kui Ren · 2025
Read Privacy-Preserving Inference for Quantized BERT Models by Tianpei Lu, Bingsheng Zhang, Lekun Peng, Bowen Zheng, Lichun Li, Kui Ren (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-01636.txt.
Introduction
Tianpei Lu and co-authors make private use of the BERT language model far faster by mixing secret sharing with slim number formats, in a paper from two thousand twenty-five. Private inference lets several parties run a model without seeing each other's data, but decimal math makes it slow. Slim formats cut cost, yet past work leaked the slimming settings or handled curves poorly.
Layer-by-layer slimming plus one-bit weight layers run safely in the new design. Table lookups handle the softmax curve and format switches with no trim loss. Speed beat three prior systems by large multiples on BERT tests. Readers in secure AI get a practical path to private text services.
Catalog record
| CREATOR | Tianpei Lu, Bingsheng Zhang, Lekun Peng, Bowen Zheng, Lichun Li, Kui Ren |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-01636
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Privacy-Preserving Inference for Quantized BERT Models Tianpei Lu, Bingsheng Zhang, Lekun Peng, Bowen Zheng, Lichun Li, Kui Ren Abstract With the increasing deployment of generative machine learning models in privacy-sensitive domains such as healthcare and personalized services, ensuring secure inference has become a critical challenge. Secure multi-party computation (MPC) enables privacy-preserving model inference but suffers from high communication and computation overhead. The main bottleneck lies in the expensive secure evaluation of floating-point operations. Quantization offers a promising solution by converting floating-point operations into lower-precision integer computations, significantly reducing overhead. However, existing MPC-based quantized inference methods either rely on public quantization parameters—posing privacy risks—or suffer from inefficiencies, particularly in handling nonlinear functions such as activations and softmax. In this work, we propose a fine-grained, layer-wise quantization scheme and support 1-bit weight fully connected layers in a secure setting. We design a multi-input lookup table protocol to evaluate softmax efficiently and securely. Furthermore, we use dual secret sharing schemes and perform precision conversions via lookup tables, eliminating truncation overhead entirely. Experimental evaluation on BERT-base models demonstrates that our approach achieves up to 8 × 8\times speedup compared to Lu et al . (NDSS 25), 9 × 9\times speedup compared to Gupta et al . (PETS 24) and 22 × 22\times speedup compared to Knott et al . (NeurIPS 21). Introduction With the widespread deployment of machine learning models in privacy-sensitive domains such as healthcare and personalized services, ensuring data confidentiality has emerged as a paramount concern. Privacy-preserving machine learning (PPML) techniques enable model inference while maintaining confidentiality, with secure multi-party computation (MPC) being the most popular approach. State-of-the-art (SOTA) PPML platforms, including Orca ( Jawalkar et al. 2023 ) , Falcon ( Wagh et al. 2021 ) , ABY 3 ( Mohassel and Rindal 2018 ) , and SecureML ( Mohassel and Zhang 2017 ) , predominantly leverage MPC frameworks. However, PPML techniques incur substantial communication overhead, especially for large-scale language models. For instance, BERT-base with 12 transformer layers and over 110 million parameters requires billions of secure operations for a single inference. MPC-based inference can generate several gigabytes of communication, creating barriers for real-world deployment. One of the key challenges in privacy-preserving machine learning is that most machine learning models are designed to operate using floating-point arithmetic, which incurs prohibitive overhead when implemented under MPC. The most efficient approach is to approximate these computations using fixed-point arithmetic. In fixed-point representation, a real number x is encoded as an integer \lfloor x\cdot 2^{\ell}\rfloor before being processed under MPC, where \ell denotes the number of fractional bits used to represent precision. However, after performing a multiplication x\cdot y , the result becomes x\cdot y\cdot 2^{2\ell} , and thus a truncation protocol is required to reduce it back to x\cdot y\cdot 2^{\ell} . This truncation step is costly. In current implementations, probabilistic truncation ( Zhou et al. 2025 ; Mohassel and Zhang 2017 ; Mohassel and Rindal 2018 ; Rathee et al. 2020 ; Huang et al. 2022 ) is widely adopted, which necessitates reserving a large number of high-order bits as a tolerance margin to prevent truncation errors 1 1 1 The truncation error is caused by the wrap bit. For example, in secret sharing, x_{1}+x_{2}\bmod 2^{\ell}=x . If a wrap occurs, then x_{1}+x_{2}=x+2^{\ell} . When truncating by k bits, we compute {x_{1}}/{2^{k}}+{x_{2}}/{2^{k}}\bmod 2^{\ell}=x/{2^{k}}+2^{\ell-k} , which introduces an error of 2^{\ell-k} . . Specifically, for a truncation error bound of 2^{-k} , k extra bits are needed; the larger k is, the lower the probability of error. For multiplication of inputs with \ell_{x} -bit precision, at least 2\ell_{x} bits are required to avoid overflow. When accounting for error tolerance in truncation, the overall bit-width expands to 2\ell_{x}+k . For example, when \ell_{x}=16 and k=32 , the computation must be performed over a 64-bit ring. Moreover, in scenarios such as matrix summation, where intermediate results may exceed 2\ell_{x} , an even larger bit-width may be necessary to ensure correctness and prevent overflow. Quantization has emerged as a promising technique to enhance the performance of privacy-preserving machine learning (PPML) ( Zhu et al. 2023 ; Luo et al. 2023 ; Zhu et al. 2021 ; Lang et al. 2023 ) , particularly in distributed settings such as federated learning. However, its application in secure multiparty computation (MPC)-based PPML remains at an early stage. Several works, including those by Riazi et al. ( Riazi et al. 2019 ) , Agrawal et al. ( Agrawal et al. 2019 ) , and Keller et al. ( Keller and Sun 2022 ) , adopt traditional MPC protocols to evaluate quantized models. To optimize performance, these studies often rely on public, fixed quantization parameters—typically powers of two—which, while computationally efficient, can degrade model accuracy and risk leaking sensitive model information. Lu et al. ( Lu et al. 2025 ) were the first to solve the truncation problem in quantized models through table lookup evaluation. While their approach ensures highly efficient online phase computation, it uses lookup tables to handle multiplication. This design results in a total of 256 bits of communication per multiplication gate (from 8-bit input to table lookup) in the offline phase, which makes the scheme challenging to deploy in practical scenarios. In this work, we address the limitations of Lu et al.’s method. Instead of using lookup tables for multiplication, which we find inefficient for matrix multiplication compared to the 2-out-of-3 protocol, we use lookup tables solely to handle truncation. Specifically, we truncate by discarding the lower bits (e.g., keeping the upper 4 bits from 16 bits) and then use a lookup table to expand the 4-bit value back to 16 bits. This avoids high-bit errors caused by truncation 2 2 2 When truncating while also reducing the modulus to 2^{\ell-k} , no truncation error occurs. If a wrap happens so that x_{1}+x_{2}=x+2^{\ell} , then {x_{1}}/{2^{k}}+{x_{2}}/{2^{k}}\bmod 2^{\ell-k}=x , eliminating the error term. ( Zhou et al. 2025 ) . In addition, by designing a multi-input lookup table protocol, we enable the efficient evaluation of functions such as Softmax. Overall, the contributions of this paper can be summarized as follows: • We analyze the impact of weight and activation precision on model performance, and train the model with 1-bit weights and 4-bit activations. Under this quantization setting, privacy-preserving inference can be efficiently supported through secure multiplication combined with lookup table protocols. • We design a multi-input lookup table protocol and, based on it, develop an efficient and systematic softmax evaluation protocol tailored for quantized models. • We employ two types of secret sharing schemes to achieve layer-specific optimal implementations. To bridge different precisions across these schemes, we use lookup table evaluation to perform share conversion between different precisions, eliminating the overhead associated with truncation. As a result, our approach achieves up to 8\times speedup compared to Lu et al . (NDSS 25), 9\times speedup compared to Gupta et al . (PETS 24) and 22\times speedup compared to Knott et al . (NeurIPS 21). Preliminaries Notations. Let \mathcal{P}:=\{P_{0},P_{1},P_{2}\} be the three MPC parties. We use := to denote the definition. We use \mathbb{Z}_{2^{\ell}} to denote the set \{0,\ldots,2^{\ell}-1\} . For a set S , S^{\ell} indicates the \ell dimension vector space over the set S , namely, S^{\ell}:=\{(s_{0},s_{2},\ldots,s_{\ell-1})\mid s_{i}\in