Practical and Private Hybrid ML Inference with Fully Homomorphic Encryption

Sayan Biswas, Philippe Chartier, Akash Dhasade, Tom Jurien, David Kerriou, [0.5ex] Anne-Marie Kerrmarec, Mohammed Lemou, Franklin Tranie, [0.5ex] Martijn de Vos, Milos Vujasinovic, [2mm] EPFL · 2025

Read Practical and Private Hybrid ML Inference with Fully Homomorphic Encryption by Sayan Biswas, Philippe Chartier, Akash Dhasade, Tom Jurien, David Kerriou, [0.5ex] Anne-Marie Kerrmarec, Mohammed Lemou, Franklin Tranie, [0.5ex] Martijn de Vos, Milos Vujasinovic, [2mm] EPFL (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-01253.txt.

Introduction

Cloud models must answer questions on private data without seeing it, and fully homomorphic encryption allows computation on locked data directly. Sayan Biswas, with nine co-authors, notes that this encryption is slowed by costly bootstrapping steps and rough stand-ins for nonlinear functions. Their framework, Safhire, splits the work between server and client.

Linear layers run under encryption on the server, while nonlinear steps are handled by the client in plain form. That split removes bootstrapping and allows exact activations, cutting cost sharply. Randomized shuffling hides intermediate values so clients cannot rebuild the model. Packing tricks cut delay further, and tests showed latency several times lower than prior approaches, while keeping both user data and the server's model private.

Catalog record

CREATORSayan Biswas, Philippe Chartier, Akash Dhasade, Tom Jurien, David Kerriou, [0.5ex] Anne-Marie Kerrmarec, Mohammed Lemou, Franklin Tranie, [0.5ex] Martijn de Vos, Milos Vujasinovic, [2mm] EPFL
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE69 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-01253

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Practical and Private Hybrid ML Inference with Fully Homomorphic Encryption Sayan Biswas, Philippe Chartier, Akash Dhasade, Tom Jurien, David Kerriou, [0.5ex] Anne-Marie Kerrmarec, Mohammed Lemou, Franklin Tranie, [0.5ex] Martijn de Vos, Milos Vujasinovic, [2mm] EPFL Abstract In contemporary cloud-based services, protecting users’ sensitive data and ensuring the confidentiality of the server’s model are critical. Fully homomorphic encryption (FHE) enables inference directly on encrypted inputs, but its practicality is hindered by expensive bootstrapping and inefficient approximations of non-linear activations. We introduce Safhire , a hybrid inference framework that executes linear layers under encryption on the server while offloading non-linearities to the client in plaintext. This design eliminates bootstrapping, supports exact activations, and significantly reduces computation. To safeguard model confidentiality despite client access to intermediate outputs, Safhire applies randomized shuffling, which obfuscates intermediate values and makes it practically impossible to reconstruct the model. To further reduce latency, Safhire incorporates advanced optimizations such as fast ciphertext packing and partial extraction. Evaluations on multiple standard models and datasets show that Safhire achieves 1.5 × 1.5\times – 10.5 × 10.5\times lower inference latency than Orion , a state-of-the-art baseline, with manageable communication overhead and comparable accuracy, thereby establishing the practicality of hybrid FHE inference. 1 Introduction ML ( ML ) has profoundly impacted industrial domains such as healthcare [ 33 ] , finance [ 23 ] , and the Internet of Things [ 37 ] . In many practical deployments, ML models are hosted on cloud servers and accessed by clients who submit private and privileged data for inference [ 52 ] . However, this paradigm, also known as MLaaS ( MLaaS ), raises serious privacy concerns as users must share their inference inputs, such as medical or financial data, with cloud servers who can directly use or share the data with other parties [ 46 ] . As users increasingly rely on online inference services, ensuring the privacy of user data during inference is paramount [ 36 ] . FHE ( FHE ) offers a compelling solution to this privacy concern by enabling computations directly on encrypted user data, see Figure 2 (top) [ 30 , 45 ] . FHE provides strong privacy guarantees as the server learns nothing beyond ciphertext sizes and protocol metadata. An ML inference request with FHE works as follows: a user first encrypts its inference inputs and sends the encrypted data to the server (step 1 and 2). The server then performs the computations related to the neural network forward pass in the encrypted domain (step 3) and sends back the encrypted output (step 4), after which the user decrypts the output and obtains the inference result (step 5). This way, the server is unable to infer any information from the input data since it remains encrypted during the inference request [ 31 ] . Figure 1: Latency breakdown for bootstrapping and other operations during a single sample inference request using Orion , a state-of-the-art FHE inference framework. We test three models and datasets and show the end-to-end inference duration in minutes on the right. Existing FHE inference schemes face two main obstacles that make end-to-end encrypted inference impractical at scale [ 26 , 48 ] . First, encrypted vectors can only undergo so many operations before becoming too noisy to decrypt correctly. To reset ciphertext noise, FHE schemes rely on an operation called bootstrapping . Bootstrapping enables further computations on these vectors by reducing this noise, but comes with a high computational cost [ 6 ] . Figure 1 shows the duration of bootstrapping and other operations in Orion , a state-of-the-art FHE inference framework [ 26 ] . We evaluate three CNN ( CNN ) models, ResNet-20, ResNet-18 and ResNet-34, and three datasets, CIFAR-10, Tiny and ImageNet. Across these configurations, bootstrapping takes between 62% to 85% of total inference time, highlighting the significant cost of this operation. Second, some non-linear operations commonly included in neural networks, such as ReLU , are not natively supported by FHE and must be replaced by, for example, high-order polynomial approximations or softmax functions [ 20 , 53 , 38 ] . Unfortunately, these approximations require many more computations than their native counterparts and quickly accumulate ciphertext noise, necessitating additional bootstrapping steps and further increasing inference latency. This work introduces Safhire , a novel and practical hybrid FHE scheme for ML inference that overcomes the above limitations. Safhire leverages a simple yet powerful idea: we keep the computation of linear, parameterized layers ( e . g ., convolutions and fully connected layers) on the server under encryption and push non-linear operations ( e . g ., ReLU activations) to the client which evaluates them in plaintext. Operationally, Safhire proceeds in rounds that correspond one-for-one to linear blocks bounded by non-linearities. 1 1 1 While we primarily evaluate on CNNs (in particular ResNet models), our approach applies to any architecture that alternates linear operations with non-linear ones ( e . g ., blocks that are linear up to an activation). We show the end-to-end workflow of Safhire in Figure 2 (bottom). In each round, the client encrypts and sends the current layer input (steps 1–2), the server applies the encrypted linear transform (step 3) and returns the resulting ciphertexts (step 4). Upon receipt, the client decrypts and applies the exact non-linear operation ( e . g ., ReLU ), then re-encrypts the outputs for the next round (step 5). This process repeats until the forward pass completes after which the client obtains the final result (step 6). The intermediate decryption and re-encryption at the client helps avoid bootstrapping since we decrypt before the noise becomes unmanageable. Moreover, because all non-linearities are executed in plaintext on the client, Safhire avoids any server side non-linear approximations, preventing additional bootstrapping and amplified latency. However, revealing intermediate outputs to the client poses the risk of model reconstruction wherein the client may try to reverse engineer the model weights. This undermines model confidentiality as the server model often represents proprietary intellectual property and investment due to high training and deployment costs. To prevent this, Safhire randomly shuffles the outputs of the linear operations on the server before sending them to the client. The shuffling function and the associated unshuffling function is derived by the server from a per-session secret seed and is unknown to the client. We provide formal differential privacy guarantees resulting from the shuffling operation in combination with the default noise introduced by server side operations. Thus, Safhire makes model reconstruction practically as hard as full (or non-hybrid) server-side inference, in a black box setting. At the same time, we show that the correctness of the client-side decryption remains unaffected. Under the hood, Safhire leverages the TFHE ( TFHE )-based RLWE ( RLWE ) cryptographic scheme which is widely used for practical homomorphic computation [ 12 , 8 ] . Beyond the hybrid scheme, we further increase efficiency by implementing a series of optimization efforts which reduce both computation and bandwidth, yielding practical end-to-end latency under standard security parameters. Our implementation supports inference using multi-core CPUs and GPUs. We evaluate the efficiency of our scheme using widely adopted CNN architectures and different datasets. Under realistic network conditions, Safhire reduces end-to-end latency of a single inference request

Read the complete text