Towards Deep Encrypted Training: Low-Latency, Memory-Efficient, and High-Throughput Inference for Privacy-Preserving Neural Networks

Nges Brian Njungle, Eric Jahns, Michel A. Kinsy · 2026

Read Towards Deep Encrypted Training: Low-Latency, Memory-Efficient, and High-Throughput Inference for Privacy-Preserving Neural Networks by Nges Brian Njungle, Eric Jahns, Michel A. Kinsy (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-16834.txt.

Introduction

Njungle, Jahns and Kinsy develop tuned recipes for batched HE-friendly neural networks. HE computes directly on encrypted data, but batch processing has lagged behind single-image inference. A pipeline architecture maximizes resource efficiency across batch sizes. On ResNet-20 with a batch of 512 encrypted images, amortized inference takes eight point eight six seconds per image with ninety-eight point nine six gigabytes peak memory. That is one point seven eight times faster and three point seven four times more memory-efficient than the leading design.

Catalog record

CREATORNges Brian Njungle, Eric Jahns, Michel A. Kinsy
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE66755
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.16834

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Towards Deep Encrypted Training: Low-Latency, Memory-Efficient, and High-Throughput Inference for Privacy-Preserving Neural Networks Nges Brian Njungle, Eric Jahns, Michel A. Kinsy Abstract Privacy-preserving machine learning (PPML) has become increasingly important in applications where sensitive data must remain confidential. Homomorphic Encryption (HE) enables computation directly on encrypted data, allowing neural network inference without revealing raw inputs. While prior works have largely focused on inference over a single encrypted image, batch processing of encrypted inputs lags behind, despite being critical for high-throughput inference scenarios and training-oriented workloads. In this work, we address this gap by developing optimized algorithms for batched HE-friendly neural networks. We also introduced a pipeline architecture designed to maximize resource efficiency for different batch size execution. We implemented these algorithms and evaluated our work using HE-friendly ResNet-20 and ResNet-34 models on encrypted CIFAR-10 and CIFAR-100 datasets, respectively. For ResNet-20, our approach achieves an amortized inference time of 8.86 8.86 seconds per image when processing a batch of 512 512 encrypted images, with a peak memory usage of 98.96 98.96 GB. These results represent a 1.78 × 1.78\times runtime improvement and a 3.74 × 3.74\times reduction in memory usage compared to the state-of-the-art design. For the deeper ResNet-34 model, we achieve an amortized inference time of 28.14 28.14 on a batch of 256 256 encrypted images using 246.78 246.78 GB of RAM. I Introduction Over the past decade, Machine Learning (ML) has evolved into a cornerstone of modern technology, driving innovation across nearly every sector of society [ 4 ] . From personalized healthcare and finance to autonomous driving systems and cybersecurity, ML models now underpin a wide range of data-driven applications. Advances in model architectures, computing hardware, and large-scale data availability have further propelled ML toward solving increasingly complex real-world problems, making it a critical enabler of intelligent automation and decision-making across many applications [ 39 ] . One fundamental paradigm that has accelerated the accessibility and scalability of ML is Machine Learning as a Service (MLaaS) [ 37 ] . MLaaS enables model developers and organizations to deploy highly efficient and complex models through cloud-based infrastructure, providing inference capabilities as a utility. Here, users can remotely access pre-trained models, submit queries, and receive predictions without maintaining local models. Another important direction in modern ML is collaborative training [ 40 ] . Here, multiple entities jointly train a model to leverage distributed data resources and domain expertise. This approach allows organizations to train a more intelligent and generalized model without explicitly sharing their datasets. A more recent, but increasingly relevant domain is cross-organizational model evaluation [ 42 ] . In this setting, one party develops a sophisticated model, while another party holds the data needed to test or fine-tune that model’s performance. Cross-organization evaluation enables stakeholders to assess the utility of external models without requiring full data or model transfer, thus promoting collaborative innovation and interoperability between independent entities. Although these domains differ in structure and objective, they all share a common challenge when handling sensitive data. Whether in MLaaS deployment, collaborative training, or cross-organization evaluation, the exchange of sensitive model weights and sensitive user data introduces critical privacy and security risks. These risks become especially acute when ML systems process sensitive or personally identifiable data such as genomic sequences, medical records, or financial transactions [ 34 ] . Yet, these are also the very domains that stand to benefit the most from the efficiency, scalability, and intelligence of modern ML systems. This tension between utility and confidentiality has made privacy-preserving machine learning a central focus for many recent ML works. Privacy-preserving machine learning (PPML) seeks to enable the benefits of ML while ensuring that sensitive information remains confidential throughout the computation process [ 29 ] . In the application domains discussed above, PPML techniques allow multiple parties to train, inference, or deploy models without exposing raw data or revealing proprietary model parameters. Among the various approaches used for PPML today, Homomorphic Encryption (HE) has emerged as a particularly compelling solution. HE allows computations to be performed directly on encrypted data, producing encrypted outputs that can be decrypted to obtain the same results as computations on plaintext inputs [ 1 ] . This property enables ML systems to perform inference and training without ever exposing sensitive information. Despite their promise, today’s practical HE schemes often impose substantial computational and memory overheads in applications, making encrypted neural network models orders of magnitude slower than their plaintext counterparts. Addressing these performance bottlenecks is essential to enable PPML with HE at scale, particularly in deep learning models [ 21 , 33 ] . I-A Limitations of Encrypted Neural Networks Within the context of HE-based neural networks, a primary approach is single-input processing. Here, each input is encrypted and processed individually through a HE-friendly neural network model [ 38 ] . This method is particularly valuable in scenarios that demand real-time processing of sensitive queries, such as personalized medical diagnosis, confidential financial decision support, or secure recommendations for individual users. In fact, the most advanced encrypted neural networks of today are focused on single-input processing. Recent frameworks such as FHEON [ 32 ] and Orion [ 17 ] have demonstrated stable implementations of encrypted deep neural networks and have significantly pushed the performance limits of HE inference in this setting. Despite its importance and numerous applications, this single-input processing approach is highly limited in high-throughput environments, as each input requires a complete sequence of computationally expensive homomorphic operations. The second approach to encrypted neural networks is encrypted batch input processing. In this method, multiple encrypted inputs are processed simultaneously through a HE-friendly neural network model [ 13 ] . Encrypted batch image processing can greatly improve throughput and reduce amortized latency compared to encrypted single-input processing. However, this approach is highly constrained by the complexity of packing multiple images, complexity of model design, the number of HE operations allowed within a ciphertext, and the substantial memory and computational requirements of processing these large amounts of encrypted data. The importance of encrypted batch inference becomes particularly evident in real-world applications that require processing large volumes of sensitive data at once. For example, hospitals may need to run predictions over hundreds of encrypted scans simultaneously to support clinical decision-making. Financial institutions may wish to evaluate large batches of transaction data for fraud detection or risk assessment without exposing customer information. Beyond inference, achieving efficient encrypted batch processing is also a prerequisite for practical HE-based deep neural network training. Training deep neural networks relies on batch operations for efficient gradient computation and weight updates, since models generally require large amounts of data to learn and generalize effectively [ 24 ] . Many previous works have proposed different approaches for encrypted batch input processing. The state-of-the-art work of Cheon et al. introduced the most optimized approach for this line of work [ 13 ] . Their implementations leveraged multi-threading on a server-grade computational setup, resulting in an encrypted-image amortized runtime of about 15.76 seconds in a 512-image pipeline using a HE-friendly ResNet-20 model and the CIFAR-10 dataset. While their work showed about 5\times performance improvement over the previous version, it still requires a memory footprint of 370.28 GB of RAM which is still very high. Improving encrypted

Read the complete text