Private LoRA Fine-tuning of Open-Source LLMs with Homomorphic Encryption
Jordan Frery, Roman Bredehoft, Jakub Klemsa, Arthur Meyre, Andrei Stoian · 2025
Read Private LoRA Fine-tuning of Open-Source LLMs with Homomorphic Encryption by Jordan Frery, Roman Bredehoft, Jakub Klemsa, Arthur Meyre, Andrei Stoian (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-07329.txt.
Introduction
Jordan Frery and co-authors fine-tune language models on encrypted data so remote servers never see the secrets, in work from twenty twenty-five. Sensitive fields cannot hand raw training data to outside compute. Their protocol adapts the popular LoRA method under homomorphic encryption. Encryption shields both training data and gradients on worker machines. The data owner directs training with modest local hardware.
No costly home graphics card is needed on the owner side. A small open model is fine-tuned as the feasibility proof. Encryption-friendly quantisation keeps the model converging during training. Graphics hardware benchmarks show the encrypted steps are practical. Confidential knowledge bases and private codebases become trainable targets. Legal and health documents could tune assistants without exposure.
Catalog record
| CREATOR | Jordan Frery, Roman Bredehoft, Jakub Klemsa, Arthur Meyre, Andrei Stoian |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-07329
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Private LoRA Fine-tuning of Open-Source LLMs with Homomorphic Encryption Jordan Frery, Roman Bredehoft, Jakub Klemsa, Arthur Meyre, Andrei Stoian Abstract Preserving data confidentiality during the fine-tuning of open-source Large Language Models (LLMs) is crucial for sensitive applications. This work introduces an interactive protocol adapting the Low-Rank Adaptation (LoRA) technique for private fine-tuning. Homomorphic Encryption (HE) protects the confidentiality of training data and gradients handled by remote worker nodes performing the bulk of computations involving the base model weights. The data owner orchestrates training, requiring minimal local computing power and memory, thus alleviating the need for expensive client-side GPUs. We demonstrate feasibility by fine-tuning a Llama-3.2-1B model, presenting convergence results using HE-compatible quantization and performance benchmarks for HE computations on GPU hardware. This approach enables applications such as confidential knowledge base question answering, private codebase fine-tuning for AI code assistants, AI agents for drafting emails based on a company’s email archive, and adapting models to analyze sensitive legal or healthcare documents. 1 Introduction Large Language Models (LLMs) exhibit transformative potential across a vast array of applications, from translation and summarization to classification and generation. While foundation models trained on web-scale datasets possess broad capabilities, unlocking their full potential for specific tasks or specialized domains like healthcare or law often requires fine-tuning. Fortunately, the prevalence of the transformer architecture allows many open-source LLMs to be efficiently adapted using techniques like Low-Rank Adaptation (LoRA) [ 11 ] . However, adapting these powerful tools using sensitive, private data presents a significant challenge for organizations. Healthcare providers aiming to leverage confidential patient records, financial institutions using private transaction data, legal firms working with sensitive case documents, or companies customizing AI assistants on proprietary source code all face a critical dilemma. Exposing such raw data to third-party cloud services or even internal tools without stringent privacy guarantees is frequently untenable due to regulatory constraints (e.g., HIPAA, GDPR), competitive risks, or ethical considerations. Consequently, leveraging LLMs on valuable private datasets poses a major privacy problem. While performing fine-tuning locally on client hardware avoids direct data exposure, this alternative presents its own significant challenges. State-of-the-art LLMs, even with efficient methods like LoRA, often demand substantial computational resources—particularly GPUs with high VRAM (e.g., > 24GB)—which may exceed the capabilities or budget of many users or smaller organizations. Furthermore, establishing and managing the complex software environment for LLM fine-tuning requires considerable technical expertise. Therefore, outsourcing the computationally intensive parts of the fine-tuning process is highly desirable but demands a solution where data confidentiality can be rigorously maintained. To address this critical need for secure and accessible private fine-tuning, we present a protocol where a client (data owner) interactively orchestrates LoRA fine-tuning of an open-source LLM, securely outsourcing the most demanding computations to a server (or a network of worker nodes). Our contribution lies in the design and practical integration of a system enabling this private fine-tuning, centered around several key aspects : • A client-server architecture tailored for LoRA, where the client manages private LoRA weights ( U,D ) and performs non-linear operations, while the server handles linear operations involving the public base model weights ( W ) under Homomorphic Encryption (HE). • An efficient HE protocol for the core encrypted vector-clear matrix multiplication ( W\cdot[x]_{\text{HE}} ), utilizing packed Ring Learning With Errors (RLWE) ciphertexts for input/output, modulus switching for communication efficiency, and GPU acceleration via custom CUDA kernels. • A demonstration of feasibility and convergence on a standard LLM (Llama-3.2-1B) using HE-compatible quantization. In our setting, the base LLM is public, allowing the server to perform necessary computations, while the client exclusively holds the private LoRA weights representing the model’s adaptation . Efficiency is achieved by restricting server-side HE to only the linear operations involving W , leveraging optimized GPU kernels, and using strong ciphertext compression to minimize bandwidth. While the protocol naturally supports inference, our focus here is demonstrating the feasibility of private fine-tuning . 2 Prior Work 2.1 Low-Rank Adaptation LLM architectures are almost exclusively based on the multi-head attention (MHA) mechanism. For an input sequence x (batch size B , context length C , model dimension d ), a transformer layer computes updated representations x^{\prime} using MHA and a feed-forward network (FFN). Let d_{k} be the dimension of keys and queries per attention head. The computation involves weight matrices ( W_{Q},W_{K},W_{V},W_{proj} ) for attention and FFN weights (e.g., W_{gate},W_{up},W_{down} in Llama-style models). Conceptually, for a single token (ignoring batch/context dimensions for simplicity): \begin{split}Q&=W_{Q}x,\quad K=W_{K}x,\quad V=W_{V}x\\ \text{AttnOut}&=\operatorname{Attention}(Q,K,V)=\operatorname{softmax}\left(\frac{QK^{T}}{\sqrt{d_{k}}}\right)V\\ h&=W_{proj}(\text{AttnOut})\\ \text{FFN}_{out}&=W_{down}(\operatorname{SiLU}(W_{gate}h)\odot W_{up}h)\\ x^{\prime}&=h+\text{FFN}_{out}\end{split} (1) Regular Low-Rank Adaptation (LoRA) fine-tuning [ 11 ] modifies a pre-trained weight matrix W (size d_{out}\times d_{in} ) by adding a low-rank update \Delta W=UD , where U is d_{out}\times r and D is r\times d_{in} , with rank r\ll\min(d_{in},d_{out}) . The forward pass for a matrix multiplication Wx is then modified to become y=Wx+UDx(+b\text{ if bias}) . For instance, applying LoRA to the query projection matrix W_{Q} (where d_{in}=d_{out}=d ) in Eq. ( 1 ) changes the computation to Q=W_{Q}x+U_{Q}D_{Q}x . Only the LoRA matrices U and D are trained. LoRA is typically applied to attention weights ( W_{Q},W_{K},W_{V},W_{proj} ) and FFN weights. The number of trainable parameters r(d_{in}+d_{out}) is much smaller than the original d_{in}d_{out} parameters of W . 2.2 Split Edge-Cloud LLM Fine-tuning Splitting LLM LoRA fine-tuning between an edge device and a cloud service has been explored in [ 7 , 23 ] . Both works outsource the computations involving the original model weight matrices W to a cloud, while forward and backward passes on LoRA weights U,D are kept local on the edge client. However, these approaches do not consider data confidentiality risks associated with sending intermediate activations to the cloud, and the latter work does not ensure model adaptation ( \Delta W ) confidentiality either. 2.3 Non-interactive Encrypted Training In non-interactive training, only the encrypted dataset is sent to the server, and the encrypted model is retrieved. This has the advantage of keeping bandwidth requirements to a minimum. Logistic Regression training on encrypted data was described in several works [ 19 , 16 , 9 , 3 , 4 ] . Small multi-layer perceptrons (MLPs) were studied in [ 19 , 17 , 20 ] . Since HE operates over integers, most of these works quantize weights, gradients, activations, and the error function to between 6 and 8 bits. However, fully encrypted training requires a large amount of expensive encrypted multiplications and noise-management procedures like bootstrapping, making it too costly at present for large models like LLMs. Outsourcing linear operations confidentially usually relies on HE ( [ 26 ] , [ 25 ] ) or multi-party computation (MPC) [ 8 ] . [ 26 ] relies on the Paillier cryptosystem [ 21 ] . With Paillier, ciphertexts have sizes of 4096 or 6144 bits and can pack multiple plaintexts. Encryption is performed using modular exponentiation, though it can be optimized by splitting encryption into an online and an offline stage [ 14 ] . When encrypting a large amount of data, pre-computing modular exponentiations in the offline stage may not be practical. Modular exponentiation makes encryption slow, placing a large computational burden on the client side and strongly reducing training throughput. RLWE-based approaches that pack multiple plaintexts in a