Threshold-adaptive pruning with multi-key homomorphic encryption for communication-efficient secure federated learning Authors: Jie Guo, Renjing Liu, Jinsheng Xing, Je Sen Teh, Je Sen Teh, Je Sen Teh Published: 2026 ABSTRACT Under the federated learning framework, frequent parameter interactions between edge devices and servers result in communication inefficiency, while conventional encryption methods fail to resist multi-node collusion attacks. To address these challenges, this paper proposes an optimized federated learning scheme integrating adaptive channel pruning with multi-key homomorphic encryption. First, we construct a dynamic threshold determination mechanism that automatically calibrates channel pruning rates through precision feedback during the pre-pruning phase, achieving the optimal balance between model compression and accuracy, while significantly reducing communication bandwidth consumption compared to traditional algorithms. Second, based on the Brakerski-Gentry-Vaikuntanathan (BGV) multi-key fully homomorphic encryption architecture, we design a distributed public-key encryption protocol that enables aggregation servers to securely fuse multi-source model parameters without decryption, resisting collusion attacks from up to C − 1 nodes (where C denotes the total number of devices). Experiments on MNIST and CIFAR-10 datasets demonstrate that our scheme significantly reduces communication overhead through two complementary mechanisms: adaptive pruning reduces both the computational burden of local training and the volume of parameters transmitted per round, while multi-key BGV encryption ensures privacy-preserving aggregation without decryption. This work provides a novel technical pathway for privacy-preserving federated learning in resource-constrained scenarios. Introduction In federated learning, edge devices need to engage in regular model parameter exchanges with the central server to achieve collaborative training. However, as machine learning models continue to grow in scale, communication and computational overhead issues become increasingly prominent. More complex models require greater bandwidth and computational resources to transmit and process parameters, while edge devices are typically resource-constrained, lacking sufficient bandwidth and processing power. To address these challenges, researchers have introduced model pruning techniques [1,2] in federated learning, effectively reducing the computational burden and communication overhead of edge devices. For example, Jiang et al. [3] proposed reducing communication costs by pruning weights; while Munir et al. [4] successfully shortened overall training time by pruning global models for lower-performing devices. These works primarily focus on applying unstructured pruning techniques within the federated learning framework. Although such methods can maintain high accuracy even at high compression rates, they rely on specialized computation libraries and hardware support, making them unfavorable for deployment on resource-constrained edge devices. Xu et al. [5] effectively reduced communication overhead by employing structured channel pruning to decrease model size, enabling pruned models to be trained directly using existing hardware and computation libraries. However, these studies generally adopt a one-time pruning strategy, performing only a single pruning operation on the model. In this process, selecting an appropriate pruning rate is crucial: if the rate is too low, the model retains significant redundancy; if too high, it may result in considerable accuracy loss. Furthermore, recent research [6] has revealed significant privacy and security challenges in federated learning. These challenges include: inference attacks [7], where malicious participants analyze model parameters or gradient information to deduce training data from specific clients; model reverse engineering [8], where attackers use reverse engineering techniques to reconstruct local models or approximate original data, leading to privacy breaches; and model extraction attacks [9], where malicious actors may extract partial data samples from the global model to understand data characteristics and distribution, ultimately resulting in privacy leakage. To counter these threats, researchers have proposed various privacy protection techniques, primarily including differential privacy (DP) [10], homomorphic encryption (HE) [11], and secure multi-party computation (SMC) [12]. However, these methods still face numerous challenges: federated learning based on differential privacy requires striking a delicate balance between model accuracy and privacy protection, constituting a significant challenge; while federated learning based on secure multi-party computation typically demands multiple rounds of interaction between participants to achieve secure aggregation, resulting in substantial communication overhead. In comparison, federated learning approaches based on homomorphic encryption, despite certain limitations in computational efficiency, effectively avoid model accuracy degradation and complex interactions between clients, while achieving relatively ideal privacy protection outcomes. However, current homomorphic encryption-based federated learning schemes predominantly employ a single key for encrypting and homomorphically computing model parameters, meaning all ciphertexts involved in computation correspond to the same key. This design cannot effectively resist data leakage attacks from curious internal devices, nor collusion attacks between internal devices and servers. At present, although several federated learning-related studies have separately focused on addressing privacy leakage [13] or communication overhead [14] issues, research that comprehensively tackles both challenges simultaneously remains relatively scarce. Therefore, this research proposes an Adaptive Pruning Multi-Key Federated Learning (APMKFL) scheme based on the federated learning framework. In the pruning process of this scheme, edge devices apply various pruning rates for model pre-pruning and evaluate model accuracy on validation datasets. Subsequently, pre-pruned models are ranked according to their accuracy, and the pruning rate corresponding to the highest accuracy is automatically selected for final pruning operations. While network slimming [15] and multi-key BGV homomorphic encryption [16] are individually established techniques, their integration within a federated learning framework—with a feedback-driven adaptive threshold mechanism—constitutes the core novelty of this work. Three aspects distinguish APMKFL from a straightforward combination of prior methods: (1) Prior federated pruning works [1,3,5,17,18] apply static or one-time pruning. Network slimming [15] was designed for centralized training. We reformulate its BN-based channel scoring as a per-round adaptive decision mechanism within the FL communication protocol, where the pruning structure evolves with each gradient update—a non-trivial integration challenge not addressed by existing work. (2) Prior HE-based FL schemes [11, 12] use a single shared key, enabling honest-but-curious servers to decrypt individual updates. APMKFL employs multi-key BGV [16] with threshold decryption, so the server never holds a decryption key—achieving provable C-1 collusion resistance. (3) Unlike [19], which addresses both communication and privacy but uses differential privacy (degrading accuracy), APMKFL achieves communication efficiency and cryptographic privacy simultaneously with no accuracy trade-off. The main contributions of this research are as follows: 1)Proposed an adaptive iterative channel pruning method, enabling edge devices to dynamically adjust pruning thresholds based on the accuracy of pre-pruned models, thereby achieving optimal balance between model structural complexity and prediction accuracy. Proposed an adaptive iterative channel pruning method, enabling edge devices to dynamically adjust pruning thresholds based on the accuracy of pre-pruned models, thereby achieving optimal balance between model structural complexity and prediction accuracy. 2)Employed multi-key BGV homomorphic encryption technology, allowing edge devices to encrypt model parameters through jointly generated aggregation public keys, enabling servers to perform secure aggregation of local models from edge devices in ciphertext state to update the global model. Employed multi-key BGV homomorphic encryption technology, allowing edge devices to encrypt model parameters through jointly generated aggregation public keys, enabling servers to perform secure aggregation of local models from edge devices in ciphertext state to update the global model. 3)Conducted comprehensive experimental validation on standard MNIST and CIFAR-10 datasets, with results demonstrating that the proposed scheme significantly reduces communication overhead while maintaining high model accuracy, further enhancing data privacy and security protection. Conducted comprehensive experimental validation on standard MNIST and CIFAR-10 datasets, with results demonstrating that the proposed scheme significantly reduces communication overhead while maintaining high model accuracy, further enhancing data privacy and security protection. Related works This section discusses related research works based on two major challenges faced by federated learning: high communication overhead and data privacy security. Addressing the limited computational and communication resources of edge devices in federated learning, an increasing number of studies have incorporated model compression techniques into the federated learning framework. Major model compression methods include model pruning [20], knowledge distillation [21], and parameter quantization [22]. Jeong et al. [23] proposed federated distillation, utilizing knowledge distillation to transfer knowledge from large teacher models to smaller student models in federated learning environments. Prakash et al. [24] suggested using parameter quantization techniques to reduce parameter bit-width representation, thereby effectively decreasing communication overhead. This paper primarily focuses on techniques combining federated learning with model pruning, achieving greater efficiency by directly pruning the original model. Model pruning aims to remove redundant neurons, weights, or connections to reduce model size, lower computational and storage costs, while minimizing accuracy loss. For instance, Caldas et al. [1] employed lossy compression in federated learning to compress model storage footprint, reducing communication burden between edge devices and central servers by disabling sets of low-importance parameters. Yu et al. [17] proposed using gating networks on edge devices to eliminate redundant neurons. Vahidian et al. [18] introduced a method combining structured and unstructured pruning approaches. However, these works generally adopt one-time pruning strategies. Research by Frankle et al. [25] demonstrated that iterative pruning approaches significantly reduce model accuracy loss compared to one-time pruning. Therefore, this paper conducts research from the perspective of iterative structural pruning. Additionally, federated learning faces significant challenges in data privacy security. Within the federated learning framework, edge devices share only partial model parameters or gradients with the central server, rather than raw data. However, attackers may still exploit this shared information to infer private data from specific edge devices. To enhance data privacy protection in federated learning, researchers have proposed various solutions: Wei et al. [26] introduced a federated learning scheme incorporating differential privacy techniques, achieving edge device data privacy protection at the cost of significantly reduced model performance; Fang et al. [12] proposed using additive homomorphic encryption to protect model updates through the Paillier cryptosystem for privacy-preserving federated learning; Vedaraj et al. [27] presented a decentralized system that conducts secure statistical analysis on distributed datasets by applying the ElGamal elliptic curve additive homomorphic cryptosystem. Notably, in the aforementioned works, all participating edge devices utilize the same encryption and decryption keys. This design poses potential risks—private data information may leak between edge devices. More seriously, any curious edge device colluding with the server would compromise the data privacy of other edge devices. Recently, some research has begun to simultaneously address the dual challenges of data privacy security and communication overhead in federated learning. Hu et al. [19] proposed an innovative approach that reduces communication rounds through periodic averaging while integrating secure aggregation with differential privacy techniques to effectively prevent data leakage. Drawing on the advantages of the method proposed by Hu et al. [19], our research scheme aims to tackle both fronts: reducing the communication frequency between edge devices and servers while decreasing the volume of transmitted data, thereby comprehensively lowering communication overhead while providing robust privacy protection mechanisms. Reviewing existing works, we identify four critical gaps: (1) Pruning-only methods reduce communication but offer no privacy guarantees. (2) Single-key HE schemes protect privacy but are vulnerable to collusion and do not reduce communication volume. (3) Differential privacy methods sacrifice model accuracy due to noise injection. (4) Existing joint approaches either rely on trusted server-side data or inherit DP’s accuracy penalty. APMKFL is the first scheme to jointly address all three objectives—communication efficiency, collusion-resistant privacy, and accuracy preservation—without trusted server data or noise-based privacy. Adaptive pruning multi-key federated learning framework This section provides a detailed exposition of the proposed framework. First, it systematically introduces the technical principles and implementation process of the adaptive iterative channel pruning method. Next, it thoroughly analyzes the core mechanisms of the multi-key BGV homomorphic encryption method. Finally, it comprehensively presents the complete process and key stages of the joint model training performed by edge devices. Adaptive iterative channel pruning A natural baseline is to apply iterative pruning with a fixed rate α throughout all communication rounds. However, this approach faces a fundamental dilemma: (1) If α is set too conservatively, communication savings are minimal. (2) If α is set aggressively, accuracy degrades significantly as the number of edge devices C increases, because higher C introduces more statistical heterogeneity, requiring greater model capacity to accommodate the distributional diversity across devices. (3) The optimal α differs across datasets, models, communication rounds, and device counts—making a universally good fixed choice impossible without per-experiment manual tuning. Our adaptive mechanism resolves this by automatically identifying the highest pruning rate satisfying the accuracy constraint in each round, eliminating manual selection entirely. As illustrated in Fig 1, this paper investigates a federated learning architecture comprising a central server S and a set of edge devices C={c1,c2,...,cn}. Each device c∈C maintains a private local dataset 𝒟c with |𝒟c| data samples. The global dataset is defined as 𝒟=⋃c∈C𝒟c, containing a total of N=∑c∈C|𝒟c| samples. Under the standard federated learning framework, the central server S periodically aggregates local model parameters from all edge devices and learns the global model parameters w by minimizing the global empirical risk: The overview of system model. where ℱ(w) denotes the global empirical risk function defined over the entire federated system, and fc(wc) represents the local objective function with parameters wc specific to device c. For each edge device c, the accuracy constraint for channel pruning is formally defined as follows: Given an n-layer CNN model with parameters wc and original channel configuration ℒ={l1,l2,…,ln} where li denotes the number of output channels in the i-th layer, the channel pruning aims to find an optimized channel configuration ℒ′⊆ℒ such that the pruned compact model MPruned(ℒ′,wc′) satisfies the target accuracy threshold Accg on the validation set 𝒟val: where wc′ represents the parameters of the pruned model MPruned(ℒ′,wc′), and Accg is the predefined accuracy threshold. The core objective of model pruning can be formalized as a constrained optimization problem: to find the optimal channel configuration that minimizes the model’s structural complexity while satisfying the accuracy constraint. In federated learning, the essence of model pruning lies in generating a mask structural binary matrix maskc for each edge device c, formally defined as: where wc* denotes the sparsified model parameters after pruning, and ⊙ represents the Hadamard product (element-wise multiplication). During the model pruning phase, the proposed method incorporates the channel pruning strategy presented in [15] by applying an L1 regularization constraint to the scaling factor γ of the batch normalization layer. This allows for dynamic identification of non-critical channels during training, generating a corresponding binary mask matrix where each element indicates the presence or absence of the respective channel rather than directly representing parameter pruning. Within the federated learning framework, each edge device trains a local model using its private local data and engages in a total of T communication rounds with the central server for model aggregation. Algorithm 1 proposes an adaptive threshold-based iterative channel pruning framework, which consists of two phases: 1)Local Model Training (Lines 1–3): During the t-th federated communication round, each edge device trains local model parameters wct on its private dataset 𝒟c, while simultaneously evaluating the model’s validation accuracy Acct. Local Model Training (Lines 1–3): During the t-th federated communication round, each edge device trains local model parameters wct on its private dataset 𝒟c, while simultaneously evaluating the model’s validation accuracy Acct. 2)Dynamic Channel Pruning (Lines 4–13): When Acct meet predefined accuracy constraints Accg, the pruning process proceeds through the following steps: Candidate pruning ratios α∈[0,1] are uniformly sampled. During pre-pruning operations, the system first performs a complete sorting of scaling factors γ in batch normalization layers. A pruning threshold is then calculated as y[(α×γsum)], where γsum denotes the total number of scaling factors. A binary mask matrix maskcα is generated by comparing each channel’s γ value against the threshold: channels exceeding the threshold are preserved (marked as 1) while others are pruned (marked as 0). Each candidate pruning ratio α undergoes validation set evaluation, with the corresponding inference accuracy recorded as Accα. Ultimately, the system selects the pruning configuration with the highest validation accuracy, determining the optimal mask matrix maskcα and model architecture. Dynamic Channel Pruning (Lines 4–13): When Acct meet predefined accuracy constraints Accg, the pruning process proceeds through the following steps: Candidate pruning ratios α∈[0,1] are uniformly sampled. During pre-pruning operations, the system first performs a complete sorting of scaling factors γ in batch normalization layers. A pruning threshold is then calculated as y[(α×γsum)], where γsum denotes the total number of scaling factors. A binary mask matrix maskcα is generated by comparing each channel’s γ value against the threshold: channels exceeding the threshold are preserved (marked as 1) while others are pruned (marked as 0). Each candidate pruning ratio α undergoes validation set evaluation, with the corresponding inference accuracy recorded as Accα. Ultimately, the system selects the pruning configuration with the highest validation accuracy, determining the optimal mask matrix maskcα and model architecture. Algorithm 1. Adaptive Iterative Channel Pruning Algorithm. 1: for t=1,2,…,T do 2: Train model on local dataset 𝒟c: wct←Train(𝒟c) 3: Calculate current model accuracy: Acct←Evaluate(wct) 4: if Accg ≤ Acct then 5: (y,i)←Sort(γ) ▷(y as the value of γ, i: index) 6: for select pruning rate α∈{α1,α2,…} do 7: Calculate threshold: thre←y[(α×γsum)] 8: Generate mask matrix: maskcα={1,γ>thre0,otherwise 9: Pre-pruning: wc*←wct⊙maskcα 10: Evaluate pruned model: Accα←Evaluate(wc*) 11: end for 12: Select maximum accuracy: Max Accα←max({Accα}) 13: Record optimal mask: maskcα←argmaxmaskcαAccα 14: Save pruned model: wc*←argmaxwc*Accα 15: else 16: continue 17: end if 18: end for It is important to clarify that the candidate pruning rate selection in Algorithm 1 is fundamentally different from a conventional offline hyperparameter search. In a standard hyperparameter search, a fixed set of candidate values is evaluated on a held-out validation set before training begins, and the best value is then applied statically throughout the entire training process. In contrast, our adaptive mechanism operates online, within each communication round, and its behavior changes dynamically as the model evolves. Specifically, the pruning threshold is not chosen from a pre-fixed discrete grid; rather, it is derived from the current model’s batch normalization scaling factors γ via sorted ranking. Since the distribution of γ changes with every gradient update, the set of channels identified for pruning at a given rate α is unique to each round. This means that the same nominal rate α can preserve entirely different subsets of channels in round t versus round t + 1. Therefore, the mechanism is self-calibrating: the accuracy feedback in each round directly shapes the effective pruning structure, a property that offline hyperparameter search cannot replicate. Table 1 summarizes the key distinctions. 10.1371/journal.pone.0349432.t001Table 1Comparison with conventional hyperparameter search.AspectConventional Hyperparameter SearchOur Adaptive MechanismExecution TimingBefore training, offlineDuring each training round, onlineCandidate SourceManually predefined discrete gridDynamically generated by sorted ranking of BN layer’s γPruning StructureFixed (independent of rounds)Varies per round (evolves with the γ distribution)Accuracy FeedbackUsed only to select the optimal hyperparameterDirectly determines the final pruning mask for the current roundAdaptivityNo (remains fixed after selection)Yes (adapts automatically as the model converges)Computational CostOne-time offline costIncremental per-round cost Comparison with conventional hyperparameter search. The extra computational cost introduced by the adaptive pruning decision in each communication round consists of two components. First, sorting the BN scaling factors γ incurs O(|γ|log|γ|) time, where |γ| denotes the total number of channels across all layers (a one-time operation per round regardless of k). Second, for each of the k candidate pruning rates, the algorithm performs one forward pass over the local validation set Dval to compute the pruned model’s accuracy. Each forward pass costs O(|Dval|·FLOPs(𝐰c*)), where FLOPs(𝐰c*) is the inference cost of the pruned model—which is smaller than the full model. The total additional cost per round is therefore O(|γ|log|γ|+k·|Dval|·FLOPs(𝐰c*)). Multi-key BGV homomorphic encryption mechanism Chen et al. [16] proposed the first BGV-based multi-key fully homomorphic encryption (MKFHE) methods, which operates on ring elements and derives its security from the ring learning with errors (RLWE) problem. This multi-key homomorphic encryption mechanism allows distinct edge devices to perform encryption using individual private keys, while requiring collaborative participation from all involved devices during decryption. In our federated learning framework, we specifically leverage the additive homomorphic property of MKFHE. By aggregating public keys from all edge devices to form a unified public key, the scheme achieves secure parameter aggregation. The specific implementation details of the sub-algorithm are described as follows: 1)Initialization Phase: Given a security parameter λ and an edge device set 𝒞, the system establishes cryptographic parameters through the following steps: I) Algebraic Structure Definition: Construct the cyclotomic polynomial ring R=ℤ[X]/(Xn+1) with polynomial dimension n being a power of two, ensuring compatibility with NTT (Number Theoretic Transform) computations. II) Modulus Selection: Choose coprime integers q (ciphertext modulus) and p (plaintext modulus) such that p≪q, which guarantees effective noise control in homomorphic operations. III) Noise Distribution: Define a bounded discrete Gaussian distribution χ over R with noise bound B, governing the statistical properties of encryption noise terms. IV) Public Parameter Generation: Randomly sample a public vector a←Rq, where the quotient ring Rq is constructed via R/qR≅ℤq[X]/(Xn+1). The complete public parameters are pp={R,χ,B,q,a,p}. Initialization Phase: Given a security parameter λ and an edge device set 𝒞, the system establishes cryptographic parameters through the following steps: I) Algebraic Structure Definition: Construct the cyclotomic polynomial ring R=ℤ[X]/(Xn+1) with polynomial dimension n being a power of two, ensuring compatibility with NTT (Number Theoretic Transform) computations. II) Modulus Selection: Choose coprime integers q (ciphertext modulus) and p (plaintext modulus) such that p≪q, which guarantees effective noise control in homomorphic operations. III) Noise Distribution: Define a bounded discrete Gaussian distribution χ over R with noise bound B, governing the statistical properties of encryption noise terms. IV) Public Parameter Generation: Randomly sample a public vector a←Rq, where the quotient ring Rq is constructed via R/qR≅ℤq[X]/(Xn+1). The complete public parameters are pp={R,χ,B,q,a,p}. 2)Key Generation Phase: During the key generation process for edge device c, the core private parameter zc←R3 is first randomly selected from the ternary polynomial ring R3=ℤ[x]/(xn+1,3), forming a computationally efficient private key skc=sc:=(1,−zc)∈R32. This structure fixes the leading coefficient as 1 to eliminate modular reduction in polynomial multiplication, thereby significantly reducing computational overhead. Next, a noise term ec←χ is sampled from a bounded noise distribution χ, and combined with the predefined public parameter a to generate the public key component bc=azc+pecmodq. The public key is then defined as pkc=(bc,a)∈Rq2, whose security relies on the hardness assumption of the RLWE problem. The resulting key pair {pkc,skc} guarantees ciphertext consistency, where the public key is used for data encryption and the private key exclusively serves decryption purposes. Key Generation Phase: During the key generation process for edge device c, the core private parameter zc←R3 is first randomly selected from the ternary polynomial ring R3=ℤ[x]/(xn+1,3), forming a computationally efficient private key skc=sc:=(1,−zc)∈R32. This structure fixes the leading coefficient as 1 to eliminate modular reduction in polynomial multiplication, thereby significantly reducing computational overhead. Next, a noise term ec←χ is sampled from a bounded noise distribution χ, and combined with the predefined public parameter a to generate the public key component bc=azc+pecmodq. The public key is then defined as pkc=(bc,a)∈Rq2, whose security relies on the hardness assumption of the RLWE problem. The resulting key pair {pkc,skc} guarantees ciphertext consistency, where the public key is used for data encryption and the private key exclusively serves decryption purposes. 3)Encryption Phase: To encrypt a plaintext μ∈Rp=ℤ[x]/(xn+1,p) using the public key pkc=(bc,a)∈Rq2 of edge device c, first sample a random polynomial rc←R2=ℤ[x]/(xn+1,2) and independently draw noise terms e0,e1←χ from a bounded noise distribution. The ciphertext ctc=(c0,c1)∈Rq2 is computed as: Encryption Phase: To encrypt a plaintext μ∈Rp=ℤ[x]/(xn+1,p) using the public key pkc=(bc,a)∈Rq2 of edge device c, first sample a random polynomial rc←R2=ℤ[x]/(xn+1,2) and independently draw noise terms e0,e1←χ from a bounded noise distribution. The ciphertext ctc=(c0,c1)∈Rq2 is computed as: This encryption employs a dual-masking mechanism (via rc and e0, e1) to ensure semantic security, with its safety reduced to the hardness assumption of the RLWE problem. 4)Decryption Phase: Upon receiving ciphertext ctc=(c0,c1), edge device c computes the inner product with private key skc=(1,−zc): Decryption Phase: Upon receiving ciphertext ctc=(c0,c1), edge device c computes the inner product with private key skc=(1,−zc): When the noise term e satisfies ‖p·e‖∞