SafeLM: Unified Privacy-Aware Optimization for Trustworthy Federated Large Language Models

Noor Islam S. Mohammad, Uluğ BayazıtIstanbul Technical University {islam23, ulugbayazit}@itu.edu.tr · 2026

Read SafeLM: Unified Privacy-Aware Optimization for Trustworthy Federated Large Language Models by Noor Islam S. Mohammad, Uluğ BayazıtIstanbul Technical University {islam23, ulugbayazit}@itu.edu.tr (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-16606.txt.

Introduction

Mohammad and Bayazıt present SafeLM, a single safety framework for shared language models. It covers privacy, security, false claims and hostile attacks at once. SafeLM trains models across sites with encrypted gradient methods for privacy. It adds attack defenses, grounded decoding against made-up claims, and robust averaging that limits rebuild quality. On factuality, toxicity and membership tests, it spots 98.0 percent of harmful content. It cuts traffic by 96.9 percent and drops inversion signal from 31.7 decibels to 15.1 decibels.

Catalog record

CREATORNoor Islam S. Mohammad, Uluğ BayazıtIstanbul Technical University {islam23, ulugbayazit}@itu.edu.tr
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35901
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.16606

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SafeLM: Unified Privacy-Aware Optimization for Trustworthy Federated Large Language Models Noor Islam S. Mohammad, Uluğ BayazıtIstanbul Technical University {islam23, ulugbayazit}@itu.edu.tr Abstract Large language models (LLMs) are increasingly deployed in high-stakes domains, yet a unified treatment of their overlapping safety challenges remains lacking. We present SafeLM , a framework that jointly addresses four pillars of LLM safety: privacy, security, misinformation, and adversarial robustness. SafeLM combines federated training with gradient smartification and Paillier encryption for privacy, integrates defenses against training- and inference-time attacks, employs contrastive grounding with calibrated decoding to reduce hallucinations, and introduces alignment-aware binarized aggregation to enhance robustness while maintaining bounded reconstruction quality. Across benchmarks on factuality, toxicity, and membership inference, SafeLM achieves 98.0% harmful-content detection accuracy, reduces communication by 96.9%, and lowers gradient inversion PSNR from 31.7 dB to 15.1 dB. Ablations show that each component contributes independently, whereas their integration yields a strong privacy–utility–efficiency trade-off for deploying trustworthy LLMs. 1 Introduction The rapid deployment of large language models (LLMs) has elevated safety from a research concern to an operational requirement ( 6 ; 21 ) . Four interconnected threat surfaces arise: (i) Privacy , LLMs may memorize training data, enabling extraction and membership inference, while standard federated learning (FL) remains vulnerable to gradient inversion ( 7 ; 30 ) ; (ii) Security , adversaries can inject backdoors, craft adversarial prompts, or perform model stealing ( 26 ; 23 ) ; (iii) Misinformation , instruction-tuned models hallucinate confidently, necessitating integrated grounding mechanisms ( 16 ; 18 ) ; and (iv) Adversarial Robustness , input perturbations degrade reliability under strict latency constraints. Existing solutions address these aspects in isolation, leading to incompatible defenses and unclear interactions. Contributions. We propose SafeLM , a unified framework for jointly addressing privacy, security, misinformation, and robustness in LLMs. First, we introduce a federated training and deployment pipeline that co-optimizes all four objectives. Second, we propose gradient smartification , a median-based binarization scheme achieving 32\times communication compression with bounded inversion quality (PSNR \leq 15.1 dB). Third, we develop a calibrated misinformation-detection method via contrastive grounding with temperature scaling, reducing hallucinations by 41% on TruthfulQA while preserving > 97% ROUGE-L. Finally, we provide a unified ablation and threat analysis quantifying trade-offs across safety components. 2 Background and Related Work 2.1 Privacy in Language Model Training Training-data memorization is a well-documented property of large-scale language models ( 7 ; 11 ) . Differential Privacy (DP-SGD) provides formal (\varepsilon,\delta) -guarantees ( 1 ) , but at the cost of significant perplexity degradation for \varepsilon<3 . Federated learning distributes training across data-holders ( 17 ) , yet transmitted gradients can be reverse-engineered to reconstruct training samples ( 30 ; 12 ) . 2.2 Security: Backdoors and Prompt Injection Backdoor attacks embed hidden triggers in model weights during fine-tuning, causing targeted misbehavior on trigger-containing inputs ( 8 ; 26 ) . Prompt-injection attacks exploit instructions following to override safety constraints at inference time ( 23 ) . Recent work shows that gradient-level defenses, including gradient clipping and sign-based aggregation, partially mitigate backdoor insertion during federated fine-tuning ( 25 ) . 2.3 Misinformation and Hallucination LLMs hallucinate for multiple reasons: distributional mismatch between pre-training and deployment contexts, insufficient grounding in retrieved knowledge, and overconfidence in low-probability completions ( 16 ; 18 ) . Retrieval augmentation ( 14 ) and calibration methods ( 13 ) partially address these issues but lack formal correctness guarantees. 2.4 Adversarial Robustness for NLP Adversarial examples for text exploit the discrete token space via character-level substitutions ( 10 ) , word-level replacements constrained by semantic similarity ( 3 ) , and continuous perturbations in embedding space ( 19 ) . Certified robustness via randomized smoothing ( 9 ) has been extended to NLP but remains computationally expensive at scale. 3 Threat Model and Safety Desiderata We consider a federated fine-tuning setting with K clients, a central server, and downstream users. We model three adversaries: (i) an honest-but-curious server observing ciphertexts and attempting gradient inversion; (ii) up to \lfloor K/5\rfloor malicious clients injecting poisoned updates or backdoors; and (iii) an inference-time adversary crafting prompts to elicit harmful or hallucinated outputs. Our safety desiderata are (S1) Gradient Confidentiality —client updates remain unrecoverable from server-visible information; (S2) Backdoor Resistance —the global model avoids trigger-conditioned behaviors; (S3) Factual Consistency , outputs are calibrated with hallucinations flagged or suppressed; and (S4) Adversarial Robustness , model behavior remains stable under semantically preserving perturbations. 4 The SafeLM Framework 4.1 Overview SafeLM integrates four co-designed modules within a federated fine-tuning loop to jointly address key safety objectives. The Privacy Engine (PE) combines gradient smartification with Paillier homomorphic encryption to ensure the confidentiality of gradients (S1). The Security Module (SM) employs median-based Byzantine filtering alongside trigger detection to mitigate poisoned or backdoored updates (S2). The Misinformation Guard (MG) incorporates contrastive grounding with calibrated decoding to suppress hallucinations and improve factual consistency (S3). Finally, the Robustness Head (RH) leverages adversarial training with smartified gradients to enhance stability under semantically preserving perturbations (S4). 4.2 Phase 1: Federated Fine-Tuning with LoRA Let \mathcal{S}=\{1,\ldots,K\} denote participating clients. At round r , the server broadcasts global adapter parameters W^{(r)} (LoRA rank- \rho matrices). Each client i performs E local epochs, minimizing cross-entropy on its private corpus \mathcal{D}_{i} : W_{i}^{(r+1)}=W_{i}^{(r)}-\eta\nabla\mathcal{L}(W_{i}^{(r)},\,\mathcal{D}_{i}). (1) the client update is \Delta_{i}^{(r)}=W_{i}^{(r+1)}-W^{(r)} . 4.3 Phase 2: Gradient Smartification To simultaneously reduce uplink bandwidth and harden against gradient inversion, we apply a median-based statistical binarization operator \Phi(\cdot) : \Delta_{i,j}^{\mathrm{bin}}=\begin{cases}+1&\text{if }\Delta_{i,j}^{(r)}\geq\theta_{i}\\ -1&\text{otherwise}\end{cases},\quad\theta_{i}=\mathrm{median}\!\left(\bigl|\Delta_{i}^{(r)}\bigr|\right). (2) This compresses each 32-bit floating-point gradient to a single bit, achieving a 32\times reduction in payload. Unlike zero-threshold signSGD ( 4 ) , our per-client adaptive threshold suppresses low-magnitude components below the empirical distribution median, reducing stochastic noise under the heavy-tailed gradient distributions characteristic of LLM fine-tuning. Section 5 proves alignment of \Delta_{i}^{\mathrm{bin}} with the true gradient under mild distributional assumptions. 4.4 Phase 3: Homomorphic Encryption Each client encrypts its binary update element-wise using the Paillier scheme \mathcal{E}(\cdot) ( 22 ) : C_{i}^{(r)}[j]=\mathcal{E}_{pk}\!\left(\Delta_{i,j}^{\mathrm{bin}}\right)=g^{\Delta_{i,j}^{\mathrm{bin}}}\cdot r_{j}^{n}\bmod n^{2}, (3) where (pk,sk)=(n,g,\lambda,\mu) is a 2048-bit Paillier keypair and r_{j}\xleftarrow{\$}\mathbb{Z}_{n}^{*} . Paillier’s additive homomorphism enables the server to aggregate without decrypting individual updates: C_{\mathrm{agg}}^{(r)}[j]=\prod_{i=1}^{K}C_{i}^{(r)}[j]\bmod n^{2}=\mathcal{E}_{pk}\!\!\left(\sum_{i=1}^{K}\Delta_{i,j}^{\mathrm{bin}}\right). (4) This satisfies IND-CPA security under the Decisional Composite Residuosity Assumption (DCRA), satisfying desideratum S1. 4.5 Phase 4: Byzantine Filtering and Global Update After decryption, the server normalizes and applies a coordinate-wise median filter to resist Byzantine poisoning from malicious clients: \hat{\Delta}_{\mathrm{agg}}^{(r)}[j]=\mathrm{median}\!\left\{\Delta_{1,j}^{\mathrm{bin}},\ldots,\Delta_{K,j}^{\mathrm{bin}}\right\}, (5) then applies the global update with Nesterov momentum: W^{(r+1)}=W^{(r)}+\alpha\cdot\hat{\Delta}_{\mathrm{agg}}^{(r)}+\mu\!\left(W^{(r)}-W^{(r-1)}\right),\quad\mu=0.9. (6) 4.6 Misinformation Guard: Contrastive Grounding At inference time, each generated claim \hat{y} is scored against a retrieved evidence set \mathcal{E}=\{e_{1},\ldots,e_{m}\} from a read-only knowledge store: \mathrm{FaithScore}(\hat{y},\mathcal{E})=\frac{1}{m}\sum_{i=1}^{m}\mathrm{NLI}\!\left(\hat{y},\,e_{i}\right)\cdot\mathrm{conf}\!\left(\hat{y}\right), (7) where \mathrm{NLI}(\cdot) is an entailment classifier and \mathrm{conf}(\hat{y}) is the temperature-calibrated model confidence. Claims with \mathrm{FaithScore}<\tau_{\mathrm{MG}} are either abstained or regenerated with retrieval-augmented prompting. 4.7 Robustness Head: Adversarial Fine-Tuning During each federated round, clients augment their local batch with adversarial examples generated via projected gradient descent in the embedding space: x^{\mathrm{adv}}=x+\delta^{*},\quad\delta^{*}=\argmax_{\|\delta\|_{\infty}\leq\varepsilon_{\mathrm{adv}}}\mathcal{L}(W_{i},x+\delta,y). (8) The mixed objective \mathcal{L}_{\mathrm{adv}}=(1-\lambda_{\mathrm{adv}})\mathcal{L}+\lambda_{\mathrm{adv}}\mathcal{L}(x^{\mathrm{adv}}) is minimized during local training, with smartified gradients transmitted as in Section 4.3 . 5 Theoretical Analysis 5.1 Convergence under Gradient Smartification Lemma 1 (Descent under Median-Threshold Smartification) . Let L(W) be L -smooth and bounded below. Let’s \tilde{g}_{t} denote the smartified gradient with cosine similarity \cos(\theta_{t})=\frac{\langle g_{t},\,\tilde{g}_{t}\rangle}{\|g_{t}\|\|\tilde{g}_{t}\|}\geq\gamma>0 . Then for

Read the complete text