MergeGuard: Efficient Thwarting of Trojan Attacks in Machine Learning Models
Soheil Zibakhsh Shabgahi * , Yaman Jandali * , Farinaz Koushanfar · 2025
Read MergeGuard: Efficient Thwarting of Trojan Attacks in Machine Learning Models by Soheil Zibakhsh Shabgahi * , Yaman Jandali * , Farinaz Koushanfar (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-04015.txt.
Introduction
Soheil Zibakhsh Shabgahi and co-authors present MergeGuard, a defense against Trojan attacks on trained models. In such an attack, a hidden trigger makes a model sort marked inputs into a class the attacker picks. Their 2025 paper notes the risk when a model comes from a third party that cannot be trusted.
MergeGuard works after training: it straightens and merges the fully connected layers of a model. In tests on Transformer models, the method held accuracy steady while the rate of successful Trojan attacks fell. It beat the standard fix of fine-tuning in their proof of concept runs. Machine learning readers get a cheap post-training shield worth testing on their own models.
Catalog record
| CREATOR | Soheil Zibakhsh Shabgahi * , Yaman Jandali * , Farinaz Koushanfar |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 30 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-04015
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MergeGuard: Efficient Thwarting of Trojan Attacks in Machine Learning Models Soheil Zibakhsh Shabgahi * , Yaman Jandali * , Farinaz Koushanfar Abstract This paper proposes MergeGuard , a novel methodology for mitigation of AI Trojan attacks. Trojan attacks on AI models cause inputs embedded with triggers to be misclassified to an adversary’s target class, posing a significant threat to model usability trained by an untrusted third party. The core of MergeGuard is a new post-training methodology for linearizing and merging fully connected layers which we show simultaneously improves model generalizability and performance. Our Proof of Concept evaluation on Transformer models demonstrates that MergeGuard maintains model accuracy while decreasing trojan attack success rate, outperforming commonly used (post-training) Trojan mitigation by fine-tuning methodologies. † † footnotetext: *Equal contribution. I Introduction Utilizing Artificial Intelligence (AI) for automation is increasingly ingrained in various technical fields. Recent research has shown that larger Deep Neural Networks (DNNs) with greater expressive capacity can more effectively approximate complex real-world functions and achieve higher accuracy [ 1 , 2 ] . As model architectures grow in size, so too do the datasets required to train these data-hungry models. To conserve resources, modern Machine Learning (ML) practitioners frequently rely on pretrained models or publicly available datasets, exposing themselves to the risk of maliciously manipulated models or tampered datasets. Numerous studies [ 3 , 4 ] have demonstrated the feasibility of training trojaned models , where an adversary injects triggers into the model to force specific inputs to be misclassified to a target label. With the increasing deployment of ML models in critical applications such as in autonomous vehicles, medical diagnostics, and financial decision-making, [ 5 , 6 , 7 ] the potential consequences of such attacks continue to become more severe. This growing concern regarding the safe deployment of AI models has prompted a surge of research focused on trojan detection and mitigation [ 4 , 8 , 9 , 10 ] . A straightforward approach involves using standard fine-tuning, which may be effective for simpler models or against more subtle attacks. However, Wu et al. [ 11 ] found that vanilla fine-tuning often struggles as a mitigation strategy when applied using smaller benign datasets against more sophisticated, powerful attacks. As a result, more advanced mitigation strategies have been proposed, typically leveraging computationally heavy techniques or enhancements of fine-tuning tailored for trojan removal. While many of these methods have shown promise in mitigating trojans for smaller models, such as PreAct-ResNet18 [ 12 , 13 ] , there remains a significant gap in research addressing the mitigation of trojans in larger transformer based architectures, such as Vision Transformers. The Vision Transformer (ViT) is growing in popularity due to its exeptional performance and ability to extend beyond vision to audio, video, and other applications [ 14 , 15 ] . Subramanya et al. [ 16 ] observed that vision transformer architectures are especially vulnerable to trojan attacks, a finding corroborated by our own experiments. Given the substantial size of transformer-based models and the extensive datasets required for their training, these models and their training datasets are frequently obtained from online sources. This reliance on external pretrained models and datasets makes them especially prone to manipulation. Therefore, it is crucial to develop effective mitigation strategies that reliably generalize to larger architectures, such as those of transformer based models, to ensure the safe deployment of ML models in real-world applications. While prior state-of-the-art methods have shown promise in detecting and mitigating trojans in smaller convolutional models, they often struggle when extended to transformer architectures. In this work, we propose a novel architecture-agnostic mitigation strategy, MergeGuard , which we demonstrate to be both effective and computationally efficient for vision transformers as well as CNN architectures. Our approach not only shows enhanced resilience against trojan attacks but also preserves the high performance of the underlying model, making it suitable for deployment in high-stakes environments. Additionally, we show that MergeGuard serves a dual purpose of post-training compression, achieving up to a 15% reduction in model size for transformers without compromising accuracy. MergeGuard acts as an effective trojan mitigation strategy by means of post-training compression, removing the layers containing backdoor-related neurons. A regularization term controls the rate of compression, optimizing for model size and performance simultaneously. This regularizer reduces the model depth by adaptively linearizing the activation functions between fully connected layers, allowing for the fusion of adjacent linear layers. This effectively reduces the overall depth and complexity of the model without significantly impacting its performance. In summary, our contributions are as follows: • We demonstrate the poor generalizability of a number of works targeted at cleansing CNN-based models applied to trojaned transformer models. • We introduce MergeGuard , a novel, model-agnostic trojan mitigation method relying on post-processing and showcase its robust performance in mitigating a range of trojan attacks. • We evaluate the effectiveness of MergeGuard across different architectures, including CNNs, establishing its versatility beyond transformer models. • We show that MergeGuard achieves significant computational efficiency, offering speedups up to 17.7x compared to other methods, making it a practical solution for large-scale deployment. • We demonstrate MergeGuard ’s capability as a compression-aware regularizer, achieving up to 15\% parameter reduction and 14\% MAC reduction in transformer models without compromising accuracy. • Our code is accessible to support reproducibility and further advancements in trojan mitigation research. 1 1 1 https://github.com/yjandali/BackdoorBench-MergeGuard II Background and Related Work II-A Trojan Attacks Trojan attacks - also referred to as backdoor attacks - have emerged as a tangible and critical threat to the safe use of deep neural networks in the real world. While adversarial attacks refer to those which perturb images at inference time [ 17 ] , trojan attacks involve training a model to misclassify inputs embedded with triggers to an attacker’s target class. These attacks have become especially concerning in higher stakes scenarios such as in medical diagnostics or self driving systems [ 5 , 6 ] . A classic scenario of trojaning, as discussed in the foundational paper BadNets [ 18 ] , is the misclassification of a stop sign to a speed limit sign. Upon deployment, simply placing a trigger item in view can cause a trojaned model to behave maliciously. This example is shown in Figure 1 and demonstrates real-world vulnerabilities. Even discrete and inconspicuous modifications to physical objects in images can exploit trojans within machine learning models, leading to misclassifications with severe consequences. (a) Clean Image (b) Trigger Embedded Image Fig. 1 : A machine learning model may be trained to wrongly classify images with triggers such as post-it notes to speed limit signs. This could lead to dangerous misclassifications of stop signs, potentially preventing a car from stopping if used in an autonomous vehicle. II-B Types of Attacks Gu et al. [ 18 ] introduce one of the earliest backdoor attacks in deep learning, BadNets . This attack involves the poisoning of a dataset. Given a training dataset \mathcal{D}=\{(\mathbf{x}_{i},y_{i})\} , a subset of samples is selected and a trigger pattern \mathbf{t} is embedded into these images to create poisoned samples \mathbf{x}_{i}^{\prime}=\mathbf{x}_{i}+\mathbf{t} . These modified samples are then relabeled with the attacker-specified target label y_{t} . The poisoned dataset \mathcal{D}^{\prime}=\{(\mathbf{x}_{i}^{\prime},y_{t})\}\cup\{(\mathbf{x}_{i},y_{i})\} is then used for training the trojaned