D 2 R: dual regularization loss with collaborative adversarial generation for model robustness

Zhenyu Liu, Huizhi Liang, Rajiv Ranjan, Zhanxing Zhu, Vaclav Snasel, Varun Ojha · 2025

Read D 2 R: dual regularization loss with collaborative adversarial generation for model robustness by Zhenyu Liu, Huizhi Liang, Rajiv Ranjan, Zhanxing Zhu, Vaclav Snasel, Varun Ojha (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-07056.txt.

Introduction

Zhenyu Liu and five co-authors strengthen in 2025 neural networks against adversarial attacks with a dual regularization loss and team-based attack generation. Tiny crafted tweaks can fool image models, and defenses keep playing catch-up. Prior team-learning defenses guided the target model weakly and generated attacks without cooperation.

Their loss steers the model on both clean and attack distributions, using the best features of several loss shapes. Attacks are then crafted by a guide model and the target working from shared gradients. Across three image benchmarks and two network types, the pairing produced markedly tougher models. Robustness gains of this kind matter wherever vision systems guard money, vehicles, or doors.

Catalog record

CREATORZhenyu Liu, Huizhi Liang, Rajiv Ranjan, Zhanxing Zhu, Vaclav Snasel, Varun Ojha
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-07056

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

D 2 R: dual regularization loss with collaborative adversarial generation for model robustness Zhenyu Liu, Huizhi Liang, Rajiv Ranjan, Zhanxing Zhu, Vaclav Snasel, Varun Ojha Abstract The robustness of Deep Neural Network models is crucial for defending models against adversarial attacks. Recent defense methods have employed collaborative learning frameworks to enhance model robustness. Two key limitations of existing methods are (i) insufficient guidance of the target model via loss functions and (ii) non-collaborative adversarial generation. We, therefore, propose a dual regularization loss (D 2 R Loss) method and a collaborative adversarial generation (CAG) strategy for adversarial training. D 2 R loss includes two optimization steps. The adversarial distribution and clean distribution optimizations enhance the target model’s robustness by leveraging the strengths of different loss functions obtained via a suitable function space exploration to focus more precisely on the target model’s distribution. CAG generates adversarial samples using a gradient-based collaboration between guidance and target models. We conducted extensive experiments on three benchmark databases, including CIFAR-10, CIFAR-100, Tiny ImageNet, and two popular target models, WideResNet34-10 and PreActResNet18. Our results show that D 2 R loss with CAG produces highly robust models. Our code is available at https://github.com/lusti-Yu/D2R.git. 1 Introduction Deep Neural Networks (DNNs) are crucial in solving real-world problems. However, DNNs are vulnerable to adversarial attacks [ 14 , 19 ] . Researchers have been actively investigating defense strategies to improve the robustness of DNN models against adversarial attacks. One of the most effective and well-known defense methods is adversarial training (AT) [ 11 ] . Several studies have shown that adversarial training on large models offers greater robustness (due to large learnable parameters) than the small models [ 12 ] . Similarly, adversarial distillation (AD) methods use a pre-trained teacher model to enhance the robustness of a student model (a target model) [ 10 , 2 ] . Alternative to AD methods is a set of adversarial defense strategies that involves using a small model as a guide to support a target model. These small guide and target models are simultaneously trained to enhance the robustness of the target model. LBGAT is an example of a guide and target framework [ 1 ] . The advantage of using this small learnable guide model to support a target model is (a) its capacity to adaptively engage in the adversarial training process, (b) the non-requirement of a large pre-trained model as a guide, and (c) a learnable guide model can support adversarial training of any size of the target model. However, there are two main limitations in previous works. (i) Constraints based on a wider loss function space exploration, i.e., most works use squared error loss to implement adversarial training by leveraging the properties of a learnable guide model (e.g., [ 1 ] [ 8 ] ). This makes AT dependent on the selected loss function and hinders it from achieving optimal performance. That is, a non-optimal choice of loss function makes insufficient guidance of the target model. (ii) Constraints based on adversarial example generation, i.e., most defense methods (such as TRADES [ 18 ] and PGD-AT [ 11 ] ) use a single-model and independent adversarial example generation techniques. This non-collaborative adversarial generation hinders AT, based on learnable guide-target architecture, from achieving optimal performance since it ignores collaboration between the guide and the target models. Figure 1 : Overview of our D 2 R-CAG framework. (a) Proposed collaborative adversarial generation process, where the guide and target models create perturbations. These perturbations are applied to images to generate adversarial image examples. (b) D 2 R Loss. \mathcal{L}{ce} represents clean training loss on the guide model, \mathcal{L}_{MSE} denotes the mean squared error between the target and guide models, \mathcal{L}_{KL_{a}} is the optimization of adversarial distributions via KL divergence, and \mathcal{L}_{SKL_{c}} is the optimization of clean distributions using the symmetric KL divergence function. The guide and target models are trained with synchronized inputs throughout the adversarial training process. We propose a Dual Regularization Loss (D 2 R Loss) methodology to alleviate the limitations of loss function selection in the guide-target framework (see Fig. 1 (b) that shows adversarial training process). D 2 R loss (Fig. 1 (b)) leverages the strengths of different loss functions to focus precisely on the classification distributions of guide and target models to enhance the target model’s robustness. We decompose the D 2 R loss process into two steps: (i) adversarial distribution optimization and (ii) clean distribution optimization. Specifically, a) an adversarial KL divergence ( \mathcal{L}_{KL_{a}} ) is employed for a precise alignment between the clean classification output distributions (logits) of the guide model and adversarial classification output distributions (logits) of the target model; and b) a clean KL divergence ( \mathcal{L}_{SKL_{a}} ) helps with precise alignment between the clean classification output distributions (logits) of the guide model and clean classification output distributions (logits) of the target model. We exploit the asymmetry of KL divergence to slightly and dynamically estimate the clean distribution gap between the guide and target models, which enhances model generalization and thereby improves robustness. We propose a learnable Collaborative Adversarial Generation (CAG) strategy to incorporate the adversarial example generation process into the overall adversarial training framework (see Fig. 1 (a) that shows adversarial sample generation process) instead of letting the adversarial example generation happen in silos, as done in previous works. CAG strategy is based on global performance involving both guide and target model distributions. Under such adversarial example generation, we additionally leverage the learnable guide model’s high confidence in its correct predictions on clean samples. As the training progresses, both the guide and target models collaboratively generate adversarial examples. This collaborative method aims to produce adaptive adversarial examples, thereby enhancing the robustness of the models. In summary, the main contribution of this work is as follows: • We propose a Dual Regularization Loss (D 2 R Loss) designed to progressively enhance the robustness of the target model. • We propose a collaborative adversarial generation (CAG) method that dynamically creates adversarial examples for subsequent adversarial training. • We show that our method (D 2 R Loss with CAG) outperforms other AT methods on an extensive set of experiments. 2 Related work Adversarial training is the most widely used defense method. For example, Wu et al. [ 16 ] proposed an effective adversarial training method called adversarial weight perturbation (AWP) to improve the DNNs’ robustness. Similarly, the AT method RAT [ 6 ] demonstrates model robustness by adding random noise to DNN weights and a misclassification-aware adversarial training (MART) [ 15 ] analyzes the misclassification sample to improve the robustness of DNN models. LAS-AT [ 5 ] uses a single strategy model to inject an attack into a target model rather than manually crafting the attack, as is commonly done in conventional AT. TRADES [ 18 ] optimizes the trade-off between accuracy and robustness, generating adversarial examples for training through the KL-divergence loss. Relying on a single model for adversarial generation plays an important role in enhancing the robustness of target DNNs. However, adversarial generation using a single model suffers in achieving optimal performance in various scenarios. Therefore, our work focuses on using a learnable guide model to adaptively collaborate with the target model for generating adversarial examples.

Read the complete text