ProARD: Progressive Adversarial Robustness Distillation: Provide Wide Range of Robust Students

Seyedhamidreza Mousavi, Seyedali Mousavi, Masoud Daneshtalab · 2025

Read ProARD: Progressive Adversarial Robustness Distillation: Provide Wide Range of Robust Students by Seyedhamidreza Mousavi, Seyedali Mousavi, Masoud Daneshtalab (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-07666.txt.

Introduction

Seyedhamidreza Mousavi, Seyedali Mousavi, and Masoud Daneshtalab present a 2025 arXiv paper on hardening small neural networks against attack. Tiny networks on phones and sensors can be fooled by crafted inputs, so a large hardened teacher network often trains a small robust student. The snag is that each new device size needs a fresh student trained from scratch, which wastes time and energy.

Their method, ProARD, trains one flexible network whose width and depth can shrink or grow after training. The largest version acts as the teacher, and smaller students inside it learn at the same time through shared weights. Picking which students to train in each round turns out to matter: random picks fail to yield strong students. The approach promises one training run that serves many devices at once.

Catalog record

CREATORSeyedhamidreza Mousavi, Seyedali Mousavi, Masoud Daneshtalab
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-07666

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ProARD: Progressive Adversarial Robustness Distillation: Provide Wide Range of Robust Students Seyedhamidreza Mousavi, Seyedali Mousavi, Masoud Daneshtalab Abstract Adversarial Robustness Distillation (ARD) has emerged as an effective method to enhance the robustness of lightweight deep neural networks against adversarial attacks. Current ARD approaches have leveraged a large robust teacher network to train one robust lightweight student. However, due to the diverse range of edge devices and resource constraints, current approaches require training a new student network from scratch to meet specific constraints, leading to substantial computational costs and increased CO 2 emissions. This paper proposes Progressive Adversarial Robustness Distillation (ProARD), enabling the efficient one-time training of a dynamic network that supports a diverse range of accurate and robust student networks without requiring retraining. We first make a dynamic deep neural network based on dynamic layers by encompassing variations in width, depth, and expansion in each design stage to support a wide range of architectures ( > 10 19 >10^{19} ). Then, we consider the student network with the largest size as the dynamic teacher network. ProARD trains this dynamic network using a weight-sharing mechanism to jointly optimize the dynamic teacher network and its internal student networks. However, due to the high computational cost of calculating exact gradients for all the students within the dynamic network, a sampling mechanism is required to select a subset of students. We show that random student sampling in each iteration fails to produce accurate and robust students. ProARD employs a progressive sampling strategy that gradually reduces the size of student networks in three steps during training while applying robustness distillation between the dynamic teacher network and the selected students. Finally, we leverage a multi-objective evolutionary algorithm based on a proposed accuracy-robustness predictor to identify optimal architectures that balance accuracy, robustness, and efficiency. Through the experiments, we show that ProARD reduces the computational cost by 60 × 60\times and improves accuracy and robustness by 13 % 13\% and 14 % 14\% , respectively, compared to random sampling. We also demonstrate that our accuracy-robustness predictor can estimate the accuracy and robustness of test student networks with root mean squared errors of 0.0073 0.0073 and 0.0072 0.0072 , respectively. Keywords Adversarial Robustness, Robustness Distillation, Dynamic Network 1 Introduction Deep Neural Networks (DNNs) are highly effective in solving complex tasks, including image classification [ 39 ] , object detection [ 40 ] , and image segmentation [ 41 ] . Recently, DNNs have been widely used in security-critical applications, such as self-driving cars [ 42 ] , face recognition [ 43 ] , and medical diagnosis [ 71 ] , where robustness to input perturbations is the primary concern. Despite their strengths, DNNs are susceptible to subtle and imperceptible changes in input data, referred to as adversarial attacks [ 44 ] . These vulnerabilities pose significant challenges for security-critical applications. Different strategies have been suggested to counter adversarial attacks and enhance the robustness of DNNs. Among them, Adversarial Training (AT) is currently the most effective method for robustification [ 5 ] . AT methods aim to train an adversarially robust DNN such that its predictions are locally invariant to a small neighborhood of inputs. However, AT requires a well-designed, high-capacity network architecture, and its effectiveness is limited in lightweight networks due to their limited capacity [ 45 ] . To address this issue, the Adversarial Robustness Distillation (ARD) technique has been designed to transfer robustness from a high-capacity robust teacher network to a lightweight student network [ 80 ] . Current ARD methods are limited to training a single student network and primarily focus on finding the best training loss function based on the alignment between teacher and student networks [ 81 ] . Consequently, addressing the heterogeneity of edge devices and resource constraints requires retraining various network architectures tailored to each deployment scenario. Therefore, as the number of deployment scenarios ( N ) increases, the total cost of designing a lightweight network grows linearly (i.e. O(N) ). This approach poses significant challenges for practical implementation across varying hardware configurations, while also incurring substantial computational costs and contributing to increased CO 2 emissions. Given this limitation, we pose the following question: How can we take advantage of adversarial robustness distillation to provide a wide range of robust students efficiently without retraining them? This paper introduces a Progressive Adversarial Robustness Distillation (ProARD) training strategy to train a wide range of student network architectures suitable for different deployment scenarios. ProARD makes a dynamic deep neural network based on dynamic layers, including variations in width (kernel size), depth, and expansion in each design stage to support a wide range of architectures. To train this dynamic network, we consider the largest configuration of the dynamic network as the dynamic teacher and jointly train the dynamic teacher and students in a weight-sharing mechanism. However, due to the huge number of students in the dynamic network, it is computationally prohibitive to compute the exact gradient based on all students (more than 10^{19} students). We address this by sampling a subset of students by progressively reducing their size in each training iteration, and we show that this method works better than random sampling. After training the dynamic network by ProARD, we utilize a multi-objective evolutionary search algorithm alongside an accuracy-robustness predictor to identify the optimal student effectively. Our contributions can be summarized as follows: • We present a novel method called Progressive Adversarial Robustness Distillation (ProARD), designed to train a dynamic network including a wide range of students by progressive sampling. • We propose a joint accuracy-robustness predictor to efficiently estimate the robustness and accuracy, leveraging it in the multi-objective evolutionary algorithm to quickly find the best architecture based on the resource constraints without retraining. • We extensively evaluate the distribution of the robustness and accuracy of students and the effectiveness of our accuracy-robustness predictor and multi-objective search mechanism on the CIFAR-10 and CIFAR-100 datasets with the dynamic networks based on ResNet and MobileNet architectures. 2 Related Work 2.1 Adversarial Attacks & Training Adversarial attacks create subtle alterations in input data to intentionally mislead DNNs into making incorrect predictions. Common techniques for generating these attacks include the Fast Gradient Sign Method (FGSM) [ 2 ] , Projected Gradient Descent (PGD) [ 4 ] , and the Carlini & Wagner (CW) attack [ 13 ] , among others [ 18 ] . Furthermore, AutoAttack (AA) [ 14 ] , a set of four techniques combining white-box and black-box methods, has shown remarkable effectiveness in generating adversarial attacks. Adversarial training is the current state-of-the-art defense method against adversarial attacks, aiming to create robust DNNs. The earliest adversarial training approach leveraged both clean and adversarial images to train a robust DNN [ 2 ] . Adversarial training can be reformulated as a min-max optimization problem, where the network is trained exclusively on adversarial examples [ 4 ] . TRADES [ 5 ] regularizes the loss function for clean data by incorporating a robust loss term and making a trade-off between them. TRADES variants have also been proposed to consider regularization terms and reduce the distance between the distribution of natural data and their adversarial counterparts [ 6 , 15 ] . Although adversarial training methods work well for high-capacity

Read the complete text