Diffusion Models are Robust Pretrainers
Mika Yagoda, Shady Abu-Hussein, Raja Giryes · 2025
Read Diffusion Models are Robust Pretrainers by Mika Yagoda, Shady Abu-Hussein, Raja Giryes (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-02793.txt.
Introduction
Mika Yagoda, Shady Abu-Hussein, and Raja Giryes ask whether image generators can double as armor against adversarial attacks. Such attacks nudge an image just enough to fool a classifier. Training a model to resist them is costly. Their 2025 study instead builds classifiers and detectors on top of ready-made diffusion models, the kind used to generate images.
Only light heads are trained, on frozen features, with no full adversarial training. On ImageNet, CIFAR-10, and PASCAL VOC, the models gained real robustness at low compute cost. Their clean and attacked accuracy still trailed the best adversarially trained networks. Even so, the tradeoff suits places with tight budgets, and the work opens a path for using diffusion models in robust systems.
Catalog record
| CREATOR | Mika Yagoda, Shady Abu-Hussein, Raja Giryes |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 21 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-02793
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Diffusion Models are Robust Pretrainers Mika Yagoda, Shady Abu-Hussein, Raja Giryes Abstract Diffusion models have gained significant attention for high-fidelity image generation. Our work investigates the potential of exploiting diffusion models for adversarial robustness in image classification and object detection. Adversarial attacks challenge standard models in these tasks by perturbing inputs to force incorrect predictions. To address this issue, many approaches use training schemes for forcing the robustness of the models, which increase training costs. In this work, we study models built on top of off-the-shelf diffusion models and demonstrate their practical significance: they provide a low-cost path to robust representations, allowing lightweight heads to be trained on frozen features without full adversarial training. Our empirical evaluations on ImageNet, CIFAR-10, and PASCAL VOC show that diffusion-based classifiers and detectors achieve meaningful adversarial robustness with minimal compute. While clean and adversarial accuracies remain below state-of-the-art adversarially trained CNNs or ViTs, diffusion pretraining offers a favorable tradeoff between efficiency and robustness. This work opens a promising avenue for integrating diffusion models into resource-constrained robust deployments. Code is available at https://github.com/yagodamika/Diffusion-Models-are-Robust-Pretrainers I Introduction Diffusion models have recently achieved state-of-the-art performance in high-fidelity image generation [ 1 , 2 , 3 , 4 ] , outperforming prior generative models [ 5 ] with more stable training and scalability to higher resolutions and diverse datasets [ 6 ] . Diffusion models define a Markovian process that starts with clean images on the one end and pure noise of a known distribution on the other end. The forward process progressively adds noise to images, while the reverse process trains a neural network to denoise, generating images from noise to the clean domain. Adversarial robustness refers to the ability of neural networks to resist adversarial attacks. These attacks involve manipulating input data in ways that are imperceptible to humans but cause models to make err. For example, in image classification, an attacker perturbs the input image with small, carefully crafted noise that is indistinguishable to humans but leads the model to make an incorrect class prediction [ 7 , 8 , 9 ] . Multiple approaches have been proposed for robustifying deep neural networks against adversarial attacks. One direct approach integrates the adversarial examples within the training of the model, often referred to as ”adversarial training’” [ 10 ] . Other approaches suggest to use some sort of regularization [ 11 , 12 , 13 , 14 , 15 , 16 ] , or employ a specific network architecture [ 17 , 18 ] . Self-supervised learning (SSL) pretrains models on large unlabeled datasets [ 19 , 20 , 21 ] , producing rich representations for downstream tasks. It has been demonstrated [ 22 ] that adversarial training can be incorporated into the unsupervised training stage, resulting in significant performance improvements compared to conventional end-to-end adversarial training baselines [ 22 ] . Recently, pretrained diffusion models have also been successfully transferred to downstream tasks, surpassing other generative pretrainers [ 23 ] , but their robustness remains unexplored. [ 24 ] shows that diffusion classifiers exploiting conditional-unconditional score differences exhibit inherent robustness. Yet, they rely on labeled training and in this work we focus on unconditional diffusion models. We investigate whether diffusion models pretrained without labels provide robust features against adversarial attacks. We train lightweight classification heads on top of frozen features from off-the-shelf unconditional diffusion models and extend this method to object detection. Our results show that diffusion-based features offer robustness for both tasks. The main benefit of our approach is efficiency: it avoids costly adversarial training and requires only a lightweight head on frozen features. This makes it especially attractive for low-compute deployments or scenarios with limited labeled data. While diffusion features provide robustness “for free,” clean and adversarial accuracies remain below those of state-of-the-art adversarially trained CNNs or ViTs, positioning our method as a computationally efficient alternative that offers robustness without additional training overhead. Experiments are conducted on CIFAR-10 [ 25 ] and ImageNet [ 26 ] for classification, and PASCAL VOC for object detection. We study the effect of layer (block) and diffusion timestep choices on robustness, revealing that timestep choice has a greater impact on robustness than layer selection. Fig. 1: An overview of the used diffusion-based classification method. We examine the robustness of these models with respect to the U-Net block number and the diffusion noise time step. We use lightweight architectures for feature classification, including linear and attention-based heads. II Background II-A Adversarial Attacks We examine a deep classifier model f_{\gamma}:\mathbb{R}^{N}\rightarrow\mathbb{R}^{C} , where N , C , and \gamma denote the input image dimension, the number of classes, and the classifier parameters respectively. Adversarial examples are inputs intentionally crafted by an attacker to induce incorrect predictions by f_{\gamma} . These examples are generated by adding a small perturbation to the input image, such that is indistinguishable perceptually, yet leads to totally different class prediction. Formally, given an input x , its true label y , and a threat set \Delta=\{\delta:\|\delta\|_{n\in\{2,\infty\}}\leq\epsilon\} , an adversarial example \hat{x} is given by \hat{x}=x+\delta\quad\text{where }\delta\in\Delta\text{ and }f_{\gamma}(\hat{x})\neq y, The process of generating such examples is referred to as an adversarial attack and can be categorized into untargeted or targeted attacks. Untargeted attacks aim to generate \hat{x} leading to misclassifications without a specific target class. While targeted attacks aim to create \hat{x} inducing the classifier to predict \hat{x} as some traget class \hat{y}\neq y . There are various methods for creating adversarial examples. One known attack is the Projected Gradient Descent (PGD) attack [ 10 ] , outlined by the following iterative scheme \text{Repeat $n$ times: }\delta=\Pi_{\epsilon}\left(\delta+\alpha\nabla_{\delta}L(f_{\gamma}(x+\delta),\hat{y})\right), where \Pi_{\epsilon} denotes the projection operator onto \Delta , \alpha is the step size, n is the number of steps, and L(\cdot) denotes the cross-entropy classification loss. A popular approach to improve robustness to these attacks is using adversarial training. Yet, it is costly and increases the training time. Thus, we focus on methods that do not use it in the fine-tuning stage and show that our proposed approach can lead to good robustness even without it. II-B Diffusion Models Diffusion models define a forward noising process, which involves progressively adding Gaussian noise to an image x_{0} sampled from the data distribution q(x_{0}) . This results in a fully noised image x_{T} after T steps. The forward process is structured as a Markov chain with latent variables x_{1},x_{2},\ldots,x_{T-1},x_{T} , where each x_{t} denotes an image affected by an increasing noise level. Formally, the forward diffusion process can be expressed as: q(x_{1},\ldots,x_{T}|x_{0}):=\prod_{t=1}^{T}q(x_{t}|x_{t-1}), where q(x_{t}|x_{t-1}):=\mathcal{N}\left(x_{t};\sqrt{1-\beta_{t}}x_{t-1},\beta_{t}I\right). and \{\beta_{t}\}_{t=1}^{T} controls the noise variance scheduler. By defining \alpha_{t}:=1-\beta_{t} and \bar{\alpha}_{t}:=\prod_{i=0}^{t}\alpha_{i} , one can directly sample a noised image x_{t} at diffusion step t from the original image x_{0} according to the parameterization: x_{t}=\sqrt{\bar{\alpha}_{t}}x_{0}+\sqrt{1-\bar{\alpha}_{t}}\epsilon,\quad\epsilon\sim\mathcal{N}(0,I). (1) The reverse diffusion process seeks to invert the forward process and sample from the posterior distribution q(x_{t-1}|x_{t}) . It is performed by denoising x_{t} and then adding a noise perturbation according to the noise scheduler at step t-1 . By starting from x_{T} and progressively running the backward process until reaching x_{0} , one may obtain a clean image. This allows sampling from the original data distribution q(x_{0}) . The denoising step is approximated using a neural network with parameters \Theta