Zero-Sacrifice Persistent-Robustness Adversarial Defense for Pre-Trained Encoders Zhuxin Lei, Ziyuan Yang, Yi Zhang Abstract The widespread use of publicly available pre-trained encoders from self-supervised learning (SSL) has exposed a critical vulnerability: their susceptibility to downstream-agnostic adversarial examples (DAEs), which are crafted without knowledge of the downstream tasks but capable of misleading downstream models. While several defense methods have been explored recently, they rely primarily on task-specific adversarial fine-tuning, which inevitably limits generalizability and causes catastrophic forgetting and deteriorates benign performance. Different with previous works, we propose a more rigorous defense goal that requires only a single tuning for diverse downstream tasks to defend against DAEs and preserve benign performance. To achieve this defense goal, we introduce Ze ro-Sacrifice P ersistent-Robustness A dversarial D efense ( ZePAD ), which is inspired by the inherent sensitivity of neural networks to data characteristics. Specifically, ZePAD is a dual-branch structure, which consists of a Multi-Pattern Adversarial Enhancement Branch (MPAE-Branch) that uses two adversarially fine-tuned encoders to strengthen adversarial resistance. The Benign Memory Preservation Branch (BMP-Branch) is trained on local data to ensure adversarial robustness does not compromise benign performance. Surprisingly, we find that ZePAD can directly detect DAEs by evaluating branch confidence, without introducing any adversarial exsample identification task in training. Notably, by enriching feature diversity, our method enables a single adversarial fine-tuning to defend against DAEs across downstream tasks, thereby achieving persistent robustness. Extensive experiments on 11 SSL methods and 6 datasets validate its effectiveness. In certain cases, it achieves a 29.20% improvement in benign performance and a 73.86% gain in adversarial robustness, highlighting its zero-sacrifice property. To facilitate reproducibility, the code is publicly available 1 1 1 https://github.com/Lawliet0o/ZePAD . 1 Introduction Training robust models is resource-intensive. Consequently, using public pre-trained models from self-supervised learning (SSL), such as CLIP ( 40 ) and GPT ( 4 ) , is a common paradigm for downstream tasks. These encoders learn powerful representations, fostering advancements in fields like classification ( 23 ) and segmentation ( 34 ) . However, these encoders are vulnerable to Downstream-Agnostic Adversarial Examples (DAEs) ( 33 ) , which are crafted without knowledge of the downstream tasks but capable of misleading downstream models. Even when the model has been task-specific tuned on local data, the attack remains effective. For example, PAP ( 3 ) and AdvEncoder ( 50 ) can generate effective DAEs without knowledge of the pre-training or downstream data. To address this security concern, researchers have proposed several secure fine-tuning methods against DAEs, but these methods lead to a significant drop in benign performance, making them impractical. For instance, Gen-AF ( 51 ) improves robust accuracy from 35.58% to 62.28% but reduces benign accuracy from 81.99% to 68.69%. Meanwhile, these methods require re-tuning for each task. This leads us to ask: “Is it possible to achieve a zero-sacrifice persistent-robustness defense that requires only a single tuning for various downstream tasks, without compromising the performance of benign samples?” We find that neural networks inherently exhibit higher confidence in inputs that resemble the training data, a behavior attributed to the memorization of the data’s characteristics 45 . Inspired by this tendency, we shift our focus from traditional adversarial fine-tuning to evaluating the confidence levels of different encoders. This insight motivates us to rethink the defense process, shifting the focus from finding a tradeoff between adversarial robustness and performance decline to achieving a zero-sacrifice approach. We propose Ze ro-Sacrifice L ifelong A dversarial D efense ( ZePAD ), which uses two branches to enhance robustness while maintaining or even improving benign performance: i) Multi-Pattern Adversarial Enhancement Branch (MPAE-Branch): To counter DAEs that exploit specific patterns, this branch adversarially fine-tunes encoders trained with diverse SSL methods. This diversity makes it difficult for attackers to find common vulnerabilities. ii) Benign Memory Preservation Branch (BMP-Branch): To mitigate the performance drop on clean data from adversarial fine-tuning, this branch is trained only on benign samples, preserving the model’s sensitivity to them. To effectively integrate the different branches and achieve zero-sacrifice, we propose a R obust F ederal D ecision M echanism ( RFDM ). Each branch’s confidence adapts to the alignment between the input and its training dataset distribution, and RFDM makes a decision based on this evaluation. Specifically, for benign samples, the BMP-Branch assigns higher confidence than the other branches, as the latter are adversarially fine-tuned and more biased towards adversarial exsamples. Since adversarial exsamples were not encountered during BMP-Branch training, they fall outside its distribution, resulting in lower confidence. Similarly, the MPAE-Branch, trained with adversarial examples, exhibits higher confidence when handling such samples. Existing defense methods typically require task-specific adversarial fine-tuning. In this paper, we show that introducing diversification through multiple heterogeneous encoders enhances robustness beyond single-encoder fine-tuning. As a result, ZePAD requires only a single fine-tuning process to provide strong, versatile representations across all downstream tasks, making it a persistent-robustness defense method. Surprisingly, the aforementioned inherent characteristics of neural networks allow ZePAD to detect adversarial exsamples based solely on confidence scores, without requiring specific training for this purpose. Our main contributions are: • We propose a zero-sacrifice, persistent-robustness adversarial defense method that not only maintains but also improves benign performance, while enhancing adversarial robustness. Besides, our method only requires a single-time adversarial tuning to achieve robust performance across various downstream tasks. • Although adversarial exsample detection is not the focus of our work, our method can directly detect adversarial exsamples, without requiring any specific training for this purpose. • Extensive experiments across 11 SSL methods and 6 datasets validate ZePAD’s effectiveness, showing improvements as high as 29.20% in benign performance and 73.05% in robust performance in some scenarios. 2 Related Works Self-Supervised Learning. SSL pre-trains a generic encoder on vast unlabeled data for various downstream tasks. This paradigm overcomes labeled data limitations, allowing users to adapt powerful pre-trained encoders through fine-tuning without training from scratch. Common adaptation methods include linear evaluation ( 1 ; 27 ) , which adjusts only the last layer, and full fine-tuning ( 21 ; 36 ) , which tunes all layers. Adversarial Attacks. Deep neural networks (DNNs) are vulnerable to adversarial examples ( 18 ; 22 ) , including universal adversarial perturbations (UAPs) 12 where a single perturbation affects multiple samples. Attacks are categorized as white-box ( 38 ) , where the attacker has full model access, or black-box ( 31 ) , with access only to inputs and outputs. Recent research ( 3 ; 50 ; 47 ) explores attacks on pre-trained encoders, which are downstream-agnostic and thus pose a significant security risk to the pre-trained paradigm. Adversarial Defenses. Adversarial defense methods include input filtering ( 20 ) and enhancing model robustness, often through adversarial training ( 52 ; 51 ; 30 ; 32 ) . However, it typically harms generalization, reducing performance on clean samples. While recent work ( 28 ; 51 ) aims to balance robustness and generalization, no existing method enhances robustness without compromising—or while even improving—generalization ability. The detailed related works are discussed in Appendix A . 3 Problem Formulation and Threat Modeling 3.1 Problem Statement Let \mathcal{E}_{\theta} be a pre-trained encoder, parameterized by \theta . We define \mathcal{D}_{p} as the dataset used to pre-train \mathcal{E}_{\theta} , where the benign input is represented as (x,y)\in\mathcal{D}_{p} , with x as the input and y as the corresponding label. The output feature of \mathcal{E}_{\theta} is denoted as v=\mathcal{E}_{\theta}(x) , where v\in V and V represents the feature space. The downstream model is denoted as \mathcal{F}_{\mathcal{\phi}} , where \mathcal{\phi} represents its parameters. The input to \mathcal{F}_{\mathcal{\phi}} is the output of \mathcal{E}_{\theta} , and the process can be formulated as \mathcal{F}_{\mathcal{\phi}}(v) . In adversarial attacks, an attacker aims to exploit vulnerabilities in \mathcal{E}_{\theta} by adding small perturbations \delta to the input x . The perturbation \delta is constrained by an upper bound \epsilon based on the l_{p} -norm, which ensures the noise remains imperceptible. The adversarial attack can be described as: \mathcal{E}_{\theta}(x)\neq\mathcal{E}_{\theta}(x+\delta),\;\text{subject to}\;\|\delta\|_{p}\leq\epsilon. (1) The ultimate goal of the attacker is to deceive the downstream model \mathcal{F}_{\mathcal{\phi}} . Specifically, for a classification task, the attacker aims to manipulate \mathcal{F}_{\mathcal{\phi}} to output different predictions for benign and adversarial examples. This can be formalized as follows: \mathcal{F}_{\mathcal{\phi}}(\mathcal{E}_{\theta}(x))\neq\mathcal{F}_{\mathcal{\phi}}(\mathcal{E}_{\theta}(x+\delta)),\;\text{subject to}\;\|\delta\|_{p}\leq\epsilon. (2) As defenders, our goal is to create a defense mechanism that prevents the adversarial attack from succeeding while preserving or improving benign performance. Specifically, we aim to ensure that the predictions for both the original input x and the adversarial exsample x+\delta remain the same. 3.2 Threat Model This paper investigates the vulnerability of pre-trained encoders in downstream classification tasks. We assume attackers exploit publicly available encoders, whcih can be obtained from the public platforms, to craft adversarial examples that transfer to any downstream model utilizing the encoder. These attacks are non-targeted, universal across downstream tasks, effective even after fine-tuning, and stealthy, requiring perturbations to remain visually imperceptible. For defenders, they cannot control the pre-training stage and are unaware of the original training data or SSL method used in pre-training. However, they have full access to the released encoder and complete control over downstream fine-tuning, including architectural adaptation. We propose a stronger defender’s objective: improving downstream robustness without sacrificing benign performance. Our defender’s goals can be summarized as: 1) Zero-Sacrifice Accuracy: Maintain (or even improve) benign performance without degrading clean accuracy; 2) Adversarial Robustness: Resist adversarial examples generated from pre-trained encoders and maintain correct predictions; 3) Resource Efficiency: Avoid reliance on external data or additional computational costs beyond standard fine-tuning; 4) Persistent Robustness: Enable a single adversarially-tuned encoder to remain robust across diverse downstream tasks without task-specific retraining. 3.3 Challenges Unlike previous works, we aim to enhance adversarial robustness without sacrificing benign performance, and even expect to improve it. Furthermore, we aim to establish a persistent-robustness paradigm that enables a one-time tuning process while ensuring robustness across all downstream tasks. However, just re-tuning the pre-trained encoder may compromise its powerful feature extraction capabilities. Hence, achieveing our goal is challenging, which can be outlined as: (1) Challenge I: Zero-Sacrifice Conflict: Existing adversarial fine-tuning methods inevitably degrade benign accuracy due to the inherent conflict between robustness optimization and clean performance. Our work targets the more challenging zero-sacrifice objective that achieving robustness improvement without harming benign accuracy, despite limited access to the pre-training process. (2) Challenge II: Persistent Robustness Across Tasks: Most defenses require task-specific tuning and cannot generalize across downstream tasks. We introduce persistent robustness, aiming for “tune once, defend all” capability, enabling a single tuned encoder to remain robust across diverse downstream tasks without re-training or task-specific assumptions. The detailed threat model and challenges are provided in Appendix B . 4 Methodology 4.1 ZePAD: A Complete Illustration In this section, we introduce ZePAD, the first zero-sacrifice, persistent-robustness defense method specifically designed for pre-trained encoders. The pipeline of the proposed method is illustrated in Figure 1 , and it comprises two primary steps: encoder preparation and RFDM. These steps are elaborated upon in the following subsections. Figure 1: The overview of the proposed method. The red and green arrows denote the attack flow and normal flow in the inference stage, respectively. 4.1.1 Step 1: Encoder Preparation The core motivation behind ZePAD in achieving persistent-robustness zero-sacrifice defense against DAEs is to leverage the inherent confidence bias within neural networks. Therefore, the first step in ZePAD is encoder preparation. Since the defender can modify the local encoder, the goal of this step is to build a feature extractor with both strong representation capabilities and robust adversarial resilience. Specifically, our encoder consists of the MPAE-Branch and the BMP-Branch, which are designed to enhance adversarial robustness and improve benign performance, respectively. MPAE-Branch: DAEs exploit the model’s sensitivity to specific patterns and textures, but it is hard to learn common sensitive patterns across multiple encoders, trained with different SSL methods. The diversity in feature representations learned by different encoders makes it difficult for DAEs to pinpoint shared vulnerabilities. This significantly complicates the task of exploiting common weaknesses across all encoders. Hence, our MPAE-Branch consists of two public pre-trained encoders. To ensure adversarial robustness, both encoders must be fine-tuned to strengthen their resilience to adversarial attacks. However, pre-trained encoders are highly sensitive to fine-tuning, which presents a dilemma: how to improve adversarial robustness while preserving the original pre-trained knowledge. To address this challenge, we propose a hybrid loss that strikes a balance between preserving the pre-trained encoders’ knowledge and enhancing the adversarial robustness. The loss function is formulated as follows: \mathcal{L}=\mathcal{L}_{c}+\lambda\mathcal{L}_{f}, (3) where \mathcal{L}_{c} represents the classification loss of adversarial exsamples, and \mathcal{L}_{f} represents the encoding feature loss of adversarial exsamples. \lambda is a hyperparameter. Then, \mathcal{L}_{c} is calculated as follows: \mathcal{L}_{c}(x,y)=\mathcal{L}_{CE}\left(\mathcal{F}_{\mathcal{\phi}}\left(\mathcal{E}_{\mathcal{\theta}}\left(x+\delta\right)\right),y\right), (4) where \mathcal{E} denotes the encoder, and \mathcal{F} represents the downstream classification model. x and y stand for the data and labels, respectively. To better describe the loss caused by encoding features, we first define the distance between two encoding features. We use cosine distance to measure the feature differences between them. Additionally, we remove the nearest features to prevent local high-density data from overly influencing the loss value and to better describe the global structure. This process is formulated as follows: D_{ij}=\frac{2-\left(\cos(x_{i},x_{j})-\rho_{x_{j}}\right)}{\sum_{k=1,k\neq j}^{N}\left(2-\left(\cos(x_{i},x_{k}))-\rho_{x_{k}}\right)\right)}, (5) where \text{cos}(x_{i},x_{j}) denotes the cosine distance between x_{i} and x_{j} , and \rho_{x_{j}} denotes the minimum distance between x_{j} and all samples in the training dataset. Then, the loss can be formulated as: \mathcal{L}_{f}=\sum_{i}\sum_{j}\left[D_{ij}^{b}\log\left(\frac{{D}_{ij}^{b}}{D_{ij}^{a}}\right)+\left(1-D_{ij}^{b}\right)\log\left(\frac{1-D_{ij}^{b}}{1-{D}_{ij}^{a}}\right)\right], (6) where D_{ij}^{a} and D_{ij}^{b} represent the distances between the adversarial exsample pair (x_{i},x_{j}) and their corresponding benign sample pair, respectively. This makes the feature distribution of adversarial exsamples after encoding similar to that of benign samples. BMP-Branch: To address this challenge, we introduce a benign branch. In this step, we train an additional encoder locally using only benign data. By focusing on local data, the benign branch can memorize and capture crucial features that are essential for maintaining strong performance on clean, non-adversarial inputs. Overall, our encoder \mathcal{E} consists of three sub-encoders: \mathcal{E}_{a,1} , \mathcal{E}_{a,2} , and \mathcal{E}_{b} . Here, \mathcal{E}_{a,1} and \mathcal{E}_{a,2} represent the publicly shared and adversarially tuned pre-trained encoders, while \mathcal{E}_{b} refers to the locally trained benign encoder. As assumed, the downstream task is a classification problem. The classification heads can be trained based on these sub-encoders using the local data. This process allows us to obtain classification heads \mathcal{H}_{a,1} , \mathcal{H}_{a,2} , and \mathcal{H}_{b} , which are responsible for mapping the features from \mathcal{E}_{a,1} , \mathcal{E}_{a,2} , and \mathcal{E}_{b} , respectively. A key challenge now is how to effectively combine these sub-encoders to achieve the zero-sacrifice goal. 4.1.2 Step 2: Robust Federal Decision Mechanism Our method builds on the observation that neural networks implicitly estimate posterior confidence. Consequently, they assign higher confidence to samples aligned with their training distribution and lower confidence to out-of-distribution or adversarial inputs. This inherent behavior leads to natural confidence separation between branches for different kinds of data. To support our observation, the theoretical proof can be found in Appendix C . To fully leverage this observation, we utilize two branches trained for different sample types: the MPAE-Branch for adversarial exsamples and the BMP-Branch for benign ones. This design leverages the principle that neural networks show higher confidence on inputs consistent with their training data ( 5 ) . The MPAE-Branch is adversarially fine-tuned for robustness, while the BMP-Branch, trained solely on benign samples, is highly sensitive to clean data. Our experiments also support our assumption, as illustrated in Figure 2 . On benign samples, the BMP-Branch consistently exhibits higher confidence. Conversely, the MPAE-Branch’s adversarial training, while enhancing robustness, slightly compromises its generalization on clean data, resulting in lower confidence. Figure 2: The confidence distributions of each branch under different conditions. C-C indicates fine-tuning on CIFAR10 and downstream testing on CIFAR10; C-S indicates fine-tuning on CIFAR10 and downstream testing on STL10. On adversarial exsamples, the roles reverse. The BMP-Branch’s confidence drops significantly as it is vulnerable to the versatile attacks. In contrast, the MPAE-Branch maintains the highest confidence, as its adversarial fine-tuning makes it resilient to these specific inputs. This observed phenomenon, where the branches exhibit complementary strengths, is the foundation of our proposed approach. We leverage this insight to combine their outputs, aiming to maximize both adversarial robustness and performance on benign data. The decision process is formulated as follows: \mathcal{W}_{i}=\frac{c_{i}}{\sum_{k=1}^{3}c_{k}}, (7) where \mathcal{W}_{i} represents the confidence score of the i -th branch, while c_{i} represents the confidence value of the i -th branch. Specifically, the calculation method for c_{i} is as follows: c_{i}=m_{i}e^{3(m_{i}-0.5)}, (8) where m_{i} represents the maximum value in the probability vector output of i -th branch. However, we argue that the confidence value does not change linearly with the maximum probability. Instead, we posit the maximum probability value is positively correlated with the confidence value. Besides, as the maximum probability value approaches 1, the rate at which the confidence value increases should accelerate. To help readers follow our work, the algorithm can be found in Appendix D . Table 1: BA (%) comparison between the baseline method and ZePAD in the semi-black-box setting, where \mathcal{D}_{p} , \mathcal{D}_{f} and \mathcal{D}_{d} denote the pre-training, adversarialt adversarial training and downstream datasets, respectively. \mathcal{D}_{p} \mathcal{D}_{f} and \mathcal{D}_{d} Method W-MSE BYOL NNCLR SimCLR MoCo2+ MoCo3 SupCon RESSL DINO SwAV VibCreg AVG baseline 90.17 94.39 93.77 93.56 95.03 94.71 96.22 92.73 91.43 92.29 92.39 93.34 ours 93.42 93.49 93.96 95.05 93.79 94.27 92.39 92.87 93.36 93.51 93.12 93.57 CIFAR10 \Delta +3.25 -0.90 +0.19 +1.49 -1.24 -0.44 -3.83 +0.14 +1.93 +1.22 +0.73 +0.23 baseline 78.49 83.08 83.09 81.94 84.46 84.00 85.29 81.87 83.54 81.53 81.64 82.63 ours 81.43 82.30 82.52 84.38 81.52 83.51 82.90 82.10 81.42 81.87 83.65 82.51 STL10 \Delta +2.94 -0.78 -0.57 +2.44 -2.94 -0.49 -2.39 +0.23 -2.12 +0.34 +2.01 -0.12 baseline 80.39 90.22 91.22 90.82 88.03 88.05 88.23 87.86 85.01 89.16 94.70 88.52 ours 97.35 98.29 97.84 97.92 98.56 98.41 97.79 98.67 97.76 98.07 98.86 98.14 ANIMALS10 \Delta +16.96 +8.07 +6.62 +7.10 +10.53 +10.36 +9.56 +10.81 +12.75 +8.91 +4.16 +9.62 baseline 73.76 75.99 79.06 78.61 76.37 81.11 81.65 76.03 80.63 85.51 82.25 79.18 ours 95.34 93.58 94.20 94.99 94.78 94.38 94.23 94.12 94.41 95.54 94.09 94.51 GTSRB \Delta +21.58 +17.59 +15.14 +16.38 +18.41 +13.27 +12.58 +18.09 +13.78 +10.03 +11.84 +15.34 baseline 59.01 67.47 65.94 57.74 66.22 65.00 58.87 61.95 66.45 63.01 62.31 63.09 ours 73.93 72.88 73.09 74.08 74.45 74.45 72.54 72.39 73.35 72.94 73.41 73.41 ImageNet20 \Delta +14.92 +5.41 +7.15 +16.34 +8.23 +9.45 +13.67 +10.44 +6.90 +9.93 +11.10 +10.32 baseline 57.92 58.59 66.43 64.81 64.67 64.32 69.88 67.65 70.38 72.57 73.26 66.41 ours 94.08 94.70 94.84 94.92 95.39 94.75 94.22 94.80 94.40 94.52 93.91 94.59 CIFAR10 SVHN \Delta +36.16 +36.11 +28.41 +30.11 +30.72 +30.43 +24.34 +27.15 +24.02 +21.95 +20.65 +28.19 baseline 56.52 68.60 72.38 70.56 70.35 72.37 71.35 71.20 71.75 68.46 73.35 69.72 ours 93.03 93.05 93.39 93.07 93.46 93.88 93.44 92.43 93.05 93.02 93.12 93.18 CIFAR10 \Delta +36.51 +24.45 +21.01 +22.51 +23.11 +21.51 +22.09 +21.23 +21.30 +24.56 +19.77 +23.46 baseline 51.99 63.37 65.41 66.14 64.78 63.21 64.79 64.89 64.37 62.90 68.76 63.69 ours 80.08 76.82 81.14 82.07 81.60 82.94 79.35 80.22 80.77 80.58 81.35 80.63 STL10 \Delta +28.09 +13.45 +15.73 +15.93 +16.82 +19.73 +14.56 +15.33 +16.40 +17.68 +12.59 +16.94 baseline 48.53 63.82 68.81 70.69 65.86 68.31 74.65 64.75 62.19 61.75 76.57 65.99 ours 97.28 97.70 98.30 97.86 97.70 98.35 98.55 98.05 98.18 97.73 98.61 98.03 ANIMALS10 \Delta +48.75 +33.88 +29.49 +27.17 +31.84 +30.04 +23.90 +33.30 +35.99 +35.98 +22.04 +32.03 baseline 52.17 67.65 72.63 69.36 67.21 74.88 78.05 70.09 68.94 72.17 77.96 70.10 ours 94.56 92.84 94.34 94.98 94.12 94.33 94.87 93.79 94.55 94.76 94.44 94.33 GTSRB \Delta +42.39 +25.19 +21.71 +25.62 +26.91 +19.45 +16.82 +23.70 +25.61 +22.59 +16.48 +24.22 baseline 35.64 48.76 56.18 56.44 52.35 54.36 47.40 51.49 51.13 47.84 54.06 50.51 ours 72.80 72.75 72.99 73.97 75.34 75.69 71.20 73.16 72.33 71.91 69.81 72.90 ImageNet20 \Delta +37.16 +23.99 +16.81 +17.53 +22.99 +21.33 +23.80 +21.67 +21.20 +24.07 +15.75 +22.39 baseline 50.45 62.86 68.40 69.25 62.50 64.31 67.40 63.95 71.31 66.72 69.51 65.15 ours 94.16 93.60 94.05 94.92 95.20 94.75 94.36 94.57 94.21 94.03 94.03 94.35 ImageNet SVHN \Delta +43.71 +30.74 +25.65 +25.67 +32.70 +30.44 +26.96 +30.62 +22.90 +27.31 +24.52 +29.20 5 Experiments 5.1 Experimental Setting Attacks. We evaluate the effectiveness of our work using the following five general adversarial attacks: AdvEncoder ( 50 ) , PAP ( 3 ) , UAP ( 37 ) , UAPGD ( 14 ) and SSP ( 38 ) . Defense. We evaluate the effectiveness of our work using the following four adversarial training mechanisms: Gen-AF ( 51 ) , PGD-AT ( 35 ) , TRADES ( 46 ) and MART ( 44 ) . Datasets and models. We evaluate our method on the six datasets: CIFAR10 ( 26 ) , STL10 ( 13 ) , ANIMALS10 ( 2 ) , GTSRB ( 42 ) , ImageNet20 ( 41 ) , and SVHN ( 39 ) . We use the publicly available pre-trained encoders from solo-learn 2 2 2 https://github.com/vturrisi/solo-learn , a well-established SSL library, as victim encoders. Following ( 50 ; 51 ) , we select eleven SSL methods (W-MSE ( 16 ) , BYOL ( 19 ) , NNCLR ( 15 ) , SimCLR ( 8 ) , MoCo v2+ ( 9 ) , MoCo v3 ( 11 ) , SupCon ( 24 ) , RESSL ( 49 ) , DINO ( 7 ) , SwAV ( 6 ) , and VibCreg ( 29 ) ). All encoders are pre-trained on either CIFAR10 ( 26 ) or ImageNet ( 41 ) , using ResNet18 as the backbone. Table 2: RA (%) comparison between the baseline method and ZePAD in the semi-black-box setting, where D_{p} , D_{f} and D_{d} denote thepre-training, adversarialt adversarial training and downstream datasets, respectively D_{p} D_{f} and D_{d} Method W-MSE BYOL NNCLR SimCLR MoCo2+ MoCo3 SupCon RESSL DINO SwAV VibCreg AVG baseline 10.09 12.47 10.71 47.13 45.01 36.86 16.00 12.05 10.46 31.67 14.07 22.41 ours 86.84 80.18 79.84 74.70 69.70 72.49 77.99 84.58 80.44 74.61 79.62 78.27 CIFAR10 \Delta +76.75 +67.71 +69.13 +27.57 +24.69 +35.63 +61.99 +72.53 +69.98 +42.94 +65.55 +55.86 baseline 41.16 46.85 44.05 67.52 65.55 64.28 56.89 51.69 31.77 59.55 43.07 52.03 ours 70.28 71.30 72.66 77.34 74.01 74.89 75.17 73.74 71.73 72.12 71.18 73.13 STL10 \Delta +29.12 +24.45 +28.61 +9.82 +8.46 +10.61 +18.28 +22.05 +39.96 +12.57 +28.11 +21.09 baseline 48.31 49.25 49.23 63.48 63.03 60.32 58.48 54.92 35.46 63.75 50.19 54.22 ours 91.94 90.49 86.71 87.94 88.34 84.58 88.52 89.02 89.01 87.88 88.65 88.46 ANIMALS10 \Delta +43.63 +41.24 +37.48 +24.46 +25.31 +24.26 +30.04 +34.10 +53.55 +24.13 +38.46 +34.24 baseline 10.85 12.41 11.29 24.63 27.02 27.60 13.22 12.37 10.62 27.82 11.56 17.22 ours 75.00 79.23 73.95 74.59 74.15 73.54 72.15 72.47 80.89 76.04 72.18 74.93 GTSRB \Delta +64.15 +66.82 +62.66 +49.96 +47.13 +45.94 +58.93 +60.10 +70.27 +48.22 +60.62 +57.71 baseline 27.60 34.41 29.82 38.24 42.87 41.60 28.97 35.10 21.24 37.69 27.13 33.15 ours 60.07 64.84 64.54 61.69 62.60 62.68 63.74 63.55 65.17 62.46 55.99 62.48 ImageNet20 \Delta +32.47 +30.43 +34.72 +23.45 +19.73 +21.08 +34.77 +28.45 +43.93 +24.77 +28.86 +29.33 baseline 9.16 11.68 14.99 14.58 21.51 16.38 11.10 9.27 8.95 16.08 10.55 13.11 ours 75.65 89.17 88.52 91.26 86.68 90.75 90.53 86.67 91.39 86.67 79.43 86.97 CIFAR10 SVHN \Delta +66.49 +77.49 +73.53 +76.68 +65.17 +74.37 +79.43 +77.40 +82.44 +70.59 +68.88 +73.86 baseline 11.78 11.78 12.80 24.31 16.49 12.66 14.00 11.33 15.80 12.44 20.42 14.89 ours 74.00 66.69 66.03 78.07 68.01 69.37 71.21 81.77 78.72 86.01 67.22 73.37 CIFAR10 \Delta +62.22 +54.91 +53.23 +53.76 +51.52 +56.71 +57.21 +70.44 +62.92 +73.57 +46.80 +58.48 baseline 26.76 33.02 22.42 46.23 40.43 42.37 25.12 41.98 33.68 42.77 40.84 35.97 ours 70.35 67.80 64.67 73.26 74.30 76.78 67.00 71.90 70.18 75.18 69.36 70.98 STL10 \Delta +43.59 +34.78 +42.25 +27.03 +33.87 +34.41 +41.88 +29.92 +36.50 +32.41 +28.52 +35.01 baseline 32.68 43.20 37.47 53.24 47.71 46.21 37.43 44.26 40.70 41.23 48.36 42.95 ours 87.54 84.00 83.23 88.63 86.55 87.31 91.28 83.52 89.49 91.70 84.94 87.11 ANIMALS10 \Delta +54.86 +40.80 +45.76 +35.39 +38.84 +41.10 +53.85 +39.26 +48.79 +50.47 +36.58 +44.15 baseline 8.78 15.17 10.23 19.23 15.63 14.36 10.99 12.75 10.50 21.49 17.32 14.22 ours 69.38 66.69 60.51 69.33 73.65 76.34 69.11 69.03 67.39 81.41 68.12 70.09 GTSRB \Delta +60.60 +51.52 +50.28 +50.10 +58.02 +61.98 +58.12 +56.28 +56.89 +59.92 +50.80 +55.86 baseline 19.85 25.37 12.65 30.69 29.44 28.34 15.81 28.47 21.95 21.74 24.44 23.52 ours 63.06 59.42 60.26 61.07 61.78 64.78 62.46 60.20 65.00 67.42 58.18 62.15 ImageNet20 \Delta +43.21 +34.05 +47.61 +30.38 +32.34 +36.44 +46.65 +31.73 +43.05 +45.68 +33.74 +38.63 baseline 6.44 6.26 6.72 26.38 12.97 11.88 9.04 6.55 19.58 15.44 19.42 12.79 ours 87.88 85.36 66.21 90.36 84.44 89.36 92.81 84.44 82.31 89.49 91.53 85.84 ImageNet SVHN \Delta +81.44 +79.10 +59.49 +63.98 +71.47 +77.48 +83.77 +77.89 +62.73 +74.05 +72.11 +73.05 Evaluation Metrics . To evaluate the effectiveness of our proposed method, we examine the model’s robustness and generalization capability using the following three metrics. Benign Test Accuracy: Benign Test Accuracy (BA) denotes the classification accuracy on clean samples. Higher values indicate better generalization ability. Robust Test Accuracy: Robust Test Accuracy (RA) measures the classification accuracy on adversarial examples. Higher values signify greater robustness. Attack Success Rate: Attack Success Rate (ASR) is a metric from the attacker’s perspective, which provides insights into model robustness. It represents the proportion of adversarial examples where the model’s prediction differs from that of the corresponding clean sample. Higher values reflect stronger attack performance. Implementation details. Adversarial tuning is performed for 20 epochs with a batch size of 256, optimized by the Adam optimizer ( 25 ) . The classification head is composed of three fully connected layers. Following the previous work ( 50 ) , we select CIFAR10 as the dataset for the attacker and set the perturbation limit to 10/255 . The other experimental settings can be found in Appendix E . The default experimental setting follows a semi-black-box scenario. When the victim encoder in the MPAE-Branch is not BYOL, we select BYOL as the adversarial auxiliary encoder for the MPAE-Branch. Conversely, if the victim encoder is BYOL, we choose NNCLR as the adversarial auxiliary encoder (AdvAu-Model). The encoder in the BMP-Branch is trained using SimCLR with local datasets. Besides, we also validate our method under the white-box scenario, and the experimental results can be found in Appendix F . 5.2 Semi-Black-Box Scenario In this scenario, the attacker knows the SSL method employed by the victim model, as well as the pre-training dataset. We evaluate the performance of ZePAD across 11 SSL methods and 6 datasets, selecting AdvEncoder as the attack method due to its status as a SOTA downstream-agnostic adversarial examples method. This method enables effective attacks without knowledge of the pre-trained dataset or downstream tasks. Table 3: BA(%) comparison between the baseline method and ZePAD in the semi-black-box setting, where \mathcal{D}_{f} and \mathcal{D}_{d} denote the adversarial training and downstream datasets, respectively. \mathcal{D}_{f} \mathcal{D}_{d} Method W-MSE BYOL NNCLR SimCLR MoCo2+ MoCo3 SupCon RESSL DINO SwAV VibCreg AVG baseline 78.49 83.08 83.09 81.94 84.46 84.00 85.29 81.87 83.54 81.53 81.64 82.63 ours 84.84 84.88 85.91 86.52 85.34 85.21 82.93 84.23 84.88 84.56 84.11 84.86 STL10 \Delta +6.35 +1.80 +2.82 +4.58 +0.88 +1.21 -2.36 +2.36 +1.34 +3.03 +2.47 +2.23 baseline 80.39 90.22 91.22 90.82 88.03 88.05 88.23 87.86 85.01 89.16 94.70 88.52 ours 91.31 92.37 93.18 92.43 92.01 92.64 91.52 91.43 91.66 92.00 92.92 92.13 ANIMALS10 \Delta +10.92 +2.15 +1.96 +1.61 +3.98 +4.59 +3.29 +3.57 +6.65 +2.84 -1.78 +3.62 baseline 73.76 75.99 79.06 78.61 76.37 81.11 81.65 76.03 80.63 85.51 82.25 79.18 ours 82.50 83.17 83.86 84.36 83.10 84.49 82.13 82.71 82.25 84.33 83.41 83.30 GTSRB \Delta +8.74 +7.18 +4.80 +5.75 +6.73 +3.38 +0.48 +6.68 +1.62 -1.18 +1.16 +4.12 baseline 59.01 67.47 65.94 57.74 66.22 65.00 58.87 61.95 66.45 63.01 62.31 63.09 ours 64.89 66.23 66.86 66.10 66.21 66.49 62.56 64.47 65.42 66.63 62.75 65.33 ImageNet20 \Delta +5.88 -1.24 +0.92 +8.36 -0.01 +1.49 +3.69 +2.52 -1.03 +3.62 +0.44 +2.24 baseline 57.92 58.59 66.43 64.81 64.67 64.32 69.88 67.65 70.38 72.57 73.26 66.41 ours 75.12 78.62 80.31 79.43 73.14 77.73 75.39 77.55 77.87 75.88 77.32 77.12 CIFAR10 SVHN \Delta +17.20 +20.03 +13.88 +14.62 +8.47 +13.41 +5.51 +9.90 +7.49 +3.31 +4.06 +10.72 baseline 56.52 68.60 72.38 70.56 70.35 72.37 71.35 71.20 71.75 68.46 73.35 69.72 ours 91.20 91.07 90.92 94.40 91.54 91.24 91.11 90.91 91.06 91.53 90.65 91.42 CIFAR10 \Delta +34.68 +22.47 +18.54 +23.84 +21.19 +18.87 +19.76 +19.71 +19.31 +23.07 +17.30 +21.70 baseline 51.99 63.37 65.41 66.14 64.78 63.21 64.79 64.89 64.37 62.90 68.76 63.69 ours 81.19 80.38 80.64 82.98 80.80 81.19 80.24 80.68 80.79 80.91 79.05 80.80 STL10 \Delta +29.20 +17.01 +15.23 +16.84 +16.02 +17.98 +15.45 +15.79 +16.42 +18.01 +10.29 +17.11 baseline 48.53 63.82 68.81 70.69 65.86 68.31 74.65 64.75 62.19 61.75 76.57 65.99 ours 89.75 90.07 90.29 89.05 89.66 90.69 89.79 90.11 89.10 89.80 89.63 89.81 ANIMALS10 \Delta +41.22 +26.25 +21.48 +18.36 +23.80 +22.38 +15.14 +25.36 +26.91 +28.05 +13.06 +23.82 baseline 35.64 48.76 56.18 56.44 52.35 54.36 47.40 51.49 51.13 47.84 54.06 50.51 ours 61.61 60.66 60.09 59.96 59.52 60.71 60.32 59.87 60.00 61.17 57.41 60.12 ImageNet20 \Delta +25.97 +11.90 +3.91 +3.52 +7.17 +6.35 +12.92 +8.38 +8.87 +13.33 +3.35 +9.61 baseline 50.45 62.86 68.40 69.25 62.50 64.31 67.40 63.95 71.31 66.72 69.51 65.15 ours 81.67 80.36 82.15 82.42 80.87 82.96 79.53 81.19 82.87 81.80 82.29 81.65 GTSRB SVHN \Delta +31.22 +17.50 +13.75 +13.17 +18.37 +18.65 +12.13 +17.24 +11.56 +15.08 +12.78 +16.50 Figure 3: t-SNE visualization of feature spaces for the baseline and ZePAD under benign and adversarial examples. Different colors indicate different classes. We present comparisons between ZePAD and the baseline in Tables 1 and 2 , where the adversarial training dataset and downstream task dataset are identical. The results show that ZePAD significantly enhances task performance on benign samples, with average accuracy improvements of 29.20% and 32.03% on the SVHN and ANIMALS10 datasets, respectively, when the pre-training dataset is ImageNet. In contrast, the baseline’s accuracy drops significantly on adversarial exsamples, with RA falling below 10% in some experimental settings, underscoring the threat posed by AdvEncoder-generated DAEs to pre-trained encoders. As shown in Table 4 , ZePAD significantly improves RA. For instance, when the pre-training dataset is CIFAR10 and the downstream task is SVHN, RA increases by 73.86% compared to the baseline, reaching an average RA of 86.97%. This demonstrates that ZePAD can effectively enhance model robustness while improving generalization, achieving Zero-Sacrifice Adversarial Defense . To investigate ZePAD’s cross-task defense capabilities, we design experiments where the adversarial training dataset differs from the downstream task dataset, with CIFAR10 uniformly selected as the pre-training dataset. The experimental results are provided in Tables 3 and 4 . The results demonstrate that ZePAD can still enhance BA compared to baseline, even when the adversarial training dataset and downstream dataset are mismatched. For example, when the fine-tuning dataset is GTSRB, BA improves by an average of 21.70% and 23.82% on CIFAR10 and ANIMAL10 downstream tasks, respectively. Against adversarial examples, ZePAD continues to perform well, significantly boosting RA. When fine-tuning on CIFAR10, RA averages 73.19% on STL10 and 73.94% on ANIMALS10, showing robust defense. This confirms ZePAD’s effectiveness across tasks, achieving Persistent-Robustness adversarial defense. To intuitively understand ZePAD’s impact on model generalization and robustness, we conducted t-SNE visualization when the pre-trained dataset was CIFAR10 and the downstream task was SVHN, with results shown in Figure 3 . It is clear that ZePAD sharpens classification boundaries on benign samples. When facing adversarial exsamples, the baseline method almost loses classification capability, but ZePAD can still correctly classify samples. Table 4: RA (%) comparison between the baseline method and ZePAD in the semi-black-box setting, where \mathcal{D}_{f} and \mathcal{D}_{d} denote the adversarial training and downstream datasets, respectively. \mathcal{D}_{f} \mathcal{D}_{d} Method W-MSE BYOL NNCLR SimCLR MoCo2+ MoCo3 SupCon RESSL DINO SwAV VibCreg AVG baseline 41.16 46.85 44.05 67.52 65.55 64.28 56.89 51.69 31.77 59.55 43.07 52.03 ours 75.60 73.28 72.84 77.12 70.54 72.45 70.76 73.99 73.67 74.04 70.83 73.19 STL10 \Delta +34.44 +26.43 +28.79 +9.60 +4.99 +8.17 +13.87 +22.30 +41.90 +14.49 +27.76 +21.16 baseline 48.31 49.25 49.23 63.48 63.03 60.32 58.48 54.92 35.46 63.75 50.19 54.22 ours 75.49 72.15 74.21 75.23 69.35 70.76 74.60 77.62 74.71 73.33 75.90 73.94 ANIMALS10 \Delta +27.18 +22.90 +24.98 +11.75 +6.32 +10.44 +16.12 +22.70 +39.25 +9.58 +25.71 +19.72 baseline 10.85 12.41 11.29 24.63 27.02 27.60 13.22 12.37 10.62 27.82 11.56 17.22 ours 60.50 56.16 53.71 54.24 48.10 52.58 53.83 59.50 56.48 54.03 55.87 55.00 GTSRB \Delta +49.65 +43.75 +42.42 +29.61 +21.08 +24.98 +40.61 +47.13 +45.86 +26.21 +44.31 +37.78 baseline 27.60 34.41 29.82 38.24 42.87 41.60 28.97 35.10 21.24 37.69 27.13 33.15 ours 48.78 53.97 55.15 52.18 50.16 49.40 50.98 52.97 52.05 52.71 46.34 51.34 ImageNet20 \Delta +21.18 +19.56 +25.33 +13.94 +7.29 +7.80 +22.01 +17.87 +30.81 +15.02 +19.21 +18.18 baseline 9.16 11.68 14.99 14.58 21.51 16.38 11.10 9.27 8.95 16.08 10.55 13.11 ours 55.84 61.37 52.97 64.75 50.86 59.39 62.74 68.26 64.37 58.24 55.34 59.47 CIFAR10 SVHN \Delta +46.68 +49.69 +37.98 +50.17 +29.35 +43.01 +51.64 +58.99 +55.42 +42.16 +44.79 +46.35 baseline 11.78 11.78 12.80 24.31 16.49 12.66 14.00 11.33 15.80 12.44 20.42 14.89 ours 53.62 62.57 64.43 60.52 63.91 67.67 60.76 61.94 64.32 65.74 63.80 62.66 CIFAR10 \Delta +41.84 +50.79 +51.63 +36.21 +47.42 +55.01 +46.76 +50.61 +48.52 +53.30 +43.38 +47.77 baseline 26.76 33.02 22.42 46.23 40.43 42.37 25.12 41.98 33.68 42.77 40.84 35.97 ours 57.90 65.88 67.69 73.11 68.58 69.58 66.16 66.77 68.18 68.13 65.32 67.03 STL10 \Delta +31.14 +32.86 +45.27 +26.88 +28.15 +27.21 +41.04 +24.79 +34.50 +25.36 +24.48 +31.06 baseline 32.68 43.20 37.47 53.24 47.71 46.21 37.43 44.26 40.70 41.23 48.36 42.95 ours 65.49 70.52 72.78 72.67 70.85 71.77 71.65 73.86 71.47 72.37 71.96 71.40 ANIMALS10 \Delta +32.81 +27.32 +35.31 +19.43 +23.14 +25.56 +34.22 +29.60 +30.77 +31.14 +23.60 +28.45 baseline 19.85 25.37 12.65 30.69 29.44 28.34 15.81 28.47 21.95 21.74 24.44 23.52 ours 39.44 45.67 48.10 48.81 47.53 48.83 46.13 46.30 48.30 47.61 43.83 46.41 ImageNet20 \Delta +19.59 +20.30 +35.45 +18.12 +18.09 +20.49 +30.32 +17.83 +26.35 +25.87 +19.39 +22.89 baseline 6.44 6.26 6.72 26.38 12.97 11.88 9.04 6.55 19.58 15.44 19.42 12.79 ours 55.72 68.17 71.15 63.38 65.98 67.89 62.79 69.54 74.75 63.19 57.09 65.42 GTSRB SVHN \Delta +49.28 +61.91 +64.43 +37.00 +53.01 +56.01 +53.75 +62.99 +55.17 +47.75 +37.67 +52.63 Table 5: The ablation study of the proposed ZePAD. Adversarial Fine-Tuning MPAD Branch BMP Branch BA RA ASR \times \times \times 85.01 35.46 63.35 \times \checkmark \checkmark 90.84 59.40 38.89 \checkmark \times \checkmark 91.57 70.35 26.71 \checkmark \checkmark \times 84.51 72.47 19.36 \checkmark \checkmark \checkmark 91.66 74.71 21.99 5.3 Ablation Study In this section, we investigate the impact of different components of ZePAD. We use a pre-trained DINO encoder on CIFAR10 and conduct experiments on ANIMALS10. We analyze the effects of adversarial training, the MPAD-Branch, and the BMP-Branch on the overall method, with results in Table 5 . Removing adversarial training significantly reduces RA and increases ASR, indicating its effectiveness in enhancing robustness. Nonetheless, BA and RA still improve compared to standard testing, showing that pre-trained encoders from different self-supervised methods have complementary feature extraction patterns. When MPAD-Branch is removed, RA decreases with little change in BA, highlighting its role in boosting robustness without harming generalization. If BMP-Branch is removed, BA drops below standard testing levels, indicating that adversarial training can negatively affect generalization. Thus, BMP-Branch is crucial for maintaining and enhancing the model’s generalization capability. The other ablation studies can be found in Appendix G . 5.4 Detecting DAEs by Confidence Scores We find that the significant confidence score differences between the MPAE and BMP branches for benign and adversarial exsamples, as illustrated in Figure 2 . Then, we are curious that if we can utilize the inherent sensitivity to detect DAEs directly without additional DAE identification training. Specifically, we define a sample as a DAE if the confidence of the MPAE-Branch exceeds that of the BMP-Branch by a threshold of 0.1. The adversarial exsamples in this experiment were generated from AdvEncoder ( 50 ) . The results in Table 6 demonstrate high effectiveness. We achieve an average accuracy of 82.52% when the fine-tuning and downstream datasets are consistent (CIFAR10) and 79.04% when they are not (ANIMALS10). The higher accuracy with consistent datasets is attributed to smoother confidence score distributions, which allows for a clearer distinction between benign and adversarial inputs. Additionally, to further discuss the extracted patterns from different branches for various inputs, the visualization can be found in Appendix H . Table 6: DAE sample detection accuracy(%), where \mathcal{D}_{f} and \mathcal{D}_{d} denote the adversarial training and downstream datasets. \mathcal{D}_{f} and \mathcal{D}_{d} W-MSE BYOL NNCLR SimCLR MoCo2+ MoCo3 SupCon RESSL DINO SwAV VibCreg AVG CIFAR10 86.20 83.71 83.46 86.18 81.53 82.81 84.27 83.06 70.97 81.50 84.01 82.52 STL10 72.96 76.41 75.05 79.65 72.38 75.96 78.93 73.59 59.75 72.98 79.43 74.28 ANIMALS10 80.91 81.55 80.31 81.98 74.57 76.66 83.32 81.46 71.02 73.11 84.55 79.04 GTSRB 73.65 75.03 71.67 76.96 73.93 73.15 75.69 70.29 56.09 69.96 76.62 72.09 ImageNet20 72.25 77.99 76.65 81.23 73.49 76.48 82.67 77.71 64.08 72.21 83.09 76.17 SVHN 59.26 64.93 63.82 70.75 62.17 66.69 67.75 62.69 52.38 64.07 68.45 63.91 Table 7: Comparison study(%). Model Method BA UAP RA UAPGD RA SSP RA PAP RA AdvEncoder RA PGD-AT(2018 ICLR) 22.19 21.23 23.00 21.79 20.97 19.04 MART(2019 ICLR) 42.79 42.50 42.65 42.60 42.24 42.36 TRADES(2019 PMLR) 50.79 50.40 50.51 50.95 50.59 50.66 Gen-AF(2024 S&P) 68.69 64.43 68.65 65.92 65.48 62.28 ZePAD(ours) 82.07 77.56 76.51 72.93 72.10 73.26 SimCLR \Delta +13.38 +13.13 +7.86 +7.01 +6.62 +10.98 PGD-AT(2018 ICLR) 13.72 12.15 13.48 12.30 13.02 12.70 MART(2019 ICLR) 42.79 41.46 41.52 41.28 41.76 41.36 TRADES(2019 PMLR) 50.50 52.40 50.60 52.37 52.60 52.28 Gen-AF(2024 S&P) 69.09 64.98 68.15 59.41 60.47 66.00 ZePAD(ours) 81.60 76.05 74.83 72.15 67.96 74.30 MoCo v2+ \Delta +12.51 +11.07 +6.68 +12.74 +7.49 +8.30 “ \Delta ” represents the improvement ratio compared to the second-best metric. 5.5 Comparison Study In this section, we compare the proposed ZePAD with other advanced adversarial defense methods: Gen-AF, TRADES, MART, and PGD-AT. We use ImageNet for pre-training and STL10 as the downstream dataset. As shown in Table 7 , ZePAD outperforms existing methods in both robustness and generalization. ZePAD’s BA remains above 80%, indicating its effectiveness in boosting model generalization. In terms of robustness, ZePAD achieves over 70% RA against six attacks, outperforming all other methods and further demonstrating its zero-sacrifice characteristic. Besides, the computational cost comparison can be found in Appendix I . Finally, we extend our defense method to other tasks, and the corresponding discussions and experiments are provided in Appendix J . 6 Conclusion In this paper, we introduced ZePAD, the first zero-sacrifice persistent-robustness adversarial defense method designed for pre-trained encoders. The core idea behind ZePAD is to enhance adversarial robustness and benign performance by simultaneously leveraging two complementary branches. Our experimental results across various SSL methods and multiple datasets demonstrate that ZePAD provides robust performance in different settings, effectively enhancing adversarial robustness without sacrificing benign performance. Additionally, ZePAD’s persistent-robustness capability enables it to adapt to new tasks with a single tuning, making it scalable and adaptable to real-world applications. Further investigation is required in several areas. One important direction for future work is improving the interpretability of the ZePAD framework and DAEs. Gaining insights into them could help refine the model and offer more transparency. 7 Ethics Statement This work adheres to the ICLR Code of Ethics. In this study, no human subjects or animal experimentation was involved. All datasets used, including CIFAR10, STL10, ANIMALS10, SVHN, ImageNet and GTSRB were sourced in compliance with relevant usage guidelines, ensuring no violation of privacy. We have taken care to avoid any biases or discriminatory outcomes in our research process. No personally identifiable information was used, and no experiments were conducted that could raise privacy or security concerns. We are committed to maintaining transparency and integrity throughout the research process. 8 Reproducibility Statement We have made every effort to ensure that the results presented in this paper are reproducible. Specifically, our method is detailed in Section 4 , and the main steps are provided in Appendix D to facilitate reproducibility. Moreover, all code will be made publicly available after the review cycle to ensure the reproducibility. Details of the experimental settings, including datasets, models, SSL methods, training configurations, and hyperparameter choices, are presented in Section 5 and Appendix E . All datasets used in our experiments are publicly available, and the download links for all open-source pre-trained models involved in the experiments are released together with our code. 9