An Empirical Study of Accuracy-Robustness Tradeoff and Training Efficiency in Self-Supervised Learning
Fatemeh Ghofrani, Pooyan Jamshidi · 2025
Read An Empirical Study of Accuracy-Robustness Tradeoff and Training Efficiency in Self-Supervised Learning by Fatemeh Ghofrani, Pooyan Jamshidi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-03507.txt.
Introduction
Fatemeh Ghofrani and Pooyan Jamshidi studied in 2025 how to train image models without labels, faster and with better resistance to attacks. Self-taught methods often need many passes to settle, and attack-proof training makes that worse. The team revisited a robust frame that uses several crops of each image.
More crops per image sped learning in that frame, which also adds a sameness term and a penalty term to cut epochs. A new variant adds free attack training, which reuses work to save time. It raised both clean accuracy and attack resistance while training faster. Tests used both plain classifiers and a patch-averaging way to judge the models.
Catalog record
| CREATOR | Fatemeh Ghofrani, Pooyan Jamshidi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-03507
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
An Empirical Study of Accuracy-Robustness Tradeoff and Training Efficiency in Self-Supervised Learning Fatemeh Ghofrani, Pooyan Jamshidi Abstract Self-supervised learning (SSL) has significantly advanced in learning image representations, yet efficiency challenges persist, particularly under adversarial training. Many SSL methods require extensive training epochs to achieve convergence, a demand further amplified in adversarial settings. To address this inefficiency, we revisit the robust EMP-SSL framework, emphasizing the crucial role of increasing the number of crops per image instance to accelerate the learning process. Unlike conventional contrastive learning, robust EMP-SSL leverages multiple crops per image, integrates an invariance term and regularization, and significantly reduces the required training epochs, enhancing time efficiency. Additionally, robust EMP-SSL is evaluated using both standard linear classifiers and multi-patch embedding aggregation, providing new insights into SSL evaluation strategies. This paper investigates these methodological improvements with a focus on adversarial robustness, combining theoretical analysis with comprehensive experimental evaluation. Our results demonstrate that robust crop-based EMP-SSL, when evaluated with standard linear classifiers, not only accelerates convergence but also achieves a superior balance between clean accuracy and adversarial robustness, outperforming multi-crop embedding aggregation. Furthermore, we extend this approach by applying free adversarial training to the Multi-Crop Self-Supervised Learning algorithm, resulting in the Cost-Free Adversarial Multi-Crop Self-Supervised Learning (CF-AMC-SSL) method. This method shows the effectiveness of free adversarial training in self-supervised learning, particularly when the number of epochs is reduced. Our findings underscore the efficacy of CF-AMC-SSL in simultaneously improving clean accuracy and adversarial robustness within a reduced training time, offering promising avenues for practical applications of SSL methodologies. Our code is released at https://github.com/softsys4ai/CF-AMC-SSL . I Introduction In recent years, progress in self-supervised learning (SSL) [ 1 ] has produced representations that match or exceed those achieved by supervised learning in classification tasks [ 6 , 14 ] . This advancement has led to state-of-the-art performance in various applications (e.g., models like BERT and GPT-3) [ 4 , 10 ] . A prominent method in the realm of Self-Supervised Learning (SSL), namely joint-embedding SSL [ 2 , 6 , 26 ] , is primarily concerned with the generation of uniform representations for image augmentations. As delineated by Wu [ 25 ] , the principle of instance contrastive learning requires the training of dual networks to produce analogous embeddings for disparate views of an identical image within a joint embedding framework. Originating from the Siamese network architecture [ 3 ] , these methodologies deal with a fundamental obstacle, colloquially termed ‘collapse,’ where all representations converge to uniformity, thereby disregarding input heterogeneity. To counteract this collapse phenomenon, two principal strategies have been proposed: contrastive and information maximization. Contrastive learning methodologies aim to identify dissimilar samples employing current branches or memory banks [ 6 , 15 ] . On the contrary, noncontrastive methodologies such as TCR [ 19 ] , Barlow Twins [ 26 ] , and VICReg [ 2 ] exploit covariance regularization as a means to avert the collapse of representations. Simultaneously, the technique of Swapping Assignments between Views (SwAV) [ 5 ] delves into the application of multi-crop techniques in self-supervised learning, favoring a combination of views with diverse resolutions over a pair of full-resolution views. This groundbreaking research underscores the potential for enhanced SSL performance via multi-view augmentation, thereby offering a potent strategy to combat the issue of collapsing representations. Despite the considerable potential displayed by self-supervised learning to derive effective representations from extensive unlabeled data, its vulnerability to adversarial attacks remains a substantial concern within the field [ 12 , 17 , 24 ] . In response to these attacks, adversarial training is one of the most reliable and effective methods to address this challenge. This technique can be formulated as a form of Min-Max Optimization [ 20 ] , where the model parameters are iteratively updated to minimize training loss while simultaneously dealing with adversarial perturbations created by optimizing a specific adversary loss function. In recent years, there has been considerable focus on exploring how adversarial training affects the robustness of various self-supervised learning approaches [ 7 , 12 ] . Kim et al. [ 17 ] pioneered the use of contrastive loss to create adversarial examples without relying on any labels, to strengthen the robustness of the SimCLR [ 6 ] . Subsequently, Moshavash et al. [ 21 ] , Wahed et al. [ 24 ] , and Gowal et al. [ 13 ] adopted this approach for Momentum Contrast [ 15 ] , SwAV [ 5 ] , and BYOL [ 14 ] , respectively. Fan et al. [ 11 ] introduced an additional regularization term within the contrastive loss to improve the transferability of cross-task robustness. They employed a method of generating pseudo-labels during adversarial training for downstream tasks. Similarly, Jiang et al. [ 16 ] explored the robustness under different pair selection scenarios by considering a linear combination of two contrastive loss functions. TABLE I: CF-AMC-SSL trains efficiently in fewer epochs, thereby reducing overall training time. By effectively employing multi-crop augmentations during base encoder training, it enhances both clean accuracy and robustness against PGD attacks. Note that the highest values are indicated in red, while the second highest values are highlighted in blue. Models CIFAR-10 CIFAR-100 Time Base Encoder Clean PGD(4/255) PGD(8/255) Clean PGD(4/255) PGD(8/255) (min) Patch-based EMP-SSL (baseline) (16 patches, 5-step PGD, 30 epochs) 61 37.65 16.95 39.26 14.38 4.22 530 Crop-based EMP-SSL (16 crops, 5-step PGD, 30 epochs) 76.55 53.3 28.49 51.71 33.88 19.35 530 Crop-based SimCLR (baseline) (2 crops, 5-step PGD, 500 epochs) 72.86 47.98 16.81 44.57 19.84 5.68 934 Patch-based SimCLR (2 patches, 5-step PGD, 500 epochs) 65.44 41.85 17.19 43.71 21.87 8.33 934 Patch-based EMP-FreeAdv (16 patches, m=3, 10 epochs) 61.83 42.28 21.53 40.31 23.78 12.13 97 Crop-based SimCLR-FreeAdv (2 crops, m=3, 167 epochs) 70.25 48.34 24.5 47.64 26.53 11.7 157 Crop-based EMP-FreeAdv (CF-AMC-SSL) (16 crops, m=3, 10 epochs) 75.88 55.97 33.34 50.74 31.73 17.19 97 Despite advancements in self-supervised learning (SSL), achieving adversarial robustness remains challenging due to two primary factors: 1. High computational cost: Adversarial training requires additional gradient computations to generate adversarial examples, significantly increasing training time. 2. Trade-off between clean accuracy and robustness: Adversarial training often compromises clean accuracy to improve robustness. Extreme-Multi-Patch Self-Supervised Learning (EMP-SSL) [ 23 ] offers a promising approach by reducing training epochs and leveraging fixed-size image patches instead of traditional multi-scale cropping. Building on EMP-SSL, this work explores the interplay between adversarial training and SSL, focusing on the following central questions: 1. Can multiple crops or patches compensate for fewer training epochs in self-supervised adversarial training? We examine whether increasing data diversity via multiple crops or patches can reduce computational overhead while maintaining performance. 2. How does crop diversity impact clean accuracy and robustness? By integrating EMP-SSL’s mechanisms with adversarial training, we analyze whether multiple crops can better balance the trade-off compared to SimCLR, which uses only two augmentations per image. 3. What is the impact of augmentation strategy (multi-scale crops vs. fixed-size patches) on robustness? We investigate how adopting diverse cropping strategies affects model robustness within the EMP-SSL framework. 4. How do different evaluation strategies perform for adversarially trained models? We compare robust Multi-Crop Embedding Aggregation (averaging embeddings from multiple crops) with the standard linear classifier applied to a single whole-image