Pre-train to Gain: Robust Learning Without Clean Labels
David Szczecina · 2025
Read Pre-train to Gain: Robust Learning Without Clean Labels by David Szczecina (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-20844.txt.
Introduction
David Szczecina offers in 2025 a simple defence against a common training problem: datasets whose labels contain errors. Deep networks memorise wrong labels, wrecking their accuracy on clean data, and most existing fixes assume a small clean subset exists to learn from. Szczecina's recipe needs none. First, a feature extractor learns from the images alone through self-supervised learning, which invents its own training signals; then supervised training runs on the noisy labels.
He tested two self-supervised methods, SimCLR and Barlow Twins, on the CIFAR-10 and CIFAR-100 image datasets. Noise came in two forms: synthetic, and the real-world noise of the CIFAR-N variants. Across every noise rate, pre-training improved classification accuracy and made label errors easier to detect afterwards. The advantage grew as noise increased. At low noise the approach merely matched models pre-trained on ImageNet, but at high noise it substantially outperformed them.
Catalog record
| CREATOR | David Szczecina |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 19 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-20844
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Pre-train to Gain: Robust Learning Without Clean Labels David Szczecina Abstract Training deep networks with noisy labels leads to poor generalization and degraded accuracy due to overfitting to label noise. Existing approaches for learning with noisy labels often rely on the availability of a clean subset of data. By pre-training a feature extractor backbone without labels using self-supervised learning (SSL), followed by standard supervised training on the noisy dataset, we can train a more noise robust model without requiring a subset with clean labels. We evaluate the use of SimCLR and Barlow Twins as SSL methods on CIFAR-10 and CIFAR-100 under synthetic and real world noise. Across all noise rates, self-supervised pre-training consistently improves classification accuracy and enhances downstream label-error detection (F1 and Balanced Accuracy). The performance gap widens as the noise rate increases, demonstrating improved robustness. Notably, our approach achieves comparable results to ImageNet pre-trained models at low noise levels, while substantially outperforming them under high noise conditions. 1 Introduction Deep neural networks have achieved remarkable success across a wide range of supervised learning tasks [ 12 , 7 ] . However, their performance is heavily dependent on the quality of labeled data. In real-world settings, datasets often contain noisy labels due to human annotation errors, automated labeling processes, or ambiguous data, where \eta\in[0,1] denotes the fraction of labels that are incorrect [ 27 , 16 ] . Training directly on such noisy labels typically leads to poor generalization, as deep networks are prone to memorizing the noise [ 20 , 26 , 19 ] , especially in the absence of explicit noise-handling mechanisms. Figure 1 : Classification accuracy across varying label noise rates, for increasing durations of self-supervised pre-training on CIFAR-10 using SimCLR. Mean over 5 seeds with standard error is graphed. As the number of SSL pre-training epochs increases, downstream accuracy consistently improves across all corruption levels. A common strategy to mitigate this issue is to identify and correct noisy labels using robust loss functions or noise-aware training strategies [ 13 , 5 , 22 , 28 ] . Many of these methods, however, assume access to a small subset of clean labels ( \eta=0 ) or require prior knowledge about the noise distribution [ 13 , 14 , 5 ] , assumptions that are often unrealistic in practical scenarios. Self-supervised learning (SSL) offers a promising alternative by allowing feature representations to be learned without using labels at all [ 1 , 25 , 4 , 2 , 6 , 21 ] . SSL methods such as SimCLR [ 1 ] and Barlow Twins [ 25 ] leverage data augmentations and contrastive or redundancy-reduction objectives to train encoders that capture semantically meaningful representations. By using SSL, feature extractors can be created without considering labels, thereby avoiding any negative influences of mislabelled data. In this work, we investigate whether self-supervised pre-training can enhance a model’s noise robustness when training on noisy-labelled datasets, without relying on any clean subset of data. We perform experiments on CIFAR-10 and CIFAR-100 [ 11 ] , both with synthetically introduced label noise, and with real-world noisy labels from the CIFAR-10N and CIFAR-100N datasets [ 23 ] . We compare a standard supervised training pipeline against a two-stage approach where a ResNet18 [ 8 ] backbone is first pre-trained using self-supervised learning and then fine-tuned with supervised training on the same noisy dataset. As seen by results in Figure 1 , self-supervised pre-training consistently improves model robustness. Specifically, we observe higher test accuracy, evaluated on clean test sets, and improved label error detection capability when predicting on noisy test sets. We further compare our SSL pre-training scheme against the use of ImageNet pre-trained weights for model initialization [ 3 ] , finding comparable performance at low rates of label noise, but the effectiveness of ImageNet weights diminishes with higher noise rates. In contrast, self-supervised pre-training continues to offer substantial performance improvements under severe label noise. This study demonstrates that self-supervised pre-training on the target dataset can serve as a simple yet powerful method for improving models robustness to label errors when training on noisy datasets. 2 Background Learning with Noisy Labels Supervised learning assumes access to correctly labelled data, but in real-world scenarios, label noise is common [ 15 ] . Training deep neural networks directly on noisy data often results in memorization of incorrect labels, leading to poor generalization and overfitting [ 20 , 26 , 19 ] . Numerous strategies have been proposed to address training with label noise, including: • Robust loss functions that reduce the model’s sensitivity to incorrect labels [ 17 , 27 , 24 ] . • Sample re-weighting to prioritize clean samples [ 5 , 9 ] . • Semi-supervised and weakly supervised approaches that leverage a clean subset to guide training [ 13 , 14 ] . A recurring limitation across many of these approaches is the reliance on a small subset of data with clean labels, which may not be feasible in large-scale or low-resource settings [ 15 ] . Self-Supervised Learning (SSL) Self-supervised learning has emerged as a powerful method for representation learning without requiring any labels [ 1 , 25 , 4 ] . In computer vision, SSL methods typically use data augmentations and auxiliary pretext tasks to learn meaningful image embeddings. Two widely used approaches are SimCLR, a contrastive learning method [ 1 ] , and Barlow Twins, which is a redundancy reduction method [ 25 ] . These methods have been shown to learn generalizable and semantically rich features, even without labels. Importantly, when used for pre-training, these SSL representations can significantly improve downstream performance by providing better model initialization. Label Error Detection Identifying mislabelled instances is a critical step in improving training with noisy datasets. Frameworks for detecting label errors typically require well-trained and well-generalized models to perform effectively. Several techniques have been proposed for this purpose, ranging from training dynamics based methods [ 19 ] to confidence-based filtering. Confident Learning [ 16 ] is a prominent framework that estimates the probability that each example’s given label is incorrect. It does so using a combination of predicted probabilities and the confusion matrix estimated from the model’s outputs. Related Works Contrast to Divide (C2D) [ 28 ] identified a key limitation in learning with noisy labels, the “warm-up obstacle”, where models quickly begin memorizing incorrect labels during initial supervised training. C2D addresses this by introducing a two-stage framework: a self-supervised contrastive pre-training phase followed by integration with existing learning-with-noisy-labels (LNL) algorithms such as DivideMix [ 13 ] or ELR+ [ 14 ] . This pre-training step provides noise-invariant features that improve label separation and classification accuracy. In contrast to C2D, our work isolates the effect of self-supervised pre-training itself, without coupling it to any LNL algorithm or specialized loss. Rather than improving a specific warm-up mechanism, we show that stand-alone SSL pre-training on the same noisy dataset consistently enhances downstream performance for both classification accuracy and the models ability to detect label-errors. 3 Method To evaluate the effectiveness of self-supervised pre-training in enhancing robustness to label noise, we compare a baseline standard supervised training method against a two-stage training pipeline that incorporates SSL pre-training. • Baseline Supervised Method : The model