Exploring Structural Degradation in Dense Representations for Self-supervised Learning
Siran Dai Qianqian Xu Peisong Wen Yang Liu Qingming Huang Institute of Information Engineering, Chinese Academy of Sciences School of Cyber Security, University of Chinese Academy of Sciences State Key Laboratory of AI Safety, Institute of Computing Technology, CAS Peng Cheng Laboratory School of Co · 2025
Read Exploring Structural Degradation in Dense Representations for Self-supervised Learning by Siran Dai Qianqian Xu Peisong Wen Yang Liu Qingming Huang Institute of Information Engineering, Chinese Academy of Sciences School of Cyber Security, University of Chinese Academy of Sciences State Key Laboratory of AI Safety, Institute of Computing Technology, CAS Peng Cheng Laboratory School of Co (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-17299.txt.
Introduction
Longer training is usually assumed to help a learning model, but Siran Dai and four co-authors report the opposite for detailed prediction tasks. In self-supervised learning, where models learn from unlabelled data, extra training hurt tasks like marking each pixel of an image by class. The team names this effect self-supervised dense degradation and finds it across many leading methods.
To spot the problem without labels, Dai and her co-authors build an estimator from two measures: how relevant hidden units are to classes, and how many useful directions the representation really uses. Picking models by that score raised accuracy by about three points on average. A training penalty based on the same score also eased the decline.
Catalog record
| CREATOR | Siran Dai Qianqian Xu Peisong Wen Yang Liu Qingming Huang Institute of Information Engineering, Chinese Academy of Sciences School of Cyber Security, University of Chinese Academy of Sciences State Key Laboratory of AI Safety, Institute of Computing Technology, CAS Peng Cheng Laboratory School of Co |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 117 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-17299
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Exploring Structural Degradation in Dense Representations for Self-supervised Learning Siran Dai Qianqian Xu Peisong Wen Yang Liu Qingming Huang Institute of Information Engineering, Chinese Academy of Sciences School of Cyber Security, University of Chinese Academy of Sciences State Key Laboratory of AI Safety, Institute of Computing Technology, CAS Peng Cheng Laboratory School of Computer Science and Tech., University of Chinese Academy of Sciences [email protected] [email protected] {wenpeisong, qmhuang}@ucas.ac.cn [email protected] Abstract In this work, we observe a counterintuitive phenomenon in self-supervised learning (SSL): longer training may impair the performance of dense prediction tasks (e.g., semantic segmentation). We refer to this phenomenon as Self-supervised Dense Degradation (SDD) and demonstrate its consistent presence across sixteen state-of-the-art SSL methods with various losses, architectures, and datasets. When the model performs suboptimally on dense tasks at the end of training, measuring the performance during training becomes essential. However, evaluating dense performance effectively without annotations remains an open challenge. To tackle this issue, we introduce a Dense representation Structure Estimator (DSE), composed of a class-relevance measure and an effective dimensionality measure. The proposed DSE is both theoretically grounded and empirically validated to be closely correlated with the downstream performance. Based on this metric, we introduce a straightforward yet effective model selection strategy and a DSE-based regularization method. Experiments on sixteen SSL methods across four benchmarks confirm that model selection improves mIoU by 3.0 % 3.0\% on average with negligible computational cost. Additionally, DSE regularization consistently mitigates the effects of dense degradation. Code is available at https://github.com/EldercatSAM/SSL-Degradation . 1 Introduction Self-Supervised Learning (SSL) has greatly benefited from advancements in training algorithms, larger datasets and models, and extended training periods, leading to significant success in image-level representation learning [ 10 , 16 , 3 , 29 ] . However, dense (patch or pixel-level) representation learning remains challenging with only slight improvements [ 70 , 5 ] . While training models for extended periods to extract high-quality representations has been a common practice in SSL, we identify and study a counterintuitive phenomenon, named Self-supervised Dense Degradation (SDD), which helps explain the challenges in self-supervised dense representation learning. As illustrated in Fig. 1 , although the training loss converges and classification performance steadily improves, the dense performance declines at the later stages of training. Consequently, the final checkpoint exhibits a significant performance gap compared to the best performance observed during training. Loss Cls Accuracy Seg mIoU (a) MoCo v3 [ 16 ] (b) DINO [ 10 ] Figure 1 : The Self-supervised Dense Degradation (SDD) phenomenon. While the training loss converges and classification performance gradually increases, the segmentation performance declines and the final checkpoint is not optimal. The metrics are normalized to 1 for a better illustration. Extensive experiments on sixteen state-of-the-art methods across four benchmarks confirm that the SDD phenomenon consistently appears across diverse training approaches and evaluation protocols. More importantly, it persists even when training and evaluation are conducted on the same dataset. This demonstrates that SDD highlights the performance inconsistency between different tasks rather than overfitting to the data distribution, introducing a new challenge to the SSL community. Because of SDD, the common practice of training until convergence results in suboptimal dense performance. Finding a metric to predict downstream dense performance thus becomes essential for both understanding the cause of SDD and reducing its negative impact. Typically, models are evaluated on a labeled validation set; however, the cost of evaluating even a single checkpoint can exceed that of one full epoch of pretraining, making this impractical. Moreover, SSL typically lacks access to downstream data or labels. Although previous studies have tried estimating SSL downstream performance in an unsupervised manner [ 2 , 24 , 69 , 68 , 36 , 60 ] , these metrics mainly focus on image-level tasks and are negatively correlated with dense-level performance in our experiments. To guide the development of a better metric, we first provide a theoretical analysis based on error rate decomposition. We show that the downstream error rate is small if 1) intra-class representation radius is smaller than inter-class distance, and 2) effective dimensionality of representations is large. Based on this analysis, we propose a Dense Representation Structure Estimator (DSE), which consists of measures for class-separability and effective dimensionality, directly corresponding to our theoretical findings. Using the DSE metric, we propose two strategies to counteract SDD. In the off-the-shelf setting, we suggest a simple yet effective method of selecting the checkpoint with the highest local DSE. In the online training setting, we integrate the DSE metric as a regularizer. Empirical evaluations demonstrate that the proposed DSE accurately predicts downstream performance, significantly outperforming existing metrics. By visualizing components of the DSE, we explain that the cause of the SDD phenomenon is either a loss of class separability or dimensional collapse. For example, in Fig. 1 , MoCo v3’s performance drop is due to dense dimensional collapse, while DINO’s degradation arises from decreased class separability. Moreover, our proposed checkpoint selection approach improves the mIoU by 3.0% on average, and incorporating DSE into training further improves both DSE scores and downstream dense performance, effectively eliminating SDD’s negative effects. We summarize our contributions as follows: • We identify the SDD phenomenon in SSL, revealing an inconsistency between image-level and dense-level performance, which is prevalent in state-of-the-art methods and negatively impacts dense tasks. • We introduce the Dense Representation Structure Estimator (DSE), which accurately and efficiently predicts downstream dense performance without relying on downstream data. • Using the DSE metric, we propose model selection and regularization strategies that effectively reduce the negative impact of the SDD phenomenon. • Extensive experiments on sixteen leading SSL methods across four benchmarks demonstrate the precision of DSE and the effectiveness of the proposed approaches in addressing the SDD phenomenon. 2 Related Work 2.1 Self-supervised Learning Contrastive Learning. Contrastive self-supervised methods have shown significant progress in recent years [ 11 , 73 , 28 , 61 , 62 ] . These methods typically construct positive examples (different augmented views of an image) and negative examples (samples from other images), then use contrastive loss [ 49 ] to train models to differentiate between them [ 12 , 15 , 78 , 4 , 32 , 27 , 30 ] . Non-contrastive Learning. Recent advances in self-supervised learning avoid explicit negative examples. These methods often employ siamese networks and aim to achieve cross-view consistency by aligning representations between teacher and student networks [ 26 , 10 , 83 , 50 , 14 , 9 ] . Masked Image Modeling. Masked image modeling can be framed as a generative task. Models are trained to reconstruct original images from masked inputs [ 29 , 74 , 3 , 13 ] . Recent works [ 83 , 50 ] achieved great success by combining latent-space reconstruction with self-distillation. Dense Representation Learning. Another line of work focuses on learning dense representations. Using techniques such as dense alignment [ 86 , 70 , 31 , 71 ] , clustering [ 84 , 58 , 40 , 59 , 5 ] , and reconstruction [ 83 , 50 ] , these methods achieve strong results on dense prediction tasks. However, despite optimizing dense representations directly, performance degradation is still observedduring training. 2.2 Unsupervised Transferability Estimation Recently, \alpha -REQ [