Probability-Density-Aware Semi-supervised Learning
Shuyang Liu, Ruiqiu Zheng, Zhou Yu, Yunhang Shen, Zhou Yu, Ke Li, Xing Sun, Shaohui Lin · 2024
Read Probability-Density-Aware Semi-supervised Learning by Shuyang Liu, Ruiqiu Zheng, Zhou Yu, Yunhang Shen, Zhou Yu, Ke Li, Xing Sun, Shaohui Lin (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-17547.txt.
Introduction
Shuyang Liu and co-authors put probability density at the center of semi supervised learning in a 2024 paper. Such learning leans on two beliefs: neighbors share a label, and points in separate clusters differ. Most methods chase neighbors by similarity and neglect the cluster belief. The team first build a theoretical case for why density matters to the cluster assumption.
They then define a density aware similarity measure for finding true neighbors. A new label propagation algorithm built on it spreads labels through the data with clusters in mind. They also prove that classic pseudo labeling is a special case of their method. Across experiments, their approach beat recent rivals. The authors present density awareness as the missing ingredient in using unlabeled data well.
Catalog record
| CREATOR | Shuyang Liu, Ruiqiu Zheng, Zhou Yu, Yunhang Shen, Zhou Yu, Ke Li, Xing Sun, Shaohui Lin |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 61 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-17547
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Probability-Density-Aware Semi-supervised Learning Shuyang Liu, Ruiqiu Zheng, Zhou Yu, Yunhang Shen, Zhou Yu, Ke Li, Xing Sun, Shaohui Lin Abstract Semi-supervised learning (SSL) assumes that neighbor points lie in the same category ( neighbor assumption ), and points in different clusters belong to various categories ( cluster assumption ). Existing methods usually rely on similarity measures to retrieve the similar neighbor points, ignoring cluster assumption , which may not utilize unlabeled information sufficiently and effectively. This paper first provides a systematical investigation into the significant role of probability density in SSL and lays a solid theoretical foundation for cluster assumption . To this end, we introduce a Probability-Density-Aware Measure (PM) to discern the similarity between neighbor points. To further improve Label Propagation, we also design a Probability-Density-Aware Measure Label Propagation (PMLP) algorithm to fully consider the cluster assumption in label propagation. Last, but not least, we prove that traditional pseudo-labeling could be viewed as a particular case of PMLP, which provides a comprehensive theoretical understanding of PMLP’s superior performance. Extensive experiments demonstrate that PMLP achieves outstanding performance compared with other recent methods. Probability-Density-Aware Semi-supervised Learning Shuyang Liu , Ruiqiu Zheng 1 1 1 Equally contribution with the first author. , Yunhang Shen 2 2 2 The corresponding author, Tencent Youtu Laboratory. , Zhou Yu 3 3 3 The corresponding author, East China Normal University. , Ke Li , Xing Sun , Shaohui Lin zyu @ stat.ecnu.edu.cn, shenyunhang01 @ gmail.com Introduction Machine Learning’s remarkable breakthroughs rely on constructing high-quality, complex labeled datasets. However, dataset annotations are increasingly costly or even infeasible in many professional areas ( e.g. , medical and astronomical fields ( Chen et al. 2020b ; Xu et al. 2022 ) ). Semi-supervised learning (SSL) has been proposed to mitigate the demand for large-scale labels by extracting information from unlabeled data with guidance from a few labeled data. The existing SSL methods are designed based on the widely accepted assumption of consistency , which assumes that close data points probably have the same label and data points lie in the same cluster tend to have the same labels ( Zhou et al. 2003 ; Chapelle and Zien 2005 ; Iscen et al. 2019 ; Li et al. 2020 ; Zhao et al. 2022 ) . The latter prior that points from the same category should lie in the same cluster is specifically called cluster assumption ( Zhou et al. 2003 ; Iscen et al. 2019 ; Li et al. 2020 ) . In SSL, the assumption of consistency is usually considered by assigning the similarity between neighbor points. An enormous similarity indicates that the points tend to be in the same category. Some works use similarity to train a consistent encoder, considering that neighbor input should have neighbor features ( Sohn et al. 2020 ) . Contrastive learning learns consistent visual representations with unlabeled images ( Chen et al. 2020a ) . Adversarial training ( Miyato et al. 2018 ) and data augmentation ( Berthelot et al. 2019 ; Sohn et al. 2020 ) are also widely applied to gain more training samples and improve the learning of representation. High-confidence pseudo-labels are propagated with close neighbor points’ predictions and labels ( Iscen et al. 2019 ; Zhao et al. 2022 ; Elezi et al. 2018 ) . Combining consistent encoder and pseudo-labeling significantly improves the model performance ( Iscen et al. 2019 ) . Recently, the widely-used SSL framework often consists of supervised loss to take full consideration of labeled data, unsupervised loss to improve the quality of pseudo labels, and consistency loss to encourage a consistent encoder ( Zhao et al. 2022 ; Berthelot et al. 2019 ; Zheng et al. 2022 ) . Figure 1: The procedure of PMLP. First, we select the neighbor points and extract their features. Then, we calculate the densities on the path; the density information is used to construct the Probability-density-aware Measure(PM). PM can fully consider the cluster assumption . Finally, high-confidence predictions are used for pseudo-labeling with an affinity matrix. However, these works fail to fully consider assumption of consistency , for they do not use the prior of cluster assumption . When calculating the similarity between points, their measurements seldom consider the cluster prior. Considering that the points close to the decision boundary may be closer to points from other categories than some points in the same categories, these algorithms may be misled by wrongly assigning neighbors’ labels. Our proposed solution, the Probability-Density-Aware Measurement(PM), has the potential to fully utilize the cluster assumption through the incorporation of density information and could address the limitations of existing algorithms. A natural thought goes that for two points with different clusters, their path tends to traverse low-density regions. We statistically prove the thought and demonstrate that the path probably goes through low-density areas. The theorem indicates that density information significantly helps to consider the cluster assumption and severs as the keystone of PM. With PM, we can better measure the similarity between points. We deploy PM into Label Propagation to select better neighbors and generate higher-confidence pseudo-labels. The new algorithm is named Probability-Density-Aware Label-Propagation(PMLP). To show PM’s superiority and generality, we prove that traditional pseudo-labeling is a particular case of PMLP, which indicates that PMLP should always be better with just minor optimization. PM can be easily implemented to complement many popular SSL methods involving label propagation ( Zhou et al. 2003 ; Iscen et al. 2019 ; Zhao et al. 2022 ; Elezi et al. 2018 ) ; we deploy PM to the best performance LPA algorithm ( Zhao et al. 2022 ) to implement the experiment. Through comprehensive experiments on benchmark datasets, we show that PMLP achieves noticeable performance improvement compared to some of the most powerful approaches. In general, our contribution can be summarized into three points: • We first propose a theoretical description to cluster assumption , which reveals that probability density can help make full use of SSL’s prior; • We introduce a new Probablity-Density-Aware similarity measurement(PM) to fully consider SSL’s cluster prior. Probability-Density-Aware Label-Propagation(PMLP) is proposed based on PM. We mathematically prove PMLP’s superiority and generality; • Extensive experiments are conducted to validate the effectiveness of PMLP. For example, on CIFAR100 with 400 labeled samples, we can surpass the baseline model by 3.52\% and surpass the second-best algorithm by 1.21\% . Related Works Consistency Regularization The assumption of consistency is widely accepted in SSL. Its foundation is generating well-consistency models that exhibit similar features for similar inputs. The works in ( Hinton, Osindero, and Teh 2006 ; Bengio et al. 2006 ) improved consistency via self-training and finetuning the model with labeled data. \Pi-model ( Laine and Aila 2016 ) reduces the consistency loss between images and their augmentations, or between outputs and temporal average outputs. Mean-teacher ( Tarvainen and Valpola 2017 ) reduces consistency loss between current weights and temporal average weights. Data augmentation can help to extract more feature information ( Miyato et al. 2018 ; Elezi et al. 2018 ) . Recent algorithm ( Chen et al. 2020b ) involved contrastive learning to improve the model consistency, and LASSL ( Zhao et al. 2022 ) generated class-aware contrastive learning by limiting the comparisons within