SemiOccam: A Robust Semi-Supervised Image Recognition Network Using Sparse Labels
Rui Yann, Xianglei Xing, General Artificial Intelligence Laboratory · 2025
Read SemiOccam: A Robust Semi-Supervised Image Recognition Network Using Sparse Labels by Rui Yann, Xianglei Xing, General Artificial Intelligence Laboratory (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-03582.txt.
Introduction
SemiOccam is a network for image recognition that learns from very few labeled examples. The authors note that rival methods often need complex training setups and hundreds of GPU hours, and they still do poorly when labels are rare.
Their design builds a layered mixture-density classifier that maximizes mutual information between image features and target classes, squeezing out redundant detail while keeping what sets classes apart. On three common datasets the network scores above 95% accuracy on two of them with only four labeled samples per class, and training takes minutes. The team also reports a data leakage flaw in the STL-10 dataset, removes the duplicates, and releases a cleaned version for fair testing.
Catalog record
| CREATOR | Rui Yann, Xianglei Xing, General Artificial Intelligence Laboratory |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-03582
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SemiOccam: A Robust Semi-Supervised Image Recognition Network Using Sparse Labels Rui Yann, Xianglei Xing, General Artificial Intelligence Laboratory Abstract We present SemiOccam , an image recognition network that leverages semi-supervised learning in a highly efficient manner. Existing works often rely on complex training techniques and architectures, requiring hundreds of GPU hours for training, while their generalization ability when dealing with extremely limited labeled data remains to be improved. To address these limitations, we construct a hierarchical mixture density classification decision mechanism by optimizing mutual information between feature representations and target classes, compressing redundant information while retaining crucial discriminative components. Experimental results demonstrate that our method achieves state-of-the-art performance on various datasets when using negligible labeled samples , and its simple architecture keeps training time to minute-level . Notably, this paper reveals a long-overlooked data leakage issue in the STL-10 dataset for semi-supervised learning tasks and removes duplicates to ensure the reliability of experimental results. We also release the deduplicated CleanSTL-10 dataset to facilitate fair and reliable research in future semi-supervised learning. 1 1 1 Code available at https://github.com/Shu1L0n9/SemiOccam . 2 2 2 CleanSTL-10 available at https://huggingface.co/datasets/Shu1L0n9/CleanSTL-10 . Keywords: Semi-Supervised Learning, Image Recognition, Vision Transformer, Gaussian Mixture Model 1 Introduction Deep learning has achieved remarkable success in image classification tasks. However, its performance is highly dependent on large-scale annotated datasets. In real-world applications, labeled data is often scarce and expensive to acquire . Therefore, semi-supervised learning (SSL) has received widespread attention in recent years as an effective solution. The goal of semi-supervised learning is to train models using a small amount of labeled data and large amount of unlabeled data, thereby improving the model’s generalization ability. Nevertheless, most existing SSL methods, such as consistency regularization and pseudo-labeling, rely on complex training strategies and network designs , leading to training costs of hundreds of GPU hours even on high-performance computing resources like NVIDIA V100, and their generalization ability when labeled data are extremely limited still needs improvement . Moreover, although some recent works have explored Vision Transformers (ViT) in SSL, they have yet to fully exploit its advantages. Gaussian Mixture Models (GMMs) are powerful probabilistic models that assume data is generated from a mixture of multiple Gaussian distributions. The inspiration for this research comes from Miller & Uyar 1996 ’s systematic study of the “Mixture of Experts (MoE)” structure based on the joint probability model of features and labels, as well as the principle of Occam’s razor. According to Occam’s razor, among competing hypotheses that explain the data well, the simplest one is preferred. This motivates us to design a model that balances complexity and interpretability, ensuring efficient and effective feature representation without unnecessary overfitting. In our research, we found that the performance of GMMs is highly related to the quality of feature vectors . If the features lack representativeness or have insufficient discriminative power, the classification performance will be severely affected. To address this , we designed a feature encoding framework based on a self-attention mechanism to fully exploit the global contextual information in images and generate more discriminative feature representations. Simultaneously , we incorporated semi-supervised learning strategies, utilizing a pseudo-labeling mechanism to efficiently leverage unlabeled data. This approach achieves superior classification performance within the framework of mixture density estimation, significantly overcomes the limitations of traditional methods in terms of feature representation capability, training cost and model generalization performance. Taking the Dino with ViT-large backbone as our feature extraction model, we validated the proposed semi-supervised Gaussian mixture model (SGMM) method on various datasets, results show that our method achieves state-of-the-art performance within minutes on a single Tesla T4 GPU and exhibits good stability and robustness, with strong potential for edge deployment. Figure 1: Performance Comparison. Comparison of our method with recent semi-supervised learning models on datasets with different numbers of labels, achieving state-of-the-art performance with low training cost. It is worth mentioning that we have revealed a serious problem that has been overlooked by existing semi-supervised learning research: there are 7,500+ duplicate samples between the training samples and the testing samples in STL-10 , which is undoubtedly a serious data leakage problem when directly used for semi-supervised learning tasks. To address this, we release the deduplicated CleanSTL-10 dataset for fair and reliable evaluation. Our main contributions can be summarized as follows: • We introduce the powerful ViT as a feature extractor into the classical yet often underappreciated MoE classifier for image recognition tasks. • We design a simple but performant semi-supervised learning strategy that effectively leverages negligible labels, achieves state-of-the-art performance. • We reveal the longstanding, yet overlooked issue of training set and test set contamination in the STL-10 dataset, which has not been addressed in other works. • We release the deduplicated CleanSTL-10 dataset to promote fair and reliable future research. The remainder of this paper is organized as follows: Section 2 reviews related works; Section 3 introduces our SemiOccam network; Section 4 demonstrates and analyzes experimental results, including both classification performance and training cost; Section 5 discusses the significance and limitations of the research findings and outlines future research directions. 2 Related Works 2.1 Semi-Supervised Learning and Gaussian Mixture Models Semi-Supervised Learning has been widely applied in tasks such as image classification and speech recognition in recent years, especially in scenarios where labeled data is scarce. Early SSL methods mainly relied on generating pseudo-labels and consistency regularization. Pseudo-Labeling is one of the classic methods in SSL, with the core idea of using an existing model to predict unlabeled data Yarowsky 1995 and adding its predicted labels to the training set as pseudo-labels. Representative works include Lee et al. 2013 and Zhang et al. 2021 , which iteratively enhance the labeled dataset through self-training. Although pseudo-labeling methods can effectively improve model performance, they are often affected by the quality of pseudo-labels and can easily lead to model overfitting. The study by Mishra et al. 2024 explores the issue of overconfidence caused by pseudo-labels in semi-supervised learning, analyzes the source of calibration errors, and aims to improve the reliability of the model by evaluating and potentially improving methods. Consistency Regularization methods improve the generalization ability of models by forcing them to maintain consistency on unlabeled data. The Virtual Adversarial Training (VAT) method proposed by Miyato et al. 2018 enhances the robustness of the model by perturbing the input data and maintaining the stability of the prediction results. The Mean Teacher method proposed by Tarvainen & Valpola 2017 also further improves model performance by using the consistency between the predictions of the teacher model and the output of the student model. Although these methods have achieved good results in many tasks, they often rely on complex training strategies and architectural designs, and their generalization ability across different datasets still needs to be improved. Semi-Supervised Gaussian Mixture Model is the basic probabilistic framework of our method. It combines GMM with unlabeled data and proposes a learning method that simultaneously considers both labeled and unlabeled data by maximizing the joint log-likelihood function. Our method is inspired by the classifier structure and learning algorithm proposed by Miller & Uyar 1996 , which effectively leverages unlabeled data to improve performance. In addition, many impressive works, such as Zong et al. 2018 , have extended GMMs by