Explorations in Self-Supervised Learning: Dataset Composition Testing for Object Classification

Raynor Kirkson E. Chavez* 1, Kyle Gabriel M. Reynoso* 1 Prospero C. Naval Jr. 1, Carlo R. Raquel 1 · 2024

Read Explorations in Self-Supervised Learning: Dataset Composition Testing for Object Classification by Raynor Kirkson E. Chavez* 1, Kyle Gabriel M. Reynoso* 1 Prospero C. Naval Jr. 1, Carlo R. Raquel 1 (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-00770.txt.

Introduction

Raynor Kirkson E. Chavez and co-authors ask how the makeup of a training set shapes self-supervised learning for image sorting. Their 2024 study pretains a SimCLR model on apartment images drawn from the Omnidata platform. The image sets differ in type, brightness, size, and camera view, and the learned features then feed a ResNet-50 classifier.

Through A/B tests, the team found that depth-based pretraining worked best on low-resolution images. RGB pretraining did better when images were sharper. Raising the brightness of training images also helped models handle low-resolution inputs, with no harm to their work on sharper ones. The results give practical hints for picking pretraining data when the target images are poor in quality.

Catalog record

CREATORRaynor Kirkson E. Chavez* 1, Kyle Gabriel M. Reynoso* 1 Prospero C. Naval Jr. 1, Carlo R. Raquel 1
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE17 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-00770

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Explorations in Self-Supervised Learning: Dataset Composition Testing for Object Classification Raynor Kirkson E. Chavez* 1, Kyle Gabriel M. Reynoso* 1 Prospero C. Naval Jr. 1, Carlo R. Raquel 1 Abstract This paper investigates the impact of sampling and pretraining using datasets with different image characteristics on the performance of self-supervised learning (SSL) models for object classification. To do this, we sample two apartment datasets from the Omnidata platform based on modality, luminosity, image size, and camera field of view and use them to pretrain a SimCLR model. The encodings generated from the pretrained model are then transferred to a supervised Resnet-50 model for object classification. Through A/B testing, we find that depth pretrained models are more effective on low resolution images, while RGB pretrained models perform better on higher resolution images. We also discover that increasing the luminosity of training images can improve the performance of models on low resolution images without negatively affecting their performance on higher resolution images. † † tocauthor: Raynor Kirkson E. Chavez, Kyle Gabriel M. Reynoso, Prospero C. Naval Jr., Carlo R. Raquel † † institute: University of the Philippines - Diliman, Quezon City, Philippines, *equal contribution 1 Introduction Self-supervised learning (SSL) has garnered increasing attention in the field of machine learning due to its ability to train on large amounts of unlabeled data and achieve good performance on a wide range of tasks through transfer learning. Some examples of computer vision tasks that have been successfully tackled using SSL include image classification, object detection, cross-model retrieval, and instance segmentation han2021 . SSL methods can be classified into two categories based on their architecture: contrastive and non-contrastive. Recently, researchers have focused on scaling SSL algorithms to work with even larger datasets and improving their performance by optimizing the encoding layer architecture which maps data points to the feature space fang2022 . However, the performance of these methods is still highly dependent on the characteristics of the input data, particularly the variation of features in the dataset. To address this issue, researchers have begun to study the performance of SSL algorithms across different datasets and to generate training sets with specific variations in the input data jing2019 . In addition to data augmentation techniques, some researchers have explored the idea of training multimodal models while removing certain modalities, with the goal of avoiding overfitting on the original dataset. This approach has shown promising results in some cases, but the degree of improvement can vary greatly depending on the characteristics of the dataset alayrac2020 . These findings suggest that there may be other factors, beyond the procedural details of the model itself, that influence the performance of SSL algorithms ma2022 . Further research is needed to understand these connections and identify potential ways to improve the performance of these methods. This paper explored different variations of input datasets to improve the performance of a (baseline) model. 2 Methodology In this study, we propose a method for self-supervised learning that leverages pre-generated datasets from the Omnidata platform omnidata as input for pretraining with the SimCLR model. We chose the SimCLR model for its computational efficiency, as it requires fewer operations than state-of-the-art SSL models simclr , allowing us to work with limited hardware (1 NVIDIA V100 Core). The datasets we use consist of apartment images generated from 3D models of various rooms. After pretraining, the learned encodings are transferred to a supervised learning model for finetuning. 2.1 Datasets Used For pretraining, we used the frl_apartment_0 and allensville datasets from the Omnidata starter dataset. These datasets consist of point-randomized simulated camera captures of 3D apartment room models. We used 512 x 512 RGB and depth images for model training, and also included associated camera information such as angle, distance, obliqueness, and point of view to characterize the datasets. 2.2 Pretraining The two apartment image variations are represented as classes and fed as input into the SimCLR model which uses contrastive loss to learn features by comparison across classes. Seven models were trained using different dataset variations. From the frl_apartment_0 and allensville datasets, only 3000 images (25% of the generated samples) were sampled for each variation due to hardware constraints. The SimCLR model runs for 50 epochs with a batch size of 32 for every training run. The checkpoint of each distinct model trained under a variation is saved for the finetuning step. The first two runs were varied across their data domain: RGB and depth. The first model was trained on RGB data while the second was fed black-and-white encoded depth data corresponding to the same images. The next run tested the application of the Brightness (2X) augmentation on the RGB image taken as input. The remaining runs tested variations in luminosity and field of view. To determine variations of these features, K-means clustering was performed on the one-dimensional numerical values denoting these qualities for three groups (k=3): high, medium, and low. Luminosity is defined as the mean pixel average luminosity of the generated RGB images while field of view (in radians) is indicated in the associated camera properties file of the RGB images. Four distinct models were trained based on the following clusters: low luminosity, high luminosity, low field of view and high field of view. 2.3 Transfer Learning The seven resulting models were tested on the object classification downstream task. The encoding from pretraining checkpoints were transferred to a ResNet 50 model whose last layer is unfrozen to allow for supervised learning. The models were run for 100 epochs with a batch size of 256 on two standard labelled image datasets: CIFAR-10 and STL-10. The CIFAR-10 dataset cifar10 consists of 60000 32x32 RGB images classified into 10 classes while the STL-10 dataset stl10 is a modified version of the CIFAR-10 dataset which contains fewer classified samples at a higher resolution (96x96) but more unlabelled instances of the same classes. Fourteen finetuned models resulted from running the seven models on two different datasets. An additional 2 models were generated by downsampling the STL-10 dataset into 32x32 pixel images as baseline for performance across data modalities (RGB and depth) for low resolution pictures. The top 1 train and test, and top 5 train accuracies of each epoch is recorded and plotted as a graph to compare model reliability across iterations. The convergence of these graphs toward their peak as well as their comparative accuracies were analyzed to validate the claim that changes in the distribution of datasets affect model performance. From the results, the researchers also hypothesized about underlying factors causing these fluctuations that may be explored by further studies. 3 Experimental Results 3.1 Modality Tests Figure 1: Top 5 Training accuracy of the SimCLR model pretrained on 3000 RGB or depth images and finetuned and tested on the CIFAR10 dataset. Figure 2: Top 5 Training accuracy of the SimCLR model pretrained on 3000 RGB or depth images and finetuned and tested on the STL10 dataset. Our experimental results (Fig. 1 and Fig. 2) show that both SimCLR models pretrained on depth and RGB data perform well above average on the image classification downstream task at epoch 0, indicating that both models learned useful representations from the images during the pretraining step. When evaluating the models on the CIFAR10

Read the complete text