Exploring internal representation of self-supervised networks: few-shot learning abilities and comparison with human semantics and recognition of objects

Asaki Kataoka, Yoshihiro Nagano, Masafumi Oizumi · 2025

Read Exploring internal representation of self-supervised networks: few-shot learning abilities and comparison with human semantics and recognition of objects by Asaki Kataoka, Yoshihiro Nagano, Masafumi Oizumi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-20364.txt.

Introduction

Asaki Kataoka, Yoshihiro Nagano, and Masafumi Oizumi study what self taught networks learn about objects in a 2025 paper. Self supervised methods train networks without labels, which makes them useful when labeled data are scarce. They may also hint at how the brain learns from its surroundings without a teacher.

The team tested networks trained with a contrast based method on few shot tasks, where new classes must be learned from a few examples. Those tests reveal how the inner patterns group objects into classes. They then compared the results with human meaning and human object recognition. The inner patterns matched human cognition well. The authors argue that this kind of self teaching may model parts of human learning.

Catalog record

CREATORAsaki Kataoka, Yoshihiro Nagano, Masafumi Oizumi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-20364

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Exploring internal representation of self-supervised networks: few-shot learning abilities and comparison with human semantics and recognition of objects Asaki Kataoka, Yoshihiro Nagano, Masafumi Oizumi Abstract Recent advances in self-supervised learning have attracted significant attention from both machine learning and neuroscience. This is primarily because self-supervised methods do not require annotated supervisory information, making them applicable to training artificial networks without relying on large amounts of curated data, and potentially offering insights into how the brain adapts to its environment in an unsupervised manner. Although several previous studies have elucidated the correspondence between neural representations in deep convolutional neural networks (DCNNs) and biological systems, the extent to which unsupervised or self-supervised learning can explain the human-like acquisition of categorically structured information remains less explored. In this study, we investigate the correspondence between the internal representations of DCNNs trained using a self-supervised contrastive learning algorithm and human semantics and recognition. To this end, we employ a few-shot learning evaluation procedure, which measures the ability of DCNNs to recognize novel concepts from limited exposure, to examine the inter-categorical structure of the learned representations. Two comparative approaches are used to relate the few-shot learning outcomes to human semantics and recognition, with results suggesting that the representations acquired through contrastive learning are well aligned with human cognition. These findings underscore the potential of self-supervised contrastive learning frameworks to model learning mechanisms similar to those of the human brain, particularly in scenarios where explicit supervision is unavailable, such as in human infants prior to language acquisition. K eywords Contrastive learning \cdot Few-shot learning \cdot Human semantics \cdot Human recognition \cdot Similarity \cdot Self-supervised learning 1 Introduction Self-supervised learning has recently gained significant attention from both the machine learning and neuroscience communities. Unlike supervised learning, which requires explicit task-specific labels, self-supervised learning relies on inherent structures within the data itself and does not require manual supervision. This property makes it particularly advantageous in machine learning, enabling models to be trained on vast amounts of uncurated (unlabeled) data. Recent studies have demonstrated the effectiveness of self-supervised learning as a powerful method for representation learning [ 1 , 2 , 3 , 4 , 51 , 5 , 6 , 7 , 8 , 9 , 10 ] . In neuroscience, it is equally important to investigate the characteristics of neural representations that emerge from self-supervised learning, as this can provide insights into learning mechanisms in the brain. Given that self-supervised learning does not require labeled input, it offers a plausible framework for brain-like learning. In particular, since language is considered a major source of supervision in humans [ 11 , 12 , 13 ] , self-supervised learning may play a central role in the brains of human infants before language acquisition, as well as in non-linguistic animals. When applied to the study of neural learning and information representation, self-supervised learning may help explain empirical findings showing that prelinguistic infants exhibit cognitive abilities–such as categorical representation–similar to those of adults [ 14 , 15 , 16 , 17 , 18 , 19 ] . Deep convolutional neural networks (DCNNs) have frequently been used as computational models of neural circuits to study such neural representations. Earlier studies have shown that DCNNs trained with supervised learning exhibit representational similarities to the visual systems of humans and animals [ 20 , 21 , 22 , 23 , 24 , 25 , 26 , 27 , 28 , 29 , 30 , 31 , 32 , 33 ] . Building on this foundation, recent work has demonstrated that DCNNs trained using self-supervised algorithms also show representational similarities to biological visual systems [ 34 , 35 , 36 , 37 , 38 , 39 , 40 , 41 ] , further supporting their plausibility as models of the visual system. In this study, we investigate the internal representations of deep convolutional neural networks (DCNNs) through the lens of inter-category relationship structures, as revealed by few-shot learning performance. Few-shot learning refers to the ability to recognize novel, previously unseen categories using only a limited number of examples. While prior studies have highlighted the similarity between DCNN representations and those of humans and animals, the structure of category-level representations has not been thoroughly explored. A recent study [ 42 ] evaluated the few-shot learning capabilities of DCNNs trained with both supervised and self-supervised methods. Building upon this work, we compare the category structures revealed through few-shot learning in DCNNs with human semantic organization and recognition performance, aiming to further clarify the nature of internal representations learned without explicit supervision. In this paper, we pursue three objectives: (i) to confirm that DCNNs trained with self-supervised learning can perform few-shot learning accurately, and (ii) to investigate their internal representations by comparing them to human semantic organization and (iii) human recognition performance. The few-shot learning ability (i) is evaluated based on the linear separability of categories within the neural representation space. From this evaluation, we derive category-wise confusion matrices for each trained DCNN. These matrices are then used to analyze the inter-category structure of the representations and to compare them to (ii) human semantic structures and (iii) human recognition patterns. While prior studies have investigated self-supervised DCNNs [ 2 , 42 , 43 ] , our novel contribution lies in the comparative analyses involving human-level semantics and recognition—specifically, objectives (ii) and (iii). Our experimental results indicate that the internal representations arising from self-supervised learning in DCNNs closely resemble human semantic structures and recognition patterns. These findings suggest that inter-category structures similar to those found in human cognition can emerge even before the application of explicit supervision, thereby supporting both the biological plausibility and practical utility of self-supervised learning in brain-like systems. 2 Materials and Methods 2.1 Overview Figure 1: Schematic illustration of the experimental procedure in this study. (A) A two-step methodology for evaluating the few-shot learning performance of DCNNs. In Step 1 (pre-training), the network is trained using self-supervised contrastive learning. In Step 2, pairwise few-shot classification is performed, and performance is assessed using error pattern matrices, where each cell represents the classification error rate between a pair of novel categories. (B) Clusters of object categories derived from the error pattern matrices are compared to coarse-grained category groupings based on human semantic relationships. (C) Similarity is evaluated between the error pattern matrices of the DCNNs and the confusion matrix obtained from human participants performing an object classification task. In this study, we adopt a two-step procedure to evaluate the few-shot learning performance of DCNNs trained using a self-supervised learning algorithm (Fig. 1 ). In the first step, the DCNNs are pre-trained with a self-supervised contrastive learning method. To assess the impact of the absence of supervision, we also train a separate DCNN using a supervised object classification task, which serves as a baseline for comparison. The evaluation of few-shot learning performance is conducted in the second step. In the initial pre-training step (Fig. 1 A, top), we train a DCNN using a self-supervised contrastive learning algorithm [ 44 , 45 ] . During this step, the synaptic weight parameters of the network are optimized to minimize an objective function. The loss function is designed such that it does not require explicit supervision signal during its training procedure. In

Read the complete text