Brain-Inspired Stochastic Joint Embedding Representation Learning
Makoto Yamada, Kian Ming A. Chai, Ayoub Rhim, Satoki Ishikawa, Mohammad Sabokrou, Yao-Hung Hubert Tsai · 2025
Read Brain-Inspired Stochastic Joint Embedding Representation Learning by Makoto Yamada, Kian Ming A. Chai, Ayoub Rhim, Satoki Ishikawa, Mohammad Sabokrou, Yao-Hung Hubert Tsai (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-11129.txt.
Introduction
Machines that learn from images usually need heavy editing of training pictures, while the human eye learns from plain sequences. Makoto Yamada and co-authors take a step toward the brain's way in this 2025 paper with a model called PhiNet v2. The model learns from a stream of images in time order, without strong editing of the input.
Its learning rule comes from a form of reasoning that weighs what is likely, called variational inference. In tests, PhiNet v2 matched strong recent models on image tasks while still learning from sequences. The authors frame the work as progress toward vision systems that take in the world more as people do. They stress that the model keeps its strength even when the usual tricks for editing training images are dropped.
Catalog record
| CREATOR | Makoto Yamada, Kian Ming A. Chai, Ayoub Rhim, Satoki Ishikawa, Mohammad Sabokrou, Yao-Hung Hubert Tsai |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-11129
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Brain-Inspired Stochastic Joint Embedding Representation Learning Makoto Yamada, Kian Ming A. Chai, Ayoub Rhim, Satoki Ishikawa, Mohammad Sabokrou, Yao-Hung Hubert Tsai Abstract Representation learning is one of the key research topics in machine learning, and the framework of self-supervised learning (SSL) has revolutionized computer vision. However, these approaches have not yet fully leveraged insights from biological visual processing systems. In this paper, we introduce PhiNet v2, a novel architecture that processes temporal visual input (i.e., sequences of images) without relying on strong data augmentation, enabling it to learn robust visual representations in a manner similar to human visual processing. Our learning objective is derived from variational inference. Through extensive experiments, we demonstrate that PhiNet v2 achieves competitive performance compared to state-of-the-art vision representation models, including RSP and CropMAE, while retaining the ability to learn effectively from sequential input without strong data augmentation. This work represents a step toward more biologically plausible computer vision systems that process visual information in a manner more aligned with human cognitive processes. 1 Introduction Self-supervised learning (SSL) has emerged as a powerful paradigm in computer vision, advancing significantly through vision foundation models like SimCLR [ Chen et al., 2020a ] , MoCo [ Chen et al., 2020b ] , DINOs [ Caron et al., 2021 , Oquab et al., 2024 , Siméoni et al., 2025 ] , and Masked Autoencoders (MAE) [ He et al., 2022 ] . While these methods have achieved impressive results through architectural innovations and learning objectives, they have largely overlooked one of nature’s most efficient learning systems; the human brain. Humans excel at learning visual representations through their ability to process continuous streams of visual information, suggesting that combining brain-inspired mechanisms with sequential processing can enhance machine learning approaches. PhiNet [ Ishikawa et al., 2025 ] (Figure 1(b) ) marks a significant step toward brain-inspired learning. Drawing from biological circuitry in entorhinal cortex (EC), hippocampus and neocortex (Figure 1(a) ), PhiNet combines the temporal prediction hypothesis [ Chen et al., 2024 ] with Complementary Learning Systems (CLS) theory [ McClelland et al., 1995 ] . The model uses ResNet-based encoders, two predictors, and exponential moving average (EMA) modules, all trained end-to-end via backpropagation. This model, henceforth called PhiNet v1, has matched or exceeded the performance of leading ResNet-based methods including SimSiam [ Chen and He, 2021 ] , BYOL [ Grill et al., 2020 ] , and Barlow Twins [ Zbontar et al., 2021 ] , while maintaining robustness to weight decay choices. While PhiNet v1 has shown promise in bridging machine learning with brain-inspired principles, it is not designed for sequential learning. In contrast, a truly comprehensive brain-inspired system should inherently support sequential processing, that is, the ability to learn from continuous streams of visual input, similar to how humans process visual information. In addition, PhiNet v1 has two other limitations. First, its ResNet-based architecture could be improved by incorporating a more powerful backbone (i.e., Transformers) to enhance representation learning. From a neuroscience perspective, recent studies suggest that Transformers closely resemble current models of the hippocampus [ Whittington et al., 2022 ] , making them a promising architectural choice for the next iteration of PhiNet. Second, its reliance, like SimSiam and BYOL, on strong data augmentation techniques limits its generalizability, particularly when learning from sequential input, where effective augmentations are harder to define. We present PhiNet v2 (Figure 2(b) ), a Transformer-based model that learns robust representations from sequential visual input. Our choice of input brings it closer to the work of Chen et al. [2024] , which is an inspiration for PhiNet v1. We also replace PhiNet v1’s ResNet-based encoder and predictor with Transformer-based modules, and incorporated an uncertainty model to capture the stochasticity in visual input [ Jang et al., 2024 ] . By careful architectural design incorporating these changes, we achieve stable and effective training. Our extensive experiments on standard computer vision tasks show that PhiNet v2 outperforms recent strong baselines, including DINO [ Caron et al., 2021 ] , SiamMAE [ Gupta et al., 2023 ] , and RSP [ Jang et al., 2024 ] . Ablation studies confirm that each architectural component of PhiNet v2 is crucial for stable and robust training. Our contributions are summarized as follows: • We propose PhiNet v2, the first brain-inspired vision representation learning based on a Transformer architecture. • PhiNet v2 is capable of learning from sequential visual input without relying on strong data augmentation and masking, which is required by most existing vision foundation models. • We formulate a variational learning objective for PhiNet v2, bridging brain-inspired modeling with probabilistic learning. • PhiNet v2 achieves competitive or superior performance compared to strong pretraining methods such as DINO [ Caron et al., 2021 ] , SiamMAE [ Gupta et al., 2023 ] , CropMAE [ Eymaël et al., 2024 ] , and RSP [ Jang et al., 2024 ] . 2 Related work In this section, we review the evolution of self-supervised learning methods in computer vision, from contrastive learning approaches to transformer-based architectures, leading to our brain-inspired representation learning approach. Self-supervised learning (SSL) has emerged as a powerful paradigm for learning representations without explicit labels. A widely adopted approach is contrastive learning, where models are trained to pull together similar pairs (augmentations of the same image) while separating dissimilar pairs. SimCLR [ Chen et al., 2020a ] is a representative example, employing large batch sizes to mine hard negatives effectively. MoCo [ He et al., 2020 ] introduces a momentum encoder and a queue-based dictionary, enabling the use of a dynamic memory bank to decouple batch size from the number of negative samples. MoCo v2 [ Chen et al., 2020b ] further improves this framework with stronger data augmentations and architectural changes. To overcome the reliance on negative samples, several non-contrastive methods have been proposed. BYOL [ Grill et al., 2020 ] and SimSiam [ Chen and He, 2021 ] learn representations via asymmetric architectures, leveraging stop-gradient operations or momentum encoders to prevent representational collapse. Barlow Twins [ Zbontar et al., 2021 ] introduces a redundancy reduction objective that encourages decorrelated feature representations without requiring negative samples. VICReg [ Bardes et al., 2022 ] is an another regularization based method to avoid collapse. These methods are typically implemented using convolutional backbones such as ResNet. With the advent of Vision Transformers (ViTs), transformer-based SSL methods have gained attention. DINO [ Caron et al., 2021 ] adopts a self-distillation framework inspired by BYOL, while the Masked Autoencoder (MAE) [ He et al., 2022 ] reconstructs masked image patches in an autoencoder-like setup. I-JEPA [ Assran et al., 2023 ] generalizes these ideas by predicting contextual latent representations, bridging BYOL and MAE paradigms. Recently, there have been many attempts to learn from sequences of images (i.e., videos). One of the early works is TimeSformer [ Bertasius et al., 2021 ] , a supervised learning method for video understanding. VideoMAE [ Tong et al., 2022 ] introduced a self-supervised learning (SSL) approach using Transformer and masked autoencoding for video data. SiamMAE [ Gupta et al., 2023 ] incorporates a Siamese architecture to enhance feature learning, achieving significant improvements over VideoMAE. CropMAE [ Eymaël et al., 2024 ] further explores data augmentation