Learning Mask Invariant Mutual Information for Masked Image Modeling
Tao Huang, Yanxiang Ma, Shan You, Chang Xu · 2025
Read Learning Mask Invariant Mutual Information for Masked Image Modeling by Tao Huang, Yanxiang Ma, Shan You, Chang Xu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-19718.txt.
Introduction
Tao Huang, Yanxiang Ma, Shan You, and Chang Xu gave a fresh account of why masked image models work. Their paper, from twenty twenty-five, deals with masked autoencoders. These models learn by hiding parts of an image and guessing them back. They score well, but why they work has stayed dim. Past reads lean on contrast or feature views that give hints, not plain truth.
The pair frame the trick through the information bottleneck idea. Their proofs show that the key is a balance: keep the facts that matter, drop the rest. From that they built MI-MAE, which tunes that balance through mutual information terms. Readers in vision get both a theory and a tweak for a core self taught model.
Catalog record
| CREATOR | Tao Huang, Yanxiang Ma, Shan You, Chang Xu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 55 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-19718
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Learning Mask Invariant Mutual Information for Masked Image Modeling Tao Huang, Yanxiang Ma, Shan You, Chang Xu Abstract Masked autoencoders (MAEs) represent a prominent self-supervised learning paradigm in computer vision. Despite their empirical success, the underlying mechanisms of MAEs remain insufficiently understood. Recent studies have attempted to elucidate the functioning of MAEs through contrastive learning and feature representation analysis, yet these approaches often provide only implicit insights. In this paper, we propose a new perspective for understanding MAEs by leveraging the information bottleneck principle in information theory. Our theoretical analyses reveal that optimizing the latent features to balance relevant and irrelevant information is key to improving MAE performance. Building upon our proofs, we introduce MI-MAE, a novel method that optimizes MAEs through mutual information maximization and minimization. By enhancing latent features to retain maximal relevant information between them and the output, and minimizing irrelevant information between them and the input, our approach achieves better performance. Extensive experiments on standard benchmarks show that MI-MAE significantly outperforms MAE models in tasks such as image classification, object detection, and semantic segmentation. Our findings validate the theoretical framework and highlight the practical advantages of applying the information bottleneck principle to MAEs, offering deeper insights for developing more powerful self-supervised learning models. 1 Introduction Masked autoencoders (MAEs) ( He et al., 2022 ; Xie et al., 2022 ; Bao et al., 2022 ) have emerged as a powerful self-supervised learning paradigm, particularly in the realm of computer vision. Inspired by the success of masked language models like BERT ( Devlin et al., 2019 ) in natural language processing, MAEs leverage a similar masking and reconstruction strategy to learn meaningful visual representations. The fundamental concept involves masking a portion of the input image and training a model to predict the missing parts, thereby enabling the model to capture the underlying structure and semantics of the visual data. This approach has proven effective in numerous applications ( Li et al., 2022b ; Kirillov et al., 2023 ; Tong et al., 2022 ; Fang et al., 2023b ) , showcasing the potential of MAEs to learn robust and generalizable features from unlabeled data. Despite their success, the understanding of how MAEs function and why they perform well remains an open question. Recent research has sought to demystify the inner workings of MAEs, providing valuable insights into their operation. Several studies have approached this task from various perspectives, including contrastive learning ( Zhang et al., 2022 ; Kong & Zhang, 2023 ; Huang et al., 2023 ) and feature representation analysis ( Xie et al., 2023 ; Pan et al., 2023 ) . Specifically, ( Kong & Zhang, 2023 ) proposed that MAEs inherently learn occlusion-invariant features by treating masked patches as a form of data augmentation. This approach, also suggested by ( Zhang et al., 2022 ; Yue et al., 2023 ) , aligns MAEs with contrastive learning frameworks, where the models learn to align features between different masked views of the same image. Other studies such as ( Xie et al., 2023 ; Pan et al., 2023 ) analysed the latent feature representations learned by MAEs to understand how these models capture and organize visual information. However, these efforts often provide only implicit insights and do not fully address the need for a comprehensive and systematic understanding of the learning objectives and framework of MAEs. In this paper, we propose a new perspective for understanding MAEs by leveraging information theory, specifically the information bottleneck (IB) principle ( Tishby & Zaslavsky, 2015 ) . This perspective provides a systematic and comprehensive framework for optimizing MAEs, offering theoretical insights that can guide the development of more effective models. The IB principle posits that any deep neural network can be understood as a system that balances the trade-off between retaining relevant information and compressing irrelevant information. By applying this principle to MAEs, we aim to provide a more robust understanding of their mechanisms and to identify key areas for improvement. Based on our findings, we introduce a novel masked image modeling method, dubbed MI-MAE, to learn Mask Invariant Mutual Information for MAEs through the lens of the information bottleneck theory. Our method systematically optimizes the latent features produced by the encoder, ensuring that they contain maximal relevant information and minimal irrelevant information on the information bottleneck of MAE. Concretely, we introduce two aspects of mutual information based losses on the latent feature: (1) Mutual information maximization. To optimize the autoencoder in the latent space, we derive a loss to maximize the mutual information between the latent features of multiple orthogonal masks 1 1 1 Here, “orthogonal” means the inner productions between each mask are 0, which means we are completely dividing the image into visible parts of multiple masks. . (2) Mutual information minimization. We optimize an upper bound of mutual information between the input and latent space to minimize the irrelevant information in the latent features and thus maximize the capacity of relevant information. This comprehensive optimization strategy helps in achieving better feature representations and improved performance. We conduct a series of evaluations on standard benchmarks, showing that our method performs significant improvements over MAE in various tasks, including image classification, object detection, and semantic segmentation. For example, our 400-epoch model achieves 83.9\% accuracy on ImageNet-1K, surpassing the 1600 -epoch MAE by 0.5%. The experimental results validate our theoretical findings and highlight the practical benefits of applying the information bottleneck principle to masked autoencoders. Additionally, by providing a new perspective and a rigorous analytical framework, our work paves the way for future research in this area, offering insights that can drive the development of even more powerful self-supervised learning models. 2 Related Works Contrastive learning. Contrastive learning ( Chen et al., 2020 ; He et al., 2020 ; Chen & He, 2021 ; Grill et al., 2020 ; Caron et al., 2021 ) stands out as the leading self-supervised representation learning approach in computer vision, achieving invariance by comparing different augmentations of the same image. A notable example is SimCLR ( Chen et al., 2020 ) , which enhances semantic representations by increasing the similarity between various views of the same image in the latent space. MoCo v3 ( Chen et al., 2021 ) applies contrastive learning techniques to pre-train vision transformers. DINO ( Caron et al., 2021 ) delves into novel properties of self-supervised vision transformers. Masked image modeling. Masked image modeling (MIM) has gained significant traction in the field of computer vision as an effective self-supervised learning paradigm. Recently, with the widespread use of vision transformers (ViTs) ( Dosovitskiy et al., 2021 ; Liu et al., 2021 ) , a series of notable methods such as BEiT ( Bao et al., 2022 ) , MAE ( He et al., 2022 ) , and SimMIM ( Xie et al., 2022 ) have been proposed to pre-train ViTs following the BERT-style masked modeling paradigm used in natural language processing (NLP) ( Devlin et al., 2019 ; Liu et al., 2019 ) . Many follow-up works extend masked pre-training by exploring data augmentations ( Chen et al., 2023 ; Fang et al., 2023a ) , mask strategies ( Li et al., 2022a ; Wang et al., 2023 ; Wang et