SparseJEPA: Sparse Representation Learning of Joint Embedding Predictive Architectures
Max Hartman, Lav R. Varshney · 2025
Read SparseJEPA: Sparse Representation Learning of Joint Embedding Predictive Architectures by Max Hartman, Lav R. Varshney (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-16140.txt.
Introduction
Max Hartman and Lav Varshney make self-taught image models sparser and clearer in this 2025 paper. The models, joint embedding predictive architectures, learn by guessing hidden parts of their own input codes. Their dense codes are hard to read and wasteful.
Their variant, SparseJEPA, adds a penalty that pushes code units to be shared by features with strong meaning ties. Prediction quality stays while the code thins out. They trained it on a hundred-class image set with a light vision transformer. Thinned codes then transferred well to new tasks with simple probes. The work aims for representations people can interpret, not just use.
Catalog record
| CREATOR | Max Hartman, Lav R. Varshney |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 19 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-16140
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SparseJEPA: Sparse Representation Learning of Joint Embedding Predictive Architectures Max Hartman, Lav R. Varshney Abstract Joint Embedding Predictive Architectures (JEPA) have emerged as a powerful framework for learning general-purpose representations. However, these models often lack interpretability and suffer from inefficiencies due to dense embedding representations. We propose SparseJEPA, an extension that integrates sparse representation learning into the JEPA framework to enhance the quality of learned representations. SparseJEPA employs a penalty method that encourages latent space variables to be shared among data features with strong semantic relationships, while maintaining predictive performance. We demonstrate the effectiveness of SparseJEPA by training on the CIFAR-100 dataset and pre-training a lightweight Vision Transformer. The improved embeddings are utilized in linear-probe transfer learning for both image classification and low-level tasks, showcasing the architecture’s versatility across different transfer tasks. Furthermore, we provide a theoretical proof that demonstrates that the grouping mechanism enhances representation quality. This was done by displaying that grouping reduces Multiinformation among latent-variables, including proofing the Data Processing Inequality for Multiinformation. Our results indicate that incorporating sparsity not only refines the latent space but also facilitates the learning of more meaningful and interpretable representations. In further work, hope to further extend this method by finding new ways to leverage the grouping mechanism through object-centric representation learning. Keywords: Representation Learning, Joint Embedding Predictive Architecture 1 Introduction Joint Embedding Predictive Architectures (JEPA) seek to learn comprehensive representations by predicting masked target blocks provided a single context block ( Assran et al. 2023 ) . These architectures learn such representations by making predictions in the latent-space of the data they seek to reconstruct. In the case of images, for example, the context block and target blocks are embedded via a Vision Transformer. These models have seen great results in Computer Vision downstream tasks such as Image Classification, Object Counting, and Depth Perception, compared to similar Self-Supervised Architectures like Masked Autoencoders (MAE) ( He et al. 2021 ) , data2vec ( Baevski et al. 2022 ) , and Context Auto Encoders (CAE) ( Chen et al. 2023 ) Despite the success of these models to learn abstract representations, they rely on dense embeddings. This high density also poses a challenge from an interpretability standpoint. For future work on learning representations for World Knowledge, understanding how representations group and store knowledge will be critical for further downstream tasks across Robotics and Computer Vision. Despite their effectiveness, the dense embedding representations in JEPA act as ”black boxes,” making it difficult to understand the factors driving their predictions. Sparse representation learning provides an opportunity to make these embeddings more interpretable, yielding insights into the underlying structure of the data. Sparse representation learning offers a transformative approach to addressing inefficiencies and interpretability challenges in JEPA. By introducing sparsity into the latent space, models can focus on the most relevant features, enabling them to learn meaningful representations that uncover intrinsic structures within the data. Sparse representations go beyond mere computational benefits by naturally aligning the latent space with meaningful patterns in the data. In the case of SparseJEPA, our results demonstrate that sparsity helps the model learn a latent space that captures specific groupings of shared strong semantic relationships across input images. This grouping reflects the underlying structure of the data, allowing embeddings to emphasize regions or features that share semantic or informational similarities. For example, in visual datasets, sparsity can lead to embeddings that cluster objects or textures based on shared attributes, revealing deeper insights into the data distribution. This alignment of sparse embeddings with shared semantic relationships enhances both the interpretability and utility of the model In the broader field of machine learning, techniques such as L_{1} -regularization, structured sparsity, and group sparsity have been instrumental in promoting sparse solutions. Output-Interpretable Variational Autoencoders have been seen to enhance latent-space representations, while maintaining model performance ( Ainsworth et al. 2018 ) . L_{1} -regularization penalizes the magnitude of latent variables, effectively shrinking less relevant features to zero, while structured sparsity encourages hierarchical or grouped patterns in embeddings. These methods have demonstrated success in producing compact and interpretable representations. However, their integration into JEPA frameworks remains relatively unexplored, representing a key opportunity to leverage these advancements in a joint embedding context. This work proposes an architecture extending JEPA by including a sparsity loss penalty to enhance interpretability and efficiency in the learned latent representations. The sparsity loss penalty employed is adapted from oi-VAE: Output Interpretable VAEs for Nonlinear Group Factor Analysis. This loss encourages the latent space to reflect groupings of shared semantic relationships by penalizing redundant or uninformative latent variables, promoting a structured and interpretable embedding space. By integrating this loss into the JEPA framework, the proposed architecture not only improves computational efficiency but also aligns the latent space with meaningful data patterns, enabling better insights and downstream utility. 2 Background 2.1 Joint Embedding Predictive Architecture (JEPA) JEPA is a self-supervised learning framework designed to learn data representations. JEPA learns a shared embedding space by predicting the embedding of target patches (masked regions of an image) using the embeddings of context patches (visible regions), encouraging the model to infer relationships and semantic structures within the input data. This distinguishes JEPA from traditional reconstruction-based methods, as it directly learns from the latent space rather than raw pixel-level details ( Assran et al. 2023 ) . 2.2 Output Interpretable Variational Autoencoders (oi-VAE) oi-VAE ( Ainsworth et al. 2018 ) is a framework designed to achieve interpretable and disentangled latent representations for nonlinear group factor analysis. It extends traditional variational autoencoders (VAEs) by grouping associating latent variables with specific components of the input data. This is achieved by decomposing the data into distinct outputs, each corresponding to a separate group of latent variables. The oi-VAE assumes that the observed data can be factorized into components, where each component is generated from a distinct latent variable. These latent variables are disentangled into group-specific and shared (global) factors, enabling the model to capture both unique and common structures in the data. By linking latent variables directly to data components, the oi-VAE achieves interpretability, allowing each latent dimension to correspond to a specific aspect of the data. 2.3 Sparse Autoencoders Sparse Autoencoders are a type of neural network designed to learn efficient and meaningful representations of data by introducing a sparsity constraint in the hidden layer. This constraint encourages the network to activate only a small number of neurons in response to an input, resulting in compact and interpretable representations. Sparse Autoencoders aim to learn a representation where only a small subset of hidden units is active for any given input. This sparsity is typically enforced through a regularization term in the loss function, such as the KL divergence between the average activation of the hidden units and a desired sparsity target, or via L_{1} -regularization on the hidden activations ( Le et al. 2012 ) . 3 Sparse grouping of latent variables reduces redundant multiinformation In this section, we build up to a theorem which states that the grouping mechanism of SparseJEPA and oi-VAE enhances learned representations by reducing multiinformation among the data. We follow Raman et al. 2017 and Studený and Vejnarová 1998 by defining multiiformation as: I(X_{1};\ldots;X_{n})=D_{\mathrm{KL}}\bigl(p(x_{1},\ldots,x_{n})\,\big\|\,\prod_{i=1}^{n}p(x_{i})\bigr). where we let X_{1},\ldots,X_{n} be random variables with joint