Disentangling Latent Embeddings with Sparse Linear Concept Subspaces (SLiCS)

Zhi Li, Hau Phan, Matthew Emigh, Austin J. Brockmeier · 2025

Read Disentangling Latent Embeddings with Sparse Linear Concept Subspaces (SLiCS) by Zhi Li, Hau Phan, Matthew Emigh, Austin J. Brockmeier (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-20322.txt.

Introduction

Li, Phan, Emigh, and Brockmeier split in twenty twenty-five the inner codes of pretrained vision models into concept parts a user can inspect and recombine. A scene's code mixes many ideas at once, which blurs search and editing. Their method learns a dictionary where each concept owns a cone of building blocks, and each image code becomes a sparse, non-negative mix of a few concepts, steered by labels.

Training alternates between fitting codes and updating the dictionary, with an autoencoder variant too. On several embedding types and datasets, concept-filtered search beat raw search, even finding fine details never labelled in training. Parts could be deleted or swapped between images, and fed into prompt-based generation. A label-free version also worked from neuron activity alone.

Catalog record

CREATORZhi Li, Hau Phan, Matthew Emigh, Austin J. Brockmeier
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE66 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-20322

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Disentangling Latent Embeddings with Sparse Linear Concept Subspaces (SLiCS) Zhi Li, Hau Phan, Matthew Emigh, Austin J. Brockmeier Abstract Vision-language co-embedding networks, such as CLIP, provide a latent embedding space with semantic information that is useful for downstream tasks. We hypothesize that the embedding space can be disentangled to separate the information on the content of complex scenes by decomposing the embedding into multiple concept-specific component vectors that lie in different subspaces. We propose a supervised dictionary learning approach to estimate a linear synthesis model consisting of sparse, non-negative combinations of groups of vectors in the dictionary (atoms), whose group-wise activity matches the multi-label information. Each concept-specific component is a non-negative combination of atoms associated to a label. The group-structured dictionary is optimized through a novel alternating optimization with guaranteed convergence. Exploiting the text co-embeddings, we detail how semantically meaningful descriptions can be found based on text embeddings of words best approximated by a concept’s group of atoms, and unsupervised dictionary learning can exploit zero-shot classification of training set images using the text embeddings of concept labels to provide instance-wise multi-labels. We show that the disentangled embeddings provided by our sparse linear concept subspaces (SLiCS) enable concept-filtered image retrieval (and conditional generation using image-to-prompt) that is more precise. We also apply SLiCS to highly-compressed autoencoder embeddings from TiTok and the latent embedding from self-supervised DINOv2. Quantitative and qualitative results highlight the improved precision of the concept-filtered image retrieval for all embeddings. 1 Introduction Deep vision-language models trained on large datasets capture both visual and semantic features from the original inputs, encoding them into a dense vector space to generate image and text embeddings. These models are widely applied in tasks such as cross-modal retrieval and conditional generative modeling. In particular, the pre-trained CLIP model (Contrastive Language-Image Pre-Training) ( Radford et al. 2021 ) uses separate image and text encoders to embed inputs into the same vector space using contrastive learning ( Poole et al. 2019 ; Gutmann and Hyvärinen 2010 ) . Even though CLIP embeddings are semantically rich, the dimensions of the dense vectors are not directly interpretable. For complex scene imagery, the ability to decompose an embedding vector into components corresponding to interpretable concepts would enable concept-filtered retrieval and conditional generation. Towards this goal, we propose Sparse Linear Concept Subspaces (SLiCS) to disentangle any given embedding vector into a set of component vectors, each a non-negative combination of basis vectors (atoms) defining a subspace for a coherent concept. The disentangled components provide an understanding of the distinct concepts present in an image. That is, each component subspace attempts to isolate and capture the possible variation of its associated concept within the embedding space. In order to associate a human-interpretable concept to each group, we exploit the fact that the image and text embedding spaces are aligned; we select descriptive words for each concept by using the word embeddings that are well-reconstructed by the concept’s atoms. We assume that a deep image embedding encompasses the entire scene, densely preserving information about every object and background element. In retrieval, this holistic representation of the query image may not be ideal when we are interested in matching only a subset of aspects of the query image, as irrelevant aspects may adversely affect the retrieval. Suppose a query image contains multiple distinct concepts; then the embedding can be decomposed so that each concept-labeled component can be used to retrieve images similar to the aspects of the query image that are contained within that concept. For example, consider a complex query image containing a dog; to retrieve other images containing similar dogs, the “animal” component of the image embedding can be used to query the image database. Thus, concept-filtered retrieval, where the user can define which concepts should be focused on, has the potential to make image retrieval more flexible and precise. An overview of our proposed concept-filtered framework is presented in Fig. 1 . Figure 1: The pipeline of SLiCS-aided concept-filtered image retrieval. SLiCS decomposes a query image embedding \mathbf{x}_{\star} into multiple components \mathbf{v}_{1}^{\star},\mathbf{v}_{2}^{\star},\ldots , one for each concept subspace \mathcal{V}_{1},\mathcal{V}_{2},\ldots , such that \mathbf{x}_{\star}\approx\mathbf{v}_{1}^{\star}+\mathbf{v}_{2}^{\star}+\cdots , where the j th component \mathbf{v}_{j}^{\star}\in\mathcal{V}_{j} is a non-negative combination of the M_{j} atoms \mathbf{b}_{j,1},\ldots,\mathbf{b}_{j,M_{j}} associated to the j th concept. Image retrieval can be performed on each disentangled component to retrieve the most similar concept-specific images. Given a concept-labeled image dataset, SLiCS can learn a subspace for each concept. For vision-language models, the labels can be obtained from zero-shot classification of text embeddings of concepts. This provides for an unsupervised version of SLiCS. However, SLiCS does not rely on the presence of text embeddings for supervised training. Deep vision embedding models trained using self-supervised objectives ( Chen et al. 2020b ; Chen et al. 2020a ; Chen et al. 2021 ; Caron et al. 2020 ; Oquab et al. 2023 ; Yu et al. 2024 ) and high-quality auto-encoders ( Van Den Oord et al. 2017 ; Esser et al. 2021 ) with more compressed embeddings ( Yu et al. 2024 ) also provide semantically rich embeddings, and can benefit from SLiCS-based disentangling for more precise retrieval. In summary, we propose SLiCS as an approach for learning to disentangle dense embeddings to enable concept-filtered retrieval and image generation in cases where the embedding can be directly decoded or used as a conditioning prompt. 2 Related Work The emergence of CLIP has enabled various types of tasks involving the embedding of images and texts. Gandelsman et al. 2024 explore the interpretability of the intermediate representations in CLIP, breaking the vision transformer image encoder into patches and attention heads, trying to find the text description for each attention head. However, it does not directly handle decomposition. Other methods implement decomposition with the help of text embeddings. Yuksekgonul et al. 2023 integrates the idea of subspaces into decomposition, although each concept is only spanned by one-word CLIP text embeddings. The projection of an image embedding onto a set of concepts gives a magnitude vector that can be further used in supervised classification. Similarly, SpLiCE ( Bhalla et al. 2024 ) , decomposes an image embedding on a set of fixed word embeddings with sparse coding. It operates in an unsupervised fashion, but the lack of subspaces construction limits the interpretation to instance level. Furthermore, both these methods are limited because of their reliance on the selection of word tokens. More similar to our approach, Kobs et al. 2023 use PCA to create a subspace with the guidance of a set of word embeddings that describe the same concept or aspect. Compared with these methods, SLiCS operates solely on image embeddings, only using text embeddings to construct pseudo-labels to aid the decomposition. CLIP has inspired a number of image retrieval methods using text ( Chaudhary et al. 2020 ) or multi-modal ( Baldrati et al. 2022 ; Couairon et al. 2022 ; Liu et al. 2021 ) queries. Multi-modal approaches have used a combination of image and text to retrieve images similar to the contents of a query image modified by the provided query text. However, exploiting the CLIP image embeddings to perform concept-filtered retrieval remains unexplored. More generally, our proposed SLiCS approach is a particular form of constrained

Read the complete text