Towards an Explainable Comparison and Alignment of Feature Embeddings

Mohammad Jalali, Bahar Dibaei Nia, Farzan Farnia · 2025

Read Towards an Explainable Comparison and Alignment of Feature Embeddings by Mohammad Jalali, Bahar Dibaei Nia, Farzan Farnia (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-06231.txt.

Introduction

Mohammad Jalali, Bahar Dibaei Nia, and Farzan Farnia proposed SPEC in 2025, a framework to compare feature embeddings in a way people can explain. Embedding models turn data into vectors, and they are usually judged only by downstream scores. That hides where two models group the same samples in unlike ways, which is what a user often needs to know.

Their method builds kernel matrices from two embeddings and studies the spectrum of their difference. Clusters that one embedding captures and the other misses stand out in that spectrum. The implementation scales linearly with sample size, and an alignment step follows from the same machinery. Readers get a diagnostic lens for model choice: not just which embedding scores higher, but where the two actually disagree.

Catalog record

CREATORMohammad Jalali, Bahar Dibaei Nia, Farzan Farnia
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE69 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-06231

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Towards an Explainable Comparison and Alignment of Feature Embeddings Mohammad Jalali, Bahar Dibaei Nia, Farzan Farnia Abstract While several feature embedding models have been developed in the literature, comparisons of these embeddings have largely focused on their numerical performance in classification-related downstream applications. However, an interpretable comparison of different embeddings requires identifying and analyzing mismatches between sample groups clustered within the embedding spaces. In this work, we propose the Spectral Pairwise Embedding Comparison (SPEC) framework to compare embeddings and identify their differences in clustering a reference dataset. Our approach examines the kernel matrices derived from two embeddings and leverages the eigendecomposition of the difference kernel matrix to detect sample clusters that are captured differently by the two embeddings. We present a scalable implementation of this kernel-based approach, with computational complexity that grows linearly with the sample size. Furthermore, we introduce an optimization problem using this framework to align two embeddings, ensuring that clusters identified in one embedding are also captured in the other model. We provide numerical results demonstrating the SPEC’s application to compare and align embeddings on large-scale datasets such as ImageNet and MS-COCO. The project page is available at https://mjalali.github.io/SPEC/ . 1 Introduction Several mainstream frameworks in computer vision and natural language processing rely on embedding models to map raw image and text inputs into spaces with semantically meaningful features ( Radford et al. 2021 ; Liu et al. 2023 ; Alayrac et al. 2022 ; Li et al. 2023 ) . The application of pre-trained embeddings has enabled scalable solutions for many downstream tasks, particularly in scenarios where the available sample size and compute resources are significantly limited. In such cases, the embedded data can be used to train simple models, such as linear or k-nearest neighbors (KNN) classifiers, to achieve satisfactory results. Additionally, features extracted by standard embedding models are widely employed for the automated evaluation of generative models ( Heusel et al. 2017 ; Kynkäänniemi et al. 2023 ; Stein et al. 2023 ) , providing accurate rankings of generative modeling architectures without requiring time-intensive human assessments. While recent advancements in the machine learning community have introduced various embedding models that achieve remarkable results on standard image, text, and video domains, comparisons of these embeddings have primarily focused on evaluating their performance in standard downstream tasks, such as classification accuracy on benchmark datasets (e.g. ImageNet). However, such comparisons often lack interpretability and do not reveal how differently the embeddings behave in recognizing various sample types ( Boggust et al. 2022 ) . A more fine-grained comparison is necessary to disclose explainable differences between embedding models, particularly in identifying which samples are clustered differently according to the models. Understanding these differences can aid in interpreting and debugging embeddings and can also be leveraged to align multiple embeddings ( Simhi & Markovitch 2023 ; Dar et al. 2023 ) . Furthermore, interpreting the discrepancies between embeddings can be utilized to select representation models for downstream applications such as generative model evaluation ( Stein et al. 2023 ; Kynkäänniemi et al. 2023 ) . Figure 1: Overview of the Spectral Pairwise Embedding Comparison (SPEC) framework: The SPEC performs an eigendecomposition of the difference of kernel matrices following the two compared embeddings (e.g., DINOv2 and CLIP image embeddings) on a given reference dataset. Every eigenvector can be interpreted as a differently captured sample cluster by the embeddings, and the corresponding eigenvalue quantifies the difference between the cluster frequencies in the embedding spaces. In this work, we propose a spectral approach called Spectral Pairwise Embedding Comparison (SPEC) for the fine-grained comparison of two embeddings. The SPEC framework detects differences in sample clusters assigned by two embeddings, identifying major data groups that are clustered differently by one embedding compared to the other one. To achieve this, we adopt standard spectral clustering, which leverages the eigendecomposition of the kernel similarity matrix, and propose analyzing the principal eigenvectors of the difference of embeddings’ kernel matrices to interpret the differences in cluster assignments. Our analysis suggests that the SPEC framework can effectively detect cluster differences between two embeddings. To address the computational challenges of performing eigendecomposition on large-scale datasets, we develop a scalable implementation of SPEC. A direct eigendecomposition of the n\times n difference kernel matrix requires O(n^{3}) computations for a dataset with n samples, which is computationally expensive for large datasets. Assuming a bounded feature dimension d for the applied kernel function, we prove that the eigenspace of the difference kernel matrix can be computed using O(\max\{d^{3},n\}) operations, resulting in a scalable algorithm under a moderate dimension d value. Furthermore, we extend this scalable computation method to shift-invariant kernel functions, e.g. the Gaussian kernel, by employing the framework of random Fourier features (RFF) ( Rahimi & Recht 2007a ) , where the size of RFF proxy-feature map can be controlled for a more efficient application of SPEC. We also explore the application of the SPEC framework to define a distance measure between two embeddings. We define the \mathrm{SPEC} - \mathrm{diff} distance as the spectral radius of the kernel difference matrix, which aims to quantify the weight of the most differently captured cluster in one embedding that is not strongly clustered by the other model. We discuss scalable computations of this distance and its gradient with respect to the embedding parameters. Using the power method and the calculated left and right eigenvectors of the differential covariance matrix, we enable gradient-based optimization of the distance measure for aligning embedding models. This gradient-based approach leads to a method we call SPEC-align , aligning embedding models by minimizing their differences in clustering a reference dataset. SPEC-align is particularly useful for aligning cross-modality embeddings, such as CLIP ( Radford et al. 2021 ) , with a state-of-the-art single-modality embedding. Such a spectral alignment can improve the performance of cross-modality embeddings in capturing concepts specific to individual modalities. Finally, we present numerical experiments on several standard image and text embeddings using benchmark datasets. Our results demonstrate the scalability of the SPEC framework in revealing differences in sample clusters across embeddings over large-scale datasets. In our experiments, we tested the SPEC algorithm’s application with both cosine similarity and shift-invariant Gaussian kernels, where we leverage random Fourier features for the latter case. Additionally, we discuss the application of SPEC-align to align the CLIP model with single-modality embeddings. The empirical results highlight the effectiveness of SPEC-align in reducing the differences between CLIP’s image embeddings and specialized image-domain embeddings. The following is a summary of our work’s main contributions: • Proposing the SPEC framework for explainable comparison of two embeddings, • Providing a scalable SPEC implementation with linearly growing computational cost to the sample size, • Developing the gradient-based SPEC-align method to align two embeddings and matching their sample clusters, • Demonstrating the successful application of SPEC in comparing and aligning embeddings on benchmark datasets. 2 Related Work Spectral Clustering, Kernel PCA, and Random Fourier Features . Kernel PCA ( Schölkopf et al. 1998 ) is a widely recognized technique for dimensionality reduction that relies on the eigendecomposition of the kernel matrix. Several studies ( Bengio et al. 2003b ; Bengio et al. 2003a ) have explored the relationship between kernel PCA and spectral clustering. Also, the analysis of random Fourier features (

Read the complete text