AKRMap : Adaptive Kernel Regression for Trustworthy Visualization of Cross-Modal Embeddings

Yilin Ye, Junchao Huang, Xingchen Zeng, Jiazhi Xia, Wei Zeng · 2025

Read AKRMap : Adaptive Kernel Regression for Trustworthy Visualization of Cross-Modal Embeddings by Yilin Ye, Junchao Huang, Xingchen Zeng, Jiazhi Xia, Wei Zeng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-14664.txt.

Introduction

Yilin Ye and co-authors build a map-maker in 2025 for the joint number spaces where AI links text and images. Standard map tools flatten such spaces by looking at one kind of data at a time, ignoring how well the two kinds match by shared scores. Their tool learns the score landscape itself as it flattens.

It trains a small network to place points so that a smooth fit over the flat map matches true match scores, with kernels that adapt during training, they report. Zoom and overlay features support close study. In tests it beat classic flatteners on accuracy and trust measures. Judging a map by the match scores it preserves gives a truer picture of how a two-kind model really sorts the world.

Catalog record

CREATORYilin Ye, Junchao Huang, Xingchen Zeng, Jiazhi Xia, Wei Zeng
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-14664

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

AKRMap : Adaptive Kernel Regression for Trustworthy Visualization of Cross-Modal Embeddings Yilin Ye, Junchao Huang, Xingchen Zeng, Jiazhi Xia, Wei Zeng Abstract Cross-modal embeddings form the foundation for multi-modal models. However, visualization methods for interpreting cross-modal embeddings have been primarily confined to traditional dimensionality reduction (DR) techniques like PCA and t-SNE. These DR methods primarily focus on feature distributions within a single modality, whilst failing to incorporate metrics (e.g., CLIPScore) across multiple modalities. This paper introduces AKRMap , a new DR technique designed to visualize cross-modal embeddings metric with enhanced accuracy by learning kernel regression of the metric landscape in the projection space. Specifically, AKRMap constructs a supervised projection network guided by a post-projection kernel regression loss, and employs adaptive generalized kernels that can be jointly optimized with the projection. This approach enables AKRMap to efficiently generate visualizations that capture complex metric distributions, while also supporting interactive features such as zoom and overlay for deeper exploration. Quantitative experiments demonstrate that AKRMap outperforms existing DR methods in generating more accurate and trustworthy visualizations. We further showcase the effectiveness of AKRMap in visualizing and comparing cross-modal embeddings for text-to-image models. Code and demo are available at https://github.com/yilinye/AKRMap . 1 Introduction Cross-modal embeddings play a fundamental role for multi-modal models, functioning as cross-modal encoders ( Rombach et al., 2022 ) , objective functions ( Ramesh et al., 2022 ) , or evaluation metrics ( Hessel et al., 2021 ; Wu et al., 2023a ) for tasks like text-to-image (T2I) generation. To assess the alignment of multi-modal models, various evaluation metrics based on these embeddings have been introduced, such as CLIPScore ( Hessel et al., 2021 ) and Human Preference Score (HPS) ( Wu et al., 2023a ; Wu et al., 2023b ; Zhang et al., 2024 ) . Despite their utility, embedding-based evaluations are often difficult to interpret, as the metrics are typically reported as aggregated values, without providing insights into instance-level performance. This deficiency undermines the trustworthiness and transparency of the evaluation. To address this limitation, dimensionality reduction (DR) visualization serves as a crucial tool, offering a way to reveal the landscape of cross-modal metrics and enabling a more comprehensive understanding of model performance. Recent studies ( Liang et al., 2022 ; Wang et al., 2023b ; Wang et al., 2023c ) have explored the use of DR methods for cross-modal embeddings. These efforts often leverage established techniques such as PCA ( Li et al., 2018 ) , UMAP ( McInnes et al., 2018 ) and t-SNE ( Van der Maaten & Hinton, 2008 ) , as well as autoencoder-based approaches ( Le et al., 2018 ; Elhamod & Karpatne, 2024 ) . However, existing DR methods are primarily designed to depict feature distributions within a single modality. When applied to cross-modal metrics, these methods often produce dense neighborhoods where points with significantly different metric values are positioned close, leading to overlap and local occlusion (Figure 1 (b)). Moreover, multi-modal models are typically evaluated on large-scale datasets containing millions of data points. This creates a need for rendering contour maps that can reveal continuous metric distributions. However, local neighboring points with a mix of high and low metric values may be misrepresented, as the contour map depicts only a single aggregated value, causing inaccurate contour mapping as illustrated in Figure 1 (c). To construct trustworthy visualizations for cross-modal embeddings, a key consideration is to enhance the accuracy of metric contour mapping in the projected 2D space. As contour estimation typically relies on radial basis function (RBF) kernel (e.g., Gaussian kernel), we seek to enhance the kernel-based mapping by coordinating it with the DR process. Drawing inspiration from supervised t-SNE ( Hajderanj et al., 2019 ) , we introduce Adaptive Kernel Regression Map ( AKRMap ), a supervised DR method that leverages adaptive kernel regression to effectively visualize the distribution of cross-modal embedding metrics. Specifically, AKRMap first constructs a supervised DR network explicitly guided by post-projection kernel regression loss with neighborhood constraint (Sect. 4.1.1 ). Next, to account for the gap between high-dimensional and low dimensional kernels, we improve the flexibility of the post-projection kernel through adaptive generalized kernel jointly learned with the projection (Sect. 4.1.2 ). This approach enables the generation of scatterplots for discrete data points and contour maps for continuous metric landscapes, while also supporting advanced features such as overlay views and zooming for interactive interpretation (Sect. 4.2 ). Both the DR method and the visualization tool have been implemented as a Python package and made publicly available. Figure 1: CLIPScore distribution on the COCO dataset by t-SNE (a). The visualization shows dense neighboring points with significantly different metric values, causing overlapping and occlusion (b) and highly inaccurate contour mapping (c). We conduct quantitative experiments to evaluate AKRMap against traditional and autoencoder-based DR visualizations (Sect. 5.1 ). The results highlight the superior performance of AKRMap in accurately mapping cross-modal metrics for both in-sample and out-of-sample data points. We further demonstrate the practical applications of AKRMap across three distinct scenarios (Sect. 5.2 ): 1) visual exploration of human preference dataset (HPD) ( Wu et al., 2023a ) , 2) visual comparison of diffusion-based and auto-regressive T2I models, and 3) visual examination of the global impact for fine-tuning. These applications illustrate that AKRMap effectively enables human-in-the-loop interpretation of model performance on large-scale datasets. Our method can also be potentially generalized to other modalities such as text-to-video task, with additional quantitative experiments shown in Appendix F . In summary, our contributions are as follows: • We propose a novel DR method for cross-modal metric visualization, which jointly learns the projection and metric contour mapping through kernel regression supervised DR with adaptive generalized kernel. • We develop a tool for trustworthy visualization of cross-modal metrics, incorporating visualization features such as a scatterplot view and a contour map, along with interactive features like zooming and overlaying. • We conduct quantitative experiments to demonstrate the superior performance of AKRMap in generating more accurate visualizations of cross-modal metric, and highlight its applications across three scenarios to enhance the trustworthiness of T2I model evaluation. 2 Related Work 2.1 Cross-Modal Embedding-based Evaluation Cross-modal embeddings like CLIP ( Radford et al., 2021 ) and ALIGN ( Jia et al., 2021 ) form the foundation of multi-modal learning. Specifically, many evaluation methods for multi-modal models, such as T2I models, rely on these embeddings. Conventional metrics like Inception Score ( Salimans et al., 2016 ) and Fréchet Inception Distance ( Heusel et al., 2017 ) compute the average distance or distributional difference between generated and reference images in the embedding space, yet they are insufficient to measure cross-modal alignment and instance-level performance. With the advancement of multi-modal AI, cross-modal embedding metrics like CLIPScore ( Hessel et al., 2021 ) have emerged to measure the alignment between prompts and generated images. While being effective in capturing the alignment, CLIPScore fails to model human preferences. To address this limitation, recent studies have proposed training specialized human preference models such as HPS ( Wu et al., 2023a ; Wu et al., 2023b ; Zhang et al., 2024 ) , PickScore ( Kirstain et al., 2023 ) , and ImageReward ( Xu et al., 2024 ) , to better align with human judgments. For example, ImageReward

Read the complete text