Fusing Cross-modal and Uni-modal Representations: A Kronecker Product Approach
Youqi Wu, Jingwei Zhang, Farzan Farnia · 2025
Read Fusing Cross-modal and Uni-modal Representations: A Kronecker Product Approach by Youqi Wu, Jingwei Zhang, Farzan Farnia (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-08645.txt.
Introduction
Youqi Wu, Jingwei Zhang, and Farzan Farnia fuse in a 2025 study rival embedding models by multiplying their similarity scores, letting each model's strengths cover the other's gaps. One image model may read texture finely, another grasps object shape. Multiplying the two similarity maps yields a fused space whose clusters unite both views.
The math runs through the Kronecker product of the feature maps, a huge space they tame with random projections in a variant called RP-KrossFuse. Fusing joint image-text models with single-mode experts lifted each side's skill while keeping cross-modal alignment. Borrowed strengths, the study shows, beat lone models.
Catalog record
| CREATOR | Youqi Wu, Jingwei Zhang, Farzan Farnia |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 81 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-08645
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Fusing Cross-modal and Uni-modal Representations: A Kronecker Product Approach Youqi Wu, Jingwei Zhang, Farzan Farnia Abstract Cross-modal embeddings, such as CLIP, BLIP and their variants, have achieved promising results in aligning representations across modalities. However, these embeddings could underperform compared to state-of-the-art single-modality embeddings on modality-specific tasks. On the other hand, single-modality embeddings excel in their domains but lack cross-modal alignment capabilities. In this work, we focus on the problem of unifying cross-modality and single-modality embeddings to achieve the performance of modality-expert embedding within individual modalities while preserving cross-modal alignment. To this end, we propose RP-KrossFuse , a method that leverages a random projection-based Kronecker product to integrate cross-modal embeddings with single-modality embeddings. RP-KrossFuse aims to fuse the sample-pairwise similarity scores of the fused embeddings and operates efficiently in a specified kernel space and supports scalable implementations via random Fourier features for shift-invariant kernels such as the Gaussian kernel. We demonstrate the effectiveness of RP-KrossFuse through several numerical experiments, combining CLIP embeddings with uni-modal image and text embeddings. Our numerical results indicate that RP-KrossFuse achieves competitive modality-specific performance while retaining cross-modal alignment, bridging the gap between cross-modal and single-modality embeddings. 1 Introduction The emergence of cross-modal embeddings has significantly advanced the scope of multi-modal deep learning frameworks, enabling models to bridge the gap between different data modalities, particularly text and images. Cross-modal models, including CLIP [ 1 ] , ALIGN [ 2 ] and BLIP [ 3 ] , have demonstrated remarkable results in cross-modal tasks including zero-shot image classification, image-text retrieval, and image captioning. Their success stems from their ability to learn joint representations in which corresponding text and image pairs are aligned in the embedding space. This capability to align modalities has been applied to numerous tasks in machine learning [ 4 , 5 , 6 , 7 ] . Despite their strengths in multi-modal tasks, the cross-modal embedding models often do not match the performance of state-of-the-art uni-modal embeddings within individual modalities. For example, while CLIP achieves promising zero-shot classification results as a benchmark cross-modality task, it is outperformed by specialized image embedding models such as DINOv2 [ 8 ] on image-specific classification benchmarks, e.g. ImageNet classification accuracy with simple linear or KNN models. Similarly, while CLIP’s text encoder is effective in distinguishing different text data, it does not achieve the same level of performance as state-of-the-art text embedding models such as E5 [ 9 ] and DeBERTaV3 [ 10 ] on several text dataset benchmarks. These comparisons suggest a trade-off between optimizing for cross-modal alignment and maximizing single-modality performance. Figure 1 : Heatmaps of RBF kernel similarity matrices for an image dataset with four groundtruth clusters (two dog classes in ImageNet and two traffic sign classes in GTSRB) (Left) K_{1} for CLIP, (Middle) K_{2} for DINOv2, (Right) K_{1}\odot K_{2} elementwise product for CLIP and DINOv2’s Kroncker product. Unlike CLIP and DINOv2, their Kronecker product could cluster the four image classes. The above discussion raises the following question: is it possible to combine the strengths of cross-modal and uni-modal approaches, forming embeddings that perform nearly optimally in individual modalities while maintaining cross-modal alignment? In this work, our goal is to address this question and characterize a joint embedding space that captures the fine-grained details necessary for state-of-the-art performance within each modality, similar to specialized uni-modal models, while it can preserve the ability to align information across different modalities, which is offered by cross-modal models like CLIP. Our approach to reach the above goal is to fuse the uni-modal and cross-modal embeddings. By combining the representation captured by the embedding maps, we can arrive at a unified representation that simultaneously achieves modality alignment and nearly optimal performance scores within each modality. Therefore, such a fusion approach can lead to an embedding space that benefits from both the cross-modal alignment of the cross-modal encoders, such as CLIP, and the modality-specific expertise of uni-modal models, such as DINOv2 and E5 for image and text modalities, respectively. In the fusion process, we propose a kernel-based approach and, within each modality, we require the kernel similarity score of the fused embedding to capture the differences between two samples, as long as one of the embeddings deems the two samples dissimilar. Mathematically, given the kernel similarity scores k_{\psi_{1}}(x,y) and k_{\psi_{2}}(x,y) for data points x,y and embedding maps \psi_{1},\psi_{2} , we require the fused embedding \psi_{\text{fused}} to assign a low similarity score k_{\psi_{\text{fuse}}}(x,y) if either of k_{\psi_{1}}(x,y) and k_{\psi_{2}}(x,y) assigns a low simialrty score. This property allows a fine-grained spectral clustering of the data that captures the differences of kernel-based clustering of both of the embeddings. To achieve the above kernel similarity score property, we propose the fused kernel similarity score to be the product of the kernel function of the embedding maps, i.e. we require k_{\psi_{\text{fuse}}}(x,y)\,=\,k_{\psi_{1}}(x,y)\times k_{\psi_{2}}(x,y) (1) Since the standard cosine similarity and Gaussian (RBF) kernels take values bounded by 1 , the product kernel will assign a closer to zero similarity score compared to either of the embeddings \psi_{1} and \psi_{2} . This property is illustrated in Figure 1 where the product of the image embeddings of CLIP and DINOv2 inherits the block structure of each individual kernel matrix, leading to separating the four underlying clusters of the dataset that could not be fully separated by either of the CLIP and DINOv2 embeddings. Proposition 1 highlights that the embedding to achieve this property follows the Kronecker product of the two embeddings \psi_{\text{fuse}}(x):=\psi_{1}(x)\otimes\psi_{2}(x) when both the embedding maps are existing for the modality. However, the application of the Kronecker product approach to fuse cross-modal and uni-modal embeddings requires addressing the missing embedding map for the non-shared modality in the uni-modal embedding. For example, in fusing cross-modal CLIP and uni-modal DINOv2, the text embedding is missing in DINOv2, hindering the Kronecker product method. We extend the Kronecker-product-based fusion to this case and propose the fusion method KrossFuse for a Kronecker-product-based integration of a cross-modal embedding with a unimodal embedding as exhibited in Figure 2 . In the KrossFuse fusion, the symmaetrized non-shared embedding map is Kronecker multiplied by the identity map, which, as we show, maintains the assigned similarity scores between the shared and non-shared modalities unchanged while fusing the embedding maps of the shared modality. While the KrossFuse method addresses the challenge of the missing modality of the uni-modal embedding, the Kronecker product in the method leads to a significant computational cost. Note that the dimensionality of the Kronecker fused embedding will be the product of the dimensions of the individual embeddings. For example, combining CLIP embedding (512-dimensional) with DINOv2 embedding (768-dimensional) will result in a Kross-fuse map with 2\times 512\times 768= 393,216 dimensions. Such high dimensionality would make the KrossFuse approach impractical for large-scale datasets where the dimensions’ product could be excessively large. To overcome this challenge, we introduce the Random Projection KrossFuse ( RP-KrossFuse ) method. By a well-designed application of random projections to the Kronecker product of the embedding vectors, we show that RP-KrossFuse can effectively reduce the dimensionality of the fused representation while preserving the kernel similarity values between the KrossFuse maps. As a result, RP-KrossFuse can provide a scalable embedding unification approach for cross-modal and uni-modal embedding maps. Figure 2 : The Kronecker product-based fusion of embeddings