A TRIANGLE Enables Multimodal Alignment Beyond Cosine Similarity

Giordano Cicchetti, Eleonora Grassucci, Danilo Comminiello · 2025

Read A TRIANGLE Enables Multimodal Alignment Beyond Cosine Similarity by Giordano Cicchetti, Eleonora Grassucci, Danilo Comminiello (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-24734.txt.

Introduction

Matching text, image, and sound in one shared space works better with a triangle than with pairwise scores, Giordano Cicchetti, Eleonora Grassucci, and Danilo Comminiello argue in a 2025 paper. Standard multimodal models align pairs with cosine scores, which can leave one mode stranded. Their measure rates how well three embeddings form a tight triangle in high-dimensional space.

Swapped into contrastive training losses, the triangle score lifts joint alignment across all three modes at once. No extra fusion layers are needed. Systems that search across media, from video captions to voice-tagged photos, depend on every mode pulling together, and a three-way check catches drift that pairwise scores miss.

Catalog record

CREATORGiordano Cicchetti, Eleonora Grassucci, Danilo Comminiello
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-24734

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A TRIANGLE Enables Multimodal Alignment Beyond Cosine Similarity Giordano Cicchetti, Eleonora Grassucci, Danilo Comminiello Abstract Multimodal learning plays a pivotal role in advancing artificial intelligence systems by incorporating information from multiple modalities to build a more comprehensive representation. Despite its importance, current state-of-the-art models still suffer from severe limitations that prevent the successful development of a fully multimodal model. Such methods may not provide indicators that all the involved modalities are effectively aligned. As a result, some modalities may not be aligned, undermining the effectiveness of the model in downstream tasks where multiple modalities should provide additional information that the model fails to exploit. In this paper, we present TRIANGLE: TRI-modAl Neural Geometric LEarning, the novel proposed similarity measure that is directly computed in the higher-dimensional space spanned by the modality embeddings. TRIANGLE improves the joint alignment of three modalities via a triangle - area similarity, avoiding additional fusion layers or pairwise similarities. When incorporated in contrastive losses replacing cosine similarity, TRIANGLE significantly boosts the performance of multimodal modeling, while yielding interpretable alignment rationales. Extensive evaluation in three-modal tasks such as video-text and audio-text retrieval or audio-video classification, demonstrates that TRIANGLE achieves state-of-the-art results across different datasets improving the performance of cosine-based methods up to 9 points of Recall@1. Code and checkpoints available at https://github.com/ispamm/TRIANGLE/ . 1 Introduction Humans perceive reality as a mixture of signals coming from different modalities registered through diverse senses, such as sounds perceived by the ears or vision by the eyes, and process these multimodal inputs to understand the scene. In the last few years, foundational models have attempted to emulate this understanding system by aligning pairs of modalities using contrastive learning approaches. The first valuable contribution in this sense, CLIP Radford et al. 2021 , aligns image and textual latent representation with a contrastive loss based on the cosine similarity between the two vectors. CLIP established the de-facto receipt for aligning multimodal latent features and the subsequent methods, such as CLAP for audio-text alignment Elizalde et al. 2023 , have followed the same scheme. Figure 1: Current state-of-the-art (SOTA) methods struggle to incorporate the third audio modality and fail in video-to-text tasks, incorrectly assigning captions such as “ A dog is howling ", as the frames may be similar. On the contrary, the proposed TRIANGLE method effectively leverages all modalities together, bringing crucial information to discriminate among the captions, and retrieving the correct caption “ A dog is barking ". Later, several works focused on extending the CLIP approach to more than two modalities, incorporating audio, depth, or other modalities to build a more representative embedding space Girdhar et al. 2023 ; Zhu et al. 2024 ; Ruan et al. 2023 ; Wang et al. 2024 . All these methods still rely on the pairwise cosine similarity between modalities. Specifically, they select an anchor modality, which may be image Girdhar et al. 2023 , audio Ruan et al. 2023 , or language Zhu et al. 2024 , and then align the embeddings of all the other modalities one-by-one to the anchor embedding. However, this alignment only ensures that all modalities are aligned with the anchor, without providing any indicators for the alignment between non-anchor modalities. Therefore, if the video is aligned with the language as the anchor modality, and the audio is also aligned with the language, there are no geometric indicators that the audio and video modalities are aligned with each other. Such a limitation may undermine the effective applicability of these models in real-world multimodal scenarios. Indeed, they mostly perform experiments on two-modal tasks where they just test the alignment between one modality and the anchor one, as their pairwise similarity loss does not allow them to test three modalities at once. However, even in these scenarios, such models often fail when sample data includes a third modality that may be crucial to accurately discriminate and truly understand the content Yoon et al. 2023 . As an example, in widespread video-text retrieval tasks, conventional state-of-the-art (SOTA) models focus on video frames to solve the task, while the audio information is instead often crucial to properly retrieve the correct caption. Figure 1 shows an example of this task in which the SOTA model fails as it is not able to leverage the audio information, while the proposed one exploits both the modalities and correctly retrieves the caption. To partially overcome the limitation due to the use of only two modalities, other works have proposed to fuse multiple modalities via MLP layers Chen et al. 2023b , supplementary loss functions Wang et al. 2024 ; Chen et al. 2023b or additional architectural strategies Li et al. 2021 . Very recently, GRAM Cicchetti et al. 2025 and Symile Saporta et al. 2024 proposed general alternative similarity objectives that scale to an arbitrary number n of modalities by minimizing, respectively, the parallelotope volume and a total - correlation bound. Unfortunately, while these methods obtain improved alignment performance, some of them still lack geometrical indicators and fail to provide insights on how each modality impacts the final outcome, thereby reducing the interpretability of the results as well. Additionally, the majority of the recent works come up with their own datasets Zhu et al. 2024 ; Chen et al. 2023b ; Chen et al. 2023a ; Saporta et al. 2024 , placing a strong emphasis on data quality and on scaling up neural models to billions of parameters Wang et al. 2024 , yet still relying on the same similarity measure. In this paper, we present TRIANGLE: TRI-modAl Neural Geometric LEarning, a novel method capable of addressing the aforementioned limitations. The proposed method directly works within the space spanned by all embedding vectors, aligning them altogether. In this space, the extremities of the vectors form the vertices of a 2D polygon, whose shape and area show interesting insights into the semantic affinity among modalities. In the most common case of video, audio, and language modalities, such a polygon is a triangle, whose area is strictly related to the alignment of all the modalities. Geometrically, the smaller the area of the triangle, the closer (i.e., the more aligned) the vectors of the three modalities are to each other. Conversely, a larger area indicates that the vectors are more orthogonal, suggesting they point in opposite directions and probably represent misaligned data. Therefore, TRIANGLE proposes using triangle area minimization as the similarity metric to align three-modal vectors in their higher-dimensional space, without relying on pairwise comparisons. This method effectively integrates the information from the three modalities, as shown in Fig. 1 , where TRIANGLE exploits both the video and audio information to retrieve the correct caption. TRIANGLE sets a new state of the art in video-text retrieval across datasets such as MSR-VTT, DiDeMo, ActivityNet, and VATEX, as well as in audio-text retrieval and classification in AudioCaps and VGGSound, all without requiring new datasets or additional layers, underscoring its superiority over conventional methods relying on cosine similarities between pairs of modalities and over generic methods designed for joint alignment. In summary, our contributions are: 1. TRIANGLE, a three-modal alignment method is introduced. TRIANGLE encourages alignment among three modalities, leveraging all the modalities together to perform downstream tasks. 2. TRIANGLE offers an easily

Read the complete text