RE-TRIANGLE: Does TRIANGLE Enable Multimodal Alignment Beyond Cosine Similarity in Retrieval? Conference: Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 20–24, 2026; Melbourne, VIC, Australia Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’26), July 20–24, 2026, Melbourne, VIC, Australia DOI: 10.1145/3805712.3808552 ISBN: 979-8-4007-2599-9/2026/07 CCS: Information systems Multimedia and multimodal retrieval
Arijit Ghosh, Aritra Bandyopadhyay, Chiranjeev Bindra, Jingfen Qiao · 2026
Read RE-TRIANGLE: Does TRIANGLE Enable Multimodal Alignment Beyond Cosine Similarity in Retrieval? Conference: Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 20–24, 2026; Melbourne, VIC, Australia Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’26), July 20–24, 2026, Melbourne, VIC, Australia DOI: 10.1145/3805712.3808552 ISBN: 979-8-4007-2599-9/2026/07 CCS: Information systems Multimedia and multimodal retrieval by Arijit Ghosh, Aritra Bandyopadhyay, Chiranjeev Bindra, Jingfen Qiao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-27436.txt.
Introduction
Arijit Ghosh and colleagues test whether the TRIANGLE method aligns modalities beyond cosine similarity. TRIANGLE shrinks triplet areas on a hypersphere to force shared consistency across side modalities. The redo confirms zero-shot gains of about 9 points over pairwise baselines, but fails to redo learning-from-scratch runs, blaming joint optimization instability.
Catalog record
| CREATOR | Arijit Ghosh, Aritra Bandyopadhyay, Chiranjeev Bindra, Jingfen Qiao |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43300 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.27436
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
RE-TRIANGLE: Does TRIANGLE Enable Multimodal Alignment Beyond Cosine Similarity in Retrieval? Conference: Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 20–24, 2026; Melbourne, VIC, Australia Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’26), July 20–24, 2026, Melbourne, VIC, Australia DOI: 10.1145/3805712.3808552 ISBN: 979-8-4007-2599-9/2026/07 CCS: Information systems Multimedia and multimodal retrieval Arijit Ghosh, Aritra Bandyopadhyay, Chiranjeev Bindra, Jingfen Qiao Abstract . Multimodal alignment is critical for bridging the semantic gap in information retrieval. However, traditional pairwise strategies introduce a geometric blind spot: while they align anchor modalities (e.g., text) with others, they lack constraints to enforce mutual consistency between peripheral modalities (e.g., video and audio). The TRIANGLE framework addresses this by minimizing the area of modality triplets on a hypersphere to enforce holistic alignment. In this reproducibility study, we verify the robustness of this geometric objective for retrieval tasks. We confirm that TRIANGLE outperforms pairwise baselines in zero-shot settings, achieving Recall@1 gains of up to +8.7 points, though benefits are domain-dependent. However, we fail to reproduce the reported learning-from-scratch results. Analysis using a synthetic toy dataset attributes this to instability when jointly optimizing geometric alignment with Data-Text Matching (DTM) loss. Furthermore, we find that cosine regularization primarily stabilizes text-to-video retrieval, and fine-tuning with domain supervision amplifies geometric benefits but reduces cross-dataset generalization. Our findings support the efficacy of geometric alignment while highlighting critical optimization sensitivities. 1 1 1 Code available at https://github.com/ARIJIT00171/RE-TRIANGLE † † cc-license: by 1. Introduction Figure 1 . TRIANGLE loss aims to minimize the Triangle area formed by the embedding vectors on a unit hypersphere between three modalities - Video (V1), Audio (A1) and Text(T1). Multimodal alignment is a fundamental challenge in Multimodal Information Retrieval, where the goal is to bridge the semantic gap between multiple co-occurrent signals. This field was revolutionized by foundational dual-encoder models like CLIP ( 18 ) and ALIGN ( 12 ) , which leverage pairwise contrastive learning to align images and text. Following this paradigm, numerous works have focused on refining alignment mechanics ( 21 ; 24 ) or extending the architecture to specific modality pairs, such as audio-text in CLAP ( 8 ) , video-text in CLIP4Clip ( 15 ) , and point cloud-text in PointCLIP ( 25 ) . This approach has been scaled to more than two modalities by selecting a central anchor and aligning all other representations to it one by one, such as ImageBind ( 11 ) and VAST ( 4 ) . However, this pairwise alignment introduces a significant geometric blind spot ( 5 ) . While the anchor modality aligns with others, there is no explicit constraint ensuring that peripheral modalities (e.g., Video and Audio) are effectively aligned with one another. Similarly, fusion-based methods ( 4 ) reduce the tri-modal problem back to a pairwise objective (Fused Vector vs. Text), effectively masking the fine-grained geometric structure of the latent space. Recent efforts to model the joint distribution of N modalities, such as Symile ( 19 ) (optimizing total correlation) and GRAM ( 6 ) (minimizing Gramian volume), offer theoretical alternatives. Yet, these often suffer from computational instability or the "vanishing volume" problem in high dimensions, failing to outperform specialized baselines in tri-modal tasks. To address this, ( 5 ) proposed TRIANGLE: TRI-modAl Neural Geometric LEarning. Rather than relying on independent 2D projections via cosine similarity, TRIANGLE computes a similarity measure directly in the higher-dimensional space spanned by three modality embeddings. This similarity is defined by the area of the triangle formed by the three modality embeddings lying on a unit hypersphere, replacing the standard cosine similarity metric. By minimizing this triangle area, the model encourages a holistic, joint alignment among all three modalities simultaneously. The authors claim this approach provides a reliable, interpretable metric for modality alignment and achieves state-of-the-art results, improving the performance of cosine-based methods by up to 9 points in Recall@1 Despite the theoretical elegance and reported performance gains of the TRIANGLE objective, its practical robustness and generalizability require independent verification. In this paper, we present a comprehensive reproducibility study motivated by three core research objectives: • RQ1 (Alignment Effectiveness in Multimodal Retrieval): Can a tri-modal geometric objective outperform independent pairwise alignment in multimodal retrieval tasks? • RQ2 (Out of domain performance): Can the pre-trained tri-modal geometric objective effectively adapt to out-of-domain distributions through continuous fine-tuning? • RQ3 (Effectiveness of Learning from Scratch): Does tri-modal geometric objective also helps when training from scratch? • RQ4 (Interpretability of Triangle Similarity): Does the triangle area provide a consistent and geometrically unambiguous indicator of modality alignment during training and inference? In response to these questions, we present the following findings: • We corroborate the original finding that the tri-modal geometric objective outperforms independent pairwise alignment in zero-shot retrieval tasks across diverse datasets. • We extend the evaluation to YouCook2 ( 26 ) to test if the tri-modal geometric objective holds up in out-of-domain video retrieval. We find that the objective is highly domain-sensitive, significantly underperforming in the unseen homogeneous YouCook2 domain compared to the baselines. • Investigating the reproducibility issues for the learning-the-space-from-scratch experiment by curating a small toy dataset consisting of geometric shapes for a controlled study. • Our qualitative analysis validates the Triangle area as an interpretable similarity metric. Our visualizations reveal that while text and video align closely, audio remains geometrically distant, explaining the model’s uneven cross-modal performance . 2. Overview of Triangle Alignment 2.1. Preliminaries: Pairwise Alignment Conventional multimodal retrieval relies on maximizing pairwise cosine similarity between an anchor (e.g., Text) and other modalities (e.g., Audio/Video). Formally, given a batch size B and a temperature parameter \tau , the symmetric pairwise contrastive loss between embeddings x and y is computed in Equation 1 . (1) \mathcal{L}_{Pair}=-\frac{1}{2B}\sum_{(x,y)}\sum_{i=1}^{B}\log\frac{\exp\left(\mathbf{x}_{i}^{\top}\mathbf{y_{i}}/\tau\right)}{\sum_{j=1}^{B}\exp\left(\mathbf{x}_{i}^{\top}\mathbf{y}_{j}/\tau\right)} where the outer summation iterates over the directional pairs (x,y)\in\{(\mathbf{a},\mathbf{t}),(\mathbf{t},\mathbf{a}),(\mathbf{v},\mathbf{t}),(\mathbf{t},\mathbf{v})\} . The \mathbf{t} , \mathbf{a} , and \mathbf{v} denote the normalized embeddings for text, audio, and video, respectively. In this setting, the peripheral relationships (e.g., Audio-Video) are not explicitly optimized. To address this, VAST aggregates video \mathbf{v} and audio \mathbf{a} representations into a unified vector via a MLP fusion layer, and the contrastive loss is applied between this single fused vector and the text. However, this approach collapses the distinct geometric structure of the individual modalities into a single point before alignment. 2.2. Geometric Tri-modal Alignment To address the limitations of the independent pairwise alignment, TRIANGLE proposed to align three modalities concurrently by maintaining their distinct positions in the joint latent space, and posits that the area of this triangle serves as a superior similarity metric. Specifically, it utilizes three backbone encoders to map different modality inputs into \mathbf{x} , \mathbf{y} , and \mathbf{z} embeddings of fixed dimensionality. These vectors are normalized to the unit norm, constraining them to lie on the surface of a unit hypersphere. Triangle Area Similarity . As illustrated in Figure 1 (and further detailed in Figure 4), the extremities of the three embedding vectors define the vertices of a triangle. Let \mathbf{u}=\mathbf{x}-\mathbf{y} and \mathbf{w}=\mathbf{x}-\mathbf{z} are two triangle sides computed among the three embeddings x, y, and z of the three modalities. The alignment similarity A is derived from the triangle area, augmented by a