Rethinking Model Selection in VLM Through the Lens of Gromov-Wasserstein Distance
Muyang Li Yucheng Liu Jianbo Ma, Elliot Osborne Bo Han Tongliang Liu · 2026
Read Rethinking Model Selection in VLM Through the Lens of Gromov-Wasserstein Distance by Muyang Li Yucheng Liu Jianbo Ma, Elliot Osborne Bo Han Tongliang Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-01325.txt.
Introduction
Muyang Li and five co-authors ask a practical question: when building a vision-language model, which pre-trained image encoder should you pair with the language model? Builders often pick the biggest encoder or the one with the best solo image scores, and this study tests whether those rules work.
Across 19 encoders and more than 60 full training runs, size and solo score predict final model quality only weakly. What does track quality is how alike the structure of the image and text spaces is, measured with the Gromov-Wasserstein distance. The authors show in theory why this link should hold for learning a map between the two spaces. Their score needs no training, so it can flag good pairings before costly full runs begin.
Catalog record
| CREATOR | Muyang Li Yucheng Liu Jianbo Ma, Elliot Osborne Bo Han Tongliang Liu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.01325
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Rethinking Model Selection in VLM Through the Lens of Gromov-Wasserstein Distance Muyang Li Yucheng Liu Jianbo Ma, Elliot Osborne Bo Han Tongliang Liu Abstract Vision-Language Models (VLMs) have enhanced traditional LLMs with visual capabilities through the integration of vision encoders. While recent works have explored various combinations of vision encoders and LLMs, there still lacks a principled understanding of what makes a vision encoder suitable for VLM alignment. In this paper, we systematically investigate this question via comprehensive experiments on a curated collection of 18 pre-trained vision encoders from diverse sources. We first demonstrate that common practices, such as choosing encoders with the largest size or highest zero-shot accuracy, consistently fail to identify optimal models. In fact, these metrics show only weak to moderate correlation with VLM performance. This intriguing finding begs a fundamental question: What factors of vision-encoders matter in VLM? Through comprehensive analysis, we identify that the structural similarity across modalities plays a crucial but previously overlooked role in vision-encoder selection, which we measure using the Gromov-Wasserstein distance as a proxy. From a theoretical perspective, we show that the learnability of cross-modality mapping can be provably associated with the Gromov-Wasserstein distance. Empirical verification on 60+ full VLM training runs shows that our proposed inference-only metric performs significantly better than alternative model selection strategies and exhibits a much stronger correlation with final VLM performance, thereby enabling efficient and effective prediction of VLM performance before full training. 1 Introduction Vision-language models (VLMs) have become an indispensable component of modern AI systems, ranging from state-of-the-art proprietary AI systems such as ChatGPT-5, to open-sourced popular models such as LLaVa ( 28 ) , Cambrian ( 44 ) , etc. The dominant paradigm for current state-of-the-art (SOTA) VLMs ( 2 ; 44 ; 9 ; 20 ; 50 ; 57 ) can be depicted by the seminal work of Visual Instruction Tuning ( 28 ) , which is typically characterized as a “pre-training then fine-tuning” process: a projector is trained to semantically align the representation spaces of the vision encoder and LLM, and subsequently, the model undergoes joint full-parameter fine-tuning of both towers for a cohesive multi-modal understanding and instruction-following capabilities. Despite the widespread success and the adoption of this simple paradigm, many questions remain unclear in this process. Specifically, prior studies have explored numerous combinations of LLM and vision-encoders ( 44 ; 8 ) , and a mismatch between the model size and performance after fine-tuning has been observed ( 22 ) . As a foundational step, this paper presents a holistic evaluation of 18 state-of-the-art (SOTA) vision encoders from diverse sources, by performing complete visual instruction tuning 27 loop on all of them. We show that traditional wisdom such as selecting the model with the largest size or highest zero-shot accuracy consistently fails to identify best choice of vision encoders. Moreover, our correlation analysis suggests that there is even no statistically meaningful correlation between the capability of vision encoders and final VLM performances. This intriguing observation begs the question of: What makes a good vision encoder for VLM? Figure 1 : From left to right, we can see the correlation analysis of zero-shot classification accuracy, vision encoder size, and GW distance against the final VLM performances, respectively. r : Pearson correlation, \rho : Spearman correlation, R 2 : R-squared coefficient. As we can see, the correlation between GW distance and final VLM performances is notably higher than naive baselines. In this paper, we aim to gain a holistic understanding to this question: if the performances of pre-trained vision encoders cannot explain the performances of VLM after visual instruction tuning, what could be the factors that are actually crucial? We begin with a simple intuition: beyond raw visual capability, there exists a notion of compatibility between the vision encoder and the LLM. Depending on the pre-training data, model architecture, and training strategy, the induced visual representations can be more or less heterogeneous with respect to the LLM’s representation space, directly affecting the difficulty of aligning them. We hypothesize that a key aspect of this compatibility is the structural similarity between visual and textual representations, which we formalize as an appropriate similarity measure between their induced distributions. When a vision encoder produces representations that are structurally incompatible with those of the LLM (i.e., the distance between the visual and textual representation distributions is large), the resulting VLM may perform poorly after fine-tuning, even if the encoder itself exhibits strong standalone visual capability. However, one challenge emerging from the attempt to compute distributional discrepancy between different modalities is that: distribution of different modality representations resides in different metric-measure space (mm-space). That is: (i) their measures are different, since the measurable function that generates those representations are different, in other words, naive distance of representations from different modalities have no canonical meaning; (ii) their metric function are different, for example, representations from different modalities have different dimension, which means that no unified metric function can naively compute their distance. Motivated by this, we seek the notion of Gromov-Wasserstein (GW) distance 31 as a proxy to measure the distance between distributions from different mm-spaces (modalities). Instead of calculating the inter-distribution distance, GW distance measures how similar two mm-spaces are by finding a correspondence between their points that minimizes the distortion of pairwise distances. In other words, it compares structural similarity up to isometry without needing a common ambient space. We summarize our contributions as follows: (i) we systematically study the problem of pre-trained vision encoder selection in VLM training, unveiling the fact that there currently lacks a plausible explanatory variable to correlate the factors of vision encoder to final VLM performances; (ii) we unveil the crucial factor in the success of VLM - the distributional compatibility between vision and text representations, which we instantiate through GW distance; (iii) we prove that the upper-bound of Lipschitz constant of the Bayes optimal hypothesis of cross-modality projector is associated with the \infty -norm GW distance, justifying that, if vision and text representations are of small GW distance, then the optimal mapping between them can essentially be learned with simpler hypothesis; (iv) through comprehensive experimental verifications with 60+ full VLM training runs, our proposed metrics show consistent performance gains compared to baselines, and showing strong Pearson correlation to the final VLM performances, whereas baselines’ correlations are only weak to moderate. 2 Related Works Gromov-Wasserstein Distance. The Gromov-Wasserstein (GW) distance 31 was proposed to quantify the similarity between different spaces, with its application in Graph Matching 48 , Flow Matching Diffusion 18 , Saliency Detection 54 , etc. Essentially, in areas where we wish to measure the similarity across heterogeneous domains, GW distance could be useful since it is invariant to choice of metric function or measure equipped with that domain. Subsequent methodological improvements over vanilla GW distance including Entropic Gromov-Wasserstein distance 39 , which applied entropic penalization to transport plan for more smoothness, etc. The Platonic Representation Hypothesis. A notable existing work relevant to our study is the Platonic Representation Hypothesis 15 . This hypothesis stipulates that as the size of vision encoders and LLMs scales, their representations across modalities effectively converge. This implies that, despite being trained with significantly different objective functions, data, and hypothesis classes, different models eventually