Bring Reason to Vision: Understanding Perception and Reasoning through Model Merging
Shiqi Chen, Jinghan Zhang, Tongyao Zhu, Wei Liu, Siyang Gao, Miao Xiong, Manling Li, Junxian He · 2025
Read Bring Reason to Vision: Understanding Perception and Reasoning through Model Merging by Shiqi Chen, Jinghan Zhang, Tongyao Zhu, Wei Liu, Siyang Gao, Miao Xiong, Manling Li, Junxian He (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-05464.txt.
Introduction
Chen and seven co-authors fuse two models to pour thought skill into a sight-tongue model in this twenty twenty-five paper. Sight models see well, while tongue models think well, and the join is still dim. Fuse-by-weights is their bridge, with no fresh drill rounds at all.
Inside, fused models gain thought skill from the tongue side in test after test. The team then maps where sight and thought live in the stack of layers. Sight skill sits most in the first layers of the stack in their maps. Thought skill sits most in the mid-to-late layers of the stack. Past the fuse, all layers pitch in on thought, while sight spread stays put.
Catalog record
| CREATOR | Shiqi Chen, Jinghan Zhang, Tongyao Zhu, Wei Liu, Siyang Gao, Miao Xiong, Manling Li, Junxian He |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-05464
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Bring Reason to Vision: Understanding Perception and Reasoning through Model Merging Shiqi Chen, Jinghan Zhang, Tongyao Zhu, Wei Liu, Siyang Gao, Miao Xiong, Manling Li, Junxian He Abstract Vision-Language Models (VLMs) combine visual perception with the general capabilities, such as reasoning, of Large Language Models (LLMs). However, the mechanisms by which these two abilities can be combined and contribute remain poorly understood. In this work, we explore to compose perception and reasoning through model merging that connects parameters of different models. Unlike previous works that often focus on merging models of the same kind, we propose merging models across modalities , enabling the incorporation of the reasoning capabilities of LLMs into VLMs. Through extensive experiments, we demonstrate that model merging offers a successful pathway to transfer reasoning abilities from LLMs to VLMs in a training-free manner. Moreover, we utilize the merged models to understand the internal mechanism of perception and reasoning and how merging affects it. We find that perception capabilities are predominantly encoded in the early layers of the model, whereas reasoning is largely facilitated by the middle-to-late layers. After merging, we observe that all layers begin to contribute to reasoning, whereas the distribution of perception abilities across layers remains largely unchanged. These observations shed light on the potential of model merging as a tool for multimodal integration and interpretation. Our code is publicly available at: https://github.com/shiqichen17/VLM_Merging . † † affiliationnotice: Equal contribution 1 Introduction Multimodal reasoning is crucial for a variety of important applications, such as interpreting charts and figures in scientific publications and government reports. Despite the great successes of Vision-Language Models (VLMs) in tasks requiring perceptual and linguistic integration ( Li et al., 2024 ; Liu et al., 2024 ; Bai et al., 2023 ; Wang et al., 2024b ) , these models struggle with complex multimodal reasoning tasks ( Lu et al., 2024 ; Zhang et al., 2024b ) . This limitation – partly due to the scarcity of multimodal reasoning data – leaves them lagging far behind their language model counterpart which has made remarkable advancement in reasoning tasks ( Yang et al., 2024 ; DeepSeek-AI et al., 2025 ) . Figure 1 : Illustration of our work investigating how model merging works when transferring reasoning ability from a Math-specific LLM to the VLM, showcasing the effects and results of the merged model, as well as the interpretation of layer-wise abilities. The abilities are represented in corresponding colors: red indicates perception ability hidden in early layers , while blue denotes reasoning ability hidden in relatively later layers . Perception and reasoning are two fundamental components in this context. While language models primarily represent the reasoning ability, VLMs demand both to succeed. Therefore, it is natural to ask: can we incorporate the reasoning ability of LMs into VLMs? Achieving such a combination is challenging, as the interaction between perception and reasoning within VLMs remains poorly understood. In this work, we investigate these questions through the lens of model merging ( Ilharco et al., 2023 ) , a straightforward approach to explore whether perception and reasoning can be combined across modalities and how these two abilities are embedded within VLMs. Concretely, model merging generates a new model by performing arithmetic operations on the parameters of existing models, without requiring additional training. This strategy works based on the assumption that models fine-tuned from a shared initialization reside in a connected subspace of the parameter space. While previous works on model merging focus on models of the same kind ( Yadav et al., 2023 ; Yu et al., 2024b ) , it remains unknown that whether models across different modalities are connectable to yield benefits. In this work, we specifically focus on the textual components of the VLMs and select LLMs with task-specific reasoning abilities that match the VLM’s configuration, performing a weighted average operation on their parameters as demonstrated in Figure 1 . We conduct extensive experiments by merging commonly used VLMs with LLMs trained on diverse math reasoning datasets (§ 4 ). Our findings demonstrate that model merging consistently improves the reasoning capabilities of VLMs across all math benchmarks, with minimal impact on the perception-dominant tasks. For instance, merging with Dart ( Tong et al., 2024 ) enhances LLaVA’s performance on MathVista’s ( Lu et al., 2024 ) math-related subset, yielding a 3.6-point absolute improvement. Even in Vision-Only mode of MathVerse ( Zhang et al., 2025 ) , where questions are presented in images, it also achieves a 1.4-point absolute improvement. Experiments on multiple reasoning-related benchmarks and various VLMs consistently show improvement. These results underscore the potential of parameter merging as a simple yet powerful mechanism for capability transfer across model architectures. Furthermore, we delve deeper into understanding how model merging works and how perception and reasoning interplay in this context. Specifically, we analyze parameter changes during the merging process to investigate whether different capabilities, such as high-level reasoning and low-level perception, are disentangled within distinct subspaces of the model’s parameter space (§ 5 ). Through knockout analysis, we identify two key observations: (1) Image perceptual abilities and world knowledge are predominantly embedded in the early layers of the model, whereas mathematical reasoning skills are concentrated in the middle-to-late layers; and (2) Merging with reasoning models brings reasoning abilities to all the layers, while having a minimal impact on the layer distribution of perception ability. These findings contribute to a better understanding of how reasoning can be transferred between models and provide insights into model compositionality, offering a promising approach for enhancing multi-modal reasoning systems. Category Name Size Base Model VLMs LLaVA-NeXT ( Liu et al., 2024 ) 8B LLaMA-series Idefics2-8B ( Laurençon et al., 2024 ) 8B Mistral-series InternVL2 ( Chen et al., 2024b ) 76B LLaMA-series Domain Base Model Task Vectors Dart-Math ( Tong et al., 2024 ) Math Domain LLaMA/Mistral-series MetaMath ( Yu et al., 2024a ) Math Domain LLaMA-series MAmmoTH-1 ( Yue et al., 2024a ) Math Domain LLaMA-series MAmmoTH-2 ( Yue et al., 2024b ) General Domain LLaMA/Mistral-series Magpie-v0.3 ( Xu et al., 2024 ) Math Domain LLaMA-series Deepseek-R1-Distill ( DeepSeek-AI et al., 2025 ) Math Domain LLaMA-series Table 1 : Overview of Vision-Language Models (VLMs) and Task Vectors with the attributes -size, base models and domains. 2 Model Merging Across Modalities In this section, we introduce model merging for transferring the reasoning abilities of textual LMs to VLMs. A typical VLM consists of three key components: a vision tower, a language Model, and a projector that bridges these two parts. The vision tower processes images, enabling the model to “see” visual content, while the language model serves as the reasoning engine, processing knowledge and generating responses. Therefore, we target the language model ( \theta_{\text{vlm}} ) for merging while keeping the vision tower and projector unchanged. Model merging has emerged as a promising “free lunch” technique, enabling performance improvements by reusing existing models through simple arithmetic operations on their parameters, without requiring additional training. For simplicity, we adopt linear merging ( Ilharco et al., 2023 ) , a widely used and robust merging strategy, in our main experiments. We also experiment with TIES merging ( Yadav et al., 2023 ) in some cases to compare both methods in Appendix C .