Evaluating and Steering Modality Preferences in Multimodal Large Language Model

Yu Zhang, Jinlong Ma, Yongshuai Hou, Xuefeng Bai, Kehai Chen, Yang Xiang, Jun Yu, Min Zhang · 2025

Read Evaluating and Steering Modality Preferences in Multimodal Large Language Model by Yu Zhang, Jinlong Ma, Yongshuai Hou, Xuefeng Bai, Kehai Chen, Yang Xiang, Jun Yu, Min Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-20977.txt.

Introduction

Yu Zhang and co-authors study whether models that read text and images favor one input type over the other. Their 2025 paper calls this pull modality preference. When text and image facts clash, a model may trust one source by habit, and that habit was little explored.

The team built a test set with staged clashes between the two sources. All 20 models they tried showed clear preferences for one type. Those habits also hinted at how well a model would do on later tasks. The pull could be shifted by plain instructions and read from the model inner states. A probe-and-steer method then pushed the pull either way, lifting scores on several tasks with no retraining.

Catalog record

CREATORYu Zhang, Jinlong Ma, Yongshuai Hou, Xuefeng Bai, Kehai Chen, Yang Xiang, Jun Yu, Min Zhang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE77 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-20977

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Evaluating and Steering Modality Preferences in Multimodal Large Language Model Yu Zhang, Jinlong Ma, Yongshuai Hou, Xuefeng Bai, Kehai Chen, Yang Xiang, Jun Yu, Min Zhang Abstract Multimodal large language models (MLLMs) have achieved remarkable performance on complex tasks with multimodal context. However, it is still understudied whether they exhibit modality preference when processing multimodal contexts. To study this question, we first build a MC 2 benchmark under controlled evidence conflict scenarios to systematically evaluate modality preference, which is the tendency to favor one modality over another when making decisions based on multimodal conflicting evidence. Our extensive evaluation reveals that all 18 tested MLLMs generally demonstrate clear modality bias, and modality preference can be influenced by external interventions. An in-depth analysis reveals that the preference direction can be captured within the latent representations of MLLMs. Built on this, we propose a probing and steering method based on representation engineering to explicitly control modality preference without additional fine-tuning or carefully crafted prompts. Our method effectively amplifies modality preference toward a desired direction and applies to downstream tasks such as hallucination mitigation and multimodal machine translation, yielding promising improvements. † † footnotetext: The code and data are released at https://github.com/EchoDreamer/Modality-Preference . 1 Introduction Multimodal Large Language Models (MLLMs) [ 3 , 41 , 43 , 50 ] have emerged as a powerful paradigm for processing and reasoning across heterogeneous data modalities (e.g., text, images, video). Recent advances demonstrate their exceptional capabilities, achieving remarkable performance on complex tasks with multimodal contexts, including autonomous web browsing, graphical user interface understanding, and multimodal dialogue system [ 16 , 18 , 40 ] . Despite impressive performance, fundamental questions remain about their modality preference , particularly whether they solve problems by genuinely integrating cross-modal information or exhibiting modality bias by over-relying on a single dominant modality . To investigate the modality preferences of MLLMs, a line of research evaluates MLLMs by comparing their performance on unimodal inputs (text-only vs. image-only presentations of the same problem) [ 12 ] . However, this approach introduces inherent bias, as it fails to assess the model’s ability to process and integrate truly multimodal inputs where both images and text are simultaneously present. Another line of research analyzes the relative contributions of textual instructions and visual context using Shapley value metric [ 4 , 37 , 36 ] . However, we contend that both parts are indispensable for task resolution, making this framework less adequate for determining true modality preferences across multi-modal contexts. In this paper, we provide a controlled setup to study the modality preference in MLLMs. As shown in the left panel of Figure 1 , we introduce a modality context conflict evaluation setting, where an MLLM is asked to answer a question based on a pair of contrasting evidence. In this way, we can determine the modality preference based on the answer given by MLLMs. Moreover, to dismiss the effect of several confounding factors, including question comprehension, single-modality perception, and the internal knowledge of MLLMs, we utilize widely adopted images and select samples that demonstrate accurate question comprehension and reliable single-modality perception. Building upon this, we introduce a semi-automated annotation framework to construct the refined Modality Context Conflict dataset, MC 2 , which covers eight recognition-level tasks with 2,000 carefully selected samples. Using MC 2 , we conduct a comprehensive analysis of modality preference across a diverse set of 18 representative MLLMs. Our study reveals several intriguing findings: • All tested MLLMs generally demonstrate clear modality bias, as illustrated in the right penal of Figure 1 , and the modality bias is affected by model-intrinsic factors and task-specific characteristics. • The direction of modality preference in MLLMs can be modulated through external interventions, such as modality degradation or instructional guidance. • Latent space analysis confirms that modality preferences manifest as geometrically separable patterns in MLLMs’ representations, which can be captured by distinct vectors. Built on these, we propose a modality preference probing and steering method based on representation engineering to explicitly amplify the direction of modality preference without the need for additional fine-tuning or prompt design. Experimental results show that the proposed method leads to notable performance improvements on both hallucination mitigation and multimodal machine translation. Our main contributions are summarized as follows: • We introduce a Modality Context Conflict benchmark for the systematic evaluation of modality preference in MLLMs. • Our systematic analysis reveals intrinsic modality preferences in MLLMs, and we find that such preferences are steerable and can be identified through latent representations, offering insights into reasoning under multimodal conflicts. • Inspired by these findings, we propose a method to steer modality preference via representation-level intervention, enabling controllable preference adjustment on MC 2 and improving performance on downstream tasks. Figure 1 : Illustrations of evaluating modality preference. Left: Use multimodal conflict pairs to evaluate modality preference. Right: Quantified results for modality preferences. Vision and Text represent vision and text preference, other represents irrelevant prediction. 2 Related Work Multimodal Large language Models. In recent years, Large Language Models (LLMs) have achieved impressive results across a wide spectrum of tasks [ 33 , 53 , 15 , 47 , 19 ] . Building upon these, Multimodal Large Language Models (MLLMs) have extended such capabilities, demonstrating strong performance in visual question answering, structured information understanding, and visual navigation [ 55 , 51 , 32 , 44 , 52 ] . Recent research has turned to systematically evaluating their biases and failure modes [ 30 , 1 , 5 , 4 , 9 ] , aiming to better understand their limitations and guide the development of more reliable models. Modality Preference in MLLMs. Modality preference—whether MLLMs prioritize one modality over another and to what extent they favor a specific modality when resolving multimodal inputs—can reveal how MLLMs jointly reason over different modalities. Some studies have also observed that MLLMs often favor textual inputs while neglecting visual information, which can result in hallucinations [ 8 , 34 , 54 ] . This highlights the importance of investigating modality preference. To study the modality preferences of MLLMs, one common approach involves evaluating model performance on unimodal inputs—comparing results when only text or only image inputs are provided for the same task [ 12 ] . However, this method inherently overlooks how models handle genuinely multimodal scenarios where both modalities are presented simultaneously, thus limiting its effectiveness in capturing true modality preference. Another research direction assesses the relative influence of textual and visual inputs using Shapley value-based attribution methods [ 4 , 36 , 37 ] . Yet, we argue that both modalities are typically essential for solving multimodal tasks, making this approach insufficient for identifying a model’s inherent modality preference in realistic multimodal contexts. Concurrent work evaluates the model’s ability to detect conflict under scenarios involving conflicting multimodal contexts [ 45 ] . However, conflict detection is only one aspect of multimodal reasoning and does not fully reflect a model’s overall ability to jointly reason across modalities. In this work, in order to investigate their modality preferences, we simulate multimodal reasoning by examining the behavior of MLLMs in response to questions under scenarios involving conflicting multimodal contexts. Compared to prior work, we carefully control for potential confounding factors that may influence modality

Read the complete text