The Role of Visual Modality in Multimodal Mathematical Reasoning: Challenges and Insights
Yufang Liu, Yao Du, Tao Ji, Jianing Wang, Yang Liu, Yuanbin Wu, Aimin Zhou, Mengdi Zhang, Xunliang Cai [4pt] School of Computer Science and Technology, East China Normal University Meituan Inc. Fudan University Pazhou Laboratory [email protected] [email protected] [email protected] · 2025
Read The Role of Visual Modality in Multimodal Mathematical Reasoning: Challenges and Insights by Yufang Liu, Yao Du, Tao Ji, Jianing Wang, Yang Liu, Yuanbin Wu, Aimin Zhou, Mengdi Zhang, Xunliang Cai [4pt] School of Computer Science and Technology, East China Normal University Meituan Inc. Fudan University Pazhou Laboratory [email protected] [email protected] [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-04167.txt.
Introduction
Yufang Liu and eight co-authors ask whether multimodal AI models really look at the picture when they solve math problems, in this 2025 paper. Most benchmarks pair word problems with diagrams. The team found that top models barely use the images: change or remove them, and scores hardly move, because text and answer options leak the answer.
To force honest looking, they built a new dataset where the image is essential and where near-twin images flip the correct answer. Leading models failed to catch those subtle visual differences. Stacking more image encoders, a common trick from general visual question answering, did not help math scores either. The authors conclude that current models see too coarsely for math, and they release their benchmark and code to push better visual grounding.
Catalog record
| CREATOR | Yufang Liu, Yao Du, Tao Ji, Jianing Wang, Yang Liu, Yuanbin Wu, Aimin Zhou, Mengdi Zhang, Xunliang Cai [4pt] School of Computer Science and Technology, East China Normal University Meituan Inc. Fudan University Pazhou Laboratory [email protected] [email protected] [email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-04167
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The Role of Visual Modality in Multimodal Mathematical Reasoning: Challenges and Insights Yufang Liu, Yao Du, Tao Ji, Jianing Wang, Yang Liu, Yuanbin Wu, Aimin Zhou, Mengdi Zhang, Xunliang Cai [4pt] School of Computer Science and Technology, East China Normal University Meituan Inc. Fudan University Pazhou Laboratory [email protected] [email protected] [email protected] Abstract Recent research has increasingly focused on multimodal mathematical reasoning, particularly emphasizing the creation of relevant datasets and benchmarks. Despite this, the role of visual information in reasoning has been underexplored. Our findings show that existing multimodal mathematical models minimally leverage visual information, and model performance remains largely unaffected by changes to or removal of images in the dataset. We attribute this to the dominance of textual information and answer options that inadvertently guide the model to correct answers. To improve evaluation methods, we introduce the HC-M3D dataset, specifically designed to require image reliance for problem-solving and to challenge models with similar, yet distinct, images that change the correct answer. In testing leading models, their failure to detect these subtle visual differences suggests limitations in current visual perception capabilities. Additionally, we observe that the common approach of improving general VQA capabilities by combining various types of image encoders does not contribute to math reasoning performance. This finding also presents a challenge to enhancing visual reliance during math reasoning. Our benchmark and code would be available at https://github.com/Yufang-Liu/visual_modality_role . 1 Introduction Recent advancements in Large Vision-Language Models (LVLMs) Zhang et al. 2023 have demonstrated remarkable potential in tasks that require the seamless integration of visual and linguistic understanding, such as image captioning Lin et al. 2014 , visual question answering Antol et al. 2015 , visual grounding Yu et al. 2016 , and autonomous agents Xi et al. 2023 ; Durante et al. 2024 . Among these, the domain of multimodal mathematical reasoning Wang et al. 2024a ; Lu et al. 2024 ; Zhang et al. 2024b emerges as a particularly challenging and pivotal application. This task requires models to accurately interpret and abstractly represent mathematical concepts, such as geometry, through a tight collaboration between textual descriptions and visual elements in diagrams. The ability to align these modalities and resolve complex problems lies at the heart of LVLMs’ capabilities, positioning multimodal mathematical reasoning as a critical field for advancing their effectiveness in abstract and logic-driven applications. Figure 1: An illustrative example of GPT-4o on the HC-M3D. When modifying the image (by swapping the positions of points G and H) while keeping the original question unchanged, the model’s output failed to correspondingly adjust, resulting in an incorrect response. There are three common approaches to enhancing mathematical reasoning capabilities in LVLMs: prompting, fine-tuning, and chain-of-thought (CoT) reasoning. The prompting approach Lu et al. 2024 ; Wang et al. 2024b seeks to unlock the potential of LVLMs through carefully designed prompts. Fine-tuning Shi et al. 2024 ; Gao et al. 2023 , on the other hand, focuses on constructing higher-quality and more diverse datasets to enhance model performance. Chain-of-thought Hu et al. 2024 reasoning mimics human reasoning by generating detailed step-by-step thought processes to tackle complex mathematical tasks. These methods rely heavily on training data tailored for mathematical understanding. Existing research primarily emphasizes improving the diversity of problem types Gao et al. 2023 , refining the conciseness and clarity of questions Zhang et al. 2024c , and providing more detailed answer explanations Peng et al. 2024 , with model performance evaluated on various benchmarks. However, current datasets often fail to establish a strong connection between mathematical images and text. In this work, we explore whether images are genuinely learned and utilized by models for mathematical reasoning within current multimodal mathematical methods and examines the role they play in the process. We observe that in existing multimodal math models, the benefit brought by image patterns is minimal. Even if the images in the training set are shuffled or even removed, the impact on model performance is not significant (On some evaluation datasets, perturbing the visual modality even resulted in higher scores). By analyzing existing evaluation methods, we suspect that there are two main reasons why the role of visual patterns in multimodal math reasoning has been overestimated: first, the textual information is overly rich, and the model can make correct predictions based on text alone; second, the options leak the answers, and the model can guess the correct answer based on the options. Hence, we propose the HC-M3D dataset, a h uman- c rafted m uli m odal m athematical d ataset, comprising 1,851 samples meticulously selected by humans, that ensures that the questions depend on the images and provides an additional image that looks similar but changes the correct answer for over 400 problems. We observe whether models can identify these subtle differences in the images and make correct predictions. In evaluating existing leading models, we find that they fail to accurately identify these differences, and in more than half of the cases, they stick to their original predictions. Figure 1 presents a representative example. Furthermore, we attempt to delve into the causes of this phenomenon. Considering that the model’s predictions remain unchanged after image alterations, we infer that the image encoder (mainly the CLIP encoder) fails to effectively recognize such subtle differences, a phenomenon also observed in other works Liu et al. 2024b ; Tong et al. 2024a . However, we observe that the common approach of improving general VQA capabilities by combining various types of image encoders does not contribute to math reasoning performance. This finding also presents a challenge to enhancing visual reliance during math reasoning. Our contributions can be summarized as follows: • We show that image shuffling or removal has minimal impact on performance, revealing an overestimation of these models’ reliance on visual input. • We introduce the HC-M3D dataset with 1,851 human-annotated samples, ensuring image-dependent questions. For 429 questions, similar but altered images are included, yet most mathematical LVLMs fail to adjust predictions, often remaining unchanged. • We demonstrate the challenge of enhancing visual reliance in mathematical reasoning, as combining image encoders proves ineffective. Based on these observations, we suggest several potential directions for future research: constructing higher-quality datasets with a stronger reliance on visual data (e.g., in the image-caption pre-training task, describing the differences between two images rather than generating captions for a single image.), improving image encoders to capture more fine-grained mathematical information, and designing better loss functions to enhance the model’s dependence on the visual modality. 2 Related Work Mathematical reasoning has garnered increasing attention from researchers and has become a critical benchmark for evaluating LLMs. Specialized LLMs such as MetaMath Yu et al. 2024a , Math-Shepherd Wang et al. 2024c , WizardMath Luo et al. 2023 , and DeepSeekMath Shao et al. 2024 have been developed to tackle these tasks. One of the challenges in multimodal mathematical reasoning lies in visual-textual reasoning, involving geometric diagrams, scientific charts, and function graphs. Due to the scarcity of training data, existing methods often employ the LLaVA architecture for training, aiming to generate higher-quality and more diverse training data. For instance, G-LLaVA Gao et al. 2023 obtains image caption data for alignment by rewriting the original QA data and paraphrases the remaining QA data to augment the SFT