VisNec: Measuring and Leveraging Visual Necessity for Multimodal Instruction Tuning

Mingkang Dong, Hongyi Cai, Jie Li, Sifan Zhou, Bin Ren, Kunyu Peng, Yuqian Fu · 2026

Read VisNec: Measuring and Leveraging Visual Necessity for Multimodal Instruction Tuning by Mingkang Dong, Hongyi Cai, Jie Li, Sifan Zhou, Bin Ren, Kunyu Peng, Yuqian Fu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-01195.txt.

Introduction

Dong and six colleagues select multimodal training data by visual necessity, since many instruction samples can be solved from text alone or carry misaligned supervision. VisNec scores each sample by how much its loss changes with versus without visual input. It flags vision-critical, redundant, or misaligned cases, then picks high-necessity samples within semantic clusters to preserve diversity. Training on just 15 percent of a large dataset matches full-data performance, and on a smaller set it beats full training by about 16 percent.

Catalog record

CREATORMingkang Dong, Hongyi Cai, Jie Li, Sifan Zhou, Bin Ren, Kunyu Peng, Yuqian Fu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38588
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.01195

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

VisNec: Measuring and Leveraging Visual Necessity for Multimodal Instruction Tuning Mingkang Dong, Hongyi Cai, Jie Li, Sifan Zhou, Bin Ren, Kunyu Peng, Yuqian Fu Abstract The effectiveness of multimodal instruction tuning depends not only on dataset scale, but critically on whether training samples genuinely require visual reasoning. However, existing instruction datasets often contain a substantial portion of visually redundant samples (solvable from text alone), as well as multimodally misaligned supervision that can degrade learning. To address this, we propose VisNec (Visual Necessity Score) , a principled data selection framework that measures the marginal contribution of visual input during instruction tuning. By comparing predictive loss with and without visual context, VisNec identifies whether a training instance is vision-critical, redundant, or misaligned. To preserve task diversity, we combine VisNec with semantic clustering and select high-necessity samples within each cluster. Across 10 downstream benchmarks, training on only 15% of the LLaVA-665K dataset selected by VisNec achieves 100.2% of full-data performance. On the smaller Vision-Flan-186K dataset, our selection not only further reduces data size but surpasses full-data training by 15.8%. These results demonstrate that measuring and leveraging visual necessity provides an effective solution for both efficient and robust multimodal instruction tuning. Codes and selected subsets will be released upon acceptance. Figure 1 : Illustration of the VisNec score , which measures the contribution of visual input in multimodal instruction tuning. Samples are categorized as Misaligned (VisNec < 0, visual input degrades prediction), Redundant ( \approx 0 , visual input provides no additional benefit), or Vision-Critical (> 0, visual input substantially improves prediction). 1 Introduction Multimodal instruction tuning [ 27 , 17 ] plays a central role in training multimodal large language models (MLLMs), enabling them to follow complex vision-language instructions and generalize across diverse tasks [ 29 , 12 , 28 , 9 , 37 , 36 , 23 , 30 , 15 , 19 , 32 , 31 ] . Beyond architectural design, the quality of instruction data fundamentally determines whether models develop genuine cross-modal reasoning or rely on superficial correlations. Consequently, curating high-quality multimodal instruction data is critical for robust and reliable performance. While large-scale instruction datasets collected from diverse sources have successfully trained strong MLLMs, directly using all available data presents two notable limitations. First, it is inefficient. As datasets scale to hundreds of thousands or millions of samples, training costs grow substantially, yet not all samples contribute equally to learning. Second, large-scale instruction data inevitably contains a significant proportion of redundant or misaligned samples. Visually redundant instances can often be answered through linguistic shortcuts alone (e.g., predicting “green” for “the color of grass”), providing little incentive for visual grounding. When such samples dominate training, models tend to exploit textual correlations rather than integrate visual evidence, weakening cross-modal dependence. In contrast, multimodal misalignment, arising from annotation errors or noisy data pipelines, produces image-text pairs with inconsistent supervision. Training on such samples can actively degrade visual reasoning and amplify hallucinations at inference time. These observations raise a fundamental question: Can we identify a small yet truly valuable subset of multimodal instruction data that is both efficient and effective? Existing data selection approaches primarily rely on generic importance or diversity signals, such as gradient influence [ 33 ] , transferability [ 21 ] , or clustering-based coverage [ 13 ] . However, these methods treat multimodal samples holistically and do not explicitly distinguish the independent contribution of the visual modality. As a result, selected subsets may still favor linguistically easy samples or retain harmful misaligned supervision. In contrast, we posit that the value of a multimodal instruction sample largely depends on whether visual input is necessary for resolving predictive uncertainty. This perspective shifts the focus from generic sample importance to visual necessity as the guiding principle for multimodal data selection. To implement this principle, we draw inspiration from the theory of V-usable information [ 7 ] , which formalizes how much a variable reduces predictive uncertainty beyond what is already available from other contexts. While V-usable information has been used to assess textual data quality, extending this principle to multimodal instruction data remains underexplored. We argue that this perspective provides the right lens for evaluating visual supervision: a training instance is only as valuable as the uncertainty that can be uniquely resolved by visual input. Grounded in this insight, we introduce Visual Necessity Score (VisNec) , a lightweight data selection framework that explicitly quantifies the marginal contribution of the visual modality. By contrasting model losses with and without visual input, VisNec produces a principled per-sample measure of visual necessity, identifying instances that are vision-critical, redundant, or misaligned. Moreover, considering that visual dependency varies across task types e.g., geometric reasoning naturally demands stronger visual grounding than knowledge-based QA, we integrate stratified semantic clustering with intra-cluster VisNec ranking. This ensures that the selected subset maintains both visual indispensability and broad task coverage. Extensive experiments across ten benchmarks demonstrate that VisNec recovers or even surpasses full-data performance using only 15% of the training data, achieving 100.2% on LLaVA-665K and 115.8% on Vision-Flan-186K. Besides, our method consistently generalizes across model scales from 3B to 32B. 1. We identify a critical yet overlooked limitation in existing multimodal data selection: the neglect of the visual modality’s independent contribution, which leads to widespread “pseudo-multimodal” samples that reinforce linguistic shortcuts and hinder true cross-modal reasoning. 2. We propose VisNec , a lightweight and model-relative data selection framework that quantifies the marginal contribution of visual input, ensuring the selected subset is both visually indispensable and task-diverse. 3. We achieve state-of-the-art performance across diverse training datasets and benchmarks, demonstrating that measuring visual necessity enables both data efficiency and improved multimodal reasoning robustness. 2 Related Work Multimodal Instruction Tuning. To enable Multimodal Large Language Models (MLLMs) to effectively handle diverse downstream tasks, instruction tuning has emerged as a fundamental stage in the training pipeline. Established frameworks such as LLaVA [ 17 ] , InstructBLIP [ 6 ] , and Qwen-VL [ 1 ] demonstrate that fine-tuning on a broad range of instruction-following pairs is critical for aligning multimodal representations with human intent. However, the effectiveness of this alignment typically relies on the availability of large scale, high quality datasets. As these collections scale to hundreds of thousands of samples, the computational resources and training time required for fine-tuning become significant bottlenecks. Furthermore, large-scale instruction datasets, whether automatically collected or model-generated, inevitably include redundant and misaligned samples, which can weaken visual supervision and potentially degrade multimodal reasoning. Information-Theoretic Data Quality. A fundamental question in data-centric learning is how to quantify the value of individual training samples. V-usable information [ 7 ] formalizes this as the reduction in predictive uncertainty contributed by a variable beyond existing context, providing a principled view of feature-level informativeness. Building on this perspective, dataset difficulty measures [ 24 ] and loss-based scoring methods [ 14 ] treat predictive loss as an indicator of sample quality in LLM fine-tuning. These works demonstrate that loss-based signals are effective in single-modal settings. However, by operating on text alone, existing approaches cannot disentangle visual informativeness from linguistic ambiguity. A sample may appear difficult due to annotation noise or textual complexity rather than genuine visual contribution. By contrast, our work extends this information-theoretic perspective to multimodal

Read the complete text