DataProphet : Demystifying Supervision Data Generalization in Multimodal LLMs

Xuan Qi γ \gamma τ \tau, Luxi He ρ \rho, Dan Roth γ \gamma, Xingyu Fu γ \gamma ρ \rho · 2026

Read DataProphet : Demystifying Supervision Data Generalization in Multimodal LLMs by Xuan Qi γ \gamma τ \tau, Luxi He ρ \rho, Dan Roth γ \gamma, Xingyu Fu γ \gamma ρ \rho (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-19688.txt.

Introduction

Qi and co-authors present DataProphet, a training-free metric for picking supervision data for multimodal language models. The folk rule says pick data that looks like the target benchmark. Their study of transfer across 14 data sets and seven tasks shows looks mislead. Transfer depends on the specific data set, not its broad task type. DataProphet mixes perplexity, similarity, and diversity into one score. Its rankings match post-training gains with a Kendall tau of 86 percent. Picks from the metric beat uniform selection by up to about 7 percent.

Catalog record

CREATORXuan Qi γ \gamma τ \tau, Luxi He ρ \rho, Dan Roth γ \gamma, Xingyu Fu γ \gamma ρ \rho
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39021
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.19688

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DataProphet : Demystifying Supervision Data Generalization in Multimodal LLMs Xuan Qi γ \gamma τ \tau, Luxi He ρ \rho, Dan Roth γ \gamma, Xingyu Fu γ \gamma ρ \rho Abstract Conventional wisdom in selecting supervision data for multimodal large language models (MLLMs) is to prioritize datasets that are intuitively similar to the target task ( e.g . text-rich v.s. vision-centric). However, it remains unclear how reliably such similarity translates into improved performance on the test benchmarks. In this paper, we take the first step to study the problem in MLLMs: can we predict a training data’s influence on a target benchmark even before any training takes place? To answer this question, we first conduct an in-depth analysis using 14 vision-language datasets covering 7 diverse tasks. Our analysis shows that intuitive task similarity is unreliable in predicting task generalizability, and that transfer depends on the specific dataset rather than the broader task category. We propose DataProphet , a training-free, simple yet effective metric based on multimodal perplexity, similarity, and data diversity.Our experiments demonstrate that the influence rankings for different supervision datasets derived from DataProphet is strongly-correlated with rankings based on the actual performance increase after training, with a Kendall’s τ \tau correlation coefficient of 86.0%. Moreover, we show that DataProphet can help select better supervision data, achieving up to 6.9% improvement in average over uniform selection, 1.4% over SoTA training-based baseline, and 0.2% higher than oracle experiment performance-based selection. Our code and data will be released. 1 Introduction Training data is one of the most important deciding factors for Multimodal Large Language Model (MLLM) performance ( 18 ; 2 ; 29 ; 3 ; 37 ; 4 ; 44 ; 26 ; 33 ) . Given the vast amount of multimodal supervision datasets, how should we effectively utilize them towards certain training targets? Some prior work has explored using high-quality data and similar tasks’ data during MLLM supervised fine-tuning stage ( 39 ; 38 ) . However, these methods still require some training and mainly improve supervision performance on target tasks by removing irrelevant, redundant, or low-quality data from the training mixture. In this work, we further demystify how training data affects a test benchmark’s performance. We pose the following research question in an MLLM setting: Given a training dataset D_{i} , can we predict the influence (measured by relative performance change) on a target benchmark T_{j} even before any training takes place? To answer this question, we first conduct a comprehensive experiment utilizing 14 vision-language fine-tuning datasets covering 7 diverse tasks (2 for each task): OCR, chart understanding, spatial reasoning, counting, knowledge-based QA, document understanding, and map understanding. We surprisingly find that different training datasets’ influence on a target benchmark may contradict human intuition. For example, humans may intuitively assume that training the model on OCR task data will help its performance on chart tasks more than spatial reasoning tasks, since OCR and chart both require extracting text and numbers in an image. However, experiment results as in Figure 1 (a) and (b) indicate the reverse: training on OCR improves spatial reasoning tasks more than chart tasks. This counter-intuitive finding suggests that model’s generalization across different tasks depends on more than surface-level similarity. We also find that performance improvement are not decided by task category but are dependent on individual datasets: datasets from the same task category do not necessarily help each other the most, and do not share same influence on the same target benchmark. Figure 1: Three major takeaways in DataProphet : (a) Surprisingly, human intuition about similarity between training data and test data turns out to be a poor indicator of model performance influence. In contrast, our designed DataProphet metric more reliably predicts the influence of training data on test benchmarks; (b) the impact of multimodal supervision is decided by specific individual datasets, rather than by broad task categories: datasets from the same task category do not necessarily help each other the most, and do not share similar influence on the same target benchmark. Here, OCR-VQA ( 25 ) and Screen-QA ( 11 ) are both OCR data, ChartQA ( 23 ) and Chart2Text ( 16 ) are both chart question answering benchmarks, and Open-Spatial 7 and CLEVR spatial ( 14 ) are both spatial reasoning benchmarks; (c) DataProphet provides an effective approach for training-free data selection under fixed compute budgets ( i.e .fixed number of total training samples). We compute influence of each supervision dataset based on a combined set of testing benchmarks and select data according to the influence ratio, yielding consistent improvements across 14 tasks, with average gains of +3.4% and +6.9% under real and synthetic data settings. Based on our analysis, we extract factors that truly matter for predicting such influence. We introduce DataProphet , an extremely simple yet effective, training-free metric which uses text and visual similarity, multimodal perplexity, and data diversity to predict the data influence ( Equation 3 ). We evaluate the ranking of 14 training datasets’ influence compared to ranking using real training effect, measured by the benchmark’s relative improvement over baseline. Our experiments show strong correlation between the two rankings (86.0% Kendall’s \tau ), demonstrating the usefulness of DataProphet in predicting data influence rankings in scale. Notably, the most important factors for prediction are multimodal perplexity and visual similarity, increasing 37.3% and 23.5% of Kendall’s \tau respectively. In addition to predicting source data usefulness, we also show that given target benchmarks, DataProphet can be applied to curate good instruction-tuning data under fixed compute and data budget. We conduct experiments under two different settings: selecting instruction-tuning data from (1) real data pool and (2) synthetic data pool. For real data, DataProphet performs 3.4% better than uniform selection and is 1.4% better than ICONS ( 38 ) , a SoTA selection requiring training. We observe even larger performance gain in synthetic data selection, reaching almost 6.9% performance increase compared to the uniform selection baseline and 1.2% improvement over ICONS. These show promise for DataProphet as a light-weight yet effective measure for choosing supervision data for MLLMs. Overall, this work performs a deep analysis of different source datasets’ influence on target benchmarks’ performance, with counter-intuitive findings. Our analysis motivates DataProphet , a simple training-free yet effective metric to predict the usefulness of different training datasets on the target. We further demonstrate the promising effect of DataProphet to guide MLLM instruction-tuning data selection that outperforms SoTA training-based selection. 2 Data Influence Analysis In this section, we demonstrate how we quantitatively analyze data influence across multiple visual question answering (VQA) tasks. Specifically, we first select a base Multimodal Large Language Model (MLLM): InternVL3 ( 44 ) , and 14 diverse VQA datasets that comes with both train (source) and test (target) sets. Then, we conduct supervised fine-tuning on our base model with each source dataset individually, and evaluate relative performance gains on all target sets. The goal is to demystify data influence from real experiment results. We will expand this section by introducing the experiment setup, followed with results and observations. 2.1 Analysis Setups Benchmark and Task Selection. We try to include as diverse benchmarks as possible and mainly select three types of tasks: text-rich tasks such as OCR, chart, and document; general

Read the complete text