Reasoning Multimodal Large Language Model: Data Contamination and Dynamic Evaluation
Ming Liu, Wensheng Zhang · 2025
Read Reasoning Multimodal Large Language Model: Data Contamination and Dynamic Evaluation by Ming Liu, Wensheng Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-07202.txt.
Introduction
Ming Liu and Wensheng Zhang propose dynamic evaluation to expose data contamination in multimodal reasoning models. Top benchmark scores may reflect leaked test data, not true skill. Reasoning models tuned from tainted bases inherit that risk.
Instead of altering inputs, their framework alters the task itself. The same image is used across a family of tasks. Questions, captions, question posing, and checks probe distinct skills. Models that overfit one task falter under such shifts. Genuine models hold up across the task family. An automated pipeline scores open answers with a calibrated judge. Paraphrase and corruption sampling steady those scores. The method tests generalization more honestly.
Catalog record
| CREATOR | Ming Liu, Wensheng Zhang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-07202
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Reasoning Multimodal Large Language Model: Data Contamination and Dynamic Evaluation Ming Liu, Wensheng Zhang Abstract Multimodal Large Language Models (MLLMs) show impressive vision-language benchmark performance, yet growing concerns about data contamination (test set exposure during training) risk masking true generalization. This concern extends to reasoning MLLMs, often fine-tuned via reinforcement learning from potentially contaminated base models. We propose a novel dynamic evaluation framework to rigorously assess MLLM generalization, moving beyond static benchmarks. Instead of perturbing inputs, we perturb the task itself. Using the same visual input, models are evaluated across a family of tasks (e.g., QA, captioning, question posing, verification) to probe diverse capabilities. This task perturbation reveals whether model performance is robust or reliant on superficial task-specific cues. Our approach is analogous to loss landscape sharpness: models overfit or contaminated for a single task (sharp minima) falter under task shifts, unlike models with generalizable solutions (flatter minima). We developed an automated pipeline with a calibrated judge scoring open-ended generations (captions, questions) using paraphrase and corruption sampling. Applying this framework to leading image/video MLLMs on benchmarks including MME, RealWorldQA, and CVRR-ES, we analyze each model’s cross-task “ability vector”. We demonstrate that fine-tuning on simulated test data (extreme contamination) drastically sharpens task-specific performance but harms overall generalization. Our dynamic task perturbation offers deeper insights into MLLM generalization, distinguishing genuine understanding from spurious leakage or overfitting. 1 Introduction Figure 1 : To mitigate contamination, we augment task and report scoring/sharpness metrics. Multimodal Large Language Models (MLLMs), combining vision and language, have advanced rapidly, with models such as GPT-4.1 ( openai2024gpt41 ) and open-source counterparts such as Qwen2-VL ( qwen_team2025qwen25vl ) showing strong performance in tasks such as visual question answering ( fu2023mme ) . More recently, reasoning-capable MLLMs, such as GPT-o4 mini ( openai2024o3o4mini ) , have attracted significant attention. However, the evaluation reliability of these MLLMs is increasingly challenged by benchmark data contamination ( chen2024we ) . As shown in Figure 1 , training on vast web-scale corpora makes it likely that test examples from popular benchmarks leak into training data ( golchin2024time ; oren2024proving ) . Consequently, models may achieve high scores by memorizing answers rather than demonstrating true understanding or reasoning, thus misrepresenting genuine generalization. This contamination extends to vision-language benchmarks ( yang2025dynamic ; chen2024we ) . A contaminated model might correctly answer previously seen questions, undermining reported accuracy. In short, performance gains on static benchmarks may overstate an MLLM’s true generalization ability , necessitating new evaluation methodologies. One proposed solution is dynamic evaluation on input space , which stress-tests models with input variations to assess true generalisation. Prior work explored input-level perturbations such as rephrasing, adding irrelevant context, or image modifications (e.g., cropping, noise) to measure consistency ( zhu2024dynamic ; yang2025dynamic ; liu2025robustnessmultimodallanguagemodel ) . However, these input perturbation approaches face significant limitations. Small perturbations (e.g., minor wording changes, slight image noise) often have negligible impact on output, particularly for robust LLM-based models adept at handling superficial prompt variations ( liu2025robustnessmultimodallanguagemodel ) . Memorizing contaminated models may still recognize slightly altered queries. Conversely, large perturbations risk altering the query’s semantic intent or introducing ambiguity. For instance, substantial rewording or inserted objects in image might create a new, potentially unclear question, making performance drops difficult to interpret: is it a failure of generalization or a consequence of a changed task? It is possible that simple input modifications such as synonym replacement do not reliably preserve the original evaluation objective, and their efficacy in eliminating memorized cues lacks rigorous validation. Thus, existing dynamic evaluation focusing on input perturbations struggles to provide a sensitive yet faithful test : minor changes can show false stability, while major changes could make the input ambiguous. We propose a fundamentally different paradigm for dynamic evaluation: perturb the task, not the input . Our insight is to hold the visual input constant, thereby preserving semantic content, while varying what we ask the model to do with that content. Evaluating an MLLM on a family of related tasks for the same image or video provides a multifaceted view of its understanding. Specifically, we define four tasks probing complementary reasoning skills with the same input: (1) Visual Question Answering (QA) , (2) Image/Video Captioning , (3) Question Generation (Question Proposing) , and (4) Answer Verification . For a given visual input, the model might answer a specific question (QA), describe it freely (captioning), generate a relevant question (generation), or verify a candidate answer (verification). Figure 2 conceptually illustrates this (same input, different prompts). These tasks are chosen as each requires overlapping visual knowledge but exercises distinct capabilities. By evaluating across this task spectrum , we better assess MLLM generalization. A robust model with an accurate internal visual representation should perform well across all tasks or show graceful degradation. In contrast, a model achieving high scores on a single-task benchmark (e.g., QA) via data leakage or narrow overfitting may exhibit a stark imbalance: succeeding at answering questions but failing at coherent description or relevant question generation for the same image. In essence, task perturbation is a stress test for model adaptability . If a model merely matches questions to answers by recall, this skill offers little help for generating a question or verifying an answer . Our dynamic evaluation thus exposes performance variations that static single-task evaluations miss. A theoretical perspective explains why task perturbation reveals contamination. If a model is fine-tuned on a leaked task t_{0} (e.g., QA), its parameters \theta are near a loss minimum L_{t_{0}}(\theta) , leading to memorization and a sharp minimum . Small deviations from t_{0} sharply increase error. When tested on a perturbed task t_{1} (e.g., captioning), an overfit model shows a large performance drop and high L_{t_{1}}(\theta) , with a large gradient \nabla_{\theta}L_{t_{1}} . In contrast, a model with better generalisation occupies a flatter region , with modest increases in L_{t_{1}}(\theta) , showing stable performance. Thus, task-space sharpness distinguishes memorization from general capability. Our framework uses the cross-task performance vector (\text{Score}_{t_{0}},\text{Score}_{t_{1}},\dots) to infer sharpness. Contaminated models show spiky profiles (high score on t_{0} , low on others), while robust models show flatter profiles. Fine-tuning on t_{0} ’s test set maximizes contamination, perfecting t_{0} ’s performance but degrading others, revealing the sharpness gap. Practical implementation of this dynamic evaluation faces several challenges. First, scoring subjective generative tasks (captioning, question generation) is non-trivial, unlike objective tasks (QA, answer verification) with ground truth. We address this by using a judge model . An reasoning MLLM (e.g., VL-Rethinker) are used to determine response quality, enabling quantitative scoring of captions and questions proposed with reasoning process. Second, a unified per-input pipeline integrates all four task evaluations. An offline script per visual tests: (a) QA (vs. ground truth), (b) captioning (judge-scored), (c) question generation (judge-scored), and (d) answer verification (vs. varied ground-truth pairs). This yields rich per-model, per-input metrics. Finally, results are aggregated for the cross-task analysis . Contributions. Our primary contributions are: • New Evaluation Framework: Introduction of dynamic task perturbation , a novel MLLM dynamic evaluation that perturbs the task (objective), not the input, to probe generalization. This general method is extendable. • Task Family Design: Design of a four-task family (QA, captioning, question generation, answer verification) using the same visual input to test complementary reasoning