MMCR: Advancing Visual Language Model in Multimodal Multi-Turn Contextual Reasoning
Dawei Yan, Yang Li, Qingguo Chen, Weihua Luo, Peng Wang, Haokui Zhang, Chunhua Shen · 2025
Read MMCR: Advancing Visual Language Model in Multimodal Multi-Turn Contextual Reasoning by Dawei Yan, Yang Li, Qingguo Chen, Weihua Luo, Peng Wang, Haokui Zhang, Chunhua Shen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-18533.txt.
Introduction
Seven researchers build a data set for multi image, multi turn chat in a 2025 paper. Real talks with AI weave many turns and many images, which trains models to reason with context. Most vision language models still train and test on single turns.
Their set, MMCR, holds 310 thousand contextual dialogues, each with one to four images and four or eight turns. A bench set spans eight domains and forty sub topics for diagnosis. Models tuned on the big set gained clear contextual accuracy on the bench. Gains also showed on three known benchmarks, which the team reads as broad value.
Catalog record
| CREATOR | Dawei Yan, Yang Li, Qingguo Chen, Weihua Luo, Peng Wang, Haokui Zhang, Chunhua Shen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-18533
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MMCR: Advancing Visual Language Model in Multimodal Multi-Turn Contextual Reasoning Dawei Yan, Yang Li, Qingguo Chen, Weihua Luo, Peng Wang, Haokui Zhang, Chunhua Shen Abstract Compared to single-turn dialogue, multi-turn dialogue involving multiple images better aligns with the needs of real-world human-AI interactions. Additionally, as training data, it provides richer contextual reasoning information, thereby guiding the model to achieve better performance. However, existing vision-language models (VLMs) primarily rely on single-turn dialogue training and evaluation benchmarks. In this paper, following the characteristics of human dialogue, such as focused topics and concise, clear content, we present MMCR (Multimodal Multi-turn Contextual Reasoning), a novel dataset comprising: (1) MMCR-310k- the largest multi-image multi-turn instruction tuning dataset with 310K contextual dialogues, each covering 1-4 images and 4 or 8 dialogue turns; and (2) MMCR-Bench - a diagnostic benchmark featuring dialogues, spanning 8 domains (Humanities, Natural, Science, Education, etc.) and 40 sub-topics. Extensive evaluations demonstrate that models fine-tuned with MMCR-310k achieve 5.2% higher contextual accuracy on MMCR-Bench, while showing consistent improvements on existing benchmarks (+1.1% on AI2D, +1.2% on MMMU and MMVet). MMCR and prompt engineering will be released publicly. 1 1 footnotetext: Corresponding author. 1 Introduction Figure 1 : Samples comparison. On the left side: two examples corresponding to a good contextual reasoning sample and a bad contextual reasoning sample with errors during multi-turn dialogue. On the right side: a single-image, single-turn dialogue sample from MMBench [ 30 ] , a multi-turn, multi-image dialogue sample from MMDU [ 32 ] , and a sample from our proposed MMCR. The pursuit of establishing Artificial General Intelligence (AGI) capable of providing expert-level responses has long been a pressing goal within the academic community. Such mature agents are expected to engage in multi-turn interactions with human users, process long-context dialogue information, and demonstrate multi-turn instruction following ability, which are also increasingly common requirements in real-world scenarios. In response to the current state, numerous outstanding studies and benchmarks in the domain of Large Language Models (LLMs) have been developed to enhance their ability to handle long-context multi-turn dialogue scenarios, including but not limited to sparse and efficient attention mechanisms [ 9 , 27 ] , position encoding extrapolation and dynamic adjustment [ 37 ] , context summarization and memory bank construction, as well as various multi-turn dialogue fine-tuning datasets and evaluation benchmarks [ 5 ] . However, for open-source VLMs, the mainstream training data and evaluation benchmarks [ 30 ] are largely confined to handling single-image, single-interaction modes. However, in real-world human-AI interactions, multi-turn dialogues involving multiple images better align with human conversational habits and practical application needs. Furthermore, multi-turn and multi-image dialogue datasets contain richer contextual reasoning information, which significantly aids in enhancing the performance of data-driven VLMs. To overcome this problem, some researchers have attempted to address this issue from the perspective of building a data foundation [ 8 , 5 ] . However, most of these efforts focus solely on dialogue content. Very recently, MMDU [ 32 ] has constructed high-quality training data and benchmarks using GPT-4o based on Wikipedia to enhance and evaluate VLMs in such scenarios, but it does not emphasize strengthening contextual logic, maintaining cross-turn dialogue consistency, or accurately parsing long-range contextual dependencies. These capabilities are essential for deploying AGI in real-world applications involving continuous multimodal inputs. Moreover, logical reasoning with long-contexts also contributes to enhancing the model’s performance on public benchmarks. Therefore, there is an urgent need in academia to develop a multi-dimensional training data and evaluation framework that includes complex multi-turn dialogues, strong contextual associations, and dynamic scene adaptation. Building on this, we introduce MMCR (Multimodal Multi-turn Contextual Reasoning), which includes MMCR-310k and MMCR-Bench, a multi-turn instruction fine-tuning dataset with a mix of single-image and multi-image interactions, and an evaluation benchmark. This instruction fine-tuning dataset and evaluation benchmark are constructed based on the multimodal text-image interleaved dataset OmniCorpus [ 24 ] , though GPT-4o [ 3 ] with the guidance of carefully designed prompt engineering. The generated multi-turn dialogue datasets, which feature strong contextual logic and simulate real-world user-agent interactions, also conduct evaluations on specific dimensions of multi-turn dialogue contextual reasoning. Specifically, MMCR features the following characteristics: (1) Multi-turn Interaction and Realism: MMCR-310k contains 210k single-image and 100k multi-image multi-turn dialogue data, with dialogues focused on images over four and eight rounds, respectively, aiming to simulate real-world user-chatbot interactions. (2) Strong Contextual Relevance and Logical Progression: We emphasize that each round of dialogue should progressively delve into image details, inter-image relationships, and related themes, while maintaining a clear contextual association. Subsequent questions must build upon prior requests or responses. (3) Evaluation Openness and Extensiveness: We utilize multi-level annotators provided by [ 44 ] to label images from OmniCorpus [ 24 ] , selecting labeled samples from eight major domains—Humanities, Nature, Education, Science, Mechanics, Animals, Plants, and Architecture—across 40 sub-topics for the evaluation set. Using GPT-4o as the evaluator, we score the model’s responses across five dimensions: Precision and Conciseness, Consistency of Contextual References, Logical Contextual Relationship, Clarity of Dialogue Theme, and Absence of Redundancy. This comprehensive evaluation assesses the model’s performance in long-context multi-turn dialogues. Several samples from difference datasets are compared in Fig. 1 We validate MMCR using Ovis [ 36 ] , a state-of-the-art (SOTA) VLM architecture that innovatively employs a learnable visual embedding table for visual encoding. After instruction fine-tuning with MMCR, Ovis achieved obviously improvement on MMCR-Bench, which revealed the limitations of existing models in logical contextual reasoning in multi-turn dialogues, and validated the necessity of constructing such a dataset. Besides, incorporating our proposed MMCR into the supervised fine-tuning stage further enhances performance on existing benchmarks. For instance, building on Ovis’s significant outperformance over other open-source models—even surpassing GPT-4o on some metrics—it achieves additional improvements on multiple benchmarks such as AI2D [ 20 ] , MMMU [ 49 ] , and MMV [ 48 ] . Furthermore, based on our detailed analysis of the experimental results, we identify a subtle yet critical point to consider when training large models, which we refer to as the “less is more” phenomenon. During the high-quality supervised fine-tuning phase, simply adding more data does not always lead to better performance. Maintaining a balanced proportion of data across different task types is equally important. In fact, to preserve this balance, reducing the amount of data for certain task types can sometimes yield better results. This phenomenon is particularly pronounced for models with fewer parameters. This finding provides valuable insights for subsequent large model training, particularly in supervised fine-tuning. In summary, the major contributions are: • We propose MMCR, a multimodal collection featuring a hybrid instruction-tuning dataset (310K dialogues with 4-8 turns) and a GPT-4o-grounded benchmark, specifically designed to enhance and assess the multi-turn dialogue contextual reasoning ability of VLMs in real-world human-AI dialogues. • Systematic evaluation shows that MMCR enables precise and effective improvements and assessments in the context of multimodal multi-turn dialogues. It also shows enhanced performance on existing benchmarks. • We discover the “Less is More” phenomenon in supervised fine-tuning of large models and demonstrated it through experiments. This phenomenon challenges the conventional belief that “more data is always better” and emphasizes the importance of data distribution in addition to data volume. It provides valuable insights for future large model