SMiR: Efficient S ynthetic Data Pipeline to Improve M ulti- i mage R easoning
Andrew Li, Rahul Thapa, Rahul ChalamalaQingyang WuKezhen Chen, James Zou Together AI · 2025
Read SMiR: Efficient S ynthetic Data Pipeline to Improve M ulti- i mage R easoning by Andrew Li, Rahul Thapa, Rahul ChalamalaQingyang WuKezhen Chen, James Zou Together AI (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-03675.txt.
Introduction
Six authors build a synthetic data pipeline for multi-image reasoning in this 2025 paper. Vision-language models do well on single images, but reasoning across several images at once lacks both training data and good test sets in the open community.
Their pipeline, SMiR, finds related images through multimodal embeddings, merges image and text information, and uses open language models to write reasoning instructions. It produced 160,000 training samples at low cost. The team also built a benchmark of 200 examples across seven hard tasks, judged by a model over multi-turn answers. Fine tuning open models on the synthetic data improved their scores on that benchmark.
Catalog record
| CREATOR | Andrew Li, Rahul Thapa, Rahul ChalamalaQingyang WuKezhen Chen, James Zou Together AI |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-03675
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SMiR: Efficient S ynthetic Data Pipeline to Improve M ulti- i mage R easoning Andrew Li, Rahul Thapa, Rahul ChalamalaQingyang WuKezhen Chen, James Zou Together AI Abstract Vision-Language Models (VLMs) excel at understanding single images, aided by high-quality instruction datasets. However, multi-image reasoning remains underexplored in the open-source community due to two key challenges: (1) scaling datasets with correlated images and complex reasoning instructions is resource-intensive, and (2) robust evaluation benchmarks for multi-image tasks are lacking. To address this, we introduce SMiR , a synthetic data-generation pipeline for multi-image reasoning, along with a high-quality dataset generated using this pipeline. SMiR efficiently extracts correlated images via multimodal embeddings, integrates visual and descriptive information, and leverages open-source LLMs to generate quality instructions. Using this approach, we produce 160K synthetic training samples, offering a cost-effective alternative to closed-source solutions. Additionally, we present SMiR-Bench , a multi-image reasoning benchmark comprising 200 diverse examples across seven complex reasoning tasks. SMiR-Bench is multi-turn and employs a VLM judge to evaluate free-form responses, providing a comprehensive assessment of model expressiveness and reasoning capability across modalities. We demonstrate the effectiveness of SMiR by fine-tuning open-source VLMs and evaluating them on SMiR-Bench . Our codebase is available at https://github.com/togethercomputer/SMiR . 1 Introduction Vision-Language Models (VLMs) have demonstrated strong performance on single-image tasks, particularly open-source models trained with high-quality instruction datasets ( Laurençon et al., 2024b ; Zhang et al., 2023 ; Xu et al., 2022 ) . However, open-source VLMs struggle with multi-image reasoning tasks—such as comparing and analyzing relationships between multiple images—lagging behind closed-source models like GPT-4 ( Achiam et al., 2023 ) , Claude 3.5 Sonnet ( Anthropic, 2024 ) , and Gemini 1.5 ( Reid et al., 2024 ) . This performance gap likely stems from the superior multi-image datasets available to closed-source models, which are essential for effective reasoning across multiple images. A key challenge in multi-image reasoning is collecting and curating large-scale datasets with highly correlated images. Existing datasets such as MANTIS ( Jiang et al., 2024 ) often pair unrelated images, simplifying the task and limiting the complexity of reasoning. MMDU-45K ( Liu et al., 2024f ) improves upon this by clustering images based on captions and category tags but overlooks purely visual relationships. Consequently, datasets that explicitly ensure image correlation are needed to advance complex multi-image reasoning. Scaling such datasets further exacerbates the challenge. Human curation, as required in datasets like MultiInstruct ( Xu et al., 2022 ) , is resource-intensive. While synthetic datasets generated via GPT-4 ( Peng et al., 2023 ; Wang et al., 2023 ) offer a scalable alternative ( Liu et al., 2024e ; Li et al., 2024a ; Li et al., 2023a ) , they remain expensive and difficult to scale effectively. Another challenge is evaluation. Existing multi-image benchmarks, such as MMIU ( Meng et al., 2024 ) and MMMU ( Yue et al., 2024 ) , rely on multiple-choice questions, failing to assess the depth of a model’s reasoning. While some benchmarks incorporate free-form responses ( Liu et al., 2024c ) , they lack multi-turn interactions, which are essential for evaluating complex reasoning processes. To address these issues, we introduce SMiR , a synthetic data generation pipeline for multi-image reasoning, and SMiR-Bench , a multi-image evaluation benchmark. Our key contributions are: • We propose SMiR , a pipeline that efficiently identifies correlated images using multimodal embeddings, applying cluster sampling and graph iteration sampling to ensure diversity and robustness in instruction tuning. • We develop a scalable synthetic data generation pipeline using open-source LLMs such as Meta Llama 3.1 70B Instruct Turbo ( Dubey et al., 2024 ) , reducing costs by up to 50x and increasing speed by up to 10x compared to closed-source alternatives ( Kirkovska, 2024 ) . • We introduce SMiR-Bench , a multi-turn evaluation benchmark that assesses models via free-form responses and reasoning justifications, using GPT-4-Turbo and other baselines for comparison. Models trained on SMiR improve win rates by up to 8% over existing instruction-tuned models. By advancing both dataset generation and evaluation, SMiR facilitates the development of stronger open-source VLMs for multi-image reasoning. Table 1: Comparison of datasets highlighting key characteristics and methodologies. Datasets Multimodal Embedding Correlated Images No Human-Annotation Open-Source LLM Mantis Jiang et al. (2024) ✗ ✗ ✗ ✗ MMDU Liu et al. (2024f) ✗ ✓ ✗ ✗ SMiR ✓ ✓ ✓ ✓ Figure 1: Our end-to-end pipeline converts image-caption pairs into synthetic multi-turn conversations using multimodal embeddings, strategic sampling, and LLM prompting. The example, based on a sports scenario, illustrates how the pipeline generates contextually rich dialogues by leveraging visual relationships. Figure 2: Comparison of an example from the MANTIS dataset, which includes samples from ShareGPT4V-PT where unrelated images such as animals and other objects are concatenated (top), vs. an example from the SMiR dataset, where only related images are grouped together (bottom), for multi-image reasoning. 2 SMiR: Synthetic Multi-Image Reasoning Data Pipeline Starting with a large-scale image-caption dataset D containing N image-caption pairs (I_{i},C_{i})_{i=1}^{N} , SMiR first constructs multimodal embeddings for each pair. Then, sampling algorithms identify correlations between these embeddings. Finally, open-source LLMs are used to generate complex question-answer pairs based on the sampled correlations. The end-to-end pipeline is shown in Fig. 1 . 2.1 Multimodal Embedding Construction Embedding-based approaches have been widely used to model relationships between images and text. However, relying solely on image or caption embeddings limits the ability to fully capture the connections between visual content and descriptive text. A more comprehensive approach requires integrating both modalities. To achieve this, we construct a multimodal embedding by combining SigLIP or CLIP image embeddings with their corresponding caption embeddings: E_{\text{multimodal}}=E_{\text{image}}+c\cdot E_{\text{caption}} (1) where E_{\text{multimodal}} represents the fused embedding, and c adjusts the weight of the caption embedding. This parameter is tuned empirically based on small-scale human evaluations. For the ShareGPT4V dataset ( Chen et al., 2023 ) , c=0.2 provided the best balance between visual and textual contributions. The optimal value may vary depending on dataset quality and caption descriptiveness. This multimodal approach enables better identification and clustering of correlated images by capturing both visual and textual relationships. We apply UMAP ( McInnes et al., 2018 ) for dimensionality reduction, preserving essential data structures while facilitating efficient analysis and visualization. 2.2 Random Sampling with Iteration After constructing multimodal embeddings, we apply a random sampling algorithm to select semantically related images, ensuring diversity while preserving coherence. Given a set of embeddings X=\{x_{1},...,x_{n}\} , the algorithm proceeds as follows: 1. Randomly selects an initial embedding and initializes the selected set S . 2. Iteratively selects embeddings until reaching N samples, using the probability distribution: p(x_{j})\propto\frac{1}{\sum_{u\in S}\|x_{j}-x_{u}\|^{k}+\epsilon},\quad x_{j}\in X\setminus S (2) 3. Samples the next embedding based on this probability, favoring closer points. Here, k (default: 12) controls the emphasis on proximity—higher values prioritize nearby points more strongly by amplifying distance differences. This approach balances semantic coherence and diversity, allowing for meaningful multi-image relationships while introducing some randomness. For example, as shown in Fig. 6 ( Section C.2 ), this method effectively clusters related animal images while avoiding the disconnected groupings seen in simpler sampling methods. The full algorithm and pseudocode are provided in Algorithm 2 ( Section B.2 ). By assembling related images before prompting the