Migician: Revealing the Magic of Free-Form Multi-Image Grounding in Multimodal Large Language Models
You Li, Heyu Huang, Chi Chen, Kaiyu Huang, Chao Huang, Zonghao Guo, Zhiyuan Liu, Jinan Xu, Yuhua Li, Ruixuan Li, Maosong Sun · 2025
Read Migician: Revealing the Magic of Free-Form Multi-Image Grounding in Multimodal Large Language Models by You Li, Heyu Huang, Chi Chen, Kaiyu Huang, Chao Huang, Zonghao Guo, Zhiyuan Liu, Jinan Xu, Yuhua Li, Ruixuan Li, Maosong Sun (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-05767.txt.
Introduction
You Li and ten colleagues build the first model that can point to objects across many images from free-form text. Multimodal models read single images well and grasp groups loosely. Precise pointing inside a group, though, has lagged. Chain-of-thought tricks that chain single-image pointing prove shaky.
Their model, Migician, learns the skill end to end. A new data set of over six hundred thousand examples covers many multi-image pointing tasks. It adds fresh free-form instructions generated for this purpose. A dedicated benchmark, MIG-Bench, scores the skill directly. Migician beat the best existing models by a wide margin on it. It even passed models many times its size. Code, model, data, and benchmark are all released openly.
Catalog record
| CREATOR | You Li, Heyu Huang, Chi Chen, Kaiyu Huang, Chao Huang, Zonghao Guo, Zhiyuan Liu, Jinan Xu, Yuhua Li, Ruixuan Li, Maosong Sun |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-05767
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Migician: Revealing the Magic of Free-Form Multi-Image Grounding in Multimodal Large Language Models You Li, Heyu Huang, Chi Chen, Kaiyu Huang, Chao Huang, Zonghao Guo, Zhiyuan Liu, Jinan Xu, Yuhua Li, Ruixuan Li, Maosong Sun Abstract The recent advancement of Multimodal Large Language Models (MLLMs) has significantly improved their fine-grained perception of single images and general comprehension across multiple images. However, existing MLLMs still face challenges in achieving precise grounding in complex multi-image scenarios. To address this, we first explore a Chain-of-Thought (CoT) framework that integrates single-image grounding with multi-image comprehension. While partially effective, it remains unstable and struggles to capture abstract visual information due to its non-end-to-end nature. Therefore, we introduce Migician , the first multi-image grounding model capable of performing free-form and accurate grounding across multiple images. To support this, we present the MGrounding-630k dataset, which comprises data for several multi-image grounding tasks derived from existing datasets, along with newly generated free-form grounding instruction-following data. Furthermore, we propose MIG-Bench, a comprehensive benchmark specifically designed for evaluating multi-image grounding capabilities. Experimental results demonstrate that our model achieves significantly superior multi-image grounding capabilities, outperforming the best existing MLLMs by 24.94% and even surpassing much larger 70B models. Our code, model, dataset, and benchmark are fully open-sourced at https://migician-vg.github.io/ . Figure 1 : Left : Examples of free-form multi-image grounding. The task is to identify and localize relevant visual regions across multiple images based on a free-form query. Right : Our proposed model, Migician, significantly outperforms other MLLMs on various multi-image grounding tasks. 1 Introduction Multimodal Large Language Models (MLLMs) have exhibited significant advancements recently, demonstrating exceptional cross-modal understanding capabilities and achieving outstanding performance in various vision-language tasks [ 49 , 13 , 7 , 16 , 30 , 1 , 21 ] . As these models continue to evolve, their capabilities have expanded beyond image-level understanding to include fine-grained visual grounding [ 40 , 5 , 51 ] . This enables MLLMs to process region-specific inputs and outputs, unlocking a broader spectrum of real-world multimodal application scenarios [ 35 ] . Despite the promising visual grounding capabilities demonstrated by existing MLLMs, these abilities are largely confined to single-image scenarios [ 20 , 51 ] . The potential of MLLMs in free-form multi-image grounding (MIG) remains underexplored. Free-form MIG challenges the model to perform grounding across multiple images effectively, where the input queries and image contexts can be organized in arbitrary forms, enabling flexible and dynamic interactions. For instance, as shown in Figure 1 , the model must understand the white car in the query image and relate it to the textual prompt "black in color" to identify the corresponding target in the target image. This capability unlocks a wide range of applications, such as fine-grained environmental perception in autonomous driving [ 42 ] , anomaly detection in surveillance systems [ 2 ] , and target localization for embodied robotics [ 12 ] . To address the free-form MIG, the model needs to possess the capability for visual grounding while achieving cross-image understanding. As a result, a question naturally arises: Can we integrate the single-image grounding and multi-image understanding capabilities of existing MLLMs to tackle the MIG task? In this work, we propose a Chain-of-Thought (CoT) framework that first leverages multi-image understanding to generate a textual referring query, and then utilizes it for localization through single-image grounding. This approach has been proven highly effective for MIG tasks, particularly in simple scenarios where textual descriptions are sufficiently distinctive, demonstrating the potential of current MLLMs in handling such tasks. However, the proposed CoT framework struggles with describing abstract visual semantics in multi-image scenarios, and the two-step process results in a doubling of the inference time. To address this, we further propose Migician, a competitive MLLM capable of free-form and accurate grounding across multiple images, serving as an end-to-end solution for MIG. To progressively establish flexible grounding capabilities, we employ a two-stage training procedure based on our proposed large-scale MIG dataset (MGrounding-630k). First, the grounding ability of Migician is enhanced through a combination of data from MIG tasks and general tasks. Then, Migician is further refined using high-quality free-form MIG instruction data. In addition, to evaluate the challenges of the free-form MIG scenario, we construct a comprehensive multi-image grounding benchmark, MIG-bench, comprising a total of 10 different tasks, 5.9k diverse images and more than 4.2k test instances. We observe a significant gap between the performance of existing mainstream MLLMs and human performance on the MIG-bench. In contrast, Migician can effectively alleviate this gap and improve the performance of free-form MIG. To sum up, our contributions can be concluded as follows: • We explore the task of multi-image grounding for MLLMs and reveal the potential and challenges of current MLLMs by through a proposed CoT framework. • We introduce Migician, the first MLLM capable of effectively performing free-form MIG. We also present MGrounding-630k, the first large-scale MIG instruction tuning dataset for training this model. • We introduce MIG-Bench, a comprehensive benchmark for evaluating multi-image grounding capabilities. Experimental results demonstrate that Migician significantly outperforms the current best methods. 2 Related Work Multimodal Large Language Models Recent developments in multimodal large language models (MLLMs) have shifted from single image-text understanding towards more versatile capabilities [ 3 , 48 , 41 , 24 ] . Among these efforts, some focus on enabling models to achieve fine-grained visual grounding, either through simple instruction tuning [ 5 , 35 ] or by integrating additional auxiliary visual components [ 51 , 54 , 4 ] . However, these models primarily focus on visual grounding within a single image. Some other studies explore multi-image understanding tasks, such as multi-image comparison, reasoning, and temporal comprehension [ 18 , 25 , 50 , 24 , 3 , 48 ] . Nevertheless, fine-grained visual grounding at the multi-image level remains an underexplored area. To the best of our knowledge, our proposed Migician is the first MLLM designed to address the challenge of multi-image grounding. MLLM Benchmarks Most existing benchmarks for evaluating MLLMs focus on single-image tasks [ 9 , 23 ] . A few recent benchmarks have started assessing the performance of MLLMs on multi-image understanding [ 18 , 31 , 11 , 39 , 28 ] , but they primarily emphasize image-level comprehension. The most relevant benchmark to our work is MC-Bench [ 45 ] , a contemporaneous study. MC-Bench evaluates the multi-context grounding capabilities of MLLMs by asking them to accurately locate the corresponding object based on a text prompt in the correct image from a given pair. However, it exhibits limitations in the fixed number of input images and the restricted forms of queries. In contrast, the proposed MIG-Bench in this work offers more flexible task formats, focusing on evaluating models’ capabilities in free-form multi-image understanding. Figure 2 : An illustration of the multi-image grounding tasks included in MIG-Bench. These tasks are divided into two categories: spontaneous grounding and referential grounding, depending on the whether there are explicit referential requirements. 3 Task Definition The task of free-form multi-image grounding is to identify and localize relevant visual regions across a set of images based on a free-form query. Unlike traditional grounding tasks with fixed input formats, the query in free-form multi-image grounding can