LIRA: Inferring Segmentation in Large Multi-modal Models with Local Interleaved Region Assistance
Zhang Li, Biao Yang, Qiang Liu, Shuo Zhang, Zhiyin Ma, Shuo Zhang, Liang Yin, Linger Deng, Yabo Sun, Yuliang Liu, Xiang Bai · 2025
Read LIRA: Inferring Segmentation in Large Multi-modal Models with Local Interleaved Region Assistance by Zhang Li, Biao Yang, Qiang Liu, Shuo Zhang, Zhiyin Ma, Shuo Zhang, Liang Yin, Linger Deng, Yabo Sun, Yuliang Liu, Xiang Bai (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-06272.txt.
Introduction
Zhang Li and co-authors present LIRA, a framework that helps large multimodal models segment objects more faithfully. Such models often outline the wrong region or invent details. Weak visual grasp and coarse perception cause both faults.
LIRA joins comprehension and segmentation so each aids the other. One part fuses semantic and pixel features for better attributes. Better attributes lead to tighter segmentation masks. A second part writes local descriptions from each masked region. Those descriptions give fine supervision that cuts hallucination. The team also links segmentation precision to token semantics. Their measure quantifies that hidden relationship. The work targets grounded vision-language systems for real tasks.
Catalog record
| CREATOR | Zhang Li, Biao Yang, Qiang Liu, Shuo Zhang, Zhiyin Ma, Shuo Zhang, Liang Yin, Linger Deng, Yabo Sun, Yuliang Liu, Xiang Bai |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-06272
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LIRA: Inferring Segmentation in Large Multi-modal Models with Local Interleaved Region Assistance Zhang Li, Biao Yang, Qiang Liu, Shuo Zhang, Zhiyin Ma, Shuo Zhang, Liang Yin, Linger Deng, Yabo Sun, Yuliang Liu, Xiang Bai Abstract While large multi-modal models (LMMs) demonstrate promising capabilities in segmentation and comprehension, they still struggle with two limitations: inaccurate segmentation and hallucinated comprehension. These challenges stem primarily from constraints in weak visual comprehension and a lack of fine-grained perception. To alleviate these limitations, we propose LIRA, a framework that capitalizes on the complementary relationship between visual comprehension and segmentation via two key components: (1) Semantic-Enhanced Feature Extractor (SEFE) improves object attribute inference by fusing semantic and pixel-level features, leading to more accurate segmentation; (2) Interleaved Local Visual Coupling (ILVC) autoregressively generates local descriptions after extracting local features based on segmentation masks, offering fine-grained supervision to mitigate hallucinations. Furthermore, we find that the precision of object segmentation is positively correlated with the latent related semantics of the <seg> token. To quantify this relationship and the model’s potential semantic inferring ability, we introduce the Attributes Evaluation (AttrEval) dataset. Our experiments show that LIRA achieves state-of-the-art performance in both segmentation and comprehension tasks. Code will be available at https://github.com/echo840/LIRA . † † footnotetext: ∗ Corresponding author 1 Introduction With the development of large multi-modal models, their capabilities have expanded from visual comprehension to pixel-wise segmentation by integrating LMMs with segmentation modules. The pioneering work LISA [ 19 ] first introduced the embedding-as-mask paradigm to unlock segmentation capabilities and proposed the task of reasoning segmentation, which requires models to generate binary segmentation masks based on implicit text queries. Recently, OMG-LLaVA [ 71 ] used a universal segmentation method as a visual encoder, integrating image information and perception priors into visual tokens provided to large language models (LLM), thus better supporting both segmentation and comprehension. Figure 1 : The performance of LIRA on various comprehension and segmentation tasks compared with existing models. The x-axis represents the scores obtained by normalizing MME [ 9 ] to a percentage scale. The y-axis represents the average performance on the RefCOCO [ 68 ] , RefCOCO+ [ 68 ] , and RefCOCOg [ 39 ] datasets. LIRA achieves excellent performance on both tasks. Despite significant advancements, previous methods often failed to accurately segment objects when meeting complex scenarios. As shown in Fig. 2 , OMG-LLaVA failed to segment “the red bus closest to the white car”. To investigate the causes of segmentation errors, we extracted the embedding of the <seg> token from LMM using the image in the first column and applied it to segment the images in the second and third columns without further interaction with LLM. Interestingly, we observed that the bus on the left was consistently segmented in all images in row (1). Upon analyzing the logits of the <seg> token, we found that the higher logits for “left” caused the bus on the left to be segmented. We suspect that the error occurred because the LMM failed to effectively embed accurate positional information into the <seg> token due to the limited visual comprehension of the LMM. Additionally, previous methods typically rely on position queries to indicate object locations and do not always establish a clear connection between local descriptions and the corresponding local image features, which may lead to hallucinations. This prompts an important question: Would it be more effective to directly input local image features into the LLM, enabling the model to generate region-specific descriptions and establish a more explicit mapping between visual features and their corresponding descriptions? In this paper, we propose LIRA, L ocal I nterleaved R egion A ssistance, incorporating two key components: 1) The Semantic-Enhanced Feature Extractor (SEFE) fuses high-level semantic comprehension with fine-grained pixel features through parametric fusion, enhancing visual comprehension for more precise object segmentation; 2) The Interleaved Local Visual Coupling (ILVC) enhances alignment between visual cues and textual descriptions by coupling image regions with corresponding text, reducing hallucination and enabling more accurate responses. As illustrated in Fig. 2 , LIRA accurately interprets queries such as “the red bus closest to the white car” as referring to the “right bus”, allowing for precise segmentation. This process involves understanding object attributes based on both the query and the image to achieve accurate segmentation, which we refer to as inferring segmentation. This may differ from the definition used in LISA Reasoning Segmentation, which relies on external or commonsense knowledge to reason about implicit queries (e.g., “the food rich in Vitamin C”). As shown in Fig. 1 , LIRA demonstrates strong performance across a range of standard comprehension and segmentation tasks. We summarize the main contributions as follows: • Improved Segmentation Accuracy . LIRA incorporates the Semantic-Enhanced Feature Extractor (SEFE), which effectively integrates high-level semantic features with fine-grained pixel-level information. This fusion significantly enhances the model’s ability to infer object attributes, resulting in more precise segmentation outcomes. • Reduced Hallucination . Through the Interleaved Local Visual Coupling (ILVC) mechanism, LIRA establishes a robust alignment between local image regions and their corresponding textual descriptions. By explicitly coupling visual features with their semantic counterparts, LIRA reduces hallucinated comprehension and ensures the generation of precise and accurate image descriptions. • Strong Performance in Both Segmentation and Comprehension Tasks . LIRA demonstrates strong performance on both segmentation and visual comprehension benchmarks. Specifically, experiments show that LIRA results in only a slight decrease compared to training with comprehension data alone (75.2% vs. 75.3%), significantly outperforms the previous best method, OMG-LLaVA, which shows a larger decline of (55.6% vs. 69.9%) across five comprehension datasets. Figure 2 : “Please segment the red bus closest to the white car.” is used for the (1) and (3) rows of images in the first column, and “Please segment the red bus farthest away from the white car.” is used for the (2) and (4) rows of images in the first column. The dashed lines indicate that the three images share the <seg> token of the image in the first column. The right of the image represents the logits of the <seg> token of the images in the first column. Figure 3 : The overall architecture of LIRA. Global and local features are extracted using SEFE, with ILVC introduced to align <mask, region, text> correspondences. Note that in this pipeline, the <seg> token passing through a projector and pixel decoder for mask generation is omitted for simplicity. The orange line indicates the use of only the semantic encoder in SEFE, avoiding excessive zooming of local images in the pixel encoder. 2 Related Work 2.1 Comprehension-Oriented LMMs In recent years, the development of LMMs has made remarkable strides, improving the visual comprehension capabilities of LLMs. Early works [ 1 , 21 , 29 , 73 , 65 ] explore various methods to integrate visual features into Large Language Models (LLMs). Building on these foundations, subsequent approaches have focused on refining the quality of visual input, including techniques such as curriculum learning [ 2 , 6 ] , high-resolution image cropping [ 63 , 25 , 28 , 60 , 31 ] , and separating high- and low-resolution branches [ 12 , 36 ] . Additionally, advancements like chain-of-thought reasoning [ 58 , 41 , 34 ] and the