Multi-task Visual Grounding with Coarse-to-Fine Consistency Constraints

Ming Dai, Jian Li, Jiedong Zhuang, Xian Zhang, Wankou Yang · 2025

Read Multi-task Visual Grounding with Coarse-to-Fine Consistency Constraints by Ming Dai, Jian Li, Jiedong Zhuang, Xian Zhang, Wankou Yang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-06710.txt.

Introduction

Ming Dai and co-authors present in 2025 a two stage frame that finds and cuts out a named thing in a photo from a text cue. Past top frames fuse sight and text in one pass. A blur between the find task and the cut task then yields split calls.

Stage one gives rough find and cut calls from split coders. Stage two refines them with a mask led link block and a two way match loss. That loss forces the find and cut calls to agree. Pre trained sight text models add depth to the text read. Tests back the two stage gains.

Catalog record

CREATORMing Dai, Jian Li, Jiedong Zhuang, Xian Zhang, Wankou Yang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-06710

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Multi-task Visual Grounding with Coarse-to-Fine Consistency Constraints Ming Dai, Jian Li, Jiedong Zhuang, Xian Zhang, Wankou Yang Abstract Multi-task visual grounding involves the simultaneous execution of localization and segmentation in images based on textual expressions. The majority of advanced methods predominantly focus on transformer-based multimodal fusion, aiming to extract robust multimodal representations. However, ambiguity between referring expression comprehension (REC) and referring image segmentation (RIS) is error-prone, leading to inconsistencies between multi-task predictions. Besides, insufficient multimodal understanding directly contributes to biased target perception. To overcome these challenges, we propose a Coarse-to-fine Consistency Constraints Visual Grounding architecture ( C 3 ​ VG \text{C}^{3}\text{VG} ), which integrates implicit and explicit modeling approaches within a two-stage framework. Initially, query and pixel decoders are employed to generate preliminary detection and segmentation outputs, a process referred to as the Rough Semantic Perception (RSP) stage. These coarse predictions are subsequently refined through the proposed Mask-guided Interaction Module (MIM) and a novel explicit bidirectional consistency constraint loss to ensure consistent representations across tasks, which we term the Refined Consistency Interaction (RCI) stage. Furthermore, to address the challenge of insufficient multimodal understanding, we leverage pre-trained models based on visual-linguistic fusion representations. Empirical evaluations on the RefCOCO, RefCOCO+, and RefCOCOg datasets demonstrate the efficacy and soundness of C 3 ​ VG \text{C}^{3}\text{VG} , which significantly outperforms state-of-the-art REC and RIS methods by a substantial margin. Code and model will be available at https://github.com/Dmmm1997/C3VG . Introduction Visual grounding is a critical task within the vision-language domain, aimed at establishing a fine-grained correspondence between images and text by grounding a given referring expression within an image ( Li et al. 2022b ) . This task is typically divided into two sub-tasks based on the grounding approach: referring expression comprehension (REC) ( Yu et al. 2018 ; Kamath et al. 2021 ) and referring image segmentation (RIS) ( Kim et al. 2022 ; Tang et al. 2023 ) . Traditionally, REC and RIS have been treated as separate tasks with distinct technological pathways, necessitating complex, task-specific designs. However, REC and RIS exhibit significant similarities and offer complementary strengths, making their unification both logical and advantageous. Recently, multi-task visual grounding has gained prominence as it eliminates the need for task-specific network designs and enables the leveraging of data across both tasks to mutually enhance performance. MCN ( Luo et al. 2020 ) was the first approach to jointly train the REC and RIS tasks, employing a learnable method to establish consistency in attention maps. Recent research has primarily focused on enhancing the interaction across different modalities ( Li and Sigal 2021 ; Su et al. 2023 ) and exploring auto-regressive approaches to achieve both detection and segmentation ( Zhu et al. 2022 ; Cheng et al. 2024 ; Liu et al. 2023a ) . In this paper, we address two overlooked issues: 1) How to effectively leverage the complementarity of multi-task predictions to mitigate inconsistencies in results. 2) How to overcome the challenge of insufficient multimodal understanding to enhance perception in complex image-text scenarios. Figure 1: (a) Three examples of inconsistent results between multi-task outputs. (b) Two examples of failure in identifying targets due to insufficient multi-modal understanding. Figure 2: (a) Examples of the intermediate process in the proposed coarse-to-fine consistency constraint framework. (b) Two pretraining architectures: the left diagram illustrates separate encodings for image and text modalities followed by fusion, using single-modal pretraining; the right diagram shows a fused encoding architecture with multimodal pretraining. Inconsistent predictions between multi-task primarily arise due to the lack of effective constraints linking different tasks. This issue can be exemplified by three scenarios depicted in Fig. 1 (a): (1) accurate segmentation but erroneous detection, (2) inaccurate segmentation but correct detection, and (3) both segmentation and detection being incorrect yet providing complementary information. The traditional REC is a one-to-one detection task. When uncertainties arise during optimization, the detected result tends to be positioned between potential targets, leading to local optima. Conversely, the RIS task, involving finer-grained pixel-level predictions, can more precisely identify the target but often lacks sufficient spatial awareness. Thus, it becomes essential to introduce a multi-task consistency constraint to guide the model in supplementing information, thereby enhancing recognition in ambiguous situations. To this end, we propose a coarse-to-fine architecture for multi-task visual grounding, named \text{C}^{3}\text{VG} . The structure is shown in Fig. 3 . Initially, we employ a pixel decoder and a query decoder to independently generate coarse foreground semantics and localization regions in the Rough Semantic Perception (RSP) stage. Subsequently, the Refined Consistency Interaction (RCI) stage refines them and enforces consistency across the multi-task outcomes. Within the RCI stage, we introduce a Mask-guided Interaction Module (MIM) to implicitly integrate the multi-task results from the RSP stage. Furthermore, we apply a bidirectional consistency constraint loss to explicitly enforce consistency across tasks. As illustrated in Fig. 2 (a), the RSP stage delivers coarse localization and semantic results. Building on these priors, the RCI stage applies consistency constraints to produce higher-quality predictions. Insufficient multimodal understanding primarily manifests as an inability to effectively capture the semantic associations between modalities in downstream tasks, particularly when data is limited. Fig. 1 (b) shows two instances of identification errors caused by inadequate multimodal understanding: (1) The model incorrectly identifies ‘egg cup’ by focusing only on ‘cup’; (2) The model misinterprets ‘iMac’ due to the absence of prior knowledge. Recently, SimVG ( Dai et al. 2024 ) has confirmed the importance of employing a pretrained multi-modality encoder for improving referential understanding. However, this paper aims to further extend this structure from a single detection task to a multi-task learning framework to validate its broader effectiveness. As shown on the left side of Fig. 2 (a), previous methods typically utilize single-modal pretrained models as feature encoders and rely on limited downstream data to learn vision-language fusion representations. Recently, SimVG ( Dai et al. 2024 ) has decoupled the downstream multimodal fusion process and incorporated it into upstream pretraining, resulting in significant performance improvements for the REC task. Fig. 2 (b) illustrates the direct integration of the two modalities during the upstream pretraining process, leveraging advances in vision-language pretraining research ( Kim, Son, and Kim 2021 ; Wang et al. 2023 ) . This paper extends the conclusions of SimVG ( Dai et al. 2024 ) , demonstrating that the integration of multimodal pretrained models significantly enhances both convergence speed and accuracy in RIS and multi-task visual grounding tasks. Our main contributions are summarized as follows: 1. We introduce an innovative and efficient coarse-to-fine architecture, \text{C}^{3}\text{VG} , specifically designed for multi-task visual grounding. 2. We design a mask-guided interaction module and a bidirectional consistency constraint loss to address the challenge of multi-task prediction inconsistency. These components facilitate implicit interaction and provide explicit supervision for multi-task predictions, respectively. 3. We extend the pretrained multi-modality encoder from a single-task setting to a multi-task joint training framework and validate its impact on addressing the issue of inadequate multimodal understanding. 4. The proposed \text{C}^{3}\text{VG} framework significantly outperforms state-of-the-art methods on RefCOCO/+/g datasets for both REC and RIS tasks, while requiring only half or fewer training epochs. Related Work Visual Grounding Referring Expression Comprehension (REC) ( Liu et al. 2019 ; Yang et al. 2020 ; Yang et al. 2024 ;

Read the complete text