ResAgent: Entropy-based Prior Point Discovery and Visual Reasoning for Referring Expression Segmentation

Yihao Wang, Jusheng Zhang, Ziyi Tang, Keze Wang, Meng Yang · 2026

Read ResAgent: Entropy-based Prior Point Discovery and Visual Reasoning for Referring Expression Segmentation by Yihao Wang, Jusheng Zhang, Ziyi Tang, Keze Wang, Meng Yang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-16394.txt.

Introduction

Yihao Wang and four coauthors find the mark for phrase based cuts with entropy cues. Their 2026 preprint deals with cuts that use free phrase cues. The task, called RES, cuts the thing a phrase names in a pic. It backs tools like pic edit and robot sight. Past tools oft miss the true thing when pics hold more than one fit.

The team uses entropy to find prime points that pin the thing. Points then guide sight thought to the right zone. Sharp zone thought should lift cut grade on hard pics. The work, named ResAgent, adds a point find step. It seeks cuts that match the phrase sense well.

Catalog record

CREATORYihao Wang, Jusheng Zhang, Ziyi Tang, Keze Wang, Meng Yang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE84 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-16394

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ResAgent: Entropy-based Prior Point Discovery and Visual Reasoning for Referring Expression Segmentation Yihao Wang, Jusheng Zhang, Ziyi Tang, Keze Wang, Meng Yang Abstract Referring Expression Segmentation (RES) is a core vision-language segmentation task that enables pixel-level understanding of targets via free-form linguistic expressions, supporting critical applications such as human-robot interaction and augmented reality. Despite the progress of Multimodal Large Language Model (MLLM)-based approaches, existing RES methods still suffer from two key limitations: first, the coarse bounding boxes from MLLMs lead to redundant or non-discriminative point prompts; second, the prevalent reliance on textual coordinate reasoning is unreliable, as it fails to distinguish targets from visually similar distractors. To address these issues, we propose ResAgent , a novel RES framework integrating E ntropy- B ased Point D iscovery ( EBD ) and V ision- B ased R easoning ( VBR ). Specifically, EBD identifies high-information candidate points by modeling spatial uncertainty within coarse bounding boxes, treating point selection as an information maximization process. VBR verifies point correctness through joint visual-semantic alignment, abandoning text-only coordinate inference for more robust validation. Built on these components, ResAgent implements a coarse-to-fine workflow: bounding box initialization, entropy-guided point discovery, vision-based validation, and mask decoding. Extensive evaluations on four benchmark datasets (RefCOCO, RefCOCO+, RefCOCOg, and ReasonSeg) demonstrate that ResAgent achieves new state-of-the-art performance across all four benchmarks, highlighting its effectiveness in generating accurate and semantically grounded segmentation masks with minimal prompts. 1 Introduction R eferring E xpression S egmentation (RES) aims to bridge visual perception and natural language understanding by generating segmentation masks for objects referenced by free-form linguistic expressions 17 ; 12 ; 50 ; 33 . As a core vision–language grounding task, RES supports key applications such as human–robot interaction 45 ; 11 ; 23 , augmented reality 10 ; 37 ; 43 , and content-based retrieval 1 ; 16 ; 40 , where models must interpret fine-grained textual cues and precisely localize targets under cluttered or ambiguous visual contexts. Traditional RES approaches based on convolutional architectures 3 ; 9 ; 14 ; 27 or early cross-modal attention mechanisms 6 ; 7 ; 18 ; 29 ; 26 often struggle to model the rich, fine-grained semantic alignment needed for robust grounding. With the rise of Multimodal Large Language Models (MLLMs), recent RES methods 42 ; 44 ; 21 ; 34 ; 30 ; 4 ; 22 ; 55 leverage MLLMs to guide segmentation models such as SAM 19 ; 35 , typically through coarse bounding box prediction or point-level prompting, thereby improving high-level reasoning and linguistic understanding. Figure 1 : Top: Accuracy (left) and confidence (right) of coordinate-based versus visual reasoning for point inference in a zero-shot setting. Bottom: Example queries highlighting the fundamental difference—textual coordinates lose spatial cues, while visual reasoning provides semantic anchoring. However, current MLLM-based RES pipelines still face two central limitations. First , bounding boxes predicted by MLLMs are coarse and offer only approximate spatial localization, requiring point prompts for fine-grained segmentation; yet widely used point sampling strategies (e.g., random or uniform sampling) provide redundant or non-discriminative cues. Second , MLLMs treat textualized coordinates as ordinary language tokens (e.g., “(175, 60)”→“1–7–5–6–0”), interpreting them as discrete symbols rather than grounded spatial cues. This tokenization disrupts geometric continuity and visual context, leading to unreliable point reasoning and noisy prompts. As shown in Fig. 1 , the zero-shot textual coordinate reasoning exhibits an overall accuracy below 60 %. Given these limitations, a pivotal question naturally arises: (Q) How can we select semantically informative point prompts and verify their correctness in a way that is grounded in visual–semantic alignment, rather than relying on coarse region priors or text-only coordinate reasoning? To answer this question, we treat the coarse bounding box not merely as a spatial constraint, but as a structured uncertainty prior . Regions within the box exhibit varying semantic relevance, suggesting that point prompts should be selected according to information gain rather than spatial uniformity. Meanwhile, verifying a point’s correctness should occur within the joint visual–language representation space , rather than through coordinate text tokens—a design rationale strongly corroborated by our pilot study (Fig. 1 ), where vision-based reasoning significantly outperformed textual coordinate parsing in both accuracy and confidence under a zero-shot setting. Guided by these two principles, we propose a pairing of entropy-based discovery and vision-based semantic validation . Guided by this insight, we propose ResAgent, a framework for Referring Expression Segmentation via Entropy-based Prior Point Discovery with a Visual Reasoning Agent . ResAgent is built upon two key components: (1) an Entropy-Based Point Discovery (EBD) strategy that identifies high-information candidate points by modeling spatial uncertainty (Sec. 3.3 ), and (2) a Vision-Based Reasoning (VBR) module that verifies candidate correctness through visual–semantic alignment instead of textual coordinate inference (Sec. 3.4 ). Building on these components, ResAgent performs RES through a four-stage coarse-to-fine workflow: bounding box initialization (Sec. 3.2 ), entropy-guided point discovery (Sec. 3.3 ), vision-based point validation (Sec. 3.4 ), and final mask decoding with probability aggregation (Sec. 3.5 ). This formulation casts point prompting as an information maximization process, improving grounding accuracy while requiring minimal prompts. We evaluate ResAgent on RefCOCO, RefCOCO+, RefCOCOg, and ReasonSeg. ResAgent achieves new state-of-the-art performance across all benchmarks, demonstrating its ability to produce accurate and semantically grounded segmentations. Our contributions are summarized as follows: • We propose an Entropy-Based Point Discovery (EBD) strategy that leverages spatial uncertainty to identify semantically critical prompt candidates. • We introduce a Vision-Based Reasoning (VBR) mechanism that verifies candidate correctness through visual–semantic alignment rather than text-only reasoning. • ResAgent achieves state-of-the-art performance on RefCOCO, RefCOCO+, RefCOCOg, and ReasonSeg. 2 Related Work 2.1 Referring Expression Segmentation Referring Expression Segmentation (RES) 12 ; 17 ; 33 ; 50 ; 31 evolves from its precursor, Referring Expression Comprehension (REC) 13 ; 28 ; 47 ; 39 ; 48 ; 49 , by advancing from coarse bounding box localization to precise pixel-level segmentation. This shift places stricter demands on fine-grained visual-linguistic alignment. Early methods primarily depended on convolutional architectures 3 ; 9 ; 14 ; 27 or vanilla attention mechanisms 6 ; 7 ; 18 ; 29 ; 26 , which often inadequately captured complex cross-modal interactions. Recent studies have enhanced segmentation performance by incorporating more sophisticated mechanisms into the models, such as the cross-attention in ReLA 25 , the vision-language token fusion in RefSegformer 42 , and the counting-aware hierarchical decoding in CoHD 32 . Nevertheless, these methods still struggle to address the intricate challenges posed by RES, especially failing to accurately capture the nuanced relationships between textual referring and their corresponding object regions. 2.2 Pixel Understanding with MLLMs Pioneering the integration of Multimodal Large Language Models (MLLMs) with segmentation tasks, LISA 21 established the "one embedding as all mask" paradigm by augmenting the MLLM vocabulary with a special [SEG] token to elicit masks from decoders like SAM 19 ; 35 . Building upon this, subsequent works like GSVA 44 extend this ideal to address more complex scenarios, such as handling the rejection of null targets through a [REJ] token. Other efforts enhance versatility through dedicated decoding mechanisms, such as GLaMM’s integration of masks with language responses for multi-granular prompts 34 and PSALM’s unified framework with an external decoder for diverse tasks 54 . Unfortunately,

Read the complete text