Can Segmentation Models Understand the World? Towards Proactive Affordance Reasoning via Visual Chain-of-Thought
Yuchen Guo, Junli Gong, Hongmin Cai, Yiu-ming Cheung, Weifeng Su · 2026
Read Can Segmentation Models Understand the World? Towards Proactive Affordance Reasoning via Visual Chain-of-Thought by Yuchen Guo, Junli Gong, Hongmin Cai, Yiu-ming Cheung, Weifeng Su (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-27764.txt.
Introduction
Yuchen Guo, Junli Gong, Hongmin Cai, Yiu-ming Cheung, and Weifeng Su introduce SegWorld, a segmentation model that reasons before committing to a mask. Before any instruction arrives, it watches the scene first, noting visible objects and guessing events they may support. Given an intent-level instruction, it chains from the key object through the needed action to the physical touch site, lifting mask prediction on intent-level tasks.
Catalog record
| CREATOR | Yuchen Guo, Junli Gong, Hongmin Cai, Yiu-ming Cheung, Weifeng Su |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36591 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.27764
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Can Segmentation Models Understand the World? Towards Proactive Affordance Reasoning via Visual Chain-of-Thought Yuchen Guo, Junli Gong, Hongmin Cai, Yiu-ming Cheung, Weifeng Su Abstract Recent segmentation models couple large language models (LLMs) with mask decoders to ground complex language expressions into masks, yet their instructions remain target-referential: they describe, constrain, or imply the region to be segmented. However, in real-world embodied interaction, human instructions are often at the intent-level, which includes the desired outcome without naming the region that enables it. To bridge this gap, we introduce SegWorld , where the model reasons about the scene through a multi-level visual chain-of-thought (CoT) before committing to a mask. Before receiving any instructions, it proactively observes the scene, describing visible objects and inferring plausible events they may support. Given an instruction, it continues the chain: from the object relevant to the intent, through the action that satisfies it, to the physical interaction site, the object part that affords the action. We formalize SegWorld as probabilistic inference, in which proactive observation supplies a linguistic scene context that improves mask prediction when instructions are given at the level of intent. We construct an intent-to-part benchmark for evaluating affordance-bearing part segmentation from high-level goals. Experiments show SegWorld matches instruction-driven baselines on target-referential instructions and improves substantially on intent-level ones. Figure 1: Given a human-centric conversational intent that does not name the target region, SegWorld first proactively observes the scene, then grounds the intent to a relevant object and action, reasons about the physical interaction site, and finally segments the affordance-bearing part that enables the intended action. 1 Introduction Language-driven segmentation has expanded the role of segmentation from category recognition to instruction following. Recent models integrate LLMs or multimodal LLMs with mask decoders, enabling them to ground complex natural-language expressions, including spatial relations, object attributes, and implicit reasoning cues, into masks ( 10 ; 14 ; 13 ; 11 ; 6 ) . These instructions are commonly studied as explicit referring expressions ( 9 ; 26 ; 12 ) or implicit reasoning instructions ( 10 ; 7 ; 24 ) . Recent works further introduce visual chain-of-thought reasoning into segmentation, using textual rationales, point or region cues, or positional priors to connect reasoning with grounding ( 8 ; 27 ) . Despite this progress, the dominant paradigm remains query-conditioned: the model is given an instruction and is expected to return the corresponding region. The visual scene becomes relevant only after a question is asked, and the instruction is treated as a specification of the target. But this is not how human instructions work in the real world 21 . Human-centric instructions are pragmatic and underspecified, requiring the listener or agent to infer the intended goal from context ( 20 ; 23 ; 15 ) . Models shouldn’t view scenes through the narrow aperture of current instructions. Before being asked anything, it can already take in what is present, what could happen, and how things can be used. The instruction then serves not as a full description of the target, but as a way of selecting one actionable possibility from this prior understanding. This distinction becomes critical when instructions are expressed at the level of intent. Consider a tabletop scene with a mug, a kettle, and a plate, and the instruction “I want to drink water.” The desired mask is not named by the instruction at all. A query-conditioned model may ground it to the mug or the kettle, both semantically related to drinking, yet neither specifies the physical site through which the intended action should be executed. Reaching the right region requires several steps of everyday reasoning, and the region it arrives at is often not a whole object but a part. This is consistent with affordance and part-level grounding studies, where interaction is localized to functional regions rather than entire objects ( 5 ; 17 ; 4 ; 16 ; 24 ; 22 ) . We call this setting intent-level segmentation : recovering an affordance-bearing part that lies several reasoning steps below a high-level goal. We introduce SegWorld , which replaces query-conditioned mapping with a proactive, multi-level reasoning process. Rather than waiting for an instruction, SegWorld first reads the scene on its own: it describes the visible objects and the events they could plausibly support, producing a linguistic scene context before any instruction arrives. When the instruction is given, SegWorld does not search the image from scratch; it traverses this context through an eight-level visual chain-of-thought organized into three stages. The first stage performs proactive observation. The second identifies the object implicated by the intent and the action that would fulfill it. The third resolves the object part that affords this action and the affordance it carries. Each level is expressed in language and remains explicit and inspectable, turning a single opaque mapping from instruction to pixels into a traceable reasoning process. This design is principled, not merely procedural. We formalize SegWorld as probabilistic inference: the proactive stage draws a scene context, and the instruction-conditioned stages estimate the mask conditioned on both the instruction and that context. The scene context acts as an informative prior. When an instruction is target-referential, this prior matters little, since the instruction already localizes the answer. But when the instruction is intent-level, the answer is underdetermined by the instruction alone, and the scene context is what makes the posterior tractable. Proactive observation, in other words, is most valuable exactly where query-conditioned models are weakest. To study this setting, we construct Intent2Part , a benchmark for intent-level segmentation in which instructions name a high-level goal and the target is an affordance-bearing object part. On Intent2Part , SegWorld matches strong query-conditioned baselines when instructions directly specify their target, and improves over them substantially when the target must be recovered from intent. Beyond accuracy, its visual chain-of-thought exposes where a prediction comes from, making both correct and incorrect masks interpretable. Figure 2: Examples from Intent2Part . Each sample pairs a human-centric intent-level instruction with the latent reasoning chain needed to recover the target: object, action, part, and affordance. The benchmark also retains target-referential referring and reasoning expressions, allowing models to be evaluated under both direct target descriptions and high-level intents. Our contributions are summarized as follows: • We identify intent segmentation , a setting where instructions name a goal rather than a target, and where the answer is an affordance-bearing object part lying several reasoning steps below the instruction. • We propose SegWorld , a proactive segmentation framework that performs a eight-level visual chain-of-thought, and formalize it as probabilistic inference in which a proactive scene context serves as a prior for mask prediction. • We construct Intent2Part , a benchmark for evaluating intent-level segmentation, and show that SegWorld matches query-conditioned baselines on direct instructions while improving substantially on intent-level ones. 2 Related Work Language-driven segmentation. With the rapid development of LLMs, recent studies integrate LLMs into segmentation models 14 ; 13 ; 10 , enabling them to segment objects from natural-language instructions rather than single-word category labels 11 ; 6 . Such instructions are broadly categorized into explicit referring expressions ( 9 ; 26 ; 12 ) and implicit reasoning instructions ( 10 ; 7 ; 24 ) . To strengthen reasoning, some