CoT-Seg: Rethinking Segmentation with Chain-of-Thought Reasoning and Self-Correction
Shiu-hong Kao Chak Ho Huang Huaiqian Liu Yu-Wing Tai Chi-Keung Tang, Equal contribution · 2026
Read CoT-Seg: Rethinking Segmentation with Chain-of-Thought Reasoning and Self-Correction by Shiu-hong Kao Chak Ho Huang Huaiqian Liu Yu-Wing Tai Chi-Keung Tang, Equal contribution (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-17420.txt.
Introduction
Shiu-hong Kao and four co-authors present CoT-Seg, a method to cut out image regions named in text. The task is called reasoning segmentation. It fails most on hard queries and new image types. The team borrows chain-of-thought, the idea that hard problems need longer thought. Their method thinks in steps before it cuts. It also checks and corrects its own output.
Self-correction gives the model a second pass at weak spots. Long thought helps when a query hides its target in words. Better segmentation aids photo edit, medical scans, and robot sight. The work aims at the messy cases. Those are where simple one-pass tools break down most.
Catalog record
| CREATOR | Shiu-hong Kao Chak Ho Huang Huaiqian Liu Yu-Wing Tai Chi-Keung Tang, Equal contribution |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-17420
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CoT-Seg: Rethinking Segmentation with Chain-of-Thought Reasoning and Self-Correction Shiu-hong Kao Chak Ho Huang Huaiqian Liu Yu-Wing Tai Chi-Keung Tang, Equal contribution Abstract Existing works of reasoning segmentation often fall short in complex cases, particularly when addressing complicated queries and out-of-domain images. Inspired by the chain-of-thought reasoning, where harder problems require longer thinking steps/time, this paper aims to explore a system that can think step-by-step, look up information if needed, generate results, self-evaluate its own results, and refine the results, in the same way humans approach harder questions. We introduce CoT-Seg , a training-free framework that rethinks reasoning segmentation by combining chain-of-thought reasoning with self-correction . Instead of fine-tuning, CoT-Seg leverages the inherent reasoning ability of pre-trained MLLMs ( e.g., GPT-4o) to decompose queries into meta-instructions, extract fine-grained semantics from images, and identify target objects even under implicit or complex prompts. Moreover, CoT-Seg incorporates a self-correction stage: the model evaluates its own segmentation against the original query and reasoning trace, identifies mismatches, and iteratively refines the mask. This tight integration of reasoning and correction significantly improves reliability and robustness, especially in ambiguous or error-prone cases. Furthermore, our CoT-Seg framework allows easy incorporation of retrieval-augmented reasoning, enabling the system to access external knowledge when the input lacks sufficient information. To showcase CoT-Seg’s ability to handle very challenging cases, we introduce a new dataset ReasonSeg-Hard . Our results highlight that combining chain-of-thought reasoning, self-correction, offers a powerful paradigm for vision language integration driven segmentation. Our project website is available at https://danielshkao.github.io/cot-seg.html . 1 Introduction Figure 1 : Finding the first violinist (concertmaster) is challenging among similar-looking musicians. CoT-Seg reasons that they sit to the conductor’s left and generates a meta-query with relevant spatial information, enabling more accurate segmentation than LISA and Vision Reasoner (No self-correction was needed). Figure 2 : CoT-Seg reasons about the user’s query to segment surfers in the correct pose , capturing only those who have popped up and are riding waves, unlike LISA and Vision Reasoner (No self-correction was needed). Figure 3 : CoT-Seg identifies the gym equipment matching the user’s query for biceps, e.g., the preacher’s curl, reasoning about its function without any training (Self-correction was needed). Figure 4 : CoT-Seg reasons about the arrangement of dumbbells to segment those that are unracked , a more challenging task than simple detection (No self-correction was needed). Reasoning segmentation represents a promising step toward vision-language integration, where a system generates a segmentation mask from complex and often implicit language queries. Recent progress has been driven by fine-tuning Multimodal Large Language Models (MLLMs), such as LISA ( 21 ) , Seg-Zero ( 28 ) and Vision Reasoner ( 29 ) , to produce segmentation outputs. Despite their success, these methods struggle with cases that require nuanced reasoning, domain knowledge, or contextual inference which are the major challenges that humans naturally handle. Consider the examples in Figures 1 – 4 . Locating the first-chair violinist requires knowledge of orchestra seating arrangements, not just visual similarity. Differentiating surfers by posture demands reasoning about dynamic body positions. Selecting the correct gym equipment for bicep training requires understanding functional affordances. Identifying unracked dumbbells requires contextual analysis of their relation to the rack. How can we approach robust reasoning segmentation? To improve the segmentation quality, existing work mainly focus on the techniques to connect MLLM with vision foundation model, e.g. LISA introduces a new [SEG] token in MLLM vocabulary to extract segmentation-based features, Seg-Zero utilizes reinforcement learning to finetune MLLM for point and bounding box outputs. However, generating difficult mask from implicit text in one pass is even challenging for human beings. We rethink the task, aiming to explore a system capable of approaching the prediction step-by-step. This system allows the model to address harder problems with longer reasoning steps/time. To achieve this, we aim to propose a coherent mechanism to analyze implicit semantics in depth, evaluate its own predictions, and refine its own mistakes. In this work, we introduce CoT-Seg , a training-free framework that revisits reasoning segmentation by integrating chain-of-thought (CoT) reasoning with a dedicated self-correction mechanism . Although CoT has been used in other reasoning tasks, to the best of our knowledge it has not been explored for reasoning-driven segmentation, where the system must jointly interpret linguistic instructions, visual context, and object relationships. Designing a CoT workflow in this setting is non-trivial, since the reasoning must both decompose the query and reveal the semantic structure required for spatially grounded segmentation. CoT-Seg leverages the latent reasoning capability of pre-trained MLLMs (e.g., GPT-4o) to convert queries into meta-instructions, extract fine-grained semantics, and generate initial segmentation maps, all without additional training. Crucially, CoT-Seg introduces a self-correction stage: the model checks its predictions against the query and reasoning trace, detects inconsistencies, and refines the results through automatically generated meta-queries. This closed-loop design enables the system not only to reason about segmentation but also to critique and repair its outputs, showing that CoT becomes genuinely impactful when adapted to the challenges of reasoning-based segmentation. Furthermore, we extend CoT-Seg with retrieval-augmented reasoning . When the query and image lack sufficient information, CoT-Seg calls an external agent to retrieve relevant knowledge from the web, integrating it into the reasoning process. This augmentation further strengthens its ability to tackle ambiguous or knowledge-intensive cases. Through extensive experiments on ReasonSeg 21 , we demonstrate that CoT-Seg outperforms existing methods while requiring no additional training. To examine our work in extremely difficult cases, we also propose a new reasoning segmentation benchmark called ReasonSeg-Hard , where CoT-Seg showcases additional improvements against state of the arts. Our results show that integrating CoT reasoning, self-correction, and retrieval augmentation provides a powerful paradigm for advancing reasoning-driven segmentation toward human-level reliability. 2 Related Work Image Segmentation and Reasoning Segmentation. Image segmentation has evolved from early graphical-model-based methods, such as Conditional Random Fields (CRFs) ( 20 ; 2 ) and region growing ( 5 ) , to deep learning approaches that utilize encoder-decoder architectures ( 1 ) , dilated convolutions ( 43 ) , pyramid pooling ( 47 ) , and non-local operators ( 27 ) . Instance segmentation ( 11 ; 4 ) and panoptic segmentation ( 16 ; 3 ) further pushed the boundary to finer-grained understanding. The emergence of foundation models for segmentation, especially the Segment Anything Model (SAM) ( 17 ) , has revolutionized the field. By training on billions of masks and images, SAM enables promptable, zero-shot segmentation with multimodal inputs like points or bounding boxes. Leveraging SAM with Multimodal Large Language Models (MLLMs) has led to a new line of works on reasoning segmentation ( 22 ; 40 ; 44 ; 10 ; 42 ) . These approaches generate segmentation masks conditioned on implicit or complex textual queries. However, combining MLLMs with SAM directly often fails in challenging scenarios, such as queries requiring domain knowledge, occluded objects, or intricate structures. In contrast, our work shows that integrating chain-of-thought reasoning and self-correction can substantially enhance robustness and accuracy in these difficult cases. Chain-of-Thought Reasoning in LLMs and MLLMs. Chain-of-Thought (CoT) reasoning improves reasoning performance in large language models by decomposing complex tasks into intermediate steps ( 39 ; 37 ; 45 ; 32 ; 18 ) . While CoT has been extensively explored in text-only