CoT-Segmenter: Enhancing OOD Detection in Dense Road Scenes via Chain-of-Thought Reasoning

Jeonghyo Song, Kimin Yun, DaeUng Jo, Jinyoung Kim, Youngjoon Yoo · 2025

Read CoT-Segmenter: Enhancing OOD Detection in Dense Road Scenes via Chain-of-Thought Reasoning by Jeonghyo Song, Kimin Yun, DaeUng Jo, Jinyoung Kim, Youngjoon Yoo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-03984.txt.

Introduction

Self-driving systems must spot objects they were never trained on, such as debris or strange vehicles on the road. Jeonghyo Song and co-authors found that current detectors fail in three recurring situations: objects packed tightly together, small objects far away, and single objects that fill most of the frame. Their system, CoT-Segmenter, attacks all three by borrowing the step-by-step reasoning style of large language models.

The method prompts a foundation model like GPT-4 to reason about a scene in stages before deciding which pixels are anomalous. That reasoning is tuned to the three failure patterns the team catalogued. In tests on standard benchmarks and on a harder subset of a road anomaly dataset, the framework beat existing methods. The authors present it as a more reliable and more interpretable way to flag the unexpected in dense driving scenes.

Catalog record

CREATORJeonghyo Song, Kimin Yun, DaeUng Jo, Jinyoung Kim, Youngjoon Yoo
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE21 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-03984

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CoT-Segmenter: Enhancing OOD Detection in Dense Road Scenes via Chain-of-Thought Reasoning Thanks: Corresponding author Jeonghyo Song, Kimin Yun, DaeUng Jo, Jinyoung Kim, Youngjoon Yoo Abstract Effective Out-of-Distribution (OOD) detection is critical for ensuring the reliability of semantic segmentation models, particularly in complex road environments where safety and accuracy are paramount. Despite recent advancements in large language models (LLMs), notably GPT-4, which significantly enhanced multimodal reasoning through Chain-of-Thought (CoT) prompting, the application of CoT-based visual reasoning for OOD semantic segmentation remains largely unexplored. In this paper, through extensive analyses of the road scene anomalies, we identify three challenging scenarios where current state-of-the-art OOD segmentation methods consistently struggle: (1) densely packed and overlapping objects, (2) distant scenes with small objects, and (3) large foreground-dominant objects. To address the presented challenges, we propose a novel CoT-based framework targeting OOD detection in road anomaly scenes. Our method leverages the extensive knowledge and reasoning capabilities of foundation models, such as GPT-4, to enhance OOD detection through improved image understanding and prompt-based reasoning aligned with observed problematic scene attributes. Extensive experiments show that our framework consistently outperforms state-of-the-art methods on both standard benchmarks and our newly defined challenging subset of the RoadAnomaly dataset, offering a robust and interpretable solution for OOD semantic segmentation in complex driving environments. 1 Introduction Reliable Out-of-Distribution (OOD) detection is essential for robust semantic segmentation, especially in safety-critical domains like autonomous driving, where failing to identify unfamiliar objects can have severe consequences. In the road scene scenario, OOD detection aims to distinguish inputs that differ significantly from the training distribution, ensuring that autonomous systems can safely and effectively respond to unexpected or rare scenarios. Despite recent advancements, accurately identifying and segmenting OOD objects remains challenging due to real-world complexities and inherent limitations of existing models. Figure 1: Visualization of three challenging scenarios affecting performance degradation in OOD segmentation models, observed in the current state-of-the-art method S2M [ 26 ] . From top to bottom: (1) densely packed and overlapping objects, (2) distant scenes with small objects, and (3) large foreground-dominant objects. Specifically, we identify critical failure cases in Figure 1 in which existing OOD segmentation methods consistently fail: (1) densely clustered and overlapping objects, (2) small and distant objects, and (3) scenes dominated by large foreground objects from the RoadAnomaly dataset [ 12 ] , a representative benchmark for road scene understanding. These challenging scenarios reveal a fundamental limitation in existing OOD segmentation approaches: their inability to comprehend visual scenes contextually. Current methods predominantly rely on local pixel-level uncertainty metrics or global scene statistics, rather than reasoning about object relationships, scene composition, and spatial dependencies—critical elements for understanding fine-grained semantics in complex environments. To address the problems, motivated by the recent breakthroughs in Large Language Models (LLMs) such as GPT-4 [ 1 ] and their reasoning capability, we explore the application of Chain-of-Thought (CoT) [ 25 ] reasoning to the relatively underexplored area of OOD semantic segmentation in road scene understanding. We introduce a novel framework that employs GPT-4-driven CoT prompting specifically tailored for GroundedSAM [ 21 ] , a combination of GroundingDINO [ 14 ] and Segment Anything Model (SAM) [ 10 ] . Our proposed framework exploits the extensive knowledge and advanced logical reasoning capabilities of GPT-4 [ 1 ] to generate highly effective text prompts. These prompts provide critical contextual cues that effectively guide the GroundedSAM model in identifying regions of interest, thereby significantly improving both segmentation performance and interpretability. The primary contributions of our research are: • Clearly identifying and defining the problematic scenarios where current OOD semantic segmentation models consistently underperform, verified through comprehensive analysis of the RoadAnomaly dataset. • Proposing a GPT-4-based CoT prompting framework that leverages advanced prompt engineering techniques to improve OOD detection and segmentation performance. • The proposed framework enhances interpretability and robustness of OOD segmentation by combining CoT-guided reasoning with open-vocabulary detectors. 2 Related work 2.1 OOD Semantic Segmentation Out-of-distribution (OOD) semantic segmentation [ 26 , 24 , 15 , 11 , 18 , 20 , 8 , 23 ] aims to segment regions belonging to unknown classes not encountered during training, a critical task in safety-sensitive domains like autonomous driving. Recent methods include entropy-based, score-based, hybrid, and mask-based. Entropy-based methods, such as Meta-OoD [ 4 ] , model prediction uncertainty through entropy maximization. Score-based methods like PEBAL [ 24 ] and RPL [ 15 ] employ pixel-level anomaly scoring via energy functions or residual features. Hybrid approaches, notably DenseHybrid [ 7 ] , integrate discriminative and generative signals, though often sensitive to threshold tuning. Mask-based methods, including RbA [ 18 ] , Mask2Anomaly [ 20 ] , and S2M [ 26 ] , achieve strong object-level localization through mask classification or abstention strategies, effectively reducing false positives near ambiguous regions. Unlike previous methods, we propose a novel inference-driven framework leveraging Chain-of-Thought (CoT) reasoning [ 25 ] via Large Language Models (LLMs), such as GPT-4 [ 1 ] . Our approach guides segmentation through structured visual reasoning with semantically grounded prompts, enabling robust OOD detection in both general and challenging scenarios. 2.2 CoT Reasoning in Vision Tasks Using LLMs Large Vision-Language Models (LVLMs), such as LLaVA [ 13 ] , GPT-series [ 1 , 3 ] , Kosmos [ 19 ] , and Flamingo [ 2 ] , have significantly advanced multimodal understanding through visual perception and high-level language reasoning. These models excel in vision-language tasks such as visual question answering and image captioning. To improve interpretability and localized reasoning, recent studies have extended Chain-of-Thought (CoT) [ 25 ] prompting to vision tasks. Works such as Visual CoT [ 22 ] , Chain-of-Spot [ 16 ] and CCoT [ 17 ] guide the reasoning process via intermediate rationales linked to spatial regions, while Shikra [ 5 ] enables fine-grained coordinate-level grounding within dialogues. Inspired by these approaches, we propose a framework that leverages step-by-step reasoning to infer task-relevant Regions of Interest (ROIs) in complex road scenes. By explicitly directing attention to visually challenging areas, our method enhances OOD object identification and improves segmentation robustness in safety-critical scenarios. 3 Method Figure 2: Overview of the proposed CoT-Segmenter framework. Given a road scene image, our CoT-Based Prompt Generation Module (right) performs step-by-step reasoning via GPT-4 to identify semantic anomalies and outputs a concise text prompt in two formats: V1 (state + noun) and V2 (noun only). Each prompt is independently fed into the Grounded Segmentation Module (left), which employs open-vocabulary detectors (GroundedSAM) to produce segmentation masks. The final OOD prediction is obtained by taking the union of the pixel regions from both prompts. 3.1 Overview We propose a novel two-stage framework for OOD object segmentation in road scenes. The core idea is to exploit the reasoning capability of a large vision-language model via Chain-of-Thought (CoT) [ 25 ] prompting to generate text-based soft supervision, which guides a powerful segmentation backbone, GroundedSAM [ 21 ] . Our framework consists of two main modules: CoT-Based Prompt Generation Module: Given an input road scene image denoted by I , we use a large language model to infer OOD object candidates through step-by-step visual reasoning. This process unfolds over three structured stages, producing intermediate reasoning outputs S_{1} , S_{2} , and S_{3} . Based on the final reasoning

Read the complete text