Multi-modality Anomaly Segmentation on the Road
Heng Gao 1 , Zhuolin He 2 , Shoumeng Qiu 2 , Xiangyang Xue 2,∗ , Jian Pu 1 · 2025
Read Multi-modality Anomaly Segmentation on the Road by Heng Gao 1 , Zhuolin He 2 , Shoumeng Qiu 2 , Xiangyang Xue 2,∗ , Jian Pu 1 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-17712.txt.
Introduction
Heng Gao and co-authors present, in a 2025 paper, a multi-sense method for flagging unexpected obstacles in road scenes for self-driving cars. A car must spot not only known object classes but anything dangerous it never trained on. Single-camera systems, the authors found, cry wolf on harmless regions.
Their framework adds a text channel, using a vision-language encoder, to steady the image channel's judgment. An ensemble module further sharpens the result. Tests ran on a road anomaly benchmark built for this task. False alarms erode trust and can trigger needless braking. Teaching the car to say unusual only when several senses agree is a safety feature, not a luxury.
Catalog record
| CREATOR | Heng Gao 1 , Zhuolin He 2 , Shoumeng Qiu 2 , Xiangyang Xue 2,∗ , Jian Pu 1 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 23 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-17712
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Multi-modality Anomaly Segmentation on the Road Heng Gao 1 , Zhuolin He 2 , Shoumeng Qiu 2 , Xiangyang Xue 2,∗ , Jian Pu 1 Abstract Semantic segmentation allows autonomous driving cars to understand the surroundings of the vehicle comprehensively. However, it is also crucial for the model to detect obstacles that may jeopardize the safety of autonomous driving systems. Based on our experiments, we find that current uni-modal anomaly segmentation frameworks tend to produce high anomaly scores for non-anomalous regions in images. Motivated by this empirical finding, we develop a multi-modal uncertainty-based anomaly segmentation framework, named MMRAS+, for autonomous driving systems. MMRAS+ effectively reduces the high anomaly outputs of non-anomalous classes by introducing text-modal using the CLIP text encoder. Indeed, MMRAS+ is the first multi-modal anomaly segmentation solution for autonomous driving. Moreover, we develop an ensemble module to further boost the anomaly segmentation performance. Experiments on RoadAnomaly, SMIYC, and Fishyscapes validation datasets demonstrate the superior performance of our method. The code is available in https://github.com/HengGao12/MMRAS_plus . I Introduction Semantic segmentation [ 1 , 2 , 3 , 4 , 5 , 6 , 7 ] is a powerful tool for autonomous driving systems to understand the surrounding environments of the vehicle. However, when deployed in the real world, autonomous driving models may encounter unknown objects that they have not seen during training. This problem could pose risks to the safety of autonomous driving cars. Therefore, in recent years, researchers have proposed many anomaly segmentation algorithms [ 8 , 9 , 10 , 11 , 12 ] to help the model segment anomalies on the road. Nevertheless, based on our experiments, we observe that the current uni-modal based anomaly segmentation methods [ 13 , 14 , 11 ] tend to produce high anomaly scores for non-anomalous regions in autonomous driving scenes. For instance, as shown in the second row of Fig. 1 , using the architecture proposed in [ 11 ] , both MSP and Mask2Anomaly output high anomaly scores for the tree and sky classes, which belong to non-anomalous objects. Fig. 1: The limitation of current uni-modal based anomaly segmentation approaches for autonomous driving. The uni-modal-based methods may tend to produce high anomaly predictions for non-anomalous regions in the image. See the second row in the above figure, MSP [ 13 ] and Mask2Anomaly [ 11 ] output high anomaly scores for the sky and trees classes in the image on RoadAnomaly [ 15 ] validation dataset. Note that in the Ground Truth image, the red parts represent anomalous objects and the blue parts represent non-anomalous regions. Besides, the closer the color is to red, the more likely it is to be an anomaly, the closer the color is to blue, the less likely it is to be an anomaly . Therefore, from the perspective of multi-modal learning, we propose introducing text-modal into the anomaly segmentation framework to mitigate this issue. The intuition here is to leverage the information from in-distribution classes to “tell” the model in which objects in the image are within the training distribution. Besides, the main challenge in achieving our goal is to find a feasible way to incorporate the text branch into the current image-based anomaly segmentation network. To address this challenge, we present a M ulti- M odal uncertainty-based R oad A nomaly S egmentation framework, named MMRAS+, for autonomous driving scenes. In MMRAS+, we first use a CLIP [ 16 ] text encoder to extract text embeddings from text prompts. Afterward, we take these embeddings as the weight of a segmentation head to obtain the text-image logits. Then, we perform a weighted summation of the text-image logits and the outputs of Mask2Anomaly to get the final logits. We use the final logits to compute the anomaly score maps. To the best of our knowledge, we present the first multi-modal anomaly segmentation solution for autonomous driving scenes. Moreover, to further enhance the anomaly segmentation performance, we develop an ensemble module that also computes a weighted sum of several different anomaly score outputs. The idea of this ensemble module stems from a classic machine learning algorithm called AdaBoost [ 17 ] , which ensembles many weakly learnable algorithms to attain a strongly learnable algorithm. Herein, similarly, we create a strong anomaly detector by integrating different “weak” anomaly scores. The quantitative and qualitative experiments on RoadAnomaly, SegmentMeIfYouCan (SMIYC) and Fishyscapes validation datasets demonstrate that our multi-modal anomaly segmentation framework effectively reduces the anomaly scores of non-anomalous regions in images of autonomous driving scenes. In brief, the main contributions of our paper are: • We propose a multi-modal uncertainty-based anomaly segmentation approach named MMRAS+, which mitigates the issue of producing high anomaly scores for non-anomalous regions in autonomous driving scenes by introducing text-modal into the uni-modal anomaly segmentation framework. • We investigate the application of MaskCLIP [ 18 ] in segmenting anomalies for road scenes. In addition, we develop an ensemble module to further boost the anomaly segmentation performance of our framework. • The quantitative and qualitative experiments on RoadAnomaly, SMIYC, and Fishyscapes validation datasets demonstrate the effectiveness of our method. Fig. 2: The overview of MMRAS for anomaly segmentation in autonomous driving scenes. MMRAS first uses CLIP [ 16 ] text encoder to extract text embeddings from text prompts. Then, we take these text embeddings as the weight of a 1\times 1 convolutional segmentation head to obtain the text-image logits. We use the text-image logits to enhance the original logits output to help the model better detect the anomaly classes in the image. II Related Work II-A Anomaly segmentation In general, the anomaly segmentation approaches can be classified into three categories: (i) uncertainty-based ; (ii) image resynthesis-based ; (iii) outlier exposure-based . The uncertainty-based approaches, originated from image-level Out-of-Distribution (OoD) detection, leverage statistics of logits and softmax probability to detect anomaly objects in the image. In [ 13 ] , Hendrycks et al. propose a strong baseline, named MSP, to detect image-level OoD data. Afterward, to scale MSP to large-scale datasets and real-world cases, they propose MaxLogits [ 19 ] , which even has applications in autonomous driving scenes (the CAOS benchmark, also proposed in [ 19 ] ). To mitigate the degradation caused by the significant differences between the distribution of max logits of each predicted class, Jung et al. propose SML (Standardized Max Logits) [ 10 ] to better apply MaxLogits [ 19 ] to segment anomalies in urban scenes. Besides, they also develop an iterative boundary suppression method to further improve the performance of SML. The image resynthesis-based methods accomplish anomaly segmentation by using the reconstruction error between the input image and its reconstructions. The main challenge here is how to design an effective reconstruction framework. In [ 20 ] , Creusot et al. propose to detect the anomalies by using a Restricted Boltzmann Machine neural network to reconstruct the appearance of the road. In recent work, [ 15 ] propose to reconstruct images from the semantic maps using generative networks [ 21 ] . In SynBoost [ 12 ] , the authors propose an anomaly segmentation framework that produces robust predictions by combining resynthesis-based approaches with uncertainty-based approaches. Moreover, the outlier exposure-based methods leverage auxiliary outlier datasets to train the network