Vision Also You Need: Navigating Out-of-Distribution Detection with Multimodal Large Language Model
Haoran Xu, Yanlin Liu, Zizhao Tong, Jiaze Li, Kexue Fu, Yuyang Zhang, Longxiang Gao, Shuaiguang Li, Xingyu Li, Yanran Xu, Changwei Wang · 2026
Read Vision Also You Need: Navigating Out-of-Distribution Detection with Multimodal Large Language Model by Haoran Xu, Yanlin Liu, Zizhao Tong, Jiaze Li, Kexue Fu, Yuyang Zhang, Longxiang Gao, Shuaiguang Li, Xingyu Li, Yanran Xu, Changwei Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-14052.txt.
Introduction
Haoran Xu and a large team use multimodal AI to spot data that does not fit in 2026. Out-of-distribution detection flags inputs unlike those a system was trained on. CLIP sparked much work on zero-shot detection with no extra training. Current methods draw expert knowledge from large language models to name outliers.
Their work, Vision Also You Need, navigates detection with a multimodal large language model. Such a model reads both images and text together. Better outlier flags protect systems from silent failure on odd inputs. Medical, driving, and security tools all need that guard.
Catalog record
| CREATOR | Haoran Xu, Yanlin Liu, Zizhao Tong, Jiaze Li, Kexue Fu, Yuyang Zhang, Longxiang Gao, Shuaiguang Li, Xingyu Li, Yanran Xu, Changwei Wang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 55 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-14052
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Vision Also You Need: Navigating Out-of-Distribution Detection with Multimodal Large Language Model Haoran Xu, Yanlin Liu, Zizhao Tong, Jiaze Li, Kexue Fu, Yuyang Zhang, Longxiang Gao, Shuaiguang Li, Xingyu Li, Yanran Xu, Changwei Wang Abstract Out-of-Distribution (OOD) detection is a critical task that has garnered significant attention. The emergence of CLIP has spurred extensive research into zero-shot OOD detection, often employing a training-free approach. Current methods leverage expert knowledge from large language models (LLMs) to identify potential outliers. However, these approaches tend to over-rely on knowledge in the text space, neglecting the inherent challenges involved in detecting out-of-distribution samples in the image space. In this paper, we propose a novel pipeline, MM-OOD, which leverages the multimodal reasoning capabilities of MLLMs and their ability to conduct multi-round conversations for enhanced outlier detection. Our method is designed to improve performance in both near OOD and far OOD tasks. Specifically, (1) for near OOD tasks, we directly feed ID images and corresponding text prompts into MLLMs to identify potential outliers; and (2) for far OOD tasks, we introduce the sketch-generate-elaborate framework: first, we sketch outlier exposure using text prompts, then generate corresponding visual OOD samples, and finally elaborate by using multimodal prompts. Experiments demonstrate that our method achieves significant improvements on widely used multimodal datasets such as Food-101, while also validating its scalability on ImageNet-1K. I Introduction Amidst the continuous developments in deep learning, various [ 46 ] expert models have achieved corresponding success across diverse domains. However, there is a rising focus on addressing model generalization in current research. It is well known that expert models in deep learning are often trained on closed datasets and then tested on expert-specific tasks which requires that training and testing datasets share identical distribution. Nevertheless, in complex open-world scenarios, the distribution of the training set and the test set is often considerable inconsistency, even substantial divergence, which often leads to the failure of the expert model in the real scenario. In some special scenarios, such as autonomous driving scenarios [ 18 ] , the impact of these errors and failures can be profoundly serious. Therefore, it is equally important to identify out-of-distribution (OOD) samples and maintain good performance on ID samples. Previous methods usually focus on detecting OOD examples only using closed dataset in training to predict OOD samples by lower confidence or higher energy. However, these methods are often less effective in the real scene. ATOM [ 7 ] claimed that selecting more challenging outlier samples can help the model learn a better decision boundary [ 26 ] between ID and OOD. ID-like [ 4 ] proposed that the sampling ID-like samples come from the ID samples without auxiliary outliers dataset, because most challenging OOD samples can improve the performance. Recently, MCM [ 30 ] introduces the zero-shot OOD detection method which borrows large-scale VLMs, i.e., CLIP [ 33 ] . (a) ATOM (b) ID-like (c) EOE (d) MM-OOD (Ours) Fig. 1: Illustrations of (a) ATOM, (b) ID-like, (c) EOE, and (d) MM-OOD (Ours). Previous methods such as (a), (b), and (c) typically utilized knowledge from a single modality (either text or vision) or relied on auxiliary datasets. Our approach stimulates the multimodal reasoning and multi-round dialogue capabilities of MLLMs. Further, EOE [ 6 ] extends the training-free framework [ 28 ] by addressing the limitation of MCM in understanding out-of-distribution (OOD) samples. It leverages a large language model (LLM) to generate textual descriptions for OOD samples, thereby enhancing the model’s ability to envision potential outlier exposure under a zero-shot OOD detection setting. To sum up, the key to improve OOD detection is how to effectively generate outlier samples, regardless of whether the approach is training-free or involves fine-tuning. Hence, EOE attempts to leverage the reasoning ability of LLMs to address this challenge without other auxiliary datasets, but it overlooks one of the most remarkable properties of LLMs: multi-round conversations. Meanwhile, with the advancement of multimodal large language models (MLLMs), multimodal reasoning has demonstrated significant potential. In this paper, we take a step further in EOE and employ multimodal large language models (MLLMs) to address this challenge. For near OOD task and far OOD task, we proposal a detailed and targeted pipeline which harnesses the Multimodal reasoning and Multi-round conversation capabilities [ 54 ] to envision potential outlier exposure for Out-Of-Distribution detection, termed MM-OOD. Technically, we straightforwardly feed ID images and corresponding text prompt into MLLMs for near OOD task to propose potential outlier exposure. Nonetheless, in far OOD tasks, we find that the outlier samples generated by MLLMs when feeding images tend to be very similar to the input image space, leading to the failure of the pipeline in handling far OOD tasks. Upon analysis, we infer that the outlier exposure provided by MLLMs is biased toward sampling near the input image space. When ID samples are fed into MLLMs, this bias restricts the model’s imaginative exploration of its stored knowledge. This observation motivates us to employ pretrained generative models to produce OOD samples into MLLMs. Thus, we design the sketch-generate-elaborate framework which is the form of multi-round conversations: sketch outlier exposure by text prompts, generate corresponding visual OOD samples, and elaborate by visual OOD samples and fine-grained text prompts. Our contributions can be summarized as follows. First, we propose leveraging the multi-modal reasoning and multi-round conversation capability of multimodal large language models (MLLMs) to enhance outlier exposure. Second, we introduce MM-OOD, which incorporates differentiated designs for near OOD and far OOD tasks: for the former, we feed the in-distribution (ID) image and text prompt directly into MLLMs; for the latter, we propose a sketch-generate-elaborate framework. Third, our method MM-OOD consistently outperforms current state-of-the-art baselines across extensive experiments. II Related Works II-A Traditional OOD Detection. Traditional out-of-distribution (OOD) detection has been extensively studied in both visual [ 24 ] and textual [ 13 , 21 , 3 ] classification tasks. Existing methods improve OOD detection through various strategies, including logit-based scoring [ 12 , 25 ] , distance-based detectors [ 31 ] , robust representation learning [ 51 ] , and pseudo-OOD sample generation [ 1 ] . These approaches can be broadly categorized into classification-based [ 47 , 17 ] , density-based [ 34 , 45 ] , and reconstruction-based [ 52 ] techniques. Classification-based methods rely on a well-trained in-distribution (ID) classifier and construct a scoring function to identify OOD samples. This scoring function may be derived from the input, hidden [ 37 , 22 , 36 , 53 , 42 ] , output [ 38 , 12 , 25 ] , or gradient spaces [ 15 , 35 ] . However, when the label space of the test data differs from that of the training data, conventional OOD detection methods typically require either retraining the model from scratch or fine-tuning, which incurs substantial computational overhead. II-B Zero-Shot OOD Detection In the zero-shot setting, which leverages vision-language models (VLMs) without requiring further training on in-distribution (ID) data, zero-shot out-of-distribution OOD detection methods have shown promising results. However, recent works such as MCM [ 29 ] rely on closed-set ID labels and underutilize CLIP’s open-world potential. ZOC [ 10 ] and CLIPN [ 41 ] address the open-world setting but require additional