Large Language Models Facilitate Vision Reflection in Image Classification

Guoyuan An, JaeYoon Kim, SungEui Yoon · 2025

Read Large Language Models Facilitate Vision Reflection in Image Classification by Guoyuan An, JaeYoon Kim, SungEui Yoon (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-06525.txt.

Introduction

Guoyuan An, JaeYoon Kim, and SungEui Yoon study in 2025 how large models that see and read can double-check image labels. A vision model first names what is in a picture. Then a big language model reviews that guess, a step the authors call vision reflection.

Surprisingly, this review raised accuracy even on hard standard tests, though big models alone often lag behind specialist vision tools. Looking inside, the team found the model turns image features into a few clear text ideas, then reasons about them with common sense. Most image tokens could be swapped for a handful of text tokens with little loss. A simple add-on connector also lifted fine-grained naming, with no extra training.

Catalog record

CREATORGuoyuan An, JaeYoon Kim, SungEui Yoon
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-06525

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Large Language Models Facilitate Vision Reflection in Image Classification Guoyuan An, JaeYoon Kim, SungEui Yoon Abstract This paper presents several novel findings on the explainability of vision reflection on recognition in large multimodal models (LMMs). First, we show that prompting an LMM to verify the prediction of a specialized vision model can improve recognition accuracy—even on benchmarks like ImageNet—despite prior evidence that LMMs typically underperform dedicated vision encoders. Second, we analyze the internal behavior of vision reflection and find that the vision-language connector maps visual features into explicit textual concepts, allowing the language model to reason about prediction plausibility using commonsense knowledge. We further observe that replacing a large number of vision tokens with only a few text tokens still enables LLaVA to generate similar answers, suggesting that LMMs may rely primarily on a compact set of distilled textual representations rather than raw vision features. Third, we show that a training-free connector can enhance LMM performance in fine-grained recognition tasks, without extensive feature-alignment training. Together, these findings offer new insights into the explainability of vision-language models and suggest that vision reflection is a promising strategy for achieving robust and interpretable visual recognition. 1 Introduction. The capacity for reflection , or self-correction , is increasingly recognized as a hallmark of intelligence in large language models (LLMs), allowing them to scrutinize outputs, identify errors, and iteratively refine responses, thus enhancing performance ( Shinn et al. 2024 ; Kumar et al. 2024 ; Huang et al. 2024 ) . Recently, a few studies have begun to explore reflection in multimodal tasks ( Cheng et al. 2024 ; Yao et al. 2024 ) , where visual perception serves as the primary information source. For example, accurately recognizing an obstacle on the road is essential for making reliable decisions in autonomous driving. While these works show that reflection can improve LMM performance in multimodal reasoning, they do not demonstrate improvements in core visual recognition accuracy during the reflection process. In contrast, numerous recent studies consistently report that LMMs typically underperform specialized vision encoders on direct classification tasks ( Zhang et al. 2024 ; Wu et al. 2023a ; Hu et al. 2024 ; Yu et al. 2025 ) . For example, even advanced models like GPT-4V achieve lower accuracy (60.6%) on ImageNet compared to dedicated vision encoders such as CLIP-L (74.8%) ( Zhang et al. 2024 ) . In this paper, we propose a two-stage classification pipeline: (1) an initial inference generated solely by a vision encoder, followed by (2) a reflective verification prompted to the LMM (e.g., “Does the picture have a/an {initially predicted category}?”) (Fig. 1 ). We empirically demonstrate, for the first time, that this vision reflection effectively enhances accuracy on challenging benchmarks such as ImageNet. This finding is notable given prior evidence that LMMs generally perform worse than specialized vision encoders on such tasks ( Zhang et al. 2024 ; Wu et al. 2023a ; Yu et al. 2025 ) . Moreover, previous efforts to improve LMM recognition through data augmentation and tuning have yielded only limited success ( Zhang et al. 2024 ) . Thus, the ability of LMMs to improve vision encoder predictions via reflection presents a surprising and promising direction. Figure 1: We propose integrating a reflection stage into the image recognition pipeline to address potential errors in the initial predictions made by the vision encoder, which may lead to undesirable outcomes. Powered by a LMM, the reflection stage allows the system to reevaluate its initial predictions and actively seek additional information, thereby mitigating risks and enhancing decision-making. To understand how LMMs accomplish vision reflection despite limitations in direct classification, we investigate internal mechanisms using the LLaVA architecture as an exemplar ( Liu et al. 2024a ) . We design a reverse embedding layer to map vision tokens to their most relevant textual descriptors. Applying this reverse embedding layer to vision tokens reveals that the key textual terms used by the LMM to describe an image are directly encoded in these vision tokens. We replace the original image input with the extrated key textual descriptors and observe that the LMM generates similar responses. This shows that vision token features encapsulate most (though not all) of the detailed visual information in an explicit and interpretable manner, rather than relying solely on implicit representations. This insight sheds light on the limitations of LMMs in classification tasks: the vision-language connector module translates visual features into descriptive textual concepts. While these concepts may lack granularity for precise direct classification, they provide sufficient context for common sense reasoning about prediction plausibility. To further substantiate our explanation, we propose and validate a training-free approach for constructing the vision-to-language connector. This method demonstrates that by explicitly mapping image features to a vocabulary of relevant textual terms—without additional feature-alignment training—an LMM can effectively improve performance on fine-grained categories, supporting our hypothesis that LMMs transform visual features into “text-encoding” vision tokens through the connector. Our primary contributions are threefold: 1. We provide the first evidence that LMM reflection effectively enhances image recognition accuracy on standard benchmarks like ImageNet. 2. We discover that the connector in LMMs maps vision tokens into explicit and interpretable textual concepts. Remarkably, replacing a large number of vision tokens with just a few text tokens still enables LLaVA to generate similar answers. This suggests that the LMM may primarily rely on a compact set of distilled textual concepts from the visual input. This finding also provides a novel explanation for how LMMs can perform reflective reasoning over visual predictions. 3. We demonstrate that a training-free connector can enhance LMM performance in fine-grained recognition tasks, even without extensive feature-alignment training. 2 Related Works. Reflection and vision reasoning. Reflection , or self-correction , is a highly desirable capability in large language models (LLMs), referring to their ability to detect and revise their own outputs to progressively enhance response quality. Reflexion ( Shinn et al. 2024 ) demonstrates that converting scalar feedback from the environment into textual summaries allows LLMs to learn complex tasks within just a few trials. ScoRe ( Kumar et al. 2024 ) adopts multi-turn reinforcement learning to train LLMs to correct their own mistakes; unlike Reflexion, it does not rely on ground-truth feedback during the reflection process. Huang et al. ( Huang et al. 2024 ) argue that LLMs contain latent “hidden knowledge” and show that self-improvement can be achieved through either supervised fine-tuning (SFT) or reinforcement learning with human feedback (RLHF). Some recent studies explore visual reasoning ( Diwan et al. 2022 ; Thrush et al. 2022 ; Wu et al. 2023b ; Huang et al. 2025 ; Wang et al. 2024 ) , which requires step-by-step reasoning to answer multimodal queries (e.g., solving math problems with diagrams ( Wang et al. 2024 ) or aligning paired images and captions ( Diwan et al. 2022 ) ). However, no existing work has demonstrated that large multimodal models (LMMs) can improve pure visual recognition performance through reflection. This paper provides the first empirical evidence that vision reflection can enhance visual recognition performance, and offers novel insights into the explainability of this process. LMM on vision recognition. Large multimodal models (LMMs) ( Achiam et al. 2023 ; Team et al. 2023 ) combine

Read the complete text