VisualQuest: A Diverse Image Dataset for Evaluating Visual Recognition in LLMs
Kelaiti Xiao 1,2 , Liang Yang 1 , PAERHATI Tulajiang 1,2 , Hongfei Lin 1 · 2025
Read VisualQuest: A Diverse Image Dataset for Evaluating Visual Recognition in LLMs by Kelaiti Xiao 1,2 , Liang Yang 1 , PAERHATI Tulajiang 1,2 , Hongfei Lin 1 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-19936.txt.
Introduction
Kelaiti Xiao and colleagues built a benchmark of stylized images that probe abstract visual reasoning in a 2025 paper. VisualQuest holds thousands of non-photographic images across public figures, popular culture, linguistic expressions, and literary works, each paired with questions demanding symbolic, cultural, and language knowledge rather than plain object naming. Captioning skill alone cannot crack a visual pun or an emoji riddle, which is exactly the point of the set.
Ten leading multimodal models were tested. Only two, named in the paper, performed strongly overall, and a small slice of images defeated every model. One leader excelled at stylized public figures, the other at linguistic puzzles. The dataset maps where machine visual understanding still thins out.
Catalog record
| CREATOR | Kelaiti Xiao 1,2 , Liang Yang 1 , PAERHATI Tulajiang 1,2 , Hongfei Lin 1 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 23 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-19936
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
VisualQuest: A Diverse Image Dataset for Evaluating Visual Recognition in LLMs Kelaiti Xiao 1,2 , Liang Yang 1 , PAERHATI Tulajiang 1,2 , Hongfei Lin 1 Abstract This paper introduces VisualQuest, a novel image dataset designed to assess the ability of large language models (LLMs) to interpret non-traditional, stylized imagery. Unlike conventional photographic benchmarks, VisualQuest challenges models with images that incorporate abstract, symbolic, and metaphorical elements, requiring the integration of domain-specific knowledge and advanced reasoning. The dataset was meticulously curated through multiple stages of filtering, annotation, and standardization to ensure high quality and diversity. Our evaluations using several state-of-the-art multimodal LLMs reveal significant performance variations that underscore the importance of both factual background knowledge and inferential capabilities in visual recognition tasks. VisualQuest thus provides a robust and comprehensive benchmark for advancing research in multimodal reasoning and model architecture design. † † address: 1 Computer Science and Technology, Dalian University of Technology, Dalian, China 2 School of Computer Science and Technology, Xinjiang Normal University, Urumqi, China Email: {xklt@mail.,liang@,prht@mail.,hflin@}dlut.edu.cn 1 Introduction Large Language Models (LLMs) have demonstrated remarkable success in processing complex textual data, and their capabilities have recently extended into multimodal tasks that integrate visual and textual inputs [ 1 ] . Applications such as automated captioning of everyday scenes and the analysis of medical imagery have showcased these models’ enhanced capacity to interpret visual information when combined with robust learning paradigms [ 2 ] . Nevertheless, most existing multimodal benchmarks predominantly focus on direct image captioning or coarse-grained classification tasks using real-world datasets like ImageNet [ 3 ] . While valuable, these approaches do not adequately assess the ability of LLMs to leverage domain-specific background knowledge in more sophisticated recognition scenarios. To bridge this gap, we introduce “VisualQuest,” a dataset specifically designed to evaluate the performance of LLMs on images that deviate from conventional photographic representations. Unlike traditional datasets, VisualQuest centers on visuals infused with stylized, symbolic, and at times surrealistic elements. Popular imagery—such as caricatures, minimalistic designs, emoji-based symbols, and creative illustrations—presents unique recognition challenges that extend beyond simple object detection. Instead, successful interpretation demands familiarity with cultural, literary, or domain-specific references and the ability to decode abstract visual cues [ 4 ] . VisualQuest comprises 3,529 images organized into four thematic categories: (1) Public Figures, (2) Popular Media, (3) Linguistic Expressions, and (4) Literature Works. Each category features artistic renditions—including memoji, caricature, symbolism, and visual puns—that intensify the complexity of the recognition task. For instance, cartoonish portrayals of renowned authors necessitate an understanding of literary figures, while minimalistic movie posters require comprehension of key cinematic elements. Moreover, when images incorporate symbolic idioms, proverbs, or slang, the recognition task relies on linking abstract visual clues with the appropriate cultural or linguistic concepts. In summary, VisualQuest offers a systematic benchmark that transcends standard object recognition by integrating domain-specific knowledge with abstract visual reasoning. By challenging LLMs to interpret non-traditional imagery, our work aims to contribute a valuable foundation for future advancements in multimodal reasoning and model architecture design. 2 Related Works Multimodal benchmarks and datasets have played a crucial role in advancing the field of vision-language models (VLMs) [ 5 ] . Widely used datasets such as MS COCO [ 6 ] , Flickr30k [ 7 ] , and Visual Genome [ 8 ] have been instrumental in evaluating tasks like image captioning and visual question answering . These benchmarks have facilitated significant progress in multimodal learning, enabling models to generate descriptive captions and answer questions based on visual content [ 9 ] . However, they predominantly focus on real-world photographic representations, which may not fully challenge the interpretive capabilities of large language models (LLMs) when it comes to non-traditional images. This limitation highlights the need for datasets like VisualQuest, which provide a platform to assess LLMs’ ability to interpret stylized, symbolic, and surrealistic images, thereby pushing the boundaries of current multimodal benchmarks. Before LLMs, Language-vision models such as ViLBERT [ 10 ] , LXMERT [ 11 ] , and UNITER [ 12 ] have demonstrated impressive performance on existing multimodal benchmarks. These models leverage transformer-based architectures to process and integrate visual and textual data, enabling them to handle complex visual reasoning tasks. For instance, they excel in object detection and scene understanding tasks [ 13 ] . However, despite their capabilities, these models often struggle with incorporating domain-specific knowledge and interpreting stylized or symbolic images. Their performance tends to degrade when faced with abstract visual cues that demand contextual understanding beyond the literal content. This limitation underscores the gap that VisualQuest aims to address by providing a dataset that challenges models to go beyond simple object detection and engage in deeper interpretive tasks, particularly when confronted with non-conventional visual representations. Recent research has explored visual reasoning tasks that require domain-specific knowledge, such as understanding visual metaphors [ 14 ] , interpreting artistic styles [ 15 ] , and recognizing cultural references in images [ 16 ] . These tasks present unique challenges, as they demand a nuanced understanding of context and symbolism that current models often lack. For example, the field of Abstract Visual Reasoning (AVR) investigates problems that require formulating analogies between abstract visual concepts, often necessitating the application of previously gained skills and knowledge in new settings [ 17 ] . Additionally, neurosymbolic approaches have been proposed to enhance visual reasoning capabilities by combining neural networks with symbolic reasoning, leveraging scene graphs and common sense knowledge to better understand and interpret complex visual scenes [ 18 ] . Despite these advancements, significant challenges remain in tasks that require deep cognitive understanding and domain-specific reasoning. The introduction of benchmarks like NTSEBench highlights the gaps in current models’ abilities to handle complex reasoning tasks [ 19 ] . VisualQuest positions itself as a dataset that bridges the gap between visual perception and domain-specific knowledge, enabling a more comprehensive assessment of LLMs’ multimodal reasoning capabilities, particularly in interpreting stylized and symbolic visual content. 3 Dataset Details This section describes the construction of the VisualQuest dataset, detailing the data collection process, filtering and curation pipelines, presentation method labeling, as well as annotation steps for questions and standardized labels. The final dataset consists of 3,529 images across four main categories and highlights multiple presentation styles specifically chosen to challenge conventional image-recognition capabilities in LLMs. 3.1 Data Collection We recruited 10 student collectors, each instructed to gather 500 images from various online resources. The images were required to fulfill the following criteria: • Belong to one of four targeted categories: Public Figures, Popular Media, Linguistic Expressions , and Literature Works . • Use non-realistic or stylized presentation methods—examples included caricature , illustration , emoji combination , or visual pun (among others). • Include sufficient cultural, literary, or symbolic elements that deviate from straightforward photographic representations. Throughout this stage, example images were provided to each collector to illustrate how stylized or symbolic depictions might appear. This example set helped ensure that the collected images reflected the dataset’s emphasis on non-traditional, symbolic cues. 3.2 Data Curation and Annotation We began with an initial pool of 5,000 images sourced from decentralized collection efforts. To produce a concise and high-quality dataset, we applied several filtering and annotation steps described below, ultimately arriving at 3,529 images. Duplicate Removal