Texture or Semantics? Vision-Language Models Get Lost in Font Recognition
Zhecheng Li † Guoxian Song ∥ Yujun Cai Zhen Xiong Junsong Yuan ⋆ Yiwei Wang ‡ † University of California, San Diego ∥ ByteDance · 2025
Read Texture or Semantics? Vision-Language Models Get Lost in Font Recognition by Zhecheng Li † Guoxian Song ∥ Yujun Cai Zhen Xiong Junsong Yuan ⋆ Yiwei Wang ‡ † University of California, San Diego ∥ ByteDance (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-23768.txt.
Introduction
Zhecheng Li and five co-authors test whether vision-language models can recognize fonts in a 2025 paper. People often meet a pleasing typeface in a magazine or brand design and want its name. Multimodal models seem like handy tools for that job, but fine visual detail may trip them up.
Their benchmark renders text in fifteen common fonts at two difficulty levels. The hard set prints the font names themselves, creating a clash between word meaning and letter shape. Models must report the shape, not read the word. Results probe whether the systems lean on texture or on semantics. The study warns that broad vision skill does not guarantee success on fine-grained design tasks.
Catalog record
| CREATOR | Zhecheng Li † Guoxian Song ∥ Yujun Cai Zhen Xiong Junsong Yuan ⋆ Yiwei Wang ‡ † University of California, San Diego ∥ ByteDance |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-23768
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Texture or Semantics? Vision-Language Models Get Lost in Font Recognition Zhecheng Li † Guoxian Song ∥ Yujun Cai Zhen Xiong Junsong Yuan ⋆ Yiwei Wang ‡ † University of California, San Diego ∥ ByteDance Abstract Modern Vision-Language Models (VLMs) exhibit remarkable visual and linguistic capabilities, achieving impressive performance in various tasks such as image recognition and object localization. However, their effectiveness in fine-grained tasks remains an open question. In everyday scenarios, individuals encountering design materials, such as magazines, typography tutorials, research papers, or branding content, may wish to identify aesthetically pleasing fonts used in the text. Given their multimodal capabilities and free accessibility, many VLMs are often considered potential tools for font recognition. This raises a fundamental question: Do VLMs truly possess the capability to recognize fonts? To investigate this, we introduce the Font Recognition Benchmark (FRB) , a compact and well-structured dataset comprising 15 commonly used fonts. FRB includes two versions: (i) an easy version , where 10 sentences are rendered in different fonts, and (ii) a hard version , where each text sample consists of the names of the 15 fonts themselves, introducing a stroop effect that challenges model perception. Through extensive evaluation of various VLMs on font recognition tasks, we arrive at the following key findings: (i) Current VLMs exhibit limited font recognition capabilities, with many state-of-the-art models failing to achieve satisfactory performance and being easily affected by the stroop effect introduced by textual information. (ii) Few-shot learning and Chain-of-Thought (CoT) prompting provide minimal benefits in improving font recognition accuracy across different VLMs. (iii) Attention analysis sheds light on the inherent limitations of VLMs in capturing semantic features. 1 Introduction Font recognition is a fundamental visual task with applications across various domains. It is particularly relevant to graphic design, publishing, advertising, and digital content creation, where typography plays a crucial role in visual communication ( Zramdini & Ingold, 1998 ; Chen et al., 2014b ; Zhu et al., 2001 ) . For example, researchers may seek to identify aesthetically appealing fonts in academic papers or images, while designers sometimes encounter unique typefaces they wish to integrate into their projects. Statistics indicate that over 60% of websites utilize custom or non-system fonts, highlighting the significance of font identification in web design. Additionally, Google Fonts are used on more than 50 million websites, demonstrating the extensive adoption of diverse typefaces ( Solomons, 2023 ) . This widespread demand underscores the need for efficient and accurate font recognition tools. Traditionally, Convolutional Neural Networks (CNNs) have been the predominant architecture for font recognition ( Chen et al., 2021 ; Li et al., 2022b ) . While CNNs achieve high accuracy, they rely on large and well-annotated datasets for training ( Wang & Zong, 2023 ; Zhang, 2023 ; Tonmoy et al., 2024 ) . The process of data collection, annotation, and Figure 1: An illustrative example where the image displays the word ’Arial’ rendered in the ’Monoton’ font. The vision-language model incorrectly predicts the font as ’Arial’, indicating an over-reliance on textual content and limited sensitivity to font semantics. curation is time-consuming and resource-intensive, limiting the scalability and adaptability of CNN-based systems in real-world applications. In contrast, VLMs, pre-trained on large-scale multimodal datasets, have demonstrated remarkable capabilities in processing both visual and textual information ( Zhang et al., 2024c ; Bordes et al., 2024 ; OpenAI et al., 2024b ; OpenAI, 2024 ; Team et al., 2024 ; Google, 2025 ) . Their ability to generalize across diverse visual tasks without extensive task-specific training makes them highly versatile. Additionally, many VLMs are open-weight and freely accessible, positioning them as practical alternatives for everyday font recognition. Although VLMs have the potential to aid in font recognition, there has been no comprehensive analysis or systematic evaluation of their capabilities in this domain. In this paper, we introduce a font recognition benchmark comprising 15 commonly used fonts, with images containing textual content rendered in these typefaces. To ensure a rigorous evaluation, we construct two dataset versions: an easy version, where fonts are recognized from images containing sentences, and a hard version, which introduces the stroop effect ( Besner et al., 1997 ; MacLeod, 1991 ; Verhaeghen & De Meersman, 1998 ) by displaying font names in mismatched typefaces, adding an additional layer of difficulty. This benchmark provides a systematic assessment of state-of-the-art VLMs’ font recognition abilities. We conduct extensive evaluations on 13 open-weight and closed-source VLMs. All tested models are struggle to differentiate between font types and exhibit high sensitivity to the textual content within images, as illustrated in Figure 1 . The best-performing model achieves only around 30% accuracy on the easy version of the benchmark, while performance on the hard version is even more challenging, with accuracy dropping to approximately 15%. Additionally, chain-of-thought (CoT) prompting ( Wei et al., 2023 ; Kojima et al., 2023 ) provides limited performance gains in this task. Furthermore, we incorporate single-letter images as demonstrations in few-shot learning scenarios to help VLMs identify font similarities. Interestingly, despite the widespread familiarity of these fonts and the relatively low difficulty of the task for humans, where few-shot examples significantly aid in font recognition, current VLMs continue to exhibit poor performance. In summary, our contributions are threefold: (i) We propose a font recognition benchmark with two versions to systematically evaluate the capabilities of VLMs and conduct a comprehensive assessment of 13 different models. (ii) Our results reveal that current VLMs perform poorly in font recognition, with neither Chain-of-Thought prompting nor few-shot learning providing significant improvements. (iii) We analyze attention matrices across image patches to gain deeper insights into their failure modes and susceptibility to texture-based perturbations. 2 Related Works 2.1 Vision-Language Models With the foundation laid by Transformer and CLIP ( Vaswani et al., 2023 ; Radford et al., 2021 ) , Vision-Language Models (VLMs) have emerged as powerful multimodal systems that seamlessly integrate both visual and linguistic modalities. These models bridges the gap between image and text understanding, enabling them to process and interpret complex multimodal data efficiently ( OpenAI et al., 2024a ; Bai et al., 2025 ; Team et al., 2024 ; Li et al., 2024 ; Liu et al., 2023 ; Caffagni et al., 2024 ; Zhang et al., 2024a ; Yin et al., 2024 ; Team et al., 2025 ; Chen et al., 2024 ) . Recent advancements in large-scale image-text datasets and contrastive learning have significantly improved VLM performance. These models now exhibit strong general representation capabilities, effectively aligning visual and textual information and achieving state-of-the-art results across multiple benchmarks, establishing themselves as key technologies in multimodal AI research ( Zhang et al., 2024c ; Bordes et al., 2024 ; Schuhmann et al., 2022 ; Li et al., 2023 ; Chen et al., 2025 ; Chen et al., 2020 ) . The strong multimodal capabilities of VLMs make them valuable tools in real-world applications. Models such as BLIP ( Li et al., 2022a ) , Flamingo ( Alayrac et al., 2022 ) and later GPT-4o ( OpenAI et al., 2024b ) exhibit impressive performance in visual question answering, image captioning, and multimodal dialogue, enabling applications in interactive AI agents and assistive technologies for visually impaired individuals ( Agrawal et al., 2016 ; Zhang et