Text Speaks Louder than Vision: ASCII Art Reveals Textual Biases in Vision-Language Models
Zhaochen Wang, Bryan Hooi, Yiwei Wang, Ming-Hsuan Yang, Zi Huang, Yujun Cai · 2025
Read Text Speaks Louder than Vision: ASCII Art Reveals Textual Biases in Vision-Language Models by Zhaochen Wang, Bryan Hooi, Yiwei Wang, Ming-Hsuan Yang, Zi Huang, Yujun Cai (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-01589.txt.
Introduction
Zhaochen Wang and colleagues show that ASCII art exposes text bias in vision-language models. Their 2025 arXiv paper studies how models settle clashes between signals. ASCII art builds pictures out of characters with their own meanings. Letter-level sense can thus fight the global visual pattern. The team built a test suite of adversarial ASCII images.
Five leading models faced the crafted art pieces. Models consistently favored the text reading over the picture. Visual recognition fell sharply as semantic complexity grew. Tuning image settings and prompts brought only small gains. The authors say the flaw sits deep in model design. Fixes will likely need architectural change, not surface tweaks. The work warns against trusting multimodal models on mixed signals.
Catalog record
| CREATOR | Zhaochen Wang, Bryan Hooi, Yiwei Wang, Ming-Hsuan Yang, Zi Huang, Yujun Cai |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-01589
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Text Speaks Louder than Vision: ASCII Art Reveals Textual Biases in Vision-Language Models Zhaochen Wang, Bryan Hooi, Yiwei Wang, Ming-Hsuan Yang, Zi Huang, Yujun Cai Abstract Vision-language models (VLMs) have advanced rapidly in processing multimodal information, but their ability to reconcile conflicting signals across modalities remains underexplored. This work investigates how VLMs process ASCII art, a unique medium where textual elements collectively form visual patterns, potentially creating semantic-visual conflicts. We introduce a novel evaluation framework that systematically challenges five state-of-the-art models (including GPT-4o, Claude, and Gemini) using adversarial ASCII art, where character-level semantics deliberately contradict global visual patterns. Our experiments reveal a strong text-priority bias: VLMs consistently prioritize textual information over visual patterns, with visual recognition ability declining dramatically as semantic complexity increases. Various mitigation attempts through visual parameter tuning and prompt engineering yielded only modest improvements, suggesting that this limitation requires architectural-level solutions. These findings uncover fundamental flaws in how current VLMs integrate multimodal information, providing important guidance for future model development while highlighting significant implications for content moderation systems vulnerable to adversarial examples. Warning: this paper contains examples of toxic language used for research purposes. 1 Introduction ASCII art is a unique digital medium that combines textual and visual information. From early BBS forums to modern social media, ASCII art remains a tool for emotional expression and creative display. However, we frequently encounter adversarial ASCII art in online comments, such as skull patterns densely arranged with positive characters like \star and \heartsuit , or negative emotional words constructed using positive words (e.g., “love”, “joy”). This contradiction between text and visual content creates metaphorical or ironic expressions of latent negativity. Such art is dangerous because its unclear boundaries make it hard for automated systems to detect. This allows hate speech or discrimination to spread unnoticed online, making the problem harder to control than direct verbal attacks. Figure 1 illustrates an example of adversarial ASCII art. A natural question is whether visual-language models (VLMs) can recognize and understand such “adversarial ASCII art”. Prior research ( Todd et al. 2023 ; Dąbkowski & Beguš 2023 ) indicates that large language models can understand and generate simple ASCII art (e.g., box diagrams) but struggle with more complex forms ( Wang et al. 2023 ) . Intuitively, VLMs should outperform text-only models by “seeing” the visual patterns rather than processing ASCII art merely as textual input. However, empirical evidence for this assumption is notably lacking, particularly for cases where textual and visual information conflict. To address this gap, we systematically evaluate how VLMs handle ASCII art. We test VLMs using ASCII art images constructed from diverse character sets—ranging from simple symbols and random letters to adversarial text—to examine how textual meaning affects visual understanding. Our experiments with five state-of-the-art VLMs reveal a strong text-priority bias: these models predominantly rely on character-level semantics rather than overall visual patterns when interpreting ASCII art. This bias becomes particularly severe as textual content carries richer semantic meaning, revealing the models’ tendency to ignore visual content in favor of textual information. Following our main experiments, we explored mitigation strategies, including adjusting visual parameters (e.g., font size) and prompting techniques (e.g., few-shot Chain-of-Thought). Although these methods improved VLMs’ performance to some extent, their effectiveness remained limited for samples containing semantically rich characters. Figure 1 : An example of adversarial ASCII art: the VLM can recognize text at the detail level but fails to perceive the macro-visual structure (the ’BAD’ ASCII art). The main contributions of this work are: • We evaluate five state-of-the-art VLMs on ASCII art tasks and introduce “adversarial ASCII art” as a challenging test case. • We identify a strong text-priority bias in VLMs when processing ASCII art and find that this bias becomes more severe when the characters are semantically complex. • We test mitigation strategies such as visual adjustments and prompting techniques but find only limited improvements, underscoring the need for architectural changes. The paper is structured as follows: Section 2 reviews related work; Section 3 covers the dataset and evaluation framework; Section 4 presents experiments and results; Section 5 summarizes findings and future directions for VLMs. 2 Related Work 2.1 Research on ASCII Art In the 1980s-90s, with the rise of BBS (electronic bulletin board systems), ASCII art flourished and created a unique digital culture ( Danet 2020 ) . Narrow ASCII art can only use the 95 printable characters defined by the 1963 ASCII standard (128 total) ( Xu et al. 2016 ) . Broad ASCII art, in contrast, has a variety of types and styles, such as “Typewriter-style” lettering, line art, solid art and so on. Early ASCII art research primarily focused on developing effective generation methods. Initial approaches focused on tone-based generation techniques that mapped grayscale values from natural images to ASCII character densities ( Prabagar & Vasuki 2012 ; O’Grady & Rickard 2008 ) . Subsequent advancements introduced structure-based generation, which optimizes character selection through structural analysis of source images ( Xu et al. 2010 ; Miyake et al. 2011 ; Xu et al. 2015 ) . Comparative studies demonstrate that structure-based methods surpass tone-based approaches in achieving visual simplicity and maintaining recognizability at low resolutions. Recent scholarship explores ASCII art’s emerging role in large language model (LLM) research. While multiple studies confirm LLMs’ basic capacity to recognize and generate ASCII art ( Dąbkowski & Beguš 2023 ; Todd et al. 2023 ; Bayani 2023 ; Jia et al. 2024 ; Wu et al. 2024 ) , critical limitations persist. Notably, Wang et al. 2023 reveals a stark performance gap: LLMs achieve merely 8% accuracy versus humans’ 94% in ASCII art reasoning tasks, with minimal improvement from Chain-of-Thought prompting or Python API integration. This deficiency has inspired novel applications, including Jiang et al. 2024 ’s ASCII art jailbreak prompts that circumvent LLM alignment safeguards, and Berezin et al. 2024 ’s demonstration of LLMs’ failure to detect toxic content encoded in ASCII art. 2.2 VLM and language bias Vision-language models (VLMs) have advanced rapidly in recent years. Advanced models like CLIP ( Radford et al. 2021 ) , GPT-4o ( Yang et al. 2023 ) , Claude ( Anthropic 2024 ) and Flamingo ( Alayrac et al. 2022 ) achieve state-of-the-art performance on many multimodal tasks. However, VLMs still face many challenges. A persistent challenge is multimodal alignment: VLMs often generate descriptions that misinterpret or incompletely capture visual content ( Wang et al. 2024b ) . This misalignment originates from the difference in training data and training process for different modalities ( Bartsch et al. 2023 ; Rabinovich et al. 2023 ) . Language bias is a consequence of multimodal misalignment. Many studies have found language bias in VLMs ( Niu et al. 2021 ; Zhang et al. 2024 ; Wang et al. 2024a ; Goyal et al. 2017 ; Thrush et al. 2022 ) : Model tendency to rely primarily on linguistic correlations or language patterns for decision making and ignore visual information. Additionally, Cao et al. 2024 , Goh et al. 2021 and Azuma & Matsui 2023 all mentioned the concept of typographic attack: VLMs systematically misclassified images that add adversarial text. 3 Methodology 3.1 Text-Visual Conflict in ASCII Art ASCII art presents a unique opportunity for examining