Reading, Not Thinking: Understanding and Bridging the Modality Gap When Text Becomes Pixels in Multimodal LLMs
Kaiser Sun, Xiaochuang Yuan ⋆, Hongjun Liu ⋆, Chen Zhao, Cheng Zhang, Mark Dredze, Fan Bai · 2026
Read Reading, Not Thinking: Understanding and Bridging the Modality Gap When Text Becomes Pixels in Multimodal LLMs by Kaiser Sun, Xiaochuang Yuan ⋆, Hongjun Liu ⋆, Chen Zhao, Cheng Zhang, Mark Dredze, Fan Bai (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-09095.txt.
Introduction
Kaiser Sun and colleagues study why multimodal models do worse when the same text arrives as an image rather than as tokens. Across seven models and seven benchmarks, the gap swings with font and resolution, and real document pages often show far smaller gaps. Reading thousands of errors, they find the main cause: image input alone makes models reason far less, giving much shorter replies. Fine-tuning on the model's own text-mode traces closes the gap and carries to new tests.
Catalog record
| CREATOR | Kaiser Sun, Xiaochuang Yuan ⋆, Hongjun Liu ⋆, Chen Zhao, Cheng Zhang, Mark Dredze, Fan Bai |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 62927 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.09095
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Reading, Not Thinking: Understanding and Bridging the Modality Gap When Text Becomes Pixels in Multimodal LLMs Kaiser Sun, Xiaochuang Yuan ⋆, Hongjun Liu ⋆, Chen Zhao, Cheng Zhang, Mark Dredze, Fan Bai Abstract Multimodal large language models (MLLMs) can process text presented as images, yet they often perform worse than when the same content is provided as textual tokens. We systematically diagnose this “modality gap” by evaluating seven MLLMs across seven benchmarks in five input modes, spanning both synthetically rendered text and realistic document images from arXiv PDFs to Wikipedia pages. We find that the modality gap is task- and data-dependent. For example, math tasks degrade by over 60 points on synthetic renderings, while natural document images often match or exceed text-mode performance. Rendering choices such as font and resolution are strong confounds, with font alone swinging accuracy by up to 47 percentage points. To understand this, we conduct a grounded-theory error analysis of over 4,000 examples, revealing that image mode selectively amplifies reading errors (calculation and formatting failures) while leaving knowledge and reasoning errors largely unchanged, and that some models exhibit a chain-of-thought reasoning collapse under visual input. Motivated by these findings, we propose a self-distillation method that trains the model on its own pure text reasoning traces paired with image inputs, raising image-mode accuracy on GSM8K from 30.71% to 92.72% and transferring to unseen benchmarks without catastrophic forgetting. Overall, our study provides a systematic understanding of the modality gap and suggests a practical path toward improving visual text understanding in multimodal language models. 1 Introduction Humans read text through vision: we perceive words as visual patterns on pages, screens, and signs, not as sequences of abstract symbols [ 16 , 17 ] . Modern multimodal large language models (MLLMs), however, process text and images through separate pathways, compressing visual input into tokens before fusing it with textual representations [ 25 , 3 , 9 , 1 , 36 ] . When the same textual content is presented as an image rather than as text tokens, these models often produce different, and frequently worse, predictions ( Figure 1 ). This “modality gap” has been documented across math, coding, and knowledge tasks [ 27 , 35 ] , and motivates a growing line of work on visual text processing, including screenshot language models [ 32 , 18 ] , vision-centric tokenization [ 43 , 10 ] , and unified multimodal architectures [ 40 ] . Yet existing work predominantly focuses on either documenting the gap or building new systems to circumvent it, without systematically explaining why the gap arises in current MLLMs, characterizing when visual input helps versus hinders, or showing how the gap can be closed with minimal intervention. A further limitation is that existing evaluations rely almost exclusively on synthetically rendered text images, potentially confounding rendering artifacts with genuine perceptual deficits. In this work, we diagnose the modality gap and show it can be substantially narrowed. We evaluate seven MLLMs across seven benchmarks in five input modes ( Figure 2 ), spanning synthetically rendered text and realistic document images from arXiv PDFs and Wikipedia pages. Two OCR-based modes decompose the visual pipeline into reading and reasoning stages, isolating their respective contributions to the gap. We find that the gap is task- and data-dependent, not universal. On synthetic benchmarks, math tasks show gaps exceeding 60 points while knowledge-intensive tasks show modest drops. On natural document images, MLLMs often match or exceed their text-mode performance, suggesting the gap partly reflects distributional mismatch between synthetic renderings and pretraining data. Consistent with this, rendering choices turn out to be a critical confound: font choice alone swings accuracy by up to 47%, and resolution sensitivity varies dramatically across architectures. To understand why image mode degrades performance, we conduct a grounded-theory error analysis of over 4,000 errors. The taxonomy reveals that image mode selectively amplifies perceptual errors (calculation and format errors increase by 1.5\times ) while leaving conceptual and reasoning errors largely unchanged. We also identify a chain-of-thought reasoning collapse, where models produce dramatically shorter outputs in image mode, bypassing step-by-step reasoning. Together, these findings establish that image mode impairs reading, not thinking . Figure 1 : Humans read text through vision from diverse sources such as books, webpages, and documents, whereas MLLMs may yield different predictions when the same textual content is presented in different input forms. Figure 2 : Diagnosing the modality gap in visual text understanding. We evaluate MLLMs across five input modes, including pure text, rendered text images, real-world visual text, and two OCR-based diagnostic settings ( OCR-1P and OCR-2P ). Our error taxonomy reveals that while image modality amplifies reading and calculation errors, it leaves underlying reasoning capabilities largely intact but also reduces the likelihood of triggering a reasoning chain. We further propose remedies to bridge the performance gap, including rendering specification control, resolution-aware preprocessing, and LM-only self-distillation. Motivated by this diagnosis, we propose a self-distillation approach: training the model on its own text-mode reasoning traces paired with image inputs. This raises image-mode accuracy on GSM8K from 30.71% to 92.72%, nearly matching text-mode performance. Both vision encoder and language model adaptation contribute, with the language model playing a larger role, and the gains transfer to unseen benchmarks without catastrophic forgetting. 2 Related Work Visual Text Processing. A growing body of work explores processing language through visual representations rather than discrete tokens. Screenshot language models [ 32 , 18 ] train vision transformers on rendered text with masked patch prediction, progressively closing the gap with text-only models such as BERT on standard NLU benchmarks. On the efficiency front, vision-centric tokenization methods render text as images to reduce token counts: SeeTok [ 43 ] achieves competitive performance with 4.43 \times fewer tokens, while Glyph [ 10 ] demonstrates 3–4 \times context compression via continual pre-training with rendering search. At the architectural level, Emu3 [ 40 ] shows that a single decoder-only transformer performing next-token prediction over a shared discrete space of images, text, and video can match compositional encoder–LLM architectures, establishing unified token prediction as a viable foundation for multimodal learning. These works focus on building better systems , including new training objectives, tokenization schemes, and unified architectures. In contrast, our work takes existing MLLMs as given and first diagnoses why they fail when reading text as images, then shows that the gap can be narrowed with minimal, targeted training rather than architectural redesign. Evaluating MLLMs on Visual Text. Unlike scene-text understanding [ 34 , 5 , 33 ] , which primarily tests text recognition in natural images, recent work has turned to evaluating whether MLLMs can reason over textual content presented visually, documenting a “modality gap” between text and image presentation. PixelWorld [ 27 ] reports that the gap is task-dependent: semantic tasks are largely preserved while math and coding degrade. VISTA-Bench [ 26 ] evaluates over 20 VLMs across perception, reasoning, and knowledge tasks. Li et al. [ 24 ] show that rendering long contexts as images can halve decoder token counts while maintaining accuracy on retrieval and summarization. Most closely related, Sprang et al. [ 35 ] introduce the REST benchmarks to measure cross-modal consistency , i.e., whether models produce the same answer regardless of input modality, finding