How Do Large Vision-Language Models See Text in Image? Unveiling the Distinctive Role of OCR Heads

Ingeol Baek, Hwan Chang, Sunghyun Ryu, Hwanhee Lee · 2025

Read How Do Large Vision-Language Models See Text in Image? Unveiling the Distinctive Role of OCR Heads by Ingeol Baek, Hwan Chang, Sunghyun Ryu, Hwanhee Lee (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-15865.txt.

Introduction

Ingeol Baek, Hwan Chang, Sunghyun Ryu, and Hwanhee Lee probe in 2025 how large vision-language models read text that sits inside an image. Such models now handle image and text at once, yet how they find and read words in a picture was not clear. The team hunts for the inner heads, the parts of the model that focus on set spots, that do that reading.

They find that text reading calls on many heads at once, far more than the few heads used to fetch other facts. Those heads also look unlike standard fetch heads in their traits. How often a head fires tracks its text reading score in a steady way. Tests that hide or boost these heads shift task scores as the account predicts. Moving a sink value inside them lifts results.

Catalog record

CREATORIngeol Baek, Hwan Chang, Sunghyun Ryu, Hwanhee Lee
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-15865

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

How Do Large Vision-Language Models See Text in Image? Unveiling the Distinctive Role of OCR Heads Ingeol Baek, Hwan Chang, Sunghyun Ryu, Hwanhee Lee Abstract Despite significant advancements in Large Vision Language Models (LVLMs), a gap remains, particularly regarding their interpretability and how they locate and interpret textual information within images. In this paper, we explore various LVLMs to identify the specific heads responsible for recognizing text from images, which we term the Optical Character Recognition Head ( OCR Head ). Our findings regarding these heads are as follows: (1) Less Sparse: Unlike previous retrieval heads, a large number of heads are activated to extract textual information from images. (2) Qualitatively Distinct: OCR heads possess properties that differ significantly from general retrieval heads, exhibiting low similarity in their characteristics. (3) Statically Activated: The frequency of activation for these heads closely aligns with their OCR scores. We validate our findings in downstream tasks by applying Chain-of-Thought (CoT) to both OCR and conventional retrieval heads and by masking these heads. We also demonstrate that redistributing sink-token values within the OCR heads improves performance. These insights provide a deeper understanding of the internal mechanisms LVLMs employ in processing embedded textual information in images. † † footnotetext: \dagger Corresponding author. 1 Introduction Figure 1: Visualization of image attention maps from InternVL2-8B. L and H denote the layer and attention head of the LVLM, respectively. Large Vision-Language Models (LVLMs) Liu et al. (2023) ; Li et al. (2023) ; Dong et al. (2024) ; Liu et al. (2024a) extend the capabilities of Large Language Models (LLMs) Zhao et al. (2023) ; Achiam et al. (2023) ; Grattafiori et al. (2024) into multimodal domains. Typically, an LVLM integrates a pretrained vision encoder with pretrained LLM decoder via an adapter, enabling it to generate coherent text grounded in both image and text inputs. The power of LVLMs lies in their deep understanding of image-text relationships, which is evident in tasks like VQA Antol et al. (2015) ; Goyal et al. (2017) and particularly those relying on text embedded in images, such as OCR-VQA. LVLMs have demonstrated impressive performance on these OCR-centric tasks Mathew et al. (2021) ; Huang et al. (2024) ; Reddy et al. (2024) . Despite these advancements, there is limited interpretability research investigating the internal mechanisms driving this performance, specifically how LVLMs handle textual information within images. Recent interpretability work has identified retrieval heads in LLMs—attention units that copy relevant tokens from long textual contexts to support long-context factuality Wu et al. (2024b) —and visual grounding heads that localize image regions in response to textual queries Xiao et al. (2024) ; Wu et al. (2024a) ; Kang et al. (2025b) . However, these studies do not address how LVLMs internally locate and extract embedded text within images. We hypothesize that LVLMs contain a distinct class of attention heads— Optical Character Recognition (OCR) heads —that operate independently of copy-paste retrieval heads. While retrieval heads attend to text tokens in the input sequence and replicate them in the output, OCR heads should selectively attend to visual patches corresponding to characters or words and directly guide text extraction from images. To test this hypothesis, we convert the Passkey and Needle-in-a-Haystack (NIAH) Kamradt (2023) benchmarks into a multi-image QA setup requiring model to answer by copying text from rendered image patches. By analyzing attention distributions during answer generation (see Figure 1 ), we observe that certain heads consistently concentrate on the ground-truth text regions, supporting the existence of specialized OCR heads. Based on these observations, we formulate two core research questions: RQ1. How can we identify the OCR heads? RQ2. How do OCR heads differ from existing retrieval heads? To answer these questions, we introduce an algorithm for detecting OCR heads in multi-image LVLMs. Our experiments reveal three key properties that distinguish OCR heads from retrieval heads: (1) Less Sparsity: OCR heads activate more densely across instances compared to the 3–5% active rate of conventional retrieval heads. (2) Qualitative Distinctiveness: OCR heads exhibit low overlap with retrieval heads, indicating they form an independent functional group. (3) Static Activation Patterns: OCR head activations remain consistent across diverse contexts, unlike context-sensitive retrieval heads. To validate the specialized role of OCR heads, we evaluate their behavior in downstream tasks via CoT Wei et al. (2022) ; Li et al. (2025) prompting, attention masking, and attention redistribution. These studies not only confirm the mechanistic role of OCR heads in reading embedded text but also demonstrate that manipulating their attention distributions can improve OCR-VQA performance. Our findings fill a critical gap in LVLM interpretability and provide actionable insights for enhancing multimodal reasoning and reducing hallucination in OCR-centric applications. We summarize our contributions as follows: • We propose a scoring-based method for automatically identifying OCR heads in LVLMs. • We demonstrate that OCR heads exhibit reduced sparsity, qualitative distinctiveness, and static activation patterns across diverse contexts. • Through CoT prompting, targeted head masking, and strategic sink-token redistribution, we confirm OCR heads’ specialized role in text extraction from images. 2 Preliminaries 2.1 Retrieval Heads Retrieval heads are identified as a specific type of attention head primarily responsible for identifying and copying relevant information from the input context to the generated output. Wu et al. (2024b) These heads are understood to play a crucial role in the model’s ability to retrieve factual information, particularly from long input sequences. They are mechanistically linked to the process of extracting and repeating input tokens. 2.2 Retrieval Score To quantitatively identify which attention heads are responsible for copying relevant tokens from the input, a retrieval score has been defined. Wu et al. (2024b) This score serves to measure how consistently a specific attention head engages in this copy-paste behavior. Given a question q , an answer k , and a context x , the model addresses a NIAH task—locating k within x based on q . Let h denote attention head, w the generated token, and a\in\mathcal{R}^{|x|} attention score; then two conditions must be satisfied: (1) the generated token w is part of the target sentence k , and (2) the input token x_{j} with the highest attention aligns exactly with w , formally x_{j}=w,\quad j=\text{argmax}(a) . Under these conditions, the retrieval score for a head h is calculated as: \text{Retrieval Score for head }h=\frac{|g_{h}\cap k|}{|k|} (1) where g_{h} represents the set of all correctly identified (copied-and-pasted) tokens by head h . Intuitively, this retrieval score measures the token-level recall capability of a specific head. Retrieval score is computed over many NIAH instances, and heads achieving a score above 0.1 are classified as retrieval heads. 3 Identifying Heads for Text Recognition Figure 2: The figure illustrates the image input format designed to identify tokens responsible for copy-pasting text within the image. Tokens corresponding to the patch-to-bounding box intersection ratio serve as evidence patch tokens. The retrieval score computation utilizes attention between generated tokens and evidence patch tokens. In this section, we describe our method for identifying attention heads within LVLMs that are primarily responsible for retrieving textual information from images. We term these specialized heads as Optical Character Recognition Heads (OCR heads). We adapt the concept of the retrieval score Wu et al. (2024b) , originally used to quantify copy-paste

Read the complete text