Unveil: Unified Visual-Textual Integration and Distillation for Multi-modal Document Retrieval

Hao Sun, Yingyan Hou, Jiayan Guo, Bo Wang, Chunyu Yang, Jinsong Ni, Yan Zhang · 2026

Read Unveil: Unified Visual-Textual Integration and Distillation for Multi-modal Document Retrieval by Hao Sun, Yingyan Hou, Jiayan Guo, Bo Wang, Chunyu Yang, Jinsong Ni, Yan Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-24530.txt.

Introduction

Hao Sun and colleagues present Unveil, a 2026 system for finding multimodal documents that mixes words and layout instead of text parsing alone. Text-only tools ignore layout and often misread, while image-only tools miss fine word meaning in text-heavy pages. Unveil first learns a joint word-and-image embedding, then distills it into an image-only model for fast parsing-free search. Tests show the joint model beating prior methods, with distillation closing most of the semantic gap.

Catalog record

CREATORHao Sun, Yingyan Hou, Jiayan Guo, Bo Wang, Chunyu Yang, Jinsong Ni, Yan Zhang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34977
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.24530

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Unveil: Unified Visual-Textual Integration and Distillation for Multi-modal Document Retrieval Hao Sun, Yingyan Hou, Jiayan Guo, Bo Wang, Chunyu Yang, Jinsong Ni, Yan Zhang Abstract Document retrieval in real-world scenarios faces significant challenges due to diverse document formats and modalities. Traditional text-based approaches rely on tailored parsing techniques that disregard layout information and are prone to errors, while recent parsing-free visual methods often struggle to capture fine-grained textual semantics in text-rich scenarios. To address these limitations, we propose Unveil , a novel visual-textual embedding framework that effectively integrates textual and visual features for robust document representation. Through knowledge distillation, we transfer the semantic understanding capabilities from the visual-textual embedding model to a purely visual model, enabling efficient parsing-free retrieval while preserving semantic fidelity. Experimental results demonstrate that our visual-textual embedding method surpasses existing approaches, while knowledge distillation successfully bridges the performance gap between visual-textual and visual-only methods, improving both retrieval accuracy and efficiency. 1 Introduction Document retrieval for real-world applications remains a challenging task due to the need to effectively handle diverse document formats, including text, images, charts, and complex visual layouts. As shown in Figure 1 , traditional document retrieval predominantly relies on Optical Character Recognition (OCR) to convert scanned or image-based documents into machine-readable text. Subsequently, approaches such as the lexical-based BM25 [ 23 ] and embedding-based techniques like Dense Passage Retrieval (DPR) [ 9 ] are utilized to model the semantic relevance between queries and documents. However, OCR-dependent pipelines come with significant limitations. They not only add computational overhead but also introduce potential recognition errors. Furthermore, these approaches often miss crucial visual contextual elements, which are essential for comprehending document content [ 34 , 5 , 15 ] . Figure 1: Comparison of document retrieval methods. Traditional approaches parse text and use text encoders for embeddings, while parsing-free methods directly process document screenshots with visual language models. Our visual-textual approach leverages both modalities, effectively addressing diverse scenarios. Recent research has shifted toward parsing-free techniques that directly utilize visual inputs such as document screenshots [ 5 , 15 , 35 , 20 , 14 , 12 ] . These methods leverage Vision-Language Models (VLMs) and Multi-Modal Large Language Models (MLLMs) to process entire pages directly, preserving rich structural and graphical information [ 3 , 32 , 29 ] . While these approaches circumvent the computational overhead and complexity associated with OCR, our empirical analysis reveals significant limitations in their textual understanding capabilities. As shown in Figure 2 , our comparison of text-based and visual-based methods across both text-rich scenarios (web-page retrieval) and visual-rich scenarios (visual document retrieval) reveals distinct performance patterns. In visual-rich scenarios where layout and graphical elements are crucial, these visual-based methods outperform traditional text-based approaches, highlighting their superior ability to process spatial and structural information [ 17 , 25 , 26 ] . However, when handling text-rich contexts, visual-based methods struggle to capture semantic details that text-based methods process effectively. This observation underscores a fundamental challenge: text-based methods excel in modeling linguistic semantics but overlook crucial layout and graphical details, while purely visual methods preserve visual context but struggle with fine-grained language understanding [ 5 , 34 , 20 , 15 ] . To address this limitation, we propose Unveil ( Un ified V isual-T e xt I ntegration and Disti l lation), a novel framework that bridges the gap between textual and visual document understanding. Our approach consists of two key components: First, we develop a visual-textual embedding approach that integrates both textual and visual inputs, leveraging the complementary strengths of both modalities for comprehensive document representations. Second, we conduct knowledge distillation to transfer semantic understanding from the teacher model (visual-textual embedding model) to the student model (purely visual model), enabling enhanced text comprehension without OCR dependency during inference. Specifically, we propose several techniques to facilitate this distillation process: (1) Representation Alignment: The student model is trained to replicate the teacher model’s representations by minimizing the distance between their query and document representations. (2) Soft Label Distillation: We utilize the teacher model to provide a fine-grained label distribution for the student model. (3) Adaptive Re-Weighting: We dynamically identify instances where discrepancies exist between the teacher and student models, assigning higher weights to these instances. Figure 2: Empirical analysis on retrieval performance under text-rich and visual-rich scenarios. Our framework offers a flexible retrieval system. For text-rich scenarios that require precise semantic nuances, the visual-textual model—which incorporates both textual and visual inputs—can be employed. Alternatively, in scenarios where efficiency or OCR-free processing is preferred, the distilled visual-only model serves as a practical alternative while maintaining comparable semantic understanding. We validate our approach on 12 datasets encompassing both text-rich and visual-rich scenarios. Experimental results indicate that the visual-textual embedding model consistently outperforms both text-based and visual-based methods. Furthermore, the distillation process effectively reduces the gap between visual-textual and visual-only approaches, enhancing retrieval accuracy and efficiency. In summary, our contributions are as follows: • We identify that existing text-based and visual-based methods struggle to adapt across different scenarios. To address this, we introduce Unveil, a visual-textual embedding approach that integrates textual and visual features for comprehensive document understanding. • We propose several knowledge distillation strategies to transfer the visual-textual model’s robust textual understanding to a purely visual model, enabling parsing-free retrieval without compromising accuracy. • Extensive experiments demonstrate that our visual-textual embedding method outperforms existing text-based and visual-based methods. Additionally, the knowledge distillation effectively reduces the gap between visual-textual and visual-only approaches, enhancing retrieval accuracy and efficiency. Figure 3: Unveil consists of: (a) a visual-textual embedding model that jointly processes document images and OCR text, and (b) a purely visual model that operates on document images only. During training, knowledge distillation is employed to transfer semantic understanding from the teacher (visual-textual) to the student (visual-only) model. At inference time, the framework offers flexibility to choose between the two models based on efficiency requirements. 2 Methodology Our proposed method seeks to bridge the gap between visual-textual and visual-only approaches. Initially, Unveil learns a textual-visual embedding model that leverages both OCR-derived text and visual inputs. Subsequently, it distills the strong capacity of the visual-textual embedding model into a purely visual model. The distilled visual model thus retains the semantic richness characteristic of visual-textual embeddings while achieving high efficiency without the need for textual input. 2.1 Task Definition Given a query q and a corpus \mathcal{C} comprising visual documents \{d_{1},d_{2},\ldots,d_{n}\} , the task of multi-modal document retrieval is to identify the k visual documents most relevant to the query q . Relevance is assessed using a similarity metric to measure the similarity between the query and document embeddings. Here, a visual document represents a complete information snippet (e.g., a web article or a PPT page), while the query is purely textual. 2.2 Unveil Framework As illustrated in Figure 3 , Unveil comprises two components: the teacher model (Visual-Textual model) and the student model (Visual-Only model). We initially train these models independently, followed by knowledge distillation to produce a visual-only model capable of robust document retrieval without OCR reliance during inference. Unified View of Retrieval Models Both the visual-textual and visual-only models employ a dual encoder architecture to model the similarity between queries and documents. The key distinction lies in the input

Read the complete text