Any Information Is Just Worth One Single Screenshot: Unifying Search With Visualized Information Retrieval
Ze Liu, Zhengyang Liang, Junjie Zhou, Zheng Liu, Defu Lian · 2025
Read Any Information Is Just Worth One Single Screenshot: Unifying Search With Visualized Information Retrieval by Ze Liu, Zhengyang Liang, Junjie Zhou, Zheng Liu, Defu Lian (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-11431.txt.
Introduction
Ze Liu and co-authors propose in 2025 that all information, text, tables, charts, and images alike, be searched as pictures. Their idea, visualized information retrieval, renders everything as screenshots in one common format. Search then works on the pictures themselves, whatever the source was.
The team built a large dataset of screenshots in captioned and question-answer forms. On it they trained a family of retrieval models that let screenshots search, or be searched by, any kind of content. A broad benchmark test showed existing multimodal retrievers struggle with this format, while their models improved results substantially. Code and data are shared to seed the new field.
Catalog record
| CREATOR | Ze Liu, Zhengyang Liang, Junjie Zhou, Zheng Liu, Defu Lian |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 64 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-11431
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Any Information Is Just Worth One Single Screenshot: Unifying Search With Visualized Information Retrieval Ze Liu, Zhengyang Liang, Junjie Zhou, Zheng Liu, Defu Lian Abstract With the popularity of multimodal techniques, it receives growing interests to acquire useful information in visual forms. In this work, we formally define an emerging IR paradigm called Visualized Information Retrieval , or Vis-IR , where multimodal information, such as texts, images, tables and charts, is jointly represented by a unified visual format called Screenshots , for various retrieval applications. We further make three key contributions for Vis-IR. First, we create VIRA (Vis-IR Aggregation), a large-scale dataset comprising a vast collection of screenshots from diverse sources, carefully curated into captioned and question-answer formats. Second, we develop UniSE (Universal Screenshot Embeddings), a family of retrieval models that enable screenshots to query or be queried across arbitrary data modalities. Finally, we construct MVRB (Massive Visualized IR Benchmark), a comprehensive benchmark covering a variety of task forms and application scenarios. Through extensive evaluations on MVRB, we highlight the deficiency from existing multimodal retrievers and the substantial improvements made by UniSE. Our work will be shared with the community, laying a solid foundation for this emerging field. Figure 1: A use case of Vis-IR. Users take a screenshot of their interested news by circling and selection, and search for relevant news reports associated with “Nvidia” based on a query conditioned on the screenshot. 1 Introduction Information retrieval has experienced tremendous progresses in recent years, driven by breakthroughs in foundation models. The growing capacity of language models has enabled precise and generalized text retrieval Xiao et al. 2024 ; Wang et al. 2022 ; Wang et al. 2023 , while the development in vision-language models has extended information retrieval to a broader range of data modalities Wei et al. 2024 ; Zhang et al. 2024a ; Zhou et al. 2024b . With the recent popularity of multimodal techniques Li et al. 2023 ; Liu et al. 2023 ; Team et al. 2023 , there is growing interest in uniformly representing multimodal data in visual forms and leveraging these visual representations for diverse information retrieval tasks (shown in Figure 1 ) Faysse et al. 2024 ; Ma et al. 2024 ; Zhang et al. 2024b . For example, researchers can circle and select a part of a paper, which may include texts, figures, equations, and figures, and search for relevant literature to address a specific question about the selected content. Similarly, people take a screenshot of an advertisement on their cellphones and retrieve descriptions of the corresponding product 1 1 1 https://blog.google/products/search/google-circle-to-search-android/ . These innovative paradigms will greatly enhance the flexibility of information retrieval in real-world scenarios, leading to a unified paradigm for search engines. In this paper, we formally define these problems as Visualized Information Retrieval , or Vis-IR for brevity. We also define the visual representation for a mixture of multimodal data as a screenshot , which is treated as a unified entity in the retrieval process. Despite that common multimodal retrievers can be applied for Vis-IR in a zero-shot manner, and there are preliminary works focused on improving representations of specific multimodal data, like screenshots of Wiki webpages Ma et al. 2024 , several fundamental challenges still persist for this emerging problem. Particularly, there are no tailored models which can offer unified, high-quality support for various Vis-IR applications. Besides, there is a lack of comprehensive benchmarks to evaluate a retriever’s performance on Vis-IR tasks. Finally, no specialized datasets are curated and published for training competent Vis-IR models. To address the above challenges, we present the following three resources, which are critical to the technical advancement of Vis-IR. First, we create a large-scale dataset, namely VIRA (Vis-IR, Aggregation), which comprises a vast collection of 13 million screenshot data from diverse sources, like webpages from news websites, shopping platforms, and Wikipedia, homepages from GitHub, research papers from ArXiv, general PDF documents, and various forms of charts. Each screenshot is assigned with a fine-grained caption by either extracting from meta data or annotating with sophisticated OCR tools. We further produce two types of question-answer data for the well-captioned screenshots: 1) q2s tuples, which comprises a screenshot and a query about the screenshot, 2) sq2s triples, which contain source screenshots, conditioned queries about the source, and target screenshots. Altogether, over 20 million data samples are included by VIRA. Second, we develop a family of embedding models, called UniSE (Universal Screenshot Embeddings), built on top of our created dataset. These models enable screenshots to either serve as queries or be queried by various data formats. For example, using a screenshot to search for an image or a document, or retrieving a screenshot based on a textual question. We adopt two alternative model architectures for UniSE: one based on CLIP Radford et al. 2021 , which is more efficient, the other one leveraging MLLMs Liu et al. 2023 , which is more expressive. This allows users to flexibly choose the model that best suits their own needs. Third, we construct MVRB (Massive Visualized IR Benchmark) to evaluate multimodal retrievers’ performance on general Vis-IR tasks. The benchmark includes various task types, such as screenshot-based multimodal retrieval (screenshot to anything, anything to screenshot) and screenshot-conditioned retrieval (e.g., searching for documents using queries conditioned on screenshots). It also covers a variety of important domains, including news, products, papers, and charts. We conduct a comprehensive experimental study using MVRB, which highlights significant deficiencies in existing methods and uncovers key factors influencing the development of Vis-IR retrievers. The results also demonstrate the effectiveness of UniSE in both in-domain and out-of-domain applications. To summarize, we define the Vis-IR problem, which holds promising prospects. We present three key resources: a large-scale dataset, powerful models, and a comprehensive benchmark, which are foundational to Vis-IR and will be shared with the public to facilitate technical advancement. Figure 2: Creation process of VIRA dataset, including 1) comprehensive screenshot collection from various sources, 2) fine-grained screenshot captioning, 3) similar screenshots mining, 4) q2s annotation, and 5) sq2s annotation. 2 Dataset: VIRA In this section, we present the VIRA (Vis-IR Aggregation) dataset, which offers diverse and large-scale training data for developing Vis-IR models. While creating the dataset, we emphasize two main factors: 1. comprehensiveness of screenshots, 2. utility of annotation. The creation’s outline is introduced as follows. Due to the space limitation, the entire details are provided in Appendix A . 2.1 Screenshot collection We perform massive collection of screenshot data spanning seven categories , which include News : from popular websites like BBC, CNN, and Fox, Products : from Amazon, Research Papers : from ArXiv, General Documents : from PDF Association, Charts : from ArXiv, Common Knowledge : from Wikipedia; Project Homepages : from GitHub. The data is evenly distributed across these categories, leading to 13 million screenshots in total. The collected data encompasses a rich variety of formats, showing diverse combinations of text, figures, and structures presented in various layouts. These features substantially enhance the generalization for models trained on corresponding data. 2.2 Screenshot Annotation We perform two types of annotations for the collected data. First, we assign each screenshot with a fine-grained caption , which paraphrases its detailed semantic. The caption is produced