When Vision Meets Texts in Listwise Reranking DOI: XXXXXXX.XXXXXXX Conference: Full Conference Name; Month DD–DD, 2025; City, Country ISBN: 978-1-4503-XXXX-X/2025/06 CCS: Information systems Retrieval models and ranking
Hongyi Cai · 2026
Read When Vision Meets Texts in Listwise Reranking DOI: XXXXXXX.XXXXXXX Conference: Full Conference Name; Month DD–DD, 2025; City, Country ISBN: 978-1-4503-XXXX-X/2025/06 CCS: Information systems Retrieval models and ranking by Hongyi Cai (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-20623.txt.
Introduction
Hongyi Cai studies listwise reranking for documents that mix images and text. Recent retrieval work shows promise in joining visual and textual signals. Yet reranking image-text documents stays hard, because of the modality gap and a scarcity of aligned datasets. Existing approaches often use very large models with reasoning-based distillation.
That size brings heavy compute cost while focusing mostly on text. Cai's work questions whether the visual side gets its due in reranking. Better multimodal reranking would sharpen search in commerce, media libraries, and any archive where pictures and words travel together.
Catalog record
| CREATOR | Hongyi Cai |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-20623
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
When Vision Meets Texts in Listwise Reranking DOI: XXXXXXX.XXXXXXX Conference: Full Conference Name; Month DD–DD, 2025; City, Country ISBN: 978-1-4503-XXXX-X/2025/06 CCS: Information systems Retrieval models and ranking Hongyi Cai Abstract . Recent advancements in information retrieval have highlighted the potential of integrating visual and textual information, yet effective reranking for image-text documents remains challenging due to the modality gap and scarcity of aligned datasets. Meanwhile, existing approaches often rely on large models (7B–32B parameters) with reasoning-based distillation, incurring unnecessary computational overhead while primarily focusing on textual modalities. In this paper, we propose Rank-Nexus , a multimodal image-text document reranker that performs listwise qualitative reranking on retrieved lists incorporating both images and texts. To bridge the modality gap, we introduce a progressive cross-modal training strategy. We first train modalities separately: leveraging abundant text reranking data, we distill knowledge into the text branch. For images, where data is scarce, we construct distilled pairs from multimodal large language model (MLLM) captions on image retrieval benchmarks. Subsequently, we distilled a joint image-text reranking dataset. Rank-Nexus achieves outstanding performance on text reranking benchmarks (TREC, BEIR) and the challenging image reranking benchmark (INQUIRE, MMDocIR), using only a lightweight 2B pretrained visual-language model. This efficient design ensures strong generalization across diverse multimodal scenarios without excessive parameters or reasoning overhead. Figure 1. Comparison of three reranking paradigms. (a) Pairwise reranking ( 23 ; 27 ) evaluates each passage independently against the query using an LLM to determine relevance with binary yes/no decisions. (b) Listwise reranking ( 22 ; 25 ) processes all candidate passages simultaneously, and directly outputs them in sorted ranking. (c) Our proposed Rank-Nexus approach extends listwise reranking to multimodal scenarios, jointly processing both textual passages and images to generate comprehensive rankings that leverage visual and textual information. Comparison of three reranking paradigms for information retrieval. 1. Introduction "A picture is worth a thousand words," – advertising executive Frederick R. Barnard. Large Language Models (LLMs) ( 5 ; 35 ; 36 ) have been proven effective document rerankers from the advantages of its strong context understanding and pre-training prior knowledge. Using LLMs to rank document with specific instructions is categorized into two downstream tasks: pairwise reranking ( 23 ; 27 ) and listwise reranking ( 22 ; 25 ) . Both settings in tandem leverages LLMs output and format them with patterns, which pairwise reranking focuses on single question relativity identification, while listwise reranking involves LLMs with long contexts to select the ranking for each document. However, most data today are documents mixed with tables, illustration and captions. While these approaches have advanced textual retrieval, their extension to multimodal documents that have both text and visual elements, remains fundamentally limited, as illustrated in Fig. 1 . There are several challenges of training a Vision-Language Model (VLM) reranker at the moment; First, the gap needs to be bridged between images and texts to understand each priority relevant to the query, as how the VLM model process multimodal data in the same embedding space to achieve reranking ( 32 ) . More radicially, since most popular methods of reranking models are based on text ( 19 ) , distilling from MS-MARCO ( 4 ) , ignoring visual content like illustration and tables, the datasets for both training and evaluating multimodal information retrieval are scarce, especially on listwise reranking, which focuses more on the overall quality of ranking results. Although it is still possible to re-rank documents without pre-training, the large burdening computational cost and inference time from listwise reranking of text results in inefficiency, let alone image involved reranking, nearly impossible to be deployed in production. Therefore, we found and aim to solve one question: Where can we seek the trade-off on reranking documents, each contains a vast amount of pictures and words? To address data issues, we propose a complete further data distillation pipeline along with efficient coreset selection on reranking training data, as well as integrating image-text multimodal documents from one of popular document retrieval dataset, MMDocIR ( 11 ) . We demonstrate that only the core part of reranking training data are significant to be involved, although the existing dataset RankGPT ( 22 ) have been extending to 10k scale. We also observe the diminishing return of dataset up-scaling and adopted greedy diversity-based searching strategy for fine-graining qualitative data to reduce unnecessary computation, which is agnostic to the modality–used in both text and images. Therefore, only 7.5% text information retrieval and image information retrieval (1000 pairwise samples + 2100 listwise samples) are used during training. Building upon this foundation of high-quality multimodal ranking data, we introduce a progressive training strategy to train our VLM reranker model, Rank-Nexus , which systematically transitions the model from text-only to multimodal listwise reranking. This curriculum progresses through three stages: the model first learns fundamental ranking on pairwise text data, then acquires basic visual-text alignment through pairwise image ranking, and finally masters global document ordering via listwise image ranking. Crucially, our approach departs from currently trending reasoning-based rerankers, Rank-R1 ( 44 ) built upon 7B+ LLMs, whose extensive inference overhead often negates their marginal relevance gains. Instead, our compact 2B multimodal reranker operates through direct forward computation without expensive reasoning chains ( 15 ) , achieving superior inference efficiency—completing MSMARCO passage reranking in just 2 seconds compared to 8.92 seconds (RankZephyr ( 26 ) ) and 16 seconds (RankR1)—while simultaneously attaining higher ranking accuracy, as quantitatively demonstrated in Fig. 2 . This synergistic combination of progressive multimodal training, lightweight architecture, and efficient inference enables state-of-the-art reranking performance with dramatically reduced computational requirements across both training and deployment. We conduct the extensive experiment to verify the effectiveness of Rank-Nexus ranking on both text ranking benchmark (DL19 ( 8 ) , DL20 ( 9 ) ) and image ranking benchmark (INQUIRE ( 37 ) , MMDocIR ( 11 ) ). Our 2B model outperforms the majority of open-source distilled instruction models. Our contribution can be summarized below: • We introduce Rank-Nexus, the lightweight model for end-to-end listwise reranking of mixed image-text documents, directly optimizing global ordering in a unified framework. • We develop a data curation and progressive training strategy that achieves strong performance using minimal high-quality data, reducing training costs. • Our model enables fast inference through direct computation, significantly outperforming reasoning-based rerankers in speed while maintaining competitive accuracy on benchmarks. Figure 2. The effectivenss (left) and latency (right) of Rank-Nexus and other baseline method on document listwise reranking inference. We experimented and illustrated DL-19 benchmark for evaluation and one of distilled MSMARCO dataset as inference time example. Rank-R1 takes most latency among 3 models, strongly dependent on reasoning time. While Rank-Nexus are the fastest as it possesses smallest parameter size and outputs without ad-hoc thinking. 2. Related Work 2.1. Document Reranking Document reranking has evolved from traditional learning-to-rank methods ( 20 ) to modern neural approaches leveraging pre-trained language models ( 19 ) . Early neural rerankers employed direct scoring based on points ( 23 ; 45 ) that evaluates each document independently, while pairwise methods ( 24 ; 27 ) compare document pairs for relative relevance. Recent advances focus on listwise reranking ( 22 ; 25 ) , which processes entire candidate lists simultaneously to optimize global ranking quality.