LITTA: Late-Interaction and Test-Time Alignment for Visually-Grounded Multimodal Retrieval

Seonok Kim · 2026

Read LITTA: Late-Interaction and Test-Time Alignment for Visually-Grounded Multimodal Retrieval by Seonok Kim (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-26683.txt.

Introduction

Kim presents LITTA, a retrieval framework that finds evidence pages inside visually rich documents. Textbooks, reports, and manuals mix long text with complex layouts, and user questions often share few words with the right page. LITTA asks a large language model for extra query variants, then searches with a frozen vision retriever for each one. Reciprocal rank fusion merges the candidate lists into one ranking. Across computer science, drug, and industrial manual sets, the multi-query route lifts top-k accuracy, recall, and MRR.

Catalog record

CREATORSeonok Kim
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40539
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.26683

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LITTA: Late-Interaction and Test-Time Alignment for Visually-Grounded Multimodal Retrieval Seonok Kim Abstract Retrieving relevant evidence from visually rich documents such as textbooks, technical reports, and manuals is challenging due to long context, complex layouts, and weak lexical overlap between user questions and supporting pages. We propose LITTA, a query-expansion-centric retrieval framework for evidence page retrieval that improves multimodal document retrieval without retriever retraining. Given a user query, LITTA generates complementary query variants using a large language model and retrieves candidate pages for each variant using a frozen vision retriever with late-interaction scoring. Candidates from expanded queries are then aggregated through reciprocal rank fusion to improve evidence coverage and reduce sensitivity to any single phrasing. This simple test-time strategy significantly improves retrieval robustness while remaining compatible with existing multimodal embedding indices. We evaluate LITTA on visually grounded document retrieval tasks across three domains: computer science, pharmaceuticals, and industrial manuals. Multi-query retrieval consistently improves top- k k accuracy, recall, and MRR compared to single-query retrieval, with particularly large gains in domains with high visual and semantic variability. Moreover, the accuracy–efficiency trade-off is directly controllable by the number of query variants, making LITTA practical for deployment under latency constraints. These results demonstrate that query expansion provides a simple yet effective mechanism for improving visually grounded multimodal retrieval. 1 Introduction Retrieving relevant evidence from visually rich documents remains a challenging problem in multimodal information retrieval. Documents such as textbooks, technical manuals, and scientific reports often contain complex layouts, diagrams, tables, and long contextual dependencies. As a result, evidence relevant to a user query may appear in visually structured regions of a page rather than in plain text passages. Recent multimodal retrieval systems address this challenge by encoding document pages with vision-language models and performing retrieval over page-level embeddings. In this work, we emphasize evidence page retrieval: the retrieval target is not merely the correct document, but the specific page(s) that contain the supporting table, figure, or procedure. This page-level view is crucial for visually grounded downstream reasoning, because the decisive evidence is often localized and cannot be recovered from neighboring pages without retrieving it first. Evidence page retrieval is difficult not only because of visual complexity, but also because the “right” page is frequently described in ways that are indirect from the user’s perspective. A query may refer to an outcome (e.g., a failure mode), while the document uses part numbers and schematic labels; a question about a regulation may be answered by a table header rather than a paragraph; and a procedural question may be supported by a diagram plus a short caption. In these cases, retrieval must bridge a gap between the user-facing phrasing and the document’s internal terminology and visual conventions. Despite these advances, most retrieval systems rely on a single query representation to retrieve candidate pages. However, user questions often underspecify the relevant visual evidence, especially in domains with weak lexical overlap between the query and the document content. For example, a question about a technical concept may correspond to figures, diagrams, or descriptions that use different terminology or visual explanations. As a result, single-query retrieval may fail to retrieve relevant evidence pages even when the underlying retriever is strong. Visually rich documents introduce failure modes that are not well captured by single-shot querying. First, user terminology often mismatches the phrasing used in manuals and regulatory reports, including abbreviations, part names, and compliance jargon. Second, evidence frequently appears in tables and figures, so the key semantics may be only weakly expressed in surrounding text. Third, a relevant page may support the answer implicitly through diagrams or parameter tables, leaving few lexical cues even when the page is decisive evidence. Late-interaction multi-vector retrievers mitigate some of these issues by preserving fine-grained token–region interactions via ColBERT-style MaxSim scoring [ 18 ] . Yet even strong retrievers can be brittle when constrained to a single query: the query may fail to express the right terminology or constraints needed to surface the evidence page. In this work, we propose LITTA, a retrieval framework that improves multimodal document retrieval through query expansion. Given a user query, LITTA generates a small set of complementary query variants that preserve the original intent while emphasizing alternative terminology, constraints, and contextual cues. Each query variant retrieves candidate pages using a frozen late-interaction vision retriever, and we fuse the ranked lists across variants using Reciprocal Rank Fusion (RRF) [ 8 ] to promote pages that are consistently ranked highly across variants. This test-time strategy increases the likelihood that at least one query variant aligns with the evidence page, improving coverage without retriever retraining. While multi-query retrieval increases online cost roughly linearly in the number of variants, small values (e.g., Q{=}3 ) provide a strong accuracy–efficiency trade-off. Our pipeline is designed for practical deployment. Page embeddings are precomputed offline and stored in a compact index, while online computation scales primarily with the number of query variants and per-variant candidate depth. The retrieval cost is therefore controllable, enabling a favorable accuracy–efficiency trade-off in real-world settings. Beyond improving raw retrieval metrics, we view query expansion as a robustness mechanism for visually grounded retrieval. When evidence is concentrated in visually dense regions (tables, plots, schematics), minor lexical shifts can substantially change which page regions are activated under late interaction. By issuing a small set of diverse but faithful variants, LITTA reduces sensitivity to any single phrasing and instead aggregates consistent retrieval signals across variants. We evaluate LITTA on visually grounded document retrieval tasks across three domains: computer science, pharmaceuticals, and industrial manuals. Our experiments show that multi-query retrieval consistently improves top- k accuracy, recall, and ranking metrics compared to single-query baselines. In particular, gains are larger in industrial manuals where terminology is heterogeneous and evidence is frequently embedded in figures and procedural tables, while improvements remain consistent in textbooks and pharmaceutical reports. Our contributions are summarized as follows: • We propose LITTA, a simple query-expansion framework for improving multimodal document retrieval without retriever retraining. • We introduce a multi-query retrieval strategy that pools candidates across query variants using late-interaction scoring and robust rank-level aggregation. • Extensive experiments across three visually rich document domains demonstrate consistent improvements over strong multimodal retrieval baselines, with particularly large gains in terminology- and layout-diverse manuals, and highlight practical accuracy–cost trade-offs. 2 Related Work 2.1 Visual Document Retrieval and Late Interaction Visually rich document retrieval requires grounding queries to page-level evidence that may be concentrated in localized regions such as tables, figures, or diagrams. Recent work has explored model- and system-level strategies for cross-modal retrieval and document understanding, spanning generative or coarse-to-fine retrieval formulations and unified visual–textual integration for document retrieval [ 11 , 40 , 27 ] . Parallel progress in long-document and multi-page understanding further highlights the need for robust evidence localization under complex layouts and long contexts [ 16 , 38 , 42 , 10 ] . Late-interaction multi-vector retrieval is particularly well-suited to this setting because it preserves token/region-level interactions rather than collapsing pages into a single embedding. Strong late-interaction embedding models for visual document retrieval, such as Nemotron ColEmbed V2 [ 9 ] , enable efficient MaxSim-style matching and fine-grained evidence alignment. More broadly, structured retrieval and reasoning pipelines based on graphs and multi-agent systems

Read the complete text