Pixel-Grounded Retrieval for Knowledgeable Large Multimodal Models

Jeonghwan Kim, Renjie Tao, Sanat Sharma, Jiaqi Wang, Kai Sun, Zhaojiang Lin, Seungwhan Moon, Lambert Mathias, Anuj Kumar, Heng Ji, Xin Luna Dong · 2026

Read Pixel-Grounded Retrieval for Knowledgeable Large Multimodal Models by Jeonghwan Kim, Renjie Tao, Sanat Sharma, Jiaqi Wang, Kai Sun, Zhaojiang Lin, Seungwhan Moon, Lambert Mathias, Anuj Kumar, Heng Ji, Xin Luna Dong (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-19060.txt.

Introduction

Jeonghwan Kim and ten co-authors teach a visual model to fetch facts tied to the exact spot in a photo. Answering questions about images often needs both sharp sight and outside facts. Past systems that fetch facts lack an inner rule for when to fetch and how to aim the search.

Their model, PixSearch, joins region level sight with fetch based reasoning in one end to end system. It can point at the part of the image the question is about, then pull facts for that part. Grounded fetching could cut the made up answers that plague visual question answering today.

Catalog record

CREATORJeonghwan Kim, Renjie Tao, Sanat Sharma, Jiaqi Wang, Kai Sun, Zhaojiang Lin, Seungwhan Moon, Lambert Mathias, Anuj Kumar, Heng Ji, Xin Luna Dong
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE62 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-19060

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Pixel-Grounded Retrieval for Knowledgeable Large Multimodal Models Jeonghwan Kim, Renjie Tao, Sanat Sharma, Jiaqi Wang, Kai Sun, Zhaojiang Lin, Seungwhan Moon, Lambert Mathias, Anuj Kumar, Heng Ji, Xin Luna Dong Abstract Visual Question Answering (VQA) often requires coupling fine-grained perception with factual knowledge beyond the input image. Prior multimodal Retrieval-Augmented Generation (MM-RAG) systems improve factual grounding but lack an internal policy for when and how to retrieve. We propose PixSearch , the first end-to-end Segmenting Large Multimodal Model (LMM) that unifies region-level perception and retrieval-augmented reasoning. During encoding, PixSearch emits <search> tokens to trigger retrieval, selects query modalities (text, image, or region), and generates pixel-level masks that directly serve as visual queries, eliminating the reliance on modular pipelines (detectors, segmenters, captioners, etc.). A two-stage supervised fine-tuning regimen with search-interleaved supervision teaches retrieval timing and query selection while preserving segmentation ability. On egocentric and entity-centric VQA benchmarks, PixSearch substantially improves factual consistency and generalization, yielding a 19.7% relative gain in accuracy on CRAG-MM compared to whole image retrieval, while retaining competitive reasoning performance on various VQA and text-only QA tasks. † † affiliation: Meta Reality Labs † † affiliation: University of Illinois Urbana-Champaign † † contribution: Work done at Meta † † date: August 10, 2026 † † correspondence: First Author at [email protected], [email protected] † † Code: https://github.com/wjdghks950/PixSearch 1 Introduction Figure 1: Egocentric images from wearables devices often render entities smaller than it appears because of wide-angle cameras. MM-RAG methods that rely on full-image search or caption-only queries can introduce retrieval noises and degrade QA quality. PixSearch , an end-to-end segmenting LMM, learns when to issue a query, how to route among text, whole-image, and region-level queries, and how to reason over retrieved evidence for answer generation. Our work also compares against pipeline, tool-based approaches. On CRAG-MM ( 47 ) , PixSearch (full) improves accuracy by 26% and reduces hallucination by 39%. Entity-centric visual question answering (VQA) sits at the nexus of perception and reasoning: it demands recognizing specific entities in an image, leveraging factual knowledge about those entities, and when needed, composing related evidence to answer the question. VQA on egocentric images from wearable devices such as smart glasses is even harder: as illustrated in Figure 1 , wide-angle viewpoints render entities small, and the entities themselves are often long-tail or niche, making them unlikely to be reliably covered by an LLM’s internal knowledge. Multimodal Retrieval-Augmented Generation (MM-RAG) has strengthened factual grounding in VQA ( 30 ; 26 ; 17 ) , but two limitations persist. First, most MM-RAG systems either retrieve with the full image ( 41 ; 30 ; 53 ; 51 ; 55 ; 13 ; 43 ) , or use text-only queries that simply paraphrase the image ( 32 ; 10 ; 9 ; 38 ) . Full-image retrieval pulls in distracting background, while text-only cues (e.g., “car”) lack the specificity needed for fine-grained entity grounding and enrichment. Second, MM-RAG pipelines are often modular—detectors, segmenters, captioners, etc.—to form queries, thus can introduce cross-modal translation errors, struggle with composing multiple queries, and add latency when retrieval is not truly necessary. We present PixSearch , the first end-to-end framework for retrieval-augmented reasoning. During generation, PixSearch (i) learns when to retrieve by emitting <search> tokens, (ii) decides how to retrieve by routing among text, whole-image, and region-level queries via token outputs, and (iii) grounds answers in the retrieved evidence, supporting multi-step search. Built on segmenting LMMs (Large Multi-modal Models with segmentation capabilities), PixSearch natively produces segmentation masks without external detection/segmentation APIs, and uses these masks directly as retrieval queries. This yields pixel-level, context-aware grounding that surpasses modular, text- or tool-driven pipelines. Integrating the aforementioned capabilities, nonetheless, is non-trivial. It either requires reinforcement learning (RL)-based tuning as in previous work ( 18 ) or requires a supervised finetuning (SFT) dataset to teach the model such behaviors. Nonetheless, the field currently lacks such data that interleaves retrieval triggering, query type assignment and reasoning into a single model output sequence. To this end, we propose an effective two-stage supervised finetuning strategy, together with a training data construction pipeline. We leverage diverse VQA datasets ( 45 ; 7 ; 15 ; 47 ; 5 ; 31 ; 40 ) to teach the model to trigger retrieval only when needed and to select appropriate query types, enabling effective multimodal RAG for entity-centric VQA while preserving segmentation performance. In summary, our paper makes the following three contributions. 1. Framework: We introduce PixSearch , the first end-to-end segmenting LMM that autonomously triggers retrieval and performs region-level grounding. 2. Training : We devise a two-stage supervised fine-tuning regimen and a multimodal training dataset that teaches when to retrieve and how to form text, image, or region queries via search-interleaved trajectories, and meanwhile preserving segmentation quality. 3. Experiments : Through comprehensive experimental results, we demonstrate that PixSearch substantially improves factual consistency and generalization across egocentric and entity-centric VQA benchmarks, achieving 19.7% accuracy improvement in CRAG-MM ( 47 ) , and in particular 24.3% improvement on egocentric images (Figure 1 ). 2 Related Work 2.1 Knowledge-based Visual Question Answering Prior research in knowledge-based visual question answering (KB-VQA) has recognized the need to go beyond parametric model knowledge by incorporating external retrieval modules. Previous work such as KRISP ( 30 ) combined implicit knowledge from vision-language transformers with explicit symbolic knowledge retrieved for detected objects. Similarly, KAT ( 12 ) leveraged retrieved knowledge snippets aligned to object regions for VQA, and MAVEx ( 50 ) proposed answer-conditioned retrieval to validate candidate answers with external evidence. These systems demonstrated that combining visual inputs with external knowledge improves factual grounding, but they largely rely on text-based representations of detected entities, e.g., object tags or noun phrases, as the query. More recent approaches ( 26 ) incorporated object detector outputs as region-specific queries, or leveraged LLMs to extract referring expressions for visual entity grounding ( 17 ) , showing that retrieving knowledge about individual detected objects significantly outperforms whole-image retrieval. While these approaches highlight the promise of region-centric retrieval, they nonetheless depend on external object detectors, separate segmentation modules, or hand-engineered pipelines. 2.2 Multimodal Retrieval-Augmented Generation. In parallel, retrieval-augmented generation (RAG) has been extended to multimodal domains. EchoSight ( 51 ) combined visual search with textual retrieval to answer encyclopedic VQA, while VisRAG ( 55 ) introduced document-level multimodal retrieval for visually rich pages. RA-CM3 ( 54 ) unified text and image retrieval for generation, and frameworks like UniIR/M-BEIR ( 49 ) benchmarked multimodal retrieval performance. While these works demonstrate the benefits of grounding multimodal models in external knowledge, their queries are typically either text-only or whole-image embeddings, limiting their precision when a specific entity in the scene is most relevant. 2.3 Search Trigger and Interleaving External Knowledge Another open challenge is teaching models not just to use retrieval, but to decide when retrieval is needed and how to query. Self-RAG ( 2 ) introduced reflection tokens for adaptive retrieval, Toolformer ( 39 ) trained LLMs to call external APIs at the right time, and RePlug ( 42 ) optimized retrievers based on LM likelihood. More recent line of work, Search-R1 ( 18 ) uses PPO/GRPO to enable LLMs to interleave search with auto-regressive reasoning. These methods show that adaptive retrieval policies can

Read the complete text