SORCE: Small Object Retrieval in Complex Environments

Chunxu Liu, Chi Xie, Xiaxu Chen, Wei Li, Feng Zhu, Rui Zhao, Limin Wang · 2025

Read SORCE: Small Object Retrieval in Complex Environments by Chunxu Liu, Chi Xie, Xiaxu Chen, Wei Li, Feng Zhu, Rui Zhao, Limin Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-24441.txt.

Introduction

Chunxu Liu and six co-authors set a new search task: find a small object in a busy photo from a text query. Most photo search tests dwell on the big, clear subject of a scene. Real needs often target the small thing in the corner. Their 2025 work names that gap and builds a test for it.

Their test set holds a thousand scenes packed with detail, and queries that point at minor objects with few hints. Known methods did poorly, since they squeeze a whole scene into one code. The team instead codes each photo in several parts, guided by region prompts from a large vision-language model. That split code beat all rivals on the new test. The work opens a fresh track for photo search.

Catalog record

CREATORChunxu Liu, Chi Xie, Xiaxu Chen, Wei Li, Feng Zhu, Rui Zhao, Limin Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-24441

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SORCE: Small Object Retrieval in Complex Environments Chunxu Liu, Chi Xie, Xiaxu Chen, Wei Li, Feng Zhu, Rui Zhao, Limin Wang Abstract Text-to-Image Retrieval (T2IR) is a highly valuable task that aims to match a given textual query to images in a gallery. Existing benchmarks primarily focus on textual queries describing overall image semantics or foreground salient objects, possibly overlooking inconspicuous small objects, especially in complex environments. Such small object retrieval is crucial, as in real-world applications, the targets of interest are not always prominent in the image. Thus, we introduce SORCE ( S mall O bject R etrieval in C omplex E nvironments), a new subfield of T2IR, focusing on retrieving small objects in complex images with textual queries. We propose a new benchmark, SORCE-1K, consisting of images with complex environments and textual queries describing less conspicuous small objects with minimal contextual cues from other salient objects. Preliminary analysis on SORCE-1K finds that existing T2IR methods struggle to capture small objects and encode all the semantics into a single embedding, leading to poor retrieval performance on SORCE-1K. Therefore, we propose to represent each image with multiple distinctive embeddings. We leverage Multimodal Large Language Models (MLLMs) to extract multiple embeddings for each image instructed by a set of Regional Prompts (ReP). Experimental results show that our multi-embedding approach through MLLM and ReP significantly outperforms existing T2IR methods on SORCE-1K. Our experiments validate the effectiveness of SORCE-1K for benchmarking SORCE performances, highlighting the potential of multi-embedding representation and text-customized MLLM features for addressing this task. 1 Introduction Figure 1 : Instances from different retrieval benchmarks. We draw the bounding boxes of the referred objects in the image. While traditional Text-to-Image Retrieval (T2IR) focuses on retrieving images that match a given textual query, an important yet often overlooked challenge is retrieving small objects within complex scenes. In many real-world applications, the target of interest is not the dominant subject, but a small, inconspicuous object in a cluttered background. Despite significant advancements in T2IR methods [ 25 , 38 , 43 , 39 ] , existing frameworks struggle with such fine-grained retrieval, highlighting the need for dedicated research. The challenge of retrieving small objects in complex environments with textual queries is highly relevant to various real-world applications. In video surveillance, it is essential for discovering inconspicuous yet critical items, such as concealed weapons in crowded areas or abandoned luggage in transportation hubs; in smart city systems, it helps detect traffic violations (e.g., unlicensed vehicles) and monitor infrastructure integrity (e.g., cracks in bridges). In cybersecurity, it can help discover tampered watermarks and manipulated content; in consumer applications, it enables users to locate specific small objects within personal photo collections efficiently. Despite the importance of small object retrieval, existing T2IR benchmarks primarily focus on retrieving images based on holistic descriptions rather than identifying fine-grained, non-salient objects. Popular T2IR benchmarks like Flickr30K [ 36 ] and COCO [ 15 ] highlights the main object or the whole picture in the image. Other datasets with image-text pairs [ 23 , 2 ] provide more detailed descriptions, but still focus on describing the full image rather than specific small objects. To address this gap, we introduce S mall O bject R etrieval in C omplex E nvironments ( SORCE ), a task focusing on retrieving the target image containing a specific small object. Here, small objects refer to the objects that are visually and resolution-wise small in the image, not semantically small. For this task, we construct the SORCE-1K benchmark by collecting images from the SA-1B dataset [ 44 ] . As illustrated in Fig. 1 , it is a high-resolution segmentation dataset with 1B mask annotations across 11M images, guaranteeing a high level of environment complexity. For each image, we identify a small target object and carefully craft a descriptive caption that uniquely specifies it, minimizing ambiguity within the benchmark. The resulting dataset, SORCE-1K , is a comprehensive benchmark for evaluating small object retrieval in complex environments. In image retrieval, candidate images are typically pre-encoded into features, and retrieval is performed based on feature similarity with the query. Consequently, the image feature extraction is independent of the query content. This presents a significant challenge for feature extraction to capture every fine-grained detail within a single feature representation, making small object retrieval particularly difficult. Especially, the objects of interest in our SORCE-1K are often non-salient and embedded in complex scenes. Thus, relying on a single image feature for SORCE is likely unsatisfying. To tackle the challenges of the SORCE task, we propose a simple yet effective starting point: representing an image with multiple features instead of a single embedding. Conventional feature extractors tend to prioritize the most salient objects, often overlooking fine-grained details. Extracting multiple feature representations can help mitigate this issue by capturing different aspects of an image. This can be naturally achieved using feature extractors based on Multimodal Large Language Models (MLLMs). Unlike traditional encoders, MLLMs can generate different feature embeddings based on different prompts [ 31 , 8 , 42 ] , allowing them to focus on different aspects of the image while preserving global context. By leveraging this property, we can obtain a more comprehensive representation that improves retrieval performance for small, non-salient objects in complex environments. Our main contributions are summarized as follows: 1. We introduce SORCE, a new subfield of Text-to-Image Retrieval (T2IR) that addresses the challenge of retrieving small, less conspicuous objects within images with complex environments based on textual queries. 2. We develop a new benchmark, SORCE-1K, for facilitating research on the SORCE task. SORCE-1K comprises 1,023 carefully curated images with complex backgrounds and textual queries that describe less prominent small objects with minimum surrounding context. 3. We propose an MLLM-based multi-embeddings approach, with Regional Prompts (ReP) to extract multiple image features focusing on different aspects. Superior performances over existing T2IR methods on SORCE-1K highlight the potential of multi-embedding representation and text-guided MLLM features for addressing the SORCE task. 2 Related Work 2.1 Text-to-Image Retrieval Benchmarks Text-to-Image Retrieval (T2IR) aims to retrieve relevant images from a candidate pool based on a given textual query. In practice, the candidate pool consists of pre-extracted image features, agnostic of the query content. The common approach is to first extract the text query embedding, then compute the cosine similarity between this text embedding and the embeddings of the candidate images. The top-ranked images with the highest similarity scores are retrieved as the final results. Common benchmarks such as Flickr30K [ 36 ] and COCO [ 15 ] provide five brief captions for each image, describing the main object or event. Other datasets such as ShareGPT4V [ 2 ] and DOCCI [ 23 ] further extend the captions of images to be more detailed descriptions. Urban-1K [ 40 ] is another retrieval dataset for urban images with detailed descriptions. However, all of the above retrieval benchmarks are dedicated to enriching the query to be a detailed description of a whole image or the obvious objects in the image. In contrast, our benchmark SORCE-1K introduces a novel setting, which only contains the minimum required context in the query description. This approach aims to prevent retrieval models from taking shortcuts by recognizing and relying on non-target contextual

Read the complete text